From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from localhost (localhost [127.0.0.1]) by arlo.cworth.org (Postfix) with ESMTP id E3A5C6DE0183 for ; Thu, 28 Dec 2017 18:01:01 -0800 (PST) X-Virus-Scanned: Debian amavisd-new at cworth.org X-Spam-Flag: NO X-Spam-Score: 0 X-Spam-Level: X-Spam-Status: No, score=0 tagged_above=-999 required=5 tests=[AWL=0.011, SPF_PASS=-0.001, T_RP_MATCHES_RCVD=-0.01] autolearn=disabled Received: from arlo.cworth.org ([127.0.0.1]) by localhost (arlo.cworth.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id qobbnrYRPz24 for ; Thu, 28 Dec 2017 18:01:00 -0800 (PST) Received: from fethera.tethera.net (fethera.tethera.net [198.245.60.197]) by arlo.cworth.org (Postfix) with ESMTPS id AFE9C6DE014D for ; Thu, 28 Dec 2017 18:01:00 -0800 (PST) Received: from remotemail by fethera.tethera.net with local (Exim 4.89) (envelope-from ) id 1eUjyj-0003sO-Nd; Thu, 28 Dec 2017 21:00:57 -0500 Received: (nullmailer pid 3420 invoked by uid 1000); Fri, 29 Dec 2017 02:00:56 -0000 From: David Bremner To: David Edmondson , notmuch@notmuchmail.org Subject: Re: Xapian exception leading to database corruption In-Reply-To: References: Date: Thu, 28 Dec 2017 22:00:56 -0400 Message-ID: <87608q1eef.fsf@tethera.net> MIME-Version: 1.0 Content-Type: text/plain X-BeenThere: notmuch@notmuchmail.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Use and development of the notmuch mail system." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 29 Dec 2017 02:01:02 -0000 David Edmondson writes: > Using current git notmuch on Debian testing a rebuild from scratch of my > database fails: > >> agrajag-testing ~/s/notmuch % ./notmuch new >> Found 605510 total files (that's not much mail). >> add_file: A Xapian exception occurred 28m 37s remaining). >> A Xapian exception occurred adding message: Unexpected end of posting list for 'G0000000000014364'. >> Processed 137296 total files in 8h 4m 45s (4 files/sec.). >> Added 135950 new messages to the database. >> Note: A fatal error was encountered: A Xapian exception occurred >> agrajag-testing ~/s/notmuch % > I can't duplicate this (probably not surprising) on debian testing. I also have about 600k files, and I'm running debian testing. My total index time is about 1h on a not-very-recent machine with an SSD. I'm guessing you have a hard disk (or something is deeply wrong to take that long). Even for a hard disk, 8h to index 137k files seems slow. Did you happen to check for disk errors? > I see that bremner reported something like this in #xapian, but not any > resolution. I guess your best bet is to write to xapian-discuss@lists.xapian.org > > Any suggestions? Is it possible to force a new chert database to be > created rather than glass? (Mostly I'd like to get back to work!) According to Olly Betts, With 1.4 you can pass Xapian::DB_BACKEND_CHERT in the flags when constructing the WritableDatabase object. So you'd have to hack the notmuch source, I'm guessing around line 55 of database.cc #define DB_ACTION (Xapian::DB_CREATE_OR_OPEN | Xapian::DB_RETRY_LOCK | Xapian::DB_BACKEND_CHERT) Do move the existing database out of the way, or apparently xapian can get confused.