From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from localhost (localhost [127.0.0.1]) by arlo.cworth.org (Postfix) with ESMTP id 0ADA26DE022C for ; Sun, 29 Apr 2018 03:22:52 -0700 (PDT) X-Virus-Scanned: Debian amavisd-new at cworth.org X-Spam-Flag: NO X-Spam-Score: -0.594 X-Spam-Level: X-Spam-Status: No, score=-0.594 tagged_above=-999 required=5 tests=[AWL=0.116, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_LOW=-0.7, SPF_PASS=-0.001, T_RP_MATCHES_RCVD=-0.01] autolearn=disabled Received: from arlo.cworth.org ([127.0.0.1]) by localhost (arlo.cworth.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Cgfc2kCBXSVP for ; Sun, 29 Apr 2018 03:22:50 -0700 (PDT) X-Greylist: delayed 325 seconds by postgrey-1.36 at arlo; Sun, 29 Apr 2018 03:22:50 PDT Received: from mout.gmx.net (mout.gmx.net [212.227.15.18]) by arlo.cworth.org (Postfix) with ESMTPS id 32D0B6DE01FF for ; Sun, 29 Apr 2018 03:22:49 -0700 (PDT) Received: from len.workgroup ([46.94.156.122]) by mail.gmx.com (mrgmx002 [212.227.17.190]) with ESMTPSA (Nemesis) id 0LtaDM-1eC0M01XtM-010veS; Sun, 29 Apr 2018 12:17:20 +0200 From: Gregor Zattler To: notmuch@notmuchmail.org Cc: xapian-discuss@lists.xapian.org Subject: Re: Database corruption after clean rebuild In-Reply-To: <5bde652e-8ffe-d43d-106d-a43c7a1b4126@gmail.com> References: <8a311263-a46f-3327-4054-d8d863fa0169@gmail.com> <874lknwztk.fsf@tethera.net> <87y3hzuw5z.fsf@tethera.net> <5bde652e-8ffe-d43d-106d-a43c7a1b4126@gmail.com> Mail-Followup-To: notmuch@notmuchmail.org, xapian-discuss@lists.xapian.org Date: Sun, 29 Apr 2018 12:17:12 +0200 Message-ID: <87po2iqpuf.fsf@len.workgroup> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Provags-ID: V03:K1:+6QU0MvU+NKVSeEx8FBr7My47uOJ7Qn+afVOmuotYjAfWI3Qz0t pMHmCSrS48IcQsZH4kg4LdCM46iFc2YolCAzubQkLm1CRmDrQoqKNJRGDSgEzNr699I6f3k gy0tpQ5RTGk+n3Z/JsEpKwtGxUFGQPe8Flaq6NuIRVqkLJa4x3WvIFlRF9hqKLlbi0O1lgw CMMKaJSiFWk1VDR0I3X/A== X-UI-Out-Filterresults: notjunk:1;V01:K0:XKkH+lhk+pE=:Tfg5pRR/sc8GwGadDxxO3K zCZ0cHiX3VtiKYYfIBy0Q0F+n1mGMtXmm+6OLphJWSaegdnMYq44JZt9DPhrv7GdWmreixvFP kTrYRhlMpRb6kLzkKmHzOWukxZfluz9YlIo3VTd2NHprmIbAEnLBH9Ap9JHdntBl4626zNh9k Bq+ezM7FIE0Ka1B36G9LmAkKkEreAQqxzBGuHf0z49AsoX6QA1lBdRpLf/QOp4701fLzZAKlI wHiNamOaiiRNdct/S/FBflwqJFsMplT6qA8AVmveMKcNIcJiIYhEwKcnNU1BsFs2VzN287PcT zYsmjclo0ba5DzBZh6gFyQ+Lw/PhO8uDoZEyrQb/mJTAVI02et3a614tqZ4dOVOIyz+rmk8ol 3Z33YanM4KYx2im+5UVZLi+1rtpcpZQHZp+6rH/Ke7zHRPYqc234KlvvIkdQwE6KSk083j2Eh 6Ljx2cuNSINIkJwLQ2UlR5WPvBe/pvAZhPmXi9zBSUvMJRun/a1yS5+fZ5u2JV4UqAeNJIRfv G30+YnAZWy2KI6QZwXcMsdAF/8tUTwn8u9JRIeis1MEkWahcuvb9rS0l+B++jNnkbi+nsnHVR s35kdAG87wnF4gdo3sM+OfctYTLhUqOad9D6t7wx95ZAKdlC5IWwGRx/7HETV7v7IWxJg1ugF Yi3vtYvBrKRBAWTXxyp50keHJa4GHGBnIiRD40VoZnHqTs2eiL2cQf0AVpLWd1i/Sb6Qu9+ql a82PhL338e73rQzQ7KHbQHRHfPr+8AP4ivgf0Iu4fe3WK1OjTYTSlSw7P/PU5ws/+B1h/A0Kw mFOwsRI X-BeenThere: notmuch@notmuchmail.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: "Use and development of the notmuch mail system." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 29 Apr 2018 10:22:52 -0000 Hi notmuch developers, I also had this database corruption, I waited for the fix to land in notmuch 0.26.2, build it, moved the xapian directory away, did a notmuch new and restored the tags from a dump. But the problem remains: ~$ xapian-check ~/Mail/.notmuch/xapian docdata: blocksize=3D8K items=3D10841 firstunused=3D75 revision=3D82 levels=3D1 root= =3D2 B-tree checked okay docdata table structure checked OK termlist: blocksize=3D8K items=3D1893162 firstunused=3D368983 revision=3D82 levels=3D= 3 root=3D177608 xapian-check: DatabaseError: 1 unused block(s) missing from the free list, = first is 0 this is very similar to the old database which I had moved away: ~$ xapian-check ~/Mail/.notmuch/xapian-2018-04-29-00-22/ docdata: blocksize=3D8K items=3D10863 firstunused=3D78 revision=3D59623 levels=3D1 r= oot=3D2 B-tree checked okay docdata table structure checked OK termlist: blocksize=3D8K items=3D1894648 firstunused=3D360821 revision=3D59623 levels= =3D3 root=3D360580 xapian-check: DatabaseError: 1 unused block(s) missing from the free list, = first is 0 Now I did notmuch compact and the database check says there are no errors. This seems to me as if the fix had not helped or there is another problem. $ notmuch --version notmuch 0.26.2+26~g9e158fb ~$ xapian-compact --version xapian-compact - xapian-core 1.4.3 Thanks for developing notmuch, Gregor * Javier Garcia [2018-04-07; 17:09]: > Unfortunately I can't share my emails without the approval of other > parties. The minimum subsets that trigger the error are in the range of > 1000-5000 mails, so asking each and everyone of them is out of my reach. > I tried to replicate the problem using just spam folders without success. > > The following is a solid workaround I've stumbled upon. Afew no longer > complains and database corruption is gone. > > $ notmuch compact > $ xapian-check ~/.mail/.notmuch/xapian > =C2=A0=C2=A0 > =C2=A0=C2=A0 No errors found > > I built xapian-core 1.50 but I can't compile notmuch 0.26.1 against it. > I will wait and test again in a few weeks. > > If you are interested in my setup, the error happens with this minimal > configuration. > > #~/.config/afew/config > [Filter.1] > query =3D 'folder:"//(INBOX|Inbox|inbox)$/" AND (NOT tag:inbox)' > tags =3D +inbox;-new > message =3D Messages in INBOX folder are tagged as inbox > > [Filter.2] > query =3D '(NOT folder:"//(INBOX|Inbox|inbox)$/") AND (tag:inbox)' > tags =3D -inbox > message =3D Messages not in INBOX folder cannot be inbox > > #~/.notmuch-config > [database] > path=3D/home-path/.mail > [new] > tags=3Dnew > > On 07/04/18 12:51, David Bremner wrote: >> Javier Garcia writes: >> >>> I've applied the path to notmuch 0.26.1 without success. >>> >>> $ rm -rf ~/.mail/.notmuch >>> $ LD_LIBRARY_PATH=3D/hidden-path/notmuch-0.26.1/lib/:$LD_LIBRARY_PATH >>> ./notmuch new >>> =C2=A0=C2=A0 Found 20065 total files (that's not much mail). >>> =C2=A0=C2=A0 Processed 20065 total files in 58s (341 files/sec.). >>> =C2=A0=C2=A0 Added 19605 new messages to the database. >>> >>> $ xapian-check .mail/.notmuch/xapian/ >>> =C2=A0=C2=A0 docdata: >>> =C2=A0=C2=A0 blocksize=3D8K items=3D63 firstunused=3D1 revision=3D2 lev= els=3D0 root=3D0 >>> =C2=A0=C2=A0 B-tree checked okay >>> =C2=A0=C2=A0 docdata table structure checked OK >>> =C2=A0=C2=A0 termlist: >>> =C2=A0=C2=A0 blocksize=3D8K items=3D43520 firstunused=3D8293 revision= =3D2 levels=3D2 root=3D748 >>> =C2=A0=C2=A0 xapian-check: DatabaseError: 1 unused block(s) missing fro= m the free >>> list, first is 0 >> OK, so probably not related to reference loops (although that patch is >> not very well tested). It's not clear how notmuch is triggering it, but >> this looks like the same bug in Xapian that olly fixed recently [1]. >> >> A possible next step is to try building xapian master, and linking >> notmuch against that. >> >> Maybe Patrick or Justus (in copy) has some idea why you're only seeing >> problems in afew. >> >> Another debugging direction is to try to duplicate your problem with >> some subset of mail that you're willing to share (bisection is the usual >> strategy). >> >> [1] https://notmuchmail.org/pipermail/notmuch/2018/026369.html > > _______________________________________________ > notmuch mailing list > notmuch@notmuchmail.org > https://notmuchmail.org/mailman/listinfo/notmuch Ciao; Gregor --=20 -... --- .-. . -.. ..--.. ...-.-