From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from localhost (localhost [127.0.0.1]) by arlo.cworth.org (Postfix) with ESMTP id 1B5E36DE301E for ; Fri, 14 Jul 2017 03:14:54 -0700 (PDT) X-Virus-Scanned: Debian amavisd-new at cworth.org X-Spam-Flag: NO X-Spam-Score: -0.542 X-Spam-Level: X-Spam-Status: No, score=-0.542 tagged_above=-999 required=5 tests=[AWL=0.379, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.211, SPF_PASS=-0.001, T_RP_MATCHES_RCVD=-0.01] autolearn=disabled Received: from arlo.cworth.org ([127.0.0.1]) by localhost (arlo.cworth.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id A_4e7zS22sG0 for ; Fri, 14 Jul 2017 03:14:52 -0700 (PDT) Received: from mout.gmx.net (mout.gmx.net [212.227.17.21]) by arlo.cworth.org (Postfix) with ESMTPS id A878E6DE301C for ; Fri, 14 Jul 2017 03:14:51 -0700 (PDT) Received: from len.workgroup ([84.189.136.132]) by mail.gmx.com (mrgmx101 [212.227.17.168]) with ESMTPSA (Nemesis) id 0LyR1G-1dklSA0PyA-015naa for ; Fri, 14 Jul 2017 12:14:48 +0200 Date: Fri, 14 Jul 2017 12:14:36 +0200 From: Gregor Zattler To: notmuch Subject: Re: Bug: fatal error with notmuch new, second run starts indexing all over again Message-ID: <20170714101436.dnacwh3rbwva5htp@len.workgroup> Mail-Followup-To: notmuch References: <20170703220750.dkornyh4ho7b2azy@len.workgroup> <87inj9w3nu.fsf@tethera.net> <20170706093101.cdjxdgvfy57a3kkb@len.workgroup> <87o9st3hz0.fsf@tethera.net> <20170709142147.czeett6stskxrnkp@len.workgroup> <87k23h3cgg.fsf@tethera.net> <20170713070444.tr4n2yfot5t4j663@len.workgroup> <87bmooebfs.fsf@tethera.net> <20170713135947.nqdagxnxe5sdubko@len.workgroup> <87y3rsp517.fsf@tesseract.cs.unb.ca> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <87y3rsp517.fsf@tesseract.cs.unb.ca> X-Provags-ID: V03:K0:KkNVtIT/3TGRlOaI0RdMJy/1TrR+6eu+2B5TR0OLegFxxe+b0rF YyARFjfTbUHY3bxMcKP5Q5r1p5htOIEBF+C27GA7iQf0FlVgJyQGzzThJtn09sKU1ONWPW0 Cuz63TKf9PMRt8IWHARcmXWT2gYs8hx8JEZFDPNhLBTB76iW6WKbolSQ+nZM8b6srOJvlum dUAK8VZn7PR2uLqI1/XaA== X-UI-Out-Filterresults: notjunk:1;V01:K0:M5KqOERPrsk=:8wLwHNTMzMzfSVS4BYdkUC zhGoWOOrQyPrYia143KeOsC8/rwh+7pvLneURaFCCWgg/haBnIDEz2dYDL9yXCKUD3iiWyqGy 9X8q8P+VN2r8O7n2wMYXe9VmaJ4dVyFb2zhdztywdb5cJqhqBFRf2WfqBGGnqEDaO0/cmddCw O06x0O4Tbuhp0ZV8CdtAF0s/byIY8oMJeN2Chhp1VcA8Dcz2/IbIafQI98KP6Xe4VoHek6Z1r L4FY+6mbz4WS6hIb5KyCJ92sMXPYsAOR9L3wNvTrFma8pYGb3T2zzB0XDpT1QbSdihmPQqKnw pWorKsaV8rFz94t20zt2t6MJkXrpSF53ux9HgwkP+Wd+9RQ/62ihuDRae05gz2OlmESRNgiWy OcdmsPUJ+K4vmcxSLvRGVBi4KWhBaw/DuOMPkUq5z9PJ9Bz1pGsnd0f3PehQuFfDCKnNIe6nl M3iCX+3rN5UxkDBXF/cBysODxh96iyDpO7w+b/MBg2JZFQJIpSkE46oY2rm0h4O3RNpkUmNXx madAiJ0O5mhLSzr8A1GWoAQFntEq/3OiV6lv4/ZBVExmGUWY3Jivop8a+lP3CPwe7QLBqInvk ZsTdFs9PfJfMuKoz0J7b4raiI00d01sk673iXnO01ityfj5yHUtHJFsdLI4w+at/d9Cuna6Qt KgZeRYA5q6V1IyWEIq95LEfqXwITk7Rn1pxeteFCYcIqpr0IuyTlxBvcAnr0YToVb2tuTM8yC ederXhwmgMmwrzS5TUqvFAJG1LzXjvgVbjSnEktwaowLUBijVULALU7Rw2t4T8Z/AJd7sBq9v 5C/NhLvA6F0+A/vQB/bIiog8GO5+g== X-BeenThere: notmuch@notmuchmail.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Use and development of the notmuch mail system." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 14 Jul 2017 10:14:54 -0000 Hi David, * David Bremner [13. Jul. 2017]: >> * David Bremner [13. Jul. 2017]: >>> Gregor Zattler writes: >>>> I then downloaded parts of the archive, merged and sliced it and >>>> now I have a sample of 25001 total files (that's not much mail), >>>> on which notmuch new produces a xapian exeption: [...] >>> For me this corpus does not produce an exception (although it does have >>> one thread with 5067 messages). >>> I'm running Debian testing, so there could be slightly different >>> versions of xapian and gmime. What versions of libxapian-dev and >>> libgmime-2.6-dev have you built notmuch against? [...] > I tried on a debian stretch with the same library versions, and I still > can't duplicate the crash. This is strange. I rerun the self compiled notmuch new on this corpus and it did not throw an exeption, nore did the debian stretch binary. I have no idea why this is different with the first run. On my way reducing the corpus I had an intermediate corpus of ~43000 files on which notmuch new produced a xapian exeption. I kept this corpus but did not include the test results in my email because the later, smaller corpus seemed more interesting. I redid the test on this mid-sized corpus and I got an exeption (again): 0 (master *) grfz@len:~$ gdb --args /home/grfz/src/notmuch/notmuch new GNU gdb (Debian 7.12-6) 7.12.0.20161007-git Copyright (C) 2016 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: . Find the GDB manual and other documentation resources online at: . For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from /home/grfz/src/notmuch/notmuch...done. (gdb) b _notmuch_database_log Breakpoint 1 at 0x1f6e0: file lib/database.cc, line 426. (gdb) run Starting program: /home/grfz/src/notmuch/notmuch new [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". Found 43255 total files (that's not much mail). Breakpoint 1, _notmuch_database_log (notmuch=0x5555557d2820, format=0x55555558d258 "A Xapian exception occurred adding message: %s.\n") at lib/database.cc:426 426 { (gdb) bt #0 _notmuch_database_log (notmuch=0x5555557d2820, format=0x55555558d258 "A Xapian exception occurred adding message: %s.\n") at lib/database.cc:426 #1 0x0000555555577f58 in notmuch_database_add_message (notmuch=notmuch@entry=0x5555557d2820, filename=filename@entry=0x555556fa4870 "/tmp/reduced-sample.xapian-exeption/new/1499897795.R10126604875560291557.len", message_ret=message_ret@entry=0x7fffffffd2e8) at lib/database.cc:2597 #2 0x000055555556802f in add_file (state=0x7fffffffd560, filename=0x555556fa4870 "/tmp/reduced-sample.xapian-exeption/new/1499897795.R10126604875560291557.len", notmuch=0x5555557d2820) at notmuch-new.c:264 #3 add_files (notmuch=notmuch@entry=0x5555557d2820, path=path@entry=0x5555557d26e0 "/tmp/reduced-sample.xapian-exeption/new", state=state@entry=0x7fffffffd560) at notmuch-new.c:599 #4 0x0000555555567b44 in add_files (notmuch=0x5555557d2820, path=path@entry=0x5555557cf730 "/tmp/reduced-sample.xapian-exeption", state=state@entry=0x7fffffffd560) at notmuch-new.c:483 #5 0x00005555555689ed in notmuch_new_command (config=0x5555557ce1d0, argc=, argv=) at notmuch-new.c:1099 #6 0x0000555555561a27 in main (argc=, argv=0x7fffffffda88) at notmuch.c:456 (gdb) continue Continuing. Error: A Xapian exception occurred. Halting processing. Processed 27611 total files in 21m 2s (21 files/sec.). Added 27481 new messages to the database. Note: A fatal error was encountered: A Xapian exception occurred [Inferior 1 (process 6752) exited with code 01] (gdb) I redid the test with this medium sized corpus with the notmuch binary from debian stretch, the xapian exeption hits after the same number of scanned/added files: 0 (master *) grfz@len:~$ gdb --args /usr/bin/notmuch new GNU gdb (Debian 7.12-6) 7.12.0.20161007-git Copyright (C) 2016 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: . Find the GDB manual and other documentation resources online at: . For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from /usr/bin/notmuch...(no debugging symbols found)...done. (gdb) b _notmuch_database_log Function "_notmuch_database_log" not defined. Make breakpoint pending on future shared library load? (y or [n]) y Breakpoint 1 (_notmuch_database_log) pending. (gdb) run Starting program: /usr/bin/notmuch new [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". Found 43255 total files (that's not much mail). Error: A Xapian exception occurred. Halting processing. Processed 27611 total files in 10m 21s (44 files/sec.). Added 27481 new messages to the database. Note: A fatal error was encountered: A Xapian exception occurred [Inferior 1 (process 10408) exited with code 01] (gdb) bt No stack. (gdb) continue The program is not being run. (gdb) quite Undefined command: "quite". Try "help". (gdb) quit In case your' not exhausted, the mid-sized corpus is here: https://giku.de/reduced-sample-midsized.xapian-exeption.tar.xz Regards, Gregor