From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from localhost (localhost [127.0.0.1]) by olra.theworths.org (Postfix) with ESMTP id 3C627431FBC for ; Wed, 12 Dec 2012 12:49:28 -0800 (PST) X-Virus-Scanned: Debian amavisd-new at olra.theworths.org X-Spam-Flag: NO X-Spam-Score: -0.7 X-Spam-Level: X-Spam-Status: No, score=-0.7 tagged_above=-999 required=5 tests=[RCVD_IN_DNSWL_LOW=-0.7] autolearn=disabled Received: from olra.theworths.org ([127.0.0.1]) by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id RFgI5+-oFAbI for ; Wed, 12 Dec 2012 12:49:27 -0800 (PST) Received: from dmz-mailsec-scanner-3.mit.edu (DMZ-MAILSEC-SCANNER-3.MIT.EDU [18.9.25.14]) by olra.theworths.org (Postfix) with ESMTP id AD9C7431FB6 for ; Wed, 12 Dec 2012 12:49:27 -0800 (PST) X-AuditID: 1209190e-b7f516d0000008e4-4d-50c8edd70b77 Received: from mailhub-auth-1.mit.edu ( [18.9.21.35]) by dmz-mailsec-scanner-3.mit.edu (Symantec Messaging Gateway) with SMTP id 75.B0.02276.7DDE8C05; Wed, 12 Dec 2012 15:49:27 -0500 (EST) Received: from outgoing.mit.edu (OUTGOING-AUTH.MIT.EDU [18.7.22.103]) by mailhub-auth-1.mit.edu (8.13.8/8.9.2) with ESMTP id qBCKnQwm023632; Wed, 12 Dec 2012 15:49:26 -0500 Received: from awakening.csail.mit.edu (awakening.csail.mit.edu [18.26.4.91]) (authenticated bits=0) (User authenticated as amdragon@ATHENA.MIT.EDU) by outgoing.mit.edu (8.13.6/8.12.4) with ESMTP id qBCKnNrk024766 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT); Wed, 12 Dec 2012 15:49:25 -0500 (EST) Received: from amthrax by awakening.csail.mit.edu with local (Exim 4.80) (envelope-from ) id 1TitF4-0007I4-SE; Wed, 12 Dec 2012 15:49:22 -0500 Date: Wed, 12 Dec 2012 15:49:22 -0500 From: Austin Clements To: Patrick Totzke Subject: Re: notmuch python bindings corrupt db index (was: gmail importer script) Message-ID: <20121212204922.GB6187@mit.edu> References: <20121211182638.27237.98903@brick.lan> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFuphleLIzCtJLcpLzFFi42IR4hRV1r3+9kSAwfkV/BYPrkpZXL85k9ni 4hdZB2aPnbPusns8W3WL2WPy36fMAcxRXDYpqTmZZalF+nYJXBkbm26zFMzirNh57BNjA+NC 9i5GDg4JAROJBZvVuxg5gUwxiQv31rN1MXJxCAnsY5S4f3clC4SzgVGi83oDI4RzkUni3/lG dghnCaPErSeHmUH6WQRUJZ4/fskIYrMJaEhs278czBYR0JF4uGAnE4jNLOAsMWnPbRYQW1gg ROJldwtYnFdAW+Jjy0GoDQcYJa5MfwKVEJQ4OfMJC0SzlsSNfy+ZQO5mFpCWWP6PAyTMKRAo 8XjaKbAbRAVUJKac3MY2gVFoFpLuWUi6ZyF0L2BkXsUom5JbpZubmJlTnJqsW5ycmJeXWqRr rJebWaKXmlK6iREc6JJ8Oxi/HlQ6xCjAwajEw3vh5fEAIdbEsuLK3EOMkhxMSqK8K5+cCBDi S8pPqcxILM6ILyrNSS0+xCjBwawkwut2ACjHm5JYWZValA+TkuZgURLnvZJy019IID2xJDU7 NbUgtQgmK8PBoSTBqw6MaCHBotT01Iq0zJwShDQTByfIcB6g4dwgNbzFBYm5xZnpEPlTjIpS 4rzn3wAlBEASGaV5cL2wRPSKURzoFWFeDpB2HmASg+t+BTSYCWhw3KXjIINLEhFSUg2Mwj42 uc8P+zw4/qxd57NIsGPXTNOtlYUN65Jzv6rX+7DGShXVsGzlnPHj3MnpOQ96hMzeXLDcl3Qq MVzHhSF/ukToNQ024wNn93i7H+mz32l1IKu4WeLQs7fcfffskidmu9nNVpIXPVkmlsWpe+zb rrJ76fsccgvuHXYtuTErtm7ZT0GWrYZKLMUZiYZazEXFiQA9XRU0HwMAAA== Cc: notmuch@notmuchmail.org X-BeenThere: notmuch@notmuchmail.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: "Use and development of the notmuch mail system." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 12 Dec 2012 20:49:28 -0000 Quoth Jason A. Donenfeld on Dec 12 at 7:35 pm: > On Tue, Dec 11, 2012 at 7:26 PM, Patrick Totzke wrote: > > i am kind of reluctant to reproduce this again (on my work-desktop). > > It'd really be a huge help if you could backup and try... > > > All i did was start your script, writing to the index in another terminal in between (using alot) > > possibly my cronjob fired a "notmuch new; afew.." in between also. > > when I killed your script the index war broken. > > > > Can i turn on some global logging for notmuch to debug this? > > Maybe libnotmuch had trouble with so many db-writes at a time? > > That is very wrong and bizarre. I open the database in read/write > mode, which is supposed to lock other programs from accessing it in > this way... Very strange. There should be no way to corrupt the database at this level through the Xapian API, which means nothing libnotmuch can do (much less users of libnotmuch) should be able to corrupt the database. If you can reproduce the problem, it's probably a serious bug in Xapian, but it could also have been a file system bug or even random file system corruption.