From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp0 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms11 with LMTPS id 6PzBDQ4W1l8FSAAA0tVLHw (envelope-from ) for ; Sun, 13 Dec 2020 13:24:30 +0000 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp0 with LMTPS id cZ+PCQ4W1l/IdwAA1q6Kng (envelope-from ) for ; Sun, 13 Dec 2020 13:24:30 +0000 Received: from mail.notmuchmail.org (nmbug.tethera.net [IPv6:2607:5300:201:3100::1657]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (2048 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id C382294042C for ; Sun, 13 Dec 2020 13:24:29 +0000 (UTC) Received: from nmbug.tethera.net (localhost [127.0.0.1]) by mail.notmuchmail.org (Postfix) with ESMTP id 7E26428C03; Sun, 13 Dec 2020 08:24:23 -0500 (EST) X-Greylist: delayed 304 seconds by postgrey-1.36 at nmbug; Sun, 13 Dec 2020 08:24:19 EST Received: from mout.gmx.net (mout.gmx.net [212.227.17.22]) by mail.notmuchmail.org (Postfix) with ESMTPS id E037528539 for ; Sun, 13 Dec 2020 08:24:19 -0500 (EST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=gmx.net; s=badeba3b8450; t=1607865856; bh=NsmqJVPHkz8x/w8TKQuBe/uQuYzSK60muDiw09dKptI=; h=X-UI-Sender-Class:Date:From:To:Subject; b=luqlRP8FRZECACX9Q3MGa16aSaCLfEH+HR/ROV0Rfb2xyfhE7BdRYdw0kc+LLTmu0 j1sGmIoBk9oDMMUP15YIWfq3VduiLRbBOr0c+OYRbt4gzFr1qylHdRZt7FF3092Bgw +qKfZva3ARhfUCXVcif7LyGGh46Fn1Q9U3qrE/RY= X-UI-Sender-Class: 01bb95c1-4bf8-414a-932a-4f6e2808ef9c Received: from no.speedport.ip ([84.185.97.253]) by mail.gmx.com (mrgmx105 [212.227.17.168]) with ESMTPSA (Nemesis) id 1MLiCu-1kWuOX2Vii-00HioZ for ; Sun, 13 Dec 2020 14:19:13 +0100 Date: Sun, 13 Dec 2020 14:19:09 +0100 From: Gregor Zattler To: notmuch Subject: consistent database corruption with notmuch new Message-ID: <20201213131909.GD21521@no.workgroup> Mail-Followup-To: notmuch MIME-Version: 1.0 Content-Disposition: inline X-Provags-ID: V03:K1:bNlUKNEgy8PO7p68Lw/t6v9Q0noqkm+xz+byXt26WfAnfgrqC6U 71pU7bmPDwjT5XpoXXPmVtwoAfNYfqcosRPNU9JUXTXmRM/pKVXGfpR7Nr+vg+2QLRi7vBL K8e+K788xOAMOol0IULVPDobM0j21MewdHE32CsBcNxhsJKiJd5KH52N896SAXAVpcHVB1g uR68e292+BgKmQcw2BCzg== X-Spam-Flag: NO X-UI-Out-Filterresults: notjunk:1;V03:K0:Hpp4kuQGWl4=:pS0U+P0+SpSUmLuYdkFbTX eBBaBLpBBYy4gkOuEg/7TE6y8Iu0hR0zIMpbFm9oTYPljpg/y70JsDXN+JBtF/NJUnVheXtcw cfWa1YkAHilGrxlyADN73Eo3WCtroYIUU6QLs5VLf1nTVcpyY1yK8tub+ouilAP2D5yKsthJC /Fe/FZFwDMU6p2bk5b6uV1MxLChoZczxh+Ex3Xgnpqsci1qHItdcmvUsCjy6uZdAV9I/VYLhh ZyWACKhODCnja6/tq+5QleroDD3wRSUy5tyg3LOKnKgzAvrrFlNFrF8ny9F7QY+xrywRJs57V xl5vKcG22EzLrFiHU8troIbZwIIJDhucyAU5rAuizsf78PMM4oAdlZfQuqB0W3/g0y6xVL4cM W8u0TmBz8wSNvbejKQUOb1jWImFrMEIZCRsWjRLZWEfeqVYzlMNmu4v3f1ZMPNK4QP/udI7mP Z7ABzBV6NmPQup68G/rtWROMkC4KYqL2ozOkCWoLKWjRmZvvGVFIu4Lm07thM2JOtfx1vMDFC gseumb7PdlD8IRMS94oVLqLAyXTfjhVEmhoZoHOBnjLUXp2SFFnWKperxlcwUoZZhq5EFOg3L ofAyJOUB59iOqw+ZDisezL34BeuyTcA4k2jqkmjTQkmrzpvEwXXhYwbCShZciMveZ9ug611RM joaxDimgH8X3IvdebNiuu6djmx3FBSXjM8FehaMM0eYMOvDldYQk1BIbOxeggHTclt4IJOjTG X+iRJCikE59tg+t8mgttkZZDFfQ+ecV9khePUHtjG8UJQPw7l8ByDEvLtxmHCwqwNqa/fJwmD oIzfKfX87xNCRFARFiUC8M8T5Y8S7lxN805xKUSHxsXrpHo3LjnsLwdHUU43TnQCiiGAv63E1 egjrDo6aucF1yUMX2PpQ== Message-ID-Hash: J6QN7QZWDQI2NPTBJGNY2N5AU7TW3NR3 X-Message-ID-Hash: J6QN7QZWDQI2NPTBJGNY2N5AU7TW3NR3 X-MailFrom: telegraph@gmx.net X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; header-match-notmuch.notmuchmail.org-0; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; suspicious-header X-Mailman-Version: 3.2.1 Precedence: list List-Id: "Use and development of the notmuch mail system." List-Help: List-Post: List-Subscribe: List-Unsubscribe: Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Migadu-Flow: FLOW_IN X-Migadu-Spam-Score: 0.83 Authentication-Results: aspmx1.migadu.com; dkim=fail (body hash did not verify) header.d=gmx.net header.s=badeba3b8450 header.b=luqlRP8F; dmarc=fail reason="SPF not aligned (relaxed)" header.from=gmx.net (policy=none); spf=pass (aspmx1.migadu.com: domain of notmuch-bounces@notmuchmail.org designates 2607:5300:201:3100::1657 as permitted sender) smtp.mailfrom=notmuch-bounces@notmuchmail.org X-Migadu-Queue-Id: C382294042C X-Spam-Score: 0.83 X-Migadu-Scanner: scn1.migadu.com X-TUID: snPeQ7fdyI6N Dear notmuch developers, I need help because notmuch new on my configured notmuch system without index/database consistently produces a corrupted database. On Friday my notmuch database got corrupted. xapian-check xapian F did not fix the database. I moved the xapian directory to xapian-corrupted and did a notmuch new. This consistently yields a corrupted database after some hours (no other notmuch running, no mail delivery, no cron jobs etc): 1 (master *) grfz@no:~/Mail/.notmuch$ rm -rf xapian/ 0 (master *) grfz@no:~/Mail/.notmuch$ time nice ionice -c3 notmuch new ; time nice ionice -c3 notmuch new ; time nice ionice -c3 notmuch new ; zcat notmuch.dump.2020-12-11.1607641607 | notmuch tag --batch ; notmuch tag --input=notmuch-doit-batch-flow-via-procmail-since-11 ; notmuch tag --input=notmuch-doit-batch-flow-via-procmail ; notmuch tag --input=notmuch-doit-batch-flow-via-procmail.with-failure ; time nice ionice -c3 notmuch new ; time nice ionice -c3 notmuch new ; time nice ionice -c3 notmuch new Welcome to a new version of notmuch! Your database will now be upgraded. This process is safe to interrupt. Backing up tags to /home/grfz/Mail/.notmuch/dump-20201212T212701.gz... Your notmuch database has now been upgraded. Note: Ignoring non-mail file: /home/grfz/Mail/drafts.mbox Note: Ignoring non-mail file: /home/grfz/Mail/postponed.mbox Processed 1175458 total files in 2h 47m 16s (117 files/sec.). Added 1101746 new messages to the database. real 167m16,589s user 79m23,083s sys 6m19,064s Processed 124685 total files in 29m 21s (70 files/sec.). Added 83646 new messages to the database. real 29m25,353s user 10m24,737s sys 2m14,977s add_file: A Xapian exception occurred). A Xapian exception occurred at lib/message.cc:1182: Expected block 325393 to be level 0, not 1 Processed 122555 total files in 30m 46s (66 files/sec.). Added 83636 new messages to the database. Note: A fatal error was encountered: A Xapian exception occurred real 30m49,774s user 9m44,716s sys 2m3,859s adding tag frommeMessage-ID: (null) Status: A Xapian exception occurred add_file: A Xapian exception occurred). A Xapian exception occurred at lib/message.cc:1182: Too few chunks of compressed data Processed 122555 total files in 30m 38s (66 files/sec.). Added 83636 new messages to the database. Note: A fatal error was encountered: A Xapian exception occurred real 30m42,427s user 10m22,614s sys 2m2,737s add_file: A Xapian exception occurred). A Xapian exception occurred at lib/message.cc:1182: Too few chunks of compressed data Processed 122555 total files in 31m 39s (64 files/sec.). Added 83636 new messages to the database. Note: A fatal error was encountered: A Xapian exception occurred real 31m43,529s user 10m33,959s sys 2m7,378s add_file: A Xapian exception occurred). A Xapian exception occurred at lib/message.cc:1182: Too few chunks of compressed data Processed 122555 total files in 30m 44s (66 files/sec.). Added 83636 new messages to the database. Note: A fatal error was encountered: A Xapian exception occurred real 30m48,443s user 10m14,171s sys 2m8,209s While the first notmuch new run seems OK, there at least is the question why the second one adds another 83646 new messages. This happened several times in a row. This is a Thinkpad x240, 8GB RAM on debian buster but with notmuch compiled from source (0.31.2+28~gadfded9). I think the SSD is OK (no hints to SSD failures in log files, smart tests OK, I even filled the free rest of the SSD with randomm data and after that compared it to the source with no difference. Any idea what to do in order to get a running notmuch mail system going? If it's interesting to nomuch development, what happens here, I'm happy to so some digging if someone gives me instructions. I'm not a developer and have no clue how to debug this. Ciao, Gregor -- -... --- .-. . -.. ..--.. ...-.-