From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp2 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms11 with LMTPS id wF4ODxEgFWBIMQAA0tVLHw (envelope-from ) for ; Sat, 30 Jan 2021 09:00:01 +0000 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp2 with LMTPS id mC3VChEgFWBiFAAAB5/wlQ (envelope-from ) for ; Sat, 30 Jan 2021 09:00:01 +0000 Received: from mail.notmuchmail.org (nmbug.tethera.net [144.217.243.247]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (2048 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 9CC949401C0 for ; Sat, 30 Jan 2021 08:59:59 +0000 (UTC) Received: from nmbug.tethera.net (localhost [127.0.0.1]) by mail.notmuchmail.org (Postfix) with ESMTP id 61CC228CCE; Sat, 30 Jan 2021 03:59:49 -0500 (EST) X-Greylist: delayed 304 seconds by postgrey-1.36 at nmbug; Sat, 30 Jan 2021 03:59:45 EST Received: from mout.gmx.net (mout.gmx.net [212.227.17.20]) by mail.notmuchmail.org (Postfix) with ESMTPS id 6902928CB4 for ; Sat, 30 Jan 2021 03:59:45 -0500 (EST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=gmx.net; s=badeba3b8450; t=1611997183; bh=m7Xf32g8aazj1gK0cS6PxE+LL4T9gZr6FNikyOdruhA=; h=X-UI-Sender-Class:Date:From:To:Subject:References:In-Reply-To; b=JBhVxuXghDuCES3p0DPGjuT3x0+Y7KMMiW4Zm9FVEKi4hX7u31xAv4/5v77olDN8n VntOqwODYcj2Ve/y6gGz1LxyMdMUL1S62VzwjDqpbOkH0Pe1RetIhNfMU9xTMa9XaI 8g9quX/yN8UJfuPa+gJ02stSumO4przFkqVcKHpg= X-UI-Sender-Class: 01bb95c1-4bf8-414a-932a-4f6e2808ef9c Received: from no.workgroup ([84.185.97.253]) by mail.gmx.net (mrgmx104 [212.227.17.168]) with ESMTPSA (Nemesis) id 1MmlTC-1lnNyz22uU-00jr8W for ; Sat, 30 Jan 2021 09:54:39 +0100 Date: Sat, 30 Jan 2021 09:54:32 +0100 From: Gregor Zattler To: notmuch Subject: out of memory on idle machine (was: Re: consistent database corruption with notmuch new) Message-ID: <20210130085432.GA14025@no.workgroup> Mail-Followup-To: notmuch References: <20201213131909.GD21521@no.workgroup> <87zh2hhk15.fsf@tethera.net> <20201213141543.GE21521@no.workgroup> <20201213151336.GF21521@no.workgroup> <20201213212252.GH21521@no.workgroup> <20201214192251.GA7858@no.workgroup> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20201214192251.GA7858@no.workgroup> X-Provags-ID: V03:K1:vuse5s9mZoKcLtUhSQb4/ErpShJhOxTO7ExIU05jjqYmn9oK6lh Zto1toHYjeJfxfmAi1TaHSdVubcPatrHagz4NU7dy/MmUvvheeXtTO98Nsl2waM7+N+E2NS e2dc/+kVll4V0qo+ZnFkKgXnAorhrwDaw0UGUgzxLNQwlbLmyGAa6aDfheC1ipEs0uJtUUi yj9utOH8sdnnb+a17a7sQ== X-Spam-Flag: NO X-UI-Out-Filterresults: notjunk:1;V03:K0:4YwGe4hQFsU=:v0npwBDZUOGIHSFwifhocv bus46hstKmYDJm2Ryd68ffXYB43YXSp0NuJHblPcf4eAaZsmIcNOGFtaCkMDoDzXrEVu68NS9 X7U7QTa6lT5cPVC398BiP1tRqGWHLYFVciLf55JQ6eWeMfrQ24W1EhskofVhi65MQG93Q9F+q BVbV45mePcQ6w14WAHI83B3WnhRpQHZijO2RSiEDnoTLnax2zZCwAXelRmJ8xiRGMcY1xgRcO gW8xuZ6Q0G8Q+wQ89oZ50LNYusIHW60YPNUbPkz5K7ySEPBWGsNze+zMKpktpJq7TMWOxNAt2 aQGtyBrFBeTqpRo1knjVorKiv5Svslx314YGQfDLnbnkQ6UcJH7m3qvl4I6sA6b054DIxdqga gufsidPb94UaoQglyq9lxYDd9SvSKSNEXbGp1b7zWTv5/BHzlBJoB5NNRjy56JVIWEGJ9hx31 8B2sM1RokwW1xwygRrLJQLMrnTFcJmOIJXGvLDAIqG6mY/RNFPjDrf9nF9KnnrHvnXTaTsb9m JTqXBN9CJ/crJRTFqEf0nA/PmTvjXlQdUUx1gcoDO8r9Jw1shjEqO8knYG1afmsOi/06iAsmA fF3eEIjlnbj/CpUtH5xPYqIHSXFHmwWggNQHTyMm+AIWdCijoqul8taEcHhK21iMva+XgjxXa 9UrdwKVNx66L8MOzwh9W1EHIWoEe21EguQiOQyAKQdDwOU7v50Uf+qepdfUZ+xQ3/G7xanzc3 7YUFGswFQckGHoqD2scVNVw3aIh22xQA6tq+zWTQNHWx1Jki6Bnx3mPwU9IuKBdijDwo9+e+R KThOq7GNwL0YkLBNWRi1pEb1LND7h9aJAs0lfv0nEs83qq6jwwtQFDq5RGJ4GeW4UeOsK4DwS ccvMxJ2oKUJTP5F+OTrA== Message-ID-Hash: QIEIGCSMKJTNQBTJ3PGCGYYT7W4NELAO X-Message-ID-Hash: QIEIGCSMKJTNQBTJ3PGCGYYT7W4NELAO X-MailFrom: telegraph@gmx.net X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; header-match-notmuch.notmuchmail.org-0; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; suspicious-header X-Mailman-Version: 3.2.1 Precedence: list List-Id: "Use and development of the notmuch mail system." List-Help: List-Post: List-Subscribe: List-Unsubscribe: Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Migadu-Flow: FLOW_IN X-Migadu-Spam-Score: -0.97 Authentication-Results: aspmx1.migadu.com; dkim=fail ("body hash did not verify") header.d=gmx.net header.s=badeba3b8450 header.b=JBhVxuXg; dmarc=fail reason="SPF not aligned (relaxed)" header.from=gmx.net (policy=none); spf=pass (aspmx1.migadu.com: domain of notmuch-bounces@notmuchmail.org designates 144.217.243.247 as permitted sender) smtp.mailfrom=notmuch-bounces@notmuchmail.org X-Migadu-Queue-Id: 9CC949401C0 X-Spam-Score: -0.97 X-Migadu-Scanner: scn0.migadu.com X-TUID: hczcU6MHKl/E Hi notmuch developers,, * Gregor Zattler [14. Dez. 2020]: > notmuch new still corrupts the database, the second notmuch new > invocation finds emails the first did not find. I'm still searching for the reason notmuch chokes on my mails. I assembled a HP MicroServer, installed basic debian buster and notmuch from the debian buster repo, rsynced my mail to a separate file system symlinked to the same location as on my laptop. There are now grfz@mic:~/Mail$ find -type f | wc -l 1209419 files on this file system. no other process touches this file system, actually the machine is otherwise ilde. I did notmuch new several times in a row: grfz@mic:~/Mail/.notmuch$ rm -rf xapian grfz@mic:~/Mail/.notmuch$ notmuch new Welcome to a new version of notmuch! Your database will now be upgraded. This process is safe to interrupt. Backing up tags to /home/grfz/Mail/.notmuch/dump-20210127T114210.gz... Your notmuch database has now been upgraded. Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607947606.8134_1.no:2, Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607940473.9509_1.no:2,S Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607969276.21046_1.no:2, Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607987211.1395_1.no:2, Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607979988.4942_1.no:2, Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607972847.4857_1.no:2, Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607943993.24776_1.no:2, Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607976389.23296_1.no:2, Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607983586.19063_1.no:2, Note: Ignoring non-mail file: /home/grfz/Mail/drafts.mbox Note: Ignoring non-mail file: /home/grfz/Mail/postponed.mbox Processed 1183682 total files in 16h 43m 27s (19 files/sec.). Added 1091038 new messages to the database. grfz@mic:~/Mail/.notmuch$ notmuch new Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607947606.8134_1.no:2, Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607940473.9509_1.no:2,S Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607969276.21046_1.no:2, Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607987211.1395_1.no:2, Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607979988.4942_1.no:2, Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607972847.4857_1.no:2, Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607943993.24776_1.no:2, Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607976389.23296_1.no:2, Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607983586.19063_1.no:2, Processed 1169095 total files in 16h 52m 48s (19 files/sec.). Added 1077686 new messages to the database. grfz@mic:~/Mail/.notmuch$ notmuch new Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607947606.8134_1.no:2, Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607940473.9509_1.no:2,S Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607969276.21046_1.no:2, Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607987211.1395_1.no:2, Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607979988.4942_1.no:2, Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607972847.4857_1.no:2, Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607943993.24776_1.no:2, Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607976389.23296_1.no:2, Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607983586.19063_1.no:2, Processed 1151900 total files in 16h 59m 36s (18 files/sec.). Added 1050106 new messages to the database. grfz@mic:~/Mail/.notmuch$ notmuch new add_file: Out of memory files/sec.). Processed 205 total files in 6s (29 files/sec.). Added 193 new messages to the database. Note: A fatal error was encountered: Out of memory grfz@mic:~/Mail/.notmuch$ notmuch new processes not all files, later invocations re-add most of the files and the fourth notmuch new invocation runs out of memory after a few seconds. This machine has grfz@mic:~/Mail$ free total used free shared buff/cache available Mem: 16394760 222272 2293404 5464 13879084 15831448 Swap: 15622140 15104 15607036 enough memory. It's ECC Memory. Any ideas on what is the cause of this? Even if we ignore the out-of-memory condition: Why does notmuch re-add a million already added files? Ciao; Gregor + my-fetchmail.services stop + /home/grfz/bin/notmuch-new.screen-locked--low-load + sed -i -e 's/^#exit 0$/exit 0/' /home/grfz/bin/notmuch-new.screen-locked--low-load + sleep 77 + my-fetchmail.services stop /home/grfz/bin/my-notmuch-renew: line 4: my-fetchmail.services: command not found + /home/grfz/bin/notmuch-new.screen-locked--low-load + sed -i -e 's/^#exit 0$/exit 0/' /home/grfz/bin/notmuch-new.screen-locked--low-load + sleep 77 ++ date +%F.%s + TAGSpreREINDEX=/home/grfz/Mail/.notmuch/notmuch.dump.2019-05-29.1559166634 + notmuch dump --gzip --format=batch-tag --output=/home/grfz/Mail/.notmuch/notmuch.dump.2019-05-29.1559166634 ++ date +%F.%s + mv /home/grfz/Mail/.notmuch/xapian /home/grfz/Mail/.notmuch/xapian.2019-05-29.1559166684 + notmuch config set query.all '(is:spam OR NOT is:spam OR is:deleted OR NOT is:deleted)' + notmuch config set query.izt-all 'query:all AND (is:izt OR (NOT is:izt AND (path:IZT-EDV/** OR path:IZT/**)))' + notmuch config set query.izt-edv '(is:izt OR path:IZT-EDV/**) AND (from:science-computing OR to:science-computing OR from:ATOS OR to:ATOS OR to:izt@2cal.de OR from:info@ralf-voegtle.de OR from:info@andresedv.de)' + notmuch config set query.izt-robots 'is:izt AND NOT path:IZT/** AND NOT path:sent/** AND NOT query:izt-edv' + notmuch config set query.izt '(is:izt OR path:IZT/**) AND NOT path:IZT-EDV/** AND NOT query:izt-edv)' + notmuch config set index.header.List List-Id + notmuch config set index.header.Spamgrfz X-Spam-grfz-Status + notmuch config set index.header.UserAgent User-Agent + notmuch config set index.header.XLabel X-Label + notmuch new Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1519322404.860_1.len:2, Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1525795204.16507_1.len:2,S Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1514242825.4116_1.len:2,S Note: Ignoring non-mail file: /home/grfz/Mail/~ml/linux-crypto@nl.linux.org/cur/1054419557.21641_405.pit:2, Note: Ignoring non-mail file: /home/grfz/Mail/~ml/linux-crypto@nl.linux.org/cur/1054419557.21641_525.pit:2, Note: Ignoring non-mail file: /home/grfz/Mail/~ml/linux-crypto@nl.linux.org/cur/1054419557.21641_685.pit:2, Note: Ignoring non-mail file: /home/grfz/Mail/rmlint.sh~ Note: Ignoring non-mail file: /home/grfz/Mail/postponed.mbox Note: Ignoring non-mail file: /home/grfz/Mail/drafts.mbox Note: Ignoring non-mail file: /home/grfz/Mail/rmlint.sh Welcome to a new version of notmuch! Your database will now be upgraded. This process is safe to interrupt. Backing up tags to /home/grfz/Mail/.notmuch/dump-20190529T215124.gz... Your notmuch database has now been upgraded. Processed 1036956 total files in 2h 21m 25s (122 files/sec.). Added 945879 new messages to the database. + notmuch tag --batch -new -inbox -unread '*' Can't specify both cmdline and stdin! + notmuch restore --accumulate --format=batch-tag --input=/home/grfz/Mail/.notmuch/notmuch.dump.2019-05-29.1559166634 Warning: cannot apply tags to missing message: notmuch-sha1-e584281f598522d172e73617da020e2f914ccb7e Warning: cannot apply tags to missing message: notmuch-sha1-5b56ca3568715b006eddd35a597ca7654be045f9 Warning: cannot apply tags to missing message: notmuch-sha1-00ab3ac6e631287f0f6dffaf0037bb7ef86f9e47 Warning: cannot apply tags to missing message: notmuch-sha1-72bde4c427b04c1cbac4a2c2b3a94402616917d7 Warning: cannot apply tags to missing message: notmuch-sha1-dbd41b3c6736943ef17eff9c6862dfa3703af020 Warning: cannot apply tags to missing message: notmuch-sha1-f17b642c650ba54b67b9d4ff5a338143a73f8b7b Warning: cannot apply tags to missing message: notmuch-sha1-e1cf1219c1fb10a55ba30ca590b93eaf33fdae27 Warning: cannot apply tags to missing message: notmuch-sha1-0d5aaa420dc9e9505cced7c5d8f4c15e0c88c5ef Warning: cannot apply tags to missing message: notmuch-sha1-a31480a6ef2520153b778dc9531f2c982a79ba32 Warning: cannot apply tags to missing message: notmuch-sha1-606e94dbe6e46b9df83d77ffa788429523e39646 Warning: cannot apply tags to missing message: notmuch-sha1-bb586b264c94f23d9e826071d8841d542ca95c75 Warning: cannot apply tags to missing message: notmuch-sha1-06d4d1492c70aa544b1deeaf7895ca85fe634670 Warning: cannot apply tags to missing message: notmuch-sha1-253991559a5e3c2d5cf1234b761e52ea09aa693f ++ date +%F.%s + notmuch dump --gzip --output=/home/grfz/Mail/.notmuch/notmuch.dump.2019-05-30.1559177807 ++ date +%F.%s + notmuch compact --backup=/home/grfz/Mail/.notmuch/xapian.2019-05-30.1559177848 Compacting database... compacting table postlist Reduced by 36% 908856K (2488312K -> 1579456K) compacting table docdata Reduced by 49% 312K (632K -> 320K) compacting table termlist Reduced by 47% 1701792K (3551384K -> 1849592K) compacting table position Reduced by 45% 3375368K (7372720K -> 3997352K) compacting table spelling doesn't exist compacting table synonym doesn't exist The old database has been moved to /home/grfz/Mail/.notmuch/xapian.2019-05-30.1559177848. Done. + sed -i -e 's/^exit 0$/#exit 0/' /home/grfz/bin/notmuch-new.screen-locked--low-load + my-fetchmail.services stop + /home/grfz/bin/notmuch-new.screen-locked--low-load + sed -i -e 's/^#exit 0$/exit 0/' /home/grfz/bin/notmuch-new.screen-locked--low-load + sleep 77 ++ date +%F.%s + TAGSpreREINDEX=/home/grfz/Mail/.notmuch/notmuch.dump.2019-05-30.1559214465 + notmuch dump --gzip --format=batch-tag --output=/home/grfz/Mail/.notmuch/notmuch.dump.2019-05-30.1559214465 ++ date +%F.%s + mv /home/grfz/Mail/.notmuch/xapian /home/grfz/Mail/.notmuch/xapian.2019-05-30.1559214505 + notmuch config set query.all '(is:spam OR NOT is:spam OR is:deleted OR NOT is:deleted)' + notmuch config set query.izt-all 'query:all AND (is:izt OR (NOT is:izt AND (path:IZT-EDV/** OR path:IZT/**)))' + notmuch config set query.izt-edv '(is:izt OR path:IZT-EDV/**) AND (from:science-computing OR to:science-computing OR from:ATOS OR to:ATOS OR to:izt@2cal.de OR from:info@ralf-voegtle.de OR from:info@andresedv.de)' + notmuch config set query.izt-robots 'is:izt AND NOT path:IZT/** AND NOT path:sent/** AND NOT query:izt-edv' + notmuch config set query.izt '(is:izt OR path:IZT/**) AND NOT path:IZT-EDV/** AND NOT query:izt-edv)' + notmuch config set index.header.List List-Id + notmuch config set index.header.Spamgrfz X-Spam-grfz-Status + notmuch config set index.header.UserAgent User-Agent + notmuch config set index.header.XLabel X-Label + notmuch config set index.header.DeliveredTo Delivered-To + notmuch config set index.header.DeliveryDate Delivery-date + notmuch new Stopping... Stopping... Stopping... Stopping... Stopping... Stopping... Stopping... Stopping... Stopping... Stopping... Stopping... Stopping... Stopping... Stopping... Stopping... Stopping... Stopping... Stopping... Stopping... Stopping... Stopping... Stopping... Stopping... Stopping... Stopping... Stopping... Stopping... Stopping... Stopping... Stopping... Stopping... Stopping... Stopping... Stopping... Stopping... Stopping... Stopping... Stopping... Stopping... Stopping... Stopping... Stopping... Stopping... Stopping... Stopping... Stopping... Stopping... + my-fetchmail.services stop + rm -rf /home/grfz/Mail/.notmuch/xapian + notmuch config set query.all '(is:spam OR NOT is:spam OR is:deleted OR NOT is:deleted)' + notmuch config set query.izt-all 'query:all AND (is:izt OR (NOT is:izt AND (path:IZT-EDV/** OR path:IZT/**)))' + notmuch config set query.izt-edv '(is:izt OR path:IZT-EDV/**) AND (from:science-computing OR to:science-computing OR from:ATOS OR to:ATOS OR to:izt@2cal.de OR from:info@ralf-voegtle.de OR from:info@andresedv.de)' + notmuch config set query.izt-robots 'is:izt AND NOT path:IZT/** AND NOT path:sent/** AND NOT query:izt-edv' + notmuch config set query.izt '(is:izt OR path:IZT/**) AND NOT path:IZT-EDV/** AND NOT query:izt-edv)' + notmuch config set index.header.List List-Id + notmuch config set index.header.Spamgrfz X-Spam-grfz-Status + notmuch config set index.header.UserAgent User-Agent + notmuch config set index.header.XLabel X-Label + notmuch config set index.header.DeliveredTo Delivered-To + notmuch config set index.header.DeliveryDate Delivery-date + notmuch config set index.header.To To + notmuch config set index.header.Cc Cc + notmuch config set index.header.Sender Sender + notmuch config set index.header.InReplyTo In-Reply-To + notmuch config set index.header.ReplyTo Reply-To + notmuch new Note: Ignoring non-mail file: /home/grfz/Mail/rmlint.sh~ Note: Ignoring non-mail file: /home/grfz/Mail/postponed.mbox Note: Ignoring non-mail file: /home/grfz/Mail/drafts.mbox Note: Ignoring non-mail file: /home/grfz/Mail/rmlint.sh Welcome to a new version of notmuch! Your database will now be upgraded. This process is safe to interrupt. Backing up tags to /home/grfz/Mail/.notmuch/dump-20190530T135332.gz... Your notmuch database has now been upgraded. Processed 1037014 total files in 2h 42m 32s (106 files/sec.). Added 945937 new messages to the database. + notmuch tag -new -inbox -unread '*' + notmuch restore --accumulate --format=batch-tag --input=/home/grfz/Mail/.notmuch/notmuch.dump.2019-05-30.1559214465 ++ date +%F.%s + notmuch dump --gzip --output=/home/grfz/Mail/.notmuch/notmuch.dump.2019-05-30.1559241909 ++ date +%F.%s + notmuch compact --backup=/home/grfz/Mail/.notmuch/xapian.2019-05-30.1559241950 Compacting database... compacting table postlist Reduced by 36% 1001496K (2707600K -> 1706104K) compacting table docdata Reduced by 51% 336K (656K -> 320K) compacting table termlist Reduced by 47% 1837496K (3836392K -> 1998896K) compacting table position Reduced by 45% 3875240K (8440616K -> 4565376K) compacting table spelling doesn't exist compacting table synonym doesn't exist The old database has been moved to /home/grfz/Mail/.notmuch/xapian.2019-05-30.1559241950. Done. + sed -i -e 's/^exit 0$/#exit 0/' /home/grfz/bin/notmuch-new.screen-locked--low-load ++ date + echo Start @ Sa 1. Jun 16:50:23 CEST 2019 Start @ Sa 1. Jun 16:50:23 CEST 2019 + /home/grfz/bin/my-fetchmail.services stop + /home/grfz/bin/notmuch-new.screen-locked--low-load + sed -i -e 's/^#exit 0$/exit 0/' /home/grfz/bin/notmuch-new.screen-locked--low-load + sleep 77 ++ date +%F.%s + TAGSpreREINDEX=/home/grfz/Mail/.notmuch/notmuch.dump.2019-06-01.1559400706 + notmuch dump --gzip --format=batch-tag --output=/home/grfz/Mail/.notmuch/notmuch.dump.2019-06-01.1559400706 ++ date +%F.%s + mv /home/grfz/Mail/.notmuch/xapian /home/grfz/Mail/.notmuch/xapian.2019-06-01.1559400744 + rm -rf /home/grfz/Mail/.notmuch/xapian + notmuch config set query.all '(is:spam OR NOT is:spam OR is:deleted OR NOT is:deleted)' + notmuch config set query.izt-all 'query:all AND (is:izt OR (NOT is:izt AND (path:IZT-EDV/** OR path:IZT/**)))' + notmuch config set query.izt-edv '(is:izt OR path:IZT-EDV/**) AND (from:science-computing OR to:science-computing OR from:ATOS OR to:ATOS OR to:izt@2cal.de OR from:info@ralf-voegtle.de OR from:info@andresedv.de)' + notmuch config set query.izt-robots 'is:izt AND NOT path:IZT/** AND NOT path:sent/** AND NOT query:izt-edv' + notmuch config set query.izt '(is:izt OR path:IZT/**) AND NOT path:IZT-EDV/** AND NOT query:izt-edv)' + notmuch config set index.header.List List-Id + notmuch config set index.header.Spamgrfz X-Spam-grfz-Status + notmuch config set index.header.UserAgent User-Agent + notmuch config set index.header.XLabel X-Label + notmuch config set index.header.DeliveredTo Delivered-To + notmuch config set index.header.DeliveryDate Delivery-date + notmuch config set index.header.To To + notmuch config set index.header.Cc Cc + notmuch config set index.header.Sender Sender + notmuch config set index.header.InReplyTo In-Reply-To + notmuch config set index.header.ReplyTo Reply-To + notmuch config set index.header.Received Received + notmuch new Note: Ignoring non-mail file: /home/grfz/Mail/rmlint.sh~ Note: Ignoring non-mail file: /home/grfz/Mail/postponed.mbox Note: Ignoring non-mail file: /home/grfz/Mail/drafts.mbox Note: Ignoring non-mail file: /home/grfz/Mail/rmlint.sh Welcome to a new version of notmuch! Your database will now be upgraded. This process is safe to interrupt. Backing up tags to /home/grfz/Mail/.notmuch/dump-20190601T145224.gz... Your notmuch database has now been upgraded. Processed 1037381 total files in 2h 54m 22s (99 files/sec.). Added 946288 new messages to the database. + notmuch tag -inbox -unread '*' + notmuch restore --accumulate --format=batch-tag --input=/home/grfz/Mail/.notmuch/notmuch.dump.2019-06-01.1559400706 ++ date +%F.%s + notmuch dump --gzip --output=/home/grfz/Mail/.notmuch/notmuch.dump.2019-06-01.1559419342 ++ date +%F.%s + notmuch compact --backup=/home/grfz/Mail/.notmuch/xapian.2019-06-01.1559419387 Compacting database... compacting table postlist Reduced by 37% 1077536K (2904400K -> 1826864K) compacting table docdata Reduced by 48% 304K (624K -> 320K) compacting table termlist Reduced by 48% 2004432K (4166496K -> 2162064K) compacting table position Reduced by 46% 4498968K (9748816K -> 5249848K) compacting table spelling doesn't exist compacting table synonym doesn't exist The old database has been moved to /home/grfz/Mail/.notmuch/xapian.2019-06-01.1559419387. Done. + sed -i -e 's/^exit 0$/#exit 0/' /home/grfz/bin/notmuch-new.screen-locked--low-load + sleep 77 + my-fetchmail.services start /home/grfz/bin/my-notmuch-renew: line 48: my-fetchmail.services: command not found ++ date + echo End @ Sa 1. Jun 22:10:51 CEST 2019 End @ Sa 1. Jun 22:10:51 CEST 2019 ++ date + echo Start @ So 2. Jun 00:45:52 CEST 2019 Start @ So 2. Jun 00:45:52 CEST 2019 + /home/grfz/bin/my-fetchmail.services stop + /home/grfz/bin/notmuch-new.screen-locked--low-load + sed -i -e 's/^#exit 0$/exit 0/' /home/grfz/bin/notmuch-new.screen-locked--low-load + sleep 77 ++ date +%F.%s + TAGSpreREINDEX=/home/grfz/Mail/.notmuch/notmuch.dump.2019-06-02.1559429235 + notmuch dump --gzip --format=batch-tag --output=/home/grfz/Mail/.notmuch/notmuch.dump.2019-06-02.1559429235 ++ date + echo Start @ Di 4. Jun 20:54:03 CEST 2019 Start @ Di 4. Jun 20:54:03 CEST 2019 + /home/grfz/bin/my-fetchmail.services stop + /home/grfz/bin/notmuch-new.screen-locked--low-load + sed -i -e 's/^#exit 0$/exit 0/' /home/grfz/bin/notmuch-new.screen-locked--low-load + sleep 77 ++ date +%F.%s + TAGSpreREINDEX=/home/grfz/Mail/.notmuch/notmuch.dump.2019-06-04.1559674534 + notmuch dump --gzip --format=batch-tag --output=/home/grfz/Mail/.notmuch/notmuch.dump.2019-06-04.1559674534 ++ date +%F.%s + mv /home/grfz/Mail/.notmuch/xapian /home/grfz/Mail/.notmuch/xapian.2019-06-04.1559674574 + rm -rf /home/grfz/Mail/.notmuch/xapian + notmuch config set query.all '(is:spam OR NOT is:spam OR is:deleted OR NOT is:deleted)' + notmuch config set query.izt-all 'query:all AND (is:izt OR (NOT is:izt AND (path:IZT-EDV/** OR path:IZT/**)))' + notmuch config set query.izt-edv '(is:izt OR path:IZT-EDV/**) AND (from:science-computing OR to:science-computing OR from:ATOS OR to:ATOS OR to:izt@2cal.de OR from:info@ralf-voegtle.de OR from:info@andresedv.de)' + notmuch config set query.izt-robots 'is:izt AND NOT path:IZT/** AND NOT path:sent/** AND NOT query:izt-edv' + notmuch config set query.izt '(is:izt OR path:IZT/**) AND NOT path:IZT-EDV/** AND NOT query:izt-edv)' + notmuch config set index.header.List List-Id + notmuch config set index.header.Spamgrfz X-Spam-grfz-Status + notmuch config set index.header.UserAgent User-Agent + notmuch config set index.header.XLabel X-Label + notmuch config set index.header.DeliveredTo Delivered-To + notmuch config set index.header.DeliveryDate Delivery-date + notmuch config set index.header.To To + notmuch config set index.header.Cc Cc + notmuch config set index.header.Sender Sender + notmuch config set index.header.InReplyTo In-Reply-To + notmuch config set index.header.ReplyTo Reply-To + notmuch config set index.header.Received Received + notmuch new Note: Ignoring non-mail file: /home/grfz/Mail/postponed.mbox Note: Ignoring non-mail file: /home/grfz/Mail/drafts.mbox Welcome to a new version of notmuch! Your database will now be upgraded. This process is safe to interrupt. Backing up tags to /home/grfz/Mail/.notmuch/dump-20190604T185615.gz... Your notmuch database has now been upgraded. Processed 1034784 total files in 3h 16m 21s (87 files/sec.). Added 944525 new messages to the database. + notmuch tag -inbox -unread '*' + notmuch restore --accumulate --format=batch-tag --input=/home/grfz/Mail/.notmuch/notmuch.dump.2019-06-04.1559674534 ++ date +%F.%s + notmuch dump --gzip --output=/home/grfz/Mail/.notmuch/notmuch.dump.2019-06-05.1559694905 ++ date +%F.%s + notmuch compact --backup=/home/grfz/Mail/.notmuch/xapian.2019-06-05.1559694949 Compacting database... compacting table postlist Reduced by 37% 1073696K (2900248K -> 1826552K) compacting table docdata Reduced by 50% 328K (648K -> 320K) compacting table termlist Reduced by 48% 2003640K (4164728K -> 2161088K) compacting table position Reduced by 46% 4495744K (9743384K -> 5247640K) compacting table spelling doesn't exist compacting table synonym doesn't exist The old database has been moved to /home/grfz/Mail/.notmuch/xapian.2019-06-05.1559694949. Done. + sed -i -e 's/^exit 0$/#exit 0/' /home/grfz/bin/notmuch-new.screen-locked--low-load + sleep 77 + my-fetchmail.services start ++ date + echo End @ Mi 5. Jun 02:44:58 CEST 2019 End @ Mi 5. Jun 02:44:58 CEST 2019 From: Gregor Zattler To: notmuch Cc: Bcc: Gregor Zattler Subject: out of memory on "empty" machine (was: Re: consistent database corruption with notmuch new) Reply-To: In-Reply-To: <20201214192251.GA7858@no.workgroup> Hi notmuch, * Gregor Zattler [14. Dez. 2020]: > Hi notmuch developers, > * Gregor Zattler [13. Dez. 2020]: >> * Gregor Zattler [13. Dez. 2020]: >>> I now do a notmuch new with libxapian30 version 1.4.17-1 >>> and will report back in a few hours. >> >> The result is only slightly different from version 1.4.11: > > actually now I realized, that notmuch was not linked against > libxapian v1.4.17-1. > > Now I build notmuch from master and linked it with > libxapian 1.4.17-1. The result is almost the same, though: > 0 (master *) grfz@no:~/Mail/.notmuch$ nice ionice -c3 notmuch new --full-scan ; nice ionice -c3 notmuch new --full-scan ; nice ionice -c3 notmuch new --full-scan > Welcome to a new version of notmuch! Your database will now be upgraded. > This process is safe to interrupt. > Backing up tags to /home/grfz/Mail/.notmuch/dump-20201214T124836.gz... > Your notmuch database has now been upgraded. > Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607940473.9509_1.no:2,S > Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607943993.24776_1.no:2, > Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607947606.8134_1.no:2, > Note: Ignoring non-mail file: /home/grfz/Mail/drafts.mbox > Note: Ignoring non-mail file: /home/grfz/Mail/postponed.mbox > Processed 1176599 total files in 3h 31m 23s (92 files/sec.). > Added 1102787 new messages to the database. > Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607940473.9509_1.no:2,S > Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607943993.24776_1.no:2, > Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607947606.8134_1.no:2, > Note: Ignoring non-mail file: /home/grfz/Mail/drafts.mbox > Note: Ignoring non-mail file: /home/grfz/Mail/postponed.mbox > Processed 125008 total files in 44m 59s (46 files/sec.). > Added 83898 new messages to the database. > Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607940473.9509_1.no:2,S > Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607943993.24776_1.no:2, > Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607947606.8134_1.no:2, > add_file: A Xapian exception occurred). > A Xapian exception occurred at lib/message.cc:1182: Too few chunks of compressed data > Processed 122480 total files in 42m 1s (48 files/sec.). > Added 83907 new messages to the database. > Note: A fatal error was encountered: A Xapian exception occurred > 1 (master *) grfz@no:~/Mail/.notmuch$ > > notmuch new still corrupts the database, the second notmuch new > invocation finds emails the first did not find. > > > Ciao, Gregor > -- > -... --- .-. . -.. ..--.. ...-.- > > Ciao, Gregor -- -... --- .-. . -.. ..--.. ...-.-