From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp1 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms11 with LMTPS id WwrdMjqQGmAGJgAA0tVLHw (envelope-from ) for ; Wed, 03 Feb 2021 11:59:54 +0000 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp1 with LMTPS id GCBdLjqQGmAcMQAAbx9fmQ (envelope-from ) for ; Wed, 03 Feb 2021 11:59:54 +0000 Received: from mail.notmuchmail.org (nmbug.tethera.net [144.217.243.247]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (2048 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id C30DF9403EA for ; Wed, 3 Feb 2021 11:59:53 +0000 (UTC) Received: from nmbug.tethera.net (localhost [127.0.0.1]) by mail.notmuchmail.org (Postfix) with ESMTP id 8318C1FBA6; Wed, 3 Feb 2021 06:59:47 -0500 (EST) Received: from fethera.tethera.net (fethera.tethera.net [IPv6:2607:5300:60:c5::1]) by mail.notmuchmail.org (Postfix) with ESMTP id ED0A21FBA4 for ; Wed, 3 Feb 2021 06:59:44 -0500 (EST) Received: by fethera.tethera.net (Postfix, from userid 1001) id D8A5D6081A; Wed, 3 Feb 2021 06:59:44 -0500 (EST) Received: (nullmailer pid 758311 invoked by uid 1000); Wed, 03 Feb 2021 11:59:43 -0000 From: David Bremner To: Gregor Zattler , notmuch Subject: Re: out of memory on idle machine In-Reply-To: <20210131081638.GA5640@no.workgroup> References: <20201213131909.GD21521@no.workgroup> <87zh2hhk15.fsf@tethera.net> <20201213141543.GE21521@no.workgroup> <20201213151336.GF21521@no.workgroup> <20201213212252.GH21521@no.workgroup> <20201214192251.GA7858@no.workgroup> <20210130085432.GA14025@no.workgroup> <87bld6shrk.fsf@tethera.net> <20210131081638.GA5640@no.workgroup> X-List-To: notmuch Date: Wed, 03 Feb 2021 07:59:43 -0400 Message-ID: <87sg6dweds.fsf@tethera.net> MIME-Version: 1.0 Message-ID-Hash: CY6EX4T7IGG7EHUVQ4BHSKFPKWXSIPUW X-Message-ID-Hash: CY6EX4T7IGG7EHUVQ4BHSKFPKWXSIPUW X-MailFrom: david@tethera.net X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; header-match-notmuch.notmuchmail.org-0; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; suspicious-header CC: xapian-discuss@lists.xapian.org X-Mailman-Version: 3.2.1 Precedence: list List-Id: "Use and development of the notmuch mail system." List-Help: List-Post: List-Subscribe: List-Unsubscribe: Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Migadu-Flow: FLOW_IN X-Migadu-Spam-Score: -1.99 Authentication-Results: aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of notmuch-bounces@notmuchmail.org designates 144.217.243.247 as permitted sender) smtp.mailfrom=notmuch-bounces@notmuchmail.org X-Migadu-Queue-Id: C30DF9403EA X-Spam-Score: -1.99 X-Migadu-Scanner: scn0.migadu.com X-TUID: FgpsKqy7Q9oi Gregor Zattler writes: > > Installed notmuch-dbgsym (0.28.4-1) and gdb. > > grfz@mic:/etc$ gdb --args notmuch new > [...] > (gdb) b notmuch-new.c:420 > Breakpoint 1 at 0x10601: file notmuch-new.c, line 421. > (gdb) run > Starting program: /usr/bin/notmuch new > [Thread debugging using libthread_db enabled] > Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". > add_file: A Xapian exception occurred > A Xapian exception occurred finding message: Db block overwritten - are there multiple writers?. > Processed 24 total files in almost no time. > Added 23 new messages to the database. > Note: A fatal error was encountered: A Xapian exception occurred > [Inferior 1 (process 22756) exited with code 01] > (gdb) > > This time it's no OOM it's a xapian exeption again. > > I have included the Xapian list in copy in case that message rings a bell. I guess you know there are not multiple writers in your setup. Olly Betts mentioned in a different thread that he will build a version of xapian 1.4.18 for buster backports, so trying with that is probably a good step when it is available. > grfz@mic:~/Mail/.notmuch$ mv xapian xapian-corrupted > grfz@mic:~/Mail/.notmuch$ notmuch new > Welcome to a new version of notmuch! Your database will now be upgraded. > This process is safe to interrupt. > Backing up tags to /home/grfz/Mail/.notmuch/dump-20210130T170349.gz... > Your notmuch database has now been upgraded. > Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607947606.8134_1.no:2, > Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607940473.9509_1.no:2,S > Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607969276.21046_1.no:2, > Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607987211.1395_1.no:2, > Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607979988.4942_1.no:2, > Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607972847.4857_1.no:2, > Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607943993.24776_1.no:2, > Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607976389.23296_1.no:2, > Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607983586.19063_1.no:2, > Note: Ignoring non-mail file: /home/grfz/Mail/drafts.mbox > Note: Ignoring non-mail file: /home/grfz/Mail/postponed.mbox > Processed 1183682 total files in 13h 38m 31s (24 files/sec.). > Added 1091038 new messages to the database. > > I then installed xapian-tools amd64 1.4.11-1. > > grfz@mic:~/Mail/.notmuch$ stat --format "%Y" ~/Mail/inbox/cur > 1611646289 > > grfz@mic:~/Mail/.notmuch$ quest -bdir:XDIRECTORY -d ~/Mail/.notmuch/xapian/ dir:inbox/cur > Parsed Query: Query(0 * XDIRECTORYinbox/cur) > MSet: > > That's it, there is data missing in the database. > This could either be a logic error in Notmuch, You can get a complete list of all of the directory documents in the notmuch database with % xapian-delve -1 -A XDIRECTORY ~/Mail/.notmuch/xapian | sort -u > delve.txt You can get a list of the actual directories with % find ~/Mail -type d -not empty | sed s,/home/grfz/Mail/,XDIRECTORY, |sort -u >find.txt Comparing those two lists may give you some hints. Any directory that shows up in the second list but not the first should have no files in it (but potentially other directories) or be ignored either implicitly (.notmuch, .notmuch/xapian) or explicitely by your configuration.