From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp1 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms11 with LMTPS id QLHyIMeJGmDzZQAA0tVLHw (envelope-from ) for ; Wed, 03 Feb 2021 11:32:23 +0000 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp1 with LMTPS id CJCbHMeJGmDgIAAAbx9fmQ (envelope-from ) for ; Wed, 03 Feb 2021 11:32:23 +0000 Received: from mail.notmuchmail.org (nmbug.tethera.net [IPv6:2607:5300:201:3100::1657]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (2048 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 115B7940480 for ; Wed, 3 Feb 2021 11:32:22 +0000 (UTC) Received: from nmbug.tethera.net (localhost [127.0.0.1]) by mail.notmuchmail.org (Postfix) with ESMTP id EF9E328CF6; Wed, 3 Feb 2021 06:32:14 -0500 (EST) Received: from fethera.tethera.net (fethera.tethera.net [198.245.60.197]) by mail.notmuchmail.org (Postfix) with ESMTP id 5B0B728CA2 for ; Wed, 3 Feb 2021 06:32:11 -0500 (EST) Received: by fethera.tethera.net (Postfix, from userid 1001) id D00EA6081A; Wed, 3 Feb 2021 06:32:09 -0500 (EST) Received: (nullmailer pid 757550 invoked by uid 1000); Wed, 03 Feb 2021 11:32:08 -0000 From: David Bremner To: Gregor Zattler , notmuch Subject: Re: out of memory on idle machine In-Reply-To: <20210131202139.GC5640@no.workgroup> References: <20201213131909.GD21521@no.workgroup> <87zh2hhk15.fsf@tethera.net> <20201213141543.GE21521@no.workgroup> <20201213151336.GF21521@no.workgroup> <20201213212252.GH21521@no.workgroup> <20201214192251.GA7858@no.workgroup> <20210130085432.GA14025@no.workgroup> <87bld6shrk.fsf@tethera.net> <20210131081638.GA5640@no.workgroup> <20210131202139.GC5640@no.workgroup> Date: Wed, 03 Feb 2021 07:32:08 -0400 Message-ID: <87v9b9wfnr.fsf@tethera.net> MIME-Version: 1.0 Message-ID-Hash: 2C55PBGJL4P4SS3FFAJBPNA57CXPKO2J X-Message-ID-Hash: 2C55PBGJL4P4SS3FFAJBPNA57CXPKO2J X-MailFrom: david@tethera.net X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; header-match-notmuch.notmuchmail.org-0; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; suspicious-header X-Mailman-Version: 3.2.1 Precedence: list List-Id: "Use and development of the notmuch mail system." List-Help: List-Post: List-Subscribe: List-Unsubscribe: Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Migadu-Flow: FLOW_IN X-Migadu-Spam-Score: -1.92 Authentication-Results: aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of notmuch-bounces@notmuchmail.org designates 2607:5300:201:3100::1657 as permitted sender) smtp.mailfrom=notmuch-bounces@notmuchmail.org X-Migadu-Queue-Id: 115B7940480 X-Spam-Score: -1.92 X-Migadu-Scanner: scn1.migadu.com X-TUID: XOO/SbpKcS1O Gregor Zattler writes: > > 0 (master *) grfz@no:~/Mail/.notmuch$ stat --format "%Y" ~/Mail/inbox/cur > 1612091679 > 0 (master *) grfz@no:~/Mail/.notmuch$ stat --format "%y" ~/Mail/inbox/cur > 2021-01-31 12:14:39.771049424 +0100 > 0 (master *) grfz@no:~/Mail/.notmuch$ > 0 (master *) grfz@no:~/Mail/.notmuch$ quest -bdir:XDIRECTORY -d ~/Mail/.notmuch/xapian/ dir:inbox/cur > bash: quest: command not found > 127 (master *) grfz@no:~/Mail/.notmuch$ quest -bdir:XDIRECTORY -d ~/Mail/.notmuch/xapian/ dir:inbox/cur > Parsed Query: Query(0 * XDIRECTORYinbox/cur) > MSet: > 1114128: [0] > inbox/cur > > 0 (master *) grfz@no:~/Mail/.notmuch$ xapian-delve -r 1114128 -VS0 ~/Mail/.notmuch/xapian > Value 0 for record #1114128: 1.61208e+09 > Term List for record #1114128: XDDIRENTRY1114127:cur XDIRECTORYinbox/cur > > > So I think that's OK on my laptop, after the first notmuch > new. Now I do another one. There is the rounding error in the last digit (8 vs. 9), but I'm not sure if that matters, or if it's just in the output formatting. > Processed 121712 total files in 45m 4s (45 files/sec.). > Added 85345 new messages to the database. Removed 3 messages. > 0 (master *) grfz@no:~/Mail/.notmuch$ > > The Problem remains, but at a different scale: In between > these two notmuch new runs there were only a few hours, > there's no way I received 121712 or 85345 emails in this time > frame. If you use "notmuch new --debug" for your second run that may give you more information about what files are being (re)-added. Be warned that will probably generate copious output, so better to redirect. > 0 (master *) grfz@no:~/Mail/.notmuch$ xapian-delve -r 1114128 -VS0 ~/Mail/.notmuch/xapian-2 > Value 0 for record #1114128: 1.61209e+09 > Term List for record #1114128: XDDIRENTRY1114127:cur XDIRECTORYinbox/cur > 0 (master *) grfz@no:~/Mail/.notmuch$ stat --format "%Y" ~/Mail/inbox/cur > 1612091679 > 0 (master *) grfz@no:~/Mail/.notmuch$ quest -bdir:XDIRECTORY -d ~/Mail/.notmuch/xapian-2/ dir:inbox/cur > Parsed Query: Query(0 * XDIRECTORYinbox/cur) > MSet: > 1114128: [0] > inbox/cur > 0 (master *) grfz@no:~/Mail/.notmuch$ xapian-delve -r 1114128 -VS0 ~/Mail/.notmuch/xapian-2 > Value 0 for record #1114128: 1.61209e+09 > Term List for record #1114128: XDDIRENTRY1114127:cur XDIRECTORYinbox/cur > 0 (master *) grfz@no:~/Mail/.notmuch$ Here I don't see the rounding error, which is (potentially) interesting. Perhaps after a run with --debug we will have some idea of what directory is being rescanned. d