From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp1 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms11 with LMTPS id 8LlaCnxnFmBlSQAA0tVLHw (envelope-from ) for ; Sun, 31 Jan 2021 08:17:00 +0000 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp1 with LMTPS id MFULBnxnFmAjAgAAbx9fmQ (envelope-from ) for ; Sun, 31 Jan 2021 08:17:00 +0000 Received: from mail.notmuchmail.org (nmbug.tethera.net [IPv6:2607:5300:201:3100::1657]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (2048 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 1C651940111 for ; Sun, 31 Jan 2021 08:16:59 +0000 (UTC) Received: from nmbug.tethera.net (localhost [127.0.0.1]) by mail.notmuchmail.org (Postfix) with ESMTP id 9CA9828CDF; Sun, 31 Jan 2021 03:16:50 -0500 (EST) Received: from mout.gmx.net (mout.gmx.net [212.227.15.19]) by mail.notmuchmail.org (Postfix) with ESMTPS id D818028CD0 for ; Sun, 31 Jan 2021 03:16:47 -0500 (EST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=gmx.net; s=badeba3b8450; t=1612081006; bh=LFW2AsDPwp05za/L0k2bE6VMKDDxWpiESy5AdlWuwh4=; h=X-UI-Sender-Class:Date:From:To:Subject:References:In-Reply-To; b=Xyd7Ao01JDJ1x3xJhLtBWZarwbej8dKbgL45AJExJWtnjz65lIZTBrspsTs0EyhjX 84wWJw7MVg2nBGA4WBTT3/GseQpLcfg72SS8rOv91Hss/yd/UZTrTx+o9SCdbjDdWC Kw2UNmdevJ89OBPwZ2KUDZSvWe+PT+Tqturhf2Ro= X-UI-Sender-Class: 01bb95c1-4bf8-414a-932a-4f6e2808ef9c Received: from no.workgroup ([84.185.97.253]) by mail.gmx.net (mrgmx004 [212.227.17.190]) with ESMTPSA (Nemesis) id 1MSt8W-1lWyPr3Rjq-00UKiY for ; Sun, 31 Jan 2021 09:16:45 +0100 Date: Sun, 31 Jan 2021 09:16:39 +0100 From: Gregor Zattler To: notmuch Subject: Re: out of memory on idle machine Message-ID: <20210131081638.GA5640@no.workgroup> Mail-Followup-To: notmuch References: <20201213131909.GD21521@no.workgroup> <87zh2hhk15.fsf@tethera.net> <20201213141543.GE21521@no.workgroup> <20201213151336.GF21521@no.workgroup> <20201213212252.GH21521@no.workgroup> <20201214192251.GA7858@no.workgroup> <20210130085432.GA14025@no.workgroup> <87bld6shrk.fsf@tethera.net> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <87bld6shrk.fsf@tethera.net> X-Provags-ID: V03:K1:jsstYOphuIoeDw/0IPzFGGz5c9MBqA8jdcna6BN9LtzN4/BDmZC gPl+ckm0EwwWbkQr66MimEJ94DXBCPbZ5YLE6CZehpLGhhopOe/klI5Y9GtPYUQcQ2lkbSJ eBcIpd6HyZ9/Fz8h4QRCSJbif8dAxH9Zl3WavAHLxaPXvuL2I9NHiab1va7Qp7QiOCxkwnn N2FkDyjT6p+uZwzZ0+z2Q== X-Spam-Flag: NO X-UI-Out-Filterresults: notjunk:1;V03:K0:AG5/amDPwaY=:biiZX6uAp/F8gRTvgsYjkM Hftsr37oMB73NZUdNHBuJmEN0dcNkOnQHwFE6YUpT8y3lrsenTDuLeuyh+EP3ymgiotVb69rm PBfHye/REeMMYwj9FTh8QPWBjOcVjM/OS+gGPAiJO2OAWnysILny4cjlajMP9PAETOhwjaYGX RCI5LU+Q8yYS8lK1HUQge+305F9jqlCN117s63Ro4bsskMUn/J1O1Nuc+ageDVcLPXoHi7Azw +waUgn+IfsWX7c5vFrlT3cOYX/zGmDDsXMqFtJkWQLBKb627DGALUyLURGXp/Aws4ed0OvXHs 4rFyDQ6VMFAqtDiA7oXakuFwYrdAFRYZKzo6OvR/iZcpHcPUCdRz+ekCixAm0h0dTMvLd8J12 c+hlDYtgaM89UId3G/ZwOJTTL22ClLKwrQUf022sX2b1tAd+yZxTQkd37nmDY1t6A1A8E9Kc7 HCnJz+ryZSVZwrfUuXJXstbvypPppIDe+MymH9hPesvh5/7EVnuwHzm1kdRgD+ErDcg+CMiyn PNMQ1xevAfSykcsL5z2VMQSEn8N4Jg0HBNUiYFjf9cukHRjuE40+AWFrQZWJLlZhOLzkg30XI xihLcKCJizzimp5w1E4PCRWmUh4ymSZRTEye/ELAXLOm0lxwzETQCRzaqq0jOhOVt2/s0KSSb w4/G3s4c1chN9RGZLH3CzmiLEFlgke6GxZtgXNJF5mKZg+mbDAqQAA2+NBFyaCtuGky1qxjHm Po8DBzn3pg4ryqqrfd8JMVET/be4hhw7O/iDcw4TvGJY6PgENf66FCmeezicsYK9mY5qBMYEd hVIL+Hna7pYUzCAiJg952tVXE5fF4NDzgzybeD11EhZMS3SK2CKuwXtWI4GMkMqzLj8H+ZJFa ix2vcajVy7J7i01wFOfg== Message-ID-Hash: FKY6GEMZBZIIVPNESY2D2VLD5UEYYF5B X-Message-ID-Hash: FKY6GEMZBZIIVPNESY2D2VLD5UEYYF5B X-MailFrom: telegraph@gmx.net X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; header-match-notmuch.notmuchmail.org-0; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; suspicious-header X-Mailman-Version: 3.2.1 Precedence: list List-Id: "Use and development of the notmuch mail system." List-Help: List-Post: List-Subscribe: List-Unsubscribe: Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Migadu-Flow: FLOW_IN X-Migadu-Spam-Score: -0.81 Authentication-Results: aspmx1.migadu.com; dkim=fail ("body hash did not verify") header.d=gmx.net header.s=badeba3b8450 header.b=Xyd7Ao01; dmarc=fail reason="SPF not aligned (relaxed)" header.from=gmx.net (policy=none); spf=pass (aspmx1.migadu.com: domain of notmuch-bounces@notmuchmail.org designates 2607:5300:201:3100::1657 as permitted sender) smtp.mailfrom=notmuch-bounces@notmuchmail.org X-Migadu-Queue-Id: 1C651940111 X-Spam-Score: -0.81 X-Migadu-Scanner: scn1.migadu.com X-TUID: 0hxl8k6Dvd5P Hi David, notmuch developers, thanks a lot: your second idea this time showed a database corruption instead of an OOM. The first idea showed, that there is data missing in the database: * David Bremner [30. Jan. 2021]: Since idea #1 involves removing the xapian database I first tried idea #2: > Idea #2 > ------- > > Try to figure out if some specific file is causing the OOM. > > Run notmuch-new in gdb > > There is a check for NOTMUCH_STATUS_OUT_OF_MEMORY around line 419/420 of > notmuch-new.c. If I understand correctly, that is where things are > failing. The following is untested; you will need the package > notmuch-dbgsym installed [1] > > % gdb --args notmuch new > (gdb) b notmuch-new.c:420 > (gdb) run > (gdb) p filename > [1]: https://wiki.debian.org/AutomaticDebugPackages Installed notmuch-dbgsym (0.28.4-1) and gdb. grfz@mic:/etc$ gdb --args notmuch new [...] (gdb) b notmuch-new.c:420 Breakpoint 1 at 0x10601: file notmuch-new.c, line 421. (gdb) run Starting program: /usr/bin/notmuch new [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". add_file: A Xapian exception occurred A Xapian exception occurred finding message: Db block overwritten - are there multiple writers?. Processed 24 total files in almost no time. Added 23 new messages to the database. Note: A fatal error was encountered: A Xapian exception occurred [Inferior 1 (process 22756) exited with code 01] (gdb) This time it's no OOM it's a xapian exeption again. Now for idea #1: > Idea #1 > ------- > > There are several mysteries here, but maybe we should begin at the > beginning. Something is wrong if notmuch scans your entire mail tree the > second time you run notmuch new. > > Notmuch checks the mtime of directories against the time stored in the > database. As a sanitity check, maybe you can do that for one of your > directories with many messages. This needs "quest" and "xapian-delve", > from the package xapian-tools. > > Unfortunately this should probably be done after the first notmuch > new. I have another idea to try (below) in the state after several news where > you are getting OOM. > > I'll use real paths for my system; you'll need to update them. > > This gives a time in seconds > > % stat --format "%Y" ~/Maildir/tethera/cur > 1612008734 > > Now let us find the database document for that directory > > % quest -bdir:XDIRECTORY -d ~/Maildir/.notmuch/xapian/ dir:tethera/cur > > Parsed Query: Query(0 * XDIRECTORYtethera/cur) > Exactly 1 matches > MSet: > 431067: [0] > tethera/cur > > Grabbing the record number from the output of quest: > > % xapian-delve -r 431067 -VS0 ~/Maildir/.notmuch/xapian > > Value 0 for record #431067: 1.61201e+09 > Term List for record #431067: XDDIRENTRY387045:cur XDIRECTORYtethera/cur > > You can see the value matches the mtime up to 6 decimal places. grfz@mic:~/Mail/.notmuch$ mv xapian xapian-corrupted grfz@mic:~/Mail/.notmuch$ notmuch new Welcome to a new version of notmuch! Your database will now be upgraded. This process is safe to interrupt. Backing up tags to /home/grfz/Mail/.notmuch/dump-20210130T170349.gz... Your notmuch database has now been upgraded. Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607947606.8134_1.no:2, Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607940473.9509_1.no:2,S Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607969276.21046_1.no:2, Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607987211.1395_1.no:2, Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607979988.4942_1.no:2, Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607972847.4857_1.no:2, Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607943993.24776_1.no:2, Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607976389.23296_1.no:2, Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607983586.19063_1.no:2, Note: Ignoring non-mail file: /home/grfz/Mail/drafts.mbox Note: Ignoring non-mail file: /home/grfz/Mail/postponed.mbox Processed 1183682 total files in 13h 38m 31s (24 files/sec.). Added 1091038 new messages to the database. I then installed xapian-tools amd64 1.4.11-1. grfz@mic:~/Mail/.notmuch$ stat --format "%Y" ~/Mail/inbox/cur 1611646289 grfz@mic:~/Mail/.notmuch$ quest -bdir:XDIRECTORY -d ~/Mail/.notmuch/xapian/ dir:inbox/cur Parsed Query: Query(0 * XDIRECTORYinbox/cur) MSet: That's it, there is data missing in the database. I have no clue why this is the case. Just in case here is grfz@mic:~$ dpkg -l notmuch libnotmuch5 libc6 libglib2.0-0 libgmime-3.0-0 libtalloc2 zlib1g libxapian30 Desired=Unknown/Install/Remove/Purge/Hold | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad) ||/ Name Version Architecture Description +++-====================-================-============-====================================================== ii libc6:amd64 2.28-10 amd64 GNU C Library: Shared libraries ii libglib2.0-0:amd64 2.58.3-2+deb10u2 amd64 GLib library of C routines ii libgmime-3.0-0:amd64 3.2.1-1 amd64 MIME message parser and creator library ii libnotmuch5 0.28.4-1 amd64 thread-based email index, search and tagging (runtime) ii libtalloc2:amd64 2.1.14-2 amd64 hierarchical pool based memory allocator ii libxapian30:amd64 1.4.11-1 amd64 Search engine library ii notmuch 0.28.4-1 amd64 thread-based email index, search and tagging ii zlib1g:amd64 1:1.2.11.dfsg-1 amd64 compression library - runtime and grfz@mic:~$ sudo tune2fs -l /dev/mapper/bcache0p2_crypt tune2fs 1.44.5 (15-Dec-2018) Filesystem volume name: data Last mounted on: /home/grfz/data Filesystem UUID: a15890ca-4817-418b-923a-fa6849855a1e Filesystem magic number: 0xEF53 Filesystem revision #: 1 (dynamic) Filesystem features: has_journal ext_attr resize_inode dir_index sparse_super2 filetype needs_recovery extent 64bit mmp flex_bg sparse_super large_file huge_file dir_nlink extra_isize metadata_csum Filesystem flags: signed_directory_hash Default mount options: user_xattr acl Filesystem state: clean Errors behavior: Continue Filesystem OS type: Linux Inode count: 58589184 Block count: 234355200 Reserved block count: 11717760 Free blocks: 230413509 Free inodes: 58589173 First block: 0 Block size: 4096 Fragment size: 4096 Group descriptor size: 64 Reserved GDT blocks: 1024 Blocks per group: 32768 Fragments per group: 32768 Inodes per group: 8192 Inode blocks per group: 512 Flex block group size: 16 Filesystem created: Mon Jan 25 21:44:28 2021 Last mount time: Mon Jan 25 22:13:54 2021 Last write time: Mon Jan 25 22:13:54 2021 Mount count: 3 Maximum mount count: -1 Last checked: Mon Jan 25 21:44:28 2021 Check interval: 0 () Lifetime writes: 1032 MB Reserved blocks uid: 0 (user root) Reserved blocks gid: 0 (group root) First inode: 11 Inode size: 256 Required extra isize: 32 Desired extra isize: 32 Journal inode: 8 Default directory hash: half_md4 Directory Hash Seed: ea7647de-8706-4db8-b45b-a0ba39fc193b Journal backup: inode blocks Backup block groups: 1 7151 MMP block number: 9367 MMP update interval: 5 Checksum type: crc32c Checksum: 0x62d3b045 As you see, it's pretty new. If you think the non-standard features play a role here, I can simply reformat the pratition and start over. The machine has no other purpose atm. I'll redo idea #2 on my laptop and will report it's results. Ciao, Gregor -- -... --- .-. . -.. ..--.. ...-.-