Re: out of memory on idle machine

[Gregor Zattler <telegraph@gmx.net>]

Hi David, notmuch developers,

thanks a lot: your second idea this time showed a database
corruption instead of an OOM.

The first idea showed, that there is data missing in the
database:

* David Bremner <david@tethera.net> [30. Jan. 2021]:
Since idea #1 involves removing the xapian database I first
tried idea #2:
> Idea #2
> -------
>
> Try to figure out if some specific file is causing the OOM.
>
> Run notmuch-new in gdb
>
> There is a check for NOTMUCH_STATUS_OUT_OF_MEMORY around line 419/420 of
> notmuch-new.c. If I understand correctly, that is where things are
> failing. The following is untested; you will need the package
> notmuch-dbgsym installed [1]
>
> % gdb --args notmuch new
> (gdb) b notmuch-new.c:420
> (gdb) run
> (gdb) p filename
> [1]: https://wiki.debian.org/AutomaticDebugPackages

Installed notmuch-dbgsym (0.28.4-1) and gdb.

grfz@mic:/etc$ gdb --args notmuch new
[...]
(gdb) b notmuch-new.c:420
Breakpoint 1 at 0x10601: file notmuch-new.c, line 421.
(gdb) run
Starting program: /usr/bin/notmuch new
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
add_file: A Xapian exception occurred
A Xapian exception occurred finding message: Db block overwritten - are there multiple writers?.
Processed 24 total files in almost no time.
Added 23 new messages to the database.
Note: A fatal error was encountered: A Xapian exception occurred
[Inferior 1 (process 22756) exited with code 01]
(gdb)

This time it's no OOM it's a xapian exeption again.



Now for idea #1:

> Idea #1
> -------
>
> There are several mysteries here, but maybe we should begin at the
> beginning. Something is wrong if notmuch scans your entire mail tree the
> second time you run notmuch new.
>
> Notmuch checks the mtime of directories against the time stored in the
> database. As a sanitity check, maybe you can do that for one of your
> directories with many messages. This needs "quest" and "xapian-delve",
> from the package xapian-tools.
>
> Unfortunately this should probably be done after the first notmuch
> new. I have another idea to try (below) in the state after several news where
> you are getting OOM.
>
> I'll use real paths for my system; you'll need to update them.
>
> This gives a time in seconds
>
> % stat --format "%Y" ~/Maildir/tethera/cur
> 1612008734
>
> Now let us find the database document for that directory
>
> % quest -bdir:XDIRECTORY -d ~/Maildir/.notmuch/xapian/ dir:tethera/cur
>
> Parsed Query: Query(0 * XDIRECTORYtethera/cur)
> Exactly 1 matches
> MSet:
> 431067: [0]
> tethera/cur
>
> Grabbing the record number from the output of quest:
>
> % xapian-delve -r 431067 -VS0 ~/Maildir/.notmuch/xapian
>
> Value 0 for record #431067: 1.61201e+09
> Term List for record #431067: XDDIRENTRY387045:cur XDIRECTORYtethera/cur
>
> You can see the value matches the mtime up to 6 decimal places.

grfz@mic:~/Mail/.notmuch$ mv xapian xapian-corrupted
grfz@mic:~/Mail/.notmuch$ notmuch new
Welcome to a new version of notmuch! Your database will now be upgraded.
This process is safe to interrupt.
Backing up tags to /home/grfz/Mail/.notmuch/dump-20210130T170349.gz...
Your notmuch database has now been upgraded.
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607947606.8134_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607940473.9509_1.no:2,S
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607969276.21046_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607987211.1395_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607979988.4942_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607972847.4857_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607943993.24776_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607976389.23296_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607983586.19063_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/drafts.mbox
Note: Ignoring non-mail file: /home/grfz/Mail/postponed.mbox
Processed 1183682 total files in 13h 38m 31s (24 files/sec.).
Added 1091038 new messages to the database.

I then installed xapian-tools amd64 1.4.11-1.

grfz@mic:~/Mail/.notmuch$ stat --format "%Y"  ~/Mail/inbox/cur
1611646289

grfz@mic:~/Mail/.notmuch$ quest -bdir:XDIRECTORY -d ~/Mail/.notmuch/xapian/ dir:inbox/cur
Parsed Query: Query(0 * XDIRECTORYinbox/cur)
MSet:

That's it, there is data missing in the database.



I have no clue why this is the case.  Just in case here is

grfz@mic:~$ dpkg -l notmuch libnotmuch5 libc6 libglib2.0-0 libgmime-3.0-0 libtalloc2 zlib1g libxapian30
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                 Version          Architecture Description
+++-====================-================-============-======================================================
ii  libc6:amd64          2.28-10          amd64        GNU C Library: Shared libraries
ii  libglib2.0-0:amd64   2.58.3-2+deb10u2 amd64        GLib library of C routines
ii  libgmime-3.0-0:amd64 3.2.1-1          amd64        MIME message parser and creator library
ii  libnotmuch5          0.28.4-1         amd64        thread-based email index, search and tagging (runtime)
ii  libtalloc2:amd64     2.1.14-2         amd64        hierarchical pool based memory allocator
ii  libxapian30:amd64    1.4.11-1         amd64        Search engine library
ii  notmuch              0.28.4-1         amd64        thread-based email index, search and tagging
ii  zlib1g:amd64         1:1.2.11.dfsg-1  amd64        compression library - runtime


and

grfz@mic:~$ sudo tune2fs -l /dev/mapper/bcache0p2_crypt
tune2fs 1.44.5 (15-Dec-2018)
Filesystem volume name:   data
Last mounted on:          /home/grfz/data
Filesystem UUID:          a15890ca-4817-418b-923a-fa6849855a1e
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal ext_attr resize_inode dir_index sparse_super2 filetype needs_recovery extent 64bit mmp flex_bg sparse_super large_file huge_file dir_nlink extra_isize metadata_csum
Filesystem flags:         signed_directory_hash
Default mount options:    user_xattr acl
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              58589184
Block count:              234355200
Reserved block count:     11717760
Free blocks:              230413509
Free inodes:              58589173
First block:              0
Block size:               4096
Fragment size:            4096
Group descriptor size:    64
Reserved GDT blocks:      1024
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         8192
Inode blocks per group:   512
Flex block group size:    16
Filesystem created:       Mon Jan 25 21:44:28 2021
Last mount time:          Mon Jan 25 22:13:54 2021
Last write time:          Mon Jan 25 22:13:54 2021
Mount count:              3
Maximum mount count:      -1
Last checked:             Mon Jan 25 21:44:28 2021
Check interval:           0 (<none>)
Lifetime writes:          1032 MB
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:               256
Required extra isize:     32
Desired extra isize:      32
Journal inode:            8
Default directory hash:   half_md4
Directory Hash Seed:      ea7647de-8706-4db8-b45b-a0ba39fc193b
Journal backup:           inode blocks
Backup block groups:      1 7151
MMP block number:         9367
MMP update interval:      5
Checksum type:            crc32c
Checksum:                 0x62d3b045


As you see, it's pretty new.  If you think the non-standard
features play a role here, I can simply reformat the
pratition and start over.  The machine has no other purpose
atm.

I'll redo idea #2 on my laptop and will report it's results.

Ciao, Gregor
--
 -... --- .-. . -.. ..--.. ...-.-