Re: out of memory on idle machine

[David Bremner <david@tethera.net>]

Gregor Zattler <telegraph@gmx.net> writes:
>
> Installed notmuch-dbgsym (0.28.4-1) and gdb.
>
> grfz@mic:/etc$ gdb --args notmuch new
> [...]
> (gdb) b notmuch-new.c:420
> Breakpoint 1 at 0x10601: file notmuch-new.c, line 421.
> (gdb) run
> Starting program: /usr/bin/notmuch new
> [Thread debugging using libthread_db enabled]
> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
> add_file: A Xapian exception occurred
> A Xapian exception occurred finding message: Db block overwritten - are there multiple writers?.
> Processed 24 total files in almost no time.
> Added 23 new messages to the database.
> Note: A fatal error was encountered: A Xapian exception occurred
> [Inferior 1 (process 22756) exited with code 01]
> (gdb)
>
> This time it's no OOM it's a xapian exeption again.
>
>

I have included the Xapian list in copy in case that message rings a
bell. I guess you know there are not multiple writers in your setup.
Olly Betts mentioned in a different thread that he will build a version
of xapian 1.4.18 for buster backports, so trying with that is probably a
good step when it is available.


> grfz@mic:~/Mail/.notmuch$ mv xapian xapian-corrupted
> grfz@mic:~/Mail/.notmuch$ notmuch new
> Welcome to a new version of notmuch! Your database will now be upgraded.
> This process is safe to interrupt.
> Backing up tags to /home/grfz/Mail/.notmuch/dump-20210130T170349.gz...
> Your notmuch database has now been upgraded.
> Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607947606.8134_1.no:2,
> Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607940473.9509_1.no:2,S
> Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607969276.21046_1.no:2,
> Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607987211.1395_1.no:2,
> Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607979988.4942_1.no:2,
> Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607972847.4857_1.no:2,
> Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607943993.24776_1.no:2,
> Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607976389.23296_1.no:2,
> Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607983586.19063_1.no:2,
> Note: Ignoring non-mail file: /home/grfz/Mail/drafts.mbox
> Note: Ignoring non-mail file: /home/grfz/Mail/postponed.mbox
> Processed 1183682 total files in 13h 38m 31s (24 files/sec.).
> Added 1091038 new messages to the database.
>
> I then installed xapian-tools amd64 1.4.11-1.
>
> grfz@mic:~/Mail/.notmuch$ stat --format "%Y"  ~/Mail/inbox/cur
> 1611646289
>
> grfz@mic:~/Mail/.notmuch$ quest -bdir:XDIRECTORY -d ~/Mail/.notmuch/xapian/ dir:inbox/cur
> Parsed Query: Query(0 * XDIRECTORYinbox/cur)
> MSet:
>
> That's it, there is data missing in the database.
>

This could either be a logic error in Notmuch, 

You can get a complete list of all of the directory documents in the
notmuch database with

% xapian-delve -1 -A XDIRECTORY ~/Mail/.notmuch/xapian | sort -u > delve.txt

You can get a list of the actual directories with

% find ~/Mail -type d -not empty | sed s,/home/grfz/Mail/,XDIRECTORY, |sort -u  >find.txt

Comparing those two lists may give you some hints. Any directory that
shows up in the second list but not the first should have no files in it
(but potentially other directories) or be ignored either implicitly
(.notmuch, .notmuch/xapian) or explicitely by your configuration.