consistent database corruption with notmuch new

[Gregor Zattler <telegraph@gmx.net>]

Dear notmuch developers, I need help because notmuch
new on my configured notmuch system without
index/database consistently produces a corrupted
database.

On Friday my notmuch database got corrupted.
xapian-check xapian F did not fix the database.

I moved the xapian directory to xapian-corrupted and
did a notmuch new.  This consistently yields a
corrupted database after some hours (no other notmuch
running, no mail delivery, no cron jobs etc):

    1 (master *) grfz@no:~/Mail/.notmuch$ rm -rf xapian/
    0 (master *) grfz@no:~/Mail/.notmuch$ time nice ionice -c3 notmuch new ; time nice ionice -c3 notmuch new ; time nice ionice -c3 notmuch new ; zcat notmuch.dump.2020-12-11.1607641607 | notmuch tag --batch ; notmuch tag --input=notmuch-doit-batch-flow-via-procmail-since-11 ; notmuch tag --input=notmuch-doit-batch-flow-via-procmail ; notmuch tag --input=notmuch-doit-batch-flow-via-procmail.with-failure ; time nice ionice -c3 notmuch new ; time nice ionice -c3 notmuch new ; time nice ionice -c3 notmuch new
    Welcome to a new version of notmuch! Your database will now be upgraded.
    This process is safe to interrupt.
    Backing up tags to /home/grfz/Mail/.notmuch/dump-20201212T212701.gz...
    Your notmuch database has now been upgraded.
    Note: Ignoring non-mail file: /home/grfz/Mail/drafts.mbox
    Note: Ignoring non-mail file: /home/grfz/Mail/postponed.mbox
    Processed 1175458 total files in 2h 47m 16s (117 files/sec.).
    Added 1101746 new messages to the database.

    real    167m16,589s
    user    79m23,083s
    sys     6m19,064s
    Processed 124685 total files in 29m 21s (70 files/sec.).
    Added 83646 new messages to the database.

    real    29m25,353s
    user    10m24,737s
    sys     2m14,977s
    add_file: A Xapian exception occurred).
    A Xapian exception occurred at lib/message.cc:1182: Expected block 325393 to be level 0, not 1
    Processed 122555 total files in 30m 46s (66 files/sec.).
    Added 83636 new messages to the database.
    Note: A fatal error was encountered: A Xapian exception occurred

    real    30m49,774s
    user    9m44,716s
    sys     2m3,859s
    adding tag frommeMessage-ID: (null)
    Status: A Xapian exception occurred
    add_file: A Xapian exception occurred).
    A Xapian exception occurred at lib/message.cc:1182: Too few chunks of compressed data
    Processed 122555 total files in 30m 38s (66 files/sec.).
    Added 83636 new messages to the database.
    Note: A fatal error was encountered: A Xapian exception occurred

    real    30m42,427s
    user    10m22,614s
    sys     2m2,737s
    add_file: A Xapian exception occurred).
    A Xapian exception occurred at lib/message.cc:1182: Too few chunks of compressed data
    Processed 122555 total files in 31m 39s (64 files/sec.).
    Added 83636 new messages to the database.
    Note: A fatal error was encountered: A Xapian exception occurred

    real    31m43,529s
    user    10m33,959s
    sys     2m7,378s
    add_file: A Xapian exception occurred).
    A Xapian exception occurred at lib/message.cc:1182: Too few chunks of compressed data
    Processed 122555 total files in 30m 44s (66 files/sec.).
    Added 83636 new messages to the database.
    Note: A fatal error was encountered: A Xapian exception occurred

    real    30m48,443s
    user    10m14,171s
    sys     2m8,209s


While the first notmuch new run seems OK, there at least
is the question why the second one adds another 83646
new messages.

This happened several times in a row.

This is a Thinkpad x240, 8GB RAM on debian buster but with
notmuch compiled from source (0.31.2+28~gadfded9).

I think the SSD is OK (no hints to SSD failures in log
files, smart tests OK, I even filled the free rest of
the SSD with randomm data and after that compared it
to the source with no difference.


Any idea what to do in order to get a running notmuch
mail system going?  If it's interesting to nomuch
development, what happens here, I'm happy to so some
digging if someone gives me instructions.  I'm not a
developer and have no clue how to debug this.

Ciao, Gregor
--
 -... --- .-. . -.. ..--.. ...-.-