consistent database corruption with notmuch new

consistent database corruption with notmuch new

[Gregor Zattler]

Dear notmuch developers, I need help because notmuch
new on my configured notmuch system without
index/database consistently produces a corrupted
database.

On Friday my notmuch database got corrupted.
xapian-check xapian F did not fix the database.

I moved the xapian directory to xapian-corrupted and
did a notmuch new.  This consistently yields a
corrupted database after some hours (no other notmuch
running, no mail delivery, no cron jobs etc):

    1 (master *) grfz@no:~/Mail/.notmuch$ rm -rf xapian/
    0 (master *) grfz@no:~/Mail/.notmuch$ time nice ionice -c3 notmuch new ; time nice ionice -c3 notmuch new ; time nice ionice -c3 notmuch new ; zcat notmuch.dump.2020-12-11.1607641607 | notmuch tag --batch ; notmuch tag --input=notmuch-doit-batch-flow-via-procmail-since-11 ; notmuch tag --input=notmuch-doit-batch-flow-via-procmail ; notmuch tag --input=notmuch-doit-batch-flow-via-procmail.with-failure ; time nice ionice -c3 notmuch new ; time nice ionice -c3 notmuch new ; time nice ionice -c3 notmuch new
    Welcome to a new version of notmuch! Your database will now be upgraded.
    This process is safe to interrupt.
    Backing up tags to /home/grfz/Mail/.notmuch/dump-20201212T212701.gz...
    Your notmuch database has now been upgraded.
    Note: Ignoring non-mail file: /home/grfz/Mail/drafts.mbox
    Note: Ignoring non-mail file: /home/grfz/Mail/postponed.mbox
    Processed 1175458 total files in 2h 47m 16s (117 files/sec.).
    Added 1101746 new messages to the database.

    real    167m16,589s
    user    79m23,083s
    sys     6m19,064s
    Processed 124685 total files in 29m 21s (70 files/sec.).
    Added 83646 new messages to the database.

    real    29m25,353s
    user    10m24,737s
    sys     2m14,977s
    add_file: A Xapian exception occurred).
    A Xapian exception occurred at lib/message.cc:1182: Expected block 325393 to be level 0, not 1
    Processed 122555 total files in 30m 46s (66 files/sec.).
    Added 83636 new messages to the database.
    Note: A fatal error was encountered: A Xapian exception occurred

    real    30m49,774s
    user    9m44,716s
    sys     2m3,859s
    adding tag frommeMessage-ID: (null)
    Status: A Xapian exception occurred
    add_file: A Xapian exception occurred).
    A Xapian exception occurred at lib/message.cc:1182: Too few chunks of compressed data
    Processed 122555 total files in 30m 38s (66 files/sec.).
    Added 83636 new messages to the database.
    Note: A fatal error was encountered: A Xapian exception occurred

    real    30m42,427s
    user    10m22,614s
    sys     2m2,737s
    add_file: A Xapian exception occurred).
    A Xapian exception occurred at lib/message.cc:1182: Too few chunks of compressed data
    Processed 122555 total files in 31m 39s (64 files/sec.).
    Added 83636 new messages to the database.
    Note: A fatal error was encountered: A Xapian exception occurred

    real    31m43,529s
    user    10m33,959s
    sys     2m7,378s
    add_file: A Xapian exception occurred).
    A Xapian exception occurred at lib/message.cc:1182: Too few chunks of compressed data
    Processed 122555 total files in 30m 44s (66 files/sec.).
    Added 83636 new messages to the database.
    Note: A fatal error was encountered: A Xapian exception occurred

    real    30m48,443s
    user    10m14,171s
    sys     2m8,209s


While the first notmuch new run seems OK, there at least
is the question why the second one adds another 83646
new messages.

This happened several times in a row.

This is a Thinkpad x240, 8GB RAM on debian buster but with
notmuch compiled from source (0.31.2+28~gadfded9).

I think the SSD is OK (no hints to SSD failures in log
files, smart tests OK, I even filled the free rest of
the SSD with randomm data and after that compared it
to the source with no difference.


Any idea what to do in order to get a running notmuch
mail system going?  If it's interesting to nomuch
development, what happens here, I'm happy to so some
digging if someone gives me instructions.  I'm not a
developer and have no clue how to debug this.

Ciao, Gregor
--
 -... --- .-. . -.. ..--.. ...-.-

Re: consistent database corruption with notmuch new

[David Bremner]

Gregor Zattler <telegraph@gmx.net> writes:

>
> This is a Thinkpad x240, 8GB RAM on debian buster but with
> notmuch compiled from source (0.31.2+28~gadfded9).
>
> I think the SSD is OK (no hints to SSD failures in log
> files, smart tests OK, I even filled the free rest of
> the SSD with randomm data and after that compared it
> to the source with no difference.

What version of Xapian are you linking with? Are you using the 1.4.11-1
in Debian buster?

Re: consistent database corruption with notmuch new

[Gregor Zattler]

Hi David,
* David Bremner <david@tethera.net> [13. Dez. 2020]:
> Gregor Zattler <telegraph@gmx.net> writes:
>
>>
>> This is a Thinkpad x240, 8GB RAM on debian buster but with
>> notmuch compiled from source (0.31.2+28~gadfded9).
>>
>> I think the SSD is OK (no hints to SSD failures in log
>> files, smart tests OK, I even filled the free rest of
>> the SSD with randomm data and after that compared it
>> to the source with no difference.
>
> What version of Xapian are you linking with? Are you using the 1.4.11-1
> in Debian buster?


yes.

Ciao, Gregor
--
 -... --- .-. . -.. ..--.. ...-.-

Re: consistent database corruption with notmuch new

[Gregor Zattler]

Hi David, notmuch developers,
* Gregor Zattler <telegraph@gmx.net> [13. Dez. 2020]:
> * David Bremner <david@tethera.net> [13. Dez. 2020]:
>> What version of Xapian are you linking with? Are you using the 1.4.11-1
>> in Debian buster?
>
> yes.

I now got the bullseye version of xapian-core source
package, build it on my buster machine (all 4 tests
passed, expected tests failed).

xapian-check F version 1.4.17-1 did not succed in fixing
the last corrupted database

I now do a notmuch new with libxapian30 version 1.4.17-1
and will report back in a few hours.

Ciao, Gregor
--
 -... --- .-. . -.. ..--.. ...-.-

Re: consistent database corruption with notmuch new

[David Bremner]

Gregor Zattler <telegraph@gmx.net> writes:
>
> I now do a notmuch new with libxapian30 version 1.4.17-1
> and will report back in a few hours.
>

I have just uploaded 0.31.3 to buster-backports in Debian. Unfortunately
it has to have some manual processing because of the new python
bindings, but hopefully in a few days it will be available in Debian
backports. This would be another potential test, to see if the backport
is working well for you.

d

Re: consistent database corruption with notmuch new

[David Bremner]

David Bremner <david@tethera.net> writes:

> Gregor Zattler <telegraph@gmx.net> writes:
>>
>> I now do a notmuch new with libxapian30 version 1.4.17-1
>> and will report back in a few hours.
>>
>
> I have just uploaded 0.31.3 to buster-backports

Sorry, I meant 0.31.2-3.

Re: consistent database corruption with notmuch new

[Gregor Zattler]

Hi David, notmuch developers,
* Gregor Zattler <telegraph@gmx.net> [13. Dez. 2020]:
> I now got the bullseye version of xapian-core source
> package, build it on my buster machine (all 4 tests
> passed, expected tests failed).
>
> xapian-check F version 1.4.17-1 did not succed in fixing
> the last corrupted database
>
> I now do a notmuch new with libxapian30 version 1.4.17-1
> and will report back in a few hours.

The result is only slightly different from version 1.4.11:

0 (master *) grfz@no:~/Mail/.notmuch$ nice ionice -c3 notmuch new ; nice ionice -c3 notmuch new ; nice ionice -c3 notmuch new                                                                                                                                                                                                                                                                  Welcome to a new version of notmuch! Your database will now be upgraded.
This process is safe to interrupt.
Backing up tags to /home/grfz/Mail/.notmuch/dump-20201213T144839.gz...
Your notmuch database has now been upgraded.
Note: Ignoring non-mail file: /home/grfz/Mail/drafts.mbox
Note: Ignoring non-mail file: /home/grfz/Mail/postponed.mbox
Processed 1175849 total files in 3h 1m 40s (107 files/sec.).
Added 1102076 new messages to the database.
Note: Ignoring non-mail file: /home/grfz/Mail/drafts.mbox
Note: Ignoring non-mail file: /home/grfz/Mail/postponed.mbox
Processed 143329 total files in 1h 21m 23s (29 files/sec.).
Added 83756 new messages to the database. Removed 18389 messages. Detected 96 file renames.
Processed 124614 total files in 39m 23s (52 files/sec.).
Added 83712 new messages to the database.
0 (master *) grfz@no:~/Mail/.notmuch$ nice ionice -c3 notmuch new ; nice ionice -c3 notmuch new ; nice ionice -c3 notmuch new
add_file: A Xapian exception occurred
A Xapian exception occurred at lib/message.cc:1182: No termlist for document 1114342
Processed 697 total files in 14s (46 files/sec.).
Added 299 new messages to the database.
Note: A fatal error was encountered: A Xapian exception occurred
add_file: A Xapian exception occurred
A Xapian exception occurred at lib/message.cc:1182: No termlist for document 1114342
Processed 697 total files in 2s (250 files/sec.).
Added 299 new messages to the database.
Note: A fatal error was encountered: A Xapian exception occurred
add_file: A Xapian exception occurred
A Xapian exception occurred at lib/message.cc:1182: No termlist for document 1114342
Processed 697 total files in 2s (252 files/sec.).
Added 299 new messages to the database.
Note: A fatal error was encountered: A Xapian exception occurred
1 (master *) grfz@no:~/Mail/.notmuch$


This time email werde delivered while notmuch
new was running but there were certainly not
83756 new messages or 18389 messages removed
ones.

Since I now for the frist time in years read
email in mutt emails were moved from new/ to cur/
and renamed at the same time, but there were
neither such many new ones nore were thousands
removed.

Next I will test with notmuch from git master,
but this will have to wait a day or two.


Ciao, Gregor
--
 -... --- .-. . -.. ..--.. ...-.-

Re: consistent database corruption with notmuch new

[David Edmondson]

Which filesystem are you using for the mail store?

I had similar looking problems when using ZFS and never got to the
bottom of it.

dme.
-- 
Why does it have to be like this? I can never tell.

Re: consistent database corruption with notmuch new

[Gregor Zattler]

Hi David, notmuch developers,
* David Edmondson <dme@dme.org> [14. Dez. 2020]:
> Which filesystem are you using for the mail store?
>
> I had similar looking problems when using ZFS and never got to the
> bottom of it.

this is a ext4 file system, which gets checked
every time I boot my laptop.

I also run memtest 8.4 (but since it's the gratis
version only for 4 passes) without errors.


Ciao, Gregor
--
 -... --- .-. . -.. ..--.. ...-.-

Re: consistent database corruption with notmuch new

[David Bremner]

David Bremner <david@tethera.net> writes:

> David Bremner <david@tethera.net> writes:
>
>> Gregor Zattler <telegraph@gmx.net> writes:
>>>
>>> I now do a notmuch new with libxapian30 version 1.4.17-1
>>> and will report back in a few hours.
>>>
>>
>> I have just uploaded 0.31.3 to buster-backports
>
> Sorry, I meant 0.31.2-3.

version 0.31.2-3~bpo10+2 is now available in Debian backports, at least
for amd64. Other architectures should follow.  I'd be a bit surprised if
this fixed your issues, but it seems worth testing.

d

Re: consistent database corruption with notmuch new

[Gregor Zattler]

Hi notmuch developers,
* Gregor Zattler <telegraph@gmx.net> [13. Dez. 2020]:
> * Gregor Zattler <telegraph@gmx.net> [13. Dez. 2020]:
>> I now do a notmuch new with libxapian30 version 1.4.17-1
>> and will report back in a few hours.
>
> The result is only slightly different from version 1.4.11:

actually now I realized, that notmuch was not linked against
libxapian v1.4.17-1.

Now I build notmuch from master and linked it with
libxapian 1.4.17-1.  The result is almost the same, though:
0 (master *) grfz@no:~/Mail/.notmuch$ nice ionice -c3 notmuch new --full-scan ; nice ionice -c3 notmuch new --full-scan ; nice ionice -c3 notmuch new --full-scan
Welcome to a new version of notmuch! Your database will now be upgraded.
This process is safe to interrupt.
Backing up tags to /home/grfz/Mail/.notmuch/dump-20201214T124836.gz...
Your notmuch database has now been upgraded.
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607940473.9509_1.no:2,S
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607943993.24776_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607947606.8134_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/drafts.mbox
Note: Ignoring non-mail file: /home/grfz/Mail/postponed.mbox
Processed 1176599 total files in 3h 31m 23s (92 files/sec.).
Added 1102787 new messages to the database.
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607940473.9509_1.no:2,S
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607943993.24776_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607947606.8134_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/drafts.mbox
Note: Ignoring non-mail file: /home/grfz/Mail/postponed.mbox
Processed 125008 total files in 44m 59s (46 files/sec.).
Added 83898 new messages to the database.
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607940473.9509_1.no:2,S
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607943993.24776_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607947606.8134_1.no:2,
add_file: A Xapian exception occurred).
A Xapian exception occurred at lib/message.cc:1182: Too few chunks of compressed data
Processed 122480 total files in 42m 1s (48 files/sec.).
Added 83907 new messages to the database.
Note: A fatal error was encountered: A Xapian exception occurred
1 (master *) grfz@no:~/Mail/.notmuch$

notmuch new still corrupts the database, the second notmuch new
invocation finds emails the first did not find.


Ciao, Gregor
--
 -... --- .-. . -.. ..--.. ...-.-

out of memory on idle machine (was: Re: consistent database corruption with notmuch new)

[Gregor Zattler]

Hi notmuch developers,,
* Gregor Zattler <telegraph@gmx.net> [14. Dez. 2020]:
> notmuch new still corrupts the database, the second notmuch new
> invocation finds emails the first did not find.

I'm still searching for the reason notmuch chokes on my mails.

I assembled a HP MicroServer, installed basic debian buster and
notmuch from the debian buster repo, rsynced my mail to a
separate file system symlinked to the same location as on my
laptop.

There are now
grfz@mic:~/Mail$ find -type f | wc -l
1209419
files on this file system.  no other process touches this
file system, actually the machine is otherwise ilde.

I did notmuch new several times in a row:

grfz@mic:~/Mail/.notmuch$ rm -rf xapian
grfz@mic:~/Mail/.notmuch$ notmuch new
Welcome to a new version of notmuch! Your database will now be upgraded.
This process is safe to interrupt.
Backing up tags to /home/grfz/Mail/.notmuch/dump-20210127T114210.gz...
Your notmuch database has now been upgraded.
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607947606.8134_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607940473.9509_1.no:2,S
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607969276.21046_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607987211.1395_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607979988.4942_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607972847.4857_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607943993.24776_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607976389.23296_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607983586.19063_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/drafts.mbox
Note: Ignoring non-mail file: /home/grfz/Mail/postponed.mbox
Processed 1183682 total files in 16h 43m 27s (19 files/sec.).
Added 1091038 new messages to the database.
grfz@mic:~/Mail/.notmuch$ notmuch new
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607947606.8134_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607940473.9509_1.no:2,S
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607969276.21046_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607987211.1395_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607979988.4942_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607972847.4857_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607943993.24776_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607976389.23296_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607983586.19063_1.no:2,
Processed 1169095 total files in 16h 52m 48s (19 files/sec.).
Added 1077686 new messages to the database.
grfz@mic:~/Mail/.notmuch$ notmuch new
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607947606.8134_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607940473.9509_1.no:2,S
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607969276.21046_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607987211.1395_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607979988.4942_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607972847.4857_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607943993.24776_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607976389.23296_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607983586.19063_1.no:2,
Processed 1151900 total files in 16h 59m 36s (18 files/sec.).
Added 1050106 new messages to the database.
grfz@mic:~/Mail/.notmuch$ notmuch new
add_file: Out of memory files/sec.).
Processed 205 total files in 6s (29 files/sec.).
Added 193 new messages to the database.
Note: A fatal error was encountered: Out of memory
grfz@mic:~/Mail/.notmuch$


notmuch new processes not all files, later invocations
re-add most of the files and the fourth notmuch new
invocation runs out of memory after a few seconds.

This machine has

grfz@mic:~/Mail$ free
total        used        free      shared  buff/cache   available
Mem:       16394760      222272     2293404        5464    13879084    15831448
Swap:      15622140       15104    15607036

enough memory.  It's ECC Memory.


Any ideas on what is the cause of this?  Even if we ignore the
out-of-memory condition: Why does notmuch re-add a million
already added files?


Ciao; Gregor














































































+ my-fetchmail.services stop
+ /home/grfz/bin/notmuch-new.screen-locked--low-load
+ sed -i -e 's/^#exit 0$/exit 0/' /home/grfz/bin/notmuch-new.screen-locked--low-load
+ sleep 77
+ my-fetchmail.services stop
/home/grfz/bin/my-notmuch-renew: line 4: my-fetchmail.services: command not found
+ /home/grfz/bin/notmuch-new.screen-locked--low-load
+ sed -i -e 's/^#exit 0$/exit 0/' /home/grfz/bin/notmuch-new.screen-locked--low-load
+ sleep 77
++ date +%F.%s
+ TAGSpreREINDEX=/home/grfz/Mail/.notmuch/notmuch.dump.2019-05-29.1559166634
+ notmuch dump --gzip --format=batch-tag --output=/home/grfz/Mail/.notmuch/notmuch.dump.2019-05-29.1559166634
++ date +%F.%s
+ mv /home/grfz/Mail/.notmuch/xapian /home/grfz/Mail/.notmuch/xapian.2019-05-29.1559166684
+ notmuch config set query.all '(is:spam OR NOT is:spam OR is:deleted OR NOT is:deleted)'
+ notmuch config set query.izt-all 'query:all AND (is:izt OR (NOT is:izt AND (path:IZT-EDV/** OR path:IZT/**)))'
+ notmuch config set query.izt-edv '(is:izt OR path:IZT-EDV/**) AND (from:science-computing OR to:science-computing OR from:ATOS OR to:ATOS OR to:izt@2cal.de OR from:info@ralf-voegtle.de OR from:info@andresedv.de)'
+ notmuch config set query.izt-robots 'is:izt AND NOT path:IZT/** AND NOT path:sent/** AND NOT query:izt-edv'
+ notmuch config set query.izt '(is:izt OR path:IZT/**) AND NOT path:IZT-EDV/**  AND NOT query:izt-edv)'
+ notmuch config set index.header.List List-Id
+ notmuch config set index.header.Spamgrfz X-Spam-grfz-Status
+ notmuch config set index.header.UserAgent User-Agent
+ notmuch config set index.header.XLabel X-Label
+ notmuch new
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1519322404.860_1.len:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1525795204.16507_1.len:2,S
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1514242825.4116_1.len:2,S
Note: Ignoring non-mail file: /home/grfz/Mail/~ml/linux-crypto@nl.linux.org/cur/1054419557.21641_405.pit:2,
Note: Ignoring non-mail file: /home/grfz/Mail/~ml/linux-crypto@nl.linux.org/cur/1054419557.21641_525.pit:2,
Note: Ignoring non-mail file: /home/grfz/Mail/~ml/linux-crypto@nl.linux.org/cur/1054419557.21641_685.pit:2,
Note: Ignoring non-mail file: /home/grfz/Mail/rmlint.sh~
Note: Ignoring non-mail file: /home/grfz/Mail/postponed.mbox
Note: Ignoring non-mail file: /home/grfz/Mail/drafts.mbox
Note: Ignoring non-mail file: /home/grfz/Mail/rmlint.sh
Welcome to a new version of notmuch! Your database will now be upgraded.
This process is safe to interrupt.
Backing up tags to /home/grfz/Mail/.notmuch/dump-20190529T215124.gz...
Your notmuch database has now been upgraded.
Processed 1036956 total files in 2h 21m 25s (122 files/sec.).
Added 945879 new messages to the database.
+ notmuch tag --batch -new -inbox -unread '*'
Can't specify both cmdline and stdin!
+ notmuch restore --accumulate --format=batch-tag --input=/home/grfz/Mail/.notmuch/notmuch.dump.2019-05-29.1559166634
Warning: cannot apply tags to missing message: notmuch-sha1-e584281f598522d172e73617da020e2f914ccb7e
Warning: cannot apply tags to missing message: notmuch-sha1-5b56ca3568715b006eddd35a597ca7654be045f9
Warning: cannot apply tags to missing message: notmuch-sha1-00ab3ac6e631287f0f6dffaf0037bb7ef86f9e47
Warning: cannot apply tags to missing message: notmuch-sha1-72bde4c427b04c1cbac4a2c2b3a94402616917d7
Warning: cannot apply tags to missing message: notmuch-sha1-dbd41b3c6736943ef17eff9c6862dfa3703af020
Warning: cannot apply tags to missing message: notmuch-sha1-f17b642c650ba54b67b9d4ff5a338143a73f8b7b
Warning: cannot apply tags to missing message: notmuch-sha1-e1cf1219c1fb10a55ba30ca590b93eaf33fdae27
Warning: cannot apply tags to missing message: notmuch-sha1-0d5aaa420dc9e9505cced7c5d8f4c15e0c88c5ef
Warning: cannot apply tags to missing message: notmuch-sha1-a31480a6ef2520153b778dc9531f2c982a79ba32
Warning: cannot apply tags to missing message: notmuch-sha1-606e94dbe6e46b9df83d77ffa788429523e39646
Warning: cannot apply tags to missing message: notmuch-sha1-bb586b264c94f23d9e826071d8841d542ca95c75
Warning: cannot apply tags to missing message: notmuch-sha1-06d4d1492c70aa544b1deeaf7895ca85fe634670
Warning: cannot apply tags to missing message: notmuch-sha1-253991559a5e3c2d5cf1234b761e52ea09aa693f
++ date +%F.%s
+ notmuch dump --gzip --output=/home/grfz/Mail/.notmuch/notmuch.dump.2019-05-30.1559177807
++ date +%F.%s
+ notmuch compact --backup=/home/grfz/Mail/.notmuch/xapian.2019-05-30.1559177848
Compacting database...
compacting table postlist
     Reduced by 36% 908856K (2488312K -> 1579456K)
compacting table docdata
     Reduced by 49% 312K (632K -> 320K)
compacting table termlist
     Reduced by 47% 1701792K (3551384K -> 1849592K)
compacting table position
     Reduced by 45% 3375368K (7372720K -> 3997352K)
compacting table spelling
     doesn't exist
compacting table synonym
     doesn't exist
The old database has been moved to /home/grfz/Mail/.notmuch/xapian.2019-05-30.1559177848.
Done.
+ sed -i -e 's/^exit 0$/#exit 0/' /home/grfz/bin/notmuch-new.screen-locked--low-load
+ my-fetchmail.services stop
+ /home/grfz/bin/notmuch-new.screen-locked--low-load
+ sed -i -e 's/^#exit 0$/exit 0/' /home/grfz/bin/notmuch-new.screen-locked--low-load
+ sleep 77
++ date +%F.%s
+ TAGSpreREINDEX=/home/grfz/Mail/.notmuch/notmuch.dump.2019-05-30.1559214465
+ notmuch dump --gzip --format=batch-tag --output=/home/grfz/Mail/.notmuch/notmuch.dump.2019-05-30.1559214465
++ date +%F.%s
+ mv /home/grfz/Mail/.notmuch/xapian /home/grfz/Mail/.notmuch/xapian.2019-05-30.1559214505
+ notmuch config set query.all '(is:spam OR NOT is:spam OR is:deleted OR NOT is:deleted)'
+ notmuch config set query.izt-all 'query:all AND (is:izt OR (NOT is:izt AND (path:IZT-EDV/** OR path:IZT/**)))'
+ notmuch config set query.izt-edv '(is:izt OR path:IZT-EDV/**) AND (from:science-computing OR to:science-computing OR from:ATOS OR to:ATOS OR to:izt@2cal.de OR from:info@ralf-voegtle.de OR from:info@andresedv.de)'
+ notmuch config set query.izt-robots 'is:izt AND NOT path:IZT/** AND NOT path:sent/** AND NOT query:izt-edv'
+ notmuch config set query.izt '(is:izt OR path:IZT/**) AND NOT path:IZT-EDV/**  AND NOT query:izt-edv)'
+ notmuch config set index.header.List List-Id
+ notmuch config set index.header.Spamgrfz X-Spam-grfz-Status
+ notmuch config set index.header.UserAgent User-Agent
+ notmuch config set index.header.XLabel X-Label
+ notmuch config set index.header.DeliveredTo Delivered-To
+ notmuch config set index.header.DeliveryDate Delivery-date
+ notmuch new
Stopping...
Stopping...
Stopping...
Stopping...
Stopping...
Stopping...
Stopping...
Stopping...
Stopping...
Stopping...
Stopping...
Stopping...
Stopping...
Stopping...
Stopping...
Stopping...
Stopping...
Stopping...
Stopping...
Stopping...
Stopping...
Stopping...
Stopping...
Stopping...
Stopping...
Stopping...
Stopping...
Stopping...
Stopping...
Stopping...
Stopping...
Stopping...
Stopping...
Stopping...
Stopping...
Stopping...
Stopping...
Stopping...
Stopping...
Stopping...
Stopping...
Stopping...
Stopping...
Stopping...
Stopping...
Stopping...
Stopping...
+ my-fetchmail.services stop
+ rm -rf /home/grfz/Mail/.notmuch/xapian
+ notmuch config set query.all '(is:spam OR NOT is:spam OR is:deleted OR NOT is:deleted)'
+ notmuch config set query.izt-all 'query:all AND (is:izt OR (NOT is:izt AND (path:IZT-EDV/** OR path:IZT/**)))'
+ notmuch config set query.izt-edv '(is:izt OR path:IZT-EDV/**) AND (from:science-computing OR to:science-computing OR from:ATOS OR to:ATOS OR to:izt@2cal.de OR from:info@ralf-voegtle.de OR from:info@andresedv.de)'
+ notmuch config set query.izt-robots 'is:izt AND NOT path:IZT/** AND NOT path:sent/** AND NOT query:izt-edv'
+ notmuch config set query.izt '(is:izt OR path:IZT/**) AND NOT path:IZT-EDV/**  AND NOT query:izt-edv)'
+ notmuch config set index.header.List List-Id
+ notmuch config set index.header.Spamgrfz X-Spam-grfz-Status
+ notmuch config set index.header.UserAgent User-Agent
+ notmuch config set index.header.XLabel X-Label
+ notmuch config set index.header.DeliveredTo Delivered-To
+ notmuch config set index.header.DeliveryDate Delivery-date
+ notmuch config set index.header.To To
+ notmuch config set index.header.Cc Cc
+ notmuch config set index.header.Sender Sender
+ notmuch config set index.header.InReplyTo In-Reply-To
+ notmuch config set index.header.ReplyTo Reply-To
+ notmuch new
Note: Ignoring non-mail file: /home/grfz/Mail/rmlint.sh~
Note: Ignoring non-mail file: /home/grfz/Mail/postponed.mbox
Note: Ignoring non-mail file: /home/grfz/Mail/drafts.mbox
Note: Ignoring non-mail file: /home/grfz/Mail/rmlint.sh
Welcome to a new version of notmuch! Your database will now be upgraded.
This process is safe to interrupt.
Backing up tags to /home/grfz/Mail/.notmuch/dump-20190530T135332.gz...
Your notmuch database has now been upgraded.
Processed 1037014 total files in 2h 42m 32s (106 files/sec.).
Added 945937 new messages to the database.
+ notmuch tag -new -inbox -unread '*'
+ notmuch restore --accumulate --format=batch-tag --input=/home/grfz/Mail/.notmuch/notmuch.dump.2019-05-30.1559214465
++ date +%F.%s
+ notmuch dump --gzip --output=/home/grfz/Mail/.notmuch/notmuch.dump.2019-05-30.1559241909
++ date +%F.%s
+ notmuch compact --backup=/home/grfz/Mail/.notmuch/xapian.2019-05-30.1559241950
Compacting database...
compacting table postlist
     Reduced by 36% 1001496K (2707600K -> 1706104K)
compacting table docdata
     Reduced by 51% 336K (656K -> 320K)
compacting table termlist
     Reduced by 47% 1837496K (3836392K -> 1998896K)
compacting table position
     Reduced by 45% 3875240K (8440616K -> 4565376K)
compacting table spelling
     doesn't exist
compacting table synonym
     doesn't exist
The old database has been moved to /home/grfz/Mail/.notmuch/xapian.2019-05-30.1559241950.
Done.
+ sed -i -e 's/^exit 0$/#exit 0/' /home/grfz/bin/notmuch-new.screen-locked--low-load
++ date
+ echo Start @ Sa 1. Jun 16:50:23 CEST 2019
Start @ Sa 1. Jun 16:50:23 CEST 2019
+ /home/grfz/bin/my-fetchmail.services stop
+ /home/grfz/bin/notmuch-new.screen-locked--low-load
+ sed -i -e 's/^#exit 0$/exit 0/' /home/grfz/bin/notmuch-new.screen-locked--low-load
+ sleep 77
++ date +%F.%s
+ TAGSpreREINDEX=/home/grfz/Mail/.notmuch/notmuch.dump.2019-06-01.1559400706
+ notmuch dump --gzip --format=batch-tag --output=/home/grfz/Mail/.notmuch/notmuch.dump.2019-06-01.1559400706
++ date +%F.%s
+ mv /home/grfz/Mail/.notmuch/xapian /home/grfz/Mail/.notmuch/xapian.2019-06-01.1559400744
+ rm -rf /home/grfz/Mail/.notmuch/xapian
+ notmuch config set query.all '(is:spam OR NOT is:spam OR is:deleted OR NOT is:deleted)'
+ notmuch config set query.izt-all 'query:all AND (is:izt OR (NOT is:izt AND (path:IZT-EDV/** OR path:IZT/**)))'
+ notmuch config set query.izt-edv '(is:izt OR path:IZT-EDV/**) AND (from:science-computing OR to:science-computing OR from:ATOS OR to:ATOS OR to:izt@2cal.de OR from:info@ralf-voegtle.de OR from:info@andresedv.de)'
+ notmuch config set query.izt-robots 'is:izt AND NOT path:IZT/** AND NOT path:sent/** AND NOT query:izt-edv'
+ notmuch config set query.izt '(is:izt OR path:IZT/**) AND NOT path:IZT-EDV/**  AND NOT query:izt-edv)'
+ notmuch config set index.header.List List-Id
+ notmuch config set index.header.Spamgrfz X-Spam-grfz-Status
+ notmuch config set index.header.UserAgent User-Agent
+ notmuch config set index.header.XLabel X-Label
+ notmuch config set index.header.DeliveredTo Delivered-To
+ notmuch config set index.header.DeliveryDate Delivery-date
+ notmuch config set index.header.To To
+ notmuch config set index.header.Cc Cc
+ notmuch config set index.header.Sender Sender
+ notmuch config set index.header.InReplyTo In-Reply-To
+ notmuch config set index.header.ReplyTo Reply-To
+ notmuch config set index.header.Received Received
+ notmuch new
Note: Ignoring non-mail file: /home/grfz/Mail/rmlint.sh~
Note: Ignoring non-mail file: /home/grfz/Mail/postponed.mbox
Note: Ignoring non-mail file: /home/grfz/Mail/drafts.mbox
Note: Ignoring non-mail file: /home/grfz/Mail/rmlint.sh
Welcome to a new version of notmuch! Your database will now be upgraded.
This process is safe to interrupt.
Backing up tags to /home/grfz/Mail/.notmuch/dump-20190601T145224.gz...
Your notmuch database has now been upgraded.
Processed 1037381 total files in 2h 54m 22s (99 files/sec.).
Added 946288 new messages to the database.
+ notmuch tag -inbox -unread '*'
+ notmuch restore --accumulate --format=batch-tag --input=/home/grfz/Mail/.notmuch/notmuch.dump.2019-06-01.1559400706
++ date +%F.%s
+ notmuch dump --gzip --output=/home/grfz/Mail/.notmuch/notmuch.dump.2019-06-01.1559419342
++ date +%F.%s
+ notmuch compact --backup=/home/grfz/Mail/.notmuch/xapian.2019-06-01.1559419387
Compacting database...
compacting table postlist
     Reduced by 37% 1077536K (2904400K -> 1826864K)
compacting table docdata
     Reduced by 48% 304K (624K -> 320K)
compacting table termlist
     Reduced by 48% 2004432K (4166496K -> 2162064K)
compacting table position
     Reduced by 46% 4498968K (9748816K -> 5249848K)
compacting table spelling
     doesn't exist
compacting table synonym
     doesn't exist
The old database has been moved to /home/grfz/Mail/.notmuch/xapian.2019-06-01.1559419387.
Done.
+ sed -i -e 's/^exit 0$/#exit 0/' /home/grfz/bin/notmuch-new.screen-locked--low-load
+ sleep 77
+ my-fetchmail.services start
/home/grfz/bin/my-notmuch-renew: line 48: my-fetchmail.services: command not found
++ date
+ echo End @ Sa 1. Jun 22:10:51 CEST 2019
End @ Sa 1. Jun 22:10:51 CEST 2019
++ date
+ echo Start @ So 2. Jun 00:45:52 CEST 2019
Start @ So 2. Jun 00:45:52 CEST 2019
+ /home/grfz/bin/my-fetchmail.services stop
+ /home/grfz/bin/notmuch-new.screen-locked--low-load
+ sed -i -e 's/^#exit 0$/exit 0/' /home/grfz/bin/notmuch-new.screen-locked--low-load
+ sleep 77
++ date +%F.%s
+ TAGSpreREINDEX=/home/grfz/Mail/.notmuch/notmuch.dump.2019-06-02.1559429235
+ notmuch dump --gzip --format=batch-tag --output=/home/grfz/Mail/.notmuch/notmuch.dump.2019-06-02.1559429235
++ date
+ echo Start @ Di 4. Jun 20:54:03 CEST 2019
Start @ Di 4. Jun 20:54:03 CEST 2019
+ /home/grfz/bin/my-fetchmail.services stop
+ /home/grfz/bin/notmuch-new.screen-locked--low-load
+ sed -i -e 's/^#exit 0$/exit 0/' /home/grfz/bin/notmuch-new.screen-locked--low-load
+ sleep 77
++ date +%F.%s
+ TAGSpreREINDEX=/home/grfz/Mail/.notmuch/notmuch.dump.2019-06-04.1559674534
+ notmuch dump --gzip --format=batch-tag --output=/home/grfz/Mail/.notmuch/notmuch.dump.2019-06-04.1559674534
++ date +%F.%s
+ mv /home/grfz/Mail/.notmuch/xapian /home/grfz/Mail/.notmuch/xapian.2019-06-04.1559674574
+ rm -rf /home/grfz/Mail/.notmuch/xapian
+ notmuch config set query.all '(is:spam OR NOT is:spam OR is:deleted OR NOT is:deleted)'
+ notmuch config set query.izt-all 'query:all AND (is:izt OR (NOT is:izt AND (path:IZT-EDV/** OR path:IZT/**)))'
+ notmuch config set query.izt-edv '(is:izt OR path:IZT-EDV/**) AND (from:science-computing OR to:science-computing OR from:ATOS OR to:ATOS OR to:izt@2cal.de OR from:info@ralf-voegtle.de OR from:info@andresedv.de)'
+ notmuch config set query.izt-robots 'is:izt AND NOT path:IZT/** AND NOT path:sent/** AND NOT query:izt-edv'
+ notmuch config set query.izt '(is:izt OR path:IZT/**) AND NOT path:IZT-EDV/**  AND NOT query:izt-edv)'
+ notmuch config set index.header.List List-Id
+ notmuch config set index.header.Spamgrfz X-Spam-grfz-Status
+ notmuch config set index.header.UserAgent User-Agent
+ notmuch config set index.header.XLabel X-Label
+ notmuch config set index.header.DeliveredTo Delivered-To
+ notmuch config set index.header.DeliveryDate Delivery-date
+ notmuch config set index.header.To To
+ notmuch config set index.header.Cc Cc
+ notmuch config set index.header.Sender Sender
+ notmuch config set index.header.InReplyTo In-Reply-To
+ notmuch config set index.header.ReplyTo Reply-To
+ notmuch config set index.header.Received Received
+ notmuch new
Note: Ignoring non-mail file: /home/grfz/Mail/postponed.mbox
Note: Ignoring non-mail file: /home/grfz/Mail/drafts.mbox
Welcome to a new version of notmuch! Your database will now be upgraded.
This process is safe to interrupt.
Backing up tags to /home/grfz/Mail/.notmuch/dump-20190604T185615.gz...
Your notmuch database has now been upgraded.
Processed 1034784 total files in 3h 16m 21s (87 files/sec.).
Added 944525 new messages to the database.
+ notmuch tag -inbox -unread '*'
+ notmuch restore --accumulate --format=batch-tag --input=/home/grfz/Mail/.notmuch/notmuch.dump.2019-06-04.1559674534
++ date +%F.%s
+ notmuch dump --gzip --output=/home/grfz/Mail/.notmuch/notmuch.dump.2019-06-05.1559694905
++ date +%F.%s
+ notmuch compact --backup=/home/grfz/Mail/.notmuch/xapian.2019-06-05.1559694949
Compacting database...
compacting table postlist
     Reduced by 37% 1073696K (2900248K -> 1826552K)
compacting table docdata
     Reduced by 50% 328K (648K -> 320K)
compacting table termlist
     Reduced by 48% 2003640K (4164728K -> 2161088K)
compacting table position
     Reduced by 46% 4495744K (9743384K -> 5247640K)
compacting table spelling
     doesn't exist
compacting table synonym
     doesn't exist
The old database has been moved to /home/grfz/Mail/.notmuch/xapian.2019-06-05.1559694949.
Done.
+ sed -i -e 's/^exit 0$/#exit 0/' /home/grfz/bin/notmuch-new.screen-locked--low-load
+ sleep 77
+ my-fetchmail.services start
++ date
+ echo End @ Mi 5. Jun 02:44:58 CEST 2019
End @ Mi 5. Jun 02:44:58 CEST 2019
From: Gregor Zattler <telegraph@gmx.net>
To: notmuch <notmuch@notmuchmail.org>
Cc:
Bcc: Gregor Zattler <grfz@localhost>
Subject: out of memory on "empty" machine (was: Re: consistent database
 corruption with notmuch new)
Reply-To:
In-Reply-To: <20201214192251.GA7858@no.workgroup>

Hi notmuch,
* Gregor Zattler <telegraph@gmx.net> [14. Dez. 2020]:
> Hi notmuch developers,
> * Gregor Zattler <telegraph@gmx.net> [13. Dez. 2020]:
>> * Gregor Zattler <telegraph@gmx.net> [13. Dez. 2020]:
>>> I now do a notmuch new with libxapian30 version 1.4.17-1
>>> and will report back in a few hours.
>>
>> The result is only slightly different from version 1.4.11:
>
> actually now I realized, that notmuch was not linked against
> libxapian v1.4.17-1.
>
> Now I build notmuch from master and linked it with
> libxapian 1.4.17-1.  The result is almost the same, though:
> 0 (master *) grfz@no:~/Mail/.notmuch$ nice ionice -c3 notmuch new --full-scan ; nice ionice -c3 notmuch new --full-scan ; nice ionice -c3 notmuch new --full-scan
> Welcome to a new version of notmuch! Your database will now be upgraded.
> This process is safe to interrupt.
> Backing up tags to /home/grfz/Mail/.notmuch/dump-20201214T124836.gz...
> Your notmuch database has now been upgraded.
> Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607940473.9509_1.no:2,S
> Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607943993.24776_1.no:2,
> Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607947606.8134_1.no:2,
> Note: Ignoring non-mail file: /home/grfz/Mail/drafts.mbox
> Note: Ignoring non-mail file: /home/grfz/Mail/postponed.mbox
> Processed 1176599 total files in 3h 31m 23s (92 files/sec.).
> Added 1102787 new messages to the database.
> Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607940473.9509_1.no:2,S
> Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607943993.24776_1.no:2,
> Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607947606.8134_1.no:2,
> Note: Ignoring non-mail file: /home/grfz/Mail/drafts.mbox
> Note: Ignoring non-mail file: /home/grfz/Mail/postponed.mbox
> Processed 125008 total files in 44m 59s (46 files/sec.).
> Added 83898 new messages to the database.
> Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607940473.9509_1.no:2,S
> Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607943993.24776_1.no:2,
> Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607947606.8134_1.no:2,
> add_file: A Xapian exception occurred).
> A Xapian exception occurred at lib/message.cc:1182: Too few chunks of compressed data
> Processed 122480 total files in 42m 1s (48 files/sec.).
> Added 83907 new messages to the database.
> Note: A fatal error was encountered: A Xapian exception occurred
> 1 (master *) grfz@no:~/Mail/.notmuch$
>
> notmuch new still corrupts the database, the second notmuch new
> invocation finds emails the first did not find.
>
>
> Ciao, Gregor
> --
>  -... --- .-. . -.. ..--.. ...-.-
>
>

Ciao, Gregor
--
 -... --- .-. . -.. ..--.. ...-.-

Re: out of memory on idle machine (was: Re: consistent database corruption with notmuch new)

[David Bremner]

Gregor Zattler <telegraph@gmx.net> writes:

> Hi notmuch developers,,
> * Gregor Zattler <telegraph@gmx.net> [14. Dez. 2020]:
>> notmuch new still corrupts the database, the second notmuch new
>> invocation finds emails the first did not find.
>
> I'm still searching for the reason notmuch chokes on my mails.

>
> I assembled a HP MicroServer, installed basic debian buster and
> notmuch from the debian buster repo, rsynced my mail to a
> separate file system symlinked to the same location as on my
> laptop.
>
> There are now
> grfz@mic:~/Mail$ find -type f | wc -l
> 1209419
> files on this file system.  no other process touches this
> file system, actually the machine is otherwise ilde.
>
> I did notmuch new several times in a row:
>
> grfz@mic:~/Mail/.notmuch$ rm -rf xapian
> grfz@mic:~/Mail/.notmuch$ notmuch new
> Welcome to a new version of notmuch! Your database will now be upgraded.
> This process is safe to interrupt.
> Backing up tags to /home/grfz/Mail/.notmuch/dump-20210127T114210.gz...
> Your notmuch database has now been upgraded.
> Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607947606.8134_1.no:2,
> Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607940473.9509_1.no:2,S
> Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607969276.21046_1.no:2,
> Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607987211.1395_1.no:2,
> Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607979988.4942_1.no:2,
> Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607972847.4857_1.no:2,
> Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607943993.24776_1.no:2,
> Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607976389.23296_1.no:2,
> Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607983586.19063_1.no:2,
> Note: Ignoring non-mail file: /home/grfz/Mail/drafts.mbox
> Note: Ignoring non-mail file: /home/grfz/Mail/postponed.mbox
> Processed 1183682 total files in 16h 43m 27s (19 files/sec.).
> Added 1091038 new messages to the database.
> grfz@mic:~/Mail/.notmuch$ notmuch new
> Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607947606.8134_1.no:2,
> Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607940473.9509_1.no:2,S
> Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607969276.21046_1.no:2,
> Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607987211.1395_1.no:2,
> Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607979988.4942_1.no:2,
> Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607972847.4857_1.no:2,
> Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607943993.24776_1.no:2,
> Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607976389.23296_1.no:2,
> Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607983586.19063_1.no:2,
> Processed 1169095 total files in 16h 52m 48s (19 files/sec.).
> Added 1077686 new messages to the database.

Idea #1
-------

There are several mysteries here, but maybe we should begin at the
beginning. Something is wrong if notmuch scans your entire mail tree the
second time you run notmuch new.

Notmuch checks the mtime of directories against the time stored in the
database. As a sanitity check, maybe you can do that for one of your
directories with many messages. This needs "quest" and "xapian-delve",
from the package xapian-tools.

Unfortunately this should probably be done after the first notmuch
new. I have another idea to try (below) in the state after several news where
you are getting OOM.

I'll use real paths for my system; you'll need to update them.

This gives a time in seconds

% stat --format "%Y" ~/Maildir/tethera/cur
1612008734

Now let us find the database document for that directory

% quest -bdir:XDIRECTORY -d ~/Maildir/.notmuch/xapian/ dir:tethera/cur

Parsed Query: Query(0 * XDIRECTORYtethera/cur)
Exactly 1 matches
MSet:
431067: [0]
tethera/cur

Grabbing the record number from the output of quest:

% xapian-delve -r 431067 -VS0 ~/Maildir/.notmuch/xapian

Value 0 for record #431067: 1.61201e+09
Term List for record #431067: XDDIRENTRY387045:cur XDIRECTORYtethera/cur

You can see the value matches the mtime up to 6 decimal places.

Idea #2
-------

Try to figure out if some specific file is causing the OOM.

Run notmuch-new in gdb

There is a check for NOTMUCH_STATUS_OUT_OF_MEMORY around line 419/420 of
notmuch-new.c. If I understand correctly, that is where things are
failing. The following is untested; you will need the package
notmuch-dbgsym installed [1]

% gdb --args notmuch new
(gdb) b notmuch-new.c:420
(gdb) run
(gdb) p filename



[1]: https://wiki.debian.org/AutomaticDebugPackages

Re: out of memory on idle machine

[Gregor Zattler]

Hi David, notmuch developers,

thanks a lot: your second idea this time showed a database
corruption instead of an OOM.

The first idea showed, that there is data missing in the
database:

* David Bremner <david@tethera.net> [30. Jan. 2021]:
Since idea #1 involves removing the xapian database I first
tried idea #2:
> Idea #2
> -------
>
> Try to figure out if some specific file is causing the OOM.
>
> Run notmuch-new in gdb
>
> There is a check for NOTMUCH_STATUS_OUT_OF_MEMORY around line 419/420 of
> notmuch-new.c. If I understand correctly, that is where things are
> failing. The following is untested; you will need the package
> notmuch-dbgsym installed [1]
>
> % gdb --args notmuch new
> (gdb) b notmuch-new.c:420
> (gdb) run
> (gdb) p filename
> [1]: https://wiki.debian.org/AutomaticDebugPackages

Installed notmuch-dbgsym (0.28.4-1) and gdb.

grfz@mic:/etc$ gdb --args notmuch new
[...]
(gdb) b notmuch-new.c:420
Breakpoint 1 at 0x10601: file notmuch-new.c, line 421.
(gdb) run
Starting program: /usr/bin/notmuch new
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
add_file: A Xapian exception occurred
A Xapian exception occurred finding message: Db block overwritten - are there multiple writers?.
Processed 24 total files in almost no time.
Added 23 new messages to the database.
Note: A fatal error was encountered: A Xapian exception occurred
[Inferior 1 (process 22756) exited with code 01]
(gdb)

This time it's no OOM it's a xapian exeption again.



Now for idea #1:

> Idea #1
> -------
>
> There are several mysteries here, but maybe we should begin at the
> beginning. Something is wrong if notmuch scans your entire mail tree the
> second time you run notmuch new.
>
> Notmuch checks the mtime of directories against the time stored in the
> database. As a sanitity check, maybe you can do that for one of your
> directories with many messages. This needs "quest" and "xapian-delve",
> from the package xapian-tools.
>
> Unfortunately this should probably be done after the first notmuch
> new. I have another idea to try (below) in the state after several news where
> you are getting OOM.
>
> I'll use real paths for my system; you'll need to update them.
>
> This gives a time in seconds
>
> % stat --format "%Y" ~/Maildir/tethera/cur
> 1612008734
>
> Now let us find the database document for that directory
>
> % quest -bdir:XDIRECTORY -d ~/Maildir/.notmuch/xapian/ dir:tethera/cur
>
> Parsed Query: Query(0 * XDIRECTORYtethera/cur)
> Exactly 1 matches
> MSet:
> 431067: [0]
> tethera/cur
>
> Grabbing the record number from the output of quest:
>
> % xapian-delve -r 431067 -VS0 ~/Maildir/.notmuch/xapian
>
> Value 0 for record #431067: 1.61201e+09
> Term List for record #431067: XDDIRENTRY387045:cur XDIRECTORYtethera/cur
>
> You can see the value matches the mtime up to 6 decimal places.

grfz@mic:~/Mail/.notmuch$ mv xapian xapian-corrupted
grfz@mic:~/Mail/.notmuch$ notmuch new
Welcome to a new version of notmuch! Your database will now be upgraded.
This process is safe to interrupt.
Backing up tags to /home/grfz/Mail/.notmuch/dump-20210130T170349.gz...
Your notmuch database has now been upgraded.
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607947606.8134_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607940473.9509_1.no:2,S
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607969276.21046_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607987211.1395_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607979988.4942_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607972847.4857_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607943993.24776_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607976389.23296_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607983586.19063_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/drafts.mbox
Note: Ignoring non-mail file: /home/grfz/Mail/postponed.mbox
Processed 1183682 total files in 13h 38m 31s (24 files/sec.).
Added 1091038 new messages to the database.

I then installed xapian-tools amd64 1.4.11-1.

grfz@mic:~/Mail/.notmuch$ stat --format "%Y"  ~/Mail/inbox/cur
1611646289

grfz@mic:~/Mail/.notmuch$ quest -bdir:XDIRECTORY -d ~/Mail/.notmuch/xapian/ dir:inbox/cur
Parsed Query: Query(0 * XDIRECTORYinbox/cur)
MSet:

That's it, there is data missing in the database.



I have no clue why this is the case.  Just in case here is

grfz@mic:~$ dpkg -l notmuch libnotmuch5 libc6 libglib2.0-0 libgmime-3.0-0 libtalloc2 zlib1g libxapian30
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                 Version          Architecture Description
+++-====================-================-============-======================================================
ii  libc6:amd64          2.28-10          amd64        GNU C Library: Shared libraries
ii  libglib2.0-0:amd64   2.58.3-2+deb10u2 amd64        GLib library of C routines
ii  libgmime-3.0-0:amd64 3.2.1-1          amd64        MIME message parser and creator library
ii  libnotmuch5          0.28.4-1         amd64        thread-based email index, search and tagging (runtime)
ii  libtalloc2:amd64     2.1.14-2         amd64        hierarchical pool based memory allocator
ii  libxapian30:amd64    1.4.11-1         amd64        Search engine library
ii  notmuch              0.28.4-1         amd64        thread-based email index, search and tagging
ii  zlib1g:amd64         1:1.2.11.dfsg-1  amd64        compression library - runtime


and

grfz@mic:~$ sudo tune2fs -l /dev/mapper/bcache0p2_crypt
tune2fs 1.44.5 (15-Dec-2018)
Filesystem volume name:   data
Last mounted on:          /home/grfz/data
Filesystem UUID:          a15890ca-4817-418b-923a-fa6849855a1e
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal ext_attr resize_inode dir_index sparse_super2 filetype needs_recovery extent 64bit mmp flex_bg sparse_super large_file huge_file dir_nlink extra_isize metadata_csum
Filesystem flags:         signed_directory_hash
Default mount options:    user_xattr acl
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              58589184
Block count:              234355200
Reserved block count:     11717760
Free blocks:              230413509
Free inodes:              58589173
First block:              0
Block size:               4096
Fragment size:            4096
Group descriptor size:    64
Reserved GDT blocks:      1024
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         8192
Inode blocks per group:   512
Flex block group size:    16
Filesystem created:       Mon Jan 25 21:44:28 2021
Last mount time:          Mon Jan 25 22:13:54 2021
Last write time:          Mon Jan 25 22:13:54 2021
Mount count:              3
Maximum mount count:      -1
Last checked:             Mon Jan 25 21:44:28 2021
Check interval:           0 (<none>)
Lifetime writes:          1032 MB
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:               256
Required extra isize:     32
Desired extra isize:      32
Journal inode:            8
Default directory hash:   half_md4
Directory Hash Seed:      ea7647de-8706-4db8-b45b-a0ba39fc193b
Journal backup:           inode blocks
Backup block groups:      1 7151
MMP block number:         9367
MMP update interval:      5
Checksum type:            crc32c
Checksum:                 0x62d3b045


As you see, it's pretty new.  If you think the non-standard
features play a role here, I can simply reformat the
pratition and start over.  The machine has no other purpose
atm.

I'll redo idea #2 on my laptop and will report it's results.

Ciao, Gregor
--
 -... --- .-. . -.. ..--.. ...-.-

Re: out of memory on idle machine

[Gregor Zattler]

Hi David, notmuch developers,
* Gregor Zattler <telegraph@gmx.net> [31. Jan. 2021]:
> I'll redo idea #2 on my laptop and will report it's results.

Now on the laptop:

grfz@no:~/Mail/.notmuch$ nice ionice -c 3 notmuch new
Welcome to a new version of notmuch! Your database will now be upgraded.
This process is safe to interrupt.
Backing up tags to /home/grfz/Mail/.notmuch/dump-20210131T080546.gz...
Your notmuch database has now been upgraded.
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607940473.9509_1.no:2,S
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607943993.24776_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607947606.8134_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1611742380.14576_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607969276.21046_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607972847.4857_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607976389.23296_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607979988.4942_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607983586.19063_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607987211.1395_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/drafts.mbox
Processed 1184742 total files in 3h 33m 35s (92 files/sec.).
Added 1115147 new messages to the database.

0 (master *) grfz@no:~/Mail/.notmuch$ stat --format "%Y"  ~/Mail/inbox/cur
1612091679
0 (master *) grfz@no:~/Mail/.notmuch$ stat --format "%y"  ~/Mail/inbox/cur
2021-01-31 12:14:39.771049424 +0100
0 (master *) grfz@no:~/Mail/.notmuch$
0 (master *) grfz@no:~/Mail/.notmuch$ quest -bdir:XDIRECTORY -d ~/Mail/.notmuch/xapian/ dir:inbox/cur
bash: quest: command not found
127 (master *) grfz@no:~/Mail/.notmuch$ quest -bdir:XDIRECTORY -d ~/Mail/.notmuch/xapian/ dir:inbox/cur
Parsed Query: Query(0 * XDIRECTORYinbox/cur)
MSet:
1114128: [0]
inbox/cur

0 (master *) grfz@no:~/Mail/.notmuch$ xapian-delve -r 1114128 -VS0 ~/Mail/.notmuch/xapian
Value 0 for record #1114128: 1.61208e+09
Term List for record #1114128: XDDIRENTRY1114127:cur XDIRECTORYinbox/cur


So I think that's OK on my laptop, after the first notmuch
new.  Now I do another one.

0 (master *) grfz@no:~/Mail/.notmuch$ nice ionice -c 3 notmuch new
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607940473.9509_1.no:2,S
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607943993.24776_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607947606.8134_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607969276.21046_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607972847.4857_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607976389.23296_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607979988.4942_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607983586.19063_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607987211.1395_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1611742380.14576_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/drafts.mbox
Processed 121712 total files in 45m 4s (45 files/sec.).
Added 85345 new messages to the database. Removed 3 messages.
0 (master *) grfz@no:~/Mail/.notmuch$

The Problem remains, but at a different scale: In between
these two notmuch new runs there were only a few hours,
there's no way I received 121712 or 85345 emails in this time
frame.

0 (master *) grfz@no:~/Mail/.notmuch$ nice ionice -c 3 notmuch new
Note: Ignoring non-mail file: /home/grfz/Mail/spam/new/1612101986.719_1.no
Note: Ignoring non-mail file: /home/grfz/Mail/spam/new/1612105584.10821_1.no
Note: Ignoring non-mail file: /home/grfz/Mail/spam/new/1612109187.21153_1.no
Note: Ignoring non-mail file: /home/grfz/Mail/spam/new/1612112775.31326_1.no
Note: Ignoring non-mail file: /home/grfz/Mail/spam/new/1612116375.9385_1.no
Note: Ignoring non-mail file: /home/grfz/Mail/spam/new/1612120297.21133_1.no
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607940473.9509_1.no:2,S
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607943993.24776_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607947606.8134_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607969276.21046_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607972847.4857_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607976389.23296_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607979988.4942_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607983586.19063_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607987211.1395_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1611742380.14576_1.no:2,
terminate called after throwing an instance of 'Xapian::DatabaseCorruptError'
Aborted
134 (master *) grfz@no:~/Mail/.notmuch$

So here is the database corruption again, while doing the
third notmuch new.


I only now realize I did not do the tests after the second
notmuch new, but I copied the database so here is it:

0 (master *) grfz@no:~/Mail/.notmuch$ xapian-delve -r 1114128 -VS0 ~/Mail/.notmuch/xapian-2
Value 0 for record #1114128: 1.61209e+09
Term List for record #1114128: XDDIRENTRY1114127:cur XDIRECTORYinbox/cur
0 (master *) grfz@no:~/Mail/.notmuch$ stat --format "%Y"  ~/Mail/inbox/cur
1612091679
0 (master *) grfz@no:~/Mail/.notmuch$ quest -bdir:XDIRECTORY -d ~/Mail/.notmuch/xapian-2/ dir:inbox/cur
Parsed Query: Query(0 * XDIRECTORYinbox/cur)
MSet:
1114128: [0]
inbox/cur
0 (master *) grfz@no:~/Mail/.notmuch$ xapian-delve -r 1114128 -VS0 ~/Mail/.notmuch/xapian-2
Value 0 for record #1114128: 1.61209e+09
Term List for record #1114128: XDDIRENTRY1114127:cur XDIRECTORYinbox/cur
0 (master *) grfz@no:~/Mail/.notmuch$


I don't have a clue on how to interpret this.  Any ideas?

Thanks for your attention, Gregor


P.S.: These are the properties of the laptops root filesystem:

0 (master *) grfz@no:~$ sudo tune2fs -l /dev/mapper/vg-lv--root
[sudo] password for grfz:
tune2fs 1.44.5 (15-Dec-2018)
Filesystem volume name:   root
Last mounted on:          /
Filesystem UUID:          801d2942-610d-4f43-9824-b275642e75de
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal ext_attr resize_inode dir_index filetype needs_recovery extent 64bit flex_bg sparse_super large_file huge_file dir_nlink extra_isize metadata_csum
Filesystem flags:         signed_directory_hash
Default mount options:    user_xattr acl
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              14082048
Block count:              56308736
Reserved block count:     2815436
Free blocks:              23262312
Free inodes:              11957831
First block:              0
Block size:               4096
Fragment size:            4096
Group descriptor size:    64
Reserved GDT blocks:      1024
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         8192
Inode blocks per group:   512
RAID stripe width:        8191
Flex block group size:    16
Filesystem created:       Sat Nov 23 11:59:12 2019
Last mount time:          Sun Jan 31 08:43:15 2021
Last write time:          Sun Jan 31 08:43:00 2021
Mount count:              1
Maximum mount count:      23
Last checked:             Sun Jan 31 08:43:00 2021
Check interval:           0 (<none>)
Lifetime writes:          7448 GB
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:               256
Required extra isize:     32
Desired extra isize:      32
Journal inode:            8
First orphan inode:       13902671
Default directory hash:   half_md4
Directory Hash Seed:      22422f4f-fec4-4a9b-8ee6-5be99ecb21f4
Journal backup:           inode blocks
Checksum type:            crc32c
Checksum:                 0x69629743

Re: out of memory on idle machine

[David Bremner]

Gregor Zattler <telegraph@gmx.net> writes:
>
> 0 (master *) grfz@no:~/Mail/.notmuch$ stat --format "%Y"  ~/Mail/inbox/cur
> 1612091679
> 0 (master *) grfz@no:~/Mail/.notmuch$ stat --format "%y"  ~/Mail/inbox/cur
> 2021-01-31 12:14:39.771049424 +0100
> 0 (master *) grfz@no:~/Mail/.notmuch$
> 0 (master *) grfz@no:~/Mail/.notmuch$ quest -bdir:XDIRECTORY -d ~/Mail/.notmuch/xapian/ dir:inbox/cur
> bash: quest: command not found
> 127 (master *) grfz@no:~/Mail/.notmuch$ quest -bdir:XDIRECTORY -d ~/Mail/.notmuch/xapian/ dir:inbox/cur
> Parsed Query: Query(0 * XDIRECTORYinbox/cur)
> MSet:
> 1114128: [0]
> inbox/cur
>
> 0 (master *) grfz@no:~/Mail/.notmuch$ xapian-delve -r 1114128 -VS0 ~/Mail/.notmuch/xapian
> Value 0 for record #1114128: 1.61208e+09
> Term List for record #1114128: XDDIRENTRY1114127:cur XDIRECTORYinbox/cur
>
>
> So I think that's OK on my laptop, after the first notmuch
> new.  Now I do another one.

There is the rounding error in the last digit (8 vs. 9), but I'm not
sure if that matters, or if it's just in the output formatting.

> Processed 121712 total files in 45m 4s (45 files/sec.).
> Added 85345 new messages to the database. Removed 3 messages.
> 0 (master *) grfz@no:~/Mail/.notmuch$
>
> The Problem remains, but at a different scale: In between
> these two notmuch new runs there were only a few hours,
> there's no way I received 121712 or 85345 emails in this time
> frame.

If you use "notmuch new --debug" for your second run that may give you
more information about what files are being (re)-added. Be warned that
will probably generate copious output, so better to redirect.

> 0 (master *) grfz@no:~/Mail/.notmuch$ xapian-delve -r 1114128 -VS0 ~/Mail/.notmuch/xapian-2
> Value 0 for record #1114128: 1.61209e+09
> Term List for record #1114128: XDDIRENTRY1114127:cur XDIRECTORYinbox/cur
> 0 (master *) grfz@no:~/Mail/.notmuch$ stat --format "%Y"  ~/Mail/inbox/cur
> 1612091679
> 0 (master *) grfz@no:~/Mail/.notmuch$ quest -bdir:XDIRECTORY -d ~/Mail/.notmuch/xapian-2/ dir:inbox/cur
> Parsed Query: Query(0 * XDIRECTORYinbox/cur)
> MSet:
> 1114128: [0]
> inbox/cur
> 0 (master *) grfz@no:~/Mail/.notmuch$ xapian-delve -r 1114128 -VS0 ~/Mail/.notmuch/xapian-2
> Value 0 for record #1114128: 1.61209e+09
> Term List for record #1114128: XDDIRENTRY1114127:cur XDIRECTORYinbox/cur
> 0 (master *) grfz@no:~/Mail/.notmuch$

Here I don't see the rounding error, which is (potentially)
interesting. Perhaps after a run with --debug we will have some idea of
what directory is being rescanned.

d

Re: out of memory on idle machine

[David Bremner]

Gregor Zattler <telegraph@gmx.net> writes:
>
> Installed notmuch-dbgsym (0.28.4-1) and gdb.
>
> grfz@mic:/etc$ gdb --args notmuch new
> [...]
> (gdb) b notmuch-new.c:420
> Breakpoint 1 at 0x10601: file notmuch-new.c, line 421.
> (gdb) run
> Starting program: /usr/bin/notmuch new
> [Thread debugging using libthread_db enabled]
> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
> add_file: A Xapian exception occurred
> A Xapian exception occurred finding message: Db block overwritten - are there multiple writers?.
> Processed 24 total files in almost no time.
> Added 23 new messages to the database.
> Note: A fatal error was encountered: A Xapian exception occurred
> [Inferior 1 (process 22756) exited with code 01]
> (gdb)
>
> This time it's no OOM it's a xapian exeption again.
>
>

I have included the Xapian list in copy in case that message rings a
bell. I guess you know there are not multiple writers in your setup.
Olly Betts mentioned in a different thread that he will build a version
of xapian 1.4.18 for buster backports, so trying with that is probably a
good step when it is available.


> grfz@mic:~/Mail/.notmuch$ mv xapian xapian-corrupted
> grfz@mic:~/Mail/.notmuch$ notmuch new
> Welcome to a new version of notmuch! Your database will now be upgraded.
> This process is safe to interrupt.
> Backing up tags to /home/grfz/Mail/.notmuch/dump-20210130T170349.gz...
> Your notmuch database has now been upgraded.
> Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607947606.8134_1.no:2,
> Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607940473.9509_1.no:2,S
> Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607969276.21046_1.no:2,
> Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607987211.1395_1.no:2,
> Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607979988.4942_1.no:2,
> Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607972847.4857_1.no:2,
> Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607943993.24776_1.no:2,
> Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607976389.23296_1.no:2,
> Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607983586.19063_1.no:2,
> Note: Ignoring non-mail file: /home/grfz/Mail/drafts.mbox
> Note: Ignoring non-mail file: /home/grfz/Mail/postponed.mbox
> Processed 1183682 total files in 13h 38m 31s (24 files/sec.).
> Added 1091038 new messages to the database.
>
> I then installed xapian-tools amd64 1.4.11-1.
>
> grfz@mic:~/Mail/.notmuch$ stat --format "%Y"  ~/Mail/inbox/cur
> 1611646289
>
> grfz@mic:~/Mail/.notmuch$ quest -bdir:XDIRECTORY -d ~/Mail/.notmuch/xapian/ dir:inbox/cur
> Parsed Query: Query(0 * XDIRECTORYinbox/cur)
> MSet:
>
> That's it, there is data missing in the database.
>

This could either be a logic error in Notmuch, 

You can get a complete list of all of the directory documents in the
notmuch database with

% xapian-delve -1 -A XDIRECTORY ~/Mail/.notmuch/xapian | sort -u > delve.txt

You can get a list of the actual directories with

% find ~/Mail -type d -not empty | sed s,/home/grfz/Mail/,XDIRECTORY, |sort -u  >find.txt

Comparing those two lists may give you some hints. Any directory that
shows up in the second list but not the first should have no files in it
(but potentially other directories) or be ignored either implicitly
(.notmuch, .notmuch/xapian) or explicitely by your configuration.

Re: out of memory on idle machine

[Gregor Zattler]

Hi David, notmuch and xapian developers,
* David Bremner <david@tethera.net> [03. Feb. 2021]:
> Gregor Zattler <telegraph@gmx.net> writes:
>>
>> Installed notmuch-dbgsym (0.28.4-1) and gdb.
>>
>> grfz@mic:/etc$ gdb --args notmuch new
>> [...]
>> (gdb) b notmuch-new.c:420
>> Breakpoint 1 at 0x10601: file notmuch-new.c, line 421.
>> (gdb) run
>> Starting program: /usr/bin/notmuch new
>> [Thread debugging using libthread_db enabled]
>> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
>> add_file: A Xapian exception occurred
>> A Xapian exception occurred finding message: Db block overwritten - are there multiple writers?.
>> Processed 24 total files in almost no time.
>> Added 23 new messages to the database.
>> Note: A fatal error was encountered: A Xapian exception occurred
>> [Inferior 1 (process 22756) exited with code 01]
>> (gdb)
>>
>> This time it's no OOM it's a xapian exeption again.
>>
>>
>
> I have included the Xapian list in copy in case that message rings a
> bell. I guess you know there are not multiple writers in your setup.

Absolutely.  This is an otherwise unused minimal debian
buster system, the emails are a static copy from from my
production system (laptop) some days ago on a mounted
filesystem.  There are no other writes to this file system
besides notmuch new.

When I do subsequent notmuch new, most of the files are
reindexed in the second and third run, in the fourth run
there is a OOM although the system has 16GB RAM, 16GB swap.
The fourth notmuch new invocation throws an exeption:

grfz@mic:~/Mail/.notmuch$ nice ionice -c 3 notmuch new
Welcome to a new version of notmuch! Your database will now be upgraded.
This process is safe to interrupt.
Backing up tags to /home/grfz/Mail/.notmuch/dump-20210202T075743.gz...
Your notmuch database has now been upgraded.
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607947606.8134_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607940473.9509_1.no:2,S
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607969276.21046_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607987211.1395_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607979988.4942_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607972847.4857_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607943993.24776_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607976389.23296_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607983586.19063_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/drafts.mbox
Note: Ignoring non-mail file: /home/grfz/Mail/postponed.mbox
Processed 1183682 total files in 11h 33m 56s (28 files/sec.).
Added 1091038 new messages to the database.
grfz@mic:~/Mail/.notmuch$ cp -a xapian xapian-1
grfz@mic:~/Mail/.notmuch$ nice ionice -c 3 notmuch new
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607947606.8134_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607940473.9509_1.no:2,S
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607969276.21046_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607987211.1395_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607979988.4942_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607972847.4857_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607943993.24776_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607976389.23296_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607983586.19063_1.no:2,
Processed 1169095 total files in 13h 13m 16s (24 files/sec.).
Added 1077686 new messages to the database.
grfz@mic:~/Mail/.notmuch$ cp -a xapian xapian-2
grfz@mic:~/Mail/.notmuch$ nice ionice -c 3 notmuch new
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607947606.8134_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607940473.9509_1.no:2,S
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607969276.21046_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607987211.1395_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607979988.4942_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607972847.4857_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607943993.24776_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607976389.23296_1.no:2,
Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607983586.19063_1.no:2,
Processed 1151900 total files in 12h 55m 51s (24 files/sec.).
Added 1050106 new messages to the database.
grfz@mic:~/Mail/.notmuch$ cp -a xapian xapian-3
grfz@mic:~/Mail/.notmuch$ nice ionice -c 3 notmuch new
add_file: Out of memory files/sec.).
Processed 205 total files in 4s (42 files/sec.).
Added 193 new messages to the database.
Note: A fatal error was encountered: Out of memory
grfz@mic:~/Mail/.notmuch$

htop shows no memory pressure.  I reboot, move xapian
directory away, copy xapian-3 back to xapian and try again:

grfz@mic:~/Mail/.notmuch$ mv xapian xapian-OOM2
grfz@mic:~/Mail/.notmuch$ cp -a xapian-3 xapian
grfz@mic:~/Mail/.notmuch$ nice ionice -c 3 notmuch new                                                                                                                                         add_file: Out of memory files/sec.).
Processed 205 total files in 10s (19 files/sec.).
Added 193 new messages to the database.
Note: A fatal error was encountered: Out of memory
grfz@mic:~/Mail/.notmuch$ free
total        used        free      shared  buff/cache   available
Mem:       16394744      234788      234400        8724    15925556    15815084
Swap:      15622140         512    15621628
grfz@mic:~/Mail/.notmuch$


> Olly Betts mentioned in a different thread that he will build a version
> of xapian 1.4.18 for buster backports, so trying with that is probably a
> good step when it is available.

I'll do so.

>> grfz@mic:~/Mail/.notmuch$ mv xapian xapian-corrupted
>> grfz@mic:~/Mail/.notmuch$ notmuch new
>> Welcome to a new version of notmuch! Your database will now be upgraded.
>> This process is safe to interrupt.
>> Backing up tags to /home/grfz/Mail/.notmuch/dump-20210130T170349.gz...
>> Your notmuch database has now been upgraded.
>> Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607947606.8134_1.no:2,
>> Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607940473.9509_1.no:2,S
>> Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607969276.21046_1.no:2,
>> Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607987211.1395_1.no:2,
>> Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607979988.4942_1.no:2,
>> Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607972847.4857_1.no:2,
>> Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607943993.24776_1.no:2,
>> Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607976389.23296_1.no:2,
>> Note: Ignoring non-mail file: /home/grfz/Mail/spam-old/cur/1607983586.19063_1.no:2,
>> Note: Ignoring non-mail file: /home/grfz/Mail/drafts.mbox
>> Note: Ignoring non-mail file: /home/grfz/Mail/postponed.mbox
>> Processed 1183682 total files in 13h 38m 31s (24 files/sec.).
>> Added 1091038 new messages to the database.
>>
>> I then installed xapian-tools amd64 1.4.11-1.
>>
>> grfz@mic:~/Mail/.notmuch$ stat --format "%Y"  ~/Mail/inbox/cur
>> 1611646289
>>
>> grfz@mic:~/Mail/.notmuch$ quest -bdir:XDIRECTORY -d ~/Mail/.notmuch/xapian/ dir:inbox/cur
>> Parsed Query: Query(0 * XDIRECTORYinbox/cur)
>> MSet:
>>
>> That's it, there is data missing in the database.
>>
>
> This could either be a logic error in Notmuch,
>
> You can get a complete list of all of the directory documents in the
> notmuch database with
>
> % xapian-delve -1 -A XDIRECTORY ~/Mail/.notmuch/xapian | sort -u > delve.txt
>
> You can get a list of the actual directories with
>
> % find ~/Mail -type d -not empty | sed s,/home/grfz/Mail/,XDIRECTORY, |sort -u  >find.txt
>
> Comparing those two lists may give you some hints. Any directory that
> shows up in the second list but not the first should have no files in it
> (but potentially other directories) or be ignored either implicitly
> (.notmuch, .notmuch/xapian) or explicitely by your configuration.


I tried this after the second OOM (see above):

grfz@mic:~/Mail/.notmuch$ rm -rf xapian
grfz@mic:~/Mail/.notmuch$ cp -a xapian-3 xapian

grfz@mic:~/Mail/.notmuch$ xapian-delve -1 -A XDIRECTORY ~/Mail/.notmuch/xapian | sort -u > /tmp/delve.txt
grfz@mic:~/Mail/.notmuch$ find ~/Mail/  -type d -not -empty | sed s,/home/grfz/Mail/,XDIRECTORY, |sort -u  >/tmp/find.txt

As expected the find.txt contains a huge amount of ../cur
directories.

I searched for the directories in find.txt, which are not in
delve.txt:

grfz@mic:~/Mail/.notmuch$ while read ; do grep -qF "$REPLY" /tmp/delve.txt 2>/dev/null || echo $REPLY ; done < /tmp/find.txt > /tmp/out

It contains a huge amount of ../cur directories.

I searched if some of these ../cur directories do not
contain files:

grfz@mic:~/Mail/.notmuch$ cat /tmp/out | grep "/cur$" | sed -e "s,XDIRECTORY,/home/grfz/Mail/," | while read ; do cd "$REPLY" ; test $(find -type f | wc -l) = 0 && echo "$REPLY" ; done
/home/grfz/Mail/findex/cur

This one directory only contains symbolic links to maildir
mail files.  It is populated by mairx.

Otherwise there are no empty ../cur directories.  This was
to be expected.

grfz@mic:~/Mail/.notmuch$ grep "/cur$" /tmp/find.txt | while read ; do grep -qa "$REPLY" /tmp/delve.txt || echo "$REPLY" ; done | wc -l
5258

grfz@mic:~/Mail/.notmuch$ grep "/cur$" /tmp/find.txt | while read ; do grep -qa "$REPLY" /tmp/delve.txt || echo "$REPLY" ; done | sed -e "s,^XDIRECTORY,/home/grfz/Mail/," | while read ; do test $(find "$REPLY" | wc -l) -gt 1 || echo "$REPLY" ; done
find: ‘Binary file /tmp/find.txt matches’: No such file or directory
Binary file /tmp/find.txt matches
grfz@mic:~/Mail/.notmuch$


So there are 5258 directories which are in find.txt but not
in delve.txt and all of them at the same time have files in
them.

I don't really grasp the details, but to me it seems that a)
5288 directories are not indexed; and b) there is no
explanation for why almost all files were re-indexed several
times.


Ciao, Gregor\r

Re: out of memory on idle machine

[Olly Betts]

On Wed, Feb 03, 2021 at 07:59:43AM -0400, David Bremner wrote:
> Gregor Zattler <telegraph@gmx.net> writes:
> > A Xapian exception occurred finding message: Db block overwritten - are there multiple writers?.
> 
> I have included the Xapian list in copy in case that message rings a
> bell.

There was a bug fixed in 1.4.7 which incorrectly resulted in this error
message, but it seems from the quoted text you're using 1.4.11.

> I guess you know there are not multiple writers in your setup.

There's a lock file locked by fcntl() which protects against multiple
writers, so someone/something would have need to have deleted that
behind Xapian's back, or else a bug somewhere in the locking code stack.

(Aside from that bug, probably the most common case here over time has
been that someone deleted the lock file thinking it's "stale", but it's
not the mere presence of the file that means the lock is held.  It's
not at all frequent, but perhaps we should adjust this message to better
reflect that.)

Have you tried xapian-check on this database?

> Olly Betts mentioned in a different thread that he will build a version
> of xapian 1.4.18 for buster backports, so trying with that is probably a
> good step when it is available.

Yes - 1.4.18 packages are now in Debian testing, so hopefully I can get
this done soon.

> % xapian-delve -1 -A XDIRECTORY ~/Mail/.notmuch/xapian | sort -u > delve.txt

FWIW, the output should be sorted and unique already (sorted by byte
order, so equivalent to `LC_ALL=C sort`).

Cheers,
    Olly

Re: out of memory on idle machine

[David Bremner]

Gregor Zattler <telegraph@gmx.net> writes:

> Hi David, notmuch and xapian developers,
> * David Bremner <david@tethera.net> [03. Feb. 2021]:
>
>
>> Olly Betts mentioned in a different thread that he will build a version
>> of xapian 1.4.18 for buster backports, so trying with that is probably a
>> good step when it is available.
>
> I'll do so.
>

At this point I don't really have any good ideas, so I'm waiting for
results from the 1.4.18 trial.

>>
>> Comparing those two lists may give you some hints. Any directory that
>> shows up in the second list but not the first should have no files in it
>> (but potentially other directories) or be ignored either implicitly
>> (.notmuch, .notmuch/xapian) or explicitely by your configuration.
>
>
> I tried this after the second OOM (see above):
>
> grfz@mic:~/Mail/.notmuch$ rm -rf xapian
> grfz@mic:~/Mail/.notmuch$ cp -a xapian-3 xapian
>
> grfz@mic:~/Mail/.notmuch$ xapian-delve -1 -A XDIRECTORY ~/Mail/.notmuch/xapian | sort -u > /tmp/delve.txt
> grfz@mic:~/Mail/.notmuch$ find ~/Mail/  -type d -not -empty | sed s,/home/grfz/Mail/,XDIRECTORY, |sort -u  >/tmp/find.txt
>
> As expected the find.txt contains a huge amount of ../cur
> directories.
>
> I searched for the directories in find.txt, which are not in
> delve.txt:
>
> grfz@mic:~/Mail/.notmuch$ while read ; do grep -qF "$REPLY" /tmp/delve.txt 2>/dev/null || echo $REPLY ; done < /tmp/find.txt > /tmp/out
>
> It contains a huge amount of ../cur directories.
>
> I searched if some of these ../cur directories do not
> contain files:
>
> grfz@mic:~/Mail/.notmuch$ cat /tmp/out | grep "/cur$" | sed -e "s,XDIRECTORY,/home/grfz/Mail/," | while read ; do cd "$REPLY" ; test $(find -type f | wc -l) = 0 && echo "$REPLY" ; done
> /home/grfz/Mail/findex/cur

I don't have any /cur directories in my version. I do have a few (3 or 4) /tmp
directories that are apparently not indexed. That's a bit mysterious,
but nothing on the scale of what you are seeing.

d

Re: out of memory on idle machine

[David Bremner]

David Bremner <david@tethera.net> writes:

> Gregor Zattler <telegraph@gmx.net> writes:

> I don't have any /cur directories in my version. I do have a few (3 or 4) /tmp
> directories that are apparently not indexed. That's a bit mysterious,
> but nothing on the scale of what you are seeing.

Not so mysterious as it turns out. The tmp dirs in question had
filenames duplicated elsewhere in maildir (a violation of the maildir
spec), and where ignored by notmuch because they were in tmp/.  It
doesn't seem like either of these issues is relevant to your situation.

As a kind of desperation move, you could try bisecting your mailstore,
to see how small of a set of messages you can duplicate the problem
with.

d

Re: out of memory on idle machine

[Olly Betts]

On Thu, Feb 11, 2021 at 06:53:27AM -0400, David Bremner wrote:
> At this point I don't really have any good ideas, so I'm waiting for
> results from the 1.4.18 trial.

I've uploaded a backport, but it's the first backport of xapian-core to
buster so it'll need manual approval.  Hopefully that'll happen over the
weekend, but it could take longer.

Cheers,
    Olly

Re: out of memory on idle machine

[Gregor Zattler]

Hi Olly, notmuch and xapian developers,

sorry for late answer, I had problems with the test system:
* Olly Betts <olly@survex.com> [09. Feb. 2021]:
> On Wed, Feb 03, 2021 at 07:59:43AM -0400, David Bremner wrote:
>> Gregor Zattler <telegraph@gmx.net> writes:
>>> A Xapian exception occurred finding message: Db block overwritten - are there multiple writers?.
>>
>> I have included the Xapian list in copy in case that message rings a
>> bell.
>
> There was a bug fixed in 1.4.7 which incorrectly resulted in this error
> message, but it seems from the quoted text you're using 1.4.11.

yes.

>> I guess you know there are not multiple writers in your setup.
>
> There's a lock file locked by fcntl() which protects against multiple
> writers, so someone/something would have need to have deleted that
> behind Xapian's back, or else a bug somewhere in the locking code stack.

There is no other writer.  The system is used only for this
test atm, the emails are on a dedicated data partition.

> (Aside from that bug, probably the most common case here over time has
> been that someone deleted the lock file thinking it's "stale", but it's
> not the mere presence of the file that means the lock is held.  It's
> not at all frequent, but perhaps we should adjust this message to better
> reflect that.)
>
> Have you tried xapian-check on this database?

not this time.  Yust wait.  I have different databases from
different runs of notmuch new, "xapian-3" being identical
with "xapian-3"


grfz@mic:~/Mail/.notmuch$ for name in xapian  xapian-3 xapian-2 xapian-1 ; do echo "===== $name"; xapian-check $name ; done > /tmp/xapian-checks 2>&1

results in:

===== xapian
docdata:
blocksize=8K items=577 firstunused=10 revision=4 levels=1 root=6
B-tree checked okay
docdata table structure checked OK

termlist:
blocksize=8K items=68006 firstunused=20302 revision=4 levels=2 root=687
xapian-check: DatabaseError: Block 17959: Used block also in freelist
===== xapian-3
docdata:
blocksize=8K items=577 firstunused=10 revision=4 levels=1 root=6
B-tree checked okay
docdata table structure checked OK

termlist:
blocksize=8K items=68006 firstunused=20302 revision=4 levels=2 root=687
xapian-check: DatabaseError: Block 17959: Used block also in freelist
===== xapian-2
docdata:
blocksize=8K items=577 firstunused=10 revision=3 levels=1 root=6
B-tree checked okay
docdata table structure checked OK

termlist:
blocksize=8K items=68006 firstunused=20302 revision=3 levels=2 root=8651
B-tree checked okay
termlist table structure checked OK

postlist:
blocksize=8K items=1971726 firstunused=25177 revision=3 levels=2 root=6651
B-tree checked okay
postlist table structure checked OK

position:
blocksize=8K items=9589984 firstunused=52204 revision=3 levels=2 root=18102
B-tree checked okay
position table structure checked OK

spelling:
Lazily created, and not yet used.

synonym:
Lazily created, and not yet used.

No errors found
===== xapian-1
docdata:
blocksize=8K items=371 firstunused=4 revision=2 levels=1 root=2
B-tree checked okay
docdata table structure checked OK

termlist:
blocksize=8K items=32426 firstunused=8650 revision=2 levels=2 root=687
B-tree checked okay
termlist table structure checked OK

postlist:
blocksize=8K items=562665 firstunused=6650 revision=2 levels=2 root=380
B-tree checked okay
postlist table structure checked OK

position:
blocksize=8K items=4359147 firstunused=18099 revision=2 levels=2 root=377
B-tree checked okay
position table structure checked OK

spelling:
Lazily created, and not yet used.

synonym:
Lazily created, and not yet used.

No errors found



There are no problems reported on earlier databases although
they resulted in reindexing of almost all of the emails and
in case of the corrupted xapian database not all file are
checked.


This is not fixable:
grfz@mic:~/Mail/.notmuch$ cp -a xapian xapian-checked
grfz@mic:~/Mail/.notmuch$ xapian-check xapian-checked F
docdata:
B-tree checked okay
docdata table structure checked OK

termlist:
xapian-check: DatabaseError: Block 17959: Used block also in freelist

grfz@mic:~/Mail/.notmuch$ xapian-check xapian-checked
docdata:
blocksize=8K items=577 firstunused=10 revision=4 levels=1 root=6
B-tree checked okay
docdata table structure checked OK

termlist:
blocksize=8K items=68006 firstunused=20302 revision=4 levels=2 root=687
xapian-check: DatabaseError: Block 17959: Used block also in freelist


>> Olly Betts mentioned in a different thread that he will build a version
>> of xapian 1.4.18 for buster backports, so trying with that is probably a
>> good step when it is available.
>
> Yes - 1.4.18 packages are now in Debian testing, so hopefully I can get
> this done soon.

OK, I'll wait for that.

>> % xapian-delve -1 -A XDIRECTORY ~/Mail/.notmuch/xapian | sort -u > delve.txt
>
> FWIW, the output should be sorted and unique already (sorted by byte
> order, so equivalent to `LC_ALL=C sort`).

ok, thanks.

Ciao, Gregor
--
 -... --- .-. . -.. ..--.. ...-.-

Re: out of memory on idle machine

[Gregor Zattler]

Hi Olly, David, xapian and notmuch developers,
* Olly Betts <olly@survex.com> [12. Feb. 2021]:
> On Thu, Feb 11, 2021 at 06:53:27AM -0400, David Bremner wrote:
>> At this point I don't really have any good ideas, so I'm waiting for
>> results from the 1.4.18 trial.
>
> I've uploaded a backport, but it's the first backport of xapian-core to
> buster so it'll need manual approval.  Hopefully that'll happen over the
> weekend, but it could take longer.

I installed version 1.4.18 and the errors are the same.  I
will take Davids advice and try to bisect my mail store.
This will take some time.  I'll report back.


Ciao, Gregor
--
 -... --- .-. . -.. ..--.. ...-.-

bug: chokes on long directory names (was: Re: out of memory on idle machine)

[Gregor Zattler]

Hi David, Olly, notmuch and xapian developers,
* David Bremner <david@tethera.net> [11. Feb. 2021]:
> David Bremner <david@tethera.net> writes:
> As a kind of desperation move, you could try bisecting your mailstore,
> to see how small of a set of messages you can duplicate the problem
> with.

this I did, somehow.  I found the culprit: It's a maildir
with one single mail in it.  The name of the maildir is
exceptionally long [because generated from a List-Id:
-Header] and the mail arrived at the very day, my notmuch
database corrupted.  This maildir alone provokes that every
next notmuch new will rescan all (?) files.

Then I tried to only index this maildir, it showed the same
strange re-indexing but even when running notmuch new for a
while in a loop (>1000 times), the database showed no
corruption.

When instead I shorten the name of the maildir to three
characters with the very same email file in it, nothing
happens, it indexes the file once and not again.

Then I prolonged the name of the file instead of the
directory and even with the longest possible filename (or
path?)

/home/grfz/Mail/nuk/new/1607641473.31514_2.no1607641473.31514_2.no1607641473.31514_2.no1607641473.31514_2.no1607641473.31514_2.no1607641473.31514_2.no1607641473.31514_2.no1607641473.31514_2.no1607641473.31514_2.no1607641473.31514_2.no1607641473.31514_2.no16076414734160.14_2.no

notmuch has no problem indexing this and not to reindex it
in the next run.


So notmuch or xapian (I don't know) chokes on extreme long
directory names.  I consider this to be a bug.



My scripts create this long names from List-Id and some
such.  The one which triggered the problems is from an online
shop:

u+mq6tamjqhe3cm2j5giydembrgiytamrtga2deojogexdsmzygm4egnbuifatcnrsgazdejjugbzgkylmfvxw43djnzsxg2dpoaxgizjgna6ton3bg4zdsobsgmytczlcme3dentehaydmnjxmy4doyrwha4tgobgoi6xizlmmvtxeylqnastimdhnv4c43tfoqthipldovzxi33nmvzhgllxmvwgg33nmu@real-onlineshop.de/

Since, as I tested, this can be reproduced with the simplest
of email in a maildir with an extremly long name, I do not
attach the maildir in question.  But if anyone wants it I
can send it.



I then had a look at other long directory names and there is
another one which also triggers the problem, it also has
only one email in it and arrived on 12th of January:

u+mq6wcodfgmygcjtjhuzdamrrgaytemjrhe2dqmbqfyys4mbxgazugnbsie3doobsgfcdmobfgqygg5ltorxw2zlsomxgo2lunrqweltdn5wsm2b5mu3tkmddhbrdoyrwgvsgeobymi2dszbtg4zdamztmm4dsmzvgjssm4r5orswyzlhojqxa2bfgqygo3lyfzxgk5bgoq6xa4tjozqwg6i@customers.gitlab.com


Since I removed both on my laptop, notmuch new works again,
yeah!  Now I will have a look on my .procmailrc.

Thanks for your attention, thanks for notmuch and for xapian,
Grgeor

--
 -... --- .-. . -.. ..--.. ...-.-

[PATCH] test: add known broken test for long directory bug

[David Bremner]

In [1] Gregor Zattler explained the results of his hard working
tracking down a bug in notmuch with long directories. This test
duplicates the bug.

[1]: id:20210317194728.GB5561@no.workgroup
---
 test/T050-new.sh | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/test/T050-new.sh b/test/T050-new.sh
index 76bda959..f84dc2b0 100755
--- a/test/T050-new.sh
+++ b/test/T050-new.sh
@@ -339,6 +339,20 @@ test_expect_code 1 "NOTMUCH_NEW --debug 2>&1"
 
 notmuch config set new.tags $OLDCONFIG
 
+test_begin_subtest "Long directory names don't cause rescan"
+test_subtest_known_broken
+name=$(printf 'z%.0s' {1..234})
+generate_message [dir]=$name
+NOTMUCH_NEW  > OUTPUT
+notmuch new  >> OUTPUT
+rm -r ${MAIL_DIR}/${name}
+notmuch new >> OUTPUT
+cat <<EOF > EXPECTED
+Added 1 new message to the database.
+No new mail.
+No new mail. Removed 1 message.
+EOF
+test_expect_equal_file EXPECTED OUTPUT
 
 test_begin_subtest "Xapian exception: read only files"
 chmod u-w ${MAIL_DIR}/.notmuch/xapian/*.*
-- 
2.30.2

Re: bug: chokes on long directory names (was: Re: out of memory on idle machine)

[David Bremner]

Gregor Zattler <telegraph@gmx.net> writes:

> Hi David, Olly, notmuch and xapian developers,
> * David Bremner <david@tethera.net> [11. Feb. 2021]:
>> David Bremner <david@tethera.net> writes:
>> As a kind of desperation move, you could try bisecting your mailstore,
>> to see how small of a set of messages you can duplicate the problem
>> with.
>
> this I did, somehow.  I found the culprit: It's a maildir
> with one single mail in it.  The name of the maildir is
> exceptionally long [because generated from a List-Id:
> -Header] and the mail arrived at the very day, my notmuch
> database corrupted.  This maildir alone provokes that every
> next notmuch new will rescan all (?) files.

Hi Gregor;

I am very impressed with your persistence. I suspect it is a bug in
notmuch. I don't know all the details yet, but in the normal case the
directory name is added to the database prefixed with XDIRECTORY. I
noticed this isn't happening in the case of directories 234 or
longer. That is roughly the Xapian term limit of 245 characters in
total. I'm not sure why the discrepency of one character, but the main
point is that notmuch is probably improperly ignoring an error from
Xapian when adding these overlong terms.

Thanks again for the debugging, I suspect would have never found this
bug on my own.

David

Re: [PATCH] test: add known broken test for long directory bug

[Tomi Ollila]

On Wed, Mar 17 2021, David Bremner wrote:

> In [1] Gregor Zattler explained the results of his hard working
> tracking down a bug in notmuch with long directories. This test
> duplicates the bug.
>
> [1]: id:20210317194728.GB5561@no.workgroup
> ---
>  test/T050-new.sh | 14 ++++++++++++++
>  1 file changed, 14 insertions(+)
>
> diff --git a/test/T050-new.sh b/test/T050-new.sh
> index 76bda959..f84dc2b0 100755
> --- a/test/T050-new.sh
> +++ b/test/T050-new.sh
> @@ -339,6 +339,20 @@ test_expect_code 1 "NOTMUCH_NEW --debug 2>&1"
>  
>  notmuch config set new.tags $OLDCONFIG
>  
> +test_begin_subtest "Long directory names don't cause rescan"
> +test_subtest_known_broken
> +name=$(printf 'z%.0s' {1..234})

could do printf -v name 'z%.0s' {1..234}

> +generate_message [dir]=$name
> +NOTMUCH_NEW  > OUTPUT
> +notmuch new  >> OUTPUT

2 spaces in lines above

apart from those 2 spaces lgtm.

Tomi

> +rm -r ${MAIL_DIR}/${name}
> +notmuch new >> OUTPUT
> +cat <<EOF > EXPECTED
> +Added 1 new message to the database.
> +No new mail.
> +No new mail. Removed 1 message.
> +EOF
> +test_expect_equal_file EXPECTED OUTPUT
>  
>  test_begin_subtest "Xapian exception: read only files"
>  chmod u-w ${MAIL_DIR}/.notmuch/xapian/*.*
> -- 
> 2.30.2

Re: [PATCH] test: add known broken test for long directory bug

[David Bremner]

Tomi Ollila <tomi.ollila@iki.fi> writes:

> On Wed, Mar 17 2021, David Bremner wrote:
>
> could do printf -v name 'z%.0s' {1..234}
>
>> +generate_message [dir]=$name
>> +NOTMUCH_NEW  > OUTPUT
>> +notmuch new  >> OUTPUT
>
> 2 spaces in lines above
>
> apart from those 2 spaces lgtm.
>
> Tomi

Thanks. Pushed with those changes.

d

[PATCH] lib/n_d_index_file: check return value from _n_m_add_filename

[David Bremner]

Ignoring this return value seems like a bad idea in general, and in
particular it has been hiding one or more bugs related to handling
long directory names.
---

This is not a fix for the aforementioned bugs, but it at least makes
clear part of the problem.  The XDDIRENTRYnnnnn: terms are not checked
for length in the same way as XDIRECTORY terms. It isn't clear the
same hashing strategy will work, as the XDDIRECTORY terms are used to
create lists of child directories. It may be the best we can do is
enforce a limit on the length of path elements in trees indexed by
notmuch.

 lib/add-message.cc | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/lib/add-message.cc b/lib/add-message.cc
index 485debad..0c34d318 100644
--- a/lib/add-message.cc
+++ b/lib/add-message.cc
@@ -529,7 +529,9 @@ notmuch_database_index_file (notmuch_database_t *notmuch,
 	    goto DONE;
 	}
 
-	_notmuch_message_add_filename (message, filename);
+	ret = _notmuch_message_add_filename (message, filename);
+	if (ret)
+	    goto DONE;
 
 	if (is_new || is_ghost) {
 	    _notmuch_message_add_term (message, "type", "mail");
-- 
2.30.2

Re: [PATCH] lib/n_d_index_file: check return value from _n_m_add_filename

[David Bremner]

David Bremner <david@tethera.net> writes:

> Ignoring this return value seems like a bad idea in general, and in
> particular it has been hiding one or more bugs related to handling
> long directory names.
> ---
>
> This is not a fix for the aforementioned bugs, but it at least makes
> clear part of the problem.  The XDDIRENTRYnnnnn: terms are not checked
> for length in the same way as XDIRECTORY terms. It isn't clear the
> same hashing strategy will work, as the XDDIRECTORY terms are used to
> create lists of child directories. It may be the best we can do is
> enforce a limit on the length of path elements in trees indexed by
> notmuch.

Applied to master.

d