Re: out of memory on idle machine

[Gregor Zattler <telegraph@gmx.net>]

Hi Olly, notmuch and xapian developers,

sorry for late answer, I had problems with the test system:
* Olly Betts <olly@survex.com> [09. Feb. 2021]:
> On Wed, Feb 03, 2021 at 07:59:43AM -0400, David Bremner wrote:
>> Gregor Zattler <telegraph@gmx.net> writes:
>>> A Xapian exception occurred finding message: Db block overwritten - are there multiple writers?.
>>
>> I have included the Xapian list in copy in case that message rings a
>> bell.
>
> There was a bug fixed in 1.4.7 which incorrectly resulted in this error
> message, but it seems from the quoted text you're using 1.4.11.

yes.

>> I guess you know there are not multiple writers in your setup.
>
> There's a lock file locked by fcntl() which protects against multiple
> writers, so someone/something would have need to have deleted that
> behind Xapian's back, or else a bug somewhere in the locking code stack.

There is no other writer.  The system is used only for this
test atm, the emails are on a dedicated data partition.

> (Aside from that bug, probably the most common case here over time has
> been that someone deleted the lock file thinking it's "stale", but it's
> not the mere presence of the file that means the lock is held.  It's
> not at all frequent, but perhaps we should adjust this message to better
> reflect that.)
>
> Have you tried xapian-check on this database?

not this time.  Yust wait.  I have different databases from
different runs of notmuch new, "xapian-3" being identical
with "xapian-3"


grfz@mic:~/Mail/.notmuch$ for name in xapian  xapian-3 xapian-2 xapian-1 ; do echo "===== $name"; xapian-check $name ; done > /tmp/xapian-checks 2>&1

results in:

===== xapian
docdata:
blocksize=8K items=577 firstunused=10 revision=4 levels=1 root=6
B-tree checked okay
docdata table structure checked OK

termlist:
blocksize=8K items=68006 firstunused=20302 revision=4 levels=2 root=687
xapian-check: DatabaseError: Block 17959: Used block also in freelist
===== xapian-3
docdata:
blocksize=8K items=577 firstunused=10 revision=4 levels=1 root=6
B-tree checked okay
docdata table structure checked OK

termlist:
blocksize=8K items=68006 firstunused=20302 revision=4 levels=2 root=687
xapian-check: DatabaseError: Block 17959: Used block also in freelist
===== xapian-2
docdata:
blocksize=8K items=577 firstunused=10 revision=3 levels=1 root=6
B-tree checked okay
docdata table structure checked OK

termlist:
blocksize=8K items=68006 firstunused=20302 revision=3 levels=2 root=8651
B-tree checked okay
termlist table structure checked OK

postlist:
blocksize=8K items=1971726 firstunused=25177 revision=3 levels=2 root=6651
B-tree checked okay
postlist table structure checked OK

position:
blocksize=8K items=9589984 firstunused=52204 revision=3 levels=2 root=18102
B-tree checked okay
position table structure checked OK

spelling:
Lazily created, and not yet used.

synonym:
Lazily created, and not yet used.

No errors found
===== xapian-1
docdata:
blocksize=8K items=371 firstunused=4 revision=2 levels=1 root=2
B-tree checked okay
docdata table structure checked OK

termlist:
blocksize=8K items=32426 firstunused=8650 revision=2 levels=2 root=687
B-tree checked okay
termlist table structure checked OK

postlist:
blocksize=8K items=562665 firstunused=6650 revision=2 levels=2 root=380
B-tree checked okay
postlist table structure checked OK

position:
blocksize=8K items=4359147 firstunused=18099 revision=2 levels=2 root=377
B-tree checked okay
position table structure checked OK

spelling:
Lazily created, and not yet used.

synonym:
Lazily created, and not yet used.

No errors found



There are no problems reported on earlier databases although
they resulted in reindexing of almost all of the emails and
in case of the corrupted xapian database not all file are
checked.


This is not fixable:
grfz@mic:~/Mail/.notmuch$ cp -a xapian xapian-checked
grfz@mic:~/Mail/.notmuch$ xapian-check xapian-checked F
docdata:
B-tree checked okay
docdata table structure checked OK

termlist:
xapian-check: DatabaseError: Block 17959: Used block also in freelist

grfz@mic:~/Mail/.notmuch$ xapian-check xapian-checked
docdata:
blocksize=8K items=577 firstunused=10 revision=4 levels=1 root=6
B-tree checked okay
docdata table structure checked OK

termlist:
blocksize=8K items=68006 firstunused=20302 revision=4 levels=2 root=687
xapian-check: DatabaseError: Block 17959: Used block also in freelist


>> Olly Betts mentioned in a different thread that he will build a version
>> of xapian 1.4.18 for buster backports, so trying with that is probably a
>> good step when it is available.
>
> Yes - 1.4.18 packages are now in Debian testing, so hopefully I can get
> this done soon.

OK, I'll wait for that.

>> % xapian-delve -1 -A XDIRECTORY ~/Mail/.notmuch/xapian | sort -u > delve.txt
>
> FWIW, the output should be sorted and unique already (sorted by byte
> order, so equivalent to `LC_ALL=C sort`).

ok, thanks.

Ciao, Gregor
--
 -... --- .-. . -.. ..--.. ...-.-