Xapian exception leading to database corruption

Xapian exception leading to database corruption

[David Edmondson]

Using current git notmuch on Debian testing a rebuild from scratch of my
database fails:

> agrajag-testing ~/s/notmuch % ./notmuch new
> Found 605510 total files (that's not much mail).    
> add_file: A Xapian exception occurred 28m 37s remaining).
> A Xapian exception occurred adding message: Unexpected end of posting list for 'G0000000000014364'.
> Processed 137296 total files in 8h 4m 45s (4 files/sec.).
> Added 135950 new messages to the database.
> Note: A fatal error was encountered: A Xapian exception occurred
> agrajag-testing ~/s/notmuch %                          

Checking the database at this point seems fine:

> agrajag-testing ~/s/notmuch % xapian-check ~/Maildir/.notmuch/xapian 
> docdata:
> blocksize=8K items=0 firstunused=1 revision=1 levels=0 root=(faked)
> void B-tree checked okay
> docdata table structure checked OK
> 
> termlist:
> blocksize=8K items=0 firstunused=2 revision=1 levels=0 root=(faked)
> void B-tree checked okay
> termlist table structure checked OK
> 
> postlist:
> blocksize=8K items=2 firstunused=1 revision=1 levels=0 root=0
> B-tree checked okay
> postlist table structure checked OK
> 
> position:
> blocksize=8K items=0 firstunused=1 revision=1 levels=0 root=(faked)
> void B-tree checked okay
> position table structure checked OK
> 
> spelling:
> Lazily created, and not yet used.
> 
> synonym:
> Lazily created, and not yet used.
> 
> No errors found
> agrajag-testing ~/s/notmuch % 

If I run “notmuch new” again then the exception repeats, except the
database is listed as corrupt (this is actually from an earlier
cycle of testing, but it is reproducible):

> agrajag-testing ~/s/notmuch % ./notmuch new
> add_file: A Xapian exception occurred
> A Xapian exception occurred adding message: Unexpected end of posting list for 'G0000000000013f27'.
> Processed 16 total files in 59s (0 files/sec.).
> Added 15 new messages to the database.
> Note: A fatal error was encountered: A Xapian exception occurred
> agrajag-testing ~/s/notmuch % xapian-check ~/Maildir/.notmuch/xapian  
> docdata:
> blocksize=8K items=15 firstunused=3 revision=9 levels=0 root=2
> B-tree checked okay
> docdata table structure checked OK
> 
> termlist:
> blocksize=8K items=301848 firstunused=68658 revision=9 levels=2 root=14019
> xapian-check: DatabaseError: 1 unused block(s) missing from the free list, first is 0
> agrajag-testing ~/s/notmuch % 

I see that bremner reported something like this in #xapian, but not any
resolution.

Any suggestions? Is it possible to force a new chert database to be
created rather than glass? (Mostly I'd like to get back to work!)

dme.
-- 
But uh oh, I love her because, she moves in her own way.

Re: Xapian exception leading to database corruption

[David Bremner]

David Edmondson <dme@dme.org> writes:

> Using current git notmuch on Debian testing a rebuild from scratch of my
> database fails:
>
>> agrajag-testing ~/s/notmuch % ./notmuch new
>> Found 605510 total files (that's not much mail).    
>> add_file: A Xapian exception occurred 28m 37s remaining).
>> A Xapian exception occurred adding message: Unexpected end of posting list for 'G0000000000014364'.
>> Processed 137296 total files in 8h 4m 45s (4 files/sec.).
>> Added 135950 new messages to the database.
>> Note: A fatal error was encountered: A Xapian exception occurred
>> agrajag-testing ~/s/notmuch %                          
>

I can't duplicate this (probably not surprising) on debian testing. I
also have about 600k files, and I'm running debian testing. My total
index time is about 1h on a not-very-recent machine with an SSD. I'm
guessing you have a hard disk (or something is deeply wrong to take
that long). Even for a hard disk, 8h to index 137k files seems slow.
Did you happen to check for disk errors?

> I see that bremner reported something like this in #xapian, but not any
> resolution.

I guess your best bet is to write to

  xapian-discuss@lists.xapian.org

>
> Any suggestions? Is it possible to force a new chert database to be
> created rather than glass? (Mostly I'd like to get back to work!)

According to Olly Betts,

   With 1.4 you can pass Xapian::DB_BACKEND_CHERT in the flags when
   constructing the WritableDatabase object.

So you'd have to hack the notmuch source, I'm guessing around line 55 of
database.cc

#define DB_ACTION (Xapian::DB_CREATE_OR_OPEN | Xapian::DB_RETRY_LOCK | Xapian::DB_BACKEND_CHERT)

Do move the existing database out of the way, or apparently xapian can
get confused.

Re: Xapian exception leading to database corruption

[David Edmondson]

On Thursday, 2017-12-28 at 22:00:56 -0400, David Bremner wrote:

> David Edmondson <dme@dme.org> writes:
>
>> Using current git notmuch on Debian testing a rebuild from scratch of my
>> database fails:
>>
>>> agrajag-testing ~/s/notmuch % ./notmuch new
>>> Found 605510 total files (that's not much mail).    
>>> add_file: A Xapian exception occurred 28m 37s remaining).
>>> A Xapian exception occurred adding message: Unexpected end of posting list for 'G0000000000014364'.
>>> Processed 137296 total files in 8h 4m 45s (4 files/sec.).
>>> Added 135950 new messages to the database.
>>> Note: A fatal error was encountered: A Xapian exception occurred
>>> agrajag-testing ~/s/notmuch %                          
>>
>
> I can't duplicate this (probably not surprising) on debian testing. I
> also have about 600k files, and I'm running debian testing. My total
> index time is about 1h on a not-very-recent machine with an SSD. I'm
> guessing you have a hard disk (or something is deeply wrong to take
> that long). Even for a hard disk, 8h to index 137k files seems slow.
> Did you happen to check for disk errors?

That was running under valgrind (after seeing the discussion in
#xapian). valgrind only complains about some things in debugger.c, which
seems mildly ironic.

Without valgrind the time estimate for 600k files is around 2.5 hours,
though I suspect in reality it will be more than this.

The files are stored on a ZFS mirror of two SSDs. The machine is an AMD
X3216, so it doesn't have particularly quick CPUs and “only” 8G RAM.

When I said “Debian testing”, perhaps I should have been more explicit -
it's a Debian testing docker container running on Debian stable. I don't
think that this really matters - the failure happens with the version of
notmuch from stable backports on the host. The error messages are just
better from notmuch git and I'm trying to avoid installing all of the
development tools on the host.

>> I see that bremner reported something like this in #xapian, but not any
>> resolution.
>
> I guess your best bet is to write to
>
>   xapian-discuss@lists.xapian.org

I'll go there, thanks.

>>
>> Any suggestions? Is it possible to force a new chert database to be
>> created rather than glass? (Mostly I'd like to get back to work!)
>
> According to Olly Betts,
>
>    With 1.4 you can pass Xapian::DB_BACKEND_CHERT in the flags when
>    constructing the WritableDatabase object.
>
> So you'd have to hack the notmuch source, I'm guessing around line 55 of
> database.cc
>
> #define DB_ACTION (Xapian::DB_CREATE_OR_OPEN | Xapian::DB_RETRY_LOCK | Xapian::DB_BACKEND_CHERT)
>
> Do move the existing database out of the way, or apparently xapian can
> get confused.

Thanks.

dme.
-- 
But whichever way I go, I come back to the place you are.

Re: Xapian exception leading to database corruption

[David Edmondson]

On Friday, 2017-12-29 at 10:40:02 UTC, David Edmondson wrote:

>>> Any suggestions? Is it possible to force a new chert database to be
>>> created rather than glass? (Mostly I'd like to get back to work!)
>>
>> According to Olly Betts,
>>
>>    With 1.4 you can pass Xapian::DB_BACKEND_CHERT in the flags when
>>    constructing the WritableDatabase object.
>>
>> So you'd have to hack the notmuch source, I'm guessing around line 55 of
>> database.cc
>>
>> #define DB_ACTION (Xapian::DB_CREATE_OR_OPEN | Xapian::DB_RETRY_LOCK | Xapian::DB_BACKEND_CHERT)

This worked and the initial scan ran through to completion (making
faster progress than the glass based version).

dme.
-- 
You can't hide from the flipside.