Error while compacting: Bad position key

Error while compacting: Bad position key

[Mike Hommey]

Hi,

When running `notmuch compact` today, it stopped with the following
output:

Compacting database...
compacting table postlist
     Reduced by 25% 648656K (2498904K -> 1850248K)
compacting table docdata
     Reduced by 15% 24K (152K -> 128K)
compacting table termlist
     Reduced by 1% 27008K (2211800K -> 2184792K)
compacting table position
Error while compacting: Bad position key

Compaction failed: A Xapian exception occurred

Running xapian-check says:

docdata:
blocksize=8K items=2677 firstunused=19 revision=15425 levels=1 root=17
B-tree checked okay
docdata table structure checked OK

termlist:
blocksize=8K items=2986940 firstunused=276475 revision=15425 levels=2 root=271786
B-tree checked okay
doclen not within bounds
(...)
doclen not within bounds
termlist table errors found: 107982

postlist:
blocksize=8K items=16090818 firstunused=312363 revision=15425 levels=3 root=249894
B-tree checked okay
postlist table structure checked OK

position:
blocksize=8K items=236476398 firstunused=653990 revision=15425 levels=3 root=598684
xapian-check: DatabaseError: Block 459158 item 179: not in sorted order


Is there something I can do, or do I essentially need to completely
rebuild the database (and if so, what's the best way to do it?)

Cheers,

Mike

Re: Error while compacting: Bad position key

[David Bremner]

Mike Hommey <mh@glandium.org> writes:

> Hi,
>
> When running `notmuch compact` today, it stopped with the following
> output:
>
> Compacting database...
> compacting table postlist
>      Reduced by 25% 648656K (2498904K -> 1850248K)
> compacting table docdata
>      Reduced by 15% 24K (152K -> 128K)
> compacting table termlist
>      Reduced by 1% 27008K (2211800K -> 2184792K)
> compacting table position
> Error while compacting: Bad position key

I had not seen anything like this before, but when I run xapian-check
from xapian 1.4.6-2, I see


termlist:
B-tree checked okay
doclen not within bounds
doclen not within bounds
doclen not within bounds
doclen not within bounds
doclen not within bounds
doclen not within bounds
doclen not within bounds
doclen not within bounds

[repeated 52498 times]

OTOH, notmuch compact completes for me, so that might be unrelated.
I'm copying the Xapian list for input.

> Compaction failed: A Xapian exception occurred
>
> Running xapian-check says:
>
> docdata:
> blocksize=8K items=2677 firstunused=19 revision=15425 levels=1 root=17
> B-tree checked okay
> docdata table structure checked OK
>
> termlist:
> blocksize=8K items=2986940 firstunused=276475 revision=15425 levels=2 root=271786
> B-tree checked okay
> doclen not within bounds
> (...)
> doclen not within bounds
> termlist table errors found: 107982
>
> postlist:
> blocksize=8K items=16090818 firstunused=312363 revision=15425 levels=3 root=249894
> B-tree checked okay
> postlist table structure checked OK
>
> position:
> blocksize=8K items=236476398 firstunused=653990 revision=15425 levels=3 root=598684
> xapian-check: DatabaseError: Block 459158 item 179: not in sorted order
>

What version of Xapian is this?  I guess an obvious question is if
there is some potential external cause of filesystem corruption
(e.g. smart errrors, power loss at an inconvenient time).

> Is there something I can do, or do I essentially need to completely
> rebuild the database (and if so, what's the best way to do it?)
>

xapian-check has an option F to attempt to fix things. It might not
help, but it shouldn't hurt. Otherwise something like

% notmuch dump > backup.txt
% mv Maildir/.notmuch $backup
% notmuch new
% notmuch restore < backup.txt

will rebuild the database.

Re: Error while compacting: Bad position key

[Olly Betts]

On Thu, Jul 12, 2018 at 08:22:44AM -0300, David Bremner wrote:
> Mike Hommey <mh@glandium.org> writes:
> > When running `notmuch compact` today, it stopped with the following
> > output:
> >
> > Compacting database...
> > compacting table postlist
> >      Reduced by 25% 648656K (2498904K -> 1850248K)
> > compacting table docdata
> >      Reduced by 15% 24K (152K -> 128K)
> > compacting table termlist
> >      Reduced by 1% 27008K (2211800K -> 2184792K)
> > compacting table position
> > Error while compacting: Bad position key
> 
> I had not seen anything like this before, but when I run xapian-check
> from xapian 1.4.6-2, I see
> 
> 
> termlist:
> B-tree checked okay
> doclen not within bounds
> [repeated 52498 times]

That's a bug introduced in 1.4.6.  I've pushed a fix to master and will
backport for 1.4.7 (which I think will probably be this coming week as
there are several annoying issues found in 1.4.6).

> OTOH, notmuch compact completes for me, so that might be unrelated.

Yeah, that's unrelated to Mike's problem.

> > Running xapian-check says:
[...]
> > termlist:
> > blocksize=8K items=2986940 firstunused=276475 revision=15425 levels=2 root=271786
> > B-tree checked okay
> > doclen not within bounds
> > (...)
> > doclen not within bounds
> > termlist table errors found: 107982
[...]
> > position:
> > blocksize=8K items=236476398 firstunused=653990 revision=15425 levels=3 root=598684
> > xapian-check: DatabaseError: Block 459158 item 179: not in sorted order
> 
> What version of Xapian is this?  I guess an obvious question is if
> there is some potential external cause of filesystem corruption
> (e.g. smart errrors, power loss at an inconvenient time).

The "doclen not within bounds" message above means this must be 1.4.6
(though the database was probably built by an older version originally).

I don't think I've ever seen a report of "not in sorted order" before,
so if it's a Xapian bug it's either very rarely triggered or this is
an unusual manifestation.

I could make a patch which reports more details if you're happy to
build xapian-core from source.

> xapian-check has an option F to attempt to fix things.

It won't fix this case though.  Currently what it can do is regenerate
the small files in the database directory (which are sometimes truncated
to zero size after an OS crash).

Cheers,
    Olly