Exception: Expected block 59423 to be level 2, not 0

Exception: Expected block 59423 to be level 2, not 0

[Kevin Hilman]

I recently started seeing this error for `lei up`:

  $ lei up -v dummy
  Exception: Expected block 59423 to be level 2, not 0

The same thing happens when trying an edit-search:

  $ lei edit-search dummy
  Exception: Expected block 59423 to be level 2, not 0

The same thing happens whether "dummy" is a valid saved-search or not.
A bit of searching suggests that this is a xapian error.  I have
xapian-tools installed on this server, should I run some checks on the
xapian db?  If so, where is it stored?  Everything I find under
~/.local/share/lei seems to be sqlite3 db.

Thanks for any pointers,

Kevin

Re: Exception: Expected block 59423 to be level 2, not 0

[Kevin Hilman]

On Fri, Aug 30, 2024 at 12:40 PM Kevin Hilman <khilman@baylibre.com> wrote:
>
> I recently started seeing this error for `lei up`:
>
>   $ lei up -v dummy
>   Exception: Expected block 59423 to be level 2, not 0
>
> The same thing happens when trying an edit-search:
>
>   $ lei edit-search dummy
>   Exception: Expected block 59423 to be level 2, not 0
>
> The same thing happens whether "dummy" is a valid saved-search or not.
> A bit of searching suggests that this is a xapian error.  I have
> xapian-tools installed on this server, should I run some checks on the
> xapian db?  If so, where is it stored?  Everything I find under
> ~/.local/share/lei seems to be sqlite3 db.

I forgot to mention, I'm running the v1.9.0 tag of public-inbox, but
also tried with the master branch and got the same results.

Kevin

Re: Exception: Expected block 59423 to be level 2, not 0

[Eric Wong]

Kevin Hilman <khilman@baylibre.com> wrote:
> On Fri, Aug 30, 2024 at 12:40 PM Kevin Hilman <khilman@baylibre.com> wrote:
> >
> > I recently started seeing this error for `lei up`:
> >
> >   $ lei up -v dummy
> >   Exception: Expected block 59423 to be level 2, not 0
> >
> > The same thing happens when trying an edit-search:
> >
> >   $ lei edit-search dummy
> >   Exception: Expected block 59423 to be level 2, not 0
> >
> > The same thing happens whether "dummy" is a valid saved-search or not.
> > A bit of searching suggests that this is a xapian error.  I have
> > xapian-tools installed on this server, should I run some checks on the
> > xapian db?  If so, where is it stored?  Everything I find under
> > ~/.local/share/lei seems to be sqlite3 db.
> 
> I forgot to mention, I'm running the v1.9.0 tag of public-inbox, but
> also tried with the master branch and got the same results.

This looks like file corruption of some sort with Xapian...
Not my area of expertise.

Perhaps xapian-check(1) on each of the 0..$N directories under
~/.local/share/lei/store/ei15 would help (where $N is 3 on most
SMP systems, YMMV).

Any failing HW (SMART errors, dmesg), dirty shutdowns, or non-ECC RAM?
Otherwise it might be something the xapian-discuss list can help with...

Re: Exception: Expected block 59423 to be level 2, not 0

[Kevin Hilman]

Eric Wong <e@80x24.org> writes:

> Kevin Hilman <khilman@baylibre.com> wrote:
>> On Fri, Aug 30, 2024 at 12:40 PM Kevin Hilman <khilman@baylibre.com> wrote:
>> >
>> > I recently started seeing this error for `lei up`:
>> >
>> >   $ lei up -v dummy
>> >   Exception: Expected block 59423 to be level 2, not 0
>> >
>> > The same thing happens when trying an edit-search:
>> >
>> >   $ lei edit-search dummy
>> >   Exception: Expected block 59423 to be level 2, not 0
>> >
>> > The same thing happens whether "dummy" is a valid saved-search or not.
>> > A bit of searching suggests that this is a xapian error.  I have
>> > xapian-tools installed on this server, should I run some checks on the
>> > xapian db?  If so, where is it stored?  Everything I find under
>> > ~/.local/share/lei seems to be sqlite3 db.
>> 
>> I forgot to mention, I'm running the v1.9.0 tag of public-inbox, but
>> also tried with the master branch and got the same results.
>
> This looks like file corruption of some sort with Xapian...
> Not my area of expertise.
>
> Perhaps xapian-check(1) on each of the 0..$N directories under
> ~/.local/share/lei/store/ei15 would help (where $N is 3 on most
> SMP systems, YMMV).

Indeed, xapian-check found a corrupt database in
~/.local/share/lei/store/ei15/2, but and spits out the same error as
$SUBJECT.  However, it is not able to correct it[1]  :(

Assuming my xapian db is unrecoverable, is there any way to rebuild it?
or do I just need to delete it & re-run all my saved searches,
presumably losing all the past mail from any searches.

> Any failing HW (SMART errors, dmesg), dirty shutdowns, or non-ECC RAM?

Not that I know of.

> Otherwise it might be something the xapian-discuss list can help with...

OK, thanks for the pointer.

Kevin

[1]
$ xapian-check 2 F
Database couldn't be opened for reading: DatabaseCorruptError: Expected block 59423 to be level 2, not 0
Continuing check anyway
docdata:
B-tree checked okay
docdata table structure checked OK

termlist:
xapian-check: DatabaseCorruptError: Block overwritten - run xapian-check on this database

Re: Exception: Expected block 59423 to be level 2, not 0

[Eric Wong]

Kevin Hilman <khilman@baylibre.com> wrote:
> Indeed, xapian-check found a corrupt database in
> ~/.local/share/lei/store/ei15/2, but and spits out the same error as
> $SUBJECT.  However, it is not able to correct it[1]  :(
> 
> Assuming my xapian db is unrecoverable, is there any way to rebuild it?
> or do I just need to delete it & re-run all my saved searches,
> presumably losing all the past mail from any searches.

I haven't tried it, but probably removing the Xapian directories
and `lei reindex' with current public-inbox.git

I just pushed out some changes the other week which should make
it more robust against missing Xapian docs, but have yet to test
against non-existent Xapian.

> > Any failing HW (SMART errors, dmesg), dirty shutdowns, or non-ECC RAM?
> 
> Not that I know of.

I do wonder what caused it, AFAIK there's nothing in our Xapian
library usage which might cause this.  Perhaps an FS-level fsck
would be helpful.  Similarly, I would do git fsck and/or gc
and try the integrity_check pragma in SQLite just in case...