Re: Bug#808610: libxapian22: xapian database corruption causes recollindex loop

Re: Bug#808610: libxapian22: xapian database corruption causes recollindex loop

[Eric Wong]

Olly Betts <olly@survex.com> wrote:
> On Mon, Dec 28, 2015 at 02:45:43AM +0000, Olly Betts wrote:
> > There haven't been many backend changes since 1.2.19 - my guess is it's
> > this one:
> > 
> > http://trac.xapian.org/changeset/826d1a19cc356e7bf66c1681626e70af32967447/git
> > 
> > Can you test 1.2.19 plus that patch to see if this solves your problem?
> > 
> > (If that is the fix, then testing and unstable aren't affected).
> 
> Still awaiting your response on this.

I encountered a problem with Jessie, too, and this might be it.
I'm testing the above change on the public-inbox.git tree
(instructions below).  Will update in 24-48 hours...

Meanwhile, in case anybody isn't familiar with patching + building
Debian packages.... (and most of my Debian knowledge is 15 years old,
so maybe I'm not up-to-date with latest practices).

Download the unified diff from the above Trac URL, it should
save to a file like:

changeset_826d1a19cc356e7bf66c1681626e70af32967447.diff

Make sure deb-src lines for jessie are available in
/etc/apt/sources.list (or a file in /etc/apt/sources.list.d/)

	deb-src http://http.us.debian.org/debian/ jessie-updates main
	deb-src http://http.us.debian.org/debian/ jessie main
	deb-src http://http.us.debian.org/debian/ jessie-backports main

# install build prereqs (need root):
sudo apt-get update # make sure deb-src sources are known
sudo apt-get install build-essential fakeroot
sudo apt-get build-dep xapian-core

# grab the source (as a normal user)
apt-get source xapian-core=1.2.19-1

# enter the unpatched source directory
cd xapian-core-1.2.19

# apply the patch you downloaded above
# you'll see some warnings about fuzz which seem harmless for this:
patch -p1 </path/to/changeset_826d1a19cc356e7bf66c1681626e70af32967447.diff

# Still in the xapian-core-1.2.19 dir, run the build and wait a while:
debian/rules build

# build the Debian binary packages (.deb files)
fakeroot debian/rules binary

# At the end, you should see messages about .deb files being
# built in the parent directory, install the libxapian22 deb:
sudo dpkg -i ../libxapian22_1.2.19-1_amd64.deb


I'm testing this against t/search.t in the public-inbox.git sources

# fetch prereqs and sources:
sudo apt-get install libsearch-xapian-perl libemail-mime-perl
git clone git://80x24.org/public-inbox

# enter the source tree
cd public-inbox

# reset to the HEAD as of this writing
git reset --hard 9bae4c858d983d0f0cf7314b50f613818a58d5c5

# Audit t/search.t and any files it uses/requires (in lib)
# to ensure it won't eat your hard drive or send me all your
# personal info :)

# test in a loop:
while perl -I lib t/search.t; do :; done
# let the loop run for a few hours/days...

Even without the patch, it can take thousands of tries before a
failure manifests along the lines of:

  Exception: find_entry failed to find any entry at all!

Or something with "basic_string" in it...

I never had any problems under Debian wheezy.

Re: Bug#808610: libxapian22: xapian database corruption causes recollindex loop

[Eric Wong]

Eric Wong <e@80x24.org> wrote:
> > On Mon, Dec 28, 2015 at 02:45:43AM +0000, Olly Betts wrote:
> > > There haven't been many backend changes since 1.2.19 - my guess is it's
> > > this one:
> > > 
> > > http://trac.xapian.org/changeset/826d1a19cc356e7bf66c1681626e70af32967447/git
> 
> I encountered a problem with Jessie, too, and this might be it.
> I'm testing the above change on the public-inbox.git tree
> (instructions below).  Will update in 24-48 hours...

28 hours or so and my loop (below) is still going! \o/
I think the longest it ran previously was maybe an hour or two.
I'll let it run another day to be certain, but I'm fairly
certain this needs to be backported for Jessie users.

<snip>

> while perl -I lib t/search.t; do :; done

Re: Bug#808610: libxapian22: xapian database corruption causes recollindex loop

[Eric Wong]

> > > On Mon, Dec 28, 2015 at 02:45:43AM +0000, Olly Betts wrote:
> > > > http://trac.xapian.org/changeset/826d1a19cc356e7bf66c1681626e70af32967447/git

It's been longer than 24 hours, but yes, everything is still
good with the above change.

I would greatly appreciate a backport to fix this in Jessie.
Thanks

Stopping the test now since I've burned enough cycles for
now :)

Re: Bug#808610: libxapian22: xapian database corruption causes recollindex loop

[Olly Betts]

Control: tags -1 -moreinfo
Control: tags -1 +wheezy
Control: severity -1 grave

On Thu, Mar 03, 2016 at 01:52:45AM +0000, Eric Wong wrote:
> > > > On Mon, Dec 28, 2015 at 02:45:43AM +0000, Olly Betts wrote:
> > > > > http://trac.xapian.org/changeset/826d1a19cc356e7bf66c1681626e70af32967447/git
> 
> It's been longer than 24 hours, but yes, everything is still
> good with the above change.
> 
> I would greatly appreciate a backport to fix this in Jessie.

A backport would only fix it for people who take special action and
install from backports (that said, it's probably time to do a backport
so there's a more up to date upstream version easily available for
those who want it).

This database corruption means data loss, so I think this deserves a
stable update with just the patch above.  It doesn't touch many places,
comes with a regression testcase, and has been in use without reported
issues for 10 months.

In https://trac.xapian.org/ticket/645 the recoll upstream maintainer
indicated that this happens with 1.2.8 too, and wheezy has 1.2.12-2
currently, so this issue also affects wheezy.

> Stopping the test now since I've burned enough cycles for
> now :)

Thanks for the testing.

Cheers,
    Olly

Re: Bug#808610: libxapian22: xapian database corruption causes recollindex loop

[Eric Wong]

Olly Betts <olly@survex.com> wrote:
> On Thu, Mar 03, 2016 at 01:52:45AM +0000, Eric Wong wrote:
> > 
> > I would greatly appreciate a backport to fix this in Jessie.
> 
> A backport would only fix it for people who take special action and
> install from backports (that said, it's probably time to do a backport
> so there's a more up to date upstream version easily available for
> those who want it).

Errm, sorry, I meant "backport" as a general term for getting
a patch into jessie for all jessie users.
I forgot the Debian meaning of it is slightly different.

> In https://trac.xapian.org/ticket/645 the recoll upstream
> maintainer indicated that this happens with 1.2.8 too, and
> wheezy has 1.2.12-2 currently, so this issue also affects
> wheezy.

Interesting.  I haven't used Xapian extensively, but have never
seen problems on wheezy using the same hardware that was able
to reproduce the bug on jessie.