Backport of Xapian term update optimisation

Backport of Xapian term update optimisation

[Olly Betts]

I've backported the term update optimisation patches
<http://trac.xapian.org/ticket/250> to Xapian's 1.0 branch, and you can
find snapshot tarballs including these changes here:

http://oligarchy.co.uk/xapian/branches/1.0/

Xapian's testsuite passes (including the additional test coverage which I
also backported), and I looked over each change carefully, but I would be
interested to see some real world testing, particularly in the situation
which these changes are intended to improve (i.e. speed of tagging in
notmuch).

So if you're so inclined, try it and report how you got on (on this list
is fine).

Cheers,
    Olly

Re: Backport of Xapian term update optimisation

[Jameson Rollins]

[-- Attachment #1: Type: text/plain, Size: 1422 bytes --]

On Thu, 28 Jan 2010 00:06:59 +0000 (UTC), Olly Betts <olly@survex.com> wrote:
> I've backported the term update optimisation patches
> <http://trac.xapian.org/ticket/250> to Xapian's 1.0 branch, and you can
> find snapshot tarballs including these changes here:
> 
> http://oligarchy.co.uk/xapian/branches/1.0/
> 
> Xapian's testsuite passes (including the additional test coverage which I
> also backported), and I looked over each change carefully, but I would be
> interested to see some real world testing, particularly in the situation
> which these changes are intended to improve (i.e. speed of tagging in
> notmuch).

Hey, Olly.  Thanks so much for backporting this patch and uploading a
patched package to Debian experimental (which is now available):

servo:~ 0$ apt-cache policy libxapian-dev
libxapian-dev:
  Installed: 1.0.17+svn13879-1
  Candidate: 1.0.17+svn13879-1
  Version table:
 *** 1.0.17+svn13879-1 0
          1 http://ftp.us.debian.org experimental/main Packages
        100 /var/lib/dpkg/status
     1.0.17-1 0
        500 http://debian.lcs.mit.edu testing/main Packages
        200 http://ftp.us.debian.org unstable/main Packages
servo:~ 0$ 

I just installed this new version from a Debian experimental repo,
rebuilt notmuch against the new installation, and everything seems to be
working great.  I'll report back any issues to the BTS.  Thanks again.

jamie.

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: Backport of Xapian term update optimisation

[Olly Betts]

On Wed, Feb 03, 2010 at 02:35:14PM -0500, Jameson Rollins wrote:
> On Thu, 28 Jan 2010 00:06:59 +0000 (UTC), Olly Betts <olly@survex.com> wrote:
> > I've backported the term update optimisation patches
> > <http://trac.xapian.org/ticket/250> to Xapian's 1.0 branch, and you can
> > find snapshot tarballs including these changes here:
> > 
> > http://oligarchy.co.uk/xapian/branches/1.0/
> > 
> > Xapian's testsuite passes (including the additional test coverage which I
> > also backported), and I looked over each change carefully, but I would be
> > interested to see some real world testing, particularly in the situation
> > which these changes are intended to improve (i.e. speed of tagging in
> > notmuch).
> 
> Hey, Olly.  Thanks so much for backporting this patch and uploading a
> patched package to Debian experimental (which is now available):

It hasn't built for all Debian architectures yet, but is available for at
least amd64 and x86, which are probably the most popular two.

If you aren't sure how to pull in packages from experimental, see:

http://wiki.debian.org/DebianExperimental

I've also put it in a Launchpad PPA for all currently supported Ubuntu
releases, which has built for all of them already:

https://launchpad.net/~ojwb/+archive/experimental/

> I just installed this new version from a Debian experimental repo,
> rebuilt notmuch against the new installation, and everything seems to be
> working great.  I'll report back any issues to the BTS.  Thanks again.

Thanks.  Are you seeing the expected speed improvement?

Cheers,
    Olly

Re: Backport of Xapian term update optimisation

[Jameson Rollins]

[-- Attachment #1: Type: text/plain, Size: 362 bytes --]

On Thu, 4 Feb 2010 02:45:24 +0000, Olly Betts <olly@survex.com> wrote:
> Thanks.  Are you seeing the expected speed improvement?

Yes, absolutely.  Everything is working great.  I'm now using a notmuch
built against the xapian libraries from the debian experimental package,
instead of the one I built from upstream, and everything is working very
fast.

jamie.

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: Backport of Xapian term update optimisation

[micah anderson]

[-- Attachment #1: Type: text/plain, Size: 900 bytes --]

On Thu, 4 Feb 2010 02:45:24 +0000, Olly Betts <olly@survex.com> wrote:
> On Wed, Feb 03, 2010 at 02:35:14PM -0500, Jameson Rollins wrote:
> > On Thu, 28 Jan 2010 00:06:59 +0000 (UTC), Olly Betts <olly@survex.com> wrote:
> >
> > I just installed this new version from a Debian experimental repo,
> > rebuilt notmuch against the new installation, and everything seems to be
> > working great.  I'll report back any issues to the BTS.  Thanks again.
> 
> Thanks.  Are you seeing the expected speed improvement?

Once this is available in unstable, the notmuch package should be
re-uploaded with a build-dependency on this version so that the package
users see the speed improvement. As it is now, the debian package is
pretty slow.

What is the expected time-frame for moving this out of an experimental
stage into unstable? Presumably this is a more proper release by Xapian?

micah

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: Backport of Xapian term update optimisation

[Olly Betts]

On Thu, Feb 04, 2010 at 11:55:44AM -0500, micah anderson wrote:
> Once this is available in unstable, the notmuch package should be
> re-uploaded with a build-dependency on this version so that the package
> users see the speed improvement. As it is now, the debian package is
> pretty slow.

There's no need to rebuild notmuch to benefit from this improvement.  Just
install the updated libxapian15 package.

> What is the expected time-frame for moving this out of an experimental
> stage into unstable?

Probably a week or two.  It would be good to get this into Ubuntu Lucid (though oddly notmuch doesn't seem to be there yet) which means pretty soon.

If I get a lot of positive feedback on the snapshot version, that'll encourage
me to push ahead with it.  I've also announced it on the sup mailing list (as
they have similar issues with adding tags) and will on the Xapian list later
today.

> Presumably this is a more proper release by Xapian?

I would indeed prefer to put it in a new Xapian release and then package
that - if the code isn't "good enough" for an upstream release, then I'd
find it hard to justify it as "good enough" for Debian.

Cheers,
    Olly

Re: Backport of Xapian term update optimisation

[David Edmondson]

On Thu, 4 Feb 2010 23:23:56 +0000, Olly Betts <olly@survex.com> wrote:
> If I get a lot of positive feedback on the snapshot version, that'll encourage
> me to push ahead with it.  I've also announced it on the sup mailing list (as
> they have similar issues with adding tags) and will on the Xapian list later
> today.

I'm using the version from the experimental repository and it's working
well for me so far.

dme.
-- 
David Edmondson, http://dme.org

Re: Backport of Xapian term update optimisation

[Olly Betts]

Thanks for all the testing feedback folks.

I've made a new Xapian release (1.0.18) and packaged it for Debian.  If you
are using unstable, and not on mips or hurd, it's now built:

https://buildd.debian.org/status/package.php?p=xapian-core

I haven't updated the xapian.org download page, etc yet - I wanted to get the
Debian package uploaded promptly so there's more chance it'll get into the
next Ubuntu release - but you can find tarballs in the usual place:

http://oligarchy.co.uk/xapian/1.0.18/

Cheers,
    Olly