Notmuch success: Xapian database corrupt

Notmuch success: Xapian database corrupt

[John Fremlin]

First off, thanks for making notmuch, it's a really good idea and it
works generally very well.

Running notmuch new, it processes mails nicely for a while 

Processed 58 files (19 files/sec.)

then after crunching through not many emails

terminate called after throwing an instance of 'Xapian::DatabaseCorruptError'
Aborted (core dumped)

Is there any way to recover the database? Notmuch search works well and
it takes absolutely ages (one or two days) to add my mail to it; and I
would suspect that it might happen again . . . I'm on Ubuntu lucid with
an Intel SSD.

Re: Notmuch success: Xapian database corrupt

[Sebastian Spaeth]

On 2010-04-18, John Fremlin wrote:
> Processed 58 files (19 files/sec.)

That seems exceptionally low. I get about 60-70 files/sec on a laptop
hard disk.
> 
> Is there any way to recover the database?

I am no expert with xapian databases, and that might seem obvious, but
you did a notmuch dump to save your tags, did you? If that works you can
nuke your database directory and after a notmuch new (1-2 days?!!) you
can notmuch restore your tags. This way, at least, you won't have any
data loss.

Sebastian

Re: Notmuch success: Xapian database corrupt

[Carl Worth]

[-- Attachment #1: Type: text/plain, Size: 1455 bytes --]

On Sun, 18 Apr 2010 14:18:09 +0000, John Fremlin <john@fremlin.org> wrote:
> terminate called after throwing an instance of 'Xapian::DatabaseCorruptError'
> Aborted (core dumped)
> 
> Is there any way to recover the database? Notmuch search works well and
> it takes absolutely ages (one or two days) to add my mail to it; and I
> would suspect that it might happen again . . . I'm on Ubuntu lucid with
> an Intel SSD.

Hi John,

Welcome to notmuch, and I'm so sorry to hear that your initial attempt
to use it was so frustrating.

I'm not aware of any bugs in notmuch that can result in a corrupt Xapian
database. In fact, this can't be a bug in notmuch alone (since Xapian is
detecting the corruption). There must at least be a bug in Xapian or
else some lower-level failure is occurring (disk full?) that Xapian
can't deal with.

I've not yet encountered a corrupt Xapian database, so I'm afraid I
don't have any tips to help you with that.

But I'm also surprised to hear that it takes you days to incorporate
your mail into a notmuch database. I have over 600 thousand messages
myself, and it takes a few hours (maybe 4?) to incorporate all of these
messages, but not days, (also with an Intel SSD).

So there's some performance problem that you're having in addition to
the database corruption. Hopefully we can figure that out. What kernel
and filesystem are you using? Are you using an encrypted partition?

-Carl

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: Notmuch success: Xapian database corrupt

[Ben Gamari]

On Wed, 21 Apr 2010 17:17:16 -0700, Carl Worth <cworth@cworth.org> wrote:
>
> I'm not aware of any bugs in notmuch that can result in a corrupt Xapian
> database. In fact, this can't be a bug in notmuch alone (since Xapian is
> detecting the corruption). There must at least be a bug in Xapian or
> else some lower-level failure is occurring (disk full?) that Xapian
> can't deal with.
> 
> I've not yet encountered a corrupt Xapian database, so I'm afraid I
> don't have any tips to help you with that.
> 
Nor have I experienced any corruption issues. I'd say just hope that it
was an isolated incident.

> But I'm also surprised to hear that it takes you days to incorporate
> your mail into a notmuch database. I have over 600 thousand messages
> myself, and it takes a few hours (maybe 4?) to incorporate all of these
> messages, but not days, (also with an Intel SSD).
> 
6e5 messages / 4 hours = ~40 messages/s. I don't believe I have ever
seen more than 0 messages per second average on my box (granted, with a
spinning disk, but I'm generally getting 0.05 messages/second or so), so
you are not the only one experiencing such abysmal performance. I sent a
message[1] to the list about this a few weeks ago, and Olly and others
had some productive input, but nothing that seemed too promising as far
as fixing the issue. I then took the issue to the LKML[2], although this
hasn't resulted in much progress. I recently switched from ext4 to btrfs
and both are quite poor when it comes to notmuch performance, so I'm
honestly not entirely convinced the problem can be placed exclusively on
the file system.

I know that the disk is capable of 20MByte/second sustained (peak of
60MByte/second), however I'm lucky to see a throughput of several
hundred kByte/second under the workload presented by notmuch.  I have
plenty of perf/blktrace data of notmuch new sessions if anyone is
interested, but there were unfortunately no takers on the lkml.

I am under the impression that Xapian is doing some really
knuckle-headed things when it comes to fsync()ing and the like, but I
really have a difficult time believing that is the sole issue while
others are getting perfectly acceptable performance with spinning disks.

I would love to get this issue solved, but my experience is definitely
quite limited in the file system/block I/O department and the semester
is definitely severely limiting the amount of time I am able to invest
in the problem, so I find myself pretty much at the mercy of whoever has
time to parse the data. If you are that person, I would be elated to
provide you with whatever data you might want/need

- Ben


[1] id:20100315090401.GA29891@glaive.weftsoar.net
[2] id:4b9fa440.12135e0a.7fc8.ffffe745@mx.google.com

Re: Notmuch success: Xapian database corrupt

[John Fremlin]

After the encouraging message from Sebastian. I deleted the
.notmuch/xapian dir and started again.

It went off a good rate (300+ files/sec) and here was the final score

Processed 494764 total files in 2h 54m 41s (47 files/sec.).                 
Added 226817 new messages to the database.

This is much faster than before. As I haven't changed the storage or the
filesystem (ext4,data=ordered over encrypted aes-xts-plain), I just
don't know what made the difference. My kernel is now 2.6.32-21-generic
#32-Ubuntu and I had an older one the first try a month or so ago.

Carl Worth <cworth@cworth.org> writes:
[...]
> Welcome to notmuch, and I'm so sorry to hear that your initial attempt
> to use it was so frustrating.

Thanks for the welcome! I was initially impressed by it but rather
worried about relying on it after the database corruption.

> I'm not aware of any bugs in notmuch that can result in a corrupt Xapian
> database. In fact, this can't be a bug in notmuch alone (since Xapian is
> detecting the corruption). There must at least be a bug in Xapian or
> else some lower-level failure is occurring (disk full?) that Xapian
> can't deal with.

Disk full is quite likely. I'll try to avoid that in future.

[...]
> So there's some performance problem that you're having in addition to
> the database corruption. Hopefully we can figure that out. What kernel
> and filesystem are you using? Are you using an encrypted partition?

Happy to say (though frustrating for you), this time it's much
faster. Maybe because I had more disk free this time round so the Xapian
database became less fragmented? (Speculation, no evidence.) 

Re: Notmuch success: Xapian database corrupt

[Carl Worth]

[-- Attachment #1: Type: text/plain, Size: 870 bytes --]

On Thu, 22 Apr 2010 07:19:58 +0000, John Fremlin <john@fremlin.org> wrote:
> After the encouraging message from Sebastian. I deleted the
> .notmuch/xapian dir and started again.
> 
> It went off a good rate (300+ files/sec) and here was the final score
> 
> Processed 494764 total files in 2h 54m 41s (47 files/sec.).                 
> Added 226817 new messages to the database.

Nice. That's much more like what I'm accustomed to getting.

> This is much faster than before. As I haven't changed the storage or the
> filesystem (ext4,data=ordered over encrypted aes-xts-plain), I just
> don't know what made the difference. My kernel is now 2.6.32-21-generic
> #32-Ubuntu and I had an older one the first try a month or so ago.

Thanks for the details at least. Maybe other people having performance
problems can start finding correlations.

-Carl

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]