notmuch python bindings corrupt db index (was: gmail importer script)

notmuch python bindings corrupt db index (was: gmail importer script)

[Jason A. Donenfeld]

On Tue, Dec 11, 2012 at 11:04 AM, Patrick Totzke
<patricktotzke@gmail.com> wrote:
> I ended up killing the process after i saw that ETA was >3h.

Gmail's bottleneck. Very unfortunate. Leave it going over night.

> This broke my index!
>
> afterwards, a `notmuch new` printed
>
> A Xapian exception occurred creating a directory: Expected block
> 485 to be level 1, not 0.
>
> I wasnt able to properly read the index anymore and had to
> restore an old index dump. not cool!


Can you provide any more information about this? The script properly
uses atomic transactions. Any corruption here is the result of a
notmuch library bug, which should be investigated.

Re: notmuch python bindings corrupt db index (was: gmail importer script)

[Patrick Totzke]

[-- Attachment #1: Type: text/plain, Size: 1538 bytes --]

Quoting Jason A. Donenfeld (2012-12-11 13:47:32)
> On Tue, Dec 11, 2012 at 11:04 AM, Patrick Totzke
> <patricktotzke@gmail.com> wrote:
> > I ended up killing the process after i saw that ETA was >3h.
> 
> Gmail's bottleneck. Very unfortunate. Leave it going over night.

OK, i assumed i'd not have to download all messages because i already have them
on disk -- synced via offlineimap. but then i remembered that OI stores uses
its own naming scheme for the files in a maildir. so thats probably why your script
did not realize theyre already there. (maybe it should? it could look up the MIDs from notmuch)

> > This broke my index!
> >
> > afterwards, a `notmuch new` printed
> >
> > A Xapian exception occurred creating a directory: Expected block
> > 485 to be level 1, not 0.
> >
> > I wasnt able to properly read the index anymore and had to
> > restore an old index dump. not cool!
> 
> 
> Can you provide any more information about this? The script properly
> uses atomic transactions. Any corruption here is the result of a
> notmuch library bug, which should be investigated.

i am kind of reluctant to reproduce this again (on my work-desktop).
All i did was start your script, writing to the index in another terminal in between (using alot)
possibly my cronjob fired a "notmuch new; afew.." in between also.
when I killed your script the index war broken.

Can i turn on some global logging for notmuch to debug this?
Maybe libnotmuch had trouble with so many db-writes at a time?

/p



[-- Attachment #2: signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)

iEYEABECAAYFAlDHet4ACgkQlDQDZ9fWxap4BgCdFxBrgs+jIadzi/xvsAVB9mnY
s2UAoJZXSab5ftLEa1+d7ORHFiHsJy62
=WT0i
-----END PGP SIGNATURE-----

Re: notmuch python bindings corrupt db index (was: gmail importer script)

[Jason A. Donenfeld]

On Tue, Dec 11, 2012 at 7:26 PM, Patrick Totzke <patricktotzke@gmail.com> wrote:
> OK, i assumed i'd not have to download all messages because i already have them
> on disk -- synced via offlineimap. but then i remembered that OI stores uses
> its own naming scheme for the files in a maildir. so thats probably why your script
> did not realize theyre already there. (maybe it should? it could look up the MIDs from notmuch)

Gmail bandwidth restrictions make this unfeasible. Requesting mail
header IDs via IMAP will rate limit the listing to 2 mails per second.
The best gmail will allow us to do at quasi rapid speed (~180kB/s) is
ask for the IMAP sequence number and the unique gmail 64bit number.
So, the maildir scheme winds up being "${gmail-unique-number}.gmail",
and the script doesn't redownload based on the preexistence of a file
with the corresponding name.

> i am kind of reluctant to reproduce this again (on my work-desktop).

It'd really be a huge help if you could backup and try...

> All i did was start your script, writing to the index in another terminal in between (using alot)
> possibly my cronjob fired a "notmuch new; afew.." in between also.
> when I killed your script the index war broken.
>
> Can i turn on some global logging for notmuch to debug this?
> Maybe libnotmuch had trouble with so many db-writes at a time?

That is very wrong and bizarre. I open the database in read/write
mode, which is supposed to lock other programs from accessing it in
this way... Very strange.

Re: notmuch python bindings corrupt db index (was: gmail importer script)

[Austin Clements]

Quoth Jason A. Donenfeld on Dec 12 at  7:35 pm:
> On Tue, Dec 11, 2012 at 7:26 PM, Patrick Totzke <patricktotzke@gmail.com> wrote:
> > i am kind of reluctant to reproduce this again (on my work-desktop).
> 
> It'd really be a huge help if you could backup and try...
> 
> > All i did was start your script, writing to the index in another terminal in between (using alot)
> > possibly my cronjob fired a "notmuch new; afew.." in between also.
> > when I killed your script the index war broken.
> >
> > Can i turn on some global logging for notmuch to debug this?
> > Maybe libnotmuch had trouble with so many db-writes at a time?
> 
> That is very wrong and bizarre. I open the database in read/write
> mode, which is supposed to lock other programs from accessing it in
> this way... Very strange.

There should be no way to corrupt the database at this level through
the Xapian API, which means nothing libnotmuch can do (much less users
of libnotmuch) should be able to corrupt the database.  If you can
reproduce the problem, it's probably a serious bug in Xapian, but it
could also have been a file system bug or even random file system
corruption.

Re: notmuch python bindings corrupt db index (was: gmail importer script)

[David Bremner]

Austin Clements <amdragon@MIT.EDU> writes:

>
> There should be no way to corrupt the database at this level through
> the Xapian API, which means nothing libnotmuch can do (much less users
> of libnotmuch) should be able to corrupt the database.  If you can
> reproduce the problem, it's probably a serious bug in Xapian, but it
> could also have been a file system bug or even random file system
> corruption.

One thing that Olly Betts mentioned is that there is a recently fixed
bug in xapian with respect to multiple threads in the same process

http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=695542

I don't _think_ this is relevant here, but I mention it for
completeness.

d

Re: notmuch python bindings corrupt db index (was: gmail importer script)

[Jason A. Donenfeld]

On Wed, Dec 12, 2012 at 10:26 PM, David Bremner <david@tethera.net> wrote:
>
> One thing that Olly Betts mentioned is that there is a recently fixed
> bug in xapian with respect to multiple threads in the same process

The script's actually single threaded.

Re: notmuch python bindings corrupt db index (was: gmail importer script)

[Jason A. Donenfeld]

On Wed, Dec 12, 2012 at 9:49 PM, Austin Clements <amdragon@mit.edu> wrote:
> There should be no way to corrupt the database at this level through
> the Xapian API, which means nothing libnotmuch can do (much less users
> of libnotmuch) should be able to corrupt the database.  If you can
> reproduce the problem, it's probably a serious bug in Xapian, but it
> could also have been a file system bug or even random file system
> corruption.

Well that's... troubling.

Patrick: could you please backup and try to reproduce? Otherwise I'll
assume this was a one-off situation.


Austin-- think you could do a quick review of the script to double
check and confirm I'm not doing anything nefarious?
http://git.zx2c4.com/gmail-notmuch/tree/gmail-notmuch.py

Re: notmuch python bindings corrupt db index (was: gmail importer script)

[Patrick Totzke]

Quoting Jason A. Donenfeld (2012-12-13 14:32:53)
> On Wed, Dec 12, 2012 at 9:49 PM, Austin Clements <amdragon@mit.edu> wrote:
> > There should be no way to corrupt the database at this level through
> > the Xapian API, which means nothing libnotmuch can do (much less users
> > of libnotmuch) should be able to corrupt the database.  If you can
> > reproduce the problem, it's probably a serious bug in Xapian, but it
> > could also have been a file system bug or even random file system
> > corruption.
> 
> Well that's... troubling.
> 
> Patrick: could you please backup and try to reproduce? Otherwise I'll
> assume this was a one-off situation.


I tried it again, concurrently triggered some modifications
and killed the process afer a while.
(my modifications were not flushed during the run of your script as
the index was locked)
The index seems to be OK. So I'm afraid I cannot reproduce this reliably.
/p

Re: notmuch python bindings corrupt db index (was: gmail importer script)

[Austin Clements]

Quoth Jason A. Donenfeld on Dec 13 at  3:32 pm:
> On Wed, Dec 12, 2012 at 9:49 PM, Austin Clements <amdragon@mit.edu> wrote:
> > There should be no way to corrupt the database at this level through
> > the Xapian API, which means nothing libnotmuch can do (much less users
> > of libnotmuch) should be able to corrupt the database.  If you can
> > reproduce the problem, it's probably a serious bug in Xapian, but it
> > could also have been a file system bug or even random file system
> > corruption.
> 
> Well that's... troubling.
> 
> Patrick: could you please backup and try to reproduce? Otherwise I'll
> assume this was a one-off situation.
> 
> 
> Austin-- think you could do a quick review of the script to double
> check and confirm I'm not doing anything nefarious?
> http://git.zx2c4.com/gmail-notmuch/tree/gmail-notmuch.py

In theory the only way you could cause corruption besides tickling a
bug would be to access the same database object concurrently from
different threads (since it's not thread-safe), but you don't appear
to be doing that.

I did spot something that could corrupt delivered email, though.  The
way you deliver to the Maildir is resilient to process termination,
but not to system failures such as power outages.  In particular, you
need to at least os.fsync before the os.link.  I'd recommend looking
at Python's mailbox module, which has a robust Maildir delivery
implementation (though it appears it doesn't let you control the file
name, so you probably can't use it directly).

Re: notmuch python bindings corrupt db index (was: gmail importer script)

[Jason A. Donenfeld]

[-- Attachment #1: Type: text/plain, Size: 248 bytes --]

On Sat, Dec 15, 2012 at 7:18 AM, Austin Clements <amdragon@mit.edu> wrote:
>
> need to at least os.fsync before the os.link.


Fixed, thanks for the suggestion.
http://git.zx2c4.com/gmail-notmuch/commit/?id=3f9646058bfd91d7d0e2eda035521f97de92eabc

[-- Attachment #2: Type: text/html, Size: 623 bytes --]