Xapian::DatabaseError on notmuch new upgrade

Xapian::DatabaseError on notmuch new upgrade

[Jed Brown]

I rebuilt notmuch and got a Xapian exception while notmuch new was
upgrading my database.  This seemed like a decent time to try the latest
Xapian, so I built a copy and the crash remained (and maintainer-mode
still optimizes so I built Xapian again so I could get decent
debugging).  Here is a trace from

  XAPIAN_PREFER_CHERT=1 notmuch new

I got the same error without setting XAPIAN_PREFER_CHERT.  It's not
clear to me whether this is a usage issue or Xapian problem.  The core
is 245 MB, I can provide it if someone wants to try debugging.

Jed

(gdb) bt
#0  0x00007fa687d9e035 in raise () from /lib/libc.so.6
#1  0x00007fa687d9f460 in abort () from /lib/libc.so.6
#2  0x00007fa688622925 in __gnu_cxx::__verbose_terminate_handler() () from /usr/lib/libstdc++.so.6
#3  0x00007fa688620d56 in __cxxabiv1::__terminate(void (*)()) () from /usr/lib/libstdc++.so.6
#4  0x00007fa688620d83 in std::terminate() () from /usr/lib/libstdc++.so.6
#5  0x00007fa688620e7e in __cxa_throw () from /usr/lib/libstdc++.so.6
#6  0x00007fa688a221bc in FlintTable::read_block (this=0x16395c8, n=2569, p=0xe3fd8e0 "") at backends/flint/flint_table.cc:243
#7  0x00007fa688a22bda in FlintTable::block_to_cursor (this=0x16395c8, C_=0xe4074e0, j=0, n=2569) at backends/flint/flint_table.cc:393
#8  0x00007fa688a299fc in FlintTable::next_default (this=0x16395c8, C_=0xe4074e0, j=1) at backends/flint/flint_table.cc:2219
#9  0x00007fa688a29983 in FlintTable::next_default (this=0x16395c8, C_=0xe4074e0, j=0) at backends/flint/flint_table.cc:2214
#10 0x00007fa688a022c5 in FlintTable::next (this=0x16395c8, C_=0xe4074e0, j=0) at backends/flint/flint_table.h:692
#11 0x00007fa688a020d5 in FlintCursor::read_tag (this=0xe43ece0, keep_compressed=false) at backends/flint/flint_cursor.cc:264
#12 0x00007fa688a1a6d8 in FlintPostListTable::get_chunk (this=0x16395c8, tname=..., did=82407, adding=false, from=0x7fff849be358, to=0x7fff849be350) at backends/flint/flint_postlist.cc:946
#13 0x00007fa688a1b5e3 in FlintPostListTable::merge_changes (this=0x16395c8, mod_plists=..., doclens=..., freq_deltas=...) at backends/flint/flint_postlist.cc:1100
#14 0x00007fa688a08d89 in FlintWritableDatabase::flush_postlist_changes (this=0x1639590) at backends/flint/flint_database.cc:1053
#15 0x00007fa688a0bb5f in FlintWritableDatabase::replace_document (this=0x1639590, did=45341, document=...) at backends/flint/flint_database.cc:1425
#16 0x00007fa688975b7d in Xapian::WritableDatabase::replace_document (this=0x16393a0, did=45341, document=...) at api/omdatabase.cc:817
#17 0x0000000000416c6f in _notmuch_message_sync (message=0xe327130) at lib/message.cc:609
#18 0x0000000000410ca5 in notmuch_database_upgrade (notmuch=0x1639460, progress_notify=0x40a890 <upgrade_print_progress>, closure=0x7fff849bef20) at lib/database.cc:751
#19 0x000000000040ad4a in notmuch_new_command (ctx=0x162b120, argc=0, argv=0x7fff849bf0f8) at notmuch-new.c:746
#20 0x000000000040862e in main (argc=2, argv=0x7fff849bf0e8) at notmuch.c:449
(gdb) f 17
#17 0x0000000000416c6f in _notmuch_message_sync (message=0xe327130) at lib/message.cc:609
609         db->replace_document (message->doc_id, message->doc);
Current language:  auto
The current source language is "auto; currently c++".
(gdb) p *message
$1 = {notmuch = 0x1639460, doc_id = 45341, frozen = 0, message_id = 0x0, thread_id = 0x0, in_reply_to = 0x0, filename = 0x0, message_file = 0x0, replies = 0xbaea960, flags = 0, doc = {internal = {dest = 0xe3816c0}}}

Re: Xapian::DatabaseError on notmuch new upgrade

[Jed Brown]

Looks like this was just a bug in Xapian HEAD, today's build works just
fine and I'm enjoying the faster CHERT backend.  Bulk tagging is really
loads faster now, and the interactivity is better than before, but SPC
still has a noticeable pause (at least 200 ms) since (apparently) it's
doing an fsync, and this is worse when the disk is getting hit by some
other job (like when I'm reading email while something big compiles).
It wouldn't bother me at all if I lost my last few seconds of
interactive tagging due to something catastrophic like losing power.  I
think there is still (post #250) a case for supporting some asynchronous
operations.

Jed

asynch operations protocol

[David Bremner]

On Fri, 15 Jan 2010 14:49:14 +0100, Jed Brown <jed@59A2.org> wrote:
> It wouldn't bother me at all if I lost my last few seconds of
> interactive tagging due to something catastrophic like losing power.  I
> think there is still (post #250) a case for supporting some asynchronous
> operations.

I was wondering what the protocol would be like for non-blocking
commands to a notmuch daemon. I have no experience with these things,
but I was thinking something along the lines of (not worrying about wire
encoding)

open_session  - basically just generates a unique id to allow grabbing
              results of commands to be collected.

queue command session arguments
     
        I guess the command dispatcher could just be a thin replacement for
        command-line argument parsing.

gather session
       return all output from session 

flush session

close session

Is this over/under engineered?  I spent roughly as long on the design as
it took me to type :). Maybe the whole session id thing is redundant and
could be done at the socket level. Or, getting more serious about the
whole thing, maybe each queue operation should return an identifier.

OK, discuss amongst yourselves ;)

d

Re: asynch operations protocol

[Jed Brown]

On Fri, 15 Jan 2010 09:59:54 -0400, David Bremner <bremner@unb.ca> wrote:
> Is this over/under engineered?  I spent roughly as long on the design as
> it took me to type :). Maybe the whole session id thing is redundant and
> could be done at the socket level. Or, getting more serious about the
> whole thing, maybe each queue operation should return an identifier.

The asynchronous interface I work with most is MPI.  There you get a
Request object when the operation is initiated and you can
{test,block}{one,some,any,all}, where the latter takes a list of
requests.  These variants are all useful, but of course they could be
implemented as needed.  I don't think that being able to support these
variants places any particular burden on the design.

I believe in performing operations with appropriate granularity, so I
wouldn't expect cases where you need to manage thousands of active
requests, thus I'm not sure the "session" grouping offers any real
benefit.  In any case, I'm not in favor of a single global flush.

Jed