[python] segfaults at Message.get_date

[python] segfaults at Message.get_date

[Patrick Totzke]

[-- Attachment #1: Type: text/plain, Size: 1534 bytes --]

Hi all!
First off: Thanks Sebastian for your recent work on the python bindings.
It all makes a little bit more sense now without the __len__ and all.
As some of you might have read on the IRC channel, I'm facing really
strange behaviour with Message.get_date() which unfortunately,
I cannot reproduce reliably: I get it on one machine, not on the other,
despite using the same distribution (ubuntu natty - standard python
install), and notmuch and bindings from master on both boxes:

Unfortunately, as it segfaults, I don't get a stacktrace here.
The line that seems to cause the segfault is this one:
https://github.com/pazz/notmuch-gui/blob/master/alot/db.py#L181

To explain the circumstances in my code:
I crete these alot.Message objects by recursively calling
notmuch.Message.get_replies at line 153 in the same file.
The error occurs only at recursion depth 1, so to reproduce
one needs to open a thread with at least one reply.

Did anyone experience something similar? It would help if someone 
could try and run my code and report if it works for him/her:
 git clone https://github.com/pazz/notmuch-gui 
 python alot/init.py # to run directly 
 open a large enough thread by hitting enter

Just to be clear: I don't expect anybody to debug my broken code.
I think it might help to report segfaults, as it looks like it is 
a problem with the C-lib or the bindings.
Of course, I'd be grateful for any pointers (pun not intended *caught*)
or helpful hints to debug this.

thanks,
/patrick


[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [python] segfaults at Message.get_date

[Sebastian Spaeth]

[-- Attachment #1: Type: text/plain, Size: 601 bytes --]

On Thu, 16 Jun 2011 22:54:39 +0100, Patrick Totzke wrote:
> Unfortunately, as it segfaults, I don't get a stacktrace here.
> The line that seems to cause the segfault is this one:
> https://github.com/pazz/notmuch-gui/blob/master/alot/db.py#L181

Just some followup. get_date() calls libnotmuch's:

notmuch_message_get_date(...)

and expects a "c_long" as result (notmuch.h says it returns time_t, so
long should be appropriate, shouldn't it?)

I had this segfault on me once, but it was not reproducable.

I have no clue what's going on there, especially as it's not reliable reproducable.
Sebastian

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: [python] segfaults at Message.get_date

[Patrick Totzke]

[-- Attachment #1: Type: text/plain, Size: 2975 bytes --]

Thanks to amdragon's hint on how to get a stacktrace, here it is.
I startet gdb --args python, imported my stuff and ran it directly.
I can't really interpret this, but it seems as if its libxapians fault
doesn't it?
best,
/p


(gdb) bt
#0  0x006eb87d in Xapian::Document::Internal::get_value(unsigned int) const () from /usr/lib/sse2/libxapian.so.22
#1  0x006eb952 in Xapian::Document::get_value(unsigned int) const () from /usr/lib/sse2/libxapian.so.22
#2  0x00523963 in notmuch_message_get_date () from /usr/local/lib/libnotmuch.so.1
#3  0x0050c767 in ffi_call_SYSV () from /usr/lib/python2.7/lib-dynload/_ctypes.so
#4  0x0050c4bf in ffi_call () from /usr/lib/python2.7/lib-dynload/_ctypes.so
#5  0x00504e63 in _ctypes_callproc () from /usr/lib/python2.7/lib-dynload/_ctypes.so
#6  0x004fce6e in ?? () from /usr/lib/python2.7/lib-dynload/_ctypes.so
#7  0x080a464a in PyObject_Call ()
#8  0x080dad43 in PyEval_EvalFrameEx ()
#9  0x080dae8b in PyEval_EvalFrameEx ()
#10 0x080e11e7 in PyEval_EvalCodeEx ()
#11 0x08105a61 in ?? ()
#12 0x080a464a in PyObject_Call ()
#13 0x080a603f in ?? ()
#14 0x080a464a in PyObject_Call ()
#15 0x080da034 in PyEval_CallObjectWithKeywords ()
#16 0x080a6fff in PyInstance_New ()
#17 0x080a464a in PyObject_Call ()
#18 0x080dad43 in PyEval_EvalFrameEx ()
#19 0x080dae8b in PyEval_EvalFrameEx ()
#20 0x080dae8b in PyEval_EvalFrameEx ()
#21 0x080dae8b in PyEval_EvalFrameEx ()
#22 0x080dae8b in PyEval_EvalFrameEx ()
#23 0x080e11e7 in PyEval_EvalCodeEx ()
#24 0x08105a61 in ?? ()
#25 0x080a464a in PyObject_Call ()
#26 0x080a603f in ?? ()
#27 0x080a464a in PyObject_Call ()
#28 0x080da034 in PyEval_CallObjectWithKeywords ()
#29 0x080a6fff in PyInstance_New ()
#30 0x080a464a in PyObject_Call ()
#31 0x080dad43 in PyEval_EvalFrameEx ()
#32 0x080dae8b in PyEval_EvalFrameEx ()
#33 0x080dae8b in PyEval_EvalFrameEx ()
#34 0x080dae8b in PyEval_EvalFrameEx ()
#35 0x080dae8b in PyEval_EvalFrameEx ()
#36 0x080dae8b in PyEval_EvalFrameEx ()
#37 0x080dae8b in PyEval_EvalFrameEx ()
#38 0x080e16ee in PyEval_EvalCodeEx ()
#39 0x080dac2a in PyEval_EvalFrameEx ()
#40 0x080dae8b in PyEval_EvalFrameEx ()
#41 0x080dae8b in PyEval_EvalFrameEx ()
#42 0x080dae8b in PyEval_EvalFrameEx ()
#43 0x080e11e7 in PyEval_EvalCodeEx ()
#44 0x080dac2a in PyEval_EvalFrameEx ()
#45 0x080dae8b in PyEval_EvalFrameEx ()
#46 0x080e11e7 in PyEval_EvalCodeEx ()
#47 0x08105a61 in ?? ()
#48 0x080a464a in PyObject_Call ()
#49 0x080a603f in ?? ()
#50 0x080a464a in PyObject_Call ()
#51 0x080da034 in PyEval_CallObjectWithKeywords ()
#52 0x080a6fff in PyInstance_New ()
#53 0x080a464a in PyObject_Call ()
#54 0x080dad43 in PyEval_EvalFrameEx ()
#55 0x080dae8b in PyEval_EvalFrameEx ()
#56 0x080e11e7 in PyEval_EvalCodeEx ()
#57 0x0812c477 in PyEval_EvalCode ()
#58 0x0813c010 in ?? ()
#59 0x08070d09 in PyRun_InteractiveOneFlags ()
#60 0x08070e0b in PyRun_InteractiveLoopFlags ()
#61 0x0807124e in PyRun_AnyFileExFlags ()
#62 0x0805c069 in Py_Main ()
#63 0x0805b25b in main ()


[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [python] segfaults at Message.get_date

[Sebastian Spaeth]

[-- Attachment #1: Type: text/plain, Size: 1046 bytes --]

On Fri, 17 Jun 2011 17:10:24 +0100, Patrick Totzke :
> #0  0x006eb87d in Xapian::Document::Internal::get_value(unsigned int) const () from /usr/lib/sse2/libxapian.so.22
> #1  0x006eb952 in Xapian::Document::get_value(unsigned int) const () from /usr/lib/sse2/libxapian.so.22
> #2  0x00523963 in notmuch_message_get_date () from /usr/local/lib/libnotmuch.so.1

One question, what type is libnotmuch really returning here? The code:


time_t
notmuch_message_get_date (notmuch_message_t *message)
{   ...
    return Xapian::sortable_unserialise (value);
}

But Xapian API says that sortable_unserialise() returns floating type "double"

http://xapian.org/docs/apidoc/html/namespaceXapian.html#326fe2d6b0ee59ac9536f3960e8fd99b
"Convert a string encoded using sortable_serialise back to a floating
point number."

But time_t is usually a (signed) long and not floating point. Obviously
things have worked just fine so far, but is libnotmuch really returning
the right type here? Sorry, I expose my total lack of basic C++ knowledge
here...

Sebastian

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: [python] segfaults at Message.get_date

[Dmitry Kurochkin]

Hi Sebastian, Patrick.

On Sat, 18 Jun 2011 12:30:01 +0200, Sebastian Spaeth <Sebastian@SSpaeth.de> wrote:
> On Fri, 17 Jun 2011 17:10:24 +0100, Patrick Totzke :
> > #0  0x006eb87d in Xapian::Document::Internal::get_value(unsigned int) const () from /usr/lib/sse2/libxapian.so.22
> > #1  0x006eb952 in Xapian::Document::get_value(unsigned int) const () from /usr/lib/sse2/libxapian.so.22
> > #2  0x00523963 in notmuch_message_get_date () from /usr/local/lib/libnotmuch.so.1
> 
> One question, what type is libnotmuch really returning here? The code:
> 
> 
> time_t
> notmuch_message_get_date (notmuch_message_t *message)
> {   ...
>     return Xapian::sortable_unserialise (value);
> }
> 
> But Xapian API says that sortable_unserialise() returns floating type "double"
> 
> http://xapian.org/docs/apidoc/html/namespaceXapian.html#326fe2d6b0ee59ac9536f3960e8fd99b
> "Convert a string encoded using sortable_serialise back to a floating
> point number."
> 
> But time_t is usually a (signed) long and not floating point. Obviously
> things have worked just fine so far, but is libnotmuch really returning
> the right type here? Sorry, I expose my total lack of basic C++ knowledge
> here...
> 

Converting double to time_t does not look good.  Notmuch converts
between time_t and double both when setting and getting the date.  I
guess it should work good in most cases at least.  Perhaps Carl knows
better that it is safe.

This is not relevant to the segfault.  The failure happens in:

  value = message->doc.get_value (NOTMUCH_VALUE_TIMESTAMP);

before conversion from double to time_t.

Sebastian, are you able to reproduce the issue?  A python script that
triggers the bug, perhaps?  I would look at this if I can reproduce the
problem.

Patrick, can you install debugging symbols for libxapian and get the
backtrace again?  On Debian you need to install libxapian22-dbg.

Regards,
  Dmitry

> Sebastian
> _______________________________________________
> notmuch mailing list
> notmuch@notmuchmail.org
> http://notmuchmail.org/mailman/listinfo/notmuch

Re: [python] segfaults at Message.get_date

[Sebastian Spaeth]

[-- Attachment #1: Type: text/plain, Size: 357 bytes --]

On Sun, 19 Jun 2011 13:45:07 +0400, Dmitry Kurochkin wrote:
> Sebastian, are you able to reproduce the issue?  A python script that
> triggers the bug, perhaps?  I would look at this if I can reproduce the
> problem.

No, not reproducable at all. I only had it twice actually. Patrick had it
several times but he can't reliably reproduce either.

Sebastian

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: [python] segfaults at Message.get_date

[Austin Clements]

On Sun, Jun 19, 2011 at 5:45 AM, Dmitry Kurochkin
<dmitry.kurochkin@gmail.com> wrote:
> Hi Sebastian, Patrick.
>
> On Sat, 18 Jun 2011 12:30:01 +0200, Sebastian Spaeth <Sebastian@SSpaeth.de> wrote:
>> On Fri, 17 Jun 2011 17:10:24 +0100, Patrick Totzke :
>> > #0  0x006eb87d in Xapian::Document::Internal::get_value(unsigned int) const () from /usr/lib/sse2/libxapian.so.22
>> > #1  0x006eb952 in Xapian::Document::get_value(unsigned int) const () from /usr/lib/sse2/libxapian.so.22
>> > #2  0x00523963 in notmuch_message_get_date () from /usr/local/lib/libnotmuch.so.1
>>
>> One question, what type is libnotmuch really returning here? The code:
>>
>>
>> time_t
>> notmuch_message_get_date (notmuch_message_t *message)
>> {   ...
>>     return Xapian::sortable_unserialise (value);
>> }
>>
>> But Xapian API says that sortable_unserialise() returns floating type "double"
>>
>> http://xapian.org/docs/apidoc/html/namespaceXapian.html#326fe2d6b0ee59ac9536f3960e8fd99b
>> "Convert a string encoded using sortable_serialise back to a floating
>> point number."
>>
>> But time_t is usually a (signed) long and not floating point. Obviously
>> things have worked just fine so far, but is libnotmuch really returning
>> the right type here? Sorry, I expose my total lack of basic C++ knowledge
>> here...
>>
>
> Converting double to time_t does not look good.  Notmuch converts
> between time_t and double both when setting and getting the date.  I
> guess it should work good in most cases at least.  Perhaps Carl knows
> better that it is safe.

A double will precisely represent integers up to 2^53, so this
conversion shouldn't be a problem until the year 285422109 or so.

Re: [python] segfaults at Message.get_date

[Sebastian Spaeth]

[-- Attachment #1: Type: text/plain, Size: 561 bytes --]

On Sun, 19 Jun 2011 19:51:11 -0400, Austin Clements <amdragon@mit.edu> wrote:
> A double will precisely represent integers up to 2^53, so this
> conversion shouldn't be a problem until the year 285422109 or so.

But given that it works, is it actually necessary, that xapian
apparently pulls an int from the database, returns a std::string to
libnotmuch, which calls a helper function to unserialize it to a double
and casts it to time_t before handing it to the user how probably uses
it as a long?

Can't we easily put in longs and get longs back?

Sebastian

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: [python] segfaults at Message.get_date

[Austin Clements]

Quoth Sebastian Spaeth on Jun 20 at  9:29 am:
> On Sun, 19 Jun 2011 19:51:11 -0400, Austin Clements <amdragon@mit.edu> wrote:
> > A double will precisely represent integers up to 2^53, so this
> > conversion shouldn't be a problem until the year 285422109 or so.
> 
> But given that it works, is it actually necessary, that xapian
> apparently pulls an int from the database, returns a std::string to
> libnotmuch, which calls a helper function to unserialize it to a double
> and casts it to time_t before handing it to the user how probably uses
> it as a long?
> 
> Can't we easily put in longs and get longs back?

Xapian only knows about strings, so there has to be serialization
somewhere and the serialized form has to have the same collation order
as the numerical order of the original numbers.  You could do this by
storing the bytes of the long in big-endian form, but that's basically
what sortable_serialise does: roughly, the first byte stores the
number of bits needed to represent the number, followed by enough
bytes to hold those bits in big-endian.  Since POSIX permits a wide
range of types to implement time_t, sortable_serialise also has the
major advantage that it can handle any of them.  So, taking
portability and time_t as assumptions, there are no unnecessary
conversion steps here (and, really, it's just string->double->time_t
on Linux and just string->time_t on a platform that uses floating
point time_t's).

Alternatively, notmuch could switch to using long instead of time_t
for dates, but that seems like a lot of effort for little gain and
results in portability friction whenever notmuch wants to use the
standard C time API's.