Python binding SIGABRT/SIGSEGV

Python binding SIGABRT/SIGSEGV

[Austin Lund]

I'm clearly doing this python code wrong by not using the iterator correctly:

> import notmuch2
> 
> d = notmuch2.Database()
> m = list(d.messages("since:today"))
> p = m[0].path
> print(p)

But I seem to be getting a SIGABRT instead of a python stack trace.  Is
this the expected behaviour?

Re: Python binding SIGABRT/SIGSEGV

[Michael J Gruber]

Austin Lund venit, vidit, dixit 2022-02-10 06:56:12:
> I'm clearly doing this python code wrong by not using the iterator correctly:
> 
> > import notmuch2
> > 
> > d = notmuch2.Database()
> > m = list(d.messages("since:today"))
> > p = m[0].path
> > print(p)
> 
> But I seem to be getting a SIGABRT instead of a python stack trace.  Is
> this the expected behaviour?

You didn't expect it :)

And this can be confusing. d.messages() returns an iterator through
Message objects whose lifetime depends on the iterator. In contrast,
thread.get_messages() returns on iterator through OwnedMessage objects
whose lifetime depends on the thread.

As soon as the iterator is depleted, the returned objects are (possibly)
gone. (Well, because it's return by reference in Python, and ...)

If you're interested in m[0] only you can "cheat" by not depleting the
iterator:

mm = next(d.messages("since:today"))

p = mm.path

This never frees the object (I think).

My attempts with notmuch2._message.OwnedMessage (and db as parent)
failed. There must be a better way, possibly using a context manager or
search.

I guess usually people just use the iterator in a for loop and do
something with the message inside the loop (while the iterator is not
depleted), such as converting it into a proper email.Message object
(i.e. instantiating a new object from it).

Michael

Re: Python binding SIGABRT/SIGSEGV

[Floris Bruynooghe]

On Thu 10 Feb 2022 at 13:16 +0100, Michael J Gruber wrote:

> Austin Lund venit, vidit, dixit 2022-02-10 06:56:12:
>> I'm clearly doing this python code wrong by not using the iterator correctly:
>> 
>> > import notmuch2
>> > 
>> > d = notmuch2.Database()
>> > m = list(d.messages("since:today"))
>> > p = m[0].path
>> > print(p)
>> 
>> But I seem to be getting a SIGABRT instead of a python stack trace.  Is
>> this the expected behaviour?
>
> You didn't expect it :)
>
> And this can be confusing. d.messages() returns an iterator through
> Message objects whose lifetime depends on the iterator. In contrast,
> thread.get_messages() returns on iterator through OwnedMessage objects
> whose lifetime depends on the thread.
>
> As soon as the iterator is depleted, the returned objects are (possibly)
> gone. (Well, because it's return by reference in Python, and ...)
>
> If you're interested in m[0] only you can "cheat" by not depleting the
> iterator:
>
> mm = next(d.messages("since:today"))
>
> p = mm.path
>
> This never frees the object (I think).
>
> My attempts with notmuch2._message.OwnedMessage (and db as parent)
> failed. There must be a better way, possibly using a context manager or
> search.
>
> I guess usually people just use the iterator in a for loop and do
> something with the message inside the loop (while the iterator is not
> depleted), such as converting it into a proper email.Message object
> (i.e. instantiating a new object from it).

Hum, to be fair I consider this a serious bug in the notmuch2 bindings.
You should not be able to crash whatever you do and not need to
understand the internal notmuch memory model.  I tried to build the
bindings so they would keep alive what they need, but it seems it fails
here.  It would be good to figure out if this can be fixed.

Cheers,
Floris

Re: Python binding SIGABRT/SIGSEGV

[Austin Lund]

On Thu, Feb 10, 2022 at 01:12:47PM +0100, Michael J Gruber wrote:
> Austin Lund venit, vidit, dixit 2022-02-10 06:56:12:
> > I'm clearly doing this python code wrong by not using the iterator correctly:
> > 
> > > import notmuch2
> > > 
> > > d = notmuch2.Database()
> > > m = list(d.messages("since:today"))
> > > p = m[0].path
> > > print(p)
> > 
> > But I seem to be getting a SIGABRT instead of a python stack trace.  Is
> > this the expected behaviour?
> 
> You didn't expect it :)
> 
> And this can be confusing. d.messages() returns an iterator through
> Message objects whose lifetime depends on the iterator. In contrast,
> thread.get_messages() returns on iterator through OwnedMessage objects
> whose lifetime depends on the thread.

I guess I didn't say it explicitly, but I would 'expect' the python
interpreter to raise an exception rather than having an unhandled
exception terminate the program.  Perhaps raising a MemoryError or
ReferenceError or some other exception would be better than an unhandled
SIGABRT.

Re: Python binding SIGABRT/SIGSEGV

[Michael J Gruber]

Austin Lund venit, vidit, dixit 2022-02-10 23:21:58:
> On Thu, Feb 10, 2022 at 01:12:47PM +0100, Michael J Gruber wrote:
> > Austin Lund venit, vidit, dixit 2022-02-10 06:56:12:
> > > I'm clearly doing this python code wrong by not using the iterator correctly:
> > > 
> > > > import notmuch2
> > > > 
> > > > d = notmuch2.Database()
> > > > m = list(d.messages("since:today"))
> > > > p = m[0].path
> > > > print(p)
> > > 
> > > But I seem to be getting a SIGABRT instead of a python stack trace.  Is
> > > this the expected behaviour?
> > 
> > You didn't expect it :)
> > 
> > And this can be confusing. d.messages() returns an iterator through
> > Message objects whose lifetime depends on the iterator. In contrast,
> > thread.get_messages() returns on iterator through OwnedMessage objects
> > whose lifetime depends on the thread.
> 
> I guess I didn't say it explicitly, but I would 'expect' the python
> interpreter to raise an exception rather than having an unhandled
> exception terminate the program.  Perhaps raising a MemoryError or
> ReferenceError or some other exception would be better than an unhandled
> SIGABRT.
> 

In fact you did. Sorry for overlooking this. (I still find db.messages()
vs thread.get_messages() confusing.)

The way memory handling works, one could even expect a more specific
notmuch2.ObjectDestroyedError.

As for the SIGABRT: I guess this is what "python -X faulthandler" is
for, especially in the context of C extension modules:

LANG=C python -X faulthandler t.py
Fatal Python error: Aborted

Current thread 0x00007f5bf667e740 (most recent call first):
  File "/usr/lib64/python3.10/site-packages/notmuch2/_message.py", line
131 in path
  File "/tmp/t.py", line 5 in <module>

Extension modules: _cffi_backend (total: 1)
Abgebrochen (Speicherabzug geschrieben)

(Yes, it disrespects LANG)

Line 131 is

ret = capi.lib.notmuch_message_get_filename(self._msg_p)

in the path method, and since _msg_p is a base.MemoryPointer() I would
have hoped (now) to get a notmuch2.ObjectDestroyedError. Maybe _msg_p is
destroyed but the Message object is not? Fishing in the dark here ...

Michael

P.S.: Sorry for the repeated message. I had problems with list bounces
in the past, then remembered the original alias wrong, then replied to
the reply to the wrong (only off-list-delivered) post. git@grubix.eu is
the one subscribed to the list. I'll remember now.

Re: Python binding SIGABRT/SIGSEGV

[Tomi Ollila]

On Fri, Feb 11 2022, Michael J. Gruber wrote:

> Austin Lund venit, vidit, dixit 2022-02-10 23:21:58:
>> On Thu, Feb 10, 2022 at 01:12:47PM +0100, Michael J Gruber wrote:
>> > Austin Lund venit, vidit, dixit 2022-02-10 06:56:12:
>> > > I'm clearly doing this python code wrong by not using the iterator correctly:
>> > > 
>> > > > import notmuch2
>> > > > 
>> > > > d = notmuch2.Database()
>> > > > m = list(d.messages("since:today"))
>> > > > p = m[0].path
>> > > > print(p)
>> > > 
>> > > But I seem to be getting a SIGABRT instead of a python stack trace.  Is
>> > > this the expected behaviour?
>> > 
>> > You didn't expect it :)
>> > 
>> > And this can be confusing. d.messages() returns an iterator through
>> > Message objects whose lifetime depends on the iterator. In contrast,
>> > thread.get_messages() returns on iterator through OwnedMessage objects
>> > whose lifetime depends on the thread.
>> 
>> I guess I didn't say it explicitly, but I would 'expect' the python
>> interpreter to raise an exception rather than having an unhandled
>> exception terminate the program.  Perhaps raising a MemoryError or
>> ReferenceError or some other exception would be better than an unhandled
>> SIGABRT.
>> 
>
> In fact you did. Sorry for overlooking this. (I still find db.messages()
> vs thread.get_messages() confusing.)
>
> The way memory handling works, one could even expect a more specific
> notmuch2.ObjectDestroyedError.
>
> As for the SIGABRT: I guess this is what "python -X faulthandler" is
> for, especially in the context of C extension modules:
>
> LANG=C python -X faulthandler t.py
> Fatal Python error: Aborted
>
> Current thread 0x00007f5bf667e740 (most recent call first):
>   File "/usr/lib64/python3.10/site-packages/notmuch2/_message.py", line
> 131 in path
>   File "/tmp/t.py", line 5 in <module>
>
> Extension modules: _cffi_backend (total: 1)
> Abgebrochen (Speicherabzug geschrieben)
>
> (Yes, it disrespects LANG)

You'd probably need LC_ALL=C for "overwriting" whatever in other LC_*
variables has been set. IIRC LANG just set default in case particular
LC_* is not set

Unfortunately this does not help solving SIGABRT case...

Tomi

>
> Line 131 is
>
> ret = capi.lib.notmuch_message_get_filename(self._msg_p)
>
> in the path method, and since _msg_p is a base.MemoryPointer() I would
> have hoped (now) to get a notmuch2.ObjectDestroyedError. Maybe _msg_p is
> destroyed but the Message object is not? Fishing in the dark here ...
>
> Michael
>
> P.S.: Sorry for the repeated message. I had problems with list bounces
> in the past, then remembered the original alias wrong, then replied to
> the reply to the wrong (only off-list-delivered) post. git@grubix.eu is
> the one subscribed to the list. I'll remember now.