segfault using python bindings

segfault using python bindings

[David Čepelík]

[-- Attachment #1: Type: text/plain, Size: 3737 bytes --]

Hello Notmuch devs,

I'm facing an issue trying to use the Python bindings. This trivial
piece of code segfaults:

    import notmuch

    database = notmuch.Database()
    threads = database.create_query('tag:inbox and not tag:killed').search_threads()
    
    for t in threads:
        print("Thread:", t)
        msgs = t.get_toplevel_messages()
        for m in msgs:
            print("Message:", m)
        msgs = t.get_toplevel_messages()
        next(msgs)

The problem is triggered by the call to next. Doing this instead works:

    database = notmuch.Database()

    threads = database.create_query('tag:inbox and not tag:killed').search_threads()
    for t in threads:
        print("Thread:", t)
        msgs = t.get_toplevel_messages()
        for m in msgs:
            print("Message:", m)

    threads = database.create_query('tag:inbox and not tag:killed').search_threads()
    for t in threads:
        print("Thread:", t)
        msgs = t.get_toplevel_messages()
        for m in msgs:
            print("Message:", m)

It seems that the problem is caused by calling get_toplevel_messages
twice on the same Threads object.

I've been able to narrow the problem down using gdb. The first few
frames of the stack-trace are:

    #0  0x000055555557da90 in  ()
    #1  0x00007ffff665db5a in Xapian::Document::Internal::get_value[abi:cxx11](unsigned int) const () at /usr/lib/libxapian.so.30
    #2  0x00007ffff665db91 in Xapian::Document::get_value[abi:cxx11](unsigned int) const () at /usr/lib/libxapian.so.30
    #3  0x00007ffff6e3165a in notmuch_message_get_header(notmuch_message_t*, char const*)
        (message=0x5555556e6920, header=0x7ffff7195f78 "from") at lib/message.cc:549
    #4  0x00007ffff6efb1c8 in ffi_call_unix64 () at /usr/lib/libffi.so.6

The () seems to denote C++'s tuple class:

    (gdb) frame 0
    #0  0x000055555557da90 in ?? ()
    (gdb) lis
    1	// <tuple> -*- C++ -*-
    2	
    3	// Copyright (C) 2007-2018 Free Software Foundation, Inc.
    4	//
    5	// This file is part of the GNU ISO C++ Library.  This library is free
    6	// software; you can redistribute it and/or modify it under the
    7	// terms of the GNU General Public License as published by the
    8	// Free Software Foundation; either version 3, or (at your option)
    9	// any later version.
    10	

Searching further, I've arrived at the following piece of Xapian code
(xapian-core/backends/documentinternal.h):

    390     std::string get_value(Xapian::valueno slot) const {
    391         if (values) {
    392             auto i = values->find(slot);
    393             if (i != values->end())
    394                 return i->second;
    395             return std::string();
    396         }
    397
    398         return fetch_value(slot);
    399     }

Since the invalid dereference indicates a tuple, I suspect the crash
stems from the use `second' on line 394. (The (->) dereference likely
does not cause the crash since otherwise we wouldn't arrive at the
tuple.)

When I use alot with this version of notmuch/python bindings, it works
just fine, but I suspect that's because alot wraps all the provided
objects to avoid this sort of bugs (and allow for repeated iteration
over Threads objects, etc), and hence avoid calling get_toplevel_messages()
twice.

Does the problem lie with my fundamental misunderstanding of how the
bindings work, or is this a bug?

Linux x1 4.18.16-arch1-1-ARCH #1 SMP PREEMPT Sat Oct 20 22:06:45 UTC 2018 x86_64 GNU/Linux
Python 3.7.1
built from upstream @ 7f726c6 (AUR: notmuch-git 3:0.28.2.7.g7f726c6e-1)


                                            Regards, David

[-- Attachment #2: signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

-----BEGIN PGP SIGNATURE-----

iQEzBAABCAAdFiEE1W8XnnLsjcC3H4ylq55LF2s9c50FAlvlnmYACgkQq55LF2s9
c51CBwf/Tkm6EenR46QKCjgfbpC9yK6/S/9CQRjrsFXGdPQDZRqiszpARabOO16f
orflzuvmNkE7HomaS3U+fpFQiJs1HvSPFFM7uzl/39he9fofFJ8hfZXJZeVi9msg
nApO3aV1ei5+4TWx27MxI52FpXiy5ggNhT0NhgtUh6IYHGC2rn2PqfS6NfCVcqZ+
pLU7Y68Vb9eSPXwRfG2hInmGFcut7KTSGJRJ7r1WG0XG2Kdvh2mX8uVgWtJsQdls
jLKrKKfJyOq4SKKYifGBDZKgR/qJKllKIETiOmPVpVCwMUqGuo/wGXPcCj2KaAdw
8kVJfTGwsskQxJqoF0mLOdW1lIPOsQ==
=9oon
-----END PGP SIGNATURE-----

Re: segfault using python bindings

[David Bremner]

David Čepelík <d@dcepelik.cz> writes:

> Hello Notmuch devs,
>
> I'm facing an issue trying to use the Python bindings. This trivial
> piece of code segfaults:
>
>     import notmuch

I don't remember the details [1], but there are known conflicts between
recent versions of python3 and the way the notmuch python bindings
manage memory. So it could be that. There was also an initiative to
rewrite at (python3 only?) version of the bindings that did not have
this problem. I haven't heard much about that recently.


[1]: There is some discussion in the list archives.

Re: segfault using python bindings

[Gaute Hope]

David Bremner writes on November 11, 2018 21:16:
> [1]: There is some discussion in the list archives.

See id:87lg5z74l3.fsf@tethera.net

Regards, Gaute

Re: segfault using python bindings

[Floris Bruynooghe]

Hi,

On Sun 11 Nov 2018 at 16:16 -0400, David Bremner wrote:

> David Čepelík <d@dcepelik.cz> writes:
>
>> Hello Notmuch devs,
>>
>> I'm facing an issue trying to use the Python bindings. This trivial
>> piece of code segfaults:
>>
>>     import notmuch
>
> I don't remember the details [1], but there are known conflicts between
> recent versions of python3 and the way the notmuch python bindings
> manage memory. So it could be that. There was also an initiative to
> rewrite at (python3 only?) version of the bindings that did not have
> this problem. I haven't heard much about that recently.

These are at https://github.com/flub/notmuch/tree/cffi/bindings/python-cffi

I'm not really convinced of the way forward last time it was discussed
on how to get them merged into notmuch itself so have failed to put in
the not insignificant effort.

I've since wondered if just getting them standalone on pypi is perhaps a
useful service in the mean time as it's relatively little effort.  And
if there eventually is a desire again to get them merged in some way
that could still be done.


Cheers,
Floris

Re: segfault using python bindings

[Brian May]

Floris Bruynooghe <flub@devork.be> writes:

> I've since wondered if just getting them standalone on pypi is perhaps a
> useful service in the mean time as it's relatively little effort.  And
> if there eventually is a desire again to get them merged in some way
> that could still be done.

Standalone on pypi would be my preferred option.

It is defacto Python standard to refer to all dependancies in something
like requirements.txt or Pipfile from pypi.
-- 
Brian May <brian@linuxpenguins.xyz>
https://linuxpenguins.xyz/brian/

Re: segfault using python bindings

[David Bremner]

Brian May <brian@linuxpenguins.xyz> writes:

> Floris Bruynooghe <flub@devork.be> writes:
>
>> I've since wondered if just getting them standalone on pypi is perhaps a
>> useful service in the mean time as it's relatively little effort.  And
>> if there eventually is a desire again to get them merged in some way
>> that could still be done.
>
> Standalone on pypi would be my preferred option.
>
> It is defacto Python standard to refer to all dependancies in something
> like requirements.txt or Pipfile from pypi.

As I mentioned last time this was discussed, the python bindings are
currently more or less a core part of notmuch as both the test
suite and developement need them.

d

Re: segfault using python bindings

[Floris Bruynooghe]

On Fri 16 Nov 2018 at 06:29 -0400, David Bremner wrote:

> Brian May <brian@linuxpenguins.xyz> writes:
>
>> Floris Bruynooghe <flub@devork.be> writes:
>>
>>> I've since wondered if just getting them standalone on pypi is perhaps a
>>> useful service in the mean time as it's relatively little effort.  And
>>> if there eventually is a desire again to get them merged in some way
>>> that could still be done.
>>
>> Standalone on pypi would be my preferred option.
>>
>> It is defacto Python standard to refer to all dependancies in something
>> like requirements.txt or Pipfile from pypi.
>
> As I mentioned last time this was discussed, the python bindings are
> currently more or less a core part of notmuch as both the test
> suite and developement need them.

Sure, I think pypi publishing is orthogonal to this however.  Either or
both versions of the bindings could be published on pypi in addition to
being in the main repo.  As Brian mentions it would improve
discoverability and improves integration on the python side.  There's
even tooling to bundle the library these days with the manylinux1
wheels.  So there's no need to stop anyone who'd like to do this.

Re: segfault using python bindings

[David Bremner]

Floris Bruynooghe <flub@devork.be> writes:

>> As I mentioned last time this was discussed, the python bindings are
>> currently more or less a core part of notmuch as both the test
>> suite and developement need them.
>
> Sure, I think pypi publishing is orthogonal to this however.  Either or
> both versions of the bindings could be published on pypi in addition to
> being in the main repo.  As Brian mentions it would improve
> discoverability and improves integration on the python side.  There's
> even tooling to bundle the library these days with the manylinux1
> wheels.  So there's no need to stop anyone who'd like to do this.

Well, I agree with all that (and did in the previous thread too). But
the context was Florian's idea of publishing on pypi instead of/before
integrating with notmuch. That's of course his right to do, but my main
(selfish) interest is in having python bindings shipping with notmuch
that work properly with recent python3. I guess even having a separate
set of incompatible python3 only bindings would be better than the
current situation. We could just ship the two bindings in parallel,
deprecate the python2 bindings, and give people a year or so to
transition.

Re: segfault using python bindings

[David Bremner]

Floris Bruynooghe <flub@devork.be> writes:

>
> These are at https://github.com/flub/notmuch/tree/cffi/bindings/python-cffi
>
> I'm not really convinced of the way forward last time it was discussed
> on how to get them merged into notmuch itself so have failed to put in
> the not insignificant effort.
>
> I've since wondered if just getting them standalone on pypi is perhaps a
> useful service in the mean time as it's relatively little effort.  And
> if there eventually is a desire again to get them merged in some way
> that could still be done.
>

What effort are you referring to specifically? Integration with the
notmuch test suite?

Re: segfault using python bindings

[Daniel Kahn Gillmor]

[-- Attachment #1: Type: text/plain, Size: 1082 bytes --]

On Fri 2018-11-16 06:27:12 -0400, David Bremner wrote:
> Floris Bruynooghe <flub@devork.be> writes:
>
>> These are at https://github.com/flub/notmuch/tree/cffi/bindings/python-cffi
>>
>> I'm not really convinced of the way forward last time it was discussed
>> on how to get them merged into notmuch itself so have failed to put in
>> the not insignificant effort.
>>
>> I've since wondered if just getting them standalone on pypi is perhaps a
>> useful service in the mean time as it's relatively little effort.  And
>> if there eventually is a desire again to get them merged in some way
>> that could still be done.
>
> What effort are you referring to specifically? Integration with the
> notmuch test suite?

My recollection is that the main question was about supporting the old
python interface with the new bindings, so that consumers would have a
smooth upgrade path.  Is that not right?

Floris, i really appreciate the work you put in here, and i'd love to
see notmuch be able to adopt it directly.   can we figure out what is
needed to take these changes?

       --dkg

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

Re: segfault using python bindings

[Floris Bruynooghe]

On Fri 16 Nov 2018 at 07:15 -0500, Daniel Kahn Gillmor wrote:

> On Fri 2018-11-16 06:27:12 -0400, David Bremner wrote:
>> Floris Bruynooghe <flub@devork.be> writes:
>>
>>> These are at https://github.com/flub/notmuch/tree/cffi/bindings/python-cffi
>>>
>>> I'm not really convinced of the way forward last time it was discussed
>>> on how to get them merged into notmuch itself so have failed to put in
>>> the not insignificant effort.
>>>
>>> I've since wondered if just getting them standalone on pypi is perhaps a
>>> useful service in the mean time as it's relatively little effort.  And
>>> if there eventually is a desire again to get them merged in some way
>>> that could still be done.
>>
>> What effort are you referring to specifically? Integration with the
>> notmuch test suite?
>
> My recollection is that the main question was about supporting the old
> python interface with the new bindings, so that consumers would have a
> smooth upgrade path.  Is that not right?

That's indeed what I was referring to, integration with the test suite
is fine as was discussed last time imho.

> Floris, i really appreciate the work you put in here, and i'd love to
> see notmuch be able to adopt it directly.   can we figure out what is
> needed to take these changes?

Thanks.  I think mainly when the technical approach was discussed [0] no
actual users of the current Python API weighed in with if they'd be
interested in a migration of the API and if so, how it might work for
them.  So while the gradual approach described there is technically
somewhat nice I have no idea if anyone would benefit from it, or whether
the benefits outweigh all the work involved.

As I was recently thinking however, maybe there's nothing wrong with new
bindings being published as a 3rd party package on pypi.  It'd make it
more discoverable and if people start to adopt it maybe there'd be more
demand for integrating it back with more clarity over how smooth a
transition path needs to be.

Also lastly an apology.  I could have done more to move this forward,
but I simply haven't found^Wmade the time for it.

Cheers,
Floris


[0] id:py3imv2cjp28.fsf@devork.be

Re: segfault using python bindings

[David Bremner]

Floris Bruynooghe <flub@devork.be> writes:

> These are at https://github.com/flub/notmuch/tree/cffi/bindings/python-cffi
>
> I'm not really convinced of the way forward last time it was discussed
> on how to get them merged into notmuch itself so have failed to put in
> the not insignificant effort.
>
> I've since wondered if just getting them standalone on pypi is perhaps a
> useful service in the mean time as it's relatively little effort.  And
> if there eventually is a desire again to get them merged in some way
> that could still be done.

Can you remind me what the percieved blockers are for merging into the
main notmuch tree? I'm less hung up on python2 compatibility than I used
to be, fwiw. I'd be ok with shipping the old python2 bindings in contrib
for a bit for those who still need/want them, but concentrate our
maintenance effort on the cffi bindings.

d

Re: segfault using python bindings

[Floris Bruynooghe]

On Wed 14 Aug 2019 at 16:20 -0300, David Bremner wrote:

> Floris Bruynooghe <flub@devork.be> writes:
>
>> These are at https://github.com/flub/notmuch/tree/cffi/bindings/python-cffi
>>
>> I'm not really convinced of the way forward last time it was discussed
>> on how to get them merged into notmuch itself so have failed to put in
>> the not insignificant effort.
>>
>> I've since wondered if just getting them standalone on pypi is perhaps a
>> useful service in the mean time as it's relatively little effort.  And
>> if there eventually is a desire again to get them merged in some way
>> that could still be done.
>
> Can you remind me what the percieved blockers are for merging into the
> main notmuch tree? I'm less hung up on python2 compatibility than I used
> to be, fwiw. I'd be ok with shipping the old python2 bindings in contrib
> for a bit for those who still need/want them, but concentrate our
> maintenance effort on the cffi bindings.

IIRC it was mostly about how to support transitioning from one API to
the other since currently there's no compatibility.  I guess there's
nothing stopping one from using both APIs in one codebase though, AFAIK
Xapian handles the required locking.  But some of the discussions
suggested being able to create a new Message object from an old one etc,
allowing you to do more mixing during a transition period.  This is the
part that I said is possible but a lot of work and questionable if no
one thought they'd be using it.

Cheers,
Floris

Re: segfault using python bindings

[David Bremner]

Floris Bruynooghe <flub@devork.be> writes:

> On Wed 14 Aug 2019 at 16:20 -0300, David Bremner wrote:
>>
>> Can you remind me what the percieved blockers are for merging into the
>> main notmuch tree? I'm less hung up on python2 compatibility than I used
>> to be, fwiw. I'd be ok with shipping the old python2 bindings in contrib
>> for a bit for those who still need/want them, but concentrate our
>> maintenance effort on the cffi bindings.
>
> IIRC it was mostly about how to support transitioning from one API to
> the other since currently there's no compatibility.  I guess there's
> nothing stopping one from using both APIs in one codebase though, AFAIK
> Xapian handles the required locking.  But some of the discussions
> suggested being able to create a new Message object from an old one etc,
> allowing you to do more mixing during a transition period.  This is the
> part that I said is possible but a lot of work and questionable if no
> one thought they'd be using it.
>

Ah right. Well, given the impending removal of python2 from various
places (e.g. debian testing), I don't think we should be that
fussy/ambitious.

I'd propose

- add the new cffi based bindings, using a distinct name (as you already
  have).

- deprecate the old ones

- port any internal dependencies to the new bindings

- finally drop the old bindings or move them to contrib/ for people slow
  in switching.

I think for the first step we only need a reasonable looking patch
(series?) from you.

Re: segfault using python bindings

[Floris Bruynooghe]

On Thu 15 Aug 2019 at 09:28 -0300, David Bremner wrote:

> Floris Bruynooghe <flub@devork.be> writes:
>
>> On Wed 14 Aug 2019 at 16:20 -0300, David Bremner wrote:
>>>
>>> Can you remind me what the percieved blockers are for merging into the
>>> main notmuch tree? I'm less hung up on python2 compatibility than I used
>>> to be, fwiw. I'd be ok with shipping the old python2 bindings in contrib
>>> for a bit for those who still need/want them, but concentrate our
>>> maintenance effort on the cffi bindings.
>>
>> IIRC it was mostly about how to support transitioning from one API to
>> the other since currently there's no compatibility.  I guess there's
>> nothing stopping one from using both APIs in one codebase though, AFAIK
>> Xapian handles the required locking.  But some of the discussions
>> suggested being able to create a new Message object from an old one etc,
>> allowing you to do more mixing during a transition period.  This is the
>> part that I said is possible but a lot of work and questionable if no
>> one thought they'd be using it.
>>
>
> Ah right. Well, given the impending removal of python2 from various
> places (e.g. debian testing), I don't think we should be that
> fussy/ambitious.
>
> I'd propose
>
> - add the new cffi based bindings, using a distinct name (as you already
>   have).
>
> - deprecate the old ones
>
> - port any internal dependencies to the new bindings
>
> - finally drop the old bindings or move them to contrib/ for people slow
>   in switching.
>
> I think for the first step we only need a reasonable looking patch
> (series?) from you.

Sounds reasonable, just a quick note that I'm on holiday at the moment
and generally won't be too quick.  But I guess there's no rush.  I was
also trying to improve the docs but got sidetracked at some point.
There should be already reasonable good docstrings though.

For path series what did you have in mind?  One single patch with the
whole lot?  The original history at https:://github/flub/notdb?
Something in between?

I also recall some comments about the naming not being liked much
(notdb).  I'm open to some bikeshedding on the naming if it bothers
people.  I was only aiming for something short but under a different
import name, which are probably still useful features to keep.

Cheers,
Floris

Re: segfault using python bindings

[Daniel Kahn Gillmor]

[-- Attachment #1: Type: text/plain, Size: 1461 bytes --]

On Tue 2019-08-20 19:20:30 +0200, Floris Bruynooghe wrote:
> Sounds reasonable, just a quick note that I'm on holiday at the moment
> and generally won't be too quick.  But I guess there's no rush.  I was
> also trying to improve the docs but got sidetracked at some point.
> There should be already reasonable good docstrings though.

thanks for having done all this work already!

> For path series what did you have in mind?  One single patch with the
> whole lot?  The original history at https:://github/flub/notdb?
> Something in between?

I would be happy with a series of patches, but the cited github history
has only one commit in the first place :P

> I also recall some comments about the naming not being liked much
> (notdb).  I'm open to some bikeshedding on the naming if it bothers
> people.  I was only aiming for something short but under a different
> import name, which are probably still useful features to keep.

fwiw, "notdb" doesn't really resonate with me, but i wouldn't block you
if you decided to stick with it.

I'd generally prefer something like "notmuch2" because it makes it much
clearer that it is associated with notmuch, and that it is the successor
to the original notmuch bindings.

(i also find "notmuch2" rather unsatisfying, but python doesn't have a
meaningful versioning system for backwards-incompatible API changes for
its modules, so this kind of name augmentation is the only strategy i'm
aware of).

      --dkg

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

Re: segfault using python bindings

[David Bremner]

Daniel Kahn Gillmor <dkg@fifthhorseman.net> writes:

>
> (i also find "notmuch2" rather unsatisfying, but python doesn't have a
> meaningful versioning system for backwards-incompatible API changes for
> its modules, so this kind of name augmentation is the only strategy i'm
> aware of).

Naming is hard. What about "notmuch3" to hint that it's for python3? A
transition plan could be to have notmuch load the new bindings, after a
deprecation period.

d

Re: segfault using python bindings

[Rollins, Jameson]

On Thu, Aug 22 2019, David Bremner <david@tethera.net> wrote:
> Daniel Kahn Gillmor <dkg@fifthhorseman.net> writes:
>
>> (i also find "notmuch2" rather unsatisfying, but python doesn't have a
>> meaningful versioning system for backwards-incompatible API changes for
>> its modules, so this kind of name augmentation is the only strategy i'm
>> aware of).
>
> Naming is hard. What about "notmuch3" to hint that it's for python3? A
> transition plan could be to have notmuch load the new bindings, after a
> deprecation period.

Ug, this naming issue is unfortunate.  I don't really like "notmuch3" as
a reference to python 3, honestly.

What about making these new bindings only for python3, and the old ones
relegating to python2, and then just using the same name?  Is that too
confusing?  Do we need to maintain both concurrently?

jamie.

Re: segfault using python bindings

[David Bremner]

"Rollins, Jameson" <jrollins@caltech.edu> writes:

> What about making these new bindings only for python3, and the old
> ones relegating to python2, and then just using the same name?  Is
> that too confusing?

I imagine it can be done to have 'import notmuch' redirect to different
things under python3 and python2.  Conceptually the two uses would
something like

,----
| #!/usr/bin/python2.7
| 
| import notmuch
| 
| code1
`----

vs.

,----
| #!/usr/bin/python3
| 
| import notmuch
| 
| code2
`----

It would break the naming logjam, but there are a few potential points
of confusion.

1) it requires gluing together the two sets bindings somehow,
complicating things like making a sensible setup.py.

2) if I remember correctly, the differences in the API are going to
force code1 and code2 to be pretty different (hopefully Floris can
correct me here if I'm wrong). That might also make errors a bit
confusing if people get the interpreter wrong.

3) It would also probably complicate documenting both sets of
bindings. How would we make clear in the installed documentation which
was which?

> Do we need to maintain both concurrently?

I guess it depends a bit what you mean by 'maintain'. I do think we need
a transition period before we stop shipping the current bindings, as
several projects (and probably lots of smaller scripts) rely on them.
We might not be too motivated to add new features to the python2
bindings, but I guess we would probably fix any serious-enough bugs.

Re: segfault using python bindings

[Daniel Kahn Gillmor]

[-- Attachment #1: Type: text/plain, Size: 1427 bytes --]

On Thu 2019-08-22 19:37:34 +0000, Rollins, Jameson wrote:
> Ug, this naming issue is unfortunate.  I don't really like "notmuch3" as
> a reference to python 3, honestly.

how about notmuch3000? :P

> What about making these new bindings only for python3, and the old ones
> relegating to python2, and then just using the same name?  Is that too
> confusing?  Do we need to maintain both concurrently?

I don't like the idea of the same-named module with radically different
syntax and semantics.  That'd be fine if python modules had a way to
indicate backward incompatibility, but they don't :/ Just naming the new
module "notmuch" when existing code expects a certain API will result in
a real mess of downstream breakage.

The other possibility would be to implement the old "notmuch" API on top
of the new one with explicitly logged deprecations. But iirc, the
semantics and object lifecycle/ownership issues are subtly different
enough that this would be a non-trivial project.  I'd be happy to be
wrong though, perhaps someone closer to the systems (someone actively
using the current python bindings on an ongoing project?) could look
more closely?

here's what i see as plausible/possible names for the new module if we
can't replicate the old API on top of it:

 * notdb
 * notmuch2
 * notmuch3
 * notmuch_ng (ng == "next generation")
 * not2much
 * notmuchmail
 * notmuch3000
 * notmuch_cffi
 * pynotmuch

     --dkg

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

Re: segfault using python bindings

[David Bremner]

Daniel Kahn Gillmor <dkg@fifthhorseman.net> writes:

> The other possibility would be to implement the old "notmuch" API on top
> of the new one with explicitly logged deprecations. But iirc, the
> semantics and object lifecycle/ownership issues are subtly different
> enough that this would be a non-trivial project.  I'd be happy to be
> wrong though, perhaps someone closer to the systems (someone actively
> using the current python bindings on an ongoing project?) could look
> more closely?

IIUC, it is precisely this compatibility layer that has stalled things
for a year or so. I don't think we can afford to wait longer.

d

Re: segfault using python bindings

[Daniel Kahn Gillmor]

[-- Attachment #1: Type: text/plain, Size: 1019 bytes --]

On Fri 2019-08-23 09:07:12 -0300, David Bremner wrote:
> Daniel Kahn Gillmor <dkg@fifthhorseman.net> writes:
>
>> The other possibility would be to implement the old "notmuch" API on top
>> of the new one with explicitly logged deprecations. But iirc, the
>> semantics and object lifecycle/ownership issues are subtly different
>> enough that this would be a non-trivial project.  I'd be happy to be
>> wrong though, perhaps someone closer to the systems (someone actively
>> using the current python bindings on an ongoing project?) could look
>> more closely?
>
> IIUC, it is precisely this compatibility layer that has stalled things
> for a year or so. I don't think we can afford to wait longer.

that matches my understanding too.  i'd love to move forward with the
improved bindings, under a different name.

if someone later figures out the compatibility layer that allows
implementation of the traditional "notmuch" module api atop the
notmuch2(?)  module, then we can always reclaim the name.

         --dkg

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

Re: segfault using python bindings

[Rollins, Jameson]

On Fri, Aug 23 2019, Daniel Kahn Gillmor <dkg@fifthhorseman.net> wrote:
> that matches my understanding too.  i'd love to move forward with the
> improved bindings, under a different name.
>
> if someone later figures out the compatibility layer that allows
> implementation of the traditional "notmuch" module api atop the
> notmuch2(?)  module, then we can always reclaim the name.

It would be shame if after this deprecation of the original "notmuch"
bindings we're left unable to "import notmuch" to get the notmuch
bindings.

jamie.

Re: segfault using python bindings

[Floris Bruynooghe]

On Wed 21 Aug 2019 at 12:02 -0400, Daniel Kahn Gillmor wrote:

> On Tue 2019-08-20 19:20:30 +0200, Floris Bruynooghe wrote:
>> For path series what did you have in mind?  One single patch with the
>> whole lot?  The original history at https:://github/flub/notdb?
>> Something in between?
>
> I would be happy with a series of patches, but the cited github history
> has only one commit in the first place :P

hah!  Perhaps the actual history is somewhere in a hg repo, but I can't
even find that.  Not that it matters much.

>> I also recall some comments about the naming not being liked much
>> (notdb).  I'm open to some bikeshedding on the naming if it bothers
>> people.  I was only aiming for something short but under a different
>> import name, which are probably still useful features to keep.
>
> fwiw, "notdb" doesn't really resonate with me, but i wouldn't block you
> if you decided to stick with it.
>
> I'd generally prefer something like "notmuch2" because it makes it much
> clearer that it is associated with notmuch, and that it is the successor
> to the original notmuch bindings.

From the ensuing conversation I'm concluding that "notmuch2" is the most
liked option, so I'll go with that (and yes, I share many of the
disappointed sentiments raised on that naming convention, but it's how
things are in python).


I'll try and send a first patch series in the coming week.

Cheers,
Floris

Re: segfault using python bindings

[David Bremner]

Floris Bruynooghe <flub@devork.be> writes:

>
> I'll try and send a first patch series in the coming week.
>

Sounds great. Thanks for you work on this, and for not being scared of
by the bikeshedding ;).

d

Re: segfault using python bindings

[Dirk Van Haerenborgh]

Hi,

I've encountered something very similar just today. But not with python,
but my rust bindings. I could very easily reproduce it using:

for (messages = notmuch_thread_get_messages (thread);
notmuch_messages_valid (messages);
notmuch_messages_move_to_next (messages))
{
notmuch_message_t *message = notmuch_messages_get (messages);
const char *mid = notmuch_message_get_message_id(message);
fprintf(stdout, "Message: %s\n", mid);
notmuch_message_destroy(message);
}

for (messages = notmuch_thread_get_messages (thread);
notmuch_messages_valid (messages);
notmuch_messages_move_to_next (messages))
{
notmuch_message_t *message = notmuch_messages_get (messages);
const char *mid = notmuch_message_get_message_id(message);
fprintf(stdout, "Message: %s\n", mid);
notmuch_message_destroy(message);
}

This is not a typo. I deliberately duplicated that bit.
The second time it runs that loop, it will always segfault, unless you
omit the first 'notmuch_message_destroy' call.

Given your example, I suspect the Python3 bindings to do something very
similar. I would think that this is the correct way to use the API?

Kind regards,
-Dirk


On Fri, 09 Nov 2018 15:49:13 +0100, David Čepelík wrote:

> Hello Notmuch devs,
>
> I'm facing an issue trying to use the Python bindings. This trivial
> piece of code segfaults:
>
> import notmuch
>
> database = notmuch.Database()
> threads = database.create_query('tag:inbox and not
> tag:killed').search_threads()
>
> for t in threads:
> print("Thread:", t)
> msgs = t.get_toplevel_messages()
> for m in msgs:
> print("Message:", m)
> msgs = t.get_toplevel_messages()
> next(msgs)
>
> The problem is triggered by the call to next. Doing this instead works:
>
> database = notmuch.Database()
>
> threads = database.create_query('tag:inbox and not
> tag:killed').search_threads()
> for t in threads:
> print("Thread:", t)
> msgs = t.get_toplevel_messages()
> for m in msgs:
> print("Message:", m)
>
> threads = database.create_query('tag:inbox and not
> tag:killed').search_threads()
> for t in threads:
> print("Thread:", t)
> msgs = t.get_toplevel_messages()
> for m in msgs:
> print("Message:", m)
>
> It seems that the problem is caused by calling get_toplevel_messages
> twice on the same Threads object.
>
> I've been able to narrow the problem down using gdb. The first few
> frames of the stack-trace are:
>
> #0 0x000055555557da90 in ()
> #1 0x00007ffff665db5a in
> Xapian::Document::Internal::get_value[abi:cxx11](unsigned int) const
> () at /usr/lib/libxapian.so.30 #2 0x00007ffff665db91 in
> Xapian::Document::get_value[abi:cxx11](unsigned int) const () at
> /usr/lib/libxapian.so.30 #3 0x00007ffff6e3165a in
> notmuch_message_get_header(notmuch_message_t*, char const*)
> (message=0x5555556e6920, header=0x7ffff7195f78 "from") at
> lib/message.cc:549
> #4 0x00007ffff6efb1c8 in ffi_call_unix64 () at /usr/lib/libffi.so.6
>
> The () seems to denote C++'s tuple class:
>
> (gdb) frame 0 #0 0x000055555557da90 in ?? ()
> (gdb) lis 1 // <tuple> -*- C++ -*-
> 2
> 3 // Copyright (C) 2007-2018 Free Software Foundation, Inc.
> 4 //
> 5 // This file is part of the GNU ISO C++ Library. This library is
> free 6 // software; you can redistribute it and/or modify it under
> the 7 // terms of the GNU General Public License as published by
the
> 8 // Free Software Foundation; either version 3, or (at your option)
> 9 // any later version.
> 10
>
> Searching further, I've arrived at the following piece of Xapian code
> (xapian-core/backends/documentinternal.h):
>
> 390 std::string get_value(Xapian::valueno slot) const {
> 391 if (values) {
> 392 auto i = values->find(slot);
> 393 if (i != values->end())
> 394 return i->second;
> 395 return std::string();
> 396 }
> 397 398 return fetch_value(slot);
> 399 }
>
> Since the invalid dereference indicates a tuple, I suspect the crash
> stems from the use `second' on line 394. (The (->) dereference likely
> does not cause the crash since otherwise we wouldn't arrive at the
> tuple.)
>
> When I use alot with this version of notmuch/python bindings, it works
> just fine, but I suspect that's because alot wraps all the provided
> objects to avoid this sort of bugs (and allow for repeated iteration
> over Threads objects, etc), and hence avoid calling
> get_toplevel_messages()
> twice.
>
> Does the problem lie with my fundamental misunderstanding of how the
> bindings work, or is this a bug?
>
> Linux x1 4.18.16-arch1-1-ARCH #1 SMP PREEMPT Sat Oct 20 22:06:45 UTC
> 2018 x86_64 GNU/Linux Python 3.7.1 built from upstream @ 7f726c6 (AUR:
> notmuch-git 3:0.28.2.7.g7f726c6e-1)
>
>
> Regards, David

Re: segfault using python bindings

[David Bremner]

Dirk Van Haerenborgh <dirk.vanhaerenborgh@senso2.me> writes:


> This is not a typo. I deliberately duplicated that bit.
> The second time it runs that loop, it will always segfault, unless you
> omit the first 'notmuch_message_destroy' call.
>
> Given your example, I suspect the Python3 bindings to do something very
> similar. I would think that this is the correct way to use the API?
>
> Kind regards,
> -Dirk

Unless I misremember / misunderstand something, calling
notmuch_message_destroy is not a good idea, unless you are going to
recreate the thread object. The thread has an internal list of messages,
which you are freeing.

Possibly the documentation could be improved on this point.

d

Re: segfault using python bindings

[Dirk Van Haerenborgh]

[-- Attachment #1: Type: text/plain, Size: 1451 bytes --]

Thanks,

The documentation regarding lifetime of pointers is indeed at some points
rather limited.
Is there ever a need for calling thread/message_destroy?

From what I understood, lifetime is like this:
Database > Query > Threads > Thread > Messages > Message

I assumed that destroying a 'Message' would only invalidate it for the
'Messages' iterator. Given that you can't destroy a message without
invalidating it for the Thread itself, it seems weird that the
documentation states that the 'Message' is only valid for the lifetime of
the iterator.

Can someone clarify this for me? It would simplify things a lot in
notmuch-rs.

-Dirk

On Mon, 19 Nov 2018 at 00:34, David Bremner <david@tethera.net> wrote:

> Dirk Van Haerenborgh <dirk.vanhaerenborgh@senso2.me> writes:
>
>
> > This is not a typo. I deliberately duplicated that bit.
> > The second time it runs that loop, it will always segfault, unless you
> > omit the first 'notmuch_message_destroy' call.
> >
> > Given your example, I suspect the Python3 bindings to do something very
> > similar. I would think that this is the correct way to use the API?
> >
> > Kind regards,
> > -Dirk
>
> Unless I misremember / misunderstand something, calling
> notmuch_message_destroy is not a good idea, unless you are going to
> recreate the thread object. The thread has an internal list of messages,
> which you are freeing.
>
> Possibly the documentation could be improved on this point.
>
> d
>

[-- Attachment #2: Type: text/html, Size: 2033 bytes --]