From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from localhost (localhost [127.0.0.1]) by arlo.cworth.org (Postfix) with ESMTP id 5DB876DE0198 for ; Sun, 18 Nov 2018 13:01:21 -0800 (PST) X-Virus-Scanned: Debian amavisd-new at cworth.org X-Spam-Flag: NO X-Spam-Score: 1.803 X-Spam-Level: * X-Spam-Status: No, score=1.803 tagged_above=-999 required=5 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, FREEMAIL_FORGED_REPLYTO=2.503, RCVD_IN_DNSWL_LOW=-0.7] autolearn=disabled Received: from arlo.cworth.org ([127.0.0.1]) by localhost (arlo.cworth.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Z8mSLRVpHARx for ; Sun, 18 Nov 2018 13:01:19 -0800 (PST) X-Greylist: delayed 2330 seconds by postgrey-1.36 at arlo; Sun, 18 Nov 2018 13:01:19 PST Received: from out002.mailprotect.be (out002.mailprotect.be [83.217.72.86]) by arlo.cworth.org (Postfix) with ESMTPS id 9D4026DE00D4 for ; Sun, 18 Nov 2018 13:01:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=mailprotect.be; s=mail; h=Content-Transfer-Encoding:Content-Type: In-Reply-To:MIME-Version:Date:Message-ID:From:Reply-To:Subject:References:To: sender:cc:bcc; bh=oTMu3Kh2xpg8KT5bQMjswjG/LBjA2ACKcz6+U92+Hwg=; b=iVP0sL+ssVQ ou2+l0Av1Q2IW6zvCLJf2+MG5ZGlRXUSSkU8s1s0fG+b+GKre2pbalhnODtI4SkU7Mu0l3ddyL5xQ g98jd3V3DkHFgDDwSnq4fNwSud3HZbRGRtym0x/Eh+xYtG/AZzEi0kySFe70RioFVIDwgHdpHMH/M 8ShxNanWX5qEB21ODpdsZu5X7KMuX8Lo9OiDV4JA7lONn3ipvXi0qSkL2Uwx9DBSvdF16A8rqD/yr 96TeMZ4af9Iw1eXpTAVZoMjI5XVeOvLBUFgi3o5g3oYahmiyiwO0FGZoIVS5jnF7VxuOLtjph3KVx jxymv/ZwV8xZ7a0ESoIoivw==; Received: from smtp-auth.mailprotect.be ([178.208.39.155]) by com-mpt-out002.mailprotect.be with esmtp (Exim 4.89) (envelope-from ) id 1gOTaK-0004lP-D6 for notmuch@notmuchmail.org; Sun, 18 Nov 2018 21:22:25 +0100 Received: from [192.168.2.60] (d5153a70a.access.telenet.be [81.83.167.10]) by smtp-auth.mailprotect.be (Postfix) with ESMTPA id 7F745C0645 for ; Sun, 18 Nov 2018 21:22:17 +0100 (CET) To: notmuch@notmuchmail.org References: <154177495352.5588.12072713055654441286@x1.localdomain> Subject: Re: segfault using python bindings Reply-To: Dirk Van Haerenborgh From: Dirk Van Haerenborgh Openpgp: preference=signencrypt Autocrypt: addr=dirk.vanhaerenborgh@senso2.me; prefer-encrypt=mutual; keydata= xsFNBFkZcRUBEADxkmC2bDsHycFQyOO6YGRmPAy/aNZNQ7gLtq8zhG25lD0oNkdRBTN3vRCp ZCbPaevh/5pM7b+pSp9oGzLc5AlivZS+AUt25h+Sr16cRK6Du/0GpH0BqxZA29oTxsKssJIl 7JJXj7FxECPBSFGBh7Brb65WqXDbVfhkU/N5Q9j4Zf0xyjFp5ERIq6CerCVm8SxNLYDEV7lo IMlsK2yN9Uv9AezqsYXIGBe/LNO69TNcOTWZzhi6u83zfLl/X9prrakNVtIgo7MEDeJvVTZc Deh0UGAgfWkHz6/xaLqrFPp7nhpi65iC0hi03ZLs3NQI0IGCNFm95Idc01vIOBUAmOpp+xVG JmKgrh4dyw5FRKTO6hw4SyIJa8yk68v2whA3sIIX8qKBec9ggph6Tb9LA7gCsIc36su3dq9s /quYLa14Gg14sS53xgJTFJoLmRK1zMplinKr1jNKxPHd23YEjfSQiguTxwc1uVgg32ODfCmS vpUoG+rvgR64JxPGbkRsPooM5iOX32tCbKUF+g/++F8qVTAgvHkWIh7/44MejEQvfqgGHQCO 7j7lS/2LmKWeK+Dc50bW0uGYKEbH5cxC0tWJ52PetKxFjNiS/tnh2fVulEvFYX30BJL1a1jJ SkMTtIrQPlZ02na8Id3f51OT8IqDlXKxsbAykLhzidG/1q6FoQARAQABzTREaXJrIFZhbiBI YWVyZW5ib3JnaCA8ZGlyay52YW5oYWVyZW5ib3JnaEBzZW5zbzIubWU+wsF9BBMBCAAnBQJZ GXEVAhsjBQkJZgGABQsJCAcCBhUICQoLAgQWAgMBAh4BAheAAAoJEE+41lvdPmrdfwwP/jiQ O1RXSzjTWXPFBCtcqwiynfx+GxvzzsEeNt5S3Qe2P7NVWLD3LJvjserPKqleskG0SoJRhpDz +xvA3jRpsfvZfr9qwEdGbm2epGyw5sqex8IM5cE31qT8qPtSGi4psEzem/uu3VhQq8JllDYq LCs5kgn3TrPNf2M+4ksiJOAf5llTNxLKKj16GPuLedeDBRPegwtsRO8yr7WkgA+3AYvKzjEC m2UFrodeUHeYR5LY/Facevu9dHgt3jItYbugnqw9ri2Pt+CXAVnIGl6a1Hc51kPC8jN30zxS e5xV40bP1rbzDN51jcB98O/hIJvwOYAM/aGxQEoynaM9wxmnD+XBplASwyivPK4ANGMBKAdx Y0PFfD0QosVxCxyy5Jux5BSFpGQmq0JXsYZdK95+kd3JCkLV/gpmPIGuOOiqBE5Tlp07cL32 LizyyF8CJ5srLQD7GSKJUvj/Vyf9fj5BakJIZ3pARzVENDJUh0DaWj3plfMgTBr2JDaYQRPp JnU4FOGwxyF52ZzteTo4ouuteXeim9WjqGYHDjpxw4aaHXPvQgTg9lfmN1qwbAGg7mjOKldw qI47wsfMTJdcaoSDMvYzGLnoufYK81hAFGJTgGj5Niwr0nbkbibX9YolV2+eyDLS0ckBZYKh uQqWoR0vSRfI1FWFijjIr/l9iTom+LZ/zsFNBFkZcRUBEADrQQpaUyOv5Bg81rFqGIs/kIBg TDeXjEHEUnaZVC+8/zlG8hDECO+YGR9Rxc7hYIGNGZjktoeCO+n8g0kdPY51M4U3QG7ITcae ih3CthAC1BTC43X22r0iARuw3a4Bhwze4vDaOg0cNAAnk/Zze+Ik67vpKrOyfsiHJST7h5xv LBGb3o7enLBvOQbIWupMBl5N2S9MMPJT5kgoTSwheSWdx7Ned+g9Ir5aQyFfvpXshMYAgzr1 RMZqJPQdFjlTKk1bmG8mS8Azz0s7RyoNhuEMv8m0BX04oehLbK4xURGdEK7kWjQnhrtqqD7U 9X7kD4YX25exbJJTJD8I0Vx+F9pGjPqSi8GhtfEqOhq+HhdvtSSZUafemspfKHcqfcsrbJG9 SQBjJV7UnYh0xptYvZhVi++Qyt2x5pCt5PF60n07C4Iu8TI3oRrrvIEQreKKd42IXp7qDGMi NZNuWq2OdpzdyPvoNxRvengPVgCBKA/tMU9eeujdaI0vRwCHq9J1VORR16AdYYgjvwbe5ySF xso6LdqqAxaw36KewLAZqLLHXUxhMNIokMhwMpggXic5YuigJLmteF7SB5/ra7ULlS5n4tdc EDAy8xFBzWkVvMmt8BJKacCUDsPcvjkLPZzelwcOnKrDpzBtXdaOxKVgZtmduRyUXPv0tBYk z1e3igO4kwARAQABwsFlBBgBCAAPBQJZGXEVAhsMBQkJZgGAAAoJEE+41lvdPmrdpAYP/2yk p2voNoVAyOFEomitPdJEhLNHU/gU7LkC4pjUvxTSnGUKazPRJL1K+0R1ysPJGL4zWWpLCCa5 60PnM0wS3wGx656d3RXeacYCCecoxKNBEjGIT1U8PFl0orysr/F1UtwTRURAwPZ3M4scVO5M +AbXiJ/g88cLHzKyBzbbyJi7ypH/eR98Mlo7xyRpeGsq4TILzMBy7TFvGQdLy+ANKMCjJRZW 66/nfmZ5mndAFJ/ta08zfnWbJd6bVhL0KAnTGr88jwXjF75FzYYzBfZX4vqiy++ZRSJMLrCA qIjx2tcdqCMn+zVpmaoNgF5O2D9sRMUeQjvFZaDuNYf+lMWlqViGgNdW21lpLglG6HqMefBo wDNxgkZ/S60IJk1FMq8O5zmb/Ex6rzYhG/f0h3j1icP9qwyYPCw6erhJTpKuV0CZIXjOpOZC ICc1cfQKE6yUHjd36K1exrz9s0CkanB0WcD9TTsnPG5Xk4zb3i6XrV5ttMzdTW8xcSZO5EhS AhIGG1FXJN4soelfa/MenBULng4hgQbkvcv5+Kdb36yLzivjMP0eizfyOo/G+fxUEZplqwSz RIry26uNPv6SIJl1WHMMcCHbf671uiIFfh2jqzRhQlbieFsCI0vUGEPaHjKNEJ2DlM8uuzrL lLDeNzRw696+ullUqFw/7AeLLmeC3zz2 Organization: Senso2Me Message-ID: Date: Sun, 18 Nov 2018 21:22:15 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.3.0 MIME-Version: 1.0 In-Reply-To: <154177495352.5588.12072713055654441286@x1.localdomain> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit Content-Language: en-US X-Originating-IP: 178.208.39.155 X-SpamExperts-Domain: mailprotect.be X-SpamExperts-Username: 178.208.39.128/27 Authentication-Results: mailprotect.be; auth=pass smtp.auth=178.208.39.128/27@mailprotect.be X-SpamExperts-Outgoing-Class: ham X-SpamExperts-Outgoing-Evidence: Combined (0.11) X-Recommended-Action: accept X-Filter-ID: EX5BVjFpneJeBchSMxfU5t9ED8W8QYvRUkcWW2WBsE5602E9L7XzfQH6nu9C/Fh9KJzpNe6xgvOx q3u0UDjvO6A1i+D2biAhVFH6fjDeRy9R51/rLBJvzDjIczWbRLlJvpf3JjS9poTbC9IODXxixmPQ HoQrBGTAejKCyFbhreVy0vKNdJMHZ2K0qgciRU0qONRxRTOF7WipAhWSwUnloVK7hDWwk3SlM88Y 8RxvUUJgkjwqr3ytcoT8+IMec5F3QuMb7P+4hoAyoWeOQ+b4oTQrzKM+ea/cZerB51gfCwtuBNu/ Tx4Fu+kFXr+ng+N+xZAsWducJfkrhp2hCEbtetapCu+NsuMAnrPBU2QFddkiuJj+IkuT/Kmmlqzk Ckr7CiQQ4kqBtO0YcEZfXjILsnmdARK1kLxGGp0urJMZncONDgfBsXhpZpGXRrEDnbmi5YE5enyc cp7RH4WQio3uGaHI2iuff/uZWO1cdCI6aFayNKvOeLJBg69UhVs/U6xGeamVvJiHj7DTd6zfY7ON 9KjkmHCT6ExMpimm4DYFpZ6KlM9YwcZSq44U2lb1Hqcx31MusX2Xci0Z8D9aucAtXiQsncMJhgwW oc/PBBv7wDlh2nmrUpfzWc0sUv4FNFzRANXMKKxBjlwTDfQ/eeNkHtySxDGtICXCdNpMgIxlFlC4 2IPZn/fX2Zv3VhQ2TlBD4mzfL6uQrnIMi0X9aiLKl+CRCse6/EUT2Y/Vj4dRKzfpfj5sSQhb3cc4 IkKXIhy+k3fwpqGT/H3p7CoyYF2PHD52ATWO/pGVwiLnw8ckWdgLkxZSyECAnkEAgYV65ENp8ZZD w6MG3E9U62qnI+GE+oErWMddmgnZiYJZW8Mx5OLMzrS+ipRTMma8gsCSR+Ag53AchVbc0JpeNrhF rx64zcw4Oh2LkwsmIC+tUiJAU2FsIH8EF3joBheiindVvWjMi8Q2MHdFe9rI0hiXVMpEqC8/3bIm oUNQxaBiRh+ELzQ7TjzYpbt4xxgmOeDi/ycL1nkyvqm+OpheAIxpeuwPhPyLh6kclJi0kfUq3h56 1kc= X-Report-Abuse-To: spam@com-mpt-mgt001.mailprotect.be X-BeenThere: notmuch@notmuchmail.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Use and development of the notmuch mail system." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 18 Nov 2018 21:01:21 -0000 Hi, I've encountered something very similar just today. But not with python, but my rust bindings. I could very easily reproduce it using: for (messages = notmuch_thread_get_messages (thread); notmuch_messages_valid (messages); notmuch_messages_move_to_next (messages)) { notmuch_message_t *message = notmuch_messages_get (messages); const char *mid = notmuch_message_get_message_id(message); fprintf(stdout, "Message: %s\n", mid); notmuch_message_destroy(message); } for (messages = notmuch_thread_get_messages (thread); notmuch_messages_valid (messages); notmuch_messages_move_to_next (messages)) { notmuch_message_t *message = notmuch_messages_get (messages); const char *mid = notmuch_message_get_message_id(message); fprintf(stdout, "Message: %s\n", mid); notmuch_message_destroy(message); } This is not a typo. I deliberately duplicated that bit. The second time it runs that loop, it will always segfault, unless you omit the first 'notmuch_message_destroy' call. Given your example, I suspect the Python3 bindings to do something very similar. I would think that this is the correct way to use the API? Kind regards, -Dirk On Fri, 09 Nov 2018 15:49:13 +0100, David Čepelík wrote: > Hello Notmuch devs, > > I'm facing an issue trying to use the Python bindings. This trivial > piece of code segfaults: > > import notmuch > > database = notmuch.Database() > threads = database.create_query('tag:inbox and not > tag:killed').search_threads() > > for t in threads: > print("Thread:", t) > msgs = t.get_toplevel_messages() > for m in msgs: > print("Message:", m) > msgs = t.get_toplevel_messages() > next(msgs) > > The problem is triggered by the call to next. Doing this instead works: > > database = notmuch.Database() > > threads = database.create_query('tag:inbox and not > tag:killed').search_threads() > for t in threads: > print("Thread:", t) > msgs = t.get_toplevel_messages() > for m in msgs: > print("Message:", m) > > threads = database.create_query('tag:inbox and not > tag:killed').search_threads() > for t in threads: > print("Thread:", t) > msgs = t.get_toplevel_messages() > for m in msgs: > print("Message:", m) > > It seems that the problem is caused by calling get_toplevel_messages > twice on the same Threads object. > > I've been able to narrow the problem down using gdb. The first few > frames of the stack-trace are: > > #0 0x000055555557da90 in () > #1 0x00007ffff665db5a in > Xapian::Document::Internal::get_value[abi:cxx11](unsigned int) const > () at /usr/lib/libxapian.so.30 #2 0x00007ffff665db91 in > Xapian::Document::get_value[abi:cxx11](unsigned int) const () at > /usr/lib/libxapian.so.30 #3 0x00007ffff6e3165a in > notmuch_message_get_header(notmuch_message_t*, char const*) > (message=0x5555556e6920, header=0x7ffff7195f78 "from") at > lib/message.cc:549 > #4 0x00007ffff6efb1c8 in ffi_call_unix64 () at /usr/lib/libffi.so.6 > > The () seems to denote C++'s tuple class: > > (gdb) frame 0 #0 0x000055555557da90 in ?? () > (gdb) lis 1 // -*- C++ -*- > 2 > 3 // Copyright (C) 2007-2018 Free Software Foundation, Inc. > 4 // > 5 // This file is part of the GNU ISO C++ Library. This library is > free 6 // software; you can redistribute it and/or modify it under > the 7 // terms of the GNU General Public License as published by the > 8 // Free Software Foundation; either version 3, or (at your option) > 9 // any later version. > 10 > > Searching further, I've arrived at the following piece of Xapian code > (xapian-core/backends/documentinternal.h): > > 390 std::string get_value(Xapian::valueno slot) const { > 391 if (values) { > 392 auto i = values->find(slot); > 393 if (i != values->end()) > 394 return i->second; > 395 return std::string(); > 396 } > 397 398 return fetch_value(slot); > 399 } > > Since the invalid dereference indicates a tuple, I suspect the crash > stems from the use `second' on line 394. (The (->) dereference likely > does not cause the crash since otherwise we wouldn't arrive at the > tuple.) > > When I use alot with this version of notmuch/python bindings, it works > just fine, but I suspect that's because alot wraps all the provided > objects to avoid this sort of bugs (and allow for repeated iteration > over Threads objects, etc), and hence avoid calling > get_toplevel_messages() > twice. > > Does the problem lie with my fundamental misunderstanding of how the > bindings work, or is this a bug? > > Linux x1 4.18.16-arch1-1-ARCH #1 SMP PREEMPT Sat Oct 20 22:06:45 UTC > 2018 x86_64 GNU/Linux Python 3.7.1 built from upstream @ 7f726c6 (AUR: > notmuch-git 3:0.28.2.7.g7f726c6e-1) > > > Regards, David