From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from localhost (localhost [127.0.0.1]) by olra.theworths.org (Postfix) with ESMTP id B170A431FAF for ; Thu, 12 Apr 2012 09:57:50 -0700 (PDT) X-Virus-Scanned: Debian amavisd-new at olra.theworths.org X-Spam-Flag: NO X-Spam-Score: -0.7 X-Spam-Level: X-Spam-Status: No, score=-0.7 tagged_above=-999 required=5 tests=[RCVD_IN_DNSWL_LOW=-0.7] autolearn=disabled Received: from olra.theworths.org ([127.0.0.1]) by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Ks05ixbL1F5s for ; Thu, 12 Apr 2012 09:57:50 -0700 (PDT) Received: from dmz-mailsec-scanner-4.mit.edu (DMZ-MAILSEC-SCANNER-4.MIT.EDU [18.9.25.15]) by olra.theworths.org (Postfix) with ESMTP id E21AC431FAE for ; Thu, 12 Apr 2012 09:57:49 -0700 (PDT) X-AuditID: 1209190f-b7f8a6d000000914-bb-4f87098b16b6 Received: from mailhub-auth-1.mit.edu ( [18.9.21.35]) by dmz-mailsec-scanner-4.mit.edu (Symantec Messaging Gateway) with SMTP id 12.0B.02324.B89078F4; Thu, 12 Apr 2012 12:57:47 -0400 (EDT) Received: from outgoing.mit.edu (OUTGOING-AUTH.MIT.EDU [18.7.22.103]) by mailhub-auth-1.mit.edu (8.13.8/8.9.2) with ESMTP id q3CGvkWV021095; Thu, 12 Apr 2012 12:57:47 -0400 Received: from awakening.csail.mit.edu (awakening.csail.mit.edu [18.26.4.91]) (authenticated bits=0) (User authenticated as amdragon@ATHENA.MIT.EDU) by outgoing.mit.edu (8.13.6/8.12.4) with ESMTP id q3CGviOd024327 (version=TLSv1/SSLv3 cipher=AES256-SHA bits=256 verify=NOT); Thu, 12 Apr 2012 12:57:45 -0400 (EDT) Received: from amthrax by awakening.csail.mit.edu with local (Exim 4.77) (envelope-from ) id 1SINL6-0004IU-2r; Thu, 12 Apr 2012 12:57:44 -0400 Date: Thu, 12 Apr 2012 12:57:44 -0400 From: Austin Clements To: Justus Winter <4winter@informatik.uni-hamburg.de> Subject: Re: [RFC] Split notmuch_database_close into two functions Message-ID: <20120412165744.GF13549@mit.edu> References: <1332291311-28954-1-git-send-email-4winter@informatik.uni-hamburg.de> <20120401032323.GH5949@mit.edu> <20120412090533.2074.78211@thinkbox.jade-hamburg.de> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20120412090533.2074.78211@thinkbox.jade-hamburg.de> User-Agent: Mutt/1.5.21 (2010-09-15) X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFmphleLIzCtJLcpLzFFi42IR4hRV1u3mbPc3uDFZ32J26w8mi+s3ZzI7 MHlMPH+azePZqlvMAUxRXDYpqTmZZalF+nYJXBl/Ln1jLNgpUfH2u1cDY4NIFyMnh4SAicTc tl5mCFtM4sK99WxdjFwcQgL7GCVu3XkI5WxglDhyfSY7hHOSSeLezqVMEM4SRonmnd3sIP0s AqoSl59tYQWx2QQ0JLbtX84IYosImEpsePAArIZZQFri2+9mJhBbWMBJYs2FSWBxXgEdiWln vkINXcUoMW3Tb6iEoMTJmU9YIJq1JG78ewlUxAE2aPk/DpAwp4CjxOOek2B7RQVUJKac3MY2 gVFoFpLuWUi6ZyF0L2BkXsUom5JbpZubmJlTnJqsW5ycmJeXWqRropebWaKXmlK6iREc2JL8 Oxi/HVQ6xCjAwajEw/viWZu/EGtiWXFl7iFGSQ4mJVHe1ezt/kJ8SfkplRmJxRnxRaU5qcWH GCU4mJVEeP88BirnTUmsrEotyodJSXOwKInzqmm98xMSSE8sSc1OTS1ILYLJynBwKEnwigMj WEiwKDU9tSItM6cEIc3EwQkynAdoOC9IDW9xQWJucWY6RP4Uo6KUOO9PDqCEAEgiozQPrheW eF4xigO9IswrBNLOA0xacN2vgAYzAQ3mUwC5urgkESEl1cBo8yDVwjU/fb6HVONxhSCHW58N P7uaFur1GKhI+3WGv9FRCl8WMX8D4w3eax1d2cuLtz2bphCQGNeY7nnHuDLD8ihT96+Ct73z Krg9b/1iOMVhmiK2eOeZRz8LVi399sPnzaNtTZbHLV9+YJgc/y0nI54nTlBp5ZcDbh3njRiv fI503pY8/6gSS3FGoqEWc1FxIgAexVgYFwMAAA== Cc: notmuch@notmuchmail.org X-BeenThere: notmuch@notmuchmail.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: "Use and development of the notmuch mail system." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 12 Apr 2012 16:57:50 -0000 Quoth Justus Winter on Apr 12 at 11:05 am: > Quoting Austin Clements (2012-04-01 05:23:23) > >Quoth Justus Winter on Mar 21 at 1:55 am: > >> I propose to split the function notmuch_database_close into > >> notmuch_database_close and notmuch_database_destroy so that long > >> running processes like alot can close the database while still using > >> data obtained from queries to that database. > > > >Is this actually safe? My understanding of Xapian::Database::close is > >that, once you've closed the database, basically anything can throw a > >Xapian exception. A lot of data is retrieved lazily, both by notmuch > >and by Xapian, so simply having, say, a notmuch_message_t object isn't > >enough to guarantee that you'll be able to get data out of it after > >closing the database. Hence, I don't see how this interface could be > >used correctly. > > I do not know how, but both alot and afew (and occasionally the > notmuch binary) are somehow safely using this interface on my box for > the last three weeks. I see. TL;DR: This isn't safe, but that's okay if we document it. The bug report [0] you pointed to was quite informative. At its core, this is really a memory management issue. To sum up for the record (and to check my own thinking): It sounds like alot is careful not to use any notmuch objects after closing the database. The problem is that, currently, closing the database also talloc_free's it, which recursively free's everything derived from it. Python later GCs the wrapper objects, which *also* try to free their underlying objects, resulting in a double free. Before the change to expose notmuch_database_close, the Python bindings would only talloc_free from destructors. Furthermore, they prevented the library from recursively freeing things at other times by internally maintaining a reverse reference for every library talloc reference (e.g., message is a sub-allocation of query, so the bindings keep a reference from each message to its query to ensure the query doesn't get freed). The ability to explicitly talloc_free the database subverts this mechanism. So, I've come around to thinking that splitting notmuch_database_close and _destroy is okay. It certainly parallels the rest of the API better. However, notmuch_database_close needs a big warning similar to Xapian::Database::close's warning that retrieving information from objects derived from this database may not work after calling close. notmuch_database_close is really a specialty interface, and about the only thing you can guarantee after closing the database is that you can destroy other objects. This is also going to require a SONAME major version bump, as mentioned by others. Which, to be fair, would be a good opportunity to fix some other issues, too, like how notmuch_database_open can't return errors and how notmuch_database_get_directory is broken on read-only databases. The actual bump should be done at release time, but maybe we should drop a note somewhere (NEWS?) so we don't forget. [0] https://github.com/pazz/alot/issues/413