From: Austin Clements <amdragon@MIT.EDU>
To: Mark Walters <markwalters1009@gmail.com>
Cc: notmuch@notmuchmail.org
Subject: Re: [RFC] Split notmuch_database_close into two functions
Date: Wed, 18 Apr 2012 13:54:50 -0400 [thread overview]
Message-ID: <20120418175450.GR13549@mit.edu> (raw)
In-Reply-To: <87mx6a4uls.fsf@qmul.ac.uk>
Quoth Mark Walters on Apr 17 at 9:42 am:
> On Thu, 12 Apr 2012, Austin Clements <amdragon@MIT.EDU> wrote:
> > Quoth Justus Winter on Apr 12 at 11:05 am:
> >> Quoting Austin Clements (2012-04-01 05:23:23)
> >> >Quoth Justus Winter on Mar 21 at 1:55 am:
> >> >> I propose to split the function notmuch_database_close into
> >> >> notmuch_database_close and notmuch_database_destroy so that long
> >> >> running processes like alot can close the database while still using
> >> >> data obtained from queries to that database.
> >> >
> >> >Is this actually safe? My understanding of Xapian::Database::close is
> >> >that, once you've closed the database, basically anything can throw a
> >> >Xapian exception. A lot of data is retrieved lazily, both by notmuch
> >> >and by Xapian, so simply having, say, a notmuch_message_t object isn't
> >> >enough to guarantee that you'll be able to get data out of it after
> >> >closing the database. Hence, I don't see how this interface could be
> >> >used correctly.
> >>
> >> I do not know how, but both alot and afew (and occasionally the
> >> notmuch binary) are somehow safely using this interface on my box for
> >> the last three weeks.
> >
> > I see. TL;DR: This isn't safe, but that's okay if we document it.
> >
> > The bug report [0] you pointed to was quite informative. At its core,
> > this is really a memory management issue. To sum up for the record
> > (and to check my own thinking): It sounds like alot is careful not to
> > use any notmuch objects after closing the database. The problem is
> > that, currently, closing the database also talloc_free's it, which
> > recursively free's everything derived from it. Python later GCs the
> > wrapper objects, which *also* try to free their underlying objects,
> > resulting in a double free.
> >
> > Before the change to expose notmuch_database_close, the Python
> > bindings would only talloc_free from destructors. Furthermore, they
> > prevented the library from recursively freeing things at other times
> > by internally maintaining a reverse reference for every library talloc
> > reference (e.g., message is a sub-allocation of query, so the bindings
> > keep a reference from each message to its query to ensure the query
> > doesn't get freed). The ability to explicitly talloc_free the
> > database subverts this mechanism.
> >
> >
> > So, I've come around to thinking that splitting notmuch_database_close
> > and _destroy is okay. It certainly parallels the rest of the API
> > better. However, notmuch_database_close needs a big warning similar
> > to Xapian::Database::close's warning that retrieving information from
> > objects derived from this database may not work after calling close.
> > notmuch_database_close is really a specialty interface, and about the
> > only thing you can guarantee after closing the database is that you
> > can destroy other objects. This is also going to require a SONAME
> > major version bump, as mentioned by others. Which, to be fair, would
> > be a good opportunity to fix some other issues, too, like how
> > notmuch_database_open can't return errors and how
> > notmuch_database_get_directory is broken on read-only databases. The
> > actual bump should be done at release time, but maybe we should drop a
> > note somewhere (NEWS?) so we don't forget.
>
> Can I just check that there is no way to reopen the Xapian database
> readonly? (I may be using the wrong term: I mean is there a way of
> switching an open read-write database to read-only without losing the
> attached structures/messages/threads etc) If I understand it this would
> be sufficient as it would free the lock, but could be more generally
> useful for long lived notmuch processes.
That would be handy and perfect for this situation, but no (I
double-checked with Olly on IRC, which you probably saw). We might be
able to lobby for this capability if it seems more generally useful.
On the other hand, I think it would probably mix poorly with Xapian's
optimistic snapshot isolation if we tried to use it for anything
non-trivial (combined with real snapshot isolation it would be
awesome).
prev parent reply other threads:[~2012-04-18 17:54 UTC|newest]
Thread overview: 43+ messages / expand[flat|nested] mbox.gz Atom feed top
2012-03-21 0:55 [RFC] Split notmuch_database_close into two functions Justus Winter
2012-03-21 0:55 ` [PATCH 1/7] " Justus Winter
2012-03-31 17:17 ` Mark Walters
2012-03-31 17:29 ` David Bremner
2012-04-16 21:51 ` Justus Winter
2012-04-17 8:37 ` Mark Walters
2012-03-21 0:55 ` [PATCH 2/7] NEWS: Document the notmuch_database_close split Justus Winter
2012-03-21 0:55 ` [PATCH 3/7] Use notmuch_database_destroy instead of notmuch_database_close Justus Winter
2012-03-21 0:55 ` [PATCH 4/7] " Justus Winter
2012-03-21 0:55 ` [PATCH 5/7] go: " Justus Winter
2012-03-21 0:55 ` [PATCH 6/7] ruby: " Justus Winter
2012-03-21 0:55 ` [PATCH 7/7] python: wrap and use notmuch_database_destroy as destructor Justus Winter
2012-04-12 17:02 ` Austin Clements
2012-04-20 13:10 ` Sebastian Spaeth
2012-04-22 12:06 ` Justus Winter
2012-04-22 12:07 ` [PATCH 1/7] Split notmuch_database_close into two functions Justus Winter
2012-04-22 12:07 ` [PATCH 2/7] NEWS: Document the notmuch_database_close split Justus Winter
2012-04-22 15:09 ` Felipe Contreras
2012-04-22 12:07 ` [PATCH 3/7] Use notmuch_database_destroy instead of notmuch_database_close Justus Winter
2012-04-22 12:07 ` [PATCH 4/7] " Justus Winter
2012-04-22 12:07 ` [PATCH 5/7] go: " Justus Winter
2012-04-22 12:07 ` [PATCH 6/7] ruby: " Justus Winter
2012-04-23 12:36 ` Felipe Contreras
2012-04-23 12:49 ` Justus Winter
2012-04-25 13:39 ` Austin Clements
2012-04-22 12:07 ` [PATCH 7/7] python: wrap and use notmuch_database_destroy as destructor Justus Winter
2012-04-22 18:01 ` [PATCH 1/7] Split notmuch_database_close into two functions Austin Clements
2012-04-25 13:20 ` Justus Winter
2012-04-25 13:34 ` Austin Clements
2012-04-28 12:54 ` David Bremner
2012-04-22 18:06 ` Austin Clements
2012-03-21 8:57 ` [RFC] " Patrick Totzke
2012-03-24 9:07 ` Tomi Ollila
2012-03-27 8:19 ` Justus Winter
2012-03-27 8:19 ` [PATCH 1/7] " Justus Winter
2012-04-01 3:23 ` [RFC] " Austin Clements
2012-04-12 9:05 ` Justus Winter
2012-04-12 16:57 ` Austin Clements
2012-04-12 17:19 ` Justus Winter
[not found] ` <20120413083358.13321.66680@megatron>
2012-04-16 21:45 ` Justus Winter
2012-04-17 4:56 ` Tomi Ollila
2012-04-17 8:42 ` Mark Walters
2012-04-18 17:54 ` Austin Clements [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: https://notmuchmail.org/
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20120418175450.GR13549@mit.edu \
--to=amdragon@mit.edu \
--cc=markwalters1009@gmail.com \
--cc=notmuch@notmuchmail.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://yhetil.org/notmuch.git/
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).