From: Austin Clements <amdragon@mit.edu>
To: Carl Worth <cworth@cworth.org>
Cc: notmuch@notmuchmail.org
Subject: Re: [PATCH v6 00/17] Fix 'notmuch new' atomicity issues
Date: Wed, 29 Jun 2011 02:37:00 -0400 [thread overview]
Message-ID: <CAH-f9Wt9ZH0rMsaCK1bMfsccTGhYvcBBAAH=dSL5ps+KhyHtwA@mail.gmail.com> (raw)
In-Reply-To: <BANLkTimeWnn5YtKjarFB8f8Fowvz0PxK2Q@mail.gmail.com>
Just found one more atomicity bug in notmuch-new. I would prefer to
not update this patch series yet again, but rather to follow up with a
separate fix for this. The full bug is described below, but the gist
is that how add_files_recursive computes new_directory from the mtime
isn't sound. The simplest fix is to remove the new_directory
optimization, but then we wouldn't scan files in inode order. The
best fix is probably to add an out-argument to
notmuch_database_get_directory that indicates if the directory really
was just created in the database (and hence has no files or subdirs).
Unfortunately, that requires an API change. If that's undesirable, an
alternate would be to record that bit in the notmuch_directory_t and
add some notmuch_directory_is_new API that returns it. Thoughts?
Bug description:
add_files_recursive determines whether a directory is "new" by
inspecting the database mtime returned by notmuch_directory_get_mtime.
However, if there's an interruption after adding
messages/subdirectories from a new directory but before updating the
directory's mtime, it will be considered a new directory again on the
next run. As a result, on the next run, db_files and db_subdirs will
not be loaded from the database (even though they *wouldn't* be NULL).
As a result, we'll miss removing messages or entire subdirectories
that have been deleted from the "new" directory. (We'll also re-add
the messages in this directory to the database rather than skipping
them, which will throw off the notmuch new statistics, but that's
fine.)
This didn't show up in the atomicity test because throwing off
anything besides the statistics requires *two* interruptions. In
fact, I don't even know how I would write an automated test for this.
next prev parent reply other threads:[~2011-06-29 6:37 UTC|newest]
Thread overview: 51+ messages / expand[flat|nested] mbox.gz Atom feed top
2011-02-18 7:58 [PATCH 00/10] Fix 'notmuch new' atomicity issues Austin Clements
2011-02-18 7:58 ` [PATCH 01/10] test: Test atomicity of notmuch new Austin Clements
2011-05-04 20:32 ` [PATCH 01/10 v2] " Austin Clements
2011-02-18 7:58 ` [PATCH 02/10] new: Don't loose messages on SIGINT Austin Clements
2011-02-18 7:58 ` [PATCH 03/10] new: Defer updating directory mtimes until the end Austin Clements
2011-02-18 7:58 ` [PATCH 04/10] lib: Make _notmuch_message_sync capable of deleting a message Austin Clements
2011-02-18 7:58 ` [PATCH 05/10] lib: Indicate if there are more filenames after removal Austin Clements
2011-02-18 7:58 ` [PATCH 06/10] lib: Add API's to find by filename and remove a filename from a message Austin Clements
2011-02-18 7:58 ` [PATCH 07/10] new: Use the new filename removal API Austin Clements
2011-02-18 7:58 ` [PATCH 08/10] new: Synchronize maildir flags eagerly Austin Clements
2011-02-18 7:58 ` [PATCH 09/10] lib: Add notmuch_database_{begin,end}_atomic Austin Clements
2011-02-18 7:59 ` [PATCH 10/10] new: Wrap adding a message in an atomic section Austin Clements
2011-04-26 4:13 ` [PATCH 00/10] Fix 'notmuch new' atomicity issues Austin Clements
2011-05-04 20:30 ` Austin Clements
2011-05-29 2:51 ` Austin Clements
2011-06-08 22:05 ` Carl Worth
2011-06-10 21:11 ` Austin Clements
2011-06-10 22:02 ` Carl Worth
2011-06-10 22:27 ` Austin Clements
2011-06-11 20:04 ` [PATCH v6 00/17] " Austin Clements
2011-06-11 20:04 ` [PATCH 02/17] test: Report test failures from test_expect_* Austin Clements
2011-06-11 20:04 ` [PATCH 03/17] lib: Add missing status check in _notmuch_message_remove_filename Austin Clements
2011-06-11 20:04 ` [PATCH 04/17] test: Test atomicity of notmuch new Austin Clements
2011-06-11 20:04 ` [PATCH 05/17] new: Don't lose messages on SIGINT Austin Clements
2011-06-11 20:04 ` [PATCH 06/17] new: Defer updating directory mtimes until the end Austin Clements
2011-06-11 20:04 ` [PATCH 07/17] lib: Add notmuch_database_{begin,end}_atomic Austin Clements
2011-06-11 20:04 ` [PATCH 08/17] lib: Add support for nested atomic sections Austin Clements
2011-06-11 20:04 ` [PATCH 09/17] lib: Indicate if there are more filenames after removal Austin Clements
2011-06-11 20:04 ` [PATCH 10/17] lib: Remove message document directly after removing the last file name Austin Clements
2011-06-11 20:04 ` [PATCH 11/17] lib: Add an API to find a message by filename Austin Clements
2011-06-11 20:04 ` [PATCH 12/17] lib: Wrap notmuch_database_add_message in an atomic section Austin Clements
2011-06-11 20:04 ` [PATCH 13/17] new: Cleanup. Put removed/renamed message count in add_files_state_t Austin Clements
2011-06-11 20:04 ` [PATCH 14/17] new: Cleanup. De-duplicate file name removal code Austin Clements
2011-06-11 20:04 ` [PATCH 15/17] new: Synchronize maildir flags eagerly Austin Clements
2011-06-11 20:04 ` [PATCH 16/17] new: Wrap adding and removing messages in atomic sections Austin Clements
2011-06-11 20:04 ` [PATCH 17/17] lib: Improve notmuch_database_{add, remove}_message documentation Austin Clements
2011-06-11 20:49 ` [PATCH 01/17] test: Fix message when skipping test_expect_equal* tests Austin Clements
2011-06-11 20:53 ` [PATCH v6 00/17] Fix 'notmuch new' atomicity issues Austin Clements
2011-06-22 19:39 ` Austin Clements
2011-06-29 6:37 ` Austin Clements [this message]
2011-07-08 18:09 ` Austin Clements
2011-09-13 12:39 ` David Bremner
2011-09-24 3:14 ` David Bremner
2011-09-24 4:03 ` Austin Clements
2011-09-25 2:36 ` David Bremner
2011-09-26 22:07 ` Austin Clements
2011-09-26 23:32 ` NEWS for 0.9 David Bremner
2011-09-27 13:08 ` Ali Polatel
2011-09-28 16:36 ` [PATCH v6 00/17] Fix 'notmuch new' atomicity issues Sebastian Spaeth
2011-09-29 15:01 ` Austin Clements
2011-09-30 9:21 ` Sebastian Spaeth
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: https://notmuchmail.org/
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to='CAH-f9Wt9ZH0rMsaCK1bMfsccTGhYvcBBAAH=dSL5ps+KhyHtwA@mail.gmail.com' \
--to=amdragon@mit.edu \
--cc=cworth@cworth.org \
--cc=notmuch@notmuchmail.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://yhetil.org/notmuch.git/
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).