automatic database upgrades (was: Re: [PATCH 0/7] cli: notmuch new improvements)

[Jani Nikula <jani@nikula.org>]

On Sat, 25 Jan 2014, Mark Walters <markwalters1009@gmail.com> wrote:
> This series LGTM.

Hi Mark, thanks for the review!

> I do now recall there was some discussion on irc about the automatic
> database upgrade: it would be good to have that documented but the
> consensus was to do it, so +1 from me.

Here's some summary, as promised. Please bear in mind that the
discussion is pretty much irrelevant to this particular patch
series. (We might discuss whether a warning about upgrade should be
printed to stderr also with --quiet, but IMHO that can be a follow-up
patch.)

A database upgrade is required when the user upgrades to a new version
of notmuch that has a modified database schema. See
id:cover.1389304779.git.jani@nikula.org for an example of a proposed
database change.

A database upgrade is a rare event. Most of the time, it's okay to go
back and forth with notmuch versions on the same database, but a
database upgrade is an irreversible process after which the user must
use the new version of notmuch. To go back requires a full rebuild of
the database.

We don't have recent experience with the database upgrades. The last
time it happened was before notmuch 0.1 (yes, 0.1) was released, when
the whole upgrade mechanism was introduced:

commit 909f52bd8c4bdfa11cd3e75e3d0959e0293689bd
Author: Carl Worth <cworth@cworth.org>
Date:   Thu Jan 7 18:26:31 2010 -0800

    lib: Implement versioning in the database and provide upgrade function.

Some of the points in favor of requiring manual intervention (such as
running 'notmuch new --upgrade' or a new command 'notmuch upgrade')
before upgrading the database:

* The database upgrade is an irreversible process. The user should have
  a chance to decide whether to go ahead with that. You can't go back to
  the old notmuch and database version without rebuilding the database.

* The user should be given the chance to make backups first in case
  something goes wrong.

* The database upgrade might take a long time for large databases. The
  user should be able to choose when to go ahead with that.

Some of the points in favor of upgrading automatically:

* cworth: "One potential concern is that [requiring manual intervention]
  effectively breaks notmuch until the user intervenese and runs this
  new command. So that can complicate things for any interface that sits
  on top of notmuch."

* cworth: "In general, I'm often frustrated with programs that say "I
  cannot continue until you run command <foo>.". If command <foo> needs
  to be run, and the software knows it, why doesn't it just run it
  itself? [...] So a message like "Run 'notmuch upgrade'" seems it could
  corrode the user's trust in notmuch to maintain its own state."

* There are people who run notmuch new non-interactively. There's no
  easy answer to handling that if manual intervention is required.

* It should always be okay to kill the upgrade, and continue at a later
  time, in case it takes too long.

Reading the logs again, I'm not so confident about us reaching a
concensus. Maybe it was just me changing my mind during the course of
the discussion... so we can try to reach concensus here.


BR,
Jani.