notmuch insert: Cannot open Xapian database (0.34.1)

notmuch insert: Cannot open Xapian database (0.34.1)

[Peter J. Jones]

I'm in the process of upgrading from notmuch 0.31.4 to 0.34.1 due to an
OS upgrade (NixOS 21.11).  I have tests that verify my mail
configuration (postfix, dovecot, notmuch, etc.) and they fail with
notmuch 0.34.1.  I've tracked it down to (what I believe to be) two
possible bugs.

First, with 0.31 you could call `notmuch insert` and it would create the
database if it didn't exist.  This appears to no longer work in 0.34:

```
[nix-shell:~/src/rc/mailrc]$ NOTMUCH_CONFIG=$(pwd)/config notmuch config get database.path
/home/pjones/src/rc/mailrc/maildir
[nix-shell:~/src/rc/mailrc]$ NOTMUCH_CONFIG=$(pwd)/config notmuch insert < <(echo foo)
Error: Cannot open database at /home/pjones/src/rc/mailrc/maildir/.notmuch: No such file or directory.
```

I can see in `notmuch.c` that the insert command doesn't have the
`NOTMUCH_COMMAND_DATABASE_CREATE` bit set, so this is the expected
behavior in 0.34 and was probably a bug in 0.31.  However, I would argue
that it's useful to create the database in this situation.

In my tests I bootstrap a new VM with an account that has no mail yet.
Then mail comes in via Dovecot which is handed off to `notmuch insert`,
which fails.  On my end I can fix the test by running `notmuch new`
before Dovecot calls `notmuch insert`.

The second issue is that if the `.notmuch` directory exists, but the
`.notmuch/xapian` directory doesn't, `notmuch new/insert` will fail:

```
[nix-shell:~/src/rc/mailrc]$ NOTMUCH_CONFIG=$(pwd)/config notmuch new
Found 0 total files (that's not much mail).
No new mail.
[nix-shell:~/src/rc/mailrc]$ rm -rf maildir/.notmuch/xapian
[nix-shell:~/src/rc/mailrc]$ NOTMUCH_CONFIG=$(pwd)/config notmuch new
Cannot open Xapian database at /home/pjones/src/rc/mailrc/maildir/.notmuch/xapian: Couldn't stat '/home/pjones/src/rc/mailrc/maildir/.notmuch/xapian'
```

The reason I get into this state is because I use Home Manager to manage
my configuration files and it creates `~/mail/.notmuch` so it can
install hook scripts.  Of course, this only happens on a brand new
setup, like in my test VM where Home Manager sets up my configuration
files first, then notmuch is run to test mail delivery and indexing.

This seemed to work in 0.31 (notmuch would just create a new database).

Would it be okay to change `notmuch_database_create_with_config` to
create a new database if `.notmuch` exists but `.notmuch/xapian`
doesn't?

Thanks!


-- 
Peter Jones

Re: notmuch insert: Cannot open Xapian database (0.34.1)

[David Bremner]

"Peter J. Jones" <pjones@pmade.com> writes:

> I can see in `notmuch.c` that the insert command doesn't have the
> `NOTMUCH_COMMAND_DATABASE_CREATE` bit set, so this is the expected
> behavior in 0.34 and was probably a bug in 0.31.  However, I would argue
> that it's useful to create the database in this situation.

I have no real objection to that happening; I'm not sure if it would
break anything. 

>
> In my tests I bootstrap a new VM with an account that has no mail yet.
> Then mail comes in via Dovecot which is handed off to `notmuch insert`,
> which fails.  On my end I can fix the test by running `notmuch new`
> before Dovecot calls `notmuch insert`.
>
> The second issue is that if the `.notmuch` directory exists, but the
> `.notmuch/xapian` directory doesn't, `notmuch new/insert` will fail:
> files first, then notmuch is run to test mail delivery and indexing.
>
> This seemed to work in 0.31 (notmuch would just create a new database).
>
> Would it be okay to change `notmuch_database_create_with_config` to
> create a new database if `.notmuch` exists but `.notmuch/xapian`
> doesn't?

There is many different options for database location possible in 0.32
that were not possible in 0.31.  It might be a bit trickier to make this
change (without compromising something else) than the previous one.

Perhaps migrating to XDG database locations (see notmuch-config(1) for
details) would be an option for you?

Re: notmuch insert: Cannot open Xapian database (0.34.1)

[Peter J. Jones]

On Sat, Dec 04 2021, David Bremner wrote:
> "Peter J. Jones" <pjones@pmade.com> writes:
>>
>> In my tests I bootstrap a new VM with an account that has no mail yet.
>> Then mail comes in via Dovecot which is handed off to `notmuch insert`,
>> which fails.  On my end I can fix the test by running `notmuch new`
>> before Dovecot calls `notmuch insert`.
>>
>> The second issue is that if the `.notmuch` directory exists, but the
>> `.notmuch/xapian` directory doesn't, `notmuch new/insert` will fail:
>> files first, then notmuch is run to test mail delivery and indexing.
>>
>> This seemed to work in 0.31 (notmuch would just create a new database).
>>
>> Would it be okay to change `notmuch_database_create_with_config` to
>> create a new database if `.notmuch` exists but `.notmuch/xapian`
>> doesn't?
>
> There is many different options for database location possible in 0.32
> that were not possible in 0.31.  It might be a bit trickier to make this
> change (without compromising something else) than the previous one.
>
> Perhaps migrating to XDG database locations (see notmuch-config(1) for
> details) would be an option for you?

Thanks for the pointer.  I was able to resolve this issue by moving my
hook scripts out of `database.path` thanks to `database.hook_dir`.

-- 
Peter Jones