Breakage after updating to 0.32 (database and path issues)

Breakage after updating to 0.32 (database and path issues)

[Jack Kamm]

Updating to 0.32 broke my notmuch setup. notmuch could no longer find my
database; after I changed some paths in my config, notmuch appeared to
be able to find the database again, but wound up rebuilding it anyways,
and could no longer find any of my emails.

Any advice on fixing my setup would be appreciated. Also, I would
suggest adding information to the v0.32 release notes of any breaking
changes that were made, and how to migrate existing setups.

I'm on Archlinux, and my email is located in $HOME/mail, with each of my
accounts located under a different subfolder (e.g. $HOME/mail/gmail,
$HOME/mail/work, etc).

Below are details of the problems I ran into after updating to 0.32:

First, notmuch could no longer find my database at $HOME/mail/.notmuch:

> $ notmuch new
> Error: Cannot open database at /home/jack/.local/share/notmuch/default: No such file or directory.

According to the manpage of notmuch-config, I should be able to fix
this by unsetting database.mail_root and leaving database.path to
$HOME/mail:

> database.path
>        Notmuch will store its database here,  (in  sub-directory  named
>        .notmuch if database.mail_root is unset).
>
>        Default: $MAILDIR variable if set, otherwise $HOME/mail.
>
> database.mail_root
>        The top-level directory where your mail currently exists and to where mail will be delivered in the future. Files should be individual email messages.
> 
>        History: this configuration value was introduced in notmuch 0.32.
> 
>        Default: For compatibility with older configurations, the value of database.path is used if database.mail_root is unset.

But contrary to the manpage this didn't fix the problem:

> $ notmuch config set database.mail_root ""
> $ notmuch new                                                                                                                                              
> Error: Cannot open database at /home/jack/.local/share/notmuch/default: No such file or directory.

At this point I manually set database.path, but I got a new error this time:

> $ notmuch config set database.path $HOME/mail/.notmuch
> $ notmuch new                                                         
> Warning: failed to open directory : No such file or directory                 
> Found 0 total files (that's not much mail).                                 
> Error reading directory : No such file or directory                         
> No new mail.                                                                         
> Note: A fatal error was encountered: Something went wrong trying to read or write a file

Then I removed the setting for database.mail_root in my
.notmuch-config, at which point "notmuch new" was able to
run. However, it seems to have decided to rebuild the whole database,
and now notmuch-show is not finding any emails.

> notmuch new                                                                                                                                              
> Note: Ignoring non-mail file: /home/jack/mail/.notmuch/xapian/flintlock                                                                                                   
> Note: Ignoring non-mail file: /home/jack/mail/.notmuch/xapian/iamchert                                                                                                    
> Note: Ignoring non-mail file: /home/jack/mail/.notmuch/xapian/position.DB                                                                                                 
> Note: Ignoring non-mail file: /home/jack/mail/.notmuch/xapian/postlist.DB                                                                                                 
> Note: Ignoring non-mail file: /home/jack/mail/.notmuch/xapian/record.DB                                                                                                   
> Note: Ignoring non-mail file: /home/jack/mail/.notmuch/xapian/termlist.DB                                                                                                 
> Note: Ignoring non-mail file: /home/jack/mail/.notmuch/xapian/postlist.baseB                                                                                              
> Note: Ignoring non-mail file: /home/jack/mail/.notmuch/xapian/position.baseA                                                                                              
> Note: Ignoring non-mail file: /home/jack/mail/.notmuch/xapian/termlist.baseB                                                                                              
> Note: Ignoring non-mail file: /home/jack/mail/.notmuch/xapian/record.baseB                                                                                                
> Note: Ignoring non-mail file: /home/jack/mail/.notmuch/xapian/postlist.baseA                                                                                              
> Note: Ignoring non-mail file: /home/jack/mail/.notmuch/xapian/position.baseB                                                                                              
> Note: Ignoring non-mail file: /home/jack/mail/.notmuch/xapian/termlist.baseA                                                                                              
> Note: Ignoring non-mail file: /home/jack/mail/.notmuch/xapian/record.baseA                                                                                                
> Note: Ignoring non-mail file: /home/jack/mail/.notmuch/dump-20180526T233634.gz                                                                                            
> Processed 15 total files in 18m 24s (0 files/sec.).                                                                                                                       
> No new mail. Removed 884086 messages. Detected 12226 file renames.

I'm also quite concerned about the "Removed 884086 messages." I'm
guessing it only removed entries from the database, and doesn't seem to
have deleted any files. But I need to double check my afew filters and
mbsync configuration, to make sure no deletions propagate back to my
remote. In the meantime I'm refraining from running mbsync/notmuch until
I can verify that nothing will get deleted.

Re: Breakage after updating to 0.32 (database and path issues)

[David Bremner]

Jack Kamm <jackkamm@gmail.com> writes:

> Updating to 0.32 broke my notmuch setup. notmuch could no longer find my
> database; after I changed some paths in my config, notmuch appeared to
> be able to find the database again, but wound up rebuilding it anyways,
> and could no longer find any of my emails.
>
> Any advice on fixing my setup would be appreciated. Also, I would
> suggest adding information to the v0.32 release notes of any breaking
> changes that were made, and how to migrate existing setups.

There is a bug related to relative values. Maybe that was what you encountered?
There will be a point release soon with a fix for that bug. In the mean
time you can just set database path to an absolute path.

> I'm on Archlinux, and my email is located in $HOME/mail, with each of my
> accounts located under a different subfolder (e.g. $HOME/mail/gmail,
> $HOME/mail/work, etc).
>
> Below are details of the problems I ran into after updating to 0.32:
>
> First, notmuch could no longer find my database at $HOME/mail/.notmuch:
>
>> $ notmuch new
>> Error: Cannot open database at /home/jack/.local/share/notmuch/default: No such file or directory.

You should not have to change your configuration to work with 0.32,
other than temporarily making sure database.path has an absolute value.
If you have a configuration file that works with 0.31 and not with 0.32,
please share it as precisely as you can, including the contents of
.notmuch-config. 

>
> At this point I manually set database.path, but I got a new error this time:
>
>> $ notmuch config set database.path $HOME/mail/.notmuch
>> $ notmuch new                                                         

Note that the ".notmuch" should not be included in database.path. 

                                                
>> No new mail. Removed 884086 messages. Detected 12226 file renames.
>
> I'm also quite concerned about the "Removed 884086 messages." I'm
> guessing it only removed entries from the database, and doesn't seem to
> have deleted any files.

That's correct. Notmuch does not delete mail files.

> But I need to double check my afew filters and
> mbsync configuration, to make sure no deletions propagate back to my
> remote. In the meantime I'm refraining from running mbsync/notmuch until
> I can verify that nothing will get deleted.

Fair enough, it pays to be cautious. 

Re: Breakage after updating to 0.32 (database and path issues)

[Jack Kamm]

[-- Attachment #1: Type: text/plain, Size: 1092 bytes --]

David Bremner <david@tethera.net> writes:

> There is a bug related to relative values. Maybe that was what you encountered?
> There will be a point release soon with a fix for that bug. In the mean
> time you can just set database path to an absolute path.

Explicitly setting "database.path" to "/home/jack/mail" in my
".notmuch-config" indeed fixed the problem.

However previously, I simply had not set this variable at all in my
".notmuch-config", and "notmuch config get database.path" returned
"/home/jack/mail" (an absolute path).

So, I'm unsure if this was the underlying issue.

> You should not have to change your configuration to work with 0.32,
> other than temporarily making sure database.path has an absolute value.
> If you have a configuration file that works with 0.31 and not with 0.32,
> please share it as precisely as you can, including the contents of
> .notmuch-config. 

I've attached my .notmuch-config (slightly edited to remove some email addresses).

> That's correct. Notmuch does not delete mail files.

That's what I thought -- thanks for confirming this.


[-- Attachment #2: notmuch-config-for-bug-report.conf --]
[-- Type: text/plain, Size: 2897 bytes --]

# -*- mode: conf; -*-
# .notmuch-config - Configuration file for the notmuch mail system
#
# For more information about notmuch, see https://notmuchmail.org

# Database configuration
#
# The only value supported here is 'path' which should be the top-level
# directory where your mail currently exists and to where mail will be
# delivered in the future. Files should be individual email messages.
# Notmuch will store its database within a sub-directory of the path
# configured here named ".notmuch".
#

#[database]
#path=/home/jack/Maildir

# User configuration
#
# Here is where you can let notmuch know how you would like to be
# addressed. Valid settings are
#
#	name		Your full name.
#	primary_email	Your primary email address.
#	other_email	A list (separated by ';') of other email addresses
#			at which you receive email.
#
# Notmuch will use the various email addresses configured here when
# formatting replies. It will avoid including your own addresses in the
# recipient list of replies, and will set the From address based on the
# address to which the original email was addressed.
#

[user]
name=Jack Kamm
primary_email=jackkamm@gmail.com
other_email=xxx@xxx;xxx@xxx;xxx@xxx

# Configuration for "notmuch new"
#
# The following options are supported here:
#
#	tags	A list (separated by ';') of the tags that will be
#		added to all messages incorporated by "notmuch new".
#
#	ignore	A list (separated by ';') of file and directory names
#		that will not be searched for messages by "notmuch new".
#
#		NOTE: *Every* file/directory that goes by one of those
#		names will be ignored, independent of its depth/location
#		in the mail store.
#

[new]
tags=new;
ignore=.uidvalidity;.mbsyncstate

# Search configuration
#
# The following option is supported here:
#
#	exclude_tags
#		A ;-separated list of tags that will be excluded from
#		search results by default.  Using an excluded tag in a
#		query will override that exclusion.
#

[search]
exclude_tags=deleted;spam;

# Maildir compatibility configuration
#
# The following option is supported here:
#
#	synchronize_flags      Valid values are true and false.
#
#	If true, then the following maildir flags (in message filenames)
#	will be synchronized with the corresponding notmuch tags:
#
#		Flag	Tag
#		----	-------
#		D	draft
#		F	flagged
#		P	passed
#		R	replied
#		S	unread (added when 'S' flag is not present)
#
#	The "notmuch new" command will notice flag changes in filenames
#	and update tags, while the "notmuch tag" and "notmuch restore"
#	commands will notice tag changes and update flags in filenames
#

[maildir]
synchronize_flags=true

# Cryptography related configuration
#
# The following *deprecated* option is currently supported:
#
#	gpg_path
#		binary name or full path to invoke gpg.
#		NOTE: In a future build, this option will be ignored.
#		Setting $PATH is a better approach.
#

[crypto]
gpg_path=gpg

[-- Attachment #3: Type: text/plain, Size: 0 bytes --]

Re: Breakage after updating to 0.32 (database and path issues)

[Jack Kamm]

Update: I was able to fix my problems by explicitly setting "database.path" to "$HOME/mail" in my .notmuch-config. Then, notmuch was able to find my emails in my "$HOME/mail" as well as my database at "$HOME/mail/.notmuch".

Confusingly, if "database.path" wasn't explicitly set in .notmuch-config, then "notmuch config get database.path" still returned "$HOME/mail", but notmuch cant' find the database, instead looking in ".local/share/notmuch/default". Is this a bug?

Also, it seems like a bug that notmuch was unable to find any of my email when I had previously set "database.path" to "$HOME/mail/.notmuch". It seems like it was looking for the mail in "database.path" when it should have been looking in "database.mail_root".

Finally, the manpage of notmuch-config(1) seems to be incorrect when it says that database.path will use a ".notmuch" subdirectory when "database.mail_root" is unset. In particular, "notmuch config get database.mail_root" returns "$HOME/mail", however "database.path" is still using a ".notmuch" subdirectory.

Re: Breakage after updating to 0.32 (database and path issues)

[Jack Kamm]

Jack Kamm <jackkamm@gmail.com> writes:

> Confusingly, if "database.path" wasn't explicitly set in .notmuch-config, then "notmuch config get database.path" still returned "$HOME/mail", but notmuch cant' find the database, instead looking in ".local/share/notmuch/default". Is this a bug?

I'm no longer able to reproduce this -- when I comment out the
database.path from my .notmuch-config, notmuch continues to work
correctly now.

Also, I'm 90% sure that "notmuch config get database.path" was returning
"/home/jack/mail" when I was having problems after updating, but am
unable to confirm this anymore.

Anyways, glad that my email is working again, and thank you David for
your fast response on a Sunday, and for all your work on notmuch.

Best,
Jack

[PATCH] lib/open: restore default database path of $HOME/mail

[David Bremner]

Although this default worked for "notmuch config get", it didn't work
most other places. Restore the previous functionality, with the
wrinkle that XDG locations will shadow $HOME/mail if they exist.

This fixes a bug reported by Jack Kamm in id:87eeefdc8b.fsf@gmail.com
---
 lib/open.cc              | 19 ++++++++++++++++++-
 test/T055-path-config.sh | 22 ++++++++++++++++++++--
 test/T560-lib-error.sh   |  4 ++--
 test/T590-libconfig.sh   |  4 ++--
 4 files changed, 42 insertions(+), 7 deletions(-)

diff --git a/lib/open.cc b/lib/open.cc
index bdb695fe..d7d2bedb 100644
--- a/lib/open.cc
+++ b/lib/open.cc
@@ -209,8 +209,25 @@ _choose_database_path (void *ctx,
 	}
     }
     if (! *database_path) {
+	notmuch_status_t status;
+
 	*database_path = _xdg_dir (ctx, "XDG_DATA_HOME", ".local/share", profile);
-	*split = true;
+	status = _db_dir_exists (*database_path, message);
+	if (status) {
+	    *database_path = NULL;
+	} else {
+	    *split = true;
+	}
+    }
+
+    if (! *database_path) {
+	notmuch_status_t status;
+
+	*database_path = talloc_asprintf (ctx, "%s/mail", getenv ("HOME"));
+	status = _db_dir_exists (*database_path, message);
+	if (status) {
+	    *database_path = NULL;
+	}
     }
 
     if (*database_path == NULL) {
diff --git a/test/T055-path-config.sh b/test/T055-path-config.sh
index 2045a555..8ef76aed 100755
--- a/test/T055-path-config.sh
+++ b/test/T055-path-config.sh
@@ -16,6 +16,7 @@ restore_config () {
     unset DATABASE_PATH
     unset NOTMUCH_PROFILE
     unset XAPIAN_PATH
+    rm -f "$HOME/mail"
     cp notmuch-config-backup.${test_name} ${NOTMUCH_CONFIG}
 }
 
@@ -42,6 +43,18 @@ symlink_config () {
     unset DATABASE_PATH
 }
 
+
+home_mail_config () {
+    local dir
+    backup_config
+    dir="${HOME}/mail"
+    ln -s $MAIL_DIR $dir
+    notmuch config set database.path
+    notmuch config set database.mail_root
+    XAPIAN_PATH="$MAIL_DIR/.notmuch/xapian"
+    unset DATABASE_PATH
+}
+
 xdg_config () {
     local dir
     local profile=${1:-default}
@@ -66,7 +79,7 @@ xdg_config () {
     notmuch --config=${CONFIG_PATH} config set database.path
 }
 
-for config in traditional split XDG XDG+profile symlink; do
+for config in traditional split XDG XDG+profile symlink home_mail; do
     #start each set of tests with an known set of messages
     add_email_corpus
 
@@ -90,6 +103,9 @@ for config in traditional split XDG XDG+profile symlink; do
 	symlink)
 	    symlink_config
 	    ;;
+	home_mail)
+	    home_mail_config
+	    ;;
     esac
 
     test_begin_subtest "count ($config)"
@@ -236,7 +252,9 @@ EOF
    test_begin_subtest "Config list ($config)"
    notmuch config list | notmuch_dir_sanitize | sed -e "s/^database.backup_dir=.*$/database.backup_dir/"  \
 						    -e "s/^database.hook_dir=.*$/database.hook_dir/" \
-						    -e "s/^database.path=.*$/database.path/" > OUTPUT
+						    -e "s/^database.path=.*$/database.path/"  \
+						    -e "s,^database.mail_root=CWD/home/mail,database.mail_root=MAIL_DIR," \
+						    > OUTPUT
    cat <<EOF > EXPECTED
 built_with.compact=true
 built_with.field_processor=true
diff --git a/test/T560-lib-error.sh b/test/T560-lib-error.sh
index 89447e9a..1f4482cb 100755
--- a/test/T560-lib-error.sh
+++ b/test/T560-lib-error.sh
@@ -22,7 +22,7 @@ EOF
 cat <<'EOF' >EXPECTED
 == stdout ==
 == stderr ==
-Error: Cannot open database at CWD/home/.local/share/notmuch/default: No such file or directory.
+Error: could not locate database.
 EOF
 test_expect_equal_file EXPECTED OUTPUT
 
@@ -93,7 +93,7 @@ EOF
 cat <<'EOF' >EXPECTED
 == stdout ==
 == stderr ==
-Error: Cannot open database at CWD/home/.local/share/notmuch/default: No such file or directory.
+Error: could not locate database.
 EOF
 test_expect_equal_file EXPECTED OUTPUT
 
diff --git a/test/T590-libconfig.sh b/test/T590-libconfig.sh
index 51dd29c8..745e1bb4 100755
--- a/test/T590-libconfig.sh
+++ b/test/T590-libconfig.sh
@@ -546,8 +546,8 @@ cat <<'EOF' >EXPECTED
 == stdout ==
 == stderr ==
 error opening database
-Something went wrong trying to read or write a file
-Error: Cannot open database at CWD/home/.local/share/notmuch/default: No such file or directory.
+No database found
+Error: could not locate database.
 
 EOF
 test_expect_equal_file EXPECTED OUTPUT
-- 
2.30.2

Re: [PATCH] lib/open: restore default database path of $HOME/mail

[David Bremner]

David Bremner <david@tethera.net> writes:

> Although this default worked for "notmuch config get", it didn't work
> most other places. Restore the previous functionality, with the
> wrinkle that XDG locations will shadow $HOME/mail if they exist.

I have applied this patch to master and release

Re: Breakage after updating to 0.32 (database and path issues)

[David Bremner]

Jack Kamm <jackkamm@gmail.com> writes:

> Update: I was able to fix my problems by explicitly setting "database.path" to "$HOME/mail" in my .notmuch-config. Then, notmuch was able to find my emails in my "$HOME/mail" as well as my database at "$HOME/mail/.notmuch".
>
> Confusingly, if "database.path" wasn't explicitly set in
> .notmuch-config, then "notmuch config get database.path" still
> returned "$HOME/mail", but notmuch cant' find the database, instead
> looking in ".local/share/notmuch/default". Is this a bug?

It's a bit surprising that you can't reproduce this anymore, because I
think there really was a bug here. I have sent a patch as a followup,

> Also, it seems like a bug that notmuch was unable to find any of my
>email when I had previously set "database.path" to
>"$HOME/mail/.notmuch". It seems like it was looking for the mail in
>"database.path" when it should have been looking in
>"database.mail_root".

Notmuch has to go through some contortions to support both nested
(.notmuch inside mail dir) and split configurations, and we may just
have to call this one unsupported. It didn't work before, so there is no
loss of functionality here.

> Finally, the manpage of notmuch-config(1) seems to be incorrect when
> it says that database.path will use a ".notmuch" subdirectory when
> "database.mail_root" is unset. In particular, "notmuch config get
> database.mail_root" returns "$HOME/mail", however "database.path" is
> still using a ".notmuch" subdirectory.

Sorry, I don't follow this. I don't _think_ there's an actual error
here, but if someone wants to propose patches that
make that part of the documentation clearer, I'd be happy to take a look
at them (although probably not for the point release, which I want to
get out the door).

Re: Breakage after updating to 0.32 (database and path issues)

[Jack Kamm]

Hi David,

David Bremner <david@tethera.net> writes:

> Jack Kamm <jackkamm@gmail.com> writes:
>
>> Update: I was able to fix my problems by explicitly setting "database.path" to "$HOME/mail" in my .notmuch-config. Then, notmuch was able to find my emails in my "$HOME/mail" as well as my database at "$HOME/mail/.notmuch".
>>
>> Confusingly, if "database.path" wasn't explicitly set in
>> .notmuch-config, then "notmuch config get database.path" still
>> returned "$HOME/mail", but notmuch cant' find the database, instead
>> looking in ".local/share/notmuch/default". Is this a bug?
>
> It's a bit surprising that you can't reproduce this anymore, because I
> think there really was a bug here. I have sent a patch as a followup,

Is it possible that at this point the config variable had been set in
the database, so it remained set even though I commented it from my
dotfile, hence the inability to reproduce?

Just a wild guess, as I'm not sure how the database and config file
interact with each other.

I don't want to mess further with my email at the moment, so not going
to do more testing on this right now. But I'll keep an eye on it and
update if I notice anything wrong.

Thanks again, and sorry for the delayed response.

Best,
Jack