[kisara.moe] notmuch doesn't respect MAILDIR env for database path

[kisara.moe] notmuch doesn't respect MAILDIR env for database path

[mohsin kaleem]

notmuch doesn't seem to use the value of the $MAILDIR option when
assigning `database.path`.

On a new install from notmuch version 0.32 I ran `notmuch` to initialise
a config and then `notmuch new` which produces the output

```
Found 4 total files (that's not much mail).
Note: Ignoring non-mail file: /home/mohkale/.local/share/notmuch/default/xapian/flintlock
Note: Ignoring non-mail file: /home/mohkale/.local/share/notmuch/default/xapian/termlist.glass
Note: Ignoring non-mail file: /home/mohkale/.local/share/notmuch/default/xapian/postlist.glass
Note: Ignoring non-mail file: /home/mohkale/.local/share/notmuch/default/xapian/iamglass
Processed 4 total files in almost no time.
No new mail.
```

notmuch seems to default to using ~/.local/share/notmuch for the
`database.path` value instead of MAILDIR like the manual for
notmuch-config states.

S.N. My value of $MAILDIR is `~/docs/mail`. It exists in the maildir
format and has several mail files available already.

look in MAILDIR for database

[David Bremner]

0.32 introduced a regression where notmuch stopped finding the database of people who specified it via Maildir.

The actual code fix is in

     [PATCH 7/8] lib/open: look in MAILDIR for database, as documented.

Also user facing is the updates to documentation in

     [PATCH 8/8] doc: document database search algorithm.

The rest of the series is tweaks to the test framework and some regression tests.

I'm thinking it's probably time to do another feature release, rather
than a point release just for this series.

d

[PATCH 1/8] test/path-config: define config_subtest

[David Bremner]

This will allow adding "hook code" to be run for a particular
configuration, in particular marking all tests in a given
configuration as broken.
---
 test/T055-path-config.sh | 48 ++++++++++++++++++++++++----------------
 1 file changed, 29 insertions(+), 19 deletions(-)

diff --git a/test/T055-path-config.sh b/test/T055-path-config.sh
index bb3bf665..2061b116 100755
--- a/test/T055-path-config.sh
+++ b/test/T055-path-config.sh
@@ -79,6 +79,13 @@ xdg_config () {
     notmuch --config=${CONFIG_PATH} config set database.path
 }
 
+config_subtest () {
+    local title=$1
+    local config=$2
+
+    test_begin_subtest "$title ($config)"
+}
+
 for config in traditional split XDG XDG+profile symlink home_mail; do
     #start each set of tests with an known set of messages
     add_email_corpus
@@ -106,20 +113,23 @@ for config in traditional split XDG XDG+profile symlink home_mail; do
 	home_mail)
 	    home_mail_config
 	    ;;
+	maildir_env)
+	    maildir_env_config
+	    ;;
     esac
 
-    test_begin_subtest "count ($config)"
+    config_subtest "count" "$config"
     output=$(notmuch count '*')
     test_expect_equal "$output" '52'
 
-    test_begin_subtest "count+tag ($config)"
+    config_subtest "count+tag" "$config"
     tag="tag${RANDOM}"
     notmuch tag +$tag '*'
     output=$(notmuch count tag:$tag)
     notmuch tag -$tag '*'
     test_expect_equal "$output" '52'
 
-    test_begin_subtest "address ($config)"
+    config_subtest "address" "$config"
     notmuch address --deduplicate=no --sort=newest-first --output=sender --output=recipients path:foo >OUTPUT
     cat <<EOF >EXPECTED
 Carl Worth <cworth@cworth.org>
@@ -127,7 +137,7 @@ notmuch@notmuchmail.org
 EOF
     test_expect_equal_file EXPECTED OUTPUT
 
-    test_begin_subtest "dump ($config)"
+    config_subtest "dump" "$config"
     notmuch dump is:attachment and is:signed | sort > OUTPUT
     cat <<EOF > EXPECTED
 #notmuch-dump batch-tag:3 config,properties,tags
@@ -136,43 +146,43 @@ EOF
 EOF
     test_expect_equal_file EXPECTED OUTPUT
 
-    test_begin_subtest "dump + tag + restore ($config)"
+    config_subtest "dump + tag + restore" "$config"
     notmuch dump '*' > EXPECTED
     notmuch tag -inbox '*'
     notmuch restore < EXPECTED
     notmuch dump > OUTPUT
     test_expect_equal_file EXPECTED OUTPUT
 
-    test_begin_subtest "reindex ($config)"
+    config_subtest "reindex" "$config"
     notmuch search --output=messages '*' > EXPECTED
     notmuch reindex '*'
     notmuch search --output=messages '*' > OUTPUT
     test_expect_equal_file EXPECTED OUTPUT
 
-    test_begin_subtest "use existing database ($config)"
+    config_subtest "use existing database" "$config"
     output=$(notmuch new)
     test_expect_equal "$output" 'No new mail.'
 
-    test_begin_subtest "create database ($config)"
+    config_subtest "create database" "$config"
     rm -rf $DATABASE_PATH/{.notmuch,}/xapian
     notmuch new
     output=$(notmuch count '*')
     test_expect_equal "$output" '52'
 
-    test_begin_subtest "detect new files ($config)"
+    config_subtest "detect new files" "$config"
     generate_message
     generate_message
     notmuch new
     output=$(notmuch count '*')
     test_expect_equal "$output" '54'
 
-    test_begin_subtest "Show a raw message ($config)"
+    config_subtest "Show a raw message" "$config"
     add_message
     notmuch show --format=raw id:$gen_msg_id > OUTPUT
     test_expect_equal_file $gen_msg_filename OUTPUT
     rm -f $gen_msg_filename
 
-    test_begin_subtest "reply ($config)"
+    config_subtest "reply" "$config"
     add_message '[from]="Sender <sender@example.com>"' \
 		[to]=test_suite@notmuchmail.org \
 		[subject]=notmuch-reply-test \
@@ -190,7 +200,7 @@ On Tue, 05 Jan 2010 15:43:56 -0000, Sender <sender@example.com> wrote:
 > basic reply test
 EOF
     test_expect_equal_file EXPECTED OUTPUT
-    test_begin_subtest "insert+search ($config)"
+    config_subtest "insert+search" "$config"
     generate_message \
 	"[subject]=\"insert-subject\"" \
 	"[date]=\"Sat, 01 Jan 2000 12:00:00 -0000\"" \
@@ -201,13 +211,13 @@ EOF
     test_expect_equal_file "$cur_msg_filename" "$gen_msg_filename"
 
 
-    test_begin_subtest "compact+search ($config)"
+    config_subtest "compact+search" "$config"
     notmuch search --output=messages '*' | sort > EXPECTED
     notmuch compact
     notmuch search --output=messages '*' | sort > OUTPUT
     test_expect_equal_file EXPECTED OUTPUT
 
-    test_begin_subtest "upgrade backup ($config)"
+    config_subtest "upgrade backup" "$config"
     features=$(xapian-metadata get $XAPIAN_PATH features | grep -v "^relative directory paths")
     xapian-metadata set $XAPIAN_PATH features "$features"
     output=$(notmuch new | grep Welcome)
@@ -215,7 +225,7 @@ EOF
 	"$output" \
 	"Welcome to a new version of notmuch! Your database will now be upgraded."
 
-    test_begin_subtest "notmuch +config -database suggests notmuch new ($config)"
+    config_subtest "notmuch +config -database suggests notmuch new" "$config"
     mv "$XAPIAN_PATH" "${XAPIAN_PATH}.bak"
     notmuch > OUTPUT
 cat <<EOF > EXPECTED
@@ -231,7 +241,7 @@ EOF
 
    test_expect_equal_file EXPECTED OUTPUT
 
-   test_begin_subtest "Set config value ($config)"
+   config_subtest "Set config value" "$config"
    name=${RANDOM}
    value=${RANDOM}
    notmuch config set test${test_count}.${name} ${value}
@@ -240,7 +250,7 @@ EOF
    output2=$(notmuch config get test${test_count}.${name})
    test_expect_equal "${output}+${output2}" "${value}+"
 
-   test_begin_subtest "Set config value in database ($config)"
+   config_subtest "Set config value in database" "$config"
    name=${RANDOM}
    value=${RANDOM}
    notmuch config set --database test${test_count}.${name} ${value}
@@ -249,7 +259,7 @@ EOF
    output2=$(notmuch config get test${test_count}.${name})
    test_expect_equal "${output}+${output2}" "${value}+"
 
-   test_begin_subtest "Config list ($config)"
+   config_subtest "Config list" "$config"
    notmuch config list | notmuch_dir_sanitize | sed -e "s/^database.backup_dir=.*$/database.backup_dir/"  \
 						    -e "s/^database.hook_dir=.*$/database.hook_dir/" \
 						    -e "s/^database.path=.*$/database.path/"  \
@@ -276,7 +286,7 @@ EOF
 
    case $config in
        XDG*)
-	   test_begin_subtest "Set shadowed config value in database ($config)"
+	   config_subtest "Set shadowed config value in database" "$config"
 	   name=${RANDOM}
 	   value=${RANDOM}
 	   key=test${test_count}.${name}
-- 
2.30.2

Re: [PATCH 1/8] test/path-config: define config_subtest

[Tomi Ollila]

On Mon, Aug 02 2021, David Bremner wrote:

> This will allow adding "hook code" to be run for a particular
> configuration, in particular marking all tests in a given
> configuration as broken.

uh, see below

> ---
>  test/T055-path-config.sh | 48 ++++++++++++++++++++++++----------------
>  1 file changed, 29 insertions(+), 19 deletions(-)
>
> diff --git a/test/T055-path-config.sh b/test/T055-path-config.sh
> index bb3bf665..2061b116 100755
> --- a/test/T055-path-config.sh
> +++ b/test/T055-path-config.sh
> @@ -79,6 +79,13 @@ xdg_config () {
>      notmuch --config=${CONFIG_PATH} config set database.path
>  }
>  
> +config_subtest () {
> +    local title=$1
> +    local config=$2
> +
> +    test_begin_subtest "$title ($config)"
> +}

here w/o commit message one cannot see the point ;/

Then I looked, in one commit there were this added

      if [[ "$config" == "maildir_env" ]]; then
  	test_subtest_known_broken
      fi

after test_begin_subtest "$title ($config)", and one
config_subtest call was restored to call to test_begin_subtest

finally the above code was removed (but that one test_begin_subtest
was not re-restored to use config_subtest)

i am not too happy about this acrobacy...

btw; that one change there (later in series):

  -    config_subtest "notmuch +config -database suggests notmuch new" "$config"
  +    test_begin_subtest "notmuch +config -database suggests notmuch new" "$config"

... and another...

  -   config_subtest "Set config value" "$config"
  +   test_begin_subtest "Set config value" "$config"

are these correct, the latter was changed originally (in this patch):

  -   test_begin_subtest "Set config value ($config)"
  +   config_subtest "Set config value" "$config"

---

After thinking a while I see the point, I am not sure if this is better,
but I'd do this the following way:

begin_config_subtest () {
    test_begin_subtest "$@"
}

then it is easy to flip between begin_config_subtest and test_begin_subtest
whenever needed -- and in final commit to fix all broken patches remove
all extra code from begin_config_subtest and ensure that no calls to
test_begin_subtest are made in that particular file


Tomi


> +
>  for config in traditional split XDG XDG+profile symlink home_mail; do
>      #start each set of tests with an known set of messages
>      add_email_corpus
> @@ -106,20 +113,23 @@ for config in traditional split XDG XDG+profile symlink home_mail; do
>  	home_mail)
>  	    home_mail_config
>  	    ;;
> +	maildir_env)
> +	    maildir_env_config
> +	    ;;
>      esac
>  
> -    test_begin_subtest "count ($config)"
> +    config_subtest "count" "$config"
>      output=$(notmuch count '*')
>      test_expect_equal "$output" '52'
>  
> -    test_begin_subtest "count+tag ($config)"
> +    config_subtest "count+tag" "$config"
>      tag="tag${RANDOM}"
>      notmuch tag +$tag '*'
>      output=$(notmuch count tag:$tag)
>      notmuch tag -$tag '*'
>      test_expect_equal "$output" '52'
>  
> -    test_begin_subtest "address ($config)"
> +    config_subtest "address" "$config"
>      notmuch address --deduplicate=no --sort=newest-first --output=sender --output=recipients path:foo >OUTPUT
>      cat <<EOF >EXPECTED
>  Carl Worth <cworth@cworth.org>
> @@ -127,7 +137,7 @@ notmuch@notmuchmail.org
>  EOF
>      test_expect_equal_file EXPECTED OUTPUT
>  
> -    test_begin_subtest "dump ($config)"
> +    config_subtest "dump" "$config"
>      notmuch dump is:attachment and is:signed | sort > OUTPUT
>      cat <<EOF > EXPECTED
>  #notmuch-dump batch-tag:3 config,properties,tags
> @@ -136,43 +146,43 @@ EOF
>  EOF
>      test_expect_equal_file EXPECTED OUTPUT
>  
> -    test_begin_subtest "dump + tag + restore ($config)"
> +    config_subtest "dump + tag + restore" "$config"
>      notmuch dump '*' > EXPECTED
>      notmuch tag -inbox '*'
>      notmuch restore < EXPECTED
>      notmuch dump > OUTPUT
>      test_expect_equal_file EXPECTED OUTPUT
>  
> -    test_begin_subtest "reindex ($config)"
> +    config_subtest "reindex" "$config"
>      notmuch search --output=messages '*' > EXPECTED
>      notmuch reindex '*'
>      notmuch search --output=messages '*' > OUTPUT
>      test_expect_equal_file EXPECTED OUTPUT
>  
> -    test_begin_subtest "use existing database ($config)"
> +    config_subtest "use existing database" "$config"
>      output=$(notmuch new)
>      test_expect_equal "$output" 'No new mail.'
>  
> -    test_begin_subtest "create database ($config)"
> +    config_subtest "create database" "$config"
>      rm -rf $DATABASE_PATH/{.notmuch,}/xapian
>      notmuch new
>      output=$(notmuch count '*')
>      test_expect_equal "$output" '52'
>  
> -    test_begin_subtest "detect new files ($config)"
> +    config_subtest "detect new files" "$config"
>      generate_message
>      generate_message
>      notmuch new
>      output=$(notmuch count '*')
>      test_expect_equal "$output" '54'
>  
> -    test_begin_subtest "Show a raw message ($config)"
> +    config_subtest "Show a raw message" "$config"
>      add_message
>      notmuch show --format=raw id:$gen_msg_id > OUTPUT
>      test_expect_equal_file $gen_msg_filename OUTPUT
>      rm -f $gen_msg_filename
>  
> -    test_begin_subtest "reply ($config)"
> +    config_subtest "reply" "$config"
>      add_message '[from]="Sender <sender@example.com>"' \
>  		[to]=test_suite@notmuchmail.org \
>  		[subject]=notmuch-reply-test \
> @@ -190,7 +200,7 @@ On Tue, 05 Jan 2010 15:43:56 -0000, Sender <sender@example.com> wrote:
>  > basic reply test
>  EOF
>      test_expect_equal_file EXPECTED OUTPUT
> -    test_begin_subtest "insert+search ($config)"
> +    config_subtest "insert+search" "$config"
>      generate_message \
>  	"[subject]=\"insert-subject\"" \
>  	"[date]=\"Sat, 01 Jan 2000 12:00:00 -0000\"" \
> @@ -201,13 +211,13 @@ EOF
>      test_expect_equal_file "$cur_msg_filename" "$gen_msg_filename"
>  
>  
> -    test_begin_subtest "compact+search ($config)"
> +    config_subtest "compact+search" "$config"
>      notmuch search --output=messages '*' | sort > EXPECTED
>      notmuch compact
>      notmuch search --output=messages '*' | sort > OUTPUT
>      test_expect_equal_file EXPECTED OUTPUT
>  
> -    test_begin_subtest "upgrade backup ($config)"
> +    config_subtest "upgrade backup" "$config"
>      features=$(xapian-metadata get $XAPIAN_PATH features | grep -v "^relative directory paths")
>      xapian-metadata set $XAPIAN_PATH features "$features"
>      output=$(notmuch new | grep Welcome)
> @@ -215,7 +225,7 @@ EOF
>  	"$output" \
>  	"Welcome to a new version of notmuch! Your database will now be upgraded."
>  
> -    test_begin_subtest "notmuch +config -database suggests notmuch new ($config)"
> +    config_subtest "notmuch +config -database suggests notmuch new" "$config"
>      mv "$XAPIAN_PATH" "${XAPIAN_PATH}.bak"
>      notmuch > OUTPUT
>  cat <<EOF > EXPECTED
> @@ -231,7 +241,7 @@ EOF
>  
>     test_expect_equal_file EXPECTED OUTPUT
>  
> -   test_begin_subtest "Set config value ($config)"
> +   config_subtest "Set config value" "$config"
>     name=${RANDOM}
>     value=${RANDOM}
>     notmuch config set test${test_count}.${name} ${value}
> @@ -240,7 +250,7 @@ EOF
>     output2=$(notmuch config get test${test_count}.${name})
>     test_expect_equal "${output}+${output2}" "${value}+"
>  
> -   test_begin_subtest "Set config value in database ($config)"
> +   config_subtest "Set config value in database" "$config"
>     name=${RANDOM}
>     value=${RANDOM}
>     notmuch config set --database test${test_count}.${name} ${value}
> @@ -249,7 +259,7 @@ EOF
>     output2=$(notmuch config get test${test_count}.${name})
>     test_expect_equal "${output}+${output2}" "${value}+"
>  
> -   test_begin_subtest "Config list ($config)"
> +   config_subtest "Config list" "$config"
>     notmuch config list | notmuch_dir_sanitize | sed -e "s/^database.backup_dir=.*$/database.backup_dir/"  \
>  						    -e "s/^database.hook_dir=.*$/database.hook_dir/" \
>  						    -e "s/^database.path=.*$/database.path/"  \
> @@ -276,7 +286,7 @@ EOF
>  
>     case $config in
>         XDG*)
> -	   test_begin_subtest "Set shadowed config value in database ($config)"
> +	   config_subtest "Set shadowed config value in database" "$config"
>  	   name=${RANDOM}
>  	   value=${RANDOM}
>  	   key=test${test_count}.${name}
> -- 
> 2.30.2
> _______________________________________________
> notmuch mailing list -- notmuch@notmuchmail.org
> To unsubscribe send an email to notmuch-leave@notmuchmail.org

[PATCH 2/8] test/expect_equal_file: whitespace cleanup

[David Bremner]

No functional change, just reindent to Emacs defaults.
---
 test/test-lib.sh | 42 +++++++++++++++++++++---------------------
 1 file changed, 21 insertions(+), 21 deletions(-)

diff --git a/test/test-lib.sh b/test/test-lib.sh
index 67ad8853..10ee9b72 100644
--- a/test/test-lib.sh
+++ b/test/test-lib.sh
@@ -404,29 +404,29 @@ test_expect_equal () {
 
 # Like test_expect_equal, but takes two filenames.
 test_expect_equal_file () {
-	local file1 file2 testname basename1 basename2
-	exec 1>&6 2>&7		# Restore stdout and stderr
-	if [ -z "$inside_subtest" ]; then
-		error "bug in the test script: test_expect_equal_file without test_begin_subtest"
-	fi
-	inside_subtest=
-	test "$#" = 2 ||
+    local file1 file2 testname basename1 basename2
+    exec 1>&6 2>&7		# Restore stdout and stderr
+    if [ -z "$inside_subtest" ]; then
+	error "bug in the test script: test_expect_equal_file without test_begin_subtest"
+    fi
+    inside_subtest=
+    test "$#" = 2 ||
 	error "bug in the test script: not 2 parameters to test_expect_equal_file"
 
-	file1="$1"
-	file2="$2"
-	if ! test_skip "$test_subtest_name"
-	then
-		if diff -q "$file1" "$file2" >/dev/null ; then
-			test_ok_
-		else
-			testname=$this_test.$test_count
-			basename1=`basename "$file1"`
-			basename2=`basename "$file2"`
-			cp "$file1" "$testname.$basename1"
-			cp "$file2" "$testname.$basename2"
-			test_failure_ "$(diff -u "$testname.$basename1" "$testname.$basename2")"
-		fi
+    file1="$1"
+    file2="$2"
+    if ! test_skip "$test_subtest_name"
+    then
+	if diff -q "$file1" "$file2" >/dev/null ; then
+	    test_ok_
+	else
+	    testname=$this_test.$test_count
+	    basename1=`basename "$file1"`
+	    basename2=`basename "$file2"`
+	    cp "$file1" "$testname.$basename1"
+	    cp "$file2" "$testname.$basename2"
+	    test_failure_ "$(diff -u "$testname.$basename1" "$testname.$basename2")"
+	fi
     fi
 }
 
-- 
2.30.2

[PATCH 3/8] test: factor out test_diff_file_

[David Bremner]

A following commit will use this new function in a public test
function.
---
 test/test-lib.sh | 24 ++++++++++++++----------
 1 file changed, 14 insertions(+), 10 deletions(-)

diff --git a/test/test-lib.sh b/test/test-lib.sh
index 10ee9b72..a8e57e51 100644
--- a/test/test-lib.sh
+++ b/test/test-lib.sh
@@ -402,17 +402,8 @@ test_expect_equal () {
     fi
 }
 
-# Like test_expect_equal, but takes two filenames.
-test_expect_equal_file () {
+test_diff_file_ () {
     local file1 file2 testname basename1 basename2
-    exec 1>&6 2>&7		# Restore stdout and stderr
-    if [ -z "$inside_subtest" ]; then
-	error "bug in the test script: test_expect_equal_file without test_begin_subtest"
-    fi
-    inside_subtest=
-    test "$#" = 2 ||
-	error "bug in the test script: not 2 parameters to test_expect_equal_file"
-
     file1="$1"
     file2="$2"
     if ! test_skip "$test_subtest_name"
@@ -430,6 +421,19 @@ test_expect_equal_file () {
     fi
 }
 
+# Like test_expect_equal, but takes two filenames.
+test_expect_equal_file () {
+    exec 1>&6 2>&7		# Restore stdout and stderr
+    if [ -z "$inside_subtest" ]; then
+	error "bug in the test script: test_expect_equal_file without test_begin_subtest"
+    fi
+    inside_subtest=
+    test "$#" = 2 ||
+	error "bug in the test script: not 2 parameters to test_expect_equal_file"
+
+    test_diff_file_ "$1" "$2"
+}
+
 # Like test_expect_equal, but arguments are JSON expressions to be
 # canonicalized before diff'ing.  If an argument cannot be parsed, it
 # is used unchanged so that there's something to diff against.
-- 
2.30.2

[PATCH 4/8] test: add test_expect_equal_file_nonempty

[David Bremner]

A common bug in tests is that the code used to generate the EXPECTED
file fails, generating no output. When the code generating the OUTPUT
file fails in the same way, the test passes, even though there is a
failure being hidden. Add a new test function that guards against
this.
---
 test/test-lib.sh | 19 +++++++++++++++++++
 1 file changed, 19 insertions(+)

diff --git a/test/test-lib.sh b/test/test-lib.sh
index a8e57e51..4865032d 100644
--- a/test/test-lib.sh
+++ b/test/test-lib.sh
@@ -434,6 +434,25 @@ test_expect_equal_file () {
     test_diff_file_ "$1" "$2"
 }
 
+# Like test_expect_equal, but takes two filenames. Fails if either is empty
+test_expect_equal_file_nonempty () {
+    exec 1>&6 2>&7		# Restore stdout and stderr
+    if [ -z "$inside_subtest" ]; then
+	error "bug in the test script: test_expect_equal_file_nonempty without test_begin_subtest"
+    fi
+    inside_subtest=
+    test "$#" = 2 ||
+	error "bug in the test script: not 2 parameters to test_expect_equal_file_nonempty"
+
+    for file in "$1" "$2"; do
+	if [ ! -s "$file" ]; then
+	    test_failure_ "Missing or zero length file: $file"
+	fi
+    done
+
+    test_diff_file_ "$1" "$2"
+}
+
 # Like test_expect_equal, but arguments are JSON expressions to be
 # canonicalized before diff'ing.  If an argument cannot be parsed, it
 # is used unchanged so that there's something to diff against.
-- 
2.30.2

[PATCH 5/8] test/path-config: use test_expect_equal_file_nonempty

[David Bremner]

This is more robust against crashes when the expected output is also
generated by notmuch. In the case where the expected output is
explicit, it seems like overkill.
---
 test/T055-path-config.sh | 11 +++++------
 test/test-lib.sh         |  1 +
 2 files changed, 6 insertions(+), 6 deletions(-)

diff --git a/test/T055-path-config.sh b/test/T055-path-config.sh
index 2061b116..035276c2 100755
--- a/test/T055-path-config.sh
+++ b/test/T055-path-config.sh
@@ -151,13 +151,13 @@ EOF
     notmuch tag -inbox '*'
     notmuch restore < EXPECTED
     notmuch dump > OUTPUT
-    test_expect_equal_file EXPECTED OUTPUT
+    test_expect_equal_file_nonempty EXPECTED OUTPUT
 
     config_subtest "reindex" "$config"
     notmuch search --output=messages '*' > EXPECTED
     notmuch reindex '*'
     notmuch search --output=messages '*' > OUTPUT
-    test_expect_equal_file EXPECTED OUTPUT
+    test_expect_equal_file_nonempty EXPECTED OUTPUT
 
     config_subtest "use existing database" "$config"
     output=$(notmuch new)
@@ -179,7 +179,7 @@ EOF
     config_subtest "Show a raw message" "$config"
     add_message
     notmuch show --format=raw id:$gen_msg_id > OUTPUT
-    test_expect_equal_file $gen_msg_filename OUTPUT
+    test_expect_equal_file_nonempty $gen_msg_filename OUTPUT
     rm -f $gen_msg_filename
 
     config_subtest "reply" "$config"
@@ -208,14 +208,13 @@ EOF
     mkdir -p "$MAIL_DIR"/{cur,new,tmp}
     notmuch insert < "$gen_msg_filename"
     cur_msg_filename=$(notmuch search --output=files "subject:insert-subject")
-    test_expect_equal_file "$cur_msg_filename" "$gen_msg_filename"
-
+    test_expect_equal_file_nonempty "$cur_msg_filename" "$gen_msg_filename"
 
     config_subtest "compact+search" "$config"
     notmuch search --output=messages '*' | sort > EXPECTED
     notmuch compact
     notmuch search --output=messages '*' | sort > OUTPUT
-    test_expect_equal_file EXPECTED OUTPUT
+    test_expect_equal_file_nonempty EXPECTED OUTPUT
 
     config_subtest "upgrade backup" "$config"
     features=$(xapian-metadata get $XAPIAN_PATH features | grep -v "^relative directory paths")
diff --git a/test/test-lib.sh b/test/test-lib.sh
index 4865032d..94f9d605 100644
--- a/test/test-lib.sh
+++ b/test/test-lib.sh
@@ -447,6 +447,7 @@ test_expect_equal_file_nonempty () {
     for file in "$1" "$2"; do
 	if [ ! -s "$file" ]; then
 	    test_failure_ "Missing or zero length file: $file"
+	    return $?
 	fi
     done
 
-- 
2.30.2

[PATCH 6/8] test: add known broken tests for finding database via MAILDIR

[David Bremner]

This highlights a bug reported by several users, including
Mohsin Kaleem [1].

The use of test_begin_subtest is because these tests pass even though
the database cannot be located.

[1]: id:87bl9lx864.fsf@kisara.moe
---
 test/T055-path-config.sh | 34 ++++++++++++++++++++++++++--------
 1 file changed, 26 insertions(+), 8 deletions(-)

diff --git a/test/T055-path-config.sh b/test/T055-path-config.sh
index 035276c2..40601acd 100755
--- a/test/T055-path-config.sh
+++ b/test/T055-path-config.sh
@@ -16,6 +16,7 @@ restore_config () {
     unset DATABASE_PATH
     unset NOTMUCH_PROFILE
     unset XAPIAN_PATH
+    unset MAILDIR
     rm -f "$HOME/mail"
     cp notmuch-config-backup.${test_name} ${NOTMUCH_CONFIG}
 }
@@ -55,6 +56,18 @@ home_mail_config () {
     unset DATABASE_PATH
 }
 
+maildir_env_config () {
+    local dir
+    backup_config
+    dir="${HOME}/env_points_here"
+    ln -s $MAIL_DIR $dir
+    export MAILDIR=$dir
+    notmuch config set database.path
+    notmuch config set database.mail_root
+    XAPIAN_PATH="${MAIL_DIR}/.notmuch/xapian"
+    unset DATABASE_PATH
+}
+
 xdg_config () {
     local dir
     local profile=${1:-default}
@@ -84,9 +97,12 @@ config_subtest () {
     local config=$2
 
     test_begin_subtest "$title ($config)"
+    if [[ "$config" == "maildir_env" ]]; then
+	test_subtest_known_broken
+    fi
 }
 
-for config in traditional split XDG XDG+profile symlink home_mail; do
+for config in traditional split XDG XDG+profile symlink home_mail maildir_env; do
     #start each set of tests with an known set of messages
     add_email_corpus
 
@@ -224,7 +240,7 @@ EOF
 	"$output" \
 	"Welcome to a new version of notmuch! Your database will now be upgraded."
 
-    config_subtest "notmuch +config -database suggests notmuch new" "$config"
+    test_begin_subtest "notmuch +config -database suggests notmuch new" "$config"
     mv "$XAPIAN_PATH" "${XAPIAN_PATH}.bak"
     notmuch > OUTPUT
 cat <<EOF > EXPECTED
@@ -240,7 +256,7 @@ EOF
 
    test_expect_equal_file EXPECTED OUTPUT
 
-   config_subtest "Set config value" "$config"
+   test_begin_subtest "Set config value" "$config"
    name=${RANDOM}
    value=${RANDOM}
    notmuch config set test${test_count}.${name} ${value}
@@ -259,11 +275,13 @@ EOF
    test_expect_equal "${output}+${output2}" "${value}+"
 
    config_subtest "Config list" "$config"
-   notmuch config list | notmuch_dir_sanitize | sed -e "s/^database.backup_dir=.*$/database.backup_dir/"  \
-						    -e "s/^database.hook_dir=.*$/database.hook_dir/" \
-						    -e "s/^database.path=.*$/database.path/"  \
-						    -e "s,^database.mail_root=CWD/home/mail,database.mail_root=MAIL_DIR," \
-						    > OUTPUT
+   notmuch config list | notmuch_dir_sanitize | \
+       sed -e "s/^database.backup_dir=.*$/database.backup_dir/"  \
+	   -e "s/^database.hook_dir=.*$/database.hook_dir/" \
+	   -e "s/^database.path=.*$/database.path/"  \
+	   -e "s,^database.mail_root=CWD/home/mail,database.mail_root=MAIL_DIR," \
+	   -e "s,^database.mail_root=CWD/home/env_points_here,database.mail_root=MAIL_DIR," \
+	   > OUTPUT
    cat <<EOF > EXPECTED
 built_with.compact=true
 built_with.field_processor=true
-- 
2.30.2

[PATCH 7/8] lib/open: look in MAILDIR for database, as documented.

[David Bremner]

This fixes the bug id:87bl9lx864.fsf@kisara.moe
---
 lib/open.cc              | 4 ++++
 test/T055-path-config.sh | 3 ---
 2 files changed, 4 insertions(+), 3 deletions(-)

diff --git a/lib/open.cc b/lib/open.cc
index 7b95c5b1..280ffee3 100644
--- a/lib/open.cc
+++ b/lib/open.cc
@@ -220,6 +220,10 @@ _choose_database_path (void *ctx,
 	}
     }
 
+    if (! *database_path) {
+	*database_path = getenv ("MAILDIR");
+    }
+
     if (! *database_path) {
 	notmuch_status_t status;
 
diff --git a/test/T055-path-config.sh b/test/T055-path-config.sh
index 40601acd..774fe9c8 100755
--- a/test/T055-path-config.sh
+++ b/test/T055-path-config.sh
@@ -97,9 +97,6 @@ config_subtest () {
     local config=$2
 
     test_begin_subtest "$title ($config)"
-    if [[ "$config" == "maildir_env" ]]; then
-	test_subtest_known_broken
-    fi
 }
 
 for config in traditional split XDG XDG+profile symlink home_mail maildir_env; do
-- 
2.30.2

[PATCH 8/8] doc: document database search algorithm.

[David Bremner]

Essentially a translation of the function _choose_database_path for
human consumption. As a bonus, document environment variable
NOTMUCH_DATABASE
---
 doc/man1/notmuch-config.rst | 21 ++++++++++++++++++++-
 doc/man1/notmuch.rst        |  5 +++++
 2 files changed, 25 insertions(+), 1 deletion(-)

diff --git a/doc/man1/notmuch-config.rst b/doc/man1/notmuch-config.rst
index 82fbb44c..07a9eaf0 100644
--- a/doc/man1/notmuch-config.rst
+++ b/doc/man1/notmuch-config.rst
@@ -60,7 +60,7 @@ paths are presumed relative to `$HOME` for items in section
     sub-directory named ``.notmuch`` if **database.mail\_root**
     is unset).
 
-    Default: ``$MAILDIR`` variable if set, otherwise ``$HOME/mail``.
+    Default: see :ref:`database`
 
 **database.mail_root**
     The top-level directory where your mail currently exists and to
@@ -272,6 +272,25 @@ Notmuch configuration file search order:
    by :envvar:`NOTMUCH_PROFILE` environment variable if set,
    ``$HOME/.notmuch-config`` otherwise.
 
+.. _database:
+
+DATABASE LOCATION
+-----------------
+
+Notmuch database search order:
+
+1. Directory specified by :envvar:`NOTMUCH_DATABASE` environment variable.
+
+2. Directory specified by config key ``database.path``.
+
+3. ``$XDG_DATA_HOME/notmuch/<profile>`` where ``<profile>``
+   is defined by :envvar:`NOTMUCH_PROFILE` environment variable if
+   set, ``$XDG_DATA_HOME/notmuch/default`` otherwise.
+
+4. Directory specified by :envvar:`MAILDIR` environment variable.
+
+5. ``$HOME/mail``
+
 HOOKS
 -----
 
diff --git a/doc/man1/notmuch.rst b/doc/man1/notmuch.rst
index 6bd1dc03..de4c91ac 100644
--- a/doc/man1/notmuch.rst
+++ b/doc/man1/notmuch.rst
@@ -164,6 +164,11 @@ of notmuch.
    Specifies the location of the notmuch configuration file. See
    :any:`notmuch-config(1)` for details.
 
+.. envvar:: NOTMUCH_DATABASE
+
+   Specifies the location of the notmuch database. See
+   :any:`notmuch-config(1)` for details.
+        
 .. envvar:: NOTMUCH_PROFILE
 
    Selects among notmuch configurations. See :any:`notmuch-config(1)`
-- 
2.30.2