Notmuch removing messages from the index, when calling `new` two times in a row

Notmuch removing messages from the index, when calling `new` two times in a row

[Mayeu]

Hi,

I have hit a weird behaviour with notmuch.

I recently saw a drop of the number of messages in my index, and after
doing a backup of the current state of the index, I am able to reproduce
this behaviour fairly consistently. ie: notmuch is purgin at least 50%
of its index of messages that are still in the maildir.

I do the following:

1. remove my notmuch database completely.
2. launching `notmuch new`:

   Found 42059 total files (that's not much mail).
   Processed 42059 total files in 3m 24s (205 files/sec.).
   Added 41331 new messages to the database.

3. launching `notmuch new` right after, without having done anything else
   on my Maildir folder:

   Cleaned up 5 of 156 directories (1m 9s remaining).                                                                                                   
   No new mail. Removed 22211 messages. Detected 378 file renames.

Then I can loop back to step 1 and reproduce the behaviour every single time.

If I keep a window with Notmuch Emacs in it, I can see some e-mail
disapearing, and search not returning result anymore after the second
notmuch new.

Is it a know bug? A feature that I do not understand? Anyway, this make
notmuch almost unusable right now, since I can't have acces to my full
e-mail history, and I endup with conversations with holes in them.

Regards,

-- 
Mayeu a.k.a Matthieu
http://6x9.fr
GPG: A016 F2D8 0472 1186 1B33  A419 22B1 0496 B00D A693

Re: Notmuch removing messages from the index, when calling `new` two times in a row

[David Bremner]

Mayeu <m@6x9.fr> writes:
> 3. launching `notmuch new` right after, without having done anything else
>    on my Maildir folder:
>
>    Cleaned up 5 of 156 directories (1m 9s remaining).                                                                                                   
>    No new mail. Removed 22211 messages. Detected 378 file renames.

This message means that notmuch thinks 156 directories have been
deleted. Do you have any special mounts / symlinks / file systems that
might trigger this?

d

Re: Notmuch removing messages from the index, when calling `new` two times in a row

[Mayeu]

[-- Attachment #1: Type: text/plain, Size: 690 bytes --]

David Bremner <david@tethera.net> writes:
> This message means that notmuch thinks 156 directories have been
> deleted. Do you have any special mounts / symlinks / file systems that
> might trigger this?

Thank you for your response, my whole drive uses ZFS and I have a ZFS
volume for my whole /home/*. My Maildir in itself is just a folder in my
~, and do not have anything special.

I have encountered this behaviour also on my previous setup (migrated
last week-end from Archlinux to NixOS), and this one was using ext4, so
it does not seems to be ZFS related.

Regards,

-- 
Mayeu a.k.a Matthieu
http://6x9.fr
GPG: A016 F2D8 0472 1186 1B33  A419 22B1 0496 B00D A693

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]

Re: Notmuch removing messages from the index, when calling `new` two times in a row

[David Bremner]

Mayeu <m@6x9.fr> writes:

> David Bremner <david@tethera.net> writes:
>> This message means that notmuch thinks 156 directories have been
>> deleted. Do you have any special mounts / symlinks / file systems that
>> might trigger this?
>
> Thank you for your response, my whole drive uses ZFS and I have a ZFS
> volume for my whole /home/*. My Maildir in itself is just a folder in my
> ~, and do not have anything special.
>
> I have encountered this behaviour also on my previous setup (migrated
> last week-end from Archlinux to NixOS), and this one was using ext4, so
> it does not seems to be ZFS related.

OK. Please try running "notmuch new" (the second time) with --debug.

d

Re: Notmuch removing messages from the index, when calling `new` two times in a row

[Mayeu]

On Fri, 24 Apr 2015 07:25:28 +0900
David Bremner <david@tethera.net> wrote:
> OK. Please try running "notmuch new" (the second time) with --debug.

The output is exactly the same, no other information. I also tried to
add --verbose but same output. Both are accepted as valid since I do
net get back any error, but the output is not changed.

Is it something I should enable during the compilation of Notmuch? I
am using 0.19, and from what I see, the notmuch configuration used is
the default one on Nixos[1]. But when running ./configure --help I do
not see any --{enable,disable}-debug option.

[1] https://github.com/NixOS/nixpkgs/blob/master/pkgs/applications/networking/mailreaders/notmuch/default.nix

-- 
Mayeu a.k.a Matthieu
http://6x9.fr
GPG: A016 F2D8 0472 1186 1B33  A419 22B1 0496 B00D A693

Re: Notmuch removing messages from the index, when calling `new` two times in a row

[David Bremner]

[-- Attachment #1: Type: text/plain, Size: 599 bytes --]

Mayeu <ml+notmuch@6x9.fr> writes:

> On Fri, 24 Apr 2015 07:25:28 +0900
> David Bremner <david@tethera.net> wrote:
>> OK. Please try running "notmuch new" (the second time) with --debug.
>
> The output is exactly the same, no other information. I also tried to
> add --verbose but same output. Both are accepted as valid since I do
> net get back any error, but the output is not changed.
>

I think there is just not enough debugging output in this part of the
code. If you can compile from source, please try the attached patch
(it's against master, but I guess it should apply to 0.19 as well).


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-cli-new-add-more-debugging-output.patch --]
[-- Type: text/x-diff, Size: 1989 bytes --]

From 52975ecfdc754863e9ac9b465257a566845377c3 Mon Sep 17 00:00:00 2001
From: David Bremner <david@tethera.net>
Date: Mon, 27 Apr 2015 08:42:36 +0900
Subject: [PATCH] cli/new: add more debugging output

Try to narrow down what part of the code adds files and directories to
the queue(s) to be deleted.
---
 notmuch-new.c | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/notmuch-new.c b/notmuch-new.c
index e6c283e..39b0b9a 100644
--- a/notmuch-new.c
+++ b/notmuch-new.c
@@ -528,6 +528,10 @@ add_files (notmuch_database_t *notmuch,
 					      "%s/%s", path,
 					      notmuch_filenames_get (db_files));
 
+	    if (state->debug)
+		printf ("(D) add_files_recursive, pass 2: queuing passed file %s for deletion\n",
+			absolute);
+
 	    _filename_list_add (state->removed_files, absolute);
 
 	    notmuch_filenames_move_to_next (db_files);
@@ -542,6 +546,9 @@ add_files (notmuch_database_t *notmuch,
 	    {
 		char *absolute = talloc_asprintf (state->removed_directories,
 						  "%s/%s", path, filename);
+		if (state->debug)
+		    printf ("(D) add_files_recursive, pass 2: queuing passed directory %s for deletion\n",
+			absolute);
 
 		_filename_list_add (state->removed_directories, absolute);
 	    }
@@ -610,6 +617,9 @@ add_files (notmuch_database_t *notmuch,
 	char *absolute = talloc_asprintf (state->removed_files,
 					  "%s/%s", path,
 					  notmuch_filenames_get (db_files));
+	if (state->debug)
+	    printf ("(D) add_files_recursive, pass 3: queuing leftover file %s for deletion\n",
+		    absolute);
 
 	_filename_list_add (state->removed_files, absolute);
 
@@ -622,6 +632,10 @@ add_files (notmuch_database_t *notmuch,
 					  "%s/%s", path,
 					  notmuch_filenames_get (db_subdirs));
 
+	if (state->debug)
+	    printf ("(D) add_files_recursive, pass 3: queuing leftover directory %s for deletion\n",
+		    absolute);
+
 	_filename_list_add (state->removed_directories, absolute);
 
 	notmuch_filenames_move_to_next (db_subdirs);
-- 
2.1.4

[PATCH v2] cli/new: add more debugging output

[David Bremner]

Try to narrow down what part of the code adds files and directories to
the queue(s) to be deleted.
---

 - print slightly less scary messages
 - use git-send-email to send

 notmuch-new.c | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/notmuch-new.c b/notmuch-new.c
index 8ff1ade..ee786a3 100644
--- a/notmuch-new.c
+++ b/notmuch-new.c
@@ -528,6 +528,10 @@ add_files (notmuch_database_t *notmuch,
 					      "%s/%s", path,
 					      notmuch_filenames_get (db_files));
 
+	    if (state->debug)
+		printf ("(D) add_files_recursive, pass 2: queuing passed file %s for deletion from database\n",
+			absolute);
+
 	    _filename_list_add (state->removed_files, absolute);
 
 	    notmuch_filenames_move_to_next (db_files);
@@ -542,6 +546,9 @@ add_files (notmuch_database_t *notmuch,
 	    {
 		char *absolute = talloc_asprintf (state->removed_directories,
 						  "%s/%s", path, filename);
+		if (state->debug)
+		    printf ("(D) add_files_recursive, pass 2: queuing passed directory %s for deletion from database\n",
+			absolute);
 
 		_filename_list_add (state->removed_directories, absolute);
 	    }
@@ -610,6 +617,9 @@ add_files (notmuch_database_t *notmuch,
 	char *absolute = talloc_asprintf (state->removed_files,
 					  "%s/%s", path,
 					  notmuch_filenames_get (db_files));
+	if (state->debug)
+	    printf ("(D) add_files_recursive, pass 3: queuing leftover file %s for deletion from database\n",
+		    absolute);
 
 	_filename_list_add (state->removed_files, absolute);
 
@@ -622,6 +632,10 @@ add_files (notmuch_database_t *notmuch,
 					  "%s/%s", path,
 					  notmuch_filenames_get (db_subdirs));
 
+	if (state->debug)
+	    printf ("(D) add_files_recursive, pass 3: queuing leftover directory %s for deletion from database\n",
+		    absolute);
+
 	_filename_list_add (state->removed_directories, absolute);
 
 	notmuch_filenames_move_to_next (db_subdirs);
-- 
2.1.4

Re: [PATCH v2] cli/new: add more debugging output

[Tomi Ollila]

On Sun, Aug 02 2015, David Bremner <david@tethera.net> wrote:

> Try to narrow down what part of the code adds files and directories to
> the queue(s) to be deleted.
> ---

Looks trivial enough to me. +1

Tomi


>
>  - print slightly less scary messages
>  - use git-send-email to send
>
>  notmuch-new.c | 14 ++++++++++++++
>  1 file changed, 14 insertions(+)
>
> diff --git a/notmuch-new.c b/notmuch-new.c
> index 8ff1ade..ee786a3 100644
> --- a/notmuch-new.c
> +++ b/notmuch-new.c
> @@ -528,6 +528,10 @@ add_files (notmuch_database_t *notmuch,
>  					      "%s/%s", path,
>  					      notmuch_filenames_get (db_files));
>  
> +	    if (state->debug)
> +		printf ("(D) add_files_recursive, pass 2: queuing passed file %s for deletion from database\n",
> +			absolute);
> +
>  	    _filename_list_add (state->removed_files, absolute);
>  
>  	    notmuch_filenames_move_to_next (db_files);
> @@ -542,6 +546,9 @@ add_files (notmuch_database_t *notmuch,
>  	    {
>  		char *absolute = talloc_asprintf (state->removed_directories,
>  						  "%s/%s", path, filename);
> +		if (state->debug)
> +		    printf ("(D) add_files_recursive, pass 2: queuing passed directory %s for deletion from database\n",
> +			absolute);
>  
>  		_filename_list_add (state->removed_directories, absolute);
>  	    }
> @@ -610,6 +617,9 @@ add_files (notmuch_database_t *notmuch,
>  	char *absolute = talloc_asprintf (state->removed_files,
>  					  "%s/%s", path,
>  					  notmuch_filenames_get (db_files));
> +	if (state->debug)
> +	    printf ("(D) add_files_recursive, pass 3: queuing leftover file %s for deletion from database\n",
> +		    absolute);
>  
>  	_filename_list_add (state->removed_files, absolute);
>  
> @@ -622,6 +632,10 @@ add_files (notmuch_database_t *notmuch,
>  					  "%s/%s", path,
>  					  notmuch_filenames_get (db_subdirs));
>  
> +	if (state->debug)
> +	    printf ("(D) add_files_recursive, pass 3: queuing leftover directory %s for deletion from database\n",
> +		    absolute);
> +
>  	_filename_list_add (state->removed_directories, absolute);
>  
>  	notmuch_filenames_move_to_next (db_subdirs);
> -- 
> 2.1.4
>
> _______________________________________________
> notmuch mailing list
> notmuch@notmuchmail.org
> http://notmuchmail.org/mailman/listinfo/notmuch

test suite: add more debugging output for notmuch-new

[David Bremner]

The first patch in the series is a trivial cleanup.

I _thought_ (and tricked Tomi ;)) that the next one was trivial too,
but now I'm not so sure.

First, I forgot to update one test, but I mostly understand (or can at
least rationalize) that output. Then I had the bright idea that I
should add debugging output to other tests in T050-new. In 5 cases I
can't really understand the results. It seems like it takes several
runs of notmuch-new to remove directory entries. Maybe this is
intended, but I couldn't figure that out in a quick look.

[PATCH 1/3] test: don't claim single message mbox support is going away

[David Bremner]

We gave up on this.
---
 test/T050-new.sh | 13 ++++++++-----
 1 file changed, 8 insertions(+), 5 deletions(-)

diff --git a/test/T050-new.sh b/test/T050-new.sh
index e6c3291..66ea10f 100755
--- a/test/T050-new.sh
+++ b/test/T050-new.sh
@@ -63,15 +63,18 @@ test_begin_subtest "Renamed message"
 generate_message
 notmuch new > /dev/null
 mv "$gen_msg_filename" "${gen_msg_filename}"-renamed
-output=$(NOTMUCH_NEW)
-test_expect_equal "$output" "No new mail. Detected 1 file rename."
+output=$(NOTMUCH_NEW --debug)
+test_expect_equal "$output" "(D) add_files_recursive, pass 2: queuing passed file ${gen_msg_filename} for deletion from database
+No new mail. Detected 1 file rename."
 
 
 test_begin_subtest "Deleted message"
 
 rm "${gen_msg_filename}"-renamed
-output=$(NOTMUCH_NEW)
-test_expect_equal "$output" "No new mail. Removed 1 message."
+output=$(NOTMUCH_NEW --debug)
+test_expect_equal "$output" "(D) add_files_recursive, pass 3: queuing leftover file ${gen_msg_filename}-renamed for deletion from database
+No new mail. Removed 1 message."
+
 
 
 test_begin_subtest "Renamed directory"
@@ -163,7 +166,7 @@ rm -rf "${MAIL_DIR}"/two
 output=$(NOTMUCH_NEW)
 test_expect_equal "$output" "No new mail. Removed 3 messages."
 
-test_begin_subtest "Support single-message mbox (deprecated)"
+test_begin_subtest "Support single-message mbox"
 cat > "${MAIL_DIR}"/mbox_file1 <<EOF
 From test_suite@notmuchmail.org Fri Jan  5 15:43:57 2001
 From: Notmuch Test Suite <test_suite@notmuchmail.org>
-- 
2.1.4

[PATCH 2/3] cli/new: add more debugging output

[David Bremner]

Try to narrow down what part of the code adds files and directories to
the queue(s) to be deleted.

Update one test. The output is slightly confusing, but I believe it is
correct, resulting from a directory being discovered but containing only ignored files.
---
 notmuch-new.c    | 14 ++++++++++++++
 test/T050-new.sh |  1 +
 2 files changed, 15 insertions(+)

diff --git a/notmuch-new.c b/notmuch-new.c
index 8ff1ade..ee786a3 100644
--- a/notmuch-new.c
+++ b/notmuch-new.c
@@ -528,6 +528,10 @@ add_files (notmuch_database_t *notmuch,
 					      "%s/%s", path,
 					      notmuch_filenames_get (db_files));
 
+	    if (state->debug)
+		printf ("(D) add_files_recursive, pass 2: queuing passed file %s for deletion from database\n",
+			absolute);
+
 	    _filename_list_add (state->removed_files, absolute);
 
 	    notmuch_filenames_move_to_next (db_files);
@@ -542,6 +546,9 @@ add_files (notmuch_database_t *notmuch,
 	    {
 		char *absolute = talloc_asprintf (state->removed_directories,
 						  "%s/%s", path, filename);
+		if (state->debug)
+		    printf ("(D) add_files_recursive, pass 2: queuing passed directory %s for deletion from database\n",
+			absolute);
 
 		_filename_list_add (state->removed_directories, absolute);
 	    }
@@ -610,6 +617,9 @@ add_files (notmuch_database_t *notmuch,
 	char *absolute = talloc_asprintf (state->removed_files,
 					  "%s/%s", path,
 					  notmuch_filenames_get (db_files));
+	if (state->debug)
+	    printf ("(D) add_files_recursive, pass 3: queuing leftover file %s for deletion from database\n",
+		    absolute);
 
 	_filename_list_add (state->removed_files, absolute);
 
@@ -622,6 +632,10 @@ add_files (notmuch_database_t *notmuch,
 					  "%s/%s", path,
 					  notmuch_filenames_get (db_subdirs));
 
+	if (state->debug)
+	    printf ("(D) add_files_recursive, pass 3: queuing leftover directory %s for deletion from database\n",
+		    absolute);
+
 	_filename_list_add (state->removed_directories, absolute);
 
 	notmuch_filenames_move_to_next (db_subdirs);
diff --git a/test/T050-new.sh b/test/T050-new.sh
index 66ea10f..ab15ca1 100755
--- a/test/T050-new.sh
+++ b/test/T050-new.sh
@@ -238,6 +238,7 @@ test_expect_equal "$output" \
 (D) add_files_recursive, pass 2: explicitly ignoring ${MAIL_DIR}/one/two/ignored_file
 (D) add_files_recursive, pass 2: explicitly ignoring ${MAIL_DIR}/one/two/three/.git
 (D) add_files_recursive, pass 2: explicitly ignoring ${MAIL_DIR}/one/two/three/ignored_file
+(D) add_files_recursive, pass 3: queuing leftover directory ${MAIL_DIR}/two for deletion from database
 No new mail."
 
 
-- 
2.1.4

[PATCH 3/3] test: add debugging output to notmuch-new tests, mark 5 as broken

[David Bremner]

I marked the tests where I really couldn't understand the output as
broken. It could also be that I don't understand how directory removal
is supposed to work.
---
 test/T050-new.sh | 62 ++++++++++++++++++++++++++++++++------------------------
 1 file changed, 35 insertions(+), 27 deletions(-)

diff --git a/test/T050-new.sh b/test/T050-new.sh
index ab15ca1..ff1c354 100755
--- a/test/T050-new.sh
+++ b/test/T050-new.sh
@@ -3,25 +3,25 @@ test_description='"notmuch new" in several variations'
 . ./test-lib.sh
 
 test_begin_subtest "No new messages"
-output=$(NOTMUCH_NEW)
+output=$(NOTMUCH_NEW --debug)
 test_expect_equal "$output" "No new mail."
 
 
 test_begin_subtest "Single new message"
 generate_message
-output=$(NOTMUCH_NEW)
+output=$(NOTMUCH_NEW --debug)
 test_expect_equal "$output" "Added 1 new message to the database."
 
 
 test_begin_subtest "Multiple new messages"
 generate_message
 generate_message
-output=$(NOTMUCH_NEW)
+output=$(NOTMUCH_NEW --debug)
 test_expect_equal "$output" "Added 2 new messages to the database."
 
 
 test_begin_subtest "No new messages (non-empty DB)"
-output=$(NOTMUCH_NEW)
+output=$(NOTMUCH_NEW --debug)
 test_expect_equal "$output" "No new mail."
 
 
@@ -31,7 +31,7 @@ mkdir "${MAIL_DIR}"/def
 mkdir "${MAIL_DIR}"/ghi
 generate_message [dir]=def
 
-output=$(NOTMUCH_NEW)
+output=$(NOTMUCH_NEW --debug)
 test_expect_equal "$output" "Added 1 new message to the database."
 
 
@@ -42,7 +42,7 @@ mv "${MAIL_DIR}"/ghi "${MAIL_DIR}"/abc
 rm "${MAIL_DIR}"/def/*
 generate_message [dir]=abc
 
-output=$(NOTMUCH_NEW)
+output=$(NOTMUCH_NEW --debug)
 test_expect_equal "$output" "Added 1 new message to the database."
 
 
@@ -54,7 +54,7 @@ mkdir -p "$(dirname "$tmp_msg_filename")"
 mv "$gen_msg_filename" "$tmp_msg_filename"
 notmuch new > /dev/null
 mv "$tmp_msg_filename" "$gen_msg_filename"
-output=$(NOTMUCH_NEW)
+output=$(NOTMUCH_NEW --debug)
 test_expect_equal "$output" "Added 1 new message to the database."
 
 
@@ -87,34 +87,39 @@ notmuch new > /dev/null
 
 mv "${MAIL_DIR}"/dir "${MAIL_DIR}"/dir-renamed
 
-output=$(NOTMUCH_NEW)
-test_expect_equal "$output" "No new mail. Detected 3 file renames."
+output=$(NOTMUCH_NEW --debug)
+test_expect_equal "$output" "(D) add_files_recursive, pass 2: queuing passed directory ${MAIL_DIR}/dir for deletion from database
+No new mail. Detected 3 file renames."
 
 
 test_begin_subtest "Deleted directory"
-
+test_subtest_known_broken
 rm -rf "${MAIL_DIR}"/dir-renamed
 
-output=$(NOTMUCH_NEW)
-test_expect_equal "$output" "No new mail. Removed 3 messages."
+output=$(NOTMUCH_NEW --debug)
+test_expect_equal "$output" "(D) add_files_recursive, pass 2: queuing passed directory ${MAIL_DIR}/dir-renamed for deletion from database
+No new mail. Removed 3 messages."
 
 
 test_begin_subtest "New directory (at end of list)"
+test_subtest_known_broken
 
 generate_message [dir]=zzz
 generate_message [dir]=zzz
 generate_message [dir]=zzz
 
-output=$(NOTMUCH_NEW)
+output=$(NOTMUCH_NEW --debug)
 test_expect_equal "$output" "Added 3 new messages to the database."
 
 
 test_begin_subtest "Deleted directory (end of list)"
 
+test_subtest_known_broken
 rm -rf "${MAIL_DIR}"/zzz
 
-output=$(NOTMUCH_NEW)
-test_expect_equal "$output" "No new mail. Removed 3 messages."
+output=$(NOTMUCH_NEW --debug)
+test_expect_equal "$output" "(D) add_files_recursive, pass 3: queuing leftover directory ${MAIL_DIR}/zzz for deletion from database
+No new mail. Removed 3 messages."
 
 
 test_begin_subtest "New symlink to directory"
@@ -125,7 +130,7 @@ mv "${MAIL_DIR}" "${TMP_DIRECTORY}"/actual_maildir
 mkdir "${MAIL_DIR}"
 ln -s "${TMP_DIRECTORY}"/actual_maildir "${MAIL_DIR}"/symlink
 
-output=$(NOTMUCH_NEW)
+output=$(NOTMUCH_NEW --debug)
 test_expect_equal "$output" "Added 1 new message to the database."
 
 
@@ -135,13 +140,13 @@ external_msg_filename="${TMP_DIRECTORY}"/external/"$(basename "$gen_msg_filename
 mkdir -p "$(dirname "$external_msg_filename")"
 mv "$gen_msg_filename" "$external_msg_filename"
 ln -s "$external_msg_filename" "$gen_msg_filename"
-output=$(NOTMUCH_NEW)
+output=$(NOTMUCH_NEW --debug)
 test_expect_equal "$output" "Added 1 new message to the database."
 
 
 test_begin_subtest "Broken symlink aborts"
 ln -s does-not-exist "${MAIL_DIR}/broken"
-output=$(NOTMUCH_NEW 2>&1)
+output=$(NOTMUCH_NEW --debug 2>&1)
 test_expect_equal "$output" \
 "Error reading file ${MAIL_DIR}/broken: No such file or directory
 Note: A fatal error was encountered: Something went wrong trying to read or write a file
@@ -155,7 +160,7 @@ generate_message [dir]=two/levels
 generate_message [dir]=two/levels
 generate_message [dir]=two/levels
 
-output=$(NOTMUCH_NEW)
+output=$(NOTMUCH_NEW --debug)
 test_expect_equal "$output" "Added 3 new messages to the database."
 
 
@@ -163,10 +168,12 @@ test_begin_subtest "Deleted two-level directory"
 
 rm -rf "${MAIL_DIR}"/two
 
-output=$(NOTMUCH_NEW)
-test_expect_equal "$output" "No new mail. Removed 3 messages."
+output=$(NOTMUCH_NEW --debug)
+test_expect_equal "$output" "(D) add_files_recursive, pass 3: queuing leftover directory ${MAIL_DIR}/two for deletion from database
+No new mail. Removed 3 messages."
 
 test_begin_subtest "Support single-message mbox"
+test_subtest_known_broken
 cat > "${MAIL_DIR}"/mbox_file1 <<EOF
 From test_suite@notmuchmail.org Fri Jan  5 15:43:57 2001
 From: Notmuch Test Suite <test_suite@notmuchmail.org>
@@ -175,11 +182,12 @@ Subject: Test mbox message 1
 
 Body.
 EOF
-output=$(NOTMUCH_NEW 2>&1)
+output=$(NOTMUCH_NEW --debug 2>&1)
 test_expect_equal "$output" "Added 1 new message to the database."
 
 # This test requires that notmuch new has been run at least once.
 test_begin_subtest "Skip and report non-mail files"
+test_subtest_known_broken
 generate_message
 mkdir -p "${MAIL_DIR}"/.git && touch "${MAIL_DIR}"/.git/config
 touch "${MAIL_DIR}"/ignored_file
@@ -199,7 +207,7 @@ Subject: Test mbox message 2
 
 Body 2.
 EOF
-output=$(NOTMUCH_NEW 2>&1)
+output=$(NOTMUCH_NEW --debug 2>&1)
 test_expect_equal "$output" \
 "Note: Ignoring non-mail file: ${MAIL_DIR}/.git/config
 Note: Ignoring non-mail file: ${MAIL_DIR}/.ignored_hidden_file
@@ -267,23 +275,23 @@ OLDCONFIG=$(notmuch config get new.tags)
 
 test_begin_subtest "Empty tags in new.tags are forbidden"
 notmuch config set new.tags "foo;;bar"
-output=$(NOTMUCH_NEW 2>&1)
+output=$(NOTMUCH_NEW --debug 2>&1)
 test_expect_equal "$output" "Error: tag '' in new.tags: empty tag forbidden"
 
 test_begin_subtest "Tags starting with '-' in new.tags are forbidden"
 notmuch config set new.tags "-foo;bar"
-output=$(NOTMUCH_NEW 2>&1)
+output=$(NOTMUCH_NEW --debug 2>&1)
 test_expect_equal "$output" "Error: tag '-foo' in new.tags: tag starting with '-' forbidden"
 
 test_expect_code 1 "Invalid tags set exit code" \
-    "NOTMUCH_NEW 2>&1"
+    "NOTMUCH_NEW --debug 2>&1"
 
 notmuch config set new.tags $OLDCONFIG
 
 
 test_begin_subtest "Xapian exception: read only files"
 chmod u-w  ${MAIL_DIR}/.notmuch/xapian/*.DB
-output=$(NOTMUCH_NEW 2>&1 | sed 's/: .*$//' )
+output=$(NOTMUCH_NEW --debug 2>&1 | sed 's/: .*$//' )
 chmod u+w  ${MAIL_DIR}/.notmuch/xapian/*.DB
 test_expect_equal "$output" "A Xapian exception occurred opening database"
 
-- 
2.1.4

Re: test suite: add more debugging output for notmuch-new

[Tomi Ollila]

On Tue, Aug 04 2015, David Bremner <david@tethera.net> wrote:

> The first patch in the series is a trivial cleanup.
>
> I _thought_ (and tricked Tomi ;)) that the next one was trivial too,
> but now I'm not so sure.
>
> First, I forgot to update one test, but I mostly understand (or can at
> least rationalize) that output. Then I had the bright idea that I
> should add debugging output to other tests in T050-new. In 5 cases I
> can't really understand the results. It seems like it takes several
> runs of notmuch-new to remove directory entries. Maybe this is
> intended, but I couldn't figure that out in a quick look.

I got the same test results -- and could not figure out the reason through
a brief look at the notmuch-new.c code.

+1 for the series for now, hopefully we soon find out the resolution to the
known broken tests.

Tomi

Re: test suite: add more debugging output for notmuch-new

[David Bremner]

Tomi Ollila <tomi.ollila@iki.fi> writes:

>
> I got the same test results -- and could not figure out the reason through
> a brief look at the notmuch-new.c code.
>
> +1 for the series for now, hopefully we soon find out the resolution to the
> known broken tests.

OK, I pushed it, we'll see what happens. It'd be niced to sort this out
before the next release anyway.

d

Re: Notmuch removing messages from the index, when calling `new` two times in a row

[Jani Nikula]

On Fri, 24 Apr 2015, Mayeu <ml+notmuch@6x9.fr> wrote:
> On Fri, 24 Apr 2015 07:25:28 +0900
> David Bremner <david@tethera.net> wrote:
>> OK. Please try running "notmuch new" (the second time) with --debug.
>
> The output is exactly the same, no other information. I also tried to
> add --verbose but same output. Both are accepted as valid since I do
> net get back any error, but the output is not changed.
>
> Is it something I should enable during the compilation of Notmuch? I
> am using 0.19, and from what I see, the notmuch configuration used is
> the default one on Nixos[1]. But when running ./configure --help I do
> not see any --{enable,disable}-debug option.

I don't think we've resolved this bug yet. Do you still see it? Do all
the tests pass if you take current git, and do 'make test'?

BR,
Jani.

>
> [1] https://github.com/NixOS/nixpkgs/blob/master/pkgs/applications/networking/mailreaders/notmuch/default.nix
>
> -- 
> Mayeu a.k.a Matthieu
> http://6x9.fr
> GPG: A016 F2D8 0472 1186 1B33  A419 22B1 0496 B00D A693
> _______________________________________________
> notmuch mailing list
> notmuch@notmuchmail.org
> http://notmuchmail.org/mailman/listinfo/notmuch