[PATCH v3 0/8] cli: better tooling support for duplicate messages

[PATCH v3 0/8] cli: better tooling support for duplicate messages

[Jani Nikula]

v3 of id:cover.1370775663.git.jani@nikula.org. Just a rebase, no changes
since the latest versions of the patches posted in the previous thread.

BR,
Jani.

Jani Nikula (8):
  cli: add --duplicate=N option to notmuch search
  test: test notmuch search --duplicate=N
  man: document notmuch search --duplicate=N
  cli: add --output=files option to notmuch count
  test: test notmuch count --output=files
  man: document notmuch count --output=files
  NEWS: cli options for improved duplicate message handling
  man: clarify folder: prefix regarding duplicate files

 NEWS                            |   20 ++++++++++++
 man/man1/notmuch-count.1        |   10 +++++-
 man/man1/notmuch-search.1       |   20 ++++++++++++
 man/man7/notmuch-search-terms.7 |    8 +++--
 notmuch-count.c                 |   37 ++++++++++++++++++++++
 notmuch-search.c                |   18 +++++++----
 test/count                      |   10 ++++++
 test/search-output              |   65 +++++++++++++++++++++++++++++++++++++++
 8 files changed, 178 insertions(+), 10 deletions(-)

-- 
1.7.10.4

[PATCH v3 1/8] cli: add --duplicate=N option to notmuch search

[Jani Nikula]

Effective with --output=files, output the Nth filename associated with
each message matching the query (N is 1-based). If N is greater than
the number of files associated with the message, don't print anything.

---

v3: fix commit message, no other changes
---
 notmuch-search.c |   18 ++++++++++++------
 1 file changed, 12 insertions(+), 6 deletions(-)

diff --git a/notmuch-search.c b/notmuch-search.c
index a96f07d..d9d39ec 100644
--- a/notmuch-search.c
+++ b/notmuch-search.c
@@ -177,7 +177,8 @@ do_search_messages (sprinter_t *format,
 		    notmuch_query_t *query,
 		    output_t output,
 		    int offset,
-		    int limit)
+		    int limit,
+		    int dupe)
 {
     notmuch_message_t *message;
     notmuch_messages_t *messages;
@@ -206,14 +207,17 @@ do_search_messages (sprinter_t *format,
 	message = notmuch_messages_get (messages);
 
 	if (output == OUTPUT_FILES) {
+	    int j;
 	    filenames = notmuch_message_get_filenames (message);
 
-	    for (;
+	    for (j = 1;
 		 notmuch_filenames_valid (filenames);
-		 notmuch_filenames_move_to_next (filenames))
+		 notmuch_filenames_move_to_next (filenames), j++)
 	    {
-		format->string (format, notmuch_filenames_get (filenames));
-		format->separator (format);
+		if (dupe < 0 || dupe == j) {
+		    format->string (format, notmuch_filenames_get (filenames));
+		    format->separator (format);
+		}
 	    }
 	    
 	    notmuch_filenames_destroy( filenames );
@@ -296,6 +300,7 @@ notmuch_search_command (notmuch_config_t *config, int argc, char *argv[])
     int offset = 0;
     int limit = -1; /* unlimited */
     notmuch_exclude_t exclude = NOTMUCH_EXCLUDE_TRUE;
+    int dupe = -1;
     unsigned int i;
 
     enum {
@@ -332,6 +337,7 @@ notmuch_search_command (notmuch_config_t *config, int argc, char *argv[])
                                   { 0, 0 } } },
 	{ NOTMUCH_OPT_INT, &offset, "offset", 'O', 0 },
 	{ NOTMUCH_OPT_INT, &limit, "limit", 'L', 0  },
+	{ NOTMUCH_OPT_INT, &dupe, "duplicate", 'D', 0  },
 	{ 0, 0, 0, 0, 0 }
     };
 
@@ -414,7 +420,7 @@ notmuch_search_command (notmuch_config_t *config, int argc, char *argv[])
 	break;
     case OUTPUT_MESSAGES:
     case OUTPUT_FILES:
-	ret = do_search_messages (format, query, output, offset, limit);
+	ret = do_search_messages (format, query, output, offset, limit, dupe);
 	break;
     case OUTPUT_TAGS:
 	ret = do_search_tags (notmuch, format, query);
-- 
1.7.10.4

[PATCH v3 2/8] test: test notmuch search --duplicate=N

[Jani Nikula]

Add test for notmuch search --duplicate=N option.
---
 test/search-output |   65 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 65 insertions(+)

diff --git a/test/search-output b/test/search-output
index c2a87eb..5ccfeaf 100755
--- a/test/search-output
+++ b/test/search-output
@@ -239,6 +239,64 @@ MAIL_DIR/cur/01:2,
 EOF
 test_expect_equal_file OUTPUT EXPECTED
 
+test_begin_subtest "--output=files --duplicate=1"
+notmuch search --output=files --duplicate=1 '*' | sed -e "s,$MAIL_DIR,MAIL_DIR," >OUTPUT
+cat <<EOF >EXPECTED
+MAIL_DIR/cur/52:2,
+MAIL_DIR/cur/53:2,
+MAIL_DIR/cur/50:2,
+MAIL_DIR/cur/49:2,
+MAIL_DIR/cur/48:2,
+MAIL_DIR/cur/47:2,
+MAIL_DIR/cur/46:2,
+MAIL_DIR/cur/45:2,
+MAIL_DIR/cur/44:2,
+MAIL_DIR/cur/43:2,
+MAIL_DIR/cur/42:2,
+MAIL_DIR/cur/41:2,
+MAIL_DIR/cur/40:2,
+MAIL_DIR/cur/39:2,
+MAIL_DIR/cur/38:2,
+MAIL_DIR/cur/37:2,
+MAIL_DIR/cur/36:2,
+MAIL_DIR/cur/35:2,
+MAIL_DIR/cur/34:2,
+MAIL_DIR/cur/33:2,
+MAIL_DIR/cur/32:2,
+MAIL_DIR/cur/31:2,
+MAIL_DIR/cur/30:2,
+MAIL_DIR/cur/29:2,
+MAIL_DIR/cur/28:2,
+MAIL_DIR/cur/27:2,
+MAIL_DIR/cur/26:2,
+MAIL_DIR/cur/25:2,
+MAIL_DIR/cur/24:2,
+MAIL_DIR/cur/23:2,
+MAIL_DIR/cur/22:2,
+MAIL_DIR/cur/21:2,
+MAIL_DIR/cur/19:2,
+MAIL_DIR/cur/18:2,
+MAIL_DIR/cur/20:2,
+MAIL_DIR/cur/17:2,
+MAIL_DIR/cur/16:2,
+MAIL_DIR/cur/15:2,
+MAIL_DIR/cur/14:2,
+MAIL_DIR/cur/13:2,
+MAIL_DIR/cur/12:2,
+MAIL_DIR/cur/11:2,
+MAIL_DIR/cur/10:2,
+MAIL_DIR/cur/09:2,
+MAIL_DIR/cur/08:2,
+MAIL_DIR/cur/06:2,
+MAIL_DIR/cur/05:2,
+MAIL_DIR/cur/04:2,
+MAIL_DIR/cur/03:2,
+MAIL_DIR/cur/07:2,
+MAIL_DIR/cur/02:2,
+MAIL_DIR/cur/01:2,
+EOF
+test_expect_equal_file OUTPUT EXPECTED
+
 test_begin_subtest "--output=files --format=json"
 notmuch search --format=json --output=files '*' | sed -e "s,$MAIL_DIR,MAIL_DIR," >OUTPUT
 cat <<EOF >EXPECTED
@@ -298,6 +356,13 @@ cat <<EOF >EXPECTED
 EOF
 test_expect_equal_file OUTPUT EXPECTED
 
+test_begin_subtest "--output=files --format=json --duplicate=2"
+notmuch search --format=json --output=files --duplicate=2 '*' | sed -e "s,$MAIL_DIR,MAIL_DIR," >OUTPUT
+cat <<EOF >EXPECTED
+["MAIL_DIR/cur/51:2,"]
+EOF
+test_expect_equal_file OUTPUT EXPECTED
+
 test_begin_subtest "--output=tags"
 notmuch search --output=tags '*' >OUTPUT
 cat <<EOF >EXPECTED
-- 
1.7.10.4

[PATCH v3 3/8] man: document notmuch search --duplicate=N

[Jani Nikula]

Document the notmuch search --duplicate=N option.

---

v3: clarify dupe vs. folder prefix per Mark's suggestion
---
 man/man1/notmuch-search.1 |   20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)

diff --git a/man/man1/notmuch-search.1 b/man/man1/notmuch-search.1
index 2d6ecaf..f69a4cd 100644
--- a/man/man1/notmuch-search.1
+++ b/man/man1/notmuch-search.1
@@ -80,6 +80,10 @@ Output the filenames of all messages matching the search terms, either
 one per line (\-\-format=text), separated by null characters
 (\-\-format=text0), as a JSON array (\-\-format=json), or as an
 S-Expression list (\-\-format=sexp).
+
+Note that each message may have multiple filenames associated with it.
+All of them are included in the output, unless limited with the
+\-\-duplicate=N option.
 .RE
 .RS 4
 .TP 4
@@ -158,6 +162,22 @@ but the "match count" is the number of matching non-excluded messages in the
 thread, rather than the number of matching messages.
 .RE
 
+.RS 4
+.TP 4
+.BR \-\-duplicate=N
+
+Effective with
+.BR --output=files ,
+output the Nth filename associated with each message matching the
+query (N is 1-based). If N is greater than the number of files
+associated with the message, don't print anything.
+
+Note that this option is orthogonal with the
+.BR folder:
+search prefix. The prefix matches messages based on filenames. This
+option filters filenames of the matching messages.
+.RE
+
 .SH EXIT STATUS
 
 This command supports the following special exit status codes
-- 
1.7.10.4

[PATCH v3 4/8] cli: add --output=files option to notmuch count

[Jani Nikula]

Add support for querying the total number of files associated with the
messages matching the search. This is mostly useful with an
id:<message-id> query for a single message.
---
 notmuch-count.c |   37 +++++++++++++++++++++++++++++++++++++
 1 file changed, 37 insertions(+)

diff --git a/notmuch-count.c b/notmuch-count.c
index 8772cff..01e4e30 100644
--- a/notmuch-count.c
+++ b/notmuch-count.c
@@ -24,6 +24,7 @@
 enum {
     OUTPUT_THREADS,
     OUTPUT_MESSAGES,
+    OUTPUT_FILES,
 };
 
 /* The following is to allow future options to be added more easily */
@@ -32,6 +33,38 @@ enum {
     EXCLUDE_FALSE,
 };
 
+static unsigned int
+count_files (notmuch_query_t *query)
+{
+    notmuch_messages_t *messages;
+    notmuch_message_t *message;
+    notmuch_filenames_t *filenames;
+    unsigned int count = 0;
+
+    messages = notmuch_query_search_messages (query);
+    if (messages == NULL)
+	return 0;
+
+    for (;
+	 notmuch_messages_valid (messages);
+	 notmuch_messages_move_to_next (messages)) {
+	message = notmuch_messages_get (messages);
+	filenames = notmuch_message_get_filenames (message);
+
+	for (;
+	     notmuch_filenames_valid (filenames);
+	     notmuch_filenames_move_to_next (filenames))
+	    count++;
+
+	notmuch_filenames_destroy (filenames);
+	notmuch_message_destroy (message);
+    }
+
+    notmuch_messages_destroy (messages);
+
+    return count;
+}
+
 static int
 print_count (notmuch_database_t *notmuch, const char *query_str,
 	     const char **exclude_tags, size_t exclude_tags_length, int output)
@@ -55,6 +88,9 @@ print_count (notmuch_database_t *notmuch, const char *query_str,
     case OUTPUT_THREADS:
 	printf ("%u\n", notmuch_query_count_threads (query));
 	break;
+    case OUTPUT_FILES:
+	printf ("%u\n", count_files (query));
+	break;
     }
 
     notmuch_query_destroy (query);
@@ -102,6 +138,7 @@ notmuch_count_command (notmuch_config_t *config, int argc, char *argv[])
 	{ NOTMUCH_OPT_KEYWORD, &output, "output", 'o',
 	  (notmuch_keyword_t []){ { "threads", OUTPUT_THREADS },
 				  { "messages", OUTPUT_MESSAGES },
+				  { "files", OUTPUT_FILES },
 				  { 0, 0 } } },
 	{ NOTMUCH_OPT_KEYWORD, &exclude, "exclude", 'x',
 	  (notmuch_keyword_t []){ { "true", EXCLUDE_TRUE },
-- 
1.7.10.4

[PATCH v3 5/8] test: test notmuch count --output=files

[Jani Nikula]

Add tests for notmuch count --output=files option.
---
 test/count |   10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/test/count b/test/count
index 05713fd..da86c8c 100755
--- a/test/count
+++ b/test/count
@@ -28,6 +28,16 @@ test_expect_equal \
     "$((`notmuch search '*' | wc -l`))" \
     "`notmuch count --output=threads '*'`"
 
+test_begin_subtest "files count"
+test_expect_equal \
+    "$((`notmuch search --output=files '*' | wc -l`))" \
+    "`notmuch count --output=files '*'`"
+
+test_begin_subtest "files count for a duplicate message-id"
+test_expect_equal \
+    "2" \
+    "`notmuch count --output=files id:20091117232137.GA7669@griffis1.net`"
+
 test_begin_subtest "count with no matching messages"
 test_expect_equal \
     "0" \
-- 
1.7.10.4

[PATCH v3 6/8] man: document notmuch count --output=files

[Jani Nikula]

Document the notmuch count --output=files option.
---
 man/man1/notmuch-count.1 |   10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)

diff --git a/man/man1/notmuch-count.1 b/man/man1/notmuch-count.1
index baed25f..173e884 100644
--- a/man/man1/notmuch-count.1
+++ b/man/man1/notmuch-count.1
@@ -23,7 +23,7 @@ Supported options for
 include
 .RS 4
 .TP 4
-.B \-\-output=(messages|threads)
+.B \-\-output=(messages|threads|files)
 
 .RS 4
 .TP 4
@@ -37,6 +37,14 @@ Output the number of matching messages. This is the default.
 
 Output the number of matching threads.
 .RE
+.RS 4
+.TP 4
+.B files
+
+Output the number of files associated with matching messages. This may
+be bigger than the number of matching messages due to duplicates
+(i.e. multiple files having the same message-id).
+.RE
 .RE
 
 .RS 4
-- 
1.7.10.4

[PATCH v3 7/8] NEWS: cli options for improved duplicate message handling

[Jani Nikula]

---
 NEWS |   20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)

diff --git a/NEWS b/NEWS
index 722a35d..7f5bf6b 100644
--- a/NEWS
+++ b/NEWS
@@ -1,3 +1,23 @@
+
+Notmuch 0.16 (201x-xx-xx)
+=========================
+
+Command-Line Interface
+----------------------
+
+New options to better support handling duplicate messages
+
+  If more than one message file is associated with a message-id,
+  `notmuch search --output=files` will print all of them. A new
+  `--duplicate=N` option can be used to specify which duplicate to
+  print for each message.
+
+  `notmuch count` now supports `--output=files` option to output the
+  number of files associated with matching messages. This may be
+  bigger than the number of matching messages due to duplicates
+  (i.e. multiple files having the same message-id).
+
+
 Notmuch 0.16 (2013-08-03)
 =========================
 
-- 
1.7.10.4

Re: [PATCH v3 7/8] NEWS: cli options for improved duplicate message handling

[Tomi Ollila]

On Sat, Aug 17 2013, Jani Nikula <jani@nikula.org> wrote:

> ---
>  NEWS |   20 ++++++++++++++++++++
>  1 file changed, 20 insertions(+)
>
> diff --git a/NEWS b/NEWS
> index 722a35d..7f5bf6b 100644
> --- a/NEWS
> +++ b/NEWS
> @@ -1,3 +1,23 @@
> +
> +Notmuch 0.16 (201x-xx-xx)
> +=========================

This could be

Notmuch 0.17 (UNRELEASED)
=========================

I.e. not 0.16... I saw this UNRELEASED in some github project a while ago
and liked it as it clearer shows the release status of that version.

otherwise the whole patch series LGTM (as the one suggestion Mark had
to v2 was updates).

Tomi


> +
> +Command-Line Interface
> +----------------------
> +
> +New options to better support handling duplicate messages
> +
> +  If more than one message file is associated with a message-id,
> +  `notmuch search --output=files` will print all of them. A new
> +  `--duplicate=N` option can be used to specify which duplicate to
> +  print for each message.
> +
> +  `notmuch count` now supports `--output=files` option to output the
> +  number of files associated with matching messages. This may be
> +  bigger than the number of matching messages due to duplicates
> +  (i.e. multiple files having the same message-id).
> +
> +
>  Notmuch 0.16 (2013-08-03)
>  =========================
>  
> -- 
> 1.7.10.4
>
> _______________________________________________
> notmuch mailing list
> notmuch@notmuchmail.org
> http://notmuchmail.org/mailman/listinfo/notmuch

[PATCH v4 7/8] NEWS: cli options for improved duplicate message handling

[Jani Nikula]

---
 NEWS |   20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)

diff --git a/NEWS b/NEWS
index 722a35d..2e845f4 100644
--- a/NEWS
+++ b/NEWS
@@ -1,3 +1,23 @@
+
+Notmuch 0.17 (UNRELEASED)
+=========================
+
+Command-Line Interface
+----------------------
+
+New options to better support handling duplicate messages
+
+  If more than one message file is associated with a message-id,
+  `notmuch search --output=files` will print all of them. A new
+  `--duplicate=N` option can be used to specify which duplicate to
+  print for each message.
+
+  `notmuch count` now supports `--output=files` option to output the
+  number of files associated with matching messages. This may be
+  bigger than the number of matching messages due to duplicates
+  (i.e. multiple files having the same message-id).
+
+
 Notmuch 0.16 (2013-08-03)
 =========================
 
-- 
1.7.10.4

[PATCH v3 8/8] man: clarify folder: prefix regarding duplicate files

[Jani Nikula]

The results for folder: prefix are a source of recurring confusion.
---
 man/man7/notmuch-search-terms.7 |    8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)

diff --git a/man/man7/notmuch-search-terms.7 b/man/man7/notmuch-search-terms.7
index 314efa8..b5eb195 100644
--- a/man/man7/notmuch-search-terms.7
+++ b/man/man7/notmuch-search-terms.7
@@ -102,9 +102,11 @@ thread ID values can be seen in the first column of output from
 The
 .B folder:
 prefix can be used to search for email message files that are
-contained within particular directories within the mail store. Only
-the directory components below the top-level mail database path are
-available to be searched.
+contained within particular directories within the mail store. If the
+same email message has multiple message files associated with it, it's
+sufficient for a match that at least one of the files is contained
+within a matching directory. Only the directory components below the
+top-level mail database path are available to be searched.
 
 The
 .B date:
-- 
1.7.10.4

Re: [PATCH v3 0/8] cli: better tooling support for duplicate messages

[Tomi Ollila]

On Sat, Aug 17 2013, Jani Nikula <jani@nikula.org> wrote:

> v3 of id:cover.1370775663.git.jani@nikula.org. Just a rebase, no changes
> since the latest versions of the patches posted in the previous thread.
>
> BR,
> Jani.

This patch series LGTM. Tests pass.

>
> Jani Nikula (8):
>   cli: add --duplicate=N option to notmuch search
>   test: test notmuch search --duplicate=N
>   man: document notmuch search --duplicate=N
>   cli: add --output=files option to notmuch count
>   test: test notmuch count --output=files
>   man: document notmuch count --output=files
>   NEWS: cli options for improved duplicate message handling
>   man: clarify folder: prefix regarding duplicate files
>
>  NEWS                            |   20 ++++++++++++
>  man/man1/notmuch-count.1        |   10 +++++-
>  man/man1/notmuch-search.1       |   20 ++++++++++++
>  man/man7/notmuch-search-terms.7 |    8 +++--
>  notmuch-count.c                 |   37 ++++++++++++++++++++++
>  notmuch-search.c                |   18 +++++++----
>  test/count                      |   10 ++++++
>  test/search-output              |   65 +++++++++++++++++++++++++++++++++++++++
>  8 files changed, 178 insertions(+), 10 deletions(-)
>
> -- 
> 1.7.10.4
>
> _______________________________________________
> notmuch mailing list
> notmuch@notmuchmail.org
> http://notmuchmail.org/mailman/listinfo/notmuch

Re: [PATCH v3 0/8] cli: better tooling support for duplicate messages

[Mark Walters]

This series LGTM +1.

Best wishes

Mark

On Sat, 17 Aug 2013, Jani Nikula <jani@nikula.org> wrote:
> v3 of id:cover.1370775663.git.jani@nikula.org. Just a rebase, no changes
> since the latest versions of the patches posted in the previous thread.
>
> BR,
> Jani.
>
> Jani Nikula (8):
>   cli: add --duplicate=N option to notmuch search
>   test: test notmuch search --duplicate=N
>   man: document notmuch search --duplicate=N
>   cli: add --output=files option to notmuch count
>   test: test notmuch count --output=files
>   man: document notmuch count --output=files
>   NEWS: cli options for improved duplicate message handling
>   man: clarify folder: prefix regarding duplicate files
>
>  NEWS                            |   20 ++++++++++++
>  man/man1/notmuch-count.1        |   10 +++++-
>  man/man1/notmuch-search.1       |   20 ++++++++++++
>  man/man7/notmuch-search-terms.7 |    8 +++--
>  notmuch-count.c                 |   37 ++++++++++++++++++++++
>  notmuch-search.c                |   18 +++++++----
>  test/count                      |   10 ++++++
>  test/search-output              |   65 +++++++++++++++++++++++++++++++++++++++
>  8 files changed, 178 insertions(+), 10 deletions(-)
>
> -- 
> 1.7.10.4
>
> _______________________________________________
> notmuch mailing list
> notmuch@notmuchmail.org
> http://notmuchmail.org/mailman/listinfo/notmuch

Re: [PATCH v3 0/8] cli: better tooling support for duplicate messages

[David Bremner]

Jani Nikula <jani@nikula.org> writes:

> v3 of id:cover.1370775663.git.jani@nikula.org. Just a rebase, no
> changes since the latest versions of the patches posted in the
> previous thread.

pushed, 

d