[PATCH 0/6] cli: better tooling support for duplicate messages

[PATCH 0/6] cli: better tooling support for duplicate messages

[Jani Nikula]

Patches 1-3 are an answer to a question on IRC:

<ccxCZ> is there a way for notmuch search --output=files to only output
        one file for Message-ID instead of all?

Yes, use --duplicate=0 on the command line. ;)

I suggest this interface (instead of, say, --output=file1 which was also
contemplated) for two reasons: it allows one to choose also other files
than the first one, and the same interface could be used later on in
notmuch show to display the duplicates.

An alternative could be something like --duplicates=(all|first|rest)
which would lack the possibility to pick a specific duplicate, but would
allow one to list all the "redundant" files with --duplicates=rest (this
would correspond to a number of invocations with --duplicate=1,
--duplicate=2, etc.)

Patches 4-6 add support for querying the file counts in notmuch
count. Most useful with a single id:<message-id> query I presume.


BR,
Jani.


Jani Nikula (6):
  cli: add --duplicate=N option to notmuch search
  test: test notmuch search --duplicate=N
  man: document notmuch search --duplicate=N
  cli: add --output=files option to notmuch count
  test: test notmuch count --output=files
  man: document notmuch count --output=files

 man/man1/notmuch-count.1  |   10 ++++++-
 man/man1/notmuch-search.1 |   11 ++++++++
 notmuch-count.c           |   37 ++++++++++++++++++++++++++
 notmuch-search.c          |   18 ++++++++-----
 test/count                |   10 +++++++
 test/search-output        |   65 +++++++++++++++++++++++++++++++++++++++++++++
 6 files changed, 144 insertions(+), 7 deletions(-)

-- 
1.7.10.4

[PATCH 1/6] cli: add --duplicate=N option to notmuch search

[Jani Nikula]

Effective with --output=files, output the Nth filename associated with
each message matching the query (N is 0-based). If N is equal to or
greater than the number of files associated with the message, don't
print anything.
---
 notmuch-search.c |   18 ++++++++++++------
 1 file changed, 12 insertions(+), 6 deletions(-)

diff --git a/notmuch-search.c b/notmuch-search.c
index e658639..b113883 100644
--- a/notmuch-search.c
+++ b/notmuch-search.c
@@ -177,7 +177,8 @@ do_search_messages (sprinter_t *format,
 		    notmuch_query_t *query,
 		    output_t output,
 		    int offset,
-		    int limit)
+		    int limit,
+		    int dupe)
 {
     notmuch_message_t *message;
     notmuch_messages_t *messages;
@@ -206,14 +207,17 @@ do_search_messages (sprinter_t *format,
 	message = notmuch_messages_get (messages);
 
 	if (output == OUTPUT_FILES) {
+	    int j;
 	    filenames = notmuch_message_get_filenames (message);
 
-	    for (;
+	    for (j = 0;
 		 notmuch_filenames_valid (filenames);
-		 notmuch_filenames_move_to_next (filenames))
+		 notmuch_filenames_move_to_next (filenames), j++)
 	    {
-		format->string (format, notmuch_filenames_get (filenames));
-		format->separator (format);
+		if (dupe < 0 || dupe == j) {
+		    format->string (format, notmuch_filenames_get (filenames));
+		    format->separator (format);
+		}
 	    }
 	    
 	    notmuch_filenames_destroy( filenames );
@@ -302,6 +306,7 @@ notmuch_search_command (notmuch_config_t *config, int argc, char *argv[])
     int offset = 0;
     int limit = -1; /* unlimited */
     int exclude = EXCLUDE_TRUE;
+    int dupe = -1;
     unsigned int i;
 
     enum {
@@ -337,6 +342,7 @@ notmuch_search_command (notmuch_config_t *config, int argc, char *argv[])
                                   { 0, 0 } } },
 	{ NOTMUCH_OPT_INT, &offset, "offset", 'O', 0 },
 	{ NOTMUCH_OPT_INT, &limit, "limit", 'L', 0  },
+	{ NOTMUCH_OPT_INT, &dupe, "duplicate", 'D', 0  },
 	{ 0, 0, 0, 0, 0 }
     };
 
@@ -420,7 +426,7 @@ notmuch_search_command (notmuch_config_t *config, int argc, char *argv[])
 	break;
     case OUTPUT_MESSAGES:
     case OUTPUT_FILES:
-	ret = do_search_messages (format, query, output, offset, limit);
+	ret = do_search_messages (format, query, output, offset, limit, dupe);
 	break;
     case OUTPUT_TAGS:
 	ret = do_search_tags (notmuch, format, query);
-- 
1.7.10.4

[PATCH 2/6] test: test notmuch search --duplicate=N

[Jani Nikula]

---
 test/search-output |   65 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 65 insertions(+)

diff --git a/test/search-output b/test/search-output
index c2a87eb..3268833 100755
--- a/test/search-output
+++ b/test/search-output
@@ -239,6 +239,64 @@ MAIL_DIR/cur/01:2,
 EOF
 test_expect_equal_file OUTPUT EXPECTED
 
+test_begin_subtest "--output=files --duplicate=0"
+notmuch search --output=files --duplicate=0 '*' | sed -e "s,$MAIL_DIR,MAIL_DIR," >OUTPUT
+cat <<EOF >EXPECTED
+MAIL_DIR/cur/52:2,
+MAIL_DIR/cur/53:2,
+MAIL_DIR/cur/50:2,
+MAIL_DIR/cur/49:2,
+MAIL_DIR/cur/48:2,
+MAIL_DIR/cur/47:2,
+MAIL_DIR/cur/46:2,
+MAIL_DIR/cur/45:2,
+MAIL_DIR/cur/44:2,
+MAIL_DIR/cur/43:2,
+MAIL_DIR/cur/42:2,
+MAIL_DIR/cur/41:2,
+MAIL_DIR/cur/40:2,
+MAIL_DIR/cur/39:2,
+MAIL_DIR/cur/38:2,
+MAIL_DIR/cur/37:2,
+MAIL_DIR/cur/36:2,
+MAIL_DIR/cur/35:2,
+MAIL_DIR/cur/34:2,
+MAIL_DIR/cur/33:2,
+MAIL_DIR/cur/32:2,
+MAIL_DIR/cur/31:2,
+MAIL_DIR/cur/30:2,
+MAIL_DIR/cur/29:2,
+MAIL_DIR/cur/28:2,
+MAIL_DIR/cur/27:2,
+MAIL_DIR/cur/26:2,
+MAIL_DIR/cur/25:2,
+MAIL_DIR/cur/24:2,
+MAIL_DIR/cur/23:2,
+MAIL_DIR/cur/22:2,
+MAIL_DIR/cur/21:2,
+MAIL_DIR/cur/19:2,
+MAIL_DIR/cur/18:2,
+MAIL_DIR/cur/20:2,
+MAIL_DIR/cur/17:2,
+MAIL_DIR/cur/16:2,
+MAIL_DIR/cur/15:2,
+MAIL_DIR/cur/14:2,
+MAIL_DIR/cur/13:2,
+MAIL_DIR/cur/12:2,
+MAIL_DIR/cur/11:2,
+MAIL_DIR/cur/10:2,
+MAIL_DIR/cur/09:2,
+MAIL_DIR/cur/08:2,
+MAIL_DIR/cur/06:2,
+MAIL_DIR/cur/05:2,
+MAIL_DIR/cur/04:2,
+MAIL_DIR/cur/03:2,
+MAIL_DIR/cur/07:2,
+MAIL_DIR/cur/02:2,
+MAIL_DIR/cur/01:2,
+EOF
+test_expect_equal_file OUTPUT EXPECTED
+
 test_begin_subtest "--output=files --format=json"
 notmuch search --format=json --output=files '*' | sed -e "s,$MAIL_DIR,MAIL_DIR," >OUTPUT
 cat <<EOF >EXPECTED
@@ -298,6 +356,13 @@ cat <<EOF >EXPECTED
 EOF
 test_expect_equal_file OUTPUT EXPECTED
 
+test_begin_subtest "--output=files --format=json --duplicate=1"
+notmuch search --format=json --output=files --duplicate=1 '*' | sed -e "s,$MAIL_DIR,MAIL_DIR," >OUTPUT
+cat <<EOF >EXPECTED
+["MAIL_DIR/cur/51:2,"]
+EOF
+test_expect_equal_file OUTPUT EXPECTED
+
 test_begin_subtest "--output=tags"
 notmuch search --output=tags '*' >OUTPUT
 cat <<EOF >EXPECTED
-- 
1.7.10.4

[PATCH 3/6] man: document notmuch search --duplicate=N

[Jani Nikula]

---
 man/man1/notmuch-search.1 |   11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/man/man1/notmuch-search.1 b/man/man1/notmuch-search.1
index da2f1dd..d2a582d 100644
--- a/man/man1/notmuch-search.1
+++ b/man/man1/notmuch-search.1
@@ -154,6 +154,17 @@ but the "match count" is the number of matching non-excluded messages in the
 thread, rather than the number of matching messages.
 .RE
 
+.RS 4
+.TP 4
+.BR \-\-duplicate=N
+
+Effective with
+.BR --output=files ,
+output the Nth filename associated with each message matching the
+query (N is 0-based). If N is equal to or greater than the number of
+files associated with the message, don't print anything.
+.RE
+
 .SH EXIT STATUS
 
 This command supports the following special exit status codes
-- 
1.7.10.4

[PATCH 4/6] cli: add --output=files option to notmuch count

[Jani Nikula]

Add support for querying the total number of files associated with the
messages matching the search. This is mostly useful with an
id:<message-id> query for a single message.
---
 notmuch-count.c |   37 +++++++++++++++++++++++++++++++++++++
 1 file changed, 37 insertions(+)

diff --git a/notmuch-count.c b/notmuch-count.c
index 8772cff..01e4e30 100644
--- a/notmuch-count.c
+++ b/notmuch-count.c
@@ -24,6 +24,7 @@
 enum {
     OUTPUT_THREADS,
     OUTPUT_MESSAGES,
+    OUTPUT_FILES,
 };
 
 /* The following is to allow future options to be added more easily */
@@ -32,6 +33,38 @@ enum {
     EXCLUDE_FALSE,
 };
 
+static unsigned int
+count_files (notmuch_query_t *query)
+{
+    notmuch_messages_t *messages;
+    notmuch_message_t *message;
+    notmuch_filenames_t *filenames;
+    unsigned int count = 0;
+
+    messages = notmuch_query_search_messages (query);
+    if (messages == NULL)
+	return 0;
+
+    for (;
+	 notmuch_messages_valid (messages);
+	 notmuch_messages_move_to_next (messages)) {
+	message = notmuch_messages_get (messages);
+	filenames = notmuch_message_get_filenames (message);
+
+	for (;
+	     notmuch_filenames_valid (filenames);
+	     notmuch_filenames_move_to_next (filenames))
+	    count++;
+
+	notmuch_filenames_destroy (filenames);
+	notmuch_message_destroy (message);
+    }
+
+    notmuch_messages_destroy (messages);
+
+    return count;
+}
+
 static int
 print_count (notmuch_database_t *notmuch, const char *query_str,
 	     const char **exclude_tags, size_t exclude_tags_length, int output)
@@ -55,6 +88,9 @@ print_count (notmuch_database_t *notmuch, const char *query_str,
     case OUTPUT_THREADS:
 	printf ("%u\n", notmuch_query_count_threads (query));
 	break;
+    case OUTPUT_FILES:
+	printf ("%u\n", count_files (query));
+	break;
     }
 
     notmuch_query_destroy (query);
@@ -102,6 +138,7 @@ notmuch_count_command (notmuch_config_t *config, int argc, char *argv[])
 	{ NOTMUCH_OPT_KEYWORD, &output, "output", 'o',
 	  (notmuch_keyword_t []){ { "threads", OUTPUT_THREADS },
 				  { "messages", OUTPUT_MESSAGES },
+				  { "files", OUTPUT_FILES },
 				  { 0, 0 } } },
 	{ NOTMUCH_OPT_KEYWORD, &exclude, "exclude", 'x',
 	  (notmuch_keyword_t []){ { "true", EXCLUDE_TRUE },
-- 
1.7.10.4

[PATCH 5/6] test: test notmuch count --output=files

[Jani Nikula]

---
 test/count |   10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/test/count b/test/count
index 05713fd..da86c8c 100755
--- a/test/count
+++ b/test/count
@@ -28,6 +28,16 @@ test_expect_equal \
     "$((`notmuch search '*' | wc -l`))" \
     "`notmuch count --output=threads '*'`"
 
+test_begin_subtest "files count"
+test_expect_equal \
+    "$((`notmuch search --output=files '*' | wc -l`))" \
+    "`notmuch count --output=files '*'`"
+
+test_begin_subtest "files count for a duplicate message-id"
+test_expect_equal \
+    "2" \
+    "`notmuch count --output=files id:20091117232137.GA7669@griffis1.net`"
+
 test_begin_subtest "count with no matching messages"
 test_expect_equal \
     "0" \
-- 
1.7.10.4

[PATCH 6/6] man: document notmuch count --output=files

[Jani Nikula]

---
 man/man1/notmuch-count.1 |   10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)

diff --git a/man/man1/notmuch-count.1 b/man/man1/notmuch-count.1
index 7fc4378..60af631 100644
--- a/man/man1/notmuch-count.1
+++ b/man/man1/notmuch-count.1
@@ -23,7 +23,7 @@ Supported options for
 include
 .RS 4
 .TP 4
-.B \-\-output=(messages|threads)
+.B \-\-output=(messages|threads|files)
 
 .RS 4
 .TP 4
@@ -37,6 +37,14 @@ Output the number of matching messages. This is the default.
 
 Output the number of matching threads.
 .RE
+.RS 4
+.TP 4
+.B files
+
+Output the number of files associated with matching messages. This may
+be bigger than the number of matching messages due to duplicates
+(i.e. multiple files having the same message-id).
+.RE
 .RE
 
 .RS 4
-- 
1.7.10.4

Re: [PATCH 1/6] cli: add --duplicate=N option to notmuch search

[David Bremner]

Jani Nikula <jani@nikula.org> writes:

> Effective with --output=files, output the Nth filename associated with
> each message matching the query (N is 0-based). If N is equal to or
> greater than the number of files associated with the message, don't
> print anything.

I guess this is a proverbial bikeshed, but 1 based indexing feels more
natural to me here.

Re: [PATCH 6/6] man: document notmuch count --output=files

[David Bremner]

Other than the bikeshed about starting from zero instead of one, and not
being completely happy with the option name --duplicate (but not being
inspired to suggest a better one), this series looks OK to me.

It seems like it would be quite useful to query based on the number of
duplicates, but that is a much more ambitious task, I think.

d

Re: [PATCH 6/6] man: document notmuch count --output=files

[Mark Walters]

David Bremner <david@tethera.net> writes:

> Other than the bikeshed about starting from zero instead of one, and not
> being completely happy with the option name --duplicate (but not being
> inspired to suggest a better one), this series looks OK to me.
>
> It seems like it would be quite useful to query based on the number of
> duplicates, but that is a much more ambitious task, I think.

This series basically looks good to me too. I agree with all of the
above points though. 

Best wishes

Mark