[RFC PATCH v5 00/11] Add NOTMUCH_MESSAGE_FLAG_EXCLUDED flag

[RFC PATCH v5 00/11] Add NOTMUCH_MESSAGE_FLAG_EXCLUDED flag

[Mark Walters]

This is v5 of the exclude flag series. (v4 was at 
id:"874nv9rv79.fsf@qmul.ac.uk")

This email has 4 sections, an overview of what the patch set is 
trying to achieve, a summary of the changes from v4, some comments on 
the status of the patches in the series and some remaining queries.

OVERVIEW

The current implementation of exclude-tags does not use excludes in
notmuch-show.c (and thus not in notmuch-show.el). Thus when selecting 
a thread in the search view claiming one matched message you may get 
several matches in show all but one of which are tagged excluded.

The trivial change of adding excludes to show does not work as easily 
as one would like. For example if you try notmuch-show 
id:<deleted message> you get no results (see 
id:"871uqvgrnm.fsf@qmul.ac.uk" for more discussion).

This set moves in a different direction. It returns all the results 
but marks excluded messages with a new flag 
(NOTMUCH_MESSAGE_FLAG_EXCLUDED) and lets the consumer decide what to 
do with them. For example it could start with the message closed in 
emacs show view, it could colour the headerline differently etc.

CHANGES:
This has been rebased on top of Jani's notmuch-show command line 
parsing patch.

The function notmuch_thread_get_flag_messages function added in v4
has been removed. Unfortunately (as Austin pointed out) that patch 
broke binary compatibility so should be deferred until we wish to 
bump the library version.

I also fixed a minor bug and a style comment in the emacs part of 
the patch.

STATUS:
The first 3 patches in the series just add a --no-exclude option
to notmuch-search.c and notmuch-count.c to "turn off" the excluding.
(The 3 patches are one for the C code, one for the man pages and 
one for the tests.) I think this change is desirable independently 
of the rest of the series (and indeed Jameson had a use for it
id:"878vk943ci.fsf@servo.finestructure.net").

QUERIES

1) As with v4 the api notmuch_query_set_omit_excluded_messages
remains: without it I can't see how a user can pass a 
notmuch_messages_t object around which does not contain the 
excluded messages. See id:"87fweusabh.fsf@qmul.ac.uk". 

2)  If we have a query which overrides the excludes such as "blah and
tag:deleted" should the tag:deleted messages still be marked excluded?
The current implementation does mark them excluded but my preference 
would be not to. What do people think? At the moment 2 tests fail but 
the correct output depends on the above so I will leave them until we
have a decision on this.

There are some other queries mentioned in v4 but the two above are
the significant ones.

Best wishes

Mark



Mark Walters (11):
  cli: add --no-exclude option to count and search.
  cli: Add --no-exclude to the man pages for search and count
  test: add tests for new cli --no-exclude option
  lib: Rearrange the exclude code in query.cc
  lib: Make notmuch_query_search_messages set the exclude flag
  lib: Add the exclude flag to notmuch_query_search_threads
  cli: Make notmuch-show respect excludes.
  man: update manpage for notmuch-show --no-exclude option
  test: update tests to reflect the exclude flag
  cli: omit excluded messages in results where appropriate.
  emacs: show: recognize the exclude flag.

 emacs/notmuch-show.el     |   19 +++++++++++-
 lib/notmuch-private.h     |    8 ++++-
 lib/notmuch.h             |   16 ++++++++--
 lib/query.cc              |   74 ++++++++++++++++++++++++++++++++++++++------
 lib/thread.cc             |   18 +++++++++--
 man/man1/notmuch-count.1  |    7 ++++
 man/man1/notmuch-search.1 |    7 ++++
 man/man1/notmuch-show.1   |    7 ++++
 notmuch-count.c           |   19 ++++++++----
 notmuch-search.c          |   26 ++++++++++++----
 notmuch-show.c            |   31 ++++++++++++++++--
 test/count                |   21 +++++++++++++
 test/crypto               |    9 +++++-
 test/encoding             |    2 +-
 test/json                 |    6 ++--
 test/maildir-sync         |    1 +
 test/multipart            |    4 +-
 test/search               |    5 +++
 test/thread-naming        |   16 +++++-----
 19 files changed, 246 insertions(+), 50 deletions(-)

-- 
1.7.2.3

[RFC PATCH v5 01/11] cli: add --no-exclude option to count and search.

[Mark Walters]

This option turns off the exclusion so all matching messages are
returned. We do not need to add this to notmuch-show as that does not
(yet) exclude.
---
 notmuch-count.c  |   17 +++++++++++------
 notmuch-search.c |   17 +++++++++++------
 2 files changed, 22 insertions(+), 12 deletions(-)

diff --git a/notmuch-count.c b/notmuch-count.c
index 63459fb..5364507 100644
--- a/notmuch-count.c
+++ b/notmuch-count.c
@@ -35,8 +35,7 @@ notmuch_count_command (void *ctx, int argc, char *argv[])
     char *query_str;
     int opt_index;
     int output = OUTPUT_MESSAGES;
-    const char **search_exclude_tags;
-    size_t search_exclude_tags_length;
+    notmuch_bool_t no_exclude = FALSE;
     unsigned int i;
 
     notmuch_opt_desc_t options[] = {
@@ -44,6 +43,7 @@ notmuch_count_command (void *ctx, int argc, char *argv[])
 	  (notmuch_keyword_t []){ { "threads", OUTPUT_THREADS },
 				  { "messages", OUTPUT_MESSAGES },
 				  { 0, 0 } } },
+	{ NOTMUCH_OPT_BOOLEAN, &no_exclude, "no-exclude", 'd', 0 },
 	{ 0, 0, 0, 0, 0 }
     };
 
@@ -78,10 +78,15 @@ notmuch_count_command (void *ctx, int argc, char *argv[])
 	return 1;
     }
 
-    search_exclude_tags = notmuch_config_get_search_exclude_tags
-	(config, &search_exclude_tags_length);
-    for (i = 0; i < search_exclude_tags_length; i++)
-	notmuch_query_add_tag_exclude (query, search_exclude_tags[i]);
+    if (!no_exclude) {
+	const char **search_exclude_tags;
+	size_t search_exclude_tags_length;
+
+	search_exclude_tags = notmuch_config_get_search_exclude_tags
+	    (config, &search_exclude_tags_length);
+	for (i = 0; i < search_exclude_tags_length; i++)
+	    notmuch_query_add_tag_exclude (query, search_exclude_tags[i]);
+    }
 
     switch (output) {
     case OUTPUT_MESSAGES:
diff --git a/notmuch-search.c b/notmuch-search.c
index d504051..43ec90b 100644
--- a/notmuch-search.c
+++ b/notmuch-search.c
@@ -423,8 +423,7 @@ notmuch_search_command (void *ctx, int argc, char *argv[])
     output_t output = OUTPUT_SUMMARY;
     int offset = 0;
     int limit = -1; /* unlimited */
-    const char **search_exclude_tags;
-    size_t search_exclude_tags_length;
+    notmuch_bool_t no_exclude = FALSE;
     unsigned int i;
 
     enum { NOTMUCH_FORMAT_JSON, NOTMUCH_FORMAT_TEXT }
@@ -446,6 +445,7 @@ notmuch_search_command (void *ctx, int argc, char *argv[])
 				  { "files", OUTPUT_FILES },
 				  { "tags", OUTPUT_TAGS },
 				  { 0, 0 } } },
+        { NOTMUCH_OPT_BOOLEAN, &no_exclude, "no-exclude", 'd', 0 },
 	{ NOTMUCH_OPT_INT, &offset, "offset", 'O', 0 },
 	{ NOTMUCH_OPT_INT, &limit, "limit", 'L', 0  },
 	{ 0, 0, 0, 0, 0 }
@@ -493,10 +493,15 @@ notmuch_search_command (void *ctx, int argc, char *argv[])
 
     notmuch_query_set_sort (query, sort);
 
-    search_exclude_tags = notmuch_config_get_search_exclude_tags
-	(config, &search_exclude_tags_length);
-    for (i = 0; i < search_exclude_tags_length; i++)
-	notmuch_query_add_tag_exclude (query, search_exclude_tags[i]);
+    if (!no_exclude) {
+	const char **search_exclude_tags;
+	size_t search_exclude_tags_length;
+
+	search_exclude_tags = notmuch_config_get_search_exclude_tags
+	    (config, &search_exclude_tags_length);
+	for (i = 0; i < search_exclude_tags_length; i++)
+	    notmuch_query_add_tag_exclude (query, search_exclude_tags[i]);
+    }
 
     switch (output) {
     default:
-- 
1.7.2.3

[RFC PATCH v5 02/11] cli: Add --no-exclude to the man pages for search and count

[Mark Walters]

---
 man/man1/notmuch-count.1  |    7 +++++++
 man/man1/notmuch-search.1 |    7 +++++++
 2 files changed, 14 insertions(+), 0 deletions(-)

diff --git a/man/man1/notmuch-count.1 b/man/man1/notmuch-count.1
index 25fe329..413b405 100644
--- a/man/man1/notmuch-count.1
+++ b/man/man1/notmuch-count.1
@@ -38,6 +38,13 @@ Output the number of matching messages. This is the default.
 Output the number of matching threads.
 .RE
 .RE
+
+.RS 4
+.TP 4
+.BR \-\-no\-exclude
+
+Do not exclude the messages matching search_exclude_tags in the config file.
+.RE
 .RE
 .RE
 
diff --git a/man/man1/notmuch-search.1 b/man/man1/notmuch-search.1
index 0bc3f40..bc54b4d 100644
--- a/man/man1/notmuch-search.1
+++ b/man/man1/notmuch-search.1
@@ -112,6 +112,13 @@ result from the end.
 Limit the number of displayed results to N.
 .RE
 
+.RS 4
+.TP 4
+.BR \-\-no\-exclude
+
+Do not exclude the messages matching search_exclude_tags in the config file.
+.RE
+
 .SH SEE ALSO
 
 \fBnotmuch\fR(1), \fBnotmuch-config\fR(1), \fBnotmuch-count\fR(1),
-- 
1.7.2.3

[RFC PATCH v5 03/11] test: add tests for new cli --no-exclude option

[Mark Walters]

The tests test the new --no-exclude option to search and count.
There were no existing tests for the exclude behaviour for count so
added these too.
---
 test/count  |   21 +++++++++++++++++++++
 test/search |    5 +++++
 2 files changed, 26 insertions(+), 0 deletions(-)

diff --git a/test/count b/test/count
index 300b171..976fff1 100755
--- a/test/count
+++ b/test/count
@@ -37,4 +37,25 @@ test_expect_equal \
     "0" \
     "`notmuch count --output=threads ${SEARCH}`"
 
+test_begin_subtest "count excluding \"deleted\" messages"
+notmuch config set search.exclude_tags = deleted
+generate_message '[subject]="Not deleted"'
+generate_message '[subject]="Another not deleted"'
+generate_message '[subject]="Deleted"'
+notmuch new > /dev/null
+notmuch tag +deleted id:$gen_msg_id
+test_expect_equal \
+    "2" \
+    "`notmuch count subject:deleted`"
+
+test_begin_subtest "count \"deleted\" messages, exclude overridden"
+test_expect_equal \
+    "1" \
+    "`notmuch count subject:deleted and tag:deleted`"
+
+test_begin_subtest "count \"deleted\" messages, with --no-exclude"
+test_expect_equal \
+    "3" \
+    "`notmuch count --no-exclude subject:deleted`"
+
 test_done
diff --git a/test/search b/test/search
index 414be35..3da5d17 100755
--- a/test/search
+++ b/test/search
@@ -148,6 +148,11 @@ output=$(notmuch search subject:deleted | notmuch_search_sanitize)
 test_expect_equal "$output" "thread:XXX   2001-01-05 [1/1] Notmuch Test Suite; Not deleted (inbox unread)
 thread:XXX   2001-01-05 [1/2] Notmuch Test Suite; Not deleted reply (deleted inbox unread)"
 
+test_begin_subtest "Don't exclude \"deleted\" messages when --no-exclude specified"
+output=$(notmuch search --no-exclude subject:deleted | notmuch_search_sanitize)
+test_expect_equal "$output" "thread:XXX   2001-01-05 [1/1] Notmuch Test Suite; Not deleted (inbox unread)
+thread:XXX   2001-01-05 [2/2] Notmuch Test Suite; Deleted (deleted inbox unread)"
+
 test_begin_subtest "Don't exclude \"deleted\" messages from search if not configured"
 notmuch config set search.exclude_tags
 output=$(notmuch search subject:deleted | notmuch_search_sanitize)
-- 
1.7.2.3

[RFC PATCH v5 04/11] lib: Rearrange the exclude code in query.cc

[Mark Walters]

Slightly refactor the exclude code to give the callers access to the
exclude query itself. There should be no functional change.
---
 lib/query.cc |   29 +++++++++++++++++++----------
 1 files changed, 19 insertions(+), 10 deletions(-)

diff --git a/lib/query.cc b/lib/query.cc
index 0b36602..c25b301 100644
--- a/lib/query.cc
+++ b/lib/query.cc
@@ -122,12 +122,15 @@ _notmuch_messages_destructor (notmuch_mset_messages_t *messages)
     return 0;
 }
 
-/* Return a query that does not match messages with the excluded tags
- * registered with the query.  Any tags that explicitly appear in
- * xquery will not be excluded. */
+/* Return a query that matches messages with the excluded tags
+ * registered with query.  Any tags that explicitly appear in xquery
+ * will not be excluded. The caller of this function has to combine
+ * the returned query appropriately.*/
 static Xapian::Query
 _notmuch_exclude_tags (notmuch_query_t *query, Xapian::Query xquery)
 {
+    Xapian::Query exclude_query = Xapian::Query::MatchNothing;
+
     for (notmuch_string_node_t *term = query->exclude_terms->head; term;
 	 term = term->next) {
 	Xapian::TermIterator it = xquery.get_terms_begin ();
@@ -137,10 +140,10 @@ _notmuch_exclude_tags (notmuch_query_t *query, Xapian::Query xquery)
 		break;
 	}
 	if (it == end)
-	    xquery = Xapian::Query (Xapian::Query::OP_AND_NOT,
-				    xquery, Xapian::Query (term->string));
+	    exclude_query = Xapian::Query (Xapian::Query::OP_OR,
+				    exclude_query, Xapian::Query (term->string));
     }
-    return xquery;
+    return exclude_query;
 }
 
 notmuch_messages_t *
@@ -168,7 +171,7 @@ notmuch_query_search_messages (notmuch_query_t *query)
 	Xapian::Query mail_query (talloc_asprintf (query, "%s%s",
 						   _find_prefix ("type"),
 						   "mail"));
-	Xapian::Query string_query, final_query;
+	Xapian::Query string_query, final_query, exclude_query;
 	Xapian::MSet mset;
 	unsigned int flags = (Xapian::QueryParser::FLAG_BOOLEAN |
 			      Xapian::QueryParser::FLAG_PHRASE |
@@ -188,7 +191,10 @@ notmuch_query_search_messages (notmuch_query_t *query)
 					 mail_query, string_query);
 	}
 
-	final_query = _notmuch_exclude_tags (query, final_query);
+	exclude_query = _notmuch_exclude_tags (query, final_query);
+
+	final_query = Xapian::Query (Xapian::Query::OP_AND_NOT,
+					 final_query, exclude_query);
 
 	enquire.set_weighting_scheme (Xapian::BoolWeight());
 
@@ -449,7 +455,7 @@ notmuch_query_count_messages (notmuch_query_t *query)
 	Xapian::Query mail_query (talloc_asprintf (query, "%s%s",
 						   _find_prefix ("type"),
 						   "mail"));
-	Xapian::Query string_query, final_query;
+	Xapian::Query string_query, final_query, exclude_query;
 	Xapian::MSet mset;
 	unsigned int flags = (Xapian::QueryParser::FLAG_BOOLEAN |
 			      Xapian::QueryParser::FLAG_PHRASE |
@@ -469,7 +475,10 @@ notmuch_query_count_messages (notmuch_query_t *query)
 					 mail_query, string_query);
 	}
 
-	final_query = _notmuch_exclude_tags (query, final_query);
+	exclude_query = _notmuch_exclude_tags (query, final_query);
+
+	final_query = Xapian::Query (Xapian::Query::OP_AND_NOT,
+					 final_query, exclude_query);
 
 	enquire.set_weighting_scheme(Xapian::BoolWeight());
 	enquire.set_docid_order(Xapian::Enquire::ASCENDING);
-- 
1.7.2.3

[RFC PATCH v5 05/11] lib: Make notmuch_query_search_messages set the exclude flag

[Mark Walters]

Add a flag NOTMUCH_MESSAGE_FLAG_EXCLUDED which is set by
notmuch_query_search_messages for excluded messages. Also add an
option omit_excluded_messages to the search that we do not want the
excludes at all.

This exclude flag will be added to notmuch_query_search threads in the
next patch.
---
 lib/notmuch-private.h |    1 +
 lib/notmuch.h         |   10 ++++++++-
 lib/query.cc          |   52 +++++++++++++++++++++++++++++++++++++++++++++---
 3 files changed, 58 insertions(+), 5 deletions(-)

diff --git a/lib/notmuch-private.h b/lib/notmuch-private.h
index 7bf153e..e791bb0 100644
--- a/lib/notmuch-private.h
+++ b/lib/notmuch-private.h
@@ -401,6 +401,7 @@ typedef struct _notmuch_message_list {
  */
 struct visible _notmuch_messages {
     notmuch_bool_t is_of_list_type;
+    notmuch_doc_id_set_t *excluded_doc_ids;
     notmuch_message_node_t *iterator;
 };
 
diff --git a/lib/notmuch.h b/lib/notmuch.h
index 7929fe7..f75afae 100644
--- a/lib/notmuch.h
+++ b/lib/notmuch.h
@@ -449,6 +449,13 @@ typedef enum {
 const char *
 notmuch_query_get_query_string (notmuch_query_t *query);
 
+/* Specify whether to results should omit the excluded results rather
+ * than just marking them excluded. This is useful for passing a
+ * notmuch_messages_t not containing the excluded messages to other
+ * functions. */
+void
+notmuch_query_set_omit_excluded_messages (notmuch_query_t *query, notmuch_bool_t omit);
+
 /* Specify the sorting desired for this query. */
 void
 notmuch_query_set_sort (notmuch_query_t *query, notmuch_sort_t sort);
@@ -895,7 +902,8 @@ notmuch_message_get_filenames (notmuch_message_t *message);
 
 /* Message flags */
 typedef enum _notmuch_message_flag {
-    NOTMUCH_MESSAGE_FLAG_MATCH
+    NOTMUCH_MESSAGE_FLAG_MATCH,
+    NOTMUCH_MESSAGE_FLAG_EXCLUDED
 } notmuch_message_flag_t;
 
 /* Get a value of a flag for the email corresponding to 'message'. */
diff --git a/lib/query.cc b/lib/query.cc
index c25b301..90a71a1 100644
--- a/lib/query.cc
+++ b/lib/query.cc
@@ -28,6 +28,7 @@ struct _notmuch_query {
     const char *query_string;
     notmuch_sort_t sort;
     notmuch_string_list_t *exclude_terms;
+    notmuch_bool_t omit_excluded_messages;
 };
 
 typedef struct _notmuch_mset_messages {
@@ -57,6 +58,12 @@ struct visible _notmuch_threads {
     notmuch_doc_id_set_t match_set;
 };
 
+/* We need this in the message functions so forward declare. */
+static notmuch_bool_t
+_notmuch_doc_id_set_init (void *ctx,
+			  notmuch_doc_id_set_t *doc_ids,
+			  GArray *arr);
+
 notmuch_query_t *
 notmuch_query_create (notmuch_database_t *notmuch,
 		      const char *query_string)
@@ -79,6 +86,8 @@ notmuch_query_create (notmuch_database_t *notmuch,
 
     query->exclude_terms = _notmuch_string_list_create (query);
 
+    query->omit_excluded_messages = FALSE;
+
     return query;
 }
 
@@ -89,6 +98,12 @@ notmuch_query_get_query_string (notmuch_query_t *query)
 }
 
 void
+notmuch_query_set_omit_excluded_messages (notmuch_query_t *query, notmuch_bool_t omit)
+{
+    query->omit_excluded_messages = omit;
+}
+
+void
 notmuch_query_set_sort (notmuch_query_t *query, notmuch_sort_t sort)
 {
     query->sort = sort;
@@ -173,6 +188,7 @@ notmuch_query_search_messages (notmuch_query_t *query)
 						   "mail"));
 	Xapian::Query string_query, final_query, exclude_query;
 	Xapian::MSet mset;
+	Xapian::MSetIterator iterator;
 	unsigned int flags = (Xapian::QueryParser::FLAG_BOOLEAN |
 			      Xapian::QueryParser::FLAG_PHRASE |
 			      Xapian::QueryParser::FLAG_LOVEHATE |
@@ -190,11 +206,35 @@ notmuch_query_search_messages (notmuch_query_t *query)
 	    final_query = Xapian::Query (Xapian::Query::OP_AND,
 					 mail_query, string_query);
 	}
+	messages->base.excluded_doc_ids = NULL;
+
+	if (query->exclude_terms) {
+	    exclude_query = _notmuch_exclude_tags (query, final_query);
+	    exclude_query = Xapian::Query (Xapian::Query::OP_AND,
+					   exclude_query, final_query);
+
+	    if (query->omit_excluded_messages)
+		final_query = Xapian::Query (Xapian::Query::OP_AND_NOT,
+					     final_query, exclude_query);
+	    else {
+		enquire.set_weighting_scheme (Xapian::BoolWeight());
+		enquire.set_query (exclude_query);
+
+		mset = enquire.get_mset (0, notmuch->xapian_db->get_doccount ());
+
+		GArray *excluded_doc_ids = g_array_new (FALSE, FALSE, sizeof (unsigned int));
+
+		for (iterator = mset.begin (); iterator != mset.end (); iterator++) {
+		    unsigned int doc_id = *iterator;
+		    g_array_append_val (excluded_doc_ids, doc_id);
+		}
+		messages->base.excluded_doc_ids = talloc (query, _notmuch_doc_id_set);
+		_notmuch_doc_id_set_init (query, messages->base.excluded_doc_ids,
+					  excluded_doc_ids);
+		g_array_unref (excluded_doc_ids);
+	    }
+	}
 
-	exclude_query = _notmuch_exclude_tags (query, final_query);
-
-	final_query = Xapian::Query (Xapian::Query::OP_AND_NOT,
-					 final_query, exclude_query);
 
 	enquire.set_weighting_scheme (Xapian::BoolWeight());
 
@@ -283,6 +323,10 @@ _notmuch_mset_messages_get (notmuch_messages_t *messages)
 	INTERNAL_ERROR ("a messages iterator contains a non-existent document ID.\n");
     }
 
+    if (messages->excluded_doc_ids &&
+	_notmuch_doc_id_set_contains (messages->excluded_doc_ids, doc_id))
+	notmuch_message_set_flag (message, NOTMUCH_MESSAGE_FLAG_EXCLUDED, TRUE);
+
     return message;
 }
 
-- 
1.7.2.3

[RFC PATCH v5 06/11] lib: Add the exclude flag to notmuch_query_search_threads

[Mark Walters]

Add the NOTMUCH_MESSAGE_FLAG_EXCLUDED flag to
notmuch_query_search_threads. Implemented by inspecting the tags
directly in _notmuch_thread_create/_thread_add_message rather than as
a Xapian query for speed reasons.

Note notmuch_thread_get_matched_messages now returns the number of
non-excluded matching messages. This API is not totally desirable but
fixing it means breaking binary compatibility so we delay that.
---
 lib/notmuch-private.h |    7 +++++--
 lib/notmuch.h         |    6 ++++--
 lib/query.cc          |    1 +
 lib/thread.cc         |   18 +++++++++++++++---
 4 files changed, 25 insertions(+), 7 deletions(-)

diff --git a/lib/notmuch-private.h b/lib/notmuch-private.h
index e791bb0..ea836f7 100644
--- a/lib/notmuch-private.h
+++ b/lib/notmuch-private.h
@@ -148,6 +148,8 @@ typedef enum _notmuch_private_status {
 
 typedef struct _notmuch_doc_id_set notmuch_doc_id_set_t;
 
+typedef struct _notmuch_string_list notmuch_string_list_t;
+
 /* database.cc */
 
 /* Lookup a prefix value by name.
@@ -216,6 +218,7 @@ _notmuch_thread_create (void *ctx,
 			notmuch_database_t *notmuch,
 			unsigned int seed_doc_id,
 			notmuch_doc_id_set_t *match_set,
+			notmuch_string_list_t *excluded_terms,
 			notmuch_sort_t sort);
 
 /* message.cc */
@@ -459,11 +462,11 @@ typedef struct _notmuch_string_node {
     struct _notmuch_string_node *next;
 } notmuch_string_node_t;
 
-typedef struct visible _notmuch_string_list {
+struct visible _notmuch_string_list {
     int length;
     notmuch_string_node_t *head;
     notmuch_string_node_t **tail;
-} notmuch_string_list_t;
+};
 
 notmuch_string_list_t *
 _notmuch_string_list_create (const void *ctx);
diff --git a/lib/notmuch.h b/lib/notmuch.h
index f75afae..babd208 100644
--- a/lib/notmuch.h
+++ b/lib/notmuch.h
@@ -672,8 +672,10 @@ notmuch_thread_get_toplevel_messages (notmuch_thread_t *thread);
 /* Get the number of messages in 'thread' that matched the search.
  *
  * This count includes only the messages in this thread that were
- * matched by the search from which the thread was created. Contrast
- * with notmuch_thread_get_total_messages() .
+ * matched by the search from which the thread was created and were
+ * not excluded by any exclude tags passed in with the query (see
+ * notmuch_query_add_tag_exclude). Contrast with
+ * notmuch_thread_get_total_messages() .
  */
 int
 notmuch_thread_get_matched_messages (notmuch_thread_t *thread);
diff --git a/lib/query.cc b/lib/query.cc
index 90a71a1..e1c3977 100644
--- a/lib/query.cc
+++ b/lib/query.cc
@@ -472,6 +472,7 @@ notmuch_threads_get (notmuch_threads_t *threads)
 				   threads->query->notmuch,
 				   doc_id,
 				   &threads->match_set,
+				   threads->query->exclude_terms,
 				   threads->query->sort);
 }
 
diff --git a/lib/thread.cc b/lib/thread.cc
index 0435ee6..e976d64 100644
--- a/lib/thread.cc
+++ b/lib/thread.cc
@@ -214,7 +214,8 @@ _thread_cleanup_author (notmuch_thread_t *thread,
  */
 static void
 _thread_add_message (notmuch_thread_t *thread,
-		     notmuch_message_t *message)
+		     notmuch_message_t *message,
+		     notmuch_string_list_t *exclude_terms)
 {
     notmuch_tags_t *tags;
     const char *tag;
@@ -262,6 +263,15 @@ _thread_add_message (notmuch_thread_t *thread,
 	 notmuch_tags_move_to_next (tags))
     {
 	tag = notmuch_tags_get (tags);
+	/* Mark excluded messages. */
+	for (notmuch_string_node_t *term = exclude_terms->head; term;
+	     term = term->next) {
+	    /* We ignore initial 'K'. */
+	    if (strcmp(tag, (term->string + 1)) == 0) {
+		notmuch_message_set_flag (message, NOTMUCH_MESSAGE_FLAG_EXCLUDED, TRUE);
+		break;
+	    }
+	}
 	g_hash_table_insert (thread->tags, xstrdup (tag), NULL);
     }
 }
@@ -321,7 +331,8 @@ _thread_add_matched_message (notmuch_thread_t *thread,
 	    _thread_set_subject_from_message (thread, message);
     }
 
-    thread->matched_messages++;
+    if (!notmuch_message_get_flag (message, NOTMUCH_MESSAGE_FLAG_EXCLUDED))
+	thread->matched_messages++;
 
     if (g_hash_table_lookup_extended (thread->message_hash,
 			    notmuch_message_get_message_id (message), NULL,
@@ -392,6 +403,7 @@ _notmuch_thread_create (void *ctx,
 			notmuch_database_t *notmuch,
 			unsigned int seed_doc_id,
 			notmuch_doc_id_set_t *match_set,
+			notmuch_string_list_t *exclude_terms,
 			notmuch_sort_t sort)
 {
     notmuch_thread_t *thread;
@@ -467,7 +479,7 @@ _notmuch_thread_create (void *ctx,
 	if (doc_id == seed_doc_id)
 	    message = seed_message;
 
-	_thread_add_message (thread, message);
+	_thread_add_message (thread, message, exclude_terms);
 
 	if ( _notmuch_doc_id_set_contains (match_set, doc_id)) {
 	    _notmuch_doc_id_set_remove (match_set, doc_id);
-- 
1.7.2.3

[RFC PATCH v5 07/11] cli: Make notmuch-show respect excludes.

[Mark Walters]

This adds the excludes to notmuch-show.c. We do not exclude when only
a single message (or part) is requested. notmuch-show will output the
exclude information when either text or json format is requested. As
this changes the output from notmuch-show it breaks many tests (in a
trivial and expected fashion).
---
 notmuch-show.c |   25 +++++++++++++++++++++----
 1 files changed, 21 insertions(+), 4 deletions(-)

diff --git a/notmuch-show.c b/notmuch-show.c
index d930f94..2a634e9 100644
--- a/notmuch-show.c
+++ b/notmuch-show.c
@@ -179,9 +179,10 @@ format_message_json (const void *ctx, notmuch_message_t *message, unused (int in
     date = notmuch_message_get_date (message);
     relative_date = notmuch_time_relative_date (ctx, date);
 
-    printf ("\"id\": %s, \"match\": %s, \"filename\": %s, \"timestamp\": %ld, \"date_relative\": \"%s\", \"tags\": [",
+    printf ("\"id\": %s, \"match\": %s, \"excluded\": %s, \"filename\": %s, \"timestamp\": %ld, \"date_relative\": \"%s\", \"tags\": [",
 	    json_quote_str (ctx_quote, notmuch_message_get_message_id (message)),
 	    notmuch_message_get_flag (message, NOTMUCH_MESSAGE_FLAG_MATCH) ? "true" : "false",
+	    notmuch_message_get_flag (message, NOTMUCH_MESSAGE_FLAG_EXCLUDED) ? "true" : "false",
 	    json_quote_str (ctx_quote, notmuch_message_get_filename (message)),
 	    date, relative_date);
 
@@ -735,11 +736,12 @@ format_part_text (const void *ctx, mime_node_t *node,
 	notmuch_message_t *message = node->envelope_file;
 
 	part_type = "message";
-	printf ("\f%s{ id:%s depth:%d match:%d filename:%s\n",
+	printf ("\f%s{ id:%s depth:%d match:%d excluded:%d filename:%s\n",
 		part_type,
 		notmuch_message_get_message_id (message),
 		indent,
-		notmuch_message_get_flag (message, NOTMUCH_MESSAGE_FLAG_MATCH),
+		notmuch_message_get_flag (message, NOTMUCH_MESSAGE_FLAG_MATCH) ? 1 : 0,
+		notmuch_message_get_flag (message, NOTMUCH_MESSAGE_FLAG_EXCLUDED) ? 1 : 0,
 		notmuch_message_get_filename (message));
     } else {
 	GMimeContentDisposition *disposition = g_mime_object_get_content_disposition (meta);
@@ -1034,6 +1036,7 @@ notmuch_show_command (void *ctx, unused (int argc), unused (char *argv[]))
     notmuch_show_params_t params = { .part = -1 };
     int format_sel = NOTMUCH_FORMAT_NOT_SPECIFIED;
     notmuch_bool_t verify = FALSE;
+    notmuch_bool_t no_exclude = FALSE;
 
     notmuch_opt_desc_t options[] = {
 	{ NOTMUCH_OPT_KEYWORD, &format_sel, "format", 'f',
@@ -1046,6 +1049,7 @@ notmuch_show_command (void *ctx, unused (int argc), unused (char *argv[]))
 	{ NOTMUCH_OPT_BOOLEAN, &params.entire_thread, "entire-thread", 't', 0 },
 	{ NOTMUCH_OPT_BOOLEAN, &params.decrypt, "decrypt", 'd', 0 },
 	{ NOTMUCH_OPT_BOOLEAN, &verify, "verify", 'v', 0 },
+	{ NOTMUCH_OPT_BOOLEAN, &no_exclude, "no-exclude", 'n', 0 },
 	{ 0, 0, 0, 0, 0 }
     };
 
@@ -1133,10 +1137,23 @@ notmuch_show_command (void *ctx, unused (int argc), unused (char *argv[]))
 	return 1;
     }
 
+    /* If a single message is requested we do not use search_excludes. */
     if (params.part >= 0)
 	ret = do_show_single (ctx, query, format, &params);
-    else
+    else {
+	if (!no_exclude) {
+	    const char **search_exclude_tags;
+	    size_t search_exclude_tags_length;
+	    unsigned int i;
+
+	    search_exclude_tags = notmuch_config_get_search_exclude_tags
+		(config, &search_exclude_tags_length);
+	    for (i = 0; i < search_exclude_tags_length; i++)
+		notmuch_query_add_tag_exclude (query, search_exclude_tags[i]);
+	}
 	ret = do_show (ctx, query, format, &params);
+    }
+
 
     notmuch_query_destroy (query);
     notmuch_database_close (notmuch);
-- 
1.7.2.3

[RFC PATCH v5 08/11] man: update manpage for notmuch-show --no-exclude option

[Mark Walters]

---
 man/man1/notmuch-show.1 |    7 +++++++
 1 files changed, 7 insertions(+), 0 deletions(-)

diff --git a/man/man1/notmuch-show.1 b/man/man1/notmuch-show.1
index b2301d8..f743a53 100644
--- a/man/man1/notmuch-show.1
+++ b/man/man1/notmuch-show.1
@@ -128,6 +128,13 @@ multipart/encrypted part will be replaced by the decrypted
 content.
 .RE
 
+.RS 4
+.TP 4
+.B \-\-no-exclude
+
+Do not exclude the messages matching search_exclude_tags in the config file.
+.RE
+
 A common use of
 .B notmuch show
 is to display a single thread of email messages. For this, use a
-- 
1.7.2.3

[RFC PATCH v5 09/11] test: update tests to reflect the exclude flag

[Mark Walters]

notmuch show outputs the exclude flag so many tests using notmuch
show failed. This commit adds "excluded:0" or "excluded: false" to
the expected outputs. After this commit there should be no failing
tests.
---
 test/crypto        |    9 ++++++++-
 test/encoding      |    2 +-
 test/json          |    6 +++---
 test/maildir-sync  |    1 +
 test/multipart     |    4 ++--
 test/thread-naming |   16 ++++++++--------
 6 files changed, 23 insertions(+), 15 deletions(-)

diff --git a/test/crypto b/test/crypto
index 1dbb60a..564cf9a 100755
--- a/test/crypto
+++ b/test/crypto
@@ -43,6 +43,7 @@ output=$(notmuch show --format=json --verify subject:"test signed message 001" \
     | sed -e 's|"created": [1234567890]*|"created": 946728000|')
 expected='[[[{"id": "XXXXX",
  "match": true,
+ "excluded": false,
  "filename": "YYYYY",
  "timestamp": 946728000,
  "date_relative": "2000-01-01",
@@ -77,6 +78,7 @@ output=$(notmuch show --format=json --verify subject:"test signed message 001" \
     | sed -e 's|"created": [1234567890]*|"created": 946728000|')
 expected='[[[{"id": "XXXXX",
  "match": true,
+ "excluded": false,
  "filename": "YYYYY",
  "timestamp": 946728000,
  "date_relative": "2000-01-01",
@@ -113,6 +115,7 @@ output=$(notmuch show --format=json --verify subject:"test signed message 001" \
     | sed -e 's|"created": [1234567890]*|"created": 946728000|')
 expected='[[[{"id": "XXXXX",
  "match": true,
+ "excluded": false,
  "filename": "YYYYY",
  "timestamp": 946728000,
  "date_relative": "2000-01-01",
@@ -153,7 +156,7 @@ test_begin_subtest "decryption, --format=text"
 output=$(notmuch show --format=text --decrypt subject:"test encrypted message 001" \
     | notmuch_show_sanitize_all \
     | sed -e 's|"created": [1234567890]*|"created": 946728000|')
-expected='\fmessage{ id:XXXXX depth:0 match:1 filename:XXXXX
+expected='\fmessage{ id:XXXXX depth:0 match:1 excluded:0 filename:XXXXX
 \fheader{
 Notmuch Test Suite <test_suite@notmuchmail.org> (2000-01-01) (encrypted inbox)
 Subject: test encrypted message 001
@@ -187,6 +190,7 @@ output=$(notmuch show --format=json --decrypt subject:"test encrypted message 00
     | sed -e 's|"created": [1234567890]*|"created": 946728000|')
 expected='[[[{"id": "XXXXX",
  "match": true,
+ "excluded": false,
  "filename": "YYYYY",
  "timestamp": 946728000,
  "date_relative": "2000-01-01",
@@ -242,6 +246,7 @@ output=$(notmuch show --format=json --decrypt subject:"test encrypted message 00
     | sed -e 's|"created": [1234567890]*|"created": 946728000|')
 expected='[[[{"id": "XXXXX",
  "match": true,
+ "excluded": false,
  "filename": "YYYYY",
  "timestamp": 946728000,
  "date_relative": "2000-01-01",
@@ -277,6 +282,7 @@ output=$(notmuch show --format=json --decrypt subject:"test encrypted message 00
     | sed -e 's|"created": [1234567890]*|"created": 946728000|')
 expected='[[[{"id": "XXXXX",
  "match": true,
+ "excluded": false,
  "filename": "YYYYY",
  "timestamp": 946728000,
  "date_relative": "2000-01-01",
@@ -332,6 +338,7 @@ output=$(notmuch show --format=json --verify subject:"test signed message 001" \
     | sed -e 's|"created": [1234567890]*|"created": 946728000|')
 expected='[[[{"id": "XXXXX",
  "match": true,
+ "excluded": false,
  "filename": "YYYYY",
  "timestamp": 946728000,
  "date_relative": "2000-01-01",
diff --git a/test/encoding b/test/encoding
index 33259c1..a872345 100755
--- a/test/encoding
+++ b/test/encoding
@@ -6,7 +6,7 @@ test_begin_subtest "Message with text of unknown charset"
 add_message '[content-type]="text/plain; charset=unknown-8bit"' \
 	    "[body]=irrelevant"
 output=$(notmuch show id:${gen_msg_id} 2>&1 | notmuch_show_sanitize)
-test_expect_equal "$output" "\fmessage{ id:msg-001@notmuch-test-suite depth:0 match:1 filename:/XXX/mail/msg-001
+test_expect_equal "$output" "\fmessage{ id:msg-001@notmuch-test-suite depth:0 match:1 excluded:0 filename:/XXX/mail/msg-001
 \fheader{
 Notmuch Test Suite <test_suite@notmuchmail.org> (2001-01-05) (inbox unread)
 Subject: Test message #1
diff --git a/test/json b/test/json
index 7df4380..f95fcf8 100755
--- a/test/json
+++ b/test/json
@@ -5,7 +5,7 @@ test_description="--format=json output"
 test_begin_subtest "Show message: json"
 add_message "[subject]=\"json-show-subject\"" "[date]=\"Sat, 01 Jan 2000 12:00:00 -0000\"" "[body]=\"json-show-message\""
 output=$(notmuch show --format=json "json-show-message")
-test_expect_equal "$output" "[[[{\"id\": \"${gen_msg_id}\", \"match\": true, \"filename\": \"${gen_msg_filename}\", \"timestamp\": 946728000, \"date_relative\": \"2000-01-01\", \"tags\": [\"inbox\",\"unread\"], \"headers\": {\"Subject\": \"json-show-subject\", \"From\": \"Notmuch Test Suite <test_suite@notmuchmail.org>\", \"To\": \"Notmuch Test Suite <test_suite@notmuchmail.org>\", \"Cc\": \"\", \"Bcc\": \"\", \"Date\": \"Sat, 01 Jan 2000 12:00:00 -0000\"}, \"body\": [{\"id\": 1, \"content-type\": \"text/plain\", \"content\": \"json-show-message\n\"}]}, []]]]"
+test_expect_equal "$output" "[[[{\"id\": \"${gen_msg_id}\", \"match\": true, \"excluded\": false, \"filename\": \"${gen_msg_filename}\", \"timestamp\": 946728000, \"date_relative\": \"2000-01-01\", \"tags\": [\"inbox\",\"unread\"], \"headers\": {\"Subject\": \"json-show-subject\", \"From\": \"Notmuch Test Suite <test_suite@notmuchmail.org>\", \"To\": \"Notmuch Test Suite <test_suite@notmuchmail.org>\", \"Cc\": \"\", \"Bcc\": \"\", \"Date\": \"Sat, 01 Jan 2000 12:00:00 -0000\"}, \"body\": [{\"id\": 1, \"content-type\": \"text/plain\", \"content\": \"json-show-message\n\"}]}, []]]]"
 
 test_begin_subtest "Search message: json"
 add_message "[subject]=\"json-search-subject\"" "[date]=\"Sat, 01 Jan 2000 12:00:00 -0000\"" "[body]=\"json-search-message\""
@@ -22,7 +22,7 @@ test_expect_equal "$output" "[{\"thread\": \"XXX\",
 test_begin_subtest "Show message: json, utf-8"
 add_message "[subject]=\"json-show-utf8-body-sübjéct\"" "[date]=\"Sat, 01 Jan 2000 12:00:00 -0000\"" "[body]=\"jsön-show-méssage\""
 output=$(notmuch show --format=json "jsön-show-méssage")
-test_expect_equal "$output" "[[[{\"id\": \"${gen_msg_id}\", \"match\": true, \"filename\": \"${gen_msg_filename}\", \"timestamp\": 946728000, \"date_relative\": \"2000-01-01\", \"tags\": [\"inbox\",\"unread\"], \"headers\": {\"Subject\": \"json-show-utf8-body-sübjéct\", \"From\": \"Notmuch Test Suite <test_suite@notmuchmail.org>\", \"To\": \"Notmuch Test Suite <test_suite@notmuchmail.org>\", \"Cc\": \"\", \"Bcc\": \"\", \"Date\": \"Sat, 01 Jan 2000 12:00:00 -0000\"}, \"body\": [{\"id\": 1, \"content-type\": \"text/plain\", \"content\": \"jsön-show-méssage\n\"}]}, []]]]"
+test_expect_equal "$output" "[[[{\"id\": \"${gen_msg_id}\", \"match\": true, \"excluded\": false, \"filename\": \"${gen_msg_filename}\", \"timestamp\": 946728000, \"date_relative\": \"2000-01-01\", \"tags\": [\"inbox\",\"unread\"], \"headers\": {\"Subject\": \"json-show-utf8-body-sübjéct\", \"From\": \"Notmuch Test Suite <test_suite@notmuchmail.org>\", \"To\": \"Notmuch Test Suite <test_suite@notmuchmail.org>\", \"Cc\": \"\", \"Bcc\": \"\", \"Date\": \"Sat, 01 Jan 2000 12:00:00 -0000\"}, \"body\": [{\"id\": 1, \"content-type\": \"text/plain\", \"content\": \"jsön-show-méssage\n\"}]}, []]]]"
 
 test_begin_subtest "Show message: json, inline attachment filename"
 subject='json-show-inline-attachment-filename'
@@ -35,7 +35,7 @@ emacs_deliver_message \
      (insert \"Message-ID: <$id>\n\")"
 output=$(notmuch show --format=json "id:$id")
 filename=$(notmuch search --output=files "id:$id")
-test_expect_equal "$output" "[[[{\"id\": \"$id\", \"match\": true, \"filename\": \"$filename\", \"timestamp\": 946728000, \"date_relative\": \"2000-01-01\", \"tags\": [\"inbox\"], \"headers\": {\"Subject\": \"$subject\", \"From\": \"Notmuch Test Suite <test_suite@notmuchmail.org>\", \"To\": \"test_suite@notmuchmail.org\", \"Cc\": \"\", \"Bcc\": \"\", \"Date\": \"01 Jan 2000 12:00:00 -0000\"}, \"body\": [{\"id\": 1, \"content-type\": \"multipart/mixed\", \"content\": [{\"id\": 2, \"content-type\": \"text/plain\", \"content\": \"This is a test message with inline attachment with a filename\"}, {\"id\": 3, \"content-type\": \"application/octet-stream\", \"filename\": \"README\"}]}]}, []]]]"
+test_expect_equal "$output" "[[[{\"id\": \"$id\", \"match\": true, \"excluded\": false, \"filename\": \"$filename\", \"timestamp\": 946728000, \"date_relative\": \"2000-01-01\", \"tags\": [\"inbox\"], \"headers\": {\"Subject\": \"$subject\", \"From\": \"Notmuch Test Suite <test_suite@notmuchmail.org>\", \"To\": \"test_suite@notmuchmail.org\", \"Cc\": \"\", \"Bcc\": \"\", \"Date\": \"01 Jan 2000 12:00:00 -0000\"}, \"body\": [{\"id\": 1, \"content-type\": \"multipart/mixed\", \"content\": [{\"id\": 2, \"content-type\": \"text/plain\", \"content\": \"This is a test message with inline attachment with a filename\"}, {\"id\": 3, \"content-type\": \"application/octet-stream\", \"filename\": \"README\"}]}]}, []]]]"
 
 test_begin_subtest "Search message: json, utf-8"
 add_message "[subject]=\"json-search-utf8-body-sübjéct\"" "[date]=\"Sat, 01 Jan 2000 12:00:00 -0000\"" "[body]=\"jsön-search-méssage\""
diff --git a/test/maildir-sync b/test/maildir-sync
index d5872a5..dfa3966 100755
--- a/test/maildir-sync
+++ b/test/maildir-sync
@@ -46,6 +46,7 @@ test_begin_subtest "notmuch show works with renamed file (without notmuch new)"
 output=$(notmuch show --format=json id:${gen_msg_id} | filter_show_json)
 test_expect_equal "$output" '[[[{"id": "adding-replied-tag@notmuch-test-suite",
 "match": true,
+"excluded": false,
 "filename": "MAIL_DIR/cur/adding-replied-tag:2,RS",
 "timestamp": 978709437,
 "date_relative": "2001-01-05",
diff --git a/test/multipart b/test/multipart
index 2dd73f5..3d2dcfc 100755
--- a/test/multipart
+++ b/test/multipart
@@ -108,7 +108,7 @@ notmuch new > /dev/null
 test_begin_subtest "--format=text --part=0, full message"
 notmuch show --format=text --part=0 'id:87liy5ap00.fsf@yoom.home.cworth.org' >OUTPUT
 cat <<EOF >EXPECTED
-\fmessage{ id:87liy5ap00.fsf@yoom.home.cworth.org depth:0 match:1 filename:${MAIL_DIR}/multipart
+\fmessage{ id:87liy5ap00.fsf@yoom.home.cworth.org depth:0 match:1 excluded:0 filename:${MAIL_DIR}/multipart
 \fheader{
 Carl Worth <cworth@cworth.org> (2001-01-05) (attachment inbox signed unread)
 Subject: Multipart message
@@ -322,7 +322,7 @@ notmuch show --format=json --part=0 'id:87liy5ap00.fsf@yoom.home.cworth.org' | s
 echo >>OUTPUT # expect *no* newline at end of output
 cat <<EOF >EXPECTED
 
-{"id": "87liy5ap00.fsf@yoom.home.cworth.org", "match": true, "filename": "${MAIL_DIR}/multipart", "timestamp": 978709437, "date_relative": "2001-01-05", "tags": ["attachment","inbox","signed","unread"], "headers": {"Subject": "Multipart message", "From": "Carl Worth <cworth@cworth.org>", "To": "cworth@cworth.org", "Cc": "", "Bcc": "", "Date": "Fri, 05 Jan 2001 15:43:57 +0000"}, "body": [
+{"id": "87liy5ap00.fsf@yoom.home.cworth.org", "match": true, "excluded": false, "filename": "${MAIL_DIR}/multipart", "timestamp": 978709437, "date_relative": "2001-01-05", "tags": ["attachment","inbox","signed","unread"], "headers": {"Subject": "Multipart message", "From": "Carl Worth <cworth@cworth.org>", "To": "cworth@cworth.org", "Cc": "", "Bcc": "", "Date": "Fri, 05 Jan 2001 15:43:57 +0000"}, "body": [
 {"id": 1, "content-type": "multipart/signed", "content": [
 {"id": 2, "content-type": "multipart/mixed", "content": [
 {"id": 3, "content-type": "message/rfc822", "content": [{"headers": {"From": "Carl Worth <cworth@cworth.org>", "To": "cworth@cworth.org", "Subject": "html message", "Date": "Fri, 05 Jan 2001 15:42:57 +0000"}, "body": [
diff --git a/test/thread-naming b/test/thread-naming
index 942e593..1a1a48f 100755
--- a/test/thread-naming
+++ b/test/thread-naming
@@ -65,7 +65,7 @@ test_expect_equal "$output" "thread:XXX   2001-01-12 [6/8] Notmuch Test Suite; t
 
 test_begin_subtest 'Test order of messages in "notmuch show"'
 output=$(notmuch show thread-naming | notmuch_show_sanitize)
-test_expect_equal "$output" "\fmessage{ id:msg-$(printf "%03d" $first)@notmuch-test-suite depth:0 match:1 filename:/XXX/mail/msg-$(printf "%03d" $first)
+test_expect_equal "$output" "\fmessage{ id:msg-$(printf "%03d" $first)@notmuch-test-suite depth:0 match:1 excluded:0 filename:/XXX/mail/msg-$(printf "%03d" $first)
 \fheader{
 Notmuch Test Suite <test_suite@notmuchmail.org> (2001-01-05) (unread)
 Subject: thread-naming: Initial thread subject
@@ -79,7 +79,7 @@ This is just a test message (#$first)
 \fpart}
 \fbody}
 \fmessage}
-\fmessage{ id:msg-$(printf "%03d" $((first + 1)))@notmuch-test-suite depth:1 match:1 filename:/XXX/mail/msg-$(printf "%03d" $((first + 1)))
+\fmessage{ id:msg-$(printf "%03d" $((first + 1)))@notmuch-test-suite depth:1 match:1 excluded:0 filename:/XXX/mail/msg-$(printf "%03d" $((first + 1)))
 \fheader{
 Notmuch Test Suite <test_suite@notmuchmail.org> (2001-01-06) (inbox unread)
 Subject: thread-naming: Older changed subject
@@ -93,7 +93,7 @@ This is just a test message (#$((first + 1)))
 \fpart}
 \fbody}
 \fmessage}
-\fmessage{ id:msg-$(printf "%03d" $((first + 2)))@notmuch-test-suite depth:1 match:1 filename:/XXX/mail/msg-$(printf "%03d" $((first + 2)))
+\fmessage{ id:msg-$(printf "%03d" $((first + 2)))@notmuch-test-suite depth:1 match:1 excluded:0 filename:/XXX/mail/msg-$(printf "%03d" $((first + 2)))
 \fheader{
 Notmuch Test Suite <test_suite@notmuchmail.org> (2001-01-07) (inbox unread)
 Subject: thread-naming: Newer changed subject
@@ -107,7 +107,7 @@ This is just a test message (#$((first + 2)))
 \fpart}
 \fbody}
 \fmessage}
-\fmessage{ id:msg-$(printf "%03d" $((first + 3)))@notmuch-test-suite depth:1 match:1 filename:/XXX/mail/msg-$(printf "%03d" $((first + 3)))
+\fmessage{ id:msg-$(printf "%03d" $((first + 3)))@notmuch-test-suite depth:1 match:1 excluded:0 filename:/XXX/mail/msg-$(printf "%03d" $((first + 3)))
 \fheader{
 Notmuch Test Suite <test_suite@notmuchmail.org> (2001-01-08) (unread)
 Subject: thread-naming: Final thread subject
@@ -121,7 +121,7 @@ This is just a test message (#$((first + 3)))
 \fpart}
 \fbody}
 \fmessage}
-\fmessage{ id:msg-$(printf "%03d" $((first + 4)))@notmuch-test-suite depth:1 match:1 filename:/XXX/mail/msg-$(printf "%03d" $((first + 4)))
+\fmessage{ id:msg-$(printf "%03d" $((first + 4)))@notmuch-test-suite depth:1 match:1 excluded:0 filename:/XXX/mail/msg-$(printf "%03d" $((first + 4)))
 \fheader{
 Notmuch Test Suite <test_suite@notmuchmail.org> (2001-01-09) (inbox unread)
 Subject: Re: thread-naming: Initial thread subject
@@ -135,7 +135,7 @@ This is just a test message (#$((first + 4)))
 \fpart}
 \fbody}
 \fmessage}
-\fmessage{ id:msg-$(printf "%03d" $((first + 5)))@notmuch-test-suite depth:1 match:1 filename:/XXX/mail/msg-$(printf "%03d" $((first + 5)))
+\fmessage{ id:msg-$(printf "%03d" $((first + 5)))@notmuch-test-suite depth:1 match:1 excluded:0 filename:/XXX/mail/msg-$(printf "%03d" $((first + 5)))
 \fheader{
 Notmuch Test Suite <test_suite@notmuchmail.org> (2001-01-10) (inbox unread)
 Subject: Aw: thread-naming: Initial thread subject
@@ -149,7 +149,7 @@ This is just a test message (#$((first + 5)))
 \fpart}
 \fbody}
 \fmessage}
-\fmessage{ id:msg-$(printf "%03d" $((first + 6)))@notmuch-test-suite depth:1 match:1 filename:/XXX/mail/msg-$(printf "%03d" $((first + 6)))
+\fmessage{ id:msg-$(printf "%03d" $((first + 6)))@notmuch-test-suite depth:1 match:1 excluded:0 filename:/XXX/mail/msg-$(printf "%03d" $((first + 6)))
 \fheader{
 Notmuch Test Suite <test_suite@notmuchmail.org> (2001-01-11) (inbox unread)
 Subject: Vs: thread-naming: Initial thread subject
@@ -163,7 +163,7 @@ This is just a test message (#$((first + 6)))
 \fpart}
 \fbody}
 \fmessage}
-\fmessage{ id:msg-$(printf "%03d" $((first + 7)))@notmuch-test-suite depth:1 match:1 filename:/XXX/mail/msg-$(printf "%03d" $((first + 7)))
+\fmessage{ id:msg-$(printf "%03d" $((first + 7)))@notmuch-test-suite depth:1 match:1 excluded:0 filename:/XXX/mail/msg-$(printf "%03d" $((first + 7)))
 \fheader{
 Notmuch Test Suite <test_suite@notmuchmail.org> (2001-01-12) (inbox unread)
 Subject: Sv: thread-naming: Initial thread subject
-- 
1.7.2.3

[RFC PATCH v5 10/11] cli: omit excluded messages in results where appropriate.

[Mark Walters]

In all cases of notmuch count/search/show where the results returned
cannot reflect the exclude flag return just the matched not-excluded
results. If the caller wishes to have all the matched results (i.e.,
including the excluded ones) they should call with the
--no-exclude option.

The relevant cases are
    count: both threads and messages
    search: all cases except the summary view
    show: mbox format
---
 notmuch-count.c  |    2 ++
 notmuch-search.c |    9 +++++++++
 notmuch-show.c   |    6 ++++++
 3 files changed, 17 insertions(+), 0 deletions(-)

diff --git a/notmuch-count.c b/notmuch-count.c
index 5364507..46b76ae 100644
--- a/notmuch-count.c
+++ b/notmuch-count.c
@@ -88,6 +88,8 @@ notmuch_count_command (void *ctx, int argc, char *argv[])
 	    notmuch_query_add_tag_exclude (query, search_exclude_tags[i]);
     }
 
+    notmuch_query_set_omit_excluded_messages (query, TRUE);
+
     switch (output) {
     case OUTPUT_MESSAGES:
 	printf ("%u\n", notmuch_query_count_messages (query));
diff --git a/notmuch-search.c b/notmuch-search.c
index 43ec90b..d2b2488 100644
--- a/notmuch-search.c
+++ b/notmuch-search.c
@@ -207,6 +207,9 @@ do_search_threads (const search_format_t *format,
     int first_thread = 1;
     int i;
 
+    if (output == OUTPUT_THREADS)
+	notmuch_query_set_omit_excluded_messages (query, TRUE);
+
     if (offset < 0) {
 	offset += notmuch_query_count_threads (query);
 	if (offset < 0)
@@ -297,6 +300,8 @@ do_search_messages (const search_format_t *format,
     int first_message = 1;
     int i;
 
+    notmuch_query_set_omit_excluded_messages (query, TRUE);
+
     if (offset < 0) {
 	offset += notmuch_query_count_messages (query);
 	if (offset < 0)
@@ -368,6 +373,10 @@ do_search_tags (notmuch_database_t *notmuch,
     const char *tag;
     int first_tag = 1;
 
+    notmuch_query_set_omit_excluded_messages (query, TRUE);
+    /* should the following only special case if no excluded terms
+     * specified? */
+
     /* Special-case query of "*" for better performance. */
     if (strcmp (notmuch_query_get_query_string (query), "*") == 0) {
 	tags = notmuch_database_get_all_tags (notmuch);
diff --git a/notmuch-show.c b/notmuch-show.c
index 2a634e9..17a4ec6 100644
--- a/notmuch-show.c
+++ b/notmuch-show.c
@@ -1080,6 +1080,7 @@ notmuch_show_command (void *ctx, unused (int argc), unused (char *argv[]))
 	    fprintf (stderr, "Error: specifying parts is incompatible with mbox output format.\n");
 	    return 1;
 	}
+
 	format = &format_mbox;
 	break;
     case NOTMUCH_FORMAT_RAW:
@@ -1137,6 +1138,11 @@ notmuch_show_command (void *ctx, unused (int argc), unused (char *argv[]))
 	return 1;
     }
 
+    /* if format=mbox then we can not output excluded messages as
+     * there is no way to make the exclude flag available */
+    if (format_sel == NOTMUCH_FORMAT_MBOX)
+	notmuch_query_set_omit_excluded_messages (query, TRUE);
+
     /* If a single message is requested we do not use search_excludes. */
     if (params.part >= 0)
 	ret = do_show_single (ctx, query, format, &params);
-- 
1.7.2.3

[RFC PATCH v5 11/11] emacs: show: recognize the exclude flag.

[Mark Walters]

Show mode will recognize the exclude flag by not opening excluding
messages by default, and will start at the first matching non-excluded
message. If there are no matching non-excluded messages it will go to
the first matching (necessarily excluded) message.
---
 emacs/notmuch-show.el |   19 ++++++++++++++++++-
 1 files changed, 18 insertions(+), 1 deletions(-)

diff --git a/emacs/notmuch-show.el b/emacs/notmuch-show.el
index aa9ccee..96b0c71 100644
--- a/emacs/notmuch-show.el
+++ b/emacs/notmuch-show.el
@@ -948,7 +948,8 @@ current buffer, if possible."
 
     ;; Message visibility depends on whether it matched the search
     ;; criteria.
-    (notmuch-show-message-visible msg (plist-get msg :match))))
+    (notmuch-show-message-visible msg (and (plist-get msg :match)
+					   (not (plist-get msg :excluded))))))
 
 (defun notmuch-show-toggle-process-crypto ()
   "Toggle the processing of cryptographic MIME parts."
@@ -1079,6 +1080,10 @@ function is used."
     ;; Move straight to the first open message
     (unless (notmuch-show-message-visible-p)
       (notmuch-show-next-open-message))
+    (when (eobp)
+      (goto-char (point-min))
+      (unless (notmuch-show-get-prop :match)
+	(notmuch-show-next-matching-message)))
 
     ;; Set the header line to the subject of the first open message.
     (setq header-line-format (notmuch-show-strip-re (notmuch-show-get-pretty-subject)))
@@ -1557,6 +1562,18 @@ to show, nil otherwise."
 	(goto-char (point-max))))
     r))
 
+(defun notmuch-show-next-matching-message ()
+  "Show the next matching message."
+  (interactive)
+  (let (r)
+    (while (and (setq r (notmuch-show-goto-message-next))
+		(not (notmuch-show-get-prop :match))))
+    (if r
+	(progn
+	  (notmuch-show-mark-read)
+	  (notmuch-show-message-adjust))
+      (goto-char (point-max)))))
+
 (defun notmuch-show-previous-open-message ()
   "Show the previous open message."
   (interactive)
-- 
1.7.2.3

Re: [RFC PATCH v5 00/11] Add NOTMUCH_MESSAGE_FLAG_EXCLUDED flag

[Jameson Graef Rollins]

[-- Attachment #1: Type: text/plain, Size: 1376 bytes --]

On Wed, 15 Feb 2012 09:03:28 +0000, Mark Walters <markwalters1009@gmail.com> wrote:
> The current implementation of exclude-tags does not use excludes in
> notmuch-show.c (and thus not in notmuch-show.el). Thus when selecting 
> a thread in the search view claiming one matched message you may get 
> several matches in show all but one of which are tagged excluded.

Hey, Mark.  Thanks so much for working on this.  This is a very nice
extension to Austin's work.

> This set moves in a different direction. It returns all the results 
> but marks excluded messages with a new flag 
> (NOTMUCH_MESSAGE_FLAG_EXCLUDED) and lets the consumer decide what to 
> do with them. For example it could start with the message closed in 
> emacs show view, it could colour the headerline differently etc.

I missed some of the previous discussion on this, but rather than add a
new flag, why not just use the existing "match" flag?  If the message is
excluded, just mark "match" as "false".  I think this is basically all
we really want.  If the message is "excluded" include it in returned
threads, but just don't display it.  This is in fact exactly what the
"match" flag is currently for, and I can't see any reason not to use it
here.  We don't need to add anything new to the show output, and I
believe it will simplify this patch set considerably.

jamie.

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: [RFC PATCH v5 00/11] Add NOTMUCH_MESSAGE_FLAG_EXCLUDED flag

[Jameson Graef Rollins]

[-- Attachment #1: Type: text/plain, Size: 802 bytes --]

On Wed, 15 Feb 2012 09:46:56 -0800, Jameson Graef Rollins <jrollins@finestructure.net> wrote:
> I missed some of the previous discussion on this, but rather than add a
> new flag, why not just use the existing "match" flag?  If the message is
> excluded, just mark "match" as "false".  I think this is basically all
> we really want.  If the message is "excluded" include it in returned
> threads, but just don't display it.  This is in fact exactly what the
> "match" flag is currently for, and I can't see any reason not to use it
> here.  We don't need to add anything new to the show output, and I
> believe it will simplify this patch set considerably.

Also, using the match flag means that all the consumers that use this
will all automatically do the right thing, without modification.

jamie.

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: [RFC PATCH v5 00/11] Add NOTMUCH_MESSAGE_FLAG_EXCLUDED flag

[Mark Walters]

On Wed, 15 Feb 2012 09:46:56 -0800, Jameson Graef Rollins <jrollins@finestructure.net> wrote:
> On Wed, 15 Feb 2012 09:03:28 +0000, Mark Walters <markwalters1009@gmail.com> wrote:
> > The current implementation of exclude-tags does not use excludes in
> > notmuch-show.c (and thus not in notmuch-show.el). Thus when selecting 
> > a thread in the search view claiming one matched message you may get 
> > several matches in show all but one of which are tagged excluded.
> 
> Hey, Mark.  Thanks so much for working on this.  This is a very nice
> extension to Austin's work.
> 
> > This set moves in a different direction. It returns all the results 
> > but marks excluded messages with a new flag 
> > (NOTMUCH_MESSAGE_FLAG_EXCLUDED) and lets the consumer decide what to 
> > do with them. For example it could start with the message closed in 
> > emacs show view, it could colour the headerline differently etc.
> 
> I missed some of the previous discussion on this, but rather than add a
> new flag, why not just use the existing "match" flag?  If the message is
> excluded, just mark "match" as "false".  I think this is basically all
> we really want.  If the message is "excluded" include it in returned
> threads, but just don't display it.  This is in fact exactly what the
> "match" flag is currently for, and I can't see any reason not to use it
> here.  We don't need to add anything new to the show output, and I
> believe it will simplify this patch set considerably.

I think the difficulty is that there are lots of annoying corner cases,
but if there is a  simpler solution that would be great!

Anyway I will mention a couple of corner cases I can remember with any
thoughts I have on alternative solutions.

1) What should notmuch show id:deleted-message-id do? 

It could return the thread containing the deleted message. If it does
return a thread what subject does it assign it?  Possibly it could
return no messages and the caller would have to call it again with
--no-exclude.

2) Should notmuch search return threads which match but only in excluded
messages? 

If yes then does it sort it based on match or match-not-excluded? If the
latter what happens to threads with no match-not-excluded messages? If
not then does searching for id:deleted-message return no results? The
caller could try with --no-exclude, but then the caller would end up
returning the deleted message for search id:deleted-message but not for
search id:deleted-message or id:some-other-not-deleted-message.

Anyway with answers to the 2 main questions I can try thinking about
simpler possibilities.

Best wishes

Mark

Re: [RFC PATCH v5 00/11] Add NOTMUCH_MESSAGE_FLAG_EXCLUDED flag

[Jameson Graef Rollins]

[-- Attachment #1: Type: text/plain, Size: 2059 bytes --]

On Wed, 15 Feb 2012 21:11:15 +0000, Mark Walters <markwalters1009@gmail.com> wrote:
> I think the difficulty is that there are lots of annoying corner cases,
> but if there is a  simpler solution that would be great!

I think there is!

> 1) What should notmuch show id:deleted-message-id do? 
> 
> It could return the thread containing the deleted message. If it does
> return a thread what subject does it assign it?  Possibly it could
> return no messages and the caller would have to call it again with
> --no-exclude.

"notmuch show id:<excluded-id>" should always return the message
matching id:<excluded-id> with match=true.  In fact, any search that
references a specific id: should always process the message as if there
were no excludes at all.

Excluded messages are not directly accessible at the moment, which is
definitely a bug.  Adding the --no-excludes option will help, but I
still think we should just implement the behavior I outline above.

> 2) Should notmuch search return threads which match but only in excluded
> messages? 
> 
> If yes then does it sort it based on match or match-not-excluded? If the
> latter what happens to threads with no match-not-excluded messages? If
> not then does searching for id:deleted-message return no results? The
> caller could try with --no-exclude, but then the caller would end up
> returning the deleted message for search id:deleted-message but not for
> search id:deleted-message or id:some-other-not-deleted-message.

See the point above.  If one of the search terms is an id: then that
message should be returned matched, as if there were no excludes.

I think this is the right solution, for both search and show:

- excluded messages are just match=false

- searches that reference a specific id: are match=true no matter what
  their exclude status

- searches that reference an excluded tag are match=true

As far as I can see this should "just work", without any existing
changes to consumers.  Anyone see any issues I'm missing?

jamie.

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: [RFC PATCH v5 00/11] Add NOTMUCH_MESSAGE_FLAG_EXCLUDED flag

[Austin Clements]

Quoth Jameson Graef Rollins on Feb 15 at  2:16 pm:
> On Wed, 15 Feb 2012 21:11:15 +0000, Mark Walters <markwalters1009@gmail.com> wrote:
> > I think the difficulty is that there are lots of annoying corner cases,
> > but if there is a  simpler solution that would be great!
> 
> I think there is!

There's been a fair bit of discussion of this at
id:"871uqvgrnm.fsf@qmul.ac.uk" (already pointed out on IRC, but I
figured I'd point it out here for others).

There are some drawbacks to separating out the exclude flag, but
personally I think they're minor and that doing so opens up a lot of
interesting UI possibilities as well as providing a very clean and
consistent interface.

> > 1) What should notmuch show id:deleted-message-id do? 
> > 
> > It could return the thread containing the deleted message. If it does
> > return a thread what subject does it assign it?  Possibly it could
> > return no messages and the caller would have to call it again with
> > --no-exclude.
> 
> "notmuch show id:<excluded-id>" should always return the message
> matching id:<excluded-id> with match=true.  In fact, any search that
> references a specific id: should always process the message as if there
> were no excludes at all.

This sounds terribly inconsistent to me.  Even setting that aside, I
can't think of a reasonable way to implement this.  What does it mean
for a search to reference a specific id?  Do you special case when the
query begins with the string "id:"?  Do you parse the query?  Do you
check if the unexcluded search matches exactly one message and do
something different in that case?

Re: [RFC PATCH v5 00/11] Add NOTMUCH_MESSAGE_FLAG_EXCLUDED flag

[Mark Walters]

On Wed, 15 Feb 2012 14:16:16 -0800, Jameson Graef Rollins <jrollins@finestructure.net> wrote:
> On Wed, 15 Feb 2012 21:11:15 +0000, Mark Walters <markwalters1009@gmail.com> wrote:
> > I think the difficulty is that there are lots of annoying corner cases,
> > but if there is a  simpler solution that would be great!
> 
> I think there is!
> 
> > 1) What should notmuch show id:deleted-message-id do? 
> > 
> > It could return the thread containing the deleted message. If it does
> > return a thread what subject does it assign it?  Possibly it could
> > return no messages and the caller would have to call it again with
> > --no-exclude.
> 
> "notmuch show id:<excluded-id>" should always return the message
> matching id:<excluded-id> with match=true.  In fact, any search that
> references a specific id: should always process the message as if there
> were no excludes at all.
> 
> Excluded messages are not directly accessible at the moment, which is
> definitely a bug.  Adding the --no-excludes option will help, but I
> still think we should just implement the behavior I outline above.
> 
> > 2) Should notmuch search return threads which match but only in excluded
> > messages? 
> > 
> > If yes then does it sort it based on match or match-not-excluded? If the
> > latter what happens to threads with no match-not-excluded messages? If
> > not then does searching for id:deleted-message return no results? The
> > caller could try with --no-exclude, but then the caller would end up
> > returning the deleted message for search id:deleted-message but not for
> > search id:deleted-message or id:some-other-not-deleted-message.
> 
> See the point above.  If one of the search terms is an id: then that
> message should be returned matched, as if there were no excludes.
> 
> I think this is the right solution, for both search and show:
> 
> - excluded messages are just match=false
> 
> - searches that reference a specific id: are match=true no matter what
>   their exclude status
> 
> - searches that reference an excluded tag are match=true
> 
> As far as I can see this should "just work", without any existing
> changes to consumers.  Anyone see any issues I'm missing?

So is the following what you are suggesting:

always return all messages/threads that match the query whether or not
the match is excluded, but only mark the message as a match if it
matches and is not excluded (with some exceptions: e.g., for id: queries
and tag:deleted queries).

A different way of looking at this is that we revert to the old
pre-exclude behaviour and then implement excludes as a "turn off the
match flag" rather than the current "omit the results".

That sounds plausible. 

But I am not sure how much simpler it is. Of my patch set patches 1-3
are essentially separate (the --no-exclude bits). 

I think you would need patches 4 and 5 or something similar since we
both need to mark the excluded messages ("excluded" in my case, "not
matched" in your case).

Patch 6 could be simplified: since in your case the exclude status only
matters on matching messages we have already computed it (although the
exclude information would need to passed to create_thread)

Patch 7: loses the exclude flag bits (5 lines or so) but the bulk is the
--no-exclude for show.

Patch 8 man-page stays

Patch 9 (updating tests) might not be needed since the output is
changing less.

Patch 10 the "omit" excluded results totally option. What would you do
for notmuch search --output=messages? either it outputs all matching
messages or only those that match and are not excluded. The code can of
course check, but things like --output=tags are more annoying as that is
back into the library code. In any event I think something here is
needed, and in fact any solution here would apply to either version.

Patch 11: emacs would be a bit simpler in your case.

However, you would need to special case the id: stuff (which could
probably be done in Austin's code that examines the parsed query for
tags). 


One possibility would be for my patch to set the match status based on
match flag and not exclude flag, and separately return the whole
flag-set. This might give nicer behaviour for current callers whilst
giving the full information to new callers.

(This email was mostly written when Austin sent his)

Best wishes

Mark

Re: [RFC PATCH v5 01/11] cli: add --no-exclude option to count and search.

[Austin Clements]

Quoth Mark Walters on Feb 15 at  9:03 am:
> This option turns off the exclusion so all matching messages are
> returned. We do not need to add this to notmuch-show as that does not
> (yet) exclude.
> ---
>  notmuch-count.c  |   17 +++++++++++------
>  notmuch-search.c |   17 +++++++++++------
>  2 files changed, 22 insertions(+), 12 deletions(-)
> 
> diff --git a/notmuch-count.c b/notmuch-count.c
> index 63459fb..5364507 100644
> --- a/notmuch-count.c
> +++ b/notmuch-count.c
> @@ -35,8 +35,7 @@ notmuch_count_command (void *ctx, int argc, char *argv[])
>      char *query_str;
>      int opt_index;
>      int output = OUTPUT_MESSAGES;
> -    const char **search_exclude_tags;
> -    size_t search_exclude_tags_length;
> +    notmuch_bool_t no_exclude = FALSE;
>      unsigned int i;
>  
>      notmuch_opt_desc_t options[] = {
> @@ -44,6 +43,7 @@ notmuch_count_command (void *ctx, int argc, char *argv[])
>  	  (notmuch_keyword_t []){ { "threads", OUTPUT_THREADS },
>  				  { "messages", OUTPUT_MESSAGES },
>  				  { 0, 0 } } },
> +	{ NOTMUCH_OPT_BOOLEAN, &no_exclude, "no-exclude", 'd', 0 },
>  	{ 0, 0, 0, 0, 0 }
>      };
>  
> @@ -78,10 +78,15 @@ notmuch_count_command (void *ctx, int argc, char *argv[])
>  	return 1;
>      }
>  
> -    search_exclude_tags = notmuch_config_get_search_exclude_tags
> -	(config, &search_exclude_tags_length);
> -    for (i = 0; i < search_exclude_tags_length; i++)
> -	notmuch_query_add_tag_exclude (query, search_exclude_tags[i]);
> +    if (!no_exclude) {
> +	const char **search_exclude_tags;
> +	size_t search_exclude_tags_length;
> +
> +	search_exclude_tags = notmuch_config_get_search_exclude_tags
> +	    (config, &search_exclude_tags_length);
> +	for (i = 0; i < search_exclude_tags_length; i++)
> +	    notmuch_query_add_tag_exclude (query, search_exclude_tags[i]);
> +    }
>  
>      switch (output) {
>      case OUTPUT_MESSAGES:
> diff --git a/notmuch-search.c b/notmuch-search.c
> index d504051..43ec90b 100644
> --- a/notmuch-search.c
> +++ b/notmuch-search.c
> @@ -423,8 +423,7 @@ notmuch_search_command (void *ctx, int argc, char *argv[])
>      output_t output = OUTPUT_SUMMARY;
>      int offset = 0;
>      int limit = -1; /* unlimited */
> -    const char **search_exclude_tags;
> -    size_t search_exclude_tags_length;
> +    notmuch_bool_t no_exclude = FALSE;
>      unsigned int i;
>  
>      enum { NOTMUCH_FORMAT_JSON, NOTMUCH_FORMAT_TEXT }
> @@ -446,6 +445,7 @@ notmuch_search_command (void *ctx, int argc, char *argv[])
>  				  { "files", OUTPUT_FILES },
>  				  { "tags", OUTPUT_TAGS },
>  				  { 0, 0 } } },
> +        { NOTMUCH_OPT_BOOLEAN, &no_exclude, "no-exclude", 'd', 0 },

Indentation error.  Otherwise this patch LGTM.

>  	{ NOTMUCH_OPT_INT, &offset, "offset", 'O', 0 },
>  	{ NOTMUCH_OPT_INT, &limit, "limit", 'L', 0  },
>  	{ 0, 0, 0, 0, 0 }
> @@ -493,10 +493,15 @@ notmuch_search_command (void *ctx, int argc, char *argv[])
>  
>      notmuch_query_set_sort (query, sort);
>  
> -    search_exclude_tags = notmuch_config_get_search_exclude_tags
> -	(config, &search_exclude_tags_length);
> -    for (i = 0; i < search_exclude_tags_length; i++)
> -	notmuch_query_add_tag_exclude (query, search_exclude_tags[i]);
> +    if (!no_exclude) {
> +	const char **search_exclude_tags;
> +	size_t search_exclude_tags_length;
> +
> +	search_exclude_tags = notmuch_config_get_search_exclude_tags
> +	    (config, &search_exclude_tags_length);
> +	for (i = 0; i < search_exclude_tags_length; i++)
> +	    notmuch_query_add_tag_exclude (query, search_exclude_tags[i]);
> +    }
>  
>      switch (output) {
>      default:

Re: [RFC PATCH v5 02/11] cli: Add --no-exclude to the man pages for search and count

[Austin Clements]

Quoth Mark Walters on Feb 15 at  9:03 am:
> ---
>  man/man1/notmuch-count.1  |    7 +++++++
>  man/man1/notmuch-search.1 |    7 +++++++
>  2 files changed, 14 insertions(+), 0 deletions(-)
> 
> diff --git a/man/man1/notmuch-count.1 b/man/man1/notmuch-count.1
> index 25fe329..413b405 100644
> --- a/man/man1/notmuch-count.1
> +++ b/man/man1/notmuch-count.1
> @@ -38,6 +38,13 @@ Output the number of matching messages. This is the default.
>  Output the number of matching threads.
>  .RE
>  .RE
> +
> +.RS 4
> +.TP 4
> +.BR \-\-no\-exclude
> +
> +Do not exclude the messages matching search_exclude_tags in the config file.

Should be search.exclude_tags, since search_exclude_tags is just the
internal variable name.

> +.RE
>  .RE
>  .RE
>  
> diff --git a/man/man1/notmuch-search.1 b/man/man1/notmuch-search.1
> index 0bc3f40..bc54b4d 100644
> --- a/man/man1/notmuch-search.1
> +++ b/man/man1/notmuch-search.1
> @@ -112,6 +112,13 @@ result from the end.
>  Limit the number of displayed results to N.
>  .RE
>  
> +.RS 4
> +.TP 4
> +.BR \-\-no\-exclude
> +
> +Do not exclude the messages matching search_exclude_tags in the config file.

Same.

> +.RE
> +
>  .SH SEE ALSO
>  
>  \fBnotmuch\fR(1), \fBnotmuch-config\fR(1), \fBnotmuch-count\fR(1),

Re: [RFC PATCH v5 05/11] lib: Make notmuch_query_search_messages set the exclude flag

[Austin Clements]

Quoth Mark Walters on Feb 15 at  9:03 am:
> Add a flag NOTMUCH_MESSAGE_FLAG_EXCLUDED which is set by
> notmuch_query_search_messages for excluded messages. Also add an
> option omit_excluded_messages to the search that we do not want the
> excludes at all.
> 
> This exclude flag will be added to notmuch_query_search threads in the
> next patch.
> ---
>  lib/notmuch-private.h |    1 +
>  lib/notmuch.h         |   10 ++++++++-
>  lib/query.cc          |   52 +++++++++++++++++++++++++++++++++++++++++++++---
>  3 files changed, 58 insertions(+), 5 deletions(-)
> 
> diff --git a/lib/notmuch-private.h b/lib/notmuch-private.h
> index 7bf153e..e791bb0 100644
> --- a/lib/notmuch-private.h
> +++ b/lib/notmuch-private.h
> @@ -401,6 +401,7 @@ typedef struct _notmuch_message_list {
>   */
>  struct visible _notmuch_messages {
>      notmuch_bool_t is_of_list_type;
> +    notmuch_doc_id_set_t *excluded_doc_ids;
>      notmuch_message_node_t *iterator;
>  };
>  
> diff --git a/lib/notmuch.h b/lib/notmuch.h
> index 7929fe7..f75afae 100644
> --- a/lib/notmuch.h
> +++ b/lib/notmuch.h
> @@ -449,6 +449,13 @@ typedef enum {
>  const char *
>  notmuch_query_get_query_string (notmuch_query_t *query);
>  
> +/* Specify whether to results should omit the excluded results rather
> + * than just marking them excluded. This is useful for passing a
> + * notmuch_messages_t not containing the excluded messages to other
> + * functions. */
> +void
> +notmuch_query_set_omit_excluded_messages (notmuch_query_t *query, notmuch_bool_t omit);
> +
>  /* Specify the sorting desired for this query. */
>  void
>  notmuch_query_set_sort (notmuch_query_t *query, notmuch_sort_t sort);
> @@ -895,7 +902,8 @@ notmuch_message_get_filenames (notmuch_message_t *message);
>  
>  /* Message flags */
>  typedef enum _notmuch_message_flag {
> -    NOTMUCH_MESSAGE_FLAG_MATCH
> +    NOTMUCH_MESSAGE_FLAG_MATCH,
> +    NOTMUCH_MESSAGE_FLAG_EXCLUDED
>  } notmuch_message_flag_t;
>  
>  /* Get a value of a flag for the email corresponding to 'message'. */
> diff --git a/lib/query.cc b/lib/query.cc
> index c25b301..90a71a1 100644
> --- a/lib/query.cc
> +++ b/lib/query.cc
> @@ -28,6 +28,7 @@ struct _notmuch_query {
>      const char *query_string;
>      notmuch_sort_t sort;
>      notmuch_string_list_t *exclude_terms;
> +    notmuch_bool_t omit_excluded_messages;
>  };
>  
>  typedef struct _notmuch_mset_messages {
> @@ -57,6 +58,12 @@ struct visible _notmuch_threads {
>      notmuch_doc_id_set_t match_set;
>  };
>  
> +/* We need this in the message functions so forward declare. */
> +static notmuch_bool_t
> +_notmuch_doc_id_set_init (void *ctx,
> +			  notmuch_doc_id_set_t *doc_ids,
> +			  GArray *arr);
> +
>  notmuch_query_t *
>  notmuch_query_create (notmuch_database_t *notmuch,
>  		      const char *query_string)
> @@ -79,6 +86,8 @@ notmuch_query_create (notmuch_database_t *notmuch,
>  
>      query->exclude_terms = _notmuch_string_list_create (query);
>  
> +    query->omit_excluded_messages = FALSE;
> +
>      return query;
>  }
>  
> @@ -89,6 +98,12 @@ notmuch_query_get_query_string (notmuch_query_t *query)
>  }
>  
>  void
> +notmuch_query_set_omit_excluded_messages (notmuch_query_t *query, notmuch_bool_t omit)
> +{
> +    query->omit_excluded_messages = omit;
> +}
> +
> +void
>  notmuch_query_set_sort (notmuch_query_t *query, notmuch_sort_t sort)
>  {
>      query->sort = sort;
> @@ -173,6 +188,7 @@ notmuch_query_search_messages (notmuch_query_t *query)
>  						   "mail"));
>  	Xapian::Query string_query, final_query, exclude_query;
>  	Xapian::MSet mset;
> +	Xapian::MSetIterator iterator;
>  	unsigned int flags = (Xapian::QueryParser::FLAG_BOOLEAN |
>  			      Xapian::QueryParser::FLAG_PHRASE |
>  			      Xapian::QueryParser::FLAG_LOVEHATE |
> @@ -190,11 +206,35 @@ notmuch_query_search_messages (notmuch_query_t *query)
>  	    final_query = Xapian::Query (Xapian::Query::OP_AND,
>  					 mail_query, string_query);
>  	}
> +	messages->base.excluded_doc_ids = NULL;
> +
> +	if (query->exclude_terms) {
> +	    exclude_query = _notmuch_exclude_tags (query, final_query);
> +	    exclude_query = Xapian::Query (Xapian::Query::OP_AND,
> +					   exclude_query, final_query);
> +
> +	    if (query->omit_excluded_messages)
> +		final_query = Xapian::Query (Xapian::Query::OP_AND_NOT,
> +					     final_query, exclude_query);
> +	    else {
> +		enquire.set_weighting_scheme (Xapian::BoolWeight());
> +		enquire.set_query (exclude_query);
> +
> +		mset = enquire.get_mset (0, notmuch->xapian_db->get_doccount ());
> +
> +		GArray *excluded_doc_ids = g_array_new (FALSE, FALSE, sizeof (unsigned int));
> +
> +		for (iterator = mset.begin (); iterator != mset.end (); iterator++) {
> +		    unsigned int doc_id = *iterator;
> +		    g_array_append_val (excluded_doc_ids, doc_id);
> +		}
> +		messages->base.excluded_doc_ids = talloc (query, _notmuch_doc_id_set);

Should this have 'messages' as its talloc context?

> +		_notmuch_doc_id_set_init (query, messages->base.excluded_doc_ids,
> +					  excluded_doc_ids);
> +		g_array_unref (excluded_doc_ids);
> +	    }
> +	}
>  
> -	exclude_query = _notmuch_exclude_tags (query, final_query);
> -
> -	final_query = Xapian::Query (Xapian::Query::OP_AND_NOT,
> -					 final_query, exclude_query);
>  
>  	enquire.set_weighting_scheme (Xapian::BoolWeight());
>  
> @@ -283,6 +323,10 @@ _notmuch_mset_messages_get (notmuch_messages_t *messages)
>  	INTERNAL_ERROR ("a messages iterator contains a non-existent document ID.\n");
>      }
>  
> +    if (messages->excluded_doc_ids &&
> +	_notmuch_doc_id_set_contains (messages->excluded_doc_ids, doc_id))
> +	notmuch_message_set_flag (message, NOTMUCH_MESSAGE_FLAG_EXCLUDED, TRUE);
> +

This, at least, is pleasingly simple!

>      return message;
>  }
>  

Re: [RFC PATCH v5 08/11] man: update manpage for notmuch-show --no-exclude option

[Austin Clements]

Quoth Mark Walters on Feb 15 at  9:03 am:
> ---
>  man/man1/notmuch-show.1 |    7 +++++++
>  1 files changed, 7 insertions(+), 0 deletions(-)
> 
> diff --git a/man/man1/notmuch-show.1 b/man/man1/notmuch-show.1
> index b2301d8..f743a53 100644
> --- a/man/man1/notmuch-show.1
> +++ b/man/man1/notmuch-show.1
> @@ -128,6 +128,13 @@ multipart/encrypted part will be replaced by the decrypted
>  content.
>  .RE
>  
> +.RS 4
> +.TP 4
> +.B \-\-no-exclude
> +
> +Do not exclude the messages matching search_exclude_tags in the config file.

search.exclude_tags

> +.RE
> +
>  A common use of
>  .B notmuch show
>  is to display a single thread of email messages. For this, use a

Re: [RFC PATCH v5 09/11] test: update tests to reflect the exclude flag

[Austin Clements]

Did you mean to put this one right after the show format update,
rather than having the man update between the two?  (I would argue
that this should go in the same patch as the show format update so
that the tests never fail, but people seem to disagree about this.)

Quoth Mark Walters on Feb 15 at  9:03 am:
> notmuch show outputs the exclude flag so many tests using notmuch
> show failed. This commit adds "excluded:0" or "excluded: false" to
> the expected outputs. After this commit there should be no failing
> tests.
> ---
>  test/crypto        |    9 ++++++++-
>  test/encoding      |    2 +-
>  test/json          |    6 +++---
>  test/maildir-sync  |    1 +
>  test/multipart     |    4 ++--
>  test/thread-naming |   16 ++++++++--------
>  6 files changed, 23 insertions(+), 15 deletions(-)
> 
> diff --git a/test/crypto b/test/crypto
> index 1dbb60a..564cf9a 100755
> --- a/test/crypto
> +++ b/test/crypto
> @@ -43,6 +43,7 @@ output=$(notmuch show --format=json --verify subject:"test signed message 001" \
>      | sed -e 's|"created": [1234567890]*|"created": 946728000|')
>  expected='[[[{"id": "XXXXX",
>   "match": true,
> + "excluded": false,
>   "filename": "YYYYY",
>   "timestamp": 946728000,
>   "date_relative": "2000-01-01",
> @@ -77,6 +78,7 @@ output=$(notmuch show --format=json --verify subject:"test signed message 001" \
>      | sed -e 's|"created": [1234567890]*|"created": 946728000|')
>  expected='[[[{"id": "XXXXX",
>   "match": true,
> + "excluded": false,
>   "filename": "YYYYY",
>   "timestamp": 946728000,
>   "date_relative": "2000-01-01",
> @@ -113,6 +115,7 @@ output=$(notmuch show --format=json --verify subject:"test signed message 001" \
>      | sed -e 's|"created": [1234567890]*|"created": 946728000|')
>  expected='[[[{"id": "XXXXX",
>   "match": true,
> + "excluded": false,
>   "filename": "YYYYY",
>   "timestamp": 946728000,
>   "date_relative": "2000-01-01",
> @@ -153,7 +156,7 @@ test_begin_subtest "decryption, --format=text"
>  output=$(notmuch show --format=text --decrypt subject:"test encrypted message 001" \
>      | notmuch_show_sanitize_all \
>      | sed -e 's|"created": [1234567890]*|"created": 946728000|')
> -expected='\fmessage{ id:XXXXX depth:0 match:1 filename:XXXXX
> +expected='\fmessage{ id:XXXXX depth:0 match:1 excluded:0 filename:XXXXX
>  \fheader{
>  Notmuch Test Suite <test_suite@notmuchmail.org> (2000-01-01) (encrypted inbox)
>  Subject: test encrypted message 001
> @@ -187,6 +190,7 @@ output=$(notmuch show --format=json --decrypt subject:"test encrypted message 00
>      | sed -e 's|"created": [1234567890]*|"created": 946728000|')
>  expected='[[[{"id": "XXXXX",
>   "match": true,
> + "excluded": false,
>   "filename": "YYYYY",
>   "timestamp": 946728000,
>   "date_relative": "2000-01-01",
> @@ -242,6 +246,7 @@ output=$(notmuch show --format=json --decrypt subject:"test encrypted message 00
>      | sed -e 's|"created": [1234567890]*|"created": 946728000|')
>  expected='[[[{"id": "XXXXX",
>   "match": true,
> + "excluded": false,
>   "filename": "YYYYY",
>   "timestamp": 946728000,
>   "date_relative": "2000-01-01",
> @@ -277,6 +282,7 @@ output=$(notmuch show --format=json --decrypt subject:"test encrypted message 00
>      | sed -e 's|"created": [1234567890]*|"created": 946728000|')
>  expected='[[[{"id": "XXXXX",
>   "match": true,
> + "excluded": false,
>   "filename": "YYYYY",
>   "timestamp": 946728000,
>   "date_relative": "2000-01-01",
> @@ -332,6 +338,7 @@ output=$(notmuch show --format=json --verify subject:"test signed message 001" \
>      | sed -e 's|"created": [1234567890]*|"created": 946728000|')
>  expected='[[[{"id": "XXXXX",
>   "match": true,
> + "excluded": false,
>   "filename": "YYYYY",
>   "timestamp": 946728000,
>   "date_relative": "2000-01-01",
> diff --git a/test/encoding b/test/encoding
> index 33259c1..a872345 100755
> --- a/test/encoding
> +++ b/test/encoding
> @@ -6,7 +6,7 @@ test_begin_subtest "Message with text of unknown charset"
>  add_message '[content-type]="text/plain; charset=unknown-8bit"' \
>  	    "[body]=irrelevant"
>  output=$(notmuch show id:${gen_msg_id} 2>&1 | notmuch_show_sanitize)
> -test_expect_equal "$output" "\fmessage{ id:msg-001@notmuch-test-suite depth:0 match:1 filename:/XXX/mail/msg-001
> +test_expect_equal "$output" "\fmessage{ id:msg-001@notmuch-test-suite depth:0 match:1 excluded:0 filename:/XXX/mail/msg-001
>  \fheader{
>  Notmuch Test Suite <test_suite@notmuchmail.org> (2001-01-05) (inbox unread)
>  Subject: Test message #1
> diff --git a/test/json b/test/json
> index 7df4380..f95fcf8 100755
> --- a/test/json
> +++ b/test/json
> @@ -5,7 +5,7 @@ test_description="--format=json output"
>  test_begin_subtest "Show message: json"
>  add_message "[subject]=\"json-show-subject\"" "[date]=\"Sat, 01 Jan 2000 12:00:00 -0000\"" "[body]=\"json-show-message\""
>  output=$(notmuch show --format=json "json-show-message")
> -test_expect_equal "$output" "[[[{\"id\": \"${gen_msg_id}\", \"match\": true, \"filename\": \"${gen_msg_filename}\", \"timestamp\": 946728000, \"date_relative\": \"2000-01-01\", \"tags\": [\"inbox\",\"unread\"], \"headers\": {\"Subject\": \"json-show-subject\", \"From\": \"Notmuch Test Suite <test_suite@notmuchmail.org>\", \"To\": \"Notmuch Test Suite <test_suite@notmuchmail.org>\", \"Cc\": \"\", \"Bcc\": \"\", \"Date\": \"Sat, 01 Jan 2000 12:00:00 -0000\"}, \"body\": [{\"id\": 1, \"content-type\": \"text/plain\", \"content\": \"json-show-message\n\"}]}, []]]]"
> +test_expect_equal "$output" "[[[{\"id\": \"${gen_msg_id}\", \"match\": true, \"excluded\": false, \"filename\": \"${gen_msg_filename}\", \"timestamp\": 946728000, \"date_relative\": \"2000-01-01\", \"tags\": [\"inbox\",\"unread\"], \"headers\": {\"Subject\": \"json-show-subject\", \"From\": \"Notmuch Test Suite <test_suite@notmuchmail.org>\", \"To\": \"Notmuch Test Suite <test_suite@notmuchmail.org>\", \"Cc\": \"\", \"Bcc\": \"\", \"Date\": \"Sat, 01 Jan 2000 12:00:00 -0000\"}, \"body\": [{\"id\": 1, \"content-type\": \"text/plain\", \"content\": \"json-show-message\n\"}]}, []]]]"
>  
>  test_begin_subtest "Search message: json"
>  add_message "[subject]=\"json-search-subject\"" "[date]=\"Sat, 01 Jan 2000 12:00:00 -0000\"" "[body]=\"json-search-message\""
> @@ -22,7 +22,7 @@ test_expect_equal "$output" "[{\"thread\": \"XXX\",
>  test_begin_subtest "Show message: json, utf-8"
>  add_message "[subject]=\"json-show-utf8-body-sübjéct\"" "[date]=\"Sat, 01 Jan 2000 12:00:00 -0000\"" "[body]=\"jsön-show-méssage\""
>  output=$(notmuch show --format=json "jsön-show-méssage")
> -test_expect_equal "$output" "[[[{\"id\": \"${gen_msg_id}\", \"match\": true, \"filename\": \"${gen_msg_filename}\", \"timestamp\": 946728000, \"date_relative\": \"2000-01-01\", \"tags\": [\"inbox\",\"unread\"], \"headers\": {\"Subject\": \"json-show-utf8-body-sübjéct\", \"From\": \"Notmuch Test Suite <test_suite@notmuchmail.org>\", \"To\": \"Notmuch Test Suite <test_suite@notmuchmail.org>\", \"Cc\": \"\", \"Bcc\": \"\", \"Date\": \"Sat, 01 Jan 2000 12:00:00 -0000\"}, \"body\": [{\"id\": 1, \"content-type\": \"text/plain\", \"content\": \"jsön-show-méssage\n\"}]}, []]]]"
> +test_expect_equal "$output" "[[[{\"id\": \"${gen_msg_id}\", \"match\": true, \"excluded\": false, \"filename\": \"${gen_msg_filename}\", \"timestamp\": 946728000, \"date_relative\": \"2000-01-01\", \"tags\": [\"inbox\",\"unread\"], \"headers\": {\"Subject\": \"json-show-utf8-body-sübjéct\", \"From\": \"Notmuch Test Suite <test_suite@notmuchmail.org>\", \"To\": \"Notmuch Test Suite <test_suite@notmuchmail.org>\", \"Cc\": \"\", \"Bcc\": \"\", \"Date\": \"Sat, 01 Jan 2000 12:00:00 -0000\"}, \"body\": [{\"id\": 1, \"content-type\": \"text/plain\", \"content\": \"jsön-show-méssage\n\"}]}, []]]]"
>  
>  test_begin_subtest "Show message: json, inline attachment filename"
>  subject='json-show-inline-attachment-filename'
> @@ -35,7 +35,7 @@ emacs_deliver_message \
>       (insert \"Message-ID: <$id>\n\")"
>  output=$(notmuch show --format=json "id:$id")
>  filename=$(notmuch search --output=files "id:$id")
> -test_expect_equal "$output" "[[[{\"id\": \"$id\", \"match\": true, \"filename\": \"$filename\", \"timestamp\": 946728000, \"date_relative\": \"2000-01-01\", \"tags\": [\"inbox\"], \"headers\": {\"Subject\": \"$subject\", \"From\": \"Notmuch Test Suite <test_suite@notmuchmail.org>\", \"To\": \"test_suite@notmuchmail.org\", \"Cc\": \"\", \"Bcc\": \"\", \"Date\": \"01 Jan 2000 12:00:00 -0000\"}, \"body\": [{\"id\": 1, \"content-type\": \"multipart/mixed\", \"content\": [{\"id\": 2, \"content-type\": \"text/plain\", \"content\": \"This is a test message with inline attachment with a filename\"}, {\"id\": 3, \"content-type\": \"application/octet-stream\", \"filename\": \"README\"}]}]}, []]]]"
> +test_expect_equal "$output" "[[[{\"id\": \"$id\", \"match\": true, \"excluded\": false, \"filename\": \"$filename\", \"timestamp\": 946728000, \"date_relative\": \"2000-01-01\", \"tags\": [\"inbox\"], \"headers\": {\"Subject\": \"$subject\", \"From\": \"Notmuch Test Suite <test_suite@notmuchmail.org>\", \"To\": \"test_suite@notmuchmail.org\", \"Cc\": \"\", \"Bcc\": \"\", \"Date\": \"01 Jan 2000 12:00:00 -0000\"}, \"body\": [{\"id\": 1, \"content-type\": \"multipart/mixed\", \"content\": [{\"id\": 2, \"content-type\": \"text/plain\", \"content\": \"This is a test message with inline attachment with a filename\"}, {\"id\": 3, \"content-type\": \"application/octet-stream\", \"filename\": \"README\"}]}]}, []]]]"
>  
>  test_begin_subtest "Search message: json, utf-8"
>  add_message "[subject]=\"json-search-utf8-body-sübjéct\"" "[date]=\"Sat, 01 Jan 2000 12:00:00 -0000\"" "[body]=\"jsön-search-méssage\""
> diff --git a/test/maildir-sync b/test/maildir-sync
> index d5872a5..dfa3966 100755
> --- a/test/maildir-sync
> +++ b/test/maildir-sync
> @@ -46,6 +46,7 @@ test_begin_subtest "notmuch show works with renamed file (without notmuch new)"
>  output=$(notmuch show --format=json id:${gen_msg_id} | filter_show_json)
>  test_expect_equal "$output" '[[[{"id": "adding-replied-tag@notmuch-test-suite",
>  "match": true,
> +"excluded": false,
>  "filename": "MAIL_DIR/cur/adding-replied-tag:2,RS",
>  "timestamp": 978709437,
>  "date_relative": "2001-01-05",
> diff --git a/test/multipart b/test/multipart
> index 2dd73f5..3d2dcfc 100755
> --- a/test/multipart
> +++ b/test/multipart
> @@ -108,7 +108,7 @@ notmuch new > /dev/null
>  test_begin_subtest "--format=text --part=0, full message"
>  notmuch show --format=text --part=0 'id:87liy5ap00.fsf@yoom.home.cworth.org' >OUTPUT
>  cat <<EOF >EXPECTED
> -\fmessage{ id:87liy5ap00.fsf@yoom.home.cworth.org depth:0 match:1 filename:${MAIL_DIR}/multipart
> +\fmessage{ id:87liy5ap00.fsf@yoom.home.cworth.org depth:0 match:1 excluded:0 filename:${MAIL_DIR}/multipart
>  \fheader{
>  Carl Worth <cworth@cworth.org> (2001-01-05) (attachment inbox signed unread)
>  Subject: Multipart message
> @@ -322,7 +322,7 @@ notmuch show --format=json --part=0 'id:87liy5ap00.fsf@yoom.home.cworth.org' | s
>  echo >>OUTPUT # expect *no* newline at end of output
>  cat <<EOF >EXPECTED
>  
> -{"id": "87liy5ap00.fsf@yoom.home.cworth.org", "match": true, "filename": "${MAIL_DIR}/multipart", "timestamp": 978709437, "date_relative": "2001-01-05", "tags": ["attachment","inbox","signed","unread"], "headers": {"Subject": "Multipart message", "From": "Carl Worth <cworth@cworth.org>", "To": "cworth@cworth.org", "Cc": "", "Bcc": "", "Date": "Fri, 05 Jan 2001 15:43:57 +0000"}, "body": [
> +{"id": "87liy5ap00.fsf@yoom.home.cworth.org", "match": true, "excluded": false, "filename": "${MAIL_DIR}/multipart", "timestamp": 978709437, "date_relative": "2001-01-05", "tags": ["attachment","inbox","signed","unread"], "headers": {"Subject": "Multipart message", "From": "Carl Worth <cworth@cworth.org>", "To": "cworth@cworth.org", "Cc": "", "Bcc": "", "Date": "Fri, 05 Jan 2001 15:43:57 +0000"}, "body": [
>  {"id": 1, "content-type": "multipart/signed", "content": [
>  {"id": 2, "content-type": "multipart/mixed", "content": [
>  {"id": 3, "content-type": "message/rfc822", "content": [{"headers": {"From": "Carl Worth <cworth@cworth.org>", "To": "cworth@cworth.org", "Subject": "html message", "Date": "Fri, 05 Jan 2001 15:42:57 +0000"}, "body": [
> diff --git a/test/thread-naming b/test/thread-naming
> index 942e593..1a1a48f 100755
> --- a/test/thread-naming
> +++ b/test/thread-naming
> @@ -65,7 +65,7 @@ test_expect_equal "$output" "thread:XXX   2001-01-12 [6/8] Notmuch Test Suite; t
>  
>  test_begin_subtest 'Test order of messages in "notmuch show"'
>  output=$(notmuch show thread-naming | notmuch_show_sanitize)
> -test_expect_equal "$output" "\fmessage{ id:msg-$(printf "%03d" $first)@notmuch-test-suite depth:0 match:1 filename:/XXX/mail/msg-$(printf "%03d" $first)
> +test_expect_equal "$output" "\fmessage{ id:msg-$(printf "%03d" $first)@notmuch-test-suite depth:0 match:1 excluded:0 filename:/XXX/mail/msg-$(printf "%03d" $first)
>  \fheader{
>  Notmuch Test Suite <test_suite@notmuchmail.org> (2001-01-05) (unread)
>  Subject: thread-naming: Initial thread subject
> @@ -79,7 +79,7 @@ This is just a test message (#$first)
>  \fpart}
>  \fbody}
>  \fmessage}
> -\fmessage{ id:msg-$(printf "%03d" $((first + 1)))@notmuch-test-suite depth:1 match:1 filename:/XXX/mail/msg-$(printf "%03d" $((first + 1)))
> +\fmessage{ id:msg-$(printf "%03d" $((first + 1)))@notmuch-test-suite depth:1 match:1 excluded:0 filename:/XXX/mail/msg-$(printf "%03d" $((first + 1)))
>  \fheader{
>  Notmuch Test Suite <test_suite@notmuchmail.org> (2001-01-06) (inbox unread)
>  Subject: thread-naming: Older changed subject
> @@ -93,7 +93,7 @@ This is just a test message (#$((first + 1)))
>  \fpart}
>  \fbody}
>  \fmessage}
> -\fmessage{ id:msg-$(printf "%03d" $((first + 2)))@notmuch-test-suite depth:1 match:1 filename:/XXX/mail/msg-$(printf "%03d" $((first + 2)))
> +\fmessage{ id:msg-$(printf "%03d" $((first + 2)))@notmuch-test-suite depth:1 match:1 excluded:0 filename:/XXX/mail/msg-$(printf "%03d" $((first + 2)))
>  \fheader{
>  Notmuch Test Suite <test_suite@notmuchmail.org> (2001-01-07) (inbox unread)
>  Subject: thread-naming: Newer changed subject
> @@ -107,7 +107,7 @@ This is just a test message (#$((first + 2)))
>  \fpart}
>  \fbody}
>  \fmessage}
> -\fmessage{ id:msg-$(printf "%03d" $((first + 3)))@notmuch-test-suite depth:1 match:1 filename:/XXX/mail/msg-$(printf "%03d" $((first + 3)))
> +\fmessage{ id:msg-$(printf "%03d" $((first + 3)))@notmuch-test-suite depth:1 match:1 excluded:0 filename:/XXX/mail/msg-$(printf "%03d" $((first + 3)))
>  \fheader{
>  Notmuch Test Suite <test_suite@notmuchmail.org> (2001-01-08) (unread)
>  Subject: thread-naming: Final thread subject
> @@ -121,7 +121,7 @@ This is just a test message (#$((first + 3)))
>  \fpart}
>  \fbody}
>  \fmessage}
> -\fmessage{ id:msg-$(printf "%03d" $((first + 4)))@notmuch-test-suite depth:1 match:1 filename:/XXX/mail/msg-$(printf "%03d" $((first + 4)))
> +\fmessage{ id:msg-$(printf "%03d" $((first + 4)))@notmuch-test-suite depth:1 match:1 excluded:0 filename:/XXX/mail/msg-$(printf "%03d" $((first + 4)))
>  \fheader{
>  Notmuch Test Suite <test_suite@notmuchmail.org> (2001-01-09) (inbox unread)
>  Subject: Re: thread-naming: Initial thread subject
> @@ -135,7 +135,7 @@ This is just a test message (#$((first + 4)))
>  \fpart}
>  \fbody}
>  \fmessage}
> -\fmessage{ id:msg-$(printf "%03d" $((first + 5)))@notmuch-test-suite depth:1 match:1 filename:/XXX/mail/msg-$(printf "%03d" $((first + 5)))
> +\fmessage{ id:msg-$(printf "%03d" $((first + 5)))@notmuch-test-suite depth:1 match:1 excluded:0 filename:/XXX/mail/msg-$(printf "%03d" $((first + 5)))
>  \fheader{
>  Notmuch Test Suite <test_suite@notmuchmail.org> (2001-01-10) (inbox unread)
>  Subject: Aw: thread-naming: Initial thread subject
> @@ -149,7 +149,7 @@ This is just a test message (#$((first + 5)))
>  \fpart}
>  \fbody}
>  \fmessage}
> -\fmessage{ id:msg-$(printf "%03d" $((first + 6)))@notmuch-test-suite depth:1 match:1 filename:/XXX/mail/msg-$(printf "%03d" $((first + 6)))
> +\fmessage{ id:msg-$(printf "%03d" $((first + 6)))@notmuch-test-suite depth:1 match:1 excluded:0 filename:/XXX/mail/msg-$(printf "%03d" $((first + 6)))
>  \fheader{
>  Notmuch Test Suite <test_suite@notmuchmail.org> (2001-01-11) (inbox unread)
>  Subject: Vs: thread-naming: Initial thread subject
> @@ -163,7 +163,7 @@ This is just a test message (#$((first + 6)))
>  \fpart}
>  \fbody}
>  \fmessage}
> -\fmessage{ id:msg-$(printf "%03d" $((first + 7)))@notmuch-test-suite depth:1 match:1 filename:/XXX/mail/msg-$(printf "%03d" $((first + 7)))
> +\fmessage{ id:msg-$(printf "%03d" $((first + 7)))@notmuch-test-suite depth:1 match:1 excluded:0 filename:/XXX/mail/msg-$(printf "%03d" $((first + 7)))
>  \fheader{
>  Notmuch Test Suite <test_suite@notmuchmail.org> (2001-01-12) (inbox unread)
>  Subject: Sv: thread-naming: Initial thread subject

Re: [RFC PATCH v5 00/11] Add NOTMUCH_MESSAGE_FLAG_EXCLUDED flag

[Austin Clements]

Quoth Mark Walters on Feb 15 at  9:03 am:
> This is v5 of the exclude flag series. (v4 was at 
> id:"874nv9rv79.fsf@qmul.ac.uk")
> 
> This email has 4 sections, an overview of what the patch set is 
> trying to achieve, a summary of the changes from v4, some comments on 
> the status of the patches in the series and some remaining queries.
> 
> OVERVIEW
> 
> The current implementation of exclude-tags does not use excludes in
> notmuch-show.c (and thus not in notmuch-show.el). Thus when selecting 
> a thread in the search view claiming one matched message you may get 
> several matches in show all but one of which are tagged excluded.
> 
> The trivial change of adding excludes to show does not work as easily 
> as one would like. For example if you try notmuch-show 
> id:<deleted message> you get no results (see 
> id:"871uqvgrnm.fsf@qmul.ac.uk" for more discussion).
> 
> This set moves in a different direction. It returns all the results 
> but marks excluded messages with a new flag 
> (NOTMUCH_MESSAGE_FLAG_EXCLUDED) and lets the consumer decide what to 
> do with them. For example it could start with the message closed in 
> emacs show view, it could colour the headerline differently etc.
> 
> CHANGES:
> This has been rebased on top of Jani's notmuch-show command line 
> parsing patch.
> 
> The function notmuch_thread_get_flag_messages function added in v4
> has been removed. Unfortunately (as Austin pointed out) that patch 
> broke binary compatibility so should be deferred until we wish to 
> bump the library version.
> 
> I also fixed a minor bug and a style comment in the emacs part of 
> the patch.
> 
> STATUS:
> The first 3 patches in the series just add a --no-exclude option
> to notmuch-search.c and notmuch-count.c to "turn off" the excluding.
> (The 3 patches are one for the C code, one for the man pages and 
> one for the tests.) I think this change is desirable independently 
> of the rest of the series (and indeed Jameson had a use for it
> id:"878vk943ci.fsf@servo.finestructure.net").
> 
> QUERIES
> 
> 1) As with v4 the api notmuch_query_set_omit_excluded_messages
> remains: without it I can't see how a user can pass a 
> notmuch_messages_t object around which does not contain the 
> excluded messages. See id:"87fweusabh.fsf@qmul.ac.uk". 

This does seem like a useful simplification.  Another possibility
(which may not work in practice) would be to have a utility function
in the CLI that iterated a notmuch_messages_t to the next non-excluded
message.  This would push the knowledge of whether or not a format can
represent excluded messages into that format, but I can't see how to
do things like "count" this way.

> 2)  If we have a query which overrides the excludes such as "blah and
> tag:deleted" should the tag:deleted messages still be marked excluded?
> The current implementation does mark them excluded but my preference 
> would be not to. What do people think? At the moment 2 tests fail but 
> the correct output depends on the above so I will leave them until we
> have a decision on this.

I could go either way on this, but I think it should still be marked
excluded since it is, after all, excluded.

> There are some other queries mentioned in v4 but the two above are
> the significant ones.
> 
> Best wishes
> 
> Mark
> 
> 
> 
> Mark Walters (11):
>   cli: add --no-exclude option to count and search.
>   cli: Add --no-exclude to the man pages for search and count
>   test: add tests for new cli --no-exclude option
>   lib: Rearrange the exclude code in query.cc
>   lib: Make notmuch_query_search_messages set the exclude flag
>   lib: Add the exclude flag to notmuch_query_search_threads
>   cli: Make notmuch-show respect excludes.
>   man: update manpage for notmuch-show --no-exclude option
>   test: update tests to reflect the exclude flag
>   cli: omit excluded messages in results where appropriate.
>   emacs: show: recognize the exclude flag.
> 
>  emacs/notmuch-show.el     |   19 +++++++++++-
>  lib/notmuch-private.h     |    8 ++++-
>  lib/notmuch.h             |   16 ++++++++--
>  lib/query.cc              |   74 ++++++++++++++++++++++++++++++++++++++------
>  lib/thread.cc             |   18 +++++++++--
>  man/man1/notmuch-count.1  |    7 ++++
>  man/man1/notmuch-search.1 |    7 ++++
>  man/man1/notmuch-show.1   |    7 ++++
>  notmuch-count.c           |   19 ++++++++----
>  notmuch-search.c          |   26 ++++++++++++----
>  notmuch-show.c            |   31 ++++++++++++++++--
>  test/count                |   21 +++++++++++++
>  test/crypto               |    9 +++++-
>  test/encoding             |    2 +-
>  test/json                 |    6 ++--
>  test/maildir-sync         |    1 +
>  test/multipart            |    4 +-
>  test/search               |    5 +++
>  test/thread-naming        |   16 +++++-----
>  19 files changed, 246 insertions(+), 50 deletions(-)