another attempt to add delete functionality in emacs

another attempt to add delete functionality in emacs

[Jameson Graef Rollins]

So, after many stabs at adding the ability to "delete" messages in
emacs [0], and the corresponding heated discussions, I'm throwing
another attempt into the fray.

I try to address the concerns that have come up in previous attempts.
In particular, I include a patch that creates a new customization
variable, notmuch-search-exclude-deleted, that will exclude any
messages with the "deleted" tag from searches.  This actually makes
"deleted" messages appear effectively deleted, which is one of the
things cworth wanted to see, and one of the reasons he kept pushing
back on previous attempts at this functionality.

Also, no tags other than "deleted" are modified.  All tags should be
orthogonal, and should be handled so.

Note: this is all about handling the "deleted" tag.  No actual
deletion of message is involved in this functionality at all.  Actual
deletion of messages should always be left entirely up to the user to
handle as they see fit.

jamie.

[0] id:"1266408746-28549-1-git-send-email-Sebastian@SSpaeth.de"
    id:"87sk8qwjlt.fsf@yoom.home.cworth.org"
    id:"1271891763-10757-1-git-send-email-hohndel@infradead.org"
    id:"1310841600-28281-1-git-send-email-anarcat@koumbit.org"

[PATCH 1/4] emacs: new customization variable to exclude "deleted" messages from search

[Jameson Graef Rollins]

The new customization variable, notmuch-search-exclude-deleted, when
set to t, will exclude any messages with the "deleted" tag from
searches.

Additionally, specifying "tag:deleted" in the search directly will
override the exclusion and will included deleted messages in the
search results.
---
 emacs/notmuch.el                                   |    8 ++++
 test/emacs                                         |   42 ++++++++++++++++++++
 .../notmuch-search-tag-inbox-deleted-excluded      |   24 +++++++++++
 3 files changed, 74 insertions(+), 0 deletions(-)
 create mode 100644 test/emacs.expected-output/notmuch-search-tag-inbox-deleted-excluded

diff --git a/emacs/notmuch.el b/emacs/notmuch.el
index fde2377..c519687 100644
--- a/emacs/notmuch.el
+++ b/emacs/notmuch.el
@@ -905,6 +905,11 @@ PROMPT is the string to prompt with."
       (read-from-minibuffer prompt nil keymap nil
 			    'notmuch-query-history nil nil))))
 
+(defcustom notmuch-search-exclude-deleted nil
+  "Exclude deleted messages (with \"deleted\" tag) from search results."
+  :group 'notmuch
+  :type 'boolean)
+
 ;;;###autoload
 (defun notmuch-search (query &optional oldest-first target-thread target-line continuation)
   "Run \"notmuch search\" with the given query string and display results.
@@ -927,6 +932,9 @@ The optional parameters are used as follows:
     (set 'notmuch-search-target-thread target-thread)
     (set 'notmuch-search-target-line target-line)
     (set 'notmuch-search-continuation continuation)
+    (when (and notmuch-search-exclude-deleted
+	       (not (string-match "tag:deleted[ )]*" query)))
+      (setq query (concat query " and not tag:deleted")))
     (let ((proc (get-buffer-process (current-buffer)))
 	  (inhibit-read-only t))
       (if proc
diff --git a/test/emacs b/test/emacs
index a06c223..1d78fbe 100755
--- a/test/emacs
+++ b/test/emacs
@@ -35,6 +35,48 @@ test_emacs '(notmuch-search "tag:inbox")
 	    (test-output)'
 test_expect_equal_file OUTPUT $EXPECTED/notmuch-search-tag-inbox
 
+test_begin_subtest "Exclude \"deleted\" messages from search"
+notmuch tag +deleted id:1258506353-20352-1-git-send-email-stewart@flamingspork.com
+# we "delete" a second message that's part of a multi-message thread
+# to make sure the rest of the thread is still returned
+notmuch tag +deleted id:1258471718-6781-1-git-send-email-dottedmag@dottedmag.net
+test_emacs '(let ((notmuch-search-exclude-deleted t))
+	      (notmuch-search "tag:inbox")
+	      (notmuch-test-wait)
+	      (test-output))'
+notmuch tag -deleted id:1258506353-20352-1-git-send-email-stewart@flamingspork.com
+notmuch tag -deleted id:1258471718-6781-1-git-send-email-dottedmag@dottedmag.net
+test_expect_equal_file OUTPUT $EXPECTED/notmuch-search-tag-inbox-deleted-excluded
+
+test_begin_subtest "Exclude \"deleted\" messages from search, manual override"
+notmuch tag +deleted id:1258506353-20352-1-git-send-email-stewart@flamingspork.com
+notmuch tag +deleted id:1258471718-6781-1-git-send-email-dottedmag@dottedmag.net
+cat <<EOF >EXPECTED
+  2009-11-18 [1/1]   Stewart Smith        [notmuch] [PATCH] Fix linking with gcc to use g++ to link in C++ libs. (deleted inbox unread)
+  2009-11-17 [1/5]   Mikhail Gusarov, Carl Worth, Keith Packard  [notmuch] [PATCH 1/2] Close message file after parsing message headers (deleted inbox unread)
+End of search results.
+EOF
+test_emacs '(let ((notmuch-search-exclude-deleted t))
+	      (notmuch-search "tag:inbox and tag:deleted")
+	      (notmuch-test-wait)
+	      (test-output))'
+notmuch tag -deleted id:1258506353-20352-1-git-send-email-stewart@flamingspork.com
+notmuch tag -deleted id:1258471718-6781-1-git-send-email-dottedmag@dottedmag.net
+test_expect_equal_file OUTPUT EXPECTED
+
+test_begin_subtest "Exclude \"deleted\" messages from search, but not \"deleted*\""
+notmuch tag +deleted-patch id:1258506353-20352-1-git-send-email-stewart@flamingspork.com
+cat <<EOF >EXPECTED
+  2009-11-18 [1/1]   Stewart Smith        [notmuch] [PATCH] Fix linking with gcc to use g++ to link in C++ libs. (deleted-patch inbox unread)
+End of search results.
+EOF
+test_emacs '(let ((notmuch-search-exclude-deleted t))
+	      (notmuch-search "tag:inbox and tag:deleted-patch")
+	      (notmuch-test-wait)
+	      (test-output))'
+notmuch tag -deleted-patch id:1258506353-20352-1-git-send-email-stewart@flamingspork.com
+test_expect_equal_file OUTPUT EXPECTED
+
 test_begin_subtest "Navigation of notmuch-hello to search results"
 test_emacs '(notmuch-hello)
 	    (goto-char (point-min))
diff --git a/test/emacs.expected-output/notmuch-search-tag-inbox-deleted-excluded b/test/emacs.expected-output/notmuch-search-tag-inbox-deleted-excluded
new file mode 100644
index 0000000..39b4c51
--- /dev/null
+++ b/test/emacs.expected-output/notmuch-search-tag-inbox-deleted-excluded
@@ -0,0 +1,24 @@
+  2010-12-29 [1/1]   François Boulogne    [aur-general] Guidelines: cp, mkdir vs install (inbox unread)
+  2010-12-16 [1/1]   Olivier Berger       Essai accentué (inbox unread)
+  2009-11-18 [1/1]   Chris Wilson         [notmuch] [PATCH 1/2] Makefile: evaluate pkg-config once (inbox unread)
+  2009-11-18 [2/2]   Alex Botero-Lowry, Carl Worth  [notmuch] [PATCH] Error out if no query is supplied to search instead of going into an infinite loop (attachment inbox unread)
+  2009-11-18 [2/2]   Ingmar Vanhassel, Carl Worth  [notmuch] [PATCH] Typsos (inbox unread)
+  2009-11-18 [3/3]   Adrian Perez de Castro, Keith Packard, Carl Worth  [notmuch] Introducing myself (inbox signed unread)
+  2009-11-18 [3/3]   Israel Herraiz, Keith Packard, Carl Worth   [notmuch] New to the list (inbox unread)
+  2009-11-18 [3/3]   Jan Janak, Carl Worth        [notmuch] What a great idea! (inbox unread)
+  2009-11-18 [2/2]   Jan Janak, Carl Worth        [notmuch] [PATCH] Older versions of install do not support -C. (inbox unread)
+  2009-11-18 [3/3]   Aron Griffis, Keith Packard, Carl Worth     [notmuch] archive (inbox unread)
+  2009-11-18 [2/2]   Keith Packard, Carl Worth    [notmuch] [PATCH] Make notmuch-show 'X' (and 'x') commands remove inbox (and unread) tags (inbox unread)
+  2009-11-18 [7/7]   Lars Kellogg-Stedman, Mikhail Gusarov, Keith Packard, Carl Worth  [notmuch] Working with Maildir storage? (inbox signed unread)
+  2009-11-18 [4/5]   Mikhail Gusarov, Carl Worth, Keith Packard  [notmuch] [PATCH 1/2] Close message file after parsing message headers (deleted inbox unread)
+  2009-11-18 [2/2]   Keith Packard, Alexander Botero-Lowry    [notmuch] [PATCH] Create a default notmuch-show-hook that highlights URLs and uses word-wrap (inbox unread)
+  2009-11-18 [1/1]   Alexander Botero-Lowry  [notmuch] request for pull (inbox unread)
+  2009-11-18 [4/4]   Jjgod Jiang, Alexander Botero-Lowry      [notmuch] Mac OS X/Darwin compatibility issues (inbox unread)
+  2009-11-18 [1/1]   Rolland Santimano    [notmuch] Link to mailing list archives ? (inbox unread)
+  2009-11-18 [1/1]   Jan Janak            [notmuch] [PATCH] notmuch new: Support for conversion of spool subdirectories into tags (inbox unread)
+  2009-11-18 [1/1]   Stewart Smith        [notmuch] [PATCH] count_files: sort directory in inode order before statting (inbox unread)
+  2009-11-18 [1/1]   Stewart Smith        [notmuch] [PATCH 2/2] Read mail directory in inode number order (inbox unread)
+  2009-11-18 [2/2]   Lars Kellogg-Stedman [notmuch] "notmuch help" outputs to stderr? (attachment inbox signed unread)
+  2009-11-17 [1/1]   Mikhail Gusarov      [notmuch] [PATCH] Handle rename of message file (inbox unread)
+  2009-11-17 [2/2]   Alex Botero-Lowry, Carl Worth  [notmuch] preliminary FreeBSD support (attachment inbox unread)
+End of search results.
-- 
1.7.7.3

[PATCH 2/4] emacs: repurpose notmuch-show-archive-thread-internal function for general thread tagging

[Jameson Graef Rollins]

Instead of having a function that is only used for archiving a thread,
we instead make it useful for any tagging operation.  The new
function, notmuch-show-tag-thread-internal, now takes two more
arguments, for the "sign" of the tagging operation ("-" or "+"), and
the tag to be added or removed.  This will allow this function to be
used for any generic thread tagging operation.

The higher level functions that call this function are modified
accordingly.
---
 emacs/notmuch-show.el |   34 ++++++++++++++++++++--------------
 1 files changed, 20 insertions(+), 14 deletions(-)

diff --git a/emacs/notmuch-show.el b/emacs/notmuch-show.el
index 5502efd..1e16f05 100644
--- a/emacs/notmuch-show.el
+++ b/emacs/notmuch-show.el
@@ -1414,20 +1414,26 @@ argument, hide all of the messages."
   (interactive)
   (backward-button 1))
 
-(defun notmuch-show-archive-thread-internal (show-next)
+(defun notmuch-show-tag-thread-internal (sign tag show-next)
   ;; Remove the tag from the current set of messages.
   (goto-char (point-min))
-  (loop do (notmuch-show-remove-tag "inbox")
-	until (not (notmuch-show-goto-message-next)))
-  ;; Move to the next item in the search results, if any.
-  (let ((parent-buffer notmuch-show-parent-buffer))
-    (notmuch-kill-this-buffer)
-    (if parent-buffer
-	(progn
-	  (switch-to-buffer parent-buffer)
-	  (forward-line)
-	  (if show-next
-	      (notmuch-search-show-thread))))))
+  (let ((tag-function))
+    (cond
+     ((string= sign "-")
+      (setq tag-function 'notmuch-show-remove-tag))
+     ((string= sign "+")
+      (setq tag-function 'notmuch-show-add-tag)))
+    (loop do (funcall tag-function tag)
+	  until (not (notmuch-show-goto-message-next)))
+    ;; Move to the next item in the search results, if any.
+    (let ((parent-buffer notmuch-show-parent-buffer))
+      (notmuch-kill-this-buffer)
+      (if parent-buffer
+	  (progn
+	    (switch-to-buffer parent-buffer)
+	    (forward-line)
+	    (if show-next
+		(notmuch-search-show-thread)))))))
 
 (defun notmuch-show-archive-thread ()
   "Archive each message in thread, then show next thread from search.
@@ -1441,12 +1447,12 @@ being delivered to the same thread. It does not archive the
 entire thread, but only the messages shown in the current
 buffer."
   (interactive)
-  (notmuch-show-archive-thread-internal t))
+  (notmuch-show-tag-thread-internal "-" "inbox" t))
 
 (defun notmuch-show-archive-thread-then-exit ()
   "Archive each message in thread, then exit back to search results."
   (interactive)
-  (notmuch-show-archive-thread-internal nil))
+  (notmuch-show-tag-thread-internal "-" "inbox" nil))
 
 (defun notmuch-show-stash-cc ()
   "Copy CC field of current message to kill-ring."
-- 
1.7.7.3

[PATCH 3/4] emacs: add ability to "delete" messages and threads

[Jameson Graef Rollins]

This completely mimics the behavior of archiving, but instead of
remove the inbox tag it add the "deleted" tag.  The functionality is
bound to the 'd' key in both search and show mode.

Note: this does *not* actually delete any messages.  That is still
left entirely up to the user to figure out how they want to handle
that.  This *just* adds the "deleted" tag to messages, no more.  If
the notmuch-search-exclude-deleted config variable is set, though,
"deleted" messages will be excluded from search results.
---
 emacs/notmuch-show.el |   15 +++++++++++++++
 emacs/notmuch.el      |   12 ++++++++++++
 2 files changed, 27 insertions(+), 0 deletions(-)

diff --git a/emacs/notmuch-show.el b/emacs/notmuch-show.el
index 1e16f05..e1d15f4 100644
--- a/emacs/notmuch-show.el
+++ b/emacs/notmuch-show.el
@@ -944,6 +944,7 @@ thread id.  If a prefix is given, crypto processing is toggled."
 	(define-key map "+" 'notmuch-show-add-tag)
 	(define-key map "x" 'notmuch-show-archive-thread-then-exit)
 	(define-key map "a" 'notmuch-show-archive-thread)
+	(define-key map "d" 'notmuch-show-delete-thread)
 	(define-key map "N" 'notmuch-show-next-message)
 	(define-key map "P" 'notmuch-show-previous-message)
 	(define-key map "n" 'notmuch-show-next-open-message)
@@ -1454,6 +1455,20 @@ buffer."
   (interactive)
   (notmuch-show-tag-thread-internal "-" "inbox" nil))
 
+(defun notmuch-show-delete-thread ()
+  "Delete each message in thread, then show next thread from search.
+
+Delete each message currently shown by adding the \"deleted\"
+tag to each. Then kill this buffer and show the next thread
+from the search from which this thread was originally shown.
+
+Note: This command is safe from any race condition of new messages
+being delivered to the same thread. It does not archive the
+entire thread, but only the messages shown in the current
+buffer."
+  (interactive)
+  (notmuch-show-tag-thread-internal "+" "deleted" t))
+
 (defun notmuch-show-stash-cc ()
   "Copy CC field of current message to kill-ring."
   (interactive)
diff --git a/emacs/notmuch.el b/emacs/notmuch.el
index c519687..2b860f4 100644
--- a/emacs/notmuch.el
+++ b/emacs/notmuch.el
@@ -218,6 +218,7 @@ For a mouse binding, return nil."
     (define-key map [mouse-1] 'notmuch-search-show-thread)
     (define-key map "*" 'notmuch-search-operate-all)
     (define-key map "a" 'notmuch-search-archive-thread)
+    (define-key map "d" 'notmuch-search-delete-thread)
     (define-key map "-" 'notmuch-search-remove-tag)
     (define-key map "+" 'notmuch-search-add-tag)
     (define-key map (kbd "RET") 'notmuch-search-show-thread)
@@ -605,6 +606,17 @@ This function advances the next thread when finished."
   (notmuch-search-remove-tag-thread "inbox")
   (forward-line))
 
+(defun notmuch-search-delete-thread ()
+  "Delete the currently selected thread (add the \"deleted\" tag).
+
+This function advances the next thread when finished.
+
+If notmuch-search-exclude-deleted is set, \"deleted\" messages
+will be excluded from searches."
+  (interactive)
+  (notmuch-search-add-tag "deleted")
+  (forward-line))
+
 (defvar notmuch-search-process-filter-data nil
   "Data that has not yet been processed.")
 (make-variable-buffer-local 'notmuch-search-process-filter-data)
-- 
1.7.7.3

[PATCH 4/4] emacs: modify help message for notmuch-search-line-faces to reflect preferred "deleted" tag name.

[Jameson Graef Rollins]

No functional change here.  The help message previously referred to
the "delete" tag, but "deleted" is now preferred, so hopefully this
will reduce any potential confusion.
---
 emacs/notmuch.el |    4 ++--
 1 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/emacs/notmuch.el b/emacs/notmuch.el
index 2b860f4..ebdc7d1 100644
--- a/emacs/notmuch.el
+++ b/emacs/notmuch.el
@@ -658,12 +658,12 @@ will be excluded from searches."
 Here is an example of how to color search results based on tags.
  (the following text would be placed in your ~/.emacs file):
 
- (setq notmuch-search-line-faces '((\"delete\" . (:foreground \"red\"
+ (setq notmuch-search-line-faces '((\"deleted\" . (:foreground \"red\"
 						  :background \"blue\"))
                                    (\"unread\" . (:foreground \"green\"))))
 
 The attributes defined for matching tags are merged, with later
-attributes overriding earlier. A message having both \"delete\"
+attributes overriding earlier. A message having both \"deleted\"
 and \"unread\" tags with the above settings would have a green
 foreground and blue background."
   :type '(alist :key-type (string) :value-type (custom-face-edit))
-- 
1.7.7.3

change to default archive/delete key bindings

[Jameson Graef Rollins]

While working on the delete message handling patches, I was reminded
how much I really dislike the default show-mode key bindings.  Why
can't I just archive/delete the current message, without archiving the
entire thread?  It doesn't make any sense.

Here we add two new functions to archive and delete just the single
message, and then move to the next open message.  We also add an
option to the -next-open-message function so that it will pop back out
to the parent search buffer when reaching the end of the thread.  This
should make message processing flow much smoother.

Patches 1,2 and 4 can be applied even if the consensus is to not
change the default key bindings, to make it easier for users to
achieve the desired functionality without having to write their own
functions.

jamie.

[PATCH 1/4] emacs: add show-mode functions to archive/delete only current message

[Jameson Graef Rollins]

This adds two new function, notmuch-show-{archive,delete}-message,
that archive/delete the current message, and then move to the next
open one.
---
 emacs/notmuch-show.el |   24 ++++++++++++++++++++++++
 1 files changed, 24 insertions(+), 0 deletions(-)

diff --git a/emacs/notmuch-show.el b/emacs/notmuch-show.el
index e1d15f4..8bb052e 100644
--- a/emacs/notmuch-show.el
+++ b/emacs/notmuch-show.el
@@ -1436,6 +1436,18 @@ argument, hide all of the messages."
 	    (if show-next
 		(notmuch-search-show-thread)))))))
 
+(defun notmuch-show-archive-message ()
+  "Archive the current message and advance.
+
+After the last message is reached, either the buffer will be
+closed and the cursor will move to the search result if
+available, or the cursor will move to the end of the current
+thread.
+"
+    (interactive)
+    (notmuch-show-remove-tag "inbox")
+    (notmuch-show-next-open-message)))
+
 (defun notmuch-show-archive-thread ()
   "Archive each message in thread, then show next thread from search.
 
@@ -1455,6 +1467,18 @@ buffer."
   (interactive)
   (notmuch-show-tag-thread-internal "-" "inbox" nil))
 
+(defun notmuch-show-delete-message ()
+  "Delete the current message and advance.
+
+After the last message is reached, either the buffer will be
+closed and the cursor will move to the search result if
+available, or the cursor will move to the end of the current
+thread.
+"
+    (interactive)
+    (notmuch-show-add-tag "deleted")
+    (notmuch-show-next-open-message)))
+
 (defun notmuch-show-delete-thread ()
   "Delete each message in thread, then show next thread from search.
 
-- 
1.7.7.3

[PATCH 2/4] emacs: add option to notmuch-show-next-open-message to pop out to parent buffer if at end

[Jameson Graef Rollins]

This will allow for keybindings that achieve a smoother message
processing flow by reducing the number of key presses needed for most
common operations.
---
 emacs/notmuch-show.el |   12 +++++++++---
 1 files changed, 9 insertions(+), 3 deletions(-)

diff --git a/emacs/notmuch-show.el b/emacs/notmuch-show.el
index 8bb052e..e7bb958 100644
--- a/emacs/notmuch-show.el
+++ b/emacs/notmuch-show.el
@@ -1264,17 +1264,23 @@ any effects from previous calls to
   (notmuch-show-mark-read)
   (notmuch-show-message-adjust))
 
-(defun notmuch-show-next-open-message ()
+(defun notmuch-show-next-open-message (&optional pop-at-end)
   "Show the next message."
   (interactive)
-  (let (r)
+  (let ((r)
+	(parent-buffer notmuch-show-parent-buffer))
     (while (and (setq r (notmuch-show-goto-message-next))
 		(not (notmuch-show-message-visible-p))))
     (if r
 	(progn
 	  (notmuch-show-mark-read)
 	  (notmuch-show-message-adjust))
-      (goto-char (point-max)))))
+      (if (and parent-buffer pop-at-end)
+	  (progn
+	    (kill-this-buffer)
+	    (switch-to-buffer parent-buffer)
+	    (forward-line 1))
+	(goto-char (point-max))))))
 
 (defun notmuch-show-previous-open-message ()
   "Show the previous message."
-- 
1.7.7.3

[PATCH 3/4] emacs: modify the default show-mode key bindings for archiving/deleting

[Jameson Graef Rollins]

This changes the default key bindings for the 'a' and 'd' keys in
notmuch-show mode.  Instead of archiving/deleting the entire thread,
they now just archive/delete the current message, and then advance to
the next open message.

'A' and 'D' are rebound to the previous archive/delete-thread
functions.
---
 emacs/notmuch-show.el |    6 ++++--
 1 files changed, 4 insertions(+), 2 deletions(-)

diff --git a/emacs/notmuch-show.el b/emacs/notmuch-show.el
index e7bb958..4c2b507 100644
--- a/emacs/notmuch-show.el
+++ b/emacs/notmuch-show.el
@@ -943,8 +943,10 @@ thread id.  If a prefix is given, crypto processing is toggled."
 	(define-key map "-" 'notmuch-show-remove-tag)
 	(define-key map "+" 'notmuch-show-add-tag)
 	(define-key map "x" 'notmuch-show-archive-thread-then-exit)
-	(define-key map "a" 'notmuch-show-archive-thread)
-	(define-key map "d" 'notmuch-show-delete-thread)
+	(define-key map "a" 'notmuch-show-archive-message)
+	(define-key map "A" 'notmuch-show-archive-thread)
+	(define-key map "d" 'notmuch-show-delete-message)
+	(define-key map "D" 'notmuch-show-delete-thread)
 	(define-key map "N" 'notmuch-show-next-message)
 	(define-key map "P" 'notmuch-show-previous-message)
 	(define-key map "n" 'notmuch-show-next-open-message)
-- 
1.7.7.3

[PATCH 4/4] emacs: use pop-at-end functionality in archive/delete-message functions

[Jameson Graef Rollins]

This provides a smoother message processing flow by reducing the
number of key presses needed for these common operations.
---
 emacs/notmuch-show.el |    4 ++--
 1 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/emacs/notmuch-show.el b/emacs/notmuch-show.el
index 4c2b507..7103c23 100644
--- a/emacs/notmuch-show.el
+++ b/emacs/notmuch-show.el
@@ -1454,7 +1454,7 @@ thread.
 "
     (interactive)
     (notmuch-show-remove-tag "inbox")
-    (notmuch-show-next-open-message)))
+    (notmuch-show-next-open-message t)))
 
 (defun notmuch-show-archive-thread ()
   "Archive each message in thread, then show next thread from search.
@@ -1485,7 +1485,7 @@ thread.
 "
     (interactive)
     (notmuch-show-add-tag "deleted")
-    (notmuch-show-next-open-message)))
+    (notmuch-show-next-open-message t)))
 
 (defun notmuch-show-delete-thread ()
   "Delete each message in thread, then show next thread from search.
-- 
1.7.7.3

[PATCH 4/4 v2] emacs: use pop-at-end functionality in archive/delete-message functions

[Jameson Graef Rollins]

This provides a smoother message processing flow by reducing the
number of key presses needed for these common operations.
---
Sorry, there were some errant extra parens at the end of these
function definitions.

 emacs/notmuch-show.el |    4 ++--
 1 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/emacs/notmuch-show.el b/emacs/notmuch-show.el
index 4c2b507..a706637 100644
--- a/emacs/notmuch-show.el
+++ b/emacs/notmuch-show.el
@@ -1454,7 +1454,7 @@ thread.
 "
     (interactive)
     (notmuch-show-remove-tag "inbox")
-    (notmuch-show-next-open-message)))
+    (notmuch-show-next-open-message t))
 
 (defun notmuch-show-archive-thread ()
   "Archive each message in thread, then show next thread from search.
@@ -1485,7 +1485,7 @@ thread.
 "
     (interactive)
     (notmuch-show-add-tag "deleted")
-    (notmuch-show-next-open-message)))
+    (notmuch-show-next-open-message t))
 
 (defun notmuch-show-delete-thread ()
   "Delete each message in thread, then show next thread from search.
-- 
1.7.7.3

[PATCH 1/4 v2] emacs: add show-mode functions to archive/delete only current message

[Jameson Graef Rollins]

This adds two new function, notmuch-show-{archive,delete}-message,
that archive/delete the current message, and then move to the next
open one.
---
Sorry, there were some errant extra parens at the end of these
function definitions.

 emacs/notmuch-show.el |   24 ++++++++++++++++++++++++
 1 files changed, 24 insertions(+), 0 deletions(-)

diff --git a/emacs/notmuch-show.el b/emacs/notmuch-show.el
index e1d15f4..b2e7829 100644
--- a/emacs/notmuch-show.el
+++ b/emacs/notmuch-show.el
@@ -1436,6 +1436,18 @@ argument, hide all of the messages."
 	    (if show-next
 		(notmuch-search-show-thread)))))))
 
+(defun notmuch-show-archive-message ()
+  "Archive the current message and advance.
+
+After the last message is reached, either the buffer will be
+closed and the cursor will move to the search result if
+available, or the cursor will move to the end of the current
+thread.
+"
+    (interactive)
+    (notmuch-show-remove-tag "inbox")
+    (notmuch-show-next-open-message))
+
 (defun notmuch-show-archive-thread ()
   "Archive each message in thread, then show next thread from search.
 
@@ -1455,6 +1467,18 @@ buffer."
   (interactive)
   (notmuch-show-tag-thread-internal "-" "inbox" nil))
 
+(defun notmuch-show-delete-message ()
+  "Delete the current message and advance.
+
+After the last message is reached, either the buffer will be
+closed and the cursor will move to the search result if
+available, or the cursor will move to the end of the current
+thread.
+"
+    (interactive)
+    (notmuch-show-add-tag "deleted")
+    (notmuch-show-next-open-message))
+
 (defun notmuch-show-delete-thread ()
   "Delete each message in thread, then show next thread from search.
 
-- 
1.7.7.3

[PATCH 4/4 v3] emacs: use pop-at-end functionality in archive/delete-message functions

[Jameson Graef Rollins]

This provides a smoother message processing flow by reducing the
number of key presses needed for these common operations.
---
Sorry sorry.  I originally missed the problem in patch 1/4, so I had
to modify this patch again to apply against the new version of the
previous.  So sorry for the noise, and thanks to amdragon for pointing
all of this out.

 emacs/notmuch-show.el |    4 ++--
 1 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/emacs/notmuch-show.el b/emacs/notmuch-show.el
index b9ec584..a706637 100644
--- a/emacs/notmuch-show.el
+++ b/emacs/notmuch-show.el
@@ -1454,7 +1454,7 @@ thread.
 "
     (interactive)
     (notmuch-show-remove-tag "inbox")
-    (notmuch-show-next-open-message))
+    (notmuch-show-next-open-message t))
 
 (defun notmuch-show-archive-thread ()
   "Archive each message in thread, then show next thread from search.
@@ -1485,7 +1485,7 @@ thread.
 "
     (interactive)
     (notmuch-show-add-tag "deleted")
-    (notmuch-show-next-open-message))
+    (notmuch-show-next-open-message t))
 
 (defun notmuch-show-delete-thread ()
   "Delete each message in thread, then show next thread from search.
-- 
1.7.7.3

Re: [PATCH 2/4] emacs: repurpose notmuch-show-archive-thread-internal function for general thread tagging

[Aaron Ecay]

Jameson,

Some comments below:

On Sat,  7 Jan 2012 14:28:12 -0800, Jameson Graef Rollins <jrollins@finestructure.net> wrote:
> Instead of having a function that is only used for archiving a thread,
> we instead make it useful for any tagging operation.  The new
> function, notmuch-show-tag-thread-internal, now takes two more
> arguments, for the "sign" of the tagging operation ("-" or "+"), and
> the tag to be added or removed.  This will allow this function to be
> used for any generic thread tagging operation.
> 
> The higher level functions that call this function are modified
> accordingly.
> ---
>  emacs/notmuch-show.el |   34 ++++++++++++++++++++--------------
>  1 files changed, 20 insertions(+), 14 deletions(-)
> 
> diff --git a/emacs/notmuch-show.el b/emacs/notmuch-show.el
> index 5502efd..1e16f05 100644
> --- a/emacs/notmuch-show.el
> +++ b/emacs/notmuch-show.el
> @@ -1414,20 +1414,26 @@ argument, hide all of the messages."
>    (interactive)
>    (backward-button 1))
>  
> -(defun notmuch-show-archive-thread-internal (show-next)
> +(defun notmuch-show-tag-thread-internal (sign tag show-next)

A couple of comments on the arguments:
- It would be good to make show-next &optional.  This will enable code
  to call the fn with only two arguments, and not showing next will be
  the default behavior.
- A more lispy way of specifying the sign would be to use a
  boolean.  Perhaps you could call this “remove”; a value of ‘t’ would
  remove the tag; ‘nil’ would add it.  Moving this argument after ‘tag’
  and also making it &optional woudl allow this fn to be called with one
  arg to add a tag.  (Maybe this is too minimalist and API, however.) 

>    ;; Remove the tag from the current set of messages.
>    (goto-char (point-min))
> -  (loop do (notmuch-show-remove-tag "inbox")
> -	until (not (notmuch-show-goto-message-next)))
> -  ;; Move to the next item in the search results, if any.
> -  (let ((parent-buffer notmuch-show-parent-buffer))
> -    (notmuch-kill-this-buffer)
> -    (if parent-buffer
> -	(progn
> -	  (switch-to-buffer parent-buffer)
> -	  (forward-line)
> -	  (if show-next
> -	      (notmuch-search-show-thread))))))
> +  (let ((tag-function))

No second set of parens is needed around tag-function.

> +    (cond
> +     ((string= sign "-")
> +      (setq tag-function 'notmuch-show-remove-tag))
> +     ((string= sign "+")
> +      (setq tag-function 'notmuch-show-add-tag)))
> +    (loop do (funcall tag-function tag)
> +	  until (not (notmuch-show-goto-message-next)))
> +    ;; Move to the next item in the search results, if any.

Does it make sense to separate the tagging and the movement?  It seems
plausible that some code somewhere might want to add/remove a tag from
all messages in the thread w/o changing the display.

> +    (let ((parent-buffer notmuch-show-parent-buffer))
> +      (notmuch-kill-this-buffer)
> +      (if parent-buffer
> +	  (progn
> +	    (switch-to-buffer parent-buffer)
> +	    (forward-line)
> +	    (if show-next
> +		(notmuch-search-show-thread)))))))

-- 
Aaron Ecay

Re: [PATCH 2/4] emacs: add option to notmuch-show-next-open-message to pop out to parent buffer if at end

[Aaron Ecay]

Jameson,

One small, stylistic/nitpicky comment :)

On Sat,  7 Jan 2012 17:26:53 -0800, Jameson Graef Rollins <jrollins@finestructure.net> wrote:
> This will allow for keybindings that achieve a smoother message
> processing flow by reducing the number of key presses needed for most
> common operations.
> ---
>  emacs/notmuch-show.el |   12 +++++++++---
>  1 files changed, 9 insertions(+), 3 deletions(-)
> 
> diff --git a/emacs/notmuch-show.el b/emacs/notmuch-show.el
> index 8bb052e..e7bb958 100644
> --- a/emacs/notmuch-show.el
> +++ b/emacs/notmuch-show.el
> @@ -1264,17 +1264,23 @@ any effects from previous calls to
>    (notmuch-show-mark-read)
>    (notmuch-show-message-adjust))
>  
> -(defun notmuch-show-next-open-message ()
> +(defun notmuch-show-next-open-message (&optional pop-at-end)
>    "Show the next message."
>    (interactive)
> -  (let (r)
> +  (let ((r)
> +	(parent-buffer notmuch-show-parent-buffer))

No second set of parentheses is needed around r.  Also, it is more
idiomatic to put the initialized variable (i.e. parent-buffer) before
the uninitialized one (r).

>      (while (and (setq r (notmuch-show-goto-message-next))
>  		(not (notmuch-show-message-visible-p))))
>      (if r
>  	(progn
>  	  (notmuch-show-mark-read)
>  	  (notmuch-show-message-adjust))
> -      (goto-char (point-max)))))
> +      (if (and parent-buffer pop-at-end)
> +	  (progn
> +	    (kill-this-buffer)
> +	    (switch-to-buffer parent-buffer)
> +	    (forward-line 1))
> +	(goto-char (point-max))))))
>  
>  (defun notmuch-show-previous-open-message ()
>    "Show the previous message."
> -- 
> 1.7.7.3
> 
> _______________________________________________
> notmuch mailing list
> notmuch@notmuchmail.org
> http://notmuchmail.org/mailman/listinfo/notmuch

-- 
Aaron Ecay

Re: [PATCH 1/4] emacs: new customization variable to exclude "deleted" messages from search

[Aaron Ecay]

Jameson,

One comment

On Sat,  7 Jan 2012 14:28:11 -0800, Jameson Graef Rollins <jrollins@finestructure.net> wrote:
> The new customization variable, notmuch-search-exclude-deleted, when
> set to t, will exclude any messages with the "deleted" tag from
> searches.
> 
> Additionally, specifying "tag:deleted" in the search directly will
> override the exclusion and will included deleted messages in the
> search results.
> ---
>  emacs/notmuch.el                                   |    8 ++++
>  test/emacs                                         |   42 ++++++++++++++++++++
>  .../notmuch-search-tag-inbox-deleted-excluded      |   24 +++++++++++
>  3 files changed, 74 insertions(+), 0 deletions(-)
>  create mode 100644 test/emacs.expected-output/notmuch-search-tag-inbox-deleted-excluded
> 
> diff --git a/emacs/notmuch.el b/emacs/notmuch.el
> index fde2377..c519687 100644
> --- a/emacs/notmuch.el
> +++ b/emacs/notmuch.el
> @@ -905,6 +905,11 @@ PROMPT is the string to prompt with."
>        (read-from-minibuffer prompt nil keymap nil
>  			    'notmuch-query-history nil nil))))
>  
> +(defcustom notmuch-search-exclude-deleted nil
> +  "Exclude deleted messages (with \"deleted\" tag) from search results."
> +  :group 'notmuch
> +  :type 'boolean)
> +
>  ;;;###autoload
>  (defun notmuch-search (query &optional oldest-first target-thread target-line continuation)
>    "Run \"notmuch search\" with the given query string and display results.
> @@ -927,6 +932,9 @@ The optional parameters are used as follows:
>      (set 'notmuch-search-target-thread target-thread)
>      (set 'notmuch-search-target-line target-line)
>      (set 'notmuch-search-continuation continuation)
> +    (when (and notmuch-search-exclude-deleted
> +	       (not (string-match "tag:deleted[ )]*" query)))

“is:” is a synonym for “tag:” in searches – so this section of the code
should look for it too.

-- 
Aaron Ecay

Re: [PATCH 1/4] emacs: new customization variable to exclude "deleted" messages from search

[Austin Clements]

> > @@ -927,6 +932,9 @@ The optional parameters are used as follows:
> >      (set 'notmuch-search-target-thread target-thread)
> >      (set 'notmuch-search-target-line target-line)
> >      (set 'notmuch-search-continuation continuation)
> > +    (when (and notmuch-search-exclude-deleted
> > +	       (not (string-match "tag:deleted[ )]*" query)))
> 
> “is:” is a synonym for “tag:” in searches – so this section of the code
> should look for it too.

There are several other things that could also trip up this regexp.
xtag:deletedx would be falsely matched, as would a quoted phrase
containing "tag:deleted", while tag:"deleted" and tag:(deleted) would
incorrectly not be matched.  Getting this right is hard, though I'd be
happy with

  "\\<\\(tag\\|is\\):deleted\\>"

or maybe

  "\\<\\(tag\\|is\\):\\(\"?\\)deleted\\>\\2"

Implicit exclusions like this were actually one of my target features
for the custom query parser, but I think hacking around that by
inspecting the query string is a fine interim solution.  (One of these
months I'll dust off the query parser, really!)

Re: [PATCH 1/4] emacs: new customization variable to exclude "deleted" messages from search

[Jameson Graef Rollins]

[-- Attachment #1: Type: text/plain, Size: 2163 bytes --]

On Sun, 8 Jan 2012 20:49:38 -0500, Austin Clements <amdragon@MIT.EDU> wrote:
> > > @@ -927,6 +932,9 @@ The optional parameters are used as follows:
> > >      (set 'notmuch-search-target-thread target-thread)
> > >      (set 'notmuch-search-target-line target-line)
> > >      (set 'notmuch-search-continuation continuation)
> > > +    (when (and notmuch-search-exclude-deleted
> > > +	       (not (string-match "tag:deleted[ )]*" query)))
> > 
> > “is:” is a synonym for “tag:” in searches – so this section of the code
> > should look for it too.
> 
> There are several other things that could also trip up this regexp.
> xtag:deletedx would be falsely matched, as would a quoted phrase
> containing "tag:deleted", while tag:"deleted" and tag:(deleted) would
> incorrectly not be matched.

Thanks so much for the review, guys.  I should have mentioned in this
patch that the my regex skills are very weak, and that it was surely
incomplete.  I always forget about the is: prefix as well.

> Getting this right is hard, though I'd be happy with
> 
>   "\\<\\(tag\\|is\\):deleted\\>"

Every time I think I start to understand regex I am reminded that it's
black magic and I really know nothing.  For instance, I am not familiar
with "<" or ">", although they appear to be a "word" boundaries
(although I'm not sure how "word" is defined).  Also, why is all the \\
(double?)  escaping needed?  I'll certainly take your word for it,
though.

> or maybe
> 
>   "\\<\\(tag\\|is\\):\\(\"?\\)deleted\\>\\2"

After staring at this for 10 minutes I think I'm getting the extra bits
here.  It matches an initial \", and then a second at the end if the
first matched.  That's clever.  Why 

  \\>\\2

instead of

 \\2\\>

?

I'm definitely confused by why so much apparent escaping is needed,
though.

> Implicit exclusions like this were actually one of my target features
> for the custom query parser, but I think hacking around that by
> inspecting the query string is a fine interim solution.  (One of these
> months I'll dust off the query parser, really!)

Very much looking forward to it!

jamie.

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: [PATCH 1/4] emacs: new customization variable to exclude "deleted" messages from search

[Austin Clements]

Quoth Jameson Graef Rollins on Jan 08 at  6:34 pm:
> On Sun, 8 Jan 2012 20:49:38 -0500, Austin Clements <amdragon@MIT.EDU> wrote:
> > > > @@ -927,6 +932,9 @@ The optional parameters are used as follows:
> > > >      (set 'notmuch-search-target-thread target-thread)
> > > >      (set 'notmuch-search-target-line target-line)
> > > >      (set 'notmuch-search-continuation continuation)
> > > > +    (when (and notmuch-search-exclude-deleted
> > > > +	       (not (string-match "tag:deleted[ )]*" query)))
> > > 
> > > “is:” is a synonym for “tag:” in searches – so this section of the code
> > > should look for it too.
> > 
> > There are several other things that could also trip up this regexp.
> > xtag:deletedx would be falsely matched, as would a quoted phrase
> > containing "tag:deleted", while tag:"deleted" and tag:(deleted) would
> > incorrectly not be matched.
> 
> Thanks so much for the review, guys.  I should have mentioned in this
> patch that the my regex skills are very weak, and that it was surely
> incomplete.  I always forget about the is: prefix as well.
> 
> > Getting this right is hard, though I'd be happy with
> > 
> >   "\\<\\(tag\\|is\\):deleted\\>"
> 
> Every time I think I start to understand regex I am reminded that it's
> black magic and I really know nothing.  For instance, I am not familiar
> with "<" or ">", although they appear to be a "word" boundaries
> (although I'm not sure how "word" is defined).  Also, why is all the \\
> (double?)  escaping needed?  I'll certainly take your word for it,
> though.

I'm not positive, but I think \> matches on the transition from a
"word-constituent" character to a non-word-constituent character, as
defined by Emacs' active syntax table.

The slashes are all doubled because I was writing it as an Emacs
string for easy pasting (sorry, I should have been explicit about
that).  The regexp itself is

  \<\(tag\|is\):deleted\>

> > or maybe
> > 
> >   "\\<\\(tag\\|is\\):\\(\"?\\)deleted\\>\\2"
> 
> After staring at this for 10 minutes I think I'm getting the extra bits
> here.  It matches an initial \", and then a second at the end if the
> first matched.  That's clever.  Why 

Exactly.

>   \\>\\2
> 
> instead of
> 
>  \\2\\>
> 
> ?

Okay, that can qualify as black magic.  The problem is that a " will
mess up the word-boundary matching because " isn't a word constituent
character.  So, if it is looking for a quote at the end, the \2 in
\2\> would match and consume the ", but then the \> wouldn't match.

Re: [PATCH 2/4] emacs: repurpose notmuch-show-archive-thread-internal function for general thread tagging

[Jameson Graef Rollins]

[-- Attachment #1: Type: text/plain, Size: 1466 bytes --]

Thanks so much for the review, Aaron.

On Sun, 08 Jan 2012 20:08:59 -0500, Aaron Ecay <aaronecay@gmail.com> wrote:
> A couple of comments on the arguments:
> - It would be good to make show-next &optional.  This will enable code
>   to call the fn with only two arguments, and not showing next will be
>   the default behavior.

That's a nice idea.  Probably better for a separate patch, though.

> - A more lispy way of specifying the sign would be to use a
>   boolean.  Perhaps you could call this “remove”; a value of ‘t’ would
>   remove the tag; ‘nil’ would add it.  Moving this argument after ‘tag’
>   and also making it &optional woudl allow this fn to be called with one
>   arg to add a tag.  (Maybe this is too minimalist and API, however.) 

That might be more lispy, but it seems a lot less clear to me.  It might
save a few keystrokes when coding, but it would definitely make the code
a lot harder to read ("remove" to add a tag?).  I think I would prefer
people to give the sign explicitly.

> No second set of parens is needed around tag-function.

Yeah, I've seen this either way.  I guess it's just a stylistic choice.

I think it might make sense, but again I think that's out of the scope
of this patch series.  The point was to make a minimal set of
modifications here.  If we want to separate out the functionality, we
should do that in a separate patch.

Thanks again for the review.

jamie.

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: [PATCH 1/4] emacs: new customization variable to exclude "deleted" messages from search

[Austin Clements]

Quoth myself on Jan 08 at  8:49 pm:
> > > @@ -927,6 +932,9 @@ The optional parameters are used as follows:
> > >      (set 'notmuch-search-target-thread target-thread)
> > >      (set 'notmuch-search-target-line target-line)
> > >      (set 'notmuch-search-continuation continuation)
> > > +    (when (and notmuch-search-exclude-deleted
> > > +	       (not (string-match "tag:deleted[ )]*" query)))
> > 
> > “is:” is a synonym for “tag:” in searches – so this section of the code
> > should look for it too.
> 
> There are several other things that could also trip up this regexp.
> xtag:deletedx would be falsely matched, as would a quoted phrase
> containing "tag:deleted", while tag:"deleted" and tag:(deleted) would
> incorrectly not be matched.  Getting this right is hard, though I'd be
> happy with
> 
>   "\\<\\(tag\\|is\\):deleted\\>"
> 
> or maybe
> 
>   "\\<\\(tag\\|is\\):\\(\"?\\)deleted\\>\\2"

For the record, here's a More Correct (TM) version

  "\\(^\\|[-+ ()]\\)\\(tag\\|is\\):\\(\"?\\)deleted\\3\\($\\|[ ()]\\)"

However, as we discussed on IRC, it's probably better to fix this in
the CLI/library by adding a config option for auto-excluded tags, an
API to register these with the library (probably part of the query
API), and to iterate over the terms in the parsed query to determine
which tag exclusions should be automatically added.

Re: [PATCH 2/4] emacs: repurpose notmuch-show-archive-thread-internal function for general thread tagging

[Aaron Ecay]

On Sun, 08 Jan 2012 18:49:56 -0800, Jameson Graef Rollins <jrollins@finestructure.net> wrote:
> Thanks so much for the review, Aaron.
> 
> On Sun, 08 Jan 2012 20:08:59 -0500, Aaron Ecay <aaronecay@gmail.com> wrote:
> > A couple of comments on the arguments:
> > - It would be good to make show-next &optional.  This will enable code
> >   to call the fn with only two arguments, and not showing next will be
> >   the default behavior.
> 
> That's a nice idea.  Probably better for a separate patch, though.

This patch introduces show-next as a new argument to the function.  So it
can and should make it &optional, if that is the appropriate semantics
for it to have.

> 
> > - A more lispy way of specifying the sign would be to use a
> >   boolean.  Perhaps you could call this “remove”; a value of ‘t’ would
> >   remove the tag; ‘nil’ would add it.  Moving this argument after ‘tag’
> >   and also making it &optional woudl allow this fn to be called with one
> >   arg to add a tag.  (Maybe this is too minimalist and API, however.) 
> 
> That might be more lispy, but it seems a lot less clear to me.  It might
> save a few keystrokes when coding, but it would definitely make the code
> a lot harder to read ("remove" to add a tag?).  I think I would prefer
> people to give the sign explicitly.

Using a string value makes it harder to interface with other code.  For
example, the prefix argument (C-u) is delivered to emacs commands as a
boolean value (nil if no arg, something truthy if the arg is given).  A
plausible custom end user function/keybinding would be one to add a tag
to the open messages, or remove that tag if the prefix arg is given to
the same command.  (So that ‘d’ deletes and ‘C-u d’ undeletes, for
example.)  In order to do that, the user’s code has to convert the
prefix arg into a string.  Making something “lispy” isn’t just about
code readability/saving keystrokes, but also refers to how well the
code interfaces with the conventions used by the rest of the emacs
environment.

That said, here’s an alternate proposal: provide two functions as the
“external” API, namely ‘notmuch-show-{add,remove}-tag-thread’ (by
parallelism with ‘notmuch-show-{add,remove}-tag’).  These could be
thin wrappers around ‘notmuch-show-tag-thread-internal’, which would
then not be intended to be called by user code.

> 
> > No second set of parens is needed around tag-function.
> 
> Yeah, I've seen this either way.  I guess it's just a stylistic
> choice.

Using double parens is semantically correct, but makes the code less
idiomatic and harder to read (IMO).  To test my intuition, I looked at
‘let’ invocations in the Emacs source that have a non-initialized
variable (because the multiple variable case is hard to grep for).
Double parens are used only 3% of the time (44 double vs 1468 single).
Make of this data what you will.

-- 
Aaron Ecay

Re: [PATCH 1/4 v2] emacs: add show-mode functions to archive/delete only current message

[David Edmondson]

[-- Attachment #1: Type: text/plain, Size: 257 bytes --]

On Sun,  8 Jan 2012 11:09:09 -0800, Jameson Graef Rollins <jrollins@finestructure.net> wrote:
> This adds two new function, notmuch-show-{archive,delete}-message,
> that archive/delete the current message, and then move to the next
> open one.

Looks fine.

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: another attempt to add delete functionality in emacs

[David Edmondson]

[-- Attachment #1: Type: text/plain, Size: 1235 bytes --]

On Sat,  7 Jan 2012 14:28:10 -0800, Jameson Graef Rollins <jrollins@finestructure.net> wrote:
> I try to address the concerns that have come up in previous attempts.
> In particular, I include a patch that creates a new customization
> variable, notmuch-search-exclude-deleted, that will exclude any
> messages with the "deleted" tag from searches.  This actually makes
> "deleted" messages appear effectively deleted, which is one of the
> things cworth wanted to see, and one of the reasons he kept pushing
> back on previous attempts at this functionality.

I honestly don't understand the reason for this. If someone wants to not
see messages that they have tagged as 'deleted', they add 'and not
tag:deleted' to the end of the search expression.

The messages aren't actually deleted because they have the 'deleted'
tag, so why would they not be shown?

If it's really intended that those messages not be shown then perhaps
the tag should be called 'not-shown' or something.

> Also, no tags other than "deleted" are modified.  All tags should be
> orthogonal, and should be handled so.

+1

The name 'deleted' for the tag should be a configuration option, as a
non-English speaker (for example) might want to use something else.

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: change to default archive/delete key bindings

[David Edmondson]

[-- Attachment #1: Type: text/plain, Size: 793 bytes --]

On Sat,  7 Jan 2012 17:26:51 -0800, Jameson Graef Rollins <jrollins@finestructure.net> wrote:
> While working on the delete message handling patches, I was reminded
> how much I really dislike the default show-mode key bindings.  Why
> can't I just archive/delete the current message, without archiving the
> entire thread?  It doesn't make any sense.
> 
> Here we add two new functions to archive and delete just the single
> message, and then move to the next open message.  We also add an
> option to the -next-open-message function so that it will pop back out
> to the parent search buffer when reaching the end of the thread.  This
> should make message processing flow much smoother.

This seems good to me, though it will confuse me for a while when first
introduced :-/

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: another attempt to add delete functionality in emacs

[David Bremner]

On Tue, 10 Jan 2012 07:47:23 +0000, David Edmondson <dme@dme.org> wrote:
> On Sat,  7 Jan 2012 14:28:10 -0800, Jameson Graef Rollins <jrollins@finestructure.net> wrote:
> > I try to address the concerns that have come up in previous attempts.
> > In particular, I include a patch that creates a new customization
> > variable, notmuch-search-exclude-deleted, that will exclude any
> > messages with the "deleted" tag from searches.  

[snip]

> I honestly don't understand the reason for this. If someone wants to not
> see messages that they have tagged as 'deleted', they add 'and not
> tag:deleted' to the end of the search expression.

Just thinking out loud here, but it does seem a bit unfortunate to me
that it represents a pretty fundamental divergence between the CLI and
the emacs interface. Mind you, I guess one could make the same argument
about the libs versus the CLI. Lack of configuration information in the
library (possibly among other reasons) makes this not too nice to
support in the current library either.

d

Re: another attempt to add delete functionality in emacs

[Jameson Graef Rollins]

[-- Attachment #1: Type: text/plain, Size: 514 bytes --]

On Tue, 10 Jan 2012 07:47:23 +0000, David Edmondson <dme@dme.org> wrote:
> I honestly don't understand the reason for this. If someone wants to not
> see messages that they have tagged as 'deleted', they add 'and not
> tag:deleted' to the end of the search expression.

Adding "and not tag:deleted" in all of your searches vs. having it done
automatically?  That a pretty big usability difference to me.

The config variable is included so that people can turn the
functionality on or off as they see fit.

jamie.

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: [PATCH 2/4] emacs: repurpose notmuch-show-archive-thread-internal function for general thread tagging

[Jameson Graef Rollins]

[-- Attachment #1: Type: text/plain, Size: 1376 bytes --]

On Mon, 09 Jan 2012 00:02:20 -0500, Aaron Ecay <aaronecay@gmail.com> wrote:
> On Sun, 08 Jan 2012 18:49:56 -0800, Jameson Graef Rollins <jrollins@finestructure.net> wrote:
> > On Sun, 08 Jan 2012 20:08:59 -0500, Aaron Ecay <aaronecay@gmail.com> wrote:
> > >
> > > - It would be good to make show-next &optional.  This will enable code
> > >   to call the fn with only two arguments, and not showing next will be
> > >   the default behavior.
> > 
> > That's a nice idea.  Probably better for a separate patch, though.
> 
> This patch introduces show-next as a new argument to the function.  So it
> can and should make it &optional, if that is the appropriate semantics
> for it to have.

Actually, the show-next argument was already part of the function.  I
did not introduce it.  And it wasn't optional originally, so if we want
to change that behavior we should probably do so in a separate patch.

> That said, here’s an alternate proposal: provide two functions as the
> “external” API, namely ‘notmuch-show-{add,remove}-tag-thread’ (by
> parallelism with ‘notmuch-show-{add,remove}-tag’).  These could be
> thin wrappers around ‘notmuch-show-tag-thread-internal’, which would
> then not be intended to be called by user code.

I think that's a better idea.  In the next version I'll add something
like this instead.

jamie.

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: another attempt to add delete functionality in emacs

[Jameson Graef Rollins]

[-- Attachment #1: Type: text/plain, Size: 1782 bytes --]

On Tue, 10 Jan 2012 16:01:32 -0400, David Bremner <david@tethera.net> wrote:
> Just thinking out loud here, but it does seem a bit unfortunate to me
> that it represents a pretty fundamental divergence between the CLI and
> the emacs interface. Mind you, I guess one could make the same argument
> about the libs versus the CLI. Lack of configuration information in the
> library (possibly among other reasons) makes this not too nice to
> support in the current library either.

I think a consensus has formed that this functionality (automatically
suppressing messages with certain tags from searches) is better left to
the CLI, rather than implementing it just in the emacs UI.
Unfortunately I'm not going to get to that any time soon.

However, without that functionality, I really see no reason why we
should be adding any built-in support for adding "deleted" tags in the
emacs UI.  Without the CLI change, "deleted" tags aren't handled any
differently than any other tag, so why should the default emacs UI care.
If users want to bind keys to special tagging operations, they can do so
for themselves [0].

In fact, I'm now starting to think we don't need to add any support for
special tagging operations (such as "deleted") even if we *do* have
support for suppressing them in the CLI.  Special tagging operations
should only be supported if the tagged messages are somehow handled
specially.  If not, then again, we should just leave them up to the user
[0].

All that said, I think I'll just resubmit the couple small changes to
the emacs UI that I think we should consider adopting anyway.

jamie.


[0]
(define-key notmuch-show-mode-map "d"
  (lambda ()
    "Delete message"
    (interactive)
    (notmuch-show-add-tag "deleted")
    (notmuch-show-next-open-message)))

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

[PATCH 0/3] Automatic tag-based exclusion

[Austin Clements]

This implements essentially the same idea as Jamie's patch, but does
it in the library/CLI and operates on the parsed query rather than
using an ad hoc regexp.  Which tags are automatically excluded is set
in the config file and defaults to "deleted" and "spam".

My comment for the config file is clunky.  I'd love suggestions for
improvements.

The first patch can be applied without the other two.

[PATCH 1/3] count: Convert to new-style argument parsing

[Austin Clements]

---
 notmuch-count.c |   53 +++++++++++++++++++++++++----------------------------
 1 files changed, 25 insertions(+), 28 deletions(-)

diff --git a/notmuch-count.c b/notmuch-count.c
index 20ce334..fb7401b 100644
--- a/notmuch-count.c
+++ b/notmuch-count.c
@@ -21,6 +21,11 @@
 
 #include "notmuch-client.h"
 
+typedef enum {
+    OUTPUT_THREADS,
+    OUTPUT_MESSAGES,
+} output_t;
+
 int
 notmuch_count_command (void *ctx, int argc, char *argv[])
 {
@@ -28,35 +33,23 @@ notmuch_count_command (void *ctx, int argc, char *argv[])
     notmuch_database_t *notmuch;
     notmuch_query_t *query;
     char *query_str;
-    int i;
-    notmuch_bool_t output_messages = TRUE;
+    int opt_index;
+    output_t output = OUTPUT_MESSAGES;
 
-    argc--; argv++; /* skip subcommand argument */
+    notmuch_opt_desc_t options[] = {
+	{ NOTMUCH_OPT_KEYWORD, &output, "output", 'o',
+	  (notmuch_keyword_t []){ { "threads", OUTPUT_THREADS },
+				  { "messages", OUTPUT_MESSAGES },
+				  { 0, 0 } } },
+	{ 0, 0, 0, 0, 0 }
+    };
 
-    for (i = 0; i < argc && argv[i][0] == '-'; i++) {
-	if (strcmp (argv[i], "--") == 0) {
-	    i++;
-	    break;
-	}
-	if (STRNCMP_LITERAL (argv[i], "--output=") == 0) {
-	    const char *opt = argv[i] + sizeof ("--output=") - 1;
-	    if (strcmp (opt, "threads") == 0) {
-		output_messages = FALSE;
-	    } else if (strcmp (opt, "messages") == 0) {
-		output_messages = TRUE;
-	    } else {
-		fprintf (stderr, "Invalid value for --output: %s\n", opt);
-		return 1;
-	    }
-	} else {
-	    fprintf (stderr, "Unrecognized option: %s\n", argv[i]);
-	    return 1;
-	}
+    opt_index = parse_arguments (argc, argv, options, 1);
+
+    if (opt_index < 0) {
+	return 1;
     }
 
-    argc -= i;
-    argv += i;
-
     config = notmuch_config_open (ctx, NULL, NULL);
     if (config == NULL)
 	return 1;
@@ -66,7 +59,7 @@ notmuch_count_command (void *ctx, int argc, char *argv[])
     if (notmuch == NULL)
 	return 1;
 
-    query_str = query_string_from_args (ctx, argc, argv);
+    query_str = query_string_from_args (ctx, argc-opt_index, argv+opt_index);
     if (query_str == NULL) {
 	fprintf (stderr, "Out of memory.\n");
 	return 1;
@@ -82,10 +75,14 @@ notmuch_count_command (void *ctx, int argc, char *argv[])
 	return 1;
     }
 
-    if (output_messages)
+    switch (output) {
+    case OUTPUT_MESSAGES:
 	printf ("%u\n", notmuch_query_count_messages (query));
-    else
+	break;
+    case OUTPUT_THREADS:
 	printf ("%u\n", notmuch_query_count_threads (query));
+	break;
+    }
 
     notmuch_query_destroy (query);
     notmuch_database_close (notmuch);
-- 
1.7.7.3

[PATCH 2/3] lib: Add support for automatically excluding tags from queries

[Austin Clements]

This is useful for tags like "deleted" and "spam" that people
generally want to exclude from query results.  These exclusions will
be overridden if a tag is explicitly mentioned in a query.
---
 lib/notmuch.h |    6 ++++++
 lib/query.cc  |   33 +++++++++++++++++++++++++++++++++
 2 files changed, 39 insertions(+), 0 deletions(-)

diff --git a/lib/notmuch.h b/lib/notmuch.h
index 9f23a10..0a3ae2b 100644
--- a/lib/notmuch.h
+++ b/lib/notmuch.h
@@ -457,6 +457,12 @@ notmuch_query_set_sort (notmuch_query_t *query, notmuch_sort_t sort);
 notmuch_sort_t
 notmuch_query_get_sort (notmuch_query_t *query);
 
+/* Add a tag that will be excluded by default from the query results.
+ * This exclusion will be overridden if this tag appears explicitly in
+ * the query. */
+void
+notmuch_query_add_tag_exclude (notmuch_query_t *query, const char *tag);
+
 /* Execute a query for threads, returning a notmuch_threads_t object
  * which can be used to iterate over the results. The returned threads
  * object is owned by the query and as such, will only be valid until
diff --git a/lib/query.cc b/lib/query.cc
index b6c0f12..716db1c 100644
--- a/lib/query.cc
+++ b/lib/query.cc
@@ -27,6 +27,7 @@ struct _notmuch_query {
     notmuch_database_t *notmuch;
     const char *query_string;
     notmuch_sort_t sort;
+    notmuch_string_list_t *exclude_terms;
 };
 
 typedef struct _notmuch_mset_messages {
@@ -76,6 +77,8 @@ notmuch_query_create (notmuch_database_t *notmuch,
 
     query->sort = NOTMUCH_SORT_NEWEST_FIRST;
 
+    query->exclude_terms = _notmuch_string_list_create (query);
+
     return query;
 }
 
@@ -97,6 +100,13 @@ notmuch_query_get_sort (notmuch_query_t *query)
     return query->sort;
 }
 
+void
+notmuch_query_add_tag_exclude (notmuch_query_t *query, const char *tag)
+{
+    char *term = talloc_asprintf (query, "%s%s", _find_prefix ("tag"), tag);
+    _notmuch_string_list_append (query->exclude_terms, term);
+}
+
 /* We end up having to call the destructors explicitly because we had
  * to use "placement new" in order to initialize C++ objects within a
  * block that we allocated with talloc. So C++ is making talloc
@@ -112,6 +122,25 @@ _notmuch_messages_destructor (notmuch_mset_messages_t *messages)
     return 0;
 }
 
+static Xapian::Query
+_notmuch_exclude_tags (notmuch_query_t *query, Xapian::Query xquery)
+{
+    Xapian::TermIterator end = xquery.get_terms_end ();
+
+    for (notmuch_string_node_t *term = query->exclude_terms->head; term;
+	 term = term->next) {
+	Xapian::TermIterator it = xquery.get_terms_begin ();
+	for (; it != end; it++) {
+	    if (*it == term->string)
+		break;
+	}
+	if (it == end)
+	    xquery = Xapian::Query (Xapian::Query::OP_AND_NOT,
+				    xquery, Xapian::Query (term->string));
+    }
+    return xquery;
+}
+
 notmuch_messages_t *
 notmuch_query_search_messages (notmuch_query_t *query)
 {
@@ -157,6 +186,8 @@ notmuch_query_search_messages (notmuch_query_t *query)
 					 mail_query, string_query);
 	}
 
+	final_query = _notmuch_exclude_tags (query, final_query);
+
 	enquire.set_weighting_scheme (Xapian::BoolWeight());
 
 	switch (query->sort) {
@@ -436,6 +467,8 @@ notmuch_query_count_messages (notmuch_query_t *query)
 					 mail_query, string_query);
 	}
 
+	final_query = _notmuch_exclude_tags (query, final_query);
+
 	enquire.set_weighting_scheme(Xapian::BoolWeight());
 	enquire.set_docid_order(Xapian::Enquire::ASCENDING);
 
-- 
1.7.7.3

[PATCH 3/3] search: Support automatic tag exclusions

[Austin Clements]

This adds a "search" section to the config file and an
"auto_tag_exclusions" setting in that section.  The search and count
commands pass tag tags from the configuration to the library.
---
 notmuch-client.h |    8 ++++++++
 notmuch-config.c |   42 ++++++++++++++++++++++++++++++++++++++++++
 notmuch-count.c  |    8 ++++++++
 notmuch-search.c |    8 ++++++++
 test/search      |   18 ++++++++++++++++++
 5 files changed, 84 insertions(+), 0 deletions(-)

diff --git a/notmuch-client.h b/notmuch-client.h
index 517c010..62ede28 100644
--- a/notmuch-client.h
+++ b/notmuch-client.h
@@ -235,6 +235,14 @@ void
 notmuch_config_set_maildir_synchronize_flags (notmuch_config_t *config,
 					      notmuch_bool_t synchronize_flags);
 
+const char **
+notmuch_config_get_auto_exclude_tags (notmuch_config_t *config, size_t *length);
+
+void
+notmuch_config_set_auto_exclude_tags (notmuch_config_t *config,
+				      const char *list[],
+				      size_t length);
+
 int
 notmuch_run_hook (const char *db_path, const char *hook);
 
diff --git a/notmuch-config.c b/notmuch-config.c
index d697138..6c3123b 100644
--- a/notmuch-config.c
+++ b/notmuch-config.c
@@ -84,6 +84,15 @@ static const char maildir_config_comment[] =
     "\tand update tags, while the \"notmuch tag\" and \"notmuch restore\"\n"
     "\tcommands will notice tag changes and update flags in filenames\n";
 
+static const char search_config_comment[] =
+    " Search configuration\n"
+    "\n"
+    " The following option is supported here:\n"
+    "\n"
+    "\tauto_exclude_tags      A ;-separated list of tags that will be\n"
+    "\t excluded from queries by default.  This can be overridden by including\n"
+    "\t these tags in a query.\n";
+
 struct _notmuch_config {
     char *filename;
     GKeyFile *key_file;
@@ -96,6 +105,8 @@ struct _notmuch_config {
     const char **new_tags;
     size_t new_tags_length;
     notmuch_bool_t maildir_synchronize_flags;
+    const char **auto_exclude_tags;
+    size_t auto_exclude_tags_length;
 };
 
 static int
@@ -221,6 +232,7 @@ notmuch_config_open (void *ctx,
     int file_had_new_group;
     int file_had_user_group;
     int file_had_maildir_group;
+    int file_had_search_group;
 
     if (is_new_ret)
 	*is_new_ret = 0;
@@ -252,6 +264,8 @@ notmuch_config_open (void *ctx,
     config->new_tags = NULL;
     config->new_tags_length = 0;
     config->maildir_synchronize_flags = TRUE;
+    config->auto_exclude_tags = NULL;
+    config->auto_exclude_tags_length = 0;
 
     if (! g_key_file_load_from_file (config->key_file,
 				     config->filename,
@@ -295,6 +309,7 @@ notmuch_config_open (void *ctx,
     file_had_new_group = g_key_file_has_group (config->key_file, "new");
     file_had_user_group = g_key_file_has_group (config->key_file, "user");
     file_had_maildir_group = g_key_file_has_group (config->key_file, "maildir");
+    file_had_search_group = g_key_file_has_group (config->key_file, "search");
 
 
     if (notmuch_config_get_database_path (config) == NULL) {
@@ -345,6 +360,11 @@ notmuch_config_open (void *ctx,
 	notmuch_config_set_new_tags (config, tags, 2);
     }
 
+    if (notmuch_config_get_auto_exclude_tags (config, &tmp) == NULL) {
+	const char *tags[] = { "deleted", "spam" };
+	notmuch_config_set_auto_exclude_tags (config, tags, 2);
+    }
+
     error = NULL;
     config->maildir_synchronize_flags =
 	g_key_file_get_boolean (config->key_file,
@@ -387,6 +407,11 @@ notmuch_config_open (void *ctx,
 				maildir_config_comment, NULL);
     }
 
+    if (! file_had_search_group) {
+	g_key_file_set_comment (config->key_file, "search", NULL,
+				search_config_comment, NULL);
+    }
+
     if (is_new_ret)
 	*is_new_ret = is_new;
 
@@ -597,6 +622,23 @@ notmuch_config_set_new_tags (notmuch_config_t *config,
 		     &(config->new_tags));
 }
 
+const char **
+notmuch_config_get_auto_exclude_tags (notmuch_config_t *config, size_t *length)
+{
+    return _config_get_list (config, "search", "auto_exclude_tags",
+			     &(config->auto_exclude_tags),
+			     &(config->auto_exclude_tags_length), length);
+}
+
+void
+notmuch_config_set_auto_exclude_tags (notmuch_config_t *config,
+				      const char *list[],
+				      size_t length)
+{
+    _config_set_list (config, "search", "auto_exclude_tags", list, length,
+		      &(config->auto_exclude_tags));
+}
+
 /* Given a configuration item of the form <group>.<key> return the
  * component group and key. If any error occurs, print a message on
  * stderr and return 1. Otherwise, return 0.
diff --git a/notmuch-count.c b/notmuch-count.c
index fb7401b..494619f 100644
--- a/notmuch-count.c
+++ b/notmuch-count.c
@@ -35,6 +35,9 @@ notmuch_count_command (void *ctx, int argc, char *argv[])
     char *query_str;
     int opt_index;
     output_t output = OUTPUT_MESSAGES;
+    const char **auto_exclude_tags;
+    size_t auto_exclude_tags_length;
+    unsigned int i;
 
     notmuch_opt_desc_t options[] = {
 	{ NOTMUCH_OPT_KEYWORD, &output, "output", 'o',
@@ -75,6 +78,11 @@ notmuch_count_command (void *ctx, int argc, char *argv[])
 	return 1;
     }
 
+    auto_exclude_tags = notmuch_config_get_auto_exclude_tags
+	(config, &auto_exclude_tags_length);
+    for (i = 0; i < auto_exclude_tags_length; i++)
+	notmuch_query_add_tag_exclude (query, auto_exclude_tags[i]);
+
     switch (output) {
     case OUTPUT_MESSAGES:
 	printf ("%u\n", notmuch_query_count_messages (query));
diff --git a/notmuch-search.c b/notmuch-search.c
index 4baab56..8867aab 100644
--- a/notmuch-search.c
+++ b/notmuch-search.c
@@ -423,6 +423,9 @@ notmuch_search_command (void *ctx, int argc, char *argv[])
     output_t output = OUTPUT_SUMMARY;
     int offset = 0;
     int limit = -1; /* unlimited */
+    const char **auto_exclude_tags;
+    size_t auto_exclude_tags_length;
+    unsigned int i;
 
     enum { NOTMUCH_FORMAT_JSON, NOTMUCH_FORMAT_TEXT }
 	format_sel = NOTMUCH_FORMAT_TEXT;
@@ -490,6 +493,11 @@ notmuch_search_command (void *ctx, int argc, char *argv[])
 
     notmuch_query_set_sort (query, sort);
 
+    auto_exclude_tags = notmuch_config_get_auto_exclude_tags
+	(config, &auto_exclude_tags_length);
+    for (i = 0; i < auto_exclude_tags_length; i++)
+	notmuch_query_add_tag_exclude (query, auto_exclude_tags[i]);
+
     switch (output) {
     default:
     case OUTPUT_SUMMARY:
diff --git a/test/search b/test/search
index a7a0b18..f421ae3 100755
--- a/test/search
+++ b/test/search
@@ -129,4 +129,22 @@ add_message '[subject]="utf8-message-body-subject"' '[date]="Sat, 01 Jan 2000 12
 output=$(notmuch search "bödý" | notmuch_search_sanitize)
 test_expect_equal "$output" "thread:XXX   2000-01-01 [1/1] Notmuch Test Suite; utf8-message-body-subject (inbox unread)"
 
+test_begin_subtest "Search hides deleted"
+generate_message '[subject]="Not deleted"'
+generate_message '[subject]="Deleted"'
+notmuch new > /dev/null
+notmuch tag +deleted id:$gen_msg_id
+output=$(notmuch search subject:deleted | notmuch_search_sanitize)
+test_expect_equal "$output" "thread:XXX   2001-01-05 [1/1] Notmuch Test Suite; Not deleted (inbox unread)"
+
+test_begin_subtest "Search shows deleted if requested"
+output=$(notmuch search subject:deleted and tag:deleted | notmuch_search_sanitize)
+test_expect_equal "$output" "thread:XXX   2001-01-05 [1/1] Notmuch Test Suite; Deleted (deleted inbox unread)"
+
+test_begin_subtest "Search hides deleted in threads"
+add_message '[subject]="Not deleted reply"' '[in-reply-to]="<$gen_msg_id>"'
+output=$(notmuch search subject:deleted | notmuch_search_sanitize)
+test_expect_equal "$output" "thread:XXX   2001-01-05 [1/1] Notmuch Test Suite; Not deleted (inbox unread)
+thread:XXX   2001-01-05 [1/2] Notmuch Test Suite; Not deleted reply (deleted inbox unread)"
+
 test_done
-- 
1.7.7.3

Re: another attempt to add delete functionality in emacs

[Jani Nikula]

On Tue, 10 Jan 2012 19:12:16 -0800, Jameson Graef Rollins <jrollins@finestructure.net> wrote:
> On Tue, 10 Jan 2012 16:01:32 -0400, David Bremner <david@tethera.net> wrote:
> > Just thinking out loud here, but it does seem a bit unfortunate to me
> > that it represents a pretty fundamental divergence between the CLI and
> > the emacs interface. Mind you, I guess one could make the same argument
> > about the libs versus the CLI. Lack of configuration information in the
> > library (possibly among other reasons) makes this not too nice to
> > support in the current library either.
> 
> I think a consensus has formed that this functionality (automatically
> suppressing messages with certain tags from searches) is better left to
> the CLI, rather than implementing it just in the emacs UI.
> Unfortunately I'm not going to get to that any time soon.

I could have a go at it, but I can't make any promises about getting to
that any time soon either. So what if emacs ui goes head first and does
something that should be done in the CLI in a perfect world? If it's
added properly, it can be taken out if/when this pops up in the CLI.

Also, there already *is* filtering for "all tags" list. See
notmuch-hello-tag-list-make-query. How about having something like that
for saved searches? I know it's not the same as your original, but it's
middle ground...

> However, without that functionality, I really see no reason why we
> should be adding any built-in support for adding "deleted" tags in the
> emacs UI.  Without the CLI change, "deleted" tags aren't handled any
> differently than any other tag, so why should the default emacs UI care.
> If users want to bind keys to special tagging operations, they can do so
> for themselves [0].

In fact, "deleted" used to be special, but that was, err, deleted
because it had problems: 2c262042ac174d7bc96d6035ab9c88bd0abe7f35. If
that ever gets fixed, "deleted" would be special again.


BR,
Jani.

Re: another attempt to add delete functionality in emacs

[Austin Clements]

Quoth Jani Nikula on Jan 11 at  7:16 am:
> On Tue, 10 Jan 2012 19:12:16 -0800, Jameson Graef Rollins <jrollins@finestructure.net> wrote:
> > On Tue, 10 Jan 2012 16:01:32 -0400, David Bremner <david@tethera.net> wrote:
> > > Just thinking out loud here, but it does seem a bit unfortunate to me
> > > that it represents a pretty fundamental divergence between the CLI and
> > > the emacs interface. Mind you, I guess one could make the same argument
> > > about the libs versus the CLI. Lack of configuration information in the
> > > library (possibly among other reasons) makes this not too nice to
> > > support in the current library either.
> > 
> > I think a consensus has formed that this functionality (automatically
> > suppressing messages with certain tags from searches) is better left to
> > the CLI, rather than implementing it just in the emacs UI.
> > Unfortunately I'm not going to get to that any time soon.
> 
> I could have a go at it, but I can't make any promises about getting to
> that any time soon either. So what if emacs ui goes head first and does
> something that should be done in the CLI in a perfect world? If it's
> added properly, it can be taken out if/when this pops up in the CLI.

Too late.  See id:"1326258173-21163-1-git-send-email-amdragon@mit.edu"

Re: [PATCH 2/4] emacs: repurpose notmuch-show-archive-thread-internal function for general thread tagging

[Aaron Ecay]

On Tue, 10 Jan 2012 18:56:29 -0800, Jameson Graef Rollins <jrollins@finestructure.net> wrote:
> Actually, the show-next argument was already part of the function.  I
> did not introduce it.  And it wasn't optional originally, so if we want
> to change that behavior we should probably do so in a separate patch.

Hrm.  I didn’t communicate as clearly as I could have – you are correct
that the show-next argument to the notmuch-show-archive-internal function
was pre-existing.  But notmuch-show-tag-thread-internal is a new function,
with potentially expanded usefulness to third-party code.  Thus I think
I’m in the clear to bikeshed about its calling convention.  :) It’s your
patch, though, so it’s your call if you feel that the &optional goes best
in a new change.

-- 
Aaron Ecay

Re: [PATCH 0/3] Automatic tag-based exclusion

[Jameson Graef Rollins]

[-- Attachment #1: Type: text/plain, Size: 513 bytes --]

On Wed, 11 Jan 2012 00:02:50 -0500, Austin Clements <amdragon@MIT.EDU> wrote:
> This implements essentially the same idea as Jamie's patch, but does
> it in the library/CLI and operates on the parsed query rather than
> using an ad hoc regexp.  Which tags are automatically excluded is set
> in the config file and defaults to "deleted" and "spam".

Yay Austin!  Nice fast work!  That's really great.  Don't have time to
review right now, but asap.

I'll resend my emacs patches as soon as I can as well.

jamie.

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: [PATCH 2/4] emacs: repurpose notmuch-show-archive-thread-internal function for general thread tagging

[Jameson Graef Rollins]

[-- Attachment #1: Type: text/plain, Size: 1026 bytes --]

On Wed, 11 Jan 2012 00:53:35 -0500, Aaron Ecay <aaronecay@gmail.com> wrote:
> On Tue, 10 Jan 2012 18:56:29 -0800, Jameson Graef Rollins <jrollins@finestructure.net> wrote:
> > Actually, the show-next argument was already part of the function.  I
> > did not introduce it.  And it wasn't optional originally, so if we want
> > to change that behavior we should probably do so in a separate patch.
> 
> Hrm.  I didn’t communicate as clearly as I could have – you are correct
> that the show-next argument to the notmuch-show-archive-internal function
> was pre-existing.  But notmuch-show-tag-thread-internal is a new function,
> with potentially expanded usefulness to third-party code.  Thus I think
> I’m in the clear to bikeshed about its calling convention.  :) It’s your
> patch, though, so it’s your call if you feel that the &optional goes best
> in a new change.

No, I see your point, Aaron.  I'll rework it to make it a little clean
and more flexible when I resend.  Thanks again.

jamie.

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: [PATCH 1/3] count: Convert to new-style argument parsing

[Jani Nikula]

On Wed, 11 Jan 2012 00:02:51 -0500, Austin Clements <amdragon@MIT.EDU> wrote:
> ---
>  notmuch-count.c |   53 +++++++++++++++++++++++++----------------------------
>  1 files changed, 25 insertions(+), 28 deletions(-)
> 
> diff --git a/notmuch-count.c b/notmuch-count.c
> index 20ce334..fb7401b 100644
> --- a/notmuch-count.c
> +++ b/notmuch-count.c
> @@ -21,6 +21,11 @@
>  
>  #include "notmuch-client.h"
>  
> +typedef enum {
> +    OUTPUT_THREADS,
> +    OUTPUT_MESSAGES,
> +} output_t;
> +
>  int
>  notmuch_count_command (void *ctx, int argc, char *argv[])
>  {
> @@ -28,35 +33,23 @@ notmuch_count_command (void *ctx, int argc, char *argv[])
>      notmuch_database_t *notmuch;
>      notmuch_query_t *query;
>      char *query_str;
> -    int i;
> -    notmuch_bool_t output_messages = TRUE;
> +    int opt_index;
> +    output_t output = OUTPUT_MESSAGES;
>  
> -    argc--; argv++; /* skip subcommand argument */
> +    notmuch_opt_desc_t options[] = {
> +	{ NOTMUCH_OPT_KEYWORD, &output, "output", 'o',
> +	  (notmuch_keyword_t []){ { "threads", OUTPUT_THREADS },
> +				  { "messages", OUTPUT_MESSAGES },
> +				  { 0, 0 } } },

To be pedantic, parse_arguments() expects 'output_var' to be a pointer
to int for NOTMUCH_OPT_KEYWORD. sizeof(enum) is implementation
dependent. I would forget about output_t typedef, use plain enum, and
int type for 'output'.

In fact, this is exactly what I did in commits
e6d89ad723f775952d89d4f81b6072617c5caf18 and
be5619cca32452f1a398add85908d78d6e72f469. In the latter I could have
used notmuch_bool_t because it's a typedeffed int, but then I would've
depended on some external typedef not changing (it probably won't, but
this is defensive programming).

Otherwise, looks good.

BR,
Jani.


> +	{ 0, 0, 0, 0, 0 }
> +    };
>  
> -    for (i = 0; i < argc && argv[i][0] == '-'; i++) {
> -	if (strcmp (argv[i], "--") == 0) {
> -	    i++;
> -	    break;
> -	}
> -	if (STRNCMP_LITERAL (argv[i], "--output=") == 0) {
> -	    const char *opt = argv[i] + sizeof ("--output=") - 1;
> -	    if (strcmp (opt, "threads") == 0) {
> -		output_messages = FALSE;
> -	    } else if (strcmp (opt, "messages") == 0) {
> -		output_messages = TRUE;
> -	    } else {
> -		fprintf (stderr, "Invalid value for --output: %s\n", opt);
> -		return 1;
> -	    }
> -	} else {
> -	    fprintf (stderr, "Unrecognized option: %s\n", argv[i]);
> -	    return 1;
> -	}
> +    opt_index = parse_arguments (argc, argv, options, 1);
> +
> +    if (opt_index < 0) {
> +	return 1;
>      }
>  
> -    argc -= i;
> -    argv += i;
> -
>      config = notmuch_config_open (ctx, NULL, NULL);
>      if (config == NULL)
>  	return 1;
> @@ -66,7 +59,7 @@ notmuch_count_command (void *ctx, int argc, char *argv[])
>      if (notmuch == NULL)
>  	return 1;
>  
> -    query_str = query_string_from_args (ctx, argc, argv);
> +    query_str = query_string_from_args (ctx, argc-opt_index, argv+opt_index);
>      if (query_str == NULL) {
>  	fprintf (stderr, "Out of memory.\n");
>  	return 1;
> @@ -82,10 +75,14 @@ notmuch_count_command (void *ctx, int argc, char *argv[])
>  	return 1;
>      }
>  
> -    if (output_messages)
> +    switch (output) {
> +    case OUTPUT_MESSAGES:
>  	printf ("%u\n", notmuch_query_count_messages (query));
> -    else
> +	break;
> +    case OUTPUT_THREADS:
>  	printf ("%u\n", notmuch_query_count_threads (query));
> +	break;
> +    }
>  
>      notmuch_query_destroy (query);
>      notmuch_database_close (notmuch);
> -- 
> 1.7.7.3
> 
> _______________________________________________
> notmuch mailing list
> notmuch@notmuchmail.org
> http://notmuchmail.org/mailman/listinfo/notmuch

Re: another attempt to add delete functionality in emacs

[David Edmondson]

[-- Attachment #1: Type: text/plain, Size: 727 bytes --]

On Tue, 10 Jan 2012 19:12:16 -0800, Jameson Graef Rollins <jrollins@finestructure.net> wrote:
> In fact, I'm now starting to think we don't need to add any support
> for special tagging operations (such as "deleted") even if we *do*
> have support for suppressing them in the CLI.  Special tagging
> operations should only be supported if the tagged messages are somehow
> handled specially.  If not, then again, we should just leave them up
> to the user [0].

Agreed.

The more interesting patch is the one that maps from the 'deleted' tag
to the 'T' bit, at least as someone who is mostly maildir-ignorant it
seems that way.

The keybindings are easy for a user to play with. The behaviour of the
core library much less so.

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: [PATCH 2/3] lib: Add support for automatically excluding tags from queries

[Jani Nikula]

On Wed, 11 Jan 2012 00:02:52 -0500, Austin Clements <amdragon@MIT.EDU> wrote:
> This is useful for tags like "deleted" and "spam" that people
> generally want to exclude from query results.  These exclusions will
> be overridden if a tag is explicitly mentioned in a query.
> ---
>  lib/notmuch.h |    6 ++++++
>  lib/query.cc  |   33 +++++++++++++++++++++++++++++++++
>  2 files changed, 39 insertions(+), 0 deletions(-)
> 
> diff --git a/lib/notmuch.h b/lib/notmuch.h
> index 9f23a10..0a3ae2b 100644
> --- a/lib/notmuch.h
> +++ b/lib/notmuch.h
> @@ -457,6 +457,12 @@ notmuch_query_set_sort (notmuch_query_t *query, notmuch_sort_t sort);
>  notmuch_sort_t
>  notmuch_query_get_sort (notmuch_query_t *query);
>  
> +/* Add a tag that will be excluded by default from the query results.
> + * This exclusion will be overridden if this tag appears explicitly in
> + * the query. */
> +void
> +notmuch_query_add_tag_exclude (notmuch_query_t *query, const char *tag);
> +
>  /* Execute a query for threads, returning a notmuch_threads_t object
>   * which can be used to iterate over the results. The returned threads
>   * object is owned by the query and as such, will only be valid until
> diff --git a/lib/query.cc b/lib/query.cc
> index b6c0f12..716db1c 100644
> --- a/lib/query.cc
> +++ b/lib/query.cc
> @@ -27,6 +27,7 @@ struct _notmuch_query {
>      notmuch_database_t *notmuch;
>      const char *query_string;
>      notmuch_sort_t sort;
> +    notmuch_string_list_t *exclude_terms;
>  };
>  
>  typedef struct _notmuch_mset_messages {
> @@ -76,6 +77,8 @@ notmuch_query_create (notmuch_database_t *notmuch,
>  
>      query->sort = NOTMUCH_SORT_NEWEST_FIRST;
>  
> +    query->exclude_terms = _notmuch_string_list_create (query);
> +
>      return query;
>  }
>  
> @@ -97,6 +100,13 @@ notmuch_query_get_sort (notmuch_query_t *query)
>      return query->sort;
>  }
>  
> +void
> +notmuch_query_add_tag_exclude (notmuch_query_t *query, const char *tag)
> +{
> +    char *term = talloc_asprintf (query, "%s%s", _find_prefix ("tag"), tag);
> +    _notmuch_string_list_append (query->exclude_terms, term);
> +}
> +
>  /* We end up having to call the destructors explicitly because we had
>   * to use "placement new" in order to initialize C++ objects within a
>   * block that we allocated with talloc. So C++ is making talloc
> @@ -112,6 +122,25 @@ _notmuch_messages_destructor (notmuch_mset_messages_t *messages)
>      return 0;
>  }
>  

I'd like to have a comment here, or inline in the code, explaining the
following function a little bit.

> +static Xapian::Query
> +_notmuch_exclude_tags (notmuch_query_t *query, Xapian::Query xquery)
> +{
> +    Xapian::TermIterator end = xquery.get_terms_end ();
> +
> +    for (notmuch_string_node_t *term = query->exclude_terms->head; term;
> +	 term = term->next) {
> +	Xapian::TermIterator it = xquery.get_terms_begin ();
> +	for (; it != end; it++) {
> +	    if (*it == term->string)

[This is a double reminder to me why I'm not that enthusiastic about
operator overloading in C++.]

> +		break;
> +	}
> +	if (it == end)
> +	    xquery = Xapian::Query (Xapian::Query::OP_AND_NOT,
> +				    xquery, Xapian::Query (term->string));

I briefly dug into Xapian documentation and source code, and became none
the wiser whether this copies the queries passed to it or not, i.e. does
this leak memory or not. I just presume you know what you're doing. ;)

I think the function fails if someone is stupid enough to exclude the
same tag twice. I'm not sure if you should care. If you think so, you
could just check double add in notmuch_query_add_tag_exclude().

Otherwise, looks good.

> +    }
> +    return xquery;
> +}
> +
>  notmuch_messages_t *
>  notmuch_query_search_messages (notmuch_query_t *query)
>  {
> @@ -157,6 +186,8 @@ notmuch_query_search_messages (notmuch_query_t *query)
>  					 mail_query, string_query);
>  	}
>  
> +	final_query = _notmuch_exclude_tags (query, final_query);
> +
>  	enquire.set_weighting_scheme (Xapian::BoolWeight());
>  
>  	switch (query->sort) {
> @@ -436,6 +467,8 @@ notmuch_query_count_messages (notmuch_query_t *query)
>  					 mail_query, string_query);
>  	}
>  
> +	final_query = _notmuch_exclude_tags (query, final_query);
> +
>  	enquire.set_weighting_scheme(Xapian::BoolWeight());
>  	enquire.set_docid_order(Xapian::Enquire::ASCENDING);
>  
> -- 
> 1.7.7.3
> 
> _______________________________________________
> notmuch mailing list
> notmuch@notmuchmail.org
> http://notmuchmail.org/mailman/listinfo/notmuch

Re: [PATCH 1/3] count: Convert to new-style argument parsing

[Austin Clements]

Quoth Jani Nikula on Jan 11 at  8:17 am:
> On Wed, 11 Jan 2012 00:02:51 -0500, Austin Clements <amdragon@MIT.EDU> wrote:
> > ---
> >  notmuch-count.c |   53 +++++++++++++++++++++++++----------------------------
> >  1 files changed, 25 insertions(+), 28 deletions(-)
> > 
> > diff --git a/notmuch-count.c b/notmuch-count.c
> > index 20ce334..fb7401b 100644
> > --- a/notmuch-count.c
> > +++ b/notmuch-count.c
> > @@ -21,6 +21,11 @@
> >  
> >  #include "notmuch-client.h"
> >  
> > +typedef enum {
> > +    OUTPUT_THREADS,
> > +    OUTPUT_MESSAGES,
> > +} output_t;
> > +
> >  int
> >  notmuch_count_command (void *ctx, int argc, char *argv[])
> >  {
> > @@ -28,35 +33,23 @@ notmuch_count_command (void *ctx, int argc, char *argv[])
> >      notmuch_database_t *notmuch;
> >      notmuch_query_t *query;
> >      char *query_str;
> > -    int i;
> > -    notmuch_bool_t output_messages = TRUE;
> > +    int opt_index;
> > +    output_t output = OUTPUT_MESSAGES;
> >  
> > -    argc--; argv++; /* skip subcommand argument */
> > +    notmuch_opt_desc_t options[] = {
> > +	{ NOTMUCH_OPT_KEYWORD, &output, "output", 'o',
> > +	  (notmuch_keyword_t []){ { "threads", OUTPUT_THREADS },
> > +				  { "messages", OUTPUT_MESSAGES },
> > +				  { 0, 0 } } },
> 
> To be pedantic, parse_arguments() expects 'output_var' to be a pointer
> to int for NOTMUCH_OPT_KEYWORD. sizeof(enum) is implementation
> dependent. I would forget about output_t typedef, use plain enum, and
> int type for 'output'.

Pedantic or not, that's a good point.  We should fix this in
notmuch-search.c, too (which is where I shamelessly copied this from).
Or make the argument parser more type-safe.

Re: [PATCH 1/3] count: Convert to new-style argument parsing

[Jani Nikula]

On Wed, 11 Jan 2012 08:17:05 +0000, Jani Nikula <jani@nikula.org> wrote:
> In fact, this is exactly what I did in commits
> e6d89ad723f775952d89d4f81b6072617c5caf18 and
> be5619cca32452f1a398add85908d78d6e72f469. In the latter I could have
> used notmuch_bool_t because it's a typedeffed int, but then I would've
> depended on some external typedef not changing (it probably won't, but
> this is defensive programming).

Please disregard the commit ids above. The correct references are
2d1385e141b1799b4b3942a3fe2dcbcdbf92dc38 and
id:"d32e63c8115b1f2868acf96c83b30697be88ff10.1326224339.git.jani@nikula.org".

Sorry for the confusion.

BR,
Jani.

Re: [PATCH 2/3] lib: Add support for automatically excluding tags from queries

[Austin Clements]

Quoth Jani Nikula on Jan 11 at 10:11 am:
> On Wed, 11 Jan 2012 00:02:52 -0500, Austin Clements <amdragon@MIT.EDU> wrote:
> > This is useful for tags like "deleted" and "spam" that people
> > generally want to exclude from query results.  These exclusions will
> > be overridden if a tag is explicitly mentioned in a query.
> > ---
> >  lib/notmuch.h |    6 ++++++
> >  lib/query.cc  |   33 +++++++++++++++++++++++++++++++++
> >  2 files changed, 39 insertions(+), 0 deletions(-)
> > 
> > diff --git a/lib/notmuch.h b/lib/notmuch.h
> > index 9f23a10..0a3ae2b 100644
> > --- a/lib/notmuch.h
> > +++ b/lib/notmuch.h
> > @@ -457,6 +457,12 @@ notmuch_query_set_sort (notmuch_query_t *query, notmuch_sort_t sort);
> >  notmuch_sort_t
> >  notmuch_query_get_sort (notmuch_query_t *query);
> >  
> > +/* Add a tag that will be excluded by default from the query results.
> > + * This exclusion will be overridden if this tag appears explicitly in
> > + * the query. */
> > +void
> > +notmuch_query_add_tag_exclude (notmuch_query_t *query, const char *tag);
> > +
> >  /* Execute a query for threads, returning a notmuch_threads_t object
> >   * which can be used to iterate over the results. The returned threads
> >   * object is owned by the query and as such, will only be valid until
> > diff --git a/lib/query.cc b/lib/query.cc
> > index b6c0f12..716db1c 100644
> > --- a/lib/query.cc
> > +++ b/lib/query.cc
> > @@ -27,6 +27,7 @@ struct _notmuch_query {
> >      notmuch_database_t *notmuch;
> >      const char *query_string;
> >      notmuch_sort_t sort;
> > +    notmuch_string_list_t *exclude_terms;
> >  };
> >  
> >  typedef struct _notmuch_mset_messages {
> > @@ -76,6 +77,8 @@ notmuch_query_create (notmuch_database_t *notmuch,
> >  
> >      query->sort = NOTMUCH_SORT_NEWEST_FIRST;
> >  
> > +    query->exclude_terms = _notmuch_string_list_create (query);
> > +
> >      return query;
> >  }
> >  
> > @@ -97,6 +100,13 @@ notmuch_query_get_sort (notmuch_query_t *query)
> >      return query->sort;
> >  }
> >  
> > +void
> > +notmuch_query_add_tag_exclude (notmuch_query_t *query, const char *tag)
> > +{
> > +    char *term = talloc_asprintf (query, "%s%s", _find_prefix ("tag"), tag);
> > +    _notmuch_string_list_append (query->exclude_terms, term);
> > +}
> > +
> >  /* We end up having to call the destructors explicitly because we had
> >   * to use "placement new" in order to initialize C++ objects within a
> >   * block that we allocated with talloc. So C++ is making talloc
> > @@ -112,6 +122,25 @@ _notmuch_messages_destructor (notmuch_mset_messages_t *messages)
> >      return 0;
> >  }
> >  
> 
> I'd like to have a comment here, or inline in the code, explaining the
> following function a little bit.

Done.

> > +static Xapian::Query
> > +_notmuch_exclude_tags (notmuch_query_t *query, Xapian::Query xquery)
> > +{
> > +    Xapian::TermIterator end = xquery.get_terms_end ();
> > +
> > +    for (notmuch_string_node_t *term = query->exclude_terms->head; term;
> > +	 term = term->next) {
> > +	Xapian::TermIterator it = xquery.get_terms_begin ();
> > +	for (; it != end; it++) {
> > +	    if (*it == term->string)
> 
> [This is a double reminder to me why I'm not that enthusiastic about
> operator overloading in C++.]

Actually, that's a good point.  I had originally done this to prevent
std::string from performing any internal allocation or copying (which
calling c_str could do), but since it took me some digging to figure
out if I had *actually* avoided this (turns out there's an overloaded
string==char*, so I had), I've switched to using string.compare, which
I think makes the behavior and performance more obvious.

> > +		break;
> > +	}
> > +	if (it == end)
> > +	    xquery = Xapian::Query (Xapian::Query::OP_AND_NOT,
> > +				    xquery, Xapian::Query (term->string));
> 
> I briefly dug into Xapian documentation and source code, and became none
> the wiser whether this copies the queries passed to it or not, i.e. does
> this leak memory or not. I just presume you know what you're doing. ;)

Since my code isn't doing any memory allocation, it can't leak memory.
It could do a lot of copying, but it turns out that isn't a problem
either because Xapian objects are internally reference-counted
handles.  So, in other words, if you write the obvious thing, it will
do the right thing, which is totally contrary to what C++ has
conditioned us to expect.  ]:--8)

> I think the function fails if someone is stupid enough to exclude the
> same tag twice. I'm not sure if you should care. If you think so, you
> could just check double add in notmuch_query_add_tag_exclude().

It handles this correctly for two reasons.  Suppose tag x is excluded
twice.  At worst, if tag:x doesn't appear in the query, the returned
query will get two AND NOT tag:x's, but that doesn't affect the
results.  It turns out it doesn't even get doubled up for a slightly
subtle reason: when the loop reaches the second x in the exclude list,
it will iterate over the new query, which already has the AND NOT
tag:x in it, so it will see the tag:x as if it were in the original
query and not further modify the query.

However, this did point out a bug.  I was using the end iterator from
the original query even when I started iterating over the modified
query.

I'll send out an updated series after someone looks at the third
patch.

> Otherwise, looks good.
> 
> > +    }
> > +    return xquery;
> > +}
> > +
> >  notmuch_messages_t *
> >  notmuch_query_search_messages (notmuch_query_t *query)
> >  {
> > @@ -157,6 +186,8 @@ notmuch_query_search_messages (notmuch_query_t *query)
> >  					 mail_query, string_query);
> >  	}
> >  
> > +	final_query = _notmuch_exclude_tags (query, final_query);
> > +
> >  	enquire.set_weighting_scheme (Xapian::BoolWeight());
> >  
> >  	switch (query->sort) {
> > @@ -436,6 +467,8 @@ notmuch_query_count_messages (notmuch_query_t *query)
> >  					 mail_query, string_query);
> >  	}
> >  
> > +	final_query = _notmuch_exclude_tags (query, final_query);
> > +
> >  	enquire.set_weighting_scheme(Xapian::BoolWeight());
> >  	enquire.set_docid_order(Xapian::Enquire::ASCENDING);
> >  
> 

Re: [PATCH 3/3] search: Support automatic tag exclusions

[Jani Nikula]

On Wed, 11 Jan 2012 00:02:53 -0500, Austin Clements <amdragon@MIT.EDU> wrote:
> This adds a "search" section to the config file and an
> "auto_tag_exclusions" setting in that section.  The search and count
> commands pass tag tags from the configuration to the library.

Looks good.

Perhaps a few subtests like this in test/count on a corpus with some
"deleted" or "spam" tags would be in order:

test_begin_subtest "message count with --output=messages"
test_expect_equal \
    "`notmuch search --output=messages ${SEARCH} | wc -l`" \
    "`notmuch count --output=messages ${SEARCH}`"

test_begin_subtest "thread count with --output=threads"
test_expect_equal \
    "`notmuch search --output=threads ${SEARCH} | wc -l`" \
    "`notmuch count --output=threads ${SEARCH}`"

That's copy-paste from test/count; doing the same after some exclude
tagging (and making sure it actually affects count) should be enough.


BR,
Jani.




> ---
>  notmuch-client.h |    8 ++++++++
>  notmuch-config.c |   42 ++++++++++++++++++++++++++++++++++++++++++
>  notmuch-count.c  |    8 ++++++++
>  notmuch-search.c |    8 ++++++++
>  test/search      |   18 ++++++++++++++++++
>  5 files changed, 84 insertions(+), 0 deletions(-)
> 
> diff --git a/notmuch-client.h b/notmuch-client.h
> index 517c010..62ede28 100644
> --- a/notmuch-client.h
> +++ b/notmuch-client.h
> @@ -235,6 +235,14 @@ void
>  notmuch_config_set_maildir_synchronize_flags (notmuch_config_t *config,
>  					      notmuch_bool_t synchronize_flags);
>  
> +const char **
> +notmuch_config_get_auto_exclude_tags (notmuch_config_t *config, size_t *length);
> +
> +void
> +notmuch_config_set_auto_exclude_tags (notmuch_config_t *config,
> +				      const char *list[],
> +				      size_t length);
> +
>  int
>  notmuch_run_hook (const char *db_path, const char *hook);
>  
> diff --git a/notmuch-config.c b/notmuch-config.c
> index d697138..6c3123b 100644
> --- a/notmuch-config.c
> +++ b/notmuch-config.c
> @@ -84,6 +84,15 @@ static const char maildir_config_comment[] =
>      "\tand update tags, while the \"notmuch tag\" and \"notmuch restore\"\n"
>      "\tcommands will notice tag changes and update flags in filenames\n";
>  
> +static const char search_config_comment[] =
> +    " Search configuration\n"
> +    "\n"
> +    " The following option is supported here:\n"
> +    "\n"
> +    "\tauto_exclude_tags      A ;-separated list of tags that will be\n"
> +    "\t excluded from queries by default.  This can be overridden by including\n"
> +    "\t these tags in a query.\n";
> +
>  struct _notmuch_config {
>      char *filename;
>      GKeyFile *key_file;
> @@ -96,6 +105,8 @@ struct _notmuch_config {
>      const char **new_tags;
>      size_t new_tags_length;
>      notmuch_bool_t maildir_synchronize_flags;
> +    const char **auto_exclude_tags;
> +    size_t auto_exclude_tags_length;
>  };
>  
>  static int
> @@ -221,6 +232,7 @@ notmuch_config_open (void *ctx,
>      int file_had_new_group;
>      int file_had_user_group;
>      int file_had_maildir_group;
> +    int file_had_search_group;
>  
>      if (is_new_ret)
>  	*is_new_ret = 0;
> @@ -252,6 +264,8 @@ notmuch_config_open (void *ctx,
>      config->new_tags = NULL;
>      config->new_tags_length = 0;
>      config->maildir_synchronize_flags = TRUE;
> +    config->auto_exclude_tags = NULL;
> +    config->auto_exclude_tags_length = 0;
>  
>      if (! g_key_file_load_from_file (config->key_file,
>  				     config->filename,
> @@ -295,6 +309,7 @@ notmuch_config_open (void *ctx,
>      file_had_new_group = g_key_file_has_group (config->key_file, "new");
>      file_had_user_group = g_key_file_has_group (config->key_file, "user");
>      file_had_maildir_group = g_key_file_has_group (config->key_file, "maildir");
> +    file_had_search_group = g_key_file_has_group (config->key_file, "search");
>  
>  
>      if (notmuch_config_get_database_path (config) == NULL) {
> @@ -345,6 +360,11 @@ notmuch_config_open (void *ctx,
>  	notmuch_config_set_new_tags (config, tags, 2);
>      }
>  
> +    if (notmuch_config_get_auto_exclude_tags (config, &tmp) == NULL) {
> +	const char *tags[] = { "deleted", "spam" };
> +	notmuch_config_set_auto_exclude_tags (config, tags, 2);
> +    }
> +
>      error = NULL;
>      config->maildir_synchronize_flags =
>  	g_key_file_get_boolean (config->key_file,
> @@ -387,6 +407,11 @@ notmuch_config_open (void *ctx,
>  				maildir_config_comment, NULL);
>      }
>  
> +    if (! file_had_search_group) {
> +	g_key_file_set_comment (config->key_file, "search", NULL,
> +				search_config_comment, NULL);
> +    }
> +
>      if (is_new_ret)
>  	*is_new_ret = is_new;
>  
> @@ -597,6 +622,23 @@ notmuch_config_set_new_tags (notmuch_config_t *config,
>  		     &(config->new_tags));
>  }
>  
> +const char **
> +notmuch_config_get_auto_exclude_tags (notmuch_config_t *config, size_t *length)
> +{
> +    return _config_get_list (config, "search", "auto_exclude_tags",
> +			     &(config->auto_exclude_tags),
> +			     &(config->auto_exclude_tags_length), length);
> +}
> +
> +void
> +notmuch_config_set_auto_exclude_tags (notmuch_config_t *config,
> +				      const char *list[],
> +				      size_t length)
> +{
> +    _config_set_list (config, "search", "auto_exclude_tags", list, length,
> +		      &(config->auto_exclude_tags));
> +}
> +
>  /* Given a configuration item of the form <group>.<key> return the
>   * component group and key. If any error occurs, print a message on
>   * stderr and return 1. Otherwise, return 0.
> diff --git a/notmuch-count.c b/notmuch-count.c
> index fb7401b..494619f 100644
> --- a/notmuch-count.c
> +++ b/notmuch-count.c
> @@ -35,6 +35,9 @@ notmuch_count_command (void *ctx, int argc, char *argv[])
>      char *query_str;
>      int opt_index;
>      output_t output = OUTPUT_MESSAGES;
> +    const char **auto_exclude_tags;
> +    size_t auto_exclude_tags_length;
> +    unsigned int i;
>  
>      notmuch_opt_desc_t options[] = {
>  	{ NOTMUCH_OPT_KEYWORD, &output, "output", 'o',
> @@ -75,6 +78,11 @@ notmuch_count_command (void *ctx, int argc, char *argv[])
>  	return 1;
>      }
>  
> +    auto_exclude_tags = notmuch_config_get_auto_exclude_tags
> +	(config, &auto_exclude_tags_length);
> +    for (i = 0; i < auto_exclude_tags_length; i++)
> +	notmuch_query_add_tag_exclude (query, auto_exclude_tags[i]);
> +
>      switch (output) {
>      case OUTPUT_MESSAGES:
>  	printf ("%u\n", notmuch_query_count_messages (query));
> diff --git a/notmuch-search.c b/notmuch-search.c
> index 4baab56..8867aab 100644
> --- a/notmuch-search.c
> +++ b/notmuch-search.c
> @@ -423,6 +423,9 @@ notmuch_search_command (void *ctx, int argc, char *argv[])
>      output_t output = OUTPUT_SUMMARY;
>      int offset = 0;
>      int limit = -1; /* unlimited */
> +    const char **auto_exclude_tags;
> +    size_t auto_exclude_tags_length;
> +    unsigned int i;
>  
>      enum { NOTMUCH_FORMAT_JSON, NOTMUCH_FORMAT_TEXT }
>  	format_sel = NOTMUCH_FORMAT_TEXT;
> @@ -490,6 +493,11 @@ notmuch_search_command (void *ctx, int argc, char *argv[])
>  
>      notmuch_query_set_sort (query, sort);
>  
> +    auto_exclude_tags = notmuch_config_get_auto_exclude_tags
> +	(config, &auto_exclude_tags_length);
> +    for (i = 0; i < auto_exclude_tags_length; i++)
> +	notmuch_query_add_tag_exclude (query, auto_exclude_tags[i]);
> +
>      switch (output) {
>      default:
>      case OUTPUT_SUMMARY:
> diff --git a/test/search b/test/search
> index a7a0b18..f421ae3 100755
> --- a/test/search
> +++ b/test/search
> @@ -129,4 +129,22 @@ add_message '[subject]="utf8-message-body-subject"' '[date]="Sat, 01 Jan 2000 12
>  output=$(notmuch search "bödý" | notmuch_search_sanitize)
>  test_expect_equal "$output" "thread:XXX   2000-01-01 [1/1] Notmuch Test Suite; utf8-message-body-subject (inbox unread)"
>  
> +test_begin_subtest "Search hides deleted"
> +generate_message '[subject]="Not deleted"'
> +generate_message '[subject]="Deleted"'
> +notmuch new > /dev/null
> +notmuch tag +deleted id:$gen_msg_id
> +output=$(notmuch search subject:deleted | notmuch_search_sanitize)
> +test_expect_equal "$output" "thread:XXX   2001-01-05 [1/1] Notmuch Test Suite; Not deleted (inbox unread)"
> +
> +test_begin_subtest "Search shows deleted if requested"
> +output=$(notmuch search subject:deleted and tag:deleted | notmuch_search_sanitize)
> +test_expect_equal "$output" "thread:XXX   2001-01-05 [1/1] Notmuch Test Suite; Deleted (deleted inbox unread)"
> +
> +test_begin_subtest "Search hides deleted in threads"
> +add_message '[subject]="Not deleted reply"' '[in-reply-to]="<$gen_msg_id>"'
> +output=$(notmuch search subject:deleted | notmuch_search_sanitize)
> +test_expect_equal "$output" "thread:XXX   2001-01-05 [1/1] Notmuch Test Suite; Not deleted (inbox unread)
> +thread:XXX   2001-01-05 [1/2] Notmuch Test Suite; Not deleted reply (deleted inbox unread)"
> +
>  test_done
> -- 
> 1.7.7.3
> 
> _______________________________________________
> notmuch mailing list
> notmuch@notmuchmail.org
> http://notmuchmail.org/mailman/listinfo/notmuch

[PATCH v2 0/3]

[Austin Clements]

This addresses Jani's comments and improves some of the text and code
comments.

[PATCH v2 1/3] count: Convert to new-style argument parsing

[Austin Clements]

---
 notmuch-count.c |   53 +++++++++++++++++++++++++----------------------------
 1 files changed, 25 insertions(+), 28 deletions(-)

diff --git a/notmuch-count.c b/notmuch-count.c
index 20ce334..0982f99 100644
--- a/notmuch-count.c
+++ b/notmuch-count.c
@@ -21,6 +21,11 @@
 
 #include "notmuch-client.h"
 
+enum {
+    OUTPUT_THREADS,
+    OUTPUT_MESSAGES,
+};
+
 int
 notmuch_count_command (void *ctx, int argc, char *argv[])
 {
@@ -28,35 +33,23 @@ notmuch_count_command (void *ctx, int argc, char *argv[])
     notmuch_database_t *notmuch;
     notmuch_query_t *query;
     char *query_str;
-    int i;
-    notmuch_bool_t output_messages = TRUE;
+    int opt_index;
+    int output = OUTPUT_MESSAGES;
 
-    argc--; argv++; /* skip subcommand argument */
+    notmuch_opt_desc_t options[] = {
+	{ NOTMUCH_OPT_KEYWORD, &output, "output", 'o',
+	  (notmuch_keyword_t []){ { "threads", OUTPUT_THREADS },
+				  { "messages", OUTPUT_MESSAGES },
+				  { 0, 0 } } },
+	{ 0, 0, 0, 0, 0 }
+    };
 
-    for (i = 0; i < argc && argv[i][0] == '-'; i++) {
-	if (strcmp (argv[i], "--") == 0) {
-	    i++;
-	    break;
-	}
-	if (STRNCMP_LITERAL (argv[i], "--output=") == 0) {
-	    const char *opt = argv[i] + sizeof ("--output=") - 1;
-	    if (strcmp (opt, "threads") == 0) {
-		output_messages = FALSE;
-	    } else if (strcmp (opt, "messages") == 0) {
-		output_messages = TRUE;
-	    } else {
-		fprintf (stderr, "Invalid value for --output: %s\n", opt);
-		return 1;
-	    }
-	} else {
-	    fprintf (stderr, "Unrecognized option: %s\n", argv[i]);
-	    return 1;
-	}
+    opt_index = parse_arguments (argc, argv, options, 1);
+
+    if (opt_index < 0) {
+	return 1;
     }
 
-    argc -= i;
-    argv += i;
-
     config = notmuch_config_open (ctx, NULL, NULL);
     if (config == NULL)
 	return 1;
@@ -66,7 +59,7 @@ notmuch_count_command (void *ctx, int argc, char *argv[])
     if (notmuch == NULL)
 	return 1;
 
-    query_str = query_string_from_args (ctx, argc, argv);
+    query_str = query_string_from_args (ctx, argc-opt_index, argv+opt_index);
     if (query_str == NULL) {
 	fprintf (stderr, "Out of memory.\n");
 	return 1;
@@ -82,10 +75,14 @@ notmuch_count_command (void *ctx, int argc, char *argv[])
 	return 1;
     }
 
-    if (output_messages)
+    switch (output) {
+    case OUTPUT_MESSAGES:
 	printf ("%u\n", notmuch_query_count_messages (query));
-    else
+	break;
+    case OUTPUT_THREADS:
 	printf ("%u\n", notmuch_query_count_threads (query));
+	break;
+    }
 
     notmuch_query_destroy (query);
     notmuch_database_close (notmuch);
-- 
1.7.7.3

[PATCH v2 2/3] lib: Add support for automatically excluding tags from queries

[Austin Clements]

This is useful for tags like "deleted" and "spam" that people
generally want to exclude from query results.  These exclusions will
be overridden if a tag is explicitly mentioned in a query.
---
 lib/notmuch.h |    6 ++++++
 lib/query.cc  |   35 +++++++++++++++++++++++++++++++++++
 2 files changed, 41 insertions(+), 0 deletions(-)

diff --git a/lib/notmuch.h b/lib/notmuch.h
index 9f23a10..7929fe7 100644
--- a/lib/notmuch.h
+++ b/lib/notmuch.h
@@ -457,6 +457,12 @@ notmuch_query_set_sort (notmuch_query_t *query, notmuch_sort_t sort);
 notmuch_sort_t
 notmuch_query_get_sort (notmuch_query_t *query);
 
+/* Add a tag that will be excluded from the query results by default.
+ * This exclusion will be overridden if this tag appears explicitly in
+ * the query. */
+void
+notmuch_query_add_tag_exclude (notmuch_query_t *query, const char *tag);
+
 /* Execute a query for threads, returning a notmuch_threads_t object
  * which can be used to iterate over the results. The returned threads
  * object is owned by the query and as such, will only be valid until
diff --git a/lib/query.cc b/lib/query.cc
index b6c0f12..0b36602 100644
--- a/lib/query.cc
+++ b/lib/query.cc
@@ -27,6 +27,7 @@ struct _notmuch_query {
     notmuch_database_t *notmuch;
     const char *query_string;
     notmuch_sort_t sort;
+    notmuch_string_list_t *exclude_terms;
 };
 
 typedef struct _notmuch_mset_messages {
@@ -76,6 +77,8 @@ notmuch_query_create (notmuch_database_t *notmuch,
 
     query->sort = NOTMUCH_SORT_NEWEST_FIRST;
 
+    query->exclude_terms = _notmuch_string_list_create (query);
+
     return query;
 }
 
@@ -97,6 +100,13 @@ notmuch_query_get_sort (notmuch_query_t *query)
     return query->sort;
 }
 
+void
+notmuch_query_add_tag_exclude (notmuch_query_t *query, const char *tag)
+{
+    char *term = talloc_asprintf (query, "%s%s", _find_prefix ("tag"), tag);
+    _notmuch_string_list_append (query->exclude_terms, term);
+}
+
 /* We end up having to call the destructors explicitly because we had
  * to use "placement new" in order to initialize C++ objects within a
  * block that we allocated with talloc. So C++ is making talloc
@@ -112,6 +122,27 @@ _notmuch_messages_destructor (notmuch_mset_messages_t *messages)
     return 0;
 }
 
+/* Return a query that does not match messages with the excluded tags
+ * registered with the query.  Any tags that explicitly appear in
+ * xquery will not be excluded. */
+static Xapian::Query
+_notmuch_exclude_tags (notmuch_query_t *query, Xapian::Query xquery)
+{
+    for (notmuch_string_node_t *term = query->exclude_terms->head; term;
+	 term = term->next) {
+	Xapian::TermIterator it = xquery.get_terms_begin ();
+	Xapian::TermIterator end = xquery.get_terms_end ();
+	for (; it != end; it++) {
+	    if ((*it).compare (term->string) == 0)
+		break;
+	}
+	if (it == end)
+	    xquery = Xapian::Query (Xapian::Query::OP_AND_NOT,
+				    xquery, Xapian::Query (term->string));
+    }
+    return xquery;
+}
+
 notmuch_messages_t *
 notmuch_query_search_messages (notmuch_query_t *query)
 {
@@ -157,6 +188,8 @@ notmuch_query_search_messages (notmuch_query_t *query)
 					 mail_query, string_query);
 	}
 
+	final_query = _notmuch_exclude_tags (query, final_query);
+
 	enquire.set_weighting_scheme (Xapian::BoolWeight());
 
 	switch (query->sort) {
@@ -436,6 +469,8 @@ notmuch_query_count_messages (notmuch_query_t *query)
 					 mail_query, string_query);
 	}
 
+	final_query = _notmuch_exclude_tags (query, final_query);
+
 	enquire.set_weighting_scheme(Xapian::BoolWeight());
 	enquire.set_docid_order(Xapian::Enquire::ASCENDING);
 
-- 
1.7.7.3

[PATCH v2 3/3] search: Support automatic tag exclusions

[Austin Clements]

This adds a "search" section to the config file and an
"auto_tag_exclusions" setting in that section.  The search and count
commands pass tag tags from the configuration to the library.
---
 notmuch-client.h |    8 ++++++++
 notmuch-config.c |   42 ++++++++++++++++++++++++++++++++++++++++++
 notmuch-count.c  |    8 ++++++++
 notmuch-search.c |    8 ++++++++
 test/search      |   18 ++++++++++++++++++
 5 files changed, 84 insertions(+), 0 deletions(-)

diff --git a/notmuch-client.h b/notmuch-client.h
index 517c010..62ede28 100644
--- a/notmuch-client.h
+++ b/notmuch-client.h
@@ -235,6 +235,14 @@ void
 notmuch_config_set_maildir_synchronize_flags (notmuch_config_t *config,
 					      notmuch_bool_t synchronize_flags);
 
+const char **
+notmuch_config_get_auto_exclude_tags (notmuch_config_t *config, size_t *length);
+
+void
+notmuch_config_set_auto_exclude_tags (notmuch_config_t *config,
+				      const char *list[],
+				      size_t length);
+
 int
 notmuch_run_hook (const char *db_path, const char *hook);
 
diff --git a/notmuch-config.c b/notmuch-config.c
index d697138..3d4d5b9 100644
--- a/notmuch-config.c
+++ b/notmuch-config.c
@@ -84,6 +84,15 @@ static const char maildir_config_comment[] =
     "\tand update tags, while the \"notmuch tag\" and \"notmuch restore\"\n"
     "\tcommands will notice tag changes and update flags in filenames\n";
 
+static const char search_config_comment[] =
+    " Search configuration\n"
+    "\n"
+    " The following option is supported here:\n"
+    "\n"
+    "\tauto_exclude_tags      A ;-separated list of tags that will be\n"
+    "\t excluded from search results by default.  Using an excluded tag\n"
+    "\t in a query will override that exclusion.\n";
+
 struct _notmuch_config {
     char *filename;
     GKeyFile *key_file;
@@ -96,6 +105,8 @@ struct _notmuch_config {
     const char **new_tags;
     size_t new_tags_length;
     notmuch_bool_t maildir_synchronize_flags;
+    const char **auto_exclude_tags;
+    size_t auto_exclude_tags_length;
 };
 
 static int
@@ -221,6 +232,7 @@ notmuch_config_open (void *ctx,
     int file_had_new_group;
     int file_had_user_group;
     int file_had_maildir_group;
+    int file_had_search_group;
 
     if (is_new_ret)
 	*is_new_ret = 0;
@@ -252,6 +264,8 @@ notmuch_config_open (void *ctx,
     config->new_tags = NULL;
     config->new_tags_length = 0;
     config->maildir_synchronize_flags = TRUE;
+    config->auto_exclude_tags = NULL;
+    config->auto_exclude_tags_length = 0;
 
     if (! g_key_file_load_from_file (config->key_file,
 				     config->filename,
@@ -295,6 +309,7 @@ notmuch_config_open (void *ctx,
     file_had_new_group = g_key_file_has_group (config->key_file, "new");
     file_had_user_group = g_key_file_has_group (config->key_file, "user");
     file_had_maildir_group = g_key_file_has_group (config->key_file, "maildir");
+    file_had_search_group = g_key_file_has_group (config->key_file, "search");
 
 
     if (notmuch_config_get_database_path (config) == NULL) {
@@ -345,6 +360,11 @@ notmuch_config_open (void *ctx,
 	notmuch_config_set_new_tags (config, tags, 2);
     }
 
+    if (notmuch_config_get_auto_exclude_tags (config, &tmp) == NULL) {
+	const char *tags[] = { "deleted", "spam" };
+	notmuch_config_set_auto_exclude_tags (config, tags, 2);
+    }
+
     error = NULL;
     config->maildir_synchronize_flags =
 	g_key_file_get_boolean (config->key_file,
@@ -387,6 +407,11 @@ notmuch_config_open (void *ctx,
 				maildir_config_comment, NULL);
     }
 
+    if (! file_had_search_group) {
+	g_key_file_set_comment (config->key_file, "search", NULL,
+				search_config_comment, NULL);
+    }
+
     if (is_new_ret)
 	*is_new_ret = is_new;
 
@@ -597,6 +622,23 @@ notmuch_config_set_new_tags (notmuch_config_t *config,
 		     &(config->new_tags));
 }
 
+const char **
+notmuch_config_get_auto_exclude_tags (notmuch_config_t *config, size_t *length)
+{
+    return _config_get_list (config, "search", "auto_exclude_tags",
+			     &(config->auto_exclude_tags),
+			     &(config->auto_exclude_tags_length), length);
+}
+
+void
+notmuch_config_set_auto_exclude_tags (notmuch_config_t *config,
+				      const char *list[],
+				      size_t length)
+{
+    _config_set_list (config, "search", "auto_exclude_tags", list, length,
+		      &(config->auto_exclude_tags));
+}
+
 /* Given a configuration item of the form <group>.<key> return the
  * component group and key. If any error occurs, print a message on
  * stderr and return 1. Otherwise, return 0.
diff --git a/notmuch-count.c b/notmuch-count.c
index 0982f99..f77861e 100644
--- a/notmuch-count.c
+++ b/notmuch-count.c
@@ -35,6 +35,9 @@ notmuch_count_command (void *ctx, int argc, char *argv[])
     char *query_str;
     int opt_index;
     int output = OUTPUT_MESSAGES;
+    const char **auto_exclude_tags;
+    size_t auto_exclude_tags_length;
+    unsigned int i;
 
     notmuch_opt_desc_t options[] = {
 	{ NOTMUCH_OPT_KEYWORD, &output, "output", 'o',
@@ -75,6 +78,11 @@ notmuch_count_command (void *ctx, int argc, char *argv[])
 	return 1;
     }
 
+    auto_exclude_tags = notmuch_config_get_auto_exclude_tags
+	(config, &auto_exclude_tags_length);
+    for (i = 0; i < auto_exclude_tags_length; i++)
+	notmuch_query_add_tag_exclude (query, auto_exclude_tags[i]);
+
     switch (output) {
     case OUTPUT_MESSAGES:
 	printf ("%u\n", notmuch_query_count_messages (query));
diff --git a/notmuch-search.c b/notmuch-search.c
index 4baab56..8867aab 100644
--- a/notmuch-search.c
+++ b/notmuch-search.c
@@ -423,6 +423,9 @@ notmuch_search_command (void *ctx, int argc, char *argv[])
     output_t output = OUTPUT_SUMMARY;
     int offset = 0;
     int limit = -1; /* unlimited */
+    const char **auto_exclude_tags;
+    size_t auto_exclude_tags_length;
+    unsigned int i;
 
     enum { NOTMUCH_FORMAT_JSON, NOTMUCH_FORMAT_TEXT }
 	format_sel = NOTMUCH_FORMAT_TEXT;
@@ -490,6 +493,11 @@ notmuch_search_command (void *ctx, int argc, char *argv[])
 
     notmuch_query_set_sort (query, sort);
 
+    auto_exclude_tags = notmuch_config_get_auto_exclude_tags
+	(config, &auto_exclude_tags_length);
+    for (i = 0; i < auto_exclude_tags_length; i++)
+	notmuch_query_add_tag_exclude (query, auto_exclude_tags[i]);
+
     switch (output) {
     default:
     case OUTPUT_SUMMARY:
diff --git a/test/search b/test/search
index a7a0b18..bf965e7 100755
--- a/test/search
+++ b/test/search
@@ -129,4 +129,22 @@ add_message '[subject]="utf8-message-body-subject"' '[date]="Sat, 01 Jan 2000 12
 output=$(notmuch search "bödý" | notmuch_search_sanitize)
 test_expect_equal "$output" "thread:XXX   2000-01-01 [1/1] Notmuch Test Suite; utf8-message-body-subject (inbox unread)"
 
+test_begin_subtest "Exclude \"deleted\" messages from search"
+generate_message '[subject]="Not deleted"'
+generate_message '[subject]="Deleted"'
+notmuch new > /dev/null
+notmuch tag +deleted id:$gen_msg_id
+output=$(notmuch search subject:deleted | notmuch_search_sanitize)
+test_expect_equal "$output" "thread:XXX   2001-01-05 [1/1] Notmuch Test Suite; Not deleted (inbox unread)"
+
+test_begin_subtest "Exclude \"deleted\" messages from search, overridden"
+output=$(notmuch search subject:deleted and tag:deleted | notmuch_search_sanitize)
+test_expect_equal "$output" "thread:XXX   2001-01-05 [1/1] Notmuch Test Suite; Deleted (deleted inbox unread)"
+
+test_begin_subtest "Exclude \"deleted\" messages from threads"
+add_message '[subject]="Not deleted reply"' '[in-reply-to]="<$gen_msg_id>"'
+output=$(notmuch search subject:deleted | notmuch_search_sanitize)
+test_expect_equal "$output" "thread:XXX   2001-01-05 [1/1] Notmuch Test Suite; Not deleted (inbox unread)
+thread:XXX   2001-01-05 [1/2] Notmuch Test Suite; Not deleted reply (deleted inbox unread)"
+
 test_done
-- 
1.7.7.3

Re: [PATCH v2 1/3] count: Convert to new-style argument parsing

[David Bremner]

On Fri, 13 Jan 2012 18:07:02 -0500, Austin Clements <amdragon@MIT.EDU> wrote:
> ---
>  notmuch-count.c |   53 +++++++++++++++++++++++++----------------------------
>  1 files changed, 25 insertions(+), 28 deletions(-)

This seems independent of the rest of the series, and kindof obvious by
this point, so I pushed it.

d

Re: [PATCH v2 0/3]

[Jameson Graef Rollins]

[-- Attachment #1: Type: text/plain, Size: 292 bytes --]

On Fri, 13 Jan 2012 18:07:01 -0500, Austin Clements <amdragon@MIT.EDU> wrote:
> This addresses Jani's comments and improves some of the text and code
> comments.

This is a really nice feature addition, Austin.  And it works like a
charm.  ++1.

A couple of small comments to follow.

jamie.

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: [PATCH v2 2/3] lib: Add support for automatically excluding tags from queries

[Jameson Graef Rollins]

[-- Attachment #1: Type: text/plain, Size: 1391 bytes --]

It looks like something in this patch is causing the following build
warning:

CXX -O2 lib/query.o
lib/query.cc:26:8: warning: ‘_notmuch_query’ declared with greater visibility than the type of its field
‘_notmuch_query::exclude_terms’ [-Wattributes]

However, I can't quite figure out what's causing it.

> +void
> +notmuch_query_add_tag_exclude (notmuch_query_t *query, const char *tag)
> +{
> +    char *term = talloc_asprintf (query, "%s%s", _find_prefix ("tag"), tag);
> +    _notmuch_string_list_append (query->exclude_terms, term);
> +}

This is really not an issue with this patch at all, and it should NOT
prevent it from being applied, but this came up briefly on IRC and I'm
curious, so I'll ask about it here.

Are terms ALWAYS lower cased?  If not, it seems to me it's possible to
have an indexed term 'Kspam' that would get confused with the term
'spam' prefixed with the keyword prefix 'K' (which we use for tags).
Maybe this degeneracy is broken by the query parser somehow (or maybe by
the fact that tags are boolean terms?), but I wonder if it's not safer
to use the built-in xapian prefix separator ':', ie:

  ... talloc_asprintf (query, "%s:%s", _find_prefix ("tag"), tag);

I guess fixing that globally would require a database rebuild...

Ok, that's totally just an aside, and should not be a blocker for this
patch.

jamie.

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: [PATCH v2 3/3] search: Support automatic tag exclusions

[Jameson Graef Rollins]

[-- Attachment #1: Type: text/plain, Size: 1142 bytes --]

This patch looks fine.  Philosophical UI discussion to follow:

On Fri, 13 Jan 2012 18:07:04 -0500, Austin Clements <amdragon@MIT.EDU> wrote:
> +    if (notmuch_config_get_auto_exclude_tags (config, &tmp) == NULL) {
> +	const char *tags[] = { "deleted", "spam" };
> +	notmuch_config_set_auto_exclude_tags (config, tags, 2);
> +    }

This creates the config section with the exclude list pre-set to
"deleted;spam".  I personally have no problem with this, since I was
going to be setting exactly that anyway.  However, assuming we decide to
have this be the default in the CLI, should we therefore add support for
it in the emacs UI?  I've been going back and forth on this (as readers
are well aware), and have most recently rejected the idea that we should
add delete support to the emacs UI.  However, if we are excluding
"deleted" tags by default, then I'm going to go back and say that we
should include the keybindings to "delete" messages.  Comments?

If people think we should exclude "deleted;spam" by default, and agree
that we should also add delete support in the emacs UI, I'll go ahead
and rework my keybinding patches.

jamie.

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: [PATCH v2 2/3] lib: Add support for automatically excluding tags from queries

[Austin Clements]

Quoth Jameson Graef Rollins on Jan 14 at  3:38 pm:
> It looks like something in this patch is causing the following build
> warning:
> 
> CXX -O2 lib/query.o
> lib/query.cc:26:8: warning: ‘_notmuch_query’ declared with greater visibility than the type of its field
> ‘_notmuch_query::exclude_terms’ [-Wattributes]
> 
> However, I can't quite figure out what's causing it.

The problem is that notmuch_query_t is a "visible" symbol because the
type is declared (though not defined) in lib/notmuch.h.  The actual
definition is tucked away in lib/query.cc, but GCC doesn't seem to
care.  I added a field of type notmuch_string_list_t to the struct's
definition, but notmuch_string_list_t is declared between the "hidden"
pragmas in lib/notmuch-private.h because the type is private to the
library.  This field with a hidden type in a visible type is what GCC
is complaining about.

I'm rather confused by the whole type visibility thing since type
symbols aren't linkable anyway.  However, while puzzling over how you
could possibly use hidden types if they can only be used in other
hidden types, I discovered that Carl had solved this exact problem
last May in d5523ead by adding a visibility("default") attribute to
the offending hidden type.

> > +void
> > +notmuch_query_add_tag_exclude (notmuch_query_t *query, const char *tag)
> > +{
> > +    char *term = talloc_asprintf (query, "%s%s", _find_prefix ("tag"), tag);
> > +    _notmuch_string_list_append (query->exclude_terms, term);
> > +}
> 
> This is really not an issue with this patch at all, and it should NOT
> prevent it from being applied, but this came up briefly on IRC and I'm
> curious, so I'll ask about it here.
> 
> Are terms ALWAYS lower cased?  If not, it seems to me it's possible to
> have an indexed term 'Kspam' that would get confused with the term
> 'spam' prefixed with the keyword prefix 'K' (which we use for tags).
> Maybe this degeneracy is broken by the query parser somehow (or maybe by
> the fact that tags are boolean terms?), but I wonder if it's not safer
> to use the built-in xapian prefix separator ':', ie:
> 
>   ... talloc_asprintf (query, "%s:%s", _find_prefix ("tag"), tag);
> 
> I guess fixing that globally would require a database rebuild...

We discussed this on IRC, but to summarize for the list, the tag
prefix is a single character, so Xapian's ':' rule doesn't apply.
There are several places where we *do* get this wrong and use a
multi-character term prefix with a term that may start with a capital
letter but they're all terms you can't search anyway and, unless I'm
mistaken, we're completely consistent about where we violate or do not
violate the ':' rule.

> Ok, that's totally just an aside, and should not be a blocker for this
> patch.
> 
> jamie.

Re: [PATCH v2 3/3] search: Support automatic tag exclusions

[Austin Clements]

Quoth Jameson Graef Rollins on Jan 14 at  3:40 pm:
> This patch looks fine.  Philosophical UI discussion to follow:
> 
> On Fri, 13 Jan 2012 18:07:04 -0500, Austin Clements <amdragon@MIT.EDU> wrote:
> > +    if (notmuch_config_get_auto_exclude_tags (config, &tmp) == NULL) {
> > +	const char *tags[] = { "deleted", "spam" };
> > +	notmuch_config_set_auto_exclude_tags (config, tags, 2);
> > +    }
> 
> This creates the config section with the exclude list pre-set to
> "deleted;spam".  I personally have no problem with this, since I was
> going to be setting exactly that anyway.  However, assuming we decide to
> have this be the default in the CLI, should we therefore add support for
> it in the emacs UI?  I've been going back and forth on this (as readers
> are well aware), and have most recently rejected the idea that we should
> add delete support to the emacs UI.  However, if we are excluding
> "deleted" tags by default, then I'm going to go back and say that we
> should include the keybindings to "delete" messages.  Comments?

It's not that Emacs doesn't support the deleted tag.  You can always
+deleted<RET>, and this even seems like a pretty natural thing to do.
To me, the question is whether there should be a shortcut to do this.
I'm probably not one to answer this, since I don't plan to use the
deleted tag and hence using this binding would only ever be an
accident (though I will use the spam tag and I don't think I need a
binding for that; perhaps I would feel differently if my spam filters
were less effective).

> If people think we should exclude "deleted;spam" by default, and agree
> that we should also add delete support in the emacs UI, I'll go ahead
> and rework my keybinding patches.
> 
> jamie.

[PATCH v3 0/2] Automatic tag-based exclusion

[Austin Clements]

This fixes the symbol visibility warning Jamie pointed out.

[PATCH v3 1/2] lib: Add support for automatically excluding tags from queries

[Austin Clements]

This is useful for tags like "deleted" and "spam" that people
generally want to exclude from query results.  These exclusions will
be overridden if a tag is explicitly mentioned in a query.
---
 lib/notmuch-private.h |    2 +-
 lib/notmuch.h         |    6 ++++++
 lib/query.cc          |   35 +++++++++++++++++++++++++++++++++++
 3 files changed, 42 insertions(+), 1 deletions(-)

diff --git a/lib/notmuch-private.h b/lib/notmuch-private.h
index 60a932f..7bf153e 100644
--- a/lib/notmuch-private.h
+++ b/lib/notmuch-private.h
@@ -458,7 +458,7 @@ typedef struct _notmuch_string_node {
     struct _notmuch_string_node *next;
 } notmuch_string_node_t;
 
-typedef struct _notmuch_string_list {
+typedef struct visible _notmuch_string_list {
     int length;
     notmuch_string_node_t *head;
     notmuch_string_node_t **tail;
diff --git a/lib/notmuch.h b/lib/notmuch.h
index 9f23a10..7929fe7 100644
--- a/lib/notmuch.h
+++ b/lib/notmuch.h
@@ -457,6 +457,12 @@ notmuch_query_set_sort (notmuch_query_t *query, notmuch_sort_t sort);
 notmuch_sort_t
 notmuch_query_get_sort (notmuch_query_t *query);
 
+/* Add a tag that will be excluded from the query results by default.
+ * This exclusion will be overridden if this tag appears explicitly in
+ * the query. */
+void
+notmuch_query_add_tag_exclude (notmuch_query_t *query, const char *tag);
+
 /* Execute a query for threads, returning a notmuch_threads_t object
  * which can be used to iterate over the results. The returned threads
  * object is owned by the query and as such, will only be valid until
diff --git a/lib/query.cc b/lib/query.cc
index b6c0f12..0b36602 100644
--- a/lib/query.cc
+++ b/lib/query.cc
@@ -27,6 +27,7 @@ struct _notmuch_query {
     notmuch_database_t *notmuch;
     const char *query_string;
     notmuch_sort_t sort;
+    notmuch_string_list_t *exclude_terms;
 };
 
 typedef struct _notmuch_mset_messages {
@@ -76,6 +77,8 @@ notmuch_query_create (notmuch_database_t *notmuch,
 
     query->sort = NOTMUCH_SORT_NEWEST_FIRST;
 
+    query->exclude_terms = _notmuch_string_list_create (query);
+
     return query;
 }
 
@@ -97,6 +100,13 @@ notmuch_query_get_sort (notmuch_query_t *query)
     return query->sort;
 }
 
+void
+notmuch_query_add_tag_exclude (notmuch_query_t *query, const char *tag)
+{
+    char *term = talloc_asprintf (query, "%s%s", _find_prefix ("tag"), tag);
+    _notmuch_string_list_append (query->exclude_terms, term);
+}
+
 /* We end up having to call the destructors explicitly because we had
  * to use "placement new" in order to initialize C++ objects within a
  * block that we allocated with talloc. So C++ is making talloc
@@ -112,6 +122,27 @@ _notmuch_messages_destructor (notmuch_mset_messages_t *messages)
     return 0;
 }
 
+/* Return a query that does not match messages with the excluded tags
+ * registered with the query.  Any tags that explicitly appear in
+ * xquery will not be excluded. */
+static Xapian::Query
+_notmuch_exclude_tags (notmuch_query_t *query, Xapian::Query xquery)
+{
+    for (notmuch_string_node_t *term = query->exclude_terms->head; term;
+	 term = term->next) {
+	Xapian::TermIterator it = xquery.get_terms_begin ();
+	Xapian::TermIterator end = xquery.get_terms_end ();
+	for (; it != end; it++) {
+	    if ((*it).compare (term->string) == 0)
+		break;
+	}
+	if (it == end)
+	    xquery = Xapian::Query (Xapian::Query::OP_AND_NOT,
+				    xquery, Xapian::Query (term->string));
+    }
+    return xquery;
+}
+
 notmuch_messages_t *
 notmuch_query_search_messages (notmuch_query_t *query)
 {
@@ -157,6 +188,8 @@ notmuch_query_search_messages (notmuch_query_t *query)
 					 mail_query, string_query);
 	}
 
+	final_query = _notmuch_exclude_tags (query, final_query);
+
 	enquire.set_weighting_scheme (Xapian::BoolWeight());
 
 	switch (query->sort) {
@@ -436,6 +469,8 @@ notmuch_query_count_messages (notmuch_query_t *query)
 					 mail_query, string_query);
 	}
 
+	final_query = _notmuch_exclude_tags (query, final_query);
+
 	enquire.set_weighting_scheme(Xapian::BoolWeight());
 	enquire.set_docid_order(Xapian::Enquire::ASCENDING);
 
-- 
1.7.7.3

[PATCH v3 2/2] search: Support automatic tag exclusions

[Austin Clements]

This adds a "search" section to the config file and an
"auto_tag_exclusions" setting in that section.  The search and count
commands pass tag tags from the configuration to the library.
---
 notmuch-client.h |    8 ++++++++
 notmuch-config.c |   42 ++++++++++++++++++++++++++++++++++++++++++
 notmuch-count.c  |    8 ++++++++
 notmuch-search.c |    8 ++++++++
 test/search      |   18 ++++++++++++++++++
 5 files changed, 84 insertions(+), 0 deletions(-)

diff --git a/notmuch-client.h b/notmuch-client.h
index 517c010..62ede28 100644
--- a/notmuch-client.h
+++ b/notmuch-client.h
@@ -235,6 +235,14 @@ void
 notmuch_config_set_maildir_synchronize_flags (notmuch_config_t *config,
 					      notmuch_bool_t synchronize_flags);
 
+const char **
+notmuch_config_get_auto_exclude_tags (notmuch_config_t *config, size_t *length);
+
+void
+notmuch_config_set_auto_exclude_tags (notmuch_config_t *config,
+				      const char *list[],
+				      size_t length);
+
 int
 notmuch_run_hook (const char *db_path, const char *hook);
 
diff --git a/notmuch-config.c b/notmuch-config.c
index d697138..3d4d5b9 100644
--- a/notmuch-config.c
+++ b/notmuch-config.c
@@ -84,6 +84,15 @@ static const char maildir_config_comment[] =
     "\tand update tags, while the \"notmuch tag\" and \"notmuch restore\"\n"
     "\tcommands will notice tag changes and update flags in filenames\n";
 
+static const char search_config_comment[] =
+    " Search configuration\n"
+    "\n"
+    " The following option is supported here:\n"
+    "\n"
+    "\tauto_exclude_tags      A ;-separated list of tags that will be\n"
+    "\t excluded from search results by default.  Using an excluded tag\n"
+    "\t in a query will override that exclusion.\n";
+
 struct _notmuch_config {
     char *filename;
     GKeyFile *key_file;
@@ -96,6 +105,8 @@ struct _notmuch_config {
     const char **new_tags;
     size_t new_tags_length;
     notmuch_bool_t maildir_synchronize_flags;
+    const char **auto_exclude_tags;
+    size_t auto_exclude_tags_length;
 };
 
 static int
@@ -221,6 +232,7 @@ notmuch_config_open (void *ctx,
     int file_had_new_group;
     int file_had_user_group;
     int file_had_maildir_group;
+    int file_had_search_group;
 
     if (is_new_ret)
 	*is_new_ret = 0;
@@ -252,6 +264,8 @@ notmuch_config_open (void *ctx,
     config->new_tags = NULL;
     config->new_tags_length = 0;
     config->maildir_synchronize_flags = TRUE;
+    config->auto_exclude_tags = NULL;
+    config->auto_exclude_tags_length = 0;
 
     if (! g_key_file_load_from_file (config->key_file,
 				     config->filename,
@@ -295,6 +309,7 @@ notmuch_config_open (void *ctx,
     file_had_new_group = g_key_file_has_group (config->key_file, "new");
     file_had_user_group = g_key_file_has_group (config->key_file, "user");
     file_had_maildir_group = g_key_file_has_group (config->key_file, "maildir");
+    file_had_search_group = g_key_file_has_group (config->key_file, "search");
 
 
     if (notmuch_config_get_database_path (config) == NULL) {
@@ -345,6 +360,11 @@ notmuch_config_open (void *ctx,
 	notmuch_config_set_new_tags (config, tags, 2);
     }
 
+    if (notmuch_config_get_auto_exclude_tags (config, &tmp) == NULL) {
+	const char *tags[] = { "deleted", "spam" };
+	notmuch_config_set_auto_exclude_tags (config, tags, 2);
+    }
+
     error = NULL;
     config->maildir_synchronize_flags =
 	g_key_file_get_boolean (config->key_file,
@@ -387,6 +407,11 @@ notmuch_config_open (void *ctx,
 				maildir_config_comment, NULL);
     }
 
+    if (! file_had_search_group) {
+	g_key_file_set_comment (config->key_file, "search", NULL,
+				search_config_comment, NULL);
+    }
+
     if (is_new_ret)
 	*is_new_ret = is_new;
 
@@ -597,6 +622,23 @@ notmuch_config_set_new_tags (notmuch_config_t *config,
 		     &(config->new_tags));
 }
 
+const char **
+notmuch_config_get_auto_exclude_tags (notmuch_config_t *config, size_t *length)
+{
+    return _config_get_list (config, "search", "auto_exclude_tags",
+			     &(config->auto_exclude_tags),
+			     &(config->auto_exclude_tags_length), length);
+}
+
+void
+notmuch_config_set_auto_exclude_tags (notmuch_config_t *config,
+				      const char *list[],
+				      size_t length)
+{
+    _config_set_list (config, "search", "auto_exclude_tags", list, length,
+		      &(config->auto_exclude_tags));
+}
+
 /* Given a configuration item of the form <group>.<key> return the
  * component group and key. If any error occurs, print a message on
  * stderr and return 1. Otherwise, return 0.
diff --git a/notmuch-count.c b/notmuch-count.c
index 0982f99..f77861e 100644
--- a/notmuch-count.c
+++ b/notmuch-count.c
@@ -35,6 +35,9 @@ notmuch_count_command (void *ctx, int argc, char *argv[])
     char *query_str;
     int opt_index;
     int output = OUTPUT_MESSAGES;
+    const char **auto_exclude_tags;
+    size_t auto_exclude_tags_length;
+    unsigned int i;
 
     notmuch_opt_desc_t options[] = {
 	{ NOTMUCH_OPT_KEYWORD, &output, "output", 'o',
@@ -75,6 +78,11 @@ notmuch_count_command (void *ctx, int argc, char *argv[])
 	return 1;
     }
 
+    auto_exclude_tags = notmuch_config_get_auto_exclude_tags
+	(config, &auto_exclude_tags_length);
+    for (i = 0; i < auto_exclude_tags_length; i++)
+	notmuch_query_add_tag_exclude (query, auto_exclude_tags[i]);
+
     switch (output) {
     case OUTPUT_MESSAGES:
 	printf ("%u\n", notmuch_query_count_messages (query));
diff --git a/notmuch-search.c b/notmuch-search.c
index 4baab56..8867aab 100644
--- a/notmuch-search.c
+++ b/notmuch-search.c
@@ -423,6 +423,9 @@ notmuch_search_command (void *ctx, int argc, char *argv[])
     output_t output = OUTPUT_SUMMARY;
     int offset = 0;
     int limit = -1; /* unlimited */
+    const char **auto_exclude_tags;
+    size_t auto_exclude_tags_length;
+    unsigned int i;
 
     enum { NOTMUCH_FORMAT_JSON, NOTMUCH_FORMAT_TEXT }
 	format_sel = NOTMUCH_FORMAT_TEXT;
@@ -490,6 +493,11 @@ notmuch_search_command (void *ctx, int argc, char *argv[])
 
     notmuch_query_set_sort (query, sort);
 
+    auto_exclude_tags = notmuch_config_get_auto_exclude_tags
+	(config, &auto_exclude_tags_length);
+    for (i = 0; i < auto_exclude_tags_length; i++)
+	notmuch_query_add_tag_exclude (query, auto_exclude_tags[i]);
+
     switch (output) {
     default:
     case OUTPUT_SUMMARY:
diff --git a/test/search b/test/search
index a7a0b18..bf965e7 100755
--- a/test/search
+++ b/test/search
@@ -129,4 +129,22 @@ add_message '[subject]="utf8-message-body-subject"' '[date]="Sat, 01 Jan 2000 12
 output=$(notmuch search "bödý" | notmuch_search_sanitize)
 test_expect_equal "$output" "thread:XXX   2000-01-01 [1/1] Notmuch Test Suite; utf8-message-body-subject (inbox unread)"
 
+test_begin_subtest "Exclude \"deleted\" messages from search"
+generate_message '[subject]="Not deleted"'
+generate_message '[subject]="Deleted"'
+notmuch new > /dev/null
+notmuch tag +deleted id:$gen_msg_id
+output=$(notmuch search subject:deleted | notmuch_search_sanitize)
+test_expect_equal "$output" "thread:XXX   2001-01-05 [1/1] Notmuch Test Suite; Not deleted (inbox unread)"
+
+test_begin_subtest "Exclude \"deleted\" messages from search, overridden"
+output=$(notmuch search subject:deleted and tag:deleted | notmuch_search_sanitize)
+test_expect_equal "$output" "thread:XXX   2001-01-05 [1/1] Notmuch Test Suite; Deleted (deleted inbox unread)"
+
+test_begin_subtest "Exclude \"deleted\" messages from threads"
+add_message '[subject]="Not deleted reply"' '[in-reply-to]="<$gen_msg_id>"'
+output=$(notmuch search subject:deleted | notmuch_search_sanitize)
+test_expect_equal "$output" "thread:XXX   2001-01-05 [1/1] Notmuch Test Suite; Not deleted (inbox unread)
+thread:XXX   2001-01-05 [1/2] Notmuch Test Suite; Not deleted reply (deleted inbox unread)"
+
 test_done
-- 
1.7.7.3

Re: [PATCH v2 3/3] search: Support automatic tag exclusions

[David Edmondson]

[-- Attachment #1: Type: text/plain, Size: 793 bytes --]

On Sat, 14 Jan 2012 15:40:26 -0800, Jameson Graef Rollins <jrollins@finestructure.net> wrote:
> This patch looks fine.  Philosophical UI discussion to follow:
> 
> On Fri, 13 Jan 2012 18:07:04 -0500, Austin Clements <amdragon@MIT.EDU> wrote:
> > +    if (notmuch_config_get_auto_exclude_tags (config, &tmp) == NULL) {
> > +	const char *tags[] = { "deleted", "spam" };
> > +	notmuch_config_set_auto_exclude_tags (config, tags, 2);
> > +    }
> 
> This creates the config section with the exclude list pre-set to
> "deleted;spam".

I don't think that anything should be excluded from the search results
by default, at least not as a default behaviour of the 'notmuch'
binary.

Having "deleted" and "spam" as default settings in the configuration
file might be more reasonable.

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: [PATCH v2 3/3] search: Support automatic tag exclusions

[Austin Clements]

Quoth David Edmondson on Jan 16 at  9:12 am:
> On Sat, 14 Jan 2012 15:40:26 -0800, Jameson Graef Rollins <jrollins@finestructure.net> wrote:
> > This patch looks fine.  Philosophical UI discussion to follow:
> > 
> > On Fri, 13 Jan 2012 18:07:04 -0500, Austin Clements <amdragon@MIT.EDU> wrote:
> > > +    if (notmuch_config_get_auto_exclude_tags (config, &tmp) == NULL) {
> > > +	const char *tags[] = { "deleted", "spam" };
> > > +	notmuch_config_set_auto_exclude_tags (config, tags, 2);
> > > +    }
> > 
> > This creates the config section with the exclude list pre-set to
> > "deleted;spam".
> 
> I don't think that anything should be excluded from the search results
> by default, at least not as a default behaviour of the 'notmuch'
> binary.
> 
> Having "deleted" and "spam" as default settings in the configuration
> file might be more reasonable.

Sorry, I'm confused.  Are you saying deleted;spam should or should not
be the default?

Re: [PATCH v2 3/3] search: Support automatic tag exclusions

[Jameson Graef Rollins]

[-- Attachment #1: Type: text/plain, Size: 536 bytes --]

On Mon, 16 Jan 2012 09:12:38 +0000, David Edmondson <dme@dme.org> wrote:
> I don't think that anything should be excluded from the search results
> by default, at least not as a default behaviour of the 'notmuch'
> binary.
> 
> Having "deleted" and "spam" as default settings in the configuration
> file might be more reasonable.

Yes, I agree completely.  The CLI does not, and should not, exclude
anything by default.  I only meant that the patch adds the configuration
variable by default, which seems ok to me.

jamie.

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: [PATCH v3 0/2] Automatic tag-based exclusion

[Jameson Graef Rollins]

[-- Attachment #1: Type: text/plain, Size: 202 bytes --]

On Sat, 14 Jan 2012 19:17:32 -0500, Austin Clements <amdragon@MIT.EDU> wrote:
> This fixes the symbol visibility warning Jamie pointed out.

LGTM.  I'm using this now and it works like a charm.

jamie.

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: [PATCH v2 3/3] search: Support automatic tag exclusions

[Jeremy Nickurak]

On Mon, Jan 16, 2012 at 12:28, Austin Clements <amdragon@mit.edu> wrote:
> Quoth David Edmondson on Jan 16 at  9:12 am:
>> Having "deleted" and "spam" as default settings in the configuration
>> file might be more reasonable.
>
> Sorry, I'm confused.  Are you saying deleted;spam should or should not
> be the default?

If I read correctly:

1) If no exclude options are in the config file, none should be used.
2) On notmuch setup, "deleted" and "spam" should be added to .notmuch-config

Re: [PATCH v2 3/3] search: Support automatic tag exclusions

[Jameson Graef Rollins]

[-- Attachment #1: Type: text/plain, Size: 262 bytes --]

On Mon, 16 Jan 2012 15:18:18 -0700, Jeremy Nickurak <not-much@trk.nickurak.ca> wrote:
> 1) If no exclude options are in the config file, none should be used.
> 2) On notmuch setup, "deleted" and "spam" should be added to .notmuch-config

That's correct.

jamie.

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: [PATCH v3 0/2] Automatic tag-based exclusion

[David Bremner]

On Sat, 14 Jan 2012 19:17:32 -0500, Austin Clements <amdragon@MIT.EDU> wrote:
> This fixes the symbol visibility warning Jamie pointed out.

Third time the charm.

pushed. 

Please (somebody) consider writing NEWS and man page updates.

d

Re: [PATCH v2 3/3] search: Support automatic tag exclusions

[David Edmondson]

[-- Attachment #1: Type: text/plain, Size: 537 bytes --]

On Mon, 16 Jan 2012 15:16:24 -0700, Jeremy Nickurak <jeremy@nickurak.ca> wrote:
> On Mon, Jan 16, 2012 at 12:28, Austin Clements <amdragon@mit.edu> wrote:
> >> Having "deleted" and "spam" as default settings in the configuration
> >> file might be more reasonable.
> 
> If I read correctly:
> 
> 1) If no exclude options are in the config file, none should be used.

Yes.

> 2) On notmuch setup, "deleted" and "spam" should be added to .notmuch-config

I might argue between 'should' and 'could', but the sense is correct.

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: [PATCH v2 3/3] search: Support automatic tag exclusions

[Austin Clements]

Quoth David Edmondson on Jan 17 at  9:08 am:
> On Mon, 16 Jan 2012 15:16:24 -0700, Jeremy Nickurak <jeremy@nickurak.ca> wrote:
> > On Mon, Jan 16, 2012 at 12:28, Austin Clements <amdragon@mit.edu> wrote:
> > >> Having "deleted" and "spam" as default settings in the configuration
> > >> file might be more reasonable.
> > 
> > If I read correctly:
> > 
> > 1) If no exclude options are in the config file, none should be used.
> 
> Yes.
> 
> > 2) On notmuch setup, "deleted" and "spam" should be added to .notmuch-config
> 
> I might argue between 'should' and 'could', but the sense is correct.

Oh, I think I see.  I don't know if I can do precisely that, since the
config code doesn't know if it's being called from setup, but is
something like this essentially what you're suggesting?

    if (notmuch_config_get_auto_exclude_tags (config, &tmp) == NULL) {
        if (is_new) {
            const char *tags[] = { "deleted", "spam" };
            notmuch_config_set_auto_exclude_tags (config, tags, 2);
        } else {
            notmuch_config_set_auto_exclude_tags (config, NULL, 0);
        }
    }

(where is_new is TRUE if this is a brand-new config file)

Re: [PATCH v2 3/3] search: Support automatic tag exclusions

[David Edmondson]

[-- Attachment #1: Type: text/plain, Size: 1987 bytes --]

On Tue, 17 Jan 2012 15:32:11 -0500, Austin Clements <amdragon@MIT.EDU> wrote:
> Quoth David Edmondson on Jan 17 at  9:08 am:
> > On Mon, 16 Jan 2012 15:16:24 -0700, Jeremy Nickurak <jeremy@nickurak.ca> wrote:
> > > On Mon, Jan 16, 2012 at 12:28, Austin Clements <amdragon@mit.edu> wrote:
> > > >> Having "deleted" and "spam" as default settings in the configuration
> > > >> file might be more reasonable.
> > > 
> > > If I read correctly:
> > > 
> > > 1) If no exclude options are in the config file, none should be used.
> > 
> > Yes.
> > 
> > > 2) On notmuch setup, "deleted" and "spam" should be added to .notmuch-config
> > 
> > I might argue between 'should' and 'could', but the sense is correct.
> 
> Oh, I think I see.  I don't know if I can do precisely that, since the
> config code doesn't know if it's being called from setup, but is
> something like this essentially what you're suggesting?
> 
>     if (notmuch_config_get_auto_exclude_tags (config, &tmp) == NULL) {
>         if (is_new) {
>             const char *tags[] = { "deleted", "spam" };
>             notmuch_config_set_auto_exclude_tags (config, tags, 2);
>         } else {
>             notmuch_config_set_auto_exclude_tags (config, NULL, 0);
>         }
>     }
> 
> (where is_new is TRUE if this is a brand-new config file)

I'm not sure, as I haven't looked at the configuration code at
all, sorry.

Something must create the initial configuration file if none exists. I'd
be okay with that code adding 'deleted' and 'spam' to the excluded list.

This would mean that an existing user would see no change without taking
some action (adding the tags to the configuration file) and a new user
would see the new behaviour (automatic exclusion).

I'm not completely sure that automatically adding the exclusion of the
specified tags via the configuration file for new users is a great
idea. It seems as though it will lead to confusion for someone at some
point.

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: [PATCH v2 3/3] search: Support automatic tag exclusions

[Jameson Graef Rollins]

[-- Attachment #1: Type: text/plain, Size: 1125 bytes --]

On Wed, 18 Jan 2012 08:38:23 +0000, David Edmondson <dme@dme.org> wrote:
> Something must create the initial configuration file if none exists. I'd
> be okay with that code adding 'deleted' and 'spam' to the excluded list.
>
> This would mean that an existing user would see no change without taking
> some action (adding the tags to the configuration file) and a new user
> would see the new behaviour (automatic exclusion).

What you describe is indeed how it currently works.  For new users or
old users who rerun setup, the config file will automatically include
the exclusions.  Otherwise, users will see no change.

> I'm not completely sure that automatically adding the exclusion of the
> specified tags via the configuration file for new users is a great
> idea. It seems as though it will lead to confusion for someone at some
> point.

Without any keys pre-bound to add "deleted" or "spam" tags, it probably
won't make much difference for new users.  And as long as it's
documented, users will be warned of the behavior.  Reading the config
file would also make it clear how the variable changes behavior.

jamie.

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: [PATCH v2 3/3] search: Support automatic tag exclusions

[David Edmondson]

[-- Attachment #1: Type: text/plain, Size: 1568 bytes --]

On Wed, 18 Jan 2012 00:52:09 -0800, Jameson Graef Rollins <jrollins@finestructure.net> wrote:
> On Wed, 18 Jan 2012 08:38:23 +0000, David Edmondson <dme@dme.org> wrote:
> > Something must create the initial configuration file if none exists. I'd
> > be okay with that code adding 'deleted' and 'spam' to the excluded list.
> >
> > This would mean that an existing user would see no change without taking
> > some action (adding the tags to the configuration file) and a new user
> > would see the new behaviour (automatic exclusion).
> 
> What you describe is indeed how it currently works.  For new users or
> old users who rerun setup, the config file will automatically include
> the exclusions.  Otherwise, users will see no change.

Good, thanks.

> > I'm not completely sure that automatically adding the exclusion of the
> > specified tags via the configuration file for new users is a great
> > idea. It seems as though it will lead to confusion for someone at some
> > point.
> 
> Without any keys pre-bound to add "deleted" or "spam" tags, it probably
> won't make much difference for new users.  And as long as it's
> documented, users will be warned of the behavior.  Reading the config
> file would also make it clear how the variable changes behavior.

I agree that as long as no keys are pre-bound it will make little
difference. That just transfers the discussion to the thread about
adding the bindings, which seems silly.

Anyway, I'm not too worried - I just won't answer any of the questions
about it on the list :-)

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: [PATCH v2 3/3] search: Support automatic tag exclusions

[Jameson Graef Rollins]

[-- Attachment #1: Type: text/plain, Size: 551 bytes --]

On Wed, 18 Jan 2012 09:52:52 +0000, David Edmondson <dme@dme.org> wrote:
> I agree that as long as no keys are pre-bound it will make little
> difference. That just transfers the discussion to the thread about
> adding the bindings, which seems silly.

I think that's ok.  The tag exclusion is in, which is great.  The
next question, of whether we should setup excludes by default, is
intimately related to whether or not we support key bindings to add
those tags.  So I think it's fine to transfer the rest of this
discussion to that thread.

jamie.

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

[PATCH] News for tag exclusion

[Austin Clements]

---
 NEWS |   15 +++++++++++++++
 1 files changed, 15 insertions(+), 0 deletions(-)

diff --git a/NEWS b/NEWS
index 1e561a9..6afa912 100644
--- a/NEWS
+++ b/NEWS
@@ -11,6 +11,13 @@ Reply to sender
   to all. The feature is available through the new command line option
   --reply-to=(all|sender).
 
+Tag exclusion
+
+  Tags can be automatically excluded from search results unless they
+  appear explicitly in a query.  By default, notmuch excludes the tags
+  deleted and spam.  This can be changed using the new config setting
+  search.auto_exclude_tags.
+
 Emacs Interface
 ---------------
 
@@ -21,6 +28,14 @@ Reply to sender
   and search modes, 'r' has been bound to reply to sender, replacing
   reply to all, which now has key binding 'R'.
 
+Library changes
+---------------
+
+New functions
+
+  notmuch_query_add_tag_exclude supports the new tag exclusion
+  feature.
+
 Notmuch 0.11 (2012-01-13)
 =========================
 
-- 
1.7.7.3

Re: [PATCH v3 2/2] search: Support automatic tag exclusions

[Pieter Praet]

Nice feature!  I won't be using it myself, but I can imagine it being
*very* useful for those who still feel the need to "delete" email :).


Nitpicking:

- All other config-related functions and args include the section title
  in their name [1], so for the sake of consistency, we might want to
  mirror that.  Also, the "auto"matic part is pretty much a given.

  So I'd like to suggest replacing all occurences of "auto_exclude_tags"
  with "search_exclude_tags" (and simply "exclude_tags" in the args to
  `_config_get_list' and `_config_set_list', of course).

  Unfortunately, this would also partially invalidate your recent NEWS
  submission [2].

- If the 'search.exclude_tags' option is missing from the config file,
  its value is automatically set to "deleted;spam;", which probably isn't
  a sane default.  Luckily, you've already provided the solution [3].

- To make new users aware of the config option's existence, we should
  prompt them to configure it during setup.

Patches follow.


Peace


[1] Eg. `notmuch_config_get_user_name', `notmuch_config_get_new_tags',
        `notmuch_config_get_maildir_synchronize_flags', ...

[2] id:"1326920330-31496-1-git-send-email-amdragon@mit.edu"

[3] id:"20120117203211.GQ16740@mit.edu"

[PATCH 1/4] search: rename auto_exclude_tags to {search, }exclude_tags

[Pieter Praet]

All other config-related functions and args include the section
title in their name, so for the sake of consistency, mirror that.

Also, the "auto"matic part is a given, so that was dropped.
---
 notmuch-client.h |    4 ++--
 notmuch-config.c |   32 ++++++++++++++++----------------
 notmuch-count.c  |   12 ++++++------
 notmuch-search.c |   12 ++++++------
 4 files changed, 30 insertions(+), 30 deletions(-)

diff --git a/notmuch-client.h b/notmuch-client.h
index 62ede28..4bc7320 100644
--- a/notmuch-client.h
+++ b/notmuch-client.h
@@ -236,10 +236,10 @@ notmuch_config_set_maildir_synchronize_flags (notmuch_config_t *config,
 					      notmuch_bool_t synchronize_flags);
 
 const char **
-notmuch_config_get_auto_exclude_tags (notmuch_config_t *config, size_t *length);
+notmuch_config_get_search_exclude_tags (notmuch_config_t *config, size_t *length);
 
 void
-notmuch_config_set_auto_exclude_tags (notmuch_config_t *config,
+notmuch_config_set_search_exclude_tags (notmuch_config_t *config,
 				      const char *list[],
 				      size_t length);
 
diff --git a/notmuch-config.c b/notmuch-config.c
index 3d4d5b9..687bd76 100644
--- a/notmuch-config.c
+++ b/notmuch-config.c
@@ -89,9 +89,9 @@ static const char search_config_comment[] =
     "\n"
     " The following option is supported here:\n"
     "\n"
-    "\tauto_exclude_tags      A ;-separated list of tags that will be\n"
-    "\t excluded from search results by default.  Using an excluded tag\n"
-    "\t in a query will override that exclusion.\n";
+    "\texclude_tags      A list (separated by ';') of the tags that will be\n"
+    "\t\texcluded from search results by default.  Using an excluded tag in a\n"
+    "\t\tquery will override that exclusion.\n";
 
 struct _notmuch_config {
     char *filename;
@@ -105,8 +105,8 @@ struct _notmuch_config {
     const char **new_tags;
     size_t new_tags_length;
     notmuch_bool_t maildir_synchronize_flags;
-    const char **auto_exclude_tags;
-    size_t auto_exclude_tags_length;
+    const char **search_exclude_tags;
+    size_t search_exclude_tags_length;
 };
 
 static int
@@ -264,8 +264,8 @@ notmuch_config_open (void *ctx,
     config->new_tags = NULL;
     config->new_tags_length = 0;
     config->maildir_synchronize_flags = TRUE;
-    config->auto_exclude_tags = NULL;
-    config->auto_exclude_tags_length = 0;
+    config->search_exclude_tags = NULL;
+    config->search_exclude_tags_length = 0;
 
     if (! g_key_file_load_from_file (config->key_file,
 				     config->filename,
@@ -360,9 +360,9 @@ notmuch_config_open (void *ctx,
 	notmuch_config_set_new_tags (config, tags, 2);
     }
 
-    if (notmuch_config_get_auto_exclude_tags (config, &tmp) == NULL) {
+    if (notmuch_config_get_search_exclude_tags (config, &tmp) == NULL) {
 	const char *tags[] = { "deleted", "spam" };
-	notmuch_config_set_auto_exclude_tags (config, tags, 2);
+	notmuch_config_set_search_exclude_tags (config, tags, 2);
     }
 
     error = NULL;
@@ -623,20 +623,20 @@ notmuch_config_set_new_tags (notmuch_config_t *config,
 }
 
 const char **
-notmuch_config_get_auto_exclude_tags (notmuch_config_t *config, size_t *length)
+notmuch_config_get_search_exclude_tags (notmuch_config_t *config, size_t *length)
 {
-    return _config_get_list (config, "search", "auto_exclude_tags",
-			     &(config->auto_exclude_tags),
-			     &(config->auto_exclude_tags_length), length);
+    return _config_get_list (config, "search", "exclude_tags",
+			     &(config->search_exclude_tags),
+			     &(config->search_exclude_tags_length), length);
 }
 
 void
-notmuch_config_set_auto_exclude_tags (notmuch_config_t *config,
+notmuch_config_set_search_exclude_tags (notmuch_config_t *config,
 				      const char *list[],
 				      size_t length)
 {
-    _config_set_list (config, "search", "auto_exclude_tags", list, length,
-		      &(config->auto_exclude_tags));
+    _config_set_list (config, "search", "exclude_tags", list, length,
+		      &(config->search_exclude_tags));
 }
 
 /* Given a configuration item of the form <group>.<key> return the
diff --git a/notmuch-count.c b/notmuch-count.c
index f77861e..63459fb 100644
--- a/notmuch-count.c
+++ b/notmuch-count.c
@@ -35,8 +35,8 @@ notmuch_count_command (void *ctx, int argc, char *argv[])
     char *query_str;
     int opt_index;
     int output = OUTPUT_MESSAGES;
-    const char **auto_exclude_tags;
-    size_t auto_exclude_tags_length;
+    const char **search_exclude_tags;
+    size_t search_exclude_tags_length;
     unsigned int i;
 
     notmuch_opt_desc_t options[] = {
@@ -78,10 +78,10 @@ notmuch_count_command (void *ctx, int argc, char *argv[])
 	return 1;
     }
 
-    auto_exclude_tags = notmuch_config_get_auto_exclude_tags
-	(config, &auto_exclude_tags_length);
-    for (i = 0; i < auto_exclude_tags_length; i++)
-	notmuch_query_add_tag_exclude (query, auto_exclude_tags[i]);
+    search_exclude_tags = notmuch_config_get_search_exclude_tags
+	(config, &search_exclude_tags_length);
+    for (i = 0; i < search_exclude_tags_length; i++)
+	notmuch_query_add_tag_exclude (query, search_exclude_tags[i]);
 
     switch (output) {
     case OUTPUT_MESSAGES:
diff --git a/notmuch-search.c b/notmuch-search.c
index 8867aab..d504051 100644
--- a/notmuch-search.c
+++ b/notmuch-search.c
@@ -423,8 +423,8 @@ notmuch_search_command (void *ctx, int argc, char *argv[])
     output_t output = OUTPUT_SUMMARY;
     int offset = 0;
     int limit = -1; /* unlimited */
-    const char **auto_exclude_tags;
-    size_t auto_exclude_tags_length;
+    const char **search_exclude_tags;
+    size_t search_exclude_tags_length;
     unsigned int i;
 
     enum { NOTMUCH_FORMAT_JSON, NOTMUCH_FORMAT_TEXT }
@@ -493,10 +493,10 @@ notmuch_search_command (void *ctx, int argc, char *argv[])
 
     notmuch_query_set_sort (query, sort);
 
-    auto_exclude_tags = notmuch_config_get_auto_exclude_tags
-	(config, &auto_exclude_tags_length);
-    for (i = 0; i < auto_exclude_tags_length; i++)
-	notmuch_query_add_tag_exclude (query, auto_exclude_tags[i]);
+    search_exclude_tags = notmuch_config_get_search_exclude_tags
+	(config, &search_exclude_tags_length);
+    for (i = 0; i < search_exclude_tags_length; i++)
+	notmuch_query_add_tag_exclude (query, search_exclude_tags[i]);
 
     switch (output) {
     default:
-- 
1.7.8.1

[PATCH 2/4] test: only exclude "deleted" messages from search if explicitly configured

[Pieter Praet]

If the 'search.exclude_tags' option is missing from the config file,
its value is automatically set to "deleted;spam;".  Taking PoLS/DWIM
into account, this should probably only happen during setup.

---
 test/search |    8 ++++++++
 1 files changed, 8 insertions(+), 0 deletions(-)

diff --git a/test/search b/test/search
index bf965e7..99d94bd 100755
--- a/test/search
+++ b/test/search
@@ -130,6 +130,7 @@ output=$(notmuch search "bödý" | notmuch_search_sanitize)
 test_expect_equal "$output" "thread:XXX   2000-01-01 [1/1] Notmuch Test Suite; utf8-message-body-subject (inbox unread)"
 
 test_begin_subtest "Exclude \"deleted\" messages from search"
+notmuch config set search.exclude_tags = deleted
 generate_message '[subject]="Not deleted"'
 generate_message '[subject]="Deleted"'
 notmuch new > /dev/null
@@ -147,4 +148,11 @@ output=$(notmuch search subject:deleted | notmuch_search_sanitize)
 test_expect_equal "$output" "thread:XXX   2001-01-05 [1/1] Notmuch Test Suite; Not deleted (inbox unread)
 thread:XXX   2001-01-05 [1/2] Notmuch Test Suite; Not deleted reply (deleted inbox unread)"
 
+test_begin_subtest "Don't exclude \"deleted\" messages from search if not configured"
+test_subtest_known_broken
+notmuch config set search.exclude_tags
+output=$(notmuch search subject:deleted | notmuch_search_sanitize)
+test_expect_equal "$output" "thread:XXX   2001-01-05 [1/1] Notmuch Test Suite; Not deleted (inbox unread)
+thread:XXX   2001-01-05 [2/2] Notmuch Test Suite; Deleted (deleted inbox unread)"
+
 test_done
-- 
1.7.8.1

[PATCH 3/4] config: only set search.exclude_tags to "deleted; spam; " during setup

[Pieter Praet]

If the 'search.exclude_tags' option is missing from the config file,
its value is automatically set to "deleted;spam;".  Taking PoLS/DWIM
into account, this should probably only happen during setup.

This patch is actually Austin Clements' work:
  id:"20120117203211.GQ16740@mit.edu"

---
 notmuch-config.c |    8 ++++++--
 test/search      |    1 -
 2 files changed, 6 insertions(+), 3 deletions(-)

diff --git a/notmuch-config.c b/notmuch-config.c
index 687bd76..1d9e842 100644
--- a/notmuch-config.c
+++ b/notmuch-config.c
@@ -361,8 +361,12 @@ notmuch_config_open (void *ctx,
     }
 
     if (notmuch_config_get_search_exclude_tags (config, &tmp) == NULL) {
-	const char *tags[] = { "deleted", "spam" };
-	notmuch_config_set_search_exclude_tags (config, tags, 2);
+	if (is_new) {
+	    const char *tags[] = { "deleted", "spam" };
+	    notmuch_config_set_search_exclude_tags (config, tags, 2);
+	} else {
+	    notmuch_config_set_search_exclude_tags (config, NULL, 0);
+	}
     }
 
     error = NULL;
diff --git a/test/search b/test/search
index 99d94bd..414be35 100755
--- a/test/search
+++ b/test/search
@@ -149,7 +149,6 @@ test_expect_equal "$output" "thread:XXX   2001-01-05 [1/1] Notmuch Test Suite; N
 thread:XXX   2001-01-05 [1/2] Notmuch Test Suite; Not deleted reply (deleted inbox unread)"
 
 test_begin_subtest "Don't exclude \"deleted\" messages from search if not configured"
-test_subtest_known_broken
 notmuch config set search.exclude_tags
 output=$(notmuch search subject:deleted | notmuch_search_sanitize)
 test_expect_equal "$output" "thread:XXX   2001-01-05 [1/1] Notmuch Test Suite; Not deleted (inbox unread)
-- 
1.7.8.1

[PATCH 4/4] setup: prompt user for search.exclude_tags value

[Pieter Praet]

Allow users to customize the search.exclude_tags option during setup.

---
 notmuch-setup.c |   36 ++++++++++++++++++++++++++++++++++++
 1 files changed, 36 insertions(+), 0 deletions(-)

diff --git a/notmuch-setup.c b/notmuch-setup.c
index c3ea937..44d4aaa 100644
--- a/notmuch-setup.c
+++ b/notmuch-setup.c
@@ -101,6 +101,8 @@ notmuch_setup_command (unused (void *ctx),
     int is_new;
     const char **new_tags;
     size_t new_tags_len;
+    const char **search_exclude_tags;
+    size_t search_exclude_tags_len;
 
 #define prompt(format, ...)					\
     do {							\
@@ -195,6 +197,40 @@ notmuch_setup_command (unused (void *ctx),
 	g_ptr_array_free (tags, TRUE);
     }
 
+    search_exclude_tags = notmuch_config_get_search_exclude_tags (config, &search_exclude_tags_len);
+
+    printf ("Tags to exclude when searching messages (separated by spaces) [");
+
+    for (i = 0; i < search_exclude_tags_len; i++) {
+	if (i != 0)
+	    printf (" ");
+	printf ("%s", search_exclude_tags[i]);
+    }
+
+    prompt ("]: ");
+
+    if (strlen (response)) {
+	GPtrArray *tags = g_ptr_array_new ();
+	char *tag = response;
+	char *space;
+
+	while (tag && *tag) {
+	    space = strchr (tag, ' ');
+	    if (space)
+		g_ptr_array_add (tags, talloc_strndup (ctx, tag, space - tag));
+	    else
+		g_ptr_array_add (tags, talloc_strdup (ctx, tag));
+	    tag = space;
+	    while (tag && *tag == ' ')
+		tag++;
+	}
+
+	notmuch_config_set_search_exclude_tags (config, (const char **) tags->pdata,
+				     tags->len);
+
+	g_ptr_array_free (tags, TRUE);
+    }
+
     if (! notmuch_config_save (config)) {
 	if (is_new)
 	  welcome_message_post_setup ();
-- 
1.7.8.1

Re: [PATCH v3 2/2] search: Support automatic tag exclusions

[Austin Clements]

Quoth Pieter Praet on Jan 19 at  8:19 pm:
> Nice feature!  I won't be using it myself, but I can imagine it being
> *very* useful for those who still feel the need to "delete" email :).

Same here.  I probably will use the spam tag, though.

> Nitpicking:
> 
> - All other config-related functions and args include the section title
>   in their name [1], so for the sake of consistency, we might want to
>   mirror that.  Also, the "auto"matic part is pretty much a given.
> 
>   So I'd like to suggest replacing all occurences of "auto_exclude_tags"
>   with "search_exclude_tags" (and simply "exclude_tags" in the args to
>   `_config_get_list' and `_config_set_list', of course).

You are technically correct, the best kind of correct.  I'd completely
missed this pattern.  This should get fixed ASAP, while this feature
still has limited adoption.

>   Unfortunately, this would also partially invalidate your recent NEWS
>   submission [2].

No worries, though maybe you want to tack an updated version of that
patch on the end of your series?

> - If the 'search.exclude_tags' option is missing from the config file,
>   its value is automatically set to "deleted;spam;", which probably isn't
>   a sane default.  Luckily, you've already provided the solution [3].

I'm good either way.  I got lost in the discussion of defaults but
Jamie assured me everything was okay, so I took the path of least
resistance and left things as they were.

> - To make new users aware of the config option's existence, we should
>   prompt them to configure it during setup.

Sure.

> Patches follow.
> 
> 
> Peace
> 
> 
> [1] Eg. `notmuch_config_get_user_name', `notmuch_config_get_new_tags',
>         `notmuch_config_get_maildir_synchronize_flags', ...
> 
> [2] id:"1326920330-31496-1-git-send-email-amdragon@mit.edu"
> 
> [3] id:"20120117203211.GQ16740@mit.edu"
> 

Re: [PATCH 1/4] search: rename auto_exclude_tags to {search,}exclude_tags

[Austin Clements]

Quoth Pieter Praet on Jan 19 at  8:19 pm:
> All other config-related functions and args include the section
> title in their name, so for the sake of consistency, mirror that.
> 
> Also, the "auto"matic part is a given, so that was dropped.

LGTM other than one nit, below.

> --- a/notmuch-config.c
> +++ b/notmuch-config.c
> @@ -89,9 +89,9 @@ static const char search_config_comment[] =
>      "\n"
>      " The following option is supported here:\n"
>      "\n"
> -    "\tauto_exclude_tags      A ;-separated list of tags that will be\n"
> -    "\t excluded from search results by default.  Using an excluded tag\n"
> -    "\t in a query will override that exclusion.\n";
> +    "\texclude_tags      A list (separated by ';') of the tags that will be\n"
> +    "\t\texcluded from search results by default.  Using an excluded tag in a\n"
> +    "\t\tquery will override that exclusion.\n";

I'd propose that any tag whose name is more than seven characters long
should have its description start under it, indented by a tab, like I
did in id:"1326920205-31296-1-git-send-email-amdragon@mit.edu".  That
looks nice and keeps the description indented consistently without
ridiculous amounts of negative space and narrow wrapping.

Re: [PATCH 4/4] setup: prompt user for search.exclude_tags value

[Austin Clements]

Quoth Pieter Praet on Jan 19 at  8:19 pm:
> Allow users to customize the search.exclude_tags option during setup.
> 
> ---
>  notmuch-setup.c |   36 ++++++++++++++++++++++++++++++++++++
>  1 files changed, 36 insertions(+), 0 deletions(-)
> 
> diff --git a/notmuch-setup.c b/notmuch-setup.c
> index c3ea937..44d4aaa 100644
> --- a/notmuch-setup.c
> +++ b/notmuch-setup.c
> @@ -101,6 +101,8 @@ notmuch_setup_command (unused (void *ctx),
>      int is_new;
>      const char **new_tags;
>      size_t new_tags_len;
> +    const char **search_exclude_tags;
> +    size_t search_exclude_tags_len;
>  
>  #define prompt(format, ...)					\
>      do {							\
> @@ -195,6 +197,40 @@ notmuch_setup_command (unused (void *ctx),
>  	g_ptr_array_free (tags, TRUE);
>      }
>  
> +    search_exclude_tags = notmuch_config_get_search_exclude_tags (config, &search_exclude_tags_len);
> +
> +    printf ("Tags to exclude when searching messages (separated by spaces) [");
> +
> +    for (i = 0; i < search_exclude_tags_len; i++) {
> +	if (i != 0)
> +	    printf (" ");
> +	printf ("%s", search_exclude_tags[i]);
> +    }
> +
> +    prompt ("]: ");
> +
> +    if (strlen (response)) {
> +	GPtrArray *tags = g_ptr_array_new ();
> +	char *tag = response;
> +	char *space;
> +
> +	while (tag && *tag) {
> +	    space = strchr (tag, ' ');
> +	    if (space)
> +		g_ptr_array_add (tags, talloc_strndup (ctx, tag, space - tag));
> +	    else
> +		g_ptr_array_add (tags, talloc_strdup (ctx, tag));
> +	    tag = space;
> +	    while (tag && *tag == ' ')
> +		tag++;
> +	}
> +
> +	notmuch_config_set_search_exclude_tags (config, (const char **) tags->pdata,
> +				     tags->len);
> +
> +	g_ptr_array_free (tags, TRUE);
> +    }
> +

Holy code duplication.  Can we move most of this (at least the
response parsing part and maybe the prompt printing) into a function
and use it for both new tags and exclude tags?

>      if (! notmuch_config_save (config)) {
>  	if (is_new)
>  	  welcome_message_post_setup ();

Re: [PATCH v3 2/2] search: Support automatic tag exclusions

[markwalters1009]

> >   So I'd like to suggest replacing all occurences of "auto_exclude_tags"
> >   with "search_exclude_tags" (and simply "exclude_tags" in the args to
> >   `_config_get_list' and `_config_set_list', of course).
> 
> You are technically correct, the best kind of correct.  I'd completely
> missed this pattern.  This should get fixed ASAP, while this feature
> still has limited adoption.

I would actually like make a different suggestion: extend
auto_exclude_tags to notmuch-show as well. I was quite surprised to see
my deleted (ie hidden rather than actually deleted) messages return when
viewing a thread.

Best wishes

Mark

Re: [PATCH v3 2/2] search: Support automatic tag exclusions

[Aaron Ecay]

On Thu, 19 Jan 2012 20:06:27 +0000, markwalters1009@gmail.com wrote:
> I would actually like make a different suggestion: extend
> auto_exclude_tags to notmuch-show as well. I was quite surprised to see
> my deleted (ie hidden rather than actually deleted) messages return when
> viewing a thread.

It might be best to show the messages in the thread – after all, you
might want to “undelete” (or “unspam”) those messages in light of later
replies to them.  (And if you are sure you really want them gone, you
should be deleting them or moving them out of your maildir periodically,
rather than relying on notmuch to not show them to you ever.)

(Perhaps another application of this functionality would be to “mute”
certain threads on a mailing list.  Then you could use a query for
“word-i-am-interested-in AND tag:muted” to find out if something
interesting pops up in a muted thread.  Then in the show view, you will
want to unmute the other thread messages.)

Perhaps this could be an application of dme’s header-line collapsing
functionality in id:"cun8vl5m610.fsf@hotblack-desiato.hh.sledj.net"
(suitably cleaned up), such that you see only a single line in the show
buffer that says:
----- 6 messages hidden -----
instead of the 6 (or however many) individual messages.

-- 
Aaron Ecay

Re: [PATCH v3 2/2] search: Support automatic tag exclusions

[Mark Walters]

> On Thu, 19 Jan 2012 20:06:27 +0000, markwalters1009@gmail.com wrote:
> > I would actually like make a different suggestion: extend
> > auto_exclude_tags to notmuch-show as well. I was quite surprised to see
> > my deleted (ie hidden rather than actually deleted) messages return when
> > viewing a thread.
> 
> It might be best to show the messages in the thread – after all, you
> might want to “undelete” (or “unspam”) those messages in light of later
> replies to them.  (And if you are sure you really want them gone, you
> should be deleting them or moving them out of your maildir periodically,
> rather than relying on notmuch to not show them to you ever.)

I am happy with them appearing as a non matching message, but currently
they appear as a full open message. (The patch to achieve this is
trivial: essentially transpose part of Austin's patch of
notmuch-search.c into notmuch-show.c)

Best wishes

Mark

Re: [PATCH v3 2/2] search: Support automatic tag exclusions

[Austin Clements]

Quoth Mark Walters on Jan 19 at  8:23 pm:
> 
> > On Thu, 19 Jan 2012 20:06:27 +0000, markwalters1009@gmail.com wrote:
> > > I would actually like make a different suggestion: extend
> > > auto_exclude_tags to notmuch-show as well. I was quite surprised to see
> > > my deleted (ie hidden rather than actually deleted) messages return when
> > > viewing a thread.
> > 
> > It might be best to show the messages in the thread – after all, you
> > might want to “undelete” (or “unspam”) those messages in light of later
> > replies to them.  (And if you are sure you really want them gone, you
> > should be deleting them or moving them out of your maildir periodically,
> > rather than relying on notmuch to not show them to you ever.)
> 
> I am happy with them appearing as a non matching message, but currently
> they appear as a full open message. (The patch to achieve this is
> trivial: essentially transpose part of Austin's patch of
> notmuch-search.c into notmuch-show.c)

This definitely sounds like the right thing to do.  We can argue about
more sophisticated UIs, but it should do this at a minimum.

Re: [PATCH 1/4] search: rename auto_exclude_tags to {search, }exclude_tags

[Pieter Praet]

On Thu, 19 Jan 2012 14:41:15 -0500, Austin Clements <amdragon@MIT.EDU> wrote:
> Quoth Pieter Praet on Jan 19 at  8:19 pm:
> > All other config-related functions and args include the section
> > title in their name, so for the sake of consistency, mirror that.
> > 
> > Also, the "auto"matic part is a given, so that was dropped.
> 
> LGTM other than one nit, below.
> 
> > --- a/notmuch-config.c
> > +++ b/notmuch-config.c
> > @@ -89,9 +89,9 @@ static const char search_config_comment[] =
> >      "\n"
> >      " The following option is supported here:\n"
> >      "\n"
> > -    "\tauto_exclude_tags      A ;-separated list of tags that will be\n"
> > -    "\t excluded from search results by default.  Using an excluded tag\n"
> > -    "\t in a query will override that exclusion.\n";
> > +    "\texclude_tags      A list (separated by ';') of the tags that will be\n"
> > +    "\t\texcluded from search results by default.  Using an excluded tag in a\n"
> > +    "\t\tquery will override that exclusion.\n";
> 
> I'd propose that any tag whose name is more than seven characters long
> should have its description start under it, indented by a tab, like I
> did in id:"1326920205-31296-1-git-send-email-amdragon@mit.edu".  That
> looks nice and keeps the description indented consistently without
> ridiculous amounts of negative space and narrow wrapping.

Oh, nice.  Looks like I should first work down to inbox zero
(still got a looong way to go :/...) before submitting patches.

I'll wait for yours to be pushed before resubmitting this series.


Peace

-- 
Pieter

Re: [PATCH 4/4] setup: prompt user for search.exclude_tags value

[Pieter Praet]

On Thu, 19 Jan 2012 14:44:37 -0500, Austin Clements <amdragon@MIT.EDU> wrote:
> Quoth Pieter Praet on Jan 19 at  8:19 pm:
> > Allow users to customize the search.exclude_tags option during setup.
> > 
> > ---
> >  notmuch-setup.c |   36 ++++++++++++++++++++++++++++++++++++
> >  1 files changed, 36 insertions(+), 0 deletions(-)
> > 
> > diff --git a/notmuch-setup.c b/notmuch-setup.c
> > index c3ea937..44d4aaa 100644
> > --- a/notmuch-setup.c
> > +++ b/notmuch-setup.c
> > @@ -101,6 +101,8 @@ notmuch_setup_command (unused (void *ctx),
> >      int is_new;
> >      const char **new_tags;
> >      size_t new_tags_len;
> > +    const char **search_exclude_tags;
> > +    size_t search_exclude_tags_len;
> >  
> >  #define prompt(format, ...)					\
> >      do {							\
> > @@ -195,6 +197,40 @@ notmuch_setup_command (unused (void *ctx),
> >  	g_ptr_array_free (tags, TRUE);
> >      }
> >  
> > +    search_exclude_tags = notmuch_config_get_search_exclude_tags (config, &search_exclude_tags_len);
> > +
> > +    printf ("Tags to exclude when searching messages (separated by spaces) [");
> > +
> > +    for (i = 0; i < search_exclude_tags_len; i++) {
> > +	if (i != 0)
> > +	    printf (" ");
> > +	printf ("%s", search_exclude_tags[i]);
> > +    }
> > +
> > +    prompt ("]: ");
> > +
> > +    if (strlen (response)) {
> > +	GPtrArray *tags = g_ptr_array_new ();
> > +	char *tag = response;
> > +	char *space;
> > +
> > +	while (tag && *tag) {
> > +	    space = strchr (tag, ' ');
> > +	    if (space)
> > +		g_ptr_array_add (tags, talloc_strndup (ctx, tag, space - tag));
> > +	    else
> > +		g_ptr_array_add (tags, talloc_strdup (ctx, tag));
> > +	    tag = space;
> > +	    while (tag && *tag == ' ')
> > +		tag++;
> > +	}
> > +
> > +	notmuch_config_set_search_exclude_tags (config, (const char **) tags->pdata,
> > +				     tags->len);
> > +
> > +	g_ptr_array_free (tags, TRUE);
> > +    }
> > +
> 
> Holy code duplication.  Can we move most of this (at least the
> response parsing part and maybe the prompt printing) into a function
> and use it for both new tags and exclude tags?
> 

Depends on who "we" is... :)  I would gladly do it if I could, but I
think this uber1337 copy-paste coding job serves as *very* convincing
proof that I'm pretty much in the dark (with a single wet match) when it
comes to C :)


> >      if (! notmuch_config_save (config)) {
> >  	if (is_new)
> >  	  welcome_message_post_setup ();


Peace

-- 
Pieter

Re: [PATCH v3 2/2] search: Support automatic tag exclusions

[Pieter Praet]

On Thu, 19 Jan 2012 14:36:47 -0500, Austin Clements <amdragon@MIT.EDU> wrote:
> Quoth Pieter Praet on Jan 19 at  8:19 pm:
> > Nice feature!  I won't be using it myself, but I can imagine it being
> > *very* useful for those who still feel the need to "delete" email :).
> 
> Same here.  I probably will use the spam tag, though.
> 

Really?  Then how will you be able to reply to my emails?!?  ;)


> > Nitpicking:
> > 
> > - All other config-related functions and args include the section title
> >   in their name [1], so for the sake of consistency, we might want to
> >   mirror that.  Also, the "auto"matic part is pretty much a given.
> > 
> >   So I'd like to suggest replacing all occurences of "auto_exclude_tags"
> >   with "search_exclude_tags" (and simply "exclude_tags" in the args to
> >   `_config_get_list' and `_config_set_list', of course).
> 
> You are technically correct, the best kind of correct.  I'd completely
> missed this pattern.  This should get fixed ASAP, while this feature
> still has limited adoption.
> 
> >   Unfortunately, this would also partially invalidate your recent NEWS
> >   submission [2].
> 
> No worries, though maybe you want to tack an updated version of that
> patch on the end of your series?
> 

Will do.


> > - If the 'search.exclude_tags' option is missing from the config file,
> >   its value is automatically set to "deleted;spam;", which probably isn't
> >   a sane default.  Luckily, you've already provided the solution [3].
> 
> I'm good either way.  I got lost in the discussion of defaults but
> Jamie assured me everything was okay, so I took the path of least
> resistance and left things as they were.
> 

Hmm.  I probably haven't read everything pertaining to that issue yet,
but IMO it violates the Principle of Least Surprise with a vengeance.

Tagging stuff some way or another shouldn't make it disappear by default.

What if "spam" means "read it when you're bored", or "deleted" means
"processed request for user deletion" (pretty far-fetched, but still...)
for someone?


> > - To make new users aware of the config option's existence, we should
> >   prompt them to configure it during setup.
> 
> Sure.
> 
> > Patches follow.
> > 
> > 
> > Peace
> > 
> > 
> > [1] Eg. `notmuch_config_get_user_name', `notmuch_config_get_new_tags',
> >         `notmuch_config_get_maildir_synchronize_flags', ...
> > 
> > [2] id:"1326920330-31496-1-git-send-email-amdragon@mit.edu"
> > 
> > [3] id:"20120117203211.GQ16740@mit.edu"
> > 


Peace

-- 
Pieter

Re: [PATCH v3 2/2] search: Support automatic tag exclusions

[Mark Walters]

> > I am happy with them appearing as a non matching message, but currently
> > they appear as a full open message. (The patch to achieve this is
> > trivial: essentially transpose part of Austin's patch of
> > notmuch-search.c into notmuch-show.c)
> 
> This definitely sounds like the right thing to do.  We can argue about
> more sophisticated UIs, but it should do this at a minimum.

I will post the trivial patch in a moment (trivial but I could easily
have done something stupid).

I haven't written a test yet: should it have one, and if so, could
someone point me to the appropriate place to put it?

Best wishes

Mark

[PATCH] Automatically exclude tags in notmuch-show

[Mark Walters]

Add the use of auto_exclude_tags in notmuch-show.c.  As with Austin's
patch (commit 42a907992823030f070fc395a174f779998ca6f5) it just adds
the excluded tags to the query so the excluded messages will still
appear in the emacs interface, but as a single header line rather than
as a matching message.
---
 notmuch-show.c |    8 ++++++++
 1 files changed, 8 insertions(+), 0 deletions(-)

diff --git a/notmuch-show.c b/notmuch-show.c
index d14dac9..925dfd6 100644
--- a/notmuch-show.c
+++ b/notmuch-show.c
@@ -948,9 +948,12 @@ notmuch_show_command (void *ctx, unused (int argc), unused (char *argv[]))
     char *opt;
     const notmuch_show_format_t *format = &format_text;
     notmuch_show_params_t params;
+    const char **auto_exclude_tags;
+    size_t auto_exclude_tags_length;
     int mbox = 0;
     int format_specified = 0;
     int i;
+    unsigned int j;
 
     params.entire_thread = 0;
     params.raw = 0;
@@ -1040,6 +1043,11 @@ notmuch_show_command (void *ctx, unused (int argc), unused (char *argv[]))
 	return 1;
     }
 
+    auto_exclude_tags = notmuch_config_get_auto_exclude_tags
+        (config, &auto_exclude_tags_length);
+    for (j = 0; j < auto_exclude_tags_length; j++)
+        notmuch_query_add_tag_exclude (query, auto_exclude_tags[j]);
+
     /* if part was requested and format was not specified, use format=raw */
     if (params.part >= 0 && !format_specified)
 	format = &format_raw;
-- 
1.7.2.3

Re: [PATCH v3 2/2] search: Support automatic tag exclusions

[Pieter Praet]

On Thu, 19 Jan 2012 22:01:27 +0000, Mark Walters <markwalters1009@gmail.com> wrote:
> 
> > > I am happy with them appearing as a non matching message, but currently
> > > they appear as a full open message. (The patch to achieve this is
> > > trivial: essentially transpose part of Austin's patch of
> > > notmuch-search.c into notmuch-show.c)
> > 
> > This definitely sounds like the right thing to do.  We can argue about
> > more sophisticated UIs, but it should do this at a minimum.
> 
> I will post the trivial patch in a moment (trivial but I could easily
> have done something stupid).
> 
> I haven't written a test yet: should it have one, [...]

That would be preferrable, yes.

> [...] and if so, could
> someone point me to the appropriate place to put it?
> 

In "test/emacs", right after "Hiding message with visible citation in
notmuch-show view" would be perfect IMO.

As you can see, that test uses `test-visible-output', which should
provide the functionality you need.

> Best wishes
> 
> Mark


Thanks!


Peace

-- 
Pieter

Re: [PATCH] Automatically exclude tags in notmuch-show

[Austin Clements]

LGTM, but should definitely come with a test.

Also, this won't commute with Pieter's patch
(id:"1327000744-25463-2-git-send-email-pieter@praet.org"), so one or
the other will have to get updated.

Quoth Mark Walters on Jan 19 at 10:03 pm:
> Add the use of auto_exclude_tags in notmuch-show.c.  As with Austin's
> patch (commit 42a907992823030f070fc395a174f779998ca6f5) it just adds
> the excluded tags to the query so the excluded messages will still
> appear in the emacs interface, but as a single header line rather than
> as a matching message.
> ---
>  notmuch-show.c |    8 ++++++++
>  1 files changed, 8 insertions(+), 0 deletions(-)
> 
> diff --git a/notmuch-show.c b/notmuch-show.c
> index d14dac9..925dfd6 100644
> --- a/notmuch-show.c
> +++ b/notmuch-show.c
> @@ -948,9 +948,12 @@ notmuch_show_command (void *ctx, unused (int argc), unused (char *argv[]))
>      char *opt;
>      const notmuch_show_format_t *format = &format_text;
>      notmuch_show_params_t params;
> +    const char **auto_exclude_tags;
> +    size_t auto_exclude_tags_length;
>      int mbox = 0;
>      int format_specified = 0;
>      int i;
> +    unsigned int j;

Hah.  The original patch series updated 'count' to use the new
argument parsing solely so I could steal 'i' for the tag exclude code.

>  
>      params.entire_thread = 0;
>      params.raw = 0;
> @@ -1040,6 +1043,11 @@ notmuch_show_command (void *ctx, unused (int argc), unused (char *argv[]))
>  	return 1;
>      }
>  
> +    auto_exclude_tags = notmuch_config_get_auto_exclude_tags
> +        (config, &auto_exclude_tags_length);
> +    for (j = 0; j < auto_exclude_tags_length; j++)
> +        notmuch_query_add_tag_exclude (query, auto_exclude_tags[j]);
> +
>      /* if part was requested and format was not specified, use format=raw */
>      if (params.part >= 0 && !format_specified)
>  	format = &format_raw;

Re: [PATCH] Automatically exclude tags in notmuch-show

[Pieter Praet]

On Thu, 19 Jan 2012 17:59:10 -0500, Austin Clements <amdragon@MIT.EDU> wrote:
> LGTM, but should definitely come with a test.
> 

[...]

> Also, this won't commute with Pieter's patch
> (id:"1327000744-25463-2-git-send-email-pieter@praet.org"), so one or
> the other will have to get updated.
> 

No problem, I'll have to resubmit my entire series anyway,
so go right ahead.

> Quoth Mark Walters on Jan 19 at 10:03 pm:
> > Add the use of auto_exclude_tags in notmuch-show.c.  As with Austin's
> > patch (commit 42a907992823030f070fc395a174f779998ca6f5) it just adds
> > the excluded tags to the query so the excluded messages will still
> > appear in the emacs interface, but as a single header line rather than
> > as a matching message.
> > ---
> >  notmuch-show.c |    8 ++++++++
> >  1 files changed, 8 insertions(+), 0 deletions(-)
> > 
> > diff --git a/notmuch-show.c b/notmuch-show.c
> > index d14dac9..925dfd6 100644
> > --- a/notmuch-show.c
> > +++ b/notmuch-show.c
> > @@ -948,9 +948,12 @@ notmuch_show_command (void *ctx, unused (int argc), unused (char *argv[]))
> >      char *opt;
> >      const notmuch_show_format_t *format = &format_text;
> >      notmuch_show_params_t params;
> > +    const char **auto_exclude_tags;
> > +    size_t auto_exclude_tags_length;
> >      int mbox = 0;
> >      int format_specified = 0;
> >      int i;
> > +    unsigned int j;
> 
> Hah.  The original patch series updated 'count' to use the new
> argument parsing solely so I could steal 'i' for the tag exclude code.
> 
> >  
> >      params.entire_thread = 0;
> >      params.raw = 0;
> > @@ -1040,6 +1043,11 @@ notmuch_show_command (void *ctx, unused (int argc), unused (char *argv[]))
> >  	return 1;
> >      }
> >  
> > +    auto_exclude_tags = notmuch_config_get_auto_exclude_tags
> > +        (config, &auto_exclude_tags_length);
> > +    for (j = 0; j < auto_exclude_tags_length; j++)
> > +        notmuch_query_add_tag_exclude (query, auto_exclude_tags[j]);
> > +
> >      /* if part was requested and format was not specified, use format=raw */
> >      if (params.part >= 0 && !format_specified)
> >  	format = &format_raw;
> _______________________________________________
> notmuch mailing list
> notmuch@notmuchmail.org
> http://notmuchmail.org/mailman/listinfo/notmuch


Peace

-- 
Pieter

Re: [PATCH] Automatically exclude tags in notmuch-show

[Mark Walters]

Ok Having said this is trivial I have found a problem. What should
notmuch do if you do something like

notmuch show id:<some-id>
and that message is marked with a deleted tag? To be consistent with the
other cases (where a deleted message is in a matched thread) we might
want to return the message with the not-matched flag set (eg in
JSON). But my patch doesn't, as it never even sees the thread since it
doesn't match.

Looking at notmuch-show.c I think we should not apply the exclude tags
to do_show_single, but usually should apply it to do_show. One solution
which is simple and is at least close to right would be to get do_show
to return the number of threads found. If this is zero then retry the
query without the excludes (possible setting the match_flag to zero on
each message since we know it does not match)

This is not a completely correct solution as if you ask notmuch-show to
show more than one thread it might  threads which only contain deleted
messages.

I can't see other good possibilities without slowing down the normal
path a lot (eg find all threads that match the original query and then
apply the argument above).

Any thoughts?

Incidentally, is there something strange at the end of notmuch-show.c: I
can't see how we could ever reach the last half dozen lines.

Best wishes

Mark

Re: [PATCH 4/4] setup: prompt user for search.exclude_tags value

[Austin Clements]

Quoth Pieter Praet on Jan 19 at 10:16 pm:
> On Thu, 19 Jan 2012 14:44:37 -0500, Austin Clements <amdragon@MIT.EDU> wrote:
> > Quoth Pieter Praet on Jan 19 at  8:19 pm:
> > > Allow users to customize the search.exclude_tags option during setup.
> > > 
> > > ---
> > >  notmuch-setup.c |   36 ++++++++++++++++++++++++++++++++++++
> > >  1 files changed, 36 insertions(+), 0 deletions(-)
> > > 
> > > diff --git a/notmuch-setup.c b/notmuch-setup.c
> > > index c3ea937..44d4aaa 100644
> > > --- a/notmuch-setup.c
> > > +++ b/notmuch-setup.c
> > > @@ -101,6 +101,8 @@ notmuch_setup_command (unused (void *ctx),
> > >      int is_new;
> > >      const char **new_tags;
> > >      size_t new_tags_len;
> > > +    const char **search_exclude_tags;
> > > +    size_t search_exclude_tags_len;
> > >  
> > >  #define prompt(format, ...)					\
> > >      do {							\
> > > @@ -195,6 +197,40 @@ notmuch_setup_command (unused (void *ctx),
> > >  	g_ptr_array_free (tags, TRUE);
> > >      }
> > >  
> > > +    search_exclude_tags = notmuch_config_get_search_exclude_tags (config, &search_exclude_tags_len);
> > > +
> > > +    printf ("Tags to exclude when searching messages (separated by spaces) [");
> > > +
> > > +    for (i = 0; i < search_exclude_tags_len; i++) {
> > > +	if (i != 0)
> > > +	    printf (" ");
> > > +	printf ("%s", search_exclude_tags[i]);
> > > +    }
> > > +
> > > +    prompt ("]: ");
> > > +
> > > +    if (strlen (response)) {
> > > +	GPtrArray *tags = g_ptr_array_new ();
> > > +	char *tag = response;
> > > +	char *space;
> > > +
> > > +	while (tag && *tag) {
> > > +	    space = strchr (tag, ' ');
> > > +	    if (space)
> > > +		g_ptr_array_add (tags, talloc_strndup (ctx, tag, space - tag));
> > > +	    else
> > > +		g_ptr_array_add (tags, talloc_strdup (ctx, tag));
> > > +	    tag = space;
> > > +	    while (tag && *tag == ' ')
> > > +		tag++;
> > > +	}
> > > +
> > > +	notmuch_config_set_search_exclude_tags (config, (const char **) tags->pdata,
> > > +				     tags->len);
> > > +
> > > +	g_ptr_array_free (tags, TRUE);
> > > +    }
> > > +
> > 
> > Holy code duplication.  Can we move most of this (at least the
> > response parsing part and maybe the prompt printing) into a function
> > and use it for both new tags and exclude tags?
> > 
> 
> Depends on who "we" is... :)  I would gladly do it if I could, but I
> think this uber1337 copy-paste coding job serves as *very* convincing
> proof that I'm pretty much in the dark (with a single wet match) when it
> comes to C :)

Hah, okay.  How about this as an initial minor refactoring of the code
for new.tags?

diff --git a/notmuch-setup.c b/notmuch-setup.c
index c3ea937..dcfa607 100644
--- a/notmuch-setup.c
+++ b/notmuch-setup.c
@@ -87,6 +87,38 @@ welcome_message_post_setup (void)
 "have sufficient storage space available now.\n\n");
 }
 
+static void
+print_tag_list (const char **tags, size_t tags_len)
+{
+    unsigned int i;
+    for (i = 0; i < tags_len; i++) {
+	if (i != 0)
+	    printf (" ");
+	printf ("%s", tags[i]);
+    }
+}
+
+static GPtrArray *
+parse_tag_list (void *ctx, char *response)
+{
+    GPtrArray *tags = g_ptr_array_new ();
+    char *tag = response;
+    char *space;
+
+    while (tag && *tag) {
+	space = strchr (tag, ' ');
+	if (space)
+	    g_ptr_array_add (tags, talloc_strndup (ctx, tag, space - tag));
+	else
+	    g_ptr_array_add (tags, talloc_strdup (ctx, tag));
+	tag = space;
+	while (tag && *tag == ' ')
+	    tag++;
+    }
+
+    return tags;
+}
+
 int
 notmuch_setup_command (unused (void *ctx),
 		       unused (int argc), unused (char *argv[]))
@@ -164,30 +196,11 @@ notmuch_setup_command (unused (void *ctx),
     new_tags = notmuch_config_get_new_tags (config, &new_tags_len);
 
     printf ("Tags to apply to all new messages (separated by spaces) [");
-
-    for (i = 0; i < new_tags_len; i++) {
-	if (i != 0)
-	    printf (" ");
-	printf ("%s", new_tags[i]);
-    }
-
+    print_tag_list(new_tags, new_tags_len);
     prompt ("]: ");
 
     if (strlen (response)) {
-	GPtrArray *tags = g_ptr_array_new ();
-	char *tag = response;
-	char *space;
-
-	while (tag && *tag) {
-	    space = strchr (tag, ' ');
-	    if (space)
-		g_ptr_array_add (tags, talloc_strndup (ctx, tag, space - tag));
-	    else
-		g_ptr_array_add (tags, talloc_strdup (ctx, tag));
-	    tag = space;
-	    while (tag && *tag == ' ')
-		tag++;
-	}
+	GPtrArray *tags = parse_tag_list (ctx, response);
 
 	notmuch_config_set_new_tags (config, (const char **) tags->pdata,
 				     tags->len);

Re: [PATCH] Automatically exclude tags in notmuch-show

[Austin Clements]

Quoth Mark Walters on Jan 20 at 12:10 am:
> 
> Ok Having said this is trivial I have found a problem. What should
> notmuch do if you do something like
> 
> notmuch show id:<some-id>
> and that message is marked with a deleted tag? To be consistent with the
> other cases (where a deleted message is in a matched thread) we might
> want to return the message with the not-matched flag set (eg in
> JSON). But my patch doesn't, as it never even sees the thread since it
> doesn't match.
> 
> Looking at notmuch-show.c I think we should not apply the exclude tags
> to do_show_single, but usually should apply it to do_show. One solution
> which is simple and is at least close to right would be to get do_show
> to return the number of threads found. If this is zero then retry the
> query without the excludes (possible setting the match_flag to zero on
> each message since we know it does not match)
> 
> This is not a completely correct solution as if you ask notmuch-show to
> show more than one thread it might  threads which only contain deleted
> messages.
> 
> I can't see other good possibilities without slowing down the normal
> path a lot (eg find all threads that match the original query and then
> apply the argument above).
> 
> Any thoughts?

Oh dear.

Well, here's one idea.  Instead of doing a single thread query in
show, do a thread query without the exclusions and then a message
query with the exclusions.  Output all of the messages from the first
query, but use the results of the second query to determine which
messages are "matched".  The same could be accomplished in the library
somewhat more efficiently, but it's not obvious to me what the API
would be.

> Incidentally, is there something strange at the end of notmuch-show.c: I
> can't see how we could ever reach the last half dozen lines.

Yes, I've wondered about that before, too.  I think none of those
technically matter since they're all cleaning up resources that the OS
is about to clean up for us.  It would be a problem if the database
was open in write mode because Xapian's write lock hangs around for a
split second after the process terminates if you don't close the
database yourself, but in read mode it doesn't take any locks.  Not
that this excuses the code.

Re: [PATCH] Automatically exclude tags in notmuch-show

[Mark Walters]

On Fri, 20 Jan 2012 12:18:01 -0500, Austin Clements <amdragon@MIT.EDU> wrote:
> 
> Oh dear.
> 
> Well, here's one idea.  Instead of doing a single thread query in
> show, do a thread query without the exclusions and then a message
> query with the exclusions.  Output all of the messages from the first
> query, but use the results of the second query to determine which
> messages are "matched".  The same could be accomplished in the library
> somewhat more efficiently, but it's not obvious to me what the API
> would be.

I have been thinking about this and one question is what should the sort
order be? If I understand it correctly notmuch sorts the threads
by the oldest/newest matching message, so the "correct" behaviour if no
message matches is unclear. Perhaps all threads with a matching
non-excluded message sorted by the matching-non-excluded message
followed by all threads that match only on excluded messages with sort
based on the matching excluded message?

Best wishes

Mark

Re: [PATCH 4/4] setup: prompt user for search.exclude_tags value

[Pieter Praet]

On Thu, 19 Jan 2012 23:19:02 -0500, Austin Clements <amdragon@MIT.EDU> wrote:
> [...]
> Hah, okay.  How about this as an initial minor refactoring of the code
> for new.tags?
> [...]

Great, thanks!


Would the following commit message be satisfactory? :

  #+begin_quote
    setup: move tag printing and parsing into separate functions

    * notmuch-setup.c (notmuch_setup_command):
      Break tag printing and response parsing out into separate functions
      called `print_tag_list' respectively `parse_tag_list', for reuse
      with the 'search.exclude_tags' option.
  #+end_quote

(if not, please suggest an alternative; it'll be your name at the top)


Thanks again!


Peace

-- 
Pieter

Re: [PATCH 4/4] setup: prompt user for search.exclude_tags value

[Austin Clements]

Quoth Pieter Praet on Jan 22 at  7:55 am:
> On Thu, 19 Jan 2012 23:19:02 -0500, Austin Clements <amdragon@MIT.EDU> wrote:
> > [...]
> > Hah, okay.  How about this as an initial minor refactoring of the code
> > for new.tags?
> > [...]
> 
> Great, thanks!
> 
> 
> Would the following commit message be satisfactory? :
> 
>   #+begin_quote
>     setup: move tag printing and parsing into separate functions
> 
>     * notmuch-setup.c (notmuch_setup_command):
>       Break tag printing and response parsing out into separate functions
>       called `print_tag_list' respectively `parse_tag_list', for reuse
>       with the 'search.exclude_tags' option.
>   #+end_quote
> 
> (if not, please suggest an alternative; it'll be your name at the top)

Sure.

Re: [PATCH] Automatically exclude tags in notmuch-show

[Austin Clements]

Quoth Mark Walters on Jan 22 at 12:38 am:
> 
> On Fri, 20 Jan 2012 12:18:01 -0500, Austin Clements <amdragon@MIT.EDU> wrote:
> > 
> > Oh dear.
> > 
> > Well, here's one idea.  Instead of doing a single thread query in
> > show, do a thread query without the exclusions and then a message
> > query with the exclusions.  Output all of the messages from the first
> > query, but use the results of the second query to determine which
> > messages are "matched".  The same could be accomplished in the library
> > somewhat more efficiently, but it's not obvious to me what the API
> > would be.
> 
> I have been thinking about this and one question is what should the sort
> order be? If I understand it correctly notmuch sorts the threads
> by the oldest/newest matching message, so the "correct" behaviour if no
> message matches is unclear. Perhaps all threads with a matching
> non-excluded message sorted by the matching-non-excluded message
> followed by all threads that match only on excluded messages with sort
> based on the matching excluded message?

I don't think show sorts in any particular way.  Or are you saying
that search also needs to know the difference between excluded and
non-excluded matched messages?

Re: [PATCH] Automatically exclude tags in notmuch-show

[Austin Clements]

Quoth myself on Jan 20 at 12:18 pm:
> Quoth Mark Walters on Jan 20 at 12:10 am:
> > 
> > Ok Having said this is trivial I have found a problem. What should
> > notmuch do if you do something like
> > 
> > notmuch show id:<some-id>
> > and that message is marked with a deleted tag? To be consistent with the
> > other cases (where a deleted message is in a matched thread) we might
> > want to return the message with the not-matched flag set (eg in
> > JSON). But my patch doesn't, as it never even sees the thread since it
> > doesn't match.
> > 
> > Looking at notmuch-show.c I think we should not apply the exclude tags
> > to do_show_single, but usually should apply it to do_show. One solution
> > which is simple and is at least close to right would be to get do_show
> > to return the number of threads found. If this is zero then retry the
> > query without the excludes (possible setting the match_flag to zero on
> > each message since we know it does not match)
> > 
> > This is not a completely correct solution as if you ask notmuch-show to
> > show more than one thread it might  threads which only contain deleted
> > messages.
> > 
> > I can't see other good possibilities without slowing down the normal
> > path a lot (eg find all threads that match the original query and then
> > apply the argument above).
> > 
> > Any thoughts?
> 
> Oh dear.
> 
> Well, here's one idea.  Instead of doing a single thread query in
> show, do a thread query without the exclusions and then a message
> query with the exclusions.  Output all of the messages from the first
> query, but use the results of the second query to determine which
> messages are "matched".  The same could be accomplished in the library
> somewhat more efficiently, but it's not obvious to me what the API
> would be.

Here's a slightly crazier idea that's more library-invasive than the
original approach, but probably better in the long run.

Have notmuch_query_search_* return everything and make exclusion a
message flag like NOTMUCH_MESSAGE_FLAG_MATCH.  Tweak the definition of
"matched" to mean "matched and not excluded" (specifically, a message
would have the match flag or the excluded flag or neither, but not
both).  Search would skip threads with zero matched messages and I
think show would Just Work.

I can think of two ways to implement this.  notmuch_query_search_*
could perform both the original query and the query with exclusions
and use the docid set from the second to compute the "excluded"
message flag.  Alternatively, it could examine the tags of each
message directly to compute the flag.  The latter is probably easier
to implement, but probably slower.

Thoughts?

Re: [PATCH] Automatically exclude tags in notmuch-show

[Mark Walters]

On Sun, 22 Jan 2012 13:16:09 -0500, Austin Clements <amdragon@MIT.EDU> wrote:
> Quoth myself on Jan 20 at 12:18 pm:
> > Quoth Mark Walters on Jan 20 at 12:10 am:
> > > 
> > > Ok Having said this is trivial I have found a problem. What should
> > > notmuch do if you do something like
> > > 
> > > notmuch show id:<some-id>
> > > and that message is marked with a deleted tag? To be consistent with the
> > > other cases (where a deleted message is in a matched thread) we might
> > > want to return the message with the not-matched flag set (eg in
> > > JSON). But my patch doesn't, as it never even sees the thread since it
> > > doesn't match.
> > > 
> > > Looking at notmuch-show.c I think we should not apply the exclude tags
> > > to do_show_single, but usually should apply it to do_show. One solution
> > > which is simple and is at least close to right would be to get do_show
> > > to return the number of threads found. If this is zero then retry the
> > > query without the excludes (possible setting the match_flag to zero on
> > > each message since we know it does not match)
> > > 
> > > This is not a completely correct solution as if you ask notmuch-show to
> > > show more than one thread it might  threads which only contain deleted
> > > messages.
> > > 
> > > I can't see other good possibilities without slowing down the normal
> > > path a lot (eg find all threads that match the original query and then
> > > apply the argument above).
> > > 
> > > Any thoughts?
> > 
> > Oh dear.
> > 
> > Well, here's one idea.  Instead of doing a single thread query in
> > show, do a thread query without the exclusions and then a message
> > query with the exclusions.  Output all of the messages from the first
> > query, but use the results of the second query to determine which
> > messages are "matched".  The same could be accomplished in the library
> > somewhat more efficiently, but it's not obvious to me what the API
> > would be.
> 
> Here's a slightly crazier idea that's more library-invasive than the
> original approach, but probably better in the long run.
> 
> Have notmuch_query_search_* return everything and make exclusion a
> message flag like NOTMUCH_MESSAGE_FLAG_MATCH.  Tweak the definition of
> "matched" to mean "matched and not excluded" (specifically, a message
> would have the match flag or the excluded flag or neither, but not
> both).  Search would skip threads with zero matched messages and I
> think show would Just Work.
> 
> I can think of two ways to implement this.  notmuch_query_search_*
> could perform both the original query and the query with exclusions
> and use the docid set from the second to compute the "excluded"
> message flag.  Alternatively, it could examine the tags of each
> message directly to compute the flag.  The latter is probably easier
> to implement, but probably slower.

I really like the idea of returning two flags. I think your first
suggestion works better for sorting reasons: we want to return a thread
which has a match-not-excluded message and also a match-excluded message
to be sorted based on the match-not-excluded message. Hence in
notmuch_query_search_threads we can create the list of docids to iterate
over as the list generated by query with exclusions followed by the list
without exclusions.  This list contains lots of messages twice but that
doesn't matter since we have to check whether we have already output
the message in an earlier thread anyway. 

Incidentally, it might not take very much more code to allow
notmuch_query_search_threads to take two arbitrary queries and return
all threads which match the first case but mark as matched those that
match the second: i.e. a step on the way towards "thread based and".

Best wishes

Mark

Re: [PATCH v3 2/2] search: Support automatic tag exclusions

[Xavier Maillard]

Hey Pieter,

On Thu, 19 Jan 2012 20:19:00 +0100, Pieter Praet <pieter@praet.org> wrote:
> Nice feature!  I won't be using it myself, but I can imagine it being
> *very* useful for those who still feel the need to "delete" email :).

Adding a 'deleted' tag does not mean there will be a delete/purge
process ;) (currently I got 5k messages with the tag deleted ;). 

> Nitpicking:
> 

[ ... ]

>   So I'd like to suggest replacing all occurences of "auto_exclude_tags"
>   with "search_exclude_tags" (and simply "exclude_tags" in the args to
>   `_config_get_list' and `_config_set_list', of course).

+1
 
>   Unfortunately, this would also partially invalidate your recent NEWS
>   submission [2].
> 
> - If the 'search.exclude_tags' option is missing from the config file,
>   its value is automatically set to "deleted;spam;", which probably isn't
>   a sane default.  Luckily, you've already provided the solution [3].

I am against doing something /unsafe/ in the user's back. If there is no
option set intentionnaly by the user, there is nothing notmuch should
do -i.e no exclusion -

> - To make new users aware of the config option's existence, we should
>   prompt them to configure it during setup.

+1

/Xavier

Re: [PATCH 3/4] config: only set search.exclude_tags to "deleted; spam; " during setup

[Xavier Maillard]

On Thu, 19 Jan 2012 20:19:03 +0100, Pieter Praet <pieter@praet.org> wrote:
> If the 'search.exclude_tags' option is missing from the config file,
> its value is automatically set to "deleted;spam;".  Taking PoLS/DWIM
> into account, this should probably only happen during setup.
> 
> This patch is actually Austin Clements' work:
>   id:"20120117203211.GQ16740@mit.edu"

I do not think this is a sane default. As I told it in another post. I
do not expect notmuch to skew my search queries not that I specifically
asked.

/Xavier

Re: [PATCH 3/4] config: only set search.exclude_tags to "deleted; spam; " during setup

[Jameson Graef Rollins]

[-- Attachment #1: Type: text/plain, Size: 1300 bytes --]

On Sun, 22 Jan 2012 23:14:13 +0100, Xavier Maillard <xavier@maillard.im> wrote:
> 
> On Thu, 19 Jan 2012 20:19:03 +0100, Pieter Praet <pieter@praet.org> wrote:
> > If the 'search.exclude_tags' option is missing from the config file,
> > its value is automatically set to "deleted;spam;".  Taking PoLS/DWIM
> > into account, this should probably only happen during setup.
> > 
> > This patch is actually Austin Clements' work:
> >   id:"20120117203211.GQ16740@mit.edu"
> 
> I do not think this is a sane default. As I told it in another post. I
> do not expect notmuch to skew my search queries not that I specifically
> asked.

Hi, Xavier.  Do you currently mark things as "deleted" or "spam"?  If
not, this would have no affect on your search results.  If you do, do
you currently expect those messages to show up in searches?  If so, why
did you mark them as "deleted" or "spam" to begin with?

I agree with your point in principle (ie. I don't generally want my
searches tampered with behind the scenes) but the issue here is about
messages that have been explicitly tagged as a form of "trash".  Trash
is by it's nature something you're trying to get rid of.  If you wanted
to find something in the future, why would you put it in the trash in
the first place?

jamie.

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: [PATCH] Automatically exclude tags in notmuch-show

[Mark Walters]

On Sun, 22 Jan 2012 13:16:09 -0500, Austin Clements <amdragon@MIT.EDU> wrote:
> Quoth myself on Jan 20 at 12:18 pm:
> > Quoth Mark Walters on Jan 20 at 12:10 am:
> > > 
> > > Ok Having said this is trivial I have found a problem. What should
> > > notmuch do if you do something like
> > > 
> > > notmuch show id:<some-id>
> > > and that message is marked with a deleted tag? To be consistent with the
> > > other cases (where a deleted message is in a matched thread) we might
> > > want to return the message with the not-matched flag set (eg in
> > > JSON). But my patch doesn't, as it never even sees the thread since it
> > > doesn't match.
> > > 
> > > Looking at notmuch-show.c I think we should not apply the exclude tags
> > > to do_show_single, but usually should apply it to do_show. One solution
> > > which is simple and is at least close to right would be to get do_show
> > > to return the number of threads found. If this is zero then retry the
> > > query without the excludes (possible setting the match_flag to zero on
> > > each message since we know it does not match)
> > > 
> > > This is not a completely correct solution as if you ask notmuch-show to
> > > show more than one thread it might  threads which only contain deleted
> > > messages.
> > > 
> > > I can't see other good possibilities without slowing down the normal
> > > path a lot (eg find all threads that match the original query and then
> > > apply the argument above).
> > > 
> > > Any thoughts?
> > 
> > Oh dear.
> > 
> > Well, here's one idea.  Instead of doing a single thread query in
> > show, do a thread query without the exclusions and then a message
> > query with the exclusions.  Output all of the messages from the first
> > query, but use the results of the second query to determine which
> > messages are "matched".  The same could be accomplished in the library
> > somewhat more efficiently, but it's not obvious to me what the API
> > would be.
> 
> Here's a slightly crazier idea that's more library-invasive than the
> original approach, but probably better in the long run.
> 
> Have notmuch_query_search_* return everything and make exclusion a
> message flag like NOTMUCH_MESSAGE_FLAG_MATCH.  Tweak the definition of
> "matched" to mean "matched and not excluded" (specifically, a message
> would have the match flag or the excluded flag or neither, but not
> both).  Search would skip threads with zero matched messages and I
> think show would Just Work.
> 
> I can think of two ways to implement this.  notmuch_query_search_*
> could perform both the original query and the query with exclusions
> and use the docid set from the second to compute the "excluded"
> message flag.  Alternatively, it could examine the tags of each
> message directly to compute the flag.  The latter is probably easier
> to implement, but probably slower.
> 
> Thoughts?

I have now thought about this some more and think I understand your idea
(and how it would work) rather better now. 

I would suggest one small change: the flags for the messages returned
should be "independent": so a message can match the query or not, and it
can be excluded or not, with all 4 combinations being possible. (The
consumer of notmuch_query_search_* would extract the information it
wanted.)

I have thought about some implementation ideas but I think sorting is
going to be the deciding factor: what order should
notmuch_query_search_* return messages/threads? 

For notmuch_query_search_messages either it returns them all together
with the excluded messages marked, or returns all included ones, and
then all excluded one.

For notmuch_query_search_threads it is less clear. Currently it returns
threads in order of first matching message. It is not clear what
matching means now: is matching and included, or just matching? If the
former then we will be returning some threads with no matching and
included messages so we need to decide where to put them in the order.

If we sort in both cases just on matching then we have the same
output/sort as notmuch pre-excluded flags, just the frontends
notmuch-search/show can decide to omit some lines/results. Note that
after omitting "excluded" lines the thread sort would be different from
the current notmuch-with-excluded implementation.

Whereas if we sort based on matching and included, we keep the current
sort order with some stuff appended.

As regards implementation I think notmuch_query_search_messages is the
crucial place: once that returns one of its two orders the rest sort of
takes care of itself.

Best wishes

Mark

Re: [PATCH] Automatically exclude tags in notmuch-show

[Austin Clements]

Quoth Mark Walters on Jan 23 at  1:13 am:
> On Sun, 22 Jan 2012 13:16:09 -0500, Austin Clements <amdragon@MIT.EDU> wrote:
> > Quoth myself on Jan 20 at 12:18 pm:
> > > Quoth Mark Walters on Jan 20 at 12:10 am:
> > > > 
> > > > Ok Having said this is trivial I have found a problem. What should
> > > > notmuch do if you do something like
> > > > 
> > > > notmuch show id:<some-id>
> > > > and that message is marked with a deleted tag? To be consistent with the
> > > > other cases (where a deleted message is in a matched thread) we might
> > > > want to return the message with the not-matched flag set (eg in
> > > > JSON). But my patch doesn't, as it never even sees the thread since it
> > > > doesn't match.
> > > > 
> > > > Looking at notmuch-show.c I think we should not apply the exclude tags
> > > > to do_show_single, but usually should apply it to do_show. One solution
> > > > which is simple and is at least close to right would be to get do_show
> > > > to return the number of threads found. If this is zero then retry the
> > > > query without the excludes (possible setting the match_flag to zero on
> > > > each message since we know it does not match)
> > > > 
> > > > This is not a completely correct solution as if you ask notmuch-show to
> > > > show more than one thread it might  threads which only contain deleted
> > > > messages.
> > > > 
> > > > I can't see other good possibilities without slowing down the normal
> > > > path a lot (eg find all threads that match the original query and then
> > > > apply the argument above).
> > > > 
> > > > Any thoughts?
> > > 
> > > Oh dear.
> > > 
> > > Well, here's one idea.  Instead of doing a single thread query in
> > > show, do a thread query without the exclusions and then a message
> > > query with the exclusions.  Output all of the messages from the first
> > > query, but use the results of the second query to determine which
> > > messages are "matched".  The same could be accomplished in the library
> > > somewhat more efficiently, but it's not obvious to me what the API
> > > would be.
> > 
> > Here's a slightly crazier idea that's more library-invasive than the
> > original approach, but probably better in the long run.
> > 
> > Have notmuch_query_search_* return everything and make exclusion a
> > message flag like NOTMUCH_MESSAGE_FLAG_MATCH.  Tweak the definition of
> > "matched" to mean "matched and not excluded" (specifically, a message
> > would have the match flag or the excluded flag or neither, but not
> > both).  Search would skip threads with zero matched messages and I
> > think show would Just Work.
> > 
> > I can think of two ways to implement this.  notmuch_query_search_*
> > could perform both the original query and the query with exclusions
> > and use the docid set from the second to compute the "excluded"
> > message flag.  Alternatively, it could examine the tags of each
> > message directly to compute the flag.  The latter is probably easier
> > to implement, but probably slower.
> > 
> > Thoughts?
> 
> I have now thought about this some more and think I understand your idea
> (and how it would work) rather better now. 
> 
> I would suggest one small change: the flags for the messages returned
> should be "independent": so a message can match the query or not, and it
> can be excluded or not, with all 4 combinations being possible. (The
> consumer of notmuch_query_search_* would extract the information it
> wanted.)

I'd initially approached it this way, but went with redefining a
"matched" messages because it had much less impact on the API.  For
example, with the redefined "match",
notmuch_thread_get_matched_messages still does the right thing for
search and things like the thread subject can still be based on
"matched" messages.  If we orthongonalize these flags, then we at
least need to count matched non-excluded messages and provide an API
to access this (while I don't have a solid argument against such an
API it just seems weirdly specific to me).

My other concern is performance.  In thread queries, marking
non-matched messages as excluded would require either an extra query
per thread or a single query to match all excluded messages (not
filtered by the primary query).  The former is prohibitive, though the
latter might be acceptable (that might depend on how many things
people mark as spam or deleted).  If the cost is too high, this
suggests that we shouldn't mark non-matched messages as excluded, but
then we're back to effectively having three levels of matching: not
matched, matched but not excluded, and matched but excluded.

> I have thought about some implementation ideas but I think sorting is
> going to be the deciding factor: what order should
> notmuch_query_search_* return messages/threads? 

Yes.  This is exactly what I've been puzzling over, too.

> For notmuch_query_search_messages either it returns them all together
> with the excluded messages marked, or returns all included ones, and
> then all excluded one.

I would prefer them intermingled.  I feel like returning one and then
the other is just exposing implementation details.  Plus, it's unclear
if the order of the two groups should depend on the sort order, be
configurable, or what.  Intermingling them seems like the obvious
answer.

> For notmuch_query_search_threads it is less clear. Currently it returns
> threads in order of first matching message. It is not clear what
> matching means now: is matching and included, or just matching? If the
> former then we will be returning some threads with no matching and
> included messages so we need to decide where to put them in the order.

I would argue that, if the caller cares about the sort order of the
results, it only makes sense for it to skip over threads consisting
only of excluded messages, and if the caller doesn't care about the
sort order, we can choose whatever's most convenient.

> If we sort in both cases just on matching then we have the same
> output/sort as notmuch pre-excluded flags, just the frontends
> notmuch-search/show can decide to omit some lines/results. Note that
> after omitting "excluded" lines the thread sort would be different from
> the current notmuch-with-excluded implementation.
> 
> Whereas if we sort based on matching and included, we keep the current
> sort order with some stuff appended.
> 
> As regards implementation I think notmuch_query_search_messages is the
> crucial place: once that returns one of its two orders the rest sort of
> takes care of itself.

I think that comes down to exactly how we want to handle the message
flags.

> Best wishes
> 
> Mark

Re: [PATCH v3 2/2] search: Support automatic tag exclusions

[Pieter Praet]

On Sun, 22 Jan 2012 23:09:30 +0100, Xavier Maillard <xavier@maillard.im> wrote:
> Hey Pieter,
> 

Hi!

> On Thu, 19 Jan 2012 20:19:00 +0100, Pieter Praet <pieter@praet.org> wrote:
> > Nice feature!  I won't be using it myself, but I can imagine it being
> > *very* useful for those who still feel the need to "delete" email :).
> 
> Adding a 'deleted' tag does not mean there will be a delete/purge
> process ;) (currently I got 5k messages with the tag deleted ;). 
> 

Very true, that's why I put quotes around "delete".

Deleting email for real is *so* old-fashioned... :D ;)

> > Nitpicking:
> > 
> 
> [ ... ]
> 
> >   So I'd like to suggest replacing all occurences of "auto_exclude_tags"
> >   with "search_exclude_tags" (and simply "exclude_tags" in the args to
> >   `_config_get_list' and `_config_set_list', of course).
> 
> +1
>  
> >   Unfortunately, this would also partially invalidate your recent NEWS
> >   submission [2].
> > 
> > - If the 'search.exclude_tags' option is missing from the config file,
> >   its value is automatically set to "deleted;spam;", which probably isn't
> >   a sane default.  Luckily, you've already provided the solution [3].
> 
> I am against doing something /unsafe/ in the user's back. If there is no
> option set intentionnaly by the user, there is nothing notmuch should
> do -i.e no exclusion -
> 

Absolutely.  Actually, that's *exactly* what I meant.

I thought that would be pretty clear, but perhaps it wasn't.
I've reworded some of the commit messages, and will pay more
attention to it in the future.

> > - To make new users aware of the config option's existence, we should
> >   prompt them to configure it during setup.
> 
> +1
> 
> /Xavier

Thanks for your input!


Peace

-- 
Pieter

Re: [PATCH 3/4] config: only set search.exclude_tags to "deleted; spam; " during setup

[Pieter Praet]

On Sun, 22 Jan 2012 23:14:13 +0100, Xavier Maillard <xavier@maillard.im> wrote:
> 
> On Thu, 19 Jan 2012 20:19:03 +0100, Pieter Praet <pieter@praet.org> wrote:
> > If the 'search.exclude_tags' option is missing from the config file,
> > its value is automatically set to "deleted;spam;".  Taking PoLS/DWIM
> > into account, this should probably only happen during setup.
> > 
> > This patch is actually Austin Clements' work:
> >   id:"20120117203211.GQ16740@mit.edu"
> 
> I do not think this is a sane default. As I told it in another post. I
> do not expect notmuch to skew my search queries not that I specifically
> asked.
> 

I agree 100%.  I've responded in more detail @ your previous reply:

  id:"87ty3nawbt.fsf@praet.org"

> /Xavier


Peace

-- 
Pieter

Re: [PATCH 4/4] setup: prompt user for search.exclude_tags value

[Pieter Praet]

On Sun, 22 Jan 2012 12:08:15 -0500, Austin Clements <amdragon@MIT.EDU> wrote:
> Quoth Pieter Praet on Jan 22 at  7:55 am:
> > On Thu, 19 Jan 2012 23:19:02 -0500, Austin Clements <amdragon@MIT.EDU> wrote:
> > > [...]
> > > Hah, okay.  How about this as an initial minor refactoring of the code
> > > for new.tags?
> > > [...]
> > 
> > Great, thanks!
> > 
> > 
> > Would the following commit message be satisfactory? :
> > 
> >   #+begin_quote
> >     setup: move tag printing and parsing into separate functions
> > 
> >     * notmuch-setup.c (notmuch_setup_command):
> >       Break tag printing and response parsing out into separate functions
> >       called `print_tag_list' respectively `parse_tag_list', for reuse
> >       with the 'search.exclude_tags' option.
> >   #+end_quote
> > 
> > (if not, please suggest an alternative; it'll be your name at the top)
> 
> Sure.

Great!

Updated patch series follows.


Peace

-- 
Pieter

[PATCH v2 1/6] search: rename auto_exclude_tags to {search, }exclude_tags

[Pieter Praet]

All other config-related functions and args include the section
title in their name, so for the sake of consistency, mirror that.

Also, the "auto"matic part is a given, so that was dropped.
---
 notmuch-client.h |    4 ++--
 notmuch-config.c |   28 ++++++++++++++--------------
 notmuch-count.c  |   12 ++++++------
 notmuch-search.c |   12 ++++++------
 4 files changed, 28 insertions(+), 28 deletions(-)

diff --git a/notmuch-client.h b/notmuch-client.h
index 9c1d383..f5414f6 100644
--- a/notmuch-client.h
+++ b/notmuch-client.h
@@ -252,10 +252,10 @@ notmuch_config_set_maildir_synchronize_flags (notmuch_config_t *config,
 					      notmuch_bool_t synchronize_flags);
 
 const char **
-notmuch_config_get_auto_exclude_tags (notmuch_config_t *config, size_t *length);
+notmuch_config_get_search_exclude_tags (notmuch_config_t *config, size_t *length);
 
 void
-notmuch_config_set_auto_exclude_tags (notmuch_config_t *config,
+notmuch_config_set_search_exclude_tags (notmuch_config_t *config,
 				      const char *list[],
 				      size_t length);
 
diff --git a/notmuch-config.c b/notmuch-config.c
index 8dcfe86..39da888 100644
--- a/notmuch-config.c
+++ b/notmuch-config.c
@@ -89,7 +89,7 @@ static const char search_config_comment[] =
     "\n"
     " The following option is supported here:\n"
     "\n"
-    "\tauto_exclude_tags\n"
+    "\texclude_tags\n"
     "\t\tA ;-separated list of tags that will be excluded from\n"
     "\t\tsearch results by default.  Using an excluded tag in a\n"
     "\t\tquery will override that exclusion.\n";
@@ -106,8 +106,8 @@ struct _notmuch_config {
     const char **new_tags;
     size_t new_tags_length;
     notmuch_bool_t maildir_synchronize_flags;
-    const char **auto_exclude_tags;
-    size_t auto_exclude_tags_length;
+    const char **search_exclude_tags;
+    size_t search_exclude_tags_length;
 };
 
 static int
@@ -265,8 +265,8 @@ notmuch_config_open (void *ctx,
     config->new_tags = NULL;
     config->new_tags_length = 0;
     config->maildir_synchronize_flags = TRUE;
-    config->auto_exclude_tags = NULL;
-    config->auto_exclude_tags_length = 0;
+    config->search_exclude_tags = NULL;
+    config->search_exclude_tags_length = 0;
 
     if (! g_key_file_load_from_file (config->key_file,
 				     config->filename,
@@ -361,9 +361,9 @@ notmuch_config_open (void *ctx,
 	notmuch_config_set_new_tags (config, tags, 2);
     }
 
-    if (notmuch_config_get_auto_exclude_tags (config, &tmp) == NULL) {
+    if (notmuch_config_get_search_exclude_tags (config, &tmp) == NULL) {
 	const char *tags[] = { "deleted", "spam" };
-	notmuch_config_set_auto_exclude_tags (config, tags, 2);
+	notmuch_config_set_search_exclude_tags (config, tags, 2);
     }
 
     error = NULL;
@@ -624,20 +624,20 @@ notmuch_config_set_new_tags (notmuch_config_t *config,
 }
 
 const char **
-notmuch_config_get_auto_exclude_tags (notmuch_config_t *config, size_t *length)
+notmuch_config_get_search_exclude_tags (notmuch_config_t *config, size_t *length)
 {
-    return _config_get_list (config, "search", "auto_exclude_tags",
-			     &(config->auto_exclude_tags),
-			     &(config->auto_exclude_tags_length), length);
+    return _config_get_list (config, "search", "exclude_tags",
+			     &(config->search_exclude_tags),
+			     &(config->search_exclude_tags_length), length);
 }
 
 void
-notmuch_config_set_auto_exclude_tags (notmuch_config_t *config,
+notmuch_config_set_search_exclude_tags (notmuch_config_t *config,
 				      const char *list[],
 				      size_t length)
 {
-    _config_set_list (config, "search", "auto_exclude_tags", list, length,
-		      &(config->auto_exclude_tags));
+    _config_set_list (config, "search", "exclude_tags", list, length,
+		      &(config->search_exclude_tags));
 }
 
 /* Given a configuration item of the form <group>.<key> return the
diff --git a/notmuch-count.c b/notmuch-count.c
index f77861e..63459fb 100644
--- a/notmuch-count.c
+++ b/notmuch-count.c
@@ -35,8 +35,8 @@ notmuch_count_command (void *ctx, int argc, char *argv[])
     char *query_str;
     int opt_index;
     int output = OUTPUT_MESSAGES;
-    const char **auto_exclude_tags;
-    size_t auto_exclude_tags_length;
+    const char **search_exclude_tags;
+    size_t search_exclude_tags_length;
     unsigned int i;
 
     notmuch_opt_desc_t options[] = {
@@ -78,10 +78,10 @@ notmuch_count_command (void *ctx, int argc, char *argv[])
 	return 1;
     }
 
-    auto_exclude_tags = notmuch_config_get_auto_exclude_tags
-	(config, &auto_exclude_tags_length);
-    for (i = 0; i < auto_exclude_tags_length; i++)
-	notmuch_query_add_tag_exclude (query, auto_exclude_tags[i]);
+    search_exclude_tags = notmuch_config_get_search_exclude_tags
+	(config, &search_exclude_tags_length);
+    for (i = 0; i < search_exclude_tags_length; i++)
+	notmuch_query_add_tag_exclude (query, search_exclude_tags[i]);
 
     switch (output) {
     case OUTPUT_MESSAGES:
diff --git a/notmuch-search.c b/notmuch-search.c
index 8867aab..d504051 100644
--- a/notmuch-search.c
+++ b/notmuch-search.c
@@ -423,8 +423,8 @@ notmuch_search_command (void *ctx, int argc, char *argv[])
     output_t output = OUTPUT_SUMMARY;
     int offset = 0;
     int limit = -1; /* unlimited */
-    const char **auto_exclude_tags;
-    size_t auto_exclude_tags_length;
+    const char **search_exclude_tags;
+    size_t search_exclude_tags_length;
     unsigned int i;
 
     enum { NOTMUCH_FORMAT_JSON, NOTMUCH_FORMAT_TEXT }
@@ -493,10 +493,10 @@ notmuch_search_command (void *ctx, int argc, char *argv[])
 
     notmuch_query_set_sort (query, sort);
 
-    auto_exclude_tags = notmuch_config_get_auto_exclude_tags
-	(config, &auto_exclude_tags_length);
-    for (i = 0; i < auto_exclude_tags_length; i++)
-	notmuch_query_add_tag_exclude (query, auto_exclude_tags[i]);
+    search_exclude_tags = notmuch_config_get_search_exclude_tags
+	(config, &search_exclude_tags_length);
+    for (i = 0; i < search_exclude_tags_length; i++)
+	notmuch_query_add_tag_exclude (query, search_exclude_tags[i]);
 
     switch (output) {
     default:
-- 
1.7.8.1

[PATCH v2 2/6] test: only exclude "deleted" messages from search if explicitly configured

[Pieter Praet]

Currently, the 'search.exclude_tags' option is automatically set to
"deleted;spam;" if it's missing from the config file.

This violates the Principle of Least Surprise, so update the tests to
*only* expect the exclusion of messages which are tagged "deleted" if the
'search.exclude_tags' option is explicitly set *and* contains that tag.
---
 test/search |    8 ++++++++
 1 files changed, 8 insertions(+), 0 deletions(-)

diff --git a/test/search b/test/search
index bf965e7..99d94bd 100755
--- a/test/search
+++ b/test/search
@@ -130,6 +130,7 @@ output=$(notmuch search "bödý" | notmuch_search_sanitize)
 test_expect_equal "$output" "thread:XXX   2000-01-01 [1/1] Notmuch Test Suite; utf8-message-body-subject (inbox unread)"
 
 test_begin_subtest "Exclude \"deleted\" messages from search"
+notmuch config set search.exclude_tags = deleted
 generate_message '[subject]="Not deleted"'
 generate_message '[subject]="Deleted"'
 notmuch new > /dev/null
@@ -147,4 +148,11 @@ output=$(notmuch search subject:deleted | notmuch_search_sanitize)
 test_expect_equal "$output" "thread:XXX   2001-01-05 [1/1] Notmuch Test Suite; Not deleted (inbox unread)
 thread:XXX   2001-01-05 [1/2] Notmuch Test Suite; Not deleted reply (deleted inbox unread)"
 
+test_begin_subtest "Don't exclude \"deleted\" messages from search if not configured"
+test_subtest_known_broken
+notmuch config set search.exclude_tags
+output=$(notmuch search subject:deleted | notmuch_search_sanitize)
+test_expect_equal "$output" "thread:XXX   2001-01-05 [1/1] Notmuch Test Suite; Not deleted (inbox unread)
+thread:XXX   2001-01-05 [2/2] Notmuch Test Suite; Deleted (deleted inbox unread)"
+
 test_done
-- 
1.7.8.1

[PATCH v2 3/6] config: only exclude messages if 'search.exclude_tags' is explicitly set

[Pieter Praet]

Currently, the 'search.exclude_tags' option is automatically
set to "deleted;spam;" if it's missing from the config file.

This violates the Principle of Least Surprise, so *only* set
'search.exclude_tags' to "deleted;spam;" if we didn't find a
configuration file at all.

This patch is actually Austin Clements' work:
  id:"20120117203211.GQ16740@mit.edu"
---
 notmuch-config.c |    8 ++++++--
 test/search      |    1 -
 2 files changed, 6 insertions(+), 3 deletions(-)

diff --git a/notmuch-config.c b/notmuch-config.c
index 39da888..0ded6d7 100644
--- a/notmuch-config.c
+++ b/notmuch-config.c
@@ -362,8 +362,12 @@ notmuch_config_open (void *ctx,
     }
 
     if (notmuch_config_get_search_exclude_tags (config, &tmp) == NULL) {
-	const char *tags[] = { "deleted", "spam" };
-	notmuch_config_set_search_exclude_tags (config, tags, 2);
+	if (is_new) {
+	    const char *tags[] = { "deleted", "spam" };
+	    notmuch_config_set_search_exclude_tags (config, tags, 2);
+	} else {
+	    notmuch_config_set_search_exclude_tags (config, NULL, 0);
+	}
     }
 
     error = NULL;
diff --git a/test/search b/test/search
index 99d94bd..414be35 100755
--- a/test/search
+++ b/test/search
@@ -149,7 +149,6 @@ test_expect_equal "$output" "thread:XXX   2001-01-05 [1/1] Notmuch Test Suite; N
 thread:XXX   2001-01-05 [1/2] Notmuch Test Suite; Not deleted reply (deleted inbox unread)"
 
 test_begin_subtest "Don't exclude \"deleted\" messages from search if not configured"
-test_subtest_known_broken
 notmuch config set search.exclude_tags
 output=$(notmuch search subject:deleted | notmuch_search_sanitize)
 test_expect_equal "$output" "thread:XXX   2001-01-05 [1/1] Notmuch Test Suite; Not deleted (inbox unread)
-- 
1.7.8.1

[PATCH v2 4/6] setup: move tag printing and parsing into separate functions

[Pieter Praet]

From: Austin Clements <amdragon@MIT.EDU>

* notmuch-setup.c (notmuch_setup_command):
  Break tag printing and response parsing out into separate functions
  called `print_tag_list' respectively `parse_tag_list', for reuse
  with the 'search.exclude_tags' option.

---
 notmuch-setup.c |   55 ++++++++++++++++++++++++++++++++++---------------------
 1 files changed, 34 insertions(+), 21 deletions(-)

diff --git a/notmuch-setup.c b/notmuch-setup.c
index c3ea937..dcfa607 100644
--- a/notmuch-setup.c
+++ b/notmuch-setup.c
@@ -87,6 +87,38 @@ welcome_message_post_setup (void)
 "have sufficient storage space available now.\n\n");
 }
 
+static void
+print_tag_list (const char **tags, size_t tags_len)
+{
+    unsigned int i;
+    for (i = 0; i < tags_len; i++) {
+	if (i != 0)
+	    printf (" ");
+	printf ("%s", tags[i]);
+    }
+}
+
+static GPtrArray *
+parse_tag_list (void *ctx, char *response)
+{
+    GPtrArray *tags = g_ptr_array_new ();
+    char *tag = response;
+    char *space;
+
+    while (tag && *tag) {
+	space = strchr (tag, ' ');
+	if (space)
+	    g_ptr_array_add (tags, talloc_strndup (ctx, tag, space - tag));
+	else
+	    g_ptr_array_add (tags, talloc_strdup (ctx, tag));
+	tag = space;
+	while (tag && *tag == ' ')
+	    tag++;
+    }
+
+    return tags;
+}
+
 int
 notmuch_setup_command (unused (void *ctx),
 		       unused (int argc), unused (char *argv[]))
@@ -164,30 +196,11 @@ notmuch_setup_command (unused (void *ctx),
     new_tags = notmuch_config_get_new_tags (config, &new_tags_len);
 
     printf ("Tags to apply to all new messages (separated by spaces) [");
-
-    for (i = 0; i < new_tags_len; i++) {
-	if (i != 0)
-	    printf (" ");
-	printf ("%s", new_tags[i]);
-    }
-
+    print_tag_list(new_tags, new_tags_len);
     prompt ("]: ");
 
     if (strlen (response)) {
-	GPtrArray *tags = g_ptr_array_new ();
-	char *tag = response;
-	char *space;
-
-	while (tag && *tag) {
-	    space = strchr (tag, ' ');
-	    if (space)
-		g_ptr_array_add (tags, talloc_strndup (ctx, tag, space - tag));
-	    else
-		g_ptr_array_add (tags, talloc_strdup (ctx, tag));
-	    tag = space;
-	    while (tag && *tag == ' ')
-		tag++;
-	}
+	GPtrArray *tags = parse_tag_list (ctx, response);
 
 	notmuch_config_set_new_tags (config, (const char **) tags->pdata,
 				     tags->len);
-- 
1.7.8.1

[PATCH v2 5/6] setup: prompt user for search.exclude_tags value

[Pieter Praet]

Allow users to customize the search.exclude_tags option during setup.
---
 notmuch-setup.c |   21 +++++++++++++++++++++
 1 files changed, 21 insertions(+), 0 deletions(-)

diff --git a/notmuch-setup.c b/notmuch-setup.c
index dcfa607..0d75adc 100644
--- a/notmuch-setup.c
+++ b/notmuch-setup.c
@@ -133,6 +133,8 @@ notmuch_setup_command (unused (void *ctx),
     int is_new;
     const char **new_tags;
     size_t new_tags_len;
+    const char **search_exclude_tags;
+    size_t search_exclude_tags_len;
 
 #define prompt(format, ...)					\
     do {							\
@@ -208,6 +210,25 @@ notmuch_setup_command (unused (void *ctx),
 	g_ptr_array_free (tags, TRUE);
     }
 
+
+    search_exclude_tags = notmuch_config_get_search_exclude_tags (config, &search_exclude_tags_len);
+
+    printf ("Tags to exclude when searching messages (separated by spaces) [");
+    print_tag_list(search_exclude_tags, search_exclude_tags_len);
+    prompt ("]: ");
+
+    if (strlen (response)) {
+	GPtrArray *tags = parse_tag_list (ctx, response);
+
+	notmuch_config_set_search_exclude_tags (config,
+						(const char **)
+						tags->pdata,
+						tags->len);
+
+	g_ptr_array_free (tags, TRUE);
+    }
+
+
     if (! notmuch_config_save (config)) {
 	if (is_new)
 	  welcome_message_post_setup ();
-- 
1.7.8.1

[PATCH v2 6/6] NEWS: update "Tag exclusion" section

[Pieter Praet]

---
 NEWS |   14 ++++++++++----
 1 files changed, 10 insertions(+), 4 deletions(-)

diff --git a/NEWS b/NEWS
index 2c2d9e9..a0b7929 100644
--- a/NEWS
+++ b/NEWS
@@ -13,10 +13,16 @@ Reply to sender
 
 Tag exclusion
 
-  Tags can be automatically excluded from search results unless they
-  appear explicitly in a query.  By default, notmuch excludes the tags
-  deleted and spam.  This can be changed using the new config setting
-  search.auto_exclude_tags.
+  Tags can be automatically excluded from search results by adding them
+  to the new 'search.exclude_tags' option in the Notmuch config file.
+
+  This behaviour can be overridden by explicitly including an excluded
+  tag in your query, for example:
+
+    notmuch search $your_query and tag:$excluded_tag
+
+  Existing users will probably want to run "notmuch setup" again to add
+  the new well-commented [search] section to the configuration file.
 
 Emacs Interface
 ---------------
-- 
1.7.8.1

Re: [PATCH v2 5/6] setup: prompt user for search.exclude_tags value

[Austin Clements]

Quoth Pieter Praet on Jan 23 at  5:22 am:
> Allow users to customize the search.exclude_tags option during setup.
> ---
>  notmuch-setup.c |   21 +++++++++++++++++++++
>  1 files changed, 21 insertions(+), 0 deletions(-)
> 
> diff --git a/notmuch-setup.c b/notmuch-setup.c
> index dcfa607..0d75adc 100644
> --- a/notmuch-setup.c
> +++ b/notmuch-setup.c
> @@ -133,6 +133,8 @@ notmuch_setup_command (unused (void *ctx),
>      int is_new;
>      const char **new_tags;
>      size_t new_tags_len;
> +    const char **search_exclude_tags;
> +    size_t search_exclude_tags_len;
>  
>  #define prompt(format, ...)					\
>      do {							\
> @@ -208,6 +210,25 @@ notmuch_setup_command (unused (void *ctx),
>  	g_ptr_array_free (tags, TRUE);
>      }
>  
> +
> +    search_exclude_tags = notmuch_config_get_search_exclude_tags (config, &search_exclude_tags_len);
> +
> +    printf ("Tags to exclude when searching messages (separated by spaces) [");
> +    print_tag_list(search_exclude_tags, search_exclude_tags_len);

Missing space before paren.

> +    prompt ("]: ");
> +
> +    if (strlen (response)) {
> +	GPtrArray *tags = parse_tag_list (ctx, response);
> +
> +	notmuch_config_set_search_exclude_tags (config,
> +						(const char **)
> +						tags->pdata,

No newline is needed between the case and the value.

> +						tags->len);
> +
> +	g_ptr_array_free (tags, TRUE);
> +    }
> +
> +
>      if (! notmuch_config_save (config)) {
>  	if (is_new)
>  	  welcome_message_post_setup ();

Re: [PATCH v2 6/6] NEWS: update "Tag exclusion" section

[Austin Clements]

Quoth Pieter Praet on Jan 23 at  5:22 am:
> ---
>  NEWS |   14 ++++++++++----
>  1 files changed, 10 insertions(+), 4 deletions(-)
> 
> diff --git a/NEWS b/NEWS
> index 2c2d9e9..a0b7929 100644
> --- a/NEWS
> +++ b/NEWS
> @@ -13,10 +13,16 @@ Reply to sender
>  
>  Tag exclusion
>  
> -  Tags can be automatically excluded from search results unless they
> -  appear explicitly in a query.  By default, notmuch excludes the tags
> -  deleted and spam.  This can be changed using the new config setting
> -  search.auto_exclude_tags.
> +  Tags can be automatically excluded from search results by adding them
> +  to the new 'search.exclude_tags' option in the Notmuch config file.
> +
> +  This behaviour can be overridden by explicitly including an excluded
> +  tag in your query, for example:
> +
> +    notmuch search $your_query and tag:$excluded_tag
> +
> +  Existing users will probably want to run "notmuch setup" again to add
> +  the new well-commented [search] section to the configuration file.

Should probably add something about the default for new
configurations.  This also gives existing users an idea of what to set
this to if they want to follow the conventions used by new
configurations.  Maybe just add

In new configurations, this defaults to "deleted" and "spam".

>  
>  Emacs Interface
>  ---------------

Re: [PATCH 3/4] config: only set search.exclude_tags to "deleted; spam; " during setup

[Pieter Praet]

On Sun, 22 Jan 2012 14:53:41 -0800, Jameson Graef Rollins <jrollins@finestructure.net> wrote:
> On Sun, 22 Jan 2012 23:14:13 +0100, Xavier Maillard <xavier@maillard.im> wrote:
> > 
> > On Thu, 19 Jan 2012 20:19:03 +0100, Pieter Praet <pieter@praet.org> wrote:
> > > If the 'search.exclude_tags' option is missing from the config file,
> > > its value is automatically set to "deleted;spam;".  Taking PoLS/DWIM
> > > into account, this should probably only happen during setup.
> > > 
> > > This patch is actually Austin Clements' work:
> > >   id:"20120117203211.GQ16740@mit.edu"
> > 
> > I do not think this is a sane default. As I told it in another post. I
> > do not expect notmuch to skew my search queries not that I specifically
> > asked.
> 
> Hi, Xavier.  Do you currently mark things as "deleted" or "spam"?  If
> not, this would have no affect on your search results.  If you do, do
> you currently expect those messages to show up in searches?  If so, why
> did you mark them as "deleted" or "spam" to begin with?
> 
> I agree with your point in principle (ie. I don't generally want my
> searches tampered with behind the scenes) but the issue here is about
> messages that have been explicitly tagged as a form of "trash".  Trash
> is by it's nature something you're trying to get rid of.  If you wanted
> to find something in the future, why would you put it in the trash in
> the first place?
> 

You definitely have a point, but then again, who are we to assume that
the terms "deleted" and "spam" have the *exact* same meaning for
everyone?  (also see id:"8739bbo0br.fsf@praet.org")

IMHO, this is one of those options that should remain disabled until
*explicitly* set by the user.

> jamie.


Peace

-- 
Pieter

Re: [PATCH v2 4/6] setup: move tag printing and parsing into separate functions

[Austin Clements]

Quoth Pieter Praet on Jan 23 at  5:22 am:
> From: Austin Clements <amdragon@MIT.EDU>
> 
> * notmuch-setup.c (notmuch_setup_command):
>   Break tag printing and response parsing out into separate functions
>   called `print_tag_list' respectively `parse_tag_list', for reuse
>   with the 'search.exclude_tags' option.

Since I'm revising this patch a little bit anyway, how about

setup: Create functions for tag list printing and parsing

This refactors the tag list printing and parsing currently used for
new.tags so that both can be reused for the new search.exclude_tags
option.

> 
> ---
>  notmuch-setup.c |   55 ++++++++++++++++++++++++++++++++++---------------------
>  1 files changed, 34 insertions(+), 21 deletions(-)
> 
> diff --git a/notmuch-setup.c b/notmuch-setup.c
> index c3ea937..dcfa607 100644
> --- a/notmuch-setup.c
> +++ b/notmuch-setup.c
> @@ -87,6 +87,38 @@ welcome_message_post_setup (void)
>  "have sufficient storage space available now.\n\n");
>  }
>  
> +static void
> +print_tag_list (const char **tags, size_t tags_len)
> +{
> +    unsigned int i;
> +    for (i = 0; i < tags_len; i++) {
> +	if (i != 0)
> +	    printf (" ");
> +	printf ("%s", tags[i]);
> +    }
> +}
> +
> +static GPtrArray *
> +parse_tag_list (void *ctx, char *response)
> +{
> +    GPtrArray *tags = g_ptr_array_new ();
> +    char *tag = response;
> +    char *space;
> +
> +    while (tag && *tag) {
> +	space = strchr (tag, ' ');
> +	if (space)
> +	    g_ptr_array_add (tags, talloc_strndup (ctx, tag, space - tag));
> +	else
> +	    g_ptr_array_add (tags, talloc_strdup (ctx, tag));
> +	tag = space;
> +	while (tag && *tag == ' ')
> +	    tag++;
> +    }
> +
> +    return tags;
> +}
> +
>  int
>  notmuch_setup_command (unused (void *ctx),
>  		       unused (int argc), unused (char *argv[]))
> @@ -164,30 +196,11 @@ notmuch_setup_command (unused (void *ctx),
>      new_tags = notmuch_config_get_new_tags (config, &new_tags_len);
>  
>      printf ("Tags to apply to all new messages (separated by spaces) [");
> -
> -    for (i = 0; i < new_tags_len; i++) {
> -	if (i != 0)
> -	    printf (" ");
> -	printf ("%s", new_tags[i]);
> -    }
> -
> +    print_tag_list(new_tags, new_tags_len);

Missing space before paren.  (Sorry, my fault.)

>      prompt ("]: ");
>  
>      if (strlen (response)) {
> -	GPtrArray *tags = g_ptr_array_new ();
> -	char *tag = response;
> -	char *space;
> -
> -	while (tag && *tag) {
> -	    space = strchr (tag, ' ');
> -	    if (space)
> -		g_ptr_array_add (tags, talloc_strndup (ctx, tag, space - tag));
> -	    else
> -		g_ptr_array_add (tags, talloc_strdup (ctx, tag));
> -	    tag = space;
> -	    while (tag && *tag == ' ')
> -		tag++;
> -	}
> +	GPtrArray *tags = parse_tag_list (ctx, response);
>  
>  	notmuch_config_set_new_tags (config, (const char **) tags->pdata,
>  				     tags->len);

Re: [PATCH 3/4] config: only set search.exclude_tags to "deleted; spam; " during setup

[Jameson Graef Rollins]

[-- Attachment #1: Type: text/plain, Size: 1995 bytes --]

On Mon, 23 Jan 2012 06:05:27 +0100, Pieter Praet <pieter@praet.org> wrote:
> You definitely have a point, but then again, who are we to assume that
> the terms "deleted" and "spam" have the *exact* same meaning for
> everyone?  (also see id:"8739bbo0br.fsf@praet.org")

Hrm.  I'm not sure I buy this.  Words already have meanings.  If we're
going to start down a rabbit hole where we have to assume that users are
making up crazy alternate meanings for words, we're going to run into a
lot of problems.

Notmuch, or at least the emacs interface, already assumes a specific
meaning for certain terms, like most notably "inbox".  Given that we're
dealing with english here, I think we have to assume common usage
meanings for any of the words we're using to describe anything.

This argument breaks a little in regards to "delete" since we're not
actually deleting anything in the sense of rm'ing it form the
filesystem, so we're already changing the meaning a bit.  But see below.

> IMHO, this is one of those options that should remain disabled until
> *explicitly* set by the user.

Ok, but then we're back to the incredibly prolonged discussion we've
been having about adding "delete" keys.  If we disable this by default,
but add "delete" keys, the user might be in for a different surprise if
"deleted" messages keep showing up in searches.

Basically we have two options as I see it:

- add keys bindings to add "deleted" tags, and then *also* exclude
  "tag:deleted" by default.

- don't exclude anything by default, but then don't add any special keys
  to handle "deleted" tags.

There seemed to be a consensus forming that we in fact did want to add
the "deleted" key bindings.  If we do that, then I think we should
generate the config file to exclude "deleted" tags by default.

jamie.

PS: when I say "exclude tags by default" I actually mean that the
setting should be added to the config file upon (re)generation.  Nothing
should be excluded if nothing is set in the config file.

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

[PATCH v3 5/6] setup: prompt user for search.exclude_tags value

[Pieter Praet]

Allow users to customize the search.exclude_tags option during setup.

---

v2:
- less copy-paste coding... :)

v3: Austin's corrections [1]
- @ `print_tag_list':
  add space before paren.

- @ `notmuch_config_set_search_exclude_tags':
  remove \n between type cast and value.

[1] id:"20120123043435.GR16740@mit.edu"

 notmuch-setup.c |   20 ++++++++++++++++++++
 1 files changed, 20 insertions(+), 0 deletions(-)

diff --git a/notmuch-setup.c b/notmuch-setup.c
index dcfa607..2941c52 100644
--- a/notmuch-setup.c
+++ b/notmuch-setup.c
@@ -133,6 +133,8 @@ notmuch_setup_command (unused (void *ctx),
     int is_new;
     const char **new_tags;
     size_t new_tags_len;
+    const char **search_exclude_tags;
+    size_t search_exclude_tags_len;
 
 #define prompt(format, ...)					\
     do {							\
@@ -208,6 +210,24 @@ notmuch_setup_command (unused (void *ctx),
 	g_ptr_array_free (tags, TRUE);
     }
 
+
+    search_exclude_tags = notmuch_config_get_search_exclude_tags (config, &search_exclude_tags_len);
+
+    printf ("Tags to exclude when searching messages (separated by spaces) [");
+    print_tag_list (search_exclude_tags, search_exclude_tags_len);
+    prompt ("]: ");
+
+    if (strlen (response)) {
+	GPtrArray *tags = parse_tag_list (ctx, response);
+
+	notmuch_config_set_search_exclude_tags (config,
+						(const char **) tags->pdata,
+						tags->len);
+
+	g_ptr_array_free (tags, TRUE);
+    }
+
+
     if (! notmuch_config_save (config)) {
 	if (is_new)
 	  welcome_message_post_setup ();
-- 
1.7.8.1

[PATCH v3 6/6] NEWS: update "Tag exclusion" section

[Pieter Praet]

---

Added note re new configurations, as suggested by Austin:
  id:"20120123044108.GS16740@mit.edu"

 NEWS |   19 +++++++++++++++----
 1 files changed, 15 insertions(+), 4 deletions(-)

diff --git a/NEWS b/NEWS
index 2c2d9e9..2acdce5 100644
--- a/NEWS
+++ b/NEWS
@@ -13,10 +13,21 @@ Reply to sender
 
 Tag exclusion
 
-  Tags can be automatically excluded from search results unless they
-  appear explicitly in a query.  By default, notmuch excludes the tags
-  deleted and spam.  This can be changed using the new config setting
-  search.auto_exclude_tags.
+  Tags can be automatically excluded from search results by adding them
+  to the new 'search.exclude_tags' option in the Notmuch config file.
+
+  This behaviour can be overridden by explicitly including an excluded
+  tag in your query, for example:
+
+    notmuch search $your_query and tag:$excluded_tag
+
+  Existing users will probably want to run "notmuch setup" again to add
+  the new well-commented [search] section to the configuration file.
+
+  For new configurations, accepting the default setting will cause the
+  tags "deleted" and "spam" to be excluded, equivalent to running:
+
+    notmuch config set search.exclude_tags deleted spam
 
 Emacs Interface
 ---------------
-- 
1.7.8.1

[PATCH v3 4/6] setup: Create functions for tag list printing and parsing

[Pieter Praet]

From: Austin Clements <amdragon@MIT.EDU>

This refactors the tag list printing and parsing currently used for
new.tags so that both can be reused for the new search.exclude_tags
option.

---
 notmuch-setup.c |   55 ++++++++++++++++++++++++++++++++++---------------------
 1 files changed, 34 insertions(+), 21 deletions(-)

diff --git a/notmuch-setup.c b/notmuch-setup.c
index c3ea937..f85e0eb 100644
--- a/notmuch-setup.c
+++ b/notmuch-setup.c
@@ -87,6 +87,38 @@ welcome_message_post_setup (void)
 "have sufficient storage space available now.\n\n");
 }
 
+static void
+print_tag_list (const char **tags, size_t tags_len)
+{
+    unsigned int i;
+    for (i = 0; i < tags_len; i++) {
+	if (i != 0)
+	    printf (" ");
+	printf ("%s", tags[i]);
+    }
+}
+
+static GPtrArray *
+parse_tag_list (void *ctx, char *response)
+{
+    GPtrArray *tags = g_ptr_array_new ();
+    char *tag = response;
+    char *space;
+
+    while (tag && *tag) {
+	space = strchr (tag, ' ');
+	if (space)
+	    g_ptr_array_add (tags, talloc_strndup (ctx, tag, space - tag));
+	else
+	    g_ptr_array_add (tags, talloc_strdup (ctx, tag));
+	tag = space;
+	while (tag && *tag == ' ')
+	    tag++;
+    }
+
+    return tags;
+}
+
 int
 notmuch_setup_command (unused (void *ctx),
 		       unused (int argc), unused (char *argv[]))
@@ -164,30 +196,11 @@ notmuch_setup_command (unused (void *ctx),
     new_tags = notmuch_config_get_new_tags (config, &new_tags_len);
 
     printf ("Tags to apply to all new messages (separated by spaces) [");
-
-    for (i = 0; i < new_tags_len; i++) {
-	if (i != 0)
-	    printf (" ");
-	printf ("%s", new_tags[i]);
-    }
-
+    print_tag_list (new_tags, new_tags_len);
     prompt ("]: ");
 
     if (strlen (response)) {
-	GPtrArray *tags = g_ptr_array_new ();
-	char *tag = response;
-	char *space;
-
-	while (tag && *tag) {
-	    space = strchr (tag, ' ');
-	    if (space)
-		g_ptr_array_add (tags, talloc_strndup (ctx, tag, space - tag));
-	    else
-		g_ptr_array_add (tags, talloc_strdup (ctx, tag));
-	    tag = space;
-	    while (tag && *tag == ' ')
-		tag++;
-	}
+	GPtrArray *tags = parse_tag_list (ctx, response);
 
 	notmuch_config_set_new_tags (config, (const char **) tags->pdata,
 				     tags->len);
-- 
1.7.8.1

Re: [PATCH 3/4] config: only set search.exclude_tags to "deleted; spam; " during setup

[Jani Nikula]

On Mon, 23 Jan 2012 06:05:27 +0100, Pieter Praet <pieter@praet.org> wrote:
> On Sun, 22 Jan 2012 14:53:41 -0800, Jameson Graef Rollins <jrollins@finestructure.net> wrote:
> > On Sun, 22 Jan 2012 23:14:13 +0100, Xavier Maillard <xavier@maillard.im> wrote:
> > > 
> > > On Thu, 19 Jan 2012 20:19:03 +0100, Pieter Praet <pieter@praet.org> wrote:
> > > > If the 'search.exclude_tags' option is missing from the config file,
> > > > its value is automatically set to "deleted;spam;".  Taking PoLS/DWIM
> > > > into account, this should probably only happen during setup.
> > > > 
> > > > This patch is actually Austin Clements' work:
> > > >   id:"20120117203211.GQ16740@mit.edu"
> > > 
> > > I do not think this is a sane default. As I told it in another post. I
> > > do not expect notmuch to skew my search queries not that I specifically
> > > asked.
> > 
> > Hi, Xavier.  Do you currently mark things as "deleted" or "spam"?  If
> > not, this would have no affect on your search results.  If you do, do
> > you currently expect those messages to show up in searches?  If so, why
> > did you mark them as "deleted" or "spam" to begin with?
> > 
> > I agree with your point in principle (ie. I don't generally want my
> > searches tampered with behind the scenes) but the issue here is about
> > messages that have been explicitly tagged as a form of "trash".  Trash
> > is by it's nature something you're trying to get rid of.  If you wanted
> > to find something in the future, why would you put it in the trash in
> > the first place?
> > 
> 
> You definitely have a point, but then again, who are we to assume that
> the terms "deleted" and "spam" have the *exact* same meaning for
> everyone?  (also see id:"8739bbo0br.fsf@praet.org")

"deleted" used to be a tag recognized by notmuch, and it used to sync to
the T (trashed) maildir flag. Even if notmuch won't delete any of your
mails now, I don't think you should use "deleted" on messages you want
to see again. Please let's not split hairs about this.

There really should be a definitive list of tags that are special to
lib/cli/emacs (like "inbox", "unread", "deleted", ...), or are
recommended for specific purposes (like "new" as an intermediate tag
before more sophisticated tagging), to avoid prolonged discussions like
this.

BR,
Jani.

Re: [PATCH 3/4] config: only set search.exclude_tags to "deleted; spam; " during setup

[Pieter Praet]

On Sun, 22 Jan 2012 21:34:00 -0800, Jameson Graef Rollins <jrollins@finestructure.net> wrote:
> On Mon, 23 Jan 2012 06:05:27 +0100, Pieter Praet <pieter@praet.org> wrote:
> > You definitely have a point, but then again, who are we to assume that
> > the terms "deleted" and "spam" have the *exact* same meaning for
> > everyone?  (also see id:"8739bbo0br.fsf@praet.org")
> 
> Hrm.  I'm not sure I buy this.  Words already have meanings.  If we're
> going to start down a rabbit hole where we have to assume that users are
> making up crazy alternate meanings for words, we're going to run into a
> lot of problems.
> 

True, but we're talking about tags here.  Flexibility is their raison
d'être.  If we're going to impose semi-arbitrary limitations, we do away
with alot of the benefits we were looking for in the first place.

> Notmuch, or at least the emacs interface, already assumes a specific
> meaning for certain terms, like most notably "inbox".  Given that we're
> dealing with english here, I think we have to assume common usage
> meanings for any of the words we're using to describe anything.
> 

Actually, we're only a small step away from being free to use whatever
tag(s) we want for this.  The term "inbox" is hardcoded in only 4
places, all of which are trivial to fix:

- @ binary
  - `notmuch_config_open', where it's only used as a fallback when
    'new.tags' isn't set (and reused to suggest a default value for
    'new.tags' when running setup).

- @ Emacs UI
  - `notmuch-saved-searches' (the function), where it's only used as
    a fallback when `notmuch-saved-searches' (the var) isn't set.
  - `notmuch-search-archive-thread'
  - `notmuch-show-archive-thread-internal'

> This argument breaks a little in regards to "delete" since we're not
> actually deleting anything in the sense of rm'ing it form the
> filesystem, so we're already changing the meaning a bit.  But see below.
> 

[...]

> > IMHO, this is one of those options that should remain disabled until
> > *explicitly* set by the user.
> 
> Ok, but then we're back to the incredibly prolonged discussion we've
> been having about adding "delete" keys.  If we disable this by default,
> but add "delete" keys, the user might be in for a different surprise if
> "deleted" messages keep showing up in searches.
> 
> Basically we have two options as I see it:
> 
> - add keys bindings to add "deleted" tags, and then *also* exclude
>   "tag:deleted" by default.
> 
> - don't exclude anything by default, but then don't add any special keys
>   to handle "deleted" tags.
> 

Adding a key to "delete" messages doesn't necessarily have to mean that
the tag it adds should be hardcoded to "deleted".

For example, our config file could contain something like this:

  #+begin_src conf
    [new]
    tags=unread;inbox;

    [tag]
    deleted=deleted;-inbox;
    archive=-inbox;

    [search]
    exclude_tags=deleted;spam;
  #+end_src

  (where all entries under the [tag] section could be used as
   aliases for "complex" tagging operations)

... and then we could "delete" messages using something like:

  #+begin_src emacs-lisp
    (define-key notmuch-show-mode-map "d"
      (lambda()
        (interactive)
        (notmuch-show-mod-tags
          (split-string
            (notmuch-config-get "tag.deleted") "\n"))
        (notmuch-show-next-open-message)))
  #+end_src

  (`notmuch-show-mod-tags' doesn't exist, of course, but see
   `notmuch-message-mark-replied' for a good example of how
   it could be work...)

> There seemed to be a consensus forming that we in fact did want to add
> the "deleted" key bindings.  [...]

+1.

> [...] If we do that, then I think we should
> generate the config file to exclude "deleted" tags by default.
> 
> jamie.
> 
> PS: when I say "exclude tags by default" I actually mean that the
> setting should be added to the config file upon (re)generation.  Nothing
> should be excluded if nothing is set in the config file.

Exactly!  This is actually the *only* reason I involved myself with this
whole conversation in the first place :)


Peace

-- 
Pieter

Re: [PATCH 3/4] config: only set search.exclude_tags to "deleted; spam; " during setup

[Jameson Graef Rollins]

[-- Attachment #1: Type: text/plain, Size: 682 bytes --]

On Mon, 23 Jan 2012 07:22:25 +0000, Jani Nikula <jani@nikula.org> wrote:
> There really should be a definitive list of tags that are special to
> lib/cli/emacs (like "inbox", "unread", "deleted", ...), or are
> recommended for specific purposes (like "new" as an intermediate tag
> before more sophisticated tagging), to avoid prolonged discussions like
> this.

Just to be clear: the lib doesn't assign any special meaning to any tag
(as it shouldn't).  The cli does, but only in the sense that it creates
config files that designate certain tags for certain operations by
default.  It's really in emacs where certain tags currently have
unconfigurable meanings ("inbox").

jamie.

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: [PATCH 3/4] config: only set search.exclude_tags to "deleted; spam; " during setup

[Pieter Praet]

On Mon, 23 Jan 2012 07:22:25 +0000, Jani Nikula <jani@nikula.org> wrote:
> On Mon, 23 Jan 2012 06:05:27 +0100, Pieter Praet <pieter@praet.org> wrote:
> > On Sun, 22 Jan 2012 14:53:41 -0800, Jameson Graef Rollins <jrollins@finestructure.net> wrote:
> > > On Sun, 22 Jan 2012 23:14:13 +0100, Xavier Maillard <xavier@maillard.im> wrote:
> > > > 
> > > > On Thu, 19 Jan 2012 20:19:03 +0100, Pieter Praet <pieter@praet.org> wrote:
> > > > > If the 'search.exclude_tags' option is missing from the config file,
> > > > > its value is automatically set to "deleted;spam;".  Taking PoLS/DWIM
> > > > > into account, this should probably only happen during setup.
> > > > > 
> > > > > This patch is actually Austin Clements' work:
> > > > >   id:"20120117203211.GQ16740@mit.edu"
> > > > 
> > > > I do not think this is a sane default. As I told it in another post. I
> > > > do not expect notmuch to skew my search queries not that I specifically
> > > > asked.
> > > 
> > > Hi, Xavier.  Do you currently mark things as "deleted" or "spam"?  If
> > > not, this would have no affect on your search results.  If you do, do
> > > you currently expect those messages to show up in searches?  If so, why
> > > did you mark them as "deleted" or "spam" to begin with?
> > > 
> > > I agree with your point in principle (ie. I don't generally want my
> > > searches tampered with behind the scenes) but the issue here is about
> > > messages that have been explicitly tagged as a form of "trash".  Trash
> > > is by it's nature something you're trying to get rid of.  If you wanted
> > > to find something in the future, why would you put it in the trash in
> > > the first place?
> > > 
> > 
> > You definitely have a point, but then again, who are we to assume that
> > the terms "deleted" and "spam" have the *exact* same meaning for
> > everyone?  (also see id:"8739bbo0br.fsf@praet.org")
> 
> "deleted" used to be a tag recognized by notmuch, and it used to sync to
> the T (trashed) maildir flag. Even if notmuch won't delete any of your
> mails now, I don't think you should use "deleted" on messages you want
> to see again. Please let's not split hairs about this.
> 

Agreed, but it might be nice to make a clear distinction between
concepts and the actual tags mapped to them.  I'm not suggestion we
redefine the term "deleted", but from an internationalization
standpoint, we shouldn't prevent users from mapping e.g. "verwijderd",
"supprimé", "gelöscht", ... to the concept "deleted".

> There really should be a definitive list of tags that are special to
> lib/cli/emacs (like "inbox", "unread", "deleted", ...), or are
> recommended for specific purposes (like "new" as an intermediate tag
> before more sophisticated tagging), to avoid prolonged discussions like
> this.
> 

A list of recommended tags would definitely be nice, as long as they
remain recommendations (as opposed to obligations), especially since
there's really no reason to designate certain tags as being "special".

> BR,
> Jani.


Peace

-- 
Pieter

Re: [PATCH 3/4] config: only set search.exclude_tags to "deleted; spam; " during setup

[Jani Nikula]

On Sun, 22 Jan 2012 23:38:40 -0800, Jameson Graef Rollins <jrollins@finestructure.net> wrote:
> On Mon, 23 Jan 2012 07:22:25 +0000, Jani Nikula <jani@nikula.org> wrote:
> > There really should be a definitive list of tags that are special to
> > lib/cli/emacs (like "inbox", "unread", "deleted", ...), or are
> > recommended for specific purposes (like "new" as an intermediate tag
> > before more sophisticated tagging), to avoid prolonged discussions like
> > this.
> 
> Just to be clear: the lib doesn't assign any special meaning to any tag
> (as it shouldn't).  The cli does, but only in the sense that it creates
> config files that designate certain tags for certain operations by
> default.  It's really in emacs where certain tags currently have
> unconfigurable meanings ("inbox").

The lib *does* assign special meaning to the tags it syncs to maildir
flags: draft, flagged, passed, replied, unread. (deleted used to be part
of the list.) The cli does have to request the syncing, but the mapping
is in the lib (flag2tag array in lib/message.cc).

BR,
Jani.

Re: [PATCH 3/4] config: only set search.exclude_tags to "deleted; spam; " during setup

[Jani Nikula]

On Mon, 23 Jan 2012 09:03:42 +0100, Pieter Praet <pieter@praet.org> wrote:
> On Mon, 23 Jan 2012 07:22:25 +0000, Jani Nikula <jani@nikula.org> wrote:
> > On Mon, 23 Jan 2012 06:05:27 +0100, Pieter Praet <pieter@praet.org> wrote:
> > > On Sun, 22 Jan 2012 14:53:41 -0800, Jameson Graef Rollins <jrollins@finestructure.net> wrote:
> > > > On Sun, 22 Jan 2012 23:14:13 +0100, Xavier Maillard <xavier@maillard.im> wrote:
> > > > > 
> > > > > On Thu, 19 Jan 2012 20:19:03 +0100, Pieter Praet <pieter@praet.org> wrote:
> > > > > > If the 'search.exclude_tags' option is missing from the config file,
> > > > > > its value is automatically set to "deleted;spam;".  Taking PoLS/DWIM
> > > > > > into account, this should probably only happen during setup.
> > > > > > 
> > > > > > This patch is actually Austin Clements' work:
> > > > > >   id:"20120117203211.GQ16740@mit.edu"
> > > > > 
> > > > > I do not think this is a sane default. As I told it in another post. I
> > > > > do not expect notmuch to skew my search queries not that I specifically
> > > > > asked.
> > > > 
> > > > Hi, Xavier.  Do you currently mark things as "deleted" or "spam"?  If
> > > > not, this would have no affect on your search results.  If you do, do
> > > > you currently expect those messages to show up in searches?  If so, why
> > > > did you mark them as "deleted" or "spam" to begin with?
> > > > 
> > > > I agree with your point in principle (ie. I don't generally want my
> > > > searches tampered with behind the scenes) but the issue here is about
> > > > messages that have been explicitly tagged as a form of "trash".  Trash
> > > > is by it's nature something you're trying to get rid of.  If you wanted
> > > > to find something in the future, why would you put it in the trash in
> > > > the first place?
> > > > 
> > > 
> > > You definitely have a point, but then again, who are we to assume that
> > > the terms "deleted" and "spam" have the *exact* same meaning for
> > > everyone?  (also see id:"8739bbo0br.fsf@praet.org")
> > 
> > "deleted" used to be a tag recognized by notmuch, and it used to sync to
> > the T (trashed) maildir flag. Even if notmuch won't delete any of your
> > mails now, I don't think you should use "deleted" on messages you want
> > to see again. Please let's not split hairs about this.
> > 
> 
> Agreed, but it might be nice to make a clear distinction between
> concepts and the actual tags mapped to them.  I'm not suggestion we
> redefine the term "deleted", but from an internationalization
> standpoint, we shouldn't prevent users from mapping e.g. "verwijderd",
> "supprimé", "gelöscht", ... to the concept "deleted".
> 
> > There really should be a definitive list of tags that are special to
> > lib/cli/emacs (like "inbox", "unread", "deleted", ...), or are
> > recommended for specific purposes (like "new" as an intermediate tag
> > before more sophisticated tagging), to avoid prolonged discussions like
> > this.
> > 
> 
> A list of recommended tags would definitely be nice, as long as they
> remain recommendations (as opposed to obligations), especially since
> there's really no reason to designate certain tags as being "special".

Whether there's reason or not, certain tags are special, for a fact, and
they are not just recommendations. Perhaps one day someone will
contribute patches to make them configurable, and separate the concepts
from the actual tags, but in the mean time it will be easier to just
document them for what they are.

BR,
Jani.

Re: [PATCH 3/4] config: only set search.exclude_tags to "deleted; spam; " during setup

[Jameson Graef Rollins]

[-- Attachment #1: Type: text/plain, Size: 393 bytes --]

On Mon, 23 Jan 2012 08:24:44 +0000, Jani Nikula <jani@nikula.org> wrote:
> The lib *does* assign special meaning to the tags it syncs to maildir
> flags: draft, flagged, passed, replied, unread. (deleted used to be part
> of the list.) The cli does have to request the syncing, but the mapping
> is in the lib (flag2tag array in lib/message.cc).

Ah, you're absolutely right.  My bad.

jamie.

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: [PATCH v3 6/6] NEWS: update "Tag exclusion" section

[Austin Clements]

Series LGTM.

Quoth Pieter Praet on Jan 23 at  6:41 am:
> ---
> 
> Added note re new configurations, as suggested by Austin:
>   id:"20120123044108.GS16740@mit.edu"
> 
>  NEWS |   19 +++++++++++++++----
>  1 files changed, 15 insertions(+), 4 deletions(-)
> 
> diff --git a/NEWS b/NEWS
> index 2c2d9e9..2acdce5 100644
> --- a/NEWS
> +++ b/NEWS
> @@ -13,10 +13,21 @@ Reply to sender
>  
>  Tag exclusion
>  
> -  Tags can be automatically excluded from search results unless they
> -  appear explicitly in a query.  By default, notmuch excludes the tags
> -  deleted and spam.  This can be changed using the new config setting
> -  search.auto_exclude_tags.
> +  Tags can be automatically excluded from search results by adding them
> +  to the new 'search.exclude_tags' option in the Notmuch config file.
> +
> +  This behaviour can be overridden by explicitly including an excluded
> +  tag in your query, for example:
> +
> +    notmuch search $your_query and tag:$excluded_tag
> +
> +  Existing users will probably want to run "notmuch setup" again to add
> +  the new well-commented [search] section to the configuration file.
> +
> +  For new configurations, accepting the default setting will cause the
> +  tags "deleted" and "spam" to be excluded, equivalent to running:
> +
> +    notmuch config set search.exclude_tags deleted spam
>  
>  Emacs Interface
>  ---------------

Re: [PATCH v2 1/6] search: rename auto_exclude_tags to {search, }exclude_tags

[David Bremner]

On Mon, 23 Jan 2012 05:22:32 +0100, Pieter Praet <pieter@praet.org> wrote:
> All other config-related functions and args include the section
> title in their name, so for the sake of consistency, mirror that.
> 

All six pushed,

d

Re: [PATCH] Automatically exclude tags in notmuch-show

[Mark Walters]

On Sun, 22 Jan 2012 20:52:22 -0500, Austin Clements <amdragon@MIT.EDU> wrote:
> Quoth Mark Walters on Jan 23 at  1:13 am:
> > On Sun, 22 Jan 2012 13:16:09 -0500, Austin Clements <amdragon@MIT.EDU> wrote:
> > > Quoth myself on Jan 20 at 12:18 pm:
> > > > Quoth Mark Walters on Jan 20 at 12:10 am:
> > > > > 
> > > > > Ok Having said this is trivial I have found a problem. What should
> > > > > notmuch do if you do something like
> > > > > 
> > > > > notmuch show id:<some-id>
> > > > > and that message is marked with a deleted tag? To be consistent with the
> > > > > other cases (where a deleted message is in a matched thread) we might
> > > > > want to return the message with the not-matched flag set (eg in
> > > > > JSON). But my patch doesn't, as it never even sees the thread since it
> > > > > doesn't match.
> > > > > 
> > > > > Looking at notmuch-show.c I think we should not apply the exclude tags
> > > > > to do_show_single, but usually should apply it to do_show. One solution
> > > > > which is simple and is at least close to right would be to get do_show
> > > > > to return the number of threads found. If this is zero then retry the
> > > > > query without the excludes (possible setting the match_flag to zero on
> > > > > each message since we know it does not match)
> > > > > 
> > > > > This is not a completely correct solution as if you ask notmuch-show to
> > > > > show more than one thread it might  threads which only contain deleted
> > > > > messages.
> > > > > 
> > > > > I can't see other good possibilities without slowing down the normal
> > > > > path a lot (eg find all threads that match the original query and then
> > > > > apply the argument above).
> > > > > 
> > > > > Any thoughts?
> > > > 
> > > > Oh dear.
> > > > 
> > > > Well, here's one idea.  Instead of doing a single thread query in
> > > > show, do a thread query without the exclusions and then a message
> > > > query with the exclusions.  Output all of the messages from the first
> > > > query, but use the results of the second query to determine which
> > > > messages are "matched".  The same could be accomplished in the library
> > > > somewhat more efficiently, but it's not obvious to me what the API
> > > > would be.
> > > 
> > > Here's a slightly crazier idea that's more library-invasive than the
> > > original approach, but probably better in the long run.
> > > 
> > > Have notmuch_query_search_* return everything and make exclusion a
> > > message flag like NOTMUCH_MESSAGE_FLAG_MATCH.  Tweak the definition of
> > > "matched" to mean "matched and not excluded" (specifically, a message
> > > would have the match flag or the excluded flag or neither, but not
> > > both).  Search would skip threads with zero matched messages and I
> > > think show would Just Work.
> > > 
> > > I can think of two ways to implement this.  notmuch_query_search_*
> > > could perform both the original query and the query with exclusions
> > > and use the docid set from the second to compute the "excluded"
> > > message flag.  Alternatively, it could examine the tags of each
> > > message directly to compute the flag.  The latter is probably easier
> > > to implement, but probably slower.
> > > 
> > > Thoughts?
> > 
> > I have now thought about this some more and think I understand your idea
> > (and how it would work) rather better now. 
> > 
> > I would suggest one small change: the flags for the messages returned
> > should be "independent": so a message can match the query or not, and it
> > can be excluded or not, with all 4 combinations being possible. (The
> > consumer of notmuch_query_search_* would extract the information it
> > wanted.)
> 
> I'd initially approached it this way, but went with redefining a
> "matched" messages because it had much less impact on the API.  For
> example, with the redefined "match",
> notmuch_thread_get_matched_messages still does the right thing for
> search and things like the thread subject can still be based on
> "matched" messages.  If we orthongonalize these flags, then we at
> least need to count matched non-excluded messages and provide an API
> to access this (while I don't have a solid argument against such an
> API it just seems weirdly specific to me).

Ok I have an initial implementation of this which I will post as a reply
to this thread: it does make the flags orthogonal but that would be easy
to change. If we do want to keep match to mean match and not excluded
then I would argue for a third flag so that the emacs frontend could see
all 4 possibilities. Note that in your suggestion we still need to do
something in notmuch_thread_get_matched_messages to set the subject etc
in threads with no matching non-excluded messages.

> My other concern is performance.  In thread queries, marking
> non-matched messages as excluded would require either an extra query
> per thread or a single query to match all excluded messages (not
> filtered by the primary query).  The former is prohibitive, though the
> latter might be acceptable (that might depend on how many things
> people mark as spam or deleted).  If the cost is too high, this
> suggests that we shouldn't mark non-matched messages as excluded, but
> then we're back to effectively having three levels of matching: not
> matched, matched but not excluded, and matched but excluded.

The implementation follows the second suggestion. I'll discuss
performance below.

> > I have thought about some implementation ideas but I think sorting is
> > going to be the deciding factor: what order should
> > notmuch_query_search_* return messages/threads? 
> 
> Yes.  This is exactly what I've been puzzling over, too.
> 
> > For notmuch_query_search_messages either it returns them all together
> > with the excluded messages marked, or returns all included ones, and
> > then all excluded one.
> 
> I would prefer them intermingled.  I feel like returning one and then
> the other is just exposing implementation details.  Plus, it's unclear
> if the order of the two groups should depend on the sort order, be
> configurable, or what.  Intermingling them seems like the obvious
> answer.
> 
> > For notmuch_query_search_threads it is less clear. Currently it returns
> > threads in order of first matching message. It is not clear what
> > matching means now: is matching and included, or just matching? If the
> > former then we will be returning some threads with no matching and
> > included messages so we need to decide where to put them in the order.
> 
> I would argue that, if the caller cares about the sort order of the
> results, it only makes sense for it to skip over threads consisting
> only of excluded messages, and if the caller doesn't care about the
> sort order, we can choose whatever's most convenient.

I have gone with this intermingled suggestion.

One other implementation detail: the code uses the exclude query
generated by your _notmuch_exclude_tags function; thus if the user
over-rides the exclude with tag:deleted then these messages will not be
marked excluded. I can't work out whether that is the right behaviour.

The code as it stands should finish where we started, but have the
infrastructure to show excluded messages. 

I think a nice interface would be for the emacs notmuch-search frontend
to take all messages and hide all threads with 0 matching messages, but
make them unhideable. (Possibly if there are no non-hidden threads it
could offer to show them.) The notmuch-show frontend could take all
messages but show only a header line for excluded messages (perhaps in a
different colour). It would jump to the first matching non excluded
message unless there aren't any, in which case it would go to the first
matching (necessarily excluded) message instead.

Performance: the code/notmuch is generally fast. On my computer/email
archive of circa 70000 the time to constuct the list of excluded
messages is of the order of 20 milliseconds. (Inherently it should not
be much slower since the thread return already has to construct a
complete list of all messages matching the query before it can return
any threads.)

However, in some case it will appear slower: if there are lots of
threads that only match in excluded messages then this code will waste
time constructing these "hidden threads". In other words the code will
be comparable to the pre-exclude notmuch: since it is constructing the
same threads.

Finally, the code is obviously an early draft to see what people
think. I know there are things that need freeing, and probably some
error returns that should be checked. 

Best wishes

Mark 

Re: [PATCH] Automatically exclude tags in notmuch-show

[Austin Clements]

Quoth Mark Walters on Jan 24 at  1:05 am:
> 
> On Sun, 22 Jan 2012 20:52:22 -0500, Austin Clements <amdragon@MIT.EDU> wrote:
> > Quoth Mark Walters on Jan 23 at  1:13 am:
> > > On Sun, 22 Jan 2012 13:16:09 -0500, Austin Clements <amdragon@MIT.EDU> wrote:
> > > > Quoth myself on Jan 20 at 12:18 pm:
> > > > > Quoth Mark Walters on Jan 20 at 12:10 am:
> > > > > > 
> > > > > > Ok Having said this is trivial I have found a problem. What should
> > > > > > notmuch do if you do something like
> > > > > > 
> > > > > > notmuch show id:<some-id>
> > > > > > and that message is marked with a deleted tag? To be consistent with the
> > > > > > other cases (where a deleted message is in a matched thread) we might
> > > > > > want to return the message with the not-matched flag set (eg in
> > > > > > JSON). But my patch doesn't, as it never even sees the thread since it
> > > > > > doesn't match.
> > > > > > 
> > > > > > Looking at notmuch-show.c I think we should not apply the exclude tags
> > > > > > to do_show_single, but usually should apply it to do_show. One solution
> > > > > > which is simple and is at least close to right would be to get do_show
> > > > > > to return the number of threads found. If this is zero then retry the
> > > > > > query without the excludes (possible setting the match_flag to zero on
> > > > > > each message since we know it does not match)
> > > > > > 
> > > > > > This is not a completely correct solution as if you ask notmuch-show to
> > > > > > show more than one thread it might  threads which only contain deleted
> > > > > > messages.
> > > > > > 
> > > > > > I can't see other good possibilities without slowing down the normal
> > > > > > path a lot (eg find all threads that match the original query and then
> > > > > > apply the argument above).
> > > > > > 
> > > > > > Any thoughts?
> > > > > 
> > > > > Oh dear.
> > > > > 
> > > > > Well, here's one idea.  Instead of doing a single thread query in
> > > > > show, do a thread query without the exclusions and then a message
> > > > > query with the exclusions.  Output all of the messages from the first
> > > > > query, but use the results of the second query to determine which
> > > > > messages are "matched".  The same could be accomplished in the library
> > > > > somewhat more efficiently, but it's not obvious to me what the API
> > > > > would be.
> > > > 
> > > > Here's a slightly crazier idea that's more library-invasive than the
> > > > original approach, but probably better in the long run.
> > > > 
> > > > Have notmuch_query_search_* return everything and make exclusion a
> > > > message flag like NOTMUCH_MESSAGE_FLAG_MATCH.  Tweak the definition of
> > > > "matched" to mean "matched and not excluded" (specifically, a message
> > > > would have the match flag or the excluded flag or neither, but not
> > > > both).  Search would skip threads with zero matched messages and I
> > > > think show would Just Work.
> > > > 
> > > > I can think of two ways to implement this.  notmuch_query_search_*
> > > > could perform both the original query and the query with exclusions
> > > > and use the docid set from the second to compute the "excluded"
> > > > message flag.  Alternatively, it could examine the tags of each
> > > > message directly to compute the flag.  The latter is probably easier
> > > > to implement, but probably slower.
> > > > 
> > > > Thoughts?
> > > 
> > > I have now thought about this some more and think I understand your idea
> > > (and how it would work) rather better now. 
> > > 
> > > I would suggest one small change: the flags for the messages returned
> > > should be "independent": so a message can match the query or not, and it
> > > can be excluded or not, with all 4 combinations being possible. (The
> > > consumer of notmuch_query_search_* would extract the information it
> > > wanted.)
> > 
> > I'd initially approached it this way, but went with redefining a
> > "matched" messages because it had much less impact on the API.  For
> > example, with the redefined "match",
> > notmuch_thread_get_matched_messages still does the right thing for
> > search and things like the thread subject can still be based on
> > "matched" messages.  If we orthongonalize these flags, then we at
> > least need to count matched non-excluded messages and provide an API
> > to access this (while I don't have a solid argument against such an
> > API it just seems weirdly specific to me).
> 
> Ok I have an initial implementation of this which I will post as a reply
> to this thread: it does make the flags orthogonal but that would be easy
> to change. If we do want to keep match to mean match and not excluded
> then I would argue for a third flag so that the emacs frontend could see
> all 4 possibilities. Note that in your suggestion we still need to do
> something in notmuch_thread_get_matched_messages to set the subject etc
> in threads with no matching non-excluded messages.

Cool.  I was starting to hack together an implementation too, but I'll
put that on hold.

Since notmuch_thread_get_matched_messages just counts matched
messages, it doesn't need anything special for threads with no matched
messages, but you're right that something has to be done about the
subjects of threads containing only excluded messages.

Could you CC me on the patch?  The list has been stuck all day.

> > My other concern is performance.  In thread queries, marking
> > non-matched messages as excluded would require either an extra query
> > per thread or a single query to match all excluded messages (not
> > filtered by the primary query).  The former is prohibitive, though the
> > latter might be acceptable (that might depend on how many things
> > people mark as spam or deleted).  If the cost is too high, this
> > suggests that we shouldn't mark non-matched messages as excluded, but
> > then we're back to effectively having three levels of matching: not
> > matched, matched but not excluded, and matched but excluded.
> 
> The implementation follows the second suggestion. I'll discuss
> performance below.
> 
> > > I have thought about some implementation ideas but I think sorting is
> > > going to be the deciding factor: what order should
> > > notmuch_query_search_* return messages/threads? 
> > 
> > Yes.  This is exactly what I've been puzzling over, too.
> > 
> > > For notmuch_query_search_messages either it returns them all together
> > > with the excluded messages marked, or returns all included ones, and
> > > then all excluded one.
> > 
> > I would prefer them intermingled.  I feel like returning one and then
> > the other is just exposing implementation details.  Plus, it's unclear
> > if the order of the two groups should depend on the sort order, be
> > configurable, or what.  Intermingling them seems like the obvious
> > answer.
> > 
> > > For notmuch_query_search_threads it is less clear. Currently it returns
> > > threads in order of first matching message. It is not clear what
> > > matching means now: is matching and included, or just matching? If the
> > > former then we will be returning some threads with no matching and
> > > included messages so we need to decide where to put them in the order.
> > 
> > I would argue that, if the caller cares about the sort order of the
> > results, it only makes sense for it to skip over threads consisting
> > only of excluded messages, and if the caller doesn't care about the
> > sort order, we can choose whatever's most convenient.
> 
> I have gone with this intermingled suggestion.
> 
> One other implementation detail: the code uses the exclude query
> generated by your _notmuch_exclude_tags function; thus if the user
> over-rides the exclude with tag:deleted then these messages will not be
> marked excluded. I can't work out whether that is the right behaviour.

I think that's reasonable.  Otherwise we need yet another state,
matched-but-excluded-but-overridden.

> The code as it stands should finish where we started, but have the
> infrastructure to show excluded messages. 
> 
> I think a nice interface would be for the emacs notmuch-search frontend
> to take all messages and hide all threads with 0 matching messages, but
> make them unhideable. (Possibly if there are no non-hidden threads it
> could offer to show them.) The notmuch-show frontend could take all

That does sound nice, though where would they go in the list?  Just at
the end?

> messages but show only a header line for excluded messages (perhaps in a
> different colour). It would jump to the first matching non excluded
> message unless there aren't any, in which case it would go to the first
> matching (necessarily excluded) message instead.

That sounds reasonable.  Alternatively, show could parallel search and
hide these messages, but point out that there are excluded messages
and offer to show them.

> Performance: the code/notmuch is generally fast. On my computer/email
> archive of circa 70000 the time to constuct the list of excluded
> messages is of the order of 20 milliseconds. (Inherently it should not
> be much slower since the thread return already has to construct a
> complete list of all messages matching the query before it can return
> any threads.)

I think this time will depend heavily on how many excluded messages
you have.  My performance concern is that it's not bounded by how many
messages match the original query.

> However, in some case it will appear slower: if there are lots of
> threads that only match in excluded messages then this code will waste
> time constructing these "hidden threads". In other words the code will
> be comparable to the pre-exclude notmuch: since it is constructing the
> same threads.
> 
> Finally, the code is obviously an early draft to see what people
> think. I know there are things that need freeing, and probably some
> error returns that should be checked. 
> 
> Best wishes
> 
> Mark 

[RFC PATCH 1/4] Add NOTMUCH_MESSAGE_FLAG_EXCLUDED flag

[Mark Walters]

Slightly refactor the exclude code to give the callers access to the
exclude query itself. There should be no functional change.

---
 lib/query.cc |   29 +++++++++++++++++++----------
 1 files changed, 19 insertions(+), 10 deletions(-)

diff --git a/lib/query.cc b/lib/query.cc
index 0b36602..c25b301 100644
--- a/lib/query.cc
+++ b/lib/query.cc
@@ -122,12 +122,15 @@ _notmuch_messages_destructor (notmuch_mset_messages_t *messages)
     return 0;
 }
 
-/* Return a query that does not match messages with the excluded tags
- * registered with the query.  Any tags that explicitly appear in
- * xquery will not be excluded. */
+/* Return a query that matches messages with the excluded tags
+ * registered with query.  Any tags that explicitly appear in xquery
+ * will not be excluded. The caller of this function has to combine
+ * the returned query appropriately.*/
 static Xapian::Query
 _notmuch_exclude_tags (notmuch_query_t *query, Xapian::Query xquery)
 {
+    Xapian::Query exclude_query = Xapian::Query::MatchNothing;
+
     for (notmuch_string_node_t *term = query->exclude_terms->head; term;
 	 term = term->next) {
 	Xapian::TermIterator it = xquery.get_terms_begin ();
@@ -137,10 +140,10 @@ _notmuch_exclude_tags (notmuch_query_t *query, Xapian::Query xquery)
 		break;
 	}
 	if (it == end)
-	    xquery = Xapian::Query (Xapian::Query::OP_AND_NOT,
-				    xquery, Xapian::Query (term->string));
+	    exclude_query = Xapian::Query (Xapian::Query::OP_OR,
+				    exclude_query, Xapian::Query (term->string));
     }
-    return xquery;
+    return exclude_query;
 }
 
 notmuch_messages_t *
@@ -168,7 +171,7 @@ notmuch_query_search_messages (notmuch_query_t *query)
 	Xapian::Query mail_query (talloc_asprintf (query, "%s%s",
 						   _find_prefix ("type"),
 						   "mail"));
-	Xapian::Query string_query, final_query;
+	Xapian::Query string_query, final_query, exclude_query;
 	Xapian::MSet mset;
 	unsigned int flags = (Xapian::QueryParser::FLAG_BOOLEAN |
 			      Xapian::QueryParser::FLAG_PHRASE |
@@ -188,7 +191,10 @@ notmuch_query_search_messages (notmuch_query_t *query)
 					 mail_query, string_query);
 	}
 
-	final_query = _notmuch_exclude_tags (query, final_query);
+	exclude_query = _notmuch_exclude_tags (query, final_query);
+
+	final_query = Xapian::Query (Xapian::Query::OP_AND_NOT,
+					 final_query, exclude_query);
 
 	enquire.set_weighting_scheme (Xapian::BoolWeight());
 
@@ -449,7 +455,7 @@ notmuch_query_count_messages (notmuch_query_t *query)
 	Xapian::Query mail_query (talloc_asprintf (query, "%s%s",
 						   _find_prefix ("type"),
 						   "mail"));
-	Xapian::Query string_query, final_query;
+	Xapian::Query string_query, final_query, exclude_query;
 	Xapian::MSet mset;
 	unsigned int flags = (Xapian::QueryParser::FLAG_BOOLEAN |
 			      Xapian::QueryParser::FLAG_PHRASE |
@@ -469,7 +475,10 @@ notmuch_query_count_messages (notmuch_query_t *query)
 					 mail_query, string_query);
 	}
 
-	final_query = _notmuch_exclude_tags (query, final_query);
+	exclude_query = _notmuch_exclude_tags (query, final_query);
+
+	final_query = Xapian::Query (Xapian::Query::OP_AND_NOT,
+					 final_query, exclude_query);
 
 	enquire.set_weighting_scheme(Xapian::BoolWeight());
 	enquire.set_docid_order(Xapian::Enquire::ASCENDING);
-- 
1.7.2.3

[RFC PATCH 2/4] Add NOTMUCH_MESSAGE_FLAG_EXCLUDED flag

[Mark Walters]

Form excluded doc_ids set and use that to exclude messages.
Should be no functional change.

---
 lib/notmuch-private.h |    1 +
 lib/query.cc          |   28 ++++++++++++++++++++++++++--
 2 files changed, 27 insertions(+), 2 deletions(-)

diff --git a/lib/notmuch-private.h b/lib/notmuch-private.h
index 7bf153e..e791bb0 100644
--- a/lib/notmuch-private.h
+++ b/lib/notmuch-private.h
@@ -401,6 +401,7 @@ typedef struct _notmuch_message_list {
  */
 struct visible _notmuch_messages {
     notmuch_bool_t is_of_list_type;
+    notmuch_doc_id_set_t *excluded_doc_ids;
     notmuch_message_node_t *iterator;
 };
 
diff --git a/lib/query.cc b/lib/query.cc
index c25b301..92fa834 100644
--- a/lib/query.cc
+++ b/lib/query.cc
@@ -57,6 +57,11 @@ struct visible _notmuch_threads {
     notmuch_doc_id_set_t match_set;
 };
 
+static notmuch_bool_t
+_notmuch_doc_id_set_init (void *ctx,
+			  notmuch_doc_id_set_t *doc_ids,
+			  GArray *arr);
+
 notmuch_query_t *
 notmuch_query_create (notmuch_database_t *notmuch,
 		      const char *query_string)
@@ -173,6 +178,7 @@ notmuch_query_search_messages (notmuch_query_t *query)
 						   "mail"));
 	Xapian::Query string_query, final_query, exclude_query;
 	Xapian::MSet mset;
+	Xapian::MSetIterator iterator;
 	unsigned int flags = (Xapian::QueryParser::FLAG_BOOLEAN |
 			      Xapian::QueryParser::FLAG_PHRASE |
 			      Xapian::QueryParser::FLAG_LOVEHATE |
@@ -193,8 +199,21 @@ notmuch_query_search_messages (notmuch_query_t *query)
 
 	exclude_query = _notmuch_exclude_tags (query, final_query);
 
-	final_query = Xapian::Query (Xapian::Query::OP_AND_NOT,
-					 final_query, exclude_query);
+	enquire.set_weighting_scheme (Xapian::BoolWeight());
+	enquire.set_query (exclude_query);
+
+	mset = enquire.get_mset (0, notmuch->xapian_db->get_doccount ());
+
+	GArray *excluded_doc_ids = g_array_new (FALSE, FALSE, sizeof (unsigned int));
+
+	for (iterator = mset.begin (); iterator != mset.end (); iterator++)
+	{
+	    unsigned int doc_id = *iterator;
+	    g_array_append_val (excluded_doc_ids, doc_id);
+	}
+	messages->base.excluded_doc_ids = talloc (query, _notmuch_doc_id_set);
+	_notmuch_doc_id_set_init (query, messages->base.excluded_doc_ids,
+				  excluded_doc_ids);
 
 	enquire.set_weighting_scheme (Xapian::BoolWeight());
 
@@ -294,6 +313,11 @@ _notmuch_mset_messages_move_to_next (notmuch_messages_t *messages)
     mset_messages = (notmuch_mset_messages_t *) messages;
 
     mset_messages->iterator++;
+
+    while ((mset_messages->iterator != mset_messages->iterator_end) &&
+	   (_notmuch_doc_id_set_contains (messages->excluded_doc_ids,
+					  *mset_messages->iterator)))
+	mset_messages->iterator++;
 }
 
 static notmuch_bool_t
-- 
1.7.2.3

[RFC PATCH 3/4] Add NOTMUCH_MESSAGE_FLAG_EXCLUDED flag

[Mark Walters]

Add the actual NOTMUCH_MESSAGE_FLAG_EXCLUDED flag.

---
 lib/notmuch-private.h |    1 +
 lib/notmuch.h         |    3 ++-
 lib/query.cc          |   11 +++++++----
 lib/thread.cc         |   14 ++++++++++----
 4 files changed, 20 insertions(+), 9 deletions(-)

diff --git a/lib/notmuch-private.h b/lib/notmuch-private.h
index e791bb0..cb3eca6 100644
--- a/lib/notmuch-private.h
+++ b/lib/notmuch-private.h
@@ -216,6 +216,7 @@ _notmuch_thread_create (void *ctx,
 			notmuch_database_t *notmuch,
 			unsigned int seed_doc_id,
 			notmuch_doc_id_set_t *match_set,
+			notmuch_doc_id_set_t *excluded_doc_ids,
 			notmuch_sort_t sort);
 
 /* message.cc */
diff --git a/lib/notmuch.h b/lib/notmuch.h
index 7929fe7..cf0d45d 100644
--- a/lib/notmuch.h
+++ b/lib/notmuch.h
@@ -895,7 +895,8 @@ notmuch_message_get_filenames (notmuch_message_t *message);
 
 /* Message flags */
 typedef enum _notmuch_message_flag {
-    NOTMUCH_MESSAGE_FLAG_MATCH
+    NOTMUCH_MESSAGE_FLAG_MATCH,
+    NOTMUCH_MESSAGE_FLAG_EXCLUDED
 } notmuch_message_flag_t;
 
 /* Get a value of a flag for the email corresponding to 'message'. */
diff --git a/lib/query.cc b/lib/query.cc
index 92fa834..69e32bd 100644
--- a/lib/query.cc
+++ b/lib/query.cc
@@ -55,6 +55,7 @@ struct visible _notmuch_threads {
     /* The set of matched docid's that have not been assigned to a
      * thread. Initially, this contains every docid in doc_ids. */
     notmuch_doc_id_set_t match_set;
+    notmuch_doc_id_set_t *excluded_doc_ids;
 };
 
 static notmuch_bool_t
@@ -302,6 +303,9 @@ _notmuch_mset_messages_get (notmuch_messages_t *messages)
 	INTERNAL_ERROR ("a messages iterator contains a non-existent document ID.\n");
     }
 
+    if (_notmuch_doc_id_set_contains (messages->excluded_doc_ids, doc_id))
+	notmuch_message_set_flag (message, NOTMUCH_MESSAGE_FLAG_EXCLUDED, TRUE);
+
     return message;
 }
 
@@ -314,10 +318,6 @@ _notmuch_mset_messages_move_to_next (notmuch_messages_t *messages)
 
     mset_messages->iterator++;
 
-    while ((mset_messages->iterator != mset_messages->iterator_end) &&
-	   (_notmuch_doc_id_set_contains (messages->excluded_doc_ids,
-					  *mset_messages->iterator)))
-	mset_messages->iterator++;
 }
 
 static notmuch_bool_t
@@ -403,6 +403,8 @@ notmuch_query_search_threads (notmuch_query_t *query)
 	notmuch_messages_move_to_next (messages);
     }
     threads->doc_id_pos = 0;
+    /* the excluded messages are in query context so this should be ok */
+    threads->excluded_doc_ids = messages->excluded_doc_ids;
 
     talloc_free (messages);
 
@@ -452,6 +454,7 @@ notmuch_threads_get (notmuch_threads_t *threads)
 				   threads->query->notmuch,
 				   doc_id,
 				   &threads->match_set,
+				   threads->excluded_doc_ids,
 				   threads->query->sort);
 }
 
diff --git a/lib/thread.cc b/lib/thread.cc
index 0435ee6..6ea2a44 100644
--- a/lib/thread.cc
+++ b/lib/thread.cc
@@ -302,7 +302,8 @@ _thread_set_subject_from_message (notmuch_thread_t *thread,
 static void
 _thread_add_matched_message (notmuch_thread_t *thread,
 			     notmuch_message_t *message,
-			     notmuch_sort_t sort)
+			     notmuch_sort_t sort,
+			     notmuch_bool_t excluded)
 {
     time_t date;
     notmuch_message_t *hashed_message;
@@ -321,7 +322,8 @@ _thread_add_matched_message (notmuch_thread_t *thread,
 	    _thread_set_subject_from_message (thread, message);
     }
 
-    thread->matched_messages++;
+    if (!excluded)
+	thread->matched_messages++;
 
     if (g_hash_table_lookup_extended (thread->message_hash,
 			    notmuch_message_get_message_id (message), NULL,
@@ -392,6 +394,7 @@ _notmuch_thread_create (void *ctx,
 			notmuch_database_t *notmuch,
 			unsigned int seed_doc_id,
 			notmuch_doc_id_set_t *match_set,
+			notmuch_doc_id_set_t *excluded_doc_ids,
 			notmuch_sort_t sort)
 {
     notmuch_thread_t *thread;
@@ -456,7 +459,9 @@ _notmuch_thread_create (void *ctx,
      * oldest or newest subject is desired. */
     notmuch_query_set_sort (thread_id_query, NOTMUCH_SORT_OLDEST_FIRST);
 
-    for (messages = notmuch_query_search_messages (thread_id_query);
+    messages = notmuch_query_search_messages (thread_id_query);
+    messages->excluded_doc_ids = excluded_doc_ids;
+    for (;
 	 notmuch_messages_valid (messages);
 	 notmuch_messages_move_to_next (messages))
     {
@@ -471,7 +476,8 @@ _notmuch_thread_create (void *ctx,
 
 	if ( _notmuch_doc_id_set_contains (match_set, doc_id)) {
 	    _notmuch_doc_id_set_remove (match_set, doc_id);
-	    _thread_add_matched_message (thread, message, sort);
+	    _thread_add_matched_message (thread, message, sort,
+					 _notmuch_doc_id_set_contains (excluded_doc_ids, doc_id));
 	}
 
 	_notmuch_message_close (message);
-- 
1.7.2.3

[PATCH 4/4] Add NOTMUCH_MESSAGE_FLAG_EXCLUDED flag

[Mark Walters]

Make notmuch-search.c respect the NOTMUCH_MESSAGE_FLAG_EXCLUDED.  All tests
should pass: i.e., the behaviour should be the same as before the
series.

---
 notmuch-search.c |    5 ++++-
 1 files changed, 4 insertions(+), 1 deletions(-)

diff --git a/notmuch-search.c b/notmuch-search.c
index 8867aab..b005abf 100644
--- a/notmuch-search.c
+++ b/notmuch-search.c
@@ -227,7 +227,7 @@ do_search_threads (const search_format_t *format,
 
 	thread = notmuch_threads_get (threads);
 
-	if (i < offset) {
+	if ((i < offset) || (notmuch_thread_get_matched_messages (thread) == 0)) {
 	    notmuch_thread_destroy (thread);
 	    continue;
 	}
@@ -318,6 +318,9 @@ do_search_messages (const search_format_t *format,
 
 	message = notmuch_messages_get (messages);
 
+	if (notmuch_message_get_flag (message, NOTMUCH_MESSAGE_FLAG_EXCLUDED))
+	    continue;
+
 	if (output == OUTPUT_FILES) {
 	    filenames = notmuch_message_get_filenames (message);
 
-- 
1.7.2.3

Re: [RFC PATCH 2/4] Add NOTMUCH_MESSAGE_FLAG_EXCLUDED flag

[Austin Clements]

The overall structure of this series looks great.  There's obviously a
lot of clean up to do, but I'll reply with a few high-level comments.

Quoth Mark Walters on Jan 24 at  1:18 am:
> Form excluded doc_ids set and use that to exclude messages.
> Should be no functional change.
> 
> ---
>  lib/notmuch-private.h |    1 +
>  lib/query.cc          |   28 ++++++++++++++++++++++++++--
>  2 files changed, 27 insertions(+), 2 deletions(-)
> 
> diff --git a/lib/notmuch-private.h b/lib/notmuch-private.h
> index 7bf153e..e791bb0 100644
> --- a/lib/notmuch-private.h
> +++ b/lib/notmuch-private.h
> @@ -401,6 +401,7 @@ typedef struct _notmuch_message_list {
>   */
>  struct visible _notmuch_messages {
>      notmuch_bool_t is_of_list_type;
> +    notmuch_doc_id_set_t *excluded_doc_ids;
>      notmuch_message_node_t *iterator;
>  };
>  
> diff --git a/lib/query.cc b/lib/query.cc
> index c25b301..92fa834 100644
> --- a/lib/query.cc
> +++ b/lib/query.cc
> @@ -57,6 +57,11 @@ struct visible _notmuch_threads {
>      notmuch_doc_id_set_t match_set;
>  };
>  
> +static notmuch_bool_t
> +_notmuch_doc_id_set_init (void *ctx,
> +			  notmuch_doc_id_set_t *doc_ids,
> +			  GArray *arr);
> +
>  notmuch_query_t *
>  notmuch_query_create (notmuch_database_t *notmuch,
>  		      const char *query_string)
> @@ -173,6 +178,7 @@ notmuch_query_search_messages (notmuch_query_t *query)
>  						   "mail"));
>  	Xapian::Query string_query, final_query, exclude_query;
>  	Xapian::MSet mset;
> +	Xapian::MSetIterator iterator;
>  	unsigned int flags = (Xapian::QueryParser::FLAG_BOOLEAN |
>  			      Xapian::QueryParser::FLAG_PHRASE |
>  			      Xapian::QueryParser::FLAG_LOVEHATE |
> @@ -193,8 +199,21 @@ notmuch_query_search_messages (notmuch_query_t *query)
>  
>  	exclude_query = _notmuch_exclude_tags (query, final_query);
>  
> -	final_query = Xapian::Query (Xapian::Query::OP_AND_NOT,
> -					 final_query, exclude_query);
> +	enquire.set_weighting_scheme (Xapian::BoolWeight());
> +	enquire.set_query (exclude_query);
> +
> +	mset = enquire.get_mset (0, notmuch->xapian_db->get_doccount ());
> +
> +	GArray *excluded_doc_ids = g_array_new (FALSE, FALSE, sizeof (unsigned int));
> +
> +	for (iterator = mset.begin (); iterator != mset.end (); iterator++)
> +	{
> +	    unsigned int doc_id = *iterator;
> +	    g_array_append_val (excluded_doc_ids, doc_id);
> +	}
> +	messages->base.excluded_doc_ids = talloc (query, _notmuch_doc_id_set);
> +	_notmuch_doc_id_set_init (query, messages->base.excluded_doc_ids,
> +				  excluded_doc_ids);

This might be inefficient for message-only queries, since it will
fetch *all* excluded docids.  This highlights a basic difference
between message and thread search: thread search can return messages
that don't match the original query and hence needs to know all
potentially excluded messages, while message search can only return
messages that match the original query.

It's entirely possible this doesn't matter because Xapian probably
still needs to fetch the full posting lists of the excluded terms, but
it would be worth doing a quick/hacky benchmark to verify this, with
enough excluded messages to make the cost non-trivial.

If it does matter, you could pass in a flag indicating if the exclude
query should be limited by the original query or not.  Or you could do
the limited exclude query in notmuch_query_search_messages and a
separate open-ended exclude query in notmuch_query_search_threads.

>  
>  	enquire.set_weighting_scheme (Xapian::BoolWeight());
>  
> @@ -294,6 +313,11 @@ _notmuch_mset_messages_move_to_next (notmuch_messages_t *messages)
>      mset_messages = (notmuch_mset_messages_t *) messages;
>  
>      mset_messages->iterator++;
> +
> +    while ((mset_messages->iterator != mset_messages->iterator_end) &&
> +	   (_notmuch_doc_id_set_contains (messages->excluded_doc_ids,
> +					  *mset_messages->iterator)))
> +	mset_messages->iterator++;

This seemed a little weird, since you remove it in the next patch.  Is
this just to keep the tests happy?  (If so, it would be worth
mentioning in the commit message; other reviewers will definitely have
the same question.)

>  }
>  
>  static notmuch_bool_t

Re: [RFC PATCH 3/4] Add NOTMUCH_MESSAGE_FLAG_EXCLUDED flag

[Austin Clements]

Quoth Mark Walters on Jan 24 at  1:18 am:
> Add the actual NOTMUCH_MESSAGE_FLAG_EXCLUDED flag.
> 
> ---
>  lib/notmuch-private.h |    1 +
>  lib/notmuch.h         |    3 ++-
>  lib/query.cc          |   11 +++++++----
>  lib/thread.cc         |   14 ++++++++++----
>  4 files changed, 20 insertions(+), 9 deletions(-)
> 
> diff --git a/lib/notmuch-private.h b/lib/notmuch-private.h
> index e791bb0..cb3eca6 100644
> --- a/lib/notmuch-private.h
> +++ b/lib/notmuch-private.h
> @@ -216,6 +216,7 @@ _notmuch_thread_create (void *ctx,
>  			notmuch_database_t *notmuch,
>  			unsigned int seed_doc_id,
>  			notmuch_doc_id_set_t *match_set,
> +			notmuch_doc_id_set_t *excluded_doc_ids,
>  			notmuch_sort_t sort);
>  
>  /* message.cc */
> diff --git a/lib/notmuch.h b/lib/notmuch.h
> index 7929fe7..cf0d45d 100644
> --- a/lib/notmuch.h
> +++ b/lib/notmuch.h
> @@ -895,7 +895,8 @@ notmuch_message_get_filenames (notmuch_message_t *message);
>  
>  /* Message flags */
>  typedef enum _notmuch_message_flag {
> -    NOTMUCH_MESSAGE_FLAG_MATCH
> +    NOTMUCH_MESSAGE_FLAG_MATCH,
> +    NOTMUCH_MESSAGE_FLAG_EXCLUDED
>  } notmuch_message_flag_t;
>  
>  /* Get a value of a flag for the email corresponding to 'message'. */
> diff --git a/lib/query.cc b/lib/query.cc
> index 92fa834..69e32bd 100644
> --- a/lib/query.cc
> +++ b/lib/query.cc
> @@ -55,6 +55,7 @@ struct visible _notmuch_threads {
>      /* The set of matched docid's that have not been assigned to a
>       * thread. Initially, this contains every docid in doc_ids. */
>      notmuch_doc_id_set_t match_set;
> +    notmuch_doc_id_set_t *excluded_doc_ids;
>  };
>  
>  static notmuch_bool_t
> @@ -302,6 +303,9 @@ _notmuch_mset_messages_get (notmuch_messages_t *messages)
>  	INTERNAL_ERROR ("a messages iterator contains a non-existent document ID.\n");
>      }
>  
> +    if (_notmuch_doc_id_set_contains (messages->excluded_doc_ids, doc_id))
> +	notmuch_message_set_flag (message, NOTMUCH_MESSAGE_FLAG_EXCLUDED, TRUE);
> +
>      return message;
>  }
>  
> @@ -314,10 +318,6 @@ _notmuch_mset_messages_move_to_next (notmuch_messages_t *messages)
>  
>      mset_messages->iterator++;
>  
> -    while ((mset_messages->iterator != mset_messages->iterator_end) &&
> -	   (_notmuch_doc_id_set_contains (messages->excluded_doc_ids,
> -					  *mset_messages->iterator)))
> -	mset_messages->iterator++;
>  }
>  
>  static notmuch_bool_t
> @@ -403,6 +403,8 @@ notmuch_query_search_threads (notmuch_query_t *query)
>  	notmuch_messages_move_to_next (messages);
>      }
>      threads->doc_id_pos = 0;
> +    /* the excluded messages are in query context so this should be ok */
> +    threads->excluded_doc_ids = messages->excluded_doc_ids;
>  
>      talloc_free (messages);
>  
> @@ -452,6 +454,7 @@ notmuch_threads_get (notmuch_threads_t *threads)
>  				   threads->query->notmuch,
>  				   doc_id,
>  				   &threads->match_set,
> +				   threads->excluded_doc_ids,
>  				   threads->query->sort);
>  }
>  
> diff --git a/lib/thread.cc b/lib/thread.cc
> index 0435ee6..6ea2a44 100644
> --- a/lib/thread.cc
> +++ b/lib/thread.cc
> @@ -302,7 +302,8 @@ _thread_set_subject_from_message (notmuch_thread_t *thread,
>  static void
>  _thread_add_matched_message (notmuch_thread_t *thread,
>  			     notmuch_message_t *message,
> -			     notmuch_sort_t sort)
> +			     notmuch_sort_t sort,
> +			     notmuch_bool_t excluded)
>  {
>      time_t date;
>      notmuch_message_t *hashed_message;
> @@ -321,7 +322,8 @@ _thread_add_matched_message (notmuch_thread_t *thread,
>  	    _thread_set_subject_from_message (thread, message);
>      }
>  
> -    thread->matched_messages++;
> +    if (!excluded)
> +	thread->matched_messages++;

I interpret notmuch_thread_get_matched_messages as returning the
number of messages with the "match" flag, which this approach changes.
I would suggest introducing a new, more flexible API along the lines
of

/* Return the number of messages in thread for which
 * notmuch_message_get_flag(msg) & msg == match.
 */
int
notmuch_thread_count_flags (notmuch_thread_t *thread, int mask, int match);

This would subsume the existing
notmuch_thread_get_{matched,total}_messages APIs.  It could either
iterate over the message list (which, curiously, we don't currently
track), or it could proactively count messages in an array indexed by
the message's flags (which wouldn't scale to large numbers of flags
but, for now, would only be of length 4).

>  
>      if (g_hash_table_lookup_extended (thread->message_hash,
>  			    notmuch_message_get_message_id (message), NULL,
> @@ -392,6 +394,7 @@ _notmuch_thread_create (void *ctx,
>  			notmuch_database_t *notmuch,
>  			unsigned int seed_doc_id,
>  			notmuch_doc_id_set_t *match_set,
> +			notmuch_doc_id_set_t *excluded_doc_ids,
>  			notmuch_sort_t sort)
>  {
>      notmuch_thread_t *thread;
> @@ -456,7 +459,9 @@ _notmuch_thread_create (void *ctx,
>       * oldest or newest subject is desired. */
>      notmuch_query_set_sort (thread_id_query, NOTMUCH_SORT_OLDEST_FIRST);
>  
> -    for (messages = notmuch_query_search_messages (thread_id_query);
> +    messages = notmuch_query_search_messages (thread_id_query);
> +    messages->excluded_doc_ids = excluded_doc_ids;
> +    for (;
>  	 notmuch_messages_valid (messages);
>  	 notmuch_messages_move_to_next (messages))
>      {
> @@ -471,7 +476,8 @@ _notmuch_thread_create (void *ctx,
>  
>  	if ( _notmuch_doc_id_set_contains (match_set, doc_id)) {
>  	    _notmuch_doc_id_set_remove (match_set, doc_id);
> -	    _thread_add_matched_message (thread, message, sort);
> +	    _thread_add_matched_message (thread, message, sort,
> +					 _notmuch_doc_id_set_contains (excluded_doc_ids, doc_id));
>  	}
>  
>  	_notmuch_message_close (message);

Re: [RFC PATCH 2/4] Add NOTMUCH_MESSAGE_FLAG_EXCLUDED flag

[Mark Walters]

On Mon, 23 Jan 2012 21:45:21 -0500, Austin Clements <amdragon@MIT.EDU> wrote:
> The overall structure of this series looks great.  There's obviously a
> lot of clean up to do, but I'll reply with a few high-level comments.
> 
> Quoth Mark Walters on Jan 24 at  1:18 am:
> > Form excluded doc_ids set and use that to exclude messages.
> > Should be no functional change.
> > 
> > ---
> >  lib/notmuch-private.h |    1 +
> >  lib/query.cc          |   28 ++++++++++++++++++++++++++--
> >  2 files changed, 27 insertions(+), 2 deletions(-)
> > 
> > diff --git a/lib/notmuch-private.h b/lib/notmuch-private.h
> > index 7bf153e..e791bb0 100644
> > --- a/lib/notmuch-private.h
> > +++ b/lib/notmuch-private.h
> > @@ -401,6 +401,7 @@ typedef struct _notmuch_message_list {
> >   */
> >  struct visible _notmuch_messages {
> >      notmuch_bool_t is_of_list_type;
> > +    notmuch_doc_id_set_t *excluded_doc_ids;
> >      notmuch_message_node_t *iterator;
> >  };
> >  
> > diff --git a/lib/query.cc b/lib/query.cc
> > index c25b301..92fa834 100644
> > --- a/lib/query.cc
> > +++ b/lib/query.cc
> > @@ -57,6 +57,11 @@ struct visible _notmuch_threads {
> >      notmuch_doc_id_set_t match_set;
> >  };
> >  
> > +static notmuch_bool_t
> > +_notmuch_doc_id_set_init (void *ctx,
> > +			  notmuch_doc_id_set_t *doc_ids,
> > +			  GArray *arr);
> > +
> >  notmuch_query_t *
> >  notmuch_query_create (notmuch_database_t *notmuch,
> >  		      const char *query_string)
> > @@ -173,6 +178,7 @@ notmuch_query_search_messages (notmuch_query_t *query)
> >  						   "mail"));
> >  	Xapian::Query string_query, final_query, exclude_query;
> >  	Xapian::MSet mset;
> > +	Xapian::MSetIterator iterator;
> >  	unsigned int flags = (Xapian::QueryParser::FLAG_BOOLEAN |
> >  			      Xapian::QueryParser::FLAG_PHRASE |
> >  			      Xapian::QueryParser::FLAG_LOVEHATE |
> > @@ -193,8 +199,21 @@ notmuch_query_search_messages (notmuch_query_t *query)
> >  
> >  	exclude_query = _notmuch_exclude_tags (query, final_query);
> >  
> > -	final_query = Xapian::Query (Xapian::Query::OP_AND_NOT,
> > -					 final_query, exclude_query);
> > +	enquire.set_weighting_scheme (Xapian::BoolWeight());
> > +	enquire.set_query (exclude_query);
> > +
> > +	mset = enquire.get_mset (0, notmuch->xapian_db->get_doccount ());
> > +
> > +	GArray *excluded_doc_ids = g_array_new (FALSE, FALSE, sizeof (unsigned int));
> > +
> > +	for (iterator = mset.begin (); iterator != mset.end (); iterator++)
> > +	{
> > +	    unsigned int doc_id = *iterator;
> > +	    g_array_append_val (excluded_doc_ids, doc_id);
> > +	}
> > +	messages->base.excluded_doc_ids = talloc (query, _notmuch_doc_id_set);
> > +	_notmuch_doc_id_set_init (query, messages->base.excluded_doc_ids,
> > +				  excluded_doc_ids);
> 
> This might be inefficient for message-only queries, since it will
> fetch *all* excluded docids.  This highlights a basic difference
> between message and thread search: thread search can return messages
> that don't match the original query and hence needs to know all
> potentially excluded messages, while message search can only return
> messages that match the original query.
> 
> It's entirely possible this doesn't matter because Xapian probably
> still needs to fetch the full posting lists of the excluded terms, but
> it would be worth doing a quick/hacky benchmark to verify this, with
> enough excluded messages to make the cost non-trivial.
> 
> If it does matter, you could pass in a flag indicating if the exclude
> query should be limited by the original query or not.  Or you could do
> the limited exclude query in notmuch_query_search_messages and a
> separate open-ended exclude query in notmuch_query_search_threads.

Yes I will benchmark that: I am just importing a large archive into
notmuch for testing. 

> >  	enquire.set_weighting_scheme (Xapian::BoolWeight());
> >  
> > @@ -294,6 +313,11 @@ _notmuch_mset_messages_move_to_next (notmuch_messages_t *messages)
> >      mset_messages = (notmuch_mset_messages_t *) messages;
> >  
> >      mset_messages->iterator++;
> > +
> > +    while ((mset_messages->iterator != mset_messages->iterator_end) &&
> > +	   (_notmuch_doc_id_set_contains (messages->excluded_doc_ids,
> > +					  *mset_messages->iterator)))
> > +	mset_messages->iterator++;
> 
> This seemed a little weird, since you remove it in the next patch.  Is
> this just to keep the tests happy?  (If so, it would be worth
> mentioning in the commit message; other reviewers will definitely have
> the same question.)

Essentially just to keep tests happy: or rather to try and make it easy
for a reviewer to see that the individual patch does not make any
functional change.

Best wishes 

Mark

Re: [PATCH 3/4] config: only set search.exclude_tags to "deleted; spam; " during setup

[Pieter Praet]

On Mon, 23 Jan 2012 08:31:23 +0000, Jani Nikula <jani@nikula.org> wrote:
> On Mon, 23 Jan 2012 09:03:42 +0100, Pieter Praet <pieter@praet.org> wrote:
> > On Mon, 23 Jan 2012 07:22:25 +0000, Jani Nikula <jani@nikula.org> wrote:
> > > On Mon, 23 Jan 2012 06:05:27 +0100, Pieter Praet <pieter@praet.org> wrote:
> > > > On Sun, 22 Jan 2012 14:53:41 -0800, Jameson Graef Rollins <jrollins@finestructure.net> wrote:
> > > > > On Sun, 22 Jan 2012 23:14:13 +0100, Xavier Maillard <xavier@maillard.im> wrote:
> > > > > > 
> > > > > > On Thu, 19 Jan 2012 20:19:03 +0100, Pieter Praet <pieter@praet.org> wrote:
> > > > > > > If the 'search.exclude_tags' option is missing from the config file,
> > > > > > > its value is automatically set to "deleted;spam;".  Taking PoLS/DWIM
> > > > > > > into account, this should probably only happen during setup.
> > > > > > > 
> > > > > > > This patch is actually Austin Clements' work:
> > > > > > >   id:"20120117203211.GQ16740@mit.edu"
> > > > > > 
> > > > > > I do not think this is a sane default. As I told it in another post. I
> > > > > > do not expect notmuch to skew my search queries not that I specifically
> > > > > > asked.
> > > > > 
> > > > > Hi, Xavier.  Do you currently mark things as "deleted" or "spam"?  If
> > > > > not, this would have no affect on your search results.  If you do, do
> > > > > you currently expect those messages to show up in searches?  If so, why
> > > > > did you mark them as "deleted" or "spam" to begin with?
> > > > > 
> > > > > I agree with your point in principle (ie. I don't generally want my
> > > > > searches tampered with behind the scenes) but the issue here is about
> > > > > messages that have been explicitly tagged as a form of "trash".  Trash
> > > > > is by it's nature something you're trying to get rid of.  If you wanted
> > > > > to find something in the future, why would you put it in the trash in
> > > > > the first place?
> > > > > 
> > > > 
> > > > You definitely have a point, but then again, who are we to assume that
> > > > the terms "deleted" and "spam" have the *exact* same meaning for
> > > > everyone?  (also see id:"8739bbo0br.fsf@praet.org")
> > > 
> > > "deleted" used to be a tag recognized by notmuch, and it used to sync to
> > > the T (trashed) maildir flag. Even if notmuch won't delete any of your
> > > mails now, I don't think you should use "deleted" on messages you want
> > > to see again. Please let's not split hairs about this.
> > > 
> > 
> > Agreed, but it might be nice to make a clear distinction between
> > concepts and the actual tags mapped to them.  I'm not suggestion we
> > redefine the term "deleted", but from an internationalization
> > standpoint, we shouldn't prevent users from mapping e.g. "verwijderd",
> > "supprimé", "gelöscht", ... to the concept "deleted".
> > 
> > > There really should be a definitive list of tags that are special to
> > > lib/cli/emacs (like "inbox", "unread", "deleted", ...), or are
> > > recommended for specific purposes (like "new" as an intermediate tag
> > > before more sophisticated tagging), to avoid prolonged discussions like
> > > this.
> > > 
> > 
> > A list of recommended tags would definitely be nice, as long as they
> > remain recommendations (as opposed to obligations), especially since
> > there's really no reason to designate certain tags as being "special".
> 
> Whether there's reason or not, certain tags are special, for a fact, and
> they are not just recommendations. [...]

My mistake.  Thanks for the correction!


> [...] Perhaps one day someone will
> contribute patches to make them configurable, and separate the concepts
> from the actual tags, but in the mean time it will be easier to just
> document them for what they are.
> 

Agreed.


> BR,
> Jani.


Peace

-- 
Pieter

Re: [PATCH 3/4] config: only set search.exclude_tags to "deleted; spam; " during setup

[Pieter Praet]

On Mon, 23 Jan 2012 08:24:44 +0000, Jani Nikula <jani@nikula.org> wrote:
> On Sun, 22 Jan 2012 23:38:40 -0800, Jameson Graef Rollins <jrollins@finestructure.net> wrote:
> > On Mon, 23 Jan 2012 07:22:25 +0000, Jani Nikula <jani@nikula.org> wrote:
> > > There really should be a definitive list of tags that are special to
> > > lib/cli/emacs (like "inbox", "unread", "deleted", ...), or are
> > > recommended for specific purposes (like "new" as an intermediate tag
> > > before more sophisticated tagging), to avoid prolonged discussions like
> > > this.
> > 
> > Just to be clear: the lib doesn't assign any special meaning to any tag
> > (as it shouldn't).  The cli does, but only in the sense that it creates
> > config files that designate certain tags for certain operations by
> > default.  It's really in emacs where certain tags currently have
> > unconfigurable meanings ("inbox").
> 
> The lib *does* assign special meaning to the tags it syncs to maildir
> flags: draft, flagged, passed, replied, unread. (deleted used to be part
> of the list.) The cli does have to request the syncing, but the mapping
> is in the lib (flag2tag array in lib/message.cc).
> 

Hmmm, right.

And seeing as how we can't simply replace the hardcoded tag names in
the flag2tag array with calls to the `notmuch_config_get_*' functions
(or move the Maildir sync stuff out of the lib altogether), custom
tag-to-concept mapping probably won't become a reality anytime soon.

Too bad...


> BR,
> Jani.


Peace

-- 
Pieter

Re: [RFC PATCH 2/4] Add NOTMUCH_MESSAGE_FLAG_EXCLUDED flag

[Mark Walters]

> >  	exclude_query = _notmuch_exclude_tags (query, final_query);
> >  
> > -	final_query = Xapian::Query (Xapian::Query::OP_AND_NOT,
> > -					 final_query, exclude_query);
> > +	enquire.set_weighting_scheme (Xapian::BoolWeight());
> > +	enquire.set_query (exclude_query);
> > +
> > +	mset = enquire.get_mset (0, notmuch->xapian_db->get_doccount ());
> > +
> > +	GArray *excluded_doc_ids = g_array_new (FALSE, FALSE, sizeof (unsigned int));
> > +
> > +	for (iterator = mset.begin (); iterator != mset.end (); iterator++)
> > +	{
> > +	    unsigned int doc_id = *iterator;
> > +	    g_array_append_val (excluded_doc_ids, doc_id);
> > +	}
> > +	messages->base.excluded_doc_ids = talloc (query, _notmuch_doc_id_set);
> > +	_notmuch_doc_id_set_init (query, messages->base.excluded_doc_ids,
> > +				  excluded_doc_ids);
> 
> This might be inefficient for message-only queries, since it will
> fetch *all* excluded docids.  This highlights a basic difference
> between message and thread search: thread search can return messages
> that don't match the original query and hence needs to know all
> potentially excluded messages, while message search can only return
> messages that match the original query.

I now have some benchmarks (not run enough times to be hugely accurate
so ignore minor differences). The full results are below. The summary
is:

Large-archive = 1 100 000 messages in 290 000 threads (about 10 years of
lkml). I mark 1 000 000 deleted
Small-archive = 70 000 messages in 35 000 threads. 10 000 marked
deleted.

Doing the initial exclude work on the big collection takes about 0.8s
and on the small collection about 0.01s. So any query to the big
collection takes at least 0.8s longer and this all occurs before any
results appear.

I then implemented the exclude doing it once for each thread query in
_notmuch_create_thread. Roughly this made any query 50% slower.

In normal front end use even the 0.8s is not totally unusable, but it is
totally unacceptable in the backend where a user might do something like

for i in ` notmuch search --output=threads  from:xxx ` ; 
do 
   notmuch search --output=messages $i; 
done

to list all messages in all matching threads.

So I think my conclusions are:

(1) message only queries must be done without the full exclude.
(2) thread queries which only match one message should not do the full
exclude
(3) it would be nice to switch between the two approaches depending on
size but I don't see how to do that without extra(!) queries
(4) One possible might be do something that say does thirty threads with
the by thread method and then if not finished does the full exclude.
(5) thread-by-thread might be best for  Jani's limit-match 
id:"1327692900-22926-1-git-send-email-jani@nikula.org" 

Obviously, anything setting an exclude flag like this will be slower
(since it is doing more work): the question is are either of these (or a
combination like (4) above) acceptable?

I now have a mostly working implementation from library to
emacs frontend and I do like the overall outcome.

The complete benchmarks are below

Best wishes

Mark

LARGE COLLECTION is 1,100,000 messages 290,000 threads 1,000,000 deleted
SMALL COLLECTION is 70,000 messages in 35,000 threads 10,000 deleted

benchmarks: all times in seconds, x/y/z means a query which matches x
threads with y matching messages and z messages in total. Ig or ignore
means with the tag-exclude turned off (i.e. with a query matching the
excluded tag). list all messages is the time for the for loop listed
above giving all message-ids for all messages in any thread matching a
query.

Finally the three columns are master with exclude code disabled,
thread-thread is doing excludes once per thread construction, and
in-advance does all the exclude work in advance as in the patches I posted.

In most cases the benchmark is the average of a lot of runs so the
database should have been as cached as one could hope.

			master-(all)	thread-thread	in-advance
LARGE COLLECTION		   			
show single message	0.016		0.018		0.78
search single message	0.015		0.016		0.78
search single with tag	0.015		0.015		0.009
945/2627/20000
query ignore		2.9		n/a		3
query 			2.9		4.2		3.8
list all messages (ig)	13		n/a		13
list all messages 	13		14		12mins
4754/13000/110000
query ignore		15.9		n/a		17
query 			15.9		22		17.6
only messages		1.25		1.26		1.9
177/483/1752		
query			0.3		0.42		1.1

search '*'		20mins		28mins		21.5mins			

SMALL COLLECTION
1500/2800/5600
query			1.8		2.7		2
list all messages	14.5		16.4		30
single message		0.008		0.008		0.018

search '*'		28		49		32

Re: [RFC PATCH 2/4] Add NOTMUCH_MESSAGE_FLAG_EXCLUDED flag

[Austin Clements]

Quoth Mark Walters on Jan 28 at 10:51 am:
> 
> > >  	exclude_query = _notmuch_exclude_tags (query, final_query);
> > >  
> > > -	final_query = Xapian::Query (Xapian::Query::OP_AND_NOT,
> > > -					 final_query, exclude_query);
> > > +	enquire.set_weighting_scheme (Xapian::BoolWeight());
> > > +	enquire.set_query (exclude_query);
> > > +
> > > +	mset = enquire.get_mset (0, notmuch->xapian_db->get_doccount ());
> > > +
> > > +	GArray *excluded_doc_ids = g_array_new (FALSE, FALSE, sizeof (unsigned int));
> > > +
> > > +	for (iterator = mset.begin (); iterator != mset.end (); iterator++)
> > > +	{
> > > +	    unsigned int doc_id = *iterator;
> > > +	    g_array_append_val (excluded_doc_ids, doc_id);
> > > +	}
> > > +	messages->base.excluded_doc_ids = talloc (query, _notmuch_doc_id_set);
> > > +	_notmuch_doc_id_set_init (query, messages->base.excluded_doc_ids,
> > > +				  excluded_doc_ids);
> > 
> > This might be inefficient for message-only queries, since it will
> > fetch *all* excluded docids.  This highlights a basic difference
> > between message and thread search: thread search can return messages
> > that don't match the original query and hence needs to know all
> > potentially excluded messages, while message search can only return
> > messages that match the original query.
> 
> I now have some benchmarks (not run enough times to be hugely accurate
> so ignore minor differences). The full results are below. The summary
> is:
> 
> Large-archive = 1 100 000 messages in 290 000 threads (about 10 years of
> lkml). I mark 1 000 000 deleted
> Small-archive = 70 000 messages in 35 000 threads. 10 000 marked
> deleted.
> 
> Doing the initial exclude work on the big collection takes about 0.8s
> and on the small collection about 0.01s. So any query to the big
> collection takes at least 0.8s longer and this all occurs before any
> results appear.

Interesting.  Do you know where that time is spent?

Also, it might be reasonable to assume that no more than, say, 10% of
a person's mail store is excluded, but maybe that depends on how
people use this feature.

> I then implemented the exclude doing it once for each thread query in
> _notmuch_create_thread. Roughly this made any query 50% slower.

That's not terrible.

> In normal front end use even the 0.8s is not totally unusable, but it is
> totally unacceptable in the backend where a user might do something like
> 
> for i in ` notmuch search --output=threads  from:xxx ` ; 
> do 
>    notmuch search --output=messages $i; 
> done
> 
> to list all messages in all matching threads.
> 
> So I think my conclusions are:
> 
> (1) message only queries must be done without the full exclude.
> (2) thread queries which only match one message should not do the full
> exclude
> (3) it would be nice to switch between the two approaches depending on
> size but I don't see how to do that without extra(!) queries
> (4) One possible might be do something that say does thirty threads with
> the by thread method and then if not finished does the full exclude.
> (5) thread-by-thread might be best for  Jani's limit-match 
> id:"1327692900-22926-1-git-send-email-jani@nikula.org" 
> 
> Obviously, anything setting an exclude flag like this will be slower
> (since it is doing more work): the question is are either of these (or a
> combination like (4) above) acceptable?

Or only mark matched messages as excluded.

Here's another idea (actually, a rehash of an old idea).  For message
search do two queries, the original query and "<original> AND
<exclude>", and use this to keep everything in order and mark excluded
messages.  For thread search, use message search results so it's easy
to both sort by unexcluded messages and include fully-excluded
threads, but compute the excluded flag (either just for unmatched
messages or for all messages) by examining each message's tags
directly (which thread_add_message already iterates over, so this is
easy and won't add any overhead).  If the excluded query is fast,
which I think it will be, I think this should get the best of all
worlds and be fairly straightforward to implement (no asymmetries
between the queries used for message and thread search).  It would be
easy and worth it to run the excluded query by hand on your test
corpus; I suspect it will be much faster than 0.8s because the query
already uses "Tmail", which is huge and doesn't seem to slow things
down.

> I now have a mostly working implementation from library to
> emacs frontend and I do like the overall outcome.

Awesome.

> The complete benchmarks are below
> 
> Best wishes
> 
> Mark
> 
> LARGE COLLECTION is 1,100,000 messages 290,000 threads 1,000,000 deleted
> SMALL COLLECTION is 70,000 messages in 35,000 threads 10,000 deleted
> 
> benchmarks: all times in seconds, x/y/z means a query which matches x
> threads with y matching messages and z messages in total. Ig or ignore
> means with the tag-exclude turned off (i.e. with a query matching the
> excluded tag). list all messages is the time for the for loop listed
> above giving all message-ids for all messages in any thread matching a
> query.
> 
> Finally the three columns are master with exclude code disabled,
> thread-thread is doing excludes once per thread construction, and
> in-advance does all the exclude work in advance as in the patches I posted.
> 
> In most cases the benchmark is the average of a lot of runs so the
> database should have been as cached as one could hope.
> 
> 			master-(all)	thread-thread	in-advance
> LARGE COLLECTION		   			
> show single message	0.016		0.018		0.78
> search single message	0.015		0.016		0.78
> search single with tag	0.015		0.015		0.009
> 945/2627/20000
> query ignore		2.9		n/a		3
> query 			2.9		4.2		3.8
> list all messages (ig)	13		n/a		13
> list all messages 	13		14		12mins
> 4754/13000/110000
> query ignore		15.9		n/a		17
> query 			15.9		22		17.6
> only messages		1.25		1.26		1.9
> 177/483/1752		
> query			0.3		0.42		1.1
> 
> search '*'		20mins		28mins		21.5mins			
> 
> SMALL COLLECTION
> 1500/2800/5600
> query			1.8		2.7		2
> list all messages	14.5		16.4		30
> single message		0.008		0.008		0.018
> 
> search '*'		28		49		32
> 

Re: [RFC PATCH 2/4] Add NOTMUCH_MESSAGE_FLAG_EXCLUDED flag

[Mark Walters]

On Sat, 28 Jan 2012 13:33:40 -0500, Austin Clements <amdragon@MIT.EDU> wrote:
> Quoth Mark Walters on Jan 28 at 10:51 am:
> > 
> > > >  	exclude_query = _notmuch_exclude_tags (query, final_query);
> > > >  
> > > > -	final_query = Xapian::Query (Xapian::Query::OP_AND_NOT,
> > > > -					 final_query, exclude_query);
> > > > +	enquire.set_weighting_scheme (Xapian::BoolWeight());
> > > > +	enquire.set_query (exclude_query);
> > > > +
> > > > +	mset = enquire.get_mset (0, notmuch->xapian_db->get_doccount ());
> > > > +
> > > > +	GArray *excluded_doc_ids = g_array_new (FALSE, FALSE, sizeof (unsigned int));
> > > > +
> > > > +	for (iterator = mset.begin (); iterator != mset.end (); iterator++)
> > > > +	{
> > > > +	    unsigned int doc_id = *iterator;
> > > > +	    g_array_append_val (excluded_doc_ids, doc_id);
> > > > +	}
> > > > +	messages->base.excluded_doc_ids = talloc (query, _notmuch_doc_id_set);
> > > > +	_notmuch_doc_id_set_init (query, messages->base.excluded_doc_ids,
> > > > +				  excluded_doc_ids);
> > > 
> > > This might be inefficient for message-only queries, since it will
> > > fetch *all* excluded docids.  This highlights a basic difference
> > > between message and thread search: thread search can return messages
> > > that don't match the original query and hence needs to know all
> > > potentially excluded messages, while message search can only return
> > > messages that match the original query.
> > 
> > I now have some benchmarks (not run enough times to be hugely accurate
> > so ignore minor differences). The full results are below. The summary
> > is:
> > 
> > Large-archive = 1 100 000 messages in 290 000 threads (about 10 years of
> > lkml). I mark 1 000 000 deleted
> > Small-archive = 70 000 messages in 35 000 threads. 10 000 marked
> > deleted.
> > 
> > Doing the initial exclude work on the big collection takes about 0.8s
> > and on the small collection about 0.01s. So any query to the big
> > collection takes at least 0.8s longer and this all occurs before any
> > results appear.
> 
> Interesting.  Do you know where that time is spent?
> 
> Also, it might be reasonable to assume that no more than, say, 10% of
> a person's mail store is excluded, but maybe that depends on how
> people use this feature.
> 
> > I then implemented the exclude doing it once for each thread query in
> > _notmuch_create_thread. Roughly this made any query 50% slower.
> 
> That's not terrible.
> 
> > In normal front end use even the 0.8s is not totally unusable, but it is
> > totally unacceptable in the backend where a user might do something like
> > 
> > for i in ` notmuch search --output=threads  from:xxx ` ; 
> > do 
> >    notmuch search --output=messages $i; 
> > done
> > 
> > to list all messages in all matching threads.
> > 
> > So I think my conclusions are:
> > 
> > (1) message only queries must be done without the full exclude.
> > (2) thread queries which only match one message should not do the full
> > exclude
> > (3) it would be nice to switch between the two approaches depending on
> > size but I don't see how to do that without extra(!) queries
> > (4) One possible might be do something that say does thirty threads with
> > the by thread method and then if not finished does the full exclude.
> > (5) thread-by-thread might be best for  Jani's limit-match 
> > id:"1327692900-22926-1-git-send-email-jani@nikula.org" 
> > 
> > Obviously, anything setting an exclude flag like this will be slower
> > (since it is doing more work): the question is are either of these (or a
> > combination like (4) above) acceptable?
> 
> Or only mark matched messages as excluded.
> 
> Here's another idea (actually, a rehash of an old idea).  For message
> search do two queries, the original query and "<original> AND
> <exclude>", and use this to keep everything in order and mark excluded
> messages.  For thread search, use message search results so it's easy
> to both sort by unexcluded messages and include fully-excluded
> threads, but compute the excluded flag (either just for unmatched
> messages or for all messages) by examining each message's tags
> directly (which thread_add_message already iterates over, so this is
> easy and won't add any overhead).  If the excluded query is fast,
> which I think it will be, I think this should get the best of all
> worlds and be fairly straightforward to implement (no asymmetries
> between the queries used for message and thread search).  It would be
> easy and worth it to run the excluded query by hand on your test
> corpus; I suspect it will be much faster than 0.8s because the query
> already uses "Tmail", which is huge and doesn't seem to slow things
> down.

I have tried your suggestion (still marking all messages) and it does
seem the way to go: the difference in speed is small from master is
small: between 0 and 10% for most of the tests. 

The code seems to work and I will post it in reply to this thread. 

The library code is reasonable (although whether messages matching an
exclude tag that has been specified in the query should be marked as
excluded is unclear).

The cli stuff needs thought (about what it should do rather than
how to do it).

I won't post the emacs stuff yet but I when I merge my various bits
together I should get different colour headerlines for excluded messages
and that they are initially shown collapsed.

Best wishes

Mark

[PATCH 1/4] Add exclude flag

[Mark Walters]

Slightly refactor the exclude code to give the callers access to the
exclude query itself. There should be no functional change.

---
 lib/query.cc |   29 +++++++++++++++++++----------
 1 files changed, 19 insertions(+), 10 deletions(-)

diff --git a/lib/query.cc b/lib/query.cc
index 0b36602..c25b301 100644
--- a/lib/query.cc
+++ b/lib/query.cc
@@ -122,12 +122,15 @@ _notmuch_messages_destructor (notmuch_mset_messages_t *messages)
     return 0;
 }
 
-/* Return a query that does not match messages with the excluded tags
- * registered with the query.  Any tags that explicitly appear in
- * xquery will not be excluded. */
+/* Return a query that matches messages with the excluded tags
+ * registered with query.  Any tags that explicitly appear in xquery
+ * will not be excluded. The caller of this function has to combine
+ * the returned query appropriately.*/
 static Xapian::Query
 _notmuch_exclude_tags (notmuch_query_t *query, Xapian::Query xquery)
 {
+    Xapian::Query exclude_query = Xapian::Query::MatchNothing;
+
     for (notmuch_string_node_t *term = query->exclude_terms->head; term;
 	 term = term->next) {
 	Xapian::TermIterator it = xquery.get_terms_begin ();
@@ -137,10 +140,10 @@ _notmuch_exclude_tags (notmuch_query_t *query, Xapian::Query xquery)
 		break;
 	}
 	if (it == end)
-	    xquery = Xapian::Query (Xapian::Query::OP_AND_NOT,
-				    xquery, Xapian::Query (term->string));
+	    exclude_query = Xapian::Query (Xapian::Query::OP_OR,
+				    exclude_query, Xapian::Query (term->string));
     }
-    return xquery;
+    return exclude_query;
 }
 
 notmuch_messages_t *
@@ -168,7 +171,7 @@ notmuch_query_search_messages (notmuch_query_t *query)
 	Xapian::Query mail_query (talloc_asprintf (query, "%s%s",
 						   _find_prefix ("type"),
 						   "mail"));
-	Xapian::Query string_query, final_query;
+	Xapian::Query string_query, final_query, exclude_query;
 	Xapian::MSet mset;
 	unsigned int flags = (Xapian::QueryParser::FLAG_BOOLEAN |
 			      Xapian::QueryParser::FLAG_PHRASE |
@@ -188,7 +191,10 @@ notmuch_query_search_messages (notmuch_query_t *query)
 					 mail_query, string_query);
 	}
 
-	final_query = _notmuch_exclude_tags (query, final_query);
+	exclude_query = _notmuch_exclude_tags (query, final_query);
+
+	final_query = Xapian::Query (Xapian::Query::OP_AND_NOT,
+					 final_query, exclude_query);
 
 	enquire.set_weighting_scheme (Xapian::BoolWeight());
 
@@ -449,7 +455,7 @@ notmuch_query_count_messages (notmuch_query_t *query)
 	Xapian::Query mail_query (talloc_asprintf (query, "%s%s",
 						   _find_prefix ("type"),
 						   "mail"));
-	Xapian::Query string_query, final_query;
+	Xapian::Query string_query, final_query, exclude_query;
 	Xapian::MSet mset;
 	unsigned int flags = (Xapian::QueryParser::FLAG_BOOLEAN |
 			      Xapian::QueryParser::FLAG_PHRASE |
@@ -469,7 +475,10 @@ notmuch_query_count_messages (notmuch_query_t *query)
 					 mail_query, string_query);
 	}
 
-	final_query = _notmuch_exclude_tags (query, final_query);
+	exclude_query = _notmuch_exclude_tags (query, final_query);
+
+	final_query = Xapian::Query (Xapian::Query::OP_AND_NOT,
+					 final_query, exclude_query);
 
 	enquire.set_weighting_scheme(Xapian::BoolWeight());
 	enquire.set_docid_order(Xapian::Enquire::ASCENDING);
-- 
1.7.2.3

[PATCH 2/4] Add exclude flag

[Mark Walters]

Make notmuch_query_search_messages set the exclude flag

Exclude flag will be added to notmuch_query_search threads later.
---
 lib/notmuch.h |    3 ++-
 lib/query.cc  |   34 +++++++++++++++++++++++++++++++---
 2 files changed, 33 insertions(+), 4 deletions(-)

diff --git a/lib/notmuch.h b/lib/notmuch.h
index 7929fe7..cf0d45d 100644
--- a/lib/notmuch.h
+++ b/lib/notmuch.h
@@ -895,7 +895,8 @@ notmuch_message_get_filenames (notmuch_message_t *message);
 
 /* Message flags */
 typedef enum _notmuch_message_flag {
-    NOTMUCH_MESSAGE_FLAG_MATCH
+    NOTMUCH_MESSAGE_FLAG_MATCH,
+    NOTMUCH_MESSAGE_FLAG_EXCLUDED
 } notmuch_message_flag_t;
 
 /* Get a value of a flag for the email corresponding to 'message'. */
diff --git a/lib/query.cc b/lib/query.cc
index c25b301..8ffafe5 100644
--- a/lib/query.cc
+++ b/lib/query.cc
@@ -57,6 +57,12 @@ struct visible _notmuch_threads {
     notmuch_doc_id_set_t match_set;
 };
 
+/* we need this in the message functions so forward declare */
+static notmuch_bool_t
+_notmuch_doc_id_set_init (void *ctx,
+			  notmuch_doc_id_set_t *doc_ids,
+			  GArray *arr);
+
 notmuch_query_t *
 notmuch_query_create (notmuch_database_t *notmuch,
 		      const char *query_string)
@@ -173,6 +179,7 @@ notmuch_query_search_messages (notmuch_query_t *query)
 						   "mail"));
 	Xapian::Query string_query, final_query, exclude_query;
 	Xapian::MSet mset;
+	Xapian::MSetIterator iterator;
 	unsigned int flags = (Xapian::QueryParser::FLAG_BOOLEAN |
 			      Xapian::QueryParser::FLAG_PHRASE |
 			      Xapian::QueryParser::FLAG_LOVEHATE |
@@ -190,11 +197,28 @@ notmuch_query_search_messages (notmuch_query_t *query)
 	    final_query = Xapian::Query (Xapian::Query::OP_AND,
 					 mail_query, string_query);
 	}
+	if (query->exclude_terms) {
+	    exclude_query = _notmuch_exclude_tags (query, final_query);
+	    exclude_query = Xapian::Query (Xapian::Query::OP_AND,
+					   exclude_query, final_query);
 
-	exclude_query = _notmuch_exclude_tags (query, final_query);
+	    enquire.set_weighting_scheme (Xapian::BoolWeight());
+	    enquire.set_query (exclude_query);
 
-	final_query = Xapian::Query (Xapian::Query::OP_AND_NOT,
-					 final_query, exclude_query);
+	    mset = enquire.get_mset (0, notmuch->xapian_db->get_doccount ());
+
+	    GArray *excluded_doc_ids = g_array_new (FALSE, FALSE, sizeof (unsigned int));
+
+	    for (iterator = mset.begin (); iterator != mset.end (); iterator++)
+	    {
+		unsigned int doc_id = *iterator;
+		g_array_append_val (excluded_doc_ids, doc_id);
+	    }
+	    messages->base.excluded_doc_ids = talloc (query, _notmuch_doc_id_set);
+	    _notmuch_doc_id_set_init (query, messages->base.excluded_doc_ids,
+				  excluded_doc_ids);
+	} else
+	    messages->base.excluded_doc_ids = NULL;
 
 	enquire.set_weighting_scheme (Xapian::BoolWeight());
 
@@ -283,6 +307,10 @@ _notmuch_mset_messages_get (notmuch_messages_t *messages)
 	INTERNAL_ERROR ("a messages iterator contains a non-existent document ID.\n");
     }
 
+    if ((messages->excluded_doc_ids) &&
+	(_notmuch_doc_id_set_contains (messages->excluded_doc_ids, doc_id)))
+	notmuch_message_set_flag (message, NOTMUCH_MESSAGE_FLAG_EXCLUDED, TRUE);
+
     return message;
 }
 
-- 
1.7.2.3

[PATCH 3/4] Add exclude flag

[Mark Walters]

Add the exclude flag to notmuch_query_search_threads

---
 lib/notmuch-private.h |   17 +++++++++++------
 lib/query.cc          |    1 +
 lib/thread.cc         |   18 +++++++++++++++---
 3 files changed, 27 insertions(+), 9 deletions(-)

diff --git a/lib/notmuch-private.h b/lib/notmuch-private.h
index 7bf153e..56b87c6 100644
--- a/lib/notmuch-private.h
+++ b/lib/notmuch-private.h
@@ -211,12 +211,8 @@ _notmuch_directory_get_document_id (notmuch_directory_t *directory);
 
 /* thread.cc */
 
-notmuch_thread_t *
-_notmuch_thread_create (void *ctx,
-			notmuch_database_t *notmuch,
-			unsigned int seed_doc_id,
-			notmuch_doc_id_set_t *match_set,
-			notmuch_sort_t sort);
+/* Definition of _notmuch_thread_create moved later since now uses
+ * string_list_t */
 
 /* message.cc */
 
@@ -401,6 +397,7 @@ typedef struct _notmuch_message_list {
  */
 struct visible _notmuch_messages {
     notmuch_bool_t is_of_list_type;
+    notmuch_doc_id_set_t *excluded_doc_ids;
     notmuch_message_node_t *iterator;
 };
 
@@ -491,6 +488,14 @@ notmuch_filenames_t *
 _notmuch_filenames_create (const void *ctx,
 			   notmuch_string_list_t *list);
 
+notmuch_thread_t *
+_notmuch_thread_create (void *ctx,
+			notmuch_database_t *notmuch,
+			unsigned int seed_doc_id,
+			notmuch_doc_id_set_t *match_set,
+			notmuch_string_list_t *excluded_terms,
+			notmuch_sort_t sort);
+
 #pragma GCC visibility pop
 
 NOTMUCH_END_DECLS
diff --git a/lib/query.cc b/lib/query.cc
index 8ffafe5..de95745 100644
--- a/lib/query.cc
+++ b/lib/query.cc
@@ -456,6 +456,7 @@ notmuch_threads_get (notmuch_threads_t *threads)
 				   threads->query->notmuch,
 				   doc_id,
 				   &threads->match_set,
+				   threads->query->exclude_terms,
 				   threads->query->sort);
 }
 
diff --git a/lib/thread.cc b/lib/thread.cc
index 0435ee6..6d65d52 100644
--- a/lib/thread.cc
+++ b/lib/thread.cc
@@ -214,7 +214,8 @@ _thread_cleanup_author (notmuch_thread_t *thread,
  */
 static void
 _thread_add_message (notmuch_thread_t *thread,
-		     notmuch_message_t *message)
+		     notmuch_message_t *message,
+		     notmuch_string_list_t *exclude_terms)
 {
     notmuch_tags_t *tags;
     const char *tag;
@@ -262,6 +263,15 @@ _thread_add_message (notmuch_thread_t *thread,
 	 notmuch_tags_move_to_next (tags))
     {
 	tag = notmuch_tags_get (tags);
+	/* mark excluded messages */
+	for (notmuch_string_node_t *term = exclude_terms->head; term;
+	     term = term->next) {
+	    /* we ignore initial 'K' */
+	    if (strcmp(tag, (term->string + 1)) == 0) {
+		notmuch_message_set_flag (message, NOTMUCH_MESSAGE_FLAG_EXCLUDED, TRUE);
+		break;
+	    }
+	}
 	g_hash_table_insert (thread->tags, xstrdup (tag), NULL);
     }
 }
@@ -321,7 +331,8 @@ _thread_add_matched_message (notmuch_thread_t *thread,
 	    _thread_set_subject_from_message (thread, message);
     }
 
-    thread->matched_messages++;
+    if (!notmuch_message_get_flag (message, NOTMUCH_MESSAGE_FLAG_EXCLUDED))
+	thread->matched_messages++;
 
     if (g_hash_table_lookup_extended (thread->message_hash,
 			    notmuch_message_get_message_id (message), NULL,
@@ -392,6 +403,7 @@ _notmuch_thread_create (void *ctx,
 			notmuch_database_t *notmuch,
 			unsigned int seed_doc_id,
 			notmuch_doc_id_set_t *match_set,
+			notmuch_string_list_t *exclude_terms,
 			notmuch_sort_t sort)
 {
     notmuch_thread_t *thread;
@@ -467,7 +479,7 @@ _notmuch_thread_create (void *ctx,
 	if (doc_id == seed_doc_id)
 	    message = seed_message;
 
-	_thread_add_message (thread, message);
+	_thread_add_message (thread, message, exclude_terms);
 
 	if ( _notmuch_doc_id_set_contains (match_set, doc_id)) {
 	    _notmuch_doc_id_set_remove (match_set, doc_id);
-- 
1.7.2.3

[PATCH 4/4] Add exclude flag

[Mark Walters]

Make notmuch-show.c respect the EXCLUDE flag.

---
 notmuch-show.c |   20 +++++++++++++++++---
 1 files changed, 17 insertions(+), 3 deletions(-)

diff --git a/notmuch-show.c b/notmuch-show.c
index dec799c..b55d2ba 100644
--- a/notmuch-show.c
+++ b/notmuch-show.c
@@ -193,10 +193,12 @@ _get_one_line_summary (const void *ctx, notmuch_message_t *message)
 static void
 format_message_text (unused (const void *ctx), notmuch_message_t *message, int indent)
 {
-    printf ("id:%s depth:%d match:%d filename:%s\n",
+    /* Could changing this could break users ? */
+    printf ("id:%s depth:%d match:%d excluded:%d filename:%s\n",
 	    notmuch_message_get_message_id (message),
 	    indent,
 	    notmuch_message_get_flag (message, NOTMUCH_MESSAGE_FLAG_MATCH),
+	    notmuch_message_get_flag (message, NOTMUCH_MESSAGE_FLAG_EXCLUDED),
 	    notmuch_message_get_filename (message));
 }
 
@@ -212,9 +214,10 @@ format_message_json (const void *ctx, notmuch_message_t *message, unused (int in
     date = notmuch_message_get_date (message);
     relative_date = notmuch_time_relative_date (ctx, date);
 
-    printf ("\"id\": %s, \"match\": %s, \"filename\": %s, \"timestamp\": %ld, \"date_relative\": \"%s\", \"tags\": [",
+    printf ("\"id\": %s, \"match\": %s, \"excluded\": %s, \"filename\": %s, \"timestamp\": %ld, \"date_relative\": \"%s\", \"tags\": [",
 	    json_quote_str (ctx_quote, notmuch_message_get_message_id (message)),
 	    notmuch_message_get_flag (message, NOTMUCH_MESSAGE_FLAG_MATCH) ? "true" : "false",
+	    notmuch_message_get_flag (message, NOTMUCH_MESSAGE_FLAG_EXCLUDED) ? "true" : "false",
 	    json_quote_str (ctx_quote, notmuch_message_get_filename (message)),
 	    date, relative_date);
 
@@ -293,6 +296,7 @@ _is_from_line (const char *line)
  *
  * http://qmail.org/qmail-manual-html/man5/mbox.html
  */
+/* should this do something with the exclude flag? */
 static void
 format_message_mbox (const void *ctx,
 		     notmuch_message_t *message,
@@ -947,9 +951,11 @@ do_show_single (void *ctx,
     notmuch_messages_t *messages;
     notmuch_message_t *message;
 
+    /* we need to fix something in notmuch_query_count for now just
+     * comment this out*/
     if (notmuch_query_count_messages (query) != 1) {
 	fprintf (stderr, "Error: search term did not match precisely one message.\n");
-	return 1;
+	/* return 1; */
     }
 
     messages = notmuch_query_search_messages (query);
@@ -1059,9 +1065,12 @@ notmuch_show_command (void *ctx, unused (int argc), unused (char *argv[]))
     char *opt;
     const notmuch_show_format_t *format = &format_text;
     notmuch_show_params_t params;
+    const char **search_exclude_tags;
+    size_t search_exclude_tags_length;
     int mbox = 0;
     int format_specified = 0;
     int i;
+    unsigned int j;
 
     params.entire_thread = 0;
     params.raw = 0;
@@ -1158,6 +1167,11 @@ notmuch_show_command (void *ctx, unused (int argc), unused (char *argv[]))
 	return 1;
     }
 
+    search_exclude_tags = notmuch_config_get_search_exclude_tags
+        (config, &search_exclude_tags_length);
+    for (j = 0; j < search_exclude_tags_length; j++)
+        notmuch_query_add_tag_exclude (query, search_exclude_tags[j]);
+
     /* if part was requested and format was not specified, use format=raw */
     if (params.part >= 0 && !format_specified)
 	format = &format_raw;
-- 
1.7.2.3

Re: [RFC PATCH 2/4] Add NOTMUCH_MESSAGE_FLAG_EXCLUDED flag

[Mark Walters]

> The cli stuff needs thought (about what it should do rather than
> how to do it).

Ok I have thought about the cli interface. My thoughts are as follows:

count/search/show should all have a --do-not-exclude option.

notmuch count:

        messages: just output count of matching not-excluded

        threads: count threads matching in a non-excluded message

notmuch search: 

        default/summary should output line as in the current patch with
        number matching being number non-excluded matching (so some will
        be zero)

        messages/files/tags should be matching not-excluded

        threads:  show thread ids of messages matching in a non-excluded message

notmuch show: 

        raw and part both deal with a single message so should output it
        regardless of exclude flags.

        text/json can give out the results including the exclude flag

        mbox only include matching not-excluded

The rationale is that all formats which can return an exclude flag do;
those that cannot omit the excluded results since the caller can just
call without setting the excludes if they want the full results.

Does that seem reasonable?

Best wishes

Mark

Re: [RFC PATCH 2/4] Add NOTMUCH_MESSAGE_FLAG_EXCLUDED flag

[Mark Walters]

Ok I now have a patch set which might be complete enough to be worth
reviewing. It is essentially complete and appears to work.

Things that still need doing: 
       updating the test suite. The series changes notmuch-show to
       output the exclude flag so several tests need updating. Of
       course, the new functionality needs some tests too.
       
       emacs/notmuch.el I think it would be nice to hide (make
       invisible) threads with no matching non-excluded messages (with a
       toggle for visibility) but that is definitely beyond my elisp
       skills.

The first patch of the series is not really part of the series: it adds
a --do-not-exclude option to tell the command not to exclude. I think
this is useful anyway, but it also simplifies behaviour decisions with
the excludes. For example notmuch count will only count matching
non-excluded messages but notmuch count --do-not-exclude will count 
all matching messages excluded or not.

One outstanding issue is that raised in
id:"20120124025331.GZ16740@mit.edu". I will need to think about
that. 

Best wishes

Mark

[PATCH 1/7] cli: add --do-not-exclude option to count and search.

[Mark Walters]

This option turns off the exclusion so all matching messages are
returned. We do not need to add this to show as notmuch-show does not
(yet) exclude.
---
 notmuch-count.c  |   12 ++++++++----
 notmuch-search.c |   12 ++++++++----
 2 files changed, 16 insertions(+), 8 deletions(-)

diff --git a/notmuch-count.c b/notmuch-count.c
index 63459fb..c88975e 100644
--- a/notmuch-count.c
+++ b/notmuch-count.c
@@ -37,6 +37,7 @@ notmuch_count_command (void *ctx, int argc, char *argv[])
     int output = OUTPUT_MESSAGES;
     const char **search_exclude_tags;
     size_t search_exclude_tags_length;
+    notmuch_bool_t do_not_exclude = FALSE;
     unsigned int i;
 
     notmuch_opt_desc_t options[] = {
@@ -44,6 +45,7 @@ notmuch_count_command (void *ctx, int argc, char *argv[])
 	  (notmuch_keyword_t []){ { "threads", OUTPUT_THREADS },
 				  { "messages", OUTPUT_MESSAGES },
 				  { 0, 0 } } },
+	{ NOTMUCH_OPT_BOOLEAN,  &do_not_exclude, "do-not-exclude", 'd', 0 },
 	{ 0, 0, 0, 0, 0 }
     };
 
@@ -78,10 +80,12 @@ notmuch_count_command (void *ctx, int argc, char *argv[])
 	return 1;
     }
 
-    search_exclude_tags = notmuch_config_get_search_exclude_tags
-	(config, &search_exclude_tags_length);
-    for (i = 0; i < search_exclude_tags_length; i++)
-	notmuch_query_add_tag_exclude (query, search_exclude_tags[i]);
+    if (!do_not_exclude) {
+	search_exclude_tags = notmuch_config_get_search_exclude_tags
+	    (config, &search_exclude_tags_length);
+	for (i = 0; i < search_exclude_tags_length; i++)
+	    notmuch_query_add_tag_exclude (query, search_exclude_tags[i]);
+    }
 
     switch (output) {
     case OUTPUT_MESSAGES:
diff --git a/notmuch-search.c b/notmuch-search.c
index d504051..084dd05 100644
--- a/notmuch-search.c
+++ b/notmuch-search.c
@@ -425,6 +425,7 @@ notmuch_search_command (void *ctx, int argc, char *argv[])
     int limit = -1; /* unlimited */
     const char **search_exclude_tags;
     size_t search_exclude_tags_length;
+    notmuch_bool_t do_not_exclude = FALSE;
     unsigned int i;
 
     enum { NOTMUCH_FORMAT_JSON, NOTMUCH_FORMAT_TEXT }
@@ -446,6 +447,7 @@ notmuch_search_command (void *ctx, int argc, char *argv[])
 				  { "files", OUTPUT_FILES },
 				  { "tags", OUTPUT_TAGS },
 				  { 0, 0 } } },
+        { NOTMUCH_OPT_BOOLEAN,  &do_not_exclude, "do-not-exclude", 'd', 0 },
 	{ NOTMUCH_OPT_INT, &offset, "offset", 'O', 0 },
 	{ NOTMUCH_OPT_INT, &limit, "limit", 'L', 0  },
 	{ 0, 0, 0, 0, 0 }
@@ -493,10 +495,12 @@ notmuch_search_command (void *ctx, int argc, char *argv[])
 
     notmuch_query_set_sort (query, sort);
 
-    search_exclude_tags = notmuch_config_get_search_exclude_tags
-	(config, &search_exclude_tags_length);
-    for (i = 0; i < search_exclude_tags_length; i++)
-	notmuch_query_add_tag_exclude (query, search_exclude_tags[i]);
+    if (!do_not_exclude) {
+	search_exclude_tags = notmuch_config_get_search_exclude_tags
+	    (config, &search_exclude_tags_length);
+	for (i = 0; i < search_exclude_tags_length; i++)
+	    notmuch_query_add_tag_exclude (query, search_exclude_tags[i]);
+    }
 
     switch (output) {
     default:
-- 
1.7.2.3

[PATCH 2/7] lib: Rearrange the exclude code in query.cc

[Mark Walters]

Slightly refactor the exclude code to give the callers access to the
exclude query itself. There should be no functional change.
---
 lib/query.cc |   29 +++++++++++++++++++----------
 1 files changed, 19 insertions(+), 10 deletions(-)

diff --git a/lib/query.cc b/lib/query.cc
index 0b36602..c25b301 100644
--- a/lib/query.cc
+++ b/lib/query.cc
@@ -122,12 +122,15 @@ _notmuch_messages_destructor (notmuch_mset_messages_t *messages)
     return 0;
 }
 
-/* Return a query that does not match messages with the excluded tags
- * registered with the query.  Any tags that explicitly appear in
- * xquery will not be excluded. */
+/* Return a query that matches messages with the excluded tags
+ * registered with query.  Any tags that explicitly appear in xquery
+ * will not be excluded. The caller of this function has to combine
+ * the returned query appropriately.*/
 static Xapian::Query
 _notmuch_exclude_tags (notmuch_query_t *query, Xapian::Query xquery)
 {
+    Xapian::Query exclude_query = Xapian::Query::MatchNothing;
+
     for (notmuch_string_node_t *term = query->exclude_terms->head; term;
 	 term = term->next) {
 	Xapian::TermIterator it = xquery.get_terms_begin ();
@@ -137,10 +140,10 @@ _notmuch_exclude_tags (notmuch_query_t *query, Xapian::Query xquery)
 		break;
 	}
 	if (it == end)
-	    xquery = Xapian::Query (Xapian::Query::OP_AND_NOT,
-				    xquery, Xapian::Query (term->string));
+	    exclude_query = Xapian::Query (Xapian::Query::OP_OR,
+				    exclude_query, Xapian::Query (term->string));
     }
-    return xquery;
+    return exclude_query;
 }
 
 notmuch_messages_t *
@@ -168,7 +171,7 @@ notmuch_query_search_messages (notmuch_query_t *query)
 	Xapian::Query mail_query (talloc_asprintf (query, "%s%s",
 						   _find_prefix ("type"),
 						   "mail"));
-	Xapian::Query string_query, final_query;
+	Xapian::Query string_query, final_query, exclude_query;
 	Xapian::MSet mset;
 	unsigned int flags = (Xapian::QueryParser::FLAG_BOOLEAN |
 			      Xapian::QueryParser::FLAG_PHRASE |
@@ -188,7 +191,10 @@ notmuch_query_search_messages (notmuch_query_t *query)
 					 mail_query, string_query);
 	}
 
-	final_query = _notmuch_exclude_tags (query, final_query);
+	exclude_query = _notmuch_exclude_tags (query, final_query);
+
+	final_query = Xapian::Query (Xapian::Query::OP_AND_NOT,
+					 final_query, exclude_query);
 
 	enquire.set_weighting_scheme (Xapian::BoolWeight());
 
@@ -449,7 +455,7 @@ notmuch_query_count_messages (notmuch_query_t *query)
 	Xapian::Query mail_query (talloc_asprintf (query, "%s%s",
 						   _find_prefix ("type"),
 						   "mail"));
-	Xapian::Query string_query, final_query;
+	Xapian::Query string_query, final_query, exclude_query;
 	Xapian::MSet mset;
 	unsigned int flags = (Xapian::QueryParser::FLAG_BOOLEAN |
 			      Xapian::QueryParser::FLAG_PHRASE |
@@ -469,7 +475,10 @@ notmuch_query_count_messages (notmuch_query_t *query)
 					 mail_query, string_query);
 	}
 
-	final_query = _notmuch_exclude_tags (query, final_query);
+	exclude_query = _notmuch_exclude_tags (query, final_query);
+
+	final_query = Xapian::Query (Xapian::Query::OP_AND_NOT,
+					 final_query, exclude_query);
 
 	enquire.set_weighting_scheme(Xapian::BoolWeight());
 	enquire.set_docid_order(Xapian::Enquire::ASCENDING);
-- 
1.7.2.3

[PATCH 3/7] lib: Make notmuch_query_search_messages set the exclude flag

[Mark Walters]

Add a flag NOTMUCH_MESSAGE_FLAG_EXCLUDED which is set by
notmuch_query_search_messages for excluded messages. Also add an
option omit_excluded_messages to the search that we do not want the
excludes at all.

This exclude flag will be added to notmuch_query_search threads in the
next patch.
---
 lib/notmuch-private.h |    1 +
 lib/notmuch.h         |    8 ++++++-
 lib/query.cc          |   52 +++++++++++++++++++++++++++++++++++++++++++++---
 3 files changed, 56 insertions(+), 5 deletions(-)

diff --git a/lib/notmuch-private.h b/lib/notmuch-private.h
index 7bf153e..e791bb0 100644
--- a/lib/notmuch-private.h
+++ b/lib/notmuch-private.h
@@ -401,6 +401,7 @@ typedef struct _notmuch_message_list {
  */
 struct visible _notmuch_messages {
     notmuch_bool_t is_of_list_type;
+    notmuch_doc_id_set_t *excluded_doc_ids;
     notmuch_message_node_t *iterator;
 };
 
diff --git a/lib/notmuch.h b/lib/notmuch.h
index 7929fe7..740d005 100644
--- a/lib/notmuch.h
+++ b/lib/notmuch.h
@@ -449,6 +449,11 @@ typedef enum {
 const char *
 notmuch_query_get_query_string (notmuch_query_t *query);
 
+/* specify whether to results should omit the excluded results rather
+ * than just marking them excluded */
+void
+notmuch_query_set_omit_excluded_messages (notmuch_query_t *query, notmuch_bool_t omit);
+
 /* Specify the sorting desired for this query. */
 void
 notmuch_query_set_sort (notmuch_query_t *query, notmuch_sort_t sort);
@@ -895,7 +900,8 @@ notmuch_message_get_filenames (notmuch_message_t *message);
 
 /* Message flags */
 typedef enum _notmuch_message_flag {
-    NOTMUCH_MESSAGE_FLAG_MATCH
+    NOTMUCH_MESSAGE_FLAG_MATCH,
+    NOTMUCH_MESSAGE_FLAG_EXCLUDED
 } notmuch_message_flag_t;
 
 /* Get a value of a flag for the email corresponding to 'message'. */
diff --git a/lib/query.cc b/lib/query.cc
index c25b301..7d165d2 100644
--- a/lib/query.cc
+++ b/lib/query.cc
@@ -28,6 +28,7 @@ struct _notmuch_query {
     const char *query_string;
     notmuch_sort_t sort;
     notmuch_string_list_t *exclude_terms;
+    notmuch_bool_t omit_excluded_messages;
 };
 
 typedef struct _notmuch_mset_messages {
@@ -57,6 +58,12 @@ struct visible _notmuch_threads {
     notmuch_doc_id_set_t match_set;
 };
 
+/* we need this in the message functions so forward declare */
+static notmuch_bool_t
+_notmuch_doc_id_set_init (void *ctx,
+			  notmuch_doc_id_set_t *doc_ids,
+			  GArray *arr);
+
 notmuch_query_t *
 notmuch_query_create (notmuch_database_t *notmuch,
 		      const char *query_string)
@@ -79,6 +86,8 @@ notmuch_query_create (notmuch_database_t *notmuch,
 
     query->exclude_terms = _notmuch_string_list_create (query);
 
+    query->omit_excluded_messages = FALSE;
+
     return query;
 }
 
@@ -89,6 +98,12 @@ notmuch_query_get_query_string (notmuch_query_t *query)
 }
 
 void
+notmuch_query_set_omit_excluded_messages (notmuch_query_t *query, notmuch_bool_t omit)
+{
+    query->omit_excluded_messages = omit;
+}
+
+void
 notmuch_query_set_sort (notmuch_query_t *query, notmuch_sort_t sort)
 {
     query->sort = sort;
@@ -173,6 +188,7 @@ notmuch_query_search_messages (notmuch_query_t *query)
 						   "mail"));
 	Xapian::Query string_query, final_query, exclude_query;
 	Xapian::MSet mset;
+	Xapian::MSetIterator iterator;
 	unsigned int flags = (Xapian::QueryParser::FLAG_BOOLEAN |
 			      Xapian::QueryParser::FLAG_PHRASE |
 			      Xapian::QueryParser::FLAG_LOVEHATE |
@@ -190,11 +206,35 @@ notmuch_query_search_messages (notmuch_query_t *query)
 	    final_query = Xapian::Query (Xapian::Query::OP_AND,
 					 mail_query, string_query);
 	}
+	messages->base.excluded_doc_ids = NULL;
+
+	if (query->exclude_terms) {
+	    exclude_query = _notmuch_exclude_tags (query, final_query);
+	    exclude_query = Xapian::Query (Xapian::Query::OP_AND,
+					   exclude_query, final_query);
+
+	    if (query->omit_excluded_messages)
+		final_query = Xapian::Query (Xapian::Query::OP_AND_NOT,
+					     final_query, exclude_query);
+	    else {
+		enquire.set_weighting_scheme (Xapian::BoolWeight());
+		enquire.set_query (exclude_query);
+
+		mset = enquire.get_mset (0, notmuch->xapian_db->get_doccount ());
+
+		GArray *excluded_doc_ids = g_array_new (FALSE, FALSE, sizeof (unsigned int));
+
+		for (iterator = mset.begin (); iterator != mset.end (); iterator++)
+		{
+		    unsigned int doc_id = *iterator;
+		    g_array_append_val (excluded_doc_ids, doc_id);
+		}
+		messages->base.excluded_doc_ids = talloc (query, _notmuch_doc_id_set);
+		_notmuch_doc_id_set_init (query, messages->base.excluded_doc_ids,
+					  excluded_doc_ids);
+	    }
+	}
 
-	exclude_query = _notmuch_exclude_tags (query, final_query);
-
-	final_query = Xapian::Query (Xapian::Query::OP_AND_NOT,
-					 final_query, exclude_query);
 
 	enquire.set_weighting_scheme (Xapian::BoolWeight());
 
@@ -283,6 +323,10 @@ _notmuch_mset_messages_get (notmuch_messages_t *messages)
 	INTERNAL_ERROR ("a messages iterator contains a non-existent document ID.\n");
     }
 
+    if ((messages->excluded_doc_ids) &&
+	(_notmuch_doc_id_set_contains (messages->excluded_doc_ids, doc_id)))
+	notmuch_message_set_flag (message, NOTMUCH_MESSAGE_FLAG_EXCLUDED, TRUE);
+
     return message;
 }
 
-- 
1.7.2.3

[PATCH 4/7] lib: Add the exclude flag to notmuch_query_search_threads

[Mark Walters]

Add the NOTMUCH_MESSAGE_FLAG_EXCLUDED flag to
notmuch_query_search_threads. Implemented by inspecting the tags
directly in _notmuch_thread_create/_thread_add_message rather than as
a Xapian query for speed reasons.
---
 lib/notmuch-private.h |   16 ++++++++++------
 lib/query.cc          |    1 +
 lib/thread.cc         |   18 +++++++++++++++---
 3 files changed, 26 insertions(+), 9 deletions(-)

diff --git a/lib/notmuch-private.h b/lib/notmuch-private.h
index e791bb0..56b87c6 100644
--- a/lib/notmuch-private.h
+++ b/lib/notmuch-private.h
@@ -211,12 +211,8 @@ _notmuch_directory_get_document_id (notmuch_directory_t *directory);
 
 /* thread.cc */
 
-notmuch_thread_t *
-_notmuch_thread_create (void *ctx,
-			notmuch_database_t *notmuch,
-			unsigned int seed_doc_id,
-			notmuch_doc_id_set_t *match_set,
-			notmuch_sort_t sort);
+/* Definition of _notmuch_thread_create moved later since now uses
+ * string_list_t */
 
 /* message.cc */
 
@@ -492,6 +488,14 @@ notmuch_filenames_t *
 _notmuch_filenames_create (const void *ctx,
 			   notmuch_string_list_t *list);
 
+notmuch_thread_t *
+_notmuch_thread_create (void *ctx,
+			notmuch_database_t *notmuch,
+			unsigned int seed_doc_id,
+			notmuch_doc_id_set_t *match_set,
+			notmuch_string_list_t *excluded_terms,
+			notmuch_sort_t sort);
+
 #pragma GCC visibility pop
 
 NOTMUCH_END_DECLS
diff --git a/lib/query.cc b/lib/query.cc
index 7d165d2..dee7ec0 100644
--- a/lib/query.cc
+++ b/lib/query.cc
@@ -472,6 +472,7 @@ notmuch_threads_get (notmuch_threads_t *threads)
 				   threads->query->notmuch,
 				   doc_id,
 				   &threads->match_set,
+				   threads->query->exclude_terms,
 				   threads->query->sort);
 }
 
diff --git a/lib/thread.cc b/lib/thread.cc
index 0435ee6..6d65d52 100644
--- a/lib/thread.cc
+++ b/lib/thread.cc
@@ -214,7 +214,8 @@ _thread_cleanup_author (notmuch_thread_t *thread,
  */
 static void
 _thread_add_message (notmuch_thread_t *thread,
-		     notmuch_message_t *message)
+		     notmuch_message_t *message,
+		     notmuch_string_list_t *exclude_terms)
 {
     notmuch_tags_t *tags;
     const char *tag;
@@ -262,6 +263,15 @@ _thread_add_message (notmuch_thread_t *thread,
 	 notmuch_tags_move_to_next (tags))
     {
 	tag = notmuch_tags_get (tags);
+	/* mark excluded messages */
+	for (notmuch_string_node_t *term = exclude_terms->head; term;
+	     term = term->next) {
+	    /* we ignore initial 'K' */
+	    if (strcmp(tag, (term->string + 1)) == 0) {
+		notmuch_message_set_flag (message, NOTMUCH_MESSAGE_FLAG_EXCLUDED, TRUE);
+		break;
+	    }
+	}
 	g_hash_table_insert (thread->tags, xstrdup (tag), NULL);
     }
 }
@@ -321,7 +331,8 @@ _thread_add_matched_message (notmuch_thread_t *thread,
 	    _thread_set_subject_from_message (thread, message);
     }
 
-    thread->matched_messages++;
+    if (!notmuch_message_get_flag (message, NOTMUCH_MESSAGE_FLAG_EXCLUDED))
+	thread->matched_messages++;
 
     if (g_hash_table_lookup_extended (thread->message_hash,
 			    notmuch_message_get_message_id (message), NULL,
@@ -392,6 +403,7 @@ _notmuch_thread_create (void *ctx,
 			notmuch_database_t *notmuch,
 			unsigned int seed_doc_id,
 			notmuch_doc_id_set_t *match_set,
+			notmuch_string_list_t *exclude_terms,
 			notmuch_sort_t sort)
 {
     notmuch_thread_t *thread;
@@ -467,7 +479,7 @@ _notmuch_thread_create (void *ctx,
 	if (doc_id == seed_doc_id)
 	    message = seed_message;
 
-	_thread_add_message (thread, message);
+	_thread_add_message (thread, message, exclude_terms);
 
 	if ( _notmuch_doc_id_set_contains (match_set, doc_id)) {
 	    _notmuch_doc_id_set_remove (match_set, doc_id);
-- 
1.7.2.3

[PATCH 5/7] cli: Make notmuch-show respect excludes.

[Mark Walters]

This adds the excludes to notmuch-show.c. We do not exclude when only
a single message (or part) is requested. notmuch-show will output the
exclude information when either text or json format is requested. As
this changes the output from notmuch-show it breaks many tests (in a
trivial and expected fashion).
---
 notmuch-show.c |   26 +++++++++++++++++++++-----
 1 files changed, 21 insertions(+), 5 deletions(-)

diff --git a/notmuch-show.c b/notmuch-show.c
index dec799c..681827f 100644
--- a/notmuch-show.c
+++ b/notmuch-show.c
@@ -193,10 +193,12 @@ _get_one_line_summary (const void *ctx, notmuch_message_t *message)
 static void
 format_message_text (unused (const void *ctx), notmuch_message_t *message, int indent)
 {
-    printf ("id:%s depth:%d match:%d filename:%s\n",
+    /* Could changing this could break users ? */
+    printf ("id:%s depth:%d match:%d excluded:%d filename:%s\n",
 	    notmuch_message_get_message_id (message),
 	    indent,
-	    notmuch_message_get_flag (message, NOTMUCH_MESSAGE_FLAG_MATCH),
+	    notmuch_message_get_flag (message, NOTMUCH_MESSAGE_FLAG_MATCH) ? 1 : 0,
+	    notmuch_message_get_flag (message, NOTMUCH_MESSAGE_FLAG_EXCLUDED) ? 1 : 0,
 	    notmuch_message_get_filename (message));
 }
 
@@ -212,9 +214,10 @@ format_message_json (const void *ctx, notmuch_message_t *message, unused (int in
     date = notmuch_message_get_date (message);
     relative_date = notmuch_time_relative_date (ctx, date);
 
-    printf ("\"id\": %s, \"match\": %s, \"filename\": %s, \"timestamp\": %ld, \"date_relative\": \"%s\", \"tags\": [",
+    printf ("\"id\": %s, \"match\": %s, \"excluded\": %s, \"filename\": %s, \"timestamp\": %ld, \"date_relative\": \"%s\", \"tags\": [",
 	    json_quote_str (ctx_quote, notmuch_message_get_message_id (message)),
 	    notmuch_message_get_flag (message, NOTMUCH_MESSAGE_FLAG_MATCH) ? "true" : "false",
+	    notmuch_message_get_flag (message, NOTMUCH_MESSAGE_FLAG_EXCLUDED) ? "true" : "false",
 	    json_quote_str (ctx_quote, notmuch_message_get_filename (message)),
 	    date, relative_date);
 
@@ -1059,9 +1062,13 @@ notmuch_show_command (void *ctx, unused (int argc), unused (char *argv[]))
     char *opt;
     const notmuch_show_format_t *format = &format_text;
     notmuch_show_params_t params;
+    const char **search_exclude_tags;
+    size_t search_exclude_tags_length;
     int mbox = 0;
     int format_specified = 0;
     int i;
+    notmuch_bool_t do_not_exclude = FALSE;
+    unsigned int j;
 
     params.entire_thread = 0;
     params.raw = 0;
@@ -1098,6 +1105,8 @@ notmuch_show_command (void *ctx, unused (int argc), unused (char *argv[]))
 	    params.part = atoi(argv[i] + sizeof ("--part=") - 1);
 	} else if (STRNCMP_LITERAL (argv[i], "--entire-thread") == 0) {
 	    params.entire_thread = 1;
+	} else if (STRNCMP_LITERAL (argv[i], "--do-not-exclude") == 0) {
+	    do_not_exclude = TRUE;
 	} else if ((STRNCMP_LITERAL (argv[i], "--verify") == 0) ||
 		   (STRNCMP_LITERAL (argv[i], "--decrypt") == 0)) {
 	    if (params.cryptoctx == NULL) {
@@ -1105,7 +1114,7 @@ notmuch_show_command (void *ctx, unused (int argc), unused (char *argv[]))
 		/* TODO: GMimePasswordRequestFunc */
 		if (NULL == (params.cryptoctx = g_mime_gpg_context_new(NULL, "gpg")))
 #else
-		GMimeSession* session = g_object_new(g_mime_session_get_type(), NULL);
+		    GMimeSession* session = g_object_new(g_mime_session_get_type(), NULL);
 		if (NULL == (params.cryptoctx = g_mime_gpg_context_new(session, "gpg")))
 #endif
 		    fprintf (stderr, "Failed to construct gpg context.\n");
@@ -1167,10 +1176,17 @@ notmuch_show_command (void *ctx, unused (int argc), unused (char *argv[]))
     if (params.raw && params.part < 0)
 	params.part = 0;
 
+    /* if a single message is requested we do not use search_excludes */
     if (params.part >= 0)
 	return do_show_single (ctx, query, format, &params);
     else
-	return do_show (ctx, query, format, &params);
+	if (!do_not_exclude) {
+	    search_exclude_tags = notmuch_config_get_search_exclude_tags
+		(config, &search_exclude_tags_length);
+	    for (j = 0; j < search_exclude_tags_length; j++)
+		notmuch_query_add_tag_exclude (query, search_exclude_tags[j]);
+	    return do_show (ctx, query, format, &params);
+	}
 
     notmuch_query_destroy (query);
     notmuch_database_close (notmuch);
-- 
1.7.2.3

[PATCH 6/7] cli: omit excluded messages in results where appropriate.

[Mark Walters]

In all cases of notmuch count/search/show where the results returned
cannot reflect the exclude flag return just the matched not-excluded
results. If the caller wishes to have all the matched results (i.e.,
including the excluded ones) they should call with the
--do-not-exclude option.

The relevant cases are
    count: both threads and messages
    search: all cases except the summary view
    show: mbox format
---
 notmuch-count.c  |    2 ++
 notmuch-search.c |    9 +++++++++
 notmuch-show.c   |    5 +++++
 3 files changed, 16 insertions(+), 0 deletions(-)

diff --git a/notmuch-count.c b/notmuch-count.c
index c88975e..1aa3f1a 100644
--- a/notmuch-count.c
+++ b/notmuch-count.c
@@ -87,6 +87,8 @@ notmuch_count_command (void *ctx, int argc, char *argv[])
 	    notmuch_query_add_tag_exclude (query, search_exclude_tags[i]);
     }
 
+    notmuch_query_set_omit_excluded_messages (query, TRUE);
+
     switch (output) {
     case OUTPUT_MESSAGES:
 	printf ("%u\n", notmuch_query_count_messages (query));
diff --git a/notmuch-search.c b/notmuch-search.c
index 084dd05..339ce82 100644
--- a/notmuch-search.c
+++ b/notmuch-search.c
@@ -207,6 +207,9 @@ do_search_threads (const search_format_t *format,
     int first_thread = 1;
     int i;
 
+    if (output == OUTPUT_THREADS)
+	notmuch_query_set_omit_excluded_messages (query, TRUE);
+
     if (offset < 0) {
 	offset += notmuch_query_count_threads (query);
 	if (offset < 0)
@@ -297,6 +300,8 @@ do_search_messages (const search_format_t *format,
     int first_message = 1;
     int i;
 
+    notmuch_query_set_omit_excluded_messages (query, TRUE);
+
     if (offset < 0) {
 	offset += notmuch_query_count_messages (query);
 	if (offset < 0)
@@ -368,6 +373,10 @@ do_search_tags (notmuch_database_t *notmuch,
     const char *tag;
     int first_tag = 1;
 
+    notmuch_query_set_omit_excluded_messages (query, TRUE);
+    /* should the following only special case if no excluded terms
+     * specified? */
+
     /* Special-case query of "*" for better performance. */
     if (strcmp (notmuch_query_get_query_string (query), "*") == 0) {
 	tags = notmuch_database_get_all_tags (notmuch);
diff --git a/notmuch-show.c b/notmuch-show.c
index 681827f..5d98724 100644
--- a/notmuch-show.c
+++ b/notmuch-show.c
@@ -1167,6 +1167,11 @@ notmuch_show_command (void *ctx, unused (int argc), unused (char *argv[]))
 	return 1;
     }
 
+    /* if format=mbox then we can not output excluded messages as
+     * there is no way to make the exclude flag available */
+    if (mbox)
+	notmuch_query_set_omit_excluded_messages (query, TRUE);
+
     /* if part was requested and format was not specified, use format=raw */
     if (params.part >= 0 && !format_specified)
 	format = &format_raw;
-- 
1.7.2.3

[PATCH 7/7] emacs: show: recognize the exclude flag.

[Mark Walters]

Show mode will recognize the exclude flag by not opening excluding
messages by default, and will start at the first matching non-excluded
message. If there are no matching non-excluded messages it will go to
the first matching (necessarily excluded) message.
---
 emacs/notmuch-show.el |   18 +++++++++++++++++-
 1 files changed, 17 insertions(+), 1 deletions(-)

diff --git a/emacs/notmuch-show.el b/emacs/notmuch-show.el
index 84ac624..8efadf6 100644
--- a/emacs/notmuch-show.el
+++ b/emacs/notmuch-show.el
@@ -905,7 +905,8 @@ current buffer, if possible."
 
     ;; Message visibility depends on whether it matched the search
     ;; criteria.
-    (notmuch-show-message-visible msg (plist-get msg :match))))
+    (notmuch-show-message-visible msg (and (plist-get msg :match)
+					   (not (plist-get msg :excluded))))))
 
 (defun notmuch-show-insert-tree (tree depth)
   "Insert the message tree TREE at depth DEPTH in the current thread."
@@ -1015,6 +1016,9 @@ buffer."
     ;; Move straight to the first open message
     (unless (notmuch-show-message-visible-p)
       (notmuch-show-next-open-message))
+    (when (eq (point) (point-max))
+      (goto-char (point-min))
+      (notmuch-show-next-matching-message))
 
     ;; Set the header line to the subject of the first open message.
     (setq header-line-format (notmuch-show-strip-re (notmuch-show-get-subject)))
@@ -1417,6 +1421,18 @@ any effects from previous calls to
 	  (notmuch-show-message-adjust))
       (goto-char (point-max)))))
 
+(defun notmuch-show-next-matching-message ()
+  "Show the next matching message."
+  (interactive)
+  (let (r)
+    (while (and (setq r (notmuch-show-goto-message-next))
+		(not (notmuch-show-get-prop :match))))
+    (if r
+	(progn
+	  (notmuch-show-mark-read)
+	  (notmuch-show-message-adjust))
+      (goto-char (point-max)))))
+
 (defun notmuch-show-previous-open-message ()
   "Show the previous message."
   (interactive)
-- 
1.7.2.3

Re: [PATCH 1/7] cli: add --do-not-exclude option to count and search.

[Austin Clements]

Quoth Mark Walters on Jan 29 at  6:39 pm:
> This option turns off the exclusion so all matching messages are
> returned. We do not need to add this to show as notmuch-show does not
> (yet) exclude.
> ---
>  notmuch-count.c  |   12 ++++++++----
>  notmuch-search.c |   12 ++++++++----
>  2 files changed, 16 insertions(+), 8 deletions(-)
> 
> diff --git a/notmuch-count.c b/notmuch-count.c
> index 63459fb..c88975e 100644
> --- a/notmuch-count.c
> +++ b/notmuch-count.c
> @@ -37,6 +37,7 @@ notmuch_count_command (void *ctx, int argc, char *argv[])
>      int output = OUTPUT_MESSAGES;
>      const char **search_exclude_tags;
>      size_t search_exclude_tags_length;
> +    notmuch_bool_t do_not_exclude = FALSE;
>      unsigned int i;
>  
>      notmuch_opt_desc_t options[] = {
> @@ -44,6 +45,7 @@ notmuch_count_command (void *ctx, int argc, char *argv[])
>  	  (notmuch_keyword_t []){ { "threads", OUTPUT_THREADS },
>  				  { "messages", OUTPUT_MESSAGES },
>  				  { 0, 0 } } },
> +	{ NOTMUCH_OPT_BOOLEAN,  &do_not_exclude, "do-not-exclude", 'd', 0 },

Maybe just "no-exclude"?  "do-not-exclude" seems needlessly verbose.

Also, you have an extra space after the first comma.

>  	{ 0, 0, 0, 0, 0 }
>      };
>  
> @@ -78,10 +80,12 @@ notmuch_count_command (void *ctx, int argc, char *argv[])
>  	return 1;
>      }
>  
> -    search_exclude_tags = notmuch_config_get_search_exclude_tags
> -	(config, &search_exclude_tags_length);
> -    for (i = 0; i < search_exclude_tags_length; i++)
> -	notmuch_query_add_tag_exclude (query, search_exclude_tags[i]);
> +    if (!do_not_exclude) {

You could move search_exclude_tags and search_exclude_tags_length in
here now that it's a block (but you don't have to).

> +	search_exclude_tags = notmuch_config_get_search_exclude_tags
> +	    (config, &search_exclude_tags_length);
> +	for (i = 0; i < search_exclude_tags_length; i++)
> +	    notmuch_query_add_tag_exclude (query, search_exclude_tags[i]);
> +    }
>  
>      switch (output) {
>      case OUTPUT_MESSAGES:
> diff --git a/notmuch-search.c b/notmuch-search.c
> index d504051..084dd05 100644
> --- a/notmuch-search.c
> +++ b/notmuch-search.c
> @@ -425,6 +425,7 @@ notmuch_search_command (void *ctx, int argc, char *argv[])
>      int limit = -1; /* unlimited */
>      const char **search_exclude_tags;
>      size_t search_exclude_tags_length;
> +    notmuch_bool_t do_not_exclude = FALSE;
>      unsigned int i;
>  
>      enum { NOTMUCH_FORMAT_JSON, NOTMUCH_FORMAT_TEXT }
> @@ -446,6 +447,7 @@ notmuch_search_command (void *ctx, int argc, char *argv[])
>  				  { "files", OUTPUT_FILES },
>  				  { "tags", OUTPUT_TAGS },
>  				  { 0, 0 } } },
> +        { NOTMUCH_OPT_BOOLEAN,  &do_not_exclude, "do-not-exclude", 'd', 0 },

Same.

>  	{ NOTMUCH_OPT_INT, &offset, "offset", 'O', 0 },
>  	{ NOTMUCH_OPT_INT, &limit, "limit", 'L', 0  },
>  	{ 0, 0, 0, 0, 0 }
> @@ -493,10 +495,12 @@ notmuch_search_command (void *ctx, int argc, char *argv[])
>  
>      notmuch_query_set_sort (query, sort);
>  
> -    search_exclude_tags = notmuch_config_get_search_exclude_tags
> -	(config, &search_exclude_tags_length);
> -    for (i = 0; i < search_exclude_tags_length; i++)
> -	notmuch_query_add_tag_exclude (query, search_exclude_tags[i]);
> +    if (!do_not_exclude) {
> +	search_exclude_tags = notmuch_config_get_search_exclude_tags
> +	    (config, &search_exclude_tags_length);
> +	for (i = 0; i < search_exclude_tags_length; i++)
> +	    notmuch_query_add_tag_exclude (query, search_exclude_tags[i]);
> +    }
>  
>      switch (output) {
>      default:

Re: [PATCH 3/7] lib: Make notmuch_query_search_messages set the exclude flag

[Austin Clements]

Quoth Mark Walters on Jan 29 at  6:39 pm:
> Add a flag NOTMUCH_MESSAGE_FLAG_EXCLUDED which is set by
> notmuch_query_search_messages for excluded messages. Also add an
> option omit_excluded_messages to the search that we do not want the
> excludes at all.
> 
> This exclude flag will be added to notmuch_query_search threads in the
> next patch.
> ---
>  lib/notmuch-private.h |    1 +
>  lib/notmuch.h         |    8 ++++++-
>  lib/query.cc          |   52 +++++++++++++++++++++++++++++++++++++++++++++---
>  3 files changed, 56 insertions(+), 5 deletions(-)
> 
> diff --git a/lib/notmuch-private.h b/lib/notmuch-private.h
> index 7bf153e..e791bb0 100644
> --- a/lib/notmuch-private.h
> +++ b/lib/notmuch-private.h
> @@ -401,6 +401,7 @@ typedef struct _notmuch_message_list {
>   */
>  struct visible _notmuch_messages {
>      notmuch_bool_t is_of_list_type;
> +    notmuch_doc_id_set_t *excluded_doc_ids;

I might be following the diff wrong, but shouldn't this be a field of
notmuch_mset_messages_t?  (Then it also doesn't have to be a pointer,
which is really how notmuch_doc_id_set_t was designed to be used.)

>      notmuch_message_node_t *iterator;
>  };
>  
> diff --git a/lib/notmuch.h b/lib/notmuch.h
> index 7929fe7..740d005 100644
> --- a/lib/notmuch.h
> +++ b/lib/notmuch.h
> @@ -449,6 +449,11 @@ typedef enum {
>  const char *
>  notmuch_query_get_query_string (notmuch_query_t *query);
>  
> +/* specify whether to results should omit the excluded results rather
> + * than just marking them excluded */
> +void
> +notmuch_query_set_omit_excluded_messages (notmuch_query_t *query, notmuch_bool_t omit);
> +

I don't think we should add this API.  The library behavior will not
change for library users that don't use excludes and library users
that do use excludes should by aware of the excluded flag and do the
appropriate thing.

I can see why this is handy in some cases, but I don't think it
provides enough utility to warrant becoming part of the permanent and
minimal library interface.

>  /* Specify the sorting desired for this query. */
>  void
>  notmuch_query_set_sort (notmuch_query_t *query, notmuch_sort_t sort);
> @@ -895,7 +900,8 @@ notmuch_message_get_filenames (notmuch_message_t *message);
>  
>  /* Message flags */
>  typedef enum _notmuch_message_flag {
> -    NOTMUCH_MESSAGE_FLAG_MATCH
> +    NOTMUCH_MESSAGE_FLAG_MATCH,
> +    NOTMUCH_MESSAGE_FLAG_EXCLUDED
>  } notmuch_message_flag_t;
>  
>  /* Get a value of a flag for the email corresponding to 'message'. */
> diff --git a/lib/query.cc b/lib/query.cc
> index c25b301..7d165d2 100644
> --- a/lib/query.cc
> +++ b/lib/query.cc
> @@ -28,6 +28,7 @@ struct _notmuch_query {
>      const char *query_string;
>      notmuch_sort_t sort;
>      notmuch_string_list_t *exclude_terms;
> +    notmuch_bool_t omit_excluded_messages;
>  };
>  
>  typedef struct _notmuch_mset_messages {
> @@ -57,6 +58,12 @@ struct visible _notmuch_threads {
>      notmuch_doc_id_set_t match_set;
>  };
>  
> +/* we need this in the message functions so forward declare */

Comments should start with a capital letter and end with a period.
(The code isn't completely consistent about this, but it is something
we're codifying in the upcoming style guide.)

> +static notmuch_bool_t
> +_notmuch_doc_id_set_init (void *ctx,
> +			  notmuch_doc_id_set_t *doc_ids,
> +			  GArray *arr);
> +
>  notmuch_query_t *
>  notmuch_query_create (notmuch_database_t *notmuch,
>  		      const char *query_string)
> @@ -79,6 +86,8 @@ notmuch_query_create (notmuch_database_t *notmuch,
>  
>      query->exclude_terms = _notmuch_string_list_create (query);
>  
> +    query->omit_excluded_messages = FALSE;
> +
>      return query;
>  }
>  
> @@ -89,6 +98,12 @@ notmuch_query_get_query_string (notmuch_query_t *query)
>  }
>  
>  void
> +notmuch_query_set_omit_excluded_messages (notmuch_query_t *query, notmuch_bool_t omit)
> +{
> +    query->omit_excluded_messages = omit;
> +}
> +
> +void
>  notmuch_query_set_sort (notmuch_query_t *query, notmuch_sort_t sort)
>  {
>      query->sort = sort;
> @@ -173,6 +188,7 @@ notmuch_query_search_messages (notmuch_query_t *query)
>  						   "mail"));
>  	Xapian::Query string_query, final_query, exclude_query;
>  	Xapian::MSet mset;
> +	Xapian::MSetIterator iterator;
>  	unsigned int flags = (Xapian::QueryParser::FLAG_BOOLEAN |
>  			      Xapian::QueryParser::FLAG_PHRASE |
>  			      Xapian::QueryParser::FLAG_LOVEHATE |
> @@ -190,11 +206,35 @@ notmuch_query_search_messages (notmuch_query_t *query)
>  	    final_query = Xapian::Query (Xapian::Query::OP_AND,
>  					 mail_query, string_query);
>  	}
> +	messages->base.excluded_doc_ids = NULL;
> +
> +	if (query->exclude_terms) {
> +	    exclude_query = _notmuch_exclude_tags (query, final_query);
> +	    exclude_query = Xapian::Query (Xapian::Query::OP_AND,
> +					   exclude_query, final_query);
> +
> +	    if (query->omit_excluded_messages)
> +		final_query = Xapian::Query (Xapian::Query::OP_AND_NOT,
> +					     final_query, exclude_query);
> +	    else {
> +		enquire.set_weighting_scheme (Xapian::BoolWeight());
> +		enquire.set_query (exclude_query);
> +
> +		mset = enquire.get_mset (0, notmuch->xapian_db->get_doccount ());
> +
> +		GArray *excluded_doc_ids = g_array_new (FALSE, FALSE, sizeof (unsigned int));
> +
> +		for (iterator = mset.begin (); iterator != mset.end (); iterator++)
> +		{

No newline before the brace.

> +		    unsigned int doc_id = *iterator;
> +		    g_array_append_val (excluded_doc_ids, doc_id);
> +		}
> +		messages->base.excluded_doc_ids = talloc (query, _notmuch_doc_id_set);
> +		_notmuch_doc_id_set_init (query, messages->base.excluded_doc_ids,
> +					  excluded_doc_ids);

Don't forget to g_array_unref excluded_doc_ids.

> +	    }
> +	}
>  
> -	exclude_query = _notmuch_exclude_tags (query, final_query);
> -
> -	final_query = Xapian::Query (Xapian::Query::OP_AND_NOT,
> -					 final_query, exclude_query);
>  
>  	enquire.set_weighting_scheme (Xapian::BoolWeight());
>  
> @@ -283,6 +323,10 @@ _notmuch_mset_messages_get (notmuch_messages_t *messages)
>  	INTERNAL_ERROR ("a messages iterator contains a non-existent document ID.\n");
>      }
>  
> +    if ((messages->excluded_doc_ids) &&
> +	(_notmuch_doc_id_set_contains (messages->excluded_doc_ids, doc_id)))

No need for so many parens (just a nit).

> +	notmuch_message_set_flag (message, NOTMUCH_MESSAGE_FLAG_EXCLUDED, TRUE);
> +
>      return message;
>  }
>  

Re: [PATCH 4/7] lib: Add the exclude flag to notmuch_query_search_threads

[Austin Clements]

Quoth Mark Walters on Jan 29 at  6:39 pm:
> Add the NOTMUCH_MESSAGE_FLAG_EXCLUDED flag to
> notmuch_query_search_threads. Implemented by inspecting the tags
> directly in _notmuch_thread_create/_thread_add_message rather than as
> a Xapian query for speed reasons.
> ---
>  lib/notmuch-private.h |   16 ++++++++++------
>  lib/query.cc          |    1 +
>  lib/thread.cc         |   18 +++++++++++++++---
>  3 files changed, 26 insertions(+), 9 deletions(-)
> 
> diff --git a/lib/notmuch-private.h b/lib/notmuch-private.h
> index e791bb0..56b87c6 100644
> --- a/lib/notmuch-private.h
> +++ b/lib/notmuch-private.h
> @@ -211,12 +211,8 @@ _notmuch_directory_get_document_id (notmuch_directory_t *directory);
>  
>  /* thread.cc */
>  
> -notmuch_thread_t *
> -_notmuch_thread_create (void *ctx,
> -			notmuch_database_t *notmuch,
> -			unsigned int seed_doc_id,
> -			notmuch_doc_id_set_t *match_set,
> -			notmuch_sort_t sort);
> +/* Definition of _notmuch_thread_create moved later since now uses
> + * string_list_t */

Naw, leave the definition here along with the other things from
thread.cc and just add a

  typedef struct _notmuch_string_list notmuch_string_list_t;

along with the typedef for notmuch_doc_id_set_t near the top.  (You
might also have to tweak the typedef of notmuch_string_list_t later so
it's just the struct definition.)

>  
>  /* message.cc */
>  
> @@ -492,6 +488,14 @@ notmuch_filenames_t *
>  _notmuch_filenames_create (const void *ctx,
>  			   notmuch_string_list_t *list);
>  
> +notmuch_thread_t *
> +_notmuch_thread_create (void *ctx,
> +			notmuch_database_t *notmuch,
> +			unsigned int seed_doc_id,
> +			notmuch_doc_id_set_t *match_set,
> +			notmuch_string_list_t *excluded_terms,
> +			notmuch_sort_t sort);
> +
>  #pragma GCC visibility pop
>  
>  NOTMUCH_END_DECLS
> diff --git a/lib/query.cc b/lib/query.cc
> index 7d165d2..dee7ec0 100644
> --- a/lib/query.cc
> +++ b/lib/query.cc
> @@ -472,6 +472,7 @@ notmuch_threads_get (notmuch_threads_t *threads)
>  				   threads->query->notmuch,
>  				   doc_id,
>  				   &threads->match_set,
> +				   threads->query->exclude_terms,
>  				   threads->query->sort);
>  }
>  
> diff --git a/lib/thread.cc b/lib/thread.cc
> index 0435ee6..6d65d52 100644
> --- a/lib/thread.cc
> +++ b/lib/thread.cc
> @@ -214,7 +214,8 @@ _thread_cleanup_author (notmuch_thread_t *thread,
>   */
>  static void
>  _thread_add_message (notmuch_thread_t *thread,
> -		     notmuch_message_t *message)
> +		     notmuch_message_t *message,
> +		     notmuch_string_list_t *exclude_terms)
>  {
>      notmuch_tags_t *tags;
>      const char *tag;
> @@ -262,6 +263,15 @@ _thread_add_message (notmuch_thread_t *thread,
>  	 notmuch_tags_move_to_next (tags))
>      {
>  	tag = notmuch_tags_get (tags);
> +	/* mark excluded messages */

Capital and period.

> +	for (notmuch_string_node_t *term = exclude_terms->head; term;
> +	     term = term->next) {
> +	    /* we ignore initial 'K' */

Same.

> +	    if (strcmp(tag, (term->string + 1)) == 0) {
> +		notmuch_message_set_flag (message, NOTMUCH_MESSAGE_FLAG_EXCLUDED, TRUE);
> +		break;
> +	    }
> +	}
>  	g_hash_table_insert (thread->tags, xstrdup (tag), NULL);
>      }
>  }
> @@ -321,7 +331,8 @@ _thread_add_matched_message (notmuch_thread_t *thread,
>  	    _thread_set_subject_from_message (thread, message);
>      }
>  
> -    thread->matched_messages++;
> +    if (!notmuch_message_get_flag (message, NOTMUCH_MESSAGE_FLAG_EXCLUDED))
> +	thread->matched_messages++;

I'd still say this warrants a better API.

>  
>      if (g_hash_table_lookup_extended (thread->message_hash,
>  			    notmuch_message_get_message_id (message), NULL,
> @@ -392,6 +403,7 @@ _notmuch_thread_create (void *ctx,
>  			notmuch_database_t *notmuch,
>  			unsigned int seed_doc_id,
>  			notmuch_doc_id_set_t *match_set,
> +			notmuch_string_list_t *exclude_terms,
>  			notmuch_sort_t sort)
>  {
>      notmuch_thread_t *thread;
> @@ -467,7 +479,7 @@ _notmuch_thread_create (void *ctx,
>  	if (doc_id == seed_doc_id)
>  	    message = seed_message;
>  
> -	_thread_add_message (thread, message);
> +	_thread_add_message (thread, message, exclude_terms);
>  
>  	if ( _notmuch_doc_id_set_contains (match_set, doc_id)) {
>  	    _notmuch_doc_id_set_remove (match_set, doc_id);

Re: [PATCH 5/7] cli: Make notmuch-show respect excludes.

[Austin Clements]

Quoth Mark Walters on Jan 29 at  6:39 pm:
> This adds the excludes to notmuch-show.c. We do not exclude when only
> a single message (or part) is requested. notmuch-show will output the
> exclude information when either text or json format is requested. As
> this changes the output from notmuch-show it breaks many tests (in a
> trivial and expected fashion).
> ---
>  notmuch-show.c |   26 +++++++++++++++++++++-----
>  1 files changed, 21 insertions(+), 5 deletions(-)
> 
> diff --git a/notmuch-show.c b/notmuch-show.c
> index dec799c..681827f 100644
> --- a/notmuch-show.c
> +++ b/notmuch-show.c
> @@ -193,10 +193,12 @@ _get_one_line_summary (const void *ctx, notmuch_message_t *message)
>  static void
>  format_message_text (unused (const void *ctx), notmuch_message_t *message, int indent)
>  {
> -    printf ("id:%s depth:%d match:%d filename:%s\n",
> +    /* Could changing this could break users ? */

I don't think anybody seriously tries to parse the text format, so I
wouldn't worry about breaking anything.

> +    printf ("id:%s depth:%d match:%d excluded:%d filename:%s\n",
>  	    notmuch_message_get_message_id (message),
>  	    indent,
> -	    notmuch_message_get_flag (message, NOTMUCH_MESSAGE_FLAG_MATCH),
> +	    notmuch_message_get_flag (message, NOTMUCH_MESSAGE_FLAG_MATCH) ? 1 : 0,
> +	    notmuch_message_get_flag (message, NOTMUCH_MESSAGE_FLAG_EXCLUDED) ? 1 : 0,
>  	    notmuch_message_get_filename (message));
>  }
>  
> @@ -212,9 +214,10 @@ format_message_json (const void *ctx, notmuch_message_t *message, unused (int in
>      date = notmuch_message_get_date (message);
>      relative_date = notmuch_time_relative_date (ctx, date);
>  
> -    printf ("\"id\": %s, \"match\": %s, \"filename\": %s, \"timestamp\": %ld, \"date_relative\": \"%s\", \"tags\": [",
> +    printf ("\"id\": %s, \"match\": %s, \"excluded\": %s, \"filename\": %s, \"timestamp\": %ld, \"date_relative\": \"%s\", \"tags\": [",

I wonder if it would be better to switch to an array of flag names...
That obviously would break consumers, but it's worth thinking about in
the longer term.

>  	    json_quote_str (ctx_quote, notmuch_message_get_message_id (message)),
>  	    notmuch_message_get_flag (message, NOTMUCH_MESSAGE_FLAG_MATCH) ? "true" : "false",
> +	    notmuch_message_get_flag (message, NOTMUCH_MESSAGE_FLAG_EXCLUDED) ? "true" : "false",
>  	    json_quote_str (ctx_quote, notmuch_message_get_filename (message)),
>  	    date, relative_date);
>  
> @@ -1059,9 +1062,13 @@ notmuch_show_command (void *ctx, unused (int argc), unused (char *argv[]))
>      char *opt;
>      const notmuch_show_format_t *format = &format_text;
>      notmuch_show_params_t params;
> +    const char **search_exclude_tags;
> +    size_t search_exclude_tags_length;
>      int mbox = 0;
>      int format_specified = 0;
>      int i;
> +    notmuch_bool_t do_not_exclude = FALSE;
> +    unsigned int j;
>  
>      params.entire_thread = 0;
>      params.raw = 0;
> @@ -1098,6 +1105,8 @@ notmuch_show_command (void *ctx, unused (int argc), unused (char *argv[]))
>  	    params.part = atoi(argv[i] + sizeof ("--part=") - 1);
>  	} else if (STRNCMP_LITERAL (argv[i], "--entire-thread") == 0) {
>  	    params.entire_thread = 1;
> +	} else if (STRNCMP_LITERAL (argv[i], "--do-not-exclude") == 0) {
> +	    do_not_exclude = TRUE;

"no-exclude" if you change the others.

>  	} else if ((STRNCMP_LITERAL (argv[i], "--verify") == 0) ||
>  		   (STRNCMP_LITERAL (argv[i], "--decrypt") == 0)) {
>  	    if (params.cryptoctx == NULL) {
> @@ -1105,7 +1114,7 @@ notmuch_show_command (void *ctx, unused (int argc), unused (char *argv[]))
>  		/* TODO: GMimePasswordRequestFunc */
>  		if (NULL == (params.cryptoctx = g_mime_gpg_context_new(NULL, "gpg")))
>  #else
> -		GMimeSession* session = g_object_new(g_mime_session_get_type(), NULL);
> +		    GMimeSession* session = g_object_new(g_mime_session_get_type(), NULL);

Accidental reindent?

>  		if (NULL == (params.cryptoctx = g_mime_gpg_context_new(session, "gpg")))
>  #endif
>  		    fprintf (stderr, "Failed to construct gpg context.\n");
> @@ -1167,10 +1176,17 @@ notmuch_show_command (void *ctx, unused (int argc), unused (char *argv[]))
>      if (params.raw && params.part < 0)
>  	params.part = 0;
>  
> +    /* if a single message is requested we do not use search_excludes */

Capital and period.

>      if (params.part >= 0)
>  	return do_show_single (ctx, query, format, &params);
>      else
> -	return do_show (ctx, query, format, &params);
> +	if (!do_not_exclude) {
> +	    search_exclude_tags = notmuch_config_get_search_exclude_tags
> +		(config, &search_exclude_tags_length);
> +	    for (j = 0; j < search_exclude_tags_length; j++)
> +		notmuch_query_add_tag_exclude (query, search_exclude_tags[j]);
> +	    return do_show (ctx, query, format, &params);
> +	}

I don't think this is the control flow you meant.  With
--do-not-exclude, there won't be any output.

>  
>      notmuch_query_destroy (query);
>      notmuch_database_close (notmuch);

Cool.  That was easy!

Re: [PATCH 4/7] lib: Add the exclude flag to notmuch_query_search_threads

[Austin Clements]

Quoth Mark Walters on Jan 29 at  6:39 pm:
> Add the NOTMUCH_MESSAGE_FLAG_EXCLUDED flag to
> notmuch_query_search_threads. Implemented by inspecting the tags
> directly in _notmuch_thread_create/_thread_add_message rather than as
> a Xapian query for speed reasons.

Hmm.  Won't the thread sort be influenced by excluded messages?

It's not completely obvious to me if it should or shouldn't be.  If
excluded messages are counted for thread sorting, it would be very
natural to toggle their visibility in the search view; otherwise we
probably need to perform the entire query again with exclusions
disabled.

If we do want to ignore excluded messages for sorting, I think
notmuch_query_search_threads can simply shuffle excluded messages to
the end of doc_ids (which is slightly awkward to implement, but not
too bad).

Re: [RFC PATCH 2/4] Add NOTMUCH_MESSAGE_FLAG_EXCLUDED flag

[Austin Clements]

This is looking really good.  I think this overall approach is
significantly better than the initial exclude support and the UI
aspects look like they should be much more pleasant.

Quoth Mark Walters on Jan 29 at  6:36 pm:
> 
> Ok I now have a patch set which might be complete enough to be worth
> reviewing. It is essentially complete and appears to work.
> 
> Things that still need doing: 
>        updating the test suite. The series changes notmuch-show to
>        output the exclude flag so several tests need updating. Of
>        course, the new functionality needs some tests too.
>        
>        emacs/notmuch.el I think it would be nice to hide (make
>        invisible) threads with no matching non-excluded messages (with a
>        toggle for visibility) but that is definitely beyond my elisp
>        skills.

I'm happy to do that once this goes in.

> The first patch of the series is not really part of the series: it adds
> a --do-not-exclude option to tell the command not to exclude. I think
> this is useful anyway, but it also simplifies behaviour decisions with
> the excludes. For example notmuch count will only count matching
> non-excluded messages but notmuch count --do-not-exclude will count 
> all matching messages excluded or not.
> 
> One outstanding issue is that raised in
> id:"20120124025331.GZ16740@mit.edu". I will need to think about
> that. 
> 
> Best wishes
> 
> Mark
> 

Re: [PATCH 1/7] cli: add --do-not-exclude option to count and search.

[Mark Walters]

Thanks for the review. Almost all of it (for all all the patches) I
agree with and will just fix but I do have a couple of queries.

On Mon, 30 Jan 2012 23:17:32 -0500, Austin Clements <amdragon@MIT.EDU> wrote:
> Quoth Mark Walters on Jan 29 at  6:39 pm:
> > This option turns off the exclusion so all matching messages are
> > returned. We do not need to add this to show as notmuch-show does not
> > (yet) exclude.
> > ---
> >  notmuch-count.c  |   12 ++++++++----
> >  notmuch-search.c |   12 ++++++++----
> >  2 files changed, 16 insertions(+), 8 deletions(-)
> > 
> > diff --git a/notmuch-count.c b/notmuch-count.c
> > index 63459fb..c88975e 100644
> > --- a/notmuch-count.c
> > +++ b/notmuch-count.c
> > @@ -37,6 +37,7 @@ notmuch_count_command (void *ctx, int argc, char *argv[])
> >      int output = OUTPUT_MESSAGES;
> >      const char **search_exclude_tags;
> >      size_t search_exclude_tags_length;
> > +    notmuch_bool_t do_not_exclude = FALSE;
> >      unsigned int i;
> >  
> >      notmuch_opt_desc_t options[] = {
> > @@ -44,6 +45,7 @@ notmuch_count_command (void *ctx, int argc, char *argv[])
> >  	  (notmuch_keyword_t []){ { "threads", OUTPUT_THREADS },
> >  				  { "messages", OUTPUT_MESSAGES },
> >  				  { 0, 0 } } },
> > +	{ NOTMUCH_OPT_BOOLEAN,  &do_not_exclude, "do-not-exclude", 'd', 0 },
> 
> Maybe just "no-exclude"?  "do-not-exclude" seems needlessly verbose.

The reason I went for verbose do-not-exclude was to try and avoid the
double negative ambiguity: does no-exclude mean do-not-exclude or
do-note-return-excluded-messages. Possibly I am worrying needlessly, and
obviously I am quite happy to change.

> Also, you have an extra space after the first comma.

Will fix.

> 
> >  	{ 0, 0, 0, 0, 0 }
> >      };
> >  
> > @@ -78,10 +80,12 @@ notmuch_count_command (void *ctx, int argc, char *argv[])
> >  	return 1;
> >      }
> >  
> > -    search_exclude_tags = notmuch_config_get_search_exclude_tags
> > -	(config, &search_exclude_tags_length);
> > -    for (i = 0; i < search_exclude_tags_length; i++)
> > -	notmuch_query_add_tag_exclude (query, search_exclude_tags[i]);
> > +    if (!do_not_exclude) {
> 
> You could move search_exclude_tags and search_exclude_tags_length in
> here now that it's a block (but you don't have to).

Will fix

> > +	search_exclude_tags = notmuch_config_get_search_exclude_tags
> > +	    (config, &search_exclude_tags_length);
> > +	for (i = 0; i < search_exclude_tags_length; i++)
> > +	    notmuch_query_add_tag_exclude (query, search_exclude_tags[i]);
> > +    }
> >  
> >      switch (output) {
> >      case OUTPUT_MESSAGES:
> > diff --git a/notmuch-search.c b/notmuch-search.c
> > index d504051..084dd05 100644
> > --- a/notmuch-search.c
> > +++ b/notmuch-search.c
> > @@ -425,6 +425,7 @@ notmuch_search_command (void *ctx, int argc, char *argv[])
> >      int limit = -1; /* unlimited */
> >      const char **search_exclude_tags;
> >      size_t search_exclude_tags_length;
> > +    notmuch_bool_t do_not_exclude = FALSE;
> >      unsigned int i;
> >  
> >      enum { NOTMUCH_FORMAT_JSON, NOTMUCH_FORMAT_TEXT }
> > @@ -446,6 +447,7 @@ notmuch_search_command (void *ctx, int argc, char *argv[])
> >  				  { "files", OUTPUT_FILES },
> >  				  { "tags", OUTPUT_TAGS },
> >  				  { 0, 0 } } },
> > +        { NOTMUCH_OPT_BOOLEAN,  &do_not_exclude, "do-not-exclude", 'd', 0 },
> 
> Same.

Will fix

> 
> >  	{ NOTMUCH_OPT_INT, &offset, "offset", 'O', 0 },
> >  	{ NOTMUCH_OPT_INT, &limit, "limit", 'L', 0  },
> >  	{ 0, 0, 0, 0, 0 }
> > @@ -493,10 +495,12 @@ notmuch_search_command (void *ctx, int argc, char *argv[])
> >  
> >      notmuch_query_set_sort (query, sort);
> >  
> > -    search_exclude_tags = notmuch_config_get_search_exclude_tags
> > -	(config, &search_exclude_tags_length);
> > -    for (i = 0; i < search_exclude_tags_length; i++)
> > -	notmuch_query_add_tag_exclude (query, search_exclude_tags[i]);
> > +    if (!do_not_exclude) {
> > +	search_exclude_tags = notmuch_config_get_search_exclude_tags
> > +	    (config, &search_exclude_tags_length);
> > +	for (i = 0; i < search_exclude_tags_length; i++)
> > +	    notmuch_query_add_tag_exclude (query, search_exclude_tags[i]);
> > +    }
> >  
> >      switch (output) {
> >      default:

Thanks

Mark

Re: [PATCH 3/7] lib: Make notmuch_query_search_messages set the exclude flag

[Mark Walters]

On Mon, 30 Jan 2012 23:43:52 -0500, Austin Clements <amdragon@MIT.EDU> wrote:
> Quoth Mark Walters on Jan 29 at  6:39 pm:
> > Add a flag NOTMUCH_MESSAGE_FLAG_EXCLUDED which is set by
> > notmuch_query_search_messages for excluded messages. Also add an
> > option omit_excluded_messages to the search that we do not want the
> > excludes at all.
> > 
> > This exclude flag will be added to notmuch_query_search threads in the
> > next patch.
> > ---
> >  lib/notmuch-private.h |    1 +
> >  lib/notmuch.h         |    8 ++++++-
> >  lib/query.cc          |   52 +++++++++++++++++++++++++++++++++++++++++++++---
> >  3 files changed, 56 insertions(+), 5 deletions(-)
> > 
> > diff --git a/lib/notmuch-private.h b/lib/notmuch-private.h
> > index 7bf153e..e791bb0 100644
> > --- a/lib/notmuch-private.h
> > +++ b/lib/notmuch-private.h
> > @@ -401,6 +401,7 @@ typedef struct _notmuch_message_list {
> >   */
> >  struct visible _notmuch_messages {
> >      notmuch_bool_t is_of_list_type;
> > +    notmuch_doc_id_set_t *excluded_doc_ids;
> 
> I might be following the diff wrong, but shouldn't this be a field of
> notmuch_mset_messages_t?  (Then it also doesn't have to be a pointer,
> which is really how notmuch_doc_id_set_t was designed to be used.)

I will need to think about that.

> >      notmuch_message_node_t *iterator;
> >  };
> >  
> > diff --git a/lib/notmuch.h b/lib/notmuch.h
> > index 7929fe7..740d005 100644
> > --- a/lib/notmuch.h
> > +++ b/lib/notmuch.h
> > @@ -449,6 +449,11 @@ typedef enum {
> >  const char *
> >  notmuch_query_get_query_string (notmuch_query_t *query);
> >  
> > +/* specify whether to results should omit the excluded results rather
> > + * than just marking them excluded */
> > +void
> > +notmuch_query_set_omit_excluded_messages (notmuch_query_t *query, notmuch_bool_t omit);
> > +
> 
> I don't think we should add this API.  The library behavior will not
> change for library users that don't use excludes and library users
> that do use excludes should by aware of the excluded flag and do the
> appropriate thing.
> 
> I can see why this is handy in some cases, but I don't think it
> provides enough utility to warrant becoming part of the permanent and
> minimal library interface.

This is really a performance improvement: suppose that there are lots of
threads that only match in excluded messages. Then without this flag we
will spend lots of time constructing the thread only for it to be
ignored. (In contrived situations this could be arbitrarily slower.)

Note the benchmarks were against master with the exclude code switched
off so that I was comparing the creation of the same threads. Sorry if I
didn't make that clear.

> >  /* Specify the sorting desired for this query. */
> >  void
> >  notmuch_query_set_sort (notmuch_query_t *query, notmuch_sort_t sort);
> > @@ -895,7 +900,8 @@ notmuch_message_get_filenames (notmuch_message_t *message);
> >  
> >  /* Message flags */
> >  typedef enum _notmuch_message_flag {
> > -    NOTMUCH_MESSAGE_FLAG_MATCH
> > +    NOTMUCH_MESSAGE_FLAG_MATCH,
> > +    NOTMUCH_MESSAGE_FLAG_EXCLUDED
> >  } notmuch_message_flag_t;
> >  
> >  /* Get a value of a flag for the email corresponding to 'message'. */
> > diff --git a/lib/query.cc b/lib/query.cc
> > index c25b301..7d165d2 100644
> > --- a/lib/query.cc
> > +++ b/lib/query.cc
> > @@ -28,6 +28,7 @@ struct _notmuch_query {
> >      const char *query_string;
> >      notmuch_sort_t sort;
> >      notmuch_string_list_t *exclude_terms;
> > +    notmuch_bool_t omit_excluded_messages;
> >  };
> >  
> >  typedef struct _notmuch_mset_messages {
> > @@ -57,6 +58,12 @@ struct visible _notmuch_threads {
> >      notmuch_doc_id_set_t match_set;
> >  };
> >  
> > +/* we need this in the message functions so forward declare */
> 
> Comments should start with a capital letter and end with a period.
> (The code isn't completely consistent about this, but it is something
> we're codifying in the upcoming style guide.)

Will fix

> > +static notmuch_bool_t
> > +_notmuch_doc_id_set_init (void *ctx,
> > +			  notmuch_doc_id_set_t *doc_ids,
> > +			  GArray *arr);
> > +
> >  notmuch_query_t *
> >  notmuch_query_create (notmuch_database_t *notmuch,
> >  		      const char *query_string)
> > @@ -79,6 +86,8 @@ notmuch_query_create (notmuch_database_t *notmuch,
> >  
> >      query->exclude_terms = _notmuch_string_list_create (query);
> >  
> > +    query->omit_excluded_messages = FALSE;
> > +
> >      return query;
> >  }
> >  
> > @@ -89,6 +98,12 @@ notmuch_query_get_query_string (notmuch_query_t *query)
> >  }
> >  
> >  void
> > +notmuch_query_set_omit_excluded_messages (notmuch_query_t *query, notmuch_bool_t omit)
> > +{
> > +    query->omit_excluded_messages = omit;
> > +}
> > +
> > +void
> >  notmuch_query_set_sort (notmuch_query_t *query, notmuch_sort_t sort)
> >  {
> >      query->sort = sort;
> > @@ -173,6 +188,7 @@ notmuch_query_search_messages (notmuch_query_t *query)
> >  						   "mail"));
> >  	Xapian::Query string_query, final_query, exclude_query;
> >  	Xapian::MSet mset;
> > +	Xapian::MSetIterator iterator;
> >  	unsigned int flags = (Xapian::QueryParser::FLAG_BOOLEAN |
> >  			      Xapian::QueryParser::FLAG_PHRASE |
> >  			      Xapian::QueryParser::FLAG_LOVEHATE |
> > @@ -190,11 +206,35 @@ notmuch_query_search_messages (notmuch_query_t *query)
> >  	    final_query = Xapian::Query (Xapian::Query::OP_AND,
> >  					 mail_query, string_query);
> >  	}
> > +	messages->base.excluded_doc_ids = NULL;
> > +
> > +	if (query->exclude_terms) {
> > +	    exclude_query = _notmuch_exclude_tags (query, final_query);
> > +	    exclude_query = Xapian::Query (Xapian::Query::OP_AND,
> > +					   exclude_query, final_query);
> > +
> > +	    if (query->omit_excluded_messages)
> > +		final_query = Xapian::Query (Xapian::Query::OP_AND_NOT,
> > +					     final_query, exclude_query);
> > +	    else {
> > +		enquire.set_weighting_scheme (Xapian::BoolWeight());
> > +		enquire.set_query (exclude_query);
> > +
> > +		mset = enquire.get_mset (0, notmuch->xapian_db->get_doccount ());
> > +
> > +		GArray *excluded_doc_ids = g_array_new (FALSE, FALSE, sizeof (unsigned int));
> > +
> > +		for (iterator = mset.begin (); iterator != mset.end (); iterator++)
> > +		{
> 
> No newline before the brace.

Will fix.

> 
> > +		    unsigned int doc_id = *iterator;
> > +		    g_array_append_val (excluded_doc_ids, doc_id);
> > +		}
> > +		messages->base.excluded_doc_ids = talloc (query, _notmuch_doc_id_set);
> > +		_notmuch_doc_id_set_init (query, messages->base.excluded_doc_ids,
> > +					  excluded_doc_ids);
> 
> Don't forget to g_array_unref excluded_doc_ids.

Yes I will add that.

> > +	    }
> > +	}
> >  
> > -	exclude_query = _notmuch_exclude_tags (query, final_query);
> > -
> > -	final_query = Xapian::Query (Xapian::Query::OP_AND_NOT,
> > -					 final_query, exclude_query);
> >  
> >  	enquire.set_weighting_scheme (Xapian::BoolWeight());
> >  
> > @@ -283,6 +323,10 @@ _notmuch_mset_messages_get (notmuch_messages_t *messages)
> >  	INTERNAL_ERROR ("a messages iterator contains a non-existent document ID.\n");
> >      }
> >  
> > +    if ((messages->excluded_doc_ids) &&
> > +	(_notmuch_doc_id_set_contains (messages->excluded_doc_ids, doc_id)))
> 
> No need for so many parens (just a nit).

Will fix.

> > +	notmuch_message_set_flag (message, NOTMUCH_MESSAGE_FLAG_EXCLUDED, TRUE);
> > +
> >      return message;
> >  }
> >  

Thanks

Mark

Re: [PATCH 4/7] lib: Add the exclude flag to notmuch_query_search_threads

[Mark Walters]

On Mon, 30 Jan 2012 23:50:20 -0500, Austin Clements <amdragon@MIT.EDU> wrote:
> Quoth Mark Walters on Jan 29 at  6:39 pm:
> > Add the NOTMUCH_MESSAGE_FLAG_EXCLUDED flag to
> > notmuch_query_search_threads. Implemented by inspecting the tags
> > directly in _notmuch_thread_create/_thread_add_message rather than as
> > a Xapian query for speed reasons.
> > ---
> >  lib/notmuch-private.h |   16 ++++++++++------
> >  lib/query.cc          |    1 +
> >  lib/thread.cc         |   18 +++++++++++++++---
> >  3 files changed, 26 insertions(+), 9 deletions(-)
> > 
> > diff --git a/lib/notmuch-private.h b/lib/notmuch-private.h
> > index e791bb0..56b87c6 100644
> > --- a/lib/notmuch-private.h
> > +++ b/lib/notmuch-private.h
> > @@ -211,12 +211,8 @@ _notmuch_directory_get_document_id (notmuch_directory_t *directory);
> >  
> >  /* thread.cc */
> >  
> > -notmuch_thread_t *
> > -_notmuch_thread_create (void *ctx,
> > -			notmuch_database_t *notmuch,
> > -			unsigned int seed_doc_id,
> > -			notmuch_doc_id_set_t *match_set,
> > -			notmuch_sort_t sort);
> > +/* Definition of _notmuch_thread_create moved later since now uses
> > + * string_list_t */
> 
> Naw, leave the definition here along with the other things from
> thread.cc and just add a
> 
>   typedef struct _notmuch_string_list notmuch_string_list_t;
> 
> along with the typedef for notmuch_doc_id_set_t near the top.  (You
> might also have to tweak the typedef of notmuch_string_list_t later so
> it's just the struct definition.)

Will do.

> >  /* message.cc */
> >  
> > @@ -492,6 +488,14 @@ notmuch_filenames_t *
> >  _notmuch_filenames_create (const void *ctx,
> >  			   notmuch_string_list_t *list);
> >  
> > +notmuch_thread_t *
> > +_notmuch_thread_create (void *ctx,
> > +			notmuch_database_t *notmuch,
> > +			unsigned int seed_doc_id,
> > +			notmuch_doc_id_set_t *match_set,
> > +			notmuch_string_list_t *excluded_terms,
> > +			notmuch_sort_t sort);
> > +
> >  #pragma GCC visibility pop
> >  
> >  NOTMUCH_END_DECLS
> > diff --git a/lib/query.cc b/lib/query.cc
> > index 7d165d2..dee7ec0 100644
> > --- a/lib/query.cc
> > +++ b/lib/query.cc
> > @@ -472,6 +472,7 @@ notmuch_threads_get (notmuch_threads_t *threads)
> >  				   threads->query->notmuch,
> >  				   doc_id,
> >  				   &threads->match_set,
> > +				   threads->query->exclude_terms,
> >  				   threads->query->sort);
> >  }
> >  
> > diff --git a/lib/thread.cc b/lib/thread.cc
> > index 0435ee6..6d65d52 100644
> > --- a/lib/thread.cc
> > +++ b/lib/thread.cc
> > @@ -214,7 +214,8 @@ _thread_cleanup_author (notmuch_thread_t *thread,
> >   */
> >  static void
> >  _thread_add_message (notmuch_thread_t *thread,
> > -		     notmuch_message_t *message)
> > +		     notmuch_message_t *message,
> > +		     notmuch_string_list_t *exclude_terms)
> >  {
> >      notmuch_tags_t *tags;
> >      const char *tag;
> > @@ -262,6 +263,15 @@ _thread_add_message (notmuch_thread_t *thread,
> >  	 notmuch_tags_move_to_next (tags))
> >      {
> >  	tag = notmuch_tags_get (tags);
> > +	/* mark excluded messages */
> 
> Capital and period.

Will fix.

> > +	for (notmuch_string_node_t *term = exclude_terms->head; term;
> > +	     term = term->next) {
> > +	    /* we ignore initial 'K' */
> 
> Same.

Will fix

> > +	    if (strcmp(tag, (term->string + 1)) == 0) {
> > +		notmuch_message_set_flag (message, NOTMUCH_MESSAGE_FLAG_EXCLUDED, TRUE);
> > +		break;
> > +	    }
> > +	}
> >  	g_hash_table_insert (thread->tags, xstrdup (tag), NULL);
> >      }
> >  }
> > @@ -321,7 +331,8 @@ _thread_add_matched_message (notmuch_thread_t *thread,
> >  	    _thread_set_subject_from_message (thread, message);
> >      }
> >  
> > -    thread->matched_messages++;
> > +    if (!notmuch_message_get_flag (message, NOTMUCH_MESSAGE_FLAG_EXCLUDED))
> > +	thread->matched_messages++;
> 
> I'd still say this warrants a better API.

I agree: I will think about that.

> >      if (g_hash_table_lookup_extended (thread->message_hash,
> >  			    notmuch_message_get_message_id (message), NULL,
> > @@ -392,6 +403,7 @@ _notmuch_thread_create (void *ctx,
> >  			notmuch_database_t *notmuch,
> >  			unsigned int seed_doc_id,
> >  			notmuch_doc_id_set_t *match_set,
> > +			notmuch_string_list_t *exclude_terms,
> >  			notmuch_sort_t sort)
> >  {
> >      notmuch_thread_t *thread;
> > @@ -467,7 +479,7 @@ _notmuch_thread_create (void *ctx,
> >  	if (doc_id == seed_doc_id)
> >  	    message = seed_message;
> >  
> > -	_thread_add_message (thread, message);
> > +	_thread_add_message (thread, message, exclude_terms);
> >  
> >  	if ( _notmuch_doc_id_set_contains (match_set, doc_id)) {
> >  	    _notmuch_doc_id_set_remove (match_set, doc_id);

Re: [PATCH 5/7] cli: Make notmuch-show respect excludes.

[Mark Walters]

On Mon, 30 Jan 2012 23:56:24 -0500, Austin Clements <amdragon@MIT.EDU> wrote:
> Quoth Mark Walters on Jan 29 at  6:39 pm:
> > This adds the excludes to notmuch-show.c. We do not exclude when only
> > a single message (or part) is requested. notmuch-show will output the
> > exclude information when either text or json format is requested. As
> > this changes the output from notmuch-show it breaks many tests (in a
> > trivial and expected fashion).
> > ---
> >  notmuch-show.c |   26 +++++++++++++++++++++-----
> >  1 files changed, 21 insertions(+), 5 deletions(-)
> > 
> > diff --git a/notmuch-show.c b/notmuch-show.c
> > index dec799c..681827f 100644
> > --- a/notmuch-show.c
> > +++ b/notmuch-show.c
> > @@ -193,10 +193,12 @@ _get_one_line_summary (const void *ctx, notmuch_message_t *message)
> >  static void
> >  format_message_text (unused (const void *ctx), notmuch_message_t *message, int indent)
> >  {
> > -    printf ("id:%s depth:%d match:%d filename:%s\n",
> > +    /* Could changing this could break users ? */
> 
> I don't think anybody seriously tries to parse the text format, so I
> wouldn't worry about breaking anything.

Right: I will remove the comment.

> > +    printf ("id:%s depth:%d match:%d excluded:%d filename:%s\n",
> >  	    notmuch_message_get_message_id (message),
> >  	    indent,
> > -	    notmuch_message_get_flag (message, NOTMUCH_MESSAGE_FLAG_MATCH),
> > +	    notmuch_message_get_flag (message, NOTMUCH_MESSAGE_FLAG_MATCH) ? 1 : 0,
> > +	    notmuch_message_get_flag (message, NOTMUCH_MESSAGE_FLAG_EXCLUDED) ? 1 : 0,
> >  	    notmuch_message_get_filename (message));
> >  }
> >  
> > @@ -212,9 +214,10 @@ format_message_json (const void *ctx, notmuch_message_t *message, unused (int in
> >      date = notmuch_message_get_date (message);
> >      relative_date = notmuch_time_relative_date (ctx, date);
> >  
> > -    printf ("\"id\": %s, \"match\": %s, \"filename\": %s, \"timestamp\": %ld, \"date_relative\": \"%s\", \"tags\": [",
> > +    printf ("\"id\": %s, \"match\": %s, \"excluded\": %s, \"filename\": %s, \"timestamp\": %ld, \"date_relative\": \"%s\", \"tags\": [",
> 
> I wonder if it would be better to switch to an array of flag names...
> That obviously would break consumers, but it's worth thinking about in
> the longer term.

Unless you have a strong feeling for this I will leave that for a later
patch.

> >  	    json_quote_str (ctx_quote, notmuch_message_get_message_id (message)),
> >  	    notmuch_message_get_flag (message, NOTMUCH_MESSAGE_FLAG_MATCH) ? "true" : "false",
> > +	    notmuch_message_get_flag (message, NOTMUCH_MESSAGE_FLAG_EXCLUDED) ? "true" : "false",
> >  	    json_quote_str (ctx_quote, notmuch_message_get_filename (message)),
> >  	    date, relative_date);
> >  
> > @@ -1059,9 +1062,13 @@ notmuch_show_command (void *ctx, unused (int argc), unused (char *argv[]))
> >      char *opt;
> >      const notmuch_show_format_t *format = &format_text;
> >      notmuch_show_params_t params;
> > +    const char **search_exclude_tags;
> > +    size_t search_exclude_tags_length;
> >      int mbox = 0;
> >      int format_specified = 0;
> >      int i;
> > +    notmuch_bool_t do_not_exclude = FALSE;
> > +    unsigned int j;
> >  
> >      params.entire_thread = 0;
> >      params.raw = 0;
> > @@ -1098,6 +1105,8 @@ notmuch_show_command (void *ctx, unused (int argc), unused (char *argv[]))
> >  	    params.part = atoi(argv[i] + sizeof ("--part=") - 1);
> >  	} else if (STRNCMP_LITERAL (argv[i], "--entire-thread") == 0) {
> >  	    params.entire_thread = 1;
> > +	} else if (STRNCMP_LITERAL (argv[i], "--do-not-exclude") == 0) {
> > +	    do_not_exclude = TRUE;
> 
> "no-exclude" if you change the others.

See comment on first patch: will make sure they are consistent.

> >  	} else if ((STRNCMP_LITERAL (argv[i], "--verify") == 0) ||
> >  		   (STRNCMP_LITERAL (argv[i], "--decrypt") == 0)) {
> >  	    if (params.cryptoctx == NULL) {
> > @@ -1105,7 +1114,7 @@ notmuch_show_command (void *ctx, unused (int argc), unused (char *argv[]))
> >  		/* TODO: GMimePasswordRequestFunc */
> >  		if (NULL == (params.cryptoctx = g_mime_gpg_context_new(NULL, "gpg")))
> >  #else
> > -		GMimeSession* session = g_object_new(g_mime_session_get_type(), NULL);
> > +		    GMimeSession* session = g_object_new(g_mime_session_get_type(), NULL);
> 
> Accidental reindent?

Yes will fix.

> >  		if (NULL == (params.cryptoctx = g_mime_gpg_context_new(session, "gpg")))
> >  #endif
> >  		    fprintf (stderr, "Failed to construct gpg context.\n");
> > @@ -1167,10 +1176,17 @@ notmuch_show_command (void *ctx, unused (int argc), unused (char *argv[]))
> >      if (params.raw && params.part < 0)
> >  	params.part = 0;
> >  
> > +    /* if a single message is requested we do not use search_excludes */
> 
> Capital and period.

Will fix.

> >      if (params.part >= 0)
> >  	return do_show_single (ctx, query, format, &params);
> >      else
> > -	return do_show (ctx, query, format, &params);
> > +	if (!do_not_exclude) {
> > +	    search_exclude_tags = notmuch_config_get_search_exclude_tags
> > +		(config, &search_exclude_tags_length);
> > +	    for (j = 0; j < search_exclude_tags_length; j++)
> > +		notmuch_query_add_tag_exclude (query, search_exclude_tags[j]);
> > +	    return do_show (ctx, query, format, &params);
> > +	}
> 
> I don't think this is the control flow you meant.  With
> --do-not-exclude, there won't be any output.

Good catch! Will fix.

Thanks 

Mark

Re: [PATCH 1/7] cli: add --do-not-exclude option to count and search.

[Austin Clements]

Quoth Mark Walters on Jan 31 at 11:40 am:
> 
> Thanks for the review. Almost all of it (for all all the patches) I
> agree with and will just fix but I do have a couple of queries.
> 
> On Mon, 30 Jan 2012 23:17:32 -0500, Austin Clements <amdragon@MIT.EDU> wrote:
> > Quoth Mark Walters on Jan 29 at  6:39 pm:
> > > This option turns off the exclusion so all matching messages are
> > > returned. We do not need to add this to show as notmuch-show does not
> > > (yet) exclude.
> > > ---
> > >  notmuch-count.c  |   12 ++++++++----
> > >  notmuch-search.c |   12 ++++++++----
> > >  2 files changed, 16 insertions(+), 8 deletions(-)
> > > 
> > > diff --git a/notmuch-count.c b/notmuch-count.c
> > > index 63459fb..c88975e 100644
> > > --- a/notmuch-count.c
> > > +++ b/notmuch-count.c
> > > @@ -37,6 +37,7 @@ notmuch_count_command (void *ctx, int argc, char *argv[])
> > >      int output = OUTPUT_MESSAGES;
> > >      const char **search_exclude_tags;
> > >      size_t search_exclude_tags_length;
> > > +    notmuch_bool_t do_not_exclude = FALSE;
> > >      unsigned int i;
> > >  
> > >      notmuch_opt_desc_t options[] = {
> > > @@ -44,6 +45,7 @@ notmuch_count_command (void *ctx, int argc, char *argv[])
> > >  	  (notmuch_keyword_t []){ { "threads", OUTPUT_THREADS },
> > >  				  { "messages", OUTPUT_MESSAGES },
> > >  				  { 0, 0 } } },
> > > +	{ NOTMUCH_OPT_BOOLEAN,  &do_not_exclude, "do-not-exclude", 'd', 0 },
> > 
> > Maybe just "no-exclude"?  "do-not-exclude" seems needlessly verbose.
> 
> The reason I went for verbose do-not-exclude was to try and avoid the
> double negative ambiguity: does no-exclude mean do-not-exclude or
> do-note-return-excluded-messages. Possibly I am worrying needlessly, and
> obviously I am quite happy to change.

Oh.  Hmm.  --no-exclusions?  --unexcluded?  --all?
--include-excludes?  Maybe --do-not-exclude is best.

Re: [PATCH 3/7] lib: Make notmuch_query_search_messages set the exclude flag

[Austin Clements]

Quoth Mark Walters on Jan 31 at 11:45 am:
> On Mon, 30 Jan 2012 23:43:52 -0500, Austin Clements <amdragon@MIT.EDU> wrote:
> > Quoth Mark Walters on Jan 29 at  6:39 pm:
> > >      notmuch_message_node_t *iterator;
> > >  };
> > >  
> > > diff --git a/lib/notmuch.h b/lib/notmuch.h
> > > index 7929fe7..740d005 100644
> > > --- a/lib/notmuch.h
> > > +++ b/lib/notmuch.h
> > > @@ -449,6 +449,11 @@ typedef enum {
> > >  const char *
> > >  notmuch_query_get_query_string (notmuch_query_t *query);
> > >  
> > > +/* specify whether to results should omit the excluded results rather
> > > + * than just marking them excluded */
> > > +void
> > > +notmuch_query_set_omit_excluded_messages (notmuch_query_t *query, notmuch_bool_t omit);
> > > +
> > 
> > I don't think we should add this API.  The library behavior will not
> > change for library users that don't use excludes and library users
> > that do use excludes should by aware of the excluded flag and do the
> > appropriate thing.
> > 
> > I can see why this is handy in some cases, but I don't think it
> > provides enough utility to warrant becoming part of the permanent and
> > minimal library interface.
> 
> This is really a performance improvement: suppose that there are lots of
> threads that only match in excluded messages. Then without this flag we
> will spend lots of time constructing the thread only for it to be
> ignored. (In contrived situations this could be arbitrarily slower.)

I would prefer to keep the public API minimal and only introduce
something like this if people actually experience performance
problems.

Re: [PATCH 1/7] cli: add --do-not-exclude option to count and search.

[Jameson Graef Rollins]

[-- Attachment #1: Type: text/plain, Size: 534 bytes --]

On Tue, 31 Jan 2012 11:40:00 +0000, Mark Walters <markwalters1009@gmail.com> wrote:
> The reason I went for verbose do-not-exclude was to try and avoid the
> double negative ambiguity: does no-exclude mean do-not-exclude or
> do-note-return-excluded-messages. Possibly I am worrying needlessly, and
> obviously I am quite happy to change.

I also think "no-exclude" or "no-excludes" would be more appropriate.
It's not a command that people will likely use that frequently, so as
long as it's well documented it will be fine.

jamie.

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: [PATCH 3/7] lib: Make notmuch_query_search_messages set the exclude flag

[Mark Walters]

On Tue, 31 Jan 2012 11:25:06 -0500, Austin Clements <amdragon@MIT.EDU> wrote:
> Quoth Mark Walters on Jan 31 at 11:45 am:
> > On Mon, 30 Jan 2012 23:43:52 -0500, Austin Clements <amdragon@MIT.EDU> wrote:
> > > Quoth Mark Walters on Jan 29 at  6:39 pm:
> > > >      notmuch_message_node_t *iterator;
> > > >  };
> > > >  
> > > > diff --git a/lib/notmuch.h b/lib/notmuch.h
> > > > index 7929fe7..740d005 100644
> > > > --- a/lib/notmuch.h
> > > > +++ b/lib/notmuch.h
> > > > @@ -449,6 +449,11 @@ typedef enum {
> > > >  const char *
> > > >  notmuch_query_get_query_string (notmuch_query_t *query);
> > > >  
> > > > +/* specify whether to results should omit the excluded results rather
> > > > + * than just marking them excluded */
> > > > +void
> > > > +notmuch_query_set_omit_excluded_messages (notmuch_query_t *query, notmuch_bool_t omit);
> > > > +
> > > 
> > > I don't think we should add this API.  The library behavior will not
> > > change for library users that don't use excludes and library users
> > > that do use excludes should by aware of the excluded flag and do the
> > > appropriate thing.
> > > 
> > > I can see why this is handy in some cases, but I don't think it
> > > provides enough utility to warrant becoming part of the permanent and
> > > minimal library interface.
> > 
> > This is really a performance improvement: suppose that there are lots of
> > threads that only match in excluded messages. Then without this flag we
> > will spend lots of time constructing the thread only for it to be
> > ignored. (In contrived situations this could be arbitrarily slower.)
> 
> I would prefer to keep the public API minimal and only introduce
> something like this if people actually experience performance
> problems.

I have looked at removing this api and it does make things
inconvenient. It is nice to be able to pass a notmuch_messages_t around
without having to teach every function that takes notmuch_messages_t
(e.g., notmuch_messages_collect_tags) about the exclude flag.

An alternative might be to make it possible to set an omit excludes flag
on a notmuch_messages_t object but that seems worse than the current
patches.

Any thoughts?

Mark

PS I think my current version fixes all the other queries so I will post
that later today (in a new thread!).

Re: [PATCH 1/7] cli: add --do-not-exclude option to count and search.

[Jameson Graef Rollins]

[-- Attachment #1: Type: text/plain, Size: 678 bytes --]

On Sun, 29 Jan 2012 18:39:48 +0000, Mark Walters <markwalters1009@gmail.com> wrote:
> This option turns off the exclusion so all matching messages are
> returned. We do not need to add this to show as notmuch-show does not
> (yet) exclude.

Hi.  What is the status of this patch?  It looks like we're waiting on a
new version.  I found a bug in the emacs interface that actually
requires the --no-excludes search option to fix.  It we can get another
version of the patch I'll work my bug fix on top of it.

I'm tempted to ask for the next version in a new thread, since this
thread has gotten obscenely long and includes what are essentially
multiple separate threads.

jamie.

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: [PATCH 1/7] cli: add --do-not-exclude option to count and search.

[Austin Clements]

Quoth Jameson Graef Rollins on Feb 11 at 10:44 am:
> On Sun, 29 Jan 2012 18:39:48 +0000, Mark Walters <markwalters1009@gmail.com> wrote:
> > This option turns off the exclusion so all matching messages are
> > returned. We do not need to add this to show as notmuch-show does not
> > (yet) exclude.
> 
> Hi.  What is the status of this patch?  It looks like we're waiting on a
> new version.  I found a bug in the emacs interface that actually
> requires the --no-excludes search option to fix.  It we can get another
> version of the patch I'll work my bug fix on top of it.
> 
> I'm tempted to ask for the next version in a new thread, since this
> thread has gotten obscenely long and includes what are essentially
> multiple separate threads.

It has moved to a new thread: id:"874nv9rv79.fsf@qmul.ac.uk".  It's
waiting for a rebase on Jani's command line parsing change and
probably needs some more review, too.

Re: [PATCH 1/7] cli: add --do-not-exclude option to count and search.

[Jameson Graef Rollins]

[-- Attachment #1: Type: text/plain, Size: 290 bytes --]

On Sat, 11 Feb 2012 13:50:23 -0500, Austin Clements <amdragon@MIT.EDU> wrote:
> It has moved to a new thread: id:"874nv9rv79.fsf@qmul.ac.uk".  It's
> waiting for a rebase on Jani's command line parsing change and
> probably needs some more review, too.

So it has!  Thanks, Austin.

jamie.

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]