[PATCH v3 0/3] Better id: link buttonization

[PATCH v3 0/3] Better id: link buttonization

[Austin Clements]

This is v3 of id:"1352609175-12790-1-git-send-email-amdragon@mit.edu".
Tomi pointed out in [1] that I'd reversed OUTPUT and EXPECTED in the
test, which this version fixes.  There are no other changes, so I'm
marking this patch ready since Ethan and Tomi approved v2.

[1] id:m2obj0ega4.fsf@guru.guru-group.fi

[PATCH v3 1/3] test: Test buttonization of id: links

[Austin Clements]

This matches the current behavior of the buttonizer, so it passes, but
many of these cases are not what you'd want (and some of them aren't
even valid Xapian queries).  The next patch will fix the handling of
these cases and update the test.
---
 test/emacs-show  |   50 ++++++++++++++++++++++++++++++++++++++++++++++++++
 test/test-lib.el |   14 ++++++++++++++
 2 files changed, 64 insertions(+)

diff --git a/test/emacs-show b/test/emacs-show
index 64c38d3..5d7151b 100755
--- a/test/emacs-show
+++ b/test/emacs-show
@@ -106,5 +106,55 @@ test_emacs '(notmuch-search "from:lars@seas.harvard.edu and subject:\"Maildir st
 	(test-visible-output)'
 test_expect_equal_file OUTPUT $EXPECTED/notmuch-show-indent-thread-content-off
 
+test_begin_subtest "id buttonization"
+add_message '[body]="
+id:abc
+id:abc.def. id:abc,def, id:abc;def; id:abc:def:
+id:foo@bar.?baz? id:foo@bar!.baz!
+(id:foo@bar.baz) [id:foo@bar.baz]
+id:foo@bar.baz...
+id:2+2=5
+id:=_-:/.[]@$%+
+id:abc)def
+id:ab\"c def
+id:\"abc\"def
+id:\"ab\"\"c\"def
+id:\"ab c\"def
+id:\"abc\".def
+id:\"abc
+\"
+id:)
+id:
+cid:xxx"'
+test_emacs '(notmuch-show "id:'$gen_msg_id'")
+	(notmuch-test-mark-links)
+	(test-visible-output)'
+cat <<EOF >EXPECTED
+Notmuch Test Suite <test_suite@notmuchmail.org> (2001-01-05) (inbox)
+Subject: id buttonization
+To: Notmuch Test Suite <test_suite@notmuchmail.org>
+Date: Fri, 05 Jan 2001 15:43:57 +0000
+
+<<id:abc>>
+<<id:abc.def.>> <<id:abc,def,>> <<id:abc;def;>> <<id:abc:def:>>
+<<id:foo@bar.?baz?>> <<id:foo@bar!.baz!>>
+(<<id:foo@bar.baz)>> [<<id:foo@bar.baz]>>
+<<id:foo@bar.baz...>>
+<<id:2+2=5>>
+<<id:=_-:/.[]@$%+>>
+<<id:abc)def>>
+<<id:ab>>"c def
+<<id:"abc">>def
+<<id:"ab">>"c"def
+id:"ab c"def
+<<id:"abc">>.def
+id:"abc
+"
+<<id:)>>
+id:
+c<<id:xxx>>
+EOF
+test_expect_equal_file OUTPUT EXPECTED
+
 
 test_done
diff --git a/test/test-lib.el b/test/test-lib.el
index fa3380c..dece811 100644
--- a/test/test-lib.el
+++ b/test/test-lib.el
@@ -107,6 +107,20 @@ nothing."
 	       (ad-set-arg 1 (char-to-string char))
 	       ad-do-it))))
 
+(defun notmuch-test-mark-links ()
+  "Enclose links in the current buffer with << and >>."
+  ;; Links are often created by jit-lock functions
+  (jit-lock-fontify-now)
+  (save-excursion
+    (let ((inhibit-read-only t))
+      (goto-char (point-min))
+      (let ((button))
+	(while (setq button (next-button (point)))
+	  (goto-char (button-start button))
+	  (insert "<<")
+	  (goto-char (button-end button))
+	  (insert ">>"))))))
+
 (defmacro notmuch-test-run (&rest body)
   "Evaluate a BODY of test expressions and output the result."
   `(with-temp-buffer
-- 
1.7.10.4

[PATCH v3 2/3] emacs: Improve the regexp used to match id:'s in messages

[Austin Clements]

This regexp agrees with Xapian query syntax much more closely, though
we specifically disallow various cases that would be confusing in the
context of an email body (e.g., punctuation at the end of an id: link
is not considered part of the id: link because it's probably part of
the surrounding text).

In particular, this handles id: links that are not surrounded by
quotes much better, which stash is much more likely to generate now
that we don't quote id's that don't need to be quoted.  It also
handles quoted id: links better.

We update the buttonization test to reflect the new pattern.
---
 emacs/notmuch-show.el |   20 +++++++++++++++++++-
 test/emacs-show       |   20 ++++++++++----------
 2 files changed, 29 insertions(+), 11 deletions(-)

diff --git a/emacs/notmuch-show.el b/emacs/notmuch-show.el
index d061367..49961fb 100644
--- a/emacs/notmuch-show.el
+++ b/emacs/notmuch-show.el
@@ -996,6 +996,24 @@ message at DEPTH in the current thread."
   "Insert the forest of threads FOREST."
   (mapc (lambda (thread) (notmuch-show-insert-thread thread 0)) forest))
 
+(defvar notmuch-id-regexp
+  (concat
+   ;; Match the id: prefix only if it begins a word (to disallow, for
+   ;; example, matching cid:).
+   "\\<id:\\("
+   ;; If the term starts with a ", then parse Xapian's quoted boolean
+   ;; term syntax, which allows for anything as long as embedded
+   ;; double quotes escaped by doubling them.  We also disallow
+   ;; newlines (which Xapian allows) to prevent runaway terms.
+   "\"\\([^\"\n]\\|\"\"\\)*\""
+   ;; Otherwise, parse Xapian's unquoted syntax, which goes up to the
+   ;; next space or ).  We disallow [.,;] as the last character
+   ;; because these are probably part of the surrounding text, and not
+   ;; part of the id.  This doesn't match single character ids; meh.
+   "\\|[^\"[:space:])][^[:space:])]*[^])[:space:].,:;?!]"
+   "\\)")
+  "The regexp used to match id: links in messages.")
+
 (defun notmuch-show-buttonise-links (start end)
   "Buttonise URLs and mail addresses between START and END.
 
@@ -1004,7 +1022,7 @@ a corresponding notmuch search."
   (goto-address-fontify-region start end)
   (save-excursion
     (goto-char start)
-    (while (re-search-forward "id:\\(\"?\\)[^[:space:]\"]+\\1" end t)
+    (while (re-search-forward notmuch-id-regexp end t)
       ;; remove the overlay created by goto-address-mode
       (remove-overlays (match-beginning 0) (match-end 0) 'goto-address t)
       (make-text-button (match-beginning 0) (match-end 0)
diff --git a/test/emacs-show b/test/emacs-show
index 5d7151b..e2d7c70 100755
--- a/test/emacs-show
+++ b/test/emacs-show
@@ -136,23 +136,23 @@ To: Notmuch Test Suite <test_suite@notmuchmail.org>
 Date: Fri, 05 Jan 2001 15:43:57 +0000
 
 <<id:abc>>
-<<id:abc.def.>> <<id:abc,def,>> <<id:abc;def;>> <<id:abc:def:>>
-<<id:foo@bar.?baz?>> <<id:foo@bar!.baz!>>
-(<<id:foo@bar.baz)>> [<<id:foo@bar.baz]>>
-<<id:foo@bar.baz...>>
+<<id:abc.def>>. <<id:abc,def>>, <<id:abc;def>>; <<id:abc:def>>:
+<<id:foo@bar.?baz>>? <<id:foo@bar!.baz>>!
+(<<id:foo@bar.baz>>) [<<id:foo@bar.baz>>]
+<<id:foo@bar.baz>>...
 <<id:2+2=5>>
 <<id:=_-:/.[]@$%+>>
-<<id:abc)def>>
-<<id:ab>>"c def
+<<id:abc>>)def
+<<id:ab"c>> def
 <<id:"abc">>def
-<<id:"ab">>"c"def
-id:"ab c"def
+<<id:"ab""c">>def
+<<id:"ab c">>def
 <<id:"abc">>.def
 id:"abc
 "
-<<id:)>>
+id:)
 id:
-c<<id:xxx>>
+cid:xxx
 EOF
 test_expect_equal_file OUTPUT EXPECTED
 
-- 
1.7.10.4

[PATCH v3 3/3] emacs: Buttonize mid: links

[Austin Clements]

This adds support for RFC 2392 mid: message ID links.
---
 emacs/notmuch-show.el |   45 +++++++++++++++++++++++++++++++++------------
 test/emacs-show       |    8 +++++++-
 2 files changed, 40 insertions(+), 13 deletions(-)

diff --git a/emacs/notmuch-show.el b/emacs/notmuch-show.el
index 49961fb..5b3e70e 100644
--- a/emacs/notmuch-show.el
+++ b/emacs/notmuch-show.el
@@ -1014,23 +1014,44 @@ message at DEPTH in the current thread."
    "\\)")
   "The regexp used to match id: links in messages.")
 
+(defvar notmuch-mid-regexp
+  ;; goto-address-url-regexp matched cid: links, which have the same
+  ;; grammar as the message ID part of a mid: link.  Construct the
+  ;; regexp using the same technique as goto-address-url-regexp.
+  (concat "\\<mid:\\(" thing-at-point-url-path-regexp "\\)")
+  "The regexp used to match mid: links in messages.
+
+See RFC 2392.")
+
 (defun notmuch-show-buttonise-links (start end)
   "Buttonise URLs and mail addresses between START and END.
 
-This also turns id:\"<message id>\"-parts into buttons for
-a corresponding notmuch search."
+This also turns id:\"<message id>\"-parts and mid: links into
+buttons for a corresponding notmuch search."
   (goto-address-fontify-region start end)
   (save-excursion
-    (goto-char start)
-    (while (re-search-forward notmuch-id-regexp end t)
-      ;; remove the overlay created by goto-address-mode
-      (remove-overlays (match-beginning 0) (match-end 0) 'goto-address t)
-      (make-text-button (match-beginning 0) (match-end 0)
-			'action `(lambda (arg)
-				   (notmuch-show ,(match-string-no-properties 0)))
-			'follow-link t
-			'help-echo "Mouse-1, RET: search for this message"
-			'face goto-address-mail-face))))
+    (let (links)
+      (goto-char start)
+      (while (re-search-forward notmuch-id-regexp end t)
+	(push (list (match-beginning 0) (match-end 0)
+		    (match-string-no-properties 0)) links))
+      (goto-char start)
+      (while (re-search-forward notmuch-mid-regexp end t)
+	(let* ((mid-cid (match-string-no-properties 1))
+	       (mid (save-match-data
+		      (string-match "^[^/]*" mid-cid)
+		      (url-unhex-string (match-string 0 mid-cid)))))
+	  (push (list (match-beginning 0) (match-end 0)
+		      (notmuch-id-to-query mid)) links)))
+      (dolist (link links)
+	;; Remove the overlay created by goto-address-mode
+	(remove-overlays (first link) (second link) 'goto-address t)
+	(make-text-button (first link) (second link)
+			  'action `(lambda (arg)
+				     (notmuch-show ,(third link)))
+			  'follow-link t
+			  'help-echo "Mouse-1, RET: search for this message"
+			  'face goto-address-mail-face)))))
 
 ;;;###autoload
 (defun notmuch-show (thread-id &optional parent-buffer query-context buffer-name)
diff --git a/test/emacs-show b/test/emacs-show
index e2d7c70..ec3be8b 100755
--- a/test/emacs-show
+++ b/test/emacs-show
@@ -125,7 +125,10 @@ id:\"abc
 \"
 id:)
 id:
-cid:xxx"'
+cid:xxx
+mid:abc mid:abc/def
+mid:abc%20def
+mid:abc. mid:abc, mid:abc;"'
 test_emacs '(notmuch-show "id:'$gen_msg_id'")
 	(notmuch-test-mark-links)
 	(test-visible-output)'
@@ -153,6 +156,9 @@ id:"abc
 id:)
 id:
 cid:xxx
+<<mid:abc>> <<mid:abc/def>>
+<<mid:abc%20def>>
+<<mid:abc>>. <<mid:abc>>, <<mid:abc>>;
 EOF
 test_expect_equal_file OUTPUT EXPECTED
 
-- 
1.7.10.4

Re: [PATCH v3 0/3] Better id: link buttonization

[David Bremner]

Austin Clements <amdragon@MIT.EDU> writes:

> This is v3 of id:"1352609175-12790-1-git-send-email-amdragon@mit.edu".
> Tomi pointed out in [1] that I'd reversed OUTPUT and EXPECTED in the
> test, which this version fixes.  There are no other changes, so I'm
> marking this patch ready since Ethan and Tomi approved v2.

Pushed. I wonder if we should support mid: on the command line?

d

Re: [PATCH v3 0/3] Better id: link buttonization

[Austin Clements]

Quoth David Bremner on Nov 15 at  6:24 pm:
> Austin Clements <amdragon@MIT.EDU> writes:
> 
> > This is v3 of id:"1352609175-12790-1-git-send-email-amdragon@mit.edu".
> > Tomi pointed out in [1] that I'd reversed OUTPUT and EXPECTED in the
> > test, which this version fixes.  There are no other changes, so I'm
> > marking this patch ready since Ethan and Tomi approved v2.
> 
> Pushed. I wonder if we should support mid: on the command line?

Currently that would require adding an mid:-encoded copy of every
message ID to the database.  This could be more approachable in the
future with a custom query parser or Xapian's rumored field parser
support.