[PATCH v2 0/3] Better id: link buttonization

[PATCH v2 0/3] Better id: link buttonization

[Austin Clements]

This is v2 of id:"1351650561-7331-1-git-send-email-amdragon@mit.edu".
This makes Jani's suggested additions to the regexp and adds support
for RFC 2392 mid: links, as suggested by Sascha.

[PATCH v2 1/3] test: Test buttonization of id: links

[Austin Clements]

This matches the current behavior of the buttonizer, so it passes, but
many of these cases are not what you'd want (and some of them aren't
even valid Xapian queries).  The next patch will fix the handling of
these cases and update the test.
---
 test/emacs-show  |   50 ++++++++++++++++++++++++++++++++++++++++++++++++++
 test/test-lib.el |   14 ++++++++++++++
 2 files changed, 64 insertions(+)

diff --git a/test/emacs-show b/test/emacs-show
index 64c38d3..9712633 100755
--- a/test/emacs-show
+++ b/test/emacs-show
@@ -106,5 +106,55 @@ test_emacs '(notmuch-search "from:lars@seas.harvard.edu and subject:\"Maildir st
 	(test-visible-output)'
 test_expect_equal_file OUTPUT $EXPECTED/notmuch-show-indent-thread-content-off
 
+test_begin_subtest "id buttonization"
+add_message '[body]="
+id:abc
+id:abc.def. id:abc,def, id:abc;def; id:abc:def:
+id:foo@bar.?baz? id:foo@bar!.baz!
+(id:foo@bar.baz) [id:foo@bar.baz]
+id:foo@bar.baz...
+id:2+2=5
+id:=_-:/.[]@$%+
+id:abc)def
+id:ab\"c def
+id:\"abc\"def
+id:\"ab\"\"c\"def
+id:\"ab c\"def
+id:\"abc\".def
+id:\"abc
+\"
+id:)
+id:
+cid:xxx"'
+test_emacs '(notmuch-show "id:'$gen_msg_id'")
+	(notmuch-test-mark-links)
+	(test-visible-output)'
+cat <<EOF >EXPECTED
+Notmuch Test Suite <test_suite@notmuchmail.org> (2001-01-05) (inbox)
+Subject: id buttonization
+To: Notmuch Test Suite <test_suite@notmuchmail.org>
+Date: Fri, 05 Jan 2001 15:43:57 +0000
+
+<<id:abc>>
+<<id:abc.def.>> <<id:abc,def,>> <<id:abc;def;>> <<id:abc:def:>>
+<<id:foo@bar.?baz?>> <<id:foo@bar!.baz!>>
+(<<id:foo@bar.baz)>> [<<id:foo@bar.baz]>>
+<<id:foo@bar.baz...>>
+<<id:2+2=5>>
+<<id:=_-:/.[]@$%+>>
+<<id:abc)def>>
+<<id:ab>>"c def
+<<id:"abc">>def
+<<id:"ab">>"c"def
+id:"ab c"def
+<<id:"abc">>.def
+id:"abc
+"
+<<id:)>>
+id:
+c<<id:xxx>>
+EOF
+test_expect_equal_file EXPECTED OUTPUT
+
 
 test_done
diff --git a/test/test-lib.el b/test/test-lib.el
index fa3380c..dece811 100644
--- a/test/test-lib.el
+++ b/test/test-lib.el
@@ -107,6 +107,20 @@ nothing."
 	       (ad-set-arg 1 (char-to-string char))
 	       ad-do-it))))
 
+(defun notmuch-test-mark-links ()
+  "Enclose links in the current buffer with << and >>."
+  ;; Links are often created by jit-lock functions
+  (jit-lock-fontify-now)
+  (save-excursion
+    (let ((inhibit-read-only t))
+      (goto-char (point-min))
+      (let ((button))
+	(while (setq button (next-button (point)))
+	  (goto-char (button-start button))
+	  (insert "<<")
+	  (goto-char (button-end button))
+	  (insert ">>"))))))
+
 (defmacro notmuch-test-run (&rest body)
   "Evaluate a BODY of test expressions and output the result."
   `(with-temp-buffer
-- 
1.7.10.4

[PATCH v2 2/3] emacs: Improve the regexp used to match id:'s in messages

[Austin Clements]

This regexp agrees with Xapian query syntax much more closely, though
we specifically disallow various cases that would be confusing in the
context of an email body (e.g., punctuation at the end of an id: link
is not considered part of the id: link because it's probably part of
the surrounding text).

In particular, this handles id: links that are not surrounded by
quotes much better, which stash is much more likely to generate now
that we don't quote id's that don't need to be quoted.  It also
handles quoted id: links better.

We update the buttonization test to reflect the new pattern.
---
 emacs/notmuch-show.el |   20 +++++++++++++++++++-
 test/emacs-show       |   20 ++++++++++----------
 2 files changed, 29 insertions(+), 11 deletions(-)

diff --git a/emacs/notmuch-show.el b/emacs/notmuch-show.el
index d061367..49961fb 100644
--- a/emacs/notmuch-show.el
+++ b/emacs/notmuch-show.el
@@ -996,6 +996,24 @@ message at DEPTH in the current thread."
   "Insert the forest of threads FOREST."
   (mapc (lambda (thread) (notmuch-show-insert-thread thread 0)) forest))
 
+(defvar notmuch-id-regexp
+  (concat
+   ;; Match the id: prefix only if it begins a word (to disallow, for
+   ;; example, matching cid:).
+   "\\<id:\\("
+   ;; If the term starts with a ", then parse Xapian's quoted boolean
+   ;; term syntax, which allows for anything as long as embedded
+   ;; double quotes escaped by doubling them.  We also disallow
+   ;; newlines (which Xapian allows) to prevent runaway terms.
+   "\"\\([^\"\n]\\|\"\"\\)*\""
+   ;; Otherwise, parse Xapian's unquoted syntax, which goes up to the
+   ;; next space or ).  We disallow [.,;] as the last character
+   ;; because these are probably part of the surrounding text, and not
+   ;; part of the id.  This doesn't match single character ids; meh.
+   "\\|[^\"[:space:])][^[:space:])]*[^])[:space:].,:;?!]"
+   "\\)")
+  "The regexp used to match id: links in messages.")
+
 (defun notmuch-show-buttonise-links (start end)
   "Buttonise URLs and mail addresses between START and END.
 
@@ -1004,7 +1022,7 @@ a corresponding notmuch search."
   (goto-address-fontify-region start end)
   (save-excursion
     (goto-char start)
-    (while (re-search-forward "id:\\(\"?\\)[^[:space:]\"]+\\1" end t)
+    (while (re-search-forward notmuch-id-regexp end t)
       ;; remove the overlay created by goto-address-mode
       (remove-overlays (match-beginning 0) (match-end 0) 'goto-address t)
       (make-text-button (match-beginning 0) (match-end 0)
diff --git a/test/emacs-show b/test/emacs-show
index 9712633..8944e73 100755
--- a/test/emacs-show
+++ b/test/emacs-show
@@ -136,23 +136,23 @@ To: Notmuch Test Suite <test_suite@notmuchmail.org>
 Date: Fri, 05 Jan 2001 15:43:57 +0000
 
 <<id:abc>>
-<<id:abc.def.>> <<id:abc,def,>> <<id:abc;def;>> <<id:abc:def:>>
-<<id:foo@bar.?baz?>> <<id:foo@bar!.baz!>>
-(<<id:foo@bar.baz)>> [<<id:foo@bar.baz]>>
-<<id:foo@bar.baz...>>
+<<id:abc.def>>. <<id:abc,def>>, <<id:abc;def>>; <<id:abc:def>>:
+<<id:foo@bar.?baz>>? <<id:foo@bar!.baz>>!
+(<<id:foo@bar.baz>>) [<<id:foo@bar.baz>>]
+<<id:foo@bar.baz>>...
 <<id:2+2=5>>
 <<id:=_-:/.[]@$%+>>
-<<id:abc)def>>
-<<id:ab>>"c def
+<<id:abc>>)def
+<<id:ab"c>> def
 <<id:"abc">>def
-<<id:"ab">>"c"def
-id:"ab c"def
+<<id:"ab""c">>def
+<<id:"ab c">>def
 <<id:"abc">>.def
 id:"abc
 "
-<<id:)>>
+id:)
 id:
-c<<id:xxx>>
+cid:xxx
 EOF
 test_expect_equal_file EXPECTED OUTPUT
 
-- 
1.7.10.4

[PATCH v2 3/3] emacs: Buttonize mid: links

[Austin Clements]

This adds support for RFC 2392 mid: message ID links.
---
 emacs/notmuch-show.el |   45 +++++++++++++++++++++++++++++++++------------
 test/emacs-show       |    8 +++++++-
 2 files changed, 40 insertions(+), 13 deletions(-)

diff --git a/emacs/notmuch-show.el b/emacs/notmuch-show.el
index 49961fb..5b3e70e 100644
--- a/emacs/notmuch-show.el
+++ b/emacs/notmuch-show.el
@@ -1014,23 +1014,44 @@ message at DEPTH in the current thread."
    "\\)")
   "The regexp used to match id: links in messages.")
 
+(defvar notmuch-mid-regexp
+  ;; goto-address-url-regexp matched cid: links, which have the same
+  ;; grammar as the message ID part of a mid: link.  Construct the
+  ;; regexp using the same technique as goto-address-url-regexp.
+  (concat "\\<mid:\\(" thing-at-point-url-path-regexp "\\)")
+  "The regexp used to match mid: links in messages.
+
+See RFC 2392.")
+
 (defun notmuch-show-buttonise-links (start end)
   "Buttonise URLs and mail addresses between START and END.
 
-This also turns id:\"<message id>\"-parts into buttons for
-a corresponding notmuch search."
+This also turns id:\"<message id>\"-parts and mid: links into
+buttons for a corresponding notmuch search."
   (goto-address-fontify-region start end)
   (save-excursion
-    (goto-char start)
-    (while (re-search-forward notmuch-id-regexp end t)
-      ;; remove the overlay created by goto-address-mode
-      (remove-overlays (match-beginning 0) (match-end 0) 'goto-address t)
-      (make-text-button (match-beginning 0) (match-end 0)
-			'action `(lambda (arg)
-				   (notmuch-show ,(match-string-no-properties 0)))
-			'follow-link t
-			'help-echo "Mouse-1, RET: search for this message"
-			'face goto-address-mail-face))))
+    (let (links)
+      (goto-char start)
+      (while (re-search-forward notmuch-id-regexp end t)
+	(push (list (match-beginning 0) (match-end 0)
+		    (match-string-no-properties 0)) links))
+      (goto-char start)
+      (while (re-search-forward notmuch-mid-regexp end t)
+	(let* ((mid-cid (match-string-no-properties 1))
+	       (mid (save-match-data
+		      (string-match "^[^/]*" mid-cid)
+		      (url-unhex-string (match-string 0 mid-cid)))))
+	  (push (list (match-beginning 0) (match-end 0)
+		      (notmuch-id-to-query mid)) links)))
+      (dolist (link links)
+	;; Remove the overlay created by goto-address-mode
+	(remove-overlays (first link) (second link) 'goto-address t)
+	(make-text-button (first link) (second link)
+			  'action `(lambda (arg)
+				     (notmuch-show ,(third link)))
+			  'follow-link t
+			  'help-echo "Mouse-1, RET: search for this message"
+			  'face goto-address-mail-face)))))
 
 ;;;###autoload
 (defun notmuch-show (thread-id &optional parent-buffer query-context buffer-name)
diff --git a/test/emacs-show b/test/emacs-show
index 8944e73..91706ea 100755
--- a/test/emacs-show
+++ b/test/emacs-show
@@ -125,7 +125,10 @@ id:\"abc
 \"
 id:)
 id:
-cid:xxx"'
+cid:xxx
+mid:abc mid:abc/def
+mid:abc%20def
+mid:abc. mid:abc, mid:abc;"'
 test_emacs '(notmuch-show "id:'$gen_msg_id'")
 	(notmuch-test-mark-links)
 	(test-visible-output)'
@@ -153,6 +156,9 @@ id:"abc
 id:)
 id:
 cid:xxx
+<<mid:abc>> <<mid:abc/def>>
+<<mid:abc%20def>>
+<<mid:abc>>. <<mid:abc>>, <<mid:abc>>;
 EOF
 test_expect_equal_file EXPECTED OUTPUT
 
-- 
1.7.10.4

Re: [PATCH v2 0/3] Better id: link buttonization

[Ethan Glasser-Camp]

Austin Clements <amdragon@MIT.EDU> writes:

> This is v2 of id:"1351650561-7331-1-git-send-email-amdragon@mit.edu".
> This makes Jani's suggested additions to the regexp and adds support
> for RFC 2392 mid: links, as suggested by Sascha.

This series looks fine to me.

Ethan

Re: [PATCH v2 0/3] Better id: link buttonization

[Tomi Ollila]

On Wed, Nov 14 2012, Ethan Glasser-Camp wrote:

> Austin Clements <amdragon@MIT.EDU> writes:
>
>> This is v2 of id:"1351650561-7331-1-git-send-email-amdragon@mit.edu".
>> This makes Jani's suggested additions to the regexp and adds support
>> for RFC 2392 mid: links, as suggested by Sascha.
>
> This series looks fine to me.

LGTM too. 

although this is different than others in test/emacs:

 +test_expect_equal_file EXPECTED OUTPUT

(grep test_expect_equal_file test/emacs)

The diff output in case of failure makes more sense when the args
are in this order -- but probably that should be changed
bu changing the arg handling order in test_expect_equal_file
(to be consistent with test_expect_equal) and then change the order
in the "remaining" uses of test_expect_equal_file EXPECTED OUTPUT.

> Ethan

Tomi