bug#17699: Various tildify.el improvements

bug#17699: Various tildify.el improvements

[Michal Nazarewicz]

Various patches to lisp/textmodes/tildify.el file.  The changes do not
introduce any significant functionality changes.

I have one more patch addinc `auto-tildify-mode' but I'm currently
testing it and not yet ready to submit.

The changes do not contain `tildify-ignored-environments-alist' format
changes discussed with Stefan[1].  Those are left for maybe a future
patch.

[1] http://lists.gnu.org/archive/html/emacs-devel/2014-06/msg00056.html

The whole series, including the `auto-tildify-mode' patch, can also be
seen at:
    https://github.com/mina86/emacs/tree/tildify
    git pull git://github.com/mina86/emacs tildify

Michal Nazarewicz (7):
  tildify.el: Fix end-regex building in `tildify-find-env'
  tildify.el: Fix matched group indexes in end-regex building
  tildify.el: Improve defcustom's types.
  tildify.el: Better support for XML
  tildify.el: Optimise environments regexes
  tildify.el: Rewrite `tildify-region' and co., add foreach function.
  * tests/automated/tildify-tests.el (tildify-test--test): Optimise    
    the test slightly by reusing the same temporary buffer across    
    multiple test cases.

 lisp/ChangeLog                  | 105 +++++++++++++++++
 lisp/textmodes/tildify.el       | 245 ++++++++++++++++++++--------------------
 test/ChangeLog                  |  25 ++++
 test/automated/tildify-tests.el |  47 ++++++--
 4 files changed, 290 insertions(+), 132 deletions(-)

-- 
2.0.0.526.g5318336

bug#17699: [PATCH 1/7] tildify.el: Fix end-regex building in `tildify-find-env'

[Michal Nazarewicz]

* lisp/textmodes/tildify.el (tildify-find-env): The
`tildify-ignored-environments-alist' allows the end-regex
to be provided not as a static string but mix of strings and
indexes of groups matched the begin-regex.  For example, the
“\verb!…!” TeX-command (where “!” is an arbitrary character)
is handled using:

    ("\\\\verb\\*?\\(.\\)" . (1))

In the same way, the following should be supported as well:

    ("open-\\(.\\)" . ("end-" 1))

However the tildify-find-env function fails at

    (concat result
            (if (stringp (setq aux (car expression)))
                 expression  ; BUG: expression is a list
               (regexp-quote (match-string aux))))

where the string part is handled incorrectly.

The most trivial fix would be to replace `expression'
in the true-part of the if-statement with `aux', but
instead, this commit optimises `tildify-find-env' by
changing it to use `mapconcat' rather than open-coded
while-loop.

* tests/automated/tildify-tests.el (tildify-test-find-env-end-re-bug):
New test validating fix to the above bug.
---
 lisp/ChangeLog                  | 31 +++++++++++++++++++++++++++++++
 lisp/textmodes/tildify.el       | 40 ++++++++++++++++++----------------------
 test/ChangeLog                  |  6 ++++++
 test/automated/tildify-tests.el | 11 +++++++++++
 4 files changed, 66 insertions(+), 22 deletions(-)

diff --git a/lisp/ChangeLog b/lisp/ChangeLog
index a6ae6f8..dc3c42a 100644
--- a/lisp/ChangeLog
+++ b/lisp/ChangeLog
@@ -1,3 +1,34 @@
+2014-06-05  Michal Nazarewicz  <mina86@mina86.com>
+
+	* textmodes/tildify.el (tildify-find-env): Fix end-regex building
+	in `tildify-find-env'
+
+	The `tildify-ignored-environments-alist' allows the end-regex to
+	be provided not as a static string but mix of strings and indexes
+	of groups matched the begin-regex.  For example, the “\verb!…!”
+	TeX-command (where “!” is an arbitrary character) is handled
+	using:
+
+	    ("\\\\verb\\*?\\(.\\)" . (1))
+
+	In the same way, the following should be supported as well:
+
+	    ("open-\\(.\\)" . ("end-" 1))
+
+	However the tildify-find-env function fails at
+
+	    (concat result
+	            (if (stringp (setq aux (car expression)))
+	                 expression  ; BUG: expression is a list
+	               (regexp-quote (match-string aux))))
+
+	where the string part is handled incorrectly.
+
+	The most trivial fix would be to replace `expression' in the
+	true-part of the if-statement with `aux', but instead, this commit
+	optimises `tildify-find-env' by changing it to use `mapconcat'
+	rather than open-coded while-loop.
+
 2014-06-03  Rüdiger Sonderfeld  <ruediger@c-plusplus.de>
 
 	* register.el: Add link to Emacs manual in Commentary.
diff --git a/lisp/textmodes/tildify.el b/lisp/textmodes/tildify.el
index 339f900..7e4b39b 100644
--- a/lisp/textmodes/tildify.el
+++ b/lisp/textmodes/tildify.el
@@ -3,7 +3,7 @@
 ;; Copyright (C) 1997-2014 Free Software Foundation, Inc.
 
 ;; Author:     Milan Zamazal <pdm@zamazal.org>
-;; Version:    4.5.1
+;; Version:    4.5.2
 ;; Keywords:   text, TeX, SGML, wp
 
 ;; This file is part of GNU Emacs.
@@ -270,27 +270,23 @@ won't be prompted for confirmation of each substitution."
 Return regexp for the end of the environment or nil if no environment was
 found."
   ;; Find environment
-  (if (re-search-forward regexp nil t)
-      ;; Build end-env regexp
-      (let ((match (match-string 0))
-	    (alist (tildify-mode-alist tildify-ignored-environments-alist))
-	    expression)
-	(save-match-data
-	  (while (not (eq (string-match (caar alist) match) 0))
-	    (setq alist (cdr alist))))
-	(if (stringp (setq expression (cdar alist)))
-	    expression
-	  (let ((result "")
-		aux)
-	    (while expression
-	      (setq result (concat result
-				   (if (stringp (setq aux (car expression)))
-				       expression
-				     (regexp-quote (match-string aux)))))
-	      (setq expression (cdr expression)))
-	    result)))
-    ;; Return nil if not found
-    nil))
+  (when (re-search-forward regexp nil t)
+    ;; Build end-env regexp
+    (let ((match (match-string 0))
+          (alist (tildify-mode-alist tildify-ignored-environments-alist)))
+      (save-match-data
+        (while (not (eq (string-match (caar alist) match) 0))
+          (setq alist (cdr alist))))
+      (let ((expression (cdar alist)))
+        (if (stringp expression)
+            expression
+          (mapconcat
+           (lambda (expr)
+             (if (stringp expr)
+                 expr
+               (regexp-quote (match-string expr))))
+           expression
+           ""))))))
 
 (defun tildify-tildify (beg end ask)
   "Add tilde characters in the region between BEG and END.
diff --git a/test/ChangeLog b/test/ChangeLog
index b6b3dd3..db32aae 100644
--- a/test/ChangeLog
+++ b/test/ChangeLog
@@ -1,3 +1,9 @@
+2014-06-05  Michal Nazarewicz  <mina86@mina86.com>
+
+	* automated/tildify-tests.el (tildify-test-find-env-end-re-bug): New
+	test checking end-regex building in `tildify-find-env' function when
+	integers (denoting capture groups) and strings are mixed together.
+
 2014-06-02  Michael Albinus  <michael.albinus@gmx.de>
 
 	* automated/tramp-tests.el (tramp-remote-process-environment): Declare.
diff --git a/test/automated/tildify-tests.el b/test/automated/tildify-tests.el
index 4223029..25e9f22 100644
--- a/test/automated/tildify-tests.el
+++ b/test/automated/tildify-tests.el
@@ -101,6 +101,17 @@ latter is missing, SENTENCE will be used in all placeholder positions."
                         (tildify-test--example-tex sentence sentence)
                         (tildify-test--example-tex sentence with-nbsp))))
 
+
+(ert-deftest tildify-test-find-env-end-re-bug ()
+    "Tests generation of end-regex using mix of indexes and strings"
+  (with-temp-buffer
+    (let ((tildify-ignored-environments-alist
+           `((,major-mode ("foo\\|bar" . ("end-" 0))))))
+      (insert "foo whatever end-foo")
+      (goto-char (point-min))
+      (should (string-equal "end-foo" (tildify-find-env "foo\\|bar"))))))
+
+
 (provide 'tildify-tests)
 
 ;;; tildify-tests.el ends here
-- 
2.0.0.526.g5318336

bug#17699: [PATCH 2/7] tildify.el: Fix matched group indexes in end-regex building

[Michal Nazarewicz]

* lisp/textmodes/tildifi.el (tildify-find-env): When looking for
a start of an ignore-environment, the regex is built by
concatenating regexes of all the environments configured in
`tildify-ignored-environments-alist'.  So for example, the following
list could be used to match TeX's \verb and \verb* commands:

    (("\\\\verb\\(.\\)" . (1))
     ("\\\\verb\\*\\(.\\)" . (1)))

This would result in the following regex being used to find the start
of any of the variants of the \verb command:

    \\\\verb\\(.\\)\\|\\\\verb\\*\\(.\\)

But now, if “\\\\verb\\*\\(.\\)” matches, the first capture group
won't match anything, and thus (match-string 1) will be nil, which
will cause building of the end-matching regex to fail.

Fix this by using capture groups from the time when the opening
regexes are matched individually.

* tests/automated/tildify-tests.el (tildify-test-find-env-group-index-bug):
New test validating fix to the above bug.
---
 lisp/ChangeLog                  | 23 +++++++++++++++++++++++
 lisp/textmodes/tildify.el       | 30 +++++++++++++++---------------
 test/ChangeLog                  |  4 ++++
 test/automated/tildify-tests.el | 12 ++++++++++++
 4 files changed, 54 insertions(+), 15 deletions(-)

diff --git a/lisp/ChangeLog b/lisp/ChangeLog
index dc3c42a..523b2a9 100644
--- a/lisp/ChangeLog
+++ b/lisp/ChangeLog
@@ -1,5 +1,28 @@
 2014-06-05  Michal Nazarewicz  <mina86@mina86.com>
 
+	* textmodes/tildify.el (tildify-find-env): Fix matched group
+	indexes in end-regex building
+
+	When looking for a start of an ignore-environment, the regex is built
+	by concatenating regexes of all the environments configured in
+	`tildify-ignored-environments-alist'.  So for example, the following
+	list could be used to match TeX's \verb and \verb* commands:
+
+	    (("\\\\verb\\(.\\)" . (1))
+	     ("\\\\verb\\*\\(.\\)" . (1)))
+
+	This would result in the following regex being used to find the start
+	of any of the variants of the \verb command:
+
+	    \\\\verb\\(.\\)\\|\\\\verb\\*\\(.\\)
+
+	But now, if “\\\\verb\\*\\(.\\)” matches, the first capture group
+	won't match anything, and thus (match-string 1) will be nil, which
+	will cause building of the end-matching regex to fail.
+
+	Fix this by using capture groups from the time when the opening
+	regexes are matched individually.
+
 	* textmodes/tildify.el (tildify-find-env): Fix end-regex building
 	in `tildify-find-env'
 
diff --git a/lisp/textmodes/tildify.el b/lisp/textmodes/tildify.el
index 7e4b39b..7aa338e 100644
--- a/lisp/textmodes/tildify.el
+++ b/lisp/textmodes/tildify.el
@@ -271,22 +271,22 @@ Return regexp for the end of the environment or nil if no environment was
 found."
   ;; Find environment
   (when (re-search-forward regexp nil t)
-    ;; Build end-env regexp
-    (let ((match (match-string 0))
-          (alist (tildify-mode-alist tildify-ignored-environments-alist)))
-      (save-match-data
+    (save-match-data
+      ;; Build end-env regexp
+      (let ((match (match-string 0))
+            (alist (tildify-mode-alist tildify-ignored-environments-alist)))
         (while (not (eq (string-match (caar alist) match) 0))
-          (setq alist (cdr alist))))
-      (let ((expression (cdar alist)))
-        (if (stringp expression)
-            expression
-          (mapconcat
-           (lambda (expr)
-             (if (stringp expr)
-                 expr
-               (regexp-quote (match-string expr))))
-           expression
-           ""))))))
+          (setq alist (cdr alist)))
+        (let ((expression (cdar alist)))
+          (if (stringp expression)
+              expression
+            (mapconcat
+             (lambda (expr)
+               (if (stringp expr)
+                   expr
+                 (regexp-quote (match-string expr match))))
+             expression
+             "")))))))
 
 (defun tildify-tildify (beg end ask)
   "Add tilde characters in the region between BEG and END.
diff --git a/test/ChangeLog b/test/ChangeLog
index db32aae..93ef098 100644
--- a/test/ChangeLog
+++ b/test/ChangeLog
@@ -1,5 +1,9 @@
 2014-06-05  Michal Nazarewicz  <mina86@mina86.com>
 
+	* automated/tildify-tests.el (tildify-test-find-env-group-index-bug):
+	New test checking end-regex building when multiple environment pairs
+	use integers to refer to capture groups.
+
 	* automated/tildify-tests.el (tildify-test-find-env-end-re-bug): New
 	test checking end-regex building in `tildify-find-env' function when
 	integers (denoting capture groups) and strings are mixed together.
diff --git a/test/automated/tildify-tests.el b/test/automated/tildify-tests.el
index 25e9f22..6fee28b 100644
--- a/test/automated/tildify-tests.el
+++ b/test/automated/tildify-tests.el
@@ -112,6 +112,18 @@ latter is missing, SENTENCE will be used in all placeholder positions."
       (should (string-equal "end-foo" (tildify-find-env "foo\\|bar"))))))
 
 
+(ert-deftest tildify-test-find-env-group-index-bug ()
+    "Tests generation of match-string indexes"
+  (with-temp-buffer
+    (let ((tildify-ignored-environments-alist
+           `((,major-mode ("start-\\(foo\\|bar\\)" . ("end-" 1))
+                          ("open-\\(foo\\|bar\\)" . ("close-" 1)))))
+          (beg-re "start-\\(foo\\|bar\\)\\|open-\\(foo\\|bar\\)"))
+      (insert "open-foo whatever close-foo")
+      (goto-char (point-min))
+      (should (string-equal "close-foo" (tildify-find-env beg-re))))))
+
+
 (provide 'tildify-tests)
 
 ;;; tildify-tests.el ends here
-- 
2.0.0.526.g5318336

bug#17699: [PATCH 3/7] tildify.el: Improve defcustom's types

[Michal Nazarewicz]

* lisp/textmodes/tildify.el (tildify-pattern-alist)
(tildify-string-alist, tildify-ignored-environments-alist):
Add more tags explaining what each value means and replace
“sexp” used in `tildify-ignored-environments-alist' with
a full type declaration.
---
 lisp/ChangeLog            |  6 ++++++
 lisp/textmodes/tildify.el | 32 +++++++++++++++++++++++++++++---
 2 files changed, 35 insertions(+), 3 deletions(-)

diff --git a/lisp/ChangeLog b/lisp/ChangeLog
index 523b2a9..487e21c 100644
--- a/lisp/ChangeLog
+++ b/lisp/ChangeLog
@@ -1,5 +1,11 @@
 2014-06-05  Michal Nazarewicz  <mina86@mina86.com>
 
+	* textmodes/tildify.el (tildify-pattern-alist)
+	(tildify-string-alist, tildify-ignored-environments-alist):
+	Improve defcustom's types by adding more tags explaining what each
+	value means and replace “sexp” used in
+	`tildify-ignored-environments-alist' with a full type declaration.
+
 	* textmodes/tildify.el (tildify-find-env): Fix matched group
 	indexes in end-regex building
 
diff --git a/lisp/textmodes/tildify.el b/lisp/textmodes/tildify.el
index 7aa338e..d61998a 100644
--- a/lisp/textmodes/tildify.el
+++ b/lisp/textmodes/tildify.el
@@ -77,7 +77,13 @@ by the hard space character.
 The form (MAJOR-MODE . SYMBOL) defines alias item for MAJOR-MODE.  For this
 mode, the item for the mode SYMBOL is looked up in the alist instead."
   :group 'tildify
-  :type '(repeat (choice (list symbol regexp integer) (cons symbol symbol))))
+  :type '(repeat (cons :tag "Entry for major mode"
+                       (choice (const  :tag "Default" t)
+                               (symbol :tag "Major mode"))
+                       (choice (list   :tag "Regexp"
+                                       regexp
+                                       (integer :tag "Group "))
+                               (symbol :tag "Like other")))))
 
 (defcustom tildify-string-alist
   '((latex-mode . "~")
@@ -104,7 +110,12 @@ for SGML.
 The form (MAJOR-MODE . SYMBOL) defines alias item for MAJOR-MODE.  For this
 mode, the item for the mode SYMBOL is looked up in the alist instead."
   :group 'tildify
-  :type '(repeat (cons symbol (choice string symbol))))
+  :type '(repeat (cons :tag "Entry for major mode"
+                       (choice (const  :tag "Default" t)
+                               (symbol :tag "Major mode"))
+                       (choice (const  :tag "No-break space (U+00A0)" "\u00A0")
+                               (string :tag "String    ")
+                               (symbol :tag "Like other")))))
 
 (defcustom tildify-ignored-environments-alist
   '((latex-mode
@@ -160,7 +171,22 @@ END-REGEX defines end of the corresponding text part and can be either:
   subexpressions of BEG-REGEX (this is used to solve cases like
   \\\\verb<character> in TeX)."
   :group 'tildify
-  :type '(repeat (cons symbol (choice symbol (repeat sexp)))))
+  :type '(repeat
+          (cons :tag "Entry for major mode"
+                (choice (const  :tag "Default" t)
+                        (symbol :tag "Major mode"))
+                (choice
+                 (const  :tag "None")
+                 (repeat
+                  :tag "Environments"
+                  (cons :tag "Regexp pair"
+                        (regexp :tag "Open ")
+                        (choice :tag "Close"
+                                (regexp :tag "Regexp")
+                                (list :tag "Regexp and groups (concatenated)"
+                                      (choice (regexp  :tag "Regexp")
+                                              (integer :tag "Group "))))))
+                 (symbol :tag "Like other")))))
 
 
 ;;; *** Internal variables ***
-- 
2.0.0.526.g5318336

bug#17699: [PATCH 4/7] tildify.el: Better support for XML

[Michal Nazarewicz]

* lisp/textmodes/tildify.el  (tildify-string-alist)
(tildify-ignored-environments-alist): Add `nxml-mode' to the list of
supported modes since `xml-mode' is no longer a thing but just an
alias to the former.  Also include comments and insides of tags in
`tildify-ignored-environments-alist' for XML modes.  Finally, since
XML does not define “ ”[1], use a numeric reference for
a no-break space (namely “ ”)

[1] XML specification defines only a handful of predefined entities.
    The list is at <http://www.w3.org/TR/REC-xml/#sec-predefined-ent>
    and includes only &lt;, &gt;, &amp;, &apos; and &quot; (meaning <,
    >, &, ' and " respectively).  This is in contrast to HTML and even
    XHTML which defined a whole bunch of entities including “&nbsp;”.

* automated/tildify-tests.el (tildify-test--example-html): Add support
for generating XML code, so that…
(tildify-test-xml) …test can be added to check handling of XML
documents.
---
 lisp/ChangeLog                  | 14 ++++++++++++++
 lisp/textmodes/tildify.el       |  7 ++++++-
 test/ChangeLog                  |  5 +++++
 test/automated/tildify-tests.el | 15 ++++++++++++---
 4 files changed, 37 insertions(+), 4 deletions(-)

diff --git a/lisp/ChangeLog b/lisp/ChangeLog
index 487e21c..c662add 100644
--- a/lisp/ChangeLog
+++ b/lisp/ChangeLog
@@ -1,5 +1,19 @@
 2014-06-05  Michal Nazarewicz  <mina86@mina86.com>
 
+	* textmodes/tildify.el (tildify-string-alist)
+	(tildify-ignored-environments-alist): Add `nxml-mode' to the list
+	of supported modes since `xml-mode' is no longer a thing but just
+	an alias to the former.  Also include comments and insides of tags
+	in `tildify-ignored-environments-alist' for XML modes.  Finally,
+	since XML does not define “&nbsp;”[1], use a numeric reference for
+	a no-break space (namely “&#160;”)
+
+	[1] XML specification defines only a handful of predefined entities.
+	    The list is at <http://www.w3.org/TR/REC-xml/#sec-predefined-ent>
+	    and includes only &lt;, &gt;, &amp;, &apos; and &quot; (meaning <,
+	    >, &, ' and " respectively).  This is in contrast to HTML and even
+	    XHTML which defined a whole bunch of entities including “&nbsp;”.
+
 	* textmodes/tildify.el (tildify-pattern-alist)
 	(tildify-string-alist, tildify-ignored-environments-alist):
 	Improve defcustom's types by adding more tags explaining what each
diff --git a/lisp/textmodes/tildify.el b/lisp/textmodes/tildify.el
index d61998a..6dd471d 100644
--- a/lisp/textmodes/tildify.el
+++ b/lisp/textmodes/tildify.el
@@ -90,8 +90,9 @@ mode, the item for the mode SYMBOL is looked up in the alist instead."
     (tex-mode . latex-mode)
     (plain-tex-mode . latex-mode)
     (sgml-mode . "&nbsp;")
-    (xml-mode . sgml-mode)
     (html-mode . sgml-mode)
+    (xml-mode . "&#160;") ; XML does not define &nbsp;, use numeric reference
+    (nxml-mode . xml-mode)
     (t . " "))
   "Alist specifying what is a hard space in the current major mode.
 
@@ -149,6 +150,10 @@ mode, the item for the mode SYMBOL is looked up in the alist instead."
      ("<! *--" . "-- *>")
      ("<" . ">"))
     (sgml-mode . html-mode)
+    (xml-mode
+     ("<! *--" . "-- *>")
+     ("<" . ">"))
+    (nxml-mode . xml-mode)
     (t nil))
   "Alist specifying ignored structured text environments.
 Parts of text defined in this alist are skipped without performing hard space
diff --git a/test/ChangeLog b/test/ChangeLog
index 93ef098..38a4feb 100644
--- a/test/ChangeLog
+++ b/test/ChangeLog
@@ -1,5 +1,10 @@
 2014-06-05  Michal Nazarewicz  <mina86@mina86.com>
 
+	* automated/tildify-tests.el (tildify-test--example-html): Add support
+	for generating XML code, so that…
+	(tildify-test-xml) …test can be added to check handling of XML
+	documents.
+
 	* automated/tildify-tests.el (tildify-test-find-env-group-index-bug):
 	New test checking end-regex building when multiple environment pairs
 	use integers to refer to capture groups.
diff --git a/test/automated/tildify-tests.el b/test/automated/tildify-tests.el
index 6fee28b..dd404fc 100644
--- a/test/automated/tildify-tests.el
+++ b/test/automated/tildify-tests.el
@@ -36,14 +36,15 @@
           "consectetur adipiscing elit."))
 
 
-(defun tildify-test--example-html (sentence &optional with-nbsp)
+(defun tildify-test--example-html (sentence &optional with-nbsp is-xml)
   "Return an example HTML code.
 SENTENCE is placed where spaces should not be replaced with hard spaces, and
 WITH-NBSP is placed where spaces should be replaced with hard spaces.  If the
-latter is missing, SENTENCE will be used in all placeholder positions."
+latter is missing, SENTENCE will be used in all placeholder positions.
+If IS-XML is non-nil, <pre> tag is not treated specially."
   (let ((with-nbsp (or with-nbsp sentence)))
     (concat "<p>" with-nbsp "</p>\n"
-            "<pre>" sentence "</pre>\n"
+            "<pre>" (if is-xml with-nbsp sentence) "</pre>\n"
             "<! -- " sentence " -- >\n"
             "<p>" with-nbsp "</p>\n"
             "<" sentence ">\n")))
@@ -77,6 +78,14 @@ after `tildify-buffer' is run."
                         (tildify-test--example-html sentence sentence)
                         (tildify-test--example-html sentence with-nbsp))))
 
+(ert-deftest tildify-test-xml ()
+  "Tests tildification in an XML document"
+  (let* ((sentence (tildify-test--example-sentence " "))
+         (with-nbsp (tildify-test--example-sentence "&#160;")))
+    (tildify-test--test '(nxml-mode)
+                        (tildify-test--example-html sentence sentence t)
+                        (tildify-test--example-html sentence with-nbsp t))))
+
 
 (defun tildify-test--example-tex (sentence &optional with-nbsp)
   "Return an example (La)Tex code.
-- 
2.0.0.526.g5318336

bug#17699: [PATCH 5/7] tildify.el: Optimise environments regexes

[Michal Nazarewicz]

* lisp/textmodes/tildify.el (tildify-ignored-environments-alist):
Each time beginning of an environment to ignore is found,
`tildify-find-env' needs to identify regexp for the ending
of the environment.  This is done by trying all the opening
regexes on matched text in a loop, so to speed that up, this
loop should have fewer things to match, which can be done by
using alternatives in the opening regexes.

Coincidentally, this should make matching of the opening
regexp faster as well thanks to the use of `regexp-opt' and
having common prefix pulled from many regexes.
---
 lisp/ChangeLog            | 14 ++++++++++++++
 lisp/textmodes/tildify.el | 37 +++++++++++++++----------------------
 2 files changed, 29 insertions(+), 22 deletions(-)

diff --git a/lisp/ChangeLog b/lisp/ChangeLog
index c662add..0351d54 100644
--- a/lisp/ChangeLog
+++ b/lisp/ChangeLog
@@ -1,5 +1,19 @@
 2014-06-05  Michal Nazarewicz  <mina86@mina86.com>
 
+	* textmodes/tildify.el (tildify-ignored-environments-alist):
+	Optimise environments regexes
+
+	Each time beginning of an environment to ignore is found,
+	`tildify-find-env' needs to identify regexp for the ending
+	of the environment.  This is done by trying all the opening
+	regexes on matched text in a loop, so to speed that up, this
+	loop should have fewer things to match, which can be done by
+	using alternatives in the opening regexes.
+
+	Coincidentally, this should make matching of the opening
+	regexp faster as well thanks to the use of `regexp-opt' and
+	having common prefix pulled from many regexes.
+
 	* textmodes/tildify.el (tildify-string-alist)
 	(tildify-ignored-environments-alist): Add `nxml-mode' to the list
 	of supported modes since `xml-mode' is no longer a thing but just
diff --git a/lisp/textmodes/tildify.el b/lisp/textmodes/tildify.el
index 6dd471d..39ccad7 100644
--- a/lisp/textmodes/tildify.el
+++ b/lisp/textmodes/tildify.el
@@ -119,42 +119,35 @@ mode, the item for the mode SYMBOL is looked up in the alist instead."
                                (symbol :tag "Like other")))))
 
 (defcustom tildify-ignored-environments-alist
-  '((latex-mode
+  `((latex-mode
      ("\\\\\\\\" . "")		; do not remove this
-     ("\\\\begin{verbatim}" . "\\\\end{verbatim}")
+     (,(eval-when-compile (concat
+                           "\\\\begin{\\("
+                           (regexp-opt '("verbatim" "math" "displaymath"
+                                         "equation" "eqnarray" "eqnarray*"))
+                           "\\)}"))
+      . ("\\\\end{" 1 "}"))
      ("\\\\verb\\*?\\(.\\)" . (1))
-     ("\\$\\$" . "\\$\\$")
-     ("\\$" . "\\$")
+     ("\\$\\$?" . (0))
      ("\\\\(" . "\\\\)")
      ("\\\\[[]" . "\\\\[]]")
-     ("\\\\begin{math}" . "\\\\end{math}")
-     ("\\\\begin{displaymath}" . "\\\\end{displaymath}")
-     ("\\\\begin{equation}" . "\\\\end{equation}")
-     ("\\\\begin{eqnarray\\*?}" . "\\\\end{eqnarray\\*?}")
      ("\\\\[a-zA-Z]+\\( +\\|{}\\)[a-zA-Z]*" . "")
      ("%" . "$"))
     (plain-tex-mode . latex-mode)
     (html-mode
-     ("<pre[^>]*>" . "</pre>")
-     ("<dfn>" . "</dfn>")
-     ("<code>" . "</code>")
-     ("<samp>" . "</samp>")
-     ("<kbd>" . "</kbd>")
-     ("<var>" . "</var>")
-     ("<PRE[^>]*>" . "</PRE>")
-     ("<DFN>" . "</DFN>")
-     ("<CODE>" . "</CODE>")
-     ("<SAMP>" . "</SAMP>")
-     ("<KBD>" . "</KBD>")
-     ("<VAR>" . "</VAR>")
+     (,(eval-when-compile (concat
+                           "<\\("
+                           (regexp-opt '("pre" "dfn" "code" "samp" "kbd" "var"
+                                         "PRE" "DFN" "CODE" "SAMP" "KBD" "VAR"))
+                           "\\)\\>[^>]*>"))
+      . ("</" 1 ">"))
      ("<! *--" . "-- *>")
      ("<" . ">"))
     (sgml-mode . html-mode)
     (xml-mode
      ("<! *--" . "-- *>")
      ("<" . ">"))
-    (nxml-mode . xml-mode)
-    (t nil))
+    (nxml-mode . xml-mode))
   "Alist specifying ignored structured text environments.
 Parts of text defined in this alist are skipped without performing hard space
 insertion on them.  These setting allow skipping text parts like verbatim or
-- 
2.0.0.526.g5318336

bug#17699: [PATCH 6/7] tildify.el: Rewrite `tildify-region' and co., add foreach function.

[Michal Nazarewicz]

* lisp/textmodes/tildify.el (tildify-foreach-region-outside-env): New
function which calls a callback on portions of the buffer that are
outside of ignored environments.
(tildify-build-regexp): Remove function since it is now
incorporated in `tildify-foreach-region-outside-env' where it is
optimised and simplified by the use of `mapconcat'.
(tildify-tildify): Return number of substitutions made so that…
(tildify-count): …can be removed.
(tildify-find-env): Accept a new PAIRS argument which was
previously looked up in `tildify-ignored-environments-alist' each
time the function was called.  With this change, the lookup is
performed only once in `tildify-foreach-region-outside-env'.
(tildify-region): Greatly simplify the function since now most of
the work is done by `tildify-foreach-region-outside-env'.
(tildify-mode-alist): Simplify slightly by avoiding if and setq
and instead using or.

* tests/automated/tildify-tests.el (tildify-test-find-env-end-re-bug)
(tildify-test-find-env-group-index-bug): Update to support new
signature of the `tildify-foreach-region-outside-env' function.
Namely, it now takes pairs as an argument instead of looking it up in
`tildify-ignored-environments-alist'.
---
 lisp/ChangeLog                  |  17 +++++
 lisp/textmodes/tildify.el       | 145 +++++++++++++++++-----------------------
 test/ChangeLog                  |   6 ++
 test/automated/tildify-tests.el |  17 +++--
 4 files changed, 92 insertions(+), 93 deletions(-)

diff --git a/lisp/ChangeLog b/lisp/ChangeLog
index 0351d54..075a424 100644
--- a/lisp/ChangeLog
+++ b/lisp/ChangeLog
@@ -1,5 +1,22 @@
 2014-06-05  Michal Nazarewicz  <mina86@mina86.com>
 
+	* textmodes/tildify.el (tildify-foreach-region-outside-env): New
+	function which calls a callback on portions of the buffer that are
+	outside of ignored environments.
+	(tildify-build-regexp): Remove function since it is now
+	incorporated in `tildify-foreach-region-outside-env' where it is
+	optimised and simplified by the use of `mapconcat'.
+	(tildify-tildify): Return number of substitutions made so that…
+	(tildify-count): …can be removed.
+	(tildify-find-env): Accept a new PAIRS argument which was
+	previously looked up in `tildify-ignored-environments-alist' each
+	time the function was called.  With this change, the lookup is
+	performed only once in `tildify-foreach-region-outside-env'.
+	(tildify-region): Greatly simplify the function since now most of
+	the work is done by `tildify-foreach-region-outside-env'.
+	(tildify-mode-alist): Simplify slightly by avoiding if and setq
+	and instead using or.
+
 	* textmodes/tildify.el (tildify-ignored-environments-alist):
 	Optimise environments regexes
 
diff --git a/lisp/textmodes/tildify.el b/lisp/textmodes/tildify.el
index 39ccad7..50fee2f 100644
--- a/lisp/textmodes/tildify.el
+++ b/lisp/textmodes/tildify.el
@@ -3,7 +3,8 @@
 ;; Copyright (C) 1997-2014 Free Software Foundation, Inc.
 
 ;; Author:     Milan Zamazal <pdm@zamazal.org>
-;; Version:    4.5.2
+;;             Michal Nazarewicz <mina86@mina86.com>
+;; Version:    4.5.3
 ;; Keywords:   text, TeX, SGML, wp
 
 ;; This file is part of GNU Emacs.
@@ -187,12 +188,6 @@ END-REGEX defines end of the corresponding text part and can be either:
                  (symbol :tag "Like other")))))
 
 
-;;; *** Internal variables ***
-
-(defvar tildify-count nil
-  "Counter for replacements.")
-
-
 ;;; *** Interactive functions ***
 
 ;;;###autoload
@@ -205,51 +200,16 @@ This function performs no refilling of the changed text.
 If DONT-ASK is set, or called interactively with prefix argument, user
 won't be prompted for confirmation of each substitution."
   (interactive "*rP")
-  (setq tildify-count 0)
-  (let (a
-	z
-	(marker-end (copy-marker end))
-	end-env
-	finish
-	(ask (not dont-ask))
-	(case-fold-search nil)
-	(regexp (tildify-build-regexp))	; beginnings of environments
-	aux)
-    (if regexp
-	;; Yes, ignored environments exist for the current major mode,
-	;; tildify just texts outside them
-	(save-excursion
-	  (save-restriction
-	    (widen)
-	    (goto-char (point-min))
-	    (while (not finish)
-	      (setq a (point))
-	      (setq end-env (tildify-find-env regexp))
-	      (setq z (copy-marker (if end-env (1- (point)) (point-max))))
-	      (if (>= (marker-position z) beg)
-		  (progn
-		    (or (>= a beg) (setq a beg))
-		    (or (<= (marker-position z) (marker-position marker-end))
-			(setq z marker-end))
-		    (setq aux (tildify-tildify a (marker-position z) ask))
-		    (if (eq aux 'force)
-			(setq ask nil)
-		      (if (eq aux nil)
-			  (setq finish t)))))
-	      (if (>= (marker-position z) (marker-position marker-end))
-		  (setq finish t))
-	      (or (>= (point) (marker-position z))
-		  (goto-char (marker-position z)))
-	      (if (not finish)
-		  (if (re-search-forward end-env nil t)
-		      (if (> (point) (marker-position marker-end))
-			  (setq finish t))
-		    (message
-		     "End of environment not found: %s" end-env)
-		    (setq finish t))))))
-      ;; No ignored environments, tildify directly
-      (tildify-tildify beg end ask)))
-  (message "%d spaces replaced." tildify-count))
+  (let (case-fold-search (count 0) (ask (not dont-ask)))
+    (tildify-foreach-region-outside-env beg end
+      (lambda (beg end)
+        (let ((aux (tildify-tildify beg end ask)))
+          (setq count (+ count (car aux)))
+          (if (not (eq (cdr aux) 'force))
+              (cdr aux)
+            (setq ask nil)
+            t))))
+    (message "%d spaces replaced." count)))
 
 ;;;###autoload
 (defun tildify-buffer (&optional dont-ask)
@@ -266,42 +226,58 @@ won't be prompted for confirmation of each substitution."
 
 ;;; *** Auxiliary functions ***
 
-(defun tildify-build-regexp ()
-  "Build start of environment regexp."
-  (let ((alist (tildify-mode-alist tildify-ignored-environments-alist))
-	regexp)
-    (when alist
-      (setq regexp (caar alist))
-      (setq alist (cdr alist))
-      (while alist
-	(setq regexp (concat regexp "\\|" (caar alist)))
-	(setq alist (cdr alist)))
-      regexp)))
-
 (defun tildify-mode-alist (mode-alist &optional mode)
   "Return alist item for the MODE-ALIST in the current major MODE."
-  (if (null mode)
-      (setq mode major-mode))
-  (let ((alist (cdr (or (assoc mode mode-alist)
+  (let ((alist (cdr (or (assoc (or mode major-mode) mode-alist)
 			(assoc t mode-alist)))))
     (if (and alist
 	     (symbolp alist))
 	(tildify-mode-alist mode-alist alist)
       alist)))
 
-(defun tildify-find-env (regexp)
+(defun tildify-foreach-region-outside-env (beg end callback)
+  "Scan region from BEG to END calling CALLBACK on portions out of environments.
+Call CALLBACK on each region outside of environment to ignore.
+CALLBACK will only be called for regions which have intersection
+with [BEG END].  It must be a function that takes two point
+arguments specifying the region to operate on.  Stop scanning the
+region as soon as CALLBACK returns nil.  Environments to ignore
+are determined from `tildify-ignored-environments-alist'."
+  (declare (indent 2))
+  (let ((pairs (tildify-mode-alist tildify-ignored-environments-alist)))
+    (if (not pairs)
+        (funcall callback beg end)
+      (let ((func (lambda (b e)
+                    (let ((b (max b beg)) (e (min e end)))
+                    (if (< b e) (funcall callback b e) t))))
+            (beg-re (concat "\\(?:"
+                            (mapconcat 'car pairs "\\)\\|\\(?:")
+                            "\\)"))
+            p end-re)
+        (save-excursion
+          (save-restriction
+            (widen)
+            (goto-char (point-min))
+            (while (and (< (setq p (point)) end)
+                        (if (not (setq end-re
+                                       (tildify-find-env beg-re pairs)))
+                            (progn (funcall func p end) nil)
+                          (funcall func p (match-beginning 0))
+                          (when (< (point) end)
+                            (setq p (point))
+                            (re-search-forward end-re nil t)))))))))))
+
+(defun tildify-find-env (regexp pairs)
   "Find environment using REGEXP.
-Return regexp for the end of the environment or nil if no environment was
-found."
+Return regexp for the end of the environment found in PAIRS or nil if
+no environment was found."
   ;; Find environment
   (when (re-search-forward regexp nil t)
     (save-match-data
-      ;; Build end-env regexp
-      (let ((match (match-string 0))
-            (alist (tildify-mode-alist tildify-ignored-environments-alist)))
-        (while (not (eq (string-match (caar alist) match) 0))
-          (setq alist (cdr alist)))
-        (let ((expression (cdar alist)))
+      (let ((match (match-string 0)))
+        (while (not (eq (string-match (caar pairs) match) 0))
+          (setq pairs (cdr pairs)))
+        (let ((expression (cdar pairs)))
           (if (stringp expression)
               expression
             (mapconcat
@@ -319,8 +295,9 @@ macros.
 
 If ASK is nil, perform replace without asking user for confirmation.
 
-Returns one of symbols: t (all right), nil (quit), force (replace without
-further questions)."
+Returns (count . response) cons where count is number of string
+replacements done and response is one of symbols: t (all right), nil
+(quit), force (replace without further questions)."
   (save-excursion
     (goto-char beg)
     (let* ((alist (tildify-mode-alist tildify-pattern-alist))
@@ -332,7 +309,8 @@ further questions)."
 	   bad-answer
 	   replace
 	   quit
-	   (message-log-max nil))
+	   (message-log-max nil)
+	   (count 0))
       (while (and (not quit)
 		  (re-search-forward regexp (marker-position end-marker) t))
 	(when (or (not ask)
@@ -359,12 +337,11 @@ further questions)."
 		      (setq bad-answer t)))
 		    replace))
 	  (replace-match tilde t t nil match-number)
-	  (setq tildify-count (1+ tildify-count))))
+	  (setq count (1+ count))))
       ;; Return value
-      (cond
-       (quit nil)
-       ((not ask) 'force)
-       (t t)))))
+      (cons count (cond (quit nil)
+                        ((not ask) 'force)
+                        (t t))))))
 
 
 ;;; *** Announce ***
diff --git a/test/ChangeLog b/test/ChangeLog
index 38a4feb..6248d6c 100644
--- a/test/ChangeLog
+++ b/test/ChangeLog
@@ -1,5 +1,11 @@
 2014-06-05  Michal Nazarewicz  <mina86@mina86.com>
 
+	* automated/tildify-tests.el (tildify-test-find-env-end-re-bug)
+	(tildify-test-find-env-group-index-bug): Update to support new
+	signature of the `tildify-foreach-region-outside-env' function.
+	Namely, it now takes pairs as an argument instead of looking it up in
+	`tildify-ignored-environments-alist'.
+
 	* automated/tildify-tests.el (tildify-test--example-html): Add support
 	for generating XML code, so that…
 	(tildify-test-xml) …test can be added to check handling of XML
diff --git a/test/automated/tildify-tests.el b/test/automated/tildify-tests.el
index dd404fc..cf18320 100644
--- a/test/automated/tildify-tests.el
+++ b/test/automated/tildify-tests.el
@@ -114,23 +114,22 @@ latter is missing, SENTENCE will be used in all placeholder positions."
 (ert-deftest tildify-test-find-env-end-re-bug ()
     "Tests generation of end-regex using mix of indexes and strings"
   (with-temp-buffer
-    (let ((tildify-ignored-environments-alist
-           `((,major-mode ("foo\\|bar" . ("end-" 0))))))
-      (insert "foo whatever end-foo")
-      (goto-char (point-min))
-      (should (string-equal "end-foo" (tildify-find-env "foo\\|bar"))))))
+    (insert "foo whatever end-foo")
+    (goto-char (point-min))
+    (should (string-equal "end-foo"
+                          (tildify-find-env "foo\\|bar"
+                                            '(("foo\\|bar" . ("end-" 0))))))))
 
 
 (ert-deftest tildify-test-find-env-group-index-bug ()
     "Tests generation of match-string indexes"
   (with-temp-buffer
-    (let ((tildify-ignored-environments-alist
-           `((,major-mode ("start-\\(foo\\|bar\\)" . ("end-" 1))
-                          ("open-\\(foo\\|bar\\)" . ("close-" 1)))))
+    (let ((pairs '(("start-\\(foo\\|bar\\)" . ("end-" 1))
+                   ("open-\\(foo\\|bar\\)" . ("close-" 1))))
           (beg-re "start-\\(foo\\|bar\\)\\|open-\\(foo\\|bar\\)"))
       (insert "open-foo whatever close-foo")
       (goto-char (point-min))
-      (should (string-equal "close-foo" (tildify-find-env beg-re))))))
+      (should (string-equal "close-foo" (tildify-find-env beg-re pairs))))))
 
 
 (provide 'tildify-tests)
-- 
2.0.0.526.g5318336

bug#17699: [PATCH 7/7] * tests/automated/tildify-tests.el (tildify-test--test): Optimise the test slightly by reusing the same temporary buffer across multiple test cases.

[Michal Nazarewicz]

---
 test/ChangeLog                  |  4 ++++
 test/automated/tildify-tests.el | 10 +++++-----
 2 files changed, 9 insertions(+), 5 deletions(-)

diff --git a/test/ChangeLog b/test/ChangeLog
index 6248d6c..bb8ed89 100644
--- a/test/ChangeLog
+++ b/test/ChangeLog
@@ -1,5 +1,9 @@
 2014-06-05  Michal Nazarewicz  <mina86@mina86.com>
 
+	* automated/tildify-tests.el (tildify-test--test): Optimise the test
+	slightly by reusing the same temporary buffer across multiple test
+	cases.
+
 	* automated/tildify-tests.el (tildify-test-find-env-end-re-bug)
 	(tildify-test-find-env-group-index-bug): Update to support new
 	signature of the `tildify-foreach-region-outside-env' function.
diff --git a/test/automated/tildify-tests.el b/test/automated/tildify-tests.el
index cf18320..86c83d7 100644
--- a/test/automated/tildify-tests.el
+++ b/test/automated/tildify-tests.el
@@ -54,16 +54,16 @@ If IS-XML is non-nil, <pre> tag is not treated specially."
   "Test tildify running in MODES.
 INPUT is the initial content of the buffer and EXPECTED is expected result
 after `tildify-buffer' is run."
-  (dolist (mode modes)
-    (with-temp-buffer
+  (with-temp-buffer
+    (dolist (mode modes)
+      (erase-buffer)
       (funcall mode)
       (let ((header (concat "Testing `tildify-buffer' in "
                             (symbol-name mode) "\n")))
         (insert header input)
         (tildify-buffer t)
-        (should (string-equal (concat header expected) (buffer-string)))))
-    (with-temp-buffer
-      (funcall mode)
+        (should (string-equal (concat header expected) (buffer-string))))
+      (erase-buffer)
       (let ((header (concat "Testing `tildify-region' in "
                             (symbol-name mode) "\n")))
         (insert header input)
-- 
2.0.0.526.g5318336

bug#17699: Various tildify.el improvements

[Stefan Monnier]

> Various patches to lisp/textmodes/tildify.el file.  The changes do not
> introduce any significant functionality changes.

They look fine to me, feel free to install them into `trunk'.


        Stefan

bug#17699: Various tildify.el improvements

[Michal Nazarewicz]

On Thu, Jun 05 2014, Stefan Monnier <monnier@iro.umontreal.ca> wrote:
>> Various patches to lisp/textmodes/tildify.el file.  The changes do not
>> introduce any significant functionality changes.
>
> They look fine to me, feel free to install them into `trunk'.

Pushed.

-- 
Best regards,                                         _     _
.o. | Liege of Serenely Enlightened Majesty of      o' \,=./ `o
..o | Computer Science,  Michał “mina86” Nazarewicz    (o o)
ooo +--<mpn@google.com>--<xmpp:mina86@jabber.org>--ooO--(_)--Ooo--

bug#17699: Various tildify.el improvements

[Glenn Morris]

Please now close the bug report, by sending a mail to 17699-done@debbugs.
(I like to put "Version: 24.5" in the first line of the message body,
to indicate which Emacs version the fix is expected to appear in.)

bug#17699: Various tildify.el improvements

[Michal Nazarewicz]

Version: 24.5