[RFC] Change \[..\] and $$...$$ into elements

[RFC] Change \[..\] and $$...$$ into elements

[Nicolas Goaziou]

[-- Attachment #1: Type: text/plain, Size: 770 bytes --]

Hello,

The following patch changes status for \[..\] and $$...$$ from objects
to elements.

There are two consequences to this :

  1. \[...\] and $$...$$ are not filled anymore with M-q.

  2. \[...\] and $$...$$ cannot be inlined, i.e, they must start at the
     beginning of a line.

In particular, the second point may introduce incompatibilities with
previous Org revisions since, e.g.,

  Text \[...\] and so on...

is not recognized anymore as a LaTeX object.

Also, I'm not keen on allowing both inline and block state for the same
syntax, as it would create create a precedent.

This patch is missing documentation updates and an ORG-NEWS entry.


Feedback is welcome.


Regards,

-- 
Nicolas Goaziou                                                0x80A93738

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Make-.-and-.-elements-instead-of-objects.patch --]
[-- Type: text/x-patch, Size: 9018 bytes --]

From 946eb109ac581fc25a67af4c57afb35a09d25c18 Mon Sep 17 00:00:00 2001
From: Nicolas Goaziou <mail@nicolasgoaziou.fr>
Date: Sat, 9 May 2015 11:29:48 +0200
Subject: [PATCH] Make \[...\] and $$...$$ elements instead of objects

* lisp/org-element.el (org-element--set-regexps): Update regexp.
(org-element--latex-begin-environment):
(org-element--latex-end-environment): Remove variable.  Change it into
a function.
(org-element-latex-environment-parser):
(org-element-paragraph-parser): Use function.
(org-element-latex-fragment-parser): Remove \[...\] and $$...$$
constructs.

* lisp/org.el (org-latex-regexps):
* lisp/ox-latex.el (org-latex--wrap-latex-math-block): Propagate
  changes.

* testing/lisp/test-org-element.el (test-org-element/latex-environment-parser):
  Add a test.
(test-org-element/latex-fragment-parser): Update test.

With this change \[...\] and $$...$$ are considered elements and cannot
be inlined anymore.  As a consequence, they will not be filled anymore
but will stop paragraphs.
---
 lisp/org-element.el              | 58 +++++++++++++++++++---------------------
 lisp/org.el                      |  4 +--
 lisp/ox-latex.el                 |  2 +-
 testing/lisp/test-org-element.el | 27 ++++++++++++++-----
 4 files changed, 50 insertions(+), 41 deletions(-)

diff --git a/lisp/org-element.el b/lisp/org-element.el
index d6695c8..24ca339 100644
--- a/lisp/org-element.el
+++ b/lisp/org-element.el
@@ -173,7 +173,7 @@ specially in `org-element--object-lex'.")
 		;; Horizontal rules.
 		"-\\{5,\\}[ \t]*$" "\\|"
 		;; LaTeX environments.
-		"\\\\begin{\\([A-Za-z0-9*]+\\)}" "\\|"
+		"\\\\\\(begin{\\([A-Za-z0-9*]+\\)}\\|\\[\\|\\$\\$\\)" "\\|"
 		;; Clock lines.
 		(regexp-quote org-clock-string) "\\|"
 		;; Lists.
@@ -216,7 +216,7 @@ specially in `org-element--object-lex'.")
 		      ;; Objects starting with "$": latex fragment.
 		      "\\$"
 		      ;; Objects starting with "\": line break,
-		      ;; entity, latex fragment.
+		      ;; entity.
 		      "\\\\\\(?:[a-zA-Z[(]\\|\\\\[ \t]*$\\|_ +\\)"
 		      ;; Objects starting with raw text: inline Babel
 		      ;; source block, inline Babel call.
@@ -2112,16 +2112,19 @@ containing `:key', `:value', `:begin', `:end', `:post-blank' and
 ;;;; Latex Environment
 
 (defconst org-element--latex-begin-environment
-  "^[ \t]*\\\\begin{\\([A-Za-z0-9*]+\\)}"
+  "^[ \t]*\\(\\\\\\(?:begin{[A-Za-z0-9*]+}\\|\\[\\)\\|\\$\\$\\)"
   "Regexp matching the beginning of a LaTeX environment.
-The environment is captured by the first group.
-
-See also `org-element--latex-end-environment'.")
-
-(defconst org-element--latex-end-environment
-  "\\\\end{%s}[ \t]*$"
-  "Format string matching the ending of a LaTeX environment.
-See also `org-element--latex-begin-environment'.")
+The environment is captured by the first group.  See also
+`org-element--latex-end-environment'.")
+
+(defun org-element--latex-end-environment (environment)
+  "Build regexp matching the end of a LaTeX environment.
+See also `org-element--latex-begin-environment'."
+  (concat (pcase environment
+	    ("$$" "\\$\\$")
+	    ("\\[" "\\]")
+	    (e (format "\\\\end{%s}" (regexp-quote (substring e 7 -1)))))
+	  "[ \t]*$"))
 
 (defun org-element-latex-environment-parser (limit affiliated)
   "Parse a LaTeX environment.
@@ -2140,9 +2143,8 @@ Assume point is at the beginning of the latex environment."
     (let ((case-fold-search t)
 	  (code-begin (point)))
       (looking-at org-element--latex-begin-environment)
-      (if (not (re-search-forward (format org-element--latex-end-environment
-					  (regexp-quote (match-string 1)))
-				  limit t))
+      (if (not (re-search-forward
+		(org-element--latex-end-environment (match-string 1)) limit t))
 	  ;; Incomplete latex environment: parse it as a paragraph.
 	  (org-element-paragraph-parser limit affiliated)
 	(let* ((code-end (progn (forward-line) (point)))
@@ -2242,8 +2244,7 @@ Assume point is at the beginning of the paragraph."
 		       ((looking-at org-element--latex-begin-environment)
 			(save-excursion
 			  (re-search-forward
-			   (format org-element--latex-end-environment
-				   (regexp-quote (match-string 1)))
+			   (org-element--latex-end-environment (match-string 1))
 			   limit t)))
 		       ((looking-at "[ \t]*#\\+\\(\\S-+\\)\\[.*\\]:")
 			(member-ignore-case (match-string 1)
@@ -2928,22 +2929,17 @@ Assume point is at the beginning of the LaTeX fragment."
       (let* ((begin (point))
 	     (after-fragment
 	      (if (eq (char-after) ?$)
-		  (if (eq (char-after (1+ (point))) ?$)
-		      (search-forward "$$" nil t 2)
-		    (and (not (eq (char-before) ?$))
-			 (search-forward "$" nil t 2)
-			 (not (memq (char-before (match-beginning 0))
-				    '(?\s ?\t ?\n ?, ?.)))
-			 (looking-at "\\(\\s.\\|\\s-\\|\\s(\\|\\s)\\|\\s\"\\|$\\)")
-			 (point)))
-		(pcase (char-after (1+ (point)))
-		  (?\( (search-forward "\\)" nil t))
-		  (?\[ (search-forward "\\]" nil t))
-		  (_
-		   ;; Macro.
-		   (and (looking-at "\\\\[a-zA-Z]+\\*?\\(\\(\\[[^][\n{}]*\\]\\)\
+		  (and (search-forward "$" nil t 2)
+		       (not (memq (char-before (match-beginning 0))
+				  '(?\s ?\t ?\n ?, ?.)))
+		       (looking-at "\\(\\s.\\|\\s-\\|\\s(\\|\\s)\\|\\s\"\\|$\\)")
+		       (point))
+		(if (eq (char-after (1+ (point))) ?\()
+		    (search-forward "\\)" nil t)
+		  ;; Macro.
+		  (and (looking-at "\\\\[a-zA-Z]+\\*?\\(\\(\\[[^][\n{}]*\\]\\)\
 \\|\\({[^{}\n]*}\\)\\)*")
-			(match-end 0))))))
+		       (match-end 0)))))
 	     (post-blank (if (not after-fragment) (throw 'no-object nil)
 			   (goto-char after-fragment)
 			   (skip-chars-forward " \t")))
diff --git a/lisp/org.el b/lisp/org.el
index dd5d7ac..417329e 100755
--- a/lisp/org.el
+++ b/lisp/org.el
@@ -529,8 +529,8 @@ An entry can be toggled between COMMENT and normal with
     ("$1" "\\([^$]\\|^\\)\\(\\$[^ 	\r\n,;.$]\\$\\)\\(\\s.\\|\\s-\\|\\s(\\|\\s)\\|\\s\"\\|\000\\|$\\)" 2 nil)
     ("$"  "\\([^$]\\|^\\)\\(\\(\\$\\([^ 	\r\n,;.$][^$\n\r]*?\\(\n[^$\n\r]*?\\)\\{0,2\\}[^ 	\r\n,.$]\\)\\$\\)\\)\\(\\s.\\|\\s-\\|\\s(\\|\\s)\\|\\s\"\\|\000\\|$\\)" 2 nil)
     ("\\(" "\\\\([^\000]*?\\\\)" 0 nil)
-    ("\\[" "\\\\\\[[^\000]*?\\\\\\]" 0 nil)
-    ("$$" "\\$\\$[^\000]*?\\$\\$" 0 nil))
+    ("\\[" "^[ \t]*\\(\\\\\\[[^\000]*?\\\\\\]\\)[ \t]*$" 1 nil)
+    ("$$" "^[ \t]*\\(\\$\\$[^\000]*?\\$\\$\\)[ \t]*$" 1 nil))
   "Regular expressions for matching embedded LaTeX.")
 
 ;;;; Node Property
diff --git a/lisp/ox-latex.el b/lisp/ox-latex.el
index 5b4af9e..1d006ca 100644
--- a/lisp/ox-latex.el
+++ b/lisp/ox-latex.el
@@ -2651,7 +2651,7 @@ containing export options.  Modify DATA by side-effect and return it."
 	     (`latex-fragment
 	      (let ((value (org-element-property :value obj)))
 		(or (string-prefix-p "\\(" value)
-		    (string-match-p "\\`\\$[^$]" value))))
+		    (string-prefix-p "$" value))))
 	     ((or `subscript `superscript) t)))))
     (org-element-map data '(entity latex-fragment subscript superscript)
       (lambda (object)
diff --git a/testing/lisp/test-org-element.el b/testing/lisp/test-org-element.el
index 12cf2dd..158da08 100644
--- a/testing/lisp/test-org-element.el
+++ b/testing/lisp/test-org-element.el
@@ -1381,9 +1381,21 @@ Paragraph"
 
 (ert-deftest test-org-element/latex-environment-parser ()
   "Test `latex-environment' parser."
+  ;; Standard test.
+  (should
+   (eq 'latex-environment
+       (org-test-with-temp-text
+	   "\\begin{equation}\ne^{i\\pi}+1=0\n\\end{equation}"
+	 (org-element-type (org-element-at-point)))))
+  ;; Allow shorthand for "equation*" environment.
   (should
-   (org-test-with-temp-text "\\begin{equation}\ne^{i\\pi}+1=0\n\\end{equation}"
-     (org-element-map (org-element-parse-buffer) 'latex-environment 'identity)))
+   (eq 'latex-environment
+       (org-test-with-temp-text "\\[e^{i\\pi}+1=0\n\\]"
+	 (org-element-type (org-element-at-point)))))
+  (should
+   (eq 'latex-environment
+       (org-test-with-temp-text "$$e^{i\\pi}+1=0\n$$"
+	 (org-element-type (org-element-at-point)))))
   ;; Allow nested environments.
   (should
    (equal
@@ -1426,7 +1438,7 @@ e^{i\\pi}+1=0
   (should-not
    (eq 'latex-environment
        (org-test-with-temp-text "\\begin{env*}{arg}something\\end{env}"
-				(org-element-type (org-element-at-point)))))
+	 (org-element-type (org-element-at-point)))))
   ;; LaTeX environments must be on separate lines.
   (should-not
    (eq 'latex-environment
@@ -1476,13 +1488,14 @@ e^{i\\pi}+1=0
 	 (org-element-type (org-element-context)))))
   (should
    (eq 'latex-fragment
-       (org-test-with-temp-text "$$a$$"
+       (org-test-with-temp-text "\\(a\\)"
 	 (org-element-type (org-element-context)))))
-  (should
+  ;; The following constructs are elements, not objects.
+  (should-not
    (eq 'latex-fragment
-       (org-test-with-temp-text "\\(a\\)"
+       (org-test-with-temp-text "$$a$$"
 	 (org-element-type (org-element-context)))))
-  (should
+  (should-not
    (eq 'latex-fragment
        (org-test-with-temp-text "\\[a\\]"
 	 (org-element-type (org-element-context)))))
-- 
2.6.4

Re: [RFC] Change \[..\] and $$...$$ into elements

[Ethan Ligon]

Nicolas Goaziou <mail <at> nicolasgoaziou.fr> writes:

> 
> Hello,
> 
> The following patch changes status for \[..\] and $$...$$ from objects
> to elements.
> 

I think I've already indicated my enthusiastic support for this change, but 
let me do it again!  In /The TeXbook/ Donald Knuth writes that the markup for 
display math mode allows "your manuscript to mirror what you  are trying to 
accomplish."  I write all my manuscripts in  org-mode these days, in large 
part because the mark-up is lighter-weight than it is for LaTeX, but at 
present the preferred markup for display equations does *not* really allow 
one to make the manuscript mirror what one is trying to do.

Enthusiastic thanks and support for this patch!

-Ethan 

Re: [RFC] Change \[..\] and $$...$$ into elements

[Rasmus]

Ethan Ligon <ligon@berkeley.edu> writes:

> Nicolas Goaziou <mail <at> nicolasgoaziou.fr> writes:
>
>> 
>> Hello,
>> 
>> The following patch changes status for \[..\] and $$...$$ from objects
>> to elements.
>> 
>
> I think I've already indicated my enthusiastic support for this change, but 
> let me do it again!  In /The TeXbook/ Donald Knuth writes that the markup for 
> display math mode allows "your manuscript to mirror what you  are trying to 
> accomplish."  I write all my manuscripts in  org-mode these days, in large 
> part because the mark-up is lighter-weight than it is for LaTeX, but at 
> present the preferred markup for display equations does *not* really allow 
> one to make the manuscript mirror what one is trying to do.

What's wrong with equation (potentially coupled with mathtools for
handling numbering automatically)?  Is it worth breaking old documents
(irrespective of org-lint) for this inconvenience?

Rasmus

-- 
One thing that is clear: it's all down hill from here 

Re: [RFC] Change \[..\] and $$...$$ into elements

[Nicolas Goaziou]

Hello,

Rasmus <rasmus@gmx.us> writes:

> What's wrong with equation (potentially coupled with mathtools for
> handling numbering automatically)?  Is it worth breaking old documents
> (irrespective of org-lint) for this inconvenience?

Actually, the idea behind this patch is to mimic the expected LaTeX
behaviour,i.e., treat \[..\] as an equivalent
to \begin{equation*}...\end{equation*}.

This was requested at least once before, so I guess it may make sense to
follow the path of least surprise.

Breaking old-documents is obviously a concern, but is the following
construct really common?

  Text \[math formula\] ...


Regards,

-- 
Nicolas Goaziou

Re: [RFC] Change \[..\] and $$...$$ into elements

[Rasmus]

Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

> Hello,
>
> Rasmus <rasmus@gmx.us> writes:
>
>> What's wrong with equation (potentially coupled with mathtools for
>> handling numbering automatically)?  Is it worth breaking old documents
>> (irrespective of org-lint) for this inconvenience?
>
> Actually, the idea behind this patch is to mimic the expected LaTeX
> behaviour,i.e., treat \[..\] as an equivalent
> to \begin{equation*}...\end{equation*}.
>
> This was requested at least once before, so I guess it may make sense to
> follow the path of least surprise.

And I'm saying that people should just write:

     #+latex_header: \usepackage{mathtools} \mathtoolsset{showonlyrefs}
     aaa
     \begin{equation}
     bbb
     \end{equation}
     ccc

Org isn't LaTeX and there’s no difference in the output from \[⋯\]
and \begin{equation*}⋯\end{equation*} or the above.

Nonetheless, I don't feel strongly about the (non)issue so if this
behavior is better, somehow, go for it.

Rasmus

-- 
This is the kind of tedious nonsense up with which I will not put

Re: [RFC] Change \[..\] and $$...$$ into elements

[Ethan Ligon]

Rasmus <rasmus <at> gmx.us> writes:

> 
> Nicolas Goaziou <mail <at> nicolasgoaziou.fr> writes:
> 
> > Hello,
> >
> > Rasmus <rasmus <at> gmx.us> writes:
> >
> >> What's wrong with equation (potentially coupled with mathtools for
> >> handling numbering automatically)?  Is it worth breaking old documents
> >> (irrespective of org-lint) for this inconvenience?
> >
> > Actually, the idea behind this patch is to mimic the expected LaTeX
> > behaviour,i.e., treat \[..\] as an equivalent
> > to \begin{equation*}...\end{equation*}.
> >
> > This was requested at least once before, so I guess it may make sense 
to
> > follow the path of least surprise.
> 
> And I'm saying that people should just write:
> 
>      #+latex_header: \usepackage{mathtools} \mathtoolsset{showonlyrefs}
>      aaa
>      \begin{equation}
>      bbb
>      \end{equation}
>      ccc
> 
> Org isn't LaTeX and there’s no difference in the output from \[⋯\]
> and \begin{equation*}⋯\end{equation*} or the above.
> 
> Nonetheless, I don't feel strongly about the (non)issue so if this
> behavior is better, somehow, go for it.
> 
> Rasmus
> 

At the risk of sounding flip, I've gotta say that the question of whether
#+begin_example
aaa
\[
bbb
\]
ccc
#+end_example

is "somehow better" than

#+begin_example
#+latex_header: \usepackage{mathtools} \mathtoolsset{showonlyrefs}
aaa
\begin{equation}
bbb
\end{equation}
ccc
#+end_example

sort of answers itself.  It's not so much that we must make Org mimic 
LaTeX; it's that people actually use Org to compose documents with 
equations, and making this easier and simpler is a good thing.

-Ethan