[bug] `org-latex-line-break-safe' breaks the export of verse blocks to LaTeX

[bug] `org-latex-line-break-safe' breaks the export of verse blocks to LaTeX

[Juan Manuel Macías]

The recent addition of the `org-latex-line-break-safe' constant makes it
impossible to compile verse blocks. The reason is that now a \\ is
inserted between each stanza (aka each paragraph), instead of the
`\vspace{1em}' as before. That's wrong, as this compile error message
says:

---------
A \newline or \\ command appears between paragraphs, where it makes no
sense. If you're trying to ``leave a blank line'', use a \vspace
command.
---------

You can test yourself with this example:

#+begin_verse
lorem ipsum dolor
lorem ipsum dolor

lorem ipsum dolor
lorem ipsum dolor

lorem ipsum dolor
lorem ipsum dolor
#+end_verse

On the other hand, I have a few reservations about the usefulness of
org-latex-line-break-safe. To begin with, it is a particular solution
applied in a general way. This can have unexpected consequences, as has
happened in verse blocks. It's like putting your whole body in a
cast because of a broken finger.

I think if the reason for this addition is to prevent problems with
square brackets, in certain contexts, why not just apply a point
solution? For example, putting {[}, which is a common LaTeX way to
protect a square bracket. Use a macro, for example, or even
define a new entity, something like:

(setq org-entities-user
      '(("lbrack" "{[}" nil "[" "[" "[" "[")
	("rbrack" "{]}" nil "]" "]" "]" "]")))

And finally, I think that applying a general solution to this problem is
something that should be done(IMHO) in LaTeX and not in Org, for
example, with some new package that would protect certain signs or
through ad hoc LaTeX/Lua code.

Best regards,

Juan Manuel 

-- 
--
------------------------------------------------------
Juan Manuel Macías 

https://juanmanuelmacias.com

https://lunotipia.juanmanuelmacias.com

https://gnutas.juanmanuelmacias.com

Re: [bug] `org-latex-line-break-safe' breaks the export of verse blocks to LaTeX

[Ihor Radchenko]

Juan Manuel Macías <maciaschain@posteo.net> writes:

> The recent addition of the `org-latex-line-break-safe' constant makes it
> impossible to compile verse blocks. The reason is that now a \\ is
> inserted between each stanza (aka each paragraph), instead of the
> `\vspace{1em}' as before. That's wrong, as this compile error message
> says:

Thanks for reporting!
Fixed on main.
https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=5fa66c7ffc4efbea11357fe0d05bc570855cfa91

This was an omission when replacing "\\" instances with safer version in
ox-latex.el.

> On the other hand, I have a few reservations about the usefulness of
> org-latex-line-break-safe. To begin with, it is a particular solution
> applied in a general way. This can have unexpected consequences, as has
> happened in verse blocks. It's like putting your whole body in a
> cast because of a broken finger.

The idea is to use "\\\someescapeLaTeXcommand" rather than "\\". The
latter can take arguments unexpectedly. See
https://orgmode.org/list/8735c0kpb9.fsf@localhost

> I think if the reason for this addition is to prevent problems with
> square brackets, in certain contexts, why not just apply a point
> solution? For example, putting {[}, which is a common LaTeX way to
> protect a square bracket.

Because people export tables not only to LaTeX, but also to other
formats. {[} makes no sense then.

> Use a macro, for example, or even
> define a new entity, something like:
>
> (setq org-entities-user
>       '(("lbrack" "{[}" nil "[" "[" "[" "[")
> 	("rbrack" "{]}" nil "]" "]" "]" "]")))

It will be unclear when to use such entity. Note that it is not obvious
at all that LaTeX export specifically will stumble on

| [1, 2] |
| [3, 4] |

without knowing LaTeX internals and checking the exported LaTeX
intermediate source closely.

We already ask users to deal with Org parser oddities. If we also demand
knowing LaTeX oddities, it will be too much. Better modify parser to
export valid Org syntax into valid LaTeX.

> And finally, I think that applying a general solution to this problem is
> something that should be done(IMHO) in LaTeX and not in Org, for
> example, with some new package that would protect certain signs or
> through ad hoc LaTeX/Lua code.

Either way, we will need to use the new package in ox-latex.el. And our
current approach is somewhat better because users can still use direct
LaTeX @@latex:\\@@ if they want to get "\\" exactly.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [bug] `org-latex-line-break-safe' breaks the export of verse blocks to LaTeX

[Juan Manuel Macías]

Ihor Radchenko writes:

>> And finally, I think that applying a general solution to this problem is
>> something that should be done(IMHO) in LaTeX and not in Org, for
>> example, with some new package that would protect certain signs or
>> through ad hoc LaTeX/Lua code.
>
> Either way, we will need to use the new package in ox-latex.el. And our
> current approach is somewhat better because users can still use direct
> LaTeX @@latex:\\@@ if they want to get "\\" exactly.

I'm afraid we differ completely on what Org's priorities are at this
point. The way I see it, Org shouldn't try to solve LaTeX problems,
because Org isn't proficient in LaTeX. This is a known issue in LaTeX,
where a leading bracket can be confused with the optional argument of a
preceding macro (such as \\ or \linebreak). And it doesn't just happen
in this context. It can also happen with environments that have optional
arguments, although the case of \\ is the most common and the case of
environments depends on how the environment is defined. For this
problem, LaTeX has been recommending various *specific* solutions
(because the probability of it appearing is very low, luckily), such as
enclosing the brackets in two {} or putting a \relax before .

In this case Org intends to solve the LaTeX problem, and it does it, in
my opinion, poorly, by generalizing a particular solution for what is
even an extremely rare problem. This is the typical case that can be
easily solved by the user with a macro or an export filter, as I have
been doing for a long time, and at no time have I needed anything else.

Org must worry about exporting well and offering the necessary tools for
that to happen, as it has been doing until now. And the duty of the
users is to know reasonably the language to which they export, to the
extent of what they want to obtain when they export. Org should not be a
substitute for LaTeX: whoever wants to get special things in LaTeX, must
know LaTeX. It's that easy. If Org is going to start trying to solve at
its own risk all the *specific* problems that can appear in the typical
compilation of a LaTeX document, then I prefer to stop using Org and use
pure LaTeX from the beginning.

And as for the ox-latex.el file, in its current state it is completely
unusable for me. At least the user could be offered the possibility of
choosing.

Re: [bug] `org-latex-line-break-safe' breaks the export of verse blocks to LaTeX

[Max Nikulin]

On 16/10/2022 04:35, Juan Manuel Macías wrote:
>
> To begin with, it is a particular solution
> applied in a general way. This can have unexpected consequences, as has
> happened in verse blocks.

In my opinion Org has to apply a general solution merely because a table 
may be result of evaluation of an org-babel code block. I may be wrong, 
but I suspect that some users do not care what is the intermediate 
format when they need a PDF file.

> And finally, I think that applying a general solution to this problem is
> something that should be done(IMHO) in LaTeX and not in Org,

Org has a bit more information concerning context of square brackets. 
They may be just text of a table cell or contents of an export snippet. 
In a LaTeX file they are just square brackets and there is no way to 
distinguish them.

I believe, it is a design flaw in LaTeX that the \\ command does not 
have a counterpart with no optional * and [] arguments. For humans \\ is 
enough, but it is fragile when LaTeX is used as an intermediate export 
format. A part of the problem is that all 3rd party packages should be 
adapted for the robust \\ sibling. Likely it may be solved by additional 
pass in the exporter code but it will significantly increase its 
complexity. I have no idea what is appropriate place to discuss such 
issue with LaTeX developers.

The following is irrelevant to the recent changes. I have tried

---- >8 ----
text
#+begin_verse

a b
c d

e f
g h
#+end_verse
---- 8< ----

With the fix Ihor committed today I have got

---- >8 ----
text
\begin{verse}
\vspace*{1em}
a b\\\empty
c d\\\empty
\vspace*{1em}
e f\\\empty
g h\\\empty
\end{verse}
---- 8< ----

My expectation was
---- >8 ----
\begin{verse}
a b\\\empty
c d

e f\\\empty
g h
\end{verse}
---- 8< ----

I am surprised that \vspace is added instead of empty line between 
stanzas. Is there a reason to override LaTeX defaults? If such reason 
exists would not it better to add

     \parskip=1em plus 0.5em minus 0.25em\relax

before first verse line? Is hard coded rigid vertical space acceptable 
when high quality output is desired? \vspace before first line looks 
like a bug. Frankly speaking, nested calls of `replace-regexp-in-string' 
makes the code hard to read.

P.S. I have not found exact citation, but I noticed a mention: Lamport 
expected that verse environment would get negative feedback from poets.

Verse block and separations (was: [bug] `org-latex-line-break-safe' breaks the export of verse blocks to LaTeX)

[Juan Manuel Macías]

Max Nikulin writes:

> The following is irrelevant to the recent changes. I have tried
>
> ---- >8 ----
> text
>
> #+begin_verse
>
> a b
> c d
>
> e f
> g h
> #+end_verse
>
> ---- 8< ----
>
> With the fix Ihor committed today I have got
>
> ---- >8 ----
> text
> \begin{verse}
> \vspace*{1em}
> a b\\\empty
> c d\\\empty
> \vspace*{1em}
> e f\\\empty
> g h\\\empty
> \end{verse}
> ---- 8< ----
>
> My expectation was
> ---- >8 ----
> \begin{verse}
> a b\\\empty
> c d
>
> e f\\\empty
> g h
> \end{verse}
> ---- 8< ----
>
> I am surprised that \vspace is added instead of empty line between 
> stanzas. Is there a reason to override LaTeX defaults? If such reason 
> exists would not it better to add
>
>      \parskip=1em plus 0.5em minus 0.25em\relax
>
> before first verse line? Is hard coded rigid vertical space acceptable 
> when high quality output is desired? \vspace before first line looks 
> like a bug. Frankly speaking, nested calls of `replace-regexp-in-string' 
> makes the code hard to read.
>
> P.S. I have not found exact citation, but I noticed a mention: Lamport 
> expected that verse environment would get negative feedback from poets.

First of all, I'm afraid that in my previous post I mixed up two
different issues: the verse block bug and my personal opinions about the
new added constant. I should have separated these topics, sorry.

I answer here what you comment about the verse blocks and later I will
open a different thread for the other issue.

Yes, you're right: that \vspace is the normal behavior of the block when
exporting to LaTeX, and it is certainly not quite correct. In LaTeX,
both the out-of-the-box verse environment and the one provided by the
'verse' package (which, typographically speaking, is the correct way to
represent verses in LaTeX) do not add a \vspace between stanzas. In the
LaTeX input the separation between stanzas is expressed by an (at least)
empty line. In fact, that space can be controlled by the verse package
with the stanzaskip \length. I remember that I commented on this anomaly
here on the list, a long time ago, but I can't find the original
message...

In my init I have redefined the verse block like this, so that there is
a white line between stanzas, not \vspace (I have also added some things
for the verse package, so that the numbering of verses is not broken).
So your example would produce a white line under \begin{verse}, not the
current \vspace, which would be irrelevant to LaTeX. But this is only a
hack:

  (defun org-latex-verse-block (verse-block contents info)
    "Transcode a VERSE-BLOCK element from Org to LaTeX.
      CONTENTS is verse block contents.  INFO is a plist holding
      contextual information."
    (let*
	((lin (org-export-read-attribute :attr_latex verse-block :lines))
	 (latcode (org-export-read-attribute :attr_latex verse-block :latexcode))
	 (cent (org-export-read-attribute :attr_latex verse-block :center))
	 (attr (concat
		(if cent "[\\versewidth]" "")
		(if lin (format "\n\\poemlines{%s}" lin) "")
		(if latcode (format "\n%s" latcode) "")))
	 (versewidth (org-export-read-attribute :attr_latex verse-block :versewidth))
	 (vwidth (if versewidth (format "\\settowidth{\\versewidth}{%s}\n" versewidth) ""))
	 (linreset (if lin "\n\\poemlines{0}" "")))
      (org-latex--wrap-label
       verse-block
       ;; In a verse environment, add a line break to each newline
       ;; character and change each white space at beginning of a line
       ;; into a space of 1 em.  Also change each blank line with
       ;; a vertical space of 1 em.
       (format "%s\\begin{verse}%s\n%s\\end{verse}%s"
	       vwidth
	       attr
	       (replace-regexp-in-string
		"^[ \t]+" (lambda (m) (format "\\hspace*{%dem}" (length m)))
		(replace-regexp-in-string
		 "\\(\\\\\\\\\n\\)+\\([ \t]*\\\\\\\\\\)+" "\n"
		 (replace-regexp-in-string
		  "\\([ \t]*\\\\\\\\\\)?[ \t]*\n" "\\\\\n"
		  (setq contents
			(if lin
			    (replace-regexp-in-string "\\(\n\\)+\\([ \t]*\n\\)+" "\\\\\\\\!\n\n"
						      contents)
			  contents)) nil t) nil t) nil t) linreset)
       info)))

Re: Verse block and separations (was: [bug] `org-latex-line-break-safe' breaks the export of verse blocks to LaTeX)

[Ihor Radchenko]

Juan Manuel Macías <maciaschain@posteo.net> writes:

> Yes, you're right: that \vspace is the normal behavior of the block when
> exporting to LaTeX, and it is certainly not quite correct. In LaTeX,
> both the out-of-the-box verse environment and the one provided by the
> 'verse' package (which, typographically speaking, is the correct way to
> represent verses in LaTeX) do not add a \vspace between stanzas. In the
> LaTeX input the separation between stanzas is expressed by an (at least)
> empty line. In fact, that space can be controlled by the verse package
> with the stanzaskip \length. I remember that I commented on this anomaly
> here on the list, a long time ago, but I can't find the original
> message...

I checked git log and the current approach for verse export dates back
to initial versions of ox-latex. I am pretty sure that it should be
improved.

> In my init I have redefined the verse block like this, so that there is
> a white line between stanzas, not \vspace (I have also added some things
> for the verse package, so that the numbering of verses is not broken).
> So your example would produce a white line under \begin{verse}, not the
> current \vspace, which would be irrelevant to LaTeX. But this is only a
> hack:

Could you provide a patch?

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Verse block and separations

[Juan Manuel Macías]

Ihor Radchenko writes:

>> In my init I have redefined the verse block like this, so that there is
>> a white line between stanzas, not \vspace (I have also added some things
>> for the verse package, so that the numbering of verses is not broken).
>> So your example would produce a white line under \begin{verse}, not the
>> current \vspace, which would be irrelevant to LaTeX. But this is only a
>> hack:
>
> Could you provide a patch?

Sorry, I missed this message...

Sure, I think by the end of this week I'll be able to prepare a patch. I
put it on the agenda.

Re: Verse block and separations (was: [bug] `org-latex-line-break-safe' breaks the export of verse blocks to LaTeX)

[Max Nikulin]

On 16/10/2022 23:33, Juan Manuel Macías wrote:
> Max Nikulin writes:
>>
>> I am surprised that \vspace is added instead of empty line between
>> stanzas.
> 
> First of all, I'm afraid that in my previous post I mixed up two
> different issues: the verse block bug and my personal opinions about the
> new added constant. I should have separated these topics, sorry.

It was me who raised issues with verse exporter in discussion of newline 
commands. I was impressed by hard to read elisp code and its result.

> I answer here what you comment about the verse blocks and later I will
> open a different thread for the other issue.

I am trying to read with enough attention this discussion, for a while I 
have a couple of remarks specific to poetry.

> with the stanzaskip \length. I remember that I commented on this anomaly
> here on the list, a long time ago, but I can't find the original
> message...

https://list.orgmode.org/87k0tjasty.fsf@posteo.net/
Juan Manuel Macías. [Small suggestion] on exporting verse blocks to 
LaTeX and vertical spaces. Tue, 15 Dec 2020 18:21:13 +0100

https://list.orgmode.org/?q=verse+vspace

> In my init I have redefined the verse block like this, so that there is
> a white line between stanzas, not \vspace

Do you still have real problems with current main HEAD state after 
adjusting patterns in your code?

>         (format "%s\\begin{verse}%s\n%s\\end{verse}%s"
> 	       vwidth

I have not verified it, but since I am not aware of a group wrapping 
this fragment, I suspect that :versewidth may have global effect, not 
limited to the current environment. Org variant of the function is 
rather close at this point.

Re: Verse block and separations (was: [bug] `org-latex-line-break-safe' breaks the export of verse blocks to LaTeX)

[Max Nikulin]

On 16/10/2022 23:33, Juan Manuel Macías wrote:
> 	       (replace-regexp-in-string
> 		"^[ \t]+" (lambda (m) (format "\\hspace*{%dem}" (length m)))
> 		(replace-regexp-in-string
> 		 "\\(\\\\\\\\\n\\)+\\([ \t]*\\\\\\\\\\)+" "\n"
> 		 (replace-regexp-in-string
> 		  "\\([ \t]*\\\\\\\\\\)?[ \t]*\n" "\\\\\n"
> 		  (setq contents
> 			(if lin
> 			    (replace-regexp-in-string "\\(\n\\)+\\([ \t]*\n\\)+" "\\\\\\\\!\n\n"
> 						      contents)
> 			  contents)) nil t) nil t) nil t) linreset)

I had a hope, it is possible to do it in a single pass of 
`replace-regexp-in-string', but unfortunately the function does not 
allow to make conditional substitution based on (rx (optional (group 
string-start))) (a bug?).

I still prefer to avoid replacement of latex newlines back to empty 
string. Though I am really happy with the following code, I expected a 
more concise snippet. Unit tests may found bugs in it.

(let ((contents "\n\n 0  \n\n\na b\nc d e  \n\n\nf g  \n   h i\n\n"))
   ;; Strip leading newlines.
   (setq contents
	(replace-regexp-in-string
	 (rx string-start (1+ (0+ blank) ?\n)) ""
	 contents 'fixed-case 'literal))
   ;; Add explicit line breaks and strip trailing spaces.
   (setq contents
	(replace-regexp-in-string
	 (rx (0+ blank) ?\n
	     (optional (group (1+ (0+ blank) ?\n)))
	     (optional (group (0+ blank) (not (any blank ?\n)))))
	 (lambda (m)
	   (let ((par (match-string 1 m))
		 (next (match-string 2 m)))
	     (if next
		 (concat (if par "\n\n" "\\\\\n")
			 next)
	       "")))
	 contents 'fixed-case 'literal))
   ;; Indented lines.
   (replace-regexp-in-string
    (rx line-start (1+ blank))
    (lambda (m) (format "\\hspace*{%dem}" (length m)))
    contents 'fixed-case 'literal))

Feel free to use it for inspiration during your work on a patch.

Re: Verse block and separations

[Juan Manuel Macías]

Max Nikulin writes:

> I still prefer to avoid replacement of latex newlines back to empty
> string. Though I am really happy with the following code, I expected a
> more concise snippet. Unit tests may found bugs in it.
>
> (let ((contents "\n\n 0  \n\n\na b\nc d e  \n\n\nf g  \n   h i\n\n"))
>   ;; Strip leading newlines.
>   (setq contents
> 	(replace-regexp-in-string
> 	 (rx string-start (1+ (0+ blank) ?\n)) ""
> 	 contents 'fixed-case 'literal))
>   ;; Add explicit line breaks and strip trailing spaces.
>   (setq contents
> 	(replace-regexp-in-string
> 	 (rx (0+ blank) ?\n
> 	     (optional (group (1+ (0+ blank) ?\n)))
> 	     (optional (group (0+ blank) (not (any blank ?\n)))))
> 	 (lambda (m)
> 	   (let ((par (match-string 1 m))
> 		 (next (match-string 2 m)))
> 	     (if next
> 		 (concat (if par "\n\n" "\\\\\n")
> 			 next)
> 	       "")))
> 	 contents 'fixed-case 'literal))
>   ;; Indented lines.
>   (replace-regexp-in-string
>    (rx line-start (1+ blank))
>    (lambda (m) (format "\\hspace*{%dem}" (length m)))
>    contents 'fixed-case 'literal))
>
> Feel free to use it for inspiration during your work on a patch.

OK, thanks a lot. (BTW, I have yet to reply to some interesting things
you mention in this thread about the verse environment).

Line breaks and brackets in LaTeX export (was: [bug] `org-latex-line-break-safe' breaks the export of verse blocks to LaTeX)

[Juan Manuel Macías]

Max Nikulin writes:

> In my opinion Org has to apply a general solution merely because a table 
> may be result of evaluation of an org-babel code block. I may be wrong, 
> but I suspect that some users do not care what is the intermediate 
> format when they need a PDF file.
>
>> And finally, I think that applying a general solution to this problem is
>> something that should be done(IMHO) in LaTeX and not in Org,
>
> Org has a bit more information concerning context of square brackets. 
> They may be just text of a table cell or contents of an export snippet. 
> In a LaTeX file they are just square brackets and there is no way to 
> distinguish them.
>
> I believe, it is a design flaw in LaTeX that the \\ command does not 
> have a counterpart with no optional * and [] arguments. For humans \\ is 
> enough, but it is fragile when LaTeX is used as an intermediate export 
> format. A part of the problem is that all 3rd party packages should be 
> adapted for the robust \\ sibling. Likely it may be solved by additional 
> pass in the exporter code but it will significantly increase its 
> complexity. I have no idea what is appropriate place to discuss such 
> issue with LaTeX developers.

The solution of adding \relax or \empty is a good one. The problem is
when it is applied massively and indiscriminately, bearing in mind that
it is to prevent a problem that is, frankly, very rare. \empty is
unlikely to have any side effects, but still, I don't like Org making
these decisions for me and adding LaTeX code that I didn't ask for, in
form or quantity. Isn't it possible to be more selective, like Pandoc
does (I think), and add what needs to be added only when the problem is
present? I don't know if the Pandoc thing has already been mentioned in
past discussions, because these days I haven't been paying attention to
the list.

In any case, I would prefer: a) a selective solution; and if a) is not
possible, at least b) a defcustom.

This is an example I came up with of how it could be fixed selectively
in LaTeX (big, big caveat: it's just a proof of concept and likely to
produce errors in other contexts. I think if a LaTeX package to fix this
isn't already out, it is either because the problem is very rare and the
solution for particular cases is known, or because there are drawbacks
inherent to LaTeX that I can't figure out right now):

\makeatletter

\def\my@protectlbrack{
  \@ifnextchar{[}{\relax}{}
  \@ifnextchar{*}{\relax}{}
}

\let\old@break\\
\def\\{%
  \old@break\my@protectlbrack}

%% for tables

\let\old@tabularcr\@tabularcr
\def\@tabularcr{%
  \old@tabularcr\my@protectlbrack}

% for use the old commands

\newcommand\oldbreak{\old@break}
\newcommand\oldbreakt{\old@tabularcr}

\makeatother

\begin{document}

lorem\\
[ipsum]

lorem\\
*ipsum

\begin{tabular}{cccc}
a & a & a & a \\
[a] & a & a & a \\
*a & a & a & a \\
\end{tabular}


\end{document}

Re: Line breaks and brackets in LaTeX export (was: [bug] `org-latex-line-break-safe' breaks the export of verse blocks to LaTeX)

[Ihor Radchenko]

Juan Manuel Macías <maciaschain@posteo.net> writes:

> The solution of adding \relax or \empty is a good one. The problem is
> when it is applied massively and indiscriminately, bearing in mind that
> it is to prevent a problem that is, frankly, very rare. \empty is
> unlikely to have any side effects, but still, I don't like Org making
> these decisions for me and adding LaTeX code that I didn't ask for, in
> form or quantity. Isn't it possible to be more selective, like Pandoc
> does (I think), and add what needs to be added only when the problem is
> present? I don't know if the Pandoc thing has already been mentioned in
> past discussions, because these days I haven't been paying attention to
> the list.
>
> In any case, I would prefer: a) a selective solution; and if a) is not
> possible, at least b) a defcustom.

I tried to add the safe version of page break everywhere where there is
a risk of a page break followed by square brackets.

The key point: A valid Org file must be exported to a valid LaTeX file.

If you know a selective solution, patches are welcome. (I am also CCing
Daniel).

I see no issue with defcustom in general, but can you explain what
practical problems it is going to solve? I am not talking about
aesthetics of the exported LaTeX.

> This is an example I came up with of how it could be fixed selectively
> in LaTeX (big, big caveat: it's just a proof of concept and likely to
> produce errors in other contexts. I think if a LaTeX package to fix this
> isn't already out, it is either because the problem is very rare and the
> solution for particular cases is known, or because there are drawbacks
> inherent to LaTeX that I can't figure out right now):
>
> \makeatletter
>
> \def\my@protectlbrack{
>   \@ifnextchar{[}{\relax}{}
>   \@ifnextchar{*}{\relax}{}
> }
>
> \let\old@break\\
> \def\\{%
>   \old@break\my@protectlbrack}
>
> %% for tables
>
> \let\old@tabularcr\@tabularcr
> \def\@tabularcr{%
>   \old@tabularcr\my@protectlbrack}
>
> % for use the old commands
>
> \newcommand\oldbreak{\old@break}
> \newcommand\oldbreakt{\old@tabularcr}
>
> \makeatother
>
> \begin{document}
>
> lorem\\
> [ipsum]
>
> lorem\\
> *ipsum
>
> \begin{tabular}{cccc}
> a & a & a & a \\
> [a] & a & a & a \\
> *a & a & a & a \\
> \end{tabular}
>
>
> \end{document}

Won't it make it impossible to use
@@latex:\\@@
@@latex:[1em]@@

deliberately?

Note that we may also replace \\\empty with a single custom command. It
will look a bit cleaner. Not sure if it is any different from your
approach.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Line breaks and brackets in LaTeX export

[Juan Manuel Macías]

Ihor Radchenko writes:

> I see no issue with defcustom in general, but can you explain what
> practical problems it is going to solve? I am not talking about
> aesthetics of the exported LaTeX.

In my opinion it is not so much a question of aesthetics as of (let's
say) a certain conformity with what a LaTeX document should be. I think,
for example, in the case of a user who must submit his/her work in LaTeX
format to a publication, following the rules of that publication. It is
one thing for a LaTeX document to compile without error and quite
another for a LaTeX document to be or not to be formed according to good
practice. Would there be a problem sharing LaTeX documents with
unnecessary commands like \empty after all occurrences of \\?

Anyway. As for the compilation, it is highly unlikely that \empty will
cause any unexpected error. But LaTeX and its over 6000 packages is
unpredictable. It also seemed unlikely that \relax would cause any
problems, and catching up on the last discussion, it had to be replaced
by \empty because it returned an error just before \hline. \relax is one
of the recommended solutions from LaTeX, because it tells LaTeX that the
previous macro has finished expanding, but it is recommended keeping in
mind that the user will apply it only when needed, not everywhere. And
before \hline it doesn't make sense because there will never be an
'\\[...]' error. So, in the current situation, we can ask ourselves: is
\empty everywhere safe? Everything points to yes. Can we be 100% sure?
...?

The only thing I can think of, for a non-selective solution like the
current one, is the following: if \\ has an optional argument that must
be a length, then let's give it, but with a value of zero: \\[0pt], which
is equivalent to putting the value by default (zero) explicitly.

>> This is an example I came up with of how it could be fixed selectively
>> in LaTeX (big, big caveat: it's just a proof of concept and likely to
>> produce errors in other contexts. I think if a LaTeX package to fix this
>> isn't already out, it is either because the problem is very rare and the
>> solution for particular cases is known, or because there are drawbacks
>> inherent to LaTeX that I can't figure out right now):
>>
>> \makeatletter
>>
>> \def\my@protectlbrack{
>>   \@ifnextchar{[}{\relax}{}
>>   \@ifnextchar{*}{\relax}{}
>> }
>>
>> \let\old@break\\
>> \def\\{%
>>   \old@break\my@protectlbrack}
>>
>> %% for tables
>>
>> \let\old@tabularcr\@tabularcr
>> \def\@tabularcr{%
>>   \old@tabularcr\my@protectlbrack}
>>
>> % for use the old commands
>>
>> \newcommand\oldbreak{\old@break}
>> \newcommand\oldbreakt{\old@tabularcr}
>>
>> \makeatother
>>
>> \begin{document}
>>
>> lorem\\
>> [ipsum]
>>
>> lorem\\
>> *ipsum
>>
>> \begin{tabular}{cccc}
>> a & a & a & a \\
>> [a] & a & a & a \\
>> *a & a & a & a \\
>> \end{tabular}
>>
>>
>> \end{document}
>
> Won't it make it impossible to use
> @@latex:\\@@
> @@latex:[1em]@@
>
> deliberately?

What I've done in this code is redefine \\ so that if the next character
is a [ or a * it doesn't do anything. To use the macro with the old
behavior, you would have to use the new macros \oldbreak and \oldbreakt
(this one for tables):

@@latex:\oldbreak@@
@@latex:[1em]@@

Anyway, like I said, it's a proof of concept. In the tests I've done it
works very well, but on a large scale I can't be sure of the results.

Best regards,

Juan Manuel 

Re: Line breaks and brackets in LaTeX export

[Ihor Radchenko]

Juan Manuel Macías <maciaschain@posteo.net> writes:

> Ihor Radchenko writes:
>
>> I see no issue with defcustom in general, but can you explain what
>> practical problems it is going to solve? I am not talking about
>> aesthetics of the exported LaTeX.
>
> In my opinion it is not so much a question of aesthetics as of (let's
> say) a certain conformity with what a LaTeX document should be. I think,
> for example, in the case of a user who must submit his/her work in LaTeX
> format to a publication, following the rules of that publication. It is
> one thing for a LaTeX document to compile without error and quite
> another for a LaTeX document to be or not to be formed according to good
> practice. Would there be a problem sharing LaTeX documents with
> unnecessary commands like \empty after all occurrences of \\?

I haven't seen any publication rule that prevents using valid LaTeX
commands like this. Do you have concrete examples? If not, one could
argue that any auto-generated output could break some imaginary rule.

I am also wondering how LaTeX documents generated from LyX or TeXmacs
look like. Are they not using some obvious machine-generated constructs?

> Anyway. As for the compilation, it is highly unlikely that \empty will
> cause any unexpected error. But LaTeX and its over 6000 packages is
> unpredictable. It also seemed unlikely that \relax would cause any
> problems, and catching up on the last discussion, it had to be replaced
> by \empty because it returned an error just before \hline. \relax is one
> of the recommended solutions from LaTeX, because it tells LaTeX that the
> previous macro has finished expanding, but it is recommended keeping in
> mind that the user will apply it only when needed, not everywhere. And
> before \hline it doesn't make sense because there will never be an
> '\\[...]' error. So, in the current situation, we can ask ourselves: is
> \empty everywhere safe? Everything points to yes. Can we be 100% sure?
> ...?

My answer is: "\\" is 100% not universally safe. Simply because we have
that bug report with [ ... ] items in tables.
So, anything with no concrete counter-example is better as long as we
are reasonably sure that we are not breaking LaTeX conventions.

> The only thing I can think of, for a non-selective solution like the
> current one, is the following: if \\ has an optional argument that must
> be a length, then let's give it, but with a value of zero: \\[0pt], which
> is equivalent to putting the value by default (zero) explicitly.

This has been proposed and then rejected by Max in
https://list.orgmode.org/orgmode/ti5tdb$rd2$1@ciao.gmane.io/ and he
concluded that some side effects are present when using \\[0pt]:

Max>       \\[0pt]
Max> 
Max> causes insertion of some code for negative vertical skip (of zero height 
Max> this case). It should not be really harmful, but I would avoid this hack.

> What I've done in this code is redefine \\ so that if the next character
> is a [ or a * it doesn't do anything. To use the macro with the old
> behavior, you would have to use the new macros \oldbreak and \oldbreakt
> (this one for tables):
>
> @@latex:\oldbreak@@
> @@latex:[1em]@@

It means that LaTeX packages that redefine \\ may be affected. I'd
prefer to avoid such side effects.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Line breaks and brackets in LaTeX export

[Juan Manuel Macías]

Ihor Radchenko writes:

> I haven't seen any publication rule that prevents using valid LaTeX
> commands like this. Do you have concrete examples? If not, one could
> argue that any auto-generated output could break some imaginary rule.

No, I don't have any concrete example. But it is one thing to use a
valid LaTeX command and another to use it unnecessarily. It's like
putting \vspace{0pt} between each paragraph.

> I am also wondering how LaTeX documents generated from LyX or TeXmacs
> look like. Are they not using some obvious machine-generated constructs?

I don't know, because I haven't seen them. But I'd bet (at the risk of
losing the bet) that none of those machine-generated constructs produce
anything as unnecessary as Org's present solution. The situation now
becomes the following:

Pandoc: selective solution. In specific cases it returns {[}...{]}

Org: non-selective solution = ugly LaTeX code.

>> Anyway. As for the compilation, it is highly unlikely that \empty will
>> cause any unexpected error. But LaTeX and its over 6000 packages is
>> unpredictable. It also seemed unlikely that \relax would cause any
>> problems, and catching up on the last discussion, it had to be replaced
>> by \empty because it returned an error just before \hline. \relax is one
>> of the recommended solutions from LaTeX, because it tells LaTeX that the
>> previous macro has finished expanding, but it is recommended keeping in
>> mind that the user will apply it only when needed, not everywhere. And
>> before \hline it doesn't make sense because there will never be an
>> '\\[...]' error. So, in the current situation, we can ask ourselves: is
>> \empty everywhere safe? Everything points to yes. Can we be 100% sure?
>> ...?
>
> My answer is: "\\" is 100% not universally safe. Simply because we have
> that bug report with [ ... ] items in tables.
> So, anything with no concrete counter-example is better as long as we
> are reasonably sure that we are not breaking LaTeX conventions.

As I said, this is a known LaTeX problem that occurs in a rare case, for
which there is a known solution (or several), which should be applied in
the specific case. And Org already provides the necessary tools to apply
it.

>> The only thing I can think of, for a non-selective solution like the
>> current one, is the following: if \\ has an optional argument that must
>> be a length, then let's give it, but with a value of zero: \\[0pt], which
>> is equivalent to putting the value by default (zero) explicitly.
>
> This has been proposed and then rejected by Max in
> https://list.orgmode.org/orgmode/ti5tdb$rd2$1@ciao.gmane.io/ and he
> concluded that some side effects are present when using \\[0pt]:
>
> Max>       \\[0pt]
> Max> 
> Max> causes insertion of some code for negative vertical skip (of zero height 
> Max> this case). It should not be really harmful, but I would avoid this

I would like to see some concrete example where this solution was
problematic. \\[0pt] is exactly the same as \\ (as for the effects).
Redundant but valid.

Re: Line breaks and brackets in LaTeX export

[Juan Manuel Macías]

Juan Manuel Macías writes:

> So, in the current situation, we can ask ourselves: is
> \empty everywhere safe? Everything points to yes. Can we be 100% sure?
> ...?

I think I've found a case where \empty 'everywhere' can produce unexpected
results. I don't know if I'm missing something, because I've never used
this package, but taking a couple of random examples from the tabularray
documentation, you can see it, compiling the following snippet (by the
way, putting \\[0pt] instead of \\\empty the result is correct). Here in
both cases the problem is in the last \empty:

\documentclass{article}
\usepackage{tabularray}

\begin{document}

\section{Normal}

\begin{tblr}{lccr}
\hline
Alpha & Beta & Gamma & Delta \\
\hline
Epsilon & Zeta & Eta & Theta \\
\hline
Iota & Kappa & Lambda & Mu\\
\hline
\end{tblr}

\section{With `empty'}

\begin{tblr}{lccr}
\hline
Alpha & Beta & Gamma & Delta \\\empty
\hline
Epsilon & Zeta & Eta & Theta \\\empty
\hline
Iota & Kappa & Lambda & Mu\\\empty
\hline
\end{tblr}

\section{Normal}

\begin{tblr}[m]{hlines}
Alpha & Beta & Gamma \\
Epsilon & Zeta & Eta\\
Iota & Kappa & Lambda\\
\end{tblr}

\section{With `empty'}

\begin{tblr}[m]{hlines}
Alpha & Beta & Gamma \\\empty
Epsilon & Zeta & Eta\\\empty
Iota & Kappa & Lambda \\\empty
\end{tblr}

Re: Line breaks and brackets in LaTeX export

[Max Nikulin]

On 17/10/2022 22:01, Juan Manuel Macías wrote:
> 
> \documentclass{article}
> \usepackage{tabularray}

LaTeX I have installed is too old for this package. It is marked as 
"Experimental LaTeX3", so I am unsure, maybe you have found a bug in 
this package.
https://ctan.org/pkg/tabularray

You believe that an issue with brackets is extremely rare. It may be 
true for humanitarian texts. For some users it may be a constant source 
of pain e.g. in the case of interval notation as [a,b]. I have already 
mentioned tables generated by code blocks, not typed directly. I can not 
say that I often need to export my notes, but I was afraid that I will 
be bitten by this bug because I may try to put dates close to left margin:

- Something\\
   [2022-10-17 Mon]

By default dates wrapped into \textit, but it may be customized to just 
"%s".

Selectively adding some workaround require complete reimplementation of 
exporters. I have some curiosity concerning pandoc approach, but I am 
unsure if I will reserve some time to read its code.

An idea how to avoid complete redesign: at first add something like
    %__ORG_PROTECT_NEWLINE__
after each \\, later at an optimizing pass remove the comment if next 
line does not start from a star or a square bracket, otherwise use some 
workaround, e.g. "{[}". \relax may be suitable as well (in the beginning 
of rows, not after \\).

I do not like approach with a custom command. It is effectively the same 
as adding \empty but \\ may be redefined by some packages, so I strongly 
prefer to keep \\ in exported markup. Redefining of \\ is a way to new 
issues. I do not feel firm ground with a command that will expand to \\. 
I am afraid that \\ may still consume following "[" unless \empty or its 
equivalent is added after it. I am completely unsure concerning 
tabularray parser that does not rely on TeX primitives.

I found \empty when I was looking for an approach with minimal overhead. 
I expect that e.g. \\[0pt] may have higher performance penalty since it 
is expanded to several commands. When the idea with "\\\relax" failed I 
was choosing between "\\{}" and "\\\empty". I decided that the latter 
minimizes risk to add spurious space.

Some problems:

I do not see a way to add some LaTeX code between table lines.

Semi-verbatim environment may be a source of new bugs. They may be 
rather selective in respect to what they consider a command and what is 
passed as raw text. tabularray perhaps is similar in this sense.

Re: Line breaks and brackets in LaTeX export

[Juan Manuel Macías]

Max Nikulin writes:

> On 17/10/2022 22:01, Juan Manuel Macías wrote:
>> \documentclass{article}
>> \usepackage{tabularray}
>
> LaTeX I have installed is too old for this package. It is marked as
> "Experimental LaTeX3", so I am unsure, maybe you have found a bug in
> this package.
> https://ctan.org/pkg/tabularray

As I said, I don't use this package. Once I wanted to start using it,
because it has many interesting features, but I gave up on it because
this package does not (at the moment) have support for the CMYK color
space (necessary for print publication) with the xcolors package.

Maybe it's a bug, but the situation is that compiling the copied
examples as they are in the package manual, the result is correct (as it
is also shown in the manual); but adding an unexpected element (an
\empty on the last line) produces a bad result. Since you can't test the
package, I've taken this screenshot:

https://i.imgur.com/jkSHUMP.png

I think there is a tabularray user on the list, Vikas Rawal. In fact, I
also remember that I provided a patch so that he could use this package
in Org
(https://list.orgmode.org/CALtzAB1yM+uG_xHghCxTLRX5mgbzNvT5+PO=DuaBB28nCsVqEA@mail.gmail.com/).
I've cc'd him in case he wants to join the discussion.

> You believe that an issue with brackets is extremely rare. It may be
> true for humanitarian texts. For some users it may be a constant
> source of pain e.g. in the case of interval notation as [a,b]. I have
> already mentioned tables generated by code blocks, not typed directly.
> I can not say that I often need to export my notes, but I was afraid
> that I will be bitten by this bug because I may try to put dates close
> to left margin:
>
> - Something\\
>   [2022-10-17 Mon]
>
> By default dates wrapped into \textit, but it may be customized to
> just "%s".
>
> Selectively adding some workaround require complete reimplementation
> of exporters. I have some curiosity concerning pandoc approach, but I
> am unsure if I will reserve some time to read its code.

I see. If the selective solution is going to involve rewriting the
exporters, I find that it is unaffordable in the present circumstances.
It's a shame, because pandoc's solution seems ideal to me. What I
couldn't say is how pandoc does it and if it does it whenever expected,
because I use very little pandoc.

> I found \empty when I was looking for an approach with minimal
> overhead. I expect that e.g. \\[0pt] may have higher performance
> penalty since it is expanded to several commands. When the idea with
> "\\\relax" failed I was choosing between "\\{}" and "\\\empty". I
> decided that the latter minimizes risk to add spurious space.

Assuming that there is currently no alternative to the non-selective
solution, and that, as you say, the presence of brackets may be more
common than I initially expected, if I had to choose between \empty and
[0pt], I would say that [0pt] is the safest, as it is an expected
argument to \\ and equals the default space. I can't think of any
unexpected results from this, but of course, it also depends on there
being no package redefining \\ with another argument structure on its
own. I think it would be a bad idea for a package developer, but LaTeX
(and the LaTeX packages) is horribly unpredictable.

Best regards,

Juan Manuel 

Re: Line breaks and brackets in LaTeX export

[Ihor Radchenko]

Juan Manuel Macías <maciaschain@posteo.net> writes:

> Assuming that there is currently no alternative to the non-selective
> solution, and that, as you say, the presence of brackets may be more
> common than I initially expected, if I had to choose between \empty and
> [0pt], I would say that [0pt] is the safest, as it is an expected
> argument to \\ and equals the default space. I can't think of any
> unexpected results from this, but of course, it also depends on there
> being no package redefining \\ with another argument structure on its
> own. I think it would be a bad idea for a package developer, but LaTeX
> (and the LaTeX packages) is horribly unpredictable.

It is easy to change \\\empty constant to \\[0pt] if necessary. I have
no objection either way. Though I do not feel like we are in rush. I'd
like to hear from ox-latex maintainer.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Line breaks and brackets in LaTeX export

[Juan Manuel Macías]

Ihor Radchenko writes:

>> Assuming that there is currently no alternative to the non-selective
>> solution, and that, as you say, the presence of brackets may be more
>> common than I initially expected, if I had to choose between \empty and
>> [0pt], I would say that [0pt] is the safest, as it is an expected
>> argument to \\ and equals the default space. I can't think of any
>> unexpected results from this, but of course, it also depends on there
>> being no package redefining \\ with another argument structure on its
>> own. I think it would be a bad idea for a package developer, but LaTeX
>> (and the LaTeX packages) is horribly unpredictable.
>
> It is easy to change \\\empty constant to \\[0pt] if necessary. I have
> no objection either way. Though I do not feel like we are in rush. I'd
> like to hear from ox-latex maintainer.

Today I have tried with the latest version of tabularray (2022C, the one
I tried yesterday was 2022A, included in TeX Live 2022), and the bad
results persist. Also, it now returns a compile error when an \empty
precedes a \hline. I suspect this package does a pretty drastic
redefinition of \\. The [0pt] option still works fine here, though.

Re: Line breaks and brackets in LaTeX export

[Ihor Radchenko]

[-- Attachment #1: Type: text/plain, Size: 721 bytes --]

Juan Manuel Macías <maciaschain@posteo.net> writes:

>> It is easy to change \\\empty constant to \\[0pt] if necessary. I have
>> no objection either way. Though I do not feel like we are in rush. I'd
>> like to hear from ox-latex maintainer.
>
> Today I have tried with the latest version of tabularray (2022C, the one
> I tried yesterday was 2022A, included in TeX Live 2022), and the bad
> results persist. Also, it now returns a compile error when an \empty
> precedes a \hline. I suspect this package does a pretty drastic
> redefinition of \\. The [0pt] option still works fine here, though.

Then [0pt] should it be. At least for now, before we have a cleaner
solution.

See the attached patch.


[-- Attachment #2: 0001-org-latex-line-break-safe-Use-safer-value-of-0pt.patch --]
[-- Type: text/x-patch, Size: 4217 bytes --]

From b060f63078d65758f8fd2ab7725fbcf8b2de0057 Mon Sep 17 00:00:00 2001
Message-Id: <b060f63078d65758f8fd2ab7725fbcf8b2de0057.1666151751.git.yantar92@posteo.net>
From: Ihor Radchenko <yantar92@posteo.net>
Date: Wed, 19 Oct 2022 11:48:26 +0800
Subject: [PATCH] org-latex-line-break-safe: Use safer value of "\\[0pt]"
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

* lisp/ox-latex.el (org-latex-line-break-safe):
(org-latex-table-row):
Change \empty ending to explicit optional argument.  \empty still has
undesired side effects in some cases.

* testing/lisp/test-org-table.el (test-org-table/to-latex):
* testing/lisp/test-ox-latex.el (test-ox-latex/verse): Update tests.

Reported-by: Juan Manuel Macías <maciaschain@posteo.net>
Link: https://orgmode.org/list/87o7u9rz1a.fsf@posteo.net
---
 lisp/ox-latex.el               | 12 ++++++------
 testing/lisp/test-org-table.el |  6 +++---
 testing/lisp/test-ox-latex.el  | 12 ++++++------
 3 files changed, 15 insertions(+), 15 deletions(-)

diff --git a/lisp/ox-latex.el b/lisp/ox-latex.el
index dc8477d14..a5652fd78 100644
--- a/lisp/ox-latex.el
+++ b/lisp/ox-latex.el
@@ -278,17 +278,17 @@ (defconst org-latex-language-alist
 
 - `:lang-name' the actual name of the language.")
 
-(defconst org-latex-line-break-safe "\\\\\\empty"
+(defconst org-latex-line-break-safe "\\\\[0pt]"
   "Linebreak protecting the following [...].
 
-Without \"\\empty\" it would be interpreted as an optional argument to
+Without \"[0pt]\" it would be interpreted as an optional argument to
 the \\\\.
 
 This constant, for example, makes the below code not err:
 
 \\begin{tabular}{c|c}
-    [t] & s\\\\\\empty
-    [I] & A\\\\\\empty
+    [t] & s\\\\[0pt]
+    [I] & A\\\\[0pt]
     [m] & kg
 \\end{tabular}")
 
@@ -4005,9 +4005,9 @@ (defun org-latex-table-row (table-row contents info)
 			      (org-export-get-parent-table table-row) info))))
 	   (format "%s
 \\endfirsthead
-\\multicolumn{%d}{l}{%s} \\\\\\empty
+\\multicolumn{%d}{l}{%s} \\\\[0pt]
 %s
-%s \\\\\\empty\n
+%s \\\\[0pt]\n
 %s
 \\endhead
 %s\\multicolumn{%d}{r}{%s} \\\\
diff --git a/testing/lisp/test-org-table.el b/testing/lisp/test-org-table.el
index 722c37ea4..573179878 100644
--- a/testing/lisp/test-org-table.el
+++ b/testing/lisp/test-org-table.el
@@ -1635,11 +1635,11 @@ (ert-deftest test-org-table/to-generic ()
 (ert-deftest test-org-table/to-latex ()
   "Test `orgtbl-to-latex' specifications."
   (should
-   (equal "\\begin{tabular}{l}\na\\\\\\empty\n\\end{tabular}"
+   (equal "\\begin{tabular}{l}\na\\\\[0pt]\n\\end{tabular}"
 	  (orgtbl-to-latex (org-table-to-lisp "| a |") nil)))
   ;; Test :environment parameter.
   (should
-   (equal "\\begin{tabularx}{l}\na\\\\\\empty\n\\end{tabularx}"
+   (equal "\\begin{tabularx}{l}\na\\\\[0pt]\n\\end{tabularx}"
 	  (orgtbl-to-latex (org-table-to-lisp "| a |")
 			   '(:environment "tabularx"))))
   ;; Test :booktabs parameter.
@@ -1648,7 +1648,7 @@ (ert-deftest test-org-table/to-latex ()
     "\\toprule" (orgtbl-to-latex (org-table-to-lisp "| a |") '(:booktabs t))))
   ;; Handle LaTeX snippets.
   (should
-   (equal "\\begin{tabular}{l}\n\\(x\\)\\\\\\empty\n\\end{tabular}"
+   (equal "\\begin{tabular}{l}\n\\(x\\)\\\\[0pt]\n\\end{tabular}"
 	  (orgtbl-to-latex (org-table-to-lisp "| $x$ |") nil)))
   ;; Test pseudo objects and :raw parameter.
   (should
diff --git a/testing/lisp/test-ox-latex.el b/testing/lisp/test-ox-latex.el
index 4fb9f2888..adb3a60ea 100644
--- a/testing/lisp/test-ox-latex.el
+++ b/testing/lisp/test-ox-latex.el
@@ -60,14 +60,14 @@ (ert-deftest test-ox-latex/verse ()
     (should
      (search-forward
       "\\begin{verse}
-lorem ipsum dolor\\\\\\empty
-lorem ipsum dolor\\\\\\empty
+lorem ipsum dolor\\\\[0pt]
+lorem ipsum dolor\\\\[0pt]
 \\vspace*{1em}
-lorem ipsum dolor\\\\\\empty
-lorem ipsum dolor\\\\\\empty
+lorem ipsum dolor\\\\[0pt]
+lorem ipsum dolor\\\\[0pt]
 \\vspace*{1em}
-lorem ipsum dolor\\\\\\empty
-lorem ipsum dolor\\\\\\empty
+lorem ipsum dolor\\\\[0pt]
+lorem ipsum dolor\\\\[0pt]
 \\end{verse}"))))
 
 (provide 'test-ox-latex)
-- 
2.35.1


[-- Attachment #3: Type: text/plain, Size: 225 bytes --]



-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Line breaks and brackets in LaTeX export

[Max Nikulin]

On 19/10/2022 10:57, Ihor Radchenko wrote:
> Juan Manuel Macías writes:
>>
>> Today I have tried with the latest version of tabularray (2022C, the one
>> I tried yesterday was 2022A, included in TeX Live 2022), and the bad
>> results persist. Also, it now returns a compile error when an \empty
>> precedes a \hline. I suspect this package does a pretty drastic
>> redefinition of \\. The [0pt] option still works fine here, though.
> 
> Then [0pt] should it be. At least for now, before we have a cleaner
> solution.

It seems when I had a look into latex.ltx first time, I confused which 
branch is executed when length is less than or equal to zero and decided 
that it is the heavier \@xargarraycr. Actually \@yargarraycr do not 
really worry me, so degree of my objection concerning \\[0pt] 
significantly decreased.

\def\@argarraycr[#1]{%
   \ifnum0=`{\fi}${}\ifdim #1>\z@ \@xargarraycr{#1}\else
    \@yargarraycr{#1}\fi}

\def\@xargarraycr#1{\@tempdima #1\advance\@tempdima \dp \@arstrutbox
    \vrule \@height\z@ \@depth\@tempdima \@width\z@ \cr}
\def\@yargarraycr#1{\cr\noalign{\vskip #1}}

I have realized that

| / | <                                | > |
|   | a                                | b |
|   | @@latex:\noalign{\vskip 1em}@@ c | d |

is not a workaround to increase local interval between rows. It may 
cause disrupted vertical rules. Another recipe should be used:

| / | < |                                 > |
|   | a | b @@latex:\rule[-1em]{0pt}{1em}@@ |
|   | c | d                                 |

I believe that a more convenient way to override [0pt] to some other 
length for particular row should exist, but I have no idea which syntax 
should be used.

As to tabulararray, I still consider it as an experimental package. 
Perhaps I will install a more modern container. I am curious what code 
handles \\[0pt]. Likely I should read docs to get impression related to 
design goals and approaches to implement them. The bug tracker of the 
project looks like an appropriate place to ask a question concerning \\ 
variant safe for dumb exporters.

For a while I have the following question. Is \\{} have the same effect 
on tabularray parser as \\\empty?

Re: Line breaks and brackets in LaTeX export

[Juan Manuel Macías]

Max Nikulin writes:

> Another recipe should be used:
>
> | / | < |                                 > |
> |   | a | b @@latex:\rule[-1em]{0pt}{1em}@@ |
> |   | c | d                                 |
>
> I believe that a more convenient way to override [0pt] to some other
> length for particular row should exist, but I have no idea which
> syntax should be used.

I think this new recipe you propose works fine, although I haven't
done a thorough test yet. In principle, it shouldn't be a problem.

By the way, now I remember that the package verse adds a series of extra
arguments to \\ (p. 6 in the documentation:
https://www.ctan.org/pkg/verse). As a user, the only way I can think of
to add arguments instead of [0pt] or \empty is by using a small export
filter to act on the final output. Perhaps putting something like
!!new-argument!! on the affected row/verse/line to guide the filter.

> As to tabulararray, I still consider it as an experimental package.
> Perhaps I will install a more modern container. I am curious what code
> handles \\[0pt]. Likely I should read docs to get impression related
> to design goals and approaches to implement them. The bug tracker of
> the project looks like an appropriate place to ask a question
> concerning \\ variant safe for dumb exporters.

Keep in mind that tabularray is a latex3 package and, in principle,
everything that is latex3 is here to stay. Certainly, it is a blessing
compared to the classic tabular environments, because it is tremendously
versatile. That is why it already has a significant number of users:
https://tex.stackexchange.com/questions/tagged/tabularray

And Org already provides what is necessary to use its syntax. I've had a
first look at tabularray.sty, but I'm not familiar enough with the new
LaTeX 3 syntax and, frankly, I'm lost...

I've tried all the packages involved in tables that I can think of
(longtable, siunitx, tabularx, booktabs, array, and I don't know if I
forgot any) and in all of them the \empty solution works fine. It seems
that tabularray is the black sheep here.

> For a while I have the following question. Is \\{} have the same
> effect on tabularray parser as \\\empty?

Throw an error before \hline.

Re: Line breaks and brackets in LaTeX export

[Juan Manuel Macías]

Juan Manuel Macías writes:

>> For a while I have the following question. Is \\{} have the same
>> effect on tabularray parser as \\\empty?
>
> Throw an error before \hline.

To expand on my answer: \\{} and \\\empty, before \hline, throw the same
error:

------------------------------
ERROR: Misplaced \noalign.

--- TeX said ---
\hline ->\noalign 
{\ifnum 0=`}\fi \hrule \@height \arrayrulewidth \futurelet...

l.17 \end
{tblr}
--- HELP ---
From the .log file...

I expect to see \noalign only after the \cr of
an alignment. Proceed, and I'll ignore this case.
------------------------------

Re: Line breaks and brackets in LaTeX export

[Max Nikulin]

>>> For a while I have the following question. Is \\{} have the same
>>> effect on tabularray parser as \\\empty?
>>
>> Throw an error before \hline.

It is possible to use
      \begin{tblr}[expand=\empty]{rl}
but I would prefer to reserve such feature for a more nobble purpose.

> I've had a
> first look at tabularray.sty, but I'm not familiar enough with the new
> LaTeX 3 syntax and, frankly, I'm lost...

While it is not easy for me to read code LaTeX2e core and packages, I 
should say that I have no guess at all when I am looking at tabularray 
sources. It is a completely different language.

I have started a discussion requesting for a \\-like command without 
optional arguments. Maybe somebody will suggest a better workaround instead.
https://github.com/lvjr/tabularray/discussions/321

> I've tried all the packages involved in tables that I can think of
> (longtable, siunitx, tabularx, booktabs, array, and I don't know if I
> forgot any) and in all of them the \empty solution works fine. It seems
> that tabularray is the black sheep here.

I think tabularray is unique with a regexp-based parser. I had a hope 
that new approach does not allow newline between \\ and its arguments, 
but unfortunately compatibility with older code is preserved in this aspect.

 From LaTeX companion I remember supertabular as an alternative for 
longtable, but I am unsure if it is alive yet.

Re: Line breaks and brackets in LaTeX export

[Juan Manuel Macías]

Max Nikulin writes:

> I have started a discussion requesting for a \\-like command without
> optional arguments. Maybe somebody will suggest a better workaround
> instead.
> https://github.com/lvjr/tabularray/discussions/321

I've had a look at the thread. What do you think of that
\NewTableCommand\empty{} workaround mentioned at
https://github.com/lvjr/tabularray/discussions/321#discussioncomment-3920957?

Since the \empty option only has problems in tabularray, maybe we could
keep it, and put in the documentation some recommendations for
tabularray users. I imagine they would have to add a @@latex:\empty@@
before each row that follows a line. A bit laborious, I'm afraid.

Another possibility that occurs to me is that the string reserved for
\empty, [0pt], etc., is a defcustom, with a value of \empty by default.
So the user would choose what suits him best.

By the way (a little digression), I was curious to see if these age-old
LaTeX problems with line breaking exist in ConTeXt as well. Since I'm
completely unfamiliar with ConTeXt, the quickest thing to do has been to
see what code ox-context returns for the org tables. The answer is that
there are no such problems, and one can safely put a square bracket at
the beginning of a row. It is also true that the table syntax in ConTeXt
is radically different from that in LaTeX. And there is also no problem
if I put @@context:a\[b]@@. Some screenshots:

https://i.imgur.com/2k1TaU9.png

https://i.imgur.com/8i9qlEH.png

Many times I've been tempted to give ConTeXt a try, but I've always run
into two things: ConTeXt's perennially experimental status and a
horrible lack of documentation. In addition, backward compatibility is
not usually respected, since ConTeXt, although it is free software, does
not have community development as a priority, but rather the company
behind it, Pragma.

>> I've tried all the packages involved in tables that I can think of
>> (longtable, siunitx, tabularx, booktabs, array, and I don't know if I
>> forgot any) and in all of them the \empty solution works fine. It seems
>> that tabularray is the black sheep here.
>
> I think tabularray is unique with a regexp-based parser. I had a hope
> that new approach does not allow newline between \\ and its arguments,
> but unfortunately compatibility with older code is preserved in this
> aspect.
>
> From LaTeX companion I remember supertabular as an alternative for
> longtable, but I am unsure if it is alive yet.

True, I had forgotten about this package (I don't think I've ever used
it). It looks like it has a 2020 new version:

@manual{supertabular,
  title = {The \texttt{supertabular} package},
  subtitle = {A multi-page tables package},
  author = {Johannes L. Braams},
  date = {2020-02-02},
  version = {4.1g},
  license = {lppl1.3c},
  url = {https://mirror.ctan.org/macros/latex/contrib/supertabular},
  pkgurl = {https://ctan.org/pkg/supertabular},
  }

(bibtex entry obtained from: https://www.ctan.org/pkg/ctan-bibdata)

Re: Line breaks and brackets in LaTeX export

[Ihor Radchenko]

Juan Manuel Macías <maciaschain@posteo.net> writes:

> I've had a look at the thread. What do you think of that
> \NewTableCommand\empty{} workaround mentioned at
> https://github.com/lvjr/tabularray/discussions/321#discussioncomment-3920957?
>
> Since the \empty option only has problems in tabularray, maybe we could
> keep it, and put in the documentation some recommendations for
> tabularray users. I imagine they would have to add a @@latex:\empty@@
> before each row that follows a line. A bit laborious, I'm afraid.

I see the tabularray issue simply as an example that \empty is not as
reliable as we thought. There might be other LaTeX packages throwing
errors on \\\empty.

The proposed workaround may be enough for tabularray, but may not be for
other packages.

> Another possibility that occurs to me is that the string reserved for
> \empty, [0pt], etc., is a defcustom, with a value of \empty by default.
> So the user would choose what suits him best.

I do not think that users will clearly understand the purpose of such
defcustom. It is solving some very narrow edge case, and it is unclear
why we would need to advertise changing it in customize interface.

I'd prefer to keep it as defconst, but maybe mention in the docstring
that it can still be set to "\\empty" as another semi-safe value. At the
user's own risk.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Line breaks and brackets in LaTeX export

[Max Nikulin]

On 21/10/2022 10:34, Ihor Radchenko wrote:
> Juan Manuel Macías writes:
> 
> I see the tabularray issue simply as an example that \empty is not as
> reliable as we thought. There might be other LaTeX packages throwing
> errors on \\\empty.

My impression is that tabularray has an ambitious goal to replace all 
current table packages. I have no idea if other packages will adopt 
similar approach with regexp-based parsing instead of usual expanding of 
TeX commands.

I do not like necessity to add \NewTableCommand\empty{} to documents 
somehow (only if tabularray is loaded). I do not have an idea better 
than \\[0pt] and an optimizing filter to remove [0pt] in almost all cases.

Re: Line breaks and brackets in LaTeX export

[Juan Manuel Macías]

Max Nikulin writes:

> My impression is that tabularray has an ambitious goal to replace all
> current table packages. I have no idea if other packages will adopt
> similar approach with regexp-based parsing instead of usual expanding
> of TeX commands.

Yes, that's the impression I have too. Tabularray certainly solves a lot
of traditional LaTeX table problems (unfortunately not the one we're
dealing with). Time will tell if it also creates other new problems. In
any case, it brings a lot of flexibility to a part of LaTeX, tables,
that has always suffered from a certain constriction. We'll see what
happens. LaTeX is becoming very complex and now several layers coexist,
since the jump to LaTeX 3 is going to be gradual. On the other hand, I
don't know if the latex core developers have a cleaner \\ command in
their roadmap, without those absurd current problems (and that LaTeX has
been carrying for almost 40 years).

> I do not like necessity to add \NewTableCommand\empty{} to documents
> somehow (only if tabularray is loaded). I do not have an idea better
> than \\[0pt] and an optimizing filter to remove [0pt] in almost all
> cases.

I totally agree.

Best regards,

Juan Manuel 

Re: Line breaks and brackets in LaTeX export

[Juan Manuel Macías]

Max Nikulin writes:

> A workaround for tabularray:
>
>     \NewTableCommand\empty{}
>
> I am unsure if we should use it.

Ah, I had just commented on this in the email I just sent...

>> By the way, now I remember that the package verse adds a series of
>> extra
>> arguments to \\ (p. 6 in the documentation:
>
> \\! command between stanzas to get proper line count makes things a
> bit more complicated. I believed that some TeX trick may be used
> instead of requirement of the explicit command.

In my custom code that I commented on in another email, \\! is
automatically inserted if verse numbering is active. My intention is to
incorporate that into my patch (work-in-progress) over the spaces between
verses.

Re: Line breaks and brackets in LaTeX export

[Ihor Radchenko]

Ihor Radchenko <yantar92@posteo.net> writes:

> Then [0pt] should it be. At least for now, before we have a cleaner
> solution.
>
> See the attached patch.

Applied onto main.
https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=b93a61af9c93d21c56cf883630e52f36076e40bd

I will look into filtering out unnecessary [0pt] instances later.
The patch here is more urgent as it avoids the known case of breakage.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Line breaks and brackets in LaTeX export

[Max Nikulin]

On 29/10/2022 09:36, Ihor Radchenko wrote:
> Ihor Radchenko writes:
> 
>> Then [0pt] should it be. At least for now, before we have a cleaner
>> solution.
>>
>> See the attached patch.
> 
> Applied onto main.
> https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=b93a61af9c93d21c56cf883630e52f36076e40bd
> 
> I will look into filtering out unnecessary [0pt] instances later.
> The patch here is more urgent as it avoids the known case of breakage.

Tabularray became aware of \empty:
https://github.com/lvjr/tabularray/issues/328

So in future this command may be used instead of [0pt].

Re: Line breaks and brackets in LaTeX export

[Ihor Radchenko]

Max Nikulin <manikulin@gmail.com> writes:

> Selectively adding some workaround require complete reimplementation of 
> exporters. I have some curiosity concerning pandoc approach, but I am 
> unsure if I will reserve some time to read its code.

Maybe or maybe not. We have org-export-get-next-element and a number of
exporters doing something special for the first/last object inside a
container. So, an alternative workaround might be possible. But
certainly more complex and fragile.

> An idea how to avoid complete redesign: at first add something like
>     %__ORG_PROTECT_NEWLINE__
> after each \\, later at an optimizing pass remove the comment if next 
> line does not start from a star or a square bracket, otherwise use some 
> workaround, e.g. "{[}". \relax may be suitable as well (in the beginning 
> of rows, not after \\).

I am not sure if it is a good idea. It may interfere with export filters.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Line breaks and brackets in LaTeX export

[Max Nikulin]

On 18/10/2022 11:39, Ihor Radchenko wrote:
> Max Nikulin writes:
> 
>> Selectively adding some workaround require complete reimplementation of
>> exporters. I have some curiosity concerning pandoc approach, but I am
>> unsure if I will reserve some time to read its code.
> 
> Maybe or maybe not. We have org-export-get-next-element and a number of
> exporters doing something special for the first/last object inside a
> container.

I have an impression that markup generated by children often added using 
%s to some other construct. I may be wrong.

>> An idea how to avoid complete redesign: at first add something like
>>      %__ORG_PROTECT_NEWLINE__
>> after each \\, later at an optimizing pass remove the comment if next
>> line does not start from a star or a square bracket, otherwise use some
>> workaround, e.g. "{[}". \relax may be suitable as well (in the beginning
>> of rows, not after \\).
> 
> I am not sure if it is a good idea. It may interfere with export filters.

I do not expect negative effect from a comment added at the end of line. 
Optimizing pass may be performed prior to user filters.

As another approach text properties may be used as a communication 
channel unless they are stripped by ox.

Re: Line breaks and brackets in LaTeX export

[Ihor Radchenko]

Max Nikulin <manikulin@gmail.com> writes:

>>> Selectively adding some workaround require complete reimplementation of
>>> exporters. I have some curiosity concerning pandoc approach, but I am
>>> unsure if I will reserve some time to read its code.
>> 
>> Maybe or maybe not. We have org-export-get-next-element and a number of
>> exporters doing something special for the first/last object inside a
>> container.
>
> I have an impression that markup generated by children often added using 
> %s to some other construct. I may be wrong.

You are right, but missing subtle detail.
Export transcoders for container elements are called with 3 arguments:
data, contents, and info.

DATA contains the actual AST of the transcoded element, including its
children available via org-element-contents. CONTENTS is the string
representations of the exported children elements. Before CONTENTS is
generated, relevant transcoders are also called for each child of the
DATA.

When transcoding children (e.g. table rows), the sibling rows can always
be accessed using org-export-get-previous-element and
org-export-get-next-element.

>>> An idea how to avoid complete redesign: at first add something like
>>>      %__ORG_PROTECT_NEWLINE__
>>> after each \\, later at an optimizing pass remove the comment if next
>>> line does not start from a star or a square bracket, otherwise use some
>>> workaround, e.g. "{[}". \relax may be suitable as well (in the beginning
>>> of rows, not after \\).
>> 
>> I am not sure if it is a good idea. It may interfere with export filters.
>
> I do not expect negative effect from a comment added at the end of line. 
> Optimizing pass may be performed prior to user filters.

Filters are applied by ox.el immediately after transcoding each element.
See the final expressions in org-export-data. The filters may alter the
exported string, including removing the %__ORG_PROTECT_NEWLINE__ we put
there. Of course, we can warn users about this particular caveat, but I
do not like such design -- it is too fragile.

> As another approach text properties may be used as a communication 
> channel unless they are stripped by ox.

I am not sure what you are referring to. If modifying exported string,
it will suffer from the same problems as your idea with comment.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Line breaks and brackets in LaTeX export

[Max Nikulin]

An argument in favor of \\[0pt] escaping. Unlike \empty, [0pt] is rather 
artificial, so such pattern may be removed (I hope in a safe way) by an 
export filter if it is not followed by a bracket or a star.

On 20/10/2022 12:07, Ihor Radchenko wrote:
> When transcoding children (e.g. table rows), the sibling rows can always
> be accessed using org-export-get-previous-element and
> org-export-get-next-element.

Decision if escaping is necessary should be based on export result, not 
on the source element.

> Max Nikulin writes:

>> As another approach text properties may be used as a communication
>> channel unless they are stripped by ox.
> 
> I am not sure what you are referring to. If modifying exported string,
> it will suffer from the same problems as your idea with comment.

It is a brain storm idea, I still unsure it is feasible.

I mean info "(elisp) Text Properties". I do not think they are usually 
ignored by filters. \\ substrings added by ox-latex may have some unique 
property. Export filter searches for occurrences of \\ to check if 
escaping is necessary, but it ignores particular location if the text 
does not have the specific property. I would prefer to mark output of 
export snippets with another property value and do not escape them. It 
is a kind of out of band communication. Unsure if text properties may be 
lost due to string operations in an exporter or vice versa to propagate 
to the source buffer.

Re: Line breaks and brackets in LaTeX export

[Ihor Radchenko]

Max Nikulin <manikulin@gmail.com> writes:

> On 20/10/2022 12:07, Ihor Radchenko wrote:
>> When transcoding children (e.g. table rows), the sibling rows can always
>> be accessed using org-export-get-previous-element and
>> org-export-get-next-element.
>
> Decision if escaping is necessary should be based on export result, not 
> on the source element.

I am not sure. The export result may originate from constructs like

#+MACRO: pagebreak @@latex:\\@@

Some paragraph {{{pagebreak}}}
src_elisp[:results raw]{(format "@@latex:[%dpt]@@" (or my-spacing-value 0))}

The point is: it can be done on purpose for some reason.

>> I am not sure what you are referring to. If modifying exported string,
>> it will suffer from the same problems as your idea with comment.
>
> It is a brain storm idea, I still unsure it is feasible.
>
> I mean info "(elisp) Text Properties". I do not think they are usually 
> ignored by filters. \\ substrings added by ox-latex may have some unique 
> property. Export filter searches for occurrences of \\ to check if 
> escaping is necessary, but it ignores particular location if the text 
> does not have the specific property. I would prefer to mark output of 
> export snippets with another property value and do not escape them. It 
> is a kind of out of band communication. Unsure if text properties may be 
> lost due to string operations in an exporter or vice versa to propagate 
> to the source buffer.

I see. I feel like it is too fragile to rely upon. At least, I'd use it
only if there are no better options.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Line breaks and brackets in LaTeX export

[Max Nikulin]

On 21/10/2022 10:41, Ihor Radchenko wrote:
> Max Nikulin writes:
> 
>> On 20/10/2022 12:07, Ihor Radchenko wrote:
>>> When transcoding children (e.g. table rows), the sibling rows can always
>>> be accessed using org-export-get-previous-element and
>>> org-export-get-next-element.
>>
>> Decision if escaping is necessary should be based on export result, not
>> on the source element.
> 
> I am not sure. The export result may originate from constructs like
> 
> #+MACRO: pagebreak @@latex:\\@@
> 
> Some paragraph {{{pagebreak}}}
> src_elisp[:results raw]{(format "@@latex:[%dpt]@@" (or my-spacing-value 0))}
> 
> The point is: it can be done on purpose for some reason.

I agree that it can be done for purpose and I am not happy that due to 
table syntax it is not easy to emit custom code between rows.

Since you declined an idea to mark default line breaks with some unique 
comment to use it in search pattern in a filter, I may suggest to change 
custom code to either remove newline before brackets or to add a comment 
after \\

Some paragraph
src_elisp[:results raw]{(format "@@latex:\\\\[%dpt]@@" (or 
my-spacing-value 0))}

or #+MACRO: pagebreak @@latex:\\%@@

With any variant such line break will be ignored by "optimizing" filter.

Concerning element vs. exported text, consider a derived backend that 
ignores italics markers if a paragraph has some attribute. Usually
      Paragraph \\
      \emph{[something]}
does not cause any problem, however if italics is ignored it is an error
      Paragraph\\
      [something]
That is why namely exported code of adjacent leaf node should be 
examined. Ideally there should be a possibility to add some attributes 
or properties to distinguish raw export snippet. Unfortunately it 
requires complete redesign of org-export.

Re: Line breaks and brackets in LaTeX export

[Ihor Radchenko]

Max Nikulin <manikulin@gmail.com> writes:

> Concerning element vs. exported text, consider a derived backend that 
> ignores italics markers if a paragraph has some attribute. Usually
>       Paragraph \\
>       \emph{[something]}
> does not cause any problem, however if italics is ignored it is an error
>       Paragraph\\
>       [something]
> That is why namely exported code of adjacent leaf node should be 
> examined. Ideally there should be a possibility to add some attributes 
> or properties to distinguish raw export snippet.

This is a valid example.

> Unfortunately it requires complete redesign of org-export.

Thinking about it more, we may actually do post-processing of the
exported text. Unless I miss something, constructs like "\\[0pt]\n" can
_always_ be replaced with "\\\n" as long as the next line does not match
"^[ \t]*\\[". Even when such construct is deliberately placed by the
user. There will be no difference in the generated pdf.

Or, to be completely safe, we can introduce a defcustom that will
control such clean-up (clean up by default).

I propose the following:
1. Merge my patch with \\[0pt] safe page breaks
2. Modify org-latex-template to replace unnecessary occurrences of
   "\\[0pt]" in CONTENTS when org-latex-compact-latex (you may propose
   other defcustom names) is non-nil.

WDYT?

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Line breaks and brackets in LaTeX export

[Juan Manuel Macías]

Ihor Radchenko writes:

> Or, to be completely safe, we can introduce a defcustom that will
> control such clean-up (clean up by default).
>
> I propose the following:
> 1. Merge my patch with \\[0pt] safe page breaks
> 2. Modify org-latex-template to replace unnecessary occurrences of
>    "\\[0pt]" in CONTENTS when org-latex-compact-latex (you may propose
>    other defcustom names) is non-nil.
>
> WDYT?

+1

Best regards,

Juan Manuel 

Re: Line breaks and brackets in LaTeX export

[Max Nikulin]

On 22/10/2022 12:15, Ihor Radchenko wrote:
> as long as the next line does not match
> "^[ \t]*\\["

Verse package defines \\! and \\>.

The only precaution that search pattern should ignore \\\[0pt]\] that is 
a display equation "0pt]"

> I propose the following:
> 1. Merge my patch with \\[0pt] safe page breaks
> 2. Modify org-latex-template to replace unnecessary occurrences of
>     "\\[0pt]" in CONTENTS when org-latex-compact-latex (you may propose
>     other defcustom names) is non-nil.

I believe, it is better to introduce a list of filter functions that may 
perform such optimizing.

The only problem that I see with such approach is that if defined as 
defcustom then users changed its value will not see default filters 
added in new Org version. If an alist mapping hook name to 
enabled/disabled state is used then I am unsure how to determine order 
of hooks during mixing of standard and user defined functions. I raised 
a similar question during discussion of org-file-apps.

I do not have a better proposal.

Re: Line breaks and brackets in LaTeX export

[Ihor Radchenko]

Max Nikulin <manikulin@gmail.com> writes:

> On 22/10/2022 12:15, Ihor Radchenko wrote:
>> as long as the next line does not match
>> "^[ \t]*\\["
>
> Verse package defines \\! and \\>.
>
> The only precaution that search pattern should ignore \\\[0pt]\] that is 
> a display equation "0pt]"

Can you please elaborate?

>> 2. Modify org-latex-template to replace unnecessary occurrences of
>>     "\\[0pt]" in CONTENTS when org-latex-compact-latex (you may propose
>>     other defcustom names) is non-nil.
>
> I believe, it is better to introduce a list of filter functions that may 
> perform such optimizing.

I don't think so. Users can already define filters. What we discuss here
is a built-in filter. It should be separate from user customization and
only expose on/off options.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Line breaks and brackets in LaTeX export

[Max Nikulin]

On 01/11/2022 08:51, Ihor Radchenko wrote:
> Max Nikulin writes:
> 
>> On 22/10/2022 12:15, Ihor Radchenko wrote:
>>> as long as the next line does not match
>>> "^[ \t]*\\["
>>
>> Verse package defines \\! and \\>.
>>
>> The only precaution that search pattern should ignore \\\[0pt]\] that is
>> a display equation "0pt]"
> 
> Can you please elaborate?

LaTeX verse package require "\\!" as stanza separator to get proper line 
count, so square bracket on the next line is not the only character that 
may change meaning of "\\". So "[*!>" (depending on context) should be 
handled.

Search pattern should not match odd number of backslashes. It is a 
common mistake to not care what character precedes the pattern. 
Accordingly to Org syntax \\ line break may appear at the end of line 
only, but a user may add for some reason @@latex:...@.

Another corner case when "\\[0pt]\n" should not be blindly replaced to 
"\\\n" (I do not escape "\"). Imagine that you are going to describe 
this optimizing filter

---- >8 ----
Optimizing filter will replace
#+begin_example
First\\[0pt]
Second
#+end_example
with
#+begin_example
First\\
Second
#+end_example
---- 8< -----

>>> 2. Modify org-latex-template to replace unnecessary occurrences of
>>>      "\\[0pt]" in CONTENTS when org-latex-compact-latex (you may propose
>>>      other defcustom names) is non-nil.
>>
>> I believe, it is better to introduce a list of filter functions that may
>> perform such optimizing.
> 
> I don't think so. Users can already define filters. What we discuss here
> is a built-in filter. It should be separate from user customization and
> only expose on/off options.

If a separate defcustom is used for each built-in filter then it will be 
inconvenient to review which filters are active from easy customization UI.

It might be better to use single defcustom of list type allowing to 
disable particular built-in filters and to add user filters and to 
define their order in respect to built-in ones. I am unsure concerning 
convenient enough UI.

Re: Line breaks and brackets in LaTeX export

[Ihor Radchenko]

Max Nikulin <manikulin@gmail.com> writes:

> Another corner case when "\\[0pt]\n" should not be blindly replaced to 
> "\\\n" (I do not escape "\"). Imagine that you are going to describe 
> this optimizing filter
>
> ---- >8 ----
> Optimizing filter will replace
> #+begin_example
> First\\[0pt]
> Second
> #+end_example
> with
> #+begin_example
> First\\
> Second
> #+end_example
> ---- 8< -----

This is a strong argument against auto-removing the \\[0pt].
Should we, instead of using exact "\\[0pt]" string for line breaks,
define a new LaTeX command and then clean it up? This will distinguish
between \\[0pt] added by users explicitly and the ones generated
automatically by Org.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Line breaks and brackets in LaTeX export

[Ihor Radchenko]

Ihor Radchenko <yantar92@posteo.net> writes:

> Should we, instead of using exact "\\[0pt]" string for line breaks,
> define a new LaTeX command and then clean it up? This will distinguish
> between \\[0pt] added by users explicitly and the ones generated
> automatically by Org.

I guess it will not work for that fancy table package we discussed
above. Maybe the Max's idea with "\\[0pt]%some comment indicator"?
I am just worried that % does have side effects in LaTeX.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Line breaks and brackets in LaTeX export

[Max Nikulin]

On 02/11/2022 13:46, Ihor Radchenko wrote:
> Ihor Radchenko writes:
> 
>> Should we, instead of using exact "\\[0pt]" string for line breaks,
>> define a new LaTeX command and then clean it up? This will distinguish
>> between \\[0pt] added by users explicitly and the ones generated
>> automatically by Org.
> 
> I guess it will not work for that fancy table package we discussed
> above.

tabularray package will require \NewTableCommand

> Maybe the Max's idea with "\\[0pt]%some comment indicator"?
> I am just worried that % does have side effects in LaTeX.

TeX reads

"a% comment
   b"

as "ab", dropping newline and starting spaces. Of course, advanced users 
may redefine category of "%" from comment to something else (regular 
character, command prefix like \, etc.) like it is done inside verbatim 
environment.

Re: Line breaks and brackets in LaTeX export

[Ihor Radchenko]

Max Nikulin <manikulin@gmail.com> writes:

> On 02/11/2022 13:46, Ihor Radchenko wrote:
>> Ihor Radchenko writes:
>> 
>>> Should we, instead of using exact "\\[0pt]" string for line breaks,
>>> define a new LaTeX command and then clean it up? This will distinguish
>>> between \\[0pt] added by users explicitly and the ones generated
>>> automatically by Org.
>> 
>> I guess it will not work for that fancy table package we discussed
>> above.
>
> tabularray package will require \NewTableCommand
>
>> Maybe the Max's idea with "\\[0pt]%some comment indicator"?
>> I am just worried that % does have side effects in LaTeX.
>
> TeX reads
>
> "a% comment
>    b"
>
> as "ab", dropping newline and starting spaces. Of course, advanced users 
> may redefine category of "%" from comment to something else (regular 
> character, command prefix like \, etc.) like it is done inside verbatim 
> environment.

These arguments mean that auto-cleaning \\[0pt] is not always safe and
may be a subject of surrounding LaTeX context. Moreover, there is no
clear alternative to \\[0pt] that is guaranteed to work.
Thus, the whole idea with cleaning up the generated LaTeX cannot be
enabled by default, and I am not even sure if it is something we want to
implement in the core.

Juan, maybe you have some good alternative suggestions?

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Line breaks and brackets in LaTeX export

[Juan Manuel Macías]

Ihor Radchenko writes:

> These arguments mean that auto-cleaning \\[0pt] is not always safe and
> may be a subject of surrounding LaTeX context. Moreover, there is no
> clear alternative to \\[0pt] that is guaranteed to work.
> Thus, the whole idea with cleaning up the generated LaTeX cannot be
> enabled by default, and I am not even sure if it is something we want to
> implement in the core.
>
> Juan, maybe you have some good alternative suggestions?

I'm afraid not. I've been trying to follow the thread these past few
days and frankly I can't think of anything beyond all the options that
have already been discussed.

I agree that the safest solution (or the least compromised, depending on
how we look at it) is to add [0pt] in all cases. I believe this should
not bring any unexpected consequences. At least I've tried all the
tabular environments related packages that I know of and there don't
seem to be any problems. And in the case of the verse package and verse
numberings, the solution is ad hoc with the patch that I have planned (I
hope to find a gap these days to finish it and test it...): The idea of
the patch is to replace the current separation between stanzas that Org
adds (\vspace) with a white line, and (only) in that case it would be
necessary to ensure that the last verse of the stanza ends in \\!.

On the other hand, if 'factory-cleaning' unnecessary instances of [0pt]
is going to be problematic, I also agree not to add that action to the
exporter. I think it would be better to leave these things up to the
users, depending on their use cases. After all, it is not difficult to
do it using a specific export filter. Maybe add some tips and examples
of basic filters in the documentation, especially for users who don't
have a lot of experience writing filters?

BTW, I also hope that one day this LaTeX problem, which has been there
for so many years, will be solved (because it should be considered a
LaTeX bug, or a LaTeX design flaw, as Maxim commented in another
message). 

Best regards,

Juan Manuel 

Re: Line breaks and brackets in LaTeX export

[Max Nikulin]

> Ihor Radchenko writes:
> 
>> These arguments mean that auto-cleaning \\[0pt] is not always safe and
>> may be a subject of surrounding LaTeX context.

I still believe that

     something\\[0pt]%__ORG_EXPORT__

is quite safe to remove (depending on the following character) and 
unlikely harmful if remained for some reason. Even tabularray 
regexp-based parser ignores comments. To be perfect, 
"\\[0pt]%__ORG_EXPORT__" emitted by user code should be escaped to e.g. 
"\\[0pd]%__ORG_EXPORT____ORG_EXPORT__", but I have no idea if it can be 
implemented in a reliable way.

An alternative is a new export framework.

On 03/11/2022 22:00, Juan Manuel Macías wrote:
> 
> BTW, I also hope that one day this LaTeX problem, which has been there
> for so many years, will be solved (because it should be considered a
> LaTeX bug, or a LaTeX design flaw, as Maxim commented in another
> message).

I have not managed to convince even the tabularray developer. And I have 
not tried to find a way to reach more LaTeX developers.

Re: Line breaks and brackets in LaTeX export

[Juan Manuel Macías]

Max Nikulin writes:

> I have not managed to convince even the tabularray developer. And I
> have not tried to find a way to reach more LaTeX developers.

I think that people from the LaTeX team and the authors of the most
popular packages are often very active on tex.stackexchange.com

And the repo on GitHub for the LaTeX project: https://github.com/latex3

Re: Line breaks and brackets in LaTeX export

[Ihor Radchenko]

Max Nikulin <manikulin@gmail.com> writes:

>> Ihor Radchenko writes:
>> 
>>> These arguments mean that auto-cleaning \\[0pt] is not always safe and
>>> may be a subject of surrounding LaTeX context.
>
> I still believe that
>
>      something\\[0pt]%__ORG_EXPORT__
>
> is quite safe to remove (depending on the following character) and 
> unlikely harmful if remained for some reason.

What about the side effect you mentioned in a previous email?

>		   TeX reads
>
>		   "a% comment
>		      b"
>
>		   as "ab", dropping newline and starting spaces.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Line breaks and brackets in LaTeX export

[Max Nikulin]

On 04/11/2022 11:23, Ihor Radchenko wrote:
> Max Nikulin writes:
>>> Ihor Radchenko writes:
>>>
>>>> These arguments mean that auto-cleaning \\[0pt] is not always safe and
>>>> may be a subject of surrounding LaTeX context.
>>
>> I still believe that
>>
>>       something\\[0pt]%__ORG_EXPORT__
>>
>> is quite safe to remove (depending on the following character) and
>> unlikely harmful if remained for some reason.
> 
> What about the side effect you mentioned in a previous email?
> 
>> 		   TeX reads
>>
>> 		   "a% comment
>> 		      b"
>>
>> 		   as "ab", dropping newline and starting spaces.

Unlike in Org, in TeX "a\\b" still contains a line break. Besides exotic 
packages and user setups it should just work. After all, during normal 
operation all "%__ORG_EXPORT__" should be stripped by a filter. They are 
just marks where a decision is necessary whether to retain "[0pt]" or to 
remove it.

On 03/11/2022 22:48, Juan Manuel Macías wrote:
> Max Nikulin writes:
> 
>> I have not managed to convince even the tabularray developer. And I
>> have not tried to find a way to reach more LaTeX developers.
> 
> I think that people from the LaTeX team and the authors of the most
> popular packages are often very active on tex.stackexchange.com

Stackexchange is for questions, not for discussions and bugs. At certain 
point I was going to ask a question there, I even prepared a draft. 
After that I decided to try a command expanding to nothing and found 
\empty. I was not aware that tabularray uses a different approach to 
parsing.

> And the repo on GitHub for the LaTeX project: https://github.com/latex3

https://github.com/latex3/latex3/issues may be an appropriate place, 
thank you.

Re: Line breaks and brackets in LaTeX export

[Ihor Radchenko]

Max Nikulin <manikulin@gmail.com> writes:

>>> I still believe that
>>>
>>>       something\\[0pt]%__ORG_EXPORT__
>>>
>>> is quite safe to remove (depending on the following character) and
>>> unlikely harmful if remained for some reason.
>> 
>> What about the side effect you mentioned in a previous email?
>> 
>>> 		   TeX reads
>>>
>>> 		   "a% comment
>>> 		      b"
>>>
>>> 		   as "ab", dropping newline and starting spaces.
>
> Unlike in Org, in TeX "a\\b" still contains a line break. Besides exotic 
> packages and user setups it should just work. After all, during normal 
> operation all "%__ORG_EXPORT__" should be stripped by a filter. They are 
> just marks where a decision is necessary whether to retain "[0pt]" or to 
> remove it.

Thanks for the clarification.

So, we can

1. Change org-latex-line-break-safe to "\\\\[0pt]%__ORG_EXPORT__",
   making it unique to distinguish Org-generated line breaks.
2. Establish a regexp criterion when it is safe to strip
   "[0pt]%__ORG_EXPORT__" from the page break (I am still unsure what
   will be the most reliable criteria here). If not safe, we just strip
   the "%__ORG_EXPORT__" part.
3. Write a filter that replaces the above from final output.
   Note: If we go __ORG_EXPORT__ way, it might be safe to leave the
   filter on by default as long as we can find a really robust criteria
   on when it is safe to drop [0pt] part.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Line breaks and brackets in LaTeX export

[Juan Manuel Macías]

Max Nikulin writes:

> LaTeX verse package require "\\!" as stanza separator to get proper line count,
> so square bracket on the next line is not the only character that may change
> meaning of "\\". So "[*!>" (depending on context) should be handled.

Only when the separation between stanzas is made by means of a blank
line. With the current behavior of verse blocks in the export to LaTeX,
where the separation between stanzas is done by a \vspace, there is no
problem.

When, months ago, I contributed the LaTeX verse numbering patch, I took
this into account, and it didn't seem like the right time to change the
default behavior of the verse block.

I hope to be able to send a new patch soon in the few days (according to
what we discussed in past messages) where the separation by stanzas is
applied by means of a blank line (which is the right thing to do in
LaTeX). I'm rewriting my old code for verse numbering so that a \\! is
automatically added in the last line of each stanza when line numbering
is on.

Best regards,

Juan Manuel