bug#71364: Fix Table.el export

bug#71364: Fix Table.el export

[Pranshu]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: table.diff --]
[-- Type: text/x-diff, Size: 3649 bytes --]

diff --git a/lisp/textmodes/table.el b/lisp/textmodes/table.el
index 19c6a8d7c4..28401084a6 100644
--- a/lisp/textmodes/table.el
+++ b/lisp/textmodes/table.el
@@ -697,6 +697,19 @@ table-word-continuation-char
   :type 'character
   :group 'table)
 
+(defcustom table-source-latex-escape-characters (cons "[#$~_^%{}&]" t)
+  "A cons cell containing which charecters to escape in the latex source
+of `table-generate-source'.  The head of the list, if non-nil contains a
+regexp that matches all text that is to be adding a preceding backslash
+to the matching text. If nil, no non-backslash charecters will be
+escaped.  The tail, if non-nil, escapes all the backslashes in the latex
+source."
+  :tag "Source Latex Escape Charecters Regexp"
+  :type '(cons (radio (regexp :tag "regexp")
+                        (const :tag "Off" nil))
+               boolean)
+  :group 'table)
+
 (defcustom table-detect-cell-alignment t
   "Detect cell contents alignment automatically.
 When non-nil cell alignment is automatically determined by the
@@ -3264,19 +3277,29 @@ table--generate-source-scan-lines
                   (lambda (from to)
                     (let ((line (table--buffer-substring-and-trim
                                  (table--goto-coordinate (cons from y))
-                                 (table--goto-coordinate (cons to y)))))
+                                 (table--goto-coordinate (cons to y))))
+                          (escape-char-reg
+                           (apply 'concat (append (and (car table-source-latex-escape-characters)
+                                               (list "\\("
+                                                     (car table-source-latex-escape-characters)
+                                                     "\\)"
+                                                     (and (cdr table-source-latex-escape-characters)
+                                                          "\\|")))
+                                   (and (cdr table-source-latex-escape-characters)
+                                        (list"\\(\\\\\\)"))))))
                       ;; escape special characters
                       (with-temp-buffer
                         (insert line)
                         (goto-char (point-min))
-                        (while (re-search-forward "\\([#$~_^%{}&]\\)\\|\\(\\\\\\)\\|\\([<>|]\\)" nil t)
-                          (if (match-beginning 1)
-                              (save-excursion
-                                (goto-char (match-beginning 1))
-                                (insert  "\\"))
-                            (if (match-beginning 2)
-                                (replace-match "$\\backslash$" t t)
-                              (replace-match (concat "$" (match-string 3) "$")) t t)))
+                        (when (or (car table-source-latex-escape-characters)
+                                  (cdr table-source-latex-escape-characters))
+                          (while (re-search-forward escape-char-reg nil t)
+                            (if (and (car table-source-latex-escape-characters)
+                                     (match-beginning 1))
+                                    (save-excursion
+                                      (goto-char (match-beginning 1))
+                                      (insert  "\\"))
+                              (replace-match "$\\backslash$" t t))))
                         (setq line (buffer-substring (point-min) (point-max))))
                       ;; insert a column separator and column/multicolumn contents
                       (with-current-buffer dest-buffer

[-- Attachment #2: Type: text/plain, Size: 262 bytes --]


Hello all,

When table.el exports to latex, charecters like '$' are all escaped.
This makes table.el tables practically useless for any sorts of latex
export.

Attached is a diff that adds a variable that controls what should be
escaped.

Kind Regards,
Pranshu

bug#71364: Fix Table.el export

[Eli Zaretskii]

> Cc: 71364@debbugs.gnu.org
> From: Pranshu <pranshusharma366@gmail.com>
> Date: Tue, 04 Jun 2024 21:22:00 +1000
> 
> When table.el exports to latex, charecters like '$' are all escaped.
> This makes table.el tables practically useless for any sorts of latex
> export.
> 
> Attached is a diff that adds a variable that controls what should be
> escaped.

Could you please explain in some more details the problem and the
solution?  I'm afraid I couldn't follow your logic, perhaps because
I'm not a frequent user of LaTeX.

Thanks.

bug#71364: Fix Table.el export

[Pranshu]

[1. application/pdf; thing.pdf]...


All good, basiclly in latex a equation is formatted in something like:
$\Delta Q = r^2$
Will become something like: "ΔQ = r²"

The key takeway is the dollar symbol is needed to tell the latex
compiler that it is typesetting maths.

Now, the problem is if you have a table like the following:
+---------+-------+--------+
|$\delta$ |x      |y       |
+---------+-------+--------+
|x        | $x^2$ |$xy$    |
+---------+-------+--------+
|y        | $xy$  | $y^2$  |
+---------+-------+--------+

Now if you export to latex using "C-^" or M-x table-generate-source you
get:

% This LaTeX table template is generated by emacs 30.0.50
\begin{tabular}{|l|l|l|}
\hline
\$$\backslash$delta\$ & x & y \\
\hline
x & \$x\^2\$ & \$xy\$ \\
\hline
y & \$xy\$ & \$y\^2\$ \\
\hline
\end{tabular}

The problem with this is that since the all the dollar and caret signs
are all escaped, the maths will not render.  The rendered pdf from the
above latex is attached, you can see that the equations were not
rendered properly.  This is because the dollar signs and carets were
automaticly escaped

What the diff does, is that it adds a variable that allows the user to
change what will be escaped.  Also as you can see in the above latex
table, the backslash had to be escaped in a different way, so using one
regexp variable would not be enough.  That is why the solution contains
a variable called 'table-source-latex-escape-characters', which is a
cons cell with the documentation:

A cons cell containing which charecters to escape in the latex source
of ‘table-generate-source’.  The head of the list, if non-nil contains a
regexp that matches all text that is to be adding a preceding backslash
to the matching text. If nil, no non-backslash charecters will be
escaped.  The tail, if non-nil, escapes all the backslashes in the latex
source.

Kind Regards
Pranshu

Eli Zaretskii <eliz@gnu.org> writes:

>> Cc: 71364@debbugs.gnu.org
>> From: Pranshu <pranshusharma366@gmail.com>
>> Date: Tue, 04 Jun 2024 21:22:00 +1000
>> 
>> When table.el exports to latex, charecters like '$' are all escaped.
>> This makes table.el tables practically useless for any sorts of latex
>> export.
>> 
>> Attached is a diff that adds a variable that controls what should be
>> escaped.
>
> Could you please explain in some more details the problem and the
> solution?  I'm afraid I couldn't follow your logic, perhaps because
> I'm not a frequent user of LaTeX.
>
> Thanks.

bug#71364: Fix Table.el export

[Eli Zaretskii]

> From: Pranshu <pranshusharma366@gmail.com>
> Cc: 71364@debbugs.gnu.org
> Date: Fri, 07 Jun 2024 17:00:13 +1000
> 
> Now, the problem is if you have a table like the following:
> +---------+-------+--------+
> |$\delta$ |x      |y       |
> +---------+-------+--------+
> |x        | $x^2$ |$xy$    |
> +---------+-------+--------+
> |y        | $xy$  | $y^2$  |
> +---------+-------+--------+
> 
> Now if you export to latex using "C-^" or M-x table-generate-source you
> get:
> 
> % This LaTeX table template is generated by emacs 30.0.50
> \begin{tabular}{|l|l|l|}
> \hline
> \$$\backslash$delta\$ & x & y \\
> \hline
> x & \$x\^2\$ & \$xy\$ \\
> \hline
> y & \$xy\$ & \$y\^2\$ \\
> \hline
> \end{tabular}
> 
> The problem with this is that since the all the dollar and caret signs
> are all escaped, the maths will not render.

So you are saying that exporting to latex is basically completely
broken in the current code?  Is that command use to export only to
latex, or is the fact that the export is to latex is just one of the
possibilities?

> What the diff does, is that it adds a variable that allows the user to
> change what will be escaped.  Also as you can see in the above latex
> table, the backslash had to be escaped in a different way, so using one
> regexp variable would not be enough.  That is why the solution contains
> a variable called 'table-source-latex-escape-characters', which is a
> cons cell with the documentation:

Why would a user need to customize this on the level of characters?
Shouldn't there be a single boolean that causes latex-specific
characters to be escape or not to be escaped, all of them or none?

> A cons cell containing which charecters to escape in the latex source
> of ‘table-generate-source’.  The head of the list, if non-nil contains a
> regexp that matches all text that is to be adding a preceding backslash
> to the matching text. If nil, no non-backslash charecters will be
> escaped.  The tail, if non-nil, escapes all the backslashes in the latex
> source.

What is the purpose of the "tail", i.e. why would a user want to
escape all the backslashes?

Thanks.

bug#71364: Fix Table.el export

[Pranshu]

> So you are saying that exporting to latex is basically completely
> broken in the current code?  Is that command use to export only to
> latex, or is the fact that the export is to latex is just one of the
> possibilities?

Yes.  The function that does the escaping is
`table--generate-source-scan-lines', which is a misleading name because
it is only called by table-generate-source when the language is exported
to latex.

>> What the diff does, is that it adds a variable that allows the user to
>> change what will be escaped.  Also as you can see in the above latex
>> table, the backslash had to be escaped in a different way, so using one
>> regexp variable would not be enough.  That is why the solution contains
>> a variable called 'table-source-latex-escape-characters', which is a
>> cons cell with the documentation:
>
> Why would a user need to customize this on the level of characters?
> Shouldn't there be a single boolean that causes latex-specific
> characters to be escape or not to be escaped, all of them or none?
>

The problem with that is that there are latex charecters such as '%',
which is comment, that will cause the whole table to break of not
escaped.

And since the variable is a regexp, it can possibly escape macros, such
as 'LaTeX' -> '\LaTeX'. 

>> A cons cell containing which charecters to escape in the latex source
>> of ‘table-generate-source’.  The head of the list, if non-nil contains a
>> regexp that matches all text that is to be adding a preceding backslash
>> to the matching text. If nil, no non-backslash charecters will be
>> escaped.  The tail, if non-nil, escapes all the backslashes in the latex
>> source.
>
> What is the purpose of the "tail", i.e. why would a user want to
> escape all the backslashes?

Mainly because some people might only use the latex export to turn their
table into a pdf, and escaping backslashes is not as simple as typing
two backslashes.  And also since it was the default behaviour, so those
who have already made tables will not need to escape all those
backslashes to make their documents exportable.

If you feel that is unnecessary, I can modify the diff not include
backslash escaping.

>
> Thanks.

bug#71364: Fix Table.el export

[Eli Zaretskii]

> From: Pranshu <pranshusharma366@gmail.com>
> Cc: 71364@debbugs.gnu.org
> Date: Sat, 08 Jun 2024 12:10:32 +1000
> 
> > So you are saying that exporting to latex is basically completely
> > broken in the current code?  Is that command use to export only to
> > latex, or is the fact that the export is to latex is just one of the
> > possibilities?
> 
> Yes.  The function that does the escaping is
> `table--generate-source-scan-lines', which is a misleading name because
> it is only called by table-generate-source when the language is exported
> to latex.

It is strange to hear that this is so basically broken.  This code
exists since more than 20 years ago, and several people contributed
changes to the LaTeX export, so I'd expect it to be at least somewhat
useful and working.

> >> What the diff does, is that it adds a variable that allows the user to
> >> change what will be escaped.  Also as you can see in the above latex
> >> table, the backslash had to be escaped in a different way, so using one
> >> regexp variable would not be enough.  That is why the solution contains
> >> a variable called 'table-source-latex-escape-characters', which is a
> >> cons cell with the documentation:
> >
> > Why would a user need to customize this on the level of characters?
> > Shouldn't there be a single boolean that causes latex-specific
> > characters to be escape or not to be escaped, all of them or none?
> >
> 
> The problem with that is that there are latex charecters such as '%',
> which is comment, that will cause the whole table to break of not
> escaped.

Is % the only character with such problems, or are there others?

> And since the variable is a regexp, it can possibly escape macros, such
> as 'LaTeX' -> '\LaTeX'. 

Not sure I understand: is it a Good Thing that it could escape macros,
or is it a Bad Thing (which should be avoided)?

More generally, given these tricky considerations, how would a user
determine which characters to include in the regexp?  If it's only by
examining the texts in the table cells, then it could be not very
practical, since a table could be very large.

> >> A cons cell containing which charecters to escape in the latex source
> >> of ‘table-generate-source’.  The head of the list, if non-nil contains a
> >> regexp that matches all text that is to be adding a preceding backslash
> >> to the matching text. If nil, no non-backslash charecters will be
> >> escaped.  The tail, if non-nil, escapes all the backslashes in the latex
> >> source.
> >
> > What is the purpose of the "tail", i.e. why would a user want to
> > escape all the backslashes?
> 
> Mainly because some people might only use the latex export to turn their
> table into a pdf, and escaping backslashes is not as simple as typing
> two backslashes.  And also since it was the default behaviour, so those
> who have already made tables will not need to escape all those
> backslashes to make their documents exportable.
> 
> If you feel that is unnecessary, I can modify the diff not include
> backslash escaping.

I think it should be controlled by a separate variable.

I also added to the discussion two people who in the past contributed
changes to table.el in the LaTeX-related areas, so they probably have
some experience with exporting to LaTeX, and could help us figure out
how best to handle this issue.

Reuben and Vladimir, would you please comment on the usability of
exporting to LaTeX in table.el, and on the proposed improvements?

Thanks.

bug#71364: Fix Table.el export

[Pranshu]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Pranshu <pranshusharma366@gmail.com>
>> Cc: 71364@debbugs.gnu.org
>> Date: Sat, 08 Jun 2024 12:10:32 +1000
>> 
>> > So you are saying that exporting to latex is basically completely
>> > broken in the current code?  Is that command use to export only to
>> > latex, or is the fact that the export is to latex is just one of the
>> > possibilities?
>> 
>> Yes.  The function that does the escaping is
>> `table--generate-source-scan-lines', which is a misleading name because
>> it is only called by table-generate-source when the language is exported
>> to latex.
>
> It is strange to hear that this is so basically broken.  This code
> exists since more than 20 years ago, and several people contributed
> changes to the LaTeX export, so I'd expect it to be at least somewhat
> useful and working.

Yes, initially I was surprised as well.  Initially I was surprised as
well.  I found 2 questions on stack overflow in which they also came to
the same dead end, of not being able to put equations in table(.el)s.

>> >> What the diff does, is that it adds a variable that allows the user to
>> >> change what will be escaped.  Also as you can see in the above latex
>> >> table, the backslash had to be escaped in a different way, so using one
>> >> regexp variable would not be enough.  That is why the solution contains
>> >> a variable called 'table-source-latex-escape-characters', which is a
>> >> cons cell with the documentation:
>> >
>> > Why would a user need to customize this on the level of characters?
>> > Shouldn't there be a single boolean that causes latex-specific
>> > characters to be escape or not to be escaped, all of them or none?
>> >
>> 
>> The problem with that is that there are latex charecters such as '%',
>> which is comment, that will cause the whole table to break of not
>> escaped.
>
> Is % the only character with such problems, or are there others?
>

These characters: "#$~_^%{}&", will all mess something up to different
extents.  For some it's mostly harmless as with the '~', but characters
like '&' will fully mess up the table, as it is used to separate the
columns.

>> And since the variable is a regexp, it can possibly escape macros, such
>> as 'LaTeX' -> '\LaTeX'. 
>
> Not sure I understand: is it a Good Thing that it could escape macros,
> or is it a Bad Thing (which should be avoided)?

Yes, it is a good thing.  "Escaping macros" was bad wording, I meant
turning words into macros.  Normally it will not do that, but it can be
customised to do that.

> More generally, given these tricky considerations, how would a user
> determine which characters to include in the regexp?  If it's only by
> examining the texts in the table cells, then it could be not very
> practical, since a table could be very large.
>
>> >
>> > What is the purpose of the "tail", i.e. why would a user want to
>> > escape all the backslashes?
>> 
>> Mainly because some people might only use the latex export to turn their
>> table into a pdf, and escaping backslashes is not as simple as typing
>> two backslashes.  And also since it was the default behaviour, so those
>> who have already made tables will not need to escape all those
>> backslashes to make their documents exportable.
>> 
>> If you feel that is unnecessary, I can modify the diff not include
>> backslash escaping.
>
> I think it should be controlled by a separate variable.
>

Yeah, you probably know better about that then I do.  I will just say
that variables about table.el are extremely hard to find, since of you
"C-h v table ?", you will see 100s of variables thanks to org-mode
tables.  Personally I would avoid creating more variables with 'table' in
their name, but as I said you know a lot more about emacs than I do.

> I also added to the discussion two people who in the past contributed
> changes to table.el in the LaTeX-related areas, so they probably have
> some experience with exporting to LaTeX, and could help us figure out
> how best to handle this issue.
>

all good, more perspectives could help.

bug#71364: Fix Table.el export

[Reuben Thomas via Bug reports for GNU Emacs, the Swiss army knife of text editors]

[-- Attachment #1: Type: text/plain, Size: 2726 bytes --]

On Sat, 8 Jun 2024 at 07:38, Eli Zaretskii <eliz@gnu.org> wrote:

>
> It is strange to hear that this is so basically broken.  This code
> exists since more than 20 years ago, and several people contributed
> changes to the LaTeX export, so I'd expect it to be at least somewhat
> useful and working.
>

I think the explanation is that it is somewhat useful and working, and also
basically broken.

Specifically, it works fine if, like me, you do not want to put LaTeX
markup in tables. This was certainly my case, as I was using it with
org-mode. In this case, any character that is active in LaTeX can and
should simply be escaped on output.

What is basically broken is the ability to export to LaTeX when the table
contains LaTeX markup.


> Reuben and Vladimir, would you please comment on the usability of
> exporting to LaTeX in table.el, and on the proposed improvements?
>

Personally, I think trying to make this work is a mug's game, and having
wrestled as a maintainer with other Emacs code that is over-complicated
because it tries to understand the syntax of e.g. LaTeX or mail buffers
(for example, ispell.el), I would resist adding further support for
specific syntaxes to code that is not part of an editing package for that
syntax.

Again, taking ispell.el, with which I am much more familiar, what I would
like to do there is to strip out all the support for LaTeX and mail
buffers, and instead require LaTeX- or mail-editing modes to call the
spell-checking routines. That would require a different approach globally:
some of the existing ispell commands work on a whole buffer, so they cannot
interact with language-specific code. I guess the code was originally
written like this because it could be created from scratch without changing
anything about other modes.

I think this example of ispell.el is relevant to table.el because much the
same considerations apply: table.el was originally written independent of
other code, but that's not a good design for interacting with other
syntaxes, as you end up implementing half-baked support for them, rather
than allowing an editing mode that already understands the syntax to
interact with the table routines.

Therefore, I recommend not attempting to improve table.el's LaTeX support,
documenting clearly that it doesn't support embedded LaTeX, and designing
functions that allow table.el to cooperate with other packages to export
different embedded syntaxes. Otherwise, we will frustrate users (who will
find the code broken and, if they try to understand it, baffling) and
maintainers (who will spend effort for many years maintaining code that is
half-baked, complicates the modules it lives in, but has to be maintained).

-- 
https://rrt.sc3d.org

[-- Attachment #2: Type: text/html, Size: 4679 bytes --]

bug#71364: Fix Table.el export

[Eli Zaretskii]

> From: Reuben Thomas <rrt@sc3d.org>
> Date: Sat, 8 Jun 2024 11:05:25 +0100
> Cc: Pranshu <pranshusharma366@gmail.com>, Vladimir Nikishkin <lockywolf@gmail.com>, 
> 	71364@debbugs.gnu.org
> 
> Specifically, it works fine if, like me, you do not want to put LaTeX markup in tables. This was certainly my
> case, as I was using it with org-mode. In this case, any character that is active in LaTeX can and should simply
> be escaped on output.
> 
> What is basically broken is the ability to export to LaTeX when the table contains LaTeX markup.

But the function being discussed is said to be specific to export to
LaTeX, so it should expect LaTeX markup in the cells, no?

> Again, taking ispell.el, with which I am much more familiar, what I would like to do there is to strip out all the
> support for LaTeX and mail buffers, and instead require LaTeX- or mail-editing modes to call the
> spell-checking routines. That would require a different approach globally: some of the existing ispell
> commands work on a whole buffer, so they cannot interact with language-specific code. I guess the code was
> originally written like this because it could be created from scratch without changing anything about other
> modes.
> 
> I think this example of ispell.el is relevant to table.el because much the same considerations apply: table.el
> was originally written independent of other code, but that's not a good design for interacting with other
> syntaxes, as you end up implementing half-baked support for them, rather than allowing an editing mode that
> already understands the syntax to interact with the table routines.
> 
> Therefore, I recommend not attempting to improve table.el's LaTeX support, documenting clearly that it
> doesn't support embedded LaTeX, and designing functions that allow table.el to cooperate with other
> packages to export different embedded syntaxes. Otherwise, we will frustrate users (who will find the code
> broken and, if they try to understand it, baffling) and maintainers (who will spend effort for many years
> maintaining code that is half-baked, complicates the modules it lives in, but has to be maintained).

Again, the function being discussed is specific to LaTeX, AFAIU, so
this is unlike the more-general issue of spell-checking an arbitrary
buffer.  It is more like a hypothetical command ispell-latex-buffer
(if it were to exist).  Don't you agree?

bug#71364: Fix Table.el export

[Reuben Thomas via Bug reports for GNU Emacs, the Swiss army knife of text editors]

[-- Attachment #1: Type: text/plain, Size: 1289 bytes --]

On Sat, 8 Jun 2024 at 11:51, Eli Zaretskii <eliz@gnu.org> wrote:

> > From: Reuben Thomas <rrt@sc3d.org>
> > Date: Sat, 8 Jun 2024 11:05:25 +0100
> > Cc: Pranshu <pranshusharma366@gmail.com>, Vladimir Nikishkin <
> lockywolf@gmail.com>,
> >       71364@debbugs.gnu.org
> >
> > Specifically, it works fine if, like me, you do not want to put LaTeX
> markup in tables. This was certainly my
> > case, as I was using it with org-mode. In this case, any character that
> is active in LaTeX can and should simply
> > be escaped on output.
> >
> > What is basically broken is the ability to export to LaTeX when the
> table contains LaTeX markup.
>
> But the function being discussed is said to be specific to export to
> LaTeX, so it should expect LaTeX markup in the cells, no?
>

No, it is supposed to export arbitrary text as valid LaTeX. At least, that
is my understanding.

Again, the function being discussed is specific to LaTeX, AFAIU, so
> this is unlike the more-general issue of spell-checking an arbitrary
> buffer.  It is more like a hypothetical command ispell-latex-buffer
> (if it were to exist).  Don't you agree?
>

The point I was trying to make is that LaTeX-specific code belongs in
LaTeX-specific modes and modules, not in ispell.el or table.el.

-- 
https://rrt.sc3d.org

[-- Attachment #2: Type: text/html, Size: 2488 bytes --]

bug#71364: Fix Table.el export

[Eli Zaretskii]

> From: Reuben Thomas <rrt@sc3d.org>
> Date: Sat, 8 Jun 2024 11:59:52 +0100
> Cc: pranshusharma366@gmail.com, lockywolf@gmail.com, 71364@debbugs.gnu.org
> 
> On Sat, 8 Jun 2024 at 11:51, Eli Zaretskii <eliz@gnu.org> wrote:
> 
>  > From: Reuben Thomas <rrt@sc3d.org>
>  > Date: Sat, 8 Jun 2024 11:05:25 +0100
>  > Cc: Pranshu <pranshusharma366@gmail.com>, Vladimir Nikishkin <lockywolf@gmail.com>, 
>  >       71364@debbugs.gnu.org
>  > 
>  > Specifically, it works fine if, like me, you do not want to put LaTeX markup in tables. This was certainly
>  my
>  > case, as I was using it with org-mode. In this case, any character that is active in LaTeX can and
>  should simply
>  > be escaped on output.
>  > 
>  > What is basically broken is the ability to export to LaTeX when the table contains LaTeX markup.
> 
>  But the function being discussed is said to be specific to export to
>  LaTeX, so it should expect LaTeX markup in the cells, no?
> 
> No, it is supposed to export arbitrary text as valid LaTeX. At least, that is my understanding.

So it's a YES, because I was saying the same thing.  Or maybe we
disagree about whether a function written for exporting to LaTeX
should or should not expect LaTeX markup.

>  Again, the function being discussed is specific to LaTeX, AFAIU, so
>  this is unlike the more-general issue of spell-checking an arbitrary
>  buffer.  It is more like a hypothetical command ispell-latex-buffer
>  (if it were to exist).  Don't you agree?
> 
> The point I was trying to make is that LaTeX-specific code belongs in LaTeX-specific modes and modules, not
> in ispell.el or table.el.

I don't think I agree with such a radical approach.  From where I
stand, there's nothing wrong with having LaTeX-specific features in
modes that are more general.

In any case, this is beyond the immediate issues discussed in this bug
report, I think.  Broken code must be fixed.

bug#71364: Fix Table.el export

[Reuben Thomas via Bug reports for GNU Emacs, the Swiss army knife of text editors]

[-- Attachment #1: Type: text/plain, Size: 1486 bytes --]

On Sat, 8 Jun 2024 at 12:19, Eli Zaretskii <eliz@gnu.org> wrote:

> > From: Reuben Thomas <rrt@sc3d.org>
> > Date: Sat, 8 Jun 2024 11:59:52 +0100
> > Cc: pranshusharma366@gmail.com, lockywolf@gmail.com,
> 71364@debbugs.gnu.org
> >
> > On Sat, 8 Jun 2024 at 11:51, Eli Zaretskii <eliz@gnu.org> wrote:
> >
> >  > From: Reuben Thomas <rrt@sc3d.org>
> >  > Date: Sat, 8 Jun 2024 11:05:25 +0100
> >  > Cc: Pranshu <pranshusharma366@gmail.com>, Vladimir Nikishkin <
> lockywolf@gmail.com>,
> >  >       71364@debbugs.gnu.org
> >  >
> >  > Specifically, it works fine if, like me, you do not want to put LaTeX
> markup in tables. This was certainly
> >  my
> >  > case, as I was using it with org-mode. In this case, any character
> that is active in LaTeX can and
> >  should simply
> >  > be escaped on output.
> >  >
> >  > What is basically broken is the ability to export to LaTeX when the
> table contains LaTeX markup.
> >
> >  But the function being discussed is said to be specific to export to
> >  LaTeX, so it should expect LaTeX markup in the cells, no?
> >
> > No, it is supposed to export arbitrary text as valid LaTeX. At least,
> that is my understanding.
>
> So it's a YES, because I was saying the same thing.  Or maybe we
> disagree about whether a function written for exporting to LaTeX
> should or should not expect LaTeX markup.
>

It looks like I was wrong: `table--generate-source-scan-lines` is expecting
LaTeX input.

Good luck with that!

-- 
https://rrt.sc3d.org

[-- Attachment #2: Type: text/html, Size: 3117 bytes --]

bug#71364: Fix Table.el export

[Pranshu]

Now that this is sorted, do you want me to send an updated diff with a
seperate variable for backslash?

bug#71364: Fix Table.el export

[Eli Zaretskii]

> From: Pranshu <pranshusharma366@gmail.com>
> Cc: Reuben Thomas <rrt@sc3d.org>,  lockywolf@gmail.com,  71364@debbugs.gnu.org
> Date: Sun, 09 Jun 2024 14:33:28 +1000
> 
> 
> Now that this is sorted, do you want me to send an updated diff with a
> seperate variable for backslash?

It isn't sorted yet, so please wait.

bug#71364: Fix Table.el export

[Pranshu]

[-- Attachment #1: Type: text/plain, Size: 414 bytes --]

no problem

On Sun, 9 Jun 2024 at 18:16, Eli Zaretskii <eliz@gnu.org> wrote:

> > From: Pranshu <pranshusharma366@gmail.com>
> > Cc: Reuben Thomas <rrt@sc3d.org>,  lockywolf@gmail.com,
> 71364@debbugs.gnu.org
> > Date: Sun, 09 Jun 2024 14:33:28 +1000
> >
> >
> > Now that this is sorted, do you want me to send an updated diff with a
> > seperate variable for backslash?
>
> It isn't sorted yet, so please wait.
>

[-- Attachment #2: Type: text/html, Size: 982 bytes --]

bug#71364: Fix Table.el export

[Pranshu]

[-- Attachment #1: Type: text/plain, Size: 444 bytes --]

How about now, it has been about 2 weeks

On Sun, 9 Jun 2024 at 18:16, Eli Zaretskii <eliz@gnu.org> wrote:

> > From: Pranshu <pranshusharma366@gmail.com>
> > Cc: Reuben Thomas <rrt@sc3d.org>,  lockywolf@gmail.com,
> 71364@debbugs.gnu.org
> > Date: Sun, 09 Jun 2024 14:33:28 +1000
> >
> >
> > Now that this is sorted, do you want me to send an updated diff with a
> > seperate variable for backslash?
>
> It isn't sorted yet, so please wait.
>

[-- Attachment #2: Type: text/html, Size: 1012 bytes --]

bug#71364: Fix Table.el export

[Eli Zaretskii]

> From: Pranshu <pranshusharma366@gmail.com>
> Date: Fri, 21 Jun 2024 18:25:46 +1000
> Cc: rrt@sc3d.org, lockywolf@gmail.com, 71364@debbugs.gnu.org
> 
> How about now, it has been about 2 weeks

I'd like to hear from Vladimir as well.

Vladimir, would you please chime in and comment on this proposal?

bug#71364: Fix Table.el export

[Eli Zaretskii]

Ping!  Vladimir, could you please respond?

> Cc: 71364@debbugs.gnu.org, rrt@sc3d.org
> Date: Sat, 22 Jun 2024 12:39:58 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> 
> > From: Pranshu <pranshusharma366@gmail.com>
> > Date: Fri, 21 Jun 2024 18:25:46 +1000
> > Cc: rrt@sc3d.org, lockywolf@gmail.com, 71364@debbugs.gnu.org
> > 
> > How about now, it has been about 2 weeks
> 
> I'd like to hear from Vladimir as well.
> 
> Vladimir, would you please chime in and comment on this proposal?
> 
> 
> 
> 

bug#71364: Fix Table.el export

[Lockywolf Laptop]

[-- Attachment #1: Type: text/plain, Size: 5066 bytes --]


Apologies for a delay. My life has been a little hectic recently.

While I have, indeed, contributed a bit to the table.el's support for
latex, I can't say that I am a big guru either on table.el or on latex.

I used table.el instead of native org tables because table.el supports
multiline cells, which I needed to write parallel texts in several
languages, naturally having different number of lines per passage.

So it is, indeed, "somewhat useful and working" for people not using
mathematics.

I have never tried writing latex code into table cells, and although I
agree that multiline tables might be useful with support for
mathematics, I also quite agree that making LaTeX work as expected is
hard, even when not using language-to-language transformations.

I can see why everything was escaped in the past, because it is the
easiest way to make sure that "everything at least compiles" even if the
result is not what one would expect naively.

(As a side-note, using the dollar signs in LaTeX mathematics is a bad
practice, because identical delimiters serving different roles when in
an even or odd position easily confuse the parser. Using \(\) and \[\]
is strongly preferred.)

>So it's a YES, because I was saying the same thing.

If you look at the const called table-source-languages, it should imply
that there "should" be export to languages other than LaTeX, but as
already mentioned, table--generate-source-scan-lines does not support
any language other than LaTeX, because nobody implemented, I guess.

>should or should not expect LaTeX markup

How much of LaTeX markup?
What if you add some mysterious \makeatletter there?

I generally agree with Reuben that making this work is a thing hard to
get right.

Moreover, as I mentioned above, I only used table.el with org-mode, but
I can't remember for sure what happens if one uses org-links in the
table.el cells. Do they become latex href links? Or links to images? Do
they turn into \includegraphics? Latex math is part of org, so I
presume, it is expected to somehow work, but I am not sure how.

>In any case, this is beyond the immediate issues discussed in this bug
>report, I think.  Broken code must be fixed.

I don't think it is a bug. It is an intentional limitation of scope,
which implements a crude but simple way of producing valid outputs,
albeit not 100% functional.

However I do not want to outright recommend rejection of the patch.
After all, implementing "simple math" is a valid desire for someone who
wants to print a table of integrals.

If I may, I would ask Mr Pranshu to spend a bit more work on this patch,
and do some straightforward, albeit a little tedious work to make this
patch more pleasant. May I?

1. Add a check to table--generate-source-scan-lines, which would
actually check that the variable _language equals to latex, and bail out
if it is not. Silently doing export to latex if html output is requested
is even worse than politely refusing.
2. Implement a defcustom table-el-cell-export-function-latex, which would be
called on each cell.
3. Implement a default value for this function, which should basically
amount to factoring out the code already in
table--generate-source-scan-lines. This way nothing will break.
This function maybe called table-el-cell-export-latex-default
3. Implement a copy of this function, with support for additional
escaping tricks needed by Mr Pranshu. It can be called
table-el-cell-export-latex. I do not think that a special defcustom
for the regexp is needed, because such export functions can be expected
to be written by people who need them necessarily in an ad-hoc way, and
customisations might not be well described by just a regexp.

It is a bit of work, but at least it will add some necessary extension
points. And org-mode, when exporting to table.el used in org files,
could override that cell export function to its own benefit.

(By the way, I haven't checked, but probably org-mode already has some
additional support for exporting table.el? I remember successfully
exporting org-files with table.el tables into html, which is seemingly
not supported in table.el itself.)

P.S.

Thank you Reuben for workign on ispell.el.
I think that Emacs would benefit from improved spellchecking, but I
myself didn't have time to explore it with sufficient assiduity.

Eli Zaretskii <eliz@gnu.org> writes:

> Ping!  Vladimir, could you please respond?
>
>> Cc: 71364@debbugs.gnu.org, rrt@sc3d.org
>> Date: Sat, 22 Jun 2024 12:39:58 +0300
>> From: Eli Zaretskii <eliz@gnu.org>
>> 
>> > From: Pranshu <pranshusharma366@gmail.com>
>> > Date: Fri, 21 Jun 2024 18:25:46 +1000
>> > Cc: rrt@sc3d.org, lockywolf@gmail.com, 71364@debbugs.gnu.org
>> > 
>> > How about now, it has been about 2 weeks
>> 
>> I'd like to hear from Vladimir as well.
>> 
>> Vladimir, would you please chime in and comment on this proposal?
>> 
>> 
>> 
>> 


-- 
Your sincerely,
Vladimir Nikishkin (MiEr, lockywolf)
(Laptop)

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

bug#71364: Fix Table.el export

[Jeremy Bryant via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Lockywolf Laptop <lockywolf@gmail.com> writes:

>>should or should not expect LaTeX markup
>
> How much of LaTeX markup?
> What if you add some mysterious \makeatletter there?
>
> I generally agree with Reuben that making this work is a thing hard to
> get right.
>
> Moreover, as I mentioned above, I only used table.el with org-mode, but
> I can't remember for sure what happens if one uses org-links in the
> table.el cells. Do they become latex href links? Or links to images? Do
> they turn into \includegraphics? Latex math is part of org, so I
> presume, it is expected to somehow work, but I am not sure how.
>
>>In any case, this is beyond the immediate issues discussed in this bug
>>report, I think.  Broken code must be fixed.
>
> I don't think it is a bug. It is an intentional limitation of scope,
> which implements a crude but simple way of producing valid outputs,
> albeit not 100% functional.
>

I concur - why is it a bug? (as opposed to a feature extension)
To write LaTeX documents, including with tables, there are dedicated
major modes.

On the other hand, table.el's Commentary states

;; This package provides text based table creation and editing
;; feature.  With this package Emacs is capable of editing tables that
;; are embedded inside a text document, the feature similar to the

The primary input appears plain text, with the export to LaTeX a
convenience.

So it appears that the problem is approached the wrong way around?

From this point of view how practical or desirable (feature duplication)
is it to allow the processing of a smaller or larger subset of LaTeX
within table.el?

bug#71364: Fix Table.el export

[Pranshu]

Hello Vladimir

Lockywolf Laptop <lockywolf@gmail.com> writes:

>>should or should not expect LaTeX markup
>
> How much of LaTeX markup?
> What if you add some mysterious \makeatletter there?
>

My idea was that the default should still be that all latex specific
charecters are escaped by default, and the variable is customized by
those who know what the specific function or markup does.

>
> If I may, I would ask Mr Pranshu to spend a bit more work on this patch,
> and do some straightforward, albeit a little tedious work to make this
> patch more pleasant. May I?
>

Yeah sure, but not rn.  I am kind of busy for 2 weeks, then I will be
able to do it.

> 1. Add a check to table--generate-source-scan-lines, which would
> actually check that the variable _language equals to latex, and bail out
> if it is not. Silently doing export to latex if html output is requested
> is even worse than politely refusing.

Is that really neccasary, because all that the function does it latex
specific. Why not just rename the function to something latex specific,
eg table--generate-source-scan-lines-latex.

> 2. Implement a defcustom table-el-cell-export-function-latex, which would be
> called on each cell.
> 3. Implement a default value for this function, which should basically
> amount to factoring out the code already in
> table--generate-source-scan-lines. This way nothing will break.
> This function maybe called table-el-cell-export-latex-default
> 3. Implement a copy of this function, with support for additional
> escaping tricks needed by Mr Pranshu. It can be called
> table-el-cell-export-latex. I do not think that a special defcustom
> for the regexp is needed, because such export functions can be expected
> to be written by people who need them necessarily in an ad-hoc way, and
> customisations might not be well described by just a regexp.
>
> It is a bit of work, but at least it will add some necessary extension
> points. And org-mode, when exporting to table.el used in org files,
> could override that cell export function to its own benefit.
>
> (By the way, I haven't checked, but probably org-mode already has some
> additional support for exporting table.el? I remember successfully
> exporting org-files with table.el tables into html, which is seemingly
> not supported in table.el itself.)

I haven't really played with the html org export, but as of now table.el
does have an html export.  All org mode does is calles the appropiate
export, so it does no sort checking text inside the table.  For example
you cannot do org itallics, citing, and any of that stuff inside the table.

To change this I think table.el would need a major rewrite that allows
for something like your idea of having a function called to all of the
text inside the table.

Instead of just improving the latex export and leaving the others in the
dust, how about we do what you suggested, but instead of having seperate
functions for each export, we make the exports a list.

All of the table exports basiclly do the same thing:
- Add some header text
- Replace column and row delimeters
- add end text
- optional: escape text



Honestly I think table.el is due for a rewrite, if you have used it
before yk what I mean, on average it's about half a second latency to
type one charecter.  Personally I think that instead of having a
table.el version of every basic command, eg self insert, kill line, it
would be better to add a post command hook and do damage control
depending on the last command.

Kind Regards,
Pranshu