The less ambiguous math delimiters in tables

The less ambiguous math delimiters in tables

[Rudolf Adamkovič]

We know that

  \(...\) math delimiters are superior to $...$

in that

  they remove parsing ambiguities.

So, I updated my notes to the new delimiters, but

  Org still struggles with | within \(...\) in tables,

such as

  | \(|x|\) |

Is this a feature or a bug?

I was hoping to reap the benefits of \(...\) in this case.

P.S. I know about '\vert' in LaTeX.

Rudy
-- 
"Programming reliably -- must be an activity of an undeniably
mathematical nature […] You see, mathematics is about thinking, and
doing mathematics is always trying to think as well as possible."
--- Edsger W. Dijkstra, 1981

Rudolf Adamkovič <rudolf@adamkovic.org> [he/him]
http://adamkovic.org

Re: The less ambiguous math delimiters in tables

[Ihor Radchenko]

Rudolf Adamkovič <rudolf@adamkovic.org> writes:
>   Org still struggles with | within \(...\) in tables,
>
> such as
>
>   | \(|x|\) |
>
> Is this a feature or a bug?

It is a syntax limitation.
Org parser is outer-inner - the table row is parsed first.
So, | are unconditionally used as table delimiters, *before* verbatim
LaTeX markup is recognized.

\vert is one possible workaround, but it does not work inside verbatim
text.

The only sane way would be adding some kind of alternative table
delimiter syntax to aid the situations like in your example.

-- 
Ihor Radchenko // yantar92,
Org mode maintainer,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: The less ambiguous math delimiters in tables

[Rudolf Adamkovič]

Ihor Radchenko <yantar92@posteo.net> writes:

> It is a syntax limitation.
> Org parser is outer-inner - the table row is parsed first.
> So, | are unconditionally used as table delimiters, *before* verbatim
> LaTeX markup is recognized.

How is that not a parser bug?

I thought that, with the noisy \(...\) delimiters,

  the parser finally has enough information

to not screw up.

P.S. It is impossible to write, e.g. \|, which is vector length (norm).

Rudy
-- 
"I would prefer an intelligent hell to a stupid paradise."
--- Blaise Pascal

Rudolf Adamkovič <rudolf@adamkovic.org> [he/him]
http://adamkovic.org

Re: The less ambiguous math delimiters in tables

[Ihor Radchenko]

Rudolf Adamkovič <rudolf@adamkovic.org> writes:

>> It is a syntax limitation.
>> Org parser is outer-inner - the table row is parsed first.
>> So, | are unconditionally used as table delimiters, *before* verbatim
>> LaTeX markup is recognized.
>
> How is that not a parser bug?
>
> I thought that, with the noisy \(...\) delimiters,
>
>   the parser finally has enough information
>
> to not screw up.

Your example shows ambiguous markup that can be
interpreted in multiple ways:

1. <begin cell> \(<end cell><begin cell>x<end cell><begin cell>\) <end cell>
2. <begin cell> <begin latex>|x|<end latex> <end cell>

Org parser chooses one. It has to choose some.
Org parser also chooses a simpler interpretation that does not require
backtracking.

> P.S. It is impossible to write, e.g. \|, which is vector length (norm).

Yes, it is impossible. That's why I call it markup limitation.
We need to improve the markup to address this known problematic scenario.

-- 
Ihor Radchenko // yantar92,
Org mode maintainer,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: The less ambiguous math delimiters in tables

[Rudolf Adamkovič]

Ihor Radchenko <yantar92@posteo.net> writes:

> Your example shows ambiguous markup that can be
> interpreted in multiple ways:
>
> 1. <begin cell> \(<end cell><begin cell>x<end cell><begin cell>\) <end cell>
> 2. <begin cell> <begin latex>|x|<end latex> <end cell>
>
> Org parser chooses one. It has to choose some.
> Org parser also chooses a simpler interpretation that does not require
> backtracking.

But (2) is a *much, much, much* better choice (for the user).

How often does a table row contain

  - a sole '\(` in one cell and
  - a sole '\)` in another cell?

Virtually never [excluding verbatim/code, which is a different problem.]

How often does LaTeX include `|'?  Often:

  - absolute value,
  - parallel lines,
  - vector length,
  - set cardinality,
  - various norms,

on and on.

The use starts at elementary school level, so

  it is like Org reserving the minus sign in LaTeX for itself,

which would be a similar usability disaster.

Rudy
-- 
"It is no paradox to say that in our most theoretical moods we may be
nearest to our most practical applications."  --- Alfred North
Whitehead, 1861-1947

Rudolf Adamkovič <rudolf@adamkovic.org> [he/him]
http://adamkovic.org

Re: The less ambiguous math delimiters in tables

[Ihor Radchenko]

Rudolf Adamkovič <rudolf@adamkovic.org> writes:

>> 1. <begin cell> \(<end cell><begin cell>x<end cell><begin cell>\) <end cell>
>> 2. <begin cell> <begin latex>|x|<end latex> <end cell>
>>
>> Org parser chooses one. It has to choose some.
>> Org parser also chooses a simpler interpretation that does not require
>> backtracking.
>
> But (2) is a *much, much, much* better choice (for the user).

Maybe, but it is also much more complex in terms of parser.
Backtracking will introduce non-linear complexity to the parser,
degrading the performance significantly. It will also make Org syntax
much, much harder in more complex cases - there will still be
ambiguities when you have more than 2 interpretations: e.g.

 | \(|x|\) | \(|x|\) |

this one has 3 possibilities:

1. <cell> \(</cell><cell>x...
2. <cell> <latex>|x|\) | \(|x|</latex> </cell>
3. <cell> <latex>|x|</latex> </cell><cell> <latex>|x|</latex> </cell>

And there will be similar situations with even more possibilities. In
fact, the number of ambiguous alternatives can blow up pretty quickly
when the text is complex enough combination of literal and non-literal
markups.

In any case, the way Org parser works in this example is one of the most
fundamental design decisions in the Org markup. We cannot change it at
this point without breaking all the historical documents + third-party
parsers. That's why I am talking about providing markup extension to
address the issue rather than altering the existing parser fundamentals.

-- 
Ihor Radchenko // yantar92,
Org mode maintainer,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: The less ambiguous math delimiters in tables

[Rudolf Adamkovič]

Ihor Radchenko <yantar92@posteo.net> writes:

> Maybe, but it is also much more complex in terms of parser.
> Backtracking will introduce non-linear complexity to the parser,
> degrading the performance significantly.

Is that so?  I thought it is all about simple precedence rules.  In this
case, once the parser finds the opening \(, it interprets everything as
LaTeX, until it finds the closing \).

> It will also make Org syntax much, much harder in more complex cases -
> there will still be ambiguities when you have more than 2
> interpretations: e.g.
>
>  | \(|x|\) | \(|x|\) |
>
> this one has 3 possibilities:
>
> 1. <cell> \(</cell><cell>x...
> 2. <cell> <latex>|x|\) | \(|x|</latex> </cell>
> 3. <cell> <latex>|x|</latex> </cell><cell> <latex>|x|</latex> </cell>

Again, if \(...\) has a higher precedence than the table |, then this is
not a problem.  There is no ambiguity, right?

> We cannot change it at this point without breaking all the historical
> documents + third-party parsers. That's why I am talking about
> providing markup extension to address the issue rather than altering
> the existing parser fundamentals.

It would only break the documents that have one-sided \( or \) in the
cells of the same column, no?  And that is ... virtually never?

Rudy
-- 
"I do not fear death.  I had been dead for billions and billions of years
before I was born, and had not suffered the slightest inconvenience from it."
--- Mark Twain, paraphrased

Rudolf Adamkovič <rudolf@adamkovic.org> [he/him]
http://adamkovic.org

Re: The less ambiguous math delimiters in tables

[Ihor Radchenko]

Rudolf Adamkovič <rudolf@adamkovic.org> writes:

>> Maybe, but it is also much more complex in terms of parser.
>> Backtracking will introduce non-linear complexity to the parser,
>> degrading the performance significantly.
>
> Is that so?  I thought it is all about simple precedence rules.  In this
> case, once the parser finds the opening \(, it interprets everything as
> LaTeX, until it finds the closing \).

Yes, it is. Please check how Org parser works in
`org-element--parse-elements' and `org-element--parse-objects'.

>> We cannot change it at this point without breaking all the historical
>> documents + third-party parsers. That's why I am talking about
>> providing markup extension to address the issue rather than altering
>> the existing parser fundamentals.
>
> It would only break the documents that have one-sided \( or \) in the
> cells of the same column, no?  And that is ... virtually never?

In a theoretical case if we agree to what you are suggesting, it should
not be just for tables. There are similar cases with other markup, like

 *foo =* *= bar*

-- 
Ihor Radchenko // yantar92,
Org mode maintainer,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: The less ambiguous math delimiters in tables

[Rudolf Adamkovič]

Ihor Radchenko <yantar92@posteo.net> writes:

> In a theoretical case if we agree to what you are suggesting, it should
> not be just for tables. There are similar cases with other markup, like
>
>  *foo =* *= bar*

Agreed!  We could introduce a kind of escaping that means "this MUST be
interpreted as markup" and/or "this MUST NOT be interpreted as markup",
but that could lead to documents that are hard to read for humans.  Or,
we could add structured markup that is unambiguous and takes precedence
over all unstructured markup.  For example:

  emphasis{...}
  verbatim{...}
  table[...]{...}
  src[...]{...}

The last one already exists. :)

That said, as for my original problem, I still think that \(...\) should
take precedence over |.  Even if we added structured latex{...} markup,
it should not be necessary in my case, as Org should not severely break
basic LaTeX within tables in the first place.

Rudy
-- 
"All you have to do is write one true sentence.  Write the truest
sentence that you know."  --- Ernest Miller Hemingway (1899-1961)

Rudolf Adamkovič <rudolf@adamkovic.org> [he/him]
http://adamkovic.org

Re: The less ambiguous math delimiters in tables

[Ihor Radchenko]

Rudolf Adamkovič <rudolf@adamkovic.org> writes:

>> In a theoretical case if we agree to what you are suggesting, it should
>> not be just for tables. There are similar cases with other markup, like
>>
>>  *foo =* *= bar*
>
> Agreed!  We could introduce a kind of escaping that means "this MUST be
> interpreted as markup" and/or "this MUST NOT be interpreted as markup",
> but that could lead to documents that are hard to read for humans.  Or,
> we could add structured markup that is unambiguous and takes precedence
> over all unstructured markup.  For example:
>
>   emphasis{...}
>   verbatim{...}
>   table[...]{...}
>   src[...]{...}
>
> The last one already exists. :)

See https://list.orgmode.org/875xwqj4tl.fsf@localhost/

> That said, as for my original problem, I still think that \(...\) should
> take precedence over |.  Even if we added structured latex{...} markup,
> it should not be necessary in my case, as Org should not severely break
> basic LaTeX within tables in the first place.

Sorry, but no.
Basically, what you propose is a rabbit hole that will introduce new
parser bugs and, worse, new systematic problems with syntax.

For context, I proposed similar ideas to Nicolas, the author of
org-element parser, in the past, and he rejected them firmly.

FYI, my approach to solve this problem is different - I want
(eventually) to allow some kind of alternative syntax for tables that
will allow bypassing similar situations. For example, we can allow
multiple || to serve as delimiters:

|  this |  is |  a     |  normal       |  table  |  row |
|| here || we || allow || verbatim "|" || inside ||    ||

-- 
Ihor Radchenko // yantar92,
Org mode maintainer,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: The less ambiguous math delimiters in tables

[Leo Butler]

On Wed, Dec 25 2024, Rudolf Adamkovič <rudolf@adamkovic.org> wrote:

> Ihor Radchenko <yantar92@posteo.net> writes:
>
>> It is a syntax limitation.
>> Org parser is outer-inner - the table row is parsed first.
>> So, | are unconditionally used as table delimiters, *before* verbatim
>> LaTeX markup is recognized.
>
> How is that not a parser bug?
>
> I thought that, with the noisy \(...\) delimiters,
>
>   the parser finally has enough information
>
> to not screw up.
>
> P.S. It is impossible to write, e.g. \|, which is vector length (norm).

Rudy,
Since you are using Org -> LaTeX, I suggest using something like the
following:

| \([x]\) |

Depending on the flavour of LaTeX you use, you can use a package to
typeset the left/right square brackets as |, e.g.

\newunicodechar{[}{|}
\newunicodechar{]}{|}

Leo

Re: The less ambiguous math delimiters in tables

[Rudolf Adamkovič]

Rudolf Adamkovič <rudolf@adamkovic.org> writes:

> P.S. I know about '\vert' in LaTeX.

But, I was not able to figure out how to put

  \(\|\vec{u}\|\)

into a table.

[The \| sequence typesets as || but with different spacing than \vert\vert.]

Rudy
-- 
"Logic is a science of the necessary laws of thought, without which no
employment of the understanding and the reason takes place."
--- Immanuel Kant, 1785

Rudolf Adamkovič <rudolf@adamkovic.org> [he/him]
http://adamkovic.org