Parsing nested markup objects

Parsing nested markup objects

[Laith Bahodi]

Hi! I'm working on a parser for org and I noticed something about
nested markup in the syntax spec: markup starting at the limit of
another object seems like it shouldn't be interpreted as markup. The
spec says that the precondition characters are:

PRE MARKER ...
PRE
  Either a whitespace character, -, (, {, ', ", or the beginning of a line.
https://orgmode.org/worg/dev/org-syntax-edited.html#Emphasis_Markers

With links, since `[` isn't in that list, the spec seems to imply the
following wouldn't contain an italic block, but it does:

[[https://example.com][/according to the spec, this shouldn't be marked up/]]

same goes for  */abc/* (since `*` isn't in the set defined by PRE)

I understand why it works like this since it's pretty convenient, but
I feel like the details about this interaction should probably be
clarified since it's not quite obvious from the spec (unless I missed
something!).

Also, how are these objects intended to be handled? I couldn't
pinpoint where in `org-element` this is approached.

Re: Parsing nested markup objects

[Ihor Radchenko]

[-- Attachment #1: Type: text/plain, Size: 871 bytes --]

Laith Bahodi <laithbahodi@gmail.com> writes:

> Hi! I'm working on a parser for org and I noticed something about
> nested markup in the syntax spec: markup starting at the limit of
> another object seems like it shouldn't be interpreted as markup. The
> spec says that the precondition characters are:
>
> PRE MARKER ...
> PRE
>   Either a whitespace character, -, (, {, ', ", or the beginning of a line.
> https://orgmode.org/worg/dev/org-syntax-edited.html#Emphasis_Markers
>
> With links, since `[` isn't in that list, the spec seems to imply the
> following wouldn't contain an italic block, but it does:
>
> [[https://example.com][/according to the spec, this shouldn't be marked up/]]
>
> same goes for  */abc/* (since `*` isn't in the set defined by PRE)

Fair point. We indeed missed this detail in the syntax blueprint.

Does the attached patch clarify things?

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-org-syntax.org-Special-tokens-Clarify-recursive-matc.patch --]
[-- Type: text/x-patch, Size: 1211 bytes --]

From 974786c726e151ea6bb709bd508faab1cc155fcf Mon Sep 17 00:00:00 2001
Message-Id: <974786c726e151ea6bb709bd508faab1cc155fcf.1686222568.git.yantar92@posteo.net>
From: Ihor Radchenko <yantar92@posteo.net>
Date: Thu, 8 Jun 2023 14:08:30 +0300
Subject: [PATCH] * org-syntax.org (Special tokens): Clarify recursive matching

Explain that contained objects are only matched against parent
contents.

Reported-by: Laith Bahodi <laithbahodi@gmail.com>
Link: https://orgmode.org/list/CABhQr0TRuGeJMn+G5xzqJuX=UhvwiBHJr1+P8qqDRhA8KUpPfQ@mail.gmail.com
---
 org-syntax.org | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/org-syntax.org b/org-syntax.org
index ee05c5a8..7073442b 100644
--- a/org-syntax.org
+++ b/org-syntax.org
@@ -161,6 +161,13 @@ *** Special tokens
 PRE TOKEN POST
 #+end_example
 
+/PRE/ and /POST/ token are only matched against the contents of the
+containing object. For example, /bold/ object within link description is
+only matched against the description text =*bold* description=, not
+against the full containing link text:
+
+: [[https://orgmode.org][*bold* description]]
+
 *** Case significance
 
 In this document, unless specified otherwise, case is insignificant.
-- 
2.40.0


[-- Attachment #3: Type: text/plain, Size: 495 bytes --]


> Also, how are these objects intended to be handled? I couldn't
> pinpoint where in `org-element` this is approached.

The parser narrows to the contents and starts from there, ignoring all
the surrounding text.  See `narrow-to-region' in
`org-element--parse-objects'.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Parsing nested markup objects

[Ihor Radchenko]

[-- Attachment #1: Type: text/plain, Size: 871 bytes --]

Laith Bahodi <laithbahodi@gmail.com> writes:

> Hi! I'm working on a parser for org and I noticed something about
> nested markup in the syntax spec: markup starting at the limit of
> another object seems like it shouldn't be interpreted as markup. The
> spec says that the precondition characters are:
>
> PRE MARKER ...
> PRE
>   Either a whitespace character, -, (, {, ', ", or the beginning of a line.
> https://orgmode.org/worg/dev/org-syntax-edited.html#Emphasis_Markers
>
> With links, since `[` isn't in that list, the spec seems to imply the
> following wouldn't contain an italic block, but it does:
>
> [[https://example.com][/according to the spec, this shouldn't be marked up/]]
>
> same goes for  */abc/* (since `*` isn't in the set defined by PRE)

Fair point. We indeed missed this detail in the syntax blueprint.

Does the attached patch clarify things?

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-org-syntax.org-Special-tokens-Clarify-recursive-matc.patch --]
[-- Type: text/x-patch, Size: 1211 bytes --]

From 974786c726e151ea6bb709bd508faab1cc155fcf Mon Sep 17 00:00:00 2001
Message-Id: <974786c726e151ea6bb709bd508faab1cc155fcf.1686222568.git.yantar92@posteo.net>
From: Ihor Radchenko <yantar92@posteo.net>
Date: Thu, 8 Jun 2023 14:08:30 +0300
Subject: [PATCH] * org-syntax.org (Special tokens): Clarify recursive matching

Explain that contained objects are only matched against parent
contents.

Reported-by: Laith Bahodi <laithbahodi@gmail.com>
Link: https://orgmode.org/list/CABhQr0TRuGeJMn+G5xzqJuX=UhvwiBHJr1+P8qqDRhA8KUpPfQ@mail.gmail.com
---
 org-syntax.org | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/org-syntax.org b/org-syntax.org
index ee05c5a8..7073442b 100644
--- a/org-syntax.org
+++ b/org-syntax.org
@@ -161,6 +161,13 @@ *** Special tokens
 PRE TOKEN POST
 #+end_example
 
+/PRE/ and /POST/ token are only matched against the contents of the
+containing object. For example, /bold/ object within link description is
+only matched against the description text =*bold* description=, not
+against the full containing link text:
+
+: [[https://orgmode.org][*bold* description]]
+
 *** Case significance
 
 In this document, unless specified otherwise, case is insignificant.
-- 
2.40.0


[-- Attachment #3: Type: text/plain, Size: 495 bytes --]


> Also, how are these objects intended to be handled? I couldn't
> pinpoint where in `org-element` this is approached.

The parser narrows to the contents and starts from there, ignoring all
the surrounding text.  See `narrow-to-region' in
`org-element--parse-objects'.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Parsing nested markup objects

[Ihor Radchenko]

Ihor Radchenko <yantar92@posteo.net> writes:

> Fair point. We indeed missed this detail in the syntax blueprint.
>
> Does the attached patch clarify things?
> From 974786c726e151ea6bb709bd508faab1cc155fcf Mon Sep 17 00:00:00 2001
> Message-Id: <974786c726e151ea6bb709bd508faab1cc155fcf.1686222568.git.yantar92@posteo.net>
> From: Ihor Radchenko <yantar92@posteo.net>
> Date: Thu, 8 Jun 2023 14:08:30 +0300
> Subject: [PATCH] * org-syntax.org (Special tokens): Clarify recursive matching

Applied, onto master.
https://git.sr.ht/~bzg/worg/commit/7b98eb36

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>