[DISCUSSION] Refactoring fontification system

[DISCUSSION] Refactoring fontification system

[Ihor Radchenko]

Dear all,

Recently, there have been multiple issues related to incorrect
fontification:
- https://list.orgmode.org/orgmode/23707.20428.546749.44853@frac.u-strasbg.fr/
- https://list.orgmode.org/orgmode/87fsujp7mc.fsf@web.de/
- https://list.orgmode.org/orgmode/87czvqxdn9.fsf@gmail.com/
- https://list.orgmode.org/8735nsv9qo.fsf@nicolasgoaziou.fr/T/#me1c44b6e493dd280cca4f042b833c24749ae4fe0

These issues keep appearing because our current fontification code is
based on regexps and only approximates the actual Org grammar elements.
It is largely a legacy from the times when org-element parser was not a
thing.

Maybe it is a time to upgrade the fontification according to our
state-of-art parser?

Instead of fontifying elements individually via regexps, we can leverage
org-element-map, org-element-parse-buffer, org-element-cache, and
jit-lock-mode. Each type of Org element/object can be assigned with a
fontification function accepting a single argument - the element datum.

Also, the fontification code can be move to a separate library.

WDYT?

Best,
Ihor

Re: [DISCUSSION] Refactoring fontification system

[Bruce D'Arcus]

On Fri, Nov 19, 2021 at 9:11 AM Ihor Radchenko <yantar92@gmail.com> wrote:

> WDYT?

I don't understand all the technical details, but it sounds like a
GREAT idea to me in general!

Bruce

Re: [DISCUSSION] Refactoring fontification system

[Tim Cross]

Ihor Radchenko <yantar92@gmail.com> writes:

> Dear all,
>
> Recently, there have been multiple issues related to incorrect
> fontification:
> - https://list.orgmode.org/orgmode/23707.20428.546749.44853@frac.u-strasbg.fr/
> - https://list.orgmode.org/orgmode/87fsujp7mc.fsf@web.de/
> - https://list.orgmode.org/orgmode/87czvqxdn9.fsf@gmail.com/
> - https://list.orgmode.org/8735nsv9qo.fsf@nicolasgoaziou.fr/T/#me1c44b6e493dd280cca4f042b833c24749ae4fe0
>
> These issues keep appearing because our current fontification code is
> based on regexps and only approximates the actual Org grammar elements.
> It is largely a legacy from the times when org-element parser was not a
> thing.
>
> Maybe it is a time to upgrade the fontification according to our
> state-of-art parser?
>
> Instead of fontifying elements individually via regexps, we can leverage
> org-element-map, org-element-parse-buffer, org-element-cache, and
> jit-lock-mode. Each type of Org element/object can be assigned with a
> fontification function accepting a single argument - the element datum.
>
> Also, the fontification code can be move to a separate library.
>
> WDYT?
>

Sounds like a better approach to me. In addition to being
more accurate, this would mean we don't need to keep 2 separate
definitions in sync or have confusing font locking where the regexp and element
definitions are different. Should gives us increased consistency and
less maintenance. 

Re: [DISCUSSION] Refactoring fontification system

[Nicolas Goaziou]

Hello,

Ihor Radchenko <yantar92@gmail.com> writes:

> Recently, there have been multiple issues related to incorrect
> fontification:
> - https://list.orgmode.org/orgmode/23707.20428.546749.44853@frac.u-strasbg.fr/
> - https://list.orgmode.org/orgmode/87fsujp7mc.fsf@web.de/
> - https://list.orgmode.org/orgmode/87czvqxdn9.fsf@gmail.com/
> - https://list.orgmode.org/8735nsv9qo.fsf@nicolasgoaziou.fr/T/#me1c44b6e493dd280cca4f042b833c24749ae4fe0
>
> These issues keep appearing because our current fontification code is
> based on regexps and only approximates the actual Org grammar elements.
> It is largely a legacy from the times when org-element parser was not a
> thing.
>
> Maybe it is a time to upgrade the fontification according to our
> state-of-art parser?
>
> Instead of fontifying elements individually via regexps, we can leverage
> org-element-map, org-element-parse-buffer, org-element-cache, and
> jit-lock-mode. Each type of Org element/object can be assigned with a
> fontification function accepting a single argument - the element datum.
>
> Also, the fontification code can be move to a separate library.
>
> WDYT?

I wholeheartedly agree with all these points.

Regards,
-- 
Nicolas Goaziou

Re: [DISCUSSION] Refactoring fontification system

[Ihor Radchenko]

Ihor Radchenko <yantar92@gmail.com> writes:

> Instead of fontifying elements individually via regexps, we can leverage
> org-element-map, org-element-parse-buffer, org-element-cache, and
> jit-lock-mode. Each type of Org element/object can be assigned with a
> fontification function accepting a single argument - the element datum.

I have been working on the new fontification library for the last
several months and I have noticed several confusing things in the
current fontification settings.

I'd like to hear if anyone has any idea on how to interpret the
following:

1. org-protecting-blocks is an internal auxiliary variable used to
   determine which blocks should be fontified using different major
   mode.
   It's value is ("src" "example" "export")
   So, #+begin_src lang and #+begin_export lang are fontified according
   to LANG. Makes sense.
   However, what about #+begin_example?
   org-element-example-block-parser does not appear to expect language
   specification in the example blocks. Only switches seems to be
   allowed. Am I missing something and Org actually allows example
   blocks to specify language? Or was it the case in the distant past
   versions of Org?

2. org-script-display is an internal auxiliary variable used to display
   sub/superscripts. Note that it's default value holds 4 possibilities.
   Two for each type of script. For example, for superscripts we have
   two options:
   ((raise 0.3) (height 0.7)) and ((raise 0.5))
   The first one looks more compact (does not change the line height)
   and reduces size of the superscript.
   The second one makes superscript have the same size with the main
   text an increases the line height x1.5.

   org-raise-scripts fontifies sub/superscripts differently inside and
   outside the tables:
   From org-raise-scripts:
   (nth (if table-p 3 1) org-script-display)

   However, it currently uses x1.5 line height for tables creating empty
   space between vertical | separators. It looks like a typo for me. It
   would make more sense to make table lines compact, not vice versa. Am
   I missing something?

3. org-fontify-meta-lines-and-blocks-1 creates a special face for
   ("+title:" "+subtitle:" "+author:" "+email:" "+date:")
   The face name is org-document-info.
   But what about, say, +description: or +language:?
   Would it make more sense to fontify all the keywords from
   org-options-keywords instead?
   
Best,
Ihor

Re: [DISCUSSION] Refactoring fontification system

[Ted Reed via General discussions about Org-mode.]

Ihor Radchenko <yantar92@gmail.com> writes:

> 1. org-protecting-blocks is an internal auxiliary variable used to
>    determine which blocks should be fontified using different major
>    mode.
>    It's value is ("src" "example" "export")
>    So, #+begin_src lang and #+begin_export lang are fontified according
>    to LANG. Makes sense.
>    However, what about #+begin_example?
>    org-element-example-block-parser does not appear to expect language
>    specification in the example blocks. Only switches seems to be
>    allowed. Am I missing something and Org actually allows example
>    blocks to specify language? Or was it the case in the distant past
>    versions of Org?
>

Speaking as a user, I've used fontified example blocks for json
responses when using ob-http:

#+begin_src http :pretty :wrap example json
GET http://es-master.service.consul:9200/_cluster/settings?pretty
#+end_src

#+RESULTS:
#+begin_example json
{
  "persistent": {
    "cluster": {
      "routing": {
        "allocation": {
          "node_concurrent_incoming_recoveries": "20",
          "enable": "all",
          "node_concurrent_outgoing_recoveries": "20"
        }
      }
    },
    "xpack": {
      "monitoring": {
        "collection": {
          "enabled": "true"
        }
      }
    }
  },
  "transient": {}
}
#+end_example

In this case, ob-http seems to default to generating example blocks. I
realized I could add a bit there to get it to fontify as JSON.

I'm unsure if this is correct usage or merely a side effect, or if
there's a better block to be using for this purpose, but I figured I
could at least contribute a user report about this feature.

I'm on org-mode 9.5.3 if that matters.

-- Ted Reed

Re: [DISCUSSION] Refactoring fontification system

[Ihor Radchenko]

Ted Reed <treed@zenithia.net> writes:

> #+begin_src http :pretty :wrap example json
> ...
> In this case, ob-http seems to default to generating example blocks. I
> realized I could add a bit there to get it to fontify as JSON.

Have you tried
#+begin_src http :pretty :wrap "src json :eval no"
?

It should give the same effect.

Best,
Ihor

Re: [DISCUSSION] Refactoring fontification system

[Ted Reed via General discussions about Org-mode.]

Ihor Radchenko <yantar92@gmail.com> writes:

> Have you tried
> #+begin_src http :pretty :wrap "src json :eval no"
> ?

It does work the same, thanks!

-- Ted Reed

Re: [DISCUSSION] Refactoring fontification system

[Tim Cross]

Ihor Radchenko <yantar92@gmail.com> writes:

> Ihor Radchenko <yantar92@gmail.com> writes:
>
>> Instead of fontifying elements individually via regexps, we can leverage
>> org-element-map, org-element-parse-buffer, org-element-cache, and
>> jit-lock-mode. Each type of Org element/object can be assigned with a
>> fontification function accepting a single argument - the element datum.
>
> I have been working on the new fontification library for the last
> several months and I have noticed several confusing things in the
> current fontification settings.
>

Very pleased your working on this. I suspect this will, once combined
with the other fine work you have done, generate some very rewarding
results. 

> I'd like to hear if anyone has any idea on how to interpret the
> following:
>
> 1. org-protecting-blocks is an internal auxiliary variable used to
>    determine which blocks should be fontified using different major
>    mode.
>    It's value is ("src" "example" "export")
>    So, #+begin_src lang and #+begin_export lang are fontified according
>    to LANG. Makes sense.
>    However, what about #+begin_example?
>    org-element-example-block-parser does not appear to expect language
>    specification in the example blocks. Only switches seems to be
>    allowed. Am I missing something and Org actually allows example
>    blocks to specify language? Or was it the case in the distant past
>    versions of Org?
>

I don't recall ever being able to specify language type with source
blocks. The only benefit I can see for being able to would be to have
blocks which are not 'executed' (tangled), but you can already achieve
that just using source blocks. So, from my perspective, I would want
example blocks (which I view mainly as a type of verbatim block)
fontified in a distinct manner, but no need for language specific
font-locking - if I want that, I would just use a src block. 

> 2. org-script-display is an internal auxiliary variable used to display
>    sub/superscripts. Note that it's default value holds 4 possibilities.
>    Two for each type of script. For example, for superscripts we have
>    two options:
>    ((raise 0.3) (height 0.7)) and ((raise 0.5))
>    The first one looks more compact (does not change the line height)
>    and reduces size of the superscript.
>    The second one makes superscript have the same size with the main
>    text an increases the line height x1.5.
>
>    org-raise-scripts fontifies sub/superscripts differently inside and
>    outside the tables:
>    From org-raise-scripts:
>    (nth (if table-p 3 1) org-script-display)
>
>    However, it currently uses x1.5 line height for tables creating empty
>    space between vertical | separators. It looks like a typo for me. It
>    would make more sense to make table lines compact, not vice versa. Am
>    I missing something?
>

I need superscript/subscript so rarely I really don't have much of a
position. However, I do find it annoying when a superscript/subscript
character alters the line spacing for the line it happens to display in. 

> 3. org-fontify-meta-lines-and-blocks-1 creates a special face for
>    ("+title:" "+subtitle:" "+author:" "+email:" "+date:")
>    The face name is org-document-info.
>    But what about, say, +description: or +language:?
>    Would it make more sense to fontify all the keywords from
>    org-options-keywords instead?
> 

Yes, I think that would be the right approach. I suspect most people
won't really worry too much. Absent font locking is less of an issue
than incorrect font locking. Soemthing too often seen with regexp based
font-locking which I'm hoping will be avoided with a version based more
on parsed elements etc. 
   

Re: [DISCUSSION] Refactoring fontification system

[Max Nikulin]

On 03/06/2022 16:45, Ihor Radchenko wrote:
> 
> 1. org-protecting-blocks is an internal auxiliary variable used to
>     determine which blocks should be fontified using different major
>     mode.
>     It's value is ("src" "example" "export")
>     So, #+begin_src lang and #+begin_export lang are fontified according
>     to LANG. Makes sense.
>     However, what about #+begin_example?

I have a lot of #+begin_example with language in my notes. In the 
beginning I accidentally discovered that code is highlighted accordingly 
to language syntax, so I continued to use this feature for snippets that 
are not intended to be evaluated. Later it was a bad surprise that 
export does not preserves highlights. So for me #+begin_example is a 
shortcut for #+begin_src with :eval never (that can be specified as a 
property).

 From my point of view exporters may reuse code block formatters for 
examples. It would allow to pass options to e.g. LaTeX verbatim environment:
Pedro Andres Aranda Gutierrez. Re: A question/bug report(?) Wed, 30 Mar 
2022 07:14:54 +0200 
https://list.orgmode.org/orgmode/CAO48Bk_dJs1=5zgpcZwODaTsRqyRsKq1ALj6WpaXCc4bDjW3FQ@mail.gmail.com/

> 2. org-script-display is an internal auxiliary variable used to display
>     sub/superscripts. Note that it's default value holds 4 possibilities.
>     Two for each type of script. For example, for superscripts we have
>     two options:
>     ((raise 0.3) (height 0.7)) and ((raise 0.5))
>     The first one looks more compact (does not change the line height)

Notice that subscript still increases inter-line space a bit.

>     However, it currently uses x1.5 line height for tables creating empty
>     space between vertical | separators. It looks like a typo for me. It
>     would make more sense to make table lines compact, not vice versa. Am
>     I missing something?

git blame gives 0618aeafb39dbf78e753348eaeaddbb7f8104cd0
It seems smaller font breaks horizontal alignment in tables.

Re: [DISCUSSION] Refactoring fontification system

[Ihor Radchenko]

Max Nikulin <manikulin@gmail.com> writes:

> On 03/06/2022 16:45, Ihor Radchenko wrote:
>> 
>> 1. org-protecting-blocks is an internal auxiliary variable used to
>>     determine which blocks should be fontified using different major
>>     mode.
>>     It's value is ("src" "example" "export")
>>     So, #+begin_src lang and #+begin_export lang are fontified according
>>     to LANG. Makes sense.
>>     However, what about #+begin_example?
>
> I have a lot of #+begin_example with language in my notes. In the 
> beginning I accidentally discovered that code is highlighted accordingly 
> to language syntax, so I continued to use this feature for snippets that 
> are not intended to be evaluated. Later it was a bad surprise that 
> export does not preserves highlights. So for me #+begin_example is a 
> shortcut for #+begin_src with :eval never (that can be specified as a 
> property).
>
>  From my point of view exporters may reuse code block formatters for 
> examples. It would allow to pass options to e.g. LaTeX verbatim environment:
> Pedro Andres Aranda Gutierrez. Re: A question/bug report(?) Wed, 30 Mar 
> 2022 07:14:54 +0200 
> https://list.orgmode.org/orgmode/CAO48Bk_dJs1=5zgpcZwODaTsRqyRsKq1ALj6WpaXCc4bDjW3FQ@mail.gmail.com/

Among other goals, the new fontification is aiming to avoid such bad
surprises.

As for lang parameter support in example blocks, would you mind creating
a separate feature request thread? Extending export blocks export will
require changing in parser syntax and thus should be discussed carefully
in a separate thread.

>> 2. org-script-display is an internal auxiliary variable used to display
>>     sub/superscripts. Note that it's default value holds 4 possibilities.
>>     Two for each type of script. For example, for superscripts we have
>>     two options:
>>     ((raise 0.3) (height 0.7)) and ((raise 0.5))
>>     The first one looks more compact (does not change the line height)
>
> Notice that subscript still increases inter-line space a bit.

Can you (and other interested users) try the following value of
org-script-display and let me know if it looks acceptable:

(defconst org-script-display  '(;; The values are tweaked to not change
                                ;; the line height.
                                ((raise -0.1)  (height 0.7))
				((raise 0.25)  (height 0.7))
                                ;; Alternative properties for tables.
                                ;; FIXME: We cannot change the text
                                ;; height because it will alter the
                                ;; symbol width and thus break the
                                ;; table alignment (at least, until
                                ;; org table are aligned via pixel
                                ;; width).
				((raise -0.5))
				((raise 0.5)))
  "Display properties for showing superscripts and subscripts.")

>>     However, it currently uses x1.5 line height for tables creating empty
>>     space between vertical | separators. It looks like a typo for me. It
>>     would make more sense to make table lines compact, not vice versa. Am
>>     I missing something?
>
> git blame gives 0618aeafb39dbf78e753348eaeaddbb7f8104cd0
> It seems smaller font breaks horizontal alignment in tables.

Thanks! Now it is crystal clear what was the reason behind the different
fontification inside/outside the tables. I will add the explanation to
comments.

Best,
Ihor

Re: [DISCUSSION] Refactoring fontification system

[Tom Gillespie]

> As for lang parameter support in example blocks, would you mind creating
> a separate feature request thread? Extending export blocks export will
> require changing in parser syntax and thus should be discussed carefully
> in a separate thread.

I would strongly caution against allowing an optional #+begin_example lang
syntax. It will lead to extreme confusion, even when users know to use org-lint.
The reason for this is that example blocks do not have (and frankly should not
have) full org-babel support. Babel is already complex enough as is without
having to explain to a user that yes they can noweb an example block into
a src block, but that they cannot noweb a source block into an example block.

One of the most powerful features of src blocks is that they can go from being
dumb examples all the way up to fully executable programs. Example blocks
cannot do that, and adding features that overlap with code blocks is inviting
duplicated effort and will confuse and frustrate users if they have
the misfortune
to start with an example block an then have to change mid way through to a
code block.

I also think that adding a parameter #+begin_example :lang bash to example
blocks will also lead to confusion because now there are two different ways
to specify what lang a block is. To me the answer should be to just use source
blocks if you need highlighting, example blocks should not highlight at all in
order to make the distinction clear.

Best,
Tom

Re: [DISCUSSION] Refactoring fontification system

[Tim Cross]

Tom Gillespie <tgbugs@gmail.com> writes:

>> As for lang parameter support in example blocks, would you mind creating
>> a separate feature request thread? Extending export blocks export will
>> require changing in parser syntax and thus should be discussed carefully
>> in a separate thread.
>
> I would strongly caution against allowing an optional #+begin_example lang
> syntax. It will lead to extreme confusion, even when users know to use org-lint.
> The reason for this is that example blocks do not have (and frankly should not
> have) full org-babel support. Babel is already complex enough as is without
> having to explain to a user that yes they can noweb an example block into
> a src block, but that they cannot noweb a source block into an example block.
>
> One of the most powerful features of src blocks is that they can go from being
> dumb examples all the way up to fully executable programs. Example blocks
> cannot do that, and adding features that overlap with code blocks is inviting
> duplicated effort and will confuse and frustrate users if they have
> the misfortune
> to start with an example block an then have to change mid way through to a
> code block.
>
> I also think that adding a parameter #+begin_example :lang bash to example
> blocks will also lead to confusion because now there are two different ways
> to specify what lang a block is. To me the answer should be to just use source
> blocks if you need highlighting, example blocks should not highlight at all in
> order to make the distinction clear.
>

+1. I hold the same view. 

I'm happy if example blocks have a highlighting which distinguishes them
as a 'block of something' i.e. slightly different background, smaller or different
font etc. However, they don't need font-locking style highlighting or
highlighting which is determined by a language setting. If you want that
level of highlighting, just use a src block, possibly disabling eval
when warranted.

Re: [DISCUSSION] Refactoring fontification system

[Max Nikulin]

On 08/06/2022 11:23, Tom Gillespie wrote:
>> As for lang parameter support in example blocks, would you mind creating
>> a separate feature request thread? Extending export blocks export will
>> require changing in parser syntax and thus should be discussed carefully
>> in a separate thread.
> 
> I would strongly caution against allowing an optional #+begin_example lang
> syntax. It will lead to extreme confusion, even when users know to use org-lint.
> The reason for this is that example blocks do not have (and frankly should not
> have) full org-babel support. Babel is already complex enough as is without
> having to explain to a user that yes they can noweb an example block into
> a src block, but that they cannot noweb a source block into an example block.

My expectation is that treating #+begin_example almost as #+begin_src 
will simplify code. E.g. `org-latex-src-block' has a fallback to plain 
\begin{verbatim} if language is unknown or is not specified, so it may 
handle examples as well. The difference is that for #+begin_example 
evaluation and noweb are disabled, either it is hardcoded (my 
preference) or default values for :noweb and :eval header argument are 
adjusted.

On 09/06/2022 12:37, Pedro Andres Aranda Gutierrez wrote:
> OK, just to add to the discussion. The original intent of my message was
> to give _me_ control over what I want to colour, not delegating that to
> the fontifying engine in Emacs.
> 
> I need limited and controlled colouring for text-books and lab manuals
> where colouring elements of a listing goes against the publishing
> standards. I only need custom colouring to "emulate" terminal output,
> not for the code as such. And I could do that with the #+ATTR: if supported.

Fontification and an attribute to specify an option for environment are 
independent features. My point is that if example and src blocks reused 
the same code than it would be enough to implement :options attribute 
(similar to #+begin_quote) once and both types of blocks would be more 
flexible.

If #+begin_example becomes more dumb, unsure if it is reasonable to 
allow to generate "\begin{verbatim}[commandchars=\\\{\}]" for example 
block but not for source blocks.

Side note: I can not say that #+begin_example with language is widely 
used, but one case may be found in org-manual.org and a dozen in worg.

[PATCH] #+begin_example lang used in manual and worg (was: [DISCUSSION] Refactoring fontification system)

[Ihor Radchenko]

[-- Attachment #1: Type: text/plain, Size: 331 bytes --]

Max Nikulin <manikulin@gmail.com> writes:

> Side note: I can not say that #+begin_example with language is widely 
> used, but one case may be found in org-manual.org and a dozen in worg.

Which should probably be fixed. LANG parameter is ignored on export
anyway. See the attached patches for WORG and emacs-manual.

Best,
Ihor


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-org-manual-Fix-example-block-with-unsupported-langua.patch --]
[-- Type: text/x-patch, Size: 940 bytes --]

From 1f5419fb4c249d6e53940ff16e27e16da55c5dbc Mon Sep 17 00:00:00 2001
Message-Id: <1f5419fb4c249d6e53940ff16e27e16da55c5dbc.1654826607.git.yantar92@gmail.com>
From: Ihor Radchenko <yantar92@gmail.com>
Date: Fri, 10 Jun 2022 10:02:51 +0800
Subject: [PATCH] org-manual: Fix example block with unsupported language name
 parameter

* doc/org-manual.org (Breaking Down Tasks into Subtasks): Remove
unsupported LANG parameter from #+begin_example LANG.
---
 doc/org-manual.org | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/doc/org-manual.org b/doc/org-manual.org
index 32a45f884..51aa4d77e 100644
--- a/doc/org-manual.org
+++ b/doc/org-manual.org
@@ -4488,7 +4488,7 @@ ** Breaking Down Tasks into Subtasks
 include the word =recursive= into the value of the =COOKIE_DATA=
 property.
 
-#+begin_example org
+#+begin_example
 ,* Parent capturing statistics [2/20]
   :PROPERTIES:
   :COOKIE_DATA: todo recursive
-- 
2.35.1


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #3: 0001-Fix-example-blocks-with-unsupported-language-name-pa.patch --]
[-- Type: text/x-patch, Size: 4254 bytes --]

From 4a43d570b5424dadfda74996bddc4982aadafe37 Mon Sep 17 00:00:00 2001
Message-Id: <4a43d570b5424dadfda74996bddc4982aadafe37.1654826554.git.yantar92@gmail.com>
From: Ihor Radchenko <yantar92@gmail.com>
Date: Fri, 10 Jun 2022 10:01:10 +0800
Subject: [PATCH] Fix example blocks with unsupported language name parameter

* org-contrib/babel/languages/ob-doc-shell.org (Requirements and Setup):
* org-hacks.org (Reschedule agenda items to today with a single command):
* org-tutorials/org-jekyll.org (Creating an org File to be Published with Jekyll):
(Showing Blog Posts on the Front Page):
(Creating Archive Pages):
(Inserting Image):
(Using Text Markup in Front Matte): Remove LANG in #+begin_example LANG.
---
 org-contrib/babel/languages/ob-doc-shell.org |  2 +-
 org-hacks.org                                |  2 +-
 org-tutorials/org-jekyll.org                 | 16 ++++++++--------
 3 files changed, 10 insertions(+), 10 deletions(-)

diff --git a/org-contrib/babel/languages/ob-doc-shell.org b/org-contrib/babel/languages/ob-doc-shell.org
index 730f63d7..e7d728b3 100644
--- a/org-contrib/babel/languages/ob-doc-shell.org
+++ b/org-contrib/babel/languages/ob-doc-shell.org
@@ -179,7 +179,7 @@ * Requirements and Setup
 2. The Org Babel language facility must be set to load shell
    functionality[fn:5]:
 
-    #+begin_example emacs-lisp
+    #+begin_example
     ;; active Babel languages
     (org-babel-do-load-languages
     'org-babel-load-languages
diff --git a/org-hacks.org b/org-hacks.org
index 968e511d..d446de9b 100644
--- a/org-hacks.org
+++ b/org-hacks.org
@@ -2196,7 +2196,7 @@ *** Reschedule agenda items to today with a single command
 #+index: Agenda!Reschedule
 This was suggested by Carsten in reply to David Abrahams:
 
-#+begin_example emacs-lisp
+#+begin_example
 (defun org-agenda-reschedule-to-today ()
   (interactive)
   (flet ((org-read-date (&rest rest) (current-time)))
diff --git a/org-tutorials/org-jekyll.org b/org-tutorials/org-jekyll.org
index 4da2d5ad..a0c57991 100644
--- a/org-tutorials/org-jekyll.org
+++ b/org-tutorials/org-jekyll.org
@@ -172,7 +172,7 @@ * Creating an org File to be Published with Jekyll
 
 Below is a short extract from one of my org files showing my setup:
 
-#+BEGIN_EXAMPLE org
+#+BEGIN_EXAMPLE
 #+STARTUP: showall indent
 #+STARTUP: hidestars
 #+BEGIN_EXPORT html
@@ -254,7 +254,7 @@ ** Showing Blog Posts on the Front Page
 Most blogs show the latest posts on their front page. The example
 below shows the title and an excerpt for the five latest posts:
 
-#+BEGIN_EXAMPLE html
+#+BEGIN_EXAMPLE
 <ul class="posts">
 {% for post in site.posts limit: 5 %}
   <div class="post_info">
@@ -274,7 +274,7 @@ ** Creating Archive Pages
 available. You can create a simple list of all blog posts using the
 following markup:
 
-#+begin_example html
+#+begin_example
 <ul>
   {% for post in site.posts %}
   <li>
@@ -295,7 +295,7 @@ * Inserting Image
 You will probably want to insert some images into your blog posts. I
 use the following method:
 
-#+BEGIN_EXAMPLE html
+#+BEGIN_EXAMPLE
 <img src ="/images/skiddaw.jpg"
 alt="John and Ella on Skiddaw" align="left" width="300" height="250"
 title="John and Ella on Skiddaw" class="img"</img>
@@ -304,7 +304,7 @@ * Inserting Image
 Note that the class attribute refers to the class used to style the
 image tag in your css. My css contains:
 
-#+BEGIN_EXAMPLE css
+#+BEGIN_EXAMPLE
 img {
     margin: 15px;
     border: 1px solid blue;
@@ -323,7 +323,7 @@ * Inserting Image
 
 In your =.org= file use the following html to embed the picture:
 
-#+BEGIN_EXAMPLE html
+#+BEGIN_EXAMPLE
 <div class="photofloatr">
   <p><img src="myphoto.jpg" width="300"
     height="150" alt="My Mug Shot"></p>
@@ -333,7 +333,7 @@ * Inserting Image
 
 Now you need to add some information to your style sheet:
 
-#+BEGIN_EXAMPLE css
+#+BEGIN_EXAMPLE
 div.photofloatr {
     float: right;
     border: thin silver solid;
@@ -362,7 +362,7 @@ * Using Text Markup in Front Matte
 I use this to format my page excerpts, which I include in my org files
 Front Matter markup. So in my sites index.html I have:
 
-#+begin_example html
+#+begin_example
 <li>
   <a href="{{ post.url }}">{{ post.title }}</a>
   <span>({{ post.date | date:"%Y-%m-%d" }})</span>
-- 
2.35.1

Re: [PATCH] #+begin_example lang used in manual and worg (was: [DISCUSSION] Refactoring fontification system)

[Max Nikulin]

On 10/06/2022 09:06, Ihor Radchenko wrote:
> Max Nikulin writes:
> 
>> Side note: I can not say that #+begin_example with language is widely
>> used, but one case may be found in org-manual.org and a dozen in worg.
> 
> Which should probably be fixed. LANG parameter is ignored on export
> anyway. See the attached patches for WORG and emacs-manual.

Thank you for your intent to fix it.

> diff --git a/doc/org-manual.org b/doc/org-manual.org
> index 32a45f884..51aa4d77e 100644
> --- a/doc/org-manual.org
> +++ b/doc/org-manual.org
> @@ -4488,7 +4488,7 @@ ** Breaking Down Tasks into Subtasks
>  include the word =recursive= into the value of the =COOKIE_DATA=
>  property.
>  
> -#+begin_example org
> +#+begin_example
>  ,* Parent capturing statistics [2/20]
>    :PROPERTIES:
>    :COOKIE_DATA: todo recursive

It is consistent with other examples in the manual. Only one snippet is 
wrapped into "#+begin_src org" and it is a recent addition caused a long 
discussion on Shakespeare's poetry. I am curious why #+begin_src is used 
for elisp examples, but not for org markup.

For worg pages #+begin_example to #+begin_src substitution may be a 
better option than dropping language.

> --- a/org-tutorials/org-jekyll.org
> +++ b/org-tutorials/org-jekyll.org
> @@ -172,7 +172,7 @@ * Creating an org File to be Published with Jekyll
>  
>  Below is a short extract from one of my org files showing my setup:
>  
> -#+BEGIN_EXAMPLE org
> +#+BEGIN_EXAMPLE
>  #+STARTUP: showall indent
>  #+STARTUP: hidestars
>  #+BEGIN_EXPORT html

It is not the scope of this patch but looks like missed commas to escape 
leading "#".

Re: [PATCH] #+begin_example lang used in manual and worg (was: [DISCUSSION] Refactoring fontification system)

[Ihor Radchenko]

Max Nikulin <manikulin@gmail.com> writes:

>> -#+begin_example org
>> +#+begin_example
>>  ,* Parent capturing statistics [2/20]
>>    :PROPERTIES:
>>    :COOKIE_DATA: todo recursive
>
> It is consistent with other examples in the manual. Only one snippet is 
> wrapped into "#+begin_src org" and it is a recent addition caused a long 
> discussion on Shakespeare's poetry. I am curious why #+begin_src is used 
> for elisp examples, but not for org markup.

I am not sure. In theory, the only significant difference could be that
#+begin_src org could be fontified natively on export (some time in
future; they are not now), which might be slightly confusing.

In any case, I applied the Org manual patch onto main via 0cc4f492f.

> For worg pages #+begin_example to #+begin_src substitution may be a 
> better option than dropping language.

You are right. Would you mind checking if there are any #+begin_example
cases in worg with no language specified that are also worth converting
into #+begin_src?

Best,
Ihor

[BUG] Unescaped #+ lines in WORG example blocks (was: [PATCH] #+begin_example lang used in manual and worg (was: [DISCUSSION] Refactoring fontification system))

[Ihor Radchenko]

Max Nikulin <manikulin@gmail.com> writes:

>> --- a/org-tutorials/org-jekyll.org
>> +++ b/org-tutorials/org-jekyll.org
>> @@ -172,7 +172,7 @@ * Creating an org File to be Published with Jekyll
>>  
>>  Below is a short extract from one of my org files showing my setup:
>>  
>> -#+BEGIN_EXAMPLE org
>> +#+BEGIN_EXAMPLE
>>  #+STARTUP: showall indent
>>  #+STARTUP: hidestars
>>  #+BEGIN_EXPORT html
>
> It is not the scope of this patch but looks like missed commas to escape 
> leading "#".

You are right. We need someone to look through worg pages and spot all
such instances.
(Help appreciated)

Best,
Ihor

Re: [BUG] Unescaped #+ lines in WORG example blocks (was: [PATCH] #+begin_example lang used in manual and worg (was: [DISCUSSION] Refactoring fontification system))

[Tim Cross]

Ihor Radchenko <yantar92@gmail.com> writes:

> Max Nikulin <manikulin@gmail.com> writes:
>
>>> --- a/org-tutorials/org-jekyll.org
>>> +++ b/org-tutorials/org-jekyll.org
>>> @@ -172,7 +172,7 @@ * Creating an org File to be Published with Jekyll
>>>  
>>>  Below is a short extract from one of my org files showing my setup:
>>>  
>>> -#+BEGIN_EXAMPLE org
>>> +#+BEGIN_EXAMPLE
>>>  #+STARTUP: showall indent
>>>  #+STARTUP: hidestars
>>>  #+BEGIN_EXPORT html
>>
>> It is not the scope of this patch but looks like missed commas to escape 
>> leading "#".
>
> You are right. We need someone to look through worg pages and spot all
> such instances.
> (Help appreciated)
>

Just FYI I plan to go through all the worg files and run org-lint over
them as well as fix up the many problems I've found so far with broken
links, missing images, missing macros, examples not in example blocks
etc. So while I'd appreciate any help on offer, don't worry too much
about fixing some of these as I will get to them eventually. 

One area I would definitely like some help with though are those org
files not in english. For example, there is an org tutorial written in
Spanish which is generating bad link errors. Unfortunately, I'm limited
to only one language, so not 100% sure on the right fix (it looks like
examples of  how to do links in org, but they are not in example blocks,
so org throws an error when doing the export to html. . 

Re: [BUG] Unescaped #+ lines in WORG example blocks (was: [PATCH] #+begin_example lang used in manual and worg (was: [DISCUSSION] Refactoring fontification system))

[Ihor Radchenko]

Ihor Radchenko <yantar92@gmail.com> writes:

>>> -#+BEGIN_EXAMPLE org
>>> +#+BEGIN_EXAMPLE
>>>  #+STARTUP: showall indent
>>>  #+STARTUP: hidestars
>>>  #+BEGIN_EXPORT html
>>
>> It is not the scope of this patch but looks like missed commas to escape 
>> leading "#".
>
> You are right. We need someone to look through worg pages and spot all
> such instances.
> (Help appreciated)

https://git.sr.ht/~bzg/worg/commit/729b9679

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [DISCUSSION] Refactoring fontification system

[Max Nikulin]

On 08/06/2022 09:02, Ihor Radchenko wrote:
> Max Nikulin writes:
> 
> Can you (and other interested users) try the following value of
> org-script-display and let me know if it looks acceptable:
> 
> (defconst org-script-display  '(;; The values are tweaked to not change
>                                  ;; the line height.
>                                  ((raise -0.1)  (height 0.7))
> 				((raise 0.25)  (height 0.7))

I do not mind. High symbols like "(|)" cross middle line a bit, but 
since it is not TeX and superscript can not be above subscript it should 
not cause any problem with overlapping.

>                                  ;; Alternative properties for tables.
>                                  ;; FIXME: We cannot change the text
>                                  ;; height because it will alter the
>                                  ;; symbol width and thus break the
>                                  ;; table alignment (at least, until
>                                  ;; org table are aligned via pixel
>                                  ;; width).
> 				((raise -0.5))
> 				((raise 0.5)))

The issue with increased vertical space between vertical bars might be 
alleviated a bit by using e.g. 0.35 instead of 0.5. I hope, subscripts 
and superscripts are still distinguishable.

>>>      However, it currently uses x1.5 line height for tables creating empty
>>>      space between vertical | separators. It looks like a typo for me. It
>>>      would make more sense to make table lines compact, not vice versa. Am
>>>      I missing something?
>>
>> git blame gives 0618aeafb39dbf78e753348eaeaddbb7f8104cd0
>> It seems smaller font breaks horizontal alignment in tables.
> 
> Thanks! Now it is crystal clear what was the reason behind the different
> fontification inside/outside the tables. I will add the explanation to
> comments.

Re: [DISCUSSION] Refactoring fontification system

[Phil Estival]

[2022-06-03 Wed 11:45] Ihor Radchenko <yantar92@gmail.com>:
 >
 > I'd like to hear if anyone has any idea on how to interpret the
 > following:
 >
 > 1. org-protecting-blocks is an internal auxiliary variable used to
 >    determine which blocks should be fontified using different major
 >    mode.
 >    It's value is ("src" "example" "export")
 >    So, #+begin_src lang and #+begin_export lang are fontified according
 >    to LANG. Makes sense.
 >    However, what about #+begin_example?
 >    org-element-example-block-parser does not appear to expect language
 >    specification in the example blocks. Only switches seems to be
 >    allowed. Am I missing something and Org actually allows example
 >    blocks to specify language? Or was it the case in the distant past
 >    versions of Org?


  - org-fontify-meta-lines-and-blocks-1
    is looking for begin_
    what comes after (src) is optional and can be anything

    Next it looks for "language" (match-string 5 to 7
    — it could be helpful to have comments indicating
    the number matching of the groups next to them).
    What gets fontified like a source block turns out to be:
    ,#+begin_{\w} <language> [<switches> <header arguments>]

    So this is fontified:

    #+begin_quote python
    def sss(): pass
    #+end_quote

    and this too:

    #+begin_fly awk
    BEGIN { woosh }
    #+end_fly

    Which is nice, but not interpreted like so
    by any export backend.


 >
 > 3. org-fontify-meta-lines-and-blocks-1 creates a special face for
 >    ("+title:" "+subtitle:" "+author:" "+email:" "+date:")
 >    The face name is org-document-info.
 >    But what about, say, +description: or +language:?
 >    Would it make more sense to fontify all the keywords from
 >    org-options-keywords instead?
 >

Makes more sense, yes.
I would have named them "directives"
rather than "keywords", but it's too late now.


Regards,
Phil

[RFC] Refactoring org-element API (was: [DISCUSSION] Refactoring fontification system)

[Ihor Radchenko]

[-- Attachment #1: Type: text/plain, Size: 3156 bytes --]

Ihor Radchenko <yantar92@gmail.com> writes:

> Ihor Radchenko <yantar92@gmail.com> writes:
>
>> Instead of fontifying elements individually via regexps, we can leverage
>> org-element-map, org-element-parse-buffer, org-element-cache, and
>> jit-lock-mode. Each type of Org element/object can be assigned with a
>> fontification function accepting a single argument - the element datum.
>
> I have been working on the new fontification library for the last
> several months and I have noticed several confusing things in the
> current fontification settings.

It has been a while since the last update on the fontification... for a
reason.

As the fontification should be very tightly optimized to avoid Emacs
freezes, I stumbled upon a need to improve the performance of Org parser
further than we already have. This triggered a number of significant
changes in listp/org-element.el. So many that the current logic of the
library becomes a mess.

In order not to turn org-element into another org-agenda, I am proposing
to factor out Org syntax tree API into a separate file
org-element-ast.el and add a number of breaking changes how the syntax
tree is structured.

The most important changes are the following:

1. Frequently used element properties will no longer be stored directly
   in the property list. Instead, they will be placed inside a special
   vector that is much faster to access. `org-element-property' and
   other accessor functions are changed accordingly to inline the
   property queries into `aref' calls, when possible.

   This is a major breaking change.

2. Org parser will now be able to parse elements partially, with some
   parts of the parser executed later, only when necessary.
   The downside is relying upon the original buffer to be live even
   after parsing.

   This is also a major breaking change.

3. The order of multiple affiliated keywords in Org parse tree will be
   reversed. The main purpose here is to avoid special cases when fine
   details of Org syntax had to be accounted for when traversing Org
   parse trees.

   This is a breaking change.

4. Org cache will be available even when `org-element-use-cache' is
   non-nil. This is not a breaking change and simplifies Org code
   greatly without compromising performance.

5. The usage of regular expressions is by Org parser is now tightly
   optimized. See the discussion in https://debbugs.gnu.org/cgi/bugreport.cgi?bug=63225
   This is also not a breaking change.

6. Memory footprint is reduced for Org AST. This is achieved using
   shared string objects in Org parser. With low probability, this might
   cause problems if third-party code modifies these string objects by
   side effect.

This refactoring is large, spanning >80 commits.
The full commit log can be found at
https://git.sr.ht/~yantar92/org-mode/log/feature/org-element-ast-tidy

Here, I am attaching ORG-NEWS diff, summarizing important changes.
I am also adding the commentary section of org-element-ast.el with more
details about the new syntax tree structure and about the new concept of
deferred parsed values.

I am sure that I missed things, so comments are welcome.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: ORG-NEWS.diff --]
[-- Type: text/x-patch, Size: 9763 bytes --]

diff --git a/etc/ORG-NEWS b/etc/ORG-NEWS
index ddf1e9110..396935283 100644
--- a/etc/ORG-NEWS
+++ b/etc/ORG-NEWS
@@ -13,6 +13,187 @@ Please send Org bug reports to mailto:emacs-orgmode@gnu.org.
 
 * Version 9.7 (not released yet)
 ** Important announcements and breaking changes
+*** Major changes and additions to Org API
+**** New term: "syntax node"
+
+To reduce confusion with "element" referring to both "syntax element"
+and "element/object" class, we now prefer using "syntax node" when
+referring to generic Org syntax elements.  "Elements" and "objects"
+now refer to different syntax node classes of paragraph-like nodes and
+markup-like nodes.
+
+**** New element type ~anonymous~
+
+Secondary strings can now be recognized as ~anonymous~ type to
+distinguish from non-elements.  With a new optional argument for
+~org-element-type~ will return ~anonymous~ for secondary strings
+instead of nil.
+
+The new element type can be used in ~org-element-lineage~,
+~org-element-map~, and other functions that filter by element type.
+**** Internal structure of Org parse tree has been changed
+
+The code relying upon the previously used =(TYPE PROPERTIES-PLIST CONTENTS-LIST)=
+structure may no longer work.  Please use ~org-element-create~,
+~org-element-property~, and other Org element API functions to work
+with Org syntax trees.
+
+Some syntax node properties are no longer stored as property list elements.
+Instead, they are kept in a special vector value of a new
+=:standard-properties= property.  This is done to improve performance.
+
+Properties and their values can now be deferred to avoid overheads when
+parsing.  They are calculated lazily, when the value/property is
+requested by getters.
+
+New special property =:secondary= is used internally to record which
+properties store secondary objects.
+
+New special property =:deferred= is used to keep information how to
+calculate property names lazily.
+
+See the commentary in =lisp/org-element-ast.el= for more details.
+
+**** Multiple affiliated keyword values are now stored in the order they appear in buffer
+
+Previously,
+
+: #+caption: foo
+: #+caption: bar
+: Paragraph
+
+would have its =:caption= property set to ~(("bar") ("foo"))~ in reverse order.
+
+Now, the order is not reversed: ~(("foo") ("bar"))~.
+
+**** Some property values may now be calculated lazily and require original Org buffer to be live
+
+~org-element-at-point~, ~org-element-context~, and
+~org-element-at-point-no-context~ may now not calculate all the
+property values at the call time.  Instead, the calculation will be
+deferred until ~org-element-property~ or the equivalent getter
+function is called.  The property names may not all be calculated as
+well.
+
+It may often be necessary to have the original Org buffer open when
+resolving the deferred values.
+
+One can ensure that all the deferred values are resolved using new
+function ~org-element-resolve-deferred~ and new optional argument for
+~org-element-property~.
+
+~org-element-parse-buffer~ and ~org-element-parse-secondary-string~
+will resolve all the deferred values by default.  No adjustment is
+needed for their users.
+
+**** New API functions and macros
+***** New property accessors and setters
+
+New functions to retrieve and set (via ~setf~) commonly used element properties:
+- =:begin= :: ~org-element-begin~
+- =:end= :: ~org-element-end~
+- =:contents-begin= :: ~org-element-contents-begin~
+- =:contents-end= :: ~org-element-contents-end~
+- =:contents-post-affiliated= :: ~org-element-post-affiliated~
+- =:contents-post-blank= :: ~org-element-post-blank~
+- =:parent= :: ~org-element-parent~
+ 
+***** New macro ~org-element-with-enabled-cache~
+
+The macro arranges the element cache to be active during =BODY= execution.
+When cache is enabled, the macro is identical to ~progn~.  When cache
+is disabled, the macro arranges a new fresh cache that is discarded
+upon completion of =BODY=.
+
+***** New function ~org-element-property-1~
+
+This function is like ~org-element-property~ but does not try to
+resolve deferred properties.
+
+~org-element-property-1~ can be used with ~setf~.
+
+***** New function ~org-element-put-property-2~
+
+Like ~org-element-put-property~, but the argument list is changed to have
+=NODE= as the last argument.  Useful with threading macros like
+~thread-last~.
+
+***** New function ~org-element-properties-resolve~
+
+This function resolves all the deferred values in a =NODE=, modifying
+the =NODE= for side effect.
+
+***** New functions ~org-element-properties-map~ and ~org-element-properties-mapc~
+
+New functions to map over =NODE= properties.
+
+***** New function ~org-element-ast-map~
+
+This is a more general equivalent of ~org-element-map~.  It allows
+more precise control over recursion into secondary strings.
+
+***** New function ~org-element-lineage-map~
+
+Traverse syntax tree ancestor list, applying arbitrary function to
+each ancestor.
+
+***** New function ~org-element-property-inherited~
+
+Like ~org-element-property~, but can be used to retrieve and combine
+multiple different properties for a given =NODE= and its parents.
+
+**** ~org-element-cache-map~ can now be used even when element cache is disabled
+**** =org-element= API functions and macros can now accept syntax elements as =POM= argument
+
+The following functions are updated:
+- ~org-agenda-entry-get-agenda-timestamp~
+- ~org-element-at-point~
+- ~org-is-habit-p~
+- ~org-id-get~
+- ~org-with-point-at~
+- ~org-entry-properties~
+- ~org-entry-get~
+- ~org-entry-delete~
+- ~org-entry-add-to-multivalued-property~
+- ~org-entry-remove-from-multivalued-property~
+- ~org-entry-member-in-multivalued-property~
+- ~org-entry-put-multivalued-property~
+- ~org-entry-get-with-inheritance~
+- ~org-entry-put~
+- ~org-read-property-value~
+- ~org-property-get-allowed-values~
+
+**** ~org-element-map~ now traverses main value in dual keywords before the secondary value
+
+The traverse order for dual keywords is reversed.  The main value is
+now traversed first, followed by the secondary value.
+
+**** Org parse tree is now non-printable
+
+Org parser now assigns a new property =:buffer= that holds
+non-printable buffer object.  This makes syntax tree non-printable.
+Using ~print~/~read~ is no longer safe.
+
+**** Some Org API functions no longer preserve match data
+
+~org-element-at-point~, ~org-element-context~, ~org-get-category~, ~org-get-tags~
+
+The relevant function docstrings now explicitly mention that match
+data may be modified.
+**** ~org-element-create~ now treats a single ~anonymous~ =CHILDREN= argument as a list of child nodes
+
+When =CHILDREN= is a single anonymous node, use its contents as children
+nodes.  This way,
+
+: (org-element-create 'section nil (org-element-contents node))
+
+will yield expected results with contents of another node adopted into
+a newly created one.
+
+Previously, one had to use
+
+: (apply #'org-element-create 'section nil (org-element-contents node))
+
 *** "Priority" used to sort items in agenda is renamed to "urgency"
 
 Previously, ~priority-up~ and ~priority-down~ in
@@ -225,7 +406,64 @@ editing with Emacs while a ~:session~ block executes.
 
 When ~org-return-follows-link~ is non-nil and cursor is over an
 org-cite citation, ~org-return~ will call ~org-open-at-point~.
+** New functions and changes in function arguments
+*** =TYPES= argument in ~org-element-lineage~ can now be a symbol
+
+When =TYPES= is symbol, only check syntax nodes of that type.
+
+*** New optional argument =KEEP-CONTENTS= for ~org-element-copy~
+
+With the new argument, the contents is copied recursively.
+
+*** ~org-element-property~ can now be used with ~setf~
+*** New optional arguments for ~org-element-property~
+
+The value of the new optional argument =DFLT= is returned if the
+property with given name is not present.  Same as =DEFAULT= argument
+for ~alist-get~.
+
+New optional argument =FORCE-UNDEFER= modifies the =NODE=, storing the
+resolved deferred values.
+
+*** New optional argument =NO-UNDEFER= in ~org-element-map~ and changed argument conventions
+
+New optional argument =NO-UNDEFER=, when non-nil, will make
+~org-element-map~ keep deferred secondary string values in their raw form.
+
+=TYPES= argument can now be set to ~t~.  This will match all the
+syntax nodes when traversing the tree.
+
+~FUN~ can now be a lisp form that will be evaluated with symbol ~node~
+assigned to the current syntax node.
+
+~FUN~ can now throw ~:org-element-skip~ signal to skip recursing into
+current element children and secondary strings.
+
+*** New optional argument =KEEP-DEFERRED= in ~org-element-parse-buffer~
+
+When non-nil, the deferred values and properties will not be resolved.
+
+*** New optional argument =ANONYMOUS= for ~org-element-type~
+
+When the new argument is non-nil, return symbol ~anonymous~ for anonymous elements.
+
+*** ~org-element-adopt-elements~ is renamed to ~org-element-adopt~
+
+The old name is kept as an alias.  The new name creates less confusion
+as the function can also act on objects.
+
+*** ~org-element-extract-element~ is renamed to ~org-element-extract~
+
+The old name is kept as an alias.  The new name creates less confusion
+as the function can also act on objects.
+
+*** ~org-element-set-element~ is renamed to ~org-element-set~
+
+The old name is kept as an alias.  The new name creates less confusion
+as the function can also act on objects.
 
+*** ~org-export-get-parent~ is renamed to ~org-element-parent~ and moved to =lisp/org-element.el=
+*** ~org-export-get-parent-element~ is renamed to ~org-element-parent-element~ and moved to =lisp/org-element.el=
 ** Miscellaneous
 *** =org-crypt.el= now applies initial visibility settings to decrypted entries
 

[-- Attachment #3: Type: text/plain, Size: 7620 bytes --]


;;; org-element-ast.el --- Abstract syntax tree for Org  -*- lexical-binding: t; -*-

;; ...
;; This file implements Org abstract syntax tree (AST) data structure.
;;
;; Only the most generic aspect of the syntax tree are considered
;; below.  The fine details of Org syntax are implemented elsewhere.
;;
;; Org AST is composed of nested syntax nodes.
;; Within actual Org syntax, the nodes can be either headings,
;; elements, or objects.  However, historically, we often call syntax
;; nodes simply "elements", unless the context requires clarification
;; about the node type.  In particular, many functions below will have
;; naming pattern `org-element-X', implying `org-element-node-X' --
;; they will apply to all the node types, not just to elements.
;;
;; 1. Syntax nodes
;; ------------------
;; Each Org syntax node can be represented as a string or list.
;;
;; The main node representation follows the pattern
;; (TYPE PROPERTIES CONTENTS), where
;;   TYPE is a symbol describing the node type.
;;   PROPERTIES is the property list attached to it.
;;   CONTENTS is a list of child syntax nodes contained within the
;;            current node, when applicable.
;;
;;; For example, "*bold text*  " node can be represented as
;;
;;    (bold (:begin 1 :end 14 :post-blank 2 ...) "bold text")
;;
;; TYPE can be any symbol, including symbol not explicitly defined by
;; Org syntax.  If TYPE is not a part of the syntax, the syntax
;; node is called "pseudo element/object", but otherwise considered a
;; valid part of Org syntax tree.  Search "Pseudo objects and
;; elements" in lisp/ox-latex.el for an example of using pseudo
;; elements.
;;
;; PROPERTIES is a property list (:property1 value1 :property2 value2 ...)
;; holding properties and value.
;;
;; `:standard-properties', `:parent', `:deferred', and `:secondary'
;; properties are treated specially in the code below.
;;
;; `:standard-properties' holds an array with
;; `org-element--standard-properties' values, in the same order.  The
;; values in the array have priority over the same properties
;; specified in the property list.  You should not rely on the value
;; of `org-element--standard-propreties' in the code.
;; `:standard-properties' may or may not be actually present in
;; PROPERTIES.  It is mostly used to speed up property access in
;; performance-critical code, as most of the code requesting property
;; values by constant name is inlined.
;;
;; The previous example can also be presented in more compact form as:
;;
;;    (bold (:standard-properties [1 10 ... 2 ...]) "bold text")
;;
;; Using an array allows faster access to frequently used properties.
;;
;; `:parent' holds the containing node, for a child node within the
;; AST.  It may or may not be present in PROPERTIES.
;;
;; `:secondary' holds a list of properties that may contain extra AST
;; nodes, in addition to the node contents.
;;
;; `deferred' property describes how to update not-yet-calculated
;; properties on request.
;;
;;
;; Syntax node can also be represented by a string.  Strings always
;; represent syntax node of `plain-text' type with contents being nil
;; and properties represented as string properties at position 0.
;; `:standard-properties' are not considered for `plain-text' nodes as
;; `plain-text' nodes tend to hold much fewer properties.
;;
;; In the above example, `plain-text' node "bold text" is more
;; accurately represented as
;;
;;    #("bold text" 0 9 (:parent (bold ...)))
;;
;; with :parent property value pointing back to the containing `bold'
;; node.
;;
;; `anonymous' syntax node is represented as a list with `car'
;; containing another syntax node.  Such node has nil type, does not
;; have properties, and its contents is a list of the contained syntax
;; node.  `:parent' property of the contained nodes point back to the
;; list itself, except when `anonymous' node holds secondary value
;; (see below), in which case the `:parent' property is set to be the
;; containing node in the AST.
;;
;; Any node representation other then described above is not
;; considered as Org syntax node.
;;
;; 2. Deferred values
;; ------------------
;; Sometimes, it is computationally expensive or even not possible to
;; calculate property values when creating an AST node.  The value
;; calculation can be deferred to the time the value is requested.
;;
;; Property values and contained nodes may have a special value of
;; `org-element-deferred' type.  Such values are computed dynamically.
;; Either every time the property value is requested or just the first
;; time.  In the latter case, the `org-element-deferred' property
;; value is auto-replaced with the dynamically computed result.
;;
;; Sometimes, even property names (not just property values) cannot, or
;; should not be computed in advance.  If a special property
;; `:deferred' has the value of `org-element-deferred-type', it is
;; first resolved for side effects of setting the missing properties.
;; The resolved value is re-assigned to the `:deferred' property.
;;
;; Note that `org-element-copy' unconditionally resolves deferred
;; properties.  This is useful to generate pure (in functional sense)
;; AST.
;;
;; The properties listed in `org-element--standard-properties', except
;; `:deferred' and `:parent' are never considered to have deferred value.
;; This constraint makes org-element API significantly faster.
;;
;; 3. Org document representation
;; ------------------------------
;; Document AST is represented by nested Org syntax nodes.
;;
;; Each node in the AST can hold the contained node in its CONTENTS or
;; as values of properties.
;;
;; For example, (bold (...) "bold text") `bold' node contains
;; `plain-text' node in CONTENTS.
;;
;; The containing node is called "parent node".
;;
;; The contained nodes held inside CONTENTS are called "child nodes".
;; They must have their `:parent' property set to the containing
;; parent node.
;;
;; The contained nodes can also be held as property values.  Such
;; nodes are called "secondary nodes".  Only certain properties
;; can contribute to AST - the property names listed as the value of
;; special property `:secondary'
;;
;; For example,
;;
;;   (headline ((:secondary (:title)
;;               :title (#("text" 0 4 (:parent (headline ...)))))))
;;
;; is a parent headline node containing "text" secondary string node
;; inside `:title' property.  Note that `:title' is listed in
;; `:secondary' value.
;;
;; The following example illustrates an example AST for Org document:
;;
;; ---- Org document --------
;; * Heading with *bold* text
;; Paragraph.
;; ---- end -----------------
;;
;; (org-data (...) ; `org-data' node.
;;   (headline
;;     (
;;      ;; `:secondary' property lists property names that contain other
;;      ;; syntax tree nodes.
;;
;;      :secondary (:title)
;;
;;      ;; `:title' property is set to anonymous node containing:
;;      ;; `plain-text', `bold', `plain-text'.
;;
;;      :title ("Heading with " (bold (:post-blank 1 ...) "bold") "text"))
;;
;;      ;; `headline' contents
;;     (section (...)
;;       (paragraph
;;         ;; `:parent' property set to the containing section.
;;         (:parent (section ...))
;;         ;; paragraph contents is a `plain-text' node.
;;         "Paragraph1."))))
;;
;; Try calling M-: (org-element-parse-buffer) on the above example Org
;; document to explore a more complete version of Org AST.


-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [RFC] Refactoring org-element API (was: [DISCUSSION] Refactoring fontification system)

[Ihor Radchenko]

Hi,

Heads up to org-ml developers as the discussed change is very likely to
break org-ml (https://github.com/ndwarshuis/org-ml).

See https://list.orgmode.org/874jnudps5.fsf@localhost/T/#u

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [RFC] Refactoring org-element API

[Stefan Nobis]

Hi Ihor,

thank you for all your great work. I did not dive deep into all the
details, but the overview you provided sounds sensible.

One minor nitpick: I would prefer a function like
~org-element-property-1~ to be named e.g. ~org-element-property-lazy~
(or something similar), because the "-1" is just obfuscation to me.

--
Until the next mail...,
Stefan.

Re: [RFC] Refactoring org-element API

[Ihor Radchenko]

Stefan Nobis <stefan-ml@snobis.de> writes:

> One minor nitpick: I would prefer a function like
> ~org-element-property-1~ to be named e.g. ~org-element-property-lazy~
> (or something similar), because the "-1" is just obfuscation to me.

foo-...-1 is actually frequently used in Emacs code for semi-internal
helper functions.

I do not mind alternative name, although I feel that
~org-element-property-lazy~ is not very descriptive.
I'd prefer to see several more ideas about the alternative names, if
possible.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [RFC] Refactoring org-element API

[Ihor Radchenko]

Ihor Radchenko <yantar92@posteo.net> writes:

>> One minor nitpick: I would prefer a function like
>> ~org-element-property-1~ to be named e.g. ~org-element-property-lazy~
>> (or something similar), because the "-1" is just obfuscation to me.
> ...
> I'd prefer to see several more ideas about the alternative names, if
> possible.

What about org-element-property-raw?

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [RFC] Refactoring org-element API

[Ihor Radchenko]

Ihor Radchenko <yantar92@posteo.net> writes:

>>> One minor nitpick: I would prefer a function like
>>> ~org-element-property-1~ to be named e.g. ~org-element-property-lazy~
>>> (or something similar), because the "-1" is just obfuscation to me.
>> ...
>> I'd prefer to see several more ideas about the alternative names, if
>> possible.
>
> What about org-element-property-raw?

Now renamed to `org-element-property-raw'.

Since one month have passed and no major concerns have been raised, I
plan to push the branch onto main tomorrow.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [RFC] Refactoring org-element API (was: [DISCUSSION] Refactoring fontification system)

[Ihor Radchenko]

Ihor Radchenko <yantar92@posteo.net> writes:

> In order not to turn org-element into another org-agenda, I am proposing
> to factor out Org syntax tree API into a separate file
> org-element-ast.el and add a number of breaking changes how the syntax
> tree is structured.

Applied, onto main.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>