[PATCH] org-manual: Describe export process flow

[PATCH] org-manual: Describe export process flow

[Ihor Radchenko]

[-- Attachment #1: Type: text/plain, Size: 318 bytes --]

Hi,

I'd like to add a new section to Org mode manual.
The new section will describe all the steps performed by Org export
process. This should hopefully create a more clear picture on how
various export hooks and filters are used.

The patch is attached.
I'd appreciate feedback from people not familiar with ox.el.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-doc-org-manual.org-Describe-export-flow.patch --]
[-- Type: text/x-patch, Size: 3477 bytes --]

From 57d290b710c85e0411855937e224d2cecdbd52da Mon Sep 17 00:00:00 2001
Message-ID: <57d290b710c85e0411855937e224d2cecdbd52da.1703600243.git.yantar92@posteo.net>
From: Ihor Radchenko <yantar92@posteo.net>
Date: Tue, 26 Dec 2023 15:15:23 +0100
Subject: [PATCH] doc/org-manual.org: Describe export flow

* doc/org-manual.org (Summary of the export process): Explain how the
export process is handled in Org mode.
---
 doc/org-manual.org | 54 ++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 54 insertions(+)

diff --git a/doc/org-manual.org b/doc/org-manual.org
index 7db69cbb6..1a9432b4d 100644
--- a/doc/org-manual.org
+++ b/doc/org-manual.org
@@ -16391,6 +16391,60 @@ ** Advanced Export Configuration
 :DESCRIPTION: Fine-tuning the export output.
 :END:
 
+*** Summary of the export process
+
+During export, Org mode processes the source Org buffer in multiple
+steps:
+
+1. The source Org mode buffer is copied into temporary throwaway
+   buffer that can be edited by export hooks
+2. ~org-export-before-processing-hook~ is executed (see [[*Export hooks]])
+3. All the =#+include= keywords are expanded
+4. All the commented headings are removed
+5. All the macros are replaced in buffer
+6. When ~org-export-use-babel~ is non-nil (default), all the src
+   blocks and babel calls that are not inside archived headings are
+   processed
+7. ~org-export-before-parsing-hook~ is executed (see [[*Export hooks]])
+8. Export option values are calculated, according to in-buffer
+   keywords, =#+BIND= keywords, buffer-local and global
+   customizations.
+9. Files contributing to bibliography are calculated
+10. ~org-export-filter-options-functions~ is executed
+11. The buffer is parsed, generating abstract syntax tree (AST)
+12. The AST is cleaned from buffer elements that should not be
+    exported:
+    - Heading are removed according to =SELECT_TAGS= and
+      =EXCLUDE_TAGS= export keywords; =task=, =inline=, =arch= export
+      options
+    - All the comments are removed
+    - Clocks, drawers, fixed-width environments, footnotes, latex
+      environments and fragments, node properties, planning lines,
+      property drawers, statistics cookies, and timestamps are removed
+      or kept according to the corresponding export options
+    - Table rows containing [[*Column Width and Alignment][width and alignment markers]] are removed
+    - Table columns containing [[*Advanced features][recalc marks]] are removed
+13. Environment variables are expanded in all the file links when
+    =expand-links= export option is set
+14. ~org-export-filter-parse-tree-functions~ is executed
+15. All the citations are processed according to the chosen citation
+    backend
+16. =#+print_bibliography= keywords are replaced with bibliography
+    listings
+17. AST is transcoded according to the chosen export backend
+    - The export happens recursively, depth-first
+    - Each transcoded AST node, as a string, is passed to the
+      corresponding export filter (see [[*Filters]])
+18. The transcoded AST body is formatted according to backend's
+    "inner" template
+19. The resulting body is passed to
+    ~org-export-filter-body-functions~
+20. The filtered body is formatted according to backend's outer
+    template
+21. The resulting output is processed by citation backend finalizer
+22. ~org-export-filter-final-output-functions~ are applied to the
+    final output
+
 *** Export hooks
 :PROPERTIES:
 :UNNUMBERED: notoc
-- 
2.42.0


[-- Attachment #3: Type: text/plain, Size: 225 bytes --]



-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [PATCH] org-manual: Describe export process flow

[Thomas S. Dye]

Aloha Ihor,

Ihor Radchenko <yantar92@posteo.net> writes:

> Hi,
>
> I'd like to add a new section to Org mode manual.
> The new section will describe all the steps performed by Org 
> export
> process. This should hopefully create a more clear picture on 
> how
> various export hooks and filters are used.
>
> The patch is attached.
> I'd appreciate feedback from people not familiar with ox.el.

I'm not too familiar with ox.el.

I edited mostly to use an active voice. I put author queries in 
parentheses.  I haven't paid attention to manual formatting 
conventions.

IMO, more links would likely be helpful.

* Suggested revision

Org mode export is a multi-step process that works on a temporary 
copy of the buffer as follows:
    1. Execute ~org-export-before-processing-hook~ (see 
    [[*Hooks]]);
    2. Expand =#+include= keywords;
    3. Remove commented headings; (author: clarify whether the 
    body is also removed?)
    4. Replace macros;
    5. Export source code blocks, conditional on 
    ~org-export-use-babel~ and, if
       necessary, the individual ~exports~ header arguments (see 
       [[*Exporting Code Blocks]]);
    6. Evaluate source code blocks (see [[*Evaluating Code 
    Blocks]]);
    7. Execute ~org-export-before-parsing-hook~ (see [[*Hooks]]);
    8. Calculate export option values according to in-buffer 
    keywords,
       =#+BIND= keywords, and buffer-local and global 
       customizations;
    9. Calculate (author: Determine instead of Calculate?) 
    bibliography file paths;
    10. Execute ~org-export-filter-options-functions~;
    11. Parse the temporary (author: is this correct?) buffer to 
    generate an
        abstract syntax tree (AST);
    12. Remove elements that will not be exported from the AST:
        1. Headings according to =SELECT_TAGS= and =EXCLUDE_TAGS= 
        export
           keywords, and =task=, =inline=, =arch= export options;
        2. Comments;
        3. Clocks, drawers, fixed-width environments, footnotes, 
        LaTeX environments and fragments, node properties, 
        planning lines, property drawers, statistics cookies, and 
        timestamps according to their corresponding export 
        options;
        4. Table rows containing [[*Column Width and 
        Alignment][width and alignment markers]];
        5. Table columns containing [[*Advanced features][recalc 
        marks]];
    13. Expand environment variables in file links according to 
    the =expand-links= export option (author: are we still 
    operating on AST here?);
    14. Execute ~org-export-filter-parse-tree-functions~ (author: 
    ditto);
    15. Create bibliography listing(s) from citation(s) (author: 
    ditto);
    16. Replace =+print_bibliography= keyword(s) with bibliography 
    listing(s) (author: ditto);
    17. Transcode the AST according to export backend using 
    recursive, depth-first search, and passing each transcoded 
    node as a string to the export filter (see [[*Filters]]):
    18. Format transcoded AST according to the backend's "inner" 
    template;
    19. Execute ~org-export-filter-body-functions~ on the 
    transcoded and formatted AST;
    20. Format the result according to the backend's "outer" 
    template;
    21. Finalize bibliography listing(s) (author: correct?) and 
    citation(s); and
    22. Execute ~org-export-filter-final-output-functions~.

Let me know if you have questions.

All the best,
Tom
-- 
Thomas S. Dye
https://tsdye.online/tsdye

Re: [PATCH] org-manual: Describe export process flow

[Karthik Chikmagalur]

> The patch is attached.
> I'd appreciate feedback from people not familiar with ox.el.

- When exporting a sub-tree, at what stage of the export process is the
  buffer narrowed to the sub-tree?
- Are "inner" and "outer" templates described in the manual, and if they
  are could you add a link to those sections when mentioning them in
  this summary?  I only found references to the plist properties
  BEAMER_INNER_THEME etc.

Karthik

Re: [PATCH] org-manual: Describe export process flow

[Matt]

This looks useful.

My only understanding of the problem this addresses is what you wrote:

 > create a more clear picture on how various export hooks and filters
 > are used.

With this understanding, my first question was, "What hooks exist?"  I
see that [[https://www.gnu.org/software/emacs/manual/html_node/org/Advanced-Export-Configuration.html][Advanced Export Configuration]] gives only
~org-export-before-processing-hook~ and
~org-export-before-parsing-hook~.  The section below it gives a table
of filters.

Here are all the hooks and functions for org-export (via =C-h v
org-export--hooks TAB= and =C-h v org-export--function TAB=).  I see
59 of them.

| Hooks                                           | Manual (60f357e8b) | Patch |
|-------------------------------------------------+--------------------+-------|
| org-export-before-processing-hook (obsolete)    | X                  | X     |
| org-export-before-parsing-hook                  | X                  | X     |
| org-export-stack-mode-hook                      |                    |       |
| org-export-filter-options-functions             | X                  | X     |
| org-export-filter-parse-tree-functions          | X                  | X     |
| org-export-filter-body-functions                | X                  | X     |
| org-export-filter-final-output-functions        | X                  | X     |
| org-export-before-parsing-functions             |                    |       |
| org-export-before-processing-functions          |                    |       |
| org-export-filter-babel-call-functions          | X                  |       |
| org-export-filter-bold-functions                | X                  |       |
| org-export-filter-center-block-functions        | X                  |       |
| org-export-filter-clock-functions               | X                  |       |
| org-export-filter-code-functions                | X                  |       |
| org-export-filter-diary-sexp-functions          | X                  |       |
| org-export-filter-drawer-functions              | X                  |       |
| org-export-filter-dynamic-block-functions       | X                  |       |
| org-export-filter-entity-functions              | X                  |       |
| org-export-filter-example-block-functions       | X                  |       |
| org-export-filter-export-block-functions        | X                  |       |
| org-export-filter-export-snippet-functions      | X                  |       |
| org-export-filter-fixed-width-functions         | X                  |       |
| org-export-filter-footnote-definition-functions | X                  |       |
| org-export-filter-footnote-reference-functions  | X                  |       |
| org-export-filter-headline-functions            | X                  |       |
| org-export-filter-horizontal-rule-functions     | X                  |       |
| org-export-filter-inline-babel-call-functions   | X                  |       |
| org-export-filter-inline-src-block-functions    | X                  |       |
| org-export-filter-inlinetask-functions          | X                  |       |
| org-export-filter-italic-functions              | X                  |       |
| org-export-filter-item-functions                | X                  |       |
| org-export-filter-keyword-functions             | X                  |       |
| org-export-filter-latex-environment-functions   | X                  |       |
| org-export-filter-latex-fragment-functions      | X                  |       |
| org-export-filter-line-break-functions          | X                  |       |
| org-export-filter-link-functions                | X                  |       |
| org-export-filter-node-property-functions       | X                  |       |
| org-export-filter-paragraph-functions           | X                  |       |
| org-export-filter-plain-list-functions          | X                  |       |
| org-export-filter-plain-text-functions          | X                  |       |
| org-export-filter-planning-functions            | X                  |       |
| org-export-filter-property-drawer-functions     | X                  |       |
| org-export-filter-quote-block-functions         | X                  |       |
| org-export-filter-radio-target-functions        | X                  |       |
| org-export-filter-section-functions             | X                  |       |
| org-export-filter-special-block-functions       | X                  |       |
| org-export-filter-src-block-functions           | X                  |       |
| org-export-filter-statistics-cookie-functions   | X                  |       |
| org-export-filter-strike-through-functions      | X                  |       |
| org-export-filter-subscript-functions           | X                  |       |
| org-export-filter-superscript-functions         | X                  |       |
| org-export-filter-table-cell-functions          | X                  |       |
| org-export-filter-table-functions               | X                  |       |
| org-export-filter-table-row-functions           | X                  |       |
| org-export-filter-target-functions              | X                  |       |
| org-export-filter-timestamp-functions           | X                  |       |
| org-export-filter-underline-functions           | X                  |       |
| org-export-filter-verbatim-functions            | X                  |       |
| org-export-filter-verse-block-functions         | X                  |       |
|-------------------------------------------------+--------------------+-------|

* Feedback 1:
How are the functions not present in the patch handled?

I have no clue if they're relevant here or if someone would be
confused by their absence.  I only noticed that some of them were used
in the patch and that the purpose of the patch also deals with
filters.

* Feedback 2:
If I read the patch correctly, the it places the summary *before* the
"Export hooks" section.  This means that the summary will refer to the
hooks before they're introduced (AFAICT, they're only ever mentioned
in the "Export hooks" section).  The summary should come after the
hooks.  Maybe before "Extending an existing back-end" or before the
next Chapter (Publishing)?

* Feedback 3:
These are nitpicky things.

 > 1. The source Org mode buffer is copied into temporary throwaway
 >   buffer that can be edited by export hooks

"throwaway" is unnecessary because "temporary" implies it.

 > 6. When ~org-export-use-babel~ is non-nil (default), all the src
 >   blocks and babel calls that are not inside archived headings are
 >   processed

I would write out "src" as "source".  Do we have an official way to
refer to source blocks?  For example, we standardize on "Org":
https://git.savannah.gnu.org/cgit/emacs/org-mode.git/tree/doc/Documentation_Standards.org#n47

Capitalize "babel" since it's a name.

 > 8. Export option values are calculated, according to in-buffer
 >    keywords, =#+BIND= keywords, buffer-local and global
 >    customizations.

The first comma is not necessary.  The option values are calculated
according to the list that follows (in-buffer keywords, BIND, etc.).

Remove the trailing period or add periods to all the others.  I tend
leave the period of the last sentence of a list.  I'm not sure of a
style guide that recommends one or the other.  Maybe someone knows
what's "right"?

 >     - Heading are removed according to =SELECT_TAGS= and
 >       =EXCLUDE_TAGS= export keywords; =task=, =inline=, =arch= export
 >       options

"Heading" should be "Headings"

 > 17. AST is transcoded according to the chosen export backend
 >     - The export happens recursively, depth-first
 >     - Each transcoded AST node, as a string, is passed to the
 >       corresponding export filter (see [[*Filters]])
 > 18. The transcoded AST body is formatted according to backend's
 >     "inner" template

Maybe use "converted" instead of "transcoded"?  I'm a native speaker
but I wonder if "converted" is a simpler word for people who aren't.

The sentence is simpler by removing the commas and rearranging,

  "Each transcoded AST node is passed as a string to the corresponding
  export filter (see [[*Filters]])""

What's an "inner" template and why does that matter?

 > 20. The filtered body is formatted according to backend's outer
 >     template

What's an "outer" template and why does it matter?

Should "outer" be quoted like "inner" was?  Or, should "inner" be
unquoted?

 > 21. The resulting output is processed by citation backend finalizer

"Finalizer" is vague.  What's its significance?  Does it correspond to
a function the end-user might look up?

Overall, great work!  For each "negative" bit of feedback, there were
at least two things that deserve "positive" comments.  I saw no
spelling errors, you wrote out abbreviations (AST) before using them,
and there are no voice issues.  Everything was to the point.

At first I was confused why this is being added to the manual and not
to Worg.  However, after reading it more thoroughly, I believe it will
be a welcome and helpful addition to the manual.  The export process
is complex!

--
Matt Trzcinski
Emacs Org contributor (ob-shell)
Learn more about Org mode at https://orgmode.org
Support Org development at https://liberapay.com/org-mode

Re: [PATCH] org-manual: Describe export process flow

[Matt]

 ---- On Tue, 26 Dec 2023 22:56:00 +0100  Matt  wrote --- 
 
 > I tend leave the period of the last sentence of a list.

Typo.  I meant to write "I tend TO leave the period OFF the last sentence of a list."  That is, the item ends without a period.

--
Matt Trzcinski
Emacs Org contributor (ob-shell)
Learn more about Org mode at https://orgmode.org
Support Org development at https://liberapay.com/org-mode

[PATCH v2] org-manual: Describe export process flow

[Ihor Radchenko]

[-- Attachment #1: Type: text/plain, Size: 3258 bytes --]

Ihor Radchenko <yantar92@posteo.net> writes:

> I'd like to add a new section to Org mode manual.
> The new section will describe all the steps performed by Org export
> process. This should hopefully create a more clear picture on how
> various export hooks and filters are used.
>
> The patch is attached.
> I'd appreciate feedback from people not familiar with ox.el.

Thank you all for the feedback!
I am attaching revised version of the patch with most of the comments
addressed.

I will put more detailed responses inline.


"Thomas S. Dye" <tsd@tsdye.online> writes:
> I'm not too familiar with ox.el.
>
> I edited mostly to use an active voice. I put author queries in 
> parentheses.  I haven't paid attention to manual formatting 
> conventions.
>
> IMO, more links would likely be helpful.
>
> * Suggested revision
> ...

I believe that I have addressed everything you commented on.

Karthik Chikmagalur <karthikchikmagalur@gmail.com> writes:
> - When exporting a sub-tree, at what stage of the export process is the
>   buffer narrowed to the sub-tree?

I added a clarification on subtree export now.

> - Are "inner" and "outer" templates described in the manual, and if they
>   are could you add a link to those sections when mentioning them in
>   this summary?  I only found references to the plist properties
>   BEAMER_INNER_THEME etc.

This is internal terminology. I changed the wording, expanding on what
inner and outer template do.

Matt <matt@excalamus.com> writes:
> Here are all the hooks and functions for org-export (via =C-h v
> org-export--hooks TAB= and =C-h v org-export--function TAB=).  I see
> 59 of them.
> ...
> * Feedback 1:
> How are the functions not present in the patch handled?

- I fixed the obsolete variable names.
- `org-export-stack-mode-hook' is not directly relevant to the export
  process - it is for asynchronous export listing buffer
- Syntax-specific filters are applied according to the corresponding Org
  syntax element. I tried to make it more clear.
- Special filters, like `org-export-before-parsing-functions' are
  described separately. I think I have mentioned all of them.

> I would write out "src" as "source".  Do we have an official way to
> refer to source blocks?  For example, we standardize on "Org":
> https://git.savannah.gnu.org/cgit/emacs/org-mode.git/tree/doc/Documentation_Standards.org#n47

We use "source block", "code block", and "source code block" across the
manual. Not "src block" though.
I went with "code block" in the patch.

I am not sure if it is necessary to standardize the above three terms.
They are all equally understandable I believe.

> Remove the trailing period or add periods to all the others.  I tend
> leave the period of the last sentence of a list.  I'm not sure of a
> style guide that recommends one or the other.  Maybe someone knows
> what's "right"?

No idea. We do not have a consistency in the manual either.  I went with
";" as Thomas suggested.

> Maybe use "converted" instead of "transcoded"?  I'm a native speaker
> but I wonder if "converted" is a simpler word for people who aren't.

"transcoded" is closer to what we use in the code. I tried to use
"converted" in more general description, but still hint on the
internal terminology.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: v2-0001-doc-org-manual.org-Fix-some-obsolete-variable-nam.patch --]
[-- Type: text/x-patch, Size: 1919 bytes --]

From efee8fb5e8aca473b1b80aacc2b38951421225cc Mon Sep 17 00:00:00 2001
Message-ID: <efee8fb5e8aca473b1b80aacc2b38951421225cc.1703683799.git.yantar92@posteo.net>
From: Ihor Radchenko <yantar92@posteo.net>
Date: Wed, 27 Dec 2023 14:23:29 +0100
Subject: [PATCH v2 1/2] doc/org-manual.org: Fix some obsolete variable names

* doc/org-manual.org (Export hooks): Use the new
`org-export-before-processing-functions' and
`org-export-before-parsing-functions' instead of their obsolete
aliases.
---
 doc/org-manual.org | 12 +++++++-----
 1 file changed, 7 insertions(+), 5 deletions(-)

diff --git a/doc/org-manual.org b/doc/org-manual.org
index 23f250fa7..b35a83434 100644
--- a/doc/org-manual.org
+++ b/doc/org-manual.org
@@ -16397,12 +16397,14 @@ *** Export hooks
 :END:
 
 #+vindex: org-export-before-processing-hook
+#+vindex: org-export-before-processing-functions
 #+vindex: org-export-before-parsing-hook
 The export process executes two hooks before the actual exporting
-begins.  The first hook, ~org-export-before-processing-hook~, runs
-before any expansions of macros, Babel code, and include keywords in
-the buffer.  The second hook, ~org-export-before-parsing-hook~, runs
-before the buffer is parsed.
+begins.  The first hook, ~org-export-before-processing-functions~,
+runs before any expansions of macros, Babel code, and include keywords
+in the buffer.  The second hook,
+~org-export-before-parsing-functions~, runs before the buffer is
+parsed.
 
 Functions added to these hooks are called with a single argument: the
 export backend actually used, as a symbol.  You may use them for
@@ -16421,7 +16423,7 @@ *** Export hooks
      ;; the docstring of `org-map-entries' for details.
      (setq org-map-continue-from (point)))))
 
-(add-hook 'org-export-before-parsing-hook #'my-headline-removal)
+(add-hook 'org-export-before-parsing-functions #'my-headline-removal)
 #+end_src
 
 *** Filters
-- 
2.42.0


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #3: v2-0002-doc-org-manual.org-Describe-export-flow.patch --]
[-- Type: text/x-patch, Size: 6504 bytes --]

From 704eda63d6abf847efcd79cd97e3560ba4729921 Mon Sep 17 00:00:00 2001
Message-ID: <704eda63d6abf847efcd79cd97e3560ba4729921.1703683799.git.yantar92@posteo.net>
In-Reply-To: <efee8fb5e8aca473b1b80aacc2b38951421225cc.1703683799.git.yantar92@posteo.net>
References: <efee8fb5e8aca473b1b80aacc2b38951421225cc.1703683799.git.yantar92@posteo.net>
From: Ihor Radchenko <yantar92@posteo.net>
Date: Tue, 26 Dec 2023 15:15:23 +0100
Subject: [PATCH v2 2/2] doc/org-manual.org: Describe export flow

* doc/org-manual.org (Summary of the export process): Explain how the
export process is handled in Org mode.
---
 doc/org-manual.org | 149 +++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 149 insertions(+)

diff --git a/doc/org-manual.org b/doc/org-manual.org
index b35a83434..12e7d031a 100644
--- a/doc/org-manual.org
+++ b/doc/org-manual.org
@@ -16505,6 +16505,155 @@ *** Defining filters for individual files
 ,#+END_SRC
 #+end_example
 
+*** Summary of the export process
+:PROPERTIES:
+:UNNUMBERED: notoc
+:END:
+
+Org mode export is a multi-step process that works on a temporary copy
+of the buffer.  On high-level, the export process consists of 4 major
+steps:
+
+1. Process the temporary copy, making necessary changes to the buffer
+   text;
+
+2. Parse the buffer, converting plain Org markup into abstract syntax
+   tree (AST);
+
+3. Convert the AST to text, as prescribed by the selected export
+   backend;
+
+4. Post-process the resulting exported text.
+
+
+#+texinfo: @noindent
+Process temporary copy of the source Org buffer [fn::Unless
+otherwise specified, each step of the export process only operates on
+the accessible portion of the buffer.  When subtree export is selected
+(see [[*The Export Dispatcher]]), the buffer is narrowed to the body of
+the selected subtree, so that the rest of the buffer text, except
+export keywords, does not contribute to the export output.]:
+
+1. Execute ~org-export-before-processing-functions~ (see [[*Export hooks]]);
+
+2. Expand =#+include= keywords in the whole buffer (see
+   [[*Include Files]]);
+
+3. Remove commented subtrees in the whole buffer (see [[*Comment
+   Lines]]);
+
+4. Replace macros in the whole buffer (see [[*Macro Replacement]]);
+
+5. When ~org-export-use-babel~ is non-nil (default), process code
+   blocks:
+
+   - Leave code blocks inside archived subtrees (see [[*Internal
+     archiving]]) as is;
+
+   - Evaluate all the other code blocks according to code block
+     headers (see [[*Limit code block evaluation]]);
+
+   - Remove code, results of evaluation, both, or neither according
+     to =:exports= header argument (see [[*Exporting Code Blocks]]).
+
+
+#+texinfo: @noindent
+Parse the temporary buffer, creating AST:
+
+1. Execute ~org-export-before-parsing-functions~ (see [[*Export hooks]]).
+   The hook functions may still modify the buffer;
+
+2. Calculate export option values according to subtree-specific export
+   settings, in-buffer keywords, =#+BIND= keywords, and buffer-local
+   and global customization.  The whole buffer is considered;
+
+3. Determine contributing bibliographies and record them into export
+   options (see [[*Citations]]).  The whole buffer is considered;
+
+4. Execute ~org-export-filter-options-functions~;
+
+5. Parse the accessible portion of the temporary buffer to generate
+   AST.  The AST is a nested list of lists representing Org syntax
+   elements (see [[https://orgmode.org/worg/dev/org-element-api.html][Org Element API]] for more details):
+
+   : (org-data ...
+   :  (heading
+   :   (section
+   :    (paragraph (plain-text) (bold (plain-text))))
+   :   (heading)
+   :   (heading (section ...))))
+
+   Past this point, modifications in the temporary buffer copy no
+   longer affect export; Org export works only with the AST;
+
+6. Remove elements that will not be exported from the AST:
+
+   - Headings according to =SELECT_TAGS= and =EXCLUDE_TAGS= export
+     keywords, and =task=, =inline=, =arch= export options (see
+     [[*Export Settings]]);
+
+   - Comments;
+
+   - Clocks, drawers, fixed-width environments, footnotes, LaTeX
+     environments and fragments, node properties, planning lines,
+     property drawers, statistics cookies, timestamps, timestamps,
+     etc according to =#+OPTIONS= keyword (see [[*Export Settings]]);
+
+   - Table rows containing width and alignment markers (see [[*Column
+     Width and Alignment]]);
+
+     - Table columns containing recalc marks (see [[*Advanced features]]).
+
+7. Expand environment variables in file link AST nodes, according to
+   the =expand-links= export option (see [[*Export Settings]]);
+
+8. Execute ~org-export-filter-parse-tree-functions~.  These
+   functions can modify AST by side effect;
+
+9. Replace citation AST nodes and =#+print_bibliography= keyword AST
+   nodes as prescribed by the selected citation export processor
+   (see [[*Citation export processors]]).
+
+
+#+texinfo: @noindent
+Convert the AST to text by traversing the AST nodes, depth-first:
+
+1. Convert the leaf nodes (without children) to text as prescribed
+   by "transcoders" in the selected export backend
+   [fn:: See transcoders and ~:translate-alist~ in the docstrings
+   of ~org-export-define-backend~ and ~org-export-define-derived-backend~.];
+
+2. Pass the converted nodes through the corresponding export
+   filters (see [[*Filters]]);
+
+3. Concatenate all the converted child nodes to produce parent
+   node contents;
+
+4. Convert the nodes with children to text, passing the nodes
+   themselves and their contents to the corresponding transcoders
+   and then to export filters (see [[*Filters]]).
+
+
+#+texinfo: @noindent
+Post-process the exported text:
+
+  1. Post-process the converted AST, as prescribed by the export
+     backend. [fn:: See ~inner-template~ in the docstring of ~org-export-define-backend~.]
+     This step usually adds generated content (like Table of Contents)
+     to the exported text;
+
+  2. Execute ~org-export-filter-body-functions~;
+
+  3. Unless body-only export is selected (see [[*The Export Dispatcher]]),
+     add the necessary metadata to the final document, as prescribed
+     by the export backend.  Examples: Document author/title; HTML
+     headers/footers; LaTeX preamble;
+
+  4. Add bibliography metadata, as prescribed by the citation export
+     processor;
+
+  5. Execute ~org-export-filter-final-output-functions~.
+
 *** Extending an existing backend
 :PROPERTIES:
 :UNNUMBERED: notoc
-- 
2.42.0


[-- Attachment #4: Type: text/plain, Size: 224 bytes --]


-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [PATCH v2] org-manual: Describe export process flow

[Matt]

 ---- On Wed, 27 Dec 2023 14:41:32 +0100  Ihor Radchenko  wrote --- 
 
 > Thank you all for the feedback!

Thanks for taking the time to write and revise the patches.

You didn't explicitly ask for more feedback, so I hope the following feedback on the new patches is in order.

Both patches look good.  I see no issues with
v2-0001-doc-org-manual.org-Fix-some-obsolete-variable-nam.patch

For v2-0002-doc-org-manual.org-Describe-export-flow.patch here are
some quibbles:

> On high-level, the export process consists of 4 major steps:

I don't think "On high level" is necessary (it's also missing an 'a';
"On *a* high level").  "Major" implies high-level.

"The export process consists of 4 major steps:"

If you wanted to keep "high-level", you could replace major with it:

"The export process consists of 4 high-level steps:"

 > 2. Parse the buffer, converting plain Org markup into abstract syntax
 >    tree (AST);

Missing "an":

"...converting plain Org markup into *an* abstract syntax..."

 > 5. Parse the accessible portion of the temporary buffer to generate
 >    AST.  The AST is a nested list of lists representing Org syntax
 >    elements

"...of the temporary buffer to generate *the* AST."

or

"...of the temporary buffer to generate *an* AST."

 >    Past this point, modifications in the temporary buffer copy no
 >    longer affect export; Org export works only with the AST;

Replace "in the" with "to the".

"no longer affect export" -> "no longer affect *the* export"

Replace "temporary buffer copy" with "temporary buffer" or "temporary
copy".

"Past this point, modifications to the temporary buffer no longer
affect the export; Org export works only with the AST;"

> 6. Remove elements that will not be exported from the AST:

"Remove elements that are not exported from the AST:"

or

"Remove elements not exported from the AST:"

 >    - Headings according to =SELECT_TAGS= and =EXCLUDE_TAGS= export
 >      keywords, and =task=, =inline=, =arch= export options (see
 >      [[*Export Settings]]);

Remove comma after "keywords":

"Headings according to =SELECT_TAGS= and =EXCLUDE_TAGS= export
keywords and =task=, =inline=, =arch= export options (see [[*Export
Settings]]);"

 >    - Clocks, drawers, fixed-width environments, footnotes, LaTeX
 >      environments and fragments, node properties, planning lines,
 >      property drawers, statistics cookies, timestamps, timestamps,
 >      etc according to =#+OPTIONS= keyword (see [[*Export Settings]]);

"timestamps" is written twice.

>     - Table columns containing recalc marks (see [[*Advanced features]]).

This is indented two spaces more than the previous item.  Is that what you
want?

 > 7. Expand environment variables in file link AST nodes, according to
 >    the =expand-links= export option (see [[*Export Settings]]);

Comma after "nodes" not necessary.

"Expand environment variables in file link AST nodes according to the
=expand-links= export option (see [[*Export Settings]]);"

 > 8. Execute ~org-export-filter-parse-tree-functions~.  These
 >    functions can modify AST by side effect;

I would say "*the* AST" and make "side effect" plural:

"...can modify *the* AST by side effect*s*;"

 > 4. Convert the nodes with children to text, passing the nodes
 >    themselves and their contents to the corresponding transcoders
 >    and then to export filters (see [[*Filters]]).

Add a "the":

"...and then to *the* export filters (see [[*Filters]])."

-----

Checking ox.el, I see that the commentary says,

";; See <https://orgmode.org/worg/dev/org-export-reference.html> for
;; more information."

Maybe also let maintainers know about this manual section:

"See:
<https://orgmode.org/worg/dev/org-export-reference.html>
<link to Summary of the export process>"

-----

Again, good work.  AFAIU, you used "affect" and "effect" correctly.
Breaking the big list into smaller ones makes it much easier to read.

--
Matt Trzcinski
Emacs Org contributor (ob-shell)
Learn more about Org mode at https://orgmode.org
Support Org development at https://liberapay.com/org-mode

[PATCH v3] org-manual: Describe export process flow

[Ihor Radchenko]

[-- Attachment #1: Type: text/plain, Size: 1920 bytes --]

Matt <matt@excalamus.com> writes:

>  > Thank you all for the feedback!
>
> Thanks for taking the time to write and revise the patches.
>
> You didn't explicitly ask for more feedback, so I hope the following feedback on the new patches is in order.

Of course. Any patch posted on the mailing list is open to feedback.
Especially mine, since I can push it directly if I think that the patch
is in a good shape.

> For v2-0002-doc-org-manual.org-Describe-export-flow.patch here are
> some quibbles:

Thanks for proofreading!
I accepted all but one suggestion on grammar.
The new iteration of the patch is attached.

>  >    - Headings according to =SELECT_TAGS= and =EXCLUDE_TAGS= export
>  >      keywords, and =task=, =inline=, =arch= export options (see
>  >      [[*Export Settings]]);
>
> Remove comma after "keywords":
>
> "Headings according to =SELECT_TAGS= and =EXCLUDE_TAGS= export
> keywords and =task=, =inline=, =arch= export options (see [[*Export
> Settings]]);"

Here, "export keywords" and "export options" are two different groups. I
wanted to indicate this and went with ";":

   - Headings according to =SELECT_TAGS= and =EXCLUDE_TAGS= export
     keywords; =task=, =inline=, =arch= export options (see
     [[*Export Settings]]);

> Checking ox.el, I see that the commentary says,
>
> ";; See <https://orgmode.org/worg/dev/org-export-reference.html> for
> ;; more information."
>
> Maybe also let maintainers know about this manual section:
>
> "See:
> <https://orgmode.org/worg/dev/org-export-reference.html>
> <link to Summary of the export process>"

AFAIU, you are asking to add it to WORG page. However, it is too early.
The online Org manual that we can link to is only for stable Org
version. So, only after the next release.

For now, I added a link to the WORG page to "Extending an existing
backend" section. It should be a good reference for people writing
non-trivial derived backends.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: v3-0001-doc-org-manual.org-Fix-some-obsolete-variable-nam.patch --]
[-- Type: text/x-patch, Size: 1919 bytes --]

From 6f6a1a190724176da7373e3d25f9df0bf60dbac9 Mon Sep 17 00:00:00 2001
Message-ID: <6f6a1a190724176da7373e3d25f9df0bf60dbac9.1703696392.git.yantar92@posteo.net>
From: Ihor Radchenko <yantar92@posteo.net>
Date: Wed, 27 Dec 2023 14:23:29 +0100
Subject: [PATCH v3 1/3] doc/org-manual.org: Fix some obsolete variable names

* doc/org-manual.org (Export hooks): Use the new
`org-export-before-processing-functions' and
`org-export-before-parsing-functions' instead of their obsolete
aliases.
---
 doc/org-manual.org | 12 +++++++-----
 1 file changed, 7 insertions(+), 5 deletions(-)

diff --git a/doc/org-manual.org b/doc/org-manual.org
index ff1b9cffb..377706ee7 100644
--- a/doc/org-manual.org
+++ b/doc/org-manual.org
@@ -16397,12 +16397,14 @@ *** Export hooks
 :END:
 
 #+vindex: org-export-before-processing-hook
+#+vindex: org-export-before-processing-functions
 #+vindex: org-export-before-parsing-hook
 The export process executes two hooks before the actual exporting
-begins.  The first hook, ~org-export-before-processing-hook~, runs
-before any expansions of macros, Babel code, and include keywords in
-the buffer.  The second hook, ~org-export-before-parsing-hook~, runs
-before the buffer is parsed.
+begins.  The first hook, ~org-export-before-processing-functions~,
+runs before any expansions of macros, Babel code, and include keywords
+in the buffer.  The second hook,
+~org-export-before-parsing-functions~, runs before the buffer is
+parsed.
 
 Functions added to these hooks are called with a single argument: the
 export backend actually used, as a symbol.  You may use them for
@@ -16421,7 +16423,7 @@ *** Export hooks
      ;; the docstring of `org-map-entries' for details.
      (setq org-map-continue-from (point)))))
 
-(add-hook 'org-export-before-parsing-hook #'my-headline-removal)
+(add-hook 'org-export-before-parsing-functions #'my-headline-removal)
 #+end_src
 
 *** Filters
-- 
2.42.0


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #3: v3-0002-doc-org-manual.org-Describe-export-flow.patch --]
[-- Type: text/x-patch, Size: 6479 bytes --]

From 786e3e78d4c6846188477452a529132aba76149f Mon Sep 17 00:00:00 2001
Message-ID: <786e3e78d4c6846188477452a529132aba76149f.1703696392.git.yantar92@posteo.net>
In-Reply-To: <6f6a1a190724176da7373e3d25f9df0bf60dbac9.1703696392.git.yantar92@posteo.net>
References: <6f6a1a190724176da7373e3d25f9df0bf60dbac9.1703696392.git.yantar92@posteo.net>
From: Ihor Radchenko <yantar92@posteo.net>
Date: Tue, 26 Dec 2023 15:15:23 +0100
Subject: [PATCH v3 2/3] doc/org-manual.org: Describe export flow

* doc/org-manual.org (Summary of the export process): Explain how the
export process is handled in Org mode.
---
 doc/org-manual.org | 148 +++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 148 insertions(+)

diff --git a/doc/org-manual.org b/doc/org-manual.org
index 377706ee7..66a582eae 100644
--- a/doc/org-manual.org
+++ b/doc/org-manual.org
@@ -16505,6 +16505,154 @@ *** Defining filters for individual files
 ,#+END_SRC
 #+end_example
 
+*** Summary of the export process
+:PROPERTIES:
+:UNNUMBERED: notoc
+:END:
+
+Org mode export is a multi-step process that works on a temporary copy
+of the buffer.  The export process consists of 4 major steps:
+
+1. Process the temporary copy, making necessary changes to the buffer
+   text;
+
+2. Parse the buffer, converting plain Org markup into an abstract
+   syntax tree (AST);
+
+3. Convert the AST to text, as prescribed by the selected export
+   backend;
+
+4. Post-process the resulting exported text.
+
+
+#+texinfo: @noindent
+Process temporary copy of the source Org buffer [fn::Unless
+otherwise specified, each step of the export process only operates on
+the accessible portion of the buffer.  When subtree export is selected
+(see [[*The Export Dispatcher]]), the buffer is narrowed to the body of
+the selected subtree, so that the rest of the buffer text, except
+export keywords, does not contribute to the export output.]:
+
+1. Execute ~org-export-before-processing-functions~ (see [[*Export hooks]]);
+
+2. Expand =#+include= keywords in the whole buffer (see
+   [[*Include Files]]);
+
+3. Remove commented subtrees in the whole buffer (see [[*Comment
+   Lines]]);
+
+4. Replace macros in the whole buffer (see [[*Macro Replacement]]);
+
+5. When ~org-export-use-babel~ is non-nil (default), process code
+   blocks:
+
+   - Leave code blocks inside archived subtrees (see [[*Internal
+     archiving]]) as is;
+
+   - Evaluate all the other code blocks according to code block
+     headers (see [[*Limit code block evaluation]]);
+
+   - Remove code, results of evaluation, both, or neither according
+     to =:exports= header argument (see [[*Exporting Code Blocks]]).
+
+
+#+texinfo: @noindent
+Parse the temporary buffer, creating AST:
+
+1. Execute ~org-export-before-parsing-functions~ (see [[*Export hooks]]).
+   The hook functions may still modify the buffer;
+
+2. Calculate export option values according to subtree-specific export
+   settings, in-buffer keywords, =#+BIND= keywords, and buffer-local
+   and global customization.  The whole buffer is considered;
+
+3. Determine contributing bibliographies and record them into export
+   options (see [[*Citations]]).  The whole buffer is considered;
+
+4. Execute ~org-export-filter-options-functions~;
+
+5. Parse the accessible portion of the temporary buffer to generate an
+   AST.  The AST is a nested list of lists representing Org syntax
+   elements (see [[https://orgmode.org/worg/dev/org-element-api.html][Org Element API]] for more details):
+
+   : (org-data ...
+   :  (heading
+   :   (section
+   :    (paragraph (plain-text) (bold (plain-text))))
+   :   (heading)
+   :   (heading (section ...))))
+
+   Past this point, modifications to the temporary buffer no longer
+   affect the export; Org export works only with the AST;
+
+6. Remove elements that are not exported from the AST:
+
+   - Headings according to =SELECT_TAGS= and =EXCLUDE_TAGS= export
+     keywords; =task=, =inline=, =arch= export options (see
+     [[*Export Settings]]);
+
+   - Comments;
+
+   - Clocks, drawers, fixed-width environments, footnotes, LaTeX
+     environments and fragments, node properties, planning lines,
+     property drawers, statistics cookies, timestamps, etc according
+     to =#+OPTIONS= keyword (see [[*Export Settings]]);
+
+   - Table rows containing width and alignment markers (see [[*Column
+     Width and Alignment]]);
+
+   - Table columns containing recalc marks (see [[*Advanced features]]).
+
+7. Expand environment variables in file link AST nodes according to
+   the =expand-links= export option (see [[*Export Settings]]);
+
+8. Execute ~org-export-filter-parse-tree-functions~.  These
+   functions can modify the AST by side effects;
+
+9. Replace citation AST nodes and =#+print_bibliography= keyword AST
+   nodes as prescribed by the selected citation export processor
+   (see [[*Citation export processors]]).
+
+
+#+texinfo: @noindent
+Convert the AST to text by traversing the AST nodes, depth-first:
+
+1. Convert the leaf nodes (without children) to text as prescribed
+   by "transcoders" in the selected export backend
+   [fn:: See transcoders and ~:translate-alist~ in the docstrings
+   of ~org-export-define-backend~ and ~org-export-define-derived-backend~.];
+
+2. Pass the converted nodes through the corresponding export
+   filters (see [[*Filters]]);
+
+3. Concatenate all the converted child nodes to produce parent
+   node contents;
+
+4. Convert the nodes with children to text, passing the nodes
+   themselves and their contents to the corresponding transcoders
+   and then to the export filters (see [[*Filters]]).
+
+
+#+texinfo: @noindent
+Post-process the exported text:
+
+  1. Post-process the converted AST, as prescribed by the export
+     backend. [fn:: See ~inner-template~ in the docstring of ~org-export-define-backend~.]
+     This step usually adds generated content (like Table of Contents)
+     to the exported text;
+
+  2. Execute ~org-export-filter-body-functions~;
+
+  3. Unless body-only export is selected (see [[*The Export Dispatcher]]),
+     add the necessary metadata to the final document, as prescribed
+     by the export backend.  Examples: Document author/title; HTML
+     headers/footers; LaTeX preamble;
+
+  4. Add bibliography metadata, as prescribed by the citation export
+     processor;
+
+  5. Execute ~org-export-filter-final-output-functions~.
+
 *** Extending an existing backend
 :PROPERTIES:
 :UNNUMBERED: notoc
-- 
2.42.0


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #4: v3-0003-doc-org-manual.org-Add-link-to-WORG-export-refere.patch --]
[-- Type: text/x-patch, Size: 1227 bytes --]

From 66d2b5f68e0a4c62be3ed65a8d06300c0c2c5083 Mon Sep 17 00:00:00 2001
Message-ID: <66d2b5f68e0a4c62be3ed65a8d06300c0c2c5083.1703696392.git.yantar92@posteo.net>
In-Reply-To: <6f6a1a190724176da7373e3d25f9df0bf60dbac9.1703696392.git.yantar92@posteo.net>
References: <6f6a1a190724176da7373e3d25f9df0bf60dbac9.1703696392.git.yantar92@posteo.net>
From: Ihor Radchenko <yantar92@posteo.net>
Date: Wed, 27 Dec 2023 17:58:48 +0100
Subject: [PATCH v3 3/3] doc/org-manual.org: Add link to WORG export reference
 page

* doc/org-manual.org (Extending an existing backend): Link to WORG
page describing more details about writing export backends.
---
 doc/org-manual.org | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/doc/org-manual.org b/doc/org-manual.org
index 66a582eae..bb1bad673 100644
--- a/doc/org-manual.org
+++ b/doc/org-manual.org
@@ -16709,7 +16709,9 @@ *** Extending an existing backend
 
 Further steps to consider would be an interactive function,
 self-installing an item in the export dispatcher menu, and other
-user-friendly improvements.
+user-friendly improvements.  See
+<https://orgmode.org/worg/dev/org-export-reference.html> for more
+details.
 
 ** Export Region
 :PROPERTIES:
-- 
2.42.0


[-- Attachment #5: Type: text/plain, Size: 224 bytes --]


-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [PATCH v3] org-manual: Describe export process flow

[Matt]

 ---- On Wed, 27 Dec 2023 18:05:05 +0100  Ihor Radchenko  wrote --- 
 
 > Matt matt@excalamus.com> writes:
 > > Checking ox.el, I see that the commentary says,
 > >
 > > ";; See https://orgmode.org/worg/dev/org-export-reference.html> for
 > > ;; more information."
 > >
 > > Maybe also let maintainers know about this manual section:
 > >
 > > "See:
 > >  https://orgmode.org/worg/dev/org-export-reference.html
 > >  some-kind-of-reference-to-the-new-manual-section
 > > "
 > 
 > AFAIU, you are asking to add it to WORG page. However, it is too early.
 > The online Org manual that we can link to is only for stable Org
 > version. So, only after the next release.

I was making a different point.  I was trying to say that it may be a
good idea for the ox.el commentary section to specifically reference
the new manual section (next to where it references Worg).

The new manual section is unique in that it gives detailed steps about
the order of execution.  Without knowing the code (so I may be
mistaken), it seems like a risk that the ox.el code and manual could
get out of sync.  This is why I wouldn't suggest documenting the
execution order for something like source blocks.  However, since
end-users have a lot of flexiblity in customizing the export process,
it's helpful for them to understand it.  It makes sense to add the new
manual section.

Mentioning the new manual section in the ox.el commentary would help
someone new to the ox.el code.  It may also help keep the new manual
section and ox.el aligned (should anything in the execution
order/process change in ox.el) by reminding someone changing the code
that there's a manual section specifically about *how* the code works.

--
Matt Trzcinski
Emacs Org contributor (ob-shell)
Learn more about Org mode at https://orgmode.org
Support Org development at https://liberapay.com/org-mode

[PATCH v4] org-manual: Describe export process flow

[Ihor Radchenko]

[-- Attachment #1: Type: text/plain, Size: 434 bytes --]

Matt <matt@excalamus.com> writes:

> I was making a different point.  I was trying to say that it may be a
> good idea for the ox.el commentary section to specifically reference
> the new manual section (next to where it references Worg).

Makes sense. See the attached.
I added reference to the manual in `org-export-as' docstring and made it
explicit that the manual section is related to `org-export-as' by adding
#+findex entry.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: v4-0001-doc-org-manual.org-Fix-some-obsolete-variable-nam.patch --]
[-- Type: text/x-patch, Size: 1919 bytes --]

From b17d57c1afeb160783da36647f56462ee67c3558 Mon Sep 17 00:00:00 2001
Message-ID: <b17d57c1afeb160783da36647f56462ee67c3558.1703765064.git.yantar92@posteo.net>
From: Ihor Radchenko <yantar92@posteo.net>
Date: Wed, 27 Dec 2023 14:23:29 +0100
Subject: [PATCH v4 1/5] doc/org-manual.org: Fix some obsolete variable names

* doc/org-manual.org (Export hooks): Use the new
`org-export-before-processing-functions' and
`org-export-before-parsing-functions' instead of their obsolete
aliases.
---
 doc/org-manual.org | 12 +++++++-----
 1 file changed, 7 insertions(+), 5 deletions(-)

diff --git a/doc/org-manual.org b/doc/org-manual.org
index ff1b9cffb..377706ee7 100644
--- a/doc/org-manual.org
+++ b/doc/org-manual.org
@@ -16397,12 +16397,14 @@ *** Export hooks
 :END:
 
 #+vindex: org-export-before-processing-hook
+#+vindex: org-export-before-processing-functions
 #+vindex: org-export-before-parsing-hook
 The export process executes two hooks before the actual exporting
-begins.  The first hook, ~org-export-before-processing-hook~, runs
-before any expansions of macros, Babel code, and include keywords in
-the buffer.  The second hook, ~org-export-before-parsing-hook~, runs
-before the buffer is parsed.
+begins.  The first hook, ~org-export-before-processing-functions~,
+runs before any expansions of macros, Babel code, and include keywords
+in the buffer.  The second hook,
+~org-export-before-parsing-functions~, runs before the buffer is
+parsed.
 
 Functions added to these hooks are called with a single argument: the
 export backend actually used, as a symbol.  You may use them for
@@ -16421,7 +16423,7 @@ *** Export hooks
      ;; the docstring of `org-map-entries' for details.
      (setq org-map-continue-from (point)))))
 
-(add-hook 'org-export-before-parsing-hook #'my-headline-removal)
+(add-hook 'org-export-before-parsing-functions #'my-headline-removal)
 #+end_src
 
 *** Filters
-- 
2.42.0


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #3: v4-0002-doc-org-manual.org-Describe-export-flow.patch --]
[-- Type: text/x-patch, Size: 6479 bytes --]

From b195caf687793af228a2ea9a484f2fb8cb87ddab Mon Sep 17 00:00:00 2001
Message-ID: <b195caf687793af228a2ea9a484f2fb8cb87ddab.1703765065.git.yantar92@posteo.net>
In-Reply-To: <b17d57c1afeb160783da36647f56462ee67c3558.1703765064.git.yantar92@posteo.net>
References: <b17d57c1afeb160783da36647f56462ee67c3558.1703765064.git.yantar92@posteo.net>
From: Ihor Radchenko <yantar92@posteo.net>
Date: Tue, 26 Dec 2023 15:15:23 +0100
Subject: [PATCH v4 2/5] doc/org-manual.org: Describe export flow

* doc/org-manual.org (Summary of the export process): Explain how the
export process is handled in Org mode.
---
 doc/org-manual.org | 148 +++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 148 insertions(+)

diff --git a/doc/org-manual.org b/doc/org-manual.org
index 377706ee7..66a582eae 100644
--- a/doc/org-manual.org
+++ b/doc/org-manual.org
@@ -16505,6 +16505,154 @@ *** Defining filters for individual files
 ,#+END_SRC
 #+end_example
 
+*** Summary of the export process
+:PROPERTIES:
+:UNNUMBERED: notoc
+:END:
+
+Org mode export is a multi-step process that works on a temporary copy
+of the buffer.  The export process consists of 4 major steps:
+
+1. Process the temporary copy, making necessary changes to the buffer
+   text;
+
+2. Parse the buffer, converting plain Org markup into an abstract
+   syntax tree (AST);
+
+3. Convert the AST to text, as prescribed by the selected export
+   backend;
+
+4. Post-process the resulting exported text.
+
+
+#+texinfo: @noindent
+Process temporary copy of the source Org buffer [fn::Unless
+otherwise specified, each step of the export process only operates on
+the accessible portion of the buffer.  When subtree export is selected
+(see [[*The Export Dispatcher]]), the buffer is narrowed to the body of
+the selected subtree, so that the rest of the buffer text, except
+export keywords, does not contribute to the export output.]:
+
+1. Execute ~org-export-before-processing-functions~ (see [[*Export hooks]]);
+
+2. Expand =#+include= keywords in the whole buffer (see
+   [[*Include Files]]);
+
+3. Remove commented subtrees in the whole buffer (see [[*Comment
+   Lines]]);
+
+4. Replace macros in the whole buffer (see [[*Macro Replacement]]);
+
+5. When ~org-export-use-babel~ is non-nil (default), process code
+   blocks:
+
+   - Leave code blocks inside archived subtrees (see [[*Internal
+     archiving]]) as is;
+
+   - Evaluate all the other code blocks according to code block
+     headers (see [[*Limit code block evaluation]]);
+
+   - Remove code, results of evaluation, both, or neither according
+     to =:exports= header argument (see [[*Exporting Code Blocks]]).
+
+
+#+texinfo: @noindent
+Parse the temporary buffer, creating AST:
+
+1. Execute ~org-export-before-parsing-functions~ (see [[*Export hooks]]).
+   The hook functions may still modify the buffer;
+
+2. Calculate export option values according to subtree-specific export
+   settings, in-buffer keywords, =#+BIND= keywords, and buffer-local
+   and global customization.  The whole buffer is considered;
+
+3. Determine contributing bibliographies and record them into export
+   options (see [[*Citations]]).  The whole buffer is considered;
+
+4. Execute ~org-export-filter-options-functions~;
+
+5. Parse the accessible portion of the temporary buffer to generate an
+   AST.  The AST is a nested list of lists representing Org syntax
+   elements (see [[https://orgmode.org/worg/dev/org-element-api.html][Org Element API]] for more details):
+
+   : (org-data ...
+   :  (heading
+   :   (section
+   :    (paragraph (plain-text) (bold (plain-text))))
+   :   (heading)
+   :   (heading (section ...))))
+
+   Past this point, modifications to the temporary buffer no longer
+   affect the export; Org export works only with the AST;
+
+6. Remove elements that are not exported from the AST:
+
+   - Headings according to =SELECT_TAGS= and =EXCLUDE_TAGS= export
+     keywords; =task=, =inline=, =arch= export options (see
+     [[*Export Settings]]);
+
+   - Comments;
+
+   - Clocks, drawers, fixed-width environments, footnotes, LaTeX
+     environments and fragments, node properties, planning lines,
+     property drawers, statistics cookies, timestamps, etc according
+     to =#+OPTIONS= keyword (see [[*Export Settings]]);
+
+   - Table rows containing width and alignment markers (see [[*Column
+     Width and Alignment]]);
+
+   - Table columns containing recalc marks (see [[*Advanced features]]).
+
+7. Expand environment variables in file link AST nodes according to
+   the =expand-links= export option (see [[*Export Settings]]);
+
+8. Execute ~org-export-filter-parse-tree-functions~.  These
+   functions can modify the AST by side effects;
+
+9. Replace citation AST nodes and =#+print_bibliography= keyword AST
+   nodes as prescribed by the selected citation export processor
+   (see [[*Citation export processors]]).
+
+
+#+texinfo: @noindent
+Convert the AST to text by traversing the AST nodes, depth-first:
+
+1. Convert the leaf nodes (without children) to text as prescribed
+   by "transcoders" in the selected export backend
+   [fn:: See transcoders and ~:translate-alist~ in the docstrings
+   of ~org-export-define-backend~ and ~org-export-define-derived-backend~.];
+
+2. Pass the converted nodes through the corresponding export
+   filters (see [[*Filters]]);
+
+3. Concatenate all the converted child nodes to produce parent
+   node contents;
+
+4. Convert the nodes with children to text, passing the nodes
+   themselves and their contents to the corresponding transcoders
+   and then to the export filters (see [[*Filters]]).
+
+
+#+texinfo: @noindent
+Post-process the exported text:
+
+  1. Post-process the converted AST, as prescribed by the export
+     backend. [fn:: See ~inner-template~ in the docstring of ~org-export-define-backend~.]
+     This step usually adds generated content (like Table of Contents)
+     to the exported text;
+
+  2. Execute ~org-export-filter-body-functions~;
+
+  3. Unless body-only export is selected (see [[*The Export Dispatcher]]),
+     add the necessary metadata to the final document, as prescribed
+     by the export backend.  Examples: Document author/title; HTML
+     headers/footers; LaTeX preamble;
+
+  4. Add bibliography metadata, as prescribed by the citation export
+     processor;
+
+  5. Execute ~org-export-filter-final-output-functions~.
+
 *** Extending an existing backend
 :PROPERTIES:
 :UNNUMBERED: notoc
-- 
2.42.0


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #4: v4-0003-doc-org-manual.org-Add-link-to-WORG-export-refere.patch --]
[-- Type: text/x-patch, Size: 1227 bytes --]

From bbd6f3613d3d8ca1196c258ee1fcc9ec4df1d9a1 Mon Sep 17 00:00:00 2001
Message-ID: <bbd6f3613d3d8ca1196c258ee1fcc9ec4df1d9a1.1703765065.git.yantar92@posteo.net>
In-Reply-To: <b17d57c1afeb160783da36647f56462ee67c3558.1703765064.git.yantar92@posteo.net>
References: <b17d57c1afeb160783da36647f56462ee67c3558.1703765064.git.yantar92@posteo.net>
From: Ihor Radchenko <yantar92@posteo.net>
Date: Wed, 27 Dec 2023 17:58:48 +0100
Subject: [PATCH v4 3/5] doc/org-manual.org: Add link to WORG export reference
 page

* doc/org-manual.org (Extending an existing backend): Link to WORG
page describing more details about writing export backends.
---
 doc/org-manual.org | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/doc/org-manual.org b/doc/org-manual.org
index 66a582eae..bb1bad673 100644
--- a/doc/org-manual.org
+++ b/doc/org-manual.org
@@ -16709,7 +16709,9 @@ *** Extending an existing backend
 
 Further steps to consider would be an interactive function,
 self-installing an item in the export dispatcher menu, and other
-user-friendly improvements.
+user-friendly improvements.  See
+<https://orgmode.org/worg/dev/org-export-reference.html> for more
+details.
 
 ** Export Region
 :PROPERTIES:
-- 
2.42.0


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #5: v4-0004-org-export-as-Add-reference-to-export-flow-in-the.patch --]
[-- Type: text/x-patch, Size: 1131 bytes --]

From 5ad0e8d6e05a312960ea006958f986097441cd5a Mon Sep 17 00:00:00 2001
Message-ID: <5ad0e8d6e05a312960ea006958f986097441cd5a.1703765065.git.yantar92@posteo.net>
In-Reply-To: <b17d57c1afeb160783da36647f56462ee67c3558.1703765064.git.yantar92@posteo.net>
References: <b17d57c1afeb160783da36647f56462ee67c3558.1703765064.git.yantar92@posteo.net>
From: Ihor Radchenko <yantar92@posteo.net>
Date: Thu, 28 Dec 2023 12:59:22 +0100
Subject: [PATCH v4 4/5] org-export-as: Add reference to export flow in the
 manual

* lisp/ox.el (org-export-as): Update docstring.
---
 lisp/ox.el | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/lisp/ox.el b/lisp/ox.el
index 46994f0e2..2cf364227 100644
--- a/lisp/ox.el
+++ b/lisp/ox.el
@@ -2923,6 +2923,9 @@ (defun org-export-as
     (backend &optional subtreep visible-only body-only ext-plist)
   "Transcode current Org buffer into BACKEND code.
 
+See info node `(org)Advanced Export Configuration' for the details of
+the transcoding process.
+
 BACKEND is either an export backend, as returned by, e.g.,
 `org-export-create-backend', or a symbol referring to
 a registered backend.
-- 
2.42.0


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #6: v4-0005-doc-org-manual.org-Summary-of-the-export-process-.patch --]
[-- Type: text/x-patch, Size: 988 bytes --]

From 1c89b2e32816c0ad95d344aa6b05a2ad65b2d172 Mon Sep 17 00:00:00 2001
Message-ID: <1c89b2e32816c0ad95d344aa6b05a2ad65b2d172.1703765065.git.yantar92@posteo.net>
In-Reply-To: <b17d57c1afeb160783da36647f56462ee67c3558.1703765064.git.yantar92@posteo.net>
References: <b17d57c1afeb160783da36647f56462ee67c3558.1703765064.git.yantar92@posteo.net>
From: Ihor Radchenko <yantar92@posteo.net>
Date: Thu, 28 Dec 2023 13:04:06 +0100
Subject: [PATCH v4 5/5] doc/org-manual.org (Summary of the export process):
 Reference `org-export-as'

---
 doc/org-manual.org | 1 +
 1 file changed, 1 insertion(+)

diff --git a/doc/org-manual.org b/doc/org-manual.org
index bb1bad673..de2194868 100644
--- a/doc/org-manual.org
+++ b/doc/org-manual.org
@@ -16510,6 +16510,7 @@ *** Summary of the export process
 :UNNUMBERED: notoc
 :END:
 
+#+findex: org-export-as
 Org mode export is a multi-step process that works on a temporary copy
 of the buffer.  The export process consists of 4 major steps:
 
-- 
2.42.0


[-- Attachment #7: Type: text/plain, Size: 224 bytes --]


-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [PATCH v4] org-manual: Describe export process flow

[Matt]

 ---- On Thu, 28 Dec 2023 13:05:27 +0100  Ihor Radchenko  wrote --- 

 > See the attached.

All looks good to me.  

--
Matt Trzcinski
Emacs Org contributor (ob-shell)
Learn more about Org mode at https://orgmode.org
Support Org development at https://liberapay.com/org-mode

Re: [PATCH v4] org-manual: Describe export process flow

[Ihor Radchenko]

Ihor Radchenko <yantar92@posteo.net> writes:

> ... See the attached.
> I added reference to the manual in `org-export-as' docstring and made it
> explicit that the manual section is related to `org-export-as' by adding
> #+findex entry.

Applied, onto main.
https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=1ff72e091
https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=46cf76259
https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=1087a7b4a
https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=f9fc9d95c
https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=a70f216e3

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>