[PATCH v3] org-manual: Describe export process flow

[Ihor Radchenko <yantar92@posteo.net>]

[-- Attachment #1: Type: text/plain, Size: 1920 bytes --]

Matt <matt@excalamus.com> writes:

>  > Thank you all for the feedback!
>
> Thanks for taking the time to write and revise the patches.
>
> You didn't explicitly ask for more feedback, so I hope the following feedback on the new patches is in order.

Of course. Any patch posted on the mailing list is open to feedback.
Especially mine, since I can push it directly if I think that the patch
is in a good shape.

> For v2-0002-doc-org-manual.org-Describe-export-flow.patch here are
> some quibbles:

Thanks for proofreading!
I accepted all but one suggestion on grammar.
The new iteration of the patch is attached.

>  >    - Headings according to =SELECT_TAGS= and =EXCLUDE_TAGS= export
>  >      keywords, and =task=, =inline=, =arch= export options (see
>  >      [[*Export Settings]]);
>
> Remove comma after "keywords":
>
> "Headings according to =SELECT_TAGS= and =EXCLUDE_TAGS= export
> keywords and =task=, =inline=, =arch= export options (see [[*Export
> Settings]]);"

Here, "export keywords" and "export options" are two different groups. I
wanted to indicate this and went with ";":

   - Headings according to =SELECT_TAGS= and =EXCLUDE_TAGS= export
     keywords; =task=, =inline=, =arch= export options (see
     [[*Export Settings]]);

> Checking ox.el, I see that the commentary says,
>
> ";; See <https://orgmode.org/worg/dev/org-export-reference.html> for
> ;; more information."
>
> Maybe also let maintainers know about this manual section:
>
> "See:
> <https://orgmode.org/worg/dev/org-export-reference.html>
> <link to Summary of the export process>"

AFAIU, you are asking to add it to WORG page. However, it is too early.
The online Org manual that we can link to is only for stable Org
version. So, only after the next release.

For now, I added a link to the WORG page to "Extending an existing
backend" section. It should be a good reference for people writing
non-trivial derived backends.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: v3-0001-doc-org-manual.org-Fix-some-obsolete-variable-nam.patch --]
[-- Type: text/x-patch, Size: 1919 bytes --]

From 6f6a1a190724176da7373e3d25f9df0bf60dbac9 Mon Sep 17 00:00:00 2001
Message-ID: <6f6a1a190724176da7373e3d25f9df0bf60dbac9.1703696392.git.yantar92@posteo.net>
From: Ihor Radchenko <yantar92@posteo.net>
Date: Wed, 27 Dec 2023 14:23:29 +0100
Subject: [PATCH v3 1/3] doc/org-manual.org: Fix some obsolete variable names

* doc/org-manual.org (Export hooks): Use the new
`org-export-before-processing-functions' and
`org-export-before-parsing-functions' instead of their obsolete
aliases.
---
 doc/org-manual.org | 12 +++++++-----
 1 file changed, 7 insertions(+), 5 deletions(-)

diff --git a/doc/org-manual.org b/doc/org-manual.org
index ff1b9cffb..377706ee7 100644
--- a/doc/org-manual.org
+++ b/doc/org-manual.org
@@ -16397,12 +16397,14 @@ *** Export hooks
 :END:
 
 #+vindex: org-export-before-processing-hook
+#+vindex: org-export-before-processing-functions
 #+vindex: org-export-before-parsing-hook
 The export process executes two hooks before the actual exporting
-begins.  The first hook, ~org-export-before-processing-hook~, runs
-before any expansions of macros, Babel code, and include keywords in
-the buffer.  The second hook, ~org-export-before-parsing-hook~, runs
-before the buffer is parsed.
+begins.  The first hook, ~org-export-before-processing-functions~,
+runs before any expansions of macros, Babel code, and include keywords
+in the buffer.  The second hook,
+~org-export-before-parsing-functions~, runs before the buffer is
+parsed.
 
 Functions added to these hooks are called with a single argument: the
 export backend actually used, as a symbol.  You may use them for
@@ -16421,7 +16423,7 @@ *** Export hooks
      ;; the docstring of `org-map-entries' for details.
      (setq org-map-continue-from (point)))))
 
-(add-hook 'org-export-before-parsing-hook #'my-headline-removal)
+(add-hook 'org-export-before-parsing-functions #'my-headline-removal)
 #+end_src
 
 *** Filters
-- 
2.42.0


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #3: v3-0002-doc-org-manual.org-Describe-export-flow.patch --]
[-- Type: text/x-patch, Size: 6479 bytes --]

From 786e3e78d4c6846188477452a529132aba76149f Mon Sep 17 00:00:00 2001
Message-ID: <786e3e78d4c6846188477452a529132aba76149f.1703696392.git.yantar92@posteo.net>
In-Reply-To: <6f6a1a190724176da7373e3d25f9df0bf60dbac9.1703696392.git.yantar92@posteo.net>
References: <6f6a1a190724176da7373e3d25f9df0bf60dbac9.1703696392.git.yantar92@posteo.net>
From: Ihor Radchenko <yantar92@posteo.net>
Date: Tue, 26 Dec 2023 15:15:23 +0100
Subject: [PATCH v3 2/3] doc/org-manual.org: Describe export flow

* doc/org-manual.org (Summary of the export process): Explain how the
export process is handled in Org mode.
---
 doc/org-manual.org | 148 +++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 148 insertions(+)

diff --git a/doc/org-manual.org b/doc/org-manual.org
index 377706ee7..66a582eae 100644
--- a/doc/org-manual.org
+++ b/doc/org-manual.org
@@ -16505,6 +16505,154 @@ *** Defining filters for individual files
 ,#+END_SRC
 #+end_example
 
+*** Summary of the export process
+:PROPERTIES:
+:UNNUMBERED: notoc
+:END:
+
+Org mode export is a multi-step process that works on a temporary copy
+of the buffer.  The export process consists of 4 major steps:
+
+1. Process the temporary copy, making necessary changes to the buffer
+   text;
+
+2. Parse the buffer, converting plain Org markup into an abstract
+   syntax tree (AST);
+
+3. Convert the AST to text, as prescribed by the selected export
+   backend;
+
+4. Post-process the resulting exported text.
+
+
+#+texinfo: @noindent
+Process temporary copy of the source Org buffer [fn::Unless
+otherwise specified, each step of the export process only operates on
+the accessible portion of the buffer.  When subtree export is selected
+(see [[*The Export Dispatcher]]), the buffer is narrowed to the body of
+the selected subtree, so that the rest of the buffer text, except
+export keywords, does not contribute to the export output.]:
+
+1. Execute ~org-export-before-processing-functions~ (see [[*Export hooks]]);
+
+2. Expand =#+include= keywords in the whole buffer (see
+   [[*Include Files]]);
+
+3. Remove commented subtrees in the whole buffer (see [[*Comment
+   Lines]]);
+
+4. Replace macros in the whole buffer (see [[*Macro Replacement]]);
+
+5. When ~org-export-use-babel~ is non-nil (default), process code
+   blocks:
+
+   - Leave code blocks inside archived subtrees (see [[*Internal
+     archiving]]) as is;
+
+   - Evaluate all the other code blocks according to code block
+     headers (see [[*Limit code block evaluation]]);
+
+   - Remove code, results of evaluation, both, or neither according
+     to =:exports= header argument (see [[*Exporting Code Blocks]]).
+
+
+#+texinfo: @noindent
+Parse the temporary buffer, creating AST:
+
+1. Execute ~org-export-before-parsing-functions~ (see [[*Export hooks]]).
+   The hook functions may still modify the buffer;
+
+2. Calculate export option values according to subtree-specific export
+   settings, in-buffer keywords, =#+BIND= keywords, and buffer-local
+   and global customization.  The whole buffer is considered;
+
+3. Determine contributing bibliographies and record them into export
+   options (see [[*Citations]]).  The whole buffer is considered;
+
+4. Execute ~org-export-filter-options-functions~;
+
+5. Parse the accessible portion of the temporary buffer to generate an
+   AST.  The AST is a nested list of lists representing Org syntax
+   elements (see [[https://orgmode.org/worg/dev/org-element-api.html][Org Element API]] for more details):
+
+   : (org-data ...
+   :  (heading
+   :   (section
+   :    (paragraph (plain-text) (bold (plain-text))))
+   :   (heading)
+   :   (heading (section ...))))
+
+   Past this point, modifications to the temporary buffer no longer
+   affect the export; Org export works only with the AST;
+
+6. Remove elements that are not exported from the AST:
+
+   - Headings according to =SELECT_TAGS= and =EXCLUDE_TAGS= export
+     keywords; =task=, =inline=, =arch= export options (see
+     [[*Export Settings]]);
+
+   - Comments;
+
+   - Clocks, drawers, fixed-width environments, footnotes, LaTeX
+     environments and fragments, node properties, planning lines,
+     property drawers, statistics cookies, timestamps, etc according
+     to =#+OPTIONS= keyword (see [[*Export Settings]]);
+
+   - Table rows containing width and alignment markers (see [[*Column
+     Width and Alignment]]);
+
+   - Table columns containing recalc marks (see [[*Advanced features]]).
+
+7. Expand environment variables in file link AST nodes according to
+   the =expand-links= export option (see [[*Export Settings]]);
+
+8. Execute ~org-export-filter-parse-tree-functions~.  These
+   functions can modify the AST by side effects;
+
+9. Replace citation AST nodes and =#+print_bibliography= keyword AST
+   nodes as prescribed by the selected citation export processor
+   (see [[*Citation export processors]]).
+
+
+#+texinfo: @noindent
+Convert the AST to text by traversing the AST nodes, depth-first:
+
+1. Convert the leaf nodes (without children) to text as prescribed
+   by "transcoders" in the selected export backend
+   [fn:: See transcoders and ~:translate-alist~ in the docstrings
+   of ~org-export-define-backend~ and ~org-export-define-derived-backend~.];
+
+2. Pass the converted nodes through the corresponding export
+   filters (see [[*Filters]]);
+
+3. Concatenate all the converted child nodes to produce parent
+   node contents;
+
+4. Convert the nodes with children to text, passing the nodes
+   themselves and their contents to the corresponding transcoders
+   and then to the export filters (see [[*Filters]]).
+
+
+#+texinfo: @noindent
+Post-process the exported text:
+
+  1. Post-process the converted AST, as prescribed by the export
+     backend. [fn:: See ~inner-template~ in the docstring of ~org-export-define-backend~.]
+     This step usually adds generated content (like Table of Contents)
+     to the exported text;
+
+  2. Execute ~org-export-filter-body-functions~;
+
+  3. Unless body-only export is selected (see [[*The Export Dispatcher]]),
+     add the necessary metadata to the final document, as prescribed
+     by the export backend.  Examples: Document author/title; HTML
+     headers/footers; LaTeX preamble;
+
+  4. Add bibliography metadata, as prescribed by the citation export
+     processor;
+
+  5. Execute ~org-export-filter-final-output-functions~.
+
 *** Extending an existing backend
 :PROPERTIES:
 :UNNUMBERED: notoc
-- 
2.42.0


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #4: v3-0003-doc-org-manual.org-Add-link-to-WORG-export-refere.patch --]
[-- Type: text/x-patch, Size: 1227 bytes --]

From 66d2b5f68e0a4c62be3ed65a8d06300c0c2c5083 Mon Sep 17 00:00:00 2001
Message-ID: <66d2b5f68e0a4c62be3ed65a8d06300c0c2c5083.1703696392.git.yantar92@posteo.net>
In-Reply-To: <6f6a1a190724176da7373e3d25f9df0bf60dbac9.1703696392.git.yantar92@posteo.net>
References: <6f6a1a190724176da7373e3d25f9df0bf60dbac9.1703696392.git.yantar92@posteo.net>
From: Ihor Radchenko <yantar92@posteo.net>
Date: Wed, 27 Dec 2023 17:58:48 +0100
Subject: [PATCH v3 3/3] doc/org-manual.org: Add link to WORG export reference
 page

* doc/org-manual.org (Extending an existing backend): Link to WORG
page describing more details about writing export backends.
---
 doc/org-manual.org | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/doc/org-manual.org b/doc/org-manual.org
index 66a582eae..bb1bad673 100644
--- a/doc/org-manual.org
+++ b/doc/org-manual.org
@@ -16709,7 +16709,9 @@ *** Extending an existing backend
 
 Further steps to consider would be an interactive function,
 self-installing an item in the export dispatcher menu, and other
-user-friendly improvements.
+user-friendly improvements.  See
+<https://orgmode.org/worg/dev/org-export-reference.html> for more
+details.
 
 ** Export Region
 :PROPERTIES:
-- 
2.42.0


[-- Attachment #5: Type: text/plain, Size: 224 bytes --]


-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>