[PATCH v4] org-manual: Describe export process flow

[Ihor Radchenko <yantar92@posteo.net>]

[-- Attachment #1: Type: text/plain, Size: 434 bytes --]

Matt <matt@excalamus.com> writes:

> I was making a different point.  I was trying to say that it may be a
> good idea for the ox.el commentary section to specifically reference
> the new manual section (next to where it references Worg).

Makes sense. See the attached.
I added reference to the manual in `org-export-as' docstring and made it
explicit that the manual section is related to `org-export-as' by adding
#+findex entry.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: v4-0001-doc-org-manual.org-Fix-some-obsolete-variable-nam.patch --]
[-- Type: text/x-patch, Size: 1919 bytes --]

From b17d57c1afeb160783da36647f56462ee67c3558 Mon Sep 17 00:00:00 2001
Message-ID: <b17d57c1afeb160783da36647f56462ee67c3558.1703765064.git.yantar92@posteo.net>
From: Ihor Radchenko <yantar92@posteo.net>
Date: Wed, 27 Dec 2023 14:23:29 +0100
Subject: [PATCH v4 1/5] doc/org-manual.org: Fix some obsolete variable names

* doc/org-manual.org (Export hooks): Use the new
`org-export-before-processing-functions' and
`org-export-before-parsing-functions' instead of their obsolete
aliases.
---
 doc/org-manual.org | 12 +++++++-----
 1 file changed, 7 insertions(+), 5 deletions(-)

diff --git a/doc/org-manual.org b/doc/org-manual.org
index ff1b9cffb..377706ee7 100644
--- a/doc/org-manual.org
+++ b/doc/org-manual.org
@@ -16397,12 +16397,14 @@ *** Export hooks
 :END:
 
 #+vindex: org-export-before-processing-hook
+#+vindex: org-export-before-processing-functions
 #+vindex: org-export-before-parsing-hook
 The export process executes two hooks before the actual exporting
-begins.  The first hook, ~org-export-before-processing-hook~, runs
-before any expansions of macros, Babel code, and include keywords in
-the buffer.  The second hook, ~org-export-before-parsing-hook~, runs
-before the buffer is parsed.
+begins.  The first hook, ~org-export-before-processing-functions~,
+runs before any expansions of macros, Babel code, and include keywords
+in the buffer.  The second hook,
+~org-export-before-parsing-functions~, runs before the buffer is
+parsed.
 
 Functions added to these hooks are called with a single argument: the
 export backend actually used, as a symbol.  You may use them for
@@ -16421,7 +16423,7 @@ *** Export hooks
      ;; the docstring of `org-map-entries' for details.
      (setq org-map-continue-from (point)))))
 
-(add-hook 'org-export-before-parsing-hook #'my-headline-removal)
+(add-hook 'org-export-before-parsing-functions #'my-headline-removal)
 #+end_src
 
 *** Filters
-- 
2.42.0


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #3: v4-0002-doc-org-manual.org-Describe-export-flow.patch --]
[-- Type: text/x-patch, Size: 6479 bytes --]

From b195caf687793af228a2ea9a484f2fb8cb87ddab Mon Sep 17 00:00:00 2001
Message-ID: <b195caf687793af228a2ea9a484f2fb8cb87ddab.1703765065.git.yantar92@posteo.net>
In-Reply-To: <b17d57c1afeb160783da36647f56462ee67c3558.1703765064.git.yantar92@posteo.net>
References: <b17d57c1afeb160783da36647f56462ee67c3558.1703765064.git.yantar92@posteo.net>
From: Ihor Radchenko <yantar92@posteo.net>
Date: Tue, 26 Dec 2023 15:15:23 +0100
Subject: [PATCH v4 2/5] doc/org-manual.org: Describe export flow

* doc/org-manual.org (Summary of the export process): Explain how the
export process is handled in Org mode.
---
 doc/org-manual.org | 148 +++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 148 insertions(+)

diff --git a/doc/org-manual.org b/doc/org-manual.org
index 377706ee7..66a582eae 100644
--- a/doc/org-manual.org
+++ b/doc/org-manual.org
@@ -16505,6 +16505,154 @@ *** Defining filters for individual files
 ,#+END_SRC
 #+end_example
 
+*** Summary of the export process
+:PROPERTIES:
+:UNNUMBERED: notoc
+:END:
+
+Org mode export is a multi-step process that works on a temporary copy
+of the buffer.  The export process consists of 4 major steps:
+
+1. Process the temporary copy, making necessary changes to the buffer
+   text;
+
+2. Parse the buffer, converting plain Org markup into an abstract
+   syntax tree (AST);
+
+3. Convert the AST to text, as prescribed by the selected export
+   backend;
+
+4. Post-process the resulting exported text.
+
+
+#+texinfo: @noindent
+Process temporary copy of the source Org buffer [fn::Unless
+otherwise specified, each step of the export process only operates on
+the accessible portion of the buffer.  When subtree export is selected
+(see [[*The Export Dispatcher]]), the buffer is narrowed to the body of
+the selected subtree, so that the rest of the buffer text, except
+export keywords, does not contribute to the export output.]:
+
+1. Execute ~org-export-before-processing-functions~ (see [[*Export hooks]]);
+
+2. Expand =#+include= keywords in the whole buffer (see
+   [[*Include Files]]);
+
+3. Remove commented subtrees in the whole buffer (see [[*Comment
+   Lines]]);
+
+4. Replace macros in the whole buffer (see [[*Macro Replacement]]);
+
+5. When ~org-export-use-babel~ is non-nil (default), process code
+   blocks:
+
+   - Leave code blocks inside archived subtrees (see [[*Internal
+     archiving]]) as is;
+
+   - Evaluate all the other code blocks according to code block
+     headers (see [[*Limit code block evaluation]]);
+
+   - Remove code, results of evaluation, both, or neither according
+     to =:exports= header argument (see [[*Exporting Code Blocks]]).
+
+
+#+texinfo: @noindent
+Parse the temporary buffer, creating AST:
+
+1. Execute ~org-export-before-parsing-functions~ (see [[*Export hooks]]).
+   The hook functions may still modify the buffer;
+
+2. Calculate export option values according to subtree-specific export
+   settings, in-buffer keywords, =#+BIND= keywords, and buffer-local
+   and global customization.  The whole buffer is considered;
+
+3. Determine contributing bibliographies and record them into export
+   options (see [[*Citations]]).  The whole buffer is considered;
+
+4. Execute ~org-export-filter-options-functions~;
+
+5. Parse the accessible portion of the temporary buffer to generate an
+   AST.  The AST is a nested list of lists representing Org syntax
+   elements (see [[https://orgmode.org/worg/dev/org-element-api.html][Org Element API]] for more details):
+
+   : (org-data ...
+   :  (heading
+   :   (section
+   :    (paragraph (plain-text) (bold (plain-text))))
+   :   (heading)
+   :   (heading (section ...))))
+
+   Past this point, modifications to the temporary buffer no longer
+   affect the export; Org export works only with the AST;
+
+6. Remove elements that are not exported from the AST:
+
+   - Headings according to =SELECT_TAGS= and =EXCLUDE_TAGS= export
+     keywords; =task=, =inline=, =arch= export options (see
+     [[*Export Settings]]);
+
+   - Comments;
+
+   - Clocks, drawers, fixed-width environments, footnotes, LaTeX
+     environments and fragments, node properties, planning lines,
+     property drawers, statistics cookies, timestamps, etc according
+     to =#+OPTIONS= keyword (see [[*Export Settings]]);
+
+   - Table rows containing width and alignment markers (see [[*Column
+     Width and Alignment]]);
+
+   - Table columns containing recalc marks (see [[*Advanced features]]).
+
+7. Expand environment variables in file link AST nodes according to
+   the =expand-links= export option (see [[*Export Settings]]);
+
+8. Execute ~org-export-filter-parse-tree-functions~.  These
+   functions can modify the AST by side effects;
+
+9. Replace citation AST nodes and =#+print_bibliography= keyword AST
+   nodes as prescribed by the selected citation export processor
+   (see [[*Citation export processors]]).
+
+
+#+texinfo: @noindent
+Convert the AST to text by traversing the AST nodes, depth-first:
+
+1. Convert the leaf nodes (without children) to text as prescribed
+   by "transcoders" in the selected export backend
+   [fn:: See transcoders and ~:translate-alist~ in the docstrings
+   of ~org-export-define-backend~ and ~org-export-define-derived-backend~.];
+
+2. Pass the converted nodes through the corresponding export
+   filters (see [[*Filters]]);
+
+3. Concatenate all the converted child nodes to produce parent
+   node contents;
+
+4. Convert the nodes with children to text, passing the nodes
+   themselves and their contents to the corresponding transcoders
+   and then to the export filters (see [[*Filters]]).
+
+
+#+texinfo: @noindent
+Post-process the exported text:
+
+  1. Post-process the converted AST, as prescribed by the export
+     backend. [fn:: See ~inner-template~ in the docstring of ~org-export-define-backend~.]
+     This step usually adds generated content (like Table of Contents)
+     to the exported text;
+
+  2. Execute ~org-export-filter-body-functions~;
+
+  3. Unless body-only export is selected (see [[*The Export Dispatcher]]),
+     add the necessary metadata to the final document, as prescribed
+     by the export backend.  Examples: Document author/title; HTML
+     headers/footers; LaTeX preamble;
+
+  4. Add bibliography metadata, as prescribed by the citation export
+     processor;
+
+  5. Execute ~org-export-filter-final-output-functions~.
+
 *** Extending an existing backend
 :PROPERTIES:
 :UNNUMBERED: notoc
-- 
2.42.0


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #4: v4-0003-doc-org-manual.org-Add-link-to-WORG-export-refere.patch --]
[-- Type: text/x-patch, Size: 1227 bytes --]

From bbd6f3613d3d8ca1196c258ee1fcc9ec4df1d9a1 Mon Sep 17 00:00:00 2001
Message-ID: <bbd6f3613d3d8ca1196c258ee1fcc9ec4df1d9a1.1703765065.git.yantar92@posteo.net>
In-Reply-To: <b17d57c1afeb160783da36647f56462ee67c3558.1703765064.git.yantar92@posteo.net>
References: <b17d57c1afeb160783da36647f56462ee67c3558.1703765064.git.yantar92@posteo.net>
From: Ihor Radchenko <yantar92@posteo.net>
Date: Wed, 27 Dec 2023 17:58:48 +0100
Subject: [PATCH v4 3/5] doc/org-manual.org: Add link to WORG export reference
 page

* doc/org-manual.org (Extending an existing backend): Link to WORG
page describing more details about writing export backends.
---
 doc/org-manual.org | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/doc/org-manual.org b/doc/org-manual.org
index 66a582eae..bb1bad673 100644
--- a/doc/org-manual.org
+++ b/doc/org-manual.org
@@ -16709,7 +16709,9 @@ *** Extending an existing backend
 
 Further steps to consider would be an interactive function,
 self-installing an item in the export dispatcher menu, and other
-user-friendly improvements.
+user-friendly improvements.  See
+<https://orgmode.org/worg/dev/org-export-reference.html> for more
+details.
 
 ** Export Region
 :PROPERTIES:
-- 
2.42.0


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #5: v4-0004-org-export-as-Add-reference-to-export-flow-in-the.patch --]
[-- Type: text/x-patch, Size: 1131 bytes --]

From 5ad0e8d6e05a312960ea006958f986097441cd5a Mon Sep 17 00:00:00 2001
Message-ID: <5ad0e8d6e05a312960ea006958f986097441cd5a.1703765065.git.yantar92@posteo.net>
In-Reply-To: <b17d57c1afeb160783da36647f56462ee67c3558.1703765064.git.yantar92@posteo.net>
References: <b17d57c1afeb160783da36647f56462ee67c3558.1703765064.git.yantar92@posteo.net>
From: Ihor Radchenko <yantar92@posteo.net>
Date: Thu, 28 Dec 2023 12:59:22 +0100
Subject: [PATCH v4 4/5] org-export-as: Add reference to export flow in the
 manual

* lisp/ox.el (org-export-as): Update docstring.
---
 lisp/ox.el | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/lisp/ox.el b/lisp/ox.el
index 46994f0e2..2cf364227 100644
--- a/lisp/ox.el
+++ b/lisp/ox.el
@@ -2923,6 +2923,9 @@ (defun org-export-as
     (backend &optional subtreep visible-only body-only ext-plist)
   "Transcode current Org buffer into BACKEND code.
 
+See info node `(org)Advanced Export Configuration' for the details of
+the transcoding process.
+
 BACKEND is either an export backend, as returned by, e.g.,
 `org-export-create-backend', or a symbol referring to
 a registered backend.
-- 
2.42.0


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #6: v4-0005-doc-org-manual.org-Summary-of-the-export-process-.patch --]
[-- Type: text/x-patch, Size: 988 bytes --]

From 1c89b2e32816c0ad95d344aa6b05a2ad65b2d172 Mon Sep 17 00:00:00 2001
Message-ID: <1c89b2e32816c0ad95d344aa6b05a2ad65b2d172.1703765065.git.yantar92@posteo.net>
In-Reply-To: <b17d57c1afeb160783da36647f56462ee67c3558.1703765064.git.yantar92@posteo.net>
References: <b17d57c1afeb160783da36647f56462ee67c3558.1703765064.git.yantar92@posteo.net>
From: Ihor Radchenko <yantar92@posteo.net>
Date: Thu, 28 Dec 2023 13:04:06 +0100
Subject: [PATCH v4 5/5] doc/org-manual.org (Summary of the export process):
 Reference `org-export-as'

---
 doc/org-manual.org | 1 +
 1 file changed, 1 insertion(+)

diff --git a/doc/org-manual.org b/doc/org-manual.org
index bb1bad673..de2194868 100644
--- a/doc/org-manual.org
+++ b/doc/org-manual.org
@@ -16510,6 +16510,7 @@ *** Summary of the export process
 :UNNUMBERED: notoc
 :END:
 
+#+findex: org-export-as
 Org mode export is a multi-step process that works on a temporary copy
 of the buffer.  The export process consists of 4 major steps:
 
-- 
2.42.0


[-- Attachment #7: Type: text/plain, Size: 224 bytes --]


-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>