Re: multipage html output

[Orm Finnendahl <orm.finnendahl@selma.hfmdk-frankfurt.de>]

Hi Ihor,

 thanks a lot for the patches and explanations. Your assessment makes
sense and I agree that we should stick to the current design as much
as possible. I will go ahead and adapt my ox-html.el code the way you
propose and use your patched ox.el for this. It shouldn't take too
much effort and I will get back as soon as I have results (or
questions ;-).

--
Orm

Am Mittwoch, den 24. Juli 2024 um 10:20:16 Uhr (+0000) schrieb Ihor Radchenko:
> Orm Finnendahl <orm.finnendahl@selma.hfmdk-frankfurt.de> writes:
> 
> > To recapitulate: In my code, org-export-as calls process-multipage in
> > the backend. This function:
> >
> > - collects and adds information necessary for org-multipage to do its
> >   job, splitting the document into different parts, etc. and
> >
> > - then calls org-export-data on the subtrees and exports each returned
> >   string to an individual file.
> >
> > - It finally issues a done string and executes a browser open/visit
> >   file or simply exits nil.
> 
> Currently, org-export-as does the following:
> 
> 1. Compute global export attributes, according to the selected export backend
> 2. Copy original buffer into working copy
> 3. Process and parse the copy, generating AST
> 4. Do the actual export
> 
> You plugged your multipage processing into (4), but what it actually
> does involves (3), (4), and also a new kind of post-processing.
> I do not think that it is a good design from the point of view of ox.el.
> I prefer to reuse or extend the existing mechanisms if at all possible -
> this makes new features less confusing for users and backend developers.
> 
> > - collects and adds information necessary for org-multipage to do its
> >   job, splitting the document into different parts, etc. and
> 
> What you describe here is more or less what :filter-parse-tree filters
> do - they can rearrange the parse tree before passing it to the
> transcoders. Why not reusing it for multipage export?
> 
> > - then calls org-export-data on the subtrees and exports each returned
> >   string to an individual file.
> 
> And you simply call `org-export-transcode-page' for this, followed by
> writing the returned string to file.
> 
> The first part can fit within `org-export-as', but writing to file is
> going a step beyond, duplicating what `org-export-to-file' does.
> 
> > - It finally issues a done string and executes a browser open/visit
> >   file or simply exits nil.
> 
> ... which again steps beyond `org-export-as' scope - post-processing is
> currently done as a part of `org-export-to-file'/`org-export-to-buffer'.
> 
> ----
> 
> Let me propose the following changes to ox.el:
> 
> 1. org-data will be transcoded using `org-export-transcode-org-data',
>    which can be overridden by setting org-data transcoders in the
>    individual backends.
> 
> 2. org-export-as will understand transcoded output to be a list of
>    strings and will transfer INFO plist as text property in the return
>    values
> 
> 3. org-export-to-file will make use of the text properties to retrieve
>    the file name to write.  This way, export backend itself can assign
>    the file names where each exporter string should go.
> 
> I believe that my changes should allow you to implement multipage export
> in the following way:
> 
> 1. You can use :filter-parse-tree in ox-html backend to replace the
>    original (org-data ...) AST with a list of
>    ((org-page ...) (org-page ...) ...) pseudo-elements and populate INFO
>    channel with auxiliary information you now compute in `org-html-process-multipage'
> 
> 2. You can define org-page transcoder to render individual pages as
> needed
> 
> 3. You can assign :output-file text property to the returned org-page
>    strings and use org-export-to-file to generate the multipage output
>    on disk
> 
> 4. You can handle opening exported files by augmenting POST-PROCESS
>    argument in `org-html-export-to-multipage-html' and calling
>    `org-export-file' instead of `org-export-as'.
> 
> The tentative patches (against Org mode main branch) implementing my
> changes are attached.
> 

> From 540c8ef21c26df79cf48f58afb4e88130985e2f7 Mon Sep 17 00:00:00 2001
> Message-ID: <540c8ef21c26df79cf48f58afb4e88130985e2f7.1721815865.git.yantar92@posteo.net>
> From: Ihor Radchenko <yantar92@posteo.net>
> Date: Wed, 24 Jul 2024 11:40:57 +0200
> Subject: [PATCH 1/3] ox: Factor out org-data transcoding into dedicated
>  overrideable transcoder
> 
> * lisp/ox.el (org-export-transcode-org-data): New function serving as
> the default transcoder for org-data export.
> (org-export-transcoder): Use `org-export-transcode-org-data' when no
> org-data transcoder is defined.
> (org-export-as): Rely upon org-data transcoder to do its job.
> ---
>  lisp/ox.el | 55 +++++++++++++++++++++++++++++-------------------------
>  1 file changed, 30 insertions(+), 25 deletions(-)
> 
> diff --git a/lisp/ox.el b/lisp/ox.el
> index fbd9bb0df..bdee71082 100644
> --- a/lisp/ox.el
> +++ b/lisp/ox.el
> @@ -1883,9 +1883,11 @@ (defun org-export-transcoder (blob info)
>  INFO is a plist containing export directives."
>    (let ((type (org-element-type blob)))
>      ;; Return contents only for complete parse trees.
> -    (if (eq type 'org-data) (lambda (_datum contents _info) contents)
> -      (let ((transcoder (cdr (assq type (plist-get info :translate-alist)))))
> -	(and (functionp transcoder) transcoder)))))
> +    (let ((transcoder (cdr (assq type (plist-get info :translate-alist)))))
> +      (cond
> +       ((functionp transcoder) transcoder)
> +       ;; Use default org-data transcoder unless specified.
> +       ((eq type 'org-data) #'org-export-transcode-org-data)))))
>  
>  (defun org-export--keep-spaces (data info)
>    "Non-nil, when post-blank spaces after removing DATA should be preserved.
> @@ -3004,31 +3006,34 @@ (defun org-export-as
>                         backend info subtreep visible-only ext-plist))
>  	   ;; Eventually transcode TREE.  Wrap the resulting string into
>  	   ;; a template.
> -	   (let* ((body (org-element-normalize-string
> -		         (or (org-export-data (plist-get info :parse-tree) info)
> -                             "")))
> -		  (inner-template (cdr (assq 'inner-template
> -					     (plist-get info :translate-alist))))
> -		  (full-body (org-export-filter-apply-functions
> -			      (plist-get info :filter-body)
> -			      (if (not (functionp inner-template)) body
> -			        (funcall inner-template body info))
> -			      info))
> -		  (template (cdr (assq 'template
> -				       (plist-get info :translate-alist))))
> -                  (output
> -                   (if (or (not (functionp template)) body-only) full-body
> -	             (funcall template full-body info))))
> +	   (let ((output
> +                  (or (org-export-data (plist-get info :parse-tree) info)
> +                      "")))
>               ;; Call citation export finalizer.
>               (when (plist-get info :with-cite-processors)
>                 (setq output (org-cite-finalize-export output info)))
> -	     ;; Remove all text properties since they cannot be
> -	     ;; retrieved from an external process.  Finally call
> -	     ;; final-output filter and return result.
> -	     (org-no-properties
> -	      (org-export-filter-apply-functions
> -	       (plist-get info :filter-final-output)
> -	       output info)))))))))
> +             (let ((filters (plist-get info :filter-final-output)))
> +               ;; Remove all text properties since they cannot be
> +	       ;; retrieved from an external process.  Finally call
> +	       ;; final-output filter and return result.
> +               (org-no-properties
> +                (org-export-filter-apply-functions filters output info))))))))))
> +
> +(defun org-export-transcode-org-data (_ body info)
> +  "Transcode `org-data' node with BODY.  Return transcoded string.
> +INFO is the communication channel plist."
> +  (let* ((inner-template (cdr (assq 'inner-template
> +				    (plist-get info :translate-alist))))
> +	 (full-body (org-export-filter-apply-functions
> +		     (plist-get info :filter-body)
> +		     (if (not (functionp inner-template)) body
> +		       (funcall inner-template body info))
> +		     info))
> +	 (template (cdr (assq 'template
> +			      (plist-get info :translate-alist))))
> +         (body-only (memq 'body-only (plist-get info :export-options))))
> +    (if (or (not (functionp template)) body-only) full-body
> +      (funcall template full-body info))))
>  
>  (defun org-export--annotate-info (backend info &optional subtreep visible-only ext-plist)
>    "Annotate the INFO plist according to the BACKEND.
> -- 
> 2.45.2
> 

> From 1b0b331f92abc1ca7e04f71fe7ff60da57c719b8 Mon Sep 17 00:00:00 2001
> Message-ID: <1b0b331f92abc1ca7e04f71fe7ff60da57c719b8.1721815865.git.yantar92@posteo.net>
> In-Reply-To: <540c8ef21c26df79cf48f58afb4e88130985e2f7.1721815865.git.yantar92@posteo.net>
> References: <540c8ef21c26df79cf48f58afb4e88130985e2f7.1721815865.git.yantar92@posteo.net>
> From: Ihor Radchenko <yantar92@posteo.net>
> Date: Wed, 24 Jul 2024 11:51:21 +0200
> Subject: [PATCH 2/3] org-export-as: Allow the return value to be a list of
>  strings; add INFO
> 
> * lisp/ox.el (org-export-as): Allow the transcoders to return list of
> strings and return it.  When returning a string, put INFO plist as
> text property.  Do not remove text properties assigned by the
> transcoders.
> (org-export-data): Document that list of strings may be returned.
> ---
>  lisp/ox.el | 28 ++++++++++++++++++++--------
>  1 file changed, 20 insertions(+), 8 deletions(-)
> 
> diff --git a/lisp/ox.el b/lisp/ox.el
> index bdee71082..a76b3b353 100644
> --- a/lisp/ox.el
> +++ b/lisp/ox.el
> @@ -1930,7 +1930,7 @@ (defun org-export-data (data info)
>  
>  The `:filter-parse-tree' filters are not applied.
>  
> -Return a string."
> +Return a string or a list of strings."
>    (or (gethash data (plist-get info :exported-data))
>        ;; Handle broken links according to
>        ;; `org-export-with-broken-links'.
> @@ -2969,7 +2969,9 @@ (defun org-export-as
>  with external parameters overriding Org default settings, but
>  still inferior to file-local settings.
>  
> -Return code as a string."
> +Return code as a string or a list of strings.
> +The returned strings will have their `org-export-info' property set to
> +export information channel."
>    (when (symbolp backend) (setq backend (org-export-get-backend backend)))
>    (org-export-barf-if-invalid-backend backend)
>    (org-fold-core-ignore-modifications
> @@ -3009,15 +3011,25 @@ (defun org-export-as
>  	   (let ((output
>                    (or (org-export-data (plist-get info :parse-tree) info)
>                        "")))
> +             (setq output (ensure-list output))
>               ;; Call citation export finalizer.
>               (when (plist-get info :with-cite-processors)
> -               (setq output (org-cite-finalize-export output info)))
> +               (setq output
> +                     (mapcar
> +                      (lambda (o) (org-cite-finalize-export o info))
> +                      output)))
>               (let ((filters (plist-get info :filter-final-output)))
> -               ;; Remove all text properties since they cannot be
> -	       ;; retrieved from an external process.  Finally call
> -	       ;; final-output filter and return result.
> -               (org-no-properties
> -                (org-export-filter-apply-functions filters output info))))))))))
> +               ;; Call final-output filter and return result.
> +               (setq output
> +                     (mapcar
> +                      (lambda (o) (org-export-filter-apply-functions filters o info))
> +                      output)))
> +             ;; Apply org-export-info property.
> +             (setq output
> +                   (mapcar
> +                    (lambda (o) (org-add-props o nil 'org-export-info info))
> +                    output))
> +             (if (length= output 1) (car output) output))))))))
>  
>  (defun org-export-transcode-org-data (_ body info)
>    "Transcode `org-data' node with BODY.  Return transcoded string.
> -- 
> 2.45.2
> 

> From 6fa2efadd229a667fba1b18aecc9d1ead5f284ac Mon Sep 17 00:00:00 2001
> Message-ID: <6fa2efadd229a667fba1b18aecc9d1ead5f284ac.1721815865.git.yantar92@posteo.net>
> In-Reply-To: <540c8ef21c26df79cf48f58afb4e88130985e2f7.1721815865.git.yantar92@posteo.net>
> References: <540c8ef21c26df79cf48f58afb4e88130985e2f7.1721815865.git.yantar92@posteo.net>
> From: Ihor Radchenko <yantar92@posteo.net>
> Date: Wed, 24 Jul 2024 12:09:36 +0200
> Subject: [PATCH 3/3] org-export-to-file: Derive file name to write from export
>  output
> 
> * lisp/ox.el (org-export--write-output): New helper function
> performing writing an export output or a list of outputs to file.  It
> derives the file name from :output-file property in the output string
> or INFO plist stored in the output string.
> (org-export-to-file): Handle export output being a list of strings.
> Use `org-export--write-output'.
> ---
>  lisp/ox.el | 61 ++++++++++++++++++++++++++++++++++--------------------
>  1 file changed, 38 insertions(+), 23 deletions(-)
> 
> diff --git a/lisp/ox.el b/lisp/ox.el
> index a76b3b353..d78c04998 100644
> --- a/lisp/ox.el
> +++ b/lisp/ox.el
> @@ -6830,6 +6830,31 @@   (defun org-latex-export-as-latex
>  	(switch-to-buffer-other-window buffer))
>        buffer)))
>  
> +(defun org-export--write-output (output encoding)
> +  "Write OUTPUT to file with ENCODING.
> +OUTPUT may be a string or a list of strings.
> +The target file is retrieved from :output-file OUTPUT property or
> +:output-file property in plist stored in `org-export-info' property of
> +each string.
> +
> +Return the file name or a list of file names."
> +  (if (listp output) (mapcar #'org-export--write-output output)
> +    (let ((file (or
> +                 (get-text-property 0 :output-file output)
> +                 (plist-get
> +                  (get-text-property 0 'org-export-info output)
> +                  :output-file))))
> +      (with-temp-buffer
> +        (insert output)
> +        ;; Ensure final newline.  This is what was done
> +        ;; historically, when we used `write-file'.
> +        ;; Note that adding a newline is only safe for
> +        ;; non-binary data.
> +        (unless (bolp) (insert "\n"))
> +        (let ((coding-system-for-write encoding))
> +	  (write-region nil nil file))
> +        file))))
> +
>  ;;;###autoload
>  (defun org-export-to-file
>      (backend file &optional async subtreep visible-only body-only ext-plist
> @@ -6878,33 +6903,23 @@   (defun org-latex-export-to-latex
>  	    `(let ((output
>  		    (org-export-as
>  		     ',backend ,subtreep ,visible-only ,body-only
> -		     ',ext-plist)))
> -	       (with-temp-buffer
> -		 (insert output)
> -                 ;; Ensure final newline.  This is what was done
> -                 ;; historically, when we used `write-file'.
> -                 ;; Note that adding a newline is only safe for
> -                 ;; non-binary data.
> -                 (unless (bolp) (insert "\n"))
> -		 (let ((coding-system-for-write ',encoding))
> -		   (write-region nil nil ,file)))
> -	       (or (ignore-errors (funcall ',post-process ,file)) ,file)))
> +		     ',ext-plist))
> +                   file)
> +               (setq file (org-export--write-output output ',encoding))
> +               (let ((post (lambda (f) (or (ignore-errors (funcall ',post-process f)) f))))
> +                 (if (listp file) (mapcar post file) (funcall post file)))))
>          (let ((output (org-export-as
> -                       backend subtreep visible-only body-only ext-plist)))
> -          (with-temp-buffer
> -            (insert output)
> -            ;; Ensure final newline.  This is what was done
> -            ;; historically, when we used `write-file'.
> -            ;; Note that adding a newline is only safe for
> -            ;; non-binary data.
> -            (unless (bolp) (insert "\n"))
> -            (let ((coding-system-for-write encoding))
> -	      (write-region nil nil file)))
> +                       backend subtreep visible-only body-only ext-plist))
> +              file)
> +          (setq file (org-export--write-output output encoding))
>            (when (and (org-export--copy-to-kill-ring-p) (org-string-nw-p output))
>              (org-kill-new output))
>            ;; Get proper return value.
> -          (or (and (functionp post-process) (funcall post-process file))
> -	      file))))))
> +          (let ((post (lambda (f)
> +                        (or (and (functionp post-process)
> +                                 (funcall post-process f))
> +	                    f))))
> +            (if (listp file) (mapcar post file) (funcall post file))))))))
>  
>  (defun org-export-output-file-name (extension &optional subtreep pub-dir)
>    "Return output file's name according to buffer specifications.
> -- 
> 2.45.2
> 

> 
> -- 
> Ihor Radchenko // yantar92,
> Org mode contributor,
> Learn more about Org mode at <https://orgmode.org/>.
> Support Org development at <https://liberapay.com/org-mode>,
> or support my work at <https://liberapay.com/yantar92>