[tip] org-publish to work with (very) large books

[tip] org-publish to work with (very) large books

[Juan Manuel Macías]

Hi all,

- tl; dr: I describe here my workflow with org-publish to work with long
books.

—

I discovered a long time ago that `org-publish' not only works very well
for managing websites but also for working with long and complex books
with many parts, with output to LaTeX/PDF. I developed a workflow around
it that has been quite productive for me, and that I briefly describe
here in case someone finds it useful and wants to try it or modify/adapt
it to their needs. I usually use it for my typesetting work, but I think
it can also be useful for personal projects, such as doctoral theses.

First of all, each folder of my project-books has the same structure:
two subdirectories named `/org' and `/tex', for the source `org' files
and for the output `.tex' documents, respectively. And, inside the `org'
directory I include a `setup' file, an `elisp' file (for export
filters), and another `/img' directory for image files. Each `org' file
is a part of the book, and usually begins simply with the directives:

┌────
│ #+SETUPFILE: xxx.setup
│ #+INCLUDE: "elisp"
└────

`Org-publish' exports the subdocuments (body only!) as `.tex' documents
in the `/tex' folder, but they are not compiled.

What gets compiled is a master `.org' file, which is also inside the
`org' folder. I compile this master file using an asynchronous function
that calls `latexmk'. I put all the LaTeX configuration, the packages to
load, the (re)defined commands and macros, the necessary Lua code, etc.
in a `.sty' file that I load at the very beginning of the master
document. Subdocuments are loaded into this file by the LaTeX command
(\input), not by the org #+INCLUDE directive. So the master file usually
looks like this:

┌────
│ #+LaTeX_CLASS: my-custom-latex-class
│ #+LaTeX_Header: \input{my-custom-conf.sty}
│ #+SETUPFILE: xxxx.setup
│ #+INCLUDE: "elisp"
│ 
│ * Part 1
│ ** Chapter 1
│ #+LaTeX: \input{chapter1.tex}
└────

When I eval my function, `latexmk' compiles the entire book with the
`-pvc' option, which keeps the session open, and if it detects any
changes to the entire document, it recompiles and refresh the pdf view.
For example, if I edit one of the subdocuments and run
`org-publish-current-file', everything is automatically recompiled.

When I have the project folder ready, I add this to
`org-publish-project-alist' (this is an example from one of the books I
did recently):

┌────
│ ("cartas-org"
│  :base-directory "~/Git/cartas/libro/org/"
│  :base-extension "org"
│  ;; Directorio para los ficheros *.tex
│  :publishing-directory "~/Git/cartas/libro/tex/"
│  :publishing-function org-latex-publish-to-latex
│  :body-only t ;; this is important!
│  :exclude "cartas-master\\.org\\|bibliografia-cartas\\.org"
│  :recursive t)
│ 
│ ("cartas-img"
│  :base-directory "~/Git/cartas/libro/org/img/"
│  :base-extension "jpg\\|png"
│  :publishing-directory "~/Git/cartas/libro/tex/img/"
│  :recursive t
│  :publishing-function org-publish-attachment)
│ 
│ ("cartas" :components ("cartas-tex" "cartas-img"))
└────

And finally, this is the function that compiles everything (each project
usually has some local variables, like the value of ’jobname’ or the
status of the printing proofs).

Nota Bene: The reason for using async is that in some projects,
especially bilingual editions, I need to pre-compile some files first.
Under normal conditions I don't think it's necessary to use async, since
org-publish just exports everything to .tex documents (short timeout)
and then start-process-shell-command run latexmk asynchronously.

Best regards,

Juan Manuel 

┌────
│ (require 'async)
│ (require 'projectile)
│ 
│ (defun latexmk-compile-project-async ()
│   (interactive)
│   (let*
│       ((project-root (projectile-project-root))
│        (master-file (read-file-name "Compile: "
│ 				    (concat project-root "libro/org/")))
│        (master-file-tex (file-name-sans-extension
│ 			 (expand-file-name master-file)))
│        (dir-tex (file-name-directory
│ 		 (expand-file-name
│ 		  (replace-regexp-in-string "/org/" "/tex/" master-file)))))
│     ;; save the master document
│     (with-current-buffer
│ 	(find-file-noselect master-file)
│       (save-buffer))
│     (async-start
│      (lambda ()
│        (load "~/.emacs")
│        (with-current-buffer
│ 	   (find-file-noselect master-file)
│ 	 (org-show-all)
│ 	 (save-buffer)
│ 	 (org-latex-export-to-latex nil nil nil nil nil))
│        ;; remove all old auxiliary files before compiling
│        (shell-command (concat "rm -r " dir-tex (file-name-base master-file-tex) "*"))
│        (shell-command (concat "mv " master-file-tex ".tex" " " dir-tex))
│        "Document exported")
│      (lambda (resultado)
│        (message resultado)
│        (let
│ 	   ((default-directory dir-tex)
│ 	    (jobname (if (and jobname-local printing-proofs-state)
│ 			 (concat jobname-local "_" printing-proofs-state "_"
│ 				 (format-time-string "%d-%m-%y"))
│ 		       (concat (file-name-sans-extension
│ 				(file-name-nondirectory master-file-tex))
│ 			       "_"
│ 			       (format-time-string "%d-%m-%y")))))
│ 	 (start-process-shell-command
│ 	  "project"
│ 	  "*project*"
│ 	  (concat
│ 	   "latexmk"
│ 	   " -jobname="
│ 	   jobname
│ 	   " -pvc -lualatex -e '$lualatex=q/lualatex %O -shell-escape %S/' "
│ 	   (file-name-nondirectory master-file-tex)
│ 	   ".tex")))))))
└────

Re: [tip] org-publish to work with (very) large books

[Christian Moe]

Thanks for this, really interesting.

Do I understand correctly that the main advantage of this approach (over
#+INCLUDE) is the ability to continuously update preview of the whole
book with latexmk -pvc even if you only re-export one chapter from
Org-mode?

I couldn't find the :body-only publishing option in the docs ...?

Yours,
Christian

Juan Manuel Macías writes:

> Hi all,
>
> - tl; dr: I describe here my workflow with org-publish to work with long
> books.
>
> —
>
> I discovered a long time ago that `org-publish' not only works very well
> for managing websites but also for working with long and complex books
> with many parts, with output to LaTeX/PDF. I developed a workflow around
> it that has been quite productive for me, and that I briefly describe
> here in case someone finds it useful and wants to try it or modify/adapt
> it to their needs. I usually use it for my typesetting work, but I think
> it can also be useful for personal projects, such as doctoral theses.
>
> First of all, each folder of my project-books has the same structure:
> two subdirectories named `/org' and `/tex', for the source `org' files
> and for the output `.tex' documents, respectively. And, inside the `org'
> directory I include a `setup' file, an `elisp' file (for export
> filters), and another `/img' directory for image files. Each `org' file
> is a part of the book, and usually begins simply with the directives:
>
> ┌────
> │ #+SETUPFILE: xxx.setup
> │ #+INCLUDE: "elisp"
> └────
>
> `Org-publish' exports the subdocuments (body only!) as `.tex' documents
> in the `/tex' folder, but they are not compiled.
>
> What gets compiled is a master `.org' file, which is also inside the
> `org' folder. I compile this master file using an asynchronous function
> that calls `latexmk'. I put all the LaTeX configuration, the packages to
> load, the (re)defined commands and macros, the necessary Lua code, etc.
> in a `.sty' file that I load at the very beginning of the master
> document. Subdocuments are loaded into this file by the LaTeX command
> (\input), not by the org #+INCLUDE directive. So the master file usually
> looks like this:
>
> ┌────
> │ #+LaTeX_CLASS: my-custom-latex-class
> │ #+LaTeX_Header: \input{my-custom-conf.sty}
> │ #+SETUPFILE: xxxx.setup
> │ #+INCLUDE: "elisp"
> │
> │ * Part 1
> │ ** Chapter 1
> │ #+LaTeX: \input{chapter1.tex}
> └────
>
> When I eval my function, `latexmk' compiles the entire book with the
> `-pvc' option, which keeps the session open, and if it detects any
> changes to the entire document, it recompiles and refresh the pdf view.
> For example, if I edit one of the subdocuments and run
> `org-publish-current-file', everything is automatically recompiled.
>
> When I have the project folder ready, I add this to
> `org-publish-project-alist' (this is an example from one of the books I
> did recently):
>
> ┌────
> │ ("cartas-org"
> │  :base-directory "~/Git/cartas/libro/org/"
> │  :base-extension "org"
> │  ;; Directorio para los ficheros *.tex
> │  :publishing-directory "~/Git/cartas/libro/tex/"
> │  :publishing-function org-latex-publish-to-latex
> │  :body-only t ;; this is important!
> │  :exclude "cartas-master\\.org\\|bibliografia-cartas\\.org"
> │  :recursive t)
> │
> │ ("cartas-img"
> │  :base-directory "~/Git/cartas/libro/org/img/"
> │  :base-extension "jpg\\|png"
> │  :publishing-directory "~/Git/cartas/libro/tex/img/"
> │  :recursive t
> │  :publishing-function org-publish-attachment)
> │
> │ ("cartas" :components ("cartas-tex" "cartas-img"))
> └────
>
> And finally, this is the function that compiles everything (each project
> usually has some local variables, like the value of ’jobname’ or the
> status of the printing proofs).
>
> Nota Bene: The reason for using async is that in some projects,
> especially bilingual editions, I need to pre-compile some files first.
> Under normal conditions I don't think it's necessary to use async, since
> org-publish just exports everything to .tex documents (short timeout)
> and then start-process-shell-command run latexmk asynchronously.
>
> Best regards,
>
> Juan Manuel
>
> ┌────
> │ (require 'async)
> │ (require 'projectile)
> │
> │ (defun latexmk-compile-project-async ()
> │   (interactive)
> │   (let*
> │       ((project-root (projectile-project-root))
> │        (master-file (read-file-name "Compile: "
> │ 				    (concat project-root "libro/org/")))
> │        (master-file-tex (file-name-sans-extension
> │ 			 (expand-file-name master-file)))
> │        (dir-tex (file-name-directory
> │ 		 (expand-file-name
> │ 		  (replace-regexp-in-string "/org/" "/tex/" master-file)))))
> │     ;; save the master document
> │     (with-current-buffer
> │ 	(find-file-noselect master-file)
> │       (save-buffer))
> │     (async-start
> │      (lambda ()
> │        (load "~/.emacs")
> │        (with-current-buffer
> │ 	   (find-file-noselect master-file)
> │ 	 (org-show-all)
> │ 	 (save-buffer)
> │ 	 (org-latex-export-to-latex nil nil nil nil nil))
> │        ;; remove all old auxiliary files before compiling
> │        (shell-command (concat "rm -r " dir-tex (file-name-base master-file-tex) "*"))
> │        (shell-command (concat "mv " master-file-tex ".tex" " " dir-tex))
> │        "Document exported")
> │      (lambda (resultado)
> │        (message resultado)
> │        (let
> │ 	   ((default-directory dir-tex)
> │ 	    (jobname (if (and jobname-local printing-proofs-state)
> │ 			 (concat jobname-local "_" printing-proofs-state "_"
> │ 				 (format-time-string "%d-%m-%y"))
> │ 		       (concat (file-name-sans-extension
> │ 				(file-name-nondirectory master-file-tex))
> │ 			       "_"
> │ 			       (format-time-string "%d-%m-%y")))))
> │ 	 (start-process-shell-command
> │ 	  "project"
> │ 	  "*project*"
> │ 	  (concat
> │ 	   "latexmk"
> │ 	   " -jobname="
> │ 	   jobname
> │ 	   " -pvc -lualatex -e '$lualatex=q/lualatex %O -shell-escape %S/' "
> │ 	   (file-name-nondirectory master-file-tex)
> │ 	   ".tex")))))))
> └────

Re: [tip] org-publish to work with (very) large books

[Ihor Radchenko]

Christian Moe <mail@christianmoe.com> writes:

> Do I understand correctly that the main advantage of this approach (over
> #+INCLUDE) is the ability to continuously update preview of the whole
> book with latexmk -pvc even if you only re-export one chapter from
> Org-mode?

I am not sure why Juan did not use include. Include would not require
LaTeX to re-compile unchanged files. See
https://tex.stackexchange.com/questions/246/when-should-i-use-input-vs-include

> I couldn't find the :body-only publishing option in the docs ...?

See [[info:org#The Export Dispatcher][org#The Export Dispatcher]]

Best,
Ihor

Re: [tip] org-publish to work with (very) large books

[Christian Moe]

I see, thanks.

Ought this to be documented at [[info:org#Publishing options]], perhaps?

Yours,
Christian

Ihor Radchenko writes:

> Christian Moe <mail@christianmoe.com> writes:
>
>> Do I understand correctly that the main advantage of this approach (over
>> #+INCLUDE) is the ability to continuously update preview of the whole
>> book with latexmk -pvc even if you only re-export one chapter from
>> Org-mode?
>
> I am not sure why Juan did not use include. Include would not require
> LaTeX to re-compile unchanged files. See
> https://tex.stackexchange.com/questions/246/when-should-i-use-input-vs-include
>
>> I couldn't find the :body-only publishing option in the docs ...?
>
> See [[info:org#The Export Dispatcher][org#The Export Dispatcher]]
>
> Best,
> Ihor

Re: [tip] org-publish to work with (very) large books

[Ihor Radchenko]

Christian Moe <mail@christianmoe.com> writes:

> I see, thanks.
>
> Ought this to be documented at [[info:org#Publishing options]], perhaps?

Maybe. I think org-publish-project-alist docstring has many more
details. Someone™ should update the manual. Patches are welcome :)

Best,
Ihor

Re: [tip] org-publish to work with (very) large books

[Juan Manuel Macías]

Hi Ihor and Christian,

Ihor Radchenko writes:

> Christian Moe <mail@christianmoe.com> writes:
>
>> Do I understand correctly that the main advantage of this approach (over
>> #+INCLUDE) is the ability to continuously update preview of the whole
>> book with latexmk -pvc even if you only re-export one chapter from
>> Org-mode?
>
> I am not sure why Juan did not use include. Include would not require
> LaTeX to re-compile unchanged files. See
> https://tex.stackexchange.com/questions/246/when-should-i-use-input-vs-include
>
>> I couldn't find the :body-only publishing option in the docs ...?
>
> See [[info:org#The Export Dispatcher][org#The Export Dispatcher]]
>
> Best,
> Ihor
>

Sorry for not explaining the \input part in more detail. I think the
essential part here is that all the .tex files (the subdocuments) are
already created by org-publish before I compile the master document. The
master document simply stores all the subdocuments: I use
\input{subdocument.tex} instead of the org #+INCLUDE directive (not the
LaTeX \include command). The master document calls ready-made TeX files,
not Org files. And it is independent of the whole org-publish process,
which is responsible for creating only the parts of the book. This
procedure, apart from being able to compile parts of the book in real
time with latexmk -pvc, allows me to have more control over these parts.
But it makes more sense to use it when dealing with very long books. The
first time I used it was in a book of more than 1000 pages :-)

The skeleton of the process is that subdocuments are produced with
org-publish (as uncompiled tex files) and the master document is
exported to tex from org and then compiled with latexmk inside /tex
directory.

Best regards,

Juan Manuel 

Re: [tip] org-publish to work with (very) large books

[Christian Moe]

Thanks, Juan!

Yours,
Christian

Juan Manuel Macías writes:

> Hi Ihor and Christian,
>
> Ihor Radchenko writes:
>
>> Christian Moe <mail@christianmoe.com> writes:
>>
>>> Do I understand correctly that the main advantage of this approach (over
>>> #+INCLUDE) is the ability to continuously update preview of the whole
>>> book with latexmk -pvc even if you only re-export one chapter from
>>> Org-mode?
>>
>> I am not sure why Juan did not use include. Include would not require
>> LaTeX to re-compile unchanged files. See
>> https://tex.stackexchange.com/questions/246/when-should-i-use-input-vs-include
>>
>>> I couldn't find the :body-only publishing option in the docs ...?
>>
>> See [[info:org#The Export Dispatcher][org#The Export Dispatcher]]
>>
>> Best,
>> Ihor
>>
>
> Sorry for not explaining the \input part in more detail. I think the
> essential part here is that all the .tex files (the subdocuments) are
> already created by org-publish before I compile the master document. The
> master document simply stores all the subdocuments: I use
> \input{subdocument.tex} instead of the org #+INCLUDE directive (not the
> LaTeX \include command). The master document calls ready-made TeX files,
> not Org files. And it is independent of the whole org-publish process,
> which is responsible for creating only the parts of the book. This
> procedure, apart from being able to compile parts of the book in real
> time with latexmk -pvc, allows me to have more control over these parts.
> But it makes more sense to use it when dealing with very long books. The
> first time I used it was in a book of more than 1000 pages :-)
>
> The skeleton of the process is that subdocuments are produced with
> org-publish (as uncompiled tex files) and the master document is
> exported to tex from org and then compiled with latexmk inside /tex
> directory.
>
> Best regards,
>
> Juan Manuel 

Re: [tip] org-publish to work with (very) large books

[Ihor Radchenko]

Juan Manuel Macías <maciaschain@posteo.net> writes:

> Sorry for not explaining the \input part in more detail. I think the
> essential part here is that all the .tex files (the subdocuments) are
> already created by org-publish before I compile the master document. The
> master document simply stores all the subdocuments: I use
> \input{subdocument.tex} instead of the org #+INCLUDE directive (not the
> LaTeX \include command). The master document calls ready-made TeX files,
> not Org files. And it is independent of the whole org-publish process,
> which is responsible for creating only the parts of the book.

> This
> procedure, apart from being able to compile parts of the book in real
> time with latexmk -pvc, allows me to have more control over these parts.
> But it makes more sense to use it when dealing with very long books. The
> first time I used it was in a book of more than 1000 pages :-)

I am not sure if I understand correctly. Do you mean that you only
preview the book parts you are currently working on via latexmk -pvc?
What kind of more control are you referring to?

Best,
Ihor

Re: [tip] org-publish to work with (very) large books

[Juan Manuel Macías]

Ihor Radchenko writes:

> I am not sure if I understand correctly. Do you mean that you only
> preview the book parts you are currently working on via latexmk -pvc?
> What kind of more control are you referring to?

The -pvc flag means that if latexmk detects any modification to any
document involved in the current job (a subdocument, the .sty file, a
.bib file, or whatever), it reruns the appropriate builds to bring the
pdf up to date, and it only stops when everything is up to date. I can
focus that action on parts of the book by commenting or uncommenting
elements in the master file.

The moment one breaks down a large piece of work into specialized parts,
one gains more control over that piece of work. And org-publish helps
manage all of that. It is about managing a large book as a website (via
org-publish). In short, the combination of org-publish, projectile and
latexmk is quite productive for me in this type of work.

Anyway, as they say that a picture is worth a thousand words, I have
made this short example video. This is a dictionary I produced a year
ago. Each dictionary entry has its own separate bibliographic list, so I
had to manage more than 100 separate bib files. I have all these files
inside an Org document, and I create them using org-babel-tangle. The
video shows editing a field in a bib file. I've removed the build time
from the video, as the entire book is almost a thousand pages long.

https://cloud.disroot.org/s/PiSaHqWZr25GfJY

Best regards,

Juan Manuel 

Re: [tip] org-publish to work with (very) large books

[Ihor Radchenko]

Juan Manuel Macías <maciaschain@posteo.net> writes:

> Ihor Radchenko writes:
>
>> I am not sure if I understand correctly. Do you mean that you only
>> preview the book parts you are currently working on via latexmk -pvc?
>> What kind of more control are you referring to?
>
> The -pvc flag means that if latexmk detects any modification to any
> document involved in the current job (a subdocument, the .sty file, a
> .bib file, or whatever), it reruns the appropriate builds to bring the
> pdf up to date, and it only stops when everything is up to date. I can
> focus that action on parts of the book by commenting or uncommenting
> elements in the master file.

> Anyway, as they say that a picture is worth a thousand words, I have
> made this short example video. This is a dictionary I produced a year
> ago. Each dictionary entry has its own separate bibliographic list, so I
> had to manage more than 100 separate bib files. I have all these files
> inside an Org document, and I create them using org-babel-tangle. The
> video shows editing a field in a bib file. I've removed the build time
> from the video, as the entire book is almost a thousand pages long.
>
> https://cloud.disroot.org/s/PiSaHqWZr25GfJY

Thanks for the clarification! So, you are previewing the whole book with
some \input statements commented out. It is an ok approach, unless you
need cross-references between chapters.

A more advanced approach would be using
\include + \includeonly instead of \input:

https://web.archive.org/web/20160627050806/http://www.howtotex.com/tips-tricks/faster-latex-part-i-compile-only-parts/

Also, FYI:

https://web.archive.org/web/20160712215709/http://www.howtotex.com:80/tips-tricks/faster-latex-part-iv-use-a-precompiled-preamble/

> The moment one breaks down a large piece of work into specialized parts,
> one gains more control over that piece of work. And org-publish helps
> manage all of that. It is about managing a large book as a website (via
> org-publish). In short, the combination of org-publish, projectile and
> latexmk is quite productive for me in this type of work.

This is a bit confusing. You still keep the book in a single giant Org
file. It indeed does not mean anything given that we can always narrow
to subtree, but I fail to see where you break the book into specialized
parts then (LaTeX performance trickery aside).

Best,
Ihor

Re: [tip] org-publish to work with (very) large books

[Juan Manuel Macías]

Ihor Radchenko writes:

> A more advanced approach would be using
> \include + \includeonly instead of \input:
>
> https://web.archive.org/web/20160627050806/http://www.howtotex.com/tips-tricks/faster-latex-part-i-compile-only-parts/

Yeah, \include and \includeonly save the .aux files for each part.
However, I think choosing between \input, \include or \includeonly is
not the important part here. I usually use \input for convenience,
because I have not needed in the work done to make references between
parts. You can choose any of the options, according to needs. Also this
procedure can be made more complex. For example, sometimes (when it
comes to a bilingual edition with facing pages), I also start from
precompiled documents together with tex (subdocument) files. The
precompiled documents are placed on the odd and even pages of the
bilingual part:

https://i.imgur.com/Jbjutmf.jpg

> Also, FYI:
>
> https://web.archive.org/web/20160712215709/http://www.howtotex.com:80/tips-tricks/faster-latex-part-iv-use-a-precompiled-preamble/

Using a precompiled preamble can improve compilation sometimes, but
other times it's not worth it. Also, I use a lot of code in Lua. When it
comes to a very complex preamble, with lots of code, it is usually more
practical to create a .sty file (that is, a package, in LaTeX parlance).
The difference is that I prefer to use org and org-babel-tangle instead
of the 'official' LaTeX suite docstript for writing packages, which I
find horribly hard, especially compared to the ease of Org :-)

Improving performance and compile time in TeX is an old topic, and there
are a few tricks here and there. But TeX is what Emacs is, both are
venerably old; and both are single-thread. There are more ''modern''
approaches, like Patoline or Sile (of course, based heavily on TeX,
which is the father of everything). Sile, especially, is very
interesting and from time to time I like to play with it. The problem
with these new projects is that they don't have the LaTeX package
ecosystem, and they are poorly documented. Well, Sile in particular is
the work of a single person. Links:

https://patoline.github.io/#documentation

https://sile-typesetter.org/

As for LuaTeX, which is the state of the art today in the TeX ecosystem,
it is nothing more than TeX + a lua interpreter + the implementation of
advanced features from previous engines like pdfTeX and the experimental
Omega/Alef. It has the advantage that it is a scriptable TeX (TeX
primitives can be controlled by Lua scripts, and truly amazing things[1]
can be achieved with very little effort[2]); it has the disadvantage that
the scripting language is Lua. The ideal would have been a Lisp-TeX ;-)

[1] The chickenize package contains many examples, some of them somewhat
absurd and not very useful in
appearance: https://www.ctan.org/pkg/chickenize

[2] https://tug.org/TUGboat/tb31-3/tb99isambert.pdf

>> The moment one breaks down a large piece of work into specialized parts,
>> one gains more control over that piece of work. And org-publish helps
>> manage all of that. It is about managing a large book as a website (via
>> org-publish). In short, the combination of org-publish, projectile and
>> latexmk is quite productive for me in this type of work.
>
> This is a bit confusing. You still keep the book in a single giant Org
> file. It indeed does not mean anything given that we can always narrow
> to subtree, but I fail to see where you break the book into specialized
> parts then (LaTeX performance trickery aside).

I think this is inaccurate. The book is split across multiple
subdocuments. The master file is just the 'outline' of the book.

Best regards,

Juan Manuel 

Re: [tip] org-publish to work with (very) large books

[Ihor Radchenko]

Juan Manuel Macías <maciaschain@posteo.net> writes:

> Improving performance and compile time in TeX is an old topic, and there
> are a few tricks here and there. But TeX is what Emacs is, both are
> venerably old; and both are single-thread.

Yet, the information is surprisingly scattered. I was unable to find a
single guide on the available possibilities. Mostly unanswered or
partially answered questions from users.

> There are more ''modern''
> approaches, like Patoline or Sile (of course, based heavily on TeX,
> which is the father of everything). Sile, especially, is very
> interesting and from time to time I like to play with it. The problem
> with these new projects is that they don't have the LaTeX package
> ecosystem, and they are poorly documented. Well, Sile in particular is
> the work of a single person. Links:
>
> https://patoline.github.io/#documentation
>
> https://sile-typesetter.org/

Thanks! This is interesting.

> As for LuaTeX, which is the state of the art today in the TeX ecosystem,
> it is nothing more than TeX + a lua interpreter + the implementation of
> advanced features from previous engines like pdfTeX and the experimental
> Omega/Alef. It has the advantage that it is a scriptable TeX (TeX
> primitives can be controlled by Lua scripts, and truly amazing things[1]
> can be achieved with very little effort[2]); it has the disadvantage that
> the scripting language is Lua. The ideal would have been a Lisp-TeX ;-)
>
> [1] The chickenize package contains many examples, some of them somewhat
> absurd and not very useful in
> appearance: https://www.ctan.org/pkg/chickenize
>
> [2] https://tug.org/TUGboat/tb31-3/tb99isambert.pdf

For me, the main problem with LuaTeX is that it is generally not
supported by publishers I deal with. Mostly, LaTeX is the requirement.
Some even demand Word documents ):

Hence, all the advanced features of LuaTeX cannot be used in real my
real publications and I cannot convince myself to dedicate time for
playing around with LuaTeX.

Do you have anything from LuaTeX in mind that could improve the current
ox-latex pdf export when LuaTeX is used as the TeX engine?

>>> The moment one breaks down a large piece of work into specialized parts,
>>> one gains more control over that piece of work. And org-publish helps
>>> manage all of that. It is about managing a large book as a website (via
>>> org-publish). In short, the combination of org-publish, projectile and
>>> latexmk is quite productive for me in this type of work.
>>
>> This is a bit confusing. You still keep the book in a single giant Org
>> file. It indeed does not mean anything given that we can always narrow
>> to subtree, but I fail to see where you break the book into specialized
>> parts then (LaTeX performance trickery aside).
>
> I think this is inaccurate. The book is split across multiple
> subdocuments. The master file is just the 'outline' of the book.

I see. After watching the video more carefully, I do see the your org
file only had the bibliographies. Not the actual book text.

Best,
Ihor

Re: [tip] org-publish to work with (very) large books

[Juan Manuel Macías]

Ihor Radchenko writes:

> Yet, the information is surprisingly scattered. I was unable to find a
> single guide on the available possibilities. Mostly unanswered or
> partially answered questions from users.

Yes you're right. In addition, what I have been testing is not a panacea
either. In general, when it comes to long and complex documents, there
is no other solution than to arm yourself with patience, launch
asynchronous processes and dedicate yourself to doing another task while
LaTeX does its job. And, of course, trust that there are no errors. The
only advantage to debugging the code of a LaTeX document is how much you
learn about LaTeX and TeX in the process. But many times it is something
that can become frustrating, and the log file can be more cryptic than a
Sumerian inscription. The cause/effect relationship in LaTeX errors can
be the most surrealistic things in the world :-D.

Luckily, there's texstackexchange, where the LaTeX core and LaTeX
package developers themselves write, which is an endless source of
help...

> Do you have anything from LuaTeX in mind that could improve the current
> ox-latex pdf export when LuaTeX is used as the TeX engine?

I've thought about it sometimes, but haven't been able to find anything
concrete for Org. LuaLaTeX cares that a well-formed LaTeX document is
delivered to it, and Org already does that very well. In LuaTeX you can
insert lua code between La(TeX) code. For example:

\begin{luacode}
local x = "foo"
local y = "bar"
tex.print (x .. " and " .. y)
\end{luacode}

But in Org we have all the power of Babel: Org wins.

In LuaTeX you can write functions as pre-process filters, and associate
these functions with different callbacks. For example, there is a
callback_input_buffer, but we already have something like that in Org,
and with a larger scope and not limited to output to LaTeX.

In general, the advanced features of LuaTeX are more typographical and
micro-typographical in nature, and I guess they are of little use to
Org. For example, I recently wrote this function that highlights in red
the text that is in a language other than the main language of the
document (in my case, Spanish, langid 80). Act low-level on the line
node, just before LuaTeX does the line break to create the paragraph:

\directlua{
w = node.new("whatsit","pdf_literal")
w.data = "1 0 0 rg"
z = node.new("whatsit","pdf_literal")
z.data = "0 g"
function other_langs(h,c)
   for n in node.traverse_id(0,h) do
      for x in node.traverse_id(node.id("glyph"),n.head) do
	 if x.lang < 80 or x.lang > 80 then
	    local before, after = node.copy(w), node.copy(z)
	    n.head = node.insert_before(n.head,x,before)
	    n = node.insert_after(n,x,after)
	 end
      end
   end
   return h
   end

luatexbase.add_to_callback('post_linebreak_filter', other_langs, 'other_langs')
}

According to the LuaTeX manual, "TeX’s nodes are represented in Lua as
userdata objects with a variable set of fields". What this function does
is simply manipulate the .lang field of the glyph nodes in an hlist node
(the line with its components).

Functions associated with the post_linebreak_filter callback are very
useful and productive, but from a purely typographical point of view.

At the pure LaTeX level, LuaLaTeX is not very different from LaTeX. Any
LaTeX document, generally speaking, can be compiled with LuaLaTeX, as
long as it is in utf8 and does not contain some pdfLaTeX- or
XelaTeX-specific commands. Today the compatibility between engines is
reasonably good, and more and more packages designed exclusively for
LuaTeX are uploaded to CTAN. The TeX ecosystem is notorious for its
slowness and conformism, but LuaTeX is meant to be the natural
replacement for pdfTeX. Sometime in the uncertain future :-)

Best regards,

Juan Manuel