* [tip] org-publish to work with (very) large books @ 2022-05-26 10:01 Juan Manuel Macías 2022-05-26 12:46 ` Christian Moe 0 siblings, 1 reply; 13+ messages in thread From: Juan Manuel Macías @ 2022-05-26 10:01 UTC (permalink / raw) To: orgmode Hi all, - tl; dr: I describe here my workflow with org-publish to work with long books. — I discovered a long time ago that `org-publish' not only works very well for managing websites but also for working with long and complex books with many parts, with output to LaTeX/PDF. I developed a workflow around it that has been quite productive for me, and that I briefly describe here in case someone finds it useful and wants to try it or modify/adapt it to their needs. I usually use it for my typesetting work, but I think it can also be useful for personal projects, such as doctoral theses. First of all, each folder of my project-books has the same structure: two subdirectories named `/org' and `/tex', for the source `org' files and for the output `.tex' documents, respectively. And, inside the `org' directory I include a `setup' file, an `elisp' file (for export filters), and another `/img' directory for image files. Each `org' file is a part of the book, and usually begins simply with the directives: ┌──── │ #+SETUPFILE: xxx.setup │ #+INCLUDE: "elisp" └──── `Org-publish' exports the subdocuments (body only!) as `.tex' documents in the `/tex' folder, but they are not compiled. What gets compiled is a master `.org' file, which is also inside the `org' folder. I compile this master file using an asynchronous function that calls `latexmk'. I put all the LaTeX configuration, the packages to load, the (re)defined commands and macros, the necessary Lua code, etc. in a `.sty' file that I load at the very beginning of the master document. Subdocuments are loaded into this file by the LaTeX command (\input), not by the org #+INCLUDE directive. So the master file usually looks like this: ┌──── │ #+LaTeX_CLASS: my-custom-latex-class │ #+LaTeX_Header: \input{my-custom-conf.sty} │ #+SETUPFILE: xxxx.setup │ #+INCLUDE: "elisp" │ │ * Part 1 │ ** Chapter 1 │ #+LaTeX: \input{chapter1.tex} └──── When I eval my function, `latexmk' compiles the entire book with the `-pvc' option, which keeps the session open, and if it detects any changes to the entire document, it recompiles and refresh the pdf view. For example, if I edit one of the subdocuments and run `org-publish-current-file', everything is automatically recompiled. When I have the project folder ready, I add this to `org-publish-project-alist' (this is an example from one of the books I did recently): ┌──── │ ("cartas-org" │ :base-directory "~/Git/cartas/libro/org/" │ :base-extension "org" │ ;; Directorio para los ficheros *.tex │ :publishing-directory "~/Git/cartas/libro/tex/" │ :publishing-function org-latex-publish-to-latex │ :body-only t ;; this is important! │ :exclude "cartas-master\\.org\\|bibliografia-cartas\\.org" │ :recursive t) │ │ ("cartas-img" │ :base-directory "~/Git/cartas/libro/org/img/" │ :base-extension "jpg\\|png" │ :publishing-directory "~/Git/cartas/libro/tex/img/" │ :recursive t │ :publishing-function org-publish-attachment) │ │ ("cartas" :components ("cartas-tex" "cartas-img")) └──── And finally, this is the function that compiles everything (each project usually has some local variables, like the value of ’jobname’ or the status of the printing proofs). Nota Bene: The reason for using async is that in some projects, especially bilingual editions, I need to pre-compile some files first. Under normal conditions I don't think it's necessary to use async, since org-publish just exports everything to .tex documents (short timeout) and then start-process-shell-command run latexmk asynchronously. Best regards, Juan Manuel ┌──── │ (require 'async) │ (require 'projectile) │ │ (defun latexmk-compile-project-async () │ (interactive) │ (let* │ ((project-root (projectile-project-root)) │ (master-file (read-file-name "Compile: " │ (concat project-root "libro/org/"))) │ (master-file-tex (file-name-sans-extension │ (expand-file-name master-file))) │ (dir-tex (file-name-directory │ (expand-file-name │ (replace-regexp-in-string "/org/" "/tex/" master-file))))) │ ;; save the master document │ (with-current-buffer │ (find-file-noselect master-file) │ (save-buffer)) │ (async-start │ (lambda () │ (load "~/.emacs") │ (with-current-buffer │ (find-file-noselect master-file) │ (org-show-all) │ (save-buffer) │ (org-latex-export-to-latex nil nil nil nil nil)) │ ;; remove all old auxiliary files before compiling │ (shell-command (concat "rm -r " dir-tex (file-name-base master-file-tex) "*")) │ (shell-command (concat "mv " master-file-tex ".tex" " " dir-tex)) │ "Document exported") │ (lambda (resultado) │ (message resultado) │ (let │ ((default-directory dir-tex) │ (jobname (if (and jobname-local printing-proofs-state) │ (concat jobname-local "_" printing-proofs-state "_" │ (format-time-string "%d-%m-%y")) │ (concat (file-name-sans-extension │ (file-name-nondirectory master-file-tex)) │ "_" │ (format-time-string "%d-%m-%y"))))) │ (start-process-shell-command │ "project" │ "*project*" │ (concat │ "latexmk" │ " -jobname=" │ jobname │ " -pvc -lualatex -e '$lualatex=q/lualatex %O -shell-escape %S/' " │ (file-name-nondirectory master-file-tex) │ ".tex"))))))) └──── ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [tip] org-publish to work with (very) large books 2022-05-26 10:01 [tip] org-publish to work with (very) large books Juan Manuel Macías @ 2022-05-26 12:46 ` Christian Moe 2022-05-26 13:11 ` Ihor Radchenko 0 siblings, 1 reply; 13+ messages in thread From: Christian Moe @ 2022-05-26 12:46 UTC (permalink / raw) To: emacs-orgmode Thanks for this, really interesting. Do I understand correctly that the main advantage of this approach (over #+INCLUDE) is the ability to continuously update preview of the whole book with latexmk -pvc even if you only re-export one chapter from Org-mode? I couldn't find the :body-only publishing option in the docs ...? Yours, Christian Juan Manuel Macías writes: > Hi all, > > - tl; dr: I describe here my workflow with org-publish to work with long > books. > > — > > I discovered a long time ago that `org-publish' not only works very well > for managing websites but also for working with long and complex books > with many parts, with output to LaTeX/PDF. I developed a workflow around > it that has been quite productive for me, and that I briefly describe > here in case someone finds it useful and wants to try it or modify/adapt > it to their needs. I usually use it for my typesetting work, but I think > it can also be useful for personal projects, such as doctoral theses. > > First of all, each folder of my project-books has the same structure: > two subdirectories named `/org' and `/tex', for the source `org' files > and for the output `.tex' documents, respectively. And, inside the `org' > directory I include a `setup' file, an `elisp' file (for export > filters), and another `/img' directory for image files. Each `org' file > is a part of the book, and usually begins simply with the directives: > > ┌──── > │ #+SETUPFILE: xxx.setup > │ #+INCLUDE: "elisp" > └──── > > `Org-publish' exports the subdocuments (body only!) as `.tex' documents > in the `/tex' folder, but they are not compiled. > > What gets compiled is a master `.org' file, which is also inside the > `org' folder. I compile this master file using an asynchronous function > that calls `latexmk'. I put all the LaTeX configuration, the packages to > load, the (re)defined commands and macros, the necessary Lua code, etc. > in a `.sty' file that I load at the very beginning of the master > document. Subdocuments are loaded into this file by the LaTeX command > (\input), not by the org #+INCLUDE directive. So the master file usually > looks like this: > > ┌──── > │ #+LaTeX_CLASS: my-custom-latex-class > │ #+LaTeX_Header: \input{my-custom-conf.sty} > │ #+SETUPFILE: xxxx.setup > │ #+INCLUDE: "elisp" > │ > │ * Part 1 > │ ** Chapter 1 > │ #+LaTeX: \input{chapter1.tex} > └──── > > When I eval my function, `latexmk' compiles the entire book with the > `-pvc' option, which keeps the session open, and if it detects any > changes to the entire document, it recompiles and refresh the pdf view. > For example, if I edit one of the subdocuments and run > `org-publish-current-file', everything is automatically recompiled. > > When I have the project folder ready, I add this to > `org-publish-project-alist' (this is an example from one of the books I > did recently): > > ┌──── > │ ("cartas-org" > │ :base-directory "~/Git/cartas/libro/org/" > │ :base-extension "org" > │ ;; Directorio para los ficheros *.tex > │ :publishing-directory "~/Git/cartas/libro/tex/" > │ :publishing-function org-latex-publish-to-latex > │ :body-only t ;; this is important! > │ :exclude "cartas-master\\.org\\|bibliografia-cartas\\.org" > │ :recursive t) > │ > │ ("cartas-img" > │ :base-directory "~/Git/cartas/libro/org/img/" > │ :base-extension "jpg\\|png" > │ :publishing-directory "~/Git/cartas/libro/tex/img/" > │ :recursive t > │ :publishing-function org-publish-attachment) > │ > │ ("cartas" :components ("cartas-tex" "cartas-img")) > └──── > > And finally, this is the function that compiles everything (each project > usually has some local variables, like the value of ’jobname’ or the > status of the printing proofs). > > Nota Bene: The reason for using async is that in some projects, > especially bilingual editions, I need to pre-compile some files first. > Under normal conditions I don't think it's necessary to use async, since > org-publish just exports everything to .tex documents (short timeout) > and then start-process-shell-command run latexmk asynchronously. > > Best regards, > > Juan Manuel > > ┌──── > │ (require 'async) > │ (require 'projectile) > │ > │ (defun latexmk-compile-project-async () > │ (interactive) > │ (let* > │ ((project-root (projectile-project-root)) > │ (master-file (read-file-name "Compile: " > │ (concat project-root "libro/org/"))) > │ (master-file-tex (file-name-sans-extension > │ (expand-file-name master-file))) > │ (dir-tex (file-name-directory > │ (expand-file-name > │ (replace-regexp-in-string "/org/" "/tex/" master-file))))) > │ ;; save the master document > │ (with-current-buffer > │ (find-file-noselect master-file) > │ (save-buffer)) > │ (async-start > │ (lambda () > │ (load "~/.emacs") > │ (with-current-buffer > │ (find-file-noselect master-file) > │ (org-show-all) > │ (save-buffer) > │ (org-latex-export-to-latex nil nil nil nil nil)) > │ ;; remove all old auxiliary files before compiling > │ (shell-command (concat "rm -r " dir-tex (file-name-base master-file-tex) "*")) > │ (shell-command (concat "mv " master-file-tex ".tex" " " dir-tex)) > │ "Document exported") > │ (lambda (resultado) > │ (message resultado) > │ (let > │ ((default-directory dir-tex) > │ (jobname (if (and jobname-local printing-proofs-state) > │ (concat jobname-local "_" printing-proofs-state "_" > │ (format-time-string "%d-%m-%y")) > │ (concat (file-name-sans-extension > │ (file-name-nondirectory master-file-tex)) > │ "_" > │ (format-time-string "%d-%m-%y"))))) > │ (start-process-shell-command > │ "project" > │ "*project*" > │ (concat > │ "latexmk" > │ " -jobname=" > │ jobname > │ " -pvc -lualatex -e '$lualatex=q/lualatex %O -shell-escape %S/' " > │ (file-name-nondirectory master-file-tex) > │ ".tex"))))))) > └──── ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [tip] org-publish to work with (very) large books 2022-05-26 12:46 ` Christian Moe @ 2022-05-26 13:11 ` Ihor Radchenko 2022-05-26 13:29 ` Christian Moe 2022-05-26 13:48 ` Juan Manuel Macías 0 siblings, 2 replies; 13+ messages in thread From: Ihor Radchenko @ 2022-05-26 13:11 UTC (permalink / raw) To: Christian Moe; +Cc: emacs-orgmode Christian Moe <mail@christianmoe.com> writes: > Do I understand correctly that the main advantage of this approach (over > #+INCLUDE) is the ability to continuously update preview of the whole > book with latexmk -pvc even if you only re-export one chapter from > Org-mode? I am not sure why Juan did not use include. Include would not require LaTeX to re-compile unchanged files. See https://tex.stackexchange.com/questions/246/when-should-i-use-input-vs-include > I couldn't find the :body-only publishing option in the docs ...? See [[info:org#The Export Dispatcher][org#The Export Dispatcher]] Best, Ihor ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [tip] org-publish to work with (very) large books 2022-05-26 13:11 ` Ihor Radchenko @ 2022-05-26 13:29 ` Christian Moe 2022-05-26 14:15 ` Ihor Radchenko 2022-05-26 13:48 ` Juan Manuel Macías 1 sibling, 1 reply; 13+ messages in thread From: Christian Moe @ 2022-05-26 13:29 UTC (permalink / raw) To: Ihor Radchenko; +Cc: Christian Moe, emacs-orgmode I see, thanks. Ought this to be documented at [[info:org#Publishing options]], perhaps? Yours, Christian Ihor Radchenko writes: > Christian Moe <mail@christianmoe.com> writes: > >> Do I understand correctly that the main advantage of this approach (over >> #+INCLUDE) is the ability to continuously update preview of the whole >> book with latexmk -pvc even if you only re-export one chapter from >> Org-mode? > > I am not sure why Juan did not use include. Include would not require > LaTeX to re-compile unchanged files. See > https://tex.stackexchange.com/questions/246/when-should-i-use-input-vs-include > >> I couldn't find the :body-only publishing option in the docs ...? > > See [[info:org#The Export Dispatcher][org#The Export Dispatcher]] > > Best, > Ihor ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [tip] org-publish to work with (very) large books 2022-05-26 13:29 ` Christian Moe @ 2022-05-26 14:15 ` Ihor Radchenko 0 siblings, 0 replies; 13+ messages in thread From: Ihor Radchenko @ 2022-05-26 14:15 UTC (permalink / raw) To: Christian Moe; +Cc: emacs-orgmode Christian Moe <mail@christianmoe.com> writes: > I see, thanks. > > Ought this to be documented at [[info:org#Publishing options]], perhaps? Maybe. I think org-publish-project-alist docstring has many more details. Someone™ should update the manual. Patches are welcome :) Best, Ihor ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [tip] org-publish to work with (very) large books 2022-05-26 13:11 ` Ihor Radchenko 2022-05-26 13:29 ` Christian Moe @ 2022-05-26 13:48 ` Juan Manuel Macías 2022-05-26 17:47 ` Christian Moe 2022-05-27 4:19 ` Ihor Radchenko 1 sibling, 2 replies; 13+ messages in thread From: Juan Manuel Macías @ 2022-05-26 13:48 UTC (permalink / raw) To: Ihor Radchenko; +Cc: Christian Moe, orgmode Hi Ihor and Christian, Ihor Radchenko writes: > Christian Moe <mail@christianmoe.com> writes: > >> Do I understand correctly that the main advantage of this approach (over >> #+INCLUDE) is the ability to continuously update preview of the whole >> book with latexmk -pvc even if you only re-export one chapter from >> Org-mode? > > I am not sure why Juan did not use include. Include would not require > LaTeX to re-compile unchanged files. See > https://tex.stackexchange.com/questions/246/when-should-i-use-input-vs-include > >> I couldn't find the :body-only publishing option in the docs ...? > > See [[info:org#The Export Dispatcher][org#The Export Dispatcher]] > > Best, > Ihor > Sorry for not explaining the \input part in more detail. I think the essential part here is that all the .tex files (the subdocuments) are already created by org-publish before I compile the master document. The master document simply stores all the subdocuments: I use \input{subdocument.tex} instead of the org #+INCLUDE directive (not the LaTeX \include command). The master document calls ready-made TeX files, not Org files. And it is independent of the whole org-publish process, which is responsible for creating only the parts of the book. This procedure, apart from being able to compile parts of the book in real time with latexmk -pvc, allows me to have more control over these parts. But it makes more sense to use it when dealing with very long books. The first time I used it was in a book of more than 1000 pages :-) The skeleton of the process is that subdocuments are produced with org-publish (as uncompiled tex files) and the master document is exported to tex from org and then compiled with latexmk inside /tex directory. Best regards, Juan Manuel ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [tip] org-publish to work with (very) large books 2022-05-26 13:48 ` Juan Manuel Macías @ 2022-05-26 17:47 ` Christian Moe 2022-05-27 4:19 ` Ihor Radchenko 1 sibling, 0 replies; 13+ messages in thread From: Christian Moe @ 2022-05-26 17:47 UTC (permalink / raw) To: Juan Manuel Macías; +Cc: Ihor Radchenko, Christian Moe, orgmode Thanks, Juan! Yours, Christian Juan Manuel Macías writes: > Hi Ihor and Christian, > > Ihor Radchenko writes: > >> Christian Moe <mail@christianmoe.com> writes: >> >>> Do I understand correctly that the main advantage of this approach (over >>> #+INCLUDE) is the ability to continuously update preview of the whole >>> book with latexmk -pvc even if you only re-export one chapter from >>> Org-mode? >> >> I am not sure why Juan did not use include. Include would not require >> LaTeX to re-compile unchanged files. See >> https://tex.stackexchange.com/questions/246/when-should-i-use-input-vs-include >> >>> I couldn't find the :body-only publishing option in the docs ...? >> >> See [[info:org#The Export Dispatcher][org#The Export Dispatcher]] >> >> Best, >> Ihor >> > > Sorry for not explaining the \input part in more detail. I think the > essential part here is that all the .tex files (the subdocuments) are > already created by org-publish before I compile the master document. The > master document simply stores all the subdocuments: I use > \input{subdocument.tex} instead of the org #+INCLUDE directive (not the > LaTeX \include command). The master document calls ready-made TeX files, > not Org files. And it is independent of the whole org-publish process, > which is responsible for creating only the parts of the book. This > procedure, apart from being able to compile parts of the book in real > time with latexmk -pvc, allows me to have more control over these parts. > But it makes more sense to use it when dealing with very long books. The > first time I used it was in a book of more than 1000 pages :-) > > The skeleton of the process is that subdocuments are produced with > org-publish (as uncompiled tex files) and the master document is > exported to tex from org and then compiled with latexmk inside /tex > directory. > > Best regards, > > Juan Manuel ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [tip] org-publish to work with (very) large books 2022-05-26 13:48 ` Juan Manuel Macías 2022-05-26 17:47 ` Christian Moe @ 2022-05-27 4:19 ` Ihor Radchenko 2022-05-27 11:39 ` Juan Manuel Macías 1 sibling, 1 reply; 13+ messages in thread From: Ihor Radchenko @ 2022-05-27 4:19 UTC (permalink / raw) To: Juan Manuel Macías; +Cc: Christian Moe, orgmode Juan Manuel Macías <maciaschain@posteo.net> writes: > Sorry for not explaining the \input part in more detail. I think the > essential part here is that all the .tex files (the subdocuments) are > already created by org-publish before I compile the master document. The > master document simply stores all the subdocuments: I use > \input{subdocument.tex} instead of the org #+INCLUDE directive (not the > LaTeX \include command). The master document calls ready-made TeX files, > not Org files. And it is independent of the whole org-publish process, > which is responsible for creating only the parts of the book. > This > procedure, apart from being able to compile parts of the book in real > time with latexmk -pvc, allows me to have more control over these parts. > But it makes more sense to use it when dealing with very long books. The > first time I used it was in a book of more than 1000 pages :-) I am not sure if I understand correctly. Do you mean that you only preview the book parts you are currently working on via latexmk -pvc? What kind of more control are you referring to? Best, Ihor ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [tip] org-publish to work with (very) large books 2022-05-27 4:19 ` Ihor Radchenko @ 2022-05-27 11:39 ` Juan Manuel Macías 2022-05-28 3:02 ` Ihor Radchenko 0 siblings, 1 reply; 13+ messages in thread From: Juan Manuel Macías @ 2022-05-27 11:39 UTC (permalink / raw) To: Ihor Radchenko; +Cc: orgmode Ihor Radchenko writes: > I am not sure if I understand correctly. Do you mean that you only > preview the book parts you are currently working on via latexmk -pvc? > What kind of more control are you referring to? The -pvc flag means that if latexmk detects any modification to any document involved in the current job (a subdocument, the .sty file, a .bib file, or whatever), it reruns the appropriate builds to bring the pdf up to date, and it only stops when everything is up to date. I can focus that action on parts of the book by commenting or uncommenting elements in the master file. The moment one breaks down a large piece of work into specialized parts, one gains more control over that piece of work. And org-publish helps manage all of that. It is about managing a large book as a website (via org-publish). In short, the combination of org-publish, projectile and latexmk is quite productive for me in this type of work. Anyway, as they say that a picture is worth a thousand words, I have made this short example video. This is a dictionary I produced a year ago. Each dictionary entry has its own separate bibliographic list, so I had to manage more than 100 separate bib files. I have all these files inside an Org document, and I create them using org-babel-tangle. The video shows editing a field in a bib file. I've removed the build time from the video, as the entire book is almost a thousand pages long. https://cloud.disroot.org/s/PiSaHqWZr25GfJY Best regards, Juan Manuel ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [tip] org-publish to work with (very) large books 2022-05-27 11:39 ` Juan Manuel Macías @ 2022-05-28 3:02 ` Ihor Radchenko 2022-05-28 8:59 ` Juan Manuel Macías 0 siblings, 1 reply; 13+ messages in thread From: Ihor Radchenko @ 2022-05-28 3:02 UTC (permalink / raw) To: Juan Manuel Macías; +Cc: orgmode Juan Manuel Macías <maciaschain@posteo.net> writes: > Ihor Radchenko writes: > >> I am not sure if I understand correctly. Do you mean that you only >> preview the book parts you are currently working on via latexmk -pvc? >> What kind of more control are you referring to? > > The -pvc flag means that if latexmk detects any modification to any > document involved in the current job (a subdocument, the .sty file, a > .bib file, or whatever), it reruns the appropriate builds to bring the > pdf up to date, and it only stops when everything is up to date. I can > focus that action on parts of the book by commenting or uncommenting > elements in the master file. > Anyway, as they say that a picture is worth a thousand words, I have > made this short example video. This is a dictionary I produced a year > ago. Each dictionary entry has its own separate bibliographic list, so I > had to manage more than 100 separate bib files. I have all these files > inside an Org document, and I create them using org-babel-tangle. The > video shows editing a field in a bib file. I've removed the build time > from the video, as the entire book is almost a thousand pages long. > > https://cloud.disroot.org/s/PiSaHqWZr25GfJY Thanks for the clarification! So, you are previewing the whole book with some \input statements commented out. It is an ok approach, unless you need cross-references between chapters. A more advanced approach would be using \include + \includeonly instead of \input: https://web.archive.org/web/20160627050806/http://www.howtotex.com/tips-tricks/faster-latex-part-i-compile-only-parts/ Also, FYI: https://web.archive.org/web/20160712215709/http://www.howtotex.com:80/tips-tricks/faster-latex-part-iv-use-a-precompiled-preamble/ > The moment one breaks down a large piece of work into specialized parts, > one gains more control over that piece of work. And org-publish helps > manage all of that. It is about managing a large book as a website (via > org-publish). In short, the combination of org-publish, projectile and > latexmk is quite productive for me in this type of work. This is a bit confusing. You still keep the book in a single giant Org file. It indeed does not mean anything given that we can always narrow to subtree, but I fail to see where you break the book into specialized parts then (LaTeX performance trickery aside). Best, Ihor ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [tip] org-publish to work with (very) large books 2022-05-28 3:02 ` Ihor Radchenko @ 2022-05-28 8:59 ` Juan Manuel Macías 2022-05-29 12:15 ` Ihor Radchenko 0 siblings, 1 reply; 13+ messages in thread From: Juan Manuel Macías @ 2022-05-28 8:59 UTC (permalink / raw) To: Ihor Radchenko; +Cc: orgmode Ihor Radchenko writes: > A more advanced approach would be using > \include + \includeonly instead of \input: > > https://web.archive.org/web/20160627050806/http://www.howtotex.com/tips-tricks/faster-latex-part-i-compile-only-parts/ Yeah, \include and \includeonly save the .aux files for each part. However, I think choosing between \input, \include or \includeonly is not the important part here. I usually use \input for convenience, because I have not needed in the work done to make references between parts. You can choose any of the options, according to needs. Also this procedure can be made more complex. For example, sometimes (when it comes to a bilingual edition with facing pages), I also start from precompiled documents together with tex (subdocument) files. The precompiled documents are placed on the odd and even pages of the bilingual part: https://i.imgur.com/Jbjutmf.jpg > Also, FYI: > > https://web.archive.org/web/20160712215709/http://www.howtotex.com:80/tips-tricks/faster-latex-part-iv-use-a-precompiled-preamble/ Using a precompiled preamble can improve compilation sometimes, but other times it's not worth it. Also, I use a lot of code in Lua. When it comes to a very complex preamble, with lots of code, it is usually more practical to create a .sty file (that is, a package, in LaTeX parlance). The difference is that I prefer to use org and org-babel-tangle instead of the 'official' LaTeX suite docstript for writing packages, which I find horribly hard, especially compared to the ease of Org :-) Improving performance and compile time in TeX is an old topic, and there are a few tricks here and there. But TeX is what Emacs is, both are venerably old; and both are single-thread. There are more ''modern'' approaches, like Patoline or Sile (of course, based heavily on TeX, which is the father of everything). Sile, especially, is very interesting and from time to time I like to play with it. The problem with these new projects is that they don't have the LaTeX package ecosystem, and they are poorly documented. Well, Sile in particular is the work of a single person. Links: https://patoline.github.io/#documentation https://sile-typesetter.org/ As for LuaTeX, which is the state of the art today in the TeX ecosystem, it is nothing more than TeX + a lua interpreter + the implementation of advanced features from previous engines like pdfTeX and the experimental Omega/Alef. It has the advantage that it is a scriptable TeX (TeX primitives can be controlled by Lua scripts, and truly amazing things[1] can be achieved with very little effort[2]); it has the disadvantage that the scripting language is Lua. The ideal would have been a Lisp-TeX ;-) [1] The chickenize package contains many examples, some of them somewhat absurd and not very useful in appearance: https://www.ctan.org/pkg/chickenize [2] https://tug.org/TUGboat/tb31-3/tb99isambert.pdf >> The moment one breaks down a large piece of work into specialized parts, >> one gains more control over that piece of work. And org-publish helps >> manage all of that. It is about managing a large book as a website (via >> org-publish). In short, the combination of org-publish, projectile and >> latexmk is quite productive for me in this type of work. > > This is a bit confusing. You still keep the book in a single giant Org > file. It indeed does not mean anything given that we can always narrow > to subtree, but I fail to see where you break the book into specialized > parts then (LaTeX performance trickery aside). I think this is inaccurate. The book is split across multiple subdocuments. The master file is just the 'outline' of the book. Best regards, Juan Manuel ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [tip] org-publish to work with (very) large books 2022-05-28 8:59 ` Juan Manuel Macías @ 2022-05-29 12:15 ` Ihor Radchenko 2022-05-29 18:01 ` Juan Manuel Macías 0 siblings, 1 reply; 13+ messages in thread From: Ihor Radchenko @ 2022-05-29 12:15 UTC (permalink / raw) To: Juan Manuel Macías; +Cc: orgmode Juan Manuel Macías <maciaschain@posteo.net> writes: > Improving performance and compile time in TeX is an old topic, and there > are a few tricks here and there. But TeX is what Emacs is, both are > venerably old; and both are single-thread. Yet, the information is surprisingly scattered. I was unable to find a single guide on the available possibilities. Mostly unanswered or partially answered questions from users. > There are more ''modern'' > approaches, like Patoline or Sile (of course, based heavily on TeX, > which is the father of everything). Sile, especially, is very > interesting and from time to time I like to play with it. The problem > with these new projects is that they don't have the LaTeX package > ecosystem, and they are poorly documented. Well, Sile in particular is > the work of a single person. Links: > > https://patoline.github.io/#documentation > > https://sile-typesetter.org/ Thanks! This is interesting. > As for LuaTeX, which is the state of the art today in the TeX ecosystem, > it is nothing more than TeX + a lua interpreter + the implementation of > advanced features from previous engines like pdfTeX and the experimental > Omega/Alef. It has the advantage that it is a scriptable TeX (TeX > primitives can be controlled by Lua scripts, and truly amazing things[1] > can be achieved with very little effort[2]); it has the disadvantage that > the scripting language is Lua. The ideal would have been a Lisp-TeX ;-) > > [1] The chickenize package contains many examples, some of them somewhat > absurd and not very useful in > appearance: https://www.ctan.org/pkg/chickenize > > [2] https://tug.org/TUGboat/tb31-3/tb99isambert.pdf For me, the main problem with LuaTeX is that it is generally not supported by publishers I deal with. Mostly, LaTeX is the requirement. Some even demand Word documents ): Hence, all the advanced features of LuaTeX cannot be used in real my real publications and I cannot convince myself to dedicate time for playing around with LuaTeX. Do you have anything from LuaTeX in mind that could improve the current ox-latex pdf export when LuaTeX is used as the TeX engine? >>> The moment one breaks down a large piece of work into specialized parts, >>> one gains more control over that piece of work. And org-publish helps >>> manage all of that. It is about managing a large book as a website (via >>> org-publish). In short, the combination of org-publish, projectile and >>> latexmk is quite productive for me in this type of work. >> >> This is a bit confusing. You still keep the book in a single giant Org >> file. It indeed does not mean anything given that we can always narrow >> to subtree, but I fail to see where you break the book into specialized >> parts then (LaTeX performance trickery aside). > > I think this is inaccurate. The book is split across multiple > subdocuments. The master file is just the 'outline' of the book. I see. After watching the video more carefully, I do see the your org file only had the bibliographies. Not the actual book text. Best, Ihor ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [tip] org-publish to work with (very) large books 2022-05-29 12:15 ` Ihor Radchenko @ 2022-05-29 18:01 ` Juan Manuel Macías 0 siblings, 0 replies; 13+ messages in thread From: Juan Manuel Macías @ 2022-05-29 18:01 UTC (permalink / raw) To: Ihor Radchenko; +Cc: orgmode Ihor Radchenko writes: > Yet, the information is surprisingly scattered. I was unable to find a > single guide on the available possibilities. Mostly unanswered or > partially answered questions from users. Yes you're right. In addition, what I have been testing is not a panacea either. In general, when it comes to long and complex documents, there is no other solution than to arm yourself with patience, launch asynchronous processes and dedicate yourself to doing another task while LaTeX does its job. And, of course, trust that there are no errors. The only advantage to debugging the code of a LaTeX document is how much you learn about LaTeX and TeX in the process. But many times it is something that can become frustrating, and the log file can be more cryptic than a Sumerian inscription. The cause/effect relationship in LaTeX errors can be the most surrealistic things in the world :-D. Luckily, there's texstackexchange, where the LaTeX core and LaTeX package developers themselves write, which is an endless source of help... > Do you have anything from LuaTeX in mind that could improve the current > ox-latex pdf export when LuaTeX is used as the TeX engine? I've thought about it sometimes, but haven't been able to find anything concrete for Org. LuaLaTeX cares that a well-formed LaTeX document is delivered to it, and Org already does that very well. In LuaTeX you can insert lua code between La(TeX) code. For example: \begin{luacode} local x = "foo" local y = "bar" tex.print (x .. " and " .. y) \end{luacode} But in Org we have all the power of Babel: Org wins. In LuaTeX you can write functions as pre-process filters, and associate these functions with different callbacks. For example, there is a callback_input_buffer, but we already have something like that in Org, and with a larger scope and not limited to output to LaTeX. In general, the advanced features of LuaTeX are more typographical and micro-typographical in nature, and I guess they are of little use to Org. For example, I recently wrote this function that highlights in red the text that is in a language other than the main language of the document (in my case, Spanish, langid 80). Act low-level on the line node, just before LuaTeX does the line break to create the paragraph: \directlua{ w = node.new("whatsit","pdf_literal") w.data = "1 0 0 rg" z = node.new("whatsit","pdf_literal") z.data = "0 g" function other_langs(h,c) for n in node.traverse_id(0,h) do for x in node.traverse_id(node.id("glyph"),n.head) do if x.lang < 80 or x.lang > 80 then local before, after = node.copy(w), node.copy(z) n.head = node.insert_before(n.head,x,before) n = node.insert_after(n,x,after) end end end return h end luatexbase.add_to_callback('post_linebreak_filter', other_langs, 'other_langs') } According to the LuaTeX manual, "TeX’s nodes are represented in Lua as userdata objects with a variable set of fields". What this function does is simply manipulate the .lang field of the glyph nodes in an hlist node (the line with its components). Functions associated with the post_linebreak_filter callback are very useful and productive, but from a purely typographical point of view. At the pure LaTeX level, LuaLaTeX is not very different from LaTeX. Any LaTeX document, generally speaking, can be compiled with LuaLaTeX, as long as it is in utf8 and does not contain some pdfLaTeX- or XelaTeX-specific commands. Today the compatibility between engines is reasonably good, and more and more packages designed exclusively for LuaTeX are uploaded to CTAN. The TeX ecosystem is notorious for its slowness and conformism, but LuaTeX is meant to be the natural replacement for pdfTeX. Sometime in the uncertain future :-) Best regards, Juan Manuel ^ permalink raw reply [flat|nested] 13+ messages in thread
end of thread, other threads:[~2022-05-29 18:02 UTC | newest] Thread overview: 13+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2022-05-26 10:01 [tip] org-publish to work with (very) large books Juan Manuel Macías 2022-05-26 12:46 ` Christian Moe 2022-05-26 13:11 ` Ihor Radchenko 2022-05-26 13:29 ` Christian Moe 2022-05-26 14:15 ` Ihor Radchenko 2022-05-26 13:48 ` Juan Manuel Macías 2022-05-26 17:47 ` Christian Moe 2022-05-27 4:19 ` Ihor Radchenko 2022-05-27 11:39 ` Juan Manuel Macías 2022-05-28 3:02 ` Ihor Radchenko 2022-05-28 8:59 ` Juan Manuel Macías 2022-05-29 12:15 ` Ihor Radchenko 2022-05-29 18:01 ` Juan Manuel Macías
Code repositories for project(s) associated with this public inbox https://git.savannah.gnu.org/cgit/emacs/org-mode.git This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).