Understanding Publish (ox-publish.el)

Understanding Publish (ox-publish.el)

[jman]

Hello,

I'm trying to learn more about the publish export option or Org mode. Specifically I am using the 
function `org-latex-publish-to-pdf` to export Orgmode files into PDF.

Since this export backend is transparently using Latex, there is stuff going on behind my back that 
makes it a bit difficult debugging or customizing the formatting.

My header in the Orgmode file started as simple as:

  #+TITLE: Title of the document
  #+OPTIONS: toc:nil date:nil author:nil
  #+latex_header: \usepackage[a4paper,top=4cm,bottom=4cm]{geometry}

The Orgmode file is nothing but a long ordered list of items:

  * Title
  ** Subtitle
  (Lots of text)
  ** Subtlte II
  * Title II
  (Lots of text)
  ... more ...

and so on. After compiling to PDF, I learn that the default "template" (how is it called in Latex 
lingo? `\documentclass`?) used by `org-latex-publish-to-pdf` does not handle content exceeding the 
page length so I have to manually add some page breaks, clearly a hack:

  * Title
  ** Subtitle
  (Lots of text)
  #+latex: \clearpage
  ** Subtlte II
  #+latex: \clearpage
  * Title II
  (Lots of text)
  ... more ...

Why isn't this handled automatically for me? Is there a setting I should add in the org file? What's 
the default template used by `org-latex-publish-to-pdf`?

Second issue: I want hyperlinks to other documents to not have a border. On Latex the solution seems 
to use the hyperref package:

  +latex_header: \usepackage[hidelinks]{hyperref}

again, compiling triggers this error:

  ! LaTeX Error: Option clash for package hyperref.

This error hints at some defaults I cannot see, making the document customization difficult. Where 
are these defaults?

How can I get a feeling of what's happening when using this PDF publish option? I think I need a bit 
of a high-level overview, hope the author David O’Toole is around for some support :-)

Ideally I'm looking into a simple way to export from Orgmode to PDF, without needing to learn Latex.

Thank you for suggestions, ideas, etc.

Best,

Re: Understanding Publish (ox-publish.el)

[Christian Moe]

Hi,

The default document class is article.

1. First issue:

Not quite sure what you mean by saying that it "does not handle content
exceeding the page length". If the problem is with placing images,
tables etc., latex does its best, and manually adding \clearpage or
\newpage as needed tends to be the way it's done.

If what you want is automatic page breaks before every section and
subsection, you can try this trick at the beginning of the document:

  #+latex: \AddToHook{cmd/section/before}{\clearpage}
  #+latex: \AddToHook{cmd/subsection/before}{\clearpage}

(This only kicks in after the table of contents; if you want a page break
before the ToC as well, replace #+latex with #+latex_header.)

If you have an old latex installation, the above may not work. Try this instead:

  #+latex: \let\oldsection\section\renewcommand\section{\clearpage\oldsection}
  #+latex: \let\oldsubsection\subsection\renewcommand\subsection{\clearpage\oldsubsection}

2. Second issue: Getting rid of link borders.

To use hidelinks without options clashing, try this:

  #+latex_class_options: [hidelinks]

However, if you just want the ugly borders gone but would prefer the
links to stand out, you can instead customize the variable
org-latex-hyperref-template by adding `colorlinks=true'. You can play with
the color options as well; to get you started:

(setq org-latex-hyperref-template 
   "\\hypersetup{\n pdfauthor={%a},\n pdftitle={%t},\n
pdfkeywords={%k},\n pdfsubject={%d},\n pdfcreator={%c},\n
pdflang={%L},\n colorlinks=true,\n urlcolor=blue,\n linkcolor=blue\n,
citecolor=green}")


Yours,
Christian


jman <emacs-orgmode@city17.xyz> writes:

> Hello,
>
> I'm trying to learn more about the publish export option or Org mode. Specifically I am using the 
> function `org-latex-publish-to-pdf` to export Orgmode files into PDF.
>
> Since this export backend is transparently using Latex, there is stuff going on behind my back that 
> makes it a bit difficult debugging or customizing the formatting.
>
> My header in the Orgmode file started as simple as:
>
>   #+TITLE: Title of the document
>   #+OPTIONS: toc:nil date:nil author:nil
>
>   #+latex_header: \usepackage[a4paper,top=4cm,bottom=4cm]{geometry}
>
> The Orgmode file is nothing but a long ordered list of items:
>
>   * Title
>   ** Subtitle
>   (Lots of text)
>   ** Subtlte II
>   * Title II
>   (Lots of text)
>   ... more ...
>
> and so on. After compiling to PDF, I learn that the default "template" (how is it called in Latex 
> lingo? `\documentclass`?) used by `org-latex-publish-to-pdf` does not handle content exceeding the 
> page length so I have to manually add some page breaks, clearly a hack:
>
>   * Title
>   ** Subtitle
>   (Lots of text)
>
>   #+latex: \clearpage
>   ** Subtlte II
>
>   #+latex: \clearpage
>   * Title II
>
>   (Lots of text)
>   ... more ...
>
> Why isn't this handled automatically for me? Is there a setting I should add in the org file? What's 
> the default template used by `org-latex-publish-to-pdf`?
>
> Second issue: I want hyperlinks to other documents to not have a border. On Latex the solution seems 
> to use the hyperref package:
>
>   +latex_header: \usepackage[hidelinks]{hyperref}
>
> again, compiling triggers this error:
>
>   ! LaTeX Error: Option clash for package hyperref.
>
> This error hints at some defaults I cannot see, making the document customization difficult. Where 
> are these defaults?
>
> How can I get a feeling of what's happening when using this PDF publish option? I think I need a bit 
> of a high-level overview, hope the author David O’Toole is around for some support :-)
>
> Ideally I'm looking into a simple way to export from Orgmode to PDF, without needing to learn Latex.
>
> Thank you for suggestions, ideas, etc.
>
> Best,

Re: Understanding Publish (ox-publish.el)

[Dr. Arne Babenhauserheide]

[-- Attachment #1: Type: text/plain, Size: 818 bytes --]

jman <emacs-orgmode@city17.xyz> writes:
> I'm trying to learn more about the publish export option or Org mode.
> Specifically I am using the function `org-latex-publish-to-pdf` to
> export Orgmode files into PDF.
>
> Since this export backend is transparently using Latex, there is stuff
> going on behind my back that makes it a bit difficult debugging or
> customizing the formatting.

As a general hint when debugging PDF export: when you hit an error, then
first export to LaTeX and convert the file to PDF with pdflatex
(following the commands defined in the variable org-latex-pdf-process).

You can then fix it directly in LaTeX-side and afterwards bring the
change into the org-mode document.

Best wishes,
Arne
-- 
Unpolitisch sein
heißt politisch sein,
ohne es zu merken.
draketo.de

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 1125 bytes --]

Re: Understanding Publish (ox-publish.el)

[jman]

Christian Moe <mail@christianmoe.com> writes:

> 1. First issue:
>
> Not quite sure what you mean by saying that it "does not handle content
> exceeding the page length". If the problem is with placing images,
> tables etc., latex does its best, and manually adding \clearpage or
> \newpage as needed tends to be the way it's done.

Hey thanks for the tips!

To be clearer about what I mean I'll provide two sample documents. I hope they're good to reproduce 
the issue.

1) Compiling with ox-publish the following document produces the expected result. Text is 
automatically distributed on subsequent pages
https://paste.sr.ht/~jman/06239fa6624a1af23edfbb5154f5d9dcfdf8427a

2) Compiling this other document will produce strange empty pages. The content of the ordered list 
doesn't fit one page and instead of filling the next page, it will just "overflow" and "disappear":
https://paste.sr.ht/~jman/35434b0fdf08164abf6c59a413d60ea6a4d0e943

These samples are compiled with:

$ emacs --batch <file>.org --load ol-pdf.el --funcall org-publish-current-file

`ol-pdf.el` is nothing special, I think:

(require 'ol)
(require 'ox-publish)
(setq org-publish-project-alist
      '(("org-to-pdf"
         :base-directory "~/src"
         :base-extension "org"
         :publishing-directory "~/pdf"
         :publishing-function org-latex-publish-to-pdf
         :recursive t
         :with-toc nil
         :with-date nil
         :with-author nil
         )))

Additional info:
- I am using Emacs 29.4 (Org mode 9.6.15)
- I compile these file using the Debian/testing texlive-latex-* (2024.20240829-2) packages 
  (pdfTeX-1.40.26)
- I also tried compiling the PDF using `latexmk` (4.85) but the end result doesn't change

> 2. Second issue: Getting rid of link borders.
[...]

Thanks! Your suggestions gave me a few ideas to explore and this is now understood.

Best,

Re: Understanding Publish (ox-publish.el)

[jman]

jman <emacs-orgmode@city17.xyz> writes:

> 2) Compiling this other document will produce strange empty pages. The content of the ordered list
> doesn't fit one page and instead of filling the next page, it will just "overflow" and 
> "disappear":
> https://paste.sr.ht/~jman/35434b0fdf08164abf6c59a413d60ea6a4d0e943

I think the problem is that for the above file is a single ordered list and org-mode (or Latex?) for 
some reason are unable to split it. For an automatic split to happen, the single list must be split.

So another hack is to add random text in between! Example:

#+TITLE: Dokument Titel
#+LATEX_HEADER: \usepackage[paper=a4paper,top=3cm,bottom=3cm]{geometry}
* Title 1
** Subtitle
*** Subsubtitle
* Title 2
** Subtitle
*** Subsubtitle
* Title 3
** Subtitle
*** Subsubtitle
This text here is only to split the ordered list.
* Title 4
** Subtitle
*** Subsubtitle
* Title 5
** Subtitle
*** Subsubtitle
This text here is only to split the ordered list.
(etc.)

Now we have multiple lists and the page split happens according to the top and bottom margins set in 
the header.

I am not sure if there is a better way to work around this.

Best,

Re: Understanding Publish (ox-publish.el)

[Christian Moe]

It's due to Latex refusing to commit the typographic crime of leaving a
heading at the bottom of the page.

See:
https://tex.stackexchange.com/questions/57857/overfull-box-and-broken-pagination-with-page-that-contains-only-section-headings

(That one was also asked by an Org user -- the ease of outlining a
header structure in Org does tempt people into this situation, which one
might wish that Latex could handle with less logic and more grace.)

Note that this does /not/ happen with a list. (Try changing your long
outline from headers to a list -- you can do so quickly by including the
whole outline in the region and pressing `C-c -' -- and exporting it;
you get page breaks as expected.) So it doesn't prevent you from
exporting an outline to PDF.

Yours,
Christian


jman <emacs-orgmode@city17.xyz> writes:

> jman <emacs-orgmode@city17.xyz> writes:
>
>> 2) Compiling this other document will produce strange empty pages. The content of the ordered list
>> doesn't fit one page and instead of filling the next page, it will just "overflow" and 
>> "disappear":
>> https://paste.sr.ht/~jman/35434b0fdf08164abf6c59a413d60ea6a4d0e943
>
> I think the problem is that for the above file is a single ordered list and org-mode (or Latex?) for 
> some reason are unable to split it. For an automatic split to happen, the single list must be split.
>
> So another hack is to add random text in between! Example:
>
> #+TITLE: Dokument Titel
> #+LATEX_HEADER: \usepackage[paper=a4paper,top=3cm,bottom=3cm]{geometry}
>
> * Title 1
> ** Subtitle
> *** Subsubtitle
> * Title 2
> ** Subtitle
> *** Subsubtitle
> * Title 3
> ** Subtitle
> *** Subsubtitle
> This text here is only to split the ordered list.
> * Title 4
> ** Subtitle
> *** Subsubtitle
> * Title 5
> ** Subtitle
> *** Subsubtitle
> This text here is only to split the ordered list.
> (etc.)
>
> Now we have multiple lists and the page split happens according to the top and bottom margins set in 
> the header.
>
> I am not sure if there is a better way to work around this.
>
> Best,

Re: Understanding Publish (ox-publish.el)

[jman]

Christian Moe <mail@christianmoe.com> writes:

> It's due to Latex refusing to commit the typographic crime of leaving a
> heading at the bottom of the page.

Wow. I had no clue something like this could happen.

> Note that this does /not/ happen with a list. (Try changing your long
> outline from headers to a list)

Interesting. A "mixed" approach like the following work too:

* Title 1
  - Subtitle
    - Subsubtitle
* Title 2
  - Subtitle
    - Subsubtitle
(etc.)

As you point out, I just need to avoid having /only/ headings.

Thank you so much for hand-holding me to fully understanding the
problem!

Cheers,

Re: Understanding Publish (ox-publish.el)

[Christian Moe]

> As you point out, I just need to avoid having /only/ headings.

Or have only headings, but insert a strategic \clearpage here and there,
if you *want* the outline to have the look of headings.

> Thank you so much for hand-holding me to fully understanding the
> problem!

You're welcome!

Re: Understanding Publish (ox-publish.el)

[Rens Oliemans]

Christian Moe <mail@christianmoe.com> writes:

>> As you point out, I just need to avoid having /only/ headings.
>
> Or have only headings, but insert a strategic \clearpage here and there,
> if you *want* the outline to have the look of headings.

Or have only headings, but have some text in some of the sections: LaTeX is fine with
breaking after a paragraph. See:

https://paste.sr.ht/~rensoliemans/ff5bff303e02c559c53c3cd98e36b03a7d1dc00a

Which breaks and looks OK.

Best

Re: Understanding Publish (ox-publish.el)

[Ihor Radchenko]

Christian Moe <mail@christianmoe.com> writes:

> It's due to Latex refusing to commit the typographic crime of leaving a
> heading at the bottom of the page.
> ...

and is committing another typographic crime :)
I am wondering if this may somehow be addressed.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Understanding Publish (ox-publish.el)

[Christian Moe]

Ihor Radchenko <yantar92@posteo.net> writes:

> Christian Moe <mail@christianmoe.com> writes:
>
>> It's due to Latex refusing to commit the typographic crime of leaving a
>> heading at the bottom of the page.
>> ...
>
> and is committing another typographic crime :)
> I am wondering if this may somehow be addressed.

Indeed. The best place to address it would be in Latex, I suppose. Kind
of surprising that it hasn't been.

I don't suppose there's any way for Org to know beforehand how much
space an empty outline will take on the PDF page.

But Org could perhaps just check the outline and insert a page break every so
often after a run of empty headers? Something like this hook:

#+BEGIN_SRC elisp
  (defvar my/org-latex-pagebreak-every 20
    "Number of empty (sub)headings after which to insert pagebreak.")

  (defun my/org-latex-insert-pagebreaks (backend)
    "Insert page breaks in an empty outline on LaTeX export.
  A page break is inserted for every nth consecutive empty (sub)heading,
  where n is given by `my/org-latex-pagebreak-after', to allow an
  empty outline to break gracefully in PDF export."
    (when (eq backend 'latex)
      (let ((c 0))
        (org-map-entries
         (lambda ()
  	 ;; if entry is empty
  	 (unless (string-match (org-get-entry) "[[:graph:]]")
  	   (setq c (1+ c))
  	   (when (= c my/org-latex-pagebreak-every)
               (setq c 0) ; reset counter
  	     (org-end-of-line)
  	     (insert "\n\\clearpage\n"))))))))

  (add-hook 'org-export-before-parsing-functions #'my/org-latex-insert-pagebreaks)
#+END_SRC

Yours,
Christian

Re: Understanding Publish (ox-publish.el)

[Ihor Radchenko]

Christian Moe <mail@christianmoe.com> writes:

>> and is committing another typographic crime :)
>> I am wondering if this may somehow be addressed.
>
> Indeed. The best place to address it would be in Latex, I suppose. Kind
> of surprising that it hasn't been.

That's what I was hoping for...
Maybe some low-level TeX command or something.

> I don't suppose there's any way for Org to know beforehand how much
> space an empty outline will take on the PDF page.
>
> But Org could perhaps just check the outline and insert a page break every so
> often after a run of empty headers? Something like this hook:
> ...

I suspect that it is going to be fragile. Whether your idea work or not
will depend on the documentclass, fonts, font sizes, page sizes, etc.
I am afraid that such a partial solution is not worth the hassle - it
will complicate the exporter code for questionable gain.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>