Some hacks on top of org-publish

Some hacks on top of org-publish

[Nikolaos Chatzikonstantinou]

Hello,

I spent the past couple of days hacking together this new blog based
on org-publish that I'd like to show you (comments welcome!):

Blog: <https://createyourpersonalaccount.github.io/blog-v2/>
git repo: <https://github.com/createyourpersonalaccount/blog-v2>

The highlights are that I got

1) MathJax to work
2) TikZ commutative diagrams to work as SVG files
3) Source code blocks + colors (server-side; not a .js coloring client-side)
4) A nice navigation sidebar
5) I get to preview locally if I want but also I have a GitHub
workflow that automatically builds my blog; both views are the same.

In this format all I have to do is add .org files under the
content/pages/ directory (and .tex files under the content/img/
directory for the commutative diagrams) and run `make` and the static
blog is built.

I know there's several ways to improve but I want to bring a couple of
points where you might be able to help me:

1) I got colors in source blocks by post-processing with node.js +
highlight.js the raw HTML. Is it possible to write my own backend that
feeds the source block into my node app?
2) I got several issues with how the org files are exported as HTML,
for example I may want to insert an <article> tag, I may want a div
around content and a div around nav bar, etc. Not everything can be
solved with #+HTML: tags because I don't have precise control of where
those go. Can the HTML backend be edited or do I have to write my own
backend?
3) I am very inexperienced with org backends. What good resources are
out there for this subject?

Finally a highlight of the hacks I had to pull off: apparently
#+INCLUDE: will strip away inline backend information such as
@@html:<b>@@. I figured out that if I write something like
@@html@@a:@@:<b>@@, only the inner @@a:@@ is recognized and stripped,
which leaves me with what I originally wanted! (see
<https://github.com/createyourpersonalaccount/blog-v2/blob/main/content/publish.el#L14-L20>)

Regards,
Nikolaos Chatzikonstantinou

Re: Some hacks on top of org-publish

[Ihor Radchenko]

Nikolaos Chatzikonstantinou <nchatz314@gmail.com> writes:

> 3) Source code blocks + colors (server-side; not a .js coloring client-side)

Note that ox-html can fontify src blocks by itself. Via htmlize + CSS styles.

> 1) I got colors in source blocks by post-processing with node.js +
> highlight.js the raw HTML. Is it possible to write my own backend that
> feeds the source block into my node app?

Yes, of course. See my next answers. The whole export proces is hackable
on multiple levels.

> 2) I got several issues with how the org files are exported as HTML,
> for example I may want to insert an <article> tag, I may want a div
> around content and a div around nav bar, etc. Not everything can be
> solved with #+HTML: tags because I don't have precise control of where
> those go. Can the HTML backend be edited or do I have to write my own
> backend?

Check out https://orgmode.org/manual/Advanced-Export-Configuration.html

> 3) I am very inexperienced with org backends. What good resources are
> out there for this subject?

See https://orgmode.org/manual/Adding-Export-Backends.html
Org export backends can use inheritance. So, if you need to re-define a
part of backend (if filters are not enough), you can always create a
"subclass" by creating a "derived" backend.

> Finally a highlight of the hacks I had to pull off: apparently
> #+INCLUDE: will strip away inline backend information such as
> @@html:<b>@@. I figured out that if I write something like
> @@html@@a:@@:<b>@@, only the inner @@a:@@ is recognized and stripped,
> which leaves me with what I originally wanted! (see
> <https://github.com/createyourpersonalaccount/blog-v2/blob/main/content/publish.el#L14-L20>)

May you show more concrete example?
#+INCLUDE by itself does not remove export snippets.

-- 
Ihor Radchenko // yantar92,
Org mode maintainer,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Some hacks on top of org-publish

[Nikolaos Chatzikonstantinou]

[-- Attachment #1: Type: text/plain, Size: 1317 bytes --]

On Wed, Dec 25, 2024 at 8:47 AM Ihor Radchenko <yantar92@posteo.net> wrote:
>
> Nikolaos Chatzikonstantinou <nchatz314@gmail.com> writes:
>
> > Finally a highlight of the hacks I had to pull off: apparently
> > #+INCLUDE: will strip away inline backend information such as
> > @@html:<b>@@. I figured out that if I write something like
> > @@html@@a:@@:<b>@@, only the inner @@a:@@ is recognized and stripped,
> > which leaves me with what I originally wanted! (see
> > <https://github.com/createyourpersonalaccount/blog-v2/blob/main/content/publish.el#L14-L20>)
>
> May you show more concrete example?
> #+INCLUDE by itself does not remove export snippets.

It must not be #+INCLUDE then, but sitemap.org generation itself, I
misremembered when I wrote the e-mail. Here's a reproducer, see
attachment. Have this index.org file in its own directory and publish
it. Then view public/sitemap.html. You will see that the @@BACKEND:@@
stuff has been stripped away in the first sitemap entry instance,
while in the second it is there. When inspecting the generated
sitemap.org you will see the following line:

  - Without italics: [[file:./index.org][index]], 2024-12-26 -- With
italics: [[file:./index.org][index]],
@@html:<i>@@2024-12-26@@html:</i>@@

Regards,
Nikolaos Chatzikonstantinou

[-- Attachment #2: index.org --]
[-- Type: text/org, Size: 874 bytes --]

* Hello world

This is a test.

#+begin_comment
Local Variables:
org-publish-project-alist:
  (("test-project"
    :components ("test-component"))
   ("test-component"
    :auto-sitemap t
    :base-directory "."
    :base-extension "org"
    :publishing-function org-html-publish-to-html
    :publishing-directory "public"
    :sitemap-format-entry
    (lambda (entry sitemap-style project)
      (let* ((title (org-publish-find-title entry project))
             (date (org-publish-find-date entry project))
             (date-s (format-time-string "%Y-%m-%d" date)))
        (format (concat "Without italics: [[./%s][%s]], @@html:<i>@@%s@@html:</i>@@"
                        " -- "
                        "With italics: [[./%s][%s]], @@html@@a:@@:<i>@@%s@@html@@a:@@:</i>@@")
                entry title date-s
                entry title date-s)))))
End:
#+end_comment

Re: Some hacks on top of org-publish

[Ihor Radchenko]

Nikolaos Chatzikonstantinou <nchatz314@gmail.com> writes:

>> > Finally a highlight of the hacks I had to pull off: apparently
>> > #+INCLUDE: will strip away inline backend information such as
>> > @@html:<b>@@. I figured out that if I write something like
>> > @@html@@a:@@:<b>@@, only the inner @@a:@@ is recognized and stripped,
>> > which leaves me with what I originally wanted! (see
>> > <https://github.com/createyourpersonalaccount/blog-v2/blob/main/content/publish.el#L14-L20>)
>>
>> May you show more concrete example?
>> #+INCLUDE by itself does not remove export snippets.
>
> It must not be #+INCLUDE then, but sitemap.org generation itself, I
> misremembered when I wrote the e-mail. Here's a reproducer, see
> attachment. Have this index.org file in its own directory and publish
> it. Then view public/sitemap.html. You will see that the @@BACKEND:@@
> stuff has been stripped away in the first sitemap entry instance,
> while in the second it is there. When inspecting the generated
> sitemap.org you will see the following line:

I see.
What happens here is double export:
1. The default `org-publish-sitemap-default' first exports the sitemap
   items to Org file
2. The resulting Org file is exported to html

During the first step, @@html:...@@ is stripped because we export to
org.

You may want to use a custom :sitemap-function.

-- 
Ihor Radchenko // yantar92,
Org mode maintainer,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Some hacks on top of org-publish

[Nikolaos Chatzikonstantinou]

On Fri, Dec 27, 2024 at 12:31 PM Ihor Radchenko <yantar92@posteo.net> wrote:
>
> Nikolaos Chatzikonstantinou <nchatz314@gmail.com> writes:
>
> >> > Finally a highlight of the hacks I had to pull off: apparently
> >> > #+INCLUDE: will strip away inline backend information such as
> >> > @@html:<b>@@. I figured out that if I write something like
> >> > @@html@@a:@@:<b>@@, only the inner @@a:@@ is recognized and stripped,
> >> > which leaves me with what I originally wanted! (see
> >> > <https://github.com/createyourpersonalaccount/blog-v2/blob/main/content/publish.el#L14-L20>)
> >>
> >> May you show more concrete example?
> >> #+INCLUDE by itself does not remove export snippets.
> >
> > It must not be #+INCLUDE then, but sitemap.org generation itself, I
> > misremembered when I wrote the e-mail. Here's a reproducer, see
> > attachment. Have this index.org file in its own directory and publish
> > it. Then view public/sitemap.html. You will see that the @@BACKEND:@@
> > stuff has been stripped away in the first sitemap entry instance,
> > while in the second it is there. When inspecting the generated
> > sitemap.org you will see the following line:
>
> I see.
> What happens here is double export:
> 1. The default `org-publish-sitemap-default' first exports the sitemap
>    items to Org file
> 2. The resulting Org file is exported to html
>
> During the first step, @@html:...@@ is stripped because we export to
> org.
>
> You may want to use a custom :sitemap-function.

That makes sense. The org backend strips the HTML backend. You're
right, I should be using a custom :sitemap-function (didn't realize
there was one). Well, my hack for now remains. I might fix it later
down the road.

Regards,
Nikolaos Chatzikonstantinou

Re: Some hacks on top of org-publish

[Nikolaos Chatzikonstantinou]

On Wed, Dec 25, 2024 at 8:47 AM Ihor Radchenko <yantar92@posteo.net> wrote:
>
> Nikolaos Chatzikonstantinou <nchatz314@gmail.com> writes:
>
> > 3) Source code blocks + colors (server-side; not a .js coloring client-side)
>
> Note that ox-html can fontify src blocks by itself. Via htmlize + CSS styles.
>
> > 1) I got colors in source blocks by post-processing with node.js +
> > highlight.js the raw HTML. Is it possible to write my own backend that
> > feeds the source block into my node app?
>
> Yes, of course. See my next answers. The whole export proces is hackable
> on multiple levels.
>
> > 2) I got several issues with how the org files are exported as HTML,
> > for example I may want to insert an <article> tag, I may want a div
> > around content and a div around nav bar, etc. Not everything can be
> > solved with #+HTML: tags because I don't have precise control of where
> > those go. Can the HTML backend be edited or do I have to write my own
> > backend?
>
> Check out https://orgmode.org/manual/Advanced-Export-Configuration.html
>
> > 3) I am very inexperienced with org backends. What good resources are
> > out there for this subject?
>
> See https://orgmode.org/manual/Adding-Export-Backends.html
> Org export backends can use inheritance. So, if you need to re-define a
> part of backend (if filters are not enough), you can always create a
> "subclass" by creating a "derived" backend.

I should probably try to do this, but I got inspired by Worg's website
which does not do any backend hacking as far as I can tell, but
instead puts some javascript to collapse the table of contents. If you
check my blog now, it looks almost OK in mobile phones. I do want to
write the backend stuff, but it feels like such a chore. The
ox-latex.el (which the manual gives as an example of a not-derived
backend) is a bit overengineered for my taste, in the sense that I
need to pay a lot of attention to figure it out.

Do you have a particular backend for me to look at as a recommendation
of an easy source? I don't want it to be derived because I want to
understand the whole process.

Regards,
Nikolaos Chatzikonstantinou

Re: Some hacks on top of org-publish

[Ihor Radchenko]

Nikolaos Chatzikonstantinou <nchatz314@gmail.com> writes:

> ... I got inspired by Worg's website
> which does not do any backend hacking as far as I can tell, but
> instead puts some javascript to collapse the table of contents.

WORG does not use JS. The TOC is implemented using pure CSS.

> Do you have a particular backend for me to look at as a recommendation
> of an easy source? I don't want it to be derived because I want to
> understand the whole process.

Try ox-org.

-- 
Ihor Radchenko // yantar92,
Org mode maintainer,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Some hacks on top of org-publish

[Max Nikulin]

On 29/12/2024 11:26, Nikolaos Chatzikonstantinou wrote:
> I should probably try to do this, but I got inspired by Worg's website
> which does not do any backend hacking as far as I can tell, but
> instead puts some javascript to collapse the table of contents.

Not all users are happy with current CSS with :hover to show the table 
of contents on Worg:

Samuel Wales. [accessibility] worg obscures text. Wed, 8 Jun 2022 
14:51:55 -0700.
<https://list.orgmode.org/CAJcAo8uTOpEazLNCr0t1kFqPGTLz=G=aDnKLhuo1-Ura8-m7-Q@mail.gmail.com>

There is a trick with a hidden checkbox to implement menu without 
JavaScript. See languages selector in Wikipedia
<https://en.wikipedia.org/wiki/Org_mode>
and the main Org site top menu when accessed from a mobile device
<https://orgmode.org>

Perhaps a similar effect may be achieved with <details>/<summary>.

> If you check my blog now, it looks almost OK in mobile phones.

In desktop version of Firefox, when responsive design mode is enabled, 
the sidebar is rendered over main text (at least with active NoScript).

Re: Some hacks on top of org-publish

[Nikolaos Chatzikonstantinou]

On Sat, Jan 4, 2025 at 10:39 AM Max Nikulin <manikulin@gmail.com> wrote:
>
> On 29/12/2024 11:26, Nikolaos Chatzikonstantinou wrote:
> > I should probably try to do this, but I got inspired by Worg's website
> > which does not do any backend hacking as far as I can tell, but
> > instead puts some javascript to collapse the table of contents.
>
> Not all users are happy with current CSS with :hover to show the table
> of contents on Worg:
>
> Samuel Wales. [accessibility] worg obscures text. Wed, 8 Jun 2022
> 14:51:55 -0700.
> <https://list.orgmode.org/CAJcAo8uTOpEazLNCr0t1kFqPGTLz=G=aDnKLhuo1-Ura8-m7-Q@mail.gmail.com>
>
> There is a trick with a hidden checkbox to implement menu without
> JavaScript. See languages selector in Wikipedia
> <https://en.wikipedia.org/wiki/Org_mode>
> and the main Org site top menu when accessed from a mobile device
> <https://orgmode.org>
>
> Perhaps a similar effect may be achieved with <details>/<summary>.

org-publish (or is it org-html?) should just structure the HTML
document properly, as there is already an <aside> element, and even if
those semantic elements are not used, at least if the <div>'s of the
various parts of the page are placed sequentially one should be able
to deal with the rest. As it stands now the div is inside the article
content and it's hard to position it right. There's an urge to send
the entire org file contents inside the main content of the HTML page,
but then it makes it difficult to incorporate the sitemap with
#+INCLUDE.

Proper placement of elements would solve the issue addressed in my
e-mail and in your link as well. Then the user can just use flexbox to
get everything to be responsive.

> > If you check my blog now, it looks almost OK in mobile phones.
>
> In desktop version of Firefox, when responsive design mode is enabled,
> the sidebar is rendered over main text (at least with active NoScript).

The sidebar's default property is to be unexpanded, accounting for
those without javascript enabled. If it's running over the text it's
because your screen is too small. It's far from perfect, but I haven't
gotten around to hacking the org backends yet.

Regards,
Nikolaos Chatzikonstantinou