no syntax highlighting for code blocks with org-publish

no syntax highlighting for code blocks with org-publish

[M. Pger]

[-- Attachment #1: Type: text/plain, Size: 452 bytes --]

Dear All,

While syntax highlighting for code blocks is correctly implemented when I export a Org document with M-x org-html-export-to-html​, it does not work when the same document is exported as part of an org-publish project defined using org-publish-project-alist​.

Org-version: 9.5.4 (also tried with 9.5.2)
Htmlize-version: 1.57

Note that including :htmlized-source t​ does not solve the problem.

Any idea of what's happening?

Best,

MP

[-- Attachment #2: Type: text/html, Size: 1635 bytes --]

Re: no syntax highlighting for code blocks with org-publish

[Ihor Radchenko]

"M. Pger" <mpger@protonmail.com> writes:

> While syntax highlighting for code blocks is correctly implemented when I export a Org document with M-x org-html-export-to-html​, it does not work when the same document is exported as part of an org-publish project defined using org-publish-project-alist​.
>
> Org-version: 9.5.4 (also tried with 9.5.2)
> Htmlize-version: 1.57
>
> Note that including :htmlized-source t​ does not solve the problem.
>
> Any idea of what's happening?

May you provide an example reproducer?

Best,
Ihor

Re: no syntax highlighting for code blocks with org-publish

[M. Pger]

Thank you for your answer. Here it is:

1. Create the following directory structure (3 directories):
~/test/
├── content
├── html
└── .packages

2. Create the .el script to build the website (=~/test/build.el=):

#+begin_src elisp
  ;; * Set the package installation directory (in order not to overwrite the standard ~/emacs.d)
  (require 'package)
  (setq package-user-dir (expand-file-name "./.packages"))
  (setq package-archives '(("melpa" . "https://melpa.org/packages/")))
  (add-to-list 'package-archives '("elpa" . "https://elpa.gnu.org/packages/"))

  ;; * Initialize the package system
  (package-initialize)
  (unless package-archive-contents
    (package-refresh-contents))

  ;; * Install dependencies
  ;; ** since org is builtin, by default Emacs does not try to install the latest version from Elpa (9.5.4)
  ;; the following solves the issue:
  (defun mpger-ignore-builtin (pkg)
    (assq-delete-all pkg package--builtins)
    (assq-delete-all pkg package--builtin-versions))
  (mpger-ignore-builtin 'org)
  ;; ** install packages:
  (package-install 'org)
  (package-install 'htmlize)

  ;; * Load the publishing system:
  (require 'org)
  (require 'htmlize)
  (require 'ox-publish)

  ;; * Define the project
  (setq org-publish-project-alist
	(list
	 (list "pages"
	       :recursive t
	       :htmlized-source t
	       :base-directory "./content/"
	       :base-extension "org"
	       :publishing-directory "./html/"
	       :publishing-function 'org-html-publish-to-html
	       :with-creator t
	       :with-toc t
	       :section-numbers nil
	       :time-stamp-file nil)
	 ))

  ;; * Generate the site output
  (org-publish-all t)

  (message "Done!")
#+end_src

3. In =~/test/content/=, create a simple test.org file to be published as html (=~/test/content/test.org=):

#+begin_example
* Here's some text

Lorem ipsum.

* Here's some code

#+begin_src R :results output :exports both
  df <- mtcars ## a comment
  library(parallel)
#+end_src
#+end_example

4. Run =~/test/build.el= (e.g. with ~emacs -Q --script ~/test/build.el~) and compare with the output from ~C-c C-e h o~. The latter has syntax highlighting, the former has not.

Best,

M





------- Original Message -------
On Monday, July 18th, 2022 at 1:42 AM, Ihor Radchenko <yantar92@gmail.com> wrote:


> "M. Pger" mpger@protonmail.com writes:
>
> > While syntax highlighting for code blocks is correctly implemented when I export a Org document with M-x org-html-export-to-html, it does not work when the same document is exported as part of an org-publish project defined using org-publish-project-alist.
> >
> > Org-version: 9.5.4 (also tried with 9.5.2)
> > Htmlize-version: 1.57
> >
> > Note that including :htmlized-source t does not solve the problem.
> >
> > Any idea of what's happening?
>
>
> May you provide an example reproducer?
>
> Best,
> Ihor

Re: no syntax highlighting for code blocks with org-publish

[Tim Cross]

"M. Pger" <mpger@protonmail.com> writes:

> Thank you for your answer. Here it is:
>
> 1. Create the following directory structure (3 directories):
> ~/test/
> ├── content
> ├── html
> └── .packages
>
> 2. Create the .el script to build the website (=~/test/build.el=):
>
> #+begin_src elisp
>   ;; * Set the package installation directory (in order not to overwrite the standard ~/emacs.d)
>   (require 'package)
>   (setq package-user-dir (expand-file-name "./.packages"))
>   (setq package-archives '(("melpa" . "https://melpa.org/packages/")))
>   (add-to-list 'package-archives '("elpa" . "https://elpa.gnu.org/packages/"))
>
>   ;; * Initialize the package system
>   (package-initialize)
>   (unless package-archive-contents
>     (package-refresh-contents))
>
>   ;; * Install dependencies
>   ;; ** since org is builtin, by default Emacs does not try to install the latest version from Elpa (9.5.4)
>   ;; the following solves the issue:
>   (defun mpger-ignore-builtin (pkg)
>     (assq-delete-all pkg package--builtins)
>     (assq-delete-all pkg package--builtin-versions))
>   (mpger-ignore-builtin 'org)
>   ;; ** install packages:
>   (package-install 'org)
>   (package-install 'htmlize)
>
>   ;; * Load the publishing system:
>   (require 'org)
>   (require 'htmlize)
>   (require 'ox-publish)
>
>   ;; * Define the project
>   (setq org-publish-project-alist
> 	(list
> 	 (list "pages"
> 	       :recursive t
> 	       :htmlized-source t
> 	       :base-directory "./content/"
> 	       :base-extension "org"
> 	       :publishing-directory "./html/"
> 	       :publishing-function 'org-html-publish-to-html
> 	       :with-creator t
> 	       :with-toc t
> 	       :section-numbers nil
> 	       :time-stamp-file nil)
> 	 ))
>
>   ;; * Generate the site output
>   (org-publish-all t)
>
>   (message "Done!")
> #+end_src
>
>
> 3. In =~/test/content/=, create a simple test.org file to be published as html (=~/test/content/test.org=):
>
> #+begin_example
> * Here's some text
>
> Lorem ipsum.
>
> * Here's some code
>
> #+begin_src R :results output :exports both
>   df <- mtcars ## a comment
>   library(parallel)
> #+end_src
>
> #+end_example
>
> 4. Run =~/test/build.el= (e.g. with ~emacs -Q --script ~/test/build.el~) and compare with
> the output from ~C-c C-e h o~. The latter has syntax highlighting, the former has not.
>

Just a shot in the dark. Does syntax highlighting work if the source
block is something like emacs-lisp rather than R?

I know that syntax highlighting is based on the syntax highlighting from
the mode used for a specific language. I'm wondering if your not getting
syntax highlighting because in yhour publish script, R mode is not
loaded, but when you open the org file and do a 'normal' export, R is
loaded and so you get syntax highlighting. Therefore, I would try the
same experiment, but instead of a source block of R code, I would try a
source block of emacs-lisp code as we know that emacs-lisp mode will be
loaded.

Could be completely off track though!

Re: no syntax highlighting for code blocks with org-publish

[M. Pger]

Thanks for your suggestion. I added the following:

#+begin_src elisp :eval no :exports code
  (setq my-var "org mailing list")
  (message "Hello, %s" my-var)
#+end_src

When exported with ~C-c C-e h o~, syntax highlighting is implemented (with colors). When exported with org-publish interestingly I have no color, but =setq= is in bold. Would it be possible that ox-publish implements some kind of really basic builtin syntax highlighting and ignores htmlize?


------- Original Message -------
On Monday, July 18th, 2022 at 11:50 PM, Tim Cross <theophilusx@gmail.com> wrote:


> "M. Pger" mpger@protonmail.com writes:
>
> > Thank you for your answer. Here it is:
> >
> > 1. Create the following directory structure (3 directories):
> > ~/test/
> > ├── content
> > ├── html
> > └── .packages
> >
> > 2. Create the .el script to build the website (=~/test/build.el=):
> >
> > #+begin_src elisp
> > ;; * Set the package installation directory (in order not to overwrite the standard ~/emacs.d)
> > (require 'package)
> > (setq package-user-dir (expand-file-name "./.packages"))
> > (setq package-archives '(("melpa" . "https://melpa.org/packages/")))
> > (add-to-list 'package-archives '("elpa" . "https://elpa.gnu.org/packages/"))
> >
> > ;; * Initialize the package system
> > (package-initialize)
> > (unless package-archive-contents
> > (package-refresh-contents))
> >
> > ;; * Install dependencies
> > ;; ** since org is builtin, by default Emacs does not try to install the latest version from Elpa (9.5.4)
> > ;; the following solves the issue:
> > (defun mpger-ignore-builtin (pkg)
> > (assq-delete-all pkg package--builtins)
> > (assq-delete-all pkg package--builtin-versions))
> > (mpger-ignore-builtin 'org)
> > ;; ** install packages:
> > (package-install 'org)
> > (package-install 'htmlize)
> >
> > ;; * Load the publishing system:
> > (require 'org)
> > (require 'htmlize)
> > (require 'ox-publish)
> >
> > ;; * Define the project
> > (setq org-publish-project-alist
> > (list
> > (list "pages"
> > :recursive t
> > :htmlized-source t
> > :base-directory "./content/"
> > :base-extension "org"
> > :publishing-directory "./html/"
> > :publishing-function 'org-html-publish-to-html
> > :with-creator t
> > :with-toc t
> > :section-numbers nil
> > :time-stamp-file nil)
> > ))
> >
> > ;; * Generate the site output
> > (org-publish-all t)
> >
> > (message "Done!")
> > #+end_src
> >
> > 3. In =~/test/content/=, create a simple test.org file to be published as html (=~/test/content/test.org=):
> >
> > #+begin_example
> > * Here's some text
> >
> > Lorem ipsum.
> >
> > * Here's some code
> >
> > #+begin_src R :results output :exports both
> > df <- mtcars ## a comment
> > library(parallel)
> > #+end_src
> >
> > #+end_example
> >
> > 4. Run =~/test/build.el= (e.g. with ~emacs -Q --script ~/test/build.el~) and compare with
> > the output from ~C-c C-e h o~. The latter has syntax highlighting, the former has not.
>
>
> Just a shot in the dark. Does syntax highlighting work if the source
> block is something like emacs-lisp rather than R?
>
> I know that syntax highlighting is based on the syntax highlighting from
> the mode used for a specific language. I'm wondering if your not getting
> syntax highlighting because in yhour publish script, R mode is not
> loaded, but when you open the org file and do a 'normal' export, R is
> loaded and so you get syntax highlighting. Therefore, I would try the
> same experiment, but instead of a source block of R code, I would try a
> source block of emacs-lisp code as we know that emacs-lisp mode will be
> loaded.
>
> Could be completely off track though!

Re: no syntax highlighting for code blocks with org-publish

[Tim Cross]

"M. Pger" <mpger@protonmail.com> writes:

> Thanks for your suggestion. I added the following:
>
> #+begin_src elisp :eval no :exports code
>   (setq my-var "org mailing list")
>   (message "Hello, %s" my-var)
> #+end_src
>
> When exported with ~C-c C-e h o~, syntax highlighting is implemented (with colors). When
> exported with org-publish interestingly I have no color, but =setq= is in bold. Would it
> be possible that ox-publish implements some kind of really basic builtin syntax
> highlighting and ignores htmlize?
>
>

I tried both exporting an org file into html and publishing and didn't
get any syntax highlighting for either case. Had a closer look and
noticed it didn't look like htmlize was being loaded. Did a (require
'htmlize) and did both an export and publish, betting syntax
highlighting for both.

My suspicion is that for the publish option, htmlize wasn't loaded?
Maybe worth doing an explicit require and then call org-publish directly
and see if that makes any difference.

Below is the basic publish alist setting I used. Doubt all the slots are
relevant - it was just a snippet from another project I grabbed to get
setup. 

I also run M-x org-publish <ret> orgfiles <ret> rather than the export menu.

(setq org-publish-project-alist
      '(("orgfiles"
         :base-directory "~/playground/org/"
         :base-extension "org"
         :recursive t
         :publishing-directory "~/Public"
         :publishing-function org-html-publish-to-html
         :exclude "PrivatePage.org"
         :html-style-default ""
         :html-scripts ""
         :html-htmlize-output-type 'inline-css
         :html-doctype "html5"
         :html-html5-fancy t
         :html-validation-link nil
         ))) 

Re: no syntax highlighting for code blocks with org-publish

[M. Pger]

I tried to publish using `M-x org-publish RET <name_of_project> RET` as you suggested, and syntax highlighting was actually implemented, thanks.

Actually the problem is coming from the fact that I use `emacs -Q --script build.el` (as a bash script) to publish the project. According to https://list.orgmode.org/AE5693F1-F63D-4383-8840-0FD2DBAAC5D6@gmail.com/ (rather old, but apparently this has not changed), font-lock-mode is not enabled by default in --batch mode. Since `--script` "run Emacs in batch mode, like ‘--batch’, and then read and execute the Lisp code in file" (cf the doc), it seems that this is exactly my problem.

A workaround is suggested in https://list.orgmode.org/AE5693F1-F63D-4383-8840-0FD2DBAAC5D6@gmail.com/, but unfortunately for me it does not work (yet). I am going to dig further into that.

Best,

M



Sent with Proton Mail secure email.

------- Original Message -------
On Tuesday, July 19th, 2022 at 9:51 AM, Tim Cross <theophilusx@gmail.com> wrote:


> "M. Pger" mpger@protonmail.com writes:
>
> > Thanks for your suggestion. I added the following:
> >
> > #+begin_src elisp :eval no :exports code
> > (setq my-var "org mailing list")
> > (message "Hello, %s" my-var)
> > #+end_src
> >
> > When exported with ~C-c C-e h o~, syntax highlighting is implemented (with colors). When
> > exported with org-publish interestingly I have no color, but =setq= is in bold. Would it
> > be possible that ox-publish implements some kind of really basic builtin syntax
> > highlighting and ignores htmlize?
>
>
> I tried both exporting an org file into html and publishing and didn't
> get any syntax highlighting for either case. Had a closer look and
> noticed it didn't look like htmlize was being loaded. Did a (require
> 'htmlize) and did both an export and publish, betting syntax
> highlighting for both.
>
> My suspicion is that for the publish option, htmlize wasn't loaded?
> Maybe worth doing an explicit require and then call org-publish directly
> and see if that makes any difference.
>
> Below is the basic publish alist setting I used. Doubt all the slots are
> relevant - it was just a snippet from another project I grabbed to get
> setup.
>
> I also run M-x org-publish <ret> orgfiles <ret> rather than the export menu.
>
>
> (setq org-publish-project-alist
> '(("orgfiles"
> :base-directory "~/playground/org/"
> :base-extension "org"
> :recursive t
> :publishing-directory "~/Public"
> :publishing-function org-html-publish-to-html
> :exclude "PrivatePage.org"
> :html-style-default ""
> :html-scripts ""
> :html-htmlize-output-type 'inline-css
> :html-doctype "html5"
> :html-html5-fancy t
> :html-validation-link nil
> )))

Re: no syntax highlighting for code blocks with org-publish

[Tim Cross]

"M. Pger" <mpger@protonmail.com> writes:

> I tried to publish using `M-x org-publish RET <name_of_project> RET` as you suggested, and
> syntax highlighting was actually implemented, thanks.
>
> Actually the problem is coming from the fact that I use `emacs -Q --script build.el` (as a
> bash script) to publish the project. According to
> https://list.orgmode.org/AE5693F1-F63D-4383-8840-0FD2DBAAC5D6@gmail.com/ (rather old, but
> apparently this has not changed), font-lock-mode is not enabled by default in --batch
> mode. Since `--script` "run Emacs in batch mode, like ‘--batch’, and then read and execute
> the Lisp code in file" (cf the doc), it seems that this is exactly my problem.
>
> A workaround is suggested in
> https://list.orgmode.org/AE5693F1-F63D-4383-8840-0FD2DBAAC5D6@gmail.com/, but
> unfortunately for me it does not work (yet). I am going to dig further into that.
>

I would be interested to hear how you go. I have a use case coming up
where I need to publish a large number of org files using a batch job,
so expect to run into the same issue. Right now, I'm still getting my
head around how to best get a consistent style when the input comes from
multiple org files from different sources and I want to minimise editing
the sources, plus provide a simple way to update/change the styling
later.

Re: no syntax highlighting for code blocks with org-publish

[M. Pger]

I solved the issue by creating the following bash script in my project folder:

#!/bin/sh
TERM=xterm export TERM
emacs -q -nw --load=build.el --eval="(eval-buffer)" --eval="(save-buffers-kill-terminal)"

This avoid using --script (and thus batch) and thus allows font-lock to be enabled when my publishing elisp script (build.el) is run. Thanks to `TERM=xterm export TERM`, this bash script can be run from the Emacs shell (otherwise problematic because of the -nw option).

To have 'extended' syntax highlighting for R, note that I have to load ESS from build.el.

Hope this will help! Thanks again for your feedback and your insights.

Best,

M

------- Original Message -------
On Wednesday, July 20th, 2022 at 12:33 AM, Tim Cross <theophilusx@gmail.com> wrote:


> "M. Pger" mpger@protonmail.com writes:
>
> > I tried to publish using `M-x org-publish RET <name_of_project> RET` as you suggested, and
> > syntax highlighting was actually implemented, thanks.
> >
> > Actually the problem is coming from the fact that I use `emacs -Q --script build.el` (as a
> > bash script) to publish the project. According to
> > https://list.orgmode.org/AE5693F1-F63D-4383-8840-0FD2DBAAC5D6@gmail.com/ (rather old, but
> > apparently this has not changed), font-lock-mode is not enabled by default in --batch
> > mode. Since `--script` "run Emacs in batch mode, like ‘--batch’, and then read and execute
> > the Lisp code in file" (cf the doc), it seems that this is exactly my problem.
> >
> > A workaround is suggested in
> > https://list.orgmode.org/AE5693F1-F63D-4383-8840-0FD2DBAAC5D6@gmail.com/, but
> > unfortunately for me it does not work (yet). I am going to dig further into that.
>
>
> I would be interested to hear how you go. I have a use case coming up
> where I need to publish a large number of org files using a batch job,
> so expect to run into the same issue. Right now, I'm still getting my
> head around how to best get a consistent style when the input comes from
> multiple org files from different sources and I want to minimise editing
> the sources, plus provide a simple way to update/change the styling
> later.

Re: no syntax highlighting for code blocks with org-publish

[M. Pger]

Actually, using -nw implies that no theme is loaded, so the face colors 'extracted' by htmlize are the flashy default colors of Emacs in terminal.

To have the colors associated with the theme I use, I had to load the theme through build.el and to change my bash script to:

#!/bin/sh
emacs -q --load=build.el --eval="(eval-buffer)" --eval="(save-buffers-kill-terminal)"

The advantage is of course to have the desired face colors; the minor disadvantage is to see an Emacs frame appearing, loading the theme and then disappearing.

Best,

M


------- Original Message -------
On Wednesday, July 20th, 2022 at 1:14 AM, M. Pger <mpger@protonmail.com> wrote:


> I solved the issue by creating the following bash script in my project folder:
>
> #!/bin/sh
> TERM=xterm export TERM
> emacs -q -nw --load=build.el --eval="(eval-buffer)" --eval="(save-buffers-kill-terminal)"
>
> This avoid using --script (and thus batch) and thus allows font-lock to be enabled when my publishing elisp script (build.el) is run. Thanks to `TERM=xterm export TERM`, this bash script can be run from the Emacs shell (otherwise problematic because of the -nw option).
>
> To have 'extended' syntax highlighting for R, note that I have to load ESS from build.el.
>
> Hope this will help! Thanks again for your feedback and your insights.
>
> Best,
>
> M
>
> ------- Original Message -------
> On Wednesday, July 20th, 2022 at 12:33 AM, Tim Cross theophilusx@gmail.com wrote:
>
>
>
> > "M. Pger" mpger@protonmail.com writes:
> >
> > > I tried to publish using `M-x org-publish RET <name_of_project> RET` as you suggested, and
> > > syntax highlighting was actually implemented, thanks.
> > >
> > > Actually the problem is coming from the fact that I use `emacs -Q --script build.el` (as a
> > > bash script) to publish the project. According to
> > > https://list.orgmode.org/AE5693F1-F63D-4383-8840-0FD2DBAAC5D6@gmail.com/ (rather old, but
> > > apparently this has not changed), font-lock-mode is not enabled by default in --batch
> > > mode. Since `--script` "run Emacs in batch mode, like ‘--batch’, and then read and execute
> > > the Lisp code in file" (cf the doc), it seems that this is exactly my problem.
> > >
> > > A workaround is suggested in
> > > https://list.orgmode.org/AE5693F1-F63D-4383-8840-0FD2DBAAC5D6@gmail.com/, but
> > > unfortunately for me it does not work (yet). I am going to dig further into that.
> >
> > I would be interested to hear how you go. I have a use case coming up
> > where I need to publish a large number of org files using a batch job,
> > so expect to run into the same issue. Right now, I'm still getting my
> > head around how to best get a consistent style when the input comes from
> > multiple org files from different sources and I want to minimise editing
> > the sources, plus provide a simple way to update/change the styling
> > later.

Re: no syntax highlighting for code blocks with org-publish

[Ihor Radchenko]

"M. Pger" <mpger@protonmail.com> writes:

> Actually, using -nw implies that no theme is loaded, so the face colors 'extracted' by htmlize are the flashy default colors of Emacs in terminal.
>
> To have the colors associated with the theme I use, I had to load the theme through build.el and to change my bash script to:
>
> #!/bin/sh
> emacs -q --load=build.el --eval="(eval-buffer)" --eval="(save-buffers-kill-terminal)"
>
> The advantage is of course to have the desired face colors; the minor disadvantage is to see an Emacs frame appearing, loading the theme and then disappearing.

You may also try --daemon. It should not create frame.

Best,
Ihor