[PATCH] ob-ditaa.el: custom var name, ditaa executable, SVG output, and chararacter encoding

[PATCH] ob-ditaa.el: custom var name, ditaa executable, SVG output, and chararacter encoding

[Jarmo Hurri]

[-- Attachment #1: Type: text/plain, Size: 265 bytes --]


Greetings.

Please find attached a patch written mainly to allow a ditaa executable
to be used instead of a JAR file. Assuming that this patch is
(eventually) accepted, I can also volunteer to be a maintainer for this
file if one is needed.

All the best,

Jarmo


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: ditaa exec patch --]
[-- Type: text/x-patch, Size: 14233 bytes --]

From 090beafd82518380aa59eecf0e35110566b26d41 Mon Sep 17 00:00:00 2001
From: Jarmo Hurri <jarmo.hurri@iki.fi>
Date: Sun, 3 Nov 2024 15:22:33 +0200
Subject: [PATCH] ob-ditaa.el: custom var name, ditaa executable, SVG output,
 and chararacter encoding

* lisp/ob-ditaa.el (org-babel-default-header-args:ditaa) Include new header arg `:encoding`.
(org-ditaa-default-exec-mode): Define new customizable variable for controlling ditaa execution via jar or executable.
(org-ditaa-exec): Define new customizable variable for controlling path to ditaa executable.
(org-ditaa-java-exec): Rename old customizable variable to conform to ditaa variable naming (not containing word `babel`).
(org-ditaa-ensure-jar-file): Write a small helper function checking existence of jar file.
(org-babel-execute:ditaa): Add support for executable. Add support for SVG output. Automatically deduce output type if possible. Override type with header argument. Set default type to PNG. Check that no more than one result type is specified. Echo shell commands in message buffer. Clarify code structure and local variable naming.
* doc/org-manual.org (List of contributors): Remove reference to non-existing location of ditaa.jar in org contrib, refer to ditaa github page instead.
* etc/ORG-NEWS (=ob-ditaa=: custom variable name, ditaa executable, SVG output, and chararacter encoding): Document breaking change and new features.

There was a mismatch between what ob-ditaa expected and what some operating systems provide. In particular, ob-ditaa expected a JAR executable via `java -jar`, while some operating systems provide a shell script which executes the JAR in a more complicated manner. Therefore support for executing ditaa source code blocks directly via an executable was added.

Newer versions of ditaa can generate SVG output, which was not supported by ob-ditaa. This is now fixed. Output type is deduced automatically from file suffix if possible, can be overridden by header argument, and defaults to PNG.

Character encoding of ditaa source code blocks was passed via Java, while the encoding can be specified directly as a parameter to ditaa. Character encoding is now passed directly to ditaa, with a corresponding header argument `:encoding`.
---
 doc/org-manual.org |   5 +-
 etc/ORG-NEWS       |  23 ++++++
 lisp/ob-ditaa.el   | 180 +++++++++++++++++++++++++++++++++------------
 3 files changed, 157 insertions(+), 51 deletions(-)

diff --git a/doc/org-manual.org b/doc/org-manual.org
index 451fd72cd..8562fbd74 100644
--- a/doc/org-manual.org
+++ b/doc/org-manual.org
@@ -23020,8 +23020,9 @@ be complete if the ones above were not mentioned in this manual.
 - Ilya Shlyakhter proposed the Archive Sibling, line numbering in
   literal examples, and remote highlighting for referenced code lines.
 
-- Stathis Sideris wrote the =ditaa.jar= ASCII to PNG converter that is
-  now packaged into the [[https://git.sr.ht/~bzg/org-contrib][org-contrib]] repository.
+- Stathis Sideris wrote the =ditaa.jar= ASCII to PNG/SVG converter
+  that is available as a package in some operating systems or can be
+  downloaded from [[https://github.com/stathissideris/ditaa]].
 
 - Daniel Sinder came up with the idea of internal archiving by locking
   subtrees.
diff --git a/etc/ORG-NEWS b/etc/ORG-NEWS
index de4f11b25..8a19f3fdf 100644
--- a/etc/ORG-NEWS
+++ b/etc/ORG-NEWS
@@ -71,6 +71,29 @@ This should have minimal impact on non-iCalendar exporters, since
 users who manually set ~org-export-with-timestamps~ to ~active~ will
 now have diary timestamps included as well.
 
+*** =ob-ditaa=: custom variable name, ditaa executable, SVG output, and chararacter encoding
+
+To align with other customizable variable names, which do not contain
+the word =babel=, variable =org-babel-ditaa-java-cmd= has been renamed
+to =org-ditaa-java-exec=.
+
+In order to use an executable instead of a JAR file, you can set
+=org-ditaa-default-exec-mode= to ='ditaa=.  The location of the
+executable can be configured via =org-ditaa-exec=.
+
+SVG output can be generated by using =:svg= header argument, or by
+specifying a result file with =.svg= suffix; note, however, that this
+requires a ditaa version of at least 0.11.0.  Result file type is
+deduced from result file suffix, can be overruled by a header
+argument, and is PNG by default.  An error is signalled if more than
+one output type is specified via header arguments.
+
+Character encoding is passed directly to ditaa (earlier to Java) and
+can be controlled by new header argument =:encoding=.
+
+Shell commands run when executing a ditaa code block are echoed in
+the message buffer.
+
 ** New features
 
 # We list the most important features, and the features that may
diff --git a/lisp/ob-ditaa.el b/lisp/ob-ditaa.el
index cc01d7e28..8931e9a6b 100644
--- a/lisp/ob-ditaa.el
+++ b/lisp/ob-ditaa.el
@@ -2,7 +2,7 @@
 
 ;; Copyright (C) 2009-2024 Free Software Foundation, Inc.
 
-;; Author: Eric Schulte
+;; Authors: Eric Schulte, Jarmo Hurri
 ;; Keywords: literate programming, reproducible research
 ;; URL: https://orgmode.org
 
@@ -25,15 +25,62 @@
 
 ;; Org-Babel support for evaluating ditaa source code.
 ;;
-;; This differs from most standard languages in that
+;; Source code blocks of type ditaa have some special features:
 ;;
-;; 1) there is no such thing as a "session" in ditaa
+;; - there is no such thing as a "session"
 ;;
-;; 2) we are generally only going to return results of type "file"
+;; - there is no "output", so source code blocks always export "results"
 ;;
-;; 3) we are adding the "file" and "cmdline" header arguments
+;; - only results of type "file" are returned
 ;;
-;; 4) there are no variables (at least for now)
+;; - there are no variables
+;;
+;; - output file type is deduced from result file suffix by default,
+;;   but can be overwritten; if no type is specified and none can be
+;;   deduced, the default is png
+;;
+;; - three different variants of "ditaa" exist: a ditaa executable
+;;   (shell script), ditaa.jar Java archive and DitaaEPS.jar Java
+;;   archive; the third one is a fork generating eps output, and is
+;;   also a prerequisite for producing pdf output; ob-ditaa supports
+;;   all three of these; if ditaa.jar or DitaaEPS.jar is used, paths
+;;   to file(s) must be set; the following table summarizes which
+;;   variant is used in which case; column mode refers to
+;;   `org-ditaa-default-exec-mode'
+;;
+;;   | mode           | output   | command                                             |
+;;   |----------------+----------+-----------------------------------------------------|
+;;   | `ditaa'        | png, svg | `org-ditaa-exec'                                    |
+;;   | `jar'          | png, svg | `org-ditaa-java-exec' -jar `org-ditaa-jar-path'     |
+;;   | `ditaa', `jar' | eps      | `org-ditaa-java-exec' -jar `org-ditaa-eps-jar-path' |
+;;   | `ditaa', `jar' | pdf      | `org-ditaa-java-exec' -jar `org-ditaa-eps-jar-path' |
+;;
+;; - the following header arguments are added:
+;;   "file" : path to generated result file
+;;   "png" : toggle for generating png file (default)
+;;   "svg" : toggle for generating svg file (requires ditaa version >= 0.11.0)
+;;   "eps" : toggle for generating eps file (requires org-ditaa-eps-jar-path)
+;;   "pdf" : toggle for generating pdf file (requires org-ditaa-eps-jar-path and epstopdf)
+;;   "cmdline" : command line parameters passed to ditaa
+;;   "encoding" : character encoding (default UTF-8)
+;;   "java" : additional parameters passed to java if ditaa run via a jar
+;;
+
+;;; Requirements:
+
+;; at least one of the following:
+;;
+;; ditaa (executable)
+;; - packaged in some distributions
+;; - configurable via `org-ditaa-exec'
+;;
+;; ditaa.jar | when exec mode is `jar'
+;; - `org-ditaa-jar-path' must point to this jar file
+;; - see https://github.com/stathissideris/ditaa
+;;
+;; DitaaEps.jar | when generating eps or pdf output
+;; - `org-ditaa-eps-jar-path' must point to this jar file
+;; - see https://sourceforge.net/projects/ditaa-addons/files/DitaaEps/
 
 ;;; Code:
 
@@ -46,9 +93,33 @@
 (defvar org-babel-default-header-args:ditaa
   '((:results . "file")
     (:exports . "results")
-    (:java . "-Dfile.encoding=UTF-8"))
+    (:encoding . "UTF-8"))
   "Default arguments for evaluating a ditaa source block.")
 
+(defcustom org-ditaa-default-exec-mode 'jar
+  "Method to use for ditaa diagram generation when generating png or svg output.
+`jar' means to use java together with a JAR.
+The JAR must be set via `org-ditaa-jar-path'.
+
+`ditaa' means to use the ditaa executable.
+The executable can be configured via `org-ditaa-exec'."
+
+  :group 'org-babel
+  :package-version '(Org . "9.8")
+  :type 'symbol
+  :options '(ditaa jar))
+
+(defcustom org-ditaa-exec "ditaa"
+  "File name of the ditaa executable."
+  :group 'org-babel
+  :package-version '(Org . "9.8")
+  :type 'string)
+
+(defcustom org-ditaa-java-exec "java"
+  "Java executable to use when evaluating ditaa blocks using a JAR."
+  :group 'org-babel
+  :type 'string)
+
 (defcustom org-ditaa-jar-path (expand-file-name
 			       "ditaa.jar"
 			       (file-name-as-directory
@@ -58,64 +129,75 @@
 				  (expand-file-name
 				   "../contrib"
 				   (file-name-directory (org-find-library-dir "org")))))))
-  "Path to the ditaa jar executable."
-  :group 'org-babel
-  :type 'string)
-
-(defcustom org-babel-ditaa-java-cmd "java"
-  "Java executable to use when evaluating ditaa blocks."
+  "Path to the ditaa.jar file."
   :group 'org-babel
   :type 'string)
 
 (defcustom org-ditaa-eps-jar-path
   (expand-file-name "DitaaEps.jar" (file-name-directory org-ditaa-jar-path))
-  "Path to the DitaaEps.jar executable."
+  "Path to the DitaaEps.jar executable.
+Used when generating eps or pdf output."
   :group 'org-babel
   :version "24.4"
   :package-version '(Org . "8.0")
   :type 'string)
 
-(defcustom org-ditaa-jar-option "-jar"
-  "Option for the ditaa jar file.
-Do not leave leading or trailing spaces in this string."
-  :group 'org-babel
-  :version "24.1"
-  :type 'string)
+;;; small helper function returning file if it exists and signalling
+;;; error otherwise
+(defun org-ditaa-ensure-jar-file (file)
+  (if (file-exists-p file)
+      file
+    (error "could not find jar file %s" file)))
 
 (defun org-babel-execute:ditaa (body params)
-  "Execute BODY of Ditaa code with org-babel according to PARAMS.
+  "Execute BODY of ditaa code with org-babel according to PARAMS.
 This function is called by `org-babel-execute-src-block'."
   (let* ((out-file (or (cdr (assq :file params))
 		       (error
-			"Ditaa code block requires :file header argument")))
-	 (cmdline (cdr (assq :cmdline params)))
-	 (java (cdr (assq :java params)))
-	 (in-file (org-babel-temp-file "ditaa-"))
+			"ditaa code block requires :file header argument")))
+	 (png (cdr (assq :png params)))
+	 (svg (cdr (assq :svg params)))
 	 (eps (cdr (assq :eps params)))
-	 (eps-file (when eps
-		     (org-babel-process-file-name (concat in-file ".eps"))))
-	 (pdf-cmd (when (and (or (string= (file-name-extension out-file) "pdf")
-				 (cdr (assq :pdf params))))
-		    (concat
-		     "epstopdf"
-		     " " eps-file
-		     " -o=" (org-babel-process-file-name out-file))))
-	 (cmd (concat org-babel-ditaa-java-cmd
-		      " " java " " org-ditaa-jar-option " "
-		      (shell-quote-argument
-		       (expand-file-name
-			(if eps org-ditaa-eps-jar-path org-ditaa-jar-path)))
-		      " " cmdline
-		      " " (org-babel-process-file-name in-file)
-		      " " (if pdf-cmd
-			      eps-file
-			    (org-babel-process-file-name out-file)))))
-    (unless (file-exists-p org-ditaa-jar-path)
-      (error "Could not find ditaa.jar at %s" org-ditaa-jar-path))
-    (with-temp-file in-file (insert body))
-    (shell-command cmd)
-    (when pdf-cmd (shell-command pdf-cmd))
-    nil)) ;; signal that output has already been written to file
+	 (pdf (cdr (assq :pdf params)))
+         (num-out-types-set (seq-count 'identity (list png svg eps pdf))))
+    (when (> num-out-types-set 1)
+      (error "more than one output type specified for ditaa code block"))
+    (when (= num-out-types-set 0)
+      ;; deduce outfile type from suffix or set to default
+      (let ((out-file-suffix (file-name-extension out-file)))
+        (setq svg (string= out-file-suffix "svg"))
+        (setq eps (string= out-file-suffix "eps"))
+        (setq pdf (string= out-file-suffix "pdf"))
+        (setq png (or (string= out-file-suffix "png")
+                      (not svg eps pdf)))))
+    (let* ((ditaa-options (cdr (assq :cmdline params)))
+           (ditaa-encoding (cdr (assq :encoding params)))
+	   (java-options (cdr (assq :java params)))
+           (use-eps-jar (or eps pdf))
+           (exec-form (if (or (equal org-ditaa-default-exec-mode 'jar) use-eps-jar)
+                          (concat org-ditaa-java-exec
+                                  (when java-options (concat " " java-options))
+                                  " " "-jar" " "
+                                  (shell-quote-argument
+                                   (org-ditaa-ensure-jar-file (if use-eps-jar org-ditaa-eps-jar-path
+                                                                org-ditaa-jar-path))))
+                        org-ditaa-exec))
+	   (in-file (org-babel-temp-file "ditaa-"))
+           (ditaa-out-file (org-babel-process-file-name (if pdf (concat in-file ".eps") out-file)))
+	   (cmd (concat exec-form
+                        (when ditaa-options (concat " " ditaa-options))
+                        (when svg (concat " " "--svg"))
+                        (when ditaa-encoding (concat " " "-e " ditaa-encoding))
+                        " " in-file " " ditaa-out-file)))
+      (with-temp-file in-file (insert body))
+      (message cmd)
+      (shell-command cmd)
+      (when pdf
+	(let ((pdf-cmd (concat "epstopdf" " " ditaa-out-file " "
+		              "-o=" (org-babel-process-file-name out-file))))
+          (message pdf-cmd)
+          (shell-command pdf-cmd)))
+      nil))) ;; signal that output has already been written to file
 
 (defun org-babel-prep-session:ditaa (_session _params)
   "Return an error because ditaa does not support sessions."
-- 
2.47.0

Re: [PATCH] ob-ditaa.el: custom var name, ditaa executable, SVG output, and chararacter encoding

[Ihor Radchenko]

Jarmo Hurri <jarmo.hurri@iki.fi> writes:

> Please find attached a patch written mainly to allow a ditaa executable
> to be used instead of a JAR file. Assuming that this patch is
> (eventually) accepted, I can also volunteer to be a maintainer for this
> file if one is needed.

Thanks for the patch and for volunteering to be a maintainer!

Note that we will also need to update
https://orgmode.org/worg/org-contrib/babel/languages/ob-doc-ditaa.html
after we finalize changes in the code.

See some initial comments below.

> -(defcustom org-babel-ditaa-java-cmd "java"

> +(defcustom org-ditaa-java-exec "java"
> +  "Java executable to use when evaluating ditaa blocks using a JAR."
> +  :group 'org-babel
> +  :type 'string)

We generally do not rename variables irreversibly.
Please leave an obsolete alias for `org-babel-ditaa-java-cmd' pointing
to the new variable name. Otherwise, the existing configs that were
using the old variable name will be broken.

> +;;; small helper function returning file if it exists and signalling
> +;;; error otherwise
> +(defun org-ditaa-ensure-jar-file (file)
> +  (if (file-exists-p file)
> +      file
> +    (error "could not find jar file %s" file)))

Rather than writing what the function does in the comment, please do it
in the docstring. We might also make this function internal.

Also, the error sounds very generic. It would be nicer to indicate to
the user that the problem is related to ob-ditaa.
  
> +	 (png (cdr (assq :png params)))
> +	 (svg (cdr (assq :svg params)))
>  	 (eps (cdr (assq :eps params)))

I am wondering if we could instead deprecate the :png/:eps parameters
and instead use the :file extension to decide.

> +      (message cmd)
> +      (shell-command cmd)
> +      (when pdf
> +	(let ((pdf-cmd (concat "epstopdf" " " ditaa-out-file " "
> +		              "-o=" (org-babel-process-file-name out-file))))
> +          (message pdf-cmd)

Why message?

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [PATCH] ob-ditaa.el: custom var name, ditaa executable, SVG output, and chararacter encoding

[Jarmo Hurri]

Greetings Ihor.

Thanks for your feedback. A couple of notes and questions before I can
proceed to format the next version of the patch.

Ihor Radchenko <yantar92@posteo.net> writes:

> Note that we will also need to update
> https://orgmode.org/worg/org-contrib/babel/languages/ob-doc-ditaa.html
> after we finalize changes in the code.

Noted.

>> -(defcustom org-babel-ditaa-java-cmd "java"
>
>> +(defcustom org-ditaa-java-exec "java"
>> +  "Java executable to use when evaluating ditaa blocks using a JAR."
>> +  :group 'org-babel
>> +  :type 'string)
>
> We generally do not rename variables irreversibly.  Please leave an
> obsolete alias for `org-babel-ditaa-java-cmd' pointing to the new
> variable name. Otherwise, the existing configs that were using the old
> variable name will be broken.

Will do so. This will also move the contents in ORG-NEWS to a different
section, since there will no longer be any "breaking changes."

>> +;;; small helper function returning file if it exists and signalling
>> +;;; error otherwise
>> +(defun org-ditaa-ensure-jar-file (file)
>> +  (if (file-exists-p file)
>> +      file
>> +    (error "could not find jar file %s" file)))
>
> Rather than writing what the function does in the comment, please do
> it in the docstring. We might also make this function internal.

Check.

> Also, the error sounds very generic. It would be nicer to indicate to
> the user that the problem is related to ob-ditaa.

Check.

>> +	 (png (cdr (assq :png params)))
>> +	 (svg (cdr (assq :svg params)))
>>  	 (eps (cdr (assq :eps params)))
>
> I am wondering if we could instead deprecate the :png/:eps parameters
> and instead use the :file extension to decide.

This could be done, but I do not see much harm in providing an
override. Note that the file extension is used by default.

So, your choice: is it
a) file extension only
b) file extension with possibility to override with parameters?

>> +      (message cmd)
>> +      (shell-command cmd)
>> +      (when pdf
>> +	(let ((pdf-cmd (concat "epstopdf" " " ditaa-out-file " "
>> +		              "-o=" (org-babel-process-file-name out-file))))
>> +          (message pdf-cmd)
>
> Why message?

I was originally directed to ob-plantuml, which message's its
command. During the testing of this patch I found messaging useful to
observe what was happening.

So, your choice:
a) no messaging
b) message always
c) defcustom a toggle for messaging?

And, finally, should I add myself as the maintainer?

All the best,

Jarmo

Re: [PATCH] ob-ditaa.el: custom var name, ditaa executable, SVG output, and chararacter encoding

[Ihor Radchenko]

Jarmo Hurri <jarmo.hurri@iki.fi> writes:

>>> +	 (png (cdr (assq :png params)))
>>> +	 (svg (cdr (assq :svg params)))
>>>  	 (eps (cdr (assq :eps params)))
>>
>> I am wondering if we could instead deprecate the :png/:eps parameters
>> and instead use the :file extension to decide.
>
> This could be done, but I do not see much harm in providing an
> override. Note that the file extension is used by default.
>
> So, your choice: is it
> a) file extension only
> b) file extension with possibility to override with parameters?

AFAIU, the current situation is not ideal: if we have something like
#+begin_src ditaa :pdf :file foo.eps, the results will be very strange.

Moreover, we have API function `org-babel-graphical-output-file' (not
used by ob-ditaa) that makes use of :file-ext standard header argument
that is serving similar purpose with :eps/:pdf in ob-ditaa (see 16.5
Evaluating Code Blocks section of the manual).

So, I view these :ext options that are still used by several babel
backends as candidates for deprecation. Certainly, I do not want to make
them proliferate in the new features.

>>> +      (message cmd)
>>> +      (shell-command cmd)
>>> +      (when pdf
>>> +	(let ((pdf-cmd (concat "epstopdf" " " ditaa-out-file " "
>>> +		              "-o=" (org-babel-process-file-name out-file))))
>>> +          (message pdf-cmd)
>>
>> Why message?
>
> I was originally directed to ob-plantuml, which message's its
> command. During the testing of this patch I found messaging useful to
> observe what was happening.
>
> So, your choice:
> a) no messaging
> b) message always
> c) defcustom a toggle for messaging?

I'd prefer no messaging as it serves no clear purpose to the user.
For debugging purposes, one can always do M-x trace-function shell-command

> And, finally, should I add myself as the maintainer?

That would be great. Thanks!
I assume that you already have access to savannah as you are
listed as the maintainer of ob-processing.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [PATCH] ob-ditaa.el: custom var name, ditaa executable, SVG output, and chararacter encoding

[Ihor Radchenko]

Hi,

Just a quick followup since this thread got no new replies within a
month.

Do you need any extra help with your patch?

-- 
Ihor Radchenko // yantar92,
Org mode maintainer,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>