Best way to include METAPOST in ConTeXt exporter

Best way to include METAPOST in ConTeXt exporter

[Jason Ross]

[-- Attachment #1: Type: text/plain, Size: 701 bytes --]

Hello,

I'd like to include METAPOST figures in the ConTeXt exporter backend I'm
developing. However, I don't know of an idiomatic way to add captions and
references for the figures.

Currently, I export METAPOST with `#+BEGIN_EXPORT metapost` / `#+END_EXPORT`
tags. However, this feature seems to be intended for completely "raw"
outputs
with no markup or tagging in the resulting export. I'm interested in
supporting
at least `#+NAME` and `#+CAPTION` keywords for METAPOST figures so that they
can be referred to in the Org file and also in the exported pdf.

What are some better ways of doing something like this? Source blocks? How
would
a user expect to use a feature like this?

Thanks,

Jason

[-- Attachment #2: Type: text/html, Size: 804 bytes --]

Re: Best way to include METAPOST in ConTeXt exporter

[Ihor Radchenko]

Jason Ross <jasonross1024@gmail.com> writes:

> Hello,
>
> I'd like to include METAPOST figures in the ConTeXt exporter backend I'm
> developing. However, I don't know of an idiomatic way to add captions and
> references for the figures.

You can use affiliated keywords:

(defconst org-element-affiliated-keywords
  '("CAPTION" "DATA" "HEADER" "HEADERS" "LABEL" "NAME" "PLOT" "RESNAME" "RESULT"
    "RESULTS" "SOURCE" "SRCNAME" "TBLNAME")
  "List of affiliated keywords as strings.
By default, all keywords setting attributes (e.g., \"ATTR_LATEX\")
are affiliated keywords and need not to be in this list.")

> Currently, I export METAPOST with `#+BEGIN_EXPORT metapost` / `#+END_EXPORT`
> tags. However, this feature seems to be intended for completely "raw"
> outputs
> with no markup or tagging in the resulting export. I'm interested in
> supporting
> at least `#+NAME` and `#+CAPTION` keywords for METAPOST figures so that they
> can be referred to in the Org file and also in the exported pdf.



> What are some better ways of doing something like this? Source blocks? How
> would
> a user expect to use a feature like this?


You can use a special block:
#+caption: Sample caption
#+attr_context: :field value
#+begin_metapost
...
#+end_metapost

The parsed representation of this element in exporter will be something like:

(special-block (:type "metapost" :begin 292 :end 382 :contents-begin 364 :contents-end 368 :post-blank 0 :post-affiliated 347 :caption (((#("Sample caption" 0 14 (:parent #4))))) :attr_context (":field value")))


Best,
Ihor

Re: Best way to include METAPOST in ConTeXt exporter

[Jason Ross]

I had considered using special blocks; they match my mental model the best.
However, they don't provide any support for syntax highlighting or opening
the block in a new major mode buffer. I'm not sure if it's worth giving up
language features in order to use the block that's most intuitive to users.

I'm also considering writing an Org Babel module for METAPOST. This could
allow METAPOST figures to be included with any export backend. However,
if I go this route, there are still some challenges:
1. Getting captions and tags attached to the resulting figures is clumsy.
   As far as I know,  this could be done with the :post header argument
   and a wrapper source block, but this would require boilerplate code
   in documents that use this feature
2. I don't know if there's a reasonable way to leverage ConTeXt's native support
   for METAPOST this way; if I ask Org Babel to generate an SVG, ConTeXt then
   has to parse the SVG and (internally) convert it back into METAPOST to
   render in the document. ConTeXt's SVG support (particularly with mathematical
   symbols) is missing some features, so this will be a lossy process.
   Essentially I'd want the module to return a file most of the time, but
   return raw METAPOST (wrapped in some tags) if the ConTeXt backend is used.

Here's a minimal implementation of the source block concept:

    # Define a macro to add header arguments UNLESS ConTeXt backend is used
    #+MACRO: conditional-header (eval (when (not (eq
org-export-current-backend 'context)) (concat  "#+HEADER: :results
file :file " $1)))

    # Define a wrapper block to annotate source block outputs with
    # caption and name
    #+NAME: wrap_metapost
    #+BEGIN_SRC emacs-lisp :var data="" :var caption="" :var name=""
    (concat
     (when (org-string-nw-p name) (format "#+NAME: %s\n" name))
     (when (org-string-nw-p caption) (format "#+CAPTION: %s\n" caption))
     (if (eq org-export-current-backend 'context)
         (format "#+BEGIN_METAPOST \n%s\n#+END_METAPOST" data)
       data))
    #+END_SRC

    # Minimal Org Babel implementation for METAPOST
    #+BEGIN_SRC emacs-lisp
    (defvar org-babel-default-header-args:metapost
      '((:exports . "results"))
      "Default arguments to use when evaluating a dot source block.")
    (defun org-babel-execute:metapost (body params)
      "Execute a block of METAPOST code with org-babel.
    This function is called by `org-babel-execute-src-block'."
      (if (cdr (assq :file params))
          (let* ((out-file (cdr (assq :file params)))
                 (cmdline (or (cdr (assq :cmdline params))
                              (format "-T%s" (file-name-extension out-file))))
                 (cmd (or (cdr (assq :cmd params)) "mpost"))
                 (coding-system-for-read 'utf-8) ;use utf-8 with sub-processes
                 (coding-system-for-write 'utf-8)
                 (in-file (org-babel-temp-file "metapost-")))
            (with-temp-file in-file
              (insert (org-babel-expand-body:generic body params)))
            (org-babel-eval
             (concat cmd
                     " -s 'outputformat=\"svg\"'"
                     (format " -s 'outputtemplate=\"%s\"'"
(org-babel-process-file-name out-file))
                     " " (org-babel-process-file-name in-file)) "")
            nil)
        body))
    #+END_SRC

    # Example Usage
    {{{conditional-header(foo.svg)}}}
    #+BEGIN_SRC metapost :results drawer :post
wrap_metapost(name="my-name", caption="my-caption", data=*this*)
    beginfig(1);
    draw origin--(100,100)--(200,0)--cycle;
    endfig;
    end;
    #+END_SRC

This kind of works: It exports to HTML with a nice SVG figure and sends raw
METAPOST code to the backend, with #+CAPTION and #+NAME information
attached. Also, links to the figures work in the buffer as well as the pdf.

However, there's a lot I don't like about this.
1. No SVG preview in the buffer since the `conditional-header` macro doesn't
   get expanded until export time
2. Boilerplate; both the `conditional-header` and `wrap_metapost` definitions
   need to be included in the org file
3. Inconvenient to use since I have to add a macro call and a :post argument
   with weird syntax to every METAPOST figure, and the way captions and tags
   are specified is different.



On Mon, Oct 4, 2021 at 1:39 AM Ihor Radchenko <yantar92@gmail.com> wrote:
>
> Jason Ross <jasonross1024@gmail.com> writes:
>
> > Hello,
> >
> > I'd like to include METAPOST figures in the ConTeXt exporter backend I'm
> > developing. However, I don't know of an idiomatic way to add captions and
> > references for the figures.
>
> You can use affiliated keywords:
>
> (defconst org-element-affiliated-keywords
>   '("CAPTION" "DATA" "HEADER" "HEADERS" "LABEL" "NAME" "PLOT" "RESNAME" "RESULT"
>     "RESULTS" "SOURCE" "SRCNAME" "TBLNAME")
>   "List of affiliated keywords as strings.
> By default, all keywords setting attributes (e.g., \"ATTR_LATEX\")
> are affiliated keywords and need not to be in this list.")
>
> > Currently, I export METAPOST with `#+BEGIN_EXPORT metapost` / `#+END_EXPORT`
> > tags. However, this feature seems to be intended for completely "raw"
> > outputs
> > with no markup or tagging in the resulting export. I'm interested in
> > supporting
> > at least `#+NAME` and `#+CAPTION` keywords for METAPOST figures so that they
> > can be referred to in the Org file and also in the exported pdf.
>
>
>
> > What are some better ways of doing something like this? Source blocks? How
> > would
> > a user expect to use a feature like this?
>
>
> You can use a special block:
> #+caption: Sample caption
> #+attr_context: :field value
> #+begin_metapost
> ...
> #+end_metapost
>
> The parsed representation of this element in exporter will be something like:
>
> (special-block (:type "metapost" :begin 292 :end 382 :contents-begin 364 :contents-end 368 :post-blank 0 :post-affiliated 347 :caption (((#("Sample caption" 0 14 (:parent #4))))) :attr_context (":field value")))
>
>
> Best,
> Ihor

Re: Best way to include METAPOST in ConTeXt exporter

[Ihor Radchenko]

Jason Ross <jasonross1024@gmail.com> writes:

> I had considered using special blocks; they match my mental model the best.
> However, they don't provide any support for syntax highlighting or opening
> the block in a new major mode buffer. I'm not sure if it's worth giving up
> language features in order to use the block that's most intuitive to users.
>
> I'm also considering writing an Org Babel module for METAPOST. This could
> allow METAPOST figures to be included with any export backend. However,
> if I go this route, there are still some challenges:

If you are in control of the export backend, you can directly process
the metapost source blocks during export and ignore/filter their output
as needed.

>    #+MACRO: conditional-header (eval (when (not (eq
> org-export-current-backend 'context)) (concat  "#+HEADER: :results
> file :file " $1)))

>    #+BEGIN_SRC metapost :results drawer :post
> wrap_metapost(name="my-name", caption="my-caption", data=*this*)

Or you can use "raw" results by default and format everything as you
wish in your Org Babel module programatically.

If above is not sufficient, you can install global export filters that
can transform the metapost blocks/source blocks/their results as you
need in other export backends.

Best,
Ihor

Re: Best way to include METAPOST in ConTeXt exporter

[Jason Ross]

(Apologies to Ihor who I already replied to without cc'ing the list)
=======================================================
> If you are in control of the export backend, you can directly process
> the metapost source blocks during export and ignore/filter their output
> as needed.

This is definitely possible, but I don't want to commit to bypassing the
entire Org Babel system right now. I will explore this a bit.

> Or you can use "raw" results by default and format everything as you
> wish in your Org Babel module programatically.

I'm not sure I understand this yet. Would this emit METAPOST code to the
Org buffer? Ideally, I'd like to emit METAPOST code if the backend is
`context`, otherwise, emit a file.

> If above is not sufficient, you can install global export filters that
> can transform the metapost blocks/source blocks/their results as you
> need in other export backends.

This sounds like it might be a good solution. Is it possible to have an
export filter that changes the header arguments of a source block before
Org Babel sees them? If so, ox-context could change all METAPOST source
block header arguments to give raw results wrapped in appropriate tags.

Also, I'm still curious if there's a better way to add #+CAPTION and
#+NAME tags to images generated by source blocks (in any context). It
would be extremely handy to reference figures generated with
matplotlib throughout the document, not just METAPOST drawings.

Thanks,
Jason

On Mon, Oct 4, 2021 at 9:44 AM Ihor Radchenko <yantar92@gmail.com> wrote:
>
> Jason Ross <jasonross1024@gmail.com> writes:
>
> > I had considered using special blocks; they match my mental model the best.
> > However, they don't provide any support for syntax highlighting or opening
> > the block in a new major mode buffer. I'm not sure if it's worth giving up
> > language features in order to use the block that's most intuitive to users.
> >
> > I'm also considering writing an Org Babel module for METAPOST. This could
> > allow METAPOST figures to be included with any export backend. However,
> > if I go this route, there are still some challenges:
>
> If you are in control of the export backend, you can directly process
> the metapost source blocks during export and ignore/filter their output
> as needed.
>
> >    #+MACRO: conditional-header (eval (when (not (eq
> > org-export-current-backend 'context)) (concat  "#+HEADER: :results
> > file :file " $1)))
>
> >    #+BEGIN_SRC metapost :results drawer :post
> > wrap_metapost(name="my-name", caption="my-caption", data=*this*)
>
> Or you can use "raw" results by default and format everything as you
> wish in your Org Babel module programatically.
>
> If above is not sufficient, you can install global export filters that
> can transform the metapost blocks/source blocks/their results as you
> need in other export backends.
>
> Best,
> Ihor

Re: Best way to include METAPOST in ConTeXt exporter

[Jason Ross]

[-- Attachment #1: Type: text/plain, Size: 1044 bytes --]

Here's a hook that modifies the source blocks to wrap their
output in #+BEGIN/END_METAPOST tags if the ConTeXt backend is used,
before Org Babel gets to them, but otherwise leaves them alone.

I wonder if anyone has any better ideas of how to do this. I'm
modifying the Org source with the hook before the document gets parsed
so that it can be more backend-agnostic but it seems like it would
be better if there was a way to modify the document parse tree
directly instead. I don't like that I'm effectively parsing and
rebuilding (hopefully) the same string in order to change the :result
type.

I also don't like that I don't really have a clean way of turning
the hook on and off with document keywords. This is kind of a nasty
thing to do to a document and users should probably have to explicitly
opt in.

I also found an old answer that describes how to add captions
to figures generated by source blocks:
https://www.mail-archive.com/emacs-orgmode@gnu.org/msg68100.html
Probably not news to many other people on this list but myself :)

[-- Attachment #2: metapost-handler.org --]
[-- Type: text/plain, Size: 5796 bytes --]

#+TITLE: Metapost Handler

This is a basic handler for METAPOST that exports as raw code
when the ConTeXt exporter is used but otherwise does whatever you
tell it to.

#+NAME: hooks
#+BEGIN_SRC emacs-lisp :exports none :results none
(defun format-src-block-arguments (arguments)
  "Returns a formatted plist of header arguments"
  (mapconcat
   (lambda (argument)
     (let ((kw (car argument))
           (vals (cdr argument)))
       (concat (format "%s" kw)
               " "
               (format "%s" vals))))
   arguments
   " "))
(defun metapost-process-hook (backend)
  "If BACKEND is `context', change metapost code blocks to output
raw code wrapped in #+BEGIN_METAPOST/#+END_METAPOST tags."
  ;; TODO This should be controlled by a flag.
  ;; TODO Check buffer info to see if we are allowed to do this.
  (when (string= backend "context")
    (goto-char (point-min))
    (let ((case-fold-search t)
          ;; Search for source code with a regex
          (regexp "^[ \t]*#\\+BEGIN_SRC"))
      (while (re-search-forward regexp nil t)
        (let* ((objectp (match-end 1))
               (tree (org-element-parse-buffer))
               ;; Get the buffer info plist (need this to export a caption)
               (info (org-combine-plists
                     (org-export--get-export-attributes)
                     (org-export-get-environment)))
               (info (progn
                      (org-export--prune-tree tree info)
                      (org-export--remove-uninterpreted-data tree info)
                      (org-combine-plists info
                                          (org-export--collect-tree-properties
                                           tree info))))
               ;; Get a code element
               (element
                (save-match-data
                  (if objectp (org-element-context) (org-element-at-point))))
               (caption (org-element-property :caption element))
               (type (org-element-type element))
               (begin (copy-marker (org-element-property :begin element)))
               (end (copy-marker
                     (save-excursion
                       (goto-char (org-element-property :end element))
                       (skip-chars-backward " \r\t\n")
                       (point))))
               (block-info (org-babel-get-src-block-info t))
               (language (nth 0 block-info))
               (body (nth 1 block-info))
               (arguments (nth 2 block-info))
               (arguments (delq (assoc :file arguments) arguments))
               (switches (nth 3 block-info))
               (name (nth 4 block-info))
               (start (nth 5 block-info))
               (coderef (nth 6 block-info)))

          (when (or t (string= (downcase language) "metapost"))
            ;; Remove "file" from `results' setting
            (setf (alist-get :results arguments)
                  (mapconcat
                   #'identity
                   (seq-filter
                    (lambda (a) (not (string= a "file")) )
                    (split-string (alist-get :results arguments)))
                   " "))
            ;; Add a wrap argument to wrap in a METAPOST special block
            (setf (alist-get :wrap arguments) "METAPOST")
            (pcase type
              (`src-block
               (progn
                 (delete-region begin end)
                 (goto-char begin)
                 (insert
                  (concat
                   ;; Captions and names got deleted; add them back
                   (when (org-string-nw-p name)
                     (format "#+NAME: %s \n" name))
                   (when caption
                     (format "#+CAPTION: %s\n"
                             (org-string-nw-p
                              (org-trim
                               (org-export-data
                                (or
                                 (org-export-get-caption element t)
                                 (org-export-get-caption element))
                                info)))))
                   ;; Add the (modified) header arguments back
                   (format "#+BEGIN_SRC metapost %s\n%s\n#+END_SRC"
                           (format-src-block-arguments arguments)
                           body)
                   "\n"))))))))
      (goto-char (point-min)))))

(remove-hook 'org-export-before-processing-hook 'metapost-process-hook)
(add-hook 'org-export-before-processing-hook 'metapost-process-hook)
#+END_SRC

#+NAME: metapost-export
#+BEGIN_SRC emacs-lisp :exports none :results none
(defun org-babel-execute:metapost (body params)
  "Execute a block of metapost code with org-babel.
This function is called by `org-babel-execute-src-block'."
  (if (cdr (assq :file params))
      (let* ((out-file (cdr (assq :file params)))
             (cmdline (or (cdr (assq :cmdline params))
                          (format "-T%s" (file-name-extension out-file))))
             (cmd (or (cdr (assq :cmd params)) "mpost"))
             (coding-system-for-read 'utf-8) ;use utf-8 with sub-processes
             (coding-system-for-write 'utf-8)
             (in-file (org-babel-temp-file "metapost-")))
        (with-temp-file in-file
          (insert (org-babel-expand-body:generic body params)))
        (org-babel-eval
         (concat cmd
                 " -s 'outputformat=\"svg\"'"
                 (format " -s 'outputtemplate=\"%s\"'" (org-babel-process-file-name out-file))
                 " " (org-babel-process-file-name in-file)) "")
        nil)
    body))
#+END_SRC


#+NAME: some-name
#+BEGIN_SRC metapost :results file :file foo.svg :exports results
beginfig(1);
draw origin--(100,100)--(200,0)--cycle;
endfig;
end;
#+END_SRC

#+CAPTION: Some caption
#+RESULTS: some-name
[[file:foo.svg]]

Re: Best way to include METAPOST in ConTeXt exporter

[Ihor Radchenko]

Jason Ross <jasonross1024@gmail.com> writes:

> Here's a hook that modifies the source blocks to wrap their
> output in #+BEGIN/END_METAPOST tags if the ConTeXt backend is used,
> before Org Babel gets to them, but otherwise leaves them alone.
>
> I wonder if anyone has any better ideas of how to do this. I'm
> modifying the Org source with the hook before the document gets parsed
> so that it can be more backend-agnostic but it seems like it would
> be better if there was a way to modify the document parse tree
> directly instead. I don't like that I'm effectively parsing and
> rebuilding (hopefully) the same string in order to change the :result
> type.

I am not sure if there is an easy way. ob-exp gets run before backend
filters and you do not yet have access to the document parse tree.

You may consider a patch for ob-exp to get better control over code
execution during export.

Best,
Ihor

Re: Best way to include METAPOST in ConTeXt exporter

[Ihor Radchenko]

Jason Ross <jasonross1024@gmail.com> writes:

>> Or you can use "raw" results by default and format everything as you
>> wish in your Org Babel module programatically.
>
> I'm not sure I understand this yet. Would this emit METAPOST code to the
> Org buffer? Ideally, I'd like to emit METAPOST code if the backend is
> `context`, otherwise, emit a file.

A hackish way would be detecting if you are executing code during export
and emitting different results (say, you may look at
org-export-current-backend).  However, it will do nothing when user
specifies :results file

Best,
Ihor