Re: Experimental public branch for inline special blocks

[Ihor Radchenko <yantar92@posteo.net>]

Juan Manuel Macías <maciaschain@posteo.net> writes:

> Finally, I have made public on GitLab my experimental branch for the new
> (possible) inline-special-block element:
>
> <https://gitlab.com/maciaschain/org-mode.git>

I have finally finished my review of the previous related discussions.
See my comments and proposals below.

[ Aside: It looks like your public branch is not up-to-date as the time
  of writing this email - I do not see commits changing the syntax to
  @foo{...} and @@{...} yet, as discussed in
  https://list.orgmode.org/orgmode/87sf121e9t.fsf@localhost/)
  I have edited your original message in the quotes to reflect upon the
  newly agreed syntax. ]

---------------------

> The basic syntax:
> @foo{lorem ipsum dolor}

> There is also an anonymous variant:
> @@{lorem ipsum dolor}

> Optional attributes in square brackets are supported
> @foo[:key1 value1 :key2 value2 ...]{lorem ipsum dolor}

Let me list the main goals of the new inline markup within the scope of
overall Org markup and discuss whether the proposed syntax can work to
achieve them.

1. Introduce user-configurable ad-hoc syntax where exporting and
   fontification are fully configurable on Elisp level, per-document
   level, and maybe even per-markup level. Something combining
   flexibility of links and special blocks, but without their
   limitations:

   - In contrast to links, we should be able to nest inline markup
     without restrictions:

     @foo{@bar{text}} ([[foo:a][[[bar:b]]]] is not possible)

     ***This goal is covered by the branch***

   - We should be able to have arbitrary text inside inline markup.
     No sequence of symbols should be prohibited:

     @foo{ can contain leading and trailing spaces }
     @foo{can contain "}" inside, especially inside =verbatim }= text} <-- **ambiguous**
     @foo{can even have "}" at the end boundary}} <-- **ambiguous**
     This is unlike links that prohibit closing "]".

     ***For the above examples, the proposed syntax is ambiguous***

    - We should be able to define special markup for code, where the
      contents is not parsed. Something like

      @code{ unlike =code=, we can have leading and trailing spaces }
      @code{ @foo{is not interpreted inside}}

      ***This goal is not addressed by the experimental branch for now***

2. Allow attaching auxiliary attributes to the on object level, as an
   equivalent of affiliated keywords on element level

   - For example, we should allow assigning height per-link without the
     awkward kludge we have with special handling of
     
     #+attr_html: :height 300
     [[file:image.png]] has a height of 300, but what if we want a
     different height in [[file:another-image.png]]?

     We can thus do
     
     #+attr_html: :height 300
     [[file:image.png]] has a height of 300, but what if we want a
     different height in @@[:html-height 300]{[[file:another-image.png]]}?

     Note how @@{...} markup assigns attributes to objects inside - the
     attributes should be somehow inherited.

     Another example is the proposed @@[:lang "en"]{...} attribute - we
     may want to assign it even beyond current paragraph; for example
     for the whole subtree (aka mark subtree language to be "English").

     ***This is not covered by the branch***

   - We should also be able to assign attributes that contain Org markup
     themselves:

     @@[:alt This image shows a _cat_ climbing a /wall/]{<file:cat.png>}

     ***This is not covered by the branch***

3. The new syntax should allow us to do anything Texinfo markup can do
   (as per RMS request https://list.orgmode.org/orgmode/87bkqx4jyg.fsf@localhost/)

   - Multi-argument markup in Texinfo like for abbreviations
     @abbr{abbreviation[,meaning]}

     Example: @abbr{Comput. J., Computer Journal}
     (https://www.gnu.org/software/texinfo/manual/texinfo/texinfo.html#g_t_0040abbr)

     We can equivalently (if markup inside attributes is supported) do

     @abbr[:meaning Computer Journal]{Comput. J.}
     or even
     @abbr[Computer Journal]{Comput. J.}

     In theory, we may need even more than 2 arguments.

     ***This is partially covered***

4. The new syntax should allow working around ambiguities of the *DWIM*
   markup without using the awkward zero-width space kludge

   - Intra-word markup

      foo@bold{bar}baz
      
      ***This is covered already***

   - Simultaneous markup
   
     @bold{foo}@italic{bar}
     @@{*foo*}@@{/bar/}

      ***This is covered already***

   - Solving the undesired first-fit-wins parser behaviour:

     /italics stops [[early/?here]], but not later/
     need to *escape stars* like that one*
     你好。*我*不会些这个

     @italic{italics stops [[early/?here]], but not later}
     need to @bold{escape stars* like that one}
     你好。@bold{我}不会些这个

      ***This is covered already***


--------
Proposed modifications to the inline markup syntax
-------

1. @foo{basic form remains the same}

2. @foo{{but we allow arbitrary number of "{{..." to open the markup.
   Then, } inside does not end the @foo markup; we only end it at the
   matching number of }s; "{" and "}" inside do not have to be balanced.}}

3. @foo[alternative brackets]

   In an edge case, when the contents itself is {...}, allow alternative
   brackets

   @foo[{contents surrounded by curly braces}]

4. We also have a special form of syntax where the contents is not at all
   parsed

   @code{this is *verbatim* code; it may contain = or even have
   whitespace at the boundaries                    }

   This is equivalent to #+begin_example example blocks vs.
   #+begin_<arbitrary> special blocks.

   @code{{can also contain "}" inside, using multiple bracket
   alternative}}

   @code[{or be like this}]

5. Change the @foo[:key1 value1 :key2 value2 ...]{lorem ipsum dolor}
   syntax:

   @foo[can have][multiple][arguments]{the last argument is considered object contents}

   Every argument is parsed

   @title[This is an image title, with *markup*]{<file:image.png>}
   @@[:lang cn :alt Hello]{你好}
   @@[:lang cn][:alt Hello]{你好}

   The exporters are responsible to interpret the arguments as needed,
   if keyword arguments are passed.

   We should also not layer additional rules for escaping delimiters
   apart from the existing Org mode markup:

   @@[:lang cn :alt @@{can have :keyword-like inside, insulated inside =@@{...}=}]{...}
   @@[:lang cn][:attr_latex @@{:color blue}]{...}

   This is akin affiliated keywords that are never split into
   keyword/value pairs by org-element, leaving this job downstream to
   specific commands and libraries.
   (The difference is that affiliated keywords are mostly not parsed,
   but parsing is necessary here to address multi-argument markup syntax
   like @abbrev)

-------
Attribute inheritance
-------

As I described in @@[:html-height 300]{[[file:another-image.png]]}
example, we should have some form of attribute inheritance, so that we
are able to assign attributes to arbitrary Org markup, not just to
inline markup objects.

Furthermore, attributes like @@[:lang cn][好工作] may need to be
assigned to the text spanning across multiple paragraphs or even
headings.

I suggest following what we already do for source blocks:

1. org-babel-default-header-args sets global attributes applicable to
   everything globally
2. org-babel-default-header-args:<lang> sets global attributes
   per-language
3. #+PROPERTY: header-args <attributes> sets global attributes per-file
4. #+PROPERTY: header-args:<lang> <attributes> sets language-specific
   attributes
5. :HEADER-ARGS: or :HEADER-ARGS:<lang>: set attributes per-subtree
6. #+HEADERS: <attributes> sets src block attributes
7. Attribute values are merged together:

   org-babel-default-header-args = :cache yes :export none
   #+HEADERS: :cache no :var x=1
   combines into
   :export none :cache no :var x=1

I propose the following naming for inline markup:

  org-default-inline-attributes
  org-default-inline-attributes:<foo>
  #+PROPERTY: inline_attr
  #+PROPERTY: inline_attr:<foo>
  #+INLINE_ATTR: affiliated
  #+INLINE_ATTR:<foo>:

In addition, attributes from @@[:alt Text]{<file:image.png>} are applied
to all the markup inside, including normal non-inline markup objects.

Further, we may allow a special attribute :inherit that will allow more
fine-grained control on where to inherit the attributes from:

#+inline_attr:var: :inherit example
@example[:color gray]{(defvar @var{variable-name} nil)}

As an option, I am not yet sure about, we may also allow per-keyword
control like

@bold[:export latex rest*]{and some more @bold[:export+ html]{inside}}

so that :export and :export+ are combined.

(Despite me using :export example here, what I am proposing is not
limited to :export keyword that have been discussed in-depth in
https://list.orgmode.org/orgmode/87bk7k7tuf.fsf@posteo.net/)

> Optional attributes in square brackets are supported. There are a series
> of 'universal' attributes, common to each backend. At the moment:
> `:lang', `:color' and `:smallcaps'. Example:

-------
On :lang attribute
-------

This attribute is special - language of a text is actually _not_ a
markup. It is more global, and it does not quite fit within the framework
of per-markup type inheritance.

In contrast to something like
@foo[:export latex rest*]{@bar{should be exported everywhere}}
:lang attribute applies to all types of markup inside
@foo{:lang cn}{@bar{公共汽车}}

Moreover, we may want to apply it per-paragraph or per-subtree.

So, we may want to

1. Combine all the nested inline :lang attributes regardless of the
   inline markup types nesting
2. Introduce a new affiliated keyword #+LANGUAGE
3. Introduce a new subtree property :LANGUAGE:
4. Re-use #+LANGUAGE: per-document keyword
5. Use the previous 4 rules to :lang attributes as a special case.

Then, export backends should implement :lang attribute support and ox.el
should provide the means to query it, performing all the necessary
inheritance calculations.

-------
Configurable inline markup export
-------

Given that the discussed inline markup is supposed to improve on the
link export flexibility, and also add more ad-hoc export setting
options, I have the following necessary features in mind:

1. We should have an equivalent of :export link parameter
   This will be Elisp level interface for inline markup, allowing to
   write Org mode syntax extensions (like the one necessary for Texinfo
   feature-parity)

   ***This is not covered by the branch***

2. We should expose per-document ad-hoc markup customization that does
   not require Elisp. Something similar to link abbreviations or macros.

   However, I do not want the inline markup to turn into another variant
   of macros. So, I do like the proposal to define only attributes:

> ┌────
> │ #+options: inline-special-block-aliases:(("latin" :lang "la" :latex-command "textit" :html-tag "em"))
> │ 
> │ Caesar's famous quote: @latin{Alea iacta est}
> │ 
> │ Caesar's famous quote: @latin[:smallcaps t :color blue]{Alea iacta est}
> └────

However, I am not a big fan of the way it is implemented - Elisp syntax
with the need to layer all the (...) and "...".

We may instead re-use my attribute inheritance idea:

#+PROPERTY: inline_attr:latin :lang la :latex-command textit :html-tag em

-------
On :color and :smallcaps attributes or user-customized markup
-------

These proposed attributes are nothing but additional markups that will
become a part of Org mode syntax. I see them as nothing different from
@smallcals{...} and @color{blue}{...} inline markups.

I am not in favour of adding such extra markup to Org mode syntax.
Instead, we can make it a part of inline markup extensions, in parallel
with what we need to support Texinfo-specific markup like @var, @defun,
etc.

I also do not see any clear benefit of adding :color and :smallcaps as
_attributes_. Just markup will do. If one needs to combine
smallcaps/color with other markup, it is a job for ad-hoc markup
definitions.

-------
pre/post-latex (aka templates)
-------

> Specific to the LaTeX backend we have the `:prelatex' and `:postlatex'
> attributes (which introduce arbitrary code before and after the content)
> and `:latex-command', which overrides the exported command.
> `:latex-command nocommand' does not export a command flag. Examples:
>
> ┌────
> │ @foo[:prelatex [bar] :postlatex {baz} :lang it :latex-command blah]{lorem ipsum dolor}
> └────

I do not like the idea of pre/post-<exporter> attributes.
I think that we discussed this particular idea in the past, when talking
about prepending/appending commands like \newpage to headings during
LaTeX export
(https://list.orgmode.org/orgmode/87czbna4e4.fsf@localhost/)

There, I proposed to use a more generalized approach, allowing custom
export templates:

* headline
:PROPERTIES:
:ATTR_BACKEND: :export_template "\begin{myenv}\n%s\n\end{myenv}"
:ATTR_BACKEND+: "The %%s instances are replaced by the exported element"
:ATTR_BACKEND+: (concat "arbitrary sexp, the exported element is bound to: " *this*)
:ATTR_BACKEND+: babel_block_name(exported=*this*)
:ATTR_BACKEND+: "the property lines are concatenated with \" \" (space),"
:ATTR_BACKEND+: "just like the usual approach in `org-export-read-attribute'"
:END:

We can apply the same approach when defining ad-hoc markup, extending it
a little. Rather than using %s, we introduce more placeholders:

   - %contents :: like CONTENTS argument in the export transcoders
   - %transcoded :: the result of what the export backend returns normally
   - %:<number> :: reference to last Nth optional argument. For example
        in @abbrev{Computer Science}{CS} %contents == CS and %:1 =
        Computer Science
   - %:attribute :: the value of attribute

This way,

@@[:prelatex \foo{bar} :color red]{lorem ipsum dolor}

will become

@foo[:latex-template @code[{\color{red}\foo{bar}%contents}]]{lorem ipsum dolor}

-------
On :export attribute
-------

> I'm thinking about adding a new global attribute, `:export', that
> would granularly control whether or not to export the object and how to
> export it.
>
> Possible values: "noexport", "contents" (it would export only the content)
> or the backends to which you want to export, separated by spaces. Each
> backend should be followed by a "+" sign (= export all) or "*" (export
> content only). For example:
>
> @foo[:color red :export latex+ odt* html*]{lorem ipsum dolor}

Similar to :lang attribute, I see this idea to be more general than
inline markup scope. It does not have to be applicable only to inline
markup. We can extend selective export further - to paragraph-level
elements and to headings.

Recently, we even got a feature request to
exclude/include certain subtrees from export for specific backends:
https://list.orgmode.org/orgmode/1008678740.100388.1708624390334@fidget.co-bxl/

So, we can approach this just like for :lang attribute, except that we
do not need special handling with unconditional inheritance for all the
inline markup

1. Allow :lang attributes and make them follow standard inheritance
   rules 
2. Introduce a new affiliated keyword #+EXPORT
3. Introduce a new subtree property :EXPORT:
4. Introduce #+EXPORT: per-document keyword
5. (optional) Introduce new #+SELECT_TAGS:BACKEND:
   #+EXCLUDE_TAGS:BACKEND to allow excluding/selecting subtrees by tag
   (more visible compared to properties)

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>