[ANN] Org Export in contrib

[ANN] Org Export in contrib

[Nicolas Goaziou]

Hello,

I've pushed org-export.el to contrib. It's a general export engine,
built on top of org-elements aiming at simplifying life of both
developers and maintainers (and, therefore, of end-users).

It predates org-exp.el and org-special-blocks.el. Though, keep it mind
that, as advanced as it is, it isn't yet a drop-in replacement for
them. It still lacks an interface (à la `org-export'), back-ends, and
tons of testing and improvements. That being said, it's usable anyway
and one can already write back-ends for it. I'll show a silly example
later in this mail.

Now, let's have a peek into the guts of that beast.

Besides the parser, the generic exporter is made of three distinct
parts:

- The communication channel consists in a property list, which is
  created and updated during the process.  Its use is to offer every
  piece of information, would it be export options or contextual data,
  all in a single place.  The exhaustive list of properties is given in
  "The Communication Channel" section of the file.

- The transcoder walks the parse tree, ignores or treat as plain text
  elements and objects according to export options, and eventually calls
  back-end specific functions to do the real transcoding, concatenating
  their return value along the way.

- The filter system is activated at the very beginning and the very end
  of the export process, and each time an element or an object has been
  converted.  It is the entry point to fine-tune standard output from
  back-end transcoders.

The core function is `org-export-as'.  It returns the transcoded buffer
as a string.

In order to derive an exporter out of this generic implementation, one
can define a transcode function for each element or object.  Such
function should return a string for the corresponding element, without
any trailing space, or nil.  It must accept three arguments:

  1. the element or object itself,
  2. its contents, or nil when it isn't recursive,
  3. the property list used as a communication channel.

If no such function is found, that element or object type will simply be
ignored, along with any separating blank line.  The same will happen if
the function returns the nil value.  If that function returns the empty
string, the type will be ignored, but the blank lines will be kept.

Contents, when not nil, are stripped from any global indentation
(although the relative one is preserved).  They also always end with
a single newline character.

These functions must follow a strict naming convention:
`org-BACKEND-TYPE' where, obviously, BACKEND is the name of the export
back-end and TYPE the type of the element or object handled.

Moreover, two additional functions can be defined.  On the one hand,
`org-BACKEND-template' returns the final transcoded string, and can be
used to add a preamble and a postamble to document's body.  It must
accept two arguments: the transcoded string and the property list
containing export options.  On the other hand, `org-BACKEND-plain-text',
when defined, is to be called on every text not recognized as an element
or an object.  It must accept two arguments: the text string and the
information channel.

Any back-end can define its own variables.  Among them, those
customizables should belong to the `org-export-BACKEND' group.  Also,
a special variable, `org-BACKEND-option-alist', allows to define buffer
keywords and "#+options:" items specific to that back-end.  See
`org-export-option-alist' for supported defaults and syntax.

Tools for common tasks across back-ends are implemented in the last
part of the file.


* For Maintainers

  To word it differently, this exporter doesn't rely on any
  text-property during the process. Thus, it makes
  `org-if-unprotected-at' and alike obsolete in the whole code base. Org
  core doesn't have to bother anymore about its exporter weaknesses.

  Also, buffer's pre-processing is reduced to its strict minimum: Babel
  code expansion. No footnote normalization, list markers to add and
  remove...

  Being only a beefed-up parse tree reader, any element or object added
  to Elements is available through the exporter with no further
  modification. Back-end just have to create the appropriate new
  transcoders, unless that element or object should be ignored anyway.


* For Developers

  All data needed is available in two places: the properties associated
  to the element being transcoded, through the use of
  `org-element-get-property', and the communication channel, with the
  help of `plist-get'. Period.

  Also, the exporter takes care about all the filtering required by
  options, and enforces the same number of blank lines in the Org buffer
  and in the source code (though this can be overcome with the use of
  filters). It's easier this way to concentrate on the shape of the
  output.

  Tools for common tasks (like building table of contents or listings,
  or numbering headlines) are provided in the library.


* For Users

  Hooks are gone. Sorry. Most of them happened during a pre-process part
  that doesn't exist anymore.

  Though, there are three way to configure the output, in increasing
  power:

  1. Variables (customizable or not) are still there, provided by either
     the exporter itself or its back-ends.

  2. Filter sets are provided to fine-tune output of any
     back-end. A filter set is a list of functions, applied in a FIFO
     order, whose signature is the resulting string of the previous
     function (or the back-end output for the first one) and the
     back-end as a symbol. The return value of the last function
     replaces back-end's output. If one of the return values is nil, the
     element or object on which the filters are applied is ignored in
     the final output.

     Also, three special filter sets apply on the parse tree, on plain
     text, and on the final output.

     For example, the LaTeX back-end has the bad habit to "boldify"
     deadline, scheduled and closed strings close to time-stamps in the
     buffer. I'd rather have them emphasized. Obviously, I don't want to
     annoy other back-ends with this. The following will do the trick.

     #+begin_src emacs-lisp
     (add-to-list 'org-export-filter-time-stamp-functions
                  (lambda (ts backend)
                    (if (not (eq backend 'latex))
                        ts
                      (replace-regexp-in-string "textbf" "emph" ts))))
     #+end_src

  3. Whole parts of any back-end can be redefined (or advised).  For
     example, if I don't like the way the LaTeX back-end transcodes
     verbatim text, I can always create an `org-latex-verbatim' function
     of my own.


* A (silly) Back-End: `summary'

  I want a back-end, `summary', which only exports headlines of the
  current buffer, in a markdown format. I would like to have the
  opportunity to add a few lines of text before the first headline.  It
  should also delimit beginning and end of output by ASCII scissors. Oh,
  and scissors string should be customizable!

  As I only want headlines, I only need to implement an
  `org-summary-headline' function. Though, text before the first
  headline in my buffer will be ignored (it isn't an headline).

  So this back-end will have to define its own buffer keyword:
  "#+PREAMBLE:". I need to be able to encounter this keyword more than
  once in the buffer as my preamble will probably span on more than one
  line. The following snippet will do this, and provide the text as the
  value of the `:preamble' property in the communication channel. It
  also creates a customizable `org-summary-scissors' variable, which is
  rightfully added to the `org-export-summary' group.

  #+begin_src emacs-lisp
  (defcustom org-summary-scissors "--%<--------------------------------\n"
    "String used as a delimiter for summary output.
  It should end with a newline character."
    :group 'org-export-summary
    :type 'string)
  (defvar org-summary-option-alist)
  (add-to-list 'org-summary-option-alist
               '(:preamble "PREAMBLE" nil nil newline))
  #+end_src

  Now onto the headline transcoder. A quick look at the
  `org-element-headline-parser' function tell me that `:raw-value'
  property should be enough, as I need no fancy transformation. I might
  want to also use `:level' to get the number of "equal" signs before
  the text, but a longer look at the list of properties offered in the
  communication channel tells me that `org-export-get-relative-level'
  may be more adequate. So be it.

  #+begin_src emacs-lisp
  (defun org-summary-headline (headline contents info)
    "Return HEADLINE in a Markdown syntax.
  CONTENTS is the contents of the headline.  INFO is the property
  list used as a communication channel."
    (let ((title (org-element-get-property :raw-value headline))
          (pre-blank (org-element-get-property :pre-blank headline))
          (level (org-export-get-relative-level headline info))
          ;; Depth of 6 is a hard limit in HTML (and therefore Markdown)
          ;; syntax.
          (limit (min (plist-get info :headline-levels) 6)))
      (when (<= level limit)
        (concat (make-string level ?=) " " title
                (make-string (1+ pre-blank) ?\n)
                contents))))
  #+end_src

  This should be sufficient to take care of document's body. Now, I only
  need to add the scissors, the preamble, and the title in the final
  output. This all happens with the help of the `org-summary-template'
  function.

  I remember that "#+TITLE:" belongs to `org-element-parsed-keywords',
  which means that its value isn't a string but a secondary string. As
  I don't want to transcode it (my back-end only knows about headline),
  I'll get the raw value back with `org-element-interpret-secondary'
  function (If I had wanted to transcode it, I would have used
  `org-export-secondary-string' instead).

  #+begin_src emacs-lisp
  (defun org-summary-template (contents info)
    "Return complete document transcoded with summary back-end.
  CONTENTS is the body of the document.  INFO is the plist used as
  a communication channel."
    (let ((title (org-element-interpret-secondary (plist-get info :title)))
          (preamble (plist-get info :preamble)))
      (concat org-summary-scissors
              (upcase title) "\n\n"
              preamble "\n\n"
              contents
              org-summary-scissors)))
  #+end_src

  Now, I can test all of this by with M-: (org-export-as 'summary) on
  a test buffer[1]. So far, so good. But I know better and define an
  interactive function for that important action. While I'm at it, it
  will display my summary in a buffer named "*Summary*".

  #+begin_src emacs-lisp
  (defun org-export-as-summary ()
    "Create the summary of the current Org buffer.
  Summary is displayed in a buffer called \"*Summary*\"."
    (interactive)
    (when (eq major-mode 'org-mode)
      (switch-to-buffer (org-export-to-buffer 'summary "*Summary*"))))
  #+end_src

  That's all, folks.


I'll try to package its first back-end, org-latex.el, into experimental/
before monday.

Feedback, as always, is welcome.


Some text that will probably be ignored.

* Head 1

  some text
  
** Head 1.1

   some text too

*** Head 1.1.1

    Some text again

** Head 1.2

   some text
   
* Head 2                                                                :noexport:

  some text
--8<---------------cut here---------------end--------------->8---


Regards,

[1] For example, this one:
--8<---------------cut here---------------start------------->8---
#+Title: My Summary Test
#+Preamble: I hope
#+Preamble: that it is working
#+Options: H:2

-- 
Nicolas Goaziou

Re: [ANN] Org Export in contrib

[Nicolas Goaziou]

Completing myself, I'll add a few notes about the differences between
the current exporter and this one. While it tries to mimic most of the
behaviours of its ancestor, some points just couldn't fit in the new
scheme, would it be for a technical reason or by design.

Here follows a rough list stating how the new engine stands with regards
to the older one.  When possible, the reasons justifying that the
difference remains are given.

1. It's slower.  Indeed, it has to first parse completely the area to
   transcode, even if it doesn't transcode it after (i.e. archive
   trees).  It's the job of the exporter to decide what should or
   shouldn't be exported.  Anticipating filtering would be disastrous
   for some exporters (i.e. the Org one) and wouldn't fit the modular
   design.

2. The document title cannot be obtained anymore from the first line of
   text.  It's either explicitely defined with the =TITLE= keyword, or
   derived from buffer's name.

3. =TEXT= keywords at the beginning of the document are unneeded, and,
   as such, no longer have an effect.

4. "[TABLE OF CONTENTS]" as a placeholder for the table of contents has
   been removed.  Though, a new keyword, =TOC=, achieves the same
   effect, and can even take more values, like "contents", "tables",
   "figures" and "listings".  Tools are provided to make them available
   to any exporter to come.

5. "-t" option at a src or example block isn't supported anymore.  This
   is an HTML only option that could fit in =attr_html= affiliated
   keyword instead, with a "textarea" value.  A new HTML back-end isn't
   available yet, but should develop in that direction anyway.

6. Macros have been under powered a little.  They cannot live anymore in
   comments, example blocks, or even src blocks.  In fact, one cannot
   find a macro where the text isn't parsed.  Macros are Org syntax.
   Using such syntax where there is, by definition, none is just
   nonsensical.

7. For the same reason, invisible targets in comments have been
   deprecated.  For the same result, one can now use a =TARGET= keyword
   instead.

8. =INCLUDE= keyword :prefix, :prefix1 and :minlevel properties support
   has been dropped. Though, the keyword, which doesn't have to be at
   column 0 anymore, is aware about its environment.  If it belongs to
   a list, the whole file will be in the current item.  Also, including
   a file from within a sub-tree will always make the top-level headline
   of the included file is a direct child of that sub-tree.  One
   technical drawback is that no Babel block is executed in the included
   file.

9. Table.el tables will always use their own export back-end.  In other
   words, no Org syntax will be recognized in such table anymore.
   A table.el is an extraneous element while the parser is meant to
   parse Org syntax.  Now, an interesting idea from /Jambunathan K./,
   namely his "list-tables", might fill the gap between the Org
   spreadsheet and the table.el table.  Nothing is implemented about it
   right now, but we're talking about a modular and extensible parser,
   after all.

10. Calling an export function with a numeric argument doesn't change
    headline maximum level, by default.  There are enough ways to change
    this export property already and the argument may be required for
    something more important in the future.

11. =org-export-with-TeX-macros= has been replaced with the more
    appropriate =org-export-with-entities=, and the associated =OPTIONS=
    keyword's symbol changed from =TeX:nil= to =e:nil=.

12. About variables changes, =org-export-author-info=,
    =org-export-creator-info= and =org-export-email-info= have been
    replaced with, respectively, =org-export-with-author=,
    =org-export-with-creator= ad =org-export-with-email=, for the sake
    of consistency with other opt-in variables.


Regards,

-- 
Nicolas Goaziou

Re: [ANN] Org Export in contrib

[Carsten Dominik]

Hi Nicolas,

a few comments:

On 25.11.2011, at 19:57, Nicolas Goaziou wrote:

> 
> Completing myself, I'll add a few notes about the differences between
> the current exporter and this one. While it tries to mimic most of the
> behaviours of its ancestor, some points just couldn't fit in the new
> scheme, would it be for a technical reason or by design.
> 
> Here follows a rough list stating how the new engine stands with regards
> to the older one.  When possible, the reasons justifying that the
> difference remains are given.
> 
> 1. It's slower.  Indeed, it has to first parse completely the area to
>   transcode, even if it doesn't transcode it after (i.e. archive
>   trees).  It's the job of the exporter to decide what should or
>   shouldn't be exported.  Anticipating filtering would be disastrous
>   for some exporters (i.e. the Org one) and wouldn't fit the modular
>   design.

I am curious to see how slow it will be.  If necessary, export can be
pushed to a background process, but I do not think this will be
necessary.

> 2. The document title cannot be obtained anymore from the first line of
>   text.  It's either explicitely defined with the =TITLE= keyword, or
>   derived from buffer's name.

This is actually good, I think.  Much of the stuff like initial text,
title from text etc are leftovers from early time.  It might mean that
we will have to keep the old exporters around somewhere, if someone
needs compatibility.

> 
> 3. =TEXT= keywords at the beginning of the document are unneeded, and,
>   as such, no longer have an effect.
> 
> 4. "[TABLE OF CONTENTS]" as a placeholder for the table of contents has
>   been removed.  Though, a new keyword, =TOC=, achieves the same
>   effect, and can even take more values, like "contents", "tables",
>   "figures" and "listings".  Tools are provided to make them available
>   to any exporter to come.

What do you mean by "keyword"?  Can you provide an example of how
to place the TOC?

> 
> 5. "-t" option at a src or example block isn't supported anymore.  This
>   is an HTML only option that could fit in =attr_html= affiliated
>   keyword instead, with a "textarea" value.  A new HTML back-end isn't
>   available yet, but should develop in that direction anyway.

Yes, this is a good alternative.

> 
> 6. Macros have been under powered a little.  They cannot live anymore in
>   comments, example blocks, or even src blocks.  In fact, one cannot
>   find a macro where the text isn't parsed.  Macros are Org syntax.
>   Using such syntax where there is, by definition, none is just
>   nonsensical.

I know that Stefan Vollmar is using macros in complicated and
extensive ways.  I also know that he is using the index facilities,
which so far have depended heavily on the preprocessor.  I am curious
how indexing will work with org-elements.  Have you put any thought
into this?

> 
> 7. For the same reason, invisible targets in comments have been
>   deprecated.  For the same result, one can now use a =TARGET= keyword
>   instead.

This is actually better...

> 
> 8. =INCLUDE= keyword :prefix, :prefix1 and :minlevel properties support
>   has been dropped. Though, the keyword, which doesn't have to be at
>   column 0 anymore, is aware about its environment.  If it belongs to
>   a list, the whole file will be in the current item.  Also, including
>   a file from within a sub-tree will always make the top-level headline
>   of the included file is a direct child of that sub-tree.  One
>   technical drawback is that no Babel block is executed in the included
>   file.

Again, no loss, this was a weird feature anyway.

> 
> 9. Table.el tables will always use their own export back-end.  In other
>   words, no Org syntax will be recognized in such table anymore.
>   A table.el is an extraneous element while the parser is meant to
>   parse Org syntax.  Now, an interesting idea from /Jambunathan K./,
>   namely his "list-tables", might fill the gap between the Org
>   spreadsheet and the table.el table.  Nothing is implemented about it
>   right now, but we're talking about a modular and extensible parser,
>   after all.

Maybe we should see if there is a hook in table.el which could
be called to format text in a backend specific way.  If it is not
there, maybe we can simply introduce it ourselves, or add some
advice for this purpose....

> 
> 10. Calling an export function with a numeric argument doesn't change
>    headline maximum level, by default.  There are enough ways to change
>    this export property already and the argument may be required for
>    something more important in the future.

Again, less is more.

> 
> 11. =org-export-with-TeX-macros= has been replaced with the more
>    appropriate =org-export-with-entities=, and the associated =OPTIONS=
>    keyword's symbol changed from =TeX:nil= to =e:nil=.

I do like the change, but maybe it would be good to support TeX
for backward compatibility...  Then, maybe, we are breaking a few
things anyway.

> 
> 12. About variables changes, =org-export-author-info=,
>    =org-export-creator-info= and =org-export-email-info= have been
>    replaced with, respectively, =org-export-with-author=,
>    =org-export-with-creator= ad =org-export-with-email=, for the sake
>    of consistency with other opt-in variables.

Are you adding defvaralias for compatibility, or are you arguing for
a clean break here?

Regards

- Carsten

Re: [ANN] Org Export in contrib

[Nicolas Goaziou]

Hello,

Carsten Dominik <carsten.dominik@gmail.com> writes:

>> 2. The document title cannot be obtained anymore from the first line of
>>   text.  It's either explicitely defined with the =TITLE= keyword, or
>>   derived from buffer's name.
>
> This is actually good, I think.  Much of the stuff like initial text,
> title from text etc are leftovers from early time.  It might mean that
> we will have to keep the old exporters around somewhere, if someone
> needs compatibility.

We're not there yet. But I hope there will be no backward compatibility.
We really should remove from all export unrelated code the export
related text-properties.

>> 4. "[TABLE OF CONTENTS]" as a placeholder for the table of contents has
>>   been removed.  Though, a new keyword, =TOC=, achieves the same
>>   effect, and can even take more values, like "contents", "tables",
>>   "figures" and "listings".  Tools are provided to make them available
>>   to any exporter to come.
>
> What do you mean by "keyword"?  Can you provide an example of how
> to place the TOC?

Just put "#+toc: headlines", "#+toc: tables", "#+toc: listings", "#+toc:
figures" anywhere in the document (at least with the LaTeX back-end).

>> 6. Macros have been under powered a little.  They cannot live anymore in
>>   comments, example blocks, or even src blocks.  In fact, one cannot
>>   find a macro where the text isn't parsed.  Macros are Org syntax.
>>   Using such syntax where there is, by definition, none is just
>>   nonsensical.
>
> I know that Stefan Vollmar is using macros in complicated and
> extensive ways.  I also know that he is using the index facilities,
> which so far have depended heavily on the preprocessor.  I am curious
> how indexing will work with org-elements.  Have you put any thought
> into this?

I don't think that macros will be a source of problems since comments
and example blocks were weird locations for them anyway.

In the LaTeX exporter, "#+index: something" will be transcoded into
"\index{something}". That's about it.

Should the generic export build a list of all "#+index:" values and
store it in a `:index' property (accessible through the communication
channel)?

>> 9. Table.el tables will always use their own export back-end.  In other
>>   words, no Org syntax will be recognized in such table anymore.
>>   A table.el is an extraneous element while the parser is meant to
>>   parse Org syntax.

> Maybe we should see if there is a hook in table.el which could
> be called to format text in a backend specific way.  If it is not
> there, maybe we can simply introduce it ourselves, or add some
> advice for this purpose....

Ok. If anyone can look at it and determine the right thing to do, I will
merge it into the exporter code.

>> 11. =org-export-with-TeX-macros= has been replaced with the more
>>    appropriate =org-export-with-entities=, and the associated =OPTIONS=
>>    keyword's symbol changed from =TeX:nil= to =e:nil=.
>
> I do like the change, but maybe it would be good to support TeX
> for backward compatibility...  Then, maybe, we are breaking a few
> things anyway.

TeX is still supported as a "latex-fragment". In the long run, though,
I think TeX commands that are not meant for dvipng/mathjax should be
called through an export snippet (that is "@l{...}" temporary syntax).

>> 12. About variables changes, =org-export-author-info=,
>>    =org-export-creator-info= and =org-export-email-info= have been
>>    replaced with, respectively, =org-export-with-author=,
>>    =org-export-with-creator= ad =org-export-with-email=, for the sake
>>    of consistency with other opt-in variables.
>
> Are you adding defvaralias for compatibility, or are you arguing for
> a clean break here?

There is no defvaralias. Though, I don't mind adding them.


Regards,

-- 
Nicolas Goaziou

Re: [ANN] Org Export in contrib

[Carsten Dominik]

On Nov 27, 2011, at 8:54 PM, Nicolas Goaziou wrote:

> Hello,
> 
> Carsten Dominik <carsten.dominik@gmail.com> writes:
> 
>>> 2. The document title cannot be obtained anymore from the first line of
>>>  text.  It's either explicitely defined with the =TITLE= keyword, or
>>>  derived from buffer's name.
>> 
>> This is actually good, I think.  Much of the stuff like initial text,
>> title from text etc are leftovers from early time.  It might mean that
>> we will have to keep the old exporters around somewhere, if someone
>> needs compatibility.
> 
> We're not there yet. But I hope there will be no backward compatibility.
> We really should remove from all export unrelated code the export
> related text-properties.

Yes, I agree.

> 
>>> 4. "[TABLE OF CONTENTS]" as a placeholder for the table of contents has
>>>  been removed.  Though, a new keyword, =TOC=, achieves the same
>>>  effect, and can even take more values, like "contents", "tables",
>>>  "figures" and "listings".  Tools are provided to make them available
>>>  to any exporter to come.
>> 
>> What do you mean by "keyword"?  Can you provide an example of how
>> to place the TOC?
> 
> Just put "#+toc: headlines", "#+toc: tables", "#+toc: listings", "#+toc:
> figures" anywhere in the document (at least with the LaTeX back-end).

OK, thanks.

> 
>>> 6. Macros have been under powered a little.  They cannot live anymore in
>>>  comments, example blocks, or even src blocks.  In fact, one cannot
>>>  find a macro where the text isn't parsed.  Macros are Org syntax.
>>>  Using such syntax where there is, by definition, none is just
>>>  nonsensical.
>> 
>> I know that Stefan Vollmar is using macros in complicated and
>> extensive ways.  I also know that he is using the index facilities,
>> which so far have depended heavily on the preprocessor.  I am curious
>> how indexing will work with org-elements.  Have you put any thought
>> into this?
> 
> I don't think that macros will be a source of problems since comments
> and example blocks were weird locations for them anyway.
> 
> In the LaTeX exporter, "#+index: something" will be transcoded into
> "\index{something}". That's about it.
> 
> Should the generic export build a list of all "#+index:" values and
> store it in a `:index' property (accessible through the communication
> channel)?

Yes, I think so!

> 
>>> 9. Table.el tables will always use their own export back-end.  In other
>>>  words, no Org syntax will be recognized in such table anymore.
>>>  A table.el is an extraneous element while the parser is meant to
>>>  parse Org syntax.
> 
>> Maybe we should see if there is a hook in table.el which could
>> be called to format text in a backend specific way.  If it is not
>> there, maybe we can simply introduce it ourselves, or add some
>> advice for this purpose....
> 
> Ok. If anyone can look at it and determine the right thing to do, I will
> merge it into the exporter code.

I will try to look at this possibility.

Cheers

- Carsten

> 
>>> 11. =org-export-with-TeX-macros= has been replaced with the more
>>>   appropriate =org-export-with-entities=, and the associated =OPTIONS=
>>>   keyword's symbol changed from =TeX:nil= to =e:nil=.
>> 
>> I do like the change, but maybe it would be good to support TeX
>> for backward compatibility...  Then, maybe, we are breaking a few
>> things anyway.
> 
> TeX is still supported as a "latex-fragment". In the long run, though,
> I think TeX commands that are not meant for dvipng/mathjax should be
> called through an export snippet (that is "@l{...}" temporary syntax).
> 
>>> 12. About variables changes, =org-export-author-info=,
>>>   =org-export-creator-info= and =org-export-email-info= have been
>>>   replaced with, respectively, =org-export-with-author=,
>>>   =org-export-with-creator= ad =org-export-with-email=, for the sake
>>>   of consistency with other opt-in variables.
>> 
>> Are you adding defvaralias for compatibility, or are you arguing for
>> a clean break here?
> 
> There is no defvaralias. Though, I don't mind adding them.
> 
> 
> Regards,
> 
> -- 
> Nicolas Goaziou

- Carsten

Re: [ANN] Org Export in contrib

[Nicolas Goaziou]

Carsten Dominik <carsten.dominik@gmail.com> writes:

>>>> 9. Table.el tables will always use their own export back-end.  In other
>>>>  words, no Org syntax will be recognized in such table anymore.
>>>>  A table.el is an extraneous element while the parser is meant to
>>>>  parse Org syntax.
>> 
>>> Maybe we should see if there is a hook in table.el which could
>>> be called to format text in a backend specific way.  If it is not
>>> there, maybe we can simply introduce it ourselves, or add some
>>> advice for this purpose....
>> 
>> Ok. If anyone can look at it and determine the right thing to do, I will
>> merge it into the exporter code.
>
> I will try to look at this possibility.

Also, note that tables come with the `:type' property, which tells
whether it is a table-el or not (it is either `org' or `table.el').

Thus, back-ends can already implement special actions when type is
`table.el'.

Isn't is sufficient?


Regards,

-- 
Nicolas Goaziou

Re: [ANN] Org Export in contrib

[Carsten Dominik]

Hi everyone,

is there anyone who is planning to implement a texinfo
exporter based on org-elements?  If not, I would write
this exporter...

- Carsten

On 25.11.2011, at 18:32, Nicolas Goaziou wrote:

> Hello,
> 
> I've pushed org-export.el to contrib. It's a general export engine,
> built on top of org-elements aiming at simplifying life of both
> developers and maintainers (and, therefore, of end-users).
> 
> It predates org-exp.el and org-special-blocks.el. Though, keep it mind
> that, as advanced as it is, it isn't yet a drop-in replacement for
> them. It still lacks an interface (à la `org-export'), back-ends, and
> tons of testing and improvements. That being said, it's usable anyway
> and one can already write back-ends for it. I'll show a silly example
> later in this mail.
> 
> Now, let's have a peek into the guts of that beast.
> 
> Besides the parser, the generic exporter is made of three distinct
> parts:
> 
> - The communication channel consists in a property list, which is
>  created and updated during the process.  Its use is to offer every
>  piece of information, would it be export options or contextual data,
>  all in a single place.  The exhaustive list of properties is given in
>  "The Communication Channel" section of the file.
> 
> - The transcoder walks the parse tree, ignores or treat as plain text
>  elements and objects according to export options, and eventually calls
>  back-end specific functions to do the real transcoding, concatenating
>  their return value along the way.
> 
> - The filter system is activated at the very beginning and the very end
>  of the export process, and each time an element or an object has been
>  converted.  It is the entry point to fine-tune standard output from
>  back-end transcoders.
> 
> The core function is `org-export-as'.  It returns the transcoded buffer
> as a string.
> 
> In order to derive an exporter out of this generic implementation, one
> can define a transcode function for each element or object.  Such
> function should return a string for the corresponding element, without
> any trailing space, or nil.  It must accept three arguments:
> 
>  1. the element or object itself,
>  2. its contents, or nil when it isn't recursive,
>  3. the property list used as a communication channel.
> 
> If no such function is found, that element or object type will simply be
> ignored, along with any separating blank line.  The same will happen if
> the function returns the nil value.  If that function returns the empty
> string, the type will be ignored, but the blank lines will be kept.
> 
> Contents, when not nil, are stripped from any global indentation
> (although the relative one is preserved).  They also always end with
> a single newline character.
> 
> These functions must follow a strict naming convention:
> `org-BACKEND-TYPE' where, obviously, BACKEND is the name of the export
> back-end and TYPE the type of the element or object handled.
> 
> Moreover, two additional functions can be defined.  On the one hand,
> `org-BACKEND-template' returns the final transcoded string, and can be
> used to add a preamble and a postamble to document's body.  It must
> accept two arguments: the transcoded string and the property list
> containing export options.  On the other hand, `org-BACKEND-plain-text',
> when defined, is to be called on every text not recognized as an element
> or an object.  It must accept two arguments: the text string and the
> information channel.
> 
> Any back-end can define its own variables.  Among them, those
> customizables should belong to the `org-export-BACKEND' group.  Also,
> a special variable, `org-BACKEND-option-alist', allows to define buffer
> keywords and "#+options:" items specific to that back-end.  See
> `org-export-option-alist' for supported defaults and syntax.
> 
> Tools for common tasks across back-ends are implemented in the last
> part of the file.
> 
> 
> * For Maintainers
> 
>  To word it differently, this exporter doesn't rely on any
>  text-property during the process. Thus, it makes
>  `org-if-unprotected-at' and alike obsolete in the whole code base. Org
>  core doesn't have to bother anymore about its exporter weaknesses.
> 
>  Also, buffer's pre-processing is reduced to its strict minimum: Babel
>  code expansion. No footnote normalization, list markers to add and
>  remove...
> 
>  Being only a beefed-up parse tree reader, any element or object added
>  to Elements is available through the exporter with no further
>  modification. Back-end just have to create the appropriate new
>  transcoders, unless that element or object should be ignored anyway.
> 
> 
> * For Developers
> 
>  All data needed is available in two places: the properties associated
>  to the element being transcoded, through the use of
>  `org-element-get-property', and the communication channel, with the
>  help of `plist-get'. Period.
> 
>  Also, the exporter takes care about all the filtering required by
>  options, and enforces the same number of blank lines in the Org buffer
>  and in the source code (though this can be overcome with the use of
>  filters). It's easier this way to concentrate on the shape of the
>  output.
> 
>  Tools for common tasks (like building table of contents or listings,
>  or numbering headlines) are provided in the library.
> 
> 
> * For Users
> 
>  Hooks are gone. Sorry. Most of them happened during a pre-process part
>  that doesn't exist anymore.
> 
>  Though, there are three way to configure the output, in increasing
>  power:
> 
>  1. Variables (customizable or not) are still there, provided by either
>     the exporter itself or its back-ends.
> 
>  2. Filter sets are provided to fine-tune output of any
>     back-end. A filter set is a list of functions, applied in a FIFO
>     order, whose signature is the resulting string of the previous
>     function (or the back-end output for the first one) and the
>     back-end as a symbol. The return value of the last function
>     replaces back-end's output. If one of the return values is nil, the
>     element or object on which the filters are applied is ignored in
>     the final output.
> 
>     Also, three special filter sets apply on the parse tree, on plain
>     text, and on the final output.
> 
>     For example, the LaTeX back-end has the bad habit to "boldify"
>     deadline, scheduled and closed strings close to time-stamps in the
>     buffer. I'd rather have them emphasized. Obviously, I don't want to
>     annoy other back-ends with this. The following will do the trick.
> 
>     #+begin_src emacs-lisp
>     (add-to-list 'org-export-filter-time-stamp-functions
>                  (lambda (ts backend)
>                    (if (not (eq backend 'latex))
>                        ts
>                      (replace-regexp-in-string "textbf" "emph" ts))))
>     #+end_src
> 
>  3. Whole parts of any back-end can be redefined (or advised).  For
>     example, if I don't like the way the LaTeX back-end transcodes
>     verbatim text, I can always create an `org-latex-verbatim' function
>     of my own.
> 
> 
> * A (silly) Back-End: `summary'
> 
>  I want a back-end, `summary', which only exports headlines of the
>  current buffer, in a markdown format. I would like to have the
>  opportunity to add a few lines of text before the first headline.  It
>  should also delimit beginning and end of output by ASCII scissors. Oh,
>  and scissors string should be customizable!
> 
>  As I only want headlines, I only need to implement an
>  `org-summary-headline' function. Though, text before the first
>  headline in my buffer will be ignored (it isn't an headline).
> 
>  So this back-end will have to define its own buffer keyword:
>  "#+PREAMBLE:". I need to be able to encounter this keyword more than
>  once in the buffer as my preamble will probably span on more than one
>  line. The following snippet will do this, and provide the text as the
>  value of the `:preamble' property in the communication channel. It
>  also creates a customizable `org-summary-scissors' variable, which is
>  rightfully added to the `org-export-summary' group.
> 
>  #+begin_src emacs-lisp
>  (defcustom org-summary-scissors "--%<--------------------------------\n"
>    "String used as a delimiter for summary output.
>  It should end with a newline character."
>    :group 'org-export-summary
>    :type 'string)
>  (defvar org-summary-option-alist)
>  (add-to-list 'org-summary-option-alist
>               '(:preamble "PREAMBLE" nil nil newline))
>  #+end_src
> 
>  Now onto the headline transcoder. A quick look at the
>  `org-element-headline-parser' function tell me that `:raw-value'
>  property should be enough, as I need no fancy transformation. I might
>  want to also use `:level' to get the number of "equal" signs before
>  the text, but a longer look at the list of properties offered in the
>  communication channel tells me that `org-export-get-relative-level'
>  may be more adequate. So be it.
> 
>  #+begin_src emacs-lisp
>  (defun org-summary-headline (headline contents info)
>    "Return HEADLINE in a Markdown syntax.
>  CONTENTS is the contents of the headline.  INFO is the property
>  list used as a communication channel."
>    (let ((title (org-element-get-property :raw-value headline))
>          (pre-blank (org-element-get-property :pre-blank headline))
>          (level (org-export-get-relative-level headline info))
>          ;; Depth of 6 is a hard limit in HTML (and therefore Markdown)
>          ;; syntax.
>          (limit (min (plist-get info :headline-levels) 6)))
>      (when (<= level limit)
>        (concat (make-string level ?=) " " title
>                (make-string (1+ pre-blank) ?\n)
>                contents))))
>  #+end_src
> 
>  This should be sufficient to take care of document's body. Now, I only
>  need to add the scissors, the preamble, and the title in the final
>  output. This all happens with the help of the `org-summary-template'
>  function.
> 
>  I remember that "#+TITLE:" belongs to `org-element-parsed-keywords',
>  which means that its value isn't a string but a secondary string. As
>  I don't want to transcode it (my back-end only knows about headline),
>  I'll get the raw value back with `org-element-interpret-secondary'
>  function (If I had wanted to transcode it, I would have used
>  `org-export-secondary-string' instead).
> 
>  #+begin_src emacs-lisp
>  (defun org-summary-template (contents info)
>    "Return complete document transcoded with summary back-end.
>  CONTENTS is the body of the document.  INFO is the plist used as
>  a communication channel."
>    (let ((title (org-element-interpret-secondary (plist-get info :title)))
>          (preamble (plist-get info :preamble)))
>      (concat org-summary-scissors
>              (upcase title) "\n\n"
>              preamble "\n\n"
>              contents
>              org-summary-scissors)))
>  #+end_src
> 
>  Now, I can test all of this by with M-: (org-export-as 'summary) on
>  a test buffer[1]. So far, so good. But I know better and define an
>  interactive function for that important action. While I'm at it, it
>  will display my summary in a buffer named "*Summary*".
> 
>  #+begin_src emacs-lisp
>  (defun org-export-as-summary ()
>    "Create the summary of the current Org buffer.
>  Summary is displayed in a buffer called \"*Summary*\"."
>    (interactive)
>    (when (eq major-mode 'org-mode)
>      (switch-to-buffer (org-export-to-buffer 'summary "*Summary*"))))
>  #+end_src
> 
>  That's all, folks.
> 
> 
> I'll try to package its first back-end, org-latex.el, into experimental/
> before monday.
> 
> Feedback, as always, is welcome.
> 
> 
> Some text that will probably be ignored.
> 
> * Head 1
> 
>  some text
> 
> ** Head 1.1
> 
>   some text too
> 
> *** Head 1.1.1
> 
>    Some text again
> 
> ** Head 1.2
> 
>   some text
> 
> * Head 2                                                                :noexport:
> 
>  some text
> --8<---------------cut here---------------end--------------->8---
> 
> 
> Regards,
> 
> [1] For example, this one:
> --8<---------------cut here---------------start------------->8---
> #+Title: My Summary Test
> #+Preamble: I hope
> #+Preamble: that it is working
> #+Options: H:2
> 
> -- 
> Nicolas Goaziou
> 

Re: [ANN] Org Export in contrib

[Robert Klein]

On 11/25/2011 06:32 PM, Nicolas Goaziou wrote:
> Hello,
>
> I've pushed org-export.el to contrib. It's a general export engine,
> built on top of org-elements aiming at simplifying life of both
> developers and maintainers (and, therefore, of end-users).
>

Hi Nicolas,

I'd like to make same remarks regarding the Wikipedia exporter included in
the generic exporter:

Is the exporter supposed to export a whole wiki or a sub-wiki from a
org-file or a single wiki page?

If it is to be only a single wiki page, could you change the
variables :body-section-header-prefix and :body-section-header-suffix to
begin with "==" instead of "="?  The level 1 headings are supposed to be
only used for the page title
(cf. http://www.mediawiki.org/wiki/Help:Formatting and the note on the
talks page http://www.mediawiki.org/wiki/Help_talk:Formatting#Level_1).


Second, when I format my org file like this:

* heading
   some text. some text. some text.

the text currently gets exported with the leading blanks.  This causes the
Mediawiki to render the text as preformatted.  Is there a way around this?

Thanks you very much.

Best regards
Robert

Re: [ANN] Org Export in contrib

[Robert Klein]

Umm, sorry,

I just found out, I was talking about the other generic exporter :(

Robert

On 11/29/2011 07:15 AM, Robert Klein wrote:
> On 11/25/2011 06:32 PM, Nicolas Goaziou wrote:
>> Hello,
>>
>> I've pushed org-export.el to contrib. It's a general export engine,
>> built on top of org-elements aiming at simplifying life of both
>> developers and maintainers (and, therefore, of end-users).
>>
>
> Hi Nicolas,
>
> I'd like to make same remarks regarding the Wikipedia exporter 
> included in
> the generic exporter:
>
> Is the exporter supposed to export a whole wiki or a sub-wiki from a
> org-file or a single wiki page?
>
> If it is to be only a single wiki page, could you change the
> variables :body-section-header-prefix and :body-section-header-suffix to
> begin with "==" instead of "="?  The level 1 headings are supposed to be
> only used for the page title
> (cf. http://www.mediawiki.org/wiki/Help:Formatting and the note on the
> talks page http://www.mediawiki.org/wiki/Help_talk:Formatting#Level_1).
>
>
> Second, when I format my org file like this:
>
> * heading
>   some text. some text. some text.
>
> the text currently gets exported with the leading blanks.  This causes 
> the
> Mediawiki to render the text as preformatted.  Is there a way around 
> this?
>
> Thanks you very much.
>
> Best regards
> Robert
>