Source block processing changes

Source block processing changes

[Ista Zahn]

Hi all,

I've been silently suffering since the changes to source block
processing announced at the end of September[1]. I had been using
(abusing?) ':results org' to write results that would be processed as
orgmode syntax and exported to both latex and html. For example, this

#+name: tst
#+begin_src R :results output org :exports results
  library(ascii)
  cat("* First ten\n")
  print(ascii(head(mtcars)), type="org")
  cat("\n* Last ten\n")
  print(ascii(tail(mtcars)), type="org")
#+end_src

used to give me heading "First ten" followed by a table nicely
formatted in html or latex depending on the export target, followed by
a heading "Last ten", again followed by a nicely formatted table.
Changing the data or the code in the block would update the contents.
This was very convenient, and I relied heavily on this to generate
reports.

Now it seems that this very useful feature no longer exists, and try
as I might I cannot find a way to duplicate it. Using ':results
drawer' or ':results org' gives me verbatim export (i.e., * First ten
is not converted to a headline, but displayed in highlighted syntax
similar to the way it appears in the buffer. The only way I can get
the headlines and tables to be exported properly is to set ':results
raw', but then I get duplicate results every time I evaluate the R
source block.

I understand from the original announcement that "#+begin org / #+end
org" blocks don't really make sense because these kind of blocks
correspond to backends, and there is no org backend, but it was very
useful. Is there any way to produce the old behavior in the current
development version of org? If not, what are the chances of this very
useful functionality being re-implemented?

Thanks,
Ista


[1] http://lists.gnu.org/archive/html/emacs-orgmode/2012-09/msg01230.html

Re: Source block processing changes

[Nicolas Goaziou]

Hello,

Ista Zahn <istazahn@gmail.com> writes:

> The only way I can get the headlines and tables to be exported
> properly is to set ':results raw', but then I get duplicate results
> every time I evaluate the R source block.

You can use ":cache yes" in order to avoid duplicating results.

> Is there any way to produce the old behavior in the current
> development version of org?

Besides using :results raw? You may also generate a file containing your
results and include it in the buffer.

> If not, what are the chances of this very useful functionality being
> re-implemented?

I'd like to avoid re-implementing this hack, if possible. But if it had
to be done, I think it would require to treat specially "begin_org"
blocks during export by replacing them with their contents just before
parsing (i.e. just after included files have been expanded).

Hence,

    #+begin_src org
    ,* Headline
    #+end_src

would become

    * Headline

just before parser kicks in.

But, again, let's see first if no solution can be found without creating
yet another special case.


Regards,

-- 
Nicolas Goaziou

Re: Source block processing changes

[Ista Zahn]

Thank you Nicolas for your quick response!

On Mon, Nov 5, 2012 at 12:25 PM, Nicolas Goaziou <n.goaziou@gmail.com> wrote:
> Hello,
>
> Ista Zahn <istazahn@gmail.com> writes:
>
>> The only way I can get the headlines and tables to be exported
>> properly is to set ':results raw', but then I get duplicate results
>> every time I evaluate the R source block.
>
> You can use ":cache yes" in order to avoid duplicating results.

I gave up on using ':cache yes' a long time ago -- the problem is that
results don't update when the input data changes, as I describe here:
http://lists.gnu.org/archive/html/emacs-orgmode/2010-09/msg01152.html

>
>> Is there any way to produce the old behavior in the current
>> development version of org?
>
> Besides using :results raw?

Yes, so that evaluating the result multiple times does not produce
duplicate output, while re-evaluating if the input data changes.

You may also generate a file containing your
> results and include it in the buffer.

This does work, but has the drawback that I can't see the output in my
main org buffer. This is one of the things that drew me away from
Sweave/Knitr, i.e., I could see both the code and the results in the
same buffer, without having to compile the document or switch to
another buffer.

>
>> If not, what are the chances of this very useful functionality being
>> re-implemented?
>
> I'd like to avoid re-implementing this hack, if possible. But if it had
> to be done, I think it would require to treat specially "begin_org"
> blocks during export by replacing them with their contents just before
> parsing (i.e. just after included files have been expanded).
>
> Hence,
>
>     #+begin_src org
>     ,* Headline
>     #+end_src
>
> would become
>
>     * Headline
>
> just before parser kicks in.
>
> But, again, let's see first if no solution can be found without creating
> yet another special case.

Yes, of course. Thanks again for considering my use-case!

Best,
Ista

>
>
> Regards,
>
> --
> Nicolas Goaziou

Re: Source block processing changes

[Sebastien Vauban]

Hi Ista and Nicolas,

Ista Zahn wrote:
> On Mon, Nov 5, 2012 at 12:25 PM, Nicolas Goaziou <n.goaziou-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>> Ista Zahn <istazahn-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> writes:
>>
>>> The only way I can get the headlines and tables to be exported
>>> properly is to set ':results raw', but then I get duplicate results
>>> every time I evaluate the R source block.
>>
>> You can use ":cache yes" in order to avoid duplicating results.
>
> I gave up on using ':cache yes' a long time ago -- the problem is that
> results don't update when the input data changes, as I describe here:
> http://lists.gnu.org/archive/html/emacs-orgmode/2010-09/msg01152.html

That's normal with raw results: as I wrote it in
http://lists.gnu.org/archive/html/emacs-orgmode/2012-09/msg01273.html, as
there are *no obvious markers to delimit the results* in the Org mode file,
there is no way to know where raw results begin or end: raw results *cannot be
removed* [NOR UPDATED] from the buffer.

>>> Is there any way to produce the old behavior in the current
>>> development version of org?
>>
>> Besides using :results raw?

I share Ista's point of view in thinking that we've lost a useful
functionality, but I've not been blocked these days because of that, and
(because of lack of time) I really couldn't yet come up with a case showing it
more clearly.

Best regards,
  Seb

-- 
Sebastien Vauban

Re: Source block processing changes

[Nicolas Goaziou]

Hello,

"Sebastien Vauban"
<wxhgmqzgwmuf-geNee64TY+gS+FvcfC7Uqw@public.gmane.org> writes:

> That's normal with raw results: as I wrote it in
> http://lists.gnu.org/archive/html/emacs-orgmode/2012-09/msg01273.html, as
> there are *no obvious markers to delimit the results* in the Org mode file,
> there is no way to know where raw results begin or end: raw results *cannot be
> removed* [NOR UPDATED] from the buffer.

The sole purpose of raw results is to allow inserting an headline (I
mean a real headline, not comma protected) in the buffer, because
headlines cannot be contained in anything else than headlines.

If the generated code doesn't contain an headline, you don't need raw
results.


Regards,

-- 
Nicolas Goaziou

Re: Source block processing changes

[Achim Gratz]

Sebastien Vauban writes:
> That's normal with raw results: as I wrote it in
> http://lists.gnu.org/archive/html/emacs-orgmode/2012-09/msg01273.html, as
> there are *no obvious markers to delimit the results* in the Org mode file,
> there is no way to know where raw results begin or end: raw results *cannot be
> removed* [NOR UPDATED] from the buffer.

So, it seems that there should be a way to specify a delimiter for this
purpose that is interpreted as a comment line on export.  How about
":results delimited" ?


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptations for KORG EX-800 and Poly-800MkII V0.9:
http://Synth.Stromeko.net/Downloads.html#KorgSDada

Re: Source block processing changes

[Eric Schulte]

Ista Zahn <istazahn@gmail.com> writes:

> Hi all,
>
> I've been silently suffering since the changes to source block
> processing announced at the end of September[1]. I had been using
> (abusing?) ':results org' to write results that would be processed as
> orgmode syntax and exported to both latex and html. For example, this
>
> #+name: tst
> #+begin_src R :results output org :exports results
>   library(ascii)
>   cat("* First ten\n")
>   print(ascii(head(mtcars)), type="org")
>   cat("\n* Last ten\n")
>   print(ascii(tail(mtcars)), type="org")
> #+end_src
>
> used to give me heading "First ten" followed by a table nicely
> formatted in html or latex depending on the export target, followed by
> a heading "Last ten", again followed by a nicely formatted table.
> Changing the data or the code in the block would update the contents.
> This was very convenient, and I relied heavily on this to generate
> reports.
>
> Now it seems that this very useful feature no longer exists, and try
> as I might I cannot find a way to duplicate it. Using ':results
> drawer' or ':results org' gives me verbatim export

This was a bug in ":results drawer", which should *not* escape its
results.  I've just pushed up a fix for this so ":results drawer" should
now be a drop-in replacement for your previous use-case of ":results
org" (as was the original intent of this change).

Best,

-- 
Eric Schulte
http://cs.unm.edu/~eschulte

Re: Source block processing changes

[Ista Zahn]

On Tue, Nov 6, 2012 at 8:22 AM, Eric Schulte <schulte.eric@gmail.com> wrote:
> Ista Zahn <istazahn@gmail.com> writes:
>
>> Hi all,
>>
>> I've been silently suffering since the changes to source block
>> processing announced at the end of September[1]. I had been using
>> (abusing?) ':results org' to write results that would be processed as
>> orgmode syntax and exported to both latex and html. For example, this
>>
>> #+name: tst
>> #+begin_src R :results output org :exports results
>>   library(ascii)
>>   cat("* First ten\n")
>>   print(ascii(head(mtcars)), type="org")
>>   cat("\n* Last ten\n")
>>   print(ascii(tail(mtcars)), type="org")
>> #+end_src
>>
>> used to give me heading "First ten" followed by a table nicely
>> formatted in html or latex depending on the export target, followed by
>> a heading "Last ten", again followed by a nicely formatted table.
>> Changing the data or the code in the block would update the contents.
>> This was very convenient, and I relied heavily on this to generate
>> reports.
>>
>> Now it seems that this very useful feature no longer exists, and try
>> as I might I cannot find a way to duplicate it. Using ':results
>> drawer' or ':results org' gives me verbatim export
>
> This was a bug in ":results drawer", which should *not* escape its
> results.  I've just pushed up a fix for this so ":results drawer" should
> now be a drop-in replacement for your previous use-case of ":results
> org" (as was the original intent of this change).

Thank you, all is working as desired now! One small niggle: the
'::RESULTS:' and ':END:' strings are included in the exported html.
Not sure if this is intentional, but it seems to me that they should
be removed (or at lease given a class so we can hide them via css)

Thanks again!
Ista
>
> Best,
>
> --
> Eric Schulte
> http://cs.unm.edu/~eschulte

Re: Source block processing changes

[Nicolas Goaziou]

Ista Zahn <istazahn@gmail.com> writes:

> Thank you, all is working as desired now! One small niggle: the
> '::RESULTS:' and ':END:' strings are included in the exported html.
> Not sure if this is intentional, but it seems to me that they should
> be removed (or at lease given a class so we can hide them via css)

That's because you're inserting a headline within a drawer, which isn't
valid Org syntax. You cannot use a drawer for your specific use-case.

Regards,

Re: Source block processing changes

[Ista Zahn]

On Tue, Nov 6, 2012 at 11:44 AM, Nicolas Goaziou <n.goaziou@gmail.com> wrote:
> Ista Zahn <istazahn@gmail.com> writes:
>
>> Thank you, all is working as desired now! One small niggle: the
>> '::RESULTS:' and ':END:' strings are included in the exported html.
>> Not sure if this is intentional, but it seems to me that they should
>> be removed (or at lease given a class so we can hide them via css)
>
> That's because you're inserting a headline within a drawer, which isn't
> valid Org syntax. You cannot use a drawer for your specific use-case.

Well, it works at the moment (I can live with the extra ':RESULTS:'
strings). So currently I _can_ use a drawer for my use case -- the
question is whether this is likely to break in the future.

Another question is what is and is not allowed in a drawer? Is this documented?

Best,
Ista
>
> Regards,

Re: Source block processing changes

[Nicolas Goaziou]

Ista Zahn <istazahn@gmail.com> writes:

> On Tue, Nov 6, 2012 at 11:44 AM, Nicolas Goaziou <n.goaziou@gmail.com> wrote:
>> Ista Zahn <istazahn@gmail.com> writes:
>>
>>> Thank you, all is working as desired now! One small niggle: the
>>> '::RESULTS:' and ':END:' strings are included in the exported html.
>>> Not sure if this is intentional, but it seems to me that they should
>>> be removed (or at lease given a class so we can hide them via css)
>>
>> That's because you're inserting a headline within a drawer, which isn't
>> valid Org syntax. You cannot use a drawer for your specific use-case.
>
> Well, it works at the moment (I can live with the extra ':RESULTS:'
> strings). So currently I _can_ use a drawer for my use case -- the
> question is whether this is likely to break in the future.

This is already broken. In Org syntax,

  :RESULTS:
  * Headline
  :END:

means a paragraph containing ":RESULTS:" text, then an headline
containing a section with a paragraph containing ":END:" text. In other
words, headlines have precedence over drawers.

Expecting Org to recognize a drawer here is like playing with fire. You
cannot know when it will burn you.

> Another question is what is and is not allowed in a drawer? Is this
> documented?

A drawer can contain anything but a headline and another drawer. I'm not
talking about properties drawer which is special. This is not explicitly
documented.


Regards,

Re: Source block processing changes

[Ista Zahn]

On Tue, Nov 6, 2012 at 12:16 PM, Nicolas Goaziou <n.goaziou@gmail.com> wrote:
> Ista Zahn <istazahn@gmail.com> writes:
>
>> On Tue, Nov 6, 2012 at 11:44 AM, Nicolas Goaziou <n.goaziou@gmail.com> wrote:
>>> Ista Zahn <istazahn@gmail.com> writes:
>>>
>>>> Thank you, all is working as desired now! One small niggle: the
>>>> '::RESULTS:' and ':END:' strings are included in the exported html.
>>>> Not sure if this is intentional, but it seems to me that they should
>>>> be removed (or at lease given a class so we can hide them via css)
>>>
>>> That's because you're inserting a headline within a drawer, which isn't
>>> valid Org syntax. You cannot use a drawer for your specific use-case.
>>
>> Well, it works at the moment (I can live with the extra ':RESULTS:'
>> strings). So currently I _can_ use a drawer for my use case -- the
>> question is whether this is likely to break in the future.
>
> This is already broken. In Org syntax,
>
>   :RESULTS:
>   * Headline
>   :END:
>
> means a paragraph containing ":RESULTS:" text, then an headline
> containing a section with a paragraph containing ":END:" text. In other
> words, headlines have precedence over drawers.
>
> Expecting Org to recognize a drawer here is like playing with fire. You
> cannot know when it will burn you.
>
>> Another question is what is and is not allowed in a drawer? Is this
>> documented?
>
> A drawer can contain anything but a headline and another drawer.

OK, great. The ability to generate headlines from source blocks is not
crucial, I'll change them to plain lists.

 I'm not
> talking about properties drawer which is special. This is not explicitly
> documented.
>
>
> Regards,

Re: Source block processing changes

[Ista Zahn]

On Tue, Nov 6, 2012 at 8:22 AM, Eric Schulte <schulte.eric@gmail.com> wrote:
> Ista Zahn <istazahn@gmail.com> writes:

>> Using ':results
>> drawer' or ':results org' gives me verbatim export
>
> This was a bug in ":results drawer", which should *not* escape its
> results.  I've just pushed up a fix for this so ":results drawer" should
> now be a drop-in replacement for your previous use-case of ":results
> org" (as was the original intent of this change).

I'm still getting escaped results with ':results drawer', unless I
generate a headline, which Nicolas tells me is illegal. Can you take
another look at this?

Thanks!
Ista

>
> Best,
>
> --
> Eric Schulte
> http://cs.unm.edu/~eschulte

Re: Source block processing changes

[Ista Zahn]

On Tue, Nov 6, 2012 at 12:48 PM, Ista Zahn <istazahn@gmail.com> wrote:
> On Tue, Nov 6, 2012 at 8:22 AM, Eric Schulte <schulte.eric@gmail.com> wrote:
>> Ista Zahn <istazahn@gmail.com> writes:
>
>>> Using ':results
>>> drawer' or ':results org' gives me verbatim export
>>
>> This was a bug in ":results drawer", which should *not* escape its
>> results.  I've just pushed up a fix for this so ":results drawer" should
>> now be a drop-in replacement for your previous use-case of ":results
>> org" (as was the original intent of this change).
>
> I'm still getting escaped results with ':results drawer', unless I
> generate a headline, which Nicolas tells me is illegal. Can you take
> another look at this?

Clarification: ':results drawer' exports correctly using the new
exporter (accessed through M-x org-export-dispatch). However, when
exporting using the old exporter (C-c-e) the results drawer is escaped
so that it is formatted as an example or source block. For example a
buffer with this content


#+name: setup
#+begin_src emacs-lisp :results drawer :exports both
  (message "hello world")
#+end_src

#+RESULTS: setup
:RESULTS:
hello world
:END:


exported to html with 'C-c-e h' gives me 'hello world' in a box with
boarders and a gray rectangle. Exporting to html with 'M-x
org-export-dispatch h h' gives me "hello world" as regular text.

I guess this is the push I need to start using the new exporter!

Best,
Ista

>
> Thanks!
> Ista
>
>>
>> Best,
>>
>> --
>> Eric Schulte
>> http://cs.unm.edu/~eschulte