[RFC] new :post header argument for post-processing of code block results

[RFC] new :post header argument for post-processing of code block results

[Eric Schulte]

[-- Attachment #1: Type: text/plain, Size: 514 bytes --]

Hi,

I've been wanting to add the ability to post-process the results of a
code block for some time, and some recent threads (e.g., [1] and [2])
could both have benefited from post-processing of code block output.

The attached patch [3] adds a new :post header argument, which may be
used to specify a code block reference to use to post-process the
results of the current code block.  The following example illustrates
how this header argument could be used to add attr_latex lines to table
code block results.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: example.org --]
[-- Type: text/x-org, Size: 1640 bytes --]

#+Title: Proposed =post= Header Argument
#+Author: Eric Schulte

* a code block which does the post processing
This code block adds latex width attributes to whatever =data= is
passed to it.
#+name: add-attr
#+begin_src sh :var data="" :results output drawer :var width="\textwidth"
  echo "#+attr_latex: width=$width"
  echo "$data"
#+end_src

* two code blocks which use the post processing
The following two code blocks demonstrate the use of the new =post=
header argument, whcih we could add if it seems generally useful and
not too confusing.

This works by lexically binding the =*this*= variable to the results
of the current code block, and then calling the =add-attr= code block
to post-process these results.

#+begin_src sh :results output wrap :post add-attr(width="5cm",data=(identity *this*))
cat <<EOF
| 1 | 2 |
| 3 | 4 |
EOF
#+end_src

#+RESULTS:
:RESULTS:
#+attr_latex: width=5cm
| 1 | 2 |
| 3 | 4 |
:END:

This code block does the same thing as the previous one, but using
different values and arguments.

#+header: :post add-attr(width="0.5\\textwidth",data=(identity *this*))
#+begin_src sh :results output wrap
cat <<EOF
| 5 | 6 |
| 7 | 8 |
EOF
#+end_src

#+RESULTS:
:RESULTS:
#+attr_latex: width=0.5\textwidth
| 5 | 6 |
| 7 | 8 |
:END:

* another unrelated example
Say we only care about the size of our results, then we could use the
following.
#+name: wc
#+begin_src sh data=""
  echo "$data"|wc
#+end_src

This code block checks the size of some Emacs Lisp data (in this case
my =load-path= variable).
#+begin_src emacs-lisp :post wc(data=(identity *this*))
  load-path
#+end_src

#+RESULTS:
: 160     160    6783

[-- Attachment #3: Type: text/plain, Size: 274 bytes --]


Does this new header argument seem useful?  Any suggestions for better
syntax which don't add too much conceptual or code complexity?

Thanks,

Footnotes: 
[1]  http://thread.gmane.org/gmane.emacs.orgmode/69339

[2]  http://thread.gmane.org/gmane.emacs.orgmode/68700

[3]  

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #4: 0001-post-header-arg-post-processes-code-block-results.patch --]
[-- Type: text/x-patch, Size: 1407 bytes --]

From f83f31d82c0c660c74a2af7114d4e23c9b37c095 Mon Sep 17 00:00:00 2001
From: Eric Schulte <schulte.eric@gmail.com>
Date: Sun, 31 Mar 2013 19:02:11 -0600
Subject: [PATCH] :post header arg post-processes code block results

* lisp/ob-core.el (org-babel-common-header-args-w-values): Add :post to
  the list of header arguments.
  (org-babel-execute-src-block): Post process results when the :post
  header argument has been supplied.
---
 lisp/ob-core.el | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/lisp/ob-core.el b/lisp/ob-core.el
index 723aa9d..e1321eb 100644
--- a/lisp/ob-core.el
+++ b/lisp/ob-core.el
@@ -456,6 +456,7 @@ then run `org-babel-pop-to-session'."
     (noweb-ref	. :any)
     (noweb-sep  . :any)
     (padline	. ((yes no)))
+    (post       . :any)
     (results	. ((file list vector table scalar verbatim)
 		   (raw html latex org code pp drawer)
 		   (replace silent none append prepend)
@@ -625,6 +626,11 @@ block."
 				      (not (listp result)))
 				 (list (list result)) result))
 			   (funcall cmd body params)))
+		    ;; possibly perform post process provided its appropriate
+		    (when (cdr (assoc :post params))
+		      (let ((*this* result))
+			(setq result (org-babel-ref-resolve
+				      (cdr (assoc :post params))))))
 		    ;; if non-empty result and :file then write to :file
 		    (when (cdr (assoc :file params))
 		      (when result
-- 
1.8.2


[-- Attachment #5: Type: text/plain, Size: 47 bytes --]



-- 
Eric Schulte
http://cs.unm.edu/~eschulte

Re: [RFC] new :post header argument for post-processing of code block results

[Aaron Ecay]

Hi Eric

2013ko martxoak 31an, Eric Schulte-ek idatzi zuen:
> 
> Hi,
> 
> I've been wanting to add the ability to post-process the results of a
> code block for some time, and some recent threads (e.g., [1] and [2])
> could both have benefited from post-processing of code block output.

This looks very nice!

> 
> Does this new header argument seem useful?  Any suggestions for better
> syntax which don't add too much conceptual or code complexity?

See below.

> @@ -625,6 +626,11 @@ block."
>  				      (not (listp result)))
>  				 (list (list result)) result))
>  			   (funcall cmd body params)))
> +		    ;; possibly perform post process provided its appropriate
> +		    (when (cdr (assoc :post params))
> +		      (let ((*this* result))
> +			(setq result (org-babel-ref-resolve
> +				      (cdr (assoc :post params))))))

What if you did some string surgery on the :post string, to insert
",data=\"the result\"" into the call?  That way users could just write
:post add-width(width=5cm), which would be automatically transformed
into add-width(width=5cm,data="[[graph.png]]") before being passed to
o-b-ref-resolve.

(I guess you’d have to take special care to handle things like ":post
no-args()" and ":post no-args" properly, stripping the initial comma in
the first case and adding parens in the second.)

This requires that all :post code blocks take a data
argument, but I don’t think that’s more onerous than stipulating the
*this* variable at the lisp level.

Also, I’m unclear on whether elisp is supported (or should be).  Do we
want to allow ":post (message *this*)"?

-- 
Aaron Ecay

Re: [RFC] new :post header argument for post-processing of code block results

[Eric Schulte]

Aaron Ecay <aaronecay@gmail.com> writes:

> Hi Eric
>
> 2013ko martxoak 31an, Eric Schulte-ek idatzi zuen:
>> 
>> Hi,
>> 
>> I've been wanting to add the ability to post-process the results of a
>> code block for some time, and some recent threads (e.g., [1] and [2])
>> could both have benefited from post-processing of code block output.
>
> This looks very nice!
>
>> 
>> Does this new header argument seem useful?  Any suggestions for better
>> syntax which don't add too much conceptual or code complexity?
>
> See below.
>
>> @@ -625,6 +626,11 @@ block."
>>  				      (not (listp result)))
>>  				 (list (list result)) result))
>>  			   (funcall cmd body params)))
>> +		    ;; possibly perform post process provided its appropriate
>> +		    (when (cdr (assoc :post params))
>> +		      (let ((*this* result))
>> +			(setq result (org-babel-ref-resolve
>> +				      (cdr (assoc :post params))))))
>
> What if you did some string surgery on the :post string, to insert
> ",data=\"the result\"" into the call?  That way users could just write
> :post add-width(width=5cm), which would be automatically transformed
> into add-width(width=5cm,data="[[graph.png]]") before being passed to
> o-b-ref-resolve.
>

I don't like this idea, because then it becomes "magic" which value is
assigned the result of the current code block, rather than the current
case in which it is very explicit.

>
> (I guess you’d have to take special care to handle things like ":post
> no-args()" and ":post no-args" properly, stripping the initial comma in
> the first case and adding parens in the second.)
>
> This requires that all :post code blocks take a data
> argument, but I don’t think that’s more onerous than stipulating the
> *this* variable at the lisp level.
>

I think it is more onerous, I also think it reduces flexibility (the
writing of the called block needs to know exactly which argument will
want to be set by later code blocks).

>
> Also, I’m unclear on whether elisp is supported

Yes it is

> (or should be)

Yes again

> .  Do we want to allow ":post (message *this*)"?

Yes again

If your issue is that (identity *this*) is cumbersome, then I would
agree.  What about if we change `org-babel-read' as with the attached
patch s.t. *any* variable with ear-muffs will be read as Emacs Lisp,
allowing this simpler alternative.

    #+name: val-wrap
    #+begin_src sh :input="" :results verbatim
      echo "--------------------"
      echo "$input"
      echo "--------------------"
    #+end_src

    #+begin_src sh :post val-wrap(input=*this*)
      echo "foo"
    #+end_src

    #+RESULTS:
    : --------------------
    : foo
    : --------------------

Cheers,

-- 
Eric Schulte
http://cs.unm.edu/~eschulte

Re: [RFC] new :post header argument for post-processing of code block results

[Eric Schulte]

[-- Attachment #1: Type: text/plain, Size: 676 bytes --]

>
> If your issue is that (identity *this*) is cumbersome, then I would
> agree.  What about if we change `org-babel-read' as with the attached
> patch s.t. *any* variable with ear-muffs will be read as Emacs Lisp,
> allowing this simpler alternative.
>

The attached patches change to the more natural *this* syntax, make
:post processing work intuitively with the :file header argument, and
adds documentation.  I'm posting them as patches here, rather than
simply committing them, because I don't want to push in a change like
this at the 11th hour before the release of 8.0.

Bastien, please commit these whenever you want, or let me know and I can
commit them.

Cheers,


[-- Attachment #2: post-patches.tar.bz2 --]
[-- Type: application/x-bzip2, Size: 3091 bytes --]

[-- Attachment #3: Type: text/plain, Size: 46 bytes --]


-- 
Eric Schulte
http://cs.unm.edu/~eschulte

Re: [RFC] new :post header argument for post-processing of code block results

[Bastien]

Hi Eric,

Eric Schulte <schulte.eric@gmail.com> writes:

> Bastien, please commit these whenever you want, or let me know and I can
> commit them.

Please go ahead.

If you have one minute, you can clone orgweb.git and send a patch
against Changes.org:

  ~$ git clone orgweb@orgmode.org:orgweb.git

Otherwise I'll do it myself, with a small description of the new
feature.

Thanks,

-- 
 Bastien

Re: [RFC] new :post header argument for post-processing of code block results

[Eric Schulte]

Bastien <bzg@altern.org> writes:

> Hi Eric,
>
> Eric Schulte <schulte.eric@gmail.com> writes:
>
>> Bastien, please commit these whenever you want, or let me know and I can
>> commit them.
>
> Please go ahead.
>

Done.

>
> If you have one minute, you can clone orgweb.git and send a patch
> against Changes.org:
>
>   ~$ git clone orgweb@orgmode.org:orgweb.git
>
> Otherwise I'll do it myself, with a small description of the new
> feature.
>

Committed.

Cheers,

>
> Thanks,

-- 
Eric Schulte
http://cs.unm.edu/~eschulte

Re: [RFC] new :post header argument for post-processing of code block results

[Bastien]

Eric Schulte <schulte.eric@gmail.com> writes:

> Done.

Thanks!

-- 
 Bastien

Re: [RFC] new :post header argument for post-processing of code block results

[Eric S Fraga]

Eric Schulte <schulte.eric@gmail.com> writes:

> Hi,
>
> I've been wanting to add the ability to post-process the results of a
> code block for some time, and some recent threads (e.g., [1] and [2])
> could both have benefited from post-processing of code block output.

[...]

> Does this new header argument seem useful?  Any suggestions for better
> syntax which don't add too much conceptual or code complexity?

Very useful indeed!  I don't have a chance to try this out properly now
but I know of several previous org files where this would have been very
useful.

One question: can one have a sequence of forward chained blocks with
length > 2?

Thanks,
eric
-- 
: Eric S Fraga, GnuPG: 0xC89193D8FFFCF67D
: in Emacs 24.3.50.1 and Org release_7.9.3f-1199-g3a0e55

Re: [RFC] new :post header argument for post-processing of code block results

[Eric Schulte]

[-- Attachment #1: Type: text/plain, Size: 835 bytes --]

Eric S Fraga <e.fraga@ucl.ac.uk> writes:

> Eric Schulte <schulte.eric@gmail.com> writes:
>
>> Hi,
>>
>> I've been wanting to add the ability to post-process the results of a
>> code block for some time, and some recent threads (e.g., [1] and [2])
>> could both have benefited from post-processing of code block output.
>
> [...]
>
>> Does this new header argument seem useful?  Any suggestions for better
>> syntax which don't add too much conceptual or code complexity?
>
> Very useful indeed!  I don't have a chance to try this out properly now
> but I know of several previous org files where this would have been very
> useful.
>

Great, I'm happy I wasn't the only one.

>
> One question: can one have a sequence of forward chained blocks with
> length > 2?
>
> Thanks,
> eric

Yes, this is now just part of the :var machinery.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: post-chain.org --]
[-- Type: text/x-org, Size: 286 bytes --]

#+Title: :post header example

#+name: mult
#+begin_src emacs-lisp :var in=0
  (* 2 in)
#+end_src

#+name: add
#+begin_src emacs-lisp :var in=0
  (+ 1 in)
#+end_src

Putting the previous two together we get.
#+begin_src emacs-lisp :post mult(add(*this*))
  4
#+end_src

#+RESULTS:
: 10

[-- Attachment #3: Type: text/plain, Size: 46 bytes --]


-- 
Eric Schulte
http://cs.unm.edu/~eschulte

Re: [RFC] new :post header argument for post-processing of code block results

[Andreas Leha]

Eric Schulte <schulte.eric@gmail.com> writes:

> Eric S Fraga <e.fraga@ucl.ac.uk> writes:
>
>> Eric Schulte <schulte.eric@gmail.com> writes:
>>
>>> Hi,
>>>
>>> I've been wanting to add the ability to post-process the results of a
>>> code block for some time, and some recent threads (e.g., [1] and [2])
>>> could both have benefited from post-processing of code block output.
>>
>> [...]
>>
>>> Does this new header argument seem useful?  Any suggestions for better
>>> syntax which don't add too much conceptual or code complexity?
>>
>> Very useful indeed!  I don't have a chance to try this out properly now
>> but I know of several previous org files where this would have been very
>> useful.
>>
>
> Great, I'm happy I wasn't the only one.
>

No, you were not.  Adding lines to tables is one of the cases, I will
use that cool feature from now on.

Regards,
Andreas

Re: [RFC] new :post header argument for post-processing of code block results

[Christian Nybø]

Eric Schulte <schulte.eric <at> gmail.com> writes:

> 
> Aaron Ecay <aaronecay <at> gmail.com> writes:

> Yes again
> 
> > .  Do we want to allow ":post (message *this*)"?
> 
> Yes again

Eric, 

would you mind posting a sample that shows how to post-process the result
with emacs-lisp?

If I create a buffer test.org, add the following to the buffer, place the
cursor at the end of the line containing "date" and type C-c C-c, I get 
"Symbol's value as variable is void: *this*"

#+begin_src sh :post (message *this*)
date
#+end_src

Org-mode version 8.3beta (release_8.3beta-642-gb8d790 @
/home/chr/extsrc/org-mode/lisp/)

Regards,

Christian Nybø