org-babel order of evaluation

org-babel order of evaluation

[Rick Frankel]

There is a problem with the order of execution of interspersed source
and call blocks will not be executed in order because of the way
org-babel-execute-buffer is written (first all the source blocks, then
all the call blocks). 

Therefore, when executing an entire buffer, there is no way to have
the execution of a call block dependent on the prior execution of a
source block.

Given the following:

	#+name: one(v="one")
	#+begin_src elisp
	v
	#+end_src

	#+call: one("two")

	#+name: three(v="three")
	#+begin_src elisp
	v
	#+end_src

The message buffer shows:

	executing Elisp code block (one)...

	(v (quote "one"))

	Code block evaluation complete.
	executing Elisp code block (three)...

	(v (quote "three"))

	Code block evaluation complete.
	executing Elisp code block (one)...

	(v (quote "two"))

	"two"
	executing Emacs-Lisp code block...

	(results (quote "two"))

Re: org-babel order of evaluation

[Eric Schulte]

Rick Frankel <rick@rickster.com> writes:

> There is a problem with the order of execution of interspersed source
> and call blocks will not be executed in order because of the way
> org-babel-execute-buffer is written (first all the source blocks, then
> all the call blocks). 
>
> Therefore, when executing an entire buffer, there is no way to have
> the execution of a call block dependent on the prior execution of a
> source block.
>

It would be better to make the dependency explicit by passing the
results of the call line as a (potentially unused) variable to the code
block.  For example;

  #+name: three(v="three")
  #+begin_src elisp :var foo=one("two")
  v
  #+end_src

There is (at least currently) no guarantee that evaluation order will be
buffer order.

-- 
Eric Schulte
http://cs.unm.edu/~eschulte/

Re: org-babel order of evaluation

[Leo Alekseyev]

>> Therefore, when executing an entire buffer, there is no way to have
>> the execution of a call block dependent on the prior execution of a
>> source block.
>>
>
> It would be better to make the dependency explicit by passing the
> results of the call line as a (potentially unused) variable to the code
> block.  For example;
[snip]
>
> There is (at least currently) no guarantee that evaluation order will be
> buffer order.

I've been extremely confused by this in the past; this should be
prominently documented.  In the long run, I would like to see this
behavior changed.  One would intuitively expect all the source code in
the file to be evaluated in order.  This is how it works in pretty
much any other interpreter, why should org-babel be different?

(I'm a big fan of the principle of least surprise, and this behavior
violates it with vengeance :)  )

This is particularly nasty because many users start by treating an
org-babel file as a fancier version of the original source code with
nice annotations and outline levels; typically in a single language.
Thus, operationally, there isn't a distinction between tangling the
blocks into a single source file and feeding that to the interpreter
and running execute on the whole buffer.  But then, of course, one
might start using named blocks, variables, and #+call directives.  It
achieves the same effect as writing wrapper functions (or issuing
statements like source("somefile")) in the original language.  So,
when it results in a completely different execution order, it's a huge
surprise.

Even if this can be fixed by putting dummy dependencies in by hand,
this fix  seems inelegant and hacky.

Is there some deep rationale for the current behavior that I'm not
seeing?  Are there big obstacles to enforcing ligeral execution order?

--Leo

Re: org-babel order of evaluation

[Rick Frankel]

On Thu, Jan 12, 2012 at 04:35:31PM -0600, Leo Alekseyev wrote:
> >> Therefore, when executing an entire buffer, there is no way to have
> >> the execution of a call block dependent on the prior execution of a
> >> source block.
> >>
> >
> > It would be better to make the dependency explicit by passing the
> > results of the call line as a (potentially unused) variable to the code
> > block.  For example;
> [snip]

The problem w/ this is that the (potentially time consuming) dependent
will be executed twice when doing a buffer eval.

> > There is (at least currently) no guarantee that evaluation order will be
> > buffer order.
> 
> Is there some deep rationale for the current behavior that I'm not
> seeing?  Are there big obstacles to enforcing ligeral execution order?

It's because prior to 7.8, call blocks were not executed during a
buffer execute. The solution was to execute all the call blocks after
executing the src block. (Eric would have to comment on how hard it
would be to merge the two functions :).

rick

Re: org-babel order of evaluation

[Eric Schulte]

[-- Attachment #1: Type: text/plain, Size: 1379 bytes --]

Rick Frankel <rick@rickster.com> writes:

> On Thu, Jan 12, 2012 at 04:35:31PM -0600, Leo Alekseyev wrote:
>> >> Therefore, when executing an entire buffer, there is no way to have
>> >> the execution of a call block dependent on the prior execution of a
>> >> source block.
>> >>
>> >
>> > It would be better to make the dependency explicit by passing the
>> > results of the call line as a (potentially unused) variable to the code
>> > block.  For example;
>> [snip]
>
> The problem w/ this is that the (potentially time consuming) dependent
> will be executed twice when doing a buffer eval.
>
>> > There is (at least currently) no guarantee that evaluation order will be
>> > buffer order.
>> 
>> Is there some deep rationale for the current behavior that I'm not
>> seeing?  Are there big obstacles to enforcing ligeral execution order?
>
> It's because prior to 7.8, call blocks were not executed during a
> buffer execute. The solution was to execute all the call blocks after
> executing the src block. (Eric would have to comment on how hard it
> would be to merge the two functions :).
>

Turns out it was not that difficult to change this behavior.  You and
Leo are both correct that in-buffer-order evaluation is more natural and
expected than the previous behavior.  I've just pushed up a fix after
which evaluating the following


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: execute-in-order.org --]
[-- Type: text/x-org, Size: 211 bytes --]

#+Title: Execute all executables in Order
#+Property: results silent

#+name: foo
#+BEGIN_SRC sh :var it="one"
  echo $it >> debug
#+END_SRC

#+call: foo("two")

#+BEGIN_SRC sh
  echo "three" >> debug
#+END_SRC

[-- Attachment #3: Type: text/plain, Size: 196 bytes --]


results in the creation of a "debug" file in the same directory reading;

,----
| one
| two
| three
`----

Thanks for bringing this up,

>
> rick
>

-- 
Eric Schulte
http://cs.unm.edu/~eschulte/

Re: org-babel order of evaluation

[Rick Frankel]

On Thu, Jan 12, 2012 at 06:07:41PM -0700, Eric Schulte wrote:
> Rick Frankel <rick@rickster.com> writes:
> 
> Turns out it was not that difficult to change this behavior.  You and
> Leo are both correct that in-buffer-order evaluation is more natural and
> expected than the previous behavior.  I've just pushed up a fix after
> which evaluating the following
> 

Brillant! thank you Eric. This saves me much pain.

rick

Re: org-babel order of evaluation

[Leo Alekseyev]

[-- Attachment #1: Type: text/plain, Size: 1125 bytes --]

On Thu, Jan 12, 2012 at 7:52 PM, Rick Frankel <rick@rickster.com> wrote:
> On Thu, Jan 12, 2012 at 06:07:41PM -0700, Eric Schulte wrote:
>> Rick Frankel <rick@rickster.com> writes:
>>
>> Turns out it was not that difficult to change this behavior.  You and
>> Leo are both correct that in-buffer-order evaluation is more natural and
>> expected than the previous behavior.  I've just pushed up a fix after
>> which evaluating the following

Eric,
The fix doesn't seem to be working for me when I export the buffer to
HTML.  The ordering of call and source blocks once again becomes
randomized, and in general, exported file is missing a bunch of stuff
unless I run org-babel-execute-buffer prior to export.  Since the
export engine does its own evaluation, it doesn't seem like
org-babel-execute-buffer should be a necessity.  But I can't run
org-babel-execute-buffer on anything with a src_<language> inline
block as it gives me an error.

I'm attaching two files which do not export correctly, at least when
one doesn't run org-babel-execute-buffer; just do C-c C-e h and look
at the output.

--Leo

[-- Attachment #2: test-export4.org --]
[-- Type: application/octet-stream, Size: 1101 bytes --]

#+property: session *R-babel*

* export fail

#+NAME: elo_sim_analysis
#+HEADER: :var idx=1 :var print="TRUE" :var plot.filename="conv1.png" :var plot.disp="FALSE"
#+BEGIN_SRC R :results output replace 
  if (print == TRUE) {cat("some output\n")}
  
  img.dir = "images"
  cat.fname.link <- function() { cat(img.dir,"/",plot.filename,"\n",sep="") } 
  
  cat(img.dir,"/",plot.filename,"\n",sep="")
  
#+END_SRC


Should have all conv1 stuff

#+call: elo_sim_analysis(idx=1,print="TRUE",plot.filename="conv1.png",plot.disp="FALSE")

src_R[:results output replace]{ cat.fname.link() }

Should have all conv2 stuff 

#+call: elo_sim_analysis(idx=8,print="TRUE",plot.filename="conv2.png",plot.disp="FALSE")

src_R[:results output replace]{ cat.fname.link() } 

Should have all conv3 stuff
#+call: elo_sim_analysis(idx=1,print="TRUE",plot.filename="conv3.png",plot.disp="FALSE")

src_R[:results output replace]{ cat.fname.link() } 

Should have all conv4 stuff

#+call: elo_sim_analysis(idx=1,print="TRUE",plot.filename="conv4.png",plot.disp="FALSE")

src_R[:results output replace]{ cat.fname.link() } 



[-- Attachment #3: test-export6.org --]
[-- Type: application/octet-stream, Size: 1573 bytes --]


#+startup: noindent
#+property: session *R-babel*
#+property: tangle yes

* code

#+NAME: elo_sim_analysis
#+HEADER: :var idx=1 :var print="TRUE" :var plot.filename="conv1.png" :var plot.disp="FALSE"
#+BEGIN_SRC R :results output replace 
  gen.sample.plots <- function(idx) {
    hist(rnorm(100))
    title(paste("Plot for index ",idx, sep=""))
  }
  
  if (print == TRUE) {cat("some output\n")}
  
  img.dir = "images"
  cat.fname.link <- function() { cat("\nfile:",img.dir,"/",plot.filename,"\n",sep="") } 
  
  cat("file:",img.dir,"/",plot.filename,"\n",sep="")
  
#+END_SRC

#+RESULTS: elo_sim_analysis
:  some output
:  file:images/conv1.png

#+NAME: gen_link
#+HEADER: :var plot.filename="conv1.png" 
#+BEGIN_SRC R :results output replace 
  img.dir = "images"
  cat.fname.link <- function() { cat("\nfile:",img.dir,"/",plot.filename,"\n",sep="") } 
  cat.fname.link()
#+END_SRC

* ordered execution fail

Should have all conv1 stuff

#+call: elo_sim_analysis(idx=1,print="TRUE",plot.filename="conv1.png",plot.disp="FALSE")

#+call: gen_link(plot.filename="conv1.png")

Should have all conv2 stuff 

#+call: elo_sim_analysis(idx=8,print="TRUE",plot.filename="conv2.png",plot.disp="FALSE")

#+call: gen_link(plot.filename="conv2.png")

Should have all conv3 stuff

#+call: elo_sim_analysis(idx=1,print="TRUE",plot.filename="conv3.png",plot.disp="FALSE")

#+call: gen_link(plot.filename="conv3.png")

Should have all conv4 stuff

#+call: elo_sim_analysis(idx=1,print="TRUE",plot.filename="conv4.png",plot.disp="FALSE")

#+call: gen_link(plot.filename="conv4.png")

Re: org-babel order of evaluation

[Eric Schulte]

Leo Alekseyev <dnquark@gmail.com> writes:

> On Thu, Jan 12, 2012 at 7:52 PM, Rick Frankel <rick@rickster.com> wrote:
>> On Thu, Jan 12, 2012 at 06:07:41PM -0700, Eric Schulte wrote:
>>> Rick Frankel <rick@rickster.com> writes:
>>>
>>> Turns out it was not that difficult to change this behavior.  You and
>>> Leo are both correct that in-buffer-order evaluation is more natural and
>>> expected than the previous behavior.  I've just pushed up a fix after
>>> which evaluating the following
>
> Eric,
> The fix doesn't seem to be working for me when I export the buffer to
> HTML.

My previous fix only set the order of evaluation for interactive buffer
evaluation.  I've just pushed up another fix which sets the order of
evaluation during export.

Best,

> The ordering of call and source blocks once again becomes randomized,
> and in general, exported file is missing a bunch of stuff unless I run
> org-babel-execute-buffer prior to export.  Since the export engine
> does its own evaluation, it doesn't seem like org-babel-execute-buffer
> should be a necessity.  But I can't run org-babel-execute-buffer on
> anything with a src_<language> inline block as it gives me an error.
>
> I'm attaching two files which do not export correctly, at least when
> one doesn't run org-babel-execute-buffer; just do C-c C-e h and look
> at the output.
>
> --Leo
>
>

-- 
Eric Schulte
http://cs.unm.edu/~eschulte/