[babel] Trouble with :cache yes

[babel] Trouble with :cache yes

[Ken.Williams]

Hi,

I'm having some trouble getting ":cache yes" to behave the way I think
it's supposed to.  As a test, I have a simple example containing just a
title and one source block:

#+source: testcache
#+begin_src R :cache yes :exports both :results output
dat <-  matrix(runif(12), 3, 4)
print(dat)
#+end_src


If I export this document to HTML (C-c C-e b), Emacs asks me "Evaluate
this R code block (testcache) on your system?"  If I say 'y' it
re-evaluates, if I say 'n' it doesn't, so it doesn't seem like there's any
role that caching gets to play here.

In addition, when I export the document as above, the results are not
saved in the original org-mode buffer, so whatever "#+results" block is
there (or not there) from doing C-c C-c is neither used nor overwritten -
and therefore the exported document contains different results than the
source document.

The behavior I expected (please let me know if my expectation is
incorrect) was for the result of the computation to be cached in the Emacs
buffer when I do the first export, and for that saved result to be
included in the exported content for subsequent exports, until either the
code/inputs change or I delete the results block.

My configuration is:

Emacs  : GNU Emacs 23.2.50.1 (i386-apple-darwin9.8.0, NS
apple-appkit-949.54)
 of 2010-08-18 on braeburn.aquamacs.org - Aquamacs Distribution 2.1
Package: Org-mode version 7.5

Thanks!


--
Ken Williams
Senior Research Scientist
Thomson Reuters
http://labs.thomsonreuters.com

Re: [babel] Trouble with :cache yes

[Eric Schulte]

Hi Ken,

In order for caching to work, the results of the code block must exist
in the org-mode file.  For example, the following code block will be
evaluated when triggered either interactively or during export

#+begin_src emacs-lisp :cache yes
  (+ 2 2)
#+end_src

alternately, this block will not be evaluated when triggered either
interactively or on export, because the cached results are present

#+begin_src emacs-lisp :cache yes
  (+ 2 2)
#+end_src

#+results[9b77429d6cea71daf37e21ee09170810b9905066]:
: 4

In your example, for the code block to not be evaluated as part of the
export process, you must first evaluate it manually within the Org-mode
file, leaving the results (with the hash tag) saved in the Org-mode
file.

Best -- Eric

<Ken.Williams@thomsonreuters.com> writes:

> Hi,
>
> I'm having some trouble getting ":cache yes" to behave the way I think
> it's supposed to.  As a test, I have a simple example containing just a
> title and one source block:
>
> #+source: testcache
> #+begin_src R :cache yes :exports both :results output
> dat <-  matrix(runif(12), 3, 4)
> print(dat)
> #+end_src
>
>
> If I export this document to HTML (C-c C-e b), Emacs asks me "Evaluate
> this R code block (testcache) on your system?"  If I say 'y' it
> re-evaluates, if I say 'n' it doesn't, so it doesn't seem like there's any
> role that caching gets to play here.
>
> In addition, when I export the document as above, the results are not
> saved in the original org-mode buffer, so whatever "#+results" block is
> there (or not there) from doing C-c C-c is neither used nor overwritten -
> and therefore the exported document contains different results than the
> source document.
>
> The behavior I expected (please let me know if my expectation is
> incorrect) was for the result of the computation to be cached in the Emacs
> buffer when I do the first export, and for that saved result to be
> included in the exported content for subsequent exports, until either the
> code/inputs change or I delete the results block.
>
> My configuration is:
>
> Emacs  : GNU Emacs 23.2.50.1 (i386-apple-darwin9.8.0, NS
> apple-appkit-949.54)
>  of 2010-08-18 on braeburn.aquamacs.org - Aquamacs Distribution 2.1
> Package: Org-mode version 7.5
>
> Thanks!
>
>
> --
> Ken Williams
> Senior Research Scientist
> Thomson Reuters
> http://labs.thomsonreuters.com

Re: [babel] Trouble with :cache yes

[Rainer M Krug]

On Wed, Mar 23, 2011 at 3:50 AM, Eric Schulte <schulte.eric@gmail.com> wrote:
> Hi Ken,
>
> In order for caching to work, the results of the code block must exist
> in the org-mode file.  For example, the following code block will be
> evaluated when triggered either interactively or during export
>
> #+begin_src emacs-lisp :cache yes
>  (+ 2 2)
> #+end_src
>
> alternately, this block will not be evaluated when triggered either
> interactively or on export, because the cached results are present
>
> #+begin_src emacs-lisp :cache yes
>  (+ 2 2)
> #+end_src
>
> #+results[9b77429d6cea71daf37e21ee09170810b9905066]:
> : 4
>
> In your example, for the code block to not be evaluated as part of the
> export process, you must first evaluate it manually within the Org-mode
> file, leaving the results (with the hash tag) saved in the Org-mode
> file.

I will chime in here, because I had a similar problem.
This does not work for me. When I evaluate manually (C-c c) I get the following:

* test
#+source: testcache
#+begin_src R :cache yes :exports both :results output
  dat <-  matrix(runif(12), 3, 4)
  print(dat)
#+end_src

#+results[cad298135e53df633545d6a32a8b2aab5201721c]: testcache
:           [,1]      [,2]      [,3]      [,4]
: [1,] 0.4470891 0.2016197 0.1383083 0.6214485
: [2,] 0.1598936 0.3819967 0.3527698 0.5124687
: [3,] 0.3040325 0.5906898 0.1611272 0.1702849

Which I saved (just to be save).

When exporting to a pdf, I get the following matrix in the pdf:

          [,1]        [,2]     [,3]      [,4]
[1,] 0.5626863 0.8397120 0.9886886 0.2233873
[2,] 0.8697064 0.1101432 0.1372992 0.4114674
[3,] 0.3548678 0.5658843 0.1608864 0.5809167

And it is different after each export. So it seems that the code block
*is* actually evaluated, despite of the cached info.

Rainer


>
> Best -- Eric
>
> <Ken.Williams@thomsonreuters.com> writes:
>
>> Hi,
>>
>> I'm having some trouble getting ":cache yes" to behave the way I think
>> it's supposed to.  As a test, I have a simple example containing just a
>> title and one source block:
>>
>> #+source: testcache
>> #+begin_src R :cache yes :exports both :results output
>> dat <-  matrix(runif(12), 3, 4)
>> print(dat)
>> #+end_src
>>
>>
>> If I export this document to HTML (C-c C-e b), Emacs asks me "Evaluate
>> this R code block (testcache) on your system?"  If I say 'y' it
>> re-evaluates, if I say 'n' it doesn't, so it doesn't seem like there's any
>> role that caching gets to play here.
>>
>> In addition, when I export the document as above, the results are not
>> saved in the original org-mode buffer, so whatever "#+results" block is
>> there (or not there) from doing C-c C-c is neither used nor overwritten -
>> and therefore the exported document contains different results than the
>> source document.
>>
>> The behavior I expected (please let me know if my expectation is
>> incorrect) was for the result of the computation to be cached in the Emacs
>> buffer when I do the first export, and for that saved result to be
>> included in the exported content for subsequent exports, until either the
>> code/inputs change or I delete the results block.
>>
>> My configuration is:
>>
>> Emacs  : GNU Emacs 23.2.50.1 (i386-apple-darwin9.8.0, NS
>> apple-appkit-949.54)
>>  of 2010-08-18 on braeburn.aquamacs.org - Aquamacs Distribution 2.1
>> Package: Org-mode version 7.5
>>
>> Thanks!
>>
>>
>> --
>> Ken Williams
>> Senior Research Scientist
>> Thomson Reuters
>> http://labs.thomsonreuters.com
>
>



-- 
NEW GERMAN FAX NUMBER!!!

Rainer M. Krug, PhD (Conservation Ecology, SUN), MSc (Conservation
Biology, UCT), Dipl. Phys. (Germany)

Centre of Excellence for Invasion Biology
Natural Sciences Building
Office Suite 2039
Stellenbosch University
Main Campus, Merriman Avenue
Stellenbosch
South Africa

Cell:           +27 - (0)83 9479 042
Fax:            +27 - (0)86 516 2782
Fax:            +49 - (0)321 2125 2244
email:          Rainer@krugs.de

Skype:          RMkrug
Google:         R.M.Krug@gmail.com

Re: [babel] Trouble with :cache yes

[Ken.Williams]

On 3/23/11 1:46 AM, "Rainer M Krug" <r.m.krug@gmail.com> wrote:

>
>When exporting to a pdf, I get the following matrix in the pdf:
>
>          [,1]        [,2]     [,3]      [,4]
>[1,] 0.5626863 0.8397120 0.9886886 0.2233873
>[2,] 0.8697064 0.1101432 0.1372992 0.4114674
>[3,] 0.3548678 0.5658843 0.1608864 0.5809167
>
>And it is different after each export. So it seems that the code block
>*is* actually evaluated, despite of the cached info.
>
>Rainer


Yes, that's exactly the behavior I'm seeing too.  Caching seems to have no
effect on HTML export, no matter whether the block already has a
"#+results" section or not.

 -Ken

Re: [babel] Trouble with :cache yes

[Eric Schulte]

Rainer M Krug <r.m.krug@gmail.com> writes:

> On Wed, Mar 23, 2011 at 3:50 AM, Eric Schulte <schulte.eric@gmail.com> wrote:
>> Hi Ken,
>>
>> In order for caching to work, the results of the code block must exist
>> in the org-mode file.  For example, the following code block will be
>> evaluated when triggered either interactively or during export
>>
>> #+begin_src emacs-lisp :cache yes
>>  (+ 2 2)
>> #+end_src
>>
>> alternately, this block will not be evaluated when triggered either
>> interactively or on export, because the cached results are present
>>
>> #+begin_src emacs-lisp :cache yes
>>  (+ 2 2)
>> #+end_src
>>
>> #+results[9b77429d6cea71daf37e21ee09170810b9905066]:
>> : 4
>>
>> In your example, for the code block to not be evaluated as part of the
>> export process, you must first evaluate it manually within the Org-mode
>> file, leaving the results (with the hash tag) saved in the Org-mode
>> file.
>
> I will chime in here, because I had a similar problem.  This does not
> work for me. When I evaluate manually (C-c c) I get the following:
>

Thanks for pointing this out, your example doesn't work for me either.
I tracked this down to a problem of not finding the cached results of
named code blocks.  I've just pushed up a simple fix for this issue, so
caching should now work as expected.

Best -- Eric

>
> * test
> #+source: testcache
> #+begin_src R :cache yes :exports both :results output
>   dat <-  matrix(runif(12), 3, 4)
>   print(dat)
> #+end_src
>
> #+results[cad298135e53df633545d6a32a8b2aab5201721c]: testcache
> :           [,1]      [,2]      [,3]      [,4]
> : [1,] 0.4470891 0.2016197 0.1383083 0.6214485
> : [2,] 0.1598936 0.3819967 0.3527698 0.5124687
> : [3,] 0.3040325 0.5906898 0.1611272 0.1702849
>
> Which I saved (just to be save).
>
> When exporting to a pdf, I get the following matrix in the pdf:
>
>           [,1]        [,2]     [,3]      [,4]
> [1,] 0.5626863 0.8397120 0.9886886 0.2233873
> [2,] 0.8697064 0.1101432 0.1372992 0.4114674
> [3,] 0.3548678 0.5658843 0.1608864 0.5809167
>
> And it is different after each export. So it seems that the code block
> *is* actually evaluated, despite of the cached info.
>
> Rainer
>
>
>>
>> Best -- Eric
>>
>> <Ken.Williams@thomsonreuters.com> writes:
>>
>>> Hi,
>>>
>>> I'm having some trouble getting ":cache yes" to behave the way I think
>>> it's supposed to.  As a test, I have a simple example containing just a
>>> title and one source block:
>>>
>>> #+source: testcache
>>> #+begin_src R :cache yes :exports both :results output
>>> dat <-  matrix(runif(12), 3, 4)
>>> print(dat)
>>> #+end_src
>>>
>>>
>>> If I export this document to HTML (C-c C-e b), Emacs asks me "Evaluate
>>> this R code block (testcache) on your system?"  If I say 'y' it
>>> re-evaluates, if I say 'n' it doesn't, so it doesn't seem like there's any
>>> role that caching gets to play here.
>>>
>>> In addition, when I export the document as above, the results are not
>>> saved in the original org-mode buffer, so whatever "#+results" block is
>>> there (or not there) from doing C-c C-c is neither used nor overwritten -
>>> and therefore the exported document contains different results than the
>>> source document.
>>>
>>> The behavior I expected (please let me know if my expectation is
>>> incorrect) was for the result of the computation to be cached in the Emacs
>>> buffer when I do the first export, and for that saved result to be
>>> included in the exported content for subsequent exports, until either the
>>> code/inputs change or I delete the results block.
>>>
>>> My configuration is:
>>>
>>> Emacs  : GNU Emacs 23.2.50.1 (i386-apple-darwin9.8.0, NS
>>> apple-appkit-949.54)
>>>  of 2010-08-18 on braeburn.aquamacs.org - Aquamacs Distribution 2.1
>>> Package: Org-mode version 7.5
>>>
>>> Thanks!
>>>
>>>
>>> --
>>> Ken Williams
>>> Senior Research Scientist
>>> Thomson Reuters
>>> http://labs.thomsonreuters.com
>>
>>

Re: [babel] Trouble with :cache yes

[Eric Schulte]

<Ken.Williams@thomsonreuters.com> writes:

> On 3/23/11 1:46 AM, "Rainer M Krug" <r.m.krug@gmail.com> wrote:
>
>>
>>When exporting to a pdf, I get the following matrix in the pdf:
>>
>>          [,1]        [,2]     [,3]      [,4]
>>[1,] 0.5626863 0.8397120 0.9886886 0.2233873
>>[2,] 0.8697064 0.1101432 0.1372992 0.4114674
>>[3,] 0.3548678 0.5658843 0.1608864 0.5809167
>>
>>And it is different after each export. So it seems that the code block
>>*is* actually evaluated, despite of the cached info.
>>
>>Rainer
>
>
> Yes, that's exactly the behavior I'm seeing too.  Caching seems to have no
> effect on HTML export, no matter whether the block already has a
> "#+results" section or not.
>

Please try once more after updating Org-mode, I just pushed up a fix
which should resolve this problem.

Thanks -- Eric

>
>  -Ken

Re: [babel] Trouble with :cache yes

[Ken.Williams]

On 3/23/11 12:28 PM, "Eric Schulte" <schulte.eric@gmail.com> wrote:

>Thanks for pointing this out, your example doesn't work for me either.
>I tracked this down to a problem of not finding the cached results of
>named code blocks.  I've just pushed up a simple fix for this issue, so
>caching should now work as expected.

Thanks Eric.  In my case I'm seeing the [mis]behavior even when the code
block has no name - will your fix cover that case too?

 -Ken

Re: [babel] Trouble with :cache yes

[Eric Schulte]

<Ken.Williams@thomsonreuters.com> writes:

> On 3/23/11 12:28 PM, "Eric Schulte" <schulte.eric@gmail.com> wrote:
>
>>Thanks for pointing this out, your example doesn't work for me either.
>>I tracked this down to a problem of not finding the cached results of
>>named code blocks.  I've just pushed up a simple fix for this issue, so
>>caching should now work as expected.
>
> Thanks Eric.  In my case I'm seeing the [mis]behavior even when the code
> block has no name - will your fix cover that case too?
>
>  -Ken

Hi Ken,

Could you send me a minimal example the demonstrates the misbehavior?

Thanks -- Eric

Re: [babel] Trouble with :cache yes

[Ken.Williams]

On 3/23/11 12:54 PM, "Williams, Ken (TR Corp Tech)"
<Ken.Williams@thomsonreuters.com> wrote:

>
>On 3/23/11 12:28 PM, "Eric Schulte" <schulte.eric@gmail.com> wrote:
>
>>Thanks for pointing this out, your example doesn't work for me either.
>>I tracked this down to a problem of not finding the cached results of
>>named code blocks.  I've just pushed up a simple fix for this issue, so
>>caching should now work as expected.
>
>Thanks Eric.  In my case I'm seeing the [mis]behavior even when the code
>block has no name - will your fix cover that case too?

Actually, I just realized I'm mistaken.  If I manually evaluate a
(non-named) block, *then* export the entire document, I indeed get the
cached results in the export, as expected.

However, if I change the code of a :cached block somewhere in my document
(or its MD5 is otherwise invalidated) and re-export the document without
first doing C-c C-c, the export will neither use the cache (which is good)
nor save the results back to the cache (which is bad), so the export is
now out of sync with the original.  It would be great if the results in
exports could be cached in the same way they would be cached when manually
evaluating blocks.

Or perhaps, is there some command to evaluate all blocks in a document
that need to be re-evaluated, and save the results back to the buffer?  I
could do that every time before exporting, maybe.


--
Ken Williams
Senior Research Scientist
Thomson Reuters
http://labs.thomsonreuters.com

Re: [babel] Trouble with :cache yes

[Eric Schulte]

<Ken.Williams@thomsonreuters.com> writes:

> On 3/23/11 12:54 PM, "Williams, Ken (TR Corp Tech)"
> <Ken.Williams@thomsonreuters.com> wrote:
>
>>
>>On 3/23/11 12:28 PM, "Eric Schulte" <schulte.eric@gmail.com> wrote:
>>
>>>Thanks for pointing this out, your example doesn't work for me either.
>>>I tracked this down to a problem of not finding the cached results of
>>>named code blocks.  I've just pushed up a simple fix for this issue, so
>>>caching should now work as expected.
>>
>>Thanks Eric.  In my case I'm seeing the [mis]behavior even when the code
>>block has no name - will your fix cover that case too?
>
> Actually, I just realized I'm mistaken.  If I manually evaluate a
> (non-named) block, *then* export the entire document, I indeed get the
> cached results in the export, as expected.
>
> However, if I change the code of a :cached block somewhere in my document
> (or its MD5 is otherwise invalidated) and re-export the document without
> first doing C-c C-c, the export will neither use the cache (which is good)
> nor save the results back to the cache (which is bad), so the export is
> now out of sync with the original.  It would be great if the results in
> exports could be cached in the same way they would be cached when manually
> evaluating blocks.
>

One of the features of Org-mode export is that the document is not
changed by the act of exporting, so unfortunately you will have to
manually evaluate all code blocks, and save their results before
exporting.

>
> Or perhaps, is there some command to evaluate all blocks in a document
> that need to be re-evaluated, and save the results back to the buffer?  I
> could do that every time before exporting, maybe.
>

Fortunately there is such a function, org-babel-execute-buffer, bound to
C-c C-v b, try C-c C-v h in an Org-mode buffer to see all code block
specific key bindings.

Best -- Eric

>
>
> --
> Ken Williams
> Senior Research Scientist
> Thomson Reuters
> http://labs.thomsonreuters.com

Re: [babel] Trouble with :cache yes

[Ken.Williams]

On 3/23/11 1:16 PM, "Eric Schulte" <schulte.eric@gmail.com> wrote:

><Ken.Williams@thomsonreuters.com> writes:
>>Or perhaps, is there some command to evaluate all blocks in a document
>> that need to be re-evaluated, and save the results back to the buffer?
>>I
>> could do that every time before exporting, maybe.
>>
>
>Fortunately there is such a function, org-babel-execute-buffer, bound to
>C-c C-v b, try C-c C-v h in an Org-mode buffer to see all code block
>specific key bindings.


So I tried doing that, and unfortunately it looks like I'm going to have
to restructure my document a bit if I want to use it.  I'd been using
"#+begin_src" blocks in two different ways - some blocks are just "code
listings" that I don't really intend to run, whereas some blocks are "live
code" that I want to execute when exporting the document - these latter
have Babel-style arguments like ":exports both" or ":results output" and
the like.

Of course, "C-c C-v b" will treat *all* of the blocks like "live code"
blocks, so I'm looking for some way to shut off Babel processing of many
of the blocks and just treat them as styled code listings.  Is there some
flag to do that, or should I switch from "#+BEGIN_SRC sh" to
"#+BEGIN_EXAMPLE sh" or something?  Does BEGIN_EXAMPLE know about
different languages of code?



--
Ken Williams
Senior Research Scientist
Thomson Reuters
http://labs.thomsonreuters.com

Re: [babel] Trouble with :cache yes

[Erik Iverson]

Ken.Williams@thomsonreuters.com wrote:
> 
> On 3/23/11 1:16 PM, "Eric Schulte" <schulte.eric@gmail.com> wrote:
> 
>> <Ken.Williams@thomsonreuters.com> writes:
>>> Or perhaps, is there some command to evaluate all blocks in a document
>>> that need to be re-evaluated, and save the results back to the buffer?
>>> I
>>> could do that every time before exporting, maybe.
>>>
>> Fortunately there is such a function, org-babel-execute-buffer, bound to
>> C-c C-v b, try C-c C-v h in an Org-mode buffer to see all code block
>> specific key bindings.
> 
> 
> So I tried doing that, and unfortunately it looks like I'm going to have
> to restructure my document a bit if I want to use it.  I'd been using
> "#+begin_src" blocks in two different ways - some blocks are just "code
> listings" that I don't really intend to run, whereas some blocks are "live
> code" that I want to execute when exporting the document - these latter
> have Babel-style arguments like ":exports both" or ":results output" and
> the like.
> 
> Of course, "C-c C-v b" will treat *all* of the blocks like "live code"
> blocks, so I'm looking for some way to shut off Babel processing of many
> of the blocks and just treat them as styled code listings.  Is there some
> flag to do that, or should I switch from "#+BEGIN_SRC sh" to
> "#+BEGIN_EXAMPLE sh" or something?  Does BEGIN_EXAMPLE know about
> different languages of code?

Try :eval never

http://orgmode.org/org.html#eval

Don't know if that will work, but it sounds promising.

--Erik

Re: [babel] Trouble with :cache yes

[Ken.Williams]

On 3/23/11 5:00 PM, "Erik Iverson" <eriki@ccbr.umn.edu> wrote:
>
>Try :eval never
>
>http://orgmode.org/org.html#eval
>
>Don't know if that will work, but it sounds promising.

Perfect!  Thanks everyone for the help.


--
Ken Williams
Senior Research Scientist
Thomson Reuters
http://labs.thomsonreuters.com