when does :cache not cache?

when does :cache not cache?

[Eric S Fraga]

[-- Attachment #1: Type: text/plain, Size: 976 bytes --]

Hello all (again!),

in the paper I am writing, I have a number of gnuplot src blocks, some
of which process a significant amount of data so take some time to
generate the actual plots.  The data are static so caching the results
make sense.  However, even though I have ":cache yes" on each of these
named src blocks, and I have (for good measure), the property
"header-args:gnuplot" set to ":cache yes" as well, the plots are being
regenerated each time I export to PDF via LaTeX.

In trying to create a minimal example (see attached), it seems that
caching stops working as soon as I add a ":var data=..." header argument
to generate a plot from a table of data.

Why does specifying a variable to a src block violate the caching
directive?  Is this intended behaviour (assuming I've understood what is
happening correctly)?

Thank you,
eric

-- 
: Eric S Fraga via Emacs 28.0.60, Org release_9.5-163-g4eab5b
: Latest paper written in org: https://arxiv.org/abs/2106.05096

[-- Attachment #2: gnuplot.org --]
[-- Type: application/vnd.lotus-organizer, Size: 618 bytes --]

Re: when does :cache not cache?

[Ihor Radchenko]

[-- Attachment #1: Type: text/plain, Size: 1113 bytes --]

Eric S Fraga <e.fraga@ucl.ac.uk> writes:

> in the paper I am writing, I have a number of gnuplot src blocks, some
> of which process a significant amount of data so take some time to
> generate the actual plots.  The data are static so caching the results
> make sense.  However, even though I have ":cache yes" on each of these
> named src blocks, and I have (for good measure), the property
> "header-args:gnuplot" set to ":cache yes" as well, the plots are being
> regenerated each time I export to PDF via LaTeX.

I was able to reproduce. The reason why caching does not work is related
to the way :var assignments work in ob-gnuplot. We dump the table data
into temporary files and refer to those files in generated gnuplot
script body. The temporary files names change on every execution and the
gnuplot script hash is never going to be the same.

I am attaching tentative patch to fix the issue.
The patch introduces a new functionality to ob-core.el allowing more
stable temporary file names. I am not sure if my implementation is the
best way to solve the problem, so comments are welcome.

Best,
Ihor


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-ob-gnuplot.el-Make-cache-argument-work-with-var-assi.patch --]
[-- Type: text/x-diff, Size: 5773 bytes --]

From ac11b4d08edd577b29a398296364b4340096a6ae Mon Sep 17 00:00:00 2001
Message-Id: <ac11b4d08edd577b29a398296364b4340096a6ae.1639294626.git.yantar92@gmail.com>
From: Ihor Radchenko <yantar92@gmail.com>
Date: Sun, 12 Dec 2021 15:31:35 +0800
Subject: [PATCH] ob-gnuplot.el: Make :cache argument work with :var
 assignments

* lisp/ob-core.el (org-babel-temporary-stable-directory): New variable
holding a temporary directory name that does not change between Emacs
sessions.
(org-babel-remove-temporary-stable-directory): New function removing
`org-babel-temporary-stable-directory' on Emacs shutdown.
(org-babel-temp-stable-file): Generate stable temporary file name for
object storage.  The file name is constant with for equal objects.
(org-babel-execute-src-block): Explicitly identify that if the result
is cached.
* lisp/ob-gnuplot.el (org-babel-gnuplot-process-vars): Make use of
`org-babel-stable-file' to make expanded body stable with respect to
:var assignments.

Fixes https://orgmode.org/list/87mtn1o5mn.fsf@ucl.ac.uk
---
 lisp/ob-core.el    | 52 +++++++++++++++++++++++++++++++++++++++++++++-
 lisp/ob-gnuplot.el |  5 ++++-
 2 files changed, 55 insertions(+), 2 deletions(-)

diff --git a/lisp/ob-core.el b/lisp/ob-core.el
index 7a9467b0e..d572423b7 100644
--- a/lisp/ob-core.el
+++ b/lisp/ob-core.el
@@ -735,7 +735,8 @@ (defun org-babel-execute-src-block (&optional arg info params)
 	    (forward-line)
 	    (skip-chars-forward " \t")
 	    (let ((result (org-babel-read-result)))
-	      (message (replace-regexp-in-string "%" "%%" (format "%S" result)))
+	      (message (format "Cached: %s"
+                               (replace-regexp-in-string "%" "%%" (format "%S" result))))
 	      result)))
 	 ((org-babel-confirm-evaluate info)
 	  (let* ((lang (nth 0 info))
@@ -3112,6 +3113,22 @@   (defvar org-babel-temporary-directory
 Used by `org-babel-temp-file'.  This directory will be removed on
 Emacs shutdown."))
 
+(defvar org-babel-temporary-stable-directory)
+(unless (or noninteractive (boundp 'org-babel-temporary-stable-directory))
+  (defvar org-babel-temporary-stable-directory
+    (or (and (boundp 'org-babel-temporary-stable-directory)
+	     (file-exists-p org-babel-temporary-stable-directory)
+	     org-babel-temporary-stable-directory)
+        (condition-case nil
+            (make-directory
+	     (expand-file-name
+              "babel-stable"
+              (temporary-file-directory)))
+          (t nil)))
+    "Directory to hold temporary files created to execute code blocks.
+Used by `org-babel-temp-file'.  This directory will be removed on
+Emacs shutdown."))
+
 (defcustom org-babel-remote-temporary-directory "/tmp/"
   "Directory to hold temporary files on remote hosts."
   :group 'org-babel
@@ -3155,6 +3172,30 @@ (defun org-babel-temp-file (prefix &optional suffix)
 	       temporary-file-directory)))
       (make-temp-file prefix nil suffix))))
 
+(defun org-babel-temp-stable-file (data prefix &optional suffix)
+  "Create a temporary file in the `org-babel-remove-temporary-stable-directory'.
+The file name is stable with respect to DATA.  The file name is
+constructed like the following: PREFIXDATAhashSUFFIX."
+  (if (file-remote-p default-directory)
+      (let* ((prefix
+              (concat (file-remote-p default-directory)
+                      (expand-file-name
+		       prefix org-babel-temporary-stable-directory)))
+             (path (concat prefix (format "%s" (sxhash data)) (or suffix ""))))
+        (with-temp-file path)
+        path)
+    (let* ((temporary-file-directory
+	    (or (and (boundp 'org-babel-temporary-stable-directory)
+		     (file-exists-p org-babel-temporary-stable-directory)
+		     org-babel-temporary-stable-directory)
+	        temporary-file-directory))
+           (path (concat
+                  (expand-file-name
+		   prefix org-babel-temporary-stable-directory)
+                  (format "%s" (sxhash data)) (or suffix ""))))
+      (with-temp-file path)
+      path)))
+
 (defun org-babel-remove-temporary-directory ()
   "Remove `org-babel-temporary-directory' on Emacs shutdown."
   (when (and (boundp 'org-babel-temporary-directory)
@@ -3178,7 +3219,16 @@ (defun org-babel-remove-temporary-directory ()
 		    org-babel-temporary-directory
 		  "[directory not defined]"))))))
 
+(defun org-babel-remove-temporary-stable-directory ()
+  "Remove `org-babel-temporary-stable-directory' and on Emacs shutdown."
+  (when (and (boundp 'org-babel-temporary-stable-directory)
+	     (file-exists-p org-babel-temporary-stable-directory))
+    (let ((org-babel-temporary-directory
+           org-babel-temporary-stable-directory))
+      (org-babel-remove-temporary-directory))))
+
 (add-hook 'kill-emacs-hook #'org-babel-remove-temporary-directory)
+(add-hook 'kill-emacs-hook #'org-babel-remove-temporary-stable-directory)
 
 (defun org-babel-one-header-arg-safe-p (pair safe-list)
   "Determine if the PAIR is a safe babel header arg according to SAFE-LIST.
diff --git a/lisp/ob-gnuplot.el b/lisp/ob-gnuplot.el
index 8c4a5957b..f831b4996 100644
--- a/lisp/ob-gnuplot.el
+++ b/lisp/ob-gnuplot.el
@@ -94,7 +94,10 @@ (defun org-babel-gnuplot-process-vars (params)
 	       (let* ((first  (car val))
 		      (tablep (or (listp first) (symbolp first))))
 		 (if tablep val (mapcar 'list val)))
-	       (org-babel-temp-file "gnuplot-") params)
+               ;; Make temporary file name stable with respect to data.
+               ;; If we do not do it, :cache argument becomes useless.
+               (org-babel-temp-stable-file params "gnuplot-")
+               params)
 	    (if (and (stringp val)
 		     (file-remote-p val)  ;; check if val is a remote file
 		     (file-exists-p val)) ;; call to file-exists-p is slow, maybe remove it
-- 
2.32.0

Re: when does :cache not cache?

[Eric S Fraga]

Dear Ihor,

> The reason why caching does not work is related to the way :var
> assignments work in ob-gnuplot. We dump the table data into temporary
> files and refer to those files in generated gnuplot script body. The
> temporary files names change on every execution and the gnuplot script
> hash is never going to be the same.

Ah, this makes perfect sense!  Thank you for looking into this.

I will try your patch later this week (busy with end of term teaching
aspects at the moment).

-- 
: Eric S Fraga, with org release_9.5.1-243-gad53c5 in Emacs 29.0.50
: Latest paper written in org: https://arxiv.org/abs/2106.05096

Re: when does :cache not cache?

[Ihor Radchenko]

Hi Eric,

Eric S Fraga <e.fraga@ucl.ac.uk> writes:

> I will try your patch later this week (busy with end of term teaching
> aspects at the moment).

Did you have a chance to test the patch?

Best,
Ihor

Re: when does :cache not cache?

[Fraga, Eric]

On Thursday,  9 Jun 2022 at 21:35, Ihor Radchenko wrote:
> Did you have a chance to test the patch?

No. <blush> Sorry.  Got busy with other things and had temporarily
(well, permanently) solved the problem (for me) by adding :eval
no-export to all of my gnuplot blocks.

-- 
: Eric S Fraga, with org release_9.5.4-521-g1105da in Emacs 29.0.50

Re: when does :cache not cache?

[Ihor Radchenko]

Ihor Radchenko <yantar92@gmail.com> writes:

> The patch introduces a new functionality to ob-core.el allowing more
> stable temporary file names. I am not sure if my implementation is the
> best way to solve the problem, so comments are welcome.

Applied onto main via 080462198.

Best,
Ihor

Re: when does :cache not cache?

[Fraga, Eric]

[-- Attachment #1: Type: text/plain, Size: 743 bytes --]

Hi Ihor,

On Monday,  4 Jul 2022 at 21:06, Ihor Radchenko wrote:
> Ihor Radchenko <yantar92@gmail.com> writes:
>
>> The patch introduces a new functionality to ob-core.el allowing more
>> stable temporary file names. I am not sure if my implementation is the
>> best way to solve the problem, so comments are welcome.
>
> Applied onto main via 080462198.

I updated org recently (and again just now) and am getting the error
shown in the attached file.  This is related to your patch, I think
given that it fails in accessing the "stable-file", but haven't verified
it.  Before I investigate further, I thought I would raise the issue.

Thank you,
eric

-- 
Eric S Fraga, @ericsfraga:matrix.org, GnuPG: 0xc89193d8fffcf67d

[-- Attachment #2: trace.txt --]
[-- Type: text/plain, Size: 2517 bytes --]

Debugger entered--Lisp error: (wrong-type-argument stringp nil)
  file-exists-p(nil)
  org-babel-temp-stable-file(((:cache . "yes") (:colname-names (plotdata "PSO" "" "PPA" "" "GA" "")) (:eval . "no-export") (:exports . "results") (:file . "images/permdbetaevolutionplot.png") (:hlines . "no") (:noweb . "yes") (:result-params "replace" "file") (:result-type . value) (:results . "replace file") (:rowname-names) (:session) (:tangle . "no") (:var plotdata [...] )) "gnuplot-")
  #f(compiled-function (pair) #<bytecode -0x18e177498e7fdfb1> [...])
  org-babel-gnuplot-process-vars([...])
  org-babel-expand-body:gnuplot([...])
  org-babel-sha1-hash([...])
  org-babel-exp-src-block((src-block (:language "gnuplot" :switches nil :parameters nil :begin 68881 :end 69528 :number-lines nil :preserve-indent nil :retain-labels t :use-labels t :label-fmt nil :value "  reset\n  set logscale xy\n  set xlabel \"n_f\"\n  set..." :post-blank 1 :post-affiliated 69077 :name "permdbetaplotevolution" :header (":var plotdata=permdbetaresultstable :file \"images/..." ":results file" ":exports results" ":eval no-export") :mode planning :granularity element :org-element--cache-sync-key (55 . 70525) :cached t :parent (section (:begin 68881 :end 79730 :contents-begin 68881 :contents-end 79679 :robust-begin 68881 :robust-end 79677 :post-blank 51 :post-affiliated 68881 :mode section :granularity element :org-element--cache-sync-key (55 . 70524) :cached t :parent (headline (:raw-value "results" :begin 68810 :end 79730 :pre-blank 0 :contents-begin 68881 :contents-end 79679 :robust-begin 68883 :robust-end 79677 :level 3 :priority nil :tags (...) :todo-keyword nil :todo-type nil :post-blank 51 :footnote-section-p nil :archivedp nil :commentedp nil :post-affiliated 68810 :title "results" :mode nil :granularity element :org-element--cache-sync-key (55 . 68551) :cached t :parent (headline ...))))))))
  org-babel-exp-process-buffer()
  org-export-as(latex nil nil nil (:output-file "book.tex"))
  org-export-to-file(latex "book.tex" nil nil nil nil nil org-latex-compile)
  org-latex-export-to-pdf(nil nil nil nil)
  (org-open-file (org-latex-export-to-pdf nil s v b))
  (if a (org-latex-export-to-pdf t s v b) (org-open-file (org-latex-export-to-pdf nil s v b)))
  (lambda (a s v b) (if a (org-latex-export-to-pdf t s v b) (org-open-file (org-latex-export-to-pdf nil s v b))))(nil nil nil nil)
  org-export-dispatch(nil)
  funcall-interactively(org-export-dispatch nil)
  command-execute(org-export-dispatch)

Re: when does :cache not cache?

[Ihor Radchenko]

"Fraga, Eric" <e.fraga@ucl.ac.uk> writes:

> I updated org recently (and again just now) and am getting the error
> shown in the attached file.  This is related to your patch, I think
> given that it fails in accessing the "stable-file", but haven't verified
> it.  Before I investigate further, I thought I would raise the issue.

Can you check the value of org-babel-temporary-stable-directory and its
docstring?

Best,
Ihor

Re: when does :cache not cache?

[Fraga, Eric]

Please ignore.  Restarting Emacs fixed the problem.  For some reason,
re-loading org was not enough.  I can export without error again.

-- 
: Eric S Fraga, with org release_9.5.4-613-gb3b17c in Emacs 29.0.50

when does :cache not cache?

[Emmanuel Charpentier]

[-- Attachment #1: Type: text/plain, Size: 91 bytes --]

You may find this John Kitchin's blog post illuminating.

HTH,

--
Emmanuel Charpentier


[-- Attachment #2: Type: text/html, Size: 366 bytes --]

Re: when does :cache not cache?

[Eric S Fraga]

On Saturday, 23 Oct 2021 at 11:13, Emmanuel Charpentier wrote:
> You may find this John Kitchin's blog
> post [...] illuminating.

Unfortunately not but thank you.  It is a very useful blog post but
doesn't cover my case, where I use a :var header to refer to a
table.  The table is static so I still do not understand why :cache
on the src block that references it doesn't work.

I may have to dive into the code, I guess.  I may be some time... ;-)

-- 
: Eric S Fraga via Emacs 28.0.60, Org release_9.5-163-g4eab5b
: Latest paper written in org: https://arxiv.org/abs/2106.05096

Re: when does :cache not cache?

[Eric S Fraga]

Hello all,

So, following the code: every time I ask to evaluate the gnuplot src
block in the minimal example I posted yesterday, the new-hash that is
calculated is different so the cache setting is ignored.

Digging deeper leads to me getting lost.  The hash does depend on the
incorporation of the variables set using :var but I do not see why the
hash should be different if the table has not changed.

Anyway, that's as far as I got.

For the record, I have simply put ":eval never-export" on all of my
gnuplot src blocks and the export happens much more quickly now.

eric
-- 
: Eric S Fraga via Emacs 28.0.60, Org release_9.5-163-g4eab5b
: Latest paper written in org: https://arxiv.org/abs/2106.05096

when does :cache not cache?

[Emmanuel Charpentier]

[-- Attachment #1: Type: text/plain, Size: 1654 bytes --]

Workaround : cache the computations,not the plotting itself (which
should be fast,and must be made on every table, anyway...) :

#+options: toc:nil author:nil

* Main text

Use of a static table:

#+call: timeit()
#+call: testplot[:file test0.pdf](data=table0)
#+call: timeit()

Use of a slow function:

#+call: timeit()
#+call: testplot[:file test1.pdf](data=table1)
#+call: timeit()

Use of a /cached/ slow function:

#+call: timeit()
#+call: testplot[:file test2.pdf](data=table2)
#+call: timeit()

* Annexes :noexport:

This is not exported, but computes results.

#+name: timeit
#+begin_src emacs-lisp
  (format-time-string "%Hh %Mm %Ss")
#+end_src

#+name: table0
| Val | Square |
|-----+--------|
|   0 |      0 |
|   1 |      1 |
|   2 |      4 |
|   3 |      9 |
|   4 |     16 |
#+TBLFM: $2=$1^2

#+name: table1
#+begin_src emacs-lisp
  (sleep-for 5)
  (setq s ( list (list "x" "x^2") 'hline))
  (dotimes (i 5 s) (setq s (append s (list (cons i (list (* i i)))))))
  s
#+end_src

#+RESULTS[46320b31c46cef901580bad78aee7032d97ffe64]: table1
| x | x^2 |
|---+-----|
| 0 |   0 |
| 1 |   1 |
| 2 |   4 |
| 3 |   9 |
| 4 |  16 |


#+name: table2
#+begin_src emacs-lisp :cache yes
  (sleep-for 5)
  (setq s ( list (list "x" "x^2") 'hline))
  (dotimes (i 5 s) (setq s (append s (list (cons i (list (* i i)))))))
  s
#+end_src

#+name: tf
| festfile.pdf ]

#+RESULTS[46320b31c46cef901580bad78aee7032d97ffe64]: table2
| x | x^2 |
|---+-----|
| 0 |   0 |
| 1 |   1 |
| 2 |   4 |
| 3 |   9 |
| 4 |  16 |

#+name: testplot
#+begin_src gnuplot :var data=table0 :exports results
  reset
  plot data with linespoints
#+end_src

HTH,

--
Emmanuel Charpentier


[-- Attachment #2: Type: text/html, Size: 2727 bytes --]

Re: when does :cache not cache?

[Eric S Fraga]

On Sunday, 24 Oct 2021 at 13:49, Emmanuel Charpentier wrote:
> Workaround : cache the computations,not the plotting itself (which
> should be fast,and must be made on every table, anyway...) :

actually, the problem in my case is that the plotting *is* the expensive
part!  In case, problem solved/avoided by using ":eval never-export".

thank you,
eric
-- 
: Eric S Fraga via Emacs 28.0.60, Org release_9.5-163-g4eab5b
: Latest paper written in org: https://arxiv.org/abs/2106.05096