org-agenda-write taking very long (probably because of babel)

org-agenda-write taking very long (probably because of babel)

[Karl Voit]

Hi!

Org-mode 7.9.3f (692f053d8 Wed Feb 27 14:49:46 2013 +0100)

I am using these two lines to export an agenda to an iCal file:
  (org-agenda-list nil nil 60)
  (org-agenda-write "~/share/all/org-mode/org-export.ics")

Since my Org-mode update from today (from
5d467d6f8affc0afe34922e885ac6e2492ddd091 Fri Feb 15 15:28:35 2013
+0100) it takes very long to export the ics file.

I guess this relates to ...

  org-babel-exp processing... [25 times]

... which also pops up some babel result graphics which did not
happen before.

Was there a change in the default settings or is this a bug?

Thanks!

-- 
Karl Voit

Re: org-agenda-write taking very long (probably because of babel)

[Karl Voit]

* Karl Voit <devnull@Karl-Voit.at> wrote:
> Hi!
>
> Org-mode 7.9.3f (692f053d8 Wed Feb 27 14:49:46 2013 +0100)
>
> I am using these two lines to export an agenda to an iCal file:
>   (org-agenda-list nil nil 60)
>   (org-agenda-write "~/share/all/org-mode/org-export.ics")
>
> Since my Org-mode update from today (from
> 5d467d6f8affc0afe34922e885ac6e2492ddd091 Fri Feb 15 15:28:35 2013
> +0100) it takes very long to export the ics file.
>
> I guess this relates to ...
>
>   org-babel-exp processing... [25 times]
>
> ... which also pops up some babel result graphics which did not
> happen before.

What I just found out: there were open buffers "2012-01-17-R" (an R
session I used in an agenda file) and "*gnuplot*", both with
running(!) sessions.

> Was there a change in the default settings or is this a bug?
> Thanks!

-- 
Karl Voit

Re: org-agenda-write taking very long (probably because of babel)

[Bastien]

Hi Karl,

Karl Voit <devnull@Karl-Voit.at> writes:

> Was there a change in the default settings or is this a bug?

Maybe this change:
http://orgmode.org/cgit.cgi/org-mode.git/commit/?id=091bf02

Achim?

-- 
 Bastien

Re: org-agenda-write taking very long (probably because of babel)

[Achim Gratz]

Bastien writes:
>> Was there a change in the default settings or is this a bug?
>
> Maybe this change:
> http://orgmode.org/cgit.cgi/org-mode.git/commit/?id=091bf02

I cannot see how.  If the cache is current, the new code does _less_
work than it would have before the change.  If the cache was stale to
begin with, the files would have been re-made with the old code anyhow.

Does this happen each time and if yes, what source blocks are involved?


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptations for Waldorf Q V3.00R3 and Q+ V3.54R2:
http://Synth.Stromeko.net/Downloads.html#WaldorfSDada

Re: org-agenda-write taking very long (probably because of babel)

[Achim Gratz]

Achim Gratz writes:
> I cannot see how.

In fact I cannot even see how creating the agenda would run any src
block at all…


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Samples for the Waldorf Blofeld:
http://Synth.Stromeko.net/Downloads.html#BlofeldSamplesExtra

Re: org-agenda-write taking very long (probably because of babel)

[Achim Gratz]

Bastien writes:
> Maybe this change:
> http://orgmode.org/cgit.cgi/org-mode.git/commit/?id=091bf02

I think I see where this is coming from, org-babel-confirm-evaluate does
some additional things beyond what it's name implies.  I'm reverting the
original commit(s) for now until I've implemented this differently.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Samples for the Waldorf Blofeld:
http://Synth.Stromeko.net/Downloads.html#BlofeldSamplesExtra

Re: org-agenda-write taking very long (probably because of babel)

[Bastien]

Achim Gratz <Stromeko@nexgo.de> writes:

> Bastien writes:
>> Maybe this change:
>> http://orgmode.org/cgit.cgi/org-mode.git/commit/?id=091bf02
>
> I think I see where this is coming from, org-babel-confirm-evaluate does
> some additional things beyond what it's name implies.  I'm reverting the
> original commit(s) for now until I've implemented this differently.

Thanks in advance,

-- 
 Bastien

Re: org-agenda-write taking very long (probably because of babel)

[Achim Gratz]

[-- Attachment #1: Type: text/plain, Size: 626 bytes --]

Karl Voit writes:
> I guess this relates to ...
>
>   org-babel-exp processing... [25 times]
>
> ... which also pops up some babel result graphics which did not
> happen before.
>
> Was there a change in the default settings or is this a bug?

Yes there was, sorry for that.  I missed something that was going on in
an otherwise inconspicously named function and that unintentionally
changed the conditions for when source blocks are eligible for
evaluation.  Since your setup was affected, may I ask you to apply this
patch after the next update (which should include 302d3780ec) and report
if this fixes the issue you had?


[-- Attachment #2: 0001-ob-core-do-not-ask-for-confirmation-if-cached-result.patch --]
[-- Type: text/x-patch, Size: 8004 bytes --]

From ac8841d2af9fcc490e5d98cb63eaaf74d3ba73f7 Mon Sep 17 00:00:00 2001
From: Achim Gratz <Stromeko@Stromeko.DE>
Date: Wed, 27 Feb 2013 22:55:26 +0100
Subject: [PATCH] ob-core: do not ask for confirmation if cached result is
 current
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

* lisp/ob-core.el (org-babel-execute-src-block): Do not ask for
  confirmation if the cached result is current.  Since
  `org-babel-confirm-evaluate´ does additional things besides asking
  for confirmation, call it first with `org-confirm-babel-evaluate´
  bound to nil.  This has the effect that it will never ask the user,
  but will indicate if the block should be evaluated.  If yes,
  determine whether the cached result block is current (this is
  deferred until now since `org-babel-process-params´ might trigger
  expensive operations).  If `cache-current-p´ is t or
  `org-confirm-babel-evaluate´ is nil, evaluate the source block
  without asking.  In case the cache is current the evaluation will
  not actually do anything but return the cached value, so this is
  safe.  In case `org-confirm-babel-evaluate´ is nil the user would
  not be asked anyway, so the call of `org-babel-confirm-evaluate´ is
  not necessary.  Otherwise run `org-babel-confirm-evaluate´ again to
  ask permission from the user and act depending on the answer.
---
 lisp/ob-core.el | 141 +++++++++++++++++++++++++++++---------------------------
 1 file changed, 73 insertions(+), 68 deletions(-)

diff --git a/lisp/ob-core.el b/lisp/ob-core.el
index 275a4f7..8b6b2a6 100644
--- a/lisp/ob-core.el
+++ b/lisp/ob-core.el
@@ -523,80 +523,85 @@ (defun org-babel-execute-src-block (&optional arg info params)
   (interactive)
   (let* ((info (or info (org-babel-get-src-block-info)))
 	 (merged-params (org-babel-merge-params (nth 2 info) params)))
-    (when (org-babel-confirm-evaluate
-	   (let ((i info)) (setf (nth 2 i) merged-params) i))
-      (let* ((lang (nth 0 info))
-	     (params (if params
+    (when (let ((org-confirm-babel-evaluate nil)) ;; do not ask yet
+	    (org-babel-confirm-evaluate
+	     (let ((i info)) (setf (nth 2 i) merged-params) i)))
+      (let* ((params (if params
 			 (org-babel-process-params merged-params)
 		       (nth 2 info)))
 	     (cache-p (and (not arg) (cdr (assoc :cache params))
 			  (string= "yes" (cdr (assoc :cache params)))))
-	     (result-params (cdr (assoc :result-params params)))
 	     (new-hash (when cache-p (org-babel-sha1-hash info)))
 	     (old-hash (when cache-p (org-babel-current-result-hash)))
-	     (cache-current-p (and (not arg) new-hash (equal new-hash old-hash)))
-	     (body (setf (nth 1 info)
-			 (if (org-babel-noweb-p params :eval)
-			     (org-babel-expand-noweb-references info)
-			   (nth 1 info))))
-	     (dir (cdr (assoc :dir params)))
-	     (default-directory
-	       (or (and dir (file-name-as-directory (expand-file-name dir)))
-		   default-directory))
-	     (org-babel-call-process-region-original
-	      (if (boundp 'org-babel-call-process-region-original)
-		  org-babel-call-process-region-original
-		(symbol-function 'call-process-region)))
-	     (indent (car (last info)))
-	     result cmd)
-	(unwind-protect
-	    (let ((call-process-region
-		   (lambda (&rest args)
-		     (apply 'org-babel-tramp-handle-call-process-region args))))
-	      (let ((lang-check (lambda (f)
-				  (let ((f (intern (concat "org-babel-execute:" f))))
-				    (when (fboundp f) f)))))
-		(setq cmd
-		      (or (funcall lang-check lang)
-			  (funcall lang-check (symbol-name
-					       (cdr (assoc lang org-src-lang-modes))))
-			  (error "No org-babel-execute function for %s!" lang))))
-	      (if cache-current-p
-		  (save-excursion ;; return cached result
-		    (goto-char (org-babel-where-is-src-block-result nil info))
-		    (end-of-line 1) (forward-char 1)
-		    (setq result (org-babel-read-result))
-		    (message (replace-regexp-in-string
-			      "%" "%%" (format "%S" result))) result)
-		(message "executing %s code block%s..."
-			 (capitalize lang)
-			 (if (nth 4 info) (format " (%s)" (nth 4 info)) ""))
-		(if (member "none" result-params)
-		    (progn
-		      (funcall cmd body params)
-		      (message "result silenced"))
-		(setq result
-		      ((lambda (result)
-			 (if (and (eq (cdr (assoc :result-type params)) 'value)
-				  (or (member "vector" result-params)
-				      (member "table" result-params))
-				  (not (listp result)))
-			     (list (list result)) result))
-		       (funcall cmd body params)))
-		;; if non-empty result and :file then write to :file
-		(when (cdr (assoc :file params))
-		  (when result
-		    (with-temp-file (cdr (assoc :file params))
-		      (insert
-		       (org-babel-format-result
-			result (cdr (assoc :sep (nth 2 info)))))))
-		  (setq result (cdr (assoc :file params))))
-		(org-babel-insert-result
-		 result result-params info new-hash indent lang)
-		(run-hooks 'org-babel-after-execute-hook)
-		result
-		)))
-	  (setq call-process-region 'org-babel-call-process-region-original))))))
+	     (cache-current-p (and (not arg) new-hash (equal new-hash old-hash))))
+	(when (or cache-current-p
+		  (null org-confirm-babel-evaluate)
+		  (org-babel-confirm-evaluate ;; perhaps ask now
+		   (let ((i info)) (setf (nth 2 i) merged-params) i)))
+	  (let* ((lang (nth 0 info))
+		 (result-params (cdr (assoc :result-params params)))
+		 (body (setf (nth 1 info)
+			     (if (org-babel-noweb-p params :eval)
+				 (org-babel-expand-noweb-references info)
+			       (nth 1 info))))
+		 (dir (cdr (assoc :dir params)))
+		 (default-directory
+		   (or (and dir (file-name-as-directory (expand-file-name dir)))
+		       default-directory))
+		 (org-babel-call-process-region-original
+		  (if (boundp 'org-babel-call-process-region-original)
+		      org-babel-call-process-region-original
+		    (symbol-function 'call-process-region)))
+		 (indent (car (last info)))
+		 result cmd)
+	    (unwind-protect
+		(let ((call-process-region
+		       (lambda (&rest args)
+			 (apply 'org-babel-tramp-handle-call-process-region args))))
+		  (let ((lang-check (lambda (f)
+				      (let ((f (intern (concat "org-babel-execute:" f))))
+					(when (fboundp f) f)))))
+		    (setq cmd
+			  (or (funcall lang-check lang)
+			      (funcall lang-check (symbol-name
+						   (cdr (assoc lang org-src-lang-modes))))
+			      (error "No org-babel-execute function for %s!" lang))))
+		  (if cache-current-p
+		      (save-excursion ;; return cached result
+			(goto-char (org-babel-where-is-src-block-result nil info))
+			(end-of-line 1) (forward-char 1)
+			(setq result (org-babel-read-result))
+			(message (replace-regexp-in-string
+				  "%" "%%" (format "%S" result))) result)
+		    (message "executing %s code block%s..."
+			     (capitalize lang)
+			     (if (nth 4 info) (format " (%s)" (nth 4 info)) ""))
+		    (if (member "none" result-params)
+			(progn
+			  (funcall cmd body params)
+			  (message "result silenced"))
+		      (setq result
+			    ((lambda (result)
+			       (if (and (eq (cdr (assoc :result-type params)) 'value)
+					(or (member "vector" result-params)
+					    (member "table" result-params))
+					(not (listp result)))
+				   (list (list result)) result))
+			     (funcall cmd body params)))
+		      ;; if non-empty result and :file then write to :file
+		      (when (cdr (assoc :file params))
+			(when result
+			  (with-temp-file (cdr (assoc :file params))
+			    (insert
+			     (org-babel-format-result
+			      result (cdr (assoc :sep (nth 2 info)))))))
+			(setq result (cdr (assoc :file params))))
+		      (org-babel-insert-result
+		       result result-params info new-hash indent lang)
+		      (run-hooks 'org-babel-after-execute-hook)
+		      result
+		      )))
+	      (setq call-process-region 'org-babel-call-process-region-original))))))))
 
 (defun org-babel-expand-body:generic (body params &optional var-lines)
   "Expand BODY with PARAMS.
-- 
1.8.1.4


[-- Attachment #3: Type: text/plain, Size: 186 bytes --]



Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Samples for the Waldorf Blofeld:
http://Synth.Stromeko.net/Downloads.html#BlofeldSamplesExtra

Re: org-agenda-write taking very long (probably because of babel)

[Bastien]

Hi Achim,

Achim Gratz <Stromeko@nexgo.de> writes:

> * lisp/ob-core.el (org-babel-execute-src-block): Do not ask for
>   confirmation if the cached result is current.  Since
>   `org-babel-confirm-evaluate´ does additional things besides asking
>   for confirmation, call it first with `org-confirm-babel-evaluate´
>   bound to nil.  This has the effect that it will never ask the user,
>   but will indicate if the block should be evaluated.  If yes,
>   determine whether the cached result block is current (this is
>   deferred until now since `org-babel-process-params´ might trigger
>   expensive operations).  If `cache-current-p´ is t or
>   `org-confirm-babel-evaluate´ is nil, evaluate the source block
>   without asking.  In case the cache is current the evaluation will
>   not actually do anything but return the cached value, so this is
>   safe.  In case `org-confirm-babel-evaluate´ is nil the user would
>   not be asked anyway, so the call of `org-babel-confirm-evaluate´ is
>   not necessary.  Otherwise run `org-babel-confirm-evaluate´ again to
>   ask permission from the user and act depending on the answer.

Sorry to nitpick on this but please keep the Emacs-like change log
small, if not terse.  Additionnal details are more than welcome in
the commit message though.

Thanks!

-- 
 Bastien

Re: org-agenda-write taking very long (probably because of babel)

[Achim Gratz]

Bastien <bzg <at> altern.org> writes:
> Sorry to nitpick on this but please keep the Emacs-like change log
> small, if not terse.

I've thought about this, but then decided that the change does some seemingly
superfluous things and I'd better explain why they are necessary.

>  Additionnal details are more than welcome in
> the commit message though.

So I'll cut the changelog after the first sentence and relegate the rest of the
explanation to the commit message (if the patch works, of course) -- OK?


Regards,
Achim.

Re: org-agenda-write taking very long (probably because of babel)

[Bastien]

Hi Achim,

Achim Gratz <Stromeko@NexGo.DE> writes:

> So I'll cut the changelog after the first sentence and relegate the rest of the
> explanation to the commit message (if the patch works, of course) -- OK?

Yes.  Feel free to add all the useful information in the git
commit log, after the Emacs-ready change log.

Thanks!

-- 
 Bastien

Re: org-agenda-write taking very long (probably because of babel)

[Karl Voit]

Hi Achim!

Sorry, I was away from Org a couple of days. 

* Achim Gratz <Stromeko@nexgo.de> wrote:
>
> Karl Voit writes:
>> I guess this relates to ...
>>
>>   org-babel-exp processing... [25 times]
>>
>> ... which also pops up some babel result graphics which did not
>> happen before.
>>
>> Was there a change in the default settings or is this a bug?
>
> Yes there was, sorry for that.  I missed something that was going on in
> an otherwise inconspicously named function and that unintentionally
> changed the conditions for when source blocks are eligible for
> evaluation.  Since your setup was affected, may I ask you to apply this
> patch after the next update (which should include 302d3780ec) and report
> if this fixes the issue you had?

I updated to 50226db65d5cb176f3f5e027d668ef5de4937bde and tried to
apply your patch. Either because I never applied patch files before
(or I don't remember) or the code changed in between, I could not
apply it without getting errors.

So I tried to do the patch manually and I hope, I did not break
something. You can find the ob-core.el I generated on [1].

Org mode compiled and Emacs started without any error message.

When I exported my agenda, I got:
org-babel-exp processing... [24 times]
org-babel-execute-src-block: Symbol's function definition is void: cache-p

The pop-up window with generated bar charts did not appear this
time. But some test.png from one gnuplot (of three) was
re-generated.

HTH

Please reply if I can check something else for you.

  1. https://gist.github.com/novoid/6bace66e0e8c6a4d2a7f
-- 
Karl Voit

Re: org-agenda-write taking very long (probably because of babel)

[Achim Gratz]

Karl Voit writes:
> Since my Org-mode update from today (from
> 5d467d6f8affc0afe34922e885ac6e2492ddd091 Fri Feb 15 15:28:35 2013
> +0100) it takes very long to export the ics file.
>
> I guess this relates to ...
>
>   org-babel-exp processing... [25 times]
>
> ... which also pops up some babel result graphics which did not
> happen before.
>
> Was there a change in the default settings or is this a bug?

I have worked with Karl to try and find what caused this change in
behaviour and it wasn't anything to do with my patch or even recently,
but the switch from the old to the new exporter framework for producing
the iCalendar files.  The patch in question is buried in commit
0a01e52aa1:

--8<---------------cut here---------------start------------->8---
------------------------------ lisp/org-agenda.el ------------------------------
index 809287b..e9a9efc 100644
@@ -2361,11 +2361,11 @@ (defun org-agenda-mode ()
      ["Phases of the Moon" org-agenda-phases-of-moon (org-agenda-check-type nil 'agenda 'timeline)]
      ["Sunrise/Sunset" org-agenda-sunrise-sunset (org-agenda-check-type nil 'agenda 'timeline)]
      ["Holidays" org-agenda-holidays (org-agenda-check-type nil 'agenda 'timeline)]
      ["Convert" org-agenda-convert-date (org-agenda-check-type nil 'agenda 'timeline)]
      "--"
-     ["Create iCalendar File" org-export-icalendar-combine-agenda-files t])
+     ["Create iCalendar File" org-icalendar-combine-agenda-files t])
     "--"
     ["Undo Remote Editing" org-agenda-undo org-agenda-undo-list]
     "--"
     ("MobileOrg"
      ["Push Files and Views" org-mobile-push t]
@@ -3347,18 +3347,12 @@ (defun org-agenda-write (file &optional open nosettings agenda-bufname)
 			      (concat (file-name-sans-extension file) ".ps"))
 			     (expand-file-name file))
 	       (delete-file (concat (file-name-sans-extension file) ".ps"))
 	       (message "PDF written to %s" file))
 	      ((string-match "\\.ics\\'" file)
-	       (require 'org-icalendar)
-	       (let ((org-agenda-marker-table
-		      (org-create-marker-find-array
-		       (org-agenda-collect-markers)))
-		     (org-icalendar-verify-function 'org-check-agenda-marker-table)
-		     (org-combined-agenda-icalendar-file file))
-		 (apply 'org-export-icalendar 'combine
-			(org-agenda-files nil 'ifmode))))
+	       (require 'ox-icalendar)
+	       (org-icalendar-export-current-agenda (expand-file-name file)))
 	      (t
 	       (let ((bs (buffer-string)))
 		 (find-file file)
 		 (erase-buffer)
 		 (insert bs)
--8<---------------cut here---------------end--------------->8---

So, it seems that the old code did something to prevent source block
execution, while the new one does not handle this situation specially.
I know next to nothing about agendas or iCalendar export, so I would
appreciate if someone more knowledgeable could have a look.



Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Factory and User Sound Singles for Waldorf Q+, Q and microQ:
http://Synth.Stromeko.net/Downloads.html#WaldorfSounds

Re: org-agenda-write taking very long (probably because of babel)

[Bastien]

Hi Karl and Achim,

Achim Gratz <Stromeko@nexgo.de> writes:

> I have worked with Karl to try and find what caused this change in
> behaviour and it wasn't anything to do with my patch or even recently,
> but the switch from the old to the new exporter framework for producing
> the iCalendar files.  The patch in question is buried in commit
> 0a01e52aa1:

This is now fixed, thanks.

-- 
 Bastien