Confused about source code blocks evaluation when exporting

Confused about source code blocks evaluation when exporting

[Alain.Cochard]

Hello.   The manual says:

     Org evaluates code in the following circumstances:

  _Source code blocks_
       Org evaluates source code blocks in an Org file during export.
       Org also evaluates a source code block with the ‘C-c C-c’ key
       chord.

NB: each test below is performed with a new 'emacs -Q' session.  I
have: Org mode version 9.5.2 (release_9.5.2-25-gaf6f12 @
/home/cochard/usr/local/emacs-28.1/lisp/org/)

(test 1) I have an org file which only contains

   #+BEGIN_SRC emacs-lisp 
   (message "foo")
   (setq my-var 'bar)
   #+END_SRC

If I do 'C-c C-c', I am asked the following

   Evaluate this emacs-lisp code block on your system? (yes or no) 

If I answer 'yes', the following block

   #+RESULTS:
   : bar

is inserted below the source block; also I see, in the *Messages*
buffer:

   executing Emacs-Lisp code block...
   foo
   Code block evaluation complete.

and if if 'C-h v' for 'my-var', it says

   my-var’s value is ‘bar’

So far so good.  (Well almost: why does '#+RESULTS:' show 'bar' and
not 'foo'?  After all, I'm just setting 'my-var', I'm not asking to
display its value.  But that's incidental.)

(test 2) Same org file.  If I do 'C-c C-e l o', no question is asked,
no '#+RESULTS:' block is inserted, and the resulting pdf file shows:

   Contents
   (message "foo")
   (setq my-var ’bar)

The *Messages* buffer shows

   org-babel-exp process emacs-lisp at position 1...
   Saving file /home/cochard/tmp/scr.tex...
   Wrote /home/cochard/tmp/scr.tex
   Processing LaTeX file scr.tex...
   PDF file produced.
   Running /usr/bin/xdg-open /home/cochard/tmp/scr.pdf...done

and 'my-var' is not understood as a variable.

So I don't understand.  I can only conclude that the code is not
evaluated, which seems to me contradictory with "Org evaluates source
code blocks in an Org file during export".

(test 3) Same code, except that I change the 1st line to

   #+BEGIN_SRC emacs-lisp :exports results 

If I do 'C-c C-e l o', I am asked the same question as in test 1:

   Evaluate this emacs-lisp code block on your system? (yes or no) 

but even if I answer 'yes', no '#+RESULTS:' block is inserted. The
*Messages* buffer shows

   org-babel-exp process emacs-lisp at position 1...
   executing Emacs-Lisp code block...
   foo
   Code block evaluation complete.
   Saving file /home/cochard/tmp/scr.tex...
   Wrote /home/cochard/tmp/scr.tex
   Processing LaTeX file scr.tex...
   PDF file produced.

and the pdf file shows:

   Contents

   bar

and my-var is defined.  So the code is evaluated this time, but I
don't really understand the whole process either...

For example, what should I do if I want the code to be evaluated, the
'#+RESUTS:' block to be inserted in the org file, but nothing to be
shown in the exported pdf? (i.e., neither the code nor its results)

Same question for code evaluation but nothing related to this code
inserted or exported.

I've tried using the ':eval' header argument but it never worked as I
expected.


Thank you for you help.

-- 
EOST (École et Observatoire des Sciences de la Terre) 
ITE (Institut Terre & Environnement) | alain.cochard@unistra.fr
5 rue René Descartes   [bureau 106]  | Phone: +33 (0)3 68 85 50 44 
F-67084 Strasbourg Cedex, France     | [ slot available for rent ]

Re: Confused about source code blocks evaluation when exporting

[Fraga, Eric]

On Tuesday, 12 Jul 2022 at 14:53, Alain.Cochard@unistra.fr wrote:
> So far so good.  (Well almost: why does '#+RESULTS:' show 'bar' and
> not 'foo'?  After all, I'm just setting 'my-var', I'm not asking to
> display its value.  But that's incidental.)

It's not incidental.  By default, the results of a src block evaluation
is the value of the src block which is typically the value returned by
the last statement.  You can modify this by adding

:results output

for instance to the begin_src line.

> (test 2) Same org file.  If I do 'C-c C-e l o', no question is asked,
> no '#+RESULTS:' block is inserted, and the resulting pdf file shows:

On export, the default is to export the code.  If you want the results,
you need to add

:exports results

or if you want both code and results,

:exports both

-- 
: Eric S Fraga, with org release_9.5.4-613-gb3b17c in Emacs 29.0.50

Re: Confused about source code blocks evaluation when exporting

[Greg Minshall]

Alain,

let's see.  maybe look in the org manual for ":results" (with "value" or
"output" as options) and ":exports" (with "code" or "results" or "both"
as options).

that might help?

cheers, Greg

Re: Confused about source code blocks evaluation when exporting

[Alain.Cochard]

Fraga, Eric writes on Tue 12 Jul 2022 14:08:

 > On Tuesday, 12 Jul 2022 at 14:53, Alain.Cochard@unistra.fr wrote:
 > > So far so good.  (Well almost: why does '#+RESULTS:' show 'bar' and
 > > not 'foo'?  After all, I'm just setting 'my-var', I'm not asking to
 > > display its value.  But that's incidental.)
 > 
 > It's not incidental.  By default, the results of a src block evaluation
 > is the value of the src block which is typically the value returned by
 > the last statement.

I guess I didn't use 'incidental' correctly.  I meant it as "not my
main concern".  Anyway, thank you; I could check that if I switch the
2 lines I indeed get '#+RESULTS:' to show foo'.

 > You can modify this by adding
 > 
 > :results output
 > 
 > for instance to the begin_src line.
 > 
 > > (test 2) Same org file.  If I do 'C-c C-e l o', no question is asked,
 > > no '#+RESULTS:' block is inserted, and the resulting pdf file shows:
 > 
 > On export, the default is to export the code.  If you want the results,
 > you need to add
 > 
 > :exports results
 > 
 > or if you want both code and results,
 > 
 > :exports both

Greg Minshall writes on Tue 12 Jul 2022 17:22:

 > maybe look in the org manual for ":results" (with "value" or
 > "output" as options) and ":exports" (with "code" or "results" or
 > "both" as options).

Thank you both, I made progress; but the behavior is still very so
unintuitive to me.  For example, I haven't found a way to have this
block:

#+begin_src emacs-lisp <arguments to be determined>
  (message "foo")
  (message "bar")
#+END_SRC

and have the following inserted during export:

#+RESULTS:
: foo
: bar

(after all, that is was goes into the *Messages* buffer upon
evaluation with 'C-c C-c'...)


Finally I still tend to consider that the sentence "Org evaluates
source code blocks in an Org file during export" is confusing.  But
maybe that's just I who interpret this as "code evaluation is the
default behavior upon export".

Regards

-- 
EOST (École et Observatoire des Sciences de la Terre) 
ITE (Institut Terre & Environnement) | alain.cochard@unistra.fr
5 rue René Descartes   [bureau 106]  | Phone: +33 (0)3 68 85 50 44 
F-67084 Strasbourg Cedex, France     | [ slot available for rent ]

Re: Confused about source code blocks evaluation when exporting

[Fraga, Eric]

On Tuesday, 12 Jul 2022 at 23:13, Alain.Cochard@unistra.fr wrote:
> I guess I didn't use 'incidental' correctly.  I meant it as "not my
> main concern".  Anyway, thank you; I could check that if I switch the
> 2 lines I indeed get '#+RESULTS:' to show foo'.

No, you used the word correctly.  My point was that this was your main
issue!  Understanding what is the "result" of a src block evaluation is
key.

> Thank you both, I made progress; but the behavior is still very so
> unintuitive to me.  For example, I haven't found a way to have this
> block:
>
> #+begin_src emacs-lisp <arguments to be determined>
>   (message "foo")
>   (message "bar")
> #+END_SRC

If you want output, maybe use "print" instead of "message".  However,
you will get quotes around the strings.  The best alternative would be
to have a single elisp statement that returns what you want output:

#+begin_src emacs-lisp :results value
  (concat "Foo"
          "\n"
          "Bar")
#+end_src

> Finally I still tend to consider that the sentence "Org evaluates
> source code blocks in an Org file during export" is confusing.  But
> maybe that's just I who interpret this as "code evaluation is the
> default behavior upon export".

I do not quite understand the difference.  Evaluation happens (subject
to confirmation).  What may happen with that evaluation, however, might
not be what you want by default.  What is it you want on export?  If you
want the code and the results, add ":exports both" to the begin_src
line.  Straightforward?

-- 
: Eric S Fraga, with org release_9.5.4-623-gc66bdb in Emacs 29.0.50

Re: Confused about source code blocks evaluation when exporting

[Alain.Cochard]

Fraga, Eric writes on Wed 13 Jul 2022 10:57:

 > No, you used the word correctly.  My point was that this was your
 > main issue!  Understanding what is the "result" of a src block
 > evaluation is key.

Thank you for the clarification.  I can see that I am still very
confused.

 > > Thank you both, I made progress; but the behavior is still very so
 > > unintuitive to me.  For example, I haven't found a way to have this
 > > block:
 > >
 > > #+begin_src emacs-lisp <arguments to be determined>
 > >   (message "foo")
 > >   (message "bar")
 > > #+END_SRC
 > 
 > If you want output, maybe use "print" instead of "message".  However,
 > you will get quotes around the strings.  The best alternative would be
 > to have a single elisp statement that returns what you want output:
 > 
 > #+begin_src emacs-lisp :results value
 >   (concat "Foo"
 >           "\n"
 >           "Bar")
 > #+end_src

I can now also see that the issue is coupled to my unfamiliarity with
the lisp language, so below I switch to something hopefully simpler
for me.

 > > Finally I still tend to consider that the sentence "Org evaluates
 > > source code blocks in an Org file during export" is confusing.  But
 > > maybe that's just I who interpret this as "code evaluation is the
 > > default behavior upon export".
 > 
 > I do not quite understand the difference.  Evaluation happens (subject
 > to confirmation).  What may happen with that evaluation, however, might
 > not be what you want by default.  What is it you want on export?  If you
 > want the code and the results, add ":exports both" to the begin_src
 > line.  Straightforward?

Nah, sorry.  Maybe I don't understand what "evaluation" is.

I have the file with only the following block in it:

#+begin_src shell
rm -f foo
#+end_src

I load it with 'emacs -Q -l ~/.emacs.git' 

where '~/.emacs.git' only contains

(add-to-list 'load-path "~/Org/Coch-git/org-mode/lisp")
(org-babel-do-load-languages
 'org-babel-load-languages 
 '((shell . t)))

I do 'C-c C-c' within the block; I am asked "Evaluate this shell code
block on your system?", I say 'yes', and I can see that that the file
'foo', which I had just created, is no longer there.  This makes sense
to me.  "evaluation" here seems to me to be a synonym for "execution".

Now I start again, but I do 'C-c C-e l o' instead.  I am _not_ asked
whether I want to evaluate, and the 'foo' file is still there.  But
the pdf file does display

rm -f foo

Does this still qualify as "evaluation"?  I thought not, hence my
conclusion that evaluation was not performed by default upon export,
but you made me doubt it...

Regards.  Hope you appreciated your beer.


Org mode version 9.5.4 (release_9.5.4-623-gc66bdb @
/home/cochard/Org/Coch-git/org-mode/lisp/) 

-- 
EOST (École et Observatoire des Sciences de la Terre) 
ITE (Institut Terre & Environnement) | alain.cochard@unistra.fr
5 rue René Descartes   [bureau 106]  | Phone: +33 (0)3 68 85 50 44 
F-67084 Strasbourg Cedex, France     | [ slot available for rent ]

Re: Confused about source code blocks evaluation when exporting

[Rudolf Adamkovič]

"Fraga, Eric" <e.fraga@ucl.ac.uk> writes:

> By default, the results of a src block evaluation is the value of the
> src block which is typically the value returned by the last statement.

The value of the last expression, not the last statement.

Rudy
-- 
"Strange as it may sound, the power of mathematics rests on its evasion
of all unnecessary thought and on its wonderful saving of mental
operations."
-- Ernst Mach, 1838-1916

Rudolf Adamkovič <salutis@me.com> [he/him]
Studenohorská 25
84103 Bratislava
Slovakia

Re: Confused about source code blocks evaluation when exporting

[Rudolf Adamkovič]

Alain.Cochard@unistra.fr writes:

> Org evaluates source code blocks in an Org file during export.

I have just tried to export the following document:

#+begin_src emacs-lisp :exports both
(+ 1 1)
#+end_src

The exported HTML file contains the expression '(+ 1 1)' along with '2',
its value.  It follows that Emacs does evaluate source blocks on export.

(Note that Org did not add RESULTS to the Org file during export, for no
export should ever modify the source file in any way.)

> why does '#+RESULTS:' show 'bar' and not 'foo'?

By default, the Org mode captures the value of the last expression.  In
your case, the last expression has the value of 'bar'.

If you want to capture something else for the result, such as standard
output, you must tell Org Babel about it with the ':results' argument.
It makes sense, if you think about it.  The computer cannot possibly
know what you want to capture.

Rudy
-- 
"The introduction of suitable abstractions is our only mental aid to
organize and master complexity."
-- Edsger Wybe Dijkstra, 1930-2002

Rudolf Adamkovič <salutis@me.com> [he/him]
Studenohorská 25
84103 Bratislava
Slovakia

Re: Confused about source code blocks evaluation when exporting

[Alain.Cochard]

Rudolf Adamkovič writes on Thu 14 Jul 2022 08:06:
 > Alain.Cochard@unistra.fr writes:
 > 
 > > Org evaluates source code blocks in an Org file during export.
 > 
 > I have just tried to export the following document:
 > 
 > #+begin_src emacs-lisp :exports both
 > (+ 1 1)
 > #+end_src
 > 
 > The exported HTML file contains the expression '(+ 1 1)' along with
 > '2', its value.  It follows that Emacs does evaluate source blocks
 > on export.

Yes, but it is because you used ':exports both'.  Without that, I only
have '(+ 1 1)'.  Would you not still speak about the "exported HTML
file"?

a.

(Thanks for the other precisions.)


-- 
EOST (École et Observatoire des Sciences de la Terre) 
ITE (Institut Terre & Environnement) | alain.cochard@unistra.fr
5 rue René Descartes   [bureau 106]  | Phone: +33 (0)3 68 85 50 44 
F-67084 Strasbourg Cedex, France     | [ slot available for rent ]

Re: Confused about source code blocks evaluation when exporting

[Fraga, Eric]

On Wednesday, 13 Jul 2022 at 23:06, Alain.Cochard@unistra.fr wrote:
> Now I start again, but I do 'C-c C-e l o' instead.  I am _not_ asked
> whether I want to evaluate, and the 'foo' file is still there.  But
> the pdf file does display
>
> rm -f foo
>
> Does this still qualify as "evaluation"?  I thought not, hence my
> conclusion that evaluation was not performed by default upon export,
> but you made me doubt it...

If you do not specify either ":exports results" or ":exports both", then
the exporter doesn't output the results and hence there is no need to
evaluate the src block.

The manual is indeed a bit vague about this so suggestions for
improvements are always welcome.

-- 
Eric S Fraga, @ericsfraga:matrix.org, GnuPG: 0xc89193d8fffcf67d

Re: Confused about source code blocks evaluation when exporting

[Alain.Cochard]

Fraga, Eric writes on Thu 14 Jul 2022 07:06:
 > On Wednesday, 13 Jul 2022 at 23:06, Alain.Cochard@unistra.fr wrote:
 > > Now I start again, but I do 'C-c C-e l o' instead.  I am _not_ asked
 > > whether I want to evaluate, and the 'foo' file is still there.  But
 > > the pdf file does display
 > >
 > > rm -f foo
 > >
 > > Does this still qualify as "evaluation"?  I thought not, hence my
 > > conclusion that evaluation was not performed by default upon export,
 > > but you made me doubt it...
 > 
 > If you do not specify either ":exports results" or ":exports both",
 > then the exporter doesn't output the results and hence there is no
 > need to evaluate the src block.

Why "there is no need"?  It was not a theoretical issue for me.
Without latexmk installed, I had to setq a different value for
org-latex-pdf-process depending on whether I used

#+cite_export: natbib plainnat

or

#+cite_export: biblatex

Sure, I can do this is in the emacs init file but then I don't how any
better than to restart emacs.  So I tried to do it on the fly with an
src block of the form

#+begin_src emacs-lisp 
(setq org-latex-pdf-process <for natib or for biblatex>)
#+end_src

which I want to be evaluated during export but I want neither the code
not its results to appear on the exported pdf file.

Maybe there are smarter ways to achieve this goal, but it was an
opportunity for me anyway to start understanding this evaluation
business.

There are other instances where this could be useful: evaluate some
shell block for deleting a file on the system before evaluating some
fortran code depending on this code.

 > The manual is indeed a bit vague about this so suggestions for
 > improvements are always welcome.

How about changing

   Org evaluates source code blocks in an Org file during export.

to

   Org generally evaluates source code blocks in an Org file during
   export.

But ideally it would be expanded, perhaps with something like
"depending on the value of some header arguments (see xxx)".

Also, how about a 'yes' option to the eval header argument to call for
unconditional evaluation?  Sorry if this would not make sense, I'm
trying...

Thank you all and best wishes.

-- 
EOST (École et Observatoire des Sciences de la Terre) 
ITE (Institut Terre & Environnement) | alain.cochard@unistra.fr
5 rue René Descartes   [bureau 106]  | Phone: +33 (0)3 68 85 50 44 
F-67084 Strasbourg Cedex, France     | [ slot available for rent ]

Re: Confused about source code blocks evaluation when exporting

[Fraga, Eric]

On Thursday, 14 Jul 2022 at 10:09, Alain.Cochard@unistra.fr wrote:
> #+begin_src emacs-lisp
> (setq org-latex-pdf-process <for natib or for biblatex>)
> #+end_src
>
> which I want to be evaluated during export but I want neither the code
> not its results to appear on the exported pdf file.
>
> Maybe there are smarter ways to achieve this goal, but it was an
> opportunity for me anyway to start understanding this evaluation
> business.

For this kind of setting, I use file local variables and #+bind:.
Evaluating this code snippet during the export process is probably not
what you want.  You want it set prior to exporting?

-- 
Eric S Fraga, @ericsfraga:matrix.org, GnuPG: 0xc89193d8fffcf67d

Re: Confused about source code blocks evaluation when exporting

[Alain.Cochard]

Fraga, Eric writes on Thu 14 Jul 2022 08:35:
 > On Thursday, 14 Jul 2022 at 10:09, Alain.Cochard@unistra.fr wrote:
 > > #+begin_src emacs-lisp
 > > (setq org-latex-pdf-process <for natib or for biblatex>)
 > > #+end_src
 > >
 > > which I want to be evaluated during export but I want neither the code
 > > not its results to appear on the exported pdf file.
 > >
 > > Maybe there are smarter ways to achieve this goal, but it was an
 > > opportunity for me anyway to start understanding this evaluation
 > > business.
 > 
 > For this kind of setting, I use file local variables and #+bind:.
 > Evaluating this code snippet during the export process is probably not
 > what you want.  You want it set prior to exporting?

Thanks a lot.

I don't know if I want to set org-latex-pdf-process _during_ or
_prior_ exporting.  All I wanted to do is to _not_ set it in the emacs
init file, so as to be able to set a specific value depending on
whether I was going to use natbib or biblatex, without having to
restart emacs.

I tried doing something with what you suggest (using #+bind:), but
failed:

I put the following at the end of my org file (then visited the file
again):

# Local Variables:
# org-export-allow-bind-keywords: t
# End:

As I was not sure about the bind syntax for the complicated
org-latex-pdf-process variable, I tried this first test:

#+bind: org-latex-pdf-process "dummy"
# (I also tried without the quotes)
#+cite_export: natbib plainnat
#+bibliography: cite.bib
[cite:@chouet88] 

And then 'C-c C-e l o'.  There is no error but the generated pdf is
not correct, because (as mentioned some days ago)
org-latex-pdf-process does not have the proper value.

In other words, '#+bind:' does not work as expected.

(By contrast, if I use "my" method, using

#+begin_src emacs-lisp :exports results
(setq org-latex-pdf-process 'dummy)
#+end_src

then, after 'C-c C-e l o' I get "org-latex-compile: Wrong type
argument: sequencep, dummy".
)

This post

https://emacs.stackexchange.com/questions/47474/setting-variable-for-org-mode-html-export

looked promising, but it also points to this one

https://stackoverflow.com/questions/41964712/how-does-bind-really-works/42012392#42012392

which looks less so...


At any rate, and as I already said, all this business became
irrelevant for me once latexmk was installed: no need then to set
org-latex-pdf-process.  This became only a matter of better
understanding some of the org intricacies.

Thank you again for your time.



















-- 
EOST (École et Observatoire des Sciences de la Terre) 
ITE (Institut Terre & Environnement) | alain.cochard@unistra.fr
5 rue René Descartes   [bureau 106]  | Phone: +33 (0)3 68 85 50 44 
F-67084 Strasbourg Cedex, France     | [ slot available for rent ]

Re: Confused about source code blocks evaluation when exporting

[Fraga, Eric]

On Wednesday, 20 Jul 2022 at 14:12, Alain.Cochard@unistra.fr wrote:
> At any rate, and as I already said, all this business became
> irrelevant for me once latexmk was installed: no need then to set
> org-latex-pdf-process.  This became only a matter of better
> understanding some of the org intricacies.

Maybe try this for your local variables (at the end of the org file):

# Local Variables:
# org-latex-pdf-process: ("lualatex %f" "bibtex %b" "lualatex %f" "lualatex %f")
# End:

as a test (adjusting as you see fit)?  This works for me.  No need to
use BIND.

-- 
: Eric S Fraga, with org release_9.5.4-648-gdf1814 in Emacs 29.0.50

Re: Confused about source code blocks evaluation when exporting

[Alain.Cochard]

Fraga, Eric writes on Wed 20 Jul 2022 12:43:

 > Maybe try this for your local variables (at the end of the org file):
 > 
 > # Local Variables:
 > # org-latex-pdf-process: ("...")
 > # End:
 > 
 > as a test (adjusting as you see fit)?  This works for me.  No need to
 > use BIND.

Indeed, it works for me as well.  Thanks.  But the advantage of not
using 'setq' comes at the detriment of keeping related things together
in the file. Plus one must revisit the file when changing the value of
that variable.  So it seems there is no satisfactory solution...




-- 
EOST (École et Observatoire des Sciences de la Terre) 
ITE (Institut Terre & Environnement) | alain.cochard@unistra.fr
5 rue René Descartes   [bureau 106]  | Phone: +33 (0)3 68 85 50 44 
F-67084 Strasbourg Cedex, France     | [ slot available for rent ]

Re: Confused about source code blocks evaluation when exporting

[Fraga, Eric]

On Friday, 22 Jul 2022 at 06:27, Alain.Cochard@unistra.fr wrote:
> Plus one must revisit the file when changing the value of
> that variable.  

You can reload local variables by hitting C-c C-c on the #+title: line,
for instance.

-- 
: Eric S Fraga, with org release_9.5.4-557-g3bbbf7 in Emacs 29.0.50

Re: Confused about source code blocks evaluation when exporting

[Alain.Cochard]

Fraga, Eric writes on Fri 22 Jul 2022 17:09:
 > On Friday, 22 Jul 2022 at 06:27, Alain.Cochard@unistra.fr wrote:
 > > Plus one must revisit the file when changing the value of
 > > that variable.  
 > 
 > You can reload local variables by hitting C-c C-c on the #+title: line,
 > for instance.

Ah OK; thanks.  But it just dawns on me that it is faster to type

C-x C-v <RET> 

(provided save-place-mode is enabled)

-- 
EOST (École et Observatoire des Sciences de la Terre) 
ITE (Institut Terre & Environnement) | alain.cochard@unistra.fr
5 rue René Descartes   [bureau 106]  | Phone: +33 (0)3 68 85 50 44 
F-67084 Strasbourg Cedex, France     | [ slot available for rent ]