Best practice for writing and debugging shell source blocks?

Best practice for writing and debugging shell source blocks?

[Tim Landscheidt]

Hi,

inspired by "Literate DevOps"
(https://howardism.org/Technical/Emacs/literate-devops.html),
I want to document some "stuff" in an Org file with the goal
to be able to replay the steps done in the future so that I
can either recreate the same "product" or variations there-
of.

My actual topic involves (inter alia) containers, but for
simplicity, let us assume that I want to document the se-
quence of shell commands:

| $ mktemp
| /tmp/tmp.gSffc4XtQ0
| $ echo a >> /tmp/tmp.gSffc4XtQ0
| $ echo b >> /tmp/tmp.gSffc4XtQ0
| $ fgrep -x a /tmp/tmp.gSffc4XtQ0
| a
| $

In other words: An "entity" is created and its identifier
shown, this entity is then referenced by this identifier,
changed and queried.

If I put the shell commands in one source block each:

| #+NAME: create-temporary-file
| #+BEGIN_SRC sh
| mktemp
| #+END_SRC

| #+BEGIN_SRC sh :var tmpfile=create-temporary-file
| echo a >> $tmpfile
| #+END_SRC

| #+BEGIN_SRC sh :var tmpfile=create-temporary-file
| echo b >> $tmpfile
| #+END_SRC

| #+BEGIN_SRC sh :var tmpfile=create-temporary-file
| fgrep -x a $tmpfile
| #+END_SRC

and then evaluate the last source block, it calls the source
block create-temporary-file, then searches the (empty) file
and fails, i. e. the second and third source blocks are not
evaluated at all.  (Dealing with failing shell commands is
another headache—thankfully, the trick at
https://emacs.stackexchange.com/questions/59875/org-src-block-does-not-return-any-output
(provide header arguments :prologue "exec 2>&1" and
:epilogue ":") works reasonably well.)

Adding more to the confusion, if I manually evaluate the
first source block (create-temporary-file), it produces for
example:

| #+RESULTS: create-temporary-file
| : /tmp/tmp.lOKZtyJ124

If I then evaluate the last source block (again), the first
code block gets called, but its (different) result gets only
passed to the last source block and used there, while the
"#+RESULTS" section stays the same.

The same thing happens if I add ":session my-test" to all
source blocks, except that I get an additional buffer
"my-test" with the content:

| mktemp
| sh-5.2$ echo 'org_babel_sh_eoe'
| /tmp/tmp.e19GfJsH7w
| sh-5.2$ org_babel_sh_eoe
| sh-5.2$ tmpfile='/tmp/tmp.e19GfJsH7w'
| sh-5.2$ fgrep -x a $tmpfile
| sh-5.2$ echo 'org_babel_sh_eoe'
| org_babel_sh_eoe
| sh-5.2$

where the "mixed" output of "mktemp" and "echo
'org_babel_sh_eoe'" does not necessarily instill confidence
to run commands that are destructive in nature.

So what is The Right Way™ to document sequences of shell
commands so that they are evaluated in sequence, being able
to refer to the output of previous commands?  Are there any
obvious collections of examples that I missed?

TIA,
Tim

Re: Best practice for writing and debugging shell source blocks?

[Max Nikulin]

On 24/10/2023 21:54, Tim Landscheidt wrote:
> The same thing happens if I add ":session my-test" to all source blocks,

Have you tried to combine variables and session?

#+name: get-random-value
#+begin_src bash
   echo "$RANDOM"
#+end_src

#+header: :var random_var=get-random-value
#+begin_src sh :session *sh-with-random*
   echo "$random_var + 10"
#+end_src

#+RESULTS:
: 19545 + 10

#+begin_src sh :session *sh-with-random*
   echo "$random_var + 20"
#+end_src

#+RESULTS:
: 19545 + 20

#+begin_src sh :session *sh-with-random*
   echo "$random_var + 30"
#+end_src

#+RESULTS:
: 19545 + 30

Or with noweb reference


#+begin_src sh :noweb-ref use-random
   echo "$random_var + 10"
#+end_src

#+begin_src sh :noweb-ref use-random
   echo "$random_var + 20"
#+end_src

#+begin_src sh :noweb-ref use-random
   echo "$random_var + 30"
#+end_src

#+begin_src sh :var random_var=get-random-value :noweb yes
   <<use-random>>
#+end_src


P.S. For the case that you have not seen it, the following page contains 
some examples:
https://orgmode.org/worg/org-contrib/babel/languages/ob-doc-shell.html