Re: [babel] How to kill two birds with one stone?

[Dan Davison <dandavison7@gmail.com>]

Hi Seb,

Cool post. I hope someone has some good ideas in this thread. Some quick
responses / questions below.

> #+TITLE:     Document a shell script as separate blocks
> #+DATE:      2011-02-04
> #+LANGUAGE:  en_US
>
> * Abstract
>
> When writing shell scripts, I'd like to kill *two* birds with one Babel stone:
>
> - Be able to *execute the script in situ*, so that I get a copy of the results
>   stored in (and "versioned" with) my documentation.
>
> - Be able to *chain the code blocks*, so that I tangle a "natural" script into
>   a file for later execution.
>
> I guess I currently miss some points in order to reach that goal in a clean
> way. Could you give me advice on how to get a step further down the route?
>
> * Sample code
>
> For the sake of clarity, let's take a simple problem: I'd like to generate a
> DOT graph of links between all files from a directory tree.

So if file a.org contains

-----
jhasg
[[file:b.org]]
[[file:c.org]]
hbjgv
-----

then you want an arrow from node a to node b and from node a to node c, right?

>
> The procedure:
>
> 1. (Recursively) list all files inside the directory.
>
> 2. For each file, (recursively) search for its name referenced in all the
>    files.
>
> 3. Generate a DOT representation of the link between files.
>
> ** List all files under current directory
>
> #+srcname: dw-file-tree
> #+begin_src sh :results output
> cd ~/Some-Project-Dir
> find . -type f -print | grep -v .svn | head -n 5
> #+end_src
>
> #+results: dw-file-tree
> #+begin_example
> ./.cvsignore
> ./charge_dim
> ./charge_fct
> ./compte
> ./controle_config
> #+end_example
>
> Here, I voluntary limit the number of results to the first 5 files, for the
> compactness of this example. This sample does not include any file from
> subdirectories, but it doesn't matter.
>
> ** Search recursively for anything about a file
>
> Search through all files (ignoring =.svn= directories) for any reference to
> filename given as parameter.
>
> #+srcname: search-files-pointing-to-this-file
> #+begin_src sh :results output :var f="charge_dim"
> cd ~/Some-Project-Dir
> find . -not \( -name .svn -prune \) -type f -print0 |\
> xargs -0 grep -i --files-with-matches "$f"
> #+end_src
>
> #+results: search-files-pointing-to-this-file
> #+begin_example
> ./.cvsignore
> ./charge_dim
> ./compte
> ./IFP/Chrg_dim
> ./IFP/Chrg_dim.avant_simple_recovery_mode_2008_03_12
> ./principal.env
> ./29Aalst/Chrg_dim
> ./29Aalst/Chrg_dim_interactif
> ./29Aalst/Publ_dim_interactif
> #+end_example
>
> HERE, I'M GIVING A FILENAME AS DEFAULT VALUE OF =f= IN ORDER TO SEE A SAMPLE
> RESULTS.
>
> ** Convert to a DOT representation
>
> For every file pointing to the file given in parameter, generate an "arrow"
> (edge) in DOT representation.
>
> #+srcname: dot-arrow-from-files-pointing-to-this-file
> #+begin_src sh :results output :var f="charge_dim" :var data=search-files-pointing-to-this-file
> for i in $(echo "$data"); do echo "    $(basename $i) -> $f"; done
> #+end_src
>
> #+results: dot-arrow-from-files-pointing-to-this-file
> #+begin_example
>     .cvsignore -> charge_dim
>     charge_dim -> charge_dim
>     compte -> charge_dim
>     Chrg_dim -> charge_dim
>     Chrg_dim.avant_simple_recovery_mode_2008_03_12 -> charge_dim
>     principal.env -> charge_dim
>     Chrg_dim -> charge_dim
>     Chrg_dim_interactif -> charge_dim
>     Publ_dim_interactif -> charge_dim
> #+end_example
>
> HERE, I'M WORKING GIVING ONCE AGAIN THE SAME DEFAULT VALUE FOR TESTING (AND
> DOCUMENTATION) PURPOSE.
>
> * Problem
>
> All of the above nicely answer my first goal ("Be able to execute the script
> in situ, so that I get a copy of the results stored in my documentation").
>
> It does not allow for the second one: *how to chain the calls together*?
>
> For example, just for chaining steps 2 and 3 (the easier to chain, I think),
> I'd like to be able to write something like this:
>
> #+srcname: search-links-and-generate-dot-arrow
> #+begin_src sh :results output :var f="charge_dim" :var data=search-files-pointing-to-this-file :noweb yes
> for i in (
>     <<search-files-pointing-to-this-file>>);
> do echo "    $(basename $i) -> $f"; done
> #+end_src

Don't forget about C-c C-v C-v `org-babel-expand-src-block' when
debugging this problem. I've started using that a lot.

My shell scripting may be too weak to help here. (Which is one point,
I'd be tempted to bail out to another language on this one) What exactly
are you doing there? Executing the code from the other block in a
subshell and using the stdout as the set to loop over? You're sure you
don't want parentheses <<search-files-pointing-to-this-file>>() ? Or $(
... ) / backticks ? How are you going to pass arguments to the
search-files-pointing src block?

>
> #+results: search-links-and-generate-dot-arrow
>
> But it yields an error:
>
> #+begin_src text :eval no
> sh: line 14: syntax error near unexpected token `('
> sh: line 14: `for i in ('
> #+end_src
>
> Note, in the latter code block, that I did not even tried to really chain
> steps 2 and 3: I'm rewriting step 3, including step 2 inside it.
>
> *I certainly miss a smarter way* to achieve the above.

I think a relevant point here is that Org doesn't yet have the ability
to pass data on standard input to a code block. I.e. a :stdin header
arg. I don't think it's that hard, someone would just need to rework
`org-babel-eval' so that it puts the code into a temporary file, freeing
up stdin to be used for data (and therefore we would no longer be able
to use shell-command-on-region but some other command (call-process I
think?).)


Then you'd be able to do something like

#+srcname: search-links-and-generate-dot-arrow
#+header: :stdin search-files-pointing-to-this-file
#+begin_src sh :results output :var f="charge_dim"
while read f; do
    echo "    $(basename $i) -> $f";
done
#+end_src


I'll be interested to see the solution to all this.

Dan

>
> Quite important as well, is that the tangled file seems natural to read (for
> someone that would not read it from the Org file). This means we would have to
> minimize the use of "echo" and "tee" commands, among others -- but that's a
> nice to have.
>
> Do you see my point?
>
> Do you have ideas on how to go that way?
>
> Best regards,
>   Seb