From mboxrd@z Thu Jan 1 00:00:00 1970 From: Dan Davison Subject: Re: [babel] How to kill two birds with one stone? Date: Fri, 04 Feb 2011 17:43:06 +0000 Message-ID: References: <808vxv23j2.fsf@missioncriticalit.com> Mime-Version: 1.0 Content-Type: text/plain Return-path: Received: from [140.186.70.92] (port=60607 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1PlPgg-0004mx-C5 for emacs-orgmode@gnu.org; Fri, 04 Feb 2011 12:43:16 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1PlPge-00016h-29 for emacs-orgmode@gnu.org; Fri, 04 Feb 2011 12:43:13 -0500 Received: from mail-ww0-f41.google.com ([74.125.82.41]:50191) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1PlPgd-00016d-OJ for emacs-orgmode@gnu.org; Fri, 04 Feb 2011 12:43:12 -0500 Received: by wwi18 with SMTP id 18so582103wwi.0 for ; Fri, 04 Feb 2011 09:43:10 -0800 (PST) In-Reply-To: <808vxv23j2.fsf@missioncriticalit.com> (=?utf-8?Q?=22S=C3=A9b?= =?utf-8?Q?astien?= Vauban"'s message of "Fri, 04 Feb 2011 17:00:33 +0100") List-Id: "General discussions about Org-mode." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: emacs-orgmode-bounces+geo-emacs-orgmode=m.gmane.org@gnu.org Errors-To: emacs-orgmode-bounces+geo-emacs-orgmode=m.gmane.org@gnu.org To: =?utf-8?Q?S=C3=A9bastien?= Vauban Cc: emacs org-mode mailing list Hi Seb, Cool post. I hope someone has some good ideas in this thread. Some quick responses / questions below. > #+TITLE: Document a shell script as separate blocks > #+DATE: 2011-02-04 > #+LANGUAGE: en_US > > * Abstract > > When writing shell scripts, I'd like to kill *two* birds with one Babel stone: > > - Be able to *execute the script in situ*, so that I get a copy of the results > stored in (and "versioned" with) my documentation. > > - Be able to *chain the code blocks*, so that I tangle a "natural" script into > a file for later execution. > > I guess I currently miss some points in order to reach that goal in a clean > way. Could you give me advice on how to get a step further down the route? > > * Sample code > > For the sake of clarity, let's take a simple problem: I'd like to generate a > DOT graph of links between all files from a directory tree. So if file a.org contains ----- jhasg [[file:b.org]] [[file:c.org]] hbjgv ----- then you want an arrow from node a to node b and from node a to node c, right? > > The procedure: > > 1. (Recursively) list all files inside the directory. > > 2. For each file, (recursively) search for its name referenced in all the > files. > > 3. Generate a DOT representation of the link between files. > > ** List all files under current directory > > #+srcname: dw-file-tree > #+begin_src sh :results output > cd ~/Some-Project-Dir > find . -type f -print | grep -v .svn | head -n 5 > #+end_src > > #+results: dw-file-tree > #+begin_example > ./.cvsignore > ./charge_dim > ./charge_fct > ./compte > ./controle_config > #+end_example > > Here, I voluntary limit the number of results to the first 5 files, for the > compactness of this example. This sample does not include any file from > subdirectories, but it doesn't matter. > > ** Search recursively for anything about a file > > Search through all files (ignoring =.svn= directories) for any reference to > filename given as parameter. > > #+srcname: search-files-pointing-to-this-file > #+begin_src sh :results output :var f="charge_dim" > cd ~/Some-Project-Dir > find . -not \( -name .svn -prune \) -type f -print0 |\ > xargs -0 grep -i --files-with-matches "$f" > #+end_src > > #+results: search-files-pointing-to-this-file > #+begin_example > ./.cvsignore > ./charge_dim > ./compte > ./IFP/Chrg_dim > ./IFP/Chrg_dim.avant_simple_recovery_mode_2008_03_12 > ./principal.env > ./29Aalst/Chrg_dim > ./29Aalst/Chrg_dim_interactif > ./29Aalst/Publ_dim_interactif > #+end_example > > HERE, I'M GIVING A FILENAME AS DEFAULT VALUE OF =f= IN ORDER TO SEE A SAMPLE > RESULTS. > > ** Convert to a DOT representation > > For every file pointing to the file given in parameter, generate an "arrow" > (edge) in DOT representation. > > #+srcname: dot-arrow-from-files-pointing-to-this-file > #+begin_src sh :results output :var f="charge_dim" :var data=search-files-pointing-to-this-file > for i in $(echo "$data"); do echo " $(basename $i) -> $f"; done > #+end_src > > #+results: dot-arrow-from-files-pointing-to-this-file > #+begin_example > .cvsignore -> charge_dim > charge_dim -> charge_dim > compte -> charge_dim > Chrg_dim -> charge_dim > Chrg_dim.avant_simple_recovery_mode_2008_03_12 -> charge_dim > principal.env -> charge_dim > Chrg_dim -> charge_dim > Chrg_dim_interactif -> charge_dim > Publ_dim_interactif -> charge_dim > #+end_example > > HERE, I'M WORKING GIVING ONCE AGAIN THE SAME DEFAULT VALUE FOR TESTING (AND > DOCUMENTATION) PURPOSE. > > * Problem > > All of the above nicely answer my first goal ("Be able to execute the script > in situ, so that I get a copy of the results stored in my documentation"). > > It does not allow for the second one: *how to chain the calls together*? > > For example, just for chaining steps 2 and 3 (the easier to chain, I think), > I'd like to be able to write something like this: > > #+srcname: search-links-and-generate-dot-arrow > #+begin_src sh :results output :var f="charge_dim" :var data=search-files-pointing-to-this-file :noweb yes > for i in ( > <>); > do echo " $(basename $i) -> $f"; done > #+end_src Don't forget about C-c C-v C-v `org-babel-expand-src-block' when debugging this problem. I've started using that a lot. My shell scripting may be too weak to help here. (Which is one point, I'd be tempted to bail out to another language on this one) What exactly are you doing there? Executing the code from the other block in a subshell and using the stdout as the set to loop over? You're sure you don't want parentheses <>() ? Or $( ... ) / backticks ? How are you going to pass arguments to the search-files-pointing src block? > > #+results: search-links-and-generate-dot-arrow > > But it yields an error: > > #+begin_src text :eval no > sh: line 14: syntax error near unexpected token `(' > sh: line 14: `for i in (' > #+end_src > > Note, in the latter code block, that I did not even tried to really chain > steps 2 and 3: I'm rewriting step 3, including step 2 inside it. > > *I certainly miss a smarter way* to achieve the above. I think a relevant point here is that Org doesn't yet have the ability to pass data on standard input to a code block. I.e. a :stdin header arg. I don't think it's that hard, someone would just need to rework `org-babel-eval' so that it puts the code into a temporary file, freeing up stdin to be used for data (and therefore we would no longer be able to use shell-command-on-region but some other command (call-process I think?).) Then you'd be able to do something like #+srcname: search-links-and-generate-dot-arrow #+header: :stdin search-files-pointing-to-this-file #+begin_src sh :results output :var f="charge_dim" while read f; do echo " $(basename $i) -> $f"; done #+end_src I'll be interested to see the solution to all this. Dan > > Quite important as well, is that the tangled file seems natural to read (for > someone that would not read it from the Org file). This means we would have to > minimize the use of "echo" and "tee" commands, among others -- but that's a > nice to have. > > Do you see my point? > > Do you have ideas on how to go that way? > > Best regards, > Seb