Selective export of Babel code blocks

Selective export of Babel code blocks

[John Hendy]

I recently started a paper using Babel in a more full-functioned way
to display both R code as well as output. Previously I've just used
the convenience of headings and src blocks to create R plots and have
the code stored nicely for future reference. I'm exporting via LaTeX
to create something to promote R and why it's awesome in conjunction
with org-mode at work, which is a Minitab environment. Thus, I'd like
to include actual R code to familiarize others. On with the problem.
I'm also using tikzDevice. Thus, my paper so far is about like so:

----------
* Plotting this vs. that

Not we'll plot this vs. that. Here's the R code:

#+begin_src R :exports none
  tikzDevice("file.tex", width=6, height=4, standAlone=T)
#+end_src

#+begin_src R :exports both
  data <- read.csv("file.csv", header=T)
  sub <- subset(data, output > 5)
  ggplot(sub, aes(x=x, y=y)) + geom_point()
#+end_src

#+begin_src R :exports none
  dev.off()
  tools::texi2pdf("file.tex")
#+end_src
----------

Anyway, something like this. This isn't a huge poroblem... I'm
generating a lot of the same type of scatterplot just using different
variables, so I can copy and yank the blocks and just change
variable/file names. Doing this got me to thinking how it would be
quite neat to run code in a block but mark it to be omitted from
export for these kind of "setup" lines that are necessary for the
desired graphs but unnecessary for your reader to see over and over
again.

Thoughts? I'm quite a novice and very possibly missed something that
can already do this.


Thanks!
John

Re: Selective export of Babel code blocks

[suvayu ali]

Hi John,

On Mon, Jun 18, 2012 at 4:53 PM, John Hendy <jw.hendy@gmail.com> wrote:
> Doing this got me to thinking how it would be
> quite neat to run code in a block but mark it to be omitted from
> export for these kind of "setup" lines that are necessary for the
> desired graphs but unnecessary for your reader to see over and over
> again.
>
> Thoughts? I'm quite a novice and very possibly missed something that
> can already do this.

Hopefully I understood your question correctly. My solution to "setup"
code blocks is use the noweb syntax. Following is an example with Gnuplot.


  #+PROPERTY: noweb yes
  #+PROPERTY: results silent
  #+BIND: org-confirm-babel-evaluate nil

  * Gnuplot source
  #+name: preamble
  #+begin_src gnuplot
    reset
    set terminal pdfcairo color size 21cm,14.8cm
    set termoption enhanced
    set encoding utf8
    set termoption font "DejaVuSerif,8"
    # ...
  #+end_src

  #+begin_src gnuplot :noweb yes :var limits=Bpluslimits
    <<preamble>>
    plot "$limits" using 1:2 title 'Theory'  set output
  #+end_src

  * Table
  #+tblname: Bpluslimits
  |   |  Theory |
  |   |         |
  |---+---------|
  | 1 | 3.6E-14 |
  | 2 | 3.6E-14 |
  | 3 | 6.3E-13 |
  | 4 | 6.3E-13 |
  | 5 | 1.7E-14 |
  | 6 | 1.7E-14 |
  | 7 |         |
  #+tblfm: $1=(@#-2)

  * COMMENT local setup

  # Local Variables:
  # org-export-allow-BIND: t
  # End:

HTH

-- 
Suvayu

Open source is the future. It sets us free.

Re: Selective export of Babel code blocks

[John Hendy]

On Mon, Jun 18, 2012 at 11:08 AM, suvayu ali
<fatkasuvayu+linux@gmail.com> wrote:
> Hi John,
>
> On Mon, Jun 18, 2012 at 4:53 PM, John Hendy <jw.hendy@gmail.com> wrote:
>> Doing this got me to thinking how it would be
>> quite neat to run code in a block but mark it to be omitted from
>> export for these kind of "setup" lines that are necessary for the
>> desired graphs but unnecessary for your reader to see over and over
>> again.
>>
>> Thoughts? I'm quite a novice and very possibly missed something that
>> can already do this.
>
> Hopefully I understood your question correctly. My solution to "setup"
> code blocks is use the noweb syntax. Following is an example with Gnuplot.
>

I didn't get it at first as I'm not familiar with noweb, but after
reading a bit, I think I get it.

>
>  #+PROPERTY: noweb yes
>  #+PROPERTY: results silent
>  #+BIND: org-confirm-babel-evaluate nil
>
>  * Gnuplot source
>  #+name: preamble
>  #+begin_src gnuplot
>    reset
>    set terminal pdfcairo color size 21cm,14.8cm
>    set termoption enhanced
>    set encoding utf8
>    set termoption font "DejaVuSerif,8"
>    # ...
>  #+end_src
>
>  #+begin_src gnuplot :noweb yes :var limits=Bpluslimits
>    <<preamble>>
>    plot "$limits" using 1:2 title 'Theory'  set output
>  #+end_src
>
>  * Table
>  #+tblname: Bpluslimits
>  |   |  Theory |
>  |   |         |
>  |---+---------|
>  | 1 | 3.6E-14 |
>  | 2 | 3.6E-14 |
>  | 3 | 6.3E-13 |
>  | 4 | 6.3E-13 |
>  | 5 | 1.7E-14 |
>  | 6 | 1.7E-14 |
>  | 7 |         |
>  #+tblfm: $1=(@#-2)
>
>  * COMMENT local setup
>
>  # Local Variables:
>  # org-export-allow-BIND: t
>  # End:
>

Gotcha. So I can define a preamble/postamble and be all set. Still
struggling on how to make this work for each file, though. I'm
guessing I need something like:

#+name: preamble
#+begin_src R :exports none
tikzDevice("file-name")
#+end_src

And by the org-mode Babel documentation it looks like I can do something like:

<<preamble(file-name=file_1.tex)>>

Or something like this? I didn't see much coverage explaining the
whole <<example-block(a=9>> bit at the end of the noweb section:
- http://orgmode.org/worg/org-contrib/babel/intro.html#literate-programming

I think this will work; I just need the bit on variable substitution.


Thanks a ton!
John


> HTH
>
> --
> Suvayu
>
> Open source is the future. It sets us free.

Re: Selective export of Babel code blocks

[suvayu ali]

Hi John,

My R knowledge is zero, but I'll give it a shot.

On Mon, Jun 18, 2012 at 6:50 PM, John Hendy <jw.hendy@gmail.com> wrote:
> #+name: preamble
> #+begin_src R :exports none
> tikzDevice("file-name")
> #+end_src
>

Are you trying to use "file-name" as a variable? if so I think the
source block header should include this: ":var file-name" and the source
block can refer to the variable as $file-name.

> And by the org-mode Babel documentation it looks like I can do
> something like:
>
> <<preamble(file-name=file_1.tex)>>

I'm not sure about this. The syntax seems correct but I don't quite
understand the following statement from the manual:

  It is possible to include the _results_ of a code block rather than
  the body. This is done by appending parenthesis to the code block name
  which may optionally contain arguments to the code block as shown
  below.

       <<code-block-name(optional arguments)>>

My lack of knowledge of R doesn't help either. A little experimentation
should resolve it I think. :)

HTH

-- 
Suvayu

Open source is the future. It sets us free.

Re: Selective export of Babel code blocks

[Thomas S. Dye]

John Hendy <jw.hendy@gmail.com> writes:

> I recently started a paper using Babel in a more full-functioned way
> to display both R code as well as output. Previously I've just used
> the convenience of headings and src blocks to create R plots and have
> the code stored nicely for future reference. I'm exporting via LaTeX
> to create something to promote R and why it's awesome in conjunction
> with org-mode at work, which is a Minitab environment. Thus, I'd like
> to include actual R code to familiarize others. On with the problem.
> I'm also using tikzDevice. Thus, my paper so far is about like so:
>
> ----------
> * Plotting this vs. that
>
> Not we'll plot this vs. that. Here's the R code:
>
> #+begin_src R :exports none
>   tikzDevice("file.tex", width=6, height=4, standAlone=T)
> #+end_src
>
> #+begin_src R :exports both
>   data <- read.csv("file.csv", header=T)
>   sub <- subset(data, output > 5)
>   ggplot(sub, aes(x=x, y=y)) + geom_point()
> #+end_src
>
> #+begin_src R :exports none
>   dev.off()
>   tools::texi2pdf("file.tex")
> #+end_src
> ----------
>
> Anyway, something like this. This isn't a huge poroblem... I'm
> generating a lot of the same type of scatterplot just using different
> variables, so I can copy and yank the blocks and just change
> variable/file names. Doing this got me to thinking how it would be
> quite neat to run code in a block but mark it to be omitted from
> export for these kind of "setup" lines that are necessary for the
> desired graphs but unnecessary for your reader to see over and over
> again.
>
> Thoughts? I'm quite a novice and very possibly missed something that
> can already do this.
>
>
> Thanks!
> John
>
>
Hi John,

One way to do this might be to name the source code block and then use
an #+CALL: line.

#+name: clean-up
#+begin_src R :exports code
  dev.off()
  tools::texi2pdf("file.tex")
#+end_src

.... 


#+CALL: clean-up() :exports none

This way, the original code block will be exported and subsequent calls
should not be.

hth,
Tom

-- 
Thomas S. Dye
http://www.tsdye.com

Re: Selective export of Babel code blocks

[suvayu ali]

Hi Thomas,

On Mon, Jun 18, 2012 at 7:11 PM, Thomas S. Dye <tsd@tsdye.com> wrote:
> #+CALL: clean-up() :exports none
>
> This way, the original code block will be exported and subsequent calls
> should not be.

I think John's use case requires other code blocks "using" the common
code block. Can a "CALL" be done from inside a codeblock?

That said, your post gave me an idea; how about defining a function in
the first code block and then using that in the other blocks. This would
require the session feature of course. :)

-- 
Suvayu

Open source is the future. It sets us free.

Re: Selective export of Babel code blocks

[John Hendy]

On Mon, Jun 18, 2012 at 12:16 PM, suvayu ali
<fatkasuvayu+linux@gmail.com> wrote:
> Hi Thomas,
>
> On Mon, Jun 18, 2012 at 7:11 PM, Thomas S. Dye <tsd@tsdye.com> wrote:
>> #+CALL: clean-up() :exports none
>>
>> This way, the original code block will be exported and subsequent calls
>> should not be.
>
> I think John's use case requires other code blocks "using" the common
> code block. Can a "CALL" be done from inside a codeblock?
>

I think Thomas' suggestion would work. Let me try one more time to
clarify what I'm doing:

----------

First, I need to define my output file name for R -> tikz code. No code export
#+name: preamble
#+begin_src R
   tikzDevice("export-file-name.tex")
#+end_src

Now, I run my actual code, which executes statistics and generates a
plot. I want this code exported in the paper.
#+begin_src R
   ggplot(data, aex(x=x, y=y)) + geom_point()
#+end_src

Now R is sitting there with a plot in the "buffer" (not sure the
proper name). To get it to dump the plot into the tex file, I need to
execute =dev.off()=. Then I have to convert the tikz .tex file into a
pdf for including in the report:
#+name: postamble
#+begin_src R
   dev.off()
   tools::texi2pdf("export-file-name.tex")
#+end_src

For the report I'll have many, many iterations of:

<<preamble>> to define my export file
Some sort of R code doing something.
<<postamble>> to create the export file and convert to pdf.

I obviously can't use the same file name over and over as I'll be
overwriting it. Thus, I need some way to set the file name in the
preamble (something like <<preamble(file-name =
"plot-of-a-vs-b.tex">>) and a way to tell the postamble which file to
convert to pdf (the same one I created in the preamble). This if this
noweb path is going to work, I need to be able to pass file name
arguments to tikzDevice() in the first block and tools::texi2pdf() in
the last.

Is that clearer? I tried to think of an analogous situation in gnuplot
where you need the same code block again and again, but with unique
names/vars passed, but couldn't.

If #+call is doing what I think it does, that definitely could work...
again, as long as I can tell the #+call argument which filenames to
use in the "master" code blocks.

> That said, your post gave me an idea; how about defining a function in
> the first code block and then using that in the other blocks. This would
> require the session feature of course. :)

I have session feature enabled in my paper because all subsequent
blocks depend on the very first block where I load my data file. If I
didn't use session, I'd have to repeat that code again and again or
they wouldn't know what data I was accessing/plotting. Your proposal
is possible, but again, it's not entirely one "static" block -- it's a
block using the same exact functions but with different arguments
passed for the unique filenames of each plot.

Hopefully this makes sense!


Thanks for the input,
John

>
> --
> Suvayu
>
> Open source is the future. It sets us free.

Re: Selective export of Babel code blocks

[William LÉCHELLE]

> On Mon, Jun 18, 2012 at 6:50 PM, John Hendy <jw.hendy@gmail.com> wrote:
> > #+name: preamble
> > #+begin_src R :exports none
> > tikzDevice("file-name")
> > #+end_src
> 
> Are you trying to use "file-name" as a variable? if so I think the
> source block header should include this: ":var file-name" and the source
> block can refer to the variable as $file-name.

It seems to me one of both
    ,----[ insert preamble ]
    | <<preamble>>
    | with the preamble having the header
    | :var file-name="file_1.tex"
    `----
or
    ,----[ eval preamble ]
    | <<preamble(file-name="file_1.tex")>>
    | without any :var header
    `----
would be correct syntax, but using :var <variable> without assigning it should fail
(though I didn't test it).

Both wouldn't give the same result…

> > And by the org-mode Babel documentation it looks like I can do
> > something like:
> >
> > <<preamble(file-name=file_1.tex)>>
> 
> I'm not sure about this. The syntax seems correct but I don't quite
> understand the following statement from the manual:
> 
>   It is possible to include the _results_ of a code block rather than
>   the body. This is done by appending parenthesis to the code block name
>   which may optionally contain arguments to the code block as shown
>   below.
> 
>        <<code-block-name(optional arguments)>>
… as I read this as 
"the noweb reference will expand in either 
1. The litteral of the code-block if there are no parenthesis
2. The results of the block (as obtained with C-c C-c or
org-babel-execute-src-block) (which may depend of a :results header, maybe),
if parenthesis are present.
"

So if the OP wants to call the preamble block with an argument in the noweb
reference, then the preamble block might have to be more like
#+name: preamble
#+begin_src R :results output
print(tikzDevice(file-name))
#+end_src
called with 
<<preamble(file-name="file_1.tex")>>

(but I don't know any R either, I just went myself through variable
substitution over the weekend)

Last alternative, which I use, is putting the file name in some named table,
and then passing it as an argument to

    ,----[ insert preamble ]
    | <<preamble>>
    | with the preamble having the header
    | :var file-name=some-table[0,1]
    `----
(Like
#+tblname: some-table
| file        | "file_1.tex" |
)

HTH,

William

Re: Selective export of Babel code blocks

[Andreas Leha]

John Hendy <jw.hendy@gmail.com> writes:

> On Mon, Jun 18, 2012 at 12:16 PM, suvayu ali
> <fatkasuvayu+linux@gmail.com> wrote:
>> Hi Thomas,
>>
>> On Mon, Jun 18, 2012 at 7:11 PM, Thomas S. Dye <tsd@tsdye.com> wrote:
>>> #+CALL: clean-up() :exports none
>>>
>>> This way, the original code block will be exported and subsequent calls
>>> should not be.
>>
>> I think John's use case requires other code blocks "using" the common
>> code block. Can a "CALL" be done from inside a codeblock?
>>
>
> I think Thomas' suggestion would work. Let me try one more time to
> clarify what I'm doing:
>
> ----------
>
> First, I need to define my output file name for R -> tikz code. No code export
> #+name: preamble
> #+begin_src R
>    tikzDevice("export-file-name.tex")
> #+end_src
>
> Now, I run my actual code, which executes statistics and generates a
> plot. I want this code exported in the paper.
> #+begin_src R
>    ggplot(data, aex(x=x, y=y)) + geom_point()
> #+end_src
>

Not sure, whether I got you correctly.  Why do you need pdf, if you go
along the LaTeX route?  But how about sth along

#+name: preamble
#+begin_src: R
  tikzDevice(fn)
#+end_src

#+name: postamble
#+begin_src: R
  dev.off()
  tools::texi2pdf(fn)
#+end_src


#+name: statistics
#+begin_src R
  ggplot(data, aex(x=x, y=y)) + geom_point()
#+end_src


#+begin_src R :noweb yes
  fn <- "export-file-name.tex"
  <<preamble>>
  <<statistics>>
  <<postamble>>
#+end_src


Or even more org-mode-ish

#+begin_src R :noweb yes :var fn="export-file-name.tex"
  <<preamble>>
  <<statistics>>
  <<postamble>>
#+end_src

Is that closer to what you want?


BTW:  If you are not aware of that, you can have the source block return
tikz code without the manual "tikz("dtrn")"

If you combine that with a general tikz2pdf converter, you could do (untested):

#+name: tikz2pdf
#+begin_src R :var fn="foo.tex"
  tools::texi2pdf(fn)
#+end_src

#+name: dotheplot
#+begin_src R :exports code :noweb yes :results graphics :file "export-file.tex"
  ggplot(data, aex(x=x, y=y)) + geom_point()
#+end_src

#+call tikz2pdf[:var fn=dotheplot()]() :exports results :results file


Regards,
Andreas

> Now R is sitting there with a plot in the "buffer" (not sure the
> proper name). To get it to dump the plot into the tex file, I need to
> execute =dev.off()=. Then I have to convert the tikz .tex file into a
> pdf for including in the report:
> #+name: postamble
> #+begin_src R
>    dev.off()
>    tools::texi2pdf("export-file-name.tex")
> #+end_src
>
> For the report I'll have many, many iterations of:
>
> <<preamble>> to define my export file
> Some sort of R code doing something.
> <<postamble>> to create the export file and convert to pdf.
>
> I obviously can't use the same file name over and over as I'll be
> overwriting it. Thus, I need some way to set the file name in the
> preamble (something like <<preamble(file-name =
> "plot-of-a-vs-b.tex">>) and a way to tell the postamble which file to
> convert to pdf (the same one I created in the preamble). This if this
> noweb path is going to work, I need to be able to pass file name
> arguments to tikzDevice() in the first block and tools::texi2pdf() in
> the last.
>
> Is that clearer? I tried to think of an analogous situation in gnuplot
> where you need the same code block again and again, but with unique
> names/vars passed, but couldn't.
>
> If #+call is doing what I think it does, that definitely could work...
> again, as long as I can tell the #+call argument which filenames to
> use in the "master" code blocks.
>
>> That said, your post gave me an idea; how about defining a function in
>> the first code block and then using that in the other blocks. This would
>> require the session feature of course. :)
>
> I have session feature enabled in my paper because all subsequent
> blocks depend on the very first block where I load my data file. If I
> didn't use session, I'd have to repeat that code again and again or
> they wouldn't know what data I was accessing/plotting. Your proposal
> is possible, but again, it's not entirely one "static" block -- it's a
> block using the same exact functions but with different arguments
> passed for the unique filenames of each plot.
>
> Hopefully this makes sense!
>
>
> Thanks for the input,
> John
>
>>
>> --
>> Suvayu
>>
>> Open source is the future. It sets us free.

Re: Selective export of Babel code blocks

[Thomas S. Dye]

suvayu ali <fatkasuvayu+linux@gmail.com> writes:

> Hi Thomas,
>
> On Mon, Jun 18, 2012 at 7:11 PM, Thomas S. Dye <tsd@tsdye.com> wrote:
>> #+CALL: clean-up() :exports none
>>
>> This way, the original code block will be exported and subsequent calls
>> should not be.
>
> I think John's use case requires other code blocks "using" the common
> code block. Can a "CALL" be done from inside a codeblock?
>
> That said, your post gave me an idea; how about defining a function in
> the first code block and then using that in the other blocks. This would
> require the session feature of course. :)
Hi Suvayu,

Yes, there are several ways to accomplish John's goal, depending on how
he wants to structure the R code and whether or not he wants to preserve
state with one or more sessions.

I don't think it is possible to use #+CALL: inside a code block.  For
that, there is noweb syntax.  So, for John's recent code, this should
work with or without sessions:

#+name: preamble
#+begin_src R
   tikzDevice("export-file-name.tex")
#+end_src

#+begin_src R :noweb yes
   <<preamble>>
   ggplot(data, aex(x=x, y=y)) + geom_point()
#+end_src


The :noweb header argument can take several values to control when these
noweb references are expanded.  Many of these don't show in the Org-mode
manual on Worg.  Here they are in org.texi:

The @code{:noweb} header argument controls expansion of ``noweb'' syntax
references (see @ref{Noweb reference syntax}) when the code block is
evaluated, tangled, or exported.  The @code{:noweb} header argument can have
one of the five values: @code{no}, @code{yes}, @code{tangle}, or
@code{no-export} @code{strip-export}.

@itemize @bullet
@item @code{no}
The default.  ``Noweb'' syntax references in the body of the code block will
not be expanded before the code block is evaluated, tangled or exported.
@item @code{yes}
``Noweb'' syntax references in the body of the code block will be
expanded before the code block is evaluated, tangled or exported.
@item @code{tangle}
``Noweb'' syntax references in the body of the code block will be expanded
before the code block is tangled.  However, ``noweb'' syntax references will
not be expanded when the code block is evaluated or exported.
@item @code{no-export}
``Noweb'' syntax references in the body of the code block will be expanded
before the block is evaluated or tangled.  However, ``noweb'' syntax
references will not be expanded when the code block is exported.
@item @code{strip-export}
``Noweb'' syntax references in the body of the code block will be expanded
before the block is evaluated or tangled.  However, ``noweb'' syntax
references will not be removed when the code block is exported.
@item @code{eval}
``Noweb'' syntax references in the body of the code block will only be
expanded before the block is evaluated.
@end itemize

So, John might be satisfied with something like this:

#+begin_src R :noweb no-export
   <<preamble>>
   ggplot(data, aex(x=x, y=y)) + geom_point()
#+end_src

hth,
Tom

-- 
Thomas S. Dye
http://www.tsdye.com

Re: Selective export of Babel code blocks

[suvayu ali]

Hi Tom,

On Mon, Jun 18, 2012 at 9:05 PM, Thomas S. Dye <tsd@tsdye.com> wrote:
>
> The :noweb header argument can take several values to control when these
> noweb references are expanded.  Many of these don't show in the Org-mode
> manual on Worg.  Here they are in org.texi:


[...]

I had missed these options! "no-export" might be the way to go. In any
case, there seems to be a myriad of ways to go about this. The choice
depends on John's preferences. :)

Cheers,

-- 
Suvayu

Open source is the future. It sets us free.

Re: Selective export of Babel code blocks

[John Hendy]

On Tue, Jun 19, 2012 at 7:55 AM, suvayu ali <fatkasuvayu+linux@gmail.com> wrote:
> Hi Tom,
>
> On Mon, Jun 18, 2012 at 9:05 PM, Thomas S. Dye <tsd@tsdye.com> wrote:
>>
>> The :noweb header argument can take several values to control when these
>> noweb references are expanded.  Many of these don't show in the Org-mode
>> manual on Worg.  Here they are in org.texi:
>
>
> [...]
>
> I had missed these options! "no-export" might be the way to go. In any
> case, there seems to be a myriad of ways to go about this. The choice
> depends on John's preferences. :)

Indeed! Thanks for all the options. I haven't gotten to plugging in my
actual code yet but will try these ways and get back to the list!

Thanks all!
John

>
> Cheers,
>
> --
> Suvayu
>
> Open source is the future. It sets us free.