Output of R code block: only text or plot but not both? And only one "result" can be output?

Output of R code block: only text or plot but not both? And only one "result" can be output?

[Giuseppe Pagnoni]

[-- Attachment #1: Type: text/plain, Size: 1054 bytes --]

Hi,

I am trying to have an R code block output both the results of a numerical
expression AND a plot, but if I set up the header arguments to display the
plot, the numerical outcome is not displayed.  A simple example:

#+begin_src R :file example.png :results output graphics file
  x <- rnorm(100)
  print(mean(x))
  hist(x)
#+end_src

Here I get the histogram plot back in the buffer, but NOT the result of
"print(mean(x))".   Is it not possible to get both?

Furthermore, what if I wanted to display *two* or more plots that get built
by the code, for instance:

#+begin_src R :file example.png :results output graphics file
  x <- rnorm(100)
  print(mean(x))
  hist(x)
  hist(x^2)
#+end_src

Here, only the last plot gets displayed.  Is it not possible to display
more than 1 plot?

many thanks in advance for any help

giuseppe
-- 
Giuseppe Pagnoni
Dip. Scienze Biomediche, Metaboliche e Neuroscienze
Sezione Fisiologia e Neuroscienze
Univ. di Modena e Reggio Emilia
Via Campi 287
I-41125 Modena, Italy
Tel: +39-059-205-5742
Fax: +39-059-205-5363

[-- Attachment #2: Type: text/html, Size: 1486 bytes --]

Re: Output of R code block: only text or plot but not both? And only one "result" can be output?

[Berry, Charles]

Giuseppe,



> On Jun 3, 2024, at 6:47 AM, Giuseppe Pagnoni <gpagnoni@gmail.com> wrote:
> 
> Hi,
> 
> I am trying to have an R code block output both the results of a numerical expression AND a plot, but if I set up the header arguments to display the plot, the numerical outcome is not displayed.  A simple example:
> 
> #+begin_src R :file example.png :results output graphics file
>   x <- rnorm(100)
>   print(mean(x))
>   hist(x)
> #+end_src
> 


Babel is a great tool, but has some inherent limitations.

Whilst you can work around this by breaking the above block into multiple src blocks, you might have an easier time using ox-ravel [1]. Basically, you export your *.org document to one of the knitr/Rmarkdown/quarto formats (e.g. *.Rmd) and then render the final output using one of those engines. This gives you access to the capabilities of any of those engines.  The downside, of course, is that you need to learn a bit about the formatting rules in one of them to fine tune your final output.

FWIW, I copy and pasted your query to a `*.org` file, exported it to `*.Rmd` with the latest ox-ravel version and rendered it as html - displaying all the printed output and all three plots. That did not need any modifications to your markup.


> Here I get the histogram plot back in the buffer, but NOT the result of "print(mean(x))".   Is it not possible to get both?
> 
> Furthermore, what if I wanted to display *two* or more plots that get built by the code, for instance:
> 
> #+begin_src R :file example.png :results output graphics file
>   x <- rnorm(100)
>   print(mean(x))
>   hist(x)
>   hist(x^2)
> #+end_src
> 
> Here, only the last plot gets displayed.  Is it not possible to display more than 1 plot?
> 


If you want to do this with babel, do the computations in an initial block and then use subsequent blocks for each of the displayed outputs/graphs. You will need to use the `:session` header arg to retain the data in `x' across blocks. 

HTH,

Chuck

[1] https://github.com/chasberry/orgmode-accessories

[FR] Org babel: mixing multiple outputs (was: Output of R code block: only text or plot but not both? And only one "result" can be output?)

[Ihor Radchenko]

"Berry, Charles" <ccberry@health.ucsd.edu> writes:

>> I am trying to have an R code block output both the results of a numerical expression AND a plot, but if I set up the header arguments to display the plot, the numerical outcome is not displayed.  A simple example:
>> 
>> #+begin_src R :file example.png :results output graphics file
>>   x <- rnorm(100)
>>   print(mean(x))
>>   hist(x)
>> #+end_src
>> 
>
>
> Babel is a great tool, but has some inherent limitations.

While it is currently not directly possible to get a mix output and
graphics results, this is _not_ inherent limitation.

One can write a patch for ob-R to produce output type that mixes all the
block results, including textual output, graphical output, and any other
perceivable output.

Such patch would be a reasonable addition to babel backends like ob-R or ob-python.

> Whilst you can work around this by breaking the above block into
> multiple src blocks, you might have an easier time using ox-ravel
> [1]. Basically, you export your *.org document to one of the
> knitr/Rmarkdown/quarto formats (e.g. *.Rmd) and then render the final
> output using one of those engines. This gives you access to the
> capabilities of any of those engines.  The downside, of course, is
> that you need to learn a bit about the formatting rules in one of them
> to fine tune your final output.
>
> FWIW, I copy and pasted your query to a `*.org` file, exported it to
> `*.Rmd` with the latest ox-ravel version and rendered it as html -
> displaying all the printed output and all three plots. That did not
> need any modifications to your markup.

Thanks for sharing the project!
Although, I would not call going through double export, and producing
html output "easier time".

While we are here, are there any other features you find missing in Org
babel that are present in knitr/Rmarkdown/quarto?

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Output of R code block: only text or plot but not both? And only one "result" can be output?

[Ihor Radchenko]

Giuseppe Pagnoni <gpagnoni@gmail.com> writes:

> #+begin_src R :file example.png :results output graphics file
>   x <- rnorm(100)
>   print(mean(x))
>   hist(x)
>   hist(x^2)
> #+end_src
>
> Here, only the last plot gets displayed.  Is it not possible to display
> more than 1 plot?

We currently do not have automatic way to do it, so that the resulting
plots are inserted as previewable file links.  However, in addition to
the approach with several :session source blocks proposed by Berry, you
can do it by hand in R:

#+begin_src R :results output raw drawer :dir ./
  x <- rnorm(100)
  print(mean(x))
  png(file="hist1.png")
  hist(x)
  png(file="hist2.png")
  hist(x^2)
  print("[[./hist1.png]]")
  print("[[./hist2.png]]")
#+end_src

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [FR] Org babel: mixing multiple outputs (was: Output of R code block: only text or plot but not both? And only one "result" can be output?)

[Berry, Charles]

> On Jun 5, 2024, at 11:17 AM, Ihor Radchenko <yantar92@posteo.net> wrote:
> 
> "Berry, Charles" <ccberry@health.ucsd.edu> writes:
> 
>>> I am trying to have an R code block output both the results of a numerical expression AND a plot, but if I set up the header arguments to display the plot, the numerical outcome is not displayed.  A simple example:
>>> 
>>> #+begin_src R :file example.png :results output graphics file
>>>  x <- rnorm(100)
>>>  print(mean(x))
>>>  hist(x)
>>> #+end_src
>>> 
>> 
>> 
>> Babel is a great tool, but has some inherent limitations.
> 
> While it is currently not directly possible to get a mix output and
> graphics results, this is _not_ inherent limitation.
> 

Fair enough, it is not `inherent'.


> One can write a patch for ob-R to produce output type that mixes all the
> block results, including textual output, graphical output, and any other
> perceivable output.

Won't you still have issues like the `:session :results output' conundrum mentioned below?

> 
> Such patch would be a reasonable addition to babel backends like ob-R or ob-python.
> 
>> Whilst you can work around this by breaking the above block into
>> multiple src blocks, you might have an easier time using ox-ravel
>> [1]. Basically, you export your *.org document to one of the
>> knitr/Rmarkdown/quarto formats (e.g. *.Rmd) and then render the final
>> output using one of those engines. This gives you access to the
>> capabilities of any of those engines.  The downside, of course, is
>> that you need to learn a bit about the formatting rules in one of them
>> to fine tune your final output.
>> 
>> FWIW, I copy and pasted your query to a `*.org` file, exported it to
>> `*.Rmd` with the latest ox-ravel version and rendered it as html -
>> displaying all the printed output and all three plots. That did not
>> need any modifications to your markup.
> 
> Thanks for sharing the project!
> Although, I would not call going through double export, and producing
> html output "easier time".
> 

The `easier' part is that knitr/Rmarkdown requires very little markup to produce nice documents in various formats (pdf, html, Word, ...). And dealing with R code and markup of the results of R code for use in documents is what that environment is attuned to, so getting a desired result often seemed easier to me in that environment.

In terms of my usual workflow, the double export (with ox-ravel and an `org-render` helper function loaded) requires these keystrokes:

`C-c C-e r m M-x org-render RET`

or subsequently

`C-u C-c C-e M-x M-p RET`

which adds only two keystrokes. 


> While we are here, are there any other features you find missing in Org
> babel that are present in knitr/Rmarkdown/quarto?
> 

`:session :results output` handling in R lang src blocks can fail as the heuristics for finding the output of a command in the session buffer and removing the prompts have limited success. An ECM:

#+begin_src R :session :results output
  cat("2 > 1\n")
#+end_src

There are workarounds, of course.

There was discussion here years ago of using an R package (evaluate?) to better parse the session results. I can't recall why this was dropped - maybe it was the requirement of additional software.

I recall hearing that `comint.el` would someday be replaced by a hardier package, so maybe if/when that happens this can be cured. 

A big motivation for creating `ox-ravel' was to be able to cache large objects. I know the `:cache` header arg helps for small objects that require a lot of computation, but AFAICS does not help once the object size gets large. The caching options [1] of knitr and friends are flexible and powerful enough to support the genomics work I do. 

I mentioned above that knitr is attuned to working with R code/output. The handling of warnings, errors, and messages resulting from R code has a number of useful options under `Text Output` [2].

I guess a rewrite of ob-R.el to implement such features as object caching and error/warning/message handling is feasible, but would require a lot of effort. And since those features (and more - like animation support) are already implemented in the knitr/Rmarkdown domain is it really worth pursuing?

Best,

Chuck 

[1] https://yihui.org/knitr/options/#cache

[2] https://yihui.org/knitr/options/#text-output

Re: Output of R code block: only text or plot but not both? And only one "result" can be output?

[Giuseppe Pagnoni]

[-- Attachment #1: Type: text/plain, Size: 1518 bytes --]

Many thanks.  Since I am using this within an org-mode notebook, I think
the :session trick works well enough for me.

best
giuseppe

On Wed, Jun 5, 2024 at 8:35 PM Ihor Radchenko <yantar92@posteo.net> wrote:

> Giuseppe Pagnoni <gpagnoni@gmail.com> writes:
>
> > #+begin_src R :file example.png :results output graphics file
> >   x <- rnorm(100)
> >   print(mean(x))
> >   hist(x)
> >   hist(x^2)
> > #+end_src
> >
> > Here, only the last plot gets displayed.  Is it not possible to display
> > more than 1 plot?
>
> We currently do not have automatic way to do it, so that the resulting
> plots are inserted as previewable file links.  However, in addition to
> the approach with several :session source blocks proposed by Berry, you
> can do it by hand in R:
>
> #+begin_src R :results output raw drawer :dir ./
>   x <- rnorm(100)
>   print(mean(x))
>   png(file="hist1.png")
>   hist(x)
>   png(file="hist2.png")
>   hist(x^2)
>   print("[[./hist1.png]]")
>   print("[[./hist2.png]]")
> #+end_src
>
> --
> Ihor Radchenko // yantar92,
> Org mode contributor,
> Learn more about Org mode at <https://orgmode.org/>.
> Support Org development at <https://liberapay.com/org-mode>,
> or support my work at <https://liberapay.com/yantar92>
>


-- 
Giuseppe Pagnoni
Dip. Scienze Biomediche, Metaboliche e Neuroscienze
Sezione Fisiologia e Neuroscienze
Univ. di Modena e Reggio Emilia
Via Campi 287
I-41125 Modena, Italy
Tel: +39-059-205-5742
Fax: +39-059-205-5363

[-- Attachment #2: Type: text/html, Size: 2427 bytes --]

Re: Output of R code block: only text or plot but not both? And only one "result" can be output?

[Giuseppe Pagnoni]

[-- Attachment #1: Type: text/plain, Size: 1731 bytes --]

Sorry, I just sent you the reply that was meant for Berry's advice :-).
Your suggestion also looks very useful, but somehow the png files do not
get saved (in the current directory or elsewhere), and I actually do not
get any error messages in the R buffer (I also tried to change the file
type to jpeg).  Any idea why?

thanks again
giuseppe

On Wed, Jun 5, 2024 at 8:35 PM Ihor Radchenko <yantar92@posteo.net> wrote:

> Giuseppe Pagnoni <gpagnoni@gmail.com> writes:
>
> > #+begin_src R :file example.png :results output graphics file
> >   x <- rnorm(100)
> >   print(mean(x))
> >   hist(x)
> >   hist(x^2)
> > #+end_src
> >
> > Here, only the last plot gets displayed.  Is it not possible to display
> > more than 1 plot?
>
> We currently do not have automatic way to do it, so that the resulting
> plots are inserted as previewable file links.  However, in addition to
> the approach with several :session source blocks proposed by Berry, you
> can do it by hand in R:
>
> #+begin_src R :results output raw drawer :dir ./
>   x <- rnorm(100)
>   print(mean(x))
>   png(file="hist1.png")
>   hist(x)
>   png(file="hist2.png")
>   hist(x^2)
>   print("[[./hist1.png]]")
>   print("[[./hist2.png]]")
> #+end_src
>
> --
> Ihor Radchenko // yantar92,
> Org mode contributor,
> Learn more about Org mode at <https://orgmode.org/>.
> Support Org development at <https://liberapay.com/org-mode>,
> or support my work at <https://liberapay.com/yantar92>
>


-- 
Giuseppe Pagnoni
Dip. Scienze Biomediche, Metaboliche e Neuroscienze
Sezione Fisiologia e Neuroscienze
Univ. di Modena e Reggio Emilia
Via Campi 287
I-41125 Modena, Italy
Tel: +39-059-205-5742
Fax: +39-059-205-5363

[-- Attachment #2: Type: text/html, Size: 2645 bytes --]

Re: Output of R code block: only text or plot but not both? And only one "result" can be output?

[Ihor Radchenko]

Giuseppe Pagnoni <gpagnoni@gmail.com> writes:

> Sorry, I just sent you the reply that was meant for Berry's advice :-).
> Your suggestion also looks very useful, but somehow the png files do not
> get saved (in the current directory or elsewhere), and I actually do not
> get any error messages in the R buffer (I also tried to change the file
> type to jpeg).  Any idea why?

Hard to say.
Try to run my example in command line R repl.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Output of R code block: only text or plot but not both? And only one "result" can be output?

[Giuseppe Pagnoni]

[-- Attachment #1: Type: text/plain, Size: 1059 bytes --]

Same... no file gets written...

On Fri, Jun 7, 2024 at 11:18 AM Ihor Radchenko <yantar92@posteo.net> wrote:

> Giuseppe Pagnoni <gpagnoni@gmail.com> writes:
>
> > Sorry, I just sent you the reply that was meant for Berry's advice :-).
> > Your suggestion also looks very useful, but somehow the png files do not
> > get saved (in the current directory or elsewhere), and I actually do not
> > get any error messages in the R buffer (I also tried to change the file
> > type to jpeg).  Any idea why?
>
> Hard to say.
> Try to run my example in command line R repl.
>
> --
> Ihor Radchenko // yantar92,
> Org mode contributor,
> Learn more about Org mode at <https://orgmode.org/>.
> Support Org development at <https://liberapay.com/org-mode>,
> or support my work at <https://liberapay.com/yantar92>
>


-- 
Giuseppe Pagnoni
Dip. Scienze Biomediche, Metaboliche e Neuroscienze
Sezione Fisiologia e Neuroscienze
Univ. di Modena e Reggio Emilia
Via Campi 287
I-41125 Modena, Italy
Tel: +39-059-205-5742
Fax: +39-059-205-5363

[-- Attachment #2: Type: text/html, Size: 1831 bytes --]

Re: Output of R code block: only text or plot but not both? And only one "result" can be output?

[Giuseppe Pagnoni]

[-- Attachment #1: Type: text/plain, Size: 1363 bytes --]

Ok, I just found out!  You need to close the device after each plot:

#+begin_src R :results output raw drawer :dir ./
  x <- rnorm(100)
  print(mean(x))
  png(file="hist1.png")
  hist(x)
  dev.off()
  png(file="hist2.png")
  hist(x^2)
  dev.off()
  print("[[./hist1.png]]")
  print("[[./hist2.png]]")
#+end_src

thanks!

On Fri, Jun 7, 2024 at 11:18 AM Ihor Radchenko <yantar92@posteo.net> wrote:

> Giuseppe Pagnoni <gpagnoni@gmail.com> writes:
>
> > Sorry, I just sent you the reply that was meant for Berry's advice :-).
> > Your suggestion also looks very useful, but somehow the png files do not
> > get saved (in the current directory or elsewhere), and I actually do not
> > get any error messages in the R buffer (I also tried to change the file
> > type to jpeg).  Any idea why?
>
> Hard to say.
> Try to run my example in command line R repl.
>
> --
> Ihor Radchenko // yantar92,
> Org mode contributor,
> Learn more about Org mode at <https://orgmode.org/>.
> Support Org development at <https://liberapay.com/org-mode>,
> or support my work at <https://liberapay.com/yantar92>
>


-- 
Giuseppe Pagnoni
Dip. Scienze Biomediche, Metaboliche e Neuroscienze
Sezione Fisiologia e Neuroscienze
Univ. di Modena e Reggio Emilia
Via Campi 287
I-41125 Modena, Italy
Tel: +39-059-205-5742
Fax: +39-059-205-5363

[-- Attachment #2: Type: text/html, Size: 2273 bytes --]

Ideas for improving ob-R (was: [FR] Org babel: mixing multiple outputs (was: Output of R code block: only text or plot but not both? And only one "result" can be output?))

[Ihor Radchenko]

"Berry, Charles" <ccberry@health.ucsd.edu> writes:

>> Thanks for sharing the project!
>> Although, I would not call going through double export, and producing
>> html output "easier time".
>> 
>
> The `easier' part is that knitr/Rmarkdown requires very little markup
> to produce nice documents in various formats (pdf, html, Word,
> ...). And dealing with R code and markup of the results of R code for
> use in documents is what that environment is attuned to, so getting a
> desired result often seemed easier to me in that environment.

So does Org markup...
What we may be missing is nicely formatted output of R blocks, where we
need something akin ob-python that knows about how to typeset various
Python types in Org - data frames, graphics, tables, etc.

> In terms of my usual workflow, the double export (with ox-ravel and an
> `org-render` helper function loaded) requires these keystrokes:
>
> `C-c C-e r m M-x org-render RET`
>
> or subsequently
>
> `C-u C-c C-e M-x M-p RET`
>
> which adds only two keystrokes. 

I understand. My point is different - what if you do _not_ need
HTML/pdf/word output and just want to work with the interactive notebook?
Then, your export workflow is a huge overkill and complication.

>> While we are here, are there any other features you find missing in Org
>> babel that are present in knitr/Rmarkdown/quarto?
>
> `:session :results output` handling in R lang src blocks can fail as
> the heuristics for finding the output of a command in the session
> buffer and removing the prompts have limited success. An ECM:
>
> #+begin_src R :session :results output
>   cat("2 > 1\n")
> #+end_src
>
> There are workarounds, of course.

Hmm. We can address this, as we already do in ob-shell.
The idea is to change the prompt string to something unique.

Or, alternatively, we can redirect output to file using `sink'.

> I recall hearing that `comint.el` would someday be replaced by a
> hardier package, so maybe if/when that happens this can be cured.

Such replacement is not in sight, unfortunately.

> A big motivation for creating `ox-ravel' was to be able to cache large
> objects. I know the `:cache` header arg helps for small objects that
> require a lot of computation, but AFAICS does not help once the object
> size gets large. The caching options [1] of knitr and friends are
> flexible and powerful enough to support the genomics work I do.

> [1] https://yihui.org/knitr/options/#cache

Org mode uses pretty much the same caching mechanism (md5 checksum for
code+block parameters) to save code block results. Except that cached
data is not persisted between Emacs sessions. It is not too hard to make
it persistent though - we have a dedicated org-persist library for such
purposes.

Patches welcome!

I guess what is not available is caching for sessions. Can also be
implemented, but it would be a slightly more complex patch.

> I mentioned above that knitr is attuned to working with R
> code/output. The handling of warnings, errors, and messages resulting
> from R code has a number of useful options under `Text Output` [2].

> [2] https://yihui.org/knitr/options/#text-output

I looked through the available options and most of them are already
supported by Org mode.

The ones that are not supported are:

warning: (TRUE; logical) Whether to preserve warnings (produced by
         warning()) in the output.

message: ...

error: ...

Fine-grained control over warning/error/message/output separation is
something that would be nice to have indeed. Not just for ob-R, but in
general.

For ob-R, it should be fairly easy to do - we can redirect stderr/stdout
output to different files and then read the results from there.

> I guess a rewrite of ob-R.el to implement such features as object
> caching and error/warning/message handling is feasible, but would
> require a lot of effort.

I believe that sufficiently motivated person familiar with R and Elisp
can implement these features within a week, working in the evenings.
But, as usual, it is a question of volunteers.

> ... And since those features (and more - like
> animation support) are already implemented in the knitr/Rmarkdown
> domain is it really worth pursuing?

Animation support is indeed tricky. Although, we do have an attempt in
https://github.com/alejandrogallo/org-inline-webkit (any volunteers to
polish that demo up to the state mergeable upstream?)

As for "already implemented", it is a question of use
case. knitr/Rmarkdown are fine as long as they fit the workflow, but Org
mode is generally more flexible - it has more stable and sophisticated
markup and can export to more formats.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>