R terminal output does not match src block output due to ">" character in results

R terminal output does not match src block output due to ">" character in results

[John Hendy]

I just ran into an issue where results look fine in the terminal, but
not in the results of a source block. I can't share the real example
as it's work confidential, but a couple of rows across 7 columns looks
like this in the terminal:

#+begin_example terminal
dbGetQuery(con, paste0("SELECT TOP 2 * FROM table"))
1       <NA>             NA            NA <NA>         FALSE      0          5
2       <NA>             NA            NA <NA>         FALSE      0          5
#+end_example

When I use this with :exports results :results output drawer in my
document, I get:
#+begin_example src
1       <
            NA            NA <
        FALSE      0          5
2       <
            NA            NA <
        FALSE      0          5
#+end_example

I had an entirely separate email written and about to send, when it
dawned on me that the > character is the same as the R terminal prompt
and might be the cause.

My first attempt to reproduce was unsuccessful:

#+begin_src R :session :exports results :results output drawer
df <- data.frame(
  x = letters[1:3],
  y = c(1, 2, NA),
  z = c("x", NA,"z"))
df
#+end_src

#+RESULTS:
:results:
  x  y    z
1 a  1    x
2 b  2 <NA>
3 c NA    z
:end:

I'm not sure if it's about more columns or more rows, but this does the trick:

#+begin_src R :session :exports results :results output drawer
df <- data.frame(
  x = letters[1:7],
  y = c(1, 2, NA, NA, NA, NA, NA),
  z = c("x", NA, NA, NA, NA, NA, "z"),
  a = c(1, rep(NA, 6)))
df
#+end_src

#+RESULTS:
:results:
  x  y    z  a
1 a  1    x  1
2 b  2 <
NA
3 c NA <
NA
### ... shortened
:end:

If you comment out the line with z=, it works again... but that's the
column with all the <NA> values. Then again, if you end the data.frame
at z and don't add column a, it *also* works despite the presence of
many <NA> values.

#+RESULTS:
:results:
  x  y    z
1 a  1    x
2 b  2 <NA>
3 c NA <NA>
### ... shortened
:end:

I'm imagining there's a... "parser?" somewhere that typically strips
off the command prompt from the results perhaps (and recall some vague
memory of someone telling me this years ago on the list). Is there a
way to stop this from happening?

Would this line be responsible, or something else?
https://github.com/bzg/org-mode/blob/main/lisp/ob-R.el#L453


Many thanks,
John

Re: R terminal output does not match src block output due to ">" character in results

[John Hendy]

Ok, after more searching, I found that this is a duplicate issue
that's already been discussed. Not sure why all my incantations didn't
bring me to the reddit post or mailing list from google, but simply
searching "prompt blocks R" on r/orgmode directly got me here:
https://www.reddit.com/r/orgmode/comments/pt3em4/source_block_modifying_format_of_results_r/

And that links to a patch:
https://gist.github.com/gtuckerkellogg/e356d20497cfdc8e4fc683412e320e3e

And that points to this mailing list thread where the topic has already come up:
https://list.orgmode.org/87zgxc42qg.fsf@gmail.com/

It appears the discussion/fix is stalled out at this point?

Best regards,
John


On Tue, Jan 11, 2022 at 4:42 PM John Hendy <jw.hendy@gmail.com> wrote:
>
> I just ran into an issue where results look fine in the terminal, but
> not in the results of a source block. I can't share the real example
> as it's work confidential, but a couple of rows across 7 columns looks
> like this in the terminal:
>
> #+begin_example terminal
> dbGetQuery(con, paste0("SELECT TOP 2 * FROM table"))
> 1       <NA>             NA            NA <NA>         FALSE      0          5
> 2       <NA>             NA            NA <NA>         FALSE      0          5
> #+end_example
>
> When I use this with :exports results :results output drawer in my
> document, I get:
> #+begin_example src
> 1       <
>             NA            NA <
>         FALSE      0          5
> 2       <
>             NA            NA <
>         FALSE      0          5
> #+end_example
>
> I had an entirely separate email written and about to send, when it
> dawned on me that the > character is the same as the R terminal prompt
> and might be the cause.
>
> My first attempt to reproduce was unsuccessful:
>
> #+begin_src R :session :exports results :results output drawer
> df <- data.frame(
>   x = letters[1:3],
>   y = c(1, 2, NA),
>   z = c("x", NA,"z"))
> df
> #+end_src
>
> #+RESULTS:
> :results:
>   x  y    z
> 1 a  1    x
> 2 b  2 <NA>
> 3 c NA    z
> :end:
>
> I'm not sure if it's about more columns or more rows, but this does the trick:
>
> #+begin_src R :session :exports results :results output drawer
> df <- data.frame(
>   x = letters[1:7],
>   y = c(1, 2, NA, NA, NA, NA, NA),
>   z = c("x", NA, NA, NA, NA, NA, "z"),
>   a = c(1, rep(NA, 6)))
> df
> #+end_src
>
> #+RESULTS:
> :results:
>   x  y    z  a
> 1 a  1    x  1
> 2 b  2 <
> NA
> 3 c NA <
> NA
> ### ... shortened
> :end:
>
> If you comment out the line with z=, it works again... but that's the
> column with all the <NA> values. Then again, if you end the data.frame
> at z and don't add column a, it *also* works despite the presence of
> many <NA> values.
>
> #+RESULTS:
> :results:
>   x  y    z
> 1 a  1    x
> 2 b  2 <NA>
> 3 c NA <NA>
> ### ... shortened
> :end:
>
> I'm imagining there's a... "parser?" somewhere that typically strips
> off the command prompt from the results perhaps (and recall some vague
> memory of someone telling me this years ago on the list). Is there a
> way to stop this from happening?
>
> Would this line be responsible, or something else?
> https://github.com/bzg/org-mode/blob/main/lisp/ob-R.el#L453
>
>
> Many thanks,
> John

Re: R terminal output does not match src block output due to ">" character in results

[Jeremie Juste]

Hello John,

As promised, I'm coming back about the formatting of  NA_characters_.
In org-mode 9.5 NA_characters_ are not printed anymore with :results
value. See example 2 and 3 for more details. Please let me know if
this post solves your issues. Woudl updating to 9.5 be an option for
you?

Best regards,
Jeremie



#+BEGIN_SRC elisp
   (org-version)
  #+END_SRC

  #+RESULTS:
  : 9.5
  
#+BEGIN_SRC elisp
  (emacs-version)
#+end_src

#+RESULTS:
: GNU Emacs 27.2 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.24)
:  of 2021-11-26

* Example 1
#+BEGIN_SRC R :results output
data.frame(abc=NA_character_)
#+end_src

#+RESULTS:
:    abc
: 1 <NA>

* Example 2
#+BEGIN_SRC R :results value 
a <- data.frame(column1=NA_character_,column2=1:10)
#+end_src

#+RESULTS:
|   |  1 |
|   |  2 |
|   |  3 |
|   |  4 |
|   |  5 |
|   |  6 |
|   |  7 |
|   |  8 |
|   |  9 |
|   | 10 |


* Example 3
#+BEGIN_SRC R :results value :colnames yes
library(RSQLite)
con <- dbConnect(RSQLite::SQLite(), ":memory:")

dbWriteTable(con, "df", data.frame(column1=NA_character_,column2=1:10))
dbGetQuery(con, "SELECT * FROM df")
#+end_src

#+RESULTS:
| column1 | column2 |
|---------+---------|
|         |       1 |
|         |       2 |
|         |       3 |
|         |       4 |
|         |       5 |
|         |       6 |
|         |       7 |
|         |       8 |
|         |       9 |
|         |      10 |

Re: R terminal output does not match src block output due to ">" character in results

[John Hendy]

On Sun, Jan 16, 2022 at 7:30 AM Jeremie Juste <jeremiejuste@gmail.com> wrote:
>
> Hello John,
>
> As promised, I'm coming back about the formatting of  NA_characters_.
> In org-mode 9.5 NA_characters_ are not printed anymore with :results
> value. See example 2 and 3 for more details. Please let me know if
> this post solves your issues. Woudl updating to 9.5 be an option for
> you?
>
> Best regards,
> Jeremie
>

From our other thread, I *think* I'm already on 9.5? Re-pasting my
org-version answer here:

#+begin_quote
$ git log
commit 7fa8173282f85c2ca03cc7f51f28f6adfb250610 (HEAD -> master,
origin/master, origin/HEAD)
Author: Ian Martins <ianxm@jhu.edu>
Date:   Sat Jan 16 15:52:21 2021 -0500

But Mx-version:
Org mode version 9.4.4 (release_9.4.4-186-g7fa817.dirty @
/home/jwhendy/.elisp/org/lisp/)

So perhaps it appends the git commit to the system version?
#+end_quote

9.5 appears 4mos old... so I think I'm on it?
https://git.savannah.gnu.org/cgit/emacs/org-mode.git/

I'll try the example later. I'd like for this to work for :results
output and :results value. I'm pretty sure I already explored :results
value and found it to work. One thing that confused me was why I
didn't get a header row with the org table coming from :results value?

Thanks for the help and I'll get back to you early this week to
confirm on the example below.


John

> #+BEGIN_SRC elisp
>    (org-version)
>   #+END_SRC
>
>   #+RESULTS:
>   : 9.5
>
> #+BEGIN_SRC elisp
>   (emacs-version)
> #+end_src
>
> #+RESULTS:
> : GNU Emacs 27.2 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.24)
> :  of 2021-11-26
>
> * Example 1
> #+BEGIN_SRC R :results output
> data.frame(abc=NA_character_)
> #+end_src
>
> #+RESULTS:
> :    abc
> : 1 <NA>
>
> * Example 2
> #+BEGIN_SRC R :results value
> a <- data.frame(column1=NA_character_,column2=1:10)
> #+end_src
>
> #+RESULTS:
> |   |  1 |
> |   |  2 |
> |   |  3 |
> |   |  4 |
> |   |  5 |
> |   |  6 |
> |   |  7 |
> |   |  8 |
> |   |  9 |
> |   | 10 |
>
>
> * Example 3
> #+BEGIN_SRC R :results value :colnames yes
> library(RSQLite)
> con <- dbConnect(RSQLite::SQLite(), ":memory:")
>
> dbWriteTable(con, "df", data.frame(column1=NA_character_,column2=1:10))
> dbGetQuery(con, "SELECT * FROM df")
> #+end_src
>
> #+RESULTS:
> | column1 | column2 |
> |---------+---------|
> |         |       1 |
> |         |       2 |
> |         |       3 |
> |         |       4 |
> |         |       5 |
> |         |       6 |
> |         |       7 |
> |         |       8 |
> |         |       9 |
> |         |      10 |
>
>

Re: R terminal output does not match src block output due to ">" character in results

[Jack Kamm]

Hi John,

> $ git log
> commit 7fa8173282f85c2ca03cc7f51f28f6adfb250610 (HEAD -> master,
> origin/master, origin/HEAD)
> Author: Ian Martins <ianxm@jhu.edu>
> Date:   Sat Jan 16 15:52:21 2021 -0500

It looks like the last commit you're on is from about a year ago (Jan
2021), so I suspect you're on an older version of org mode. Maybe
there's an issue with the remote you're pulling from?

I tested out your example R blocks and didn't get any prompt mangling
errors, so I think this issue may have been solved already.

Jack

Re: R terminal output does not match src block output due to ">" character in results

[John Hendy]

[-- Attachment #1: Type: text/plain, Size: 1537 bytes --]

Ugh. Thanks for that. Three contributing factors: a) doing a git pull/make
and seeing output as usual without really paying attention, b) scanning for
commit date, seeing January which seemed AboutRight but completely missing
the "oh right, it's 2022 now," and c) the root cause being it'd been so
long since I fetched that I was tracking *master*, not main. Now:

$ git log
commit 70970dff8d68f009947b2167c5b2bd886c6de984 (HEAD -> main, origin/main)
Author: Ihor Radchenko <yantar92@gmail.com>
Date:   Tue Jan 25 22:53:08 2022 +0800

    org-before-first-heading-p: Use cache and clarify docstring

In addition, this makes a lot more sense:

M-x version
Org mode version 9.5.2 (release_9.5.2-323-g70970d @
/home/jwhendy/.elisp/org/lisp/)

And, most importantly, I can no longer reproduce my example from above.


Thanks again, my fault, and I appreciate you pointing out what was there in
plain sight!
John

On Sun, Jan 23, 2022 at 10:49 PM Jack Kamm <jackkamm@gmail.com> wrote:

> Hi John,
>
> > $ git log
> > commit 7fa8173282f85c2ca03cc7f51f28f6adfb250610 (HEAD -> master,
> > origin/master, origin/HEAD)
> > Author: Ian Martins <ianxm@jhu.edu>
> > Date:   Sat Jan 16 15:52:21 2021 -0500
>
> It looks like the last commit you're on is from about a year ago (Jan
> 2021), so I suspect you're on an older version of org mode. Maybe
> there's an issue with the remote you're pulling from?
>
> I tested out your example R blocks and didn't get any prompt mangling
> errors, so I think this issue may have been solved already.
>
> Jack
>

[-- Attachment #2: Type: text/html, Size: 2236 bytes --]