R output with session polluted with ascii color codes

R output with session polluted with ascii color codes

[Tyler Smith]

Hi,

The printed output of some R functions includes ascii color codes (i.e., ^[[3m^[[90m). This causes problems with R code blocks evaluated with the :session option. I've pasted a file below that demonstrates the problem. (should I attach it as a file? not sure what's preferable).

I can't find a way to disable this feature from R. However, I did find a way to filter out these codes from the babel side. The function of interest is ~org-babel-R-evaluate-session` in ob-R.el. Adding a ~(replace-regexp-in-string ansi-color-control-seq-regexp "" ...)~ form around the output strips away the color codes so they don't mess up the output.  I've included that code in the example. If that makes sense I can put that together as a patch. If that doesn't make sense, please let me know how to fix this!

Best,

Tyler

* Reproducible Example

Start with ~emacs -Q~, then evaluate each of the following code blocks
in turn.

#+BEGIN_SRC elisp setup
(require 'package)
(setq package-load-list
      '((org-plus-contrib t)
	(ess t)
	(julia-mode t)))
(package-initialize)
(require 'org)
(require 'ess)

(org-babel-do-load-languages
 'org-babel-load-languages
 '((emacs-lisp . t)
   (R . t)
   (shell . t)))
#+END_SRC

#+RESULTS:

If you don't already have ~tidyr~ and ~dplyr~ installed, you need to
do that before running the following:

#+BEGIN_SRC R :results output
library(tidyr)
library(dplyr)
as_tibble(iris)
#+END_SRC

#+RESULTS:
#+begin_example
# A tibble: 150 x 5
   Sepal.Length Sepal.Width Petal.Length Petal.Width Species
          <dbl>       <dbl>        <dbl>       <dbl> <fct>  
 1         5.10        3.50         1.40       0.200 setosa 
 2         4.90        3.00         1.40       0.200 setosa 
 3         4.70        3.20         1.30       0.200 setosa 
 4         4.60        3.10         1.50       0.200 setosa 
 5         5.00        3.60         1.40       0.200 setosa 
 6         5.40        3.90         1.70       0.400 setosa 
 7         4.60        3.40         1.40       0.300 setosa 
 8         5.00        3.40         1.50       0.200 setosa 
 9         4.40        2.90         1.40       0.200 setosa 
10         4.90        3.10         1.50       0.100 setosa 
# ... with 140 more rows
#+end_example

#+BEGIN_SRC R :results output :session RSESSION
library(tidyr)
library(dplyr)
as_tibble(iris)
#+END_SRC

#+RESULTS:
#+begin_example

Attaching package: ‘dplyr’

The following objects are masked from ‘package:stats’:

    filter, lag

The following objects are masked from ‘package:base’:

    intersect, setdiff, setequal, union
^[[90m# A tibble: 150 x 5^[[39m
   Sepal.Length Sepal.Width Petal.Length Petal.Width Species
          ^[[3m^[[90m<dbl>^[[39m^[[23m       ^[[3m^[[90m<dbl>^[[39m^[[23m        ^[[3m^[[90m<dbl>^[[39m^[[23m       ^[[3m^[[90m<dbl>^[[39m^[[23m ^[[3m^[[90m<fct>^[[39m^[[23m  
^[[90m 1^[[39m         5.10        3.50         1.40       ^[[90m0^[[39m^[[90m.^[[39m200 setosa 
^[[90m 2^[[39m         4.90        3.00         1.40       ^[[90m0^[[39m^[[90m.^[[39m200 setosa 
^[[90m 3^[[39m         4.70        3.20         1.30       ^[[90m0^[[39m^[[90m.^[[39m200 setosa 
^[[90m 4^[[39m         4.60        3.10         1.50       ^[[90m0^[[39m^[[90m.^[[39m200 setosa 
^[[90m 5^[[39m         5.00        3.60         1.40       ^[[90m0^[[39m^[[90m.^[[39m200 setosa 
^[[90m 6^[[39m         5.40        3.90         1.70       ^[[90m0^[[39m^[[90m.^[[39m400 setosa 
^[[90m 7^[[39m         4.60        3.40         1.40       ^[[90m0^[[39m^[[90m.^[[39m300 setosa 
^[[90m 8^[[39m         5.00        3.40         1.50       ^[[90m0^[[39m^[[90m.^[[39m200 setosa 
^[[90m 9^[[39m         4.40        2.90         1.40       ^[[90m0^[[39m^[[90m.^[[39m200 setosa 
^[[90m10^[[39m         4.90        3.10         1.50       ^[[90m0^[[39m^[[90m.^[[39m100 setosa 
^[[90m# ... with 140 more rows^[[39m
#+end_example

* Proposed Fix

Evaluating the following replaces the built-in function with my fix:

#+BEGIN_SRC elisp fix
(defun org-babel-R-evaluate-session
    (session body result-type result-params column-names-p row-names-p)
  "Evaluate BODY in SESSION.
If RESULT-TYPE equals `output' then return standard output as a
string.  If RESULT-TYPE equals `value' then return the value of the
last statement in BODY, as elisp."
  (cl-case result-type
    (value
     (with-temp-buffer
       (insert (org-babel-chomp body))
       (let ((ess-local-process-name
	      (process-name (get-buffer-process session)))
	     (ess-eval-visibly-p nil))
	 (ess-eval-buffer nil)))
     (let ((tmp-file (org-babel-temp-file "R-")))
       (org-babel-comint-eval-invisibly-and-wait-for-file
	session tmp-file
	(format org-babel-R-write-object-command
		(if row-names-p "TRUE" "FALSE")
		(if column-names-p
		    (if row-names-p "NA" "TRUE")
		  "FALSE")
		".Last.value" (org-babel-process-file-name tmp-file 'noquote)))
       (org-babel-R-process-value-result
	(org-babel-result-cond result-params
	  (with-temp-buffer
	    (insert-file-contents tmp-file)
	    (org-babel-chomp (buffer-string) "\n"))
	  (org-babel-import-elisp-from-file tmp-file '(16)))
	column-names-p)))
    (output
     ;; strip ansi-color-control-seq-regexp from output!!
     (replace-regexp-in-string
      ansi-color-control-seq-regexp ""
      (mapconcat
       'org-babel-chomp
       (butlast
        (delq nil
	      (mapcar
	       (lambda (line) (when (> (length line) 0) line))
	       (mapcar
	        (lambda (line) ;; cleanup extra prompts left in output
		  (if (string-match
		       "^\\([>+.]\\([ ][>.+]\\)*[ ]\\)"
		       (car (split-string line "\n")))
		      (substring line (match-end 1))
		    line))
	        (org-babel-comint-with-output (session org-babel-R-eoe-output)
		  (insert (mapconcat 'org-babel-chomp
				     (list body org-babel-R-eoe-indicator)
				     "\n"))
		  (inferior-ess-send-input)))))) "\n")))))

#+END_SRC

#+RESULTS:
: org-babel-R-evaluate-session

#+BEGIN_SRC R :results output :session RSESSION
as_tibble(iris)
#+END_SRC

#+RESULTS:
#+begin_example
# A tibble: 150 x 5
   Sepal.Length Sepal.Width Petal.Length Petal.Width Species
          <dbl>       <dbl>        <dbl>       <dbl> <fct>  
 1         5.10        3.50         1.40       0.200 setosa 
 2         4.90        3.00         1.40       0.200 setosa 
 3         4.70        3.20         1.30       0.200 setosa 
 4         4.60        3.10         1.50       0.200 setosa 
 5         5.00        3.60         1.40       0.200 setosa 
 6         5.40        3.90         1.70       0.400 setosa 
 7         4.60        3.40         1.40       0.300 setosa 
 8         5.00        3.40         1.50       0.200 setosa 
 9         4.40        2.90         1.40       0.200 setosa 
10         4.90        3.10         1.50       0.100 setosa 
# ... with 140 more rows
#+end_example

Re: R output with session polluted with ascii color codes

[William Denton]

On 16 March 2018, Tyler Smith wrote:

> ^[[90m# A tibble: 150 x 5^[[39m
>   Sepal.Length Sepal.Width Petal.Length Petal.Width Species
>          ^[[3m^[[90m<dbl>^[[39m^[[23m       ^[[3m^[[90m<dbl>^[[39m^[[23m        ^[[3m^[[90m<dbl>^[[39m^[[23m       ^[[3m^[[90m<dbl>^[[39m^[[23m ^[[3m^[[90m<fct>^[[39m^[[23m
> ^[[90m 1^[[39m         5.10        3.50         1.40       ^[[90m0^[[39m^[[90m.^[[39m200 setosa
> ^[[90m 2^[[39m         4.90        3.00         1.40       ^[[90m0^[[39m^[[90m.^[[39m200 setosa

I tried the example in my configured setup and didn't get the ASCII codes.  I 
don't know why and didn't dig into it, but I could; for now I figured I'd just 
mention this.

Bill
--
William Denton :: Toronto, Canada   ---   Listening to Art: https://listeningtoart.org/
https://www.miskatonic.org/         ---   GHG.EARTH: http://ghg.earth/
Caveat lector.                      ---   STAPLR: http://staplr.org/

Re: R output with session polluted with ascii color codes

[Tyler Smith]

On Fri, Mar 16, 2018, at 11:25 AM, William Denton wrote:
> 
> I tried the example in my configured setup and didn't get the ASCII codes.  I 
> don't know why and didn't dig into it, but I could; for now I figured I'd just 
> mention this.
> 

Thanks for checking. I should have mentioned that I'm running 27.0.50 with org 9.1.7 and ess 17.11. I see this problem with emacs -Q and regular emacs. If you're using the same versions and don't see this, then there should be something I can do in my config to fix it.

Best,

Tyler

Re: R output with session polluted with ascii color codes

[William Denton]

On 16 March 2018, Tyler Smith wrote:

> Thanks for checking. I should have mentioned that I'm running 27.0.50 with org 
> 9.1.7 and ess 17.11. I see this problem with emacs -Q and regular emacs. If 
> you're using the same versions and don't see this, then there should be 
> something I can do in my config to fix it.

I think we're the same---Org says it's 9.1.6, but I just compiled it fresh from 
source five minutes ago, so it should be up to date.  Not sure what's going on 
there.  I recompiled Emacs from fresh source, too, and updated the ESS package 
(17.11, ELPA 20180314.612).

Anyway, I tried again, and there was still no problem in Emacs.  That made me 
wonder if the difference was in R.  When I run "as_tibble(iris)" in the R 
console in a terminal I don't get the ASCII colour codes, so maybe that's where 
the difference lies?  I'm running R 3.4.2, compiled from source, on Ubuntu 
17.10.

Is there an R user on the list who sees the coloured text in the R console and 
can try the example in Org?

#+BEGIN_SRC R :results output :session RSESSION
library(tidyr)
library(dplyr)
as_tibble(iris)
#+END_SRC

Bill
--
William Denton :: Toronto, Canada   ---   Listening to Art: https://listeningtoart.org/
https://www.miskatonic.org/         ---   GHG.EARTH: http://ghg.earth/
Caveat lector.                      ---   STAPLR: http://staplr.org/

Re: R output with session polluted with ascii color codes

[Tyler Smith]

On Fri, Mar 16, 2018, at 2:55 PM, William Denton wrote:
> On 16 March 2018, Tyler Smith wrote:
> 
> Anyway, I tried again, and there was still no problem in Emacs.  That made me 
> wonder if the difference was in R.  When I run "as_tibble(iris)" in the R 
> console in a terminal I don't get the ASCII colour codes, so maybe that's where 
> the difference lies?  I'm running R 3.4.2, compiled from source, on Ubuntu 
> 17.10.

I've tried this as well, outside of Emacs. For me, it depends on which terminal emulator I use. xterm displays the colours, while gnome terminal does not. Which is odd, because I use ansi colour codes for my prompt and ls output (etc), and those do display as expected in both xterm and gnome terminal.

Weird!

Thanks,

Tyler