Help debugging R source code block output problem with :session

Help debugging R source code block output problem with :session

[Dylan Schwilk]

I returned to an org file this week and found that I am getting 
some strange source code block output for R code when I use the 
:session header. I have been able to duplicate this with a minimal 
init file. I strongly suspect it is some problem at my end but 
perhaps folks here can help me know here to look? It occurs with a 
minimal emacs initialization.

First, here is a small org file example of the problem. In the 
second results block the ">" appears to be treated as a prompt 
line to strip from output but only when a session is started. This 
happens with any ">" in output when the :session header occurs. 
the problem is dramatic when printing tibbles where the column 
modes are enclosed in angle brackets.

test.org
>>>
** Test org babel R output prompt problem
#+begin_src R :results output
print("<X> <Y> <!>")
print("one <two> three")
print("end")
#+end_src

#+RESULTS:
: [1] "<X> <Y> <!>"
: [1] "one <two> three"
: [1] "end"

#+begin_src R :results output :session "NEW"
print("<X> <Y> <!>")
print("one <two> three")
print("end")
#+end_src

#+RESULTS:
: [1] "<
: <
: <!>"
: [1] "one <
: three"
: [1] "end"
<<<

The second block produces the bad output. This occurs with a 
minimal setup.
I start emacs with
> emacs -Q -l "~/debugorg.el"

where debugorg.el is:

>>
require 'package)
add-to-list 'package-archives '("melpa" . 
"https://melpa.org/packages/") t)
add-to-list 'package-archives '("org" . 
"https://orgmode.org/elpa/") t)
package-initialize)

require 'ess-site)

(org-babel-do-load-languages
 'org-babel-load-languages
 '((R . t)))
<<

emacs version 26.3
org-mode version 9.3.7
ess version 18.10.3snapshot

-- 

Re: Help debugging R source code block output problem with :session

[Jack Kamm]

[-- Attachment #1: Type: text/plain, Size: 744 bytes --]

Hi Dylan,

I'm able to reproduce your error. I think this same bug has been
previously reported at [1]. As discussed there, the reason is that
substrings ending in ">" get stripped out by
org-babel-comint-with-output because it thinks they are prompts (in
particular, they match comint-prompt-regexp).

I'm attaching a patch to solve the issue. It replaces
org-babel-comint-with-output, with an R call to capture.output. I think
this approach is simpler and more robust, since it doesn't need to do
any parsing to remove prompts or other unwanted outputs.

I haven't committed to ob-R.el before, so thought it would be best to
solicit feedback here before merging this in.

Cheers,
Jack

[1] https://orgmode.org/list/875zgjh8wn.fsf@gmail.com/


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-ob-R-Fix-prompt-mangling-in-session-output.patch --]
[-- Type: text/x-patch, Size: 3698 bytes --]

From 1dc8e2d2cb01a4e6b82342ea8d8c965df8f5222c Mon Sep 17 00:00:00 2001
From: Jack Kamm <jackkamm@gmail.com>
Date: Fri, 28 Aug 2020 19:16:05 -0700
Subject: [PATCH] ob-R: Fix prompt mangling in session output

* lisp/ob-R.el (org-babel-R-evaluate-session): Replace
call to org-babel-comint-with-output with ess-send-to-buffer and
additional wrapper R code.

Fixes https://orgmode.org/list/875zgjh8wn.fsf@gmail.com/ and
https://orgmode.org/list/87r1rqled0.fsf@havana/
---
 lisp/ob-R.el | 79 ++++++++++++++++++++++++----------------------------
 1 file changed, 36 insertions(+), 43 deletions(-)

diff --git a/lisp/ob-R.el b/lisp/ob-R.el
index 5e9d35f58..d69cf23db 100644
--- a/lisp/ob-R.el
+++ b/lisp/ob-R.el
@@ -412,49 +412,42 @@ (defun org-babel-R-evaluate-session
 If RESULT-TYPE equals `output' then return standard output as a
 string.  If RESULT-TYPE equals `value' then return the value of the
 last statement in BODY, as elisp."
-  (cl-case result-type
-    (value
-     (with-temp-buffer
-       (insert (org-babel-chomp body))
-       (let ((ess-local-process-name
-	      (process-name (get-buffer-process session)))
-	     (ess-eval-visibly-p nil))
-	 (ess-eval-buffer nil)))
-     (let ((tmp-file (org-babel-temp-file "R-")))
-       (org-babel-comint-eval-invisibly-and-wait-for-file
-	session tmp-file
-	(format org-babel-R-write-object-command
-		(if row-names-p "TRUE" "FALSE")
-		(if column-names-p
-		    (if row-names-p "NA" "TRUE")
-		  "FALSE")
-		".Last.value" (org-babel-process-file-name tmp-file 'noquote)))
-       (org-babel-R-process-value-result
-	(org-babel-result-cond result-params
-	  (with-temp-buffer
-	    (insert-file-contents tmp-file)
-	    (org-babel-chomp (buffer-string) "\n"))
-	  (org-babel-import-elisp-from-file tmp-file '(16)))
-	column-names-p)))
-    (output
-     (mapconcat
-      'org-babel-chomp
-      (butlast
-       (delq nil
-	     (mapcar
-	      (lambda (line) (when (> (length line) 0) line))
-	      (mapcar
-	       (lambda (line) ;; cleanup extra prompts left in output
-		 (if (string-match
-		      "^\\([>+.]\\([ ][>.+]\\)*[ ]\\)"
-		      (car (split-string line "\n")))
-		     (substring line (match-end 1))
-		   line))
-	       (org-babel-comint-with-output (session org-babel-R-eoe-output)
-		 (insert (mapconcat 'org-babel-chomp
-				    (list body org-babel-R-eoe-indicator)
-				    "\n"))
-		 (inferior-ess-send-input)))))) "\n"))))
+  (let ((ess-local-process-name
+	 (process-name (get-buffer-process session)))
+	(ess-eval-visibly-p nil))
+    (cl-case result-type
+      (value
+       (with-temp-buffer
+	 (insert (org-babel-chomp body))
+	 (ess-eval-buffer nil))
+       (let ((tmp-file (org-babel-temp-file "R-")))
+	 (org-babel-comint-eval-invisibly-and-wait-for-file
+	  session tmp-file
+	  (format org-babel-R-write-object-command
+		  (if row-names-p "TRUE" "FALSE")
+		  (if column-names-p
+		      (if row-names-p "NA" "TRUE")
+		    "FALSE")
+		  ".Last.value" (org-babel-process-file-name tmp-file 'noquote)))
+	 (org-babel-R-process-value-result
+	  (org-babel-result-cond result-params
+	    (with-temp-buffer
+	      (insert-file-contents tmp-file)
+	      (org-babel-chomp (buffer-string) "\n"))
+	    (org-babel-import-elisp-from-file tmp-file '(16)))
+	  column-names-p)))
+      (output
+       (let ((tmp-file (org-babel-temp-file "R-")))
+	 (with-temp-buffer
+	   (insert (format "capture.output({
+%s
+}, file='%s')"
+			   body tmp-file))
+	   (ess-eval-buffer))
+	 (ess-wait-for-process)
+	 (with-temp-buffer
+	   (insert-file-contents tmp-file)
+	   (buffer-string)))))))
 
 (defun org-babel-R-process-value-result (result column-names-p)
   "R-specific processing of return value.
-- 
2.28.0

Re: Help debugging R source code block output problem with :session

[Dylan Schwilk]

Hi Jack,

The patch does fix that issue -- but it introduces a different bug 
for code blocks with ~:session~: the R block now only produces 
output from the last statement evaluated.

#+begin_src R :results output :exports both :session
4.0 * 3.5
log(10)  # natural log
log10(10)
(3 + 1) * 5
3^-1
1/0
#+end_src

#+results:
: [1] Inf

Without the ~:session header~ the output is as it was before the 
patch:

#+results:
: [1] 14
: [1] 2.302585
: [1] 1
: [1] 20
: [1] 0.3333333
: [1] Inf


-Dylan

Re: Help debugging R source code block output problem with :session

[Jack Kamm]

[-- Attachment #1: Type: text/plain, Size: 1125 bytes --]

Hi Dylan,

> The patch does fix that issue -- but it introduces a different bug 
> for code blocks with ~:session~: the R block now only produces 
> output from the last statement evaluated.

Of course, you're right. Good catch.

Here's another attempt. It fixes the issue by modifying the R comint
regular expression, requiring it to match at the beginning of the line.

I think this should fix most cases, including the examples you
sent. Still, it's not totally robust -- for example, it will still
mangle multiline strings, if one of the lines starts with a substring
that looks like a prompt.

I'd be interested to hear if the attached patch works for the common
cases you encounter, such as with tibbles.

As an aside, I personally use an alternative implementation of ob-R
sessions that doesn't suffer from this issue, and also provides some
other benefits like async evaluation [1]. I'm planning to submit these
changes to org-mode someday, but am not ready yet. But you may want to
check it out, it was able to solve the issue in the other thread I
linked as well.

[1] https://github.com/jackkamm/ob-session-async


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-ob-R-Fix-prompt-mangling-in-session-output.patch --]
[-- Type: text/x-patch, Size: 1505 bytes --]

From 9eaf81d708f88d06f14f9b6b9cf4182dd0fbb997 Mon Sep 17 00:00:00 2001
From: Jack Kamm <jackkamm@gmail.com>
Date: Sat, 29 Aug 2020 00:07:58 -0700
Subject: [PATCH] ob-R: Fix prompt mangling in session output

* lisp/ob-R.el (org-babel-R-evaluate-session): Force comint prompt
regexp to start at beginning of line, to prevent
org-babel-comint-with-output from splitting mid-line.

Fixes https://orgmode.org/list/875zgjh8wn.fsf@gmail.com/ and
https://orgmode.org/list/87r1rqled0.fsf@havana/
---
 lisp/ob-R.el | 12 +++++++-----
 1 file changed, 7 insertions(+), 5 deletions(-)

diff --git a/lisp/ob-R.el b/lisp/ob-R.el
index 5e9d35f58..10b3b6fe3 100644
--- a/lisp/ob-R.el
+++ b/lisp/ob-R.el
@@ -450,11 +450,13 @@ (defun org-babel-R-evaluate-session
 		      (car (split-string line "\n")))
 		     (substring line (match-end 1))
 		   line))
-	       (org-babel-comint-with-output (session org-babel-R-eoe-output)
-		 (insert (mapconcat 'org-babel-chomp
-				    (list body org-babel-R-eoe-indicator)
-				    "\n"))
-		 (inferior-ess-send-input)))))) "\n"))))
+	       (with-current-buffer session
+		 (let ((comint-prompt-regexp (concat "^" comint-prompt-regexp)))
+		   (org-babel-comint-with-output (session org-babel-R-eoe-output)
+		     (insert (mapconcat 'org-babel-chomp
+					(list body org-babel-R-eoe-indicator)
+					"\n"))
+		     (inferior-ess-send-input)))))))) "\n"))))
 
 (defun org-babel-R-process-value-result (result column-names-p)
   "R-specific processing of return value.
-- 
2.28.0

Re: Help debugging R source code block output problem with :session

[Berry, Charles via General discussions about Org-mode.]

This problem has been bugging people for years and previous attempts to solve it have always run up against creating more problems in the process of solving this one.

This workaround gives the same results with or without `:session "NEW"' and same as OPs first src block:

#+header: :prologue "capture.output({" :epilogue "})"
#+begin_src R :results value verbatim :session "NEW" 
print("<X> <Y> <!>")
print("one <two> three")
print("end")
#+end_src

If you do decide to dig into solving this, please be sure that remote sessions and graphical outputs are not broken.  test-ob-R.el does not cover those cases.  In fact, it is pretty short, so there are probably other things that could break without `make test' complaining.

HTH,

Chuck


> On Aug 29, 2020, at 12:24 AM, Jack Kamm <jackkamm@gmail.com> wrote:
> 
> Hi Dylan,
> 
>> The patch does fix that issue -- but it introduces a different bug 
>> for code blocks with ~:session~: the R block now only produces 
>> output from the last statement evaluated.
> 
> Of course, you're right. Good catch.
> 
> Here's another attempt. It fixes the issue by modifying the R comint
> regular expression, requiring it to match at the beginning of the line.

Re: Help debugging R source code block output problem with :session

[Dylan Schwilk]

Thanks for the workaround idea of grabbing the results as a 
string.

I'm not sure why this issue cropped up for me right now when I did 
not run into it in previous years. But I have run into other 
:results output issues in the past when ess changed.

Thanks,

Dylan


On Sat 29 Aug 2020 at 15:35, Berry, Charles 
<ccberry@health.ucsd.edu> wrote:

> This problem has been bugging people for years and previous 
> attempts to solve it have always run up against creating more 
> problems in the process of solving this one.
>
> This workaround gives the same results with or without `:session 
> "NEW"' and same as OPs first src block:
>
> #+header: :prologue "capture.output({" :epilogue "})"
> #+begin_src R :results value verbatim :session "NEW" 
> print("<X> <Y> <!>")
> print("one <two> three")
> print("end")
> #+end_src
>
> If you do decide to dig into solving this, please be sure that 
> remote sessions and graphical outputs are not broken. 
> test-ob-R.el does not cover those cases.  In fact, it is pretty 
> short, so there are probably other things that could break 
> without `make test' complaining.
>
> HTH,
>
> Chuck
>
>
>> On Aug 29, 2020, at 12:24 AM, Jack Kamm <jackkamm@gmail.com> 
>> wrote:
>> 
>> Hi Dylan,
>> 
>>> The patch does fix that issue -- but it introduces a different 
>>> bug 
>>> for code blocks with ~:session~: the R block now only produces 
>>> output from the last statement evaluated.
>> 
>> Of course, you're right. Good catch.
>> 
>> Here's another attempt. It fixes the issue by modifying the R 
>> comint
>> regular expression, requiring it to match at the beginning of 
>> the line.


-- 

Re: Help debugging R source code block output problem with :session

[Jack Kamm]

Hi Chuck,

"Berry, Charles" <ccberry@health.ucsd.edu> writes:

> This problem has been bugging people for years and previous attempts to solve it have always run up against creating more problems in the process of solving this one.
>
> If you do decide to dig into solving this, please be sure that remote sessions and graphical outputs are not broken.  test-ob-R.el does not cover those cases.  In fact, it is pretty short, so there are probably other things that could break without `make test' complaining.

Thanks for this info, it's helpful.

I do want to dig deeper into this, but probably won't have bandwidth in
the near future. But I've captured your note for when I return to
this. I'll be sure to test that any solution works with remote sessions
and graphics.

Best,
Jack

Re: Help debugging R source code block output problem with :session

[Jack Kamm]

[-- Attachment #1: Type: text/plain, Size: 905 bytes --]

Hi Chuck and Dylan,

"Berry, Charles" <ccberry@health.ucsd.edu> writes:

> This problem has been bugging people for years and previous attempts to solve it have always run up against creating more problems in the process of solving this one.
>
> If you do decide to dig into solving this, please be sure that remote sessions and graphical outputs are not broken.  test-ob-R.el does not cover those cases.  In fact, it is pretty short, so there are probably other things that could break without `make test' complaining.

I think I have found a robust solution to this issue, which I've
attached.

The solution avoids parsing for prompts at all, thus it should be robust
even for the case when the beginning of an output line looks like a
prompt.

I tested that the unit tests all pass, and that remote sessions and
graphical outputs still work. I also added Dylan's examples as unit
tests.

Cheers,
Jack


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-ob-R-Fix-session-output-with-substrings-matching-pro.patch --]
[-- Type: text/x-patch, Size: 4277 bytes --]

From 76d0eaa31506ce8a2f81f64eae43161db5721317 Mon Sep 17 00:00:00 2001
From: Jack Kamm <jackkamm@gmail.com>
Date: Mon, 7 Sep 2020 00:41:52 -0700
Subject: [PATCH] ob-R: Fix session output with substrings matching prompts

* lisp/ob-R.el (ess-send-string): Declare external function.
(org-babel-R-evaluate-session): New implementation for session output
results, that replaces calls to org-babel-comint-with-output with
custom code.
* testing/lisp/test-ob-R.el (test-ob-R/prompt-output): New test for
output results containing angle brackets.
(test-ob-R/output-nonprinted): New test for output results that aren't
explicitly printed.

Fixes issue reported in
https://orgmode.org/list/875zgjh8wn.fsf@gmail.com/,
https://orgmode.org/list/87r1rqled0.fsf@havana/
---
 lisp/ob-R.el              | 37 +++++++++++++++++++------------------
 testing/lisp/test-ob-R.el | 13 +++++++++++++
 2 files changed, 32 insertions(+), 18 deletions(-)

diff --git a/lisp/ob-R.el b/lisp/ob-R.el
index 5e9d35f58..b37e3965a 100644
--- a/lisp/ob-R.el
+++ b/lisp/ob-R.el
@@ -38,6 +38,8 @@ (declare-function ess-make-buffer-current "ext:ess-inf" ())
 (declare-function ess-eval-buffer "ext:ess-inf" (vis))
 (declare-function ess-wait-for-process "ext:ess-inf"
 		  (&optional proc sec-prompt wait force-redisplay))
+(declare-function ess-send-string "ext:ess-inf"
+		  (process string &optional visibly message _type))
 
 (defconst org-babel-header-args:R
   '((width		 . :any)
@@ -437,24 +439,23 @@ (defun org-babel-R-evaluate-session
 	  (org-babel-import-elisp-from-file tmp-file '(16)))
 	column-names-p)))
     (output
-     (mapconcat
-      'org-babel-chomp
-      (butlast
-       (delq nil
-	     (mapcar
-	      (lambda (line) (when (> (length line) 0) line))
-	      (mapcar
-	       (lambda (line) ;; cleanup extra prompts left in output
-		 (if (string-match
-		      "^\\([>+.]\\([ ][>.+]\\)*[ ]\\)"
-		      (car (split-string line "\n")))
-		     (substring line (match-end 1))
-		   line))
-	       (org-babel-comint-with-output (session org-babel-R-eoe-output)
-		 (insert (mapconcat 'org-babel-chomp
-				    (list body org-babel-R-eoe-indicator)
-				    "\n"))
-		 (inferior-ess-send-input)))))) "\n"))))
+     (let* ((tmp-file (org-babel-temp-file "R-")))
+       (with-temp-file tmp-file
+	 (insert (concat body "\n" org-babel-R-eoe-indicator)))
+       (with-current-buffer session
+	 (let* ((process (get-buffer-process (current-buffer)))
+		(string-buffer "")
+		(comint-output-filter-functions
+		 (cons (lambda (text) (setq string-buffer
+					    (concat string-buffer text)))
+		       comint-output-filter-functions)))
+	   (ess-send-string
+	    process (format "source('%s', print.eval=TRUE)"
+			    (org-babel-process-file-name tmp-file 'noquote)))
+	   (while (not (string-match (regexp-quote org-babel-R-eoe-output)
+				     string-buffer))
+	     (accept-process-output process))
+	   (substring string-buffer 0 (match-beginning 0))))))))
 
 (defun org-babel-R-process-value-result (result column-names-p)
   "R-specific processing of return value.
diff --git a/testing/lisp/test-ob-R.el b/testing/lisp/test-ob-R.el
index 7ce340ba4..ff7ea19d5 100644
--- a/testing/lisp/test-ob-R.el
+++ b/testing/lisp/test-ob-R.el
@@ -97,6 +97,19 @@ (ert-deftest test-ob-R/results-file ()
      (org-babel-goto-named-result "TESTSRC") (forward-line 1)
      (should (string= "[[file:junk/test.org]]"
 		      (buffer-substring-no-properties (point-at-bol) (point-at-eol)))))))
+
+(ert-deftest test-ob-R/prompt-output ()
+  (let (ess-ask-for-ess-directory ess-history-file)
+    (org-test-with-temp-text
+     "#+begin_src R :results output :session\nprint(\"<X> <Y> <!>\")\nprint(\"one <two> three\")\nprint(\"end\")\n#+end_src\n"
+     (should (string= "[1] \"<X> <Y> <!>\"\n[1] \"one <two> three\"\n[1] \"end\"\n" (org-babel-execute-src-block))))))
+
+(ert-deftest test-ob-R/output-nonprinted ()
+  (let (ess-ask-for-ess-directory ess-history-file)
+    (org-test-with-temp-text
+     "#+begin_src R :results output :session\n4.0 * 3.5\nlog(10)\nlog10(10)\n(3 + 1) * 5\n3^-1\n1/0\n#+end_src\n"
+     (should (string= "[1] 14\n[1] 2.302585\n[1] 1\n[1] 20\n[1] 0.3333333\n[1] Inf\n" (org-babel-execute-src-block))))))
+
 (provide 'test-ob-R)
 
 ;;; test-ob-R.el ends here
-- 
2.28.0

Re: Help debugging R source code block output problem with :session

[Jack Kamm]

[-- Attachment #1: Type: text/plain, Size: 190 bytes --]

I just realized my patch had an issue where it freezes if there is an
error in the source block.

I'm attaching a second patch, to be applied on top of the first one, that fixes the issue.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: fix-R-prompt-patch.patch --]
[-- Type: text/x-patch, Size: 1044 bytes --]

diff --git a/lisp/ob-R.el b/lisp/ob-R.el
index b37e3965a..5ddf0ebd1 100644
--- a/lisp/ob-R.el
+++ b/lisp/ob-R.el
@@ -441,7 +441,7 @@ (defun org-babel-R-evaluate-session
     (output
      (let* ((tmp-file (org-babel-temp-file "R-")))
        (with-temp-file tmp-file
-	 (insert (concat body "\n" org-babel-R-eoe-indicator)))
+	 (insert body))
        (with-current-buffer session
 	 (let* ((process (get-buffer-process (current-buffer)))
 		(string-buffer "")
@@ -450,8 +450,9 @@ (defun org-babel-R-evaluate-session
 					    (concat string-buffer text)))
 		       comint-output-filter-functions)))
 	   (ess-send-string
-	    process (format "source('%s', print.eval=TRUE)"
-			    (org-babel-process-file-name tmp-file 'noquote)))
+	    process (format "tryCatch(source('%s', print.eval=TRUE), finally=print(%s))"
+			    (org-babel-process-file-name tmp-file 'noquote)
+			    org-babel-R-eoe-indicator))
 	   (while (not (string-match (regexp-quote org-babel-R-eoe-output)
 				     string-buffer))
 	     (accept-process-output process))

Re: Help debugging R source code block output problem with :session

[Berry, Charles via General discussions about Org-mode.]

Jack,

This does not work for my remote session.

I run from macOS locally and on a Linux host remotely.  I use 

	M-x shell RET
	R RET
	M-x ess-remote RET R RET

to start the R session.

The problem is that tempfiles on the remote host are like "/tmp/RtmpeFHudh/file23a66d2fc1f9", but emacs tries to use '/var/folders/kb/2hchpbyj7lb6z76l0q73w_fh0000gn/T/babel-OSXKNd/R-oNOVVB'

`org-babel-temp-file' doesn't honor remote connections AFAICS.

Maybe there is some comint or tramp idiom that would solve this, but I do not know what it is.

Chuck

> On Sep 7, 2020, at 1:45 AM, Jack Kamm <jackkamm@gmail.com> wrote:
> 
> I just realized my patch had an issue where it freezes if there is an
> error in the source block.
> 
> I'm attaching a second patch, to be applied on top of the first one, that fixes the issue.
> 
> diff --git a/lisp/ob-R.el b/lisp/ob-R.el
> index b37e3965a..5ddf0ebd1 100644
> --- a/lisp/ob-R.el
> +++ b/lisp/ob-R.el
> @@ -441,7 +441,7 @@ (defun org-babel-R-evaluate-session
>     (output
>      (let* ((tmp-file (org-babel-temp-file "R-")))
>        (with-temp-file tmp-file
> -	 (insert (concat body "\n" org-babel-R-eoe-indicator)))
> +	 (insert body))
>        (with-current-buffer session
> 	 (let* ((process (get-buffer-process (current-buffer)))
> 		(string-buffer "")
> @@ -450,8 +450,9 @@ (defun org-babel-R-evaluate-session
> 					    (concat string-buffer text)))
> 		       comint-output-filter-functions)))
> 	   (ess-send-string
> -	    process (format "source('%s', print.eval=TRUE)"
> -			    (org-babel-process-file-name tmp-file 'noquote)))
> +	    process (format "tryCatch(source('%s', print.eval=TRUE), finally=print(%s))"
> +			    (org-babel-process-file-name tmp-file 'noquote)
> +			    org-babel-R-eoe-indicator))
> 	   (while (not (string-match (regexp-quote org-babel-R-eoe-output)
> 				     string-buffer))
> 	     (accept-process-output process))

Re: Help debugging R source code block output problem with :session

[Jack Kamm]

[-- Attachment #1: Type: text/plain, Size: 2282 bytes --]

Hi Chuck,

> This does not work for my remote session.

Thanks for testing this.

The remote tests I originally tried did not cover this case -- I only
tested the case where both org-file and session were remote, but didn't
test the case where org-file was local and session was remote.

I've now tested this case as well, and added some fixes for it. I'm
reattaching the patch in full.

However, this implementation does require the R session to have the
correct default-directory -- see details below.

> The problem is that tempfiles on the remote host are like "/tmp/RtmpeFHudh/file23a66d2fc1f9", but emacs tries to use '/var/folders/kb/2hchpbyj7lb6z76l0q73w_fh0000gn/T/babel-OSXKNd/R-oNOVVB'
>
> `org-babel-temp-file' doesn't honor remote connections AFAICS.

org-babel-temp-file does honor remote connections, but relies on
default-directory to do so. I've moved the call to org-babel-temp-file
so it happens when the session buffer is current -- that way, the
tempfile will be correctly handled, assuming that the session's
default-directory is on the remote location.

If the session's default-directory isn't at the remote location, then
this will break. But, in this case some other things break as well, for
example inserting links to remote plots.

> Maybe there is some comint or tramp idiom that would solve this, but I do not know what it is.

Here are some ways to start a remote R session with correct
default-directory:

1. Do "M-x R", then when it prompts for a directory, use
"/scp:hostname:/some/path". Optionally, first visit that location with
"C-x C-f /scp:hostname:/some/path", so that the default value is
already there.

2. Alternatively, start the R session by evaluating a source block with
header argument ":dir /scp:hostname:/some/path".

3. If you prefer to use "M-x shell" with "ess-remote",
first visit the remote location with "C-x C-f /scp:hostname:/some/path", before calling "M-x shell".

4. If you prefer to start "M-x shell" locally and then
ssh in, it's still possible to have default-directory set, but it
requires some configuration [1]. Or you could use "M-x cd" to set it as well.

[1] https://emacs.stackexchange.com/questions/5589/automatically-update-default-directory-when-pwd-changes-in-shell-mode-and-term-m/5592#5592

Best,
Jack


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-ob-R-Fix-session-output-with-substrings-matching-pro.patch --]
[-- Type: text/x-patch, Size: 4289 bytes --]

From e7f1a59167de88fb9a5b96a0e1ac3199f105f600 Mon Sep 17 00:00:00 2001
From: Jack Kamm <jackkamm@gmail.com>
Date: Mon, 7 Sep 2020 00:41:52 -0700
Subject: [PATCH] ob-R: Fix session output with substrings matching prompts

* lisp/ob-R.el (ess-send-string): Declare external function.
(org-babel-R-evaluate-session): New implementation for session output
results, that replaces calls to org-babel-comint-with-output with
custom code.
* testing/lisp/test-ob-R.el (test-ob-R/prompt-output): New test for
output results containing angle brackets.
(test-ob-R/output-nonprinted): New test for output results that aren't
explicitly printed.

Fixes issue reported in
https://orgmode.org/list/875zgjh8wn.fsf@gmail.com/,
https://orgmode.org/list/87r1rqled0.fsf@havana/
---
 lisp/ob-R.el              | 38 ++++++++++++++++++++------------------
 testing/lisp/test-ob-R.el | 13 +++++++++++++
 2 files changed, 33 insertions(+), 18 deletions(-)

diff --git a/lisp/ob-R.el b/lisp/ob-R.el
index 5e9d35f58..dffbbe112 100644
--- a/lisp/ob-R.el
+++ b/lisp/ob-R.el
@@ -38,6 +38,8 @@ (declare-function ess-make-buffer-current "ext:ess-inf" ())
 (declare-function ess-eval-buffer "ext:ess-inf" (vis))
 (declare-function ess-wait-for-process "ext:ess-inf"
 		  (&optional proc sec-prompt wait force-redisplay))
+(declare-function ess-send-string "ext:ess-inf"
+		  (process string &optional visibly message _type))
 
 (defconst org-babel-header-args:R
   '((width		 . :any)
@@ -437,24 +439,24 @@ (defun org-babel-R-evaluate-session
 	  (org-babel-import-elisp-from-file tmp-file '(16)))
 	column-names-p)))
     (output
-     (mapconcat
-      'org-babel-chomp
-      (butlast
-       (delq nil
-	     (mapcar
-	      (lambda (line) (when (> (length line) 0) line))
-	      (mapcar
-	       (lambda (line) ;; cleanup extra prompts left in output
-		 (if (string-match
-		      "^\\([>+.]\\([ ][>.+]\\)*[ ]\\)"
-		      (car (split-string line "\n")))
-		     (substring line (match-end 1))
-		   line))
-	       (org-babel-comint-with-output (session org-babel-R-eoe-output)
-		 (insert (mapconcat 'org-babel-chomp
-				    (list body org-babel-R-eoe-indicator)
-				    "\n"))
-		 (inferior-ess-send-input)))))) "\n"))))
+     (with-current-buffer session
+       (let* ((tmp-file (org-babel-temp-file "R-"))
+	      (process (get-buffer-process (current-buffer)))
+	      (string-buffer "")
+	      (comint-output-filter-functions
+	       (cons (lambda (text) (setq string-buffer
+					  (concat string-buffer text)))
+		     comint-output-filter-functions)))
+	 (with-temp-file tmp-file
+	   (insert body))
+	 (ess-send-string
+	  process (format "tryCatch(source('%s', print.eval=TRUE), finally=print(%s))"
+			  (org-babel-process-file-name tmp-file 'noquote)
+			  org-babel-R-eoe-indicator))
+	 (while (not (string-match (regexp-quote org-babel-R-eoe-output)
+				   string-buffer))
+	   (accept-process-output process))
+	 (substring string-buffer 0 (match-beginning 0)))))))
 
 (defun org-babel-R-process-value-result (result column-names-p)
   "R-specific processing of return value.
diff --git a/testing/lisp/test-ob-R.el b/testing/lisp/test-ob-R.el
index 7ce340ba4..ff7ea19d5 100644
--- a/testing/lisp/test-ob-R.el
+++ b/testing/lisp/test-ob-R.el
@@ -97,6 +97,19 @@ (ert-deftest test-ob-R/results-file ()
      (org-babel-goto-named-result "TESTSRC") (forward-line 1)
      (should (string= "[[file:junk/test.org]]"
 		      (buffer-substring-no-properties (point-at-bol) (point-at-eol)))))))
+
+(ert-deftest test-ob-R/prompt-output ()
+  (let (ess-ask-for-ess-directory ess-history-file)
+    (org-test-with-temp-text
+     "#+begin_src R :results output :session\nprint(\"<X> <Y> <!>\")\nprint(\"one <two> three\")\nprint(\"end\")\n#+end_src\n"
+     (should (string= "[1] \"<X> <Y> <!>\"\n[1] \"one <two> three\"\n[1] \"end\"\n" (org-babel-execute-src-block))))))
+
+(ert-deftest test-ob-R/output-nonprinted ()
+  (let (ess-ask-for-ess-directory ess-history-file)
+    (org-test-with-temp-text
+     "#+begin_src R :results output :session\n4.0 * 3.5\nlog(10)\nlog10(10)\n(3 + 1) * 5\n3^-1\n1/0\n#+end_src\n"
+     (should (string= "[1] 14\n[1] 2.302585\n[1] 1\n[1] 20\n[1] 0.3333333\n[1] Inf\n" (org-babel-execute-src-block))))))
+
 (provide 'test-ob-R)
 
 ;;; test-ob-R.el ends here
-- 
2.28.0

Re: Help debugging R source code block output problem with :session

[Berry, Charles via General discussions about Org-mode.]

Jack,

I can confirm that this works on my setup in each of the scenarios in which `default-directory' got set correctly in the session buffer.

At some point, my default-directory got reset to drop the tramp prefix "/scp:/user@host:" in one session after faithfully running the src block several times. I am unable to sort out why or even reproduce this.

Chuck

> On Sep 7, 2020, at 1:07 PM, Jack Kamm <jackkamm@gmail.com> wrote:
> 
> Hi Chuck,
> 
>> This does not work for my remote session.
> 
> Thanks for testing this.
> 
> The remote tests I originally tried did not cover this case -- I only
> tested the case where both org-file and session were remote, but didn't
> test the case where org-file was local and session was remote.
> 
> I've now tested this case as well, and added some fixes for it. I'm
> reattaching the patch in full.
> 
> However, this implementation does require the R session to have the
> correct default-directory -- see details below.

Re: Help debugging R source code block output problem with :session

[Jack Kamm]

Hi Chuck,

> I can confirm that this works on my setup in each of the scenarios in which `default-directory' got set correctly in the session buffer.
>
> At some point, my default-directory got reset to drop the tramp prefix "/scp:/user@host:" in one session after faithfully running the src block several times. I am unable to sort out why or even reproduce this.

Thanks for taking the time to test again. If you have any further
feedback at any point, please don't hesitate.

Best,
Jack

Re: Help debugging R source code block output problem with :session

[Berry, Charles via General discussions about Org-mode.]

Jack,

> On Sep 7, 2020, at 6:06 PM, Jack Kamm <jackkamm@gmail.com> wrote:
> 
> Hi Chuck,
> 
>> I can confirm that this works on my setup in each of the scenarios in which `default-directory' got set correctly in the session buffer.
>> 
>> At some point, my default-directory got reset to drop the tramp prefix "/scp:/user@host:" in one session after faithfully running the src block several times. I am unable to sort out why or even reproduce this.
> 
> Thanks for taking the time to test again. If you have any further
> feedback at any point, please don't hesitate.
> 


Thanks. This seems like it will help me.  

Also, I wonder if the `:results value' case can be handled in a similar manner, viz. 

-  (let ((tmp-file (org-babel-temp-file "R-")))
+  (let ((tmp-file (with-current-buffer session (org-babel-temp-file "R-"))))
   
Best,
Chuck

Re: Help debugging R source code block output problem with :session

[Jack Kamm]

"Berry, Charles" <ccberry@health.ucsd.edu> writes:

> Also, I wonder if the `:results value' case can be handled in a similar manner, viz. 
>
> -  (let ((tmp-file (org-babel-temp-file "R-")))
> +  (let ((tmp-file (with-current-buffer session (org-babel-temp-file "R-"))))

Yes, if we did that then tmp-file would have a prefix like
"/scp:user@hostname:", and elisp would then know to read the result file
from the remote host.

Before pasting tmp-file into R code, we should also call

(org-babel-process-file-name tmp-file 'noquote)

to remove the tramp prefix when referring to the file from R.

Re: Help debugging R source code block output problem with :session

[Berry, Charles via General discussions about Org-mode.]

Jack,

Maybe I am confused here:

> On Sep 8, 2020, at 7:51 AM, Jack Kamm <jackkamm@gmail.com> wrote:
> 
> Yes, if we did that then tmp-file would have a prefix like
> "/scp:user@hostname:", and elisp would then know to read the result file
> from the remote host.
> 
> Before pasting tmp-file into R code, we should also call
> 
> (org-babel-process-file-name tmp-file 'noquote)
> 
> to remove the tramp prefix when referring to the file from R.

this is already present in `org-babel-R-evaluate-session' in the call to `org-babel-comint-eval-invisibly-and-wait-for-file'' just a couple of lines further down in the `(cl-case result-type (value ...))' branch.

The other use of `tmp-file' in that block is the one that requires the prefix, right?

So nothing more to fix.

??

Chuck

Re: Help debugging R source code block output problem with :session

[Jack Kamm]

Hi Chuck,

> this is already present in `org-babel-R-evaluate-session' in the call to `org-babel-comint-eval-invisibly-and-wait-for-file'' just a couple of lines further down in the `(cl-case result-type (value ...))' branch.
>
> The other use of `tmp-file' in that block is the one that requires the prefix, right?
>
> So nothing more to fix.
>
> ??

You're right, I didn't notice that it's already present -- nothing more
to fix.

Jack

Re: Help debugging R source code block output problem with :session

[Dylan Schwilk]

Thanks. This patch works for my org files.


On Tue 08 Sep 2020 at 12:41, Jack Kamm <jackkamm@gmail.com> wrote:

> Hi Chuck,
>
>> this is already present in `org-babel-R-evaluate-session' in 
>> the call to 
>> `org-babel-comint-eval-invisibly-and-wait-for-file'' just a 
>> couple of lines further down in the `(cl-case result-type 
>> (value ...))' branch.
>>
>> The other use of `tmp-file' in that block is the one that 
>> requires the prefix, right?
>>
>> So nothing more to fix.
>>
>> ??
>
> You're right, I didn't notice that it's already present -- 
> nothing more
> to fix.
>
> Jack


-- 

Re: Help debugging R source code block output problem with :session

[Jack Kamm]

Hi Jeremie,

Thanks for volunteering to maintain ob-R.el :)

I'm bumping this patch [0] to see if it could be merged into ob-R.el. It
fixes some long-standing issues with prompt mangling for :session blocks
with :results output.

Elsewhere in the thread, Chuck had a suggestion [1] to allow :results
value to be properly handled in remote sessions. It's not included in
the patch, but worth adding separately IMO.

It may be helpful reading the thread in full -- Chuck had several
comments on ob-R that I found insightful, and may be useful for you to
consider as maintainer. I'd highlight [2] in particular:

> If you do decide to dig into solving this, please be sure that remote
> sessions and graphical outputs are not broken.  test-ob-R.el does not
> cover those cases.  In fact, it is pretty short, so there are probably
> other things that could break without `make test' complaining.

All the best,
Jack

[0] https://orgmode.org/list/87ft7t9wqk.fsf@gmail.com/
[1] https://orgmode.org/list/1E0046A6-1FAB-45B5-9B08-68FE1BE98180@health.ucsd.edu/
[2] https://orgmode.org/list/352C7149-743F-4944-ACA5-7A1242B5A3AA@health.ucsd.edu/

Re: Help debugging R source code block output problem with :session

[Jeremie Juste]

Hello Jack,

> Thanks for volunteering to maintain ob-R.el :)
Thanks for the encouragements and the suggestions. 

I need will take some time to dive into the existing code but
I will definitely look into it and keep you posted.

I've also noticed a [1] bug waiting to be fixed.
[1] https://orgmode.org/list/71761E05-7D0A-4FEF-8BAF-4C776A2FC17E@health.ucsd.edu/

Best regards,
Jeremie 
On Wednesday, 28 Oct 2020 at 06:13, Jack Kamm wrote:
> Hi Jeremie,
>
> Thanks for volunteering to maintain ob-R.el :)
>
> I'm bumping this patch [0] to see if it could be merged into ob-R.el. It
> fixes some long-standing issues with prompt mangling for :session blocks
> with :results output.
>
> Elsewhere in the thread, Chuck had a suggestion [1] to allow :results
> value to be properly handled in remote sessions. It's not included in
> the patch, but worth adding separately IMO.
>
> It may be helpful reading the thread in full -- Chuck had several
> comments on ob-R that I found insightful, and may be useful for you to
> consider as maintainer. I'd highlight [2] in particular:
>
>> If you do decide to dig into solving this, please be sure that remote
>> sessions and graphical outputs are not broken.  test-ob-R.el does not
>> cover those cases.  In fact, it is pretty short, so there are probably
>> other things that could break without `make test' complaining.
>
> All the best,
> Jack
>
> [0] https://orgmode.org/list/87ft7t9wqk.fsf@gmail.com/
> [1] https://orgmode.org/list/1E0046A6-1FAB-45B5-9B08-68FE1BE98180@health.ucsd.edu/
> [2] https://orgmode.org/list/352C7149-743F-4944-ACA5-7A1242B5A3AA@health.ucsd.edu/
>

Re: Help debugging R source code block output problem with :session

[Dylan Schwilk]

Yes, the modified comint-prompt-regexp works for my use cases and 
I'm back to expected output and what I had last time I taught the 
course. This was my instinct for a simple solution as well, but 
I've not tested against remote sessions and graphical outputs as 
Charles Berry warns.

I'll take a look at your ob-session-async, thanks!

-Dylan


On Sat 29 Aug 2020 at 02:24, Jack Kamm <jackkamm@gmail.com> wrote:

> Hi Dylan,
>
>> The patch does fix that issue -- but it introduces a different 
>> bug 
>> for code blocks with ~:session~: the R block now only produces 
>> output from the last statement evaluated.
>
> Of course, you're right. Good catch.
>
> Here's another attempt. It fixes the issue by modifying the R 
> comint
> regular expression, requiring it to match at the beginning of 
> the line.
>
> I think this should fix most cases, including the examples you
> sent. Still, it's not totally robust -- for example, it will 
> still
> mangle multiline strings, if one of the lines starts with a 
> substring
> that looks like a prompt.
>
> I'd be interested to hear if the attached patch works for the 
> common
> cases you encounter, such as with tibbles.
>
> As an aside, I personally use an alternative implementation of 
> ob-R
> sessions that doesn't suffer from this issue, and also provides 
> some
> other benefits like async evaluation [1]. I'm planning to submit 
> these
> changes to org-mode someday, but am not ready yet. But you may 
> want to
> check it out, it was able to solve the issue in the other thread 
> I
> linked as well.
>
> [1] https://github.com/jackkamm/ob-session-async
>
> From 9eaf81d708f88d06f14f9b6b9cf4182dd0fbb997 Mon Sep 17 
> 00:00:00 2001
> From: Jack Kamm <jackkamm@gmail.com>
> Date: Sat, 29 Aug 2020 00:07:58 -0700
> Subject: [PATCH] ob-R: Fix prompt mangling in session output
>
> * lisp/ob-R.el (org-babel-R-evaluate-session): Force comint 
> prompt
> regexp to start at beginning of line, to prevent
> org-babel-comint-with-output from splitting mid-line.
>
> Fixes https://orgmode.org/list/875zgjh8wn.fsf@gmail.com/ and
> https://orgmode.org/list/87r1rqled0.fsf@havana/
> ---
>  lisp/ob-R.el | 12 +++++++-----
>  1 file changed, 7 insertions(+), 5 deletions(-)
>
> diff --git a/lisp/ob-R.el b/lisp/ob-R.el
> index 5e9d35f58..10b3b6fe3 100644
> --- a/lisp/ob-R.el
> +++ b/lisp/ob-R.el
> @@ -450,11 +450,13 @@ (defun org-babel-R-evaluate-session
>  		      (car (split-string line "\n")))
>  		     (substring line (match-end 1))
>  		   line))
> -	       (org-babel-comint-with-output (session 
> org-babel-R-eoe-output)
> -		 (insert (mapconcat 'org-babel-chomp
> -				    (list body 
> org-babel-R-eoe-indicator)
> -				    "\n"))
> -		 (inferior-ess-send-input)))))) "\n"))))
> +	       (with-current-buffer session
> +		 (let ((comint-prompt-regexp (concat "^" 
> comint-prompt-regexp)))
> +		   (org-babel-comint-with-output (session 
> org-babel-R-eoe-output)
> +		     (insert (mapconcat 'org-babel-chomp
> +					(list body 
> org-babel-R-eoe-indicator)
> +					"\n"))
> +		     (inferior-ess-send-input)))))))) "\n"))))
>  
>  (defun org-babel-R-process-value-result (result column-names-p)
>    "R-specific processing of return value.


-- 

Re: Help debugging R source code block output problem with :session

[Jeremie Juste]

Hello,

I must apologize again for the delay. I'll be more responsive from now
on.

@Jack, I have applied the patch at the bottom of the mail. It is not
your latest patch but it works as well and is able to handle R errors.

> https://orgmode.org/list/87ft7t9wqk.fsf@gmail.com/
I have also added the 2 test cases with to make sure we are aware if
this feature breaks in the future.

> https://orgmode.org/list/87h7slgbi5.fsf@gmail.com/
> I'd be interested to hear if the attached patch works for the common
> cases you encounter, such as with tibbles.


@ Ilja, sorry for the delayed reply.
Regarding tibbles we are still not out of the woods. But the current
patch would work if crayon option is removed.

#+BEGIN_SRC R :results output :session "local1" :pre
  options(crayon.enabled=FALSE)
  library(tidyr)
  library(dplyr)
  as_tibble(iris)
#+END_SRC

#+RESULTS:
#+begin_example
# A tibble: 150 x 5
   Sepal.Length Sepal.Width Petal.Length Petal.Width Species
          <dbl>       <dbl>        <dbl>       <dbl> <fct>  
 1          5.1         3.5          1.4         0.2 setosa 
 2          4.9         3            1.4         0.2 setosa 
 3          4.7         3.2          1.3         0.2 setosa 
 4          4.6         3.1          1.5         0.2 setosa 
 5          5           3.6          1.4         0.2 setosa 
 6          5.4         3.9          1.7         0.4 setosa 
 7          4.6         3.4          1.4         0.3 setosa 
 8          5           3.4          1.5         0.2 setosa 
 9          4.4         2.9          1.4         0.2 setosa 
10          4.9         3.1          1.5         0.1 setosa 
# … with 140 more rows
#+end_example

> 

@Chuck

> https://orgmode.org/list/352C7149-743F-4944-ACA5-7A1242B5A3AA@health.ucsd.edu/
>If you do decide to dig into solving this, please be sure that remote
>sessions and graphical outputs are not broken. test-ob-R.el does not
>cover those cases. In fact, it is pretty short, so there are probably
>other things that could break without `make test' complaining.


The current patch have been tested for remote connections as well and
AFAIK, nothing breaks.

But I'm afraid that the graphical output is broken and has long been
even before the path. The test for graphical output is compromised and
does not do the right test. I will suggest new ones. 


Please feel free to contact me or reply on the mailing list if you have
see any improvements to be made.

Best regards,
Jeremie



On Saturday, 29 Aug 2020 at 00:24, Jack Kamm wrote:

>>From 9eaf81d708f88d06f14f9b6b9cf4182dd0fbb997 Mon Sep 17 00:00:00 2001
> From: Jack Kamm <jackkamm@gmail.com>
> Date: Sat, 29 Aug 2020 00:07:58 -0700
> Subject: [PATCH] ob-R: Fix prompt mangling in session output
>
> * lisp/ob-R.el (org-babel-R-evaluate-session): Force comint prompt
> regexp to start at beginning of line, to prevent
> org-babel-comint-with-output from splitting mid-line.
>
> Fixes https://orgmode.org/list/875zgjh8wn.fsf@gmail.com/ and
> https://orgmode.org/list/87r1rqled0.fsf@havana/
> ---
>  lisp/ob-R.el | 12 +++++++-----
>  1 file changed, 7 insertions(+), 5 deletions(-)
>
> diff --git a/lisp/ob-R.el b/lisp/ob-R.el
> index 5e9d35f58..10b3b6fe3 100644
> --- a/lisp/ob-R.el
> +++ b/lisp/ob-R.el
> @@ -450,11 +450,13 @@ (defun org-babel-R-evaluate-session
>  		      (car (split-string line "\n")))
>  		     (substring line (match-end 1))
>  		   line))
> -	       (org-babel-comint-with-output (session org-babel-R-eoe-output)
> -		 (insert (mapconcat 'org-babel-chomp
> -				    (list body org-babel-R-eoe-indicator)
> -				    "\n"))
> -		 (inferior-ess-send-input)))))) "\n"))))
> +	       (with-current-buffer session
> +		 (let ((comint-prompt-regexp (concat "^" comint-prompt-regexp)))
> +		   (org-babel-comint-with-output (session org-babel-R-eoe-output)
> +		     (insert (mapconcat 'org-babel-chomp
> +					(list body org-babel-R-eoe-indicator)
> +					"\n"))
> +		     (inferior-ess-send-input)))))))) "\n"))))
>  
>  (defun org-babel-R-process-value-result (result column-names-p)
>    "R-specific processing of return value.
> -- 
> 2.28.0

Re: Help debugging R source code block output problem with :session

[John Hendy]

Greetings,

Apologies for writing off-list, but I don't have the original thread
anymore to reply to. I just pulled this group in directly from the
email I found... I just got bitten by this as well and burned most of
my afternoon trying to figure out what the issue was.

Any further thoughts on a patch that could be finalized/applied?

Is the one from the thread still the best from your continued experience?
https://list.orgmode.org/87zgxc42qg.fsf@gmail.com/

This r/orgmode post was what finally got me to the existence of this
as a previous issue (I'd just sent my own email to list as my
searching did not find the above):
https://www.reddit.com/r/orgmode/comments/pt3em4/source_block_modifying_format_of_results_r/

And it links to a patch as well:
https://gist.github.com/gtuckerkellogg/e356d20497cfdc8e4fc683412e320e3e

Many thanks,
John

Re: Help debugging R source code block output problem with :session

[Jeremie Juste]

Hello John,

Many thanks for reporting.  I'm short of time right now, but I'll take a closer look at the problem over
the weekend and keep you posted. Could you let me know which version of
org-mode you are currently using?

Best regards,
Jeremie Juste


On Tuesday, 11 Jan 2022 at 17:36, John Hendy wrote:
> Greetings,
>
> Apologies for writing off-list, but I don't have the original thread
> anymore to reply to. I just pulled this group in directly from the
> email I found... I just got bitten by this as well and burned most of
> my afternoon trying to figure out what the issue was.
>
> Any further thoughts on a patch that could be finalized/applied?
>
> Is the one from the thread still the best from your continued experience?
> https://list.orgmode.org/87zgxc42qg.fsf@gmail.com/
>
> This r/orgmode post was what finally got me to the existence of this
> as a previous issue (I'd just sent my own email to list as my
> searching did not find the above):
> https://www.reddit.com/r/orgmode/comments/pt3em4/source_block_modifying_format_of_results_r/
>
> And it links to a patch as well:
> https://gist.github.com/gtuckerkellogg/e356d20497cfdc8e4fc683412e320e3e
>
> Many thanks,
> John

Re: Help debugging R source code block output problem with :session

[John Hendy]

[-- Attachment #1: Type: text/plain, Size: 2390 bytes --]

On Wed, Jan 12, 2022 at 1:50 AM Jeremie Juste <jeremiejuste@gmail.com> wrote:
>
> Hello John,
>
> Many thanks for reporting.  I'm short of time right now, but I'll take a closer look at the problem over
> the weekend and keep you posted. Could you let me know which version of
> org-mode you are currently using?

Sure thing and thanks for the quick note! I use org from git and
*thought* it used to report the most recent release tag, but I see:

$ git log
commit 7fa8173282f85c2ca03cc7f51f28f6adfb250610 (HEAD -> master,
origin/master, origin/HEAD)
Author: Ian Martins <ianxm@jhu.edu>
Date:   Sat Jan 16 15:52:21 2021 -0500

But Mx-version:
Org mode version 9.4.4 (release_9.4.4-186-g7fa817.dirty @
/home/jwhendy/.elisp/org/lisp/)

So perhaps it appends the git commit to the system version?

Btw, it's dirty as I was able to apply Jack's patches (with some mods)
successfully and am a happy camper. In case it's helpful, I attached
the patches as I applied them (first -1, then -2) to this org-version.
Nothing special, they are just Jack's patches with a couple sections
removed (one change was already present, and I removed the adjustments
to the testing file as they were failing).

Thanks so much to all of you who contributed to that, as I'm so happy
that I can at least continue on with my report at this point with good
looking output!

Many thanks,
John

>
> Best regards,
> Jeremie Juste
>
>
> On Tuesday, 11 Jan 2022 at 17:36, John Hendy wrote:
> > Greetings,
> >
> > Apologies for writing off-list, but I don't have the original thread
> > anymore to reply to. I just pulled this group in directly from the
> > email I found... I just got bitten by this as well and burned most of
> > my afternoon trying to figure out what the issue was.
> >
> > Any further thoughts on a patch that could be finalized/applied?
> >
> > Is the one from the thread still the best from your continued experience?
> > https://list.orgmode.org/87zgxc42qg.fsf@gmail.com/
> >
> > This r/orgmode post was what finally got me to the existence of this
> > as a previous issue (I'd just sent my own email to list as my
> > searching did not find the above):
> > https://www.reddit.com/r/orgmode/comments/pt3em4/source_block_modifying_format_of_results_r/
> >
> > And it links to a patch as well:
> > https://gist.github.com/gtuckerkellogg/e356d20497cfdc8e4fc683412e320e3e
> >
> > Many thanks,
> > John

[-- Attachment #2: ob-R-1.patch --]
[-- Type: text/x-patch, Size: 1724 bytes --]

diff --git a/lisp/ob-R.el b/lisp/ob-R.el
index 5e9d35f58..b37e3965a 100644
--- a/lisp/ob-R.el
+++ b/lisp/ob-R.el
@@ -437,24 +439,23 @@ (defun org-babel-R-evaluate-session
 	  (org-babel-import-elisp-from-file tmp-file '(16)))
 	column-names-p)))
     (output
-     (mapconcat
-      'org-babel-chomp
-      (butlast
-       (delq nil
-	     (mapcar
-	      (lambda (line) (when (> (length line) 0) line))
-	      (mapcar
-	       (lambda (line) ;; cleanup extra prompts left in output
-		 (if (string-match
-		      "^\\([>+.]\\([ ][>.+]\\)*[ ]\\)"
-		      (car (split-string line "\n")))
-		     (substring line (match-end 1))
-		   line))
-	       (org-babel-comint-with-output (session org-babel-R-eoe-output)
-		 (insert (mapconcat 'org-babel-chomp
-				    (list body org-babel-R-eoe-indicator)
-				    "\n"))
-		 (inferior-ess-send-input)))))) "\n"))))
+     (let* ((tmp-file (org-babel-temp-file "R-")))
+       (with-temp-file tmp-file
+	 (insert (concat body "\n" org-babel-R-eoe-indicator)))
+       (with-current-buffer session
+	 (let* ((process (get-buffer-process (current-buffer)))
+		(string-buffer "")
+		(comint-output-filter-functions
+		 (cons (lambda (text) (setq string-buffer
+					    (concat string-buffer text)))
+		       comint-output-filter-functions)))
+	   (ess-send-string
+	    process (format "source('%s', print.eval=TRUE)"
+			    (org-babel-process-file-name tmp-file 'noquote)))
+	   (while (not (string-match (regexp-quote org-babel-R-eoe-output)
+				     string-buffer))
+	     (accept-process-output process))
+	   (substring string-buffer 0 (match-beginning 0))))))))
 
 (defun org-babel-R-process-value-result (result column-names-p)
   "R-specific processing of return value.


[-- Attachment #3: ob-R-2.patch --]
[-- Type: text/x-patch, Size: 1044 bytes --]

diff --git a/lisp/ob-R.el b/lisp/ob-R.el
index b37e3965a..5ddf0ebd1 100644
--- a/lisp/ob-R.el
+++ b/lisp/ob-R.el
@@ -441,7 +441,7 @@ (defun org-babel-R-evaluate-session
     (output
      (let* ((tmp-file (org-babel-temp-file "R-")))
        (with-temp-file tmp-file
-	 (insert (concat body "\n" org-babel-R-eoe-indicator)))
+	 (insert body))
        (with-current-buffer session
 	 (let* ((process (get-buffer-process (current-buffer)))
 		(string-buffer "")
@@ -450,8 +450,9 @@ (defun org-babel-R-evaluate-session
 					    (concat string-buffer text)))
 		       comint-output-filter-functions)))
 	   (ess-send-string
-	    process (format "source('%s', print.eval=TRUE)"
-			    (org-babel-process-file-name tmp-file 'noquote)))
+	    process (format "tryCatch(source('%s', print.eval=TRUE), finally=print(%s))"
+			    (org-babel-process-file-name tmp-file 'noquote)
+			    org-babel-R-eoe-indicator))
 	   (while (not (string-match (regexp-quote org-babel-R-eoe-output)
 				     string-buffer))
 	     (accept-process-output process))