Extraneous output from Python code blocks using :session option

Extraneous output from Python code blocks using :session option

[Richard Stanton]

I'm trying to use the :session option so I can import modules, etc., just once at the beginning of my document, like with am IPython notebook. Unfortunately, the output from these code blocks contains some extraneous characters. For example:

#+BEGIN_SRC python :session :results output
a = 2
b = 3
c = 4
print 'a=      ', a
print 'b =     ', b
print 'a + b = ', a+b
#+END_SRC

#+RESULTS:
: 
: >>> >>> a=       2
: b =      3
: a + b =  5

How can I stop the production of all those ">" signs (sometimes they're dots), which don't appear if I run the same code block without the :session option?

Thanks.

Richard Stanton

Re: Extraneous output from Python code blocks using :session option

[Kyle Meyer]

[-- Attachment #1: Type: text/plain, Size: 1125 bytes --]

Richard Stanton <stanton@haas.berkeley.edu> wrote:
> I'm trying to use the :session option so I can import modules, etc.,
> just once at the beginning of my document, like with am IPython
> notebook. Unfortunately, the output from these code blocks contains
> some extraneous characters. For example:
>
> #+BEGIN_SRC python :session :results output
> a = 2
> b = 3
> c = 4
> print 'a=      ', a
> print 'b =     ', b
> print 'a + b = ', a+b
> #+END_SRC
>
> #+RESULTS:
> :
> : >>> >>> a=       2
> : b =      3
> : a + b =  5

I also see this behavior.  I asked the list some time ago [1], but I
didn't follow up beyond the initial email.

> How can I stop the production of all those ">" signs (sometimes
> they're dots), which don't appear if I run the same code block without
> the :session option?

I've attached a patch that seems to fix the example you gave and the
cases from my earlier email.  I don't know enough about babel's
internals to know if it is a good way to fix it, but at least it might
serve as a quick fix for you until there is a better solution.

[1] http://thread.gmane.org/gmane.emacs.orgmode/79014


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-ob-python.el-Strip-leading-session-characters.patch --]
[-- Type: text/x-diff, Size: 2137 bytes --]

From cfa2783e03152871c0d703a6f79e96254dbe6c44 Mon Sep 17 00:00:00 2001
From: Kyle Meyer <kyle@kyleam.com>
Date: Wed, 11 Mar 2015 14:17:09 -0400
Subject: [PATCH] ob-python.el: Strip leading session characters

* lisp/ob-python.el (org-babel-python-evaluate-session): Strip leading
  "..." and ">>>" from session output.
(org-babel-python-strip-session-chars): New function.
---
 lisp/ob-python.el | 30 ++++++++++++++++++++----------
 1 file changed, 20 insertions(+), 10 deletions(-)

diff --git a/lisp/ob-python.el b/lisp/ob-python.el
index dd3cc66..0267f22 100644
--- a/lisp/ob-python.el
+++ b/lisp/ob-python.el
@@ -306,16 +306,17 @@ (defun org-babel-python-evaluate-session
          (results
           (case result-type
             (output
-             (mapconcat
-              #'org-babel-trim
-              (butlast
-               (org-babel-comint-with-output
-                   (session org-babel-python-eoe-indicator t body)
-                 (funcall input-body body)
-                 (funcall send-wait) (funcall send-wait)
-                 (insert org-babel-python-eoe-indicator)
-                 (funcall send-wait))
-               2) "\n"))
+	     (org-babel-python-strip-session-chars
+	      (mapconcat
+	       #'org-babel-chomp
+	       (butlast
+		(org-babel-comint-with-output
+		    (session org-babel-python-eoe-indicator t body)
+		  (funcall input-body body)
+		  (funcall send-wait) (funcall send-wait)
+		  (insert org-babel-python-eoe-indicator)
+		  (funcall send-wait))
+		2) "\n")))
             (value
              (let ((tmp-file (org-babel-temp-file "python-")))
                (org-babel-comint-with-output
@@ -339,6 +340,15 @@ (defun org-babel-python-read-string (string)
       (match-string 1 string)
     string))
 
+(defun org-babel-python-strip-session-chars (string)
+  "Remove leading >>> and ... from Python session output."
+  (with-temp-buffer
+    (insert string)
+    (goto-char (point-min))
+    (when (looking-at "\\s-*\\(\\(>>> \\)*\\(\\.\\.\\. \\)*\\)")
+      (delete-region (match-beginning 1) (match-end 1)))
+    (buffer-string)))
+
 (provide 'ob-python)
 
 
-- 
2.3.2


[-- Attachment #3: Type: text/plain, Size: 9 bytes --]


--
Kyle

Re: Extraneous output from Python code blocks using :session option

[Richard Stanton]

> On Mar 11, 2015, at 11:29 AM, Kyle Meyer <kyle@kyleam.com> wrote:
> 
> Richard Stanton <stanton@haas.berkeley.edu> wrote:
>> I'm trying to use the :session option so I can import modules, etc.,
>> just once at the beginning of my document, like with am IPython
>> notebook. Unfortunately, the output from these code blocks contains
>> some extraneous characters. For example:
>> 
>> #+BEGIN_SRC python :session :results output
>> a = 2
>> b = 3
>> c = 4
>> print 'a=      ', a
>> print 'b =     ', b
>> print 'a + b = ', a+b
>> #+END_SRC
>> 
>> #+RESULTS:
>> :
>> : >>> >>> a=       2
>> : b =      3
>> : a + b =  5
> 
> I also see this behavior.  I asked the list some time ago [1], but I
> didn't follow up beyond the initial email.
> 
>> How can I stop the production of all those ">" signs (sometimes
>> they're dots), which don't appear if I run the same code block without
>> the :session option?
> 
> I've attached a patch that seems to fix the example you gave and the
> cases from my earlier email.  I don't know enough about babel's
> internals to know if it is a good way to fix it, but at least it might
> serve as a quick fix for you until there is a better solution.
> 
> [1] http://thread.gmane.org/gmane.emacs.orgmode/79014
> 
> From cfa2783e03152871c0d703a6f79e96254dbe6c44 Mon Sep 17 00:00:00 2001
> From: Kyle Meyer <kyle@kyleam.com>
> Date: Wed, 11 Mar 2015 14:17:09 -0400
> Subject: [PATCH] ob-python.el: Strip leading session characters
> 
> * lisp/ob-python.el (org-babel-python-evaluate-session): Strip leading
>  "..." and ">>>" from session output.
> (org-babel-python-strip-session-chars): New function.
> ---
> lisp/ob-python.el | 30 ++++++++++++++++++++----------
> 1 file changed, 20 insertions(+), 10 deletions(-)
> 
> diff --git a/lisp/ob-python.el b/lisp/ob-python.el
> index dd3cc66..0267f22 100644
> --- a/lisp/ob-python.el
> +++ b/lisp/ob-python.el
> @@ -306,16 +306,17 @@ (defun org-babel-python-evaluate-session
>          (results
>           (case result-type
>             (output
> -             (mapconcat
> -              #'org-babel-trim
> -              (butlast
> -               (org-babel-comint-with-output
> -                   (session org-babel-python-eoe-indicator t body)
> -                 (funcall input-body body)
> -                 (funcall send-wait) (funcall send-wait)
> -                 (insert org-babel-python-eoe-indicator)
> -                 (funcall send-wait))
> -               2) "\n"))
> +	     (org-babel-python-strip-session-chars
> +	      (mapconcat
> +	       #'org-babel-chomp
> +	       (butlast
> +		(org-babel-comint-with-output
> +		    (session org-babel-python-eoe-indicator t body)
> +		  (funcall input-body body)
> +		  (funcall send-wait) (funcall send-wait)
> +		  (insert org-babel-python-eoe-indicator)
> +		  (funcall send-wait))
> +		2) "\n")))
>             (value
>              (let ((tmp-file (org-babel-temp-file "python-")))
>                (org-babel-comint-with-output
> @@ -339,6 +340,15 @@ (defun org-babel-python-read-string (string)
>       (match-string 1 string)
>     string))
> 
> +(defun org-babel-python-strip-session-chars (string)
> +  "Remove leading >>> and ... from Python session output."
> +  (with-temp-buffer
> +    (insert string)
> +    (goto-char (point-min))
> +    (when (looking-at "\\s-*\\(\\(>>> \\)*\\(\\.\\.\\. \\)*\\)")
> +      (delete-region (match-beginning 1) (match-end 1)))
> +    (buffer-string)))
> +
> (provide 'ob-python)
> 
> 
> -- 
> 2.3.2
> 
> 
> —
> Kyle

Many thanks, Kyle. It would be great if this could be fixed in the main code, but I’ll try your patch today.

Re: Extraneous output from Python code blocks using :session option

[John Kitchin]

Nice. I hope this makes it into org-mode one day.

In the mean time, I crafted an approach with another hook function that
is described here:

http://kitchingroup.cheme.cmu.edu/blog/2015/03/12/Making-org-mode-Python-sessions-look-better/

This can be done in an init file.

Kyle Meyer writes:

> Richard Stanton <stanton@haas.berkeley.edu> wrote:
>> I'm trying to use the :session option so I can import modules, etc.,
>> just once at the beginning of my document, like with am IPython
>> notebook. Unfortunately, the output from these code blocks contains
>> some extraneous characters. For example:
>>
>> #+BEGIN_SRC python :session :results output
>> a = 2
>> b = 3
>> c = 4
>> print 'a=      ', a
>> print 'b =     ', b
>> print 'a + b = ', a+b
>> #+END_SRC
>>
>> #+RESULTS:
>> :
>> : >>> >>> a=       2
>> : b =      3
>> : a + b =  5
>
> I also see this behavior.  I asked the list some time ago [1], but I
> didn't follow up beyond the initial email.
>
>> How can I stop the production of all those ">" signs (sometimes
>> they're dots), which don't appear if I run the same code block without
>> the :session option?
>
> I've attached a patch that seems to fix the example you gave and the
> cases from my earlier email.  I don't know enough about babel's
> internals to know if it is a good way to fix it, but at least it might
> serve as a quick fix for you until there is a better solution.
>
> [1] http://thread.gmane.org/gmane.emacs.orgmode/79014

--
Professor John Kitchin
Doherty Hall A207F
Department of Chemical Engineering
Carnegie Mellon University
Pittsburgh, PA 15213
412-268-7803
@johnkitchin
http://kitchingroup.cheme.cmu.edu

Re: Extraneous output from Python code blocks using :session option

[Richard Stanton]

This looks great. While we're patching this code, why does having blank lines inside function definitions cause such problems in :session mode? It would be nice if you could just type any valid Python code and have it execute (like you can in an IPython notebook cell)


> On Mar 12, 2015, at 8:17 AM, John Kitchin <jkitchin@andrew.cmu.edu> wrote:
> 
> Nice. I hope this makes it into org-mode one day.
> 
> In the mean time, I crafted an approach with another hook function that
> is described here:
> 
> http://kitchingroup.cheme.cmu.edu/blog/2015/03/12/Making-org-mode-Python-sessions-look-better/
> 
> This can be done in an init file.
> 
> Kyle Meyer writes:
> 
>> Richard Stanton <stanton@haas.berkeley.edu> wrote:
>>> I'm trying to use the :session option so I can import modules, etc.,
>>> just once at the beginning of my document, like with am IPython
>>> notebook. Unfortunately, the output from these code blocks contains
>>> some extraneous characters. For example:
>>> 
>>> #+BEGIN_SRC python :session :results output
>>> a = 2
>>> b = 3
>>> c = 4
>>> print 'a=      ', a
>>> print 'b =     ', b
>>> print 'a + b = ', a+b
>>> #+END_SRC
>>> 
>>> #+RESULTS:
>>> :
>>> : >>> >>> a=       2
>>> : b =      3
>>> : a + b =  5
>> 
>> I also see this behavior.  I asked the list some time ago [1], but I
>> didn't follow up beyond the initial email.
>> 
>>> How can I stop the production of all those ">" signs (sometimes
>>> they're dots), which don't appear if I run the same code block without
>>> the :session option?
>> 
>> I've attached a patch that seems to fix the example you gave and the
>> cases from my earlier email.  I don't know enough about babel's
>> internals to know if it is a good way to fix it, but at least it might
>> serve as a quick fix for you until there is a better solution.
>> 
>> [1] http://thread.gmane.org/gmane.emacs.orgmode/79014
> 
> --
> Professor John Kitchin
> Doherty Hall A207F
> Department of Chemical Engineering
> Carnegie Mellon University
> Pittsburgh, PA 15213
> 412-268-7803
> @johnkitchin
> http://kitchingroup.cheme.cmu.edu

Re: Extraneous output from Python code blocks using :session option

[Kyle Meyer]

John Kitchin <jkitchin@andrew.cmu.edu> wrote:
> Nice. I hope this makes it into org-mode one day.
>
> In the mean time, I crafted an approach with another hook function that
> is described here:
>
> http://kitchingroup.cheme.cmu.edu/blog/2015/03/12/Making-org-mode-Python-sessions-look-better/

Good idea.  Thanks for the information.

--
Kyle

Re: Extraneous output from Python code blocks using :session option

[Kyle Meyer]

Richard Stanton <stanton@haas.berkeley.edu> wrote:
> This looks great. While we're patching this code, why does having
> blank lines inside function definitions cause such problems in
> :session mode?

I think this is because the lines are being sent one by one, so sending
the blank line causes an issue, like typing enter with the cursor at _:

    >>> def abc():
    >>> _

    ...
      File "<stdin>", line 2

        ^
    IndentationError: expected an indented block
    >>>

I'll take a look at fixing this.  python.el in Emacs 24 seems to send
functions with blank lines over fine, so perhaps I can use it as an
example.

--
Kyle

Re: Extraneous output from Python code blocks using :session option

[Kyle Meyer]

[-- Attachment #1: Type: text/plain, Size: 1049 bytes --]

Kyle Meyer <kyle@kyleam.com> wrote:
> Richard Stanton <stanton@haas.berkeley.edu> wrote:
>> This looks great. While we're patching this code, why does having
>> blank lines inside function definitions cause such problems in
>> :session mode?
>
> I think this is because the lines are being sent one by one, so sending
> the blank line causes an issue, like typing enter with the cursor at _:
>
>     >>> def abc():
>     >>> _
>
>     ...
>       File "<stdin>", line 2
>
>         ^
>     IndentationError: expected an indented block
>     >>>
>
> I'll take a look at fixing this.  python.el in Emacs 24 seems to send
> functions with blank lines over fine, so perhaps I can use it as an
> example.

I've attached two patches.  The first one is a proposed way to deal with
the indentation issues in sessions.  It is very similar to what
python.el does for multiline input (use a temporary file and then
execute that from the shell).  The second is an update of my previous
patch to remove shell prompt characters that are leaking into the
output.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-ob-python.el-Allow-indented-code-in-sessions.patch --]
[-- Type: text/x-diff, Size: 2950 bytes --]

From 28f271314f7bff7c54696defe8e451da69cd2d6c Mon Sep 17 00:00:00 2001
From: Kyle Meyer <kyle@kyleam.com>
Date: Fri, 13 Mar 2015 02:45:04 -0400
Subject: [PATCH 1/2] ob-python.el: Allow indented code in sessions

* lisp/ob-python.el (org-babel-python-evaluate-session): Recognize
indented code in session and treat it differently to avoid syntax
errors.

For session blocks with results set to 'output', go through an
intermediate file when there is indented code to prevent
indentation-related errors when sending code to the Python shell.  For
session blocks with results set to 'value', issue an user-error.
These (usually) would have resulted in a syntax error in the shell
anyway, and the '_' variable can't be used to get the last value here
(as it is for non-indented code) because it isn't set when executing
the code through a file.
---
 lisp/ob-python.el | 24 ++++++++++++++++++++++++
 1 file changed, 24 insertions(+)

diff --git a/lisp/ob-python.el b/lisp/ob-python.el
index dd3cc66..8c51679 100644
--- a/lisp/ob-python.el
+++ b/lisp/ob-python.el
@@ -238,6 +238,14 @@ (defconst org-babel-python-pp-wrapper-method
 
 open('%s', 'w').write( pprint.pformat(main()) )")
 
+(defconst org-babel-python-indented-lines-session-method
+  (concat "fname= '%s'; fh = open(fname); "
+	  "exec(compile(fh.read(), fname, 'exec')); "
+	  "fh.close()")
+  "Single line to execute indented Python code in session.
+%s will be formatted with the file name of the file containing
+ the code.")
+
 (defun org-babel-python-evaluate
   (session body &optional result-type result-params preamble)
   "Evaluate BODY as Python code."
@@ -303,6 +311,13 @@ (defun org-babel-python-evaluate-session
 		       (mapc (lambda (line) (insert line) (funcall send-wait))
 			     (split-string body "[\r\n]"))
 		       (funcall send-wait)))
+	 (indented-p (org-babel-python--indented-p body))
+	 (body (if indented-p
+		   (let ((tmp-file (org-babel-temp-file "python-")))
+		     (with-temp-file tmp-file (insert body))
+		     (format org-babel-python-indented-lines-session-method
+			     tmp-file))
+		 body))
          (results
           (case result-type
             (output
@@ -317,6 +332,8 @@ (defun org-babel-python-evaluate-session
                  (funcall send-wait))
                2) "\n"))
             (value
+	     (when indented-p
+	       (user-error "Value output limited to unindented lines with session"))
              (let ((tmp-file (org-babel-temp-file "python-")))
                (org-babel-comint-with-output
                    (session org-babel-python-eoe-indicator nil body)
@@ -339,6 +356,13 @@ (defun org-babel-python-read-string (string)
       (match-string 1 string)
     string))
 
+(defun org-babel-python--indented-p (input)
+  "Return true if any line in INPUT is indented."
+  (with-temp-buffer
+    (insert input)
+    (goto-char (point-min))
+    (re-search-forward "^\\s-" nil t)))
+
 (provide 'ob-python)
 
 
-- 
2.3.1


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #3: 0002-ob-python.el-Strip-leading-session-characters.patch --]
[-- Type: text/x-diff, Size: 2314 bytes --]

From 3d433fac1162f1544fbcfc7e4a8675a258764f34 Mon Sep 17 00:00:00 2001
From: Kyle Meyer <kyle@kyleam.com>
Date: Fri, 13 Mar 2015 02:46:38 -0400
Subject: [PATCH 2/2] ob-python.el: Strip leading session characters

* lisp/ob-python.el (org-babel-python-evaluate-session): Strip extra leading
  "..." and ">>>" from session output.
---
 lisp/ob-python.el | 34 ++++++++++++++++++++++++----------
 1 file changed, 24 insertions(+), 10 deletions(-)

diff --git a/lisp/ob-python.el b/lisp/ob-python.el
index 8c51679..170b6cd 100644
--- a/lisp/ob-python.el
+++ b/lisp/ob-python.el
@@ -321,16 +321,17 @@ (defun org-babel-python-evaluate-session
          (results
           (case result-type
             (output
-             (mapconcat
-              #'org-babel-trim
-              (butlast
-               (org-babel-comint-with-output
-                   (session org-babel-python-eoe-indicator t body)
-                 (funcall input-body body)
-                 (funcall send-wait) (funcall send-wait)
-                 (insert org-babel-python-eoe-indicator)
-                 (funcall send-wait))
-               2) "\n"))
+	     (org-babel-python--strip-session-chars
+	      (mapconcat
+	       #'org-babel-chomp
+	       (butlast
+		(org-babel-comint-with-output
+		    (session org-babel-python-eoe-indicator t body)
+		  (funcall input-body body)
+		  (funcall send-wait) (funcall send-wait)
+		  (insert org-babel-python-eoe-indicator)
+		  (funcall send-wait))
+		2) "\n")))
             (value
 	     (when indented-p
 	       (user-error "Value output limited to unindented lines with session"))
@@ -363,6 +364,19 @@ (defun org-babel-python--indented-p (input)
     (goto-char (point-min))
     (re-search-forward "^\\s-" nil t)))
 
+(defun org-babel-python--strip-session-chars (string)
+  "Remove leading '>>>' and '...' from Python session output.
+`org-babel-comint-with-output' splits by
+`comint-prompt-regexp' (which includes '>>>' and '...'), but, in
+some situations, these still make it through at the start of the
+output string."
+  (with-temp-buffer
+    (insert string)
+    (goto-char (point-min))
+    (when (looking-at "\\s-*\n\\(\\(>>> \\)\\|\\(\\.\\.\\. \\)\\)*")
+      (delete-region (match-beginning 0) (match-end 0)))
+    (buffer-string)))
+
 (provide 'ob-python)
 
 
-- 
2.3.1


[-- Attachment #4: Type: text/plain, Size: 9 bytes --]


--
Kyle

Re: Extraneous output from Python code blocks using :session option

[Nicolas Goaziou]

Hello,

Kyle Meyer <kyle@kyleam.com> writes:

> I've attached two patches.  The first one is a proposed way to deal with
> the indentation issues in sessions.  It is very similar to what
> python.el does for multiline input (use a temporary file and then
> execute that from the shell).  The second is an update of my previous
> patch to remove shell prompt characters that are leaking into the
> output.

Thank you.

I'll just comment about code, not functionality.

> +(defconst org-babel-python-indented-lines-session-method
> +  (concat "fname= '%s'; fh = open(fname); "
> +	  "exec(compile(fh.read(), fname, 'exec')); "
> +	  "fh.close()")
> +  "Single line to execute indented Python code in session.
> +%s will be formatted with the file name of the file containing
> + the code.")
  ^^^
spurious space

> +(defun org-babel-python--indented-p (input)
> +  "Return true if any line in INPUT is indented."

Non-nil if ...

> +  (with-temp-buffer
> +    (insert input)
> +    (goto-char (point-min))
> +    (re-search-forward "^\\s-" nil t)))

aka

  (org-string-match-p "^[ \t]" input)

> +(defun org-babel-python--strip-session-chars (string)
> +  "Remove leading '>>>' and '...' from Python session output.
> +`org-babel-comint-with-output' splits by
> +`comint-prompt-regexp' (which includes '>>>' and '...'), but, in
> +some situations, these still make it through at the start of the
> +output string."

Argument STRING is not explained in the docstring.

> +  (with-temp-buffer
> +    (insert string)
> +    (goto-char (point-min))
> +    (when (looking-at "\\s-*\n\\(\\(>>> \\)\\|\\(\\.\\.\\. \\)\\)*")
> +      (delete-region (match-beginning 0) (match-end 0)))
> +    (buffer-string)))

aka

  (replace-regexp-in-string "\\`\\s-*\n\\(>>> \\|\\.\\.\\. \\)*" "" string)


Regards,

-- 
Nicolas Goaziou

Re: Extraneous output from Python code blocks using :session option

[Kyle Meyer]

[-- Attachment #1: Type: text/plain, Size: 593 bytes --]

Nicolas Goaziou <mail@nicolasgoaziou.fr> wrote:
> Hello,
>
> Kyle Meyer <kyle@kyleam.com> writes:
>
>> I've attached two patches.  The first one is a proposed way to deal with
>> the indentation issues in sessions.  It is very similar to what
>> python.el does for multiline input (use a temporary file and then
>> execute that from the shell).  The second is an update of my previous
>> patch to remove shell prompt characters that are leaking into the
>> output.
>
> Thank you.
>
> I'll just comment about code, not functionality.

Thanks for your comments.  I've attached updated patches.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-ob-python.el-Allow-indented-code-in-sessions.patch --]
[-- Type: text/x-diff, Size: 2593 bytes --]

From 41aae465038fd789fe92611ecf1fae7e9d08ab5b Mon Sep 17 00:00:00 2001
From: Kyle Meyer <kyle@kyleam.com>
Date: Fri, 13 Mar 2015 02:45:04 -0400
Subject: [PATCH 1/2] ob-python.el: Allow indented code in sessions

* lisp/ob-python.el (org-babel-python-evaluate-session): Recognize
indented code in session and treat it differently to avoid syntax
errors.

For session blocks with results set to 'output', go through an
intermediate file when there is indented code to prevent
indentation-related errors when sending code to the Python shell.  For
session blocks with results set to 'value', issue an user-error.
These (usually) would have resulted in a syntax error in the shell
anyway, and the '_' variable can't be used to get the last value here
(as it is for non-indented code) because it isn't set when executing
the code through a file.
---
 lisp/ob-python.el | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)

diff --git a/lisp/ob-python.el b/lisp/ob-python.el
index dd3cc66..0dc74ca 100644
--- a/lisp/ob-python.el
+++ b/lisp/ob-python.el
@@ -238,6 +238,14 @@ (defconst org-babel-python-pp-wrapper-method
 
 open('%s', 'w').write( pprint.pformat(main()) )")
 
+(defconst org-babel-python-indented-lines-session-method
+  (concat "fname= '%s'; fh = open(fname); "
+	  "exec(compile(fh.read(), fname, 'exec')); "
+	  "fh.close()")
+  "Single line to execute indented Python code in session.
+%s will be formatted with the file name of the file containing
+the code.")
+
 (defun org-babel-python-evaluate
   (session body &optional result-type result-params preamble)
   "Evaluate BODY as Python code."
@@ -303,6 +311,13 @@ (defun org-babel-python-evaluate-session
 		       (mapc (lambda (line) (insert line) (funcall send-wait))
 			     (split-string body "[\r\n]"))
 		       (funcall send-wait)))
+	 (indented-p (org-string-match-p "^[ \t]" body))
+	 (body (if indented-p
+		   (let ((tmp-file (org-babel-temp-file "python-")))
+		     (with-temp-file tmp-file (insert body))
+		     (format org-babel-python-indented-lines-session-method
+			     tmp-file))
+		 body))
          (results
           (case result-type
             (output
@@ -317,6 +332,8 @@ (defun org-babel-python-evaluate-session
                  (funcall send-wait))
                2) "\n"))
             (value
+	     (when indented-p
+	       (user-error "Value output limited to unindented lines with session"))
              (let ((tmp-file (org-babel-temp-file "python-")))
                (org-babel-comint-with-output
                    (session org-babel-python-eoe-indicator nil body)
-- 
2.3.1


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #3: 0002-ob-python.el-Strip-leading-session-characters.patch --]
[-- Type: text/x-diff, Size: 2161 bytes --]

From 28b6a003e4ae1dbb1474350203444e76df9e8572 Mon Sep 17 00:00:00 2001
From: Kyle Meyer <kyle@kyleam.com>
Date: Fri, 13 Mar 2015 02:46:38 -0400
Subject: [PATCH 2/2] ob-python.el: Strip leading session characters

* lisp/ob-python.el (org-babel-python-evaluate-session): Strip extra leading
  "..." and ">>>" from session output.
---
 lisp/ob-python.el | 30 ++++++++++++++++++++----------
 1 file changed, 20 insertions(+), 10 deletions(-)

diff --git a/lisp/ob-python.el b/lisp/ob-python.el
index 0dc74ca..e976d9c 100644
--- a/lisp/ob-python.el
+++ b/lisp/ob-python.el
@@ -321,16 +321,17 @@ (defun org-babel-python-evaluate-session
          (results
           (case result-type
             (output
-             (mapconcat
-              #'org-babel-trim
-              (butlast
-               (org-babel-comint-with-output
-                   (session org-babel-python-eoe-indicator t body)
-                 (funcall input-body body)
-                 (funcall send-wait) (funcall send-wait)
-                 (insert org-babel-python-eoe-indicator)
-                 (funcall send-wait))
-               2) "\n"))
+	     (org-babel-python--strip-session-chars
+	      (mapconcat
+	       #'org-babel-chomp
+	       (butlast
+		(org-babel-comint-with-output
+		    (session org-babel-python-eoe-indicator t body)
+		  (funcall input-body body)
+		  (funcall send-wait) (funcall send-wait)
+		  (insert org-babel-python-eoe-indicator)
+		  (funcall send-wait))
+		2) "\n")))
             (value
 	     (when indented-p
 	       (user-error "Value output limited to unindented lines with session"))
@@ -356,6 +357,15 @@ (defun org-babel-python-read-string (string)
       (match-string 1 string)
     string))
 
+(defun org-babel-python--strip-session-chars (string)
+  "Remove leading '>>>' and '...' from STRING.
+`org-babel-comint-with-output' splits the Python session output
+by `comint-prompt-regexp' (which includes '>>>' and '...'), but,
+in some situations, these still make it through at the start of
+the output string."
+  (replace-regexp-in-string "\\`\\s-*\n\\(>>> \\|\\.\\.\\. \\)*" ""
+			    string))
+
 (provide 'ob-python)
 
 
-- 
2.3.1


[-- Attachment #4: Type: text/plain, Size: 9 bytes --]


--
Kyle

Re: Extraneous output from Python code blocks using :session option

[Nicolas Goaziou]

Kyle Meyer <kyle@kyleam.com> writes:

> Thanks for your comments.  I've attached updated patches.

Thanks.

> Subject: [PATCH 1/2] ob-python.el: Allow indented code in sessions
>
> * lisp/ob-python.el (org-babel-python-evaluate-session): Recognize
> indented code in session and treat it differently to avoid syntax
> errors.

You also need to list new functions and variables in commit message,
e.g.,

  (org-babel-python-indented-lines-session-method): New variable.

> * lisp/ob-python.el (org-babel-python-evaluate-session): Strip extra leading
>   "..." and ">>>" from session output.

Ditto.

  (org-babel-python--strip-session-chars): New function.

> +(defun org-babel-python--strip-session-chars (string)
> +  "Remove leading '>>>' and '...' from STRING.
> +`org-babel-comint-with-output' splits the Python session output
> +by `comint-prompt-regexp' (which includes '>>>' and '...'), but,
> +in some situations, these still make it through at the start of
> +the output string."

Wouldn't it be better to find the situations above at the source? If may
be due to some indentation problem at the beginning of the string, in
which case an `org-trim' within `org-babel-comint-with-output' could
solve it.

WDYT?


Regards,

Re: Extraneous output from Python code blocks using :session option

[Kyle Meyer]

Nicolas Goaziou <mail@nicolasgoaziou.fr> wrote:
[...]
> You also need to list new functions and variables in commit message,
> e.g.,

Thanks.  I'll update the commit messages.

> Wouldn't it be better to find the situations above at the source? If may
> be due to some indentation problem at the beginning of the string, in
> which case an `org-trim' within `org-babel-comint-with-output' could
> solve it.

I agree, but I haven't figured out a good way to do this.  As far as I
can tell, it seems like the input sent in the right format.  Instead, it
is empty prompt markers that are causing the issue (see examples at end
for details).

I think there are three separate problems with Python session output
right now.

1. The empty prompt markers from multi-line code mess up the output
   processing and leak into the final output.

   The current patch fixes this in the final combined output, but I've
   realized it doesn't cover all the cases.  (I initially thought the
   markers only leaked into the start of the output, but they can be in
   the middle as well.)

   Plus, as you mention, it seems like it'd be better to fix this more
   upstream.  Since the input seems to be sent in correctly, I think
   this leaves org-babel-comint-with-output.  Specifically, output lines
   that consist of just a prompt (like '>>> ' or '... ') could be
   filtered out before the output lines are joined and returned.
   comint-prompt-regexp could be used for this check.  One problem with
   this is that I don't know if this will have unwanted effects on the
   output for other languages that use org-babel-comint-with-output.

2. org-babel-python-evaluate-session will eat whitespace at the
   beginning of output.

   The leading spaces below would be lost in the results.

   #+begin_src python :session :results output
     print('   <-- leading spaces')
   #+end_src

   Changing org-babel-trim to org-babel-chomp in
   org-babel-python-evaluate-session fixes this.

3. Valid code that has indentation and spaces cannot be sent.

   I think this is just a limitation of the shell, since blank lines are
   used to signal the end of blocks.  One of the patches I sent deals
   with this like python.el does (by using an intermediate file).

   One issue with that patch is whether it should check for blank
   lines *and* indentation, because right now it only checks for
   indentation, meaning that some code is unnecessarily sent through an
   intermediate file.

Below are two examples that explain #1 in more detail.

* Single line

#+begin_src python :session :results output
  3
#+end_src

org-babel-python-evaluate-session splits the input by newlines,
resulting in 3 (no newline) being inserted.  It then sends this input,
followed by a few more empty inputs (not needed here, but required to
finish the block if the statement had indentation), and then inserts and
sends the babel eoe indicator.

This ends up in the Python shell as

    >>> 3
    3
    >>>
    >>>
    >>>
    >>> 'org_babel_python_eoe'
    'org_babel_python_eoe'
    >>>

org-babel-comint-with-output receives then a series of text output,

    "", "3\n>>> ", "", ">>> ", "", ">>> ", "", ">>> ", "",
    "'org_babel_python_eoe'\n>>> "

which it concatenates to

    3
    >>> >>> >>> >>> 'org_babel_python_eoe'
    >>>

It then splits this by comint-prompt-regexp, which includes '>>>'
at the start of a line, resulting in

    ("3\n" ">>> >>> >>> 'org_babel_python_eoe'\n" "")

org-babel-python-evaluate-session receives this, removes the last two
elements, and trims the first, giving the correct output:

#+RESULTS:
: 3


* Multiple lines, no indentation

Here is the same example, but with some problematic lines added to the
beginning.

#+begin_src python :session :results output
  x = 3
  y = 4
  x + y
#+end_src

org-babel-python-evaluate-session splits the input by newlines and sends
x = 3, then x = 4, and then 3, as well as the empty lines and eoe
signal.

This gets ends up in the Python shell as

    x = 3
    >>> y = 4
    >>> x + y
    7
    >>> 
    >>> 
    >>> 
    >>> 'org_babel_python_eoe'
    'org_babel_python_eoe'
    >>> 

org-babel-comint-with-output receives the a series of text output,

    "", ">>> ",  <- From 'x = 3'
    "", ">>> ",  <- From 'y = 4'
    "", "7\n>>> ", "", ">>> ", "", ">>> ", "", ">>> ", ""
    "'org_babel_python_eoe'\n>>> "

which it concatenates to

    >>> >>> 7
    >>> >>> >>> >>> 'org_babel_python_eoe'
    >>>

It then splits this by comint-prompt-regexp, which includes '>>>'
at the start of a line, resulting in

    ("" ">>> 7\n" ">>> >>> >>> 'org_babel_python_eoe'\n" "")

org-babel-python-evaluate-session receives this, removes the last two
elements, and trims the others.  Unlike the output above, this has an
empty line at the beginning due to the additional ">>> " that resulted
from "x = 3", as well as ">>> " before 7 due to the additional ">>> "
that resulted from "y = 4".

#+RESULTS:
: 
: >>> 7

--
Kyle