Re: Proposal and RFC for improving ob-python

[Kyle Meyer <kyle@kyleam.com>]

[-- Attachment #1: Type: text/plain, Size: 3629 bytes --]

Ondřej Grover <ondrej.grover@gmail.com> writes:

[...]

>> But what about when :results != value?  Doesn't your proposal only
>> handle returning the last value?
>>
> You mean :results output ? In that case it could just omit the
> "open(...).write(...) " part and capture anything the console prints before
> the primary prompt appears again.
> Or it could capture stdout into a file, but I think that's needlessly
> complicated:
>
> python -i << HEREDOC_END
> import sys
> sys.stdout = open(<TMP FILE or PIPE>)   # replace stdout with some file
> _ = block_eval("""
> <CODE BLOCK BODY>
> """)
> sys.stdout.close()
> sys.stdout = sys.__stdout__  # restore stdout
> HEREDOC_END

(I think you should rely on Org babel's shared utilities,
e.g. org-babel-eval and org-babel-eval-read-file, unless there is a
specific reason they won't do.  At any rate, right now I'm just trying
to get a picture of how block_eval would fit in.)

So these are the main options to consider for Python:

    |             | output | value |
    |-------------+--------+-------|
    | non-session | a      | b     |
    | session     | c      | d     |

a) Using block_eval here seems unhelpful here because the method in
   place is already simple:

   (org-babel-eval org-babel-python-command body)

b) block_eval replaces the need for dumping the code into a function.
   Using block_eval in org-babel-python-wrapper-method would allow you
   to get rid of the odd "return" use.

c) Currently, this is handled by inserting the body lines one at a time
   and then getting the output by splitting on the prompt.  Aside from
   race condition issues, this has limitations with handling blank lines
   in the body.

   block_eval could help here because the whole body could be sent
   through as one line of input to the prompt.  This should resolve most
   issues with prompt splitting, as well as lift the body formatting
   restrictions for sessions.

   However, the main issue I see with this is that I think it would lose
   the main distinction between session and non-session for ":results
   output" that is discussed in (info "(org) Results of evaluation"):

       #+BEGIN_SRC python :results output
        print "hello"
        2
        print "bye"
       #+END_SRC

       #+RESULTS:
       : hello
       : bye

   versus this

       #+BEGIN_SRC python :results output :session
        print "hello"
        2
        print "bye"
       #+END_SRC

       #+RESULTS:
       : hello
       : 2
       : bye

d) The variable _ is assigned the last value by the shell, so the
   current mechanism should work even if block_eval is used as described
   in c.

I've attached a quick-and-dirty patch, just for discussion.  At least
under light testing, it resolves the issues I listed in my previous
response [1].  It does have the change in behavior discussed in c, which
I see as problematic.

It also raises the issue of how to incorporate the block_eval code.  I
think there are two main options.

* Require the user have the module installed and import it.

* Make the module into a snippet to go into the ob-python.el.

The first option is what the current patch does.  The problem is that it
requires users to install an external module.  The second option avoids
that but is uglier and harder to maintain.


[1] There's a remaining issue, which I believe is specific to newer
    versions of python.el, where the first time a session block is
    executed, it doesn't wait for the shell to load up before sending
    input.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-WIP-lisp-ob-python.el-Use-block_eval.patch --]
[-- Type: text/x-diff, Size: 6676 bytes --]

From 642370236f6ca06e70ea32866eedcdba161fd2c4 Mon Sep 17 00:00:00 2001
From: Kyle Meyer <kyle@kyleam.com>
Date: Thu, 10 Dec 2015 22:09:05 -0500
Subject: [PATCH] WIP lisp/ob-python.el: Use block_eval

---
 lisp/ob-python.el | 108 +++++++++++++++++++++++++++---------------------------
 1 file changed, 54 insertions(+), 54 deletions(-)

diff --git a/lisp/ob-python.el b/lisp/ob-python.el
index 95a06f5..baaaf82 100644
--- a/lisp/ob-python.el
+++ b/lisp/ob-python.el
@@ -224,19 +224,18 @@ (defun org-babel-python-initiate-session (&optional session _params)
 
 (defvar org-babel-python-eoe-indicator "'org_babel_python_eoe'"
   "A string to indicate that evaluation has completed.")
-(defconst org-babel-python-wrapper-method
-  "
-def main():
-%s
 
-open('%s', 'w').write( str(main()) )")
+(defconst org-babel-python-block-eval-import
+  "from block_eval import block_eval as _ob_python_eval")
+
+(defconst org-babel-python-wrapper-method
+  (concat org-babel-python-block-eval-import "
+open('%s', 'w').write(str(_ob_python_eval('%s')))"))
 (defconst org-babel-python-pp-wrapper-method
-  "
+  (concat org-babel-python-block-eval-import "
 import pprint
-def main():
-%s
-
-open('%s', 'w').write( pprint.pformat(main()) )")
+from block_eval import block_eval as _ob_python_eval
+open('%s', 'w').write(pprint.pformat(str(_ob_python_eval('%s'))))"))
 
 (defun org-babel-python-evaluate
   (session body &optional result-type result-params preamble)
@@ -247,6 +246,9 @@ (defun org-babel-python-evaluate
     (org-babel-python-evaluate-external-process
      body result-type result-params preamble)))
 
+(defun org-babel-python-escape-single-quotes (s)
+  (replace-regexp-in-string "'" "\\\\'" s))
+
 (defun org-babel-python-evaluate-external-process
   (body &optional result-type result-params preamble)
   "Evaluate BODY in external python process.
@@ -262,22 +264,22 @@ (defun org-babel-python-evaluate-external-process
                     (org-babel-eval
                      org-babel-python-command
                      (concat
-                      (if preamble (concat preamble "\n") "")
+                      (if preamble (concat preamble "\\n") "")
                       (format
                        (if (member "pp" result-params)
                            org-babel-python-pp-wrapper-method
                          org-babel-python-wrapper-method)
-                       (mapconcat
-                        (lambda (line) (format "\t%s" line))
-                        (split-string
-                         (org-remove-indentation
-                          (org-babel-trim body))
-                         "[\r\n]") "\n")
-                       (org-babel-process-file-name tmp-file 'noquote))))
+                       (org-babel-process-file-name tmp-file 'noquote)
+		       (org-babel-python-escape-single-quotes
+			(mapconcat #'identity
+				   (split-string
+				    (org-remove-indentation
+				     (org-babel-trim body))
+				    "[\r\n]") "\\n")))))
                     (org-babel-eval-read-file tmp-file))))))
     (org-babel-result-cond result-params
       raw
-      (org-babel-python-table-or-string (org-babel-trim raw)))))
+      (org-babel-python-table-or-string raw))))
 
 (defun org-babel-python-evaluate-session
     (session body &optional result-type result-params)
@@ -285,49 +287,47 @@ (defun org-babel-python-evaluate-session
 If RESULT-TYPE equals `output' then return standard output as a
 string.  If RESULT-TYPE equals `value' then return the value of the
 last statement in BODY, as elisp."
-  (let* ((send-wait (lambda () (comint-send-input nil t) (sleep-for 0 5)))
-	 (dump-last-value
-	  (lambda
-	    (tmp-file pp)
-	    (mapc
-	     (lambda (statement) (insert statement) (funcall send-wait))
-	     (if pp
-		 (list
-		  "import pprint"
-		  (format "open('%s', 'w').write(pprint.pformat(_))"
-			  (org-babel-process-file-name tmp-file 'noquote)))
-	       (list (format "open('%s', 'w').write(str(_))"
-			     (org-babel-process-file-name tmp-file
-                                                          'noquote)))))))
-	 (input-body (lambda (body)
-		       (mapc (lambda (line) (insert line) (funcall send-wait))
-			     (split-string body "[\r\n]"))
-		       (funcall send-wait)))
-         (results
+  (let* ((block (concat org-babel-python-block-eval-import
+			"; _ob_python_eval('%s')"))
+	 (send (lambda (s &optional block-p)
+		 (insert
+		  (if block-p
+		      (format block
+			      (org-babel-python-escape-single-quotes
+			       (mapconcat #'identity
+					  (split-string
+					   (org-remove-indentation
+					    (org-babel-trim s))
+					   "[\r\n]") "\\n")))
+		    s))
+	    (comint-send-input nil t)))
+	 (results
           (case result-type
             (output
              (mapconcat
-              #'org-babel-trim
+	      #'identity
               (butlast
                (org-babel-comint-with-output
                    (session org-babel-python-eoe-indicator t body)
-                 (funcall input-body body)
-                 (funcall send-wait) (funcall send-wait)
-                 (insert org-babel-python-eoe-indicator)
-                 (funcall send-wait))
+		 (funcall send body t)
+                 (funcall send org-babel-python-eoe-indicator))
                2) "\n"))
             (value
-             (let ((tmp-file (org-babel-temp-file "python-")))
-               (org-babel-comint-with-output
-                   (session org-babel-python-eoe-indicator nil body)
-                 (let ((comint-process-echoes nil))
-                   (funcall input-body body)
-                   (funcall dump-last-value tmp-file
-                            (member "pp" result-params))
-                   (funcall send-wait) (funcall send-wait)
-                   (insert org-babel-python-eoe-indicator)
-                   (funcall send-wait)))
-               (org-babel-eval-read-file tmp-file))))))
+	     (let* ((tmp-file (org-babel-temp-file "python-"))
+		    (dump-statement
+		     (format
+		      (if (member "pp" result-params)
+			  (concat "import pprint; "
+				  "open('%s', 'w').write(pprint.pformat(_))")
+			"open('%s', 'w').write(str(_))")
+		      (org-babel-process-file-name tmp-file 'noquote)))
+		    (comint-process-echoes nil))
+	       (org-babel-comint-with-output
+		   (session org-babel-python-eoe-indicator nil body)
+		 (funcall send body t)
+		 (funcall send dump-statement)
+		 (funcall send org-babel-python-eoe-indicator))
+	       (org-babel-eval-read-file tmp-file))))))
     (unless (string= (substring org-babel-python-eoe-indicator 1 -1) results)
       (org-babel-result-cond result-params
 	results
-- 
2.6.3


[-- Attachment #3: Type: text/plain, Size: 9 bytes --]


--
Kyle