From mboxrd@z Thu Jan 1 00:00:00 1970 From: Kyle Meyer Subject: Re: Proposal and RFC for improving ob-python Date: Thu, 10 Dec 2015 22:39:21 -0500 Message-ID: <874mfp64d2.fsf@kyleam.com> References: Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:40633) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1a7EYb-00043B-TJ for emacs-orgmode@gnu.org; Thu, 10 Dec 2015 22:39:47 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1a7EYY-0003wX-Jq for emacs-orgmode@gnu.org; Thu, 10 Dec 2015 22:39:45 -0500 Received: from pb-smtp0.int.icgroup.com ([208.72.237.35]:60408 helo=sasl.smtp.pobox.com) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1a7EYY-0003wT-F2 for emacs-orgmode@gnu.org; Thu, 10 Dec 2015 22:39:42 -0500 List-Id: "General discussions about Org-mode." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-orgmode-bounces+geo-emacs-orgmode=m.gmane.org@gnu.org Sender: emacs-orgmode-bounces+geo-emacs-orgmode=m.gmane.org@gnu.org To: =?utf-8?Q?Ond=C5=99ej?= Grover Cc: emacs-orgmode@gnu.org --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Ond=C5=99ej Grover writes: [...] >> But what about when :results !=3D value? Doesn't your proposal only >> handle returning the last value? >> > You mean :results output ? In that case it could just omit the > "open(...).write(...) " part and capture anything the console prints befo= re > the primary prompt appears again. > Or it could capture stdout into a file, but I think that's needlessly > complicated: > > python -i << HEREDOC_END > import sys > sys.stdout =3D open() # replace stdout with some file > _ =3D block_eval(""" > > """) > sys.stdout.close() > sys.stdout =3D sys.__stdout__ # restore stdout > HEREDOC_END (I think you should rely on Org babel's shared utilities, e.g. org-babel-eval and org-babel-eval-read-file, unless there is a specific reason they won't do. At any rate, right now I'm just trying to get a picture of how block_eval would fit in.) So these are the main options to consider for Python: | | output | value | |-------------+--------+-------| | non-session | a | b | | session | c | d | a) Using block_eval here seems unhelpful here because the method in place is already simple: (org-babel-eval org-babel-python-command body) b) block_eval replaces the need for dumping the code into a function. Using block_eval in org-babel-python-wrapper-method would allow you to get rid of the odd "return" use. c) Currently, this is handled by inserting the body lines one at a time and then getting the output by splitting on the prompt. Aside from race condition issues, this has limitations with handling blank lines in the body. block_eval could help here because the whole body could be sent through as one line of input to the prompt. This should resolve most issues with prompt splitting, as well as lift the body formatting restrictions for sessions. However, the main issue I see with this is that I think it would lose the main distinction between session and non-session for ":results output" that is discussed in (info "(org) Results of evaluation"): #+BEGIN_SRC python :results output print "hello" 2 print "bye" #+END_SRC #+RESULTS: : hello : bye versus this #+BEGIN_SRC python :results output :session print "hello" 2 print "bye" #+END_SRC #+RESULTS: : hello : 2 : bye d) The variable _ is assigned the last value by the shell, so the current mechanism should work even if block_eval is used as described in c. I've attached a quick-and-dirty patch, just for discussion. At least under light testing, it resolves the issues I listed in my previous response [1]. It does have the change in behavior discussed in c, which I see as problematic. It also raises the issue of how to incorporate the block_eval code. I think there are two main options. * Require the user have the module installed and import it. * Make the module into a snippet to go into the ob-python.el. The first option is what the current patch does. The problem is that it requires users to install an external module. The second option avoids that but is uglier and harder to maintain. [1] There's a remaining issue, which I believe is specific to newer versions of python.el, where the first time a session block is executed, it doesn't wait for the shell to load up before sending input. --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=0001-WIP-lisp-ob-python.el-Use-block_eval.patch >From 642370236f6ca06e70ea32866eedcdba161fd2c4 Mon Sep 17 00:00:00 2001 From: Kyle Meyer Date: Thu, 10 Dec 2015 22:09:05 -0500 Subject: [PATCH] WIP lisp/ob-python.el: Use block_eval --- lisp/ob-python.el | 108 +++++++++++++++++++++++++++--------------------------- 1 file changed, 54 insertions(+), 54 deletions(-) diff --git a/lisp/ob-python.el b/lisp/ob-python.el index 95a06f5..baaaf82 100644 --- a/lisp/ob-python.el +++ b/lisp/ob-python.el @@ -224,19 +224,18 @@ (defun org-babel-python-initiate-session (&optional session _params) (defvar org-babel-python-eoe-indicator "'org_babel_python_eoe'" "A string to indicate that evaluation has completed.") -(defconst org-babel-python-wrapper-method - " -def main(): -%s -open('%s', 'w').write( str(main()) )") +(defconst org-babel-python-block-eval-import + "from block_eval import block_eval as _ob_python_eval") + +(defconst org-babel-python-wrapper-method + (concat org-babel-python-block-eval-import " +open('%s', 'w').write(str(_ob_python_eval('%s')))")) (defconst org-babel-python-pp-wrapper-method - " + (concat org-babel-python-block-eval-import " import pprint -def main(): -%s - -open('%s', 'w').write( pprint.pformat(main()) )") +from block_eval import block_eval as _ob_python_eval +open('%s', 'w').write(pprint.pformat(str(_ob_python_eval('%s'))))")) (defun org-babel-python-evaluate (session body &optional result-type result-params preamble) @@ -247,6 +246,9 @@ (defun org-babel-python-evaluate (org-babel-python-evaluate-external-process body result-type result-params preamble))) +(defun org-babel-python-escape-single-quotes (s) + (replace-regexp-in-string "'" "\\\\'" s)) + (defun org-babel-python-evaluate-external-process (body &optional result-type result-params preamble) "Evaluate BODY in external python process. @@ -262,22 +264,22 @@ (defun org-babel-python-evaluate-external-process (org-babel-eval org-babel-python-command (concat - (if preamble (concat preamble "\n") "") + (if preamble (concat preamble "\\n") "") (format (if (member "pp" result-params) org-babel-python-pp-wrapper-method org-babel-python-wrapper-method) - (mapconcat - (lambda (line) (format "\t%s" line)) - (split-string - (org-remove-indentation - (org-babel-trim body)) - "[\r\n]") "\n") - (org-babel-process-file-name tmp-file 'noquote)))) + (org-babel-process-file-name tmp-file 'noquote) + (org-babel-python-escape-single-quotes + (mapconcat #'identity + (split-string + (org-remove-indentation + (org-babel-trim body)) + "[\r\n]") "\\n"))))) (org-babel-eval-read-file tmp-file)))))) (org-babel-result-cond result-params raw - (org-babel-python-table-or-string (org-babel-trim raw))))) + (org-babel-python-table-or-string raw)))) (defun org-babel-python-evaluate-session (session body &optional result-type result-params) @@ -285,49 +287,47 @@ (defun org-babel-python-evaluate-session If RESULT-TYPE equals `output' then return standard output as a string. If RESULT-TYPE equals `value' then return the value of the last statement in BODY, as elisp." - (let* ((send-wait (lambda () (comint-send-input nil t) (sleep-for 0 5))) - (dump-last-value - (lambda - (tmp-file pp) - (mapc - (lambda (statement) (insert statement) (funcall send-wait)) - (if pp - (list - "import pprint" - (format "open('%s', 'w').write(pprint.pformat(_))" - (org-babel-process-file-name tmp-file 'noquote))) - (list (format "open('%s', 'w').write(str(_))" - (org-babel-process-file-name tmp-file - 'noquote))))))) - (input-body (lambda (body) - (mapc (lambda (line) (insert line) (funcall send-wait)) - (split-string body "[\r\n]")) - (funcall send-wait))) - (results + (let* ((block (concat org-babel-python-block-eval-import + "; _ob_python_eval('%s')")) + (send (lambda (s &optional block-p) + (insert + (if block-p + (format block + (org-babel-python-escape-single-quotes + (mapconcat #'identity + (split-string + (org-remove-indentation + (org-babel-trim s)) + "[\r\n]") "\\n"))) + s)) + (comint-send-input nil t))) + (results (case result-type (output (mapconcat - #'org-babel-trim + #'identity (butlast (org-babel-comint-with-output (session org-babel-python-eoe-indicator t body) - (funcall input-body body) - (funcall send-wait) (funcall send-wait) - (insert org-babel-python-eoe-indicator) - (funcall send-wait)) + (funcall send body t) + (funcall send org-babel-python-eoe-indicator)) 2) "\n")) (value - (let ((tmp-file (org-babel-temp-file "python-"))) - (org-babel-comint-with-output - (session org-babel-python-eoe-indicator nil body) - (let ((comint-process-echoes nil)) - (funcall input-body body) - (funcall dump-last-value tmp-file - (member "pp" result-params)) - (funcall send-wait) (funcall send-wait) - (insert org-babel-python-eoe-indicator) - (funcall send-wait))) - (org-babel-eval-read-file tmp-file)))))) + (let* ((tmp-file (org-babel-temp-file "python-")) + (dump-statement + (format + (if (member "pp" result-params) + (concat "import pprint; " + "open('%s', 'w').write(pprint.pformat(_))") + "open('%s', 'w').write(str(_))") + (org-babel-process-file-name tmp-file 'noquote))) + (comint-process-echoes nil)) + (org-babel-comint-with-output + (session org-babel-python-eoe-indicator nil body) + (funcall send body t) + (funcall send dump-statement) + (funcall send org-babel-python-eoe-indicator)) + (org-babel-eval-read-file tmp-file)))))) (unless (string= (substring org-babel-python-eoe-indicator 1 -1) results) (org-babel-result-cond result-params results -- 2.6.3 --=-=-= Content-Type: text/plain -- Kyle --=-=-=--