emacs-orgmode@gnu.org archives
 help / color / mirror / code / Atom feed
* Proposal and RFC for improving ob-python
@ 2015-12-05 17:17 Ondřej Grover
  2015-12-10  7:22 ` Kyle Meyer
  2016-08-29 12:53 ` Greg Minshall
  0 siblings, 2 replies; 12+ messages in thread
From: Ondřej Grover @ 2015-12-05 17:17 UTC (permalink / raw)
  To: emacs-orgmode

[-- Attachment #1: Type: text/plain, Size: 3290 bytes --]

Hello,

I've been playing around with the Org-mode Babel framework and I am
grateful to all the contributors for making this wonderful library. After
some time I noticed that Python support seems a little hacky and
inconsistent and after reading through ob-python.el and consulting Python
documentation I came up with a proposal for improving it.

The ob-ipython project tries to solve this hackiness in a different way by
using the client-server infrastructure of IPython/Jupyter. That works quite
well too, but my hope is that improving ob-python.el would also make it
simpler to use IPython as the python REPL, relying only on the core of the
Python language.

It essentially boils down to implementing progn-like eval() function in
Python which would return the result of the last statement if it is an
expression. I have come up with a prototype of such a function by diving
into Python scope internals and its AST capabilities. It was written using
Org-mode and tangling so it is thoroughly documented and explained, a test
suite is included. This interesting exercise made me appreciate Lisp even
more. Here it is
https://github.com/smartass101/python_block_eval
I haven't licensed it yet, because I'm not sure what license would be
appropriate if it was used by org-mode. Any suggestions?

My proposal is to implement an equivalent of the following bash pseudo code
for non session mode

python -i << HEREDOC_END
ret = block_eval("""
<CODE BLOCK BODY>
""")
open(<TMP FILE or PIPE>).write(str(ret))
HEREDOC_END

For session mode it would be even simpler, lines containing HEREDOC above
would be dropped and the rest piped directly into the Python REPL.

This also means that the 'org_babel_python_eoe' string indicator may not be
necessary anymore because end of evaluation would be simply shown by a new
line with the primary prompt appearing.

The reason why `python -i` (force interactive mode) is used is the IMHO
poor design choice in the CPython implementation (and other implementations
have a similar issue AFAIK) to have fast but RO access to local variable
scope in non-top-level (i.e. functions calling functions) frames/scopes. I
tried to hack my way around it in the update_locals_after_eval branch of my
repo to no avail, perhaps some Pythonista among you may know a solution.

I also favor piping input into `python -i` because it means that a
temporary file does not have to be created.

As explained in the README.org in my repo, this inconsistency can be worked
around by explicitly first evaluating the side-effect-only part of the
block and than the last expression with a direct eval() call for each. This
makes it longer by 2 lines, but has the advantage of properly handling
variable scope and separating side-effects, which could be used to e.g.
suppress output. Nevertheless, I think that for Org-mode Babel usage the
`python -i` and block_eval() approach suffices, unless someone finds a way
to use the advantages of the alternative approach to improve ob-python.el
even further.

I'm not a skilled Elisp programmer, so I wanted to ask around as to the
feasibility of this endeavor or possibly availability of helping hands
before I devote more time to this.

Kind regards,
Ondřej Grover

[-- Attachment #2: Type: text/html, Size: 3786 bytes --]

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: Proposal and RFC for improving ob-python
@ 2015-12-07 19:22 Ondřej Grover
  2015-12-07 20:27 ` Kyle Meyer
                   ` (2 more replies)
  0 siblings, 3 replies; 12+ messages in thread
From: Ondřej Grover @ 2015-12-07 19:22 UTC (permalink / raw)
  To: emacs-orgmode

[-- Attachment #1: Type: text/plain, Size: 3945 bytes --]

TL;DR for those that may have been scared off by the length of my previous
email:
I propose a method of improving ob-python.el by using a progn-like eval()
Python function which can wrap and execute source blocks.

If this list is not appropriate for discussion of improvements to
ob-python.el, could you please give me pointers how to reach people that
might be interested in helping and discussing?

Kind regards,
Ondřej Grover

On Sat, Dec 5, 2015 at 6:17 PM, Ondřej Grover <ondrej.grover@gmail.com>
wrote:

> Hello,
>
> I've been playing around with the Org-mode Babel framework and I am
> grateful to all the contributors for making this wonderful library. After
> some time I noticed that Python support seems a little hacky and
> inconsistent and after reading through ob-python.el and consulting Python
> documentation I came up with a proposal for improving it.
>
> The ob-ipython project tries to solve this hackiness in a different way by
> using the client-server infrastructure of IPython/Jupyter. That works quite
> well too, but my hope is that improving ob-python.el would also make it
> simpler to use IPython as the python REPL, relying only on the core of the
> Python language.
>
> It essentially boils down to implementing progn-like eval() function in
> Python which would return the result of the last statement if it is an
> expression. I have come up with a prototype of such a function by diving
> into Python scope internals and its AST capabilities. It was written using
> Org-mode and tangling so it is thoroughly documented and explained, a test
> suite is included. This interesting exercise made me appreciate Lisp even
> more. Here it is
> https://github.com/smartass101/python_block_eval
> I haven't licensed it yet, because I'm not sure what license would be
> appropriate if it was used by org-mode. Any suggestions?
>
> My proposal is to implement an equivalent of the following bash pseudo
> code for non session mode
>
> python -i << HEREDOC_END
> ret = block_eval("""
> <CODE BLOCK BODY>
> """)
> open(<TMP FILE or PIPE>).write(str(ret))
> HEREDOC_END
>
> For session mode it would be even simpler, lines containing HEREDOC above
> would be dropped and the rest piped directly into the Python REPL.
>
> This also means that the 'org_babel_python_eoe' string indicator may not
> be necessary anymore because end of evaluation would be simply shown by a
> new line with the primary prompt appearing.
>
> The reason why `python -i` (force interactive mode) is used is the IMHO
> poor design choice in the CPython implementation (and other implementations
> have a similar issue AFAIK) to have fast but RO access to local variable
> scope in non-top-level (i.e. functions calling functions) frames/scopes. I
> tried to hack my way around it in the update_locals_after_eval branch of my
> repo to no avail, perhaps some Pythonista among you may know a solution.
>
> I also favor piping input into `python -i` because it means that a
> temporary file does not have to be created.
>
> As explained in the README.org in my repo, this inconsistency can be
> worked around by explicitly first evaluating the side-effect-only part of
> the block and than the last expression with a direct eval() call for each.
> This makes it longer by 2 lines, but has the advantage of properly handling
> variable scope and separating side-effects, which could be used to e.g.
> suppress output. Nevertheless, I think that for Org-mode Babel usage the
> `python -i` and block_eval() approach suffices, unless someone finds a way
> to use the advantages of the alternative approach to improve ob-python.el
> even further.
>
> I'm not a skilled Elisp programmer, so I wanted to ask around as to the
> feasibility of this endeavor or possibly availability of helping hands
> before I devote more time to this.
>
> Kind regards,
> Ondřej Grover
>

[-- Attachment #2: Type: text/html, Size: 4665 bytes --]

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: Proposal and RFC for improving ob-python
  2015-12-07 19:22 Ondřej Grover
@ 2015-12-07 20:27 ` Kyle Meyer
  2015-12-08  7:43   ` Ondřej Grover
  2015-12-07 20:36 ` Thomas S. Dye
  2015-12-07 20:51 ` Achim Gratz
  2 siblings, 1 reply; 12+ messages in thread
From: Kyle Meyer @ 2015-12-07 20:27 UTC (permalink / raw)
  To: Ondřej Grover; +Cc: emacs-orgmode

Hi Ondřej,

Ondřej Grover <ondrej.grover@gmail.com> writes:

> TL;DR for those that may have been scared off by the length of my previous
> email:
> I propose a method of improving ob-python.el by using a progn-like eval()
> Python function which can wrap and execute source blocks.
>
> If this list is not appropriate for discussion of improvements to
> ob-python.el, could you please give me pointers how to reach people that
> might be interested in helping and discussing?

Thanks for your proposal.  This is the right place to discuss it.  I've
only had the chance to briefly look at this, but I'll try to take a
closer look soon.

--
Kyle

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: Proposal and RFC for improving ob-python
  2015-12-07 19:22 Ondřej Grover
  2015-12-07 20:27 ` Kyle Meyer
@ 2015-12-07 20:36 ` Thomas S. Dye
  2015-12-07 20:51 ` Achim Gratz
  2 siblings, 0 replies; 12+ messages in thread
From: Thomas S. Dye @ 2015-12-07 20:36 UTC (permalink / raw)
  To: Ondřej Grover; +Cc: emacs-orgmode

Aloha Ondřej Grover,

Ondřej Grover <ondrej.grover@gmail.com> writes:

> TL;DR for those that may have been scared off by the length of my previous
> email:
> I propose a method of improving ob-python.el by using a progn-like eval()
> Python function which can wrap and execute source blocks.
>
> If this list is not appropriate for discussion of improvements to
> ob-python.el, could you please give me pointers how to reach people that
> might be interested in helping and discussing?
>
> Kind regards,
> Ondřej Grover

This is a good list to discuss improvements to ob-python.

The authors of Babel in Org mode no longer participate in its
development and AFAICT no one has taken over this responsibility.
Instead, there have been changes to one language or another as issues
arise.

I do know that the Babel authors found ob-python difficult to write and
that problems with it have been identified from time to time on this
list.  My sense is that it could be improved, though I don't use it
often enough myself to know.

At any rate, perhaps you could propose a patch and tests?  If the patch
breaks something, then you'll certainly find users interested in
discussing ob-python :)

All the best,
Tom

-- 
Thomas S. Dye
http://www.tsdye.com

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: Proposal and RFC for improving ob-python
  2015-12-07 19:22 Ondřej Grover
  2015-12-07 20:27 ` Kyle Meyer
  2015-12-07 20:36 ` Thomas S. Dye
@ 2015-12-07 20:51 ` Achim Gratz
  2 siblings, 0 replies; 12+ messages in thread
From: Achim Gratz @ 2015-12-07 20:51 UTC (permalink / raw)
  To: emacs-orgmode

Ondřej Grover writes:
> If this list is not appropriate for discussion of improvements to
> ob-python.el, could you please give me pointers how to reach people that
> might be interested in helping and discussing?

The best way to contribute to improving ob-python is by showing a patch
that implements your ideas and still passes all existing tests.  Bonus
points if you add new tests for better coverage or any new functionality
or improvements.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptation for Waldorf microQ V2.22R2:
http://Synth.Stromeko.net/Downloads.html#WaldorfSDada

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: Proposal and RFC for improving ob-python
  2015-12-07 20:27 ` Kyle Meyer
@ 2015-12-08  7:43   ` Ondřej Grover
  0 siblings, 0 replies; 12+ messages in thread
From: Ondřej Grover @ 2015-12-08  7:43 UTC (permalink / raw)
  To: emacs-orgmode

[-- Attachment #1: Type: text/plain, Size: 1108 bytes --]

Hello,

thank you all for confirming that I'm in the right place. If I felt
confident enough that this would work, I wold have already presented a
patch (I'm not an experienced ELisp programmer). That's why I want to
discuss this approach first.

Kyle, I'll be grateful for you feedback.

Kind regards,
Ondřej Grover

On Mon, Dec 7, 2015 at 9:27 PM, Kyle Meyer <kyle@kyleam.com> wrote:

> Hi Ondřej,
>
> Ondřej Grover <ondrej.grover@gmail.com> writes:
>
> > TL;DR for those that may have been scared off by the length of my
> previous
> > email:
> > I propose a method of improving ob-python.el by using a progn-like eval()
> > Python function which can wrap and execute source blocks.
> >
> > If this list is not appropriate for discussion of improvements to
> > ob-python.el, could you please give me pointers how to reach people that
> > might be interested in helping and discussing?
>
> Thanks for your proposal.  This is the right place to discuss it.  I've
> only had the chance to briefly look at this, but I'll try to take a
> closer look soon.
>
> --
> Kyle
>

[-- Attachment #2: Type: text/html, Size: 1625 bytes --]

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: Proposal and RFC for improving ob-python
  2015-12-05 17:17 Ondřej Grover
@ 2015-12-10  7:22 ` Kyle Meyer
  2016-08-29 12:53 ` Greg Minshall
  1 sibling, 0 replies; 12+ messages in thread
From: Kyle Meyer @ 2015-12-10  7:22 UTC (permalink / raw)
  To: Ondřej Grover; +Cc: emacs-orgmode

Ondřej Grover <ondrej.grover@gmail.com> writes:

[...]

> The ob-ipython project tries to solve this hackiness in a different way by
> using the client-server infrastructure of IPython/Jupyter. That works quite
> well too, but my hope is that improving ob-python.el would also make it
> simpler to use IPython as the python REPL, relying only on the core of the
> Python language.

As I understand it (and if I'm remembering correctly), there are at
least three issues with ob-python sessions:

1. Shell prompt markers leak into the output.

2. Whitespace at the beginning of the output is swallowed.

3. Valid code that has indentation and spaces cannot be sent.

I think the first one is the race conditions that ob-ipython.el's README
refers to.

> It essentially boils down to implementing progn-like eval() function in
> Python which would return the result of the last statement if it is an
> expression.
[...]
> My proposal is to implement an equivalent of the following bash pseudo code
> for non session mode
>
> python -i << HEREDOC_END
> ret = block_eval("""
> <CODE BLOCK BODY>
> """)
> open(<TMP FILE or PIPE>).write(str(ret))
> HEREDOC_END

So for non-session code and value results, the main idea is to get rid
of the need for "return" at the end?

This seems useful to me.  Requiring a trailing "return", which is a
syntax error when the code is taken by itself, is a quirk of the
implementation (the code ends up being wrapped in a function to get the
return value).  Also, it would allow switching from ":results value" to
":results output" without needing to remove "return" in order to avoid a
syntax error.

> For session mode it would be even simpler, lines containing HEREDOC above
> would be dropped and the rest piped directly into the Python REPL.
>
> This also means that the 'org_babel_python_eoe' string indicator may not be
> necessary anymore because end of evaluation would be simply shown by a new
> line with the primary prompt appearing.

But what about when :results != value?  Doesn't your proposal only
handle returning the last value?

--
Kyle

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: Proposal and RFC for improving ob-python
@ 2015-12-10  7:47 Ondřej Grover
  2015-12-11  3:39 ` Kyle Meyer
  0 siblings, 1 reply; 12+ messages in thread
From: Ondřej Grover @ 2015-12-10  7:47 UTC (permalink / raw)
  To: emacs-orgmode

[-- Attachment #1: Type: text/plain, Size: 1682 bytes --]

Thank you for the feedback Kyle.

> As I understand it (and if I'm remembering correctly), there are at
> least three issues with ob-python sessions:

4. session and non-session code differs significantly as you have already
mentioned

> So for non-session code and value results, the main idea is to get rid
> of the need for "return" at the end?
>
Yes, that's the greatest offense to the Python code.

>
> This seems useful to me.  Requiring a trailing "return", which is a
> syntax error when the code is taken by itself, is a quirk of the
> implementation (the code ends up being wrapped in a function to get the
> return value).  Also, it would allow switching from ":results value" to
> ":results output" without needing to remove "return" in order to avoid a
> syntax error.
>
Yes, these would be the obvious benefits.

> But what about when :results != value?  Doesn't your proposal only
> handle returning the last value?
>
You mean :results output ? In that case it could just omit the
"open(...).write(...) " part and capture anything the console prints before
the primary prompt appears again.
Or it could capture stdout into a file, but I think that's needlessly
complicated:
python -i << HEREDOC_END
import sys
sys.stdout = open(<TMP FILE or PIPE>)   # replace stdout with some file
_ = block_eval("""
<CODE BLOCK BODY>
""")
sys.stdout.close()
sys.stdout = sys.__stdout__  # restore stdout
HEREDOC_END

After thinking about this, perhaps it is better to capture the block result
into "_" instead of "ret", because
a) it prevents name clashes
b) it is more in line with what "_" is used for

Kind regards,
Ondřej Grover

[-- Attachment #2: Type: text/html, Size: 2995 bytes --]

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: Proposal and RFC for improving ob-python
  2015-12-10  7:47 Proposal and RFC for improving ob-python Ondřej Grover
@ 2015-12-11  3:39 ` Kyle Meyer
  0 siblings, 0 replies; 12+ messages in thread
From: Kyle Meyer @ 2015-12-11  3:39 UTC (permalink / raw)
  To: Ondřej Grover; +Cc: emacs-orgmode

[-- Attachment #1: Type: text/plain, Size: 3629 bytes --]

Ondřej Grover <ondrej.grover@gmail.com> writes:

[...]

>> But what about when :results != value?  Doesn't your proposal only
>> handle returning the last value?
>>
> You mean :results output ? In that case it could just omit the
> "open(...).write(...) " part and capture anything the console prints before
> the primary prompt appears again.
> Or it could capture stdout into a file, but I think that's needlessly
> complicated:
>
> python -i << HEREDOC_END
> import sys
> sys.stdout = open(<TMP FILE or PIPE>)   # replace stdout with some file
> _ = block_eval("""
> <CODE BLOCK BODY>
> """)
> sys.stdout.close()
> sys.stdout = sys.__stdout__  # restore stdout
> HEREDOC_END

(I think you should rely on Org babel's shared utilities,
e.g. org-babel-eval and org-babel-eval-read-file, unless there is a
specific reason they won't do.  At any rate, right now I'm just trying
to get a picture of how block_eval would fit in.)

So these are the main options to consider for Python:

    |             | output | value |
    |-------------+--------+-------|
    | non-session | a      | b     |
    | session     | c      | d     |

a) Using block_eval here seems unhelpful here because the method in
   place is already simple:

   (org-babel-eval org-babel-python-command body)

b) block_eval replaces the need for dumping the code into a function.
   Using block_eval in org-babel-python-wrapper-method would allow you
   to get rid of the odd "return" use.

c) Currently, this is handled by inserting the body lines one at a time
   and then getting the output by splitting on the prompt.  Aside from
   race condition issues, this has limitations with handling blank lines
   in the body.

   block_eval could help here because the whole body could be sent
   through as one line of input to the prompt.  This should resolve most
   issues with prompt splitting, as well as lift the body formatting
   restrictions for sessions.

   However, the main issue I see with this is that I think it would lose
   the main distinction between session and non-session for ":results
   output" that is discussed in (info "(org) Results of evaluation"):

       #+BEGIN_SRC python :results output
        print "hello"
        2
        print "bye"
       #+END_SRC

       #+RESULTS:
       : hello
       : bye

   versus this

       #+BEGIN_SRC python :results output :session
        print "hello"
        2
        print "bye"
       #+END_SRC

       #+RESULTS:
       : hello
       : 2
       : bye

d) The variable _ is assigned the last value by the shell, so the
   current mechanism should work even if block_eval is used as described
   in c.

I've attached a quick-and-dirty patch, just for discussion.  At least
under light testing, it resolves the issues I listed in my previous
response [1].  It does have the change in behavior discussed in c, which
I see as problematic.

It also raises the issue of how to incorporate the block_eval code.  I
think there are two main options.

* Require the user have the module installed and import it.

* Make the module into a snippet to go into the ob-python.el.

The first option is what the current patch does.  The problem is that it
requires users to install an external module.  The second option avoids
that but is uglier and harder to maintain.


[1] There's a remaining issue, which I believe is specific to newer
    versions of python.el, where the first time a session block is
    executed, it doesn't wait for the shell to load up before sending
    input.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-WIP-lisp-ob-python.el-Use-block_eval.patch --]
[-- Type: text/x-diff, Size: 6676 bytes --]

From 642370236f6ca06e70ea32866eedcdba161fd2c4 Mon Sep 17 00:00:00 2001
From: Kyle Meyer <kyle@kyleam.com>
Date: Thu, 10 Dec 2015 22:09:05 -0500
Subject: [PATCH] WIP lisp/ob-python.el: Use block_eval

---
 lisp/ob-python.el | 108 +++++++++++++++++++++++++++---------------------------
 1 file changed, 54 insertions(+), 54 deletions(-)

diff --git a/lisp/ob-python.el b/lisp/ob-python.el
index 95a06f5..baaaf82 100644
--- a/lisp/ob-python.el
+++ b/lisp/ob-python.el
@@ -224,19 +224,18 @@ (defun org-babel-python-initiate-session (&optional session _params)
 
 (defvar org-babel-python-eoe-indicator "'org_babel_python_eoe'"
   "A string to indicate that evaluation has completed.")
-(defconst org-babel-python-wrapper-method
-  "
-def main():
-%s
 
-open('%s', 'w').write( str(main()) )")
+(defconst org-babel-python-block-eval-import
+  "from block_eval import block_eval as _ob_python_eval")
+
+(defconst org-babel-python-wrapper-method
+  (concat org-babel-python-block-eval-import "
+open('%s', 'w').write(str(_ob_python_eval('%s')))"))
 (defconst org-babel-python-pp-wrapper-method
-  "
+  (concat org-babel-python-block-eval-import "
 import pprint
-def main():
-%s
-
-open('%s', 'w').write( pprint.pformat(main()) )")
+from block_eval import block_eval as _ob_python_eval
+open('%s', 'w').write(pprint.pformat(str(_ob_python_eval('%s'))))"))
 
 (defun org-babel-python-evaluate
   (session body &optional result-type result-params preamble)
@@ -247,6 +246,9 @@ (defun org-babel-python-evaluate
     (org-babel-python-evaluate-external-process
      body result-type result-params preamble)))
 
+(defun org-babel-python-escape-single-quotes (s)
+  (replace-regexp-in-string "'" "\\\\'" s))
+
 (defun org-babel-python-evaluate-external-process
   (body &optional result-type result-params preamble)
   "Evaluate BODY in external python process.
@@ -262,22 +264,22 @@ (defun org-babel-python-evaluate-external-process
                     (org-babel-eval
                      org-babel-python-command
                      (concat
-                      (if preamble (concat preamble "\n") "")
+                      (if preamble (concat preamble "\\n") "")
                       (format
                        (if (member "pp" result-params)
                            org-babel-python-pp-wrapper-method
                          org-babel-python-wrapper-method)
-                       (mapconcat
-                        (lambda (line) (format "\t%s" line))
-                        (split-string
-                         (org-remove-indentation
-                          (org-babel-trim body))
-                         "[\r\n]") "\n")
-                       (org-babel-process-file-name tmp-file 'noquote))))
+                       (org-babel-process-file-name tmp-file 'noquote)
+		       (org-babel-python-escape-single-quotes
+			(mapconcat #'identity
+				   (split-string
+				    (org-remove-indentation
+				     (org-babel-trim body))
+				    "[\r\n]") "\\n")))))
                     (org-babel-eval-read-file tmp-file))))))
     (org-babel-result-cond result-params
       raw
-      (org-babel-python-table-or-string (org-babel-trim raw)))))
+      (org-babel-python-table-or-string raw))))
 
 (defun org-babel-python-evaluate-session
     (session body &optional result-type result-params)
@@ -285,49 +287,47 @@ (defun org-babel-python-evaluate-session
 If RESULT-TYPE equals `output' then return standard output as a
 string.  If RESULT-TYPE equals `value' then return the value of the
 last statement in BODY, as elisp."
-  (let* ((send-wait (lambda () (comint-send-input nil t) (sleep-for 0 5)))
-	 (dump-last-value
-	  (lambda
-	    (tmp-file pp)
-	    (mapc
-	     (lambda (statement) (insert statement) (funcall send-wait))
-	     (if pp
-		 (list
-		  "import pprint"
-		  (format "open('%s', 'w').write(pprint.pformat(_))"
-			  (org-babel-process-file-name tmp-file 'noquote)))
-	       (list (format "open('%s', 'w').write(str(_))"
-			     (org-babel-process-file-name tmp-file
-                                                          'noquote)))))))
-	 (input-body (lambda (body)
-		       (mapc (lambda (line) (insert line) (funcall send-wait))
-			     (split-string body "[\r\n]"))
-		       (funcall send-wait)))
-         (results
+  (let* ((block (concat org-babel-python-block-eval-import
+			"; _ob_python_eval('%s')"))
+	 (send (lambda (s &optional block-p)
+		 (insert
+		  (if block-p
+		      (format block
+			      (org-babel-python-escape-single-quotes
+			       (mapconcat #'identity
+					  (split-string
+					   (org-remove-indentation
+					    (org-babel-trim s))
+					   "[\r\n]") "\\n")))
+		    s))
+	    (comint-send-input nil t)))
+	 (results
           (case result-type
             (output
              (mapconcat
-              #'org-babel-trim
+	      #'identity
               (butlast
                (org-babel-comint-with-output
                    (session org-babel-python-eoe-indicator t body)
-                 (funcall input-body body)
-                 (funcall send-wait) (funcall send-wait)
-                 (insert org-babel-python-eoe-indicator)
-                 (funcall send-wait))
+		 (funcall send body t)
+                 (funcall send org-babel-python-eoe-indicator))
                2) "\n"))
             (value
-             (let ((tmp-file (org-babel-temp-file "python-")))
-               (org-babel-comint-with-output
-                   (session org-babel-python-eoe-indicator nil body)
-                 (let ((comint-process-echoes nil))
-                   (funcall input-body body)
-                   (funcall dump-last-value tmp-file
-                            (member "pp" result-params))
-                   (funcall send-wait) (funcall send-wait)
-                   (insert org-babel-python-eoe-indicator)
-                   (funcall send-wait)))
-               (org-babel-eval-read-file tmp-file))))))
+	     (let* ((tmp-file (org-babel-temp-file "python-"))
+		    (dump-statement
+		     (format
+		      (if (member "pp" result-params)
+			  (concat "import pprint; "
+				  "open('%s', 'w').write(pprint.pformat(_))")
+			"open('%s', 'w').write(str(_))")
+		      (org-babel-process-file-name tmp-file 'noquote)))
+		    (comint-process-echoes nil))
+	       (org-babel-comint-with-output
+		   (session org-babel-python-eoe-indicator nil body)
+		 (funcall send body t)
+		 (funcall send dump-statement)
+		 (funcall send org-babel-python-eoe-indicator))
+	       (org-babel-eval-read-file tmp-file))))))
     (unless (string= (substring org-babel-python-eoe-indicator 1 -1) results)
       (org-babel-result-cond result-params
 	results
-- 
2.6.3


[-- Attachment #3: Type: text/plain, Size: 9 bytes --]


--
Kyle

^ permalink raw reply related	[flat|nested] 12+ messages in thread

* Re: Proposal and RFC for improving ob-python
  2015-12-05 17:17 Ondřej Grover
  2015-12-10  7:22 ` Kyle Meyer
@ 2016-08-29 12:53 ` Greg Minshall
  2016-08-30 19:17   ` John Kitchin
  1 sibling, 1 reply; 12+ messages in thread
From: Greg Minshall @ 2016-08-29 12:53 UTC (permalink / raw)
  To: =?UTF-8?Q?Ond=C5=99ej_Grover?=; +Cc: emacs-orgmode

hi.  i recently got fed up with ob-python.el behavior w.r.t. :session
handling (startup issues; blank lines [really, a python problem, imho,
but a workaround maybe could work], and value scraping), looked, and saw
this discussion from last December, but nothing since then.  has there
been any progress, either by Ondřej or anyone else?  if my frustration
level remains high enough, i might try to see if i can do anything.
cheers, Greg Minshall

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: Proposal and RFC for improving ob-python
  2016-08-29 12:53 ` Greg Minshall
@ 2016-08-30 19:17   ` John Kitchin
  2016-08-31  2:44     ` Greg Minshall
  0 siblings, 1 reply; 12+ messages in thread
From: John Kitchin @ 2016-08-30 19:17 UTC (permalink / raw)
  To: Greg Minshall; +Cc: =?UTF-8?Q?Ond=C5=99ej_Grover?=, emacs-orgmode

now I can not remember if I sent this earlier.

I like the idea behind ob-ipython, hopefully where it might go as
ob-jupyter, which could make it able to run many other languages via the
jupyter project and it's kernels.

The last time I tried it, it only supported sessions, and I had to add
non-session support. I doubt it supported async running either.

Before any major effort went in to a new ob-python, it is probably worth
a serious look at what the jupyter project could offer. 

Greg Minshall writes:

> hi.  i recently got fed up with ob-python.el behavior w.r.t. :session
> handling (startup issues; blank lines [really, a python problem, imho,
> but a workaround maybe could work], and value scraping), looked, and saw
> this discussion from last December, but nothing since then.  has there
> been any progress, either by Ondřej or anyone else?  if my frustration
> level remains high enough, i might try to see if i can do anything.
> cheers, Greg Minshall


-- 
Professor John Kitchin
Doherty Hall A207F
Department of Chemical Engineering
Carnegie Mellon University
Pittsburgh, PA 15213
412-268-7803
@johnkitchin
http://kitchingroup.cheme.cmu.edu

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: Proposal and RFC for improving ob-python
  2016-08-30 19:17   ` John Kitchin
@ 2016-08-31  2:44     ` Greg Minshall
  0 siblings, 0 replies; 12+ messages in thread
From: Greg Minshall @ 2016-08-31  2:44 UTC (permalink / raw)
  To: John Kitchin; +Cc: =?UTF-8?Q?Ond=C5=99ej_Grover?=, emacs-orgmode

hi, John,

> I like the idea behind ob-ipython, hopefully where it might go as
> ob-jupyter, which could make it able to run many other languages via
> the jupyter project and it's kernels.

thanks for the reply.  i'm not really up to speed on Jupyter, et al.
but, from a brief look, my work is probably more directed at a more
plain, vanilla python.  were an [ij]python mode available, i can imagine
being captivated it.  but, if i were to try and hack on any code right
now, it would probably be normal python mode.

cheers, Greg

^ permalink raw reply	[flat|nested] 12+ messages in thread

end of thread, other threads:[~2016-08-31  2:44 UTC | newest]

Thread overview: 12+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2015-12-10  7:47 Proposal and RFC for improving ob-python Ondřej Grover
2015-12-11  3:39 ` Kyle Meyer
  -- strict thread matches above, loose matches on Subject: below --
2015-12-07 19:22 Ondřej Grover
2015-12-07 20:27 ` Kyle Meyer
2015-12-08  7:43   ` Ondřej Grover
2015-12-07 20:36 ` Thomas S. Dye
2015-12-07 20:51 ` Achim Gratz
2015-12-05 17:17 Ondřej Grover
2015-12-10  7:22 ` Kyle Meyer
2016-08-29 12:53 ` Greg Minshall
2016-08-30 19:17   ` John Kitchin
2016-08-31  2:44     ` Greg Minshall

Code repositories for project(s) associated with this public inbox

	https://git.savannah.gnu.org/cgit/emacs/org-mode.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).