From mboxrd@z Thu Jan 1 00:00:00 1970 From: =?UTF-8?Q?Ond=C5=99ej_Grover?= Subject: Re: Proposal and RFC for improving ob-python Date: Mon, 7 Dec 2015 20:22:26 +0100 Message-ID: Mime-Version: 1.0 Content-Type: multipart/alternative; boundary=001a11c27dac2ad9aa052653c89f Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:48508) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1a61N3-0005AY-Db for emacs-orgmode@gnu.org; Mon, 07 Dec 2015 14:22:50 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1a61N1-0003hq-L9 for emacs-orgmode@gnu.org; Mon, 07 Dec 2015 14:22:49 -0500 Received: from mail-lb0-x22f.google.com ([2a00:1450:4010:c04::22f]:33993) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1a61N1-0003hZ-9v for emacs-orgmode@gnu.org; Mon, 07 Dec 2015 14:22:47 -0500 Received: by lbbcs9 with SMTP id cs9so71861726lbb.1 for ; Mon, 07 Dec 2015 11:22:46 -0800 (PST) List-Id: "General discussions about Org-mode." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-orgmode-bounces+geo-emacs-orgmode=m.gmane.org@gnu.org Sender: emacs-orgmode-bounces+geo-emacs-orgmode=m.gmane.org@gnu.org To: emacs-orgmode@gnu.org --001a11c27dac2ad9aa052653c89f Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable TL;DR for those that may have been scared off by the length of my previous email: I propose a method of improving ob-python.el by using a progn-like eval() Python function which can wrap and execute source blocks. If this list is not appropriate for discussion of improvements to ob-python.el, could you please give me pointers how to reach people that might be interested in helping and discussing? Kind regards, Ond=C5=99ej Grover On Sat, Dec 5, 2015 at 6:17 PM, Ond=C5=99ej Grover wrote: > Hello, > > I've been playing around with the Org-mode Babel framework and I am > grateful to all the contributors for making this wonderful library. After > some time I noticed that Python support seems a little hacky and > inconsistent and after reading through ob-python.el and consulting Python > documentation I came up with a proposal for improving it. > > The ob-ipython project tries to solve this hackiness in a different way b= y > using the client-server infrastructure of IPython/Jupyter. That works qui= te > well too, but my hope is that improving ob-python.el would also make it > simpler to use IPython as the python REPL, relying only on the core of th= e > Python language. > > It essentially boils down to implementing progn-like eval() function in > Python which would return the result of the last statement if it is an > expression. I have come up with a prototype of such a function by diving > into Python scope internals and its AST capabilities. It was written usin= g > Org-mode and tangling so it is thoroughly documented and explained, a tes= t > suite is included. This interesting exercise made me appreciate Lisp even > more. Here it is > https://github.com/smartass101/python_block_eval > I haven't licensed it yet, because I'm not sure what license would be > appropriate if it was used by org-mode. Any suggestions? > > My proposal is to implement an equivalent of the following bash pseudo > code for non session mode > > python -i << HEREDOC_END > ret =3D block_eval(""" > > """) > open().write(str(ret)) > HEREDOC_END > > For session mode it would be even simpler, lines containing HEREDOC above > would be dropped and the rest piped directly into the Python REPL. > > This also means that the 'org_babel_python_eoe' string indicator may not > be necessary anymore because end of evaluation would be simply shown by a > new line with the primary prompt appearing. > > The reason why `python -i` (force interactive mode) is used is the IMHO > poor design choice in the CPython implementation (and other implementatio= ns > have a similar issue AFAIK) to have fast but RO access to local variable > scope in non-top-level (i.e. functions calling functions) frames/scopes. = I > tried to hack my way around it in the update_locals_after_eval branch of = my > repo to no avail, perhaps some Pythonista among you may know a solution. > > I also favor piping input into `python -i` because it means that a > temporary file does not have to be created. > > As explained in the README.org in my repo, this inconsistency can be > worked around by explicitly first evaluating the side-effect-only part of > the block and than the last expression with a direct eval() call for each= . > This makes it longer by 2 lines, but has the advantage of properly handli= ng > variable scope and separating side-effects, which could be used to e.g. > suppress output. Nevertheless, I think that for Org-mode Babel usage the > `python -i` and block_eval() approach suffices, unless someone finds a wa= y > to use the advantages of the alternative approach to improve ob-python.el > even further. > > I'm not a skilled Elisp programmer, so I wanted to ask around as to the > feasibility of this endeavor or possibly availability of helping hands > before I devote more time to this. > > Kind regards, > Ond=C5=99ej Grover > --001a11c27dac2ad9aa052653c89f Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
TL;DR for those that may have been scared off by the lengt= h of my previous email:
I propose a method of improving ob-python.el by= using a progn-like eval() Python function which can wrap and execute sourc= e blocks.

If this list is not appropriate for disc= ussion of improvements to ob-python.el, could you please give me pointers h= ow to reach people that might be interested in helping and discussing?

Kind regards,
Ond=C5=99ej Grover

On Sat, Dec 5, 2015 at = 6:17 PM, Ond=C5=99ej Grover <ondrej.grover@gmail.com> = wrote:
Hello,

I've been playing around with the Org-mode Babel framework and= I am grateful to all the contributors for making this wonderful library. A= fter some time I noticed that Python support seems a little hacky and incon= sistent and after reading through ob-python.el and consulting Python docume= ntation I came up with a proposal for improving it.

The ob-ipython project tries to solve this hackiness in a different way b= y using the client-server infrastructure of IPython/Jupyter. That works qui= te well too, but my hope is that improving ob-python.el would also make it = simpler to use IPython as the python REPL, relying only on the core of the = Python language.

It essentially boils down to = implementing progn-like eval() function in Python which would return the re= sult of the last statement if it is an expression. I have come up with a pr= ototype of such a function by diving into Python scope internals and its AS= T capabilities. It was written using Org-mode and tangling so it is thoroug= hly documented and explained, a test suite is included. This interesting ex= ercise made me appreciate Lisp even more. Here it is
I haven't li= censed it yet, because I'm not sure what license would be appropriate i= f it was used by org-mode. Any suggestions?

My pro= posal is to implement an equivalent of the following bash pseudo code for n= on session mode

python -i << HEREDOC_EN= D
ret =3D block_eval("""
<CODE BLOCK = BODY>
""")
open(<TMP FILE or PIPE&= gt;).write(str(ret))
HEREDOC_END

F= or session mode it would be even simpler, lines containing HEREDOC above wo= uld be dropped and the rest piped directly into the Python REPL.=C2=A0

This also means that the=C2=A0'org_babel_python_eo= e' string indicator may not be necessary anymore because end of evaluat= ion would be simply shown by a new line with the primary prompt appearing.<= /div>

The reason why `python -i` (force interactive mode= ) is used is the IMHO poor design choice in the CPython implementation (and= other implementations have a similar issue AFAIK) to have fast but RO acce= ss to local variable scope in non-top-level (i.e. functions calling functio= ns) frames/scopes. I tried to hack my way around it in the=C2=A0update_loca= ls_after_eval branch of my repo to no avail, perhaps some Pythonista among = you may know a solution.

I also favor piping input= into `python -i` because it means that a temporary file does not have to b= e created.

As explained in the README.org in my re= po, this inconsistency can be worked around by explicitly first evaluating = the side-effect-only part of the block and than the last expression with a = direct eval() call for each. This makes it longer by 2 lines, but has the a= dvantage of properly handling variable scope and separating side-effects, w= hich could be used to e.g. suppress output. Nevertheless, I think that for = Org-mode Babel usage the `python -i` and block_eval() approach suffices, un= less someone finds a way to use the advantages of the alternative approach = to improve ob-python.el even further.

I'm not = a skilled Elisp programmer, so I wanted to ask around as to the feasibility= of this endeavor or possibly availability of helping hands before I devote= more time to this.

Kind regards,
Ond=C5= =99ej Grover

--001a11c27dac2ad9aa052653c89f--