Babel Python sessions severely broken

Babel Python sessions severely broken

[Jacek Generowicz]

Hi,

Picking up a few-month-old thread ...

On Tue, 21 Jun 2011 10:26:17 -0700, Eric Schulte wrote:

> You are suggesting that code to be run "interactively" should be written
> to an external file then loaded into the interactive session.  This
> would certainly work around the syntax limitation of the current setup.
> My two concerns here are that
> 
> 1. users who use interactive babel blocks side-by-side with the session
>    may be used jumping into the session to play with code interactively
>    and debug, in such cases rather than seeing their code in the session
>    history they would only see execfile('/tmp/blahblah').

This is (almost) exactly what Emacs' python mode does, so I suspect
that many people who wish to work in this way, would be very familiar
with this state of affairs.

Actually, In modern python modes (Emacs 23, 24, maybe even 22) no clue
about any code having been injected into the session appars at all:
instead of seeing "execfile(/'tmp/blah')" they see nothing at all. I
think that IPyton's emacs-python-mode, might visually inject the whole
evaluated code into the session buffer. 

>    Note I do recall discussion on list related to user's reading
>    their session code in the inferior session buffer.

I'd say that that interacting with their code in the session buffer is
more important than reading it there (on the grounds of this being the
usual python-mode workflow), but reading it there could be good too.

> 2. similarly error messages would now point into this temporary file,
>    rather than back into the session history

Yes, this is what happens in current python modes, but M-g M-n and M-g
M-p usually manage to send you to the correct place (i.e. your genuine
source file, rather than the temporary file) anyway.

To see this in action:

1. (In Emacs 23 or 24) Create the following buffer

,----[ foo.py ]
| print 1
| print 1/0
| print 2
`----
   
2. Ensure that Python mode is enabled for the buffer (Emacs should
   have switched it on automatically, by observing the .py extension.)

3. C-c C-z (This should start an inferior Python process, and switch
   to its buffer.)

4. Go back to the foo.py buffer

5. C-c C-c (This should evaluate the whole buffer)

   The inferior Python buffer should now show something like

,----[ *Python* ]
| >>> 1
| Traceback (most recent call last):
|   File "/tmp/py79095qL", line 2, in <module>
|     print 1/0
| ZeroDivisionError: integer division or modulo by zero
| >>> 
`----

6. M-g M-n (This should jump to line 2 in the foo.py buffer)

Maybe some of the stuff that's already in python-mode can be reused ?

> Basically you would prefer more decoupling from the interpreter

I would prefer it to be useable for evaluating normally formatted
Python code. In its current state it isn't. That makes it pretty much
unuseable for literal programming, creating documentation, tutorials,
etc.

I don't understand what you mean by "more decoupling from the
interpreter". Do you mean "overcoming Python's standard REPL's
inability to receive copy-pasted standard Pyton code?". I think that
would be a *very* worthwhile thing to support, which is why pretty
much anything that pretends to support Python interactivity, goes out
of its way to make it possible (Emacs' python-mode, IPython, IDLE are
a few obvious examples that come to mind.)

> and I'm not sure for the average user if this would be a worthwhile
> exchange simply to be able to avoid syntax errors like your
> originally mentioned example (which was the first such post I've
> seen on this list).

The current situation essentially makes Python babel sessions unusable
for anything but the most trivial cases. (Or it makes non-trivial use
possible at the cost of presening code in a horribly formatted style,
one which would be a horror to read.)

> I'm disinclined to make such a change without a wider base of support
> for the request from the Babel/Python user community

Please, please, make it possible to evaluate normal Python code in
Python babel sessions.

It's really important to understand that the restrictions imposed by
Python's plain REPL really are a horrible quirk: I can't imagine
anyone preferring the broken (if standard) behaviour, over correct
behaviour.

Put another way: You won't find many people who take Python
interactivity seriously, using the standard REPL. They all use
something that gets around the horrible restrictions it
imposes. Nobody wants this.

> or at least without more complaints about the existing behavior.

Please consider this to be a *vehement* complaint about the existing
behaviour. :-)

(I'd be happy to try to contribute some code to this, but I won't have
any time to think about it for, at least, the next 3 weeks.)

Re: Babel Python sessions severely broken

[Eric Schulte]

Jacek Generowicz <jacek.generowicz@cern.ch> writes:

> Hi,
>
> Picking up a few-month-old thread ...
>
> On Tue, 21 Jun 2011 10:26:17 -0700, Eric Schulte wrote:
>
>> You are suggesting that code to be run "interactively" should be written
>> to an external file then loaded into the interactive session.  This
>> would certainly work around the syntax limitation of the current setup.
>> My two concerns here are that
>> 
>> 1. users who use interactive babel blocks side-by-side with the session
>>    may be used jumping into the session to play with code interactively
>>    and debug, in such cases rather than seeing their code in the session
>>    history they would only see execfile('/tmp/blahblah').
>
> This is (almost) exactly what Emacs' python mode does, so I suspect
> that many people who wish to work in this way, would be very familiar
> with this state of affairs.
>
> Actually, In modern python modes (Emacs 23, 24, maybe even 22) no clue
> about any code having been injected into the session appars at all:
> instead of seeing "execfile(/'tmp/blah')" they see nothing at all. I
> think that IPyton's emacs-python-mode, might visually inject the whole
> evaluated code into the session buffer. 
>
>>    Note I do recall discussion on list related to user's reading
>>    their session code in the inferior session buffer.
>
> I'd say that that interacting with their code in the session buffer is
> more important than reading it there (on the grounds of this being the
> usual python-mode workflow), but reading it there could be good too.
>
>> 2. similarly error messages would now point into this temporary file,
>>    rather than back into the session history
>
> Yes, this is what happens in current python modes, but M-g M-n and M-g
> M-p usually manage to send you to the correct place (i.e. your genuine
> source file, rather than the temporary file) anyway.
>
> To see this in action:
>
> 1. (In Emacs 23 or 24) Create the following buffer
>
> ,----[ foo.py ]
> | print 1
> | print 1/0
> | print 2
> `----
>    
> 2. Ensure that Python mode is enabled for the buffer (Emacs should
>    have switched it on automatically, by observing the .py extension.)
>
> 3. C-c C-z (This should start an inferior Python process, and switch
>    to its buffer.)
>
> 4. Go back to the foo.py buffer
>
> 5. C-c C-c (This should evaluate the whole buffer)
>
>    The inferior Python buffer should now show something like
>
> ,----[ *Python* ]
> | >>> 1
> | Traceback (most recent call last):
> |   File "/tmp/py79095qL", line 2, in <module>
> |     print 1/0
> | ZeroDivisionError: integer division or modulo by zero
> | >>> 
> `----
>
> 6. M-g M-n (This should jump to line 2 in the foo.py buffer)
>
> Maybe some of the stuff that's already in python-mode can be reused ?
>
>> Basically you would prefer more decoupling from the interpreter
>
> I would prefer it to be useable for evaluating normally formatted
> Python code. In its current state it isn't. That makes it pretty much
> unuseable for literal programming, creating documentation, tutorials,
> etc.
>
> I don't understand what you mean by "more decoupling from the
> interpreter". Do you mean "overcoming Python's standard REPL's
> inability to receive copy-pasted standard Pyton code?". I think that
> would be a *very* worthwhile thing to support, which is why pretty
> much anything that pretends to support Python interactivity, goes out
> of its way to make it possible (Emacs' python-mode, IPython, IDLE are
> a few obvious examples that come to mind.)
>
>> and I'm not sure for the average user if this would be a worthwhile
>> exchange simply to be able to avoid syntax errors like your
>> originally mentioned example (which was the first such post I've
>> seen on this list).
>
> The current situation essentially makes Python babel sessions unusable
> for anything but the most trivial cases. (Or it makes non-trivial use
> possible at the cost of presening code in a horribly formatted style,
> one which would be a horror to read.)
>
>> I'm disinclined to make such a change without a wider base of support
>> for the request from the Babel/Python user community
>
> Please, please, make it possible to evaluate normal Python code in
> Python babel sessions.
>
> It's really important to understand that the restrictions imposed by
> Python's plain REPL really are a horrible quirk: I can't imagine
> anyone preferring the broken (if standard) behaviour, over correct
> behaviour.
>
> Put another way: You won't find many people who take Python
> interactivity seriously, using the standard REPL. They all use
> something that gets around the horrible restrictions it
> imposes. Nobody wants this.
>
>> or at least without more complaints about the existing behavior.
>
> Please consider this to be a *vehement* complaint about the existing
> behaviour. :-)
>
> (I'd be happy to try to contribute some code to this, but I won't have
> any time to think about it for, at least, the next 3 weeks.)
>

Alright, I abhor python myself and played very little role in developing
the current python support and thus have no personal opinion on the
matter.  I think the sentiment in my previous email is best interpreted
as a disinclination to change working code without some /sufficient/
motivation. (whatever sufficient may mean)

I'm happy to accept and apply patches which improve over the current
behavior.  My only thoughts would be that

1. wherever it is possible to offload work to an existing python
   evaluation engine (e.g., just call a function from ipython) that will
   always be preferable to re-inventing wheels

2. I was under the impression that Emacs24 included new interactive
   python support, and if a new standard is emerging I would rather
   Python code blocks make use of the new standard than the old

Thanks for the clearly explained commentary (I feel I know more now
about the current state of affairs for interactive Python evaluation
than I did before) and I look forward to upcoming patches.

-- 
Eric Schulte
http://cs.unm.edu/~eschulte/

Re: Babel Python sessions severely broken

[Jacek Generowicz]

At Fri, 9 Mar 2012 13:31:05 -0700,
Eric Schulte wrote:

> I look forward to upcoming patches.

I've not dug around the implementation of babel before. Any pointers
on where to start?

Re: Babel Python sessions severely broken

[Eric Schulte]

Jacek Generowicz <jacek.generowicz@cern.ch> writes:

> At Fri, 9 Mar 2012 13:31:05 -0700,
> Eric Schulte wrote:
>
>> I look forward to upcoming patches.
>
> I've not dug around the implementation of babel before. Any pointers
> on where to start?
>

Sure,

The best place to look at is existing lisp/ob-*.el files both for Python
itself and other languages.  These provide examples of how to deal with
nearly every session execution contingency.  The lisp/ob-comint.el file
provides functions which abstract away some of the hairy details of
working with session comint buffer.

Largely I would hope that you'll have to spend more time with the python
library which will be performing the evaluation, rather than with the
related Org-mode functions.

On the more general elisp dev side of things, one hint which I find
useful when exploring the workings of new code is edebug, which allows
single-stepping through the evaluation of a function.

Thanks, please do let me know if I can help with advice or explanation,

-- 
Eric Schulte
http://cs.unm.edu/~eschulte/