org-babel -- Improper syntax error in session mode?

org-babel -- Improper syntax error in session mode?

[Herbert Sitz]

I have a code block that evaluates fine in non-session mode but which gives
syntax error in session mode.  Since it works fine in non-session mode I assume
this is a bug?:

----------------------------
#+begin_src python :results output :session mypy 
x = 1
for i in range(1,5):
  x = x + i
  print x
print "Did it work?"
#+end_src

#+results:
: File "<stdin>", line 4
:     print "Did it work?"
:         ^
: SyntaxError: invalid syntax
--------------------------------

Re: org-babel -- Improper syntax error in session mode?

[Eric Schulte]

Hi Herbert,

I can confirm that I see the same behavior.  Also, if I manually type
the body of the code block into the session I get the same error output
from Python, so I don't believe this is due to a problem with Babel.

Thanks -- Eric

Herbert Sitz <hsitz@nwlink.com> writes:

> I have a code block that evaluates fine in non-session mode but which gives
> syntax error in session mode.  Since it works fine in non-session mode I assume
> this is a bug?:
>
> ----------------------------
> #+begin_src python :results output :session mypy 
> x = 1
> for i in range(1,5):
>   x = x + i
>   print x
> print "Did it work?"
> #+end_src
>
> #+results:
> : File "<stdin>", line 4
> :     print "Did it work?"
> :         ^
> : SyntaxError: invalid syntax
> --------------------------------
>
>
>
>

-- 
Eric Schulte
http://cs.unm.edu/~eschulte/

Re: org-babel -- Improper syntax error in session mode?

[Herbert Sitz]

Eric Schulte <schulte.eric <at> gmail.com> writes:
> 
> Hi Herbert,
> 
> I can confirm that I see the same behavior.  Also, if I manually type
> the body of the code block into the session I get the same error output
> from Python, so I don't believe this is due to a problem with Babel.
> 

Eric -- Thanks.  Surely this is a bug that should get reported _somewhere_.  If
not Org or Babel, where?

-- Herb

Re: org-babel -- Improper syntax error in session mode?

[Herbert Sitz]

Eric Schulte <schulte.eric <at> gmail.com> writes:
> I can confirm that I see the same behavior.  Also, if I manually type
> the body of the code block into the session I get the same error output
> from Python, so I don't believe this is due to a problem with Babel.
> 

It appears the problem is that the python session is interactive and is built to
emit output after each Python "block" (e.g., the 'for' block), before another
"block" of Python is entered.  If this is the way it's designed then it seems to
me that it's Babel's obligation to feed the Python blocks to the Python session
as required and then assemble the output pieces as appropriate.  Or am I missing
something? -- Herb

Re: org-babel -- Improper syntax error in session mode?

[Nick Dokos]

Herbert Sitz <hsitz@nwlink.com> wrote:

> Eric Schulte <schulte.eric <at> gmail.com> writes:
> > I can confirm that I see the same behavior.  Also, if I manually type
> > the body of the code block into the session I get the same error output
> > from Python, so I don't believe this is due to a problem with Babel.
> > 
> 
> It appears the problem is that the python session is interactive and is built to
> emit output after each Python "block" (e.g., the 'for' block), before another
> "block" of Python is entered.  If this is the way it's designed then it seems to
> me that it's Babel's obligation to feed the Python blocks to the Python session
> as required and then assemble the output pieces as appropriate.  Or am I missing
> something? -- Herb
> 

Having babel recognize python blocks in order to feed them to the python
interpreter as complete blocks seems a bit too much to me. Of course,
what I think matters little: it's what Eric thinks that matters here.

Having said that, however, I think there *is* a problem:

If you just start the python interpreter and start typing into it:

--8<---------------cut here---------------start------------->8---
x = 1
for i in range(1,5):
  x = x + i
  print x
print "Did it work?"
--8<---------------cut here---------------end--------------->8---

the problem becomes obvious: the interpreter is still in "indented mode"
and complains about the last print, because it is not "properly"
indented.  OTOH, if you exit "indented mode" by pressing another RET
before the final print, the interpreter is happy. This is a kludge used
by the interactive interpreter to accommodate python's reliance on
indentation to delimit block structure.

That however does not work with babel: even if I leave empty lines
between the print x and the last print, the error persists. Apparently,
babel does not send the empty lines to the interpreter. If there is a
bug in babel, it seems to me this is it.

Nick

Re: org-babel -- Improper syntax error in session mode?

[Herbert Sitz]

Nick Dokos <nicholas.dokos <at> hp.com> writes:
> Having said that, however, I think there *is* a problem:
> 
> If you just start the python interpreter and start typing into it:
> 
> --8<---------------cut here---------------start------------->8---
> x = 1
> for i in range(1,5):
>   x = x + i
>   print x
> print "Did it work?"
> --8<---------------cut here---------------end--------------->8---
> 
> the problem becomes obvious: the interpreter is still in "indented mode"
> and complains about the last print, because it is not "properly"
> indented.  OTOH, if you exit "indented mode" by pressing another RET
> before the final print, the interpreter is happy. This is a kludge used
> by the interactive interpreter to accommodate python's reliance on
> indentation to delimit block structure.
> 
> That however does not work with babel: even if I leave empty lines
> between the print x and the last print, the error persists. Apparently,
> babel does not send the empty lines to the interpreter. If there is a
> bug in babel, it seems to me this is it.
> 
> Nick
> 

Yep, that's what I was thinking.  Experiment a little I found that wrapping it
in a function gets around part of the problem.  This works fine in the terminal,
although it doesn't work from the equivalent Babel source block:
--8<---------------cut here---------------start------------->8---
>>> def myfunc():
...    x=1
...    for i in range(1,5):
...       x=x+i
...       print x
...    print "Did it work?"
... 
>>> myfunc()
2
4
7
11
Did it work?
>>> 
--8<---------------cut here---------------start------------->8---

I don't know whether having Babel feed things in pieces and assemble output is
complicated for the general case.  It seems to me it wouldn't be that bad, it's
only the lines where an indent reverts to zero indent where pieces would need to
be broken up.  As a partial solution, having Babel insert the requisite blank
line after the last line of a function def would be helpful, and I think the
output would then all end up in a single block, no need to reassemble (or maybe
output reassembly is just simplified a little).

-- Herb
 

Re: org-babel -- Improper syntax error in session mode?

[Eric Schulte]

[-- Attachment #1: Type: text/plain, Size: 542 bytes --]

I've changed the python session evaluation so that it explicitly sends a
RET to the inferior Python process after every line of input.  The
attached patch makes this change.  I can confirm that this fixes the
problem in your example (when an empty line is placed between the block
and the subsequent print statement), however, could you please test it
on a variety of other python blocks to confirm it doesn't seem to break
existing behavior (I'm not a python user).

If this looks safe then I'll apply it to the main repo.

Thanks! -- Eric


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-ob-python-Send-RET-after-every-line-w-session-evalua.patch --]
[-- Type: text/x-diff, Size: 1202 bytes --]

From 1edad6a87bb5491061efaf597fa38b9190387232 Mon Sep 17 00:00:00 2001
From: Eric Schulte <schulte.eric@gmail.com>
Date: Mon, 20 Jun 2011 12:18:09 -0700
Subject: [PATCH] ob-python: Send RET after every line w/session evaluation

* lisp/ob-python.el (org-babel-python-evaluate-session): Send
  comint-send-input after every line when interacting with an
  interactive python process.
---
 lisp/ob-python.el |    8 +++++---
 1 files changed, 5 insertions(+), 3 deletions(-)

diff --git a/lisp/ob-python.el b/lisp/ob-python.el
index f3f4a03..6cb2c44 100644
--- a/lisp/ob-python.el
+++ b/lisp/ob-python.el
@@ -267,9 +267,11 @@ last statement in BODY, as elisp."
 	  (org-babel-comint-with-output
 	      (session org-babel-python-eoe-indicator t body)
 	    (let ((comint-process-echoes nil))
-	      (input-body body)
-	      (insert org-babel-python-eoe-indicator)
-	      (comint-send-input))) 2) "\n"))
+	      (mapc
+	       (lambda (line)
+		 (insert line) (comint-send-input nil t))
+	       (append (split-string body "[\n\r]") (list org-babel-python-eoe-indicator)))))
+	  2) "\n"))
        (value
 	(let ((tmp-file (org-babel-temp-file "python-")))
 	  (org-babel-comint-with-output
-- 
1.7.4.1


[-- Attachment #3: Type: text/plain, Size: 1975 bytes --]


Nick Dokos <nicholas.dokos@hp.com> writes:

> Herbert Sitz <hsitz@nwlink.com> wrote:
>
>> Eric Schulte <schulte.eric <at> gmail.com> writes:
>> > I can confirm that I see the same behavior.  Also, if I manually type
>> > the body of the code block into the session I get the same error output
>> > from Python, so I don't believe this is due to a problem with Babel.
>> > 
>> 
>> It appears the problem is that the python session is interactive and is built to
>> emit output after each Python "block" (e.g., the 'for' block), before another
>> "block" of Python is entered.  If this is the way it's designed then it seems to
>> me that it's Babel's obligation to feed the Python blocks to the Python session
>> as required and then assemble the output pieces as appropriate.  Or am I missing
>> something? -- Herb
>> 
>
> Having babel recognize python blocks in order to feed them to the python
> interpreter as complete blocks seems a bit too much to me. Of course,
> what I think matters little: it's what Eric thinks that matters here.
>
> Having said that, however, I think there *is* a problem:
>
> If you just start the python interpreter and start typing into it:
>
> x = 1
> for i in range(1,5):
>   x = x + i
>   print x
> print "Did it work?"
>
> the problem becomes obvious: the interpreter is still in "indented mode"
> and complains about the last print, because it is not "properly"
> indented.  OTOH, if you exit "indented mode" by pressing another RET
> before the final print, the interpreter is happy. This is a kludge used
> by the interactive interpreter to accommodate python's reliance on
> indentation to delimit block structure.
>
> That however does not work with babel: even if I leave empty lines
> between the print x and the last print, the error persists. Apparently,
> babel does not send the empty lines to the interpreter. If there is a
> bug in babel, it seems to me this is it.
>
> Nick
>
>

-- 
Eric Schulte
http://cs.unm.edu/~eschulte/

Re: org-babel -- Improper syntax error in session mode?

[Herbert Sitz]

Eric Schulte <schulte.eric <at> gmail.com> writes:
> I've changed the python session evaluation so that it explicitly sends a
> RET to the inferior Python process after every line of input.  The
> attached patch makes this change.  

> I can confirm that this fixes the
> problem in your example (when an empty line is placed between the block
> and the subsequent print statement)

Eric --

I did confirm that the patch works with this block:
------------------------------------------------
#+begin_src python :results output :session mypy 
    x = 1
    for i in range(1,5):
        x = x + i
        print x

    print "Did it work?"
#+end_src
------------------------------------------------

But it doesn't work with the block below, which _is_ valid Python code.  I'm not
sure whether you're aware that it isn't working with the code below (and were
thinking that the code below is _not_ valid python), or whether you were going
to require the user to input blank lines at appropriate spots:
----------------------------------------------
#+begin_src python :results output :session mypy 
    x = 1
    for i in range(1,5):
        x = x + i
        print x
    print "Did it work?"
#+end_src
-------------------------------------------

The above block is valid Python, it's just because of the quirk of the
interactive python shell that it has to have a blank line inserted before the
[print "Did it work?"] line.  

The locations where the blank lines would need to be inserted in code are
wherever the line indent goes from >0 back to 0.  A user could be required to
insert these, of course, but since it's valid python to leave them out it seems
like something org-babel should do behind the scenes.  

The '0 indent' for purposes of python execution is the smallest indent in the
org-babel source.  So even though all source code lines are indented in above
org block, the line 'x=1' has zero indent for python eval purposes.  The
relevance of this is just to show that extra lines for shell/babel evaluation
are necessary only when code indent goes back to zero, not when indent merely
becomes smaller.  So the code below works fine, even though indent shrinks after
'print y' and 'print z' statements::
------------------------------
#+begin_src python :results output :session mypy 
    x = 1
    y = 1
    z = 1
    for i in range(1,5):
        x = x + i
        print x
        for y in range(10,15):
             print y
        for z in range(20,25):
             print z

    print "Did it work?"
#+end_src
------------------------------------

Like the simpler example, though, it doesn't work with with the blank line
omitted, which it should.

I hope I'm not confusing things.  The patch does help, but doesn't address the
extra-line insertion issue.

-- Herb

Re: org-babel -- Improper syntax error in session mode?

[Eric Schulte]

Herbert Sitz <hsitz@nwlink.com> writes:

> Eric Schulte <schulte.eric <at> gmail.com> writes:
>> I've changed the python session evaluation so that it explicitly sends a
>> RET to the inferior Python process after every line of input.  The
>> attached patch makes this change.  
>
>> I can confirm that this fixes the
>> problem in your example (when an empty line is placed between the block
>> and the subsequent print statement)
>
> Eric --
>
> I did confirm that the patch works with this block:
> ------------------------------------------------
> #+begin_src python :results output :session mypy 
>     x = 1
>     for i in range(1,5):
>         x = x + i
>         print x
>
>     print "Did it work?"
> #+end_src
> ------------------------------------------------
>
> But it doesn't work with the block below, which _is_ valid Python code.

As far as I can tell the problem with the block below (missing the
space) is due to problems with the Python interpreter.

[...]

> ----------------------------------------------
> #+begin_src python :results output :session mypy 
>     x = 1
>     for i in range(1,5):
>         x = x + i
>         print x
>     print "Did it work?"
> #+end_src
> -------------------------------------------

[...]

>
> I hope I'm not confusing things.

No worries, however, I maintain that it is beyond the scope of Babel's
Python interaction to address examples such as the one given above which
do not work when typed into the Python verbatim session by the user.

> The patch does help, but doesn't address the extra-line insertion
> issue.
>

Fair enough.

Does it work for other "normal" Python interactive code blocks?  Have
you noticed any places where the previous version worked but the new
version doesn't?  If it seems safe then I would like to apply it.

Best -- Eric

>
> -- Herb
>
>
>
>
>

-- 
Eric Schulte
http://cs.unm.edu/~eschulte/

Re: org-babel -- Improper syntax error in session mode?

[Herbert Sitz]

On Mon, Jun 20, 2011 at 2:15 PM, Eric Schulte <schulte.eric@gmail.com> wrote:
>
> As far as I can tell the problem with the block below (missing the
> space) is due to problems with the Python interpreter.

It's not due to any problem with the interpreter itself, it's due to a
purposeful design decision about the way the interactive shell should work.  It
makes good sense to require that extra line in interactive mode, given Python's
reliance on carriage-returns and indents as part of its syntax, along with
decision that the interactive output should be provided at the end of every
"level 1" block.  The blank line is the only way to end a "level 1 block without
starting a second level 1 block.

>>
>> I hope I'm not confusing things.
>
> No worries, however, I maintain that it is beyond the scope of Babel's
> Python interaction to address examples such as the one given above which
> do not work when typed into the Python verbatim session by the user.
>
>> The patch does help, but doesn't address the extra-line insertion
>> issue.
>
> Fair enough.
>

Maybe adding the extra lines is beyond Babel's scope, strictly speaking.  I can
think of some good reasons for doing it though:

(1a)  Babel is not an interactive shell.

(1b)  It's not obvious to the Babel user that sessions are being processed using
Python's interactive shell.  Even if that is known (I know it's somewhere in the
docs), it's not clear to a user why Babel would require a user to insert the
extra lines (and, it turns out, _avoid_ blank lines in other cases), which make
sense in interactive environment but not within Babel's non-interactive
environment.  Even if this particular idiosyncrasy is documented somewhere it's
going to cause confusion for users who skim the docs and just expect regular
Python code to work without problems.  (If I'm first to report the issue maybe
there aren't many Org users trying to use :session mode with Python, though.)

(2)  Isn't the blank-line issue an easy fix?  I think it requires just these two
simple changes to source block before submitting to python shell:  (a)  Regex
search replace to add a blank line before any "unindented" line that is preceded
by an indented line (actually it may work fine to just put blank line before
_any_ "unindented" textline in the source-block); and (b) deletion of all blank
lines in the source-block that are followed by "indented" text on next line.  


> Does it work for other "normal" Python interactive code blocks?  Have
> you noticed any places where the previous version worked but the new
> version doesn't?  If it seems safe then I would like to apply it.

I think the <Results> start with a stray blank line before what should be the
actual output.  Otherwise It does seem to be working well with the few more
complicated things I've created to throw at it.  

I'm not really a big Python user, was just doing some testing in my
vim-Org-clone with Python when I noticed the problem.  If you end up not
addressing the line-insertion issue I may put it on my todo-list for my first
real adventure in learning elisp.

Regards,

Herb

Re: org-babel -- Improper syntax error in session mode?

[Nick Dokos]

Herbert Sitz <hsitz@nwlink.com> wrote:

> ... (and, it turns out, _avoid_ blank lines in other cases)

What are those cases?

Nick

Re: org-babel -- Improper syntax error in session mode?

[Herbert Sitz]

On Mon, Jun 20, 2011 at 5:08 PM, Nick Dokos <nicholas.dokos@hp.com> wrote:
> Herbert Sitz <hsitz@nwlink.com> wrote:
>
>> ... (and, it turns out, _avoid_ blank lines in other cases)
>
> What are those cases?
>
> Nick
>

_Every_ spot where a blank line occurs that is followed by
"unindented" line of (uncommented) code.  By design, the blank line in
interactive shell terminates a "highest-level" block and triggers its
output.  So you _can't_ have blank lines within a level-1 block, all
lines in the block must have text.  I just noticed, though, that you
can include comments and that the indent of comments is irrelevant, so
these blank lines could merely have a '# sign prepended to them to
transform them into comments.  (Existence of "unindented" comments
needs to be taken into account for the regex that adds blank lines,
though, since they're to be ignored for that purpose.)

So this works fine in interactive shell with Eric's patch:
--------------------------------------
#+begin_src python :results output :session mypy
    x = 1
    y = 1
    z = 1
    for i in range(1,2):
        x = x + i
        print x
    # comment here
        for y in range(10,11):
            print y
    # comment here
        for z in range(5,6):
            print z
            while y > 0:
                print y
                y=y-1

    print "Did it work?"
#+end_src
-------------------------------------

but this doesn't
-----------------------------
#+begin_src python :results output :session mypy
    x = 1
    y = 1
    z = 1
    for i in range(1,2):
        x = x + i
        print x

        for y in range(10,11):
            print y

        for z in range(5,6):
            print z
            while y > 0:
                print y
                y=y-1

    print "Did it work?"
#+end_src
------------------------------

Re: org-babel -- Improper syntax error in session mode?

[Nick Dokos]

Herbert Sitz <hesitz@gmail.com> wrote:

> -------------------------------------
> 
> but this doesn't
> -----------------------------
> #+begin_src python :results output :session mypy
>     x = 1
>     y = 1
>     z = 1
>     for i in range(1,2):
>         x = x + i
>         print x
> 
>         for y in range(10,11):
>             print y
> 
>         for z in range(5,6):
>             print z
>             while y > 0:
>                 print y
>                 y=y-1
> 
>     print "Did it work?"
> #+end_src
> ------------------------------
> 

It works fine if you make the empty lines "indented empty" lines:
add enough space so the newline is at the proper indentation point.

Nick

Re: org-babel -- Improper syntax error in session mode?

[Eric Schulte]

Perhaps I should explain my personal view of what Babel "sessions" are,
I'm not saying this is *the* view, just my own.

Babel sessions explicitly are thin wrappers around the interactive mode
of the language in question (whatever that may be).  That is why Babel
happily doesn't implement sessions for all languages, the contract
simply is that if a language supports interactive evaluation, Babel will
try to allow access to that functionality.

If the contract was rather simply "Babel provides session based
evaluation", then we'd be on the hook for implementing session
evaluation where it doesn't already exist and may not make sense (e.g.,
C, ditaa), and normalizing everywhere else as you're suggesting here.
Which would in turn requires us to spell out Babel-specific semantics of
session based evaluation -- something we have thus-far avoided.

Some positive points *for* the "thin wrapper" approach include;

1. It is the simplest to implement
   - easier to implement support for new language
   - easier to maintain -- especially if interactive tools change
   - easier to reason about -- thinking about bug fixing here
   - works with alternate back-ends e.g., ipython

2. It is the least surprising.  I'd maintain that for users familiar
   with using Python sessions this behavior is the easiest to quickly
   understand and use.  As opposed to if we did something fancy (even if
   it was an improvement) it would be another environment for language
   users to have to familiarize themselves with -- they'd have both
   normal session rules and babel session rules

I guess that's it, I'm happy to be convinced otherwise, or simply be
outvoted.  If you want to look at the code I'm happy to help in any way
I can, my previous patch would probably be a good place to start.  Just
be careful, if you get too comfortable with Emacs Lisp you might find it
hard to go back to working with VM's extension language. :)

a few more scattered comments below...

Herbert Sitz <hsitz@nwlink.com> writes:

[...]
>
> Maybe adding the extra lines is beyond Babel's scope, strictly speaking.  I can
> think of some good reasons for doing it though:
>
> (1a)  Babel is not an interactive shell.
>

Exactly, it is a window to a language's interactive run-time as defined
by the language's tools.

>
> (1b)  It's not obvious to the Babel user that sessions are being processed using
> Python's interactive shell.

I'd disagree, to my mind this is the simplest explanation of session
evaluation.

> Even if that is known (I know it's somewhere in the docs),

Perhaps the documentation should be tweaked to make this point more
clear.

> it's not clear to a user why Babel would require a user to insert the
> extra lines (and, it turns out, _avoid_ blank lines in other cases),
> which make sense in interactive environment but not within Babel's
> non-interactive environment.  Even if this particular idiosyncrasy is
> documented somewhere it's going to cause confusion for users who skim
> the docs and just expect regular Python code to work without problems.

But any *other* behavior would be surprising to users who are familiar
with using Python's interactive shell, and think of code blocks as
simply a place to store transcripts of interactive sessions they may
want to dump back into the session at some point in the future.

> 
> (If I'm first to report the issue maybe there aren't many Org users
> trying to use :session mode with Python, though.)
>

or (if they are regular Python users) they didn't find this behavior to
be surprising.

>
> (2)  Isn't the blank-line issue an easy fix?  I think it requires just these two
> simple changes to source block before submitting to python shell:  (a)  Regex
> search replace to add a blank line before any "unindented" line that is preceded
> by an indented line (actually it may work fine to just put blank line before
> _any_ "unindented" textline in the source-block); and (b) deletion of all blank
> lines in the source-block that are followed by "indented" text on next line.  
>

I agree, it shouldn't be an overly difficult fix, but I'm worried about
precedent.  This gives me a distinct "foot in the door" feeling, and it
breaks the /thin wrapper/ contract I spelled out above.

>
>
>> Does it work for other "normal" Python interactive code blocks?  Have
>> you noticed any places where the previous version worked but the new
>> version doesn't?  If it seems safe then I would like to apply it.
>
> I think the <Results> start with a stray blank line before what should be the
> actual output.  Otherwise It does seem to be working well with the few more
> complicated things I've created to throw at it.  
>

Great, thanks.  I appreciate you testing this out.  I've just applied
the patch since, regardless of the outcome of this thread I think we're
all agreed that the patch is better than the current behavior.

>
> I'm not really a big Python user, was just doing some testing in my
> vim-Org-clone with Python when I noticed the problem.  If you end up not
> addressing the line-insertion issue I may put it on my todo-list for my first
> real adventure in learning elisp.
>

Cool, I'm happy to help, either with your digging into elisp, or by
providing Babel information/opinions for the VM clone.  It's an
incredible undertaking to re-implement something as massive as Org-mode
in a new environment and I'm excited to hear you're getting up to the
more esoteric features like Babel.

Best -- Eric

>
> Regards,
>
> Herb
>
>
>
>

-- 
Eric Schulte
http://cs.unm.edu/~eschulte/

Re: org-babel -- Improper syntax error in session mode?

[Herbert Sitz]

Eric Schulte <schulte.eric <at> gmail.com> writes:

> 
> Babel sessions explicitly are thin wrappers around the interactive mode
> of the language in question (whatever that may be).  That is why Babel
> happily doesn't implement sessions for all languages, the contract
> simply is that if a language supports interactive evaluation, Babel will
> try to allow access to that functionality.
> 

That the Babel session is only a "thin" wrapper is not clear at all from the
docs.  The docs do mention 'interactive session'--and even that the output may
be slightly different from the non-session mode.  The docs don't mention
anything about the user needing to do extra preparation to avoid syntax errors
in evaluation of already valid code.  If I take an existing Org block of
non-session code and stick a :session on I don't expect to get syntax errors.
 If that's the expected behavior then it needs to be emphasized more in the
docs, otherwise most users will think something is broken in Org.

> If the contract was rather simply "Babel provides session based
> evaluation", then we'd be on the hook for implementing session
> evaluation where it doesn't already exist and may not make sense (e.g.,
> C, ditaa), and normalizing everywhere else as you're suggesting here.
> Which would in turn requires us to spell out Babel-specific semantics of
> session based evaluation -- something we have thus-far avoided.

Again, the docs reference 'Session based evaluation' in big bold letters and
don't indicate the thinness of the wrapper.

> 
> Some positive points *for* the "thin wrapper" approach include;
> 
> 1. It is the simplest to implement
>    - easier to implement support for new language
>    - easier to maintain -- especially if interactive tools change
>    - easier to reason about -- thinking about bug fixing here
>    - works with alternate back-ends e.g., ipython
> 

I think I have a solution for Python that gives full "session-based evaluation"
and which is better than the current "thin wrapper" for all four concerns above
(See bottom of my message for an example.)  This same approach, I think, would
work for Ruby, but I would need to do a little digging to find exact method. 
Let me know if you want me to do that digging.

> 2. It is the least surprising.  I'd maintain that for users familiar
>    with using Python sessions this behavior is the easiest to quickly
>    understand and use.  As opposed to if we did something fancy (even if
>    it was an improvement) it would be another environment for language
>    users to have to familiarize themselves with -- they'd have both
>    normal session rules and babel session rules

As a sometimes Python user it had me totally confused.  I don't think of
"session-based" as having any essential tie to "interactive shell" at all.  Why
would I worry about state being maintained between statements in an org block? 
State is already maintained between statements in non-session Org blocks. The
only difference with session-based blocks is the starting state may not be
"fresh".

When I look at Org and think of "session-based" blocks, I see a potential big
benefit in having multiple source-blocks throughout my document share the same
session.  That way I don't need to use Org's (clever and useful but) somewhat
clunky and inflexible method of passing data explicitly.  In statistics-based
docs, for example, I can imagine that the sheer amount of data would make using
Org's explicit variable-passing method unfeasible.  Using
session-based-evaluation is at the same time both simpler and more heavy-duty.

I never thought the use of session-based code blocks as something like a
scratchpad was much more than toy. If you're typing in code that's not intended
to be an integral part of an Org doc then you could just as well jump to the
shell and enter it there.

Here's an example of how to run a block of Python code of any length in the
interactive session and save its output in a file.  Note that this is _not_
interactive, even though the command runs in the interactive shell's session:

Start with this block of code:
-------------------------------------
#+begin_src python :results output :session mypy 
    x = 1
    y = 1
    z = 1
    for i in range(1,2):
        x = x + i
        print x
    
        for y in range(10,11):
            print y
    # comment here
        for z in range(5,6):
            print z
            while y > 0:
                print y
                y=y-1

    print "Did it work?"
#+end_src
------------------------------------

Strip out the block of empty space at the left side and save in a file in
current directory (say, 'pyfile.py') with a couple commands prepended and
appended:
-----------------------------
# beginning of file
# prepend code to redirect stdout to file
import sys
saveout = sys.stdout
fsock = open('pyfile-output.log','w')
sys.stdout = fsock

# code block here, with beginning space block removed
x = 1
y = 1
z = 1
for i in range(1,2):
    x = x + i
    print x

    for y in range(10,11):
        print y
# comment here
    for z in range(5,6):
        print z
        while y > 0:
            print y
            y=y-1
print "Did it work?"
# end of the org source code block

# return stdout to what it was
sys.stdout = saveout
# and close the log file
fsock.close()
# end of file
----------------------------

Now you can go into the python shell and execute the code in pyfile.py:

>>> execfile('pyfile.py')

The output will be saved in the log file in the working directory.  I think it
should be exactly the same as the output from the non-session based code
(assuming the session-based code started from a fresh session).

-- Herb

Re: org-babel -- Improper syntax error in session mode?

[Thomas S. Dye]

Aloha Herbert,

I think you're right about the potential to improve the documentation.
That's an on-going process.  Could you suggest some specific changes?

All the best,
Tom

Herbert Sitz <hsitz@nwlink.com> writes:

> Eric Schulte <schulte.eric <at> gmail.com> writes:
>
>> 
>> Babel sessions explicitly are thin wrappers around the interactive mode
>> of the language in question (whatever that may be).  That is why Babel
>> happily doesn't implement sessions for all languages, the contract
>> simply is that if a language supports interactive evaluation, Babel will
>> try to allow access to that functionality.
>> 
>
> That the Babel session is only a "thin" wrapper is not clear at all from the
> docs.  The docs do mention 'interactive session'--and even that the output may
> be slightly different from the non-session mode.  The docs don't mention
> anything about the user needing to do extra preparation to avoid syntax errors
> in evaluation of already valid code.  If I take an existing Org block of
> non-session code and stick a :session on I don't expect to get syntax errors.
>  If that's the expected behavior then it needs to be emphasized more in the
> docs, otherwise most users will think something is broken in Org.
>
>> If the contract was rather simply "Babel provides session based
>> evaluation", then we'd be on the hook for implementing session
>> evaluation where it doesn't already exist and may not make sense (e.g.,
>> C, ditaa), and normalizing everywhere else as you're suggesting here.
>> Which would in turn requires us to spell out Babel-specific semantics of
>> session based evaluation -- something we have thus-far avoided.
>
> Again, the docs reference 'Session based evaluation' in big bold letters and
> don't indicate the thinness of the wrapper.
>
>> 
>> Some positive points *for* the "thin wrapper" approach include;
>> 
>> 1. It is the simplest to implement
>>    - easier to implement support for new language
>>    - easier to maintain -- especially if interactive tools change
>>    - easier to reason about -- thinking about bug fixing here
>>    - works with alternate back-ends e.g., ipython
>> 
>
> I think I have a solution for Python that gives full "session-based evaluation"
> and which is better than the current "thin wrapper" for all four concerns above
> (See bottom of my message for an example.)  This same approach, I think, would
> work for Ruby, but I would need to do a little digging to find exact method. 
> Let me know if you want me to do that digging.
>
>> 2. It is the least surprising.  I'd maintain that for users familiar
>>    with using Python sessions this behavior is the easiest to quickly
>>    understand and use.  As opposed to if we did something fancy (even if
>>    it was an improvement) it would be another environment for language
>>    users to have to familiarize themselves with -- they'd have both
>>    normal session rules and babel session rules
>
> As a sometimes Python user it had me totally confused.  I don't think of
> "session-based" as having any essential tie to "interactive shell" at all.  Why
> would I worry about state being maintained between statements in an org block? 
> State is already maintained between statements in non-session Org blocks. The
> only difference with session-based blocks is the starting state may not be
> "fresh".
>
> When I look at Org and think of "session-based" blocks, I see a potential big
> benefit in having multiple source-blocks throughout my document share the same
> session.  That way I don't need to use Org's (clever and useful but) somewhat
> clunky and inflexible method of passing data explicitly.  In statistics-based
> docs, for example, I can imagine that the sheer amount of data would make using
> Org's explicit variable-passing method unfeasible.  Using
> session-based-evaluation is at the same time both simpler and more heavy-duty.
>
> I never thought the use of session-based code blocks as something like a
> scratchpad was much more than toy. If you're typing in code that's not intended
> to be an integral part of an Org doc then you could just as well jump to the
> shell and enter it there.
>
> Here's an example of how to run a block of Python code of any length in the
> interactive session and save its output in a file.  Note that this is _not_
> interactive, even though the command runs in the interactive shell's session:
>
> Start with this block of code:
> -------------------------------------
> #+begin_src python :results output :session mypy 
>     x = 1
>     y = 1
>     z = 1
>     for i in range(1,2):
>         x = x + i
>         print x
>     
>         for y in range(10,11):
>             print y
>     # comment here
>         for z in range(5,6):
>             print z
>             while y > 0:
>                 print y
>                 y=y-1
>
>     print "Did it work?"
> #+end_src
> ------------------------------------
>
> Strip out the block of empty space at the left side and save in a file in
> current directory (say, 'pyfile.py') with a couple commands prepended and
> appended:
> -----------------------------
> # beginning of file
> # prepend code to redirect stdout to file
> import sys
> saveout = sys.stdout
> fsock = open('pyfile-output.log','w')
> sys.stdout = fsock
>
> # code block here, with beginning space block removed
> x = 1
> y = 1
> z = 1
> for i in range(1,2):
>     x = x + i
>     print x
>
>     for y in range(10,11):
>         print y
> # comment here
>     for z in range(5,6):
>         print z
>         while y > 0:
>             print y
>             y=y-1
> print "Did it work?"
> # end of the org source code block
>
> # return stdout to what it was
> sys.stdout = saveout
> # and close the log file
> fsock.close()
> # end of file
> ----------------------------
>
> Now you can go into the python shell and execute the code in pyfile.py:
>
>>>> execfile('pyfile.py')
>
> The output will be saved in the log file in the working directory.  I think it
> should be exactly the same as the output from the non-session based code
> (assuming the session-based code started from a fresh session).
>
> -- Herb
>
>
>
>
>

-- 
Thomas S. Dye
http://www.tsdye.com

Re: org-babel -- Improper syntax error in session mode?

[Herbert Sitz]

Thomas S. Dye <tsd <at> tsdye.com> writes:
> 
> Aloha Herbert,
> 
> I think you're right about the potential to improve the documentation.
> That's an on-going process.  Could you suggest some specific changes?
> 
> All the best,
> Tom

Tom -- 

I would suggest just adding specific warnings that session-based evaluation may
throw syntax errors with valid code.

Here's what Org manual says regarding :session evaluation with :results output':

"The code is passed to the interpreter running as an interactive Emacs inferior
process. The result returned is the concaOrg tenation of the sequence of (text)
output from the interactive interpreter. Notice that this is not necessarily the
same as what would be sent to STDOUT if the same code were passed to a
non-interactive interpreter running as an external process. . . ."
[http://orgmode.org/manual/Results-of-evaluation.html#Results-of-evaluation]

All that needs to be added is a warning:

"IMPORTANT:  Note that Org provides only a thin wrapper around a language's
interactive shell, so valid code that executes properly in non-session mode may
fail in :session mode.  This is because Org feeds the lines in a :session block
to the interactive interpreter exactly as written.  In come cases the lines may
require special formatting in the source block to be executed properly in the
interactive shell.  For example: . . . "

I think the above issue should be moot for some of the main interpreted
languages (viz., Python, Perl, Ruby) where there are various ways to execute a
block of code non-interactively despite being in the interactive shell.  (One of
those methods is the example I gave in previous message.)  In that case
identical code will run the same whether it's in an interactive-interpreter
session or not.  It seems to me that switching to that approach for languages
that support it should both (1) simplify the Org Babel code, and (2) provide Org
users with more flexibility and power.

Also, it seems the real power of :session evaluation is to share state between
Org/Babel source code blocks, not merely to act as a kind of Org-internal
scratchpad.  As a user trying to take advantage of that state-sharing power I
would generally want my Org document exports to start from fresh sessions.  That
is, I would write my :session blocks so that they depended on results of code
run in previous :session blocks, BUT I would want the first :session block to
start with a fresh session,with a known state.   So on export I would generally
want any existing named session to be closed and restarted anew for an 
Org export.

I hope that all makes sense.  I should say I'm not a big Babel user, and I could
easily be misunderstanding something.  But it seems this is the way the
:session-based stuff should work, when possible.

-- Herb

Re: org-babel -- Improper syntax error in session mode?

[Thomas S. Dye]

Aloha Herb,

I think a note to that effect belongs here:
http://orgmode.org/manual/session.html#session

Perhaps the behavior related to session persistence could be described
there, as well.

Examples are probably best left to the language-specific documentation,
such as it is, at http://orgmode.org/worg/org-contrib/babel/languages/

Note that there is no language-specific documentation for Python yet.

Is it the case that the Ruby and Perl interpreters won't run code that
does run in non-session mode?

All the best,
Tom

Herbert Sitz <hsitz@nwlink.com> writes:

> Thomas S. Dye <tsd <at> tsdye.com> writes:
>> 
>> Aloha Herbert,
>> 
>> I think you're right about the potential to improve the documentation.
>> That's an on-going process.  Could you suggest some specific changes?
>> 
>> All the best,
>> Tom
>
> Tom -- 
>
> I would suggest just adding specific warnings that session-based evaluation may
> throw syntax errors with valid code.
>
> Here's what Org manual says regarding :session evaluation with :results output':
>
> "The code is passed to the interpreter running as an interactive Emacs inferior
> process. The result returned is the concaOrg tenation of the sequence of (text)
> output from the interactive interpreter. Notice that this is not necessarily the
> same as what would be sent to STDOUT if the same code were passed to a
> non-interactive interpreter running as an external process. . . ."
> [http://orgmode.org/manual/Results-of-evaluation.html#Results-of-evaluation]
>
> All that needs to be added is a warning:
>
> "IMPORTANT:  Note that Org provides only a thin wrapper around a language's
> interactive shell, so valid code that executes properly in non-session mode may
> fail in :session mode.  This is because Org feeds the lines in a :session block
> to the interactive interpreter exactly as written.  In come cases the lines may
> require special formatting in the source block to be executed properly in the
> interactive shell.  For example: . . . "
>
> I think the above issue should be moot for some of the main interpreted
> languages (viz., Python, Perl, Ruby) where there are various ways to execute a
> block of code non-interactively despite being in the interactive shell.  (One of
> those methods is the example I gave in previous message.)  In that case
> identical code will run the same whether it's in an interactive-interpreter
> session or not.  It seems to me that switching to that approach for languages
> that support it should both (1) simplify the Org Babel code, and (2) provide Org
> users with more flexibility and power.
>
> Also, it seems the real power of :session evaluation is to share state between
> Org/Babel source code blocks, not merely to act as a kind of Org-internal
> scratchpad.  As a user trying to take advantage of that state-sharing power I
> would generally want my Org document exports to start from fresh sessions.  That
> is, I would write my :session blocks so that they depended on results of code
> run in previous :session blocks, BUT I would want the first :session block to
> start with a fresh session,with a known state.   So on export I would generally
> want any existing named session to be closed and restarted anew for an 
> Org export.
>
> I hope that all makes sense.  I should say I'm not a big Babel user, and I could
> easily be misunderstanding something.  But it seems this is the way the
> :session-based stuff should work, when possible.
>
> -- Herb
>
>
>

-- 
Thomas S. Dye
http://www.tsdye.com

Re: org-babel -- Improper syntax error in session mode?

[Eric Schulte]

tsd@tsdye.com (Thomas S. Dye) writes:

> Aloha Herb,
>
> I think a note to that effect belongs here:
> http://orgmode.org/manual/session.html#session
>
> Perhaps the behavior related to session persistence could be described
> there, as well.
>
> Examples are probably best left to the language-specific documentation,
> such as it is, at http://orgmode.org/worg/org-contrib/babel/languages/
>
> Note that there is no language-specific documentation for Python yet.
>

Yes, these pages would benefit greatly from more documentation.

>
> Is it the case that the Ruby and Perl interpreters won't run code that
> does run in non-session mode?
>

No, Python and Haskell are the only two languages of which I am aware of
any difference between interactive and normal evaluation.

Best -- Eric

>
> All the best,
> Tom
>
> Herbert Sitz <hsitz@nwlink.com> writes:
>
>> Thomas S. Dye <tsd <at> tsdye.com> writes:
>>> 
>>> Aloha Herbert,
>>> 
>>> I think you're right about the potential to improve the documentation.
>>> That's an on-going process.  Could you suggest some specific changes?
>>> 
>>> All the best,
>>> Tom
>>
>> Tom -- 
>>
>> I would suggest just adding specific warnings that session-based evaluation may
>> throw syntax errors with valid code.
>>
>> Here's what Org manual says regarding :session evaluation with :results output':
>>
>> "The code is passed to the interpreter running as an interactive Emacs inferior
>> process. The result returned is the concaOrg tenation of the sequence of (text)
>> output from the interactive interpreter. Notice that this is not necessarily the
>> same as what would be sent to STDOUT if the same code were passed to a
>> non-interactive interpreter running as an external process. . . ."
>> [http://orgmode.org/manual/Results-of-evaluation.html#Results-of-evaluation]
>>
>> All that needs to be added is a warning:
>>
>> "IMPORTANT:  Note that Org provides only a thin wrapper around a language's
>> interactive shell, so valid code that executes properly in non-session mode may
>> fail in :session mode.  This is because Org feeds the lines in a :session block
>> to the interactive interpreter exactly as written.  In come cases the lines may
>> require special formatting in the source block to be executed properly in the
>> interactive shell.  For example: . . . "
>>
>> I think the above issue should be moot for some of the main interpreted
>> languages (viz., Python, Perl, Ruby) where there are various ways to execute a
>> block of code non-interactively despite being in the interactive shell.  (One of
>> those methods is the example I gave in previous message.)  In that case
>> identical code will run the same whether it's in an interactive-interpreter
>> session or not.  It seems to me that switching to that approach for languages
>> that support it should both (1) simplify the Org Babel code, and (2) provide Org
>> users with more flexibility and power.
>>
>> Also, it seems the real power of :session evaluation is to share state between
>> Org/Babel source code blocks, not merely to act as a kind of Org-internal
>> scratchpad.  As a user trying to take advantage of that state-sharing power I
>> would generally want my Org document exports to start from fresh sessions.  That
>> is, I would write my :session blocks so that they depended on results of code
>> run in previous :session blocks, BUT I would want the first :session block to
>> start with a fresh session,with a known state.   So on export I would generally
>> want any existing named session to be closed and restarted anew for an 
>> Org export.
>>
>> I hope that all makes sense.  I should say I'm not a big Babel user, and I could
>> easily be misunderstanding something.  But it seems this is the way the
>> :session-based stuff should work, when possible.
>>
>> -- Herb
>>
>>
>>

-- 
Eric Schulte
http://cs.unm.edu/~eschulte/

Re: org-babel -- Improper syntax error in session mode?

[Herbert Sitz]

Thomas S. Dye <tsd <at> tsdye.com> writes:
> 
> Aloha Herb,
> 
> [ . . . ]
> 
> Is it the case that the Ruby and Perl interpreters won't run code that
> does run in non-session mode?
> 
> All the best,
> Tom
> 

Tom -- 

I don't know about Ruby and Perl, have only done some basic testing in Ruby and
it works fine.  My guess would be that generic Ruby and Perl code in :session
mode is more likely to run without error than Python, because of Python's
special reliance on <cr> and indentation as syntax items, and the way that
affected design of the Python interactive shell.

-- Herb

Re: org-babel -- Improper syntax error in session mode?

[Eric Schulte]

Herbert Sitz <hsitz@nwlink.com> writes:

> Thomas S. Dye <tsd <at> tsdye.com> writes:
>> 
>> Aloha Herbert,
>> 
>> I think you're right about the potential to improve the documentation.
>> That's an on-going process.  Could you suggest some specific changes?
>> 
>> All the best,
>> Tom
>
> Tom -- 
>
> I would suggest just adding specific warnings that session-based evaluation may
> throw syntax errors with valid code.
>

Done.

I've added such a warning -- although not as strongly stated as your suggestion

>
[...]
>
> Also, it seems the real power of :session evaluation is to share state between
> Org/Babel source code blocks, not merely to act as a kind of Org-internal
> scratchpad.  As a user trying to take advantage of that state-sharing power I
> would generally want my Org document exports to start from fresh sessions.  That
> is, I would write my :session blocks so that they depended on results of code
> run in previous :session blocks, BUT I would want the first :session block to
> start with a fresh session,with a known state.   So on export I would generally
> want any existing named session to be closed and restarted anew for an 
> Org export.
>

There are times when I explicitly do not want a session to start fresh.
Generally this is related to caching, for example if I have some value
which is expensive (say a couple of minutes of computation) to compute,
and it is currently alive in the session, then I'd probably rather keep
my session active across multiple exports as I'm working on other (say
visual/prose tweaks) to the exported document.

Does that make sense?

If the user did want to refresh the session on _every_ export I believe
they could add such a function to the `org-export-preprocess-hook'.

Cheers -- Eric

>
> I hope that all makes sense.  I should say I'm not a big Babel user, and I could
> easily be misunderstanding something.  But it seems this is the way the
> :session-based stuff should work, when possible.
>
> -- Herb
>
>
>

-- 
Eric Schulte
http://cs.unm.edu/~eschulte/

Re: org-babel -- Improper syntax error in session mode?

[Herbert Sitz]

Eric Schulte <schulte.eric <at> gmail.com> writes:
> There are times when I explicitly do not want a session to start fresh.
> Generally this is related to caching, for example if I have some value
> which is expensive (say a couple of minutes of computation) to compute,
> and it is currently alive in the session, then I'd probably rather keep
> my session active across multiple exports as I'm working on other (say
> visual/prose tweaks) to the exported document.
> 
> Does that make sense?
> 
> If the user did want to refresh the session on _every_ export I believe
> they could add such a function to the `org-export-preprocess-hook'.
> 
> Cheers -- Eric
> 

Eric --

That makes perfect sense.  However to the extent reproducibility is a concern it
would make me uneasy to have a default that does nothing to enforce
reproducibility; and instead to have it controlled in a hook behind-the-scenes.

An acceptable alternative it seems to me, would be to put an elisp source block
at begin of document to reset the session, and then to use that block to control
whether session is reset.  That way it's clear to user exactly what's happening.

-- Herb

Re: org-babel -- Improper syntax error in session mode?

[Eric Schulte]

Hi Herb,

I think we're agreed that the documentation should be updated, which
I've just done.  As for your proposed solution.

You are suggesting that code to be run "interactively" should be written
to an external file then loaded into the interactive session.  This
would certainly work around the syntax limitation of the current setup.
My two concerns here are that

1. users who use interactive babel blocks side-by-side with the session
   may be used jumping into the session to play with code interactively
   and debug, in such cases rather than seeing their code in the session
   history they would only see execfile('/tmp/blahblah').  Note I do
   recall discussion on list related to user's reading their session
   code in the inferior session buffer.

2. similarly error messages would now point into this temporary file,
   rather than back into the session history

Basically you would prefer more decoupling from the interpreter and I'm
not sure for the average user if this would be a worthwhile exchange
simply to be able to avoid syntax errors like your originally mentioned
example (which was the first such post I've seen on this list).

I'm disinclined to make such a change without a wider base of support
for the request from the Babel/Python user community -- or at least
without more complaints about the existing behavior.

Best -- Eric

Herbert Sitz <hsitz@nwlink.com> writes:

> Eric Schulte <schulte.eric <at> gmail.com> writes:
>
>> 
>> Babel sessions explicitly are thin wrappers around the interactive mode
>> of the language in question (whatever that may be).  That is why Babel
>> happily doesn't implement sessions for all languages, the contract
>> simply is that if a language supports interactive evaluation, Babel will
>> try to allow access to that functionality.
>> 
>
> That the Babel session is only a "thin" wrapper is not clear at all from the
> docs.  The docs do mention 'interactive session'--and even that the output may
> be slightly different from the non-session mode.  The docs don't mention
> anything about the user needing to do extra preparation to avoid syntax errors
> in evaluation of already valid code.  If I take an existing Org block of
> non-session code and stick a :session on I don't expect to get syntax errors.
>  If that's the expected behavior then it needs to be emphasized more in the
> docs, otherwise most users will think something is broken in Org.
>
>> If the contract was rather simply "Babel provides session based
>> evaluation", then we'd be on the hook for implementing session
>> evaluation where it doesn't already exist and may not make sense (e.g.,
>> C, ditaa), and normalizing everywhere else as you're suggesting here.
>> Which would in turn requires us to spell out Babel-specific semantics of
>> session based evaluation -- something we have thus-far avoided.
>
> Again, the docs reference 'Session based evaluation' in big bold letters and
> don't indicate the thinness of the wrapper.
>
>> 
>> Some positive points *for* the "thin wrapper" approach include;
>> 
>> 1. It is the simplest to implement
>>    - easier to implement support for new language
>>    - easier to maintain -- especially if interactive tools change
>>    - easier to reason about -- thinking about bug fixing here
>>    - works with alternate back-ends e.g., ipython
>> 
>
> I think I have a solution for Python that gives full "session-based evaluation"
> and which is better than the current "thin wrapper" for all four concerns above
> (See bottom of my message for an example.)  This same approach, I think, would
> work for Ruby, but I would need to do a little digging to find exact method. 
> Let me know if you want me to do that digging.
>
>> 2. It is the least surprising.  I'd maintain that for users familiar
>>    with using Python sessions this behavior is the easiest to quickly
>>    understand and use.  As opposed to if we did something fancy (even if
>>    it was an improvement) it would be another environment for language
>>    users to have to familiarize themselves with -- they'd have both
>>    normal session rules and babel session rules
>
> As a sometimes Python user it had me totally confused.  I don't think of
> "session-based" as having any essential tie to "interactive shell" at all.  Why
> would I worry about state being maintained between statements in an org block? 
> State is already maintained between statements in non-session Org blocks. The
> only difference with session-based blocks is the starting state may not be
> "fresh".
>
> When I look at Org and think of "session-based" blocks, I see a potential big
> benefit in having multiple source-blocks throughout my document share the same
> session.  That way I don't need to use Org's (clever and useful but) somewhat
> clunky and inflexible method of passing data explicitly.  In statistics-based
> docs, for example, I can imagine that the sheer amount of data would make using
> Org's explicit variable-passing method unfeasible.  Using
> session-based-evaluation is at the same time both simpler and more heavy-duty.
>
> I never thought the use of session-based code blocks as something like a
> scratchpad was much more than toy. If you're typing in code that's not intended
> to be an integral part of an Org doc then you could just as well jump to the
> shell and enter it there.
>
> Here's an example of how to run a block of Python code of any length in the
> interactive session and save its output in a file.  Note that this is _not_
> interactive, even though the command runs in the interactive shell's session:
>
> Start with this block of code:
> -------------------------------------
> #+begin_src python :results output :session mypy 
>     x = 1
>     y = 1
>     z = 1
>     for i in range(1,2):
>         x = x + i
>         print x
>     
>         for y in range(10,11):
>             print y
>     # comment here
>         for z in range(5,6):
>             print z
>             while y > 0:
>                 print y
>                 y=y-1
>
>     print "Did it work?"
> #+end_src
> ------------------------------------
>
> Strip out the block of empty space at the left side and save in a file in
> current directory (say, 'pyfile.py') with a couple commands prepended and
> appended:
> -----------------------------
> # beginning of file
> # prepend code to redirect stdout to file
> import sys
> saveout = sys.stdout
> fsock = open('pyfile-output.log','w')
> sys.stdout = fsock
>
> # code block here, with beginning space block removed
> x = 1
> y = 1
> z = 1
> for i in range(1,2):
>     x = x + i
>     print x
>
>     for y in range(10,11):
>         print y
> # comment here
>     for z in range(5,6):
>         print z
>         while y > 0:
>             print y
>             y=y-1
> print "Did it work?"
> # end of the org source code block
>
> # return stdout to what it was
> sys.stdout = saveout
> # and close the log file
> fsock.close()
> # end of file
> ----------------------------
>
> Now you can go into the python shell and execute the code in pyfile.py:
>
>>>> execfile('pyfile.py')
>
> The output will be saved in the log file in the working directory.  I think it
> should be exactly the same as the output from the non-session based code
> (assuming the session-based code started from a fresh session).
>
> -- Herb
>
>
>
>

-- 
Eric Schulte
http://cs.unm.edu/~eschulte/

Re: org-babel -- Improper syntax error in session mode?

[Herbert Sitz]

Eric Schulte <schulte.eric <at> gmail.com> writes:

> You are suggesting that code to be run "interactively" should be written
> to an external file then loaded into the interactive session.  

Generally, yes, because babel's definition of "interactive" (execute an
arbitrary collection of code lines all at once) is different from an interactive
shell session's (execute a single statement of statement block at a time, giving
output after each).

> This
> would certainly work around the syntax limitation of the current setup.
> My two concerns here are that
> 
> 1. users who use interactive babel blocks side-by-side with the session
>    may be used jumping into the session to play with code interactively
>    and debug, in such cases rather than seeing their code in the session
>    history they would only see execfile('/tmp/blahblah').  Note I do
>    recall discussion on list related to user's reading their session
>    code in the inferior session buffer.

I see your point, that's a valuable tool.  Ideally, it seems to me, the session
blocks could be evaluated _either_ "interactively" (with a history in shell
session) or as a block-as-a-whole.  Having a named session (e.g., ':session
debug') designated to use the interactive mode and other :sessions use
"non-interactive" mode would be one way to do this.

> 2. similarly error messages would now point into this temporary file,
>    rather than back into the session history

I think the ':session debug' behavior in paragraph above could get around this.
 (I think there are also ways to feed the source lines through <stdin> but that
doesn't get around your issue 2., anyway.)

> 
> Basically you would prefer more decoupling from the interpreter and I'm
> not sure for the average user if this would be a worthwhile exchange
> simply to be able to avoid syntax errors like your originally mentioned
> example (which was the first such post I've seen on this list).
> 
> I'm disinclined to make such a change without a wider base of support
> for the request from the Babel/Python user community -- or at least
> without more complaints about the existing behavior.
> 

Yeah, I completely get you here.  If there are no other complaints about
existing behavior, and with limited resources, there's no good reason to change
things.

Regards,

Herb

Re: org-babel -- Improper syntax error in session mode?

[Jambunathan K]

Discalimer: I neither use python or babel.

> Eric Schulte <schulte.eric <at> gmail.com> writes:
>> I can confirm that I see the same behavior.  Also, if I manually type
>> the body of the code block into the session I get the same error output
>> from Python, so I don't believe this is due to a problem with Babel.
>> 
>
> It appears the problem is that the python session is interactive and
> is built to emit output after each Python "block" (e.g., the 'for'
> block), before another "block" of Python is entered.

> If this is the way it's designed 

ISTM you are speculating on design based on a very simple example and
just this "specific implementation".

> then it seems to me that it's Babel's obligation to feed the Python
> blocks to the Python session as required and then assemble the output
> pieces as appropriate.  Or am I missing something? -- Herb

There is a difference between feeding an interactive shell by hand and
feeding interactive shell via a program (The latter one is very
fast). The behaviour pertaining to buffering and flusing of output
buffers would not be apparent unless large volumes of output text is
spewed. 

The assumption that is being made in this thread is that:

"Python interpreter blocks until *all* output are *appears at the*
console before moving on to the next block."

While it is reasonable to assume that Python interpreter *flushes the
output buffers" it seems a bit too far-fetched to me to assume that
python interpreter can *guarantee" the appearance of the spewed block
before proceeding to the next block.

Unless python spec clearly and *positively* confirms the behaviour you
are assuming in *all* compliant-implementations, it is generally a good
idea to be conservative and not rely on observed behaviour too much.

Just my 2 cents. 

ps: Ignore if I have poorly understood the items discussed in this
thread.

Jambunathan K..


--