Re: ob-python newline & indentation behavior

[Jack Kamm <jackkamm@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 8459 bytes --]

Sorry, I should have mentioned my version info anyways. I have tested on
emacs 25.3.1 and emacs 26.0.90, and org-mode versions 9.1.2 and 9.1.3
(current master).

The same error occurs on all emacs and org-mode versions. However the error
slightly differs between Python and IPython interpreters. IPython prints
"11" (instead of the expected "20"), whereas Python raises an
IndentationError then prints "10".

I've cleaned up my patch to deal with a few edge cases. In particular:
1) Fixed a bug where I was removing blank lines (these may be needed if
they are part of a multiline string object)
2) Send to tmpfile any multiline block even if not indented. This follows
the python.el behavior and gives more consistent behavior for ":results
output"
3) Allow ":results value" to work when the block ends in several newlines,
using string-trim-right

Here is the new version of the patch:

From f009da37d3b7e2730abb8cbb10f4d07b3d456dd8 Mon Sep 17 00:00:00 2001

From: Jack Kamm <jackkamm@gmail.com>
Date: Sun, 19 Nov 2017 07:13:56 +0000
Subject: [PATCH] Squashed commit of the following:

commit d1fe88a9f61a8e7082f08b7c190a29737bb655d5
Author: Jack Kamm <jackkamm@gmail.com>
Date:   Sun Nov 19 07:08:31 2017 +0000

    fix block ending in blank lines; send multiline blocks to tmpfile

commit fcc5a7795e882716775c9d925b0cd5b657da041b
Author: Jack Kamm <jackkamm@gmail.com>
Date:   Sat Nov 18 22:40:31 2017 +0000

    fix newlines and blanklines when sending codeblock to tmpfile

commit a5d553ece9f6ee35cd1e273e554a21a19e80ec3c
Author: Jack Kamm <jackkamm@gmail.com>
Date:   Sat Nov 18 21:47:09 2017 +0000

    fix newline/indentation issues in ob-python :session
---
 lisp/ob-python.el | 29 +++++++++++++++++++++++++++++
 1 file changed, 29 insertions(+)

diff --git a/lisp/ob-python.el b/lisp/ob-python.el
index 60ec5fa47..c3dba1565 100644
--- a/lisp/ob-python.el
+++ b/lisp/ob-python.el
@@ -303,6 +303,9 @@ last statement in BODY, as elisp."
                       (mapc (lambda (line) (insert line) (funcall
send-wait))
                             (split-string body "[\r\n]"))
                       (funcall send-wait)))
+        (body (if (string-match-p ".\n+." body) ;; Multiline
+                  (org-babel-python--replace-body-tmpfile body)
+                body))
          (results
           (pcase result-type
             (`output
@@ -340,6 +343,32 @@ last statement in BODY, as elisp."
       (substring string 1 -1)
     string))

+
+(defun org-babel-python--replace-body-tmpfile (body)
+  "Place body in tmpfile, and return string to exec the tmpfile.
+If last line of body is not indented, place it at end of exec string
+instead of tmpfile, so shell can see the result"
+  (let* ((body (string-trim-right body))
+        (tmp-file (org-babel-temp-file "python-"))
+        (lines (split-string body "[\r\n]"))
+        (lastline (car (last lines)))
+        (newbody (concat
+                  (format "__pyfilename = '%s'; " tmp-file)
+                  "__pyfile = open(__pyfilename); "
+                  "exec(compile("
+                  "__pyfile.read(), __pyfilename, 'exec'"
+                  ")); "
+                  "__pyfile.close()")))
+    (if (string-match-p "^[ \t]" lastline)
+       (progn
+         (with-temp-file tmp-file (insert body))
+         newbody)
+      (with-temp-file tmp-file
+       (insert (mapconcat 'identity
+                          (butlast lines) "\n")))
+      (concat newbody "\n" lastline)))
+  )
+
 (provide 'ob-python)


-- 
2.15.0




On Sun, Nov 19, 2017 at 3:34 AM, Martin Alsinet <martin@alsinet.com.ar>
wrote:

> Sorry Jack, I overlooked the :session bit.
> Disregard my email please
>
>
> On Sat, Nov 18, 2017 at 10:27 PM Martin Alsinet <martin@alsinet.com.ar>
> wrote:
>
>> Hello Jack:
>>
>> What versions of emacs and org-mode are you using?
>>
>> I tried your example on Emacs 25.3.1 and org-mode 9.1.2-22 and I got "20"
>> as result
>> It has happened to me in the past that some bug I was seeing goes away
>> just by updating org-mode.
>>
>> Regards,
>>
>>
>> Martin
>>
>> On Sat, Nov 18, 2017 at 5:16 PM Jack Kamm <jackkamm@gmail.com> wrote:
>>
>>> Thanks Kyle, and sorry for missing that recent related thread.
>>>
>>> I adapted your old patch to the current master branch. I also extended
>>> it to work for ":results value" (the original patch only worked for
>>> ":results output"). I did this by not writing the last line of the code
>>> block to the tmpfile, unless it is indented.
>>>
>>> I've never contributed before so would appreciate any comments on this
>>> patch, and how to get it accepted. Cheers, Jack.
>>>
>>> From a5d553ece9f6ee35cd1e273e554a21a19e80ec3c Mon Sep 17 00:00:00 2001
>>>
>>> From: Jack Kamm <jackkamm@gmail.com>
>>> Date: Sat, 18 Nov 2017 21:47:09 +0000
>>> Subject: [PATCH] fix newline/indentation issues in ob-python :session
>>>
>>> ---
>>>  lisp/ob-python.el | 33 +++++++++++++++++++++++++++++++++
>>>  1 file changed, 33 insertions(+)
>>>
>>> diff --git a/lisp/ob-python.el b/lisp/ob-python.el
>>> index 60ec5fa47..6623ef5fc 100644
>>> --- a/lisp/ob-python.el
>>> +++ b/lisp/ob-python.el
>>> @@ -303,6 +303,10 @@ last statement in BODY, as elisp."
>>>                        (mapc (lambda (line) (insert line) (funcall
>>> send-wait))
>>>                              (split-string body "[\r\n]"))
>>>                        (funcall send-wait)))
>>> +        (indented-p (org-babel-python--indented-p body))
>>> +        (body (if indented-p
>>> +                  (org-babel-python--replace-body-tmpfile body)
>>> +                body))
>>>           (results
>>>            (pcase result-type
>>>              (`output
>>> @@ -340,6 +344,35 @@ last statement in BODY, as elisp."
>>>        (substring string 1 -1)
>>>      string))
>>>
>>> +
>>> +(defun org-babel-python--indented-p (input)
>>> +     "Non-nil if any line in INPUT is indented."
>>> +     (string-match-p "^[ \t]" input))
>>> +
>>> +(defun org-babel-python--replace-body-tmpfile (body)
>>> +  "Place body in tmpfile, and return string to exec the tmpfile.
>>> +If last line of body is not indented, place it at end of exec string
>>> +instead of tmpfile, so shell can see the result"
>>> +  (let* ((tmp-file (org-babel-temp-file "python-"))
>>> +        (lines (split-string body "\n" t))
>>> +        (lastline (car (last lines)))
>>> +        (newbody (concat
>>> +                  (format "__pyfilename = '%s'; " tmp-file)
>>> +                  "__pyfile = open(__pyfilename); "
>>> +                  "exec(compile("
>>> +                  "__pyfile.read(), __pyfilename, 'exec'"
>>> +                  ")); "
>>> +                  "__pyfile.close()")))
>>> +    (if (string-match-p "^[ \t]" lastline)
>>> +       (progn
>>> +         (with-temp-file tmp-file (insert body))
>>> +         newbody)
>>> +      (with-temp-file tmp-file
>>> +       (insert (mapconcat 'identity
>>> +                          (butlast lines) "\n")))
>>> +      (concat newbody "\n" lastline)))
>>> +  )
>>> +
>>>  (provide 'ob-python)
>>>
>>>
>>> --
>>> 2.15.0
>>>
>>>
>>> On Sat, Nov 18, 2017 at 3:05 PM, Kyle Meyer <kyle@kyleam.com> wrote:
>>>
>>>> Hello,
>>>>
>>>> Jack Kamm <jackkamm@gmail.com> writes:
>>>>
>>>> > ob-python newline & indentation behavior in :session is very ugly and
>>>> > possibly broken. For example, consider the following code block:
>>>>
>>>> [...]
>>>>
>>>> > There is a 2 year old patch that fixes this behavior but has not yet
>>>> > been incorporated:
>>>> > https://lists.gnu.org/archive/html/emacs-orgmode/2015-03/
>>>> msg00505.html
>>>>
>>>> [...]
>>>>
>>>> > The patch follows the python.el behavior of using a temporary file and
>>>> > executing that from the shell.
>>>> >
>>>> > Could this patch, or something similar, be incorporated into org-mode,
>>>> > to fix this behavior?
>>>>
>>>> Here's what I said in a recent post about ob-python sessions:
>>>>
>>>>     https://lists.gnu.org/archive/html/emacs-orgmode/2017-10/
>>>> msg00198.html
>>>>
>>>>     I dropped my attempt to fix it because 1) I was still having trouble
>>>>     getting a complete understanding of what the issue was and 2) I
>>>> didn't
>>>>     have the motivation to spend time digging deeper because I don't use
>>>>     ob-python (and in general am not a heavy Org-Babel user).  Perhaps
>>>> you
>>>>     or some other ob-python user could help make ob-python sessions more
>>>>     robust?
>>>>
>>>> Perhaps you are the "you"?
>>>>
>>>> --
>>>> Kyle
>>>>
>>>
>>>

[-- Attachment #2: Type: text/html, Size: 12941 bytes --]