[BUG] ob-python: async session evaluation does not support python-mode.el [9.6-pre (release_9.5.5-2661-g2d040b.dirty @ /home/yantar92/.emacs.d/straight/build/org/)]

[BUG] ob-python: async session evaluation does not support python-mode.el [9.6-pre (release_9.5.5-2661-g2d040b.dirty @ /home/yantar92/.emacs.d/straight/build/org/)]

[Ihor Radchenko]

Hi,

Commit 53fd5b774 introduced async session support in Python src blocks:

ob-comint.el, ob-python.el: Async session evaluation

Adds functionality to ob-comint.el to implement async session eval on
a per-language basis.  Adds a reference implementation for ob-python.

However, async evaluation fails when org-babel-python-mode is set to
'python-mode.

MWE:

Executing

#+begin_src python :session :async yes :results output
import time
time.sleep(.1)
print('Yep!')
#+end_src

will never yield.

This is because org-babel-python-async-evaluate-session assumes that
built-in python.el is used to initiate the python session, which is not
the case when org-babel-python-mode is set to 'python-mode.

See org-babel-python-initiate-session-by-key.

Emacs  : GNU Emacs 28.1.90 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.34, cairo version 1.16.0)
 of 2022-07-17
Package: Org mode version 9.6-pre (release_9.5.5-2661-g2d040b.dirty @ /home/yantar92/.emacs.d/straight/build/org/)
-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [BUG] ob-python: async session evaluation does not support python-mode.el [9.6-pre (release_9.5.5-2661-g2d040b.dirty @ /home/yantar92/.emacs.d/straight/build/org/)]

[Jack Kamm]

Hi Ihor,

Ihor Radchenko <yantar92@posteo.net> writes:

> However, async evaluation fails when org-babel-python-mode is set to
> 'python-mode.

As a stopgap, perhaps Org could issue an error message that async is not
implemented for python-mode.el

I was actually planning to deprecate ob-python support for
python-mode.el, because I don't think it's worth the extra complexity &
burden to support 2 python modes. For users who prefer python-mode, I
think it would be better to have a separate ob-python-mode, perhaps in
org-contrib. There was a related discussion about this [1] a couple
years ago.

However, I never got around to deprecating python-mode support, as I
have been unable to work on ob-python for the last 1.5+ years due to
employment changes and subsequent copyright problems (my employer's
lawyers have some issues with the FSF's form, and I am waiting for the
FSF to review their proposed changes). I was hoping it would be resolved
by now, but it's still pending, and I should be removed as ob-python
maintainer since my hands are tied.

[1] https://list.orgmode.org/87d0a95eyz.fsf@bzg.fr/

[DISCUSSION] Should we deprecate python-mode.el (alternative to built-in python.el) support in ob-python? (was: [BUG] ob-python: async session evaluation does not support python-mode.el [9.6-pre (release_9.5.5-2661-g2d040b.dirty @ /home/yantar92/.emacs.d/straight/build/org/)])

[Ihor Radchenko]

Jack Kamm <jackkamm@gmail.com> writes:

> I was actually planning to deprecate ob-python support for
> python-mode.el, because I don't think it's worth the extra complexity &
> burden to support 2 python modes. For users who prefer python-mode, I
> think it would be better to have a separate ob-python-mode, perhaps in
> org-contrib. There was a related discussion about this [1] a couple
> years ago.
> ...
> [1] https://list.orgmode.org/87d0a95eyz.fsf@bzg.fr/

Dear All,

We are currently supporting two possible python REPL backends in
ob-python: `org-babel-python-mode' can be either 'python or 'python-mode
for built-in python.el and third-party python-mode.el, respectively.

However, supporting python-mode.el integration is difficult for Org
team: neither the current ob-python maintainer nor the rest of Org team
are familiar with python-mode.el.

The bug discussed above already shows that parts of ob-python do not
work properly with third-party python-mode.el.

Should we deprecate the support altogether and not bother with the extra
maintenance? Or maybe someone want to volunteer maintaining
python-mode.el integration?

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [DISCUSSION] Should we deprecate python-mode.el (alternative to built-in python.el) support in ob-python? (was: [BUG] ob-python: async session evaluation does not support python-mode.el [9.6-pre (release_9.5.5-2661-g2d040b.dirty @ /home/yantar92/.emacs.d/straight/build/org/)])

[Jack Kamm]

Ihor Radchenko <yantar92@posteo.net> writes:

> However, supporting python-mode.el integration is difficult for Org
> team: neither the current ob-python maintainer nor the rest of Org team
> are familiar with python-mode.el.
>
> The bug discussed above already shows that parts of ob-python do not
> work properly with third-party python-mode.el.
>
> Should we deprecate the support altogether and not bother with the extra
> maintenance? Or maybe someone want to volunteer maintaining
> python-mode.el integration?

Thanks for this summary Ihor. In my experience, supporting both
python.el and python-mode.el adds a high maintenance burden to
ob-python. I think it would be better if ob-python could focus on
python.el only, and a separate ob-python-mode created if there is demand
for it.

One cause of the higher maintenance burden, is that a lot of
functionality needs to be implemented twice to support both modes. Or
alternatively, functions need to be written in a way that's agnostic to
the python mode -- but this also adds complexity, as working with comint
directly can be tricky and bug-prone.

Testing is another problem, as there isn't a way to use ob-python with
both python modes in the same emacs config. So a separate emacs setup
has to be maintained to test that python-mode works, which adds more
maintenance headache.

If python-mode support is removed, I'd be willing to help create a new
ob-python-mode package, so that python-mode users can keep using
Babel. However I don't normally use python-mode, so it would be best
if a python-mode user could volunteer to help with this as well.

Re: [DISCUSSION] Should we deprecate python-mode.el (alternative to built-in python.el) support in ob-python? (was: [BUG] ob-python: async session evaluation does not support python-mode.el [9.6-pre (release_9.5.5-2661-g2d040b.dirty @ /home/yantar92/.emacs.d/straight/build/org/)])

[Ihor Radchenko]

Ihor Radchenko <yantar92@posteo.net> writes:

> Should we deprecate the support altogether and not bother with the extra
> maintenance? Or maybe someone want to volunteer maintaining
> python-mode.el integration?

No objections have been received.
Also, python-mode support have been broken for a long time now and no
bug reports have been submitted.

I am leaning towards announcing the deprecation in the coming release.

Bastien, what do you think?

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [DISCUSSION] Should we deprecate python-mode.el (alternative to built-in python.el) support in ob-python?

[Bastien Guerry]

Ihor Radchenko <yantar92@posteo.net> writes:

> I am leaning towards announcing the deprecation in the coming
> release.

Agreed, let's move forward in this direction for the release.

-- 
 Bastien

Re: [DISCUSSION] Should we deprecate python-mode.el (alternative to built-in python.el) support in ob-python?

[Bastien]

Jack Kamm <jackkamm@gmail.com> writes:

> If python-mode support is removed, I'd be willing to help create a new
> ob-python-mode package

Indeed.  A suggestion: can you or Ihor send a help request to the
list, explaining the issue at hand and asking if someone wants to
create and maintain ob-python-mode.el?  

This way we can refer to this email on the list when announcing that
ob-python.el do not support python-mode.el anymore.

-- 
 Bastien

Re: [DISCUSSION] Should we deprecate python-mode.el (alternative to built-in python.el) support in ob-python?

[Ihor Radchenko]

Bastien <bzg@gnu.org> writes:

> Jack Kamm <jackkamm@gmail.com> writes:
>
>> If python-mode support is removed, I'd be willing to help create a new
>> ob-python-mode package
>
> Indeed.  A suggestion: can you or Ihor send a help request to the
> list, explaining the issue at hand and asking if someone wants to
> create and maintain ob-python-mode.el?  
>
> This way we can refer to this email on the list when announcing that
> ob-python.el do not support python-mode.el anymore.

Done: https://orgmode.org/list/87lenyjaxx.fsf@localhost

I also added NEWS entry
https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=6db75d560

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [DISCUSSION] Should we deprecate python-mode.el (alternative to built-in python.el) support in ob-python?

[Bastien]

Ihor Radchenko <yantar92@posteo.net> writes:

> Done: https://orgmode.org/list/87lenyjaxx.fsf@localhost
>
> I also added NEWS entry
> https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=6db75d560

Thanks!

-- 
 Bastien

Re: [DISCUSSION] Should we deprecate python-mode.el (alternative to built-in python.el) support in ob-python?

[Ihor Radchenko]

Bastien Guerry <bzg@gnu.org> writes:

> Ihor Radchenko <yantar92@posteo.net> writes:
>
>> I am leaning towards announcing the deprecation in the coming
>> release.
>
> Agreed, let's move forward in this direction for the release.

Marked for future removal in ob-python.el.
https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=21741a469

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [DISCUSSION] Should we deprecate python-mode.el (alternative to built-in python.el) support in ob-python?

[Bastien]

Ihor Radchenko <yantar92@posteo.net> writes:

> Marked for future removal in ob-python.el.
> https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=21741a469

Thanks!

-- 
 Bastien

Re: [DISCUSSION] Should we deprecate python-mode.el (alternative to built-in python.el) support in ob-python?

[Jack Kamm]

[-- Attachment #1: Type: text/plain, Size: 566 bytes --]

Ihor Radchenko <yantar92@posteo.net> writes:

> Marked for future removal in ob-python.el.
> https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=21741a469

Thanks Ihor.

Not sure if it's too early for this, but I've prepared a patch to remove
the support for python-mode.el, attached.

Also, to help any python-mode users with the transition, I ported the
code to support python-mode to an external package here:

https://gitlab.com/jackkamm/ob-python-mode-mode

As always, feedback is welcome, either on this patch or the external
package.

Best,
Jack


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-ob-python-Remove-python-mode.el-support.patch --]
[-- Type: text/x-patch, Size: 7948 bytes --]

From 44bdfbbd9858f4190e4404467fa61b8a3f445347 Mon Sep 17 00:00:00 2001
From: Jack Kamm <jackkamm@gmail.com>
Date: Sat, 21 Jan 2023 08:24:41 -0800
Subject: [PATCH] ob-python: Remove python-mode.el support

* lisp/ob-python.el (org-babel-python-mode): Remove customization
variable.
(org-babel-python-initiate-session-by-key): Remove code to support
python-mode.el.
(org-babel-python-send-string): Renamed from
org-babel-python--send-string, turning it into a public function to
accommodate ob-python-mode-mode which advises this function. Also,
remove some code for python-mode.el
(org-babel-python-evaluate-session): Update calls to renamed function
org-babel-python-send-string.
---
 etc/ORG-NEWS      | 10 ++++++
 lisp/ob-python.el | 88 +++++++++++++----------------------------------
 2 files changed, 34 insertions(+), 64 deletions(-)

diff --git a/etc/ORG-NEWS b/etc/ORG-NEWS
index 3ef76ec1a..ad7bd6604 100644
--- a/etc/ORG-NEWS
+++ b/etc/ORG-NEWS
@@ -12,6 +12,16 @@ See the end of the file for license conditions.
 Please send Org bug reports to mailto:emacs-orgmode@gnu.org.
 
 * Version 9.7 (not released yet)
+** Important announcements and breaking changes
+*** =python-mode.el (MELPA)= support in =ob-python.el= is removed
+
+=python-mode.el= support has been removed from =ob-python.el=. The
+related customization =org-babel-python-mode= has also been removed.
+
+If you still want to use python-mode with ob-python, you might
+consider [[https://gitlab.com/jackkamm/ob-python-mode-mode][ob-python-mode-mode]], where the code to support python-mode
+has been ported to.
+
 ** New options
 *** New options for the "csl" citation export processor's LaTeX output
 
diff --git a/lisp/ob-python.el b/lisp/ob-python.el
index 39fe5c28d..f19440e00 100644
--- a/lisp/ob-python.el
+++ b/lisp/ob-python.el
@@ -36,10 +36,6 @@ (require 'ob)
 (require 'org-macs)
 (require 'python)
 
-(declare-function py-shell "ext:python-mode" (&rest args))
-(declare-function py-choose-shell "ext:python-mode" (&optional shell))
-(declare-function py-shell-send-string "ext:python-mode" (strg &optional process))
-
 (defvar org-babel-tangle-lang-exts)
 (add-to-list 'org-babel-tangle-lang-exts '("python" . "py"))
 
@@ -52,16 +48,6 @@ (defcustom org-babel-python-command "python"
   :group 'org-babel
   :type 'string)
 
-;; FIXME: Remove third-party `python-mode' package support in the next release.
-(defcustom org-babel-python-mode
-  (if (featurep 'python-mode) 'python-mode 'python)
-  "Preferred python mode for use in running python interactively.
-This will typically be either `python' or `python-mode'."
-  :group 'org-babel
-  :version "24.4"
-  :package-version '(Org . "8.0")
-  :type 'symbol)
-
 (defcustom org-babel-python-hline-to "None"
   "Replace hlines in incoming tables with this when translating to python."
   :group 'org-babel
@@ -183,7 +169,6 @@ (defun org-babel-python-without-earmuffs (session)
 	(substring name 1 (- (length name) 1))
       name)))
 
-(defvar py-which-bufname)
 (defvar python-shell-buffer-name)
 (defvar-local org-babel-python--initialized nil
   "Flag used to mark that python session has been initialized.")
@@ -197,47 +182,25 @@ (defun org-babel-python-initiate-session-by-key (&optional session)
 	   (cmd (if (member system-type '(cygwin windows-nt ms-dos))
 		    (concat org-babel-python-command " -i")
 		  org-babel-python-command)))
-      (cond
-       ((eq 'python org-babel-python-mode) ; python.el
-	(unless py-buffer
-	  (setq py-buffer (org-babel-python-with-earmuffs session)))
-	(let ((python-shell-buffer-name
-	       (org-babel-python-without-earmuffs py-buffer)))
-	  (run-python cmd)
-          (with-current-buffer py-buffer
-            (add-hook
-             'python-shell-first-prompt-hook
-             (lambda ()
-               (setq-local org-babel-python--initialized t)
-               (message "I am running!!!"))
-             nil 'local))))
-       ((and (eq 'python-mode org-babel-python-mode)
-	     (fboundp 'py-shell)) ; python-mode.el
-	(require 'python-mode)
-	;; Make sure that py-which-bufname is initialized, as otherwise
-	;; it will be overwritten the first time a Python buffer is
-	;; created.
-	(py-choose-shell)
-	;; `py-shell' creates a buffer whose name is the value of
-	;; `py-which-bufname' with '*'s at the beginning and end
-	(let* ((bufname (if (and py-buffer (buffer-live-p py-buffer))
-			    (replace-regexp-in-string ;; zap surrounding *
-			     "^\\*\\([^*]+\\)\\*$" "\\1" py-buffer)
-			  (concat "Python-" (symbol-name session))))
-	       (py-which-bufname bufname))
-	  (setq py-buffer (org-babel-python-with-earmuffs bufname))
-	  (py-shell nil nil t org-babel-python-command py-buffer nil nil t nil)))
-       (t
-	(error "No function available for running an inferior Python")))
+      (unless py-buffer
+	(setq py-buffer (org-babel-python-with-earmuffs session)))
+      (let ((python-shell-buffer-name
+	     (org-babel-python-without-earmuffs py-buffer)))
+	(run-python cmd)
+        (with-current-buffer py-buffer
+          (add-hook
+           'python-shell-first-prompt-hook
+           (lambda ()
+             (setq-local org-babel-python--initialized t)
+             (message "I am running!!!"))
+           nil 'local)))
       ;; Wait until Python initializes.
-      (if (eq 'python org-babel-python-mode) ; python.el
-          ;; This is more reliable compared to
-          ;; `org-babel-comint-wait-for-output' as python may emit
-          ;; multiple prompts during initialization.
-          (with-current-buffer py-buffer
-            (while (not org-babel-python--initialized)
-              (org-babel-comint-wait-for-output py-buffer)))
-        (org-babel-comint-wait-for-output py-buffer))
+      ;; This is more reliable compared to
+      ;; `org-babel-comint-wait-for-output' as python may emit
+      ;; multiple prompts during initialization.
+      (with-current-buffer py-buffer
+        (while (not org-babel-python--initialized)
+          (org-babel-comint-wait-for-output py-buffer)))
       (setq org-babel-python-buffers
 	    (cons (cons session py-buffer)
 		  (assq-delete-all session org-babel-python-buffers)))
@@ -352,7 +315,7 @@ (defun org-babel-python-evaluate-external-process
       raw
       (org-babel-python-table-or-string (org-trim raw)))))
 
-(defun org-babel-python--send-string (session body)
+(defun org-babel-python-send-string (session body)
   "Pass BODY to the Python process in SESSION.
 Return output."
   (with-current-buffer session
@@ -370,12 +333,9 @@ (defun org-babel-python--send-string (session body)
     print('%s')"
 			 (org-babel-python--shift-right body 4)
 			 org-babel-python-eoe-indicator)))
-      (if (not (eq 'python-mode org-babel-python-mode))
-	  (let ((python-shell-buffer-name
-		 (org-babel-python-without-earmuffs session)))
-	    (python-shell-send-string body))
-	(require 'python-mode)
-	(py-shell-send-string body (get-buffer-process session)))
+      (let ((python-shell-buffer-name
+	     (org-babel-python-without-earmuffs session)))
+	(python-shell-send-string body))
       ;; same as `python-shell-comint-end-of-output-p' in emacs-25.1+
       (while (not (string-match
 		   org-babel-python-eoe-indicator
@@ -398,12 +358,12 @@ (defun org-babel-python-evaluate-session
 	       (let ((body (format org-babel-python--exec-tmpfile
 				   (org-babel-process-file-name
 				    tmp-src-file 'noquote))))
-		 (org-babel-python--send-string session body)))
+		 (org-babel-python-send-string session body)))
               (`value
                (let* ((tmp-results-file (org-babel-temp-file "python-"))
 		      (body (org-babel-python-format-session-value
 			     tmp-src-file tmp-results-file result-params)))
-		 (org-babel-python--send-string session body)
+		 (org-babel-python-send-string session body)
 		 (sleep-for 0 10)
 		 (org-babel-eval-read-file tmp-results-file)))))))
     (org-babel-result-cond result-params
-- 
2.39.1

Re: [DISCUSSION] Should we deprecate python-mode.el (alternative to built-in python.el) support in ob-python?

[Ihor Radchenko]

Jack Kamm <jackkamm@gmail.com> writes:

> Ihor Radchenko <yantar92@posteo.net> writes:
>
>> Marked for future removal in ob-python.el.
>> https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=21741a469
>
> Thanks Ihor.
>
> Not sure if it's too early for this, but I've prepared a patch to remove
> the support for python-mode.el, attached.

main branch is for Org 9.7. So, removing it should happen before the
next Org 9.7 release on main anyway. I have no objections on this.

> Also, to help any python-mode users with the transition, I ported the
> code to support python-mode to an external package here:
>
> https://gitlab.com/jackkamm/ob-python-mode-mode
>
> As always, feedback is welcome, either on this patch or the external
> package.

I find the need to use advise awkward. Is it necessary?

> Subject: [PATCH] ob-python: Remove python-mode.el support
>
> * lisp/ob-python.el (org-babel-python-mode): Remove customization
> variable.

Let's follow the usual approach and first deprecate it. It should become
a constant with 'python value in org-compat.el. Deprecating is not make
sure that other packages that might be testing the variable value do not
get broken.

> (org-babel-python-send-string): Renamed from
> org-babel-python--send-string, turning it into a public function to
> accommodate ob-python-mode-mode which advises this function. Also,
> remove some code for python-mode.el

Note that we usually `quote' Elisp symbols in the commit messages.

Otherwise, the patch looks good.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [DISCUSSION] Should we deprecate python-mode.el (alternative to built-in python.el) support in ob-python?

[Jack Kamm]

Ihor Radchenko <yantar92@posteo.net> writes:

> Let's follow the usual approach and first deprecate it. It should become
> a constant with 'python value in org-compat.el. Deprecating is not make
> sure that other packages that might be testing the variable value do not
> get broken.
>
> Note that we usually `quote' Elisp symbols in the commit messages.

Thank you for the review. I've incorporated your feedback and pushed
the patch as commit aa48c80fe17eaaaf83c11c9ac2f2fd864f2f3ad9

> I find the need to use advise awkward. Is it necessary?

I agree it is awkward, and also fragile to future changes in ob-python.

But unless I've missed something, I think some advice is needed to get
the old behavior. Still, it would be more robust to just advise the
main function org-babel-execute:python, instead of advising the 2
helper functions.

But, this would require a little extra work, and while I might do it
later, it's a lower priority for me right now (it's unclear whether
there are currently active python-mode users of ob-python). For the
moment, I just updated the Readme to say the current implementation is
a temporary workaround, and it will work for Org 9.7, but may not work
in future versions as ob-python evolves.

Arguably, it would be best to avoid overriding python src blocks
altogether, and instead have completely separate python-mode src
blocks. But I would leave such a design decision to whoever decides to
maintain babel+python-mode integration in future.