bug#68559: [PATCH] Improve Python shell completion

[kobarity <kobarity@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 1532 bytes --]

Mattias Engdegård wrote:
> 
> 16 feb. 2024 kl. 10.34 skrev kobarity <kobarity@gmail.com>:
> 
> > This is caused by the fact that the input is echoed back on MacOS
> > Python.
> 
> Right, that bug needs to be fixed as well, but the echo problem existed prior to the change that broke the tests.
> 
> An alternative might be to disable the tty echo altogether. If I do it right after process creation then it has no effect; presumably Python or its readline module turns on echo just a bit later on. Anyway, running
> 
>   import tty
>   tty.setraw(0)
> 
> in the python shell seems to put it right, with working completion and the annoying echo gone.
> 
> > So one workaround would be to remove the echoed back string before
> > parsing as JSON.
> 
> Yes, either that or turning off echo in the tty.

I made prototype patches for each method.  I don't use Mac so it would
be helpful if you could try these.

0001-Remove-echoed-back-string-in-python-shell-completion.patch
extracts only the last line to exclude echoed back strings.

0001-Set-tty-mode-to-raw-when-setting-up-Inferior-Python.patch sets
the Inferior Python tty to raw mode.  python-ffap-module-path-1 will
no longer need to be skipped on Mac.  If it is safe to set tty to raw
mode on all UNIX based systems, I prefer this method.

By the way, is it necessary to send
`python-shell-completion-setup-code' for every completion in
`python-shell-completion-get-completions'?  To me it seems sufficient
to send it once at initialization.

[-- Attachment #2: 0001-Remove-echoed-back-string-in-python-shell-completion.patch --]
[-- Type: application/octet-stream, Size: 1480 bytes --]

From 601419eef799d2c68e3789222f5ebe2a8c31af97 Mon Sep 17 00:00:00 2001
From: kobarity <kobarity@gmail.com>
Date: Fri, 16 Feb 2024 22:48:19 +0900
Subject: [PATCH] Remove echoed back string in
 python-shell-completion-get-completions

* lisp/progmodes/python.el (python-shell-completion-get-completions):
Remove echoed back string. (Bug#68559)
---
 lisp/progmodes/python.el | 12 +++++++-----
 1 file changed, 7 insertions(+), 5 deletions(-)

diff --git a/lisp/progmodes/python.el b/lisp/progmodes/python.el
index b7e43f3fc68..a842a498113 100644
--- a/lisp/progmodes/python.el
+++ b/lisp/progmodes/python.el
@@ -4624,11 +4624,13 @@ python-shell-completion-get-completions
   "Get completions of INPUT using PROCESS."
   (with-current-buffer (process-buffer process)
     (python--parse-json-array
-     (python-shell-send-string-no-output
-      (format "%s\nprint(__PYTHON_EL_get_completions(%s))"
-              python-shell-completion-setup-code
-              (python-shell--encode-string input))
-      process))))
+     (car (last (split-string
+                 (python-shell-send-string-no-output
+                  (format "%s\nprint(__PYTHON_EL_get_completions(%s))"
+                          python-shell-completion-setup-code
+                          (python-shell--encode-string input))
+                  process)
+                 "[\n\r]+" t))))))
 
 (defun python-shell--get-multiline-input ()
   "Return lines at a multi-line input in Python shell."
-- 
2.34.1


[-- Attachment #3: 0001-Set-tty-mode-to-raw-when-setting-up-Inferior-Python.patch --]
[-- Type: application/octet-stream, Size: 2531 bytes --]

From 31738d481333bf6c258a694cf57bce944bce7778 Mon Sep 17 00:00:00 2001
From: kobarity <kobarity@gmail.com>
Date: Fri, 16 Feb 2024 22:52:06 +0900
Subject: [PATCH] Set tty mode to raw when setting up Inferior Python

* lisp/progmodes/python.el (python-shell-setup-code): New constant.
(python-shell-comint-watch-for-first-prompt-output-filter): Send
`python-shell-setup-code' to the Inferior Python process.
* test/lisp/progmodes/python-tests.el (python-ffap-module-path-1):
Eliminate skipping on Mac. (Bug#68559)
---
 lisp/progmodes/python.el            | 11 +++++++++++
 test/lisp/progmodes/python-tests.el |  5 -----
 2 files changed, 11 insertions(+), 5 deletions(-)

diff --git a/lisp/progmodes/python.el b/lisp/progmodes/python.el
index b7e43f3fc68..5501926e69d 100644
--- a/lisp/progmodes/python.el
+++ b/lisp/progmodes/python.el
@@ -3521,6 +3521,16 @@ python-shell-first-prompt-hook
   :version "25.1"
   :type 'hook)
 
+(defconst python-shell-setup-code
+  "\
+try:
+    import tty
+except ImportError:
+    pass
+else:
+    tty.setraw(0)"
+  "Code used to setup the inferior Python processes.")
+
 (defconst python-shell-eval-setup-code
   "\
 def __PYTHON_EL_eval(source, filename):
@@ -3586,6 +3596,7 @@ python-shell-comint-watch-for-first-prompt-output-filter
                       (format "exec(%s)\n" (python-shell--encode-string string))))))
           ;; Bootstrap: the normal definition of `python-shell-send-string'
           ;; depends on the Python code sent here.
+          (python-shell-send-string-no-output python-shell-setup-code)
           (python-shell-send-string-no-output python-shell-eval-setup-code)
           (python-shell-send-string-no-output python-shell-eval-file-setup-code))
         (with-current-buffer (current-buffer)
diff --git a/test/lisp/progmodes/python-tests.el b/test/lisp/progmodes/python-tests.el
index af6c199b5bd..6c6cd9eee2b 100644
--- a/test/lisp/progmodes/python-tests.el
+++ b/test/lisp/progmodes/python-tests.el
@@ -5037,11 +5037,6 @@ python-completion-at-point-native-with-eldoc-1
 
 (ert-deftest python-ffap-module-path-1 ()
   (skip-unless (executable-find python-tests-shell-interpreter))
-  ;; Skip the test on macOS, since the standard Python installation uses
-  ;; libedit rather than readline which confuses the running of an inferior
-  ;; interpreter in this case (see bug#59477 and bug#25753).
-  (skip-when (eq system-type 'darwin))
-  (trace-function 'python-shell-output-filter)
   (python-tests-with-temp-buffer-with-shell
    "
 import abc
-- 
2.34.1