bug#68559: [PATCH] Improve Python shell completion

bug#68559: [PATCH] Improve Python shell completion

[Liu Hui]

[-- Attachment #1: Type: text/plain, Size: 856 bytes --]

Hi,

Currently python-shell-completion-at-point doesn't respect the
delimiter setting of readline completer and always split the text,
resulting in poor completions when completers that don't need word
splitting, e.g. jedi[1], is used. This patch fixes the problem.

Meanwhile, this patch adds an option 'python-shell-readline-completer'
to make Python shell provide better completion experience (e.g.
completing function parameters, dictionary keys) by default if users
are using vanilla Python interpreter with jedi or using IPython, where
the completion support for IPython has been enhanced (e.g. showing
type annotations and function signatures).

BTW, I think it may make sense to move Python snippets to a separate
file to make them easier to be maintained.

Thanks.

Best,

[1] https://jedi.readthedocs.io/en/latest/docs/usage.html#repl-completion

[-- Attachment #2: 0001-Improve-Python-shell-completion.patch --]
[-- Type: text/x-patch, Size: 21278 bytes --]

From 63a0f172bcd3a3ac882d76c7788c3637acc6ae39 Mon Sep 17 00:00:00 2001
From: Liu Hui <liuhui1610@gmail.com>
Date: Thu, 18 Jan 2024 12:00:00 +0800
Subject: [PATCH] Improve Python shell completion

* lisp/progmodes/python.el (python-shell-completion-setup-code): Fix
the completion code of IPython.  Change the return value to JSON
string and ...
(python-shell-completion-get-completions): ... simplify parsing.
(inferior-python-mode): Update docstring.
(python-shell-readline-completer): New option.
(python-shell-readline-completer-delims)
(python-shell-readline-jedi-setup-code)
(python-shell-readline-ipython-setup-code): New variables.
(python-shell-completion-native-setup): Setup a suitable readline
completer and set the completer delimiter.
(python-shell-completion-native-get-completions): Convert output
string to completions properly.
(python-shell-completion-at-point): Send text beginning from the line
start if the completion backend does not need word splitting.  Remove
the detection of import statement because it is not needed anymore.
Create proper completion table based on completions returned from
different backends.

* test/lisp/progmodes/python-tests.el (python-shell-completion-module):
(python-shell-completion-parameters):
(python-shell-completion-at-point-jedi-completer):
(python-shell-completion-at-point-ipython): New tests.

* etc/NEWS: Announce the change.
---
 etc/NEWS                            |   6 +
 lisp/progmodes/python.el            | 234 ++++++++++++++++++++++------
 test/lisp/progmodes/python-tests.el |  66 ++++++++
 3 files changed, 255 insertions(+), 51 deletions(-)

diff --git a/etc/NEWS b/etc/NEWS
index 03b8c3b517a..6fd337727c5 100644
--- a/etc/NEWS
+++ b/etc/NEWS
@@ -1029,6 +1029,12 @@ instead of:
 This allows the user to specify command line arguments to the non
 interactive Python interpreter specified by 'python-interpreter'.
 
+*** New user option 'python-shell-readline-completer'.
+This allows the user to specify the readline completer used for Python
+shell completion.  The default is 'auto', which means a suitable
+completer will be configured automatically according to the Python
+interpreter.
+
 ** use-package
 
 +++
diff --git a/lisp/progmodes/python.el b/lisp/progmodes/python.el
index ff799e1e662..2d87eb59ede 100644
--- a/lisp/progmodes/python.el
+++ b/lisp/progmodes/python.el
@@ -128,9 +128,9 @@
 ;; receiving escape sequences (with some limitations, i.e. completion
 ;; in blocks does not work).  The code executed for the "fallback"
 ;; completion can be found in `python-shell-completion-setup-code' and
-;; `python-shell-completion-string-code' variables.  Their default
-;; values enable completion for both CPython and IPython, and probably
-;; any readline based shell (it's known to work with PyPy).  If your
+;; `python-shell-completion-get-completions'.  Their default values
+;; enable completion for both CPython and IPython, and probably any
+;; readline based shell (it's known to work with PyPy).  If your
 ;; Python installation lacks readline (like CPython for Windows),
 ;; installing pyreadline (URL `https://ipython.org/pyreadline.html')
 ;; should suffice.  To troubleshoot why you are not getting any
@@ -3601,7 +3601,6 @@ inferior-python-mode
 `python-shell-prompt-block-regexp',
 `python-shell-font-lock-enable',
 `python-shell-completion-setup-code',
-`python-shell-completion-string-code',
 `python-eldoc-setup-code',
 `python-ffap-setup-code' can
 customize this mode for different Python interpreters.
@@ -4319,8 +4318,9 @@ python-shell-completion-setup-code
     completions = []
     completer = None
 
+    import json
     try:
-        import readline
+        import readline, re
 
         try:
             import __builtin__
@@ -4331,16 +4331,29 @@ python-shell-completion-setup-code
 
         is_ipython = ('__IPYTHON__' in builtins or
                       '__IPYTHON__active' in builtins)
-        splits = text.split()
-        is_module = splits and splits[0] in ('from', 'import')
-
-        if is_ipython and is_module:
-            from IPython.core.completerlib import module_completion
-            completions = module_completion(text.strip())
-        elif is_ipython and '__IP' in builtins:
-            completions = __IP.complete(text)
-        elif is_ipython and 'get_ipython' in builtins:
-            completions = get_ipython().Completer.all_completions(text)
+
+        if is_ipython and 'get_ipython' in builtins:
+            def filter_c(prefix, c):
+                if re.match('_+(i?[0-9]+)?$', c):
+                    return False
+                elif c[0] == '%' and not re.match('[%a-zA-Z]+$', prefix):
+                    return False
+                return True
+
+            import IPython
+            try:
+                if IPython.version_info[0] >= 6:
+                    from IPython.core.completer import provisionalcompleter
+                    with provisionalcompleter():
+                        completions = [
+                            [c.text, c.start, c.end, c.type or '?', c.signature or '']
+                             for c in get_ipython().Completer.completions(text, len(text))
+                             if filter_c(text, c.text)]
+                else:
+                    part, matches = get_ipython().Completer.complete(line_buffer=text)
+                    completions = [text + m[len(part):] for m in matches if filter_c(text, m)]
+            except:
+                pass
         else:
             # Try to reuse current completer.
             completer = readline.get_completer()
@@ -4363,7 +4376,7 @@ python-shell-completion-setup-code
     finally:
         if getattr(completer, 'PYTHON_EL_WRAPPED', False):
             completer.print_mode = True
-    return completions"
+    return json.dumps(completions)"
   "Code used to setup completion in inferior Python processes."
   :type 'string)
 
@@ -4404,6 +4417,79 @@ python-shell-completion-native-try-output-timeout
   :version "25.1"
   :type 'float)
 
+(defcustom python-shell-readline-completer 'auto
+  "The readline completer used for Python shell completion.
+If the value is non-nil, Python shell will setup the readline
+completer unless it has been set elsewhere (e.g. in the
+PYTHONSTARTUP file).  Below are possible values:
+- `auto': the completer is determined according to the
+interpreter.  Specifically, the IPython completer, defined in
+`python-shell-readline-ipython-setup-code', is used when the
+interpreter is ipython, otherwise the Jedi completer is used.
+- a string: Python code to setup the readline. It should define
+the function `__PYTHON_EL_setup_readline_completer'.  See
+`python-shell-readline-jedi-setup-code' for reference.
+- `nil': Python shell will do nothing.
+
+In any case, if the completer is not set successfully in the end,
+fallback to the built-in rlcompleter."
+  :type '(choice (const  :tag "Automatic" auto)
+                 (const  :tag "No configuration" nil)
+                 (string :tag "Python setup code"))
+  :version "30.1")
+
+(defvar python-shell-readline-completer-delims nil
+  "Word delimiters used by the readline completer.
+It is automatically set by Python shell.")
+
+(defconst python-shell-readline-jedi-setup-code
+  "
+def __PYTHON_EL_setup_readline_completer():
+    from jedi.utils import setup_readline
+    setup_readline()"
+  "Code used to setup readline completer with Jedi.")
+
+(defconst python-shell-readline-ipython-setup-code
+  "
+def __PYTHON_EL_setup_readline_completer():
+    import readline, re, json, IPython
+
+    class __ipython_RL:
+        def __init__(self, v):
+            self.version = v
+
+        def filter(self, prefix, c):
+            if re.match('_+(i?[0-9]+)?$', c):
+                return False
+            elif c[0] == '%' and not re.match('[%a-zA-Z]+$', prefix):
+                return False
+            return True
+
+        def complete(self, text, state):
+            if state == 0:
+                try:
+                    if self.version >= 6:
+                        from IPython.core.completer import provisionalcompleter
+                        with provisionalcompleter():
+                            self.matches = [json.dumps([
+                                [c.text, c.start, c.end, c.type or '?', c.signature or '']
+                                for c in get_ipython().Completer.completions(text, len(text))
+                                if self.filter(text, c.text)])]
+                    else:
+                        part, matches = get_ipython().Completer.complete(line_buffer=text)
+                        self.matches = [text + m[len(part):] for m in matches
+                                        if self.filter(text, m)]
+                except Exception:
+                    pass
+            try:
+                return self.matches[state]
+            except IndexError:
+                return None
+
+    readline.set_completer(__ipython_RL(IPython.version_info[0]).complete)
+    readline.set_completer_delims('')"
+  "Code used to setup readline completer for IPython.")
+
 (defvar python-shell-completion-native-redirect-buffer
   " *Python completions redirect*"
   "Buffer to be used to redirect output of readline commands.")
@@ -4427,7 +4513,20 @@ python-shell-completion-native-try
 (defun python-shell-completion-native-setup ()
   "Try to setup native completion, return non-nil on success."
   (let* ((process (python-shell-get-process))
-         (output (python-shell-send-string-no-output "
+         (completer (pcase python-shell-readline-completer
+                      ('auto
+                       (if (string-match-p "ipython[23]?\\'" python-shell-interpreter)
+                           python-shell-readline-ipython-setup-code
+                         python-shell-readline-jedi-setup-code))
+                      ((pred stringp) python-shell-readline-completer)
+                      (_ "")))
+         (output (python-shell-send-string-no-output
+                  (concat "
+try:
+    del __PYTHON_EL_setup_readline_completer
+except:
+    pass
+" completer "
 def __PYTHON_EL_native_completion_setup():
     try:
         import readline
@@ -4503,8 +4602,11 @@ python-shell-completion-native-setup
         completer = readline.get_completer()
 
         if not completer:
-            # Used as last resort to avoid breaking customizations.
-            import rlcompleter
+            try:
+                __PYTHON_EL_setup_readline_completer()
+            except:
+                # Used as last resort to avoid breaking customizations.
+                import rlcompleter
             completer = readline.get_completer()
 
         if completer and not getattr(completer, 'PYTHON_EL_WRAPPED', False):
@@ -4539,9 +4641,13 @@ python-shell-completion-native-setup
         print ('python.el: native completion setup failed, %s: %s'
                % sys.exc_info()[:2])
 
-__PYTHON_EL_native_completion_setup()" process)))
+__PYTHON_EL_native_completion_setup()") process)))
     (when (string-match-p "python\\.el: native completion setup loaded"
                           output)
+      (setq-local python-shell-readline-completer-delims
+                  (string-trim-right
+                   (python-shell-send-string-no-output
+                    "import readline; print(readline.get_completer_delims())")))
       (python-shell-completion-native-try))))
 
 (defun python-shell-completion-native-turn-off (&optional msg)
@@ -4609,6 +4715,8 @@ python-shell-completion-native-get-completions
     (let* ((original-filter-fn (process-filter process))
            (redirect-buffer (get-buffer-create
                              python-shell-completion-native-redirect-buffer))
+           (sep (if (string= python-shell-readline-completer-delims "")
+                    "[\n\r]+" "[ \f\t\n\r\v()]+"))
            (trigger "\t")
            (new-input (concat input trigger))
            (input-length
@@ -4651,28 +4759,26 @@ python-shell-completion-native-get-completions
                      process python-shell-completion-native-output-timeout
                      comint-redirect-finished-regexp)
                 (re-search-backward "0__dummy_completion__" nil t)
-                (cl-remove-duplicates
-                 (split-string
-                  (buffer-substring-no-properties
-                   (line-beginning-position) (point-min))
-                  "[ \f\t\n\r\v()]+" t)
-                 :test #'string=))))
+                (let ((str (buffer-substring-no-properties
+                            (line-beginning-position) (point-min))))
+                  (if (string= "[" (substring str 0 1))
+                      (condition-case nil
+                          (python--parse-json-array str)
+                        (t (cl-remove-duplicates (split-string str sep t)
+                                                 :test #'string=)))
+                    (cl-remove-duplicates (split-string str sep t)
+                                          :test #'string=))))))
         (set-process-filter process original-filter-fn)))))
 
 (defun python-shell-completion-get-completions (process input)
   "Get completions of INPUT using PROCESS."
   (with-current-buffer (process-buffer process)
-    (let ((completions
-           (python-util-strip-string
-            (python-shell-send-string-no-output
-             (format
-              "%s\nprint(';'.join(__PYTHON_EL_get_completions(%s)))"
+    (python--parse-json-array
+     (python-shell-send-string-no-output
+      (format "%s\nprint(__PYTHON_EL_get_completions(%s))"
               python-shell-completion-setup-code
               (python-shell--encode-string input))
-             process))))
-      (when (> (length completions) 2)
-        (split-string completions
-                      "^'\\|^\"\\|;\\|'$\\|\"$" t)))))
+      process))))
 
 (defvar-local python-shell--capf-cache nil
   "Variable to store cached completions and invalidation keys.")
@@ -4687,21 +4793,21 @@ python-shell-completion-at-point
                          ;; Working on a shell buffer: use prompt end.
                          (cdr (python-util-comint-last-prompt))
                        (line-beginning-position)))
-         (import-statement
-          (when (string-match-p
-                 (rx (* space) word-start (or "from" "import") word-end space)
-                 (buffer-substring-no-properties line-start (point)))
-            (buffer-substring-no-properties line-start (point))))
+         (no-delims (with-current-buffer (process-buffer process)
+                      (if python-shell-completion-native-enable
+                          (string= python-shell-readline-completer-delims "")
+                        (string-match-p "ipython[23]?\\'" python-shell-interpreter))))
          (start
           (if (< (point) line-start)
               (point)
             (save-excursion
-              (if (not (re-search-backward
-                        (python-rx
-                         (or whitespace open-paren close-paren
-                             string-delimiter simple-operator))
-                        line-start
-                        t 1))
+              (if (or no-delims
+                      (not (re-search-backward
+                            (python-rx
+                             (or whitespace open-paren close-paren
+                                 string-delimiter simple-operator))
+                            line-start
+                            t 1)))
                   line-start
                 (forward-char (length (match-string-no-properties 0)))
                 (point)))))
@@ -4741,18 +4847,44 @@ python-shell-completion-at-point
                   (t #'python-shell-completion-native-get-completions))))
          (prev-prompt (car python-shell--capf-cache))
          (re (or (cadr python-shell--capf-cache) regexp-unmatchable))
-         (prefix (buffer-substring-no-properties start end)))
+         (prefix (buffer-substring-no-properties start end))
+         (prefix-offset 0))
     ;; To invalidate the cache, we check if the prompt position or the
     ;; completion prefix changed.
     (unless (and (equal prev-prompt (car prompt-boundaries))
-                 (string-match re prefix))
+                 (string-match re prefix)
+                 (setq prefix-offset (- (length prefix) (match-end 1))))
       (setq python-shell--capf-cache
             `(,(car prompt-boundaries)
               ,(if (string-empty-p prefix)
                    regexp-unmatchable
-                 (concat "\\`" (regexp-quote prefix) "\\(?:\\sw\\|\\s_\\)*\\'"))
-              ,@(funcall completion-fn process (or import-statement prefix)))))
-    (list start end (cddr python-shell--capf-cache))))
+                 (concat "\\`\\(" (regexp-quote prefix) "\\)\\(?:\\sw\\|\\s_\\)*\\'"))
+              ,@(funcall completion-fn process prefix))))
+    (let ((cands (cddr python-shell--capf-cache)))
+      (cond
+       ((stringp (car cands))
+        (if no-delims
+            ;; Reduce completion candidates due to long prefix.
+            (if-let ((Lp (length prefix))
+                     ((string-match "\\(\\sw\\|\\s_\\)+\\'" prefix))
+                     (L (match-beginning 0)))
+                (list (+ start L) end (mapcar (lambda (s) (substring s L)) cands))
+              (list end end (mapcar (lambda (s) (substring s Lp)) cands)))
+          (list start end cands)))
+       ;; python-shell-completion(-native)-get-completions may produce
+       ;; a list of (text start end type signature) for completion.
+       ;; See `python-shell-readline-ipython-setup-code' and
+       ;; `python-shell-completion-setup-code'.
+       ((consp (car cands))
+        (list (+ start (nth 1 (car cands)))
+              ;; Candidates may be cached, so the end position should
+              ;; be adjusted according to current completion prefix.
+              (+ start (nth 2 (car cands)) prefix-offset)
+              cands
+              :annotation-function
+              (lambda (c) (concat " " (nth 3 (assoc c cands))))
+              :company-docsig
+              (lambda (c) (nth 4 (assoc c cands)))))))))
 
 (define-obsolete-function-alias
   'python-shell-completion-complete-at-point
diff --git a/test/lisp/progmodes/python-tests.el b/test/lisp/progmodes/python-tests.el
index 97ffd5fe20f..975baf9e576 100644
--- a/test/lisp/progmodes/python-tests.el
+++ b/test/lisp/progmodes/python-tests.el
@@ -4787,6 +4787,72 @@ python-shell-completion-at-point-native-1
      (end-of-line 0)
      (should-not (nth 2 (python-shell-completion-at-point))))))
 
+(defun python-shell-completion-module ()
+  "Check if modules can be completed in Python shell."
+  (insert "import datet")
+  (completion-at-point)
+  (beginning-of-line)
+  (should (looking-at-p "import datetime"))
+  (kill-line)
+  (insert "from datet")
+  (completion-at-point)
+  (beginning-of-line)
+  (should (looking-at-p "from datetime"))
+  (end-of-line)
+  (insert " import timed")
+  (completion-at-point)
+  (beginning-of-line)
+  (should (looking-at-p "from datetime import timedelta"))
+  (kill-line))
+
+(defun python-shell-completion-parameters ()
+  "Check if parameters can be completed in Python shell."
+  (insert "import re")
+  (comint-send-input)
+  (python-tests-shell-wait-for-prompt)
+  (insert "re.split('b', 'abc', maxs")
+  (completion-at-point)
+  (should (string= "re.split('b', 'abc', maxsplit="
+                   (buffer-substring (line-beginning-position) (point))))
+  (insert "0, ")
+  (should (python-shell-completion-at-point))
+  ;; Test if cache is used.
+  (cl-letf (((symbol-function 'python-shell-completion-get-completions)
+             'ignore)
+            ((symbol-function 'python-shell-completion-native-get-completions)
+             'ignore))
+    (insert "fla")
+    (completion-at-point)
+    (should (string= "re.split('b', 'abc', maxsplit=0, flags="
+                     (buffer-substring (line-beginning-position) (point)))))
+  (beginning-of-line)
+  (kill-line))
+
+(ert-deftest python-shell-completion-at-point-jedi-completer ()
+  "Check if Python shell completion works with Jedi."
+  (skip-unless (executable-find python-tests-shell-interpreter))
+  (python-tests-with-temp-buffer-with-shell
+   ""
+   (python-shell-with-shell-buffer
+     (python-shell-completion-native-turn-on)
+     (skip-unless (string= python-shell-readline-completer-delims ""))
+     (python-shell-completion-module)
+     (python-shell-completion-parameters))))
+
+(ert-deftest python-shell-completion-at-point-ipython ()
+  "Check if Python shell completion works for IPython."
+  (let ((python-shell-interpreter "ipython")
+        (python-shell-interpreter-args "-i --simple-prompt"))
+    (skip-unless (executable-find python-shell-interpreter))
+    (python-tests-with-temp-buffer-with-shell
+     ""
+     (python-shell-with-shell-buffer
+       (python-shell-completion-native-turn-off)
+       (python-shell-completion-module)
+       (python-shell-completion-parameters)
+       (python-shell-completion-native-turn-on)
+       (python-shell-completion-module)
+       (python-shell-completion-parameters)))))
 
 \f
 ;;; PDB Track integration
-- 
2.25.1

bug#68559: [PATCH] Improve Python shell completion

[Eli Zaretskii]

> From: Liu Hui <liuhui1610@gmail.com>
> Date: Thu, 18 Jan 2024 12:48:37 +0800
> 
> Currently python-shell-completion-at-point doesn't respect the
> delimiter setting of readline completer and always split the text,
> resulting in poor completions when completers that don't need word
> splitting, e.g. jedi[1], is used. This patch fixes the problem.
> 
> Meanwhile, this patch adds an option 'python-shell-readline-completer'
> to make Python shell provide better completion experience (e.g.
> completing function parameters, dictionary keys) by default if users
> are using vanilla Python interpreter with jedi or using IPython, where
> the completion support for IPython has been enhanced (e.g. showing
> type annotations and function signatures).
> 
> BTW, I think it may make sense to move Python snippets to a separate
> file to make them easier to be maintained.

Thanks.  Let's see what others think about this (I don't use Python
enough to have an opinion that matters).

bug#68559: [PATCH] Improve Python shell completion

[kobarity]

Eli Zaretskii wrote:
> 
> > From: Liu Hui <liuhui1610@gmail.com>
> > Date: Thu, 18 Jan 2024 12:48:37 +0800
> > 
> > Currently python-shell-completion-at-point doesn't respect the
> > delimiter setting of readline completer and always split the text,
> > resulting in poor completions when completers that don't need word
> > splitting, e.g. jedi[1], is used. This patch fixes the problem.
> > 
> > Meanwhile, this patch adds an option 'python-shell-readline-completer'
> > to make Python shell provide better completion experience (e.g.
> > completing function parameters, dictionary keys) by default if users
> > are using vanilla Python interpreter with jedi or using IPython, where
> > the completion support for IPython has been enhanced (e.g. showing
> > type annotations and function signatures).
> > 
> > BTW, I think it may make sense to move Python snippets to a separate
> > file to make them easier to be maintained.
> 
> Thanks.  Let's see what others think about this (I don't use Python
> enough to have an opinion that matters).

Thanks for the patch.  I have tried it and found it to be a very
powerful enhancement.

Is it possible to allow completion of keyword arguments in a
multi-line function call like the following?

#+begin_src python
re.split(
    'b',
    'abc',
    maxs
#+end_src

I am not sure if Python snippets should be separated.  Do other
language supports keep them separate?

bug#68559: [PATCH] Improve Python shell completion

[Liu Hui]

[-- Attachment #1: Type: text/plain, Size: 754 bytes --]

kobarity <kobarity@gmail.com> 于2024年1月21日周日 17:34写道：

> Is it possible to allow completion of keyword arguments in a
> multi-line function call like the following?
>
> #+begin_src python
> re.split(
>     'b',
>     'abc',
>     maxs
> #+end_src

I have added experimental support in the attached patch, can you test
it please?

> I am not sure if Python snippets should be separated.  Do other
> language supports keep them separate?

I think it allows to reduce code duplication (e.g. the IPython support
code existing in both python-shell-completion-setup-code and
python-shell-readline-ipython-setup-code) and make code more simple.
It seems other languages do not contain as many code snippets as
python.el.

[-- Attachment #2: 0001-Improve-Python-shell-completion.patch --]
[-- Type: text/x-patch, Size: 24720 bytes --]

From 17be2968b6c53dd9fe323b8c28c3ce45efc63318 Mon Sep 17 00:00:00 2001
From: Liu Hui <liuhui1610@gmail.com>
Date: Thu, 18 Jan 2024 12:00:00 +0800
Subject: [PATCH] Improve Python shell completion

* lisp/progmodes/python.el (python-shell-completion-setup-code): Fix
the completion code of IPython.  Change the return value to JSON
string and ...
(python-shell-completion-get-completions): ... simplify parsing.
(inferior-python-mode): Update docstring.
(python-shell-readline-completer): New option.
(python-shell-readline-completer-delims)
(python-shell-readline-jedi-setup-code)
(python-shell-readline-ipython-setup-code): New variables.
(python-shell-completion-native-setup): Setup a suitable readline
completer and set the completer delimiter.
(python-shell-completion-native-get-completions): Convert output
string to completions properly.
(python-shell--get-multiline-input):
(python-shell--extra-completion-context): New functions.
(python-shell-completion-at-point): Send text beginning from the line
start if the completion backend does not need word splitting.  Remove
the detection of import statement because it is not needed anymore.
Create proper completion table based on completions returned from
different backends.

* test/lisp/progmodes/python-tests.el (python-shell-completion-module):
(python-shell-completion-parameters):
(python-shell-completion-multi-line-function-call): New functions.
(python-shell-completion-at-point-jedi-completer):
(python-shell-completion-at-point-ipython): New tests.

* etc/NEWS: Announce the change.
---
 etc/NEWS                            |   6 +
 lisp/progmodes/python.el            | 292 +++++++++++++++++++++++-----
 test/lisp/progmodes/python-tests.el |  78 ++++++++
 3 files changed, 324 insertions(+), 52 deletions(-)

diff --git a/etc/NEWS b/etc/NEWS
index 03b8c3b517a..6fd337727c5 100644
--- a/etc/NEWS
+++ b/etc/NEWS
@@ -1029,6 +1029,12 @@ instead of:
 This allows the user to specify command line arguments to the non
 interactive Python interpreter specified by 'python-interpreter'.
 
+*** New user option 'python-shell-readline-completer'.
+This allows the user to specify the readline completer used for Python
+shell completion.  The default is 'auto', which means a suitable
+completer will be configured automatically according to the Python
+interpreter.
+
 ** use-package
 
 +++
diff --git a/lisp/progmodes/python.el b/lisp/progmodes/python.el
index ff799e1e662..f54b1b29104 100644
--- a/lisp/progmodes/python.el
+++ b/lisp/progmodes/python.el
@@ -128,9 +128,9 @@
 ;; receiving escape sequences (with some limitations, i.e. completion
 ;; in blocks does not work).  The code executed for the "fallback"
 ;; completion can be found in `python-shell-completion-setup-code' and
-;; `python-shell-completion-string-code' variables.  Their default
-;; values enable completion for both CPython and IPython, and probably
-;; any readline based shell (it's known to work with PyPy).  If your
+;; `python-shell-completion-get-completions'.  Their default values
+;; enable completion for both CPython and IPython, and probably any
+;; readline based shell (it's known to work with PyPy).  If your
 ;; Python installation lacks readline (like CPython for Windows),
 ;; installing pyreadline (URL `https://ipython.org/pyreadline.html')
 ;; should suffice.  To troubleshoot why you are not getting any
@@ -3601,7 +3601,6 @@ inferior-python-mode
 `python-shell-prompt-block-regexp',
 `python-shell-font-lock-enable',
 `python-shell-completion-setup-code',
-`python-shell-completion-string-code',
 `python-eldoc-setup-code',
 `python-ffap-setup-code' can
 customize this mode for different Python interpreters.
@@ -4319,8 +4318,9 @@ python-shell-completion-setup-code
     completions = []
     completer = None
 
+    import json
     try:
-        import readline
+        import readline, re
 
         try:
             import __builtin__
@@ -4331,16 +4331,29 @@ python-shell-completion-setup-code
 
         is_ipython = ('__IPYTHON__' in builtins or
                       '__IPYTHON__active' in builtins)
-        splits = text.split()
-        is_module = splits and splits[0] in ('from', 'import')
-
-        if is_ipython and is_module:
-            from IPython.core.completerlib import module_completion
-            completions = module_completion(text.strip())
-        elif is_ipython and '__IP' in builtins:
-            completions = __IP.complete(text)
-        elif is_ipython and 'get_ipython' in builtins:
-            completions = get_ipython().Completer.all_completions(text)
+
+        if is_ipython and 'get_ipython' in builtins:
+            def filter_c(prefix, c):
+                if re.match('_+(i?[0-9]+)?$', c):
+                    return False
+                elif c[0] == '%' and not re.match('[%a-zA-Z]+$', prefix):
+                    return False
+                return True
+
+            import IPython
+            try:
+                if IPython.version_info[0] >= 6:
+                    from IPython.core.completer import provisionalcompleter
+                    with provisionalcompleter():
+                        completions = [
+                            [c.text, c.start, c.end, c.type or '?', c.signature or '']
+                             for c in get_ipython().Completer.completions(text, len(text))
+                             if filter_c(text, c.text)]
+                else:
+                    part, matches = get_ipython().Completer.complete(line_buffer=text)
+                    completions = [text + m[len(part):] for m in matches if filter_c(text, m)]
+            except:
+                pass
         else:
             # Try to reuse current completer.
             completer = readline.get_completer()
@@ -4363,7 +4376,7 @@ python-shell-completion-setup-code
     finally:
         if getattr(completer, 'PYTHON_EL_WRAPPED', False):
             completer.print_mode = True
-    return completions"
+    return json.dumps(completions)"
   "Code used to setup completion in inferior Python processes."
   :type 'string)
 
@@ -4404,6 +4417,79 @@ python-shell-completion-native-try-output-timeout
   :version "25.1"
   :type 'float)
 
+(defcustom python-shell-readline-completer 'auto
+  "The readline completer used for Python shell completion.
+If the value is non-nil, Python shell will setup the readline
+completer unless it has been set elsewhere (e.g. in the
+PYTHONSTARTUP file).  Below are possible values:
+- `auto': the completer is determined according to the
+interpreter.  Specifically, the IPython completer, defined in
+`python-shell-readline-ipython-setup-code', is used when the
+interpreter is ipython, otherwise the Jedi completer is used.
+- a string: Python code to setup the readline. It should define
+the function `__PYTHON_EL_setup_readline_completer'.  See
+`python-shell-readline-jedi-setup-code' for reference.
+- `nil': Python shell will do nothing.
+
+In any case, if the completer is not set successfully in the end,
+fallback to the built-in rlcompleter."
+  :type '(choice (const  :tag "Automatic" auto)
+                 (const  :tag "No configuration" nil)
+                 (string :tag "Python setup code"))
+  :version "30.1")
+
+(defvar python-shell-readline-completer-delims nil
+  "Word delimiters used by the readline completer.
+It is automatically set by Python shell.")
+
+(defconst python-shell-readline-jedi-setup-code
+  "
+def __PYTHON_EL_setup_readline_completer():
+    from jedi.utils import setup_readline
+    setup_readline()"
+  "Code used to setup readline completer with Jedi.")
+
+(defconst python-shell-readline-ipython-setup-code
+  "
+def __PYTHON_EL_setup_readline_completer():
+    import readline, re, json, IPython
+
+    class __ipython_RL:
+        def __init__(self, v):
+            self.version = v
+
+        def filter(self, prefix, c):
+            if re.match('_+(i?[0-9]+)?$', c):
+                return False
+            elif c[0] == '%' and not re.match('[%a-zA-Z]+$', prefix):
+                return False
+            return True
+
+        def complete(self, text, state):
+            if state == 0:
+                try:
+                    if self.version >= 6:
+                        from IPython.core.completer import provisionalcompleter
+                        with provisionalcompleter():
+                            self.matches = [json.dumps([
+                                [c.text, c.start, c.end, c.type or '?', c.signature or '']
+                                for c in get_ipython().Completer.completions(text, len(text))
+                                if self.filter(text, c.text)])]
+                    else:
+                        part, matches = get_ipython().Completer.complete(line_buffer=text)
+                        self.matches = [text + m[len(part):] for m in matches
+                                        if self.filter(text, m)]
+                except Exception:
+                    pass
+            try:
+                return self.matches[state]
+            except IndexError:
+                return None
+
+    readline.set_completer(__ipython_RL(IPython.version_info[0]).complete)
+    readline.set_completer_delims('')"
+  "Code used to setup readline completer for IPython.")
+
 (defvar python-shell-completion-native-redirect-buffer
   " *Python completions redirect*"
   "Buffer to be used to redirect output of readline commands.")
@@ -4427,7 +4513,20 @@ python-shell-completion-native-try
 (defun python-shell-completion-native-setup ()
   "Try to setup native completion, return non-nil on success."
   (let* ((process (python-shell-get-process))
-         (output (python-shell-send-string-no-output "
+         (completer (pcase python-shell-readline-completer
+                      ('auto
+                       (if (string-match-p "ipython[23]?\\'" python-shell-interpreter)
+                           python-shell-readline-ipython-setup-code
+                         python-shell-readline-jedi-setup-code))
+                      ((pred stringp) python-shell-readline-completer)
+                      (_ "")))
+         (output (python-shell-send-string-no-output
+                  (concat "
+try:
+    del __PYTHON_EL_setup_readline_completer
+except:
+    pass
+" completer "
 def __PYTHON_EL_native_completion_setup():
     try:
         import readline
@@ -4500,11 +4599,23 @@ python-shell-completion-native-setup
                 else:
                     return completion
 
+        def is_rlcompleter(completer):
+            try:
+                if completer.__self__.__module__ == 'rlcompleter':
+                    return True
+                else:
+                    return False
+            except Exception:
+                return False
+
         completer = readline.get_completer()
 
-        if not completer:
-            # Used as last resort to avoid breaking customizations.
-            import rlcompleter
+        if not completer or is_rlcompleter(completer):
+            try:
+                __PYTHON_EL_setup_readline_completer()
+            except:
+                # Used as last resort to avoid breaking customizations.
+                import rlcompleter
             completer = readline.get_completer()
 
         if completer and not getattr(completer, 'PYTHON_EL_WRAPPED', False):
@@ -4539,9 +4650,13 @@ python-shell-completion-native-setup
         print ('python.el: native completion setup failed, %s: %s'
                % sys.exc_info()[:2])
 
-__PYTHON_EL_native_completion_setup()" process)))
+__PYTHON_EL_native_completion_setup()") process)))
     (when (string-match-p "python\\.el: native completion setup loaded"
                           output)
+      (setq-local python-shell-readline-completer-delims
+                  (string-trim-right
+                   (python-shell-send-string-no-output
+                    "import readline; print(readline.get_completer_delims())")))
       (python-shell-completion-native-try))))
 
 (defun python-shell-completion-native-turn-off (&optional msg)
@@ -4609,6 +4724,8 @@ python-shell-completion-native-get-completions
     (let* ((original-filter-fn (process-filter process))
            (redirect-buffer (get-buffer-create
                              python-shell-completion-native-redirect-buffer))
+           (sep (if (string= python-shell-readline-completer-delims "")
+                    "[\n\r]+" "[ \f\t\n\r\v()]+"))
            (trigger "\t")
            (new-input (concat input trigger))
            (input-length
@@ -4651,32 +4768,63 @@ python-shell-completion-native-get-completions
                      process python-shell-completion-native-output-timeout
                      comint-redirect-finished-regexp)
                 (re-search-backward "0__dummy_completion__" nil t)
-                (cl-remove-duplicates
-                 (split-string
-                  (buffer-substring-no-properties
-                   (line-beginning-position) (point-min))
-                  "[ \f\t\n\r\v()]+" t)
-                 :test #'string=))))
+                (let ((str (buffer-substring-no-properties
+                            (line-beginning-position) (point-min))))
+                  (if (string= "[" (substring str 0 1))
+                      (condition-case nil
+                          (python--parse-json-array str)
+                        (t (cl-remove-duplicates (split-string str sep t)
+                                                 :test #'string=)))
+                    (cl-remove-duplicates (split-string str sep t)
+                                          :test #'string=))))))
         (set-process-filter process original-filter-fn)))))
 
 (defun python-shell-completion-get-completions (process input)
   "Get completions of INPUT using PROCESS."
   (with-current-buffer (process-buffer process)
-    (let ((completions
-           (python-util-strip-string
-            (python-shell-send-string-no-output
-             (format
-              "%s\nprint(';'.join(__PYTHON_EL_get_completions(%s)))"
+    (python--parse-json-array
+     (python-shell-send-string-no-output
+      (format "%s\nprint(__PYTHON_EL_get_completions(%s))"
               python-shell-completion-setup-code
               (python-shell--encode-string input))
-             process))))
-      (when (> (length completions) 2)
-        (split-string completions
-                      "^'\\|^\"\\|;\\|'$\\|\"$" t)))))
+      process))))
 
 (defvar-local python-shell--capf-cache nil
   "Variable to store cached completions and invalidation keys.")
 
+(defun python-shell--get-multiline-input ()
+  "Return lines at a multi-line input in Python shell."
+  (save-excursion
+    (let ((p (point)) lines)
+      (when (progn
+              (beginning-of-line)
+              (looking-back python-shell-prompt-block-regexp (pos-bol)))
+        (push (buffer-substring-no-properties (point) p) lines)
+        (while (progn (comint-previous-prompt 1)
+                      (looking-back python-shell-prompt-block-regexp (pos-bol)))
+          (push (buffer-substring-no-properties (point) (pos-eol)) lines))
+        (push (buffer-substring-no-properties (point) (pos-eol)) lines))
+      lines)))
+
+(defun python-shell--extra-completion-context ()
+  "Return extra completion context of current line in Python shell.
+It currently supports multi-line function call."
+  (let ((lines (python-shell--get-multiline-input)) beg)
+    (when (not (zerop (length lines)))
+      (with-temp-buffer
+        (delay-mode-hooks
+          (insert (string-join lines "\n"))
+          (python-mode)
+          (setq beg (pos-bol))
+          (python-nav-up-list -1)
+          (when (< (point) beg)
+            (while (re-search-backward
+                    (python-rx symbol-name) (pos-bol) t))
+            (string-replace
+             "\n" "" (buffer-substring-no-properties
+                      (point)
+                      (progn (goto-char (point-max)) (pos-eol 0))))))))))
+
 (defun python-shell-completion-at-point (&optional process)
   "Function for `completion-at-point-functions' in `inferior-python-mode'.
 Optional argument PROCESS forces completions to be retrieved
@@ -4687,21 +4835,21 @@ python-shell-completion-at-point
                          ;; Working on a shell buffer: use prompt end.
                          (cdr (python-util-comint-last-prompt))
                        (line-beginning-position)))
-         (import-statement
-          (when (string-match-p
-                 (rx (* space) word-start (or "from" "import") word-end space)
-                 (buffer-substring-no-properties line-start (point)))
-            (buffer-substring-no-properties line-start (point))))
+         (no-delims (with-current-buffer (process-buffer process)
+                      (if python-shell-completion-native-enable
+                          (string= python-shell-readline-completer-delims "")
+                        (string-match-p "ipython[23]?\\'" python-shell-interpreter))))
          (start
           (if (< (point) line-start)
               (point)
             (save-excursion
-              (if (not (re-search-backward
-                        (python-rx
-                         (or whitespace open-paren close-paren
-                             string-delimiter simple-operator))
-                        line-start
-                        t 1))
+              (if (or no-delims
+                      (not (re-search-backward
+                            (python-rx
+                             (or whitespace open-paren close-paren
+                                 string-delimiter simple-operator))
+                            line-start
+                            t 1)))
                   line-start
                 (forward-char (length (match-string-no-properties 0)))
                 (point)))))
@@ -4741,18 +4889,58 @@ python-shell-completion-at-point
                   (t #'python-shell-completion-native-get-completions))))
          (prev-prompt (car python-shell--capf-cache))
          (re (or (cadr python-shell--capf-cache) regexp-unmatchable))
-         (prefix (buffer-substring-no-properties start end)))
+         (prefix (buffer-substring-no-properties start end))
+         (prefix-offset 0)
+         ;; Send extra context for cases like completing function
+         ;; parameters for multi-line function call.
+         (extra-context (and no-delims
+                             (python-shell--extra-completion-context)))
+         (extra-offset (length extra-context)))
+    (unless (zerop extra-offset)
+      (setq prefix (concat extra-context prefix)))
     ;; To invalidate the cache, we check if the prompt position or the
     ;; completion prefix changed.
     (unless (and (equal prev-prompt (car prompt-boundaries))
-                 (string-match re prefix))
+                 (string-match re prefix)
+                 (setq prefix-offset (- (length prefix) (match-end 1))))
       (setq python-shell--capf-cache
             `(,(car prompt-boundaries)
               ,(if (string-empty-p prefix)
                    regexp-unmatchable
-                 (concat "\\`" (regexp-quote prefix) "\\(?:\\sw\\|\\s_\\)*\\'"))
-              ,@(funcall completion-fn process (or import-statement prefix)))))
-    (list start end (cddr python-shell--capf-cache))))
+                 (concat "\\`\\(" (regexp-quote prefix) "\\)\\(?:\\sw\\|\\s_\\)*\\'"))
+              ,@(funcall completion-fn process prefix))))
+    (let ((cands (cddr python-shell--capf-cache)))
+      (cond
+       ((stringp (car cands))
+        (if no-delims
+            ;; Reduce completion candidates due to long prefix.
+            (if-let ((Lp (length prefix))
+                     ((string-match "\\(\\sw\\|\\s_\\)+\\'" prefix))
+                     (L (match-beginning 0)))
+                ;; If extra-offset is not zero:
+                ;;                  start              end
+                ;; o------------------o---------o-------o
+                ;; |<- extra-offset ->|
+                ;; |<----------- L ------------>|
+                ;;                          new-start
+                (list (+ start L (- extra-offset)) end
+                      (mapcar (lambda (s) (substring s L)) cands))
+              (list end end (mapcar (lambda (s) (substring s Lp)) cands)))
+          (list start end cands)))
+       ;; python-shell-completion(-native)-get-completions may produce
+       ;; a list of (text start end type signature) for completion.
+       ;; See `python-shell-readline-ipython-setup-code' and
+       ;; `python-shell-completion-setup-code'.
+       ((consp (car cands))
+        (list (+ start (nth 1 (car cands)) (- extra-offset))
+              ;; Candidates may be cached, so the end position should
+              ;; be adjusted according to current completion prefix.
+              (+ start (nth 2 (car cands)) (- extra-offset) prefix-offset)
+              cands
+              :annotation-function
+              (lambda (c) (concat " " (nth 3 (assoc c cands))))
+              :company-docsig
+              (lambda (c) (nth 4 (assoc c cands)))))))))
 
 (define-obsolete-function-alias
   'python-shell-completion-complete-at-point
diff --git a/test/lisp/progmodes/python-tests.el b/test/lisp/progmodes/python-tests.el
index 97ffd5fe20f..f6926c421fe 100644
--- a/test/lisp/progmodes/python-tests.el
+++ b/test/lisp/progmodes/python-tests.el
@@ -4787,6 +4787,84 @@ python-shell-completion-at-point-native-1
      (end-of-line 0)
      (should-not (nth 2 (python-shell-completion-at-point))))))
 
+(defun python-shell-completion-module ()
+  "Check if modules can be completed in Python shell."
+  (insert "import datet")
+  (completion-at-point)
+  (beginning-of-line)
+  (should (looking-at-p "import datetime"))
+  (kill-line)
+  (insert "from datet")
+  (completion-at-point)
+  (beginning-of-line)
+  (should (looking-at-p "from datetime"))
+  (end-of-line)
+  (insert " import timed")
+  (completion-at-point)
+  (beginning-of-line)
+  (should (looking-at-p "from datetime import timedelta"))
+  (kill-line))
+
+(defun python-shell-completion-parameters ()
+  "Check if parameters can be completed in Python shell."
+  (insert "import re")
+  (comint-send-input)
+  (python-tests-shell-wait-for-prompt)
+  (insert "re.split('b', 'abc', maxs")
+  (completion-at-point)
+  (should (string= "re.split('b', 'abc', maxsplit="
+                   (buffer-substring (line-beginning-position) (point))))
+  (insert "0, ")
+  (should (python-shell-completion-at-point))
+  ;; Test if cache is used.
+  (cl-letf (((symbol-function 'python-shell-completion-get-completions)
+             'ignore)
+            ((symbol-function 'python-shell-completion-native-get-completions)
+             'ignore))
+    (insert "fla")
+    (completion-at-point)
+    (should (string= "re.split('b', 'abc', maxsplit=0, flags="
+                     (buffer-substring (line-beginning-position) (point)))))
+  (beginning-of-line)
+  (kill-line))
+
+(defun python-shell-completion-multi-line-function-call ()
+  "Check if parameters can be completed in multi-line function call."
+  (insert "re.split('b', 'abc',")
+  (comint-send-input)
+  (python-tests-shell-wait-for-prompt)
+  (insert "maxs")
+  (completion-at-point)
+  (should (string= "maxsplit="
+                   (buffer-substring (line-beginning-position) (point)))))
+
+(ert-deftest python-shell-completion-at-point-jedi-completer ()
+  "Check if Python shell completion works with Jedi."
+  (skip-unless (executable-find python-tests-shell-interpreter))
+  (python-tests-with-temp-buffer-with-shell
+   ""
+   (python-shell-with-shell-buffer
+     (python-shell-completion-native-turn-on)
+     (skip-unless (string= python-shell-readline-completer-delims ""))
+     (python-shell-completion-module)
+     (python-shell-completion-parameters)
+     (python-shell-completion-multi-line-function-call))))
+
+(ert-deftest python-shell-completion-at-point-ipython ()
+  "Check if Python shell completion works for IPython."
+  (let ((python-shell-interpreter "ipython")
+        (python-shell-interpreter-args "-i --simple-prompt"))
+    (skip-unless (executable-find python-shell-interpreter))
+    (python-tests-with-temp-buffer-with-shell
+     ""
+     (python-shell-with-shell-buffer
+       (python-shell-completion-native-turn-off)
+       (python-shell-completion-module)
+       (python-shell-completion-parameters)
+       (python-shell-completion-native-turn-on)
+       (python-shell-completion-module)
+       (python-shell-completion-parameters)
+       (python-shell-completion-multi-line-function-call)))))
 
 \f
 ;;; PDB Track integration
-- 
2.25.1

bug#68559: [PATCH] Improve Python shell completion

[kobarity]

Liu Hui wrote:
> kobarity <kobarity@gmail.com> 于2024年1月21日周日 17:34写道：
> 
> > Is it possible to allow completion of keyword arguments in a
> > multi-line function call like the following?
> >
> > #+begin_src python
> > re.split(
> >     'b',
> >     'abc',
> >     maxs
> > #+end_src
> 
> I have added experimental support in the attached patch, can you test
> it please?

Thank you.  I tried the new patch and confirmed that the Python Shell
buffer allows keyword argument completion on multi-line function
calls.  However, my expectation is that it can be done in Python
buffer as well (after calling `python-shell-send-buffer').  Is this
also possible?

> > I am not sure if Python snippets should be separated.  Do other
> > language supports keep them separate?
> 
> I think it allows to reduce code duplication (e.g. the IPython support
> code existing in both python-shell-completion-setup-code and
> python-shell-readline-ipython-setup-code) and make code more simple.
> It seems other languages do not contain as many code snippets as
> python.el.

Sorry, I misunderstood that snippets meant skeletons.  I agree that
python.el has long Python codes, which is not easy to maintain.  Are
you proposing to separate the snippets into .py file?  If it's
acceptable from the Emacs' source code management perspective, I think
that is a good approach.

bug#68559: [PATCH] Improve Python shell completion

[Liu Hui]

[-- Attachment #1: Type: text/plain, Size: 1438 bytes --]

kobarity <kobarity@gmail.com> 于2024年1月23日周二 22:15写道：

> Thank you.  I tried the new patch and confirmed that the Python Shell
> buffer allows keyword argument completion on multi-line function
> calls.  However, my expectation is that it can be done in Python
> buffer as well (after calling `python-shell-send-buffer').  Is this
> also possible?

Thank you for the feedback. I have updated the patch to add support in
Python buffer.

> > > I am not sure if Python snippets should be separated.  Do other
> > > language supports keep them separate?
> >
> > I think it allows to reduce code duplication (e.g. the IPython support
> > code existing in both python-shell-completion-setup-code and
> > python-shell-readline-ipython-setup-code) and make code more simple.
> > It seems other languages do not contain as many code snippets as
> > python.el.
>
> Sorry, I misunderstood that snippets meant skeletons.  I agree that
> python.el has long Python codes, which is not easy to maintain.  Are
> you proposing to separate the snippets into .py file?  If it's
> acceptable from the Emacs' source code management perspective, I think
> that is a good approach.

Yes, I'd like to at least put some flags and functions shared by
python-shell-completion-native-setup and python-*-setup-code in a
separate .py file. On the other hand, they are not changed frequently,
so it is not in urgent need.

[-- Attachment #2: 0001-Improve-Python-shell-completion-bug-68559.patch --]
[-- Type: text/x-patch, Size: 24987 bytes --]

From abefec11d52875a76703c25c0998b5d775442a7a Mon Sep 17 00:00:00 2001
From: Liu Hui <liuhui1610@gmail.com>
Date: Thu, 18 Jan 2024 12:00:00 +0800
Subject: [PATCH] Improve Python shell completion (bug#68559)

* lisp/progmodes/python.el (python-shell-completion-setup-code): Fix
the completion code of IPython.  Change the return value to JSON
string and ...
(python-shell-completion-get-completions): ... simplify parsing.
(inferior-python-mode): Update docstring.
(python-shell-readline-completer): New option.
(python-shell-readline-completer-delims)
(python-shell--readline-jedi-setup-code)
(python-shell--readline-ipython-setup-code): New variables.
(python-shell-completion-native-setup): Setup a suitable readline
completer and set the completer delimiter.
(python-shell-completion-native-get-completions): Convert output
string to completions properly.
(python-shell--get-multiline-input)
(python-shell--extra-completion-context)
(python-shell-completion-extra-context): New functions.
(python-shell-completion-at-point): Send text beginning from the line
start if the completion backend does not need word splitting.  Remove
the detection of import statement because it is not needed anymore.
Create proper completion table based on completions returned from
different backends.

* test/lisp/progmodes/python-tests.el (python-tests--completion-module):
(python-tests--completion-parameters):
(python-tests--completion-extra-context): New helper functions.
(python-shell-completion-at-point-jedi-completer):
(python-shell-completion-at-point-ipython): New tests.

* etc/NEWS: Announce the change.
---
 etc/NEWS                            |   6 +
 lisp/progmodes/python.el            | 304 +++++++++++++++++++++++-----
 test/lisp/progmodes/python-tests.el |  78 +++++++
 3 files changed, 336 insertions(+), 52 deletions(-)

diff --git a/etc/NEWS b/etc/NEWS
index 03b8c3b517a..6fd337727c5 100644
--- a/etc/NEWS
+++ b/etc/NEWS
@@ -1029,6 +1029,12 @@ instead of:
 This allows the user to specify command line arguments to the non
 interactive Python interpreter specified by 'python-interpreter'.
 
+*** New user option 'python-shell-readline-completer'.
+This allows the user to specify the readline completer used for Python
+shell completion.  The default is 'auto', which means a suitable
+completer will be configured automatically according to the Python
+interpreter.
+
 ** use-package
 
 +++
diff --git a/lisp/progmodes/python.el b/lisp/progmodes/python.el
index ff799e1e662..63445ab0e50 100644
--- a/lisp/progmodes/python.el
+++ b/lisp/progmodes/python.el
@@ -128,9 +128,9 @@
 ;; receiving escape sequences (with some limitations, i.e. completion
 ;; in blocks does not work).  The code executed for the "fallback"
 ;; completion can be found in `python-shell-completion-setup-code' and
-;; `python-shell-completion-string-code' variables.  Their default
-;; values enable completion for both CPython and IPython, and probably
-;; any readline based shell (it's known to work with PyPy).  If your
+;; `python-shell-completion-get-completions'.  Their default values
+;; enable completion for both CPython and IPython, and probably any
+;; readline based shell (it's known to work with PyPy).  If your
 ;; Python installation lacks readline (like CPython for Windows),
 ;; installing pyreadline (URL `https://ipython.org/pyreadline.html')
 ;; should suffice.  To troubleshoot why you are not getting any
@@ -3601,7 +3601,6 @@ inferior-python-mode
 `python-shell-prompt-block-regexp',
 `python-shell-font-lock-enable',
 `python-shell-completion-setup-code',
-`python-shell-completion-string-code',
 `python-eldoc-setup-code',
 `python-ffap-setup-code' can
 customize this mode for different Python interpreters.
@@ -4319,8 +4318,9 @@ python-shell-completion-setup-code
     completions = []
     completer = None
 
+    import json
     try:
-        import readline
+        import readline, re
 
         try:
             import __builtin__
@@ -4331,16 +4331,29 @@ python-shell-completion-setup-code
 
         is_ipython = ('__IPYTHON__' in builtins or
                       '__IPYTHON__active' in builtins)
-        splits = text.split()
-        is_module = splits and splits[0] in ('from', 'import')
-
-        if is_ipython and is_module:
-            from IPython.core.completerlib import module_completion
-            completions = module_completion(text.strip())
-        elif is_ipython and '__IP' in builtins:
-            completions = __IP.complete(text)
-        elif is_ipython and 'get_ipython' in builtins:
-            completions = get_ipython().Completer.all_completions(text)
+
+        if is_ipython and 'get_ipython' in builtins:
+            def filter_c(prefix, c):
+                if re.match('_+(i?[0-9]+)?$', c):
+                    return False
+                elif c[0] == '%' and not re.match('[%a-zA-Z]+$', prefix):
+                    return False
+                return True
+
+            import IPython
+            try:
+                if IPython.version_info[0] >= 6:
+                    from IPython.core.completer import provisionalcompleter
+                    with provisionalcompleter():
+                        completions = [
+                            [c.text, c.start, c.end, c.type or '?', c.signature or '']
+                             for c in get_ipython().Completer.completions(text, len(text))
+                             if filter_c(text, c.text)]
+                else:
+                    part, matches = get_ipython().Completer.complete(line_buffer=text)
+                    completions = [text + m[len(part):] for m in matches if filter_c(text, m)]
+            except:
+                pass
         else:
             # Try to reuse current completer.
             completer = readline.get_completer()
@@ -4363,7 +4376,7 @@ python-shell-completion-setup-code
     finally:
         if getattr(completer, 'PYTHON_EL_WRAPPED', False):
             completer.print_mode = True
-    return completions"
+    return json.dumps(completions)"
   "Code used to setup completion in inferior Python processes."
   :type 'string)
 
@@ -4404,6 +4417,78 @@ python-shell-completion-native-try-output-timeout
   :version "25.1"
   :type 'float)
 
+(defcustom python-shell-readline-completer 'auto
+  "The readline completer used for Python shell completion.
+If the value is non-nil, Python shell will setup the readline
+completer unless it has been set elsewhere (e.g. in the
+PYTHONSTARTUP file).  Below are possible values:
+- `auto': the completer is determined according to the
+interpreter.  Specifically, the IPython completer, defined in
+`python-shell--readline-ipython-setup-code', is used when the
+interpreter is ipython, otherwise the Jedi completer is used.
+- a string: Python code to setup the readline.  See
+`python-shell--readline-jedi-setup-code' for reference.
+- `nil': Python shell will do nothing.
+
+In any case, if the completer is not set successfully in the end,
+fallback to the built-in rlcompleter."
+  :type '(choice (const  :tag "Automatic" auto)
+                 (const  :tag "No configuration" nil)
+                 (string :tag "Python setup code"))
+  :version "30.1")
+
+(defvar python-shell-readline-completer-delims nil
+  "Word delimiters used by the readline completer.
+It is automatically set by Python shell.")
+
+(defconst python-shell--readline-jedi-setup-code
+  "
+def __PYTHON_EL_setup_readline_completer():
+    from jedi.utils import setup_readline
+    setup_readline()"
+  "Code used to setup readline completer with Jedi.")
+
+(defconst python-shell--readline-ipython-setup-code
+  "
+def __PYTHON_EL_setup_readline_completer():
+    import readline, re, json, IPython
+
+    class __ipython_RL:
+        def __init__(self, v):
+            self.version = v
+
+        def filter(self, prefix, c):
+            if re.match('_+(i?[0-9]+)?$', c):
+                return False
+            elif c[0] == '%' and not re.match('[%a-zA-Z]+$', prefix):
+                return False
+            return True
+
+        def complete(self, text, state):
+            if state == 0:
+                try:
+                    if self.version >= 6:
+                        from IPython.core.completer import provisionalcompleter
+                        with provisionalcompleter():
+                            self.matches = [json.dumps([
+                                [c.text, c.start, c.end, c.type or '?', c.signature or '']
+                                for c in get_ipython().Completer.completions(text, len(text))
+                                if self.filter(text, c.text)])]
+                    else:
+                        part, matches = get_ipython().Completer.complete(line_buffer=text)
+                        self.matches = [text + m[len(part):] for m in matches
+                                        if self.filter(text, m)]
+                except Exception:
+                    pass
+            try:
+                return self.matches[state]
+            except IndexError:
+                return None
+
+    readline.set_completer(__ipython_RL(IPython.version_info[0]).complete)
+    readline.set_completer_delims('')"
+  "Code used to setup readline completer for IPython.")
+
 (defvar python-shell-completion-native-redirect-buffer
   " *Python completions redirect*"
   "Buffer to be used to redirect output of readline commands.")
@@ -4427,7 +4512,20 @@ python-shell-completion-native-try
 (defun python-shell-completion-native-setup ()
   "Try to setup native completion, return non-nil on success."
   (let* ((process (python-shell-get-process))
-         (output (python-shell-send-string-no-output "
+         (completer (pcase python-shell-readline-completer
+                      ('auto
+                       (if (string-match-p "ipython[23]?\\'" python-shell-interpreter)
+                           python-shell--readline-ipython-setup-code
+                         python-shell--readline-jedi-setup-code))
+                      ((pred stringp) python-shell-readline-completer)
+                      (_ "")))
+         (output (python-shell-send-string-no-output
+                  (concat "
+try:
+    del __PYTHON_EL_setup_readline_completer
+except:
+    pass
+" completer "
 def __PYTHON_EL_native_completion_setup():
     try:
         import readline
@@ -4500,11 +4598,23 @@ python-shell-completion-native-setup
                 else:
                     return completion
 
+        def is_rlcompleter(completer):
+            try:
+                if completer.__self__.__module__ == 'rlcompleter':
+                    return True
+                else:
+                    return False
+            except Exception:
+                return False
+
         completer = readline.get_completer()
 
-        if not completer:
-            # Used as last resort to avoid breaking customizations.
-            import rlcompleter
+        if not completer or is_rlcompleter(completer):
+            try:
+                __PYTHON_EL_setup_readline_completer()
+            except:
+                # Used as last resort to avoid breaking customizations.
+                import rlcompleter
             completer = readline.get_completer()
 
         if completer and not getattr(completer, 'PYTHON_EL_WRAPPED', False):
@@ -4539,9 +4649,13 @@ python-shell-completion-native-setup
         print ('python.el: native completion setup failed, %s: %s'
                % sys.exc_info()[:2])
 
-__PYTHON_EL_native_completion_setup()" process)))
+__PYTHON_EL_native_completion_setup()") process)))
     (when (string-match-p "python\\.el: native completion setup loaded"
                           output)
+      (setq-local python-shell-readline-completer-delims
+                  (string-trim-right
+                   (python-shell-send-string-no-output
+                    "import readline; print(readline.get_completer_delims())")))
       (python-shell-completion-native-try))))
 
 (defun python-shell-completion-native-turn-off (&optional msg)
@@ -4609,6 +4723,8 @@ python-shell-completion-native-get-completions
     (let* ((original-filter-fn (process-filter process))
            (redirect-buffer (get-buffer-create
                              python-shell-completion-native-redirect-buffer))
+           (sep (if (string= python-shell-readline-completer-delims "")
+                    "[\n\r]+" "[ \f\t\n\r\v()]+"))
            (trigger "\t")
            (new-input (concat input trigger))
            (input-length
@@ -4651,28 +4767,72 @@ python-shell-completion-native-get-completions
                      process python-shell-completion-native-output-timeout
                      comint-redirect-finished-regexp)
                 (re-search-backward "0__dummy_completion__" nil t)
-                (cl-remove-duplicates
-                 (split-string
-                  (buffer-substring-no-properties
-                   (line-beginning-position) (point-min))
-                  "[ \f\t\n\r\v()]+" t)
-                 :test #'string=))))
+                (let ((str (buffer-substring-no-properties
+                            (line-beginning-position) (point-min))))
+                  (if (string= "[" (substring str 0 1))
+                      (condition-case nil
+                          (python--parse-json-array str)
+                        (t (cl-remove-duplicates (split-string str sep t)
+                                                 :test #'string=)))
+                    (cl-remove-duplicates (split-string str sep t)
+                                          :test #'string=))))))
         (set-process-filter process original-filter-fn)))))
 
 (defun python-shell-completion-get-completions (process input)
   "Get completions of INPUT using PROCESS."
   (with-current-buffer (process-buffer process)
-    (let ((completions
-           (python-util-strip-string
-            (python-shell-send-string-no-output
-             (format
-              "%s\nprint(';'.join(__PYTHON_EL_get_completions(%s)))"
+    (python--parse-json-array
+     (python-shell-send-string-no-output
+      (format "%s\nprint(__PYTHON_EL_get_completions(%s))"
               python-shell-completion-setup-code
               (python-shell--encode-string input))
-             process))))
-      (when (> (length completions) 2)
-        (split-string completions
-                      "^'\\|^\"\\|;\\|'$\\|\"$" t)))))
+      process))))
+
+(defun python-shell--get-multiline-input ()
+  "Return lines at a multi-line input in Python shell."
+  (save-excursion
+    (let ((p (point)) lines)
+      (when (progn
+              (beginning-of-line)
+              (looking-back python-shell-prompt-block-regexp (pos-bol)))
+        (push (buffer-substring-no-properties (point) p) lines)
+        (while (progn (comint-previous-prompt 1)
+                      (looking-back python-shell-prompt-block-regexp (pos-bol)))
+          (push (buffer-substring-no-properties (point) (pos-eol)) lines))
+        (push (buffer-substring-no-properties (point) (pos-eol)) lines))
+      lines)))
+
+(defun python-shell--extra-completion-context ()
+  "Get extra completion context of current input in Python shell."
+  (let ((lines (python-shell--get-multiline-input))
+        (python-indent-guess-indent-offset nil))
+    (when (not (zerop (length lines)))
+      (with-temp-buffer
+        (delay-mode-hooks
+          (insert (string-join lines "\n"))
+          (python-mode)
+          (python-shell-completion-extra-context))))))
+
+(defun python-shell-completion-extra-context (&optional pos)
+  "Get extra completion context at position POS in Python buffer.
+If optional argument POS is nil, use current position.
+
+Readline completers could use current line as the completion
+context, which may be insufficient.  In this function, extra
+context (e.g. multi-line function call) is found and reformatted
+as one line, which is required by native completion."
+  (let (bound p)
+    (save-excursion
+      (and pos (goto-char pos))
+      (setq bound (pos-bol))
+      (python-nav-up-list -1)
+      (when (and (< (point) bound)
+                 (looking-back
+                  (python-rx (+ (or "." symbol-name))) (pos-bol) t))
+        (setq p (match-beginning 0))))
+    (when p
+      (replace-regexp-in-string
+       "\n[ \t]*" "" (buffer-substring-no-properties p (1- bound))))))
 
 (defvar-local python-shell--capf-cache nil
   "Variable to store cached completions and invalidation keys.")
@@ -4687,21 +4847,21 @@ python-shell-completion-at-point
                          ;; Working on a shell buffer: use prompt end.
                          (cdr (python-util-comint-last-prompt))
                        (line-beginning-position)))
-         (import-statement
-          (when (string-match-p
-                 (rx (* space) word-start (or "from" "import") word-end space)
-                 (buffer-substring-no-properties line-start (point)))
-            (buffer-substring-no-properties line-start (point))))
+         (no-delims (with-current-buffer (process-buffer process)
+                      (if python-shell-completion-native-enable
+                          (string= python-shell-readline-completer-delims "")
+                        (string-match-p "ipython[23]?\\'" python-shell-interpreter))))
          (start
           (if (< (point) line-start)
               (point)
             (save-excursion
-              (if (not (re-search-backward
-                        (python-rx
-                         (or whitespace open-paren close-paren
-                             string-delimiter simple-operator))
-                        line-start
-                        t 1))
+              (if (or no-delims
+                      (not (re-search-backward
+                            (python-rx
+                             (or whitespace open-paren close-paren
+                                 string-delimiter simple-operator))
+                            line-start
+                            t 1)))
                   line-start
                 (forward-char (length (match-string-no-properties 0)))
                 (point)))))
@@ -4741,18 +4901,58 @@ python-shell-completion-at-point
                   (t #'python-shell-completion-native-get-completions))))
          (prev-prompt (car python-shell--capf-cache))
          (re (or (cadr python-shell--capf-cache) regexp-unmatchable))
-         (prefix (buffer-substring-no-properties start end)))
+         (prefix (buffer-substring-no-properties start end))
+         (prefix-offset 0)
+         (extra-context (when no-delims
+                          (if is-shell-buffer
+                              (python-shell--extra-completion-context)
+                            (python-shell-completion-extra-context))))
+         (extra-offset (length extra-context)))
+    (unless (zerop extra-offset)
+      (setq prefix (concat extra-context prefix)))
     ;; To invalidate the cache, we check if the prompt position or the
     ;; completion prefix changed.
     (unless (and (equal prev-prompt (car prompt-boundaries))
-                 (string-match re prefix))
+                 (string-match re prefix)
+                 (setq prefix-offset (- (length prefix) (match-end 1))))
       (setq python-shell--capf-cache
             `(,(car prompt-boundaries)
               ,(if (string-empty-p prefix)
                    regexp-unmatchable
-                 (concat "\\`" (regexp-quote prefix) "\\(?:\\sw\\|\\s_\\)*\\'"))
-              ,@(funcall completion-fn process (or import-statement prefix)))))
-    (list start end (cddr python-shell--capf-cache))))
+                 (concat "\\`\\(" (regexp-quote prefix) "\\)\\(?:\\sw\\|\\s_\\)*\\'"))
+              ,@(funcall completion-fn process prefix))))
+    (let ((cands (cddr python-shell--capf-cache)))
+      (cond
+       ((stringp (car cands))
+        (if no-delims
+            ;; Reduce completion candidates due to long prefix.
+            (if-let ((Lp (length prefix))
+                     ((string-match "\\(\\sw\\|\\s_\\)+\\'" prefix))
+                     (L (match-beginning 0)))
+                ;; If extra-offset is not zero:
+                ;;                  start              end
+                ;; o------------------o---------o-------o
+                ;; |<- extra-offset ->|
+                ;; |<----------- L ------------>|
+                ;;                          new-start
+                (list (+ start L (- extra-offset)) end
+                      (mapcar (lambda (s) (substring s L)) cands))
+              (list end end (mapcar (lambda (s) (substring s Lp)) cands)))
+          (list start end cands)))
+       ;; python-shell-completion(-native)-get-completions may produce
+       ;; a list of (text start end type signature) for completion.
+       ;; See `python-shell-readline-ipython-setup-code' and
+       ;; `python-shell-completion-setup-code'.
+       ((consp (car cands))
+        (list (+ start (nth 1 (car cands)) (- extra-offset))
+              ;; Candidates may be cached, so the end position should
+              ;; be adjusted according to current completion prefix.
+              (+ start (nth 2 (car cands)) (- extra-offset) prefix-offset)
+              cands
+              :annotation-function
+              (lambda (c) (concat " " (nth 3 (assoc c cands))))
+              :company-docsig
+              (lambda (c) (nth 4 (assoc c cands)))))))))
 
 (define-obsolete-function-alias
   'python-shell-completion-complete-at-point
diff --git a/test/lisp/progmodes/python-tests.el b/test/lisp/progmodes/python-tests.el
index 97ffd5fe20f..ae23ff8ebe2 100644
--- a/test/lisp/progmodes/python-tests.el
+++ b/test/lisp/progmodes/python-tests.el
@@ -4787,6 +4787,84 @@ python-shell-completion-at-point-native-1
      (end-of-line 0)
      (should-not (nth 2 (python-shell-completion-at-point))))))
 
+(defun python-tests--completion-module ()
+  "Check if modules can be completed in Python shell."
+  (insert "import datet")
+  (completion-at-point)
+  (beginning-of-line)
+  (should (looking-at-p "import datetime"))
+  (kill-line)
+  (insert "from datet")
+  (completion-at-point)
+  (beginning-of-line)
+  (should (looking-at-p "from datetime"))
+  (end-of-line)
+  (insert " import timed")
+  (completion-at-point)
+  (beginning-of-line)
+  (should (looking-at-p "from datetime import timedelta"))
+  (kill-line))
+
+(defun python-tests--completion-parameters ()
+  "Check if parameters can be completed in Python shell."
+  (insert "import re")
+  (comint-send-input)
+  (python-tests-shell-wait-for-prompt)
+  (insert "re.split('b', 'abc', maxs")
+  (completion-at-point)
+  (should (string= "re.split('b', 'abc', maxsplit="
+                   (buffer-substring (line-beginning-position) (point))))
+  (insert "0, ")
+  (should (python-shell-completion-at-point))
+  ;; Test if cache is used.
+  (cl-letf (((symbol-function 'python-shell-completion-get-completions)
+             'ignore)
+            ((symbol-function 'python-shell-completion-native-get-completions)
+             'ignore))
+    (insert "fla")
+    (completion-at-point)
+    (should (string= "re.split('b', 'abc', maxsplit=0, flags="
+                     (buffer-substring (line-beginning-position) (point)))))
+  (beginning-of-line)
+  (kill-line))
+
+(defun python-tests--completion-extra-context ()
+  "Check if extra context is used for completion."
+  (insert "re.split('b', 'abc',")
+  (comint-send-input)
+  (python-tests-shell-wait-for-prompt)
+  (insert "maxs")
+  (completion-at-point)
+  (should (string= "maxsplit="
+                   (buffer-substring (line-beginning-position) (point)))))
+
+(ert-deftest python-shell-completion-at-point-jedi-completer ()
+  "Check if Python shell completion works with Jedi."
+  (skip-unless (executable-find python-tests-shell-interpreter))
+  (python-tests-with-temp-buffer-with-shell
+   ""
+   (python-shell-with-shell-buffer
+     (python-shell-completion-native-turn-on)
+     (skip-unless (string= python-shell-readline-completer-delims ""))
+     (python-tests--completion-module)
+     (python-tests--completion-parameters)
+     (python-tests--completion-extra-context))))
+
+(ert-deftest python-shell-completion-at-point-ipython ()
+  "Check if Python shell completion works for IPython."
+  (let ((python-shell-interpreter "ipython")
+        (python-shell-interpreter-args "-i --simple-prompt"))
+    (skip-unless (executable-find python-shell-interpreter))
+    (python-tests-with-temp-buffer-with-shell
+     ""
+     (python-shell-with-shell-buffer
+       (python-shell-completion-native-turn-off)
+       (python-tests--completion-module)
+       (python-tests--completion-parameters)
+       (python-shell-completion-native-turn-on)
+       (python-tests--completion-module)
+       (python-tests--completion-parameters)
+       (python-tests--completion-extra-context)))))
 
 \f
 ;;; PDB Track integration
-- 
2.25.1

bug#68559: [PATCH] Improve Python shell completion

[kobarity]

Liu Hui wrote:
> kobarity <kobarity@gmail.com> 于2024年1月23日周二 22:15写道：
> > Thank you.  I tried the new patch and confirmed that the Python Shell
> > buffer allows keyword argument completion on multi-line function
> > calls.  However, my expectation is that it can be done in Python
> > buffer as well (after calling `python-shell-send-buffer').  Is this
> > also possible?
> Thank you for the feedback. I have updated the patch to add support in
> Python buffer.

Thank you.  It's working very nice.

I apologize for reporting in the piecemeal fashion, but I have noticed
the following differences from the current completion.

1. Module names cannot be completed in Python buffer.
2. Completion is not working in comments.

I'm not sure if the completion should work in comments, but some
people might want it because the current python-mode allows it in both
Python buffer and Python Shell buffer.

bug#68559: [PATCH] Improve Python shell completion

[Liu Hui]

[-- Attachment #1: Type: text/plain, Size: 1567 bytes --]

kobarity <kobarity@gmail.com> 于2024年1月25日周四 23:38写道：
>
> Liu Hui wrote:
> > kobarity <kobarity@gmail.com> 于2024年1月23日周二 22:15写道：
> > > Thank you.  I tried the new patch and confirmed that the Python Shell
> > > buffer allows keyword argument completion on multi-line function
> > > calls.  However, my expectation is that it can be done in Python
> > > buffer as well (after calling `python-shell-send-buffer').  Is this
> > > also possible?
> > Thank you for the feedback. I have updated the patch to add support in
> > Python buffer.
>
> Thank you.  It's working very nice.
>
> I apologize for reporting in the piecemeal fashion, but I have noticed
> the following differences from the current completion.

No worries. I appreciate your valuable feedback for improving the
quality of the patch.

> 1. Module names cannot be completed in Python buffer.

Can you elaborate? The completion of module names never worked for me
except using IPython with non-native completion mechanism.

Because module names cannot be completed using vanilla python
interpreter in the terminal, they are also not completed even with the
patch, unless jedi is available.

> 2. Completion is not working in comments.
>
> I'm not sure if the completion should work in comments, but some
> people might want it because the current python-mode allows it in both
> Python buffer and Python Shell buffer.

Thanks for pointing it out. The attached patch should now restore
previous basic completion behavior in comments.

[-- Attachment #2: 0001-Improve-Python-shell-completion-bug-68559.patch --]
[-- Type: text/x-patch, Size: 25288 bytes --]

From 770b306fc9ac2406517536e5276649578a18955a Mon Sep 17 00:00:00 2001
From: Liu Hui <liuhui1610@gmail.com>
Date: Thu, 18 Jan 2024 12:00:00 +0800
Subject: [PATCH] Improve Python shell completion (bug#68559)

* lisp/progmodes/python.el (python-shell-completion-setup-code): Fix
the completion code of IPython.  Change the return value to JSON
string and ...
(python-shell-completion-get-completions): ... simplify parsing.
(inferior-python-mode): Update docstring.
(python-shell-readline-completer): New option.
(python-shell-readline-completer-delims): New variable indicating the
word delimiters of readline completer.
(python-shell--readline-jedi-setup-code)
(python-shell--readline-ipython-setup-code): New internal variables.
(python-shell-completion-native-setup): Setup a suitable readline
completer and set the completer delimiter.
(python-shell-completion-native-get-completions): Convert output
string to completions properly.
(python-shell--get-multiline-input)
(python-shell--extra-completion-context)
(python-shell-completion-extra-context): New functions.
(python-shell-completion-at-point): Send text beginning from the line
start if the completion backend does not need word splitting.  Remove
the detection of import statement because it is not needed anymore.
Create proper completion table based on completions returned from
different backends.

* test/lisp/progmodes/python-tests.el (python-tests--completion-module):
(python-tests--completion-parameters):
(python-tests--completion-extra-context): New helper functions.
(python-shell-completion-at-point-jedi-completer):
(python-shell-completion-at-point-ipython): New tests.

* etc/NEWS: Announce the change.
---
 etc/NEWS                            |   6 +
 lisp/progmodes/python.el            | 309 +++++++++++++++++++++++-----
 test/lisp/progmodes/python-tests.el |  78 +++++++
 3 files changed, 341 insertions(+), 52 deletions(-)

diff --git a/etc/NEWS b/etc/NEWS
index a1874313502..9f05bdf4705 100644
--- a/etc/NEWS
+++ b/etc/NEWS
@@ -1047,6 +1047,12 @@ instead of:
 This allows the user to specify command line arguments to the non
 interactive Python interpreter specified by 'python-interpreter'.
 
+*** New user option 'python-shell-readline-completer'.
+This allows the user to specify the readline completer used for Python
+shell completion.  The default is 'auto', which means a suitable
+completer will be configured automatically according to the Python
+interpreter.
+
 ** use-package
 
 +++
diff --git a/lisp/progmodes/python.el b/lisp/progmodes/python.el
index ff799e1e662..8e7998ee8f6 100644
--- a/lisp/progmodes/python.el
+++ b/lisp/progmodes/python.el
@@ -128,9 +128,9 @@
 ;; receiving escape sequences (with some limitations, i.e. completion
 ;; in blocks does not work).  The code executed for the "fallback"
 ;; completion can be found in `python-shell-completion-setup-code' and
-;; `python-shell-completion-string-code' variables.  Their default
-;; values enable completion for both CPython and IPython, and probably
-;; any readline based shell (it's known to work with PyPy).  If your
+;; `python-shell-completion-get-completions'.  Their default values
+;; enable completion for both CPython and IPython, and probably any
+;; readline based shell (it's known to work with PyPy).  If your
 ;; Python installation lacks readline (like CPython for Windows),
 ;; installing pyreadline (URL `https://ipython.org/pyreadline.html')
 ;; should suffice.  To troubleshoot why you are not getting any
@@ -3601,7 +3601,6 @@ inferior-python-mode
 `python-shell-prompt-block-regexp',
 `python-shell-font-lock-enable',
 `python-shell-completion-setup-code',
-`python-shell-completion-string-code',
 `python-eldoc-setup-code',
 `python-ffap-setup-code' can
 customize this mode for different Python interpreters.
@@ -4319,8 +4318,9 @@ python-shell-completion-setup-code
     completions = []
     completer = None
 
+    import json
     try:
-        import readline
+        import readline, re
 
         try:
             import __builtin__
@@ -4331,16 +4331,29 @@ python-shell-completion-setup-code
 
         is_ipython = ('__IPYTHON__' in builtins or
                       '__IPYTHON__active' in builtins)
-        splits = text.split()
-        is_module = splits and splits[0] in ('from', 'import')
-
-        if is_ipython and is_module:
-            from IPython.core.completerlib import module_completion
-            completions = module_completion(text.strip())
-        elif is_ipython and '__IP' in builtins:
-            completions = __IP.complete(text)
-        elif is_ipython and 'get_ipython' in builtins:
-            completions = get_ipython().Completer.all_completions(text)
+
+        if is_ipython and 'get_ipython' in builtins:
+            def filter_c(prefix, c):
+                if re.match('_+(i?[0-9]+)?$', c):
+                    return False
+                elif c[0] == '%' and not re.match('[%a-zA-Z]+$', prefix):
+                    return False
+                return True
+
+            import IPython
+            try:
+                if IPython.version_info[0] >= 6:
+                    from IPython.core.completer import provisionalcompleter
+                    with provisionalcompleter():
+                        completions = [
+                            [c.text, c.start, c.end, c.type or '?', c.signature or '']
+                             for c in get_ipython().Completer.completions(text, len(text))
+                             if filter_c(text, c.text)]
+                else:
+                    part, matches = get_ipython().Completer.complete(line_buffer=text)
+                    completions = [text + m[len(part):] for m in matches if filter_c(text, m)]
+            except:
+                pass
         else:
             # Try to reuse current completer.
             completer = readline.get_completer()
@@ -4363,7 +4376,7 @@ python-shell-completion-setup-code
     finally:
         if getattr(completer, 'PYTHON_EL_WRAPPED', False):
             completer.print_mode = True
-    return completions"
+    return json.dumps(completions)"
   "Code used to setup completion in inferior Python processes."
   :type 'string)
 
@@ -4404,6 +4417,78 @@ python-shell-completion-native-try-output-timeout
   :version "25.1"
   :type 'float)
 
+(defcustom python-shell-readline-completer 'auto
+  "The readline completer used for Python shell completion.
+If the value is non-nil, Python shell will setup the readline
+completer unless it has been set elsewhere (e.g. in the
+PYTHONSTARTUP file).  Below are possible values:
+- `auto': the completer is determined according to the
+interpreter.  Specifically, the IPython completer, defined in
+`python-shell--readline-ipython-setup-code', is used when the
+interpreter is ipython, otherwise the Jedi completer is used.
+- a string: Python code to setup the readline.  See
+`python-shell--readline-jedi-setup-code' for reference.
+- `nil': Python shell will do nothing.
+
+In any case, if the completer is not set successfully in the end,
+fallback to the built-in rlcompleter."
+  :type '(choice (const  :tag "Automatic" auto)
+                 (const  :tag "No configuration" nil)
+                 (string :tag "Python setup code"))
+  :version "30.1")
+
+(defvar python-shell-readline-completer-delims nil
+  "Word delimiters used by the readline completer.
+It is automatically set by Python shell.")
+
+(defconst python-shell--readline-jedi-setup-code
+  "
+def __PYTHON_EL_setup_readline_completer():
+    from jedi.utils import setup_readline
+    setup_readline()"
+  "Code used to setup readline completer with Jedi.")
+
+(defconst python-shell--readline-ipython-setup-code
+  "
+def __PYTHON_EL_setup_readline_completer():
+    import readline, re, json, IPython
+
+    class __ipython_RL:
+        def __init__(self, v):
+            self.version = v
+
+        def filter(self, prefix, c):
+            if re.match('_+(i?[0-9]+)?$', c):
+                return False
+            elif c[0] == '%' and not re.match('[%a-zA-Z]+$', prefix):
+                return False
+            return True
+
+        def complete(self, text, state):
+            if state == 0:
+                try:
+                    if self.version >= 6:
+                        from IPython.core.completer import provisionalcompleter
+                        with provisionalcompleter():
+                            self.matches = [json.dumps([
+                                [c.text, c.start, c.end, c.type or '?', c.signature or '']
+                                for c in get_ipython().Completer.completions(text, len(text))
+                                if self.filter(text, c.text)])]
+                    else:
+                        part, matches = get_ipython().Completer.complete(line_buffer=text)
+                        self.matches = [text + m[len(part):] for m in matches
+                                        if self.filter(text, m)]
+                except Exception:
+                    pass
+            try:
+                return self.matches[state]
+            except IndexError:
+                return None
+
+    readline.set_completer(__ipython_RL(IPython.version_info[0]).complete)
+    readline.set_completer_delims('')"
+  "Code used to setup readline completer for IPython.")
+
 (defvar python-shell-completion-native-redirect-buffer
   " *Python completions redirect*"
   "Buffer to be used to redirect output of readline commands.")
@@ -4427,7 +4512,20 @@ python-shell-completion-native-try
 (defun python-shell-completion-native-setup ()
   "Try to setup native completion, return non-nil on success."
   (let* ((process (python-shell-get-process))
-         (output (python-shell-send-string-no-output "
+         (completer (pcase python-shell-readline-completer
+                      ('auto
+                       (if (string-match-p "ipython[23]?\\'" python-shell-interpreter)
+                           python-shell--readline-ipython-setup-code
+                         python-shell--readline-jedi-setup-code))
+                      ((pred stringp) python-shell-readline-completer)
+                      (_ "")))
+         (output (python-shell-send-string-no-output
+                  (concat "
+try:
+    del __PYTHON_EL_setup_readline_completer
+except:
+    pass
+" completer "
 def __PYTHON_EL_native_completion_setup():
     try:
         import readline
@@ -4500,11 +4598,23 @@ python-shell-completion-native-setup
                 else:
                     return completion
 
+        def is_rlcompleter(completer):
+            try:
+                if completer.__self__.__module__ == 'rlcompleter':
+                    return True
+                else:
+                    return False
+            except Exception:
+                return False
+
         completer = readline.get_completer()
 
-        if not completer:
-            # Used as last resort to avoid breaking customizations.
-            import rlcompleter
+        if not completer or is_rlcompleter(completer):
+            try:
+                __PYTHON_EL_setup_readline_completer()
+            except:
+                # Used as last resort to avoid breaking customizations.
+                import rlcompleter
             completer = readline.get_completer()
 
         if completer and not getattr(completer, 'PYTHON_EL_WRAPPED', False):
@@ -4539,9 +4649,13 @@ python-shell-completion-native-setup
         print ('python.el: native completion setup failed, %s: %s'
                % sys.exc_info()[:2])
 
-__PYTHON_EL_native_completion_setup()" process)))
+__PYTHON_EL_native_completion_setup()") process)))
     (when (string-match-p "python\\.el: native completion setup loaded"
                           output)
+      (setq-local python-shell-readline-completer-delims
+                  (string-trim-right
+                   (python-shell-send-string-no-output
+                    "import readline; print(readline.get_completer_delims())")))
       (python-shell-completion-native-try))))
 
 (defun python-shell-completion-native-turn-off (&optional msg)
@@ -4609,6 +4723,8 @@ python-shell-completion-native-get-completions
     (let* ((original-filter-fn (process-filter process))
            (redirect-buffer (get-buffer-create
                              python-shell-completion-native-redirect-buffer))
+           (sep (if (string= python-shell-readline-completer-delims "")
+                    "[\n\r]+" "[ \f\t\n\r\v()]+"))
            (trigger "\t")
            (new-input (concat input trigger))
            (input-length
@@ -4651,28 +4767,72 @@ python-shell-completion-native-get-completions
                      process python-shell-completion-native-output-timeout
                      comint-redirect-finished-regexp)
                 (re-search-backward "0__dummy_completion__" nil t)
-                (cl-remove-duplicates
-                 (split-string
-                  (buffer-substring-no-properties
-                   (line-beginning-position) (point-min))
-                  "[ \f\t\n\r\v()]+" t)
-                 :test #'string=))))
+                (let ((str (buffer-substring-no-properties
+                            (line-beginning-position) (point-min))))
+                  (if (string= "[" (substring str 0 1))
+                      (condition-case nil
+                          (python--parse-json-array str)
+                        (t (cl-remove-duplicates (split-string str sep t)
+                                                 :test #'string=)))
+                    (cl-remove-duplicates (split-string str sep t)
+                                          :test #'string=))))))
         (set-process-filter process original-filter-fn)))))
 
 (defun python-shell-completion-get-completions (process input)
   "Get completions of INPUT using PROCESS."
   (with-current-buffer (process-buffer process)
-    (let ((completions
-           (python-util-strip-string
-            (python-shell-send-string-no-output
-             (format
-              "%s\nprint(';'.join(__PYTHON_EL_get_completions(%s)))"
+    (python--parse-json-array
+     (python-shell-send-string-no-output
+      (format "%s\nprint(__PYTHON_EL_get_completions(%s))"
               python-shell-completion-setup-code
               (python-shell--encode-string input))
-             process))))
-      (when (> (length completions) 2)
-        (split-string completions
-                      "^'\\|^\"\\|;\\|'$\\|\"$" t)))))
+      process))))
+
+(defun python-shell--get-multiline-input ()
+  "Return lines at a multi-line input in Python shell."
+  (save-excursion
+    (let ((p (point)) lines)
+      (when (progn
+              (beginning-of-line)
+              (looking-back python-shell-prompt-block-regexp (pos-bol)))
+        (push (buffer-substring-no-properties (point) p) lines)
+        (while (progn (comint-previous-prompt 1)
+                      (looking-back python-shell-prompt-block-regexp (pos-bol)))
+          (push (buffer-substring-no-properties (point) (pos-eol)) lines))
+        (push (buffer-substring-no-properties (point) (pos-eol)) lines))
+      lines)))
+
+(defun python-shell--extra-completion-context ()
+  "Get extra completion context of current input in Python shell."
+  (let ((lines (python-shell--get-multiline-input))
+        (python-indent-guess-indent-offset nil))
+    (when (not (zerop (length lines)))
+      (with-temp-buffer
+        (delay-mode-hooks
+          (insert (string-join lines "\n"))
+          (python-mode)
+          (python-shell-completion-extra-context))))))
+
+(defun python-shell-completion-extra-context (&optional pos)
+  "Get extra completion context at position POS in Python buffer.
+If optional argument POS is nil, use current position.
+
+Readline completers could use current line as the completion
+context, which may be insufficient.  In this function, extra
+context (e.g. multi-line function call) is found and reformatted
+as one line, which is required by native completion."
+  (let (bound p)
+    (save-excursion
+      (and pos (goto-char pos))
+      (setq bound (pos-bol))
+      (python-nav-up-list -1)
+      (when (and (< (point) bound)
+                 (looking-back
+                  (python-rx (+ (or "." symbol-name))) (pos-bol) t))
+        (setq p (match-beginning 0))))
+    (when p
+      (replace-regexp-in-string
+       "\n[ \t]*" "" (buffer-substring-no-properties p (1- bound))))))
 
 (defvar-local python-shell--capf-cache nil
   "Variable to store cached completions and invalidation keys.")
@@ -4687,21 +4847,26 @@ python-shell-completion-at-point
                          ;; Working on a shell buffer: use prompt end.
                          (cdr (python-util-comint-last-prompt))
                        (line-beginning-position)))
-         (import-statement
-          (when (string-match-p
-                 (rx (* space) word-start (or "from" "import") word-end space)
-                 (buffer-substring-no-properties line-start (point)))
-            (buffer-substring-no-properties line-start (point))))
+         (no-delims
+          (and (not (if is-shell-buffer
+                        (eq 'font-lock-comment-face
+                            (get-text-property (1- (point)) 'face))
+                      (python-syntax-context 'comment)))
+               (with-current-buffer (process-buffer process)
+                 (if python-shell-completion-native-enable
+                     (string= python-shell-readline-completer-delims "")
+                   (string-match-p "ipython[23]?\\'" python-shell-interpreter)))))
          (start
           (if (< (point) line-start)
               (point)
             (save-excursion
-              (if (not (re-search-backward
-                        (python-rx
-                         (or whitespace open-paren close-paren
-                             string-delimiter simple-operator))
-                        line-start
-                        t 1))
+              (if (or no-delims
+                      (not (re-search-backward
+                            (python-rx
+                             (or whitespace open-paren close-paren
+                                 string-delimiter simple-operator))
+                            line-start
+                            t 1)))
                   line-start
                 (forward-char (length (match-string-no-properties 0)))
                 (point)))))
@@ -4741,18 +4906,58 @@ python-shell-completion-at-point
                   (t #'python-shell-completion-native-get-completions))))
          (prev-prompt (car python-shell--capf-cache))
          (re (or (cadr python-shell--capf-cache) regexp-unmatchable))
-         (prefix (buffer-substring-no-properties start end)))
+         (prefix (buffer-substring-no-properties start end))
+         (prefix-offset 0)
+         (extra-context (when no-delims
+                          (if is-shell-buffer
+                              (python-shell--extra-completion-context)
+                            (python-shell-completion-extra-context))))
+         (extra-offset (length extra-context)))
+    (unless (zerop extra-offset)
+      (setq prefix (concat extra-context prefix)))
     ;; To invalidate the cache, we check if the prompt position or the
     ;; completion prefix changed.
     (unless (and (equal prev-prompt (car prompt-boundaries))
-                 (string-match re prefix))
+                 (string-match re prefix)
+                 (setq prefix-offset (- (length prefix) (match-end 1))))
       (setq python-shell--capf-cache
             `(,(car prompt-boundaries)
               ,(if (string-empty-p prefix)
                    regexp-unmatchable
-                 (concat "\\`" (regexp-quote prefix) "\\(?:\\sw\\|\\s_\\)*\\'"))
-              ,@(funcall completion-fn process (or import-statement prefix)))))
-    (list start end (cddr python-shell--capf-cache))))
+                 (concat "\\`\\(" (regexp-quote prefix) "\\)\\(?:\\sw\\|\\s_\\)*\\'"))
+              ,@(funcall completion-fn process prefix))))
+    (let ((cands (cddr python-shell--capf-cache)))
+      (cond
+       ((stringp (car cands))
+        (if no-delims
+            ;; Reduce completion candidates due to long prefix.
+            (if-let ((Lp (length prefix))
+                     ((string-match "\\(\\sw\\|\\s_\\)+\\'" prefix))
+                     (L (match-beginning 0)))
+                ;; If extra-offset is not zero:
+                ;;                  start              end
+                ;; o------------------o---------o-------o
+                ;; |<- extra-offset ->|
+                ;; |<----------- L ------------>|
+                ;;                          new-start
+                (list (+ start L (- extra-offset)) end
+                      (mapcar (lambda (s) (substring s L)) cands))
+              (list end end (mapcar (lambda (s) (substring s Lp)) cands)))
+          (list start end cands)))
+       ;; python-shell-completion(-native)-get-completions may produce
+       ;; a list of (text start end type signature) for completion.
+       ;; See `python-shell--readline-ipython-setup-code' and
+       ;; `python-shell-completion-setup-code'.
+       ((consp (car cands))
+        (list (+ start (nth 1 (car cands)) (- extra-offset))
+              ;; Candidates may be cached, so the end position should
+              ;; be adjusted according to current completion prefix.
+              (+ start (nth 2 (car cands)) (- extra-offset) prefix-offset)
+              cands
+              :annotation-function
+              (lambda (c) (concat " " (nth 3 (assoc c cands))))
+              :company-docsig
+              (lambda (c) (nth 4 (assoc c cands)))))))))
 
 (define-obsolete-function-alias
   'python-shell-completion-complete-at-point
diff --git a/test/lisp/progmodes/python-tests.el b/test/lisp/progmodes/python-tests.el
index 97ffd5fe20f..ae23ff8ebe2 100644
--- a/test/lisp/progmodes/python-tests.el
+++ b/test/lisp/progmodes/python-tests.el
@@ -4787,6 +4787,84 @@ python-shell-completion-at-point-native-1
      (end-of-line 0)
      (should-not (nth 2 (python-shell-completion-at-point))))))
 
+(defun python-tests--completion-module ()
+  "Check if modules can be completed in Python shell."
+  (insert "import datet")
+  (completion-at-point)
+  (beginning-of-line)
+  (should (looking-at-p "import datetime"))
+  (kill-line)
+  (insert "from datet")
+  (completion-at-point)
+  (beginning-of-line)
+  (should (looking-at-p "from datetime"))
+  (end-of-line)
+  (insert " import timed")
+  (completion-at-point)
+  (beginning-of-line)
+  (should (looking-at-p "from datetime import timedelta"))
+  (kill-line))
+
+(defun python-tests--completion-parameters ()
+  "Check if parameters can be completed in Python shell."
+  (insert "import re")
+  (comint-send-input)
+  (python-tests-shell-wait-for-prompt)
+  (insert "re.split('b', 'abc', maxs")
+  (completion-at-point)
+  (should (string= "re.split('b', 'abc', maxsplit="
+                   (buffer-substring (line-beginning-position) (point))))
+  (insert "0, ")
+  (should (python-shell-completion-at-point))
+  ;; Test if cache is used.
+  (cl-letf (((symbol-function 'python-shell-completion-get-completions)
+             'ignore)
+            ((symbol-function 'python-shell-completion-native-get-completions)
+             'ignore))
+    (insert "fla")
+    (completion-at-point)
+    (should (string= "re.split('b', 'abc', maxsplit=0, flags="
+                     (buffer-substring (line-beginning-position) (point)))))
+  (beginning-of-line)
+  (kill-line))
+
+(defun python-tests--completion-extra-context ()
+  "Check if extra context is used for completion."
+  (insert "re.split('b', 'abc',")
+  (comint-send-input)
+  (python-tests-shell-wait-for-prompt)
+  (insert "maxs")
+  (completion-at-point)
+  (should (string= "maxsplit="
+                   (buffer-substring (line-beginning-position) (point)))))
+
+(ert-deftest python-shell-completion-at-point-jedi-completer ()
+  "Check if Python shell completion works with Jedi."
+  (skip-unless (executable-find python-tests-shell-interpreter))
+  (python-tests-with-temp-buffer-with-shell
+   ""
+   (python-shell-with-shell-buffer
+     (python-shell-completion-native-turn-on)
+     (skip-unless (string= python-shell-readline-completer-delims ""))
+     (python-tests--completion-module)
+     (python-tests--completion-parameters)
+     (python-tests--completion-extra-context))))
+
+(ert-deftest python-shell-completion-at-point-ipython ()
+  "Check if Python shell completion works for IPython."
+  (let ((python-shell-interpreter "ipython")
+        (python-shell-interpreter-args "-i --simple-prompt"))
+    (skip-unless (executable-find python-shell-interpreter))
+    (python-tests-with-temp-buffer-with-shell
+     ""
+     (python-shell-with-shell-buffer
+       (python-shell-completion-native-turn-off)
+       (python-tests--completion-module)
+       (python-tests--completion-parameters)
+       (python-shell-completion-native-turn-on)
+       (python-tests--completion-module)
+       (python-tests--completion-parameters)
+       (python-tests--completion-extra-context)))))
 
 \f
 ;;; PDB Track integration
-- 
2.25.1

bug#68559: [PATCH] Improve Python shell completion

[kobarity]

Liu Hui wrote:
> kobarity <kobarity@gmail.com> 于2024年1月25日周四 23:38写道：
> > 1. Module names cannot be completed in Python buffer.
> Can you elaborate? The completion of module names never worked for me
> except using IPython with non-native completion mechanism.
> 
> Because module names cannot be completed using vanilla python
> interpreter in the terminal, they are also not completed even with the
> patch, unless jedi is available.

Hmm, I must have done something wrong.  I cannot reproduce the problem
of module name completion even with the previous patch.
As for the current python-mode, I was able to complete the module
names because I enabled Jedi in PYTHONSTARTUP.  Sorry for the confusion.

I'm experiencing strange behavior regarding completion of import
statement in a block in Python buffer.  If I try to type the following
lines and then try to complete it, it will fail.

#+begin_src python
try:
    im
#+end_src

However, when I try to complete at the beginning of the second line:

#+begin_src python
try:
    
#+end_src

"import" keyword also appears as a candidate.  If I cancel the
candidates and type "im" and try to complete it, it will succeed.
This behavior does not occur in Python Shell buffer.

Another thing I noticed is the multi-line import statement.  If the
import statement is one-line, each items (IGNORECASE and MULTILINE in
the example below) can be completed.

#+begin_src python
from re import IGNORECASE, MULTILINE
#+end_src

However, they cannot be completed if the import statement spans
multi-line.

#+begin_src python
from re import (
    IGN
#+end_src

This happens in both Python buffer and Python Shell buffer.  Perhaps
this is a limitation of Jedi completer?

> > 2. Completion is not working in comments.
> >
> > I'm not sure if the completion should work in comments, but some
> > people might want it because the current python-mode allows it in both
> > Python buffer and Python Shell buffer.
> 
> Thanks for pointing it out. The attached patch should now restore
> previous basic completion behavior in comments.

Thanks, I confirmed that basic completion (without keyword argument
completion) can be performed within comments.

bug#68559: [PATCH] Improve Python shell completion

[kobarity]

kobarity wrote:
> Liu Hui wrote:
> > Thanks for pointing it out. The attached patch should now restore
> > previous basic completion behavior in comments.
> 
> Thanks, I confirmed that basic completion (without keyword argument
> completion) can be performed within comments.

One additional comment regarding ERT.  pyenv installs an ipython shim
to the shims directory, if there is at least one virtualenv providing
ipython.  It can be found by `executable-find'.  However, it is not
usable unless the virtualenv is activated.  As a result, the
`python-shell-completion-at-point-ipython' ERT would fail unless a
virtualenv which provides ipython is activated.

This may be something pyenv users should take care of, but it is more
convenient for pyenv users like me to make the following changes.

diff --git a/test/lisp/progmodes/python-tests.el b/test/lisp/progmodes/python-tests.el
index ae23ff8ebe2..9452f136cfb 100644
--- a/test/lisp/progmodes/python-tests.el
+++ b/test/lisp/progmodes/python-tests.el
@@ -4854,7 +4854,10 @@ python-shell-completion-at-point-ipython
   "Check if Python shell completion works for IPython."
   (let ((python-shell-interpreter "ipython")
         (python-shell-interpreter-args "-i --simple-prompt"))
-    (skip-unless (executable-find python-shell-interpreter))
+    (skip-unless
+     (and
+      (executable-find python-shell-interpreter)
+      (eql (call-process python-shell-interpreter nil nil nil "--version") 0)))
     (python-tests-with-temp-buffer-with-shell
      ""
      (python-shell-with-shell-buffer

bug#68559: [PATCH] Improve Python shell completion

[Eli Zaretskii]

> Date: Mon, 29 Jan 2024 22:15:20 +0900
> From: kobarity <kobarity@gmail.com>
> Cc: Eli Zaretskii <eliz@gnu.org>,
> 	68559@debbugs.gnu.org
> 
> kobarity wrote:
> > Liu Hui wrote:
> > > Thanks for pointing it out. The attached patch should now restore
> > > previous basic completion behavior in comments.
> > 
> > Thanks, I confirmed that basic completion (without keyword argument
> > completion) can be performed within comments.
> 
> One additional comment regarding ERT.  pyenv installs an ipython shim
> to the shims directory, if there is at least one virtualenv providing
> ipython.  It can be found by `executable-find'.  However, it is not
> usable unless the virtualenv is activated.  As a result, the
> `python-shell-completion-at-point-ipython' ERT would fail unless a
> virtualenv which provides ipython is activated.
> 
> This may be something pyenv users should take care of, but it is more
> convenient for pyenv users like me to make the following changes.

Do we have a consensus about this issue?  Should I install the
original patch, or are you still discussing it, and an updated patch
will be posted soon?

Thanks.

bug#68559: [PATCH] Improve Python shell completion

[kobarity]

Eli Zaretskii wrote:
> > Date: Mon, 29 Jan 2024 22:15:20 +0900
> > From: kobarity <kobarity@gmail.com>
> > Cc: Eli Zaretskii <eliz@gnu.org>,
> > 	68559@debbugs.gnu.org
> > 
> > kobarity wrote:
> > > Liu Hui wrote:
> > > > Thanks for pointing it out. The attached patch should now restore
> > > > previous basic completion behavior in comments.
> > > 
> > > Thanks, I confirmed that basic completion (without keyword argument
> > > completion) can be performed within comments.
> > 
> > One additional comment regarding ERT.  pyenv installs an ipython shim
> > to the shims directory, if there is at least one virtualenv providing
> > ipython.  It can be found by `executable-find'.  However, it is not
> > usable unless the virtualenv is activated.  As a result, the
> > `python-shell-completion-at-point-ipython' ERT would fail unless a
> > virtualenv which provides ipython is activated.
> > 
> > This may be something pyenv users should take care of, but it is more
> > convenient for pyenv users like me to make the following changes.
> 
> Do we have a consensus about this issue?  Should I install the
> original patch, or are you still discussing it, and an updated patch
> will be posted soon?
> 
> Thanks.

Although I think there are a few things left unanswered, the latest
patch sent on Jan 26th is very worthwhile and stable enough to be
installed.  So, unless Liu is preparing a new patch, I agree that the
latest patch should be installed.  That way more people can try it and
find the problem, if any.

bug#68559: [PATCH] Improve Python shell completion

[Liu Hui]

kobarity <kobarity@gmail.com> 于2024年2月1日周四 22:40写道：
>
> Eli Zaretskii wrote:

> > Do we have a consensus about this issue?  Should I install the
> > original patch, or are you still discussing it, and an updated patch
> > will be posted soon?
> >
> > Thanks.
>
> Although I think there are a few things left unanswered, the latest
> patch sent on Jan 26th is very worthwhile and stable enough to be
> installed.  So, unless Liu is preparing a new patch, I agree that the
> latest patch should be installed.  That way more people can try it and
> find the problem, if any.

Sorry for the late reply. Recent events have occupied all my time. I
plan to update the patch in the next day or two to address remaining
concerns.

bug#68559: [PATCH] Improve Python shell completion

[Liu Hui]

[-- Attachment #1: Type: text/plain, Size: 3845 bytes --]


kobarity <kobarity@gmail.com> writes:

> I'm experiencing strange behavior regarding completion of import
> statement in a block in Python buffer.  If I try to type the following
> lines and then try to complete it, it will fail.
>
> #+begin_src python
> try:
>     im
> #+end_src

The problem can be reproduced by running python with jedi completer in
a terminal. The reason is that readline completer can only see the
text in the current line, i.e. "    im"; in this case, jedi does not
complete the text to "    import".

Similarly, the jedi completer cannot complete "except" after ex:

try:
    pass
ex|

> However, when I try to complete at the beginning of the second line:
>
> #+begin_src python
> try:
>
> #+end_src
>
>
> "import" keyword also appears as a candidate.  If I cancel the
> candidates and type "im" and try to complete it, it will succeed.

It is because jedi produces completions including "    import" for
blank string "    ". Due to the same prefix between "    " and
"    im", completion cache is reused for the latter. Then "    import"
can be completed.

It is more a limitation of readline completer than a problem with
jedi, as we cannot provide proper completion context for jedi. We may
define a custom completer to combine jedi and rlcompleter, e.g.

(setq python-shell-readline-completer "
def __PYTHON_EL_setup_readline_completer():
    import readline, rlcompleter
    import re, sys, os, __main__
    from jedi import Interpreter

    class MyJediRL:
        def __init__(self):
            self.rlcompleter = rlcompleter.Completer()
            self.rldelim = readline.get_completer_delims()

        def complete(self, text, state):
            if state == 0:
                sys.path.insert(0, os.getcwd())
                try:
                    interpreter = Interpreter(text, [__main__.__dict__])
                    completions = interpreter.complete(fuzzy=False)
                    self.matches = [
                        text[:len(text) - c._like_name_length] + c.name_with_symbols
                        for c in completions
                    ]

                    # try rlcompleter
                    sub = re.split('[' + re.escape(self.rldelim) + ']', text)[-1]
                    i = 0
                    while True:
                        completion = self.rlcompleter.complete(sub, i)
                        if not completion:
                            break
                        i += 1
                        completion = text[:len(text)-len(sub)] + completion.rstrip(' ()')
                        if completion not in self.matches:
                            self.matches.append(completion)
                except:
                    raise
                finally:
                    sys.path.pop(0)
            try:
                return self.matches[state]
            except IndexError:
                return None

    readline.set_completer(MyJediRL().complete)
    readline.set_completer_delims('')")

> Another thing I noticed is the multi-line import statement.  If the
> import statement is one-line, each items (IGNORECASE and MULTILINE in
> the example below) can be completed.
>
> #+begin_src python
> from re import IGNORECASE, MULTILINE
> #+end_src
>
>
> However, they cannot be completed if the import statement spans
> multi-line.
>
> #+begin_src python
> from re import (
>     IGN
> #+end_src
>
> This happens in both Python buffer and Python Shell buffer.  Perhaps
> this is a limitation of Jedi completer?

Yes. Because readline completer cannot see cross-line context, I added
the function "python-shell-completion-extra-context" in previous patch
to address the case of multi-line function call. I have updated the
attached patch to handle multi-line import statement.

The change to python-tests.el has been incorporated in the patch.
Thanks!


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Improve-Python-shell-completion-bug-68559.patch --]
[-- Type: text/x-diff, Size: 26123 bytes --]

From a5da22803e967bc7df5f227a49bb81ce44fdf204 Mon Sep 17 00:00:00 2001
From: Liu Hui <liuhui1610@gmail.com>
Date: Thu, 18 Jan 2024 12:00:00 +0800
Subject: [PATCH] Improve Python shell completion (bug#68559)

* lisp/progmodes/python.el (python-shell-completion-setup-code): Fix
the completion code of IPython.  Change the return value to JSON
string and ...
(python-shell-completion-get-completions): ... simplify parsing.
(inferior-python-mode): Update docstring.
(python-shell-readline-completer): New option.
(python-shell-readline-completer-delims): New variable indicating the
word delimiters of readline completer.
(python-shell--readline-jedi-setup-code)
(python-shell--readline-ipython-setup-code): New internal variables.
(python-shell-completion-native-setup): Setup a suitable readline
completer and set the completer delimiter.
(python-shell-completion-native-get-completions): Convert output
string to completions properly.
(python-shell--get-multiline-input)
(python-shell--extra-completion-context)
(python-shell-completion-extra-context): New functions.
(python-shell-completion-at-point): Send text beginning from the line
start if the completion backend does not need word splitting.  Remove
the detection of import statement because it is not needed anymore.
Create proper completion table based on completions returned from
different backends.

* test/lisp/progmodes/python-tests.el (python-tests--completion-module)
(python-tests--completion-parameters)
(python-tests--completion-extra-context): New helper functions.
(python-shell-completion-at-point-jedi-completer)
(python-shell-completion-at-point-ipython): New tests.

* etc/NEWS: Announce the change.
---
 etc/NEWS                            |   6 +
 lisp/progmodes/python.el            | 313 +++++++++++++++++++++++-----
 test/lisp/progmodes/python-tests.el |  91 ++++++++
 3 files changed, 358 insertions(+), 52 deletions(-)

diff --git a/etc/NEWS b/etc/NEWS
index 816613de4ec..2c8e70e4a15 100644
--- a/etc/NEWS
+++ b/etc/NEWS
@@ -1086,6 +1086,12 @@ instead of:
 This allows the user to specify command line arguments to the non
 interactive Python interpreter specified by 'python-interpreter'.
 
+*** New user option 'python-shell-readline-completer'.
+This allows the user to specify the readline completer used for Python
+shell completion.  The default is 'auto', which means a suitable
+completer will be configured automatically according to the Python
+interpreter.
+
 ** use-package
 
 +++
diff --git a/lisp/progmodes/python.el b/lisp/progmodes/python.el
index 9d840efb9da..0291f398a65 100644
--- a/lisp/progmodes/python.el
+++ b/lisp/progmodes/python.el
@@ -128,9 +128,9 @@ ;;; Commentary:
 ;; receiving escape sequences (with some limitations, i.e. completion
 ;; in blocks does not work).  The code executed for the "fallback"
 ;; completion can be found in `python-shell-completion-setup-code' and
-;; `python-shell-completion-string-code' variables.  Their default
-;; values enable completion for both CPython and IPython, and probably
-;; any readline based shell (it's known to work with PyPy).  If your
+;; `python-shell-completion-get-completions'.  Their default values
+;; enable completion for both CPython and IPython, and probably any
+;; readline based shell (it's known to work with PyPy).  If your
 ;; Python installation lacks readline (like CPython for Windows),
 ;; installing pyreadline (URL `https://ipython.org/pyreadline.html')
 ;; should suffice.  To troubleshoot why you are not getting any
@@ -3604,7 +3604,6 @@ (define-derived-mode inferior-python-mode comint-mode "Inferior Python"
 `python-shell-prompt-block-regexp',
 `python-shell-font-lock-enable',
 `python-shell-completion-setup-code',
-`python-shell-completion-string-code',
 `python-eldoc-setup-code',
 `python-ffap-setup-code' can
 customize this mode for different Python interpreters.
@@ -4244,8 +4243,9 @@ (defcustom python-shell-completion-setup-code
     completions = []
     completer = None
 
+    import json
     try:
-        import readline
+        import readline, re
 
         try:
             import __builtin__
@@ -4256,16 +4256,29 @@ (defcustom python-shell-completion-setup-code
 
         is_ipython = ('__IPYTHON__' in builtins or
                       '__IPYTHON__active' in builtins)
-        splits = text.split()
-        is_module = splits and splits[0] in ('from', 'import')
-
-        if is_ipython and is_module:
-            from IPython.core.completerlib import module_completion
-            completions = module_completion(text.strip())
-        elif is_ipython and '__IP' in builtins:
-            completions = __IP.complete(text)
-        elif is_ipython and 'get_ipython' in builtins:
-            completions = get_ipython().Completer.all_completions(text)
+
+        if is_ipython and 'get_ipython' in builtins:
+            def filter_c(prefix, c):
+                if re.match('_+(i?[0-9]+)?$', c):
+                    return False
+                elif c[0] == '%' and not re.match('[%a-zA-Z]+$', prefix):
+                    return False
+                return True
+
+            import IPython
+            try:
+                if IPython.version_info[0] >= 6:
+                    from IPython.core.completer import provisionalcompleter
+                    with provisionalcompleter():
+                        completions = [
+                            [c.text, c.start, c.end, c.type or '?', c.signature or '']
+                             for c in get_ipython().Completer.completions(text, len(text))
+                             if filter_c(text, c.text)]
+                else:
+                    part, matches = get_ipython().Completer.complete(line_buffer=text)
+                    completions = [text + m[len(part):] for m in matches if filter_c(text, m)]
+            except:
+                pass
         else:
             # Try to reuse current completer.
             completer = readline.get_completer()
@@ -4288,7 +4301,7 @@ (defcustom python-shell-completion-setup-code
     finally:
         if getattr(completer, 'PYTHON_EL_WRAPPED', False):
             completer.print_mode = True
-    return completions"
+    return json.dumps(completions)"
   "Code used to setup completion in inferior Python processes."
   :type 'string)
 
@@ -4329,6 +4342,78 @@ (defcustom python-shell-completion-native-try-output-timeout 1.0
   :version "25.1"
   :type 'float)
 
+(defcustom python-shell-readline-completer 'auto
+  "The readline completer used for Python shell completion.
+If the value is non-nil, Python shell will setup the readline
+completer unless it has been set elsewhere (e.g. in the
+PYTHONSTARTUP file).  Below are possible values:
+- `auto': the completer is determined according to the
+interpreter.  Specifically, the IPython completer, defined in
+`python-shell--readline-ipython-setup-code', is used when the
+interpreter is ipython, otherwise the Jedi completer is used.
+- a string: Python code to setup the readline.  See
+`python-shell--readline-jedi-setup-code' for reference.
+- `nil': Python shell will do nothing.
+
+In any case, if the completer is not set successfully in the end,
+fallback to the built-in rlcompleter."
+  :type '(choice (const  :tag "Automatic" auto)
+                 (const  :tag "No configuration" nil)
+                 (string :tag "Python setup code"))
+  :version "30.1")
+
+(defvar python-shell-readline-completer-delims nil
+  "Word delimiters used by the readline completer.
+It is automatically set by Python shell.")
+
+(defconst python-shell--readline-jedi-setup-code
+  "
+def __PYTHON_EL_setup_readline_completer():
+    from jedi.utils import setup_readline
+    setup_readline()"
+  "Code used to setup readline completer with Jedi.")
+
+(defconst python-shell--readline-ipython-setup-code
+  "
+def __PYTHON_EL_setup_readline_completer():
+    import readline, re, json, IPython
+
+    class __ipython_RL:
+        def __init__(self, v):
+            self.version = v
+
+        def filter(self, prefix, c):
+            if re.match('_+(i?[0-9]+)?$', c):
+                return False
+            elif c[0] == '%' and not re.match('[%a-zA-Z]+$', prefix):
+                return False
+            return True
+
+        def complete(self, text, state):
+            if state == 0:
+                try:
+                    if self.version >= 6:
+                        from IPython.core.completer import provisionalcompleter
+                        with provisionalcompleter():
+                            self.matches = [json.dumps([
+                                [c.text, c.start, c.end, c.type or '?', c.signature or '']
+                                for c in get_ipython().Completer.completions(text, len(text))
+                                if self.filter(text, c.text)])]
+                    else:
+                        part, matches = get_ipython().Completer.complete(line_buffer=text)
+                        self.matches = [text + m[len(part):] for m in matches
+                                        if self.filter(text, m)]
+                except Exception:
+                    pass
+            try:
+                return self.matches[state]
+            except IndexError:
+                return None
+
+    readline.set_completer(__ipython_RL(IPython.version_info[0]).complete)
+    readline.set_completer_delims('')"
+  "Code used to setup readline completer for IPython.")
+
 (defvar python-shell-completion-native-redirect-buffer
   " *Python completions redirect*"
   "Buffer to be used to redirect output of readline commands.")
@@ -4352,7 +4437,20 @@ (defun python-shell-completion-native-try ()
 (defun python-shell-completion-native-setup ()
   "Try to setup native completion, return non-nil on success."
   (let* ((process (python-shell-get-process))
-         (output (python-shell-send-string-no-output "
+         (completer (pcase python-shell-readline-completer
+                      ('auto
+                       (if (string-match-p "ipython[23]?\\'" python-shell-interpreter)
+                           python-shell--readline-ipython-setup-code
+                         python-shell--readline-jedi-setup-code))
+                      ((pred stringp) python-shell-readline-completer)
+                      (_ "")))
+         (output (python-shell-send-string-no-output
+                  (concat "
+try:
+    del __PYTHON_EL_setup_readline_completer
+except:
+    pass
+" completer "
 def __PYTHON_EL_native_completion_setup():
     try:
         import readline
@@ -4425,11 +4523,23 @@ (defun python-shell-completion-native-setup ()
                 else:
                     return completion
 
+        def is_rlcompleter(completer):
+            try:
+                if completer.__self__.__module__ == 'rlcompleter':
+                    return True
+                else:
+                    return False
+            except Exception:
+                return False
+
         completer = readline.get_completer()
 
-        if not completer:
-            # Used as last resort to avoid breaking customizations.
-            import rlcompleter
+        if not completer or is_rlcompleter(completer):
+            try:
+                __PYTHON_EL_setup_readline_completer()
+            except:
+                # Used as last resort to avoid breaking customizations.
+                import rlcompleter
             completer = readline.get_completer()
 
         if completer and not getattr(completer, 'PYTHON_EL_WRAPPED', False):
@@ -4464,9 +4574,13 @@ (defun python-shell-completion-native-setup ()
         print ('python.el: native completion setup failed, %s: %s'
                % sys.exc_info()[:2])
 
-__PYTHON_EL_native_completion_setup()" process)))
+__PYTHON_EL_native_completion_setup()") process)))
     (when (string-match-p "python\\.el: native completion setup loaded"
                           output)
+      (setq-local python-shell-readline-completer-delims
+                  (string-trim-right
+                   (python-shell-send-string-no-output
+                    "import readline; print(readline.get_completer_delims())")))
       (python-shell-completion-native-try))))
 
 (defun python-shell-completion-native-turn-off (&optional msg)
@@ -4534,6 +4648,8 @@ (defun python-shell-completion-native-get-completions (process input)
     (let* ((original-filter-fn (process-filter process))
            (redirect-buffer (get-buffer-create
                              python-shell-completion-native-redirect-buffer))
+           (sep (if (string= python-shell-readline-completer-delims "")
+                    "[\n\r]+" "[ \f\t\n\r\v()]+"))
            (trigger "\t")
            (new-input (concat input trigger))
            (input-length
@@ -4576,28 +4692,76 @@ (defun python-shell-completion-native-get-completions (process input)
                      process python-shell-completion-native-output-timeout
                      comint-redirect-finished-regexp)
                 (re-search-backward "0__dummy_completion__" nil t)
-                (cl-remove-duplicates
-                 (split-string
-                  (buffer-substring-no-properties
-                   (line-beginning-position) (point-min))
-                  "[ \f\t\n\r\v()]+" t)
-                 :test #'string=))))
+                (let ((str (buffer-substring-no-properties
+                            (line-beginning-position) (point-min))))
+                  (if (string= "[" (substring str 0 1))
+                      (condition-case nil
+                          (python--parse-json-array str)
+                        (t (cl-remove-duplicates (split-string str sep t)
+                                                 :test #'string=)))
+                    (cl-remove-duplicates (split-string str sep t)
+                                          :test #'string=))))))
         (set-process-filter process original-filter-fn)))))
 
 (defun python-shell-completion-get-completions (process input)
   "Get completions of INPUT using PROCESS."
   (with-current-buffer (process-buffer process)
-    (let ((completions
-           (python-util-strip-string
-            (python-shell-send-string-no-output
-             (format
-              "%s\nprint(';'.join(__PYTHON_EL_get_completions(%s)))"
+    (python--parse-json-array
+     (python-shell-send-string-no-output
+      (format "%s\nprint(__PYTHON_EL_get_completions(%s))"
               python-shell-completion-setup-code
               (python-shell--encode-string input))
-             process))))
-      (when (> (length completions) 2)
-        (split-string completions
-                      "^'\\|^\"\\|;\\|'$\\|\"$" t)))))
+      process))))
+
+(defun python-shell--get-multiline-input ()
+  "Return lines at a multi-line input in Python shell."
+  (save-excursion
+    (let ((p (point)) lines)
+      (when (progn
+              (beginning-of-line)
+              (looking-back python-shell-prompt-block-regexp (pos-bol)))
+        (push (buffer-substring-no-properties (point) p) lines)
+        (while (progn (comint-previous-prompt 1)
+                      (looking-back python-shell-prompt-block-regexp (pos-bol)))
+          (push (buffer-substring-no-properties (point) (pos-eol)) lines))
+        (push (buffer-substring-no-properties (point) (pos-eol)) lines))
+      lines)))
+
+(defun python-shell--extra-completion-context ()
+  "Get extra completion context of current input in Python shell."
+  (let ((lines (python-shell--get-multiline-input))
+        (python-indent-guess-indent-offset nil))
+    (when (not (zerop (length lines)))
+      (with-temp-buffer
+        (delay-mode-hooks
+          (insert (string-join lines "\n"))
+          (python-mode)
+          (python-shell-completion-extra-context))))))
+
+(defun python-shell-completion-extra-context (&optional pos)
+  "Get extra completion context at position POS in Python buffer.
+If optional argument POS is nil, use current position.
+
+Readline completers could use current line as the completion
+context, which may be insufficient.  In this function, extra
+context (e.g. multi-line function call) is found and reformatted
+as one line, which is required by native completion."
+  (let (bound p)
+    (save-excursion
+      (and pos (goto-char pos))
+      (setq bound (pos-bol))
+      (python-nav-up-list -1)
+      (when (and (< (point) bound)
+                 (or
+                  (looking-back
+                   (python-rx (group (+ (or "." symbol-name)))) (pos-bol) t)
+                  (progn
+                    (forward-line 0)
+                    (looking-at "^[ \t]*\\(from \\)"))))
+        (setq p (match-beginning 1))))
+    (when p
+      (replace-regexp-in-string
+       "\n[ \t]*" "" (buffer-substring-no-properties p (1- bound))))))
 
 (defvar-local python-shell--capf-cache nil
   "Variable to store cached completions and invalidation keys.")
@@ -4612,21 +4776,26 @@ (defun python-shell-completion-at-point (&optional process)
                          ;; Working on a shell buffer: use prompt end.
                          (cdr (python-util-comint-last-prompt))
                        (line-beginning-position)))
-         (import-statement
-          (when (string-match-p
-                 (rx (* space) word-start (or "from" "import") word-end space)
-                 (buffer-substring-no-properties line-start (point)))
-            (buffer-substring-no-properties line-start (point))))
+         (no-delims
+          (and (not (if is-shell-buffer
+                        (eq 'font-lock-comment-face
+                            (get-text-property (1- (point)) 'face))
+                      (python-syntax-context 'comment)))
+               (with-current-buffer (process-buffer process)
+                 (if python-shell-completion-native-enable
+                     (string= python-shell-readline-completer-delims "")
+                   (string-match-p "ipython[23]?\\'" python-shell-interpreter)))))
          (start
           (if (< (point) line-start)
               (point)
             (save-excursion
-              (if (not (re-search-backward
-                        (python-rx
-                         (or whitespace open-paren close-paren
-                             string-delimiter simple-operator))
-                        line-start
-                        t 1))
+              (if (or no-delims
+                      (not (re-search-backward
+                            (python-rx
+                             (or whitespace open-paren close-paren
+                                 string-delimiter simple-operator))
+                            line-start
+                            t 1)))
                   line-start
                 (forward-char (length (match-string-no-properties 0)))
                 (point)))))
@@ -4666,18 +4835,58 @@ (defun python-shell-completion-at-point (&optional process)
                   (t #'python-shell-completion-native-get-completions))))
          (prev-prompt (car python-shell--capf-cache))
          (re (or (cadr python-shell--capf-cache) regexp-unmatchable))
-         (prefix (buffer-substring-no-properties start end)))
+         (prefix (buffer-substring-no-properties start end))
+         (prefix-offset 0)
+         (extra-context (when no-delims
+                          (if is-shell-buffer
+                              (python-shell--extra-completion-context)
+                            (python-shell-completion-extra-context))))
+         (extra-offset (length extra-context)))
+    (unless (zerop extra-offset)
+      (setq prefix (concat extra-context prefix)))
     ;; To invalidate the cache, we check if the prompt position or the
     ;; completion prefix changed.
     (unless (and (equal prev-prompt (car prompt-boundaries))
-                 (string-match re prefix))
+                 (string-match re prefix)
+                 (setq prefix-offset (- (length prefix) (match-end 1))))
       (setq python-shell--capf-cache
             `(,(car prompt-boundaries)
               ,(if (string-empty-p prefix)
                    regexp-unmatchable
-                 (concat "\\`" (regexp-quote prefix) "\\(?:\\sw\\|\\s_\\)*\\'"))
-              ,@(funcall completion-fn process (or import-statement prefix)))))
-    (list start end (cddr python-shell--capf-cache))))
+                 (concat "\\`\\(" (regexp-quote prefix) "\\)\\(?:\\sw\\|\\s_\\)*\\'"))
+              ,@(funcall completion-fn process prefix))))
+    (let ((cands (cddr python-shell--capf-cache)))
+      (cond
+       ((stringp (car cands))
+        (if no-delims
+            ;; Reduce completion candidates due to long prefix.
+            (if-let ((Lp (length prefix))
+                     ((string-match "\\(\\sw\\|\\s_\\)+\\'" prefix))
+                     (L (match-beginning 0)))
+                ;; If extra-offset is not zero:
+                ;;                  start              end
+                ;; o------------------o---------o-------o
+                ;; |<- extra-offset ->|
+                ;; |<----------- L ------------>|
+                ;;                          new-start
+                (list (+ start L (- extra-offset)) end
+                      (mapcar (lambda (s) (substring s L)) cands))
+              (list end end (mapcar (lambda (s) (substring s Lp)) cands)))
+          (list start end cands)))
+       ;; python-shell-completion(-native)-get-completions may produce
+       ;; a list of (text start end type signature) for completion.
+       ;; See `python-shell--readline-ipython-setup-code' and
+       ;; `python-shell-completion-setup-code'.
+       ((consp (car cands))
+        (list (+ start (nth 1 (car cands)) (- extra-offset))
+              ;; Candidates may be cached, so the end position should
+              ;; be adjusted according to current completion prefix.
+              (+ start (nth 2 (car cands)) (- extra-offset) prefix-offset)
+              cands
+              :annotation-function
+              (lambda (c) (concat " " (nth 3 (assoc c cands))))
+              :company-docsig
+              (lambda (c) (nth 4 (assoc c cands)))))))))
 
 (define-obsolete-function-alias
   'python-shell-completion-complete-at-point
diff --git a/test/lisp/progmodes/python-tests.el b/test/lisp/progmodes/python-tests.el
index 59957ff0712..9f9914896be 100644
--- a/test/lisp/progmodes/python-tests.el
+++ b/test/lisp/progmodes/python-tests.el
@@ -4799,6 +4799,97 @@ (ert-deftest python-shell-completion-at-point-native-1 ()
      (end-of-line 0)
      (should-not (nth 2 (python-shell-completion-at-point))))))
 
+(defun python-tests--completion-module ()
+  "Check if modules can be completed in Python shell."
+  (insert "import datet")
+  (completion-at-point)
+  (beginning-of-line)
+  (should (looking-at-p "import datetime"))
+  (kill-line)
+  (insert "from datet")
+  (completion-at-point)
+  (beginning-of-line)
+  (should (looking-at-p "from datetime"))
+  (end-of-line)
+  (insert " import timed")
+  (completion-at-point)
+  (beginning-of-line)
+  (should (looking-at-p "from datetime import timedelta"))
+  (kill-line))
+
+(defun python-tests--completion-parameters ()
+  "Check if parameters can be completed in Python shell."
+  (insert "import re")
+  (comint-send-input)
+  (python-tests-shell-wait-for-prompt)
+  (insert "re.split('b', 'abc', maxs")
+  (completion-at-point)
+  (should (string= "re.split('b', 'abc', maxsplit="
+                   (buffer-substring (line-beginning-position) (point))))
+  (insert "0, ")
+  (should (python-shell-completion-at-point))
+  ;; Test if cache is used.
+  (cl-letf (((symbol-function 'python-shell-completion-get-completions)
+             'ignore)
+            ((symbol-function 'python-shell-completion-native-get-completions)
+             'ignore))
+    (insert "fla")
+    (completion-at-point)
+    (should (string= "re.split('b', 'abc', maxsplit=0, flags="
+                     (buffer-substring (line-beginning-position) (point)))))
+  (beginning-of-line)
+  (kill-line))
+
+(defun python-tests--completion-extra-context ()
+  "Check if extra context is used for completion."
+  (insert "re.split('b', 'abc',")
+  (comint-send-input)
+  (python-tests-shell-wait-for-prompt)
+  (insert "maxs")
+  (completion-at-point)
+  (should (string= "maxsplit="
+                   (buffer-substring (line-beginning-position) (point))))
+  (insert "0)")
+  (comint-send-input)
+  (python-tests-shell-wait-for-prompt)
+  (insert "from re import (")
+  (comint-send-input)
+  (python-tests-shell-wait-for-prompt)
+  (insert "IGN")
+  (completion-at-point)
+  (should (string= "IGNORECASE"
+                   (buffer-substring (line-beginning-position) (point)))))
+
+(ert-deftest python-shell-completion-at-point-jedi-completer ()
+  "Check if Python shell completion works with Jedi."
+  (skip-unless (executable-find python-tests-shell-interpreter))
+  (python-tests-with-temp-buffer-with-shell
+   ""
+   (python-shell-with-shell-buffer
+     (python-shell-completion-native-turn-on)
+     (skip-unless (string= python-shell-readline-completer-delims ""))
+     (python-tests--completion-module)
+     (python-tests--completion-parameters)
+     (python-tests--completion-extra-context))))
+
+(ert-deftest python-shell-completion-at-point-ipython ()
+  "Check if Python shell completion works for IPython."
+  (let ((python-shell-interpreter "ipython")
+        (python-shell-interpreter-args "-i --simple-prompt"))
+    (skip-unless
+     (and
+      (executable-find python-shell-interpreter)
+      (eql (call-process python-shell-interpreter nil nil nil "--version") 0)))
+    (python-tests-with-temp-buffer-with-shell
+     ""
+     (python-shell-with-shell-buffer
+       (python-shell-completion-native-turn-off)
+       (python-tests--completion-module)
+       (python-tests--completion-parameters)
+       (python-shell-completion-native-turn-on)
+       (python-tests--completion-module)
+       (python-tests--completion-parameters)
+       (python-tests--completion-extra-context)))))
 
 \f
 ;;; PDB Track integration
-- 
2.39.2

bug#68559: [PATCH] Improve Python shell completion

[kobarity]

Liu Hui wrote:
> kobarity <kobarity@gmail.com> writes:
> > I'm experiencing strange behavior regarding completion of import
> > statement in a block in Python buffer.  If I try to type the following
> > lines and then try to complete it, it will fail.
> >
> > #+begin_src python
> > try:
> >     im
> > #+end_src
> 
> The problem can be reproduced by running python with jedi completer in
> a terminal. The reason is that readline completer can only see the
> text in the current line, i.e. "    im"; in this case, jedi does not
> complete the text to "    import".
> 
> Similarly, the jedi completer cannot complete "except" after ex:
> 
> try:
>     pass
> ex|
> 
> > However, when I try to complete at the beginning of the second line:
> >
> > #+begin_src python
> > try:
> >
> > #+end_src
> >
> >
> > "import" keyword also appears as a candidate.  If I cancel the
> > candidates and type "im" and try to complete it, it will succeed.
> 
> It is because jedi produces completions including "    import" for
> blank string "    ". Due to the same prefix between "    " and
> "    im", completion cache is reused for the latter. Then "    import"
> can be completed.
> 
> It is more a limitation of readline completer than a problem with
> jedi, as we cannot provide proper completion context for jedi. We may
> define a custom completer to combine jedi and rlcompleter, e.g.
> 
> (setq python-shell-readline-completer "
> def __PYTHON_EL_setup_readline_completer():
>     import readline, rlcompleter
>     import re, sys, os, __main__
>     from jedi import Interpreter
> 
>     class MyJediRL:
>         def __init__(self):
>             self.rlcompleter = rlcompleter.Completer()
>             self.rldelim = readline.get_completer_delims()
> 
>         def complete(self, text, state):
>             if state == 0:
>                 sys.path.insert(0, os.getcwd())
>                 try:
>                     interpreter = Interpreter(text, [__main__.__dict__])
>                     completions = interpreter.complete(fuzzy=False)
>                     self.matches = [
>                         text[:len(text) - c._like_name_length] + c.name_with_symbols
>                         for c in completions
>                     ]
> 
>                     # try rlcompleter
>                     sub = re.split('[' + re.escape(self.rldelim) + ']', text)[-1]
>                     i = 0
>                     while True:
>                         completion = self.rlcompleter.complete(sub, i)
>                         if not completion:
>                             break
>                         i += 1
>                         completion = text[:len(text)-len(sub)] + completion.rstrip(' ()')
>                         if completion not in self.matches:
>                             self.matches.append(completion)
>                 except:
>                     raise
>                 finally:
>                     sys.path.pop(0)
>             try:
>                 return self.matches[state]
>             except IndexError:
>                 return None
> 
>     readline.set_completer(MyJediRL().complete)
>     readline.set_completer_delims('')")

Thank you for the detailed explanation and the workaround.  I
confirmed that the problem is solved by the above workaround.  Just to
confirm, are you of the opinion that this workaround should not be the
default?

> > Another thing I noticed is the multi-line import statement.  If the
> > import statement is one-line, each items (IGNORECASE and MULTILINE in
> > the example below) can be completed.
> >
> > #+begin_src python
> > from re import IGNORECASE, MULTILINE
> > #+end_src
> >
> >
> > However, they cannot be completed if the import statement spans
> > multi-line.
> >
> > #+begin_src python
> > from re import (
> >     IGN
> > #+end_src
> >
> > This happens in both Python buffer and Python Shell buffer.  Perhaps
> > this is a limitation of Jedi completer?
> 
> Yes. Because readline completer cannot see cross-line context, I added
> the function "python-shell-completion-extra-context" in previous patch
> to address the case of multi-line function call. I have updated the
> attached patch to handle multi-line import statement.

Thank you very much.  I confirmed that the new patch allows completion
of multi-line import statements.

bug#68559: [PATCH] Improve Python shell completion

[Liu Hui]

[-- Attachment #1: Type: text/plain, Size: 2949 bytes --]

On Sun, Feb 4, 2024 at 10:35 PM kobarity <kobarity@gmail.com> wrote:

> > It is more a limitation of readline completer than a problem with
> > jedi, as we cannot provide proper completion context for jedi. We may
> > define a custom completer to combine jedi and rlcompleter, e.g.
> >
> > (setq python-shell-readline-completer "
> > def __PYTHON_EL_setup_readline_completer():
> >     import readline, rlcompleter
> >     import re, sys, os, __main__
> >     from jedi import Interpreter
> >
> >     class MyJediRL:
> >         def __init__(self):
> >             self.rlcompleter = rlcompleter.Completer()
> >             self.rldelim = readline.get_completer_delims()
> >
> >         def complete(self, text, state):
> >             if state == 0:
> >                 sys.path.insert(0, os.getcwd())
> >                 try:
> >                     interpreter = Interpreter(text, [__main__.__dict__])
> >                     completions = interpreter.complete(fuzzy=False)
> >                     self.matches = [
> >                         text[:len(text) - c._like_name_length] + c.name_with_symbols
> >                         for c in completions
> >                     ]
> >
> >                     # try rlcompleter
> >                     sub = re.split('[' + re.escape(self.rldelim) + ']', text)[-1]
> >                     i = 0
> >                     while True:
> >                         completion = self.rlcompleter.complete(sub, i)
> >                         if not completion:
> >                             break
> >                         i += 1
> >                         completion = text[:len(text)-len(sub)] + completion.rstrip(' ()')
> >                         if completion not in self.matches:
> >                             self.matches.append(completion)
> >                 except:
> >                     raise
> >                 finally:
> >                     sys.path.pop(0)
> >             try:
> >                 return self.matches[state]
> >             except IndexError:
> >                 return None
> >
> >     readline.set_completer(MyJediRL().complete)
> >     readline.set_completer_delims('')")
>
> Thank you for the detailed explanation and the workaround.  I
> confirmed that the problem is solved by the above workaround.  Just to
> confirm, are you of the opinion that this workaround should not be the
> default?

I'm not sure if we should add more Python code in the form of strings
to python.el, which increases maintenance burden IMO. Maybe they could
be distributed separately at ELPA/Git forges.

Actually, I'm considering simplifying this patch to mainly fix the bug
that python shell completion doesn't respect the delimiter of readline
completer. The new patch has been attached. It should support
the completer defined in the PYTHONSTARTUP file, e.g., jedi or a
custom completer like the above one. WDYT?

[-- Attachment #2: 0001-Respect-the-delimiter-of-completer-in-Python-shell-c.patch --]
[-- Type: text/x-patch, Size: 20619 bytes --]

From 082596e908ee45d18c50c3b9f0b5020b779adb77 Mon Sep 17 00:00:00 2001
From: Liu Hui <liuhui1610@gmail.com>
Date: Thu, 18 Jan 2024 12:00:00 +0800
Subject: [PATCH] Respect the delimiter of completer in Python shell completion

* lisp/progmodes/python.el: (python-shell-completion-setup-code): Fix
the completion code of IPython.  Change the return value to JSON string
and ...
(python-shell-completion-get-completions): ... simplify parsing.
(inferior-python-mode): Update docstring.
(python-shell-readline-completer-delims): New variable indicating the
word delimiters of readline completer.
(python-shell-completion-native-setup): Set the completer delimiter.
(python-shell-completion-native-get-completions): Convert output string
to completions properly.
(python-shell--get-multiline-input)
(python-shell--extra-completion-context)
(python-shell-completion-extra-context): New functions.
(python-shell-completion-at-point): Send text beginning from the line
start if the completion backend does not need word splitting.  Remove
the detection of import statement because it is not needed anymore.
Create proper completion table based on completions returned from
different backends.

* test/lisp/progmodes/python-tests.el (python-tests--completion-module)
(python-tests--completion-parameters)
(python-tests--completion-extra-context): New helper functions.
(python-shell-completion-at-point-jedi-completer)
(python-shell-completion-at-point-ipython): New tests.

* etc/NEWS: Announce the change. (bug#68559)
---
 lisp/progmodes/python.el            | 212 ++++++++++++++++++++++------
 test/lisp/progmodes/python-tests.el |  92 ++++++++++++
 2 files changed, 257 insertions(+), 47 deletions(-)

diff --git a/lisp/progmodes/python.el b/lisp/progmodes/python.el
index 9d840efb9da..45336c8396f 100644
--- a/lisp/progmodes/python.el
+++ b/lisp/progmodes/python.el
@@ -128,9 +128,9 @@ ;;; Commentary:
 ;; receiving escape sequences (with some limitations, i.e. completion
 ;; in blocks does not work).  The code executed for the "fallback"
 ;; completion can be found in `python-shell-completion-setup-code' and
-;; `python-shell-completion-string-code' variables.  Their default
-;; values enable completion for both CPython and IPython, and probably
-;; any readline based shell (it's known to work with PyPy).  If your
+;; `python-shell-completion-get-completions'.  Their default values
+;; enable completion for both CPython and IPython, and probably any
+;; readline based shell (it's known to work with PyPy).  If your
 ;; Python installation lacks readline (like CPython for Windows),
 ;; installing pyreadline (URL `https://ipython.org/pyreadline.html')
 ;; should suffice.  To troubleshoot why you are not getting any
@@ -3604,7 +3604,6 @@ (define-derived-mode inferior-python-mode comint-mode "Inferior Python"
 `python-shell-prompt-block-regexp',
 `python-shell-font-lock-enable',
 `python-shell-completion-setup-code',
-`python-shell-completion-string-code',
 `python-eldoc-setup-code',
 `python-ffap-setup-code' can
 customize this mode for different Python interpreters.
@@ -4244,8 +4243,9 @@ (defcustom python-shell-completion-setup-code
     completions = []
     completer = None

+    import json
     try:
-        import readline
+        import readline, re

         try:
             import __builtin__
@@ -4256,16 +4256,29 @@ (defcustom python-shell-completion-setup-code

         is_ipython = ('__IPYTHON__' in builtins or
                       '__IPYTHON__active' in builtins)
-        splits = text.split()
-        is_module = splits and splits[0] in ('from', 'import')
-
-        if is_ipython and is_module:
-            from IPython.core.completerlib import module_completion
-            completions = module_completion(text.strip())
-        elif is_ipython and '__IP' in builtins:
-            completions = __IP.complete(text)
-        elif is_ipython and 'get_ipython' in builtins:
-            completions = get_ipython().Completer.all_completions(text)
+
+        if is_ipython and 'get_ipython' in builtins:
+            def filter_c(prefix, c):
+                if re.match('_+(i?[0-9]+)?$', c):
+                    return False
+                elif c[0] == '%' and not re.match('[%a-zA-Z]+$', prefix):
+                    return False
+                return True
+
+            import IPython
+            try:
+                if IPython.version_info[0] >= 6:
+                    from IPython.core.completer import provisionalcompleter
+                    with provisionalcompleter():
+                        completions = [
+                            [c.text, c.start, c.end, c.type or '?', c.signature or '']
+                             for c in get_ipython().Completer.completions(text, len(text))
+                             if filter_c(text, c.text)]
+                else:
+                    part, matches = get_ipython().Completer.complete(line_buffer=text)
+                    completions = [text + m[len(part):] for m in matches if filter_c(text, m)]
+            except:
+                pass
         else:
             # Try to reuse current completer.
             completer = readline.get_completer()
@@ -4288,7 +4301,7 @@ (defcustom python-shell-completion-setup-code
     finally:
         if getattr(completer, 'PYTHON_EL_WRAPPED', False):
             completer.print_mode = True
-    return completions"
+    return json.dumps(completions)"
   "Code used to setup completion in inferior Python processes."
   :type 'string)

@@ -4329,6 +4342,10 @@ (defcustom python-shell-completion-native-try-output-timeout 1.0
   :version "25.1"
   :type 'float)

+(defvar python-shell-readline-completer-delims nil
+  "Word delimiters used by the readline completer.
+It is automatically set by Python shell.")
+
 (defvar python-shell-completion-native-redirect-buffer
   " *Python completions redirect*"
   "Buffer to be used to redirect output of readline commands.")
@@ -4467,6 +4484,10 @@ (defun python-shell-completion-native-setup ()
 __PYTHON_EL_native_completion_setup()" process)))
     (when (string-match-p "python\\.el: native completion setup loaded"
                           output)
+      (setq-local python-shell-readline-completer-delims
+                  (string-trim-right
+                   (python-shell-send-string-no-output
+                    "import readline; print(readline.get_completer_delims())")))
       (python-shell-completion-native-try))))

 (defun python-shell-completion-native-turn-off (&optional msg)
@@ -4534,6 +4555,8 @@ (defun python-shell-completion-native-get-completions (process input)
     (let* ((original-filter-fn (process-filter process))
            (redirect-buffer (get-buffer-create
                              python-shell-completion-native-redirect-buffer))
+           (sep (if (string= python-shell-readline-completer-delims "")
+                    "[\n\r]+" "[ \f\t\n\r\v()]+"))
            (trigger "\t")
            (new-input (concat input trigger))
            (input-length
@@ -4576,28 +4599,80 @@ (defun python-shell-completion-native-get-completions (process input)
                      process python-shell-completion-native-output-timeout
                      comint-redirect-finished-regexp)
                 (re-search-backward "0__dummy_completion__" nil t)
-                (cl-remove-duplicates
-                 (split-string
-                  (buffer-substring-no-properties
-                   (line-beginning-position) (point-min))
-                  "[ \f\t\n\r\v()]+" t)
-                 :test #'string=))))
+                (let ((str (buffer-substring-no-properties
+                            (line-beginning-position) (point-min))))
+                  ;; The readline completer is allowed to return a list
+                  ;; of (text start end type signature) as a JSON
+                  ;; string.  See the return value for IPython in
+                  ;; `python-shell-completion-setup-code'.
+                  (if (string= "[" (substring str 0 1))
+                      (condition-case nil
+                          (python--parse-json-array str)
+                        (t (cl-remove-duplicates (split-string str sep t)
+                                                 :test #'string=)))
+                    (cl-remove-duplicates (split-string str sep t)
+                                          :test #'string=))))))
         (set-process-filter process original-filter-fn)))))

 (defun python-shell-completion-get-completions (process input)
   "Get completions of INPUT using PROCESS."
   (with-current-buffer (process-buffer process)
-    (let ((completions
-           (python-util-strip-string
-            (python-shell-send-string-no-output
-             (format
-              "%s\nprint(';'.join(__PYTHON_EL_get_completions(%s)))"
+    (python--parse-json-array
+     (python-shell-send-string-no-output
+      (format "%s\nprint(__PYTHON_EL_get_completions(%s))"
               python-shell-completion-setup-code
               (python-shell--encode-string input))
-             process))))
-      (when (> (length completions) 2)
-        (split-string completions
-                      "^'\\|^\"\\|;\\|'$\\|\"$" t)))))
+      process))))
+
+(defun python-shell--get-multiline-input ()
+  "Return lines at a multi-line input in Python shell."
+  (save-excursion
+    (let ((p (point)) lines)
+      (when (progn
+              (beginning-of-line)
+              (looking-back python-shell-prompt-block-regexp (pos-bol)))
+        (push (buffer-substring-no-properties (point) p) lines)
+        (while (progn (comint-previous-prompt 1)
+                      (looking-back python-shell-prompt-block-regexp (pos-bol)))
+          (push (buffer-substring-no-properties (point) (pos-eol)) lines))
+        (push (buffer-substring-no-properties (point) (pos-eol)) lines))
+      lines)))
+
+(defun python-shell--extra-completion-context ()
+  "Get extra completion context of current input in Python shell."
+  (let ((lines (python-shell--get-multiline-input))
+        (python-indent-guess-indent-offset nil))
+    (when (not (zerop (length lines)))
+      (with-temp-buffer
+        (delay-mode-hooks
+          (insert (string-join lines "\n"))
+          (python-mode)
+          (python-shell-completion-extra-context))))))
+
+(defun python-shell-completion-extra-context (&optional pos)
+  "Get extra completion context at position POS in Python buffer.
+If optional argument POS is nil, use current position.
+
+Readline completers could use current line as the completion
+context, which may be insufficient.  In this function, extra
+context (e.g. multi-line function call) is found and reformatted
+as one line, which is required by native completion."
+  (let (bound p)
+    (save-excursion
+      (and pos (goto-char pos))
+      (setq bound (pos-bol))
+      (python-nav-up-list -1)
+      (when (and (< (point) bound)
+                 (or
+                  (looking-back
+                   (python-rx (group (+ (or "." symbol-name)))) (pos-bol) t)
+                  (progn
+                    (forward-line 0)
+                    (looking-at "^[ \t]*\\(from \\)"))))
+        (setq p (match-beginning 1))))
+    (when p
+      (replace-regexp-in-string
+       "\n[ \t]*" "" (buffer-substring-no-properties p (1- bound))))))

 (defvar-local python-shell--capf-cache nil
   "Variable to store cached completions and invalidation keys.")
@@ -4612,21 +4687,26 @@ (defun python-shell-completion-at-point (&optional process)
                          ;; Working on a shell buffer: use prompt end.
                          (cdr (python-util-comint-last-prompt))
                        (line-beginning-position)))
-         (import-statement
-          (when (string-match-p
-                 (rx (* space) word-start (or "from" "import") word-end space)
-                 (buffer-substring-no-properties line-start (point)))
-            (buffer-substring-no-properties line-start (point))))
+         (no-delims
+          (and (not (if is-shell-buffer
+                        (eq 'font-lock-comment-face
+                            (get-text-property (1- (point)) 'face))
+                      (python-syntax-context 'comment)))
+               (with-current-buffer (process-buffer process)
+                 (if python-shell-completion-native-enable
+                     (string= python-shell-readline-completer-delims "")
+                   (string-match-p "ipython[23]?\\'" python-shell-interpreter)))))
          (start
           (if (< (point) line-start)
               (point)
             (save-excursion
-              (if (not (re-search-backward
-                        (python-rx
-                         (or whitespace open-paren close-paren
-                             string-delimiter simple-operator))
-                        line-start
-                        t 1))
+              (if (or no-delims
+                      (not (re-search-backward
+                            (python-rx
+                             (or whitespace open-paren close-paren
+                                 string-delimiter simple-operator))
+                            line-start
+                            t 1)))
                   line-start
                 (forward-char (length (match-string-no-properties 0)))
                 (point)))))
@@ -4666,18 +4746,56 @@ (defun python-shell-completion-at-point (&optional process)
                   (t #'python-shell-completion-native-get-completions))))
          (prev-prompt (car python-shell--capf-cache))
          (re (or (cadr python-shell--capf-cache) regexp-unmatchable))
-         (prefix (buffer-substring-no-properties start end)))
+         (prefix (buffer-substring-no-properties start end))
+         (prefix-offset 0)
+         (extra-context (when no-delims
+                          (if is-shell-buffer
+                              (python-shell--extra-completion-context)
+                            (python-shell-completion-extra-context))))
+         (extra-offset (length extra-context)))
+    (unless (zerop extra-offset)
+      (setq prefix (concat extra-context prefix)))
     ;; To invalidate the cache, we check if the prompt position or the
     ;; completion prefix changed.
     (unless (and (equal prev-prompt (car prompt-boundaries))
-                 (string-match re prefix))
+                 (string-match re prefix)
+                 (setq prefix-offset (- (length prefix) (match-end 1))))
       (setq python-shell--capf-cache
             `(,(car prompt-boundaries)
               ,(if (string-empty-p prefix)
                    regexp-unmatchable
-                 (concat "\\`" (regexp-quote prefix) "\\(?:\\sw\\|\\s_\\)*\\'"))
-              ,@(funcall completion-fn process (or import-statement prefix)))))
-    (list start end (cddr python-shell--capf-cache))))
+                 (concat "\\`\\(" (regexp-quote prefix) "\\)\\(?:\\sw\\|\\s_\\)*\\'"))
+              ,@(funcall completion-fn process prefix))))
+    (let ((cands (cddr python-shell--capf-cache)))
+      (cond
+       ((stringp (car cands))
+        (if no-delims
+            ;; Reduce completion candidates due to long prefix.
+            (if-let ((Lp (length prefix))
+                     ((string-match "\\(\\sw\\|\\s_\\)+\\'" prefix))
+                     (L (match-beginning 0)))
+                ;; If extra-offset is not zero:
+                ;;                  start              end
+                ;; o------------------o---------o-------o
+                ;; |<- extra-offset ->|
+                ;; |<----------- L ------------>|
+                ;;                          new-start
+                (list (+ start L (- extra-offset)) end
+                      (mapcar (lambda (s) (substring s L)) cands))
+              (list end end (mapcar (lambda (s) (substring s Lp)) cands)))
+          (list start end cands)))
+       ;; python-shell-completion(-native)-get-completions may produce a
+       ;; list of (text start end type signature) for completion.
+       ((consp (car cands))
+        (list (+ start (nth 1 (car cands)) (- extra-offset))
+              ;; Candidates may be cached, so the end position should
+              ;; be adjusted according to current completion prefix.
+              (+ start (nth 2 (car cands)) (- extra-offset) prefix-offset)
+              cands
+              :annotation-function
+              (lambda (c) (concat " " (nth 3 (assoc c cands))))
+              :company-docsig
+              (lambda (c) (nth 4 (assoc c cands)))))))))

 (define-obsolete-function-alias
   'python-shell-completion-complete-at-point
diff --git a/test/lisp/progmodes/python-tests.el b/test/lisp/progmodes/python-tests.el
index 59957ff0712..af6c199b5bd 100644
--- a/test/lisp/progmodes/python-tests.el
+++ b/test/lisp/progmodes/python-tests.el
@@ -4799,6 +4799,98 @@ (ert-deftest python-shell-completion-at-point-native-1 ()
      (end-of-line 0)
      (should-not (nth 2 (python-shell-completion-at-point))))))

+(defun python-tests--completion-module ()
+  "Check if modules can be completed in Python shell."
+  (insert "import datet")
+  (completion-at-point)
+  (beginning-of-line)
+  (should (looking-at-p "import datetime"))
+  (kill-line)
+  (insert "from datet")
+  (completion-at-point)
+  (beginning-of-line)
+  (should (looking-at-p "from datetime"))
+  (end-of-line)
+  (insert " import timed")
+  (completion-at-point)
+  (beginning-of-line)
+  (should (looking-at-p "from datetime import timedelta"))
+  (kill-line))
+
+(defun python-tests--completion-parameters ()
+  "Check if parameters can be completed in Python shell."
+  (insert "import re")
+  (comint-send-input)
+  (python-tests-shell-wait-for-prompt)
+  (insert "re.split('b', 'abc', maxs")
+  (completion-at-point)
+  (should (string= "re.split('b', 'abc', maxsplit="
+                   (buffer-substring (line-beginning-position) (point))))
+  (insert "0, ")
+  (should (python-shell-completion-at-point))
+  ;; Test if cache is used.
+  (cl-letf (((symbol-function 'python-shell-completion-get-completions)
+             'ignore)
+            ((symbol-function 'python-shell-completion-native-get-completions)
+             'ignore))
+    (insert "fla")
+    (completion-at-point)
+    (should (string= "re.split('b', 'abc', maxsplit=0, flags="
+                     (buffer-substring (line-beginning-position) (point)))))
+  (beginning-of-line)
+  (kill-line))
+
+(defun python-tests--completion-extra-context ()
+  "Check if extra context is used for completion."
+  (insert "re.split('b', 'abc',")
+  (comint-send-input)
+  (python-tests-shell-wait-for-prompt)
+  (insert "maxs")
+  (completion-at-point)
+  (should (string= "maxsplit="
+                   (buffer-substring (line-beginning-position) (point))))
+  (insert "0)")
+  (comint-send-input)
+  (python-tests-shell-wait-for-prompt)
+  (insert "from re import (")
+  (comint-send-input)
+  (python-tests-shell-wait-for-prompt)
+  (insert "IGN")
+  (completion-at-point)
+  (should (string= "IGNORECASE"
+                   (buffer-substring (line-beginning-position) (point)))))
+
+(ert-deftest python-shell-completion-at-point-jedi-completer ()
+  "Check if Python shell completion works when Jedi completer is used."
+  (skip-unless (executable-find python-tests-shell-interpreter))
+  (python-tests-with-temp-buffer-with-shell
+   ""
+   (python-shell-with-shell-buffer
+     (python-shell-completion-native-turn-on)
+     (skip-unless (string= python-shell-readline-completer-delims ""))
+     (python-tests--completion-module)
+     (python-tests--completion-parameters)
+     (python-tests--completion-extra-context))))
+
+(ert-deftest python-shell-completion-at-point-ipython ()
+  "Check if Python shell completion works for IPython."
+  (let ((python-shell-interpreter "ipython")
+        (python-shell-interpreter-args "-i --simple-prompt"))
+    (skip-unless
+     (and
+      (executable-find python-shell-interpreter)
+      (eql (call-process python-shell-interpreter nil nil nil "--version") 0)))
+    (python-tests-with-temp-buffer-with-shell
+     ""
+     (python-shell-with-shell-buffer
+       (python-shell-completion-native-turn-off)
+       (python-tests--completion-module)
+       (python-tests--completion-parameters)
+       (python-shell-completion-native-turn-on)
+       (skip-unless (string= python-shell-readline-completer-delims ""))
+       (python-tests--completion-module)
+       (python-tests--completion-parameters)
+       (python-tests--completion-extra-context)))))

 \f
 ;;; PDB Track integration
--
2.39.2

bug#68559: [PATCH] Improve Python shell completion

[Liu Hui]

[-- Attachment #1: Type: text/plain, Size: 949 bytes --]

On Mon, Feb 5, 2024 at 11:03 PM Liu Hui <liuhui1610@gmail.com> wrote:

> > Thank you for the detailed explanation and the workaround.  I
> > confirmed that the problem is solved by the above workaround.  Just to
> > confirm, are you of the opinion that this workaround should not be the
> > default?
>
> I'm not sure if we should add more Python code in the form of strings
> to python.el, which increases maintenance burden IMO. Maybe they could
> be distributed separately at ELPA/Git forges.
>
> Actually, I'm considering simplifying this patch to mainly fix the bug
> that python shell completion doesn't respect the delimiter of readline
> completer. The new patch has been attached. It should support
> the completer defined in the PYTHONSTARTUP file, e.g., jedi or a
> custom completer like the above one. WDYT?

I have updated documentation about readline completer and fixed an
error in commit log in the attach patch.

[-- Attachment #2: 0001-Respect-the-delimiter-of-completer-in-Python-shell-c.patch --]
[-- Type: text/x-patch, Size: 21255 bytes --]

From 529e5f6180197130ec39dbddd39b2aa2b6deff67 Mon Sep 17 00:00:00 2001
From: Liu Hui <liuhui1610@gmail.com>
Date: Thu, 18 Jan 2024 12:00:00 +0800
Subject: [PATCH] Respect the delimiter of completer in Python shell completion

* lisp/progmodes/python.el: (python-shell-completion-setup-code): Fix
the completion code of IPython.  Change the return value to JSON string
and ...
(python-shell-completion-get-completions): ... simplify parsing.
(inferior-python-mode): Update docstring.
(python-shell-readline-completer-delims): New variable indicating the
word delimiters of readline completer.
(python-shell-completion-native-setup): Set the completer delimiter.
(python-shell-completion-native-get-completions): Convert output string
to completions properly.
(python-shell--get-multiline-input)
(python-shell--extra-completion-context)
(python-shell-completion-extra-context): New functions.
(python-shell-completion-at-point): Send text beginning from the line
start if the completion backend does not need word splitting.  Remove
the detection of import statement because it is not needed anymore.
Create proper completion table based on completions returned from
different backends.

* test/lisp/progmodes/python-tests.el (python-tests--completion-module)
(python-tests--completion-parameters)
(python-tests--completion-extra-context): New helper functions.
(python-shell-completion-at-point-jedi-completer)
(python-shell-completion-at-point-ipython): New tests. (bug#68559)
---
 lisp/progmodes/python.el            | 218 ++++++++++++++++++++++------
 test/lisp/progmodes/python-tests.el |  92 ++++++++++++
 2 files changed, 263 insertions(+), 47 deletions(-)

diff --git a/lisp/progmodes/python.el b/lisp/progmodes/python.el
index 9d840efb9da..d88da4e2700 100644
--- a/lisp/progmodes/python.el
+++ b/lisp/progmodes/python.el
@@ -128,9 +128,9 @@ ;;; Commentary:
 ;; receiving escape sequences (with some limitations, i.e. completion
 ;; in blocks does not work).  The code executed for the "fallback"
 ;; completion can be found in `python-shell-completion-setup-code' and
-;; `python-shell-completion-string-code' variables.  Their default
-;; values enable completion for both CPython and IPython, and probably
-;; any readline based shell (it's known to work with PyPy).  If your
+;; `python-shell-completion-get-completions'.  Their default values
+;; enable completion for both CPython and IPython, and probably any
+;; readline based shell (it's known to work with PyPy).  If your
 ;; Python installation lacks readline (like CPython for Windows),
 ;; installing pyreadline (URL `https://ipython.org/pyreadline.html')
 ;; should suffice.  To troubleshoot why you are not getting any
@@ -141,6 +141,12 @@ ;;; Commentary:
 ;; If you see an error, then you need to either install pyreadline or
 ;; setup custom code that avoids that dependency.
 
+;; By default, the "native" completion uses the built-in rlcompleter.
+;; To use other readline completer (e.g. Jedi) or a custom one, you just
+;; need to set it in the PYTHONSTARTUP file.  You can set an
+;; Emacs-specific completer by testing the environment variable
+;; INSIDE_EMACS.
+
 ;; Shell virtualenv support: The shell also contains support for
 ;; virtualenvs and other special environment modifications thanks to
 ;; `python-shell-process-environment' and `python-shell-exec-path'.
@@ -3604,7 +3610,6 @@ (define-derived-mode inferior-python-mode comint-mode "Inferior Python"
 `python-shell-prompt-block-regexp',
 `python-shell-font-lock-enable',
 `python-shell-completion-setup-code',
-`python-shell-completion-string-code',
 `python-eldoc-setup-code',
 `python-ffap-setup-code' can
 customize this mode for different Python interpreters.
@@ -4244,8 +4249,9 @@ (defcustom python-shell-completion-setup-code
     completions = []
     completer = None
 
+    import json
     try:
-        import readline
+        import readline, re
 
         try:
             import __builtin__
@@ -4256,16 +4262,29 @@ (defcustom python-shell-completion-setup-code
 
         is_ipython = ('__IPYTHON__' in builtins or
                       '__IPYTHON__active' in builtins)
-        splits = text.split()
-        is_module = splits and splits[0] in ('from', 'import')
-
-        if is_ipython and is_module:
-            from IPython.core.completerlib import module_completion
-            completions = module_completion(text.strip())
-        elif is_ipython and '__IP' in builtins:
-            completions = __IP.complete(text)
-        elif is_ipython and 'get_ipython' in builtins:
-            completions = get_ipython().Completer.all_completions(text)
+
+        if is_ipython and 'get_ipython' in builtins:
+            def filter_c(prefix, c):
+                if re.match('_+(i?[0-9]+)?$', c):
+                    return False
+                elif c[0] == '%' and not re.match('[%a-zA-Z]+$', prefix):
+                    return False
+                return True
+
+            import IPython
+            try:
+                if IPython.version_info[0] >= 6:
+                    from IPython.core.completer import provisionalcompleter
+                    with provisionalcompleter():
+                        completions = [
+                            [c.text, c.start, c.end, c.type or '?', c.signature or '']
+                             for c in get_ipython().Completer.completions(text, len(text))
+                             if filter_c(text, c.text)]
+                else:
+                    part, matches = get_ipython().Completer.complete(line_buffer=text)
+                    completions = [text + m[len(part):] for m in matches if filter_c(text, m)]
+            except:
+                pass
         else:
             # Try to reuse current completer.
             completer = readline.get_completer()
@@ -4288,7 +4307,7 @@ (defcustom python-shell-completion-setup-code
     finally:
         if getattr(completer, 'PYTHON_EL_WRAPPED', False):
             completer.print_mode = True
-    return completions"
+    return json.dumps(completions)"
   "Code used to setup completion in inferior Python processes."
   :type 'string)
 
@@ -4329,6 +4348,10 @@ (defcustom python-shell-completion-native-try-output-timeout 1.0
   :version "25.1"
   :type 'float)
 
+(defvar python-shell-readline-completer-delims nil
+  "Word delimiters used by the readline completer.
+It is automatically set by Python shell.")
+
 (defvar python-shell-completion-native-redirect-buffer
   " *Python completions redirect*"
   "Buffer to be used to redirect output of readline commands.")
@@ -4467,6 +4490,10 @@ (defun python-shell-completion-native-setup ()
 __PYTHON_EL_native_completion_setup()" process)))
     (when (string-match-p "python\\.el: native completion setup loaded"
                           output)
+      (setq-local python-shell-readline-completer-delims
+                  (string-trim-right
+                   (python-shell-send-string-no-output
+                    "import readline; print(readline.get_completer_delims())")))
       (python-shell-completion-native-try))))
 
 (defun python-shell-completion-native-turn-off (&optional msg)
@@ -4534,6 +4561,8 @@ (defun python-shell-completion-native-get-completions (process input)
     (let* ((original-filter-fn (process-filter process))
            (redirect-buffer (get-buffer-create
                              python-shell-completion-native-redirect-buffer))
+           (sep (if (string= python-shell-readline-completer-delims "")
+                    "[\n\r]+" "[ \f\t\n\r\v()]+"))
            (trigger "\t")
            (new-input (concat input trigger))
            (input-length
@@ -4576,28 +4605,80 @@ (defun python-shell-completion-native-get-completions (process input)
                      process python-shell-completion-native-output-timeout
                      comint-redirect-finished-regexp)
                 (re-search-backward "0__dummy_completion__" nil t)
-                (cl-remove-duplicates
-                 (split-string
-                  (buffer-substring-no-properties
-                   (line-beginning-position) (point-min))
-                  "[ \f\t\n\r\v()]+" t)
-                 :test #'string=))))
+                (let ((str (buffer-substring-no-properties
+                            (line-beginning-position) (point-min))))
+                  ;; The readline completer is allowed to return a list
+                  ;; of (text start end type signature) as a JSON
+                  ;; string.  See the return value for IPython in
+                  ;; `python-shell-completion-setup-code'.
+                  (if (string= "[" (substring str 0 1))
+                      (condition-case nil
+                          (python--parse-json-array str)
+                        (t (cl-remove-duplicates (split-string str sep t)
+                                                 :test #'string=)))
+                    (cl-remove-duplicates (split-string str sep t)
+                                          :test #'string=))))))
         (set-process-filter process original-filter-fn)))))
 
 (defun python-shell-completion-get-completions (process input)
   "Get completions of INPUT using PROCESS."
   (with-current-buffer (process-buffer process)
-    (let ((completions
-           (python-util-strip-string
-            (python-shell-send-string-no-output
-             (format
-              "%s\nprint(';'.join(__PYTHON_EL_get_completions(%s)))"
+    (python--parse-json-array
+     (python-shell-send-string-no-output
+      (format "%s\nprint(__PYTHON_EL_get_completions(%s))"
               python-shell-completion-setup-code
               (python-shell--encode-string input))
-             process))))
-      (when (> (length completions) 2)
-        (split-string completions
-                      "^'\\|^\"\\|;\\|'$\\|\"$" t)))))
+      process))))
+
+(defun python-shell--get-multiline-input ()
+  "Return lines at a multi-line input in Python shell."
+  (save-excursion
+    (let ((p (point)) lines)
+      (when (progn
+              (beginning-of-line)
+              (looking-back python-shell-prompt-block-regexp (pos-bol)))
+        (push (buffer-substring-no-properties (point) p) lines)
+        (while (progn (comint-previous-prompt 1)
+                      (looking-back python-shell-prompt-block-regexp (pos-bol)))
+          (push (buffer-substring-no-properties (point) (pos-eol)) lines))
+        (push (buffer-substring-no-properties (point) (pos-eol)) lines))
+      lines)))
+
+(defun python-shell--extra-completion-context ()
+  "Get extra completion context of current input in Python shell."
+  (let ((lines (python-shell--get-multiline-input))
+        (python-indent-guess-indent-offset nil))
+    (when (not (zerop (length lines)))
+      (with-temp-buffer
+        (delay-mode-hooks
+          (insert (string-join lines "\n"))
+          (python-mode)
+          (python-shell-completion-extra-context))))))
+
+(defun python-shell-completion-extra-context (&optional pos)
+  "Get extra completion context at position POS in Python buffer.
+If optional argument POS is nil, use current position.
+
+Readline completers could use current line as the completion
+context, which may be insufficient.  In this function, extra
+context (e.g. multi-line function call) is found and reformatted
+as one line, which is required by native completion."
+  (let (bound p)
+    (save-excursion
+      (and pos (goto-char pos))
+      (setq bound (pos-bol))
+      (python-nav-up-list -1)
+      (when (and (< (point) bound)
+                 (or
+                  (looking-back
+                   (python-rx (group (+ (or "." symbol-name)))) (pos-bol) t)
+                  (progn
+                    (forward-line 0)
+                    (looking-at "^[ \t]*\\(from \\)"))))
+        (setq p (match-beginning 1))))
+    (when p
+      (replace-regexp-in-string
+       "\n[ \t]*" "" (buffer-substring-no-properties p (1- bound))))))
 
 (defvar-local python-shell--capf-cache nil
   "Variable to store cached completions and invalidation keys.")
@@ -4612,21 +4693,26 @@ (defun python-shell-completion-at-point (&optional process)
                          ;; Working on a shell buffer: use prompt end.
                          (cdr (python-util-comint-last-prompt))
                        (line-beginning-position)))
-         (import-statement
-          (when (string-match-p
-                 (rx (* space) word-start (or "from" "import") word-end space)
-                 (buffer-substring-no-properties line-start (point)))
-            (buffer-substring-no-properties line-start (point))))
+         (no-delims
+          (and (not (if is-shell-buffer
+                        (eq 'font-lock-comment-face
+                            (get-text-property (1- (point)) 'face))
+                      (python-syntax-context 'comment)))
+               (with-current-buffer (process-buffer process)
+                 (if python-shell-completion-native-enable
+                     (string= python-shell-readline-completer-delims "")
+                   (string-match-p "ipython[23]?\\'" python-shell-interpreter)))))
          (start
           (if (< (point) line-start)
               (point)
             (save-excursion
-              (if (not (re-search-backward
-                        (python-rx
-                         (or whitespace open-paren close-paren
-                             string-delimiter simple-operator))
-                        line-start
-                        t 1))
+              (if (or no-delims
+                      (not (re-search-backward
+                            (python-rx
+                             (or whitespace open-paren close-paren
+                                 string-delimiter simple-operator))
+                            line-start
+                            t 1)))
                   line-start
                 (forward-char (length (match-string-no-properties 0)))
                 (point)))))
@@ -4666,18 +4752,56 @@ (defun python-shell-completion-at-point (&optional process)
                   (t #'python-shell-completion-native-get-completions))))
          (prev-prompt (car python-shell--capf-cache))
          (re (or (cadr python-shell--capf-cache) regexp-unmatchable))
-         (prefix (buffer-substring-no-properties start end)))
+         (prefix (buffer-substring-no-properties start end))
+         (prefix-offset 0)
+         (extra-context (when no-delims
+                          (if is-shell-buffer
+                              (python-shell--extra-completion-context)
+                            (python-shell-completion-extra-context))))
+         (extra-offset (length extra-context)))
+    (unless (zerop extra-offset)
+      (setq prefix (concat extra-context prefix)))
     ;; To invalidate the cache, we check if the prompt position or the
     ;; completion prefix changed.
     (unless (and (equal prev-prompt (car prompt-boundaries))
-                 (string-match re prefix))
+                 (string-match re prefix)
+                 (setq prefix-offset (- (length prefix) (match-end 1))))
       (setq python-shell--capf-cache
             `(,(car prompt-boundaries)
               ,(if (string-empty-p prefix)
                    regexp-unmatchable
-                 (concat "\\`" (regexp-quote prefix) "\\(?:\\sw\\|\\s_\\)*\\'"))
-              ,@(funcall completion-fn process (or import-statement prefix)))))
-    (list start end (cddr python-shell--capf-cache))))
+                 (concat "\\`\\(" (regexp-quote prefix) "\\)\\(?:\\sw\\|\\s_\\)*\\'"))
+              ,@(funcall completion-fn process prefix))))
+    (let ((cands (cddr python-shell--capf-cache)))
+      (cond
+       ((stringp (car cands))
+        (if no-delims
+            ;; Reduce completion candidates due to long prefix.
+            (if-let ((Lp (length prefix))
+                     ((string-match "\\(\\sw\\|\\s_\\)+\\'" prefix))
+                     (L (match-beginning 0)))
+                ;; If extra-offset is not zero:
+                ;;                  start              end
+                ;; o------------------o---------o-------o
+                ;; |<- extra-offset ->|
+                ;; |<----------- L ------------>|
+                ;;                          new-start
+                (list (+ start L (- extra-offset)) end
+                      (mapcar (lambda (s) (substring s L)) cands))
+              (list end end (mapcar (lambda (s) (substring s Lp)) cands)))
+          (list start end cands)))
+       ;; python-shell-completion(-native)-get-completions may produce a
+       ;; list of (text start end type signature) for completion.
+       ((consp (car cands))
+        (list (+ start (nth 1 (car cands)) (- extra-offset))
+              ;; Candidates may be cached, so the end position should
+              ;; be adjusted according to current completion prefix.
+              (+ start (nth 2 (car cands)) (- extra-offset) prefix-offset)
+              cands
+              :annotation-function
+              (lambda (c) (concat " " (nth 3 (assoc c cands))))
+              :company-docsig
+              (lambda (c) (nth 4 (assoc c cands)))))))))
 
 (define-obsolete-function-alias
   'python-shell-completion-complete-at-point
diff --git a/test/lisp/progmodes/python-tests.el b/test/lisp/progmodes/python-tests.el
index 59957ff0712..af6c199b5bd 100644
--- a/test/lisp/progmodes/python-tests.el
+++ b/test/lisp/progmodes/python-tests.el
@@ -4799,6 +4799,98 @@ (ert-deftest python-shell-completion-at-point-native-1 ()
      (end-of-line 0)
      (should-not (nth 2 (python-shell-completion-at-point))))))
 
+(defun python-tests--completion-module ()
+  "Check if modules can be completed in Python shell."
+  (insert "import datet")
+  (completion-at-point)
+  (beginning-of-line)
+  (should (looking-at-p "import datetime"))
+  (kill-line)
+  (insert "from datet")
+  (completion-at-point)
+  (beginning-of-line)
+  (should (looking-at-p "from datetime"))
+  (end-of-line)
+  (insert " import timed")
+  (completion-at-point)
+  (beginning-of-line)
+  (should (looking-at-p "from datetime import timedelta"))
+  (kill-line))
+
+(defun python-tests--completion-parameters ()
+  "Check if parameters can be completed in Python shell."
+  (insert "import re")
+  (comint-send-input)
+  (python-tests-shell-wait-for-prompt)
+  (insert "re.split('b', 'abc', maxs")
+  (completion-at-point)
+  (should (string= "re.split('b', 'abc', maxsplit="
+                   (buffer-substring (line-beginning-position) (point))))
+  (insert "0, ")
+  (should (python-shell-completion-at-point))
+  ;; Test if cache is used.
+  (cl-letf (((symbol-function 'python-shell-completion-get-completions)
+             'ignore)
+            ((symbol-function 'python-shell-completion-native-get-completions)
+             'ignore))
+    (insert "fla")
+    (completion-at-point)
+    (should (string= "re.split('b', 'abc', maxsplit=0, flags="
+                     (buffer-substring (line-beginning-position) (point)))))
+  (beginning-of-line)
+  (kill-line))
+
+(defun python-tests--completion-extra-context ()
+  "Check if extra context is used for completion."
+  (insert "re.split('b', 'abc',")
+  (comint-send-input)
+  (python-tests-shell-wait-for-prompt)
+  (insert "maxs")
+  (completion-at-point)
+  (should (string= "maxsplit="
+                   (buffer-substring (line-beginning-position) (point))))
+  (insert "0)")
+  (comint-send-input)
+  (python-tests-shell-wait-for-prompt)
+  (insert "from re import (")
+  (comint-send-input)
+  (python-tests-shell-wait-for-prompt)
+  (insert "IGN")
+  (completion-at-point)
+  (should (string= "IGNORECASE"
+                   (buffer-substring (line-beginning-position) (point)))))
+
+(ert-deftest python-shell-completion-at-point-jedi-completer ()
+  "Check if Python shell completion works when Jedi completer is used."
+  (skip-unless (executable-find python-tests-shell-interpreter))
+  (python-tests-with-temp-buffer-with-shell
+   ""
+   (python-shell-with-shell-buffer
+     (python-shell-completion-native-turn-on)
+     (skip-unless (string= python-shell-readline-completer-delims ""))
+     (python-tests--completion-module)
+     (python-tests--completion-parameters)
+     (python-tests--completion-extra-context))))
+
+(ert-deftest python-shell-completion-at-point-ipython ()
+  "Check if Python shell completion works for IPython."
+  (let ((python-shell-interpreter "ipython")
+        (python-shell-interpreter-args "-i --simple-prompt"))
+    (skip-unless
+     (and
+      (executable-find python-shell-interpreter)
+      (eql (call-process python-shell-interpreter nil nil nil "--version") 0)))
+    (python-tests-with-temp-buffer-with-shell
+     ""
+     (python-shell-with-shell-buffer
+       (python-shell-completion-native-turn-off)
+       (python-tests--completion-module)
+       (python-tests--completion-parameters)
+       (python-shell-completion-native-turn-on)
+       (skip-unless (string= python-shell-readline-completer-delims ""))
+       (python-tests--completion-module)
+       (python-tests--completion-parameters)
+       (python-tests--completion-extra-context)))))
 
 \f
 ;;; PDB Track integration
-- 
2.39.2

bug#68559: [PATCH] Improve Python shell completion

[kobarity]

Liu Hui wrote:
> On Mon, Feb 5, 2024 at 11:03 PM Liu Hui <liuhui1610@gmail.com> wrote:
> 
> > > Thank you for the detailed explanation and the workaround.  I
> > > confirmed that the problem is solved by the above workaround.  Just to
> > > confirm, are you of the opinion that this workaround should not be the
> > > default?
> >
> > I'm not sure if we should add more Python code in the form of strings
> > to python.el, which increases maintenance burden IMO. Maybe they could
> > be distributed separately at ELPA/Git forges.
> >
> > Actually, I'm considering simplifying this patch to mainly fix the bug
> > that python shell completion doesn't respect the delimiter of readline
> > completer. The new patch has been attached. It should support
> > the completer defined in the PYTHONSTARTUP file, e.g., jedi or a
> > custom completer like the above one. WDYT?
> 
> I have updated documentation about readline completer and fixed an
> error in commit log in the attach patch.

Thanks, I understood your concern.  I tried the new patch and
confirmed that your MyJediRL can be used as a custom completer by
storing it in a file specified in the PYTHONSTARTUP environment
variable.  This may be the better way to go.

Sorry, I noticed one more thing.  As written in the comments below, it
would be better to use `line-beginning-position' and
`line-end-position' instead of `pos-bol' and `pos-eol'.

;; This is a GNU ELPA :core package.  Avoid functionality that is not
;; compatible with the version of Emacs recorded above.

bug#68559: [PATCH] Improve Python shell completion

[Liu Hui]

[-- Attachment #1: Type: text/plain, Size: 730 bytes --]

On Tue, Feb 6, 2024 at 11:12 PM kobarity <kobarity@gmail.com> wrote:

> Sorry, I noticed one more thing.  As written in the comments below, it
> would be better to use `line-beginning-position' and
> `line-end-position' instead of `pos-bol' and `pos-eol'.
>
> ;; This is a GNU ELPA :core package.  Avoid functionality that is not
> ;; compatible with the version of Emacs recorded above.

Thanks for pointing it out! `pos-bol' was used because it can ignore
the prompt in `python-shell--get-multiline-input'. Fortunately, I find
`pos-bol' and `pos-eol' are available in the compat package since
29.1.1.0, so it should be enough to upgrade the compat version in
Package-Requires. I have updated the attached patch.

[-- Attachment #2: 0001-Respect-the-delimiter-of-completer-in-Python-shell-c.patch --]
[-- Type: text/x-patch, Size: 21693 bytes --]

From a6da4c0b88326859de5847aaa97816f89fc9782d Mon Sep 17 00:00:00 2001
From: Liu Hui <liuhui1610@gmail.com>
Date: Thu, 18 Jan 2024 12:00:00 +0800
Subject: [PATCH] Respect the delimiter of completer in Python shell completion

* lisp/progmodes/python.el: (python-shell-completion-setup-code): Fix
the completion code of IPython.  Change the return value to JSON string
and ...
(python-shell-completion-get-completions): ... simplify parsing.
(inferior-python-mode): Update docstring.
(python-shell-readline-completer-delims): New variable indicating the
word delimiters of readline completer.
(python-shell-completion-native-setup): Set the completer delimiter.
(python-shell-completion-native-get-completions): Convert output string
to completions properly.
(python-shell--get-multiline-input)
(python-shell--extra-completion-context)
(python-shell-completion-extra-context): New functions.
(python-shell-completion-at-point): Send text beginning from the line
start if the completion backend does not need word splitting.  Remove
the detection of import statement because it is not needed anymore.
Create proper completion table based on completions returned from
different backends.

* test/lisp/progmodes/python-tests.el (python-tests--completion-module)
(python-tests--completion-parameters)
(python-tests--completion-extra-context): New helper functions.
(python-shell-completion-at-point-jedi-completer)
(python-shell-completion-at-point-ipython): New tests.  (bug#68559)
---
 lisp/progmodes/python.el            | 220 ++++++++++++++++++++++------
 test/lisp/progmodes/python-tests.el |  92 ++++++++++++
 2 files changed, 264 insertions(+), 48 deletions(-)

diff --git a/lisp/progmodes/python.el b/lisp/progmodes/python.el
index 9d840efb9da..b1654b6a5aa 100644
--- a/lisp/progmodes/python.el
+++ b/lisp/progmodes/python.el
@@ -5,7 +5,7 @@ ;;; python.el --- Python's flying circus support for Emacs -*- lexical-binding:
 ;; Author: Fabián E. Gallina <fgallina@gnu.org>
 ;; URL: https://github.com/fgallina/python.el
 ;; Version: 0.28
-;; Package-Requires: ((emacs "24.4") (compat "28.1.2.1") (seq "2.23"))
+;; Package-Requires: ((emacs "24.4") (compat "29.1.1.0") (seq "2.23"))
 ;; Maintainer: emacs-devel@gnu.org
 ;; Created: Jul 2010
 ;; Keywords: languages
@@ -128,9 +128,9 @@ ;;; Commentary:
 ;; receiving escape sequences (with some limitations, i.e. completion
 ;; in blocks does not work).  The code executed for the "fallback"
 ;; completion can be found in `python-shell-completion-setup-code' and
-;; `python-shell-completion-string-code' variables.  Their default
-;; values enable completion for both CPython and IPython, and probably
-;; any readline based shell (it's known to work with PyPy).  If your
+;; `python-shell-completion-get-completions'.  Their default values
+;; enable completion for both CPython and IPython, and probably any
+;; readline based shell (it's known to work with PyPy).  If your
 ;; Python installation lacks readline (like CPython for Windows),
 ;; installing pyreadline (URL `https://ipython.org/pyreadline.html')
 ;; should suffice.  To troubleshoot why you are not getting any
@@ -141,6 +141,12 @@ ;;; Commentary:
 ;; If you see an error, then you need to either install pyreadline or
 ;; setup custom code that avoids that dependency.
 
+;; By default, the "native" completion uses the built-in rlcompleter.
+;; To use other readline completer (e.g. Jedi) or a custom one, you just
+;; need to set it in the PYTHONSTARTUP file.  You can set an
+;; Emacs-specific completer by testing the environment variable
+;; INSIDE_EMACS.
+
 ;; Shell virtualenv support: The shell also contains support for
 ;; virtualenvs and other special environment modifications thanks to
 ;; `python-shell-process-environment' and `python-shell-exec-path'.
@@ -3604,7 +3610,6 @@ (define-derived-mode inferior-python-mode comint-mode "Inferior Python"
 `python-shell-prompt-block-regexp',
 `python-shell-font-lock-enable',
 `python-shell-completion-setup-code',
-`python-shell-completion-string-code',
 `python-eldoc-setup-code',
 `python-ffap-setup-code' can
 customize this mode for different Python interpreters.
@@ -4244,8 +4249,9 @@ (defcustom python-shell-completion-setup-code
     completions = []
     completer = None
 
+    import json
     try:
-        import readline
+        import readline, re
 
         try:
             import __builtin__
@@ -4256,16 +4262,29 @@ (defcustom python-shell-completion-setup-code
 
         is_ipython = ('__IPYTHON__' in builtins or
                       '__IPYTHON__active' in builtins)
-        splits = text.split()
-        is_module = splits and splits[0] in ('from', 'import')
-
-        if is_ipython and is_module:
-            from IPython.core.completerlib import module_completion
-            completions = module_completion(text.strip())
-        elif is_ipython and '__IP' in builtins:
-            completions = __IP.complete(text)
-        elif is_ipython and 'get_ipython' in builtins:
-            completions = get_ipython().Completer.all_completions(text)
+
+        if is_ipython and 'get_ipython' in builtins:
+            def filter_c(prefix, c):
+                if re.match('_+(i?[0-9]+)?$', c):
+                    return False
+                elif c[0] == '%' and not re.match('[%a-zA-Z]+$', prefix):
+                    return False
+                return True
+
+            import IPython
+            try:
+                if IPython.version_info[0] >= 6:
+                    from IPython.core.completer import provisionalcompleter
+                    with provisionalcompleter():
+                        completions = [
+                            [c.text, c.start, c.end, c.type or '?', c.signature or '']
+                             for c in get_ipython().Completer.completions(text, len(text))
+                             if filter_c(text, c.text)]
+                else:
+                    part, matches = get_ipython().Completer.complete(line_buffer=text)
+                    completions = [text + m[len(part):] for m in matches if filter_c(text, m)]
+            except:
+                pass
         else:
             # Try to reuse current completer.
             completer = readline.get_completer()
@@ -4288,7 +4307,7 @@ (defcustom python-shell-completion-setup-code
     finally:
         if getattr(completer, 'PYTHON_EL_WRAPPED', False):
             completer.print_mode = True
-    return completions"
+    return json.dumps(completions)"
   "Code used to setup completion in inferior Python processes."
   :type 'string)
 
@@ -4329,6 +4348,10 @@ (defcustom python-shell-completion-native-try-output-timeout 1.0
   :version "25.1"
   :type 'float)
 
+(defvar python-shell-readline-completer-delims nil
+  "Word delimiters used by the readline completer.
+It is automatically set by Python shell.")
+
 (defvar python-shell-completion-native-redirect-buffer
   " *Python completions redirect*"
   "Buffer to be used to redirect output of readline commands.")
@@ -4467,6 +4490,10 @@ (defun python-shell-completion-native-setup ()
 __PYTHON_EL_native_completion_setup()" process)))
     (when (string-match-p "python\\.el: native completion setup loaded"
                           output)
+      (setq-local python-shell-readline-completer-delims
+                  (string-trim-right
+                   (python-shell-send-string-no-output
+                    "import readline; print(readline.get_completer_delims())")))
       (python-shell-completion-native-try))))
 
 (defun python-shell-completion-native-turn-off (&optional msg)
@@ -4534,6 +4561,8 @@ (defun python-shell-completion-native-get-completions (process input)
     (let* ((original-filter-fn (process-filter process))
            (redirect-buffer (get-buffer-create
                              python-shell-completion-native-redirect-buffer))
+           (sep (if (string= python-shell-readline-completer-delims "")
+                    "[\n\r]+" "[ \f\t\n\r\v()]+"))
            (trigger "\t")
            (new-input (concat input trigger))
            (input-length
@@ -4576,28 +4605,80 @@ (defun python-shell-completion-native-get-completions (process input)
                      process python-shell-completion-native-output-timeout
                      comint-redirect-finished-regexp)
                 (re-search-backward "0__dummy_completion__" nil t)
-                (cl-remove-duplicates
-                 (split-string
-                  (buffer-substring-no-properties
-                   (line-beginning-position) (point-min))
-                  "[ \f\t\n\r\v()]+" t)
-                 :test #'string=))))
+                (let ((str (buffer-substring-no-properties
+                            (line-beginning-position) (point-min))))
+                  ;; The readline completer is allowed to return a list
+                  ;; of (text start end type signature) as a JSON
+                  ;; string.  See the return value for IPython in
+                  ;; `python-shell-completion-setup-code'.
+                  (if (string= "[" (substring str 0 1))
+                      (condition-case nil
+                          (python--parse-json-array str)
+                        (t (cl-remove-duplicates (split-string str sep t)
+                                                 :test #'string=)))
+                    (cl-remove-duplicates (split-string str sep t)
+                                          :test #'string=))))))
         (set-process-filter process original-filter-fn)))))
 
 (defun python-shell-completion-get-completions (process input)
   "Get completions of INPUT using PROCESS."
   (with-current-buffer (process-buffer process)
-    (let ((completions
-           (python-util-strip-string
-            (python-shell-send-string-no-output
-             (format
-              "%s\nprint(';'.join(__PYTHON_EL_get_completions(%s)))"
+    (python--parse-json-array
+     (python-shell-send-string-no-output
+      (format "%s\nprint(__PYTHON_EL_get_completions(%s))"
               python-shell-completion-setup-code
               (python-shell--encode-string input))
-             process))))
-      (when (> (length completions) 2)
-        (split-string completions
-                      "^'\\|^\"\\|;\\|'$\\|\"$" t)))))
+      process))))
+
+(defun python-shell--get-multiline-input ()
+  "Return lines at a multi-line input in Python shell."
+  (save-excursion
+    (let ((p (point)) lines)
+      (when (progn
+              (beginning-of-line)
+              (looking-back python-shell-prompt-block-regexp (pos-bol)))
+        (push (buffer-substring-no-properties (point) p) lines)
+        (while (progn (comint-previous-prompt 1)
+                      (looking-back python-shell-prompt-block-regexp (pos-bol)))
+          (push (buffer-substring-no-properties (point) (pos-eol)) lines))
+        (push (buffer-substring-no-properties (point) (pos-eol)) lines))
+      lines)))
+
+(defun python-shell--extra-completion-context ()
+  "Get extra completion context of current input in Python shell."
+  (let ((lines (python-shell--get-multiline-input))
+        (python-indent-guess-indent-offset nil))
+    (when (not (zerop (length lines)))
+      (with-temp-buffer
+        (delay-mode-hooks
+          (insert (string-join lines "\n"))
+          (python-mode)
+          (python-shell-completion-extra-context))))))
+
+(defun python-shell-completion-extra-context (&optional pos)
+  "Get extra completion context at position POS in Python buffer.
+If optional argument POS is nil, use current position.
+
+Readline completers could use current line as the completion
+context, which may be insufficient.  In this function, extra
+context (e.g. multi-line function call) is found and reformatted
+as one line, which is required by native completion."
+  (let (bound p)
+    (save-excursion
+      (and pos (goto-char pos))
+      (setq bound (pos-bol))
+      (python-nav-up-list -1)
+      (when (and (< (point) bound)
+                 (or
+                  (looking-back
+                   (python-rx (group (+ (or "." symbol-name)))) (pos-bol) t)
+                  (progn
+                    (forward-line 0)
+                    (looking-at "^[ \t]*\\(from \\)"))))
+        (setq p (match-beginning 1))))
+    (when p
+      (replace-regexp-in-string
+       "\n[ \t]*" "" (buffer-substring-no-properties p (1- bound))))))
 
 (defvar-local python-shell--capf-cache nil
   "Variable to store cached completions and invalidation keys.")
@@ -4612,21 +4693,26 @@ (defun python-shell-completion-at-point (&optional process)
                          ;; Working on a shell buffer: use prompt end.
                          (cdr (python-util-comint-last-prompt))
                        (line-beginning-position)))
-         (import-statement
-          (when (string-match-p
-                 (rx (* space) word-start (or "from" "import") word-end space)
-                 (buffer-substring-no-properties line-start (point)))
-            (buffer-substring-no-properties line-start (point))))
+         (no-delims
+          (and (not (if is-shell-buffer
+                        (eq 'font-lock-comment-face
+                            (get-text-property (1- (point)) 'face))
+                      (python-syntax-context 'comment)))
+               (with-current-buffer (process-buffer process)
+                 (if python-shell-completion-native-enable
+                     (string= python-shell-readline-completer-delims "")
+                   (string-match-p "ipython[23]?\\'" python-shell-interpreter)))))
          (start
           (if (< (point) line-start)
               (point)
             (save-excursion
-              (if (not (re-search-backward
-                        (python-rx
-                         (or whitespace open-paren close-paren
-                             string-delimiter simple-operator))
-                        line-start
-                        t 1))
+              (if (or no-delims
+                      (not (re-search-backward
+                            (python-rx
+                             (or whitespace open-paren close-paren
+                                 string-delimiter simple-operator))
+                            line-start
+                            t 1)))
                   line-start
                 (forward-char (length (match-string-no-properties 0)))
                 (point)))))
@@ -4666,18 +4752,56 @@ (defun python-shell-completion-at-point (&optional process)
                   (t #'python-shell-completion-native-get-completions))))
          (prev-prompt (car python-shell--capf-cache))
          (re (or (cadr python-shell--capf-cache) regexp-unmatchable))
-         (prefix (buffer-substring-no-properties start end)))
+         (prefix (buffer-substring-no-properties start end))
+         (prefix-offset 0)
+         (extra-context (when no-delims
+                          (if is-shell-buffer
+                              (python-shell--extra-completion-context)
+                            (python-shell-completion-extra-context))))
+         (extra-offset (length extra-context)))
+    (unless (zerop extra-offset)
+      (setq prefix (concat extra-context prefix)))
     ;; To invalidate the cache, we check if the prompt position or the
     ;; completion prefix changed.
     (unless (and (equal prev-prompt (car prompt-boundaries))
-                 (string-match re prefix))
+                 (string-match re prefix)
+                 (setq prefix-offset (- (length prefix) (match-end 1))))
       (setq python-shell--capf-cache
             `(,(car prompt-boundaries)
               ,(if (string-empty-p prefix)
                    regexp-unmatchable
-                 (concat "\\`" (regexp-quote prefix) "\\(?:\\sw\\|\\s_\\)*\\'"))
-              ,@(funcall completion-fn process (or import-statement prefix)))))
-    (list start end (cddr python-shell--capf-cache))))
+                 (concat "\\`\\(" (regexp-quote prefix) "\\)\\(?:\\sw\\|\\s_\\)*\\'"))
+              ,@(funcall completion-fn process prefix))))
+    (let ((cands (cddr python-shell--capf-cache)))
+      (cond
+       ((stringp (car cands))
+        (if no-delims
+            ;; Reduce completion candidates due to long prefix.
+            (if-let ((Lp (length prefix))
+                     ((string-match "\\(\\sw\\|\\s_\\)+\\'" prefix))
+                     (L (match-beginning 0)))
+                ;; If extra-offset is not zero:
+                ;;                  start              end
+                ;; o------------------o---------o-------o
+                ;; |<- extra-offset ->|
+                ;; |<----------- L ------------>|
+                ;;                          new-start
+                (list (+ start L (- extra-offset)) end
+                      (mapcar (lambda (s) (substring s L)) cands))
+              (list end end (mapcar (lambda (s) (substring s Lp)) cands)))
+          (list start end cands)))
+       ;; python-shell-completion(-native)-get-completions may produce a
+       ;; list of (text start end type signature) for completion.
+       ((consp (car cands))
+        (list (+ start (nth 1 (car cands)) (- extra-offset))
+              ;; Candidates may be cached, so the end position should
+              ;; be adjusted according to current completion prefix.
+              (+ start (nth 2 (car cands)) (- extra-offset) prefix-offset)
+              cands
+              :annotation-function
+              (lambda (c) (concat " " (nth 3 (assoc c cands))))
+              :company-docsig
+              (lambda (c) (nth 4 (assoc c cands)))))))))
 
 (define-obsolete-function-alias
   'python-shell-completion-complete-at-point
diff --git a/test/lisp/progmodes/python-tests.el b/test/lisp/progmodes/python-tests.el
index 59957ff0712..af6c199b5bd 100644
--- a/test/lisp/progmodes/python-tests.el
+++ b/test/lisp/progmodes/python-tests.el
@@ -4799,6 +4799,98 @@ (ert-deftest python-shell-completion-at-point-native-1 ()
      (end-of-line 0)
      (should-not (nth 2 (python-shell-completion-at-point))))))
 
+(defun python-tests--completion-module ()
+  "Check if modules can be completed in Python shell."
+  (insert "import datet")
+  (completion-at-point)
+  (beginning-of-line)
+  (should (looking-at-p "import datetime"))
+  (kill-line)
+  (insert "from datet")
+  (completion-at-point)
+  (beginning-of-line)
+  (should (looking-at-p "from datetime"))
+  (end-of-line)
+  (insert " import timed")
+  (completion-at-point)
+  (beginning-of-line)
+  (should (looking-at-p "from datetime import timedelta"))
+  (kill-line))
+
+(defun python-tests--completion-parameters ()
+  "Check if parameters can be completed in Python shell."
+  (insert "import re")
+  (comint-send-input)
+  (python-tests-shell-wait-for-prompt)
+  (insert "re.split('b', 'abc', maxs")
+  (completion-at-point)
+  (should (string= "re.split('b', 'abc', maxsplit="
+                   (buffer-substring (line-beginning-position) (point))))
+  (insert "0, ")
+  (should (python-shell-completion-at-point))
+  ;; Test if cache is used.
+  (cl-letf (((symbol-function 'python-shell-completion-get-completions)
+             'ignore)
+            ((symbol-function 'python-shell-completion-native-get-completions)
+             'ignore))
+    (insert "fla")
+    (completion-at-point)
+    (should (string= "re.split('b', 'abc', maxsplit=0, flags="
+                     (buffer-substring (line-beginning-position) (point)))))
+  (beginning-of-line)
+  (kill-line))
+
+(defun python-tests--completion-extra-context ()
+  "Check if extra context is used for completion."
+  (insert "re.split('b', 'abc',")
+  (comint-send-input)
+  (python-tests-shell-wait-for-prompt)
+  (insert "maxs")
+  (completion-at-point)
+  (should (string= "maxsplit="
+                   (buffer-substring (line-beginning-position) (point))))
+  (insert "0)")
+  (comint-send-input)
+  (python-tests-shell-wait-for-prompt)
+  (insert "from re import (")
+  (comint-send-input)
+  (python-tests-shell-wait-for-prompt)
+  (insert "IGN")
+  (completion-at-point)
+  (should (string= "IGNORECASE"
+                   (buffer-substring (line-beginning-position) (point)))))
+
+(ert-deftest python-shell-completion-at-point-jedi-completer ()
+  "Check if Python shell completion works when Jedi completer is used."
+  (skip-unless (executable-find python-tests-shell-interpreter))
+  (python-tests-with-temp-buffer-with-shell
+   ""
+   (python-shell-with-shell-buffer
+     (python-shell-completion-native-turn-on)
+     (skip-unless (string= python-shell-readline-completer-delims ""))
+     (python-tests--completion-module)
+     (python-tests--completion-parameters)
+     (python-tests--completion-extra-context))))
+
+(ert-deftest python-shell-completion-at-point-ipython ()
+  "Check if Python shell completion works for IPython."
+  (let ((python-shell-interpreter "ipython")
+        (python-shell-interpreter-args "-i --simple-prompt"))
+    (skip-unless
+     (and
+      (executable-find python-shell-interpreter)
+      (eql (call-process python-shell-interpreter nil nil nil "--version") 0)))
+    (python-tests-with-temp-buffer-with-shell
+     ""
+     (python-shell-with-shell-buffer
+       (python-shell-completion-native-turn-off)
+       (python-tests--completion-module)
+       (python-tests--completion-parameters)
+       (python-shell-completion-native-turn-on)
+       (skip-unless (string= python-shell-readline-completer-delims ""))
+       (python-tests--completion-module)
+       (python-tests--completion-parameters)
+       (python-tests--completion-extra-context)))))
 
 \f
 ;;; PDB Track integration
-- 
2.39.2

bug#68559: [PATCH] Improve Python shell completion

[kobarity]

Liu Hui wrote:
> On Tue, Feb 6, 2024 at 11:12 PM kobarity <kobarity@gmail.com> wrote:
> 
> > Sorry, I noticed one more thing.  As written in the comments below, it
> > would be better to use `line-beginning-position' and
> > `line-end-position' instead of `pos-bol' and `pos-eol'.
> >
> > ;; This is a GNU ELPA :core package.  Avoid functionality that is not
> > ;; compatible with the version of Emacs recorded above.
> 
> Thanks for pointing it out! `pos-bol' was used because it can ignore
> the prompt in `python-shell--get-multiline-input'. Fortunately, I find
> `pos-bol' and `pos-eol' are available in the compat package since
> 29.1.1.0, so it should be enough to upgrade the compat version in
> Package-Requires. I have updated the attached patch.

I see.  The patch looks good to me.  I appreciate your efforts!

bug#68559: [PATCH] Improve Python shell completion

[Eli Zaretskii]

> Date: Thu, 08 Feb 2024 00:19:55 +0900
> From: kobarity <kobarity@gmail.com>
> Cc: Eli Zaretskii <eliz@gnu.org>,
> 	68559@debbugs.gnu.org
> 
> Liu Hui wrote:
> > On Tue, Feb 6, 2024 at 11:12 PM kobarity <kobarity@gmail.com> wrote:
> > 
> > > Sorry, I noticed one more thing.  As written in the comments below, it
> > > would be better to use `line-beginning-position' and
> > > `line-end-position' instead of `pos-bol' and `pos-eol'.
> > >
> > > ;; This is a GNU ELPA :core package.  Avoid functionality that is not
> > > ;; compatible with the version of Emacs recorded above.
> > 
> > Thanks for pointing it out! `pos-bol' was used because it can ignore
> > the prompt in `python-shell--get-multiline-input'. Fortunately, I find
> > `pos-bol' and `pos-eol' are available in the compat package since
> > 29.1.1.0, so it should be enough to upgrade the compat version in
> > Package-Requires. I have updated the attached patch.
> 
> I see.  The patch looks good to me.  I appreciate your efforts!

Thanks, I installed it on the master branch.

What about the other patch, one you posted in

  https://debbugs.gnu.org/cgi/bugreport.cgi?bug=68559#46

Does that one need to be installed as well, or is it included in the
one I just installed?

bug#68559: [PATCH] Improve Python shell completion

[Liu Hui]

On Thu, Feb 8, 2024 at 8:13 PM Eli Zaretskii <eliz@gnu.org> wrote:

> What about the other patch, one you posted in
>
>   https://debbugs.gnu.org/cgi/bugreport.cgi?bug=68559#46
>
> Does that one need to be installed as well, or is it included in the
> one I just installed?

The installed one is complete. Thanks.

bug#68559: [PATCH] Improve Python shell completion

[Eli Zaretskii]

> From: Liu Hui <liuhui1610@gmail.com>
> Date: Thu, 8 Feb 2024 21:33:10 +0800
> Cc: kobarity <kobarity@gmail.com>, 68559@debbugs.gnu.org
> 
> On Thu, Feb 8, 2024 at 8:13 PM Eli Zaretskii <eliz@gnu.org> wrote:
> 
> > What about the other patch, one you posted in
> >
> >   https://debbugs.gnu.org/cgi/bugreport.cgi?bug=68559#46
> >
> > Does that one need to be installed as well, or is it included in the
> > one I just installed?
> 
> The installed one is complete. Thanks.

Great, thanks.  Then this bug can be closed now, right?

bug#68559: [PATCH] Improve Python shell completion

[Liu Hui]

On Thu, Feb 8, 2024 at 9:46 PM Eli Zaretskii <eliz@gnu.org> wrote:
>
> > From: Liu Hui <liuhui1610@gmail.com>
> > Date: Thu, 8 Feb 2024 21:33:10 +0800
> > Cc: kobarity <kobarity@gmail.com>, 68559@debbugs.gnu.org
> >
> > On Thu, Feb 8, 2024 at 8:13 PM Eli Zaretskii <eliz@gnu.org> wrote:
> >
> > > What about the other patch, one you posted in
> > >
> > >   https://debbugs.gnu.org/cgi/bugreport.cgi?bug=68559#46
> > >
> > > Does that one need to be installed as well, or is it included in the
> > > one I just installed?
> >
> > The installed one is complete. Thanks.
>
> Great, thanks.  Then this bug can be closed now, right?

Yes, please.

bug#68559: [PATCH] Improve Python shell completion

[Eli Zaretskii]

> From: Liu Hui <liuhui1610@gmail.com>
> Date: Thu, 8 Feb 2024 22:16:23 +0800
> Cc: kobarity@gmail.com, 68559@debbugs.gnu.org
> 
> On Thu, Feb 8, 2024 at 9:46 PM Eli Zaretskii <eliz@gnu.org> wrote:
> >
> > > From: Liu Hui <liuhui1610@gmail.com>
> > > Date: Thu, 8 Feb 2024 21:33:10 +0800
> > > Cc: kobarity <kobarity@gmail.com>, 68559@debbugs.gnu.org
> > >
> > > On Thu, Feb 8, 2024 at 8:13 PM Eli Zaretskii <eliz@gnu.org> wrote:
> > >
> > > > What about the other patch, one you posted in
> > > >
> > > >   https://debbugs.gnu.org/cgi/bugreport.cgi?bug=68559#46
> > > >
> > > > Does that one need to be installed as well, or is it included in the
> > > > one I just installed?
> > >
> > > The installed one is complete. Thanks.
> >
> > Great, thanks.  Then this bug can be closed now, right?
> 
> Yes, please.

Thanks, closing.

bug#68559: [PATCH] Improve Python shell completion

[Mattias Engdegård]

[-- Attachment #1: Type: text/plain, Size: 319 bytes --]

The recent change on master, 0b9c7148fd, causes several failures in python-tests here (macOS). Log attached.

It seems to have something to do with completion. Maybe the new code is sensitive to details of Python's command line editor? The standard Python interpreter is unlikely to use GNU readline, for example.


[-- Attachment #2: python-tests.log --]
[-- Type: application/octet-stream, Size: 48228 bytes --]

  GEN      lisp/progmodes/python-tests.log
Running 360 tests (2024-02-15 15:38:05+0100, selector `(not (or (tag :unstable) (tag :nativecomp)))')
   passed    1/360  python-auto-fill-docstring (0.001040 sec)
Fontifying  *temp*-895436...
Fontifying  *temp*-895436... (syntactically...)
Fontifying  *temp*-895436... (regexps...)
Fontifying  *temp*-895436... (regexps....)
Fontifying  *temp*-895436... (regexps.....)
Fontifying  *temp*-895436... (regexps......)
Fontifying  *temp*-895436... (regexps.......)
Fontifying  *temp*-895436... (regexps........)
Fontifying  *temp*-895436... (regexps.........)
Fontifying  *temp*-895436... (regexps..........)
Fontifying  *temp*-895436... (regexps...........)
Fontifying  *temp*-895436... (regexps............)
Fontifying  *temp*-895436... (regexps.............)
Fontifying  *temp*-895436... (regexps..............)
Fontifying  *temp*-895436... (regexps...............)
Fontifying  *temp*-895436... (regexps................)

   passed    2/360  python-bob-infloop-avoid (0.000804 sec)
Test python-completion-at-point-1 backtrace:
  json-parse-string("__PYTHON_EL_eval_file(\"/var/folders/qy/zstv16390
  python--parse-json-array("__PYTHON_EL_eval_file(\"/var/folders/qy/zs
  python-shell-completion-get-completions(#<process Python[ *temp*-713
  python-shell-completion-at-point(#<process Python[ *temp*-713815]>)
  python-completion-at-point()
  completion--capf-wrapper(python-completion-at-point all)
  run-hook-wrapped(completion--capf-wrapper python-completion-at-point
  completion-at-point()
  apply(completion-at-point nil)
  (setq value-3676 (apply fn-3674 args-3675))
  (unwind-protect (setq value-3676 (apply fn-3674 args-3675)) (setq fo
  (if (unwind-protect (setq value-3676 (apply fn-3674 args-3675)) (set
  (let (form-description-3678) (if (unwind-protect (setq value-3676 (a
  (let ((value-3676 'ert-form-evaluation-aborted-3677)) (let (form-des
  (let* ((fn-3674 #'completion-at-point) (args-3675 (condition-case er
  (let ((inhibit-message t)) (python-shell-send-buffer) (python-tests-
  (progn (run-python nil t) (insert "\nimport abc\n") (goto-char (poin
  (unwind-protect (progn (run-python nil t) (insert "\nimport abc\n") 
  (let ((python-indent-guess-indent-offset nil) (python-shell-completi
  (progn (let ((python-indent-guess-indent-offset nil) (python-shell-c
  (unwind-protect (progn (let ((python-indent-guess-indent-offset nil)
  (save-current-buffer (set-buffer temp-buffer) (unwind-protect (progn
  (let ((temp-buffer (generate-new-buffer " *temp*" t))) (save-current
  (closure (t) nil (let* ((fn-3669 #'executable-find) (args-3670 (cond
  #f(compiled-function () #<bytecode -0x3de848aaf387fce>)()
  handler-bind-1(#f(compiled-function () #<bytecode -0x3de848aaf387fce
  ert--run-test-internal(#s(ert--test-execution-info :test #s(ert-test
  ert-run-test(#s(ert-test :name python-completion-at-point-1 :documen
  ert-run-or-rerun-test(#s(ert--stats :selector ... :tests ... :test-m
  ert-run-tests((not (or (tag :unstable) (tag :nativecomp))) #f(compil
  ert-run-tests-batch((not (or (tag :unstable) (tag :nativecomp))))
  ert-run-tests-batch-and-exit((not (or (tag :unstable) (tag :nativeco
  eval((ert-run-tests-batch-and-exit '(not (or (tag :unstable) (tag :n
  command-line-1(("-L" ":../../emacs/test" "-l" "ert" "-l" "lisp/progm
  command-line()
  normal-top-level()
Test python-completion-at-point-1 condition:
    (json-parse-error "invalid token near '_'" "<string>" 1 1 1)
   FAILED    3/360  python-completion-at-point-1 (0.404730 sec) at ../../emacs/test/lisp/progmodes/python-tests.el:4903
Test python-completion-at-point-2 backtrace:
  json-parse-string("__PYTHON_EL_eval_file(\"/var/folders/qy/zstv16390
  python--parse-json-array("__PYTHON_EL_eval_file(\"/var/folders/qy/zs
  python-shell-completion-get-completions(#<process Python[ *temp*-102
  python-shell-completion-at-point(#<process Python[ *temp*-102788]>)
  python-completion-at-point()
  completion--capf-wrapper(python-completion-at-point all)
  run-hook-wrapped(completion--capf-wrapper python-completion-at-point
  completion-at-point()
  apply(completion-at-point nil)
  (setq value-3691 (apply fn-3689 args-3690))
  (unwind-protect (setq value-3691 (apply fn-3689 args-3690)) (setq fo
  (if (unwind-protect (setq value-3691 (apply fn-3689 args-3690)) (set
  (let (form-description-3693) (if (unwind-protect (setq value-3691 (a
  (let ((value-3691 'ert-form-evaluation-aborted-3692)) (let (form-des
  (let* ((fn-3689 #'completion-at-point) (args-3690 (condition-case er
  (let ((inhibit-message t)) (python-shell-send-buffer) (python-tests-
  (progn (run-python nil t) (insert "\nimport abc\n") (goto-char (poin
  (unwind-protect (progn (run-python nil t) (insert "\nimport abc\n") 
  (let ((python-indent-guess-indent-offset nil) (python-shell-completi
  (progn (let ((python-indent-guess-indent-offset nil) (python-shell-c
  (unwind-protect (progn (let ((python-indent-guess-indent-offset nil)
  (save-current-buffer (set-buffer temp-buffer) (unwind-protect (progn
  (let ((temp-buffer (generate-new-buffer " *temp*" t))) (save-current
  (closure (t) nil (let* ((fn-3684 #'executable-find) (args-3685 (cond
  #f(compiled-function () #<bytecode -0x3de848aaf387fce>)()
  handler-bind-1(#f(compiled-function () #<bytecode -0x3de848aaf387fce
  ert--run-test-internal(#s(ert--test-execution-info :test #s(ert-test
  ert-run-test(#s(ert-test :name python-completion-at-point-2 :documen
  ert-run-or-rerun-test(#s(ert--stats :selector ... :tests ... :test-m
  ert-run-tests((not (or (tag :unstable) (tag :nativecomp))) #f(compil
  ert-run-tests-batch((not (or (tag :unstable) (tag :nativecomp))))
  ert-run-tests-batch-and-exit((not (or (tag :unstable) (tag :nativeco
  eval((ert-run-tests-batch-and-exit '(not (or (tag :unstable) (tag :n
  command-line-1(("-L" ":../../emacs/test" "-l" "ert" "-l" "lisp/progm
  command-line()
  normal-top-level()
Test python-completion-at-point-2 condition:
    (json-parse-error "invalid token near '_'" "<string>" 1 1 1)
   FAILED    4/360  python-completion-at-point-2 (0.381182 sec) at ../../emacs/test/lisp/progmodes/python-tests.el:4918
Test python-completion-at-point-native-1 backtrace:
  json-parse-string("__PYTHON_EL_eval_file(\"/var/folders/qy/zstv16390
  python--parse-json-array("__PYTHON_EL_eval_file(\"/var/folders/qy/zs
  python-shell-completion-get-completions(#<process Python[ *temp*-704
  python-shell-completion-at-point(#<process Python[ *temp*-704829]>)
  python-completion-at-point()
  completion--capf-wrapper(python-completion-at-point all)
  run-hook-wrapped(completion--capf-wrapper python-completion-at-point
  completion-at-point()
  apply(completion-at-point nil)
  (setq value-3718 (apply fn-3716 args-3717))
  (unwind-protect (setq value-3718 (apply fn-3716 args-3717)) (setq fo
  (if (unwind-protect (setq value-3718 (apply fn-3716 args-3717)) (set
  (let (form-description-3720) (if (unwind-protect (setq value-3718 (a
  (let ((value-3718 'ert-form-evaluation-aborted-3719)) (let (form-des
  (let* ((fn-3716 #'completion-at-point) (args-3717 (condition-case er
  (let ((inhibit-message t)) (python-shell-completion-native-turn-on) 
  (progn (run-python nil t) (insert "\nimport abc\n") (goto-char (poin
  (unwind-protect (progn (run-python nil t) (insert "\nimport abc\n") 
  (let ((python-indent-guess-indent-offset nil) (python-shell-completi
  (progn (let ((python-indent-guess-indent-offset nil) (python-shell-c
  (unwind-protect (progn (let ((python-indent-guess-indent-offset nil)
  (save-current-buffer (set-buffer temp-buffer) (unwind-protect (progn
  (let ((temp-buffer (generate-new-buffer " *temp*" t))) (save-current
  (closure (t) nil (let* ((fn-3711 #'executable-find) (args-3712 (cond
  #f(compiled-function () #<bytecode -0x3de848aaf387fce>)()
  handler-bind-1(#f(compiled-function () #<bytecode -0x3de848aaf387fce
  ert--run-test-internal(#s(ert--test-execution-info :test #s(ert-test
  ert-run-test(#s(ert-test :name python-completion-at-point-native-1 :
  ert-run-or-rerun-test(#s(ert--stats :selector ... :tests ... :test-m
  ert-run-tests((not (or (tag :unstable) (tag :nativecomp))) #f(compil
  ert-run-tests-batch((not (or (tag :unstable) (tag :nativecomp))))
  ert-run-tests-batch-and-exit((not (or (tag :unstable) (tag :nativeco
  eval((ert-run-tests-batch-and-exit '(not (or (tag :unstable) (tag :n
  command-line-1(("-L" ":../../emacs/test" "-l" "ert" "-l" "lisp/progm
  command-line()
  normal-top-level()
Test python-completion-at-point-native-1 condition:
    (json-parse-error "invalid token near '_'" "<string>" 1 1 1)
   FAILED    5/360  python-completion-at-point-native-1 (0.407805 sec) at ../../emacs/test/lisp/progmodes/python-tests.el:4966
Test python-completion-at-point-native-2 backtrace:
  json-parse-string("__PYTHON_EL_eval_file(\"/var/folders/qy/zstv16390
  python--parse-json-array("__PYTHON_EL_eval_file(\"/var/folders/qy/zs
  python-shell-completion-get-completions(#<process Python[ *temp*-734
  python-shell-completion-at-point(#<process Python[ *temp*-734580]>)
  python-completion-at-point()
  completion--capf-wrapper(python-completion-at-point all)
  run-hook-wrapped(completion--capf-wrapper python-completion-at-point
  completion-at-point()
  apply(completion-at-point nil)
  (setq value-3733 (apply fn-3731 args-3732))
  (unwind-protect (setq value-3733 (apply fn-3731 args-3732)) (setq fo
  (if (unwind-protect (setq value-3733 (apply fn-3731 args-3732)) (set
  (let (form-description-3735) (if (unwind-protect (setq value-3733 (a
  (let ((value-3733 'ert-form-evaluation-aborted-3734)) (let (form-des
  (let* ((fn-3731 #'completion-at-point) (args-3732 (condition-case er
  (let ((inhibit-message t)) (python-shell-completion-native-turn-on) 
  (progn (run-python nil t) (insert "\nimport abc\n") (goto-char (poin
  (unwind-protect (progn (run-python nil t) (insert "\nimport abc\n") 
  (let ((python-indent-guess-indent-offset nil) (python-shell-completi
  (progn (let ((python-indent-guess-indent-offset nil) (python-shell-c
  (unwind-protect (progn (let ((python-indent-guess-indent-offset nil)
  (save-current-buffer (set-buffer temp-buffer) (unwind-protect (progn
  (let ((temp-buffer (generate-new-buffer " *temp*" t))) (save-current
  (closure (t) nil (let* ((fn-3726 #'executable-find) (args-3727 (cond
  #f(compiled-function () #<bytecode -0x3de848aaf387fce>)()
  handler-bind-1(#f(compiled-function () #<bytecode -0x3de848aaf387fce
  ert--run-test-internal(#s(ert--test-execution-info :test #s(ert-test
  ert-run-test(#s(ert-test :name python-completion-at-point-native-2 :
  ert-run-or-rerun-test(#s(ert--stats :selector ... :tests ... :test-m
  ert-run-tests((not (or (tag :unstable) (tag :nativecomp))) #f(compil
  ert-run-tests-batch((not (or (tag :unstable) (tag :nativecomp))))
  ert-run-tests-batch-and-exit((not (or (tag :unstable) (tag :nativeco
  eval((ert-run-tests-batch-and-exit '(not (or (tag :unstable) (tag :n
  command-line-1(("-L" ":../../emacs/test" "-l" "ert" "-l" "lisp/progm
  command-line()
  normal-top-level()
Test python-completion-at-point-native-2 condition:
    (json-parse-error "invalid token near '_'" "<string>" 1 1 1)
   FAILED    6/360  python-completion-at-point-native-2 (0.399853 sec) at ../../emacs/test/lisp/progmodes/python-tests.el:4982
Test python-completion-at-point-native-with-eldoc-1 backtrace:
  json-parse-string("__PYTHON_EL_eval_file(\"/var/folders/qy/zstv16390
  python--parse-json-array("__PYTHON_EL_eval_file(\"/var/folders/qy/zs
  python-shell-completion-get-completions(#<process Python[ *temp*-155
  python-shell-completion-at-point(#<process Python[ *temp*-155177]>)
  python-completion-at-point()
  completion--capf-wrapper(python-completion-at-point all)
  run-hook-wrapped(completion--capf-wrapper python-completion-at-point
  completion-at-point()
  apply(completion-at-point nil)
  (setq value-3753 (apply fn-3751 args-3752))
  (unwind-protect (setq value-3753 (apply fn-3751 args-3752)) (setq fo
  (if (unwind-protect (setq value-3753 (apply fn-3751 args-3752)) (set
  (let (form-description-3755) (if (unwind-protect (setq value-3753 (a
  (let ((value-3753 'ert-form-evaluation-aborted-3754)) (let (form-des
  (let* ((fn-3751 #'completion-at-point) (args-3752 (condition-case er
  (let ((inhibit-message t)) (python-shell-completion-native-turn-on) 
  (progn (run-python nil t) (insert "\nimport abc\n") (goto-char (poin
  (unwind-protect (progn (run-python nil t) (insert "\nimport abc\n") 
  (let ((python-indent-guess-indent-offset nil) (python-shell-completi
  (progn (let ((python-indent-guess-indent-offset nil) (python-shell-c
  (unwind-protect (progn (let ((python-indent-guess-indent-offset nil)
  (save-current-buffer (set-buffer temp-buffer) (unwind-protect (progn
  (let ((temp-buffer (generate-new-buffer " *temp*" t))) (save-current
  (closure (t) nil (let* ((fn-3746 #'executable-find) (args-3747 (cond
  #f(compiled-function () #<bytecode -0x3de848aaf387fce>)()
  handler-bind-1(#f(compiled-function () #<bytecode -0x3de848aaf387fce
  ert--run-test-internal(#s(ert--test-execution-info :test #s(ert-test
  ert-run-test(#s(ert-test :name python-completion-at-point-native-wit
  ert-run-or-rerun-test(#s(ert--stats :selector ... :tests ... :test-m
  ert-run-tests((not (or (tag :unstable) (tag :nativecomp))) #f(compil
  ert-run-tests-batch((not (or (tag :unstable) (tag :nativecomp))))
  ert-run-tests-batch-and-exit((not (or (tag :unstable) (tag :nativeco
  eval((ert-run-tests-batch-and-exit '(not (or (tag :unstable) (tag :n
  command-line-1(("-L" ":../../emacs/test" "-l" "ert" "-l" "lisp/progm
  command-line()
  normal-top-level()
Test python-completion-at-point-native-with-eldoc-1 condition:
    (json-parse-error "invalid token near '_'" "<string>" 1 1 1)
   FAILED    7/360  python-completion-at-point-native-with-eldoc-1 (0.451243 sec) at ../../emacs/test/lisp/progmodes/python-tests.el:5015
Test python-completion-at-point-native-with-ffap-1 backtrace:
  json-parse-string("__PYTHON_EL_eval_file(\"/var/folders/qy/zstv16390
  python--parse-json-array("__PYTHON_EL_eval_file(\"/var/folders/qy/zs
  python-shell-completion-get-completions(#<process Python[ *temp*-392
  python-shell-completion-at-point(#<process Python[ *temp*-392361]>)
  python-completion-at-point()
  completion--capf-wrapper(python-completion-at-point all)
  run-hook-wrapped(completion--capf-wrapper python-completion-at-point
  completion-at-point()
  apply(completion-at-point nil)
  (setq value-3743 (apply fn-3741 args-3742))
  (unwind-protect (setq value-3743 (apply fn-3741 args-3742)) (setq fo
  (if (unwind-protect (setq value-3743 (apply fn-3741 args-3742)) (set
  (let (form-description-3745) (if (unwind-protect (setq value-3743 (a
  (let ((value-3743 'ert-form-evaluation-aborted-3744)) (let (form-des
  (let* ((fn-3741 #'completion-at-point) (args-3742 (condition-case er
  (let ((inhibit-message t)) (python-shell-completion-native-turn-on) 
  (progn (run-python nil t) (insert "\nimport abc\n") (goto-char (poin
  (unwind-protect (progn (run-python nil t) (insert "\nimport abc\n") 
  (let ((python-indent-guess-indent-offset nil) (python-shell-completi
  (progn (let ((python-indent-guess-indent-offset nil) (python-shell-c
  (unwind-protect (progn (let ((python-indent-guess-indent-offset nil)
  (save-current-buffer (set-buffer temp-buffer) (unwind-protect (progn
  (let ((temp-buffer (generate-new-buffer " *temp*" t))) (save-current
  (closure (t) nil (let* ((fn-3736 #'executable-find) (args-3737 (cond
  #f(compiled-function () #<bytecode -0x3de848aaf387fce>)()
  handler-bind-1(#f(compiled-function () #<bytecode -0x3de848aaf387fce
  ert--run-test-internal(#s(ert--test-execution-info :test #s(ert-test
  ert-run-test(#s(ert-test :name python-completion-at-point-native-wit
  ert-run-or-rerun-test(#s(ert--stats :selector ... :tests ... :test-m
  ert-run-tests((not (or (tag :unstable) (tag :nativecomp))) #f(compil
  ert-run-tests-batch((not (or (tag :unstable) (tag :nativecomp))))
  ert-run-tests-batch-and-exit((not (or (tag :unstable) (tag :nativeco
  eval((ert-run-tests-batch-and-exit '(not (or (tag :unstable) (tag :n
  command-line-1(("-L" ":../../emacs/test" "-l" "ert" "-l" "lisp/progm
  command-line()
  normal-top-level()
Test python-completion-at-point-native-with-ffap-1 condition:
    (json-parse-error "invalid token near '_'" "<string>" 1 1 1)
   FAILED    8/360  python-completion-at-point-native-with-ffap-1 (0.461626 sec) at ../../emacs/test/lisp/progmodes/python-tests.el:4999
   passed    9/360  python-completion-at-point-pdb-1 (0.403759 sec)
   passed   10/360  python-completion-at-point-while-running-1 (0.200969 sec)
   passed   11/360  python-eldoc--get-doc-at-point-1 (0.409866 sec)
   passed   12/360  python-eldoc--get-doc-at-point-while-running-1 (0.196968 sec)
   passed   13/360  python-eldoc--get-symbol-at-point-1 (0.001494 sec)
   passed   14/360  python-eldoc--get-symbol-at-point-2 (0.001215 sec)
   passed   15/360  python-eldoc--get-symbol-at-point-3 (0.000355 sec)
   passed   16/360  python-eldoc--get-symbol-at-point-4 (0.000367 sec)
   passed   17/360  python-end-of-defun-1 (0.000406 sec)
  skipped   18/360  python-ffap-module-path-1 (0.000411 sec)
   passed   19/360  python-ffap-module-path-while-running-1 (0.204620 sec)
   passed   20/360  python-fill-docstring (0.001313 sec)
   passed   21/360  python-fill-paragraph-single-quoted-string-1 (0.000401 sec)
   passed   22/360  python-fill-paragraph-single-quoted-string-2 (0.000313 sec)
   passed   23/360  python-fill-paragraph-triple-quoted-string-1 (0.003809 sec)
   passed   24/360  python-font-lock-assignment-statement-1 (0.000670 sec)
   passed   25/360  python-font-lock-assignment-statement-10 (0.000358 sec)
   passed   26/360  python-font-lock-assignment-statement-11 (0.000561 sec)
   passed   27/360  python-font-lock-assignment-statement-12 (0.000589 sec)
   passed   28/360  python-font-lock-assignment-statement-13 (0.000741 sec)
   passed   29/360  python-font-lock-assignment-statement-14 (0.000586 sec)
   passed   30/360  python-font-lock-assignment-statement-15 (0.000470 sec)
   passed   31/360  python-font-lock-assignment-statement-16 (0.000378 sec)
   passed   32/360  python-font-lock-assignment-statement-17 (0.000632 sec)
   passed   33/360  python-font-lock-assignment-statement-18 (0.000620 sec)
   passed   34/360  python-font-lock-assignment-statement-2 (0.000803 sec)
   passed   35/360  python-font-lock-assignment-statement-3 (0.000593 sec)
   passed   36/360  python-font-lock-assignment-statement-4 (0.000677 sec)
   passed   37/360  python-font-lock-assignment-statement-5 (0.000494 sec)
   passed   38/360  python-font-lock-assignment-statement-6 (0.000409 sec)
   passed   39/360  python-font-lock-assignment-statement-7 (0.000516 sec)
   passed   40/360  python-font-lock-assignment-statement-8 (0.000442 sec)
   passed   41/360  python-font-lock-assignment-statement-9 (0.000669 sec)
   passed   42/360  python-font-lock-escape-sequence-bytes-newline (0.000449 sec)
   passed   43/360  python-font-lock-escape-sequence-hex-octal (0.000797 sec)
   passed   44/360  python-font-lock-escape-sequence-multiline-string (0.015089 sec)
   passed   45/360  python-font-lock-escape-sequence-string-newline (0.001174 sec)
   passed   46/360  python-font-lock-escape-sequence-unicode (0.000670 sec)
   passed   47/360  python-font-lock-keywords-level-1-1 (0.000784 sec)
   passed   48/360  python-font-lock-keywords-level-1-2 (0.000567 sec)
   passed   49/360  python-font-lock-operator-1 (0.000616 sec)
   passed   50/360  python-font-lock-operator-2 (0.000685 sec)
   passed   51/360  python-font-lock-raw-escape-sequence (0.000931 sec)
   passed   52/360  python-font-lock-string-literal-concatenation (0.000505 sec)
Hiding all blocks... 
Hiding all blocks...done
   passed   53/360  python-hideshow-hide-all-1 (0.000726 sec)
Hiding all blocks... 
Hiding all blocks...done
   passed   54/360  python-hideshow-hide-all-2 (0.000482 sec)
Hiding all blocks... 
Hiding all blocks...done
   passed   55/360  python-hideshow-hide-all-3 (0.000423 sec)
   passed   56/360  python-hideshow-hide-block-1 (0.000525 sec)
Hiding blocks ...
Hiding blocks ... done
Showing all blocks ...
Showing all blocks ... done
   passed   57/360  python-hideshow-hide-levels-1 (0.001158 sec)
Showing all blocks ...
Showing all blocks ... done
   passed   58/360  python-hideshow-hide-levels-2 (0.063466 sec)
Hiding blocks ...
Hiding blocks ... done
   passed   59/360  python-hideshow-hide-levels-3 (0.001151 sec)
Hiding blocks ...
Hiding blocks ... done
   passed   60/360  python-hideshow-hide-levels-4 (0.001123 sec)
   passed   61/360  python-imenu-create-flat-index-1 (0.000588 sec)
   passed   62/360  python-imenu-create-flat-index-2 (0.000365 sec)
   passed   63/360  python-imenu-create-index-1 (0.000564 sec)
   passed   64/360  python-imenu-create-index-2 (0.000345 sec)
   passed   65/360  python-imenu-create-index-3 (0.000337 sec)
   passed   66/360  python-imenu-create-index-4 (0.000354 sec)
   passed   67/360  python-indent-after-async-block-1 (0.000366 sec)
   passed   68/360  python-indent-after-async-block-2 (0.000356 sec)
   passed   69/360  python-indent-after-async-block-3 (0.000366 sec)
   passed   70/360  python-indent-after-backslash-1 (0.000755 sec)
   passed   71/360  python-indent-after-backslash-2 (0.001117 sec)
   passed   72/360  python-indent-after-backslash-3 (0.000536 sec)
   passed   73/360  python-indent-after-backslash-4 (0.001198 sec)
   passed   74/360  python-indent-after-backslash-5 (0.000804 sec)
   passed   75/360  python-indent-after-backslash-6 (0.000594 sec)
   passed   76/360  python-indent-after-bare-match (0.000396 sec)
   passed   77/360  python-indent-after-block-1 (0.000360 sec)
   passed   78/360  python-indent-after-block-2 (0.000345 sec)
   passed   79/360  python-indent-after-block-3 (0.000522 sec)
   passed   80/360  python-indent-after-case-block (0.000358 sec)
   passed   81/360  python-indent-after-comment-1 (0.000815 sec)
   passed   82/360  python-indent-after-comment-2 (0.001021 sec)
   passed   83/360  python-indent-after-comment-3 (0.000543 sec)
   passed   84/360  python-indent-after-match-block (0.000368 sec)
   passed   85/360  python-indent-after-re-match (0.000388 sec)
   passed   86/360  python-indent-base-case (0.000483 sec)
   passed   87/360  python-indent-block-enders-1 (0.000707 sec)
   passed   88/360  python-indent-block-enders-2 (0.001406 sec)
   passed   89/360  python-indent-block-enders-3 (0.000738 sec)
   passed   90/360  python-indent-block-enders-4 (0.000479 sec)
   passed   91/360  python-indent-block-enders-5 (0.000496 sec)
   passed   92/360  python-indent-dedent-line-backspace-1 (0.000467 sec)
   passed   93/360  python-indent-dedent-line-backspace-2 (0.000312 sec)
   passed   94/360  python-indent-dedent-line-backspace-3 (0.000580 sec)
   passed   95/360  python-indent-dedenters-1 (0.000499 sec)
Closes if hide_details:
Closes except Exception:
Closes if save:
   passed   96/360  python-indent-dedenters-2 (0.002075 sec)
Closes try:
   passed   97/360  python-indent-dedenters-3 (0.000975 sec)
Closes try:
   passed   98/360  python-indent-dedenters-4 (0.000954 sec)
Closes if save:
   passed   99/360  python-indent-dedenters-5 (0.001225 sec)
   passed  100/360  python-indent-dedenters-6 (0.000430 sec)
   passed  101/360  python-indent-dedenters-7 (0.000557 sec)
Closes if (a == 1 or
Closes if (a == 1 or
Closes if (a == 1 or
   passed  102/360  python-indent-dedenters-8 (0.054888 sec)
Closes case 1:
   passed  103/360  python-indent-dedenters-9 (0.001045 sec)
Closes if hide_details:
Closes except Exception:
Closes if save:
   passed  104/360  python-indent-dedenters-comment-else (0.002278 sec)
   passed  105/360  python-indent-electric-colon-1 (0.000376 sec)
Closes if do:
   passed  106/360  python-indent-electric-colon-2 (0.000483 sec)
Closes if do:
Closes if do:
Closes if do:
   passed  107/360  python-indent-electric-colon-3 (0.000718 sec)
Closes if True:
   passed  108/360  python-indent-electric-colon-4 (0.000566 sec)
   passed  109/360  python-indent-electric-comma-after-multiline-string (0.000423 sec)
   passed  110/360  python-indent-electric-comma-inside-multiline-string (0.000405 sec)
   passed  111/360  python-indent-hanging-close-paren (0.000351 sec)
   passed  112/360  python-indent-inside-paren-1 (0.001338 sec)
   passed  113/360  python-indent-inside-paren-2 (0.001057 sec)
   passed  114/360  python-indent-inside-paren-3 (0.000878 sec)
   passed  115/360  python-indent-inside-paren-4 (0.000625 sec)
   passed  116/360  python-indent-inside-paren-5 (0.000799 sec)
   passed  117/360  python-indent-inside-paren-6 (0.000444 sec)
   passed  118/360  python-indent-inside-paren-7 (0.000303 sec)
   passed  119/360  python-indent-inside-paren-8 (0.000416 sec)
   passed  120/360  python-indent-inside-paren-9 (0.000911 sec)
   passed  121/360  python-indent-inside-paren-block-1 (0.000900 sec)
   passed  122/360  python-indent-inside-paren-block-2 (0.000817 sec)
   passed  123/360  python-indent-inside-paren-block-3 (0.000588 sec)
   passed  124/360  python-indent-inside-paren-block-4 (0.000584 sec)
   passed  125/360  python-indent-inside-string-1 (0.000646 sec)
   passed  126/360  python-indent-inside-string-2 (0.001953 sec)
   passed  127/360  python-indent-inside-string-3 (0.000754 sec)
   passed  128/360  python-indent-pep8-1 (0.000450 sec)
   passed  129/360  python-indent-pep8-2 (0.000557 sec)
   passed  130/360  python-indent-pep8-3 (0.000930 sec)
   passed  131/360  python-indent-region-1 (0.000539 sec)
   passed  132/360  python-indent-region-2 (0.000637 sec)
   passed  133/360  python-indent-region-3 (0.000518 sec)
   passed  134/360  python-indent-region-4 (0.000454 sec)
   passed  135/360  python-indent-region-5 (0.002005 sec)
   passed  136/360  python-info-assignment-continuation-line-p-1 (0.000400 sec)
   passed  137/360  python-info-assignment-continuation-line-p-2 (0.000387 sec)
   passed  138/360  python-info-assignment-statement-p-1 (0.000417 sec)
   passed  139/360  python-info-assignment-statement-p-2 (0.000410 sec)
   passed  140/360  python-info-assignment-statement-p-3 (0.000480 sec)
   passed  141/360  python-info-beginning-of-backslash-1 (0.000776 sec)
   passed  142/360  python-info-beginning-of-block-p-1 (0.000614 sec)
   passed  143/360  python-info-beginning-of-block-p-2 (0.000437 sec)
   passed  144/360  python-info-beginning-of-statement-p-1 (0.000333 sec)
   passed  145/360  python-info-beginning-of-statement-p-2 (0.000726 sec)
   passed  146/360  python-info-block-continuation-line-p-1 (0.000608 sec)
   passed  147/360  python-info-block-continuation-line-p-2 (0.000561 sec)
   passed  148/360  python-info-continuation-line-p-1 (0.000517 sec)
   passed  149/360  python-info-current-defun-1 (0.000750 sec)
   passed  150/360  python-info-current-defun-2 (0.011937 sec)
   passed  151/360  python-info-current-defun-3 (0.008679 sec)
   passed  152/360  python-info-current-defun-4 (0.001098 sec)
   passed  153/360  python-info-current-line-comment-p-1 (0.000450 sec)
   passed  154/360  python-info-current-line-empty-p (0.000382 sec)
   passed  155/360  python-info-current-symbol-1 (0.000746 sec)
   passed  156/360  python-info-current-symbol-2 (0.000596 sec)
   failed  157/360  python-info-current-symbol-3 (0.000335 sec)
   passed  158/360  python-info-dedenter-opening-block-message-1 (0.000274 sec)
Closes try:
Closes try:
   passed  159/360  python-info-dedenter-opening-block-message-2 (0.000339 sec)
Closes except:
Closes except:
   passed  160/360  python-info-dedenter-opening-block-message-3 (0.000832 sec)
Closes else:
Closes else:
   passed  161/360  python-info-dedenter-opening-block-message-4 (0.000633 sec)
Closes if a:
Closes if a:
   passed  162/360  python-info-dedenter-opening-block-message-5 (0.000618 sec)
   passed  163/360  python-info-dedenter-opening-block-position-1 (0.001020 sec)
   passed  164/360  python-info-dedenter-opening-block-position-2 (0.000366 sec)
   passed  165/360  python-info-dedenter-opening-block-position-3 (0.001109 sec)
   passed  166/360  python-info-dedenter-opening-block-positions-1 (0.001203 sec)
   passed  167/360  python-info-dedenter-opening-block-positions-2 (0.000448 sec)
   passed  168/360  python-info-dedenter-opening-block-positions-3 (0.000665 sec)
   passed  169/360  python-info-dedenter-opening-block-positions-4 (0.000411 sec)
   passed  170/360  python-info-dedenter-opening-block-positions-5 (0.000499 sec)
   passed  171/360  python-info-dedenter-opening-block-positions-6 (0.000833 sec)
   passed  172/360  python-info-dedenter-opening-block-positions-7 (0.000914 sec)
   passed  173/360  python-info-dedenter-statement-p-1 (0.000607 sec)
   passed  174/360  python-info-dedenter-statement-p-2 (0.000506 sec)
   passed  175/360  python-info-dedenter-statement-p-3 (0.000331 sec)
   passed  176/360  python-info-dedenter-statement-p-4 (0.000340 sec)
   passed  177/360  python-info-dedenter-statement-p-5 (0.000324 sec)
   passed  178/360  python-info-dedenter-statement-p-6 (0.000377 sec)
   passed  179/360  python-info-docstring-p-1 (0.001164 sec)
   passed  180/360  python-info-docstring-p-2 (0.000981 sec)
   passed  181/360  python-info-docstring-p-3 (0.001510 sec)
   passed  182/360  python-info-docstring-p-4 (0.001840 sec)
   passed  183/360  python-info-docstring-p-5 (0.002432 sec)
   passed  184/360  python-info-docstring-p-6 (0.001988 sec)
   passed  185/360  python-info-docstring-p-7 (0.000505 sec)
   passed  186/360  python-info-docstring-p-8 (0.000400 sec)
   passed  187/360  python-info-encoding-1 (0.000639 sec)
   passed  188/360  python-info-encoding-2 (0.000430 sec)
   passed  189/360  python-info-encoding-from-cookie-1 (0.000405 sec)
   passed  190/360  python-info-encoding-from-cookie-2 (0.000408 sec)
   passed  191/360  python-info-encoding-from-cookie-3 (0.000286 sec)
   passed  192/360  python-info-encoding-from-cookie-4 (0.000264 sec)
   passed  193/360  python-info-encoding-from-cookie-5 (0.000270 sec)
   passed  194/360  python-info-encoding-from-cookie-6 (0.000265 sec)
   passed  195/360  python-info-encoding-from-cookie-7 (0.000279 sec)
   passed  196/360  python-info-end-of-block-p-1 (0.000608 sec)
   passed  197/360  python-info-end-of-block-p-2 (0.000722 sec)
   passed  198/360  python-info-end-of-statement-p-1 (0.000410 sec)
   passed  199/360  python-info-end-of-statement-p-2 (0.000461 sec)
   passed  200/360  python-info-line-ends-backslash-p-1 (0.000418 sec)
   passed  201/360  python-info-looking-at-beginning-of-block-1 (0.000500 sec)
   passed  202/360  python-info-looking-at-beginning-of-defun-1 (0.000797 sec)
   passed  203/360  python-info-looking-at-beginning-of-defun-2 (0.000608 sec)
   passed  204/360  python-info-looking-at-beginning-of-defun-3 (0.000522 sec)
   passed  205/360  python-info-statement-ends-block-p-1 (0.000447 sec)
   passed  206/360  python-info-statement-ends-block-p-2 (0.000436 sec)
   passed  207/360  python-info-statement-starts-block-p-1 (0.000346 sec)
   passed  208/360  python-info-statement-starts-block-p-2 (0.000390 sec)
   passed  209/360  python-info-triple-quoted-string-p-1 (0.000493 sec)
   passed  210/360  python-info-triple-quoted-string-p-2 (0.000383 sec)
   passed  211/360  python-info-triple-quoted-string-p-3 (0.000473 sec)
Mark set
Mark set
   passed  212/360  python-mark-defun-1 (0.001020 sec)
Mark set
Mark set
   passed  213/360  python-mark-defun-2 (0.001136 sec)
Mark set
Mark set
   passed  214/360  python-mark-defun-3 (0.000968 sec)
Mark set
Mark set
   passed  215/360  python-mark-defun-4 (0.000974 sec)
Mark set
Mark set
Mark set
Mark set
   passed  216/360  python-mark-defun-5 (0.001505 sec)
   passed  217/360  python-nav-backward-defun-1 (0.000660 sec)
   passed  218/360  python-nav-backward-defun-2 (0.000493 sec)
   passed  219/360  python-nav-backward-defun-3 (0.000345 sec)
   passed  220/360  python-nav-backward-defun-4 (0.000305 sec)
   passed  221/360  python-nav-backward-statement-1 (0.000463 sec)
   failed  222/360  python-nav-backward-statement-2 (0.000408 sec)
   failed  223/360  python-nav-backward-up-list-1 (0.000457 sec)
   passed  224/360  python-nav-beginning-of-block-1 (0.000862 sec)
   passed  225/360  python-nav-beginning-of-block-2 (0.000381 sec)
   passed  226/360  python-nav-beginning-of-defun-1 (0.001112 sec)
   passed  227/360  python-nav-beginning-of-defun-2 (0.000954 sec)
   passed  228/360  python-nav-beginning-of-defun-3 (0.000628 sec)
   passed  229/360  python-nav-beginning-of-defun-4 (0.000821 sec)
   passed  230/360  python-nav-beginning-of-defun-5 (0.000534 sec)
   passed  231/360  python-nav-beginning-of-defun-6 (0.000737 sec)
   passed  232/360  python-nav-beginning-of-statement-1 (0.000764 sec)
   passed  233/360  python-nav-end-of-block-1 (0.001861 sec)
   passed  234/360  python-nav-end-of-block-2 (0.000340 sec)
   passed  235/360  python-nav-end-of-defun-1 (0.000966 sec)
   passed  236/360  python-nav-end-of-defun-2 (0.002294 sec)
   passed  237/360  python-nav-end-of-defun-3 (0.000347 sec)
   passed  238/360  python-nav-end-of-statement-1 (0.000533 sec)
   passed  239/360  python-nav-end-of-statement-2 (0.000482 sec)
   passed  240/360  python-nav-end-of-statement-3 (0.000539 sec)
   passed  241/360  python-nav-end-of-statement-4 (0.000551 sec)
   passed  242/360  python-nav-forward-block-1 (0.000995 sec)
   passed  243/360  python-nav-forward-block-2 (0.000373 sec)
   passed  244/360  python-nav-forward-defun-1 (0.000442 sec)
   passed  245/360  python-nav-forward-defun-2 (0.000595 sec)
   passed  246/360  python-nav-forward-defun-3 (0.000470 sec)
   passed  247/360  python-nav-forward-defun-4 (0.000805 sec)
   passed  248/360  python-nav-forward-sexp-1 (0.001713 sec)
   passed  249/360  python-nav-forward-sexp-2 (0.001812 sec)
   passed  250/360  python-nav-forward-sexp-3 (0.001452 sec)
   passed  251/360  python-nav-forward-sexp-safe-1 (0.000907 sec)
   passed  252/360  python-nav-forward-statement-1 (0.000523 sec)
   passed  253/360  python-nav-up-list-1 (0.000296 sec)
   passed  254/360  python-parens-electric-indent-1 (0.001948 sec)
   passed  255/360  python-shell-buffer-substring-1 (0.000728 sec)
   passed  256/360  python-shell-buffer-substring-10 (0.000505 sec)
   passed  257/360  python-shell-buffer-substring-11 (0.000599 sec)
   passed  258/360  python-shell-buffer-substring-12 (0.000640 sec)
   passed  259/360  python-shell-buffer-substring-13 (0.000773 sec)
   passed  260/360  python-shell-buffer-substring-14 (0.000511 sec)
   passed  261/360  python-shell-buffer-substring-15 (0.000452 sec)
   passed  262/360  python-shell-buffer-substring-16 (0.000440 sec)
   passed  263/360  python-shell-buffer-substring-17 (0.000998 sec)
   passed  264/360  python-shell-buffer-substring-18 (0.000789 sec)
   passed  265/360  python-shell-buffer-substring-2 (0.000863 sec)
   passed  266/360  python-shell-buffer-substring-3 (0.000638 sec)
   passed  267/360  python-shell-buffer-substring-4 (0.000699 sec)
   passed  268/360  python-shell-buffer-substring-5 (0.000605 sec)
   passed  269/360  python-shell-buffer-substring-6 (0.000535 sec)
   passed  270/360  python-shell-buffer-substring-7 (0.000522 sec)
   passed  271/360  python-shell-buffer-substring-8 (0.039097 sec)
   passed  272/360  python-shell-buffer-substring-9 (0.000453 sec)
   passed  273/360  python-shell-calculate-exec-path-1 (0.000117 sec)
   passed  274/360  python-shell-calculate-exec-path-2 (0.000089 sec)
   passed  275/360  python-shell-calculate-exec-path-3 (0.000085 sec)
   passed  276/360  python-shell-calculate-exec-path-4 (0.001826 sec)
   passed  277/360  python-shell-calculate-exec-path-5 (0.000082 sec)
   passed  278/360  python-shell-calculate-exec-path-6 (0.000387 sec)
   passed  279/360  python-shell-calculate-process-environment-1 (0.000137 sec)
   passed  280/360  python-shell-calculate-process-environment-2 (0.000159 sec)
   passed  281/360  python-shell-calculate-process-environment-3 (0.000152 sec)
   passed  282/360  python-shell-calculate-process-environment-4 (0.000118 sec)
   passed  283/360  python-shell-calculate-process-environment-5 (0.000115 sec)
   passed  284/360  python-shell-calculate-process-environment-6 (0.000116 sec)
   passed  285/360  python-shell-calculate-process-environment-7 (0.000127 sec)
   passed  286/360  python-shell-calculate-process-environment-8 (0.000124 sec)
   passed  287/360  python-shell-calculate-pythonpath-1 (0.000108 sec)
   passed  288/360  python-shell-calculate-pythonpath-2 (0.000079 sec)
Test python-shell-completion-at-point-1 backtrace:
  signal(json-parse-error ("invalid token near '_'" "<string>" 1 1 1))
  apply(signal (json-parse-error ("invalid token near '_'" "<string>" 
  (setq value-3589 (apply fn-3587 args-3588))
  (unwind-protect (setq value-3589 (apply fn-3587 args-3588)) (setq fo
  (if (unwind-protect (setq value-3589 (apply fn-3587 args-3588)) (set
  (let (form-description-3591) (if (unwind-protect (setq value-3589 (a
  (let ((value-3589 'ert-form-evaluation-aborted-3590)) (let (form-des
  (let* ((fn-3587 #'nth) (args-3588 (condition-case err (list 2 (pytho
  (save-current-buffer (set-buffer (process-buffer shell-process)) (in
  (let ((shell-process (python-shell-get-process-or-error))) (save-cur
  (progn (run-python nil t) (insert "") (goto-char (point-min)) (pytho
  (unwind-protect (progn (run-python nil t) (insert "") (goto-char (po
  (let ((python-indent-guess-indent-offset nil) (python-shell-completi
  (progn (let ((python-indent-guess-indent-offset nil) (python-shell-c
  (unwind-protect (progn (let ((python-indent-guess-indent-offset nil)
  (save-current-buffer (set-buffer temp-buffer) (unwind-protect (progn
  (let ((temp-buffer (generate-new-buffer " *temp*" t))) (save-current
  (closure (t) nil (let* ((fn-3582 #'executable-find) (args-3583 (cond
  #f(compiled-function () #<bytecode -0x3de848aaf387fce>)()
  handler-bind-1(#f(compiled-function () #<bytecode -0x3de848aaf387fce
  ert--run-test-internal(#s(ert--test-execution-info :test #s(ert-test
  ert-run-test(#s(ert-test :name python-shell-completion-at-point-1 :d
  ert-run-or-rerun-test(#s(ert--stats :selector ... :tests ... :test-m
  ert-run-tests((not (or (tag :unstable) (tag :nativecomp))) #f(compil
  ert-run-tests-batch((not (or (tag :unstable) (tag :nativecomp))))
  ert-run-tests-batch-and-exit((not (or (tag :unstable) (tag :nativeco
  eval((ert-run-tests-batch-and-exit '(not (or (tag :unstable) (tag :n
  command-line-1(("-L" ":../../emacs/test" "-l" "ert" "-l" "lisp/progm
  command-line()
  normal-top-level()
Test python-shell-completion-at-point-1 condition:
    (json-parse-error "invalid token near '_'" "<string>" 1 1 1)
   FAILED  289/360  python-shell-completion-at-point-1 (0.394116 sec) at ../../emacs/test/lisp/progmodes/python-tests.el:4777
  skipped  290/360  python-shell-completion-at-point-ipython (0.000420 sec)
Warning (python): Your `python-shell-interpreter' doesn't seem to support readline, yet `python-shell-completion-native-enable' was t and "python3" is not part of the `python-shell-completion-native-disabled-interpreters' list.  Native completions have been disabled locally. Consider installing the python package "readline". 
  skipped  291/360  python-shell-completion-at-point-jedi-completer (0.280412 sec)
Warning (python): Your `python-shell-interpreter' doesn't seem to support readline, yet `python-shell-completion-native-enable' was t and "python3" is not part of the `python-shell-completion-native-disabled-interpreters' list.  Native completions have been disabled locally. Consider installing the python package "readline". 
Test python-shell-completion-at-point-native-1 backtrace:
  signal(json-parse-error ("invalid token near '_'" "<string>" 1 1 1))
  apply(signal (json-parse-error ("invalid token near '_'" "<string>" 
  (setq value-3604 (apply fn-3602 args-3603))
  (unwind-protect (setq value-3604 (apply fn-3602 args-3603)) (setq fo
  (if (unwind-protect (setq value-3604 (apply fn-3602 args-3603)) (set
  (let (form-description-3606) (if (unwind-protect (setq value-3604 (a
  (let ((value-3604 'ert-form-evaluation-aborted-3605)) (let (form-des
  (let* ((fn-3602 #'nth) (args-3603 (condition-case err (list 2 (pytho
  (save-current-buffer (set-buffer (process-buffer shell-process)) (in
  (let ((shell-process (python-shell-get-process-or-error))) (save-cur
  (progn (run-python nil t) (insert "") (goto-char (point-min)) (pytho
  (unwind-protect (progn (run-python nil t) (insert "") (goto-char (po
  (let ((python-indent-guess-indent-offset nil) (python-shell-completi
  (progn (let ((python-indent-guess-indent-offset nil) (python-shell-c
  (unwind-protect (progn (let ((python-indent-guess-indent-offset nil)
  (save-current-buffer (set-buffer temp-buffer) (unwind-protect (progn
  (let ((temp-buffer (generate-new-buffer " *temp*" t))) (save-current
  (closure (t) nil (let* ((fn-3597 #'executable-find) (args-3598 (cond
  #f(compiled-function () #<bytecode -0x3de848aaf387fce>)()
  handler-bind-1(#f(compiled-function () #<bytecode -0x3de848aaf387fce
  ert--run-test-internal(#s(ert--test-execution-info :test #s(ert-test
  ert-run-test(#s(ert-test :name python-shell-completion-at-point-nati
  ert-run-or-rerun-test(#s(ert--stats :selector ... :tests ... :test-m
  ert-run-tests((not (or (tag :unstable) (tag :nativecomp))) #f(compil
  ert-run-tests-batch((not (or (tag :unstable) (tag :nativecomp))))
  ert-run-tests-batch-and-exit((not (or (tag :unstable) (tag :nativeco
  eval((ert-run-tests-batch-and-exit '(not (or (tag :unstable) (tag :n
  command-line-1(("-L" ":../../emacs/test" "-l" "ert" "-l" "lisp/progm
  command-line()
  normal-top-level()
Test python-shell-completion-at-point-native-1 condition:
    (json-parse-error "invalid token near '_'" "<string>" 1 1 1)
   FAILED  292/360  python-shell-completion-at-point-native-1 (0.409009 sec) at ../../emacs/test/lisp/progmodes/python-tests.el:4790
   passed  293/360  python-shell-completion-native-interpreter-disabled-p-1 (0.000129 sec)
Can't guess python-indent-offset, using defaults: 4
   passed  294/360  python-shell-get-process-1 (0.119660 sec)
   passed  295/360  python-shell-get-process-name-1 (0.000397 sec)
Can't guess python-indent-offset, using defaults: 4
   passed  296/360  python-shell-get-process-name-2 (0.007572 sec)
Can't guess python-indent-offset, using defaults: 4
   passed  297/360  python-shell-internal-get-or-create-process-1 (0.061770 sec)
   passed  298/360  python-shell-internal-get-process-name-1 (0.000318 sec)
Can't guess python-indent-offset, using defaults: 4
   passed  299/360  python-shell-internal-get-process-name-2 (0.006945 sec)
   passed  300/360  python-shell-make-comint-1 (0.062339 sec)
   passed  301/360  python-shell-make-comint-2 (0.051213 sec)
   passed  302/360  python-shell-make-comint-3 (0.050301 sec)
   passed  303/360  python-shell-make-comint-4 (0.055261 sec)
   passed  304/360  python-shell-prompt-detect-1 (0.077066 sec)
   passed  305/360  python-shell-prompt-detect-2 (0.078614 sec)
   passed  306/360  python-shell-prompt-detect-3 (0.000390 sec)
Warning (python): Python shell prompts cannot be detected.
If your emacs session hangs when starting python shells
recover with `keyboard-quit' and then try fixing the
interactive flag for your interpreter by adjusting the
`python-shell-interpreter-interactive-arg' or add regexps
matching shell prompts in the directory-local friendly vars:
  + `python-shell-prompt-regexp'
  + `python-shell-prompt-block-regexp'
  + `python-shell-prompt-output-regexp'
Or alternatively in:
  + `python-shell-prompt-input-regexps'
  + `python-shell-prompt-output-regexps'
   passed  307/360  python-shell-prompt-detect-4 (0.083172 sec)
   passed  308/360  python-shell-prompt-detect-5 (0.079290 sec)
   passed  309/360  python-shell-prompt-detect-6 (0.001412 sec)
   passed  310/360  python-shell-prompt-set-calculated-regexps-1 (0.000211 sec)
   passed  311/360  python-shell-prompt-set-calculated-regexps-2 (0.000106 sec)
   passed  312/360  python-shell-prompt-set-calculated-regexps-3 (0.000105 sec)
   passed  313/360  python-shell-prompt-set-calculated-regexps-4 (0.000103 sec)
   passed  314/360  python-shell-prompt-set-calculated-regexps-5 (0.000100 sec)
   passed  315/360  python-shell-prompt-set-calculated-regexps-6 (0.083597 sec)
   passed  316/360  python-shell-prompt-validate-regexps-1 (0.000161 sec)
   passed  317/360  python-shell-prompt-validate-regexps-2 (0.000198 sec)
   passed  318/360  python-shell-prompt-validate-regexps-3 (0.000152 sec)
   passed  319/360  python-shell-prompt-validate-regexps-4 (0.000123 sec)
   passed  320/360  python-shell-prompt-validate-regexps-5 (0.000118 sec)
   passed  321/360  python-shell-prompt-validate-regexps-6 (0.000111 sec)
   passed  322/360  python-shell-prompt-validate-regexps-7 (0.000087 sec)
   passed  323/360  python-shell-with-environment-1 (0.000323 sec)
   passed  324/360  python-shell-with-environment-2 (0.000575 sec)
   passed  325/360  python-shell-with-environment-3 (0.000584 sec)
   passed  326/360  python-syntax-after-python-backspace (0.000540 sec)
   passed  327/360  python-syntax-context-1 (0.000579 sec)
   passed  328/360  python-tests--fill-long-first-line (0.001301 sec)
   passed  329/360  python-tests--flymake-command-output-pattern (0.000159 sec)
   passed  330/360  python-tests--run-python-selects-window (0.089348 sec)
   passed  331/360  python-tests-look-at-1 (0.000405 sec)
   passed  332/360  python-tests-look-at-2 (0.000268 sec)
   passed  333/360  python-triple-double-quote-pairing (0.002917 sec)
   passed  334/360  python-triple-single-quote-pairing (0.001202 sec)
   passed  335/360  python-ts-mode-assignement-face-2 (0.052428 sec)
   passed  336/360  python-ts-mode-builtin-call-face (0.019162 sec)
   passed  337/360  python-ts-mode-class-patterns-face (0.002167 sec)
   passed  338/360  python-ts-mode-compound-keywords-face (0.012803 sec)
   passed  339/360  python-ts-mode-disabled-string-interpolation (0.002925 sec)
   passed  340/360  python-ts-mode-dotted-decorator-face-1 (0.003119 sec)
   passed  341/360  python-ts-mode-dotted-decorator-face-2 (0.002183 sec)
   passed  342/360  python-ts-mode-interpolation-doc-string (0.001884 sec)
   passed  343/360  python-ts-mode-interpolation-nested-string (0.001666 sec)
   passed  344/360  python-ts-mode-isinstance-type-face-1 (0.001781 sec)
   passed  345/360  python-ts-mode-isinstance-type-face-2 (0.001595 sec)
   passed  346/360  python-ts-mode-isinstance-type-face-3 (0.001949 sec)
   passed  347/360  python-ts-mode-level-fontification-wo-interpolation (0.001787 sec)
   passed  348/360  python-ts-mode-named-assignement-face-1 (0.001459 sec)
   passed  349/360  python-ts-mode-nested-types-face-1 (0.001656 sec)
   passed  350/360  python-ts-mode-superclass-type-face (0.001582 sec)
   passed  351/360  python-ts-mode-types-face-1 (0.001745 sec)
   passed  352/360  python-ts-mode-types-face-2 (0.002995 sec)
   passed  353/360  python-ts-mode-types-face-3 (0.003777 sec)
   passed  354/360  python-ts-mode-union-types-face-1 (0.002735 sec)
   passed  355/360  python-ts-mode-union-types-face-2 (0.001914 sec)
   passed  356/360  python-util-clone-local-variables-1 (0.000445 sec)
   passed  357/360  python-util-forward-comment-1 (0.001760 sec)
   passed  358/360  python-util-goto-line-1 (0.000713 sec)
   passed  359/360  python-util-strip-string-1 (0.000253 sec)
   passed  360/360  python-util-valid-regexp-p-1 (0.000145 sec)

Ran 360 tests, 349 results as expected, 8 unexpected, 3 skipped (2024-02-15 15:38:14+0100, 8.953487 sec)
3 expected failures

8 unexpected results:
   FAILED  python-completion-at-point-1
   FAILED  python-completion-at-point-2
   FAILED  python-completion-at-point-native-1
   FAILED  python-completion-at-point-native-2
   FAILED  python-completion-at-point-native-with-eldoc-1
   FAILED  python-completion-at-point-native-with-ffap-1
   FAILED  python-shell-completion-at-point-1
   FAILED  python-shell-completion-at-point-native-1

3 skipped results:
  SKIPPED  python-ffap-module-path-1
  SKIPPED  python-shell-completion-at-point-ipython
  SKIPPED  python-shell-completion-at-point-jedi-completer

bug#68559: [PATCH] Improve Python shell completion

[Eli Zaretskii]

> From: Mattias Engdegård <mattias.engdegard@gmail.com>
> Date: Thu, 15 Feb 2024 15:43:32 +0100
> Cc: Eli Zaretskii <eliz@gnu.org>,
>  kobarity@gmail.com,
>  68559@debbugs.gnu.org
> The recent change on master, 0b9c7148fd, causes several failures in python-tests here (macOS). Log attached.
> 
> It seems to have something to do with completion. Maybe the new code is sensitive to details of Python's command line editor? The standard Python interpreter is unlikely to use GNU readline, for example.

The python-*-completion-at-point-* tests never worked for me, and I
always assumed it was something specific to MS-Windows.  But maybe
not.  There was no change in the tests that fail for me before and
after the recent changes related to Python.

bug#68559: [PATCH] Improve Python shell completion

[Eli Zaretskii]

> Cc: liuhui1610@gmail.com, kobarity@gmail.com, 68559@debbugs.gnu.org
> Date: Thu, 15 Feb 2024 18:37:13 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> 
> > It seems to have something to do with completion. Maybe the new code is sensitive to details of Python's command line editor? The standard Python interpreter is unlikely to use GNU readline, for example.
> 
> The python-*-completion-at-point-* tests never worked for me, and I
> always assumed it was something specific to MS-Windows.  But maybe
> not.  There was no change in the tests that fail for me before and
> after the recent changes related to Python.

And, btw, on this Gnu/Linux system:

  $ uname -a
  Linux (REDACTED) 5.15.0-94-generic #104+11.0trisquel25 SMP Sat Feb 10 06:24:53 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

all the Python tests that failed for you still pass, with the current
master.  So I wonder what is it that causes those failures on your
system.

bug#68559: [PATCH] Improve Python shell completion

[Mattias Engdegård]

15 feb. 2024 kl. 17.48 skrev Eli Zaretskii <eliz@gnu.org>:

> The python-*-completion-at-point-* tests never worked for me, and I
> always assumed it was something specific to MS-Windows.  But maybe
> not.  There was no change in the tests that fail for me before and
> after the recent changes related to Python.

Right, I don't think the Windows port is obscure enough to deserve test failures either; it's an unnecessary impediment. If it cannot easily be fixed then it's better to mark them to be skipped on Windows so that you don't see it every time you run 'make check'.

Completion in the Python shell worked on macOS before the change but now pressing TAB causes the process to hang, so something was clearly broken by the patch.

> all the Python tests that failed for you still pass, with the current
> master.  So I wonder what is it that causes those failures on your
> system.

The standard Python installation on macOS uses a libedit-based readline module and the patch author probably didn't test the changes with that configuration.

bug#68559: [PATCH] Improve Python shell completion

[Basil L. Contovounesios]

[-- Attachment #1: Type: text/plain, Size: 1075 bytes --]

Eli Zaretskii [2024-02-15 18:48 +0200] wrote:

>> Cc: liuhui1610@gmail.com, kobarity@gmail.com, 68559@debbugs.gnu.org
>> Date: Thu, 15 Feb 2024 18:37:13 +0200
>> From: Eli Zaretskii <eliz@gnu.org>
>> 
>> > It seems to have something to do with completion. Maybe the new code is
>> sensitive to details of Python's command line editor? The standard Python
>> interpreter is unlikely to use GNU readline, for example.
>> 
>> The python-*-completion-at-point-* tests never worked for me, and I
>> always assumed it was something specific to MS-Windows.  But maybe
>> not.  There was no change in the tests that fail for me before and
>> after the recent changes related to Python.
>
> And, btw, on this Gnu/Linux system:
>
>   $ uname -a
>   Linux (REDACTED) 5.15.0-94-generic #104+11.0trisquel25 SMP Sat Feb 10 06:24:53 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
>
> all the Python tests that failed for you still pass, with the current
> master.  So I wonder what is it that causes those failures on your
> system.

BTW, I have been seeing an IPython test failure on GNU/Linux:


[-- Attachment #2: ipython.txt.gz --]
[-- Type: application/gzip, Size: 6592 bytes --]

[-- Attachment #3: Type: text/plain, Size: 3322 bytes --]


Any pointers?

Thanks,
-- 
Basil

$ ipython --version
8.21.0
$ python --version
Python 3.12.2

In GNU Emacs 30.0.50 (build 1, x86_64-pc-linux-gnu, X toolkit, cairo
 version 1.18.0, Xaw3d scroll bars) of 2024-02-19 built on tia
Repository revision: 8f260bb93f534b24d9a93d3315804ffe0c1fec4f
Repository branch: master
Windowing system distributor 'The X.Org Foundation', version 11.0.12101011
System Description: Debian GNU/Linux trixie/sid

Configured using:
 'configure 'CFLAGS=-Og -ggdb3' -C --prefix=/home/blc/.local
 --enable-checking=structs --without-native-compilation
 --with-file-notification=yes --with-x-toolkit=lucid --with-x'

Configured features:
ACL CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS GPM GSETTINGS HARFBUZZ JPEG
JSON LCMS2 LIBOTF LIBSELINUX LIBSYSTEMD LIBXML2 M17N_FLT MODULES NOTIFY
INOTIFY PDUMPER PNG RSVG SECCOMP SOUND SQLITE3 THREADS TIFF
TOOLKIT_SCROLL_BARS TREE_SITTER WEBP X11 XAW3D XDBE XIM XINPUT2 XPM
LUCID ZLIB

Important settings:
  value of $LANG: en_IE.UTF-8
  value of $XMODIFIERS: @im=ibus
  locale-coding-system: utf-8-unix

Major mode: Lisp Interaction

Minor modes in effect:
  tooltip-mode: t
  global-eldoc-mode: t
  eldoc-mode: t
  show-paren-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  minibuffer-regexp-mode: t
  line-number-mode: t
  indent-tabs-mode: t
  transient-mark-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message mailcap yank-media puny dired
dired-loaddefs rfc822 mml mml-sec password-cache epa derived epg rfc6068
epg-config gnus-util text-property-search time-date subr-x mm-decode
mm-bodies mm-encode mail-parse rfc2231 mailabbrev gmm-utils mailheader
cl-loaddefs cl-lib sendmail rfc2047 rfc2045 ietf-drums mm-util
mail-prsvr mail-utils rmc iso-transl tooltip cconv eldoc paren electric
uniquify ediff-hook vc-hooks lisp-float-type elisp-mode mwheel
term/x-win x-win term/common-win x-dnd touch-screen tool-bar dnd fontset
image regexp-opt fringe tabulated-list replace newcomment text-mode
lisp-mode prog-mode register page tab-bar menu-bar rfn-eshadow isearch
easymenu timer select scroll-bar mouse jit-lock font-lock syntax
font-core term/tty-colors frame minibuffer nadvice seq simple cl-generic
indonesian philippine cham georgian utf-8-lang misc-lang vietnamese
tibetan thai tai-viet lao korean japanese eucjp-ms cp51932 hebrew greek
romanian slovak czech european ethiopic indian cyrillic chinese
composite emoji-zwj charscript charprop case-table epa-hook
jka-cmpr-hook help abbrev obarray oclosure cl-preloaded button loaddefs
theme-loaddefs faces cus-face macroexp files window text-properties
overlay sha1 md5 base64 format env code-pages mule custom widget keymap
hashtable-print-readable backquote threads dbusbind inotify lcms2
dynamic-setting system-font-setting font-render-setting cairo x-toolkit
xinput2 x multi-tty move-toolbar make-network-process emacs)

Memory information:
((conses 16 39117 8464) (symbols 48 5217 0) (strings 32 13299 1612)
 (string-bytes 1 311978) (vectors 16 9245)
 (vector-slots 8 110970 8413) (floats 8 23 25) (intervals 56 266 0)
 (buffers 984 10))

bug#68559: [PATCH] Improve Python shell completion

[Liu Hui]

On Mon, Feb 19, 2024 at 9:18 PM Basil L. Contovounesios
<basil@contovou.net> wrote:
>
> BTW, I have been seeing an IPython test failure on GNU/Linux:
>
>
> Any pointers?
>
> Thanks,

Hi,

I cannot reproduce the problem with Python 3.12 and IPython 8.21. Can
you check the results with the following steps? Thanks.

1. start Python shell with IPython interpreter, i.e.
   (setq python-shell-interpreter "ipython")
   (setq python-shell-interpreter-args "-i --simple-prompt")
   M-x run-python

2. M-x python-shell-completion-native-turn-off

3. eval code in the Python shell:

    import re
    from IPython.core.completer import provisionalcompleter
    with provisionalcompleter():
        print(list(get_ipython().Completer.completions('re.split("a",
"abc", maxs', 25)))

   expected output: [<Completion start=21 end=25 text='maxsplit='
type='param', signature='?',>]

   __PYTHON_EL_get_completions('re.split("a", "abc", maxs')

   expected output: '[["maxsplit=", 21, 25, "param", ""]]'

bug#68559: [PATCH] Improve Python shell completion

[Basil L. Contovounesios]

[-- Attachment #1: Type: text/plain, Size: 817 bytes --]

Liu Hui [2024-02-20 12:46 +0800] wrote:

> On Mon, Feb 19, 2024 at 9:18 PM Basil L. Contovounesios
> <basil@contovou.net> wrote:
>>
>> BTW, I have been seeing an IPython test failure on GNU/Linux:
>> Any pointers?
>
> I cannot reproduce the problem with Python 3.12 and IPython 8.21. Can
> you check the results with the following steps? Thanks.

Thanks.  These all work as expected in 'emacs -Q', which made me realise
what may be the problem:

- the Emacs test suite runs under HOME=/nonexistent
- but PATH is unchanged
- I have the latest IPython installed locally using pipx:
  $ ls -l $(which ipython)
  lrwxrwxrwx 1 blc blc 53 Feb 20 11:46
    /home/blc/.local/bin/ipython ->
      /home/blc/.local/share/pipx/venvs/ipython/bin/ipython

This is confirmed by the following experiment:


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: prynt.diff --]
[-- Type: text/x-diff, Size: 569 bytes --]

diff --git a/test/lisp/progmodes/python-tests.el b/test/lisp/progmodes/python-tests.el
index af6c199b5bd..3393f93542b 100644
--- a/test/lisp/progmodes/python-tests.el
+++ b/test/lisp/progmodes/python-tests.el
@@ -4883,6 +4883,7 @@ python-shell-completion-at-point-ipython
     (python-tests-with-temp-buffer-with-shell
      ""
      (python-shell-with-shell-buffer
+       (make-temp-file "my-py-out-" nil ".txt" (buffer-string))
        (python-shell-completion-native-turn-off)
        (python-tests--completion-module)
        (python-tests--completion-parameters)

[-- Attachment #3: Type: text/plain, Size: 93 bytes --]


Running 'make TEST_LOAD_EL=no test/python-tests' now results in the
following output file:


[-- Attachment #4: my-py-out-kFh7He.txt --]
[-- Type: text/plain, Size: 442 bytes --]

/home/blc/.local/share/pipx/venvs/ipython/lib/python3.12/site-packages/IPython/paths.py:69: UserWarning: IPython parent '/nonexistent' is not a writable location, using a temp directory.
  warn("IPython parent '{0}' is not a writable location,"
Python 3.12.2 (main, Feb 18 2024, 16:59:56) [GCC 13.2.0]
Type 'copyright', 'credits' or 'license' for more information
IPython 8.21.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: 

[-- Attachment #5: Type: text/plain, Size: 282 bytes --]


In this case list(get_ipython().Completer.completions(...)) is empty.

I'm not sure what the best way to detect or work around this is.
Any ideas?

[ It's a problem shared, to an extent, with Eglot tests, since LSP servers are
  often installed in one's HOME. ]

Thanks,
-- 
Basil

bug#68559: [PATCH] Improve Python shell completion

[Liu Hui]

[-- Attachment #1: Type: text/plain, Size: 1782 bytes --]

On Tue, Feb 20, 2024 at 9:15 PM Basil L. Contovounesios
<basil@contovou.net> wrote:
>
> Liu Hui [2024-02-20 12:46 +0800] wrote:
>
> > On Mon, Feb 19, 2024 at 9:18 PM Basil L. Contovounesios
> > <basil@contovou.net> wrote:
> >>
> >> BTW, I have been seeing an IPython test failure on GNU/Linux:
> >> Any pointers?
> >
> > I cannot reproduce the problem with Python 3.12 and IPython 8.21. Can
> > you check the results with the following steps? Thanks.
>
> Thanks.  These all work as expected in 'emacs -Q', which made me realise
> what may be the problem:
>
> - the Emacs test suite runs under HOME=/nonexistent
> - but PATH is unchanged
> - I have the latest IPython installed locally using pipx:
>   $ ls -l $(which ipython)
>   lrwxrwxrwx 1 blc blc 53 Feb 20 11:46
>     /home/blc/.local/bin/ipython ->
>       /home/blc/.local/share/pipx/venvs/ipython/bin/ipython
>
> This is confirmed by the following experiment:
>
>
> Running 'make TEST_LOAD_EL=no test/python-tests' now results in the
> following output file:
>
>
> In this case list(get_ipython().Completer.completions(...)) is empty.

Thank you for the investigation! I didn't realize the problem is
related to HOME=/nonexistent in 'make test'. Now I can also reproduce
it. In fact, I found that the test failure was caused by Jedi, as it
attempts to write cache to a non-existent directory.

When Jedi is directly used as the completion backend, more test
failures will be triggered:

PYTHONSTARTUP="$(python -m jedi repl)" make TEST_LOAD_EL=no test/python-tests

If the cache directory is writable, all tests will pass:

PYTHONSTARTUP="$(python -m jedi repl)" XDG_CACHE_HOME=~/.cache make
TEST_LOAD_EL=no test/python-tests

The attached patch should fix the problem.

[-- Attachment #2: 0001-Fix-Python-shell-completion-test-failures.patch --]
[-- Type: text/x-patch, Size: 1760 bytes --]

From a6773258cf6fe1acc5520f56ef948ba6975cbe32 Mon Sep 17 00:00:00 2001
From: Liu Hui <liuhui1610@gmail.com>
Date: Wed, 21 Feb 2024 12:40:06 +0800
Subject: [PATCH] Fix Python shell completion test failures

* test/lisp/progmodes/python-tests.el
(python-tests-with-temp-buffer-with-shell): Set XDG_CACHE_HOME
to a temporary directory. (bug#68559)
---
 test/lisp/progmodes/python-tests.el | 17 +++++++++++------
 1 file changed, 11 insertions(+), 6 deletions(-)

diff --git a/test/lisp/progmodes/python-tests.el b/test/lisp/progmodes/python-tests.el
index af6c199b5bd..6454cfcc2b7 100644
--- a/test/lisp/progmodes/python-tests.el
+++ b/test/lisp/progmodes/python-tests.el
@@ -60,12 +60,17 @@ python-tests-with-temp-buffer-with-shell
            (python-shell-completion-native-enable nil))
        (python-mode)
        (unwind-protect
-           (progn
-             (run-python nil t)
-             (insert ,contents)
-             (goto-char (point-min))
-             (python-tests-shell-wait-for-prompt)
-             ,@body)
+           ;; Prevent test failures when Jedi is used as a completion
+           ;; backend, either directly or indirectly (e.g., via
+           ;; IPython).  Jedi needs to store cache, but the
+           ;; "/nonexistent" HOME directory is not writable.
+           (ert-with-temp-directory cache-dir
+             (with-environment-variables (("XDG_CACHE_HOME" cache-dir))
+               (run-python nil t)
+               (insert ,contents)
+               (goto-char (point-min))
+               (python-tests-shell-wait-for-prompt)
+               ,@body))
          (when (python-shell-get-buffer)
            (python-shell-with-shell-buffer
              (let (kill-buffer-hook kill-buffer-query-functions)
-- 
2.25.1

bug#68559: [PATCH] Improve Python shell completion

[Basil L. Contovounesios]

Liu Hui [2024-02-21 18:00 +0800] wrote:

> The attached patch should fix the problem.

Thanks!  The patch fixes the error, but that's because
python-shell-completion-at-point-ipython is now skipped:
in particular, python-shell-readline-completer-delims evaluates to
"\s\t\n`~!@#$%^&*()-=+[{]}\\|;:'\",<>/?" rather than the empty string.

Any idea why that happens?

> -           (progn
> -             (run-python nil t)
> -             (insert ,contents)
> -             (goto-char (point-min))
> -             (python-tests-shell-wait-for-prompt)
> -             ,@body)
> +           ;; Prevent test failures when Jedi is used as a completion
> +           ;; backend, either directly or indirectly (e.g., via
> +           ;; IPython).  Jedi needs to store cache, but the
> +           ;; "/nonexistent" HOME directory is not writable.
> +           (ert-with-temp-directory cache-dir
                                       ^^^^^^^^^
Should this be an uninterned symbol instead?

> +             (with-environment-variables (("XDG_CACHE_HOME" cache-dir))
> +               (run-python nil t)
> +               (insert ,contents)
> +               (goto-char (point-min))
> +               (python-tests-shell-wait-for-prompt)
> +               ,@body))
>           (when (python-shell-get-buffer)
>             (python-shell-with-shell-buffer
>               (let (kill-buffer-hook kill-buffer-query-functions)

Thanks,
-- 
Basil

bug#68559: [PATCH] Improve Python shell completion

[Liu Hui]

[-- Attachment #1: Type: text/plain, Size: 1675 bytes --]

On Wed, Feb 21, 2024 at 10:55 PM Basil L. Contovounesios
<basil@contovou.net> wrote:
>
> Liu Hui [2024-02-21 18:00 +0800] wrote:
>
> > The attached patch should fix the problem.
>
> Thanks!  The patch fixes the error, but that's because
> python-shell-completion-at-point-ipython is now skipped:

Only the native completion part is skipped for the reason below.

> in particular, python-shell-readline-completer-delims evaluates to
> "\s\t\n`~!@#$%^&*()-=+[{]}\\|;:'\",<>/?" rather than the empty string.

"\s\t\n`..." is the delimiter used by rlcompleter, which is the
default completer used by the readline. rlcompleter cannot complete
module names or parameters, so in this case the native completion part
is skipped.

The test is intended to be used with Jedi as the completion backend,
e.g. setting PYTHONSTARTUP="$(python -m jedi repl)", or with a custom
IPython completer defined in the PYTHONSTARTUP file. I have updated the
patch to make the test use Jedi when possible.

> > -           (progn
> > -             (run-python nil t)
> > -             (insert ,contents)
> > -             (goto-char (point-min))
> > -             (python-tests-shell-wait-for-prompt)
> > -             ,@body)
> > +           ;; Prevent test failures when Jedi is used as a completion
> > +           ;; backend, either directly or indirectly (e.g., via
> > +           ;; IPython).  Jedi needs to store cache, but the
> > +           ;; "/nonexistent" HOME directory is not writable.
> > +           (ert-with-temp-directory cache-dir
>                                        ^^^^^^^^^
> Should this be an uninterned symbol instead?

Fixed.

[-- Attachment #2: 0001-Fix-Python-shell-completion-test-failures.patch --]
[-- Type: text/x-patch, Size: 5782 bytes --]

From 8cd85404f5627ffe4b41c19f1d466425eafe31b7 Mon Sep 17 00:00:00 2001
From: Liu Hui <liuhui1610@gmail.com>
Date: Wed, 21 Feb 2024 12:40:06 +0800
Subject: [PATCH] Fix Python shell completion test failures

* test/lisp/progmodes/python-tests.el
(python-tests-with-temp-buffer-with-shell): Set XDG_CACHE_HOME
to a temporary directory.
(python-tests--pythonstartup-file): New function.
(python-shell-completion-at-point-jedi-completer)
(python-shell-completion-at-point-ipython): Use Jedi as the
native completion backend when possible.  (bug#68559)
---
 test/lisp/progmodes/python-tests.el | 87 ++++++++++++++++++-----------
 1 file changed, 53 insertions(+), 34 deletions(-)

diff --git a/test/lisp/progmodes/python-tests.el b/test/lisp/progmodes/python-tests.el
index af6c199b5bd..c5b1ab93144 100644
--- a/test/lisp/progmodes/python-tests.el
+++ b/test/lisp/progmodes/python-tests.el
@@ -55,21 +55,27 @@ python-tests-with-temp-buffer-with-shell
 always located at the beginning of buffer.  Native completion is
 turned off.  Shell buffer will be killed on exit."
   (declare (indent 1) (debug t))
-  `(with-temp-buffer
-     (let ((python-indent-guess-indent-offset nil)
-           (python-shell-completion-native-enable nil))
-       (python-mode)
-       (unwind-protect
-           (progn
-             (run-python nil t)
-             (insert ,contents)
-             (goto-char (point-min))
-             (python-tests-shell-wait-for-prompt)
-             ,@body)
-         (when (python-shell-get-buffer)
-           (python-shell-with-shell-buffer
-             (let (kill-buffer-hook kill-buffer-query-functions)
-               (kill-buffer))))))))
+  (let ((dir (make-symbol "dir")))
+    `(with-temp-buffer
+       (let ((python-indent-guess-indent-offset nil)
+             (python-shell-completion-native-enable nil))
+         (python-mode)
+         (unwind-protect
+             ;; Prevent test failures when Jedi is used as a completion
+             ;; backend, either directly or indirectly (e.g., via
+             ;; IPython).  Jedi needs to store cache, but the
+             ;; "/nonexistent" HOME directory is not writable.
+             (ert-with-temp-directory ,dir
+               (with-environment-variables (("XDG_CACHE_HOME" ,dir))
+                 (run-python nil t)
+                 (insert ,contents)
+                 (goto-char (point-min))
+                 (python-tests-shell-wait-for-prompt)
+                 ,@body))
+           (when (python-shell-get-buffer)
+             (python-shell-with-shell-buffer
+               (let (kill-buffer-hook kill-buffer-query-functions)
+                 (kill-buffer)))))))))
 
 (defmacro python-tests-with-temp-file (contents &rest body)
   "Create a `python-mode' enabled file with CONTENTS.
@@ -4860,17 +4866,28 @@ python-tests--completion-extra-context
   (should (string= "IGNORECASE"
                    (buffer-substring (line-beginning-position) (point)))))
 
+(defun python-tests--pythonstartup-file ()
+  "Return Jedi readline setup file if PYTHONSTARTUP is not set."
+  (or (getenv "PYTHONSTARTUP")
+      (with-temp-buffer
+        (if (eql 0 (call-process python-tests-shell-interpreter
+                                 nil t nil "-m" "jedi" "repl"))
+            (string-trim (buffer-string))
+          ""))))
+
 (ert-deftest python-shell-completion-at-point-jedi-completer ()
   "Check if Python shell completion works when Jedi completer is used."
   (skip-unless (executable-find python-tests-shell-interpreter))
-  (python-tests-with-temp-buffer-with-shell
-   ""
-   (python-shell-with-shell-buffer
-     (python-shell-completion-native-turn-on)
-     (skip-unless (string= python-shell-readline-completer-delims ""))
-     (python-tests--completion-module)
-     (python-tests--completion-parameters)
-     (python-tests--completion-extra-context))))
+  (with-environment-variables
+      (("PYTHONSTARTUP" (python-tests--pythonstartup-file)))
+    (python-tests-with-temp-buffer-with-shell
+     ""
+     (python-shell-with-shell-buffer
+      (python-shell-completion-native-turn-on)
+      (skip-unless (string= python-shell-readline-completer-delims ""))
+      (python-tests--completion-module)
+      (python-tests--completion-parameters)
+      (python-tests--completion-extra-context)))))
 
 (ert-deftest python-shell-completion-at-point-ipython ()
   "Check if Python shell completion works for IPython."
@@ -4880,17 +4897,19 @@ python-shell-completion-at-point-ipython
      (and
       (executable-find python-shell-interpreter)
       (eql (call-process python-shell-interpreter nil nil nil "--version") 0)))
-    (python-tests-with-temp-buffer-with-shell
-     ""
-     (python-shell-with-shell-buffer
-       (python-shell-completion-native-turn-off)
-       (python-tests--completion-module)
-       (python-tests--completion-parameters)
-       (python-shell-completion-native-turn-on)
-       (skip-unless (string= python-shell-readline-completer-delims ""))
-       (python-tests--completion-module)
-       (python-tests--completion-parameters)
-       (python-tests--completion-extra-context)))))
+    (with-environment-variables
+        (("PYTHONSTARTUP" (python-tests--pythonstartup-file)))
+      (python-tests-with-temp-buffer-with-shell
+       ""
+       (python-shell-with-shell-buffer
+         (python-shell-completion-native-turn-off)
+         (python-tests--completion-module)
+         (python-tests--completion-parameters)
+         (python-shell-completion-native-turn-on)
+         (skip-unless (string= python-shell-readline-completer-delims ""))
+         (python-tests--completion-module)
+         (python-tests--completion-parameters)
+         (python-tests--completion-extra-context))))))
 
 \f
 ;;; PDB Track integration
-- 
2.25.1

bug#68559: [PATCH] Improve Python shell completion

[Basil L. Contovounesios]

Liu Hui [2024-02-22 18:31 +0800] wrote:

> On Wed, Feb 21, 2024 at 10:55 PM Basil L. Contovounesios
> <basil@contovou.net> wrote:
>>
>> Liu Hui [2024-02-21 18:00 +0800] wrote:
>>
>> > The attached patch should fix the problem.
>>
>> Thanks!  The patch fixes the error, but that's because
>> python-shell-completion-at-point-ipython is now skipped:
>
> Only the native completion part is skipped for the reason below.
>
>> in particular, python-shell-readline-completer-delims evaluates to
>> "\s\t\n`~!@#$%^&*()-=+[{]}\\|;:'\",<>/?" rather than the empty string.
>
> "\s\t\n`..." is the delimiter used by rlcompleter, which is the
> default completer used by the readline. rlcompleter cannot complete
> module names or parameters, so in this case the native completion part
> is skipped.
>
> The test is intended to be used with Jedi as the completion backend,
> e.g. setting PYTHONSTARTUP="$(python -m jedi repl)", or with a custom
> IPython completer defined in the PYTHONSTARTUP file. I have updated the
> patch to make the test use Jedi when possible.

Thanks, looks fine to me and runs without issue.

The python-shell-completion-at-point-ipython test is still skipped, with
or without specifying PYTHONSTARTUP="$(python -m jedi repl)", but like
you suggest that's not necessarily a problem.

I'm guessing you don't have write access to emacs.git, but have signed
the CA?  If so, and if there are no other comments/objections in a few
days, I'll apply the patch in your name.

Thanks,
-- 
Basil

bug#68559: [PATCH] Improve Python shell completion

[Liu Hui]

On Thu, Feb 22, 2024 at 9:56 PM Basil L. Contovounesios
<basil@contovou.net> wrote:
>
> Liu Hui [2024-02-22 18:31 +0800] wrote:
>
> > On Wed, Feb 21, 2024 at 10:55 PM Basil L. Contovounesios
> > <basil@contovou.net> wrote:
> >>
> >> Liu Hui [2024-02-21 18:00 +0800] wrote:
> >>
> >> > The attached patch should fix the problem.
> >>
> >> Thanks!  The patch fixes the error, but that's because
> >> python-shell-completion-at-point-ipython is now skipped:
> >
> > Only the native completion part is skipped for the reason below.
> >
> >> in particular, python-shell-readline-completer-delims evaluates to
> >> "\s\t\n`~!@#$%^&*()-=+[{]}\\|;:'\",<>/?" rather than the empty string.
> >
> > "\s\t\n`..." is the delimiter used by rlcompleter, which is the
> > default completer used by the readline. rlcompleter cannot complete
> > module names or parameters, so in this case the native completion part
> > is skipped.
> >
> > The test is intended to be used with Jedi as the completion backend,
> > e.g. setting PYTHONSTARTUP="$(python -m jedi repl)", or with a custom
> > IPython completer defined in the PYTHONSTARTUP file. I have updated the
> > patch to make the test use Jedi when possible.
>
> Thanks, looks fine to me and runs without issue.
>
> The python-shell-completion-at-point-ipython test is still skipped, with
> or without specifying PYTHONSTARTUP="$(python -m jedi repl)", but like
> you suggest that's not necessarily a problem.

I have no idea why the test is skipped. You may still check if native
completion works for IPython with following steps:

1. PYTHONSTARTUP="$(python -m jedi repl)" emacs -Q

2. start Python shell with IPython interpreter, i.e.
   (setq python-shell-interpreter "ipython")
   (setq python-shell-interpreter-args "-i --simple-prompt")
   M-x run-python

   There should be text "REPL completion using Jedi xxx" before the
   first prompt in the Python shell buffer, and a message "Shell
   native completion is enabled.".

3. type "import ab"/"open(enc" and press TAB

   expected result: "import abc"/"open(encoding="

> I'm guessing you don't have write access to emacs.git, but have signed
> the CA?  If so, and if there are no other comments/objections in a few
> days, I'll apply the patch in your name.

Yes, I've signed the CA. Thanks.

> Is your GitHub username ilupin by any chance?
>
> I ask because there is one commit in emacs.git from someone with the
> same name as you, but with the email address
> ilupin@users.noreply.github.com (I'm guessing the commit was imported
> from eglot.git).
>
> If that was from you, do you have any objection to me mapping
> ilupin@users.noreply.github.com to liuhui1610@gmail.com in the .mailmap
> file in emacs.git?  Then all commits will show up under a single name
> and email address.

Yes. I have no objection, thanks.

bug#68559: [PATCH] Improve Python shell completion

[Basil L. Contovounesios]

Liu Hui [2024-02-23 21:07 +0800] wrote:

> On Thu, Feb 22, 2024 at 9:56 PM Basil L. Contovounesios
> <basil@contovou.net> wrote:
>>
>> The python-shell-completion-at-point-ipython test is still skipped, with
>> or without specifying PYTHONSTARTUP="$(python -m jedi repl)", but like
>> you suggest that's not necessarily a problem.
>
> I have no idea why the test is skipped. You may still check if native
> completion works for IPython with following steps:
>
> 1. PYTHONSTARTUP="$(python -m jedi repl)" emacs -Q

Oops:

$ PYTHONSTARTUP="$(python -m jedi repl)" ./src/emacs -Q
/home/blc/.pyenv/versions/3.12.2/bin/python: No module named jedi

Once I 'pip install jedi' the rest works as expected, and
python-shell-completion-at-point-ipython is no longer skipped.

>> I'm guessing you don't have write access to emacs.git, but have signed
>> the CA?  If so, and if there are no other comments/objections in a few
>> days, I'll apply the patch in your name.
>
> Yes, I've signed the CA. Thanks.

Done:

Fix Python shell completion test failures
8a2d013be37 2024-02-28 15:25:56 +0100
https://git.sv.gnu.org/cgit/emacs.git/commit/?id=8a2d013be37

>> If that was from you, do you have any objection to me mapping
>> ilupin@users.noreply.github.com to liuhui1610@gmail.com in the .mailmap
>> file in emacs.git?  Then all commits will show up under a single name
>> and email address.
>
> Yes. I have no objection, thanks.

Done:

; * .mailmap: Fix GitHub address (bug#68559#170).
1ddd9c8e29f 2024-02-28 15:30:41 +0100
https://git.sv.gnu.org/cgit/emacs.git/commit/?id=1ddd9c8e29f

Thanks,
-- 
Basil

bug#68559: [PATCH] Improve Python shell completion

[Liu Hui]

On Fri, Feb 16, 2024 at 12:37 AM Eli Zaretskii <eliz@gnu.org> wrote:
>
> > From: Mattias Engdegård <mattias.engdegard@gmail.com>
> > Date: Thu, 15 Feb 2024 15:43:32 +0100
> > Cc: Eli Zaretskii <eliz@gnu.org>,
> >  kobarity@gmail.com,
> >  68559@debbugs.gnu.org
> > The recent change on master, 0b9c7148fd, causes several failures in python-tests here (macOS). Log attached.
> >
> > It seems to have something to do with completion. Maybe the new code is sensitive to details of Python's command line editor? The standard Python interpreter is unlikely to use GNU readline, for example.
>
> The python-*-completion-at-point-* tests never worked for me, and I
> always assumed it was something specific to MS-Windows.  But maybe
> not.  There was no change in the tests that fail for me before and
> after the recent changes related to Python.

The Python shell completion relies on the readline module, which is
not available for Python on MS-Windows. According to the instruction
in python.el:

;; readline based shell (it's known to work with PyPy).  If your
;; Python installation lacks readline (like CPython for Windows),
;; installing pyreadline (URL `https://ipython.org/pyreadline.html')
;; should suffice.  To troubleshoot why you are not getting any
;; completions, you can try the following in your Python shell:

;; >>> import readline, rlcompleter

;; If you see an error, then you need to either install pyreadline or
;; setup custom code that avoids that dependency.

It may be necessary to install pyreadline (for Python 2.7) or
pyreadline3 (for Python 3).

bug#68559: [PATCH] Improve Python shell completion

[Eli Zaretskii]

> From: Liu Hui <liuhui1610@gmail.com>
> Date: Fri, 16 Feb 2024 12:06:38 +0800
> Cc: Mattias Engdegård <mattias.engdegard@gmail.com>, 
> 	kobarity@gmail.com, 68559@debbugs.gnu.org
> 
> On Fri, Feb 16, 2024 at 12:37 AM Eli Zaretskii <eliz@gnu.org> wrote:
> >
> > The python-*-completion-at-point-* tests never worked for me, and I
> > always assumed it was something specific to MS-Windows.  But maybe
> > not.  There was no change in the tests that fail for me before and
> > after the recent changes related to Python.
> 
> The Python shell completion relies on the readline module, which is
> not available for Python on MS-Windows. According to the instruction
> in python.el:
> 
> ;; readline based shell (it's known to work with PyPy).  If your
> ;; Python installation lacks readline (like CPython for Windows),
> ;; installing pyreadline (URL `https://ipython.org/pyreadline.html')
> ;; should suffice.  To troubleshoot why you are not getting any
> ;; completions, you can try the following in your Python shell:
> 
> ;; >>> import readline, rlcompleter
> 
> ;; If you see an error, then you need to either install pyreadline or
> ;; setup custom code that avoids that dependency.

I don't know if I have CPython, but the above does show an error
message.

> It may be necessary to install pyreadline (for Python 2.7) or
> pyreadline3 (for Python 3).

I will see if I can do that, thanks.

Regardless, patches to the test suite to skip the tests which rely on
those modules, if they aren't installed, will be welcome.

bug#68559: [PATCH] Improve Python shell completion

[Eli Zaretskii]

> Cc: mattias.engdegard@gmail.com, kobarity@gmail.com, 68559@debbugs.gnu.org
> Date: Fri, 16 Feb 2024 09:41:37 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> 
> > It may be necessary to install pyreadline (for Python 2.7) or
> > pyreadline3 (for Python 3).
> 
> I will see if I can do that, thanks.

I've now installed pyreadline3, and I can confirm that the completion
tests which were failing before now pass.

Thanks.

> Regardless, patches to the test suite to skip the tests which rely on
> those modules, if they aren't installed, will be welcome.

This is still true.

bug#68559: [PATCH] Improve Python shell completion

[Liu Hui]

On Thu, Feb 15, 2024 at 10:43 PM Mattias Engdegård
<mattias.engdegard@gmail.com> wrote:
>
> The recent change on master, 0b9c7148fd, causes several failures in python-tests here (macOS). Log attached.
>
> It seems to have something to do with completion. Maybe the new code is sensitive to details of Python's command line editor? The standard Python interpreter is unlikely to use GNU readline, for example.

Hi,

According to the log, the problem may be related to the json library,
which is introduced by the patch to convert the completion results
from Python to elisp. What is the version of Python on macOS? AFAIK
the json library is added to the Standard library since version 2.6.
If the library is not available, could you setup a Python virtual
environment containing the library and run the python tests in it?
Thanks!

bug#68559: [PATCH] Improve Python shell completion

[kobarity]

Liu Hui wrote:
> On Thu, Feb 15, 2024 at 10:43 PM Mattias Engdegård
> <mattias.engdegard@gmail.com> wrote:
> >
> > The recent change on master, 0b9c7148fd, causes several failures in python-tests here (macOS). Log attached.
> >
> > It seems to have something to do with completion. Maybe the new code is sensitive to details of Python's command line editor? The standard Python interpreter is unlikely to use GNU readline, for example.
> 
> Hi,
> 
> According to the log, the problem may be related to the json library,
> which is introduced by the patch to convert the completion results
> from Python to elisp. What is the version of Python on macOS? AFAIK
> the json library is added to the Standard library since version 2.6.
> If the library is not available, could you setup a Python virtual
> environment containing the library and run the python tests in it?
> Thanks!

This is caused by the fact that the input is echoed back on MacOS
Python.  This is described in etc/PROBLEMS:

*** In Inferior Python mode, input is echoed and native completion doesn't work.

The following log shows that it tries to parse the echoed back string
as a JSON string.

Test python-completion-at-point-1 backtrace:
  json-parse-string("__PYTHON_EL_eval_file(\"/var/folders/qy/zstv16390
  python--parse-json-array("__PYTHON_EL_eval_file(\"/var/folders/qy/zs

I think the echoed back string were ignored before the patch is
applied.  However, after applying the patch, the returned string is
now parsed as JSON, which I believe is the reason for the error.

So one workaround would be to remove the echoed back string before
parsing as JSON.

bug#68559: [PATCH] Improve Python shell completion

[Mattias Engdegård]

16 feb. 2024 kl. 10.34 skrev kobarity <kobarity@gmail.com>:

> This is caused by the fact that the input is echoed back on MacOS
> Python.

Right, that bug needs to be fixed as well, but the echo problem existed prior to the change that broke the tests.

An alternative might be to disable the tty echo altogether. If I do it right after process creation then it has no effect; presumably Python or its readline module turns on echo just a bit later on. Anyway, running

  import tty
  tty.setraw(0)

in the python shell seems to put it right, with working completion and the annoying echo gone.

> So one workaround would be to remove the echoed back string before
> parsing as JSON.

Yes, either that or turning off echo in the tty.

bug#68559: [PATCH] Improve Python shell completion

[kobarity]

[-- Attachment #1: Type: text/plain, Size: 1532 bytes --]

Mattias Engdegård wrote:
> 
> 16 feb. 2024 kl. 10.34 skrev kobarity <kobarity@gmail.com>:
> 
> > This is caused by the fact that the input is echoed back on MacOS
> > Python.
> 
> Right, that bug needs to be fixed as well, but the echo problem existed prior to the change that broke the tests.
> 
> An alternative might be to disable the tty echo altogether. If I do it right after process creation then it has no effect; presumably Python or its readline module turns on echo just a bit later on. Anyway, running
> 
>   import tty
>   tty.setraw(0)
> 
> in the python shell seems to put it right, with working completion and the annoying echo gone.
> 
> > So one workaround would be to remove the echoed back string before
> > parsing as JSON.
> 
> Yes, either that or turning off echo in the tty.

I made prototype patches for each method.  I don't use Mac so it would
be helpful if you could try these.

0001-Remove-echoed-back-string-in-python-shell-completion.patch
extracts only the last line to exclude echoed back strings.

0001-Set-tty-mode-to-raw-when-setting-up-Inferior-Python.patch sets
the Inferior Python tty to raw mode.  python-ffap-module-path-1 will
no longer need to be skipped on Mac.  If it is safe to set tty to raw
mode on all UNIX based systems, I prefer this method.

By the way, is it necessary to send
`python-shell-completion-setup-code' for every completion in
`python-shell-completion-get-completions'?  To me it seems sufficient
to send it once at initialization.

[-- Attachment #2: 0001-Remove-echoed-back-string-in-python-shell-completion.patch --]
[-- Type: application/octet-stream, Size: 1480 bytes --]

From 601419eef799d2c68e3789222f5ebe2a8c31af97 Mon Sep 17 00:00:00 2001
From: kobarity <kobarity@gmail.com>
Date: Fri, 16 Feb 2024 22:48:19 +0900
Subject: [PATCH] Remove echoed back string in
 python-shell-completion-get-completions

* lisp/progmodes/python.el (python-shell-completion-get-completions):
Remove echoed back string. (Bug#68559)
---
 lisp/progmodes/python.el | 12 +++++++-----
 1 file changed, 7 insertions(+), 5 deletions(-)

diff --git a/lisp/progmodes/python.el b/lisp/progmodes/python.el
index b7e43f3fc68..a842a498113 100644
--- a/lisp/progmodes/python.el
+++ b/lisp/progmodes/python.el
@@ -4624,11 +4624,13 @@ python-shell-completion-get-completions
   "Get completions of INPUT using PROCESS."
   (with-current-buffer (process-buffer process)
     (python--parse-json-array
-     (python-shell-send-string-no-output
-      (format "%s\nprint(__PYTHON_EL_get_completions(%s))"
-              python-shell-completion-setup-code
-              (python-shell--encode-string input))
-      process))))
+     (car (last (split-string
+                 (python-shell-send-string-no-output
+                  (format "%s\nprint(__PYTHON_EL_get_completions(%s))"
+                          python-shell-completion-setup-code
+                          (python-shell--encode-string input))
+                  process)
+                 "[\n\r]+" t))))))
 
 (defun python-shell--get-multiline-input ()
   "Return lines at a multi-line input in Python shell."
-- 
2.34.1


[-- Attachment #3: 0001-Set-tty-mode-to-raw-when-setting-up-Inferior-Python.patch --]
[-- Type: application/octet-stream, Size: 2531 bytes --]

From 31738d481333bf6c258a694cf57bce944bce7778 Mon Sep 17 00:00:00 2001
From: kobarity <kobarity@gmail.com>
Date: Fri, 16 Feb 2024 22:52:06 +0900
Subject: [PATCH] Set tty mode to raw when setting up Inferior Python

* lisp/progmodes/python.el (python-shell-setup-code): New constant.
(python-shell-comint-watch-for-first-prompt-output-filter): Send
`python-shell-setup-code' to the Inferior Python process.
* test/lisp/progmodes/python-tests.el (python-ffap-module-path-1):
Eliminate skipping on Mac. (Bug#68559)
---
 lisp/progmodes/python.el            | 11 +++++++++++
 test/lisp/progmodes/python-tests.el |  5 -----
 2 files changed, 11 insertions(+), 5 deletions(-)

diff --git a/lisp/progmodes/python.el b/lisp/progmodes/python.el
index b7e43f3fc68..5501926e69d 100644
--- a/lisp/progmodes/python.el
+++ b/lisp/progmodes/python.el
@@ -3521,6 +3521,16 @@ python-shell-first-prompt-hook
   :version "25.1"
   :type 'hook)
 
+(defconst python-shell-setup-code
+  "\
+try:
+    import tty
+except ImportError:
+    pass
+else:
+    tty.setraw(0)"
+  "Code used to setup the inferior Python processes.")
+
 (defconst python-shell-eval-setup-code
   "\
 def __PYTHON_EL_eval(source, filename):
@@ -3586,6 +3596,7 @@ python-shell-comint-watch-for-first-prompt-output-filter
                       (format "exec(%s)\n" (python-shell--encode-string string))))))
           ;; Bootstrap: the normal definition of `python-shell-send-string'
           ;; depends on the Python code sent here.
+          (python-shell-send-string-no-output python-shell-setup-code)
           (python-shell-send-string-no-output python-shell-eval-setup-code)
           (python-shell-send-string-no-output python-shell-eval-file-setup-code))
         (with-current-buffer (current-buffer)
diff --git a/test/lisp/progmodes/python-tests.el b/test/lisp/progmodes/python-tests.el
index af6c199b5bd..6c6cd9eee2b 100644
--- a/test/lisp/progmodes/python-tests.el
+++ b/test/lisp/progmodes/python-tests.el
@@ -5037,11 +5037,6 @@ python-completion-at-point-native-with-eldoc-1
 
 (ert-deftest python-ffap-module-path-1 ()
   (skip-unless (executable-find python-tests-shell-interpreter))
-  ;; Skip the test on macOS, since the standard Python installation uses
-  ;; libedit rather than readline which confuses the running of an inferior
-  ;; interpreter in this case (see bug#59477 and bug#25753).
-  (skip-when (eq system-type 'darwin))
-  (trace-function 'python-shell-output-filter)
   (python-tests-with-temp-buffer-with-shell
    "
 import abc
-- 
2.34.1

bug#68559: [PATCH] Improve Python shell completion

[Eli Zaretskii]

> Date: Sat, 17 Feb 2024 00:24:29 +0900
> From: kobarity <kobarity@gmail.com>
> Cc: Liu Hui <liuhui1610@gmail.com>,
> 	Eli Zaretskii <eliz@gnu.org>,
> 	68559@debbugs.gnu.org
> > Right, that bug needs to be fixed as well, but the echo problem existed prior to the change that broke the tests.
> > 
> > An alternative might be to disable the tty echo altogether. If I do it right after process creation then it has no effect; presumably Python or its readline module turns on echo just a bit later on. Anyway, running
> > 
> >   import tty
> >   tty.setraw(0)
> > 
> > in the python shell seems to put it right, with working completion and the annoying echo gone.
> > 
> > > So one workaround would be to remove the echoed back string before
> > > parsing as JSON.
> > 
> > Yes, either that or turning off echo in the tty.
> 
> I made prototype patches for each method.  I don't use Mac so it would
> be helpful if you could try these.
> 
> 0001-Remove-echoed-back-string-in-python-shell-completion.patch
> extracts only the last line to exclude echoed back strings.
> 
> 0001-Set-tty-mode-to-raw-when-setting-up-Inferior-Python.patch sets
> the Inferior Python tty to raw mode.  python-ffap-module-path-1 will
> no longer need to be skipped on Mac.  If it is safe to set tty to raw
> mode on all UNIX based systems, I prefer this method.

Will this work on MS-Windows as well?  If you are unsure, would you
please tell me how to test whether this works on Windows, so I could
collect the information for you?

Thanks.

bug#68559: [PATCH] Improve Python shell completion

[Mattias Engdegård]

16 feb. 2024 kl. 16.24 skrev kobarity <kobarity@gmail.com>:

> I made prototype patches for each method.  I don't use Mac so it would
> be helpful if you could try these.

Nice, thank you! I can confirm that they both appear to work, at least in the sense that the python-tests pass (except for the ones skipped intentionally), and the python shell behaves reasonable.

Only the set-tty-mode patch eliminates echo in the interactive python shell; in that sense it's preferable.

Both produce the very annoying warning

  Warning (python): Your ‘python-shell-interpreter’ doesn’t seem to support readline, yet ‘python-shell-completion-native-enable’ was t and "python3" is not part of the ‘python-shell-completion-native-disabled-interpreters’ list.  Native completions have been disabled locally. Consider installing the python package "readline".

which is not even correct since the standard Python does have a working readline module, even if it uses libedit.

> 0001-Set-tty-mode-to-raw-when-setting-up-Inferior-Python.patch sets
> the Inferior Python tty to raw mode.  python-ffap-module-path-1 will
> no longer need to be skipped on Mac.

Right, that test runs and passes.

>  If it is safe to set tty to raw
> mode on all UNIX based systems, I prefer this method.

Same here. I see no reason why it wouldn't be safe, either.

> By the way, is it necessary to send
> `python-shell-completion-setup-code' for every completion in
> `python-shell-completion-get-completions'?  To me it seems sufficient
> to send it once at initialization.

Indeed, it does seem a bit extravagant.

bug#68559: [PATCH] Improve Python shell completion

[kobarity]

Eli Zaretskii wrote:
> > 0001-Remove-echoed-back-string-in-python-shell-completion.patch
> > extracts only the last line to exclude echoed back strings.
> > 
> > 0001-Set-tty-mode-to-raw-when-setting-up-Inferior-Python.patch sets
> > the Inferior Python tty to raw mode.  python-ffap-module-path-1 will
> > no longer need to be skipped on Mac.  If it is safe to set tty to raw
> > mode on all UNIX based systems, I prefer this method.
> 
> Will this work on MS-Windows as well?  If you are unsure, would you
> please tell me how to test whether this works on Windows, so I could
> collect the information for you?

It does not affect MS-Windows because it sets only when the tty
library can be imported.  Although the tty library exists in
MS-Windows, importing the tty library will result in an error because
there is no underlying termios library.

Mattias Engdegård wrote:
> 
> 16 feb. 2024 kl. 16.24 skrev kobarity <kobarity@gmail.com>:
> 
> > I made prototype patches for each method.  I don't use Mac so it would
> > be helpful if you could try these.
> 
> Nice, thank you! I can confirm that they both appear to work, at least in the sense that the python-tests pass (except for the ones skipped intentionally), and the python shell behaves reasonable.
> 
> Only the set-tty-mode patch eliminates echo in the interactive python shell; in that sense it's preferable.

Thank you for testing my patches.

> Both produce the very annoying warning
> 
>   Warning (python): Your ‘python-shell-interpreter’ doesn’t seem to support readline, yet ‘python-shell-completion-native-enable’ was t and "python3" is not part of the ‘python-shell-completion-native-disabled-interpreters’ list.  Native completions have been disabled locally. Consider installing the python package "readline".
> 
> which is not even correct since the standard Python does have a working readline module, even if it uses libedit.

You are right. Maybe the package name should be fixed to "gnureadline"
as described in etc/PROBLEMS.

On Mac, it might be better to set the default value of
`python-shell-completion-native-enable' to nil.

> > 0001-Set-tty-mode-to-raw-when-setting-up-Inferior-Python.patch sets
> > the Inferior Python tty to raw mode.  python-ffap-module-path-1 will
> > no longer need to be skipped on Mac.
> 
> Right, that test runs and passes.
> 
> >  If it is safe to set tty to raw
> > mode on all UNIX based systems, I prefer this method.
> 
> Same here. I see no reason why it wouldn't be safe, either.

Thanks.

> > By the way, is it necessary to send
> > `python-shell-completion-setup-code' for every completion in
> > `python-shell-completion-get-completions'?  To me it seems sufficient
> > to send it once at initialization.
> 
> Indeed, it does seem a bit extravagant.

This would be one of the items for future improvement.

Liu Hui wrote:
> 
> kobarity <kobarity@gmail.com> writes:
> 
> > 0001-Set-tty-mode-to-raw-when-setting-up-Inferior-Python.patch sets
> > the Inferior Python tty to raw mode.  python-ffap-module-path-1 will
> > no longer need to be skipped on Mac.  If it is safe to set tty to raw
> > mode on all UNIX based systems, I prefer this method.
> 
> How about the following change, which only affects libedit-based
> readline? It may enable native completion on mac, but I cannot test
> it.
> 
> diff --git a/lisp/progmodes/python.el b/lisp/progmodes/python.el
> index b7e43f3fc68..f59bc19367b 100644
> --- a/lisp/progmodes/python.el
> +++ b/lisp/progmodes/python.el
> @@ -4286,6 +4286,9 @@ (defcustom python-shell-completion-setup-code
>              except:
>                  pass
>          else:
> +            if readline.__doc__ and 'libedit' in readline.__doc__:
> +                import tty
> +                tty.setraw(0)
>              # Try to reuse current completer.
>              completer = readline.get_completer()
>              if not completer:
> @@ -4471,8 +4474,8 @@ (defun python-shell-completion-native-setup ()
>                  instance.rlcomplete = new_completer
> 
>          if readline.__doc__ and 'libedit' in readline.__doc__:
> -            raise Exception('''libedit based readline is known not to work,
> -      see etc/PROBLEMS under \"In Inferior Python mode, input is echoed\".''')
> +            import tty
> +            tty.setraw(0)
>              readline.parse_and_bind('bind ^I rl_complete')
>          else:
>              readline.parse_and_bind('tab: complete')

Disabling echo back may not be sufficient to enable native completions
on Mac. I have not tried raw mode, but have tried
readline.parse_and_bind('setty -echo') and
readline.parse_and_bind('edit on').  Native completions could be
enabled, but it was unstable.

I have no objection if Mac users check the above patch and if it is
OK.

bug#68559: [PATCH] Improve Python shell completion

[Mattias Engdegård]

17 feb. 2024 kl. 14.33 skrev kobarity <kobarity@gmail.com>:

> On Mac, it might be better to set the default value of
> `python-shell-completion-native-enable' to nil.

Not sure why the inferior Python process doesn't act on a TAB being sent to it. Is the tty somehow in a state that disables readline/libedit? Something that Emacs does when setting up the pty?

Of course from a software engineering point of view, it's silly to send what essentially are edit keystrokes to Python and then screen-scrape the output. A proper interaction protocol would be the way to go, and would work equally well on any platform including Windows.

> Disabling echo back may not be sufficient to enable native completions
> on Mac. I have not tried raw mode, but have tried
> readline.parse_and_bind('setty -echo') and
> readline.parse_and_bind('edit on').  Native completions could be
> enabled, but it was unstable.
> 
> I have no objection if Mac users check the above patch and if it is
> OK.

Afraid it wasn't.

Thanks for your patches. I suggest we apply your set-tty-raw patch on master now since it cures the test failures without breaking anything else (on Mac; I'm assuming no regression elsewhere).

Would you like me to do that for you?

bug#68559: [PATCH] Improve Python shell completion

[kobarity]

Mattias Engdegård wrote:
> 17 feb. 2024 kl. 14.33 skrev kobarity <kobarity@gmail.com>:
> > On Mac, it might be better to set the default value of
> > `python-shell-completion-native-enable' to nil.
> Not sure why the inferior Python process doesn't act on a TAB being sent to it. Is the tty somehow in a state that disables readline/libedit? Something that Emacs does when setting up the pty?

On Mac, there are two independent problems.  One problem, which
existed before Liu's patch was applied, is that native completions
cannot be enabled.  The other problem is that the test now fails after
applying Liu's patch.  Suppressing echo back solves the latter, but
not the former.  Because of the former, leaving
`python-shell-completion-native-enable' at the default setting of t
will result in the warning.

There are two completions methods, native and non-native.  If native
completions cannot be used, it falls back to non-native completions.
Native completions is triggered by sending TAB character, while
non-native completions is triggered by sending and evaluating
__PYTHON_EL_get_completions().

As far as I have tried, native completions cannot be enabled for
libedit-based readline package, even on Linux.

So, there are two ways for Mac users to avoid the warning.  One is to
use gnureadline instead of libedit, and the other is to give up native
completions and set `python-shell-completion-native-enable' to nil.
This remains correct even with echo back disabled.

> Of course from a software engineering point of view, it's silly to send what essentially are edit keystrokes to Python and then screen-scrape the output. A proper interaction protocol would be the way to go, and would work equally well on any platform including Windows.

I think the protocol between python.el and inferior Python process is
already platform independent.  Protocol violations are echo back.

> Thanks for your patches. I suggest we apply your set-tty-raw patch on master now since it cures the test failures without breaking anything else (on Mac; I'm assuming no regression elsewhere).
> 
> Would you like me to do that for you?

Yes, please.  Thank you.  I have no commit right.

bug#68559: [PATCH] Improve Python shell completion

[Mattias Engdegård]

[-- Attachment #1: Type: text/plain, Size: 958 bytes --]

21 feb. 2024 kl. 14.13 skrev kobarity <kobarity@gmail.com>:

> As far as I have tried, native completions cannot be enabled for
> libedit-based readline package, even on Linux.

Did you find out why? Python responds nicely to TAB if run from a terminal, so what is it that we do in Emacs that prevents it from doing so? A TTY setting, or an environment variable (the TERM=dumb)?

> So, there are two ways for Mac users to avoid the warning.  One is to
> use gnureadline instead of libedit, and the other is to give up native
> completions and set `python-shell-completion-native-enable' to nil.

Preferably neither. Emacs should adapt to the system environment, and in particular not warn about the default environment. A warning is an indication of a possible risk or other problem, and there is none here.

At most, python-mode could show a calm notice on the echo line but even that is a stretch. What do you think about this rough patch?


[-- Attachment #2: nowarn.diff --]
[-- Type: application/octet-stream, Size: 1402 bytes --]

diff --git a/lisp/progmodes/python.el b/lisp/progmodes/python.el
index 5501926e69d..bedc61408ef 100644
--- a/lisp/progmodes/python.el
+++ b/lisp/progmodes/python.el
@@ -4536,18 +4536,11 @@ python-shell-completion-native-turn-on-maybe
        ((python-shell-completion-native-setup)
         (when msg
           (message "Shell native completion is enabled.")))
-       (t (lwarn
-           '(python python-shell-completion-native-turn-on-maybe)
-           :warning
-           (concat
-            "Your `python-shell-interpreter' doesn't seem to "
-            "support readline, yet `python-shell-completion-native-enable' "
-            (format "was t and %S is not part of the "
-                    (file-name-nondirectory python-shell-interpreter))
-            "`python-shell-completion-native-disabled-interpreters' "
-            "list.  Native completions have been disabled locally. "
-            "Consider installing the python package \"readline\". "))
-          (python-shell-completion-native-turn-off msg))))))
+       (t
+        (when msg
+          (message (concat "Python does not use GNU readline;"
+                           " no completion in multi-line commands.")))
+        (python-shell-completion-native-turn-off nil))))))
 
 (defun python-shell-completion-native-turn-on-maybe-with-msg ()
   "Like `python-shell-completion-native-turn-on-maybe' but force messages."

[-- Attachment #3: Type: text/plain, Size: 1007 bytes --]



> I think the protocol between python.el and inferior Python process is
> already platform independent.  Protocol violations are echo back.

No, I meant a protocol that allows Emacs to act as a first-class Python front-end, not simulate a terminal, send keystrokes, use heuristics to determine what in the output stream is a prompt, REPL value, error, completion etc.

For example, it's a bit silly to input multi-line code in the REPL as a sequence of individual single-line commands, when we actually are inside a text editor that can edit multi-line Python code without a problem.

(I'm not suggesting that you or anybody in particular should do this; just that it's feasible. It would clearly be quite some work!)

>> Thanks for your patches. I suggest we apply your set-tty-raw patch on master now since it cures the test failures without breaking anything else (on Mac; I'm assuming no regression elsewhere).
>> 
>> Would you like me to do that for you?
> 
> Yes, please.

Done!

bug#68559: [PATCH] Improve Python shell completion

[kobarity]

Mattias Engdegård wrote:
> 21 feb. 2024 kl. 14.13 skrev kobarity <kobarity@gmail.com>:
> > As far as I have tried, native completions cannot be enabled for
> > libedit-based readline package, even on Linux.
> Did you find out why? Python responds nicely to TAB if run from a terminal, so what is it that we do in Emacs that prevents it from doing so? A TTY setting, or an environment variable (the TERM=dumb)?

No, not exactly, but I don't think it is related to terminal settings.
python.el does not parse the completion candidates shown on the
terminal.  Instead, it expects the candidates in a particular format,
which I call the protocol between python.el and inferior Python
process.

__PYTHON_EL_native_completion_setup() defined in
`python-shell-completion-native-setup' sets up the completer to do so.
We can test it by removing empty lines and pasting into Python REPL
outside Emacs.  An example of typing "ra" and TAB using readline would
look like this.

>>> raraise
range

0__dummy_completion__  1__dummy_completion__
>>> ra

However, when I use libedit-based readline and remove the raise line
in __PYTHON_EL_native_completion_setup(), it behaves like this.

>>> raraise
range

Please note that there is no dummy completions, and the new prompt is
not shown.  When I hit TAB again, it would be like this.

>>> raraise
range
      raise
range

0__dummy_completion__ 0__dummy_completion__ 0__dummy_completion__ 1__dummy_completion__
>>> ra

I think this difference in behavior is the reason why Native
completions does not work with libedit.

> > So, there are two ways for Mac users to avoid the warning.  One is to
> > use gnureadline instead of libedit, and the other is to give up native
> > completions and set `python-shell-completion-native-enable' to nil.
> 
> Preferably neither. Emacs should adapt to the system environment, and in particular not warn about the default environment. A warning is an indication of a possible risk or other problem, and there is none here.
> 
> At most, python-mode could show a calm notice on the echo line but even that is a stretch. What do you think about this rough patch?

Personally, I think this patch would be fine.

I think we can also improve Non-native completions.  Current
implementation sends the definition of __PYTHON_EL_get_completions()
every time.  However, sending it once during initialization should be
sufficient.  Worse, if the number of characters sent exceeds
`comint-max-line-length', it will be sent via file.  This is happening
in your environment.  Here is the log you presented.

Test python-completion-at-point-1 backtrace:
  json-parse-string("__PYTHON_EL_eval_file(\"/var/folders/qy/zstv16390

Thanks to the echo back, we can see __PYTHON_EL_eval_file() was used.

> > I think the protocol between python.el and inferior Python process is
> > already platform independent.  Protocol violations are echo back.
> 
> No, I meant a protocol that allows Emacs to act as a first-class Python front-end, not simulate a terminal, send keystrokes, use heuristics to determine what in the output stream is a prompt, REPL value, error, completion etc.
> 
> For example, it's a bit silly to input multi-line code in the REPL as a sequence of individual single-line commands, when we actually are inside a text editor that can edit multi-line Python code without a problem.
> 
> (I'm not suggesting that you or anybody in particular should do this; just that it's feasible. It would clearly be quite some work!)

I am not opposed to this approach, but as you wrote, it is very
different from the current inferior Python implementation.
Jupyter-emacs's approach may be similar to some extent.  It uses zmq
as the communication channel.

https://github.com/emacs-jupyter/jupyter

> >> Thanks for your patches. I suggest we apply your set-tty-raw patch on master now since it cures the test failures without breaking anything else (on Mac; I'm assuming no regression elsewhere).
> >> Would you like me to do that for you?
> > Yes, please.
> Done!

Thank you!

bug#68559: [PATCH] Improve Python shell completion

[Mattias Engdegård]

22 feb. 2024 kl. 17.15 skrev kobarity <kobarity@gmail.com>:

>> Did you find out why? Python responds nicely to TAB if run from a terminal, so what is it that we do in Emacs that prevents it from doing so? A TTY setting, or an environment variable (the TERM=dumb)?
> 
> No, not exactly, but I don't think it is related to terminal settings.

Well it must be something. I'll see if I can make sense of it.

> I think this difference in behavior is the reason why Native
> completions does not work with libedit.

Thanks for explaining.

> Personally, I think this patch would be fine.

Thank you, now pushed to master.

> I think we can also improve Non-native completions.  Current
> implementation sends the definition of __PYTHON_EL_get_completions()
> every time.  However, sending it once during initialization should be
> sufficient.  Worse, if the number of characters sent exceeds
> `comint-max-line-length', it will be sent via file.  This is happening
> in your environment.  Here is the log you presented.
> 
> Test python-completion-at-point-1 backtrace:
>  json-parse-string("__PYTHON_EL_eval_file(\"/var/folders/qy/zstv16390
> 
> Thanks to the echo back, we can see __PYTHON_EL_eval_file() was used.

Right. (The need for a file is just an artefact of Comint limitations, isn't it?)

> I am not opposed to this approach, but as you wrote, it is very
> different from the current inferior Python implementation.
> Jupyter-emacs's approach may be similar to some extent.  It uses zmq
> as the communication channel.

Thank you, yes, in a sense it's a (small) step towards a more notebook-like interaction model, but Comint already has some of that. This problem is not unique to Python: other REPLs have to solve the problem of multi-line input as well, and it would serve the user if they all worked in the same way (as far as the language differences allow).

In Python's case, it would liberate us from the constraints of the standard terminal REPL.

bug#68559: [PATCH] Improve Python shell completion

[kobarity]

Hi Liu,

I noticed that with Non-native completions, Jedi completion is not
enabled even if PYTHONSTARTUP is set and "REPL completion using Jedi
0.18.2" is shown in inferior Python. Is this expected behavior?  Jedi
completion is enabled regardless of PYTHONSTARTUP setting when I use
IPython with Non-native completions.

Mattias Engdegård wrote:
> > I think we can also improve Non-native completions.  Current
> > implementation sends the definition of __PYTHON_EL_get_completions()
> > every time.  However, sending it once during initialization should be
> > sufficient.  Worse, if the number of characters sent exceeds
> > `comint-max-line-length', it will be sent via file.  This is happening
> > in your environment.  Here is the log you presented.
> > 
> > Test python-completion-at-point-1 backtrace:
> >  json-parse-string("__PYTHON_EL_eval_file(\"/var/folders/qy/zstv16390
> > 
> > Thanks to the echo back, we can see __PYTHON_EL_eval_file() was used.
> 
> Right. (The need for a file is just an artefact of Comint limitations, isn't it?)

Yes, I think it is the limitation of comint.  However, if the
definition is omitted, it is unlikely to reach that limit, since all
that is required is to send the string like
'__PYTHON_EL_get_completions("subproc")'.

bug#68559: [PATCH] Improve Python shell completion

[Liu Hui]

[-- Attachment #1: Type: text/plain, Size: 1731 bytes --]

On Fri, Feb 23, 2024 at 10:39 PM kobarity <kobarity@gmail.com> wrote:
>
> Hi Liu,
>
> I noticed that with Non-native completions, Jedi completion is not
> enabled even if PYTHONSTARTUP is set and "REPL completion using Jedi
> 0.18.2" is shown in inferior Python. Is this expected behavior?  Jedi
> completion is enabled regardless of PYTHONSTARTUP setting when I use
> IPython with Non-native completions.

Thanks for pointing out the problem! The attached patch should enable
Jedi completion for the non-native case.

On Fri, Feb 16, 2024 at 3:41 PM Eli Zaretskii <eliz@gnu.org> wrote:

> > The Python shell completion relies on the readline module, which is
> > not available for Python on MS-Windows. According to the instruction
> > in python.el:
> >
> > ;; readline based shell (it's known to work with PyPy).  If your
> > ;; Python installation lacks readline (like CPython for Windows),
> > ;; installing pyreadline (URL `https://ipython.org/pyreadline.html')
> > ;; should suffice.  To troubleshoot why you are not getting any
> > ;; completions, you can try the following in your Python shell:
> >
> > ;; >>> import readline, rlcompleter
> >
> > ;; If you see an error, then you need to either install pyreadline or
> > ;; setup custom code that avoids that dependency.
>
> I don't know if I have CPython, but the above does show an error
> message.
>
> > It may be necessary to install pyreadline (for Python 2.7) or
> > pyreadline3 (for Python 3).
>
> I will see if I can do that, thanks.
>
> Regardless, patches to the test suite to skip the tests which rely on
> those modules, if they aren't installed, will be welcome.

Such tests should be skipped now with this patch.

[-- Attachment #2: 0001-Detect-the-readline-support-for-Python-shell-complet.patch --]
[-- Type: text/x-patch, Size: 8213 bytes --]

From e02e55192e68b6272dea2a23096f90cdee517e74 Mon Sep 17 00:00:00 2001
From: Liu Hui <liuhui1610@gmail.com>
Date: Mon, 26 Feb 2024 18:46:36 +0800
Subject: [PATCH] Detect the readline support for Python shell completion

* lisp/progmodes/python.el
(python-shell-readline-completer-delims): Update docstring.
(python-shell-completion-native-setup): Move the detection of
readline to ...
(python-shell-readline-detect): ... new function.
(python-shell-completion-native-turn-on-maybe): Skip if Python
has no readline support.
(python-shell-completion-at-point): Respect the delimiter of
readline completer in non-native completion.
* test/lisp/progmodes/python-tests.el
(python-shell-completion-at-point-1)
(python-shell-completion-at-point-native-1)
(python-completion-at-point-1, python-completion-at-point-2)
(python-completion-at-point-pdb-1)
(python-completion-at-point-while-running-1)
(python-completion-at-point-native-1)
(python-completion-at-point-native-2)
(python-completion-at-point-native-with-ffap-1)
(python-completion-at-point-native-with-eldoc-1): Skip tests if
Python has no readline support.
(python-shell-completion-at-point-jedi-completer): Add test for
non-native Python shell completion. (bug#68559)
---
 lisp/progmodes/python.el            | 29 ++++++++++++++++++++++-------
 test/lisp/progmodes/python-tests.el | 23 ++++++++++++++++++++++-
 2 files changed, 44 insertions(+), 8 deletions(-)

diff --git a/lisp/progmodes/python.el b/lisp/progmodes/python.el
index 587d0b36304..6c73ceba4ef 100644
--- a/lisp/progmodes/python.el
+++ b/lisp/progmodes/python.el
@@ -4439,7 +4439,24 @@ python-shell-completion-native-try-output-timeout
 
 (defvar python-shell-readline-completer-delims nil
   "Word delimiters used by the readline completer.
-It is automatically set by Python shell.")
+It is automatically set by Python shell.  A value of nil means the
+Python shell has no readline support.")
+
+(defun python-shell-readline-detect ()
+  "Detect the readline support for Python shell completion."
+  (let* ((process (python-shell-get-process))
+         (output (python-shell-send-string-no-output "
+try:
+    import readline
+    print(readline.get_completer_delims())
+except:
+    print('No readline support')" process)))
+    (unless (string-match-p "No readline support" output)
+      (setq-local python-shell-readline-completer-delims
+                  (string-trim-right output)))))
+
+(add-hook 'python-shell-first-prompt-hook
+          'python-shell-readline-detect -90)
 
 (defvar python-shell-completion-native-redirect-buffer
   " *Python completions redirect*"
@@ -4579,10 +4596,6 @@ python-shell-completion-native-setup
 __PYTHON_EL_native_completion_setup()" process)))
     (when (string-match-p "python\\.el: native completion setup loaded"
                           output)
-      (setq-local python-shell-readline-completer-delims
-                  (string-trim-right
-                   (python-shell-send-string-no-output
-                    "import readline; print(readline.get_completer_delims())")))
       (python-shell-completion-native-try))))
 
 (defun python-shell-completion-native-turn-off (&optional msg)
@@ -4611,7 +4624,8 @@ python-shell-completion-native-turn-on-maybe
       (cond
        ((python-shell-completion-native-interpreter-disabled-p)
         (python-shell-completion-native-turn-off msg))
-       ((python-shell-completion-native-setup)
+       ((and python-shell-readline-completer-delims
+             (python-shell-completion-native-setup))
         (when msg
           (message "Shell native completion is enabled.")))
        (t
@@ -4783,7 +4797,8 @@ python-shell-completion-at-point
                (with-current-buffer (process-buffer process)
                  (if python-shell-completion-native-enable
                      (string= python-shell-readline-completer-delims "")
-                   (string-match-p "ipython[23]?\\'" python-shell-interpreter)))))
+                   (or (string-match-p "ipython[23]?\\'" python-shell-interpreter)
+                       (string= python-shell-readline-completer-delims ""))))))
          (start
           (if (< (point) line-start)
               (point)
diff --git a/test/lisp/progmodes/python-tests.el b/test/lisp/progmodes/python-tests.el
index 6c6cd9eee2b..e0584f78f82 100644
--- a/test/lisp/progmodes/python-tests.el
+++ b/test/lisp/progmodes/python-tests.el
@@ -4777,6 +4777,7 @@ python-shell-completion-at-point-1
   (python-tests-with-temp-buffer-with-shell
    ""
    (python-shell-with-shell-buffer
+     (skip-when (null python-shell-readline-completer-delims))
      (insert "import abc")
      (comint-send-input)
      (python-tests-shell-wait-for-prompt)
@@ -4791,6 +4792,7 @@ python-shell-completion-at-point-native-1
    ""
    (python-shell-completion-native-turn-on)
    (python-shell-with-shell-buffer
+     (skip-when (null python-shell-readline-completer-delims))
      (insert "import abc")
      (comint-send-input)
      (python-tests-shell-wait-for-prompt)
@@ -4866,8 +4868,11 @@ python-shell-completion-at-point-jedi-completer
   (python-tests-with-temp-buffer-with-shell
    ""
    (python-shell-with-shell-buffer
-     (python-shell-completion-native-turn-on)
      (skip-unless (string= python-shell-readline-completer-delims ""))
+     (python-shell-completion-native-turn-off)
+     (python-tests--completion-module)
+     (python-tests--completion-parameters)
+     (python-shell-completion-native-turn-on)
      (python-tests--completion-module)
      (python-tests--completion-parameters)
      (python-tests--completion-extra-context))))
@@ -4905,6 +4910,8 @@ python-completion-at-point-1
 import abc
 "
    (let ((inhibit-message t))
+     (python-shell-with-shell-buffer
+       (skip-when (null python-shell-readline-completer-delims)))
      (python-shell-send-buffer)
      (python-tests-shell-wait-for-prompt)
      (goto-char (point-max))
@@ -4921,6 +4928,8 @@ python-completion-at-point-2
 import abc
 "
    (let ((inhibit-message t))
+     (python-shell-with-shell-buffer
+       (skip-when (null python-shell-readline-completer-delims)))
      (python-shell-send-buffer)
      (python-tests-shell-wait-for-prompt)
      (python-shell-with-shell-buffer
@@ -4940,6 +4949,8 @@ python-completion-at-point-pdb-1
 print('Hello')
 "
    (let ((inhibit-message t))
+     (python-shell-with-shell-buffer
+       (skip-when (null python-shell-readline-completer-delims)))
      (python-shell-send-buffer)
      (python-tests-shell-wait-for-prompt)
      (goto-char (point-max))
@@ -4956,6 +4967,8 @@ python-completion-at-point-while-running-1
 time.sleep(3)
 "
    (let ((inhibit-message t))
+     (python-shell-with-shell-buffer
+       (skip-when (null python-shell-readline-completer-delims)))
      (python-shell-send-buffer)
      (goto-char (point-max))
      (insert "time.")
@@ -4968,6 +4981,8 @@ python-completion-at-point-native-1
 import abc
 "
    (let ((inhibit-message t))
+     (python-shell-with-shell-buffer
+       (skip-when (null python-shell-readline-completer-delims)))
      (python-shell-completion-native-turn-on)
      (python-shell-send-buffer)
      (python-tests-shell-wait-for-prompt)
@@ -4985,6 +5000,8 @@ python-completion-at-point-native-2
 import abc
 "
    (let ((inhibit-message t))
+     (python-shell-with-shell-buffer
+       (skip-when (null python-shell-readline-completer-delims)))
      (python-shell-completion-native-turn-on)
      (python-shell-send-buffer)
      (python-tests-shell-wait-for-prompt)
@@ -5001,6 +5018,8 @@ python-completion-at-point-native-with-ffap-1
 import abc
 "
    (let ((inhibit-message t))
+     (python-shell-with-shell-buffer
+       (skip-when (null python-shell-readline-completer-delims)))
      (python-shell-completion-native-turn-on)
      (python-shell-send-buffer)
      (python-tests-shell-wait-for-prompt)
@@ -5017,6 +5036,8 @@ python-completion-at-point-native-with-eldoc-1
 import abc
 "
    (let ((inhibit-message t))
+     (python-shell-with-shell-buffer
+       (skip-when (null python-shell-readline-completer-delims)))
      (python-shell-completion-native-turn-on)
      (python-shell-send-buffer)
      (python-tests-shell-wait-for-prompt)
-- 
2.25.1

bug#68559: [PATCH] Improve Python shell completion

[Mattias Engdegård]

26 feb. 2024 kl. 12.06 skrev Liu Hui <liuhui1610@gmail.com>:

> Thanks for pointing out the problem! The attached patch should enable
> Jedi completion for the non-native case.

This patch does not break anything further on macOS using the standard system Python.
I just thought you would be happy to know.

bug#68559: [PATCH] Improve Python shell completion

[kobarity]

Mattias Engdegård wrote:
> 
> 26 feb. 2024 kl. 12.06 skrev Liu Hui <liuhui1610@gmail.com>:
> 
> > Thanks for pointing out the problem! The attached patch should enable
> > Jedi completion for the non-native case.
> 
> This patch does not break anything further on macOS using the standard system Python.
> I just thought you would be happy to know.

Thanks, I confirmed that the patch enables Jedi completion for the
non-native mode.

bug#68559: [PATCH] Improve Python shell completion

[Basil L. Contovounesios]

Liu Hui [2024-02-26 19:06 +0800] wrote:

> +(add-hook 'python-shell-first-prompt-hook
> +          'python-shell-readline-detect -90)

This depth currently prepends the function to the hook, but in Emacs 24
it instead appends to the hook.  Is that a problem?

> +     (skip-when (null python-shell-readline-completer-delims))

skip-unless here and below?

Thanks,
-- 
Basil

bug#68559: [PATCH] Improve Python shell completion

[Liu Hui]

[-- Attachment #1: Type: text/plain, Size: 723 bytes --]

On Wed, Feb 28, 2024 at 10:50 PM Basil L. Contovounesios
<basil@contovou.net> wrote:
>
> Liu Hui [2024-02-26 19:06 +0800] wrote:
>
> > +(add-hook 'python-shell-first-prompt-hook
> > +          'python-shell-readline-detect -90)
>
> This depth currently prepends the function to the hook, but in Emacs 24
> it instead appends to the hook.  Is that a problem?

Thanks for pointing it out!  It is a problem because
python-shell-readline-detect is expected to be executed before
python-shell-completion-native-turn-on-maybe-with-msg, i.e. another
function in the hook.  I have updated the patch.

> > +     (skip-when (null python-shell-readline-completer-delims))
>
> skip-unless here and below?

Done.

[-- Attachment #2: 0001-Detect-the-readline-support-for-Python-shell-complet.patch --]
[-- Type: text/x-patch, Size: 8894 bytes --]

From 1a24ac775d8577da0990de89016afed5bfef37fa Mon Sep 17 00:00:00 2001
From: Liu Hui <liuhui1610@gmail.com>
Date: Mon, 26 Feb 2024 18:46:36 +0800
Subject: [PATCH] Detect the readline support for Python shell completion

* lisp/progmodes/python.el
(python-shell-comint-watch-for-first-prompt-output-filter):
Detect the readline support.
(python-shell-readline-completer-delims): Update docstring.
(python-shell-completion-native-setup): Move the readline
detection code to ...
(python-shell-readline-detect): ... new function.
(python-shell-completion-native-turn-on-maybe): Skip if Python
has no readline support.
(python-shell-completion-at-point): Respect the delimiter of
readline completer in non-native completion.
* test/lisp/progmodes/python-tests.el
(python-shell-completion-at-point-1)
(python-shell-completion-at-point-native-1)
(python-completion-at-point-1, python-completion-at-point-2)
(python-completion-at-point-pdb-1)
(python-completion-at-point-while-running-1)
(python-completion-at-point-native-1)
(python-completion-at-point-native-2)
(python-completion-at-point-native-with-ffap-1)
(python-completion-at-point-native-with-eldoc-1): Skip tests if
Python has no readline support.
(python-shell-completion-at-point-jedi-completer): Add test for
non-native Python shell completion. (bug#68559)
---
 lisp/progmodes/python.el            | 27 ++++++++++++++++++-------
 test/lisp/progmodes/python-tests.el | 31 ++++++++++++++++++++++++-----
 2 files changed, 46 insertions(+), 12 deletions(-)

diff --git a/lisp/progmodes/python.el b/lisp/progmodes/python.el
index 587d0b36304..9ac462be416 100644
--- a/lisp/progmodes/python.el
+++ b/lisp/progmodes/python.el
@@ -3601,6 +3601,7 @@ python-shell-comint-watch-for-first-prompt-output-filter
           (python-shell-send-string-no-output python-shell-eval-file-setup-code))
         (with-current-buffer (current-buffer)
           (let ((inhibit-quit nil))
+            (python-shell-readline-detect)
             (run-hooks 'python-shell-first-prompt-hook))))))
   output)
 
@@ -4439,7 +4440,21 @@ python-shell-completion-native-try-output-timeout
 
 (defvar python-shell-readline-completer-delims nil
   "Word delimiters used by the readline completer.
-It is automatically set by Python shell.")
+It is automatically set by Python shell.  A value of nil means the
+Python shell has no readline support.")
+
+(defun python-shell-readline-detect ()
+  "Detect the readline support for Python shell completion."
+  (let* ((process (python-shell-get-process))
+         (output (python-shell-send-string-no-output "
+try:
+    import readline
+    print(readline.get_completer_delims())
+except:
+    print('No readline support')" process)))
+    (setq-local python-shell-readline-completer-delims
+                (unless (string-match-p "No readline support" output)
+                  (string-trim-right output)))))
 
 (defvar python-shell-completion-native-redirect-buffer
   " *Python completions redirect*"
@@ -4579,10 +4594,6 @@ python-shell-completion-native-setup
 __PYTHON_EL_native_completion_setup()" process)))
     (when (string-match-p "python\\.el: native completion setup loaded"
                           output)
-      (setq-local python-shell-readline-completer-delims
-                  (string-trim-right
-                   (python-shell-send-string-no-output
-                    "import readline; print(readline.get_completer_delims())")))
       (python-shell-completion-native-try))))
 
 (defun python-shell-completion-native-turn-off (&optional msg)
@@ -4611,7 +4622,8 @@ python-shell-completion-native-turn-on-maybe
       (cond
        ((python-shell-completion-native-interpreter-disabled-p)
         (python-shell-completion-native-turn-off msg))
-       ((python-shell-completion-native-setup)
+       ((and python-shell-readline-completer-delims
+             (python-shell-completion-native-setup))
         (when msg
           (message "Shell native completion is enabled.")))
        (t
@@ -4783,7 +4795,8 @@ python-shell-completion-at-point
                (with-current-buffer (process-buffer process)
                  (if python-shell-completion-native-enable
                      (string= python-shell-readline-completer-delims "")
-                   (string-match-p "ipython[23]?\\'" python-shell-interpreter)))))
+                   (or (string-match-p "ipython[23]?\\'" python-shell-interpreter)
+                       (string= python-shell-readline-completer-delims ""))))))
          (start
           (if (< (point) line-start)
               (point)
diff --git a/test/lisp/progmodes/python-tests.el b/test/lisp/progmodes/python-tests.el
index 1ceee690cfb..e11440cdb5b 100644
--- a/test/lisp/progmodes/python-tests.el
+++ b/test/lisp/progmodes/python-tests.el
@@ -4783,6 +4783,7 @@ python-shell-completion-at-point-1
   (python-tests-with-temp-buffer-with-shell
    ""
    (python-shell-with-shell-buffer
+     (skip-unless python-shell-readline-completer-delims)
      (insert "import abc")
      (comint-send-input)
      (python-tests-shell-wait-for-prompt)
@@ -4797,6 +4798,7 @@ python-shell-completion-at-point-native-1
    ""
    (python-shell-completion-native-turn-on)
    (python-shell-with-shell-buffer
+     (skip-unless python-shell-readline-completer-delims)
      (insert "import abc")
      (comint-send-input)
      (python-tests-shell-wait-for-prompt)
@@ -4883,11 +4885,14 @@ python-shell-completion-at-point-jedi-completer
     (python-tests-with-temp-buffer-with-shell
      ""
      (python-shell-with-shell-buffer
-      (python-shell-completion-native-turn-on)
-      (skip-unless (string= python-shell-readline-completer-delims ""))
-      (python-tests--completion-module)
-      (python-tests--completion-parameters)
-      (python-tests--completion-extra-context)))))
+       (skip-unless (string= python-shell-readline-completer-delims ""))
+       (python-shell-completion-native-turn-off)
+       (python-tests--completion-module)
+       (python-tests--completion-parameters)
+       (python-shell-completion-native-turn-on)
+       (python-tests--completion-module)
+       (python-tests--completion-parameters)
+       (python-tests--completion-extra-context)))))
 
 (ert-deftest python-shell-completion-at-point-ipython ()
   "Check if Python shell completion works for IPython."
@@ -4924,6 +4929,8 @@ python-completion-at-point-1
 import abc
 "
    (let ((inhibit-message t))
+     (python-shell-with-shell-buffer
+       (skip-unless python-shell-readline-completer-delims))
      (python-shell-send-buffer)
      (python-tests-shell-wait-for-prompt)
      (goto-char (point-max))
@@ -4940,6 +4947,8 @@ python-completion-at-point-2
 import abc
 "
    (let ((inhibit-message t))
+     (python-shell-with-shell-buffer
+       (skip-unless python-shell-readline-completer-delims))
      (python-shell-send-buffer)
      (python-tests-shell-wait-for-prompt)
      (python-shell-with-shell-buffer
@@ -4959,6 +4968,8 @@ python-completion-at-point-pdb-1
 print('Hello')
 "
    (let ((inhibit-message t))
+     (python-shell-with-shell-buffer
+       (skip-unless python-shell-readline-completer-delims))
      (python-shell-send-buffer)
      (python-tests-shell-wait-for-prompt)
      (goto-char (point-max))
@@ -4975,6 +4986,8 @@ python-completion-at-point-while-running-1
 time.sleep(3)
 "
    (let ((inhibit-message t))
+     (python-shell-with-shell-buffer
+       (skip-unless python-shell-readline-completer-delims))
      (python-shell-send-buffer)
      (goto-char (point-max))
      (insert "time.")
@@ -4987,6 +5000,8 @@ python-completion-at-point-native-1
 import abc
 "
    (let ((inhibit-message t))
+     (python-shell-with-shell-buffer
+       (skip-unless python-shell-readline-completer-delims))
      (python-shell-completion-native-turn-on)
      (python-shell-send-buffer)
      (python-tests-shell-wait-for-prompt)
@@ -5004,6 +5019,8 @@ python-completion-at-point-native-2
 import abc
 "
    (let ((inhibit-message t))
+     (python-shell-with-shell-buffer
+       (skip-unless python-shell-readline-completer-delims))
      (python-shell-completion-native-turn-on)
      (python-shell-send-buffer)
      (python-tests-shell-wait-for-prompt)
@@ -5020,6 +5037,8 @@ python-completion-at-point-native-with-ffap-1
 import abc
 "
    (let ((inhibit-message t))
+     (python-shell-with-shell-buffer
+       (skip-unless python-shell-readline-completer-delims))
      (python-shell-completion-native-turn-on)
      (python-shell-send-buffer)
      (python-tests-shell-wait-for-prompt)
@@ -5036,6 +5055,8 @@ python-completion-at-point-native-with-eldoc-1
 import abc
 "
    (let ((inhibit-message t))
+     (python-shell-with-shell-buffer
+       (skip-unless python-shell-readline-completer-delims))
      (python-shell-completion-native-turn-on)
      (python-shell-send-buffer)
      (python-tests-shell-wait-for-prompt)
-- 
2.25.1

bug#68559: [PATCH] Improve Python shell completion

[Basil L. Contovounesios]

Liu Hui [2024-03-06 18:14 +0800] wrote:

> I have updated the patch.

Thanks!  The build and tests succeed here, without skipped tests.

> +                (unless (string-match-p "No readline support" output)

Nit: why not plain 'string-search' instead of a regexp search?

> -                   (string-match-p "ipython[23]?\\'" python-shell-interpreter)))))
> +                   (or (string-match-p "ipython[23]?\\'" python-shell-interpreter)
> +                       (string= python-shell-readline-completer-delims ""))))))

Just curious: what does the empty string signify?

[ If it's not just a dumb question, perhaps the meaning could be added
  to the variable's docstring/commentary. ]

-- 
Basil

bug#68559: [PATCH] Improve Python shell completion

[Liu Hui]

[-- Attachment #1: Type: text/plain, Size: 1115 bytes --]

On Fri, Mar 8, 2024 at 11:44 PM Basil L. Contovounesios
<basil@contovou.net> wrote:
>
> Liu Hui [2024-03-06 18:14 +0800] wrote:
>
> > I have updated the patch.
>
> Thanks!  The build and tests succeed here, without skipped tests.
>
> > +                (unless (string-match-p "No readline support" output)
>
> Nit: why not plain 'string-search' instead of a regexp search?

Done.

> > -                   (string-match-p "ipython[23]?\\'" python-shell-interpreter)))))
> > +                   (or (string-match-p "ipython[23]?\\'" python-shell-interpreter)
> > +                       (string= python-shell-readline-completer-delims ""))))))
>
> Just curious: what does the empty string signify?
>
> [ If it's not just a dumb question, perhaps the meaning could be added
>   to the variable's docstring/commentary. ]

The empty string means no characters are considered delimiters and the
readline completion could consider the entire line of input without
breaking it into parts based on typical delimiters like spaces or
punctuation. The docstring is updated in the attached patch.

[-- Attachment #2: 0001-Detect-the-readline-support-for-Python-shell-complet.patch --]
[-- Type: text/x-patch, Size: 9024 bytes --]

From bb1f8f92d5cd3c19d404e023ab33315bfa8a535a Mon Sep 17 00:00:00 2001
From: Liu Hui <liuhui1610@gmail.com>
Date: Mon, 26 Feb 2024 18:46:36 +0800
Subject: [PATCH] Detect the readline support for Python shell completion

* lisp/progmodes/python.el
(python-shell-comint-watch-for-first-prompt-output-filter):
Detect the readline support.
(python-shell-readline-completer-delims): Update docstring.
(python-shell-completion-native-setup): Move the readline
detection code to ...
(python-shell-readline-detect): ... new function.
(python-shell-completion-native-turn-on-maybe): Skip if Python
has no readline support.
(python-shell-completion-at-point): Respect the delimiter of
readline completer in non-native completion.
* test/lisp/progmodes/python-tests.el
(python-shell-completion-at-point-1)
(python-shell-completion-at-point-native-1)
(python-completion-at-point-1, python-completion-at-point-2)
(python-completion-at-point-pdb-1)
(python-completion-at-point-while-running-1)
(python-completion-at-point-native-1)
(python-completion-at-point-native-2)
(python-completion-at-point-native-with-ffap-1)
(python-completion-at-point-native-with-eldoc-1): Skip tests if
Python has no readline support.
(python-shell-completion-at-point-jedi-completer): Add test for
non-native Python shell completion. (bug#68559)
---
 lisp/progmodes/python.el            | 29 ++++++++++++++++++++-------
 test/lisp/progmodes/python-tests.el | 31 ++++++++++++++++++++++++-----
 2 files changed, 48 insertions(+), 12 deletions(-)

diff --git a/lisp/progmodes/python.el b/lisp/progmodes/python.el
index 587d0b36304..0ea0b918872 100644
--- a/lisp/progmodes/python.el
+++ b/lisp/progmodes/python.el
@@ -3601,6 +3601,7 @@ python-shell-comint-watch-for-first-prompt-output-filter
           (python-shell-send-string-no-output python-shell-eval-file-setup-code))
         (with-current-buffer (current-buffer)
           (let ((inhibit-quit nil))
+            (python-shell-readline-detect)
             (run-hooks 'python-shell-first-prompt-hook))))))
   output)
 
@@ -4439,7 +4440,23 @@ python-shell-completion-native-try-output-timeout
 
 (defvar python-shell-readline-completer-delims nil
   "Word delimiters used by the readline completer.
-It is automatically set by Python shell.")
+It is automatically set by Python shell.  An empty string means no
+characters are considered delimiters and the readline completion
+considers the entire line of input.  A value of nil means the Python
+shell has no readline support.")
+
+(defun python-shell-readline-detect ()
+  "Detect the readline support for Python shell completion."
+  (let* ((process (python-shell-get-process))
+         (output (python-shell-send-string-no-output "
+try:
+    import readline
+    print(readline.get_completer_delims())
+except:
+    print('No readline support')" process)))
+    (setq-local python-shell-readline-completer-delims
+                (unless (string-search "No readline support" output)
+                  (string-trim-right output)))))
 
 (defvar python-shell-completion-native-redirect-buffer
   " *Python completions redirect*"
@@ -4579,10 +4596,6 @@ python-shell-completion-native-setup
 __PYTHON_EL_native_completion_setup()" process)))
     (when (string-match-p "python\\.el: native completion setup loaded"
                           output)
-      (setq-local python-shell-readline-completer-delims
-                  (string-trim-right
-                   (python-shell-send-string-no-output
-                    "import readline; print(readline.get_completer_delims())")))
       (python-shell-completion-native-try))))
 
 (defun python-shell-completion-native-turn-off (&optional msg)
@@ -4611,7 +4624,8 @@ python-shell-completion-native-turn-on-maybe
       (cond
        ((python-shell-completion-native-interpreter-disabled-p)
         (python-shell-completion-native-turn-off msg))
-       ((python-shell-completion-native-setup)
+       ((and python-shell-readline-completer-delims
+             (python-shell-completion-native-setup))
         (when msg
           (message "Shell native completion is enabled.")))
        (t
@@ -4783,7 +4797,8 @@ python-shell-completion-at-point
                (with-current-buffer (process-buffer process)
                  (if python-shell-completion-native-enable
                      (string= python-shell-readline-completer-delims "")
-                   (string-match-p "ipython[23]?\\'" python-shell-interpreter)))))
+                   (or (string-match-p "ipython[23]?\\'" python-shell-interpreter)
+                       (string= python-shell-readline-completer-delims ""))))))
          (start
           (if (< (point) line-start)
               (point)
diff --git a/test/lisp/progmodes/python-tests.el b/test/lisp/progmodes/python-tests.el
index 1ceee690cfb..e11440cdb5b 100644
--- a/test/lisp/progmodes/python-tests.el
+++ b/test/lisp/progmodes/python-tests.el
@@ -4783,6 +4783,7 @@ python-shell-completion-at-point-1
   (python-tests-with-temp-buffer-with-shell
    ""
    (python-shell-with-shell-buffer
+     (skip-unless python-shell-readline-completer-delims)
      (insert "import abc")
      (comint-send-input)
      (python-tests-shell-wait-for-prompt)
@@ -4797,6 +4798,7 @@ python-shell-completion-at-point-native-1
    ""
    (python-shell-completion-native-turn-on)
    (python-shell-with-shell-buffer
+     (skip-unless python-shell-readline-completer-delims)
      (insert "import abc")
      (comint-send-input)
      (python-tests-shell-wait-for-prompt)
@@ -4883,11 +4885,14 @@ python-shell-completion-at-point-jedi-completer
     (python-tests-with-temp-buffer-with-shell
      ""
      (python-shell-with-shell-buffer
-      (python-shell-completion-native-turn-on)
-      (skip-unless (string= python-shell-readline-completer-delims ""))
-      (python-tests--completion-module)
-      (python-tests--completion-parameters)
-      (python-tests--completion-extra-context)))))
+       (skip-unless (string= python-shell-readline-completer-delims ""))
+       (python-shell-completion-native-turn-off)
+       (python-tests--completion-module)
+       (python-tests--completion-parameters)
+       (python-shell-completion-native-turn-on)
+       (python-tests--completion-module)
+       (python-tests--completion-parameters)
+       (python-tests--completion-extra-context)))))
 
 (ert-deftest python-shell-completion-at-point-ipython ()
   "Check if Python shell completion works for IPython."
@@ -4924,6 +4929,8 @@ python-completion-at-point-1
 import abc
 "
    (let ((inhibit-message t))
+     (python-shell-with-shell-buffer
+       (skip-unless python-shell-readline-completer-delims))
      (python-shell-send-buffer)
      (python-tests-shell-wait-for-prompt)
      (goto-char (point-max))
@@ -4940,6 +4947,8 @@ python-completion-at-point-2
 import abc
 "
    (let ((inhibit-message t))
+     (python-shell-with-shell-buffer
+       (skip-unless python-shell-readline-completer-delims))
      (python-shell-send-buffer)
      (python-tests-shell-wait-for-prompt)
      (python-shell-with-shell-buffer
@@ -4959,6 +4968,8 @@ python-completion-at-point-pdb-1
 print('Hello')
 "
    (let ((inhibit-message t))
+     (python-shell-with-shell-buffer
+       (skip-unless python-shell-readline-completer-delims))
      (python-shell-send-buffer)
      (python-tests-shell-wait-for-prompt)
      (goto-char (point-max))
@@ -4975,6 +4986,8 @@ python-completion-at-point-while-running-1
 time.sleep(3)
 "
    (let ((inhibit-message t))
+     (python-shell-with-shell-buffer
+       (skip-unless python-shell-readline-completer-delims))
      (python-shell-send-buffer)
      (goto-char (point-max))
      (insert "time.")
@@ -4987,6 +5000,8 @@ python-completion-at-point-native-1
 import abc
 "
    (let ((inhibit-message t))
+     (python-shell-with-shell-buffer
+       (skip-unless python-shell-readline-completer-delims))
      (python-shell-completion-native-turn-on)
      (python-shell-send-buffer)
      (python-tests-shell-wait-for-prompt)
@@ -5004,6 +5019,8 @@ python-completion-at-point-native-2
 import abc
 "
    (let ((inhibit-message t))
+     (python-shell-with-shell-buffer
+       (skip-unless python-shell-readline-completer-delims))
      (python-shell-completion-native-turn-on)
      (python-shell-send-buffer)
      (python-tests-shell-wait-for-prompt)
@@ -5020,6 +5037,8 @@ python-completion-at-point-native-with-ffap-1
 import abc
 "
    (let ((inhibit-message t))
+     (python-shell-with-shell-buffer
+       (skip-unless python-shell-readline-completer-delims))
      (python-shell-completion-native-turn-on)
      (python-shell-send-buffer)
      (python-tests-shell-wait-for-prompt)
@@ -5036,6 +5055,8 @@ python-completion-at-point-native-with-eldoc-1
 import abc
 "
    (let ((inhibit-message t))
+     (python-shell-with-shell-buffer
+       (skip-unless python-shell-readline-completer-delims))
      (python-shell-completion-native-turn-on)
      (python-shell-send-buffer)
      (python-tests-shell-wait-for-prompt)
-- 
2.25.1

bug#68559: [PATCH] Improve Python shell completion

[Basil L. Contovounesios]

Liu Hui [2024-03-11 19:35 +0800] wrote:

> On Fri, Mar 8, 2024 at 11:44 PM Basil L. Contovounesios
> <basil@contovou.net> wrote:
>>
>> Liu Hui [2024-03-06 18:14 +0800] wrote:
>>
>> > -                   (string-match-p "ipython[23]?\\'" python-shell-interpreter)))))
>> > +                   (or (string-match-p "ipython[23]?\\'" python-shell-interpreter)
>> > +                       (string= python-shell-readline-completer-delims ""))))))
>>
>> Just curious: what does the empty string signify?
>>
>> [ If it's not just a dumb question, perhaps the meaning could be added
>>   to the variable's docstring/commentary. ]
>
> The empty string means no characters are considered delimiters and the
> readline completion could consider the entire line of input without
> breaking it into parts based on typical delimiters like spaces or
> punctuation. The docstring is updated in the attached patch.

Thanks!  But that makes me wonder: in the cases where we check

  (string= python-shell-readline-completer-delims "")

is there a possibility that the variable's value will be nil?
(In which case we should compare with equal instead of string=.)

-- 
Basil

bug#68559: [PATCH] Improve Python shell completion

[Liu Hui]

[-- Attachment #1: Type: text/plain, Size: 1440 bytes --]

On Tue, Mar 12, 2024 at 12:02 AM Basil L. Contovounesios
<basil@contovou.net> wrote:
>
> Liu Hui [2024-03-11 19:35 +0800] wrote:
>
> > On Fri, Mar 8, 2024 at 11:44 PM Basil L. Contovounesios
> > <basil@contovou.net> wrote:
> >>
> >> Liu Hui [2024-03-06 18:14 +0800] wrote:
> >>
> >> > -                   (string-match-p "ipython[23]?\\'" python-shell-interpreter)))))
> >> > +                   (or (string-match-p "ipython[23]?\\'" python-shell-interpreter)
> >> > +                       (string= python-shell-readline-completer-delims ""))))))
> >>
> >> Just curious: what does the empty string signify?
> >>
> >> [ If it's not just a dumb question, perhaps the meaning could be added
> >>   to the variable's docstring/commentary. ]
> >
> > The empty string means no characters are considered delimiters and the
> > readline completion could consider the entire line of input without
> > breaking it into parts based on typical delimiters like spaces or
> > punctuation. The docstring is updated in the attached patch.
>
> Thanks!  But that makes me wonder: in the cases where we check
>
>   (string= python-shell-readline-completer-delims "")
>
> is there a possibility that the variable's value will be nil?
> (In which case we should compare with equal instead of string=.)

Yes, it is a string with native completion and may be nil with
non-native completion. I have updated the patch. Thanks.

[-- Attachment #2: 0001-Detect-the-readline-support-for-Python-shell-complet.patch --]
[-- Type: text/x-patch, Size: 9022 bytes --]

From d204d5ae2439dc76662e4e13491ff103216a2d12 Mon Sep 17 00:00:00 2001
From: Liu Hui <liuhui1610@gmail.com>
Date: Mon, 26 Feb 2024 18:46:36 +0800
Subject: [PATCH] Detect the readline support for Python shell completion

* lisp/progmodes/python.el
(python-shell-comint-watch-for-first-prompt-output-filter):
Detect the readline support.
(python-shell-readline-completer-delims): Update docstring.
(python-shell-completion-native-setup): Move the readline
detection code to ...
(python-shell-readline-detect): ... new function.
(python-shell-completion-native-turn-on-maybe): Skip if Python
has no readline support.
(python-shell-completion-at-point): Respect the delimiter of
readline completer in non-native completion.
* test/lisp/progmodes/python-tests.el
(python-shell-completion-at-point-1)
(python-shell-completion-at-point-native-1)
(python-completion-at-point-1, python-completion-at-point-2)
(python-completion-at-point-pdb-1)
(python-completion-at-point-while-running-1)
(python-completion-at-point-native-1)
(python-completion-at-point-native-2)
(python-completion-at-point-native-with-ffap-1)
(python-completion-at-point-native-with-eldoc-1): Skip tests if
Python has no readline support.
(python-shell-completion-at-point-jedi-completer): Add test for
non-native Python shell completion. (bug#68559)
---
 lisp/progmodes/python.el            | 29 ++++++++++++++++++++-------
 test/lisp/progmodes/python-tests.el | 31 ++++++++++++++++++++++++-----
 2 files changed, 48 insertions(+), 12 deletions(-)

diff --git a/lisp/progmodes/python.el b/lisp/progmodes/python.el
index 587d0b36304..c94bb4fc550 100644
--- a/lisp/progmodes/python.el
+++ b/lisp/progmodes/python.el
@@ -3601,6 +3601,7 @@ python-shell-comint-watch-for-first-prompt-output-filter
           (python-shell-send-string-no-output python-shell-eval-file-setup-code))
         (with-current-buffer (current-buffer)
           (let ((inhibit-quit nil))
+            (python-shell-readline-detect)
             (run-hooks 'python-shell-first-prompt-hook))))))
   output)
 
@@ -4439,7 +4440,23 @@ python-shell-completion-native-try-output-timeout
 
 (defvar python-shell-readline-completer-delims nil
   "Word delimiters used by the readline completer.
-It is automatically set by Python shell.")
+It is automatically set by Python shell.  An empty string means no
+characters are considered delimiters and the readline completion
+considers the entire line of input.  A value of nil means the Python
+shell has no readline support.")
+
+(defun python-shell-readline-detect ()
+  "Detect the readline support for Python shell completion."
+  (let* ((process (python-shell-get-process))
+         (output (python-shell-send-string-no-output "
+try:
+    import readline
+    print(readline.get_completer_delims())
+except:
+    print('No readline support')" process)))
+    (setq-local python-shell-readline-completer-delims
+                (unless (string-search "No readline support" output)
+                  (string-trim-right output)))))
 
 (defvar python-shell-completion-native-redirect-buffer
   " *Python completions redirect*"
@@ -4579,10 +4596,6 @@ python-shell-completion-native-setup
 __PYTHON_EL_native_completion_setup()" process)))
     (when (string-match-p "python\\.el: native completion setup loaded"
                           output)
-      (setq-local python-shell-readline-completer-delims
-                  (string-trim-right
-                   (python-shell-send-string-no-output
-                    "import readline; print(readline.get_completer_delims())")))
       (python-shell-completion-native-try))))
 
 (defun python-shell-completion-native-turn-off (&optional msg)
@@ -4611,7 +4624,8 @@ python-shell-completion-native-turn-on-maybe
       (cond
        ((python-shell-completion-native-interpreter-disabled-p)
         (python-shell-completion-native-turn-off msg))
-       ((python-shell-completion-native-setup)
+       ((and python-shell-readline-completer-delims
+             (python-shell-completion-native-setup))
         (when msg
           (message "Shell native completion is enabled.")))
        (t
@@ -4783,7 +4797,8 @@ python-shell-completion-at-point
                (with-current-buffer (process-buffer process)
                  (if python-shell-completion-native-enable
                      (string= python-shell-readline-completer-delims "")
-                   (string-match-p "ipython[23]?\\'" python-shell-interpreter)))))
+                   (or (string-match-p "ipython[23]?\\'" python-shell-interpreter)
+                       (equal python-shell-readline-completer-delims ""))))))
          (start
           (if (< (point) line-start)
               (point)
diff --git a/test/lisp/progmodes/python-tests.el b/test/lisp/progmodes/python-tests.el
index 1ceee690cfb..e11440cdb5b 100644
--- a/test/lisp/progmodes/python-tests.el
+++ b/test/lisp/progmodes/python-tests.el
@@ -4783,6 +4783,7 @@ python-shell-completion-at-point-1
   (python-tests-with-temp-buffer-with-shell
    ""
    (python-shell-with-shell-buffer
+     (skip-unless python-shell-readline-completer-delims)
      (insert "import abc")
      (comint-send-input)
      (python-tests-shell-wait-for-prompt)
@@ -4797,6 +4798,7 @@ python-shell-completion-at-point-native-1
    ""
    (python-shell-completion-native-turn-on)
    (python-shell-with-shell-buffer
+     (skip-unless python-shell-readline-completer-delims)
      (insert "import abc")
      (comint-send-input)
      (python-tests-shell-wait-for-prompt)
@@ -4883,11 +4885,14 @@ python-shell-completion-at-point-jedi-completer
     (python-tests-with-temp-buffer-with-shell
      ""
      (python-shell-with-shell-buffer
-      (python-shell-completion-native-turn-on)
-      (skip-unless (string= python-shell-readline-completer-delims ""))
-      (python-tests--completion-module)
-      (python-tests--completion-parameters)
-      (python-tests--completion-extra-context)))))
+       (skip-unless (string= python-shell-readline-completer-delims ""))
+       (python-shell-completion-native-turn-off)
+       (python-tests--completion-module)
+       (python-tests--completion-parameters)
+       (python-shell-completion-native-turn-on)
+       (python-tests--completion-module)
+       (python-tests--completion-parameters)
+       (python-tests--completion-extra-context)))))
 
 (ert-deftest python-shell-completion-at-point-ipython ()
   "Check if Python shell completion works for IPython."
@@ -4924,6 +4929,8 @@ python-completion-at-point-1
 import abc
 "
    (let ((inhibit-message t))
+     (python-shell-with-shell-buffer
+       (skip-unless python-shell-readline-completer-delims))
      (python-shell-send-buffer)
      (python-tests-shell-wait-for-prompt)
      (goto-char (point-max))
@@ -4940,6 +4947,8 @@ python-completion-at-point-2
 import abc
 "
    (let ((inhibit-message t))
+     (python-shell-with-shell-buffer
+       (skip-unless python-shell-readline-completer-delims))
      (python-shell-send-buffer)
      (python-tests-shell-wait-for-prompt)
      (python-shell-with-shell-buffer
@@ -4959,6 +4968,8 @@ python-completion-at-point-pdb-1
 print('Hello')
 "
    (let ((inhibit-message t))
+     (python-shell-with-shell-buffer
+       (skip-unless python-shell-readline-completer-delims))
      (python-shell-send-buffer)
      (python-tests-shell-wait-for-prompt)
      (goto-char (point-max))
@@ -4975,6 +4986,8 @@ python-completion-at-point-while-running-1
 time.sleep(3)
 "
    (let ((inhibit-message t))
+     (python-shell-with-shell-buffer
+       (skip-unless python-shell-readline-completer-delims))
      (python-shell-send-buffer)
      (goto-char (point-max))
      (insert "time.")
@@ -4987,6 +5000,8 @@ python-completion-at-point-native-1
 import abc
 "
    (let ((inhibit-message t))
+     (python-shell-with-shell-buffer
+       (skip-unless python-shell-readline-completer-delims))
      (python-shell-completion-native-turn-on)
      (python-shell-send-buffer)
      (python-tests-shell-wait-for-prompt)
@@ -5004,6 +5019,8 @@ python-completion-at-point-native-2
 import abc
 "
    (let ((inhibit-message t))
+     (python-shell-with-shell-buffer
+       (skip-unless python-shell-readline-completer-delims))
      (python-shell-completion-native-turn-on)
      (python-shell-send-buffer)
      (python-tests-shell-wait-for-prompt)
@@ -5020,6 +5037,8 @@ python-completion-at-point-native-with-ffap-1
 import abc
 "
    (let ((inhibit-message t))
+     (python-shell-with-shell-buffer
+       (skip-unless python-shell-readline-completer-delims))
      (python-shell-completion-native-turn-on)
      (python-shell-send-buffer)
      (python-tests-shell-wait-for-prompt)
@@ -5036,6 +5055,8 @@ python-completion-at-point-native-with-eldoc-1
 import abc
 "
    (let ((inhibit-message t))
+     (python-shell-with-shell-buffer
+       (skip-unless python-shell-readline-completer-delims))
      (python-shell-completion-native-turn-on)
      (python-shell-send-buffer)
      (python-tests-shell-wait-for-prompt)
-- 
2.25.1

bug#68559: [PATCH] Improve Python shell completion

[Basil L. Contovounesios]

Liu Hui [2024-03-13 18:21 +0800] wrote:

> On Tue, Mar 12, 2024 at 12:02 AM Basil L. Contovounesios
> <basil@contovou.net> wrote:
>>
>> Thanks!  But that makes me wonder: in the cases where we check
>>
>>   (string= python-shell-readline-completer-delims "")
>>
>> is there a possibility that the variable's value will be nil?
>> (In which case we should compare with equal instead of string=.)
>
> Yes, it is a string with native completion and may be nil with
> non-native completion.

Sorry, I completely forgot that string= works on symbols too.  I was
worried that (string= nil "") would signal an error, but it actually
does what we want.

> I have updated the patch.

Thanks, pushed:

Detect the readline support for Python shell completion
a7057745f5e 2024-03-14 15:09:56 +0100
https://git.sv.gnu.org/cgit/emacs.git/commit/?id=a7057745f5e

Is there more left to do as part of this bug report?

-- 
Basil

bug#68559: [PATCH] Improve Python shell completion

[Liu Hui]

On Thu, Mar 14, 2024 at 10:24 PM Basil L. Contovounesios
<basil@contovou.net> wrote:
>
> Liu Hui [2024-03-13 18:21 +0800] wrote:
>
> > On Tue, Mar 12, 2024 at 12:02 AM Basil L. Contovounesios
> > <basil@contovou.net> wrote:
> >>
> >> Thanks!  But that makes me wonder: in the cases where we check
> >>
> >>   (string= python-shell-readline-completer-delims "")
> >>
> >> is there a possibility that the variable's value will be nil?
> >> (In which case we should compare with equal instead of string=.)
> >
> > Yes, it is a string with native completion and may be nil with
> > non-native completion.
>
> Sorry, I completely forgot that string= works on symbols too.  I was
> worried that (string= nil "") would signal an error, but it actually
> does what we want.
>
> > I have updated the patch.
>
> Thanks, pushed:
>
> Detect the readline support for Python shell completion
> a7057745f5e 2024-03-14 15:09:56 +0100
> https://git.sv.gnu.org/cgit/emacs.git/commit/?id=a7057745f5e
>
> Is there more left to do as part of this bug report?

I think there is none.

bug#68559: [PATCH] Improve Python shell completion

[Eli Zaretskii]

> From: Liu Hui <liuhui1610@gmail.com>
> Date: Sat, 16 Mar 2024 14:49:20 +0800
> Cc: kobarity <kobarity@gmail.com>, Mattias Engdegård <mattias.engdegard@gmail.com>, 
> 	Eli Zaretskii <eliz@gnu.org>, 68559@debbugs.gnu.org
> 
> On Thu, Mar 14, 2024 at 10:24 PM Basil L. Contovounesios
> <basil@contovou.net> wrote:
> >
> > Thanks, pushed:
> >
> > Detect the readline support for Python shell completion
> > a7057745f5e 2024-03-14 15:09:56 +0100
> > https://git.sv.gnu.org/cgit/emacs.git/commit/?id=a7057745f5e
> >
> > Is there more left to do as part of this bug report?
> 
> I think there is none.

Thanks, closing.

bug#68559: [PATCH] Improve Python shell completion

[Liu Hui]

kobarity <kobarity@gmail.com> writes:

> 0001-Set-tty-mode-to-raw-when-setting-up-Inferior-Python.patch sets
> the Inferior Python tty to raw mode.  python-ffap-module-path-1 will
> no longer need to be skipped on Mac.  If it is safe to set tty to raw
> mode on all UNIX based systems, I prefer this method.

How about the following change, which only affects libedit-based
readline? It may enable native completion on mac, but I cannot test
it.

diff --git a/lisp/progmodes/python.el b/lisp/progmodes/python.el
index b7e43f3fc68..f59bc19367b 100644
--- a/lisp/progmodes/python.el
+++ b/lisp/progmodes/python.el
@@ -4286,6 +4286,9 @@ (defcustom python-shell-completion-setup-code
             except:
                 pass
         else:
+            if readline.__doc__ and 'libedit' in readline.__doc__:
+                import tty
+                tty.setraw(0)
             # Try to reuse current completer.
             completer = readline.get_completer()
             if not completer:
@@ -4471,8 +4474,8 @@ (defun python-shell-completion-native-setup ()
                 instance.rlcomplete = new_completer

         if readline.__doc__ and 'libedit' in readline.__doc__:
-            raise Exception('''libedit based readline is known not to work,
-      see etc/PROBLEMS under \"In Inferior Python mode, input is echoed\".''')
+            import tty
+            tty.setraw(0)
             readline.parse_and_bind('bind ^I rl_complete')
         else:
             readline.parse_and_bind('tab: complete')

bug#68559: [PATCH] Improve Python shell completion

[Mattias Engdegård]

[-- Attachment #1: Type: text/plain, Size: 312 bytes --]

17 feb. 2024 kl. 05.36 skrev Liu Hui <liuhui1610@gmail.com>:

> How about the following change, which only affects libedit-based
> readline? It may enable native completion on mac, but I cannot test
> it.

Thank you, but that change alone doesn't seem to suffice for python-tests to pass. Log attached.


[-- Attachment #2: python-tests.log --]
[-- Type: application/octet-stream, Size: 44268 bytes --]

  ELC      ../../emacs/test/lisp/progmodes/python-tests.elc
  GEN      lisp/progmodes/python-tests.log
Running 360 tests (2024-02-17 14:08:16+0100, selector `(not (or (tag :unstable) (tag :nativecomp)))')
   passed    1/360  python-auto-fill-docstring (0.000969 sec)
Fontifying  *temp*-195786...
Fontifying  *temp*-195786... (syntactically...)
Fontifying  *temp*-195786... (regexps...)
Fontifying  *temp*-195786... (regexps....)
Fontifying  *temp*-195786... (regexps.....)
Fontifying  *temp*-195786... (regexps......)
Fontifying  *temp*-195786... (regexps.......)
Fontifying  *temp*-195786... (regexps........)
Fontifying  *temp*-195786... (regexps.........)
Fontifying  *temp*-195786... (regexps..........)
Fontifying  *temp*-195786... (regexps...........)
Fontifying  *temp*-195786... (regexps............)
Fontifying  *temp*-195786... (regexps.............)
Fontifying  *temp*-195786... (regexps..............)
Fontifying  *temp*-195786... (regexps...............)
Fontifying  *temp*-195786... (regexps................)

   passed    2/360  python-bob-infloop-avoid (0.000757 sec)
Test python-completion-at-point-1 backtrace:
  json-parse-string("__PYTHON_EL_eval_file(\"/var/folders/qy/zstv16390
  python--parse-json-array("__PYTHON_EL_eval_file(\"/var/folders/qy/zs
  python-shell-completion-get-completions(#<process Python[ *temp*-828
  python-shell-completion-at-point(#<process Python[ *temp*-828935]>)
  python-completion-at-point()
  completion--capf-wrapper(python-completion-at-point all)
  run-hook-wrapped(completion--capf-wrapper python-completion-at-point
  completion-at-point()
  apply(completion-at-point nil)
  (setq value-3676 (apply fn-3674 args-3675))
  (unwind-protect (setq value-3676 (apply fn-3674 args-3675)) (setq fo
  (if (unwind-protect (setq value-3676 (apply fn-3674 args-3675)) (set
  (let (form-description-3678) (if (unwind-protect (setq value-3676 (a
  (let ((value-3676 'ert-form-evaluation-aborted-3677)) (let (form-des
  (let* ((fn-3674 #'completion-at-point) (args-3675 (condition-case er
  (let ((inhibit-message t)) (python-shell-send-buffer) (python-tests-
  (progn (run-python nil t) (insert "\nimport abc\n") (goto-char (poin
  (unwind-protect (progn (run-python nil t) (insert "\nimport abc\n") 
  (let ((python-indent-guess-indent-offset nil) (python-shell-completi
  (progn (let ((python-indent-guess-indent-offset nil) (python-shell-c
  (unwind-protect (progn (let ((python-indent-guess-indent-offset nil)
  (save-current-buffer (set-buffer temp-buffer) (unwind-protect (progn
  (let ((temp-buffer (generate-new-buffer " *temp*" t))) (save-current
  (closure (t) nil (let* ((fn-3669 #'executable-find) (args-3670 (cond
  #f(compiled-function () #<bytecode 0x1c50d36b54ca6edf>)()
  handler-bind-1(#f(compiled-function () #<bytecode 0x1c50d36b54ca6edf
  ert--run-test-internal(#s(ert--test-execution-info :test #s(ert-test
  ert-run-test(#s(ert-test :name python-completion-at-point-1 :documen
  ert-run-or-rerun-test(#s(ert--stats :selector ... :tests ... :test-m
  ert-run-tests((not (or (tag :unstable) (tag :nativecomp))) #f(compil
  ert-run-tests-batch((not (or (tag :unstable) (tag :nativecomp))))
  ert-run-tests-batch-and-exit((not (or (tag :unstable) (tag :nativeco
  eval((ert-run-tests-batch-and-exit '(not (or (tag :unstable) (tag :n
  command-line-1(("-L" ":../../emacs/test" "-l" "ert" "-l" "lisp/progm
  command-line()
  normal-top-level()
Test python-completion-at-point-1 condition:
    (json-parse-error "invalid token near '_'" "<string>" 1 1 1)
   FAILED    3/360  python-completion-at-point-1 (0.474468 sec) at ../../emacs/test/lisp/progmodes/python-tests.el:4903
Test python-completion-at-point-2 backtrace:
  json-parse-string("__PYTHON_EL_eval_file(\"/var/folders/qy/zstv16390
  python--parse-json-array("__PYTHON_EL_eval_file(\"/var/folders/qy/zs
  python-shell-completion-get-completions(#<process Python[ *temp*-238
  python-shell-completion-at-point(#<process Python[ *temp*-238097]>)
  python-completion-at-point()
  completion--capf-wrapper(python-completion-at-point all)
  run-hook-wrapped(completion--capf-wrapper python-completion-at-point
  completion-at-point()
  apply(completion-at-point nil)
  (setq value-3691 (apply fn-3689 args-3690))
  (unwind-protect (setq value-3691 (apply fn-3689 args-3690)) (setq fo
  (if (unwind-protect (setq value-3691 (apply fn-3689 args-3690)) (set
  (let (form-description-3693) (if (unwind-protect (setq value-3691 (a
  (let ((value-3691 'ert-form-evaluation-aborted-3692)) (let (form-des
  (let* ((fn-3689 #'completion-at-point) (args-3690 (condition-case er
  (let ((inhibit-message t)) (python-shell-send-buffer) (python-tests-
  (progn (run-python nil t) (insert "\nimport abc\n") (goto-char (poin
  (unwind-protect (progn (run-python nil t) (insert "\nimport abc\n") 
  (let ((python-indent-guess-indent-offset nil) (python-shell-completi
  (progn (let ((python-indent-guess-indent-offset nil) (python-shell-c
  (unwind-protect (progn (let ((python-indent-guess-indent-offset nil)
  (save-current-buffer (set-buffer temp-buffer) (unwind-protect (progn
  (let ((temp-buffer (generate-new-buffer " *temp*" t))) (save-current
  (closure (t) nil (let* ((fn-3684 #'executable-find) (args-3685 (cond
  #f(compiled-function () #<bytecode 0x1c50d36b54ca6edf>)()
  handler-bind-1(#f(compiled-function () #<bytecode 0x1c50d36b54ca6edf
  ert--run-test-internal(#s(ert--test-execution-info :test #s(ert-test
  ert-run-test(#s(ert-test :name python-completion-at-point-2 :documen
  ert-run-or-rerun-test(#s(ert--stats :selector ... :tests ... :test-m
  ert-run-tests((not (or (tag :unstable) (tag :nativecomp))) #f(compil
  ert-run-tests-batch((not (or (tag :unstable) (tag :nativecomp))))
  ert-run-tests-batch-and-exit((not (or (tag :unstable) (tag :nativeco
  eval((ert-run-tests-batch-and-exit '(not (or (tag :unstable) (tag :n
  command-line-1(("-L" ":../../emacs/test" "-l" "ert" "-l" "lisp/progm
  command-line()
  normal-top-level()
Test python-completion-at-point-2 condition:
    (json-parse-error "invalid token near '_'" "<string>" 1 1 1)
   FAILED    4/360  python-completion-at-point-2 (0.378594 sec) at ../../emacs/test/lisp/progmodes/python-tests.el:4918
Test python-completion-at-point-native-1 backtrace:
  signal(ert-test-failed (((should (completion-at-point)) :form (compl
  ert-fail(((should (completion-at-point)) :form (completion-at-point)
  (if (unwind-protect (setq value-3718 (apply fn-3716 args-3717)) (set
  (let (form-description-3720) (if (unwind-protect (setq value-3718 (a
  (let ((value-3718 'ert-form-evaluation-aborted-3719)) (let (form-des
  (let* ((fn-3716 #'completion-at-point) (args-3717 (condition-case er
  (let ((inhibit-message t)) (python-shell-completion-native-turn-on) 
  (progn (run-python nil t) (insert "\nimport abc\n") (goto-char (poin
  (unwind-protect (progn (run-python nil t) (insert "\nimport abc\n") 
  (let ((python-indent-guess-indent-offset nil) (python-shell-completi
  (progn (let ((python-indent-guess-indent-offset nil) (python-shell-c
  (unwind-protect (progn (let ((python-indent-guess-indent-offset nil)
  (save-current-buffer (set-buffer temp-buffer) (unwind-protect (progn
  (let ((temp-buffer (generate-new-buffer " *temp*" t))) (save-current
  (closure (t) nil (let* ((fn-3711 #'executable-find) (args-3712 (cond
  #f(compiled-function () #<bytecode 0x1c50d36b54ca6edf>)()
  handler-bind-1(#f(compiled-function () #<bytecode 0x1c50d36b54ca6edf
  ert--run-test-internal(#s(ert--test-execution-info :test #s(ert-test
  ert-run-test(#s(ert-test :name python-completion-at-point-native-1 :
  ert-run-or-rerun-test(#s(ert--stats :selector ... :tests ... :test-m
  ert-run-tests((not (or (tag :unstable) (tag :nativecomp))) #f(compil
  ert-run-tests-batch((not (or (tag :unstable) (tag :nativecomp))))
  ert-run-tests-batch-and-exit((not (or (tag :unstable) (tag :nativeco
  eval((ert-run-tests-batch-and-exit '(not (or (tag :unstable) (tag :n
  command-line-1(("-L" ":../../emacs/test" "-l" "ert" "-l" "lisp/progm
  command-line()
  normal-top-level()
Test python-completion-at-point-native-1 condition:
    (ert-test-failed
     ((should (completion-at-point)) :form (completion-at-point) :value
      nil))
   FAILED    5/360  python-completion-at-point-native-1 (3.420235 sec) at ../../emacs/test/lisp/progmodes/python-tests.el:4966
Test python-completion-at-point-native-2 backtrace:
  signal(ert-test-failed (((should (completion-at-point)) :form (compl
  ert-fail(((should (completion-at-point)) :form (completion-at-point)
  (if (unwind-protect (setq value-3733 (apply fn-3731 args-3732)) (set
  (let (form-description-3735) (if (unwind-protect (setq value-3733 (a
  (let ((value-3733 'ert-form-evaluation-aborted-3734)) (let (form-des
  (let* ((fn-3731 #'completion-at-point) (args-3732 (condition-case er
  (let ((inhibit-message t)) (python-shell-completion-native-turn-on) 
  (progn (run-python nil t) (insert "\nimport abc\n") (goto-char (poin
  (unwind-protect (progn (run-python nil t) (insert "\nimport abc\n") 
  (let ((python-indent-guess-indent-offset nil) (python-shell-completi
  (progn (let ((python-indent-guess-indent-offset nil) (python-shell-c
  (unwind-protect (progn (let ((python-indent-guess-indent-offset nil)
  (save-current-buffer (set-buffer temp-buffer) (unwind-protect (progn
  (let ((temp-buffer (generate-new-buffer " *temp*" t))) (save-current
  (closure (t) nil (let* ((fn-3726 #'executable-find) (args-3727 (cond
  #f(compiled-function () #<bytecode 0x1c50d36b54ca6edf>)()
  handler-bind-1(#f(compiled-function () #<bytecode 0x1c50d36b54ca6edf
  ert--run-test-internal(#s(ert--test-execution-info :test #s(ert-test
  ert-run-test(#s(ert-test :name python-completion-at-point-native-2 :
  ert-run-or-rerun-test(#s(ert--stats :selector ... :tests ... :test-m
  ert-run-tests((not (or (tag :unstable) (tag :nativecomp))) #f(compil
  ert-run-tests-batch((not (or (tag :unstable) (tag :nativecomp))))
  ert-run-tests-batch-and-exit((not (or (tag :unstable) (tag :nativeco
  eval((ert-run-tests-batch-and-exit '(not (or (tag :unstable) (tag :n
  command-line-1(("-L" ":../../emacs/test" "-l" "ert" "-l" "lisp/progm
  command-line()
  normal-top-level()
Test python-completion-at-point-native-2 condition:
    (ert-test-failed
     ((should (completion-at-point)) :form (completion-at-point) :value
      nil))
   FAILED    6/360  python-completion-at-point-native-2 (3.411212 sec) at ../../emacs/test/lisp/progmodes/python-tests.el:4982
Test python-completion-at-point-native-with-eldoc-1 backtrace:
  signal(ert-test-failed (((should (completion-at-point)) :form (compl
  ert-fail(((should (completion-at-point)) :form (completion-at-point)
  (if (unwind-protect (setq value-3753 (apply fn-3751 args-3752)) (set
  (let (form-description-3755) (if (unwind-protect (setq value-3753 (a
  (let ((value-3753 'ert-form-evaluation-aborted-3754)) (let (form-des
  (let* ((fn-3751 #'completion-at-point) (args-3752 (condition-case er
  (let ((inhibit-message t)) (python-shell-completion-native-turn-on) 
  (progn (run-python nil t) (insert "\nimport abc\n") (goto-char (poin
  (unwind-protect (progn (run-python nil t) (insert "\nimport abc\n") 
  (let ((python-indent-guess-indent-offset nil) (python-shell-completi
  (progn (let ((python-indent-guess-indent-offset nil) (python-shell-c
  (unwind-protect (progn (let ((python-indent-guess-indent-offset nil)
  (save-current-buffer (set-buffer temp-buffer) (unwind-protect (progn
  (let ((temp-buffer (generate-new-buffer " *temp*" t))) (save-current
  (closure (t) nil (let* ((fn-3746 #'executable-find) (args-3747 (cond
  #f(compiled-function () #<bytecode 0x1c50d36b54ca6edf>)()
  handler-bind-1(#f(compiled-function () #<bytecode 0x1c50d36b54ca6edf
  ert--run-test-internal(#s(ert--test-execution-info :test #s(ert-test
  ert-run-test(#s(ert-test :name python-completion-at-point-native-wit
  ert-run-or-rerun-test(#s(ert--stats :selector ... :tests ... :test-m
  ert-run-tests((not (or (tag :unstable) (tag :nativecomp))) #f(compil
  ert-run-tests-batch((not (or (tag :unstable) (tag :nativecomp))))
  ert-run-tests-batch-and-exit((not (or (tag :unstable) (tag :nativeco
  eval((ert-run-tests-batch-and-exit '(not (or (tag :unstable) (tag :n
  command-line-1(("-L" ":../../emacs/test" "-l" "ert" "-l" "lisp/progm
  command-line()
  normal-top-level()
Test python-completion-at-point-native-with-eldoc-1 condition:
    (ert-test-failed
     ((should (completion-at-point)) :form (completion-at-point) :value
      nil))
   FAILED    7/360  python-completion-at-point-native-with-eldoc-1 (3.472809 sec) at ../../emacs/test/lisp/progmodes/python-tests.el:5015
Test python-completion-at-point-native-with-ffap-1 backtrace:
  signal(ert-test-failed (((should (completion-at-point)) :form (compl
  ert-fail(((should (completion-at-point)) :form (completion-at-point)
  (if (unwind-protect (setq value-3743 (apply fn-3741 args-3742)) (set
  (let (form-description-3745) (if (unwind-protect (setq value-3743 (a
  (let ((value-3743 'ert-form-evaluation-aborted-3744)) (let (form-des
  (let* ((fn-3741 #'completion-at-point) (args-3742 (condition-case er
  (let ((inhibit-message t)) (python-shell-completion-native-turn-on) 
  (progn (run-python nil t) (insert "\nimport abc\n") (goto-char (poin
  (unwind-protect (progn (run-python nil t) (insert "\nimport abc\n") 
  (let ((python-indent-guess-indent-offset nil) (python-shell-completi
  (progn (let ((python-indent-guess-indent-offset nil) (python-shell-c
  (unwind-protect (progn (let ((python-indent-guess-indent-offset nil)
  (save-current-buffer (set-buffer temp-buffer) (unwind-protect (progn
  (let ((temp-buffer (generate-new-buffer " *temp*" t))) (save-current
  (closure (t) nil (let* ((fn-3736 #'executable-find) (args-3737 (cond
  #f(compiled-function () #<bytecode 0x1c50d36b54ca6edf>)()
  handler-bind-1(#f(compiled-function () #<bytecode 0x1c50d36b54ca6edf
  ert--run-test-internal(#s(ert--test-execution-info :test #s(ert-test
  ert-run-test(#s(ert-test :name python-completion-at-point-native-wit
  ert-run-or-rerun-test(#s(ert--stats :selector ... :tests ... :test-m
  ert-run-tests((not (or (tag :unstable) (tag :nativecomp))) #f(compil
  ert-run-tests-batch((not (or (tag :unstable) (tag :nativecomp))))
  ert-run-tests-batch-and-exit((not (or (tag :unstable) (tag :nativeco
  eval((ert-run-tests-batch-and-exit '(not (or (tag :unstable) (tag :n
  command-line-1(("-L" ":../../emacs/test" "-l" "ert" "-l" "lisp/progm
  command-line()
  normal-top-level()
Test python-completion-at-point-native-with-ffap-1 condition:
    (ert-test-failed
     ((should (completion-at-point)) :form (completion-at-point) :value
      nil))
   FAILED    8/360  python-completion-at-point-native-with-ffap-1 (3.473754 sec) at ../../emacs/test/lisp/progmodes/python-tests.el:4999
   passed    9/360  python-completion-at-point-pdb-1 (0.514344 sec)
   passed   10/360  python-completion-at-point-while-running-1 (0.203331 sec)
   passed   11/360  python-eldoc--get-doc-at-point-1 (0.416836 sec)
   passed   12/360  python-eldoc--get-doc-at-point-while-running-1 (0.201219 sec)
   passed   13/360  python-eldoc--get-symbol-at-point-1 (0.001408 sec)
   passed   14/360  python-eldoc--get-symbol-at-point-2 (0.001300 sec)
   passed   15/360  python-eldoc--get-symbol-at-point-3 (0.000353 sec)
   passed   16/360  python-eldoc--get-symbol-at-point-4 (0.000365 sec)
   passed   17/360  python-end-of-defun-1 (0.000403 sec)
  skipped   18/360  python-ffap-module-path-1 (0.000507 sec)
   passed   19/360  python-ffap-module-path-while-running-1 (0.204248 sec)
   passed   20/360  python-fill-docstring (0.001434 sec)
   passed   21/360  python-fill-paragraph-single-quoted-string-1 (0.000909 sec)
   passed   22/360  python-fill-paragraph-single-quoted-string-2 (0.000369 sec)
   passed   23/360  python-fill-paragraph-triple-quoted-string-1 (0.003542 sec)
   passed   24/360  python-font-lock-assignment-statement-1 (0.000618 sec)
   passed   25/360  python-font-lock-assignment-statement-10 (0.000487 sec)
   passed   26/360  python-font-lock-assignment-statement-11 (0.000953 sec)
   passed   27/360  python-font-lock-assignment-statement-12 (0.000504 sec)
   passed   28/360  python-font-lock-assignment-statement-13 (0.000774 sec)
   passed   29/360  python-font-lock-assignment-statement-14 (0.000396 sec)
   passed   30/360  python-font-lock-assignment-statement-15 (0.000363 sec)
   passed   31/360  python-font-lock-assignment-statement-16 (0.000368 sec)
   passed   32/360  python-font-lock-assignment-statement-17 (0.000348 sec)
   passed   33/360  python-font-lock-assignment-statement-18 (0.000592 sec)
   passed   34/360  python-font-lock-assignment-statement-2 (0.000334 sec)
   passed   35/360  python-font-lock-assignment-statement-3 (0.000320 sec)
   passed   36/360  python-font-lock-assignment-statement-4 (0.000805 sec)
   passed   37/360  python-font-lock-assignment-statement-5 (0.000691 sec)
   passed   38/360  python-font-lock-assignment-statement-6 (0.000625 sec)
   passed   39/360  python-font-lock-assignment-statement-7 (0.000499 sec)
   passed   40/360  python-font-lock-assignment-statement-8 (0.000449 sec)
   passed   41/360  python-font-lock-assignment-statement-9 (0.000551 sec)
   passed   42/360  python-font-lock-escape-sequence-bytes-newline (0.000447 sec)
   passed   43/360  python-font-lock-escape-sequence-hex-octal (0.000793 sec)
   passed   44/360  python-font-lock-escape-sequence-multiline-string (0.013924 sec)
   passed   45/360  python-font-lock-escape-sequence-string-newline (0.001196 sec)
   passed   46/360  python-font-lock-escape-sequence-unicode (0.000698 sec)
   passed   47/360  python-font-lock-keywords-level-1-1 (0.000317 sec)
   passed   48/360  python-font-lock-keywords-level-1-2 (0.000300 sec)
   passed   49/360  python-font-lock-operator-1 (0.000813 sec)
   passed   50/360  python-font-lock-operator-2 (0.000577 sec)
   passed   51/360  python-font-lock-raw-escape-sequence (0.001108 sec)
   passed   52/360  python-font-lock-string-literal-concatenation (0.000499 sec)
Hiding all blocks... 
Hiding all blocks...done
   passed   53/360  python-hideshow-hide-all-1 (0.000739 sec)
Hiding all blocks... 
Hiding all blocks...done
   passed   54/360  python-hideshow-hide-all-2 (0.000487 sec)
Hiding all blocks... 
Hiding all blocks...done
   passed   55/360  python-hideshow-hide-all-3 (0.000418 sec)
   passed   56/360  python-hideshow-hide-block-1 (0.000531 sec)
Hiding blocks ...
Hiding blocks ... done
Showing all blocks ...
Showing all blocks ... done
   passed   57/360  python-hideshow-hide-levels-1 (0.001552 sec)
Showing all blocks ...
Showing all blocks ... done
   passed   58/360  python-hideshow-hide-levels-2 (0.000666 sec)
Hiding blocks ...
Hiding blocks ... done
   passed   59/360  python-hideshow-hide-levels-3 (0.001000 sec)
Hiding blocks ...
Hiding blocks ... done
   passed   60/360  python-hideshow-hide-levels-4 (0.001441 sec)
   passed   61/360  python-imenu-create-flat-index-1 (0.000892 sec)
   passed   62/360  python-imenu-create-flat-index-2 (0.000358 sec)
   passed   63/360  python-imenu-create-index-1 (0.000555 sec)
   passed   64/360  python-imenu-create-index-2 (0.000344 sec)
   passed   65/360  python-imenu-create-index-3 (0.000331 sec)
   passed   66/360  python-imenu-create-index-4 (0.000350 sec)
   passed   67/360  python-indent-after-async-block-1 (0.000515 sec)
   passed   68/360  python-indent-after-async-block-2 (0.000386 sec)
   passed   69/360  python-indent-after-async-block-3 (0.000364 sec)
   passed   70/360  python-indent-after-backslash-1 (0.000792 sec)
   passed   71/360  python-indent-after-backslash-2 (0.001414 sec)
   passed   72/360  python-indent-after-backslash-3 (0.000629 sec)
   passed   73/360  python-indent-after-backslash-4 (0.000913 sec)
   passed   74/360  python-indent-after-backslash-5 (0.000501 sec)
   passed   75/360  python-indent-after-backslash-6 (0.000897 sec)
   passed   76/360  python-indent-after-bare-match (0.000608 sec)
   passed   77/360  python-indent-after-block-1 (0.000547 sec)
   passed   78/360  python-indent-after-block-2 (0.000419 sec)
   passed   79/360  python-indent-after-block-3 (0.000526 sec)
   passed   80/360  python-indent-after-case-block (0.000359 sec)
   passed   81/360  python-indent-after-comment-1 (0.000820 sec)
   passed   82/360  python-indent-after-comment-2 (0.001027 sec)
   passed   83/360  python-indent-after-comment-3 (0.000379 sec)
   passed   84/360  python-indent-after-match-block (0.000493 sec)
   passed   85/360  python-indent-after-re-match (0.000623 sec)
   passed   86/360  python-indent-base-case (0.000440 sec)
   passed   87/360  python-indent-block-enders-1 (0.000678 sec)
   passed   88/360  python-indent-block-enders-2 (0.000862 sec)
   passed   89/360  python-indent-block-enders-3 (0.000453 sec)
   passed   90/360  python-indent-block-enders-4 (0.000454 sec)
   passed   91/360  python-indent-block-enders-5 (0.000614 sec)
   passed   92/360  python-indent-dedent-line-backspace-1 (0.000428 sec)
   passed   93/360  python-indent-dedent-line-backspace-2 (0.000259 sec)
   passed   94/360  python-indent-dedent-line-backspace-3 (0.000464 sec)
   passed   95/360  python-indent-dedenters-1 (0.000417 sec)
Closes if hide_details:
Closes except Exception:
Closes if save:
   passed   96/360  python-indent-dedenters-2 (0.001706 sec)
Closes try:
   passed   97/360  python-indent-dedenters-3 (0.000751 sec)
Closes try:
   passed   98/360  python-indent-dedenters-4 (0.000825 sec)
Closes if save:
   passed   99/360  python-indent-dedenters-5 (0.001327 sec)
   passed  100/360  python-indent-dedenters-6 (0.001020 sec)
   passed  101/360  python-indent-dedenters-7 (0.000662 sec)
Closes if (a == 1 or
Closes if (a == 1 or
Closes if (a == 1 or
   passed  102/360  python-indent-dedenters-8 (0.001472 sec)
Closes case 1:
   passed  103/360  python-indent-dedenters-9 (0.000582 sec)
Closes if hide_details:
Closes except Exception:
Closes if save:
   passed  104/360  python-indent-dedenters-comment-else (0.003784 sec)
   passed  105/360  python-indent-electric-colon-1 (0.000420 sec)
Closes if do:
   passed  106/360  python-indent-electric-colon-2 (0.000480 sec)
Closes if do:
Closes if do:
Closes if do:
   passed  107/360  python-indent-electric-colon-3 (0.000724 sec)
Closes if True:
   passed  108/360  python-indent-electric-colon-4 (0.000562 sec)
   passed  109/360  python-indent-electric-comma-after-multiline-string (0.000467 sec)
   passed  110/360  python-indent-electric-comma-inside-multiline-string (0.000702 sec)
   passed  111/360  python-indent-hanging-close-paren (0.000450 sec)
   passed  112/360  python-indent-inside-paren-1 (0.001556 sec)
   passed  113/360  python-indent-inside-paren-2 (0.000875 sec)
   passed  114/360  python-indent-inside-paren-3 (0.000436 sec)
   passed  115/360  python-indent-inside-paren-4 (0.000375 sec)
   passed  116/360  python-indent-inside-paren-5 (0.000495 sec)
   passed  117/360  python-indent-inside-paren-6 (0.001114 sec)
   passed  118/360  python-indent-inside-paren-7 (0.000481 sec)
   passed  119/360  python-indent-inside-paren-8 (0.000563 sec)
   passed  120/360  python-indent-inside-paren-9 (0.000931 sec)
   passed  121/360  python-indent-inside-paren-block-1 (0.000826 sec)
   passed  122/360  python-indent-inside-paren-block-2 (0.000831 sec)
   passed  123/360  python-indent-inside-paren-block-3 (0.000598 sec)
   passed  124/360  python-indent-inside-paren-block-4 (0.000804 sec)
   passed  125/360  python-indent-inside-string-1 (0.000824 sec)
   passed  126/360  python-indent-inside-string-2 (0.002007 sec)
   passed  127/360  python-indent-inside-string-3 (0.001548 sec)
   passed  128/360  python-indent-pep8-1 (0.000829 sec)
   passed  129/360  python-indent-pep8-2 (0.000796 sec)
   passed  130/360  python-indent-pep8-3 (0.000469 sec)
   passed  131/360  python-indent-region-1 (0.000342 sec)
   passed  132/360  python-indent-region-2 (0.000414 sec)
   passed  133/360  python-indent-region-3 (0.000983 sec)
   passed  134/360  python-indent-region-4 (0.000713 sec)
   passed  135/360  python-indent-region-5 (0.002212 sec)
   passed  136/360  python-info-assignment-continuation-line-p-1 (0.000397 sec)
   passed  137/360  python-info-assignment-continuation-line-p-2 (0.000438 sec)
   passed  138/360  python-info-assignment-statement-p-1 (0.000405 sec)
   passed  139/360  python-info-assignment-statement-p-2 (0.000421 sec)
   passed  140/360  python-info-assignment-statement-p-3 (0.000556 sec)
   passed  141/360  python-info-beginning-of-backslash-1 (0.000660 sec)
   passed  142/360  python-info-beginning-of-block-p-1 (0.000477 sec)
   passed  143/360  python-info-beginning-of-block-p-2 (0.000649 sec)
   passed  144/360  python-info-beginning-of-statement-p-1 (0.000376 sec)
   passed  145/360  python-info-beginning-of-statement-p-2 (0.000333 sec)
   passed  146/360  python-info-block-continuation-line-p-1 (0.000364 sec)
   passed  147/360  python-info-block-continuation-line-p-2 (0.000758 sec)
   passed  148/360  python-info-continuation-line-p-1 (0.000763 sec)
   passed  149/360  python-info-current-defun-1 (0.001010 sec)
   passed  150/360  python-info-current-defun-2 (0.012160 sec)
   passed  151/360  python-info-current-defun-3 (0.009767 sec)
   passed  152/360  python-info-current-defun-4 (0.000514 sec)
   passed  153/360  python-info-current-line-comment-p-1 (0.000302 sec)
   passed  154/360  python-info-current-line-empty-p (0.000253 sec)
   passed  155/360  python-info-current-symbol-1 (0.000486 sec)
   passed  156/360  python-info-current-symbol-2 (0.000581 sec)
   failed  157/360  python-info-current-symbol-3 (0.000336 sec)
   passed  158/360  python-info-dedenter-opening-block-message-1 (0.000273 sec)
Closes try:
Closes try:
   passed  159/360  python-info-dedenter-opening-block-message-2 (0.000333 sec)
Closes except:
Closes except:
   passed  160/360  python-info-dedenter-opening-block-message-3 (0.000334 sec)
Closes else:
Closes else:
   passed  161/360  python-info-dedenter-opening-block-message-4 (0.000800 sec)
Closes if a:
Closes if a:
   passed  162/360  python-info-dedenter-opening-block-message-5 (0.000618 sec)
   passed  163/360  python-info-dedenter-opening-block-position-1 (0.001092 sec)
   passed  164/360  python-info-dedenter-opening-block-position-2 (0.000352 sec)
   passed  165/360  python-info-dedenter-opening-block-position-3 (0.001099 sec)
   passed  166/360  python-info-dedenter-opening-block-positions-1 (0.001124 sec)
   passed  167/360  python-info-dedenter-opening-block-positions-2 (0.000468 sec)
   passed  168/360  python-info-dedenter-opening-block-positions-3 (0.000941 sec)
   passed  169/360  python-info-dedenter-opening-block-positions-4 (0.000519 sec)
   passed  170/360  python-info-dedenter-opening-block-positions-5 (0.000737 sec)
   passed  171/360  python-info-dedenter-opening-block-positions-6 (0.000408 sec)
   passed  172/360  python-info-dedenter-opening-block-positions-7 (0.000446 sec)
   passed  173/360  python-info-dedenter-statement-p-1 (0.000661 sec)
   passed  174/360  python-info-dedenter-statement-p-2 (0.000527 sec)
   passed  175/360  python-info-dedenter-statement-p-3 (0.000499 sec)
   passed  176/360  python-info-dedenter-statement-p-4 (0.000499 sec)
   passed  177/360  python-info-dedenter-statement-p-5 (0.000326 sec)
   passed  178/360  python-info-dedenter-statement-p-6 (0.000385 sec)
   passed  179/360  python-info-docstring-p-1 (0.001139 sec)
   passed  180/360  python-info-docstring-p-2 (0.001632 sec)
   passed  181/360  python-info-docstring-p-3 (0.001860 sec)
   passed  182/360  python-info-docstring-p-4 (0.001932 sec)
   passed  183/360  python-info-docstring-p-5 (0.002134 sec)
   passed  184/360  python-info-docstring-p-6 (0.001738 sec)
   passed  185/360  python-info-docstring-p-7 (0.000527 sec)
   passed  186/360  python-info-docstring-p-8 (0.000430 sec)
   passed  187/360  python-info-encoding-1 (0.000300 sec)
   passed  188/360  python-info-encoding-2 (0.000258 sec)
   passed  189/360  python-info-encoding-from-cookie-1 (0.000565 sec)
   passed  190/360  python-info-encoding-from-cookie-2 (0.000425 sec)
   passed  191/360  python-info-encoding-from-cookie-3 (0.000398 sec)
   passed  192/360  python-info-encoding-from-cookie-4 (0.000409 sec)
   passed  193/360  python-info-encoding-from-cookie-5 (0.000387 sec)
   passed  194/360  python-info-encoding-from-cookie-6 (0.000267 sec)
   passed  195/360  python-info-encoding-from-cookie-7 (0.000261 sec)
   passed  196/360  python-info-end-of-block-p-1 (0.000599 sec)
   passed  197/360  python-info-end-of-block-p-2 (0.000709 sec)
   passed  198/360  python-info-end-of-statement-p-1 (0.000414 sec)
   passed  199/360  python-info-end-of-statement-p-2 (0.000647 sec)
   passed  200/360  python-info-line-ends-backslash-p-1 (0.000586 sec)
   passed  201/360  python-info-looking-at-beginning-of-block-1 (0.000623 sec)
   passed  202/360  python-info-looking-at-beginning-of-defun-1 (0.000662 sec)
   passed  203/360  python-info-looking-at-beginning-of-defun-2 (0.000400 sec)
   passed  204/360  python-info-looking-at-beginning-of-defun-3 (0.000700 sec)
   passed  205/360  python-info-statement-ends-block-p-1 (0.039753 sec)
   passed  206/360  python-info-statement-ends-block-p-2 (0.000398 sec)
   passed  207/360  python-info-statement-starts-block-p-1 (0.000300 sec)
   passed  208/360  python-info-statement-starts-block-p-2 (0.000300 sec)
   passed  209/360  python-info-triple-quoted-string-p-1 (0.000406 sec)
   passed  210/360  python-info-triple-quoted-string-p-2 (0.000312 sec)
   passed  211/360  python-info-triple-quoted-string-p-3 (0.000388 sec)
Mark set
Mark set
   passed  212/360  python-mark-defun-1 (0.000850 sec)
Mark set
Mark set
   passed  213/360  python-mark-defun-2 (0.000827 sec)
Mark set
Mark set
   passed  214/360  python-mark-defun-3 (0.000485 sec)
Mark set
Mark set
   passed  215/360  python-mark-defun-4 (0.000526 sec)
Mark set
Mark set
Mark set
Mark set
   passed  216/360  python-mark-defun-5 (0.000672 sec)
   passed  217/360  python-nav-backward-defun-1 (0.000751 sec)
   passed  218/360  python-nav-backward-defun-2 (0.000590 sec)
   passed  219/360  python-nav-backward-defun-3 (0.000499 sec)
   passed  220/360  python-nav-backward-defun-4 (0.000459 sec)
   passed  221/360  python-nav-backward-statement-1 (0.000446 sec)
   failed  222/360  python-nav-backward-statement-2 (0.000393 sec)
   failed  223/360  python-nav-backward-up-list-1 (0.000436 sec)
   passed  224/360  python-nav-beginning-of-block-1 (0.000835 sec)
   passed  225/360  python-nav-beginning-of-block-2 (0.000391 sec)
   passed  226/360  python-nav-beginning-of-defun-1 (0.001373 sec)
   passed  227/360  python-nav-beginning-of-defun-2 (0.001348 sec)
   passed  228/360  python-nav-beginning-of-defun-3 (0.000396 sec)
   passed  229/360  python-nav-beginning-of-defun-4 (0.000531 sec)
   passed  230/360  python-nav-beginning-of-defun-5 (0.000403 sec)
   passed  231/360  python-nav-beginning-of-defun-6 (0.000319 sec)
   passed  232/360  python-nav-beginning-of-statement-1 (0.000428 sec)
   passed  233/360  python-nav-end-of-block-1 (0.002600 sec)
   passed  234/360  python-nav-end-of-block-2 (0.000368 sec)
   passed  235/360  python-nav-end-of-defun-1 (0.001172 sec)
   passed  236/360  python-nav-end-of-defun-2 (0.003453 sec)
   passed  237/360  python-nav-end-of-defun-3 (0.000532 sec)
   passed  238/360  python-nav-end-of-statement-1 (0.000500 sec)
   passed  239/360  python-nav-end-of-statement-2 (0.000266 sec)
   passed  240/360  python-nav-end-of-statement-3 (0.000272 sec)
   passed  241/360  python-nav-end-of-statement-4 (0.000298 sec)
   passed  242/360  python-nav-forward-block-1 (0.000777 sec)
   passed  243/360  python-nav-forward-block-2 (0.000327 sec)
   passed  244/360  python-nav-forward-defun-1 (0.000395 sec)
   passed  245/360  python-nav-forward-defun-2 (0.000358 sec)
   passed  246/360  python-nav-forward-defun-3 (0.000304 sec)
   passed  247/360  python-nav-forward-defun-4 (0.000280 sec)
   passed  248/360  python-nav-forward-sexp-1 (0.000837 sec)
   passed  249/360  python-nav-forward-sexp-2 (0.001896 sec)
   passed  250/360  python-nav-forward-sexp-3 (0.001493 sec)
   passed  251/360  python-nav-forward-sexp-safe-1 (0.001036 sec)
   passed  252/360  python-nav-forward-statement-1 (0.000857 sec)
   passed  253/360  python-nav-up-list-1 (0.000392 sec)
   passed  254/360  python-parens-electric-indent-1 (0.001728 sec)
   passed  255/360  python-shell-buffer-substring-1 (0.000598 sec)
   passed  256/360  python-shell-buffer-substring-10 (0.000495 sec)
   passed  257/360  python-shell-buffer-substring-11 (0.000924 sec)
   passed  258/360  python-shell-buffer-substring-12 (0.000810 sec)
   passed  259/360  python-shell-buffer-substring-13 (0.000809 sec)
   passed  260/360  python-shell-buffer-substring-14 (0.000738 sec)
   passed  261/360  python-shell-buffer-substring-15 (0.000437 sec)
   passed  262/360  python-shell-buffer-substring-16 (0.000421 sec)
   passed  263/360  python-shell-buffer-substring-17 (0.000428 sec)
   passed  264/360  python-shell-buffer-substring-18 (0.000412 sec)
   passed  265/360  python-shell-buffer-substring-2 (0.001147 sec)
   passed  266/360  python-shell-buffer-substring-3 (0.000990 sec)
   passed  267/360  python-shell-buffer-substring-4 (0.000597 sec)
   passed  268/360  python-shell-buffer-substring-5 (0.000584 sec)
   passed  269/360  python-shell-buffer-substring-6 (0.000504 sec)
   passed  270/360  python-shell-buffer-substring-7 (0.000501 sec)
   passed  271/360  python-shell-buffer-substring-8 (0.000493 sec)
   passed  272/360  python-shell-buffer-substring-9 (0.000786 sec)
   passed  273/360  python-shell-calculate-exec-path-1 (0.000152 sec)
   passed  274/360  python-shell-calculate-exec-path-2 (0.000104 sec)
   passed  275/360  python-shell-calculate-exec-path-3 (0.000099 sec)
   passed  276/360  python-shell-calculate-exec-path-4 (0.002369 sec)
   passed  277/360  python-shell-calculate-exec-path-5 (0.000091 sec)
   passed  278/360  python-shell-calculate-exec-path-6 (0.000151 sec)
   passed  279/360  python-shell-calculate-process-environment-1 (0.000071 sec)
   passed  280/360  python-shell-calculate-process-environment-2 (0.000241 sec)
   passed  281/360  python-shell-calculate-process-environment-3 (0.000165 sec)
   passed  282/360  python-shell-calculate-process-environment-4 (0.000124 sec)
   passed  283/360  python-shell-calculate-process-environment-5 (0.000116 sec)
   passed  284/360  python-shell-calculate-process-environment-6 (0.000116 sec)
   passed  285/360  python-shell-calculate-process-environment-7 (0.000129 sec)
   passed  286/360  python-shell-calculate-process-environment-8 (0.000124 sec)
   passed  287/360  python-shell-calculate-pythonpath-1 (0.000114 sec)
   passed  288/360  python-shell-calculate-pythonpath-2 (0.000117 sec)
Test python-shell-completion-at-point-1 backtrace:
  signal(json-parse-error ("invalid token near '_'" "<string>" 1 1 1))
  apply(signal (json-parse-error ("invalid token near '_'" "<string>" 
  (setq value-3589 (apply fn-3587 args-3588))
  (unwind-protect (setq value-3589 (apply fn-3587 args-3588)) (setq fo
  (if (unwind-protect (setq value-3589 (apply fn-3587 args-3588)) (set
  (let (form-description-3591) (if (unwind-protect (setq value-3589 (a
  (let ((value-3589 'ert-form-evaluation-aborted-3590)) (let (form-des
  (let* ((fn-3587 #'nth) (args-3588 (condition-case err (list 2 (pytho
  (save-current-buffer (set-buffer (process-buffer shell-process)) (in
  (let ((shell-process (python-shell-get-process-or-error))) (save-cur
  (progn (run-python nil t) (insert "") (goto-char (point-min)) (pytho
  (unwind-protect (progn (run-python nil t) (insert "") (goto-char (po
  (let ((python-indent-guess-indent-offset nil) (python-shell-completi
  (progn (let ((python-indent-guess-indent-offset nil) (python-shell-c
  (unwind-protect (progn (let ((python-indent-guess-indent-offset nil)
  (save-current-buffer (set-buffer temp-buffer) (unwind-protect (progn
  (let ((temp-buffer (generate-new-buffer " *temp*" t))) (save-current
  (closure (t) nil (let* ((fn-3582 #'executable-find) (args-3583 (cond
  #f(compiled-function () #<bytecode 0x1c50d36b54ca6edf>)()
  handler-bind-1(#f(compiled-function () #<bytecode 0x1c50d36b54ca6edf
  ert--run-test-internal(#s(ert--test-execution-info :test #s(ert-test
  ert-run-test(#s(ert-test :name python-shell-completion-at-point-1 :d
  ert-run-or-rerun-test(#s(ert--stats :selector ... :tests ... :test-m
  ert-run-tests((not (or (tag :unstable) (tag :nativecomp))) #f(compil
  ert-run-tests-batch((not (or (tag :unstable) (tag :nativecomp))))
  ert-run-tests-batch-and-exit((not (or (tag :unstable) (tag :nativeco
  eval((ert-run-tests-batch-and-exit '(not (or (tag :unstable) (tag :n
  command-line-1(("-L" ":../../emacs/test" "-l" "ert" "-l" "lisp/progm
  command-line()
  normal-top-level()
Test python-shell-completion-at-point-1 condition:
    (json-parse-error "invalid token near '_'" "<string>" 1 1 1)
   FAILED  289/360  python-shell-completion-at-point-1 (0.389104 sec) at ../../emacs/test/lisp/progmodes/python-tests.el:4777
  skipped  290/360  python-shell-completion-at-point-ipython (0.000458 sec)
Warning (python): Your `python-shell-interpreter' doesn't seem to support readline, yet `python-shell-completion-native-enable' was t and "python3" is not part of the `python-shell-completion-native-disabled-interpreters' list.  Native completions have been disabled locally. Consider installing the python package "readline". 
  skipped  291/360  python-shell-completion-at-point-jedi-completer (1.283608 sec)
Warning (python): Your `python-shell-interpreter' doesn't seem to support readline, yet `python-shell-completion-native-enable' was t and "python3" is not part of the `python-shell-completion-native-disabled-interpreters' list.  Native completions have been disabled locally. Consider installing the python package "readline". 
   passed  292/360  python-shell-completion-at-point-native-1 (1.418923 sec)
   passed  293/360  python-shell-completion-native-interpreter-disabled-p-1 (0.000124 sec)
Can't guess python-indent-offset, using defaults: 4
   passed  294/360  python-shell-get-process-1 (0.215481 sec)
   passed  295/360  python-shell-get-process-name-1 (0.000597 sec)
Can't guess python-indent-offset, using defaults: 4
   passed  296/360  python-shell-get-process-name-2 (0.011209 sec)
Can't guess python-indent-offset, using defaults: 4
   passed  297/360  python-shell-internal-get-or-create-process-1 (0.063736 sec)
   passed  298/360  python-shell-internal-get-process-name-1 (0.000444 sec)
Can't guess python-indent-offset, using defaults: 4
   passed  299/360  python-shell-internal-get-process-name-2 (0.011409 sec)
   passed  300/360  python-shell-make-comint-1 (0.052506 sec)
   passed  301/360  python-shell-make-comint-2 (0.053577 sec)
   passed  302/360  python-shell-make-comint-3 (0.050405 sec)
   passed  303/360  python-shell-make-comint-4 (0.050541 sec)
   passed  304/360  python-shell-prompt-detect-1 (0.077438 sec)
   passed  305/360  python-shell-prompt-detect-2 (0.075888 sec)
   passed  306/360  python-shell-prompt-detect-3 (0.000279 sec)
Warning (python): Python shell prompts cannot be detected.
If your emacs session hangs when starting python shells
recover with `keyboard-quit' and then try fixing the
interactive flag for your interpreter by adjusting the
`python-shell-interpreter-interactive-arg' or add regexps
matching shell prompts in the directory-local friendly vars:
  + `python-shell-prompt-regexp'
  + `python-shell-prompt-block-regexp'
  + `python-shell-prompt-output-regexp'
Or alternatively in:
  + `python-shell-prompt-input-regexps'
  + `python-shell-prompt-output-regexps'
   passed  307/360  python-shell-prompt-detect-4 (0.082467 sec)
   passed  308/360  python-shell-prompt-detect-5 (0.081498 sec)
   passed  309/360  python-shell-prompt-detect-6 (0.001290 sec)
   passed  310/360  python-shell-prompt-set-calculated-regexps-1 (0.000174 sec)
   passed  311/360  python-shell-prompt-set-calculated-regexps-2 (0.000122 sec)
   passed  312/360  python-shell-prompt-set-calculated-regexps-3 (0.000201 sec)
   passed  313/360  python-shell-prompt-set-calculated-regexps-4 (0.000169 sec)
   passed  314/360  python-shell-prompt-set-calculated-regexps-5 (0.000183 sec)
   passed  315/360  python-shell-prompt-set-calculated-regexps-6 (0.116206 sec)
   passed  316/360  python-shell-prompt-validate-regexps-1 (0.000154 sec)
   passed  317/360  python-shell-prompt-validate-regexps-2 (0.000154 sec)
   passed  318/360  python-shell-prompt-validate-regexps-3 (0.000152 sec)
   passed  319/360  python-shell-prompt-validate-regexps-4 (0.000115 sec)
   passed  320/360  python-shell-prompt-validate-regexps-5 (0.000103 sec)
   passed  321/360  python-shell-prompt-validate-regexps-6 (0.000104 sec)
   passed  322/360  python-shell-prompt-validate-regexps-7 (0.000063 sec)
   passed  323/360  python-shell-with-environment-1 (0.000364 sec)
   passed  324/360  python-shell-with-environment-2 (0.000558 sec)
   passed  325/360  python-shell-with-environment-3 (0.000565 sec)
   passed  326/360  python-syntax-after-python-backspace (0.000334 sec)
   passed  327/360  python-syntax-context-1 (0.000354 sec)
   passed  328/360  python-tests--fill-long-first-line (0.000998 sec)
   passed  329/360  python-tests--flymake-command-output-pattern (0.000234 sec)
   passed  330/360  python-tests--run-python-selects-window (0.096141 sec)
   passed  331/360  python-tests-look-at-1 (0.000453 sec)
   passed  332/360  python-tests-look-at-2 (0.000298 sec)
   passed  333/360  python-triple-double-quote-pairing (0.004338 sec)
   passed  334/360  python-triple-single-quote-pairing (0.001394 sec)
   passed  335/360  python-ts-mode-assignement-face-2 (0.051551 sec)
   passed  336/360  python-ts-mode-builtin-call-face (0.018900 sec)
   passed  337/360  python-ts-mode-class-patterns-face (0.002075 sec)
   passed  338/360  python-ts-mode-compound-keywords-face (0.004958 sec)
   passed  339/360  python-ts-mode-disabled-string-interpolation (0.002239 sec)
   passed  340/360  python-ts-mode-dotted-decorator-face-1 (0.002431 sec)
   passed  341/360  python-ts-mode-dotted-decorator-face-2 (0.002075 sec)
   passed  342/360  python-ts-mode-interpolation-doc-string (0.001603 sec)
   passed  343/360  python-ts-mode-interpolation-nested-string (0.001576 sec)
   passed  344/360  python-ts-mode-isinstance-type-face-1 (0.001774 sec)
   passed  345/360  python-ts-mode-isinstance-type-face-2 (0.003366 sec)
   passed  346/360  python-ts-mode-isinstance-type-face-3 (0.002596 sec)
   passed  347/360  python-ts-mode-level-fontification-wo-interpolation (0.001768 sec)
   passed  348/360  python-ts-mode-named-assignement-face-1 (0.001438 sec)
   passed  349/360  python-ts-mode-nested-types-face-1 (0.001659 sec)
   passed  350/360  python-ts-mode-superclass-type-face (0.001565 sec)
   passed  351/360  python-ts-mode-types-face-1 (0.001577 sec)
   passed  352/360  python-ts-mode-types-face-2 (0.003078 sec)
   passed  353/360  python-ts-mode-types-face-3 (0.001907 sec)
   passed  354/360  python-ts-mode-union-types-face-1 (0.002662 sec)
   passed  355/360  python-ts-mode-union-types-face-2 (0.002332 sec)
   passed  356/360  python-util-clone-local-variables-1 (0.000391 sec)
   passed  357/360  python-util-forward-comment-1 (0.001119 sec)
   passed  358/360  python-util-goto-line-1 (0.000271 sec)
   passed  359/360  python-util-strip-string-1 (0.000134 sec)
   passed  360/360  python-util-valid-regexp-p-1 (0.000088 sec)

Ran 360 tests, 350 results as expected, 7 unexpected, 3 skipped (2024-02-17 14:08:39+0100, 23.058396 sec)
3 expected failures

7 unexpected results:
   FAILED  python-completion-at-point-1
   FAILED  python-completion-at-point-2
   FAILED  python-completion-at-point-native-1
   FAILED  python-completion-at-point-native-2
   FAILED  python-completion-at-point-native-with-eldoc-1
   FAILED  python-completion-at-point-native-with-ffap-1
   FAILED  python-shell-completion-at-point-1

3 skipped results:
  SKIPPED  python-ffap-module-path-1
  SKIPPED  python-shell-completion-at-point-ipython
  SKIPPED  python-shell-completion-at-point-jedi-completer