bug#50459: 28.0.50; [PATCH] Python shell completion is incompatible with flex, orderless, etc.

[Augusto Stoffel <arstoffel@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 1414 bytes --]

Okay, I've attached a patch which seems to work fine.

On Fri, 10 Sep 2021 at 16:32, Dmitry Gutov <dgutov@yandex.ru> wrote:

> On 09.09.2021 19:46, Augusto Stoffel wrote:
>> To alleviate this, the completion-at-point function could implement some
>> sort of caching.  The difficult question is when to invalidate the
>> cache.  I've attached one possiblility as a draft patch.  If the
>> approach seems reasonable, then I'll format it properly.
>
> It's much simpler than what we talked about, but given that
> python-shell-completion-at-point doesn't really look past the current
> line, your approach should work fine.

Yes, and you're right --- I've did it that way because this
comint-based completion basically only works for globals anyway.

>
> I'm not very familiar with the code, so I cannot approve the exact
> patch, though, sorry.
>
> As a bonus, though, maybe add the position of prompt on the
> shell-buffer to the invalidation key? Like, if a user imports some new
> lib in there, that can bring in new completions.

Good idea.  I've added this to the patch.  It still won't show the new
completions until the user evaluates the import, but this is the best
you can get by querying an interpreter for the completions.

I've also changed a bit the way the "native completion" setup code is
sent, so that it doesn't print a message directly in the shell buffer
(there is still an echo area message).


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Implement-caching-for-python-shell-completion-at-poi.patch --]
[-- Type: text/x-patch, Size: 5953 bytes --]

From 3f27a63fe5be884464a6a68407e0b6990186c4a6 Mon Sep 17 00:00:00 2001
From: Augusto Stoffel <arstoffel@gmail.com>
Date: Fri, 10 Sep 2021 20:44:10 +0200
Subject: [PATCH] Implement caching for 'python-shell-completion-at-point'

* lisp/progmodes/python.el (python-shell-completion-at-point): cache
results, since computing them involves talking with the inferior
process and, potentially, network communications
(python-shell--capf-cache): new variable, for cache
(python-shell-completion-get-completions,
python-shell-completion-native-get-completions): 'import' argument is
not needed anymore.
(python-shell-completion-native-setup,
python-shell-completion-native-try): pass the setup code
synchronously, to avoid printing a message in the shell.
---
 lisp/progmodes/python.el | 59 ++++++++++++++++++++--------------------
 1 file changed, 30 insertions(+), 29 deletions(-)

diff --git a/lisp/progmodes/python.el b/lisp/progmodes/python.el
index 2eef52de0c..e71a8102df 100644
--- a/lisp/progmodes/python.el
+++ b/lisp/progmodes/python.el
@@ -3577,13 +3577,12 @@ python-shell-completion-native-try
          python-shell-completion-native-try-output-timeout))
     (python-shell-completion-native-get-completions
      (get-buffer-process (current-buffer))
-     nil "_")))
+     "_")))
 
 (defun python-shell-completion-native-setup ()
   "Try to setup native completion, return non-nil on success."
-  (let ((process (python-shell-get-process)))
-    (with-current-buffer (process-buffer process)
-      (python-shell-send-string "
+  (let* ((process (python-shell-get-process))
+         (output (python-shell-send-string-no-output "
 def __PYTHON_EL_native_completion_setup():
     try:
         import readline
@@ -3693,14 +3692,10 @@ python-shell-completion-native-setup
         print ('python.el: native completion setup failed, %s: %s'
                % sys.exc_info()[:2])
 
-__PYTHON_EL_native_completion_setup()" process)
-      (when (and
-             (python-shell-accept-process-output
-              process python-shell-completion-native-try-output-timeout)
-             (save-excursion
-               (re-search-backward
-                (regexp-quote "python.el: native completion setup loaded") nil t 1)))
-        (python-shell-completion-native-try)))))
+__PYTHON_EL_native_completion_setup()" process)))
+    (when (string-match-p "python\\.el: native completion setup loaded"
+                          output)
+      (python-shell-completion-native-try))))
 
 (defun python-shell-completion-native-turn-off (&optional msg)
   "Turn off shell native completions.
@@ -3760,13 +3755,10 @@ python-shell-completion-native-toggle
       (python-shell-completion-native-turn-on msg))
     python-shell-completion-native-enable))
 
-(defun python-shell-completion-native-get-completions (process import input)
-  "Get completions using native readline for PROCESS.
-When IMPORT is non-nil takes precedence over INPUT for
-completion."
+(defun python-shell-completion-native-get-completions (process input)
+  "Get completions of INPUT using native readline for PROCESS."
   (with-current-buffer (process-buffer process)
-    (let* ((input (or import input))
-           (original-filter-fn (process-filter process))
+    (let* ((original-filter-fn (process-filter process))
            (redirect-buffer (get-buffer-create
                              python-shell-completion-native-redirect-buffer))
            (trigger "\t")
@@ -3818,11 +3810,8 @@ python-shell-completion-native-get-completions
                  :test #'string=))))
         (set-process-filter process original-filter-fn)))))
 
-(defun python-shell-completion-get-completions (process import input)
-  "Do completion at point using PROCESS for IMPORT or INPUT.
-When IMPORT is non-nil takes precedence over INPUT for
-completion."
-  (setq input (or import input))
+(defun python-shell-completion-get-completions (process input)
+  "Get completions of INPUT using PROCESS."
   (with-current-buffer (process-buffer process)
     (let ((completions
            (python-util-strip-string
@@ -3836,6 +3825,9 @@ python-shell-completion-get-completions
         (split-string completions
                       "^'\\|^\"\\|;\\|'$\\|\"$" t)))))
 
+(defvar-local python-shell--capf-cache nil
+  "Variable to store cached completions and invalidation keys.")
+
 (defun python-shell-completion-at-point (&optional process)
   "Function for `completion-at-point-functions' in `inferior-python-mode'.
 Optional argument PROCESS forces completions to be retrieved
@@ -3889,12 +3881,21 @@ python-shell-completion-at-point
                        ;; it during a multiline statement (Bug#28051).
                        #'ignore
                      #'python-shell-completion-get-completions))
-                  (t #'python-shell-completion-native-get-completions)))))
-    (list start end
-          (completion-table-dynamic
-           (apply-partially
-            completion-fn
-            process import-statement)))))
+                  (t #'python-shell-completion-native-get-completions))))
+         (prev-prompt (car python-shell--capf-cache))
+         (re (or (cadr python-shell--capf-cache) regexp-unmatchable))
+         (prefix (buffer-substring-no-properties start end)))
+    ;; To invalidate the cache, we check if the prompt position or the
+    ;; completion prefix changed.
+    (unless (and (equal prev-prompt (car prompt-boundaries))
+                 (string-match re prefix))
+      (setq python-shell--capf-cache
+            `(,(car prompt-boundaries)
+              ,(if (string-empty-p prefix)
+                   regexp-unmatchable
+                 (concat "\\`" (regexp-quote prefix) "\\(?:\\sw\\|\\s_\\)*\\'"))
+              ,@(funcall completion-fn process (or import-statement prefix)))))
+    (list start end (cddr python-shell--capf-cache))))
 
 (define-obsolete-function-alias
   'python-shell-completion-complete-at-point
-- 
2.31.1