bug#68559: [PATCH] Improve Python shell completion

[kobarity <kobarity@gmail.com>]

Liu Hui wrote:
> kobarity <kobarity@gmail.com> writes:
> > I'm experiencing strange behavior regarding completion of import
> > statement in a block in Python buffer.  If I try to type the following
> > lines and then try to complete it, it will fail.
> >
> > #+begin_src python
> > try:
> >     im
> > #+end_src
> 
> The problem can be reproduced by running python with jedi completer in
> a terminal. The reason is that readline completer can only see the
> text in the current line, i.e. "    im"; in this case, jedi does not
> complete the text to "    import".
> 
> Similarly, the jedi completer cannot complete "except" after ex:
> 
> try:
>     pass
> ex|
> 
> > However, when I try to complete at the beginning of the second line:
> >
> > #+begin_src python
> > try:
> >
> > #+end_src
> >
> >
> > "import" keyword also appears as a candidate.  If I cancel the
> > candidates and type "im" and try to complete it, it will succeed.
> 
> It is because jedi produces completions including "    import" for
> blank string "    ". Due to the same prefix between "    " and
> "    im", completion cache is reused for the latter. Then "    import"
> can be completed.
> 
> It is more a limitation of readline completer than a problem with
> jedi, as we cannot provide proper completion context for jedi. We may
> define a custom completer to combine jedi and rlcompleter, e.g.
> 
> (setq python-shell-readline-completer "
> def __PYTHON_EL_setup_readline_completer():
>     import readline, rlcompleter
>     import re, sys, os, __main__
>     from jedi import Interpreter
> 
>     class MyJediRL:
>         def __init__(self):
>             self.rlcompleter = rlcompleter.Completer()
>             self.rldelim = readline.get_completer_delims()
> 
>         def complete(self, text, state):
>             if state == 0:
>                 sys.path.insert(0, os.getcwd())
>                 try:
>                     interpreter = Interpreter(text, [__main__.__dict__])
>                     completions = interpreter.complete(fuzzy=False)
>                     self.matches = [
>                         text[:len(text) - c._like_name_length] + c.name_with_symbols
>                         for c in completions
>                     ]
> 
>                     # try rlcompleter
>                     sub = re.split('[' + re.escape(self.rldelim) + ']', text)[-1]
>                     i = 0
>                     while True:
>                         completion = self.rlcompleter.complete(sub, i)
>                         if not completion:
>                             break
>                         i += 1
>                         completion = text[:len(text)-len(sub)] + completion.rstrip(' ()')
>                         if completion not in self.matches:
>                             self.matches.append(completion)
>                 except:
>                     raise
>                 finally:
>                     sys.path.pop(0)
>             try:
>                 return self.matches[state]
>             except IndexError:
>                 return None
> 
>     readline.set_completer(MyJediRL().complete)
>     readline.set_completer_delims('')")

Thank you for the detailed explanation and the workaround.  I
confirmed that the problem is solved by the above workaround.  Just to
confirm, are you of the opinion that this workaround should not be the
default?

> > Another thing I noticed is the multi-line import statement.  If the
> > import statement is one-line, each items (IGNORECASE and MULTILINE in
> > the example below) can be completed.
> >
> > #+begin_src python
> > from re import IGNORECASE, MULTILINE
> > #+end_src
> >
> >
> > However, they cannot be completed if the import statement spans
> > multi-line.
> >
> > #+begin_src python
> > from re import (
> >     IGN
> > #+end_src
> >
> > This happens in both Python buffer and Python Shell buffer.  Perhaps
> > this is a limitation of Jedi completer?
> 
> Yes. Because readline completer cannot see cross-line context, I added
> the function "python-shell-completion-extra-context" in previous patch
> to address the case of multi-line function call. I have updated the
> attached patch to handle multi-line import statement.

Thank you very much.  I confirmed that the new patch allows completion
of multi-line import statements.