bug#24401: python-shell-completion-native-try returns incorrect results with python 3.5.2

bug#24401: python-shell-completion-native-try returns incorrect results with python 3.5.2

[Clément Pit--Claudel]

[-- Attachment #1.1: Type: text/plain, Size: 3054 bytes --]

Hi all,

Calling ‘run-python’ in ‘emacs -Q’ with ‘python-shell-interpreter’ set to "python3", I get the following warning:

    Warning (python): Your ‘python-shell-interpreter’ doesn’t seem to support
    readline, yet ‘python-shell-completion-native’ was t and "python3" is not
    part of the ‘python-shell-completion-native-disabled-interpreters’ list.
    Native completions have been disabled locally.

This warning is specific to Python 3 (it doesn't happen with Python 2.7). Additionally, the warning is spurious; native completion works fine.  This whole feature is new in Emacs 25.

The problem seems to be in

    (defun python-shell-completion-native-try ()
      "Return non-nil if can trigger native completion."
      (let ((python-shell-completion-native-enable t)
            (python-shell-completion-native-output-timeout
             python-shell-completion-native-try-output-timeout))
        (python-shell-completion-native-get-completions
         (get-buffer-process (current-buffer))
         nil "")))

The last call returns nil, apparently because ‘…-get-completions’ doesn't work with an empty input string in Python 3.5.2 with Emacs 25; however, if I change that last call as follows:

    (python-shell-completion-native-get-completions
     (get-buffer-process (current-buffer))
     nil "_")
          ^ add an underscore here, or another character that has completions

Then the warning message doesn't appear any more. Put more succinctly:

    Emacs 25, Python 2.7 (no warnings):

        Python 2.7.12 (default, Jul  1 2016, 15:12:24)
        [GCC 5.4.0 20160609] on linux2
        Type "help", "copyright", "credits" or "license" for more information.
        >>> python.el: native completion setup loaded

        (python-shell-completion-native-get-completions (get-buffer-process (current-buffer)) nil "")
        ⇒ ("and" "as" "assert" "break" "class" "continue" "def" "del" "elif" "else" "except" "exec" "finally" "for" "from" …)

        (python-shell-completion-native-get-completions (get-buffer-process (current-buffer)) nil "_")
        ⇒ ("__package__" "__PYTHON_EL_native_completion_setup" "__pyfile" "__name__" "__code" "__doc__" "__import__" "__debug__")

    Emacs 25, Python 3.5 (shows a warning):

        Python 3.5.2 (default, Jul  5 2016, 12:43:10)
        [GCC 5.4.0 20160609] on linux
        Type "help", "copyright", "credits" or "license" for more information.
        >>> python.el: native completion setup loaded

        (python-shell-completion-native-get-completions (get-buffer-process (current-buffer)) nil "")
        ⇒ nil ;; ← This is the bug

        (python-shell-completion-native-get-completions (get-buffer-process (current-buffer)) nil "_")
        ⇒ ("__name__" "__package__" "__spec__" "__PYTHON_EL_native_completion_setup" "__doc__" "__loader__" "__code" "__pyfile" "__build_class__" "__import__" "__debug__")

Can we add the suggested underscore?

Cheers,
Clément.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

bug#24401: python-shell-completion-native-try returns incorrect results with python 3.5.2

[Noam Postavsky]

On Fri, Sep 9, 2016 at 7:07 PM, Clément Pit--Claudel
<clement.pitclaudel@live.com> wrote:
> Hi all,
>
> Calling ‘run-python’ in ‘emacs -Q’ with ‘python-shell-interpreter’ set to "python3", I get the following warning:
>
>     Warning (python): Your ‘python-shell-interpreter’ doesn’t seem to support
>     readline, yet ‘python-shell-completion-native’ was t and "python3" is not
>     part of the ‘python-shell-completion-native-disabled-interpreters’ list.
>     Native completions have been disabled locally.
>
> This warning is specific to Python 3 (it doesn't happen with Python 2.7). Additionally, the warning is spurious; native completion works fine.  This whole feature is new in Emacs 25.

I'm not familiar enough with Python tooling to say much about this,
but it seems similar (maybe identical) to #22897?

bug#24401: python-shell-completion-native-try returns incorrect results with python 3.5.2

[Clément Pit--Claudel]

[-- Attachment #1.1: Type: text/plain, Size: 1040 bytes --]

On 2016-09-09 19:30, Noam Postavsky wrote:
> On Fri, Sep 9, 2016 at 7:07 PM, Clément Pit--Claudel 
> <clement.pitclaudel@live.com> wrote:
>> Hi all,
>> 
>> Calling ‘run-python’ in ‘emacs -Q’ with ‘python-shell-interpreter’
>> set to "python3", I get the following warning:
>> 
>> Warning (python): Your ‘python-shell-interpreter’ doesn’t seem to
>> support readline, yet ‘python-shell-completion-native’ was t and
>> "python3" is not part of the
>> ‘python-shell-completion-native-disabled-interpreters’ list. Native
>> completions have been disabled locally.
>> 
>> This warning is specific to Python 3 (it doesn't happen with Python
>> 2.7). Additionally, the warning is spurious; native completion
>> works fine.  This whole feature is new in Emacs 25.
> 
> I'm not familiar enough with Python tooling to say much about this, 
> but it seems similar (maybe identical) to #22897?

You're right. Why do I never find these duplicates?
Does anyone object to the proposed fix?

Clément.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

bug#24401: python-shell-completion-native-try returns incorrect results with python 3.5.2

[Lukas Juhrich]

[-- Attachment #1.1.1: Type: text/plain, Size: 401 bytes --]

Hi,

On 10.09.2016 06:00, Clément Pit--Claudel wrote:
> Does anyone object to the proposed fix?
>
> Clément.
>

May I as a complete outsider to emacs development ask what the state
on this is?  To me it looks like this just hasn't been done yet.

If having a patch file makes things easier, there is one attached; Let
me know if I can help otherwise.


--
Sincerely,
Lukas Juhrich

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1.1.2: 0001-Fix-Bug-24401-python-native-completion-warning.patch --]
[-- Type: text/x-patch; name="0001-Fix-Bug-24401-python-native-completion-warning.patch", Size: 1304 bytes --]

From 5919da9446850a3830ea1702c500dd749935dd54 Mon Sep 17 00:00:00 2001
From: Lukas Juhrich <lukasjuhrich@wh2.tu-dresden.de>
Date: Thu, 27 Oct 2016 03:24:58 +0200
Subject: [PATCH] Fix Bug #24401 (python native completion warning)

This commit fixes #24401 by checking for completions using "_" instead

Checking the python processes' completion by completing "" in
`python-shell-completion-native-try` does not work, since an empty
string does not yield any completion in python (at least python3.5).
Thus, `nil` is returned and the completion is regarded unsuccessful.

This commit changes the character to "_" and thus fixes #24401.
---
 lisp/progmodes/python.el | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lisp/progmodes/python.el b/lisp/progmodes/python.el
index ad69f87..290cdc8 100644
--- a/lisp/progmodes/python.el
+++ b/lisp/progmodes/python.el
@@ -3318,7 +3318,7 @@ When a match is found, native completion is disabled."
          python-shell-completion-native-try-output-timeout))
     (python-shell-completion-native-get-completions
      (get-buffer-process (current-buffer))
-     nil "")))
+     nil "_")))
 
 (defun python-shell-completion-native-setup ()
   "Try to setup native completion, return non-nil on success."
-- 
2.10.1


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 801 bytes --]

bug#24401: python-shell-completion-native-try returns incorrect results with python 3.5.2

[Clément Pit--Claudel]

[-- Attachment #1.1.1: Type: text/plain, Size: 1131 bytes --]

Hi Lukas,

Thanks for the patch!

I prepared a similar one, but I didn't end up pushing it, because the bug has a more complex root cause.  Essentially, the issue is that completion doesn't work for empty strings in when python-shell-interpreter points to Python 3 (see also the original bug report, #22897).

A good way to experiment with this is to copy the completion code to a separate file (I attached it) and run python -i completion.py.  In python 2.7, pressing TAB shows a bunch of completions.  In 3.5, it just inserts a tab.

Can you help figure out why this happen?  Then we could fix the root of this bug.

Cheers,
Clément.

On 2016-10-26 21:34, Lukas Juhrich wrote:
> Hi,
> 
> On 10.09.2016 06:00, Clément Pit--Claudel wrote:
>> Does anyone object to the proposed fix?
>>
>> Clément.
>>
> 
> May I as a complete outsider to emacs development ask what the state
> on this is?  To me it looks like this just hasn't been done yet.
> 
> If having a patch file makes things easier, there is one attached; Let
> me know if I can help otherwise.
> 
> 
> --
> Sincerely,
> Lukas Juhrich
> 

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1.1.2: completion.py --]
[-- Type: text/x-python-script; name="completion.py", Size: 4659 bytes --]

def __PYTHON_EL_native_completion_setup():
    try:
        import readline

        try:
            import __builtin__
        except ImportError:
            # Python 3
            import builtins as __builtin__

        builtins = dir(__builtin__)
        is_ipython = ('__IPYTHON__' in builtins or
                      '__IPYTHON__active' in builtins)

        class __PYTHON_EL_Completer:
            '''Completer wrapper that prints candidates to stdout.

            It wraps an existing completer function and changes its behavior so
            that the user input is unchanged and real candidates are printed to
            stdout.

            Returned candidates are '0__dummy_completion__' and
            '1__dummy_completion__' in that order ('0__dummy_completion__' is
            returned repeatedly until all possible candidates are consumed).

            The real candidates are printed to stdout so that they can be
            easily retrieved through comint output redirect trickery.
            '''

            PYTHON_EL_WRAPPED = True

            def __init__(self, completer):
                self.completer = completer
                self.last_completion = None
                self.print_mode = True

            def __call__(self, text, state):
                if state == 0:
                    # Set the first dummy completion.
                    self.last_completion = None
                    completion = '0__dummy_completion__'
                else:
                    completion = self.completer(text, state - 1)

                if not completion:
                    if self.last_completion != '1__dummy_completion__':
                        # When no more completions are available, returning a
                        # dummy with non-sharing prefix allow ensuring output
                        # while preventing changes to current input.
                        # Coincidentally it's also the end of output.
                        completion = '1__dummy_completion__'
                elif completion.endswith('('):
                    # Remove parens on callables as it breaks completion on
                    # arguments (e.g. str(Ari<tab>)).
                    completion = completion[:-1]
                self.last_completion = completion

                if completion in (
                        '0__dummy_completion__', '1__dummy_completion__'):
                    return completion
                elif completion:
                    # For every non-dummy completion, return a repeated dummy
                    # one and print the real candidate so it can be retrieved
                    # by comint output filters.
                    if self.print_mode:
                        print (completion)
                        return '0__dummy_completion__'
                    else:
                        return completion
                else:
                    return completion

        completer = readline.get_completer()

        if not completer:
            # Used as last resort to avoid breaking customizations.
            import rlcompleter
            completer = readline.get_completer()

        if completer and not getattr(completer, 'PYTHON_EL_WRAPPED', False):
            # Wrap the existing completer function only once.
            new_completer = __PYTHON_EL_Completer(completer)
            if not is_ipython:
                readline.set_completer(new_completer)
            else:
                # Try both initializations to cope with all IPython versions.
                # This works fine for IPython 3.x but not for earlier:
                readline.set_completer(new_completer)
                # IPython<3 hacks readline such that `readline.set_completer`
                # won't work.  This workaround injects the new completer
                # function into the existing instance directly:
                instance = getattr(completer, 'im_self', completer.__self__)
                instance.rlcomplete = new_completer

        if readline.__doc__ and 'libedit' in readline.__doc__:
            print("AAA")
            readline.parse_and_bind('bind ^I rl_complete')
        else:
            print("BBB")
            readline.parse_and_bind('tab: complete')
            # Require just one tab to send output.
            readline.parse_and_bind('set show-all-if-ambiguous on')

        print ('python.el: native completion setup loaded')
    except:
        print ('python.el: native completion setup failed')

__PYTHON_EL_native_completion_setup()

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

bug#24401: python-shell-completion-native-try returns incorrect results with python 3.5.2

[Clément Pit--Claudel]

[-- Attachment #1.1: Type: text/plain, Size: 538 bytes --]

https://bugs.python.org/issue25660 might be related?

On 2016-10-26 21:34, Lukas Juhrich wrote:
> Hi,
> 
> On 10.09.2016 06:00, Clément Pit--Claudel wrote:
>> Does anyone object to the proposed fix?
>>
>> Clément.
>>
> 
> May I as a complete outsider to emacs development ask what the state
> on this is?  To me it looks like this just hasn't been done yet.
> 
> If having a patch file makes things easier, there is one attached; Let
> me know if I can help otherwise.
> 
> 
> --
> Sincerely,
> Lukas Juhrich
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

bug#24401: python-shell-completion-native-try returns incorrect results with python 3.5.2

[Clément Pit--Claudel]

[-- Attachment #1.1.1: Type: text/plain, Size: 1213 bytes --]

On 2016-09-10 00:00, Clément Pit--Claudel wrote:
> On 2016-09-09 19:30, Noam Postavsky wrote:
>> On Fri, Sep 9, 2016 at 7:07 PM, Clément Pit--Claudel 
>> <clement.pitclaudel@live.com> wrote:
>>> Hi all,
>>>
>>> Calling ‘run-python’ in ‘emacs -Q’ with ‘python-shell-interpreter’
>>> set to "python3", I get the following warning:
>>>
>>> Warning (python): Your ‘python-shell-interpreter’ doesn’t seem to
>>> support readline, yet ‘python-shell-completion-native’ was t and
>>> "python3" is not part of the
>>> ‘python-shell-completion-native-disabled-interpreters’ list. Native
>>> completions have been disabled locally.
>>>
>>> This warning is specific to Python 3 (it doesn't happen with Python
>>> 2.7). Additionally, the warning is spurious; native completion
>>> works fine.  This whole feature is new in Emacs 25.
>>
>> I'm not familiar enough with Python tooling to say much about this, 
>> but it seems similar (maybe identical) to #22897?
> 
> You're right. Why do I never find these duplicates?
> Does anyone object to the proposed fix?

If no one objects, I'll push the attached patch to master tomorrow (or should it go to emacs-25?)

Clément.

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1.1.2: 0001-python.el-Fix-detection-of-native-completion-in-Pyth.patch --]
[-- Type: text/x-diff; name="0001-python.el-Fix-detection-of-native-completion-in-Pyth.patch", Size: 1230 bytes --]

From 9dd2c4867966d39d621d4862957b90dcc39fbdf0 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Cl=C3=A9ment=20Pit--Claudel?= <clement.pitclaudel@live.com>
Date: Wed, 26 Oct 2016 22:46:28 -0400
Subject: [PATCH] python.el: Fix detection of native completion in Python 3
 (bug #24401)

With Python 3.5, (python-shell-completion-native-get-completions ... "")
would return an empty list, causing python.el to think that native
completion was unavailable.

* lisp/progmodes/python.el (python-shell-completion-native-try): Use "_"
  to check whether native completion is available instead of "".
---
 lisp/progmodes/python.el | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lisp/progmodes/python.el b/lisp/progmodes/python.el
index ad69f87..290cdc8 100644
--- a/lisp/progmodes/python.el
+++ b/lisp/progmodes/python.el
@@ -3318,7 +3318,7 @@ python-shell-completion-native-try
          python-shell-completion-native-try-output-timeout))
     (python-shell-completion-native-get-completions
      (get-buffer-process (current-buffer))
-     nil "")))
+     nil "_")))
 
 (defun python-shell-completion-native-setup ()
   "Try to setup native completion, return non-nil on success."
-- 
2.7.4


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

bug#24401: python-shell-completion-native-try returns incorrect results with python 3.5.2

[Eli Zaretskii]

> From: Clément Pit--Claudel <clement.pit@gmail.com>
> Date: Sat, 29 Oct 2016 10:12:54 -0400
> 
> If no one objects, I'll push the attached patch to master tomorrow (or should it go to emacs-25?)

Can you tell when was the problem introduced?

Thanks.

bug#24401: python-shell-completion-native-try returns incorrect results with python 3.5.2

[Clément Pit--Claudel]

[-- Attachment #1.1: Type: text/plain, Size: 444 bytes --]

On 2016-10-29 10:34, Eli Zaretskii wrote:
>> From: Clément Pit--Claudel <clement.pit@gmail.com>
>> Date: Sat, 29 Oct 2016 10:12:54 -0400
>>
>> If no one objects, I'll push the attached patch to master tomorrow (or should it go to emacs-25?)
> 
> Can you tell when was the problem introduced?

The problem is new in Emacs 25: native completion is a new feature of python mode (2cb7592275bce47e44916134223b994a75e4b861).

Clément.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

bug#24401: python-shell-completion-native-try returns incorrect results with python 3.5.2

[Eli Zaretskii]

> Cc: 24401@debbugs.gnu.org
> From: Clément Pit--Claudel <clement.pit@gmail.com>
> Date: Sat, 29 Oct 2016 11:03:31 -0400
> 
> On 2016-10-29 10:34, Eli Zaretskii wrote:
> >> From: Clément Pit--Claudel <clement.pit@gmail.com>
> >> Date: Sat, 29 Oct 2016 10:12:54 -0400
> >>
> >> If no one objects, I'll push the attached patch to master tomorrow (or should it go to emacs-25?)
> > 
> > Can you tell when was the problem introduced?
> 
> The problem is new in Emacs 25: native completion is a new feature of python mode (2cb7592275bce47e44916134223b994a75e4b861).

Then it should go to emacs-25.

Thanks.

bug#24401: python-shell-completion-native-try returns incorrect results with python 3.5.2

[Clément Pit--Claudel]

[-- Attachment #1.1: Type: text/plain, Size: 111 bytes --]

On 2016-10-29 11:08, Eli Zaretskii wrote:
> Then it should go to emacs-25.

Thanks; pushed!

Clément.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

bug#24401: python-shell-completion-native-try returns incorrect results with python 3.5.2

[npostavs]

Clément Pit--Claudel <clement.pit@gmail.com> writes:

> On 2016-10-29 11:08, Eli Zaretskii wrote:
>> Then it should go to emacs-25.
>
> Thanks; pushed!
>
> Clément.

Does that mean the bug is fixed?  Or is there still some "root cause" of
the bug to be found?

bug#24401: python-shell-completion-native-try returns incorrect results with python 3.5.2

[npostavs]

tags 24401 fixed
close 24401 25.2
quit

npostavs@users.sourceforge.net writes:

> Clément Pit--Claudel <clement.pit@gmail.com> writes:
>
>> On 2016-10-29 11:08, Eli Zaretskii wrote:
>>> Then it should go to emacs-25.
>>
>> Thanks; pushed!
>>
>> Clément.
>
> Does that mean the bug is fixed?

I'll assume yes.

bug#24401: python-shell-completion-native-try returns incorrect results with python 3.5.2

[Clément Pit--Claudel]

[-- Attachment #1.1: Type: text/plain, Size: 204 bytes --]

On 2016-11-04 21:59, npostavs@users.sourceforge.net wrote:
>> Does that mean the bug is fixed?
> 
> I'll assume yes.

Sorry about this! I missed your previous email. Yes was the correct answer :)


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

bug#24401:

[Davor Rotim]

While trying out the latest pretest this issue doesn't seem to be fixed.

GNU Emacs 25.1.90.1 (x86_64-unknown-linux-gnu, X toolkit, Xawscrollbars) of 2016-11-29
Python 3.5.2

Steps to reproduce:

Start with emacs -Q and M-x run-python, the warning will still appear:

Warning (python): Your ‘python-shell-interpreter’ doesn’t seem to support readline, yet ‘python-shell-completion-native’ was t and "python" is not part of the ‘python-shell-completion-native-disabled-interpreters’ list.  Native completions have been disabled locally.

Pressing TAB will also insert spaces instead of showing completions.

bug#24401:

[Clément Pit--Claudel]

[-- Attachment #1.1: Type: text/plain, Size: 739 bytes --]

On 2016-11-29 11:38, Davor Rotim wrote:
> While trying out the latest pretest this issue doesn't seem to be fixed.
> 
> GNU Emacs 25.1.90.1 (x86_64-unknown-linux-gnu, X toolkit, Xawscrollbars) of 2016-11-29
> Python 3.5.2
> 
> Steps to reproduce:
> 
> Start with emacs -Q and M-x run-python, the warning will still appear:
> 
> Warning (python): Your ‘python-shell-interpreter’ doesn’t seem to support readline, yet ‘python-shell-completion-native’ was t and "python" is not part of the ‘python-shell-completion-native-disabled-interpreters’ list.  Native completions have been disabled locally.
> 
> Pressing TAB will also insert spaces instead of showing completions.

Thanks; I'll investigate this today.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

bug#24401:

[Clément Pit--Claudel]

[-- Attachment #1.1: Type: text/plain, Size: 1568 bytes --]

On 2016-11-29 12:21, Clément Pit--Claudel wrote:
> On 2016-11-29 11:38, Davor Rotim wrote:
>> While trying out the latest pretest this issue doesn't seem to be fixed.
>>
>> GNU Emacs 25.1.90.1 (x86_64-unknown-linux-gnu, X toolkit, Xawscrollbars) of 2016-11-29
>> Python 3.5.2
>>
>> Steps to reproduce:
>>
>> Start with emacs -Q and M-x run-python, the warning will still appear:
>>
>> Warning (python): Your ‘python-shell-interpreter’ doesn’t seem to support readline, yet ‘python-shell-completion-native’ was t and "python" is not part of the ‘python-shell-completion-native-disabled-interpreters’ list.  Native completions have been disabled locally.
>>
>> Pressing TAB will also insert spaces instead of showing completions.

Hmm. I can't reproduce this here (I just built from the archive).  I start with

    $ src/emacs -Q --eval '(setq python-shell-interpreter "python3")'

Then I run 'M-x run-python (Python 3.5.2 (default, Nov 17 2016, 17:05:23))'; I don't get a warning.

Can you try `M-x find-function python-shell-completion-native-try' and confirm that it shows this:

(defun python-shell-completion-native-try ()
  "Return non-nil if can trigger native completion."
  (let ((python-shell-completion-native-enable t)
        (python-shell-completion-native-output-timeout
         python-shell-completion-native-try-output-timeout))
    (python-shell-completion-native-get-completions
     (get-buffer-process (current-buffer))
     nil "_")))
          ^ this is the important bit

Thanks!
Clément.



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

bug#24401:

[Davor Rotim]

I started with:

emacs -Q --eval `(setq python-shell-interpreter "python3")'

then did `M-x run-python' and I was greeted by the same warning.

Tried `M-x find-function python-shell-completion-native-try' and the "_" addition is there.

http://i.imgur.com/rEDklpI.png

bug#24401:

[Clément Pit--Claudel]

[-- Attachment #1.1: Type: text/plain, Size: 633 bytes --]

On 2016-11-29 15:40, Davor Rotim wrote:
> I started with:
> 
> emacs -Q --eval `(setq python-shell-interpreter "python3")'
> 
> then did `M-x run-python' and I was greeted by the same warning.
> 
> Tried `M-x find-function python-shell-completion-native-try' and the "_" addition is there.
> 
> http://i.imgur.com/rEDklpI.png

Hmm. Can you try the following? M-x run-python, then with the *Python* buffer current do (python-shell-completion-native-try).  Does this return a list of strings?

What about 

  (python-shell-completion-native-get-completions (get-buffer-process (current-buffer)) nil "imp")

Thanks!


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

bug#24401:

[Davor Rotim]

Started with the following:

emacs -Q --eval '(setq python-shell-interpreter "python3")'
M-x run-python

Evaluation of (python-shell-completion-native-try):

http://i.imgur.com/VbBbxuy.png

Evaluation (python-shell-completion-native-get-completions (get-buffer-process (current-buffer)) nil "imp"):

http://i.imgur.com/ttABCyP.png

The same warning and errors also happen when I use "python2" as the interpreter so this is a bit strange.
This is on Arch Linux with python 3.5.2-3 and python 2.7.12-2 installed:

Python 3.5.2 (default, Nov  7 2016, 11:31:36)
[GCC 6.2.1 20160830] on linux

Python 2.7.12 (default, Nov  7 2016, 11:55:55)
[GCC 6.2.1 20160830] on linux2

Thank you for your patience.

bug#24401:

[Noam Postavsky]

On Wed, Nov 30, 2016 at 4:39 AM, Davor Rotim <drot@firemail.cc> wrote:

> The same warning and errors also happen when I use "python2" as the interpreter so this is a bit strange.
> This is on Arch Linux with python 3.5.2-3 and python 2.7.12-2 installed:

Also running Arch here (though my packages are a few months older),
you have to evaluate those expressions in the *Python* buffer, or else
you get those errors.

I'm not getting the warning here.

bug#24401:

[Davor Rotim]

Evaluating both of those expressions in the *Python* buffer returns nil.

bug#24401:

[Clément Pit--Claudel]

[-- Attachment #1.1: Type: text/plain, Size: 289 bytes --]

On 2016-11-30 10:09, Davor Rotim wrote:
> Evaluating both of those expressions in the *Python* buffer returns nil.

I installed the release candidate in a clean Ubuntu VM, and I couldn't reproduce this.  Do you think you can you investigate this issue locally? 

Cheers,
Clément.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

bug#24401:

[Davor Rotim]

I think I have tracked down the issue.

I switched to another user and I was able to use `run-python' without any warnings.
Turns out it was my readline settings.
Bash 4.3 added a new readline feature: "set colored-stats on" which I had in .inputrc
Removing that line fixed my issue, I think this should be reproducible for all.

Once again thank you for your patience.

bug#24401:

[Clément Pit--Claudel]

[-- Attachment #1.1: Type: text/plain, Size: 676 bytes --]

On 2016-12-01 03:47, Davor Rotim wrote:
> I think I have tracked down the issue.
> 
> I switched to another user and I was able to use `run-python' without any warnings.
> Turns out it was my readline settings.
> Bash 4.3 added a new readline feature: "set colored-stats on" which I had in .inputrc
> Removing that line fixed my issue, I think this should be reproducible for all.
> 
> Once again thank you for your patience.

Wonderful! Do you know if there's a way we can disable this locally, for the Python subprocess?  Alternatively, do you want to contribute a patch that adds this hint to the warning shown if completion doesn't work?

Thanks!
Clément.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

bug#24401:

[Davor Rotim]

[-- Attachment #1: Type: text/plain, Size: 263 bytes --]

Luckily Readline has conditional constructs:

# .inputrc
$if Bash
set colored-stats on
$endif

This enables us to have color tab completion and avoid the bug with Emacs.
As for the alternative, I've provided a patch in the attachment.

Kind regards,

Davor Rotim

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Python warning message patch --]
[-- Type: text/x-diff, Size: 856 bytes --]

diff --git a/lisp/progmodes/python.el b/lisp/progmodes/python.el
index d0d4a7f..ddab9c0 100644
--- a/lisp/progmodes/python.el
+++ b/lisp/progmodes/python.el
@@ -3455,7 +3455,9 @@ With argument MSG show activation/deactivation message."
             (format "was t and %S is not part of the "
                     (file-name-nondirectory python-shell-interpreter))
             "`python-shell-completion-native-disabled-interpreters' "
-            "list.  Native completions have been disabled locally. "))
+            "list. If you are using GNU Bash check that the colored-stats "
+            "variable is conditionally enabled in your Readline init file. "
+            "Native completions have been disabled locally. "))
           (python-shell-completion-native-turn-off msg))))))
 
 (defun python-shell-completion-native-turn-on-maybe-with-msg ()

bug#24401:

[Clément Pit--Claudel]

[-- Attachment #1.1: Type: text/plain, Size: 720 bytes --]

Thanks for the patch!

> +            "variable is conditionally enabled in your Readline init file. "
I would suggest conditionally "disabled", or simply "check the value of the … variable"

Can you provide the patch in Git format, so that I can commit it in your name? Eli, since this is documentation (just a comment), I guess it goes to emacs-25?

Clément.

On 2016-12-01 15:23, Davor Rotim wrote:
> Luckily Readline has conditional constructs:
> 
> # .inputrc
> $if Bash
> set colored-stats on
> $endif
> 
> This enables us to have color tab completion and avoid the bug with Emacs.
> As for the alternative, I've provided a patch in the attachment.
> 
> Kind regards,
> 
> Davor Rotim
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

bug#24401:

[Davor Rotim]

[-- Attachment #1: Type: text/plain, Size: 188 bytes --]

I think "conditionally disabled" is a better choice since it implies it
should be disabled in the first place.
I hope the attached patch is correct (emacs-25 branch).

Thanks,
Davor Rotim

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Python warning message patch --]
[-- Type: text/x-patch, Size: 1195 bytes --]

From 829f5879e73522c2fe7d23b7a3ecb2f4469867e9 Mon Sep 17 00:00:00 2001
From: Davor Rotim <rotim.davor@gmail.com>
Date: Thu, 1 Dec 2016 22:15:56 +0100
Subject: [PATCH] Add a hint to the warning message of python-mode to avoid
 conflicts.

---
 lisp/progmodes/python.el | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/lisp/progmodes/python.el b/lisp/progmodes/python.el
index d0d4a7f..c977f0b 100644
--- a/lisp/progmodes/python.el
+++ b/lisp/progmodes/python.el
@@ -3455,7 +3455,9 @@ With argument MSG show activation/deactivation message."
             (format "was t and %S is not part of the "
                     (file-name-nondirectory python-shell-interpreter))
             "`python-shell-completion-native-disabled-interpreters' "
-            "list.  Native completions have been disabled locally. "))
+            "list. If you are using GNU Bash check that the colored-stats "
+            "variable is conditionally disabled in your Readline init file. "
+            "Native completions have been disabled locally. "))
           (python-shell-completion-native-turn-off msg))))))
 
 (defun python-shell-completion-native-turn-on-maybe-with-msg ()
-- 
2.10.2

bug#24401:

[Clément Pit--Claudel]

[-- Attachment #1.1: Type: text/plain, Size: 313 bytes --]

Thanks a lot!  I'll wait for approval before pushing it.

On 2016-12-01 16:29, Davor Rotim wrote:
> I think "conditionally disabled" is a better choice since it implies it
> should be disabled in the first place.
> I hope the attached patch is correct (emacs-25 branch).
> 
> Thanks,
> Davor Rotim
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

bug#24401:

[Eli Zaretskii]

> From: Clément Pit--Claudel <clement.pit@gmail.com>
> Date: Thu, 1 Dec 2016 16:45:13 -0500
> Cc: drot@firemail.cc
> 
> Thanks a lot!  I'll wait for approval before pushing it.

If this is meant for the emacs-25 branch, then you have my approval.

Thanks.

bug#24401:

[Clément Pit--Claudel]

[-- Attachment #1.1: Type: text/plain, Size: 671 bytes --]

On 2016-12-01 03:47, Davor Rotim wrote:
> I think I have tracked down the issue.
> 
> I switched to another user and I was able to use `run-python' without any warnings.
> Turns out it was my readline settings.
> Bash 4.3 added a new readline feature: "set colored-stats on" which I had in .inputrc
> Removing that line fixed my issue, I think this should be reproducible for all.
> 
> Once again thank you for your patience.

Sorry; just before I push the fix to the warning message: are you sure this was the issue? When I set colored-stats on in my .inputrc, I do get colors in the terminal, but completion still works fine in Emacs.

Cheers,
Clément.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

bug#24401:

[Davor Rotim]

Happens with a new user if I have this in my .inputrc:

$include /etc/inputrc # tried also without this, same result
set colored-stats on

As soon as the line is removed or the variable is set to conditionally
disabled everything works again.

This was tested on:
GNU bash, version 4.4.5(1)-release (x86_64-unknown-linux-gnu)
Readline 7.0

There's no rush to merge the patch if someone else wants to test this.

Thanks,
Davor Rotim

bug#24401:

[Clément Pit--Claudel]

[-- Attachment #1.1: Type: text/plain, Size: 836 bytes --]

On 2016-12-02 12:13, Davor Rotim wrote:
> Happens with a new user if I have this in my .inputrc:
> 
> $include /etc/inputrc # tried also without this, same result
> set colored-stats on
> 
> As soon as the line is removed or the variable is set to conditionally
> disabled everything works again.
> 
> This was tested on:
> GNU bash, version 4.4.5(1)-release (x86_64-unknown-linux-gnu)
> Readline 7.0
> 
> There's no rush to merge the patch if someone else wants to test this.

Indeed, I'd love some help testing this; I don't think the version of bash plays much of a role (since we're calling python directly), but the version of readline may.

On my machine, changing the .inputrc does change bash's behaviour, but it doesn't make a difference in Emacs' python shell (and so I don't get a warning).

Clément.



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

bug#24401:

[Davor Rotim]

So far I've got two people who could confirm this error happening on
their Arch Linux installs with Readline-7.0.
I think it's best not to pollute this bug report anymore and open a new
one to show awareness of the issue and to report further findings.

Kind regards,
Davor Rotim