bug#25753: 25.2; Python mode shell interaction not working 100%

bug#25753: 25.2; Python mode shell interaction not working 100%

[Charles A. Roelli]

In emacs 25.2 (rc1) and 25.1, interaction with the Python shell does not
seem to be working fully.  Python version is '2.7.12 (v2.7.12:d33e0cf91556,
Jun 26 2016, 12:10:39) [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)]', and
this is on Snow Leopard.

- emacs -Q
- open any Python file
- M-x run-python, and this pops up:

  Warning (python): Your ‘python-shell-interpreter’ doesn’t seem to
  support readline, yet ‘python-shell-completion-native’ was t and
  "python" is not part of the
  ‘python-shell-completion-native-disabled-interpreters’ list.  Native
  completions have been disabled locally.
  
  [and yet `import readline, rlcompleter' works fine for me]

- quit that warning, and the *Python* shell states "python.el: native
  completion setup loaded".
- test out completion in the shell, it seems to be working okay
  (`import sys RET sys . TAB' shows the attributes of `sys').  Not sure
  if this is `native' completion though.
- switch to the Python file.  Place point over a module, global
  variable, function, or function definition, and this pops up in the
  echo area (meant to be eldoc documentation, I think):

  import codecs, os;
  __pyfile = codecs.open('''/var/folders/WP/WPe0Q1iAGc0J7iI6J50jcU+++TI/-Tmp-/py1883fcQ''',encoding='''utf-8''');
  __code = __pyfile.read().encode('''utf-8''');
  __pyfile.close();
  os.remove('''/var/folders/WP/WPe0Q1iAGc0J7iI6J50jcU+++TI/-Tmp-/py1883fcQ''');
  exec(compile(__code,'''/path/to/python/buffer/here''','exec'));
  
  [some line breaks included for readability]

  The same sort of thing is returned by calling, say,
  `(python-ffap-module-path "twisted")'.  Any pointers on fixing this?




In GNU Emacs 25.2.1 (x86_64-apple-darwin10.8.0, NS appkit-1038.36 Version 10.6.8 (Build 10K549))
 of 2017-02-07 built on gray
Windowing system distributor 'Apple', version 10.3.1038
Configured using:
 'configure --with-modules'

Configured features:
JPEG RSVG NOTIFY ACL GNUTLS LIBXML2 ZLIB TOOLKIT_SCROLL_BARS NS MODULES

bug#25753: 25.2; Python mode shell interaction not working 100%

[Eli Zaretskii]

> From: charles@aurox.ch (Charles A. Roelli)
> Date: Thu, 16 Feb 2017 17:07:48 +0100
> 
> In emacs 25.2 (rc1) and 25.1, interaction with the Python shell does not
> seem to be working fully.  Python version is '2.7.12 (v2.7.12:d33e0cf91556,
> Jun 26 2016, 12:10:39) [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)]', and
> this is on Snow Leopard.

Please try the next RC (should be out in a few days), I think we fixed
that there.

Thanks.

bug#25753: 25.2; Python mode shell interaction not working 100%

[npostavs]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: charles@aurox.ch (Charles A. Roelli)
>> Date: Thu, 16 Feb 2017 17:07:48 +0100
>> 
>> In emacs 25.2 (rc1) and 25.1, interaction with the Python shell does not
>> seem to be working fully.  Python version is '2.7.12 (v2.7.12:d33e0cf91556,
>> Jun 26 2016, 12:10:39) [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)]', and
>> this is on Snow Leopard.
>
> Please try the next RC (should be out in a few days), I think we fixed
> that there.

AFAIK, the only change since rc1 is to add "ipython" to
python-shell-completion-native-disabled-interpreters.  And that should
not affect the problem reported here, which does not seem to involve
ipython.

It might be something macOS specific, because I don't see any problems
on Arch GNU/Linux with python 2.7.13.

bug#25753: 25.2; Python mode shell interaction not working 100%

[Live System User]

npostavs@users.sourceforge.net writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>>> From: charles@aurox.ch (Charles A. Roelli)
>>> Date: Thu, 16 Feb 2017 17:07:48 +0100
>>> 
>>> In emacs 25.2 (rc1) and 25.1, interaction with the Python shell does not
>>> seem to be working fully.  Python version is '2.7.12 (v2.7.12:d33e0cf91556,
>>> Jun 26 2016, 12:10:39) [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)]', and
>>> this is on Snow Leopard.
>>
>> Please try the next RC (should be out in a few days), I think we fixed
>> that there.
>
> AFAIK, the only change since rc1 is to add "ipython" to
> python-shell-completion-native-disabled-interpreters.  And that should
> not affect the problem reported here, which does not seem to involve
> ipython.
>
> It might be something macOS specific, because I don't see any problems
> on Arch GNU/Linux with python 2.7.13.

  I see the problem on GNU/Linux (Fedora):

Warning (python): Your ‘python-shell-interpreter’ doesn’t seem to
support readline, yet ‘python-shell-completion-native’ was t and
"python" is not part of the
‘python-shell-completion-native-disabled-interpreters’ list.  Native
completions have been disabled locally.

Python 2.7.13 (default, Jan 13 2017, 10:15:16) 
[GCC 6.3.1 20161221 (Red Hat 6.3.1-1)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> python.el: native completion setup loaded

>>> q<TAB>qu<TAB>
Click on a completion to select it.
In this buffer, type RET to select the completion near point.

Possible completions are:
quit
quopri


In GNU Emacs 25.1.1 (x86_64-redhat-linux-gnu, GTK+ Version 3.20.9)
 of 2016-10-13 built on buildvm-05.phx2.fedoraproject.org
Windowing system distributor 'Fedora Project', version 11.0.11803000

  Thanks.

bug#25753: 25.2; Python mode shell interaction not working 100%

[Noam Postavsky]

On Sun, Feb 19, 2017 at 10:14 AM, Live System User <nyc4bos@aol.com> wrote:
>>
>> It might be something macOS specific, because I don't see any problems
>> on Arch GNU/Linux with python 2.7.13.
>
>   I see the problem on GNU/Linux (Fedora):
>
> Warning (python): Your ‘python-shell-interpreter’ doesn’t seem to
[...]
> In GNU Emacs 25.1.1 (x86_64-redhat-linux-gnu, GTK+ Version 3.20.9)
>  of 2016-10-13 built on buildvm-05.phx2.fedoraproject.org
> Windowing system distributor 'Fedora Project', version 11.0.11803000

Do you see this with 25.2-rc1 on GNU/Linux though?

There *was* a change since 25.1 (although supposedly that only
affected python 3). Do you have "set colored-stats on" in your
~/.inputrc? There was a report in #24401 of that causing trouble.

bug#25753: 25.2; Python mode shell interaction not working 100%

[Live System User]

Noam Postavsky <npostavs@users.sourceforge.net> writes:

> On Sun, Feb 19, 2017 at 10:14 AM, Live System User <nyc4bos@aol.com> wrote:
>>>
>>> It might be something macOS specific, because I don't see any problems
>>> on Arch GNU/Linux with python 2.7.13.
>>
>>   I see the problem on GNU/Linux (Fedora):
>>
>> Warning (python): Your ‘python-shell-interpreter’ doesn’t seem to
> [...]
>> In GNU Emacs 25.1.1 (x86_64-redhat-linux-gnu, GTK+ Version 3.20.9)
>>  of 2016-10-13 built on buildvm-05.phx2.fedoraproject.org
>> Windowing system distributor 'Fedora Project', version 11.0.11803000
>
> Do you see this with 25.2-rc1 on GNU/Linux though?

  Fedora doesn't have that version available currently.
  
>
> There *was* a change since 25.1 (although supposedly that only
> affected python 3). Do you have "set colored-stats on" in your
> ~/.inputrc? There was a report in #24401 of that causing trouble.

  I don't have a ~/.input file.

  I did, however, look into the Python modules having to do with
  "readline"(completion).

  I discovered that if I used the "readline" from "pyrepl.py"instead
  of the standard default one, then even though that warning still
  occured in a *Warning* buffer in Emacs, I never saw it -- I just
  saw   the *Python* buffer and my source buffer, as expected.

  I haven't looked deeper as to why "pyrepl"'s readline is able to
  not display the *Warning* buffer while Python's standard default
  readline displays the *Warning* buffer until it is dismissed.

  Thanks.

bug#25753: 25.2; Python mode shell interaction not working 100%

[Noam Postavsky]

On Sun, Feb 19, 2017 at 2:39 PM, Live System User <nyc4bos@aol.com> wrote:
>>
>> Do you see this with 25.2-rc1 on GNU/Linux though?
>
>   Fedora doesn't have that version available currently.

Can you try it after M-x load-library python RET and then evaluate
this new definition of python-shell-completion-native-try:

(defun python-shell-completion-native-try ()
  "Return non-nil if can trigger native completion."
  (let ((python-shell-completion-native-enable t)
        (python-shell-completion-native-output-timeout
         python-shell-completion-native-try-output-timeout))
    (python-shell-completion-native-get-completions
     (get-buffer-process (current-buffer))
     nil "_")))

>
>   I did, however, look into the Python modules having to do with
>   "readline"(completion).
>
>   I discovered that if I used the "readline" from "pyrepl.py"instead
>   of the standard default one, then even though that warning still
>   occured in a *Warning* buffer in Emacs, I never saw it -- I just
>   saw   the *Python* buffer and my source buffer, as expected.

I'm not sure what "pyrepl" is or to "use" it, but just looking at web
search results, it seems to be connected to pypy, which is in the list
of python-shell-completion-native-disabled-interpreters, could that be
related?

bug#25753: 25.2; Python mode shell interaction not working 100%

[Live System User]

Noam Postavsky <npostavs@users.sourceforge.net> writes:

> On Sun, Feb 19, 2017 at 2:39 PM, Live System User <nyc4bos@aol.com> wrote:
>>>
>>> Do you see this with 25.2-rc1 on GNU/Linux though?
>>
>>   Fedora doesn't have that version available currently.
>
> Can you try it after M-x load-library python RET and then evaluate
> this new definition of python-shell-completion-native-try:
>
> (defun python-shell-completion-native-try ()
>   "Return non-nil if can trigger native completion."
>   (let ((python-shell-completion-native-enable t)
>         (python-shell-completion-native-output-timeout
>          python-shell-completion-native-try-output-timeout))
>     (python-shell-completion-native-get-completions
>      (get-buffer-process (current-buffer))
>      nil "_")))

    I was already using that (wrapped in a "with-eval-after-load
    'python") from:

    https://github.com/jorgenschaefer/elpy/issues/887#issuecomment-275175119

    to no avail.

>
>>
>>   I did, however, look into the Python modules having to do with
>>   "readline"(completion).
>>
>>   I discovered that if I used the "readline" from "pyrepl.py"instead
>>   of the standard default one, then even though that warning still
>>   occured in a *Warning* buffer in Emacs, I never saw it -- I just
>>   saw   the *Python* buffer and my source buffer, as expected.
>
> I'm not sure what "pyrepl" is or to "use" it, but just looking at web
> search results, it seems to be connected to pypy, which is in the list
> of python-shell-completion-native-disabled-interpreters, could that be
> related?

  Not really but it makes allowances on whether or not pypy.py is
  currently loaded.

  Here is an important piece of pyrepl that appears to be revelent
  to how it deals with input and output from terminals (TTYs) and
  non-terminals:


    if '__pypy__' in sys.builtin_module_names:    # PyPy

        def _old_raw_input(prompt=''):
            # sys.__raw_input__() is only called when stdin and stdout are
            # as expected and are ttys.  If it is the case, then get_reader()
            # should not really fail in _wrapper.raw_input().  If it still
            # does, then we will just cancel the redirection and call again
            # the built-in raw_input().
            try:
                del sys.__raw_input__
            except AttributeError:
                pass
            return raw_input(prompt)
        sys.__raw_input__ = _wrapper.raw_input

    else:
        # this is not really what readline.c does.  Better than nothing I guess
        import __builtin__
        _old_raw_input = __builtin__.raw_input
        __builtin__.raw_input = _wrapper.raw_input

_old_raw_input = None


  I think that this related in the fact that the Emacs interaction is
  not with a real TTY (it doesn't have defined or tigetstr-retrievable
  terminal capabilities (curses) like how to do operations like "clear",
  "cup" or "horizontal" positioning).

  Thanks.

bug#25753: 25.2; Python mode shell interaction not working 100%

[npostavs]

Live System User <nyc4bos@aol.com> writes:
>     I was already using that (wrapped in a "with-eval-after-load
>     'python") from:
>
>     https://github.com/jorgenschaefer/elpy/issues/887#issuecomment-275175119
>
>     to no avail.

Can you test the python code from python-shell-completion-native-setup
outside of Emacs, e.g., save it to a file called 'native-completion.py'
and then run 'python -i native-completion.py' and then type an
underscore and hit <tab>.

With 2.7.13, I get

    $ python2 -i native-completion.py 
    python.el: native completion setup loaded
    >>> ___package__
    __PYTHON_EL_native_completion_setup
    __name__
    __doc__
    __import__
    __debug__

    0__dummy_completion__  1__dummy_completion__  
    >>> _

With python 3.6.0 I get:

    $ python -i native-completion.py 
    python.el: native completion setup loaded
    >>> ___name__
    __doc__
    __package__
    __loader__
    __spec__
    __annotations__
    __cached__
    __PYTHON_EL_native_completion_setup
    __build_class__
    __import__
    __debug__

    0__dummy_completion__  1__dummy_completion__  
    >>> _

bug#25753: 25.2; Python mode shell interaction not working 100%

[Live System User]

npostavs@users.sourceforge.net writes:

> Live System User <nyc4bos@aol.com> writes:
>>     I was already using that (wrapped in a "with-eval-after-load
>>     'python") from:
>>
>>     https://github.com/jorgenschaefer/elpy/issues/887#issuecomment-275175119
>>
>>     to no avail.
>
> Can you test the python code from python-shell-completion-native-setup
> outside of Emacs, e.g., save it to a file called 'native-completion.py'
> and then run 'python -i native-completion.py' and then type an
> underscore and hit <tab>.
>
> With 2.7.13, I get
>
>     $ python2 -i native-completion.py 
>     python.el: native completion setup loaded
>     >>> ___package__
>     __PYTHON_EL_native_completion_setup
>     __name__
>     __doc__
>     __import__
>     __debug__
>
>     0__dummy_completion__  1__dummy_completion__  
>     >>> _

      I get the same:

$ ▸ python2 -i native-completion.py
python.el: native completion setup loaded
>>> ___package__
__PYTHON_EL_native_completion_setup
__name__
__doc__
__import__
__debug__

0__dummy_completion__  1__dummy_completion__  
>>> _

    In Emacs I still get the (visable) *Warning* buffer with contents:

Warning (python): Your ‘python-shell-interpreter’ doesn’t seem to support readline, yet ‘python-shell-completion-native’ was t and "python" is not part of the ‘python-shell-completion-native-disabled-interpreters’ list.  Native completions have been disabled locally. 


>
> With python 3.6.0 I get:
>
>     $ python -i native-completion.py 
>     python.el: native completion setup loaded
>     >>> ___name__
>     __doc__
>     __package__
>     __loader__
>     __spec__
>     __annotations__
>     __cached__
>     __PYTHON_EL_native_completion_setup
>     __build_class__
>     __import__
>     __debug__
>
>     0__dummy_completion__  1__dummy_completion__  
>     >>> _

      With Python 3.5.2 I get something a little different:

liveuser@localhost:~$ ▸  python3 -i native-completion.py
python.el: native completion setup loaded
>>> ___loader__
__spec__
__cached__
__name__
__package__
__PYTHON_EL_native_completion_setup
__doc__
__build_class__
__import__
__debug__

0__dummy_completion__  1__dummy_completion__  
>>> _

    Thanks.
    

bug#25753: 25.2; Python mode shell interaction not working 100%

[npostavs]

[-- Attachment #1: Type: text/plain, Size: 797 bytes --]


Live System User <nyc4bos@aol.com> writes:

> npostavs@users.sourceforge.net writes:
>
>> Can you test the python code from python-shell-completion-native-setup
>> outside of Emacs, e.g., save it to a file called 'native-completion.py'
>> and then run 'python -i native-completion.py' and then type an
>> underscore and hit <tab>.
>>
>
>       I get the same:
>
> $ ▸ python2 -i native-completion.py
> python.el: native completion setup loaded
>>>> ___package__
> __PYTHON_EL_native_completion_setup
> __name__
> __doc__
> __import__
> __debug__
>
> 0__dummy_completion__  1__dummy_completion__  
>>>> _

Can you try load the attached file in Emacs?  Does it show similar
output in the "*py native complete test*" buffer?

    emacs -Q -l py-native-completion.el


[-- Attachment #2: test elisp file --]
[-- Type: application/octet-stream, Size: 4776 bytes --]

(require 'python)

(with-current-buffer (get-buffer-create "*py native complete test*")
  (let ((proc (start-process "python" (current-buffer) "python2" "-i")))
    (python-shell-send-string "
def __PYTHON_EL_native_completion_setup():
    try:
        import readline

        try:
            import __builtin__
        except ImportError:
            # Python 3
            import builtins as __builtin__

        builtins = dir(__builtin__)
        is_ipython = ('__IPYTHON__' in builtins or
                      '__IPYTHON__active' in builtins)

        class __PYTHON_EL_Completer:
            '''Completer wrapper that prints candidates to stdout.

            It wraps an existing completer function and changes its behavior so
            that the user input is unchanged and real candidates are printed to
            stdout.

            Returned candidates are '0__dummy_completion__' and
            '1__dummy_completion__' in that order ('0__dummy_completion__' is
            returned repeatedly until all possible candidates are consumed).

            The real candidates are printed to stdout so that they can be
            easily retrieved through comint output redirect trickery.
            '''

            PYTHON_EL_WRAPPED = True

            def __init__(self, completer):
                self.completer = completer
                self.last_completion = None
                self.print_mode = True

            def __call__(self, text, state):
                if state == 0:
                    # Set the first dummy completion.
                    self.last_completion = None
                    completion = '0__dummy_completion__'
                else:
                    completion = self.completer(text, state - 1)

                if not completion:
                    if self.last_completion != '1__dummy_completion__':
                        # When no more completions are available, returning a
                        # dummy with non-sharing prefix allow ensuring output
                        # while preventing changes to current input.
                        # Coincidentally it's also the end of output.
                        completion = '1__dummy_completion__'
                elif completion.endswith('('):
                    # Remove parens on callables as it breaks completion on
                    # arguments (e.g. str(Ari<tab>)).
                    completion = completion[:-1]
                self.last_completion = completion

                if completion in (
                        '0__dummy_completion__', '1__dummy_completion__'):
                    return completion
                elif completion:
                    # For every non-dummy completion, return a repeated dummy
                    # one and print the real candidate so it can be retrieved
                    # by comint output filters.
                    if self.print_mode:
                        print (completion)
                        return '0__dummy_completion__'
                    else:
                        return completion
                else:
                    return completion

        completer = readline.get_completer()

        if not completer:
            # Used as last resort to avoid breaking customizations.
            import rlcompleter
            completer = readline.get_completer()

        if completer and not getattr(completer, 'PYTHON_EL_WRAPPED', False):
            # Wrap the existing completer function only once.
            new_completer = __PYTHON_EL_Completer(completer)
            if not is_ipython:
                readline.set_completer(new_completer)
            else:
                # Try both initializations to cope with all IPython versions.
                # This works fine for IPython 3.x but not for earlier:
                readline.set_completer(new_completer)
                # IPython<3 hacks readline such that `readline.set_completer`
                # won't work.  This workaround injects the new completer
                # function into the existing instance directly:
                instance = getattr(completer, 'im_self', completer.__self__)
                instance.rlcomplete = new_completer

        if readline.__doc__ and 'libedit' in readline.__doc__:
            readline.parse_and_bind('bind ^I rl_complete')
        else:
            readline.parse_and_bind('tab: complete')
            # Require just one tab to send output.
            readline.parse_and_bind('set show-all-if-ambiguous on')

        print ('python.el: native completion setup loaded')
    except:
        print ('python.el: native completion setup failed')

__PYTHON_EL_native_completion_setup()" proc)
    (process-send-string proc "_\t"))
  (display-buffer (current-buffer)))

bug#25753: 25.2; Python mode shell interaction not working 100%

[Live System User]

[-- Attachment #1: Type: text/plain, Size: 246 bytes --]

npostavs@users.sourceforge.net writes:

[...]
>
> Can you try load the attached file in Emacs?  Does it show similar
> output in the "*py native complete test*" buffer?
>
>     emacs -Q -l py-native-completion.el

      Attached are my results.


[-- Attachment #2: py-native-completion results --]
[-- Type: application/octet-stream, Size: 907 bytes --]

Python 2.7.13 (default, Jan 13 2017, 10:15:16) 
[GCC 6.3.1 20161221 (Red Hat 6.3.1-1)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> python.el: native completion setup loaded
>>> _elementtree
_randommodule
_multibytecodecmodule
_codecs_tw
_ctypes
_hotshot
_sre
_heapq
_ast
_bisectmodule
_socketmodule
__code
_weakrefset
_collectionsmodule
_codecs_iso2022
_localemodule
_tkinter
__pyfile
_LWPCookieJar
_osx_support
_csv
_curses
_codecs_hk
_hashlib
_codecs
_ssl
_sqlite3
_curses_panel
_io
_strptime
_json
_lsprof
_cryptmodule
_MozillaCookieJar
__doc__
_codecs_kr
__package__
_functoolsmodule
_symtable
_weakref
_dbus_glib_bindings
_multiprocessing
__name__
__PYTHON_EL_native_completion_setup
_threading_local
_codecs_jp
_warnings
_bsddb
_codecs_cn
_pyio
_dbus_bindings
_sysconfigdata
_struct
_abcoll
__import__
__debug__

0__dummy_completion__  1__dummy_completion__  

bug#25753: 25.2; Python mode shell interaction not working 100%

[npostavs]

Live System User <nyc4bos@aol.com> writes:

> npostavs@users.sourceforge.net writes:
>
> [...]
>>
>> Can you try load the attached file in Emacs?  Does it show similar
>> output in the "*py native complete test*" buffer?
>>
>>     emacs -Q -l py-native-completion.el
>
>       Attached are my results.

Hmm, that looks like it should be working.  Wait, when you said

    I was already using that (wrapped in a "with-eval-after-load
    'python") from:

    https://github.com/jorgenschaefer/elpy/issues/887#issuecomment-275175119

    to no avail.

You meant the code from that exact comment?  That's a broken solution,
try this instead:

    (with-eval-after-load 'python
      (defun python-shell-completion-native-try ()
        "Return non-nil if can trigger native completion."
        (let ((python-shell-completion-native-enable t)
              (python-shell-completion-native-output-timeout
               python-shell-completion-native-try-output-timeout))
          (python-shell-completion-native-get-completions
           (get-buffer-process (current-buffer))
           nil "_"))))

If it's still not working, please post the contents of buffer " *Python
completions redirect*" (note the leading space).

bug#25753: 25.2; Python mode shell interaction not working 100%

[Live System User]

npostavs@users.sourceforge.net writes:

> Live System User <nyc4bos@aol.com> writes:
>
>> npostavs@users.sourceforge.net writes:
>>
>> [...]
>>>
>>> Can you try load the attached file in Emacs?  Does it show similar
>>> output in the "*py native complete test*" buffer?
>>>
>>>     emacs -Q -l py-native-completion.el
>>
>>       Attached are my results.
>
> Hmm, that looks like it should be working.  Wait, when you said
>
>     I was already using that (wrapped in a "with-eval-after-load
>     'python") from:
>
>     https://github.com/jorgenschaefer/elpy/issues/887#issuecomment-275175119
>
>     to no avail.
>
> You meant the code from that exact comment?  That's a broken solution,
> try this instead:
>
>     (with-eval-after-load 'python
>       (defun python-shell-completion-native-try ()
>         "Return non-nil if can trigger native completion."
>         (let ((python-shell-completion-native-enable t)
>               (python-shell-completion-native-output-timeout
>                python-shell-completion-native-try-output-timeout))
>           (python-shell-completion-native-get-completions
>            (get-buffer-process (current-buffer))
>            nil "_"))))

    Your version of the solution works for me.

    Thanks for your effort!.


>
> If it's still not working, please post the contents of buffer " *Python
> completions redirect*" (note the leading space).

bug#25753: 25.2; Python mode shell interaction not working 100%

[npostavs]

npostavs@users.sourceforge.net writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>>> From: charles@aurox.ch (Charles A. Roelli)
>>> Date: Thu, 16 Feb 2017 17:07:48 +0100
>>> 
>>> In emacs 25.2 (rc1) and 25.1, interaction with the Python shell does not
>>> seem to be working fully.  Python version is '2.7.12 (v2.7.12:d33e0cf91556,
>>> Jun 26 2016, 12:10:39) [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)]', and
>>> this is on Snow Leopard.
>>
>> Please try the next RC (should be out in a few days), I think we fixed
>> that there.
>
> AFAIK, the only change since rc1 is to add "ipython" to
> python-shell-completion-native-disabled-interpreters.  And that should
> not affect the problem reported here, which does not seem to involve
> ipython.

As I've said, I don't think rc2 would change this, but since it's come
out now, please test it.  Assuming rc2 still has the problem, please try
the tests I posted in
https://debbugs.gnu.org/cgi/bugreport.cgi?bug=25753#29 and
https://debbugs.gnu.org/cgi/bugreport.cgi?bug=25753#35.  Also post the
contents of buffer " *Python completions redirect*" after doing M-x
run-python.

bug#25753: 25.2; Python mode shell interaction not working 100%

[Charles A. Roelli]

> As I've said, I don't think rc2 would change this, but since it's come
> out now, please test it. 

Tested, and it has the same problem.  Here is what *Python* normally
looks like at the start, run from M-x run-python in emacs -Q:

Python 2.7.12 (v2.7.12:d33e0cf91556, Jun 26 2016, 12:10:39) 
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import codecs, os;__pyfile = codecs.open('''/var/folders/WP/WPe0Q1iAGc0J7iI6J50jcU+++TI/-Tmp-/py16611qgD''', encoding='''utf-8''');__code = __pyfile.read().encode('''utf-8''');__pyfile.close();os.remove('''/var/folders/WP/WPe0Q1iAGc0J7iI6J50jcU+++TI/-Tmp-/py16611qgD''');exec(compile(__code, '''/var/folders/WP/WPe0Q1iAGc0J7iI6J50jcU+++TI/-Tmp-/py16611qgD''', 'exec'));
python.el: native completion setup loaded

It seems like the line starting with ">>> " should not be printed, if
<nyc4bos@aol.com>'s test output is deemed to be running more or less
correctly.  Maybe that can point us in the right direction?

> Can you test the python code from python-shell-completion-native-setup
> outside of Emacs, e.g., save it to a file called 'native-completion.py'
> and then run 'python -i native-completion.py' and then type an
> underscore and hit <tab>.

I get the following:

>>> ___package__
__PYTHON_EL_native_completion_setup
__name__
__doc__
__import__
__debug__

Interestingly, none of the dummy completions pop up.

> Can you try load the attached file in Emacs?  Does it show similar
> output in the "*py native complete test*" buffer?
> 
>     emacs -Q -l py-native-completion.el

I get no completions with this -- here is all that shows up:

Python 2.7.12 (v2.7.12:d33e0cf91556, Jun 26 2016, 12:10:39) 
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> python.el: native completion setup loaded
>>> 

> Also post the contents of buffer " *Python completions redirect*"
> after doing M-x run-python.

- emacs -Q
- M-x run-python
- C-x b " *Python completions redirect*"

It's this: "_	^H^H" (space in the middle is a tab character).

bug#25753: 25.2; Python mode shell interaction not working 100%

[npostavs]

[-- Attachment #1: Type: text/plain, Size: 1364 bytes --]

charles@aurox.ch (Charles A. Roelli) writes:
>
> Tested, and it has the same problem.  Here is what *Python* normally
> looks like at the start, run from M-x run-python in emacs -Q:
>
> Python 2.7.12 (v2.7.12:d33e0cf91556, Jun 26 2016, 12:10:39) 
> [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
> Type "help", "copyright", "credits" or "license" for more information.
>>>> import codecs, os;__pyfile =
>>>> codecs.open('''/var/folders/WP/WPe0Q1iAGc0J7iI6J50jcU+++TI/-Tmp-/py16611qgD''',
>>>> encoding='''utf-8''');__code =
>>>> __pyfile.read().encode('''utf-8''');__pyfile.close();os.remove('''/var/folders/WP/WPe0Q1iAGc0J7iI6J50jcU+++TI/-Tmp-/py16611qgD''');exec(compile(__code,
>>>> '''/var/folders/WP/WPe0Q1iAGc0J7iI6J50jcU+++TI/-Tmp-/py16611qgD''',
>>>> 'exec'));
> python.el: native completion setup loaded
>
> It seems like the line starting with ">>> " should not be printed, if
> <nyc4bos@aol.com>'s test output is deemed to be running more or less
> correctly.  Maybe that can point us in the right direction?

Hmm, it's odd, though since you're still getting the "setup loaded"
message, the code *is* getting evaluated anyway, so it's likely that
this problem is not directly related.  Let's see if we can track it down
anyway, try the loading the attached as

    emacs -Q -l py-trace-bad-output.el

and see if anything shows up in *Messages*.


[-- Attachment #2: tracing code --]
[-- Type: application/emacs-lisp, Size: 595 bytes --]

[-- Attachment #3: Type: text/plain, Size: 679 bytes --]


>> Can you test the python code from python-shell-completion-native-setup
>> outside of Emacs, e.g., save it to a file called 'native-completion.py'
>> and then run 'python -i native-completion.py' and then type an
>> underscore and hit <tab>.
>
> I get the following:
>
>>>> ___package__
> __PYTHON_EL_native_completion_setup
> __name__
> __doc__
> __import__
> __debug__
>
> Interestingly, none of the dummy completions pop up.

This seems to be the core of the problem.  I gather that macOS uses
libedit instead of readline by default, perhaps that is the source of
incompatibility.  Can you figure out how to change the python code so
that the dummy completions do show up?

bug#25753: 25.2; Python mode shell interaction not working 100%

[Charles A. Roelli]

On Thu, Feb 23 2017 at 09:20:01 am, npostavs@users.sourceforge.net wrote:

> charles@aurox.ch (Charles A. Roelli) writes:
>>
>> Tested, and it has the same problem.  Here is what *Python* normally
>> looks like at the start, run from M-x run-python in emacs -Q:
>>
>> Python 2.7.12 (v2.7.12:d33e0cf91556, Jun 26 2016, 12:10:39) 
>> [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
>> Type "help", "copyright", "credits" or "license" for more information.
>>>>> import codecs, os;__pyfile =
>>>>> codecs.open('''/var/folders/WP/WPe0Q1iAGc0J7iI6J50jcU+++TI/-Tmp-/py16611qgD''',
>>>>> encoding='''utf-8''');__code =
>>>>> __pyfile.read().encode('''utf-8''');__pyfile.close();os.remove('''/var/folders/WP/WPe0Q1iAGc0J7iI6J50jcU+++TI/-Tmp-/py16611qgD''');exec(compile(__code,
>>>>> '''/var/folders/WP/WPe0Q1iAGc0J7iI6J50jcU+++TI/-Tmp-/py16611qgD''',
>>>>> 'exec'));
>> python.el: native completion setup loaded
>>
>> It seems like the line starting with ">>> " should not be printed, if
>> <nyc4bos@aol.com>'s test output is deemed to be running more or less
>> correctly.  Maybe that can point us in the right direction?
>
> Hmm, it's odd, though since you're still getting the "setup loaded"
> message, the code *is* getting evaluated anyway, so it's likely that
> this problem is not directly related.  Let's see if we can track it down
> anyway, try the loading the attached as
>
>     emacs -Q -l py-trace-bad-output.el
>
> and see if anything shows up in *Messages*.
>
> (require 'python)
>
> (advice-add
>  'python-shell-make-comint :filter-return
>  (lambda (proc-buffer-name)
>    (with-current-buffer proc-buffer-name
>      (add-hook 'after-change-functions
>                (lambda (beg end len)
>                  (ignore-errors
>                    (save-excursion
>                      (save-match-data
>                        (when (string-prefix-p "import codecs" (buffer-substring beg end))
>                          (backtrace))))))
>                nil t))
>    proc-buffer-name))
>
> (run-python)
>
> (switch-to-buffer "*Python*")
> (display-buffer "*Messages*")
>

Here is what I got:

Loading ~/Downloads/py-trace-bad-output.el (source)...done
  backtrace()
  (progn (backtrace))
  (if (string-prefix-p "import codecs" (buffer-substring beg end)) (progn (backtrace)))
  (progn (if (string-prefix-p "import codecs" (buffer-substring beg end)) (progn (backtrace))))
  (unwind-protect (progn (if (string-prefix-p "import codecs" (buffer-substring beg end)) (progn (backtrace)))) (set-match-data save-match-data-internal (quote evaporate)))
  (let ((save-match-data-internal (match-data))) (unwind-protect (progn (if (string-prefix-p "import codecs" (buffer-substring beg end)) (progn (backtrace)))) (set-match-data save-match-data-internal (quote evaporate))))
  (save-excursion (let ((save-match-data-internal (match-data))) (unwind-protect (progn (if (string-prefix-p "import codecs" (buffer-substring beg end)) (progn (backtrace)))) (set-match-data save-match-data-internal (quote evaporate)))))
  (progn (save-excursion (let ((save-match-data-internal (match-data))) (unwind-protect (progn (if (string-prefix-p "import codecs" (buffer-substring beg end)) (progn (backtrace)))) (set-match-data save-match-data-internal (quote evaporate))))))
  (condition-case nil (progn (save-excursion (let ((save-match-data-internal (match-data))) (unwind-protect (progn (if (string-prefix-p "import codecs" ...) (progn ...))) (set-match-data save-match-data-internal (quote evaporate)))))) (error nil))
  (lambda (beg end len) (condition-case nil (progn (save-excursion (let ((save-match-data-internal (match-data))) (unwind-protect (progn (if ... ...)) (set-match-data save-match-data-internal (quote evaporate)))))) (error nil)))(191 556 0)
  comint-output-filter(#<process Python> "import codecs, os;__pyfile = codecs.open('''/var/folders/WP/WPe0Q1iAGc0J7iI6J50jcU+++TI/-Tmp-/py683UGH''', encoding='''utf-8''');__code = __pyfile.read().encode('''utf-8''');__pyfile.close();os.remove('''/var/folders/WP/WPe0Q1iAGc0J7iI6J50jcU+++TI/-Tmp-/py683UGH''');exec(compile(__code, '''/var/folders/WP/WPe0Q1iAGc0J7iI6J50jcU+++TI/-Tmp-/py683UGH''', 'exec'));
")
  accept-process-output(#<process Python> 1.0)
  python-shell-accept-process-output(#<process Python> 1.0)
  python-shell-completion-native-setup()
  python-shell-completion-native-turn-on-maybe(t)
  python-shell-completion-native-turn-on-maybe-with-msg()
  run-hooks(python-shell-first-prompt-hook)
  python-shell-comint-watch-for-first-prompt-output-filter(">>> ")
  run-hook-with-args(python-shell-comint-watch-for-first-prompt-output-filter ">>> ")
  comint-output-filter(#<process Python> ">>> ")
  read-event(nil t 2)
  sit-for(2)
  execute-extended-command(nil "load-file" "load-file")
  funcall-interactively(execute-extended-command nil "load-file" "load-file")
  call-interactively(execute-extended-command nil nil)
  command-execute(execute-extended-command)

Shell native completion is disabled, using fallback

>>> Can you test the python code from python-shell-completion-native-setup
>>> outside of Emacs, e.g., save it to a file called 'native-completion.py'
>>> and then run 'python -i native-completion.py' and then type an
>>> underscore and hit <tab>.
>>
>> I get the following:
>>
>>>>> ___package__
>> __PYTHON_EL_native_completion_setup
>> __name__
>> __doc__
>> __import__
>> __debug__
>>
>> Interestingly, none of the dummy completions pop up.
>
> This seems to be the core of the problem.  I gather that macOS uses
> libedit instead of readline by default, perhaps that is the source of
> incompatibility.  Can you figure out how to change the python code so
> that the dummy completions do show up?

Yes, I will look into it.

bug#25753: 25.2; Python mode shell interaction not working 100%

[Charles A. Roelli]

Could you please evaluate these forms with a running instance of
Python in Emacs?  My output follows each form.  It seems like the Mac
version of Python echoes the last command sent, and maybe python.el
does not expect this?  Notice in the third example, the correct result
appears, but is preceded by the code sent to the interpreter (which
should be omitted as garbage).

(python-shell-send-string-no-output "import os\nimport sys")
=> "import codecs, os;__pyfile = codecs.open('''/var/folders/WP/WPe0Q1iAGc0J7iI6J50jcU+++TI/-Tmp-/py167Tvh''', encoding='''utf-8''');__code = __pyfile.read().encode('''utf-8''');__pyfile.close();os.remove('''/var/folders/WP/WPe0Q1iAGc0J7iI6J50jcU+++TI/-Tmp-/py167Tvh''');exec(compile(__code, '''/var/folders/WP/WPe0Q1iAGc0J7iI6J50jcU+++TI/-Tmp-/py167Tvh''', 'exec'));"

(python-shell-send-string-no-output "import os; import sys")
=> "import os; import sys"

(python-ffap-module-path "os")
=> "import codecs, os;__pyfile = codecs.open('''/var/folders/WP/WPe0Q1iAGc0J7iI6J50jcU+++TI/-Tmp-/py167tDu''', encoding='''utf-8''');__code = __pyfile.read().encode('''utf-8''');__pyfile.close();os.remove('''/var/folders/WP/WPe0Q1iAGc0J7iI6J50jcU+++TI/-Tmp-/py167tDu''');exec(compile(__code, '''/var/folders/WP/WPe0Q1iAGc0J7iI6J50jcU+++TI/-Tmp-/py167tDu''', 'exec'));^M
/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/os.py"

bug#25753: 25.2; Python mode shell interaction not working 100%

[npostavs]

charles@aurox.ch (Charles A. Roelli) writes:

> Could you please evaluate these forms with a running instance of
> Python in Emacs?  My output follows each form.  It seems like the Mac
> version of Python echoes the last command sent, and maybe python.el
> does not expect this?

Yes, I was going to ask if you also get this echoing for commands typed
in at the prompt or is it just the support functions?  Possibly setting
`comint-process-echoes' could help, though I don't understand why there
is echoing in the first place.

Here is what I get:

    (python-shell-send-string-no-output "import os\nimport sys") ;=> ""
    (python-shell-send-string-no-output "import os; import sys") ;=> ""
    (python-ffap-module-path "os") ;=> "/usr/lib/python3.6/os.py"

By the way, it was reported[1] that using "homebrew" python avoids these
issues, apparently that build uses GNU readline instead of libedit[2].

[1]: https://github.com/jorgenschaefer/elpy/issues/887#issuecomment-282215656
[2]: https://github.com/jorgenschaefer/elpy/issues/887#issuecomment-282332143

bug#25753: 25.2; Python mode shell interaction not working 100%

[Charles A. Roelli]

On Sat, Feb 25 2017 at 09:34:10 am, npostavs@users.sourceforge.net wrote:

> charles@aurox.ch (Charles A. Roelli) writes:
>
>> Could you please evaluate these forms with a running instance of
>> Python in Emacs?  My output follows each form.  It seems like the Mac
>> version of Python echoes the last command sent, and maybe python.el
>> does not expect this?
>
> Yes, I was going to ask if you also get this echoing for commands typed
> in at the prompt or is it just the support functions?  

Seems to be everywhere.

> Possibly setting `comint-process-echoes' could help, though I don't
> understand why there is echoing in the first place.

Thanks for the pointer to that variable.  I ran this:

   (add-hook 'inferior-python-mode-hook (lambda () (setq comint-process-echoes t)))

and the commands at the prompt stopped echoing, but the support
functions still echoed.  So it looks like the problem has to be fixed on
the readline/libedit side of Python.

> Here is what I get:
>
>     (python-shell-send-string-no-output "import os\nimport sys") ;=> ""
>     (python-shell-send-string-no-output "import os; import sys") ;=> ""
>     (python-ffap-module-path "os") ;=> "/usr/lib/python3.6/os.py"

Thanks.  I managed to fix my setup to give equivalent results (see below).

> By the way, it was reported[1] that using "homebrew" python avoids these
> issues, apparently that build uses GNU readline instead of libedit[2].
>
> [1]: https://github.com/jorgenschaefer/elpy/issues/887#issuecomment-282215656
> [2]: https://github.com/jorgenschaefer/elpy/issues/887#issuecomment-282332143

Thanks for setting me on this trail.  It turns out there's a package you
can install via `easy_install' (part of Python's `setuptools') called
`gnureadline' [1] (formerly called just `readline') which is supposed to
shadow python2.7/lib-dynload/'s `readline.so' with a relatively
up-to-date statically linked GNU replacement (for those of us with a
wacky `libedit'-ized version).  So it would seem that `sudo easy_install
gnureadline' is the right thing to run.  However that still does not fix
the issue because python2.7/lib-dynload/ comes before
python2.7/site-packages/ (where `gnureadline' is installed) in Python's
$PATH equivalent, `sys.path'.  The solution, then, is to reorder the
path somehow or get `readline.so' out of the way (maybe by renaming it
-- cleaner suggestions welcome).  I haven't tested that yet, but it
should work as expected.

At the moment I've been running M-x run-python from the
python2.7/site-packages/ folder where `readline.py' is stored, since
Python adds the path of the current directory to the front of
`sys.path'.  That means `readline.py' gets picked from there, which is
convenient, if not a little surprising at first.  With that done, the
forms I posted previously evaluate as expected, without the code
echoing.  Eldoc also works fine.  Maybe we can add a notice about this
somewhere in python.el in emacs-25.  I'm not sure yet about the best way
to handle the path ordering issue between `readline.so' and
`gnureadline'.

[1] https://pypi.python.org/pypi/gnureadline

bug#25753: 25.2; Python mode shell interaction not working 100%

[npostavs]

charles@aurox.ch (Charles A. Roelli) writes:

> On Sat, Feb 25 2017 at 09:34:10 am, npostavs@users.sourceforge.net wrote:
>
>> Possibly setting `comint-process-echoes' could help, though I don't
>> understand why there is echoing in the first place.
>
> Thanks for the pointer to that variable.  I ran this:
>
>    (add-hook 'inferior-python-mode-hook (lambda () (setq comint-process-echoes t)))
>
> and the commands at the prompt stopped echoing, but the support
> functions still echoed.  So it looks like the problem has to be fixed on
> the readline/libedit side of Python.

This thread[1] might be somewhat relevant:

    The problem is that eshell tells subprocesses that they're running in
    a terminal (e.g., when queried via hIsTerminalDevice), but always
    echos user input itself regardless of the tty's ECHO attribute. This
    confuses libedit, which assumes that if it's connected to a terminal
    then it can turn off echoing in order to run its own rich line editor.

[1]: http://glasgow-haskell-users.haskell.narkive.com/vyeVJUEB/problem-with-echo-prompting-in-ghci-visible-in-emacs

>  So it would seem that `sudo easy_install
> gnureadline' is the right thing to run.  However that still does not fix
> the issue because python2.7/lib-dynload/ comes before
> python2.7/site-packages/ (where `gnureadline' is installed) in Python's
> $PATH equivalent, `sys.path'.  The solution, then, is to reorder the
> path somehow or get `readline.so' out of the way (maybe by renaming it
> -- cleaner suggestions welcome).

I guess renaming should have the least amount of side-effects.

bug#25753: 25.2; Python mode shell interaction not working 100%

[Charles A. Roelli]

On Sun, Feb 26 2017 at 09:14:45 pm, npostavs@users.sourceforge.net wrote:

> charles@aurox.ch (Charles A. Roelli) writes:
>
>> On Sat, Feb 25 2017 at 09:34:10 am, npostavs@users.sourceforge.net wrote:
>>
>>> Possibly setting `comint-process-echoes' could help, though I don't
>>> understand why there is echoing in the first place.
>>
>> Thanks for the pointer to that variable.  I ran this:
>>
>>    (add-hook 'inferior-python-mode-hook (lambda () (setq comint-process-echoes t)))
>>
>> and the commands at the prompt stopped echoing, but the support
>> functions still echoed.  So it looks like the problem has to be fixed on
>> the readline/libedit side of Python.
>
> This thread[1] might be somewhat relevant:
>
>     The problem is that eshell tells subprocesses that they're running in
>     a terminal (e.g., when queried via hIsTerminalDevice), but always
>     echos user input itself regardless of the tty's ECHO attribute. This
>     confuses libedit, which assumes that if it's connected to a terminal
>     then it can turn off echoing in order to run its own rich line editor.
>
> [1]:
> http://glasgow-haskell-users.haskell.narkive.com/vyeVJUEB/problem-with-echo-prompting-in-ghci-visible-in-emacs

Thanks.  FWIW, I tried this approach:

> Given this info, there's a fairly easy emacs haskell-mode work-around. I
> made a shell script "ghci-no-tty" in my ~/bin that contains
> 
> # So ghci+readline won't echo input
> cat | /usr/local/bin/ghci $*
> 
> and used "M-x customize-group" with the "haskell" group to set the "Haskell
> Program Name" variable to "/home/conal/bin/ghci-no-tty" (must be full path).
> 
> Now there's no more input echoing, and commands like automatic signature
> insertion ("\C-c\C-t") work again.

using `python-shell-interpreter' and `cat | python $'.  Python exited
immediately with code 126, so I guess this solution does not work here.
But I can't claim to understand any issues having to do with TTYs/"dumb
terminals", so maybe I am missing something.

>>  So it would seem that `sudo easy_install
>> gnureadline' is the right thing to run.  However that still does not fix
>> the issue because python2.7/lib-dynload/ comes before
>> python2.7/site-packages/ (where `gnureadline' is installed) in Python's
>> $PATH equivalent, `sys.path'.  The solution, then, is to reorder the
>> path somehow or get `readline.so' out of the way (maybe by renaming it
>> -- cleaner suggestions welcome).
>
> I guess renaming should have the least amount of side-effects.

Great.  This is what works for me:

cd /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload
mv readline.so readline.so.bak

Hopefully we can add into emacs-25 a comment about this situation,
advising Mac OS X users to install `gnureadline' with `easy_install
gnureadline', then renaming `readline.so' to something not ending in
`.so', if the native completion does not work immediately.

bug#25753: 25.2; Python mode shell interaction not working 100%

[npostavs]

charles@aurox.ch (Charles A. Roelli) writes:

> using `python-shell-interpreter' and `cat | python $'.  Python exited
> immediately with code 126, so I guess this solution does not work here.
> But I can't claim to understand any issues having to do with TTYs/"dumb
> terminals", so maybe I am missing something.

The equivalent for python.el would be approximately

(defun my-python-shell-calculate-command (&rest _)
  "sh -c \"cat | python -i\"")
(advice-add 'python-shell-calculate-command :override
            #'my-python-shell-calculate-command)

But this breaks native completion and prompt detection, due to IO
buffering I think.

>>>  So it would seem that `sudo easy_install
>>> gnureadline' is the right thing to run.  However that still does not fix
>>> the issue because python2.7/lib-dynload/ comes before
>>> python2.7/site-packages/ (where `gnureadline' is installed) in Python's
>>> $PATH equivalent, `sys.path'.  The solution, then, is to reorder the
>>> path somehow or get `readline.so' out of the way (maybe by renaming it
>>> -- cleaner suggestions welcome).
>>
>> I guess renaming should have the least amount of side-effects.
>
> Great.  This is what works for me:
>
> cd /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload
> mv readline.so readline.so.bak
>
> Hopefully we can add into emacs-25 a comment about this situation,
> advising Mac OS X users to install `gnureadline' with `easy_install
> gnureadline', then renaming `readline.so' to something not ending in
> `.so', if the native completion does not work immediately.

Does this look okay?

--- i/etc/PROBLEMS
+++ w/etc/PROBLEMS
@@ -463,8 +463,25 @@ problem by adding this to your .cshrc file:
         unset edit
         stty -icrnl -onlcr -echo susp ^Z
     endif
 
+*** In Inferior Python mode, input is echoed and native completion doesn't work.
+<https://debbugs.gnu.org/cgi/bugreport.cgi?bug=25753>
+
+This happens when python uses a libedit based readline module, which
+is the default on macOS.  This can be worked around by installing a
+GNU readline based module instead, for example, using setuptools
+
+    sudo easy_install gnureadline
+
+And then rename the system's readline so that it won't be loaded:
+
+    cd /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload
+    mv readline.so readline.so.bak
+
+See <https://pypi.python.org/pypi/gnureadline> for more details on
+installation.
+
 *** Emacs startup on GNU/Linux systems (and possibly other systems) is slow.
 
 This can happen if the system is misconfigured and Emacs can't get the
 full qualified domain name, FQDN.  You should have your FQDN in the

bug#25753: 25.2; Python mode shell interaction not working 100%

[Eli Zaretskii]

> From: npostavs@users.sourceforge.net
> Cc: 25753@debbugs.gnu.org,  Eli Zaretskii <eliz@gnu.org>
> Date: Tue, 28 Feb 2017 09:07:02 -0500
> 
> Does this look okay?
> 
> --- i/etc/PROBLEMS
> +++ w/etc/PROBLEMS

Yes, thanks.

bug#25753: 25.2; Python mode shell interaction not working 100%

[npostavs]

retitle 25753 Python with libedit (macOS default) echoes input, breaks native completion
quit

Eli Zaretskii <eliz@gnu.org> writes:

>> From: npostavs@users.sourceforge.net
>> Cc: 25753@debbugs.gnu.org,  Eli Zaretskii <eliz@gnu.org>
>> Date: Tue, 28 Feb 2017 09:07:02 -0500
>> 
>> Does this look okay?
>> 
>> --- i/etc/PROBLEMS
>> +++ w/etc/PROBLEMS
>
> Yes, thanks.

Pushed to emacs-25 [1: 6e788ef0e2].

1: 2017-03-01 17:56:20 -0500 6e788ef0e262fafc014c21f4ad52cc5dc9f1715b
  ; etc/PROBLEMS: Explain about the python+libedit problem (Bug#25753).

bug#25753: 25.2; Python mode shell interaction not working 100%

[Carlos Pita]

I’m having this issue with emacs 28 in macOS Big Sur, my python interpreter was installed using brew and has readline support:

```
Python 3.9.7 (default, Sep  3 2021, 12:37:55)
[Clang 12.0.5 (clang-1205.0.22.9)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import readline; print (readline.__doc__)
Importing this module enables command line editing using GNU readline.
```

But:

```
Warning (python): Your ‘python-shell-interpreter’ doesn’t seem to support readline, yet ‘python-shell-completion-native-enable’ was t and "python3" is not part of the ‘python-shell-completion-native-disabled-interpreters’ list.  Native completions have been disabled locally.  Disable showing Disable logging
```

Is this expected? The thread above seems to suggest that it should work, but maybe I’m misreading it.

Thanks

bug#25753: 25.2; Python mode shell interaction not working 100%

[Carlos Pita]

BTW, one nasty consequence of this is that it completely freezes emacs
when trying to execute an org-babel python block.

bug#25753: 25.2; Python mode shell interaction not working 100%

[Carlos Pita]

Ok, after a lot of debugging I realized that my current readline
configuration was introducing some control characters that wreak havoc
with the native completion detection mechanism. Now, my .inputrc has:

> cat .inputrc
set completion-ignore-case on
set completion-display-width 80
set completion-prefix-display-length 5
set show-all-if-ambiguous on
set skip-completed-text on
set colored-stats on
set blink-matching-paren on
set menu-complete-display-prefix on

And I like it to be so. So there are a lot of cases to deal with here.
One could force a reset by setting INPUTRC=ieiowueqoiw or whatever,
but I've spent countless hours debugging all this native completion
stuff every couple of years, it's too fragile and too complex, do you
think it's worth the effort? ipython doesn't use readline anymore, the
trend is to move to jupyter which has a clean protocol to deal with
all this, why don't keep things simple and get rid of the readline
magic for good? Just an opinion.

bug#25753: 25.2; Python mode shell interaction not working 100%

[Carlos Pita]

Some years ago I proposed starting a thread python-side that provided
the completions through some kind of IPC, so as to not interfere with
prompt numbering and also to avoid blocking behavior (for example, if
eldoc tries to show documentation while python is running, it will
block or at least it would at the time). I started an implementation
then, sent a prototype to Fabian but it didn't get traction. I
reiterate my proposition now, even though I believe that's what
jupyter does nowadays in a much better way than any tiny
implementation of mine hacked into defcustoms will ever do.

bug#25753: 25.2; Python mode shell interaction not working 100%

[Carlos Pita]

Regarding my issue, short of resetting the entire inputrc config
somehow, one can workaround the problem by changing:

                  (comint-redirect-finished-regexp
                   "1__dummy_completion__[[:space:]]*\n")

to:
                  (comint-redirect-finished-regexp
                   "1__dummy_completion__.*\n")

This is in order to match outputs like:

    0__dummy_completion__ [0m [K  1__dummy_completion__ [0m [K

Given the fragility of all this, I think it's better not to be too
picky regarding the matching condition, which is selective enough
without enforcing spaces at the end. So I propose that minimal change.

bug#25753: 25.2; Python mode shell interaction not working 100%

[Augusto Stoffel]

On Sun,  3 Oct 2021 at 20:55, Carlos Pita <carlosjosepita@gmail.com> wrote:

> Some years ago I proposed starting a thread python-side that provided
> the completions through some kind of IPC, so as to not interfere with
> prompt numbering and also to avoid blocking behavior (for example, if
> eldoc tries to show documentation while python is running, it will
> block or at least it would at the time). I started an implementation
> then, sent a prototype to Fabian but it didn't get traction. I
> reiterate my proposition now, even though I believe that's what
> jupyter does nowadays in a much better way than any tiny
> implementation of mine hacked into defcustoms will ever do.

There exists an Emacs interface to Jupyter:

    https://github.com/nnicandro/emacs-jupyter

Unfortunately, this package is free of bugs and hasn't seen much
development lately, so I prefer to live with the issues of the good old
Python shell.

bug#25753: 25.2; Python mode shell interaction not working 100%

[Carlos Pita]

I filed a new issue for the configuration problem:
http://debbugs.gnu.org/cgi/bugreport.cgi?bug=51010

bug#25753: 25.2; Python mode shell interaction not working 100%

[Carlos Pita]

Hi Augusto,

> Unfortunately, this package is free of bugs and hasn't seen much
> development lately, so I prefer to live with the issues of the good old
> Python shell.

Just to be clear, I'm not saying python.el should move to jupyter or
anything like that. On the contrary, I believe it should provide a
solid focused basis for other modules (elpy, lsp, emacs-jupyter, etc)
and perhaps it's already somewhat at odds with that goal. Especially
regarding native completion I don't see much to be gained vs. directly
calling the readline completer (which is now considered the fallback
case) and OTOH there is something to be lost: at least in my
experience this has often been the non "just works" factor. Moreover,
the mechanism is far from perfect (it interferes with prompt
numbering, it's potentially blocking) and native completions solve
none of its issues. So I feel like getting rid of it. My point
regarding Jupyter, LSP and the emacs frameworks around them is that to
some extent they relieve python.el of having to be that smart.

Best regards,
Carlos

bug#25753: 25.2; Python mode shell interaction not working 100%

[Augusto Stoffel]

On Mon,  4 Oct 2021 at 12:31, Carlos Pita <carlosjosepita@gmail.com> wrote:

> Hi Augusto,
>
>> Unfortunately, this package is free of bugs and hasn't seen much
>> development lately, so I prefer to live with the issues of the good old
>> Python shell.

I meant emacs-jupyter _isn't_ free of bugs, in case that wasn't clear
from context :-).

> Just to be clear, I'm not saying python.el should move to jupyter or
> anything like that. On the contrary, I believe it should provide a
> solid focused basis for other modules (elpy, lsp, emacs-jupyter, etc)
> and perhaps it's already somewhat at odds with that goal.

I personally agree.

> Especially regarding native completion I don't see much to be gained
> vs. directly calling the readline completer (which is now considered
> the fallback case) and OTOH there is something to be lost: at least in
> my experience this has often been the non "just works"
> factor. Moreover, the mechanism is far from perfect (it interferes
> with prompt numbering, it's potentially blocking) and native
> completions solve none of its issues. So I feel like getting rid of
> it. My point regarding Jupyter, LSP and the emacs frameworks around
> them is that to some extent they relieve python.el of having to be
> that smart.

That's not an unreasonable proposal.  I'd be curious as to what the
maintainers think.  For the record, here are the only 2 advantages of
the "native completion" mechanism that I'm aware of:

1. It does _not_ interfere with prompt numbering and last return value
   variables `_`.

2. It works on continuation lines.

The following facts further diminish those two advantages:

A. Every other feature that sends commands to the inferior behind the
   scenes will interfere with prompt numbering.

B. Editing continuation lines is awkward for several other reasons.  For
   instance, each continuation line becomes a separate history entry.

bug#25753: 29.1; Python mode shell interaction not working 100%

[Peter Mao]

[-- Attachment #1.1: Type: text/plain, Size: 816 bytes --]

* Synopsis
  With 29.1 on OSX and GNU/Linux(Ubuntu), opening a python session in a
buffer for the
  second time in a buffer causes Emacs to hang.  (First mentioned in
  https://debbugs.gnu.org/cgi/bugreport.cgi?bug=25753#98).

  The warning message from
https://debbugs.gnu.org/cgi/bugreport.cgi?bug=25753#5
  appears on OSX, but not on Ubuntu.  As mentioned in that email, despite
the
  warning, completions *do* work.

  The major problem to my workflow is the hanging.  Versions 27 and 28 did
not
  have this problem.

* Solutions attempted that do not solve the issue for me
  1. `pip install gnureadline`
  2. https://debbugs.gnu.org/cgi/bugreport.cgi?bug=25753#41
  3. ~/.inputrc:  I don't use that on OSX, on Ubuntu, my config is minimal.

* Recipe, system info and .initrc are in the attached org-mode file.

[-- Attachment #1.2: Type: text/html, Size: 1147 bytes --]

[-- Attachment #2: emacs_bug_25753.org --]
[-- Type: application/vnd.lotus-organizer, Size: 11024 bytes --]

bug#25753: 29.1; Python mode shell interaction not working 100%

[Peter Mao]

[-- Attachment #1: Type: text/plain, Size: 1155 bytes --]

At least my version of the problem can be traced to a bug in Org's
ob-python.el.  I bisected the commit history, found the offending commit
and reported it to the org mailing list.

Peter


On Fri, Aug 11, 2023 at 10:55 AM Peter Mao <peter.mao@gmail.com> wrote:

> * Synopsis
>   With 29.1 on OSX and GNU/Linux(Ubuntu), opening a python session in a
> buffer for the
>   second time in a buffer causes Emacs to hang.  (First mentioned in
>   https://debbugs.gnu.org/cgi/bugreport.cgi?bug=25753#98).
>
>   The warning message from
> https://debbugs.gnu.org/cgi/bugreport.cgi?bug=25753#5
>   appears on OSX, but not on Ubuntu.  As mentioned in that email, despite
> the
>   warning, completions *do* work.
>
>   The major problem to my workflow is the hanging.  Versions 27 and 28 did
> not
>   have this problem.
>
> * Solutions attempted that do not solve the issue for me
>   1. `pip install gnureadline`
>   2. https://debbugs.gnu.org/cgi/bugreport.cgi?bug=25753#41
>   3. ~/.inputrc:  I don't use that on OSX, on Ubuntu, my config is minimal.
>
> * Recipe, system info and .initrc are in the attached org-mode file.
>

[-- Attachment #2: Type: text/html, Size: 1844 bytes --]

bug#25753: 29.1; Python mode shell interaction not working 100%

[Eli Zaretskii]

> From: Peter Mao <peter.mao@gmail.com>
> Date: Thu, 24 Aug 2023 22:32:20 -0700
> 
> At least my version of the problem can be traced to a bug in Org's ob-python.el.  I bisected the
> commit history, found the offending commit and reported it to the org mailing list.

Thanks, I'm therefore closing this bug.  Feel free to reopen if
something needs to be done on the Emacs core side.