bug#45938: 28.0.50; python native completion fails with fancier readline settings

bug#45938: 28.0.50; python native completion fails with fancier readline settings

[Zoltán Vandrus]

Three problematic options I found are:

   set colored-stats on
   set colored-completion-prefix on
   set enable-bracketed-paste on

With any of these M-x run-python warns:

   Warning (python): Your ‘python-shell-interpreter’ doesn’t seem to
   support readline, yet ‘python-shell-completion-native-enable’ was t and
   "python3" is not part of the
   ‘python-shell-completion-native-disabled-interpreters’ list.  Native
   completions have been disabled locally.
   

set colored-stats on
set colored-completion-prefix on
================================

Any of these on and *Python completions redirect* buffer ends with:
0__dummy_completion__^[[0m^[[K  1__dummy_completion__^[[0m^[[K

^[[0m and ^[[K are escape sequences not understood by comint mode I
think

Related bug: #24401 recommends setting colored-stats conditionally only
for bash, because filename coloring is not that useful for the python
shell.

Even then colored-completion-prefix is useful everywhere, emacs does it
by default too.

set enable-bracketed-paste on
=============================

Fails like half of the times for me. For a different reason.

First record output:
   (defun keep-output (output)
     (setq kept (cons output kept)))

   (add-hook 'inferior-python-mode-hook
             (lambda ()
               (setq kept nil)
               (add-hook 'comint-output-filter-functions #'keep-output)))

M-x run-python

kept’s value is
("python.el: native completion setup loaded
^[[?2004h>>> " "^[[?2004l^M" "Python 3.9.0 (default, Oct  7 2020, 23:09:01) 
[GCC 10.2.0] on linux
Type \"help\", \"copyright\", \"credits\" or \"license\" for more information.
^[[?2004h>>> ")

-----------------------

python-shell-completin-native-setup assumes that after running the
python script that sets up completion the first output will be from the script:
      (when (and
             (python-shell-accept-process-output
              process python-shell-completion-native-try-output-timeout)
             (save-excursion
               (re-search-backward
                (regexp-quote "python.el: native completion setup loaded") nil t 1)))
        (python-shell-completion-native-try)))))

What can happen sometimes is that, after the first initial prompt:
   "Python 3.9.0 (default, Oct  7 2020, 23:09:01) 
   [GCC 10.2.0] on linux
   Type \"help\", \"copyright\", \"credits\" or \"license\" for more information.
   ^[[?2004h>>> "
The python script is injected.
python-shell-accept-process-output waits for next prompt
Bracketed mode is disabled instead with clearing the line: "^[[?2004l^M"
Empty line is a valid python prompt, so it's accepted, and the regexp
test fails.

#41959 is another problem with trying to parse the readline output.

--------------------------------------------------

Readline version 8.1.0

In GNU Emacs 28.0.50 (build 4, x86_64-pc-linux-gnu, GTK+ Version 3.24.24, cairo version 1.17.4)
 of 2021-01-17 built on zoli-laptop
Repository revision: 1773679af3241919a85d6174b1554070a63cca79
Repository branch: master
Windowing system distributor 'The X.Org Foundation', version 11.0.12010000
System Description: Arch Linux

Configured features:
ACL CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS GPM GSETTINGS HARFBUZZ JPEG
JSON LCMS2 LIBOTF LIBSYSTEMD LIBXML2 M17N_FLT MODULES NOTIFY INOTIFY
PDUMPER PNG RSVG SOUND THREADS TIFF TOOLKIT_SCROLL_BARS X11 XDBE XIM XPM
GTK3 ZLIB

Important settings:
  value of $LC_TIME: C
  value of $LANG: hu_HU.utf8
  locale-coding-system: utf-8-unix

Major mode: Lisp Interaction

Minor modes in effect:
  shell-dirtrack-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  transient-mark-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message rmc puny dired dired-loaddefs
rfc822 mml mml-sec epa derived epg epg-config gnus-util rmail
rmail-loaddefs mm-decode mm-bodies mm-encode mail-parse rfc2231
mailabbrev gmm-utils mailheader sendmail rfc2047 rfc2045 ietf-drums
mm-util mail-prsvr mail-utils warnings compile text-property-search
cl-extra help-mode python easymenu tramp-sh tramp tramp-loaddefs
trampver tramp-integration files-x tramp-compat shell pcomplete
parse-time iso8601 time-date ls-lisp format-spec auth-source cl-seq
eieio eieio-core cl-macs eieio-loaddefs password-cache json subr-x map
seq byte-opt gv bytecomp byte-compile cconv comint ring cl-loaddefs
cl-lib ansi-color iso-transl tooltip eldoc electric uniquify ediff-hook
vc-hooks lisp-float-type mwheel term/x-win x-win term/common-win x-dnd
tool-bar dnd fontset image regexp-opt fringe tabulated-list replace
newcomment text-mode elisp-mode lisp-mode prog-mode register page
tab-bar menu-bar rfn-eshadow isearch timer select scroll-bar mouse
jit-lock font-lock syntax facemenu font-core term/tty-colors frame
minibuffer cl-generic cham georgian utf-8-lang misc-lang vietnamese
tibetan thai tai-viet lao korean japanese eucjp-ms cp51932 hebrew greek
romanian slovak czech european ethiopic indian cyrillic chinese
composite charscript charprop case-table epa-hook jka-cmpr-hook help
simple abbrev obarray cl-preloaded nadvice button loaddefs faces
cus-face macroexp files window text-properties overlay sha1 md5 base64
format env code-pages mule custom widget hashtable-print-readable
backquote threads dbusbind inotify lcms2 dynamic-setting
system-font-setting font-render-setting cairo move-toolbar gtk x-toolkit
x multi-tty make-network-process emacs)

Memory information:
((conses 16 74431 8365)
 (symbols 48 9112 1)
 (strings 32 27268 1190)
 (string-bytes 1 989045)
 (vectors 16 15840)
 (vector-slots 8 208279 9279)
 (floats 8 47 29)
 (intervals 56 226 0)
 (buffers 984 14))

bug#45938: 28.0.50; python native completion fails with fancier readline settings

[Lars Ingebrigtsen]

Zoltán Vandrus <vandrus.zoltan@gmail.com> writes:

> Three problematic options I found are:
>
>    set colored-stats on
>    set colored-completion-prefix on
>    set enable-bracketed-paste on
>
> With any of these M-x run-python warns:
>
>    Warning (python): Your ‘python-shell-interpreter’ doesn’t seem to
>    support readline, yet ‘python-shell-completion-native-enable’ was t and
>    "python3" is not part of the
>    ‘python-shell-completion-native-disabled-interpreters’ list.  Native
>    completions have been disabled locally.

(I'm going through old bug reports that unfortunately weren't resolved
at the time.)

There seems to have been some progress here after this was reported.
The only one of these settings that lead to this warning in Emacs 29 is:

set colored-completion-prefix on

But I'm actually not sure what the commit is that fixed the other cases.
Let's see...

commit a0973fa7a6e8162033abe29084f7b255976c74bc
Author:     Carlos Pita <carlosjosepita@gmail.com>
AuthorDate: Wed Oct 13 21:09:40 2021 -0300
Commit:     Stefan Kangas <stefan@marxist.se>
CommitDate: Fri Nov 5 08:02:31 2021 +0100

    Match dummy output even with non-default rl config
    
    * lisp/progmodes/python.el
    (python-shell-completion-native-get-completions): Match dummy output
    even when readline is configured in non-default ways.  (Bug#51010)

Yes, that's the one -- reverting it makes all of those settings fail,
and not just colored-completion-prefix.

And that's because completion returns:

^[[01;35m0^[[0m^[[K__dummy_completion__  ^[[01;35m1^[[0m^[[K__dummy_completion__  

And without that setting, we get:

0__dummy_completion__  1__dummy_completion__  

So something has to strip ANSI codes here before doing the matching, and
it's not done?

I'm not really that familiar with all this stuff -- anybody know on what
level this should be fixed?  (The code is in
python-shell-completion-native-get-completions.)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#45938: [PATCH] Avoid ANSI escape characters (bug#45938)

[Evgenii Klimov via Bug reports for GNU Emacs, the Swiss army knife of text editors]

[-- Attachment #1: Type: text/plain, Size: 984 bytes --]

Tags: patch

GNU Readline variables (colored-completion-prefix and colored-stats) set
in .inputrc file by the user to improve GNU Readline output lead to
corrupted completion candidates during native completion setup.

Like this:

^[[01;35m0^[[0m^[[K__dummy_completion__  ^[[01;35m1^[[0m^[[K__dummy_completion__  

However this is expected:

0__dummy_completion__  1__dummy_completion__  



In GNU Emacs 30.0.60 (build 1, x86_64-pc-linux-gnu, GTK+ Version
3.24.41, cairo version 1.18.0)
Windowing system distributor 'The X.Org Foundation', version 11.0.12101011
System Description: Guix System

Configured using:
 'configure
 CONFIG_SHELL=/gnu/store/fl3l5wx8qynjrvx5lilz6c38hb77cf36-bash-minimal-5.1.16/bin/bash
 SHELL=/gnu/store/fl3l5wx8qynjrvx5lilz6c38hb77cf36-bash-minimal-5.1.16/bin/bash
 --prefix=/gnu/store/45nwc8hc8fn1fhvr9qw01ylkfpvzxwsw-emacs-next-30.0.60-1.4e22ef8
 --enable-fast-install --with-cairo --with-modules
 --with-native-compilation=aot --disable-build-details'


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Avoid-ANSI-escape-characters-bug-45938.patch --]
[-- Type: text/patch, Size: 1150 bytes --]

From d5fb353404814afb135ba98effd200c5733bbcc3 Mon Sep 17 00:00:00 2001
From: Evgenii Klimov <eugene.dev@lipklim.org>
Date: Sun, 25 Feb 2024 20:12:38 +0000
Subject: [PATCH] Avoid ANSI escape characters (bug#45938)

* lisp/progmodes/python.el (python-shell-completion-native-setup):
Avoid escape characters in comint output
---
 lisp/progmodes/python.el | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/lisp/progmodes/python.el b/lisp/progmodes/python.el
index a00289d6de9..7193cc19425 100644
--- a/lisp/progmodes/python.el
+++ b/lisp/progmodes/python.el
@@ -4549,6 +4549,9 @@ (defun python-shell-completion-native-setup ()
             readline.parse_and_bind('tab: complete')
             # Require just one tab to send output.
             readline.parse_and_bind('set show-all-if-ambiguous on')
+            # Avoid ANSI escape characters in the output
+            readline.parse_and_bind('set colored-completion-prefix off')
+            readline.parse_and_bind('set colored-stats off')
             # Avoid replacing common prefix with ellipsis.
             readline.parse_and_bind('set completion-prefix-display-length 0')
 
-- 
2.45.2

bug#45938: [PATCH] Avoid ANSI escape characters (bug#45938)

[Eli Zaretskii]

> Date: Tue, 27 Aug 2024 22:47:05 +0100
> From:  Evgenii Klimov via "Bug reports for GNU Emacs,
>  the Swiss army knife of text editors" <bug-gnu-emacs@gnu.org>
> 
> GNU Readline variables (colored-completion-prefix and colored-stats) set
> in .inputrc file by the user to improve GNU Readline output lead to
> corrupted completion candidates during native completion setup.
> 
> Like this:
> 
> ^[[01;35m0^[[0m^[[K__dummy_completion__  ^[[01;35m1^[[0m^[[K__dummy_completion__  
> 
> However this is expected:
> 
> 0__dummy_completion__  1__dummy_completion__  

kobarity, any comments to the suggested patch?

> diff --git a/lisp/progmodes/python.el b/lisp/progmodes/python.el
> index a00289d6de9..7193cc19425 100644
> --- a/lisp/progmodes/python.el
> +++ b/lisp/progmodes/python.el
> @@ -4549,6 +4549,9 @@ (defun python-shell-completion-native-setup ()
>              readline.parse_and_bind('tab: complete')
>              # Require just one tab to send output.
>              readline.parse_and_bind('set show-all-if-ambiguous on')
> +            # Avoid ANSI escape characters in the output
> +            readline.parse_and_bind('set colored-completion-prefix off')
> +            readline.parse_and_bind('set colored-stats off')
>              # Avoid replacing common prefix with ellipsis.
>              readline.parse_and_bind('set completion-prefix-display-length 0')
>  

Are these commands available in every version of Python?

bug#45938: [PATCH] Avoid ANSI escape characters (bug#45938)

[Evgenii Klimov via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Hi Eli,

Eli Zaretskii <eliz@gnu.org> writes:
[...]
>> diff --git a/lisp/progmodes/python.el b/lisp/progmodes/python.el
>> index a00289d6de9..7193cc19425 100644
>> --- a/lisp/progmodes/python.el
>> +++ b/lisp/progmodes/python.el
>> @@ -4549,6 +4549,9 @@ (defun python-shell-completion-native-setup ()
>>              readline.parse_and_bind('tab: complete')
>>              # Require just one tab to send output.
>>              readline.parse_and_bind('set show-all-if-ambiguous on')
>> +            # Avoid ANSI escape characters in the output
>> +            readline.parse_and_bind('set colored-completion-prefix off')
>> +            readline.parse_and_bind('set colored-stats off')
>>              # Avoid replacing common prefix with ellipsis.
>>              readline.parse_and_bind('set completion-prefix-display-length 0')
>>  
>
> Are these commands available in every version of Python?

These commands are from GNU Readline, not from Python, but anyway:
- colored-stats was first implemented in readline-6.3 [1]
- colored-completion-prefix since readline-7.0 [2]

Just for the reference, completion-prefix-display-length and
show-all-if-ambiguous, that are already used during the setup, exists
since readline-6.0 and readline-6.2 [3], respectively.

Latest version of readline is 8.2.

[1] https://git.savannah.gnu.org/cgit/readline.git/tree/CHANGES?h=master#n675
[2] https://git.savannah.gnu.org/cgit/readline.git/tree/CHANGES?h=master#n508
[3] https://git.savannah.gnu.org/cgit/readline.git/tree/CHANGES?h=master#n718

bug#45938: [PATCH] Avoid ANSI escape characters (bug#45938)

[Eli Zaretskii]

> From: Evgenii Klimov <eugene.dev@lipklim.org>
> Cc: kobarity <kobarity@gmail.com>,  45938@debbugs.gnu.org
> Date: Wed, 28 Aug 2024 17:28:30 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> [...]
> >> diff --git a/lisp/progmodes/python.el b/lisp/progmodes/python.el
> >> index a00289d6de9..7193cc19425 100644
> >> --- a/lisp/progmodes/python.el
> >> +++ b/lisp/progmodes/python.el
> >> @@ -4549,6 +4549,9 @@ (defun python-shell-completion-native-setup ()
> >>              readline.parse_and_bind('tab: complete')
> >>              # Require just one tab to send output.
> >>              readline.parse_and_bind('set show-all-if-ambiguous on')
> >> +            # Avoid ANSI escape characters in the output
> >> +            readline.parse_and_bind('set colored-completion-prefix off')
> >> +            readline.parse_and_bind('set colored-stats off')
> >>              # Avoid replacing common prefix with ellipsis.
> >>              readline.parse_and_bind('set completion-prefix-display-length 0')
> >>  
> >
> > Are these commands available in every version of Python?
> 
> These commands are from GNU Readline, not from Python

Yes, I know.  But not every Python is built with GNU Readline, right?

> - colored-stats was first implemented in readline-6.3 [1]
> - colored-completion-prefix since readline-7.0 [2]
> 
> Just for the reference, completion-prefix-display-length and
> show-all-if-ambiguous, that are already used during the setup, exists
> since readline-6.0 and readline-6.2 [3], respectively.

Which versions of Python or GNU/Linux are likely to have older
versions of Readline?  And what happens in an older Readline when
these commands are sent?

bug#45938: [PATCH] Avoid ANSI escape characters (bug#45938)

[Evgenii Klimov via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Evgenii Klimov <eugene.dev@lipklim.org>
>> Cc: kobarity <kobarity@gmail.com>,  45938@debbugs.gnu.org
>> Date: Wed, 28 Aug 2024 17:28:30 +0100
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> [...]
>> >> diff --git a/lisp/progmodes/python.el b/lisp/progmodes/python.el
>> >> index a00289d6de9..7193cc19425 100644
>> >> --- a/lisp/progmodes/python.el
>> >> +++ b/lisp/progmodes/python.el
>> >> @@ -4549,6 +4549,9 @@ (defun python-shell-completion-native-setup ()
>> >>              readline.parse_and_bind('tab: complete')
>> >>              # Require just one tab to send output.
>> >>              readline.parse_and_bind('set show-all-if-ambiguous on')
>> >> +            # Avoid ANSI escape characters in the output
>> >> +            readline.parse_and_bind('set colored-completion-prefix off')
>> >> +            readline.parse_and_bind('set colored-stats off')
>> >>              # Avoid replacing common prefix with ellipsis.
>> >>              readline.parse_and_bind('set completion-prefix-display-length 0')
>> >>  
>> >
>> > Are these commands available in every version of Python?
>> 
>> These commands are from GNU Readline, not from Python
>
> Yes, I know.  But not every Python is built with GNU Readline, right?

If Python is not built with GNU Readline, then this function
(python-shell-completion-native-setup) will not set up native completion
in any case:

    ;; Shell completion: hitting tab will try to complete the current
    ;; word.  The two built-in mechanisms depend on Python's readline
    ;; module [1]

    (defun python-shell-completion-native-setup ()
      "Try to setup native completion, return non-nil on success."
      (let* ((process (python-shell-get-process))
             (output (python-shell-send-string-no-output "
    def __PYTHON_EL_native_completion_setup():
        try:
            import readline
            ...
        except:
            import sys
            print ('python.el: native completion setup failed, %s: %s'
                   % sys.exc_info()[:2])

[...]
> Which versions of Python or GNU/Linux are likely to have older
> versions of Readline?  And what happens in an older Readline when
> these commands are sent?

I tested what happens if we pass nonexistent variable - it prints to
stdout:

    In [15]: readline.parse_and_bind("set nonexistent-variable off")
    readline: nonexistent-variable: unknown variable name

In the context of `python-shell-completion-native-setup' such output
doesn't hurt, because output string still matches with expected one.
E.g. let's say that I added nonexistent-variable:

    (defun python-shell-completion-native-setup ()
      "Try to setup native completion, return non-nil on success."
      (let* ((process (python-shell-get-process))
             (output (python-shell-send-string-no-output "
    def __PYTHON_EL_native_completion_setup():
            ...
            if readline.__doc__ and 'libedit' in readline.__doc__:
                raise Exception('''libedit based readline is known not to work,
          see etc/PROBLEMS under \"In Inferior Python mode, input is echoed\".''')
                readline.parse_and_bind('bind ^I rl_complete')
            else:
                readline.parse_and_bind('tab: complete')
                # Require just one tab to send output.
                readline.parse_and_bind('set show-all-if-ambiguous on')
->              readline.parse_and_bind('set nonexistent-variable off')
                # Avoid replacing common prefix with ellipsis.
                readline.parse_and_bind('set completion-prefix-display-length 0')

            print ('python.el: native completion setup loaded')
        except:
            import sys
            print ('python.el: native completion setup failed, %s: %s'
                   % sys.exc_info()[:2])

    __PYTHON_EL_native_completion_setup()" process)))
        (when (string-match-p "python\\.el: native completion setup loaded"
                              output)
          (python-shell-completion-native-try))))

Then the `output' would be:

     "readline: nonexistent-variable: unknown variable name\npython.el: native completion setup loaded\n"

and

    (string-match-p "python\\.el: native completion setup loaded"
                              output)

finds the match.

I tried that in CPython and IPython.

Actually, I also tried python built with readline-6.2
(`python-shell-completion-native-setup' already uses
"show-all-if-ambiguous", first appeared in that version) and found out
that that version doesn't even print anything when nonexistent variable
is passed.

    eugene@gx:~/git (gx)
    $ guix shell --with-input=readline=readline@6.2 python 
    eugene@gx:~/git (gx) [env]
    $ which python3
    /gnu/store/hhcay3d42k3k7avahrmlqv3qnjk2l6wp-profile/bin/python3
    eugene@gx:~/git (gx) [env]
    $ echo $GUIX_ENVIRONMENT 
    /gnu/store/hhcay3d42k3k7avahrmlqv3qnjk2l6wp-profile
    eugene@gx:~/git (gx) [env]
    $ python3
    Python 3.10.7 (main, Jan  1 1970, 00:00:01) [GCC 11.3.0] on linux
    Type "help", "copyright", "credits" or "license" for more information.
    >>> import readline
    >>> readline.parse_and_bind("set nonexistent-variable off")
    >>> 

[1] https://git.sv.gnu.org/cgit/emacs.git/tree/lisp/progmodes/python.el?h=master#n119

bug#45938: [PATCH] Avoid ANSI escape characters (bug#45938)

[Eli Zaretskii]

> From: Evgenii Klimov <eugene.dev@lipklim.org>
> Cc: kobarity@gmail.com,  45938@debbugs.gnu.org
> Date: Wed, 28 Aug 2024 21:39:27 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> From: Evgenii Klimov <eugene.dev@lipklim.org>
> >> Cc: kobarity <kobarity@gmail.com>,  45938@debbugs.gnu.org
> >> Date: Wed, 28 Aug 2024 17:28:30 +0100
> >> 
> >> Eli Zaretskii <eliz@gnu.org> writes:
> >> [...]
> >> >> diff --git a/lisp/progmodes/python.el b/lisp/progmodes/python.el
> >> >> index a00289d6de9..7193cc19425 100644
> >> >> --- a/lisp/progmodes/python.el
> >> >> +++ b/lisp/progmodes/python.el
> >> >> @@ -4549,6 +4549,9 @@ (defun python-shell-completion-native-setup ()
> >> >>              readline.parse_and_bind('tab: complete')
> >> >>              # Require just one tab to send output.
> >> >>              readline.parse_and_bind('set show-all-if-ambiguous on')
> >> >> +            # Avoid ANSI escape characters in the output
> >> >> +            readline.parse_and_bind('set colored-completion-prefix off')
> >> >> +            readline.parse_and_bind('set colored-stats off')
> >> >>              # Avoid replacing common prefix with ellipsis.
> >> >>              readline.parse_and_bind('set completion-prefix-display-length 0')
> >> >>  
> >> >
> >> > Are these commands available in every version of Python?
> >> 
> >> These commands are from GNU Readline, not from Python
> >
> > Yes, I know.  But not every Python is built with GNU Readline, right?
> 
> If Python is not built with GNU Readline, then this function
> (python-shell-completion-native-setup) will not set up native completion
> in any case:
> 
>     ;; Shell completion: hitting tab will try to complete the current
>     ;; word.  The two built-in mechanisms depend on Python's readline
>     ;; module [1]
> 
>     (defun python-shell-completion-native-setup ()
>       "Try to setup native completion, return non-nil on success."
>       (let* ((process (python-shell-get-process))
>              (output (python-shell-send-string-no-output "
>     def __PYTHON_EL_native_completion_setup():
>         try:
>             import readline
>             ...
>         except:
>             import sys
>             print ('python.el: native completion setup failed, %s: %s'
>                    % sys.exc_info()[:2])
> 
> [...]
> > Which versions of Python or GNU/Linux are likely to have older
> > versions of Readline?  And what happens in an older Readline when
> > these commands are sent?
> 
> I tested what happens if we pass nonexistent variable - it prints to
> stdout:
> 
>     In [15]: readline.parse_and_bind("set nonexistent-variable off")
>     readline: nonexistent-variable: unknown variable name
> 
> In the context of `python-shell-completion-native-setup' such output
> doesn't hurt, because output string still matches with expected one.
> E.g. let's say that I added nonexistent-variable:
> 
>     (defun python-shell-completion-native-setup ()
>       "Try to setup native completion, return non-nil on success."
>       (let* ((process (python-shell-get-process))
>              (output (python-shell-send-string-no-output "
>     def __PYTHON_EL_native_completion_setup():
>             ...
>             if readline.__doc__ and 'libedit' in readline.__doc__:
>                 raise Exception('''libedit based readline is known not to work,
>           see etc/PROBLEMS under \"In Inferior Python mode, input is echoed\".''')
>                 readline.parse_and_bind('bind ^I rl_complete')
>             else:
>                 readline.parse_and_bind('tab: complete')
>                 # Require just one tab to send output.
>                 readline.parse_and_bind('set show-all-if-ambiguous on')
> ->              readline.parse_and_bind('set nonexistent-variable off')
>                 # Avoid replacing common prefix with ellipsis.
>                 readline.parse_and_bind('set completion-prefix-display-length 0')
> 
>             print ('python.el: native completion setup loaded')
>         except:
>             import sys
>             print ('python.el: native completion setup failed, %s: %s'
>                    % sys.exc_info()[:2])
> 
>     __PYTHON_EL_native_completion_setup()" process)))
>         (when (string-match-p "python\\.el: native completion setup loaded"
>                               output)
>           (python-shell-completion-native-try))))
> 
> Then the `output' would be:
> 
>      "readline: nonexistent-variable: unknown variable name\npython.el: native completion setup loaded\n"
> 
> and
> 
>     (string-match-p "python\\.el: native completion setup loaded"
>                               output)
> 
> finds the match.
> 
> I tried that in CPython and IPython.
> 
> Actually, I also tried python built with readline-6.2
> (`python-shell-completion-native-setup' already uses
> "show-all-if-ambiguous", first appeared in that version) and found out
> that that version doesn't even print anything when nonexistent variable
> is passed.
> 
>     eugene@gx:~/git (gx)
>     $ guix shell --with-input=readline=readline@6.2 python 
>     eugene@gx:~/git (gx) [env]
>     $ which python3
>     /gnu/store/hhcay3d42k3k7avahrmlqv3qnjk2l6wp-profile/bin/python3
>     eugene@gx:~/git (gx) [env]
>     $ echo $GUIX_ENVIRONMENT 
>     /gnu/store/hhcay3d42k3k7avahrmlqv3qnjk2l6wp-profile
>     eugene@gx:~/git (gx) [env]
>     $ python3
>     Python 3.10.7 (main, Jan  1 1970, 00:00:01) [GCC 11.3.0] on linux
>     Type "help", "copyright", "credits" or "license" for more information.
>     >>> import readline
>     >>> readline.parse_and_bind("set nonexistent-variable off")
>     >>> 

Thanks, this all sounds good.  Let's see what kobarity has to say
about this.

bug#45938: [PATCH] Avoid ANSI escape characters (bug#45938)

[kobarity]

Eli Zaretskii wrote:
> 
> > From: Evgenii Klimov <eugene.dev@lipklim.org>
> > Cc: kobarity@gmail.com,  45938@debbugs.gnu.org
> > Date: Wed, 28 Aug 2024 21:39:27 +0100
> > 
> > Eli Zaretskii <eliz@gnu.org> writes:
> > 
> > >> From: Evgenii Klimov <eugene.dev@lipklim.org>
> > >> Cc: kobarity <kobarity@gmail.com>,  45938@debbugs.gnu.org
> > >> Date: Wed, 28 Aug 2024 17:28:30 +0100
> > >> 
> > >> Eli Zaretskii <eliz@gnu.org> writes:
> > >> [...]
> > >> >> diff --git a/lisp/progmodes/python.el b/lisp/progmodes/python.el
> > >> >> index a00289d6de9..7193cc19425 100644
> > >> >> --- a/lisp/progmodes/python.el
> > >> >> +++ b/lisp/progmodes/python.el
> > >> >> @@ -4549,6 +4549,9 @@ (defun python-shell-completion-native-setup ()
> > >> >>              readline.parse_and_bind('tab: complete')
> > >> >>              # Require just one tab to send output.
> > >> >>              readline.parse_and_bind('set show-all-if-ambiguous on')
> > >> >> +            # Avoid ANSI escape characters in the output
> > >> >> +            readline.parse_and_bind('set colored-completion-prefix off')
> > >> >> +            readline.parse_and_bind('set colored-stats off')
> > >> >>              # Avoid replacing common prefix with ellipsis.
> > >> >>              readline.parse_and_bind('set completion-prefix-display-length 0')
> > >> >>  
> > >> >
> > >> > Are these commands available in every version of Python?
> > >> 
> > >> These commands are from GNU Readline, not from Python
> > >
> > > Yes, I know.  But not every Python is built with GNU Readline, right?
> > 
> > If Python is not built with GNU Readline, then this function
> > (python-shell-completion-native-setup) will not set up native completion
> > in any case:
> > 
> >     ;; Shell completion: hitting tab will try to complete the current
> >     ;; word.  The two built-in mechanisms depend on Python's readline
> >     ;; module [1]
> > 
> >     (defun python-shell-completion-native-setup ()
> >       "Try to setup native completion, return non-nil on success."
> >       (let* ((process (python-shell-get-process))
> >              (output (python-shell-send-string-no-output "
> >     def __PYTHON_EL_native_completion_setup():
> >         try:
> >             import readline
> >             ...
> >         except:
> >             import sys
> >             print ('python.el: native completion setup failed, %s: %s'
> >                    % sys.exc_info()[:2])
> > 
> > [...]
> > > Which versions of Python or GNU/Linux are likely to have older
> > > versions of Readline?  And what happens in an older Readline when
> > > these commands are sent?
> > 
> > I tested what happens if we pass nonexistent variable - it prints to
> > stdout:
> > 
> >     In [15]: readline.parse_and_bind("set nonexistent-variable off")
> >     readline: nonexistent-variable: unknown variable name
> > 
> > In the context of `python-shell-completion-native-setup' such output
> > doesn't hurt, because output string still matches with expected one.
> > E.g. let's say that I added nonexistent-variable:
> > 
> >     (defun python-shell-completion-native-setup ()
> >       "Try to setup native completion, return non-nil on success."
> >       (let* ((process (python-shell-get-process))
> >              (output (python-shell-send-string-no-output "
> >     def __PYTHON_EL_native_completion_setup():
> >             ...
> >             if readline.__doc__ and 'libedit' in readline.__doc__:
> >                 raise Exception('''libedit based readline is known not to work,
> >           see etc/PROBLEMS under \"In Inferior Python mode, input is echoed\".''')
> >                 readline.parse_and_bind('bind ^I rl_complete')
> >             else:
> >                 readline.parse_and_bind('tab: complete')
> >                 # Require just one tab to send output.
> >                 readline.parse_and_bind('set show-all-if-ambiguous on')
> > ->              readline.parse_and_bind('set nonexistent-variable off')
> >                 # Avoid replacing common prefix with ellipsis.
> >                 readline.parse_and_bind('set completion-prefix-display-length 0')
> > 
> >             print ('python.el: native completion setup loaded')
> >         except:
> >             import sys
> >             print ('python.el: native completion setup failed, %s: %s'
> >                    % sys.exc_info()[:2])
> > 
> >     __PYTHON_EL_native_completion_setup()" process)))
> >         (when (string-match-p "python\\.el: native completion setup loaded"
> >                               output)
> >           (python-shell-completion-native-try))))
> > 
> > Then the `output' would be:
> > 
> >      "readline: nonexistent-variable: unknown variable name\npython.el: native completion setup loaded\n"
> > 
> > and
> > 
> >     (string-match-p "python\\.el: native completion setup loaded"
> >                               output)
> > 
> > finds the match.
> > 
> > I tried that in CPython and IPython.
> > 
> > Actually, I also tried python built with readline-6.2
> > (`python-shell-completion-native-setup' already uses
> > "show-all-if-ambiguous", first appeared in that version) and found out
> > that that version doesn't even print anything when nonexistent variable
> > is passed.
> > 
> >     eugene@gx:~/git (gx)
> >     $ guix shell --with-input=readline=readline@6.2 python 
> >     eugene@gx:~/git (gx) [env]
> >     $ which python3
> >     /gnu/store/hhcay3d42k3k7avahrmlqv3qnjk2l6wp-profile/bin/python3
> >     eugene@gx:~/git (gx) [env]
> >     $ echo $GUIX_ENVIRONMENT 
> >     /gnu/store/hhcay3d42k3k7avahrmlqv3qnjk2l6wp-profile
> >     eugene@gx:~/git (gx) [env]
> >     $ python3
> >     Python 3.10.7 (main, Jan  1 1970, 00:00:01) [GCC 11.3.0] on linux
> >     Type "help", "copyright", "credits" or "license" for more information.
> >     >>> import readline
> >     >>> readline.parse_and_bind("set nonexistent-variable off")
> >     >>> 
> 
> Thanks, this all sounds good.  Let's see what kobarity has to say
> about this.

I confirmed that the patch resolves the issue.  Also, Evgenii's survey
seems sufficient.  So I agree with this patch.

bug#45938: [PATCH] Avoid ANSI escape characters (bug#45938)

[Eli Zaretskii]

> Date: Thu, 29 Aug 2024 21:20:50 +0900
> From: kobarity <kobarity@gmail.com>
> Cc: 45938@debbugs.gnu.org
> 
> > Thanks, this all sounds good.  Let's see what kobarity has to say
> > about this.
> 
> I confirmed that the patch resolves the issue.  Also, Evgenii's survey
> seems sufficient.  So I agree with this patch.

Thanks, installed on the emacs-30 release branch, and closing the bug.