bug#45938: 28.0.50; python native completion fails with fancier readline settings

bug#45938: 28.0.50; python native completion fails with fancier readline settings

[Zoltán Vandrus]

Three problematic options I found are:

   set colored-stats on
   set colored-completion-prefix on
   set enable-bracketed-paste on

With any of these M-x run-python warns:

   Warning (python): Your ‘python-shell-interpreter’ doesn’t seem to
   support readline, yet ‘python-shell-completion-native-enable’ was t and
   "python3" is not part of the
   ‘python-shell-completion-native-disabled-interpreters’ list.  Native
   completions have been disabled locally.
   

set colored-stats on
set colored-completion-prefix on
================================

Any of these on and *Python completions redirect* buffer ends with:
0__dummy_completion__^[[0m^[[K  1__dummy_completion__^[[0m^[[K

^[[0m and ^[[K are escape sequences not understood by comint mode I
think

Related bug: #24401 recommends setting colored-stats conditionally only
for bash, because filename coloring is not that useful for the python
shell.

Even then colored-completion-prefix is useful everywhere, emacs does it
by default too.

set enable-bracketed-paste on
=============================

Fails like half of the times for me. For a different reason.

First record output:
   (defun keep-output (output)
     (setq kept (cons output kept)))

   (add-hook 'inferior-python-mode-hook
             (lambda ()
               (setq kept nil)
               (add-hook 'comint-output-filter-functions #'keep-output)))

M-x run-python

kept’s value is
("python.el: native completion setup loaded
^[[?2004h>>> " "^[[?2004l^M" "Python 3.9.0 (default, Oct  7 2020, 23:09:01) 
[GCC 10.2.0] on linux
Type \"help\", \"copyright\", \"credits\" or \"license\" for more information.
^[[?2004h>>> ")

-----------------------

python-shell-completin-native-setup assumes that after running the
python script that sets up completion the first output will be from the script:
      (when (and
             (python-shell-accept-process-output
              process python-shell-completion-native-try-output-timeout)
             (save-excursion
               (re-search-backward
                (regexp-quote "python.el: native completion setup loaded") nil t 1)))
        (python-shell-completion-native-try)))))

What can happen sometimes is that, after the first initial prompt:
   "Python 3.9.0 (default, Oct  7 2020, 23:09:01) 
   [GCC 10.2.0] on linux
   Type \"help\", \"copyright\", \"credits\" or \"license\" for more information.
   ^[[?2004h>>> "
The python script is injected.
python-shell-accept-process-output waits for next prompt
Bracketed mode is disabled instead with clearing the line: "^[[?2004l^M"
Empty line is a valid python prompt, so it's accepted, and the regexp
test fails.

#41959 is another problem with trying to parse the readline output.

--------------------------------------------------

Readline version 8.1.0

In GNU Emacs 28.0.50 (build 4, x86_64-pc-linux-gnu, GTK+ Version 3.24.24, cairo version 1.17.4)
 of 2021-01-17 built on zoli-laptop
Repository revision: 1773679af3241919a85d6174b1554070a63cca79
Repository branch: master
Windowing system distributor 'The X.Org Foundation', version 11.0.12010000
System Description: Arch Linux

Configured features:
ACL CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS GPM GSETTINGS HARFBUZZ JPEG
JSON LCMS2 LIBOTF LIBSYSTEMD LIBXML2 M17N_FLT MODULES NOTIFY INOTIFY
PDUMPER PNG RSVG SOUND THREADS TIFF TOOLKIT_SCROLL_BARS X11 XDBE XIM XPM
GTK3 ZLIB

Important settings:
  value of $LC_TIME: C
  value of $LANG: hu_HU.utf8
  locale-coding-system: utf-8-unix

Major mode: Lisp Interaction

Minor modes in effect:
  shell-dirtrack-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  transient-mark-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message rmc puny dired dired-loaddefs
rfc822 mml mml-sec epa derived epg epg-config gnus-util rmail
rmail-loaddefs mm-decode mm-bodies mm-encode mail-parse rfc2231
mailabbrev gmm-utils mailheader sendmail rfc2047 rfc2045 ietf-drums
mm-util mail-prsvr mail-utils warnings compile text-property-search
cl-extra help-mode python easymenu tramp-sh tramp tramp-loaddefs
trampver tramp-integration files-x tramp-compat shell pcomplete
parse-time iso8601 time-date ls-lisp format-spec auth-source cl-seq
eieio eieio-core cl-macs eieio-loaddefs password-cache json subr-x map
seq byte-opt gv bytecomp byte-compile cconv comint ring cl-loaddefs
cl-lib ansi-color iso-transl tooltip eldoc electric uniquify ediff-hook
vc-hooks lisp-float-type mwheel term/x-win x-win term/common-win x-dnd
tool-bar dnd fontset image regexp-opt fringe tabulated-list replace
newcomment text-mode elisp-mode lisp-mode prog-mode register page
tab-bar menu-bar rfn-eshadow isearch timer select scroll-bar mouse
jit-lock font-lock syntax facemenu font-core term/tty-colors frame
minibuffer cl-generic cham georgian utf-8-lang misc-lang vietnamese
tibetan thai tai-viet lao korean japanese eucjp-ms cp51932 hebrew greek
romanian slovak czech european ethiopic indian cyrillic chinese
composite charscript charprop case-table epa-hook jka-cmpr-hook help
simple abbrev obarray cl-preloaded nadvice button loaddefs faces
cus-face macroexp files window text-properties overlay sha1 md5 base64
format env code-pages mule custom widget hashtable-print-readable
backquote threads dbusbind inotify lcms2 dynamic-setting
system-font-setting font-render-setting cairo move-toolbar gtk x-toolkit
x multi-tty make-network-process emacs)

Memory information:
((conses 16 74431 8365)
 (symbols 48 9112 1)
 (strings 32 27268 1190)
 (string-bytes 1 989045)
 (vectors 16 15840)
 (vector-slots 8 208279 9279)
 (floats 8 47 29)
 (intervals 56 226 0)
 (buffers 984 14))

bug#45938: 28.0.50; python native completion fails with fancier readline settings

[Lars Ingebrigtsen]

Zoltán Vandrus <vandrus.zoltan@gmail.com> writes:

> Three problematic options I found are:
>
>    set colored-stats on
>    set colored-completion-prefix on
>    set enable-bracketed-paste on
>
> With any of these M-x run-python warns:
>
>    Warning (python): Your ‘python-shell-interpreter’ doesn’t seem to
>    support readline, yet ‘python-shell-completion-native-enable’ was t and
>    "python3" is not part of the
>    ‘python-shell-completion-native-disabled-interpreters’ list.  Native
>    completions have been disabled locally.

(I'm going through old bug reports that unfortunately weren't resolved
at the time.)

There seems to have been some progress here after this was reported.
The only one of these settings that lead to this warning in Emacs 29 is:

set colored-completion-prefix on

But I'm actually not sure what the commit is that fixed the other cases.
Let's see...

commit a0973fa7a6e8162033abe29084f7b255976c74bc
Author:     Carlos Pita <carlosjosepita@gmail.com>
AuthorDate: Wed Oct 13 21:09:40 2021 -0300
Commit:     Stefan Kangas <stefan@marxist.se>
CommitDate: Fri Nov 5 08:02:31 2021 +0100

    Match dummy output even with non-default rl config
    
    * lisp/progmodes/python.el
    (python-shell-completion-native-get-completions): Match dummy output
    even when readline is configured in non-default ways.  (Bug#51010)

Yes, that's the one -- reverting it makes all of those settings fail,
and not just colored-completion-prefix.

And that's because completion returns:

^[[01;35m0^[[0m^[[K__dummy_completion__  ^[[01;35m1^[[0m^[[K__dummy_completion__  

And without that setting, we get:

0__dummy_completion__  1__dummy_completion__  

So something has to strip ANSI codes here before doing the matching, and
it's not done?

I'm not really that familiar with all this stuff -- anybody know on what
level this should be fixed?  (The code is in
python-shell-completion-native-get-completions.)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no