bug#65089: 30.0.50; shell-command filename completion unexpected behavior change

bug#65089: 30.0.50; shell-command filename completion unexpected behavior change

[Mauro Aranda]

Before running emacs, setup a directory tree like this:
mkdir bug bug/foo-1 bug/foo-2 bug/foo-1/bar bug/foo-2/bar
under HOME, for simplicity.

Now, with emacs -Q:
M-!
ls ~/bug/bar
Put point between "/" and "b" of "bar" and type TAB
emacs says "No match", but I expected it to offer completions, foo-1 and
foo-2.

That was the behavior, at least in Emacs 28.  Reverting the following
commit, returns this behavior for me:

commit a9941269683fe50673d0aa81feefb7a9d3d8a6b9
Author: Augusto Stoffel <arstoffel@gmail.com>
Date:   Thu Sep 8 11:09:42 2022 +0200

     pcomplete: Generate completions from --help messages

     * lisp/pcomplete.el (pcomplete-from-help): New function (and hash
     table) to get pcomplete candidates from help messages.
     (pcomplete-here-using-help): Helper function to define pcomplete for
     simple commands
     (pcomplete-completions-at-point): Provide annotation-function and
     company-docsig properties.
     * lisp/pcmpl-git.el: New file, provides pcomplete for Git.
     * lisp/pcmpl-gnu.el: Add pcomplete for awk, gpg and gdb, emacs and
     emacsclient.
     * lisp/pcmpl-linux.el: Add pcomplete for systemctl and journalctl.
     * lisp/pcmpl-rpm.el: Add pcomplete for dnf.
     * lisp/pcmpl-unix.el: Add pcomplete for sudo and most commands found
     in GNU Coreutils.
     * lisp/pcmpl-x.el: Add pcomplete for tex, pdftex, latex, pdflatex,
     rigrep and rclone.
     * test/lisp/pcomplete-tests.el (pcomplete-test-parse-gpg-help,
     pcomplete-test-parse-git-help): Tests for the new functions.


And my current workaround is to move pcomplete-completions-at-point to
the end in shell-dynamic-complete-functions, following what was done in
Bug#34330.


In GNU Emacs 30.0.50 (build 1, x86_64-pc-linux-gnu, GTK+ Version
  3.24.33, cairo version 1.16.0) of 2023-08-03 built on tbb-desktop
Repository revision: 0c2152222ad16ba9e68cdb27fc281755ec187620
Repository branch: master
Windowing system distributor 'The X.Org Foundation', version 11.0.12201001
System Description: Ubuntu 22.04.2 LTS

Configured features:
ACL CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS GPM GSETTINGS HARFBUZZ JPEG
JSON LCMS2 LIBOTF LIBSELINUX LIBSYSTEMD LIBXML2 M17N_FLT MODULES NOTIFY
INOTIFY PDUMPER PNG RSVG SECCOMP SOUND SQLITE3 THREADS TIFF
TOOLKIT_SCROLL_BARS TREE_SITTER X11 XDBE XIM XINPUT2 XPM GTK3 ZLIB

Important settings:
   value of $LC_MONETARY: es_AR.UTF-8
   value of $LC_NUMERIC: es_AR.UTF-8
   value of $LC_TIME: es_AR.UTF-8
   value of $LANG: en_US.UTF-8
   value of $XMODIFIERS: @im=ibus
   locale-coding-system: utf-8-unix

Major mode: Lisp Interaction

Minor modes in effect:
   tooltip-mode: t
   global-eldoc-mode: t
   eldoc-mode: t
   show-paren-mode: t
   electric-indent-mode: t
   mouse-wheel-mode: t
   tool-bar-mode: t
   menu-bar-mode: t
   file-name-shadow-mode: t
   global-font-lock-mode: t
   font-lock-mode: t
   blink-cursor-mode: t
   line-number-mode: t
   indent-tabs-mode: t
   transient-mark-mode: t
   auto-composition-mode: t
   auto-encryption-mode: t
   auto-compression-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message mailcap yank-media puny dired
dired-loaddefs rfc822 mml mml-sec password-cache epa derived epg rfc6068
epg-config gnus-util text-property-search time-date subr-x mm-decode
mm-bodies mm-encode mail-parse rfc2231 mailabbrev gmm-utils mailheader
cl-loaddefs cl-lib sendmail rfc2047 rfc2045 ietf-drums mm-util
mail-prsvr mail-utils rmc iso-transl tooltip cconv eldoc paren electric
uniquify ediff-hook vc-hooks lisp-float-type elisp-mode mwheel
term/x-win x-win term/common-win x-dnd tool-bar dnd fontset image
regexp-opt fringe tabulated-list replace newcomment text-mode lisp-mode
prog-mode register page tab-bar menu-bar rfn-eshadow isearch easymenu
timer select scroll-bar mouse jit-lock font-lock syntax font-core
term/tty-colors frame minibuffer nadvice seq simple cl-generic
indonesian philippine cham georgian utf-8-lang misc-lang vietnamese
tibetan thai tai-viet lao korean japanese eucjp-ms cp51932 hebrew greek
romanian slovak czech european ethiopic indian cyrillic chinese
composite emoji-zwj charscript charprop case-table epa-hook
jka-cmpr-hook help abbrev obarray oclosure cl-preloaded button loaddefs
theme-loaddefs faces cus-face macroexp files window text-properties
overlay sha1 md5 base64 format env code-pages mule custom widget keymap
hashtable-print-readable backquote threads dbusbind inotify lcms2
dynamic-setting system-font-setting font-render-setting cairo gtk
x-toolkit xinput2 x multi-tty move-toolbar make-network-process emacs)

Memory information:
((conses 16 46943 13845) (symbols 48 5701 0) (strings 32 16383 1535)
  (string-bytes 1 545790) (vectors 16 11987)
  (vector-slots 8 210580 14004) (floats 8 21 23) (intervals 56 209 0)
  (buffers 984 10))

bug#65089: 30.0.50; shell-command filename completion unexpected behavior change

[Eli Zaretskii]

> Date: Sat, 5 Aug 2023 07:46:16 -0300
> From: Mauro Aranda <maurooaranda@gmail.com>
> 
> Before running emacs, setup a directory tree like this:
> mkdir bug bug/foo-1 bug/foo-2 bug/foo-1/bar bug/foo-2/bar
> under HOME, for simplicity.
> 
> Now, with emacs -Q:
> M-!
> ls ~/bug/bar
> Put point between "/" and "b" of "bar" and type TAB
> emacs says "No match", but I expected it to offer completions, foo-1 and
> foo-2.
> 
> That was the behavior, at least in Emacs 28.  Reverting the following
> commit, returns this behavior for me:
> 
> commit a9941269683fe50673d0aa81feefb7a9d3d8a6b9
> Author: Augusto Stoffel <arstoffel@gmail.com>
> Date:   Thu Sep 8 11:09:42 2022 +0200
> 
>      pcomplete: Generate completions from --help messages
> 
>      * lisp/pcomplete.el (pcomplete-from-help): New function (and hash
>      table) to get pcomplete candidates from help messages.
>      (pcomplete-here-using-help): Helper function to define pcomplete for
>      simple commands
>      (pcomplete-completions-at-point): Provide annotation-function and
>      company-docsig properties.
>      * lisp/pcmpl-git.el: New file, provides pcomplete for Git.
>      * lisp/pcmpl-gnu.el: Add pcomplete for awk, gpg and gdb, emacs and
>      emacsclient.
>      * lisp/pcmpl-linux.el: Add pcomplete for systemctl and journalctl.
>      * lisp/pcmpl-rpm.el: Add pcomplete for dnf.
>      * lisp/pcmpl-unix.el: Add pcomplete for sudo and most commands found
>      in GNU Coreutils.
>      * lisp/pcmpl-x.el: Add pcomplete for tex, pdftex, latex, pdflatex,
>      rigrep and rclone.
>      * test/lisp/pcomplete-tests.el (pcomplete-test-parse-gpg-help,
>      pcomplete-test-parse-git-help): Tests for the new functions.
> 
> 
> And my current workaround is to move pcomplete-completions-at-point to
> the end in shell-dynamic-complete-functions, following what was done in
> Bug#34330.

Adding Augusto and Stefan to the discussion.

bug#65089: 30.0.50; shell-command filename completion unexpected behavior change

[Eli Zaretskii]

Ping!

> Cc: 65089@debbugs.gnu.org
> Date: Sat, 05 Aug 2023 14:04:23 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> 
> > Date: Sat, 5 Aug 2023 07:46:16 -0300
> > From: Mauro Aranda <maurooaranda@gmail.com>
> > 
> > Before running emacs, setup a directory tree like this:
> > mkdir bug bug/foo-1 bug/foo-2 bug/foo-1/bar bug/foo-2/bar
> > under HOME, for simplicity.
> > 
> > Now, with emacs -Q:
> > M-!
> > ls ~/bug/bar
> > Put point between "/" and "b" of "bar" and type TAB
> > emacs says "No match", but I expected it to offer completions, foo-1 and
> > foo-2.
> > 
> > That was the behavior, at least in Emacs 28.  Reverting the following
> > commit, returns this behavior for me:
> > 
> > commit a9941269683fe50673d0aa81feefb7a9d3d8a6b9
> > Author: Augusto Stoffel <arstoffel@gmail.com>
> > Date:   Thu Sep 8 11:09:42 2022 +0200
> > 
> >      pcomplete: Generate completions from --help messages
> > 
> >      * lisp/pcomplete.el (pcomplete-from-help): New function (and hash
> >      table) to get pcomplete candidates from help messages.
> >      (pcomplete-here-using-help): Helper function to define pcomplete for
> >      simple commands
> >      (pcomplete-completions-at-point): Provide annotation-function and
> >      company-docsig properties.
> >      * lisp/pcmpl-git.el: New file, provides pcomplete for Git.
> >      * lisp/pcmpl-gnu.el: Add pcomplete for awk, gpg and gdb, emacs and
> >      emacsclient.
> >      * lisp/pcmpl-linux.el: Add pcomplete for systemctl and journalctl.
> >      * lisp/pcmpl-rpm.el: Add pcomplete for dnf.
> >      * lisp/pcmpl-unix.el: Add pcomplete for sudo and most commands found
> >      in GNU Coreutils.
> >      * lisp/pcmpl-x.el: Add pcomplete for tex, pdftex, latex, pdflatex,
> >      rigrep and rclone.
> >      * test/lisp/pcomplete-tests.el (pcomplete-test-parse-gpg-help,
> >      pcomplete-test-parse-git-help): Tests for the new functions.
> > 
> > 
> > And my current workaround is to move pcomplete-completions-at-point to
> > the end in shell-dynamic-complete-functions, following what was done in
> > Bug#34330.
> 
> Adding Augusto and Stefan to the discussion.
> 
> 
> 
> 

bug#65089: 30.0.50; shell-command filename completion unexpected behavior change

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>> Now, with emacs -Q:
>> M-!
>> ls ~/bug/bar
>> Put point between "/" and "b" of "bar" and type TAB
>> emacs says "No match", but I expected it to offer completions, foo-1 and
>> foo-2.
>> 
>> That was the behavior, at least in Emacs 28.  Reverting the following
>> commit, returns this behavior for me:

Hmm... I haven't tracked down why/how this worked before, but this is
a general problem with pcomplete when point is in the middle of
an argument.

The root cause is a problem in the API: `pcomplete-completions` (the
function that tries to figure out what it is we're trying to complete)
only tells us:

- you're trying to complete the string "~/bug/bar".
- the start position of the thing you're trying to complete (i.e. the
  point just before ~).
- the completion table.

In this case it seems obvious that the "~/bug/bar" returned is
the "~/bug/bar" found at the starting position, but in the general case,
the strings don't have to match and we have to make an educated guess
about how the returned string maps to the buffer contents (i.e. how
buffer positions map to positions within the string and vice-versa).

The patch below adds yet another heuristic to try and handle the
current case, but really we should rework the pcomplete API so as to
provide us the right info to start with.


        Stefan


diff --git a/lisp/pcomplete.el b/lisp/pcomplete.el
index c7ec228c1db..060ddc1d853 100644
--- a/lisp/pcomplete.el
+++ b/lisp/pcomplete.el
@@ -465,6 +465,8 @@ pcomplete-completions-at-point
            ;; rely less on c-t-subvert.
            (beg (max (- (point) (length pcomplete-stub))
                      argbeg))
+           (end (point))
+           tmp
            buftext)
       ;; Try and improve our guess of `beg' in case the difference
       ;; between pcomplete-stub and the buffer's text is simply due to
@@ -472,11 +474,19 @@ pcomplete-completions-at-point
       ;; indispensable but reduces the reliance on c-t-subvert and
       ;; improves corner case behaviors.
       (while (progn (setq buftext (pcomplete-unquote-argument
-                                   (buffer-substring beg (point))))
+                                   (buffer-substring beg end)))
                     (and (> beg argbeg)
                          (> (length pcomplete-stub) (length buftext))))
         (setq beg (max argbeg (- beg (- (length pcomplete-stub)
                                         (length buftext))))))
+      ;; Try and improve our guess of `end' in case it's not point.
+      (while (and (< (length buftext) (length pcomplete-stub))
+                  (< end (point-max))
+                  (string-prefix-p (setq tmp (pcomplete-unquote-argument
+                                              (buffer-substring beg (1+ end))))
+                                   pcomplete-stub))
+        (setq end (1+ end))
+        (setq buftext tmp))
       (when completions
         (let ((table
                (completion-table-with-quoting
@@ -510,7 +520,7 @@ pcomplete-completions-at-point
                            seen)))))))
           (when completion-ignore-case
             (setq table (completion-table-case-fold table)))
-          (list beg (point) table
+          (list beg end table
                 :annotation-function
                 (lambda (cand)
                   (when (stringp cand)

bug#65089: 30.0.50; shell-command filename completion unexpected behavior change

[Mauro Aranda]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

 >>> Now, with emacs -Q:
 >>> M-!
 >>> ls ~/bug/bar
 >>> Put point between "/" and "b" of "bar" and type TAB
 >>> emacs says "No match", but I expected it to offer completions, 
foo-1 and
 >>> foo-2.
 >>>
 >>> That was the behavior, at least in Emacs 28. Reverting the following
 >>> commit, returns this behavior for me:
 >
 > The patch below adds yet another heuristic to try and handle the
 > current case, but really we should rework the pcomplete API so as to
 > provide us the right info to start with.

I see.  Thanks for your explanation.  In the meantime, I confirm that
your patch fixes this use case.

bug#65089: 30.0.50; shell-command filename completion unexpected behavior change

[Stefan Kangas]

Mauro Aranda <maurooaranda@gmail.com> writes:

> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>
>  >>> Now, with emacs -Q:
>  >>> M-!
>  >>> ls ~/bug/bar
>  >>> Put point between "/" and "b" of "bar" and type TAB
>  >>> emacs says "No match", but I expected it to offer completions,
> foo-1 and
>  >>> foo-2.
>  >>>
>  >>> That was the behavior, at least in Emacs 28. Reverting the following
>  >>> commit, returns this behavior for me:
>  >
>  > The patch below adds yet another heuristic to try and handle the
>  > current case, but really we should rework the pcomplete API so as to
>  > provide us the right info to start with.
>
> I see.  Thanks for your explanation.  In the meantime, I confirm that
> your patch fixes this use case.

Was this installed?  I only see that the bug was left open.

bug#65089: 30.0.50; shell-command filename completion unexpected behavior change

[Mauro Aranda]

On 2/9/23 17:07, Stefan Kangas wrote:
 > Mauro Aranda <maurooaranda@gmail.com> writes:
 >
 >> Stefan Monnier <monnier@iro.umontreal.ca> writes:
 >>
 >>  >>> Now, with emacs -Q:
 >>  >>> M-!
 >>  >>> ls ~/bug/bar
 >>  >>> Put point between "/" and "b" of "bar" and type TAB
 >>  >>> emacs says "No match", but I expected it to offer completions,
 >> foo-1 and
 >>  >>> foo-2.
 >>  >>>
 >>  >>> That was the behavior, at least in Emacs 28. Reverting the 
following
 >>  >>> commit, returns this behavior for me:
 >>  >
 >>  > The patch below adds yet another heuristic to try and handle the
 >>  > current case, but really we should rework the pcomplete API so as to
 >>  > provide us the right info to start with.
 >>
 >> I see.  Thanks for your explanation.  In the meantime, I confirm that
 >> your patch fixes this use case.
 >
 > Was this installed?  I only see that the bug was left open.

It looks like it wasn't installed.

bug#65089: 30.0.50; shell-command filename completion unexpected behavior change

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>> I see.  Thanks for your explanation.  In the meantime, I confirm that
>> your patch fixes this use case.
> Was this installed?  I only see that the bug was left open.

I reluctantly pushed it now.
[ Admittedly, the code was already hideous
  before; maybe I should embrace it.  ]


        Stefan

bug#65089: 30.0.50; shell-command filename completion unexpected behavior change

[Mauro Aranda]

close 65089 30.1
quit


On 8/9/23 12:33, Stefan Monnier wrote:
 >>> I see.  Thanks for your explanation.  In the meantime, I confirm that
 >>> your patch fixes this use case.
 >> Was this installed?  I only see that the bug was left open.
 >
 > I reluctantly pushed it now.

The bug was left open, but the patch fixed it and there were no
complaints in almost a month.  So I'm closing it.