bug#49761: file-cache-minibuffer-complete has become unusable for duplicate file names in MacOSX 27.x emacs

bug#49761: file-cache-minibuffer-complete has become unusable for duplicate file names in MacOSX 27.x emacs

[Aaron Cohen]

[-- Attachment #1: Type: text/plain, Size: 1503 bytes --]

In previous versions of emacs for the last decade, `
file-cache-minibuffer-complete` (C-Tab) worked well for me.

But in the MacOSX 27.x emacs series, it ceased to be able to handle
duplicate file names with extensions.  (It could never handle duplicate
file names with no extension, but that never bothered me much, as the only
files without extensions I need to visit are the Procfile and Gemfile in
the root of the project trees, which are easy to find manually.)

For example, I usually have around 5 projects in my file cache, and
currently across those 5 projects there are 8 files named `dashboard.rb`.

Previously, as soon as I'd gotten C-Tab to complete the file name to
`dashboard.rb`, continuing to hit C-Tab would cycle through the 8 files in
their various directories.

But starting in the 27.x series, C-Tab gets "stuck" on the first file
named `dashboard.rb`, and refuses to cycle through the remaining files with
that name.  (And again, this is always the behavior it has exhibited for
files without extensions: it gets stuck on the first instance, refusing to
cycle through additional instances.)

Also, perhaps relevantly, the behavior of the point while hitting C-Tab
repeatedly on an incomplete file name changed in version 27: it now jumps
to the beginning of the file name while the name is partially complete.
 (In previous versions, it would remain at the end of incomplete file
names, or wherever I'd moved it to in the minibuffer.)

Thanks for any assistance you can provide!!

[-- Attachment #2: Type: text/html, Size: 2818 bytes --]

bug#49761: file-cache-minibuffer-complete has become unusable for duplicate file names in MacOSX 27.x emacs

[Juri Linkov]

> Thanks for any assistance you can provide!!

It would be nice to have a minimal reproducible test case.
What I've tried is to create a minimal set of files with:

  mkdir -p /tmp/a/ ; touch /tmp/a/Procfile /tmp/a/dashboard.rb
  mkdir -p /tmp/b/ ; touch /tmp/b/Procfile /tmp/b/dashboard.rb
  mkdir -p /tmp/c/ ; touch /tmp/c/Procfile /tmp/c/dashboard.rb

Then after adding them to the cache with:

  (file-cache-add-directory-list '("/tmp/a" "/tmp/b" "/tmp/c"))

typing 'C-x C-f d C-TAB' cycles them correctly:

  Find file: /tmp/c/dashboard.rb [1 of 3]
  Find file: /tmp/b/dashboard.rb [2 of 3]
  Find file: /tmp/a/dashboard.rb [3 of 3]

The same for 'C-x C-f P C-TAB' that cycles the files
without file extensions correctly as well:

  Find file: /tmp/c/Procfile [1 of 3]
  Find file: /tmp/b/Procfile [2 of 3]
  Find file: /tmp/a/Procfile [3 of 3]

This works in the development version 28.0.50 without
any significant changes in filecache.el for a long time.

Do you see the same problems when running emacs with -Q
without customization?

bug#49761: file-cache-minibuffer-complete has become unusable for duplicate file names in MacOSX 27.x emacs

[Eli Zaretskii]

> From: Juri Linkov <juri@linkov.net>
> Date: Thu, 29 Jul 2021 20:55:29 +0300
> Cc: 49761@debbugs.gnu.org
> 
> Do you see the same problems when running emacs with -Q
> without customization?

What about the second part of the report, with cursor positioning
after typing an incomplete file name followed by C-TAB: do you see
that problem in your testing?

bug#49761: file-cache-minibuffer-complete has become unusable for duplicate file names in MacOSX 27.x emacs

[Juri Linkov]

>> Do you see the same problems when running emacs with -Q
>> without customization?
>
> What about the second part of the report, with cursor positioning
> after typing an incomplete file name followed by C-TAB: do you see
> that problem in your testing?

No problem with cursor positioning.  filecache.el uses minibuffer-message,
and it works fine in testing with 'emacs -Q'.

bug#49761: file-cache-minibuffer-complete has become unusable for duplicate file names in MacOSX 27.x emacs

[Eli Zaretskii]

> From: Juri Linkov <juri@linkov.net>
> Cc: aaron@brightbytes.net,  49761@debbugs.gnu.org
> Date: Thu, 29 Jul 2021 21:31:52 +0300
> 
> >> Do you see the same problems when running emacs with -Q
> >> without customization?
> >
> > What about the second part of the report, with cursor positioning
> > after typing an incomplete file name followed by C-TAB: do you see
> > that problem in your testing?
> 
> No problem with cursor positioning.

Strange, that's not what I see.  If I type "C-x C-f", type a few
characters that should match several file names, then press C-TAB, the
cursor is placed on the first character I typed, making more typing
cumbersome, because I need first type C-e or somesuch to get to the
end of what I typed.  Isn't that what you see?

bug#49761: file-cache-minibuffer-complete has become unusable for duplicate file names in MacOSX 27.x emacs

[Juri Linkov]

>> > What about the second part of the report, with cursor positioning
>> > after typing an incomplete file name followed by C-TAB: do you see
>> > that problem in your testing?
>> 
>> No problem with cursor positioning.
>
> Strange, that's not what I see.  If I type "C-x C-f", type a few
> characters that should match several file names, then press C-TAB, the
> cursor is placed on the first character I typed, making more typing
> cumbersome, because I need first type C-e or somesuch to get to the
> end of what I typed.  Isn't that what you see?

In my testing the cursor is placed at the end of the minibuffer.

bug#49761: file-cache-minibuffer-complete has become unusable for duplicate file names in MacOSX 27.x emacs

[Aaron Cohen]

[-- Attachment #1: Type: text/plain, Size: 1851 bytes --]

The `-Q` option wasn't working for me, so I stripped everything out of my
`init.el` except a few `global-set-key` forms and the initialization of the
file cache, presented below.

Both the new cursor behavior and the failure to cycle through duplicate
file names still reproduce.

Here's the start of how I load `filecache`: the same procedure is followed
for 4 other projects:

```
;; NOTE - WHEN THE FILECACHE GOES BAD, NUKE ALL FILES MATCHING THE PATTERN
.#* IN THESE DIRECTORY TREES
(require 'filecache)
;; Add all files under dir to file
(eval-after-load
    "filecache"
  '(progn

     (file-cache-add-directory "~/git/clarity_early_warning")
     (file-cache-add-directory-using-find "~/git/clarity_early_warning/app")
     (file-cache-add-directory-using-find "~/git/clarity_early_warning/bin")
     (file-cache-add-directory-using-find
"~/git/clarity_early_warning/config")
     (file-cache-add-directory-using-find "~/git/clarity_early_warning/db")
     (file-cache-add-directory-using-find "~/git/clarity_early_warning/lib")
     (file-cache-add-directory-using-find
"~/git/clarity_early_warning/spec")

;; ... etc ...
```


On Thu, Jul 29, 2021 at 12:41 PM Juri Linkov <juri@linkov.net> wrote:

> >> > What about the second part of the report, with cursor positioning
> >> > after typing an incomplete file name followed by C-TAB: do you see
> >> > that problem in your testing?
> >>
> >> No problem with cursor positioning.
> >
> > Strange, that's not what I see.  If I type "C-x C-f", type a few
> > characters that should match several file names, then press C-TAB, the
> > cursor is placed on the first character I typed, making more typing
> > cumbersome, because I need first type C-e or somesuch to get to the
> > end of what I typed.  Isn't that what you see?
>
> In my testing the cursor is placed at the end of the minibuffer.
>

[-- Attachment #2: Type: text/html, Size: 3457 bytes --]

bug#49761: file-cache-minibuffer-complete has become unusable for duplicate file names in MacOSX 27.x emacs

[Juri Linkov]

> The `-Q` option wasn't working for me, so I stripped everything out of my
> `init.el` except a few `global-set-key` forms and the initialization of the
> file cache, presented below.
>
> Both the new cursor behavior and the failure to cycle through duplicate
> file names still reproduce.
>
> Here's the start of how I load `filecache`: the same procedure is followed
> for 4 other projects:
> [...]
>      (file-cache-add-directory "~/git/clarity_early_warning")
>      (file-cache-add-directory-using-find "~/git/clarity_early_warning/app")
> ;; ... etc ...

Thanks, now I tried to add two separate emacs/lisp trees
with file-cache-add-directory-using-find that gives such results:

1. 'filec C-TAB' cycles between 2 filecache.el files from both Emacs repos;

2. 'README C-TAB' cycles between 4 README files, two files from each tree,
   that shows that it can cycle between duplicate files without extensions;

3. 'TAGS C-TAB' gets stuck with the message [File Cache: complete but not unique]
   Do you see the same message?

4. 'make C-TAB' then the completions buffer is displayed and the cursor
   jumps to the beginning because there are completions that don't begin
   with the same prefix "make" such as flymake.el, pmake.el, ob-makefile.el.

   But the cursor doesn't jump to the beginning when all completions share
   the same prefix, e.g. 'fly C-TAB' that displays flymake.el and flyspell.el.

bug#49761: file-cache-minibuffer-complete has become unusable for duplicate file names in MacOSX 27.x emacs

[Eli Zaretskii]

> From: Juri Linkov <juri@linkov.net>
> Cc: Eli Zaretskii <eliz@gnu.org>,  49761@debbugs.gnu.org
> Date: Fri, 30 Jul 2021 01:49:18 +0300
> 
> 4. 'make C-TAB' then the completions buffer is displayed and the cursor
>    jumps to the beginning because there are completions that don't begin
>    with the same prefix "make" such as flymake.el, pmake.el, ob-makefile.el.

Is this reasonable behavior?  It seems to try second-guessing what the
user will do next, but that guess is not necessarily correct.

bug#49761: file-cache-minibuffer-complete has become unusable for duplicate file names in MacOSX 27.x emacs

[Aaron Cohen]

[-- Attachment #1: Type: text/plain, Size: 2223 bytes --]

Comments:

2. The cycling is broken if you have one file without an extension with a
given name and another file with the same name and an extension.  For
example, all 5 of my projects are currently Ruby on Rails repos.  Every RoR
project has a `Gemfile` and a `Gemfile.lock`.  When I try to find a
particular Gemfile, C-Tab gets stuck on the first one.  This has always
been the case with C-Tab, so it doesn't bother me, though if there is a
workaround I'd love to know it.

3. Yes, I see the message "[File Cache: complete but not unique]" whenever
it gets stuck.  This has always been the case for 2, above, and in Emacs
27.x (and presumably 28.x) it is the case when a duplicated file has a
duplicate and also another file with a similar ending.  In my case, there
are 9 files named `dashboard.rb` across several projects, and even more
files across several projects that follow the pattern `*_dashboard.rb`.

4. The cursor-jumping-to-the-beginning-of-the-filename behavior is new in
Emacs 27.x.  It doesn't happen in case 2, above ... though that's because
the overlap termination location in that case is at the end of the
completed file name (the dot in `Gemfile.lock`). Whereas in the new
behavior, the overlap termination location is at the beginning (e.g. in my
project, the underscore in `db_dashboard.rb` or `homepage_dashboard.rb`).

In any case, the larger issue is that a prefix overlap is now causing C-Tab
to get stuck.  The point goes where it should to assist the User in
resolving the overlap.  It's not where the point goes but rather that Emacs
gets stuck on the common suffix that is the new issue.







On Thu, Jul 29, 2021 at 10:50 PM Eli Zaretskii <eliz@gnu.org> wrote:

> > From: Juri Linkov <juri@linkov.net>
> > Cc: Eli Zaretskii <eliz@gnu.org>,  49761@debbugs.gnu.org
> > Date: Fri, 30 Jul 2021 01:49:18 +0300
> >
> > 4. 'make C-TAB' then the completions buffer is displayed and the cursor
> >    jumps to the beginning because there are completions that don't begin
> >    with the same prefix "make" such as flymake.el, pmake.el,
> ob-makefile.el.
>
> Is this reasonable behavior?  It seems to try second-guessing what the
> user will do next, but that guess is not necessarily correct.
>

[-- Attachment #2: Type: text/html, Size: 4283 bytes --]

bug#49761: file-cache-minibuffer-complete has become unusable for duplicate file names in MacOSX 27.x emacs

[Eli Zaretskii]

> From: Aaron Cohen <aaron@brightbytes.net>
> Date: Thu, 29 Jul 2021 23:41:12 -0700
> Cc: Juri Linkov <juri@linkov.net>, 49761@debbugs.gnu.org
> 
> In any case, the larger issue is that a prefix overlap is now causing C-Tab to get stuck.  The point goes
> where it should to assist the User in resolving the overlap.  It's not where the point goes but rather that
> Emacs gets stuck on the common suffix that is the new issue.

I suggest to play with your value of completion-styles: perhaps by
reordering it or by removing the more-aggressive styles from there you
will be able to get your desired behavior.

bug#49761: file-cache-minibuffer-complete has become unusable for duplicate file names in MacOSX 27.x emacs

[Juri Linkov]

>> 4. 'make C-TAB' then the completions buffer is displayed and the cursor
>>    jumps to the beginning because there are completions that don't begin
>>    with the same prefix "make" such as flymake.el, pmake.el, ob-makefile.el.
>
> Is this reasonable behavior?  It seems to try second-guessing what the
> user will do next, but that guess is not necessarily correct.

I don't know, this is the default behavior for completions
that don't share a common prefix.

bug#49761: file-cache-minibuffer-complete has become unusable for duplicate file names in MacOSX 27.x emacs

[Aaron Cohen]

[-- Attachment #1: Type: text/plain, Size: 1804 bytes --]

Well, it's now the default behavior.  It wasn't previously.  :-)

That is, I haven't changed my completion settings since the mid-90s (except
the icomplete-mode, which I set up a decade ago) , and they all worked fine
until emacs 27.

They look like this:

;;; Shows globs of possible completions in minibuffer
(setq-default completion-auto-help t)

;;; Case-insensitive file and buff
(setq read-file-name-completion-ignore-case t)
(setq read-buffer-completion-ignore-case t)

;;; Show completions in M-x
(icomplete-mode 1)

Oh, you mean
https://www.gnu.org/software/emacs/manual/html_node/emacs/Completion-Styles.html

`M-: completion-styles` shows the same thing in Emacs 26 and 27: '(basic
partial-completion emacs22)

In Emacs 27, I tried `(setq completion-styles '(basic emacs22))` and `(setq
completion-styles '(emacs22))`, but the completion behavior remained broken.

(Perhaps interestingly, when I tried `(setq completion-styles '(emacs22))`
in Emacs 26, while it still got stuck on the first Gemfile, the minibuffer
started showing me that the collision was caused by Gemfile.lock, which it
never did before.)

In any case, it seems like I'll need to stay with Emacs 26 for a while, as
C-Tab is by far my most-used minibuffer command.



On Fri, Jul 30, 2021 at 11:03 AM Juri Linkov <juri@linkov.net> wrote:

> >> 4. 'make C-TAB' then the completions buffer is displayed and the cursor
> >>    jumps to the beginning because there are completions that don't begin
> >>    with the same prefix "make" such as flymake.el, pmake.el,
> ob-makefile.el.
> >
> > Is this reasonable behavior?  It seems to try second-guessing what the
> > user will do next, but that guess is not necessarily correct.
>
> I don't know, this is the default behavior for completions
> that don't share a common prefix.
>

[-- Attachment #2: Type: text/html, Size: 3243 bytes --]

bug#49761: file-cache-minibuffer-complete has become unusable for duplicate file names in MacOSX 27.x emacs

[Eli Zaretskii]

> From: Aaron Cohen <aaron@brightbytes.net>
> Date: Fri, 30 Jul 2021 11:20:51 -0700
> Cc: Eli Zaretskii <eliz@gnu.org>, 49761@debbugs.gnu.org
> 
> Well, it's now the default behavior.  It wasn't previously.  :-)

I wasn't trying to find the reason for the change in behavior, I was
trying to help you get the behavior you want back.

bug#49761: file-cache-minibuffer-complete has become unusable for duplicate file names in MacOSX 27.x emacs

[Aaron Cohen]

[-- Attachment #1: Type: text/plain, Size: 731 bytes --]

Sorry, that remark was a response (with a smile to show benign intent) to
Juri's comment:

> ... this is the default behavior for completions that don't share a
common prefix.

... and not a response to any of your comments.  No offense meant - I
really appreciate your having spent any time responding to me!!

On Fri, Jul 30, 2021 at 11:48 AM Eli Zaretskii <eliz@gnu.org> wrote:

> > From: Aaron Cohen <aaron@brightbytes.net>
> > Date: Fri, 30 Jul 2021 11:20:51 -0700
> > Cc: Eli Zaretskii <eliz@gnu.org>, 49761@debbugs.gnu.org
> >
> > Well, it's now the default behavior.  It wasn't previously.  :-)
>
> I wasn't trying to find the reason for the change in behavior, I was
> trying to help you get the behavior you want back.
>

[-- Attachment #2: Type: text/html, Size: 1941 bytes --]

bug#49761: file-cache-minibuffer-complete has become unusable for duplicate file names in MacOSX 27.x emacs

[Juri Linkov]

tags 49761 fixed
close 49761 28.0.50
thanks

> Well, it's now the default behavior.  It wasn't previously.  :-)

And new is not always better ;-)

So now the new behavior (that became old now) was fixed
in the master branch for Emacs 28, and all test cases
that you presented work completely as expected.
Thanks for helping to understand where the problem was.

BTW, after the fix, the following additional information
is not necessary, but I discovered that in old versions
you can use the prefix argument to force C-TAB cycling.

An excerpt from etc/NEWS.20:

  ** file-cache-minibuffer-complete now accepts a prefix argument.
  With a prefix argument, it does not try to do completion of
  the file name within its directory; it only checks for other
  directories that contain the same file name.
  Thus, given the file name Makefile, and assuming that a file
  Makefile.in exists in the same directory, ordinary
  file-cache-minibuffer-complete will try to complete Makefile to
  Makefile.in and will therefore never look for other directories that
  have Makefile.  A prefix argument tells it not to look for longer
  names such as Makefile.in, so that instead it will look for other
  directories--just as if the name were already complete in its present
  directory.

and a comment from bindings.el:

  ;; The prefix argument works around a bug in the minibuffer completion.
  ;; The completion function doesn't distinguish between the states:
  ;;
  ;;   "Multiple completions of name" (eg, Makefile, Makefile.in)
  ;;   "Name available in multiple directories" (/tmp/Makefile, ~me/Makefile)
  ;;
  ;; The default is to do the former; a prefix arg forces the latter.

bug#49761: file-cache-minibuffer-complete has become unusable for duplicate file names in MacOSX 27.x emacs

[Aaron Cohen]

[-- Attachment #1: Type: text/plain, Size: 2096 bytes --]

Very much appreciated!!

And thank you for investigating and isolating the issue: I had
misidentified the actual problem and didn't provide a duplication recipe,
making your job much harder.

I promise to do better next time, and provide a test case.  :-)

Thanks again!!

On Sun, Aug 1, 2021 at 1:41 AM Juri Linkov <juri@linkov.net> wrote:

> tags 49761 fixed
> close 49761 28.0.50
> thanks
>
> > Well, it's now the default behavior.  It wasn't previously.  :-)
>
> And new is not always better ;-)
>
> So now the new behavior (that became old now) was fixed
> in the master branch for Emacs 28, and all test cases
> that you presented work completely as expected.
> Thanks for helping to understand where the problem was.
>
> BTW, after the fix, the following additional information
> is not necessary, but I discovered that in old versions
> you can use the prefix argument to force C-TAB cycling.
>
> An excerpt from etc/NEWS.20:
>
>   ** file-cache-minibuffer-complete now accepts a prefix argument.
>   With a prefix argument, it does not try to do completion of
>   the file name within its directory; it only checks for other
>   directories that contain the same file name.
>   Thus, given the file name Makefile, and assuming that a file
>   Makefile.in exists in the same directory, ordinary
>   file-cache-minibuffer-complete will try to complete Makefile to
>   Makefile.in and will therefore never look for other directories that
>   have Makefile.  A prefix argument tells it not to look for longer
>   names such as Makefile.in, so that instead it will look for other
>   directories--just as if the name were already complete in its present
>   directory.
>
> and a comment from bindings.el:
>
>   ;; The prefix argument works around a bug in the minibuffer completion.
>   ;; The completion function doesn't distinguish between the states:
>   ;;
>   ;;   "Multiple completions of name" (eg, Makefile, Makefile.in)
>   ;;   "Name available in multiple directories" (/tmp/Makefile,
> ~me/Makefile)
>   ;;
>   ;; The default is to do the former; a prefix arg forces the latter.
>

[-- Attachment #2: Type: text/html, Size: 3249 bytes --]

bug#49761: file-cache-minibuffer-complete has become unusable for duplicate file names in MacOSX 27.x emacs

[Aaron Cohen]

[-- Attachment #1: Type: text/plain, Size: 2419 bytes --]

Oh, and I've confirmed that the workaround of `C-u C-Tab` works great for
cycling through files when `C-Tab` would otherwise get stuck.  So, I'm good
until the next official release.

On Mon, Aug 2, 2021 at 1:02 PM Aaron Cohen <aaron@brightbytes.net> wrote:

> Very much appreciated!!
>
> And thank you for investigating and isolating the issue: I had
> misidentified the actual problem and didn't provide a duplication recipe,
> making your job much harder.
>
> I promise to do better next time, and provide a test case.  :-)
>
> Thanks again!!
>
> On Sun, Aug 1, 2021 at 1:41 AM Juri Linkov <juri@linkov.net> wrote:
>
>> tags 49761 fixed
>> close 49761 28.0.50
>> thanks
>>
>> > Well, it's now the default behavior.  It wasn't previously.  :-)
>>
>> And new is not always better ;-)
>>
>> So now the new behavior (that became old now) was fixed
>> in the master branch for Emacs 28, and all test cases
>> that you presented work completely as expected.
>> Thanks for helping to understand where the problem was.
>>
>> BTW, after the fix, the following additional information
>> is not necessary, but I discovered that in old versions
>> you can use the prefix argument to force C-TAB cycling.
>>
>> An excerpt from etc/NEWS.20:
>>
>>   ** file-cache-minibuffer-complete now accepts a prefix argument.
>>   With a prefix argument, it does not try to do completion of
>>   the file name within its directory; it only checks for other
>>   directories that contain the same file name.
>>   Thus, given the file name Makefile, and assuming that a file
>>   Makefile.in exists in the same directory, ordinary
>>   file-cache-minibuffer-complete will try to complete Makefile to
>>   Makefile.in and will therefore never look for other directories that
>>   have Makefile.  A prefix argument tells it not to look for longer
>>   names such as Makefile.in, so that instead it will look for other
>>   directories--just as if the name were already complete in its present
>>   directory.
>>
>> and a comment from bindings.el:
>>
>>   ;; The prefix argument works around a bug in the minibuffer completion.
>>   ;; The completion function doesn't distinguish between the states:
>>   ;;
>>   ;;   "Multiple completions of name" (eg, Makefile, Makefile.in)
>>   ;;   "Name available in multiple directories" (/tmp/Makefile,
>> ~me/Makefile)
>>   ;;
>>   ;; The default is to do the former; a prefix arg forces the latter.
>>
>

[-- Attachment #2: Type: text/html, Size: 4031 bytes --]