bug#22169: 25.0.50; File name compiletion doesn't work with non-ASCII characters on OS X

[Random832 <random832@fastmail.com>]

Eli Zaretskii <eliz@gnu.org> writes:
> I guess some code is not ready to cope with a list of candidate
> completions some of which don't match the string-to-complete.  Can you
> spot which code causes the deletion, and whether that is somehow
> related to file-name-all-completions returning all the 3 file names in
> this case?

It's almost certainly related to that. I couldn't follow all the
details about how the completion code works, but it looks like
the entire design of completion-pcm--merge-completions is based
around finding a common prefix and suffix in the returned
strings irrespective of the originally entered text.

>> I'd expect it to either offer all three filenames, or just a3.
>
> It's not really clear what is correct behavior in this case.  On other
> platforms Emacs will return only a3, but HFS+ stores decomposed
> characters precisely to allow all 3 to match.  So I think we should
> at least cause Emacs return only a3, and ideally also support the
> other behavior as an option.

I'm not aware of any published rationale for the decision to
store decomposed characters. (In my testing I did notice that
zsh and bash handle globbing differently - all of the files
match a* in bash but not zsh.)

I think maybe lax matching as an option would be better than
blindly doing comparisons based on the decomposed form. With
letters with multiple diacritics, for example, the naïve
behavior would mean that one of the one-diacritic forms would
match and the other would not. If users really want that
behavior they can after all just set the file system encoding to
utf-8 instead of utf-8-hfs.

> Btw, why is completion-ignore-case nil on HFS+?  I understand it's a
> case-insensitive file system, isn't it?

No idea. (IIRC In principle it's an option that can be disabled,
though it's case-insensitive by default)

I also feel like I should ask what provisions Emacs has for
filesystem-specific case folding - NTFS and HFS both have their
own algorithms which are different from each other and may both
be different from general-purpose case matching algorithms.

>> Why exactly does completion do matching with encoded prefix
>> against raw filenames, rather than with unicode prefix against
>> decoded filenames, anyway?
>
> Performance: we don't want to decode every file name that readdir
> returns.

I'm not sure there's a way around it if we want to be 100%
correct and consistent, given the existence of parts of the
completion system that do work with the strings in Unicode.