bug#22169: 25.0.50; File name compiletion doesn't work with non-ASCII characters on OS X

bug#22169: 25.0.50; File name compiletion doesn't work with non-ASCII characters on OS X

[Anders Lindgren]

[-- Attachment #1: Type: text/plain, Size: 3267 bytes --]

Hi!

File name completion doesn't work with non-ASCII characters on OS X.

Steps to repeat:

    In an empty directory:
    echo "alpha" > åäöalpha.txt
    echo "beta" > åäöbeta.txt
    Emacs -Q
    C-x C-f
    TAB          # Emacs correctly echoes "åäö"
    TAB          # Emacs incorrectly says "[No match]"

     C-g          # To get out of the previous C-x C-f
     C-x C-f
     å TAB      # Emacs incorrectly says "[No match]"

The OS X file system is a bit bizarre in that it stores filenames using the
decomposed UTF-8 format (for characters in specific ranges), this might
have something to do with this.

Emacs can open the files if the full name is manually specified. Also,
dired seems to work OK as well.

    -- Anders Lindgren


In GNU Emacs 25.0.50.1 (x86_64-apple-darwin10.8.0, NS appkit-1038.36
Version 10.6.8 (Build 10K549))
 of 2015-12-11
Repository revision: 83114ccf77d2a5d59fccbdbda6edefacce1b979e
Windowing system distributor 'Apple', version 10.3.1038
Configured using:
 'configure --without-dbus --with-ns'

Configured features:
ACL LIBXML2 ZLIB TOOLKIT_SCROLL_BARS NS

Important settings:
  value of $LC_CTYPE: UTF-8
  locale-coding-system: utf-8-unix

Major mode: Lisp Interaction

Minor modes in effect:
  tooltip-mode: t
  global-eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  transient-mark-mode: t

Recent messages:
For information about GNU Emacs and the GNU system, type C-h C-a.
Making completion list...

Load-path shadows:
None found.

Features:
(shadow sort gnus-util mail-extr emacsbug message dired format-spec
rfc822 mml mml-sec mm-decode mm-bodies mm-encode mail-parse rfc2231
mailabbrev gmm-utils mailheader sendmail rfc2047 rfc2045 ietf-drums
mm-util help-fns mail-prsvr mail-utils seq byte-opt gv bytecomp
byte-compile cconv cl-extra help-mode easymenu cl-loaddefs pcase cl-lib
time-date mule-util tooltip eldoc electric uniquify ediff-hook vc-hooks
lisp-float-type mwheel ns-win term/common-win tool-bar dnd fontset image
regexp-opt fringe tabulated-list newcomment elisp-mode lisp-mode
prog-mode register page menu-bar rfn-eshadow timer select scroll-bar
mouse jit-lock font-lock syntax facemenu font-core frame cl-generic cham
georgian utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao
korean japanese eucjp-ms cp51932 hebrew greek romanian slovak czech
european ethiopic indian cyrillic chinese charscript case-table epa-hook
jka-cmpr-hook help simple abbrev minibuffer cl-preloaded nadvice
loaddefs button faces cus-face macroexp files text-properties overlay
sha1 md5 base64 format env code-pages mule custom widget
hashtable-print-readable backquote cocoa ns multi-tty
make-network-process emacs)

Memory information:
((conses 16 86418 9491)
 (symbols 48 19326 0)
 (miscs 40 47 155)
 (strings 32 15120 4410)
 (string-bytes 1 446648)
 (vectors 16 11888)
 (vector-slots 8 415801 4426)
 (floats 8 150 89)
 (intervals 56 206 0)
 (buffers 976 12))

[-- Attachment #2: Type: text/html, Size: 4380 bytes --]

bug#22169: 25.0.50; File name compiletion doesn't work with non-ASCII characters on OS X

[Eli Zaretskii]

> Date: Mon, 14 Dec 2015 20:08:27 +0100
> From: Anders Lindgren <andlind@gmail.com>
> 
> In an empty directory:
> echo "alpha" > åäöalpha.txt
> echo "beta" > åäöbeta.txt
> Emacs -Q
> C-x C-f
> TAB # Emacs correctly echoes "åäö"
> TAB # Emacs incorrectly says "[No match]"
> 
> C-g # To get out of the previous C-x C-f
> C-x C-f
> å TAB # Emacs incorrectly says "[No match]"
> 
> The OS X file system is a bit bizarre in that it stores filenames using the
> decomposed UTF-8 format (for characters in specific ranges), this might have
> something to do with this.

This shouldn't cause any trouble, since Emacs encodes file names
before passing them to readdir, and decodes the results.  The encoding
and decoding process should take care of the decomposition and
composition.

What is the value of file-name-coding-system on that system?

bug#22169: 25.0.50; File name compiletion doesn't work with non-ASCII characters on OS X

[Random832]

On 2015-12-14, Anders Lindgren <andlind@gmail.com> wrote:
> The OS X file system is a bit bizarre in that it stores filenames using the
> decomposed UTF-8 format (for characters in specific ranges), this might
> have something to do with this.

Can you confirm whether:
A) the contents of the minibuffer (or e.g. a dired buffer) are
   in decomposed format (i.e. can you backspace the accent marks
   separately with DEL)?
B) the filenames are returned in decomposed format, normal
   UTF-8, or some other format from readdir()?
C) Does installing the utf-8m encoding from Carbon Emacs solve
   the problem?

bug#22169: 25.0.50; File name compiletion doesn't work with non-ASCII characters on OS X

[Eli Zaretskii]

> Date: Mon, 14 Dec 2015 21:20:09 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: 22169@debbugs.gnu.org
> 
> What is the value of file-name-coding-system on that system?

And if that is nil, as it probably should, what is the value of
default-file-name-coding-system?

bug#22169: 25.0.50; File name compiletion doesn't work with non-ASCII characters on OS X

[Anders Lindgren]

[-- Attachment #1: Type: text/plain, Size: 264 bytes --]

>
> > What is the value of file-name-coding-system on that system?
>

file-name-coding-system
utf-8-nfd



> And if that is nil, as it probably should, what is the value of
> default-file-name-coding-system?
>

default-file-name-coding-system
utf-8

    -- Anders

[-- Attachment #2: Type: text/html, Size: 909 bytes --]

bug#22169: 25.0.50; File name compiletion doesn't work with non-ASCII ch

[Anders Lindgren]

[-- Attachment #1: Type: text/plain, Size: 676 bytes --]

Hi!

> Can you confirm whether:
>

>A) the contents of the minibuffer (or e.g. a dired buffer) are
>   in decomposed format (i.e. can you backspace the accent marks
>   separately with DEL)?

When I backspace I delete a full character.


>B) the filenames are returned in decomposed format, normal
>   UTF-8, or some other format from readdir()?

The low-level C function? I'll see if I find some time in the next couple
of days.


>C) Does installing the utf-8m encoding from Carbon Emacs solve
>   the problem?

Carbon Emacs? The one that YAMAMOTO Mitsuharu is maintaining? I haven't got
it installed, but I can see if I can do it in the next couple of days.

    -- Anders

[-- Attachment #2: Type: text/html, Size: 1057 bytes --]

bug#22169: 25.0.50; File name compiletion doesn't work with non-ASCII characters on OS X

[Eli Zaretskii]

> Date: Mon, 14 Dec 2015 23:07:21 +0100
> From: Anders Lindgren <andlind@gmail.com>
> Cc: 22169@debbugs.gnu.org
> 
>     > What is the value of file-name-coding-system on that system?
>     
> 
> file-name-coding-system
> utf-8-nfd

Where is that defined?

Can you try using utf-8-hfs?  (You might need to load ucs-normalize
first.)

bug#22169: 25.0.50; File name compiletion doesn't work with non-ASCII characters on OS X

[Anders Lindgren]

[-- Attachment #1: Type: text/plain, Size: 524 bytes --]

>
> > file-name-coding-system
> > utf-8-nfd
>
> Where is that defined?
>

In ns-win.el


Can you try using utf-8-hfs?  (You might need to load ucs-normalize
> first.)
>

It behaves like the original, as far as I can tell.

I tried setting it to nil. This made completion work. However, the letters
are presented in decomposed form, so that pressing backspace first converts
"å" to "a", a second backspace deletes the "a" -- this is not how we would
like to present file names to users.

    -- Anders

[-- Attachment #2: Type: text/html, Size: 1137 bytes --]

bug#22169: 25.0.50; File name compiletion doesn't work with non-ASCII characters on OS X

[Andreas Schwab]

Anders Lindgren <andlind@gmail.com> writes:

> I tried setting it to nil. This made completion work. However, the letters
> are presented in decomposed form, so that pressing backspace first converts
> "å" to "a", a second backspace deletes the "a" -- this is not how we would
> like to present file names to users.

That's how composed characters work in Emacs.

Andreas.

-- 
Andreas Schwab, SUSE Labs, schwab@suse.de
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

bug#22169: 25.0.50; File name compiletion doesn't work with non-ASCII characters on OS X

[Anders Lindgren]

[-- Attachment #1: Type: text/plain, Size: 727 bytes --]

>
> > I tried setting it to nil. This made completion work. However, the
> letters
> > are presented in decomposed form, so that pressing backspace first
> converts
> > "å" to "a", a second backspace deletes the "a" -- this is not how we
> would
> > like to present file names to users.
>
> That's how composed characters work in Emacs.
>

Andreas,

The OS X file system *stores* filenames in a decomposed manner, that is
true. However, they should be presented to the user as normal (composed)
characters. If `file-name-coding-system' has the original value, they are.
However, the problem is that the completion mechanism fails to handle this
case, which is a bug and it should be fixed.

    -- Anders

[-- Attachment #2: Type: text/html, Size: 1078 bytes --]

bug#22169: 25.0.50; File name compiletion doesn't work with non-ASCII characters on OS X

[Eli Zaretskii]

> Date: Tue, 15 Dec 2015 06:12:47 +0100
> From: Anders Lindgren <andlind@gmail.com>
> Cc: 22169@debbugs.gnu.org
> 
>     > file-name-coding-system
>     > utf-8-nfd
>     
>     Where is that defined?
> 
> In ns-win.el

I think we should remove that, and leave behind an alias that uses
utf-8-hfs, which is provided by Emacs.  There's no reason to maintain
2 identical definitions.

> I tried setting it to nil. This made completion work. However, the letters are
> presented in decomposed form, so that pressing backspace first converts "å" to
> "a", a second backspace deletes the "a" -- this is not how we would like to
> present file names to users.

When you set file-name-coding-system to nil, Emacs uses
default-file-name-system, which is utf-8, so it doesn't
compose/decompose characters, and that's why you see what you see.
IOW, using nil is a step backward.

What does this return:

  M-: (file-name-all-completion "åäö" "/that/empty/directory/") RET

Also, what is your value of completion-ignore-case?

bug#22169: 25.0.50; File name compiletion doesn't work with non-ASCII characters on OS X

[Eli Zaretskii]

> Date: Tue, 15 Dec 2015 11:21:17 +0100
> From: Anders Lindgren <andlind@gmail.com>
> Cc: Eli Zaretskii <eliz@gnu.org>, 22169@debbugs.gnu.org
> 
>     > I tried setting it to nil. This made completion work. However, the
>     letters
>     > are presented in decomposed form, so that pressing backspace first
>     converts
>     > "å" to "a", a second backspace deletes the "a" -- this is not how we
>     would
>     > like to present file names to users.
>     
>     That's how composed characters work in Emacs.
> 
> The OS X file system *stores* filenames in a decomposed manner, that is true.
> However, they should be presented to the user as normal (composed) characters.
> If `file-name-coding-system' has the original value, they are. However, the
> problem is that the completion mechanism fails to handle this case, which is a
> bug and it should be fixed.

Andreas just pointed out that when character composition happens at
display time, the behavior you observe is normal, that's all.

The issue of why completion doesn't work still exists.

bug#22169: 25.0.50; File name compiletion doesn't work with non-ASCII characters on OS X

[Anders Lindgren]

[-- Attachment #1: Type: text/plain, Size: 983 bytes --]

>
> I think we should remove that, and leave behind an alias that uses
> utf-8-hfs, which is provided by Emacs.  There's no reason to maintain
> 2 identical definitions.
>

Sounds reasonable. The implementation is vastly different, so getting rid
of one is definitively an improvement.


When you set file-name-coding-system to nil, Emacs uses
> default-file-name-system, which is utf-8, so it doesn't
> compose/decompose characters, and that's why you see what you see.
> IOW, using nil is a step backward.
>

I couldn't agree more!


What does this return:
>
>   M-: (file-name-all-completion "åäö" "/that/empty/directory/") RET
>

It returns nil.


Also, what is your value of completion-ignore-case?
>

It's nil.

Just out of curiosity -- how does `file-name-all-completions' work? Is the
FILE argument encoded to decomposed form, is the file list converted to
composed form, or is this handled by the comparison functions?

    -- Anders

[-- Attachment #2: Type: text/html, Size: 2159 bytes --]

bug#22169: 25.0.50; File name compiletion doesn't work with non-ASCII characters on OS X

[Eli Zaretskii]

> Date: Tue, 15 Dec 2015 20:16:03 +0100
> From: Anders Lindgren <andlind@gmail.com>
> Cc: 22169@debbugs.gnu.org
> 
>     I think we should remove that, and leave behind an alias that uses
>     utf-8-hfs, which is provided by Emacs. There's no reason to maintain
>     2 identical definitions.
> 
> Sounds reasonable. The implementation is vastly different, so getting rid of
> one is definitively an improvement.

Can you write a patch to that effect, for emacs-25 branch?

>     What does this return:
>     
>     M-: (file-name-all-completion "åäö" "/that/empty/directory/") RET
> 
> It returns nil.

So this is the heart of the problem.  I assume that if you do the same
with an ASCII first argument, the result is non-nil, yes?

Then the next step is to step with a debugger through
file_name_completion, and see why this returns nil instead of a list
of files that begin.

> Just out of curiosity -- how does `file-name-all-completions' work? Is the FILE
> argument encoded to decomposed form, is the file list converted to composed
> form, or is this handled by the comparison functions?

See dired.c:file_name_completion.  In a nutshell, we do this:

  . encode the file argument
  . encode the directory argument and pass it to opendir
  . loop calling readdir, and for each file name it returns:
    . if the file name begins with the same characters as the encoded
      file argument, then:
      . decode the file name
      . cons the decoded name onto the list to be returned

The above is for file-name-all-completions; for file-name-completion
the last step is more complicated, but we should understand the
file-name-all-completions case first.

When you step through the code, please pay attention to the encoded
file names.  My guess is that somehow the call to scmp around line 500
fails, or maybe we don't count characters correctly in this case.

Thanks.

bug#22169: 25.0.50; File name compiletion doesn't work with non-ASCII characters on OS X

[Anders Lindgren]

[-- Attachment #1: Type: text/plain, Size: 789 bytes --]

Hi,


> Can you write a patch to that effect, for emacs-25 branch?
>

We have the find the cause of the problem first. But once we do that, this
should be straight forward.


>     What does this return:
> >
> >     M-: (file-name-all-completion "åäö" "/that/empty/directory/") RET
> >
> > It returns nil.
>
> So this is the heart of the problem.  I assume that if you do the same
> with an ASCII first argument, the result is non-nil, yes?
>

Yes.



> Then the next step is to step with a debugger through
> file_name_completion, and see why this returns nil instead of a list
> of files that begin.
>

Auhm, I'll see what I can do. I'm a family father and have very, very,
limited time, but I can see in I can find a time slot for it.

    -- Anders

[-- Attachment #2: Type: text/html, Size: 1474 bytes --]

bug#22169: 25.0.50; File name compiletion doesn't work with non-ASCII characters on OS X

[Random832]

Eli Zaretskii <eliz@gnu.org> writes:
>   . encode the file argument
>   . encode the directory argument and pass it to opendir
>   . loop calling readdir, and for each file name it returns:
>     . if the file name begins with the same characters as the encoded
>       file argument, then:
>       . decode the file name
>       . cons the decoded name onto the list to be returned

My guess from the symptoms is that utf-8-nfd doesn't actually
bother to make any attempt to convert to decomposed form when
encoding, since in *most* cases e.g. for opening a file, the
underlying filesystem will take care of this automatically.

This is backed up by the fact that, looking at the code, it
apparently has a post-read-conversion but no matching
pre-write-conversion.

Anders Lindgren writes:
> I tried setting it to nil. This made completion work. However,
> the letters are presented in decomposed form, so that pressing
> backspace first converts "å" to "a", a second backspace
> deletes the "a" -- this is not how we would like to present
> file names to users.

Why? That _is_, for better or worse, the filename on the
disk.  On a non-OSX system, someone might actually have such a
filename, distinct from the composed one.

For that matter, what happens if an OSX user saves or opens a
file on a non-HFS filesystem? Can Emacs handle the concept of
different filesystems having different encodings?

Ultimately, this isn't really an encoding - it is a destructive
folding operation performed by the filesystem (the same as if,
say, some filesystem stored filenames in all uppercase), and
we've decided, for some reason, that we want the filenames back
in what we've judged to be more likely to be the original form.

bug#22169: 25.0.50; File name compiletion doesn't work with non-ASCII characters on OS X

[Eli Zaretskii]

> From: Random832 <random832@fastmail.com>
> Date: Tue, 15 Dec 2015 16:53:37 -0500
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> >   . encode the file argument
> >   . encode the directory argument and pass it to opendir
> >   . loop calling readdir, and for each file name it returns:
> >     . if the file name begins with the same characters as the encoded
> >       file argument, then:
> >       . decode the file name
> >       . cons the decoded name onto the list to be returned
> 
> My guess from the symptoms is that utf-8-nfd doesn't actually
> bother to make any attempt to convert to decomposed form when
> encoding, since in *most* cases e.g. for opening a file, the
> underlying filesystem will take care of this automatically.
> 
> This is backed up by the fact that, looking at the code, it
> apparently has a post-read-conversion but no matching
> pre-write-conversion.

I certainly see a pre-write-conversion function in ucs-normalize.el:
ucs-normalize-hfs-nfd-pre-write-conversion which calls
ucs-normalize-HFS-NFD-region.  So I'm not sure I understand what you
are saying.

bug#22169: 25.0.50; File name compiletion doesn't work with non-ASCII characters on OS X

[Random832]

Eli Zaretskii <eliz@gnu.org> writes:
> I certainly see a pre-write-conversion function in ucs-normalize.el:
> ucs-normalize-hfs-nfd-pre-write-conversion which calls
> ucs-normalize-HFS-NFD-region.  So I'm not sure I understand what you
> are saying.

I was talking about the "utf-8-nfd" encoding in ns-win.el. I'd
missed the statement that the problem had been reproduced with
utf-8-hfs.

I can't actually reproduce the problem myself with utf-8-hfs.  I
thought I had it once, but only immediately after switching from
utf-8-nfd (maybe the bad completion result from utf-8-nfd was in
some kind of cache?), and now I can't even reproduce that.
Otherwise, it seems to fix the problem.  Anders, can you try
this again from a clean emacs -Q session, and in particular load
ucs-normalize and set the coding system to utf-8-hfs _before_
attempting any completion?

--

Incidentally, I do get one other bit of bizarre behavior
associated with this.  If I have multiple files that start with
the same base letter and different (or no) accents, pressing TAB
_deletes_ that letter.  E.g. files: à1 á2 a3.  C-x C-f a TAB,
deletes the "a". I'd expect it to either offer all three
filenames, or just a3.

Why exactly does completion do matching with encoded prefix
against raw filenames, rather than with unicode prefix against
decoded filenames, anyway?

bug#22169: 25.0.50; File name compiletion doesn't work with non-ASCII characters on OS X

[Eli Zaretskii]

> From: Random832 <random832@fastmail.com>
> Date: Wed, 16 Dec 2015 00:05:40 -0500
> 
> Anders, can you try this again from a clean emacs -Q session, and in
> particular load ucs-normalize and set the coding system to utf-8-hfs
> _before_ attempting any completion?

I certainly hope so, thanks for testing.

> Incidentally, I do get one other bit of bizarre behavior
> associated with this.  If I have multiple files that start with
> the same base letter and different (or no) accents, pressing TAB
> _deletes_ that letter.  E.g. files: à1 á2 a3.  C-x C-f a TAB,
> deletes the "a".

I guess some code is not ready to cope with a list of candidate
completions some of which don't match the string-to-complete.  Can you
spot which code causes the deletion, and whether that is somehow
related to file-name-all-completions returning all the 3 file names in
this case?

> I'd expect it to either offer all three filenames, or just a3.

It's not really clear what is correct behavior in this case.  On other
platforms Emacs will return only a3, but HFS+ stores decomposed
characters precisely to allow all 3 to match.  So I think we should
at least cause Emacs return only a3, and ideally also support the
other behavior as an option.

Btw, why is completion-ignore-case nil on HFS+?  I understand it's a
case-insensitive file system, isn't it?

> Why exactly does completion do matching with encoded prefix
> against raw filenames, rather than with unicode prefix against
> decoded filenames, anyway?

Performance: we don't want to decode every file name that readdir
returns.

bug#22169: 25.0.50; File name compiletion doesn't work with non-ASCII characters on OS X

[Random832]

Eli Zaretskii <eliz@gnu.org> writes:
> I guess some code is not ready to cope with a list of candidate
> completions some of which don't match the string-to-complete.  Can you
> spot which code causes the deletion, and whether that is somehow
> related to file-name-all-completions returning all the 3 file names in
> this case?

It's almost certainly related to that. I couldn't follow all the
details about how the completion code works, but it looks like
the entire design of completion-pcm--merge-completions is based
around finding a common prefix and suffix in the returned
strings irrespective of the originally entered text.

>> I'd expect it to either offer all three filenames, or just a3.
>
> It's not really clear what is correct behavior in this case.  On other
> platforms Emacs will return only a3, but HFS+ stores decomposed
> characters precisely to allow all 3 to match.  So I think we should
> at least cause Emacs return only a3, and ideally also support the
> other behavior as an option.

I'm not aware of any published rationale for the decision to
store decomposed characters. (In my testing I did notice that
zsh and bash handle globbing differently - all of the files
match a* in bash but not zsh.)

I think maybe lax matching as an option would be better than
blindly doing comparisons based on the decomposed form. With
letters with multiple diacritics, for example, the naïve
behavior would mean that one of the one-diacritic forms would
match and the other would not. If users really want that
behavior they can after all just set the file system encoding to
utf-8 instead of utf-8-hfs.

> Btw, why is completion-ignore-case nil on HFS+?  I understand it's a
> case-insensitive file system, isn't it?

No idea. (IIRC In principle it's an option that can be disabled,
though it's case-insensitive by default)

I also feel like I should ask what provisions Emacs has for
filesystem-specific case folding - NTFS and HFS both have their
own algorithms which are different from each other and may both
be different from general-purpose case matching algorithms.

>> Why exactly does completion do matching with encoded prefix
>> against raw filenames, rather than with unicode prefix against
>> decoded filenames, anyway?
>
> Performance: we don't want to decode every file name that readdir
> returns.

I'm not sure there's a way around it if we want to be 100%
correct and consistent, given the existence of parts of the
completion system that do work with the strings in Unicode.

bug#22169: 25.0.50; File name compiletion doesn't work with non-ASCII characters on OS X

[Eli Zaretskii]

> From: Random832 <random832@fastmail.com>
> Date: Wed, 16 Dec 2015 11:00:57 -0500
> 
> > It's not really clear what is correct behavior in this case.  On other
> > platforms Emacs will return only a3, but HFS+ stores decomposed
> > characters precisely to allow all 3 to match.  So I think we should
> > at least cause Emacs return only a3, and ideally also support the
> > other behavior as an option.
> 
> I'm not aware of any published rationale for the decision to
> store decomposed characters.

It cannot be anything other than the desire to support lax matches.

> I think maybe lax matching as an option would be better than
> blindly doing comparisons based on the decomposed form.

It could be, if we had the lax matching implemented in C.  But we
currently only emulate that with complex regexps, and I think it's not
a good idea to call that from dired.c.

> I'm not sure there's a way around it if we want to be 100%
> correct and consistent, given the existence of parts of the
> completion system that do work with the strings in Unicode.

I could come up with a patch if someone's interested to try it.  I
just want to hear first about the details of what happens in
file_name_completion that causes file-name-all-completions return nil
in the OP's case.  There's got to be something that I'm missing here.

bug#22169: 25.0.50; File name compiletion doesn't work with non-ASCII characters on OS X

[Random832]

Eli Zaretskii <eliz@gnu.org> writes:
>> I'm not aware of any published rationale for the decision to
>> store decomposed characters.
>
> It cannot be anything other than the desire to support lax matches.

Maybe. I half suspect it was just to make their case mapping
table (which doesn't include entries for the precomposed
characters) smaller.

>> I think maybe lax matching as an option would be better than
>> blindly doing comparisons based on the decomposed form.
>
> It could be, if we had the lax matching implemented in C.  But we
> currently only emulate that with complex regexps, and I think it's not
> a good idea to call that from dired.c.

Whether that ever gets implemented or not, what I meant to
suggest is that a half-baked lax matching that only works for a
small subset of situations and only on one platform is not a
feature worth having at all. And if people really do want it
they can have it today by setting the encoding to utf-8 and
dealing with the backspacing weirdness.

AFAICT the rationale for renormalizing filenames to NFC was that
combining characters couldn't be *displayed* on Carbon Emacs,
rather than there being anything especially undesirable about
the backspacing behavior.

> I could come up with a patch if someone's interested to try it.  I
> just want to hear first about the details of what happens in
> file_name_completion that causes file-name-all-completions return nil
> in the OP's case.  There's got to be something that I'm missing here.

Like I said, ns-win's utf-8-nfd doesn't normalize on encode.
I've since confirmed this with encode-coding-string.  I haven't
been able to confirm that ucs-normalize's utf-8-hfs exhibits the
problem behavior.

bug#22169: 25.0.50; File name compiletion doesn't work with non-ASCII characters on OS X

[Eli Zaretskii]

> From: Random832 <random832@fastmail.com>
> Date: Wed, 16 Dec 2015 13:19:20 -0500
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> >> I'm not aware of any published rationale for the decision to
> >> store decomposed characters.
> >
> > It cannot be anything other than the desire to support lax matches.
> 
> Maybe. I half suspect it was just to make their case mapping
> table (which doesn't include entries for the precomposed
> characters) smaller.

Only if they force decomposition in contexts that have nothing to do
with file names.  Otherwise, they will have to have those large case
tables anyway, for other kinds of text, right?

> AFAICT the rationale for renormalizing filenames to NFC was that
> combining characters couldn't be *displayed* on Carbon Emacs,
> rather than there being anything especially undesirable about
> the backspacing behavior.

It is generally easier and more convenient to have precomposed
characters, yes.  It's not an accident that no other filesystem does
this kind of decomposition; Windows filesystems actually compose the
characters, AFAIK.

> > I could come up with a patch if someone's interested to try it.  I
> > just want to hear first about the details of what happens in
> > file_name_completion that causes file-name-all-completions return nil
> > in the OP's case.  There's got to be something that I'm missing here.
> 
> Like I said, ns-win's utf-8-nfd doesn't normalize on encode.
> I've since confirmed this with encode-coding-string.  I haven't
> been able to confirm that ucs-normalize's utf-8-hfs exhibits the
> problem behavior.

Let's hope you are right.

bug#22169: 25.0.50; File name compiletion doesn't work with non-ASCII characters on OS X

[Anders Lindgren]

[-- Attachment #1.1: Type: text/plain, Size: 1470 bytes --]

Hi!

I think I have solved this.

The current coding system defined in ns-win.el didn't work because it only
provided a decode but no encode functions.

After revisiting the "hfs" encoder, I managed to get it to work, this time.

Below is a patch where I have dropped the old encoder and use the new
instead. The only thing noteworthy is that `ucs-normalize' is loaded by
loadup (when ns is used) and thus included in the dumped Emacs (if I
understand correctly). Unless anybody objects, I'll push it in a couple of
days.

    -- Anders

On Tue, Dec 15, 2015 at 9:05 PM, Anders Lindgren <andlind@gmail.com> wrote:

> Hi,
>
>
>> Can you write a patch to that effect, for emacs-25 branch?
>>
>
> We have the find the cause of the problem first. But once we do that, this
> should be straight forward.
>
>
> >     What does this return:
>> >
>> >     M-: (file-name-all-completion "åäö" "/that/empty/directory/") RET
>> >
>> > It returns nil.
>>
>> So this is the heart of the problem.  I assume that if you do the same
>> with an ASCII first argument, the result is non-nil, yes?
>>
>
> Yes.
>
>
>
>> Then the next step is to step with a debugger through
>> file_name_completion, and see why this returns nil instead of a list
>> of files that begin.
>>
>
> Auhm, I'll see what I can do. I'm a family father and have very, very,
> limited time, but I can see in I can find a time slot for it.
>
>     -- Anders
>
>

[-- Attachment #1.2: Type: text/html, Size: 2639 bytes --]

[-- Attachment #2: coding.diff --]
[-- Type: text/plain, Size: 3333 bytes --]

diff --git a/lisp/loadup.el b/lisp/loadup.el
index f0caa8b..dda433e 100644
--- a/lisp/loadup.el
+++ b/lisp/loadup.el
@@ -276,6 +276,7 @@
 (if (featurep 'ns)
     (progn
       (load "term/common-win")
+      (load "international/ucs-normalize")
       (load "term/ns-win")))
 (if (fboundp 'x-create-frame)
     ;; Do it after loading term/foo-win.el since the value of the
diff --git a/lisp/term/ns-win.el b/lisp/term/ns-win.el
index 0b3e3bd..9bd59fc 100644
--- a/lisp/term/ns-win.el
+++ b/lisp/term/ns-win.el
@@ -51,6 +51,7 @@
 (require 'menu-bar)
 (require 'fontset)
 (require 'dnd)
+(require 'ucs-normalize)
 
 (defgroup ns nil
   "GNUstep/Mac OS X specific features."
@@ -337,29 +338,12 @@ ns-delete-working-text
   (setq ns-working-overlay nil))
 
 
-(declare-function ns-convert-utf8-nfd-to-nfc "nsfns.m" (str))
-
-;;;; OS X file system Unicode UTF-8 NFD (decomposed form) support
-;; Lisp code based on utf-8m.el, by Seiji Zenitani, Eiji Honjoh, and
-;; Carsten Bormann.
+;; OS X file system Unicode UTF-8 NFD (decomposed form) support.
 (when (eq system-type 'darwin)
-  (defun ns-utf8-nfd-post-read-conversion (length)
-    "Calls `ns-convert-utf8-nfd-to-nfc' to compose char sequences."
-    (save-excursion
-      (save-restriction
-        (narrow-to-region (point) (+ (point) length))
-        (let ((str (buffer-string)))
-          (delete-region (point-min) (point-max))
-          (insert (ns-convert-utf8-nfd-to-nfc str))
-          (- (point-max) (point-min))))))
-
-  (define-coding-system 'utf-8-nfd
-    "UTF-8 NFD (decomposed) encoding."
-    :coding-type 'utf-8
-    :mnemonic ?U
-    :charset-list '(unicode)
-    :post-read-conversion 'ns-utf8-nfd-post-read-conversion)
-  (set-file-name-coding-system 'utf-8-nfd))
+  ;; Used prior to Emacs 25.
+  (define-coding-system-alias 'utf-8-nfd 'utf-8-hfs)
+
+  (set-file-name-coding-system 'utf-8-hfs))
 
 ;;;; Inter-app communications support.
 
diff --git a/src/nsfns.m b/src/nsfns.m
index edc02e8..5fa68c0 100644
--- a/src/nsfns.m
+++ b/src/nsfns.m
@@ -2099,39 +2099,6 @@ there was no result.  */)
 }
 
 
-DEFUN ("ns-convert-utf8-nfd-to-nfc", Fns_convert_utf8_nfd_to_nfc,
-       Sns_convert_utf8_nfd_to_nfc, 1, 1, 0,
-       doc: /* Return an NFC string that matches the UTF-8 NFD string STR.  */)
-     (Lisp_Object str)
-{
-/* TODO: If GNUstep ever implements precomposedStringWithCanonicalMapping,
-         remove this. */
-  NSString *utfStr;
-  Lisp_Object ret = Qnil;
-  NSAutoreleasePool *pool;
-
-  CHECK_STRING (str);
-  pool = [[NSAutoreleasePool alloc] init];
-  utfStr = [NSString stringWithUTF8String: SSDATA (str)];
-#ifdef NS_IMPL_COCOA
-  if (utfStr)
-    utfStr = [utfStr precomposedStringWithCanonicalMapping];
-#endif
-  if (utfStr)
-    {
-      const char *cstr = [utfStr UTF8String];
-      if (cstr)
-        ret = build_string (cstr);
-    }
-
-  [pool release];
-  if (NILP (ret))
-    error ("Invalid UTF-8");
-
-  return ret;
-}
-
-
 #ifdef NS_IMPL_COCOA
 
 /* Compile and execute the AppleScript SCRIPT and return the error
@@ -3207,7 +3174,6 @@ be used as the image of the icon representing the frame.  */);
   defsubr (&Sns_emacs_info_panel);
   defsubr (&Sns_list_services);
   defsubr (&Sns_perform_service);
-  defsubr (&Sns_convert_utf8_nfd_to_nfc);
   defsubr (&Sns_popup_font_panel);
   defsubr (&Sns_popup_color_panel);
 

bug#22169: 25.0.50; File name compiletion doesn't work with non-ASCII characters on OS X

[Random832]

Anders Lindgren <andlind@gmail.com> writes:
> Below is a patch where I have dropped the old encoder and use the new
> instead. The only thing noteworthy is that `ucs-normalize' is loaded
> by loadup (when ns is used) and thus included in the dumped Emacs (if
> I understand correctly). Unless anybody objects, I'll push it in a
> couple of days.

Out of sheer morbid curiosity, I decided to see what happens if
I create a filesystem with both NFC and NFD characters (For
thoroughness, I tested both colliding and non-colliding names,
on FAT32 and NFS.

On a FAT32 volume, Linux creates all of them fine, obviously.
OSX completely fails to do anything meaningful with the files
that are in NFD on disk: They are returned by readdir, but
cannot be opened or statted (opening one that has a name
collision with an NFC file opens the NFC file).  To my under-
standing the same behavior would be present for SMB and UDF
volumes.  The filenames are normalized to NFD when returned by
readdir, but only the filenames that are normalized to NFC on
disk are accessible.

On NFS, the story is a bit more interesting. OSX does not
perform any normalization on filenames on an NFS share.  After
being bitten by a similar bug in zsh's globbing, I was able to
determine that Emacs is able to open and save files in both
formats with utf-8-nfd (since encoding passes values through
unchanged), but _not_ with utf-8-hfs.

Arguably, for the rare users who use NFS or other filesystems
and work with characters whose representations differ, they can
simply use the utf-8 encoding and be explicit about what
filenames they want.  It is something to be aware of, though.

bug#22169: 25.0.50; File name compiletion doesn't work with non-ASCII characters on OS X

[Anders Lindgren]

[-- Attachment #1: Type: text/plain, Size: 2927 bytes --]

Hi!

I just realized that I missed parts of the ongoing discussion -- I was
under impression that I as OP should be CC:ed, but apparently I wasn't.

After reading through Random832:s comments, I also see the problem with
"åäö" and "aao" not being handled correctly. Typing "a TAB" makes Emacs
delete the "a", which seems very confusing. Typing "å TAB" or "aa TAB"
works, though. (Here `(file-name-all-completions "a" ".")' returns
`("åäöfirst.txt" "aaosecond.txt")'.

In other words, Emcas is in better shape with my than it was before, but
there is still some work to be done.

When it comes to "lax" matching -- I really don't think we should use it
for file names. I don't want to match "å" when I type "a" etc.

HFS+ file systems are case sensitive (It's possible this can be disabled,
but if so it's very rarely used). However,  many OS X desktop applications
work hard to make this invisible to users. I think that we should keep
`read-file-name-completion-ignore-case' as it is, as this corresponds to
how files really are stored.

After giving this some thought, it feels like the file name matching should
be done on decoded strings (so that an "a" doesn't match the "a" in a
decomposed "å"). However, this is a major change and needs to be discussed
further.

    -- Anders

On Thu, Dec 17, 2015 at 11:01 PM, Anders Lindgren <andlind@gmail.com> wrote:
>
> Hi!
>
> I think I have solved this.
>
> The current coding system defined in ns-win.el didn't work because it
only provided a decode but no encode functions.
>
> After revisiting the "hfs" encoder, I managed to get it to work, this
time.
>
> Below is a patch where I have dropped the old encoder and use the new
instead. The only thing noteworthy is that `ucs-normalize' is loaded by
loadup (when ns is used) and thus included in the dumped Emacs (if I
understand correctly). Unless anybody objects, I'll push it in a couple of
days.
>
>     -- Anders
>
> On Tue, Dec 15, 2015 at 9:05 PM, Anders Lindgren <andlind@gmail.com>
wrote:
>>
>> Hi,
>>
>>>
>>> Can you write a patch to that effect, for emacs-25 branch?
>>
>>
>> We have the find the cause of the problem first. But once we do that,
this should be straight forward.
>>
>>
>>> >     What does this return:
>>> >
>>> >     M-: (file-name-all-completion "åäö" "/that/empty/directory/") RET
>>> >
>>> > It returns nil.
>>>
>>> So this is the heart of the problem.  I assume that if you do the same
>>> with an ASCII first argument, the result is non-nil, yes?
>>
>>
>> Yes.
>>
>>
>>>
>>> Then the next step is to step with a debugger through
>>> file_name_completion, and see why this returns nil instead of a list
>>> of files that begin.
>>
>>
>> Auhm, I'll see what I can do. I'm a family father and have very, very,
limited time, but I can see in I can find a time slot for it.
>>
>>     -- Anders
>>
>

[-- Attachment #2: Type: text/html, Size: 3712 bytes --]

bug#22169: 25.0.50; File name compiletion doesn't work with non-ASCII characters on OS X

[Eli Zaretskii]

> Date: Fri, 18 Dec 2015 07:29:17 +0100
> From: Anders Lindgren <andlind@gmail.com>
> Cc: Eli Zaretskii <eliz@gnu.org>, random832@fastmail.com
> 
> After reading through Random832:s comments, I also see the problem with "åäö"
> and "aao" not being handled correctly. Typing "a TAB" makes Emacs delete the
> "a", which seems very confusing. Typing "å TAB" or "aa TAB" works, though.
> (Here `(file-name-all-completions "a" ".")' returns `("åäöfirst.txt"
> "aaosecond.txt")'.
> 
> In other words, Emcas is in better shape with my than it was before, but there
> is still some work to be done.
> 
> When it comes to "lax" matching -- I really don't think we should use it for
> file names. I don't want to match "å" when I type "a" etc.

I have an idea for a change that could solve this.  I will post it in
a day or two and ask you to try it.

> HFS+ file systems are case sensitive (It's possible this can be disabled, but
> if so it's very rarely used). However, many OS X desktop applications work hard
> to make this invisible to users. I think that we should keep
> `read-file-name-completion-ignore-case' as it is, as this corresponds to how
> files really are stored.

If that's what OS X users expect, fine with me.

> After giving this some thought, it feels like the file name matching should be
> done on decoded strings (so that an "a" doesn't match the "a" in a decomposed
> "å"). However, this is a major change and needs to be discussed further.

I rather think it's a non-starter, at least for Emacs 25.1.  It
probably means users of all systems will be punished by slower
directory searches, on behalf of one peculiar filesystem.  Unless
there's some clever idea that avoids decoding each file name returned
by readdir, that is.

Thanks.

bug#22169: 25.0.50; File name compiletion doesn't work with non-ASCII characters on OS X

[Eli Zaretskii]

> Date: Thu, 17 Dec 2015 23:01:01 +0100
> From: Anders Lindgren <andlind@gmail.com>
> Cc: 22169@debbugs.gnu.org
> 
> Below is a patch where I have dropped the old encoder and use the new instead.
> The only thing noteworthy is that `ucs-normalize' is loaded by loadup (when ns
> is used) and thus included in the dumped Emacs (if I understand correctly).
> Unless anybody objects, I'll push it in a couple of days.

Looks good to me, with one comment:

> diff --git a/lisp/loadup.el b/lisp/loadup.el
> index f0caa8b..dda433e 100644
> --- a/lisp/loadup.el
> +++ b/lisp/loadup.el
> @@ -276,6 +276,7 @@
>  (if (featurep 'ns)
>      (progn
>        (load "term/common-win")
> +      (load "international/ucs-normalize")
>        (load "term/ns-win")))
>  (if (fboundp 'x-create-frame)
>      ;; Do it after loading term/foo-win.el since the value of the
> diff --git a/lisp/term/ns-win.el b/lisp/term/ns-win.el
> index 0b3e3bd..9bd59fc 100644
> --- a/lisp/term/ns-win.el
> +++ b/lisp/term/ns-win.el
> @@ -51,6 +51,7 @@
>  (require 'menu-bar)
>  (require 'fontset)
>  (require 'dnd)
> +(require 'ucs-normalize)

Why do you need the 'require' if loadup will unconditionally load
ucs-normalize?

Thanks.

bug#22169: 25.0.50; File name compiletion doesn't work with non-ASCII characters on OS X

[Anders Lindgren]

[-- Attachment #1: Type: text/plain, Size: 2708 bytes --]

>
> > Below is a patch where I have dropped the old encoder and use the new
> instead.
> > The only thing noteworthy is that `ucs-normalize' is loaded by loadup
> (when ns
> > is used) and thus included in the dumped Emacs (if I understand
> correctly).
> > Unless anybody objects, I'll push it in a couple of days.
>
> Looks good to me, with one comment:
>
> > diff --git a/lisp/loadup.el b/lisp/loadup.el
> > index f0caa8b..dda433e 100644
> > --- a/lisp/loadup.el
> > +++ b/lisp/loadup.el
> > @@ -276,6 +276,7 @@
> >  (if (featurep 'ns)
> >      (progn
> >        (load "term/common-win")
> > +      (load "international/ucs-normalize")
> >        (load "term/ns-win")))
> >  (if (fboundp 'x-create-frame)
> >      ;; Do it after loading term/foo-win.el since the value of the
> > diff --git a/lisp/term/ns-win.el b/lisp/term/ns-win.el
> > index 0b3e3bd..9bd59fc 100644
> > --- a/lisp/term/ns-win.el
> > +++ b/lisp/term/ns-win.el
> > @@ -51,6 +51,7 @@
> >  (require 'menu-bar)
> >  (require 'fontset)
> >  (require 'dnd)
> > +(require 'ucs-normalize)
>
> Why do you need the 'require' if loadup will unconditionally load
> ucs-normalize?
>

I was just trying to follow the pattern in ns-win.el, there are a number of
requires at the beginning, after a comment saying ";;
Documentation-purposes only: actually loaded in loadup.el."

I can easily drop the line, if you think it's better.


> > After giving this some thought, it feels like the file name matching
should be
> > done on decoded strings (so that an "a" doesn't match the "a" in a
decomposed
> > "å"). However, this is a major change and needs to be discussed further.
>
> I rather think it's a non-starter, at least for Emacs 25.1.  It
> probably means users of all systems will be punished by slower
> directory searches, on behalf of one peculiar filesystem.  Unless
> there's some clever idea that avoids decoding each file name returned
> by readdir, that is.

The eternal question of correctness versus speed...

My gut feeling is that the time it takes to decode the file names is
dwarfed by the time it takes to read the file list from the harddisk (this
needs to be verified, of course). In addition, for systems like Linux,
encoding and decoding are no-ops (as both the source and destination is
UTF-8), so there won't be a penalty there.

I agree that this is not a project for Emacs 25.1 -- however, I think that
we should at explore this for future versions. I suggest that we push the
current patch (after dropping the `require' line), close the current issue,
and post a new bug report suggesting performing the completion on decoded
strings.

    -- Anders

[-- Attachment #2: Type: text/html, Size: 4089 bytes --]

bug#22169: 25.0.50; File name compiletion doesn't work with non-ASCII characters on OS X

[Eli Zaretskii]

> Date: Fri, 18 Dec 2015 09:38:08 +0100
> From: Anders Lindgren <andlind@gmail.com>
> Cc: 22169@debbugs.gnu.org
> 
> I was just trying to follow the pattern in ns-win.el, there are a number of
> requires at the beginning, after a comment saying ";; Documentation-purposes
> only: actually loaded in loadup.el."
> 
> I can easily drop the line, if you think it's better.

I see other files do the same, so I'm probably missing something
here.  Let's leave that as you wrote it.

> > I rather think it's a non-starter, at least for Emacs 25.1. It
> > probably means users of all systems will be punished by slower
> > directory searches, on behalf of one peculiar filesystem. Unless
> > there's some clever idea that avoids decoding each file name returned
> > by readdir, that is.
> 
> The eternal question of correctness versus speed...

No, it's correctness on one platform vs speed on all the rest.

> My gut feeling is that the time it takes to decode the file names is dwarfed by
> the time it takes to read the file list from the harddisk (this needs to be
> verified, of course).

I think you should time this.  My gut feeling is the other way around,
for several reasons:

  . reading file entries in a directory is essentially a system call,
    that is usually highly optimized code

  . modern OSes cache this stuff, so you can do that without ever
    hitting the disk

  . many modern machines have SSDs (mine does), where disk drive
    accesses, even when they are needed, are very fast

  . by contrast, decoding a non-trivial encoding might take many CPU
    cycles, especially in the utf-8-hfs case, where we call Lisp as
    part of that

Nevertheless, my gut feeling could also be false.  We should time
that.

> In addition, for systems like Linux, encoding and decoding are
> no-ops (as both the source and destination is UTF-8), so there won't
> be a penalty there.

Yes, but only in UTF-8 locales.  I won't be surprised to learn that
most of Far East uses something else, even on GNU/Linux.  And then
there are Windows volumes mounted via NFS and such likes.

> I agree that this is not a project for Emacs 25.1 -- however, I think that we
> should at explore this for future versions. I suggest that we push the current
> patch (after dropping the `require' line), close the current issue, and post a
> new bug report suggesting performing the completion on decoded strings.

I have a simpler idea for fixing the issue without decoding every file
in a directory.  Please wait for a couple of days.

(There's no need for another bug report, we could continue solving the
left-over problem as part of this one.)

bug#22169: 25.0.50; File name compiletion doesn't work with non-ASCII characters on OS X

[Random832]

Eli Zaretskii <eliz@gnu.org> writes:
> I rather think it's a non-starter, at least for Emacs 25.1.  It
> probably means users of all systems will be punished by slower
> directory searches,

How much slower do you suppose it would be? Especially for
utf-8, which I assume is fast anyway (it doesn't even seem to
reject excessively high codepoints... I'm not _entirely_ sure
utf-8 is not actually identical to emacs-internal, does anyone
know any concrete differences?)

Sometimes features, and correctness, have a performance cost. If
performance is the end-all and be-all priority, let's just
abolish all encodings and assume all filenames are in
emacs-internal.

What if it's only a 1% slow down? 5%? 10%? Or would an absolute
measure be more appropriate - i.e. define how much time it's
acceptable for it to take (on some standard directory and CPU).

> No, it's correctness on one platform vs speed on all the rest.

Strictly speaking, it's correctness for one encoding.

In trying to come up with another example, I noticed that in an
EUC-JP locale, typing "*修"TAB ("*\275\244") doesn't actually
match "文字化け" ["\312\270\273\372\262\275\244\261"] as I had
expected it to. I guess matching with embedded stars goes
through a different code path? Is there a way to simply enable
doing this for normal completion when the file system encoding
is utf-8-hfs? Or to add post-filtering [only return a filename
if it matches both the existing way *and* the decoded string
matches], only enabled by default on utf-8-hfs?

Or is even the time spent checking a boolean variable too much
of a performance penalty?

bug#22169: 25.0.50; File name compiletion doesn't work with non-ASCII characters on OS X

[Random832]

Eli Zaretskii <eliz@gnu.org> writes:
>   . modern OSes cache this stuff, so you can do that without ever
>     hitting the disk

*ever*? Surely at least once.

>   . many modern machines have SSDs (mine does), where disk drive
>     accesses, even when they are needed, are very fast

They're fast, yes, but my own gut feeling is that they're not
actually fast *enough* not to be the bottleneck.

>   . by contrast, decoding a non-trivial encoding might take many CPU
>     cycles, especially in the utf-8-hfs case, where we call Lisp as
>     part of that

I don't know why "especially in the utf-8-hfs case" - the
current code is no more correct for utf-8-hfs on Linux than for
utf-8-hfs on OSX.

> Yes, but only in UTF-8 locales.  I won't be surprised to learn that
> most of Far East uses something else, even on GNU/Linux.  And then
> there are Windows volumes mounted via NFS and such likes.

I think most people who do this (I should think it would be
SMB/CIFS rather than NFS - if it's really NFS then I suppose the
translation has to happen on the Windows side) have the file-
system translated to UTF-8 [etc] for them by the kernel.  There
are mount options "iocharset" and "codepage" (the latter for the
filesystem's coding system on 8-bit filesystems), to take care
of this.  Working with multiple different directories with
different filename encoding systems is a pathological case, and
one which as far as I know Emacs makes no attempt to deal with
(except by the user switching manually).

bug#22169: 25.0.50; File name compiletion doesn't work with non-ASCII characters on OS X

[Eli Zaretskii]

> From: Random832 <random832@fastmail.com>
> Date: Fri, 18 Dec 2015 10:26:49 -0500
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> > I rather think it's a non-starter, at least for Emacs 25.1.  It
> > probably means users of all systems will be punished by slower
> > directory searches,
> 
> How much slower do you suppose it would be?

I don't know.  That's why I suggested timing it.

> I'm not _entirely_ sure utf-8 is not actually identical to
> emacs-internal

It isn't.

> does anyone know any concrete differences?

Raw bytes and characters from Far-Eastern charsets that are not
unified get non-trivial conversions.

> Sometimes features, and correctness, have a performance cost. If
> performance is the end-all and be-all priority, let's just
> abolish all encodings and assume all filenames are in
> emacs-internal.

We need to know the cost before we make the decision.  It could be
that you are right and the cost is negligible, then the decision will
be very easy.  But it also could be not so easy.

> In trying to come up with another example, I noticed that in an
> EUC-JP locale, typing "*修"TAB ("*\275\244") doesn't actually
> match "文字化け" ["\312\270\273\372\262\275\244\261"] as I had
> expected it to. I guess matching with embedded stars goes
> through a different code path?

I'm not familiar enough with all the high-level tricks above the
dired.c primitives that were introduced lately.  I hope someone else
will answer that.  (You could try figuring that out by looking at the
calls to file_name_completion that are done by Lisp.)

> Is there a way to simply enable doing this for normal completion
> when the file system encoding is utf-8-hfs? Or to add post-filtering
> [only return a filename if it matches both the existing way *and*
> the decoded string matches], only enabled by default on utf-8-hfs?

The latter is what I had in mind, yes.

> Or is even the time spent checking a boolean variable too much
> of a performance penalty?

No, I don't think so.

bug#22169: 25.0.50; File name compiletion doesn't work with non-ASCII characters on OS X

[Eli Zaretskii]

> Date: Fri, 18 Dec 2015 09:07:39 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: random832@fastmail.com, 22169@debbugs.gnu.org
> 
> > After reading through Random832:s comments, I also see the problem with "åäö"
> > and "aao" not being handled correctly. Typing "a TAB" makes Emacs delete the
> > "a", which seems very confusing. Typing "å TAB" or "aa TAB" works, though.
> > (Here `(file-name-all-completions "a" ".")' returns `("åäöfirst.txt"
> > "aaosecond.txt")'.
> > 
> > In other words, Emcas is in better shape with my than it was before, but there
> > is still some work to be done.
> > 
> > When it comes to "lax" matching -- I really don't think we should use it for
> > file names. I don't want to match "å" when I type "a" etc.
> 
> I have an idea for a change that could solve this.  I will post it in
> a day or two and ask you to try it.

Could you please try the patch below, and see if it avoids the "lax"
matches and the confusing effect of deleting "a" in the scenario above
is avoided on OS X?

(This is not the full patch, since we need to add this code only for
some file-name encodings, such as utf-8-hfs.  If this works for you, I
will add the missing bits.  If it doesn't work, please tell where I
goofed.)

Thanks.

diff --git a/src/dired.c b/src/dired.c
index 84bf247..4ff85f1 100644
--- a/src/dired.c
+++ b/src/dired.c
@@ -641,16 +641,30 @@ file_name_completion (Lisp_Object file, Lisp_Object dirname, bool all_flag,
 
       matchcount += matchcount <= 1;
 
+      Lisp_Object zero = make_number (0);
       if (all_flag)
-	bestmatch = Fcons (name, bestmatch);
+	{
+	  Lisp_Object cmp1
+	    = Fcompare_strings (name, zero, make_number (SCHARS (name)),
+				file, zero, make_number (SCHARS (file)),
+				completion_ignore_case ? Qt : Qnil);
+	  if (EQ (cmp1, Qt) || XINT (cmp1) != -1)
+	    bestmatch = Fcons (name, bestmatch);
+	}
       else if (NILP (bestmatch))
 	{
-	  bestmatch = name;
-	  bestmatchsize = SCHARS (name);
+	  Lisp_Object cmp2
+	    = Fcompare_strings (name, zero, make_number (SCHARS (name)),
+				file, zero, make_number (SCHARS (file)),
+				completion_ignore_case ? Qt : Qnil);
+	  if (EQ (cmp2, Qt) || XINT (cmp2) != -1)
+	    {
+	      bestmatch = name;
+	      bestmatchsize = SCHARS (name);
+	    }
 	}
       else
 	{
-	  Lisp_Object zero = make_number (0);
 	  /* FIXME: This is a copy of the code in Ftry_completion.  */
 	  ptrdiff_t compare = min (bestmatchsize, SCHARS (name));
 	  Lisp_Object cmp

bug#22169: 25.0.50; File name compiletion doesn't work with non-ASCII characters on OS X

[Anders Lindgren]

[-- Attachment #1: Type: text/plain, Size: 4078 bytes --]

Hi!

Unfortunately, it still doesn't work, the "a" is still deleted. You can see
what happens here:

(file-name-all-completions "" ".")
("åäö.txt" "aao.txt" "../" "./")

(file-name-all-completions "a" ".")
("åäö.txt" "aao.txt")                     <= Incorrect result

(file-name-all-completions "å" ".")
("åäö.txt")


I gave this a bit of thinking, would the following work:

 - For each match of the current system (using encoded comparison), after
the decoding of the entry, perform a second comparison with the decoded
(original) version of "file" (when not empty).

There is no extra decoding included, as the number of entries decoded is
the same as before (even if some entries will be rejected now). The extra
comparison is only performed if "file" is not empty, so it will not affect
normal directory retrieval, only when performing a completion operation.

Concretely, in the example above, completing "a" will find both entries
which are decoded. However, the second comparison will reject "åäö.txt".

    -- Anders

On Sun, Dec 20, 2015 at 6:56 PM, Eli Zaretskii <eliz@gnu.org> wrote:

> > Date: Fri, 18 Dec 2015 09:07:39 +0200
> > From: Eli Zaretskii <eliz@gnu.org>
> > Cc: random832@fastmail.com, 22169@debbugs.gnu.org
> >
> > > After reading through Random832:s comments, I also see the problem
> with "åäö"
> > > and "aao" not being handled correctly. Typing "a TAB" makes Emacs
> delete the
> > > "a", which seems very confusing. Typing "å TAB" or "aa TAB" works,
> though.
> > > (Here `(file-name-all-completions "a" ".")' returns `("åäöfirst.txt"
> > > "aaosecond.txt")'.
> > >
> > > In other words, Emcas is in better shape with my than it was before,
> but there
> > > is still some work to be done.
> > >
> > > When it comes to "lax" matching -- I really don't think we should use
> it for
> > > file names. I don't want to match "å" when I type "a" etc.
> >
> > I have an idea for a change that could solve this.  I will post it in
> > a day or two and ask you to try it.
>
> Could you please try the patch below, and see if it avoids the "lax"
> matches and the confusing effect of deleting "a" in the scenario above
> is avoided on OS X?
>
> (This is not the full patch, since we need to add this code only for
> some file-name encodings, such as utf-8-hfs.  If this works for you, I
> will add the missing bits.  If it doesn't work, please tell where I
> goofed.)
>
> Thanks.
>
> diff --git a/src/dired.c b/src/dired.c
> index 84bf247..4ff85f1 100644
> --- a/src/dired.c
> +++ b/src/dired.c
> @@ -641,16 +641,30 @@ file_name_completion (Lisp_Object file, Lisp_Object
> dirname, bool all_flag,
>
>        matchcount += matchcount <= 1;
>
> +      Lisp_Object zero = make_number (0);
>        if (all_flag)
> -       bestmatch = Fcons (name, bestmatch);
> +       {
> +         Lisp_Object cmp1
> +           = Fcompare_strings (name, zero, make_number (SCHARS (name)),
> +                               file, zero, make_number (SCHARS (file)),
> +                               completion_ignore_case ? Qt : Qnil);
> +         if (EQ (cmp1, Qt) || XINT (cmp1) != -1)
> +           bestmatch = Fcons (name, bestmatch);
> +       }
>        else if (NILP (bestmatch))
>         {
> -         bestmatch = name;
> -         bestmatchsize = SCHARS (name);
> +         Lisp_Object cmp2
> +           = Fcompare_strings (name, zero, make_number (SCHARS (name)),
> +                               file, zero, make_number (SCHARS (file)),
> +                               completion_ignore_case ? Qt : Qnil);
> +         if (EQ (cmp2, Qt) || XINT (cmp2) != -1)
> +           {
> +             bestmatch = name;
> +             bestmatchsize = SCHARS (name);
> +           }
>         }
>        else
>         {
> -         Lisp_Object zero = make_number (0);
>           /* FIXME: This is a copy of the code in Ftry_completion.  */
>           ptrdiff_t compare = min (bestmatchsize, SCHARS (name));
>           Lisp_Object cmp
>

[-- Attachment #2: Type: text/html, Size: 5540 bytes --]

bug#22169: 25.0.50; File name compiletion doesn't work with non-ASCII characters on OS X

[Eli Zaretskii]

> Date: Sun, 20 Dec 2015 20:16:29 +0100
> From: Anders Lindgren <andlind@gmail.com>
> Cc: random832@fastmail.com, 22169@debbugs.gnu.org
> 
> Unfortunately, it still doesn't work, the "a" is still deleted. You can see
> what happens here:
> 
> (file-name-all-completions "" ".")
> ("åäö.txt" "aao.txt" "../" "./")
> 
> (file-name-all-completions "a" ".")
> ("åäö.txt" "aao.txt") <= Incorrect result

So something's wrong with the patch I wrote, because it was supposed
to reject "åäö.txt" in the last case.  Can you see why it didn't?

> I gave this a bit of thinking, would the following work:
> 
> - For each match of the current system (using encoded comparison), after the
> decoding of the entry, perform a second comparison with the decoded (original)
> version of "file" (when not empty).
> 
> There is no extra decoding included, as the number of entries decoded is the
> same as before (even if some entries will be rejected now). The extra
> comparison is only performed if "file" is not empty, so it will not affect
> normal directory retrieval, only when performing a completion operation.
> 
> Concretely, in the example above, completing "a" will find both entries which
> are decoded. However, the second comparison will reject "åäö.txt".

That's exactly what my patch was supposed to do -- it makes a second
comparison right before adding a candidate to the result.  If you can
see why it isn't working, we can take it from there.

Thanks.

bug#22169: 25.0.50; File name compiletion doesn't work with non-ASCII characters on OS X

[Anders Lindgren]

[-- Attachment #1.1: Type: text/plain, Size: 1855 bytes --]

Hi!

I managed to get the attached patch to work (when used in conjunction with
my previous patch).

I've tested:

* C-x C-f a TAB
* (find-file-all-competions "a" ".")

    -- Anders



On Sun, Dec 20, 2015 at 8:39 PM, Eli Zaretskii <eliz@gnu.org> wrote:

> > Date: Sun, 20 Dec 2015 20:16:29 +0100
> > From: Anders Lindgren <andlind@gmail.com>
> > Cc: random832@fastmail.com, 22169@debbugs.gnu.org
> >
> > Unfortunately, it still doesn't work, the "a" is still deleted. You can
> see
> > what happens here:
> >
> > (file-name-all-completions "" ".")
> > ("åäö.txt" "aao.txt" "../" "./")
> >
> > (file-name-all-completions "a" ".")
> > ("åäö.txt" "aao.txt") <= Incorrect result
>
> So something's wrong with the patch I wrote, because it was supposed
> to reject "åäö.txt" in the last case.  Can you see why it didn't?
>
> > I gave this a bit of thinking, would the following work:
> >
> > - For each match of the current system (using encoded comparison), after
> the
> > decoding of the entry, perform a second comparison with the decoded
> (original)
> > version of "file" (when not empty).
> >
> > There is no extra decoding included, as the number of entries decoded is
> the
> > same as before (even if some entries will be rejected now). The extra
> > comparison is only performed if "file" is not empty, so it will not
> affect
> > normal directory retrieval, only when performing a completion operation.
> >
> > Concretely, in the example above, completing "a" will find both entries
> which
> > are decoded. However, the second comparison will reject "åäö.txt".
>
> That's exactly what my patch was supposed to do -- it makes a second
> comparison right before adding a candidate to the result.  If you can
> see why it isn't working, we can take it from there.
>
> Thanks.
>

[-- Attachment #1.2: Type: text/html, Size: 2775 bytes --]

[-- Attachment #2: coding2.diff --]
[-- Type: text/plain, Size: 1279 bytes --]

diff --git a/src/dired.c b/src/dired.c
index 84bf247..a2f388c 100644
--- a/src/dired.c
+++ b/src/dired.c
@@ -637,6 +637,20 @@ file_name_completion (Lisp_Object file, Lisp_Object dirname, bool all_flag,
       if (!NILP (predicate) && NILP (call1 (predicate, name)))
 	continue;
 
+      /* Reject entries where the encoded strings matched but the
+         decoded doesn't.  Concretely, "a" should not match "a-ring"
+         on file system encoded using UTF-8 decomposed characters. */
+      Lisp_Object zero = make_number (0);
+      if (SCHARS (file) <= SCHARS (name))
+        {
+          Lisp_Object cmp
+            = Fcompare_strings (name, zero, make_number (SCHARS (file)),
+                                file, zero, make_number (SCHARS (file)),
+                                completion_ignore_case ? Qt : Qnil);
+          if (!EQ (cmp, Qt))
+            continue;
+        }
+
       /* Suitably record this match.  */
 
       matchcount += matchcount <= 1;
@@ -650,7 +664,6 @@ file_name_completion (Lisp_Object file, Lisp_Object dirname, bool all_flag,
 	}
       else
 	{
-	  Lisp_Object zero = make_number (0);
 	  /* FIXME: This is a copy of the code in Ftry_completion.  */
 	  ptrdiff_t compare = min (bestmatchsize, SCHARS (name));
 	  Lisp_Object cmp

bug#22169: 25.0.50; File name compiletion doesn't work with non-ASCII characters on OS X

[Eli Zaretskii]

> Date: Sun, 20 Dec 2015 23:00:40 +0100
> From: Anders Lindgren <andlind@gmail.com>
> Cc: random832@fastmail.com, 22169@debbugs.gnu.org
> 
> I managed to get the attached patch to work (when used in conjunction with my
> previous patch).
> 
> I've tested:
> 
> * C-x C-f a TAB
> * (find-file-all-competions "a" ".")

OK, thanks.

The next step is arrange for this to happen only with those values of
file-name-coding-system that require it.  My idea is to put a special
property on the coding-system's symbol, and check that inside
file_name_completion (outside of the loop).  Can you add this, or do
you want me to suggest a patch along these lines for you to test?
Note that the property check will have to be done on
file-name-coding-system if it is non-nil, otherwise on
default-file-name-coding-system (if that is non-nil).

Thanks.

bug#22169: 25.0.50; File name compiletion doesn't work with non-ASCII characters on OS X

[Anders Lindgren]

[-- Attachment #1: Type: text/plain, Size: 1491 bytes --]

Hi!

I did some simple measurements with and without this patch. I ran
`(file-name-all-completions "x" "src")' on the Emacs src directory. The
timing values were almost identical (varying between 0.001012 and 0.001080).

The way I see it, the patch doesn't do any harm in any coding system, and
it is fast. Hence, I don't really see that it's worth the effort to make
this code conditional.

However, please write a patch for this if you still thinks it's necessary.
I can test it here to make sure it works under OS X.

    -- Anders


On Mon, Dec 21, 2015 at 4:39 AM, Eli Zaretskii <eliz@gnu.org> wrote:

> > Date: Sun, 20 Dec 2015 23:00:40 +0100
> > From: Anders Lindgren <andlind@gmail.com>
> > Cc: random832@fastmail.com, 22169@debbugs.gnu.org
> >
> > I managed to get the attached patch to work (when used in conjunction
> with my
> > previous patch).
> >
> > I've tested:
> >
> > * C-x C-f a TAB
> > * (find-file-all-competions "a" ".")
>
> OK, thanks.
>
> The next step is arrange for this to happen only with those values of
> file-name-coding-system that require it.  My idea is to put a special
> property on the coding-system's symbol, and check that inside
> file_name_completion (outside of the loop).  Can you add this, or do
> you want me to suggest a patch along these lines for you to test?
> Note that the property check will have to be done on
> file-name-coding-system if it is non-nil, otherwise on
> default-file-name-coding-system (if that is non-nil).
>
> Thanks.
>

[-- Attachment #2: Type: text/html, Size: 2240 bytes --]

bug#22169: 25.0.50; File name compiletion doesn't work with non-ASCII characters on OS X

[Eli Zaretskii]

> Date: Mon, 21 Dec 2015 07:52:53 +0100
> From: Anders Lindgren <andlind@gmail.com>
> Cc: random832@fastmail.com, 22169@debbugs.gnu.org
> 
> I did some simple measurements with and without this patch. I ran
> `(file-name-all-completions "x" "src")' on the Emacs src directory. The timing
> values were almost identical (varying between 0.001012 and 0.001080).

You should try it on a larger directory, preferably one that has many
files with non-ASCII file names.

> The way I see it, the patch doesn't do any harm in any coding system, and it is
> fast. Hence, I don't really see that it's worth the effort to make this code
> conditional.

I'm surprised to hear that.  Did you look at the implementation of
Fcompare_strings?  It's highly non-trivial.  What's more, if the user
sets completion-ignore-case non-nil, Fcompare_strings will call
Fupcase on each character, which is another non-trivial function; if
you are particularly unlucky, Fupcase can even GC (if it needs to set
up the case-table), which will definitely take several hundreds of
milliseconds if not longer.

And that's today; what if tomorrow someone comes and adds to
Fcompare_strings something that makes it even more complex and slow?

I've learned long ago not to call any non-trivial API unless I really
need it.  You can never know what complexity hides in there.  Besides,
it simply looks bad in the code to do processing that is unnecessary.

> However, please write a patch for this if you still thinks it's necessary. I
> can test it here to make sure it works under OS X.

Attached (relative to the current emacs-25 branch).

Please note that the patch below attempts to solve a couple of
additional subtle aspects of this:

  . it doesn't force the extra comparison for unibyte strings (which
    include ASCII strings and unibyte non-ASCII strings), since the
    issue doesn't exist then, and ENCODE_FILE/DECODE_FILE are no-ops

  . it forces the FILE argument to have all of its characters
    precomposed, since if the caller passes us a file name with
    decomposed characters, we risk rejecting them in the code we are
    adding

Please see that these indeed o their job correctly, as I could only
test the code very superficially.

Thanks.

diff --git a/lisp/international/ucs-normalize.el b/lisp/international/ucs-normalize.el
index 8839b00..6f2fb28 100644
--- a/lisp/international/ucs-normalize.el
+++ b/lisp/international/ucs-normalize.el
@@ -627,6 +627,10 @@ 'utf-8-hfs
   :pre-write-conversion 'ucs-normalize-hfs-nfd-pre-write-conversion
   )
 
+;; This is tested in dired.c:file_name_completion in order to reject
+;; false positives due to comparison of encoded file names.
+(coding-system-put 'utf-8-hfs 'decomposed-characters 't)
+
 (provide 'ucs-normalize)
 
 ;; Local Variables:
diff --git a/src/dired.c b/src/dired.c
index 84bf247..d5628d5 100644
--- a/src/dired.c
+++ b/src/dired.c
@@ -467,6 +467,7 @@ file_name_completion (Lisp_Object file, Lisp_Object dirname, bool all_flag,
      well as "." and "..".  Until shown otherwise, assume we can't exclude
      anything.  */
   bool includeall = 1;
+  bool check_decoded = false;
   ptrdiff_t count = SPECPDL_INDEX ();
 
   elt = Qnil;
@@ -485,6 +486,28 @@ file_name_completion (Lisp_Object file, Lisp_Object dirname, bool all_flag,
      on the encoded file name.  */
   encoded_file = ENCODE_FILE (file);
   encoded_dir = ENCODE_FILE (Fdirectory_file_name (dirname));
+  if (STRING_MULTIBYTE (file))
+    {
+      Lisp_Object file_encoding = Vfile_name_coding_system;
+
+      if (NILP (Vfile_name_coding_system))
+	file_encoding = Vdefault_file_name_coding_system;
+      /* If the file-name encoding decomposes characters, as we do for
+	 HFS+ filesystems, we need to make an additional comparison of
+	 decoded names in order to filter false positives, such as "a"
+	 falsely matching "a-ring".  */
+      if (!NILP (file_encoding)
+	  && !NILP (Fplist_get (Fcoding_system_plist (file_encoding),
+				Qdecomposed_characters)))
+	{
+	  check_decoded = true;
+	  /* Recompute FILE to make sure any decomposed characters in
+	     it are re-composed by the post-read-conversion.
+	     Otherwise, any decomposed characters will be rejected by
+	     the additional check below.  */
+	  file = DECODE_FILE (encoded_file);
+	}
+    }
   int fd;
   DIR *d = open_directory (encoded_dir, &fd);
   record_unwind_protect_ptr (directory_files_internal_unwind, d);
@@ -637,6 +660,21 @@ file_name_completion (Lisp_Object file, Lisp_Object dirname, bool all_flag,
       if (!NILP (predicate) && NILP (call1 (predicate, name)))
 	continue;
 
+      /* Reject entries where the encoded strings match, but the
+         decoded don't.  For example, "a" should not match "a-ring" on
+         file systems that store decomposed characters. */
+      Lisp_Object zero = make_number (0);
+      Lisp_Object compare;
+      Lisp_Object cmp;
+      if (check_decoded && SCHARS (file) <= SCHARS (name))
+	{
+	  compare = make_number (SCHARS (file));
+ 	  cmp = Fcompare_strings (name, zero, compare, file, zero, compare,
+				  completion_ignore_case ? Qt : Qnil);
+	  if (!EQ (cmp, Qt))
+	    continue;
+	}
+
       /* Suitably record this match.  */
 
       matchcount += matchcount <= 1;
@@ -650,15 +688,13 @@ file_name_completion (Lisp_Object file, Lisp_Object dirname, bool all_flag,
 	}
       else
 	{
-	  Lisp_Object zero = make_number (0);
 	  /* FIXME: This is a copy of the code in Ftry_completion.  */
-	  ptrdiff_t compare = min (bestmatchsize, SCHARS (name));
-	  Lisp_Object cmp
-	    = Fcompare_strings (bestmatch, zero,
-				make_number (compare),
-				name, zero,
-				make_number (compare),
-				completion_ignore_case ? Qt : Qnil);
+	  compare = min (bestmatchsize, SCHARS (name));
+	  cmp = Fcompare_strings (bestmatch, zero,
+				  make_number (compare),
+				  name, zero,
+				  make_number (compare),
+				  completion_ignore_case ? Qt : Qnil);
 	  ptrdiff_t matchsize = EQ (cmp, Qt) ? compare : eabs (XINT (cmp)) - 1;
 
 	  if (completion_ignore_case)
@@ -1007,6 +1043,7 @@ syms_of_dired (void)
   DEFSYM (Qfile_attributes, "file-attributes");
   DEFSYM (Qfile_attributes_lessp, "file-attributes-lessp");
   DEFSYM (Qdefault_directory, "default-directory");
+  DEFSYM (Qdecomposed_characters, "decomposed-characters");
 
   defsubr (&Sdirectory_files);
   defsubr (&Sdirectory_files_and_attributes);

bug#22169: 25.0.50; File name compiletion doesn't work with non-ASCII characters on OS X

[Anders Lindgren]

[-- Attachment #1: Type: text/plain, Size: 7933 bytes --]

Hi!

I just tested your latest patch. Unfortunately, it doesn't work properly.

When pressing TAB, it expands the characters correctly. However,
`file-name-all-completions' doesn't work:

(file-name-all-completions "a" ".")
("åäöfirst.txt" "aaosecond.txt")

I haven't had time to see what actually happens in the code, though.
However, the "  if (STRING_MULTIBYTE (file))" looks suspicious as the
decoded value needs to be checked even for strings like "a". (However, I
don't really know what STRING_MULTIBYTE does.)

    -- Anders



On Mon, Dec 21, 2015 at 5:09 PM, Eli Zaretskii <eliz@gnu.org> wrote:

> > Date: Mon, 21 Dec 2015 07:52:53 +0100
> > From: Anders Lindgren <andlind@gmail.com>
> > Cc: random832@fastmail.com, 22169@debbugs.gnu.org
> >
> > I did some simple measurements with and without this patch. I ran
> > `(file-name-all-completions "x" "src")' on the Emacs src directory. The
> timing
> > values were almost identical (varying between 0.001012 and 0.001080).
>
> You should try it on a larger directory, preferably one that has many
> files with non-ASCII file names.
>
> > The way I see it, the patch doesn't do any harm in any coding system,
> and it is
> > fast. Hence, I don't really see that it's worth the effort to make this
> code
> > conditional.
>
> I'm surprised to hear that.  Did you look at the implementation of
> Fcompare_strings?  It's highly non-trivial.  What's more, if the user
> sets completion-ignore-case non-nil, Fcompare_strings will call
> Fupcase on each character, which is another non-trivial function; if
> you are particularly unlucky, Fupcase can even GC (if it needs to set
> up the case-table), which will definitely take several hundreds of
> milliseconds if not longer.
>
> And that's today; what if tomorrow someone comes and adds to
> Fcompare_strings something that makes it even more complex and slow?
>
> I've learned long ago not to call any non-trivial API unless I really
> need it.  You can never know what complexity hides in there.  Besides,
> it simply looks bad in the code to do processing that is unnecessary.
>
> > However, please write a patch for this if you still thinks it's
> necessary. I
> > can test it here to make sure it works under OS X.
>
> Attached (relative to the current emacs-25 branch).
>
> Please note that the patch below attempts to solve a couple of
> additional subtle aspects of this:
>
>   . it doesn't force the extra comparison for unibyte strings (which
>     include ASCII strings and unibyte non-ASCII strings), since the
>     issue doesn't exist then, and ENCODE_FILE/DECODE_FILE are no-ops
>
>   . it forces the FILE argument to have all of its characters
>     precomposed, since if the caller passes us a file name with
>     decomposed characters, we risk rejecting them in the code we are
>     adding
>
> Please see that these indeed o their job correctly, as I could only
> test the code very superficially.
>
> Thanks.
>
> diff --git a/lisp/international/ucs-normalize.el
> b/lisp/international/ucs-normalize.el
> index 8839b00..6f2fb28 100644
> --- a/lisp/international/ucs-normalize.el
> +++ b/lisp/international/ucs-normalize.el
> @@ -627,6 +627,10 @@ 'utf-8-hfs
>    :pre-write-conversion 'ucs-normalize-hfs-nfd-pre-write-conversion
>    )
>
> +;; This is tested in dired.c:file_name_completion in order to reject
> +;; false positives due to comparison of encoded file names.
> +(coding-system-put 'utf-8-hfs 'decomposed-characters 't)
> +
>  (provide 'ucs-normalize)
>
>  ;; Local Variables:
> diff --git a/src/dired.c b/src/dired.c
> index 84bf247..d5628d5 100644
> --- a/src/dired.c
> +++ b/src/dired.c
> @@ -467,6 +467,7 @@ file_name_completion (Lisp_Object file, Lisp_Object
> dirname, bool all_flag,
>       well as "." and "..".  Until shown otherwise, assume we can't exclude
>       anything.  */
>    bool includeall = 1;
> +  bool check_decoded = false;
>    ptrdiff_t count = SPECPDL_INDEX ();
>
>    elt = Qnil;
> @@ -485,6 +486,28 @@ file_name_completion (Lisp_Object file, Lisp_Object
> dirname, bool all_flag,
>       on the encoded file name.  */
>    encoded_file = ENCODE_FILE (file);
>    encoded_dir = ENCODE_FILE (Fdirectory_file_name (dirname));
> +  if (STRING_MULTIBYTE (file))
> +    {
> +      Lisp_Object file_encoding = Vfile_name_coding_system;
> +
> +      if (NILP (Vfile_name_coding_system))
> +       file_encoding = Vdefault_file_name_coding_system;
> +      /* If the file-name encoding decomposes characters, as we do for
> +        HFS+ filesystems, we need to make an additional comparison of
> +        decoded names in order to filter false positives, such as "a"
> +        falsely matching "a-ring".  */
> +      if (!NILP (file_encoding)
> +         && !NILP (Fplist_get (Fcoding_system_plist (file_encoding),
> +                               Qdecomposed_characters)))
> +       {
> +         check_decoded = true;
> +         /* Recompute FILE to make sure any decomposed characters in
> +            it are re-composed by the post-read-conversion.
> +            Otherwise, any decomposed characters will be rejected by
> +            the additional check below.  */
> +         file = DECODE_FILE (encoded_file);
> +       }
> +    }
>    int fd;
>    DIR *d = open_directory (encoded_dir, &fd);
>    record_unwind_protect_ptr (directory_files_internal_unwind, d);
> @@ -637,6 +660,21 @@ file_name_completion (Lisp_Object file, Lisp_Object
> dirname, bool all_flag,
>        if (!NILP (predicate) && NILP (call1 (predicate, name)))
>         continue;
>
> +      /* Reject entries where the encoded strings match, but the
> +         decoded don't.  For example, "a" should not match "a-ring" on
> +         file systems that store decomposed characters. */
> +      Lisp_Object zero = make_number (0);
> +      Lisp_Object compare;
> +      Lisp_Object cmp;
> +      if (check_decoded && SCHARS (file) <= SCHARS (name))
> +       {
> +         compare = make_number (SCHARS (file));
> +         cmp = Fcompare_strings (name, zero, compare, file, zero, compare,
> +                                 completion_ignore_case ? Qt : Qnil);
> +         if (!EQ (cmp, Qt))
> +           continue;
> +       }
> +
>        /* Suitably record this match.  */
>
>        matchcount += matchcount <= 1;
> @@ -650,15 +688,13 @@ file_name_completion (Lisp_Object file, Lisp_Object
> dirname, bool all_flag,
>         }
>        else
>         {
> -         Lisp_Object zero = make_number (0);
>           /* FIXME: This is a copy of the code in Ftry_completion.  */
> -         ptrdiff_t compare = min (bestmatchsize, SCHARS (name));
> -         Lisp_Object cmp
> -           = Fcompare_strings (bestmatch, zero,
> -                               make_number (compare),
> -                               name, zero,
> -                               make_number (compare),
> -                               completion_ignore_case ? Qt : Qnil);
> +         compare = min (bestmatchsize, SCHARS (name));
> +         cmp = Fcompare_strings (bestmatch, zero,
> +                                 make_number (compare),
> +                                 name, zero,
> +                                 make_number (compare),
> +                                 completion_ignore_case ? Qt : Qnil);
>           ptrdiff_t matchsize = EQ (cmp, Qt) ? compare : eabs (XINT (cmp))
> - 1;
>
>           if (completion_ignore_case)
> @@ -1007,6 +1043,7 @@ syms_of_dired (void)
>    DEFSYM (Qfile_attributes, "file-attributes");
>    DEFSYM (Qfile_attributes_lessp, "file-attributes-lessp");
>    DEFSYM (Qdefault_directory, "default-directory");
> +  DEFSYM (Qdecomposed_characters, "decomposed-characters");
>
>    defsubr (&Sdirectory_files);
>    defsubr (&Sdirectory_files_and_attributes);
>

[-- Attachment #2: Type: text/html, Size: 9720 bytes --]

bug#22169: 25.0.50; File name compiletion doesn't work with non-ASCII characters on OS X

[Eli Zaretskii]

> Date: Mon, 21 Dec 2015 23:03:38 +0100
> From: Anders Lindgren <andlind@gmail.com>
> Cc: random832@fastmail.com, 22169@debbugs.gnu.org
> 
> I haven't had time to see what actually happens in the code, though. However,
> the " if (STRING_MULTIBYTE (file))" looks suspicious as the decoded value needs
> to be checked even for strings like "a". (However, I don't really know what
> STRING_MULTIBYTE does.)

Does removing the STRING_MULTIBYTE test make it work?

bug#22169: 25.0.50; File name compiletion doesn't work with non-ASCII characters on OS X

[Anders Lindgren]

[-- Attachment #1: Type: text/plain, Size: 428 bytes --]

Hi!

> I haven't had time to see what actually happens in the code, though.
> However,
> > the " if (STRING_MULTIBYTE (file))" looks suspicious as the decoded
> value needs
> > to be checked even for strings like "a". (However, I don't really know
> what
> > STRING_MULTIBYTE does.)
>
> Does removing the STRING_MULTIBYTE test make it work?
>

Yes, it does:

(file-name-all-completions "a" ".")
("aaosecond.txt")

    -- Anders

[-- Attachment #2: Type: text/html, Size: 921 bytes --]

bug#22169: 25.0.50; File name compiletion doesn't work with non-ASCII characters on OS X

[Eli Zaretskii]

> Date: Tue, 22 Dec 2015 06:42:47 +0100
> From: Anders Lindgren <andlind@gmail.com>
> Cc: random832@fastmail.com, 22169@debbugs.gnu.org
> 
>     > I haven't had time to see what actually happens in the code, though.
>     However,
>     > the " if (STRING_MULTIBYTE (file))" looks suspicious as the decoded value
>     needs
>     > to be checked even for strings like "a". (However, I don't really know
>     what
>     > STRING_MULTIBYTE does.)
>     
>     Does removing the STRING_MULTIBYTE test make it work?
> 
> Yes, it does:
> 
> (file-name-all-completions "a" ".")
> ("aaosecond.txt")

Then please try the final patch below (again against the current
emacs-25 branch), I hope I didn't goof this time.

Thanks.

diff --git a/lisp/international/ucs-normalize.el b/lisp/international/ucs-normalize.el
index 8839b00..6f2fb28 100644
--- a/lisp/international/ucs-normalize.el
+++ b/lisp/international/ucs-normalize.el
@@ -627,6 +627,10 @@ 'utf-8-hfs
   :pre-write-conversion 'ucs-normalize-hfs-nfd-pre-write-conversion
   )
 
+;; This is tested in dired.c:file_name_completion in order to reject
+;; false positives due to comparison of encoded file names.
+(coding-system-put 'utf-8-hfs 'decomposed-characters 't)
+
 (provide 'ucs-normalize)
 
 ;; Local Variables:
diff --git a/src/dired.c b/src/dired.c
index 84bf247..89bd908 100644
--- a/src/dired.c
+++ b/src/dired.c
@@ -467,6 +467,7 @@ file_name_completion (Lisp_Object file, Lisp_Object dirname, bool all_flag,
      well as "." and "..".  Until shown otherwise, assume we can't exclude
      anything.  */
   bool includeall = 1;
+  bool check_decoded = false;
   ptrdiff_t count = SPECPDL_INDEX ();
 
   elt = Qnil;
@@ -485,6 +486,28 @@ file_name_completion (Lisp_Object file, Lisp_Object dirname, bool all_flag,
      on the encoded file name.  */
   encoded_file = ENCODE_FILE (file);
   encoded_dir = ENCODE_FILE (Fdirectory_file_name (dirname));
+
+  Lisp_Object file_encoding = Vfile_name_coding_system;
+  if (NILP (Vfile_name_coding_system))
+    file_encoding = Vdefault_file_name_coding_system;
+  /* If the file-name encoding decomposes characters, as we do for
+     HFS+ filesystems, we need to make an additional comparison of
+     decoded names in order to filter false positives, such as "a"
+     falsely matching "a-ring".  */
+  if (!NILP (file_encoding)
+      && !NILP (Fplist_get (Fcoding_system_plist (file_encoding),
+			    Qdecomposed_characters)))
+    {
+      check_decoded = true;
+      if (STRING_MULTIBYTE (file))
+	{
+	  /* Recompute FILE to make sure any decomposed characters in
+	     it are re-composed by the post-read-conversion.
+	     Otherwise, any decomposed characters will be rejected by
+	     the additional check below.  */
+	  file = DECODE_FILE (encoded_file);
+	}
+    }
   int fd;
   DIR *d = open_directory (encoded_dir, &fd);
   record_unwind_protect_ptr (directory_files_internal_unwind, d);
@@ -637,6 +660,21 @@ file_name_completion (Lisp_Object file, Lisp_Object dirname, bool all_flag,
       if (!NILP (predicate) && NILP (call1 (predicate, name)))
 	continue;
 
+      /* Reject entries where the encoded strings match, but the
+         decoded don't.  For example, "a" should not match "a-ring" on
+         file systems that store decomposed characters. */
+      Lisp_Object zero = make_number (0);
+      Lisp_Object compare;
+      Lisp_Object cmp;
+      if (check_decoded && SCHARS (file) <= SCHARS (name))
+	{
+	  compare = make_number (SCHARS (file));
+	  cmp = Fcompare_strings (name, zero, compare, file, zero, compare,
+				  completion_ignore_case ? Qt : Qnil);
+	  if (!EQ (cmp, Qt))
+	    continue;
+	}
+
       /* Suitably record this match.  */
 
       matchcount += matchcount <= 1;
@@ -650,15 +688,13 @@ file_name_completion (Lisp_Object file, Lisp_Object dirname, bool all_flag,
 	}
       else
 	{
-	  Lisp_Object zero = make_number (0);
 	  /* FIXME: This is a copy of the code in Ftry_completion.  */
-	  ptrdiff_t compare = min (bestmatchsize, SCHARS (name));
-	  Lisp_Object cmp
-	    = Fcompare_strings (bestmatch, zero,
-				make_number (compare),
-				name, zero,
-				make_number (compare),
-				completion_ignore_case ? Qt : Qnil);
+	  compare = min (bestmatchsize, SCHARS (name));
+	  cmp = Fcompare_strings (bestmatch, zero,
+				  make_number (compare),
+				  name, zero,
+				  make_number (compare),
+				  completion_ignore_case ? Qt : Qnil);
 	  ptrdiff_t matchsize = EQ (cmp, Qt) ? compare : eabs (XINT (cmp)) - 1;
 
 	  if (completion_ignore_case)
@@ -1007,6 +1043,7 @@ syms_of_dired (void)
   DEFSYM (Qfile_attributes, "file-attributes");
   DEFSYM (Qfile_attributes_lessp, "file-attributes-lessp");
   DEFSYM (Qdefault_directory, "default-directory");
+  DEFSYM (Qdecomposed_characters, "decomposed-characters");
 
   defsubr (&Sdirectory_files);
   defsubr (&Sdirectory_files_and_attributes);

bug#22169: 25.0.50; File name compiletion doesn't work with non-ASCII characters on OS X

[Anders Lindgren]

[-- Attachment #1: Type: text/plain, Size: 5787 bytes --]

Hi!

I just tried this and I can confirm that it works.

Thanks for your hard work!

I will push my patch for using the utf-8-hfs coding system (most likely)
tomorrow.

    -- Anders


On Tue, Dec 22, 2015 at 6:10 PM, Eli Zaretskii <eliz@gnu.org> wrote:

> > Date: Tue, 22 Dec 2015 06:42:47 +0100
> > From: Anders Lindgren <andlind@gmail.com>
> > Cc: random832@fastmail.com, 22169@debbugs.gnu.org
> >
> >     > I haven't had time to see what actually happens in the code,
> though.
> >     However,
> >     > the " if (STRING_MULTIBYTE (file))" looks suspicious as the
> decoded value
> >     needs
> >     > to be checked even for strings like "a". (However, I don't really
> know
> >     what
> >     > STRING_MULTIBYTE does.)
> >
> >     Does removing the STRING_MULTIBYTE test make it work?
> >
> > Yes, it does:
> >
> > (file-name-all-completions "a" ".")
> > ("aaosecond.txt")
>
> Then please try the final patch below (again against the current
> emacs-25 branch), I hope I didn't goof this time.
>
> Thanks.
>
> diff --git a/lisp/international/ucs-normalize.el
> b/lisp/international/ucs-normalize.el
> index 8839b00..6f2fb28 100644
> --- a/lisp/international/ucs-normalize.el
> +++ b/lisp/international/ucs-normalize.el
> @@ -627,6 +627,10 @@ 'utf-8-hfs
>    :pre-write-conversion 'ucs-normalize-hfs-nfd-pre-write-conversion
>    )
>
> +;; This is tested in dired.c:file_name_completion in order to reject
> +;; false positives due to comparison of encoded file names.
> +(coding-system-put 'utf-8-hfs 'decomposed-characters 't)
> +
>  (provide 'ucs-normalize)
>
>  ;; Local Variables:
> diff --git a/src/dired.c b/src/dired.c
> index 84bf247..89bd908 100644
> --- a/src/dired.c
> +++ b/src/dired.c
> @@ -467,6 +467,7 @@ file_name_completion (Lisp_Object file, Lisp_Object
> dirname, bool all_flag,
>       well as "." and "..".  Until shown otherwise, assume we can't exclude
>       anything.  */
>    bool includeall = 1;
> +  bool check_decoded = false;
>    ptrdiff_t count = SPECPDL_INDEX ();
>
>    elt = Qnil;
> @@ -485,6 +486,28 @@ file_name_completion (Lisp_Object file, Lisp_Object
> dirname, bool all_flag,
>       on the encoded file name.  */
>    encoded_file = ENCODE_FILE (file);
>    encoded_dir = ENCODE_FILE (Fdirectory_file_name (dirname));
> +
> +  Lisp_Object file_encoding = Vfile_name_coding_system;
> +  if (NILP (Vfile_name_coding_system))
> +    file_encoding = Vdefault_file_name_coding_system;
> +  /* If the file-name encoding decomposes characters, as we do for
> +     HFS+ filesystems, we need to make an additional comparison of
> +     decoded names in order to filter false positives, such as "a"
> +     falsely matching "a-ring".  */
> +  if (!NILP (file_encoding)
> +      && !NILP (Fplist_get (Fcoding_system_plist (file_encoding),
> +                           Qdecomposed_characters)))
> +    {
> +      check_decoded = true;
> +      if (STRING_MULTIBYTE (file))
> +       {
> +         /* Recompute FILE to make sure any decomposed characters in
> +            it are re-composed by the post-read-conversion.
> +            Otherwise, any decomposed characters will be rejected by
> +            the additional check below.  */
> +         file = DECODE_FILE (encoded_file);
> +       }
> +    }
>    int fd;
>    DIR *d = open_directory (encoded_dir, &fd);
>    record_unwind_protect_ptr (directory_files_internal_unwind, d);
> @@ -637,6 +660,21 @@ file_name_completion (Lisp_Object file, Lisp_Object
> dirname, bool all_flag,
>        if (!NILP (predicate) && NILP (call1 (predicate, name)))
>         continue;
>
> +      /* Reject entries where the encoded strings match, but the
> +         decoded don't.  For example, "a" should not match "a-ring" on
> +         file systems that store decomposed characters. */
> +      Lisp_Object zero = make_number (0);
> +      Lisp_Object compare;
> +      Lisp_Object cmp;
> +      if (check_decoded && SCHARS (file) <= SCHARS (name))
> +       {
> +         compare = make_number (SCHARS (file));
> +         cmp = Fcompare_strings (name, zero, compare, file, zero, compare,
> +                                 completion_ignore_case ? Qt : Qnil);
> +         if (!EQ (cmp, Qt))
> +           continue;
> +       }
> +
>        /* Suitably record this match.  */
>
>        matchcount += matchcount <= 1;
> @@ -650,15 +688,13 @@ file_name_completion (Lisp_Object file, Lisp_Object
> dirname, bool all_flag,
>         }
>        else
>         {
> -         Lisp_Object zero = make_number (0);
>           /* FIXME: This is a copy of the code in Ftry_completion.  */
> -         ptrdiff_t compare = min (bestmatchsize, SCHARS (name));
> -         Lisp_Object cmp
> -           = Fcompare_strings (bestmatch, zero,
> -                               make_number (compare),
> -                               name, zero,
> -                               make_number (compare),
> -                               completion_ignore_case ? Qt : Qnil);
> +         compare = min (bestmatchsize, SCHARS (name));
> +         cmp = Fcompare_strings (bestmatch, zero,
> +                                 make_number (compare),
> +                                 name, zero,
> +                                 make_number (compare),
> +                                 completion_ignore_case ? Qt : Qnil);
>           ptrdiff_t matchsize = EQ (cmp, Qt) ? compare : eabs (XINT (cmp))
> - 1;
>
>           if (completion_ignore_case)
> @@ -1007,6 +1043,7 @@ syms_of_dired (void)
>    DEFSYM (Qfile_attributes, "file-attributes");
>    DEFSYM (Qfile_attributes_lessp, "file-attributes-lessp");
>    DEFSYM (Qdefault_directory, "default-directory");
> +  DEFSYM (Qdecomposed_characters, "decomposed-characters");
>
>    defsubr (&Sdirectory_files);
>    defsubr (&Sdirectory_files_and_attributes);
>

[-- Attachment #2: Type: text/html, Size: 7616 bytes --]

bug#22169: 25.0.50; File name compiletion doesn't work with non-ASCII characters on OS X

[Eli Zaretskii]

> Date: Tue, 22 Dec 2015 23:29:16 +0100
> From: Anders Lindgren <andlind@gmail.com>
> Cc: random832@fastmail.com, 22169@debbugs.gnu.org
> 
> I just tried this and I can confirm that it works.

Thanks for testing.

> I will push my patch for using the utf-8-hfs coding system (most likely)
> tomorrow.

OK, I will then push this change afterwards.

bug#22169: 25.0.50; File name compiletion doesn't work with non-ASCII characters on OS X

[Anders Lindgren]

[-- Attachment #1: Type: text/plain, Size: 166 bytes --]

>
> > I will push my patch for using the utf-8-hfs coding system (most likely)
> > tomorrow.
>
> OK, I will then push this change afterwards.
>

Done!

    -- Anders

[-- Attachment #2: Type: text/html, Size: 457 bytes --]

bug#22169: 25.0.50; File name compiletion doesn't work with non-ASCII characters on OS X

[Eli Zaretskii]

> Date: Wed, 23 Dec 2015 07:17:32 +0100
> From: Anders Lindgren <andlind@gmail.com>
> Cc: random832@fastmail.com, 22169@debbugs.gnu.org
> 
>     > I will push my patch for using the utf-8-hfs coding system (most likely)
>     > tomorrow.
>     
>     OK, I will then push this change afterwards.
> 
> Done!

Thanks, I pushed my part, and I'm marking this done.

bug#22169: 25.0.50; File name compiletion doesn't work with non-ASCII characters on OS X

[Anders Lindgren]

[-- Attachment #1: Type: text/plain, Size: 798 bytes --]

Hi!

Eli, can you push or send me your fix to ucs-normalize. I'd like to verify
that my patch doesn't cause further problems before I re-publish it.

Also, I was curious about one thing. My intention was to include
ucs-normalize when bootstrapping for NextStep only -- why did this cause a
problem when building on other systems?

    -- Anders


On Wed, Dec 23, 2015 at 6:36 PM, Eli Zaretskii <eliz@gnu.org> wrote:

> > Date: Wed, 23 Dec 2015 07:17:32 +0100
> > From: Anders Lindgren <andlind@gmail.com>
> > Cc: random832@fastmail.com, 22169@debbugs.gnu.org
> >
> >     > I will push my patch for using the utf-8-hfs coding system (most
> likely)
> >     > tomorrow.
> >
> >     OK, I will then push this change afterwards.
> >
> > Done!
>
> Thanks, I pushed my part, and I'm marking this done.
>

[-- Attachment #2: Type: text/html, Size: 1439 bytes --]

bug#22169: 25.0.50; File name compiletion doesn't work with non-ASCII characters on OS X

[Anders Lindgren]

[-- Attachment #1: Type: text/plain, Size: 1795 bytes --]

Eli, I just saw you other letter in the other thread. I reply here instead:

I'm saying that it's not a disaster that a bug like that stays unfixed
> for a day or two.  (It wasn't meant to be a criticism on your behalf.)
>

Good to know, I was a bit stressed causing trouble for others and not being
able to look into the problem. (When it comes to core development, I still
feel like a beginner.) Just so you know, I didn't take it as criticism.



> Not sure what you are saying.  If you mean you want to wait for me to
> push my changes, then part of them need your patch, because they
> modify some of the code you added.
>
> I can send you my changes, so you could try them before both parts are
> installed, if you want.
>

Please send the patch to me so I can test here. Alternatively, if you feel
confident it will work, feel free to push the full patch.

    -- Anders


On Thu, Dec 24, 2015 at 8:23 PM, Anders Lindgren <andlind@gmail.com> wrote:

> Hi!
>
> Eli, can you push or send me your fix to ucs-normalize. I'd like to verify
> that my patch doesn't cause further problems before I re-publish it.
>
> Also, I was curious about one thing. My intention was to include
> ucs-normalize when bootstrapping for NextStep only -- why did this cause a
> problem when building on other systems?
>
>     -- Anders
>
>
> On Wed, Dec 23, 2015 at 6:36 PM, Eli Zaretskii <eliz@gnu.org> wrote:
>
>> > Date: Wed, 23 Dec 2015 07:17:32 +0100
>> > From: Anders Lindgren <andlind@gmail.com>
>> > Cc: random832@fastmail.com, 22169@debbugs.gnu.org
>> >
>> >     > I will push my patch for using the utf-8-hfs coding system (most
>> likely)
>> >     > tomorrow.
>> >
>> >     OK, I will then push this change afterwards.
>> >
>> > Done!
>>
>> Thanks, I pushed my part, and I'm marking this done.
>>
>
>

[-- Attachment #2: Type: text/html, Size: 3367 bytes --]

bug#22169: 25.0.50; File name compiletion doesn't work with non-ASCII characters on OS X

[Eli Zaretskii]

> Date: Thu, 24 Dec 2015 20:23:13 +0100
> From: Anders Lindgren <andlind@gmail.com>
> Cc: random832@fastmail.com, 22169-done@debbugs.gnu.org
> 
> Eli, can you push or send me your fix to ucs-normalize. I'd like to verify that
> my patch doesn't cause further problems before I re-publish it.

Will do, in a separate email.

> Also, I was curious about one thing. My intention was to include ucs-normalize
> when bootstrapping for NextStep only -- why did this cause a problem when
> building on other systems?

See src/lisp.mk and the rules in src/Makefile.in which produce that
file.  Every file that is preloaded on _any_ platform automatically
gets added to the list of Lisp files that are compiled by
bootstrap-emacs (as opposed to by emacs) before emacs is dumped after
loading those files in byte-compiled form.  Those files are compiled
and scanned for doc strings on all platforms, even if some of them are
not preloaded on a particular platform.