File name completion on Mac OS X with German umlauts

File name completion on Mac OS X with German umlauts

[Markus]

Hi,

I'm using a German Mac OS X 10.5.2. I wanted to open a file 
called "rückerstattung.txt". So I typed C-x C-f rü <tab> ... but 
it didn't work. I thought this was a bug in Aquamacs, but then I 
tried to do the same with that console emacs which is bundled 
with Mac OS X 10.5. Same issue.

Then, I remembered that some time ago, I had the same issue 
within Terminal where typing "rü<tab>" didn't autocomplete the 
file name.

So, can't the autocomplete within Emacs work because it doesn't 
work under Terminal?

I assume there must be an Emacs guru around who is running a 
German Mac who can help me out with this annoying behaviour.

Thanks in advance,
Markus

Re: File name completion on Mac OS X with German umlauts

[Peter Dyballa]

Am 09.03.2008 um 12:53 schrieb Markus:

> I'm using a German Mac OS X 10.5.2. I wanted to open a file called  
> "rückerstattung.txt". So I typed C-x C-f rü <tab> ... but it didn't  
> work. I thought this was a bug in Aquamacs, but then I tried to do  
> the same with that console emacs which is bundled with Mac OS X  
> 10.5. Same issue.

Did you check that both are the same version?

You could launch Aquamacs Emacs in Terminal, too:

	/Applications/Aquamacs\ Emacs.app/Contents/MacOS/Aquamacs\ Emacs -nw

>
> Then, I remembered that some time ago, I had the same issue within  
> Terminal where typing "rü<tab>" didn't autocomplete the file name.

File a bug report at apple: https://bugreport.apple.com/cgi-bin/ 
WebObjects/RadarWeb.woa

>
> So, can't the autocomplete within Emacs work because it doesn't  
> work under Terminal?


This is not really likely. The Emacsen do not use a shell based  
mechanism to expand a file name. Nevertheless, both failures can be  
related to the same cause: apple uses in HFS+ decomposed Unicode  
characters. This means that your file's name starts with ru¨ ... Have  
you tried

	C-x C-f r u TAB


BTW, there is a more Aquamacs Emacs related mailing list:

---------------------------- Info -----------------------------
List Post: <mailto:macosx-emacs@email.esm.psu.edu>
List Archives: <http://dir.gmane.org/gmane.emacs.macintosh.osx>

--
Mit friedvollen Grüßen

   Pete
                       ~  o
                        ~_\\_/\
                       ~  O   O

Re: File name completion on Mac OS X with German umlauts

[Nikolaj Schumacher]

Markus <mnissl@gmx.de> wrote:

> Hi,
>
> I'm using a German Mac OS X 10.5.2. I wanted to open a file called
> "rückerstattung.txt". So I typed C-x C-f rü <tab> ... but it didn't work. I
> thought this was a bug in Aquamacs, but then I tried to do the same with that
> console emacs which is bundled with Mac OS X 10.5. Same issue.

The reason is simple.  OSX uses a less common formatting for umlauts.
Instead of the code for the character ü it represents it as two
characters, one for u and one for the dots.  (Yes, that is still valid
Unicode.)

In fact, if you type "ru", Emacs and Bash will both offer rückerstattung as a
completion.

That being said, I don't know any solution.  I thought utf8-m.el would
fix it, but I must have been mistaken.  It's one of the cases where
UTF-8 isn't backwards compatible with ASCII when comparing strings, so
the fix is non-trivial.


regards,
Nikolaj Schumacher

Re: File name completion on Mac OS X with German umlauts

[Markus]

Peter Dyballa schrieb:
> 
> Am 09.03.2008 um 12:53 schrieb Markus:
> 
>> I'm using a German Mac OS X 10.5.2. I wanted to open a file called 
>> "rückerstattung.txt". So I typed C-x C-f rü <tab> ... but it didn't 
>> work. I thought this was a bug in Aquamacs, but then I tried to do the 
>> same with that console emacs which is bundled with Mac OS X 10.5. Same 
>> issue.
> 
> Did you check that both are the same version?

No, they are not: the "default" Emacs is GNU Emacs 22.1.1, 
Aquamacs is GNU Emacs 22.1.91.2. But I have that issue with *any* 
Emacs built for Mac OS X that is out there for OS X 10.4/10.5.

> You could launch Aquamacs Emacs in Terminal, too:
> 
>     /Applications/Aquamacs\ Emacs.app/Contents/MacOS/Aquamacs\ Emacs -nw

Here, typing ü yields to ??

>> Then, I remembered that some time ago, I had the same issue within 
>> Terminal where typing "rü<tab>" didn't autocomplete the file name.
> 
> File a bug report at apple: 
> https://bugreport.apple.com/cgi-bin/WebObjects/RadarWeb.woa

I did that. Maybe they'll take care of it.

>> So, can't the autocomplete within Emacs work because it doesn't work 
>> under Terminal?
> 
> This is not really likely. The Emacsen do not use a shell based 
> mechanism to expand a file name. Nevertheless, both failures can be 
> related to the same cause: apple uses in HFS+ decomposed Unicode 
> characters. This means that your file's name starts with ru¨ ... Have 
> you tried
> 
>     C-x C-f r u TAB

Yes, I knew this. This also works for Terminal. But hey: this is 
the year 2008, Emacs 22.x and Mac OS X 10.5 ... I expect German 
umlauts to work.

> BTW, there is a more Aquamacs Emacs related mailing list:
> 
> ---------------------------- Info -----------------------------
> List Post: <mailto:macosx-emacs@email.esm.psu.edu>
> List Archives: <http://dir.gmane.org/gmane.emacs.macintosh.osx>

Thanks for the hint, but as mentioned above, this is not an 
Aquamacs bug ... but if they fixed the issue, that would be great 
as well :-)

Thanks for your reply,
Markus

Re: File name completion on Mac OS X with German umlauts

[Markus]

Nikolaj Schumacher schrieb:
> That being said, I don't know any solution.  I thought utf8-m.el would
> fix it, but I must have been mistaken.  It's one of the cases where
> UTF-8 isn't backwards compatible with ASCII when comparing strings, so
> the fix is non-trivial.

Thanks for your reply. But I guess this is maybe not only an 
Emacs bug, but also one in Mac OS X. In the recent years, Linux 
finally succeeded in dealing with umlauts well. But I quit Linux 
three years ago for the Mac ... I didn't expect such annoying 
things to reappear :-(

Markus

Re: File name completion on Mac OS X with German umlauts

[Peter Dyballa]

Am 09.03.2008 um 22:44 schrieb Markus:

>> You could launch Aquamacs Emacs in Terminal, too:
>>     /Applications/Aquamacs\ Emacs.app/Contents/MacOS/Aquamacs\  
>> Emacs -nw
>
> Here, typing ü yields to ??

Is Terminal set into UTF-8 encoding? Is Aquamacs Emacs set into UTF-8  
"mood?" What value have LC_CTYPE or LANG in Terminal's shell  
environment? And finally: the Emacsen before 23.0.60 only "emulate"  
UTF-8 somehow.

You might try again, adding also a -Q to the invocation, with  
Terminal in some ISO Latin encoding (ISO 8859-15 for example) and  
also set LC_CTYPE to this value.


>>> So, can't the autocomplete within Emacs work because it doesn't  
>>> work under Terminal?
>> This is not really likely. The Emacsen do not use a shell based  
>> mechanism to expand a file name. Nevertheless, both failures can  
>> be related to the same cause: apple uses in HFS+ decomposed  
>> Unicode characters. This means that your file's name starts with  
>> ru¨ ... Have you tried
>>     C-x C-f r u TAB
>
> Yes, I knew this. This also works for Terminal. But hey: this is  
> the year 2008, Emacs 22.x and Mac OS X 10.5 ... I expect German  
> umlauts to work.

The umlauts are handled somehow in text – but HFS+ is something  
different. Emacs.app, the Cocoa/GNUSTEP implementation of GNU Emacs  
23.0.60,can't handle such file names. Or apple's own tools! Have you  
tried, just for fun or proof of apple's mis-concept, to attach your  
file to an eMail and typed ``r ü´´ in the form to find your file at  
once? In Finder itself it works better ...

GNU/Linux can be a better choice than Mac OS X.

--
Mit friedvollen Grüßen

   Pete

  "A designer knows he has arrived at perfection not when there is no  
longer anything to add, but when there is no longer anything to take  
away."
				– Antoine de Saint-Exupéry

Re: File name completion on Mac OS X with German umlauts

[Markus]

>>> You could launch Aquamacs Emacs in Terminal, too: 
>>> /Applications/Aquamacs\ Emacs.app/Contents/MacOS/Aquamacs\
>>>  Emacs -nw
>> 
>> Here, typing ü yields to ??
> 
> Is Terminal set into UTF-8 encoding?

Yes, I'm using the default settings of Mac OS X 10.5.

I just found out that I get the ü correctly displayed when
invoking Aquamacs like this: emacs -nw -Q

> Is Aquamacs Emacs set into UTF-8 "mood?"

How can I find that out?

> What value have LC_CTYPE or LANG in Terminal's shell 
> environment?

chris:~ mnissl$ locale
LANG="de_DE.UTF-8"
LC_COLLATE="de_DE.UTF-8"
LC_CTYPE="de_DE.UTF-8"
LC_MESSAGES="de_DE.UTF-8"
LC_MONETARY="de_DE.UTF-8"
LC_NUMERIC="de_DE.UTF-8"
LC_TIME="de_DE.UTF-8"
LC_ALL=

> And finally: the Emacsen before 23.0.60 only "emulate" UTF-8
> somehow.

Geeze. And I thought Emacs 22.x would have sorta reached a
feature complete state of "basic" editing.

> You might try again, adding also a -Q to the invocation, with 
> Terminal in some ISO Latin encoding (ISO 8859-15 for example) 
> and also set LC_CTYPE to this value.

I'm not fond of switching to a specific codepage if I have a 
unicode capable operating system and editor -- at least I thought 
I had ones ...

>>>> So, can't the autocomplete within Emacs work because it 
>>>> doesn't work under Terminal?
>>> This is not really likely. The Emacsen do not use a shell 
>>> based mechanism to expand a file name.

Then why does the Emacs file prompt feature the same bug as
Terminal (this happens both with Aquamacs 1.3 and the Emacs
bundled with Mac OS X 10.5):

1. C-x C-f rü<tab>    (nothing happens)
2. Backspace (to delete the ü), u<tab>     (autocompletes)
3. Delete backwards until you only have the r
4. Hit tab ... and you'll end up with r̈ckerstattung.txt

Try the same in Terminal with "ls rü<tab>" ...

> Have you tried, just for fun or proof of apple's mis-concept,
> to attach your file to an eMail and typed ``r ü´´ in the form
> to find your file at once?

Err, Mac OS file open dialogs do not feature a prompt to type the 
file name ... only mouse clicks!

> GNU/Linux can be a better choice than Mac OS X.

Well, in fact I switched to Mac OS X in order *not* to have the 
conversation we have now ... :-(

Thanks for your input,
Markus

Re: File name completion on Mac OS X with German umlauts

[Peter Dyballa]

Am 10.03.2008 um 19:57 schrieb Markus:

> I just found out that I get the ü correctly displayed when
> invoking Aquamacs like this: emacs -nw -Q

So obviously your or the system's init file sets something  
incompatible, something pre-Iraq wars ... maybe a "language- 
environment?"

>
>> Is Aquamacs Emacs set into UTF-8 "mood?"

Mode-line! It should start in the left corner with -u:. You press  
with the mouse cursor on the symbol between - and : ...

>>>>> So, can't the autocomplete within Emacs work because it doesn't  
>>>>> work under Terminal?
>>>> This is not really likely. The Emacsen do not use a shell based  
>>>> mechanism to expand a file name.
>
> Then why does the Emacs file prompt feature the same bug as
> Terminal (this happens both with Aquamacs 1.3 and the Emacs
> bundled with Mac OS X 10.5):
>
> 1. C-x C-f rü<tab>    (nothing happens)
> 2. Backspace (to delete the ü), u<tab>     (autocompletes)
> 3. Delete backwards until you only have the r
> 4. Hit tab ... and you'll end up with rckerstattung.txt
>
> Try the same in Terminal with "ls rü<tab>" ...

My test file, since Jaguar (?), is called äöü... No file name  
completion starts for me.

The shell in Terminal uses a different mechanism for file name  
completion than GNU Emacs. Is this the answer you wanted to read?  
Both mechanisms choose from the same pool of HFS+ decomposed and  
deranged character strings.

Is bash still 2.05 from the 90ies?

>
>> Have you tried, just for fun or proof of apple's mis-concept,
>> to attach your file to an eMail and typed ``r ü´´ in the form
>> to find your file at once?
>
> Err, Mac OS file open dialogs do not feature a prompt to type the  
> file name ... only mouse clicks!

That's true! And you never tried in Finder, best when you have at  
least one file starting with a or A and at least another one starting  
with ä or Ä, just to type - not in the search hole! ­ a or ä? Maybe  
followed by one or more TABs? (Try both! Does a find the ä file? Does  
ä find the a and the ä file?)

The same way you can fast select a file from select menu to choose from.

--
Mit friedvollen Grüßen

   Pete

To most people solutions mean finding the answers. But to chemists  
solutions
are things that are still all mixed up.

Re: File name completion on Mac OS X with German umlauts

[Markus]

>>> Is Aquamacs Emacs set into UTF-8 "mood?"
> 
> Mode-line! It should start in the left corner with -u:. You press with 
> the mouse cursor on the symbol between - and : ...

Err, my mode-line looks like this: 1:-- rückerstattung.txt

When hovering the 1, the tool tip says: iso-latin-1-unix buffer

But this tells you about the encoding of the file content, not 
its name, I guess!

> My test file, since Jaguar (?), is called ���... No file name completion 
> starts for me.

Hmmm, in the carbon copy e-mail that I got from your reply, I saw 
that your file was called äöü ... and not ??? ... I guess your 
message didn't make the UTF-8 way to this newsgroup ...???

> The shell in Terminal uses a different mechanism for file name 
> completion than GNU Emacs. Is this the answer you wanted to read? Both 
> mechanisms choose from the same pool of HFS+ decomposed and deranged 
> character strings.

I see. Same issue, some implications!

> And you never tried in Finder, best when you have at least 
> one file starting with a or A and at least another one starting with � 
> or �, just to type - not in the search hole! � a or �?

Well, after having replied to your message, I understood what you 
meant. So I created a couple of files called ra, rb, rc, ru and 
rü. Typing rü in Finder selects ru, not rü ... Finder ain't 
smarter here. Duh :-(

Re: File name completion on Mac OS X with German umlauts

[Peter Dyballa]

Am 10.03.2008 um 22:25 schrieb Markus:

> Err, my mode-line looks like this: 1:-- rückerstattung.txt
>
> When hovering the 1, the tool tip says: iso-latin-1-unix buffer
>
> But this tells you about the encoding of the file content, not its  
> name, I guess!

Not really! GNU Emacs reads the contents of the file in cold blood.  
Then, according to the encoding set or chosen, it *presents* this  
pile of bytes as a bunch of glyphs taken from the font used. The  
encoding maps one or more bytes to a single character. Each such  
character has its distinct character code. According to this the  
glyph is chosen from the font and displayed.

The encoding of the file contents can be "encoded" (better: recorded  
or saved) in a local variable inside the file ... as an option.

A byte \327 (octal, 215 dec, D7 hex) represents in most ISO 8859  
encodings (1, 2, 3, 4, 9, 13, 15) the MULTIPLICATION SIGN at U+00D7  
(in UTF-8 it's encoded as the two bytes C3 97). The same byte is

   in ISO 8859-5  the CYRILLIC SMALL LETTER ZE              at U+0437  
(D0 B7 in UTF-8),
   in ISO 8859-6  the ARABIC LETTER TAH                     at U+0637  
(D8 B7 in UTF-8),
   in ISO 8859-7  the GREEK CAPITAL LETTER CHI              at U+03A7  
(CE A7 in UTF-8),
   in ISO 8859-11 the THAI CHARACTER SARA UEE               at U+0E37  
(E0 B8 B7 in UTF-8),
   in ISO 8859-14 the LATIN CAPITAL LETTER T WITH DOT ABOVE at U+1E6A  
(E1 B9 AA in UTF-8),
   in ISO 8859-16 the LATIN CAPITAL LETTER S WITH ACUTE     at U+015A  
(C5 9A in UTF-8).

(Continue with Mac, NeXT, MS, ... encodings!)

Usually Emacs learns from the shell which encoding is used, or might  
be preferred. In Mac OS X (as on the NeXT) a different mechanism to  
launch or create a process is used, without shell interaction. So  
Cocoa or Carbon applications (Aqua clients?) come up as babies, as  
mentioned by Tim Buckley in his “Song To The Siren.” Since your  
shell in Terminal has the proper settings, you could launch *another*  
Aquamacs Emacs from Terminal as "/Applications/Aquamacs\ Emacs.app/ 
Contents/MacOS/Aquamacs\ Emacs &" or as single application as "open / 
Applications/Aquamacs\ Emacs.app" – without &. The variable process- 
environment has the environment in which Emacs runs. You could also  
create a *shell* buffer inside Emacs and check the *shell* buffer's  
environment. It should be quite the same.

If either of these launch methods does not create a ``-u:´´ mode- 
line, then you have some code, probably in your init file, that  
changes the well derived value to something pre-historic. A

	(prefer-coding-system 'utf-8)

could correct this (or a value of utf-8m, which will help with the  
decomposed file names). (The Emacs Wiki has nodes related to Mac OS X  
and its way to create a process with a link to apple documents.)

>> And you never tried in Finder, best when you have at least one  
>> file starting with a or A and at least another one starting with  
>> � or �, just to type - not in the search hole! � a or �?
>
> Well, after having replied to your message, I understood what you  
> meant. So I created a couple of files called ra, rb, rc, ru and rü.  
> Typing rü in Finder selects ru, not rü ... Finder ain't smarter  
> here. Duh :-(


Pressing TAB you can re-search. Try this, too!

--
Mit friedvollen Grüßen

   Pete

Encryption, n.: A powerful algorithmic encoding technique
		employed in the creation of computer manuals.

Re: File name completion on Mac OS X with German umlauts

[Markus]

Peter,

thanks a lot for your information. You seem to have a profound 
knowledge of Emacs internals!

But as mentioned earlier in this thread, I switched to the Mac 
because I don't want to know why things *work*. I don't want to 
spend my evenings worrying about why things do *not* work -- I 
did that when using Linux for eight years.

It seems that both Emacs and Mac OS X have to fix some bugs in 
regards to my issue. Hopefully some Emacs developer reads this or 
is aware of this. I also let Apple know about that bug.

But I'm out of here and won't investigate any further. I just 
deal with it until it will be hopefully fixed one day or find 
some alternative ... but is there an alternative to Emacs??? ;-))

Thank you once again,
Markus

Re: File name completion on Mac OS X with German umlauts

[Peter Dyballa]

Am 11.03.2008 um 22:24 schrieb Markus:

> But I'm out of here and won't investigate any further. I just deal  
> with it until it will be hopefully fixed one day or find some  
> alternative ... but is there an alternative to Emacs??? ;-))


There might be two: Emacs.app, based on GNU Emacs 23.0.60, the  
Unicode Emacs, and GNU Emacs 23.0.60 as X client. Both can be  
compiled from source.

Leopard, Mac OS X 10.5, offers an up-to-date X server at least – and  
it must be based on X11R7.x, probably X11R7.2. X11 can have a real  
process environment that is inherited from the X server.

Emacs.app (http://emacs-app.sourceforge.net/) does not use Carbon,  
but Cocoa. It's still unfinished, it has no SVG support. There is a  
problem with preview-mode in AUCTeX. It does neither use htmlize-view  
nor mac-print for printing. Real α!

In the next few months its code will be integrated into CVS HEAD,  
i.e. the code of GNU Emacs 23.0.60, the development version of stable  
GNU Emacs. Since GNU Emacs 23.0.x does not support Carbon builds,  
it's likely that Emacs.app will become the native Mac OS X Emacs.  
Some Carbon Emacs developers are already working on code Emacs.app  
can use.

--
Mit friedvollen Grüßen

   Pete

Some day we may discover how to make magnets that can point in any  
direction.

Re: File name completion on Mac OS X with German umlauts

[Nikolaj Schumacher]

Markus <mnissl@gmx.de> wrote:

> But I guess this is maybe not only an Emacs bug, but also one in Mac
> OS X. In the recent years, Linux finally succeeded in dealing with
> umlauts well. But I quit Linux three years ago for the Mac ... I
> didn't expect such annoying things to reappear :-(

Actually ...
As I said OSX uses a different method for representing umlauts.  But at
least it uses that representation exclusively in its FS.

GNU/Linux supports BOTH representations.  For instance, if you rsync
"rückerstattung" from a Linux machine to a Mac machine and back, you
will end up with 2 files named "rückerstattung".  Each will have a
different representation to the machine but will look indistinguishable
to the user.  Completing will work nicely on one, but have the very same
problems on the other...

I don't think either OS is to blame.  It's an app (or maybe string
library) problem.  Nobody expects the same character to have two legal
representations.

(Did someone say Unicode was the /end/ to all encoding woes?)

regards,
Nikolaj Schumacher

Re: File name completion on Mac OS X with German umlauts

[Eli Zaretskii]

> From: Nikolaj Schumacher <n_schumacher@web.de>
> Date: Wed, 12 Mar 2008 16:44:03 +0100
> Cc: help-gnu-emacs@gnu.org
> 
> GNU/Linux supports BOTH representations.  For instance, if you rsync
> "rückerstattung" from a Linux machine to a Mac machine and back, you
> will end up with 2 files named "rückerstattung".  Each will have a
> different representation to the machine but will look indistinguishable
> to the user.  Completing will work nicely on one, but have the very same
> problems on the other...
> 
> I don't think either OS is to blame.  It's an app (or maybe string
> library) problem.

It is only a ``problem'' if you accept the view that no two files in
the same directory can have names that are pronounced identically.  We
are used to think that, but only because of the particular design
decisions built into the popular filesystems.  However, this design is
by no means an axiom, and in fact we people already know how to deal
with situations where two different entities have the same name.  One
example is movies.  In fact, even case-sensitive filesystems already
present a similar problem, because README and readme are the same
word.

In other words, the name as pronounced does not have to be a unique
descriptor of a file.  There's plenty of other attributes (e.g., the
inode) to distinguish them.

Re: File name completion on Mac OS X with German umlauts

[Nikolaj Schumacher]

Eli Zaretskii <eliz@gnu.org> wrote:

> It is only a ``problem'' if you accept the view that no two files in
> the same directory can have names that are pronounced identically.

No, it's not just that.
Certainly, you could have files "X" and "Ⅹ" (the Roman numeral).  Even
if they look the same there is no problem (other than likely user
confusion) in having both.

However, the two types of "ü" are the same character, or at least
functionally equivalent characters.  They should be considered equal.
But comparing them properly requires normalization
(cf. http://www.unicode.org/unicode/reports/tr15/).

OSX does normalization in its file system.  GNU/Linux apparently does not.

Emacs must also be doing some normalization... switch-to-buffer
completion works on "rückerstattung" after all.  Only `read-file-name'
doesn't.  Hmm, maybe this /is/ an Emacs bug after all.


regards,
Nikolaj Schumacher

Re: File name completion on Mac OS X with German umlauts

[Eli Zaretskii]

> Cc: help-gnu-emacs@gnu.org
> From: Nikolaj Schumacher <n_schumacher@web.de>
> Date: Wed, 12 Mar 2008 22:25:58 +0100
> 
> Eli Zaretskii <eliz@gnu.org> wrote:
> 
> > It is only a ``problem'' if you accept the view that no two files in
> > the same directory can have names that are pronounced identically.
> 
> No, it's not just that.
> Certainly, you could have files "X" and "Ⅹ" (the Roman numeral).  Even
> if they look the same there is no problem (other than likely user
> confusion) in having both.
> 
> However, the two types of "ü" are the same character, or at least
> functionally equivalent characters.  They should be considered equal.
> But comparing them properly requires normalization
> (cf. http://www.unicode.org/unicode/reports/tr15/).

I know about normalization.  What I'm saying is that there's no
problem, conceptually, to have several files having equivalent names.
In particular, I don't see why file names with "X" and "Ⅹ" are okay
for you, but the two types of "ü" are not.  I think they are both
exhibiting the same problem.

Re: File name completion on Mac OS X with German umlauts

[Juanma Barranquero]

On Wed, Mar 12, 2008 at 10:55 PM, Eli Zaretskii <eliz@gnu.org> wrote:

>  In particular, I don't see why file names with "X" and "Ⅹ" are okay
>  for you, but the two types of "ü" are not.  I think they are both
>  exhibiting the same problem.

Well, the first two, U+0058 LATIN CAPITAL LETTER X and U+2169 ROMAN
NUMERAL TEN, are meant to be semantically distinct.

However, U+00FC LATIN SMALL LETTER U WITH DIAERESIS is supposed to be
semantically identical to U+0075 LATIN SMALL LETTER U plus U+0308
COMBINING DIAERESIS. In the first case, we're talking about different
abstract characters represented with different codepoints. In the
second case, we're talking about the same abstract character. Other
than a byte comparison, there is no distinction between both ü. You
wouldn't expect a font supporting Unicode to represent U+00FC
different that U+0075 plus U+0308.

 Juanma

Re: File name completion on Mac OS X with German umlauts

[Eli Zaretskii]

> Date: Thu, 13 Mar 2008 00:23:29 +0100
> From: "Juanma Barranquero" <lekktu@gmail.com>
> Cc: help-gnu-emacs@gnu.org
> 
> On Wed, Mar 12, 2008 at 10:55 PM, Eli Zaretskii <eliz@gnu.org> wrote:
> 
> >  In particular, I don't see why file names with "X" and "Ⅹ" are okay
> >  for you, but the two types of "ü" are not.  I think they are both
> >  exhibiting the same problem.
> 
> Well, the first two, U+0058 LATIN CAPITAL LETTER X and U+2169 ROMAN
> NUMERAL TEN, are meant to be semantically distinct.
> 
> However, U+00FC LATIN SMALL LETTER U WITH DIAERESIS is supposed to be
> semantically identical to U+0075 LATIN SMALL LETTER U plus U+0308
> COMBINING DIAERESIS. In the first case, we're talking about different
> abstract characters represented with different codepoints. In the
> second case, we're talking about the same abstract character. Other
> than a byte comparison, there is no distinction between both ü. You
> wouldn't expect a font supporting Unicode to represent U+00FC
> different that U+0075 plus U+0308.

I see no difference here: in both case the bytestreams are different.
The fact that Unicode sees one case but not the other as two
equivalent strings, is irrelevant here, because file names are not
simple text strings, they have other semantics.

Anyway, we are not talking about fonts, nor about Unicode decisions
wrt character equivalency.  We (at least I) were talking about the
``problem'' that several files have the same name if a file name is to
be interpreted as a text string.  As I said, I see no problem here,
just entrenched customs.

Re: File name completion on Mac OS X with German umlauts

[Juanma Barranquero]

On Thu, Mar 13, 2008 at 5:28 AM, Eli Zaretskii <eliz@gnu.org> wrote:

>  I see no difference here: in both case the bytestreams are different.
>  The fact that Unicode sees one case but not the other as two
>  equivalent strings, is irrelevant here, because file names are not
>  simple text strings, they have other semantics.

I think we're accidentally rehearsing a recent discussion on the git
developer's list :)

You're approaching the issue (I think) from the low-level, operating
system side, while I'm approaching it from the high-level, user side.
To me, of course the most important semantics of a filename is how it
does appear to the user. I don't want "readme" and "README" to be two
different files, for example. I'm strongly in favor of the
case-folding-to compare, case-preserving-to-store side.

>  Anyway, we are not talking about fonts, nor about Unicode decisions
>  wrt character equivalency.  We (at least I) were talking about the
>  ``problem'' that several files have the same name if a file name is to
>  be interpreted as a text string.  As I said, I see no problem here,
>  just entrenched customs.

From my point of view, of course it is a problem. But I suppose
there's no point in discussing it.

 Juanma

Re: File name completion on Mac OS X with German umlauts

[Nikolaj Schumacher]

"Juanma Barranquero" <lekktu@gmail.com> wrote:

> On Thu, Mar 13, 2008 at 5:28 AM, Eli Zaretskii <eliz@gnu.org> wrote:
>
>>  I see no difference here: in both case the bytestreams are different.
>>  The fact that Unicode sees one case but not the other as two
>>  equivalent strings, is irrelevant here, because file names are not
>>  simple text strings, they have other semantics.

I think that depends on the file system.  If the file system is ignorant
of encodings (like a .txt file is) then I agree its irrelevant.

But if the file system specifies an encoding (like an .xml file does) the
appropriate semantics /should/ apply.

> To me, of course the most important semantics of a filename is how it
> does appear to the user. I don't want "readme" and "README" to be two
> different files, for example. I'm strongly in favor of the
> case-folding-to compare, case-preserving-to-store side.

That, of course, opens another can of worms, because capitalization
isn't absolute.  "i" and "I" are equal files for you, but in Turkey
they are distinct files... Unicode equality is well-defined, at least.

> From my point of view, of course it is a problem. But I suppose
> there's no point in discussing it.

I suppose.  I didn't really want to start a discussion about the
file-system part anyway.


regards,
Nikolaj Schumacher

Re: File name completion on Mac OS X with German umlauts

[Juanma Barranquero]

On Thu, Mar 13, 2008 at 2:49 PM, Nikolaj Schumacher <n_schumacher@web.de> wrote:

>  That, of course, opens another can of worms, because capitalization
>  isn't absolute.  "i" and "I" are equal files for you, but in Turkey
>  they are distinct files...

Usually, capitalization is quite stable, for your local language
setup. Of course in a Turkish language environment the relation
between dotted and dotless latin i characters should have to be taken
into account.

 Juanma

Re: File name completion on Mac OS X with German umlauts

[Juanma Barranquero]

On Thu, Mar 13, 2008 at 3:17 PM, Juanma Barranquero <lekktu@gmail.com> wrote:

>  Usually, capitalization is quite stable

s/capitalization/case folding/

 Juanma

Re: File name completion on Mac OS X with German umlauts

[Eli Zaretskii]

> Date: Thu, 13 Mar 2008 09:18:26 +0100
> From: "Juanma Barranquero" <lekktu@gmail.com>
> Cc: help-gnu-emacs@gnu.org
> 
> You're approaching the issue (I think) from the low-level, operating
> system side, while I'm approaching it from the high-level, user side.

Actually, I did both: from the low-level point of view, the file names
are different due to different representations of the same letter;
from the high-level user point of view, I see no problem with several
files having the same name.

> I don't want "readme" and "README" to be two different files

Why not?  Do you have trouble with two different entities having the
same name?  We see such things every day in our lives.

Re: File name completion on Mac OS X with German umlauts

[Juanma Barranquero]

On Thu, Mar 13, 2008 at 9:10 PM, Eli Zaretskii <eliz@gnu.org> wrote:

>  Actually, I did both: from the low-level point of view, the file names
>  are different due to different representations of the same letter;
>  from the high-level user point of view, I see no problem with several
>  files having the same name.

You don't see a problem with a listing of the files, and I don't,
either. But if you're talking to the guy in the cubicle next to yours,
saying "the data is in the s
`Sales 2007' file. Uppercase Sales, of course; lowercase Sales is
another thing altogether" would surely prompt a comment about choosing
more sensible names. Most discussions I've seen about this issue seem
to assume file names are mostly computer generated, or for program
consumption; non-programmer users would usually choose descriptive
names and not rely in non-semantic, non-verbal cues like case.

>  Why not?  Do you have trouble with two different entities having the
>  same name?

Sometimes, yes, unless we're talking of entities in very different contexts.

> We see such things every day in our lives.

Yes. And it often causes confusion, and prompts further explanations.
"Body snatchers? Book or film? If film, which one?"

 Juanma

Re: File name completion on Mac OS X with German umlauts

[Markus]

Hey guys,

you're now having a rather philosophical discussion about file 
names, upper and lower cases and so on ... but what has this to 
do with my problem?

Imagine you create with your preferred version of Emacs using 
your preferred operating system with its preferred file system 
one day a file called rückerstattung. The next day, you would 
like to open this file with Emacs again by typing C-x C-f rü<tab> 
... either of you would expect Emacs to autocomplete it to 
rückerstattung, wouldn't you?

Regardless of how the ü is saved in the file system ... when you 
enter ü twice, don't you expect twice the same results??

This newsgroup is still gnu.emacs.HELP ... not alt.religion.emacs 
or whatever ...

Markus

Re: File name completion on Mac OS X with German umlauts

[Markus]

> There might be two: Emacs.app, based on GNU Emacs 23.0.60, the Unicode 
> Emacs, and GNU Emacs 23.0.60 as X client. Both can be compiled from source.
> 
> [...]

Peter, thanks for this information. I will probably give it a try 
again when Emacs 23.x is declared stable for public.

Re: File name completion on Mac OS X with German umlauts

[Peter Dyballa]

Am 13.03.2008 um 23:09 schrieb Markus:

> Regardless of how the ü is saved in the file system ... when you  
> enter ü twice, don't you expect twice the same results??


Exactly this is the situation on Mac OS X.

Lower case, upper case, or mixed case file names can exist in this or  
that Mac OS X supported file system. Those who need such a thing have  
more choices then the U.S. voters will have.

--
Mit friedvollen Grüßen

   Pete

They're putting dimes in the hole in my head to see the change in me.

Re: File name completion on Mac OS X with German umlauts

[Peter Dyballa]

Am 13.03.2008 um 23:11 schrieb Markus:

> I will probably give it a try again when Emacs 23.x is declared  
> stable for public.

Christmas? There have been so many changes that a release under a  
22.x version number is not likely.

--
Mit friedvollen Grüßen

   Pete

Es geht nix über eine elektrische Klobürste!

Re: File name completion on Mac OS X with German umlauts

[Markus]

>> I will probably give it a try again when Emacs 23.x is declared stable 
>> for public.
> 
> Christmas? There have been so many changes that a release under a 22.x 
> version number is not likely.

Well, there have been 6 years between Emacs 21.1 and 22.1. Is it 
already Christmas 2008 when Emacs 23 should be released?

I can't tell when an Emacs version can be considered stable 
enough unless it's officially announced. Using a version from 
HEAD CVS is not always the best thing to do.

Anyway, in your previous message, you wrote: "Emacs.app, based on 
GNU Emacs 23.0.60, the Unicode Emacs". I tried that version 
(9.0-rc3), but there was no amelioration regarding my rü<tab> 
problem. Am I missin' something here??

Markus

Re: File name completion on Mac OS X with German umlauts

[Peter Dyballa]

Am 16.03.2008 um 13:51 schrieb Markus:

> Anyway, in your previous message, you wrote: "Emacs.app, based on  
> GNU Emacs 23.0.60, the Unicode Emacs". I tried that version (9.0- 
> rc3), but there was no amelioration regarding my rü<tab> problem.  
> Am I missin' something here??


Not yet. It *has* options to improve. For this I'll file another bug  
report.

--
Mit friedvollen Grüßen

   Pete

Without vi there is only GNU Emacs

Re: File name completion on Mac OS X with German umlauts

[Piet van Oostrum]

>>>>> Nikolaj Schumacher <n_schumacher@web.de> (NS) wrote:

>NS> Eli Zaretskii <eliz@gnu.org> wrote:
>>> It is only a ``problem'' if you accept the view that no two files in
>>> the same directory can have names that are pronounced identically.

>NS> No, it's not just that.
>NS> Certainly, you could have files "X" and "Ⅹ" (the Roman numeral).  Even
>NS> if they look the same there is no problem (other than likely user
>NS> confusion) in having both.

>NS> However, the two types of "ü" are the same character, or at least
>NS> functionally equivalent characters.  They should be considered equal.
>NS> But comparing them properly requires normalization
>NS> (cf. http://www.unicode.org/unicode/reports/tr15/).

>NS> OSX does normalization in its file system.  GNU/Linux apparently does not.

Right. In GNU/Linux a filename is just a sequence of bytes without
interpretation. The interpretation is done by the programs. Nowadays modern
GNU/Linux systems tend to use UTF-8 as the default interpretation. But if
you would mount a filesystem from such a system on another one that has
Latin-1 as the preferred encoding, your filenames with non-ASCII characters
would look weird. And also a filename with ü as unnormalized UTF-8 and
another one with ü as normalised UTF-8 would be different files, but in an
ls listing they would look identical (the filenames, not the files). And I
guess the normalised one would not complete on ü, but ir will on u, and the
other one just the opposite.

On Mac OS X, however, the interpretation of the filenames as UTF-8 is part
of the filesystem. It will only use the normalized version, even when you
use the unnormalized in a system call. So you can't have both in the
filesystem. However, Emacs only uses the unnormalized version when you
enter characters in the normal way, and therefore the completion fails. For
it to succeed Emacs would have to do the normalization first (there are OS
functions for this.

>NS> Emacs must also be doing some normalization... switch-to-buffer
>NS> completion works on "rückerstattung" after all.  Only `read-file-name'
>NS> doesn't.  Hmm, maybe this /is/ an Emacs bug after all.

No, it doesn't do normalization. For buffers it is the same as for
filenames. But usually you don't have normalized buffer names (except for
those where normalized is the same as unnormalized of course). When you
create a file with name rückerstattung on OS X and open it from a directory
listing (where it shows as rückerstattung) you get a buffer name
rückerstattung. This will not complete from rü.
-- 
Piet van Oostrum <piet@cs.uu.nl>
URL: http://pietvanoostrum.com [PGP 8DAE142BE17999C4]
Private email: piet@vanoostrum.org

Re: File name completion on Mac OS X with German umlauts

[Nikolaj Schumacher]

Piet van Oostrum <piet@cs.uu.nl> wrote:

>>NS> Emacs must also be doing some normalization... switch-to-buffer
>>NS> completion works on "rückerstattung" after all.  Only `read-file-name'
>>NS> doesn't.  Hmm, maybe this /is/ an Emacs bug after all.
>
> No, it doesn't do normalization. For buffers it is the same as for
> filenames. But usually you don't have normalized buffer names (except for
> those where normalized is the same as unnormalized of course). When you
> create a file with name rückerstattung on OS X and open it from a directory
> listing (where it shows as rückerstattung) you get a buffer name
> rückerstattung. This will not complete from rü.

Ah, well, I didn't verify my claim with emacs -q.
I have

  (require 'utf-8m)
  (setq file-name-coding-system 'utf-8m))

in my .emacs, which apparently makes all of that work.


This is very interesting.  Apparently `read-file-name' doesn't use
`file-name-coding-system' when completing.

More interestingly, functions like `directory-files' do.  Thus, I was
able to hack up this alternative to `find-file':

(defun my-complete-file-name (input ignored all-p)
  (let* ((dir (or (file-name-directory input) default-directory))
         (file (file-name-nondirectory input))
         (dir-files (directory-files dir nil (concat "^" file))))
    (if all-p
        (mapcar (lambda (file) (if (file-is-dir-p file)
                                   (file-name-as-directory file)
                                 file))
                (all-completions file dir-files))
      (let ((completion (try-completion file dir-files)))
        (if (not (stringp completion))
            completion
          (setq completion (concat dir completion))
          (if (file-is-dir-p completion)
              (file-name-as-directory completion)
            completion))))))

(defun my-find-file ()
  (interactive)
  (completing-read "Find file: " 'my-complete-file-name nil nil "~/"))

It's pretty rough and much slower than the original, but it will
correctly complete "rü".


regards,
Nikolaj Schumacher

Re: File name completion on Mac OS X with German umlauts

[Eli Zaretskii]

> From: Nikolaj Schumacher <n_schumacher@web.de>
> Date: Mon, 17 Mar 2008 17:44:31 +0100
> Cc: help-gnu-emacs@gnu.org
> 
> This is very interesting.  Apparently `read-file-name' doesn't use
> `file-name-coding-system' when completing.

How do you see that?  Just looking at the sources, I see it does.

Re: File name completion on Mac OS X with German umlauts

[Nikolaj Schumacher]

Eli Zaretskii <eliz@gnu.org> wrote:

>> From: Nikolaj Schumacher <n_schumacher@web.de>
>>
>> This is very interesting.  Apparently `read-file-name' doesn't use
>> `file-name-coding-system' when completing.
>
> How do you see that?  Just looking at the sources, I see it does.

Sorry, I phrased that poorly.  I only intended to express a suspicion...

After some actual research, I the real reason appears to be that
utf-8m.el only offers one-way transformation and `read-file-name' needs
the other direction.

A naive attempt of adding that worked surprisingly well.  The only issue
I have is that Emacs always says "Complete, but not unique".  Don't know
why, yet.



(require 'utf-8m)

(defun utf-8m-pre-write-conversion (beg end)
  (save-excursion ;; conversion (NFC -> NFD)
    (save-restriction
      (narrow-to-region beg end)
      (let ((str (buffer-string)))
        (delete-region (point-min) (point-max))
        (insert
         (decode-coding-string
          (or (mac-code-convert-string
               (encode-coding-string str 'utf-8) 'utf-8 'utf-8 'NFD) "")
          'utf-8)
          ))
      )))

(make-coding-system
 'utf-8m-alt 4 ?U
 "modified UTF-8 encoding for Mac OS X hfs plus volume format."
 '(ccl-decode-mule-utf-8 . ccl-encode-mule-utf-8)
 `((safe-charsets
    ascii
    eight-bit-control
    eight-bit-graphic
    latin-iso8859-1
    mule-unicode-0100-24ff
    mule-unicode-2500-33ff
    mule-unicode-e000-ffff
    ,@(if utf-translate-cjk-mode
          utf-translate-cjk-charsets))
   (mime-charset . nil)
   (coding-category . coding-category-utf-8)
   (valid-codes (0 . 255))
   (pre-write-conversion . ,(if (fboundp 'mac-code-convert-string)
                                'utf-8m-pre-write-conversion
                              'utf-8-pre-write-conversion))
   ,(if (functionp 'mac-code-convert-string)
        '(post-read-conversion . utf-8m-e22-mac-post-read-conversion)
      '(post-read-conversion . utf-8m-e22-post-read-conversion))
   (translation-table-for-encode . utf-translation-table-for-encode)
   (dependency unify-8859-on-encoding-mode
               unify-8859-on-decoding-mode
               utf-fragment-on-decoding
               utf-translate-cjk-mode)))


(require 'utf-8m-alt)
(setq file-name-coding-system 'utf-8m-alt)


regards,
Nikolaj Schumacher