Reading national characters in filenames on Mac OS X?

Reading national characters in filenames on Mac OS X?

[Jonas Steverud]

Hello.

(In case this should be a faq, please direct me to wherever I can find
the asnwer, I've googled around but whithout success.)

Since I'm swedish speaking I find it quite natural to have 
directory names and filenames in swedish, which means that quite a
few will contain the swedish letters, åäöÅÄÖ.

The problem is that Emacs[1] cannot handle those files since Mac OS X
10.2.5 use UTF-8 (?) for encoding filenames - not in the current set-up
at least. It thinks the multibyte is two characters.

Does anyone know how to make Emacs understand that the names are UTF-8
encoded?

Apple's X11.app ß3
LANG=sv_SE
LC_CTYPE=iso_8859_1

(set-language-environment "Latin-1")

The above probably sets the following:
default-file-name-coding-system's value is iso-latin-1
file-name-coding-system's value is nil

I tried to set file-name-coding-system to 'utf-8 and 'mac-roman and
Emacs stops complaing about the file's name (with iso-latin-1 it can
walk down a directory path with non-A-Z but when I try to select a
file it claims the directory doesn't exist ["run M-x make-directory"])
but I don't see the non-A-Z character but some other two character
representation, i.e. Emacs doesn't decode the multibyte when it is viewed.

What is the recommended set-up if I want to be able to read a Mac
filesystem and edit text in Latin-1 - I have too many files using
Latin-1, esp. Gnus, to change to UTF-8. But UTF-8 for editing is an
option if all my old files won't confuse Emacs, i.e. I don't want to
lose all my mails.

Is there a (set-language-environemnt "utf-8")-equivalent? I couldn't
find anything appropriate. :-(

Any suggestions?

-------
[1]  GNU Emacs 21.2.1 (powerpc-apple-darwin, X toolkit) of 2003-01-02
on Neocortex.local. Downloaded and patched via the Fink project.
-- 
(         http://hem.bredband.net/steverud        !     Wei Wu Wei     )
(        Meaning of U2 Lyrics, Roleplaying        !  To Do Without Do  )

Re: Reading national characters in filenames on Mac OS X?

[Kai Großjohann]

Jonas Steverud <tvrud-usenet@spray.se> writes:

> I tried to set file-name-coding-system to 'utf-8 and 'mac-roman and
> Emacs stops complaing about the file's name (with iso-latin-1 it can
> walk down a directory path with non-A-Z but when I try to select a
> file it claims the directory doesn't exist ["run M-x make-directory"])
> but I don't see the non-A-Z character but some other two character
> representation, i.e. Emacs doesn't decode the multibyte when it is viewed.

(set-file-name-coding-system 'utf-8) should work, I'd think.  Could
you try that and then show an example of what happens?

-- 
file-error; Data: (Opening input file no such file or directory ~/.signature)

Re: Reading national characters in filenames on Mac OS X?

[Karl Eichwalder]

kai.grossjohann@gmx.net (Kai Großjohann) writes:

> (set-file-name-coding-system 'utf-8) should work, I'd think.

This might work up to some degree.

The crucial point is to use Emacs from CVS (trunk or unicode branch).
Version 21.3 will expose problems sooner or later.

Since some weeks I simple start Emacs (CVS, trunk) on SuSE Linux
8.2-ix86 this way:

    export LANG=de_DE.UTF-8             # set in one of the profiles
    emacs

Kenichi Handa fixed serveral issues recently.

-- 
                                                         |      ,__o
http://www.gnu.franken.de/ke/                            |    _-\_<,
ke@suse.de (work) / keichwa@gmx.net (home)               |   (*)/'(*)

Re: Reading national characters in filenames on Mac OS X?

[Jonas Steverud]

Karl Eichwalder <keichwa@gmx.net> writes:

> kai.grossjohann@gmx.net (Kai Großjohann) writes:
>
>> (set-file-name-coding-system 'utf-8) should work, I'd think.
>
> This might work up to some degree.
>
> The crucial point is to use Emacs from CVS (trunk or unicode branch).
> Version 21.3 will expose problems sooner or later.
[...]
> Kenichi Handa fixed serveral issues recently.

So what you say is that I should wait until the next version of Emacs?
21.3.x (x=?) or 21.4? (I try to avoid CVS, I got enough with alpha and
beta software as it is... And I can wait if I know that it will be
fixed within reasonable time.) What is your educated guess at the
timeframe for such a release?

I made a screendump,
<URL:http://hem.bredband.net/steverud/Temp/screendump-2.jpg>

The file I try to find is in the ~/Rollspel/Äventyr/Idéer directory.

(BTW, is "still" the convention of <URL:xxx> the preferred way of
inserting URLs or should one use some other method - mostly people
just pastes it in, i.e. xxx, but some do <xxx>. I assume the former is
a MS Lookout habit.)

-- 
(         http://hem.bredband.net/steverud        !     Wei Wu Wei     )
(        Meaning of U2 Lyrics, Roleplaying        !  To Do Without Do  )

Re: Reading national characters in filenames on Mac OS X?

[Kai Großjohann]

Karl Eichwalder <keichwa@gmx.net> writes:

> kai.grossjohann@gmx.net (Kai Großjohann) writes:
>
>> (set-file-name-coding-system 'utf-8) should work, I'd think.
>
> This might work up to some degree.
>
> The crucial point is to use Emacs from CVS (trunk or unicode branch).
> Version 21.3 will expose problems sooner or later.

Which problems?  I thought that it would work as a first
approximation, especially if you mainly need to support Latin-1
characters in UTF-8 encoding.  Of course, trying CJK stuff with the
UTF-8 support in Emacs 21.3 won't work.

-- 
file-error; Data: (Opening input file no such file or directory ~/.signature)

Re: Reading national characters in filenames on Mac OS X?

[Kai Großjohann]

Jonas Steverud <tvrud-usenet@spray.se> writes:

> I made a screendump,
> <URL:http://hem.bredband.net/steverud/Temp/screendump-2.jpg>
>
> The file I try to find is in the ~/Rollspel/Äventyr/Idéer directory.

Okay, you see empty boxes.  That's good.  It means that Emacs
recognized the characters properly, and it's just that you don't have
a Unicode font.  Or the right Unicode font.

What does C-u C-x = say on such an empty box?

(Please don't forget the C-u.)
-- 
file-error; Data: (Opening input file no such file or directory ~/.signature)

Re: Reading national characters in filenames on Mac OS X?

[Karl Eichwalder]

kai.grossjohann@gmx.net (Kai Großjohann) writes:

> Which problems?

At least there are dired problems and it will not recognize the
encoding of XML files automatically.  Cut and paste issues are probably
fixed.  So you are right, 21.3 could work if you don't depend on
special features.

-- 
                                                         |      ,__o
http://www.gnu.franken.de/ke/                            |    _-\_<,
ke@suse.de (work) / keichwa@gmx.net (home)               |   (*)/'(*)

Re: Reading national characters in filenames on Mac OS X?

[Karl Eichwalder]

Jonas Steverud <tvrud-usenet@spray.se> writes:

> So what you say is that I should wait until the next version of Emacs?
> 21.3.x (x=?) or 21.4? (I try to avoid CVS, I got enough with alpha and
> beta software as it is...

Reasonable.  If you don't have the time/mood to follow the emacs-devel
mailing list for a few days one should not go for the CVS version.

Release dates are unkown to me (and the devolpers I guess).

> (BTW, is "still" the convention of <URL:xxx> the preferred way of
> inserting URLs or should one use some other method - mostly people
> just pastes it in, i.e. xxx, but some do <xxx>. I assume the former is
> a MS Lookout habit.)

<URL:xxx> is appropriate for Gnus; using this notation you can insert
white space ad libitum.

-- 
                                                         |      ,__o
http://www.gnu.franken.de/ke/                            |    _-\_<,
ke@suse.de (work) / keichwa@gmx.net (home)               |   (*)/'(*)

Re: Reading national characters in filenames on Mac OS X?

[Kai Großjohann]

Karl Eichwalder <keichwa@gmx.net> writes:

> kai.grossjohann@gmx.net (Kai Großjohann) writes:
>
>> Which problems?
>
> At least there are dired problems

Eek, that's bad.  Those might bite the OP.  What dired problems?

I recall a posting talking about a Latin-1 environment where just one
dir was UTF-8.  But that's not the case for the OP, I gather --
there, all file names are in UTF-8.

> and it will not recognize the encoding of XML files automatically.

That can be worked around with a coding cookie.

-- 
file-error; Data: (Opening input file no such file or directory ~/.signature)

Re: Reading national characters in filenames on Mac OS X?

[Piet van Oostrum]

>>>>> kai.grossjohann@gmx.net (Kai Großjohann) (KG) wrote:

KG> Karl Eichwalder <keichwa@gmx.net> writes:
>> kai.grossjohann@gmx.net (Kai Großjohann) writes:
>> 
>>> (set-file-name-coding-system 'utf-8) should work, I'd think.
>> 
>> This might work up to some degree.
>> 
>> The crucial point is to use Emacs from CVS (trunk or unicode branch).
>> Version 21.3 will expose problems sooner or later.

KG> Which problems?  I thought that it would work as a first
KG> approximation, especially if you mainly need to support Latin-1
KG> characters in UTF-8 encoding.  Of course, trying CJK stuff with the
KG> UTF-8 support in Emacs 21.3 won't work.

You can open and create the files and directories properly with the utf-8
file-name-coding. The only thing is dired will not display the filename
properly. This is because Mac OSX uses normalised UTF-8 for the filenames,
which means that the accent character is separated from the letter and put
behind it. And Emacs UTF-8 implementation doesn't understand this.
So Äventyr comes out as A?ventyr where the ? is actually the Unicode
character 0x0308:

COMBINING DIAERESIS
= double dot above, umlaut
= Greek dialytika
= double derivative

-- 
Piet van Oostrum <piet@cs.uu.nl>
URL: http://www.cs.uu.nl/~piet [PGP]
Private email: P.van.Oostrum@hccnet.nl

Re: Reading national characters in filenames on Mac OS X?

[Karl Eichwalder]

kai.grossjohann@gmx.net (Kai Großjohann) writes:

[[dired]]

> I recall a posting talking about a Latin-1 environment where just one
> dir was UTF-8.  But that's not the case for the OP, I gather --
> there, all file names are in UTF-8.

Okay, under those circumstances it should work :-)

> That can be worked around with a coding cookie.

Yes, or loading with C-x RET c utf-8 RET .

-- 
                                                         |      ,__o
http://www.gnu.franken.de/ke/                            |    _-\_<,
ke@suse.de (work) / keichwa@gmx.net (home)               |   (*)/'(*)

Re: Reading national characters in filenames on Mac OS X?

[Kai Großjohann]

Piet van Oostrum <piet@cs.uu.nl> writes:

> You can open and create the files and directories properly with the utf-8
> file-name-coding. The only thing is dired will not display the filename
> properly. This is because Mac OSX uses normalised UTF-8 for the filenames,
> which means that the accent character is separated from the letter and put
> behind it. And Emacs UTF-8 implementation doesn't understand this.

Ah, I didn't know what was meant by «normalized UTF-8».  It's a pity
that Emacs doesn't grok it.

-- 
file-error; Data: (Opening input file no such file or directory ~/.signature)

Re: Reading national characters in filenames on Mac OS X?

[Jonas Steverud]

kai.grossjohann@gmx.net (Kai Großjohann) writes:

> Jonas Steverud <tvrud-usenet@spray.se> writes:
>
>> I made a screendump,
>> <URL:http://hem.bredband.net/steverud/Temp/screendump-2.jpg>
>>
>> The file I try to find is in the ~/Rollspel/Äventyr/Idéer directory.
>
> Okay, you see empty boxes.  That's good.  It means that Emacs
> recognized the characters properly, and it's just that you don't have
> a Unicode font.  Or the right Unicode font.
>
> What does C-u C-x = say on such an empty box?

I put the cursor on the box after the first e in Ide[]er:

  character: ́ (01211301, 332481, 0x512c1)
    charset: mule-unicode-0100-24ff
	     (Unicode characters of the range U+0100..U+24FF.)
 code point: 37 65
     syntax: word
   category:
buffer code: 0x9C 0xF4 0xA5 0xC1
  file code: not encodable by coding system iso-latin-1
       font:
	     -B&H-LucidaTypewriter-Bold-R-Normal-Sans-14-140-75-75-M-90-ISO10646-1

~/> grep -i emacs .Xdefaults 
emacs.font: -b&h-lucidatypewriter-*-*-*-*-14-*-*-*-m-*-*-*
emacs*background:       Black
emacs*cursorColor:      White
emacs*foreground:       Moccasin
emacs*geometry:         80x50

-- 
(         http://hem.bredband.net/steverud        !     Wei Wu Wei     )
(        Meaning of U2 Lyrics, Roleplaying        !  To Do Without Do  )

Re: Reading national characters in filenames on Mac OS X?

[Jonas Steverud]

kai.grossjohann@gmx.net (Kai Großjohann) writes:

> Karl Eichwalder <keichwa@gmx.net> writes:
>
>> kai.grossjohann@gmx.net (Kai Großjohann) writes:
>>
>>> Which problems?
>>
>> At least there are dired problems
>
> Eek, that's bad.  Those might bite the OP.

Hopefully not, I do not use dired - the completing reading of
find-file is sufficent, and I use the GUI of the Mac. :-)

But other packages might use dired...

> I recall a posting talking about a Latin-1 environment where just one
> dir was UTF-8.  But that's not the case for the OP, I gather --
> there, all file names are in UTF-8.

I have not problems with files which paths doesn't contain any
umlauts, rings, accents or similar "decorations" in the name.

-- 
(         http://hem.bredband.net/steverud        !     Wei Wu Wei     )
(        Meaning of U2 Lyrics, Roleplaying        !  To Do Without Do  )

Re: Reading national characters in filenames on Mac OS X?

[Kai Großjohann]

Jonas Steverud <tvrud-usenet@spray.se> writes:

> I put the cursor on the box after the first e in Ide[]er:
>
>   character: ́ (01211301, 332481, 0x512c1)
>     charset: mule-unicode-0100-24ff
> 	     (Unicode characters of the range U+0100..U+24FF.)
>  code point: 37 65

My guess (from the glyph that I'm seeing): it's a combining character
that adds an accent to the previous character.  Minutes ago I learned
that Emacs doesn't support those.

Hmpf.

But I guess it ought to be possible to change Emacs to support them,
I just don't know how.
-- 
file-error; Data: (Opening input file no such file or directory ~/.signature)

Re: Reading national characters in filenames on Mac OS X?

[Jonas Steverud]

kai.grossjohann@gmx.net (Kai Großjohann) writes:

> Piet van Oostrum <piet@cs.uu.nl> writes:
>
>> You can open and create the files and directories properly with the utf-8
>> file-name-coding. The only thing is dired will not display the filename
>> properly. This is because Mac OSX uses normalised UTF-8 for the filenames,
>> which means that the accent character is separated from the letter and put
>> behind it. And Emacs UTF-8 implementation doesn't understand this.
>
> Ah, I didn't know what was meant by «normalized UTF-8».  It's a pity
> that Emacs doesn't grok it.

So, if I understand this correctly, I will be able to edit my
non-english files if I set file-coding-system to 'utf-8 and finds an
UTF-8 font.

Correct?

If so, how do I find such a font on my system? I checked xfontsel and
the "only" selections for "encdng" *, 0, 1, 2, ..., 10, 13, 14, 15,
dectech, fontspecific, irv, r, ru, standard and symbol.

(I don't know what half of them stands for. The numbers are ISO 8859,
but the others I do not know.)

Or are there a better/easier way of selection a "UTF-8 font" for Emacs?

I presume that the fonts that comes with Mac OS X are UTF-8 encoded,
right? If so, I assume I need to tell something something that Emacs
should be able to use this too, right?

Shouldn't I set terminal-coding-system (or whatever the variabels name
is) to something too? set-language-environment sets a lot of things.

/Jonas, a little bewildered and on unchartered territory - to him.
-- 
(         http://hem.bredband.net/steverud        !     Wei Wu Wei     )
(        Meaning of U2 Lyrics, Roleplaying        !  To Do Without Do  )

Re: Reading national characters in filenames on Mac OS X?

[Kai Großjohann]

Jonas Steverud <tvrud-usenet@spray.se> writes:

> kai.grossjohann@gmx.net (Kai Großjohann) writes:
>
>> Ah, I didn't know what was meant by «normalized UTF-8».  It's a pity
>> that Emacs doesn't grok it.
>
> So, if I understand this correctly, I will be able to edit my
> non-english files if I set file-coding-system to 'utf-8 and finds an
> UTF-8 font.
>
> Correct?

Well, if I understand correctly, ä is represented by the OS as two
characters.  First comes a, and then comes a special two-dot
character that combines with the preceding character to add the two
dots.

Emacs doesn't understand this kind of Unicode, I think.

It only understands the kind of Unicode where ä is represented by one
character.

> If so, how do I find such a font on my system? I checked xfontsel and
> the "only" selections for "encdng" *, 0, 1, 2, ..., 10, 13, 14, 15,
> dectech, fontspecific, irv, r, ru, standard and symbol.
>
> (I don't know what half of them stands for. The numbers are ISO 8859,
> but the others I do not know.)

Hm.  You use X11?  There is a package called unifont.

> Or are there a better/easier way of selection a "UTF-8 font" for Emacs?

I never tried that.

I installed the unifont package and the GNU intlfonts package.  And
that gave me lots of characters.  (Both packages contain X11 fonts.
Are you using X11?)  I installed the Debian packages.

> I presume that the fonts that comes with Mac OS X are UTF-8 encoded,
> right? If so, I assume I need to tell something something that Emacs
> should be able to use this too, right?

Not sure whether X11 Emacs can use Aqua fonts.

> Shouldn't I set terminal-coding-system (or whatever the variabels name
> is) to something too? set-language-environment sets a lot of things.

I think terminal-coding-system isn't relevant in a window system.

-- 
file-error; Data: (Opening input file no such file or directory ~/.signature)

Re: Reading national characters in filenames on Mac OS X?

[Piet van Oostrum]

>>>>> kai.grossjohann@gmx.net (Kai Großjohann) (KG) wrote:

KG> Piet van Oostrum <piet@cs.uu.nl> writes:
>> You can open and create the files and directories properly with the utf-8
>> file-name-coding. The only thing is dired will not display the filename
>> properly. This is because Mac OSX uses normalised UTF-8 for the filenames,
>> which means that the accent character is separated from the letter and put
>> behind it. And Emacs UTF-8 implementation doesn't understand this.

KG> Ah, I didn't know what was meant by «normalized UTF-8».  It's a pity
KG> that Emacs doesn't grok it.

I should have said:  Normalised Unicode rather than normalised UTF-8.
Although in the Mac filesystem of course each character arising from the
normalisation is encoded as UTF-8.

The Mac OSX system has functions to do the conversion in both directions
(for filenames), so it would in principle be possible to display filenames
properly. However, dired mode just puts the output of ls in the buffer, It
just assumes that ls outputs proper characters. So to have utf-8 recognise
the `decomposed' characters would be best.
-- 
Piet van Oostrum <piet@cs.uu.nl>
URL: http://www.cs.uu.nl/~piet [PGP]
Private email: P.van.Oostrum@hccnet.nl

Re: Reading national characters in filenames on Mac OS X?

[Jonas Steverud]

Piet van Oostrum <piet@cs.uu.nl> writes:

[...]
> The Mac OSX system has functions to do the conversion in both directions
> (for filenames), so it would in principle be possible to display filenames
> properly. 

Is this possible to do in Lisp? I assume it is in C, right?

If dired works or not is of lesser importance right now.

I can live with (setq file-name-coding-system 'utf-8) and then see
"Ide[]er" instead of "Idéer" but I would prefer, if possible off
course, to have Emacs to fully understand the filename and handle,
from the user's point of view, Idéer and Foobar equally.

-- 
(         http://hem.bredband.net/steverud        !     Wei Wu Wei     )
(        Meaning of U2 Lyrics, Roleplaying        !  To Do Without Do  )