Problem with UTF-8

Problem with UTF-8

[Konstantinos Efstathiou]

It seems that emacs can not display properly a text file with UTF-8 
encoding. Specifically, I do the following:

(1) Create a new buffer with UTF-8 encoding: C-x C-f test.txt
(2) Change the input method to greek: C-u C-\ greek
(3) Write some characters. Everything appears perfectly.
(4) Toggle the input method: C-\
(5) Save the file: C-x C-s
(6) Open the file: C-x C-f test.txt
Now all the greek characters appear as squares.

I am using GNU Emacs 21.3.50.1 (powerpc-apple-darwin7.0.0) under Mac OS 
X 10.3. When I open the file with TextEdit using the UTF-8 encoding and 
everything looks fine in there.

In my .emacs I have

(set-language-environment 'greek)
(set-default-coding-systems 'utf-8)
(prefer-coding-system 'utf-8)

Note also that if I set the proper variables for using iso-8859-7 
everything works great.

Re: Problem with UTF-8

[Oliver Scholz]

Konstantinos Efstathiou <cefstat@purple.univ-littoral.fr> writes:

> It seems that emacs can not display properly a text file with UTF-8 
> encoding. Specifically, I do the following:
>
> (1) Create a new buffer with UTF-8 encoding: C-x C-f test.txt
> (2) Change the input method to greek: C-u C-\ greek
> (3) Write some characters. Everything appears perfectly.
> (4) Toggle the input method: C-\
> (5) Save the file: C-x C-s
> (6) Open the file: C-x C-f test.txt
> Now all the greek characters appear as squares.
[...]

Please move the cursor to one of those characters that should appear
as a Greek char, but doesn't. Then press `C-u x =' and post the
output.

    Oliver
-- 
12 Frimaire an 212 de la Révolution
Liberté, Egalité, Fraternité!

Re: Problem with UTF-8

[Johan Bockgård]

Oliver Scholz <alkibiades@gmx.de> writes:

> Please move the cursor to one of those characters that should appear
> as a Greek char, but doesn't. Then press `C-u x =' and post the
> output.

Oliver meant `C-u C-x ='

-- 
Johan Bockgård

Re: Problem with UTF-8

[Jeffery B. Rancier]

bojohan+news@dd.chalmers.se (Johan Bockgård) writes:

> Oliver meant `C-u C-x ='

As a side question, how would I find out what function is bound to
that key sequence?  If I type M-x describe-key, followed by C-u, all I
get is the definition of the command universal-argument.
-- 
Thanks,
Jeff

Re: Problem with UTF-8

[Eli Zaretskii]

> From: jeff.rancier@softechnics.com (Jeffery B. Rancier)
> Date: Tue, 02 Dec 2003 12:15:45 -0500
> 
> > Oliver meant `C-u C-x ='
> 
> As a side question, how would I find out what function is bound to
> that key sequence?  If I type M-x describe-key, followed by C-u, all I
> get is the definition of the command universal-argument.

You should type "M-x describe-key RET C-x =", and in the doc string
look for the description of how a numeric argument changes the
behavior of that key.  C-u simply supplies a numeric argument to the
command invoked immediately after it.  See the node "Arguments" in the
Emacs manual for more details.

Re: Problem with UTF-8

[Oliver Scholz]

jeff.rancier@softechnics.com (Jeffery B. Rancier) writes:

> bojohan+news@dd.chalmers.se (Johan Bockgård) writes:
>
>> Oliver meant `C-u C-x ='
>
> As a side question, how would I find out what function is bound to
> that key sequence?  If I type M-x describe-key, followed by C-u, all I
> get is the definition of the command universal-argument.
[...]

The function is bound to `C-x ='. If you hit `C-u', Emacs passes a
prefix to this function (i.e. a specific argument specified by the
function's `interactive' declaration is non-nil). The help for that
function (`C-h k C-x =' in this case) should tell you what it does for
which kind of prefixes (if any).

    Oliver
-- 
12 Frimaire an 212 de la Révolution
Liberté, Egalité, Fraternité!

Re: Problem with UTF-8

[Konstantinos Efstathiou]

When writing the file:

   character: ^[[alpha] (05541, 2913, 0xb61, U+03B1)^[-A
     charset: greek-iso8859-7
	     (Right-Hand Part of Latin/Greek Alphabet (ISO/IEC 8859-7): 
ISO-IR-126.)
  code point: 97
      syntax: w 	which means: word
    category: g:Greek
buffer code: 0x86 0xE1
   file code: 0xCE 0xB1 (encoded by coding system mule-utf-8)
     display: by this font (glyph code)
      -etl-fixed-medium-r-normal--14-140-75-75-m-140-iso8859-7 (0xE1)

After saving and reopening the file:

   character: ^[[square] (01211661, 332721, 0x513b1, U+03B1)^[-A
     charset: mule-unicode-0100-24ff
	     (Unicode characters of the range U+0100..U+24FF.)
  code point: 39 49
      syntax: w 	which means: word
    category: g:Greek
buffer code: 0x9C 0xF4 0xA7 0xB1
   file code: 0xCE 0xB1 (encoded by coding system mule-utf-8-unix)
     display: no font avairable

Regards

Re: Problem with UTF-8

[Oliver Scholz]

Konstantinos Efstathiou <cefstat@purple.univ-littoral.fr> writes:

> When writing the file:
>
>    character: .[alpha] (05541, 2913, 0xb61, U+03B1).-A
>      charset: greek-iso8859-7
[...]
>      display: by this font (glyph code)
>       -etl-fixed-medium-r-normal--14-140-75-75-m-140-iso8859-7 (0xE1)
>
> After saving and reopening the file:
>
>    character: .[square] (01211661, 332721, 0x513b1, U+03B1).-A
>      charset: mule-unicode-0100-24ff
[...]
>      display: no font avairable

I see. It is a font problem. You need to create a fontset in order to
solve it. Look for an appropriate ISO 10646-1 font and set Emacs up to
use it for the charset mule-unicode-0100-24ff. For example this should
work on GNU/Linux:

(create-fontset-from-fontset-spec "\ 
-b&h-lucidatypewriter-medium-r-*-*-14-*-*-*-*-*-fontset-konstantinos,\ 
latin-iso8859-15:-b&h-lucidatypewriter-medium-r-*-*-14-*-*-*-*-*-iso8859-1,\ 
mule-unicode-0100-24ff:-*-fixed-medium-r-*-*-14-*-*-*-*-*-iso10646-1") 

(set-face-font 'default "fontset-konstantinos")

Maybe there is a ISO 10646 variant of the ETL font that you are using?

Some people prefer to do it via X-ressources, but I don't use that and
I am too lazy to look the syntax up in the manual.

What is happening? Emacs internal encoding (emacs-mule) is divided
into several so-called "charsets". Every character in a buffer belongs
to a charset and some charsets contain the same (abstract)
characters. The Greek input method you use obviously inserts chars
from `greek-iso8859-7'. (Some or all (?) of the input methods for
classical Greek, for example, use `mule-unicode-0100-24ff'.) Font
resolving and similar things depend on those charsets. When you save
the file as UTF-8, then Emacs knows how to generate the appropriate
UTF-8 octets for the charset `greek-iso8859-7'. But when you visit it
again, then Emacs does not decode those UTF-8 octets to characters
belonging to `greek-iso8859-7', but to chars from
`mule-unicode-0100-24ff'. This is actually a good thing. It's just a
font issue that you don't see the right glyphs for those characters.


    Oliver
-- 
13 Frimaire an 212 de la Révolution
Liberté, Egalité, Fraternité!

Re: Problem with UTF-8

[Konstantinos Efstathiou]

Oliver Scholz writes:

> I see. It is a font problem. You need to create a fontset in order to
> solve it. Look for an appropriate ISO 10646-1 font and set Emacs up to
> use it for the charset mule-unicode-0100-24ff.
> ...
> It's just a
> font issue that you don't see the right glyphs for those characters.

I appreciate the reply and the detailed explanation. Unfortunately, 
running the Carbon version of Emacs under Mac OS X does not help. The 
problem is that for some reason Emacs/Carbon can not see any iso10646-1 
fonts. Although I have fixed width fonts like 'andale mono' which 
support WGL4 and which I have used before under Linux with encoding 
iso10646-1, Emacs says that they are available only with encoding 
mac-roman. Also the etl-* fonts which are the GNU intlfonts converted 
for use with Macs do not have the iso10646-1 encoding (but at least 
they have the iso8859-7 encoding). So basically I do not know how to 
proceed from here using the Carbon Emacs. In principle I can run the 
X11 version of Emacs but I prefer the Carbon version for many reasons.

Regards

Re: Problem with UTF-8

[Oliver Scholz]

Konstantinos Efstathiou <cefstat@purple.univ-littoral.fr> writes:

[...]
> So basically I do not know how to proceed from here using the Carbon
> Emacs. In principle I can run the X11 version of Emacs but I prefer
> the Carbon version for many reasons.
[...]

Uh, sorry. I don't know anything about Mac OS. So I can't help you
here. Basically you just need to find a font encoded in ISO 10646
which contains the necessary characters from the "Greek and Coptic"
repertoire.

But I don't know how to find such a font on Mac OS.

    Oliver
-- 
15 Frimaire an 212 de la Révolution
Liberté, Egalité, Fraternité!