MULE shows gibberish; now what?

MULE shows gibberish; now what?

[Ilya Zakharevich]

Given the amount of bad press MULE gets (AFAIU, correctly), I never
tried to actully use it.  Well, now I did.  And it does not work
(21.2.2 -no-init-file).

Is there a way for a mortal to understand/fix MULE bugs?

SPECIFICS of the problem1: when asked to show cyrillic, it shows
cyrillic indeed.  However, the shown glyphs have no relationship to
the actual Cyrillic text (e.g., in the Hello, World example).  Do not
have a slightest idea how to report it in more details...

Ilya

Re: MULE shows gibberish; now what?

[Kai Großjohann]

Ilya Zakharevich <nospam-abuse@ilyaz.org> writes:

> SPECIFICS of the problem1: when asked to show cyrillic, it shows
> cyrillic indeed.  However, the shown glyphs have no relationship to
> the actual Cyrillic text (e.g., in the Hello, World example).  Do not
> have a slightest idea how to report it in more details...

IIRC, there are two popular encodings for Cyrillic, and maybe Emacs
thinks it's using a font in the foo encoding but the real font is in
the bar encoding?

Put point on a Cyrillic character and type C-u C-x =, this tells you
stuff, including the charset that Emacs thinks this character is in.

kai
-- 
~/.signature is: umop 3p!sdn    (Frank Nobis)

Re: MULE shows gibberish; now what?

[Peter J. Acklam]

grossjoh@ls6.informatik.uni-dortmund.de (Kai Großjohann) wrote:

> Ilya Zakharevich <nospam-abuse@ilyaz.org> writes:
> 
> > SPECIFICS of the problem1: when asked to show cyrillic, it
> > shows cyrillic indeed.  However, the shown glyphs have no
> > relationship to the actual Cyrillic text (e.g., in the Hello,
> > World example).  Do not have a slightest idea how to report it
> > in more details...
> 
> IIRC, there are two popular encodings for Cyrillic, and maybe
> Emacs thinks it's using a font in the foo encoding but the real
> font is in the bar encoding?
> 
> Put point on a Cyrillic character and type C-u C-x =, this tells
> you stuff, including the charset that Emacs thinks this
> character is in.

Perhaps you also know how I can make Emacs open a UTF-8-encoded
file correctly?  In my ~/.emacs I now enabled UTF-8 in every way I
can think of

   (set-buffer-file-coding-system 'utf-8-unix)
   (set-clipboard-coding-system 'utf-8-unix)
   (set-keyboard-coding-system 'utf-8-unix)
   (set-next-selection-coding-system 'utf-8-unix)
   (set-selection-coding-system 'utf-8-unix)     
   (set-terminal-coding-system 'utf-8-unix)      
   (set-w32-system-coding-system 'utf-8-unix)    
   (prefer-coding-system 'utf-8-unix)

and I have enabled multibyte characters, but still Emacs always
opens a UTF-8 file as "unibyte no-conversion".  Using

   C-x RET c utf-8 RET C-x C-f

to open a file is rather awkward.

Peter

-- 
No electrons used in the production of this message were harmed or
mistreated in any manner.

Re: MULE shows gibberish; now what?

[Kai Großjohann]

pjacklam@online.no (Peter J. Acklam) writes:

> Perhaps you also know how I can make Emacs open a UTF-8-encoded
> file correctly?

Maybe it helps to install Mule-UCS.

kai
-- 
~/.signature is: umop 3p!sdn    (Frank Nobis)

Re: MULE shows gibberish; now what?

[Peter J. Acklam]

Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Großjohann) wrote:

> pjacklam@online.no (Peter J. Acklam) writes:
> 
> > Perhaps you also know how I can make Emacs open a
> > UTF-8-encoded file correctly?
> 
> Maybe it helps to install Mule-UCS.

What is Mule-UCS?  I have installed Leim, but I find no references
to Mule-UCS anywhere...

Peter

-- 
No electrons used in the production of this message were harmed or
mistreated in any manner.

Re: MULE shows gibberish; now what?

[Kai Großjohann]

pjacklam@online.no (Peter J. Acklam) writes:

> What is Mule-UCS?  I have installed Leim, but I find no references
> to Mule-UCS anywhere...

Google will find it.  Mule-UCS is an add-on package, available from
m17n.org I believe.  It was written before Emacs got Unicode support,
and Mule-UCS supports CJK whereas the Unicode support in Emacs
doesn't, currently.

kai
-- 
~/.signature is: umop ap!sdn    (Frank Nobis)

Re: MULE shows gibberish; now what?

[Kai Großjohann]

pjacklam@online.no (Peter J. Acklam) writes:

> I thought CJK was some Chinese stuff, but all I want it to get
> Emacs to open a simple text file with European utf-8 text.

Emacs might not auto-detect UTF-8 out of the box.  Does it work if
you say C-x RET c utf-8 RET C-x C-f /path/to/the/file RET?

If it is important, I'm sure it's possible to find out how to improve
UTF-8 autodetection in Emacs.

kai
-- 
~/.signature is: umop ap!sdn    (Frank Nobis)

Re: MULE shows gibberish; now what?

[Peter J. Acklam]

Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Großjohann) wrote:

> pjacklam@online.no (Peter J. Acklam) writes:
> 
> > I thought CJK was some Chinese stuff, but all I want it to get
> > Emacs to open a simple text file with European utf-8 text.
> 
> Emacs might not auto-detect UTF-8 out of the box.  Does it work if
> you say C-x RET c utf-8 RET C-x C-f /path/to/the/file RET?

Yes, as I mentioned in an earlier posting in this thread.  :-)
 
> If it is important, I'm sure it's possible to find out how to
> improve UTF-8 autodetection in Emacs.

It does surprise me that Emacs does not open the file in UTF-8
when I have specified that my preference is UTF-8.  There are
ambigueties when detecting the coding system used in a text file,
but when I have set the preference to UTF-8, I think Emacs should
try that first.

Peter

-- 
No electrons used in the production of this message were harmed or
mistreated in any manner.

Re: MULE shows gibberish; now what?

[Kai Großjohann]

pjacklam@online.no (Peter J. Acklam) writes:

> Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Großjohann) wrote:
>
>> pjacklam@online.no (Peter J. Acklam) writes:
>> 
>> > I thought CJK was some Chinese stuff, but all I want it to get
>> > Emacs to open a simple text file with European utf-8 text.
>> 
>> Emacs might not auto-detect UTF-8 out of the box.  Does it work if
>> you say C-x RET c utf-8 RET C-x C-f /path/to/the/file RET?
>
> Yes, as I mentioned in an earlier posting in this thread.  :-)

I must have missed that...

>> If it is important, I'm sure it's possible to find out how to
>> improve UTF-8 autodetection in Emacs.
>
> It does surprise me that Emacs does not open the file in UTF-8
> when I have specified that my preference is UTF-8.  There are
> ambigueties when detecting the coding system used in a text file,
> but when I have set the preference to UTF-8, I think Emacs should
> try that first.

Hm.  What is the value of the variable coding-category-list?  Is
coding-category-utf-8 mentioned in that list?  Maybe it helps to add
this near the beginning of the list?

kai
-- 
~/.signature is: umop ap!sdn    (Frank Nobis)

Re: MULE shows gibberish; now what?

[Peter J. Acklam]

Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Großjohann) wrote:

> What is the value of the variable coding-category-list?  Is
> coding-category-utf-8 mentioned in that list?  Maybe it helps to
> add this near the beginning of the list?

Actually, coding-category-utf-8 is the last element in the list,
but placing it first, or even letting coding-category-utf-8 be the
only element in coding-category-list doesn't help.  The buffer
still ends up as "Unibyte no-conversion buffer".

Placing "-*- coding: iso-8859-1; -*-" at the top of the file does
the trick, but I'd prefer it to happen automatically.

Peter

-- 
No electrons used in the production of this message were harmed or
mistreated in any manner.

Re: MULE shows gibberish; now what?

[Kai Großjohann]

pjacklam@online.no (Peter J. Acklam) writes:

> Placing "-*- coding: iso-8859-1; -*-" at the top of the file does
> the trick, but I'd prefer it to happen automatically.

So your file is NOT in utf-8 after all???

kai
-- 
~/.signature is: umop ap!sdn    (Frank Nobis)

Re: MULE shows gibberish; now what?

[Peter J. Acklam]

Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Großjohann) wrote:

> pjacklam@online.no (Peter J. Acklam) writes:
> 
> > Placing "-*- coding: iso-8859-1; -*-" at the top of the file
> > does the trick, but I'd prefer it to happen automatically.
> 
> So your file is NOT in utf-8 after all???

Good grief, that should be "-*- coding: utf-8; -*-".  I cut and
pasted from the wrong file.  Sorry.

Peter

-- 
No electrons used in the production of this message were harmed or
mistreated in any manner.

Re: MULE shows gibberish; now what?

[Kai Großjohann]

pjacklam@online.no (Peter J. Acklam) writes:

> Good grief, that should be "-*- coding: utf-8; -*-".

I wasn't sure.  I hope I didn't offend you.

kai
-- 
~/.signature is: umop ap!sdn    (Frank Nobis)

Re: MULE shows gibberish; now what?

[Peter J. Acklam]

Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Großjohann) wrote:
>
> pjacklam@online.no (Peter J. Acklam) writes:
> 
> > Good grief, that should be "-*- coding: utf-8; -*-".
> 
> I wasn't sure.  I hope I didn't offend you.

No problem, Kai.  I realize now that

   (set-input-method "latin-1-prefix")

on my W2K box gives me

   activate-input-method: Can't activate input method `latin-1-prefix'

but it works fine on Solaris.

Perhaps there is something seriously wrong with my W2K
installation or Emacs build.  I'll have to look closer
into this, and see if a reinstall or rebuild (recompile)
will solve the problem.

Peter

Re: MULE shows gibberish; now what?

[Stefan Monnier <foo@acm.com>]

> Actually, coding-category-utf-8 is the last element in the list,
> but placing it first, or even letting coding-category-utf-8 be the
> only element in coding-category-list doesn't help.  The buffer
> still ends up as "Unibyte no-conversion buffer".

`coding-category-list' is misleading: it should probably be read-only,
because it should only be modified via functions like `prefer-coding-system'
or `set-coding-priority' (which update internal data that's then reflected
in `coding-category-list').

I use (prefer-coding-system 'utf-8) and it works.  I also tried
(prefer-coding-system 'utf-8-unix) and it worked as well, so there
must be something else at play in your .emacs.

Note that setting LANG to <foo>.UTF-8 works as well and should
be the preferred way to set those kinds of things, although I seem
to remember that Emacs-21.[12] specifically ignores it because the
utf-8 support is considered too poor in those versions.


        Stefan

Re: MULE shows gibberish; now what?

[Miles Bader]

"Stefan Monnier <foo@acm.com>" <monnier+gnu.emacs.help/news/@flint.cs.yale.edu> writes:
> Note that setting LANG to <foo>.UTF-8 works as well and should
> be the preferred way to set those kinds of things, although I seem
> to remember that Emacs-21.[12] specifically ignores it because the
> utf-8 support is considered too poor in those versions.

... but there should be no settings which can _only_ be done via the
LANG environment variable ('cause restarting emacs is a _bad_ thing!).

-Miles
-- 
"1971 pickup truck; will trade for guns"

Re: MULE shows gibberish; now what?

[Stefan Monnier <foo@acm.com>]

>> Note that setting LANG to <foo>.UTF-8 works as well and should
>> be the preferred way to set those kinds of things, although I seem
>> to remember that Emacs-21.[12] specifically ignores it because the
>> utf-8 support is considered too poor in those versions.
> ... but there should be no settings which can _only_ be done via the
> LANG environment variable ('cause restarting emacs is a _bad_ thing!).

Of course.  But LANG should be set anyway for other applications...


        Stefan

Re: MULE shows gibberish; now what?

[Kai Großjohann]

pjacklam@online.no (Peter J. Acklam) writes:

> Actually, coding-category-utf-8 is the last element in the list,
> but placing it first, or even letting coding-category-utf-8 be the
> only element in coding-category-list doesn't help.  The buffer
> still ends up as "Unibyte no-conversion buffer".

I'm no Mule guru.  I'm sorry to say that I'm at the end of my rope
now.  I know that this behavior was intentional, though: the Emacs
21.1 and 21.2 support for UTF-8 is still less than perfect, so UTF-8
was deliberately left out of autodetection.  I don't know what needs
to be done to enable it.

[time passes]

Aha, the CVS version of Emacs also does not auto-detect UTF-8
properly (tested with Markus Kuhn's UTF-8-demo.txt).

An alternative to the coding cookies is file-coding-system-alist.

kai
-- 
~/.signature is: umop ap!sdn    (Frank Nobis)

Re: MULE shows gibberish; now what?

[Clemens Fischer]

Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Großjohann) writes:

> Emacs might not auto-detect UTF-8 out of the box.  Does it work if
> you say C-x RET c utf-8 RET C-x C-f /path/to/the/file RET?
>
> If it is important, I'm sure it's possible to find out how to improve
> UTF-8 autodetection in Emacs.

oh, i'd say this is very important.  for example, i get GPG encoded
messages where the same funny characters are displayed the OP talked
about (one funny character plus the normal character one expects to
see).  this may well be a defficiency in GPG (mailcrypt in my case):
i get the cryptogram from an emacs in one characterset, which gets
lost in transmission, and gets decoded 1:1 from the perspective of the
sender.  which turns out to be wrong on my receiving side.

but actually, i didn't understand a thing what you and hilarious ilya
talked about.  what's a glyph other than what a character looks like,
anyway?  isn't it a picture of a character?

clemens

Re: MULE shows gibberish; now what?

[Kai Großjohann]

Clemens Fischer <ino@despammed.com> writes:

> Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Grojohann) writes:
>
>> Emacs might not auto-detect UTF-8 out of the box.  Does it work if
>> you say C-x RET c utf-8 RET C-x C-f /path/to/the/file RET?
>>
>> If it is important, I'm sure it's possible to find out how to improve
>> UTF-8 autodetection in Emacs.
>
> oh, i'd say this is very important.  for example, i get GPG encoded
> messages where the same funny characters are displayed the OP talked
> about (one funny character plus the normal character one expects to
> see).

Is the funny char \201?  Then it's another problem, not related to
UTF-8.

> this may well be a defficiency in GPG (mailcrypt in my case):  i get
> the cryptogram from an emacs in one characterset, which gets lost in
> transmission, and gets decoded 1:1 from the perspective of the
> sender.  which turns out to be wrong on my receiving side.

Hm.  Are you using the old non-MIME method of using GPG?  If you use
MIME, then the newly committed (to Gnus CVS) PGG might be useful.

> but actually, i didn't understand a thing what you and hilarious ilya
> talked about.  what's a glyph other than what a character looks like,
> anyway?  isn't it a picture of a character?

Ilya expected that Emacs can display a character using a glyph from
another character set.  Hm.  How to explain...  Suppose you have a
Latin-9 document but you don't have a Latin-9 font, only a Latin-1
font.  Then you would like to display the German umlauts using the
Latin-1 fonts.  But Emacs can't do that.  It associates a character
set with each character, and it only uses a font for that character
set for displaying the character.

In Ilya's case, there are different encodings for Cyrillic, and it
would be useful to display a file encoded in iso-8859-5 with a font
that's encoded in KOI-R, say.

So this would amount to using glyph number 42 (from the KOI-R font)
to display character number 27 (from the iso-8859-5 charset).

kai
-- 
~/.signature is: umop ap!sdn    (Frank Nobis)

Re: MULE shows gibberish; now what?

[Kai Großjohann]

Ilya Zakharevich <nospam-abuse@ilyaz.org> writes:

> <grossjoh@ls6.informatik.uni-dortmund.de>], who wrote in article <vaf65wv3hjr.fsf@lucy.cs.uni-dortmund.de>:
>> Ilya Zakharevich <nospam-abuse@ilyaz.org> writes:
>> 
> , and maybe Emacs
>> thinks it's using a font in the foo encoding but the real font is in
>> the bar encoding?
>
> Of course it does, what else could it be!  But how would I detect it
> and/or fix it?

With C-u C-x =, as I said.

>> Put point on a Cyrillic character and type C-u C-x =, this tells you
>> stuff, including the charset that Emacs thinks this character is in.
>
> Hmm?  You mean that doing "Hello world" demo, one gets different
> results of your test on different machines?  Now I'm completely
> confused...

Different people might use different fonts on different machines.  In
that sense, the results could be different.  I don't know anything
about Cyrillic, so let me give a Latin example.  Character number 164
is the Euro sign in Latin-9 and the Currency sign in Latin-1.

If I now set the X11 resource Emacs.font to a Latin-9 font, then
Emacs will display Latin-1 characters with that font!  This means, if
the HELLO file includes that currency sign, I would wrongly see a euro
sign, but C-u C-x = would tell me that Emacs thinks it is displaying
a Latin-1 character.

This is a misconfiguration of Emacs, and can be corrected.

Maybe you have a similar misconfiguration of Emacs.

(It is also arguably a bug since Emacs should know from the font name
that it is not a Latin-1 font.  But not all fonts specify
"iso8859-15" in their names, so it is difficult to know.)

kai
-- 
~/.signature is: umop 3p!sdn    (Frank Nobis)

Re: MULE shows gibberish; now what?

[Kai Großjohann]

Ilya Zakharevich <nospam-abuse@ilyaz.org> writes:

> [A complimentary Cc of this posting was sent to
> Kai
>  =?iso-8859-15?q?Gro=DFjohann?=
> <Kai.Grossjohann@CS.Uni-Dortmund.DE>], who wrote in article <vafznu6yx9x.fsf@lucy.cs.uni-dortmund.de>:
>> > Of course it does, what else could it be!  But how would I detect it
>> > and/or fix it?
>> 
>> With C-u C-x =, as I said.
>
> Would it show the font and the codepoint chosen in the font?  [I'm not
> on X, so cannot check myself]

Here is an example output of C-u C-x =.   I think it will clear up a
lot of things.

/----
|   character: € (07444, 3876, 0xf24)
|     charset: latin-iso8859-15
|              (Right-Hand Part of Latin Alphabet 9 (ISO/IEC 8859-15): ISO-IR-203.)
|  code point: 36
|      syntax: w 	which means: word
|    category: l:Latin  
| buffer code: 0x8E 0xA4
|   file code: 0x8E 0xA4 (encoded by coding system emacs-mule)
|     Unicode: 20AC
|        font: -Misc-Fixed-Medium-R-Normal--14-130-75-75-C-70-ISO8859-15
| 
| There are text properties here:
|   fontified            t
\----

In the above output, it is easy to see if the glyph matches what it
should be.  I can do "man iso-8859-15" to look up what character has
code 128+36, and then I can compare it with the glyph.  And if the
"man" command told me to expect a Euro sign, but I'm seeing a
currency sign, then I know that something is wrong.

>> >> Put point on a Cyrillic character and type C-u C-x =, this tells you
>> >> stuff, including the charset that Emacs thinks this character is in.
>> >
>> > Hmm?  You mean that doing "Hello world" demo, one gets different
>> > results of your test on different machines?  Now I'm completely
>> > confused...
>
>> If I now set the X11 resource Emacs.font to a Latin-9 font, then
>> Emacs will display Latin-1 characters with that font!  This means, if
>> the HELLO file includes that currency sign, I would wrongly see a euro
>> sign, but C-u C-x = would tell me that Emacs thinks it is displaying
>> a Latin-1 character.
>
> What you say is: the "hello world" example will have the same result
> of C-u C-x = on all the machines.  So I do not see how it would help me...

No, I did NOT say that the output of C-u C-x = is the same on all
machines.  What I said is that C-u C-x = tells you which character
Emacs thinks it is displaying.

As you can easily see, the font name can be different.  And the glyph
(directly after "character: ") is of course also different.

Other parts of the output are system-independent, obviously.

>> This is a misconfiguration of Emacs, and can be corrected.
>> 
>> Maybe you have a similar misconfiguration of Emacs.
>> 
>> (It is also arguably a bug since Emacs should know from the font name
>> that it is not a Latin-1 font.  But not all fonts specify
>> "iso8859-15" in their names, so it is difficult to know.)
>
> You are discussing a default font; which is not under the control of
> Emacs.  I'm discussing Cyrillic font, which is not under *my* control
> (as far as I know now).

You can tell Emacs to use the foo font for the bar encoding.  Maybe
you have (accidentally) told Emacs to use a KOI-R font for the
iso-8859-5 encoding, or something like this.  This would explain the
strange output.  And if you use C-u C-x = you can immediately see if
this is the case.

If all else fails, you can of course also install fonts in your home
dir and tell X11 about them, and then tell Emacs to use one of those...

kai
-- 
~/.signature is: umop ap!sdn    (Frank Nobis)

Re: MULE shows gibberish; now what?

[Oliver Scholz]

Ilya Zakharevich <nospam-abuse@ilyaz.org> writes:

> [A complimentary Cc of this posting was sent to
> Kai
>  =?iso-8859-15?q?Gro=DFjohann?=
> <Kai.Grossjohann@CS.Uni-Dortmund.DE>], who wrote in article <vaf1y7huplb.fsf@lucy.cs.uni-dortmund.de>:
>> /----
>> |   character: ¤ (07444, 3876, 0xf24)
>> |     charset: latin-iso8859-15
>> |              (Right-Hand Part of Latin Alphabet 9 (ISO/IEC 8859-15): ISO-IR-203.)
>> |  code point: 36
>> |      syntax: w 	which means: word
>> |    category: l:Latin  
>> | buffer code: 0x8E 0xA4
>> |   file code: 0x8E 0xA4 (encoded by coding system emacs-mule)
>> |     Unicode: 20AC
>> |        font: -Misc-Fixed-Medium-R-Normal--14-130-75-75-C-70-ISO8859-15
>
> So it does not show which codepoint inside the font is chosen!

??? 

Have you actually read what Kai quoted? What information are you
missing exactly?

>> | There are text properties here:
>> |   fontified            t
>> \----
>> 
>> In the above output, it is easy to see if the glyph matches what it
>> should be.  I can do "man iso-8859-15" to look up what character has
>> code 128+36, and then I can compare it with the glyph.  And if the
>> "man" command told me to expect a Euro sign, but I'm seeing a
>> currency sign, then I know that something is wrong.
>
> As I said, I *know* that the output is wrong.  So how would what you
> say help me to debug/fix the problem.

Because the first step to solve a problem is to analyze what the
problem actually is. "It is wrong" is not specific enough.

[ You could of course write a program that writes random characters
in your .emacs and try if any of the character sequences thus
produced makes up for the right configuration. But personally I
prefer a proper analysis of the problem. ]

If I look at it from this point of view, it actually was very kind of
you that you told us that your problem has something to do with fonts
and encodings. You could as well have said:

"My Emacs does not work correctly." 

"What exactly goes wrong?"

"I *know* that it does not work correctly. So how would the answer to
that question help me do fix the problem?"

[...]
>>  Maybe
>> you have (accidentally) told Emacs to use a KOI-R font for the
>> iso-8859-5 encoding, or something like this.
>
> With -no-init-file?  I doubt it.  ;-)

If you already know everything, why do you ask here?

Nobody said that the misconfiguration takes places in
site-start.el. It could be in your .emacs or in your .Xdefaults, in
your .Xressources, even in /etc/X11/app-defaults/Emacs. Or your OS is
misconfigured? Who knows? Why don't you simply hit `C-u C-x =' and
show us the output instead of making a whole thread about arguing if
it is possible that this could be useful? We have no crystal ball
here, to make a remote diagnosis. To help you, we need your help and
that you apply the diagnostic tools that come with Emacs. If you are
not willing to cooperate, there is no way to help you.

Perhaps Kai is on the wrong track? Perhaps the output of `C-u C-x ='
does not reveal anything useful? Who knows? Why don't you simply show
us the output and let us see ourselves?

    -- Oliver

-- 
6 Vendémiaire an 211 de la Révolution
Liberté, Egalité, Fraternité!

Re: MULE shows gibberish; now what?

[Kai Großjohann]

Ilya Zakharevich <nospam-abuse@ilyaz.org> writes:

> [A complimentary Cc of this posting was sent to
> Kai
>  =?iso-8859-15?q?Gro=DFjohann?=
> <Kai.Grossjohann@CS.Uni-Dortmund.DE>], who wrote in article <vaf1y7huplb.fsf@lucy.cs.uni-dortmund.de>:
>> /----
>> |   character: ¤ (07444, 3876, 0xf24)
>> |     charset: latin-iso8859-15
>> |              (Right-Hand Part of Latin Alphabet 9 (ISO/IEC 8859-15): ISO-IR-203.)
>> |  code point: 36
>> |      syntax: w 	which means: word
>> |    category: l:Latin  
>> | buffer code: 0x8E 0xA4
>> |   file code: 0x8E 0xA4 (encoded by coding system emacs-mule)
>> |     Unicode: 20AC
>> |        font: -Misc-Fixed-Medium-R-Normal--14-130-75-75-C-70-ISO8859-15
>
> So it does not show which codepoint inside the font is chosen!

You see "code point: 36", and you see that it is the right-hand part
of the alphabet only.  So 128+36=164 is the actual code point you want.

>> | There are text properties here:
>> |   fontified            t
>> \----
>> 
>> In the above output, it is easy to see if the glyph matches what it
>> should be.  I can do "man iso-8859-15" to look up what character has
>> code 128+36, and then I can compare it with the glyph.  And if the
>> "man" command told me to expect a Euro sign, but I'm seeing a
>> currency sign, then I know that something is wrong.
>
> As I said, I *know* that the output is wrong.  So how would what you
> say help me to debug/fix the problem.

You can look at the font name, for example.  And you can look at the
charset and compare that with your font, to see whether they match.

>> As you can easily see, the font name can be different.  And the glyph
>> (directly after "character: ") is of course also different.
>
> Is it the same glyph as in the buffer?  So what is the point of
> looking at *that*?  ;-)

Argh.

>> You can tell Emacs to use the foo font for the bar encoding.
>
> How?  And how do I tell Emacs in which encoding this font actually is?

You can't.  But you can do it the other way round: you can tell Emacs
to use the foo font for the bar encoding.  Investigate fontsets in
the Emacs documentation.

>> Maybe
>> you have (accidentally) told Emacs to use a KOI-R font for the
>> iso-8859-5 encoding, or something like this.
>
> With -no-init-file?  I doubt it.  ;-)

With a X11 resource, for instance.  You probably set the font for
your X11 apps somewhere (otherwise it would be "fixed").

>> If all else fails, you can of course also install fonts in your home
>> dir and tell X11 about them, and then tell Emacs to use one of those...
>
> I have seen machines where X server can't see user directories.

Oh, boy.

kai
-- 
~/.signature is: umop ap!sdn    (Frank Nobis)

Re: MULE shows gibberish; now what?

[Kai Großjohann]

Ilya Zakharevich <nospam-abuse@ilyaz.org> writes:

> [A complimentary Cc of this posting was sent to
> Kai
>  =?iso-8859-15?q?Gro=DFjohann?=
> <Kai.Grossjohann@CS.Uni-Dortmund.DE>], who wrote in article <vaffzvvzm51.fsf@lucy.cs.uni-dortmund.de>:
>> >> |   character: ¤ (07444, 3876, 0xf24)
>> >> |     charset: latin-iso8859-15
>> >> |              (Right-Hand Part of Latin Alphabet 9 (ISO/IEC 8859-15): ISO-IR-203.)
>> >> |  code point: 36
>> >> |      syntax: w 	which means: word
>> >> |    category: l:Latin  
>> >> | buffer code: 0x8E 0xA4
>> >> |   file code: 0x8E 0xA4 (encoded by coding system emacs-mule)
>> >> |     Unicode: 20AC
>> >> |        font: -Misc-Fixed-Medium-R-Normal--14-130-75-75-C-70-ISO8859-15
>> >
>> > So it does not show which codepoint inside the font is chosen!
>> 
>> You see "code point: 36", and you see that it is the right-hand part
>> of the alphabet only.  So 128+36=164 is the actual code point you want.
>
> *I* do not want any *codepoint*.  What I want is that CYRILLIC CAPITAL R
> is shown with a glyph which looks like `P', not like `A'.

You said it doesn't show which codepoint is used, I showed you that
you can compute the codepoint from what it shows.  That's all.

I'm sorry if there was a misunderstanding.  I understood your "So it
does not show which codepoint ... is chosen!" as a question.  And I
said "...you want" instead of "...you wanted to know".

> So you say that the "code point" entry of report is related to the
> codepoint of the font chosen to show the glyph?  How is it
> recalculated basing on the encoding of the font?

In the charset line it tells you it's the right hand part of
Latin-9.  That's where the offset of 128 comes from.

>> You can look at the font name, for example.  And you can look at the
>> charset and compare that with your font, to see whether they match.
>
> How would I know?  Emacs case the info about the font, not me.  One
> cannot deduce the encoding from the name, but one can do it from the
> character properties table.

One cannot deduce the encoding from the name?  I thought the encoding
was in the very end of the long-font-name-with-lotsa-dashes.  But I'm
not an expert on X fonts, I just kinda guessed it from having seen
lots of fonts with -gb2312 or -iso8859-1 or -iso10646 at the end.

>> >> You can tell Emacs to use the foo font for the bar encoding.
>> >
>> > How?  And how do I tell Emacs in which encoding this font actually is?
>> 
>> You can't.
>
> Wow!  So MULE is absolutely useless indeed...  I knew that it makes it
> extremely hard to edit 8-bit text, but I was told that this is the
> price to pay for internationalization.  Now it turns out that it also
> cannot *show* foreign text...

It was just my meagre attempt at some humor to lighten up the
discussion.

In my understanding, telling Emacs "use the foo font to display
gb2312 encoded characters" amounts to pretty much the same thing as
telling Emacs "the foo font is in gb2312 encoding".

>> >> Maybe
>> >> you have (accidentally) told Emacs to use a KOI-R font for the
>> >> iso-8859-5 encoding, or something like this.
>> >
>> > With -no-init-file?  I doubt it.  ;-)
>> 
>> With a X11 resource, for instance.  You probably set the font for
>> your X11 apps somewhere (otherwise it would be "fixed").
>
> *menuFont:      -*-courier-medium-r-*-*-18-*-*-*-*-*-*-*
> Emacs*Font:     -misc-fixed-bold-*-*-*-*-140-75-75-*-*-*-1
> Emacs*paneFont: -*-courier-bold-r-*-*-18-180-*-*-*-*-*-*
> Emacs*selectionFont:    -*-courier-bold-r-*-*-16-180-*-*-*-*-*-*
> Emacs.shell.menu.popup.font:    -*-courier-bold-r-*-*-18-180-*-*-*-*-*-*
> Emacs.shell.pane.menubar.font:  -*-courier-bold-r-*-*-16-180-*-*-*-*-*-*

Okay, okay.  Hm.

But first things first.  Now you know which font Emacs used to display
the Cyrillic.  Do you have any way of finding out the encoding
of that font (with an X11 font editor, perhaps, or by running some
other program)?

From the output of C-u C-x = you also know that Emacs thought the
font was in iso-8859-5 encoding (because Emacs used that font for
that charset).

I would like to verify at this point that the Cyrillic font is indeed
in an encoding that is NOT iso-8859-5.  Could you please do that?

Now let's assume that the font is really in the "wrong" encoding (ie,
not in iso-8859-5).  Now you need to tell Emacs to use a different
font for iso-8859-5.  This works with fontsets.

I am ashamed to say that I've tried to read the documentation on
fontsets and haven't been able to understand it.  But Emacs works
well for me, even without fontsets, so I didn't spend more than 2 or
3 minutes on understanding it.

I'm sure that you can cook up a fontset specification that will tell
Emacs to use a font that's right for iso-8859-5.

kai

PS: I'm assuming that Emacs uses a very simple mapping from
    characters to glyphs in the font.  For displaying the character
    numbered 4711 in any given encoding, Emacs uses the glyph at that
    code point.

    Please correct me if I'm wrong.  Anyone?
-- 
~/.signature is: umop ap!sdn    (Frank Nobis)

Re: MULE shows gibberish; now what?

[Kai Großjohann]

Ilya Zakharevich <nospam-abuse@ilyaz.org> writes:

> [A complimentary Cc of this posting was sent to
> Kai
>  =?iso-8859-15?q?Gro=DFjohann?=
> <Kai.Grossjohann@CS.Uni-Dortmund.DE>], who wrote in article <vafelbfw4cc.fsf@lucy.cs.uni-dortmund.de>:
>
>> In my understanding, telling Emacs "use the foo font to display
>> gb2312 encoded characters" amounts to pretty much the same thing as
>> telling Emacs "the foo font is in gb2312 encoding".
>
> Well, this is not enough.  A font in gb2312 encoding *has* cyrillic
> glyphs.  Will Emacs use them if they are present?

At the moment, the functionality is lacking indeed.  There is a file
latin1-disp.el which allows you to display some latin-2 characters
using their latin-1 equivalents, and vice versa.  But I think this
doesn't work with other fonts.

Note that for Emacs, a Cyrillic character in GB2312 and a Cyrillic
character in KOI-R (and so on) are all different characters, even
though they might look the same.  I don't know how much of this will
change in Emacs 22, but at the moment, that's the way it is.  And
because Emacs considers them to be different characters, Emacs
doesn't use glyphs for one character to display another.  And
latin1-disp.el is the exception from that rule...

kai
-- 
~/.signature is: umop ap!sdn    (Frank Nobis)

Re: MULE shows gibberish; now what?

[Jason Rumney]

Ilya Zakharevich <nospam-abuse@ilyaz.org> writes:

> > I don't know how much of this will change in Emacs 22, but at the
> > moment, that's the way it is.  And because Emacs considers them to
> > be different characters, Emacs doesn't use glyphs for one
> > character to display another.  And latin1-disp.el is the exception
> > from that rule...
> 
> Note that your "because" is misplaced.  Even if buffer-filling engine
> does not consider them the same, the display engine might have
> remapped them for display purposes to the same codepoint in the same
> font.

To do so, the display engine would require mapping tables. So far,
tables have been contributed for mapping the following encodings to
and from iso8859-5: koi8, koi8-r, alternativnyj, cp855, cp866,
windows-1251. If you wish to volunteer to create mapping tables for
gb2312, jis0208 etc, I'm sure they will be accepted.

Re: MULE shows gibberish; now what?

[Ilya Zakharevich]

[A complimentary Cc of this posting was sent to
Jason Rumney 
<jasonr@btinternet.co>], who wrote in article <m28z1jp7z1.fsf@nyaumo.btinternet.com>:
> Ilya Zakharevich <nospam-abuse@ilyaz.org> writes:
> 
> > > I don't know how much of this will change in Emacs 22, but at the
> > > moment, that's the way it is.  And because Emacs considers them to
> > > be different characters, Emacs doesn't use glyphs for one
> > > character to display another.  And latin1-disp.el is the exception
> > > from that rule...
> > 
> > Note that your "because" is misplaced.  Even if buffer-filling engine
> > does not consider them the same, the display engine might have
> > remapped them for display purposes to the same codepoint in the same
> > font.
> 
> To do so, the display engine would require mapping tables. So far,
> tables have been contributed for mapping the following encodings to
> and from iso8859-5: koi8, koi8-r, alternativnyj, cp855, cp866,
> windows-1251. If you wish to volunteer to create mapping tables for
> gb2312, jis0208 etc, I'm sure they will be accepted.

I made tables for 1 jis and 1 ksc so that xterm can use such fonts for
utf stuff (not in the released versions of xterm yet :-[).  However, I
do not see a point in supporting Emacs: even if I contribute these,
they would not be used for display purposes - Emacs does not do
remapping for display purposes...

Ilya

Re: MULE shows gibberish; now what?

[Jason Rumney]

Ilya Zakharevich <nospam-abuse@ilyaz.org> writes:

> I made tables for 1 jis and 1 ksc so that xterm can use such fonts for
> utf stuff (not in the released versions of xterm yet :-[).  However, I
> do not see a point in supporting Emacs: even if I contribute these,
> they would not be used for display purposes - Emacs does not do
> remapping for display purposes...

It does, but nonetheless, I am not surprised at your response, given
your tone in the rest of the thread and in previous threads to which
you have contributed.

Re: MULE shows gibberish; now what?

[Ilya Zakharevich]

[A complimentary Cc of this posting was sent to
Jason Rumney 
<jasonr@btinternet.co>], who wrote in article <m2smzrf9op.fsf@nyaumo.btinternet.com>:
> > I made tables for 1 jis and 1 ksc so that xterm can use such fonts for
> > utf stuff (not in the released versions of xterm yet :-[).  However, I
> > do not see a point in supporting Emacs: even if I contribute these,
> > they would not be used for display purposes - Emacs does not do
> > remapping for display purposes...
> 
> It does, but nonetheless, I am not surprised at your response, given
> your tone in the rest of the thread and in previous threads to which
> you have contributed.

The other posters to this thread say that it does not, but I'm not
surprised at etc...

Ilya

Re: MULE shows gibberish; now what?

[Stefan Monnier <foo@acm.com>]

>>>>> "Ilya" == Ilya Zakharevich <nospam-abuse@ilyaz.org> writes:
> The other posters to this thread say that it does not, but I'm not
> surprised at etc...

Having a bad day ?


        Stefan

Re: MULE shows gibberish; now what?

[Jason Rumney]

Ilya Zakharevich <nospam-abuse@ilyaz.org> writes:


> > > Emacs does not do remapping for display purposes...
> > 
> > It does, but nonetheless, I am not surprised at your response, given
> > your tone in the rest of the thread and in previous threads to which
> > you have contributed.
> 
> The other posters to this thread say that it does not, but I'm not
> surprised at etc...

I don't see anywhere in this thread where anyone other than yourself
has claimed that, but maybe I missed some messages. Whoever said it is
incorrect.

Emacs comes preconfigured to remap Cyrillic characters so they can be
displayed with iso8859-5, koi8 and alternativnyj fonts. If you follow
the examples for these in lisp/languages/cyrillic.el, and use the
cp1251 table in codepage.el, you can probably come up with a mapping
for rawin-r fonts.  If such fonts are really that widespread on
non-MS-Windows systems, then it would be very useful if someone could
contribute something to make them work. MS-Windows is not a concern,
since display is actually done in UTF-16 on that platform where
possible.

Re: MULE shows gibberish; now what?

[Jason Rumney]

Ilya Zakharevich <nospam-abuse@ilyaz.org> writes:

> > But first things first.  Now you know which font Emacs used to display
> > the Cyrillic.  Do you have any way of finding out the encoding
> > of that font (with an X11 font editor, perhaps, or by running some
> > other program)?
> 
> I can see the glyphs with xfd.  Spending some time I may be able to
> identify the glyph table with an encoding.  But how will it help me?

To save you the trouble, your font is encoded in windows-1251
codepage.

There is no reason why Emacs would select this font for displaying
iso8859-5, and since your X resources do not specify to use that
font, my only guess is that some misconfiguration of you font server
is causing it to be selected when Emacs requests an iso8859-5 font.

Re: MULE shows gibberish; now what?

[Kai Großjohann]

Ilya Zakharevich <nospam-abuse@ilyaz.org> writes:

> I get

Some concrete data, at last.

>   character:  (07100, 3648, 0xe40)
>     charset: cyrillic-iso8859-5
>          (Right-Hand Part of Latin/Cyrillic Alphabet (ISO/IEC 8859-5): ISO-IR-144)

So Emacs thinks it is displaying an iso-8859-5 character.

>  code point: 64

To be precise, the character 128+64 = 192.

>      syntax: word
>    category: y:Cyrillic  
> buffer code: 0x8C 0xC0
>   file code: ESC 2C 4C 40 (encoded by coding system iso-2022-7bit-unix)
>        font: -Cronyx-Fixed-Medium-R-Normal--15-140-75-75-C-90-RAWIN-R

I don't know this font.  Is it encoded in iso8859-5, or in another
encoding?  I'd expect the font name to end in iso8859-5 if it was in
that encoding, so I suspect it's in the wrong encoding.

> Do not know how Emacs choose this font and why...

Maybe you told it via the X11 resources.  What happens when you tell
Emacs to use an iso-8859-5 font, via the same way that you now chose
that font?

kai
-- 
~/.signature is: umop ap!sdn    (Frank Nobis)

Re: MULE shows gibberish; now what?

[Stefan Monnier <foo@acm.com>]

>> >        font: -Cronyx-Fixed-Medium-R-Normal--15-140-75-75-C-90-RAWIN-R
[...]
> ???  I do not use that font.  How many times should I repeat that tt
> is *Emacs* who choses this font?  My resources for fonts are

But there is no `cronyx' or `rawin-r' anywhere in the code of Emacs, so it
must get some "help" from somewhere in order to choose this font.
Normally for charset <foo> it looks for fonts with names of the form
*-...-*-*-<foo>, so there has to be something somewhere that tells Emacs to
use `rawin-r'.

Can you do `M-x describe-face RET default RET' and tell us which
font Emacs says it's using ?
Can you then do M-x describe-fontset RET <the name> RET and show
us what that says ?


        Stefan

Re: MULE shows gibberish; now what?

[Ilya Zakharevich]

[A complimentary Cc of this posting was sent to
Stefan Monnier 
<foo@acm.com>], who wrote in article <5lwup7kw31.fsf@rum.cs.yale.edu>:
> >> >        font: -Cronyx-Fixed-Medium-R-Normal--15-140-75-75-C-90-RAWIN-R
> [...]
> > ???  I do not use that font.  How many times should I repeat that tt
> > is *Emacs* who choses this font?  My resources for fonts are
> 
> But there is no `cronyx' or `rawin-r' anywhere in the code of Emacs, so it
> must get some "help" from somewhere in order to choose this font.
> Normally for charset <foo> it looks for fonts with names of the form
> *-...-*-*-<foo>, so there has to be something somewhere that tells Emacs to
> use `rawin-r'.
> 
> Can you do `M-x describe-face RET default RET' and tell us which
> font Emacs says it's using ?
> Can you then do M-x describe-fontset RET <the name> RET and show
> us what that says ?

Maybe in a week.  Especially if you explain what is "the name" -
something from the output of describe-face?

Ilya

Re: MULE shows gibberish; now what?

[Ilya Zakharevich]

[A complimentary Cc of this posting was sent to
Stefan Monnier 
<foo@acm.com>], who wrote in article <5lwup7kw31.fsf@rum.cs.yale.edu>:
> >> >        font: -Cronyx-Fixed-Medium-R-Normal--15-140-75-75-C-90-RAWIN-R
> [...]
> > ???  I do not use that font.  How many times should I repeat that tt
> > is *Emacs* who choses this font?  My resources for fonts are
> 
> But there is no `cronyx' or `rawin-r' anywhere in the code of Emacs, so it
> must get some "help" from somewhere in order to choose this font.
> Normally for charset <foo> it looks for fonts with names of the form
> *-...-*-*-<foo>, so there has to be something somewhere that tells Emacs to
> use `rawin-r'.

"Something else" does not help.  What can it be?  I posted the
relevant parts of 'xrdb -query | grep -i font'.  What else?

> Can you do `M-x describe-face RET default RET' and tell us which
> font Emacs says it's using ?

         Family: misc-fixed
          Width: normal
         Height: 120
         Weight: bold
          Slant: normal
     Foreground: black
     Background: white
      Underline: nil
       Overline: nil
 Strike-through: nil
            Box: nil
        Inverse: nil
        Stipple: nil
Font or fontset: -misc-fixed-bold-*-*-*-*-140-75-75-*-*-*-1
        Inherit: unspecified

> Can you then do M-x describe-fontset RET <the name> RET and show
> us what that says ?

[Aside: the output buffer has no trailing NL.]

Fontset: -*-*-bold-*-*-*-*-140-75-75-*-*-fontset-startup
CHARSET or CHAR RANGE	FONT NAME
---------------------	---------
ascii					-misc-fixed-bold-r-normal--15-140-75-75-c-90-iso8859-1
     [-Adobe-Helvetica-Bold-R-Normal--8-80-75-75-P-50-ISO8859-1]
     [-Adobe-Helvetica-Bold-O-Normal--14-100-100-100-P-82-ISO8859-1]
     [-Adobe-Helvetica-Bold-R-Normal--14-140-75-75-P-82-ISO8859-1]
     [-Misc-Fixed-Bold-R-Normal--15-140-75-75-C-90-ISO8859-1]
latin-iso8859-1		-misc-fixed-*-iso8859-1
     [-Misc-Fixed-Bold-R-Normal--15-140-75-75-C-90-ISO8859-1]
latin-iso8859-2		-*-iso8859-2
     [-ETL-Fixed-Medium-R-Normal--16-160-72-72-C-80-ISO8859-2]
latin-iso8859-3		-*-iso8859-3
     [-ETL-Fixed-Medium-R-Normal--16-160-72-72-C-80-ISO8859-3]
latin-iso8859-4		-*-iso8859-4
thai-tis620			-*-*-*-tis620-*
     [-misc-fixed-medium-r-normal--16-160-72-72-m-80-tis620.2529-1]
greek-iso8859-7		-*-*-*-iso8859-7
     [-ETL-Fixed-Medium-R-Normal--16-160-72-72-C-80-ISO8859-7]
arabic-iso8859-6	-*-*-*-iso8859-6
hebrew-iso8859-8	-*-*-*-iso8859-8
     [-Misc-Fixed-Medium-R-Normal--13-120-75-75-C-80-ISO8859-8]
katakana-jisx0201	-*-*-*-jisx0201-*
     [-Misc-Fixed-Medium-R-Normal--14-130-75-75-C-70-JISX0201.1976-0]
latin-jisx0201		-*-jisx0201-*
cyrillic-iso8859-5	-*-*-*-iso8859-5
     [-Cronyx-Fixed-Medium-R-Normal--15-140-75-75-C-90-RAWIN-R]
latin-iso8859-9		-*-iso8859-9
     [-ETL-Fixed-Medium-R-Normal--16-160-72-72-C-80-ISO8859-9]
latin-iso8859-15	-*-iso8859-15
latin-iso8859-14	-*-iso8859-14
japanese-jisx0208-1978	-*-*-*-jisx0208.1978-*
chinese-gb2312			-*-*-*-gb2312.1980-*
     [-ISAS-Fangsong ti-Medium-R-Normal--16-160-72-72-c-160-GB2312.1980-0]
japanese-jisx0208    -*-*-*-jisx0208.1990-*
     [-Misc-Fixed-Medium-R-Normal--16-150-75-75-C-160-JISX0208.1990-0]
korean-ksc5601		-*-*-*-ksc5601.1989-*
     [-Daewoo-Gothic-Medium-R-Normal--16-120-100-100-C-160-KSC5601.1987-0]
japanese-jisx0212	-*-*-*-jisx0212-*
chinese-cns11643-1	-*-*-*-cns11643.1992-1
chinese-cns11643-2	-*-*-*-cns11643.1992-2
japanese-jisx0213-1	-*-*-*-jisx0213.2000-1
chinese-big5-1			-*-*-*-big5-*
     [-ETen-Fixed-Medium-R-Normal--16-150-75-75-C-160-Big5.ETen-0]
chinese-big5-2		-*-*-*-big5-*
chinese-sisheng			-*-sisheng_cwnn-*
ipa						-*-muleipa-*
vietnamese-viscii-lower				-*-viscii1.1-*
     [-ETL-Fixed-Medium-R-Normal--16-160-72-72-C-80-VISCII1.1-1]
vietnamese-viscii-upper	-*-viscii1.1-*
arabic-digit			-*-*-*-mulearabic-0
arabic-1-column				-*-*-*-mulearabic-1
     [-ETL-Fixed-Medium-R-Normal--16-160-72-72-C-80-MuleArabic-1]
ascii-right-to-left	-*-iso8859-1
lao					-*-*-*-mulelao-1
     [-misc-fixed-medium-r-normal--16-160-72-72-m-80-MuleLao-1]
arabic-2-column		-*-*-*-mulearabic-2
     [-ETL-Fixed-Medium-R-Normal--16-160-72-72-C-160-MuleArabic-2]
indian-is13194		-*-*-*-is13194-devanagari
indian-1-column			-*-*-*-muleindian-1
     [-Misc-Fixed-Medium-R-Normal--16-160-72-72-C-80-MuleIndian-1]
tibetan-1-column	-*-*-*-muletibetan-1
mule-unicode-2500-33ff	-*-iso10646-1
mule-unicode-e000-ffff	-*-iso10646-1
mule-unicode-0100-24ff	-*-iso10646-1
ethiopic			-*-*-*-ethiopic-unicode
     [-Admas-Ethiomx16f-Medium-R-Normal--16-150-100-100-M-160-Ethiopic-Unicode]
chinese-cns11643-3	-*-*-*-cns11643.1992-3
chinese-cns11643-4	-*-*-*-cns11643.1992-4
chinese-cns11643-5	-*-*-*-cns11643.1992-5
chinese-cns11643-6	-*-*-*-cns11643.1992-6
chinese-cns11643-7	-*-*-*-cns11643.1992-7
indian-2-column			-*-*-*-muleindian-2
     [-Misc-Fixed-Medium-R-Normal--16-160-72-72-C-80-MuleIndian-2]
tibetan			-*-proportional-*-muletibetan-2
     [-TibMdXA-fixed-medium-r-normal--16-160-72-72-m-160-MuleTibetan-0]
japanese-jisx0213-2	-*-*-*-jisx0213.2000-2

Re: MULE shows gibberish; now what?

[Stefan Monnier <foo@acm.com>]

>> >> >        font: -Cronyx-Fixed-Medium-R-Normal--15-140-75-75-C-90-RAWIN-R
>> [...]
>> > ???  I do not use that font.  How many times should I repeat that tt
>> > is *Emacs* who choses this font?  My resources for fonts are
>> 
>> But there is no `cronyx' or `rawin-r' anywhere in the code of Emacs, so it
>> must get some "help" from somewhere in order to choose this font.
>> Normally for charset <foo> it looks for fonts with names of the form
>> *-...-*-*-<foo>, so there has to be something somewhere that tells Emacs to
>> use `rawin-r'.
> "Something else" does not help.  What can it be?  I posted the
> relevant parts of 'xrdb -query | grep -i font'.  What else?

I don't know what else.
I assume you're using `emacs -q --no-site-file' right ?
I've seen you mention --no-init-file (aka `-q') but not --no-site-file.
Could there be some global config in site-start.el or default.el ?
Or does this Emacs come from some RPM or some other such package where
the packagers might have changed the default ?

> Font or fontset: -misc-fixed-bold-*-*-*-*-140-75-75-*-*-*-1

As specified in the Xresources: looks sane.

> cyrillic-iso8859-5	-*-*-*-iso8859-5
>      [-Cronyx-Fixed-Medium-R-Normal--15-140-75-75-C-90-RAWIN-R]

This is odd.  Maybe the other poster (can't remember who it was)
was right: your Xserver has a weird font setup with an alias
from <godknowswhat>-iso8859-5
to -Cronyx-Fixed-Medium-R-Normal--15-140-75-75-C-90-RAWIN-R.

Can you try to look for RAWIN in the output of something like

    xlsfonts -ll -fn '*-iso8859-5'


-- Stefan

Re: MULE shows gibberish; now what?

[Ilya Zakharevich]

[A complimentary Cc of this posting was sent to
Stefan Monnier 
<foo@acm.com>], who wrote in article <5lzntvijqv.fsf@rum.cs.yale.edu>:
> > "Something else" does not help.  What can it be?  I posted the
> > relevant parts of 'xrdb -query | grep -i font'.  What else?
> 
> I don't know what else.
> I assume you're using `emacs -q --no-site-file' right ?
> I've seen you mention --no-init-file (aka `-q') but not --no-site-file.
> Could there be some global config in site-start.el or default.el ?

Well, when I mentioned this, this were all the options.  ;-) But
meanwhile I tested it with -no-site-file too.

> Or does this Emacs come from some RPM or some other such package where
> the packagers might have changed the default ?

How would I find it?

> > Font or fontset: -misc-fixed-bold-*-*-*-*-140-75-75-*-*-*-1

> This is odd.  Maybe the other poster (can't remember who it was)

Hmm, do not remember seeing this message...

> was right: your Xserver has a weird font setup with an alias
> from <godknowswhat>-iso8859-5
> to -Cronyx-Fixed-Medium-R-Normal--15-140-75-75-C-90-RAWIN-R.
> 
> Can you try to look for RAWIN in the output of something like
> 
>     xlsfonts -ll -fn '*-iso8859-5'

OK, *this* I can do remotely.  ;-)  Done:

name:  -cronyx-courier-bold-o-normal--10-100-75-75-m-60-iso8859-5
  direction:            left to right
  indexing:             linear
  rows:                 0x00 thru 0x00 (0 thru 0)
  columns:              0x20 thru 0xff (32 thru 255)
  all chars exist:      no
  default char:         0x0020 (32)
  ascent:               13
  descent:              3
  font type:            Monospaced (font width not equal to AVERAGE_WIDTH)
  bounds:               width left  right  asc  desc   attr   keysym
        min               10    -1    -1    -1   -11  0x0000
        max               10     7    14    12     3  0x0000
  properties:           21
      SPACING               M
      FONTNAME_REGISTRY
      FOUNDRY               Cronyx
      FAMILY_NAME           Courier
      WEIGHT_NAME           Bold
      SLANT                 O
      SETWIDTH_NAME         Normal
      ADD_STYLE_NAME
      AVERAGE_WIDTH         60
      CHARSET_REGISTRY      RAWIN
      CHARSET_ENCODING      R
      PIXEL_SIZE            10
      POINT_SIZE            100
      RESOLUTION_X          75
      RESOLUTION_Y          75
      COPYRIGHT             Copyright (C) 1990, 1991 EWT Consulting, Portions Co
pyright (C) 1994 Cronyx Ltd.
      FONT                  -Cronyx-Courier-Bold-O-Normal--10-100-75-75-M-60-RAWIN-R
      WEIGHT                10
      RESOLUTION            132
      X_HEIGHT              13
      QUAD_WIDTH            10

Etc.  A lot of them...  In fact there are 3 fonts which have
CHARSET_REGISTRY ISO8859, they have FOUNDRY ETL.  How do I convince
emacs to use them in preference to FOUNDRY Cronyx ones?

BTW, why -etl-fixed-medium-r-normal--16-160-72-72-c-80-iso8859-5 is
not prefered to the other guy automatically?  What are the criteria?

  -etl-fixed-medium-r-normal--14-140-72-72-c-70-iso8859-5
  -misc-fixed-bold-*-*-*-*-140-75-75-*-*-*-1
  -Cronyx-Fixed-Medium-R-Normal--15-140-75-75-C-90-RAWIN-R.

What can a sysadm do to fix this misconfiguration of thinking that
rawin-r is iso8859-5 (if it is one)?

Ilya

Re: MULE shows gibberish; now what?

[Stefan Monnier <foo@acm.com>]

>> Or does this Emacs come from some RPM or some other such package where
>> the packagers might have changed the default ?
> How would I find it?

The one who installed it should know.  But if you can't ask him or if he
doesn't know, then...

>> > Font or fontset: -misc-fixed-bold-*-*-*-*-140-75-75-*-*-*-1
>> This is odd.  Maybe the other poster (can't remember who it was)
> Hmm, do not remember seeing this message...
>> was right: your Xserver has a weird font setup with an alias
>> from <godknowswhat>-iso8859-5
>> to -Cronyx-Fixed-Medium-R-Normal--15-140-75-75-C-90-RAWIN-R.
>> Can you try to look for RAWIN in the output of something like
>> xlsfonts -ll -fn '*-iso8859-5'
> OK, *this* I can do remotely.  ;-)  Done:
> name:  -cronyx-courier-bold-o-normal--10-100-75-75-m-60-iso8859-5
[...]
>       FONT                  -Cronyx-Courier-Bold-O-Normal--10-100-75-75-M-60-RAWIN-R

So that's indeed the problem.  What kind of system is that ?
Solaris ?  GNU/Linux ?  Which distribution ?  Mandrake ?  Debian ?
It would be good if you could figure out where the alias for the fontname
comes from, so we can try and make sure it won't happen again.

> Etc.  A lot of them...  In fact there are 3 fonts which have
> CHARSET_REGISTRY ISO8859, they have FOUNDRY ETL.  How do I convince
> emacs to use them in preference to FOUNDRY Cronyx ones?

I don't know enough about fontsets to figure that out.  Emacs should
have better ways to specify preferences in the choice of fonts when
several "seemingly equivalent" fonts are found.  But nobody has bothered
to write the code yet.

> BTW, why -etl-fixed-medium-r-normal--16-160-72-72-c-80-iso8859-5 is
> not prefered to the other guy automatically?  What are the criteria?

I think there's no criteria for this.  It does sort the matching fonts
according to some criteria, but it only considers the size/boldness/...
to decide which is the closest match.  The handling of `family' and
`foundry' could use some improvement.

> What can a sysadm do to fix this misconfiguration of thinking that
> rawin-r is iso8859-5 (if it is one)?

Yes, it is a misconfiguration since the two encodings are not the same.
A sysadmin can just look at the /some/where/fonts.aliases file and
remove the bogus alias.


        Stefan

Re: MULE shows gibberish; now what?

[Ilya Zakharevich]

[A complimentary Cc of this posting was sent to
Stefan Monnier 
<foo@acm.com>], who wrote in article <5lit0gj2dq.fsf@rum.cs.yale.edu>:
> > name:  -cronyx-courier-bold-o-normal--10-100-75-75-m-60-iso8859-5
> [...]
> >       FONT                  -Cronyx-Courier-Bold-O-Normal--10-100-75-75-M-60-RAWIN-R
> 
> So that's indeed the problem.  What kind of system is that ?
> Solaris ?

Yes; 2.6.

> GNU/Linux ?  Which distribution ?  Mandrake ?  Debian ?
> It would be good if you could figure out where the alias for the fontname
> comes from, so we can try and make sure it won't happen again.

Have no idea how...

> > What can a sysadm do to fix this misconfiguration of thinking that
> > rawin-r is iso8859-5 (if it is one)?
> 
> Yes, it is a misconfiguration since the two encodings are not the same.
> A sysadmin can just look at the /some/where/fonts.aliases file and
> remove the bogus alias.

Is it enough grep the fontpath reported by xset?  Or can this
fonts.aliases file be long gone, so that the traces are present in the
compiled version only?

Ilya

Re: MULE shows gibberish; now what?

[Stefan Monnier <foo@acm.com>]

>> > What can a sysadm do to fix this misconfiguration of thinking that
>> > rawin-r is iso8859-5 (if it is one)?
>> Yes, it is a misconfiguration since the two encodings are not the same.
>> A sysadmin can just look at the /some/where/fonts.aliases file and
>> remove the bogus alias.
> Is it enough grep the fontpath reported by xset?  Or can this
> fonts.aliases file be long gone, so that the traces are present in the
> compiled version only?

I think you're right: the file might indeed be long gone, included inside
the resulting `fonts.dir' file.  I'd grep through any `fonts.*' in the
fontpath directories returned by xset.


        Stefan

Re: MULE shows gibberish; now what?

[Ilya Zakharevich]

[A complimentary Cc of this posting was sent to
Stefan Monnier 
<foo@acm.com>], who wrote in article <5ladlqhuia.fsf@rum.cs.yale.edu>:
> >> > What can a sysadm do to fix this misconfiguration of thinking that
> >> > rawin-r is iso8859-5 (if it is one)?
> >> Yes, it is a misconfiguration since the two encodings are not the same.
> >> A sysadmin can just look at the /some/where/fonts.aliases file and
> >> remove the bogus alias.
> > Is it enough grep the fontpath reported by xset?  Or can this
> > fonts.aliases file be long gone, so that the traces are present in the
> > compiled version only?
> 
> I think you're right: the file might indeed be long gone, included inside
> the resulting `fonts.dir' file.  I'd grep through any `fonts.*' in the
> fontpath directories returned by xset.

Neither RAWIN, nor iso8859-5 found (grep -i).  There is also

  tcp/169.229.58.58:7100

in the fontpath, but I do not know how to grep there.  :-(

Ilya

Re: MULE shows gibberish; now what?

[Stefan Monnier <foo@acm.com>]

> Neither RAWIN, nor iso8859-5 found (grep -i).  There is also
>   tcp/169.229.58.58:7100
> in the fontpath, but I do not know how to grep there.  :-(

You'll need to log into that machine and try to figure out how
the X font server (xfs) is configured.  Do you know what kind of machine
this is ?


        Stefan

Re: MULE shows gibberish; now what?

[Ilya Zakharevich]

[A complimentary Cc of this posting was sent to
Stefan Monnier 
<foo@acm.com>], who wrote in article <5l7kgtgini.fsf@rum.cs.yale.edu>:
> > Neither RAWIN, nor iso8859-5 found (grep -i).  There is also
> >   tcp/169.229.58.58:7100
> > in the fontpath, but I do not know how to grep there.  :-(
> 
> You'll need to log into that machine


[Fortunately, I *can* log in there.  On many setups I saw I would not
be able to.]

> and try to figure out how
> the X font server (xfs) is configured.

As far as I can see, xfs is not running (from output of pt -a).  I see
the reason: its argv[0] is actually "fs" (do not know why).  (Deduced
by comparing the result of /bin/ps -eo'f uid pid ppid tty time args' |
grep " fs" with the result of /bin/ps -e | grep xfs).

Here is the part of the process tree with its argv:

root         1        ?    1:04   /etc/init -rs
root       200        ?    0:09     /usr/sbin/inetd -s
nobody   14162        ?    0:02       fs
nobody    3399        ?    0:04     fs
nobody    3808        ?    0:01     fs
nobody    5740        ?    0:01     fs
nobody    8411        ?    0:06     fs
nobody    9947        ?    0:01     fs
nobody   11558        ?    0:02     fs
nobody   12927        ?    0:01     fs
nobody   13601        ?    0:01     fs
nobody   13982        ?    0:02     fs
nobody   23850        ?    0:03     fs
nobody   25090        ?    0:16     fs
nobody   27840        ?    0:36     fs
nobody   27843        ?    0:08     fs
nobody   27847        ?    0:03     fs
nobody   27850        ?    0:05     fs
nobody   27857        ?    0:12     fs

What else can I do?

> Do you know what kind of machine this is ?

Same Solaris.  This is a very homogeneous network.  

Ilya

Re: MULE shows gibberish; RAWIN-R vs iso8859-5

[Ilya Zakharevich]

[A complimentary Cc of this posting was sent to
Stefan Monnier 
<foo@acm.com>], who wrote in article <5lzntvijqv.fsf@rum.cs.yale.edu>:
> > cyrillic-iso8859-5	-*-*-*-iso8859-5
> >      [-Cronyx-Fixed-Medium-R-Normal--15-140-75-75-C-90-RAWIN-R]
> 
> This is odd.  Maybe the other poster (can't remember who it was)
> was right: your Xserver has a weird font setup with an alias
> from <godknowswhat>-iso8859-5
> to -Cronyx-Fixed-Medium-R-Normal--15-140-75-75-C-90-RAWIN-R.

Googling for RAWIN-R explains some historic things.  The first two
links (in Russian and Engish) "explain" how it was created.  Then
there is the 5th link to some Athena file with aliases, which aliases
iso8859-5 to these fonts.  So probably this Athena file was imported
to other places at some moment.

Anybody: is there any *positive* effect of these aliases?  If I ever
find *how* to disable them (which I doubt :-[), would there be any
negative effect for the hundreds of other people on this site?

Ilya

Re: MULE shows gibberish; now what?

[Kai Großjohann]

Ilya Zakharevich <nospam-abuse@ilyaz.org> writes:

> ???  I do not use that font.  How many times should I repeat that tt
> is *Emacs* who choses this font?

Thank you for your patience.

kai
-- 
~/.signature is: umop ap!sdn    (Frank Nobis)

Re: MULE shows gibberish; now what?

[Oliver Scholz]

Ilya Zakharevich <nospam-abuse@ilyaz.org> writes:

[...]
> ???  I do not use that font.  How many times should I repeat that tt
> is *Emacs* who choses this font?  

Let me phrase it like this: I don't see that font with the stock
Emacs, Kai doesn't see that font with the stock Emacs, probably most
people here don't see that font with the stock Emacs. So someone told
Emacs somewhere on your system to use that font.

It could take place in /etc/X11/app-default/Emacs. Or maybe your
GNU/Linux of Unix- distributor patched Emacs for your system? Or your
sysadmin is to blame? Who knows?

You could try to evaluate this:

(create-fontset-from-fontset-spec
 "-*-courier-medium-r-*-*-18-*-*-*-*-*-fontset-ilya,\
ecyrillic-iso8859-5:-*-*-*-*-*-*-*-*-*-*-*-*-iso8859-5")

(set-face-font 'default "fontset-ilya")

Does that change anything? (Note: you _must_ have a ISO 8859-5
encoded font on your system for this to work. You can look if you
have one with the program xfontsel.)

    -- Oliver

-- 
7 Vendémiaire an 211 de la Révolution
Liberté, Egalité, Fraternité!

Re: MULE shows gibberish; now what?

[Oliver Scholz]

Oliver Scholz <alkibiades@gmx.de> writes:
[...]
> (create-fontset-from-fontset-spec
>  "-*-courier-medium-r-*-*-18-*-*-*-*-*-fontset-ilya,\
> ecyrillic-iso8859-5:-*-*-*-*-*-*-*-*-*-*-*-*-iso8859-5")
  ^

Typo. That should read:

(create-fontset-from-fontset-spec
 "-*-courier-medium-r-*-*-18-*-*-*-*-*-fontset-ilya,\
cyrillic-iso8859-5:-*-*-*-*-*-*-*-*-*-*-*-*-iso8859-5")



    -- Oliver

-- 
7 Vendémiaire an 211 de la Révolution
Liberté, Egalité, Fraternité!

Re: MULE shows gibberish; now what?

[Luis O. Silva]

Privet Ilya!

On Tue, 24 Sep 2002 05:30:31 +0000 (UTC), Ilya Zakharevich writes:

   Ilya> Given the amount of bad press MULE gets (AFAIU,
   Ilya> correctly), I never tried to actully use it.  Well,
   Ilya> now I did.  And it does not work (21.2.2
   Ilya> -no-init-file).

   Ilya> Is there a way for a mortal to understand/fix MULE
   Ilya> bugs?

   Ilya> SPECIFICS of the problem1: when asked to show
   Ilya> cyrillic, it shows cyrillic indeed.  However, the
   Ilya> shown glyphs have no relationship to the actual
   Ilya> Cyrillic text (e.g., in the Hello, World example).
   Ilya> Do not have a slightest idea how to report it in more
   Ilya> details...

For me MULE handles all kinds of Cyrillic encodings very
well. For years I haven't had any problems reading
Russian. I've followed the discussion on this threat and I
thought that the hits given by Kai would help you understand
the problem. I got back to your original message because you
mention there the HELLO file. In my system I do:

$ emacs -no-init-file

then I type `C-h h' I put the point on the first letter of
Здравствуйте! and type `C-u x =' I get

  character: З (07067, 3639, 0xe37)
    charset: cyrillic-iso8859-5
	     (Right-Hand Part of Latin/Cyrillic Alphabet (ISO/IEC 8859-5): ISO-IR-144)
 code point: 55
     syntax: word
   category: y:Cyrillic
buffer code: 0x8C 0xB7
  file code: ESC 2C 4C 37 (encoded by coding system iso-2022-7bit-unix)
       font: -Misc-Fixed-Medium-R-Normal--13-120-75-75-C-70-ISO8859-5

If all is right with Emacs you *must* have the same. I think
that this is the case and the problem is in your OS. What
platform are you working on? Do you have the corresponding
fonts installed?

Regards,
luis


-- 
Luis Octavio Silva P.
St. Petersburg State University.
66/3 Botanicheskaya St., Apt.119/2
Stary Peterhof
St. Petersburg, Russia.

Re: MULE shows gibberish; now what?

[Ilya Zakharevich]

On Fri, Sep 27, 2002 at 11:29:39AM +0400, Luis O. Silva wrote:
> then I type `C-h h' I put the point on the first letter of
> ·ÔàÐÒáâÒãÙâÕ! and type `C-u x =' I get
> 
>   character: · (07067, 3639, 0xe37)
>     charset: cyrillic-iso8859-5
> 	     (Right-Hand Part of Latin/Cyrillic Alphabet (ISO/IEC 8859-5): ISO-IR-144)
>  code point: 55
>      syntax: word
>    category: y:Cyrillic
> buffer code: 0x8C 0xB7
>   file code: ESC 2C 4C 37 (encoded by coding system iso-2022-7bit-unix)
>        font: -Misc-Fixed-Medium-R-Normal--13-120-75-75-C-70-ISO8859-5

I already reported what I get in the newsgroup.  Cannot generate it
again now: I'm out of X11 for some time.  But I know it is different:
the font is very strange...

> If all is right with Emacs you *must* have the same. I think
> that this is the case and the problem is in your OS. What
> platform are you working on?

Nothing fancy.  Solaris 2.6.

> Do you have the corresponding fonts installed?

What is "corresponding fonts"?

Ilya

Re: MULE shows gibberish; now what?

[Ilya Zakharevich]

[A complimentary Cc of this posting was sent to
Ilya Zakharevich 
<nospam-abuse@ilyaz.org>], who wrote in article <amot9n$24j8$1@agate.berkeley.edu>:
> Given the amount of bad press MULE gets (AFAIU, correctly), I never
> tried to actully use it.  Well, now I did.  And it does not work
> (21.2.2 -no-init-file).
> 
> Is there a way for a mortal to understand/fix MULE bugs?
> 
> SPECIFICS of the problem1: when asked to show cyrillic, it shows
> cyrillic indeed.  However, the shown glyphs have no relationship to
> the actual Cyrillic text (e.g., in the Hello, World example).  Do not
> have a slightest idea how to report it in more details...

It looks like pointing-fingers was not dead-on-the-target; while Emacs
is faulty, the principal blame should go to xrus-1251 and (allegedly)
Netscape.  An excerpt from

  http://www.siber.org/sib/russify/x-windows/#win-fonts

   Yes, now you can have CP-1251 fonts on your X Window system.

   We had to put them under the iso8859-5 because buggy Netscape would
   not allow these fonts to work under official CP-1251 name.

So what I see may be a very widespread situation.  Emacs can easily
detect that the font *is not* a iso8859-5 font (since iso8859-5 has
very few characters, and cp1251 has many).  At least it could have
given a warning...

Now Emacs blames: very little documentation for fontsets available
(this site has no Elisp manual, so maybe there is something there;
there *are* some mentions of fontsets in Emacs manual, but they are
close to useless; I rate them close to "this function would do what
you want").

Fontsets are not even mentioned in the concept index available through
help menu.

Ilya

Re: MULE shows gibberish; now what?

[Ilya Zakharevich]

I wrote in article <ao0hgi$1fim$1@agate.berkeley.edu>:

> > SPECIFICS of the problem1: when asked to show cyrillic, it shows
> > cyrillic indeed.  However, the shown glyphs have no relationship to
> > the actual Cyrillic text (e.g., in the Hello, World example).  Do not
> > have a slightest idea how to report it in more details...
> 
> It looks like pointing-fingers was not dead-on-the-target; while Emacs
> is faulty, the principal blame should go to xrus-1251 and (allegedly)
> Netscape.  An excerpt from
> 
>   http://www.siber.org/sib/russify/x-windows/#win-fonts

After some private email exchange with the knowledgeable people
(designers of the system?), I got the following fix:

  (setq face-ignored-fonts '("-cronyx-.*-iso8859-5$"))

It is not documented, but this REx is matched against the name Emacs
uses for the query, and not the "actual" font name; moreover, this REx
is case-insensitive, and is not achored.

E.g., in my case Emacs was getting a Cronyx font via an -iso8859-5
alias; the actual encoding of the font (and the tail of the "actual"
name) was reported to be RAWIN-R.  But the REx should be designed to
exclude the alias name, not the actual name.

Hope this helps,
Ilya