Charset problem

Charset problem

[Christian Schröder]

Hi newgroup!
I have a question regarding character encoding which might be an Emacs 
problem or maybe a PuTTY problem or even a general linux problem. If you 
think my problem is not Emacs related, please let me know.
My problem is as follows: I log into my SuSE 10.2 linux server from my 
Windows box using PuTTY. I then use GNU Emacs (in text mode) to edit 
files on the linux box.
I have some old files which are encoded in ISO-8859-1. When I open these 
files in Emacs the german umlauts are displayed as question marks. I 
have tried to set PuTTY's translation to both UTF-8 and ISO-8859-1, but 
this doesn't change anything. When I set PuTTY's translation to 
ISO-8859-1 and *type* an umlaut, I see this umlaut only if it is 
lower-case, but it is followed by a blank. When I save the file and look 
at it using a hex editor the caracter is encoded in UTF-8. When I type 
an upper-case umlaut I see a question mark in Emacs and the hex editor 
shows a 0xC3 byte. When I set PuTTY's translation to UTF-8 and type an 
umlaut, I see a question mark, then some graphical character and a 
blank. In the hex editor I see a 3-byte sequence.
PuTTY seems to send an ISO-8859-1 encoded character when I set the 
translation to ISO-8859-1. At least it does in raw mode -- I have 
verified this using a network packet sniffer; unfortunately, I cannot 
look inside the SSH packets during the regular communication with the 
server.
So which way do my characters take before they get into the file? How 
many translation steps are done, and by whom? Are there any Emacs 
variables that must be set?

Thanks for any help!

Regards,
	Christian

Re: Charset problem

[Eli Zaretskii]

> From: =?ISO-8859-1?Q?Christian_Schr=F6der?= <chschroe@math.uni-goettingen.de>
> Date: Thu, 13 Sep 2007 01:05:19 +0200
> 
> My problem is as follows: I log into my SuSE 10.2 linux server from my 
> Windows box using PuTTY. I then use GNU Emacs (in text mode) to edit 
> files on the linux box.

You haven't supplied enough information about your systems to
intelligently guess what might be your problem.  One piece of info
that is missing is the locales of your SuSE server and of your Windows
box.

> I have some old files which are encoded in ISO-8859-1. When I open these 
> files in Emacs the german umlauts are displayed as question marks.

Go to one of those question marks and type "C-u C-x =".  Then tell
here what Emacs has to say about those characters.

Also, what does Emacs show in the left edge of the mode line.  There's
a character there that says what Emacs thinks about the encoding of
the file, and what encoding it uses for displaying characters on the
terminal (see the user manual, node "Mode Line", for explanations
about these).  What do you see there when you visit the ISO-8859-1
encoded files?

> I have tried to set PuTTY's translation to both UTF-8 and ISO-8859-1

Are you talking about the Window->Translation page in the PuTTY's
session configuration dialog?

Did you click the "Save" button on the "Session" page, after changing
those settings, and then restarted the session?  If not, the new
settings might not be in effect.

> PuTTY seems to send an ISO-8859-1 encoded character when I set the 
> translation to ISO-8859-1.

The "translation" setting is not for sending, it's for receiving: it
tells PuTTY what font to use when it gets characters with the 8th bit
set.

> So which way do my characters take before they get into the file? How 
> many translation steps are done, and by whom? Are there any Emacs 
> variables that must be set?

Before answering these questions, we need to know the information
about your locales.  It's hard to help you before that, except by wild
guessing.

Re: Charset problem

[Christian Schröder]

Eli Zaretskii wrote:
> You haven't supplied enough information about your systems to
> intelligently guess what might be your problem.  One piece of info
> that is missing is the locales of your SuSE server and of your Windows
> box.

Sorry!

These are the locale settings on my server:
LANG=posix
LC_CTYPE="posix"
LC_NUMERIC="posix"
LC_TIME="posix"
LC_COLLATE="posix"
LC_MONETARY="posix"
LC_MESSAGES="posix"
LC_PAPER="posix"
LC_NAME="posix"
LC_ADDRESS="posix"
LC_TELEPHONE="posix"
LC_MEASUREMENT="posix"
LC_IDENTIFICATION="posix"
LC_ALL=

Do you need anything else from the linux box?

I'm not sure how to find out the locale of my Windows box, but it's a 
German windows version, if that is of any help.

> Go to one of those question marks and type "C-u C-x =".  Then tell
> here what Emacs has to say about those characters.

One example:

     character: ? (0374, 252, 0xfc)
       charset: eight-bit-graphic (8-bit graphic char (0xA0..0xFF))
    code point: 252
        syntax: whitespace
      category:
   buffer code: 0xFC
     file code: 0xFC (encoded by coding system raw-text-unix)
terminal code: not encodable

This is what I expected because the character is a german "u-umlaut".

> Also, what does Emacs show in the left edge of the mode line.  There's
> a character there that says what Emacs thinks about the encoding of
> the file, and what encoding it uses for displaying characters on the
> terminal (see the user manual, node "Mode Line", for explanations
> about these).  What do you see there when you visit the ISO-8859-1
> encoded files?

-:---F1  join.php       11:33PM 0.19   (PHP 
Abbrev)--L10--C30--Top-----------

So it seems to be "no code conversion". I tried to change the coding 
system for keyboard and/or terminal, but it did not change anything. It 
seems as if I still don't understand what's happening. :(

>> I have tried to set PuTTY's translation to both UTF-8 and ISO-8859-1
> 
> Are you talking about the Window->Translation page in the PuTTY's
> session configuration dialog?

Exactly!

> Did you click the "Save" button on the "Session" page, after changing
> those settings, and then restarted the session?  If not, the new
> settings might not be in effect.

I changed the current session's settings and I think that the settings 
actually came into effect because the behaviour changed. Unfortunately, 
I was not able to get it to behave the way I want it ...

>> PuTTY seems to send an ISO-8859-1 encoded character when I set the 
>> translation to ISO-8859-1.
> 
> The "translation" setting is not for sending, it's for receiving: it
> tells PuTTY what font to use when it gets characters with the 8th bit
> set.

Yes, I have read this in the manual, but I have also found some 
references where it says that the same encoding is used for sending. If 
not, what does PuTTY send if I type some german characters on my 
keyboard? Can I somehow influence what it sends?

>> So which way do my characters take before they get into the file? How 
>> many translation steps are done, and by whom? Are there any Emacs 
>> variables that must be set?
> 
> Before answering these questions, we need to know the information
> about your locales.  It's hard to help you before that, except by wild
> guessing.

Sorry for giving too little information and many thanks for your help!

Regards,
	Christian

Re: Charset problem

[Eli Zaretskii]

> From: =?ISO-8859-1?Q?Christian_Schr=F6der?= <chschroe@math.uni-goettingen.de>
> Date: Thu, 13 Sep 2007 23:50:52 +0200
> 
> These are the locale settings on my server:
> LANG=posix
> LC_CTYPE="posix"
> LC_NUMERIC="posix"
> LC_TIME="posix"
> LC_COLLATE="posix"
> LC_MONETARY="posix"
> LC_MESSAGES="posix"
> LC_PAPER="posix"
> LC_NAME="posix"
> LC_ADDRESS="posix"
> LC_TELEPHONE="posix"
> LC_MEASUREMENT="posix"
> LC_IDENTIFICATION="posix"
> LC_ALL=

I think this is your problem, right there: on a Posix locale, Emacs
disables all its automagic for non-ASCII characters, and expects you
to tell it explicitly what to do, which is a pain.

Can you set up an ISO-8859-1 or ISO-8859-10 locale?  If not, why not?
I'd expect that users in Germany use a German locale.

>      character: ? (0374, 252, 0xfc)
>        charset: eight-bit-graphic (8-bit graphic char (0xA0..0xFF))

That's expected on a Posix locale, I think: Emacs treats any non-ASCII
characters as meaningless bytes.

> This is what I expected because the character is a german "u-umlaut".

No, you should expect Emacs to detect that this is an ISO-8859-1
character, not an eight-bit-graphic character.

> > Also, what does Emacs show in the left edge of the mode line.  There's
> > a character there that says what Emacs thinks about the encoding of
> > the file, and what encoding it uses for displaying characters on the
> > terminal (see the user manual, node "Mode Line", for explanations
> > about these).  What do you see there when you visit the ISO-8859-1
> > encoded files?
> 
> -:---F1  join.php       11:33PM 0.19   (PHP Abbrev)--L10--C30--Top-----------
> 
> So it seems to be "no code conversion".

Also expected for a Posix locale.

> I tried to change the coding 
> system for keyboard and/or terminal, but it did not change anything. It 
> seems as if I still don't understand what's happening. :(

Try "M-x set-language-environment RET Latin-1 RET".  Does this help?

> > The "translation" setting is not for sending, it's for receiving: it
> > tells PuTTY what font to use when it gets characters with the 8th bit
> > set.
> 
> Yes, I have read this in the manual, but I have also found some 
> references where it says that the same encoding is used for sending.

Not with Emacs.  Emacs encodes everything by itself.

Re: Charset problem

[Peter Dyballa]

Am 13.09.2007 um 23:50 schrieb Christian Schröder:

> These are the locale settings on my server:
> LANG=posix
> LC_CTYPE="posix"

These settings make GNU Emacs work in a 7 bit world, as in "C" locale  
environment. Therefore umlauts are not recognised as themselves. GNU  
Emacs will work better when you can launch it as in

	env LC_CTYPE=de_DE.ISO8859-15 emacs ... &

ISO 8859-15 or ISO Latin-9 supports €.

--
Mit friedvollen Grüßen

   Pete

Der Unterschied zwischen Theorie und Praxis ist in Praxis meist  
größer als in der Theorie

Re: Charset problem

[Christian Schröder]

Eli Zaretskii wrote:
> I think this is your problem, right there: on a Posix locale, Emacs
> disables all its automagic for non-ASCII characters, and expects you
> to tell it explicitly what to do, which is a pain.
> 
> Can you set up an ISO-8859-1 or ISO-8859-10 locale?  If not, why not?
> I'd expect that users in Germany use a German locale.

As far as I know I have to set the "LANG" or "LC_CTYPE" environment 
variable, or is there any other way to change the locale?
After setting LANG to "de_DE" (-> export LANG=de_DE) the output of 
"locale" is as follows:

LANG=de_DE
LC_CTYPE="de_DE"
LC_NUMERIC="de_DE"
LC_TIME="de_DE"
LC_COLLATE="de_DE"
LC_MONETARY="de_DE"
LC_MESSAGES="de_DE"
LC_PAPER="de_DE"
LC_NAME="de_DE"
LC_ADDRESS="de_DE"
LC_TELEPHONE="de_DE"
LC_MEASUREMENT="de_DE"
LC_IDENTIFICATION="de_DE"
LC_ALL=

The locale "de_DE.ISO8859-15" (as Peter suggested) doesn't seem to be 
available on my system. At least it is not in the list that "locale -a" 
prints and no directory of this name exists under /usr/lib/locale.

When I now start Emacs and load the file I still see the question marks. 
The mode line is the same as before. The output of "C-u C-x =" has 
changed slightly:

     character: ? (0374, 252, 0xfc)
       charset: eight-bit-graphic (8-bit graphic char (0xA0..0xFF))
    code point: 252
        syntax: word
      category:
   buffer code: 0xFC
     file code: 0xFC (encoded by coding system raw-text-unix)
terminal code: not encodable

The output of "C-h C" is now:

Coding system for saving this buffer:
   t -- raw-text-unix
Default coding system (for new files):
   1 -- iso-latin-1 (alias: iso-8859-1 latin-1)
Coding system for keyboard input:
   1 -- iso-8859-1 (alias of iso-latin-1)
Coding system for terminal output:
   1 -- iso-latin-1 (alias: iso-8859-1 latin-1)
Defaults for subprocess I/O:
   decoding: 1 -- iso-latin-1 (alias: iso-8859-1 latin-1)
   encoding: 1 -- iso-latin-1 (alias: iso-8859-1 latin-1)

This is really different from before (with POSIX locale), but it did not 
solve my problem. :(

> Try "M-x set-language-environment RET Latin-1 RET".  Does this help?

Doesn't change anything, no matter of the locale I use.

Do you have any other ideas?

Regards,
	Christian

Re: Charset problem

[Eli Zaretskii]

> From: =?ISO-8859-1?Q?Christian_Schr=F6der?= <chschroe@math.uni-goettingen.de>
> Date: Fri, 14 Sep 2007 14:41:54 +0200
> 
> This is really different from before (with POSIX locale), but it did not 
> solve my problem. :(
> 
> > Try "M-x set-language-environment RET Latin-1 RET".  Does this help?
> 
> Doesn't change anything, no matter of the locale I use.
> 
> Do you have any other ideas?

Can you post the offending file (as a binary attachment), or at least
its small fragment with the characters shown as question marks?
Perhaps the file itself mixes Latin-1 and non-Latin-1 characters, or
has some other weird bytes in it.

Btw, does it help to visit the file with "C-x RET c Latin-1 RET C-x C-f"
instead of the usual "C-x C-f"?

Re: Charset problem

[Christian Schröder]

[-- Attachment #1: Type: text/plain, Size: 841 bytes --]

Eli Zaretskii wrote:
> Can you post the offending file (as a binary attachment), or at least
> its small fragment with the characters shown as question marks?

I have attached a fragment of the file to this posting. I have zipped it 
to make sure it doesn't get modified during its path through the news 
world. ;)

> Perhaps the file itself mixes Latin-1 and non-Latin-1 characters, or
> has some other weird bytes in it.

I could not see anything like this, but maybe I have overlooked something.

> Btw, does it help to visit the file with "C-x RET c Latin-1 RET C-x C-f"
> instead of the usual "C-x C-f"?

"Latin-1" did not work ("No match"), so I used "iso-latin-1" (which is 
the default when I have the "de_DE" locale active) and saw no difference 
(which did not surprise me since "iso-latin-1" is the default).

Regards,
	Christian

[-- Attachment #2: example.zip --]
[-- Type: application/x-zip-compressed, Size: 325 bytes --]

[-- Attachment #3: Type: text/plain, Size: 152 bytes --]

_______________________________________________
help-gnu-emacs mailing list
help-gnu-emacs@gnu.org
http://lists.gnu.org/mailman/listinfo/help-gnu-emacs

Re: Charset problem

[Eli Zaretskii]

> From: =?ISO-8859-1?Q?Christian_Schr=F6der?= <chschroe@math.uni-goettingen.de>
> Date: Fri, 14 Sep 2007 17:06:01 +0200
> 
> > Btw, does it help to visit the file with "C-x RET c Latin-1 RET C-x C-f"
> > instead of the usual "C-x C-f"?
> 
> "Latin-1" did not work ("No match"), so I used "iso-latin-1" (which is 
> the default when I have the "de_DE" locale active) and saw no difference 
> (which did not surprise me since "iso-latin-1" is the default).

You mean, even with "C-x RET c latin-1 C-x C-f" Emacs still claims
that the umlauts are eight-bit-graphic characters?  That is soooo
strange!

What version of Emacs is that?  What does "M-x emacs-version RET" say?

Anyway, I just logged into a GNU/Linux box from Windows using
ISO-8859-1 as encoding in Translation page, and I see the u-umlaut
characters in your example file just fine, with PuTTY version 0.58
that uses Courier New font for display.  I see 3 1's in the left edge
of the mode line, meaning that keyboard and terminal are set to use
Latin-1 and the file is decoded as Latin-1 as well.

I wonder why you see something different.

Re: Charset problem

[Tim X]

Christian Schröder <chschroe@math.uni-goettingen.de> writes:

> Eli Zaretskii wrote:
>> I think this is your problem, right there: on a Posix locale, Emacs
>> disables all its automagic for non-ASCII characters, and expects you
>> to tell it explicitly what to do, which is a pain.
>>
>> Can you set up an ISO-8859-1 or ISO-8859-10 locale?  If not, why not?
>> I'd expect that users in Germany use a German locale.
>
> As far as I know I have to set the "LANG" or "LC_CTYPE" environment
> variable, or is there any other way to change the locale?
> After setting LANG to "de_DE" (-> export LANG=de_DE) the output of "locale"
> is as follows:
>
> LANG=de_DE
> LC_CTYPE="de_DE"
> LC_NUMERIC="de_DE"
> LC_TIME="de_DE"
> LC_COLLATE="de_DE"
> LC_MONETARY="de_DE"
> LC_MESSAGES="de_DE"
> LC_PAPER="de_DE"
> LC_NAME="de_DE"
> LC_ADDRESS="de_DE"
> LC_TELEPHONE="de_DE"
> LC_MEASUREMENT="de_DE"
> LC_IDENTIFICATION="de_DE"
> LC_ALL=
>
> The locale "de_DE.ISO8859-15" (as Peter suggested) doesn't seem to be
> available on my system. At least it is not in the list that "locale -a"
> prints and no directory of this name exists under /usr/lib/locale.

Depending on your Linux distro, the additional locale files may be in a
separate package. On some systems, they even have C shared libraries with
non POSIX/C support in different packages. It is also possible that you
need to create the locale files. Unfortuatnely, as different Linux distros
tend to do this in different ways, I can't be more specific other than to
suggest you look through the packages that are not installed and see if
anything looks relevant i.e. look for packages with terms like locale,
makelocale etc.

Tim


-- 
tcross (at) rapttech dot com dot au

Re: Charset problem

[Christian Schröder]

Eli Zaretskii wrote:
> You mean, even with "C-x RET c latin-1 C-x C-f" Emacs still claims
> that the umlauts are eight-bit-graphic characters?  That is soooo
> strange!

Ok, it just became even more strange ...
I did not mention (sorry!) that I was in a "screen" session when I did 
all the tests. When I realized that this might be the reason for my 
problems I tried to simply log into the server and start Emacs directly 
and ... tadaa ... all the umlauts were there.
Ok, so screen seemed to be the culprit. To verify this suspicion I 
started screen and then Emacs ... and it still worked. So something 
seems to be wrong with the "screen" session I used for all my tests, but 
I have no idea what it could be. I dumped the whole environment to a 
file and sourced this file into a fresh shell, but it still worked, so 
the environment doesn't seem to have anything to do with my problem.
Is it worth to further investigate this issue? At least I can start to 
work now, but it would still be interesting what's wrong with the screen 
session ...

> What version of Emacs is that?  What does "M-x emacs-version RET" say?

If this information is still of any use:
GNU Emacs 21.3.1 (i586-suse-linux-gnu) of 2006-11-27 on tanajew

If I may ask one last question: I normally have my PuTTY translation set 
to UTF-8. Is it possible to make Emacs send me the ISO-8859-1 encoded 
umlauts as UTF-8 characters? Then I would not have to change the 
translation on my side.

Kind regards,
	Christian

Re: Charset problem

[Peter Dyballa]

Am 15.09.2007 um 17:03 schrieb Christian Schröder:

> Is it possible to make Emacs send me the ISO-8859-1 encoded umlauts  
> as UTF-8 characters?

Of course. When you launch GNU Emacs make it see environment variable  
LC_CTYPE be like de_DE.UTF-8. Then it would automatically switch to  
UTF-8 use. Otherwise this statement in some init file can make it:

	(prefer-coding-system	'utf-8)

Problems can arise when the file-name-coding-system is now different ...

> Then I would not have to change the translation on my side.

You would need to convert some files, I think ...

--
Mit friedvollen Grüßen

   Pete

"Reisen bildet - vor allem Staus auf Autobahnen." (Michael Schiff)

Re: Charset problem

[Eli Zaretskii]

> From: =?ISO-8859-1?Q?Christian_Schr=F6der?= <chschroe@math.uni-goettingen.de>
> Date: Sat, 15 Sep 2007 17:03:43 +0200
> 
> If I may ask one last question: I normally have my PuTTY translation set 
> to UTF-8. Is it possible to make Emacs send me the ISO-8859-1 encoded 
> umlauts as UTF-8 characters?

Yes, as long as terminal encoding is set to UTF-8 (use
set-terminal-coding-system to set if not set by default in your
environment).

Re: Charset problem

[Giorgos Keramidas]

On Fri, 14 Sep 2007 14:41:54 +0200, Christian Schröder <chschroe@math.uni-goettingen.de> wrote:
>Eli Zaretskii wrote:
>> I think this is your problem, right there: on a Posix locale, Emacs
>> disables all its automagic for non-ASCII characters, and expects you
>> to tell it explicitly what to do, which is a pain.
>>
>> Can you set up an ISO-8859-1 or ISO-8859-10 locale?  If not, why not?
>> I'd expect that users in Germany use a German locale.
>
> As far as I know I have to set the "LANG" or "LC_CTYPE" environment
> variable, or is there any other way to change the locale?  After
> setting LANG to "de_DE" (-> export LANG=de_DE) the output of "locale"
> is as follows:
>
> LANG=de_DE
> LC_CTYPE="de_DE"
> LC_NUMERIC="de_DE"
> LC_TIME="de_DE"
> LC_COLLATE="de_DE"
> LC_MONETARY="de_DE"
> LC_MESSAGES="de_DE"
> LC_PAPER="de_DE"
> LC_NAME="de_DE"
> LC_ADDRESS="de_DE"
> LC_TELEPHONE="de_DE"
> LC_MEASUREMENT="de_DE"
> LC_IDENTIFICATION="de_DE"
> LC_ALL=

That's a fairly "large" set of LC_ and LANG options.  I don't know what
sets all these environment variables, but it may be worth trying a much
simpler set of environment options.  It's probably wrong, for instance,
to use just a language identifier in LC_XXX values.  I would expect
something like "de_DE.ISO8859-15" instead.

Try unsetting all of these LC_XXX variables, and check if a simpler set
of LC_XXX and LANG options works better.  For example, I'm using the
following for Greek support:

    LANG='el_GR'
    LC_ALL='el_GR.ISO8859-7'

That's easier to grasp and remember than 20 or so odd names too :)

> The locale "de_DE.ISO8859-15" (as Peter suggested) doesn't seem to be
> available on my system. At least it is not in the list that "locale
> -a" prints and no directory of this name exists under /usr/lib/locale.

You should try installing it.

Without a proper locale environment, not only Emacs but any other
application will have problems supporting German character sets.

Re: Charset problem

[Christian Schröder]

Peter Dyballa wrote:
>> Is it possible to make Emacs send me the ISO-8859-1 encoded umlauts as 
>> UTF-8 characters?
> 
> Of course. When you launch GNU Emacs make it see environment variable 
> LC_CTYPE be like de_DE.UTF-8. Then it would automatically switch to 
> UTF-8 use. Otherwise this statement in some init file can make it:
> 
>     (prefer-coding-system    'utf-8)
> 
> Problems can arise when the file-name-coding-system is now different ...

Ok, I seem to have found the cause for all of my problems. I have been 
using an Emacs init file for several years now, without ever having a 
closer look to it. It contained the following line:

(standard-display-european 1)

which I assume was useful long ago, but was now not only obsolete, but 
even harmful. After I removed this command from the init file the system 
behaves as expected. I now also finally see a correct mode line.

Thanks a lot for all your helpful comments. At least I have learned some 
tricks about how to find out what's going on.

Kind regards,
	Christian

Re: Charset problem

[Peter Dyballa]

Am 17.09.2007 um 13:11 schrieb Christian Schröder:

> Ok, I seem to have found the cause for all of my problems. I have  
> been using an Emacs init file for several years now, without ever  
> having a closer look to it. It contained the following line:
>
> (standard-display-european 1)
>
> which I assume was useful long ago

In GNU Emacs 19, I presume ...

--
Mit friedvollen Grüßen

   Pete

"Let's face it; we don't want a free market economy either."
         James Farley, president, Coca-Cola Export Corp., 1959