problems displaying german "umlaute"

problems displaying german "umlaute"

[Rainer Stengele]

Hi all,

sorry this is surely a standard problem but I cannot find a fast solution.

I am using textfiles (actually org-mode text) both on windows emacs 23 
(emacsw32-20090226) and under different linux variants (kubuntu, sidux) 
some running emacs 22 and some emacs 23.

The problem is that the "umlauts" shown in the windows emacs without 
problems are shown as code under linux.

I am using the exact same configuration (versioned by subversion), 
.emacs etc.

So under windows I see:

Hüpfreduzierspiel rekursiv lösen 						

under linux I see:				

H\303\274pfreduzierspiel rekursiv l\366sen 					

What can I do?

Thank you for any hint!


Rainer	

Re: problems displaying german "umlaute"

[tomas]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Tue, May 19, 2009 at 11:05:44PM +0200, Rainer Stengele wrote:
> Hi all,
>
> sorry this is surely a standard problem but I cannot find a fast solution.

[...]

> The problem is that the "umlauts" shown in the windows emacs without 
> problems are shown as code under linux.
>
> I am using the exact same configuration (versioned by subversion), .emacs 
> etc.
>
> So under windows I see:
>
> Hüpfreduzierspiel rekursiv lösen 						
>
> under linux I see:				
>
> H\303\274pfreduzierspiel rekursiv l\366sen 					

This one looks a bit strange to me (the u-umlaut being encoded by two
"numbers", the o-umlaut by one).

Anyway, it seems that Emacs (under GNU/Linux) is trying to interpret the
text in a wrong encoding. You can find out what coding system Emacs is
using at the moment with the key sequence "C-h C" -- you might want to
compare the answers under Windows and GNU/Linux.

Also: what does this file look like if you pass it through "hexdump -C"?

HTH
- -- tomás
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iD8DBQFKE5IMBcgs9XrR2kYRApJVAJ9/kbJ7WInELB4yyLKjH0LuZDEBMgCeKrAY
TZ+xMFL4iCq38cwBnBlJeTc=
=evDB
-----END PGP SIGNATURE-----

Re: problems displaying german "umlaute"

[Andreas Röhler]

Rainer Stengele wrote:
> Hi all,
>
> sorry this is surely a standard problem but I cannot find a fast
> solution.
>
> I am using textfiles (actually org-mode text) both on windows emacs 23
> (emacsw32-20090226) and under different linux variants (kubuntu,
> sidux) some running emacs 22 and some emacs 23.
>
> The problem is that the "umlauts" shown in the windows emacs without
> problems are shown as code under linux.
>
> I am using the exact same configuration (versioned by subversion),
> .emacs etc.
>
> So under windows I see:
>
> Hüpfreduzierspiel rekursiv lösen                        
>
> under linux I see:               
>
> H\303\274pfreduzierspiel rekursiv l\366sen                    
>
> What can I do?
>
> Thank you for any hint!
>
>
> Rainer   
>
>
>
>

Here it happens only occassionaly when taking chunks from the internet.

Reported it years ago.

Solution for me is to copy the whole buffer into a new one, where the
display is
correct then. Save the new buffer, its gone.


HTH

Andreas

Re: problems displaying german "umlaute"

[Rainer Stengele]

Rainer Stengele schrieb:
> Hi all,
> 
> sorry this is surely a standard problem but I cannot find a fast solution.
> 
> I am using textfiles (actually org-mode text) both on windows emacs 23 
> (emacsw32-20090226) and under different linux variants (kubuntu, sidux) 
> some running emacs 22 and some emacs 23.
> 
> The problem is that the "umlauts" shown in the windows emacs without 
> problems are shown as code under linux.
> 
> I am using the exact same configuration (versioned by subversion), 
> .emacs etc.
> 
> So under windows I see:
> 
> Hüpfreduzierspiel rekursiv lösen                        
> 
> under linux I see:               
> 
> H\303\274pfreduzierspiel rekursiv l\366sen                    
> 
> What can I do?
> 
> Thank you for any hint!
> 
> 
> Rainer   
> 
> 
> 
> 

This is still coming up from time to time.
I could not find anything wrong in the "codings", as far as I understand 
anything of it.
In order to proceed I wanted to write a function to replace all the 
wrong characters.
I have this her and it does not work, although doing a "replace-string" 
is working perefctly in the buffer.
"\374" is of course a character, not a string.


Can anybody give me a hint please!


(defun replace-misthaufen ()
   "replaces im gesamten buffer:
\374  => ü
\337  => ß
\344  => ä
\366  => ö"
   (interactive
    (beginning-of-buffer)
    (while (search-forward " " nil t)
      (replace-match "ü" nil t))
    ; (beginning-of-buffer)
    (while (search-forward " " nil t)
      (replace-match "ß" nil t))
    ; (beginning-of-buffer)
    (while (search-forward " " nil t)
      (replace-match "ä" nil t))
    ; (beginning-of-buffer)
    (while (search-forward " " nil t)
      (replace-match "ö" nil t))
    ))


Rainer

Re: problems displaying german "umlaute"

[Rainer Stengele]

Rainer Stengele schrieb:
> Rainer Stengele schrieb:
>> Hi all,
>>
>> sorry this is surely a standard problem but I cannot find a fast 
>> solution.
>>
>> I am using textfiles (actually org-mode text) both on windows emacs 23 
>> (emacsw32-20090226) and under different linux variants (kubuntu, 
>> sidux) some running emacs 22 and some emacs 23.
>>
>> The problem is that the "umlauts" shown in the windows emacs without 
>> problems are shown as code under linux.
>>
>> I am using the exact same configuration (versioned by subversion), 
>> .emacs etc.
>>
>> So under windows I see:
>>
>> Hüpfreduzierspiel rekursiv lösen                       
>> under linux I see:              
>> H\303\274pfreduzierspiel rekursiv l\366sen                   
>> What can I do?
>>
>> Thank you for any hint!
>>
>>
>> Rainer  
>>
>>
>>
> 
> This is still coming up from time to time.
> I could not find anything wrong in the "codings", as far as I understand 
> anything of it.
> In order to proceed I wanted to write a function to replace all the 
> wrong characters.
> I have this her and it does not work, although doing a "replace-string" 
> is working perefctly in the buffer.
> "\374" is of course a character, not a string.
> 
> 
> Can anybody give me a hint please!
> 
> 
> (defun replace-misthaufen ()
>   "replaces im gesamten buffer:
> \374  => ü
> \337  => ß
> \344  => ä
> \366  => ö"
>   (interactive
>    (beginning-of-buffer)
>    (while (search-forward " " nil t)
>      (replace-match "ü" nil t))
>    ; (beginning-of-buffer)
>    (while (search-forward " " nil t)
>      (replace-match "ß" nil t))
>    ; (beginning-of-buffer)
>    (while (search-forward " " nil t)
>      (replace-match "ä" nil t))
>    ; (beginning-of-buffer)
>    (while (search-forward " " nil t)
>      (replace-match "ö" nil t))
>    ))
> 
> 
> Rainer
> 
> 
> 
> 
sorry this is of course

(defun replace-misthaufen ()
   "ersetzt im gesamten buffer:
\374  => ü
\337  => ß
\344  => ä
\366  => ö"
   (interactive
    (beginning-of-buffer)
    (while (search-forward " " nil t)
      (replace-match "ü" nil t))
    (beginning-of-buffer)
    (while (search-forward " " nil t)
      (replace-match "ß" nil t))
    (beginning-of-buffer)
    (while (search-forward " " nil t)
      (replace-match "ä" nil t))
    (beginning-of-buffer)
    (while (search-forward " " nil t)
      (replace-match "ö" nil t))
    ))


which does not work either.


Rainer

Re: problems displaying german "umlaute"

[Rainer Stengele]

Rainer Stengele schrieb:
> Hi all,
> 
> sorry this is surely a standard problem but I cannot find a fast solution.
> 
> I am using textfiles (actually org-mode text) both on windows emacs 23 
> (emacsw32-20090226) and under different linux variants (kubuntu, sidux) 
> some running emacs 22 and some emacs 23.
> 
> The problem is that the "umlauts" shown in the windows emacs without 
> problems are shown as code under linux.
> 
> I am using the exact same configuration (versioned by subversion), 
> .emacs etc.
> 
> So under windows I see:
> 
> Hüpfreduzierspiel rekursiv lösen                        
> 
> under linux I see:               
> 
> H\303\274pfreduzierspiel rekursiv l\366sen                    
> 
> What can I do?
> 
> Thank you for any hint!
> 
> 
> Rainer   
> 
> 
> 
> 

ok, I have now replaced all wrong characters with the correct ones. I 
restart emacs and get this:

\303\237 for "ß"
\303\244 for "ä"

and so on.

What is this?

GNU Emacs 23.1.50.1 (i386-mingw-nt5.1.2600) of 2009-06-30 on 
LENNART-69DE564 (patched)


Rainer

Re: problems displaying german "umlaute"

[TomSW]

On Jul 3, 8:07 pm, Rainer Stengele <rainer.steng...@online.de> wrote:

> >> I am using textfiles (actually org-mode text) both on windows emacs 23
> >> (emacsw32-20090226) and under different linux variants (kubuntu,
> >> sidux) some running emacs 22 and some emacs 23.
>
> >> The problem is that the "umlauts" shown in the windows emacs without
> >> problems are shown as code under linux.

It looks as if you are opening utf-8 files as ascii or some other
unsuitable encoding. Try opening them as utf-8. For example, open the
file, then use the command M-x revert-buffer-with-coding-system to
reload the file with a different encoding. Specify the coding system
as "utf-8-unix". Does this fix the problem?

If it does, the second question is why Emacs isn't using the correct
encoding to start with. Load the file again, and check the value of
the variables enable-multibyte-characters and default-enable-multibyte-
characters. They should both be t - are they?

regards,
Tom SW

Re: problems displaying german "umlaute"

[Rainer Stengele]

TomSW schrieb:
> On Jul 3, 8:07 pm, Rainer Stengele <rainer.steng...@online.de> wrote:
> 
>>>> I am using textfiles (actually org-mode text) both on windows emacs 23
>>>> (emacsw32-20090226) and under different linux variants (kubuntu,
>>>> sidux) some running emacs 22 and some emacs 23.
>>>> The problem is that the "umlauts" shown in the windows emacs without
>>>> problems are shown as code under linux.
> 
> It looks as if you are opening utf-8 files as ascii or some other
> unsuitable encoding. Try opening them as utf-8. For example, open the
> file, then use the command M-x revert-buffer-with-coding-system to
> reload the file with a different encoding. Specify the coding system
> as "utf-8-unix". Does this fix the problem?

yes!

> 
> If it does, the second question is why Emacs isn't using the correct
> encoding to start with. Load the file again, and check the value of
> the variables enable-multibyte-characters and default-enable-multibyte-
> characters. They should both be t - are they?

enable-multibyte-characters is a variable defined in `C source code'.
Its value is t
Local in buffer *Help*; global value is t

default-enable-multibyte-characters is a variable defined in `C source 
code'.
Its value is t



So yes they are both "t". Now how can I set the right config option?
Strange thing is that another file I do load (another orgmode file) 
always comes up correctly.


thanx and regards,
Rainer

> 
> regards,
> Tom SW
> 

Re: problems displaying german "umlaute"

[Rainer Stengele]

Rainer Stengele schrieb:
> TomSW schrieb:
>> On Jul 3, 8:07 pm, Rainer Stengele <rainer.steng...@online.de> wrote:
>>
>>>>> I am using textfiles (actually org-mode text) both on windows emacs 23
>>>>> (emacsw32-20090226) and under different linux variants (kubuntu,
>>>>> sidux) some running emacs 22 and some emacs 23.
>>>>> The problem is that the "umlauts" shown in the windows emacs without
>>>>> problems are shown as code under linux.
>>
>> It looks as if you are opening utf-8 files as ascii or some other
>> unsuitable encoding. Try opening them as utf-8. For example, open the
>> file, then use the command M-x revert-buffer-with-coding-system to
>> reload the file with a different encoding. Specify the coding system
>> as "utf-8-unix". Does this fix the problem?
> 
> yes!
> 
>>
>> If it does, the second question is why Emacs isn't using the correct
>> encoding to start with. Load the file again, and check the value of
>> the variables enable-multibyte-characters and default-enable-multibyte-
>> characters. They should both be t - are they?
> 
> enable-multibyte-characters is a variable defined in `C source code'.
> Its value is t
> Local in buffer *Help*; global value is t
> 
> default-enable-multibyte-characters is a variable defined in `C source 
> code'.
> Its value is t
> 
> 
> 
> So yes they are both "t". Now how can I set the right config option?
> Strange thing is that another file I do load (another orgmode file) 
> always comes up correctly.
> 
> 
> thanx and regards,
> Rainer
> 
>>
>> regards,
>> Tom SW
>>
> 
> 
> 
> 

Ok now, adding the right setting for my org files to the variable 
file-coding-system-alist does it:

(custom-set-variables
...
  '(file-coding-system-alist (quote (("\\.org\\'" . utf-8-unix) ...
...

is that the right path or just a "dirty" workaround?

regards,
Rainer

Re: problems displaying german "umlaute"

[TomSW]

On Jul 3, 9:34 pm, Rainer Stengele <rainer.steng...@online.de> wrote:

> Ok now, adding the right setting for my org files to the variable
> file-coding-system-alist does it:
>
> (custom-set-variables
> ...
>   '(file-coding-system-alist (quote (("\\.org\\'" . utf-8-unix) ...
> ...
>
> is that the right path or just a "dirty" workaround?

It's just odd that it's necessary.

Try loading Emacs without any init files ("emacs -Q") and then open a
utf-8 text file, just to eliminate a few possible causes. If the file
comes out incorrectly, try evaluating (set-buffer-multibyte t) in the
same buffer - does it make a difference?

ΩŊŁÞØŦ®Ðª©łðß»¢“!!!
Tom

Re: problems displaying german "umlaute"

[Rainer Stengele]

TomSW schrieb:
> On Jul 3, 9:34 pm, Rainer Stengele <rainer.steng...@online.de> wrote:
> 
>> Ok now, adding the right setting for my org files to the variable
>> file-coding-system-alist does it:
>>
>> (custom-set-variables
>> ...
>>   '(file-coding-system-alist (quote (("\\.org\\'" . utf-8-unix) ...
>> ...
>>
>> is that the right path or just a "dirty" workaround?
> 
> It's just odd that it's necessary.
> 
> Try loading Emacs without any init files ("emacs -Q") and then open a
> utf-8 text file, just to eliminate a few possible causes. If the file
> comes out incorrectly, try evaluating (set-buffer-multibyte t) in the
> same buffer - does it make a difference?
> 
> ΩŊŁÞØŦ®Ðª©łðß»¢“!!!
> Tom
> 
> 

OK, opening the file after "emacs -Q" gives me:

[Ps 90:12  Lehre uns bedenken, da\303\237 wir sterben m\303\274ssen, auf 
da\303\237 wir klug werden.]


After evaluating (set-buffer-multibyte t) and reloading the buffer the 
result is wrong still...


Regards and thanks,
Rainer

Re: problems displaying german "umlaute"

[TomSW]

On Jul 4, 11:14 pm, Rainer Stengele <rainer.steng...@online.de> wrote:

> OK, opening the file after "emacs -Q" gives me:
>
> [Ps 90:12  Lehre uns bedenken, da\303\237 wir sterben m\303\274ssen, auf
> da\303\237 wir klug werden.]
>
> After evaluating (set-buffer-multibyte t) and reloading the buffer the
> result is wrong still...

It looks as if Emacs is reading utf-8 files as ascii, I don't know
why. After loading the file, try the command describe-coding-system,
just to confirm what coding system is being used.

To fix the problem you can change file-coding-system-alist, or use set-
default-coding-systems, somewhere in your .emacs. Just seems odd that
it is necessary.
regards,
Tom

Re: problems displaying german "umlaute"

[Rainer Stengele]

TomSW schrieb:
> On Jul 4, 11:14 pm, Rainer Stengele <rainer.steng...@online.de> wrote:
> 
>> OK, opening the file after "emacs -Q" gives me:
>>
>> [Ps 90:12  Lehre uns bedenken, da\303\237 wir sterben m\303\274ssen, auf
>> da\303\237 wir klug werden.]
>>
>> After evaluating (set-buffer-multibyte t) and reloading the buffer the
>> result is wrong still...
> 
> It looks as if Emacs is reading utf-8 files as ascii, I don't know
> why. After loading the file, try the command describe-coding-system,
> just to confirm what coding system is being used.
> 
> To fix the problem you can change file-coding-system-alist, or use set-
> default-coding-systems, somewhere in your .emacs. Just seems odd that
> it is necessary.
> regards,
> Tom
> 
> 
Oh dear,  I am totally desperate.
The bad characters are coming up all the times and getting more and more.
No matter which coding system I use they reappear.

What am I doing wrong?
Can anybody help me setting up a correct coding system framework to work with german umlauts?

I tried to write a function to replace the characters but it does not work, something like:

(defun replace-misthaufen ()
  "ersetzt im gesamten buffer:
ü => ü
ß => ß
ä => ä
ö => ö"
  (interactive
   (beginning-of-buffer)
   (while (search-forward "ü" nil t)
     (replace-match "ü" nil t))
   (beginning-of-buffer)
   (while (search-forward "ß" nil t)
     (replace-match "ß" nil t))
   (beginning-of-buffer)
   (while (search-forward "ä" nil t)
     (replace-match "ä" nil t))
   (beginning-of-buffer)
   (while (search-forward "ö" nil t)
     (replace-match "ö" nil t))
   ))


Looks crazy I know, but the reason is that even in this buffer everything comes up wrong so I do not know how to continue.

Thanks for any help,

- rainer