umlauts (8bit characters) input

umlauts (8bit characters) input

[Hendrik Sattler]

Hi,

8bit characters work fine everywhere(bash, mcedit, etc) except with emacs.
When I press 'ü', it asks me for "Shell command on region:", 'ö' goes to
"beginning of buffer" and 'ä' does nothing.
The editing is done in KDE's konsole, locale is set to
LANG=de_DE@euro
LC_CTYPE="de_DE@euro"
LC_NUMERIC="de_DE@euro"
LC_TIME="de_DE@euro"
LC_COLLATE=C
LC_MONETARY="de_DE@euro"
LC_MESSAGES="de_DE@euro"
LC_PAPER="de_DE@euro"
LC_NAME="de_DE@euro"
LC_ADDRESS="de_DE@euro"
LC_TELEPHONE="de_DE@euro"
LC_MEASUREMENT="de_DE@euro"
LC_IDENTIFICATION="de_DE@euro"
LC_ALL=

I had to reinstall my system and took my configuration files with me. I
worked before, why not now?
Here's my ~/,emacs:
(custom-set-variables
  ;; custom-set-variables was added by Custom -- don't edit or cut/paste it!
  ;; Your init file should contain only one such instance.
 '(bar-cursor 2)
 '(case-fold-search t)
 '(current-language-environment "German")
 '(default-input-method "german-postfix")
 '(global-font-lock-mode t nil (font-lock))
 '(gnuserv-program (concat exec-directory "/gnuserv"))
 '(show-paren-mode t nil (paren))
 '(transient-mark-mode t))
(custom-set-faces
  ;; custom-set-faces was added by Custom -- don't edit or cut/paste it!
  ;; Your init file should contain only one such instance.
 )

Any suggestions?

Thanks

HS

Re: umlauts (8bit characters) input

[Peter Dyballa]

Am 31.01.2005 um 17:44 schrieb Hendrik Sattler:

>  '(default-input-method "german-postfix")

Doesn't this setting mean to input all umlauts as `a"´?

--
Mit friedvollen Grüßen

   Pete

A common mistake that people make when trying to design something 
completely foolproof is to underestimate the ingenuity of complete 
fools.

Re: umlauts (8bit characters) input

[Hendrik Sattler]

Peter Dyballa wrote:
> Am 31.01.2005 um 17:44 schrieb Hendrik Sattler:
> 
>>  '(default-input-method "german-postfix")
> 
> Doesn't this setting mean to input all umlauts as `a"??

Does that imply to ignore a real 'ä' key? I deleted that line but the
behaviour did not change.

HS

Re: umlauts (8bit characters) input

[Peter Dyballa]

Am 31.01.2005 um 23:44 schrieb Hendrik Sattler:

> Peter Dyballa wrote:
>> Am 31.01.2005 um 17:44 schrieb Hendrik Sattler:
>>
>>>  '(default-input-method "german-postfix")
>>
>> Doesn't this setting mean to input all umlauts as `a"??
>
> Does that imply to ignore a real 'ä' key? I deleted that line but the
> behaviour did not change.

Actually I can't tell -- still searching for some answers. Here are 
some answers from my .emacs:

;       (set-language-environment               'UTF-8)
         (set-default-coding-systems             'utf-8)
         (setq file-name-coding-system           'utf-8)
         (setq default-buffer-file-coding-system 'utf-8)
         (setq coding-system-for-write           'utf-8)
;       (set-keyboard-coding-system             'utf-8)
         (set-terminal-coding-system             'utf-8)
;;      (set-clipboard-coding-system            'utf-8)
;;      (set-selection-coding-system            'utf-8)
;       (set-language-environment               'German)

This is for X11 on Mac OS X. The following is for "Carbon Emacsen", 
i.e. a GNU Emacs that fits into the Aqua environment. These have some 
restrictions with fonts or fontsets, so restrict them to use ISO Latin.

         (setq file-name-coding-system           'utf-8)
         (setq default-buffer-file-coding-system 'iso-latin-9-unix)
         (set-language-environment               'German)
         (set-default-coding-systems             'iso-latin-9-unix)
         (set-keyboard-coding-system             'iso-latin-9)
         (prefer-coding-system                   'iso-latin-9-unix)

I think the most important is default-buffer-file-coding-system. It 
makes Emacs aware and work with umlauts. Using the right fontsets (or 
at least fonts) should enable you to see the umlauts too. If you here 
the umlauts when you type them you might need 
set-keyboard-coding-system and/or set-terminal-coding-system. It's 
always helpful to check what Emacs actually receives: C-h k and then 
you type the umlaut.

Probably your Linux system too uses UTF-8 as file-name-coding-system ...

--
Mit friedvollen Grüßen

   Pete

"They're putting dimes in the hole in my head to see the change in me."

Re: umlauts (8bit characters) input

[Hendrik Sattler]

Peter Dyballa wrote:
> ;       (set-language-environment               'UTF-8)
>          (set-default-coding-systems             'utf-8)
>          (setq file-name-coding-system           'utf-8)
>          (setq default-buffer-file-coding-system 'utf-8)
>          (setq coding-system-for-write           'utf-8)
> ;       (set-keyboard-coding-system             'utf-8)
>          (set-terminal-coding-system             'utf-8)
> ;;      (set-clipboard-coding-system            'utf-8)
> ;;      (set-selection-coding-system            'utf-8)
> ;       (set-language-environment               'German)

I don't understand why emacs should need all that. There are locales and
they define a system charset, emacs should use that as default.
 
> It's always helpful to check what Emacs actually receives: C-h k and then
> you type the umlaut.

It gets M-| for the 'ü', '|' is 0x7C and 'ü' ist 0xFC, so the 8th bit is
ignored.
Does emacs care for readline settings?

> Probably your Linux system too uses UTF-8 as file-name-coding-system ...

No, the locale de_DE@euro has ISO-8859-15 as character set, nothing fancy.

I really want to get that guy that said that ASCII and thus 7bit is a useful
default :-(((((

HS

Re: umlauts (8bit characters) input

[Peter Dyballa]

Am 01.02.2005 um 01:41 schrieb Hendrik Sattler:

> Peter Dyballa wrote:
>> ;       (set-language-environment               'UTF-8)
>>          (set-default-coding-systems             'utf-8)
>>          (setq file-name-coding-system           'utf-8)
>>          (setq default-buffer-file-coding-system 'utf-8)
>>          (setq coding-system-for-write           'utf-8)
>> ;       (set-keyboard-coding-system             'utf-8)
>>          (set-terminal-coding-system             'utf-8)
>> ;;      (set-clipboard-coding-system            'utf-8)
>> ;;      (set-selection-coding-system            'utf-8)
>> ;       (set-language-environment               'German)
>
> I don't understand why emacs should need all that. There are locales 
> and
> they define a system charset, emacs should use that as default.

These setting show how complicated it is to translate between the many 
systems GNU Emacs supports. It's not only Latin scripts! Have a look at 
the HELLO page: C-h H. And particularly two settings can become import 
when you're interacting with a window system: clipboard-coding-system 
and selection-coding-system. There is in the Help menu a very excessive 
entry: Describe -> Show all of Mule Status. Or simply M-x mule-diag. 
It'll give you pointers to stay for a few weeks in info-mode ...

>
>> It's always helpful to check what Emacs actually receives: C-h k and 
>> then
>> you type the umlaut.
>
> It gets M-| for the 'ü', '|' is 0x7C and 'ü' ist 0xFC, so the 8th bit 
> is
> ignored.
> Does emacs care for readline settings?

I don't think so. readline would be used in shell-mode or eshell-mode, 
if at all. Are you running Emacs freely as an X11 client or is it 
no-windows inside a terminal emulation? In the second case you're a bit 
dependant of what the terminal emulation is doing. It might convert 
8bit to 7bit. In both cases you have to set an appropriate 
input-method, as mule-diag will reveal.

Do you have test files à la 'char-representation = oct dec hex its 
description (to help you recognize what Emacs is doing to that glyph)', 
for example "ü = 374 = 252 = FC = U+00FC : LATIN SMALL LETTER U WITH 
DIAERESIS"? In any case it would be very helpful (for Emacs) to give a 
hint like "-*- mode: Text; coding: iso-8859-15; -*-" in the file's 
header to help it choose the right character set.

I do hope that others on this list have made more progress than both of 
us and add some to this thread! For example I would like to have shell 
and dired-buffers display in UTF-8 because the file names in Mac OS X 
are of that kind. In shell it works with ls when I let Emacs run in 
Apple's Terminal (emulation).

I hope that GNU Emacs 25 solves all problems.


> I really want to get that guy that said that ASCII and thus 7bit is a 
> useful
> default :-(((((

Oh no, why has the world to be so stubborn and keep writing in so my 
scripts? Why is a comic's use of script not sufficient?

--
Mit friedvollen Grüßen

   Pete      <\
              _\     O  _
             |o \  _\\_/-\='
_____________(_)|-(_)  (_)___________________________________

Re: umlauts (8bit characters) input

[Hendrik Sattler]

Peter Dyballa wrote:
> Are you running Emacs freely as an X11 client or is it
> no-windows inside a terminal emulation? In the second case you're a bit
> dependant of what the terminal emulation is doing. It might convert
> 8bit to 7bit.

Yes, it is a no-windows instance inside KDE's konsole because the X emacs
interface is just sooo ugly. Konsole does not strip of the 8th bit, as
EVERYTHING els is working just fine (bash, mcedit, even VI!). The absolutly
only exception is emacs.
And yes, if I change my locale to UTF-8, emacs shall just follow case
without me having to edit the whole .emacs file again.
Maybe I just stick with other editors for now...

Thanks for trying to help on this...

HS

Re: umlauts (8bit characters) input

[David Kastrup]

Hendrik Sattler <sattler2000@gmx.de> writes:

> Peter Dyballa wrote:
>> Are you running Emacs freely as an X11 client or is it
>> no-windows inside a terminal emulation? In the second case you're a bit
>> dependant of what the terminal emulation is doing. It might convert
>> 8bit to 7bit.
>
> Yes, it is a no-windows instance inside KDE's konsole because the X emacs
> interface is just sooo ugly. Konsole does not strip of the 8th bit, as
> EVERYTHING els is working just fine (bash, mcedit, even VI!). The absolutly
> only exception is emacs.
> And yes, if I change my locale to UTF-8, emacs shall just follow case
> without me having to edit the whole .emacs file again.

And Emacs does just that.  Throw out whatever customization of locales
and whatsoever you have in your .emacs, or start with an empty .emacs,
and it should work.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: umlauts (8bit characters) input

[Reiner Steib]

Hendrik Sattler wrote:

> 8bit characters work fine everywhere(bash, mcedit, etc) except with emacs.
> When I press 'ü', it asks me for "Shell command on region:", 'ö' goes to
> "beginning of buffer" and 'ä' does nothing.
> The editing is done in KDE's konsole, locale is set to
> LANG=de_DE@euro
> LC_CTYPE="de_DE@euro"
[...]
> LC_ALL=

[...]
> Yes, it is a no-windows instance inside KDE's konsole 

That is an important information that was missing in your previous
messages.

Does it help to set...

(set-keyboard-coding-system 'iso-8859-15)

... in your ~/.emacs file?  (You shouldn't need any other settings if
the locale is setup correctly.)

> because the X emacs interface is just sooo ugly. Konsole does not
> strip of the 8th bit, as EVERYTHING els is working just fine (bash,
> mcedit, even VI!). The absolutly only exception is emacs.
> And yes, if I change my locale to UTF-8, emacs shall just follow case
> without me having to edit the whole .emacs file again.

Emacs does initialize the setting from the locale.  Initialization of
`keyboard-coding-system' works correctly in the X version without any
other settings.  I'm not sure why `set-keyboard-coding-system' is
required in Emacs 21.3 when using the no-window mode (I assume you use
21.3; you didn't mention your Emacs version).

In the next (not yet released) version of Emacs, the initialization
works fine, both with and without X, AFAICS.

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

Re: umlauts (8bit characters) input

[Hendrik Sattler]

David Kastrup wrote:

> Hendrik Sattler <sattler2000@gmx.de> writes:
> 
>> Peter Dyballa wrote:
>>> Are you running Emacs freely as an X11 client or is it
>>> no-windows inside a terminal emulation? In the second case you're a bit
>>> dependant of what the terminal emulation is doing. It might convert
>>> 8bit to 7bit.
>>
>> Yes, it is a no-windows instance inside KDE's konsole because the X emacs
>> interface is just sooo ugly. Konsole does not strip of the 8th bit, as
>> EVERYTHING els is working just fine (bash, mcedit, even VI!). The
>> absolutly only exception is emacs.
>> And yes, if I change my locale to UTF-8, emacs shall just follow case
>> without me having to edit the whole .emacs file again.
> 
> And Emacs does just that.  Throw out whatever customization of locales
> and whatsoever you have in your .emacs, or start with an empty .emacs,
> and it should work.

Thanks, that was the right hint :)
I made a "rm -rf .emacs*" in my home dir and started emacs, still no
umlauts. However, you say it works with that so I went to a VT and started
emacs there: works.
Then I started a minimal X "startx /usr/bin/X11/xterm -- :1" and it does not
work, there. Hmm, must be some X thing, then.

I looked at all environment variables: lots of differences but the main
thing: TERM=linux works, TERM=xterm does not work.

That make me look for why this is like that and I straced emacs to find
open("/usr/share/emacs/21.3/lisp/term/xterm.el", O_RDONLY|O_LARGEFILE) = 3
Hey, that's just so easy:
/usr/share/emacs/21.3/lisp/term# mv xterm.el xterm.el.old
/usr/share/emacs/21.3/lisp/term# cp linux.el xterm.el

And yes, works now. The main difference is that xterm.el adds:
(define-key function-key-map "\e[A" [up])
(define-key function-key-map "\e[B" [down])
(define-key function-key-map "\e[C" [right])
(define-key function-key-map "\e[D" [left])
(define-key function-key-map "\e[2~" [insert])
(define-key function-key-map "\e[4~" [select])
(define-key function-key-map "\e[5~" [prior])
(define-key function-key-map "\e[6~" [next])
(define-key function-key-map "\e[11~" [f1])
(define-key function-key-map "\e[12~" [f2])
(define-key function-key-map "\e[13~" [f3])
(define-key function-key-map "\e[14~" [f4])
(define-key function-key-map "\e[15~" [f5])
(define-key function-key-map "\e[17~" [f6])
(define-key function-key-map "\e[18~" [f7])
(define-key function-key-map "\e[19~" [f8])
(define-key function-key-map "\e[20~" [f9])
(define-key function-key-map "\e[21~" [f10])
(define-key function-key-map "\e[23~" [f11])
(define-key function-key-map "\e[24~" [f12])
(define-key function-key-map "\e[29~" [print])

What do those do? I am not very familiar with escape codes but should I file
a bug report against the Debian package?
What's the bug, then: the wrong TERM env var or the above file?

HS

Re: umlauts (8bit characters) input

[Hendrik Sattler]

Reiner Steib wrote:
> Emacs does initialize the setting from the locale.  Initialization of
> `keyboard-coding-system' works correctly in the X version without any
> other settings.  I'm not sure why `set-keyboard-coding-system' is
> required in Emacs 21.3 when using the no-window mode (I assume you use
> 21.3; you didn't mention your Emacs version).

Yes, I use 21.3, I also found the cause for it, see my answer to David.

HS

Re: umlauts (8bit characters) input

[Peter Dyballa]

Am 01.02.2005 um 12:37 schrieb Hendrik Sattler:

> Konsole does not strip of the 8th bit, as
> EVERYTHING els is working just fine (bash, mcedit, even VI!). The 
> absolutly
> only exception is emacs.

Hmm, that's very interesting ... to me things behave completely 
different: in Apple's Terminal.app vi (actually vim) cannot display any 
8bit chars, I have to use GNU Emacs 21.3.50 to see them. When I launch 
the same as an X client or as a Carbon Application I can see and type 
these same umlauts, except that shell und dired-mode refuse to show me 
file names containing these umlauts.

I think terminal-coding-system can be an issue.

--
Mit friedvollen Grüßen

   Pete

Linux bietet Lösungen für Probleme unter Windows, die mit Mac OS X gar 
nicht erst auftreten.

Re: umlauts (8bit characters) input

[Peter Dyballa]

Am 01.02.2005 um 14:37 schrieb Hendrik Sattler:

> And yes, works now. The main difference is that xterm.el adds:
> (define-key function-key-map "\e[A" [up])
> (define-key function-key-map "\e[B" [down])
> (define-key function-key-map "\e[C" [right])
> (define-key function-key-map "\e[D" [left])

These and the others are the ANSI escape codes these keys generate. (If 
you need proof: type C-q and then hit the funktion or numerical keypad 
key -- you'll see such a sequence.) xterm.el simply assign these 
multiple bytes sequences a name or function that Emacs can handle these 
events when running inside a terminal emulation.

--
Mit friedvollen Grüßen

   Pete

"Computers are good at following instructions,
but not at reading your mind."
D. E. Knuth, The TeXbook, Addison-Wesley 1984, 1986, 1996, p. 9

Re: umlauts (8bit characters) input

[Hendrik Sattler]

Hi,

lisp/term/linux.el did some more than nothing:
;; The Linux console handles Latin-1 by default.
(unless (terminal-coding-system)
  (set-terminal-coding-system 'iso-latin-1))

;; Make Latin-1 input characters work, too.
;; Meta will continue to work, because the kernel
;; turns that into Escape.
(let ((value (current-input-mode)))
  ;; The third arg only matters in that it is not t or nil.
  (set-input-mode (nth 0 value) (nth 1 value) 'iso-latin-1 (nth 3 value)))

I just copied it over to my ~/.emacs and it works now. So, to correct David,
emacs does obviously NOT follow locales idea of character set (at least not
for everything).
Maybe in my previous installation, I just had TERM fixed to "linux" to solve
this, I just don't remember...

But you are right: maybe emacs gets to point where it magically
"just works".

HS

Re: umlauts (8bit characters) input

[David Kastrup]

Hendrik Sattler <sattler2000@gmx.de> writes:

> lisp/term/linux.el did some more than nothing:
> ;; The Linux console handles Latin-1 by default.
> (unless (terminal-coding-system)
>   (set-terminal-coding-system 'iso-latin-1))
>
> ;; Make Latin-1 input characters work, too.
> ;; Meta will continue to work, because the kernel
> ;; turns that into Escape.
> (let ((value (current-input-mode)))
>   ;; The third arg only matters in that it is not t or nil.
>   (set-input-mode (nth 0 value) (nth 1 value) 'iso-latin-1 (nth 3 value)))
>
> I just copied it over to my ~/.emacs and it works now. So, to correct David,
> emacs does obviously NOT follow locales idea of character set (at least not
> for everything).
> Maybe in my previous installation, I just had TERM fixed to "linux" to solve
> this, I just don't remember...
>
> But you are right: maybe emacs gets to point where it magically
> "just works".

I have just verified that the development version of Emacs will start
up 8-bit transparent if the LC_CTYPE locale is indicating an 8-bit
character set.  So this will work out of the box with Emacs-21.4.

I am afraid that I have not been able to find the manual methods from
term/linux.el in the Emacs-21.3 manual, though I am pretty sure that
at one point of time, the above somewhat arcane way via set-input-mode
was described there.  Maybe the documentation changed faster than the
code.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: umlauts (8bit characters) input

[Hendrik Sattler]

David Kastrup wrote:

> I have just verified that the development version of Emacs will start
> up 8-bit transparent if the LC_CTYPE locale is indicating an 8-bit
> character set.  So this will work out of the box with Emacs-21.4.
> 
> I am afraid that I have not been able to find the manual methods from
> term/linux.el in the Emacs-21.3 manual, though I am pretty sure that
> at one point of time, the above somewhat arcane way via set-input-mode
> was described there.  Maybe the documentation changed faster than the
> code.

Great news :-D

HS

Re: umlauts (8bit characters) input

[Peter Dyballa]

Am 01.02.2005 um 13:36 schrieb David Kastrup:

>> And yes, if I change my locale to UTF-8, emacs shall just follow case
>> without me having to edit the whole .emacs file again.
>
> And Emacs does just that.  Throw out whatever customization of locales
> and whatsoever you have in your .emacs, or start with an empty .emacs,
> and it should work.

Well, throwing away all this "valuable" customization leads in Terminal 
to dired-buffers with 'uuu' encoding -- and short and precise file 
names with real umlauts! But in shell they're gone ... a bunch of 
control codes? (I am using tcsh and see 
a\314?o\314?u\314?\303?U\314?O\314?A\314?\342?\254 instead of äöü?ÜÖÄÛ, 
shell has only 'uu-')

The X11 client is even worse: copy and paste in this same Emacs does 
not work (a simple ä is converted to a whole book volume of glyphs that 
are hard to describe), dired-mode uses a simple 'u' (mode-line goes 
'-u:%%...') and displays 'a<box>o<box>u...Û', and shell uses no mode 
'--:**...' as in Terminal and shows the name as in Terminal.

In xterm shell has '-uu-:**..' and shows the same as in Terminal. Again 
dired works fine!

So it seems I have to wait for GNU Emacs 25 -- if I'm still alive!

--
Mit friedvollen Grüßen

   Pete

Es ist, glaub' ich, nichts so heiß, wie's gerne scheinen möchte.

Re: umlauts (8bit characters) input

[David Kastrup]

Peter Dyballa <Peter_Dyballa@Web.DE> writes:

> Am 01.02.2005 um 13:36 schrieb David Kastrup:
>
>>> And yes, if I change my locale to UTF-8, emacs shall just follow case
>>> without me having to edit the whole .emacs file again.
>>
>> And Emacs does just that.  Throw out whatever customization of locales
>> and whatsoever you have in your .emacs, or start with an empty .emacs,
>> and it should work.
>
> Well, throwing away all this "valuable" customization leads in
> Terminal to dired-buffers with 'uuu' encoding -- and short and precise
> file names with real umlauts! But in shell they're gone ... a bunch of
> control codes? (I am using tcsh and see
> a\314?o\314?u\314?\303?U\314?O\314?A\314?\342?\254 instead of
> äöü?ÜÖÄÛ, shell has only 'uu-')

In "Terminal" vs "in Shell"?  What are you doing?

> The X11 client is even worse: copy and paste in this same Emacs does
> not work (a simple ä is converted to a whole book volume of glyphs
> that are hard to describe), dired-mode uses a simple 'u' (mode-line
> goes '-u:%%...') and displays 'a<box>o<box>u...Û', and shell uses no
> mode '--:**...' as in Terminal and shows the name as in Terminal.
>
> In xterm shell has '-uu-:**..' and shows the same as in
> Terminal. Again dired works fine!
>
> So it seems I have to wait for GNU Emacs 25 -- if I'm still alive!

I don't think that any of those problems are in the current code base
(to become 21.4), and apart from the X11 copy&paste (where there have
been larger changes since 21.3), I doubt that most of the stuff you
relate will happen on a clean, normal setup with consistent locale
settings.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: umlauts (8bit characters) input

[Peter Dyballa]

Am 02.02.2005 um 13:38 schrieb David Kastrup:

> In "Terminal" vs "in Shell"?  What are you doing?

I am using Mac OS X 10.3.7, also known as Panther. It has an 
application called "Terminal" or "Terminal.app" that is a terminal 
emulation. Inside it a tcsh is running -- I am too old to convert to 
bash so I stand to my very old decision to use a nice shell with 
history and substitution. And to GNU Emacs too.

--
Greetings

   Pete

Re: umlauts (8bit characters) input

[David Kastrup]

Peter Dyballa <Peter_Dyballa@Web.DE> writes:

> Am 02.02.2005 um 13:38 schrieb David Kastrup:
>
>> In "Terminal" vs "in Shell"?  What are you doing?
>
> I am using Mac OS X 10.3.7, also known as Panther.

Which is not exactly renowned as being very POSIX-compliant.

> It has an application called "Terminal" or "Terminal.app" that is a
> terminal emulation. Inside it a tcsh is running -- I am too old to
> convert to bash so I stand to my very old decision to use a nice
> shell with history and substitution. And to GNU Emacs too.

So you are using some textterminal thingy that apparently is not
overly consistent with what the locale specifies.

Try a unicode-capable xterm instead.  Maybe that helps.

I am afraid your problem is more Mac-specific than Emacs-specific.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: umlauts (8bit characters) input

[Ismael Valladolid Torres]

David Kastrup escribe:
> Peter Dyballa <Peter_Dyballa@Web.DE> writes:
> 
> > Am 02.02.2005 um 13:38 schrieb David Kastrup:
> >
> >> In "Terminal" vs "in Shell"?  What are you doing?
> >
> > I am using Mac OS X 10.3.7, also known as Panther.
> 
> Which is not exactly renowned as being very POSIX-compliant.
> 
> > It has an application called "Terminal" or "Terminal.app" that is a
> > terminal emulation. Inside it a tcsh is running -- I am too old to
> > convert to bash so I stand to my very old decision to use a nice
> > shell with history and substitution. And to GNU Emacs too.
> 
> So you are using some textterminal thingy that apparently is not
> overly consistent with what the locale specifies.
> 
> Try a unicode-capable xterm instead.  Maybe that helps.
> 
> I am afraid your problem is more Mac-specific than Emacs-specific.
> 

When I was younger so much younger than today, I never achieved Emacs
managing properly latin characters on Terminal.app nor in any of its
substitutes (sorry I don't remember their names). However I achieved
compiling CVS Emacs as a Carbon app and worked lovely.

Information [1]here was damn useful!

Cordially, Ismael

1. http://members.shaw.ca/akochoi-emacs/

Re: umlauts (8bit characters) input

[Stefan Monnier]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=utf-8, Size: 1792 bytes --]

> dired-buffers with 'uuu' encoding -- and short and precise file names with

Please read the manual that explains what the "uuu" mean.

There are many places where things can go wrong and it helps if you
understand a bit of the possible different places.

E.g. the decoding of files, the decoding of file names, the decoding of
keyboard input, and the display of chars are 4 different and
independent issues.

> real umlauts! But in shell they're gone ... a bunch of control codes? (I
> am using tcsh and see a\314?o\314?u\314?\303?U\314?O\314?A\314?\342?\254
> instead of äöü?ÜÖÄÛ, shell has only 'uu-')

Could you try posting in a more widely available charset than mac-roman?
The last char of "uu-" is meaningless in a shell buffer (it's the encoding
of the file associated with the buffer, and since shell buffers have no
associated file, ...).

> The X11 client is even worse: copy and paste in this same Emacs does not
> work (a simple ä is converted to a whole book volume of glyphs that are
> hard to describe),

Huh?  You mean you can't copy&paste from Emacs to itself correctly?
Or do you mean you can't correctly copy from Emacs to something else?
Or you can't correctly paste from something else to Emacs?

Please be super extra overly precise as if talking to complete idiots.

> and displays 'a<box>o<box>u...Û', and shell uses no mode '--:**...' as in
> Terminal and shows the name as in Terminal.

The <box> just means that it couldn't find a font to display the char.
Go to that char and hit C-u C-x = to see which char it is.  Maybe you just
need to help Emacs find the right font.

But of course, first we need to know which Emacs version you're running.
If it's Emacs-CVS, please move this discussion to emacs-devel@gnu.org or
emacs-pretest-bug@gnu.org.


        Stefan

Re: umlauts (8bit characters) input

[Peter Dyballa]

Am 02.02.2005 um 15:10 schrieb David Kastrup:

> Try a unicode-capable xterm instead.  Maybe that helps.

No, it won't. As I described before, xterm and Terminal are equally 
incapable to display UTF-8 file names in shell.

--
Greetings

   Pete

“One cannot live by television, video games, top ten CDs, and dumb 
movies alone”
       (Amiri Baraka 1999)

Re: umlauts (8bit characters) input

[Peter Dyballa]

Am 02.02.2005 um 15:59 schrieb Ismael Valladolid Torres:

> However I achieved compiling CVS Emacs as a Carbon app and worked 
> lovely.

Compiling nowadays even works in Xcode. I can easily believe that you 
don't have problems to type and read in a Latin script, this works 
because the fonts and fontsets are made to support mac-roman encoding. 
So everything's fine. But try to create file names with accented 
characters from the mac-roman range and look if you see them correctly, 
in Emacs and in Finder! The file names are encoded in UTF-8, that's the 
standard in Mac OS X (and modern Linux).

--
Greetings

   Pete

Re: umlauts (8bit characters) input

[Peter Dyballa]

Am 02.02.2005 um 16:19 schrieb Stefan Monnier:

>
>> real umlauts! But in shell they're gone ... a bunch of control codes? 
>> (I
>> am using tcsh and see 
>> a\314?o\314?u\314?\303?U\314?O\314?A\314?\342?\254
>> instead of äöü?ÜÖÄÛ, shell has only 'uu-')
>
> Could you try posting in a more widely available charset than 
> mac-roman?
> The last char of "uu-" is meaningless in a shell buffer (it's the 
> encoding
> of the file associated with the buffer, and since shell buffers have no
> associated file, ...).

Sorry, this is no mac-roman, it's exactly what I see in xterm or in 
Terminal in the shell buffer -- although I should see something 
different: äöüßÜÖÄ€. That's the file's name. (Mail.app claims that it 
is using a 7bit encoding.)

>
>> The X11 client is even worse: copy and paste in this same Emacs does 
>> not
>> work (a simple ä is converted to a whole book volume of glyphs that 
>> are
>> hard to describe),
>
> Huh?  You mean you can't copy&paste from Emacs to itself correctly?
> Or do you mean you can't correctly copy from Emacs to something else?
> Or you can't correctly paste from something else to Emacs?

My .emacs file is only 8bit, so I chose ISO Latin-15. In its header it 
is explicitly written

;;; -*- mode: Emacs-Lisp; coding: iso-8859-15; -*-

When I copy a string with characters from this set via double-clicking 
the selection and paste it either in the same or in the (Unicode) 
scratch buffer the simple accented chars become each a series of this 
character and mostly \216 -- me and others have seen similiar things in 
GNOME under Linux when copying from outside GNU Emacs. Pressing umlauts 
etc. on my keyboard enter exactly these umlauts into the buffers. C-x = 
explains in hex that they're taken from the usual code positions (for 
example Ä is 0x8c4, ä is 0x8e4).

I can send you a snapshot privately ...


>> and displays 'a<box>o<box>u...Û', and shell uses no mode '--:**...' 
>> as in
>> Terminal and shows the name as in Terminal.
>
> The <box> just means that it couldn't find a font to display the char.
> Go to that char and hit C-u C-x = to see which char it is.  Maybe you 
> just
> need to help Emacs find the right font.

There is no char. So there should not be a box. The correct string is 
äöüß... and as it's a file name, it's UTF-8 encoded. (Since the 
displayed glyphs are the chars stripped off their diaeresis the box 
could be ... but it's : "(01211310, 332488, 0x512c8, file ...). A bit 
astronomical high. æ and Æ and € are represented by themselves.

> But of course, first we need to know which Emacs version you're 
> running.
> If it's Emacs-CVS, please move this discussion to emacs-devel@gnu.org 
> or
> emacs-pretest-bug@gnu.org.

Since the Fink team decided to distribute in the unstable section an 
Emacs from CVS ... (but there was not much difference to 21.3.50, it 
might be exactly 0 in this case)

--
Greetings

   Pete

Re: umlauts (8bit characters) input

[Stefan Monnier]

[ Since you're using an Emacs-CVS version, then please, as I mentioned in
  my previous message, move this discussion elsewhere. ]

>> Could you try posting in a more widely available charset than mac-roman?
>> The last char of "uu-" is meaningless in a shell buffer (it's the encoding
>> of the file associated with the buffer, and since shell buffers have no
>> associated file, ...).

> Sorry, this is no mac-roman,

Tell that to your mailer:

Mime-Version: 1.0 (Apple Message framework v619.2)
Content-Type: text/plain; charset=X-MAC-ROMAN-LATIN1; format=flowed
Content-Transfer-Encoding: quoted-printable

> it's exactly what I see in xterm or in Terminal in the shell buffer --
> although I should see something different: äöüßÜÖÄ€.  That's the file's
> name. (Mail.app claims that it is using a 7bit encoding.)

[ This time I do see your non-ASCII chars properly, and the message is
  labelled as Content-Type: text/plain; charset=WINDOWS-1252; format=flowed ]

Actually, 7bit encoding (i.e. quoted-printable or base64) has nothing to do
with charset encoding.

>>> The X11 client is even worse: copy and paste in this same Emacs does not
>>> work (a simple ä is converted to a whole book volume of glyphs that are
>>> hard to describe),
>> 
>> Huh?  You mean you can't copy&paste from Emacs to itself correctly?
>> Or do you mean you can't correctly copy from Emacs to something else?
>> Or you can't correctly paste from something else to Emacs?

> My .emacs file is only 8bit, so I chose ISO Latin-15. In its header it is
> explicitly written
> ;;; -*- mode: Emacs-Lisp; coding: iso-8859-15; -*-

This has no influence on the behavior of Emacs, other than how it reads this
particular file.

> When I copy a string with characters from this set via double-clicking the
> selection and paste it either in the same or in the (Unicode) scratch buffer
> the simple accented chars become each a series of this character and mostly
> \216 -- me and others have seen similiar things in GNOME under Linux when
> copying from outside GNU Emacs. Pressing umlauts etc. on my keyboard enter
> exactly these umlauts into the buffers. C-x = explains in hex that they're
> taken from the usual code positions (for example Ä is 0x8c4, ä is 0x8e4).

What are you waiting for to M-x report-emacs-bug ?

> I can send you a snapshot privately ...

That wouldn't do you any good.

>>> and displays 'a<box>o<box>u...Û', and shell uses no mode '--:**...' as in
>>> Terminal and shows the name as in Terminal.
>> 
>> The <box> just means that it couldn't find a font to display the char.
>> Go to that char and hit C-u C-x = to see which char it is.  Maybe you just
>> need to help Emacs find the right font.

> There is no char.

Oh, yes, there sure is: that's what the box stands for.

> So there should not be a box. The correct string is
> äöüß... and as it's a file name, it's UTF-8 encoded. (Since the displayed
> glyphs are the chars stripped off their diaeresis the box could be ...
> but it's : "(01211310, 332488, 0x512c8, file ...).

You forgot the C-u before the C-x =

> A bit astronomical high.

What makes you think so?

> æ and Æ and € are represented by themselves.

When we're talking about non-ASCII chars, there's simply no such thing as
"represented by themselves".

>> But of course, first we need to know which Emacs version you're running.
>> If it's Emacs-CVS, please move this discussion to emacs-devel@gnu.org or
>> emacs-pretest-bug@gnu.org.

> Since the Fink team decided to distribute in the unstable section an Emacs
> from CVS ... (but there was not much difference to 21.3.50, it might be
> exactly 0 in this case)

You didn't finish your