Coding systems and Terminal.app on Mac OS X

Coding systems and Terminal.app on Mac OS X

[Stefan]

> TERM_PROGRAM=Apple_Terminal
> TERM_PROGRAM_VERSION=100.1.4

OK, so I tried my patch and it seems to work: it does set the
terminal-coding-system to utf-8 and letters with accents are then
properly displayed.

Now, the next problem is keyboard input: my meta key doesn't work in
Terminal.app and neither do accented chars.  Looking at C-h l I see that the
byte sequence that Emacs receives is odd.  E.g. for "M-x" Emacs receives (I
use the "meta-is-bit7" convention):

  C-v \342 C-v \211 C-v \210

Assuming the C-v was meant to be some kind of quoting char (as it is in
several Unix tools where it plays the same role as Emacs's C-q), we could
think of it as "\342 \211 \210", which is a valid utf-8 sequence for "≈".
After trying to insert some accented chars, it seems that indeed the input
is a kind of "utf-8 interleaved with C-v" and my meta key sends non-ASCII
chars instead of something like an ESC prefix.  Does anybody know what this
C-v stuff is about?


        Stefan


PS: Actually, the utf-8 display doesn't work 100% because Terminal.app
displays my \lambda characters as two-columns-chars and Emacs doesn't know
about it, so the display gets confused and every once in a while a bit of
C-l is needed.

Re: Coding systems and Terminal.app on Mac OS X

[Peter Dyballa]

[-- Attachment #1: Type: text/plain, Size: 2280 bytes --]

Hello!

Terminal.app is a complicated application and clearly not finished 
(with ls in Terminal I can't see UTF-8 file names correctly, only 
a??o??u????U??O??A?????, doing it in Emacs' shell I see 
a\314?o\314?u\314?\303?U\314?O\314?A\314?\342?\254 and in dired as the 
correct string äöüßÜÖÄ€ as in Finder too).

Maybe we should try to adjust our settings. I have in Terminal's 
«Window Preferences»

	in *Emulation* set
Option+Click to position cursor in command line
(non-ASCII -> Esc is not active/checked)

	in *Monitor* set
Lucida Sans Typewriter Regular (a font from the Java Runtime 
Environment)
anti-aliasing
Copy&Paste with moving/dragging
UTF-8 encoding

	in *Keyboard* /not/ set
alt/Option/⌥ as Meta//Command/⌘


Am 24.03.2005 um 18:17 schrieb Stefan:

>> TERM_PROGRAM=Apple_Terminal
>> TERM_PROGRAM_VERSION=100.1.4
>
> Now, the next problem is keyboard input: my meta key doesn't work in
> Terminal.app

That is true for GNU Emacs as X11 client, running in xterm, or running 
in Terminal. Only Carbon Emacs understand Cmd-x/Apple-x/⌘-x as M-x.

>  and neither do accented chars.  Looking at C-h l I see that the
> byte sequence that Emacs receives is odd.  E.g. for "M-x" Emacs 
> receives (I
> use the "meta-is-bit7" convention):
>
>   C-v \342 C-v \211 C-v \210
>
> Assuming the C-v was meant to be some kind of quoting char (as it is in
> several Unix tools where it plays the same role as Emacs's C-q), we 
> could
> think of it as "\342 \211 \210", which is a valid utf-8 sequence for 
> "≈".
> After trying to insert some accented chars, it seems that indeed the 
> input
> is a kind of "utf-8 interleaved with C-v" and my meta key sends 
> non-ASCII
> chars instead of something like an ESC prefix.  Does anybody know what 
> this
> C-v stuff is about?

I actually don't know what Terminal or Apple or both are trying with 
this and what it's good for (maybe it's necessary for vi/vim), but with 
the above mentioned settings I can directly input with my German 
keyboard these seven umlauts and some more with Character Palette 
utility and Emacs itself writes the month of March as "März" as I set 
it in calendar-month-name-array and in calendar-month-abbrev-array:


[-- Attachment #2: pastedGraphic3.tiff --]
[-- Type: image/tiff, Size: 23802 bytes --]

[-- Attachment #3: Type: text/plain, Size: 1597 bytes --]



> PS: Actually, the utf-8 display doesn't work 100% because Terminal.app
> displays my \lambda characters as two-columns-chars and Emacs doesn't 
> know
> about it, so the display gets confused and every once in a while a bit 
> of
> C-l is needed.

Again this comes from your settings. And there is another problem: 
since Terminal or xterm have only one font available you could miss 
some characters from this. The HELLO buffer shows in Terminal twice 
this:

	Greek (ελληνικά)                Γειά σας

because they're Lucida Sans Typewriter -- and it shows some extra 
scripts that GNU Emacs puts there: Amharic, Braille, Georgian, Lao, 
Tibetan, Tigrina, half Japanese. Hebrew, Thai, and Cyrillic are in the 
font too and visible, but not Arabic! It's right-to-left as Hebrew. (In 
xterm it looks worse: many hollow boxes and even more U+FEFF, a black 
square on one edge with a white question mark as in my screenshot.)

If you can't find the Character Palette, here is way to make it 
available: in System Preferences go to Locale (the UN flag), choose 
Keyboard Menu and click there 'Keyboard Menu in Menubar' plus choose 
from the window with the flags those keyboard layouts that you might 
like to use *and* Character Palette, Keyboard Layout. Now you can 
choose from the Menubar the 'Canadian Flag' menu which will have the 
Character Palette. Cmd-clicking you can drag/position a menu in the 
Menubar.

Have you tried to correct on Terminal's command line mis-spellings? 
C-d, C-t, C-w, Esc-BS ... work.

--
Greetings

   Pete

[-- Attachment #4: Type: text/plain, Size: 142 bytes --]

_______________________________________________
Emacs-devel mailing list
Emacs-devel@gnu.org
http://lists.gnu.org/mailman/listinfo/emacs-devel

Re: Coding systems and Terminal.app on Mac OS X

[Stefan]

> Terminal.app is a complicated application and clearly not finished (with ls
> in Terminal I can't see UTF-8 file names correctly, only
> a??o??u????U??O??A?????,

This seems to be a problem with `ls' rather than with Terminal.app.
Maybe if you try another `ls' binary or another locale it'll work better.

> doing it in Emacs' shell I see
> a\314?o\314?u\314?\303?U\314?O\314?A\314?\342?\254

You mean `ls' inside M-x shell?  If so, I doubt it can work better than `ls'
directly in Terminal.app.  OTOH M-x eshell might handle file names correctly
(as long as it doesn't executed `ls').

> and in dired as the correct string äöüßÜÖÄ€ as in Finder too).

Yes, it's because I've set default-file-name-coding-system to utf-8
on MacOSX, so Emacs should handle MacOSX's filenames correctly.

> Maybe we should try to adjust our settings.

I haven't touched the defaults of Terminal.app.

> I have in Terminal's «Window Preferences»

Where are these?  (I can't get to it right now because my rootfull XDarwin
refuses to obey my cmd-option-A escape to switch back to Quartz, but the
only preferences I remember seeing where really minimal).

>>> TERM_PROGRAM=Apple_Terminal
>>> TERM_PROGRAM_VERSION=100.1.4
>> 
>> Now, the next problem is keyboard input: my meta key doesn't work in
>> Terminal.app

> That is true for GNU Emacs as X11 client, running in xterm,

No.  My meta key works just fine with Emacs running as an X11 client or with
it running in an xterm (you just have to fix your xterm settings, see
`backarrow' in xterm's manpage).

Well, OK, meta doesn't work by default with xterm, because xterm's default
setting makes it basically impossible to handle it right.  (at least with
XDarwin; can't remember how Apple's own X11 distribution behaved).  But once
xterm is fixed, Emacs works correctly.

> I actually don't know what Terminal or Apple or both are trying with this
> and what it's good for (maybe it's necessary for vi/vim), but with the above
> mentioned settings I can directly input with my German keyboard these seven
> umlauts and some more with Character Palette utility and Emacs itself writes
> the month of March as "März" as I set it in calendar-month-name-array and in
> calendar-month-abbrev-array:

Ah... good.  So we can get things to work sensibly.  I'll try it as soon as
XDarwin lets me.


        Stefan

Re: Coding systems and Terminal.app on Mac OS X

[Peter Dyballa]

Am 25.03.2005 um 02:17 schrieb Stefan:

>> I have in Terminal's «Window Preferences»
>
> Where are these?

When in Terminal open the "Terminal" menu entry from the Menubar. Below 
"Preferences ⌘," is the item that gives you the options I described 
before. In German it says something like Window Preferences.

--
Greetings

   Pete

Remember: use logout to logout.

Re: Coding systems and Terminal.app on Mac OS X

[Stefan Monnier]

> 	in *Emulation* set
> Option+Click to position cursor in command line
> (non-ASCII -> Esc is not active/checked)

Here, the important setting is the non-ASCII -> escape thingy (which in
French is described something like "Avoid non-ASCII characters", where it
probably meant "escape non-ASCII chars".
After I turned it off, I could type in non-ASCII chars just fine, provided
I set the keyboard-coding-system to utf-8.

> 	in *Monitor* set
> Lucida Sans Typewriter Regular (a font from the Java Runtime Environment)
> anti-aliasing
> Copy&Paste with moving/dragging
> UTF-8 encoding

The utf-8 encoding is the default and it's best not to change it.  And the
other three options are just real preferences which have no impact on the
correct functioning of Emacs AFAICT.
OTOH, another option that's relevant here is the "large glyph (CJK)" option
which was the cause of my \lambda taking 2 columns.  I disabled it and now
my \lambdas work perfectly.

> 	in *Keyboard* /not/ set
> alt/Option/⌥ as Meta//Command/⌘

In French this is called "Option as a virtual key".

Now, where can we put this info so that users will find it?


        Stefan

Re: Coding systems and Terminal.app on Mac OS X

[Peter Dyballa]

Am 25.03.2005 um 15:01 schrieb Stefan Monnier:

> Now, where can we put this info so that users will find it?
>

Emacs' info file has already a Mac OS entry, it seems to be more 
related to Mac OS 9 and before. If this would be renamed accordingly 
and a new node 'Mac OS X' would be opened -- it would take some years 
until the other nodes will have their references to this one. And some 
distinction too is needed between using GNU Emacs as stand-alone X11 or 
Carbon application ... and in Terminal and/or xterm?

Carbon Emacs has different fontsets. You can drag&drop files on it -- 
but when you copy with M-w a region you can't paste it into another 
Aqua/Quartz application -- at least I can't! The same is true in the 
other direction -- M-w only seems to add entries into the 
kill-ring-buffer which is accessible under the Edit menu. You can't 
mark (hi-light) text by dragging the mouse -- well, the new background 
colour comes, but a beep too!

Since you have a Mac around try this Carbon Emacs 21.3.50 from CVS with 
some Japanese extras and interesting developments:

	http://home.att.ne.jp/alpha/z123/emacs-mac-e.html

In December YAMAMOTO-san has introduced some new font handling: 
http://lists.gnu.org/archive/html/emacs-devel/2004-12/msg00004.html -- 
but I still can't see it with better ISO Latin-1/15 or UTF-8 support, 
it just allows to scale an OTF or TTF font in the system without having 
to use the limited embedded bitmaps. After Easter I might find some 
time to investigate this ...

--
Greetings

   Pete

Re: Coding systems and Terminal.app on Mac OS X

[Sébastien Kirche]

Le 25 mar 2005, Peter Dyballa s'est exprimé ainsi :

> Carbon Emacs has different fontsets. You can drag&drop files on it -- 
> but when you copy with M-w a region you can't paste it into another
> Aqua/Quartz application -- at least I can't! The same is true in the
> other direction -- M-w only seems to add entries into the
> kill-ring-buffer which is accessible under the Edit menu. 

I have no problem to exchange text by copy/paste (resp. M-w and C-y) between
Carbon Emacs and other applications. 

For example  copying some mail address  in the address book  or Mail.app and
yanking it in Gnus  or copying urls in Emacs and pasting  it in Safari. I do
it often and without difficulty.

What do you mean exactly by «at least  I can't» ? I does nothing or you only
get some crap ? Sometimes i have little problem with accentuated french text
that is encoded in  Mac-Roman in my mac apps and need  some C-x RET c before
yanking in Emacs.

> You can't  mark (hi-light)  text by  dragging the mouse  -- well,  the new
> background colour comes, but a beep too!

Again, no problem here for selecting some text with the mouse. 

I  would not  say some  stupid thing,  but maybe  you could  take a  look to
transient-mark-mode ?

-- 
Sébastien Kirche

Re: Coding systems and Terminal.app on Mac OS X

[Peter Dyballa]

Am 25.03.2005 um 19:31 schrieb Benjamin Riefenstahl:

> Hi Peter,
>
> Peter Dyballa writes:
>> (with ls in Terminal I can't see UTF-8 file names correctly, only
>> a??o??u????U??O??A?????,
>
> Try "ls -w".
>

Hey, that's really great: adding -w makes me see my test files' names 
in Terminal and in xterm and too in the Emacsen running in them! 
(eshell shows the chracters right, but has problems with columns, in 
Carbon Emacs eshell fails, even if I set it to UTF-8)

Thank you very much for this hint!

--
Mit friedvollen Grüßen

   Pete

A child of five could understand this!  Fetch me a child of five.