Re: Rebinding international characters

Re: Rebinding international characters

[Kenichi Handa]

In article <E1BwXlc-0005CT-HX@fencepost.gnu.org>, Richard Stallman <rms@gnu.org> writes:

> The node Non-ASCII Rebinding ends with this advice:
>     If you bind 8-bit characters like this in your init file, you may find it
>     convenient to specify that it is unibyte.  @xref{Enabling Multibyte}.

> Is that still correct?

No.  I tested what is given to Emacs by typing a non-ASCII
key `ñ' in various situations.   The result is quite
confusing.

(1) (read-char)
(2) (read-key-sequence "")
(3) just type it

(A) LANG=C --multibyte -nw  (keyboard-coding-system == nil)
(A-1) 134217841 (== ?\M-q)
(A-2) "\361"
(A-3) invoke fill-paragraph

(B) LANG=C --multibyte  (keyboard-coding-system == nil)
(B-1) 2289      <- handle_one_xevent converts keysym by x-keysym-table
(B-2) [2289]
(B-3) Insert multibyte ñ (== 2289).

(C) LANG=de_DE --multibyte -nw  (keyboard-coding-system == iso-latin-1)
(C-1) 241      <- set-keyboard-coding-system sets input mode
                  to accept 8-bit as is
(C-2) [2289]   <- by key-translation-map
(C-3) Insert multibyte ñ (== 2289).

(D) LANG=de_DE --multibyte  (keyboard-coding-system == nil)
(D-1) 2289
(D-2) [2289]
(D-3) Insert multibyte ñ (== 2289).

(E) LANG=C --unibyte -nw  (keyboard-coding-system == nil)
(E-1) 134217841 (== ?\M-q)
(E-2) "\361"
(E-3) invoke fill-paragraph

(F) LANG=C --unibyte  (keyboard-coding-system == nil)
(F-1) 2289
(F-2) [2289]
(F-3) Insert unibyte ñ (== 241)

(G) LANG=de_DE --unibyte -nw  (keyboard-coding-system == iso-latin-1)
(G-1) 241
(G-2) [2289]
(G-3) Insert unibyte ñ (== 241).

(H) LANG=de_DE --unibyte  (keyboard-coding-system == nil)
(H-1) 2289
(H-2) [2289]
(H-3) Insert unibyte ñ (== 241).

I think (C-1), (F-1), (F-2), (G-2), (H-1), (H-2) are not
good.  (C-1) should be 2289, (F-1) (H-1) should be 241,
(F-2) (G-2) (H-2) should be [241].

To fix (C-1), we must handle keyboard-coding-system in
read_char (perhaps after handling keyboard-translation-table
and before handling an input method).

To fix (F-1) (F-2) (H-1) (H-2), we should call
multibyte_char_to_unibyte in handle_one_xevent.  But when?
Should we check enable-multibyte-character of the current
buffer?  Or default-enable-multibyte-character?

To fix (G-2), we should simply don't setup
keyboard-coding-system.

After those fixes, we can consider what to write in info for
non-ASCII bindings.

---
Ken'ichi HANDA
handa@m17n.org

Re: Rebinding international characters

[Stefan]

> I think (C-1), (F-1), (F-2), (G-2), (H-1), (H-2) are not
> good.  (C-1) should be 2289, (F-1) (H-1) should be 241,
> (F-2) (G-2) (H-2) should be [241].

I think that the --unicode mode is a dead-end (it made sense when it was
introduced, but in my mind it's always been a temporary workaround for cases
which the multibyte mode doesn't support correctly yet; nowadays multibyte
mode behaves sufficiently well that --unibyte mode should be strongly
discouraged).
Thus I think that neither F-1, G-1, F-2, G-2, or H-2 are wrong (maybe
they're not right either, but we shouldn't waste any time on them).

I agree that C-1 should be fixed, but that implies doing the translation at
a lower level than we currently do.  It gets us back to the previous
discussion of the relationship betwen function-key-map and
key-translation-map and that we should have a key-translation-map-like step
before function-key-map.

Also we should be careful that applications like xterm-mouse-mode which want
to read keyboard input events as binary (even if the rest of the events are
normally treated as latin-1 or utf-8) can still do that properly.
It shouldn't be a source of problems, but it's still worth remembering.


        Stefan

Re: Rebinding international characters

[Kenichi Handa]

In article <m1wtzx5z8t.fsf-monnier+emacs@gnu.org>, Stefan <monnier@iro.umontreal.ca> writes:

>>  I think (C-1), (F-1), (F-2), (G-2), (H-1), (H-2) are not
>>  good.  (C-1) should be 2289, (F-1) (H-1) should be 241,
>>  (F-2) (G-2) (H-2) should be [241].

> I think that the --unicode mode is a dead-end (it made sense when it was
                   ^^^^^^^^^--> --unibyte
> introduced, but in my mind it's always been a temporary workaround for cases
> which the multibyte mode doesn't support correctly yet; nowadays multibyte
> mode behaves sufficiently well that --unibyte mode should be strongly
> discouraged).

I agree.  Temporary unibyte buffer/string is still useful
for raw data, but, yes, --unibyte mode is just a headache.

> Thus I think that neither F-1, G-1, F-2, G-2, or H-2 are wrong (maybe
> they're not right either, but we shouldn't waste any time on them).

If Richard agree with that, I'll modify the info along that
line, i.e. clear enough for normal multibyte users.

> I agree that C-1 should be fixed, but that implies doing the translation at
> a lower level than we currently do.  It gets us back to the previous
> discussion of the relationship betwen function-key-map and
> key-translation-map and that we should have a key-translation-map-like step
> before function-key-map.

What do you think about my previous suggesion; i.e. handling
it between keyboard-translate-table and input method in
read_char?

> Also we should be careful that applications like xterm-mouse-mode which want
> to read keyboard input events as binary (even if the rest of the events are
> normally treated as latin-1 or utf-8) can still do that properly.
> It shouldn't be a source of problems, but it's still worth remembering.

Ah, yes.

---
Ken'ichi HANDA
handa@m17n.org

Re: Rebinding international characters

[Stefan Monnier]

>> I think that the --unicode mode is a dead-end (it made sense when it was
>                    ^^^^^^^^^--> --unibyte

Oops, indeed, thank you for fixing the typo.

> I agree.  Temporary unibyte buffer/string is still useful
> for raw data, but, yes, --unibyte mode is just a headache.

Yes, of course I was only talking about the unibyte *mode*: unibyte buffers
and strings will always be at least very useful if not necessary (remember
how I think of unibyte objects as "sequences of bytes" as opposed to
"sequences of chars" for multibyte objects).

>> I agree that C-1 should be fixed, but that implies doing the translation at
>> a lower level than we currently do.  It gets us back to the previous
>> discussion of the relationship betwen function-key-map and
>> key-translation-map and that we should have a key-translation-map-like step
>> before function-key-map.

> What do you think about my previous suggesion; i.e. handling
> it between keyboard-translate-table and input method in
> read_char?

I'd be happy to simply use keyboard-translate-table for keyboards coded in
latin-1, and to extend keyboard-translate-table so that an entry in the
table can be mapped to another table (thus being able to handle sequences of
chars in input) for encodings that use multiple bytes per char.

But maybe adding a separate step that directly obeys keyboard-coding-system
would be simpler  (with the addition that it makes it much easier to
temporarily change the encoding).

In general I agree that the keyboard decoding should be changed so that it
is done "within read-char" (i.e. read-char should return decoded chars).

>> Also we should be careful that applications like xterm-mouse-mode which want
>> to read keyboard input events as binary (even if the rest of the events are
>> normally treated as latin-1 or utf-8) can still do that properly.
>> It shouldn't be a source of problems, but it's still worth remembering.

> Ah, yes.

Maybe a `read-byte' function is in order.


        Stefan

Re: Rebinding international characters

[Richard Stallman]

    > I agree that C-1 should be fixed, but that implies doing the translation at
    > a lower level than we currently do.  It gets us back to the previous
    > discussion of the relationship betwen function-key-map and
    > key-translation-map and that we should have a key-translation-map-like step
    > before function-key-map.

    What do you think about my previous suggesion; i.e. handling
    it between keyboard-translate-table and input method in
    read_char?

Please don't change any of this.

Re: Rebinding international characters

[Richard Stallman]

    >     If you bind 8-bit characters like this in your init file, you may find it
    >     convenient to specify that it is unibyte.  @xref{Enabling Multibyte}.

    > Is that still correct?

    No.

Are you saying I should delete those two lines?

The tests you did are interesting, but it is not clear to me how
they relate to the question of whether to specify that your
.emacs file is a unibyte file.  Could you explain the relationship
between these two issues?

Re: Rebinding international characters

[Kenichi Handa]

In article <E1BxFLB-0003QJ-9y@fencepost.gnu.org>, Richard Stallman <rms@gnu.org> writes:

>>      If you bind 8-bit characters like this in your init file, you may find it
>>      convenient to specify that it is unibyte.  @xref{Enabling Multibyte}.

>>  Is that still correct?

>     No.

> Are you saying I should delete those two lines?

Yes.  Actually we should delete all paragraphs after this.

----------------------------------------------------------------------
If you don't specify a keyboard coding system, that approach won't
work.  ...
----------------------------------------------------------------------

because specifying a unibyte code doesn't wark anymore even
in --unibyte case.

> The tests you did are interesting, but it is not clear to me how
> they relate to the question of whether to specify that your
> .emacs file is a unibyte file.  Could you explain the relationship
> between these two issues?

In --unibyte case, read-key-sequence returns a vector of
multibyte character.  So, the key binding should also be
done for the same character instead of unibyte code.  But,
as unibyte buffer can't include such a character, what we
can do is only to specify a character code directly as this:

(global-set-key [2289] 'some-function)

But how to tell users to get the number 2289 in unibyte
mode?  Something like this?
    ESC : (read-event) RET
    and type a key

And, this stops working in emacs-unicode because the
character code is different.

Another approach is to modify read_key_sequence so that it
tries to find unibyte binding if multibyte binding was not
found (like the way for finding lowercase/uppercase).  Then,
the current info is still valid.

---
Ken'ichi HANDA
handa@m17n.org

Re: Rebinding international characters

[Stefan Monnier]

> Another approach is to modify read_key_sequence so that it
> tries to find unibyte binding if multibyte binding was not
> found (like the way for finding lowercase/uppercase).  Then,
> the current info is still valid.

Placing the encoded-kbd handling on function-key-map instead of
key-translation-map will do just that without any changes t the C code.


        Stefan

Re: Rebinding international characters

[Kenichi Handa]

In article <jwvekm4ih9s.fsf-monnier+emacs@gnu.org>, Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>  Another approach is to modify read_key_sequence so that it
>>  tries to find unibyte binding if multibyte binding was not
>>  found (like the way for finding lowercase/uppercase).  Then,
>>  the current info is still valid.

> Placing the encoded-kbd handling on function-key-map instead of
> key-translation-map will do just that without any changes t the C code.

But, as function-key-map can't override global/local map,
ESC $ ... can't be handled.

---
Ken'ichi HANDA
handa@m17n.org

Re: Rebinding international characters

[Stefan Monnier]

>> Placing the encoded-kbd handling on function-key-map instead of
>> key-translation-map will do just that without any changes t the C code.
> But, as function-key-map can't override global/local map,
> ESC $ ... can't be handled.

Huh?  Why not?  You have to bind it to an empty map anyway, no?
Or did you find some other workaround?


        Stefan


PS: I personally don't think it's a good idea to use function-key-map for
    that (I think the encoded-kdb translation should be "strong"
    (i.e. not overridable other than via things like `read-byte'), just like
    it is under X11).
    It is very unusual for people to rebind non-ASCII characters (at least
    this is my impression based on the frequency of such questions in
    gnu.emacs.help) and it has always been tricky to do and brittle (based
    on the lack of useful and correct answers those rare questions
    received), so I think that the current situation is already an
    improvement and making it work for unibyte modes is not important.

Re: Rebinding international characters

[Kenichi Handa]

In article <jwvwtzvfe49.fsf-monnier+emacs@gnu.org>, Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>  But, as function-key-map can't override global/local map,
>>  ESC $ ... can't be handled.

> Huh?  Why not?  You have to bind it to an empty map anyway, no?
> Or did you find some other workaround?

Currently, this works (typing "ESC $ a" inserts `A'):
  (local-set-key "\e$a" 'ignore)
  (define-key key-translation-map "\e$a" [?A])
but this doesn't work:
  (local-set-key "\e$a" 'ignore)
  (define-key function-key-map "\e$a" [?A])

That is what I meant in "function-key-map can't override
global/local map".

> PS: I personally don't think it's a good idea to use function-key-map for
>     that (I think the encoded-kdb translation should be "strong"
>     (i.e. not overridable other than via things like `read-byte'), just like
>     it is under X11).
>     It is very unusual for people to rebind non-ASCII characters (at least
>     this is my impression based on the frequency of such questions in
>     gnu.emacs.help) and it has always been tricky to do and brittle (based
>     on the lack of useful and correct answers those rare questions
>     received), so I think that the current situation is already an
>     improvement and making it work for unibyte modes is not important.

I agree.  So, we should adjust info to the current behaviour
instead of changing the code.

---
Ken'ichi HANDA
handa@m17n.org

Re: Rebinding international characters

[Stefan Monnier]

> Currently, this works (typing "ESC $ a" inserts `A'):
>   (local-set-key "\e$a" 'ignore)
>   (define-key key-translation-map "\e$a" [?A])
> but this doesn't work:
>   (local-set-key "\e$a" 'ignore)
>   (define-key function-key-map "\e$a" [?A])

Hmm... I was thinking of (local-set-key "\e$a" nil), not `ignore'.
Any particular reason why you use `ignore'?


        Stefan

Re: Rebinding international characters

[Kenichi Handa]

In article <jwvn00pc0u7.fsf-monnier+emacs@gnu.org>, Stefan Monnier <monnier@iro.umontreal.ca> writes:
>>  but this doesn't work:
>>    (local-set-key "\e$a" 'ignore)
>>    (define-key function-key-map "\e$a" [?A])

> Hmm... I was thinking of (local-set-key "\e$a" nil), not `ignore'.
> Any particular reason why you use `ignore'?

No.  But, anyway,

(local-set-key "\e$a" nil)
(define-key function-key-map "\e$a" [?A])

doesn't work.  When I type "ESC $ a", I just hear beep.
When I type "C-h c ESC $ a", this error is signalled.

M-$ a (translated from <escape> $ a) is undefined

---
Ken'ichi HANDA
handa@m17n.org

Re: Rebinding international characters

[Stefan Monnier]

> No.  But, anyway,

> (local-set-key "\e$a" nil)
> (define-key function-key-map "\e$a" [?A])

> doesn't work.  When I type "ESC $ a", I just hear beep.
> When I type "C-h c ESC $ a", this error is signalled.

> M-$ a (translated from <escape> $ a) is undefined

Hmmm... I'm not sure it qualifies as a bug.
But if it still doesn't work with (local-set-key "\e$" nil), then it's
clearly a bug.


        Stefan

Re: Rebinding international characters

[Kenichi Handa]

In article <87y8k80yhg.fsf-monnier+emacs@gnu.org>, Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>  No.  But, anyway,
>>  (local-set-key "\e$a" nil)
>>  (define-key function-key-map "\e$a" [?A])

>>  doesn't work.  When I type "ESC $ a", I just hear beep.
>>  When I type "C-h c ESC $ a", this error is signalled.

>>  M-$ a (translated from <escape> $ a) is undefined

> Hmmm... I'm not sure it qualifies as a bug.
> But if it still doesn't work with (local-set-key "\e$" nil), then it's
> clearly a bug.

With that, when I type "ESC $", ispell-word (defined in the
global map) is executed.  Is it really a bug?

---
Ken'ichi HANDA
handa@m17n.org

Re: Rebinding international characters

[Stefan Monnier]

>> But if it still doesn't work with (local-set-key "\e$" nil), then it's
>> clearly a bug.

> With that, when I type "ESC $", ispell-word (defined in the
> global map) is executed.  Is it really a bug?

Oh, now I finally see why you use `ignore'.
OK, I guess you were right, sorry for being stubborn.


        Stefan

Re: Rebinding international characters

[Oliver Scholz]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

[...]
>     It is very unusual for people to rebind non-ASCII characters (at least
>     this is my impression based on the frequency of such questions in
>     gnu.emacs.help) and it has always been tricky to do and brittle (based
>     on the lack of useful and correct answers those rare questions
>     received), so I think that the current situation is already an
>     improvement and making it work for unibyte modes is not important.

Maybe people don't do it, because it seems like a hastle? If one reads
gnu.emacs.help and notices the few threads on binding non-ASCII keys,
this is not very encouraging. (If I ask myself, why I don't do it,
then I'd probably say that I was too lazy yet to check for the best
way to do it.)

However, on my German keyboard I have three non-ascii keys on the
three main rows. Two of them are actually on the home row, i.e. those
keys are very easy to type. So it's actually a shame not to use them
for Emacs keybindings. This might be similar for other European
keyboards.

    Oliver
-- 
Oliver Scholz               4 Fructidor an 212 de la Révolution
Ostendstr. 61               Liberté, Egalité, Fraternité!
60314 Frankfurt a. M.       

Re: Rebinding international characters

[Stefan Monnier]

> Maybe people don't do it, because it seems like a hastle? If one reads
> gnu.emacs.help and notices the few threads on binding non-ASCII keys,
> this is not very encouraging. (If I ask myself, why I don't do it,
> then I'd probably say that I was too lazy yet to check for the best
> way to do it.)

My point was not that we should discourage people from doing it, but rather
that we shouldn't worry about people who use "obsolete" settings (like
unibyte mode) since it hasn't been working well in the past anyway.

I think the new situation is actually much better since you should be
able to put something like

   (global-set-key ?é 'foobar)

directly in your .emacs, and it should just work (since by default the
keyboard-coding-system should be the same as the file's coding system).


        Stefan

Re: Rebinding international characters

[Oliver Scholz]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> Maybe people don't do it, because it seems like a hastle? If one reads
>> gnu.emacs.help and notices the few threads on binding non-ASCII keys,
>> this is not very encouraging. (If I ask myself, why I don't do it,
>> then I'd probably say that I was too lazy yet to check for the best
>> way to do it.)
>
> My point was not that we should discourage people from doing it, but rather
> that we shouldn't worry about people who use "obsolete" settings (like
> unibyte mode) since it hasn't been working well in the past anyway.
[...]

Ah, sorry. I misunderstood your statement that rebinding those keys
were unusual as meaning that it were not important to care about it.

My bad. Sorry for the noise.

    Oliver
-- 
Oliver Scholz               5 Fructidor an 212 de la Révolution
Ostendstr. 61               Liberté, Egalité, Fraternité!
60314 Frankfurt a. M.       

Re: Rebinding international characters

[Richard Stallman]

    I think (C-1), (F-1), (F-2), (G-2), (H-1), (H-2) are not
    good.  (C-1) should be 2289, (F-1) (H-1) should be 241,
    (F-2) (G-2) (H-2) should be [241].

I don't agree.  These should not be changed.

Unibyte and multibyte are representations for text in buffers--that is
all.  They should not affect how input keys are represented.  It is
correct that we get the same results from read-event and
read-key-sequence regardless of which representation the buffer uses.