Keybindings in non-Latin layout

Keybindings in non-Latin layout

[Андрей Парамонов]

Hello!

I'm using Emacs under Debian GNU/Linux system on a daily basis for a
couple of years already. Recently I've tried NTEmacs, and I've noticed
that it solves the most annoying problem I'm having with my Emacs all
the time.

When I'm typing text in Russian, I wish that Emacs keybindings worked
without need to switch to English layout. For example, when I hit C-x
C-s, I wish it worked in both English and Russian layouts.

Following advice by Jason Rumney, I've tried using "russian-computer"
input method, and indeed the keybindings keep working in this mode.
However I don't find it very convinient to use different key combos in
Emacs and in other applications for switching keyboard layouts. Also,
the keybindings work only if the active X keyboard layout is "us", not
"ru".

Is there a way to implement such a feature in GNU Emacs or it would
only work on w32, not on GNU system? Maybe it is possible to solve
these problems using the knowledge about character conversion
(available in the input method)? Is it possible to use input method
character conversion table the other direction?

Andrey Paramonov

Keybindings in non-Latin layout

[Stephen J. Turnbull]

Андрей Парамонов writes:

 > Is there a way to implement [muscle memory friendly control key
 > bindings] in GNU Emacs or it would only work on w32, not on GNU
 > system?

IIUC, it's been done in XEmacs, I believe.  If it's not obvious how to
do it in Emacs, Aidan Kehoe is the man to ask about our implementation
(kehoa at parhasard dot net).  Caution: I know he's busy with school
right now, so he may not have much time to give to it.

Re: Keybindings in non-Latin layout

[Juri Linkov]

> Is there a way to implement such a feature in GNU Emacs or it would
> only work on w32, not on GNU system? Maybe it is possible to solve
> these problems using the knowledge about character conversion
> (available in the input method)? Is it possible to use input method
> character conversion table the other direction?

This is possible with the following patch:
http://thread.gmane.org/gmane.emacs.devel/46455/focus=46602

-- 
Juri Linkov
http://www.jurta.org/emacs/

Re: Keybindings in non-Latin layout

[Andrey Paramonov]

Juri Linkov <juri <at> jurta.org> writes:
> > Is there a way to implement such a feature in GNU Emacs or it would
> > only work on w32, not on GNU system? Maybe it is possible to solve
> > these problems using the knowledge about character conversion
> > (available in the input method)? Is it possible to use input method
> > character conversion table the other direction?
> 
> This is possible with the following patch:
> http://thread.gmane.org/gmane.emacs.devel/46455/focus=46602
> 

I've read the thread you pointed out, and the fact that devs consider the
current GNU Emacs behavior a bug gives some hope. From the discussion I
understand that the patch doesn't solve the problem completely, i.e. C-x b won't
work in Russian layout anyway.

I've looked at how Mozilla solved the similar issue in Firefox for GNU systems.
The solution works nicely for me. The diff is available here:

https://bugzilla.mozilla.org/attachment.cgi?id=289837&action=diff

Possible alternative solution is to let input methods provide backward
conversion tables, and make use of them when not doing self-insert-command. That
should work even if (possibly, only if) the input method is off.

The things to consider:

1) Do we want to keep the ability to assign keybindings to something like C-Ц? I
don't think it's reasonable.

2) If we go the Mozilla way: what if one uses dvorak and qwerty X layouts in the
same system? Don't know if it's an important issue though, as I don't use dvorak.

I think the bug is important to fix as it's a sad example of how an application
works better under non-free than under free system.

Andrey

RE: Keybindings in non-Latin layout

[Drew Adams]

> 1) Do we want to keep the ability to assign keybindings to 
> something like C-?? I don't think it's reasonable.

Caveat: I haven't followed this thread at all.

Yes, Emacs should keep the ability to assign keybindings keys such as C-?. If
that presents a problem in some context, then that context should prevent or
workaround it. But Emacs should not forego keys such as C-?.

Re: Keybindings in non-Latin layout

[Andrey Paramonov]

Drew Adams <drew.adams <at> oracle.com> writes:
> Yes, Emacs should keep the ability to assign keybindings keys such as C-?. If
> that presents a problem in some context, then that context should prevent or
> workaround it. But Emacs should not forego keys such as C-?.
> 

I didn't of course mean "C-?". I meant something like "C-[Russian letter tse]".
The question mark in your message substitutes a Russian character.

You are not the only one to sink in the Cyrillic character soup ;-)  Please see

http://czyborra.com/charsets/cyrillic.html 

Andrey

Re: Keybindings in non-Latin layout

[Jason Rumney]

Andrey Paramonov wrote:
> 1) Do we want to keep the ability to assign keybindings to something like C-Ц? I
> don't think it's reasonable.
>   

It is reasonable for keys where there is no conflict between an ASCII 
meaning of the key and the meaning in some other key map (some Latin 
languages have keyboards where non-ASCII characters are on keys that are 
not assigned any ASCII character).

Re: Keybindings in non-Latin layout

[Andrey Paramonov]

Jason Rumney <jasonr <at> gnu.org> writes:
> Andrey Paramonov wrote:
> > 1) Do we want to keep the ability to assign keybindings to something
> > like C-Ц? I don't think it's reasonable.
> >   
> 
> It is reasonable for keys where there is no conflict between an ASCII 
> meaning of the key and the meaning in some other key map (some Latin 
> languages have keyboards where non-ASCII characters are on keys that are 
> not assigned any ASCII character).
> 

Do people use input methods on such keyboards? If they do, the
backward conversion table should just not provide entries for
those Latin non-ASCII characters. If they don't, the backward
conversion table would not be applied at all, so no problem
exists.

I think the Mozilla way is not directly applicable if want to
allow such keybindings.

Andrey

RE: Keybindings in non-Latin layout

[Drew Adams]

> I didn't of course mean "C-?". I meant something like 
> "C-[Russian letter tse]".
> The question mark in your message substitutes a Russian character.
> 
> You are not the only one to sink in the Cyrillic character 
> soup ;-)  Please see http://czyborra.com/charsets/cyrillic.html 

Oops; sorry for misunderstanding. Excuse the noise.

Re: Keybindings in non-Latin layout

[Juri Linkov]

>> This is possible with the following patch:
>> http://thread.gmane.org/gmane.emacs.devel/46455/focus=46602
>
> I've read the thread you pointed out, and the fact that devs consider the
> current GNU Emacs behavior a bug gives some hope. From the discussion I
> understand that the patch doesn't solve the problem completely, i.e. C-x b won't
> work in Russian layout anyway.

As you already noticed `C-x b' won't work, and a Mozilla-like patch
won't help with a single characters like `b' in a key sequence.

Now I see one way to fix this problem.  What if we would define
a parallel branch in a keymap where every defined key sequence
has a copy translated according to a conversion table.

So, for example, a define-key call:

(define-key global-map [(control ?x) ?b] 'switch-to-buffer)

would also define [(control ?ч) ?и] with the same binding
in the same keymap as if it were defined explicitly with:

(define-key global-map [(control ?ч) ?и] 'switch-to-buffer)

-- 
Juri Linkov
http://www.jurta.org/emacs/

Re: Keybindings in non-Latin layout

[Miles Bader]

Juri Linkov <juri@jurta.org> writes:
> Now I see one way to fix this problem.  What if we would define
> a parallel branch in a keymap where every defined key sequence
> has a copy translated according to a conversion table.
>
> So, for example, a define-key call:
>
> (define-key global-map [(control ?x) ?b] 'switch-to-buffer)
>
> would also define [(control ?ч) ?и] with the same binding
> in the same keymap as if it were defined explicitly with:
>
> (define-key global-map [(control ?ч) ?и] 'switch-to-buffer)

I think in general, a "reactive" solution, like function-key-map, or the
special handling of shifted keys (mapping them to unshifted variants if
not bound) would be much more robust than a method like the above
(the problem being that it essentially stores redundant state that can
get out of sync).

-Miles

-- 
`There are more things in heaven and earth, Horatio,
 Than are dreamt of in your philosophy.'

Re: Keybindings in non-Latin layout

[Stefan Monnier]

> I think in general, a "reactive" solution, like function-key-map, or the
> special handling of shifted keys (mapping them to unshifted variants if
> not bound) would be much more robust than a method like the above
> (the problem being that it essentially stores redundant state that can
> get out of sync).

Agreed.  The only difficulty is in building the reverse mapping, and in
making it apply to all forms of the key (with arbitrary modifiers), all
this ideally without adding yet-more-ad-hoc-C-code in the
read_key_sequence monster (or even removing some of it instead).

Ideally, the same technique can be used to map mouse-4/5 to
wheel-up/down (with or without arbitrary modifiers).


        Stefan

Re: Keybindings in non-Latin layout

[Juri Linkov]

> I think in general, a "reactive" solution, like function-key-map, or the
> special handling of shifted keys (mapping them to unshifted variants if
> not bound) would be much more robust than a method like the above
> (the problem being that it essentially stores redundant state that can
> get out of sync).

Maybe something like function-key-map would work if it will translate
all keys except those bound to self-insert-command - this is the primary
requirement for this feature.

-- 
Juri Linkov
http://www.jurta.org/emacs/

Re: Keybindings in non-Latin layout

[Samuel Bronson]

On Tue, May 5, 2009 at 12:41 AM, Stefan Monnier
<monnier@iro.umontreal.ca> wrote:
> Agreed.  The only difficulty is in building the reverse mapping, and in
> making it apply to all forms of the key (with arbitrary modifiers), all
> this ideally without adding yet-more-ad-hoc-C-code in the
> read_key_sequence monster (or even removing some of it instead).

That C code is so pesky -- what do we need it for anyway :-P?

Re: Keybindings in non-Latin layout

[Andrey Paramonov]

Stefan Monnier <monnier <at> iro.umontreal.ca> writes:
> > I think in general, a "reactive" solution, like function-key-map, or the
> > special handling of shifted keys (mapping them to unshifted variants if
> > not bound) would be much more robust than a method like the above
> > (the problem being that it essentially stores redundant state that can
> > get out of sync).
> 
> Agreed. The only difficulty is in building the reverse mapping, and in
> making it apply to all forms of the key (with arbitrary modifiers), all
> this ideally without adding yet-more-ad-hoc-C-code in the
> read_key_sequence monster (or even removing some of it instead).
> 
> Ideally, the same technique can be used to map mouse-4/5 to
> wheel-up/down (with or without arbitrary modifiers).
> 

I've looked at read_key_sequence function, and it is Ж:-O indeed (given I don't
have much C experience). I could however notice that it doesn't contain
platform-specific code. How come NTEmacs works nicely, while Emacs on my
GNU/Linux system doesn't? Is NTEmacs a build of GNU Emacs or it is a different
beast?

Please forgive my low level of Emacs internal knowledge ;-)

Andrey

Re: Keybindings in non-Latin layout

[Lennart Borgman]

On Tue, May 5, 2009 at 8:29 PM, Andrey Paramonov <cmr.Pent@gmail.com> wrote:
> Stefan Monnier <monnier <at> iro.umontreal.ca> writes:
>> > I think in general, a "reactive" solution, like function-key-map, or the
>> > special handling of shifted keys (mapping them to unshifted variants if
>> > not bound) would be much more robust than a method like the above
>> > (the problem being that it essentially stores redundant state that can
>> > get out of sync).
>>
>> Agreed. The only difficulty is in building the reverse mapping, and in
>> making it apply to all forms of the key (with arbitrary modifiers), all
>> this ideally without adding yet-more-ad-hoc-C-code in the
>> read_key_sequence monster (or even removing some of it instead).
>>
>> Ideally, the same technique can be used to map mouse-4/5 to
>> wheel-up/down (with or without arbitrary modifiers).
>>
>
> I've looked at read_key_sequence function, and it is Ж:-O indeed (given I don't
> have much C experience). I could however notice that it doesn't contain
> platform-specific code. How come NTEmacs works nicely, while Emacs on my
> GNU/Linux system doesn't? Is NTEmacs a build of GNU Emacs or it is a different
> beast?

It is the same beast ...

I think the difference here may be in w32term.c but I am not sure.


> Please forgive my low level of Emacs internal knowledge ;-)
>
> Andrey
>
>
>
>
>

Re: Keybindings in non-Latin layout

[James Cloos]

>>>>> "Andrey" == Andrey Paramonov <cmr.Pent@gmail.com> writes:

Andrey> I could however notice that it doesn't contain platform-specific
Andrey> code. How come NTEmacs works nicely, while Emacs on my GNU/Linux
Andrey> system doesn't?

It is probably a function of how the key events are presented to emacs.

On X, the key event struct’s .state member is a bitmask intended to
alter the meaning of the key event.  The lower eight bits represent the
eight modifiers (caps, shift, ctrl and five unnamed modifiers which are
typicaly used for alt, meta, super, hyper, numlock).

Bits 13 and 14 (0x2000 and 0x4000) specify which of the four possible
groups are in play.  Groups were initially indended for those cases
where the keyboard has multiple columns of symbols, as seen on most
European keyboards.  (Here in the states, the most common case is the
euro symbols on recent keyboards.)  As few years ago the keyboard
database project changed its use of groups.

Now, the columns of symbols on the key caps are treated as additional
rows, and one can configure up to four separate keyboard configs at a
time, one per group.

This is convenient, as it allows one to quickly switch between up to
four keyboards, using only the keyboard itself.  But it is not how
XKB groups were intended to be used when XKB was invented.

With the current keyboard configs, it is appropriate to ignore the
group bits when the control, meta, super and/or hyper bits are set.
Some apps already do.

In Emacs, it is probably also reasonable to ignore the group bits in
the middle of sequences intiated by prefix keys.

Ignoring the group bits also might be the Right Thing To Do when the
user selects an application-level alternate keyboard, such as by use
of Emacs’ (set-input-method) or (toggle-input-method).

NT probably uses a sufficiently different method of keyboard events that
this problem does not occur; there is nothing for Emacs to work around.

(BTW, the concept of groups comes from XKB; XKB uses this:

#define XkbBuildCoreState(m,g)  ((((g)&0x3)<<13)|((m)&0xff))
#define XkbGroupForCoreState(s) (((s)>>13)&0x3)
#define XkbIsLegalGroup(g)      (((g)>=0)&&((g)<XkbNumKbdGroups))

to save the group info into the core event .state bitmask, using two
bits which are not otherwise used by the core protocol.  If Emacs is
already XKB-aware there is probably a better way to deal with this
issue.  But if Emacs just used core keyboard events, then simply
ignoring the two group bits when Emacs is in “special” keyboard modes
should work well enough.)

-JimC
-- 
James Cloos <cloos@jhcloos.com>         OpenPGP: 1024D/ED7DAEA6

Re: Keybindings in non-Latin layout

[Andrey Paramonov]

Thanks for the detailed explanation!

I've managed to get hands on NTEmacs again, and it turns out that:

1) NTEmacs uses the Mozilla way, i.e. "C-x b" is processed as "C-x и", and so
doesn't work.

2) C-q [any Cyrillic character] inserts Latin characters with umlauts (weird;
not sure this is relevant to the original problem though). 

It would be great if Emacs was smart enough to know if it's in the middle of a
key sequence, and use the Latin characters no matter what layout. On the other
hand, the Mozilla way patch
http://thread.gmane.org/gmane.emacs.devel/46455/focus=46602 would solve 80% of
the problems, and is much less intrusive.

Could anyone please point me out the place(s) in the code where input-method
conversion table is used?

Thanks,
Andrey