bug#43830: keyboard layout handling incompatible with rest of the OS

bug#43830: keyboard layout handling incompatible with rest of the OS

[Paul Pogonyshev]

[-- Attachment #1: Type: text/plain, Size: 961 bytes --]

I use English and Russian keyboard layouts. For every single application I
don't need to care which layout is currently selected for shortcuts, e.g.
Ctrl+S and Ctrl+Ы do the same (S and Ы are on the same physical key). Of
course, in Emacs it doesn't work this way: C-s triggers Isearch, but C-ы
"is undefined".

I know that Emacs reinvents the wheel by providing built-in input methods
(switchable by C-\), which does the remapping. However, it is very
user-unfriendly at least for two reasons:

- it is a different key compared to everything else I use; my normal
"Shift+Shift" way of switching keyboard layouts is already part of
mechanical memory and having to use something else is very inconvenient;

- *all* of my configuration for XKB is of course gone if I try using Emacs
input methods; e.g. special handling of the right Alt key (AltGr), which is
very important to quickly type characters only available on English layout.

Paul

[-- Attachment #2: Type: text/html, Size: 1088 bytes --]

bug#43830: keyboard layout handling incompatible with rest of the OS

[Eli Zaretskii]

> From: Paul Pogonyshev <pogonyshev@gmail.com>
> Date: Tue, 6 Oct 2020 17:34:34 +0200
> 
> I use English and Russian keyboard layouts. For every single application I don't need to care which layout is
> currently selected for shortcuts, e.g. Ctrl+S and Ctrl+Ы do the same (S and Ы are on the same physical
> key). Of course, in Emacs it doesn't work this way: C-s triggers Isearch, but C-ы "is undefined".

Do you know how to retrieve the key code corresponding to the English
keyboard layout when the active layout is something else, like
Russian?  Does someone else here know?

Or maybe the keyboard driver can be configured to return ASCII
characters when the Ctrl key is held, even when the layout is
non-English?

What Emacs does, AFAICT, when a key press event comes in is call the
Xlib function that returns the character produced by the keyboard, and
that character depends on the current keyboard layout.  We then add
the modifier keys to that character.  I don't see the "original"
English-layout character available anywhere in that code, but maybe
I'm missing something.

A workaround is to bind C-ы to the same command as C-s (and similarly
with other combinations that matter).

bug#43830: keyboard layout handling incompatible with rest of the OS

[Paul Pogonyshev]

[-- Attachment #1: Type: text/plain, Size: 2705 bytes --]

> Do you know how to retrieve the key code corresponding to the English
> keyboard layout when the active layout is something else, like
> Russian?

No, but there must be a way, because both GTK+ and KDE know how to do
this. GTK+ also can do this when run under KDE, so I'm pretty sure the
information itself comes from X (xkb), not e.g. from GNOME configuration.
No idea about the other way round, but I guess it also works.

> Or maybe the keyboard driver can be configured to return ASCII
> characters when the Ctrl key is held, even when the layout is
> non-English?

Not sure what you mean here, but certainly it doesn't need to be configured
by the user. Maybe by the application/framework (i.e. KDE, GTK+ or Emacs).

> A workaround is to bind C-ы to the same command as C-s (and similarly
> with other combinations that matter).

This is not feasible because Emacs has hundreds of shortcuts coming from
all sorts of places (the core, major and minor modes, my config and so on).
It's certainly not just about C-s.

By the way, same goes for shortcuts/keybindings even without modifiers.
E.g. I have my own keybinding <f12 s> to switch to *scratch* buffer.
<f12 ы> is, predictably, "undefined". However, if using Emacs' own Russian
input method, it works just like <f12 s>, even though the key on itself
types "ы".

Paul

On Tue, 6 Oct 2020 at 19:26, Eli Zaretskii <eliz@gnu.org> wrote:

> > From: Paul Pogonyshev <pogonyshev@gmail.com>
> > Date: Tue, 6 Oct 2020 17:34:34 +0200
> >
> > I use English and Russian keyboard layouts. For every single application
> I don't need to care which layout is
> > currently selected for shortcuts, e.g. Ctrl+S and Ctrl+Ы do the same (S
> and Ы are on the same physical
> > key). Of course, in Emacs it doesn't work this way: C-s triggers
> Isearch, but C-ы "is undefined".
>
> Do you know how to retrieve the key code corresponding to the English
> keyboard layout when the active layout is something else, like
> Russian?  Does someone else here know?
>
> Or maybe the keyboard driver can be configured to return ASCII
> characters when the Ctrl key is held, even when the layout is
> non-English?
>
> What Emacs does, AFAICT, when a key press event comes in is call the
> Xlib function that returns the character produced by the keyboard, and
> that character depends on the current keyboard layout.  We then add
> the modifier keys to that character.  I don't see the "original"
> English-layout character available anywhere in that code, but maybe
> I'm missing something.
>
> A workaround is to bind C-ы to the same command as C-s (and similarly
> with other combinations that matter).
>

[-- Attachment #2: Type: text/html, Size: 3475 bytes --]

bug#43830: keyboard layout handling incompatible with rest of the OS

[Eli Zaretskii]

> From: Paul Pogonyshev <pogonyshev@gmail.com>
> Date: Tue, 6 Oct 2020 19:48:05 +0200
> Cc: 43830@debbugs.gnu.org
> 
> > Do you know how to retrieve the key code corresponding to the English
> > keyboard layout when the active layout is something else, like
> > Russian?
> 
> No, but there must be a way, because both GTK+ and KDE know how to do
> this. GTK+ also can do this when run under KDE, so I'm pretty sure the
> information itself comes from X (xkb), not e.g. from GNOME configuration.

Then I guess we will have to wait for someone who could explain how
this can be done in Emacs.

Thanks.

bug#43830: keyboard layout handling incompatible with rest of the OS

[Juri Linkov]

> I use English and Russian keyboard layouts. For every single application I
> don't need to care which layout is currently selected for shortcuts, e.g.
> Ctrl+S and Ctrl+Ы do the same (S and Ы are on the same physical key). Of
> course, in Emacs it doesn't work this way: C-s triggers Isearch, but C-ы
> "is undefined".

You can use the package https://github.com/a13/reverse-im.el
that was created from code that I wrote to address your previous request
some years ago.

bug#43830: keyboard layout handling incompatible with rest of the OS

[Paul Pogonyshev]

[-- Attachment #1: Type: text/plain, Size: 1098 bytes --]

> You can use the package https://github.com/a13/reverse-im.el

Mostly works fine, but e.g. C-. in Russian doesn't work as C-/ (undo) for me
(in Russian layout '.' is on the same key as '/' in English).

Pretty sure the correct way would be for Emacs to support this natively,
not hack from Elisp side, which has no information about physical keys,
only typed characters.

> your previous request some years ago

Must be more than 10 years ago I guess, I used to type some Russian
text back then.

Paul


On Tue, 6 Oct 2020 at 20:47, Juri Linkov <juri@linkov.net> wrote:

> > I use English and Russian keyboard layouts. For every single application
> I
> > don't need to care which layout is currently selected for shortcuts, e.g.
> > Ctrl+S and Ctrl+Ы do the same (S and Ы are on the same physical key). Of
> > course, in Emacs it doesn't work this way: C-s triggers Isearch, but C-ы
> > "is undefined".
>
> You can use the package https://github.com/a13/reverse-im.el
> that was created from code that I wrote to address your previous request
> some years ago.
>

[-- Attachment #2: Type: text/html, Size: 1773 bytes --]

bug#43830: keyboard layout handling incompatible with rest of the OS

[Juri Linkov]

>> You can use the package https://github.com/a13/reverse-im.el
>
> Mostly works fine, but e.g. C-. in Russian doesn't work as C-/ (undo) for
> me
> (in Russian layout '.' is on the same key as '/' in English).

When your X xkb layout has some differences from an Emacs input method,
you need to adjust these mismatched keys, but this is not a big problem.

> Pretty sure the correct way would be for Emacs to support this natively,
> not hack from Elisp side, which has no information about physical keys,
> only typed characters.

Hack or not, but it works fine for many users (the git repo has 57 stars).

bug#43830: keyboard layout handling incompatible with rest of the OS

[Paul Pogonyshev]

[-- Attachment #1: Type: text/plain, Size: 1821 bytes --]

> When your X xkb layout has some differences from an Emacs input method,
> you need to adjust these mismatched keys, but this is not a big problem.

It doesn't. With switching Emacs input methods between English and Russian
I also get English '/' == Russian '.'. And so C-. in Russian input method
works
as C-/ in English, because it's the same physical key.

> Hack or not, but it works fine for many users (the git repo has 57 stars).

Yeah, one of those is mine. It doesn't work "fine", it works much better
than
nothing. But still not exactly fine. I rebound C-. as a workaround, because
anyway it has no standard binding. But I cannot do the same for M-. for
example.

Also, the same goes for many S-M-[digit] combinations, because characters
on the digit row are often different in Russian layout. For example, S-M-6
in
English layout translates to M-^, `delete-indentation', but in Russian S-M-6
becomes M-:, `eval-expression'. And so on.

So, it is a good workaround that *mostly* works. But it doesn't solve the
underlying issue. Which was successfully solved by other applications tens
of years ago.

Paul


On Tue, 6 Oct 2020 at 22:37, Juri Linkov <juri@linkov.net> wrote:

> >> You can use the package https://github.com/a13/reverse-im.el
> >
> > Mostly works fine, but e.g. C-. in Russian doesn't work as C-/ (undo) for
> > me
> > (in Russian layout '.' is on the same key as '/' in English).
>
> When your X xkb layout has some differences from an Emacs input method,
> you need to adjust these mismatched keys, but this is not a big problem.
>
> > Pretty sure the correct way would be for Emacs to support this natively,
> > not hack from Elisp side, which has no information about physical keys,
> > only typed characters.
>
> Hack or not, but it works fine for many users (the git repo has 57 stars).
>

[-- Attachment #2: Type: text/html, Size: 2595 bytes --]

bug#43830: keyboard layout handling incompatible with rest of the OS

[Juri Linkov]

>> When your X xkb layout has some differences from an Emacs input method,
>> you need to adjust these mismatched keys, but this is not a big problem.
>
> It doesn't. With switching Emacs input methods between English and Russian
> I also get English '/' == Russian '.'. And so C-. in Russian input method
> works
> as C-/ in English, because it's the same physical key.
>
> Also, the same goes for many S-M-[digit] combinations, because characters
> on the digit row are often different in Russian layout. For example, S-M-6
> in
> English layout translates to M-^, `delete-indentation', but in Russian
> S-M-6
> becomes M-:, `eval-expression'. And so on.

All these problems are because of mismatch between your X layout and
your Emacs input method.  Emacs doesn't know the X layout, so you need
to define it in Emacs by adapting an existing input method,
or defining different keys manually.

> So, it is a good workaround that *mostly* works. But it doesn't solve the
> underlying issue. Which was successfully solved by other applications tens
> of years ago.

We already discussed this 10 years ago, and the conclusion was that
it would require too fundamental changes in how Emacs processes keystrokes.

If now you have new ideas about how this would be possible to implement
by keeping backward-compatibility of the existing design,
patches that demonstrate the ideas are welcome.

bug#43830: keyboard layout handling incompatible with rest of the OS

[Eli Zaretskii]

> From: Juri Linkov <juri@linkov.net>
> Date: Wed, 07 Oct 2020 11:16:12 +0300
> Cc: 43830@debbugs.gnu.org
> 
> > So, it is a good workaround that *mostly* works. But it doesn't solve the
> > underlying issue. Which was successfully solved by other applications tens
> > of years ago.
> 
> We already discussed this 10 years ago, and the conclusion was that
> it would require too fundamental changes in how Emacs processes keystrokes.

Can you point me to that discussion?

Thanks.

bug#43830: keyboard layout handling incompatible with rest of the OS

[Juri Linkov]

>> > So, it is a good workaround that *mostly* works. But it doesn't solve the
>> > underlying issue. Which was successfully solved by other applications tens
>> > of years ago.
>>
>> We already discussed this 10 years ago, and the conclusion was that
>> it would require too fundamental changes in how Emacs processes keystrokes.
>
> Can you point me to that discussion?

https://lists.gnu.org/archive/html/emacs-devel/2005-11/msg01237.html

bug#43830: keyboard layout handling incompatible with rest of the OS

[Eli Zaretskii]

> From: Juri Linkov <juri@linkov.net>
> Cc: pogonyshev@gmail.com,  43830@debbugs.gnu.org
> Date: Wed, 07 Oct 2020 22:01:47 +0300
> 
> >> We already discussed this 10 years ago, and the conclusion was that
> >> it would require too fundamental changes in how Emacs processes keystrokes.
> >
> > Can you point me to that discussion?
> 
> https://lists.gnu.org/archive/html/emacs-devel/2005-11/msg01237.html

Thanks.

My take out of that discussion:

 . There's a patch in
   https://lists.gnu.org/archive/html/emacs-devel/2005-11/msg01384.html
   which seems to allow what Paul wanted with single characters with
   modifiers, such as C-z or M-s.  That patch has a disadvantage that
   it disables AltGr, but if we install that patch as an optional
   feature, perhaps the disadvantage is not so bad?

 . The issue is more general than just a single character with a
   modifier, because key sequences such as "C-x z" will still not
   work: the 'z' will become the corresponding non-ASCII character
   when a non-US keyboard layout is used.  Therefore, the only general
   solution is for Emacs to be aware of the keyboard layout in use,
   and map the characters internally to their ASCII equivalents using
   that layout.

(The discussions also included LEIM features, but I think that is a
separate issue.)

bug#43830: keyboard layout handling incompatible with rest of the OS

[Paul Pogonyshev]

[-- Attachment #1: Type: text/plain, Size: 2620 bytes --]

> The issue is more general than just a single character with a
> modifier, because key sequences such as "C-x z" will still not
> work: the 'z' will become the corresponding non-ASCII character
> when a non-US keyboard layout is used.  Therefore, the only general
> solution is for Emacs to be aware of the keyboard layout in use,
> and map the characters internally to their ASCII equivalents using
> that layout.

Probably yes, I don't know how other applications do it internally.
As I mentioned, LibreOffice and IDEA (both are probably Java) do
it somehow, so there is a way. Maybe I'll try to dig through it later,
since I'm very familiar with Java.

By the way, what I forgot to mention, is that Emacs input modes
perform exactly like I want (i.e. bind to physical keys, so that C-.
in Russian works as C-/ in English; also e.g. C-ч й is translated to
C-x q, so even non-modified characters inside bindings work), but
they have the advantage of knowing the layout, of course. And,
as I mentioned, there are two problems with them: 1) I have to
use C-\ to switch and 2) configuration of `xkb' is bypassed.

Paul



On Thu, 8 Oct 2020 at 10:49, Eli Zaretskii <eliz@gnu.org> wrote:

> > From: Juri Linkov <juri@linkov.net>
> > Cc: pogonyshev@gmail.com,  43830@debbugs.gnu.org
> > Date: Wed, 07 Oct 2020 22:01:47 +0300
> >
> > >> We already discussed this 10 years ago, and the conclusion was that
> > >> it would require too fundamental changes in how Emacs processes
> keystrokes.
> > >
> > > Can you point me to that discussion?
> >
> > https://lists.gnu.org/archive/html/emacs-devel/2005-11/msg01237.html
>
> Thanks.
>
> My take out of that discussion:
>
>  . There's a patch in
>    https://lists.gnu.org/archive/html/emacs-devel/2005-11/msg01384.html
>    which seems to allow what Paul wanted with single characters with
>    modifiers, such as C-z or M-s.  That patch has a disadvantage that
>    it disables AltGr, but if we install that patch as an optional
>    feature, perhaps the disadvantage is not so bad?
>
>  . The issue is more general than just a single character with a
>    modifier, because key sequences such as "C-x z" will still not
>    work: the 'z' will become the corresponding non-ASCII character
>    when a non-US keyboard layout is used.  Therefore, the only general
>    solution is for Emacs to be aware of the keyboard layout in use,
>    and map the characters internally to their ASCII equivalents using
>    that layout.
>
> (The discussions also included LEIM features, but I think that is a
> separate issue.)
>

[-- Attachment #2: Type: text/html, Size: 3703 bytes --]

bug#43830: keyboard layout handling incompatible with rest of the OS

[Paul Pogonyshev]

[-- Attachment #1: Type: text/plain, Size: 3683 bytes --]

Apparently what Java does internally is calling function
XkbTranslateKeyCode() to translate 'keycode' that corresponds
to a physical key into a character using the current XKB layout.
And then passes on its own KeyEvent object that contains both
the character and the physical typed key, with downstream
code using either of the two, depending on situation (i.e. when
text is being typed, character is used, when bindings like Alt+X
are activated X states for the physical key, which can also e.g.
type Ч in Russian layout).

However, in Elisp this is further complicated by there being no
real KeyEvent structure, instead it's a single integer as far as I
can see. Since this is not OOP, changing anything here would
mean a thousand incompatibilities all over the place, so it's
easier to just give up and live with the non-perfect workaround
of reverse-im as suggested.

Paul


On Thu, 8 Oct 2020 at 15:58, Paul Pogonyshev <pogonyshev@gmail.com> wrote:

> > The issue is more general than just a single character with a
> > modifier, because key sequences such as "C-x z" will still not
> > work: the 'z' will become the corresponding non-ASCII character
> > when a non-US keyboard layout is used.  Therefore, the only general
> > solution is for Emacs to be aware of the keyboard layout in use,
> > and map the characters internally to their ASCII equivalents using
> > that layout.
>
> Probably yes, I don't know how other applications do it internally.
> As I mentioned, LibreOffice and IDEA (both are probably Java) do
> it somehow, so there is a way. Maybe I'll try to dig through it later,
> since I'm very familiar with Java.
>
> By the way, what I forgot to mention, is that Emacs input modes
> perform exactly like I want (i.e. bind to physical keys, so that C-.
> in Russian works as C-/ in English; also e.g. C-ч й is translated to
> C-x q, so even non-modified characters inside bindings work), but
> they have the advantage of knowing the layout, of course. And,
> as I mentioned, there are two problems with them: 1) I have to
> use C-\ to switch and 2) configuration of `xkb' is bypassed.
>
> Paul
>
>
>
> On Thu, 8 Oct 2020 at 10:49, Eli Zaretskii <eliz@gnu.org> wrote:
>
>> > From: Juri Linkov <juri@linkov.net>
>> > Cc: pogonyshev@gmail.com,  43830@debbugs.gnu.org
>> > Date: Wed, 07 Oct 2020 22:01:47 +0300
>> >
>> > >> We already discussed this 10 years ago, and the conclusion was that
>> > >> it would require too fundamental changes in how Emacs processes
>> keystrokes.
>> > >
>> > > Can you point me to that discussion?
>> >
>> > https://lists.gnu.org/archive/html/emacs-devel/2005-11/msg01237.html
>>
>> Thanks.
>>
>> My take out of that discussion:
>>
>>  . There's a patch in
>>    https://lists.gnu.org/archive/html/emacs-devel/2005-11/msg01384.html
>>    which seems to allow what Paul wanted with single characters with
>>    modifiers, such as C-z or M-s.  That patch has a disadvantage that
>>    it disables AltGr, but if we install that patch as an optional
>>    feature, perhaps the disadvantage is not so bad?
>>
>>  . The issue is more general than just a single character with a
>>    modifier, because key sequences such as "C-x z" will still not
>>    work: the 'z' will become the corresponding non-ASCII character
>>    when a non-US keyboard layout is used.  Therefore, the only general
>>    solution is for Emacs to be aware of the keyboard layout in use,
>>    and map the characters internally to their ASCII equivalents using
>>    that layout.
>>
>> (The discussions also included LEIM features, but I think that is a
>> separate issue.)
>>
>

[-- Attachment #2: Type: text/html, Size: 5263 bytes --]

bug#43830: keyboard layout handling incompatible with rest of the OS

[Eli Zaretskii]

> From: Paul Pogonyshev <pogonyshev@gmail.com>
> Date: Wed, 28 Oct 2020 01:43:04 +0100
> Cc: Juri Linkov <juri@linkov.net>, 43830@debbugs.gnu.org
> 
> Apparently what Java does internally is calling function
> XkbTranslateKeyCode() to translate 'keycode' that corresponds
> to a physical key into a character using the current XKB layout.

Is XKB universally available?  AFAICT, we don't require it, but use it
if available (not for XkbTranslateKeyCode, though).

> However, in Elisp this is further complicated by there being no
> real KeyEvent structure, instead it's a single integer as far as I
> can see.

Why would you need that?  If we decide to use XkbTranslateKeyCode, we
could translate the keycode in C, according to some logic that took
into account the modifiers and perhaps also some user options, and
returned the resulting translated character.

bug#43830: keyboard layout handling incompatible with rest of the OS

[Paul Pogonyshev]

[-- Attachment #1: Type: text/plain, Size: 2383 bytes --]

> Is XKB universally available?  AFAICT, we don't require it, but use it
> if available (not for XkbTranslateKeyCode, though).

No. E.g. Java has functions to process keystrokes with and without XKB.
I have no idea how it works without: I use KDE that internally uses XKB.
But AFAIK GNOME (optionally) replaces XKB with something else.

> > However, in Elisp this is further complicated by there being no
> > real KeyEvent structure, instead it's a single integer as far as I
> > can see.
>
> Why would you need that?  If we decide to use XkbTranslateKeyCode, we
> could translate the keycode in C, according to some logic that took
> into account the modifiers and perhaps also some user options, and
> returned the resulting translated character.

The point is that the character is not enough, you need to know both
the character and the physical key (you cannot reconstruct the key
from the character alone). E.g. suppose I type 'й' in Russian layout.
If it is a self-inserting command, character 'й' should be added to the
active buffer. But if I'm typing a multikey binding, it should be
interpreted as 'q' (it's the same physical key), so that e.g. 'C-ч й' is
translated to 'C-x q'. Without looking, I'm pretty sure this goes well
into Elisp part of Emacs, and changing key events from integers to
something else would be a compatibility nightmare.

Paul


On Wed, 28 Oct 2020 at 16:06, Eli Zaretskii <eliz@gnu.org> wrote:

> > From: Paul Pogonyshev <pogonyshev@gmail.com>
> > Date: Wed, 28 Oct 2020 01:43:04 +0100
> > Cc: Juri Linkov <juri@linkov.net>, 43830@debbugs.gnu.org
> >
> > Apparently what Java does internally is calling function
> > XkbTranslateKeyCode() to translate 'keycode' that corresponds
> > to a physical key into a character using the current XKB layout.
>
> Is XKB universally available?  AFAICT, we don't require it, but use it
> if available (not for XkbTranslateKeyCode, though).
>
> > However, in Elisp this is further complicated by there being no
> > real KeyEvent structure, instead it's a single integer as far as I
> > can see.
>
> Why would you need that?  If we decide to use XkbTranslateKeyCode, we
> could translate the keycode in C, according to some logic that took
> into account the modifiers and perhaps also some user options, and
> returned the resulting translated character.
>

[-- Attachment #2: Type: text/html, Size: 3295 bytes --]

bug#43830: keyboard layout handling incompatible with rest of the OS

[Eli Zaretskii]

> From: Paul Pogonyshev <pogonyshev@gmail.com>
> Date: Wed, 28 Oct 2020 17:16:59 +0100
> Cc: Juri Linkov <juri@linkov.net>, 43830@debbugs.gnu.org
> 
> > > However, in Elisp this is further complicated by there being no
> > > real KeyEvent structure, instead it's a single integer as far as I
> > > can see.
> > 
> > Why would you need that?  If we decide to use XkbTranslateKeyCode, we
> > could translate the keycode in C, according to some logic that took
> > into account the modifiers and perhaps also some user options, and
> > returned the resulting translated character.
> 
> The point is that the character is not enough, you need to know both
> the character and the physical key (you cannot reconstruct the key
> from the character alone). E.g. suppose I type 'й' in Russian layout.
> If it is a self-inserting command, character 'й' should be added to the
> active buffer. But if I'm typing a multikey binding, it should be
> interpreted as 'q' (it's the same physical key), so that e.g. 'C-ч й' is
> translated to 'C-x q'. Without looking, I'm pretty sure this goes well
> into Elisp part of Emacs

No, AFAIR this is all done in C.

bug#43830: keyboard layout handling incompatible with rest of the OS

[Paul Pogonyshev]

[-- Attachment #1: Type: text/plain, Size: 1432 bytes --]

> No, AFAIR this is all done in C.

What about functions like `read-event'? It returns integers if I press
M-[letter] or C-[letter].

Paul

On Wed, 28 Oct 2020 at 17:31, Eli Zaretskii <eliz@gnu.org> wrote:

> > From: Paul Pogonyshev <pogonyshev@gmail.com>
> > Date: Wed, 28 Oct 2020 17:16:59 +0100
> > Cc: Juri Linkov <juri@linkov.net>, 43830@debbugs.gnu.org
> >
> > > > However, in Elisp this is further complicated by there being no
> > > > real KeyEvent structure, instead it's a single integer as far as I
> > > > can see.
> > >
> > > Why would you need that?  If we decide to use XkbTranslateKeyCode, we
> > > could translate the keycode in C, according to some logic that took
> > > into account the modifiers and perhaps also some user options, and
> > > returned the resulting translated character.
> >
> > The point is that the character is not enough, you need to know both
> > the character and the physical key (you cannot reconstruct the key
> > from the character alone). E.g. suppose I type 'й' in Russian layout.
> > If it is a self-inserting command, character 'й' should be added to the
> > active buffer. But if I'm typing a multikey binding, it should be
> > interpreted as 'q' (it's the same physical key), so that e.g. 'C-ч й' is
> > translated to 'C-x q'. Without looking, I'm pretty sure this goes well
> > into Elisp part of Emacs
>
> No, AFAIR this is all done in C.
>

[-- Attachment #2: Type: text/html, Size: 2146 bytes --]

bug#43830: keyboard layout handling incompatible with rest of the OS

[Eli Zaretskii]

> From: Paul Pogonyshev <pogonyshev@gmail.com>
> Date: Sun, 1 Nov 2020 01:19:57 +0100
> Cc: Juri Linkov <juri@linkov.net>, 43830@debbugs.gnu.org
> 
> > No, AFAIR this is all done in C.
> 
> What about functions like `read-event'? It returns integers if I press
> M-[letter] or C-[letter].

read-event is also implemented in C.  But maybe I don't understand
your question.

bug#43830: keyboard layout handling incompatible with rest of the OS

[Juri Linkov]

>> Why would you need that?  If we decide to use XkbTranslateKeyCode, we
>> could translate the keycode in C, according to some logic that took
>> into account the modifiers and perhaps also some user options, and
>> returned the resulting translated character.
>
> The point is that the character is not enough, you need to know both
> the character and the physical key (you cannot reconstruct the key
> from the character alone). E.g. suppose I type 'й' in Russian layout.
> If it is a self-inserting command, character 'й' should be added to the
> active buffer. But if I'm typing a multikey binding, it should be
> interpreted as 'q' (it's the same physical key), so that e.g. 'C-ч й' is
> translated to 'C-x q'.

What is worse is that in a writable buffer, typing 'й' should insert
this character untranslated, but in the same buffer when it's in
read-only view mode, typing the same 'й' should translate it to 'q'
and quit the buffer with the View-quit command.  When using reverse-im
with local-function-key-map, the Help buffer says:

  q (translated from й) runs the command View-quit.

So the question is whether it's possible to do the same using
XkbTranslateKeyCode?  The local-function-key-map is smart enough
to not translate self-inserting keys.  Can code for XkbTranslateKeyCode
use the same condition to detect self-inserting keys?

bug#43830: keyboard layout handling incompatible with rest of the OS

[Eli Zaretskii]

> From: Juri Linkov <juri@linkov.net>
> Cc: Eli Zaretskii <eliz@gnu.org>,  43830@debbugs.gnu.org
> Date: Sun, 01 Nov 2020 09:53:08 +0200
> 
> > The point is that the character is not enough, you need to know both
> > the character and the physical key (you cannot reconstruct the key
> > from the character alone). E.g. suppose I type 'й' in Russian layout.
> > If it is a self-inserting command, character 'й' should be added to the
> > active buffer. But if I'm typing a multikey binding, it should be
> > interpreted as 'q' (it's the same physical key), so that e.g. 'C-ч й' is
> > translated to 'C-x q'.
> 
> What is worse is that in a writable buffer, typing 'й' should insert
> this character untranslated, but in the same buffer when it's in
> read-only view mode, typing the same 'й' should translate it to 'q'
> and quit the buffer with the View-quit command.  When using reverse-im
> with local-function-key-map, the Help buffer says:
> 
>   q (translated from й) runs the command View-quit.
> 
> So the question is whether it's possible to do the same using
> XkbTranslateKeyCode?  The local-function-key-map is smart enough
> to not translate self-inserting keys.  Can code for XkbTranslateKeyCode
> use the same condition to detect self-inserting keys?

Why do you want XkbTranslateKeyCode to do this? why not the code in
keyboard.c that reads the event queue and invokes commands?  If we
make both the key and the character available to keyboard.c, it should
be able to figure out what is TRT in each situation, I think.

bug#43830: keyboard layout handling incompatible with rest of the OS

[Juri Linkov]

>> What is worse is that in a writable buffer, typing 'й' should insert
>> this character untranslated, but in the same buffer when it's in
>> read-only view mode, typing the same 'й' should translate it to 'q'
>> and quit the buffer with the View-quit command.  When using reverse-im
>> with local-function-key-map, the Help buffer says:
>> 
>>   q (translated from й) runs the command View-quit.
>> 
>> So the question is whether it's possible to do the same using
>> XkbTranslateKeyCode?  The local-function-key-map is smart enough
>> to not translate self-inserting keys.  Can code for XkbTranslateKeyCode
>> use the same condition to detect self-inserting keys?
>
> Why do you want XkbTranslateKeyCode to do this? why not the code in
> keyboard.c that reads the event queue and invokes commands?  If we
> make both the key and the character available to keyboard.c, it should
> be able to figure out what is TRT in each situation, I think.

Actually by the phrase "code for XkbTranslateKeyCode" I meant the code
in keyboard.c that could use XkbTranslateKeyCode to do the key translation
between physical keys and characters.

bug#43830: keyboard layout handling incompatible with rest of the OS

[Eli Zaretskii]

> From: Juri Linkov <juri@linkov.net>
> Cc: pogonyshev@gmail.com,  43830@debbugs.gnu.org
> Date: Sun, 01 Nov 2020 20:27:41 +0200
> 
> > Why do you want XkbTranslateKeyCode to do this? why not the code in
> > keyboard.c that reads the event queue and invokes commands?  If we
> > make both the key and the character available to keyboard.c, it should
> > be able to figure out what is TRT in each situation, I think.
> 
> Actually by the phrase "code for XkbTranslateKeyCode" I meant the code
> in keyboard.c that could use XkbTranslateKeyCode to do the key translation
> between physical keys and characters.

If we keep both the translated and the untranslated character in the
queue, we won't need to call XkbTranslateKeyCode from keyboard.c,
because we'd already have the information.

bug#43830: keyboard layout handling incompatible with rest of the OS

[Paul Pogonyshev]

[-- Attachment #1: Type: text/plain, Size: 2937 bytes --]

> Could you please give an example how you bind such key sequences as
> 'C-.' and 'C-ч й' in Emacs input modes?  How they do translation
> to physical keys without using xkb?

I don't and that's the whole point. I want that when _any_ keymap
defines `M-q', this keybinding can be activated by typing M-q or M-й,
because this is the same physical key. Automatically.

`reverse-im' achieves this but _only_ for keys that type letters in
the second layout. E.g. M-/ (which I often use) won't work in Russian
layout because that physical letter types '.' in this layout.

> > What about functions like `read-event'? It returns integers if I press
> > M-[letter] or C-[letter].
>
> read-event is also implemented in C.  But maybe I don't understand
> your question.

I mean, what about the cases where it is called from Elisp?  It is
implemented in C, but also is publicly available.

I have come up with two ideas:

1. `read-event' and its internal C implementation grow an optional
parameter that says whether to return character as if being typed (as
now) or for keybinding use (i.e. from physical keys).

2. Alternatively, if this cannot be determined in advance (i.e. before
calling `read-event' etc.), these functions could set variable named sth.
like `last-keybinding-keycode'. Then the caller would use either the
return value (as now) or, if it wants, the value of the variable instead.

Paul

On Sun, 1 Nov 2020 at 09:01, Juri Linkov <juri@linkov.net> wrote:

> >> Why would you need that?  If we decide to use XkbTranslateKeyCode, we
> >> could translate the keycode in C, according to some logic that took
> >> into account the modifiers and perhaps also some user options, and
> >> returned the resulting translated character.
> >
> > The point is that the character is not enough, you need to know both
> > the character and the physical key (you cannot reconstruct the key
> > from the character alone). E.g. suppose I type 'й' in Russian layout.
> > If it is a self-inserting command, character 'й' should be added to the
> > active buffer. But if I'm typing a multikey binding, it should be
> > interpreted as 'q' (it's the same physical key), so that e.g. 'C-ч й' is
> > translated to 'C-x q'.
>
> What is worse is that in a writable buffer, typing 'й' should insert
> this character untranslated, but in the same buffer when it's in
> read-only view mode, typing the same 'й' should translate it to 'q'
> and quit the buffer with the View-quit command.  When using reverse-im
> with local-function-key-map, the Help buffer says:
>
>   q (translated from й) runs the command View-quit.
>
> So the question is whether it's possible to do the same using
> XkbTranslateKeyCode?  The local-function-key-map is smart enough
> to not translate self-inserting keys.  Can code for XkbTranslateKeyCode
> use the same condition to detect self-inserting keys?
>

[-- Attachment #2: Type: text/html, Size: 3833 bytes --]

bug#43830: keyboard layout handling incompatible with rest of the OS

[Eli Zaretskii]

> From: Paul Pogonyshev <pogonyshev@gmail.com>
> Date: Sun, 1 Nov 2020 17:51:00 +0100
> Cc: Eli Zaretskii <eliz@gnu.org>, 43830@debbugs.gnu.org
> 
> > > What about functions like `read-event'? It returns integers if I press
> > > M-[letter] or C-[letter].
> > 
> > read-event is also implemented in C.  But maybe I don't understand
> > your question.
> 
> I mean, what about the cases where it is called from Elisp?  It is
> implemented in C, but also is publicly available.
> 
> I have come up with two ideas:
> 
> 1. `read-event' and its internal C implementation grow an optional
> parameter that says whether to return character as if being typed (as
> now) or for keybinding use (i.e. from physical keys).
> 
> 2. Alternatively, if this cannot be determined in advance (i.e. before
> calling `read-event' etc.), these functions could set variable named sth.
> like `last-keybinding-keycode'. Then the caller would use either the
> return value (as now) or, if it wants, the value of the variable instead.

What I had in mind was

 0. read-event will figure out by itself whether it should return the
 ASCII character or a non-ASCII character, and return that.

bug#43830: keyboard layout handling incompatible with rest of the OS

[Paul Pogonyshev]

[-- Attachment #1: Type: text/plain, Size: 1572 bytes --]

> read-event will figure out by itself whether it should return the
> ASCII character or a non-ASCII character, and return that.

I'm not sure that is possible, because it doesn't really know how it
is going to be processed. But you must be much more familiar with
Emacs internals.

Paul

On Sun, 1 Nov 2020 at 18:25, Eli Zaretskii <eliz@gnu.org> wrote:

> > From: Paul Pogonyshev <pogonyshev@gmail.com>
> > Date: Sun, 1 Nov 2020 17:51:00 +0100
> > Cc: Eli Zaretskii <eliz@gnu.org>, 43830@debbugs.gnu.org
> >
> > > > What about functions like `read-event'? It returns integers if I
> press
> > > > M-[letter] or C-[letter].
> > >
> > > read-event is also implemented in C.  But maybe I don't understand
> > > your question.
> >
> > I mean, what about the cases where it is called from Elisp?  It is
> > implemented in C, but also is publicly available.
> >
> > I have come up with two ideas:
> >
> > 1. `read-event' and its internal C implementation grow an optional
> > parameter that says whether to return character as if being typed (as
> > now) or for keybinding use (i.e. from physical keys).
> >
> > 2. Alternatively, if this cannot be determined in advance (i.e. before
> > calling `read-event' etc.), these functions could set variable named sth.
> > like `last-keybinding-keycode'. Then the caller would use either the
> > return value (as now) or, if it wants, the value of the variable instead.
>
> What I had in mind was
>
>  0. read-event will figure out by itself whether it should return the
>  ASCII character or a non-ASCII character, and return that.
>

[-- Attachment #2: Type: text/html, Size: 2314 bytes --]

bug#43830: keyboard layout handling incompatible with rest of the OS

[Eli Zaretskii]

> From: Paul Pogonyshev <pogonyshev@gmail.com>
> Date: Sun, 1 Nov 2020 19:56:09 +0100
> Cc: Juri Linkov <juri@linkov.net>, 43830@debbugs.gnu.org
> 
> > read-event will figure out by itself whether it should return the
> > ASCII character or a non-ASCII character, and return that.
> 
> I'm not sure that is possible, because it doesn't really know how it
> is going to be processed.

It knows the binding of the character, and it knows other things.

Nothing else makes sense to me, because exposing this info to Lisp
means every Lisp program which deals with input will have to decide
what to do with such events.

bug#43830: keyboard layout handling incompatible with rest of the OS

[Paul Pogonyshev]

[-- Attachment #1: Type: text/plain, Size: 1725 bytes --]

> It knows the binding of the character, and it knows other things.

When it knows the bindings (99% of cases) it is called internally by
the command loop, so it doesn't even go into Elisp domain here.
If something calls `read-event' or similar on the Elisp level,
presumably they don't use keymaps for whatever reasons and
process events specially. In this case `read-event' cannot possibly
know what The Right Thing is.

> Nothing else makes sense to me, because exposing this info to Lisp
> means every Lisp program which deals with input will have to decide
> what to do with such events.

As I understand it, 99% of handling is done by the main command
loop at the C level, so it is not a problem to do it once there. I still
would give Elisp access so that special users of `read-event' etc. can
follow suit, but this would not oblige them to do anything. Programs
and modes that don't do anything special (either because they are
old or do not care) will just behave as now, with no effective changes.

Paul

On Sun, 1 Nov 2020 at 20:33, Eli Zaretskii <eliz@gnu.org> wrote:

> > From: Paul Pogonyshev <pogonyshev@gmail.com>
> > Date: Sun, 1 Nov 2020 19:56:09 +0100
> > Cc: Juri Linkov <juri@linkov.net>, 43830@debbugs.gnu.org
> >
> > > read-event will figure out by itself whether it should return the
> > > ASCII character or a non-ASCII character, and return that.
> >
> > I'm not sure that is possible, because it doesn't really know how it
> > is going to be processed.
>
> It knows the binding of the character, and it knows other things.
>
> Nothing else makes sense to me, because exposing this info to Lisp
> means every Lisp program which deals with input will have to decide
> what to do with such events.
>

[-- Attachment #2: Type: text/html, Size: 2491 bytes --]

bug#43830: keyboard layout handling incompatible with rest of the OS

[Arthur Miller]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Paul Pogonyshev <pogonyshev@gmail.com>
>> Date: Sun, 1 Nov 2020 19:56:09 +0100
>> Cc: Juri Linkov <juri@linkov.net>, 43830@debbugs.gnu.org
>> 
>> > read-event will figure out by itself whether it should return the
>> > ASCII character or a non-ASCII character, and return that.
>> 
>> I'm not sure that is possible, because it doesn't really know how it
>> is going to be processed.
>
> It knows the binding of the character, and it knows other things.
>
> Nothing else makes sense to me, because exposing this info to Lisp
> means every Lisp program which deals with input will have to decide
> what to do with such events.

Couldn't Emacs just pack scancodes into some event structure similar as
raw_input in windows does? But I don't think Elisp programs really would
have use of scancodes anyway.

Would it be possible to use scancodes for the shortcuts internally in
the interpreter itself?

Shortcuts could be specified by users and scripts as they are now
(as keys/characters); but when Emacs starts, it could populate a lookup
table based on the current layout with scancodes, and then translate
shortcuts from keys to scancodes for internal usage. If user chagnes
keyboard layout on the fly, new lookup table would be generated, because
virtual keys would change but scancodes would stay same; so it wouldn't
matter if some key jumps from one physical key to another. That wouldn't
require to expose scancodes to Lisp either. But that would mean that
routines for inserting text in buffers would have to work differently
from shortcut handling routines, with regard to input; I don't know if
that is case now or if it is possible.

Is something like that possible to implement in current Emacs? Woukd it
be too much work?

I don't know if it is possible to automatically detect keyboard layout
change in all cases. I know that Emacs automatically switches to
different layout when I change keyboard with loadkeys or in vconsole.conf.

bug#43830: keyboard layout handling incompatible with rest of the OS

[Eli Zaretskii]

> From: Arthur Miller <arthur.miller@live.com>
> Cc: Paul Pogonyshev <pogonyshev@gmail.com>,  43830@debbugs.gnu.org,
>   juri@linkov.net
> Date: Mon, 02 Nov 2020 05:41:06 +0100
> 
> > Nothing else makes sense to me, because exposing this info to Lisp
> > means every Lisp program which deals with input will have to decide
> > what to do with such events.
> 
> Couldn't Emacs just pack scancodes into some event structure

That's what I was prfoposing to do, on the C level.

> Would it be possible to use scancodes for the shortcuts internally in
> the interpreter itself?

I don't think I understand what shortcuts you have in mind here.

> Shortcuts could be specified by users and scripts as they are now
> (as keys/characters); but when Emacs starts, it could populate a lookup
> table based on the current layout with scancodes, and then translate
> shortcuts from keys to scancodes for internal usage. If user chagnes
> keyboard layout on the fly, new lookup table would be generated, because
> virtual keys would change but scancodes would stay same; so it wouldn't
> matter if some key jumps from one physical key to another. That wouldn't
> require to expose scancodes to Lisp either. But that would mean that
> routines for inserting text in buffers would have to work differently
> from shortcut handling routines, with regard to input; I don't know if
> that is case now or if it is possible.
> 
> Is something like that possible to implement in current Emacs? Woukd it
> be too much work?

I think you are forgetting how many different methods are there on a
modern system to generate keyboard input.  There are system input
methods, there are virtual keyboards, etc. etc.  And I don't think we
can easily read keyboard on scancode level, either, even if we wanted
to.  But I'm not an expert.

> I don't know if it is possible to automatically detect keyboard layout
> change in all cases.

AFAIK, we only support that on MS-Windows.

> I know that Emacs automatically switches to different layout when I
> change keyboard with loadkeys or in vconsole.conf.

That's because we read characters, and those don't change.

bug#43830: keyboard layout handling incompatible with rest of the OS

[Juri Linkov]

> By the way, what I forgot to mention, is that Emacs input modes
> perform exactly like I want (i.e. bind to physical keys, so that C-.
> in Russian works as C-/ in English; also e.g. C-ч й is translated to
> C-x q, so even non-modified characters inside bindings work), but
> they have the advantage of knowing the layout, of course.

Could you please give an example how you bind such key sequences as
'C-.' and 'C-ч й' in Emacs input modes?  How they do translation
to physical keys without using xkb?

bug#43830: keyboard layout handling incompatible with rest of the OS

[Paul Pogonyshev]

[-- Attachment #1: Type: text/plain, Size: 2068 bytes --]

> All these problems are because of mismatch between your X layout and
> your Emacs input method.  Emacs doesn't know the X layout, so you need
> to define it in Emacs by adapting an existing input method,
> or defining different keys manually.

Sorry, I don't understand. I also don't see anything about this in the
documentation (README).

I want physical key that is '/' in English to type '.' in Russian (because
that is what it does in russian-computer), but invoke shortcuts bound to
C-/, M-/ etc. Is that possible with `reverse-im'?

Paul


On Wed, 7 Oct 2020 at 10:43, Juri Linkov <juri@linkov.net> wrote:

> >> When your X xkb layout has some differences from an Emacs input method,
> >> you need to adjust these mismatched keys, but this is not a big problem.
> >
> > It doesn't. With switching Emacs input methods between English and
> Russian
> > I also get English '/' == Russian '.'. And so C-. in Russian input method
> > works
> > as C-/ in English, because it's the same physical key.
> >
> > Also, the same goes for many S-M-[digit] combinations, because characters
> > on the digit row are often different in Russian layout. For example,
> S-M-6
> > in
> > English layout translates to M-^, `delete-indentation', but in Russian
> > S-M-6
> > becomes M-:, `eval-expression'. And so on.
>
> All these problems are because of mismatch between your X layout and
> your Emacs input method.  Emacs doesn't know the X layout, so you need
> to define it in Emacs by adapting an existing input method,
> or defining different keys manually.
>
> > So, it is a good workaround that *mostly* works. But it doesn't solve the
> > underlying issue. Which was successfully solved by other applications
> tens
> > of years ago.
>
> We already discussed this 10 years ago, and the conclusion was that
> it would require too fundamental changes in how Emacs processes keystrokes.
>
> If now you have new ideas about how this would be possible to implement
> by keeping backward-compatibility of the existing design,
> patches that demonstrate the ideas are welcome.
>

[-- Attachment #2: Type: text/html, Size: 2701 bytes --]

bug#43830: keyboard layout handling incompatible with rest of the OS

[Juri Linkov]

>> All these problems are because of mismatch between your X layout and
>> your Emacs input method.  Emacs doesn't know the X layout, so you need
>> to define it in Emacs by adapting an existing input method,
>> or defining different keys manually.
>
> Sorry, I don't understand. I also don't see anything about this in the
> documentation (README).
>
> I want physical key that is '/' in English to type '.' in Russian (because
> that is what it does in russian-computer), but invoke shortcuts bound to
> C-/, M-/ etc. Is that possible with `reverse-im'?

You said this issue was successfully solved by other applications tens of
years ago.  Do these apps allow you typing physical key '/' to get '.'
while keeping invoking C-/, M-/ shortcuts?  I know no such applications,
they all invoke C-. and M-., not C-/ and M-/.

But in Emacs you can do everything you want.  If `reverse-im'
doesn't handle this automatically, you can redefine key mappings for
'/', 'C-/' and 'M-/' manually (using some code from `reverse-im').

bug#43830: keyboard layout handling incompatible with rest of the OS

[Paul Pogonyshev]

[-- Attachment #1: Type: text/plain, Size: 2703 bytes --]

> You said this issue was successfully solved by other applications tens of
> years ago.  Do these apps allow you typing physical key '/' to get '.'
> while keeping invoking C-/, M-/ shortcuts?  I know no such applications,
> they all invoke C-. and M-., not C-/ and M-/.

Actually, I admit I haven't even thought about that. Not many apps have
so many shortcuts that they run out of letters, so I just assumed that they
work by physical key. But a quick test shows mixed results:

* KDE and GTK+ (at least Gimp) seem to work like `reverse-im', i.e. they
don't bind to physical keys, but to characters.

* LibreOffice and IntelliJ IDEA work like I described, i.e. as desired for
me.

* I also discovered that even some ancient X programs (xedit) are smart
enough to hande Ctrl-S and Ctrl-Ы the same, but I have no idea how to
test Ctrl-. there.

My subconciousness doesn't distinguish between letter and non-letter keys,
it remembers shorcuts by physical position of the keys. That's why I think
the second approach (i.e. LibreOffice's and IDEA's) is better.

> But in Emacs you can do everything you want.  If `reverse-im'
> doesn't handle this automatically, you can redefine key mappings for
> '/', 'C-/' and 'M-/' manually (using some code from `reverse-im').

Can I do that automagically somehow, given that I switch keyboard layout
three times a minute when I type? Doing it once won't help anything, since
that would break shortcuts for English layout, and I want them work the
same in _all layouts_.

Paul

On Wed, 7 Oct 2020 at 21:25, Juri Linkov <juri@linkov.net> wrote:

> >> All these problems are because of mismatch between your X layout and
> >> your Emacs input method.  Emacs doesn't know the X layout, so you need
> >> to define it in Emacs by adapting an existing input method,
> >> or defining different keys manually.
> >
> > Sorry, I don't understand. I also don't see anything about this in the
> > documentation (README).
> >
> > I want physical key that is '/' in English to type '.' in Russian
> (because
> > that is what it does in russian-computer), but invoke shortcuts bound to
> > C-/, M-/ etc. Is that possible with `reverse-im'?
>
> You said this issue was successfully solved by other applications tens of
> years ago.  Do these apps allow you typing physical key '/' to get '.'
> while keeping invoking C-/, M-/ shortcuts?  I know no such applications,
> they all invoke C-. and M-., not C-/ and M-/.
>
> But in Emacs you can do everything you want.  If `reverse-im'
> doesn't handle this automatically, you can redefine key mappings for
> '/', 'C-/' and 'M-/' manually (using some code from `reverse-im').
>

[-- Attachment #2: Type: text/html, Size: 3538 bytes --]

bug#43830: keyboard layout handling incompatible with rest of the OS

[Juri Linkov]

> Can I do that automagically somehow, given that I switch keyboard layout
> three times a minute when I type? Doing it once won't help anything, since
> that would break shortcuts for English layout, and I want them work the
> same in _all layouts_.

If you rebind e.g. C-. to another command to use in another layout,
it won't work with the initial layout indeed.  reverse-im works well
when you need to rebind only letters, not punctuation.  This problem
is described here: https://github.com/a13/reverse-im.el/issues/6