bug#12868: global keymap preceeds input-decode-map

bug#12868: global keymap preceeds input-decode-map

[Stefan Guath]

emacs-version: GNU Emacs 24.2.1 (x86_64-apple-darwin, NS apple-appkit-1038.36) of 2012-08-27 on bob.porkrind.org

To trigger bug:
emacs -nw -Q
(defun foo () (interactive) (message "foo"))
(global-set-key "\M-O" 'foo)

Bug symptons:
Arrow keys now inserts A,B,C,D in buffer.

Suggested bug reason:
Arrow keys sends M-O A (up), M-O B (down) etc. These should be translated to function keys <up>, <down> etc in input-decode-map before any key lookup. Somehow, the binding of M-O in the global keymap takes precedence. This would be expected with local-function-key-map as described in http://www.gnu.org/software/emacs/manual/html_node/elisp/Translation-Keymaps.html , but not with input-decode-map.

bug#12868: global keymap preceeds input-decode-map

[Stefan Monnier]

> emacs -nw -Q
> (defun foo () (interactive) (message "foo"))
> (global-set-key "\M-O" 'foo)
[...]
> Arrow keys now inserts A,B,C,D in buffer.
[...]
> Arrow keys sends M-O A (up), M-O B (down) etc. These should be
> translated to function keys <up>, <down> etc in input-decode-map
> before any key lookup.  Somehow, the binding of M-O in the global
> keymap takes precedence.

Indeed, thanks for bringing this up.  The cause for the bug is that the
rule for what goes first and what goes next is here ignored because we
stop waiting for more input as soon as a real binding is found
(i.e. right after M-O).

The predicate that decides when to stop waiting for more input would
need to say say "don't stop if we're in the middle of a potential
input-decode-map rewrite".  But then if you want to run `foo', you'd
have the problem that after hitting M-O, Emacs will not actually run
`foo' but will sit there waiting to see if the next key is one of
[ABCD].

The only way to get our cake and eat it too would be to use some kind of
timeout, which we have resisted so far.


        Stefan

bug#12868: global keymap preceeds input-decode-map

[Stefan Guath]

I understand, and agree that a timeout is a bad idea. This should be reflected in the manual though, just as it is for local-function-key-map ("Entries in local-function-key-map are ignored if they conflict with bindings made in the minor mode, local, or global keymaps"). Maybe it should also point out the consequences of binding escape sequences used as prefix for common function keys, i.e. `ESC [' and `ESC O'. One could even consider to unable colliding bindings to emphasize that you have to choose, rather than 'bugging out' silently.

BTW, the manual states that "The intent of key-translation-map is for users to map one character set to another, including ordinary characters normally bound to self-insert-command." Does this mean that key-translation-map is the recommended way to implement different layouts such as Colemak/Dvorak etc at the lowest possible level?

If yes, in the case of remapping `o' to `y' (Colemak example), M-O would never get through the input-decode-map filter and therefore never pop up to key-translation-map in order to produce M-Y. Is that correct?

/Stefan

On 12 nov 2012, at 15:32, Stefan Monnier wrote:

>> emacs -nw -Q
>> (defun foo () (interactive) (message "foo"))
>> (global-set-key "\M-O" 'foo)
> [...]
>> Arrow keys now inserts A,B,C,D in buffer.
> [...]
>> Arrow keys sends M-O A (up), M-O B (down) etc. These should be
>> translated to function keys <up>, <down> etc in input-decode-map
>> before any key lookup.  Somehow, the binding of M-O in the global
>> keymap takes precedence.
> 
> Indeed, thanks for bringing this up.  The cause for the bug is that the
> rule for what goes first and what goes next is here ignored because we
> stop waiting for more input as soon as a real binding is found
> (i.e. right after M-O).
> 
> The predicate that decides when to stop waiting for more input would
> need to say say "don't stop if we're in the middle of a potential
> input-decode-map rewrite".  But then if you want to run `foo', you'd
> have the problem that after hitting M-O, Emacs will not actually run
> `foo' but will sit there waiting to see if the next key is one of
> [ABCD].
> 
> The only way to get our cake and eat it too would be to use some kind of
> timeout, which we have resisted so far.
> 
> 
>        Stefan

bug#12868: global keymap preceeds input-decode-map

[Stefan Monnier]

> I understand, and agree that a timeout is a bad idea.  This should be
> reflected in the manual though, just as it is for local-function-key-map
> ("Entries in local-function-key-map are ignored if they conflict with
> bindings made in the minor mode, local, or global keymaps").

I find it difficult to describe in a precise way without being
too detailed, but I'll see what I can come up with.

> Maybe it should also point out the consequences of binding escape
> sequences used as prefix for common function keys, i.e. `ESC [' and
> `ESC O'.  One could even consider to unable colliding bindings to
> emphasize that you have to choose, rather than 'bugging out' silently.

Can you expand on what you mean by "unable colliding bindings".  Do you
mean that in your test case, Emacs should signal an error or at issue
a warning, after seeing the M-O?

Note that such colliding bindings already exist in the default config:
"emacs -nw -Q" and then ESC ESC ESC will run keyboard-escape-quit even
though the last ESC is a prefix of many function key escape sequences.

> BTW, the manual states that "The intent of key-translation-map is for users
> to map one character set to another, including ordinary characters normally
> bound to self-insert-command."
> Does this mean that key-translation-map is the recommended way to
> implement different layouts such as Colemak/Dvorak etc at the lowest
> possible level?

No, I don't think so.  But to tell you the truth, I do not know
understand the reasoning behind key-translation-map.

Also, I don't think there is a "recommended way" to provide such
a remapping within Emacs, currently.  Or rather than recommended way is
probably to do the remapping outside of Emacs.

> If yes, in the case of remapping `o' to `y' (Colemak example), M-O would
> never get through the input-decode-map filter and therefore never pop up to
> key-translation-map in order to produce M-Y. Is that correct?

Yes, that's right.

> /Stefan

> On 12 nov 2012, at 15:32, Stefan Monnier wrote:

>>> emacs -nw -Q
>>> (defun foo () (interactive) (message "foo"))
>>> (global-set-key "\M-O" 'foo)
>> [...]
>>> Arrow keys now inserts A,B,C,D in buffer.
>> [...]
>>> Arrow keys sends M-O A (up), M-O B (down) etc. These should be
>>> translated to function keys <up>, <down> etc in input-decode-map
>>> before any key lookup.  Somehow, the binding of M-O in the global
>>> keymap takes precedence.
>> 
>> Indeed, thanks for bringing this up.  The cause for the bug is that the
>> rule for what goes first and what goes next is here ignored because we
>> stop waiting for more input as soon as a real binding is found
>> (i.e. right after M-O).
>> 
>> The predicate that decides when to stop waiting for more input would
>> need to say say "don't stop if we're in the middle of a potential
>> input-decode-map rewrite".  But then if you want to run `foo', you'd
>> have the problem that after hitting M-O, Emacs will not actually run
>> `foo' but will sit there waiting to see if the next key is one of
>> [ABCD].
>> 
>> The only way to get our cake and eat it too would be to use some kind of
>> timeout, which we have resisted so far.
>> 
>> 
>> Stefan

bug#12868: global keymap preceeds input-decode-map

[Stefan Guath]

On 12 nov 2012, at 18:03, Stefan Monnier wrote:

>> I understand, and agree that a timeout is a bad idea.  This should be
>> reflected in the manual though, just as it is for local-function-key-map
>> ("Entries in local-function-key-map are ignored if they conflict with
>> bindings made in the minor mode, local, or global keymaps").
> 
> I find it difficult to describe in a precise way without being
> too detailed, but I'll see what I can come up with.

The input-decode-map section could just have the exact same sentence as local-function-key-map has, i.e. "Entries in input-decode-map are ignored if they conflict with bindings made in the minor mode, local, or global keymaps". Since local-function-key-map has that sentence, and input-decode-map does not, one assume that they work differently. Which is not the case.

The key-translation-map section is plain out wrong: "Just like input-decode-map, but unlike local-function-key-map, this keymap is applied regardless of whether the input key-sequence has a normal binding." No, input-decode-map nor key-translation-map has precedence over normal bindings - it's just the other way around.

>> Maybe it should also point out the consequences of binding escape
>> sequences used as prefix for common function keys, i.e. `ESC [' and
>> `ESC O'.  One could even consider to unable colliding bindings to
>> emphasize that you have to choose, rather than 'bugging out' silently.
> 
> Can you expand on what you mean by "unable colliding bindings".  Do you
> mean that in your test case, Emacs should signal an error or at issue
> a warning, after seeing the M-O?
> 
> Note that such colliding bindings already exist in the default config:
> "emacs -nw -Q" and then ESC ESC ESC will run keyboard-escape-quit even
> though the last ESC is a prefix of many function key escape sequences.

Yes, that was sort of what I meant, but I agree it was a bad idea. But the manual could at least conclude the entire section 22.14 with a reminder that since normal bindings do have precedence over all three of input-decode-map, local-function-key-map and key-translation-map, it's a bad idea to bind anything that collides with them such as `ESC O' or `ESC ['. People don't have to understand why - just prevent them from binding M-O and M-[.

>> BTW, the manual states that "The intent of key-translation-map is for users
>> to map one character set to another, including ordinary characters normally
>> bound to self-insert-command."
>> Does this mean that key-translation-map is the recommended way to
>> implement different layouts such as Colemak/Dvorak etc at the lowest
>> possible level?
> 
> No, I don't think so.  But to tell you the truth, I do not know
> understand the reasoning behind key-translation-map.

Me neither... It's confusing...

> Also, I don't think there is a "recommended way" to provide such
> a remapping within Emacs, currently.  Or rather than recommended way is
> probably to do the remapping outside of Emacs.

Thanks! I've struggled with this and always thought that I missed something obvious.

>> If yes, in the case of remapping `o' to `y' (Colemak example), M-O would
>> never get through the input-decode-map filter and therefore never pop up to
>> key-translation-map in order to produce M-Y. Is that correct?
> 
> Yes, that's right.

Great - I thought I'd misunderstood something fundamental. Thanks for all the clarifications!

bug#12868: global keymap preceeds input-decode-map

[Stefan Monnier]

> The input-decode-map section could just have the exact same sentence as
> local-function-key-map has, i.e. "Entries in input-decode-map are ignored if
> they conflict with bindings made in the minor mode, local, or global
> keymaps". Since local-function-key-map has that sentence, and
> input-decode-map does not, one assume that they work differently. Which is
> not the case.

But they do work differently: If you have a global-map binding of
"M-O A", the input-decode-map remapping will take precedence, whereas
a function-key-map remapping would be ignored.

It's a tricky business.

> Yes, that was sort of what I meant, but I agree it was a bad idea.

I don't think it's a bad idea, but it seems difficult to come up with
a way to detect actual problematic cases and brings them to the
user's attention (without being annoying, that is).

The best moment to detect those problem is when the global map is
modified, but the functions that are involved are the same as for
input-decode-map ;-)

> But the manual could at least conclude the entire section 22.14 with
> a reminder that since normal bindings do have precedence over all
> three of input-decode-map, local-function-key-map and
> key-translation-map, it's a bad idea to bind anything that collides
> with them such as `ESC O' or `ESC ['. People don't have to understand
> why - just prevent them from binding M-O and M-[.

Sounds right.  I'll see about doing that.


        Stefan

bug#12868: global keymap preceeds input-decode-map

[Stefan Monnier]

I've installed the patch below to try and better explain what's going on
and warn about possible problems.


        Stefan


=== modified file 'doc/lispref/ChangeLog'
--- doc/lispref/ChangeLog	2012-11-15 05:25:05 +0000
+++ doc/lispref/ChangeLog	2012-11-15 14:15:36 +0000
@@ -1,3 +1,8 @@
+2012-11-15  Stefan Monnier  <monnier@iro.umontreal.ca>
+
+	* keymaps.texi (Translation Keymaps): Add a subsection "Interaction
+	with normal keymaps".
+
 2012-11-15  Dmitry Antipov  <dmantipov@yandex.ru>
 
 	* internals.texi (Garbage Collection): Update descriptions

=== modified file 'doc/lispref/keymaps.texi'
--- doc/lispref/keymaps.texi	2012-10-28 14:56:51 +0000
+++ doc/lispref/keymaps.texi	2012-11-15 14:12:01 +0000
@@ -1540,14 +1540,11 @@
 being read, as it is read, against @code{input-decode-map}, then
 @code{local-function-key-map}, and then against @code{key-translation-map}.
 
-@defvar input-decode-map
-This variable holds a keymap that describes the character sequences sent
-by function keys on an ordinary character terminal.  This keymap has the
-same structure as other keymaps, but is used differently: it specifies
-translations to make while reading key sequences, rather than bindings
-for key sequences.
+These keymaps have the same structure as other keymaps, but they are used
+differently: they specify translations to make while reading key sequences,
+rather than bindings for key sequences.
 
-If @code{input-decode-map} ``binds'' a key sequence @var{k} to a vector
+If one of these keymaps ``binds'' a key sequence @var{k} to a vector
 @var{v}, then when @var{k} appears as a subsequence @emph{anywhere} in a
 key sequence, it is replaced with the events in @var{v}.
 
@@ -1562,6 +1559,10 @@
 this back into @kbd{C-c @key{PF1}}, which it returns as the vector
 @code{[?\C-c pf1]}.
 
+@defvar input-decode-map
+This variable holds a keymap that describes the character sequences sent
+by function keys on an ordinary character terminal.
+
 The value of @code{input-decode-map} is usually set up automatically
 according to the terminal's Terminfo or Termcap entry, but sometimes
 those need help from terminal-specific Lisp files.  Emacs comes with
@@ -1636,8 +1637,6 @@
   (let ((symbol (if (symbolp e) e (car e))))
     (setq symbol (intern (concat string
                                  (symbol-name symbol))))
-@end group
-@group
     (if (symbolp e)
         symbol
       (cons symbol (cdr e)))))
@@ -1647,10 +1646,30 @@
 @end example
 
   If you have enabled keyboard character set decoding using
-@code{set-keyboard-coding-system}, decoding is done after the
-translations listed above.  @xref{Terminal I/O Encoding}.  However, in
-future Emacs versions, character set decoding may be done at an
-earlier stage.
+@code{set-keyboard-coding-system}, decoding is done before the
+translations listed above.  @xref{Terminal I/O Encoding}.
+
+@subsection Interaction with normal keymaps
+
+The end of a key sequence is detected when that key sequence either is bound
+to a command, or when Emacs determines that no additional event can lead
+to a sequence that is bound to a command.
+
+This means that, while @code{input-decode-map} and @code{key-translation-map}
+apply regardless of whether the original key sequence would have a binding, the
+presence of such a binding can still prevent translation from taking place.
+For example, let us return to our VT100 example above and add a binding for
+@kbd{C-c @key{ESC}} to the global map; now when the user hits @kbd{C-c
+@key{PF1}} Emacs will fail to decode @kbd{C-c @key{ESC} O P} into @kbd{C-c
+@key{PF1}} because it will stop reading keys right after @kbd{C-x @key{ESC}},
+leaving @kbd{O P} for later.  This is in case the user really hit @kbd{C-c
+@key{ESC}}, in which case Emacs should not sit there waiting for the next key
+to decide whether the user really pressed @kbd{@key{ESC}} or @kbd{@key{PF1}}.
+
+For that reason, it is better to avoid binding commands to key sequences where
+the end of the key sequence is a prefix of a key translation.  The main such
+problematic suffixes/prefixes are @kbd{@key{ESC}}, @kbd{M-O} (which is really
+@kbd{@key{ESC} O}) and @kbd{M-[} (which is really @kbd{@key{ESC} [}).
 
 @node Key Binding Commands
 @section Commands for Binding Keys