bug#51253: 28.0.60; Meta keys broken when viper is active

[John Cummings <john@rootabega.net>]

This is a break introduced in 2020 after the 27.2 release.

emacs -Q recipe:

  (setq viper-inhibit-startup-message 't)
  (setq viper-expert-level '5)
  (setq viper-mode t)
  (require 'viper)
  ;only if/when testing this variant
  ;default is t
  ;(setq viper-no-multiple-ESC nil);

  In vi state or vi insert state, press M-x, M-s, M-:, etc.
  In terminal sessions, it will be like you pressed ESC and x
  separately and slowly.
  In graphical sessions, you will see a message like
  "M-x is undefined".


SUMMARY
This is is the latest wrinkle in a series of changes and bugs related
to viper. Let me know if you want me to update a previous bug report
instead of having this new one. I also have not included the
report-bug diag info for the Emacs versions and builds I tested with,
since this is pretty clearly fundamental to Emacs key handling and
viper functionality.  More detail is below, but I'll try to summarize
it very briefly first: Keys like M-x, M-:, etc., are broken in viper
vi/insert state since bug
https://debbugs.gnu.org/cgi/bugreport.cgi?bug=18182 was fixed in
5d522b430bd5ecfb8f082906cd634883dbb68f3e. This changed viper-ESC-key
from <escape> to ESC, which resulted in a direct key binding of (27
. 'viper-intercept-ESC-key) while in the viper vi/insert states.

In a terminal, the ESC and x events encoding M-x are processed like
two separate key sequences. This is because that direct ESC binding
makes viper-intercept-ESC-key consume the ESC event greedily, and then
the x is processed after that.

On a graphical display, the direct ESC binding prevents Emacs from
finding ESC prefix mappings that provide bindings for M- sequences
when it gets events like #x8000078 for M-x.

Since the latest change that causes this behavior was meant to make
ESC, aka ^[, work to exit vi insert state on graphical displays (as
Eli pointed out, it already worked on terminals at the time), would an
acceptable compromise in the short term be to let ^[ remain broken for
this purpose so that M- keys may work? In that bug, the reporter was
OK with the workaround Stefan suggested:
  (define-key input-decode-map [?\e] [escape])
Maybe that could also be added to all graphical sessions?



HISTORY
Rough timeline of changes that contribute to this viper behavior:

f5e1b6804dc2307983e4c55d4d6530549ddccbb7
Emacs ~24, Feb 2013
This is the starting point, where everything seemed to work in harmony.


git: 99d0d6dc23f0fd2ee6d64f0f18a33f2b791c642d
bzr?: 11521f1228e447bb68ff7a48a30148b99323c04e
Emacs ~24, Feb 2013
Changes were made to keymap handling functions in C. Prior to this
change, Even if there was direct ESC keybinding, Emacs could still
properly find the ESC prefix mappings that define M- bindings. i.e.,
you could map a command to ESC and use M- keys at the same time. After
this commit, the direct ESC binding meant that M- keys would be
undefined in graphical Emacs. So graphical Emacs could not use M-x in
viper vi/insert state, instead seeing 'M-x is undefined'. Terminal
emacs still worked OK, since that ESC binding was the viper ESC key
handler, and it could correctly identify ESC-as-meta event sequences
from their timing.

A recipe to see this specific change in behavior in graphical sessions:
   (local-set-key [27] (lambda ()
       (interactive) (insert " command ")))
Prior to this change, you could run this and still be able to run M-x.
After this change, you will get "M-x is undefined"


https://debbugs.gnu.org/cgi/bugreport.cgi?bug=13793
24.3.50; M-x broken in viper and X
Feb 2013
Frank Fischer noticed this problem and reported it, and provided some
analysis that helped me confirm the change to the keymap behavior I
mentioned in the previous paragraph. The fix for this bug was the next
change:


f1e6674bb32058c7ef683d5f8f8ac67f99e96dd2
Emacs ~24, Jul 2013
The key element of this change was to change viper-ESC-key from ESC to
<escape>. Since the (27 . 'intercept-viper-ESC-key) mapping changed to
(escape . 'intercept-viper-ESC-key), Graphical Emacs was no longer
blocked from finding the ESC prefix mappings for M- keys.  But this is
where ^[ stopped working to exit vi insert state in graphical
Emacs. Since viper--tty-ESC-filter doesn't run in graphical sessions,
and the viper mappings were now on <escape>, ^[ was just treated as a
pending ESC- prefix. In termainal sessions, viper--tty-ESC-filter
translates the ^[ to an <escape> after the timeout, and viper handles
it as designed.


https://debbugs.gnu.org/cgi/bugreport.cgi?bug=18182 reported
(to be fixed later)
24.3.92; C-[ does not work as ESC in viper-mode
iquiw noticed this break and reported it


Emacs 27.2
This release behaves the same as the previous Emacs version in this
timeline.


5d522b430bd5ecfb8f082906cd634883dbb68f3e
Emacs 28.0.50
Fixes bug#18182 by reverting a part of
f1e6674bb32058c7ef683d5f8f8ac67f99e96dd2: viper-ESC-key gets set back
to ESC, which puts the direct ESC mapping for viper-intercept-ESC-key
back in the viper minor maps.  This is where the current broken
behavior comes from with keys like M-x.  This is the first time we see
this bug in terminal sessions (AFAIK).  (I believe this is because the
changes in f1e6674 changed the points at which manual ESC events were
distinguished from ESC-as-meta events, from the original
"envelop-style" to the newer style.) In the terminal, ESC is processed
immediately by viper-intercept-ESC-key, which either exits insert
state or does nothing if not in insert state, and then the x event is
processed.  (There is one exception: when viper-no-multiple-ESC is nil
and you are in vi state, the viper-ESC function does successfully
interpret ESC x in Emacs-style as an M-x.  This appears to be the only
case when that happens.) In graphical sessions, the direct ESC binding
means M-x is undefined like it was in
99d0d6dc23f0fd2ee6d64f0f18a33f2b791c642d



ANALYSIS
It seems like the behavior in v24/git f1e6674 and Emacs 27.2 was the
most stable in recent history. As suggested earlier, since the only
viper bug (that I know of) was that ^[ wouldn't exit insert state in
graphical sessions, would tolerating that bug be an acceptable
worst-case scenario for now, as opposed to all M- keys being broken in
viper terminal/graphical?

And if you do want to fix that behavior, would it be as simple as the
workaround Stefan offered:
  (define-key input-decode-map [?\e] [escape])
for all graphical sessions?

It doesn't seem like there are many other options, since having a
binding for ^[ will break M- keys. But on that note, as Frank Fischer
wondered, wouldn't fixing that original 2013 change to the keymap C
code be the best fix? Since ESC keybindings conflict with M-
keybindings in the keymap formats, it seems right that graphical Emacs
would have had a way to process them separately. Was that behavior
deliberately removed, or could it return some day?  I'll note that, in
the timeline I looked at here, it's only graphical Emacs that ever had
the inherent ability to resolve M- bindings even when there were
direct ESC bindings.  Terminal Emacs would always invoke whatever the
ESC binding was, which is why the viper interceptors had to do the
timeout checks. I doubt that would change.