bug#60077: 29.0.60; Is xterm modifyOtherKeys support broken?

bug#60077: 29.0.60; Is xterm modifyOtherKeys support broken?

[Len Trigg]

[-- Attachment #1: Type: text/plain, Size: 1584 bytes --]

I often run emacs in the terminal, and usually this is inside wezterm as my
terminal emulator. Recently wezterm has added support for the
modifyOtherKeys xterm extension, and since then emacs M-SPC has stopped
working in the terminal (instead it inserts "~32").

I can get M-SPC working again in wezterm by preventing emacs from enabling
modifyOtherKeys, either by setting xterm-extra-capabilities to nil or by
stubbing out xterm--init-modify-other-keys. Initially we thought that
this may be a problem with wezterm, but then I tried replicating with
xterm and exactly the same thing happens. This suggests to me that the
issue is with emacs rather than xterm or wezterm.

To replicate:

1) run: xterm -e emacs -nw -Q
2) in say *scratch* check whether M-SPC works or whether it inserts ~32
(for me it inserts ~32)

To replicate disabling modifyOtherKeys:

1) put the following into $HOME/test-dir/init.el
------
;; Dirty hack to not use xterm modifyOtherKeys feature
(defun my-disable-xterm--init-modify-other-keys (orig-fun &rest args))
(advice-add #'xterm--init-modify-other-keys :around
            #'my-disable-xterm--init-modify-other-keys)
------
2) run: xterm -e emacs -nw --init-dir=$HOME/test-dir
3) in say *scratch* check whether M-SPC works or whether it inserts ~32
(for me it works correctly)

This seems to demonstrate the out of the box behaviour being broken, but
I don't understand enough about terminals etc to know whether there is
some other factor at play.

Relevant link where this was discussed:
https://github.com/CyberShadow/term-keys/issues/14

Cheers,
Len.

[-- Attachment #2: Type: text/html, Size: 1904 bytes --]

bug#60077: 29.0.60; Is xterm modifyOtherKeys support broken?

[Eli Zaretskii]

> From: Len Trigg <lenbok@gmail.com>
> Date: Thu, 15 Dec 2022 10:48:35 +1300
> 
> I often run emacs in the terminal, and usually this is inside wezterm as my
> terminal emulator. Recently wezterm has added support for the
> modifyOtherKeys xterm extension, and since then emacs M-SPC has stopped
> working in the terminal (instead it inserts "~32").
> 
> I can get M-SPC working again in wezterm by preventing emacs from enabling
> modifyOtherKeys, either by setting xterm-extra-capabilities to nil or by
> stubbing out xterm--init-modify-other-keys. Initially we thought that
> this may be a problem with wezterm, but then I tried replicating with
> xterm and exactly the same thing happens. This suggests to me that the
> issue is with emacs rather than xterm or wezterm.
> 
> To replicate:
> 
> 1) run: xterm -e emacs -nw -Q
> 2) in say *scratch* check whether M-SPC works or whether it inserts ~32
> (for me it inserts ~32)
> 
> To replicate disabling modifyOtherKeys:
> 
> 1) put the following into $HOME/test-dir/init.el
> ------
> ;; Dirty hack to not use xterm modifyOtherKeys feature
> (defun my-disable-xterm--init-modify-other-keys (orig-fun &rest args))
> (advice-add #'xterm--init-modify-other-keys :around
>             #'my-disable-xterm--init-modify-other-keys)
> ------
> 2) run: xterm -e emacs -nw --init-dir=$HOME/test-dir
> 3) in say *scratch* check whether M-SPC works or whether it inserts ~32
> (for me it works correctly)
> 
> This seems to demonstrate the out of the box behaviour being broken, but
> I don't understand enough about terminals etc to know whether there is
> some other factor at play.
> 
> Relevant link where this was discussed:
> https://github.com/CyberShadow/term-keys/issues/14

I'm not sure I understand: it sounds like you are saying that M-SPC
doesn't work in xterm, either?  (Which version of xterm. btw?)  So the
question now becomes: how did it work before that change in wezterm
for you, and why did modifyOtherKeys feature broke it?

Which terminal file in lisp/term/ was/is Emacs loading at startup when
you use wezterm?

bug#60077: 29.0.60; Is xterm modifyOtherKeys support broken?

[Len Trigg]

[-- Attachment #1: Type: text/plain, Size: 1460 bytes --]

On Thu, 15 Dec 2022 at 19:41, Eli Zaretskii <eliz@gnu.org> wrote:

> I'm not sure I understand: it sounds like you are saying that M-SPC
> doesn't work in xterm, either?  (Which version of xterm. btw?)


Yes, I am saying M-SPC does not work in xterm (which reports its version as
XTerm(353)).


So the
> question now becomes: how did it work before that change in wezterm
> for you, and why did modifyOtherKeys feature broke it?
>

I don't think it worked before the change in wezterm (I haven't used xterm
for years) -- I really just tested with xterm to see whether the issue was
specific to wezterm, and was surprised to find xterm already broken.


Which terminal file in lisp/term/ was/is Emacs loading at startup when
> you use wezterm?
>

For both xterm and wezterm I have the $TERM variable set to "xterm-direct",
so they both use lisp/term/xterm.el (which is consistent with my hack of
xterm--init-modify-other-keys affecting the behaviour of both). My
hypothesis is that under both xterm and wezterm emacs is sending the
terminal initialization code for turning on modifyOtherKeys, but the older
version of wezterm just ignored it (and M-SPC worked). But now they have
added modifyOtherKeys support, it is behaving like xterm (i.e. broken) by
sending M-SPC with an encoding that emacs doesn't recognize. Do you think
it's just a matter of the dolist on line 466 of xterm.el needing additional
entries (I don't see one there for M-SPC)?

Cheers,
Len.

[-- Attachment #2: Type: text/html, Size: 2312 bytes --]

bug#60077: 29.0.60; Is xterm modifyOtherKeys support broken?

[Len Trigg]

[-- Attachment #1: Type: text/plain, Size: 704 bytes --]

On Thu, 15 Dec 2022 at 21:38, Len Trigg <lenbok@gmail.com> wrote:

> Do you think it's just a matter of the dolist on line 466 of xterm.el
> needing additional entries (I don't see one there for M-SPC)?
>

For example, I see there looks to be an entry for C-M-SPC:
                   (7 32 [?\C-\M-\s])
And my wild speculation from looking at the example table "Other
modified-key escapes" on
https://invisible-island.net/xterm/modified-keys-us-pc105.html is that
emacs would need an entry like:
                   (3 32 [?\M-\s])
to handle the modifyOtherKeys encoding of "\E[27;3;32~" for M-SPC, but also
that there are probably many many other sequences that aren't being handled.

Cheers,
Len.




>

[-- Attachment #2: Type: text/html, Size: 1544 bytes --]

bug#60077: 29.0.60; Is xterm modifyOtherKeys support broken?

[Eli Zaretskii]

> From: Len Trigg <lenbok@gmail.com>
> Date: Thu, 15 Dec 2022 21:38:57 +1300
> Cc: 60077@debbugs.gnu.org
> 
>  Which terminal file in lisp/term/ was/is Emacs loading at startup when
>  you use wezterm?
> 
> For both xterm and wezterm I have the $TERM variable set to "xterm-direct", so they both use
> lisp/term/xterm.el (which is consistent with my hack of xterm--init-modify-other-keys affecting the behaviour of
> both). My hypothesis is that under both xterm and wezterm emacs is sending the terminal initialization code
> for turning on modifyOtherKeys, but the older version of wezterm just ignored it (and M-SPC worked). But now
> they have added modifyOtherKeys support, it is behaving like xterm (i.e. broken) by sending M-SPC with an
> encoding that emacs doesn't recognize. Do you think it's just a matter of the dolist on line 466 of xterm.el
> needing additional entries (I don't see one there for M-SPC)?

I don't really know.  But one solution is to disable modifyOtherKeys
in your init file.  Or maybe you could dig deeper into what xterm.el
does with modifyOtherKeys support and tell why M-SPC fails.

bug#60077: 29.0.60; Is xterm modifyOtherKeys support broken?

[Eli Zaretskii]

> From: Len Trigg <lenbok@gmail.com>
> Date: Thu, 15 Dec 2022 21:57:00 +1300
> Cc: 60077@debbugs.gnu.org
> 
> On Thu, 15 Dec 2022 at 21:38, Len Trigg <lenbok@gmail.com> wrote:
> 
>  Do you think it's just a matter of the dolist on line 466 of xterm.el needing additional entries (I don't see
>  one there for M-SPC)?
> 
> For example, I see there looks to be an entry for C-M-SPC:
>                    (7 32 [?\C-\M-\s])
> And my wild speculation from looking at the example table "Other modified-key escapes" on
> https://invisible-island.net/xterm/modified-keys-us-pc105.html is that emacs would need an entry like:
>                    (3 32 [?\M-\s])
> to handle the modifyOtherKeys encoding of "\E[27;3;32~" for M-SPC,

If that fixes the problem, we could install it.

> but also that there are probably
> many many other sequences that aren't being handled.

There's nothing wrong with solving this one problem at a time, right?
Especially since we don't seem to have complaints about other
sequences.

bug#60077: 29.0.60; Is xterm modifyOtherKeys support broken?

[Len Trigg]

[-- Attachment #1: Type: text/plain, Size: 1186 bytes --]

On Thu, 15 Dec 2022 at 22:10, Eli Zaretskii <eliz@gnu.org> wrote:

> > And my wild speculation from looking at the example table "Other
> modified-key escapes" on
> > https://invisible-island.net/xterm/modified-keys-us-pc105.html is that
> emacs would need an entry like:
> >                    (3 32 [?\M-\s])
> > to handle the modifyOtherKeys encoding of "\E[27;3;32~" for M-SPC,
>
> If that fixes the problem, we could install it.
>

I tried this out and it does fix the issue for me. BTW, have you tried to
replicate the issue yourself?


There's nothing wrong with solving this one problem at a time, right?
> Especially since we don't seem to have complaints about other
> sequences.
>

I suppose so, I'm just a little wary of fixing like this without
understanding why it only seems to be an issue now or whether other guards
are needed. For example, the commit log for this area shows Dan Nicolaescu
adding more bindings in c0658ad46e4928ecdf6a14904073dc8a2fcfe862 and also
bumping the xterm version number that is checked against, so he is probably
knowledgeable about how this works. (I've tried to add him to the CC list
using the email address in that commit)

Cheers,
Len.

[-- Attachment #2: Type: text/html, Size: 1958 bytes --]

bug#60077: 29.0.60; Is xterm modifyOtherKeys support broken?

[Eli Zaretskii]

> From: Len Trigg <lenbok@gmail.com>
> Date: Fri, 16 Dec 2022 08:48:54 +1300
> Cc: 60077@debbugs.gnu.org, dann@ics.uci.edu
> 
> On Thu, 15 Dec 2022 at 22:10, Eli Zaretskii <eliz@gnu.org> wrote:
> 
>  > And my wild speculation from looking at the example table "Other modified-key escapes" on
>  > https://invisible-island.net/xterm/modified-keys-us-pc105.html is that emacs would need an entry like:
>  >                    (3 32 [?\M-\s])
>  > to handle the modifyOtherKeys encoding of "\E[27;3;32~" for M-SPC,
> 
>  If that fixes the problem, we could install it.
> 
> I tried this out and it does fix the issue for me.

OK, will install that soon.

> BTW, have you tried to replicate the issue yourself?

I can't: I don't have access to a system with such a terminal
emulator.

>  There's nothing wrong with solving this one problem at a time, right?
>  Especially since we don't seem to have complaints about other
>  sequences.
>  
> I suppose so, I'm just a little wary of fixing like this without understanding why it only seems to be an issue now
> or whether other guards are needed. For example, the commit log for this area shows Dan Nicolaescu adding
> more bindings in c0658ad46e4928ecdf6a14904073dc8a2fcfe862 and also bumping the xterm version
> number that is checked against, so he is probably knowledgeable about how this works. (I've tried to add him
> to the CC list using the email address in that commit)

Yes, thanks.  Dan, if you have time to look into this and advise,
please do.

bug#60077: 29.0.60; Is xterm modifyOtherKeys support broken?

[Eli Zaretskii]

> Cc: dann@ics.uci.edu, 60077@debbugs.gnu.org
> Date: Fri, 16 Dec 2022 08:52:30 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> 
> Yes, thanks.  Dan, if you have time to look into this and advise,
> please do.

And the message bounced from Dan's last known address, so I guess we
are on our own here.

bug#60077: 29.0.60; Is xterm modifyOtherKeys support broken?

[Eli Zaretskii]

> Cc: dann@ics.uci.edu, 60077@debbugs.gnu.org
> Date: Fri, 16 Dec 2022 08:52:30 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> 
> > From: Len Trigg <lenbok@gmail.com>
> > Date: Fri, 16 Dec 2022 08:48:54 +1300
> > Cc: 60077@debbugs.gnu.org, dann@ics.uci.edu
> > 
> > On Thu, 15 Dec 2022 at 22:10, Eli Zaretskii <eliz@gnu.org> wrote:
> > 
> >  > And my wild speculation from looking at the example table "Other modified-key escapes" on
> >  > https://invisible-island.net/xterm/modified-keys-us-pc105.html is that emacs would need an entry like:
> >  >                    (3 32 [?\M-\s])
> >  > to handle the modifyOtherKeys encoding of "\E[27;3;32~" for M-SPC,
> > 
> >  If that fixes the problem, we could install it.
> > 
> > I tried this out and it does fix the issue for me.
> 
> OK, will install that soon.

Now done on the emacs-29 branch, and closing the bug.

Thanks.