bug#66765: 30.0.50; Building emacs with xinput2 breaks receiving XSendEvent events

bug#66765: 30.0.50; Building emacs with xinput2 breaks receiving XSendEvent events

[Ami Fischman]

1. Run: `emacs -Q` on an X11 display for a binary that's been built
with (the default) `--with-xinput2`
2. From another program, trigger an XSendEvent targeting the above
emacs' window sending a key event.
Expected:
3. The sent key event is seen by emacs.
Actual:
4. Emacs doesn't seem to receive the sent key event.

Simplest way to execute step 2 above is:
$ git clone http://github.com/epitron/xse.git
$ cd xse && ./configure  && xmkmf && make depend && make
$ xwininfo
(click the emacs window and note its reported "Window id" for the next command)
$ ./xse -win <EMACS_WINDOW_ID> -Debug '<Key>l'

When emacs is built with `--without-xinput2` the issue is gone, and
sent events are received properly.
Issue is not present in 27.1 or 28.1, but is present in 29.1 and HEAD.
Confirmed with a git bisect that the issue showed up when xinput2 was
enabled by default in
https://git.savannah.gnu.org/cgit/emacs.git/commit/?id=0105a4ddb8a58146f3fc71c265e57291c873af0b

I stumbled across this issue when upgrading from 27.1 to 29.1 and
noticing that ratpoison can no longer send its escape key (in my case,
this is C-a) to emacs, even though it has no problems sending the same
key to every other app on my system.

In GNU Emacs 30.0.50 (build 1, x86_64-pc-linux-gnu, GTK+ Version
 3.24.33, cairo version 1.16.0) of 2023-10-25 built on <ELIDED>
Repository revision: 70e25298f160ffec62e363056904b6238a87d4e0
Repository branch: master
Windowing system distributor 'The X.Org Foundation', version 11.0.12101004
System Description: Ubuntu 22.04.3 LTS

Configured using:
 'configure --prefix=<ELIDED> --enable-libsystemd
 --with-pop=yes
 --enable-locallisppath=/etc/emacs:/usr/local/share/emacs/29.1/site-lisp:/usr/local/share/emacs/site-lisp:/usr/share/emacs/29.1/site-lisp:/usr/share/emacs/site-lisp
 --with-sound=alsa --without-gconf --with-mailutils --with-cairo
 --with-x=yes --with-x-toolkit=gtk3 --with-toolkit-scroll-bars
 --with-imagemagick --with-json --with-tree-sitter --with-xwidgets
 'CFLAGS=-g -O2 -fstack-protector-strong -w' 'CPPFLAGS=-Wdate-time
 -D_FORTIFY_SOURCE=2' 'LDFLAGS=-Wl,-Bsymbolic-functions -Wl,-z,relro''

Configured features:
ACL CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS GPM GSETTINGS HARFBUZZ
IMAGEMAGICK JPEG JSON LCMS2 LIBOTF LIBSELINUX LIBSYSTEMD LIBXML2
M17N_FLT MODULES NOTIFY INOTIFY PDUMPER PNG RSVG SECCOMP SOUND SQLITE3
THREADS TIFF TOOLKIT_SCROLL_BARS TREE_SITTER X11 XDBE XIM XINPUT2 XPM
XWIDGETS GTK3 ZLIB

Important settings:
  value of $LC_ALL:
  value of $LANG: en_US.UTF-8
  value of $XMODIFIERS: @im=ibus
  locale-coding-system: utf-8-unix

Major mode: Lisp Interaction

Minor modes in effect:
  tooltip-mode: t
  global-eldoc-mode: t
  eldoc-mode: t
  show-paren-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  minibuffer-regexp-mode: t
  line-number-mode: t
  indent-tabs-mode: t
  transient-mark-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message mailcap yank-media puny dired
dired-loaddefs rfc822 mml mml-sec password-cache epa derived epg rfc6068
epg-config gnus-util text-property-search time-date subr-x mm-decode
mm-bodies mm-encode mail-parse rfc2231 mailabbrev gmm-utils mailheader
cl-loaddefs cl-lib sendmail rfc2047 rfc2045 ietf-drums mm-util
mail-prsvr mail-utils rmc iso-transl tooltip cconv eldoc paren electric
uniquify ediff-hook vc-hooks lisp-float-type elisp-mode mwheel
term/x-win x-win term/common-win x-dnd touch-screen tool-bar dnd fontset
image regexp-opt fringe tabulated-list replace newcomment text-mode
lisp-mode prog-mode register page tab-bar menu-bar rfn-eshadow isearch
easymenu timer select scroll-bar mouse jit-lock font-lock syntax
font-core term/tty-colors frame minibuffer nadvice seq simple cl-generic
indonesian philippine cham georgian utf-8-lang misc-lang vietnamese
tibetan thai tai-viet lao korean japanese eucjp-ms cp51932 hebrew greek
romanian slovak czech european ethiopic indian cyrillic chinese
composite emoji-zwj charscript charprop case-table epa-hook
jka-cmpr-hook help abbrev obarray oclosure cl-preloaded button loaddefs
theme-loaddefs faces cus-face macroexp files window text-properties
overlay sha1 md5 base64 format env code-pages mule custom widget keymap
hashtable-print-readable backquote threads xwidget-internal dbusbind
inotify lcms2 dynamic-setting system-font-setting font-render-setting
cairo gtk x-toolkit xinput2 x multi-tty move-toolbar
make-network-process emacs)

Memory information:
((conses 16 37691 9512) (symbols 48 5216 0) (strings 32 13286 1532)
 (string-bytes 1 375218) (vectors 16 10300)
 (vector-slots 8 156562 17668) (floats 8 38 21) (intervals 56 259 0)
 (buffers 992 10))

bug#66765: 30.0.50; Building emacs with xinput2 breaks receiving XSendEvent events

[Po Lu via Bug reports for GNU Emacs, the Swiss army knife of text editors]

tags 66765 + notabug wontfix
thanks

Ami Fischman <ami@fischman.org> writes:

> 1. Run: `emacs -Q` on an X11 display for a binary that's been built
> with (the default) `--with-xinput2`
> 2. From another program, trigger an XSendEvent targeting the above
> emacs' window sending a key event.
> Expected:
> 3. The sent key event is seen by emacs.
> Actual:
> 4. Emacs doesn't seem to receive the sent key event.
>
> Simplest way to execute step 2 above is:
> $ git clone http://github.com/epitron/xse.git
> $ cd xse && ./configure  && xmkmf && make depend && make
> $ xwininfo
> (click the emacs window and note its reported "Window id" for the next command)
> $ ./xse -win <EMACS_WINDOW_ID> -Debug '<Key>l'
>
> When emacs is built with `--without-xinput2` the issue is gone, and
> sent events are received properly.
> Issue is not present in 27.1 or 28.1, but is present in 29.1 and HEAD.
> Confirmed with a git bisect that the issue showed up when xinput2 was
> enabled by default in
> https://git.savannah.gnu.org/cgit/emacs.git/commit/?id=0105a4ddb8a58146f3fc71c265e57291c873af0b
>
> I stumbled across this issue when upgrading from 27.1 to 29.1 and
> noticing that ratpoison can no longer send its escape key (in my case,
> this is C-a) to emacs, even though it has no problems sending the same
> key to every other app on my system.

When the X input extension is in use, both Emacs and the X server ignore
core events.  It's also impossible to send extension events, so this
extension renders external event delivery effectively impossible.

This isn't a bug we can fix, sorry; but the XTEST extension is capable
of simulating real key presses, so I suggest whatever tools send Emacs
events be rewritten to make use of that extension.

The same issue can be observed in programs written to use version 3 or 4
of the GTK toolkit.

bug#66765: 30.0.50; Building emacs with xinput2 breaks receiving XSendEvent events

[Eli Zaretskii]

> Cc: 66765-done@debbugs.gnu.org
> Date: Fri, 27 Oct 2023 08:32:44 +0800
> From:  Po Lu via "Bug reports for GNU Emacs,
>  the Swiss army knife of text editors" <bug-gnu-emacs@gnu.org>
> 
> When the X input extension is in use, both Emacs and the X server ignore
> core events.  It's also impossible to send extension events, so this
> extension renders external event delivery effectively impossible.
> 
> This isn't a bug we can fix, sorry; but the XTEST extension is capable
> of simulating real key presses, so I suggest whatever tools send Emacs
> events be rewritten to make use of that extension.
> 
> The same issue can be observed in programs written to use version 3 or 4
> of the GTK toolkit.

Can you please add something to etc/PROBLEMS about this issue?

How widely is this mechanism used in the wild?  If it is used widely
enough, perhaps we should tell people not to build with XInput2 if
they have vested interests in those methods?

Also, maybe submitting bug reports to the respective developers would
cause them to improve the situation?

bug#66765: 30.0.50; Building emacs with xinput2 breaks receiving XSendEvent events

[Po Lu via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Eli Zaretskii <eliz@gnu.org> writes:

> Can you please add something to etc/PROBLEMS about this issue?
>
> How widely is this mechanism used in the wild?  If it is used widely
> enough, perhaps we should tell people not to build with XInput2 if
> they have vested interests in those methods?

XSendEvent is seldom used to send input events these days, for this and
many other reasons, and teaching X programming is outside the purview of
PROBLEMS.  Properly written programs either insert events into the
server by means of the XTEST extension, or through OS interfaces that
precede the X server in the input event delivery process.

One such reason is that it's not possible for an X client to reliably
establish the window an event should be delivered to without grabbing
the server and reading every event mask in between the root window and
the window beneath the mouse pointer, which is slow, requires a server
grab, and tedious.

> Also, maybe submitting bug reports to the respective developers would
> cause them to improve the situation?

Oh no, this is a studious design choice, and one I agree with; it's not
possible to transform core events into input extension ones, because too
much information is absent from the former.

bug#66765: 30.0.50; Building emacs with xinput2 breaks receiving XSendEvent events

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: 66765@debbugs.gnu.org,  ami@fischman.org
> Date: Fri, 27 Oct 2023 15:03:21 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Can you please add something to etc/PROBLEMS about this issue?
> >
> > How widely is this mechanism used in the wild?  If it is used widely
> > enough, perhaps we should tell people not to build with XInput2 if
> > they have vested interests in those methods?
> 
> XSendEvent is seldom used to send input events these days, for this and
> many other reasons, and teaching X programming is outside the purview of
> PROBLEMS.  Properly written programs either insert events into the
> server by means of the XTEST extension, or through OS interfaces that
> precede the X server in the input event delivery process.

The OP talked specifically about ratpoison.  There may be other
programs that use this method.  I'm asking whether we know which ones
do that, not whether it's correct or recommended.

> > Also, maybe submitting bug reports to the respective developers would
> > cause them to improve the situation?
> 
> Oh no, this is a studious design choice, and one I agree with; it's not
> possible to transform core events into input extension ones, because too
> much information is absent from the former.

You are saying that ratpoison cannot be changed to use mechanisms
compatible with XInput2?  If so, it would mean ratpoison will never be
compatible with Emacs build with XInput2.  I thought that maybe
telling ratpoison developers about this will encourage them to make
ratpoison compatible with XInput2.

bug#66765: 30.0.50; Building emacs with xinput2 breaks receiving XSendEvent events

[Po Lu via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Eli Zaretskii <eliz@gnu.org> writes:

> The OP talked specifically about ratpoison.

ratpoison doesn't send events to Emacs itself.  The OP has configured it
to call "xse", which is a command-line wrapper around the proscribed
SendEvent request.  It is easily replaced with "xdotool" or suchlike.

> There may be other programs that use this method.  I'm asking whether
> we know which ones do that, not whether it's correct or recommended.

Oh, I don't.

bug#66765: 30.0.50; Building emacs with xinput2 breaks receiving XSendEvent events

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: 66765@debbugs.gnu.org,  ami@fischman.org
> Date: Fri, 27 Oct 2023 15:38:11 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > The OP talked specifically about ratpoison.
> 
> ratpoison doesn't send events to Emacs itself.  The OP has configured it
> to call "xse", which is a command-line wrapper around the proscribed
> SendEvent request.  It is easily replaced with "xdotool" or suchlike.

I guess it would be a good idea to have this information in PROBLEMS,
then.

bug#66765: 30.0.50; Building emacs with xinput2 breaks receiving XSendEvent events

[Po Lu via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Eli Zaretskii <eliz@gnu.org> writes:

> I guess it would be a good idea to have this information in PROBLEMS,
> then.

But that's tantamount to documenting the do's and don'ts of X
development, since "xse" is nothing but a wrapper around an X request.
It's a very slippery slope, though it's ultimately your call.

bug#66765: 30.0.50; Building emacs with xinput2 breaks receiving XSendEvent events

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: 66765@debbugs.gnu.org,  ami@fischman.org
> Date: Fri, 27 Oct 2023 18:34:56 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > I guess it would be a good idea to have this information in PROBLEMS,
> > then.
> 
> But that's tantamount to documenting the do's and don'ts of X
> development, since "xse" is nothing but a wrapper around an X request.
> It's a very slippery slope, though it's ultimately your call.

I'd rather we had this information in PROBLEMS, both for those who
consult that file and for us, when we need to help others to cope with
these issues.

In general, PROBLEMS should mention any problems with Emacs that
cannot be fixed in Emacs, preferably with workarounds, if we have
them.

bug#66765: 30.0.50; Building emacs with xinput2 breaks receiving XSendEvent events

[Dmitry Gutov]

On 27/10/2023 13:34, Po Lu via Bug reports for GNU Emacs, the Swiss army 
knife of text editors wrote:
> Eli Zaretskii<eliz@gnu.org>  writes:
> 
>> I guess it would be a good idea to have this information in PROBLEMS,
>> then.
> But that's tantamount to documenting the do's and don'ts of X
> development, since "xse" is nothing but a wrapper around an X request.
> It's a very slippery slope, though it's ultimately your call.

We could say something to the effect of "to control Emacs externally, 
xse is not recommended, prefer xdotool".

bug#66765: 30.0.50; Building emacs with xinput2 breaks receiving XSendEvent events

[Eli Zaretskii]

> Cc: luangruo@yahoo.com, ami@fischman.org, 66765@debbugs.gnu.org
> Date: Fri, 27 Oct 2023 09:04:00 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> 
> > Cc: 66765-done@debbugs.gnu.org
> > Date: Fri, 27 Oct 2023 08:32:44 +0800
> > From:  Po Lu via "Bug reports for GNU Emacs,
> >  the Swiss army knife of text editors" <bug-gnu-emacs@gnu.org>
> > 
> > When the X input extension is in use, both Emacs and the X server ignore
> > core events.  It's also impossible to send extension events, so this
> > extension renders external event delivery effectively impossible.
> > 
> > This isn't a bug we can fix, sorry; but the XTEST extension is capable
> > of simulating real key presses, so I suggest whatever tools send Emacs
> > events be rewritten to make use of that extension.
> > 
> > The same issue can be observed in programs written to use version 3 or 4
> > of the GTK toolkit.
> 
> Can you please add something to etc/PROBLEMS about this issue?

Ping!  I'd really love to have something in v29.2's PROBLEMS about
this.

Thanks.

bug#66765: 30.0.50; Building emacs with xinput2 breaks receiving XSendEvent events

[Po Lu via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Eli Zaretskii <eliz@gnu.org> writes:

>> Cc: luangruo@yahoo.com, ami@fischman.org, 66765@debbugs.gnu.org
>> Date: Fri, 27 Oct 2023 09:04:00 +0300
>> From: Eli Zaretskii <eliz@gnu.org>
>> 
>> > Cc: 66765-done@debbugs.gnu.org
>> > Date: Fri, 27 Oct 2023 08:32:44 +0800
>> > From:  Po Lu via "Bug reports for GNU Emacs,
>> >  the Swiss army knife of text editors" <bug-gnu-emacs@gnu.org>
>> > 
>> > When the X input extension is in use, both Emacs and the X server ignore
>> > core events.  It's also impossible to send extension events, so this
>> > extension renders external event delivery effectively impossible.
>> > 
>> > This isn't a bug we can fix, sorry; but the XTEST extension is capable
>> > of simulating real key presses, so I suggest whatever tools send Emacs
>> > events be rewritten to make use of that extension.
>> > 
>> > The same issue can be observed in programs written to use version 3 or 4
>> > of the GTK toolkit.
>> 
>> Can you please add something to etc/PROBLEMS about this issue?
>
> Ping!  I'd really love to have something in v29.2's PROBLEMS about
> this.
>
> Thanks.

Would you please install this under the heading "General X problems"?
TIA.

*** Programs which use XSendEvent cannot send input events to Emacs.

Emacs built to use the X Input Extension cannot receive core input
events sent through the SendEvent server request, since these events
intercepted by the X server when sent to input extension clients.

For such programs to function again, Emacs must be run on an X server
where the input extension is disabled, or alternatively configured with
the "--without-xinput2" option.

bug#66765: 30.0.50; Building emacs with xinput2 breaks receiving XSendEvent events

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: ami@fischman.org,  66765@debbugs.gnu.org
> Date: Sat, 04 Nov 2023 18:25:45 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Ping!  I'd really love to have something in v29.2's PROBLEMS about
> > this.
> >
> > Thanks.
> 
> Would you please install this under the heading "General X problems"?
> TIA.
> 
> *** Programs which use XSendEvent cannot send input events to Emacs.
> 
> Emacs built to use the X Input Extension cannot receive core input
> events sent through the SendEvent server request, since these events
> intercepted by the X server when sent to input extension clients.
> 
> For such programs to function again, Emacs must be run on an X server
> where the input extension is disabled, or alternatively configured with
> the "--without-xinput2" option.

Thanks, done, and closing the bug.

bug#66765: closed (Re: bug#66765: 30.0.50; Building emacs with xinput2 breaks receiving XSendEvent events)

[Ami Fischman]

Thanks for following up, all.

> > The OP talked specifically about ratpoison.
> ratpoison doesn't send events to Emacs itself.

This is incorrect. I only included the `xse` invocation to provide
readers of the bug who do not use ratpoison normally a way to
reproduce the issue. Ratpoison _does_ call XSendEvent directly, e.g.
at https://git.savannah.nongnu.org/cgit/ratpoison.git/tree/src/actions.c?h=v1.4.9#n1414

> It is easily replaced with "xdotool" or suchlike.

For the record, xdotool (version 3.20160805.1) fails similarly to xse
and ratpoison when I try to use it like this:
xdotool key --window 0x36000d4 ctrl+a
(window ID of HEAD emacs with xinput2 results in a no-op, window ID of
emacs without xinput2 correctly receives a C-a event)
I'm curious if you have an example invocation you can share that does
result in a C-a being injected to an emacs build with XInput2 support.

bug#66765: closed (Re: bug#66765: 30.0.50; Building emacs with xinput2 breaks receiving XSendEvent events)

[Po Lu via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Ami Fischman <ami@fischman.org> writes:

> Thanks for following up, all.
>
>> > The OP talked specifically about ratpoison.
>> ratpoison doesn't send events to Emacs itself.
>
> This is incorrect. I only included the `xse` invocation to provide
> readers of the bug who do not use ratpoison normally a way to
> reproduce the issue. Ratpoison _does_ call XSendEvent directly, e.g.
> at https://git.savannah.nongnu.org/cgit/ratpoison.git/tree/src/actions.c?h=v1.4.9#n1414
>
>> It is easily replaced with "xdotool" or suchlike.
>
> For the record, xdotool (version 3.20160805.1) fails similarly to xse
> and ratpoison when I try to use it like this:
> xdotool key --window 0x36000d4 ctrl+a
> (window ID of HEAD emacs with xinput2 results in a no-op, window ID of
> emacs without xinput2 correctly receives a C-a event)
> I'm curious if you have an example invocation you can share that does
> result in a C-a being injected to an emacs build with XInput2 support.

What happens when you type:

  xdotool key ctrl+a

with Emacs selected?

bug#66765: closed (Re: bug#66765: 30.0.50; Building emacs with xinput2 breaks receiving XSendEvent events)

[Ami Fischman]

> What happens when you type:
>   xdotool key ctrl+a
> with Emacs selected?

Ratpoison is triggered b/c ctrl+a is its hotkey, and it has higher
precedence than "current window".
If instead I do `xdotool key h` in an emacs *shell* buffer then the
next command line has an `h` already present, suggesting that emacs
did indeed receive that xdotool's event.

bug#66765: closed (Re: bug#66765: 30.0.50; Building emacs with xinput2 breaks receiving XSendEvent events)

[Ami Fischman]

> If instead I do `xdotool key h` in an emacs *shell* buffer then the
> next command line has an `h` already present, suggesting that emacs
> did indeed receive that xdotool's event.

I was unable to get xdotool to send to emacs from outside of emacs, so
went digging into xdotool, and apparently it's because it only uses
XTEST when sending to the current window, otherwise it uses XSendEvent
directly: https://github.com/jordansissel/xdotool/blob/7b63eb46631815fd4a10e1fd78a37213250d1193/xdo.c#L1557
Unless I misunderstood your earlier suggestion about how to use
xdotool, it doesn't seem like a usable alternative for sending events
to emacs from outside emacs (since emacs won't be the current window).

FWIW I'm looking at patching ratpoison to use XTEST for my own use,
though I doubt it'll be accepted upstream as development has mostly
stopped there.

Stepping back a bit from the specifics of Emacs and ratpoison, naively
it seems unfortunate that X11 programs that have been working for
decades should need to be modified to use similar-but-different APIs.
Do you know whether the XInput2 people are aware of / at peace with
this limitation, or is this something worth bringing up with them?

bug#66765: closed (Re: bug#66765: 30.0.50; Building emacs with xinput2 breaks receiving XSendEvent events)

[Po Lu via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Ami Fischman <ami@fischman.org> writes:

> Stepping back a bit from the specifics of Emacs and ratpoison, naively
> it seems unfortunate that X11 programs that have been working for
> decades should need to be modified to use similar-but-different APIs.
> Do you know whether the XInput2 people are aware of / at peace with
> this limitation, or is this something worth bringing up with them?

It's an immemorial and largely intentional design decision.  Core events
lack too much information for their transformation into extension events
to be possible.