xwidget events

xwidget events

[Po Lu]

I've been experimenting with making xwidget-webkit on X11 usable as an
actual web browser.  So far I've figured out the annoying redraw issues
and the broken scroll optimization, but I'm still puzzling over event
handling.

I was able to borrow a Mac from a friend today for an hour or so, and I
used it to try the NS xwidget implementation.  It seems to handle mouse,
button and click events entirely separately from the rest of Emacs,
which I don't think is a very good idea either.  But that also means the
browser is actually useful, which is a good thing.

In the existing X11 xwidget implementation, only some events, such as
motion events are passed through (without being exposed to Lisp code
first), while with the improvements to xwidget redisplay in bug#51473,
no events are passed through to the xwidget at all.

So I think a better idea would be the ability to pass through individual
input events via Lisp code (for instance with a function named
`xwidget-do-event', which would accept a lispy event as input and then
send an equivalent event to the xwidget).  Lisp code could then take
input events in the xwidget-webkit, and then send them to the xwidget.

But that's my take as a relative outsider to Emacs input, so someone
probably has a better idea.

Any thoughts?  Thanks.

Re: xwidget events

[Aiko Kyle]

On Sat, Oct 30, 2021 at 7:13 AM Po Lu <luangruo@yahoo.com> wrote:
>
> I've been experimenting with making xwidget-webkit on X11 usable as an
> actual web browser.  So far I've figured out the annoying redraw issues
> and the broken scroll optimization, but I'm still puzzling over event
> handling.
>
> I was able to borrow a Mac from a friend today for an hour or so, and I
> used it to try the NS xwidget implementation.  It seems to handle mouse,
> button and click events entirely separately from the rest of Emacs,
> which I don't think is a very good idea either.  But that also means the
> browser is actually useful, which is a good thing.
>
> In the existing X11 xwidget implementation, only some events, such as
> motion events are passed through (without being exposed to Lisp code
> first), while with the improvements to xwidget redisplay in bug#51473,
> no events are passed through to the xwidget at all.
>
> So I think a better idea would be the ability to pass through individual
> input events via Lisp code (for instance with a function named
> `xwidget-do-event', which would accept a lispy event as input and then
> send an equivalent event to the xwidget).  Lisp code could then take
> input events in the xwidget-webkit, and then send them to the xwidget.
>
> But that's my take as a relative outsider to Emacs input, so someone
> probably has a better idea.
>
> Any thoughts?  Thanks.

Sorry I haven't had the time to look over what you've done. I'm quite
interested, but unfortunately have very limited time these days.

In emacs-webkit, I introduced an "insert" mode to allows one to pass
keyboard events directly to webkitgtk's widget. This is done by
focusing the webkitgtk widget so gtk passes input events directly to
it and emacs does not receive them. I did this since I thought it was
very bad UI to see one's cursor on a web text field, try to interact
with it, and have nothing happen. I know xwidgets has this injected JS
workaround that allows one to type in the emacs minibuffer and send
the text to the html field. However this won't work to, say, tab
between form fields or use arrow keys to make selections. I'm not sure
how you'd extend the JS solution to handle all these cases generally.

On keyboard events, there will always be a tension between the CUA
keys that webkit is hardcoded to use and emac's keys. Someone could
write the appropriate translation layer, but it'll be a nontrivial,
likely manual, task and will require constant upkeep as these things
evolve in webkitgtk. Personally as an evil and former exwm user,
making keyboard focus a modal distinction is natural.

On mouse events, I'm not sure what you will gain by exposing such
events to lisp first then passing them to webkit. Is there some use
case this would enable? I can't imagine it being very helpful without
lisp also knowing what DOM element the mouse event is associated with.
I would also be worried about latency with the additional redirection.
Additionally, until emacs learns to handle xinput 2 type events, you
might risk losing all the multitouch gestures that webkit supports
including smooth two-finger scrolling.

Re: xwidget events

[Po Lu]

Aiko Kyle <aikokyle@gmail.com> writes:

> In emacs-webkit, I introduced an "insert" mode to allows one to pass
> keyboard events directly to webkitgtk's widget. This is done by
> focusing the webkitgtk widget so gtk passes input events directly to
> it and emacs does not receive them. I did this since I thought it was
> very bad UI to see one's cursor on a web text field, try to interact
> with it, and have nothing happen. I know xwidgets has this injected JS
> workaround that allows one to type in the emacs minibuffer and send
> the text to the html field. However this won't work to, say, tab
> between form fields or use arrow keys to make selections. I'm not sure
> how you'd extend the JS solution to handle all these cases generally.

BTW, I have already implemented something similar: a function
`xwidget-perform-lispy-event' to send Lisp events to an xwidget, and an
`xwidget-webkit-edit-mode' that sends self inserting characters and
arrow keys to xwidgets-webkit.

You can find a recent version of my changes somewhere in the discussion
around bug#51473.

> On keyboard events, there will always be a tension between the CUA
> keys that webkit is hardcoded to use and emac's keys. Someone could
> write the appropriate translation layer, but it'll be a nontrivial,
> likely manual, task and will require constant upkeep as these things
> evolve in webkitgtk. Personally as an evil and former exwm user,
> making keyboard focus a modal distinction is natural.

Yes, but in the meantime I added some items to the menu bar that could
help with tasks such as saving the selection.

> On mouse events, I'm not sure what you will gain by exposing such
> events to lisp first then passing them to webkit. Is there some use
> case this would enable? I can't imagine it being very helpful without
> lisp also knowing what DOM element the mouse event is associated with.
> I would also be worried about latency with the additional redirection.

I'm not particularly worried about input latency, but it would be a bad
thing to enable this for every xwidget.  WDYT about the ability to set
input event masks for individual xwidgets, so people who write Lisp code
can decide which events get passed through right after handling the X
event, and which events get passed through Lisp code first?

> Additionally, until emacs learns to handle xinput 2 type events, you
> might risk losing all the multitouch gestures that webkit supports
> including smooth two-finger scrolling.

Unfortunately, it's impossible to support XInput 2 devices in any GTK
widget running inside an Emacs process at present: it's disabled so GTK
doesn't set an XI2 event mask that prevents Emacs from handling Core
Input events.

I do have a patch that enables XInput 2 support in Emacs, but I don't
have time to work on it right now.

Thanks.