Rethinking the design of xwidgets

[Akira Kyle <ak@akirakyle.com>]

Hi all,

If you don't care about the motivation and background for this 
problem skip the next two paragraphs.

One of my primary uses of emacs is as a scientific notebook 
utilizing org mode's support for literate programming to 
interweave prose, LaTeX, and code. I find this to be far more 
powerful and comfortable than using the Jupyter notebooks which 
have quickly become incredibly popular for such scientific 
computing. However the biggest pain point I have is due to Emacs' 
relatively poor capabilities for interfacing with objects other 
than text. Sure Emacs can display images ok, but I'm not aware of 
any way to get emacs to display more dynamic content. Furthermore 
I always want to be able to do more inside Emacs, and the one 
other application I spend as much time in as I spend in Emacs is a 
browser. Unfortunately I can't even really use eww for the 
simplest of my web browsing tasks such as looking up documentation 
as it usually fails to adequately render the variety of ways 
websites embed mathematics.

This has all led me to be interested in ways to enhance Emacs' 
display capabilities. The two projects I've come across that do 
this are the experimental xwidgets configuration and the Emacs 
Application Framework (EAF) [1]. Unfortunately EAF uses Qt to 
reparent a window into emacs and Python to do two way RPC with 
elisp. IMO it isn't a very emacsy way to enhance Emacs' display 
capabilites and is limited to just taking over a whole window 
rather than being embedded as part of a buffer (key to supporting 
dynamic content such as an interactive inline plot inside an org 
mode document). That leaves xwidgets which seems to be somewhat 
incomplete, unused, and neglected. I've been poking around the 
xwidgets code and I really like the concept, however the 
implementation seems to need some more thought and work.

I see two fundamental issues with the current xwidget code. The 
first is that in order to add a new type of widget, one must 
integrate it into the C portion of Emacs' codebase and expose 
relevant functionality to elisp. The difficulty of navigating how 
xwidgets are integrated into redisplay and how to expose the 
inherently event driven nature of GUI toolkits into elisp in a 
safe way is a big hurdle to hacking on xwidgets. In the last 
decade of this features existence no one has added any additional 
xwidgets besides the original webkit one (see make-xwidget). This 
despite the fact that xwidgets were originally designed to allow 
embedding of any gtk widgets such as buttons, sliders, image 
views, opengl contexts [2]. Recently support for webkit using the 
native cocoa framework was added for the ns toolkit demonstrating 
that this feature need not be tied to a specific GUI toolkit.  

The second issue is that most interesting applications of xwidgets 
will likely need to use some external C library. Historically, 
package maintainers have been hesitant to package emacs with 
xwidgets support due to the additional dependency on webkitgtk, 
which was often not kept current with security patches [3]. In the 
long run I don't think adding extra dependencies to emacs just for 
the sake of some specific xwidget is worth it.

That leads me to my following proposal to rethink the design of 
xwidgets. Given that Emacs now has support for dynamic modules and 
it is now enabled by default, I propose exposing an interface for 
dynamic modules to define custom xwidgets. I believe this would be 
a good way to solve the two fundamental issues with the current 
xwidgets implementation that I described above. Having a well 
defined interface means one wouldn't have to deal with debugging 
subtle redisplay and lisp interpreter issues when creating a 
custom xwidget type. Furthermore any external library needed for a 
custom xwidget type will only be required if one chooses to use 
that xwidget type and would keep such dependencies out of Emacs 
itself.

I've already done a proof of concept prototype to convince myself 
that this is possible. I first modified the xwidget display spec 
to include a field which is expected to contain a function to 
initialize an xwidget view. When redisplay acts on this spec, the 
xwidgets code calls this function with the xwidget display spec 
and expects a user_ptr Lisp_Object to be returned which wraps a 
pointer to a GtkWidget. The xwidgets code then adds this widget to 
an appropriate container that it uses to ensure the widget is 
moved when the position or clipping of its glyph changes. I then 
define this xwidget view initialization function in a dynamic 
module.

Since every window containing the xwidget has its own view, and 
hence separate GtkWidget instance, it is then up to the dynamic 
module to define the how to handle the updating of each view based 
on some common model each is supposed to represent. This works 
around the limitation that in most GUI toolkits each widget can 
only have one view. Thus for something like the webkitgtk widget, 
one implementation might choose to do offscreen rendering (as is 
currently done) so that all windows displaying the widget are 
updated together when, say, one of them is scrolled. While another 
implementation might decide each view gets a separate view of the 
same webpage and scrolling one will not scroll the other (which is 
what the cocoa webkit implementation should probably do, but it is 
currently only limited to being displayed in a single window).

I'd be interested in any thoughts anyone might have about this 
proposal and if some changes along this line might have some 
chance of being accepted in master someday. I've always wanted to 
dig into the nity grity c code of emacs and this has finally given 
me the excuse to do so! 

Akira

[1] https://github.com/manateelazycat/emacs-application-framework
[2] https://github.com/jave/xwidget-emacs
[3] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=843462