Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Akira Kyle <akira@akirakyle.com>]

On Sun, Nov 29, 2020 at 12:01 PM, Eli Zaretskii <eliz@gnu.org> 
wrote:

>> One such fundamental incompatibility stems from the fact that 
>> Emacs may have multiple views on one buffer split across 
>> various 
>> windows and/or frames. Gtk assumes that each widget will only 
>> have 
>> one view. The 'webkit xwidget type currently works around this 
>> by 
>> drawing to an offscreen window then for every window that's 
>> supposed to display the widget, creating a gtk drawing area 
>> widget 
>> and copying the offscreen window's surface to the drawing 
>> surface. I don't think gtk really intends for ofscreen widgets 
>> to 
>> be used this way and as far as I can tell offscreen window 
>> widgets 
>> have been removed in gtk4.
>>
>> Another issue is that emacs uses a GtkFixed (or a custom 
>> subclassed EmacsFixed for gtk3) as the container widget for the 
>> main frame area. There isn't a mechanism to control z-ordering 
>> of 
>> child widgets other than removing and re-adding them in the 
>> desired z-order, which is what emacs-webkit has to do in order 
>> for 
>> child frames to not render below the webkit widget.
>
> The same problems plague your experiment with a module, don't 
> they?

More or less yes. Restricting myself to just a webkit widget 
allows me to not have to solve these problems generally. For 
example I don't draw the widget offscreen, instead I take a lesson 
from exwm and try to force Emacs to only ever display at most one 
window per webkit buffer.

> Anyway, this just tells me that GTK is not a good starting point 
> for
> embedding widgets.  Do other projects build embedded widgets on 
> such
> shaky grounds?

I wouldn't say GTK isn't good for embedding widgets. In fact I'd 
say it's probably great if you're wanting to extend an existing 
GTK app with custom widgets. The problem is Emacs, despite using 
GTK as a toolkit, isn't really a GTK app. Emacs just uses GTK to 
handle getting input and drawing a surface to the screen. Because 
Emacs isn't itself composed of GTK "normal" widgets, but rather 
just one GtkFixed container it does all its drawing in, you can't 
expect GTK widgets added along side what Emacs draws in that area 
to play nicely together without significant effort. And that 
effort boils down to making GTK aware of what Emacs has drawn and 
vice versa. If all the UI components of Emacs such as windows, the 
mode line, etc, were each implemented as a sensible GTK widget, 
then this wouldn't be so much of an issue since then GTK would 
already know where and what things are.

>> > As long as the widget can be told to use a given rectangular
>> > portion of the screen, we should be fine: the current Emacs
>> > display engine is perfectly capable of handling display 
>> > elements
>> > that occupy an area of certain dimensions.
>> 
>> This is more an issue on the gtk side of things. With the 
>> webkit 
>> widget it isn't an issue since the widget doesn't define a 
>> minimum 
>> size so any size you request it to render at, it will 
>> obey. However other widgets, such as buttons, sliders, etc, 
>> have a 
>> minimum size they will render at. If you tell it to render 
>> itself 
>> smaller, it will refuse and so it's impossible to simply tell 
>> the 
>> widget "here's the space you have to go in". This becomes a 
>> problem when the widget needs to be clipped at a window 
>> boundary 
>> as gtk will happily draw the widgets across them as gtk has no 
>> internal knowledge of Emacs window boundaries being a place 
>> that 
>> widget must be clipped at.
>
> This is solvable.  We already have similar situation in the 
> display
> engine, where some display element cannot be usefully "clipped". 
> We
> either don't display it at all or display it on the next screen 
> line,
> depending on the wrap mode.

I don't doubt its solvable, but how satisfactorily and at what 
complexity? I would say not displaying an element when some 
portion could be displayed is visually very unsatisfying. What if 
the element is large and 99% of it could be visible? What if the 
element is wider than the window? What if 
`window-resize-pixelwise` or `frame-resize-pixelwise` non-nil and 
the element is on the last, partially clipped line?

>> The direction I'm thinking of going in the future since I feel 
>> like thus far the xwidget route has been fairly troublesome is 
>> to 
>> expose interfaces to the cairo surfaces used in image.c. This 
>> could be through modifying image.c or through defining a new 
>> display element type. I would try to expose the cairo API to 
>> lisp 
>> and allow dynamic modules to directly draw to a cairo surface 
>> given to it through the lisp interface. Gtk along with many 
>> other 
>> libraries (such as librsvg and poppler) support drawing to a 
>> cairo 
>> surface and then Emacs gets to have full control over how that 
>> cario surface ends up on screen. I would see this as a stepping 
>> stone to attempting to do something similar with a opengl 
>> surface.
>
> My suggestion is to think how to integrate this idea into Emacs
> proper, without introducing modules into the equation.  Modules 
> add
> another layer of complexity, so I would suggest to design the 
> solution
> inside Emacs, and only after that see how to do that from a 
> module.
> One step at a time.

Of course the first step would be to work inside Emacs proper, and 
expose lisp interfaces. However I think designing such a feature 
should keep eventual integration with modules in mind. Especially 
since in general graphics is very performance sensitive and so 
doing some drawing in elisp just might not cut it and many 
interesting applications of such a feature would require 
interfacing with an external c library. This is all pretty 
hypothetical at this point though and as I keep learning with 
Emacs C code: its hard to speculate on the best way to do 
something without actually trying something out first.