Re: Rethinking the design of xwidgets

[Akira Kyle <ak@akirakyle.com>]

On Tue, Oct 13, 2020 at 11:55 AM, Eli Zaretskii <eliz@gnu.org> 
wrote:

> I'll need to read up on those two topics, but in general, as 
> someone
> already pointed out, the Emacs display of images supports
> interactivity by letting you call functions triggered by mouse
> gestures on the varipous spots on the image.  If such a function
> produces a zoomed-in/out image and displays it instead of the 
> original
> one, wouldn't that fulfill the need?  E.g., we already support
> animated GIF images and zoom-in/out by mouse scroll, so it is
> inaccurate to say that we can only display static images.

Can actions be triggered on a click and drag as one would do to 
pan within a plot?

> This use case doesn't necessarily call for the same solution, 
> but I
> could think of a differently designed pdf-viewer that didn't 
> need to
> rasterize each page separately.  Again, the mouse-sensitivity 
> feature
> of our image display could perhaps be used for something like 
> that.  I
> reckon the hardest problem is to find an engine, be it a library 
> or a
> program, that could convert PDF into something Emacs can 
> display.

As far as I know there are really only two FLOSS PDF libraries out 
there: poppler and muPDF. Between them poppler supports more PDF 
features and seems to be more widely used. I agree this isn't 
really as compelling of a use case for xwidgets, but I seems like 
it would be easier to achieve better PDF rendering via xwidgets. I 
don't think it's about needing to rasterize each page separately, 
its more about directly rendering the rasterized image without 
having to copy it around a bunch or convert it to different 
formats. I suppose the image rendering from an elisp string could 
be made more performant or Emacs could handle pdf display natively 
but that would require adding poppler as a dependency.

> E.g., the kludge in dispnew.c around line 4365.  It disables one 
> of
> the most important redisplay optimizations in Emacs, once you 
> build
> with xwidgets enabled.

Ah, I had briefly looked at that but I wasn't sure how important 
it was. I may be able to make some progress on that but it's hard 
to understand the context for that function. It seems like that 
function is responsible for an optimization that avoids redrawing 
glyphs if the window is being scrolled by copying them, however I 
couldn't find where or how the glyphs are actually copied?

>> For example, consider svg support, which is my preferred image 
>> format. As far is I can tell, currently Emacs rasterizes svgs 
>> itself before ultimately drawing them on a cairo context.
>
> Do we actually do that?  We use librsvg, don't we?
>
> Or maybe you are talking about the generic layer of image 
> handling,
> which basically needs a bitmap/pixmap?

Yes this is what I'm talking about. In svg_load_image librsvg is 
used to end up with the rasterized svg in an Emacs_Pix_Container 
which then goes through a song and dance of pixel level color and 
transparency transformations to all eventually end back up as a 
cairo surface. Unless I'm missing something it seems like all this 
extra complexity and copying around is just for the sake of 
maintaining Emacs's own image format which was established long 
before HiDPI, deep color depth displays and libraries like cairo 
that do the hard work of transforming and rendering onto the 
glass.

> In any case, this seems to be a much more fundamental issue of 
> how we
> display images in Emacs.  Which is not necessarily the same as 
> the
> issue you raised originally.

It's at least related since the integration xwidgets into Emacs' 
display was originally modeled off of the way Emacs' displays 
images, however I see it as potentially being a simpler and more 
flexible alternative to the way Emacs currently displays images.

> The problem with that is that handling such "widgets" in our 
> display
> engine is entirely non-trivial, as the xwidgets code clearly 
> shows.
> There are basic problems in displaying xwidgets that from my POV 
> are
> left unsolved, and I'm not sure we will be able to find 
> satisfactory
> solutions for them.  The display engine has several requirements 
> that
> aren't easy to support with widgets that want to manage 
> themselves.

I think it would be good to try to either resolve such issues or 
decide that Emacs would be better off without xwidgets and instead 
focus on enhancing its existing rich display capabilities because 
as it is, the xwidgets code seems to have been lingering in this 
experimental state for a decade now. I think many like myself are 
drawn by its promise of being able to rendering complex dynamic 
content using existing optimized toolkits. There is of course the 
fact that in its current state on master the only thing it does do 
is embed a webkit browser in an Emacs window. A feature many would 
like to see enhanced while many others have voiced both security, 
privacy, and freedom concerns with such a feature. I'd just like 
to see a plan on where the community thinks xwidgets should go and 
work from there since it's been experimental for far too long.

> I see where you are coming from with the idea of dynamic 
> modules, but
> I'm not sure your assumption that it will make things easier for
> implementing "display elements" that have their own behavior
> independent of Emacs -- I'm not sure this assumption is indeed 
> true.
> The Emacs display code has a deceptively simple main design idea 
> and
> execution concept, and as long as one thinks this is all there 
> is to
> it, it could indeed seem that adding such widgets should be 
> almost
> trivial.  But the reality is that there are a lot of details to 
> go
> with the main idea, and it's the details that make implementing 
> this
> stuff more difficult than one thinks first.

Thank you for taking the time to talk through this with me. I'm 
seeing more and more that when it comes to the C portion of Emacs' 
code, nothing is ever quite so straightforward and it's difficult 
to understand all the intricacies due to the sheer amount of code.

> I'm not trying to scare you, mind you, just point out that the 
> ideas
> you have should probably be talked through in mode detail, to 
> see that
> they are workable.  For example, did you study the various
> window-redisplay optimizations, and if so, did you think how the
> widgets will adapt to them?

I'm just starting to dig my teeth into some of that. However I'm 
wondering if anyone has more recently tried to quantify the 
improvements the various redisplay optimizations make to both 
actual and perceived render time of Emacs. For example the comment 
on the scrolling_window function in dispnew.c that you pointed me 
too references that it is implementing an algorithm from 1978, 
perhaps it would be worth exploring if there are any algorithmic 
advances? It strikes me that perhaps some optimizations may no 
longer be as necessary on modern hardware or on GUI displays as 
they were when running emacs in a VT on 20 year 
hardware. Conversely there may be further optimization that could 
be done by taking advantage of more recent hardware advances such 
as offloading to a GPU or utilizing vectorized instructions. If 
there's one think I've taken away from my CS education, its that 
indeed "premature optimization is the root of all evil". I know 
Emacs prides itself on its broad portability, however when it 
comes to xwidgets we already know such a feature is restricted to 
running on the hardware that gtk3 expects. Anyways I know these 
are mostly lofty thoughts but I'd like to continue to see Emacs 
evolve.