Rethinking the design of xwidgets

Rethinking the design of xwidgets

[Akira Kyle]

Hi all,

If you don't care about the motivation and background for this 
problem skip the next two paragraphs.

One of my primary uses of emacs is as a scientific notebook 
utilizing org mode's support for literate programming to 
interweave prose, LaTeX, and code. I find this to be far more 
powerful and comfortable than using the Jupyter notebooks which 
have quickly become incredibly popular for such scientific 
computing. However the biggest pain point I have is due to Emacs' 
relatively poor capabilities for interfacing with objects other 
than text. Sure Emacs can display images ok, but I'm not aware of 
any way to get emacs to display more dynamic content. Furthermore 
I always want to be able to do more inside Emacs, and the one 
other application I spend as much time in as I spend in Emacs is a 
browser. Unfortunately I can't even really use eww for the 
simplest of my web browsing tasks such as looking up documentation 
as it usually fails to adequately render the variety of ways 
websites embed mathematics.

This has all led me to be interested in ways to enhance Emacs' 
display capabilities. The two projects I've come across that do 
this are the experimental xwidgets configuration and the Emacs 
Application Framework (EAF) [1]. Unfortunately EAF uses Qt to 
reparent a window into emacs and Python to do two way RPC with 
elisp. IMO it isn't a very emacsy way to enhance Emacs' display 
capabilites and is limited to just taking over a whole window 
rather than being embedded as part of a buffer (key to supporting 
dynamic content such as an interactive inline plot inside an org 
mode document). That leaves xwidgets which seems to be somewhat 
incomplete, unused, and neglected. I've been poking around the 
xwidgets code and I really like the concept, however the 
implementation seems to need some more thought and work.

I see two fundamental issues with the current xwidget code. The 
first is that in order to add a new type of widget, one must 
integrate it into the C portion of Emacs' codebase and expose 
relevant functionality to elisp. The difficulty of navigating how 
xwidgets are integrated into redisplay and how to expose the 
inherently event driven nature of GUI toolkits into elisp in a 
safe way is a big hurdle to hacking on xwidgets. In the last 
decade of this features existence no one has added any additional 
xwidgets besides the original webkit one (see make-xwidget). This 
despite the fact that xwidgets were originally designed to allow 
embedding of any gtk widgets such as buttons, sliders, image 
views, opengl contexts [2]. Recently support for webkit using the 
native cocoa framework was added for the ns toolkit demonstrating 
that this feature need not be tied to a specific GUI toolkit.  

The second issue is that most interesting applications of xwidgets 
will likely need to use some external C library. Historically, 
package maintainers have been hesitant to package emacs with 
xwidgets support due to the additional dependency on webkitgtk, 
which was often not kept current with security patches [3]. In the 
long run I don't think adding extra dependencies to emacs just for 
the sake of some specific xwidget is worth it.

That leads me to my following proposal to rethink the design of 
xwidgets. Given that Emacs now has support for dynamic modules and 
it is now enabled by default, I propose exposing an interface for 
dynamic modules to define custom xwidgets. I believe this would be 
a good way to solve the two fundamental issues with the current 
xwidgets implementation that I described above. Having a well 
defined interface means one wouldn't have to deal with debugging 
subtle redisplay and lisp interpreter issues when creating a 
custom xwidget type. Furthermore any external library needed for a 
custom xwidget type will only be required if one chooses to use 
that xwidget type and would keep such dependencies out of Emacs 
itself.

I've already done a proof of concept prototype to convince myself 
that this is possible. I first modified the xwidget display spec 
to include a field which is expected to contain a function to 
initialize an xwidget view. When redisplay acts on this spec, the 
xwidgets code calls this function with the xwidget display spec 
and expects a user_ptr Lisp_Object to be returned which wraps a 
pointer to a GtkWidget. The xwidgets code then adds this widget to 
an appropriate container that it uses to ensure the widget is 
moved when the position or clipping of its glyph changes. I then 
define this xwidget view initialization function in a dynamic 
module.

Since every window containing the xwidget has its own view, and 
hence separate GtkWidget instance, it is then up to the dynamic 
module to define the how to handle the updating of each view based 
on some common model each is supposed to represent. This works 
around the limitation that in most GUI toolkits each widget can 
only have one view. Thus for something like the webkitgtk widget, 
one implementation might choose to do offscreen rendering (as is 
currently done) so that all windows displaying the widget are 
updated together when, say, one of them is scrolled. While another 
implementation might decide each view gets a separate view of the 
same webpage and scrolling one will not scroll the other (which is 
what the cocoa webkit implementation should probably do, but it is 
currently only limited to being displayed in a single window).

I'd be interested in any thoughts anyone might have about this 
proposal and if some changes along this line might have some 
chance of being accepted in master someday. I've always wanted to 
dig into the nity grity c code of emacs and this has finally given 
me the excuse to do so! 

Akira

[1] https://github.com/manateelazycat/emacs-application-framework
[2] https://github.com/jave/xwidget-emacs
[3] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=843462

Re: Rethinking the design of xwidgets

[joakim]

Akira Kyle <ak@akirakyle.com> writes:

> Hi all,
>
> If you don't care about the motivation and background for this problem
> skip the next two paragraphs.
>
> One of my primary uses of emacs is as a scientific notebook utilizing
> org mode's support for literate programming to interweave prose,
> LaTeX, and code. I find this to be far more powerful and comfortable
> than using the Jupyter notebooks which have quickly become incredibly
> popular for such scientific computing. However the biggest pain point
> I have is due to Emacs' relatively poor capabilities for interfacing
> with objects other than text. Sure Emacs can display images ok, but
> I'm not aware of any way to get emacs to display more dynamic
> content. Furthermore I always want to be able to do more inside Emacs,
> and the one other application I spend as much time in as I spend in
> Emacs is a browser. Unfortunately I can't even really use eww for the
> simplest of my web browsing tasks such as looking up documentation as
> it usually fails to adequately render the variety of ways websites
> embed mathematics.
>
> This has all led me to be interested in ways to enhance Emacs' display
> capabilities. The two projects I've come across that do this are the
> experimental xwidgets configuration and the Emacs Application
> Framework (EAF) [1]. Unfortunately EAF uses Qt to reparent a window
> into emacs and Python to do two way RPC with elisp. IMO it isn't a
> very emacsy way to enhance Emacs' display capabilites and is limited
> to just taking over a whole window rather than being embedded as part
> of a buffer (key to supporting dynamic content such as an interactive
> inline plot inside an org mode document). That leaves xwidgets which
> seems to be somewhat incomplete, unused, and neglected. I've been
> poking around the xwidgets code and I really like the concept, however
> the implementation seems to need some more thought and work.
>
> I see two fundamental issues with the current xwidget code. The first
> is that in order to add a new type of widget, one must integrate it
> into the C portion of Emacs' codebase and expose relevant
> functionality to elisp. The difficulty of navigating how xwidgets are
> integrated into redisplay and how to expose the inherently event
> driven nature of GUI toolkits into elisp in a safe way is a big hurdle
> to hacking on xwidgets. In the last decade of this features existence
> no one has added any additional xwidgets besides the original webkit
> one (see make-xwidget). This despite the fact that xwidgets were
> originally designed to allow embedding of any gtk widgets such as
> buttons, sliders, image views, opengl contexts [2]. Recently support
> for webkit using the native cocoa framework was added for the ns
> toolkit demonstrating that this feature need not be tied to a specific
> GUI toolkit.  The second issue is that most interesting applications
> of xwidgets will likely need to use some external C
> library. Historically, package maintainers have been hesitant to
> package emacs with xwidgets support due to the additional dependency
> on webkitgtk, which was often not kept current with security patches
> [3]. In the long run I don't think adding extra dependencies to emacs
> just for the sake of some specific xwidget is worth it.
>
> That leads me to my following proposal to rethink the design of
> xwidgets. Given that Emacs now has support for dynamic modules and it
> is now enabled by default, I propose exposing an interface for dynamic
> modules to define custom xwidgets. I believe this would be a good way
> to solve the two fundamental issues with the current xwidgets
> implementation that I described above. Having a well defined interface
> means one wouldn't have to deal with debugging subtle redisplay and
> lisp interpreter issues when creating a custom xwidget
> type. Furthermore any external library needed for a custom xwidget
> type will only be required if one chooses to use that xwidget type and
> would keep such dependencies out of Emacs itself.
>
> I've already done a proof of concept prototype to convince myself that
> this is possible. I first modified the xwidget display spec to include
> a field which is expected to contain a function to initialize an
> xwidget view. When redisplay acts on this spec, the xwidgets code
> calls this function with the xwidget display spec and expects a
> user_ptr Lisp_Object to be returned which wraps a pointer to a
> GtkWidget. The xwidgets code then adds this widget to an appropriate
> container that it uses to ensure the widget is moved when the position
> or clipping of its glyph changes. I then define this xwidget view
> initialization function in a dynamic module.
>
> Since every window containing the xwidget has its own view, and hence
> separate GtkWidget instance, it is then up to the dynamic module to
> define the how to handle the updating of each view based on some
> common model each is supposed to represent. This works around the
> limitation that in most GUI toolkits each widget can only have one
> view. Thus for something like the webkitgtk widget, one implementation
> might choose to do offscreen rendering (as is currently done) so that
> all windows displaying the widget are updated together when, say, one
> of them is scrolled. While another implementation might decide each
> view gets a separate view of the same webpage and scrolling one will
> not scroll the other (which is what the cocoa webkit implementation
> should probably do, but it is currently only limited to being
> displayed in a single window).
>
> I'd be interested in any thoughts anyone might have about this
> proposal and if some changes along this line might have some chance of
> being accepted in master someday. I've always wanted to dig into the
> nity grity c code of emacs and this has finally given me the excuse to
> do so! Akira

As the original author of the xwidgets code, I'm happy that you are
taking an interest in moving the idea forward, and I like your ideas.

I had these ideas in the same vein originally:

- Use the gnome object system to create and manipulate xwidgets
dynamically. There was a prototype for this in the original xwidget
branch. It worked, but it was tedious figuring out how to map between
elisp types and gtk types.

- In those cases when the gobject bridge didnt behave, use the emacs
  module system for shims(the approach you describe above)

Some questions:
- The original xwidgets were designed to allow several views to the same
xwidget using gtk reflection, as you describe. When I tried using
separate gtk controllers on screen, they didnt sync very well. Did you
try this and did it work well?

When I worked on this I tried several different gtk components such as
sliders etc. Some of the worked flaky, some worked rather well. What is
you current experience trying different components?

Anyway, thanks again for having a look at this!


>
> [1] https://github.com/manateelazycat/emacs-application-framework
> [2] https://github.com/jave/xwidget-emacs
> [3] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=843462
>
-- 
Joakim Verona
joakim@verona.se

Re: Rethinking the design of xwidgets

[Stefan Monnier]

Hi Kyle,

> If you don't care about the motivation and background for this problem skip
> the next two paragraphs.

Thanks for the motivation and background.

> I'd be interested in any thoughts anyone might have about this proposal and
> if some changes along this line might have some chance of being accepted in
> master someday.  I've always wanted to dig into the nity grity c code of
> emacs and this has finally given me the excuse to do so! Akira

Your analysis sounds quite right and using `emacs-modules` seems like
a very good approach.

I don't have much to contribute to this, except to encourage you to go
at it.  It sounds exciting.


        Stefan

Re: Rethinking the design of xwidgets

[Eli Zaretskii]

> From: Akira Kyle <ak@akirakyle.com>
> Date: Mon, 12 Oct 2020 14:58:19 -0600
> 
> One of my primary uses of emacs is as a scientific notebook 
> utilizing org mode's support for literate programming to 
> interweave prose, LaTeX, and code. I find this to be far more 
> powerful and comfortable than using the Jupyter notebooks which 
> have quickly become incredibly popular for such scientific 
> computing. However the biggest pain point I have is due to Emacs' 
> relatively poor capabilities for interfacing with objects other 
> than text. Sure Emacs can display images ok, but I'm not aware of 
> any way to get emacs to display more dynamic content.

Can you tell more about the dynamic content you had in mind, and why
you thought to use xwidgets as the basis for that?  There's IMO a leap
of logic here that I cannot follow, perhaps because I don't have the
necessary background information.

From my POV, xwidgets have several problems which don't allow me to
easily see them as the basis for some much more general set of
features or infrastructure:

  . they depend on GTK and webkit, whose development leaves a lot to
    be desired, and some say parts of that have no future; they are
    definitely less portable than I'd like to see in Emacs

  . their integration in the Emacs display code "needs work", there's
    at least one or two places where the code which handles them is
    clearly wrong -- this is semi-okay for a minor niche feature, but
    not for something on which we want to build our future

So before we decide that xwidgets is the way to go, I'd like to make a
step back and see what other alternatives we can think of, if any, and
carefully consider which alternative is best in the long run.

> That leads me to my following proposal to rethink the design of 
> xwidgets. Given that Emacs now has support for dynamic modules and 
> it is now enabled by default, I propose exposing an interface for 
> dynamic modules to define custom xwidgets. I believe this would be 
> a good way to solve the two fundamental issues with the current 
> xwidgets implementation that I described above. Having a well 
> defined interface means one wouldn't have to deal with debugging 
> subtle redisplay and lisp interpreter issues when creating a 
> custom xwidget type. Furthermore any external library needed for a 
> custom xwidget type will only be required if one chooses to use 
> that xwidget type and would keep such dependencies out of Emacs 
> itself.

Here, too, I'd like first to consider a design that doesn't require
dynamic modules as a prerequisite for the feature.  If we can come up
with something that requires only Lisp (like image display does, for
example), then I think we all win, because it is much easier to write
a Lisp package than a package that requires a module in C or similar
language.

Bottom line, I'd like to understand the problem you are trying to
solve better before we conclude that xwidgets and dynamic modules are
the way to go.

Thanks.

Re: Rethinking the design of xwidgets

[Akira Kyle]

Hi Joakim,

On Mon, Oct 12, 2020 at 04:18 PM, joakim@verona.se wrote:

> As the original author of the xwidgets code, I'm happy that you 
> are
> taking an interest in moving the idea forward, and I like your 
> ideas.

Thanks! And thank you for doing the hard work of initially 
figuring out how to integrate xwidgets into Emacs' redisplay.

> I had these ideas in the same vein originally:
>
> - Use the gnome object system to create and manipulate xwidgets
> dynamically. There was a prototype for this in the original 
> xwidget
> branch. It worked, but it was tedious figuring out how to map 
> between
> elisp types and gtk types.
>
> - In those cases when the gobject bridge didnt behave, use the 
> emacs
>   module system for shims(the approach you describe above)

My current understanding is that there isn't a plan to have elisp 
support a ffi, and the dynamic module system should be used 
instead. Given the small interface of the module system, I can see 
how it would be very tedious and difficult to figure out how to 
expose the large gobject and GTK api to elisp purely through the 
dynamic module system. I think my goal is much less ambitious: 
Rather then attempting to expose buttons, sliders, cairo surfaces, 
and all the variety of gtk widgets for elisp to manipulate (which 
also would be difficult to make portable to other GUI toolkits 
like cocoa), I simply want to expose a mechanism for a module to 
add a widget to a buffer. It would be up to the module for 
defining the right level of abstraction in exposing an elisp 
interface to the widget it creates.

> Some questions:
> - The original xwidgets were designed to allow several views to 
> the same
> xwidget using gtk reflection, as you describe. When I tried 
> using
> separate gtk controllers on screen, they didnt sync very 
> well. Did you
> try this and did it work well?
>
> When I worked on this I tried several different gtk components 
> such as
> sliders etc. Some of the worked flaky, some worked rather 
> well. What is
> you current experience trying different components?

So far I've started pretty simple and just looked at buttons and 
sliders, where each view gets it's own instance of the widget (so 
no offscreen rendering). The synchronizing of state is all handled 
by connecting each view's widget to a common signal handler which 
then updates the state of all the other views. So far the slider 
appears to sync its state pretty well across the views, but I have 
yet to look at more involved widgets.

> Anyway, thanks again for having a look at this!

Of course!

Re: Rethinking the design of xwidgets

[Akira Kyle]

On Tue, Oct 13, 2020 at 07:36 AM, Stefan Monnier 
<monnier@iro.umontreal.ca> wrote:

> Hi Kyle,
>
>> If you don't care about the motivation and background for this 
>> problem skip
>> the next two paragraphs.
>
> Thanks for the motivation and background.
>
>> I'd be interested in any thoughts anyone might have about this 
>> proposal and
>> if some changes along this line might have some chance of being 
>> accepted in
>> master someday.  I've always wanted to dig into the nity grity 
>> c code of
>> emacs and this has finally given me the excuse to do so! Akira
>
> Your analysis sounds quite right and using `emacs-modules` seems 
> like
> a very good approach.
>
> I don't have much to contribute to this, except to encourage you 
> to go
> at it.  It sounds exciting.
>
>
>         Stefan

Thanks for the support!

Re: Rethinking the design of xwidgets

[Akira Kyle]

On Tue, Oct 13, 2020 at 08:16 AM, Eli Zaretskii <eliz@gnu.org> 
wrote:

> Can you tell more about the dynamic content you had in mind, and 
> why
> you thought to use xwidgets as the basis for that?  There's IMO 
> a leap
> of logic here that I cannot follow, perhaps because I don't have 
> the
> necessary background information.

Sure, so to elaborate on my motivation: I spend a lot of my time 
in emacs using org-mode's babel support for python, specifically 
through emacs-jupyter [1]. This gives me much of the power of 
Jupyter notebooks, however not all. For example when generating 
plots using matplotlib, they are simply embedded in the buffer as 
an image, however matplotlib also has support for interactive 
backends [2] which allow one to zoom and pan the plot and see it 
update in real time as each plot command is issued. It would be 
convenient to have such interactivity directly inside the buffer 
rather than having to use an external window, similar to the 
interactivity offered when using matplotlib in a jupyter 
notebook. The bokeh plotting [3] library provides even more 
interactivity, allowing one to create complex and dynamic and 
interactive visualizations. When used inside a jupyter notebook, 
the plot is embedded inline, however when used with emacs-jupyter, 
it launches an external browser to display the interactive plot 
and it's controls. It would be fantastic if I could have such 
interactivity inside Emacs.

Another use case I had in mind for my proposed improved xwidgets 
is better pdf rendering. I use pdf-tools constantly and it's one 
of the best pdf viewers out there (not just because it's in 
emacs). I'll sometimes even use it to give presentations so I 
never have to leave Emacs. However it's design inherently limits 
its rendering performance as it must rasterize each pdf page in a 
separate process then pipe that data to emacs to display as an 
image. Granted this could be avoided by turning the epdfinfo 
server process into a dynamic module, however it will still be a 
bit roundabout as the pdf page would have to first be rendered 
into a cairo context then rasterized then Emacs will draw that 
rasterized image onto another cairo context to get it 
onscreen. Having direct access to an onscreen cairo context would 
eliminate this and make it easier to support crisp, performant 
rendering onto HiDPI displays. Maybe I could even throw slide 
animations into my presentations this way!

> From my POV, xwidgets have several problems which don't allow me 
> to
> easily see them as the basis for some much more general set of
> features or infrastructure:
>
>   . they depend on GTK and webkit, whose development leaves a 
>   lot to
>     be desired, and some say parts of that have no future; they 
>     are
>     definitely less portable than I'd like to see in Emacs

My proposal should actually make it easier for xwidgets to be 
decoupled from a specific GUI toolkit. The recent addition for a 
cocoa webkit xwidget has forced me to think a bit about this as I 
modify the xwidget code, and I'm fairly convinced that xwidgets 
can be designed is such as way that any object-based and 
event-driven toolkit could fit in its abstraction.

>   . their integration in the Emacs display code "needs work", 
>   there's
>     at least one or two places where the code which handles them 
>     is
>     clearly wrong -- this is semi-okay for a minor niche 
>     feature, but
>     not for something on which we want to build our future

Could you point me to some of those places. Maybe they are issues 
I've already taken a look at.

> So before we decide that xwidgets is the way to go, I'd like to 
> make a
> step back and see what other alternatives we can think of, if 
> any, and
> carefully consider which alternative is best in the long run.

I'd be interested if you can think of any alternatives since, at 
least to me, this seems like the natural way to implement such 
features.

> Here, too, I'd like first to consider a design that doesn't 
> require
> dynamic modules as a prerequisite for the feature.  If we can 
> come up
> with something that requires only Lisp (like image display does, 
> for
> example), then I think we all win, because it is much easier to 
> write
> a Lisp package than a package that requires a module in C or 
> similar
> language.

I think my pdf use case illustrates the issue with Emacs trying to 
support every such type of content as a high level "image-like" 
feature since you either force the application to convert 
everything into a sub-optimal format that Emacs understands for 
display or Emacs' must add native for such format, increasing the 
number of external dependencies to build Emacs.

For example, consider svg support, which is my preferred image 
format. As far is I can tell, currently Emacs rasterizes svgs 
itself before ultimately drawing them on a cairo context. Since 
librsvg supports drawing directly on a cairo context, it would be 
much more efficient to do so, however adding support for svg could 
be easily moved out of emacs into a dynamic module and drawn 
directly onto it's own widget. This would allow users of the NS 
toolkit to not have to rely on librsvg if they didn't want to as I 
believe NS can render svgs itself. 

> Bottom line, I'd like to understand the problem you are trying 
> to
> solve better before we conclude that xwidgets and dynamic 
> modules are
> the way to go.

Let me know if there's anything more I can clarify about my 
motivation and proposal to improve xwidgets.

[1] https://github.com/nnicandro/emacs-jupyter
[2] https://matplotlib.org/faq/usage_faq.html#what-is-a-backend
[3] https://bokeh.org/

Re: Rethinking the design of xwidgets

[Qiantan Hong]

[-- Attachment #1: Type: text/plain, Size: 943 bytes --]

> 
> For example when generating plots using matplotlib, they are simply embedded in the buffer as an image, however matplotlib also has support for interactive backends [2] which allow one to zoom and pan the plot and see it update in real time as each plot command is issued. It would be convenient to have such interactivity directly inside the buffer rather than having to use an external window, similar to the interactivity offered when using matplotlib in a jupyter notebook. The bokeh plotting [3] library provides even more interactivity, allowing one to create complex and dynamic and interactive visualizations.
Isn’t it possible to use the 'keymap text/overlay property on the image to do what you described? At least for zooming or
moving around view port, I think you can create some simple +/-/C-npfb bindings for the image and send the command to the backend.
Or write a lisp package for any more complex interaction.


[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 1858 bytes --]

Re: Rethinking the design of xwidgets

[Eli Zaretskii]

> From: Akira Kyle <ak@akirakyle.com>
> Cc: emacs-devel@gnu.org
> Date: Tue, 13 Oct 2020 11:05:51 -0600
> 
> Sure, so to elaborate on my motivation: I spend a lot of my time 
> in emacs using org-mode's babel support for python, specifically 
> through emacs-jupyter [1]. This gives me much of the power of 
> Jupyter notebooks, however not all. For example when generating 
> plots using matplotlib, they are simply embedded in the buffer as 
> an image, however matplotlib also has support for interactive 
> backends [2] which allow one to zoom and pan the plot and see it 
> update in real time as each plot command is issued. It would be 
> convenient to have such interactivity directly inside the buffer 
> rather than having to use an external window, similar to the 
> interactivity offered when using matplotlib in a jupyter 
> notebook. The bokeh plotting [3] library provides even more 
> interactivity, allowing one to create complex and dynamic and 
> interactive visualizations. When used inside a jupyter notebook, 
> the plot is embedded inline, however when used with emacs-jupyter, 
> it launches an external browser to display the interactive plot 
> and it's controls. It would be fantastic if I could have such 
> interactivity inside Emacs.

I'll need to read up on those two topics, but in general, as someone
already pointed out, the Emacs display of images supports
interactivity by letting you call functions triggered by mouse
gestures on the varipous spots on the image.  If such a function
produces a zoomed-in/out image and displays it instead of the original
one, wouldn't that fulfill the need?  E.g., we already support
animated GIF images and zoom-in/out by mouse scroll, so it is
inaccurate to say that we can only display static images.

> Another use case I had in mind for my proposed improved xwidgets 
> is better pdf rendering. I use pdf-tools constantly and it's one 
> of the best pdf viewers out there (not just because it's in 
> emacs). I'll sometimes even use it to give presentations so I 
> never have to leave Emacs. However it's design inherently limits 
> its rendering performance as it must rasterize each pdf page in a 
> separate process then pipe that data to emacs to display as an 
> image.

This use case doesn't necessarily call for the same solution, but I
could think of a differently designed pdf-viewer that didn't need to
rasterize each page separately.  Again, the mouse-sensitivity feature
of our image display could perhaps be used for something like that.  I
reckon the hardest problem is to find an engine, be it a library or a
program, that could convert PDF into something Emacs can display.

> >   . their integration in the Emacs display code "needs work", 
> >   there's
> >     at least one or two places where the code which handles them 
> >     is
> >     clearly wrong -- this is semi-okay for a minor niche 
> >     feature, but
> >     not for something on which we want to build our future
> 
> Could you point me to some of those places. Maybe they are issues 
> I've already taken a look at.

E.g., the kludge in dispnew.c around line 4365.  It disables one of
the most important redisplay optimizations in Emacs, once you build
with xwidgets enabled.

> I think my pdf use case illustrates the issue with Emacs trying to 
> support every such type of content as a high level "image-like" 
> feature since you either force the application to convert 
> everything into a sub-optimal format that Emacs understands for 
> display or Emacs' must add native for such format, increasing the 
> number of external dependencies to build Emacs.
> 
> For example, consider svg support, which is my preferred image 
> format. As far is I can tell, currently Emacs rasterizes svgs 
> itself before ultimately drawing them on a cairo context.

Do we actually do that?  We use librsvg, don't we?

Or maybe you are talking about the generic layer of image handling,
which basically needs a bitmap/pixmap?

In any case, this seems to be a much more fundamental issue of how we
display images in Emacs.  Which is not necessarily the same as the
issue you raised originally.

> Since librsvg supports drawing directly on a cairo context, it would
> be much more efficient to do so, however adding support for svg
> could be easily moved out of emacs into a dynamic module and drawn
> directly onto it's own widget.

The problem with that is that handling such "widgets" in our display
engine is entirely non-trivial, as the xwidgets code clearly shows.
There are basic problems in displaying xwidgets that from my POV are
left unsolved, and I'm not sure we will be able to find satisfactory
solutions for them.  The display engine has several requirements that
aren't easy to support with widgets that want to manage themselves.

I see where you are coming from with the idea of dynamic modules, but
I'm not sure your assumption that it will make things easier for
implementing "display elements" that have their own behavior
independent of Emacs -- I'm not sure this assumption is indeed true.
The Emacs display code has a deceptively simple main design idea and
execution concept, and as long as one thinks this is all there is to
it, it could indeed seem that adding such widgets should be almost
trivial.  But the reality is that there are a lot of details to go
with the main idea, and it's the details that make implementing this
stuff more difficult than one thinks first.

I'm not trying to scare you, mind you, just point out that the ideas
you have should probably be talked through in mode detail, to see that
they are workable.  For example, did you study the various
window-redisplay optimizations, and if so, did you think how the
widgets will adapt to them?

Re: Rethinking the design of xwidgets

[Akira Kyle]

On Tue, Oct 13, 2020 at 11:24 AM, Qiantan Hong <qhong@mit.edu> 
wrote:

> Isn’t it possible to use the 'keymap text/overlay property on 
> the image to do what you described? At least for zooming or
> moving around view port, I think you can create some simple 
> +/-/C-npfb bindings for the image and send the command to the 
> backend.
> Or write a lisp package for any more complex interaction.

That's probably possible to get some limited interaction, however 
I'm not sure it would very satisfactory or complete. For example 
is it possible to have a live updating rectangle displayed as one 
drags the mouse to select a region to inspect? Is it possible to 
display a slider which can be dragged by the mouse to update 
values on the plot or have a drop down menu to select a data 
series? It seems it may also be difficult to find a way to have 
elisp interact with the matplotlib fast enough to not introduce 
noticable delay as such actions update the view.

Re: Rethinking the design of xwidgets

[Tomas Hlavaty]

some kind of canvas was discussed in
id:87zh9hrxfj.fsf@logand.c
and
id:83img4aegz.fsf@gnu.org

It was discussed in context of a WYSIWYG editor features, and console
based pdf viewer emacs-frabuffer and pure elisp pdf generator emacs-pdf.

Eli suggested:

   > If you want to do layout for PDF, I think one way forward would be
   > to implement a pdfterm.c "terminal" for Emacs, which produces PDF
   > like the existing *term.c backends do for supported display types.

that might be better than some kind of xwidget

> For example, consider svg support, which is my preferred image 
> format. As far is I can tell, currently Emacs rasterizes svgs 
> itself before ultimately drawing them on a cairo context. Since 
> librsvg supports drawing directly on a cairo context, it would be 
> much more efficient to do so, however adding support for svg could 
> be easily moved out of emacs into a dynamic module and drawn 
> directly onto it's own widget. This would allow users of the NS 
> toolkit to not have to rely on librsvg if they didn't want to as I 
> believe NS can render svgs itself.

It is not clear to me, why a dynamic module would be better.

For example, emacs-framebuffer can display svg images without any
dynamic module by using external program, because emacs-nox does not
have cairo or librsvg or NS toolkit.

Ideally, there would be a canvas (e.g. pdfterm) providing low-level
drawing primitives (something like html canvas API) and the rest built
on top of that in elisp.

Re: Rethinking the design of xwidgets

[Tomas Hlavaty]

On Tue 13 Oct 2020 at 20:36, Tomas Hlavaty <tom@logand.com> wrote:
> id:87zh9hrxfj.fsf@logand.c

id:87zh9hrxfj.fsf@logand.com

Re: Rethinking the design of xwidgets

[Basil L. Contovounesios]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Akira Kyle <ak@akirakyle.com>
>> Cc: emacs-devel@gnu.org
>> Date: Tue, 13 Oct 2020 11:05:51 -0600
>> 
>> Another use case I had in mind for my proposed improved xwidgets 
>> is better pdf rendering. I use pdf-tools constantly and it's one 
>> of the best pdf viewers out there (not just because it's in 
>> emacs). I'll sometimes even use it to give presentations so I 
>> never have to leave Emacs. However it's design inherently limits 
>> its rendering performance as it must rasterize each pdf page in a 
>> separate process then pipe that data to emacs to display as an 
>> image.
>
> This use case doesn't necessarily call for the same solution, but I
> could think of a differently designed pdf-viewer that didn't need to
> rasterize each page separately.  Again, the mouse-sensitivity feature
> of our image display could perhaps be used for something like that.  I
> reckon the hardest problem is to find an engine, be it a library or a
> program, that could convert PDF into something Emacs can display.

Not sure if this fits the bill, but I was hoping to eventually look into
whether Emacs+Cairo could render PDFs using the GLib API of the GPLed
Poppler library.

-- 
Basil

Re: Rethinking the design of xwidgets

[Qiantan Hong]

[-- Attachment #1.1: Type: text/plain, Size: 1759 bytes --]


>> Isn’t it possible to use the 'keymap text/overlay property on the image to do what you described? At least for zooming or
>> moving around view port, I think you can create some simple +/-/C-npfb bindings for the image and send the command to the backend.
>> Or write a lisp package for any more complex interaction.
> 
> That's probably possible to get some limited interaction, however I'm not sure it would very satisfactory or complete. For example is it possible to have a live updating rectangle displayed as one drags the mouse to select a region to inspect? Is it possible to display a slider which can be dragged by the mouse to update values on the plot or have a drop down menu to select a data series? It seems it may also be difficult to find a way to have elisp interact with the matplotlib fast enough to not introduce noticable delay as such actions update the view.

I agree that displaying a live updating rectangle/slider following the mouse is near impossible.

For selecting region, supporting two mouse click should be totally doable. For selecting
data series, read-from-minibuffer together with user’s favorite completion/fuzzy search package
should be suffice. Those are discrete event. But do we really need continuous event that
much? E.g. what useful information is exchanged between computer/user when drawing a 
rectangle following user’s mouse? It seems to me that only the starting and ending position
matters. More here http://cat-v.mit.edu/2019-10-15-mouse-considered-harmful.html <http://cat-v.mit.edu/2019-10-15-mouse-considered-harmful.html>

However, if everyone decides that every event mouse spits out matters (which is a lot),
then it is the case that image display won’t work good enough.

[-- Attachment #1.2: Type: text/html, Size: 2426 bytes --]

[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 1858 bytes --]

Re: Rethinking the design of xwidgets

[Eli Zaretskii]

> From: Akira Kyle <ak@akirakyle.com>
> Cc: Eli Zaretskii <eliz@gnu.org>, emacs-devel@gnu.org
> Date: Tue, 13 Oct 2020 12:29:38 -0600
> 
> For example is it possible to have a live updating rectangle
> displayed as one drags the mouse to select a region to inspect?

Can't that be done with a mask on the image?

> Is it possible to display a slider which can be dragged by the mouse
> to update values on the plot or have a drop down menu to select a
> data series?

You'd need to regenerate the plot with the updated values.

> It seems it may also be difficult to find a way to have elisp
> interact with the matplotlib fast enough to not introduce noticable
> delay as such actions update the view.

And doing this via a module that is called by a widget won't bump into
the same problems?  After all, all the interaction with Emacs goes
through the same mechanism, no matter what the callback does.

Re: Rethinking the design of xwidgets

[joakim]

Akira Kyle <ak@akirakyle.com> writes:

> Hi Joakim,
>
> On Mon, Oct 12, 2020 at 04:18 PM, joakim@verona.se wrote:
>
>> As the original author of the xwidgets code, I'm happy that you are
>> taking an interest in moving the idea forward, and I like your
>> ideas.
>
> Thanks! And thank you for doing the hard work of initially figuring
> out how to integrate xwidgets into Emacs' redisplay.
>
>> I had these ideas in the same vein originally:
>>
>> - Use the gnome object system to create and manipulate xwidgets
>> dynamically. There was a prototype for this in the original xwidget
>> branch. It worked, but it was tedious figuring out how to map
>> between
>> elisp types and gtk types.
>>
>> - In those cases when the gobject bridge didnt behave, use the emacs
>>   module system for shims(the approach you describe above)
>
> My current understanding is that there isn't a plan to have elisp
> support a ffi, and the dynamic module system should be used
> instead. Given the small interface of the module system, I can see how
> it would be very tedious and difficult to figure out how to expose the
> large gobject and GTK api to elisp purely through the dynamic module
> system. I think my goal is much less ambitious: Rather then attempting
> to expose buttons, sliders, cairo surfaces, and all the variety of gtk
> widgets for elisp to manipulate (which also would be difficult to make
> portable to other GUI toolkits like cocoa), I simply want to expose a
> mechanism for a module to add a widget to a buffer. It would be up to
> the module for defining the right level of abstraction in exposing an
> elisp interface to the widget it creates.


Yes, I think that sounds more realistic than my original plan to expose
the gobject api to lisp.

>
>> Some questions:
>> - The original xwidgets were designed to allow several views to the
>> same
>> xwidget using gtk reflection, as you describe. When I tried using
>> separate gtk controllers on screen, they didnt sync very well. Did
>> you
>> try this and did it work well?
>>
>> When I worked on this I tried several different gtk components such
>> as
>> sliders etc. Some of the worked flaky, some worked rather well. What
>> is
>> you current experience trying different components?
>
> So far I've started pretty simple and just looked at buttons and
> sliders, where each view gets it's own instance of the widget (so no
> offscreen rendering). The synchronizing of state is all handled by
> connecting each view's widget to a common signal handler which then
> updates the state of all the other views. So far the slider appears to
> sync its state pretty well across the views, but I have yet to look at
> more involved widgets.

BTW did you look at the widget that allows embedding of other
applications? I had some success embedding emacs inside emacs using this
widget, and maybe it has evolved during the years.

>> Anyway, thanks again for having a look at this!
>
> Of course!
>
-- 
Joakim Verona
joakim@verona.se

Re: Rethinking the design of xwidgets

[Eli Zaretskii]

> From: "Basil L. Contovounesios" <contovob@tcd.ie>
> Cc: Akira Kyle <ak@akirakyle.com>,  emacs-devel@gnu.org
> Date: Tue, 13 Oct 2020 19:42:05 +0100
> 
> Not sure if this fits the bill, but I was hoping to eventually look into
> whether Emacs+Cairo could render PDFs using the GLib API of the GPLed
> Poppler library.

If Poppler could produce pixmaps like other image libraries (perhaps
by using its own cairo layer), then this should be easy, I think.

Re: Rethinking the design of xwidgets

[Eli Zaretskii]

> From: Qiantan Hong <qhong@mit.edu>
> CC: Eli Zaretskii <eliz@gnu.org>, emacs-devel <emacs-devel@gnu.org>
> Date: Tue, 13 Oct 2020 18:44:20 +0000
> 
> I agree that displaying a live updating rectangle/slider following the mouse is near impossible.

I'd suggest to try that before reaching such conclusions.  Most
platforms support double-buffering nowadays, so the results might
surprise.

Re: Rethinking the design of xwidgets

[Akira Kyle]

On Tue, Oct 13, 2020 at 11:55 AM, Eli Zaretskii <eliz@gnu.org> 
wrote:

> I'll need to read up on those two topics, but in general, as 
> someone
> already pointed out, the Emacs display of images supports
> interactivity by letting you call functions triggered by mouse
> gestures on the varipous spots on the image.  If such a function
> produces a zoomed-in/out image and displays it instead of the 
> original
> one, wouldn't that fulfill the need?  E.g., we already support
> animated GIF images and zoom-in/out by mouse scroll, so it is
> inaccurate to say that we can only display static images.

Can actions be triggered on a click and drag as one would do to 
pan within a plot?

> This use case doesn't necessarily call for the same solution, 
> but I
> could think of a differently designed pdf-viewer that didn't 
> need to
> rasterize each page separately.  Again, the mouse-sensitivity 
> feature
> of our image display could perhaps be used for something like 
> that.  I
> reckon the hardest problem is to find an engine, be it a library 
> or a
> program, that could convert PDF into something Emacs can 
> display.

As far as I know there are really only two FLOSS PDF libraries out 
there: poppler and muPDF. Between them poppler supports more PDF 
features and seems to be more widely used. I agree this isn't 
really as compelling of a use case for xwidgets, but I seems like 
it would be easier to achieve better PDF rendering via xwidgets. I 
don't think it's about needing to rasterize each page separately, 
its more about directly rendering the rasterized image without 
having to copy it around a bunch or convert it to different 
formats. I suppose the image rendering from an elisp string could 
be made more performant or Emacs could handle pdf display natively 
but that would require adding poppler as a dependency.

> E.g., the kludge in dispnew.c around line 4365.  It disables one 
> of
> the most important redisplay optimizations in Emacs, once you 
> build
> with xwidgets enabled.

Ah, I had briefly looked at that but I wasn't sure how important 
it was. I may be able to make some progress on that but it's hard 
to understand the context for that function. It seems like that 
function is responsible for an optimization that avoids redrawing 
glyphs if the window is being scrolled by copying them, however I 
couldn't find where or how the glyphs are actually copied?

>> For example, consider svg support, which is my preferred image 
>> format. As far is I can tell, currently Emacs rasterizes svgs 
>> itself before ultimately drawing them on a cairo context.
>
> Do we actually do that?  We use librsvg, don't we?
>
> Or maybe you are talking about the generic layer of image 
> handling,
> which basically needs a bitmap/pixmap?

Yes this is what I'm talking about. In svg_load_image librsvg is 
used to end up with the rasterized svg in an Emacs_Pix_Container 
which then goes through a song and dance of pixel level color and 
transparency transformations to all eventually end back up as a 
cairo surface. Unless I'm missing something it seems like all this 
extra complexity and copying around is just for the sake of 
maintaining Emacs's own image format which was established long 
before HiDPI, deep color depth displays and libraries like cairo 
that do the hard work of transforming and rendering onto the 
glass.

> In any case, this seems to be a much more fundamental issue of 
> how we
> display images in Emacs.  Which is not necessarily the same as 
> the
> issue you raised originally.

It's at least related since the integration xwidgets into Emacs' 
display was originally modeled off of the way Emacs' displays 
images, however I see it as potentially being a simpler and more 
flexible alternative to the way Emacs currently displays images.

> The problem with that is that handling such "widgets" in our 
> display
> engine is entirely non-trivial, as the xwidgets code clearly 
> shows.
> There are basic problems in displaying xwidgets that from my POV 
> are
> left unsolved, and I'm not sure we will be able to find 
> satisfactory
> solutions for them.  The display engine has several requirements 
> that
> aren't easy to support with widgets that want to manage 
> themselves.

I think it would be good to try to either resolve such issues or 
decide that Emacs would be better off without xwidgets and instead 
focus on enhancing its existing rich display capabilities because 
as it is, the xwidgets code seems to have been lingering in this 
experimental state for a decade now. I think many like myself are 
drawn by its promise of being able to rendering complex dynamic 
content using existing optimized toolkits. There is of course the 
fact that in its current state on master the only thing it does do 
is embed a webkit browser in an Emacs window. A feature many would 
like to see enhanced while many others have voiced both security, 
privacy, and freedom concerns with such a feature. I'd just like 
to see a plan on where the community thinks xwidgets should go and 
work from there since it's been experimental for far too long.

> I see where you are coming from with the idea of dynamic 
> modules, but
> I'm not sure your assumption that it will make things easier for
> implementing "display elements" that have their own behavior
> independent of Emacs -- I'm not sure this assumption is indeed 
> true.
> The Emacs display code has a deceptively simple main design idea 
> and
> execution concept, and as long as one thinks this is all there 
> is to
> it, it could indeed seem that adding such widgets should be 
> almost
> trivial.  But the reality is that there are a lot of details to 
> go
> with the main idea, and it's the details that make implementing 
> this
> stuff more difficult than one thinks first.

Thank you for taking the time to talk through this with me. I'm 
seeing more and more that when it comes to the C portion of Emacs' 
code, nothing is ever quite so straightforward and it's difficult 
to understand all the intricacies due to the sheer amount of code.

> I'm not trying to scare you, mind you, just point out that the 
> ideas
> you have should probably be talked through in mode detail, to 
> see that
> they are workable.  For example, did you study the various
> window-redisplay optimizations, and if so, did you think how the
> widgets will adapt to them?

I'm just starting to dig my teeth into some of that. However I'm 
wondering if anyone has more recently tried to quantify the 
improvements the various redisplay optimizations make to both 
actual and perceived render time of Emacs. For example the comment 
on the scrolling_window function in dispnew.c that you pointed me 
too references that it is implementing an algorithm from 1978, 
perhaps it would be worth exploring if there are any algorithmic 
advances? It strikes me that perhaps some optimizations may no 
longer be as necessary on modern hardware or on GUI displays as 
they were when running emacs in a VT on 20 year 
hardware. Conversely there may be further optimization that could 
be done by taking advantage of more recent hardware advances such 
as offloading to a GPU or utilizing vectorized instructions. If 
there's one think I've taken away from my CS education, its that 
indeed "premature optimization is the root of all evil". I know 
Emacs prides itself on its broad portability, however when it 
comes to xwidgets we already know such a feature is restricted to 
running on the hardware that gtk3 expects. Anyways I know these 
are mostly lofty thoughts but I'd like to continue to see Emacs 
evolve.

Re: Rethinking the design of xwidgets

[Akira Kyle]

On Tue, Oct 13, 2020 at 12:44 PM, Qiantan Hong <qhong@mit.edu> 
wrote:

> I agree that displaying a live updating rectangle/slider 
> following the mouse is near impossible.
>
> For selecting region, supporting two mouse click should be 
> totally doable. For selecting
> data series, read-from-minibuffer together with user’s favorite 
> completion/fuzzy search package
> should be suffice. Those are discrete event. But do we really 
> need continuous event that
> much? E.g. what useful information is exchanged between 
> computer/user when drawing a 
> rectangle following user’s mouse? It seems to me that only the 
> starting and ending position
> matters. More here 
> http://cat-v.mit.edu/2019-10-15-mouse-considered-harmful.html

I totally agree that "Mouse considered harmful for most daily 
usage" however selecting a region is definitely one in which 
having the region highlighted as one drags the mouse is 
"continuous 2D data". This actually drives me crazy when I use 
pdf-view-set-slice-from-bounding-box in pdf-tools. As I'm always 
on my laptop, getting the bounding box of the pdf content as tight 
as possible is important but its hard to judge the full extent of 
the rectangle I made with just two points.

> However, if everyone decides that every event mouse spits out 
> matters (which is a lot),
> then it is the case that image display won’t work good enough.

Wouldn't it be fun to be able to free form draw inside emacs? :)

Re: Rethinking the design of xwidgets

[Akira Kyle]

On Tue, Oct 13, 2020 at 12:45 PM, Eli Zaretskii <eliz@gnu.org> 
wrote:

>> For example is it possible to have a live updating rectangle
>> displayed as one drags the mouse to select a region to inspect?
>
> Can't that be done with a mask on the image?

Perhaps, I have yet to try

>> Is it possible to display a slider which can be dragged by the 
>> mouse
>> to update values on the plot or have a drop down menu to select 
>> a
>> data series?
>
> You'd need to regenerate the plot with the updated values.

What if the computation that generates the plot is expensive? Yes 
one could cache the actual points that the plot draws and update 
based on that, which is what matplotlib already does. I think my 
point is more that a gtk widget already does all of this, so 
rather than reimplementing such functionality between python and 
elisp which will only be useful for one specific python plotting 
library, why not have the ability to display the dynamic widget of 
any given plotting library?

>> It seems it may also be difficult to find a way to have elisp
>> interact with the matplotlib fast enough to not introduce 
>> noticable
>> delay as such actions update the view.
>
> And doing this via a module that is called by a widget won't 
> bump into
> the same problems?  After all, all the interaction with Emacs 
> goes
> through the same mechanism, no matter what the callback does.

I don't think so. In this case the user events don't have to go 
through elisp, they can go directly through the GUI toolkit to 
code behind it. My experience is that elisp is pretty poor at 
handling real time events. Perhaps in part due to its cooperative 
threading model.

Re: Rethinking the design of xwidgets

[Aiko Kyle]

On Tue, Oct 13, 2020 at 12:36 PM, Tomas Hlavaty <tom@logand.com> 
wrote:

> some kind of canvas was discussed in
> id:87zh9hrxfj.fsf@logand.c
> and
> id:83img4aegz.fsf@gnu.org
>
> It was discussed in context of a WYSIWYG editor features, and 
> console
> based pdf viewer emacs-frabuffer and pure elisp pdf generator 
> emacs-pdf.
>
> Eli suggested:
>
>    > If you want to do layout for PDF, I think one way forward 
>    > would be
>    > to implement a pdfterm.c "terminal" for Emacs, which 
>    > produces PDF
>    > like the existing *term.c backends do for supported display 
>    > types.
>
> that might be better than some kind of xwidget

I don't immediately see the connection of this proposal to 
mine. It seems like pdfterm would be a display backend like xterm 
that renders to PDF, however this is already possible using the 
x-export-frames command which uses cairo to render the emacs frame 
to. It seems like the hard part of making Emacs a WYSIWG editor 
isn't the rendering of the display to pdf, but rather all the 
features of a WISYWIG editor such as placement of arbitrary text 
boxes on screen.

> It is not clear to me, why a dynamic module would be better.
>
> For example, emacs-framebuffer can display svg images without 
> any
> dynamic module by using external program, because emacs-nox does 
> not
> have cairo or librsvg or NS toolkit.
>
> Ideally, there would be a canvas (e.g. pdfterm) providing 
> low-level
> drawing primitives (something like html canvas API) and the rest 
> built
> on top of that in elisp.

I wasn't previously aware of emacs-framebuffer. It's an 
interesting idea but I don't think it's very relevant to my 
goal. It doesn't appear that Emacs is very "aware" of the image 
with emacs-framebuffer. In otherwords the image won't move inline 
with the rest of buffer contents will it? The line that the image 
is on won't have the height of the image?

Re: Rethinking the design of xwidgets

[Corwin Brust]

Hi Aiko & Team Emacs!  Thanks for working on this.

On Tue, Oct 13, 2020 at 4:33 PM Aiko Kyle <aikokyle@gmail.com> wrote:
>
>
> On Tue, Oct 13, 2020 at 12:36 PM, Tomas Hlavaty <tom@logand.com>
> wrote:
>
> > some kind of canvas was discussed in
> > id:87zh9hrxfj.fsf@logand.c
> > and
> > id:83img4aegz.fsf@gnu.org
> >
> > It was discussed in context of a WYSIWYG editor features, and
> > console
> > based pdf viewer emacs-frabuffer and pure elisp pdf generator
> > emacs-pdf.

This sounds wonderful fwiw.

> > Eli suggested:
> >
> >    > If you want to do layout for PDF, I think one way forward
> >    > would be
> >    > to implement a pdfterm.c "terminal" for Emacs, which
> >    > produces PDF
> >    > like the existing *term.c backends do for supported display
> >    > types.
> >
> > that might be better than some kind of xwidget
>
> I don't immediately see the connection of this proposal to
> mine. It seems like pdfterm would be a display backend like xterm
> that renders to PDF, however this is already possible using the
> x-export-frames command which uses cairo to render the emacs frame
> to. It seems like the hard part of making Emacs a WYSIWG editor
> isn't the rendering of the display to pdf, but rather all the
> features of a WISYWIG editor such as placement of arbitrary text
> boxes on screen.

I wanted to offer a sample of a few things my pet project[0][1] has
been able to do with SVG updates.  While I won't get into our aims*,
at least for now, we are finding the updates following the mouse
sufficient that I'm continuing in this direction.  Nothing attempted
with drag nor using alpha transparency thus far but this seems like an
experient I'd be in a position to conduct if that may be interesting.
We're fairly happy with a multi click interface so far, probably given
the snap-to-point effect.

Here is a direct link to a recent animated GIF.   While the sources
here are very, very ugly in my estimation a clean implementation is
clearly possible.   Moreover, it would be of keen interest if such
things are available from core by the time our game works :)

_Interactive Drawing Demo_
  https://bru.st/i/hUfqzqy48w.gif

Perhaps less to topic and more as bonus bread-crumbs for someone
willfully curious, our project is aiming to store all user data,
including game sources (monsters, maps, characters, ...), in org-mode
documents.  So far this is working well; however, that design is both
extraneous and too tightly coupled with respect to a more serious
effort to replicate (or extricate) the interactive drawing
functionality here for more general purposes.  It was somewhat in this
vein, in fact, that I recently began extracting the "draw"
functionality from dm-map.el into dm-draw.el, which also omits some
"predicate" logic used to decide which parts to render out "on the
fly".   Finally in this vein, our demos do their own scaling (and
incur the associated performance cost), relating to our use-case of
sending unscaled "spoiler" free SVG data from the host to the player
clients.

> > It is not clear to me, why a dynamic module would be better.
> >
> > For example, emacs-framebuffer can display svg images without
> > any
> > dynamic module by using external program, because emacs-nox does
> > not
> > have cairo or librsvg or NS toolkit.
> >
> > Ideally, there would be a canvas (e.g. pdfterm) providing
> > low-level
> > drawing primitives (something like html canvas API) and the rest
> > built
> > on top of that in elisp.

Again fwiw, this is a nice fit in terms of direction for dungeon-mode
- we are already rendering SVG, involving manufacturing DOM nodes,
which seems to make for efficient processing and may also mesh nicely
with unrelated work such as around an ebook reader feature iiuc.

We took a hard-parts first mentality and success creating a
progressively rendering interface to display game maps has led us to
try creating an interactive, visual editor for the maps.  I've
submitted some talks related to the project; if some of these are
accepted, and if there are any questions from this group, I would be
happy to try working in responses to them, so that more show-and-tell
experience is possible and recorded for whatever use it could be down
the road.

Corwin

[0] dungeon-mode, overview: https://directory.fsf.org/wiki/dungeon-mode
[1] dungeon-mode, src, etc: https://git.savannah.nongnu.org/cgit/dungeon.git/
[3] "dm-sketch", Monday, 2020-10-11 @ oops-its-the-am:
https://bru.st/i/hUfqzqy48w.gif

Re: Rethinking the design of xwidgets

[Akira Kyle]

On Tue, Oct 13, 2020 at 01:06 PM, joakim@verona.se wrote:

> Yes, I think that sounds more realistic than my original plan to 
> expose
> the gobject api to lisp.

Thanks! I'm glad it's at least somewhat sane sounding.

> BTW did you look at the widget that allows embedding of other
> applications? I had some success embedding emacs inside emacs 
> using this
> widget, and maybe it has evolved during the years.

I assume you're referring to the XEMBED protocol which I haven't 
really looked at. I've actually been using the pgtk port [1] in 
developing this to ensure I'm not tying myself to X as so much of 
Emacs' display code has been as it evolved. Also there's some 
rendering issue [2] with xwidgets under X which causes a lot of 
flickering of the webkitgtk widget. I have yet to look into it, 
but it doesn't seem to affect pgtk or ns.

[1] https://github.com/masm11/emacs
[2] https://debbugs.gnu.org/cgi/bugreport.cgi?bug=41609

Re: Rethinking the design of xwidgets

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > That leads me to my following proposal to rethink the design of 
  > xwidgets. Given that Emacs now has support for dynamic modules and 
  > it is now enabled by default, I propose exposing an interface for 
  > dynamic modules to define custom xwidgets.

I think that is a good idea.  However, before we develop that,
we should recheck our defenses against nonfree dynamic modules.
Are they strong enough?

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Rethinking the design of xwidgets

[Akira Kyle]

On Tue, Oct 13, 2020 at 10:38 PM, Richard Stallman <rms@gnu.org> 
wrote:

>   > That leads me to my following proposal to rethink the design 
>   > of 
>   > xwidgets. Given that Emacs now has support for dynamic 
>   > modules and 
>   > it is now enabled by default, I propose exposing an 
>   > interface for 
>   > dynamic modules to define custom xwidgets.
>
> I think that is a good idea.  However, before we develop that,
> we should recheck our defenses against nonfree dynamic modules.
> Are they strong enough?

There is the GPL compatibility symbol (plugin_is_GPL_compatible) 
that must be exported in order for Emacs to load a dynamic 
module. I'm not a lawyer but I'd say there's no way someone could 
claim in a court that they weren't willfully violating the GPL if 
they wrote a nonfree dynamic module as they'd have to explicitly 
include such a symbol in their code and it would be literally 
written into the compiled library. I suppose the dynamic module 
documentation and header file (emacs-module.h) could have more 
explicit text explaining the purpose of that symbol and why a 
dynamic module must be GPL compatible. Beyond that, I can't think 
of anything further to protect against nonfree dynamic modules.

Re: Rethinking the design of xwidgets

[Tomas Hlavaty]

On Tue 13 Oct 2020 at 15:20, Aiko Kyle <aikokyle@gmail.com> wrote:
> I don't immediately see the connection of this proposal to 
> mine.

I am questioning your need for dynamic modules.

I am questioning your need for xwidgets.

Both have issues and don't go in the right direction of solving the real
problem.  Your proposal will be a niche solution not useable for many
emacs users.  You are adressing your specific problem with your specific
solution.

I am trying to say that the real problem is that there is a general need
for some kind of low-level drawing layer (some call it canvas, some
pdfterm, call it whatever you want).

Re: Rethinking the design of xwidgets

[Eli Zaretskii]

> From: Akira Kyle <ak@akirakyle.com>
> Cc: emacs-devel@gnu.org
> Date: Tue, 13 Oct 2020 14:37:31 -0600
> 
> > I'll need to read up on those two topics, but in general, as 
> > someone
> > already pointed out, the Emacs display of images supports
> > interactivity by letting you call functions triggered by mouse
> > gestures on the varipous spots on the image.  If such a function
> > produces a zoomed-in/out image and displays it instead of the 
> > original
> > one, wouldn't that fulfill the need?  E.g., we already support
> > animated GIF images and zoom-in/out by mouse scroll, so it is
> > inaccurate to say that we can only display static images.
> 
> Can actions be triggered on a click and drag as one would do to 
> pan within a plot?

I didn't have time to try that, but just looking at the code, I don't
see anything that makes the clocks on images special, so I'd expect
the same modifiers and drag-detection code to work.  And if it
doesn't, I seed no reason why we couldn't teach it to do so.

> As far as I know there are really only two FLOSS PDF libraries out 
> there: poppler and muPDF. Between them poppler supports more PDF 
> features and seems to be more widely used. I agree this isn't 
> really as compelling of a use case for xwidgets, but I seems like 
> it would be easier to achieve better PDF rendering via xwidgets.

I can see how that is tempting: the image handling is taken care of.
What I'm saying is that this advantage should be carefully weighed
against the disadvantages and difficulties in integrating such
self-managing objects into the Emacs display code.

> > E.g., the kludge in dispnew.c around line 4365.  It disables one
> > of the most important redisplay optimizations in Emacs, once you
> > build with xwidgets enabled.
> 
> Ah, I had briefly looked at that but I wasn't sure how important 
> it was. I may be able to make some progress on that but it's hard 
> to understand the context for that function. It seems like that 
> function is responsible for an optimization that avoids redrawing 
> glyphs if the window is being scrolled by copying them

Not only if the window is scrolled, but also if the previous display
can be converted into the current display by scrolling some of the
stuff that is already on the glass.  the algorithm is a basic
comparison of the current and desired display, similar to what the
Diff utility does.

> however I couldn't find where or how the glyphs are actually copied?

See the calls to rif->scroll_run_hook.

> > I'm not trying to scare you, mind you, just point out that the
> > ideas you have should probably be talked through in mode detail,
> > to see that they are workable.  For example, did you study the
> > various window-redisplay optimizations, and if so, did you think
> > how the widgets will adapt to them?
> 
> I'm just starting to dig my teeth into some of that. However I'm 
> wondering if anyone has more recently tried to quantify the 
> improvements the various redisplay optimizations make to both 
> actual and perceived render time of Emacs. For example the comment 
> on the scrolling_window function in dispnew.c that you pointed me 
> too references that it is implementing an algorithm from 1978, 
> perhaps it would be worth exploring if there are any algorithmic 
> advances?

I don't think that algorithm is less optimal nowadays than it was back
then; the really interesting question is how much it saves us when
compared to simply redisplaying all those lines.

> It strikes me that perhaps some optimizations may no longer be as
> necessary on modern hardware or on GUI displays as they were when
> running emacs in a VT on 20 year hardware.

Those are very good questions.  I'm not aware of any such
investigations since the current redisplay optimizations were
implemented around 20 years ago: at that time, Gerd Möllmann, who
developed the current display engine, did add optimizations one by one
until he got reasonable redisplay performance.

Measuring the speedups from each of the several optimizations is an
important job, but it's a large job.  For starters, one needs to come
up with a large enough and representative enough sample of redisplay
use cases, and that is not easy.  So I'd encourage you (or anyone
else, actually) to do this important job, but be aware that it could
take non-trivial time and effort.

> Conversely there may be further optimization that could be done by
> taking advantage of more recent hardware advances such as offloading
> to a GPU or utilizing vectorized instructions.

We try not to use machine-dependent code in Emacs, because that's a
maintenance burden, what with today's fast pace of chip development
and obsolescence.  Vectorization is generally left to optimizing
compilers, and relying on special hardware, such as GPU, is not
something we should depend on directly.  We should instead hope that
the GUI toolkits and display systems we use will do that for us.

> If there's one think I've taken away from my CS education, its that
> indeed "premature optimization is the root of all evil".

The optimizations we have today were definitely NOT premature when
they were introduced.  How much they are still needed today is indeed
an interesting and important question that still awaits the motivated
investigators.  I have only indirect evidence that some of the
optimizations still do a useful job: compare the redisplay performance
when linum-mode is turned on with the performance when the native
display-line-numbers-mode is used instead.  Also, we frequently hear
complaints about redisplay performance, even if you take away the
problems with long lines.

Re: Rethinking the design of xwidgets

[Eli Zaretskii]

> From: Akira Kyle <ak@akirakyle.com>
> Cc: Eli Zaretskii <eliz@gnu.org>, emacs-devel@gnu.org
> Date: Tue, 13 Oct 2020 14:52:22 -0600
> 
> Wouldn't it be fun to be able to free form draw inside emacs? :)

It's a long-time feature request.  It is non-trivial to implement,
because this must be somehow coordinated with the display engine, so
it doesn't overwrite some of the drawing.  Work in this area would be
very welcome.

Re: Rethinking the design of xwidgets

[Eli Zaretskii]

> From: Akira Kyle <ak@akirakyle.com>
> Cc: qhong@mit.edu, emacs-devel@gnu.org
> Date: Tue, 13 Oct 2020 15:00:18 -0600
> 
> >> Is it possible to display a slider which can be dragged by the
> >> mouse to update values on the plot or have a drop down menu to
> >> select a data series?
> >
> > You'd need to regenerate the plot with the updated values.
> 
> What if the computation that generates the plot is expensive?

We don't need to worry about that until we actually see it's
expensive.  No premature optimizations, remember?

> I think my point is more that a gtk widget already does all of this,
> so rather than reimplementing such functionality between python and
> elisp which will only be useful for one specific python plotting
> library, why not have the ability to display the dynamic widget of
> any given plotting library?

Yes, it's tempting to reuse functionality that someone else developed,
but let's not forget the flip side: the need to integrate such objects
into the Emacs display code.  If that integration is hard, or if it
imposes serious limitations on displaying buffers with such objects,
we need to consider the pros vs the cons.

> >> It seems it may also be difficult to find a way to have elisp
> >> interact with the matplotlib fast enough to not introduce
> >> noticable delay as such actions update the view.
> >
> > And doing this via a module that is called by a widget won't bump
> > into the same problems?  After all, all the interaction with Emacs
> > goes through the same mechanism, no matter what the callback does.
> 
> I don't think so. In this case the user events don't have to go 
> through elisp, they can go directly through the GUI toolkit to 
> code behind it.

How is this possible?  You will intentionally exempt these events from
the "normal" Emacs customizations, keymaps, etc.?  You believe users
will let us do that?

> My experience is that elisp is pretty poor at handling real time
> events.

The input events in Emacs are handled in C, not in Lisp.  The C code
is part of the Lisp machine, but it isn't in Lisp, at least not its
lower layers.

Re: Rethinking the design of xwidgets

[Eli Zaretskii]

> From: Tomas Hlavaty <tom@logand.com>
> Date: Wed, 14 Oct 2020 09:32:55 +0200
> 
> On Tue 13 Oct 2020 at 15:20, Aiko Kyle <aikokyle@gmail.com> wrote:
> > I don't immediately see the connection of this proposal to 
> > mine.
> 
> I am questioning your need for dynamic modules.
> 
> I am questioning your need for xwidgets.
> 
> Both have issues and don't go in the right direction of solving the real
> problem.  Your proposal will be a niche solution not useable for many
> emacs users.  You are adressing your specific problem with your specific
> solution.

That sounds somewhat harsh.  Let's try being kinder to people who come
up with ideas for improving Emacs, okay?

Re: Rethinking the design of xwidgets

[Arthur Miller]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Akira Kyle <ak@akirakyle.com>
>> Cc: emacs-devel@gnu.org
>> Date: Tue, 13 Oct 2020 14:37:31 -0600
>> 
>> > I'll need to read up on those two topics, but in general, as 
>> > someone
>> > already pointed out, the Emacs display of images supports
>> > interactivity by letting you call functions triggered by mouse
>> > gestures on the varipous spots on the image.  If such a function
>> > produces a zoomed-in/out image and displays it instead of the 
>> > original
>> > one, wouldn't that fulfill the need?  E.g., we already support
>> > animated GIF images and zoom-in/out by mouse scroll, so it is
>> > inaccurate to say that we can only display static images.
>> 
>> Can actions be triggered on a click and drag as one would do to 
>> pan within a plot?
>
> I didn't have time to try that, but just looking at the code, I don't
> see anything that makes the clocks on images special, so I'd expect
> the same modifiers and drag-detection code to work.  And if it
> doesn't, I seed no reason why we couldn't teach it to do so.
>
>> As far as I know there are really only two FLOSS PDF libraries out 
>> there: poppler and muPDF. Between them poppler supports more PDF 
>> features and seems to be more widely used. I agree this isn't 
>> really as compelling of a use case for xwidgets, but I seems like 
>> it would be easier to achieve better PDF rendering via xwidgets.
>
> I can see how that is tempting: the image handling is taken care of.
> What I'm saying is that this advantage should be carefully weighed
> against the disadvantages and difficulties in integrating such
> self-managing objects into the Emacs display code.
>
>> > E.g., the kludge in dispnew.c around line 4365.  It disables one
>> > of the most important redisplay optimizations in Emacs, once you
>> > build with xwidgets enabled.
>> 
>> Ah, I had briefly looked at that but I wasn't sure how important 
>> it was. I may be able to make some progress on that but it's hard 
>> to understand the context for that function. It seems like that 
>> function is responsible for an optimization that avoids redrawing 
>> glyphs if the window is being scrolled by copying them
>
> Not only if the window is scrolled, but also if the previous display
> can be converted into the current display by scrolling some of the
> stuff that is already on the glass.  the algorithm is a basic
> comparison of the current and desired display, similar to what the
> Diff utility does.
>
>> however I couldn't find where or how the glyphs are actually copied?
>
> See the calls to rif->scroll_run_hook.
>
>> > I'm not trying to scare you, mind you, just point out that the
>> > ideas you have should probably be talked through in mode detail,
>> > to see that they are workable.  For example, did you study the
>> > various window-redisplay optimizations, and if so, did you think
>> > how the widgets will adapt to them?
>> 
>> I'm just starting to dig my teeth into some of that. However I'm 
>> wondering if anyone has more recently tried to quantify the 
>> improvements the various redisplay optimizations make to both 
>> actual and perceived render time of Emacs. For example the comment 
>> on the scrolling_window function in dispnew.c that you pointed me 
>> too references that it is implementing an algorithm from 1978, 
>> perhaps it would be worth exploring if there are any algorithmic 
>> advances?
>
> I don't think that algorithm is less optimal nowadays than it was back
> then; the really interesting question is how much it saves us when
> compared to simply redisplaying all those lines.
>
>> It strikes me that perhaps some optimizations may no longer be as
>> necessary on modern hardware or on GUI displays as they were when
>> running emacs in a VT on 20 year hardware.
>
> Those are very good questions.  I'm not aware of any such
> investigations since the current redisplay optimizations were
> implemented around 20 years ago: at that time, Gerd Möllmann, who
> developed the current display engine, did add optimizations one by one
> until he got reasonable redisplay performance.

In case where there is a discret gfx card (i.e. Nvidia/AMD) it is
probably faster to send everything to GPU and ask it to render a 
giant texture and then use it as XWindow pixmap, or something similar
then to figure out on CPU all the stuff that should not be displayed.
But Emacs will maybe run on some slow devices (atmel Emacs anyone?), so
you probably don't want to ditch away all that disp stuff.

> Measuring the speedups from each of the several optimizations is an
> important job, but it's a large job.  For starters, one needs to come
> up with a large enough and representative enough sample of redisplay
> use cases, and that is not easy.  So I'd encourage you (or anyone
> else, actually) to do this important job, but be aware that it could
> take non-trivial time and effort.
>
>> Conversely there may be further optimization that could be done by
>> taking advantage of more recent hardware advances such as offloading
>> to a GPU or utilizing vectorized instructions.
>
> We try not to use machine-dependent code in Emacs, because that's a
> maintenance burden, what with today's fast pace of chip development
> and obsolescence.  Vectorization is generally left to optimizing
> compilers, and relying on special hardware, such as GPU, is not
> something we should depend on directly.  We should instead hope that
> the GUI toolkits and display systems we use will do that for us.
Would it be considered too machine-dependent if you rendered into
off-screen OpenGL surface as your rendering target? Should be avialable
on Nvidia/AMD/Intels gpus. With offscreen egl you would have relatively
OS independent code too; but that would require explicit linking to
proprietary libs. I mean X11/Win32 already does that when proprietary
drivers are installed, so Emacs (and any other software) "implicitly"
links to those, but I don't how it is viewed if you would to explicitly
when present.

You could maybe also expose egl-surface to elisp as an "image format"
and enable rendering to an image directly instead of going through
external process.

Re: Rethinking the design of xwidgets

[Eli Zaretskii]

> From: Arthur Miller <arthur.miller@live.com>
> Cc: Akira Kyle <ak@akirakyle.com>,  emacs-devel@gnu.org
> Date: Wed, 14 Oct 2020 17:04:07 +0200
> 
> In case where there is a discret gfx card (i.e. Nvidia/AMD) it is
> probably faster to send everything to GPU and ask it to render a 
> giant texture and then use it as XWindow pixmap, or something similar
> then to figure out on CPU all the stuff that should not be displayed.

I believe this should be the job of the GUI library we use (Xlib etc.)

> But Emacs will maybe run on some slow devices (atmel Emacs anyone?), so
> you probably don't want to ditch away all that disp stuff.

Different Emacs configurations could employ different redisplay
optimizations, if we know which ones are necessary when.

> > We try not to use machine-dependent code in Emacs, because that's a
> > maintenance burden, what with today's fast pace of chip development
> > and obsolescence.  Vectorization is generally left to optimizing
> > compilers, and relying on special hardware, such as GPU, is not
> > something we should depend on directly.  We should instead hope that
> > the GUI toolkits and display systems we use will do that for us.
> Would it be considered too machine-dependent if you rendered into
> off-screen OpenGL surface as your rendering target? Should be avialable
> on Nvidia/AMD/Intels gpus. With offscreen egl you would have relatively
> OS independent code too; but that would require explicit linking to
> proprietary libs. I mean X11/Win32 already does that when proprietary
> drivers are installed, so Emacs (and any other software) "implicitly"
> links to those, but I don't how it is viewed if you would to explicitly
> when present.

Why should we do that in our own code, as opposed to using some
higher-level library?

Re: Rethinking the design of xwidgets

[Tomas Hlavaty]

On Wed 14 Oct 2020 at 18:02, Eli Zaretskii <eliz@gnu.org> wrote:
> That sounds somewhat harsh.  Let's try being kinder to people who come
> up with ideas for improving Emacs, okay?

i was simply trying to be as clear as possible.  sorry if it sounded
harsh.  it wasn't suppose to be and it is not clear to me why you think
it sounded harsh.

Re: Rethinking the design of xwidgets

[Stefan Monnier]

>> I don't think that algorithm is less optimal nowadays than it was back
>> then; the really interesting question is how much it saves us when
>> compared to simply redisplaying all those lines.

Right.  The relative performance of different kinds of code has probably
changed significantly, so it's possible that it's faster to just redraw
the whole window than to try and figure out which part to redraw (and
even try and find some scrolling opportunities in there).
Then again, it's also quite possible that those optimizations are still
very much worthwhile.

> In case where there is a discret gfx card (i.e. Nvidia/AMD) it is
> probably faster to send everything to GPU and ask it to render a 
> giant texture and then use it as XWindow pixmap, or something similar
> then to figure out on CPU all the stuff that should not be displayed.

I think it's much harder than it sounds: most of the work needs access
to data structures that aren't friendly to GPUs, so the work of
providing data to a GPU in an appropriate form would probably not be
much more less in most cases than the drawing itself.

(talking-about-things-one-does-not-understand-mode 1)

But indeed, maybe we could split the "drawing" from the glyph matrices
to the glass into a first step that draws into a "pixmap matrix" and
a second step that draws from it onto the glass.  This should make it
unnecessary to try and "scroll" (parts of) the display since it should
be "just as fast" to copy from the pixmap matrix as it is to copy from
the current glass's content.

(talking-about-things-one-does-not-understand-mode -1)

I think the performance of the redisplay is by and large poorly
understood.  While there are known cases where people experience "slow
redisplay" it's usually very unclear what that means concretely.
In many cases this can be completely unrelated to the actual redisplay
code (e.g. the time is spent running some expensive code off of
`post-command-hook` or font-lock or younameit).


        Stefan

Re: Rethinking the design of xwidgets

[Stefan Monnier]

>> Wouldn't it be fun to be able to free form draw inside emacs? :)
> It's a long-time feature request.  It is non-trivial to implement,
> because this must be somehow coordinated with the display engine, so
> it doesn't overwrite some of the drawing.  Work in this area would be
> very welcome.

Indeed, it would be highly welcome.
I can see 2 ways to integrate it into Emacs:

- do something like the SVG images.  To the extent we already have SVG
  images, the potential gains here are gains in performance, mostly, but
  there could also be potential gains in being able to easily find the
  "object" closest to a click.

- offer a "transparent" canvas that is drawn *over* the normal text.
  I.e. have the normal redisplay draw a window's content into a pixmap
  rather than onto the glass, then draw the canvas into another pixmap,
  then combine the two pixmaps onto the glass.

To be most useful, the second option would probably want to have some
kind of access to information about the text that was drawn by the
normal redisplay (e.g. maybe something similar to `posn-at-x-y` and
`posn-at-point`), so you can move objects in the canvas according to the
position of particular text elements in the buffer.


        Stefan

Re: Rethinking the design of xwidgets

[Akira Kyle]

On Wed, Oct 14, 2020 at 08:33 AM, Eli Zaretskii <eliz@gnu.org> 
wrote:

>> Can actions be triggered on a click and drag as one would do to 
>> pan within a plot?
>
> I didn't have time to try that, but just looking at the code, I 
> don't
> see anything that makes the clocks on images special, so I'd 
> expect
> the same modifiers and drag-detection code to work.  And if it
> doesn't, I seed no reason why we couldn't teach it to do so.

Good to know.

>> As far as I know there are really only two FLOSS PDF libraries 
>> out 
>> there: poppler and muPDF. Between them poppler supports more 
>> PDF 
>> features and seems to be more widely used. I agree this isn't 
>> really as compelling of a use case for xwidgets, but I seems 
>> like 
>> it would be easier to achieve better PDF rendering via 
>> xwidgets.
>
> I can see how that is tempting: the image handling is taken care 
> of.
> What I'm saying is that this advantage should be carefully 
> weighed
> against the disadvantages and difficulties in integrating such
> self-managing objects into the Emacs display code.

OK. I'm still learning about the redisplay optimizations so 
hopefully I'll have a better idea of what those disadvantages are 
as I play with the existing way xwidgets works with redisplay.

>> > E.g., the kludge in dispnew.c around line 4365.  It disables 
>> > one
>> > of the most important redisplay optimizations in Emacs, once 
>> > you
>> > build with xwidgets enabled.
>> 
>> Ah, I had briefly looked at that but I wasn't sure how 
>> important 
>> it was. I may be able to make some progress on that but it's 
>> hard 
>> to understand the context for that function. It seems like that 
>> function is responsible for an optimization that avoids 
>> redrawing 
>> glyphs if the window is being scrolled by copying them
>
> Not only if the window is scrolled, but also if the previous 
> display
> can be converted into the current display by scrolling some of 
> the
> stuff that is already on the glass.  the algorithm is a basic
> comparison of the current and desired display, similar to what 
> the
> Diff utility does.
>
>> however I couldn't find where or how the glyphs are actually 
>> copied?
>
> See the calls to rif->scroll_run_hook.

Thanks, this helps!

>> I'm just starting to dig my teeth into some of that. However 
>> I'm 
>> wondering if anyone has more recently tried to quantify the 
>> improvements the various redisplay optimizations make to both 
>> actual and perceived render time of Emacs. For example the 
>> comment 
>> on the scrolling_window function in dispnew.c that you pointed 
>> me 
>> too references that it is implementing an algorithm from 1978, 
>> perhaps it would be worth exploring if there are any 
>> algorithmic 
>> advances?
>
> I don't think that algorithm is less optimal nowadays than it 
> was back
> then; the really interesting question is how much it saves us 
> when
> compared to simply redisplaying all those lines.

Yes that is the better question to ask first.

>> It strikes me that perhaps some optimizations may no longer be 
>> as
>> necessary on modern hardware or on GUI displays as they were 
>> when
>> running emacs in a VT on 20 year hardware.
>
> Those are very good questions.  I'm not aware of any such
> investigations since the current redisplay optimizations were
> implemented around 20 years ago: at that time, Gerd Möllmann, 
> who
> developed the current display engine, did add optimizations one 
> by one
> until he got reasonable redisplay performance.
>
> Measuring the speedups from each of the several optimizations is 
> an
> important job, but it's a large job.  For starters, one needs to 
> come
> up with a large enough and representative enough sample of 
> redisplay
> use cases, and that is not easy.  So I'd encourage you (or 
> anyone
> else, actually) to do this important job, but be aware that it 
> could
> take non-trivial time and effort.

I see some redisplay tests in tests/manual/redisplay-testsuite.el 
and scroll-tests.el which seems like it would be a starting point 
for such a study?

>> Conversely there may be further optimization that could be done 
>> by
>> taking advantage of more recent hardware advances such as 
>> offloading
>> to a GPU or utilizing vectorized instructions.
>
> We try not to use machine-dependent code in Emacs, because 
> that's a
> maintenance burden, what with today's fast pace of chip 
> development
> and obsolescence.  Vectorization is generally left to optimizing
> compilers, and relying on special hardware, such as GPU, is not
> something we should depend on directly.  We should instead hope 
> that
> the GUI toolkits and display systems we use will do that for us.

That's more what I meant.

>> If there's one think I've taken away from my CS education, its 
>> that
>> indeed "premature optimization is the root of all evil".
>
> The optimizations we have today were definitely NOT premature 
> when
> they were introduced.  How much they are still needed today is 
> indeed
> an interesting and important question that still awaits the 
> motivated
> investigators.  I have only indirect evidence that some of the
> optimizations still do a useful job: compare the redisplay 
> performance
> when linum-mode is turned on with the performance when the 
> native
> display-line-numbers-mode is used instead.  Also, we frequently 
> hear
> complaints about redisplay performance, even if you take away 
> the
> problems with long lines.

I am a user who frequently wishes redisplay was better which has 
partly motivated my interest in this. Between displaying long org 
mode documents with inline images and long lines of generated 
latex, I'll often see noticeable delay in basic buffer operations 
such as scrolling. This is also exacerbated by the fact that I'm 
doing all of this on a Pinebook Pro which is an arm aarch64 system 
comparable to the raspberry pi. In fact the only way Emacs has 
been generally usable for me on this laptop is with the 
native-comp branch.

Re: Rethinking the design of xwidgets

[Eli Zaretskii]

> From: Akira Kyle <ak@akirakyle.com>
> Cc: emacs-devel@gnu.org
> Date: Wed, 14 Oct 2020 12:07:38 -0600
> 
> >> It strikes me that perhaps some optimizations may no longer be as
> >> necessary on modern hardware or on GUI displays as they were when
> >> running emacs in a VT on 20 year hardware.
> >
> > Those are very good questions.  I'm not aware of any such
> > investigations since the current redisplay optimizations were
> > implemented around 20 years ago: at that time, Gerd Möllmann, who
> > developed the current display engine, did add optimizations one by
> > one until he got reasonable redisplay performance.
> >
> > Measuring the speedups from each of the several optimizations is
> > an important job, but it's a large job.  For starters, one needs
> > to come up with a large enough and representative enough sample of
> > redisplay use cases, and that is not easy.  So I'd encourage you
> > (or anyone else, actually) to do this important job, but be aware
> > that it could take non-trivial time and effort.
> 
> I see some redisplay tests in tests/manual/redisplay-testsuite.el 
> and scroll-tests.el which seems like it would be a starting point 
> for such a study?

No, not necessarily.  Those just test some tricky situations where
there were bugs at some point.

To find the set of tests to measure the effect of the optimizations,
I'd suggest the following procedure:

  . For each of the optimizations, find a situation where each that
    optimizations gets executed.  ("M-x trace-redisplay" can help
    here.)  Then measure the performance with and without the
    optimization (most of them can be inhibited by setting the
    relevant inhibit-try-* variables; for the couple that cannot, we
    can add more such variables).
    Note that some optimizations are in xdisp.c and some in dispnew.c.

  . Each test case found above should be run with and without
    variable-size fonts, with and without text properties (font-lock),
    and with and without lots of overlays.

  . Additional test cases can be constructed by benchmarking scrolling
    through a large buffer, again with and without the relevant
    optimizations.

I guess there's more, but the above can be a good starting point, and
will produce enough data to analyze the current situation.

> I am a user who frequently wishes redisplay was better which has
> partly motivated my interest in this. Between displaying long org
> mode documents with inline images and long lines of generated latex,
> I'll often see noticeable delay in basic buffer operations such as
> scrolling. This is also exacerbated by the fact that I'm doing all
> of this on a Pinebook Pro which is an arm aarch64 system comparable
> to the raspberry pi. In fact the only way Emacs has been generally
> usable for me on this laptop is with the native-comp branch.

It is possible that a significant portion of the percepted slowness is
due to Lisp that is run either via redisplay hooks or from
post-command-hook and suchlikes.  To see who is the culprit, profile
Emacs during such slow displays, and look at the percentage taken by
redisplay_internal and vertical-motion, as opposed to other functions.

Re: Rethinking the design of xwidgets

[Aiko Kyle]

Replying to both Stefan's and Eli's reply to Arthur

On Wed, Oct 14, 2020 at 09:04 AM, Arthur Miller 
<arthur.miller@live.com> wrote:

>> In case where there is a discret gfx card (i.e. Nvidia/AMD) it 
>> is
>> probably faster to send everything to GPU and ask it to render 
>> a 
>> giant texture and then use it as XWindow pixmap, or something 
>> similar
>> then to figure out on CPU all the stuff that should not be 
>> displayed.

On Wed, Oct 14, 2020 at 10:53 AM, Stefan Monnier 
<monnier@iro.umontreal.ca> wrote:

> I think it's much harder than it sounds: most of the work needs 
> access
> to data structures that aren't friendly to GPUs, so the work of
> providing data to a GPU in an appropriate form would probably 
> not be
> much more less in most cases than the drawing itself.
>
> (talking-about-things-one-does-not-understand-mode 1)
>
> But indeed, maybe we could split the "drawing" from the glyph 
> matrices
> to the glass into a first step that draws into a "pixmap matrix" 
> and
> a second step that draws from it onto the glass.  This should 
> make it
> unnecessary to try and "scroll" (parts of) the display since it 
> should
> be "just as fast" to copy from the pixmap matrix as it is to 
> copy from
> the current glass's content.
>
> (talking-about-things-one-does-not-understand-mode -1)
>
> I think the performance of the redisplay is by and large poorly
> understood.  While there are known cases where people experience 
> "slow
> redisplay" it's usually very unclear what that means concretely.
> In many cases this can be completely unrelated to the actual 
> redisplay
> code (e.g. the time is spent running some expensive code off of
> `post-command-hook` or font-lock or younameit).
>

On Wed, Oct 14, 2020 at 09:32 AM, Eli Zaretskii <eliz@gnu.org> 
wrote:

> I believe this should be the job of the GUI library we use (Xlib 
> etc.)


As someone who has done a little bit of GPU programming, I think 
Emacs should stay as far away from needing to care about the 
architecture and optimization for GPUs as possible. It is hard and 
very non-portable. I should have clarified in my prior email that 
I really meant that Emacs could take advantage of newer hardware 
like GPUs and other SIMD vectorization units not directly because 
it is a huge PITA to work with, but through appropriate high level 
libraries. I don't think Emacs should be in the business of trying 
to actually draw primitives like lines and rectangles onto the 
glass directly whether through the frambuffer or low level gpu 
primitives like opengl. It requires specific expertise to get 
right and do efficiently that I think is beyond the scope of Emacs 
development.

Since Emacs already has its own internal abstraction for actually 
doing the drawing through the redisplay_interface, my point was 
more that Emacs should gain hardware acceleration by ensuring the 
GUI toolkits it uses take advantage of such features. I think gtk 
4 will have even more integration with hardware acceleration, and 
Emacs is better served by making the redisplay_interface 
abstraction cleaner. The Emacs pgtk port I think is a big step in 
that direction [1] and I really hope they get it in a feature 
branch here soon. Someone in the remacs project even made an Emacs 
backend to Mozilla's webrender [2] which was created with hardware 
acceleration at its core. I think too much of Emacs' current 
display design is influenced by Xlib which wasn't designed with 
such hardware acceleration in mind and this is what is currently 
limiting Emacs taking advantage of the hardware acceleration 
advances in the toolkits it uses. I think my comment on another 
Email in this thread about the current way Emacs uses librsvg on 
gtk+cairo builds illustrates this point.

Improving the redisplay_interface abstraction might then allow the 
toolkit backend to disable certain redisplay optimizations that 
are unnecessary for good performance with that toolkit and that 
may be better handled inside the toolkit than inside Emacs.

[1] https://github.com/masm11/emacs/pull/23
[2] https://github.com/remacs/remacs/pull/1581

Re: Rethinking the design of xwidgets

[Akira Kyle]

On Wed, Oct 14, 2020 at 12:32 PM, Eli Zaretskii <eliz@gnu.org> 
wrote:

>> I see some redisplay tests in 
>> tests/manual/redisplay-testsuite.el 
>> and scroll-tests.el which seems like it would be a starting 
>> point 
>> for such a study?
>
> No, not necessarily.  Those just test some tricky situations 
> where
> there were bugs at some point.
>
> To find the set of tests to measure the effect of the 
> optimizations,
> I'd suggest the following procedure:
>
>   . For each of the optimizations, find a situation where each 
>   that
>     optimizations gets executed.  ("M-x trace-redisplay" can 
>     help
>     here.)  Then measure the performance with and without the
>     optimization (most of them can be inhibited by setting the
>     relevant inhibit-try-* variables; for the couple that 
>     cannot, we
>     can add more such variables).
>     Note that some optimizations are in xdisp.c and some in 
>     dispnew.c.
>
>   . Each test case found above should be run with and without
>     variable-size fonts, with and without text properties 
>     (font-lock),
>     and with and without lots of overlays.
>
>   . Additional test cases can be constructed by benchmarking 
>   scrolling
>     through a large buffer, again with and without the relevant
>     optimizations.
>
> I guess there's more, but the above can be a good starting 
> point, and
> will produce enough data to analyze the current situation.

Thanks for the outline of how to start on this project. I'll add 
it to my list of Emacs projects I'd like to work on whenever I 
have time to do so :)

>> I am a user who frequently wishes redisplay was better which 
>> has
>> partly motivated my interest in this. Between displaying long 
>> org
>> mode documents with inline images and long lines of generated 
>> latex,
>> I'll often see noticeable delay in basic buffer operations such 
>> as
>> scrolling. This is also exacerbated by the fact that I'm doing 
>> all
>> of this on a Pinebook Pro which is an arm aarch64 system 
>> comparable
>> to the raspberry pi. In fact the only way Emacs has been 
>> generally
>> usable for me on this laptop is with the native-comp branch.
>
> It is possible that a significant portion of the percepted 
> slowness is
> due to Lisp that is run either via redisplay hooks or from
> post-command-hook and suchlikes.  To see who is the culprit, 
> profile
> Emacs during such slow displays, and look at the percentage 
> taken by
> redisplay_internal and vertical-motion, as opposed to other 
> functions.

Sorry I should've been more precise. I'm sure much of the 
perceived slowness is due to lisp given how much native-comp 
improves the situation. My wishes for redisplay to be better are 
less performance related (expect for long lines which are are a 
real killer) and more functionality wise, hence the original 
proposal about xwidgets. I also think Emacs could handle scrolling 
variable height lines much better as when there are inline images 
in a buffer and generally handle being more aware about pixels 
rather than the character matrix positions that Emacs historically 
started out only caring about.

Re: Rethinking the design of xwidgets

[Akira Kyle]

On Tue, Oct 13, 2020 at 06:12 PM, Corwin Brust <corwin@bru.st> 
wrote:

> Hi Aiko & Team Emacs!  Thanks for working on this.
>
> On Tue, Oct 13, 2020 at 4:33 PM Aiko Kyle <aikokyle@gmail.com> 
> wrote:
>>
>>
>> On Tue, Oct 13, 2020 at 12:36 PM, Tomas Hlavaty 
>> <tom@logand.com>
>> wrote:
>>
>> > some kind of canvas was discussed in
>> > id:87zh9hrxfj.fsf@logand.c
>> > and
>> > id:83img4aegz.fsf@gnu.org
>> >
>> > It was discussed in context of a WYSIWYG editor features, and
>> > console
>> > based pdf viewer emacs-frabuffer and pure elisp pdf generator
>> > emacs-pdf.
>
> This sounds wonderful fwiw.
>
>> > Eli suggested:
>> >
>> >    > If you want to do layout for PDF, I think one way 
>> >    > forward
>> >    > would be
>> >    > to implement a pdfterm.c "terminal" for Emacs, which
>> >    > produces PDF
>> >    > like the existing *term.c backends do for supported 
>> >    > display
>> >    > types.
>> >
>> > that might be better than some kind of xwidget
>>
>> I don't immediately see the connection of this proposal to
>> mine. It seems like pdfterm would be a display backend like 
>> xterm
>> that renders to PDF, however this is already possible using the
>> x-export-frames command which uses cairo to render the emacs 
>> frame
>> to. It seems like the hard part of making Emacs a WYSIWG editor
>> isn't the rendering of the display to pdf, but rather all the
>> features of a WISYWIG editor such as placement of arbitrary 
>> text
>> boxes on screen.
>
> I wanted to offer a sample of a few things my pet project[0][1] 
> has
> been able to do with SVG updates.  While I won't get into our 
> aims*,
> at least for now, we are finding the updates following the mouse
> sufficient that I'm continuing in this direction.  Nothing 
> attempted
> with drag nor using alpha transparency thus far but this seems 
> like an
> experient I'd be in a position to conduct if that may be 
> interesting.
> We're fairly happy with a multi click interface so far, probably 
> given
> the snap-to-point effect.
>
> Here is a direct link to a recent animated GIF.   While the 
> sources
> here are very, very ugly in my estimation a clean implementation 
> is
> clearly possible.   Moreover, it would be of keen interest if 
> such
> things are available from core by the time our game works :)
>
> _Interactive Drawing Demo_
>   https://bru.st/i/hUfqzqy48w.gif
>
> Perhaps less to topic and more as bonus bread-crumbs for someone
> willfully curious, our project is aiming to store all user data,
> including game sources (monsters, maps, characters, ...), in 
> org-mode
> documents.  So far this is working well; however, that design is 
> both
> extraneous and too tightly coupled with respect to a more 
> serious
> effort to replicate (or extricate) the interactive drawing
> functionality here for more general purposes.  It was somewhat 
> in this
> vein, in fact, that I recently began extracting the "draw"
> functionality from dm-map.el into dm-draw.el, which also omits 
> some
> "predicate" logic used to decide which parts to render out "on 
> the
> fly".   Finally in this vein, our demos do their own scaling 
> (and
> incur the associated performance cost), relating to our use-case 
> of
> sending unscaled "spoiler" free SVG data from the host to the 
> player
> clients.
>
>> > It is not clear to me, why a dynamic module would be better.
>> >
>> > For example, emacs-framebuffer can display svg images without
>> > any
>> > dynamic module by using external program, because emacs-nox 
>> > does
>> > not
>> > have cairo or librsvg or NS toolkit.
>> >
>> > Ideally, there would be a canvas (e.g. pdfterm) providing
>> > low-level
>> > drawing primitives (something like html canvas API) and the 
>> > rest
>> > built
>> > on top of that in elisp.
>
> Again fwiw, this is a nice fit in terms of direction for 
> dungeon-mode
> - we are already rendering SVG, involving manufacturing DOM 
> nodes,
> which seems to make for efficient processing and may also mesh 
> nicely
> with unrelated work such as around an ebook reader feature iiuc.
>
> We took a hard-parts first mentality and success creating a
> progressively rendering interface to display game maps has led 
> us to
> try creating an interactive, visual editor for the maps.  I've
> submitted some talks related to the project; if some of these 
> are
> accepted, and if there are any questions from this group, I 
> would be
> happy to try working in responses to them, so that more 
> show-and-tell
> experience is possible and recorded for whatever use it could be 
> down
> the road.
>
> Corwin
>
> [0] dungeon-mode, overview: 
> https://directory.fsf.org/wiki/dungeon-mode
> [1] dungeon-mode, src, etc: 
> https://git.savannah.nongnu.org/cgit/dungeon.git/
> [3] "dm-sketch", Monday, 2020-10-11 @ oops-its-the-am:
> https://bru.st/i/hUfqzqy48w.gif

Thanks for sharing! This is really neat and it looks like its 
coming along nicely!

Re: Rethinking the design of xwidgets

[Akira Kyle]

On Wed, Oct 14, 2020 at 10:35 AM, Tomas Hlavaty <tom@logand.com> 
wrote:

> On Wed 14 Oct 2020 at 18:02, Eli Zaretskii <eliz@gnu.org> wrote:
>> That sounds somewhat harsh.  Let's try being kinder to people 
>> who come
>> up with ideas for improving Emacs, okay?
>
> i was simply trying to be as clear as possible.  sorry if it 
> sounded
> harsh.  it wasn't suppose to be and it is not clear to me why 
> you think
> it sounded harsh.

FWIW when a statement is phrased in terms of second person when it 
can just as readily be phrased in terms of third person or without 
any use of a pronoun, it can often come across as much more 
personal.

Re: Rethinking the design of xwidgets

[Mingde (Matthew) Zeng]

Hi Akira,

> Another use case I had in mind for my proposed improved xwidgets is
> better pdf rendering. I use pdf-tools constantly and it's one of the
> best pdf viewers out there (not just because it's in emacs). I'll
> sometimes even use it to give presentations so I never have to leave
> Emacs. However it's design inherently limits its rendering performance
> as it must rasterize each pdf page in a separate process then pipe
> that data to emacs to display as an image. Granted this could be
> avoided by turning the epdfinfo server process into a dynamic module,
> however it will still be a bit roundabout as the pdf page would have
> to first be rendered into a cairo context then rasterized then Emacs
> will draw that rasterized image onto another cairo context to get it
> onscreen. Having direct access to an onscreen cairo context would
> eliminate this and make it easier to support crisp, performant
> rendering onto HiDPI displays. Maybe I could even throw slide
> animations into my presentations this way!

You might've already tried it (since you mentioned in the main thread), but I still want to point out that currently the Emacs Application Framework's PDF-viewer [1] is probably your go-to choice (for now) for best PDF rendering in emacs, as it uses muPDF in the shadow.

From your main thread:
> Unfortunately EAF uses Qt to reparent a window
> into emacs and Python to do two way RPC with elisp. IMO it isn't a
> very emacsy way to enhance Emacs' display capabilites and is limited
> to just taking over a whole window rather than being embedded as part
> of a buffer (key to supporting dynamic content such as an interactive
> inline plot inside an org mode document). That leaves xwidgets which
> seems to be somewhat incomplete, unused, and neglected. I've been
> poking around the xwidgets code and I really like the concept, however
> the implementation seems to need some more thought and work.

As the maintainer of the EAF project, I have to point out that the original motivation for the existence of EAF was to enhance Emacs browser rendering, pdf rendering, and video rendering capabilites. The QGraphicsView was designed to span over the entire buffer to make it more emacsy, to fit the "one buffer, one major mode" idea. The choice was obvious when using EAF to view modern webpages, PDF, or videos. However EAF _doesn't have to_ stay this way, it is completely viable to embed the QGraphicsView inside a buffer.

I don't completely understand your opinion on why EAF is not very emacsy, I can see why it's not lispsy, but EAF is keyboard-oriented, highly extensible and customizable. It just didn't improve Emacs itself but decided to utilize other free software. The author of EAF, manateelazycat, thought that it would take quite a long time for Emacs to improve its graphical capabilites of rendering webpages or videos to the point of other modern software dedicated for these functionalities, and emacs-devel should prioritize on making Emacs a better text-based editing environment, then the EAF's job is simply extending Emacs to the world of modern graphics.

Nevertheless, I'm very interested to see how this xwidgets idea evolves.

Matthew

[1] Emacs Application Framework repo: https://github.com/manateelazycat/emacs-application-framework

--
Mingde (Matthew) Zeng

Re: Rethinking the design of xwidgets

[Tomas Hlavaty]

On Wed 14 Oct 2020 at 13:22, Akira Kyle <ak@akirakyle.com> wrote:
> FWIW when a statement is phrased in terms of second person when it 
> can just as readily be phrased in terms of third person or without 
> any use of a pronoun, it can often come across as much more 
> personal.

i see, thanks for clarification

Re: Rethinking the design of xwidgets

[Akira Kyle]

On Wed, Oct 14, 2020 at 01:24 PM, Mingde (Matthew) Zeng 
<matthewzmd@posteo.net> wrote:

> You might've already tried it (since you mentioned in the main 
> thread), but I still want to point out that currently the Emacs 
> Application Framework's PDF-viewer [1] is probably your go-to 
> choice (for now) for best PDF rendering in emacs, as it uses 
> muPDF in the shadow.

Unfortunately it doesn't seem like it doesn't currently support 
setting slices, inverting the pdf colors, or synctex.

> As the maintainer of the EAF project, I have to point out that 
> the original motivation for the existence of EAF was to enhance 
> Emacs browser rendering, pdf rendering, and video rendering 
> capabilites. The QGraphicsView was designed to span over the 
> entire buffer to make it more emacsy, to fit the "one buffer, 
> one major mode" idea. The choice was obvious when using EAF to 
> view modern webpages, PDF, or videos. However EAF _doesn't have 
> to_ stay this way, it is completely viable to embed the 
> QGraphicsView inside a buffer.

It's not clear to me how one would go about doing that? There's 
the fact that Qt is C++ based and that AFAIU it relies on the 
window manager to behave properly in order to be placed in the 
right place.


> I don't completely understand your opinion on why EAF is not 
> very emacsy, I can see why it's not lispsy, but EAF is 
> keyboard-oriented, highly extensible and customizable. It just 
> didn't improve Emacs itself but decided to utilize other free 
> software. The author of EAF, manateelazycat, thought that it 
> would take quite a long time for Emacs to improve its graphical 
> capabilites of rendering webpages or videos to the point of 
> other modern software dedicated for these functionalities, and 
> emacs-devel should prioritize on making Emacs a better 
> text-based editing environment, then the EAF's job is simply 
> extending Emacs to the world of modern graphics.

IMO the majority of what makes Emacs emacsy is being lispy, not 
necessarily that its keyboard-oriented. But yes I grant that EAF 
looks to be quite extensible.

I'm not a huge fan of Qt or C++ which has caused me to shy away 
from EAF. Also correct me if wrong, but it doesn't look like EAF 
has a path forward to running on wayland.

> Nevertheless, I'm very interested to see how this xwidgets idea 
> evolves.

Thanks, I really hope it ends up going somewhere.

Re: Rethinking the design of xwidgets

[Akira Kyle]

Sorry I missed this when replying to all the other subthreads 
earlier.

On Wed, Oct 14, 2020 at 08:44 AM, Eli Zaretskii <eliz@gnu.org> 
wrote:

>> >> Is it possible to display a slider which can be dragged by 
>> >> the
>> >> mouse to update values on the plot or have a drop down menu 
>> >> to
>> >> select a data series?
>> >
>> > You'd need to regenerate the plot with the updated values.
>> 
>> What if the computation that generates the plot is expensive?
>
> We don't need to worry about that until we actually see it's
> expensive.  No premature optimizations, remember?

I didn't mean the code that actually draws the plot the, but 
rather the computation that was run to generate the data that the 
plot is displaying as in principle that could be very expensive, 
which is why I was talking about having to cache the data that the 
plot is working from. 

>> I think my point is more that a gtk widget already does all of 
>> this,
>> so rather than reimplementing such functionality between python 
>> and
>> elisp which will only be useful for one specific python 
>> plotting
>> library, why not have the ability to display the dynamic widget 
>> of
>> any given plotting library?
>
> Yes, it's tempting to reuse functionality that someone else 
> developed,
> but let's not forget the flip side: the need to integrate such 
> objects
> into the Emacs display code.  If that integration is hard, or if 
> it
> imposes serious limitations on displaying buffers with such 
> objects,
> we need to consider the pros vs the cons.

Certainly. I just have yet to be convinced that the integration is 
so hard or would be so limited that it isn't worth the effort to 
try, but then again, I haven't been working on this for a very 
long time.

>> >> It seems it may also be difficult to find a way to have 
>> >> elisp
>> >> interact with the matplotlib fast enough to not introduce
>> >> noticable delay as such actions update the view.
>> >
>> > And doing this via a module that is called by a widget won't 
>> > bump
>> > into the same problems?  After all, all the interaction with 
>> > Emacs
>> > goes through the same mechanism, no matter what the callback 
>> > does.
>> 
>> I don't think so. In this case the user events don't have to go 
>> through elisp, they can go directly through the GUI toolkit to 
>> code behind it.
>
> How is this possible?  You will intentionally exempt these 
> events from
> the "normal" Emacs customizations, keymaps, etc.?  You believe 
> users
> will let us do that?

I should have clarified that I was referring only to actions with 
the mouse (or pen input if one wanted to draw) where latency is 
often far more noticeable, and there is less of a reason for Emacs 
to intercept the mouse actions. I think there would be very few, 
if any, instances where an xwidget would want to capture keyboard 
events. But I think the point is that each xwidget has both the 
power and responsibility to determine which events to handle 
itself and which to let Emacs handle and I imagine that decision 
will be informed by the which events for which latency is critical 
- such as drawing on a canvas.

>> My experience is that elisp is pretty poor at handling real 
>> time
>> events.
>
> The input events in Emacs are handled in C, not in Lisp.  The C 
> code
> is part of the Lisp machine, but it isn't in Lisp, at least not 
> its
> lower layers.

I meant that when the input event must be processed by elisp code 
which then subsequently does something with that event, it can 
potentially add considerable latency, and one must be careful to 
ensure that non-deterministic behavior such as garbage collection 
is controlled.

Re: Rethinking the design of xwidgets

[Stefan Monnier]

> I didn't mean the code that actually draws the plot the, but rather the
> computation that was run to generate the data that the plot is displaying as
> in principle that could be very expensive, which is why I was talking about
> having to cache the data that the plot is working from.

Sometimes, people who have worked on the code for a long time (like
myself) get "stuck in a rut" and fail to see how a different approach
could come with all kinds of strong advantages which makes it worthwhile
(despite the many downsides that we can foresee and which you luckily
can't see as clearly).

So I encourage you to try the approach you imagine, and see where it
leads you.  It will have significant shortcomings, most likely, but that
doesn't mean it can't be tremendously useful and successful.  Just be
aware of the fact that it may also turn out to be just not
very satisfactory.


        Stefan

RE: Rethinking the design of xwidgets

[arthur miller]

[-- Attachment #1: Type: text/plain, Size: 1573 bytes --]

Can you (or someone) help me test one thing: how do I hook my own image into Emacs image? Say that I have code that gives me back an XImage, how do I hook that image directly from memory into text property  so it gets displayed in a buffer? Either XImage or Xpixmap, does not matter. I was looking in image.c long time but I didn't figured it out at the time.



-------- Originalmeddelande --------
Från: Stefan Monnier <monnier@iro.umontreal.ca>
Datum: 2020-10-15 04:08 (GMT+01:00)
Till: Akira Kyle <ak@akirakyle.com>
Kopia: Eli Zaretskii <eliz@gnu.org>, qhong@mit.edu, emacs-devel@gnu.org
Ämne: Re: Rethinking the design of xwidgets

> I didn't mean the code that actually draws the plot the, but rather the
> computation that was run to generate the data that the plot is displaying as
> in principle that could be very expensive, which is why I was talking about
> having to cache the data that the plot is working from.

Sometimes, people who have worked on the code for a long time (like
myself) get "stuck in a rut" and fail to see how a different approach
could come with all kinds of strong advantages which makes it worthwhile
(despite the many downsides that we can foresee and which you luckily
can't see as clearly).

So I encourage you to try the approach you imagine, and see where it
leads you.  It will have significant shortcomings, most likely, but that
doesn't mean it can't be tremendously useful and successful.  Just be
aware of the fact that it may also turn out to be just not
very satisfactory.


        Stefan



[-- Attachment #2: Type: text/html, Size: 2377 bytes --]

Re: Rethinking the design of xwidgets

[Mingde (Matthew) Zeng]

> Unfortunately it doesn't seem like it doesn't currently support
> setting slices, inverting the pdf colors, or synctex.

You can invert PDF by pressing "i" or set it globally with `(eaf-setq eaf-pdf-dark-mode "true")` (details and other customization options can be found in [1]).
You're right about EAF doesn't support setting slices or synctex, but it could be implemented in the future. ;-)

> It's not clear to me how one would go about doing that? There's the
> fact that Qt is C++ based and that AFAIU it relies on the window
> manager to behave properly in order to be placed in the right place.

Where to draw the box of the graphic content (QGraphicsView) is entirely determined by a Lisp function. I can imagine a similar function that displays the box similar to lsp-ui popup doc, then IMO that's a way to embed something into Emacs buffer. However if we want the text to move out of the way so that it's not overlapping with QGraphicsView, it would require additional effort of modifying Emacs' code.
If you want, you can have a look at the architectural overview in the EAF wiki [2] that might help your understanding of EAF's design.

> I'm not a huge fan of Qt or C++ which has caused me to shy away from
> EAF. Also correct me if wrong, but it doesn't look like EAF has a path
> forward to running on wayland.

That's alright. EAF should support wayland now, though we do not have a lot of people who use EAF on wayland that come back and provide us feedback. I personally use KDE and don't have many chances to test EAF on Wayland or other DE/WM.


Matthew


[1] EAF Wiki - Customization:  https://github.com/manateelazycat/emacs-application-framework/wiki/Customization
[2] EAF Wiki - Architecture: https://github.com/manateelazycat/emacs-application-framework/wiki/Hacking

--
Mingde (Matthew) Zeng

Re: Rethinking the design of xwidgets

[Arthur Miller]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>> I don't think that algorithm is less optimal nowadays than it was back
>>> then; the really interesting question is how much it saves us when
>>> compared to simply redisplaying all those lines.
>
> Right.  The relative performance of different kinds of code has probably
> changed significantly, so it's possible that it's faster to just redraw
> the whole window than to try and figure out which part to redraw (and
> even try and find some scrolling opportunities in there).
> Then again, it's also quite possible that those optimizations are still
> very much worthwhile.

I have never really thought that the graphic performance is bottleneck in
Emacs either. I thinking more in terms of code simplification.

I have read the comment in disp.c and looked a bit into the display
routine at some time just for the fun, and I am impressed and as
currently wouldn't dare to touch anything there :-).

>> In case where there is a discret gfx card (i.e. Nvidia/AMD) it is
>> probably faster to send everything to GPU and ask it to render a 
>> giant texture and then use it as XWindow pixmap, or something similar
>> then to figure out on CPU all the stuff that should not be displayed.
>
> I think it's much harder than it sounds: most of the work needs access
> to data structures that aren't friendly to GPUs, so the work of
> providing data to a GPU in an appropriate form would probably not be
> much more less in most cases than the drawing itself.

As said above, I don't consider myself as well introduced to display
code, so consider this mostly as musings and curiousity: are you sure it
is that hard? What I think of is that display routine has to get this
data from lisp structures to draw to OS window anyway. While it is done,
instead of drawing to the screen, stash it into some GPU friendly
structure, i.e. "render" it into a VBO or something similar that can be
sent to gpu.

Maybe this does not need to be pulled-out; I am not sure about this; but
where and when is rendering state updated? As I understand all colours
and other rendering data for text goes through properties? Couldn't
that data be updated at same time in rendering structure when they are
updated as lisp property? By adding a hook to propertize or whatever
does that and then mark text as dirty? Just wild thoughts, don't take it
as concrete suggestion. Maybe I completely don't understand how it works.

I am not sure about the details how rendering could be done, that is
just a thought about the portion of getting state data for text from
lisp.

> (talking-about-things-one-does-not-understand-mode 1)

Nice markers you have :-)

> But indeed, maybe we could split the "drawing" from the glyph matrices
> to the glass into a first step that draws into a "pixmap matrix" and
> a second step that draws from it onto the glass.  This should make it
> unnecessary to try and "scroll" (parts of) the display since it should
> be "just as fast" to copy from the pixmap matrix as it is to copy from
> the current glass's content.

If you think in terms of "glass" or "layers" would it be possible to do
this:

Create a list of textures (to be used as layers) for a displayed buffer
(temp buffers and maybe some special buffers that are not displayed does
not need it). Render window background into one texture. Render buffer
text into another texture, render cursor into third texture and render
current line into last texture. Renderer could then composit those into
final display.

Then for example if you enabled GL, you could just move a textured
polygon up or down which should be extremely fast. Zoom in zoom out
would be adjusting moving volume (I suppose glOrtho for projection).

Rendering highlight for current line, selections etc would be composing
the texture with a colored polygon and doing some blending operations
possibly in shader to invert rendered text (I think), alternatively
render to some other texture and then compose over the old one; I don't
know just fast thinking.

If you are familiar with Qt, I think that is similar strategy they do
in their Qml. They have a scenegraph and they render each widget's state
(I think) to a texture and then just compose/display those textures at
render time. I am not sure though, it was long time I was reading about
how they do stuff.


By the way, about egl: nice thing with it is that it enables easy render
to texture, and also headless rendering (no X11 required) on gpu, but it
can also be used to render to an OS surface directly (a window).

Consider Emacs image: emacs inserts images as character properties and
display routine is aware of it's dimensions etc. If one could insert
image as a placeholder (say some imaginary format 'egl-image')
one could then render and image as a texture via egl with some lisp
commands; some graphics api not yet existing in elisp that would render
via opengl into egl texture, and then the image buffer, a pixel array as
floats, could be copied back to Emacs and displayed in image
placeholder. I don't know how to hook it up into image routines in
Emacs, was looking at it some time ago, but I didn't found it back then,
and didn't have time to go back. 

That is less efficient since it copies data back and forth between
cpu/gpu, but it would still be much more efficient then going through
external process and file system as Emacs does now. Maybe if gl context
is enable in entire frame, it would be possible to enable directly
render opengl content to the Emacs window portion that image placeholder
is. Some translation of pixel coordinate and other synchronisation would
be needed, but I don't think it is impossible or extremenly hard to do
from the opengl point of view. 

Maybe I don't understand all the pecularities and intricacies, I
myself, certainly don't know how to hook up all this into Emacs nor if
it was even possible to pull of; I am just theorycrafting atm :-).

>
> I think the performance of the redisplay is by and large poorly
> understood.  While there are known cases where people experience "slow
> redisplay" it's usually very unclear what that means concretely.
> In many cases this can be completely unrelated to the actual redisplay
> code (e.g. the time is spent running some expensive code off of
> `post-command-hook` or font-lock or younameit).

You are probably correct about that one.

Re: Rethinking the design of xwidgets

[Arthur Miller]

Aiko Kyle <aikokyle@gmail.com> writes:

> Replying to both Stefan's and Eli's reply to Arthur
>
> On Wed, Oct 14, 2020 at 09:04 AM, Arthur Miller <arthur.miller@live.com> wrote:
>
>>> In case where there is a discret gfx card (i.e. Nvidia/AMD) it is
>>> probably faster to send everything to GPU and ask it to render a giant
>>> texture and then use it as XWindow pixmap, or something similar
>>> then to figure out on CPU all the stuff that should not be displayed.
>
> On Wed, Oct 14, 2020 at 10:53 AM, Stefan Monnier <monnier@iro.umontreal.ca>
> wrote:
>
>> I think it's much harder than it sounds: most of the work needs access
>> to data structures that aren't friendly to GPUs, so the work of
>> providing data to a GPU in an appropriate form would probably not be
>> much more less in most cases than the drawing itself.
>>
>> (talking-about-things-one-does-not-understand-mode 1)
>>
>> But indeed, maybe we could split the "drawing" from the glyph matrices
>> to the glass into a first step that draws into a "pixmap matrix" and
>> a second step that draws from it onto the glass.  This should make it
>> unnecessary to try and "scroll" (parts of) the display since it should
>> be "just as fast" to copy from the pixmap matrix as it is to copy from
>> the current glass's content.
>>
>> (talking-about-things-one-does-not-understand-mode -1)
>>
>> I think the performance of the redisplay is by and large poorly
>> understood.  While there are known cases where people experience "slow
>> redisplay" it's usually very unclear what that means concretely.
>> In many cases this can be completely unrelated to the actual redisplay
>> code (e.g. the time is spent running some expensive code off of
>> `post-command-hook` or font-lock or younameit).
>>
>
> On Wed, Oct 14, 2020 at 09:32 AM, Eli Zaretskii <eliz@gnu.org> wrote:
>
>> I believe this should be the job of the GUI library we use (Xlib etc.)
>
>
> As someone who has done a little bit of GPU programming, I think Emacs should
> stay as far away from needing to care about the architecture and optimization
> for GPUs as possible. It is hard and very non-portable.
AFAIK OpenGL runs on more systems then Emacs, and is considered as
probably THE easiest graphics API out there (Turtle graphics probably
being easier :-)).

It is superseeded by Vulkan, but Vulkan is probably too much for an
application like Emacs in terms of complexity vs benefit. 

> I should have clarified
> in my prior email that I really meant that Emacs could take advantage of newer
> hardware like GPUs and other SIMD vectorization units not directly because it is
> a huge PITA to work with, but through appropriate high level libraries. I don't
> think Emacs should be in the business of trying to actually draw primitives like
> lines and rectangles onto the glass directly whether through the frambuffer or
> low level gpu primitives like opengl. It requires specific expertise to get
> right and do efficiently that I think is beyond the scope of Emacs development.
>
> Since Emacs already has its own internal abstraction for actually doing the
> drawing through the redisplay_interface, my point was more that Emacs should
> gain hardware acceleration by ensuring the GUI toolkits it uses take advantage
> of such features. I think gtk 4 will have even more integration with hardware
> acceleration,
In my world, Emacs would be better shying away from big GUI toolkits and
ginormous dependencies.

Emacs window is (logically) one giant glorified tty in principle; tucked
into an OS window and Emacs does it's own redisplay of the major os
window area, so GUI toolkits are really just placeholders for Emacs to
render. Buttons, menus, toolbars can all be implemented by Emacs buffers
rendered as Frames, or Windows. Does not blend very well into Gnome or
KDE world, but with some styling might be pretty too look at anyway.

As I udnerstand Emacs now does it's own
font and text rendering via Freetype , so question is if needs Gtk at
all more then being pretty on Gnome desktops? Maybe Cairo alone is not
bad, since it can be used as graphics API to render to and there are
also hardware accelerated versions via opengl bindings (look in Mesa source).

Re: Rethinking the design of xwidgets

[Arthur Miller]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Arthur Miller <arthur.miller@live.com>
>> Cc: Akira Kyle <ak@akirakyle.com>,  emacs-devel@gnu.org
>> Date: Wed, 14 Oct 2020 17:04:07 +0200
>> 
>> In case where there is a discret gfx card (i.e. Nvidia/AMD) it is
>> probably faster to send everything to GPU and ask it to render a 
>> giant texture and then use it as XWindow pixmap, or something similar
>> then to figure out on CPU all the stuff that should not be displayed.
>
> I believe this should be the job of the GUI library we use (Xlib etc.)

Yes, of course. EGL is your library :-)

I am thinking more in terms of performance and simplification of
implementation. Egl runs on major OS:s + android, and opengl is very
platform independent, so one code path (mostly) on all 3 major platforms
(hopefully). Also traditional Xlib and GDI does not hardware accelerate
everything or specially well. There is some article on msdn about
Direct2D and GDI for example, different driver paths etc. They have
exposed Direct2D for applications for performance reasons, as they state
GDI renders large portions through CPU in kernel mode, while DX2D is in
user space etc. I believe Xlib is even worse in that regard. Also future
of Xlib seems a tad bit uncertain. OpenGL via EGL could be Emacs's
"directx2d". Wayland uses egl and I am correct sdl uses it as well.
Maybe Emacs could use sdl to render too, but I think it would be
unnecessary expensive in term of complexity.

>> But Emacs will maybe run on some slow devices (atmel Emacs anyone?), so
>> you probably don't want to ditch away all that disp stuff.
>
> Different Emacs configurations could employ different redisplay
> optimizations, if we know which ones are necessary when.
>
>> > We try not to use machine-dependent code in Emacs, because that's a
>> > maintenance burden, what with today's fast pace of chip development
>> > and obsolescence.  Vectorization is generally left to optimizing
>> > compilers, and relying on special hardware, such as GPU, is not
>> > something we should depend on directly.  We should instead hope that
>> > the GUI toolkits and display systems we use will do that for us.
>> Would it be considered too machine-dependent if you rendered into
>> off-screen OpenGL surface as your rendering target? Should be avialable
>> on Nvidia/AMD/Intels gpus. With offscreen egl you would have relatively
>> OS independent code too; but that would require explicit linking to
>> proprietary libs. I mean X11/Win32 already does that when proprietary
>> drivers are installed, so Emacs (and any other software) "implicitly"
>> links to those, but I don't how it is viewed if you would to explicitly
>> when present.
>
> Why should we do that in our own code, as opposed to using some
> higher-level library?
Didn't you disliked proposal to use libCurl with answer that it would make
Emacs dependent on yet another library? :-)

Cairo is a nice 2D api, and has opengl backend too. I dont' know how
effective the backend is, but sure it could work. I dislike all the
dependencies it pulls in, but in a typical GNU/Linux system they are
anyway installed. Can you divorce it from Gtk requirement? Or it is
already possible to use Cairo in Emacs without rest of Gtk?

I proposed egl, as a platform independent surface drawing that enables
use of OpenGL. Someone might one day create a scene graph (isnt't it
just a list of lists) and maybe do some cool stuff in ELisp that we are
not really thinking of now.

Re: Rethinking the design of xwidgets

[Eli Zaretskii]

> From: arthur miller <arthur.miller@live.com>
> CC: Eli Zaretskii <eliz@gnu.org>, "qhong@mit.edu" <qhong@mit.edu>,
> 	"emacs-devel@gnu.org" <emacs-devel@gnu.org>
> Date: Thu, 15 Oct 2020 03:35:23 +0000
> 
> Can you (or someone) help me test one thing: how do I hook my own image into Emacs image? Say that I
> have code that gives me back an XImage, how do I hook that image directly from memory into text property 
> so it gets displayed in a buffer? Either XImage or Xpixmap, does not matter. I was looking in image.c long
> time but I didn't figured it out at the time.  

I'm not sure I understand the difficulty.  You are ware that
create-image can accept the image data directly, not as a file to
read?

Re: Rethinking the design of xwidgets

[Arthur Miller]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: arthur miller <arthur.miller@live.com>
>> CC: Eli Zaretskii <eliz@gnu.org>, "qhong@mit.edu" <qhong@mit.edu>,
>> 	"emacs-devel@gnu.org" <emacs-devel@gnu.org>
>> Date: Thu, 15 Oct 2020 03:35:23 +0000
>> 
>> Can you (or someone) help me test one thing: how do I hook my own image into Emacs image? Say that I
>> have code that gives me back an XImage, how do I hook that image directly from memory into text property 
>> so it gets displayed in a buffer? Either XImage or Xpixmap, does not matter. I was looking in image.c long
>> time but I didn't figured it out at the time.  
>
> I'm not sure I understand the difficulty.  You are ware that
> create-image can accept the image data directly, not as a file to
> read?
No I wasn't aware of it. I am not sure how I should pass an pixel array to
create-image & co.

I am just glancing at image.c and looking at how xpm format is
implemented.

Re: Rethinking the design of xwidgets

[Eli Zaretskii]

> From: Arthur Miller <arthur.miller@live.com>
> Cc: monnier@iro.umontreal.ca,  ak@akirakyle.com,  qhong@mit.edu,
>   emacs-devel@gnu.org
> Date: Thu, 15 Oct 2020 17:20:17 +0200
> 
> > I'm not sure I understand the difficulty.  You are ware that
> > create-image can accept the image data directly, not as a file to
> > read?
> No I wasn't aware of it. I am not sure how I should pass an pixel array to
> create-image & co.

If it's a bitmap, it should be already possible.

If it's something else, I'm afraid I don't understand what you are
trying to do, and in what language.

Re: Rethinking the design of xwidgets

[Akira Kyle]

On Thu, Oct 15, 2020 at 06:48 AM, Arthur Miller 
<arthur.miller@live.com> wrote:

>> As someone who has done a little bit of GPU programming, I 
>> think Emacs should
>> stay as far away from needing to care about the architecture 
>> and optimization
>> for GPUs as possible. It is hard and very non-portable.
> AFAIK OpenGL runs on more systems then Emacs, and is considered 
> as
> probably THE easiest graphics API out there (Turtle graphics 
> probably
> being easier :-)).
>
> It is superseeded by Vulkan, but Vulkan is probably too much for 
> an
> application like Emacs in terms of complexity vs benefit. 

Just a disclaimer that I haven't actually worked with OpenGL, my 
gpu experience was centered to compute instead of graphics tasks, 
so take everything I say with that grain of salt. Just because 
OpenGL is very cross platform, doesn't means its the preferred 
graphics API on a given platform. Higher level graphics APIs will 
often target some subset of OpenGL, Vulcan, Metal, and Direct3D 
depending on what platforms they want to support and what gives 
them the best performance.  I think Apple would love to kick 
opengl out of its ecosystem and force everyone to use Metal and 
AFAIK already has done so on iOS since they now control the 
physical GPU implementation with their own custom 
silicon. Furthermore, OpenGL being a low level graphics API, 
doesn't try to make decisions for you about the most efficient way 
to render what you have. Depending on the size and number of 
objects you want to draw and the resolution of the screen you want 
them draw on, some task will be better done on the CPU while 
others will be better served on a GPU. I don't think Emacs should 
be trying to figure that out since it can be difficult to do right 
and requires good performance benchmarking abilities.

>> I should have clarified
>> in my prior email that I really meant that Emacs could take 
>> advantage of newer
>> hardware like GPUs and other SIMD vectorization units not 
>> directly because it is
>> a huge PITA to work with, but through appropriate high level 
>> libraries. I don't
>> think Emacs should be in the business of trying to actually 
>> draw primitives like
>> lines and rectangles onto the glass directly whether through 
>> the frambuffer or
>> low level gpu primitives like opengl. It requires specific 
>> expertise to get
>> right and do efficiently that I think is beyond the scope of 
>> Emacs development.
>>
>> Since Emacs already has its own internal abstraction for 
>> actually doing the
>> drawing through the redisplay_interface, my point was more that 
>> Emacs should
>> gain hardware acceleration by ensuring the GUI toolkits it uses 
>> take advantage
>> of such features. I think gtk 4 will have even more integration 
>> with hardware
>> acceleration,
> In my world, Emacs would be better shying away from big GUI 
> toolkits and
> ginormous dependencies.
>
> Emacs window is (logically) one giant glorified tty in 
> principle; tucked
> into an OS window and Emacs does it's own redisplay of the major 
> os
> window area, so GUI toolkits are really just placeholders for 
> Emacs to
> render. Buttons, menus, toolbars can all be implemented by Emacs 
> buffers
> rendered as Frames, or Windows. Does not blend very well into 
> Gnome or
> KDE world, but with some styling might be pretty too look at 
> anyway.
>
> As I udnerstand Emacs now does it's own
> font and text rendering via Freetype , so question is if needs 
> Gtk at
> all more then being pretty on Gnome desktops? Maybe Cairo alone 
> is not
> bad, since it can be used as graphics API to render to and there 
> are
> also hardware accelerated versions via opengl bindings (look in 
> Mesa source).

My understanding of Emacs rendering is that it does rely on GUI 
toolkits more that you may think. I think the NS code renders via 
macOS's native font rendering, while on when on X --with-cairo, 
Emacs uses Freetype to render text. Perhaps someone may decide to 
implement text rendering using Pango to be more inline with the 
way GTK renders text.

Since Emacs doesn't even handle drawing its own rectangles but 
rather lets each toolkit draw them it its own native way I'm not 
sure it makes sense to move Emacs towards doing even more low 
level graphics stuff then it already does. In principle I think 
one could implement a EGL redisplay_interface which would then run 
across platforms, however it would potentially be a lot more code 
to implement everything from such level drawing primitives than 
the using the toolkits and it would loose the "native look" that 
the current, higher level approach maintains. I think the mac 
people tend to think their native text rendering is superior so it 
might be a hard sell to try to move Emacs to a cross-platform 
"rendered in-house" model. I think it would be really neat to 
explore such rendering backends for Emacs, I'm just not sure how 
much it would simplify things or bring tangible speed 
improvements. The webrender backend that someone made in remacs is 
in this vein, although I think even webrender is a much higher 
level graphics API since it sits atop OpenGL.

[1] https://github.com/remacs/remacs/pull/1581

RE: Rethinking the design of xwidgets

[arthur miller]

[-- Attachment #1: Type: text/plain, Size: 1714 bytes --]

I completely missed this one :-(. Sorry. My image approach is what you mean with svg-like one.

I think both approaches would be useful for different purposes.



-------- Originalmeddelande --------
Från: Stefan Monnier <monnier@iro.umontreal.ca>
Datum: 2020-10-14 19:02 (GMT+01:00)
Till: Eli Zaretskii <eliz@gnu.org>
Kopia: qhong@mit.edu, Akira Kyle <ak@akirakyle.com>, emacs-devel@gnu.org
Ämne: Re: Rethinking the design of xwidgets

>> Wouldn't it be fun to be able to free form draw inside emacs? :)
> It's a long-time feature request.  It is non-trivial to implement,
> because this must be somehow coordinated with the display engine, so
> it doesn't overwrite some of the drawing.  Work in this area would be
> very welcome.

Indeed, it would be highly welcome.
I can see 2 ways to integrate it into Emacs:

- do something like the SVG images.  To the extent we already have SVG
  images, the potential gains here are gains in performance, mostly, but
  there could also be potential gains in being able to easily find the
  "object" closest to a click.

- offer a "transparent" canvas that is drawn *over* the normal text.
  I.e. have the normal redisplay draw a window's content into a pixmap
  rather than onto the glass, then draw the canvas into another pixmap,
  then combine the two pixmaps onto the glass.

To be most useful, the second option would probably want to have some
kind of access to information about the text that was drawn by the
normal redisplay (e.g. maybe something similar to `posn-at-x-y` and
`posn-at-point`), so you can move objects in the canvas according to the
position of particular text elements in the buffer.


        Stefan



[-- Attachment #2: Type: text/html, Size: 2608 bytes --]

Re: Rethinking the design of xwidgets

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > In case where there is a discret gfx card (i.e. Nvidia/AMD) it is
  > probably faster to send everything to GPU and ask it to render a 
  > giant texture and then use it as XWindow pixmap, or something similar
  > then to figure out on CPU all the stuff that should not be displayed.

  > But Emacs will maybe run on some slow devices (atmel Emacs anyone?), so
  > you probably don't want to ditch away all that disp stuff.

The most important targets machines for the GNU system are machines
that don't require users to install any nonfree software.

Unfortunately, Nvidia and AMD GPUs require the system to download
nonfree firmware into them.  It follows that computers with that
hardware can't get certified for Respects Your Freedom -- and we who
prize freedom won't use them.

The development of GNU Emacs has to give first priority to those
computers -- not to the more powerful computers that require your
system load to contain nonfree software.

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Rethinking the design of xwidgets

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > FWIW when a statement is phrased in terms of second person when it 
  > can just as readily be phrased in terms of third person or without 
  > any use of a pronoun, it can often come across as much more 
  > personal.

That is an important point.  Thanks.

Other tips for expressing yourself in a kind way
can be found in https://gnu.org/philosophy/kind-communication.html.

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Rethinking the design of xwidgets

[Arthur Miller]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > In case where there is a discret gfx card (i.e. Nvidia/AMD) it is
>   > probably faster to send everything to GPU and ask it to render a 
>   > giant texture and then use it as XWindow pixmap, or something similar
>   > then to figure out on CPU all the stuff that should not be displayed.
>
>   > But Emacs will maybe run on some slow devices (atmel Emacs anyone?), so
>   > you probably don't want to ditch away all that disp stuff.
>
> The most important targets machines for the GNU system are machines
> that don't require users to install any nonfree software.
>
> Unfortunately, Nvidia and AMD GPUs require the system to download
> nonfree firmware into them.  It follows that computers with that
> hardware can't get certified for Respects Your Freedom -- and we who
> prize freedom won't use them.
>
> The development of GNU Emacs has to give first priority to those
> computers -- not to the more powerful computers that require your
> system load to contain nonfree software.
Of course, I completely understand and agre that  Emacs should
prioritize free systems, but in case of resource avialable why not use
it? I would personally love to see Emacs do graphics, I have bothered
Eli with the issue at few occasions I think, and I do believe if Elisp
via Emacs let one do some cool visual stuff, like programming some
graphics, and doing drawing in the editor would make it more useful and
more efficient. Also I see it as a one brick in making Emacs more
popular. I personally think it would be super cool to see Emacs do 3D
with Elisp, and peopel being able to use it as Processing (the language)
or Python to script some visualisations directly in Elisp. I think that
Lisp's nature as a languages suites graphical processing quite well,
isn't a DAGs just list of lists? Creating a DOM tre or a scene graph in
Elisp should be quite natural thing to program. Now when native compiler
is on the way, the speed would be quite good as well, probably not as
hand chrafted C/C++ code, but good enough for many purposes. Enabling
graphics with Elisp (and emacs) might be a strategical move to make
Emacs more popular. By the way I start to hate that phrase "make Emacs
more popular" :-).

In this case I would rather say it would make Emacs more efficient.
People are already doing this stuff; they visualise and render stuff in
other software and then display images in Emacs. Instead of adding cost
to go via external process and file system, Emacs could support a lot of
coolness more efficiently if there was some drawing support in it.



Optional read below: :-)    _____________________________________________

I completely understand your point and am aware of it even before you
mention it, and I agree with you. I have given thought to ideology of
GNU and your philosophy, several times. Let me first say that I do agree
with that philosophy; yes I believe you are completely correct, I value
my freedom, and I believe that freedom to see and even modify the source
code is the better way to make software and business in the very end.

It is just that most of computers nowdays will have some sort of
graphic cards, even small pocket devices, and any graphical application
running on those will be linked to non-proprietary drivers. People don't
always choose what is best for them, and best technical or moral
solution does not always win, politics and money are often bigger
motivators then the common or even personal good.

It is probably less interesting why I agree, and more what I might
object: I think it is very platonic view of the world. Looking at some
2500 thousands year back, and at the human stupidity spent till today's
time, I am very pessimistic about :-). Plato's view of the world was how
world *should* be, Aristotle's view was how world *is*. Guess whom
Greeks went to when they needed a new constitution. People don't always
choose what is best for them, unfortunately. If it was so there wouldn't
be wars, hunger and powerty. I believe also that one should live as one
preach, so definitely Emacs should be free and should work best on free
system; I do use free OS myself for all my computing needs minus mobile
phone :-(.

Just as I see that freedom is a moral imperative, I also see not wasting
resources as a moral imperative. You wouldn't open a window every day
and throw a $100 into the wind, would you? Hopefully you don't buy food
and eat just half of it and throw away the other half? Why would anyone
in clear mind wish to not save energy when it is possible? I understand
that energy is *cheap* and it is not directly visible on the bill, but
software running on thousands or millions of computers, possibly many
times during the computer lifetime should be energy efficient.

Re: Rethinking the design of xwidgets

[Arthur Miller]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > FWIW when a statement is phrased in terms of second person when it 
>   > can just as readily be phrased in terms of third person or without 
>   > any use of a pronoun, it can often come across as much more 
>   > personal.
>
> That is an important point.  Thanks.
>
> Other tips for expressing yourself in a kind way
> can be found in https://gnu.org/philosophy/kind-communication.html.
Indeed. In some leanguages the "you" is used as a third person; where
English speaking would say "one" and for example Swedish or Germans
would say "man", some other langauages would say "you" and sometimes it
can mess with how people's words are percieved. It is what we call a
"kulturkrock" in Swedish; "culture clash" in direct translation, but I
am not sure it is 'a term' in english.

Re: Rethinking the design of xwidgets

[Dmitry Gutov]

On 16.10.2020 07:02, Richard Stallman wrote:
> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
> 
>    > In case where there is a discret gfx card (i.e. Nvidia/AMD) it is
>    > probably faster to send everything to GPU and ask it to render a
>    > giant texture and then use it as XWindow pixmap, or something similar
>    > then to figure out on CPU all the stuff that should not be displayed.
> 
>    > But Emacs will maybe run on some slow devices (atmel Emacs anyone?), so
>    > you probably don't want to ditch away all that disp stuff.
> 
> The most important targets machines for the GNU system are machines
> that don't require users to install any nonfree software.
> 
> Unfortunately, Nvidia and AMD GPUs require the system to download
> nonfree firmware into them.  It follows that computers with that
> hardware can't get certified for Respects Your Freedom -- and we who
> prize freedom won't use them.
> 
> The development of GNU Emacs has to give first priority to those
> computers -- not to the more powerful computers that require your
> system load to contain nonfree software.

Even low-end GPUs like Intel's integrated ones or Mali GPUs on ARM can 
use OpenGL well these days.

The latest releases of Mozilla Firefox include a GPU-based rendering 
engine which works very well with my integrated Intel UHD 630 on a 4K 
monitor in full screen.

If Firefox can do that (with its multitude of supported graphical 
elements, animations and effects), Emacs certainly can do it too.

Re: Rethinking the design of xwidgets

[Dmitry Gutov]

On 16.10.2020 16:03, Arthur Miller wrote:
> It is just that most of computers nowdays will have some sort of
> graphic cards, even small pocket devices, and any graphical application
> running on those will be linked to non-proprietary drivers.

AMD has been offering fully-functioning Libre drivers for their GPUs on 
GNU/Linux PCs for 5 years now.

That leaves out device firmware, I suppose, but I would treat that as a 
separate (and an unfortunately ubiquitous) problem.

Re: Rethinking the design of xwidgets

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > That leaves out device firmware, I suppose, but I would treat that as a 
  > separate (and an unfortunately ubiquitous) problem.

The need to load nonfree firmware is the reason that some of us don't
use that hardware.

It is also the reason why the FSF won't give the Respects Your Freedom
certification to that hardware.

Those facts are the reason why Emacs development must give high
priority to the machines which don't need that firmware.



-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Rethinking the design of xwidgets

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Of course, I completely understand and agre that  Emacs should
  > prioritize free systems, but in case of resource avialable why not use
  > it?

Depending on what resource it is, and how it would be used, there may
or may not be a reason not to use it.

  > do believe if Elisp
  > via Emacs let one do some cool visual stuff, like programming some
  > graphics, and doing drawing in the editor would make it more useful and
  > more efficient. Also I see it as a one brick in making Emacs more
  > popular. I personally think it would be super cool to see Emacs do 3D
  > with Elisp,

Would this be a feature that works only if there is nonfree software
in your system load?

Implementing such a feature would turn Emacs into an inducement for
people to choose systems with nonfree software in them.  We must not
do that!  So we cannot include such a feature.

In other words, we must have a policy of not enhancing nonfree software.

However, the thinkpagds that we can use do have some acceleration.
If your feature works usably on an X200 or T400 type machine,
then it is ok.


-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Rethinking the design of xwidgets

[Kai Ma]

Arthur Miller <arthur.miller@live.com> writes:

> Cairo is a nice 2D api, and has opengl backend too. I dont' know how
> effective the backend is, but sure it could work. I dislike all the
> dependencies it pulls in, but in a typical GNU/Linux system they are
> anyway installed. Can you divorce it from Gtk requirement? Or it is
> already possible to use Cairo in Emacs without rest of Gtk?
>
> I proposed egl, as a platform independent surface drawing that enables
> use of OpenGL. Someone might one day create a scene graph (isnt't it
> just a list of lists) and maybe do some cool stuff in ELisp that we are
> not really thinking of now.

Maybe I'm diverting a little but I would like to add that if we have
OpenGL (or generally, hardware acceleration), we can enable more
"fashionable eye-candies".

I once worked on a shadowy text patch [1], which was discussed here some
months ago.  Though implementing the feature for Cocoa is exceptionally
easy, I couldn't get it fast enough on Linux systems with Cairo.  I
don't know if Cairo really has an OpenGL backend, but even if it has,
the blur API can't make use of it -- because Cairo doesn't even have
this API.

And yes, I do think Emacs for Linux deserves proper hardware
acceleration. (I don't really mind if it's OpenGL or something else, though.)

[1] https://github.com/ksqsf/emacsmoe/pull/1

Regards

Re: Rethinking the design of xwidgets

[Qiantan Hong]

[-- Attachment #1.1: Type: text/plain, Size: 512 bytes --]

> 
> 
> Maybe I'm diverting a little but I would like to add that if we have
> OpenGL (or generally, hardware acceleration), we can enable more
> "fashionable eye-candies”.
> 
I will be thrilled if that’s possible!

Just for your amusement, this is a short video clip of an experimental
music system I’m developing using Emacs as front end. I’m using
overlays to create visual feedback. Having blinking blocks is already
cool, but will be much cooler if I can draw arbitrary visual effect.


[-- Attachment #1.2: out.mp4 --]
[-- Type: video/mp4, Size: 1342240 bytes --]

[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 1858 bytes --]

Re: Rethinking the design of xwidgets

[Arthur Miller]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > Of course, I completely understand and agre that  Emacs should
>   > prioritize free systems, but in case of resource avialable why not use
>   > it?
>
> Depending on what resource it is, and how it would be used, there may
> or may not be a reason not to use it.
Well, yes of course like with everything.

>   > do believe if Elisp
>   > via Emacs let one do some cool visual stuff, like programming some
>   > graphics, and doing drawing in the editor would make it more useful and
>   > more efficient. Also I see it as a one brick in making Emacs more
>   > popular. I personally think it would be super cool to see Emacs do 3D
>   > with Elisp,
>
> Would this be a feature that works only if there is nonfree software
> in your system load?
No.

OpenGL implementation can be completely software based, or just
partially hardware accelerated. Such case is Mesa which is at least open
source, how much free it is I don't know, I am not so knowledgable about
interpretting licenses. 

> Implementing such a feature would turn Emacs into an inducement for
> people to choose systems with nonfree software in them.  We must not
> do that!  So we cannot include such a feature.
People still choose such hardware, regardless of Emacs runing on them or
not; and Emacs do run on them. It is also implicitly linked to a
proprietary driver if one is present via OS and windowing system. I just
think it would be very useful to people if they could use Elisp as they
use Python today, since I think Lisp is nicer language to work with.

> In other words, we must have a policy of not enhancing nonfree software.
Yes, I agree with you on that one definitely. But we would like to
enhance Emacs, not other non-free software.

> However, the thinkpagds that we can use do have some acceleration.
> If your feature works usably on an X200 or T400 type machine,
> then it is ok.
I have hard time to imagine a piece of machinery made since 2000 without
something proprietary, but I believe you guys looked up that. I don't
know myself; I don't have access to one.

Re: Rethinking the design of xwidgets

[Stefan Monnier]

> I have hard time to imagine a piece of machinery made since 2000 without
> something proprietary, but I believe you guys looked up that.

Nowadays the main issues for laptops/desktops are:

- Wifi cards: it's not hard to find wifi cards that can run on 100%
  Free Software up to 11n, but I don't know of any for 11ac.
- GPUs: AMD's require proprietary firmware, but AFAIK Intel's GPUs don't
  and I think the Nouveau driver for Nvidia doesn't either.  I believe
  the Free drivers for ARM's GPUs don't use any proprietary
  firmware either.
- BIOS: this is harder to find, but Coreboot/Libreboot/U-boot does
  support several machines from this century.

So, I think we might be better off now than we were before 2000
in this respect.  The problem is that this is a shrinking fraction of the
overall computer market, dominated by smartphones where the situation is
completely different.

I'm writing this on my brand new Librem mini, whose proprietary software
is limited to some blob that's supposed to neutralize Intel's "ME"
horror (oh and I need a proprietary firmware if I want to use the
Bluetooth functionality integrated on the wifi card).  And my home
server is a BananaPi which doesn't use any proprietary code AFAICT.


        Stefan "of course, that's only true until you start looking at
                the code running on our SSDs, etc..."

Re: Rethinking the design of xwidgets

[Arthur Miller]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> I have hard time to imagine a piece of machinery made since 2000 without
>> something proprietary, but I believe you guys looked up that.
>
> Nowadays the main issues for laptops/desktops are:
>
> - Wifi cards: it's not hard to find wifi cards that can run on 100%
>   Free Software up to 11n, but I don't know of any for 11ac.
> - GPUs: AMD's require proprietary firmware, but AFAIK Intel's GPUs don't
>   and I think the Nouveau driver for Nvidia doesn't either.  I believe
>   the Free drivers for ARM's GPUs don't use any proprietary
>   firmware either.
> - BIOS: this is harder to find, but Coreboot/Libreboot/U-boot does
>   support several machines from this century.
>
I was looking many times at Librem stuff, but I would pay additional 25%
for customs if I ordered one to here, so it is a bit on expensive side
for me.

> So, I think we might be better off now than we were before 2000
> in this respect.
Indeed. Unfortunately my Gigabyte mobo from 2016 does not support
neither one if it's network cards nor built in audio chipset in Linux.

> The problem is that this is a shrinking fraction of the
> overall computer market, dominated by smartphones where the situation is
> completely different.
Indeed; I am really curious what is Google hiding in their Android blob.
Wonder why Ubuntu's or Firefox phones never catched up. I don't know how
it goes for librem and their phones.

> I'm writing this on my brand new Librem mini, whose proprietary software
> is limited to some blob that's supposed to neutralize Intel's "ME"
> horror (oh and I need a proprietary firmware if I want to use the
> Bluetooth functionality integrated on the wifi card).  And my home
> server is a BananaPi which doesn't use any proprietary code AFAICT.
>
>
>         Stefan "of course, that's only true until you start looking at
>                 the code running on our SSDs, etc..."
Yepp; I was mostly thinking of harddrives, chipset drivers. But sure things
are getting better. I think the situation for free (as in GNU) or at least
open source is getting much better than it was.

Re: Rethinking the design of xwidgets

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > I have hard time to imagine a piece of machinery made since 2000 without
  > something proprietary, but I believe you guys looked up that. I don't
  > know myself; I don't have access to one.

I suggest loking up the machines that Libreboot supports.
People are working hard on this.
-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Rethinking the design of xwidgets

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > do believe if Elisp
  > via Emacs let one do some cool visual stuff, like programming some
  > graphics, and doing drawing in the editor would make it more useful and
  > more efficient. Also I see it as a one brick in making Emacs more
  > popular. I personally think it would be super cool to see Emacs do 3D
  > with Elisp,

Would this be a feature that works acceptably fast
only if there is nonfree software in your system load?

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Rethinking the design of xwidgets

[Dmitry Gutov]

On 18.10.2020 07:17, Richard Stallman wrote:
>    > do believe if Elisp
>    > via Emacs let one do some cool visual stuff, like programming some
>    > graphics, and doing drawing in the editor would make it more useful and
>    > more efficient. Also I see it as a one brick in making Emacs more
>    > popular. I personally think it would be super cool to see Emacs do 3D
>    > with Elisp,
> 
> Would this be a feature that works acceptably fast
> only if there is nonfree software in your system load?

Like I described, modern Intel GPUs are reasonable fast at OpenGL 
already. So not necessarily, no.

Re: Rethinking the design of xwidgets

[Arthur Miller]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > do believe if Elisp
>   > via Emacs let one do some cool visual stuff, like programming some
>   > graphics, and doing drawing in the editor would make it more useful and
>   > more efficient. Also I see it as a one brick in making Emacs more
>   > popular. I personally think it would be super cool to see Emacs do 3D
>   > with Elisp,
>
> Would this be a feature that works acceptably fast
> only if there is nonfree software in your system load?
Well it depends what one does and on underlaying OpenGL implementation.

I don't expect that anyone would use Emacs to make AAA games to be
played on 144Hz 8K screens, though it would be certainly cool if they
did. For doing some 2D graphics like plotting, graphs or 3D visualisations,
prototyping etc, I think it would work good enough. As Dmitry says, even
built-in Intel drivers does decent in simpler cases.

Re: Rethinking the design of xwidgets

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

The question of which laptops support free software is important but
it is a tangent here.  Would people please move that to
emacs-tangents@gnu.org?

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Rethinking the design of xwidgets

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

>    > do believe if Elisp
>    > via Emacs let one do some cool visual stuff, like programming some
>    > graphics, and doing drawing in the editor would make it more useful and
>    > more efficient. Also I see it as a one brick in making Emacs more
>    > popular. I personally think it would be super cool to see Emacs do 3D
>    > with Elisp,

  > > Would this be a feature that works acceptably fast
  > > only if there is nonfree software in your system load?

  > Like I described, modern Intel GPUs are reasonable fast at OpenGL 
  > already.

In the Free World, we don't get the modern Intel GPUs.  The computers we can use -- because we can turn off the Management Engine -- are the X200 and T400.

Would this feature run fast enough on them?

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Rethinking the design of xwidgets

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > I don't expect that anyone would use Emacs to make AAA games to be
  > played on 144Hz 8K screens, though it would be certainly cool if they
  > did. For doing some 2D graphics like plotting, graphs or 3D visualisations,
  > prototyping etc, I think it would work good enough.

Maybe there is an existing free program that someone could test on a
Libreboot machine, so that you could verify, before you write code,
that it would be ok on this issue.

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Rethinking the design of xwidgets

[Dmitry Gutov]

On 19.10.2020 06:44, Richard Stallman wrote:
> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
> 
>>     > do believe if Elisp
>>     > via Emacs let one do some cool visual stuff, like programming some
>>     > graphics, and doing drawing in the editor would make it more useful and
>>     > more efficient. Also I see it as a one brick in making Emacs more
>>     > popular. I personally think it would be super cool to see Emacs do 3D
>>     > with Elisp,
> 
>    > > Would this be a feature that works acceptably fast
>    > > only if there is nonfree software in your system load?
> 
>    > Like I described, modern Intel GPUs are reasonable fast at OpenGL
>    > already.
> 
> In the Free World, we don't get the modern Intel GPUs.  The computers we can use -- because we can turn off the Management Engine -- are the X200 and T400.

Ah, okay. That makes it more of a problem.

Have you looked into the use of me_cleaner for computer vendors that 
cooperate with FSF? Here's a detailed guide, which seems dangerous for 
random users but can be fine for hardware manufacturers because they can 
standardize on specific motherboards, etc: 
https://wiki.gentoo.org/wiki/User:Sakaki/Sakaki%27s_EFI_Install_Guide/Disabling_the_Intel_Management_Engine#Modifying_Firmware_using_me_cleaner.2C_to_Disable_the_IME

Also, some vendors have started offering machines with IME disabled 
(System76, Purism, and, most recently, Dell):

https://www.extremetech.com/computing/259879-dell-now-shipping-laptops-intels-management-engine-disabled

> Would this feature run fast enough on them?

The GPU is Intel GMA 4500, right?

We can definitely say that it's possible. From my brief research, people 
have been running games on it on GNU/Linux, such as Amnesia: Dark 
Descent and Bit.Trip.Runner (one is a 3D game, another is 2D).

This GPU is limited on OpenGL 2.1 only, though, so it might cause 
problems if whatever wrapper libraries we decide to use require a higher 
version.

Ultimately, I think the question can only be answered with certainty if 
someone wrote a prototype and then tested it on one of the X200 machines.

Re: Rethinking the design of xwidgets

[Arthur Miller]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>>    > do believe if Elisp
>>    > via Emacs let one do some cool visual stuff, like programming some
>>    > graphics, and doing drawing in the editor would make it more useful and
>>    > more efficient. Also I see it as a one brick in making Emacs more
>>    > popular. I personally think it would be super cool to see Emacs do 3D
>>    > with Elisp,
>
>   > > Would this be a feature that works acceptably fast
>   > > only if there is nonfree software in your system load?
>
>   > Like I described, modern Intel GPUs are reasonable fast at OpenGL 
>   > already.
>
> In the Free World, we don't get the modern Intel GPUs.  The computers we can use -- because we can turn off the Management Engine -- are the X200 and T400.
Just a thought: if a computer is detached from the "grid"; i.e. not
connected to the Internet, is it still considered not usable? I mean if
you get a modern computer, but don't use it online, just as a computing
device. Or can there be a hidden wifi card in a cpu that connects to any
open network? 

> Would this feature run fast enough on them?
I wish I could give you an answer, but I really have no idea.

I just would like to enable OpenGL context for an Emacs frame and to
draw some textured polygons to display in an image in Emacs buffer. And
make some Lisp binding to at least few important OpenGL functions to
upload polygons and textures in a vertex buffer + some shaders to OpenGL
so they can get rendered. Seems like non-accelerated Cairo graphics work
well, there is already a minor mode to render some rounded rectangles
with SVG. I don't know how that runs on older computers.

How fast would it run depends on what people will do with it.

Re: Rethinking the design of xwidgets

[Arthur Miller]

Dmitry Gutov <dgutov@yandex.ru> writes:

> On 19.10.2020 06:44, Richard Stallman wrote:
>> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
>> [[[ whether defending the US Constitution against all enemies,     ]]]
>> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>> 
>>>     > do believe if Elisp
>>>     > via Emacs let one do some cool visual stuff, like programming some
>>>     > graphics, and doing drawing in the editor would make it more useful and
>>>     > more efficient. Also I see it as a one brick in making Emacs more
>>>     > popular. I personally think it would be super cool to see Emacs do 3D
>>>     > with Elisp,
>>    > > Would this be a feature that works acceptably fast
>>    > > only if there is nonfree software in your system load?
>>    > Like I described, modern Intel GPUs are reasonable fast at OpenGL
>>    > already.
>> In the Free World, we don't get the modern Intel GPUs.  The computers we can
>> use -- because we can turn off the Management Engine -- are the X200 and T400.
>
> Ah, okay. That makes it more of a problem.
>
> Have you looked into the use of me_cleaner for computer vendors that cooperate
> with FSF? Here's a detailed guide, which seems dangerous for random users but
> can be fine for hardware manufacturers because they can standardize on specific
> motherboards, etc:
> https://wiki.gentoo.org/wiki/User:Sakaki/Sakaki%27s_EFI_Install_Guide/Disabling_the_Intel_Management_Engine#Modifying_Firmware_using_me_cleaner.2C_to_Disable_the_IME
>
> Also, some vendors have started offering machines with IME disabled (System76,
> Purism, and, most recently, Dell):
>
> https://www.extremetech.com/computing/259879-dell-now-shipping-laptops-intels-management-engine-disabled
>
>> Would this feature run fast enough on them?
>
> The GPU is Intel GMA 4500, right?
>
> We can definitely say that it's possible. From my brief research, people have
> been running games on it on GNU/Linux, such as Amnesia: Dark Descent and
> Bit.Trip.Runner (one is a 3D game, another is 2D).
>
> This GPU is limited on OpenGL 2.1 only, though, so it might cause problems if
> whatever wrapper libraries we decide to use require a higher version.
>
> Ultimately, I think the question can only be answered with certainty if someone
> wrote a prototype and then tested it on one of the X200 machines.

OpenGL 2.1 is probably fine for many cases.

2.0 adds non power of two textures which makes it possible to work
easier with images, multiple render targets, and it has VBOs and PBOs,
which gives for lots of fun :-).

OpenGL features are queried at runtime, but I am not sure
if this feature would be *legal* in Emacs; I mean loading a function
pointer at runtime from a so/dll is essentially ffi, which I
understand is not allowed in Emacs for legal purposes.

Re: Rethinking the design of xwidgets

[Arthur Miller]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > I don't expect that anyone would use Emacs to make AAA games to be
>   > played on 144Hz 8K screens, though it would be certainly cool if they
>   > did. For doing some 2D graphics like plotting, graphs or 3D visualisations,
>   > prototyping etc, I think it would work good enough.
>
> Maybe there is an existing free program that someone could test on a
> Libreboot machine, so that you could verify, before you write code,
> that it would be ok on this issue.
Can you run glxinfo on your machine and post the output? You could also
install glx-gears and glmark2 and see how they run on your machine?
glx-gears ir in Mesa utils and glmark2 is GPL3 source code on Github (I
didn't post link because of non-free JS on Github).

Re: Rethinking the design of xwidgets

[Stefan Monnier]

>   > Like I described, modern Intel GPUs are reasonable fast at OpenGL 
>   > already.
> In the Free World, we don't get the modern Intel GPUs.  The computers we can
> use -- because we can turn off the Management Engine -- are the X200
> and T400.
> Would this feature run fast enough on them?

I don't actually what "this feature" refers to, but to the extent that
the Gnome environment works fine on a Thinkpad T60 with integrated
Intel GPU, I see no reason to expect problems.


        Stefan "who remembers playing with the funny features of compiz
                on his Thinkpad X30"

Re: Rethinking the design of xwidgets

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > I don't actually what "this feature" refers to, but to the extent that
  > the Gnome environment works fine on a Thinkpad T60 with integrated
  > Intel GPU, I see no reason to expect problems.

That is good news.

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Rethinking the design of xwidgets

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > OpenGL features are queried at runtime, but I am not sure
  > if this feature would be *legal* in Emacs; I mean loading a function
  > pointer at runtime from a so/dll is essentially ffi, which I
  > understand is not allowed in Emacs for legal purposes.

If the function you are linking to is implemented by a
GPL-3-compatible library, Emacs can link to it in any way
that is convenient.  What is the source license of OpenGL 2.1?
I expect that it is GPL-3-compatible, but it can't hurt to verify that.

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Rethinking the design of xwidgets

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Also, some vendors have started offering machines with IME disabled 
  > (System76, Purism, and, most recently, Dell):

  > https://www.extremetech.com/computing/259879-dell-now-shipping-laptops-intels-management-engine-disabled

I know about Purism, but this news about Dell may be interesting.
Thanks.

However, now that this is not directly related to Emacs development
(since that question has been cleared up), how about if those
interested (including me) discuss this privately?


-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Rethinking the design of xwidgets

[Arthur Miller]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > OpenGL features are queried at runtime, but I am not sure
>   > if this feature would be *legal* in Emacs; I mean loading a function
>   > pointer at runtime from a so/dll is essentially ffi, which I
>   > understand is not allowed in Emacs for legal purposes.
>
> If the function you are linking to is implemented by a
> GPL-3-compatible library, Emacs can link to it in any way
> that is convenient.
It depends on the OpenGL implementation the user has installed on the
system. I don't see how is that different from running Emacs on a
non-free system, since Emacs has to link to a graphics library provided
by OS, GDI on Windows for example, which in turn anyway link to same
graphics driver that OpenGL/CL/EGL are linked to. Even on a GNU/Linux if
user has an AMD/Nvidia gfx card there is some firmware blob that gets
installed and graphcis applications inclusive Emacs run on it.

> What is the source license of OpenGL 2.1?
> I expect that it is GPL-3-compatible, but it can't hurt to verify that.

OpenGL is an open standard developed by Khronos: www.opengl.org

Mesa driver you have installed on your computer is open source, but I
don't know if licence is "free" or not:

https://docs.mesa3d.org/license.html

Re: Rethinking the design of xwidgets

[Arthur Miller]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > Also, some vendors have started offering machines with IME disabled 
>   > (System76, Purism, and, most recently, Dell):
>
>   > https://www.extremetech.com/computing/259879-dell-now-shipping-laptops-intels-management-engine-disabled
>
> I know about Purism, but this news about Dell may be interesting.
> Thanks.
I just looked up at dell.se; I don't see it mentioned that they have
disabled ME; the article is from 2017. I can't even choose to buy it
without Windows.

> However, now that this is not directly related to Emacs development
> (since that question has been cleared up), how about if those
> interested (including me) discuss this privately?

Sure, we can keep it off the emacs-devel; but in a way it is related: it
decides what Emacs will/can do or not, regardless of technical aspect.

Re: Rethinking the design of xwidgets

[Dmitry Gutov]

On 20.10.2020 08:56, Arthur Miller wrote:
>>    >https://www.extremetech.com/computing/259879-dell-now-shipping-laptops-intels-management-engine-disabled
>>
>> I know about Purism, but this news about Dell may be interesting.
>> Thanks.
> I just looked up at dell.se; I don't see it mentioned that they have
> disabled ME; the article is from 2017. I can't even choose to buy it
> without Windows.

Yes, sorry. Seems this info if outdated. The model with which you could 
seemingly still do that at the beginning of this year 
(https://www.reddit.com/r/linux/comments/eidk1x/how_to_buy_a_dell_laptop_with_the_intel_me/) 
is no longer available.

That still leaves System76 and me_cleaner, though.

Re: Rethinking the design of xwidgets

[Stefan Monnier]

> Even on a GNU/Linux if user has an AMD/Nvidia gfx card there is some
> firmware blob that gets installed and graphcis applications inclusive
> Emacs run on it.

Last I checked (many years ago), the Nouveau driver for Nvidia did not
require a firmware blob.  Has it changed?


        Stefan

Re: Rethinking the design of xwidgets

[Arthur Miller]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> Even on a GNU/Linux if user has an AMD/Nvidia gfx card there is some
>> firmware blob that gets installed and graphcis applications inclusive
>> Emacs run on it.
>
> Last I checked (many years ago), the Nouveau driver for Nvidia did not
> require a firmware blob.  Has it changed?
>
>
>         Stefan

No; I don't think it changed. I was refering to proprietary nv/amd
drivers. You pointed out that Intel has open sourced drivers. I just
checked and found:

"Intel’s Open Source Graphics Drivers is one of the most widely deployed driver stacks in the industry. These drivers are integrated seamlessly into Linux* PC client distributions, Google Chromebooks*, and Valve’s SteamOS* serving tens of millions of PC users. The Intel’s Open Source Vulkan driver for 5th generation Intel® Core™ processors and 6th generation Intel® Core™ processors (code-named Broadwell and Skylake) passes the Vulkan 1.0 Conformance Test Suite on these platforms and has experimental support for older platforms. Developers can either build the drivers from source code or directly get Linux distribution packages. Please go to https://01.org/linuxgraphics/blogs/jekstrand/2016/open-source-vulkan-drivers-intel-hardware for more information."


Now that was from an Intel's blog about game dev (and Vulkan), and
Intel's platform is dismissed for other reasons, so I don't know what to
say.

By the way I have also now checked the state of AMD cpus and ARM/PI.
It seems like they all have some kind of 'management' capabilities. Am I
correct? It was some blog I red it on.

So I really don't know what is sustainability in refusing to use newer
CPUs?

Insisting on using only 15 years old computers is not a sustainable
strategy for future, not for the majority of people and I guess not for
the development of FREE software or GNU movement if I can call it so.
Nor do I see it as a consistent strategy with current development either.

Anyone, don't get me wrong; I don't like those; nor do I propose
any solution, because I don't have; it is more of a general question.
Maybe I analyse the situation wrong; but I it's an honest consideration.

Re: Rethinking the design of xwidgets

[Arthur Miller]

Dmitry Gutov <dgutov@yandex.ru> writes:

> On 20.10.2020 08:56, Arthur Miller wrote:
>>>    >https://www.extremetech.com/computing/259879-dell-now-shipping-laptops-intels-management-engine-disabled
>>>
>>> I know about Purism, but this news about Dell may be interesting.
>>> Thanks.
>> I just looked up at dell.se; I don't see it mentioned that they have
>> disabled ME; the article is from 2017. I can't even choose to buy it
>> without Windows.
>
> Yes, sorry. Seems this info if outdated. The model with which you could
> seemingly still do that at the beginning of this year
> (https://www.reddit.com/r/linux/comments/eidk1x/how_to_buy_a_dell_laptop_with_the_intel_me/)
> is no longer available.
>
> That still leaves System76 and me_cleaner, though.

I see one problem with Purism.

Consider Intel and Purism. What considers me, they are both US based
private business (I don't know how much US or private Intel is, but I
guess). So what we are doing is exchanging a blob for company A for a
blob of company B. Blob from B should disable blob from A. How do I know
B is not run secretely by some goverment organisation that likes
to spy on it's own citizens and foreign ones. Just recently we learned
there was a supposedly independent Swiss company that sold a
cryptography machines to goverments all over the world. It turned out it
was secretely collaborating with CIA that gott keys to decrypt
everyone's secrets :D. Russians never bought it, but some other
countries did.

Idea of blobs is bad; and Dr. Richard S. is completelycorrect about
blobs not being acceptable. But then, we need to live in this world as
it is; so we need a sustianable solution for the future. I don't know if
limiting what Emacs can do on capabilities of a machine from the past is
a best strategy; but I am not very wise, and certainly did not do enough
research and thinking in the area, so this is just my ramblings and
consideration; unfortunately I have no answers myself.

Re: Rethinking the design of xwidgets

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > So I really don't know what is sustainability in refusing to use newer
  > CPUs?

I agree that is a problem.  But we can't win by surrendering.
I am hoping that the various efforts to make newer, freer computers
will fix this before too late.

However, could we please move this discussion off emacs-devel?
It's not on topic here.

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Rethinking the design of xwidgets

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Sure, we can keep it off the emacs-devel; but in a way it is related: it
  > decides what Emacs will/can do or not, regardless of technical aspect.

I think we've already settled that part.  The machines that we CAN
use are fast enough to handle the graphics that is being considered.

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Rethinking the design of xwidgets

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > OpenGL is an open standard developed by Khronos: www.opengl.org

  > Mesa driver you have installed on your computer is open source, but I
  > don't know if licence is "free" or not:

I am using Trisquel, a 100% free distro.
All the software installed on my machine is free.

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Rethinking the design of xwidgets

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > And yes, I do think Emacs for Linux deserves proper hardware
  > acceleration.

GNU Emacs isn't "for Linux", it is for the GNU operating system.

Perhaps when you say "Linux" you're talking about the GNU operating
system, with Linux as the kernel.  That's what people usually mean
when they say "Linux".  See https://gnu.org/gnu/linux-and-gnu.html and
https://gnu.org/gnu/gnu-linux-faq.html, plus the history in
https://gnu.org/gnu/the-gnu-project.html.

It makes a practical difference, indirectly, which name you call it.
When people think of the GNU operating system as "Linux", they tend to
forget its goal.

The goal of the GNU operating system is to help people escape from
nonfree software.  We want that because nonfree software does
injustice to its users.  See fsf.org/tedx and
https://gnu.org/philosophy/free-software-even-more-important.html.  So
we judge any change by its effect on what we can do in the Free World.
That means, what ew can do with no nonfree programs installed.

Before 2000 or so, we didn't even have a real graphical desktop in the
Free World.  So we used our computers without them.  To run Qt
(nonfree, at the time) plus KDE would have been an advance in
convenience but a step backwards in ethics.

An operating system is not a person.  It has no rights, and it has no
interests.  To speak of what it "deserves" is a metonymy which really
means what its users deserve.

For the free software movement, what users deserve above all is
freedom.  Graphical acceleration is nice provided we don't have to pay
for it with freedom.

We determined a few weeks ago that graphics acceleration is available
in the free world.  So Emacs can make use of it, within carefully
studied limits.

The reason I'm posting this now (replying to a message I just saw) is
that discussions on what to do in developing GNU Emacs should not
forget the basic goal of work on GNU Emacs: defending and extending
the Free World.

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Akira Kyle]

Awhile ago I started a thread on rethinking the design of 
xwidgets, there was a lot of good discussion about the future of 
Emacs' rendering capabilities. It personally motivated me to try 
to better understand how Emacs' redisplay works and just overall 
better understand the C core of Emacs. As part of that exercise, I 
decided to see if I could port the existing xwidgets code into a 
dynamic module. The only xwidget Emacs currently implements is a 
webkit widget, so this effort turned into creating a dynamic 
module for embedding webkit in Emacs. It also happens that I've 
been wishing that I could never leave Emacs and a general web 
browser was the last daily app that I begrudgingly used outside of 
Emacs. Anyways here's the result: 
https://github.com/akirakyle/emacs-webkit.git (sorry for the 
github hosting, I just have yet to make the jump to another 
platform -- I guess what they say about vendor lock-in is true).

I was able to implement all of the current features of the webkit 
xwidget plus some more. I think having this as a dynamic module 
makes it feel less hacked into Emacs (although still a bit hackey 
in some places). One interesting additional feature I was able to 
implement was support for TUI emacs. Essentially it opens a 
dedicated window for displaying webkit which can be controlled 
through the keybindings in the emacs buffer corresponding to that 
webkit view (of course one needs to have running a running 
graphical session pointed to in $DISPLAY). Perhaps when this code 
stabilizes, it might make sense to remove the experimental xwidget 
support from Emacs if this is deemed a worthy successor to such 
features.

I also wanted to make a few more general comments prompted by the 
previous thread about the future of Emacs redisplay and rendering 
given what I've learned in this exercise. I now think its fairly 
foolish to try to integrate too much of a toolkit library's 
widgets into Emacs. There will always be this friction between 
Emacs and the toolkit library wanting to handle the business of 
drawing to the glass. In a lot of ways I've come away from this 
very impressed with how Emacs manages to have native toolkit ports 
to gtk, ns, win, and TUI rendering given the differences in how 
each wants to handle drawing to the screen, but on the other hand 
there's a lot of complexity and code behind making this happen, 
which presents a large barrier to entry to people like me who may 
want to hack on the lower level drawing capabilities of 
Emacs. We've also seen hardware change a lot since Emacs display 
machinery was written. CPUs have gotten faster and now everyone 
has a GPU. I think there's a potential for Emacs to become both 
snappier and more extensible with respect to its rendering. I had 
initially argued that Emacs probably shouldn't deal with rendering 
on the level of opengl, however I think as I've understood better 
the way Emacs currently handles rendering everything, that's 
exactly the direction Emacs should move in, *if it can*.

So here is my new pie-in-the-sky proposal for what the future of 
Emacs rendering could look like. Given the success and hopefully 
soon-to-be in master native-comp branch, perhaps it may not be so 
crazy to think about moving more rendering to elisp. If a clean 
elisp interface to opengl were defined and Emacs were able to 
render itself using this optimized elisp opengl interface, the low 
level code for each platform could be minimized to just providing 
the opengl surface and exposing input events. In fact there may be 
ways in which elisp is well suited to the way modern GPU graphics 
are rendered using scene graphs. In fact gtk4 is moving its 
rendering to such an approach in order to take advantage of 
hardware graphics acceleration. I think a first step would likely 
be to focus on implementing an elisp interface to cairo as Emacs 
is already using more of cairo to render itself. If it seems like 
the performance is good and elisp can represent graphical 
operations well, than that might be a good indication that such a 
model could be successful path forward towards an Emacs that could 
bring its extensibility to its own basic rendering.

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Jean Louis]

Hello Akira,

That is great to see. I will try to compile that as dynamic module. If
I remember well the webkit on my system could never compile to really
work, something was odd. I am using Hyperbola GNU/Linux-libre system.

I will have one question for you later, first I wish to try your
version of webkit.

* Akira Kyle <akira@akirakyle.com> [2020-11-22 14:05]:
>  Anyways here's the result:
> https://github.com/akirakyle/emacs-webkit.git (sorry for the github
> hosting, I just have yet to make the jump to another platform -- I
> guess what they say about vendor lock-in is true).

Let me give you and others who in future read this email, a set of
reference to switch:

GNU Project
https://www.gnu.org

Savannah, the software forge for people committed to free software
https://savannah.gnu.org

Savannah on nongnu.org
https//savannah.nongnu.org

Codeberg.org (Germany)
https://codeberg.org

Sourcehut.org
https://sourcehut.org

Trisquel GNU/Linux-libre Git Repositories
https://devel.trisquel.info/groups/trisquel

Pagure
https://pagure.io/pagure

Fosshost
https://fosshost.org/

GitGud - Fast and Free Git Hosting
https://gitgud.io/users/sign_in

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Jean Louis]

I have failed shamelessly.

Can it be to some outdated stuff on my system?

git clone https://github.com/akirakyle/emacs-webkit.git
Cloning into 'emacs-webkit'...
remote: Enumerating objects: 160, done.        
remote: Counting objects: 100% (160/160), done.        
remote: Compressing objects: 100% (54/54), done.        
remote: Total 160 (delta 104), reused 160 (delta 104), pack-reused 0        
Receiving objects: 100% (160/160), 2.35 MiB | 219.00 KiB/s, done.
Resolving deltas: 100% (104/104), done.
~/Programming/git $ cd emacs-webkit/ 
~/Programming/git/emacs-webkit $ ls
LICENSE     default.nix                hints.css       script.js  webkit-ace.el      webkit.el
Makefile    emacs-module.h             hints.js        style.css  webkit-history.el  
README.org  evil-collection-webkit.el  screencast.gif  tests.el   webkit-module.c    
~/Programming/git/emacs-webkit $ make
cc -shared -std=c99 -Wall -Wextra -Wno-unused-parameter -O3 -fpic  `pkg-config --cflags gtk+-3.0 webkit2gtk-4.0 --libs webkit2gtk-4.0` -o webkit-module.so webkit-module.c
webkit-module.c: In function ‘copy_string_contents’:
webkit-module.c:47:13: warning: implicit declaration of function ‘malloc’ [-Wimplicit-function-declaration]
   *buffer = malloc ((size_t) buffer_size);
             ^~~~~~
webkit-module.c:47:13: warning: incompatible implicit declaration of built-in function ‘malloc’
webkit-module.c:47:13: note: include ‘<stdlib.h>’ or provide a declaration of ‘malloc’
webkit-module.c:57:7: warning: implicit declaration of function ‘free’ [-Wimplicit-function-declaration]
       free (*buffer);
       ^~~~
webkit-module.c:57:7: warning: incompatible implicit declaration of built-in function ‘free’
webkit-module.c:57:7: note: include ‘<stdlib.h>’ or provide a declaration of ‘free’
webkit-module.c: In function ‘webkit_get_title’:
webkit-module.c:134:46: warning: implicit declaration of function ‘strlen’ [-Wimplicit-function-declaration]
         return env->make_string (env, title, strlen (title));
                                              ^~~~~~
webkit-module.c:134:46: warning: incompatible implicit declaration of built-in function ‘strlen’
webkit-module.c:134:46: note: include ‘<string.h>’ or provide a declaration of ‘strlen’
webkit-module.c: In function ‘webkit_get_uri’:
webkit-module.c:145:26: warning: implicit declaration of function ‘webkit_uri_for_display’ [-Wimplicit-function-declaration]
       const gchar *uri = webkit_uri_for_display (webkit_web_view_get_uri
                          ^~~~~~~~~~~~~~~~~~~~~~
webkit-module.c:145:26: warning: initialization makes pointer from integer without a cast [-Wint-conversion]
webkit-module.c:147:42: warning: incompatible implicit declaration of built-in function ‘strlen’
       return env->make_string (env, uri, strlen (uri));
                                          ^~~~~~
webkit-module.c:147:42: note: include ‘<string.h>’ or provide a declaration of ‘strlen’
webkit-module.c: In function ‘webkit_load_uri’:
webkit-module.c:161:3: warning: incompatible implicit declaration of built-in function ‘free’
   free (uri);
   ^~~~
webkit-module.c:161:3: note: include ‘<stdlib.h>’ or provide a declaration of ‘free’
webkit-module.c: In function ‘webkit_search’:
webkit-module.c:221:3: warning: incompatible implicit declaration of built-in function ‘free’
   free (text);
   ^~~~
webkit-module.c:221:3: note: include ‘<stdlib.h>’ or provide a declaration of ‘free’
webkit-module.c: In function ‘send_to_lisp’:
webkit-module.c:336:41: warning: incompatible implicit declaration of built-in function ‘strlen’
       || rio_writen (c->fd, (void *)id, strlen (id)+1) < 0
                                         ^~~~~~
webkit-module.c:336:41: note: include ‘<string.h>’ or provide a declaration of ‘strlen’
webkit-module.c: In function ‘webkit_js_finished’:
webkit-module.c:360:3: error: unknown type name ‘JSCValue’
   JSCValue *value = webkit_javascript_result_get_js_value (js_result);
   ^~~~~~~~
webkit-module.c:360:21: warning: implicit declaration of function ‘webkit_javascript_result_get_js_value’ [-Wimplicit-function-declaration]
   JSCValue *value = webkit_javascript_result_get_js_value (js_result);
                     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
webkit-module.c:360:21: warning: initialization makes pointer from integer without a cast [-Wint-conversion]
webkit-module.c:361:17: warning: implicit declaration of function ‘jsc_value_to_json’ [-Wimplicit-function-declaration]
   gchar *json = jsc_value_to_json (value, 1);
                 ^~~~~~~~~~~~~~~~~
webkit-module.c:361:17: warning: initialization makes pointer from integer without a cast [-Wint-conversion]
webkit-module.c:362:3: error: unknown type name ‘JSCException’
   JSCException *exception =
   ^~~~~~~~~~~~
webkit-module.c:363:5: warning: implicit declaration of function ‘jsc_context_get_exception’ [-Wimplicit-function-declaration]
     jsc_context_get_exception (jsc_value_get_context (value));
     ^~~~~~~~~~~~~~~~~~~~~~~~~
webkit-module.c:363:32: warning: implicit declaration of function ‘jsc_value_get_context’ [-Wimplicit-function-declaration]
     jsc_context_get_exception (jsc_value_get_context (value));
                                ^~~~~~~~~~~~~~~~~~~~~
webkit-module.c:363:5: warning: initialization makes pointer from integer without a cast [-Wint-conversion]
     jsc_context_get_exception (jsc_value_get_context (value));
     ^~~~~~~~~~~~~~~~~~~~~~~~~
In file included from /usr/include/glib-2.0/glib.h:62:0,
                 from /usr/include/gtk-3.0/gdk/gdkconfig.h:13,
                 from /usr/include/gtk-3.0/gdk/gdk.h:30,
                 from /usr/include/gtk-3.0/gtk/gtk.h:30,
                 from webkit-module.c:3:
webkit-module.c:366:16: warning: implicit declaration of function ‘jsc_exception_get_message’ [-Wimplicit-function-declaration]
                jsc_exception_get_message (exception));
                ^
/usr/include/glib-2.0/glib/gmessages.h:336:32: note: in definition of macro ‘g_warning’
                                __VA_ARGS__)
                                ^~~~~~~~~~~
webkit-module.c:365:16: warning: format ‘%s’ expects argument of type ‘char *’, but argument 4 has type ‘int’ [-Wformat=]
     g_warning ("Error running javascript: %s",
                ^
/usr/include/glib-2.0/glib/gmessages.h:336:32: note: in definition of macro ‘g_warning’
                                __VA_ARGS__)
                                ^~~~~~~~~~~
webkit-module.c:373:3: warning: incompatible implicit declaration of built-in function ‘free’
   free (cb->id);
   ^~~~
webkit-module.c:373:3: note: include ‘<stdlib.h>’ or provide a declaration of ‘free’
webkit-module.c: In function ‘webkit_execute_js’:
webkit-module.c:390:26: warning: incompatible implicit declaration of built-in function ‘malloc’
           Callback *cb = malloc (sizeof (Callback));
                          ^~~~~~
webkit-module.c:390:26: note: include ‘<stdlib.h>’ or provide a declaration of ‘malloc’
webkit-module.c:403:3: warning: incompatible implicit declaration of built-in function ‘free’
   free (script);
   ^~~~
webkit-module.c:403:3: note: include ‘<stdlib.h>’ or provide a declaration of ‘free’
webkit-module.c: In function ‘webkit_add_user_style’:
webkit-module.c:430:3: warning: incompatible implicit declaration of built-in function ‘free’
   free (style);
   ^~~~
webkit-module.c:430:3: note: include ‘<stdlib.h>’ or provide a declaration of ‘free’
webkit-module.c: In function ‘webkit_add_user_script’:
webkit-module.c:472:3: warning: incompatible implicit declaration of built-in function ‘free’
   free (script);
   ^~~~
webkit-module.c:472:3: note: include ‘<stdlib.h>’ or provide a declaration of ‘free’
webkit-module.c: In function ‘webkit_script_message_cb’:
webkit-module.c:501:3: error: unknown type name ‘JSCValue’
   JSCValue *value = webkit_javascript_result_get_js_value (js_result);
   ^~~~~~~~
webkit-module.c:501:21: warning: initialization makes pointer from integer without a cast [-Wint-conversion]
   JSCValue *value = webkit_javascript_result_get_js_value (js_result);
                     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
webkit-module.c:502:17: warning: initialization makes pointer from integer without a cast [-Wint-conversion]
   gchar *json = jsc_value_to_json (value, 1);
                 ^~~~~~~~~~~~~~~~~
webkit-module.c:503:3: error: unknown type name ‘JSCException’
   JSCException *exception =
   ^~~~~~~~~~~~
webkit-module.c:504:5: warning: initialization makes pointer from integer without a cast [-Wint-conversion]
     jsc_context_get_exception (jsc_value_get_context (value));
     ^~~~~~~~~~~~~~~~~~~~~~~~~
In file included from /usr/include/glib-2.0/glib.h:62:0,
                 from /usr/include/gtk-3.0/gdk/gdkconfig.h:13,
                 from /usr/include/gtk-3.0/gdk/gdk.h:30,
                 from /usr/include/gtk-3.0/gtk/gtk.h:30,
                 from webkit-module.c:3:
webkit-module.c:506:16: warning: format ‘%s’ expects argument of type ‘char *’, but argument 4 has type ‘int’ [-Wformat=]
     g_warning ("Error in javascript message recieve: %s",
                ^
/usr/include/glib-2.0/glib/gmessages.h:336:32: note: in definition of macro ‘g_warning’
                                __VA_ARGS__)
                                ^~~~~~~~~~~
webkit-module.c: In function ‘webkit_register_script_message’:
webkit-module.c:537:3: warning: incompatible implicit declaration of built-in function ‘free’
   free (name);
   ^~~~
webkit-module.c:537:3: note: include ‘<stdlib.h>’ or provide a declaration of ‘free’
webkit-module.c: In function ‘webkit_unregister_script_message’:
webkit-module.c:559:3: warning: incompatible implicit declaration of built-in function ‘free’
   free (name);
   ^~~~
webkit-module.c:559:3: note: include ‘<stdlib.h>’ or provide a declaration of ‘free’
webkit-module.c: In function ‘webview_notify_uri’:
webkit-module.c:612:22: warning: initialization makes pointer from integer without a cast [-Wint-conversion]
   const gchar *uri = webkit_uri_for_display (webkit_web_view_get_uri (webview));
                      ^~~~~~~~~~~~~~~~~~~~~~
webkit-module.c: In function ‘client_free’:
webkit-module.c:902:3: warning: incompatible implicit declaration of built-in function ‘free’
   free(c);
   ^~~~
webkit-module.c:902:3: note: include ‘<stdlib.h>’ or provide a declaration of ‘free’
webkit-module.c: In function ‘webkit_new’:
webkit-module.c:920:13: warning: implicit declaration of function ‘calloc’ [-Wimplicit-function-declaration]
   if (!(c = calloc (1, sizeof (Client))))
             ^~~~~~
webkit-module.c:920:13: warning: incompatible implicit declaration of built-in function ‘calloc’
webkit-module.c:920:13: note: include ‘<stdlib.h>’ or provide a declaration of ‘calloc’
webkit-module.c:963:46: warning: incompatible implicit declaration of built-in function ‘strlen’
              env->make_string (env, err_msg, strlen (err_msg)));
                                              ^~~~~~
webkit-module.c:963:46: note: include ‘<string.h>’ or provide a declaration of ‘strlen’
make: *** [Makefile:10: webkit-module.so] Error 1
~/Programming/git/emacs-webkit $ 

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[tomas]

[-- Attachment #1: Type: text/plain, Size: 1870 bytes --]

On Sun, Nov 22, 2020 at 03:04:29PM +0300, Jean Louis wrote:
> I have failed shamelessly.
> 
> Can it be to some outdated stuff on my system?
> 
> git clone https://github.com/akirakyle/emacs-webkit.git
> Cloning into 'emacs-webkit'...
> remote: Enumerating objects: 160, done.        
> remote: Counting objects: 100% (160/160), done.        
> remote: Compressing objects: 100% (54/54), done.        
> remote: Total 160 (delta 104), reused 160 (delta 104), pack-reused 0        
> Receiving objects: 100% (160/160), 2.35 MiB | 219.00 KiB/s, done.
> Resolving deltas: 100% (104/104), done.
> ~/Programming/git $ cd emacs-webkit/ 
> ~/Programming/git/emacs-webkit $ ls
> LICENSE     default.nix                hints.css       script.js  webkit-ace.el      webkit.el
> Makefile    emacs-module.h             hints.js        style.css  webkit-history.el  
> README.org  evil-collection-webkit.el  screencast.gif  tests.el   webkit-module.c    
> ~/Programming/git/emacs-webkit $ make
> cc -shared -std=c99 -Wall -Wextra -Wno-unused-parameter -O3 -fpic  `pkg-config --cflags gtk+-3.0 webkit2gtk-4.0 --libs webkit2gtk-4.0` -o webkit-module.so webkit-module.c
> webkit-module.c: In function ‘copy_string_contents’:
> webkit-module.c:47:13: warning: implicit declaration of function ‘malloc’ [-Wimplicit-function-declaration]
>    *buffer = malloc ((size_t) buffer_size);
>              ^~~~~~

[...]

Uh, oh. You seem to be missing fundamental packages for compiling
things. "malloc", for example is declared in /usr/include/stdlib.h

Does this file exist on your machine? If not, you'll have to look
for it. If yes, you'll have to adapt your compile options for your
compiler to find it.

Unless there's a configure step (but I don't see traces of it in
your files list) which would be willing to do that for you.

Cheers
 - t

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Jean Louis]

* Akira Kyle <akira@akirakyle.com> [2020-11-22 14:05]:
> 
> Awhile ago I started a thread on rethinking the design of xwidgets, there
> was a lot of good discussion about the future of Emacs' rendering
> capabilities. It personally motivated me to try to better understand how
> Emacs' redisplay works and just overall better understand the C core of
> Emacs. As part of that exercise, I decided to see if I could port the
> existing xwidgets code into a dynamic module.

There are 2 types of questions for you, one group of questions if for
dynamic module emacs-webkit:

- is it possible to make sure from start of the session not to use
  Javascript?

- is it maybe possible at compile time to add an option
  --disable-javascript ?

- is there way during the session to surely disable javascript?

- it would be very nice if webkit dynamic module could bookmark
  hyperlink in such a way that it can be programmatically captured by
  using Emacs Lisp. For example by pressing B that users can capture
  both the TITLE and URL together and store it how they wish. You
  could provide standard function and one customizable function. That
  way users could store the TITLE and URL without reloading the page
  into database, or into Org file, or other system.

And some questions not related to the subject, rather to your skills
with dynamic modules:

Developers of emacs-libpq will soon contribute the dynamic module to
GNU ELPA that offers bindings to PostgreSQL and that enhances Emacs
greatly with possibilities. Related to that in GNU there is GDBM or
GNU database https://www.gnu.org/s/gdbm

I am just asking if you could think of programming the dynamic module
that offers to users Emacs interface to GDBM. This type of database is
much simpler and users most probably have it on their systems and it
can be used for business processes.

Reference to emacs-libpq @ Github
https://github.com/anse1/emacs-libpq

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Jean Louis]

* tomas@tuxteam.de <tomas@tuxteam.de> [2020-11-22 15:20]:
> On Sun, Nov 22, 2020 at 03:04:29PM +0300, Jean Louis wrote:
> > I have failed shamelessly.
> > 
> > Can it be to some outdated stuff on my system?
> > 
> > git clone https://github.com/akirakyle/emacs-webkit.git
> > Cloning into 'emacs-webkit'...
> > remote: Enumerating objects: 160, done.        
> > remote: Counting objects: 100% (160/160), done.        
> > remote: Compressing objects: 100% (54/54), done.        
> > remote: Total 160 (delta 104), reused 160 (delta 104), pack-reused 0        
> > Receiving objects: 100% (160/160), 2.35 MiB | 219.00 KiB/s, done.
> > Resolving deltas: 100% (104/104), done.
> > ~/Programming/git $ cd emacs-webkit/ 
> > ~/Programming/git/emacs-webkit $ ls
> > LICENSE     default.nix                hints.css       script.js  webkit-ace.el      webkit.el
> > Makefile    emacs-module.h             hints.js        style.css  webkit-history.el  
> > README.org  evil-collection-webkit.el  screencast.gif  tests.el   webkit-module.c    
> > ~/Programming/git/emacs-webkit $ make
> > cc -shared -std=c99 -Wall -Wextra -Wno-unused-parameter -O3 -fpic  `pkg-config --cflags gtk+-3.0 webkit2gtk-4.0 --libs webkit2gtk-4.0` -o webkit-module.so webkit-module.c
> > webkit-module.c: In function ‘copy_string_contents’:
> > webkit-module.c:47:13: warning: implicit declaration of function ‘malloc’ [-Wimplicit-function-declaration]
> >    *buffer = malloc ((size_t) buffer_size);
> >              ^~~~~~
> 
> [...]
> 
> Uh, oh. You seem to be missing fundamental packages for compiling
> things. "malloc", for example is declared in /usr/include/stdlib.h
> 
> Does this file exist on your machine? If not, you'll have to look
> for it. If yes, you'll have to adapt your compile options for your
> compiler to find it.

Yes, it does.

> Unless there's a configure step (but I don't see traces of it in
> your files list) which would be willing to do that for you.

There is no configure, I would hoping it will work out of the box as
other Emacs modules.

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Arthur Miller]

Jean Louis <bugs@gnu.support> writes:

> I have failed shamelessly.
:-) Happens to best sometimes.

> Can it be to some outdated stuff on my system?

It looks like you should just fix some include path and maybe you need
to see over your linker options;

> webkit-module.c:920:13: warning: incompatible implicit declaration of built-in function ‘calloc’
> webkit-module.c:920:13: note: include ‘<stdlib.h>’ or provide a declaration of ‘calloc’
> webkit-module.c:963:46: warning: incompatible implicit declaration of built-in function ‘strlen’
>               env->make_string (env, err_msg, strlen (err_msg)));
>                                               ^~~~~~
> webkit-module.c:963:46: note: include ‘<string.h>’ or provide a declaration of ‘strlen’
> make: *** [Makefile:10: webkit-module.so] Error 1
The compiler already tells you to include <cstdlib.h> (which is c++
version of stdlib.h) + some other stuff. malloc, free & co are found
in stdlib.h and strlen is foound in string.h.

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[tomas]

[-- Attachment #1: Type: text/plain, Size: 2435 bytes --]

On Sun, Nov 22, 2020 at 03:57:22PM +0300, Jean Louis wrote:
> * tomas@tuxteam.de <tomas@tuxteam.de> [2020-11-22 15:20]:
> > On Sun, Nov 22, 2020 at 03:04:29PM +0300, Jean Louis wrote:
> > > I have failed shamelessly.
> > > 
> > > Can it be to some outdated stuff on my system?
> > > 
> > > git clone https://github.com/akirakyle/emacs-webkit.git
> > > Cloning into 'emacs-webkit'...
> > > remote: Enumerating objects: 160, done.        
> > > remote: Counting objects: 100% (160/160), done.        
> > > remote: Compressing objects: 100% (54/54), done.        
> > > remote: Total 160 (delta 104), reused 160 (delta 104), pack-reused 0        
> > > Receiving objects: 100% (160/160), 2.35 MiB | 219.00 KiB/s, done.
> > > Resolving deltas: 100% (104/104), done.
> > > ~/Programming/git $ cd emacs-webkit/ 
> > > ~/Programming/git/emacs-webkit $ ls
> > > LICENSE     default.nix                hints.css       script.js  webkit-ace.el      webkit.el
> > > Makefile    emacs-module.h             hints.js        style.css  webkit-history.el  
> > > README.org  evil-collection-webkit.el  screencast.gif  tests.el   webkit-module.c    
> > > ~/Programming/git/emacs-webkit $ make
> > > cc -shared -std=c99 -Wall -Wextra -Wno-unused-parameter -O3 -fpic  `pkg-config --cflags gtk+-3.0 webkit2gtk-4.0 --libs webkit2gtk-4.0` -o webkit-module.so webkit-module.c
> > > webkit-module.c: In function ‘copy_string_contents’:
> > > webkit-module.c:47:13: warning: implicit declaration of function ‘malloc’ [-Wimplicit-function-declaration]
> > >    *buffer = malloc ((size_t) buffer_size);
> > >              ^~~~~~
> > 
> > [...]
> > 
> > Uh, oh. You seem to be missing fundamental packages for compiling
> > things. "malloc", for example is declared in /usr/include/stdlib.h
> > 
> > Does this file exist on your machine? If not, you'll have to look
> > for it. If yes, you'll have to adapt your compile options for your
> > compiler to find it.
> 
> Yes, it does.

except when it doesn't :)

The error, at least, suggests that webkit-module.c isn't including
stdlib.h, for some reason.

> > Unless there's a configure step (but I don't see traces of it in
> > your files list) which would be willing to do that for you.
> 
> There is no configure, I would hoping it will work out of the box as
> other Emacs modules.

Is there an #include <stdlib.h> in webkit-module.c?

Cheers
-- t

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Jean Louis]

* tomas@tuxteam.de <tomas@tuxteam.de> [2020-11-22 19:24]:
> > > Uh, oh. You seem to be missing fundamental packages for compiling
> > > things. "malloc", for example is declared in /usr/include/stdlib.h
> > > 
> > > Does this file exist on your machine? If not, you'll have to look
> > > for it. If yes, you'll have to adapt your compile options for your
> > > compiler to find it.
> > 
> > Yes, it does.
> 
> except when it doesn't :)

You know that all better, thank you. 

I do remember being able to compile webkit in Emacs by compiling Emacs
itself, but it did not work as expected and I removed it.

$ locate stdlib.h
/usr/include/stdlib.h

but webkit-module.c inside of emacs-webkit git directory does not have
stdlib.h, there is nowhere in other files included. And I included
also string.h that was missing.

$ head webkit-module.c 
#define _POSIX_SOURCE 1

#include <gtk/gtk.h>
#include <webkit2/webkit2.h>
#include <stdbool.h>
#include <signal.h>
#include <assert.h>
#include <errno.h>

I think it should be included there and I have included those missind
but then I got these other errors. Maybe my webkit is outdated for
this one. I cannot install new webkit on this system.

webkit-module.c: In function ‘webkit_get_uri’:
webkit-module.c:147:26: warning: implicit declaration of function ‘webkit_uri_for_display’ [-Wimplicit-function-declaration]
       const gchar *uri = webkit_uri_for_display (webkit_web_view_get_uri
                          ^~~~~~~~~~~~~~~~~~~~~~
webkit-module.c:147:26: warning: initialization makes pointer from integer without a cast [-Wint-conversion]
webkit-module.c: In function ‘webkit_js_finished’:
webkit-module.c:362:3: error: unknown type name ‘JSCValue’
   JSCValue *value = webkit_javascript_result_get_js_value (js_result);
   ^~~~~~~~
webkit-module.c:362:21: warning: implicit declaration of function ‘webkit_javascript_result_get_js_value’ [-Wimplicit-function-declaration]
   JSCValue *value = webkit_javascript_result_get_js_value (js_result);
                     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
webkit-module.c:362:21: warning: initialization makes pointer from integer without a cast [-Wint-conversion]
webkit-module.c:363:17: warning: implicit declaration of function ‘jsc_value_to_json’ [-Wimplicit-function-declaration]
   gchar *json = jsc_value_to_json (value, 1);
                 ^~~~~~~~~~~~~~~~~
webkit-module.c:363:17: warning: initialization makes pointer from integer without a cast [-Wint-conversion]
webkit-module.c:364:3: error: unknown type name ‘JSCException’
   JSCException *exception =
   ^~~~~~~~~~~~
webkit-module.c:365:5: warning: implicit declaration of function ‘jsc_context_get_exception’ [-Wimplicit-function-declaration]
     jsc_context_get_exception (jsc_value_get_context (value));
     ^~~~~~~~~~~~~~~~~~~~~~~~~
webkit-module.c:365:32: warning: implicit declaration of function ‘jsc_value_get_context’ [-Wimplicit-function-declaration]
     jsc_context_get_exception (jsc_value_get_context (value));
                                ^~~~~~~~~~~~~~~~~~~~~
webkit-module.c:365:5: warning: initialization makes pointer from integer without a cast [-Wint-conversion]
     jsc_context_get_exception (jsc_value_get_context (value));
     ^~~~~~~~~~~~~~~~~~~~~~~~~
In file included from /usr/include/glib-2.0/glib.h:62:0,
                 from /usr/include/gtk-3.0/gdk/gdkconfig.h:13,
                 from /usr/include/gtk-3.0/gdk/gdk.h:30,
                 from /usr/include/gtk-3.0/gtk/gtk.h:30,
                 from webkit-module.c:3:
webkit-module.c:368:16: warning: implicit declaration of function ‘jsc_exception_get_message’ [-Wimplicit-function-declaration]
                jsc_exception_get_message (exception));
                ^
/usr/include/glib-2.0/glib/gmessages.h:336:32: note: in definition of macro ‘g_warning’
                                __VA_ARGS__)
                                ^~~~~~~~~~~
webkit-module.c:367:16: warning: format ‘%s’ expects argument of type ‘char *’, but argument 4 has type ‘int’ [-Wformat=]
     g_warning ("Error running javascript: %s",
                ^
/usr/include/glib-2.0/glib/gmessages.h:336:32: note: in definition of macro ‘g_warning’
                                __VA_ARGS__)
                                ^~~~~~~~~~~
webkit-module.c: In function ‘webkit_script_message_cb’:
webkit-module.c:503:3: error: unknown type name ‘JSCValue’
   JSCValue *value = webkit_javascript_result_get_js_value (js_result);
   ^~~~~~~~
webkit-module.c:503:21: warning: initialization makes pointer from integer without a cast [-Wint-conversion]
   JSCValue *value = webkit_javascript_result_get_js_value (js_result);
                     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
webkit-module.c:504:17: warning: initialization makes pointer from integer without a cast [-Wint-conversion]
   gchar *json = jsc_value_to_json (value, 1);
                 ^~~~~~~~~~~~~~~~~
webkit-module.c:505:3: error: unknown type name ‘JSCException’
   JSCException *exception =
   ^~~~~~~~~~~~
webkit-module.c:506:5: warning: initialization makes pointer from integer without a cast [-Wint-conversion]
     jsc_context_get_exception (jsc_value_get_context (value));
     ^~~~~~~~~~~~~~~~~~~~~~~~~
In file included from /usr/include/glib-2.0/glib.h:62:0,
                 from /usr/include/gtk-3.0/gdk/gdkconfig.h:13,
                 from /usr/include/gtk-3.0/gdk/gdk.h:30,
                 from /usr/include/gtk-3.0/gtk/gtk.h:30,
                 from webkit-module.c:3:
webkit-module.c:508:16: warning: format ‘%s’ expects argument of type ‘char *’, but argument 4 has type ‘int’ [-Wformat=]
     g_warning ("Error in javascript message recieve: %s",
                ^
/usr/include/glib-2.0/glib/gmessages.h:336:32: note: in definition of macro ‘g_warning’
                                __VA_ARGS__)
                                ^~~~~~~~~~~
webkit-module.c: In function ‘webview_notify_uri’:
webkit-module.c:614:22: warning: initialization makes pointer from integer without a cast [-Wint-conversion]
   const gchar *uri = webkit_uri_for_display (webkit_web_view_get_uri (webview));
                      ^~~~~~~~~~~~~~~~~~~~~~
make: *** [Makefile:10: webkit-module.so] Error 1

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[tomas]

[-- Attachment #1: Type: text/plain, Size: 1770 bytes --]

On Sun, Nov 22, 2020 at 08:18:12PM +0300, Jean Louis wrote:
> * tomas@tuxteam.de <tomas@tuxteam.de> [2020-11-22 19:24]:
> > > > Uh, oh. You seem to be missing fundamental packages for compiling
> > > > things [...]

> > except when it doesn't :)
> 
> You know that all better, thank you. 

Not sure, really.

> I do remember being able to compile webkit in Emacs by compiling Emacs
> itself, but it did not work as expected and I removed it.
> 
> $ locate stdlib.h
> /usr/include/stdlib.h

That seems a good place for it, yes. The compiler will
find that there.

> but webkit-module.c inside of emacs-webkit git directory does not have
> stdlib.h, there is nowhere in other files included. And I included
> also string.h that was missing.
> 
> $ head webkit-module.c 
> #define _POSIX_SOURCE 1
> 
> #include <gtk/gtk.h>
> #include <webkit2/webkit2.h>
> #include <stdbool.h>
> #include <signal.h>
> #include <assert.h>
> #include <errno.h>
> 
> I think it should be included there and I have included those missind
> but then I got these other errors. Maybe my webkit is outdated for
> this one. I cannot install new webkit on this system.

Definitely. But then it's the module's author's turn to chime in.
Obviously it works for him/her, so we must be missing something
important.

> webkit-module.c: In function ‘webkit_get_uri’:
> webkit-module.c:147:26: warning: implicit declaration of function ‘webkit_uri_for_display’ [-Wimplicit-function-declaration]
>        const gchar *uri = webkit_uri_for_display (webkit_web_view_get_uri

Yep. Perhaps your hunch is correct, and you have a different version
of webkit2.h than the author has. Perhaps the author's even includes
<stdlib.h> and friends?

Cheers
-- t

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Eli Zaretskii]

> From: Akira Kyle <akira@akirakyle.com>
> Date: Sat, 21 Nov 2020 20:35:52 -0700
> 
> Awhile ago I started a thread on rethinking the design of 
> xwidgets, there was a lot of good discussion about the future of 
> Emacs' rendering capabilities. It personally motivated me to try 
> to better understand how Emacs' redisplay works and just overall 
> better understand the C core of Emacs. As part of that exercise, I 
> decided to see if I could port the existing xwidgets code into a 
> dynamic module. The only xwidget Emacs currently implements is a 
> webkit widget, so this effort turned into creating a dynamic 
> module for embedding webkit in Emacs. It also happens that I've 
> been wishing that I could never leave Emacs and a general web 
> browser was the last daily app that I begrudgingly used outside of 
> Emacs. Anyways here's the result: 
> https://github.com/akirakyle/emacs-webkit.git

Thanks.

So, IIUC, basically you create a buffer, display it in a window, and
then let GTK draw in the visible portion of that window, is that the
idea?  If so, what happens if some other Lisp inserts some text into
that buffer? does the text overwrite the WebKit display?

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[T.V Raman]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=gb18030, Size: 236 bytes --]

It  would still be even nicer to see a lisp wrapper over webkit a la
nyxt --- then you could actually manipulate higher-level objects, rather
than bits rendered by webkit.
-- 

Thanks,

--Raman
7©4 Id: kg:/m/0285kf1  •0Ü8

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Akira Kyle]

On Sun, Nov 22, 2020 at 05:50 AM, Jean Louis <bugs@gnu.support> 
wrote:

> There are 2 types of questions for you, one group of questions 
> if for
> dynamic module emacs-webkit:
>
> - is it possible to make sure from start of the session not to 
> use
>   Javascript?

Yes,

(add-hook 'webkit-new-hook #'webkit-enable-javascript)

This should disable javascript on all 
sites. `webkit-enable-javascript` can be called interactively to 
disable javascript or enable if given a prefix arg. Although I 
just realized theres a bug here and the page load request will 
happen before the hook is applied. I'll fix this soon.

There's a thread on my post on reddit.com/r/emacs [1] where I talk 
a bit more about the process of disabling JS. To quote my comment 
there:

>> I should note that this sets the webkit 
>> `enable-javascript-markup` setting, not the 'enable-javascript' 
>> setting. I'm still figuring out how all these options are 
>> applied within webkit, but it seems like 
>> `enable-javascript-markup` disables a page from executing 
>> javascript by stripping it all out. `enable-javascript` would 
>> break the module since basic things like scrolling are handled 
>> by executing javascript in the view, which is allowed even if 
>> the page javascript is disabled with 
>> `enable-javascript-markup`. It seems like this is what email 
>> clients that use webkitgtk2 use since using javascript is 
>> really the only way to control the WebView given the WebKitDOM 
>> API is depricated. One could write the equavalent to 
>> window.scrollTo in C using the JavasScriptCore API but AFAIK 
>> that would be pretty much equivalent to what's currently 
>> happening.

> - is it maybe possible at compile time to add an option
>   --disable-javascript ?

I'm not sure if webkit has such an option to disable javascript 
and compile time. I think the JavaScriptCore must be present in 
order for webkit to even work.

> - is there way during the session to surely disable javascript?

See above.

> - it would be very nice if webkit dynamic module could bookmark
>   hyperlink in such a way that it can be programmatically 
>   captured by
>   using Emacs Lisp. For example by pressing B that users can 
>   capture
>   both the TITLE and URL together and store it how they 
>   wish. You
>   could provide standard function and one customizable 
>   function. That
>   way users could store the TITLE and URL without reloading the 
>   page
>   into database, or into Org file, or other system.

Currently C-l copies the url (open to suggestions on improvements 
to the default keybindings as I don't use them myself). And the 
title and url can be retrieved from the module with 
`webkit--get-uri` and `webkit--get-title`. I don't personally use 
emacs builtin bookmarks but enough people seem interested in such 
a feature that it'll happen.

> And some questions not related to the subject, rather to your 
> skills
> with dynamic modules:
>
> Developers of emacs-libpq will soon contribute the dynamic 
> module to
> GNU ELPA that offers bindings to PostgreSQL and that enhances 
> Emacs
> greatly with possibilities. Related to that in GNU there is GDBM 
> or
> GNU database https://www.gnu.org/s/gdbm
>
> I am just asking if you could think of programming the dynamic 
> module
> that offers to users Emacs interface to GDBM. This type of 
> database is
> much simpler and users most probably have it on their systems 
> and it
> can be used for business processes.
>
> Reference to emacs-libpq @ Github
> https://github.com/anse1/emacs-libpq

I'm not very knowledgeable about databases and don't directly 
interact them in my day-to-day so I don't have much of a personal 
motivation to work on such a problem thus I don't think I'm the 
right person for this project.

[1] 
https://www.reddit.com/r/emacs/comments/jyowe0/introducing_emacswebkit_a_successor_to/

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Akira Kyle]

What version of webkitgtk2 do you have? Can you make sure 
`pkg-config` is finding gtk+-3.0 and webkit2gtk-4.0? I think 
<stdlib.h> and co. should be included by gtk and webkit2gtk.

On Sun, Nov 22, 2020 at 10:18 AM, Jean Louis <bugs@gnu.support> 
wrote:

>> except when it doesn't :)
>
> You know that all better, thank you. 
>
> I do remember being able to compile webkit in Emacs by compiling 
> Emacs
> itself, but it did not work as expected and I removed it.
>
> $ locate stdlib.h
> /usr/include/stdlib.h
>
> but webkit-module.c inside of emacs-webkit git directory does 
> not have
> stdlib.h, there is nowhere in other files included. And I 
> included
> also string.h that was missing.
>
> $ head webkit-module.c 
> #define _POSIX_SOURCE 1
>
> #include <gtk/gtk.h>
> #include <webkit2/webkit2.h>
> #include <stdbool.h>
> #include <signal.h>
> #include <assert.h>
> #include <errno.h>
>
> I think it should be included there and I have included those 
> missind
> but then I got these other errors. Maybe my webkit is outdated 
> for
> this one. I cannot install new webkit on this system.
>
> webkit-module.c: In function ‘webkit_get_uri’:
> webkit-module.c:147:26: warning: implicit declaration of 
> function ‘webkit_uri_for_display’ 
> [-Wimplicit-function-declaration]
>        const gchar *uri = webkit_uri_for_display 
>        (webkit_web_view_get_uri
>                           ^~~~~~~~~~~~~~~~~~~~~~
> webkit-module.c:147:26: warning: initialization makes pointer 
> from integer without a cast [-Wint-conversion]
> webkit-module.c: In function ‘webkit_js_finished’:
> webkit-module.c:362:3: error: unknown type name ‘JSCValue’
>    JSCValue *value = webkit_javascript_result_get_js_value 
>    (js_result);
>    ^~~~~~~~
> webkit-module.c:362:21: warning: implicit declaration of 
> function ‘webkit_javascript_result_get_js_value’ 
> [-Wimplicit-function-declaration]
>    JSCValue *value = webkit_javascript_result_get_js_value 
>    (js_result);
>                      ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> webkit-module.c:362:21: warning: initialization makes pointer 
> from integer without a cast [-Wint-conversion]
> webkit-module.c:363:17: warning: implicit declaration of 
> function ‘jsc_value_to_json’ [-Wimplicit-function-declaration]
>    gchar *json = jsc_value_to_json (value, 1);
>                  ^~~~~~~~~~~~~~~~~
> webkit-module.c:363:17: warning: initialization makes pointer 
> from integer without a cast [-Wint-conversion]
> webkit-module.c:364:3: error: unknown type name ‘JSCException’
>    JSCException *exception =
>    ^~~~~~~~~~~~
> webkit-module.c:365:5: warning: implicit declaration of function 
> ‘jsc_context_get_exception’ [-Wimplicit-function-declaration]
>      jsc_context_get_exception (jsc_value_get_context (value));
>      ^~~~~~~~~~~~~~~~~~~~~~~~~
> webkit-module.c:365:32: warning: implicit declaration of 
> function ‘jsc_value_get_context’ 
> [-Wimplicit-function-declaration]
>      jsc_context_get_exception (jsc_value_get_context (value));
>                                 ^~~~~~~~~~~~~~~~~~~~~
> webkit-module.c:365:5: warning: initialization makes pointer 
> from integer without a cast [-Wint-conversion]
>      jsc_context_get_exception (jsc_value_get_context (value));
>      ^~~~~~~~~~~~~~~~~~~~~~~~~
> In file included from /usr/include/glib-2.0/glib.h:62:0,
>                  from /usr/include/gtk-3.0/gdk/gdkconfig.h:13,
>                  from /usr/include/gtk-3.0/gdk/gdk.h:30,
>                  from /usr/include/gtk-3.0/gtk/gtk.h:30,
>                  from webkit-module.c:3:
> webkit-module.c:368:16: warning: implicit declaration of 
> function ‘jsc_exception_get_message’ 
> [-Wimplicit-function-declaration]
>                 jsc_exception_get_message (exception));
>                 ^
> /usr/include/glib-2.0/glib/gmessages.h:336:32: note: in 
> definition of macro ‘g_warning’
>                                 __VA_ARGS__)
>                                 ^~~~~~~~~~~
> webkit-module.c:367:16: warning: format ‘%s’ expects argument of 
> type ‘char *’, but argument 4 has type ‘int’ [-Wformat=]
>      g_warning ("Error running javascript: %s",
>                 ^
> /usr/include/glib-2.0/glib/gmessages.h:336:32: note: in 
> definition of macro ‘g_warning’
>                                 __VA_ARGS__)
>                                 ^~~~~~~~~~~
> webkit-module.c: In function ‘webkit_script_message_cb’:
> webkit-module.c:503:3: error: unknown type name ‘JSCValue’
>    JSCValue *value = webkit_javascript_result_get_js_value 
>    (js_result);
>    ^~~~~~~~
> webkit-module.c:503:21: warning: initialization makes pointer 
> from integer without a cast [-Wint-conversion]
>    JSCValue *value = webkit_javascript_result_get_js_value 
>    (js_result);
>                      ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> webkit-module.c:504:17: warning: initialization makes pointer 
> from integer without a cast [-Wint-conversion]
>    gchar *json = jsc_value_to_json (value, 1);
>                  ^~~~~~~~~~~~~~~~~
> webkit-module.c:505:3: error: unknown type name ‘JSCException’
>    JSCException *exception =
>    ^~~~~~~~~~~~
> webkit-module.c:506:5: warning: initialization makes pointer 
> from integer without a cast [-Wint-conversion]
>      jsc_context_get_exception (jsc_value_get_context (value));
>      ^~~~~~~~~~~~~~~~~~~~~~~~~
> In file included from /usr/include/glib-2.0/glib.h:62:0,
>                  from /usr/include/gtk-3.0/gdk/gdkconfig.h:13,
>                  from /usr/include/gtk-3.0/gdk/gdk.h:30,
>                  from /usr/include/gtk-3.0/gtk/gtk.h:30,
>                  from webkit-module.c:3:
> webkit-module.c:508:16: warning: format ‘%s’ expects argument of 
> type ‘char *’, but argument 4 has type ‘int’ [-Wformat=]
>      g_warning ("Error in javascript message recieve: %s",
>                 ^
> /usr/include/glib-2.0/glib/gmessages.h:336:32: note: in 
> definition of macro ‘g_warning’
>                                 __VA_ARGS__)
>                                 ^~~~~~~~~~~
> webkit-module.c: In function ‘webview_notify_uri’:
> webkit-module.c:614:22: warning: initialization makes pointer 
> from integer without a cast [-Wint-conversion]
>    const gchar *uri = webkit_uri_for_display 
>    (webkit_web_view_get_uri (webview));
>                       ^~~~~~~~~~~~~~~~~~~~~~
> make: *** [Makefile:10: webkit-module.so] Error 1

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Akira Kyle]

On Sun, Nov 22, 2020 at 11:04 AM, Eli Zaretskii <eliz@gnu.org> 
wrote:

>> Awhile ago I started a thread on rethinking the design of 
>> xwidgets, there was a lot of good discussion about the future 
>> of 
>> Emacs' rendering capabilities. It personally motivated me to 
>> try 
>> to better understand how Emacs' redisplay works and just 
>> overall 
>> better understand the C core of Emacs. As part of that 
>> exercise, I 
>> decided to see if I could port the existing xwidgets code into 
>> a 
>> dynamic module. The only xwidget Emacs currently implements is 
>> a 
>> webkit widget, so this effort turned into creating a dynamic 
>> module for embedding webkit in Emacs. It also happens that I've 
>> been wishing that I could never leave Emacs and a general web 
>> browser was the last daily app that I begrudgingly used outside 
>> of 
>> Emacs. Anyways here's the result: 
>> https://github.com/akirakyle/emacs-webkit.git
>
> Thanks.
>
> So, IIUC, basically you create a buffer, display it in a window, 
> and
> then let GTK draw in the visible portion of that window, is that 
> the
> idea?  If so, what happens if some other Lisp inserts some text 
> into
> that buffer? does the text overwrite the WebKit display?

I find the GtkFixed widget that contains the windows of a frame, 
then I add the webkitgtk widget as a child of that GtkFixed. So 
everything under the widget is obscured, including any text 
written into the webkit-mode buffer. In fact I use the buffer for 
processing module->lisp async messages. I monitor for changes in 
the coordinates of the window with 
`window-size-change-functions`. Child frames are also added as 
children of the GtkFixed widget which presents a problem for 
things like posframe. I ensure that the webkit view is always 
below such child frames, as it would be if it were rendered as 
part of the buffer's window itself.

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Akira Kyle]

On Sun, Nov 22, 2020 at 11:29 AM, T.V Raman <raman@google.com> 
wrote:

> It  would still be even nicer to see a lisp wrapper over webkit 
> a la
> nyxt --- then you could actually manipulate higher-level 
> objects, rather
> than bits rendered by webkit.

Unless I'm misunderstanding you, that's pretty much what this 
is. I'm trying to not do too much in the dynamic module, just 
expose webkitgtk functions to lisp along with the ability to embed 
the webkit views in an Emacs window (or in a separate, dedicated 
gtk window if the Emacs window isn't gtk).

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[T.V Raman]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=gb18030, Size: 918 bytes --]

Akira Kyle <akira@akirakyle.com> writes:

apologies if I misunderstood what you were doing then:-)

What I really want is to be able to manipulate the DOM from the Emacs side, specifically with the goal of extracting higher-level structured information from the DOM, > On Sun, Nov 22, 2020 at 11:29 AM, T.V Raman <raman@google.com> wrote:
>
>> It  would still be even nicer to see a lisp wrapper over webkit a la
>> nyxt --- then you could actually manipulate higher-level objects,
>> rather
>> than bits rendered by webkit.
>
> Unless I'm misunderstanding you, that's pretty much what this is. I'm
> trying to not do too much in the dynamic module, just expose webkitgtk
> functions to lisp along with the ability to embed the webkit views in
> an Emacs window (or in a separate, dedicated gtk window if the Emacs
> window isn't gtk).
>

-- 

Thanks,

--Raman
7©4 Id: kg:/m/0285kf1  •0Ü8

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Jean Louis]

* Akira Kyle <akira@akirakyle.com> [2020-11-22 21:40]:
> 
> What version of webkitgtk2 do you have? Can you make sure `pkg-config` is
> finding gtk+-3.0 and webkit2gtk-4.0? I think <stdlib.h> and co. should be
> included by gtk and webkit2gtk.

pkg-config --cflags gtk+-3.0
-pthread -I/usr/include/gtk-3.0 -I/usr/include/at-spi2-atk/2.0 -I/usr/include/at-spi-2.0 -I/usr/include/dbus-1.0 -I/usr/lib/dbus-1.0/include -I/usr/include/gtk-3.0 -I/usr/include/gio-unix-2.0/ -I/usr/include/cairo -I/usr/include/pango-1.0 -I/usr/include/atk-1.0 -I/usr/include/cairo -I/usr/include/pixman-1 -I/usr/include/freetype2 -I/usr/include/libpng16 -I/usr/include/harfbuzz -I/usr/include/glib-2.0 -I/usr/lib/glib-2.0/include -I/usr/include/freetype2 -I/usr/include/harfbuzz -I/usr/include/libdrm -I/usr/include/libpng16 -I/usr/include/gdk-pixbuf-2.0 -I/usr/include/libpng16 -I/usr/include/glib-2.0 -I/usr/lib/glib-2.0/include

pkg-config --libs webkit2gtk-4.0
-lwebkit2gtk-4.0 -lgtk-3 -lgdk-3 -lpangocairo-1.0 -lpango-1.0 -latk-1.0 -lcairo-gobject -lcairo -lgdk_pixbuf-2.0 -lsoup-2.4 -lgio-2.0 -lgobject-2.0 -ljavascriptcoregtk-4.0 -lglib-2.0

I think it is finding.

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Akira Kyle]

On Sun, Nov 22, 2020 at 12:35 PM, T.V Raman <raman@google.com> 
wrote:

> What I really want is to be able to manipulate the DOM from the 
> Emacs side, specifically with the goal of extracting 
> higher-level structured information from the DOM, > On Sun, Nov 
> 22, 2020 at 11:29 AM, T.V Raman <raman@google.com> wrote:

Ah, there's currently the ability to manipulate the DOM by 
injecting JS and sending results back to Emacs as JSON. As far as 
representing and manipulating the DOM directly in elisp, I 
personally don't see that as being in scope for this project, 
however I'm not opposed if others want to explore that. Someone 
asked a similar question on the github repo: "Any plans to 
integrate parenscript or similar?" My response:

>> Not really. I personally think trying to work with parenscript 
>> for only running javascript just makes things needlessly 
>> complicated since you still need to know the whole DOM api to 
>> do anything interesting. Perhaps there's a compelling enough 
>> argument to make a parenscript equivalent in elisp, but for now 
>> I'm sticking with using JSON to communicated between JS and 
>> elisp.

I think nyxt is doing a good job with this and would refer people 
interested in a lispy DOM to their project: 
https://github.com/atlas-engineer/nyxt

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Akira Kyle]

On Sun, Nov 22, 2020 at 12:58 PM, Jean Louis <bugs@gnu.support> 
wrote:

>> What version of webkitgtk2 do you have? Can you make sure 
>> `pkg-config` is
>> finding gtk+-3.0 and webkit2gtk-4.0? I think <stdlib.h> and 
>> co. should be
>> included by gtk and webkit2gtk.
>
> pkg-config --cflags gtk+-3.0
> -pthread -I/usr/include/gtk-3.0 -I/usr/include/at-spi2-atk/2.0 
> -I/usr/include/at-spi-2.0 -I/usr/include/dbus-1.0 
> -I/usr/lib/dbus-1.0/include -I/usr/include/gtk-3.0 
> -I/usr/include/gio-unix-2.0/ -I/usr/include/cairo 
> -I/usr/include/pango-1.0 -I/usr/include/atk-1.0 
> -I/usr/include/cairo -I/usr/include/pixman-1 
> -I/usr/include/freetype2 -I/usr/include/libpng16 
> -I/usr/include/harfbuzz -I/usr/include/glib-2.0 
> -I/usr/lib/glib-2.0/include -I/usr/include/freetype2 
> -I/usr/include/harfbuzz -I/usr/include/libdrm 
> -I/usr/include/libpng16 -I/usr/include/gdk-pixbuf-2.0 
> -I/usr/include/libpng16 -I/usr/include/glib-2.0 
> -I/usr/lib/glib-2.0/include
>
> pkg-config --libs webkit2gtk-4.0
> -lwebkit2gtk-4.0 -lgtk-3 -lgdk-3 -lpangocairo-1.0 -lpango-1.0 
> -latk-1.0 -lcairo-gobject -lcairo -lgdk_pixbuf-2.0 -lsoup-2.4 
> -lgio-2.0 -lgobject-2.0 -ljavascriptcoregtk-4.0 -lglib-2.0
>
> I think it is finding.

And your webkitgtk version?

Others seem to be having success building this on their systems so 
perhaps see if you can build the minimal webkitgtk example here: 
https://wiki.gnome.org/Projects/WebKitGtk/ProgrammingGuide/Tutorial

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Jean Louis]

* Akira Kyle <akira@akirakyle.com> [2020-11-22 23:30]:
> 
> On Sun, Nov 22, 2020 at 12:58 PM, Jean Louis <bugs@gnu.support> wrote:
> 
> > > What version of webkitgtk2 do you have? Can you make sure
> > > `pkg-config` is
> > > finding gtk+-3.0 and webkit2gtk-4.0? I think <stdlib.h> and co.
> > > should be
> > > included by gtk and webkit2gtk.
> > 
> > pkg-config --cflags gtk+-3.0
> > -pthread -I/usr/include/gtk-3.0 -I/usr/include/at-spi2-atk/2.0
> > -I/usr/include/at-spi-2.0 -I/usr/include/dbus-1.0
> > -I/usr/lib/dbus-1.0/include -I/usr/include/gtk-3.0
> > -I/usr/include/gio-unix-2.0/ -I/usr/include/cairo
> > -I/usr/include/pango-1.0 -I/usr/include/atk-1.0 -I/usr/include/cairo
> > -I/usr/include/pixman-1 -I/usr/include/freetype2 -I/usr/include/libpng16
> > -I/usr/include/harfbuzz -I/usr/include/glib-2.0
> > -I/usr/lib/glib-2.0/include -I/usr/include/freetype2
> > -I/usr/include/harfbuzz -I/usr/include/libdrm -I/usr/include/libpng16
> > -I/usr/include/gdk-pixbuf-2.0 -I/usr/include/libpng16
> > -I/usr/include/glib-2.0 -I/usr/lib/glib-2.0/include
> > 
> > pkg-config --libs webkit2gtk-4.0
> > -lwebkit2gtk-4.0 -lgtk-3 -lgdk-3 -lpangocairo-1.0 -lpango-1.0 -latk-1.0
> > -lcairo-gobject -lcairo -lgdk_pixbuf-2.0 -lsoup-2.4 -lgio-2.0
> > -lgobject-2.0 -ljavascriptcoregtk-4.0 -lglib-2.0
> > 
> > I think it is finding.
> 
> And your webkitgtk version?

extra/webkit2gtk 2.16.1-3.hyperbola1 [installed]
    GTK+ Web content engine library, without geoclue2 support

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Vasilij Schneidermann]

[-- Attachment #1: Type: text/plain, Size: 852 bytes --]

> I find the GtkFixed widget that contains the windows of a frame, then I add
> the webkitgtk widget as a child of that GtkFixed. So everything under the
> widget is obscured, including any text written into the webkit-mode buffer.
> In fact I use the buffer for processing module->lisp async messages. I
> monitor for changes in the coordinates of the window with
> `window-size-change-functions`. Child frames are also added as children of
> the GtkFixed widget which presents a problem for things like posframe. I
> ensure that the webkit view is always below such child frames, as it would
> be if it were rendered as part of the buffer's window itself.

Could this approach be used to perform OpenGL drawing or obtaining a
cairo context to draw on? I'm not terribly interested in putting a
webview inside Emacs, 3D/2D graphics however...

Vasilij

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Akira Kyle]

On Sun, Nov 22, 2020 at 02:20 PM, Vasilij Schneidermann 
<mail@vasilij.de> wrote:

>> I find the GtkFixed widget that contains the windows of a 
>> frame, then I add
>> the webkitgtk widget as a child of that GtkFixed. So everything 
>> under the
>> widget is obscured, including any text written into the 
>> webkit-mode buffer.
>> In fact I use the buffer for processing module->lisp async 
>> messages. I
>> monitor for changes in the coordinates of the window with
>> `window-size-change-functions`. Child frames are also added as 
>> children of
>> the GtkFixed widget which presents a problem for things like 
>> posframe. I
>> ensure that the webkit view is always below such child frames, 
>> as it would
>> be if it were rendered as part of the buffer's window itself.
>
> Could this approach be used to perform OpenGL drawing or 
> obtaining a
> cairo context to draw on? I'm not terribly interested in putting 
> a
> webview inside Emacs, 3D/2D graphics however...
>
> Vasilij

Yes, the webkitgtk widget is ultimately just drawing itself on 
cario or opengl contexts. So if you only care about taking over 
the display of a window, this approach should work fine. I'm not 
sure it would be wise, or necessarily possible, to use this 
approach to mix elements drawn with GtkWidgets and Emacs' own 
display code.

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[T.V Raman]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=gb18030, Size: 2022 bytes --]

Akira Kyle <akira@akirakyle.com> writes:


Doing the manipulation with injected jSON would be fin e bye me.

I did look at parenscript in the context of nyxt,
and it looks interesting, but I dont have anything religious  re the
shape of delimiters I use to write code :-)

My interest stems primarily from wanting to speak Web content
intelligently and EWW gets me a lot of that, except where Web Content
is dynamically generated from JS and  given that, I'm looking to get
Webkit to build the DOM, and give it to me in a form that I can query
it, manipulate it, and finally speak the relevant bits.

> On Sun, Nov 22, 2020 at 12:35 PM, T.V Raman <raman@google.com> wrote:
>
>> What I really want is to be able to manipulate the DOM from the
>> Emacs side, specifically with the goal of extracting higher-level
>> structured information from the DOM, > On Sun, Nov 22, 2020 at 11:29
>> AM, T.V Raman <raman@google.com> wrote:
>
> Ah, there's currently the ability to manipulate the DOM by injecting
> JS and sending results back to Emacs as JSON. As far as representing
> and manipulating the DOM directly in elisp, I personally don't see
> that as being in scope for this project, however I'm not opposed if
> others want to explore that. Someone asked a similar question on the
> github repo: "Any plans to integrate parenscript or similar?" My
> response:
>
>>> Not really. I personally think trying to work with parenscript for
>>> only running javascript just makes things needlessly complicated
>>> since you still need to know the whole DOM api to do anything
>>> interesting. Perhaps there's a compelling enough argument to make a
>>> parenscript equivalent in elisp, but for now I'm sticking with
>>> using JSON to communicated between JS and elisp.
>
> I think nyxt is doing a good job with this and would refer people
> interested in a lispy DOM to their project:
> https://github.com/atlas-engineer/nyxt
>

-- 

Thanks,

--Raman
7©4 Id: kg:/m/0285kf1  •0Ü8

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[T.V Raman]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=gb18030, Size: 431 bytes --]

Got it built but it crashed on me.

Hi,

I git checked out and built emacs-webkit, built happily, no warnings or
errors. Read through readme.org, and it got me really excited.

Added the directory to load-path, and did a simple load-library of
webkit

Sadly, M-x webkit  called with either a search term, or a google.com
search URL immediately crashed emacs
-- 

Thanks,

--Raman
7©4 Id: kg:/m/0285kf1  •0Ü8

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Akira Kyle]

On Sun, Nov 22, 2020 at 05:46 PM, T.V Raman <raman@google.com> 
wrote:

> Doing the manipulation with injected jSON would be fin e bye me.
>
> I did look at parenscript in the context of nyxt,
> and it looks interesting, but I dont have anything religious  re 
> the
> shape of delimiters I use to write code :-)
>
> My interest stems primarily from wanting to speak Web content
> intelligently and EWW gets me a lot of that, except where Web 
> Content
> is dynamically generated from JS and  given that, I'm looking to 
> get
> Webkit to build the DOM, and give it to me in a form that I can 
> query
> it, manipulate it, and finally speak the relevant bits.

Here's a proof of concept of that. Don't try it on any large or 
complicated page as it'll grind Emacs to a halt trying to do the 
parsing. 

(setq webkit--to-json-js "
function toJSON(node) {
  let propFix = { for: 'htmlFor', class: 'className' };
  let specialGetters = {
    style: (node) => node.style.cssText,
  };
  let attrDefaultValues = { style: '' };
  let obj = {
    nodeType: node.nodeType,
  };
  if (node.tagName) {
    obj.tagName = node.tagName.toLowerCase();
  } else if (node.nodeName) {
    obj.nodeName = node.nodeName;
  }
  if (node.nodeValue) {
    obj.nodeValue = node.nodeValue;
  }
  let attrs = node.attributes;
  if (attrs) {
    let defaultValues = new Map();
    for (let i = 0; i < attrs.length; i++) {
      let name = attrs[i].nodeName;
      defaultValues.set(name, attrDefaultValues[name]);
    }
    // Add some special cases that might not be included by 
    enumerating
    // attributes above. Note: this list is probably not 
    exhaustive.
    switch (obj.tagName) {
      case 'input': {
        if (node.type === 'checkbox' || node.type === 'radio') {
          defaultValues.set('checked', false);
        } else if (node.type !== 'file') {
          // Don't store the value for a file input.
          defaultValues.set('value', '');
        }
        break;
      }
      case 'option': {
        defaultValues.set('selected', false);
        break;
      }
      case 'textarea': {
        defaultValues.set('value', '');
        break;
      }
    }
    let arr = [];
    for (let [name, defaultValue] of defaultValues) {
      let propName = propFix[name] || name;
      let specialGetter = specialGetters[propName];
      let value = specialGetter ? specialGetter(node) : 
      node[propName];
      if (value !== defaultValue) {
        arr.push([name, value]);
      }
    }
    if (arr.length) {
      obj.attributes = arr;
    }
  }
  let childNodes = node.childNodes;
  // Don't process children for a textarea since we used `value` 
  above.
  if (obj.tagName !== 'textarea' && childNodes && 
  childNodes.length) {
    let arr = (obj.childNodes = []);
    for (let i = 0; i < childNodes.length; i++) {
      arr[i] = toJSON(childNodes[i]);
    }
  }
  return obj;
}
toJSON(document);
")

(defun webkit--save-json (msg)
  (setq webkit--json (json-parse-string msg)))

(webkit--execute-js
 (with-current-buffer (car webkit--buffers) webkit--id)
 webkit--to-json-js "webkit--save-json")


I think through this general framework one could more efficiently 
extract the parts of the DOM one is interested in rather than 
dumping it all into elisp like I did above.

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Akira Kyle]

On Sun, Nov 22, 2020 at 08:01 PM, T.V Raman <raman@google.com> 
wrote:

> Got it built but it crashed on me.
>
> Hi,
>
> I git checked out and built emacs-webkit, built happily, no 
> warnings or
> errors. Read through readme.org, and it got me really excited.
>
> Added the directory to load-path, and did a simple load-library 
> of
> webkit
>
> Sadly, M-x webkit  called with either a search term, or a 
> google.com
> search URL immediately crashed emacs

Oh no! I assume this is a segfault? Can you try building with make 
debug and see if you get any more useful info? Perhaps a backtrace 
from gdb?

Someone posted a similar issue and it was due to them not running 
Emacs 28, but I just recently added a check to give an error 
message.

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Lars Ingebrigtsen]

Akira Kyle <akira@akirakyle.com> writes:

> Essentially it opens a dedicated window for displaying webkit which
> can be controlled through the keybindings in the emacs buffer
> corresponding to that webkit view (of course one needs to have running
> a running graphical session pointed to in $DISPLAY).

This is great news -- the demo looks awesome.

Just a couple of comments (without having actually tried it or even
looked at the code):

One thing I hoped to do with the in-tree xwidget code was to use it in
eww to display media types Emacs doesn't support -- primarily .mp4
<video> elements (and youtube videos, for those who doesn't find that
abhorrent).  I poked at it for a bit, but the current code is very much
tied to the xwidgets mode owning the buffer, and I haven't yet had the
stamina to fix that.

Is this easier with your package?  That is, the ability to plop in an
arbitrary number of widgets into any buffer?

The other thing I'm wondering about is the (pure) reliance of Cairo (if
I understood you correctly).  It seems like Cairo development is slowing
down as many projects have started using Skia instead.  Emacs supports a
number of graphics backends for a reason -- they pop up, last a few
years, and then they die off, and there's no reason to believe that
Cairo is going to be the One True Toolkit.  Toolkits come and go; Emacs
is forever.  :-)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[joakim]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Akira Kyle <akira@akirakyle.com> writes:
>
>> Essentially it opens a dedicated window for displaying webkit which
>> can be controlled through the keybindings in the emacs buffer
>> corresponding to that webkit view (of course one needs to have running
>> a running graphical session pointed to in $DISPLAY).
>
> This is great news -- the demo looks awesome.
>
> Just a couple of comments (without having actually tried it or even
> looked at the code):
>
> One thing I hoped to do with the in-tree xwidget code was to use it in
> eww to display media types Emacs doesn't support -- primarily .mp4
> <video> elements (and youtube videos, for those who doesn't find that
> abhorrent).  I poked at it for a bit, but the current code is very much
> tied to the xwidgets mode owning the buffer, and I haven't yet had the
> stamina to fix that.

The original design of xwidget were supposed to work like inserting
images in a buffer, so they werent originally supposed to own the buffer
or anything. It was possible to have several xwidgets in a buffer and so
on.

Maybe thats changed, I haven't been able to follow the development
recently. 

>
> Is this easier with your package?  That is, the ability to plop in an
> arbitrary number of widgets into any buffer?

>
> The other thing I'm wondering about is the (pure) reliance of Cairo (if
> I understood you correctly).  It seems like Cairo development is slowing
> down as many projects have started using Skia instead.  Emacs supports a
> number of graphics backends for a reason -- they pop up, last a few
> years, and then they die off, and there's no reason to believe that
> Cairo is going to be the One True Toolkit.  Toolkits come and go; Emacs
> is forever.  :-)
-- 
Joakim Verona
joakim@verona.se

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Arthur Miller]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Akira Kyle <akira@akirakyle.com> writes:
>
>> Essentially it opens a dedicated window for displaying webkit which
>> can be controlled through the keybindings in the emacs buffer
>> corresponding to that webkit view (of course one needs to have running
>> a running graphical session pointed to in $DISPLAY).
>
> This is great news -- the demo looks awesome.
>
> Just a couple of comments (without having actually tried it or even
> looked at the code):
>
> One thing I hoped to do with the in-tree xwidget code was to use it in
> eww to display media types Emacs doesn't support -- primarily .mp4
> <video> elements (and youtube videos, for those who doesn't find that
> abhorrent).
Just a curious question: have you ever looked at libmpv?
Would it be possible to embedd video via livmpv in an image container
just as jpg, png etc?

libmpv is a C library for mpv player ment to be embedded, and works in
all 3 major desktop platforms.

https://github.com/mpv-player/mpv-examples/tree/master/libmpv

Mpv itself comes with crap-load of video codecs and image formats already
built-in; it is a fork mplayer2. What is good, if one could create a
OpenGL context for an Emacs window (X11/Win32 Window - shouldn't be
hard), you could controll rendering directly, since they don't take over
your even loop:

"You will have to simulate this with the mouse/keypress/keydown/keyup commands."

It would probably fit into Emacs well.

I don't know myself how to implement "new image format" for Emacs; too
much to look through the spaghetty of if-defs in image.c, 

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Jose A. Ortega Ruiz]

[...]

> Someone posted a similar issue and it was due to them not running
> Emacs 28, but I just recently added a check to give an error message.

i have that problem in debian/sid, with an emacs compiled from today's
master branch (but not the pgtk one).  the error happens when loading
the .so file, and the error message says that a symbol dynamically
linked in it (gtk_init_<something>, i lost the trace, sorry) is not
found.  so it looks as if some ldconfig step or something of the sort
were missing, but i haven't had time to investigate it furthere.

cheers,
jao
--
A man has to live with himself, and he should see to it that he always has
good company. -Charles Evans Hughes, jurist (1862-1948)

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Alexander Adolf]

A very personal caution with respect to webkitgtk2 on macOS.

We are using webkitgtk in another OSS project, and building it in macOS
is a huge pain. You have to patch it (and devise new patches when a new
version is released), and compiling it takes hours (literally). When a
new webkitgtk version comes out, the pain starts all over again. This
has gotten to the point that currently there is no macOS build of that
OSS project.

And on top you have a webkit implementation readily available on the
macOS platform. Hence, we are actively seeking to either replace
webkitgtk with something that works cross-platform, or - in case there
isn't any such beast - to write a GTK-style wrapper for the macOS
webkit.

Thus, my very personal take would be that making webkitgtk a requirement
is probably not a promising approach if macOS is to remain on the list
of supported operating systems of this package (assuming that it would
end up being a package, of course).


Looking forward to your thoughts,

  --alexander

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Alan Third]

On Mon, Nov 23, 2020 at 03:59:31PM +0100, Arthur Miller wrote:
> Just a curious question: have you ever looked at libmpv?
> Would it be possible to embedd video via livmpv in an image container
> just as jpg, png etc?
> 
> libmpv is a C library for mpv player ment to be embedded, and works in
> all 3 major desktop platforms.
> 
> https://github.com/mpv-player/mpv-examples/tree/master/libmpv
> 
> Mpv itself comes with crap-load of video codecs and image formats already
> built-in; it is a fork mplayer2. What is good, if one could create a
> OpenGL context for an Emacs window (X11/Win32 Window - shouldn't be
> hard), you could controll rendering directly, since they don't take over
> your even loop:
> 
> "You will have to simulate this with the mouse/keypress/keydown/keyup commands."
> 
> It would probably fit into Emacs well.
> 
> I don't know myself how to implement "new image format" for Emacs; too
> much to look through the spaghetty of if-defs in image.c, 

IIRC it's actually possible to load videos using the imagemagick
backend, the trouble is that Emacs can't really handle them as it
decodes and stores each frame individually.

Emacs has no way to "remember" where it is in a file, as each image is
loaded and stored individually.

For example, to display an animated gif Emacs opens the file, decodes
frame one, closes the file and then displays the frame. When asked to
display frame two it opens the file, decodes frame two (which in the
case of a gif involves decoding frame one again) closes the file and
displays it. To display frame three it opens the file, decodes frame
three (which involves decoding frames one and two again), closes the
file and displays it. And so on and so on.

I think it's like this because it was only intended for displaying
occasional graphics like the fringe bitmaps and similar, and, IMHO, to
do anything different would require a major overhaul of image.c, and
probably the lisp image API while maintaining backwards compatibility.

-- 
Alan Third

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[T.V Raman]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=gb18030, Size: 1329 bytes --]

Alexander Adolf <alexander.adolf@condition-alpha.com> writes:

does blink have the same issues? I remember seeing something about qt
dependency re blink that I didn't quite follow.> A very personal caution with respect to webkitgtk2 on macOS.
>
> We are using webkitgtk in another OSS project, and building it in macOS
> is a huge pain. You have to patch it (and devise new patches when a new
> version is released), and compiling it takes hours (literally). When a
> new webkitgtk version comes out, the pain starts all over again. This
> has gotten to the point that currently there is no macOS build of that
> OSS project.
>
> And on top you have a webkit implementation readily available on the
> macOS platform. Hence, we are actively seeking to either replace
> webkitgtk with something that works cross-platform, or - in case there
> isn't any such beast - to write a GTK-style wrapper for the macOS
> webkit.
>
> Thus, my very personal take would be that making webkitgtk a requirement
> is probably not a promising approach if macOS is to remain on the list
> of supported operating systems of this package (assuming that it would
> end up being a package, of course).
>
>
> Looking forward to your thoughts,
>
>   --alexander
>

-- 

Thanks,

--Raman
7©4 Id: kg:/m/0285kf1  •0Ü8

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Akira Kyle]

On Mon, Nov 23, 2020 at 02:39 AM, Lars Ingebrigtsen 
<larsi@gnus.org> wrote:

> This is great news -- the demo looks awesome.

Thanks!

> Just a couple of comments (without having actually tried it or 
> even
> looked at the code):
>
> One thing I hoped to do with the in-tree xwidget code was to use 
> it in
> eww to display media types Emacs doesn't support -- primarily 
> .mp4
> <video> elements (and youtube videos, for those who doesn't find 
> that
> abhorrent).  I poked at it for a bit, but the current code is 
> very much
> tied to the xwidgets mode owning the buffer, and I haven't yet 
> had the
> stamina to fix that.
>
> Is this easier with your package?  That is, the ability to plop 
> in an
> arbitrary number of widgets into any buffer?

Possibly, but I'm not sure it would be wise to try. I originally 
started with the current xwidget code, which does support such 
capabilities. However, I found what I think are fundamental 
conceptual incompatibilities with the way Emacs' redisplay and 
gtk's event loop want to handle drawing stuff to the screen. Thus 
I feel like the only sane way to mix the two is to allow gtk to 
just take over an Emacs window which is always rectangular. No 
need to worry about scrolling and all the redisplay optimizations 
that will try to interfere with gtk attempting to draw at some 
point in the buffer. Even then Emacs and gtk don't play very 
nicely when it comes to grabbing keyboard and mouse input. Not to 
mention the fundamental differences in UI paradigms between Emacs 
being primarily textual and gtk with it's eye candy interfaces. Is 
it really so wise to try to mix the two?

Thus the second half of my previous email where I speculated about 
ways to move the fundamental ways Emacs does rendering forward. I 
think if Emacs had an open gl surface it was drawing to, it could 
handle rendering more intense graphical sources such as from a 
video. It will take a lot of work to get there, but I can't 
imagine another way that wont feel at least somewhat hacky to 
render something as graphically intensive like a video in a 
buffer.

> The other thing I'm wondering about is the (pure) reliance of 
> Cairo (if
> I understood you correctly).  It seems like Cairo development is 
> slowing
> down as many projects have started using Skia instead.  Emacs 
> supports a
> number of graphics backends for a reason -- they pop up, last a 
> few
> years, and then they die off, and there's no reason to believe 
> that
> Cairo is going to be the One True Toolkit.  Toolkits come and 
> go; Emacs
> is forever.  :-)

I think the death of Cairo has been greatly exaggerated :) The 
recent hacker news post about it brought up some good points about 
this expectation that projects have daily commits for it to be 
considered "alive". I don't see Cairo going anywhere for awhile, 
at least so long as projects like gtk and gnome rely on it. Also I 
wouldn't characterize Cairo as a "toolkit" at least in the sense 
that gtk is a toolkit or motif is a toolkit. Cairo is more like a 
nice graphics api, and the thing I've come to realize about nice 
api's is they tend to stick around in one form or another (kind of 
like emacs has).

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Akira Kyle]

On Mon, Nov 23, 2020 at 03:12 PM, Alexander Adolf 
<alexander.adolf@condition-alpha.com> wrote:

> A very personal caution with respect to webkitgtk2 on macOS.
>
> We are using webkitgtk in another OSS project, and building it 
> in macOS
> is a huge pain. You have to patch it (and devise new patches 
> when a new
> version is released), and compiling it takes hours 
> (literally). When a
> new webkitgtk version comes out, the pain starts all over 
> again. This
> has gotten to the point that currently there is no macOS build 
> of that
> OSS project.

Thankfully I haven't committed myself to ever needing to build 
webkitgtk with this project, it's just a dependency relegated to a 
package manager (unless you run GNU/Linux From Scratch).

> And on top you have a webkit implementation readily available on 
> the
> macOS platform. Hence, we are actively seeking to either replace
> webkitgtk with something that works cross-platform, or - in case 
> there
> isn't any such beast - to write a GTK-style wrapper for the 
> macOS
> webkit.
>
> Thus, my very personal take would be that making webkitgtk a 
> requirement
> is probably not a promising approach if macOS is to remain on 
> the list
> of supported operating systems of this package (assuming that it 
> would
> end up being a package, of course).
> Looking forward to your thoughts,

It's not even on the list of supported systems right now, so I 
think I'm safe. In fact part of my motivation for emacs-webkit was 
precisely because the recently merged xwidget webkit for ns 
actually seemed to work better than the webkitgtk one. Actually on 
x11 the webkitgtk xwidget is pretty much unusable because it 
constantly flickers, but so does emacs-webkit and I haven't been 
able to figure out why.

If you do write a GTK-style wrapper for macOS's ns webkit, that 
would certainly make it easier to run there, however the lisp 
portion of emacs-webkit is agnostic to the C module's 
implementation which mostly just wraps around the necessary 
functions of webkitgtk's api. Since it's webkit under the hood the 
ns webkit api is pretty much the same, just in objective-c (or 
swift). So it should just be a as straight forward as writing a 
different dynamic module for ns webkit, but which exports the same 
C functions as the current webkitgtk one. I don't run macOS 
anymore as they've increasingly made it hostile to customize and 
develop on outside of xcode, which can't hold a candle to emacs of 
course (not to mention its sooo not free). Maybe if someone sent 
me one of those shiny new apple silicon macbooks I'd make an 
emacs-webkit which worked on macOS, but I'm pretty fond of my 
current Pinebook Pro ;) 

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Akira Kyle]

On Mon, Nov 23, 2020 at 04:47 PM, Alan Third <alan@idiocy.org> 
wrote:

>> Just a curious question: have you ever looked at libmpv?
>> Would it be possible to embedd video via livmpv in an image 
>> container
>> just as jpg, png etc?
>> 
>> libmpv is a C library for mpv player ment to be embedded, and 
>> works in
>> all 3 major desktop platforms.
>> 
>> https://github.com/mpv-player/mpv-examples/tree/master/libmpv
>> 
>> Mpv itself comes with crap-load of video codecs and image 
>> formats already
>> built-in; it is a fork mplayer2. What is good, if one could 
>> create a
>> OpenGL context for an Emacs window (X11/Win32 Window - 
>> shouldn't be
>> hard), you could controll rendering directly, since they don't 
>> take over
>> your even loop:
>> 
>> "You will have to simulate this with the 
>> mouse/keypress/keydown/keyup commands."
>> 
>> It would probably fit into Emacs well.
>> 
>> I don't know myself how to implement "new image format" for 
>> Emacs; too
>> much to look through the spaghetty of if-defs in image.c, 
>
> IIRC it's actually possible to load videos using the imagemagick
> backend, the trouble is that Emacs can't really handle them as 
> it
> decodes and stores each frame individually.
>
> Emacs has no way to "remember" where it is in a file, as each 
> image is
> loaded and stored individually.
>
> For example, to display an animated gif Emacs opens the file, 
> decodes
> frame one, closes the file and then displays the frame. When 
> asked to
> display frame two it opens the file, decodes frame two (which in 
> the
> case of a gif involves decoding frame one again) closes the file 
> and
> displays it. To display frame three it opens the file, decodes 
> frame
> three (which involves decoding frames one and two again), closes 
> the
> file and displays it. And so on and so on.
>
> I think it's like this because it was only intended for 
> displaying
> occasional graphics like the fringe bitmaps and similar, and, 
> IMHO, to
> do anything different would require a major overhaul of image.c, 
> and
> probably the lisp image API while maintaining backwards 
> compatibility.

I second this and would also add that I think the emacs event loop 
would need modification as well, since my understanding is that 
it's pretty lazy, rendering things only when it needs to. So while 
its good that mpv wouldn't demand taking over the event loop, I 
think emacs might really struggle to display video frames at 30fps 
without some modifications or work around.

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Just a curious question: have you ever looked at libmpv?
  > Would it be possible to embedd video via livmpv in an image container
  > just as jpg, png etc?

Would people please present a description of what this plan aims to do?
There may be ethical questions that need to be considered, as well as
technical design questions.

For instance, I am concerned that it may lead to extending Emacs using
Javascript programs without verification of their licenses, perhaps
picking them up off the internet without checking them.

Or perhaps it doesn't do anything like that.  But I can't assume that,
since I just don't know.  Would you please fill me in?

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Lars Ingebrigtsen]

joakim@verona.se writes:

> The original design of xwidget were supposed to work like inserting
> images in a buffer, so they werent originally supposed to own the buffer
> or anything. It was possible to have several xwidgets in a buffer and so
> on.

I tried making this work in eww a few years back, but I gave up after
poking at it for a couple of hours.  It could totally be my fault for
using the wrong functions or something.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Lars Ingebrigtsen]

Arthur Miller <arthur.miller@live.com> writes:

> Just a curious question: have you ever looked at libmpv?
> Would it be possible to embedd video via livmpv in an image container
> just as jpg, png etc?

I think that would be difficult -- the Emacs image layer is very
persnickety.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Lars Ingebrigtsen]

Richard Stallman <rms@gnu.org> writes:

> Or perhaps it doesn't do anything like that.  But I can't assume that,
> since I just don't know.  Would you please fill me in?

libmpv is a library to play video files like .mp4.  There's no
Javascript involved.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Lars Ingebrigtsen]

Akira Kyle <akira@akirakyle.com> writes:

> Thus I feel like the only sane way to mix the two is to allow gtk to
> just take over an Emacs window which is always rectangular. No need to
> worry about scrolling and all the redisplay optimizations that will
> try to interfere with gtk attempting to draw at some point in the
> buffer.

I think this design reduces the use cases for emacs-webkit considerably
-- if all you can do with it is to let it take over a window completely,
then that's disappointing.  

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Arthur Miller]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]

> Would people please present a description of what this plan aims to
> do?
Yes of of course! Unfortunately; I am affraid there isn't much of a
plan.

> Or perhaps it doesn't do anything like that.  But I can't assume that,
> since I just don't know.  Would you please fill me in?

libmpv is just a library to play media files, has nothing to do with
javascript. You can check yourself:

 mpv is a free (as in freedom) media player for the command line. It
 supports a wide variety of media file formats, audio and video codecs,
 and subtitle types.

https://mpv.io/

Included codecs are Free too. I think most of codecs come from ffmpeg
which comes parts licensed in lgpl 2.1 and parts in gpl 2 license:
https://ffmpeg.org/legal.html

And list of included codecs:
https://ffmpeg.org/general.html#Supported-File-Formats_002c-Codecs-or-Features

MPV also comes in form of a shared .so with C interface, which (I think)
is well suited to interface with from Emacs. I hoped it could be
embedded into Emacs, similar to libjpeg or libpng to make Emacs display
video in a buffer in same way as it is displaying images. I was actually
looking at it myself like a two, three years or more ago, but I couldn't
figure out image.c. Now when Lars brought up shr.el and <video> tag, I
thought he can maybe bring it into Emacs for the use in shr.el, since he
is probably (much) more introduced into image.c if-defs and Emacs
internals in general then I will probably every be in soon future to
come. At least I think he is. I am using mpv as my video player normally
but I would prefer it to be embedded in Emacs framem. That was the
*master* plan :-). 

> For instance, I am concerned that it may lead to extending Emacs using
> Javascript programs without verification of their licenses, perhaps
> picking them up off the internet without checking them.
I am not sure, but I wouldn't think that playing sound and video in
Emacs would lead to more Javascript than other media formats like images
or html processing in general. By the way, how do you verify if JS is
free or not? Do you have an Emacs script that can download it and check
the license? That could be an useful part of Emacs maybe? Or maybe it
already is, I am horrible at looking what there is, I just hack what I
need when I need it, I am not so much for smörgåsbord in general.

> There may be ethical questions that need to be considered,
> as well as
> technical design questions.
Yes, I understand your concerns, but as above; I don't think there
should be more concerns about libmpv then what it is with libjpeg,
libpng, libtiff and similar. But I am not an expert in legal issues; am
actually very bad at those, so correct me if I am wrong.

Btw; I also own you an unanswered mail since few weeks ago; I am sorry,
I wanted to formulate more than just a hasty answer, but then I never
got back to it.

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Arthur Miller]

Alan Third <alan@idiocy.org> writes:

> On Mon, Nov 23, 2020 at 03:59:31PM +0100, Arthur Miller wrote:
>> Just a curious question: have you ever looked at libmpv?
>> Would it be possible to embedd video via livmpv in an image container
>> just as jpg, png etc?
>> 
>> libmpv is a C library for mpv player ment to be embedded, and works in
>> all 3 major desktop platforms.
>> 
>> https://github.com/mpv-player/mpv-examples/tree/master/libmpv
>> 
>> Mpv itself comes with crap-load of video codecs and image formats already
>> built-in; it is a fork mplayer2. What is good, if one could create a
>> OpenGL context for an Emacs window (X11/Win32 Window - shouldn't be
>> hard), you could controll rendering directly, since they don't take over
>> your even loop:
>> 
>> "You will have to simulate this with the mouse/keypress/keydown/keyup commands."
>> 
>> It would probably fit into Emacs well.
>> 
>> I don't know myself how to implement "new image format" for Emacs; too
>> much to look through the spaghetty of if-defs in image.c, 
>
> IIRC it's actually possible to load videos using the imagemagick
> backend, the trouble is that Emacs can't really handle them as it
> decodes and stores each frame individually.
>
> Emacs has no way to "remember" where it is in a file, as each image is
> loaded and stored individually.
>
> For example, to display an animated gif Emacs opens the file, decodes
> frame one, closes the file and then displays the frame. When asked to
> display frame two it opens the file, decodes frame two (which in the
> case of a gif involves decoding frame one again) closes the file and
> displays it. To display frame three it opens the file, decodes frame
> three (which involves decoding frames one and two again), closes the
> file and displays it. And so on and so on.
Ouch, that sounds like very inneficient way to do things (and very
Eamcs-y too :-)).

> I think it's like this because it was only intended for displaying
> occasional graphics like the fringe bitmaps and similar, and, IMHO, to
> do anything different would require a major overhaul of image.c, and
> probably the lisp image API while maintaining backwards compatibility.
Given a place to render, libmpv does it's own rendering, it requires
just a place to render, and it has it's own event loop so Emacs does not
need to do so much more then just provide a rectangle for it to render.
I thought it would be maybe possible to do same as with jpegs and pngs,
but I never figured out how they are placed in an Emacs frame myself :-).

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Arthur Miller]

>> new webkitgtk version comes out, the pain starts all over again.
Yeah Apple and Google are not best friends, Google diverged it's
own webkit version from Apple's they couldn't really agree with
development directions, so it is not surprising you are experiencing
problems you describe.

Btw: tried warp?

https://engineering.fb.com/2014/03/28/open-source/under-the-hood-warp-a-fast-c-and-c-preprocessor/

It could speed thing up a bit, but I don't think it will help by much.

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[tomas]

[-- Attachment #1: Type: text/plain, Size: 862 bytes --]

On Tue, Nov 24, 2020 at 08:50:24AM +0100, Arthur Miller wrote:

[...]

> libmpv is just a library to play media files, has nothing to do with
> javascript. You can check yourself:

Until some idiot embeds javascript into some video format spec
and said spec becomes indispensable because That New Platform,
say, Squitch, makes it mandatory (and cheaper videocams follow
that lead, or something).

Then either libmpv follows or we users begin to whine and moan.

You laugh? See how PDF embedded javascript.

Yes, somewhat tongue-in-cheek, but at the same time watching
with a mixture of fascination and horror how this arena around
freedom has changed radically the last 20 years. 

Not long ago the "enemy" was secret sauce and closed interfaces.
These days it's more like cripplingly complex systems and user
convenience.

Cheers
 - t

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Arthur Miller]

<tomas@tuxteam.de> writes:

> On Tue, Nov 24, 2020 at 08:50:24AM +0100, Arthur Miller wrote:
>
> [...]
>
>> libmpv is just a library to play media files, has nothing to do with
>> javascript. You can check yourself:
>
> Until some idiot embeds javascript into some video format spec
> and said spec becomes indispensable because That New Platform,
> say, Squitch, makes it mandatory (and cheaper videocams follow
> that lead, or something).
>
> Then either libmpv follows or we users begin to whine and moan.
>
> You laugh? 
Of course! :D You always make me laugh, you know that! :-)

> See how PDF embedded javascript.
Just for the record didn't that happened long before v8 came and google
made js lingua franca of web dev?

If I remember even MS implemented window host that was ment for scripting
windows (along VB). I never used it though, no idea how extensive it
was, but it was also before the Node and before JS become ubiquitious in
every application, and before applications were "apps" ... :-)

> Yes, somewhat tongue-in-cheek, but at the same time watching
> with a mixture of fascination and horror how this arena around
> freedom has changed radically the last 20 years. 
 
> Not long ago the "enemy" was secret sauce and closed interfaces.
> These days it's more like cripplingly complex systems and user
> convenience.
( ... )

Come writers and critics
Who prophesize with your pen
And keep your eyes wide
The chance won't come again
And don't speak too soon
For the wheel's still in spin
And there's no tellin' who
That it's namin'
For the loser now
Will be later to win
For the times they are a-changin'

( ... )
 -- Robert Allen Zimmerman

> Cheers
>  - t

Cheers my man :-)

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[T.V Raman]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=gb18030, Size: 1116 bytes --]

<tomas@tuxteam.de> writes:


:-) Good point. See how something as simple as a Haskell API Search tool has tangled itself up in Js land (hoogle)> On Tue, Nov 24, 2020 at 08:50:24AM +0100, Arthur Miller wrote:
>
> [...]
>
>> libmpv is just a library to play media files, has nothing to do with
>> javascript. You can check yourself:
>
> Until some idiot embeds javascript into some video format spec
> and said spec becomes indispensable because That New Platform,
> say, Squitch, makes it mandatory (and cheaper videocams follow
> that lead, or something).
>
> Then either libmpv follows or we users begin to whine and moan.
>
> You laugh? See how PDF embedded javascript.
>
> Yes, somewhat tongue-in-cheek, but at the same time watching
> with a mixture of fascination and horror how this arena around
> freedom has changed radically the last 20 years. 
>
> Not long ago the "enemy" was secret sauce and closed interfaces.
> These days it's more like cripplingly complex systems and user
> convenience.
>
> Cheers
>  - t
>

-- 

Thanks,

--Raman
7©4 Id: kg:/m/0285kf1  •0Ü8

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Akira Kyle]

On Mon, Nov 23, 2020 at 11:27 PM, Lars Ingebrigtsen 
<larsi@gnus.org> wrote:

>> Thus I feel like the only sane way to mix the two is to allow 
>> gtk to
>> just take over an Emacs window which is always rectangular. No 
>> need to
>> worry about scrolling and all the redisplay optimizations that 
>> will
>> try to interfere with gtk attempting to draw at some point in 
>> the
>> buffer.
>
> I think this design reduces the use cases for emacs-webkit 
> considerably
> -- if all you can do with it is to let it take over a window 
> completely,
> then that's disappointing.  

Maybe it's worth expanding on this since I was originally trying 
to come up a solution that would work more along the lines of what 
you're asking, but ended up where I am entirely due to technical 
reasons and perhaps that journey may be informative...

Consider that any "rich" display element you might want displayed 
among others in a buffer needs to have some display property so 
that redisplay can be aware of it's position relative to other 
elements in the buffer. Thus there ultimately needs to be some 
entry in the glyph matrix for this "rich" display element. But the 
current way redisplay happens, with its various optimizations, you 
can't expect that every time the element is moved or partially 
clipped or even removed entirely, that Emacs will ask for your 
"rich" element to draw itself.

So you have two options. One is you make your "rich" element 
behave pretty much the way images behave and just give Emacs a 
pixbuf to display. But even then you may need nontrivial redisplay 
changes due to Emacs not expecting the pixbuf data to change out 
from underneath itself. The other is you try to do something like 
what xwidget does and synchronize changes in where redisplay 
calculates the position of the "rich" element to go with drawing 
that element yourself (or letting gtk draw it).

I see this second option as ultimately being pretty brittle as it 
needs to force redisplay to draw your glyph every time redisplay 
changes some properties of its display and pass what those are to 
your "rich" element and hope that it respects them. If it doesn't 
or something in this process breaks, your element ends up floating 
in places it shouldn't be because it keeps drawing itself to the 
screen there. I think the former approach of expanding the image.c 
code or creating something modeled after it would be the better 
approach, likely with less visual bugs. Perhaps cairo could be a 
key component of this if its okay to have it as a dependency on 
all platforms. Anyways that was at least the direction my hacking 
on xwidgets was taking me before I got "sidetracked" by this 
dynamic module idea.

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > was, but it was also before the Node and before JS become ubiquitious in
  > every application,

People can use web apps if they wish, but please do not tell people
that that's all there is nowadays.  GNU/Linux has plenty of
applications which users install on their own computers and contain no
Javascript.

That old way is also the way that is good for users' control of the
software.  Distros package the program and offer their own versions.
In doing this, they audit the code.  Not completely, but it's still a
good thing.  The distros compete but also learn from each other.  That
is the way in which our community traditionally protects itself
against malware and bugs.

"Web applications", even if their code is released as free software,
structure a system of release and usage which has no room for distros,
and this interferes with users' having control over the program.

A structure which precludes this is dangerous in practice, even if in
principle it should not change anything.

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > You laugh? See how PDF embedded javascript.

I checked a few years ago, and ISTR the free PDF players did not
run the Javascript.  At least, by default they did not.

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[tomas]

[-- Attachment #1: Type: text/plain, Size: 1427 bytes --]

On Wed, Nov 25, 2020 at 12:38:37AM -0500, Richard Stallman wrote:
> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
> 
>   > was, but it was also before the Node and before JS become ubiquitious in
>   > every application,
> 
> People can use web apps if they wish, but please do not tell people
> that that's all there is nowadays.  GNU/Linux has plenty of
> applications which users install on their own computers and contain no
> Javascript.
> 
> That old way is also the way that is good for users' control of the
> software.  Distros package the program and offer their own versions.
> In doing this, they audit the code.  Not completely, but it's still a
> good thing.  The distros compete but also learn from each other.  That
> is the way in which our community traditionally protects itself
> against malware and bugs.
> 
> "Web applications", even if their code is released as free software,
> structure a system of release and usage which has no room for distros,
> and this interferes with users' having control over the program.
> 
> A structure which precludes this is dangerous in practice, even if in
> principle it should not change anything.

Can't agree more (see my answer to your other post).

Cheers
 - t

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[tomas]

[-- Attachment #1: Type: text/plain, Size: 1200 bytes --]

On Wed, Nov 25, 2020 at 12:38:56AM -0500, Richard Stallman wrote:
> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
> 
>   > You laugh? See how PDF embedded javascript.
> 
> I checked a few years ago, and ISTR the free PDF players did not
> run the Javascript.  At least, by default they did not.

That's why I use those. Problem is that this creates a sort of "soft
pressure": PDFs made by proprietary tools (Adobe, I'm looking at you)
sometimes don't "work" if the Javascript isn't executed client-side.

The user has the choice to complain to the issuer, almost always
getting blank stares (the issuer is usually many levels of responsibility,
so the user's complaint almost never reaches those who can do
anything about it) or to comply -- being pushed to proprietary
software. Sometimes changing the service provider is an option,
sometimes it isn't.

It's a variation of the "defective by design" pattern, with the
aggravating factor that the user has less and less choice these
days.

Cheers
 - t

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Eli Zaretskii]

> From: Akira Kyle <akira@akirakyle.com>
> Date: Tue, 24 Nov 2020 18:36:38 -0700
> Cc: emacs-devel@gnu.org
> 
> Consider that any "rich" display element you might want displayed 
> among others in a buffer needs to have some display property so 
> that redisplay can be aware of it's position relative to other 
> elements in the buffer. Thus there ultimately needs to be some 
> entry in the glyph matrix for this "rich" display element. But the 
> current way redisplay happens, with its various optimizations, you 
> can't expect that every time the element is moved or partially 
> clipped or even removed entirely, that Emacs will ask for your 
> "rich" element to draw itself.

It is a relatively simple matter to disable some optimizations when a
widget is displayed.  It is also relatively simple to tell the widget
to redraw itself when redisplay changed its position on the screen.

> I see this second option as ultimately being pretty brittle as it 
> needs to force redisplay to draw your glyph every time redisplay 
> changes some properties of its display and pass what those are to 
> your "rich" element and hope that it respects them.

I don't see why you consider this such a significant problem.  The
number of interfaces that such a "rich" display element needs to
support is very small and well-defined.  The problem with xwidgets in
this regard is that its interaction with the display code was left
unfinished, it basically just copy/pastes the code which supports
display of images, which is not 100% correct, since xwidgets don't
display a static bitmap.

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Arthur Miller]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > was, but it was also before the Node and before JS become ubiquitious in
>   > every application,
>
> People can use web apps if they wish, but please do not tell people
> that that's all there is nowadays.
Convenience; everything is in the browser, in one application; it is
just like Emacs. They check mail online, write their documents online,
everything is online. Browser is the Emacs of millenials. Until
connection gets cut. I have been in a war and don't trust anyone; so I
have everything on my own harddrives and prefer real computer
applications to browser, but when I really think of it; browser is
equivalent to how we use Emacs. We have our apps as "packages" written
in Lisp instead of JS.

> That old way is also the way that is good for users' control of the
> software.  Distros package the program and offer their own versions.
> In doing this, they audit the code.  Not completely, but it's still a
> good thing.  The distros compete but also learn from each other.  That
> is the way in which our community traditionally protects itself
> against malware and bugs.
>
> "Web applications", even if their code is released as free software,
> structure a system of release and usage which has no room for distros,
> and this interferes with users' having control over the program.
Indeed; And control over the program and in the very end the user, is
why companies are so in love with webapps.

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Eric S Fraga]

On Wednesday, 25 Nov 2020 at 16:26, Arthur Miller wrote:
> Browser is the Emacs of millenials. 

Maybe more "Browser is the DEC VT100/IBM 3270 for millenials." ;-)

-- 
Eric S Fraga via Emacs 28.0.50 & org 9.4 on Debian bullseye/sid

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > libmpv is a library to play video files like .mp4.  There's no
  > Javascript involved.

Ok on that score.  But why is it desirable to do this in Emacs rather
than run vlc?  It is a big nonmodularity, and that is a big drawback.

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Arthur Miller]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > libmpv is a library to play video files like .mp4.  There's no
>   > Javascript involved.
>
> Ok on that score.  But why is it desirable to do this in Emacs rather
> than run vlc?  It is a big nonmodularity, and that is a big drawback.
Why is it desirable to view jpegs in Emacs rather then run
insert-your-favourite-image-viewer-here? Or why is it desirable to
render html in emacs rather then run Firefox?

For me: less processes, less switching between applications, nice to
have things in Emacs buffers.

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[tomas]

[-- Attachment #1: Type: text/plain, Size: 1066 bytes --]

On Thu, Nov 26, 2020 at 06:25:10AM +0100, Arthur Miller wrote:
> Richard Stallman <rms@gnu.org> writes:
> 
> > [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> > [[[ whether defending the US Constitution against all enemies,     ]]]
> > [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
> >
> >   > libmpv is a library to play video files like .mp4.  There's no
> >   > Javascript involved.
> >
> > Ok on that score.  But why is it desirable to do this in Emacs rather
> > than run vlc?  It is a big nonmodularity, and that is a big drawback.
> Why is it desirable to view jpegs in Emacs rather then run
> insert-your-favourite-image-viewer-here? Or why is it desirable to
> render html in emacs rather then run Firefox?
> 
> For me: less processes, less switching between applications, nice to
> have things in Emacs buffers.

Imagine being able to say "org-store-link" at a specific frame in
a video. Or at specific (x, y) coordinates in a frame in said
video.

Just imagine :-)

Cheers
 - t

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Arthur Miller]

<tomas@tuxteam.de> writes:

> On Thu, Nov 26, 2020 at 06:25:10AM +0100, Arthur Miller wrote:
>> Richard Stallman <rms@gnu.org> writes:
>> 
>> > [[[ To any NSA and FBI agents reading my email: please consider    ]]]
>> > [[[ whether defending the US Constitution against all enemies,     ]]]
>> > [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>> >
>> >   > libmpv is a library to play video files like .mp4.  There's no
>> >   > Javascript involved.
>> >
>> > Ok on that score.  But why is it desirable to do this in Emacs rather
>> > than run vlc?  It is a big nonmodularity, and that is a big drawback.
>> Why is it desirable to view jpegs in Emacs rather then run
>> insert-your-favourite-image-viewer-here? Or why is it desirable to
>> render html in emacs rather then run Firefox?
>> 
>> For me: less processes, less switching between applications, nice to
>> have things in Emacs buffers.
>
> Imagine being able to say "org-store-link" at a specific frame in
> a video. Or at specific (x, y) coordinates in a frame in said
> video.
>
> Just imagine :-)
>
> Cheers
>  - t
Or to render text programmatically on top of video.

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Tomas Hlavaty]

On Thu 26 Nov 2020 at 09:27, <tomas@tuxteam.de> wrote:
> On Thu, Nov 26, 2020 at 06:25:10AM +0100, Arthur Miller wrote:
>> Why is it desirable to view jpegs in Emacs rather then run
>> insert-your-favourite-image-viewer-here? Or why is it desirable to
>> render html in emacs rather then run Firefox?

why does this need libraries instead of programs?

>> For me: less processes,

why is it a good thing?

>> less switching between applications, nice to have things in Emacs
>> buffers.

why does this need libraries instead of programs?

> Imagine being able to say "org-store-link" at a specific frame in
> a video. Or at specific (x, y) coordinates in a frame in said
> video.

why does this need libraries instead of programs?

> Just imagine :-)

imagine all the extra vulnerabilities linked into the emacs process

  $ ldd emacs | wc -l
  94

hmm, that is a lost cause already

libraries are easier to use but programs run in separate processes and
can be extra sandboxed

RE: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[arthur miller]

[-- Attachment #1: Type: text/plain, Size: 2021 bytes --]

Imagine all the vulnerabilities brought in by all the other libraries Emacs uses, and Emacs uses quite many. I don't see difference there.

If you are aware of some  special vulnerability in libmpv, then please rapport it to the project so they can fix it.

For the rest of your questions: answer should probably be self-evident: less processes means less resources used, faster response at runtime and programmatic control usually means more flexibility. For same reasons, I wish even dired was not usin ls program but was built on directory-files instead.

By the way, I hope you can formulate yourself ess passive aggressive in future. I perceive your mail as something  belonging rather to discussions on twitch chatt or reddit at best.


-------- Originalmeddelande --------
Från: Tomas Hlavaty <tom@logand.com>
Datum: 2020-11-26 20:12 (GMT+01:00)
Till: emacs-devel@gnu.org
Ämne: Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

On Thu 26 Nov 2020 at 09:27, <tomas@tuxteam.de> wrote:
> On Thu, Nov 26, 2020 at 06:25:10AM +0100, Arthur Miller wrote:
>> Why is it desirable to view jpegs in Emacs rather then run
>> insert-your-favourite-image-viewer-here? Or why is it desirable to
>> render html in emacs rather then run Firefox?

why does this need libraries instead of programs?

>> For me: less processes,

why is it a good thing?

>> less switching between applications, nice to have things in Emacs
>> buffers.

why does this need libraries instead of programs?

> Imagine being able to say "org-store-link" at a specific frame in
> a video. Or at specific (x, y) coordinates in a frame in said
> video.

why does this need libraries instead of programs?

> Just imagine :-)

imagine all the extra vulnerabilities linked into the emacs process

  $ ldd emacs | wc -l
  94

hmm, that is a lost cause already

libraries are easier to use but programs run in separate processes and
can be extra sandboxed


[-- Attachment #2: Type: text/html, Size: 2959 bytes --]

RE: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Tomas Hlavaty]

On Thu 26 Nov 2020 at 19:34, arthur miller <arthur.miller@live.com> wrote:
> Imagine all the vulnerabilities brought in by all the other libraries
> Emacs uses, and Emacs uses quite many. I don't see difference there.

exactly, so many libraries (i already wrote that my emacs has about 94)
in the same process written in memory unsafe language

also not only vulnerabilities but also memory leaks

> If you are aware of some special vulnerability in libmpv, then please
> rapport it to the project so they can fix it.

if you are aware of some special tumor, then please go to a doctor?

i am talking about prevention

what about not smoking in the first place?

> For the rest of your questions: answer should probably be
> self-evident: less processes means less resources used,

it's 2020 and we want to avoid processes for that reason?  strange

emacs consumes all available memory easily quite often, i don't see any
improvement on this front even with so many libraries

> response at runtime and programmatic control usually means more
> flexibility.

yes, i said that libs are easier to use

> For same reasons, I wish even dired was not usin ls program but was
> built on directory-files instead.

i wish more of emacs was written in memory safe elisp too

but this is a different question altogether

> By the way, I hope you can formulate yourself ess passive aggressive
> in future. I perceive your mail as something belonging rather to
> discussions on twitch chatt or reddit at best.

sorry that it comes that way

it's a shame that you attack me instead of discussed ideas

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Arthur Miller]

Tomas Hlavaty <tom@logand.com> writes:

> On Thu 26 Nov 2020 at 19:34, arthur miller <arthur.miller@live.com> wrote:
>> Imagine all the vulnerabilities brought in by all the other libraries
>> Emacs uses, and Emacs uses quite many. I don't see difference there.
>
> exactly, so many libraries (i already wrote that my emacs has about 94)
> in the same process written in memory unsafe language
>
> also not only vulnerabilities but also memory leaks

>> If you are aware of some special vulnerability in libmpv, then please
>> rapport it to the project so they can fix it.
>
> if you are aware of some special tumor, then please go to a doctor?
>
> i am talking about prevention
>
> what about not smoking in the first place?

>> For the rest of your questions: answer should probably be
>> self-evident: less processes means less resources used,
>
> it's 2020 and we want to avoid processes for that reason?  strange
>
> emacs consumes all available memory easily quite often, i don't see any
> improvement on this front even with so many libraries
Maybe you should help them in the work, send in your traces if it eats
all memory quite often.

>> response at runtime and programmatic control usually means more
>> flexibility.
>
> yes, i said that libs are easier to use
Easier is not same as more flexible. On contrary more flexibility
usually means more complexity. Bringing a library like libmpv into Emacs
is certainly not easy. You are free to check the docs for embedding:

https://github.com/mpv-player/mpv-examples/tree/master/libmpv

Trust me, it is much easier to shell-execute mpv player, or use Emms as
I already do; or just M-! to play music or video with mpv. 

>> For same reasons, I wish even dired was not usin ls program but was
>> built on directory-files instead.
>
> i wish more of emacs was written in memory safe elisp too
>
> but this is a different question altogether
>
>> By the way, I hope you can formulate yourself ess passive aggressive
>> in future. I perceive your mail as something belonging rather to
>> discussions on twitch chatt or reddit at best.
>
> sorry that it comes that way
>
> it's a shame that you attack me instead of discussed ideas
I don't know Thomas; maybe that is way you are communicating that I am
not used too, but I am at least honest and telling you how it sounded to
me; take it or leave it. I still percieve this email as trolling. I
don't see anything really technical you are bringing in; but the very
last thing about memory safe lisp, and I don't mean it is an attack on
you, just the way I read your mail. Maybe it's me. 

Yes in word 2020 I still care about efficiency, I see it as a moral
imperative not to waste resources. I might be weird, but that is how I am.

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

We should not link another library into Emacs unless there is a
particular reason why that library is important.

So far, we have linked in libraries for displaying images which are
part of documents, and we have linked in some libraries for network
protocols that are useful for a variety of purposes.  In these cases
there is an important reason.  Doing these jobs in some other way
would have big disadvantages.

For this job, the disadvantages of the library would impact security
(that library is big), and future maintenance, as well as installation
complexity.  And there is no big disadvantage to forking VLC, or
mplayer2, or whichever player the user prefers.  That is the way we
should do it,

The argument that "We have linked with so many external libraries that
we should not hesitate to add one more" is fundamentally misguided.
Some things are worth paying a price for.  Having bought a few of
them, which msy have been good purchases, it does not follow that we
should rush to buy the whole store.  The price of these purchases is
substantial and we should pay it only when really important.

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Arthur Miller]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
> We should not link another library into Emacs unless there is a
> particular reason why that library is important.
>
> So far, we have linked in libraries for displaying images which are
> part of documents, and we have linked in some libraries for network
> protocols that are useful for a variety of purposes.  In these cases
> there is an important reason.  Doing these jobs in some other way
> would have big disadvantages.
>
> For this job, the disadvantages of the library would impact security
> (that library is big), and future maintenance, as well as installation
> complexity.  And there is no big disadvantage to forking VLC, or
> mplayer2, or whichever player the user prefers.  That is the way we
> should do it,

> The argument that "We have linked with so many external libraries that
> we should not hesitate to add one more" is fundamentally misguided.
It depends on how you read it. If you read it as a solo argument,
a sole purpose of "we have lib A let's have lib B just for purpose of
having it" then I agree with you. But what I ment is saying that
including a library is a security risk and arguing just because of
adding it, is not very much of argument, because we would not have any
external library in Emacs. I don't argue for that one particular just
because we already have so many others. That connects to the next:

> Some things are worth paying a price for.  Having bought a few of
> them, which msy have been good purchases, it does not follow that we
> should rush to buy the whole store.  The price of these purchases is
> substantial and we should pay it only when really important.
Sure of course, not every library under the Sun should be embedded. I am
aware of price/vs cost.

To my knowledge, mpv is probably the neetiest one to bring in media
playing capabilities; it has lots of codecs, is written to be embedded,
is free and would make Emacs be able to play music and video files
without external players. Adding multimedia capabilities opens up for
lots of flexibility and creativity; people can maybe do interesting
stuff with it. I would certainly like Emacs to become a multimedia
player, I play my music with Emacs already :-).

If other people think it is too expensive in terms of implementation
cost and what it offers, and if multimedia is not desirable in Emacs, I
can have understanding with that. I might not agree, but my opinion is
just one persons opinion, and I am not even an Emacs developer, so of
course, you who make Emacs work probably know better and have precedence
in what Emacs should do/have or not. It would be wrong to claim anything
else :). 

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Eli Zaretskii]

> From: Arthur Miller <arthur.miller@live.com>
> Date: Fri, 27 Nov 2020 20:22:42 +0100
> Cc: tom@logand.com, emacs-devel@gnu.org
> 
> To my knowledge, mpv is probably the neetiest one to bring in media
> playing capabilities; it has lots of codecs, is written to be embedded,
> is free and would make Emacs be able to play music and video files
> without external players. Adding multimedia capabilities opens up for
> lots of flexibility and creativity; people can maybe do interesting
> stuff with it. I would certainly like Emacs to become a multimedia
> player, I play my music with Emacs already :-).
> 
> If other people think it is too expensive in terms of implementation
> cost and what it offers, and if multimedia is not desirable in Emacs, I
> can have understanding with that. I might not agree, but my opinion is
> just one persons opinion, and I am not even an Emacs developer, so of
> course, you who make Emacs work probably know better and have precedence
> in what Emacs should do/have or not. It would be wrong to claim anything
> else :). 

Expensive or not, talking about mpv as providing video playing
facilities for Emacs is makes little sense as long as the issue of
embedding video in a window that shows an Emacs buffer is not
resolved.  As the xwidgets experiment suggests, the problem to solve
here is not a trivial one.  Until someone comes up with some clever
idea for how to do that, let alone submits patches for that, the issue
of which library to use is not really relevant to Emacs development.

If someone wants a flexible way of playing video under Emacs control,
I think a more practically useful way at this time is to provide a
feature that runs VLC or a similar player program as an Emacs
subprocess, and controls that player via Emacs commands.  That could
need extending the players themselves, if their command-line arguments
aren't flexible enough and there's no other API that Emacs could use.

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Stefan Monnier]

> To my knowledge, mpv is probably the neetiest one to bring in media
> playing capabilities; it has lots of codecs, is written to be embedded,
> is free and would make Emacs be able to play music and video files
> without external players. Adding multimedia capabilities opens up for
> lots of flexibility and creativity; people can maybe do interesting
> stuff with it. I would certainly like Emacs to become a multimedia
> player,

The question is what benefit is there to having it linked in as
a library as opposed to accessing similar functionality via
a sub-process.

> I play my music with Emacs already :-).

This would tend to argue in favor of not having it as a library since
you can already do what you want just fine using a sub-process.


        Stefan

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Akira Kyle]

On Wed, Nov 25, 2020 at 08:11 AM, Eli Zaretskii <eliz@gnu.org> 
wrote:

>> Consider that any "rich" display element you might want 
>> displayed 
>> among others in a buffer needs to have some display property so 
>> that redisplay can be aware of it's position relative to other 
>> elements in the buffer. Thus there ultimately needs to be some 
>> entry in the glyph matrix for this "rich" display element. But 
>> the 
>> current way redisplay happens, with its various optimizations, 
>> you 
>> can't expect that every time the element is moved or partially 
>> clipped or even removed entirely, that Emacs will ask for your 
>> "rich" element to draw itself.
>
> It is a relatively simple matter to disable some optimizations 
> when a
> widget is displayed.  It is also relatively simple to tell the 
> widget
> to redraw itself when redisplay changed its position on the 
> screen.

Yes, but you had previously pointed to that fact that xwidgets 
needs to such disable such optimizations as part of the "kludge" 
that would need to be resolved for xwidgets to be considered 
viable for a path forward. Having looked at that, I'm not sure gtk 
will ever want to cooperate without doing things that would be 
considered hacks by gtk folks.

>> I see this second option as ultimately being pretty brittle as 
>> it 
>> needs to force redisplay to draw your glyph every time 
>> redisplay 
>> changes some properties of its display and pass what those are 
>> to 
>> your "rich" element and hope that it respects them.
>
> I don't see why you consider this such a significant problem. 
> The
> number of interfaces that such a "rich" display element needs to
> support is very small and well-defined.  The problem with 
> xwidgets in
> this regard is that its interaction with the display code was 
> left
> unfinished, it basically just copy/pastes the code which 
> supports
> display of images, which is not 100% correct, since xwidgets 
> don't
> display a static bitmap.

The "number interfaces" will likely be small no matter what 
solution is taken, its more about how easy you want to make it for 
a "rich" display element to really mess with Emacs' display and 
input handling. For example with xwidgets the onus is on the 
widget implementation to not totally steal keyboard focus from 
Emacs. I've found that gtk widgets really like stealing keyboard 
focus so its easy to end up in a state you can't escape out 
of. Similarly gtk widgets won't always easily draw themselves 
where you want at the size you want so its easy for widgets to 
render, say, across window boundaries. The xwidget approach 
requires non-insignificant work on the widget end to ensure the 
widgets will work within the confines of Emacs' redisplay. I'd 
think that such an interface to support "rich" display elements 
should be restrictive enough to not easily allow the "rich" 
display elements to misbehave. Especially if it becomes possible 
to define such elements from lisp. I see expanding the current 
image handling to give richer canvases that can be drawn on as the 
"safer" approach with respect to these concerns. But again much of 
this is highly speculative and doesn't mean much until some code 
is actually written. I'm just trying to voice some of the 
frustrations I had in working on the current xwidget code and how 
that might apply here.

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Arthur Miller]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> To my knowledge, mpv is probably the neetiest one to bring in media
>> playing capabilities; it has lots of codecs, is written to be embedded,
>> is free and would make Emacs be able to play music and video files
>> without external players. Adding multimedia capabilities opens up for
>> lots of flexibility and creativity; people can maybe do interesting
>> stuff with it. I would certainly like Emacs to become a multimedia
>> player,
>
> The question is what benefit is there to having it linked in as
> a library as opposed to accessing similar functionality via
> a sub-process.
One could programmatically render stuff on top of the video; embedd it
in a viewer to create a player, like there is image slide for image
mode, video tag for shr was what triggered discussion. I mean you could
code a renderer for video just as you made svg renderer; but somebody
already created a re-usable, well tested library, so why recreate what
is already done and tested. That is how I think of it.

>> I play my music with Emacs already :-).
>
> This would tend to argue in favor of not having it as a library since
> you can already do what you want just fine using a sub-process.
Yeah, I know; but just playing music is a little bit of playing with a
closed box. And it does not work with video. Also as I said it is a bit
about efficiency. I think it would be useful for Emacs to play sound
files and video files natively.

For me Emacs is long past just text editor; for me it is more like an
über-terminal or we could call it OS-browser ... I don't know. Just how
I use it, I understand if others use Emacs differently.

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Arthur Miller]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Arthur Miller <arthur.miller@live.com>
>> Date: Fri, 27 Nov 2020 20:22:42 +0100
>> Cc: tom@logand.com, emacs-devel@gnu.org
>> 
>> To my knowledge, mpv is probably the neetiest one to bring in media
>> playing capabilities; it has lots of codecs, is written to be embedded,
>> is free and would make Emacs be able to play music and video files
>> without external players. Adding multimedia capabilities opens up for
>> lots of flexibility and creativity; people can maybe do interesting
>> stuff with it. I would certainly like Emacs to become a multimedia
>> player, I play my music with Emacs already :-).
>> 
>> If other people think it is too expensive in terms of implementation
>> cost and what it offers, and if multimedia is not desirable in Emacs, I
>> can have understanding with that. I might not agree, but my opinion is
>> just one persons opinion, and I am not even an Emacs developer, so of
>> course, you who make Emacs work probably know better and have precedence
>> in what Emacs should do/have or not. It would be wrong to claim anything
>> else :). 
>
> Expensive or not, talking about mpv as providing video playing
> facilities for Emacs is makes little sense as long as the issue of
> embedding video in a window that shows an Emacs buffer is not
> resolved.  As the xwidgets experiment suggests, the problem to solve
> here is not a trivial one.
Indeed; I am aware of it, and I personally am not even sure it is
possible with Emacs renderer as it is. If you read one of my first
replay's to Lars, I pretty much hoped he understands Emacs better then I
so he could do it, because I honestly, am not sure I can, I tried before
and couldn't figure it out. :)

I am playing with it, but I am not that knowledgable about
internals. Playing video on Emacs frame is not that hard; it is just
basic X11/GDI/whatever platform we are talking about. I can just attach
opengl context and send OS window to mpv, with some thinkgering. But
playing video in an image container, as images are put as faces on
characters, is not something I currently know how and could be able to
get going.

> Until someone comes up with some clever
> idea for how to do that, let alone submits patches for that, the issue
> of which library to use is not really relevant to Emacs development.
>
> If someone wants a flexible way of playing video under Emacs control,
> I think a more practically useful way at this time is to provide a
> feature that runs VLC or a similar player program as an Emacs
> subprocess, and controls that player via Emacs commands.  That could
> need extending the players themselves, if their command-line arguments
> aren't flexible enough and there's no other API that Emacs could use.
If you mean, just as external application in it's own frame, then Emms
already does it (I think).

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Akira Kyle]

On Fri, Nov 27, 2020 at 01:07 PM, Stefan Monnier 
<monnier@iro.umontreal.ca> wrote:

>> To my knowledge, mpv is probably the neetiest one to bring in 
>> media
>> playing capabilities; it has lots of codecs, is written to be 
>> embedded,
>> is free and would make Emacs be able to play music and video 
>> files
>> without external players. Adding multimedia capabilities opens 
>> up for
>> lots of flexibility and creativity; people can maybe do 
>> interesting
>> stuff with it. I would certainly like Emacs to become a 
>> multimedia
>> player,
>
> The question is what benefit is there to having it linked in as
> a library as opposed to accessing similar functionality via
> a sub-process.

I don't think this necessarily has to be a question of Emacs links 
against yet another library, or someone wraps a program's command 
line arguments. As I'm hoping to show with `emacs-webkit`, it's 
viable to think about using dynamic modules which link against a 
library and provide the desired functionality. Then those who 
never want that functionality never have to worry about their 
Emacs being linked with it, while the lisp package providing the 
functionality is free to use the best interface to that library, 
which I often think will be through linking to it. Linking often 
provides better performance and less complexity as you avoid 
needing to constantly define a data exchange format and parse it.

I once saw someone arguing why "the Emacs way" was far superior 
"the unix way" mainly because unix insists on everything being 
thought of as just files and text, which while powerful, ends up 
with every program defining different, incompatible text 
interfaces and parsing formats. Case in point being how messy 
command line argument formats can be since there's no standard 
format imposed on the way a program should handle flags. Meanwhile 
with Emacs everything is a Sexpr, which along with being easier to 
manipulate, offers just enough structure to greatly reduce these 
kind of incompatibilities between programs trying to communicate 
with eachother.

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Stefan Monnier]

>> The question is what benefit is there to having it linked in as
>> a library as opposed to accessing similar functionality via
>> a sub-process.
> One could programmatically render stuff on top of the video;

Sounds highly hypothetical.

> I mean you could code a renderer for video just as you made svg
> renderer; but somebody already created a re-usable, well tested
> library, so why recreate what is already done and tested.  That is how
> I think of it.

Same here: just linking libmpv doesn't magically let you do those things.

Until such things are more concrete, it's hard to justify linking a library.

The emacs-application-framework is an example of what you can do with
sub-processes.

> Yeah, I know; but just playing music is a little bit of playing with a
> closed box. And it does not work with video. Also as I said it is a bit
> about efficiency. I think it would be useful for Emacs to play sound
> files and video files natively.

Yet there are also packages which let you play videos from Emacs, so
it's not limited to music.

> For me Emacs is long past just text editor; for me it is more like an
> über-terminal or we could call it OS-browser ... I don't know.  Just how
> I use it, I understand if others use Emacs differently.

I think this description applies to pretty much all of us here.  It does
not imply that linking is necessarily the better path.

Linking libraries into Emacs tends to be tricky, costly, and come with
a fair bit of consequences, so it needs to justified by fairly
solid evidence.  Linking them via the module interface can be a good
middle-ground BTW.


        Stefan

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Arthur Miller]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>> The question is what benefit is there to having it linked in as
>>> a library as opposed to accessing similar functionality via
>>> a sub-process.
>> One could programmatically render stuff on top of the video;
>
> Sounds highly hypothetical.
Why is it hypothetical? With their render api, it will take an opengl
surface to render on and that surface can later be renderd on. I don't
say it is trivial or easy; just possible.

> Same here: just linking libmpv doesn't magically let you do those things.
Of course not; I don't understand why do you say so in this case, why
would you think I think so? I was obviously aiming at the fact that
instead of  writing own interface to some lower level video decoding
library or maybe your own decoding routines, Emacs could re-use
libmpv. I don't see what is hypotethical there.

> Linking libraries into Emacs tends to be tricky, costly, and come with
> a fair bit of consequences, so it needs to justified by fairly
> solid evidence.
Of course I agree. But is also a matter of personal preferance, what is
justified and solid evidence. 

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Stefan Monnier]

>> Sounds highly hypothetical.
> Why is it hypothetical?

Because it's pretty far from what existing code can do.


        Stefan

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

Each proposed external library requires a cost/benefit judgment.
In my judgment, the benefits of linking a video library are small
because it is just a frill.

Displaying images is important because documents contain images.

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Eli Zaretskii]

> From: Akira Kyle <akira@akirakyle.com>
> Date: Fri, 27 Nov 2020 14:22:37 -0700
> Cc: emacs-devel@gnu.org, Richard Stallman <rms@gnu.org>,
>  Arthur Miller <arthur.miller@live.com>, tom@logand.com
> 
> > The question is what benefit is there to having it linked in as
> > a library as opposed to accessing similar functionality via
> > a sub-process.
> 
> I don't think this necessarily has to be a question of Emacs links 
> against yet another library, or someone wraps a program's command 
> line arguments.

But that was the question that was asked, and that was the question to
which Richard responded.

Other questions are possible, and will likely have different answers.

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Eli Zaretskii]

> From: Akira Kyle <akira@akirakyle.com>
> Cc: larsi@gnus.org, emacs-devel@gnu.org
> Date: Fri, 27 Nov 2020 13:56:25 -0700
> 
> > It is a relatively simple matter to disable some optimizations
> > when a widget is displayed.  It is also relatively simple to tell
> > the widget to redraw itself when redisplay changed its position on
> > the screen.
> 
> Yes, but you had previously pointed to that fact that xwidgets 
> needs to such disable such optimizations as part of the "kludge" 
> that would need to be resolved for xwidgets to be considered 
> viable for a path forward. Having looked at that, I'm not sure gtk 
> will ever want to cooperate without doing things that would be 
> considered hacks by gtk folks.

I'd appreciate more details on the GTK part of this.  I only said that
from the Emacs display engine side things are not too hairy.  from my
POV, xwidgets integration simply stopped short of going all the way
towards integration into the redisplay framework, and I don't think
anyone said before that this was done because better integration was
impossible or very hard.

> > I don't see why you consider this such a significant problem.  The
> > number of interfaces that such a "rich" display element needs to
> > support is very small and well-defined.  The problem with xwidgets
> > in this regard is that its interaction with the display code was
> > left unfinished, it basically just copy/pastes the code which
> > supports display of images, which is not 100% correct, since
> > xwidgets don't display a static bitmap.
> 
> The "number interfaces" will likely be small no matter what 
> solution is taken, its more about how easy you want to make it for 
> a "rich" display element to really mess with Emacs' display and 
> input handling. For example with xwidgets the onus is on the 
> widget implementation to not totally steal keyboard focus from 
> Emacs. I've found that gtk widgets really like stealing keyboard 
> focus so its easy to end up in a state you can't escape out 
> of.

Keyboard focus is a separate issue.  I only talked about integration
with redisplay.

For keyboard input integration, we'd need to develop some
infrastructure, if the existing one doesn't provide enough hooks.  If
you or someone else can describe the details, we could discuss that in
more practical terms.  I think it's too soon to give up on the
practical possibility to solve these problems before we discussed
that, or are even fully aware of the problems and the potential
solutions.

> Similarly gtk widgets won't always easily draw themselves 
> where you want at the size you want so its easy for widgets to 
> render, say, across window boundaries.

If that is true, then your method of letting the widget display in a
window won't work well either, right?  IOW, whether the screen area
given to the widget is a full Emacs window or just its part is not an
important aspect of the integration of such widgets into the Emacs
display.

> The xwidget approach requires non-insignificant work on the widget
> end to ensure the widgets will work within the confines of Emacs'
> redisplay.

As long as the widget can be told to use a given rectangular portion
of the screen, we should be fine: the current Emacs display engine is
perfectly capable of handling display elements that occupy an area of
certain dimensions.

> I'd think that such an interface to support "rich" display elements
> should be restrictive enough to not easily allow the "rich" display
> elements to misbehave.

Please tell more about the reasons for this conclusion.  I don't yet
see the difficulties you envision.

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Arthur Miller]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>> Sounds highly hypothetical.
>> Why is it hypothetical?
>
> Because it's pretty far from what existing code can do.
That's true :-) Thought you were referring to what is possible with
libmpv. Yes, no idea if/how easily those capabilites can be exploited
from Emacs; but from the libmpv point of view, it is possible.

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Arthur Miller]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
> Each proposed external library requires a cost/benefit judgment.
> In my judgment, the benefits of linking a video library are small
> because it is just a frill.
:-)

Not necessary just a frill; wouldn't <video> element be nice? But for
that one Emacs can maybe wrap ffmepg directly instead; I don't know, but
then question is still stame; external lib.

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Arthur Miller]

Eli Zaretskii <eliz@gnu.org> writes:

> As long as the widget can be told to use a given rectangular portion
> of the screen, we should be fine: the current Emacs display engine is
> perfectly capable of handling display elements that occupy an area of
> certain dimensions.

How can I tell any drawing routines to use certain rectangular area on
display so it is in harmony with Emacs display engine? I thought it was
done with faces and inserting characters with certain face; but I am not
sure how it was actually done.

Are there callbacks I can connect too if I would like to update opengl
viewport that I can hook easily into? Where do I look? I am not sure I
understand it from image.c; it was there I was looking. I never looked
into xwidgets though. Can be just me; but if you can point me to right
place to look at I am thankful. 

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Eli Zaretskii]

> From: Arthur Miller <arthur.miller@live.com>
> Cc: Akira Kyle <akira@akirakyle.com>,  larsi@gnus.org,  emacs-devel@gnu.org
> Date: Sat, 28 Nov 2020 09:57:39 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > As long as the widget can be told to use a given rectangular portion
> > of the screen, we should be fine: the current Emacs display engine is
> > perfectly capable of handling display elements that occupy an area of
> > certain dimensions.
> 
> How can I tell any drawing routines to use certain rectangular area on
> display so it is in harmony with Emacs display engine?

On the display engine level, you need to define a new kind of "display
element" (see 'enum glyph_type' in dispextern.h) and new kind of
method for producing such a display element (see 'enum it_method' in
dispextern.h).  You will also need to write a BUILD_<FOO>_GLYPH_STRING
macro that produces a "glyph string" from one or more display elements
of this new type; see BUILD_XWIDGET_GLYPH_STRING in xdisp.c as an
example.  Glyph strings are the interface between xdisp.c, which
parses buffers and strings for display and performs display layout,
and xterm.c (see below) which actually draws the resulting display
elements.

On the Lisp level, you will probably use a special 'display' property
whose value specifies the widget with the necessary parameters, like
we do with images.

> Are there callbacks I can connect too if I would like to update opengl
> viewport that I can hook easily into? Where do I look? I am not sure I
> understand it from image.c; it was there I was looking. I never looked
> into xwidgets though. Can be just me; but if you can point me to right
> place to look at I am thankful. 

The actual drawing is in xterm.c (for X; it's w32term.c for
MS-Windows, nsterm.m for NS, and term.c for TTY frames).  Each type of
display element has a function there, which is called to draw the
display element.  For example, x_draw_image_glyph_string for drawing
images.  You will need to write a new function for this new display
element, whose parameters are passed from the new it_method method
above.  Not sure if this is what you meant by "callback to connect to
in order to update opengl viewport".

(image.c just prepares the data required for drawing an image; its
functions are not called when drawing the image, we just use the
pixmap produced by image.c and various other parameters, like image
dimensions etc.)

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Arthur Miller]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Arthur Miller <arthur.miller@live.com>
>> Cc: Akira Kyle <akira@akirakyle.com>,  larsi@gnus.org,  emacs-devel@gnu.org
>> Date: Sat, 28 Nov 2020 09:57:39 +0100
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> > As long as the widget can be told to use a given rectangular portion
>> > of the screen, we should be fine: the current Emacs display engine is
>> > perfectly capable of handling display elements that occupy an area of
>> > certain dimensions.
>> 
>> How can I tell any drawing routines to use certain rectangular area on
>> display so it is in harmony with Emacs display engine?
>
> On the display engine level, you need to define a new kind of "display
> element" (see 'enum glyph_type' in dispextern.h) and new kind of
> method for producing such a display element (see 'enum it_method' in
> dispextern.h).  You will also need to write a BUILD_<FOO>_GLYPH_STRING
> macro that produces a "glyph string" from one or more display elements
> of this new type; see BUILD_XWIDGET_GLYPH_STRING in xdisp.c as an
> example.  Glyph strings are the interface between xdisp.c, which
> parses buffers and strings for display and performs display layout,
> and xterm.c (see below) which actually draws the resulting display
> elements.
>
> On the Lisp level, you will probably use a special 'display' property
> whose value specifies the widget with the necessary parameters, like
> we do with images.
>
>> Are there callbacks I can connect too if I would like to update opengl
>> viewport that I can hook easily into? Where do I look? I am not sure I
>> understand it from image.c; it was there I was looking. I never looked
>> into xwidgets though. Can be just me; but if you can point me to right
>> place to look at I am thankful. 
>
> The actual drawing is in xterm.c (for X; it's w32term.c for
> MS-Windows, nsterm.m for NS, and term.c for TTY frames).  Each type of
> display element has a function there, which is called to draw the
> display element.  For example, x_draw_image_glyph_string for drawing
> images.  You will need to write a new function for this new display
> element, whose parameters are passed from the new it_method method
> above.  Not sure if this is what you meant by "callback to connect to
> in order to update opengl viewport".
>
> (image.c just prepares the data required for drawing an image; its
> functions are not called when drawing the image, we just use the
> pixmap produced by image.c and various other parameters, like image
> dimensions etc.)
Thank you for the kind tour! This was much more then I was hoping for :-).

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Akira Kyle]

On Sat, Nov 28, 2020 at 01:38 AM, Eli Zaretskii <eliz@gnu.org> 
wrote:

>> > It is a relatively simple matter to disable some 
>> > optimizations
>> > when a widget is displayed.  It is also relatively simple to 
>> > tell
>> > the widget to redraw itself when redisplay changed its 
>> > position on
>> > the screen.
>> 
>> Yes, but you had previously pointed to that fact that xwidgets 
>> needs to such disable such optimizations as part of the 
>> "kludge" 
>> that would need to be resolved for xwidgets to be considered 
>> viable for a path forward. Having looked at that, I'm not sure 
>> gtk 
>> will ever want to cooperate without doing things that would be 
>> considered hacks by gtk folks.
>
> I'd appreciate more details on the GTK part of this.  I only 
> said that
> from the Emacs display engine side things are not too hairy. 
> from my
> POV, xwidgets integration simply stopped short of going all the 
> way
> towards integration into the redisplay framework, and I don't 
> think
> anyone said before that this was done because better integration 
> was
> impossible or very hard.

One such fundamental incompatibility stems from the fact that 
Emacs may have multiple views on one buffer split across various 
windows and/or frames. Gtk assumes that each widget will only have 
one view. The 'webkit xwidget type currently works around this by 
drawing to an offscreen window then for every window that's 
supposed to display the widget, creating a gtk drawing area widget 
and copying the offscreen window's surface to the drawing 
surface. I don't think gtk really intends for ofscreen widgets to 
be used this way and as far as I can tell offscreen window widgets 
have been removed in gtk4.

Another issue is that emacs uses a GtkFixed (or a custom 
subclassed EmacsFixed for gtk3) as the container widget for the 
main frame area. There isn't a mechanism to control z-ordering of 
child widgets other than removing and re-adding them in the 
desired z-order, which is what emacs-webkit has to do in order for 
child frames to not render below the webkit widget.

>> > I don't see why you consider this such a significant problem. 
>> > The
>> > number of interfaces that such a "rich" display element needs 
>> > to
>> > support is very small and well-defined.  The problem with 
>> > xwidgets
>> > in this regard is that its interaction with the display code 
>> > was
>> > left unfinished, it basically just copy/pastes the code which
>> > supports display of images, which is not 100% correct, since
>> > xwidgets don't display a static bitmap.
>> 
>> The "number interfaces" will likely be small no matter what 
>> solution is taken, its more about how easy you want to make it 
>> for 
>> a "rich" display element to really mess with Emacs' display and 
>> input handling. For example with xwidgets the onus is on the 
>> widget implementation to not totally steal keyboard focus from 
>> Emacs. I've found that gtk widgets really like stealing 
>> keyboard 
>> focus so its easy to end up in a state you can't escape out 
>> of.
>
> Keyboard focus is a separate issue.  I only talked about 
> integration
> with redisplay.
>
> For keyboard input integration, we'd need to develop some
> infrastructure, if the existing one doesn't provide enough 
> hooks.  If
> you or someone else can describe the details, we could discuss 
> that in
> more practical terms.  I think it's too soon to give up on the
> practical possibility to solve these problems before we 
> discussed
> that, or are even fully aware of the problems and the potential
> solutions.
>
>> Similarly gtk widgets won't always easily draw themselves 
>> where you want at the size you want so its easy for widgets to 
>> render, say, across window boundaries.
>
> If that is true, then your method of letting the widget display 
> in a
> window won't work well either, right?  IOW, whether the screen 
> area
> given to the widget is a full Emacs window or just its part is 
> not an
> important aspect of the integration of such widgets into the 
> Emacs
> display.
>
>> The xwidget approach requires non-insignificant work on the 
>> widget
>> end to ensure the widgets will work within the confines of 
>> Emacs'
>> redisplay.
>
> As long as the widget can be told to use a given rectangular 
> portion
> of the screen, we should be fine: the current Emacs display 
> engine is
> perfectly capable of handling display elements that occupy an 
> area of
> certain dimensions.

This is more an issue on the gtk side of things. With the webkit 
widget it isn't an issue since the widget doesn't define a minimum 
size so any size you request it to render at, it will 
obey. However other widgets, such as buttons, sliders, etc, have a 
minimum size they will render at. If you tell it to render itself 
smaller, it will refuse and so it's impossible to simply tell the 
widget "here's the space you have to go in". This becomes a 
problem when the widget needs to be clipped at a window boundary 
as gtk will happily draw the widgets across them as gtk has no 
internal knowledge of Emacs window boundaries being a place that 
widget must be clipped at. Thus every widget that will live in a 
buffer will need to be inside a custom container than handles this 
clipping for the widget.

>> I'd think that such an interface to support "rich" display 
>> elements
>> should be restrictive enough to not easily allow the "rich" 
>> display
>> elements to misbehave.
>
> Please tell more about the reasons for this conclusion.  I don't 
> yet
> see the difficulties you envision.

While all these issues can be overcome in one way or another, I 
think they point at the differences both the UI paradigms of Emacs 
and gtk and the subsequent differences in how each want to draw to 
the screen. In addition to the above specific examples of 
difficulties I've encountered so far with making gtk widgets work 
well with Emacs' display, there's the fundamental problem of this 
route for "richer display elements" not being portable. It'll 
always be confined to Emacs builds `--with-x` (or `--with-pgtk` 
now). While I feel this is fine for emacs-webkit, I think it would 
be advantageous to not tie such a feature to gtk itself.

The direction I'm thinking of going in the future since I feel 
like thus far the xwidget route has been fairly troublesome is to 
expose interfaces to the cairo surfaces used in image.c. This 
could be through modifying image.c or through defining a new 
display element type. I would try to expose the cairo API to lisp 
and allow dynamic modules to directly draw to a cairo surface 
given to it through the lisp interface. Gtk along with many other 
libraries (such as librsvg and poppler) support drawing to a cairo 
surface and then Emacs gets to have full control over how that 
cario surface ends up on screen. I would see this as a stepping 
stone to attempting to do something similar with a opengl surface.

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

The kind of argument you are making is entirely spurious.

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[martin rudalics]

 > Another issue is that emacs uses a GtkFixed (or a custom subclassed
 > EmacsFixed for gtk3) as the container widget for the main frame
 > area. There isn't a mechanism to control z-ordering of child widgets
 > other than removing and re-adding them in the desired z-order, which
 > is what emacs-webkit has to do in order for child frames to not render
 > below the webkit widget.

I think I see something like this in the pgtk branch - a child frame
appearing beneath the parent frame's horizontal scroll bar.

 > This is more an issue on the gtk side of things. With the webkit
 > widget it isn't an issue since the widget doesn't define a minimum
 > size so any size you request it to render at, it will obey. However
 > other widgets, such as buttons, sliders, etc, have a minimum size they
 > will render at. If you tell it to render itself smaller, it will
 > refuse and so it's impossible to simply tell the widget "here's the
 > space you have to go in". This becomes a problem when the widget needs
 > to be clipped at a window boundary as gtk will happily draw the
 > widgets across them as gtk has no internal knowledge of Emacs window
 > boundaries being a place that widget must be clipped at. Thus every
 > widget that will live in a buffer will need to be inside a custom
 > container than handles this clipping for the widget.

We are already bitten by this often enough: Toolbars that auto resize
their frame, scroll bars that extend into the next window.

martin

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Eli Zaretskii]

> From: Akira Kyle <akira@akirakyle.com>
> Cc: larsi@gnus.org, emacs-devel@gnu.org
> Date: Sat, 28 Nov 2020 18:29:08 -0700
> 
> One such fundamental incompatibility stems from the fact that 
> Emacs may have multiple views on one buffer split across various 
> windows and/or frames. Gtk assumes that each widget will only have 
> one view. The 'webkit xwidget type currently works around this by 
> drawing to an offscreen window then for every window that's 
> supposed to display the widget, creating a gtk drawing area widget 
> and copying the offscreen window's surface to the drawing 
> surface. I don't think gtk really intends for ofscreen widgets to 
> be used this way and as far as I can tell offscreen window widgets 
> have been removed in gtk4.
>
> Another issue is that emacs uses a GtkFixed (or a custom 
> subclassed EmacsFixed for gtk3) as the container widget for the 
> main frame area. There isn't a mechanism to control z-ordering of 
> child widgets other than removing and re-adding them in the 
> desired z-order, which is what emacs-webkit has to do in order for 
> child frames to not render below the webkit widget.

The same problems plague your experiment with a module, don't they?

Anyway, this just tells me that GTK is not a good starting point for
embedding widgets.  Do other projects build embedded widgets on such
shaky grounds?

> > As long as the widget can be told to use a given rectangular
> > portion of the screen, we should be fine: the current Emacs
> > display engine is perfectly capable of handling display elements
> > that occupy an area of certain dimensions.
> 
> This is more an issue on the gtk side of things. With the webkit 
> widget it isn't an issue since the widget doesn't define a minimum 
> size so any size you request it to render at, it will 
> obey. However other widgets, such as buttons, sliders, etc, have a 
> minimum size they will render at. If you tell it to render itself 
> smaller, it will refuse and so it's impossible to simply tell the 
> widget "here's the space you have to go in". This becomes a 
> problem when the widget needs to be clipped at a window boundary 
> as gtk will happily draw the widgets across them as gtk has no 
> internal knowledge of Emacs window boundaries being a place that 
> widget must be clipped at.

This is solvable.  We already have similar situation in the display
engine, where some display element cannot be usefully "clipped".  We
either don't display it at all or display it on the next screen line,
depending on the wrap mode.

> The direction I'm thinking of going in the future since I feel 
> like thus far the xwidget route has been fairly troublesome is to 
> expose interfaces to the cairo surfaces used in image.c. This 
> could be through modifying image.c or through defining a new 
> display element type. I would try to expose the cairo API to lisp 
> and allow dynamic modules to directly draw to a cairo surface 
> given to it through the lisp interface. Gtk along with many other 
> libraries (such as librsvg and poppler) support drawing to a cairo 
> surface and then Emacs gets to have full control over how that 
> cario surface ends up on screen. I would see this as a stepping 
> stone to attempting to do something similar with a opengl surface.

My suggestion is to think how to integrate this idea into Emacs
proper, without introducing modules into the equation.  Modules add
another layer of complexity, so I would suggest to design the solution
inside Emacs, and only after that see how to do that from a module.
One step at a time.

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Akira Kyle]

On Sun, Nov 29, 2020 at 12:01 PM, Eli Zaretskii <eliz@gnu.org> 
wrote:

>> One such fundamental incompatibility stems from the fact that 
>> Emacs may have multiple views on one buffer split across 
>> various 
>> windows and/or frames. Gtk assumes that each widget will only 
>> have 
>> one view. The 'webkit xwidget type currently works around this 
>> by 
>> drawing to an offscreen window then for every window that's 
>> supposed to display the widget, creating a gtk drawing area 
>> widget 
>> and copying the offscreen window's surface to the drawing 
>> surface. I don't think gtk really intends for ofscreen widgets 
>> to 
>> be used this way and as far as I can tell offscreen window 
>> widgets 
>> have been removed in gtk4.
>>
>> Another issue is that emacs uses a GtkFixed (or a custom 
>> subclassed EmacsFixed for gtk3) as the container widget for the 
>> main frame area. There isn't a mechanism to control z-ordering 
>> of 
>> child widgets other than removing and re-adding them in the 
>> desired z-order, which is what emacs-webkit has to do in order 
>> for 
>> child frames to not render below the webkit widget.
>
> The same problems plague your experiment with a module, don't 
> they?

More or less yes. Restricting myself to just a webkit widget 
allows me to not have to solve these problems generally. For 
example I don't draw the widget offscreen, instead I take a lesson 
from exwm and try to force Emacs to only ever display at most one 
window per webkit buffer.

> Anyway, this just tells me that GTK is not a good starting point 
> for
> embedding widgets.  Do other projects build embedded widgets on 
> such
> shaky grounds?

I wouldn't say GTK isn't good for embedding widgets. In fact I'd 
say it's probably great if you're wanting to extend an existing 
GTK app with custom widgets. The problem is Emacs, despite using 
GTK as a toolkit, isn't really a GTK app. Emacs just uses GTK to 
handle getting input and drawing a surface to the screen. Because 
Emacs isn't itself composed of GTK "normal" widgets, but rather 
just one GtkFixed container it does all its drawing in, you can't 
expect GTK widgets added along side what Emacs draws in that area 
to play nicely together without significant effort. And that 
effort boils down to making GTK aware of what Emacs has drawn and 
vice versa. If all the UI components of Emacs such as windows, the 
mode line, etc, were each implemented as a sensible GTK widget, 
then this wouldn't be so much of an issue since then GTK would 
already know where and what things are.

>> > As long as the widget can be told to use a given rectangular
>> > portion of the screen, we should be fine: the current Emacs
>> > display engine is perfectly capable of handling display 
>> > elements
>> > that occupy an area of certain dimensions.
>> 
>> This is more an issue on the gtk side of things. With the 
>> webkit 
>> widget it isn't an issue since the widget doesn't define a 
>> minimum 
>> size so any size you request it to render at, it will 
>> obey. However other widgets, such as buttons, sliders, etc, 
>> have a 
>> minimum size they will render at. If you tell it to render 
>> itself 
>> smaller, it will refuse and so it's impossible to simply tell 
>> the 
>> widget "here's the space you have to go in". This becomes a 
>> problem when the widget needs to be clipped at a window 
>> boundary 
>> as gtk will happily draw the widgets across them as gtk has no 
>> internal knowledge of Emacs window boundaries being a place 
>> that 
>> widget must be clipped at.
>
> This is solvable.  We already have similar situation in the 
> display
> engine, where some display element cannot be usefully "clipped". 
> We
> either don't display it at all or display it on the next screen 
> line,
> depending on the wrap mode.

I don't doubt its solvable, but how satisfactorily and at what 
complexity? I would say not displaying an element when some 
portion could be displayed is visually very unsatisfying. What if 
the element is large and 99% of it could be visible? What if the 
element is wider than the window? What if 
`window-resize-pixelwise` or `frame-resize-pixelwise` non-nil and 
the element is on the last, partially clipped line?

>> The direction I'm thinking of going in the future since I feel 
>> like thus far the xwidget route has been fairly troublesome is 
>> to 
>> expose interfaces to the cairo surfaces used in image.c. This 
>> could be through modifying image.c or through defining a new 
>> display element type. I would try to expose the cairo API to 
>> lisp 
>> and allow dynamic modules to directly draw to a cairo surface 
>> given to it through the lisp interface. Gtk along with many 
>> other 
>> libraries (such as librsvg and poppler) support drawing to a 
>> cairo 
>> surface and then Emacs gets to have full control over how that 
>> cario surface ends up on screen. I would see this as a stepping 
>> stone to attempting to do something similar with a opengl 
>> surface.
>
> My suggestion is to think how to integrate this idea into Emacs
> proper, without introducing modules into the equation.  Modules 
> add
> another layer of complexity, so I would suggest to design the 
> solution
> inside Emacs, and only after that see how to do that from a 
> module.
> One step at a time.

Of course the first step would be to work inside Emacs proper, and 
expose lisp interfaces. However I think designing such a feature 
should keep eventual integration with modules in mind. Especially 
since in general graphics is very performance sensitive and so 
doing some drawing in elisp just might not cut it and many 
interesting applications of such a feature would require 
interfacing with an external c library. This is all pretty 
hypothetical at this point though and as I keep learning with 
Emacs C code: its hard to speculate on the best way to do 
something without actually trying something out first.

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[martin rudalics]

 > This is solvable.  We already have similar situation in the display
 > engine, where some display element cannot be usefully "clipped".  We
 > either don't display it at all or display it on the next screen line,
 > depending on the wrap mode.

For some value of "solvable".  Emacs has no idea about how large the
external GTK toolbar gets so where would it wrap or clip it?

martin

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Eli Zaretskii]

> From: Akira Kyle <akira@akirakyle.com>
> Cc: larsi@gnus.org, emacs-devel@gnu.org
> Date: Sun, 29 Nov 2020 20:54:01 -0700
> 
> > Anyway, this just tells me that GTK is not a good starting point
> > for embedding widgets.  Do other projects build embedded widgets
> > on such shaky grounds?
> 
> I wouldn't say GTK isn't good for embedding widgets. In fact I'd 
> say it's probably great if you're wanting to extend an existing 
> GTK app with custom widgets. The problem is Emacs, despite using 
> GTK as a toolkit, isn't really a GTK app.

First, which of the problems you mentioned are related to Emacs not
being a GTK application?

And second, there's a branch in our repository where Emacs _is_ made
to be a GTK application, AFAIU.

> > This is solvable.  We already have similar situation in the
> > display engine, where some display element cannot be usefully
> > "clipped".  We either don't display it at all or display it on the
> > next screen line, depending on the wrap mode.
> 
> I don't doubt its solvable, but how satisfactorily and at what 
> complexity?

No complexity at all, it's almost trivial, because the display code
already does that.  As for "satisfactorily" part: what can be
unsatisfactory about displaying the widget on the next screen line?

> What if the element is large and 99% of it could be visible? What if
> the element is wider than the window?

These are marginal cases: you are talking about very large widgets
that in addition refuse to be resized.  How many are like that?  And
what else can we do with widgets that are (1) large, (2) won't resize,
and (3) cannot be clipped?

The perfect should not be the enemy of the good, right?

> What if `window-resize-pixelwise` or `frame-resize-pixelwise`
> non-nil and the element is on the last, partially clipped line?

We don't display it.  How is this different from a any other large
display element that cannot be shown in the viewport?

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Eli Zaretskii]

> Cc: larsi@gnus.org, emacs-devel@gnu.org
> From: martin rudalics <rudalics@gmx.at>
> Date: Mon, 30 Nov 2020 10:05:07 +0100
> 
>  > This is solvable.  We already have similar situation in the display
>  > engine, where some display element cannot be usefully "clipped".  We
>  > either don't display it at all or display it on the next screen line,
>  > depending on the wrap mode.
> 
> For some value of "solvable".  Emacs has no idea about how large the
> external GTK toolbar gets so where would it wrap or clip it?

That's a different matter: those decorations are not managed by the
Emacs display engine.

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Akira Kyle]

On Mon, Nov 30, 2020 at 08:39 AM, Eli Zaretskii <eliz@gnu.org> 
wrote:

>> > Anyway, this just tells me that GTK is not a good starting 
>> > point
>> > for embedding widgets.  Do other projects build embedded 
>> > widgets
>> > on such shaky grounds?
>> 
>> I wouldn't say GTK isn't good for embedding widgets. In fact 
>> I'd 
>> say it's probably great if you're wanting to extend an existing 
>> GTK app with custom widgets. The problem is Emacs, despite 
>> using 
>> GTK as a toolkit, isn't really a GTK app.
>
> First, which of the problems you mentioned are related to Emacs 
> not
> being a GTK application?

All of them.

> And second, there's a branch in our repository where Emacs _is_ 
> made
> to be a GTK application, AFAIU.

If you're referring to Yukki Harano's feature/pgtk branch, then 
that's actually what I'm primarily targeting in my development of 
emacs-webkit since the usual Emacs compiled `--with-x` has an 
additional problem from what I've previously described. It seems 
gtk widgets (or at least the webkit widget), flicker constantly on 
`--with-x`. It seems to be related to the double buffering 
implementation since setting the frame parameter 
`(inhibit-double-buffering . t)` fixes it. I haven't had the 
motivation to debug this due to the mess of xterm.c and because I 
see pgtk as the future of Emacs on GNU/Linux systems as X becomes 
increasingly obsolete.

AFAIU the pgtk version, like the ns version it was modeled after, 
doesn't implement everything in the main frame area of Emacs as 
widgets, only the tool and menu bars are implemented as proper 
widgets. The main frame area is treated as just one big canvas 
that redisplay works its magic on as it would if it were just a 
TUI.

>> > This is solvable.  We already have similar situation in the
>> > display engine, where some display element cannot be usefully
>> > "clipped".  We either don't display it at all or display it 
>> > on the
>> > next screen line, depending on the wrap mode.
>> 
>> I don't doubt its solvable, but how satisfactorily and at what 
>> complexity?
>
> No complexity at all, it's almost trivial, because the display 
> code
> already does that.  As for "satisfactorily" part: what can be
> unsatisfactory about displaying the widget on the next screen 
> line?

That's fine, and would be the expected behavior when lines are 
supposed to wrap. But when lines are not wrapped, I would not 
expect a display element to disappear as soon as one pixel of it 
exceeds the window border.

>> What if the element is large and 99% of it could be visible? 
>> What if
>> the element is wider than the window?
>
> These are marginal cases: you are talking about very large 
> widgets
> that in addition refuse to be resized.  How many are like that? 
> And
> what else can we do with widgets that are (1) large, (2) won't 
> resize,
> and (3) cannot be clipped?
>
> The perfect should not be the enemy of the good, right?

Take the drawing canvas idea, the canvas widget would be the size 
of an image. Already I find the way redisplay doesn't like to 
display images clipped at the top of the window jarring. If I 
wanted to zoom into the widget if it was displaying something 
detailed, I wouldn't want it to suddenly disappear. I think Emacs 
needs to be able to clip the elements its displays in order to not 
end up with what I would call surprising and frustrating 
behavior. Hence my original argument that the way gtk widgets are 
currently handled is not in general, a good path forward for such 
features.

>> What if `window-resize-pixelwise` or `frame-resize-pixelwise`
>> non-nil and the element is on the last, partially clipped line?
>
> We don't display it.  How is this different from a any other 
> large
> display element that cannot be shown in the viewport?

Images are the only other large display element that I'm aware of 
and images are clipped in such cases, which is good as large 
images make their line very tall so very often they end up 
partially displayed in the last line.

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Eli Zaretskii]

> From: Akira Kyle <akira@akirakyle.com>
> Cc: larsi@gnus.org, emacs-devel@gnu.org
> Date: Mon, 30 Nov 2020 10:03:55 -0700
> 
> > No complexity at all, it's almost trivial, because the display
> > code already does that.  As for "satisfactorily" part: what can be
> > unsatisfactory about displaying the widget on the next screen
> > line?
> 
> That's fine, and would be the expected behavior when lines are 
> supposed to wrap. But when lines are not wrapped, I would not 
> expect a display element to disappear as soon as one pixel of it 
> exceeds the window border.

And yet that's exactly what happens with other display elements, for
example wide characters on TTY display.

Again, AFAIU we are talking about rare use cases, which you said
cannot be displayed in any other way, so what do you expect Emacs to
do in such restrictive situations??

> Already I find the way redisplay doesn't like to display images
> clipped at the top of the window jarring.

Not sure what you are talking about, but I'm sure whatever is jarring
can be fixed by relatively simple augmentations to the display code.

> If I wanted to zoom into the widget if it was displaying something
> detailed, I wouldn't want it to suddenly disappear.

It doesn't have to, we can scroll the window instead, so that the
enlarged widget is brought into the viewport.

Again, what else do you expect Emacs to do when you yourself said
these widgets don't allow any other solution??

> I think Emacs needs to be able to clip the elements its displays in
> order to not end up with what I would call surprising and
> frustrating behavior.

But you yourself said this is not allowed in these cases, so how can
Emacs do what is not allowed??

> >> What if `window-resize-pixelwise` or `frame-resize-pixelwise`
> >> non-nil and the element is on the last, partially clipped line?
> >
> > We don't display it.  How is this different from a any other large
> > display element that cannot be shown in the viewport?
> 
> Images are the only other large display element that I'm aware of 
> and images are clipped in such cases

But you yourself said that clipping these widgets is impossible.  What
am I missing in this discussion?

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Akira Kyle]

On Mon, Nov 30, 2020 at 11:11 AM, Eli Zaretskii <eliz@gnu.org> 
wrote:

>> > No complexity at all, it's almost trivial, because the 
>> > display
>> > code already does that.  As for "satisfactorily" part: what 
>> > can be
>> > unsatisfactory about displaying the widget on the next screen
>> > line?
>> 
>> That's fine, and would be the expected behavior when lines are 
>> supposed to wrap. But when lines are not wrapped, I would not 
>> expect a display element to disappear as soon as one pixel of 
>> it 
>> exceeds the window border.
>
> And yet that's exactly what happens with other display elements, 
> for
> example wide characters on TTY display.
>
> Again, AFAIU we are talking about rare use cases, which you said
> cannot be displayed in any other way, so what do you expect 
> Emacs to
> do in such restrictive situations??
>
>> Already I find the way redisplay doesn't like to display images
>> clipped at the top of the window jarring.
>
> Not sure what you are talking about, but I'm sure whatever is 
> jarring
> can be fixed by relatively simple augmentations to the display 
> code.
>
>> If I wanted to zoom into the widget if it was displaying 
>> something
>> detailed, I wouldn't want it to suddenly disappear.
>
> It doesn't have to, we can scroll the window instead, so that 
> the
> enlarged widget is brought into the viewport.
>
> Again, what else do you expect Emacs to do when you yourself 
> said
> these widgets don't allow any other solution??
>
>> I think Emacs needs to be able to clip the elements its 
>> displays in
>> order to not end up with what I would call surprising and
>> frustrating behavior.
>
> But you yourself said this is not allowed in these cases, so how 
> can
> Emacs do what is not allowed??
>
>> >> What if `window-resize-pixelwise` or 
>> >> `frame-resize-pixelwise`
>> >> non-nil and the element is on the last, partially clipped 
>> >> line?
>> >
>> > We don't display it.  How is this different from a any other 
>> > large
>> > display element that cannot be shown in the viewport?
>> 
>> Images are the only other large display element that I'm aware 
>> of 
>> and images are clipped in such cases
>
> But you yourself said that clipping these widgets is impossible. 
> What
> am I missing in this discussion?

Sorry we've gotten a bit off the original point. I was trying to 
illustrate that for such "rich" display elements clipping would 
seem to be a necessity in order to avoid surprising or frustrating 
behavior. I understand Emacs can work around an element that 
doesn't permit being clipped, but it will be exactly that -- a 
work around to a problem Emacs shouldn't have solve. Thus clipping 
has to happen on the GTK side, which isn't impossible, but 
requires additional complexity. Overriding Widgets so they can be 
clipped is yet another example of trying to draw GTK widgets in 
ways they weren't intended to be drawn (and the GTK folks would 
probably consider this all very bad practice).

My whole point is that I've become more convinced that trying to 
integrate GTK widgets into Emacs redisplay isn't a good way to 
achieve the desired functionality and it would be worth exploring 
other, more portable solutions. I don't think we need to continue 
discussing the finer points on clipping.

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Tomas Hlavaty]

On Mon 30 Nov 2020 at 10:03, Akira Kyle <akira@akirakyle.com> wrote:
> and because I see pgtk as the future of Emacs on GNU/Linux systems as
> X becomes increasingly obsolete.

I am not sure such future is bright.

In that future, is there no way to do graphics without a widget toolkit?

> AFAIU the pgtk version, like the ns version it was modeled after, 
> doesn't implement everything in the main frame area of Emacs as 
> widgets, only the tool and menu bars are implemented as proper 
> widgets.

Also popup menus that pop up on a mouse click are gtk iirc.

And maybe scrollbars?

It may have changed since I turned these distractions off a long time
ago.

I think that they are also fundamentally flawed: How can I search in
menu?  Maybe it should be a buffer.  New Gtk hides menu under an
unintuitive button anyway so this could just display searchable Menu
buffer.

> The main frame area is treated as just one big canvas that redisplay
> works its magic on as it would if it were just a TUI.

Maybe that is a good thing.

Widget toolkit dependency is a heavy price to pay for.

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Tomas Hlavaty]

On Sun 29 Nov 2020 at 20:54, Akira Kyle <akira@akirakyle.com> wrote:
> However I think designing such a feature should keep eventual
> integration with modules in mind. Especially since in general graphics
> is very performance sensitive and so doing some drawing in elisp just
> might not cut it and many interesting applications of such a feature
> would require interfacing with an external c library.

Not necessarily.

There could also be a specialized program which could handle drawing.
In fact there already is such a program: w3mimgdisplay and is part of
the w3m web browser and pager.  It is also used in the file manager
ranger.  It works both with or without X.  I am using it in
emacs-framebuffer too to display images and documents in Emacs.

The issues I am facing have nothing to do with modules or libraries.

For example:

It would be more convenient, if there was a way to specify elisp
function to draw an image.  By default, it could just call the existing
C code but this would also allow me to specify a different elisp
function which would then for example call w3mimgdisplay.

It would be more convenient, if an image was represented as elisp data
instead of C data.  iirc there is no way to add new image types without
touching C.

Is there a way to turn off cursor for a buffer?  (Blinking cursor
disrupts the drawn image.)

Is there a way run a function after emacs changes something on the
screen in a buffer?

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Arthur Miller]

Tomas Hlavaty <tom@logand.com> writes:

> On Sun 29 Nov 2020 at 20:54, Akira Kyle <akira@akirakyle.com> wrote:
>> However I think designing such a feature should keep eventual
>> integration with modules in mind. Especially since in general graphics
>> is very performance sensitive and so doing some drawing in elisp just
>> might not cut it and many interesting applications of such a feature
>> would require interfacing with an external c library.
>
> Not necessarily.
>
> There could also be a specialized program which could handle drawing.
> In fact there already is such a program: w3mimgdisplay and is part of
> the w3m web browser and pager.  It is also used in the file manager
> ranger.  It works both with or without X.  I am using it in
> emacs-framebuffer too to display images and documents in Emacs.
>
> The issues I am facing have nothing to do with modules or libraries.
>
> For example:
>
> It would be more convenient, if there was a way to specify elisp
> function to draw an image.  By default, it could just call the existing
> C code but this would also allow me to specify a different elisp
> function which would then for example call w3mimgdisplay.
>
> It would be more convenient, if an image was represented as elisp data
> instead of C data.
:-) I don't think you have thought well about it;

but really nothing forbids you to try to reprsent images as lisp.

You can take any de-compressed image and read in raw pixels in as a byte
buffer and just shuffle around numbers and see how it
works for you.Image is nothing but a bunch of numbers + some tiny
metadata on top of it. Take some of netpbm formats and you have "textual
image" you can manipulate on per-pixel level with lisp.

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Eli Zaretskii]

> From: Tomas Hlavaty <tom@logand.com>
> Date: Tue, 01 Dec 2020 09:01:20 +0100
> 
> It would be more convenient, if there was a way to specify elisp
> function to draw an image.  By default, it could just call the existing
> C code but this would also allow me to specify a different elisp
> function which would then for example call w3mimgdisplay.

You need to be aware of how this works in Emacs.  In a nutshell,
there's no function "to draw the image": we call system APIs to do
that.  By the time we need to draw the image, it is already
preprocessed into a bitmap/pixmap, so we basically ask the GUI
subsystem of the OS to bitblt the image to some coordinates on the
screen which we compute.

The preprocessing of the images into bitmap/pixmap happens when the
display engine sees a 'display' property that specifies an image.
At that time we call some image library to convert the image's data
into bitmap/pixmap; then we cache the result and produce a handle to
the cache that the drawing routines will use.

So when you ask for the above, you need to fit into this scheme.

> It would be more convenient, if an image was represented as elisp data
> instead of C data.

create-image already knows how to accept image data.

> iirc there is no way to add new image types without touching C.

No, because you need to know how to convert that into the internal
format used to draw the image, which is platform dependent.

Why is it a problem to use one of the known image types?  E.g., SVG
(via the svg.el library).

> Is there a way to turn off cursor for a buffer?

There's the cursor-type variable.

> Is there a way run a function after emacs changes something on the
> screen in a buffer?

That depends by what you mean by "after emacs changes something on the
screen", since in Emacs redisplay is independent/separate of data
changes.  Please elaborate.

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Tomas Hlavaty]

On Tue 01 Dec 2020 at 16:36, Arthur Miller <arthur.miller@live.com> wrote:
> Tomas Hlavaty <tom@logand.com> writes:
>> It would be more convenient, if an image was represented as elisp data
>> instead of C data.
> :-) I don't think you have thought well about it;

Why would you say that?

> but really nothing forbids you to try to reprsent images as lisp.

What do you mean exactly?

> You can take any de-compressed image and read in raw pixels in as a byte
> buffer and just shuffle around numbers and see how it
> works for you.Image is nothing but a bunch of numbers + some tiny
> metadata on top of it. Take some of netpbm formats and you have "textual
> image" you can manipulate on per-pixel level with lisp.

Image in emacs is represented in C as "struct image".

Image type in emacs is represented in C as "struct image_type".

The issue is that there doesn't seem to be a way to extend those (and/or
define new) in pure elisp.

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Tomas Hlavaty]

On Tue 01 Dec 2020 at 17:58, Eli Zaretskii <eliz@gnu.org> wrote:
>> From: Tomas Hlavaty <tom@logand.com>
>> Date: Tue, 01 Dec 2020 09:01:20 +0100
>> 
>> It would be more convenient, if there was a way to specify elisp
>> function to draw an image.  By default, it could just call the existing
>> C code but this would also allow me to specify a different elisp
>> function which would then for example call w3mimgdisplay.
>
> You need to be aware of how this works in Emacs.  In a nutshell,
> there's no function "to draw the image": we call system APIs to do
> that.  By the time we need to draw the image, it is already
> preprocessed into a bitmap/pixmap, so we basically ask the GUI
> subsystem of the OS to bitblt the image to some coordinates on the
> screen which we compute.
>
> The preprocessing of the images into bitmap/pixmap happens when the
> display engine sees a 'display' property that specifies an image.
> At that time we call some image library to convert the image's data
> into bitmap/pixmap; then we cache the result and produce a handle to
> the cache that the drawing routines will use.
>
> So when you ask for the above, you need to fit into this scheme.

Thanks for the great explanation.

I do not have an issue with the way it works currently.

Think of w3mimgdisplay program

- as an alternatine "system API and GUI subsystem of the OS to bitblt
  the image to some coordinates on the screen which we compute".

- as an alternative "some image library to convert the image's data into
  bitmap/pixmap; then we cache the result and produce a handle to the
  cache that the drawing routines will use".

w3mimgdisplay can supply both functions.

w3mimgdisplay program can be run as a kind of "server", where I can say:

- draw this image to these coordinates on the screen

- or additionally, cache image's data and return a handle to the cache,
  which will make redrawing fast.

I think that the only issue for me is that there is no way to implement
an alternative "system API" and "call some image library" in pure elisp.

How much work would it be to enable alternative "system API" and "call
some image library" in pure elisp?

>> It would be more convenient, if an image was represented as elisp data
>> instead of C data.
>
> create-image already knows how to accept image data.

But those must be defined in C, see lookup_image_type.

Or is there a way to define custom image format in pure elisp?

It would be actually enough for me if my custom image was opaque to
emacs (except probably some attributes like width and height) and
w3mimgdisplay handled the rest.

>> iirc there is no way to add new image types without touching C.
>
> No, because you need to know how to convert that into the internal
> format used to draw the image, which is platform dependent.

I don't need to know that.  w3mimgdisplay program does that.  As far as
emacs is concerned, it needs to give to w3mimgdisplay filename to draw
and then use a handle to cache to ask w3mimgdisplay for redraw.

> Why is it a problem to use one of the known image types?  E.g., SVG
> (via the svg.el library).

Because those those are not universally available, e.g. on the console.

Because those might not be suitable.

>> Is there a way to turn off cursor for a buffer?
>
> There's the cursor-type variable.

Thanks.

This does not work on the console.

I suppose I need to use some console specific command/escape sequence.

Is there a way to hook a function when a buffer looses and gains focus?

>> Is there a way run a function after emacs changes something on the
>> screen in a buffer?
>
> That depends by what you mean by "after emacs changes something on the
> screen", since in Emacs redisplay is independent/separate of data
> changes.  Please elaborate.

Example: I am on the console.  I am in dired buffer.  I press my key.
w3mimgdisplay program draws my image on the screen on the specified
coordinates "over" emacs.  All is well (except the blinking cursor which
we can ignore for now).  Now I press another key to display the next
image in the dired buffer.  Emacs redraws some characters, that have
changed.  This disrupts the image drawn earlier and I need to detect,
that I need to tell w3mimgdisplay to redraw the image.  I need to hook
my redraw image function after emacs is done with its drawing.

So far I have to press a key to redraw image.

I tried using timer and redraw after say 1s but that is not so nice.

Is there such hook?  If not, would it be difficult to add it?

Thanks a lot for your help!

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Eli Zaretskii]

> From: Tomas Hlavaty <tom@logand.com>
> Date: Tue, 01 Dec 2020 18:33:56 +0100
> 
> w3mimgdisplay program can be run as a kind of "server", where I can say:
> 
> - draw this image to these coordinates on the screen
> 
> - or additionally, cache image's data and return a handle to the cache,
>   which will make redrawing fast.

How would Emacs communicate with w3mimgdisplay? via some pipe? is that
fast enough to be done in the middle of redisplay?

> I think that the only issue for me is that there is no way to implement
> an alternative "system API" and "call some image library" in pure elisp.

Why is that an issue?  And what does an implementation in pure Lisp
have to do with w3mimgdisplay?

> How much work would it be to enable alternative "system API" and "call
> some image library" in pure elisp?

If they support the same interfaces, very little.  But I'm not sure we
are on the same page yet, so we might be miscommunicating.

> >> It would be more convenient, if an image was represented as elisp data
> >> instead of C data.
> >
> > create-image already knows how to accept image data.
> 
> But those must be defined in C, see lookup_image_type.

You are supposed to submit data in one of the known formats.  Why is
that a problem? why do you care about the exact format of the image
data, and why do you insist to introduce a new type instead of using
existing types?

> Or is there a way to define custom image format in pure elisp?

No.

> It would be actually enough for me if my custom image was opaque to
> emacs (except probably some attributes like width and height) and
> w3mimgdisplay handled the rest.

That's okay, but I still don't see how all that will fit into the
current framework of displaying images.

> >> iirc there is no way to add new image types without touching C.
> >
> > No, because you need to know how to convert that into the internal
> > format used to draw the image, which is platform dependent.
> 
> I don't need to know that.  w3mimgdisplay program does that.

How can w3mimgdisplay know the internal format used by Emacs for
storing images?

> As far as emacs is concerned, it needs to give to w3mimgdisplay
> filename to draw and then use a handle to cache to ask w3mimgdisplay
> for redraw.

I suggest to take a look at x_draw_image_foreground and its
subroutines, this is where we actually put the image on the glass.
Then tell in a bit more detail how to integrate an external program
into that, keeping all the features we support in this regard, like
masks, clipping, slices, etc.

> > Why is it a problem to use one of the known image types?  E.g., SVG
> > (via the svg.el library).
> 
> Because those those are not universally available, e.g. on the console.

On the console you currently cannot have images in Emacs, period.
This is not because of image libraries, this is because the text-mode
display simply cannot currently support such display elements.

> Because those might not be suitable.

"Not suitable" in what ways?

> >> Is there a way to turn off cursor for a buffer?
> >
> > There's the cursor-type variable.
> 
> Thanks.
> 
> This does not work on the console.

(You never said you were talking about a console.)  We currently don't
expose console cursor control to Lisp.  Should be easy to add that, if
needed.

> Is there a way to hook a function when a buffer looses and gains focus?

Buffer cannot have focus.  Do you mean when the buffer's window is the
selected window?  There's window-selection-change-functions.

> >> Is there a way run a function after emacs changes something on the
> >> screen in a buffer?
> >
> > That depends by what you mean by "after emacs changes something on the
> > screen", since in Emacs redisplay is independent/separate of data
> > changes.  Please elaborate.
> 
> Example: I am on the console.  I am in dired buffer.  I press my key.
> w3mimgdisplay program draws my image on the screen on the specified
> coordinates "over" emacs.  All is well (except the blinking cursor which
> we can ignore for now).  Now I press another key to display the next
> image in the dired buffer.  Emacs redraws some characters, that have
> changed.  This disrupts the image drawn earlier and I need to detect,
> that I need to tell w3mimgdisplay to redraw the image.  I need to hook
> my redraw image function after emacs is done with its drawing.

I don't think we have such a hook, no.  But if we had it, using it
like you want would cause annoying flickering, because you'd
constantly redraw the image at very high frequency.

> Is there such hook?  If not, would it be difficult to add it?

It won't be hard to add it, but I'm not sure we want to add such a
hook, see above.

RE: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[arthur miller]

[-- Attachment #1: Type: text/plain, Size: 2562 bytes --]

"> Tomas Hlavaty <tom@logand.com> writes:
>> It would be more convenient, if an image was represented as elisp data
>> instead of C data.
> :-) I don't think you have thought well about it;

Why would you say that?"

Because of the other things you wrote before that citate about drawing images in external process. You have expressed your sentiments about year 2020, efficiency and external processes a day or two ago, and I read similar reasoning here.

If all you wish is just to display an image, sure you can do that with external application, but if you would like to perform something interesting with images, or display something many images, the performance would drop quite fast.

Since you refer to drawing images and mention image data, I assumed you are talking about real image data, i.e. pixels, not handles to lisp objects. With other words I thought you would like to use Lisp to do something interesting as processing images in Emacs on pixel level. When people mention image data that is usually what they mean. But assumptions are always fault on the one that assumes, so I apologize for that.

Otherwise, you could surely easily integrate netpbm, image/openmagic or even libgd (via python or perl wrapper) into Emacs via pure lisp and processes. I don't see what's the problem if that is what you wish.

-------- Originalmeddelande --------
Från: Tomas Hlavaty <tom@logand.com>
Datum: 2020-12-01 17:49 (GMT+01:00)
Till: emacs-devel@gnu.org
Ämne: Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

On Tue 01 Dec 2020 at 16:36, Arthur Miller <arthur.miller@live.com> wrote:
> Tomas Hlavaty <tom@logand.com> writes:
>> It would be more convenient, if an image was represented as elisp data
>> instead of C data.
> :-) I don't think you have thought well about it;

Why would you say that?

> but really nothing forbids you to try to reprsent images as lisp.

What do you mean exactly?

> You can take any de-compressed image and read in raw pixels in as a byte
> buffer and just shuffle around numbers and see how it
> works for you.Image is nothing but a bunch of numbers + some tiny
> metadata on top of it. Take some of netpbm formats and you have "textual
> image" you can manipulate on per-pixel level with lisp.

Image in emacs is represented in C as "struct image".

Image type in emacs is represented in C as "struct image_type".

The issue is that there doesn't seem to be a way to extend those (and/or
define new) in pure elisp.


[-- Attachment #2: Type: text/html, Size: 3451 bytes --]

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Tomas Hlavaty]

On Tue 01 Dec 2020 at 21:41, Eli Zaretskii <eliz@gnu.org> wrote:
> How would Emacs communicate with w3mimgdisplay? via some pipe?

Yes, pipe.

At the moment, in emacs-framebuffer, I just call it to display an image
using call-process-region.  It could be optimized with pipe, but this
has low priority for emacs-framebuffer as it is fast enough for my
needs.

If it was properly integrated with emacs at the level of image.el, then
pipe would be more optimized implementation, the way w3mimgdisplay is
actually meant to be used.

It is meant to be used this way: w3m browser spawns w3mimgdisplay and
talks to it via pipe.  w3m tells w3mimgdisplay to cache some images.
w3mimgdisplay tells w3m sizes of the images, w3m leaves space for the
images on the screen, draws text and then tells w3mimgdisplay to draw
the cached images in the right places on the screen.

> is that fast enough to be done in the middle of redisplay?

I am not exatly sure, what the concern is here.

The images do not travel via pipe.  Only meta-data like filenames,
coordinates and handles.

What exactly needs to be done in the middle of redisplay that is
relevant to drawing images?  Determining image width and height?
Something else like actually drawing the image?

The actual drawing could be postponed after Emacs has drawn its text.

>> I think that the only issue for me is that there is no way to
>> implement an alternative "system API" and "call some image library"
>> in pure elisp.
>
> Why is that an issue?

Because doing it in C requires significantly more effort and my free
time is limited.

> And what does an implementation in pure Lisp have to do with
> w3mimgdisplay?

Nothing.

Except the point above.

>> How much work would it be to enable alternative "system API" and
>> "call some image library" in pure elisp?
>
> If they support the same interfaces, very little.  But I'm not sure we
> are on the same page yet, so we might be miscommunicating.

What are the current interfaces?

>> >> It would be more convenient, if an image was represented as elisp
>> >> data instead of C data.
>> >
>> > create-image already knows how to accept image data.
>> 
>> But those must be defined in C, see lookup_image_type.
>
> You are supposed to submit data in one of the known formats.  Why is
> that a problem?

Because they do not work on the console.

And the reason they do not work on the console is because there is
hardcoded dependency on image drawing libraries which do not work on the
console.

I want Emacs to understand images even on the console.  But I have no
enough free time to do it in C.  If there was elisp level boundary
between image object and something that draws that image object, it
would be much easier.  I could just specify my own thing that draws that
image object without going so low level.  And it would work with any
existing elisp code like image.el transparently.

> why do you care about the exact format of the image data, and why do
> you insist to introduce a new type instead of using existing types?

I do not care about the exact format of the image data.

I do not care about the image data at all.

I care about extending Emacs' understanding of images in elisp at
meta-data level, not image data level.  For example, an image has kind
of type or format like png or jpeg, it has width and height, maybe some
other properties and some kind of opaque data blob.  I would like to be
able to specify and/or override how to draw images in elisp and not
having that hardcoded in C.

Additionally there is no reason why documents like pdf, epub, odt etc
could not be an image (which they basically are).  They have pages but
tiff also has more than one image.  There is also no reason why image
modes could not have similar capabilities to doc-view or pdf-tools.  All
this could be eventually unified.

>> Or is there a way to define custom image format in pure elisp?
>
> No.

That is a shame.

>> It would be actually enough for me if my custom image was opaque to
>> emacs (except probably some attributes like width and height) and
>> w3mimgdisplay handled the rest.
>
> That's okay, but I still don't see how all that will fit into the
> current framework of displaying images.

Maybe I do not need custom image type.

But I need to change the way images are drawn on the console, i.e. not
at all vs draw the image.

But the way image types are implemented currently does not allow me to
do that in elisp.

>> >> iirc there is no way to add new image types without touching C.
>> >
>> > No, because you need to know how to convert that into the internal
>> > format used to draw the image, which is platform dependent.
>> 
>> I don't need to know that.  w3mimgdisplay program does that.
>
> How can w3mimgdisplay know the internal format used by Emacs for
> storing images?

Why would w3mimgdisplay need to know that?

w3mimgdisplay just needs filename and coordinates.

>> As far as emacs is concerned, it needs to give to w3mimgdisplay
>> filename to draw and then use a handle to cache to ask w3mimgdisplay
>> for redraw.
>
> I suggest to take a look at x_draw_image_foreground and its
> subroutines, this is where we actually put the image on the glass.
> Then tell in a bit more detail how to integrate an external program
> into that, keeping all the features we support in this regard, like
> masks, clipping, slices, etc.

Thanks, I'll have a look.

>> > Why is it a problem to use one of the known image types?  E.g., SVG
>> > (via the svg.el library).
>> 
>> Because those those are not universally available, e.g. on the
>> console.
>
> On the console you currently cannot have images in Emacs, period.

Exactly.  And I want this to change.  There is no reason for such
restriction.

> This is not because of image libraries, this is because the text-mode
> display simply cannot currently support such display elements.

This is not an issue at all.  Capabilities of text-mode display are
irrelevant.  w3mimgdisplay is using framebuffer.  I do not know how to
better communicate that.

Try:

   $ w3m https://gnu.org

You will see text-mode HTML browser layout with images on the console.

>> >> Is there a way to turn off cursor for a buffer?
>> >
>> > There's the cursor-type variable.
>> 
>> Thanks.
>> 
>> This does not work on the console.
>
> (You never said you were talking about a console.)  We currently don't
> expose console cursor control to Lisp.  Should be easy to add that, if
> needed.

I see.

cursor-type does not say it does not work on the console.

I think not all values are doable and relevant.  t and nil should be
enough.

Shell I open a bug report for this?

>> Is there a way to hook a function when a buffer looses and gains
>> focus?
>
> Buffer cannot have focus.  Do you mean when the buffer's window is the
> selected window?  There's window-selection-change-functions.

Thanks, I'll look into this.

>> >> Is there a way run a function after emacs changes something on the
>> >> screen in a buffer?
>> >
>> > That depends by what you mean by "after emacs changes something on
>> > the screen", since in Emacs redisplay is independent/separate of
>> > data changes.  Please elaborate.
>> 
>> Example: I am on the console.  I am in dired buffer.  I press my key.
>> w3mimgdisplay program draws my image on the screen on the specified
>> coordinates "over" emacs.  All is well (except the blinking cursor
>> which we can ignore for now).  Now I press another key to display the
>> next image in the dired buffer.  Emacs redraws some characters, that
>> have changed.  This disrupts the image drawn earlier and I need to
>> detect, that I need to tell w3mimgdisplay to redraw the image.  I
>> need to hook my redraw image function after emacs is done with its
>> drawing.
>
> I don't think we have such a hook, no.  But if we had it, using it
> like you want would cause annoying flickering, because you'd
> constantly redraw the image at very high frequency.

I do not think it would cause flickering on the console.  The only thing
that normally "flickers" is the blinking cursor.  Emacs does not draw
anything if nothing changed.

>> Is there such hook?  If not, would it be difficult to add it?
>
> It won't be hard to add it, but I'm not sure we want to add such a
> hook, see above.

That is a shame.  If there was a hook to detect when Emacs has drawn
something on the console, I could plug image redrawing there.

If I wanted to add it, where in the code should I look?

Thanks for your help!

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Akira Kyle]

On Tue, Dec 01, 2020 at 12:44 AM, Tomas Hlavaty <tom@logand.com> 
wrote:

> On Mon 30 Nov 2020 at 10:03, Akira Kyle <akira@akirakyle.com> 
> wrote:
>> and because I see pgtk as the future of Emacs on GNU/Linux 
>> systems as
>> X becomes increasingly obsolete.
>
> I am not sure such future is bright.
>
> In that future, is there no way to do graphics without a widget 
> toolkit?

Sure there is. Under wayland you can just dump pixels into a 
shared buffer to render them to the screen or use EGL to render 
using a GPU. The difference from X is that wayland offers no 
convince functions to actually draw pixels into such a buffer, its 
up to the client to do so. In order to not reinvent the wheel, its 
much more convenient to use a toolkit to draw the elements it's 
good at drawing. 

>> AFAIU the pgtk version, like the ns version it was modeled 
>> after, 
>> doesn't implement everything in the main frame area of Emacs as 
>> widgets, only the tool and menu bars are implemented as proper 
>> widgets.
>
> Also popup menus that pop up on a mouse click are gtk iirc.
>
> And maybe scrollbars?
>
> It may have changed since I turned these distractions off a long 
> time
> ago.

I think this is really ultimately a personal preference.

> I think that they are also fundamentally flawed: How can I 
> search in
> menu?  Maybe it should be a buffer.  New Gtk hides menu under an
> unintuitive button anyway so this could just display searchable 
> Menu
> buffer.

I think it would be possible to implement such menus without GTK 
using just child frames.

>> The main frame area is treated as just one big canvas that 
>> redisplay
>> works its magic on as it would if it were just a TUI.
>
> Maybe that is a good thing.
>
> Widget toolkit dependency is a heavy price to pay for.

I don't think it's a "heavy price to pay" in general. It greatly 
simplifies many applications, which if they were to forego a 
toolkit, would likely have codebases comparable to GTK or 
Emacs. But its true that Emacs is far more decoupled, for better 
or worse, from toolkits than your average GUI app.

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Akira Kyle]

On Tue, Dec 01, 2020 at 01:01 AM, Tomas Hlavaty <tom@logand.com> 
wrote:

> On Sun 29 Nov 2020 at 20:54, Akira Kyle <akira@akirakyle.com> 
> wrote:
>> However I think designing such a feature should keep eventual
>> integration with modules in mind. Especially since in general 
>> graphics
>> is very performance sensitive and so doing some drawing in 
>> elisp just
>> might not cut it and many interesting applications of such a 
>> feature
>> would require interfacing with an external c library.
>
> Not necessarily.
>
> There could also be a specialized program which could handle 
> drawing.
> In fact there already is such a program: w3mimgdisplay and is 
> part of
> the w3m web browser and pager.  It is also used in the file 
> manager
> ranger.  It works both with or without X.  I am using it in
> emacs-framebuffer too to display images and documents in Emacs.
>
> The issues I am facing have nothing to do with modules or 
> libraries.
>
> For example:
>
> It would be more convenient, if there was a way to specify elisp
> function to draw an image.  By default, it could just call the 
> existing
> C code but this would also allow me to specify a different elisp
> function which would then for example call w3mimgdisplay.
>
> It would be more convenient, if an image was represented as 
> elisp data
> instead of C data.  iirc there is no way to add new image types 
> without
> touching C.
>
> Is there a way to turn off cursor for a buffer?  (Blinking 
> cursor
> disrupts the drawn image.)
>
> Is there a way run a function after emacs changes something on 
> the
> screen in a buffer?

I just briefly poked around w3m's codebase and it appears that the 
code paths for X and direct framebuffer are almost entirely 
independent. Its not totally clear what your ultimate goal is with 
respect to w3m's image display and emacs. Do you wish to never 
launch display server such as X or wayland but still have some of 
Emacs graphics support on the linux framebuffer? It seems like 
unless you're going to teach Emacs to entirely draw itself to the 
framebuffer, there will always be glitches like the cursor messing 
up images since TUI Emacs won't be aware of what pixels of the 
framebuffer have been drawn over independently of the linux VT 
displaying lines of Emacs' characters.

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Tomas Hlavaty]

On Wed 02 Dec 2020 at 17:24, Akira Kyle <akira@akirakyle.com> wrote:
> On Tue, Dec 01, 2020 at 12:44 AM, Tomas Hlavaty <tom@logand.com> 
> wrote:
>
>> On Mon 30 Nov 2020 at 10:03, Akira Kyle <akira@akirakyle.com> 
>> wrote:
>>> and because I see pgtk as the future of Emacs on GNU/Linux systems
>>> as X becomes increasingly obsolete.
>>
>> I am not sure such future is bright.
>>
>> In that future, is there no way to do graphics without a widget 
>> toolkit?
>
> Sure there is. Under wayland you can just dump pixels into a 
> shared buffer to render them to the screen or use EGL to render 
> using a GPU. The difference from X is that wayland offers no 
> convince functions to actually draw pixels into such a buffer, its 
> up to the client to do so. In order to not reinvent the wheel, its 
> much more convenient to use a toolkit to draw the elements it's 
> good at drawing.

"just dump pixels" and "no function to actually do that" and "use
toolkit" does seem to contradict the point of being able to do it
without toolkit.

How long will pgtk future last?

Compare that kind of future with for example ansi terminal future.
Which one do you think has longer future ahead?  Why?

>>> AFAIU the pgtk version, like the ns version it was modeled after,
>>> doesn't implement everything in the main frame area of Emacs as
>>> widgets, only the tool and menu bars are implemented as proper
>>> widgets.
>>
>> Also popup menus that pop up on a mouse click are gtk iirc.
>>
>> And maybe scrollbars?
>>
>> It may have changed since I turned these distractions off a long 
>> time
>> ago.
>
> I think this is really ultimately a personal preference.

It was not my personal preference to implement popup menus and
scrollbars as proper widgets.

>> I think that they are also fundamentally flawed: How can I search in
>> menu?  Maybe it should be a buffer.  New Gtk hides menu under an
>> unintuitive button anyway so this could just display searchable Menu
>> buffer.
>
> I think it would be possible to implement such menus without GTK using
> just child frames.

Why not also implement the rest without GTK?

>>> The main frame area is treated as just one big canvas that redisplay
>>> works its magic on as it would if it were just a TUI.
>>
>> Maybe that is a good thing.
>>
>> Widget toolkit dependency is a heavy price to pay for.
>
> I don't think it's a "heavy price to pay" in general. It greatly
> simplifies many applications, which if they were to forego a toolkit,
> would likely have codebases comparable to GTK or Emacs. But its true
> that Emacs is far more decoupled, for better or worse, from toolkits
> than your average GUI app.

Emacs already has to forego a toolkit, doesn't it?  Why?  Because it has
to work portably without any particular toolkit or with all chosen
toolkits.

It does not simplify Emacs I think, on the contrary.

Would not it be better to extend the canvas idea you mentioned above to
allow graphics in general, portable, toolkit independent way?

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Tomas Hlavaty]

On Wed 02 Dec 2020 at 17:37, Akira Kyle <akira@akirakyle.com> wrote:
> Do you wish to never launch display server such as X or wayland but
> still have some of Emacs graphics support on the linux framebuffer?

yes

> It seems like unless you're going to teach Emacs to entirely draw
> itself to the framebuffer, there will always be glitches like the
> cursor messing up images since TUI Emacs won't be aware of what pixels
> of the framebuffer have been drawn over independently of the linux VT
> displaying lines of Emacs' characters.

The cursor could be turned off.

Are there other glitches which cannot be solved?

Try

   $ w3m https://gnu.org

You'll see graphics in text only web browser.  It works even with cursor
on.

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Eli Zaretskii]

> From: Tomas Hlavaty <tom@logand.com>
> Cc: emacs-devel@gnu.org
> Date: Wed, 02 Dec 2020 21:37:08 +0100

It turns out we've been mis-communicating all the time, since I didn't
realize you were interested only in displaying images on TTY frames.
So please forget (almost) everything I said, because most of it is not
relevant.  Below I answer only the questions that are relevant to your
goal.

> At the moment, in emacs-framebuffer, I just call it to display an image
> using call-process-region.  It could be optimized with pipe, but this
> has low priority for emacs-framebuffer as it is fast enough for my
> needs.

IMNSHO, neither call-process-region nor pipes are fast enough to be an
integral part of the Emacs redisplay: they are too slow.  Especially
if you want to be able to display more than one image at the same
time.  To see how the performance is inadequate, turn on flyspell-mode
and scroll through a large text buffer, preferably with lots of
mis-spelled words (or some text full of special terminology the
general-purpose dictionary won't know about).  The effect in your case
will be very similar.

> I am not exatly sure, what the concern is here.
> 
> The images do not travel via pipe.  Only meta-data like filenames,
> coordinates and handles.

Yes, I understood that much.  But that doesn't matter: communicating
with a spell-checker in the flyspell-mode case also sends just a small
number of words in each direction, and yet the slow-down is
considerable, sometimes downright annoying.

> What exactly needs to be done in the middle of redisplay that is
> relevant to drawing images?  Determining image width and height?

Sending the image spec and receiving the dimensions and other image
attributes, yes.  And I presume that the sub-process needs some time
to calculate the answer (because at least some queries require to load
and process an image).

> > You are supposed to submit data in one of the known formats.  Why is
> > that a problem?
> 
> Because they do not work on the console.
> 
> And the reason they do not work on the console is because there is
> hardcoded dependency on image drawing libraries which do not work on the
> console.

No, that's not the main reason.  See below.

> > On the console you currently cannot have images in Emacs, period.
> 
> Exactly.  And I want this to change.  There is no reason for such
> restriction.
> 
> > This is not because of image libraries, this is because the text-mode
> > display simply cannot currently support such display elements.
> 
> This is not an issue at all.  Capabilities of text-mode display are
> irrelevant.  w3mimgdisplay is using framebuffer.  I do not know how to
> better communicate that.

I was talking about the Emacs text-mode display code, sorry for being
unclear.  The Emacs display code which handles the console is not
capable of supporting images, or any display element whose dimensions
are not the same as character cells on a console.  The assumption that
each display element takes exactly one character cell on display is
implicitly present in many places in the Emacs display code that
handles TTYs.  We cannot even display a larger font on a console.
(There is a special handling of wide characters that take two columns
on display -- but still the same height as other characters -- but
that's all.)

To be able to display images on a console within an Emacs text-mode
frame, you need to be able to tell the Emacs display code that a
certain area on display is "taken".  But since images will generally
be larger than a character, in both dimensions, you need the Emacs
text-mode display code to support display elements of arbitrary
dimensions, and that is currently not possible.  Adding that would
need serious changes in the text-mode parts of the Emacs display
engine code, and that is almost entirely in C.

This is the main obstacle we'd need to overcome if Emacs will ever be
able to display images on TTYs.  All the rest is much easier.

> Try:
> 
>    $ w3m https://gnu.org
> 
> You will see text-mode HTML browser layout with images on the console.

I know that text-mode display devices can display graphics, I had it
30 years ago on an IBM PC clone.

> > (You never said you were talking about a console.)  We currently don't
> > expose console cursor control to Lisp.  Should be easy to add that, if
> > needed.
> 
> I see.
> 
> cursor-type does not say it does not work on the console.
> 
> I think not all values are doable and relevant.  t and nil should be
> enough.
> 
> Shell I open a bug report for this?

A feature request, yes.

> > I don't think we have such a hook, no.  But if we had it, using it
> > like you want would cause annoying flickering, because you'd
> > constantly redraw the image at very high frequency.
> 
> I do not think it would cause flickering on the console.  The only thing
> that normally "flickers" is the blinking cursor.  Emacs does not draw
> anything if nothing changed.

Emacs doesn't draw anything because it knows what's on the glass and
what _should_ be on the glass, and compares the two to decide what
should be redrawn.  But you want to draw on parts of the screen behind
Emacs's back, so it won't know whether something changed.  And you
OTOH have no way of knowing where Emacs have redrawn something during
the last redisplay cycle.  So you will have to redraw your images each
time Emacs finishes a redisplay cycle, regardless of whether it
changed anything on the glass in the area of an image.  These
redisplay cycles happen at very high frequency -- usually, Emacs
enters redisplay, quickly decides what needs to be redrawn, and then
does whatever has to be done and exits redisplay.  It tries very hard
not to redraw the parts that haven't changed, and that is why there's
no flickering.  But I don't see how you could prevent the flickering
of those add-on images, because without knowing which parts of the
screen changed, you will have to redraw them very frequently.

> >> Is there such hook?  If not, would it be difficult to add it?
> >
> > It won't be hard to add it, but I'm not sure we want to add such a
> > hook, see above.
> 
> That is a shame.  If there was a hook to detect when Emacs has drawn
> something on the console, I could plug image redrawing there.
> 
> If I wanted to add it, where in the code should I look?

In update_frame and its subroutines.  But my recommendation is not use
such a kludgey approach.

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Eli Zaretskii]

> From: Tomas Hlavaty <tom@logand.com>
> Date: Thu, 03 Dec 2020 09:15:16 +0100
> Cc: emacs-devel@gnu.org
> 
> > It seems like unless you're going to teach Emacs to entirely draw
> > itself to the framebuffer, there will always be glitches like the
> > cursor messing up images since TUI Emacs won't be aware of what pixels
> > of the framebuffer have been drawn over independently of the linux VT
> > displaying lines of Emacs' characters.
> 
> The cursor could be turned off.
> 
> Are there other glitches which cannot be solved?

Even turning the cursor off doesn't guarantee that Emacs will not
trigger overwriting of the images.  For example, sometimes Emacs moves
the cursor by writing newline characters, which could potentially
overwrite the image if the console implements that by writing spaces
or clearing each line to EOL.

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Arthur Miller]

Tomas Hlavaty <tom@logand.com> writes:

> On Wed 02 Dec 2020 at 17:24, Akira Kyle <akira@akirakyle.com> wrote:
>> On Tue, Dec 01, 2020 at 12:44 AM, Tomas Hlavaty <tom@logand.com> 
>> wrote:
>>
>>> On Mon 30 Nov 2020 at 10:03, Akira Kyle <akira@akirakyle.com> 
>>> wrote:
>>>> and because I see pgtk as the future of Emacs on GNU/Linux systems
>>>> as X becomes increasingly obsolete.
>>>
>>> I am not sure such future is bright.
>>>
>>> In that future, is there no way to do graphics without a widget 
>>> toolkit?
>>
>> Sure there is. Under wayland you can just dump pixels into a 
>> shared buffer to render them to the screen or use EGL to render 
>> using a GPU. The difference from X is that wayland offers no 
>> convince functions to actually draw pixels into such a buffer, its 
>> up to the client to do so. In order to not reinvent the wheel, its 
>> much more convenient to use a toolkit to draw the elements it's 
>> good at drawing.
>
> "just dump pixels" and "no function to actually do that" and "use
> toolkit" does seem to contradict the point of being able to do it
> without toolkit.
>
> How long will pgtk future last?

> Compare that kind of future with for example ansi terminal future.
> Which one do you think has longer future ahead?  Why?
I don't think you are askinkg for ANSI terminal; I think you are asking
for Emacs on framebuffer. 

I don't know how hard it is to create it, but it is probably on same
level as concole/terminal backend, X11 backend, Win32, Gtk, etc. I don't
know, I never used framebuffer so I can't really tell; just a sense.

>>>> AFAIU the pgtk version, like the ns version it was modeled after,
>>>> doesn't implement everything in the main frame area of Emacs as
>>>> widgets, only the tool and menu bars are implemented as proper
>>>> widgets.
>>>
>>> Also popup menus that pop up on a mouse click are gtk iirc.
>>>
>> I think it would be possible to implement such menus without GTK using
>> just child frames.
I agree with you on that one. I played with the idea; if you search the
archive you will find some poorman menu I sent in pga bug with focus
with child frames. Emacs already does halv of the jobb; one can display
a text buffer, enable current line highlight and set special text properties
and there is a simulated menu; sort of. 

> Would not it be better to extend the canvas idea you mentioned above to
> allow graphics in general, portable, toolkit independent way?
svg renderer is; as I understand. Maybe I am wrong, but as it seems
Emacs has got a built-in graphics via svg. I always thought Emacs went
through external application to display svg, but I learned recently it
does not.

Other path is to use Vulkan or OpenGL; I think OpenGL is better choice
for Emacs; you can render to a texture in memory, and handle XImage or
XPixmap to Emacs (hopefully - I haven't had time yet to go through and
test Eli's kind walkthrough yet). You could actually render entire emacs
buffers via egl to textures and display them via framebuffer. I don't
think it would be easy, but possible. Would require lots of C coding; if
you have time for that :-).

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Stefan Monnier]

BTW, for the framebuffer case, I think the pgtk code is a step in the
direction of making it possible for Emacs to open up a *graphical* frame
on a framebuffer using something like Cairo-fb.


        Stefan

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Tomas Hlavaty]

On Thu 03 Dec 2020 at 16:39, Eli Zaretskii <eliz@gnu.org> wrote:
>> At the moment, in emacs-framebuffer, I just call it to display an image
>> using call-process-region.  It could be optimized with pipe, but this
>> has low priority for emacs-framebuffer as it is fast enough for my
>> needs.
>
> IMNSHO, neither call-process-region nor pipes are fast enough to be an
> integral part of the Emacs redisplay: they are too slow.  Especially
> if you want to be able to display more than one image at the same
> time.  To see how the performance is inadequate, turn on flyspell-mode
> and scroll through a large text buffer, preferably with lots of
> mis-spelled words (or some text full of special terminology the
> general-purpose dictionary won't know about).  The effect in your case
> will be very similar.
>
>> I am not exatly sure, what the concern is here.
>> 
>> The images do not travel via pipe.  Only meta-data like filenames,
>> coordinates and handles.
>
> Yes, I understood that much.  But that doesn't matter: communicating
> with a spell-checker in the flyspell-mode case also sends just a small
> number of words in each direction, and yet the slow-down is
> considerable, sometimes downright annoying.

Is it an appropriate comparison?

I am sure it is possible to find many examples where Emacs slows down to
crawl even without external process, e.g. a buffer with long lines.
Does it mean I should dismiss Emacs for editing text?

I do not think my use-case is comparable to "scrolling through a large
text buffer, preferably with lots of mis-spelled words (or some text
full of special terminology the general-purpose dictionary won't know
about)".

I do not intent to scroll through a large buffer with lots images.  Even
if I did, there is no evidence that it would suffer the problems in the
above example.  w3m manages just fine.  My use case is something like
image-mode or doc-view or pdf-tools.  emacs-framebuffer already does
that for many image and document types and I have no problems with
speed.

In any case, ability to display images and documents on the console
beats no ability at all.

>> What exactly needs to be done in the middle of redisplay that is
>> relevant to drawing images?  Determining image width and height?
>
> Sending the image spec and receiving the dimensions and other image
> attributes, yes.  And I presume that the sub-process needs some time
> to calculate the answer (because at least some queries require to load
> and process an image).

This is a non-issue.  Images usually do not change those attributes
quickly, if at all.  So this can usually be cached or even calculated
just once.  This is what I do in emacs-framebuffer.

>> > You are supposed to submit data in one of the known formats.  Why is
>> > that a problem?
>> 
>> Because they do not work on the console.
>> 
>> And the reason they do not work on the console is because there is
>> hardcoded dependency on image drawing libraries which do not work on the
>> console.
>
> No, that's not the main reason.  See below.
>
>> > On the console you currently cannot have images in Emacs, period.
>> 
>> Exactly.  And I want this to change.  There is no reason for such
>> restriction.
>> 
>> > This is not because of image libraries, this is because the text-mode
>> > display simply cannot currently support such display elements.
>> 
>> This is not an issue at all.  Capabilities of text-mode display are
>> irrelevant.  w3mimgdisplay is using framebuffer.  I do not know how to
>> better communicate that.
>
> I was talking about the Emacs text-mode display code, sorry for being
> unclear.  The Emacs display code which handles the console is not
> capable of supporting images, or any display element whose dimensions
> are not the same as character cells on a console.  The assumption that
> each display element takes exactly one character cell on display is
> implicitly present in many places in the Emacs display code that
> handles TTYs.  We cannot even display a larger font on a console.
> (There is a special handling of wide characters that take two columns
> on display -- but still the same height as other characters -- but
> that's all.)
>
> To be able to display images on a console within an Emacs text-mode
> frame, you need to be able to tell the Emacs display code that a
> certain area on display is "taken".  But since images will generally
> be larger than a character, in both dimensions, you need the Emacs
> text-mode display code to support display elements of arbitrary
> dimensions, and that is currently not possible.  Adding that would
> need serious changes in the text-mode parts of the Emacs display
> engine code, and that is almost entirely in C.
>
> This is the main obstacle we'd need to overcome if Emacs will ever be
> able to display images on TTYs.  All the rest is much easier.

I understand that.

I do not aim to change that.

Again, my use-case is like that of image-mode, doc-view and pdf-tools.
There is a file, say /tmp/a.png, I open the file in a buffer and the
buffer will show the png image.  Or there is a file, say /tmp/a.pdf, I
open the file in a buffer and the buffer will show the pdf document.

This use-case does not need to deal with the issues outlined above.  It
just needs to know size and position of the buffer, width and height of
characters and screen etc.  emacs-framebuffer already does that and it
is not that difficult, thanks to the hints I already got from you:-)

>> > (You never said you were talking about a console.)  We currently don't
>> > expose console cursor control to Lisp.  Should be easy to add that, if
>> > needed.
>> 
>> I see.
>> 
>> cursor-type does not say it does not work on the console.
>> 
>> I think not all values are doable and relevant.  t and nil should be
>> enough.
>> 
>> Shell I open a bug report for this?
>
> A feature request, yes.

ok, thank you.

>> > I don't think we have such a hook, no.  But if we had it, using it
>> > like you want would cause annoying flickering, because you'd
>> > constantly redraw the image at very high frequency.
>> 
>> I do not think it would cause flickering on the console.  The only thing
>> that normally "flickers" is the blinking cursor.  Emacs does not draw
>> anything if nothing changed.
>
> Emacs doesn't draw anything because it knows what's on the glass and
> what _should_ be on the glass, and compares the two to decide what
> should be redrawn.  But you want to draw on parts of the screen behind
> Emacs's back, so it won't know whether something changed.  And you
> OTOH have no way of knowing where Emacs have redrawn something during
> the last redisplay cycle.  So you will have to redraw your images each
> time Emacs finishes a redisplay cycle, regardless of whether it
> changed anything on the glass in the area of an image.  These
> redisplay cycles happen at very high frequency -- usually, Emacs
> enters redisplay, quickly decides what needs to be redrawn, and then
> does whatever has to be done and exits redisplay.  It tries very hard
> not to redraw the parts that haven't changed, and that is why there's
> no flickering.  But I don't see how you could prevent the flickering
> of those add-on images, because without knowing which parts of the
> screen changed, you will have to redraw them very frequently.

This is not an issue.

You can try and see it easily.

Start emacs on the console.

Start shell in Emacs: M-x shell

Execute: $ cat /dev/urandom >/dev/fb0

Observe what Emacs draws.  I can see blinking cursor.  Part of the
*shell* buffer was drawn and line and column indicators on the
mode-line.  Everything else remains unchanged.

Now ssh from a different computer and run $ cat /dev/urandom >/dev/fb0

Observe what Emacs draws.  I can see blinking cursor.  Everything else
on the screen remains unchanged.  After some time, clock on the
mode-line is drawn.

So if Emacs does not change anything, it does not draw anything on the
console and I do not need to redraw the image to the framebuffer.

The only time I need to redraw the image to the framebuffer is when
emacs changed and drawn something.  But this is a single event, not a
periodic thing.  Because there is no periodic redraw, there is no
flickering.  Unfortunately, there is no hook to run after this single
event.

>> >> Is there such hook?  If not, would it be difficult to add it?
>> >
>> > It won't be hard to add it, but I'm not sure we want to add such a
>> > hook, see above.
>> 
>> That is a shame.  If there was a hook to detect when Emacs has drawn
>> something on the console, I could plug image redrawing there.
>> 
>> If I wanted to add it, where in the code should I look?
>
> In update_frame and its subroutines.  But my recommendation is not use
> such a kludgey approach.

Thanks, I'll have a look.

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Tomas Hlavaty]

On Thu 03 Dec 2020 at 17:13, Eli Zaretskii <eliz@gnu.org> wrote:
>> > It seems like unless you're going to teach Emacs to entirely draw
>> > itself to the framebuffer, there will always be glitches like the
>> > cursor messing up images since TUI Emacs won't be aware of what pixels
>> > of the framebuffer have been drawn over independently of the linux VT
>> > displaying lines of Emacs' characters.
>> 
>> The cursor could be turned off.
>> 
>> Are there other glitches which cannot be solved?
>
> Even turning the cursor off doesn't guarantee that Emacs will not
> trigger overwriting of the images.  For example, sometimes Emacs moves
> the cursor by writing newline characters, which could potentially
> overwrite the image if the console implements that by writing spaces
> or clearing each line to EOL.

But that example is a single event and does not cause glitch or
flickering.

Emacs usually does not move the cursor by writing newline characters
because it feels like it.  And it does not do that frequently.

If I do not do anything, Emacs will not do anything (except updating
clock in the mode-line perhaps, which is outside the buffer with the
image).

Now after that single event, the image should be redrawn but that's it.

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Eli Zaretskii]

> From: Tomas Hlavaty <tom@logand.com>
> Cc: emacs-devel@gnu.org
> Date: Thu, 03 Dec 2020 22:02:47 +0100
> 
> >> The images do not travel via pipe.  Only meta-data like filenames,
> >> coordinates and handles.
> >
> > Yes, I understood that much.  But that doesn't matter: communicating
> > with a spell-checker in the flyspell-mode case also sends just a small
> > number of words in each direction, and yet the slow-down is
> > considerable, sometimes downright annoying.
> 
> Is it an appropriate comparison?

Yes, IME.  It shows what happens when Emacs needs to communicate with
an external subprocess at a high frequency.  Which is what will happen
if your proposed design will be used as a general-purpose feature for
drawing images in Emacs.

> I do not intent to scroll through a large buffer with lots images.  Even
> if I did, there is no evidence that it would suffer the problems in the
> above example.  w3m manages just fine.  My use case is something like
> image-mode or doc-view or pdf-tools.  emacs-framebuffer already does
> that for many image and document types and I have no problems with
> speed.

So you are talking about making Emacs support only your use case?  Not
about a feature that allows any other legitimate use case in Emacs
which could involve images?

> > Sending the image spec and receiving the dimensions and other image
> > attributes, yes.  And I presume that the sub-process needs some time
> > to calculate the answer (because at least some queries require to load
> > and process an image).
> 
> This is a non-issue.  Images usually do not change those attributes
> quickly, if at all.  So this can usually be cached or even calculated
> just once.

That "once" is what I'm talking about.  Imagine going through a
directory of image files one by one.

> Again, my use-case is like that of image-mode, doc-view and pdf-tools.
> There is a file, say /tmp/a.png, I open the file in a buffer and the
> buffer will show the png image.  Or there is a file, say /tmp/a.pdf, I
> open the file in a buffer and the buffer will show the pdf document.

That's a very limited use case.  So much so that I doubt if it's even
justified.  After all, you can already display images on the console,
so why do you insist to show it inside an Emacs window, of all the
places.  People are already complaining about the limitations of
doc-view and similar modes.

> > Emacs doesn't draw anything because it knows what's on the glass and
> > what _should_ be on the glass, and compares the two to decide what
> > should be redrawn.  But you want to draw on parts of the screen behind
> > Emacs's back, so it won't know whether something changed.  And you
> > OTOH have no way of knowing where Emacs have redrawn something during
> > the last redisplay cycle.  So you will have to redraw your images each
> > time Emacs finishes a redisplay cycle, regardless of whether it
> > changed anything on the glass in the area of an image.  These
> > redisplay cycles happen at very high frequency -- usually, Emacs
> > enters redisplay, quickly decides what needs to be redrawn, and then
> > does whatever has to be done and exits redisplay.  It tries very hard
> > not to redraw the parts that haven't changed, and that is why there's
> > no flickering.  But I don't see how you could prevent the flickering
> > of those add-on images, because without knowing which parts of the
> > screen changed, you will have to redraw them very frequently.
> 
> This is not an issue.
> 
> You can try and see it easily.
> 
> Start emacs on the console.
> 
> Start shell in Emacs: M-x shell
> 
> Execute: $ cat /dev/urandom >/dev/fb0
> 
> Observe what Emacs draws.  I can see blinking cursor.  Part of the
> *shell* buffer was drawn and line and column indicators on the
> mode-line.  Everything else remains unchanged.

What you don't see is the frantic activity going on in the display
code behind the scenes.

Look, you can disregard what I'm saying and then later learn this
stuff the hard way.  It's okay, it will be a fine journey and very
educational, I can assure you.  I'm just trying to warn you about the
hidden rocks under the surface.  I will stop now.

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Eli Zaretskii]

> From: Tomas Hlavaty <tom@logand.com>
> Cc: akira@akirakyle.com, emacs-devel@gnu.org
> Date: Thu, 03 Dec 2020 22:15:08 +0100
> 
> > Even turning the cursor off doesn't guarantee that Emacs will not
> > trigger overwriting of the images.  For example, sometimes Emacs moves
> > the cursor by writing newline characters, which could potentially
> > overwrite the image if the console implements that by writing spaces
> > or clearing each line to EOL.
> 
> But that example is a single event and does not cause glitch or
> flickering.

Why do you assume it's a single event?  Whenever Emacs decides that
moving the cursor from some coordinates to other coordinates should be
done by writing newlines, it will always do that when it needs to.
That could be every redisplay cycle, depending on the situation.

> Emacs usually does not move the cursor by writing newline characters
> because it feels like it.  And it does not do that frequently.

If you build your software on these assumptions, I can assure you it
will break in some use cases.

> If I do not do anything, Emacs will not do anything

That is profoundly incorrect.  First, "I do not do anything" doesn't
mean Emacs doesn't: there are timers running in Emacs all the time,
even if it's "emacs -Q".  More importantly, what looks to you like
"Emacs doesn't do anything" is the result of redisplay cycles that do
quite a lot of processing, before they decide that nothing should be
redrawn on the glass, and if you assume that processing will never
include moving a cursor, you are building your code on shaky grounds.