unofficial mirror of emacs-devel@gnu.org 
 help / color / mirror / code / Atom feed
* Rethinking the design of xwidgets
@ 2020-10-12 20:58 Akira Kyle
  2020-10-12 22:18 ` joakim
                   ` (4 more replies)
  0 siblings, 5 replies; 210+ messages in thread
From: Akira Kyle @ 2020-10-12 20:58 UTC (permalink / raw)
  To: emacs-devel

Hi all,

If you don't care about the motivation and background for this 
problem skip the next two paragraphs.

One of my primary uses of emacs is as a scientific notebook 
utilizing org mode's support for literate programming to 
interweave prose, LaTeX, and code. I find this to be far more 
powerful and comfortable than using the Jupyter notebooks which 
have quickly become incredibly popular for such scientific 
computing. However the biggest pain point I have is due to Emacs' 
relatively poor capabilities for interfacing with objects other 
than text. Sure Emacs can display images ok, but I'm not aware of 
any way to get emacs to display more dynamic content. Furthermore 
I always want to be able to do more inside Emacs, and the one 
other application I spend as much time in as I spend in Emacs is a 
browser. Unfortunately I can't even really use eww for the 
simplest of my web browsing tasks such as looking up documentation 
as it usually fails to adequately render the variety of ways 
websites embed mathematics.

This has all led me to be interested in ways to enhance Emacs' 
display capabilities. The two projects I've come across that do 
this are the experimental xwidgets configuration and the Emacs 
Application Framework (EAF) [1]. Unfortunately EAF uses Qt to 
reparent a window into emacs and Python to do two way RPC with 
elisp. IMO it isn't a very emacsy way to enhance Emacs' display 
capabilites and is limited to just taking over a whole window 
rather than being embedded as part of a buffer (key to supporting 
dynamic content such as an interactive inline plot inside an org 
mode document). That leaves xwidgets which seems to be somewhat 
incomplete, unused, and neglected. I've been poking around the 
xwidgets code and I really like the concept, however the 
implementation seems to need some more thought and work.

I see two fundamental issues with the current xwidget code. The 
first is that in order to add a new type of widget, one must 
integrate it into the C portion of Emacs' codebase and expose 
relevant functionality to elisp. The difficulty of navigating how 
xwidgets are integrated into redisplay and how to expose the 
inherently event driven nature of GUI toolkits into elisp in a 
safe way is a big hurdle to hacking on xwidgets. In the last 
decade of this features existence no one has added any additional 
xwidgets besides the original webkit one (see make-xwidget). This 
despite the fact that xwidgets were originally designed to allow 
embedding of any gtk widgets such as buttons, sliders, image 
views, opengl contexts [2]. Recently support for webkit using the 
native cocoa framework was added for the ns toolkit demonstrating 
that this feature need not be tied to a specific GUI toolkit.  

The second issue is that most interesting applications of xwidgets 
will likely need to use some external C library. Historically, 
package maintainers have been hesitant to package emacs with 
xwidgets support due to the additional dependency on webkitgtk, 
which was often not kept current with security patches [3]. In the 
long run I don't think adding extra dependencies to emacs just for 
the sake of some specific xwidget is worth it.

That leads me to my following proposal to rethink the design of 
xwidgets. Given that Emacs now has support for dynamic modules and 
it is now enabled by default, I propose exposing an interface for 
dynamic modules to define custom xwidgets. I believe this would be 
a good way to solve the two fundamental issues with the current 
xwidgets implementation that I described above. Having a well 
defined interface means one wouldn't have to deal with debugging 
subtle redisplay and lisp interpreter issues when creating a 
custom xwidget type. Furthermore any external library needed for a 
custom xwidget type will only be required if one chooses to use 
that xwidget type and would keep such dependencies out of Emacs 
itself.

I've already done a proof of concept prototype to convince myself 
that this is possible. I first modified the xwidget display spec 
to include a field which is expected to contain a function to 
initialize an xwidget view. When redisplay acts on this spec, the 
xwidgets code calls this function with the xwidget display spec 
and expects a user_ptr Lisp_Object to be returned which wraps a 
pointer to a GtkWidget. The xwidgets code then adds this widget to 
an appropriate container that it uses to ensure the widget is 
moved when the position or clipping of its glyph changes. I then 
define this xwidget view initialization function in a dynamic 
module.

Since every window containing the xwidget has its own view, and 
hence separate GtkWidget instance, it is then up to the dynamic 
module to define the how to handle the updating of each view based 
on some common model each is supposed to represent. This works 
around the limitation that in most GUI toolkits each widget can 
only have one view. Thus for something like the webkitgtk widget, 
one implementation might choose to do offscreen rendering (as is 
currently done) so that all windows displaying the widget are 
updated together when, say, one of them is scrolled. While another 
implementation might decide each view gets a separate view of the 
same webpage and scrolling one will not scroll the other (which is 
what the cocoa webkit implementation should probably do, but it is 
currently only limited to being displayed in a single window).

I'd be interested in any thoughts anyone might have about this 
proposal and if some changes along this line might have some 
chance of being accepted in master someday. I've always wanted to 
dig into the nity grity c code of emacs and this has finally given 
me the excuse to do so! 

Akira

[1] https://github.com/manateelazycat/emacs-application-framework
[2] https://github.com/jave/xwidget-emacs
[3] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=843462



^ permalink raw reply	[flat|nested] 210+ messages in thread
* RE: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)
@ 2020-12-02  2:58 arthur miller
  0 siblings, 0 replies; 210+ messages in thread
From: arthur miller @ 2020-12-02  2:58 UTC (permalink / raw)
  To: Tomas Hlavaty, emacs-devel@gnu.org

[-- Attachment #1: Type: text/plain, Size: 2562 bytes --]

"> Tomas Hlavaty <tom@logand.com> writes:
>> It would be more convenient, if an image was represented as elisp data
>> instead of C data.
> :-) I don't think you have thought well about it;

Why would you say that?"

Because of the other things you wrote before that citate about drawing images in external process. You have expressed your sentiments about year 2020, efficiency and external processes a day or two ago, and I read similar reasoning here.

If all you wish is just to display an image, sure you can do that with external application, but if you would like to perform something interesting with images, or display something many images, the performance would drop quite fast.

Since you refer to drawing images and mention image data, I assumed you are talking about real image data, i.e. pixels, not handles to lisp objects. With other words I thought you would like to use Lisp to do something interesting as processing images in Emacs on pixel level. When people mention image data that is usually what they mean. But assumptions are always fault on the one that assumes, so I apologize for that.

Otherwise, you could surely easily integrate netpbm, image/openmagic or even libgd (via python or perl wrapper) into Emacs via pure lisp and processes. I don't see what's the problem if that is what you wish.

-------- Originalmeddelande --------
Från: Tomas Hlavaty <tom@logand.com>
Datum: 2020-12-01 17:49 (GMT+01:00)
Till: emacs-devel@gnu.org
Ämne: Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

On Tue 01 Dec 2020 at 16:36, Arthur Miller <arthur.miller@live.com> wrote:
> Tomas Hlavaty <tom@logand.com> writes:
>> It would be more convenient, if an image was represented as elisp data
>> instead of C data.
> :-) I don't think you have thought well about it;

Why would you say that?

> but really nothing forbids you to try to reprsent images as lisp.

What do you mean exactly?

> You can take any de-compressed image and read in raw pixels in as a byte
> buffer and just shuffle around numbers and see how it
> works for you.Image is nothing but a bunch of numbers + some tiny
> metadata on top of it. Take some of netpbm formats and you have "textual
> image" you can manipulate on per-pixel level with lisp.

Image in emacs is represented in C as "struct image".

Image type in emacs is represented in C as "struct image_type".

The issue is that there doesn't seem to be a way to extend those (and/or
define new) in pure elisp.


[-- Attachment #2: Type: text/html, Size: 3451 bytes --]

^ permalink raw reply	[flat|nested] 210+ messages in thread

end of thread, other threads:[~2020-12-04  8:00 UTC | newest]

Thread overview: 210+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2020-10-12 20:58 Rethinking the design of xwidgets Akira Kyle
2020-10-12 22:18 ` joakim
2020-10-13 16:07   ` Akira Kyle
2020-10-13 19:06     ` joakim
2020-10-14  0:33       ` Akira Kyle
2020-10-13 13:36 ` Stefan Monnier
2020-10-13 16:09   ` Akira Kyle
2020-10-13 14:16 ` Eli Zaretskii
2020-10-13 17:05   ` Akira Kyle
2020-10-13 17:24     ` Qiantan Hong
2020-10-13 18:29       ` Akira Kyle
2020-10-13 18:44         ` Qiantan Hong
2020-10-13 19:17           ` Eli Zaretskii
2020-10-13 20:52           ` Akira Kyle
2020-10-14 14:36             ` Eli Zaretskii
2020-10-14 17:01               ` Stefan Monnier
2020-10-15 19:55                 ` arthur miller
2020-10-13 18:45         ` Eli Zaretskii
2020-10-13 21:00           ` Akira Kyle
2020-10-14 14:44             ` Eli Zaretskii
2020-10-15  0:35               ` Akira Kyle
2020-10-15  2:07                 ` Stefan Monnier
2020-10-15  3:35                   ` arthur miller
2020-10-15 15:06                     ` Eli Zaretskii
2020-10-15 15:20                       ` Arthur Miller
2020-10-15 15:48                         ` Eli Zaretskii
2020-10-13 17:55     ` Eli Zaretskii
2020-10-13 18:42       ` Basil L. Contovounesios
2020-10-13 19:10         ` Eli Zaretskii
2020-10-13 20:37       ` Akira Kyle
2020-10-14 14:33         ` Eli Zaretskii
2020-10-14 15:04           ` Arthur Miller
2020-10-14 15:32             ` Eli Zaretskii
2020-10-15 13:20               ` Arthur Miller
2020-10-17  4:26                 ` Kai Ma
2020-10-17  4:42                   ` Qiantan Hong
2020-11-14  5:40                   ` Richard Stallman
2020-10-14 16:53             ` Stefan Monnier
2020-10-14 18:56               ` Aiko Kyle
2020-10-15 12:48                 ` Arthur Miller
2020-10-15 16:25                   ` Akira Kyle
2020-10-15 12:35               ` Arthur Miller
2020-10-16  4:02             ` Richard Stallman
2020-10-16 13:03               ` Arthur Miller
2020-10-16 18:38                 ` Dmitry Gutov
2020-10-17  4:19                   ` Richard Stallman
2020-10-17  4:21                 ` Richard Stallman
2020-10-17  6:30                   ` Arthur Miller
2020-10-17 13:35                     ` Stefan Monnier
2020-10-17 19:15                       ` Arthur Miller
2020-10-19  3:44                         ` Richard Stallman
2020-10-18  4:17                     ` Richard Stallman
2020-10-18  4:17                     ` Richard Stallman
2020-10-18  9:31                       ` Dmitry Gutov
2020-10-19  3:44                         ` Richard Stallman
2020-10-19 12:37                           ` Dmitry Gutov
2020-10-19 13:43                             ` Arthur Miller
2020-10-20  5:13                               ` Richard Stallman
2020-10-20  5:47                                 ` Arthur Miller
2020-10-20 12:58                                   ` Stefan Monnier
2020-10-20 13:40                                     ` Arthur Miller
2020-10-21  4:42                                       ` Richard Stallman
2020-10-21  4:46                                   ` Richard Stallman
2020-10-20  5:14                             ` Richard Stallman
2020-10-20  5:56                               ` Arthur Miller
2020-10-20 10:45                                 ` Dmitry Gutov
2020-10-20 13:49                                   ` Arthur Miller
2020-10-21  4:46                                 ` Richard Stallman
2020-10-19 13:34                           ` Arthur Miller
2020-10-19 14:04                           ` Stefan Monnier
2020-10-20  5:13                             ` Richard Stallman
2020-10-18 14:41                       ` Arthur Miller
2020-10-19  3:48                         ` Richard Stallman
2020-10-19 13:48                           ` Arthur Miller
2020-10-16 14:54               ` Dmitry Gutov
2020-10-14 18:07           ` Akira Kyle
2020-10-14 18:32             ` Eli Zaretskii
2020-10-14 19:10               ` Akira Kyle
2020-10-13 18:36     ` Tomas Hlavaty
2020-10-13 18:38       ` Tomas Hlavaty
2020-10-13 21:20       ` Aiko Kyle
2020-10-14  0:12         ` Corwin Brust
2020-10-14 19:16           ` Akira Kyle
2020-10-14  7:32         ` Tomas Hlavaty
2020-10-14 15:02           ` Eli Zaretskii
2020-10-14 16:35             ` Tomas Hlavaty
2020-10-14 19:22               ` Akira Kyle
2020-10-14 21:29                 ` Tomas Hlavaty
2020-10-16  4:02                 ` Richard Stallman
2020-10-16 13:09                   ` Arthur Miller
2020-10-14 19:24     ` Mingde (Matthew) Zeng
2020-10-14 21:46       ` Akira Kyle
2020-10-15  5:17         ` Mingde (Matthew) Zeng
2020-10-14  4:38 ` Richard Stallman
2020-10-14  6:36   ` Akira Kyle
2020-11-22  3:35 ` Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets) Akira Kyle
2020-11-22 12:01   ` Jean Louis
2020-11-22 12:04   ` Jean Louis
2020-11-22 12:18     ` tomas
2020-11-22 12:57       ` Jean Louis
2020-11-22 16:24         ` tomas
2020-11-22 17:18           ` Jean Louis
2020-11-22 17:48             ` tomas
2020-11-22 18:40             ` Akira Kyle
2020-11-22 19:58               ` Jean Louis
2020-11-22 20:29                 ` Akira Kyle
2020-11-22 20:38                   ` Jean Louis
2020-11-22 15:27     ` Arthur Miller
2020-11-22 12:50   ` Jean Louis
2020-11-22 18:33     ` Akira Kyle
2020-11-22 18:04   ` Eli Zaretskii
2020-11-22 18:46     ` Akira Kyle
2020-11-22 21:20       ` Vasilij Schneidermann
2020-11-22 22:04         ` Akira Kyle
2020-11-22 18:29   ` T.V Raman
2020-11-22 18:53     ` Akira Kyle
2020-11-22 19:35       ` T.V Raman
2020-11-22 20:26         ` Akira Kyle
2020-11-23  0:46           ` T.V Raman
2020-11-23  3:44             ` Akira Kyle
2020-11-23  3:01       ` T.V Raman
2020-11-23  3:47         ` Akira Kyle
2020-11-23 16:29           ` Jose A. Ortega Ruiz
2020-11-23  9:39   ` Lars Ingebrigtsen
2020-11-23 13:26     ` joakim
2020-11-24  6:21       ` Lars Ingebrigtsen
2020-11-23 14:59     ` Arthur Miller
2020-11-23 23:47       ` Alan Third
2020-11-24  3:58         ` Akira Kyle
2020-11-24  8:06         ` Arthur Miller
2020-11-24  5:51       ` Richard Stallman
2020-11-24  6:24         ` Lars Ingebrigtsen
2020-11-26  4:49           ` Richard Stallman
2020-11-26  5:25             ` Arthur Miller
2020-11-26  8:27               ` tomas
2020-11-26 14:55                 ` Arthur Miller
2020-11-26 19:11                 ` Tomas Hlavaty
2020-11-26 19:34                   ` arthur miller
2020-11-27  8:04                     ` Tomas Hlavaty
2020-11-27  8:44                       ` Arthur Miller
2020-11-27 13:35                     ` Richard Stallman
2020-11-27 19:22                       ` Arthur Miller
2020-11-27 20:01                         ` Eli Zaretskii
2020-11-27 21:22                           ` Arthur Miller
2020-11-27 20:07                         ` Stefan Monnier
2020-11-27 21:09                           ` Arthur Miller
2020-11-27 21:35                             ` Stefan Monnier
2020-11-27 23:23                               ` Arthur Miller
2020-11-28  0:28                                 ` Stefan Monnier
2020-11-28  8:48                                   ` Arthur Miller
2020-11-27 21:22                           ` Akira Kyle
2020-11-28  7:31                             ` Eli Zaretskii
2020-11-28  5:55                         ` Richard Stallman
2020-11-28  8:50                           ` Arthur Miller
2020-11-29  5:24               ` Richard Stallman
2020-11-24  7:50         ` Arthur Miller
2020-11-24  8:47           ` tomas
2020-11-24  9:04             ` Arthur Miller
2020-11-25  5:38               ` Richard Stallman
2020-11-25  8:46                 ` tomas
2020-11-25 15:26                 ` Arthur Miller
2020-11-25 16:15                   ` Eric S Fraga
2020-11-24 15:43             ` T.V Raman
2020-11-25  5:38             ` Richard Stallman
2020-11-25  8:54               ` tomas
2020-11-24  6:23       ` Lars Ingebrigtsen
2020-11-24  3:33     ` Akira Kyle
2020-11-24  6:27       ` Lars Ingebrigtsen
2020-11-25  1:36         ` Akira Kyle
2020-11-25 15:11           ` Eli Zaretskii
2020-11-27 20:56             ` Akira Kyle
2020-11-28  8:38               ` Eli Zaretskii
2020-11-28  8:57                 ` Arthur Miller
2020-11-28  9:19                   ` Eli Zaretskii
2020-11-29  0:22                     ` Arthur Miller
2020-11-29  1:29                 ` Akira Kyle
2020-11-29  8:22                   ` martin rudalics
2020-11-29 19:01                   ` Eli Zaretskii
2020-11-30  3:54                     ` Akira Kyle
2020-11-30 15:39                       ` Eli Zaretskii
2020-11-30 17:03                         ` Akira Kyle
2020-11-30 18:11                           ` Eli Zaretskii
2020-11-30 18:30                             ` Akira Kyle
2020-12-01  7:44                           ` Tomas Hlavaty
2020-12-03  0:24                             ` Akira Kyle
2020-12-03  8:07                               ` Tomas Hlavaty
2020-12-03 20:34                                 ` Arthur Miller
2020-12-03 20:53                                 ` Stefan Monnier
2020-12-01  8:01                       ` Tomas Hlavaty
2020-12-01 15:36                         ` Arthur Miller
2020-12-01 16:48                           ` Tomas Hlavaty
2020-12-01 15:58                         ` Eli Zaretskii
2020-12-01 17:33                           ` Tomas Hlavaty
2020-12-01 19:41                             ` Eli Zaretskii
2020-12-02 20:37                               ` Tomas Hlavaty
2020-12-03 14:39                                 ` Eli Zaretskii
2020-12-03 21:02                                   ` Tomas Hlavaty
2020-12-04  7:48                                     ` Eli Zaretskii
2020-12-03  0:37                         ` Akira Kyle
2020-12-03  8:15                           ` Tomas Hlavaty
2020-12-03 15:13                             ` Eli Zaretskii
2020-12-03 21:15                               ` Tomas Hlavaty
2020-12-04  8:00                                 ` Eli Zaretskii
2020-11-30  9:05                     ` martin rudalics
2020-11-30 16:10                       ` Eli Zaretskii
2020-11-23 22:12   ` Alexander Adolf
2020-11-24  1:30     ` T.V Raman
2020-11-24  8:13       ` Arthur Miller
2020-11-24  3:51     ` Akira Kyle
  -- strict thread matches above, loose matches on Subject: below --
2020-12-02  2:58 arthur miller

Code repositories for project(s) associated with this public inbox

	https://git.savannah.gnu.org/cgit/emacs.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).