Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Akira Kyle]

Awhile ago I started a thread on rethinking the design of 
xwidgets, there was a lot of good discussion about the future of 
Emacs' rendering capabilities. It personally motivated me to try 
to better understand how Emacs' redisplay works and just overall 
better understand the C core of Emacs. As part of that exercise, I 
decided to see if I could port the existing xwidgets code into a 
dynamic module. The only xwidget Emacs currently implements is a 
webkit widget, so this effort turned into creating a dynamic 
module for embedding webkit in Emacs. It also happens that I've 
been wishing that I could never leave Emacs and a general web 
browser was the last daily app that I begrudgingly used outside of 
Emacs. Anyways here's the result: 
https://github.com/akirakyle/emacs-webkit.git (sorry for the 
github hosting, I just have yet to make the jump to another 
platform -- I guess what they say about vendor lock-in is true).

I was able to implement all of the current features of the webkit 
xwidget plus some more. I think having this as a dynamic module 
makes it feel less hacked into Emacs (although still a bit hackey 
in some places). One interesting additional feature I was able to 
implement was support for TUI emacs. Essentially it opens a 
dedicated window for displaying webkit which can be controlled 
through the keybindings in the emacs buffer corresponding to that 
webkit view (of course one needs to have running a running 
graphical session pointed to in $DISPLAY). Perhaps when this code 
stabilizes, it might make sense to remove the experimental xwidget 
support from Emacs if this is deemed a worthy successor to such 
features.

I also wanted to make a few more general comments prompted by the 
previous thread about the future of Emacs redisplay and rendering 
given what I've learned in this exercise. I now think its fairly 
foolish to try to integrate too much of a toolkit library's 
widgets into Emacs. There will always be this friction between 
Emacs and the toolkit library wanting to handle the business of 
drawing to the glass. In a lot of ways I've come away from this 
very impressed with how Emacs manages to have native toolkit ports 
to gtk, ns, win, and TUI rendering given the differences in how 
each wants to handle drawing to the screen, but on the other hand 
there's a lot of complexity and code behind making this happen, 
which presents a large barrier to entry to people like me who may 
want to hack on the lower level drawing capabilities of 
Emacs. We've also seen hardware change a lot since Emacs display 
machinery was written. CPUs have gotten faster and now everyone 
has a GPU. I think there's a potential for Emacs to become both 
snappier and more extensible with respect to its rendering. I had 
initially argued that Emacs probably shouldn't deal with rendering 
on the level of opengl, however I think as I've understood better 
the way Emacs currently handles rendering everything, that's 
exactly the direction Emacs should move in, *if it can*.

So here is my new pie-in-the-sky proposal for what the future of 
Emacs rendering could look like. Given the success and hopefully 
soon-to-be in master native-comp branch, perhaps it may not be so 
crazy to think about moving more rendering to elisp. If a clean 
elisp interface to opengl were defined and Emacs were able to 
render itself using this optimized elisp opengl interface, the low 
level code for each platform could be minimized to just providing 
the opengl surface and exposing input events. In fact there may be 
ways in which elisp is well suited to the way modern GPU graphics 
are rendered using scene graphs. In fact gtk4 is moving its 
rendering to such an approach in order to take advantage of 
hardware graphics acceleration. I think a first step would likely 
be to focus on implementing an elisp interface to cairo as Emacs 
is already using more of cairo to render itself. If it seems like 
the performance is good and elisp can represent graphical 
operations well, than that might be a good indication that such a 
model could be successful path forward towards an Emacs that could 
bring its extensibility to its own basic rendering.

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Jean Louis]

Hello Akira,

That is great to see. I will try to compile that as dynamic module. If
I remember well the webkit on my system could never compile to really
work, something was odd. I am using Hyperbola GNU/Linux-libre system.

I will have one question for you later, first I wish to try your
version of webkit.

* Akira Kyle <akira@akirakyle.com> [2020-11-22 14:05]:
>  Anyways here's the result:
> https://github.com/akirakyle/emacs-webkit.git (sorry for the github
> hosting, I just have yet to make the jump to another platform -- I
> guess what they say about vendor lock-in is true).

Let me give you and others who in future read this email, a set of
reference to switch:

GNU Project
https://www.gnu.org

Savannah, the software forge for people committed to free software
https://savannah.gnu.org

Savannah on nongnu.org
https//savannah.nongnu.org

Codeberg.org (Germany)
https://codeberg.org

Sourcehut.org
https://sourcehut.org

Trisquel GNU/Linux-libre Git Repositories
https://devel.trisquel.info/groups/trisquel

Pagure
https://pagure.io/pagure

Fosshost
https://fosshost.org/

GitGud - Fast and Free Git Hosting
https://gitgud.io/users/sign_in

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Jean Louis]

I have failed shamelessly.

Can it be to some outdated stuff on my system?

git clone https://github.com/akirakyle/emacs-webkit.git
Cloning into 'emacs-webkit'...
remote: Enumerating objects: 160, done.        
remote: Counting objects: 100% (160/160), done.        
remote: Compressing objects: 100% (54/54), done.        
remote: Total 160 (delta 104), reused 160 (delta 104), pack-reused 0        
Receiving objects: 100% (160/160), 2.35 MiB | 219.00 KiB/s, done.
Resolving deltas: 100% (104/104), done.
~/Programming/git $ cd emacs-webkit/ 
~/Programming/git/emacs-webkit $ ls
LICENSE     default.nix                hints.css       script.js  webkit-ace.el      webkit.el
Makefile    emacs-module.h             hints.js        style.css  webkit-history.el  
README.org  evil-collection-webkit.el  screencast.gif  tests.el   webkit-module.c    
~/Programming/git/emacs-webkit $ make
cc -shared -std=c99 -Wall -Wextra -Wno-unused-parameter -O3 -fpic  `pkg-config --cflags gtk+-3.0 webkit2gtk-4.0 --libs webkit2gtk-4.0` -o webkit-module.so webkit-module.c
webkit-module.c: In function ‘copy_string_contents’:
webkit-module.c:47:13: warning: implicit declaration of function ‘malloc’ [-Wimplicit-function-declaration]
   *buffer = malloc ((size_t) buffer_size);
             ^~~~~~
webkit-module.c:47:13: warning: incompatible implicit declaration of built-in function ‘malloc’
webkit-module.c:47:13: note: include ‘<stdlib.h>’ or provide a declaration of ‘malloc’
webkit-module.c:57:7: warning: implicit declaration of function ‘free’ [-Wimplicit-function-declaration]
       free (*buffer);
       ^~~~
webkit-module.c:57:7: warning: incompatible implicit declaration of built-in function ‘free’
webkit-module.c:57:7: note: include ‘<stdlib.h>’ or provide a declaration of ‘free’
webkit-module.c: In function ‘webkit_get_title’:
webkit-module.c:134:46: warning: implicit declaration of function ‘strlen’ [-Wimplicit-function-declaration]
         return env->make_string (env, title, strlen (title));
                                              ^~~~~~
webkit-module.c:134:46: warning: incompatible implicit declaration of built-in function ‘strlen’
webkit-module.c:134:46: note: include ‘<string.h>’ or provide a declaration of ‘strlen’
webkit-module.c: In function ‘webkit_get_uri’:
webkit-module.c:145:26: warning: implicit declaration of function ‘webkit_uri_for_display’ [-Wimplicit-function-declaration]
       const gchar *uri = webkit_uri_for_display (webkit_web_view_get_uri
                          ^~~~~~~~~~~~~~~~~~~~~~
webkit-module.c:145:26: warning: initialization makes pointer from integer without a cast [-Wint-conversion]
webkit-module.c:147:42: warning: incompatible implicit declaration of built-in function ‘strlen’
       return env->make_string (env, uri, strlen (uri));
                                          ^~~~~~
webkit-module.c:147:42: note: include ‘<string.h>’ or provide a declaration of ‘strlen’
webkit-module.c: In function ‘webkit_load_uri’:
webkit-module.c:161:3: warning: incompatible implicit declaration of built-in function ‘free’
   free (uri);
   ^~~~
webkit-module.c:161:3: note: include ‘<stdlib.h>’ or provide a declaration of ‘free’
webkit-module.c: In function ‘webkit_search’:
webkit-module.c:221:3: warning: incompatible implicit declaration of built-in function ‘free’
   free (text);
   ^~~~
webkit-module.c:221:3: note: include ‘<stdlib.h>’ or provide a declaration of ‘free’
webkit-module.c: In function ‘send_to_lisp’:
webkit-module.c:336:41: warning: incompatible implicit declaration of built-in function ‘strlen’
       || rio_writen (c->fd, (void *)id, strlen (id)+1) < 0
                                         ^~~~~~
webkit-module.c:336:41: note: include ‘<string.h>’ or provide a declaration of ‘strlen’
webkit-module.c: In function ‘webkit_js_finished’:
webkit-module.c:360:3: error: unknown type name ‘JSCValue’
   JSCValue *value = webkit_javascript_result_get_js_value (js_result);
   ^~~~~~~~
webkit-module.c:360:21: warning: implicit declaration of function ‘webkit_javascript_result_get_js_value’ [-Wimplicit-function-declaration]
   JSCValue *value = webkit_javascript_result_get_js_value (js_result);
                     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
webkit-module.c:360:21: warning: initialization makes pointer from integer without a cast [-Wint-conversion]
webkit-module.c:361:17: warning: implicit declaration of function ‘jsc_value_to_json’ [-Wimplicit-function-declaration]
   gchar *json = jsc_value_to_json (value, 1);
                 ^~~~~~~~~~~~~~~~~
webkit-module.c:361:17: warning: initialization makes pointer from integer without a cast [-Wint-conversion]
webkit-module.c:362:3: error: unknown type name ‘JSCException’
   JSCException *exception =
   ^~~~~~~~~~~~
webkit-module.c:363:5: warning: implicit declaration of function ‘jsc_context_get_exception’ [-Wimplicit-function-declaration]
     jsc_context_get_exception (jsc_value_get_context (value));
     ^~~~~~~~~~~~~~~~~~~~~~~~~
webkit-module.c:363:32: warning: implicit declaration of function ‘jsc_value_get_context’ [-Wimplicit-function-declaration]
     jsc_context_get_exception (jsc_value_get_context (value));
                                ^~~~~~~~~~~~~~~~~~~~~
webkit-module.c:363:5: warning: initialization makes pointer from integer without a cast [-Wint-conversion]
     jsc_context_get_exception (jsc_value_get_context (value));
     ^~~~~~~~~~~~~~~~~~~~~~~~~
In file included from /usr/include/glib-2.0/glib.h:62:0,
                 from /usr/include/gtk-3.0/gdk/gdkconfig.h:13,
                 from /usr/include/gtk-3.0/gdk/gdk.h:30,
                 from /usr/include/gtk-3.0/gtk/gtk.h:30,
                 from webkit-module.c:3:
webkit-module.c:366:16: warning: implicit declaration of function ‘jsc_exception_get_message’ [-Wimplicit-function-declaration]
                jsc_exception_get_message (exception));
                ^
/usr/include/glib-2.0/glib/gmessages.h:336:32: note: in definition of macro ‘g_warning’
                                __VA_ARGS__)
                                ^~~~~~~~~~~
webkit-module.c:365:16: warning: format ‘%s’ expects argument of type ‘char *’, but argument 4 has type ‘int’ [-Wformat=]
     g_warning ("Error running javascript: %s",
                ^
/usr/include/glib-2.0/glib/gmessages.h:336:32: note: in definition of macro ‘g_warning’
                                __VA_ARGS__)
                                ^~~~~~~~~~~
webkit-module.c:373:3: warning: incompatible implicit declaration of built-in function ‘free’
   free (cb->id);
   ^~~~
webkit-module.c:373:3: note: include ‘<stdlib.h>’ or provide a declaration of ‘free’
webkit-module.c: In function ‘webkit_execute_js’:
webkit-module.c:390:26: warning: incompatible implicit declaration of built-in function ‘malloc’
           Callback *cb = malloc (sizeof (Callback));
                          ^~~~~~
webkit-module.c:390:26: note: include ‘<stdlib.h>’ or provide a declaration of ‘malloc’
webkit-module.c:403:3: warning: incompatible implicit declaration of built-in function ‘free’
   free (script);
   ^~~~
webkit-module.c:403:3: note: include ‘<stdlib.h>’ or provide a declaration of ‘free’
webkit-module.c: In function ‘webkit_add_user_style’:
webkit-module.c:430:3: warning: incompatible implicit declaration of built-in function ‘free’
   free (style);
   ^~~~
webkit-module.c:430:3: note: include ‘<stdlib.h>’ or provide a declaration of ‘free’
webkit-module.c: In function ‘webkit_add_user_script’:
webkit-module.c:472:3: warning: incompatible implicit declaration of built-in function ‘free’
   free (script);
   ^~~~
webkit-module.c:472:3: note: include ‘<stdlib.h>’ or provide a declaration of ‘free’
webkit-module.c: In function ‘webkit_script_message_cb’:
webkit-module.c:501:3: error: unknown type name ‘JSCValue’
   JSCValue *value = webkit_javascript_result_get_js_value (js_result);
   ^~~~~~~~
webkit-module.c:501:21: warning: initialization makes pointer from integer without a cast [-Wint-conversion]
   JSCValue *value = webkit_javascript_result_get_js_value (js_result);
                     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
webkit-module.c:502:17: warning: initialization makes pointer from integer without a cast [-Wint-conversion]
   gchar *json = jsc_value_to_json (value, 1);
                 ^~~~~~~~~~~~~~~~~
webkit-module.c:503:3: error: unknown type name ‘JSCException’
   JSCException *exception =
   ^~~~~~~~~~~~
webkit-module.c:504:5: warning: initialization makes pointer from integer without a cast [-Wint-conversion]
     jsc_context_get_exception (jsc_value_get_context (value));
     ^~~~~~~~~~~~~~~~~~~~~~~~~
In file included from /usr/include/glib-2.0/glib.h:62:0,
                 from /usr/include/gtk-3.0/gdk/gdkconfig.h:13,
                 from /usr/include/gtk-3.0/gdk/gdk.h:30,
                 from /usr/include/gtk-3.0/gtk/gtk.h:30,
                 from webkit-module.c:3:
webkit-module.c:506:16: warning: format ‘%s’ expects argument of type ‘char *’, but argument 4 has type ‘int’ [-Wformat=]
     g_warning ("Error in javascript message recieve: %s",
                ^
/usr/include/glib-2.0/glib/gmessages.h:336:32: note: in definition of macro ‘g_warning’
                                __VA_ARGS__)
                                ^~~~~~~~~~~
webkit-module.c: In function ‘webkit_register_script_message’:
webkit-module.c:537:3: warning: incompatible implicit declaration of built-in function ‘free’
   free (name);
   ^~~~
webkit-module.c:537:3: note: include ‘<stdlib.h>’ or provide a declaration of ‘free’
webkit-module.c: In function ‘webkit_unregister_script_message’:
webkit-module.c:559:3: warning: incompatible implicit declaration of built-in function ‘free’
   free (name);
   ^~~~
webkit-module.c:559:3: note: include ‘<stdlib.h>’ or provide a declaration of ‘free’
webkit-module.c: In function ‘webview_notify_uri’:
webkit-module.c:612:22: warning: initialization makes pointer from integer without a cast [-Wint-conversion]
   const gchar *uri = webkit_uri_for_display (webkit_web_view_get_uri (webview));
                      ^~~~~~~~~~~~~~~~~~~~~~
webkit-module.c: In function ‘client_free’:
webkit-module.c:902:3: warning: incompatible implicit declaration of built-in function ‘free’
   free(c);
   ^~~~
webkit-module.c:902:3: note: include ‘<stdlib.h>’ or provide a declaration of ‘free’
webkit-module.c: In function ‘webkit_new’:
webkit-module.c:920:13: warning: implicit declaration of function ‘calloc’ [-Wimplicit-function-declaration]
   if (!(c = calloc (1, sizeof (Client))))
             ^~~~~~
webkit-module.c:920:13: warning: incompatible implicit declaration of built-in function ‘calloc’
webkit-module.c:920:13: note: include ‘<stdlib.h>’ or provide a declaration of ‘calloc’
webkit-module.c:963:46: warning: incompatible implicit declaration of built-in function ‘strlen’
              env->make_string (env, err_msg, strlen (err_msg)));
                                              ^~~~~~
webkit-module.c:963:46: note: include ‘<string.h>’ or provide a declaration of ‘strlen’
make: *** [Makefile:10: webkit-module.so] Error 1
~/Programming/git/emacs-webkit $ 

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[tomas]

[-- Attachment #1: Type: text/plain, Size: 1870 bytes --]

On Sun, Nov 22, 2020 at 03:04:29PM +0300, Jean Louis wrote:
> I have failed shamelessly.
> 
> Can it be to some outdated stuff on my system?
> 
> git clone https://github.com/akirakyle/emacs-webkit.git
> Cloning into 'emacs-webkit'...
> remote: Enumerating objects: 160, done.        
> remote: Counting objects: 100% (160/160), done.        
> remote: Compressing objects: 100% (54/54), done.        
> remote: Total 160 (delta 104), reused 160 (delta 104), pack-reused 0        
> Receiving objects: 100% (160/160), 2.35 MiB | 219.00 KiB/s, done.
> Resolving deltas: 100% (104/104), done.
> ~/Programming/git $ cd emacs-webkit/ 
> ~/Programming/git/emacs-webkit $ ls
> LICENSE     default.nix                hints.css       script.js  webkit-ace.el      webkit.el
> Makefile    emacs-module.h             hints.js        style.css  webkit-history.el  
> README.org  evil-collection-webkit.el  screencast.gif  tests.el   webkit-module.c    
> ~/Programming/git/emacs-webkit $ make
> cc -shared -std=c99 -Wall -Wextra -Wno-unused-parameter -O3 -fpic  `pkg-config --cflags gtk+-3.0 webkit2gtk-4.0 --libs webkit2gtk-4.0` -o webkit-module.so webkit-module.c
> webkit-module.c: In function ‘copy_string_contents’:
> webkit-module.c:47:13: warning: implicit declaration of function ‘malloc’ [-Wimplicit-function-declaration]
>    *buffer = malloc ((size_t) buffer_size);
>              ^~~~~~

[...]

Uh, oh. You seem to be missing fundamental packages for compiling
things. "malloc", for example is declared in /usr/include/stdlib.h

Does this file exist on your machine? If not, you'll have to look
for it. If yes, you'll have to adapt your compile options for your
compiler to find it.

Unless there's a configure step (but I don't see traces of it in
your files list) which would be willing to do that for you.

Cheers
 - t

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Jean Louis]

* Akira Kyle <akira@akirakyle.com> [2020-11-22 14:05]:
> 
> Awhile ago I started a thread on rethinking the design of xwidgets, there
> was a lot of good discussion about the future of Emacs' rendering
> capabilities. It personally motivated me to try to better understand how
> Emacs' redisplay works and just overall better understand the C core of
> Emacs. As part of that exercise, I decided to see if I could port the
> existing xwidgets code into a dynamic module.

There are 2 types of questions for you, one group of questions if for
dynamic module emacs-webkit:

- is it possible to make sure from start of the session not to use
  Javascript?

- is it maybe possible at compile time to add an option
  --disable-javascript ?

- is there way during the session to surely disable javascript?

- it would be very nice if webkit dynamic module could bookmark
  hyperlink in such a way that it can be programmatically captured by
  using Emacs Lisp. For example by pressing B that users can capture
  both the TITLE and URL together and store it how they wish. You
  could provide standard function and one customizable function. That
  way users could store the TITLE and URL without reloading the page
  into database, or into Org file, or other system.

And some questions not related to the subject, rather to your skills
with dynamic modules:

Developers of emacs-libpq will soon contribute the dynamic module to
GNU ELPA that offers bindings to PostgreSQL and that enhances Emacs
greatly with possibilities. Related to that in GNU there is GDBM or
GNU database https://www.gnu.org/s/gdbm

I am just asking if you could think of programming the dynamic module
that offers to users Emacs interface to GDBM. This type of database is
much simpler and users most probably have it on their systems and it
can be used for business processes.

Reference to emacs-libpq @ Github
https://github.com/anse1/emacs-libpq

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Jean Louis]

* tomas@tuxteam.de <tomas@tuxteam.de> [2020-11-22 15:20]:
> On Sun, Nov 22, 2020 at 03:04:29PM +0300, Jean Louis wrote:
> > I have failed shamelessly.
> > 
> > Can it be to some outdated stuff on my system?
> > 
> > git clone https://github.com/akirakyle/emacs-webkit.git
> > Cloning into 'emacs-webkit'...
> > remote: Enumerating objects: 160, done.        
> > remote: Counting objects: 100% (160/160), done.        
> > remote: Compressing objects: 100% (54/54), done.        
> > remote: Total 160 (delta 104), reused 160 (delta 104), pack-reused 0        
> > Receiving objects: 100% (160/160), 2.35 MiB | 219.00 KiB/s, done.
> > Resolving deltas: 100% (104/104), done.
> > ~/Programming/git $ cd emacs-webkit/ 
> > ~/Programming/git/emacs-webkit $ ls
> > LICENSE     default.nix                hints.css       script.js  webkit-ace.el      webkit.el
> > Makefile    emacs-module.h             hints.js        style.css  webkit-history.el  
> > README.org  evil-collection-webkit.el  screencast.gif  tests.el   webkit-module.c    
> > ~/Programming/git/emacs-webkit $ make
> > cc -shared -std=c99 -Wall -Wextra -Wno-unused-parameter -O3 -fpic  `pkg-config --cflags gtk+-3.0 webkit2gtk-4.0 --libs webkit2gtk-4.0` -o webkit-module.so webkit-module.c
> > webkit-module.c: In function ‘copy_string_contents’:
> > webkit-module.c:47:13: warning: implicit declaration of function ‘malloc’ [-Wimplicit-function-declaration]
> >    *buffer = malloc ((size_t) buffer_size);
> >              ^~~~~~
> 
> [...]
> 
> Uh, oh. You seem to be missing fundamental packages for compiling
> things. "malloc", for example is declared in /usr/include/stdlib.h
> 
> Does this file exist on your machine? If not, you'll have to look
> for it. If yes, you'll have to adapt your compile options for your
> compiler to find it.

Yes, it does.

> Unless there's a configure step (but I don't see traces of it in
> your files list) which would be willing to do that for you.

There is no configure, I would hoping it will work out of the box as
other Emacs modules.

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Arthur Miller]

Jean Louis <bugs@gnu.support> writes:

> I have failed shamelessly.
:-) Happens to best sometimes.

> Can it be to some outdated stuff on my system?

It looks like you should just fix some include path and maybe you need
to see over your linker options;

> webkit-module.c:920:13: warning: incompatible implicit declaration of built-in function ‘calloc’
> webkit-module.c:920:13: note: include ‘<stdlib.h>’ or provide a declaration of ‘calloc’
> webkit-module.c:963:46: warning: incompatible implicit declaration of built-in function ‘strlen’
>               env->make_string (env, err_msg, strlen (err_msg)));
>                                               ^~~~~~
> webkit-module.c:963:46: note: include ‘<string.h>’ or provide a declaration of ‘strlen’
> make: *** [Makefile:10: webkit-module.so] Error 1
The compiler already tells you to include <cstdlib.h> (which is c++
version of stdlib.h) + some other stuff. malloc, free & co are found
in stdlib.h and strlen is foound in string.h.

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[tomas]

[-- Attachment #1: Type: text/plain, Size: 2435 bytes --]

On Sun, Nov 22, 2020 at 03:57:22PM +0300, Jean Louis wrote:
> * tomas@tuxteam.de <tomas@tuxteam.de> [2020-11-22 15:20]:
> > On Sun, Nov 22, 2020 at 03:04:29PM +0300, Jean Louis wrote:
> > > I have failed shamelessly.
> > > 
> > > Can it be to some outdated stuff on my system?
> > > 
> > > git clone https://github.com/akirakyle/emacs-webkit.git
> > > Cloning into 'emacs-webkit'...
> > > remote: Enumerating objects: 160, done.        
> > > remote: Counting objects: 100% (160/160), done.        
> > > remote: Compressing objects: 100% (54/54), done.        
> > > remote: Total 160 (delta 104), reused 160 (delta 104), pack-reused 0        
> > > Receiving objects: 100% (160/160), 2.35 MiB | 219.00 KiB/s, done.
> > > Resolving deltas: 100% (104/104), done.
> > > ~/Programming/git $ cd emacs-webkit/ 
> > > ~/Programming/git/emacs-webkit $ ls
> > > LICENSE     default.nix                hints.css       script.js  webkit-ace.el      webkit.el
> > > Makefile    emacs-module.h             hints.js        style.css  webkit-history.el  
> > > README.org  evil-collection-webkit.el  screencast.gif  tests.el   webkit-module.c    
> > > ~/Programming/git/emacs-webkit $ make
> > > cc -shared -std=c99 -Wall -Wextra -Wno-unused-parameter -O3 -fpic  `pkg-config --cflags gtk+-3.0 webkit2gtk-4.0 --libs webkit2gtk-4.0` -o webkit-module.so webkit-module.c
> > > webkit-module.c: In function ‘copy_string_contents’:
> > > webkit-module.c:47:13: warning: implicit declaration of function ‘malloc’ [-Wimplicit-function-declaration]
> > >    *buffer = malloc ((size_t) buffer_size);
> > >              ^~~~~~
> > 
> > [...]
> > 
> > Uh, oh. You seem to be missing fundamental packages for compiling
> > things. "malloc", for example is declared in /usr/include/stdlib.h
> > 
> > Does this file exist on your machine? If not, you'll have to look
> > for it. If yes, you'll have to adapt your compile options for your
> > compiler to find it.
> 
> Yes, it does.

except when it doesn't :)

The error, at least, suggests that webkit-module.c isn't including
stdlib.h, for some reason.

> > Unless there's a configure step (but I don't see traces of it in
> > your files list) which would be willing to do that for you.
> 
> There is no configure, I would hoping it will work out of the box as
> other Emacs modules.

Is there an #include <stdlib.h> in webkit-module.c?

Cheers
-- t

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Jean Louis]

* tomas@tuxteam.de <tomas@tuxteam.de> [2020-11-22 19:24]:
> > > Uh, oh. You seem to be missing fundamental packages for compiling
> > > things. "malloc", for example is declared in /usr/include/stdlib.h
> > > 
> > > Does this file exist on your machine? If not, you'll have to look
> > > for it. If yes, you'll have to adapt your compile options for your
> > > compiler to find it.
> > 
> > Yes, it does.
> 
> except when it doesn't :)

You know that all better, thank you. 

I do remember being able to compile webkit in Emacs by compiling Emacs
itself, but it did not work as expected and I removed it.

$ locate stdlib.h
/usr/include/stdlib.h

but webkit-module.c inside of emacs-webkit git directory does not have
stdlib.h, there is nowhere in other files included. And I included
also string.h that was missing.

$ head webkit-module.c 
#define _POSIX_SOURCE 1

#include <gtk/gtk.h>
#include <webkit2/webkit2.h>
#include <stdbool.h>
#include <signal.h>
#include <assert.h>
#include <errno.h>

I think it should be included there and I have included those missind
but then I got these other errors. Maybe my webkit is outdated for
this one. I cannot install new webkit on this system.

webkit-module.c: In function ‘webkit_get_uri’:
webkit-module.c:147:26: warning: implicit declaration of function ‘webkit_uri_for_display’ [-Wimplicit-function-declaration]
       const gchar *uri = webkit_uri_for_display (webkit_web_view_get_uri
                          ^~~~~~~~~~~~~~~~~~~~~~
webkit-module.c:147:26: warning: initialization makes pointer from integer without a cast [-Wint-conversion]
webkit-module.c: In function ‘webkit_js_finished’:
webkit-module.c:362:3: error: unknown type name ‘JSCValue’
   JSCValue *value = webkit_javascript_result_get_js_value (js_result);
   ^~~~~~~~
webkit-module.c:362:21: warning: implicit declaration of function ‘webkit_javascript_result_get_js_value’ [-Wimplicit-function-declaration]
   JSCValue *value = webkit_javascript_result_get_js_value (js_result);
                     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
webkit-module.c:362:21: warning: initialization makes pointer from integer without a cast [-Wint-conversion]
webkit-module.c:363:17: warning: implicit declaration of function ‘jsc_value_to_json’ [-Wimplicit-function-declaration]
   gchar *json = jsc_value_to_json (value, 1);
                 ^~~~~~~~~~~~~~~~~
webkit-module.c:363:17: warning: initialization makes pointer from integer without a cast [-Wint-conversion]
webkit-module.c:364:3: error: unknown type name ‘JSCException’
   JSCException *exception =
   ^~~~~~~~~~~~
webkit-module.c:365:5: warning: implicit declaration of function ‘jsc_context_get_exception’ [-Wimplicit-function-declaration]
     jsc_context_get_exception (jsc_value_get_context (value));
     ^~~~~~~~~~~~~~~~~~~~~~~~~
webkit-module.c:365:32: warning: implicit declaration of function ‘jsc_value_get_context’ [-Wimplicit-function-declaration]
     jsc_context_get_exception (jsc_value_get_context (value));
                                ^~~~~~~~~~~~~~~~~~~~~
webkit-module.c:365:5: warning: initialization makes pointer from integer without a cast [-Wint-conversion]
     jsc_context_get_exception (jsc_value_get_context (value));
     ^~~~~~~~~~~~~~~~~~~~~~~~~
In file included from /usr/include/glib-2.0/glib.h:62:0,
                 from /usr/include/gtk-3.0/gdk/gdkconfig.h:13,
                 from /usr/include/gtk-3.0/gdk/gdk.h:30,
                 from /usr/include/gtk-3.0/gtk/gtk.h:30,
                 from webkit-module.c:3:
webkit-module.c:368:16: warning: implicit declaration of function ‘jsc_exception_get_message’ [-Wimplicit-function-declaration]
                jsc_exception_get_message (exception));
                ^
/usr/include/glib-2.0/glib/gmessages.h:336:32: note: in definition of macro ‘g_warning’
                                __VA_ARGS__)
                                ^~~~~~~~~~~
webkit-module.c:367:16: warning: format ‘%s’ expects argument of type ‘char *’, but argument 4 has type ‘int’ [-Wformat=]
     g_warning ("Error running javascript: %s",
                ^
/usr/include/glib-2.0/glib/gmessages.h:336:32: note: in definition of macro ‘g_warning’
                                __VA_ARGS__)
                                ^~~~~~~~~~~
webkit-module.c: In function ‘webkit_script_message_cb’:
webkit-module.c:503:3: error: unknown type name ‘JSCValue’
   JSCValue *value = webkit_javascript_result_get_js_value (js_result);
   ^~~~~~~~
webkit-module.c:503:21: warning: initialization makes pointer from integer without a cast [-Wint-conversion]
   JSCValue *value = webkit_javascript_result_get_js_value (js_result);
                     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
webkit-module.c:504:17: warning: initialization makes pointer from integer without a cast [-Wint-conversion]
   gchar *json = jsc_value_to_json (value, 1);
                 ^~~~~~~~~~~~~~~~~
webkit-module.c:505:3: error: unknown type name ‘JSCException’
   JSCException *exception =
   ^~~~~~~~~~~~
webkit-module.c:506:5: warning: initialization makes pointer from integer without a cast [-Wint-conversion]
     jsc_context_get_exception (jsc_value_get_context (value));
     ^~~~~~~~~~~~~~~~~~~~~~~~~
In file included from /usr/include/glib-2.0/glib.h:62:0,
                 from /usr/include/gtk-3.0/gdk/gdkconfig.h:13,
                 from /usr/include/gtk-3.0/gdk/gdk.h:30,
                 from /usr/include/gtk-3.0/gtk/gtk.h:30,
                 from webkit-module.c:3:
webkit-module.c:508:16: warning: format ‘%s’ expects argument of type ‘char *’, but argument 4 has type ‘int’ [-Wformat=]
     g_warning ("Error in javascript message recieve: %s",
                ^
/usr/include/glib-2.0/glib/gmessages.h:336:32: note: in definition of macro ‘g_warning’
                                __VA_ARGS__)
                                ^~~~~~~~~~~
webkit-module.c: In function ‘webview_notify_uri’:
webkit-module.c:614:22: warning: initialization makes pointer from integer without a cast [-Wint-conversion]
   const gchar *uri = webkit_uri_for_display (webkit_web_view_get_uri (webview));
                      ^~~~~~~~~~~~~~~~~~~~~~
make: *** [Makefile:10: webkit-module.so] Error 1

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[tomas]

[-- Attachment #1: Type: text/plain, Size: 1770 bytes --]

On Sun, Nov 22, 2020 at 08:18:12PM +0300, Jean Louis wrote:
> * tomas@tuxteam.de <tomas@tuxteam.de> [2020-11-22 19:24]:
> > > > Uh, oh. You seem to be missing fundamental packages for compiling
> > > > things [...]

> > except when it doesn't :)
> 
> You know that all better, thank you. 

Not sure, really.

> I do remember being able to compile webkit in Emacs by compiling Emacs
> itself, but it did not work as expected and I removed it.
> 
> $ locate stdlib.h
> /usr/include/stdlib.h

That seems a good place for it, yes. The compiler will
find that there.

> but webkit-module.c inside of emacs-webkit git directory does not have
> stdlib.h, there is nowhere in other files included. And I included
> also string.h that was missing.
> 
> $ head webkit-module.c 
> #define _POSIX_SOURCE 1
> 
> #include <gtk/gtk.h>
> #include <webkit2/webkit2.h>
> #include <stdbool.h>
> #include <signal.h>
> #include <assert.h>
> #include <errno.h>
> 
> I think it should be included there and I have included those missind
> but then I got these other errors. Maybe my webkit is outdated for
> this one. I cannot install new webkit on this system.

Definitely. But then it's the module's author's turn to chime in.
Obviously it works for him/her, so we must be missing something
important.

> webkit-module.c: In function ‘webkit_get_uri’:
> webkit-module.c:147:26: warning: implicit declaration of function ‘webkit_uri_for_display’ [-Wimplicit-function-declaration]
>        const gchar *uri = webkit_uri_for_display (webkit_web_view_get_uri

Yep. Perhaps your hunch is correct, and you have a different version
of webkit2.h than the author has. Perhaps the author's even includes
<stdlib.h> and friends?

Cheers
-- t

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Eli Zaretskii]

> From: Akira Kyle <akira@akirakyle.com>
> Date: Sat, 21 Nov 2020 20:35:52 -0700
> 
> Awhile ago I started a thread on rethinking the design of 
> xwidgets, there was a lot of good discussion about the future of 
> Emacs' rendering capabilities. It personally motivated me to try 
> to better understand how Emacs' redisplay works and just overall 
> better understand the C core of Emacs. As part of that exercise, I 
> decided to see if I could port the existing xwidgets code into a 
> dynamic module. The only xwidget Emacs currently implements is a 
> webkit widget, so this effort turned into creating a dynamic 
> module for embedding webkit in Emacs. It also happens that I've 
> been wishing that I could never leave Emacs and a general web 
> browser was the last daily app that I begrudgingly used outside of 
> Emacs. Anyways here's the result: 
> https://github.com/akirakyle/emacs-webkit.git

Thanks.

So, IIUC, basically you create a buffer, display it in a window, and
then let GTK draw in the visible portion of that window, is that the
idea?  If so, what happens if some other Lisp inserts some text into
that buffer? does the text overwrite the WebKit display?

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[T.V Raman]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=gb18030, Size: 236 bytes --]

It  would still be even nicer to see a lisp wrapper over webkit a la
nyxt --- then you could actually manipulate higher-level objects, rather
than bits rendered by webkit.
-- 

Thanks,

--Raman
7©4 Id: kg:/m/0285kf1  •0Ü8

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Akira Kyle]

On Sun, Nov 22, 2020 at 05:50 AM, Jean Louis <bugs@gnu.support> 
wrote:

> There are 2 types of questions for you, one group of questions 
> if for
> dynamic module emacs-webkit:
>
> - is it possible to make sure from start of the session not to 
> use
>   Javascript?

Yes,

(add-hook 'webkit-new-hook #'webkit-enable-javascript)

This should disable javascript on all 
sites. `webkit-enable-javascript` can be called interactively to 
disable javascript or enable if given a prefix arg. Although I 
just realized theres a bug here and the page load request will 
happen before the hook is applied. I'll fix this soon.

There's a thread on my post on reddit.com/r/emacs [1] where I talk 
a bit more about the process of disabling JS. To quote my comment 
there:

>> I should note that this sets the webkit 
>> `enable-javascript-markup` setting, not the 'enable-javascript' 
>> setting. I'm still figuring out how all these options are 
>> applied within webkit, but it seems like 
>> `enable-javascript-markup` disables a page from executing 
>> javascript by stripping it all out. `enable-javascript` would 
>> break the module since basic things like scrolling are handled 
>> by executing javascript in the view, which is allowed even if 
>> the page javascript is disabled with 
>> `enable-javascript-markup`. It seems like this is what email 
>> clients that use webkitgtk2 use since using javascript is 
>> really the only way to control the WebView given the WebKitDOM 
>> API is depricated. One could write the equavalent to 
>> window.scrollTo in C using the JavasScriptCore API but AFAIK 
>> that would be pretty much equivalent to what's currently 
>> happening.

> - is it maybe possible at compile time to add an option
>   --disable-javascript ?

I'm not sure if webkit has such an option to disable javascript 
and compile time. I think the JavaScriptCore must be present in 
order for webkit to even work.

> - is there way during the session to surely disable javascript?

See above.

> - it would be very nice if webkit dynamic module could bookmark
>   hyperlink in such a way that it can be programmatically 
>   captured by
>   using Emacs Lisp. For example by pressing B that users can 
>   capture
>   both the TITLE and URL together and store it how they 
>   wish. You
>   could provide standard function and one customizable 
>   function. That
>   way users could store the TITLE and URL without reloading the 
>   page
>   into database, or into Org file, or other system.

Currently C-l copies the url (open to suggestions on improvements 
to the default keybindings as I don't use them myself). And the 
title and url can be retrieved from the module with 
`webkit--get-uri` and `webkit--get-title`. I don't personally use 
emacs builtin bookmarks but enough people seem interested in such 
a feature that it'll happen.

> And some questions not related to the subject, rather to your 
> skills
> with dynamic modules:
>
> Developers of emacs-libpq will soon contribute the dynamic 
> module to
> GNU ELPA that offers bindings to PostgreSQL and that enhances 
> Emacs
> greatly with possibilities. Related to that in GNU there is GDBM 
> or
> GNU database https://www.gnu.org/s/gdbm
>
> I am just asking if you could think of programming the dynamic 
> module
> that offers to users Emacs interface to GDBM. This type of 
> database is
> much simpler and users most probably have it on their systems 
> and it
> can be used for business processes.
>
> Reference to emacs-libpq @ Github
> https://github.com/anse1/emacs-libpq

I'm not very knowledgeable about databases and don't directly 
interact them in my day-to-day so I don't have much of a personal 
motivation to work on such a problem thus I don't think I'm the 
right person for this project.

[1] 
https://www.reddit.com/r/emacs/comments/jyowe0/introducing_emacswebkit_a_successor_to/

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Akira Kyle]

What version of webkitgtk2 do you have? Can you make sure 
`pkg-config` is finding gtk+-3.0 and webkit2gtk-4.0? I think 
<stdlib.h> and co. should be included by gtk and webkit2gtk.

On Sun, Nov 22, 2020 at 10:18 AM, Jean Louis <bugs@gnu.support> 
wrote:

>> except when it doesn't :)
>
> You know that all better, thank you. 
>
> I do remember being able to compile webkit in Emacs by compiling 
> Emacs
> itself, but it did not work as expected and I removed it.
>
> $ locate stdlib.h
> /usr/include/stdlib.h
>
> but webkit-module.c inside of emacs-webkit git directory does 
> not have
> stdlib.h, there is nowhere in other files included. And I 
> included
> also string.h that was missing.
>
> $ head webkit-module.c 
> #define _POSIX_SOURCE 1
>
> #include <gtk/gtk.h>
> #include <webkit2/webkit2.h>
> #include <stdbool.h>
> #include <signal.h>
> #include <assert.h>
> #include <errno.h>
>
> I think it should be included there and I have included those 
> missind
> but then I got these other errors. Maybe my webkit is outdated 
> for
> this one. I cannot install new webkit on this system.
>
> webkit-module.c: In function ‘webkit_get_uri’:
> webkit-module.c:147:26: warning: implicit declaration of 
> function ‘webkit_uri_for_display’ 
> [-Wimplicit-function-declaration]
>        const gchar *uri = webkit_uri_for_display 
>        (webkit_web_view_get_uri
>                           ^~~~~~~~~~~~~~~~~~~~~~
> webkit-module.c:147:26: warning: initialization makes pointer 
> from integer without a cast [-Wint-conversion]
> webkit-module.c: In function ‘webkit_js_finished’:
> webkit-module.c:362:3: error: unknown type name ‘JSCValue’
>    JSCValue *value = webkit_javascript_result_get_js_value 
>    (js_result);
>    ^~~~~~~~
> webkit-module.c:362:21: warning: implicit declaration of 
> function ‘webkit_javascript_result_get_js_value’ 
> [-Wimplicit-function-declaration]
>    JSCValue *value = webkit_javascript_result_get_js_value 
>    (js_result);
>                      ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> webkit-module.c:362:21: warning: initialization makes pointer 
> from integer without a cast [-Wint-conversion]
> webkit-module.c:363:17: warning: implicit declaration of 
> function ‘jsc_value_to_json’ [-Wimplicit-function-declaration]
>    gchar *json = jsc_value_to_json (value, 1);
>                  ^~~~~~~~~~~~~~~~~
> webkit-module.c:363:17: warning: initialization makes pointer 
> from integer without a cast [-Wint-conversion]
> webkit-module.c:364:3: error: unknown type name ‘JSCException’
>    JSCException *exception =
>    ^~~~~~~~~~~~
> webkit-module.c:365:5: warning: implicit declaration of function 
> ‘jsc_context_get_exception’ [-Wimplicit-function-declaration]
>      jsc_context_get_exception (jsc_value_get_context (value));
>      ^~~~~~~~~~~~~~~~~~~~~~~~~
> webkit-module.c:365:32: warning: implicit declaration of 
> function ‘jsc_value_get_context’ 
> [-Wimplicit-function-declaration]
>      jsc_context_get_exception (jsc_value_get_context (value));
>                                 ^~~~~~~~~~~~~~~~~~~~~
> webkit-module.c:365:5: warning: initialization makes pointer 
> from integer without a cast [-Wint-conversion]
>      jsc_context_get_exception (jsc_value_get_context (value));
>      ^~~~~~~~~~~~~~~~~~~~~~~~~
> In file included from /usr/include/glib-2.0/glib.h:62:0,
>                  from /usr/include/gtk-3.0/gdk/gdkconfig.h:13,
>                  from /usr/include/gtk-3.0/gdk/gdk.h:30,
>                  from /usr/include/gtk-3.0/gtk/gtk.h:30,
>                  from webkit-module.c:3:
> webkit-module.c:368:16: warning: implicit declaration of 
> function ‘jsc_exception_get_message’ 
> [-Wimplicit-function-declaration]
>                 jsc_exception_get_message (exception));
>                 ^
> /usr/include/glib-2.0/glib/gmessages.h:336:32: note: in 
> definition of macro ‘g_warning’
>                                 __VA_ARGS__)
>                                 ^~~~~~~~~~~
> webkit-module.c:367:16: warning: format ‘%s’ expects argument of 
> type ‘char *’, but argument 4 has type ‘int’ [-Wformat=]
>      g_warning ("Error running javascript: %s",
>                 ^
> /usr/include/glib-2.0/glib/gmessages.h:336:32: note: in 
> definition of macro ‘g_warning’
>                                 __VA_ARGS__)
>                                 ^~~~~~~~~~~
> webkit-module.c: In function ‘webkit_script_message_cb’:
> webkit-module.c:503:3: error: unknown type name ‘JSCValue’
>    JSCValue *value = webkit_javascript_result_get_js_value 
>    (js_result);
>    ^~~~~~~~
> webkit-module.c:503:21: warning: initialization makes pointer 
> from integer without a cast [-Wint-conversion]
>    JSCValue *value = webkit_javascript_result_get_js_value 
>    (js_result);
>                      ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> webkit-module.c:504:17: warning: initialization makes pointer 
> from integer without a cast [-Wint-conversion]
>    gchar *json = jsc_value_to_json (value, 1);
>                  ^~~~~~~~~~~~~~~~~
> webkit-module.c:505:3: error: unknown type name ‘JSCException’
>    JSCException *exception =
>    ^~~~~~~~~~~~
> webkit-module.c:506:5: warning: initialization makes pointer 
> from integer without a cast [-Wint-conversion]
>      jsc_context_get_exception (jsc_value_get_context (value));
>      ^~~~~~~~~~~~~~~~~~~~~~~~~
> In file included from /usr/include/glib-2.0/glib.h:62:0,
>                  from /usr/include/gtk-3.0/gdk/gdkconfig.h:13,
>                  from /usr/include/gtk-3.0/gdk/gdk.h:30,
>                  from /usr/include/gtk-3.0/gtk/gtk.h:30,
>                  from webkit-module.c:3:
> webkit-module.c:508:16: warning: format ‘%s’ expects argument of 
> type ‘char *’, but argument 4 has type ‘int’ [-Wformat=]
>      g_warning ("Error in javascript message recieve: %s",
>                 ^
> /usr/include/glib-2.0/glib/gmessages.h:336:32: note: in 
> definition of macro ‘g_warning’
>                                 __VA_ARGS__)
>                                 ^~~~~~~~~~~
> webkit-module.c: In function ‘webview_notify_uri’:
> webkit-module.c:614:22: warning: initialization makes pointer 
> from integer without a cast [-Wint-conversion]
>    const gchar *uri = webkit_uri_for_display 
>    (webkit_web_view_get_uri (webview));
>                       ^~~~~~~~~~~~~~~~~~~~~~
> make: *** [Makefile:10: webkit-module.so] Error 1

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Akira Kyle]

On Sun, Nov 22, 2020 at 11:04 AM, Eli Zaretskii <eliz@gnu.org> 
wrote:

>> Awhile ago I started a thread on rethinking the design of 
>> xwidgets, there was a lot of good discussion about the future 
>> of 
>> Emacs' rendering capabilities. It personally motivated me to 
>> try 
>> to better understand how Emacs' redisplay works and just 
>> overall 
>> better understand the C core of Emacs. As part of that 
>> exercise, I 
>> decided to see if I could port the existing xwidgets code into 
>> a 
>> dynamic module. The only xwidget Emacs currently implements is 
>> a 
>> webkit widget, so this effort turned into creating a dynamic 
>> module for embedding webkit in Emacs. It also happens that I've 
>> been wishing that I could never leave Emacs and a general web 
>> browser was the last daily app that I begrudgingly used outside 
>> of 
>> Emacs. Anyways here's the result: 
>> https://github.com/akirakyle/emacs-webkit.git
>
> Thanks.
>
> So, IIUC, basically you create a buffer, display it in a window, 
> and
> then let GTK draw in the visible portion of that window, is that 
> the
> idea?  If so, what happens if some other Lisp inserts some text 
> into
> that buffer? does the text overwrite the WebKit display?

I find the GtkFixed widget that contains the windows of a frame, 
then I add the webkitgtk widget as a child of that GtkFixed. So 
everything under the widget is obscured, including any text 
written into the webkit-mode buffer. In fact I use the buffer for 
processing module->lisp async messages. I monitor for changes in 
the coordinates of the window with 
`window-size-change-functions`. Child frames are also added as 
children of the GtkFixed widget which presents a problem for 
things like posframe. I ensure that the webkit view is always 
below such child frames, as it would be if it were rendered as 
part of the buffer's window itself.

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Akira Kyle]

On Sun, Nov 22, 2020 at 11:29 AM, T.V Raman <raman@google.com> 
wrote:

> It  would still be even nicer to see a lisp wrapper over webkit 
> a la
> nyxt --- then you could actually manipulate higher-level 
> objects, rather
> than bits rendered by webkit.

Unless I'm misunderstanding you, that's pretty much what this 
is. I'm trying to not do too much in the dynamic module, just 
expose webkitgtk functions to lisp along with the ability to embed 
the webkit views in an Emacs window (or in a separate, dedicated 
gtk window if the Emacs window isn't gtk).

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[T.V Raman]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=gb18030, Size: 918 bytes --]

Akira Kyle <akira@akirakyle.com> writes:

apologies if I misunderstood what you were doing then:-)

What I really want is to be able to manipulate the DOM from the Emacs side, specifically with the goal of extracting higher-level structured information from the DOM, > On Sun, Nov 22, 2020 at 11:29 AM, T.V Raman <raman@google.com> wrote:
>
>> It  would still be even nicer to see a lisp wrapper over webkit a la
>> nyxt --- then you could actually manipulate higher-level objects,
>> rather
>> than bits rendered by webkit.
>
> Unless I'm misunderstanding you, that's pretty much what this is. I'm
> trying to not do too much in the dynamic module, just expose webkitgtk
> functions to lisp along with the ability to embed the webkit views in
> an Emacs window (or in a separate, dedicated gtk window if the Emacs
> window isn't gtk).
>

-- 

Thanks,

--Raman
7©4 Id: kg:/m/0285kf1  •0Ü8

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Jean Louis]

* Akira Kyle <akira@akirakyle.com> [2020-11-22 21:40]:
> 
> What version of webkitgtk2 do you have? Can you make sure `pkg-config` is
> finding gtk+-3.0 and webkit2gtk-4.0? I think <stdlib.h> and co. should be
> included by gtk and webkit2gtk.

pkg-config --cflags gtk+-3.0
-pthread -I/usr/include/gtk-3.0 -I/usr/include/at-spi2-atk/2.0 -I/usr/include/at-spi-2.0 -I/usr/include/dbus-1.0 -I/usr/lib/dbus-1.0/include -I/usr/include/gtk-3.0 -I/usr/include/gio-unix-2.0/ -I/usr/include/cairo -I/usr/include/pango-1.0 -I/usr/include/atk-1.0 -I/usr/include/cairo -I/usr/include/pixman-1 -I/usr/include/freetype2 -I/usr/include/libpng16 -I/usr/include/harfbuzz -I/usr/include/glib-2.0 -I/usr/lib/glib-2.0/include -I/usr/include/freetype2 -I/usr/include/harfbuzz -I/usr/include/libdrm -I/usr/include/libpng16 -I/usr/include/gdk-pixbuf-2.0 -I/usr/include/libpng16 -I/usr/include/glib-2.0 -I/usr/lib/glib-2.0/include

pkg-config --libs webkit2gtk-4.0
-lwebkit2gtk-4.0 -lgtk-3 -lgdk-3 -lpangocairo-1.0 -lpango-1.0 -latk-1.0 -lcairo-gobject -lcairo -lgdk_pixbuf-2.0 -lsoup-2.4 -lgio-2.0 -lgobject-2.0 -ljavascriptcoregtk-4.0 -lglib-2.0

I think it is finding.

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Akira Kyle]

On Sun, Nov 22, 2020 at 12:35 PM, T.V Raman <raman@google.com> 
wrote:

> What I really want is to be able to manipulate the DOM from the 
> Emacs side, specifically with the goal of extracting 
> higher-level structured information from the DOM, > On Sun, Nov 
> 22, 2020 at 11:29 AM, T.V Raman <raman@google.com> wrote:

Ah, there's currently the ability to manipulate the DOM by 
injecting JS and sending results back to Emacs as JSON. As far as 
representing and manipulating the DOM directly in elisp, I 
personally don't see that as being in scope for this project, 
however I'm not opposed if others want to explore that. Someone 
asked a similar question on the github repo: "Any plans to 
integrate parenscript or similar?" My response:

>> Not really. I personally think trying to work with parenscript 
>> for only running javascript just makes things needlessly 
>> complicated since you still need to know the whole DOM api to 
>> do anything interesting. Perhaps there's a compelling enough 
>> argument to make a parenscript equivalent in elisp, but for now 
>> I'm sticking with using JSON to communicated between JS and 
>> elisp.

I think nyxt is doing a good job with this and would refer people 
interested in a lispy DOM to their project: 
https://github.com/atlas-engineer/nyxt

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Akira Kyle]

On Sun, Nov 22, 2020 at 12:58 PM, Jean Louis <bugs@gnu.support> 
wrote:

>> What version of webkitgtk2 do you have? Can you make sure 
>> `pkg-config` is
>> finding gtk+-3.0 and webkit2gtk-4.0? I think <stdlib.h> and 
>> co. should be
>> included by gtk and webkit2gtk.
>
> pkg-config --cflags gtk+-3.0
> -pthread -I/usr/include/gtk-3.0 -I/usr/include/at-spi2-atk/2.0 
> -I/usr/include/at-spi-2.0 -I/usr/include/dbus-1.0 
> -I/usr/lib/dbus-1.0/include -I/usr/include/gtk-3.0 
> -I/usr/include/gio-unix-2.0/ -I/usr/include/cairo 
> -I/usr/include/pango-1.0 -I/usr/include/atk-1.0 
> -I/usr/include/cairo -I/usr/include/pixman-1 
> -I/usr/include/freetype2 -I/usr/include/libpng16 
> -I/usr/include/harfbuzz -I/usr/include/glib-2.0 
> -I/usr/lib/glib-2.0/include -I/usr/include/freetype2 
> -I/usr/include/harfbuzz -I/usr/include/libdrm 
> -I/usr/include/libpng16 -I/usr/include/gdk-pixbuf-2.0 
> -I/usr/include/libpng16 -I/usr/include/glib-2.0 
> -I/usr/lib/glib-2.0/include
>
> pkg-config --libs webkit2gtk-4.0
> -lwebkit2gtk-4.0 -lgtk-3 -lgdk-3 -lpangocairo-1.0 -lpango-1.0 
> -latk-1.0 -lcairo-gobject -lcairo -lgdk_pixbuf-2.0 -lsoup-2.4 
> -lgio-2.0 -lgobject-2.0 -ljavascriptcoregtk-4.0 -lglib-2.0
>
> I think it is finding.

And your webkitgtk version?

Others seem to be having success building this on their systems so 
perhaps see if you can build the minimal webkitgtk example here: 
https://wiki.gnome.org/Projects/WebKitGtk/ProgrammingGuide/Tutorial

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Jean Louis]

* Akira Kyle <akira@akirakyle.com> [2020-11-22 23:30]:
> 
> On Sun, Nov 22, 2020 at 12:58 PM, Jean Louis <bugs@gnu.support> wrote:
> 
> > > What version of webkitgtk2 do you have? Can you make sure
> > > `pkg-config` is
> > > finding gtk+-3.0 and webkit2gtk-4.0? I think <stdlib.h> and co.
> > > should be
> > > included by gtk and webkit2gtk.
> > 
> > pkg-config --cflags gtk+-3.0
> > -pthread -I/usr/include/gtk-3.0 -I/usr/include/at-spi2-atk/2.0
> > -I/usr/include/at-spi-2.0 -I/usr/include/dbus-1.0
> > -I/usr/lib/dbus-1.0/include -I/usr/include/gtk-3.0
> > -I/usr/include/gio-unix-2.0/ -I/usr/include/cairo
> > -I/usr/include/pango-1.0 -I/usr/include/atk-1.0 -I/usr/include/cairo
> > -I/usr/include/pixman-1 -I/usr/include/freetype2 -I/usr/include/libpng16
> > -I/usr/include/harfbuzz -I/usr/include/glib-2.0
> > -I/usr/lib/glib-2.0/include -I/usr/include/freetype2
> > -I/usr/include/harfbuzz -I/usr/include/libdrm -I/usr/include/libpng16
> > -I/usr/include/gdk-pixbuf-2.0 -I/usr/include/libpng16
> > -I/usr/include/glib-2.0 -I/usr/lib/glib-2.0/include
> > 
> > pkg-config --libs webkit2gtk-4.0
> > -lwebkit2gtk-4.0 -lgtk-3 -lgdk-3 -lpangocairo-1.0 -lpango-1.0 -latk-1.0
> > -lcairo-gobject -lcairo -lgdk_pixbuf-2.0 -lsoup-2.4 -lgio-2.0
> > -lgobject-2.0 -ljavascriptcoregtk-4.0 -lglib-2.0
> > 
> > I think it is finding.
> 
> And your webkitgtk version?

extra/webkit2gtk 2.16.1-3.hyperbola1 [installed]
    GTK+ Web content engine library, without geoclue2 support

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Vasilij Schneidermann]

[-- Attachment #1: Type: text/plain, Size: 852 bytes --]

> I find the GtkFixed widget that contains the windows of a frame, then I add
> the webkitgtk widget as a child of that GtkFixed. So everything under the
> widget is obscured, including any text written into the webkit-mode buffer.
> In fact I use the buffer for processing module->lisp async messages. I
> monitor for changes in the coordinates of the window with
> `window-size-change-functions`. Child frames are also added as children of
> the GtkFixed widget which presents a problem for things like posframe. I
> ensure that the webkit view is always below such child frames, as it would
> be if it were rendered as part of the buffer's window itself.

Could this approach be used to perform OpenGL drawing or obtaining a
cairo context to draw on? I'm not terribly interested in putting a
webview inside Emacs, 3D/2D graphics however...

Vasilij

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Akira Kyle]

On Sun, Nov 22, 2020 at 02:20 PM, Vasilij Schneidermann 
<mail@vasilij.de> wrote:

>> I find the GtkFixed widget that contains the windows of a 
>> frame, then I add
>> the webkitgtk widget as a child of that GtkFixed. So everything 
>> under the
>> widget is obscured, including any text written into the 
>> webkit-mode buffer.
>> In fact I use the buffer for processing module->lisp async 
>> messages. I
>> monitor for changes in the coordinates of the window with
>> `window-size-change-functions`. Child frames are also added as 
>> children of
>> the GtkFixed widget which presents a problem for things like 
>> posframe. I
>> ensure that the webkit view is always below such child frames, 
>> as it would
>> be if it were rendered as part of the buffer's window itself.
>
> Could this approach be used to perform OpenGL drawing or 
> obtaining a
> cairo context to draw on? I'm not terribly interested in putting 
> a
> webview inside Emacs, 3D/2D graphics however...
>
> Vasilij

Yes, the webkitgtk widget is ultimately just drawing itself on 
cario or opengl contexts. So if you only care about taking over 
the display of a window, this approach should work fine. I'm not 
sure it would be wise, or necessarily possible, to use this 
approach to mix elements drawn with GtkWidgets and Emacs' own 
display code.

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[T.V Raman]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=gb18030, Size: 2022 bytes --]

Akira Kyle <akira@akirakyle.com> writes:


Doing the manipulation with injected jSON would be fin e bye me.

I did look at parenscript in the context of nyxt,
and it looks interesting, but I dont have anything religious  re the
shape of delimiters I use to write code :-)

My interest stems primarily from wanting to speak Web content
intelligently and EWW gets me a lot of that, except where Web Content
is dynamically generated from JS and  given that, I'm looking to get
Webkit to build the DOM, and give it to me in a form that I can query
it, manipulate it, and finally speak the relevant bits.

> On Sun, Nov 22, 2020 at 12:35 PM, T.V Raman <raman@google.com> wrote:
>
>> What I really want is to be able to manipulate the DOM from the
>> Emacs side, specifically with the goal of extracting higher-level
>> structured information from the DOM, > On Sun, Nov 22, 2020 at 11:29
>> AM, T.V Raman <raman@google.com> wrote:
>
> Ah, there's currently the ability to manipulate the DOM by injecting
> JS and sending results back to Emacs as JSON. As far as representing
> and manipulating the DOM directly in elisp, I personally don't see
> that as being in scope for this project, however I'm not opposed if
> others want to explore that. Someone asked a similar question on the
> github repo: "Any plans to integrate parenscript or similar?" My
> response:
>
>>> Not really. I personally think trying to work with parenscript for
>>> only running javascript just makes things needlessly complicated
>>> since you still need to know the whole DOM api to do anything
>>> interesting. Perhaps there's a compelling enough argument to make a
>>> parenscript equivalent in elisp, but for now I'm sticking with
>>> using JSON to communicated between JS and elisp.
>
> I think nyxt is doing a good job with this and would refer people
> interested in a lispy DOM to their project:
> https://github.com/atlas-engineer/nyxt
>

-- 

Thanks,

--Raman
7©4 Id: kg:/m/0285kf1  •0Ü8

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[T.V Raman]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=gb18030, Size: 431 bytes --]

Got it built but it crashed on me.

Hi,

I git checked out and built emacs-webkit, built happily, no warnings or
errors. Read through readme.org, and it got me really excited.

Added the directory to load-path, and did a simple load-library of
webkit

Sadly, M-x webkit  called with either a search term, or a google.com
search URL immediately crashed emacs
-- 

Thanks,

--Raman
7©4 Id: kg:/m/0285kf1  •0Ü8

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Akira Kyle]

On Sun, Nov 22, 2020 at 05:46 PM, T.V Raman <raman@google.com> 
wrote:

> Doing the manipulation with injected jSON would be fin e bye me.
>
> I did look at parenscript in the context of nyxt,
> and it looks interesting, but I dont have anything religious  re 
> the
> shape of delimiters I use to write code :-)
>
> My interest stems primarily from wanting to speak Web content
> intelligently and EWW gets me a lot of that, except where Web 
> Content
> is dynamically generated from JS and  given that, I'm looking to 
> get
> Webkit to build the DOM, and give it to me in a form that I can 
> query
> it, manipulate it, and finally speak the relevant bits.

Here's a proof of concept of that. Don't try it on any large or 
complicated page as it'll grind Emacs to a halt trying to do the 
parsing. 

(setq webkit--to-json-js "
function toJSON(node) {
  let propFix = { for: 'htmlFor', class: 'className' };
  let specialGetters = {
    style: (node) => node.style.cssText,
  };
  let attrDefaultValues = { style: '' };
  let obj = {
    nodeType: node.nodeType,
  };
  if (node.tagName) {
    obj.tagName = node.tagName.toLowerCase();
  } else if (node.nodeName) {
    obj.nodeName = node.nodeName;
  }
  if (node.nodeValue) {
    obj.nodeValue = node.nodeValue;
  }
  let attrs = node.attributes;
  if (attrs) {
    let defaultValues = new Map();
    for (let i = 0; i < attrs.length; i++) {
      let name = attrs[i].nodeName;
      defaultValues.set(name, attrDefaultValues[name]);
    }
    // Add some special cases that might not be included by 
    enumerating
    // attributes above. Note: this list is probably not 
    exhaustive.
    switch (obj.tagName) {
      case 'input': {
        if (node.type === 'checkbox' || node.type === 'radio') {
          defaultValues.set('checked', false);
        } else if (node.type !== 'file') {
          // Don't store the value for a file input.
          defaultValues.set('value', '');
        }
        break;
      }
      case 'option': {
        defaultValues.set('selected', false);
        break;
      }
      case 'textarea': {
        defaultValues.set('value', '');
        break;
      }
    }
    let arr = [];
    for (let [name, defaultValue] of defaultValues) {
      let propName = propFix[name] || name;
      let specialGetter = specialGetters[propName];
      let value = specialGetter ? specialGetter(node) : 
      node[propName];
      if (value !== defaultValue) {
        arr.push([name, value]);
      }
    }
    if (arr.length) {
      obj.attributes = arr;
    }
  }
  let childNodes = node.childNodes;
  // Don't process children for a textarea since we used `value` 
  above.
  if (obj.tagName !== 'textarea' && childNodes && 
  childNodes.length) {
    let arr = (obj.childNodes = []);
    for (let i = 0; i < childNodes.length; i++) {
      arr[i] = toJSON(childNodes[i]);
    }
  }
  return obj;
}
toJSON(document);
")

(defun webkit--save-json (msg)
  (setq webkit--json (json-parse-string msg)))

(webkit--execute-js
 (with-current-buffer (car webkit--buffers) webkit--id)
 webkit--to-json-js "webkit--save-json")


I think through this general framework one could more efficiently 
extract the parts of the DOM one is interested in rather than 
dumping it all into elisp like I did above.

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Akira Kyle]

On Sun, Nov 22, 2020 at 08:01 PM, T.V Raman <raman@google.com> 
wrote:

> Got it built but it crashed on me.
>
> Hi,
>
> I git checked out and built emacs-webkit, built happily, no 
> warnings or
> errors. Read through readme.org, and it got me really excited.
>
> Added the directory to load-path, and did a simple load-library 
> of
> webkit
>
> Sadly, M-x webkit  called with either a search term, or a 
> google.com
> search URL immediately crashed emacs

Oh no! I assume this is a segfault? Can you try building with make 
debug and see if you get any more useful info? Perhaps a backtrace 
from gdb?

Someone posted a similar issue and it was due to them not running 
Emacs 28, but I just recently added a check to give an error 
message.

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Lars Ingebrigtsen]

Akira Kyle <akira@akirakyle.com> writes:

> Essentially it opens a dedicated window for displaying webkit which
> can be controlled through the keybindings in the emacs buffer
> corresponding to that webkit view (of course one needs to have running
> a running graphical session pointed to in $DISPLAY).

This is great news -- the demo looks awesome.

Just a couple of comments (without having actually tried it or even
looked at the code):

One thing I hoped to do with the in-tree xwidget code was to use it in
eww to display media types Emacs doesn't support -- primarily .mp4
<video> elements (and youtube videos, for those who doesn't find that
abhorrent).  I poked at it for a bit, but the current code is very much
tied to the xwidgets mode owning the buffer, and I haven't yet had the
stamina to fix that.

Is this easier with your package?  That is, the ability to plop in an
arbitrary number of widgets into any buffer?

The other thing I'm wondering about is the (pure) reliance of Cairo (if
I understood you correctly).  It seems like Cairo development is slowing
down as many projects have started using Skia instead.  Emacs supports a
number of graphics backends for a reason -- they pop up, last a few
years, and then they die off, and there's no reason to believe that
Cairo is going to be the One True Toolkit.  Toolkits come and go; Emacs
is forever.  :-)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[joakim]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Akira Kyle <akira@akirakyle.com> writes:
>
>> Essentially it opens a dedicated window for displaying webkit which
>> can be controlled through the keybindings in the emacs buffer
>> corresponding to that webkit view (of course one needs to have running
>> a running graphical session pointed to in $DISPLAY).
>
> This is great news -- the demo looks awesome.
>
> Just a couple of comments (without having actually tried it or even
> looked at the code):
>
> One thing I hoped to do with the in-tree xwidget code was to use it in
> eww to display media types Emacs doesn't support -- primarily .mp4
> <video> elements (and youtube videos, for those who doesn't find that
> abhorrent).  I poked at it for a bit, but the current code is very much
> tied to the xwidgets mode owning the buffer, and I haven't yet had the
> stamina to fix that.

The original design of xwidget were supposed to work like inserting
images in a buffer, so they werent originally supposed to own the buffer
or anything. It was possible to have several xwidgets in a buffer and so
on.

Maybe thats changed, I haven't been able to follow the development
recently. 

>
> Is this easier with your package?  That is, the ability to plop in an
> arbitrary number of widgets into any buffer?

>
> The other thing I'm wondering about is the (pure) reliance of Cairo (if
> I understood you correctly).  It seems like Cairo development is slowing
> down as many projects have started using Skia instead.  Emacs supports a
> number of graphics backends for a reason -- they pop up, last a few
> years, and then they die off, and there's no reason to believe that
> Cairo is going to be the One True Toolkit.  Toolkits come and go; Emacs
> is forever.  :-)
-- 
Joakim Verona
joakim@verona.se

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Arthur Miller]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Akira Kyle <akira@akirakyle.com> writes:
>
>> Essentially it opens a dedicated window for displaying webkit which
>> can be controlled through the keybindings in the emacs buffer
>> corresponding to that webkit view (of course one needs to have running
>> a running graphical session pointed to in $DISPLAY).
>
> This is great news -- the demo looks awesome.
>
> Just a couple of comments (without having actually tried it or even
> looked at the code):
>
> One thing I hoped to do with the in-tree xwidget code was to use it in
> eww to display media types Emacs doesn't support -- primarily .mp4
> <video> elements (and youtube videos, for those who doesn't find that
> abhorrent).
Just a curious question: have you ever looked at libmpv?
Would it be possible to embedd video via livmpv in an image container
just as jpg, png etc?

libmpv is a C library for mpv player ment to be embedded, and works in
all 3 major desktop platforms.

https://github.com/mpv-player/mpv-examples/tree/master/libmpv

Mpv itself comes with crap-load of video codecs and image formats already
built-in; it is a fork mplayer2. What is good, if one could create a
OpenGL context for an Emacs window (X11/Win32 Window - shouldn't be
hard), you could controll rendering directly, since they don't take over
your even loop:

"You will have to simulate this with the mouse/keypress/keydown/keyup commands."

It would probably fit into Emacs well.

I don't know myself how to implement "new image format" for Emacs; too
much to look through the spaghetty of if-defs in image.c, 

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Jose A. Ortega Ruiz]

[...]

> Someone posted a similar issue and it was due to them not running
> Emacs 28, but I just recently added a check to give an error message.

i have that problem in debian/sid, with an emacs compiled from today's
master branch (but not the pgtk one).  the error happens when loading
the .so file, and the error message says that a symbol dynamically
linked in it (gtk_init_<something>, i lost the trace, sorry) is not
found.  so it looks as if some ldconfig step or something of the sort
were missing, but i haven't had time to investigate it furthere.

cheers,
jao
--
A man has to live with himself, and he should see to it that he always has
good company. -Charles Evans Hughes, jurist (1862-1948)

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Alexander Adolf]

A very personal caution with respect to webkitgtk2 on macOS.

We are using webkitgtk in another OSS project, and building it in macOS
is a huge pain. You have to patch it (and devise new patches when a new
version is released), and compiling it takes hours (literally). When a
new webkitgtk version comes out, the pain starts all over again. This
has gotten to the point that currently there is no macOS build of that
OSS project.

And on top you have a webkit implementation readily available on the
macOS platform. Hence, we are actively seeking to either replace
webkitgtk with something that works cross-platform, or - in case there
isn't any such beast - to write a GTK-style wrapper for the macOS
webkit.

Thus, my very personal take would be that making webkitgtk a requirement
is probably not a promising approach if macOS is to remain on the list
of supported operating systems of this package (assuming that it would
end up being a package, of course).


Looking forward to your thoughts,

  --alexander

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Alan Third]

On Mon, Nov 23, 2020 at 03:59:31PM +0100, Arthur Miller wrote:
> Just a curious question: have you ever looked at libmpv?
> Would it be possible to embedd video via livmpv in an image container
> just as jpg, png etc?
> 
> libmpv is a C library for mpv player ment to be embedded, and works in
> all 3 major desktop platforms.
> 
> https://github.com/mpv-player/mpv-examples/tree/master/libmpv
> 
> Mpv itself comes with crap-load of video codecs and image formats already
> built-in; it is a fork mplayer2. What is good, if one could create a
> OpenGL context for an Emacs window (X11/Win32 Window - shouldn't be
> hard), you could controll rendering directly, since they don't take over
> your even loop:
> 
> "You will have to simulate this with the mouse/keypress/keydown/keyup commands."
> 
> It would probably fit into Emacs well.
> 
> I don't know myself how to implement "new image format" for Emacs; too
> much to look through the spaghetty of if-defs in image.c, 

IIRC it's actually possible to load videos using the imagemagick
backend, the trouble is that Emacs can't really handle them as it
decodes and stores each frame individually.

Emacs has no way to "remember" where it is in a file, as each image is
loaded and stored individually.

For example, to display an animated gif Emacs opens the file, decodes
frame one, closes the file and then displays the frame. When asked to
display frame two it opens the file, decodes frame two (which in the
case of a gif involves decoding frame one again) closes the file and
displays it. To display frame three it opens the file, decodes frame
three (which involves decoding frames one and two again), closes the
file and displays it. And so on and so on.

I think it's like this because it was only intended for displaying
occasional graphics like the fringe bitmaps and similar, and, IMHO, to
do anything different would require a major overhaul of image.c, and
probably the lisp image API while maintaining backwards compatibility.

-- 
Alan Third

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[T.V Raman]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=gb18030, Size: 1329 bytes --]

Alexander Adolf <alexander.adolf@condition-alpha.com> writes:

does blink have the same issues? I remember seeing something about qt
dependency re blink that I didn't quite follow.> A very personal caution with respect to webkitgtk2 on macOS.
>
> We are using webkitgtk in another OSS project, and building it in macOS
> is a huge pain. You have to patch it (and devise new patches when a new
> version is released), and compiling it takes hours (literally). When a
> new webkitgtk version comes out, the pain starts all over again. This
> has gotten to the point that currently there is no macOS build of that
> OSS project.
>
> And on top you have a webkit implementation readily available on the
> macOS platform. Hence, we are actively seeking to either replace
> webkitgtk with something that works cross-platform, or - in case there
> isn't any such beast - to write a GTK-style wrapper for the macOS
> webkit.
>
> Thus, my very personal take would be that making webkitgtk a requirement
> is probably not a promising approach if macOS is to remain on the list
> of supported operating systems of this package (assuming that it would
> end up being a package, of course).
>
>
> Looking forward to your thoughts,
>
>   --alexander
>

-- 

Thanks,

--Raman
7©4 Id: kg:/m/0285kf1  •0Ü8

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Akira Kyle]

On Mon, Nov 23, 2020 at 02:39 AM, Lars Ingebrigtsen 
<larsi@gnus.org> wrote:

> This is great news -- the demo looks awesome.

Thanks!

> Just a couple of comments (without having actually tried it or 
> even
> looked at the code):
>
> One thing I hoped to do with the in-tree xwidget code was to use 
> it in
> eww to display media types Emacs doesn't support -- primarily 
> .mp4
> <video> elements (and youtube videos, for those who doesn't find 
> that
> abhorrent).  I poked at it for a bit, but the current code is 
> very much
> tied to the xwidgets mode owning the buffer, and I haven't yet 
> had the
> stamina to fix that.
>
> Is this easier with your package?  That is, the ability to plop 
> in an
> arbitrary number of widgets into any buffer?

Possibly, but I'm not sure it would be wise to try. I originally 
started with the current xwidget code, which does support such 
capabilities. However, I found what I think are fundamental 
conceptual incompatibilities with the way Emacs' redisplay and 
gtk's event loop want to handle drawing stuff to the screen. Thus 
I feel like the only sane way to mix the two is to allow gtk to 
just take over an Emacs window which is always rectangular. No 
need to worry about scrolling and all the redisplay optimizations 
that will try to interfere with gtk attempting to draw at some 
point in the buffer. Even then Emacs and gtk don't play very 
nicely when it comes to grabbing keyboard and mouse input. Not to 
mention the fundamental differences in UI paradigms between Emacs 
being primarily textual and gtk with it's eye candy interfaces. Is 
it really so wise to try to mix the two?

Thus the second half of my previous email where I speculated about 
ways to move the fundamental ways Emacs does rendering forward. I 
think if Emacs had an open gl surface it was drawing to, it could 
handle rendering more intense graphical sources such as from a 
video. It will take a lot of work to get there, but I can't 
imagine another way that wont feel at least somewhat hacky to 
render something as graphically intensive like a video in a 
buffer.

> The other thing I'm wondering about is the (pure) reliance of 
> Cairo (if
> I understood you correctly).  It seems like Cairo development is 
> slowing
> down as many projects have started using Skia instead.  Emacs 
> supports a
> number of graphics backends for a reason -- they pop up, last a 
> few
> years, and then they die off, and there's no reason to believe 
> that
> Cairo is going to be the One True Toolkit.  Toolkits come and 
> go; Emacs
> is forever.  :-)

I think the death of Cairo has been greatly exaggerated :) The 
recent hacker news post about it brought up some good points about 
this expectation that projects have daily commits for it to be 
considered "alive". I don't see Cairo going anywhere for awhile, 
at least so long as projects like gtk and gnome rely on it. Also I 
wouldn't characterize Cairo as a "toolkit" at least in the sense 
that gtk is a toolkit or motif is a toolkit. Cairo is more like a 
nice graphics api, and the thing I've come to realize about nice 
api's is they tend to stick around in one form or another (kind of 
like emacs has).

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Akira Kyle]

On Mon, Nov 23, 2020 at 03:12 PM, Alexander Adolf 
<alexander.adolf@condition-alpha.com> wrote:

> A very personal caution with respect to webkitgtk2 on macOS.
>
> We are using webkitgtk in another OSS project, and building it 
> in macOS
> is a huge pain. You have to patch it (and devise new patches 
> when a new
> version is released), and compiling it takes hours 
> (literally). When a
> new webkitgtk version comes out, the pain starts all over 
> again. This
> has gotten to the point that currently there is no macOS build 
> of that
> OSS project.

Thankfully I haven't committed myself to ever needing to build 
webkitgtk with this project, it's just a dependency relegated to a 
package manager (unless you run GNU/Linux From Scratch).

> And on top you have a webkit implementation readily available on 
> the
> macOS platform. Hence, we are actively seeking to either replace
> webkitgtk with something that works cross-platform, or - in case 
> there
> isn't any such beast - to write a GTK-style wrapper for the 
> macOS
> webkit.
>
> Thus, my very personal take would be that making webkitgtk a 
> requirement
> is probably not a promising approach if macOS is to remain on 
> the list
> of supported operating systems of this package (assuming that it 
> would
> end up being a package, of course).
> Looking forward to your thoughts,

It's not even on the list of supported systems right now, so I 
think I'm safe. In fact part of my motivation for emacs-webkit was 
precisely because the recently merged xwidget webkit for ns 
actually seemed to work better than the webkitgtk one. Actually on 
x11 the webkitgtk xwidget is pretty much unusable because it 
constantly flickers, but so does emacs-webkit and I haven't been 
able to figure out why.

If you do write a GTK-style wrapper for macOS's ns webkit, that 
would certainly make it easier to run there, however the lisp 
portion of emacs-webkit is agnostic to the C module's 
implementation which mostly just wraps around the necessary 
functions of webkitgtk's api. Since it's webkit under the hood the 
ns webkit api is pretty much the same, just in objective-c (or 
swift). So it should just be a as straight forward as writing a 
different dynamic module for ns webkit, but which exports the same 
C functions as the current webkitgtk one. I don't run macOS 
anymore as they've increasingly made it hostile to customize and 
develop on outside of xcode, which can't hold a candle to emacs of 
course (not to mention its sooo not free). Maybe if someone sent 
me one of those shiny new apple silicon macbooks I'd make an 
emacs-webkit which worked on macOS, but I'm pretty fond of my 
current Pinebook Pro ;) 

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Akira Kyle]

On Mon, Nov 23, 2020 at 04:47 PM, Alan Third <alan@idiocy.org> 
wrote:

>> Just a curious question: have you ever looked at libmpv?
>> Would it be possible to embedd video via livmpv in an image 
>> container
>> just as jpg, png etc?
>> 
>> libmpv is a C library for mpv player ment to be embedded, and 
>> works in
>> all 3 major desktop platforms.
>> 
>> https://github.com/mpv-player/mpv-examples/tree/master/libmpv
>> 
>> Mpv itself comes with crap-load of video codecs and image 
>> formats already
>> built-in; it is a fork mplayer2. What is good, if one could 
>> create a
>> OpenGL context for an Emacs window (X11/Win32 Window - 
>> shouldn't be
>> hard), you could controll rendering directly, since they don't 
>> take over
>> your even loop:
>> 
>> "You will have to simulate this with the 
>> mouse/keypress/keydown/keyup commands."
>> 
>> It would probably fit into Emacs well.
>> 
>> I don't know myself how to implement "new image format" for 
>> Emacs; too
>> much to look through the spaghetty of if-defs in image.c, 
>
> IIRC it's actually possible to load videos using the imagemagick
> backend, the trouble is that Emacs can't really handle them as 
> it
> decodes and stores each frame individually.
>
> Emacs has no way to "remember" where it is in a file, as each 
> image is
> loaded and stored individually.
>
> For example, to display an animated gif Emacs opens the file, 
> decodes
> frame one, closes the file and then displays the frame. When 
> asked to
> display frame two it opens the file, decodes frame two (which in 
> the
> case of a gif involves decoding frame one again) closes the file 
> and
> displays it. To display frame three it opens the file, decodes 
> frame
> three (which involves decoding frames one and two again), closes 
> the
> file and displays it. And so on and so on.
>
> I think it's like this because it was only intended for 
> displaying
> occasional graphics like the fringe bitmaps and similar, and, 
> IMHO, to
> do anything different would require a major overhaul of image.c, 
> and
> probably the lisp image API while maintaining backwards 
> compatibility.

I second this and would also add that I think the emacs event loop 
would need modification as well, since my understanding is that 
it's pretty lazy, rendering things only when it needs to. So while 
its good that mpv wouldn't demand taking over the event loop, I 
think emacs might really struggle to display video frames at 30fps 
without some modifications or work around.

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Just a curious question: have you ever looked at libmpv?
  > Would it be possible to embedd video via livmpv in an image container
  > just as jpg, png etc?

Would people please present a description of what this plan aims to do?
There may be ethical questions that need to be considered, as well as
technical design questions.

For instance, I am concerned that it may lead to extending Emacs using
Javascript programs without verification of their licenses, perhaps
picking them up off the internet without checking them.

Or perhaps it doesn't do anything like that.  But I can't assume that,
since I just don't know.  Would you please fill me in?

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Lars Ingebrigtsen]

joakim@verona.se writes:

> The original design of xwidget were supposed to work like inserting
> images in a buffer, so they werent originally supposed to own the buffer
> or anything. It was possible to have several xwidgets in a buffer and so
> on.

I tried making this work in eww a few years back, but I gave up after
poking at it for a couple of hours.  It could totally be my fault for
using the wrong functions or something.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Lars Ingebrigtsen]

Arthur Miller <arthur.miller@live.com> writes:

> Just a curious question: have you ever looked at libmpv?
> Would it be possible to embedd video via livmpv in an image container
> just as jpg, png etc?

I think that would be difficult -- the Emacs image layer is very
persnickety.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Lars Ingebrigtsen]

Richard Stallman <rms@gnu.org> writes:

> Or perhaps it doesn't do anything like that.  But I can't assume that,
> since I just don't know.  Would you please fill me in?

libmpv is a library to play video files like .mp4.  There's no
Javascript involved.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Lars Ingebrigtsen]

Akira Kyle <akira@akirakyle.com> writes:

> Thus I feel like the only sane way to mix the two is to allow gtk to
> just take over an Emacs window which is always rectangular. No need to
> worry about scrolling and all the redisplay optimizations that will
> try to interfere with gtk attempting to draw at some point in the
> buffer.

I think this design reduces the use cases for emacs-webkit considerably
-- if all you can do with it is to let it take over a window completely,
then that's disappointing.  

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Arthur Miller]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]

> Would people please present a description of what this plan aims to
> do?
Yes of of course! Unfortunately; I am affraid there isn't much of a
plan.

> Or perhaps it doesn't do anything like that.  But I can't assume that,
> since I just don't know.  Would you please fill me in?

libmpv is just a library to play media files, has nothing to do with
javascript. You can check yourself:

 mpv is a free (as in freedom) media player for the command line. It
 supports a wide variety of media file formats, audio and video codecs,
 and subtitle types.

https://mpv.io/

Included codecs are Free too. I think most of codecs come from ffmpeg
which comes parts licensed in lgpl 2.1 and parts in gpl 2 license:
https://ffmpeg.org/legal.html

And list of included codecs:
https://ffmpeg.org/general.html#Supported-File-Formats_002c-Codecs-or-Features

MPV also comes in form of a shared .so with C interface, which (I think)
is well suited to interface with from Emacs. I hoped it could be
embedded into Emacs, similar to libjpeg or libpng to make Emacs display
video in a buffer in same way as it is displaying images. I was actually
looking at it myself like a two, three years or more ago, but I couldn't
figure out image.c. Now when Lars brought up shr.el and <video> tag, I
thought he can maybe bring it into Emacs for the use in shr.el, since he
is probably (much) more introduced into image.c if-defs and Emacs
internals in general then I will probably every be in soon future to
come. At least I think he is. I am using mpv as my video player normally
but I would prefer it to be embedded in Emacs framem. That was the
*master* plan :-). 

> For instance, I am concerned that it may lead to extending Emacs using
> Javascript programs without verification of their licenses, perhaps
> picking them up off the internet without checking them.
I am not sure, but I wouldn't think that playing sound and video in
Emacs would lead to more Javascript than other media formats like images
or html processing in general. By the way, how do you verify if JS is
free or not? Do you have an Emacs script that can download it and check
the license? That could be an useful part of Emacs maybe? Or maybe it
already is, I am horrible at looking what there is, I just hack what I
need when I need it, I am not so much for smörgåsbord in general.

> There may be ethical questions that need to be considered,
> as well as
> technical design questions.
Yes, I understand your concerns, but as above; I don't think there
should be more concerns about libmpv then what it is with libjpeg,
libpng, libtiff and similar. But I am not an expert in legal issues; am
actually very bad at those, so correct me if I am wrong.

Btw; I also own you an unanswered mail since few weeks ago; I am sorry,
I wanted to formulate more than just a hasty answer, but then I never
got back to it.

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Arthur Miller]

Alan Third <alan@idiocy.org> writes:

> On Mon, Nov 23, 2020 at 03:59:31PM +0100, Arthur Miller wrote:
>> Just a curious question: have you ever looked at libmpv?
>> Would it be possible to embedd video via livmpv in an image container
>> just as jpg, png etc?
>> 
>> libmpv is a C library for mpv player ment to be embedded, and works in
>> all 3 major desktop platforms.
>> 
>> https://github.com/mpv-player/mpv-examples/tree/master/libmpv
>> 
>> Mpv itself comes with crap-load of video codecs and image formats already
>> built-in; it is a fork mplayer2. What is good, if one could create a
>> OpenGL context for an Emacs window (X11/Win32 Window - shouldn't be
>> hard), you could controll rendering directly, since they don't take over
>> your even loop:
>> 
>> "You will have to simulate this with the mouse/keypress/keydown/keyup commands."
>> 
>> It would probably fit into Emacs well.
>> 
>> I don't know myself how to implement "new image format" for Emacs; too
>> much to look through the spaghetty of if-defs in image.c, 
>
> IIRC it's actually possible to load videos using the imagemagick
> backend, the trouble is that Emacs can't really handle them as it
> decodes and stores each frame individually.
>
> Emacs has no way to "remember" where it is in a file, as each image is
> loaded and stored individually.
>
> For example, to display an animated gif Emacs opens the file, decodes
> frame one, closes the file and then displays the frame. When asked to
> display frame two it opens the file, decodes frame two (which in the
> case of a gif involves decoding frame one again) closes the file and
> displays it. To display frame three it opens the file, decodes frame
> three (which involves decoding frames one and two again), closes the
> file and displays it. And so on and so on.
Ouch, that sounds like very inneficient way to do things (and very
Eamcs-y too :-)).

> I think it's like this because it was only intended for displaying
> occasional graphics like the fringe bitmaps and similar, and, IMHO, to
> do anything different would require a major overhaul of image.c, and
> probably the lisp image API while maintaining backwards compatibility.
Given a place to render, libmpv does it's own rendering, it requires
just a place to render, and it has it's own event loop so Emacs does not
need to do so much more then just provide a rectangle for it to render.
I thought it would be maybe possible to do same as with jpegs and pngs,
but I never figured out how they are placed in an Emacs frame myself :-).

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Arthur Miller]

>> new webkitgtk version comes out, the pain starts all over again.
Yeah Apple and Google are not best friends, Google diverged it's
own webkit version from Apple's they couldn't really agree with
development directions, so it is not surprising you are experiencing
problems you describe.

Btw: tried warp?

https://engineering.fb.com/2014/03/28/open-source/under-the-hood-warp-a-fast-c-and-c-preprocessor/

It could speed thing up a bit, but I don't think it will help by much.

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[tomas]

[-- Attachment #1: Type: text/plain, Size: 862 bytes --]

On Tue, Nov 24, 2020 at 08:50:24AM +0100, Arthur Miller wrote:

[...]

> libmpv is just a library to play media files, has nothing to do with
> javascript. You can check yourself:

Until some idiot embeds javascript into some video format spec
and said spec becomes indispensable because That New Platform,
say, Squitch, makes it mandatory (and cheaper videocams follow
that lead, or something).

Then either libmpv follows or we users begin to whine and moan.

You laugh? See how PDF embedded javascript.

Yes, somewhat tongue-in-cheek, but at the same time watching
with a mixture of fascination and horror how this arena around
freedom has changed radically the last 20 years. 

Not long ago the "enemy" was secret sauce and closed interfaces.
These days it's more like cripplingly complex systems and user
convenience.

Cheers
 - t

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Arthur Miller]

<tomas@tuxteam.de> writes:

> On Tue, Nov 24, 2020 at 08:50:24AM +0100, Arthur Miller wrote:
>
> [...]
>
>> libmpv is just a library to play media files, has nothing to do with
>> javascript. You can check yourself:
>
> Until some idiot embeds javascript into some video format spec
> and said spec becomes indispensable because That New Platform,
> say, Squitch, makes it mandatory (and cheaper videocams follow
> that lead, or something).
>
> Then either libmpv follows or we users begin to whine and moan.
>
> You laugh? 
Of course! :D You always make me laugh, you know that! :-)

> See how PDF embedded javascript.
Just for the record didn't that happened long before v8 came and google
made js lingua franca of web dev?

If I remember even MS implemented window host that was ment for scripting
windows (along VB). I never used it though, no idea how extensive it
was, but it was also before the Node and before JS become ubiquitious in
every application, and before applications were "apps" ... :-)

> Yes, somewhat tongue-in-cheek, but at the same time watching
> with a mixture of fascination and horror how this arena around
> freedom has changed radically the last 20 years. 
 
> Not long ago the "enemy" was secret sauce and closed interfaces.
> These days it's more like cripplingly complex systems and user
> convenience.
( ... )

Come writers and critics
Who prophesize with your pen
And keep your eyes wide
The chance won't come again
And don't speak too soon
For the wheel's still in spin
And there's no tellin' who
That it's namin'
For the loser now
Will be later to win
For the times they are a-changin'

( ... )
 -- Robert Allen Zimmerman

> Cheers
>  - t

Cheers my man :-)

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[T.V Raman]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=gb18030, Size: 1116 bytes --]

<tomas@tuxteam.de> writes:


:-) Good point. See how something as simple as a Haskell API Search tool has tangled itself up in Js land (hoogle)> On Tue, Nov 24, 2020 at 08:50:24AM +0100, Arthur Miller wrote:
>
> [...]
>
>> libmpv is just a library to play media files, has nothing to do with
>> javascript. You can check yourself:
>
> Until some idiot embeds javascript into some video format spec
> and said spec becomes indispensable because That New Platform,
> say, Squitch, makes it mandatory (and cheaper videocams follow
> that lead, or something).
>
> Then either libmpv follows or we users begin to whine and moan.
>
> You laugh? See how PDF embedded javascript.
>
> Yes, somewhat tongue-in-cheek, but at the same time watching
> with a mixture of fascination and horror how this arena around
> freedom has changed radically the last 20 years. 
>
> Not long ago the "enemy" was secret sauce and closed interfaces.
> These days it's more like cripplingly complex systems and user
> convenience.
>
> Cheers
>  - t
>

-- 

Thanks,

--Raman
7©4 Id: kg:/m/0285kf1  •0Ü8

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Akira Kyle]

On Mon, Nov 23, 2020 at 11:27 PM, Lars Ingebrigtsen 
<larsi@gnus.org> wrote:

>> Thus I feel like the only sane way to mix the two is to allow 
>> gtk to
>> just take over an Emacs window which is always rectangular. No 
>> need to
>> worry about scrolling and all the redisplay optimizations that 
>> will
>> try to interfere with gtk attempting to draw at some point in 
>> the
>> buffer.
>
> I think this design reduces the use cases for emacs-webkit 
> considerably
> -- if all you can do with it is to let it take over a window 
> completely,
> then that's disappointing.  

Maybe it's worth expanding on this since I was originally trying 
to come up a solution that would work more along the lines of what 
you're asking, but ended up where I am entirely due to technical 
reasons and perhaps that journey may be informative...

Consider that any "rich" display element you might want displayed 
among others in a buffer needs to have some display property so 
that redisplay can be aware of it's position relative to other 
elements in the buffer. Thus there ultimately needs to be some 
entry in the glyph matrix for this "rich" display element. But the 
current way redisplay happens, with its various optimizations, you 
can't expect that every time the element is moved or partially 
clipped or even removed entirely, that Emacs will ask for your 
"rich" element to draw itself.

So you have two options. One is you make your "rich" element 
behave pretty much the way images behave and just give Emacs a 
pixbuf to display. But even then you may need nontrivial redisplay 
changes due to Emacs not expecting the pixbuf data to change out 
from underneath itself. The other is you try to do something like 
what xwidget does and synchronize changes in where redisplay 
calculates the position of the "rich" element to go with drawing 
that element yourself (or letting gtk draw it).

I see this second option as ultimately being pretty brittle as it 
needs to force redisplay to draw your glyph every time redisplay 
changes some properties of its display and pass what those are to 
your "rich" element and hope that it respects them. If it doesn't 
or something in this process breaks, your element ends up floating 
in places it shouldn't be because it keeps drawing itself to the 
screen there. I think the former approach of expanding the image.c 
code or creating something modeled after it would be the better 
approach, likely with less visual bugs. Perhaps cairo could be a 
key component of this if its okay to have it as a dependency on 
all platforms. Anyways that was at least the direction my hacking 
on xwidgets was taking me before I got "sidetracked" by this 
dynamic module idea.

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > was, but it was also before the Node and before JS become ubiquitious in
  > every application,

People can use web apps if they wish, but please do not tell people
that that's all there is nowadays.  GNU/Linux has plenty of
applications which users install on their own computers and contain no
Javascript.

That old way is also the way that is good for users' control of the
software.  Distros package the program and offer their own versions.
In doing this, they audit the code.  Not completely, but it's still a
good thing.  The distros compete but also learn from each other.  That
is the way in which our community traditionally protects itself
against malware and bugs.

"Web applications", even if their code is released as free software,
structure a system of release and usage which has no room for distros,
and this interferes with users' having control over the program.

A structure which precludes this is dangerous in practice, even if in
principle it should not change anything.

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > You laugh? See how PDF embedded javascript.

I checked a few years ago, and ISTR the free PDF players did not
run the Javascript.  At least, by default they did not.

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[tomas]

[-- Attachment #1: Type: text/plain, Size: 1427 bytes --]

On Wed, Nov 25, 2020 at 12:38:37AM -0500, Richard Stallman wrote:
> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
> 
>   > was, but it was also before the Node and before JS become ubiquitious in
>   > every application,
> 
> People can use web apps if they wish, but please do not tell people
> that that's all there is nowadays.  GNU/Linux has plenty of
> applications which users install on their own computers and contain no
> Javascript.
> 
> That old way is also the way that is good for users' control of the
> software.  Distros package the program and offer their own versions.
> In doing this, they audit the code.  Not completely, but it's still a
> good thing.  The distros compete but also learn from each other.  That
> is the way in which our community traditionally protects itself
> against malware and bugs.
> 
> "Web applications", even if their code is released as free software,
> structure a system of release and usage which has no room for distros,
> and this interferes with users' having control over the program.
> 
> A structure which precludes this is dangerous in practice, even if in
> principle it should not change anything.

Can't agree more (see my answer to your other post).

Cheers
 - t

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[tomas]

[-- Attachment #1: Type: text/plain, Size: 1200 bytes --]

On Wed, Nov 25, 2020 at 12:38:56AM -0500, Richard Stallman wrote:
> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
> 
>   > You laugh? See how PDF embedded javascript.
> 
> I checked a few years ago, and ISTR the free PDF players did not
> run the Javascript.  At least, by default they did not.

That's why I use those. Problem is that this creates a sort of "soft
pressure": PDFs made by proprietary tools (Adobe, I'm looking at you)
sometimes don't "work" if the Javascript isn't executed client-side.

The user has the choice to complain to the issuer, almost always
getting blank stares (the issuer is usually many levels of responsibility,
so the user's complaint almost never reaches those who can do
anything about it) or to comply -- being pushed to proprietary
software. Sometimes changing the service provider is an option,
sometimes it isn't.

It's a variation of the "defective by design" pattern, with the
aggravating factor that the user has less and less choice these
days.

Cheers
 - t

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Eli Zaretskii]

> From: Akira Kyle <akira@akirakyle.com>
> Date: Tue, 24 Nov 2020 18:36:38 -0700
> Cc: emacs-devel@gnu.org
> 
> Consider that any "rich" display element you might want displayed 
> among others in a buffer needs to have some display property so 
> that redisplay can be aware of it's position relative to other 
> elements in the buffer. Thus there ultimately needs to be some 
> entry in the glyph matrix for this "rich" display element. But the 
> current way redisplay happens, with its various optimizations, you 
> can't expect that every time the element is moved or partially 
> clipped or even removed entirely, that Emacs will ask for your 
> "rich" element to draw itself.

It is a relatively simple matter to disable some optimizations when a
widget is displayed.  It is also relatively simple to tell the widget
to redraw itself when redisplay changed its position on the screen.

> I see this second option as ultimately being pretty brittle as it 
> needs to force redisplay to draw your glyph every time redisplay 
> changes some properties of its display and pass what those are to 
> your "rich" element and hope that it respects them.

I don't see why you consider this such a significant problem.  The
number of interfaces that such a "rich" display element needs to
support is very small and well-defined.  The problem with xwidgets in
this regard is that its interaction with the display code was left
unfinished, it basically just copy/pastes the code which supports
display of images, which is not 100% correct, since xwidgets don't
display a static bitmap.

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Arthur Miller]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > was, but it was also before the Node and before JS become ubiquitious in
>   > every application,
>
> People can use web apps if they wish, but please do not tell people
> that that's all there is nowadays.
Convenience; everything is in the browser, in one application; it is
just like Emacs. They check mail online, write their documents online,
everything is online. Browser is the Emacs of millenials. Until
connection gets cut. I have been in a war and don't trust anyone; so I
have everything on my own harddrives and prefer real computer
applications to browser, but when I really think of it; browser is
equivalent to how we use Emacs. We have our apps as "packages" written
in Lisp instead of JS.

> That old way is also the way that is good for users' control of the
> software.  Distros package the program and offer their own versions.
> In doing this, they audit the code.  Not completely, but it's still a
> good thing.  The distros compete but also learn from each other.  That
> is the way in which our community traditionally protects itself
> against malware and bugs.
>
> "Web applications", even if their code is released as free software,
> structure a system of release and usage which has no room for distros,
> and this interferes with users' having control over the program.
Indeed; And control over the program and in the very end the user, is
why companies are so in love with webapps.

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Eric S Fraga]

On Wednesday, 25 Nov 2020 at 16:26, Arthur Miller wrote:
> Browser is the Emacs of millenials. 

Maybe more "Browser is the DEC VT100/IBM 3270 for millenials." ;-)

-- 
Eric S Fraga via Emacs 28.0.50 & org 9.4 on Debian bullseye/sid

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > libmpv is a library to play video files like .mp4.  There's no
  > Javascript involved.

Ok on that score.  But why is it desirable to do this in Emacs rather
than run vlc?  It is a big nonmodularity, and that is a big drawback.

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Arthur Miller]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > libmpv is a library to play video files like .mp4.  There's no
>   > Javascript involved.
>
> Ok on that score.  But why is it desirable to do this in Emacs rather
> than run vlc?  It is a big nonmodularity, and that is a big drawback.
Why is it desirable to view jpegs in Emacs rather then run
insert-your-favourite-image-viewer-here? Or why is it desirable to
render html in emacs rather then run Firefox?

For me: less processes, less switching between applications, nice to
have things in Emacs buffers.

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[tomas]

[-- Attachment #1: Type: text/plain, Size: 1066 bytes --]

On Thu, Nov 26, 2020 at 06:25:10AM +0100, Arthur Miller wrote:
> Richard Stallman <rms@gnu.org> writes:
> 
> > [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> > [[[ whether defending the US Constitution against all enemies,     ]]]
> > [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
> >
> >   > libmpv is a library to play video files like .mp4.  There's no
> >   > Javascript involved.
> >
> > Ok on that score.  But why is it desirable to do this in Emacs rather
> > than run vlc?  It is a big nonmodularity, and that is a big drawback.
> Why is it desirable to view jpegs in Emacs rather then run
> insert-your-favourite-image-viewer-here? Or why is it desirable to
> render html in emacs rather then run Firefox?
> 
> For me: less processes, less switching between applications, nice to
> have things in Emacs buffers.

Imagine being able to say "org-store-link" at a specific frame in
a video. Or at specific (x, y) coordinates in a frame in said
video.

Just imagine :-)

Cheers
 - t

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Arthur Miller]

<tomas@tuxteam.de> writes:

> On Thu, Nov 26, 2020 at 06:25:10AM +0100, Arthur Miller wrote:
>> Richard Stallman <rms@gnu.org> writes:
>> 
>> > [[[ To any NSA and FBI agents reading my email: please consider    ]]]
>> > [[[ whether defending the US Constitution against all enemies,     ]]]
>> > [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>> >
>> >   > libmpv is a library to play video files like .mp4.  There's no
>> >   > Javascript involved.
>> >
>> > Ok on that score.  But why is it desirable to do this in Emacs rather
>> > than run vlc?  It is a big nonmodularity, and that is a big drawback.
>> Why is it desirable to view jpegs in Emacs rather then run
>> insert-your-favourite-image-viewer-here? Or why is it desirable to
>> render html in emacs rather then run Firefox?
>> 
>> For me: less processes, less switching between applications, nice to
>> have things in Emacs buffers.
>
> Imagine being able to say "org-store-link" at a specific frame in
> a video. Or at specific (x, y) coordinates in a frame in said
> video.
>
> Just imagine :-)
>
> Cheers
>  - t
Or to render text programmatically on top of video.

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Tomas Hlavaty]

On Thu 26 Nov 2020 at 09:27, <tomas@tuxteam.de> wrote:
> On Thu, Nov 26, 2020 at 06:25:10AM +0100, Arthur Miller wrote:
>> Why is it desirable to view jpegs in Emacs rather then run
>> insert-your-favourite-image-viewer-here? Or why is it desirable to
>> render html in emacs rather then run Firefox?

why does this need libraries instead of programs?

>> For me: less processes,

why is it a good thing?

>> less switching between applications, nice to have things in Emacs
>> buffers.

why does this need libraries instead of programs?

> Imagine being able to say "org-store-link" at a specific frame in
> a video. Or at specific (x, y) coordinates in a frame in said
> video.

why does this need libraries instead of programs?

> Just imagine :-)

imagine all the extra vulnerabilities linked into the emacs process

  $ ldd emacs | wc -l
  94

hmm, that is a lost cause already

libraries are easier to use but programs run in separate processes and
can be extra sandboxed

RE: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[arthur miller]

[-- Attachment #1: Type: text/plain, Size: 2021 bytes --]

Imagine all the vulnerabilities brought in by all the other libraries Emacs uses, and Emacs uses quite many. I don't see difference there.

If you are aware of some  special vulnerability in libmpv, then please rapport it to the project so they can fix it.

For the rest of your questions: answer should probably be self-evident: less processes means less resources used, faster response at runtime and programmatic control usually means more flexibility. For same reasons, I wish even dired was not usin ls program but was built on directory-files instead.

By the way, I hope you can formulate yourself ess passive aggressive in future. I perceive your mail as something  belonging rather to discussions on twitch chatt or reddit at best.


-------- Originalmeddelande --------
Från: Tomas Hlavaty <tom@logand.com>
Datum: 2020-11-26 20:12 (GMT+01:00)
Till: emacs-devel@gnu.org
Ämne: Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

On Thu 26 Nov 2020 at 09:27, <tomas@tuxteam.de> wrote:
> On Thu, Nov 26, 2020 at 06:25:10AM +0100, Arthur Miller wrote:
>> Why is it desirable to view jpegs in Emacs rather then run
>> insert-your-favourite-image-viewer-here? Or why is it desirable to
>> render html in emacs rather then run Firefox?

why does this need libraries instead of programs?

>> For me: less processes,

why is it a good thing?

>> less switching between applications, nice to have things in Emacs
>> buffers.

why does this need libraries instead of programs?

> Imagine being able to say "org-store-link" at a specific frame in
> a video. Or at specific (x, y) coordinates in a frame in said
> video.

why does this need libraries instead of programs?

> Just imagine :-)

imagine all the extra vulnerabilities linked into the emacs process

  $ ldd emacs | wc -l
  94

hmm, that is a lost cause already

libraries are easier to use but programs run in separate processes and
can be extra sandboxed


[-- Attachment #2: Type: text/html, Size: 2959 bytes --]

RE: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Tomas Hlavaty]

On Thu 26 Nov 2020 at 19:34, arthur miller <arthur.miller@live.com> wrote:
> Imagine all the vulnerabilities brought in by all the other libraries
> Emacs uses, and Emacs uses quite many. I don't see difference there.

exactly, so many libraries (i already wrote that my emacs has about 94)
in the same process written in memory unsafe language

also not only vulnerabilities but also memory leaks

> If you are aware of some special vulnerability in libmpv, then please
> rapport it to the project so they can fix it.

if you are aware of some special tumor, then please go to a doctor?

i am talking about prevention

what about not smoking in the first place?

> For the rest of your questions: answer should probably be
> self-evident: less processes means less resources used,

it's 2020 and we want to avoid processes for that reason?  strange

emacs consumes all available memory easily quite often, i don't see any
improvement on this front even with so many libraries

> response at runtime and programmatic control usually means more
> flexibility.

yes, i said that libs are easier to use

> For same reasons, I wish even dired was not usin ls program but was
> built on directory-files instead.

i wish more of emacs was written in memory safe elisp too

but this is a different question altogether

> By the way, I hope you can formulate yourself ess passive aggressive
> in future. I perceive your mail as something belonging rather to
> discussions on twitch chatt or reddit at best.

sorry that it comes that way

it's a shame that you attack me instead of discussed ideas

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Arthur Miller]

Tomas Hlavaty <tom@logand.com> writes:

> On Thu 26 Nov 2020 at 19:34, arthur miller <arthur.miller@live.com> wrote:
>> Imagine all the vulnerabilities brought in by all the other libraries
>> Emacs uses, and Emacs uses quite many. I don't see difference there.
>
> exactly, so many libraries (i already wrote that my emacs has about 94)
> in the same process written in memory unsafe language
>
> also not only vulnerabilities but also memory leaks

>> If you are aware of some special vulnerability in libmpv, then please
>> rapport it to the project so they can fix it.
>
> if you are aware of some special tumor, then please go to a doctor?
>
> i am talking about prevention
>
> what about not smoking in the first place?

>> For the rest of your questions: answer should probably be
>> self-evident: less processes means less resources used,
>
> it's 2020 and we want to avoid processes for that reason?  strange
>
> emacs consumes all available memory easily quite often, i don't see any
> improvement on this front even with so many libraries
Maybe you should help them in the work, send in your traces if it eats
all memory quite often.

>> response at runtime and programmatic control usually means more
>> flexibility.
>
> yes, i said that libs are easier to use
Easier is not same as more flexible. On contrary more flexibility
usually means more complexity. Bringing a library like libmpv into Emacs
is certainly not easy. You are free to check the docs for embedding:

https://github.com/mpv-player/mpv-examples/tree/master/libmpv

Trust me, it is much easier to shell-execute mpv player, or use Emms as
I already do; or just M-! to play music or video with mpv. 

>> For same reasons, I wish even dired was not usin ls program but was
>> built on directory-files instead.
>
> i wish more of emacs was written in memory safe elisp too
>
> but this is a different question altogether
>
>> By the way, I hope you can formulate yourself ess passive aggressive
>> in future. I perceive your mail as something belonging rather to
>> discussions on twitch chatt or reddit at best.
>
> sorry that it comes that way
>
> it's a shame that you attack me instead of discussed ideas
I don't know Thomas; maybe that is way you are communicating that I am
not used too, but I am at least honest and telling you how it sounded to
me; take it or leave it. I still percieve this email as trolling. I
don't see anything really technical you are bringing in; but the very
last thing about memory safe lisp, and I don't mean it is an attack on
you, just the way I read your mail. Maybe it's me. 

Yes in word 2020 I still care about efficiency, I see it as a moral
imperative not to waste resources. I might be weird, but that is how I am.

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

We should not link another library into Emacs unless there is a
particular reason why that library is important.

So far, we have linked in libraries for displaying images which are
part of documents, and we have linked in some libraries for network
protocols that are useful for a variety of purposes.  In these cases
there is an important reason.  Doing these jobs in some other way
would have big disadvantages.

For this job, the disadvantages of the library would impact security
(that library is big), and future maintenance, as well as installation
complexity.  And there is no big disadvantage to forking VLC, or
mplayer2, or whichever player the user prefers.  That is the way we
should do it,

The argument that "We have linked with so many external libraries that
we should not hesitate to add one more" is fundamentally misguided.
Some things are worth paying a price for.  Having bought a few of
them, which msy have been good purchases, it does not follow that we
should rush to buy the whole store.  The price of these purchases is
substantial and we should pay it only when really important.

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Arthur Miller]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
> We should not link another library into Emacs unless there is a
> particular reason why that library is important.
>
> So far, we have linked in libraries for displaying images which are
> part of documents, and we have linked in some libraries for network
> protocols that are useful for a variety of purposes.  In these cases
> there is an important reason.  Doing these jobs in some other way
> would have big disadvantages.
>
> For this job, the disadvantages of the library would impact security
> (that library is big), and future maintenance, as well as installation
> complexity.  And there is no big disadvantage to forking VLC, or
> mplayer2, or whichever player the user prefers.  That is the way we
> should do it,

> The argument that "We have linked with so many external libraries that
> we should not hesitate to add one more" is fundamentally misguided.
It depends on how you read it. If you read it as a solo argument,
a sole purpose of "we have lib A let's have lib B just for purpose of
having it" then I agree with you. But what I ment is saying that
including a library is a security risk and arguing just because of
adding it, is not very much of argument, because we would not have any
external library in Emacs. I don't argue for that one particular just
because we already have so many others. That connects to the next:

> Some things are worth paying a price for.  Having bought a few of
> them, which msy have been good purchases, it does not follow that we
> should rush to buy the whole store.  The price of these purchases is
> substantial and we should pay it only when really important.
Sure of course, not every library under the Sun should be embedded. I am
aware of price/vs cost.

To my knowledge, mpv is probably the neetiest one to bring in media
playing capabilities; it has lots of codecs, is written to be embedded,
is free and would make Emacs be able to play music and video files
without external players. Adding multimedia capabilities opens up for
lots of flexibility and creativity; people can maybe do interesting
stuff with it. I would certainly like Emacs to become a multimedia
player, I play my music with Emacs already :-).

If other people think it is too expensive in terms of implementation
cost and what it offers, and if multimedia is not desirable in Emacs, I
can have understanding with that. I might not agree, but my opinion is
just one persons opinion, and I am not even an Emacs developer, so of
course, you who make Emacs work probably know better and have precedence
in what Emacs should do/have or not. It would be wrong to claim anything
else :). 

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Eli Zaretskii]

> From: Arthur Miller <arthur.miller@live.com>
> Date: Fri, 27 Nov 2020 20:22:42 +0100
> Cc: tom@logand.com, emacs-devel@gnu.org
> 
> To my knowledge, mpv is probably the neetiest one to bring in media
> playing capabilities; it has lots of codecs, is written to be embedded,
> is free and would make Emacs be able to play music and video files
> without external players. Adding multimedia capabilities opens up for
> lots of flexibility and creativity; people can maybe do interesting
> stuff with it. I would certainly like Emacs to become a multimedia
> player, I play my music with Emacs already :-).
> 
> If other people think it is too expensive in terms of implementation
> cost and what it offers, and if multimedia is not desirable in Emacs, I
> can have understanding with that. I might not agree, but my opinion is
> just one persons opinion, and I am not even an Emacs developer, so of
> course, you who make Emacs work probably know better and have precedence
> in what Emacs should do/have or not. It would be wrong to claim anything
> else :). 

Expensive or not, talking about mpv as providing video playing
facilities for Emacs is makes little sense as long as the issue of
embedding video in a window that shows an Emacs buffer is not
resolved.  As the xwidgets experiment suggests, the problem to solve
here is not a trivial one.  Until someone comes up with some clever
idea for how to do that, let alone submits patches for that, the issue
of which library to use is not really relevant to Emacs development.

If someone wants a flexible way of playing video under Emacs control,
I think a more practically useful way at this time is to provide a
feature that runs VLC or a similar player program as an Emacs
subprocess, and controls that player via Emacs commands.  That could
need extending the players themselves, if their command-line arguments
aren't flexible enough and there's no other API that Emacs could use.

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Stefan Monnier]

> To my knowledge, mpv is probably the neetiest one to bring in media
> playing capabilities; it has lots of codecs, is written to be embedded,
> is free and would make Emacs be able to play music and video files
> without external players. Adding multimedia capabilities opens up for
> lots of flexibility and creativity; people can maybe do interesting
> stuff with it. I would certainly like Emacs to become a multimedia
> player,

The question is what benefit is there to having it linked in as
a library as opposed to accessing similar functionality via
a sub-process.

> I play my music with Emacs already :-).

This would tend to argue in favor of not having it as a library since
you can already do what you want just fine using a sub-process.


        Stefan

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Akira Kyle]

On Wed, Nov 25, 2020 at 08:11 AM, Eli Zaretskii <eliz@gnu.org> 
wrote:

>> Consider that any "rich" display element you might want 
>> displayed 
>> among others in a buffer needs to have some display property so 
>> that redisplay can be aware of it's position relative to other 
>> elements in the buffer. Thus there ultimately needs to be some 
>> entry in the glyph matrix for this "rich" display element. But 
>> the 
>> current way redisplay happens, with its various optimizations, 
>> you 
>> can't expect that every time the element is moved or partially 
>> clipped or even removed entirely, that Emacs will ask for your 
>> "rich" element to draw itself.
>
> It is a relatively simple matter to disable some optimizations 
> when a
> widget is displayed.  It is also relatively simple to tell the 
> widget
> to redraw itself when redisplay changed its position on the 
> screen.

Yes, but you had previously pointed to that fact that xwidgets 
needs to such disable such optimizations as part of the "kludge" 
that would need to be resolved for xwidgets to be considered 
viable for a path forward. Having looked at that, I'm not sure gtk 
will ever want to cooperate without doing things that would be 
considered hacks by gtk folks.

>> I see this second option as ultimately being pretty brittle as 
>> it 
>> needs to force redisplay to draw your glyph every time 
>> redisplay 
>> changes some properties of its display and pass what those are 
>> to 
>> your "rich" element and hope that it respects them.
>
> I don't see why you consider this such a significant problem. 
> The
> number of interfaces that such a "rich" display element needs to
> support is very small and well-defined.  The problem with 
> xwidgets in
> this regard is that its interaction with the display code was 
> left
> unfinished, it basically just copy/pastes the code which 
> supports
> display of images, which is not 100% correct, since xwidgets 
> don't
> display a static bitmap.

The "number interfaces" will likely be small no matter what 
solution is taken, its more about how easy you want to make it for 
a "rich" display element to really mess with Emacs' display and 
input handling. For example with xwidgets the onus is on the 
widget implementation to not totally steal keyboard focus from 
Emacs. I've found that gtk widgets really like stealing keyboard 
focus so its easy to end up in a state you can't escape out 
of. Similarly gtk widgets won't always easily draw themselves 
where you want at the size you want so its easy for widgets to 
render, say, across window boundaries. The xwidget approach 
requires non-insignificant work on the widget end to ensure the 
widgets will work within the confines of Emacs' redisplay. I'd 
think that such an interface to support "rich" display elements 
should be restrictive enough to not easily allow the "rich" 
display elements to misbehave. Especially if it becomes possible 
to define such elements from lisp. I see expanding the current 
image handling to give richer canvases that can be drawn on as the 
"safer" approach with respect to these concerns. But again much of 
this is highly speculative and doesn't mean much until some code 
is actually written. I'm just trying to voice some of the 
frustrations I had in working on the current xwidget code and how 
that might apply here.

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Arthur Miller]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> To my knowledge, mpv is probably the neetiest one to bring in media
>> playing capabilities; it has lots of codecs, is written to be embedded,
>> is free and would make Emacs be able to play music and video files
>> without external players. Adding multimedia capabilities opens up for
>> lots of flexibility and creativity; people can maybe do interesting
>> stuff with it. I would certainly like Emacs to become a multimedia
>> player,
>
> The question is what benefit is there to having it linked in as
> a library as opposed to accessing similar functionality via
> a sub-process.
One could programmatically render stuff on top of the video; embedd it
in a viewer to create a player, like there is image slide for image
mode, video tag for shr was what triggered discussion. I mean you could
code a renderer for video just as you made svg renderer; but somebody
already created a re-usable, well tested library, so why recreate what
is already done and tested. That is how I think of it.

>> I play my music with Emacs already :-).
>
> This would tend to argue in favor of not having it as a library since
> you can already do what you want just fine using a sub-process.
Yeah, I know; but just playing music is a little bit of playing with a
closed box. And it does not work with video. Also as I said it is a bit
about efficiency. I think it would be useful for Emacs to play sound
files and video files natively.

For me Emacs is long past just text editor; for me it is more like an
über-terminal or we could call it OS-browser ... I don't know. Just how
I use it, I understand if others use Emacs differently.

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Arthur Miller]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Arthur Miller <arthur.miller@live.com>
>> Date: Fri, 27 Nov 2020 20:22:42 +0100
>> Cc: tom@logand.com, emacs-devel@gnu.org
>> 
>> To my knowledge, mpv is probably the neetiest one to bring in media
>> playing capabilities; it has lots of codecs, is written to be embedded,
>> is free and would make Emacs be able to play music and video files
>> without external players. Adding multimedia capabilities opens up for
>> lots of flexibility and creativity; people can maybe do interesting
>> stuff with it. I would certainly like Emacs to become a multimedia
>> player, I play my music with Emacs already :-).
>> 
>> If other people think it is too expensive in terms of implementation
>> cost and what it offers, and if multimedia is not desirable in Emacs, I
>> can have understanding with that. I might not agree, but my opinion is
>> just one persons opinion, and I am not even an Emacs developer, so of
>> course, you who make Emacs work probably know better and have precedence
>> in what Emacs should do/have or not. It would be wrong to claim anything
>> else :). 
>
> Expensive or not, talking about mpv as providing video playing
> facilities for Emacs is makes little sense as long as the issue of
> embedding video in a window that shows an Emacs buffer is not
> resolved.  As the xwidgets experiment suggests, the problem to solve
> here is not a trivial one.
Indeed; I am aware of it, and I personally am not even sure it is
possible with Emacs renderer as it is. If you read one of my first
replay's to Lars, I pretty much hoped he understands Emacs better then I
so he could do it, because I honestly, am not sure I can, I tried before
and couldn't figure it out. :)

I am playing with it, but I am not that knowledgable about
internals. Playing video on Emacs frame is not that hard; it is just
basic X11/GDI/whatever platform we are talking about. I can just attach
opengl context and send OS window to mpv, with some thinkgering. But
playing video in an image container, as images are put as faces on
characters, is not something I currently know how and could be able to
get going.

> Until someone comes up with some clever
> idea for how to do that, let alone submits patches for that, the issue
> of which library to use is not really relevant to Emacs development.
>
> If someone wants a flexible way of playing video under Emacs control,
> I think a more practically useful way at this time is to provide a
> feature that runs VLC or a similar player program as an Emacs
> subprocess, and controls that player via Emacs commands.  That could
> need extending the players themselves, if their command-line arguments
> aren't flexible enough and there's no other API that Emacs could use.
If you mean, just as external application in it's own frame, then Emms
already does it (I think).

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Akira Kyle]

On Fri, Nov 27, 2020 at 01:07 PM, Stefan Monnier 
<monnier@iro.umontreal.ca> wrote:

>> To my knowledge, mpv is probably the neetiest one to bring in 
>> media
>> playing capabilities; it has lots of codecs, is written to be 
>> embedded,
>> is free and would make Emacs be able to play music and video 
>> files
>> without external players. Adding multimedia capabilities opens 
>> up for
>> lots of flexibility and creativity; people can maybe do 
>> interesting
>> stuff with it. I would certainly like Emacs to become a 
>> multimedia
>> player,
>
> The question is what benefit is there to having it linked in as
> a library as opposed to accessing similar functionality via
> a sub-process.

I don't think this necessarily has to be a question of Emacs links 
against yet another library, or someone wraps a program's command 
line arguments. As I'm hoping to show with `emacs-webkit`, it's 
viable to think about using dynamic modules which link against a 
library and provide the desired functionality. Then those who 
never want that functionality never have to worry about their 
Emacs being linked with it, while the lisp package providing the 
functionality is free to use the best interface to that library, 
which I often think will be through linking to it. Linking often 
provides better performance and less complexity as you avoid 
needing to constantly define a data exchange format and parse it.

I once saw someone arguing why "the Emacs way" was far superior 
"the unix way" mainly because unix insists on everything being 
thought of as just files and text, which while powerful, ends up 
with every program defining different, incompatible text 
interfaces and parsing formats. Case in point being how messy 
command line argument formats can be since there's no standard 
format imposed on the way a program should handle flags. Meanwhile 
with Emacs everything is a Sexpr, which along with being easier to 
manipulate, offers just enough structure to greatly reduce these 
kind of incompatibilities between programs trying to communicate 
with eachother.

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Stefan Monnier]

>> The question is what benefit is there to having it linked in as
>> a library as opposed to accessing similar functionality via
>> a sub-process.
> One could programmatically render stuff on top of the video;

Sounds highly hypothetical.

> I mean you could code a renderer for video just as you made svg
> renderer; but somebody already created a re-usable, well tested
> library, so why recreate what is already done and tested.  That is how
> I think of it.

Same here: just linking libmpv doesn't magically let you do those things.

Until such things are more concrete, it's hard to justify linking a library.

The emacs-application-framework is an example of what you can do with
sub-processes.

> Yeah, I know; but just playing music is a little bit of playing with a
> closed box. And it does not work with video. Also as I said it is a bit
> about efficiency. I think it would be useful for Emacs to play sound
> files and video files natively.

Yet there are also packages which let you play videos from Emacs, so
it's not limited to music.

> For me Emacs is long past just text editor; for me it is more like an
> über-terminal or we could call it OS-browser ... I don't know.  Just how
> I use it, I understand if others use Emacs differently.

I think this description applies to pretty much all of us here.  It does
not imply that linking is necessarily the better path.

Linking libraries into Emacs tends to be tricky, costly, and come with
a fair bit of consequences, so it needs to justified by fairly
solid evidence.  Linking them via the module interface can be a good
middle-ground BTW.


        Stefan

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Arthur Miller]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>> The question is what benefit is there to having it linked in as
>>> a library as opposed to accessing similar functionality via
>>> a sub-process.
>> One could programmatically render stuff on top of the video;
>
> Sounds highly hypothetical.
Why is it hypothetical? With their render api, it will take an opengl
surface to render on and that surface can later be renderd on. I don't
say it is trivial or easy; just possible.

> Same here: just linking libmpv doesn't magically let you do those things.
Of course not; I don't understand why do you say so in this case, why
would you think I think so? I was obviously aiming at the fact that
instead of  writing own interface to some lower level video decoding
library or maybe your own decoding routines, Emacs could re-use
libmpv. I don't see what is hypotethical there.

> Linking libraries into Emacs tends to be tricky, costly, and come with
> a fair bit of consequences, so it needs to justified by fairly
> solid evidence.
Of course I agree. But is also a matter of personal preferance, what is
justified and solid evidence. 

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Stefan Monnier]

>> Sounds highly hypothetical.
> Why is it hypothetical?

Because it's pretty far from what existing code can do.


        Stefan

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

Each proposed external library requires a cost/benefit judgment.
In my judgment, the benefits of linking a video library are small
because it is just a frill.

Displaying images is important because documents contain images.

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Eli Zaretskii]

> From: Akira Kyle <akira@akirakyle.com>
> Date: Fri, 27 Nov 2020 14:22:37 -0700
> Cc: emacs-devel@gnu.org, Richard Stallman <rms@gnu.org>,
>  Arthur Miller <arthur.miller@live.com>, tom@logand.com
> 
> > The question is what benefit is there to having it linked in as
> > a library as opposed to accessing similar functionality via
> > a sub-process.
> 
> I don't think this necessarily has to be a question of Emacs links 
> against yet another library, or someone wraps a program's command 
> line arguments.

But that was the question that was asked, and that was the question to
which Richard responded.

Other questions are possible, and will likely have different answers.

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Eli Zaretskii]

> From: Akira Kyle <akira@akirakyle.com>
> Cc: larsi@gnus.org, emacs-devel@gnu.org
> Date: Fri, 27 Nov 2020 13:56:25 -0700
> 
> > It is a relatively simple matter to disable some optimizations
> > when a widget is displayed.  It is also relatively simple to tell
> > the widget to redraw itself when redisplay changed its position on
> > the screen.
> 
> Yes, but you had previously pointed to that fact that xwidgets 
> needs to such disable such optimizations as part of the "kludge" 
> that would need to be resolved for xwidgets to be considered 
> viable for a path forward. Having looked at that, I'm not sure gtk 
> will ever want to cooperate without doing things that would be 
> considered hacks by gtk folks.

I'd appreciate more details on the GTK part of this.  I only said that
from the Emacs display engine side things are not too hairy.  from my
POV, xwidgets integration simply stopped short of going all the way
towards integration into the redisplay framework, and I don't think
anyone said before that this was done because better integration was
impossible or very hard.

> > I don't see why you consider this such a significant problem.  The
> > number of interfaces that such a "rich" display element needs to
> > support is very small and well-defined.  The problem with xwidgets
> > in this regard is that its interaction with the display code was
> > left unfinished, it basically just copy/pastes the code which
> > supports display of images, which is not 100% correct, since
> > xwidgets don't display a static bitmap.
> 
> The "number interfaces" will likely be small no matter what 
> solution is taken, its more about how easy you want to make it for 
> a "rich" display element to really mess with Emacs' display and 
> input handling. For example with xwidgets the onus is on the 
> widget implementation to not totally steal keyboard focus from 
> Emacs. I've found that gtk widgets really like stealing keyboard 
> focus so its easy to end up in a state you can't escape out 
> of.

Keyboard focus is a separate issue.  I only talked about integration
with redisplay.

For keyboard input integration, we'd need to develop some
infrastructure, if the existing one doesn't provide enough hooks.  If
you or someone else can describe the details, we could discuss that in
more practical terms.  I think it's too soon to give up on the
practical possibility to solve these problems before we discussed
that, or are even fully aware of the problems and the potential
solutions.

> Similarly gtk widgets won't always easily draw themselves 
> where you want at the size you want so its easy for widgets to 
> render, say, across window boundaries.

If that is true, then your method of letting the widget display in a
window won't work well either, right?  IOW, whether the screen area
given to the widget is a full Emacs window or just its part is not an
important aspect of the integration of such widgets into the Emacs
display.

> The xwidget approach requires non-insignificant work on the widget
> end to ensure the widgets will work within the confines of Emacs'
> redisplay.

As long as the widget can be told to use a given rectangular portion
of the screen, we should be fine: the current Emacs display engine is
perfectly capable of handling display elements that occupy an area of
certain dimensions.

> I'd think that such an interface to support "rich" display elements
> should be restrictive enough to not easily allow the "rich" display
> elements to misbehave.

Please tell more about the reasons for this conclusion.  I don't yet
see the difficulties you envision.

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Arthur Miller]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>> Sounds highly hypothetical.
>> Why is it hypothetical?
>
> Because it's pretty far from what existing code can do.
That's true :-) Thought you were referring to what is possible with
libmpv. Yes, no idea if/how easily those capabilites can be exploited
from Emacs; but from the libmpv point of view, it is possible.

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Arthur Miller]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
> Each proposed external library requires a cost/benefit judgment.
> In my judgment, the benefits of linking a video library are small
> because it is just a frill.
:-)

Not necessary just a frill; wouldn't <video> element be nice? But for
that one Emacs can maybe wrap ffmepg directly instead; I don't know, but
then question is still stame; external lib.

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Arthur Miller]

Eli Zaretskii <eliz@gnu.org> writes:

> As long as the widget can be told to use a given rectangular portion
> of the screen, we should be fine: the current Emacs display engine is
> perfectly capable of handling display elements that occupy an area of
> certain dimensions.

How can I tell any drawing routines to use certain rectangular area on
display so it is in harmony with Emacs display engine? I thought it was
done with faces and inserting characters with certain face; but I am not
sure how it was actually done.

Are there callbacks I can connect too if I would like to update opengl
viewport that I can hook easily into? Where do I look? I am not sure I
understand it from image.c; it was there I was looking. I never looked
into xwidgets though. Can be just me; but if you can point me to right
place to look at I am thankful. 

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Eli Zaretskii]

> From: Arthur Miller <arthur.miller@live.com>
> Cc: Akira Kyle <akira@akirakyle.com>,  larsi@gnus.org,  emacs-devel@gnu.org
> Date: Sat, 28 Nov 2020 09:57:39 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > As long as the widget can be told to use a given rectangular portion
> > of the screen, we should be fine: the current Emacs display engine is
> > perfectly capable of handling display elements that occupy an area of
> > certain dimensions.
> 
> How can I tell any drawing routines to use certain rectangular area on
> display so it is in harmony with Emacs display engine?

On the display engine level, you need to define a new kind of "display
element" (see 'enum glyph_type' in dispextern.h) and new kind of
method for producing such a display element (see 'enum it_method' in
dispextern.h).  You will also need to write a BUILD_<FOO>_GLYPH_STRING
macro that produces a "glyph string" from one or more display elements
of this new type; see BUILD_XWIDGET_GLYPH_STRING in xdisp.c as an
example.  Glyph strings are the interface between xdisp.c, which
parses buffers and strings for display and performs display layout,
and xterm.c (see below) which actually draws the resulting display
elements.

On the Lisp level, you will probably use a special 'display' property
whose value specifies the widget with the necessary parameters, like
we do with images.

> Are there callbacks I can connect too if I would like to update opengl
> viewport that I can hook easily into? Where do I look? I am not sure I
> understand it from image.c; it was there I was looking. I never looked
> into xwidgets though. Can be just me; but if you can point me to right
> place to look at I am thankful. 

The actual drawing is in xterm.c (for X; it's w32term.c for
MS-Windows, nsterm.m for NS, and term.c for TTY frames).  Each type of
display element has a function there, which is called to draw the
display element.  For example, x_draw_image_glyph_string for drawing
images.  You will need to write a new function for this new display
element, whose parameters are passed from the new it_method method
above.  Not sure if this is what you meant by "callback to connect to
in order to update opengl viewport".

(image.c just prepares the data required for drawing an image; its
functions are not called when drawing the image, we just use the
pixmap produced by image.c and various other parameters, like image
dimensions etc.)

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Arthur Miller]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Arthur Miller <arthur.miller@live.com>
>> Cc: Akira Kyle <akira@akirakyle.com>,  larsi@gnus.org,  emacs-devel@gnu.org
>> Date: Sat, 28 Nov 2020 09:57:39 +0100
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> > As long as the widget can be told to use a given rectangular portion
>> > of the screen, we should be fine: the current Emacs display engine is
>> > perfectly capable of handling display elements that occupy an area of
>> > certain dimensions.
>> 
>> How can I tell any drawing routines to use certain rectangular area on
>> display so it is in harmony with Emacs display engine?
>
> On the display engine level, you need to define a new kind of "display
> element" (see 'enum glyph_type' in dispextern.h) and new kind of
> method for producing such a display element (see 'enum it_method' in
> dispextern.h).  You will also need to write a BUILD_<FOO>_GLYPH_STRING
> macro that produces a "glyph string" from one or more display elements
> of this new type; see BUILD_XWIDGET_GLYPH_STRING in xdisp.c as an
> example.  Glyph strings are the interface between xdisp.c, which
> parses buffers and strings for display and performs display layout,
> and xterm.c (see below) which actually draws the resulting display
> elements.
>
> On the Lisp level, you will probably use a special 'display' property
> whose value specifies the widget with the necessary parameters, like
> we do with images.
>
>> Are there callbacks I can connect too if I would like to update opengl
>> viewport that I can hook easily into? Where do I look? I am not sure I
>> understand it from image.c; it was there I was looking. I never looked
>> into xwidgets though. Can be just me; but if you can point me to right
>> place to look at I am thankful. 
>
> The actual drawing is in xterm.c (for X; it's w32term.c for
> MS-Windows, nsterm.m for NS, and term.c for TTY frames).  Each type of
> display element has a function there, which is called to draw the
> display element.  For example, x_draw_image_glyph_string for drawing
> images.  You will need to write a new function for this new display
> element, whose parameters are passed from the new it_method method
> above.  Not sure if this is what you meant by "callback to connect to
> in order to update opengl viewport".
>
> (image.c just prepares the data required for drawing an image; its
> functions are not called when drawing the image, we just use the
> pixmap produced by image.c and various other parameters, like image
> dimensions etc.)
Thank you for the kind tour! This was much more then I was hoping for :-).

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Akira Kyle]

On Sat, Nov 28, 2020 at 01:38 AM, Eli Zaretskii <eliz@gnu.org> 
wrote:

>> > It is a relatively simple matter to disable some 
>> > optimizations
>> > when a widget is displayed.  It is also relatively simple to 
>> > tell
>> > the widget to redraw itself when redisplay changed its 
>> > position on
>> > the screen.
>> 
>> Yes, but you had previously pointed to that fact that xwidgets 
>> needs to such disable such optimizations as part of the 
>> "kludge" 
>> that would need to be resolved for xwidgets to be considered 
>> viable for a path forward. Having looked at that, I'm not sure 
>> gtk 
>> will ever want to cooperate without doing things that would be 
>> considered hacks by gtk folks.
>
> I'd appreciate more details on the GTK part of this.  I only 
> said that
> from the Emacs display engine side things are not too hairy. 
> from my
> POV, xwidgets integration simply stopped short of going all the 
> way
> towards integration into the redisplay framework, and I don't 
> think
> anyone said before that this was done because better integration 
> was
> impossible or very hard.

One such fundamental incompatibility stems from the fact that 
Emacs may have multiple views on one buffer split across various 
windows and/or frames. Gtk assumes that each widget will only have 
one view. The 'webkit xwidget type currently works around this by 
drawing to an offscreen window then for every window that's 
supposed to display the widget, creating a gtk drawing area widget 
and copying the offscreen window's surface to the drawing 
surface. I don't think gtk really intends for ofscreen widgets to 
be used this way and as far as I can tell offscreen window widgets 
have been removed in gtk4.

Another issue is that emacs uses a GtkFixed (or a custom 
subclassed EmacsFixed for gtk3) as the container widget for the 
main frame area. There isn't a mechanism to control z-ordering of 
child widgets other than removing and re-adding them in the 
desired z-order, which is what emacs-webkit has to do in order for 
child frames to not render below the webkit widget.

>> > I don't see why you consider this such a significant problem. 
>> > The
>> > number of interfaces that such a "rich" display element needs 
>> > to
>> > support is very small and well-defined.  The problem with 
>> > xwidgets
>> > in this regard is that its interaction with the display code 
>> > was
>> > left unfinished, it basically just copy/pastes the code which
>> > supports display of images, which is not 100% correct, since
>> > xwidgets don't display a static bitmap.
>> 
>> The "number interfaces" will likely be small no matter what 
>> solution is taken, its more about how easy you want to make it 
>> for 
>> a "rich" display element to really mess with Emacs' display and 
>> input handling. For example with xwidgets the onus is on the 
>> widget implementation to not totally steal keyboard focus from 
>> Emacs. I've found that gtk widgets really like stealing 
>> keyboard 
>> focus so its easy to end up in a state you can't escape out 
>> of.
>
> Keyboard focus is a separate issue.  I only talked about 
> integration
> with redisplay.
>
> For keyboard input integration, we'd need to develop some
> infrastructure, if the existing one doesn't provide enough 
> hooks.  If
> you or someone else can describe the details, we could discuss 
> that in
> more practical terms.  I think it's too soon to give up on the
> practical possibility to solve these problems before we 
> discussed
> that, or are even fully aware of the problems and the potential
> solutions.
>
>> Similarly gtk widgets won't always easily draw themselves 
>> where you want at the size you want so its easy for widgets to 
>> render, say, across window boundaries.
>
> If that is true, then your method of letting the widget display 
> in a
> window won't work well either, right?  IOW, whether the screen 
> area
> given to the widget is a full Emacs window or just its part is 
> not an
> important aspect of the integration of such widgets into the 
> Emacs
> display.
>
>> The xwidget approach requires non-insignificant work on the 
>> widget
>> end to ensure the widgets will work within the confines of 
>> Emacs'
>> redisplay.
>
> As long as the widget can be told to use a given rectangular 
> portion
> of the screen, we should be fine: the current Emacs display 
> engine is
> perfectly capable of handling display elements that occupy an 
> area of
> certain dimensions.

This is more an issue on the gtk side of things. With the webkit 
widget it isn't an issue since the widget doesn't define a minimum 
size so any size you request it to render at, it will 
obey. However other widgets, such as buttons, sliders, etc, have a 
minimum size they will render at. If you tell it to render itself 
smaller, it will refuse and so it's impossible to simply tell the 
widget "here's the space you have to go in". This becomes a 
problem when the widget needs to be clipped at a window boundary 
as gtk will happily draw the widgets across them as gtk has no 
internal knowledge of Emacs window boundaries being a place that 
widget must be clipped at. Thus every widget that will live in a 
buffer will need to be inside a custom container than handles this 
clipping for the widget.

>> I'd think that such an interface to support "rich" display 
>> elements
>> should be restrictive enough to not easily allow the "rich" 
>> display
>> elements to misbehave.
>
> Please tell more about the reasons for this conclusion.  I don't 
> yet
> see the difficulties you envision.

While all these issues can be overcome in one way or another, I 
think they point at the differences both the UI paradigms of Emacs 
and gtk and the subsequent differences in how each want to draw to 
the screen. In addition to the above specific examples of 
difficulties I've encountered so far with making gtk widgets work 
well with Emacs' display, there's the fundamental problem of this 
route for "richer display elements" not being portable. It'll 
always be confined to Emacs builds `--with-x` (or `--with-pgtk` 
now). While I feel this is fine for emacs-webkit, I think it would 
be advantageous to not tie such a feature to gtk itself.

The direction I'm thinking of going in the future since I feel 
like thus far the xwidget route has been fairly troublesome is to 
expose interfaces to the cairo surfaces used in image.c. This 
could be through modifying image.c or through defining a new 
display element type. I would try to expose the cairo API to lisp 
and allow dynamic modules to directly draw to a cairo surface 
given to it through the lisp interface. Gtk along with many other 
libraries (such as librsvg and poppler) support drawing to a cairo 
surface and then Emacs gets to have full control over how that 
cario surface ends up on screen. I would see this as a stepping 
stone to attempting to do something similar with a opengl surface.

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

The kind of argument you are making is entirely spurious.

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[martin rudalics]

 > Another issue is that emacs uses a GtkFixed (or a custom subclassed
 > EmacsFixed for gtk3) as the container widget for the main frame
 > area. There isn't a mechanism to control z-ordering of child widgets
 > other than removing and re-adding them in the desired z-order, which
 > is what emacs-webkit has to do in order for child frames to not render
 > below the webkit widget.

I think I see something like this in the pgtk branch - a child frame
appearing beneath the parent frame's horizontal scroll bar.

 > This is more an issue on the gtk side of things. With the webkit
 > widget it isn't an issue since the widget doesn't define a minimum
 > size so any size you request it to render at, it will obey. However
 > other widgets, such as buttons, sliders, etc, have a minimum size they
 > will render at. If you tell it to render itself smaller, it will
 > refuse and so it's impossible to simply tell the widget "here's the
 > space you have to go in". This becomes a problem when the widget needs
 > to be clipped at a window boundary as gtk will happily draw the
 > widgets across them as gtk has no internal knowledge of Emacs window
 > boundaries being a place that widget must be clipped at. Thus every
 > widget that will live in a buffer will need to be inside a custom
 > container than handles this clipping for the widget.

We are already bitten by this often enough: Toolbars that auto resize
their frame, scroll bars that extend into the next window.

martin

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Eli Zaretskii]

> From: Akira Kyle <akira@akirakyle.com>
> Cc: larsi@gnus.org, emacs-devel@gnu.org
> Date: Sat, 28 Nov 2020 18:29:08 -0700
> 
> One such fundamental incompatibility stems from the fact that 
> Emacs may have multiple views on one buffer split across various 
> windows and/or frames. Gtk assumes that each widget will only have 
> one view. The 'webkit xwidget type currently works around this by 
> drawing to an offscreen window then for every window that's 
> supposed to display the widget, creating a gtk drawing area widget 
> and copying the offscreen window's surface to the drawing 
> surface. I don't think gtk really intends for ofscreen widgets to 
> be used this way and as far as I can tell offscreen window widgets 
> have been removed in gtk4.
>
> Another issue is that emacs uses a GtkFixed (or a custom 
> subclassed EmacsFixed for gtk3) as the container widget for the 
> main frame area. There isn't a mechanism to control z-ordering of 
> child widgets other than removing and re-adding them in the 
> desired z-order, which is what emacs-webkit has to do in order for 
> child frames to not render below the webkit widget.

The same problems plague your experiment with a module, don't they?

Anyway, this just tells me that GTK is not a good starting point for
embedding widgets.  Do other projects build embedded widgets on such
shaky grounds?

> > As long as the widget can be told to use a given rectangular
> > portion of the screen, we should be fine: the current Emacs
> > display engine is perfectly capable of handling display elements
> > that occupy an area of certain dimensions.
> 
> This is more an issue on the gtk side of things. With the webkit 
> widget it isn't an issue since the widget doesn't define a minimum 
> size so any size you request it to render at, it will 
> obey. However other widgets, such as buttons, sliders, etc, have a 
> minimum size they will render at. If you tell it to render itself 
> smaller, it will refuse and so it's impossible to simply tell the 
> widget "here's the space you have to go in". This becomes a 
> problem when the widget needs to be clipped at a window boundary 
> as gtk will happily draw the widgets across them as gtk has no 
> internal knowledge of Emacs window boundaries being a place that 
> widget must be clipped at.

This is solvable.  We already have similar situation in the display
engine, where some display element cannot be usefully "clipped".  We
either don't display it at all or display it on the next screen line,
depending on the wrap mode.

> The direction I'm thinking of going in the future since I feel 
> like thus far the xwidget route has been fairly troublesome is to 
> expose interfaces to the cairo surfaces used in image.c. This 
> could be through modifying image.c or through defining a new 
> display element type. I would try to expose the cairo API to lisp 
> and allow dynamic modules to directly draw to a cairo surface 
> given to it through the lisp interface. Gtk along with many other 
> libraries (such as librsvg and poppler) support drawing to a cairo 
> surface and then Emacs gets to have full control over how that 
> cario surface ends up on screen. I would see this as a stepping 
> stone to attempting to do something similar with a opengl surface.

My suggestion is to think how to integrate this idea into Emacs
proper, without introducing modules into the equation.  Modules add
another layer of complexity, so I would suggest to design the solution
inside Emacs, and only after that see how to do that from a module.
One step at a time.

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Akira Kyle]

On Sun, Nov 29, 2020 at 12:01 PM, Eli Zaretskii <eliz@gnu.org> 
wrote:

>> One such fundamental incompatibility stems from the fact that 
>> Emacs may have multiple views on one buffer split across 
>> various 
>> windows and/or frames. Gtk assumes that each widget will only 
>> have 
>> one view. The 'webkit xwidget type currently works around this 
>> by 
>> drawing to an offscreen window then for every window that's 
>> supposed to display the widget, creating a gtk drawing area 
>> widget 
>> and copying the offscreen window's surface to the drawing 
>> surface. I don't think gtk really intends for ofscreen widgets 
>> to 
>> be used this way and as far as I can tell offscreen window 
>> widgets 
>> have been removed in gtk4.
>>
>> Another issue is that emacs uses a GtkFixed (or a custom 
>> subclassed EmacsFixed for gtk3) as the container widget for the 
>> main frame area. There isn't a mechanism to control z-ordering 
>> of 
>> child widgets other than removing and re-adding them in the 
>> desired z-order, which is what emacs-webkit has to do in order 
>> for 
>> child frames to not render below the webkit widget.
>
> The same problems plague your experiment with a module, don't 
> they?

More or less yes. Restricting myself to just a webkit widget 
allows me to not have to solve these problems generally. For 
example I don't draw the widget offscreen, instead I take a lesson 
from exwm and try to force Emacs to only ever display at most one 
window per webkit buffer.

> Anyway, this just tells me that GTK is not a good starting point 
> for
> embedding widgets.  Do other projects build embedded widgets on 
> such
> shaky grounds?

I wouldn't say GTK isn't good for embedding widgets. In fact I'd 
say it's probably great if you're wanting to extend an existing 
GTK app with custom widgets. The problem is Emacs, despite using 
GTK as a toolkit, isn't really a GTK app. Emacs just uses GTK to 
handle getting input and drawing a surface to the screen. Because 
Emacs isn't itself composed of GTK "normal" widgets, but rather 
just one GtkFixed container it does all its drawing in, you can't 
expect GTK widgets added along side what Emacs draws in that area 
to play nicely together without significant effort. And that 
effort boils down to making GTK aware of what Emacs has drawn and 
vice versa. If all the UI components of Emacs such as windows, the 
mode line, etc, were each implemented as a sensible GTK widget, 
then this wouldn't be so much of an issue since then GTK would 
already know where and what things are.

>> > As long as the widget can be told to use a given rectangular
>> > portion of the screen, we should be fine: the current Emacs
>> > display engine is perfectly capable of handling display 
>> > elements
>> > that occupy an area of certain dimensions.
>> 
>> This is more an issue on the gtk side of things. With the 
>> webkit 
>> widget it isn't an issue since the widget doesn't define a 
>> minimum 
>> size so any size you request it to render at, it will 
>> obey. However other widgets, such as buttons, sliders, etc, 
>> have a 
>> minimum size they will render at. If you tell it to render 
>> itself 
>> smaller, it will refuse and so it's impossible to simply tell 
>> the 
>> widget "here's the space you have to go in". This becomes a 
>> problem when the widget needs to be clipped at a window 
>> boundary 
>> as gtk will happily draw the widgets across them as gtk has no 
>> internal knowledge of Emacs window boundaries being a place 
>> that 
>> widget must be clipped at.
>
> This is solvable.  We already have similar situation in the 
> display
> engine, where some display element cannot be usefully "clipped". 
> We
> either don't display it at all or display it on the next screen 
> line,
> depending on the wrap mode.

I don't doubt its solvable, but how satisfactorily and at what 
complexity? I would say not displaying an element when some 
portion could be displayed is visually very unsatisfying. What if 
the element is large and 99% of it could be visible? What if the 
element is wider than the window? What if 
`window-resize-pixelwise` or `frame-resize-pixelwise` non-nil and 
the element is on the last, partially clipped line?

>> The direction I'm thinking of going in the future since I feel 
>> like thus far the xwidget route has been fairly troublesome is 
>> to 
>> expose interfaces to the cairo surfaces used in image.c. This 
>> could be through modifying image.c or through defining a new 
>> display element type. I would try to expose the cairo API to 
>> lisp 
>> and allow dynamic modules to directly draw to a cairo surface 
>> given to it through the lisp interface. Gtk along with many 
>> other 
>> libraries (such as librsvg and poppler) support drawing to a 
>> cairo 
>> surface and then Emacs gets to have full control over how that 
>> cario surface ends up on screen. I would see this as a stepping 
>> stone to attempting to do something similar with a opengl 
>> surface.
>
> My suggestion is to think how to integrate this idea into Emacs
> proper, without introducing modules into the equation.  Modules 
> add
> another layer of complexity, so I would suggest to design the 
> solution
> inside Emacs, and only after that see how to do that from a 
> module.
> One step at a time.

Of course the first step would be to work inside Emacs proper, and 
expose lisp interfaces. However I think designing such a feature 
should keep eventual integration with modules in mind. Especially 
since in general graphics is very performance sensitive and so 
doing some drawing in elisp just might not cut it and many 
interesting applications of such a feature would require 
interfacing with an external c library. This is all pretty 
hypothetical at this point though and as I keep learning with 
Emacs C code: its hard to speculate on the best way to do 
something without actually trying something out first.

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[martin rudalics]

 > This is solvable.  We already have similar situation in the display
 > engine, where some display element cannot be usefully "clipped".  We
 > either don't display it at all or display it on the next screen line,
 > depending on the wrap mode.

For some value of "solvable".  Emacs has no idea about how large the
external GTK toolbar gets so where would it wrap or clip it?

martin

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Eli Zaretskii]

> From: Akira Kyle <akira@akirakyle.com>
> Cc: larsi@gnus.org, emacs-devel@gnu.org
> Date: Sun, 29 Nov 2020 20:54:01 -0700
> 
> > Anyway, this just tells me that GTK is not a good starting point
> > for embedding widgets.  Do other projects build embedded widgets
> > on such shaky grounds?
> 
> I wouldn't say GTK isn't good for embedding widgets. In fact I'd 
> say it's probably great if you're wanting to extend an existing 
> GTK app with custom widgets. The problem is Emacs, despite using 
> GTK as a toolkit, isn't really a GTK app.

First, which of the problems you mentioned are related to Emacs not
being a GTK application?

And second, there's a branch in our repository where Emacs _is_ made
to be a GTK application, AFAIU.

> > This is solvable.  We already have similar situation in the
> > display engine, where some display element cannot be usefully
> > "clipped".  We either don't display it at all or display it on the
> > next screen line, depending on the wrap mode.
> 
> I don't doubt its solvable, but how satisfactorily and at what 
> complexity?

No complexity at all, it's almost trivial, because the display code
already does that.  As for "satisfactorily" part: what can be
unsatisfactory about displaying the widget on the next screen line?

> What if the element is large and 99% of it could be visible? What if
> the element is wider than the window?

These are marginal cases: you are talking about very large widgets
that in addition refuse to be resized.  How many are like that?  And
what else can we do with widgets that are (1) large, (2) won't resize,
and (3) cannot be clipped?

The perfect should not be the enemy of the good, right?

> What if `window-resize-pixelwise` or `frame-resize-pixelwise`
> non-nil and the element is on the last, partially clipped line?

We don't display it.  How is this different from a any other large
display element that cannot be shown in the viewport?

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Eli Zaretskii]

> Cc: larsi@gnus.org, emacs-devel@gnu.org
> From: martin rudalics <rudalics@gmx.at>
> Date: Mon, 30 Nov 2020 10:05:07 +0100
> 
>  > This is solvable.  We already have similar situation in the display
>  > engine, where some display element cannot be usefully "clipped".  We
>  > either don't display it at all or display it on the next screen line,
>  > depending on the wrap mode.
> 
> For some value of "solvable".  Emacs has no idea about how large the
> external GTK toolbar gets so where would it wrap or clip it?

That's a different matter: those decorations are not managed by the
Emacs display engine.

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Akira Kyle]

On Mon, Nov 30, 2020 at 08:39 AM, Eli Zaretskii <eliz@gnu.org> 
wrote:

>> > Anyway, this just tells me that GTK is not a good starting 
>> > point
>> > for embedding widgets.  Do other projects build embedded 
>> > widgets
>> > on such shaky grounds?
>> 
>> I wouldn't say GTK isn't good for embedding widgets. In fact 
>> I'd 
>> say it's probably great if you're wanting to extend an existing 
>> GTK app with custom widgets. The problem is Emacs, despite 
>> using 
>> GTK as a toolkit, isn't really a GTK app.
>
> First, which of the problems you mentioned are related to Emacs 
> not
> being a GTK application?

All of them.

> And second, there's a branch in our repository where Emacs _is_ 
> made
> to be a GTK application, AFAIU.

If you're referring to Yukki Harano's feature/pgtk branch, then 
that's actually what I'm primarily targeting in my development of 
emacs-webkit since the usual Emacs compiled `--with-x` has an 
additional problem from what I've previously described. It seems 
gtk widgets (or at least the webkit widget), flicker constantly on 
`--with-x`. It seems to be related to the double buffering 
implementation since setting the frame parameter 
`(inhibit-double-buffering . t)` fixes it. I haven't had the 
motivation to debug this due to the mess of xterm.c and because I 
see pgtk as the future of Emacs on GNU/Linux systems as X becomes 
increasingly obsolete.

AFAIU the pgtk version, like the ns version it was modeled after, 
doesn't implement everything in the main frame area of Emacs as 
widgets, only the tool and menu bars are implemented as proper 
widgets. The main frame area is treated as just one big canvas 
that redisplay works its magic on as it would if it were just a 
TUI.

>> > This is solvable.  We already have similar situation in the
>> > display engine, where some display element cannot be usefully
>> > "clipped".  We either don't display it at all or display it 
>> > on the
>> > next screen line, depending on the wrap mode.
>> 
>> I don't doubt its solvable, but how satisfactorily and at what 
>> complexity?
>
> No complexity at all, it's almost trivial, because the display 
> code
> already does that.  As for "satisfactorily" part: what can be
> unsatisfactory about displaying the widget on the next screen 
> line?

That's fine, and would be the expected behavior when lines are 
supposed to wrap. But when lines are not wrapped, I would not 
expect a display element to disappear as soon as one pixel of it 
exceeds the window border.

>> What if the element is large and 99% of it could be visible? 
>> What if
>> the element is wider than the window?
>
> These are marginal cases: you are talking about very large 
> widgets
> that in addition refuse to be resized.  How many are like that? 
> And
> what else can we do with widgets that are (1) large, (2) won't 
> resize,
> and (3) cannot be clipped?
>
> The perfect should not be the enemy of the good, right?

Take the drawing canvas idea, the canvas widget would be the size 
of an image. Already I find the way redisplay doesn't like to 
display images clipped at the top of the window jarring. If I 
wanted to zoom into the widget if it was displaying something 
detailed, I wouldn't want it to suddenly disappear. I think Emacs 
needs to be able to clip the elements its displays in order to not 
end up with what I would call surprising and frustrating 
behavior. Hence my original argument that the way gtk widgets are 
currently handled is not in general, a good path forward for such 
features.

>> What if `window-resize-pixelwise` or `frame-resize-pixelwise`
>> non-nil and the element is on the last, partially clipped line?
>
> We don't display it.  How is this different from a any other 
> large
> display element that cannot be shown in the viewport?

Images are the only other large display element that I'm aware of 
and images are clipped in such cases, which is good as large 
images make their line very tall so very often they end up 
partially displayed in the last line.

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Eli Zaretskii]

> From: Akira Kyle <akira@akirakyle.com>
> Cc: larsi@gnus.org, emacs-devel@gnu.org
> Date: Mon, 30 Nov 2020 10:03:55 -0700
> 
> > No complexity at all, it's almost trivial, because the display
> > code already does that.  As for "satisfactorily" part: what can be
> > unsatisfactory about displaying the widget on the next screen
> > line?
> 
> That's fine, and would be the expected behavior when lines are 
> supposed to wrap. But when lines are not wrapped, I would not 
> expect a display element to disappear as soon as one pixel of it 
> exceeds the window border.

And yet that's exactly what happens with other display elements, for
example wide characters on TTY display.

Again, AFAIU we are talking about rare use cases, which you said
cannot be displayed in any other way, so what do you expect Emacs to
do in such restrictive situations??

> Already I find the way redisplay doesn't like to display images
> clipped at the top of the window jarring.

Not sure what you are talking about, but I'm sure whatever is jarring
can be fixed by relatively simple augmentations to the display code.

> If I wanted to zoom into the widget if it was displaying something
> detailed, I wouldn't want it to suddenly disappear.

It doesn't have to, we can scroll the window instead, so that the
enlarged widget is brought into the viewport.

Again, what else do you expect Emacs to do when you yourself said
these widgets don't allow any other solution??

> I think Emacs needs to be able to clip the elements its displays in
> order to not end up with what I would call surprising and
> frustrating behavior.

But you yourself said this is not allowed in these cases, so how can
Emacs do what is not allowed??

> >> What if `window-resize-pixelwise` or `frame-resize-pixelwise`
> >> non-nil and the element is on the last, partially clipped line?
> >
> > We don't display it.  How is this different from a any other large
> > display element that cannot be shown in the viewport?
> 
> Images are the only other large display element that I'm aware of 
> and images are clipped in such cases

But you yourself said that clipping these widgets is impossible.  What
am I missing in this discussion?

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Akira Kyle]

On Mon, Nov 30, 2020 at 11:11 AM, Eli Zaretskii <eliz@gnu.org> 
wrote:

>> > No complexity at all, it's almost trivial, because the 
>> > display
>> > code already does that.  As for "satisfactorily" part: what 
>> > can be
>> > unsatisfactory about displaying the widget on the next screen
>> > line?
>> 
>> That's fine, and would be the expected behavior when lines are 
>> supposed to wrap. But when lines are not wrapped, I would not 
>> expect a display element to disappear as soon as one pixel of 
>> it 
>> exceeds the window border.
>
> And yet that's exactly what happens with other display elements, 
> for
> example wide characters on TTY display.
>
> Again, AFAIU we are talking about rare use cases, which you said
> cannot be displayed in any other way, so what do you expect 
> Emacs to
> do in such restrictive situations??
>
>> Already I find the way redisplay doesn't like to display images
>> clipped at the top of the window jarring.
>
> Not sure what you are talking about, but I'm sure whatever is 
> jarring
> can be fixed by relatively simple augmentations to the display 
> code.
>
>> If I wanted to zoom into the widget if it was displaying 
>> something
>> detailed, I wouldn't want it to suddenly disappear.
>
> It doesn't have to, we can scroll the window instead, so that 
> the
> enlarged widget is brought into the viewport.
>
> Again, what else do you expect Emacs to do when you yourself 
> said
> these widgets don't allow any other solution??
>
>> I think Emacs needs to be able to clip the elements its 
>> displays in
>> order to not end up with what I would call surprising and
>> frustrating behavior.
>
> But you yourself said this is not allowed in these cases, so how 
> can
> Emacs do what is not allowed??
>
>> >> What if `window-resize-pixelwise` or 
>> >> `frame-resize-pixelwise`
>> >> non-nil and the element is on the last, partially clipped 
>> >> line?
>> >
>> > We don't display it.  How is this different from a any other 
>> > large
>> > display element that cannot be shown in the viewport?
>> 
>> Images are the only other large display element that I'm aware 
>> of 
>> and images are clipped in such cases
>
> But you yourself said that clipping these widgets is impossible. 
> What
> am I missing in this discussion?

Sorry we've gotten a bit off the original point. I was trying to 
illustrate that for such "rich" display elements clipping would 
seem to be a necessity in order to avoid surprising or frustrating 
behavior. I understand Emacs can work around an element that 
doesn't permit being clipped, but it will be exactly that -- a 
work around to a problem Emacs shouldn't have solve. Thus clipping 
has to happen on the GTK side, which isn't impossible, but 
requires additional complexity. Overriding Widgets so they can be 
clipped is yet another example of trying to draw GTK widgets in 
ways they weren't intended to be drawn (and the GTK folks would 
probably consider this all very bad practice).

My whole point is that I've become more convinced that trying to 
integrate GTK widgets into Emacs redisplay isn't a good way to 
achieve the desired functionality and it would be worth exploring 
other, more portable solutions. I don't think we need to continue 
discussing the finer points on clipping.

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Tomas Hlavaty]

On Mon 30 Nov 2020 at 10:03, Akira Kyle <akira@akirakyle.com> wrote:
> and because I see pgtk as the future of Emacs on GNU/Linux systems as
> X becomes increasingly obsolete.

I am not sure such future is bright.

In that future, is there no way to do graphics without a widget toolkit?

> AFAIU the pgtk version, like the ns version it was modeled after, 
> doesn't implement everything in the main frame area of Emacs as 
> widgets, only the tool and menu bars are implemented as proper 
> widgets.

Also popup menus that pop up on a mouse click are gtk iirc.

And maybe scrollbars?

It may have changed since I turned these distractions off a long time
ago.

I think that they are also fundamentally flawed: How can I search in
menu?  Maybe it should be a buffer.  New Gtk hides menu under an
unintuitive button anyway so this could just display searchable Menu
buffer.

> The main frame area is treated as just one big canvas that redisplay
> works its magic on as it would if it were just a TUI.

Maybe that is a good thing.

Widget toolkit dependency is a heavy price to pay for.

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Tomas Hlavaty]

On Sun 29 Nov 2020 at 20:54, Akira Kyle <akira@akirakyle.com> wrote:
> However I think designing such a feature should keep eventual
> integration with modules in mind. Especially since in general graphics
> is very performance sensitive and so doing some drawing in elisp just
> might not cut it and many interesting applications of such a feature
> would require interfacing with an external c library.

Not necessarily.

There could also be a specialized program which could handle drawing.
In fact there already is such a program: w3mimgdisplay and is part of
the w3m web browser and pager.  It is also used in the file manager
ranger.  It works both with or without X.  I am using it in
emacs-framebuffer too to display images and documents in Emacs.

The issues I am facing have nothing to do with modules or libraries.

For example:

It would be more convenient, if there was a way to specify elisp
function to draw an image.  By default, it could just call the existing
C code but this would also allow me to specify a different elisp
function which would then for example call w3mimgdisplay.

It would be more convenient, if an image was represented as elisp data
instead of C data.  iirc there is no way to add new image types without
touching C.

Is there a way to turn off cursor for a buffer?  (Blinking cursor
disrupts the drawn image.)

Is there a way run a function after emacs changes something on the
screen in a buffer?

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Arthur Miller]

Tomas Hlavaty <tom@logand.com> writes:

> On Sun 29 Nov 2020 at 20:54, Akira Kyle <akira@akirakyle.com> wrote:
>> However I think designing such a feature should keep eventual
>> integration with modules in mind. Especially since in general graphics
>> is very performance sensitive and so doing some drawing in elisp just
>> might not cut it and many interesting applications of such a feature
>> would require interfacing with an external c library.
>
> Not necessarily.
>
> There could also be a specialized program which could handle drawing.
> In fact there already is such a program: w3mimgdisplay and is part of
> the w3m web browser and pager.  It is also used in the file manager
> ranger.  It works both with or without X.  I am using it in
> emacs-framebuffer too to display images and documents in Emacs.
>
> The issues I am facing have nothing to do with modules or libraries.
>
> For example:
>
> It would be more convenient, if there was a way to specify elisp
> function to draw an image.  By default, it could just call the existing
> C code but this would also allow me to specify a different elisp
> function which would then for example call w3mimgdisplay.
>
> It would be more convenient, if an image was represented as elisp data
> instead of C data.
:-) I don't think you have thought well about it;

but really nothing forbids you to try to reprsent images as lisp.

You can take any de-compressed image and read in raw pixels in as a byte
buffer and just shuffle around numbers and see how it
works for you.Image is nothing but a bunch of numbers + some tiny
metadata on top of it. Take some of netpbm formats and you have "textual
image" you can manipulate on per-pixel level with lisp.

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Eli Zaretskii]

> From: Tomas Hlavaty <tom@logand.com>
> Date: Tue, 01 Dec 2020 09:01:20 +0100
> 
> It would be more convenient, if there was a way to specify elisp
> function to draw an image.  By default, it could just call the existing
> C code but this would also allow me to specify a different elisp
> function which would then for example call w3mimgdisplay.

You need to be aware of how this works in Emacs.  In a nutshell,
there's no function "to draw the image": we call system APIs to do
that.  By the time we need to draw the image, it is already
preprocessed into a bitmap/pixmap, so we basically ask the GUI
subsystem of the OS to bitblt the image to some coordinates on the
screen which we compute.

The preprocessing of the images into bitmap/pixmap happens when the
display engine sees a 'display' property that specifies an image.
At that time we call some image library to convert the image's data
into bitmap/pixmap; then we cache the result and produce a handle to
the cache that the drawing routines will use.

So when you ask for the above, you need to fit into this scheme.

> It would be more convenient, if an image was represented as elisp data
> instead of C data.

create-image already knows how to accept image data.

> iirc there is no way to add new image types without touching C.

No, because you need to know how to convert that into the internal
format used to draw the image, which is platform dependent.

Why is it a problem to use one of the known image types?  E.g., SVG
(via the svg.el library).

> Is there a way to turn off cursor for a buffer?

There's the cursor-type variable.

> Is there a way run a function after emacs changes something on the
> screen in a buffer?

That depends by what you mean by "after emacs changes something on the
screen", since in Emacs redisplay is independent/separate of data
changes.  Please elaborate.

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Tomas Hlavaty]

On Tue 01 Dec 2020 at 16:36, Arthur Miller <arthur.miller@live.com> wrote:
> Tomas Hlavaty <tom@logand.com> writes:
>> It would be more convenient, if an image was represented as elisp data
>> instead of C data.
> :-) I don't think you have thought well about it;

Why would you say that?

> but really nothing forbids you to try to reprsent images as lisp.

What do you mean exactly?

> You can take any de-compressed image and read in raw pixels in as a byte
> buffer and just shuffle around numbers and see how it
> works for you.Image is nothing but a bunch of numbers + some tiny
> metadata on top of it. Take some of netpbm formats and you have "textual
> image" you can manipulate on per-pixel level with lisp.

Image in emacs is represented in C as "struct image".

Image type in emacs is represented in C as "struct image_type".

The issue is that there doesn't seem to be a way to extend those (and/or
define new) in pure elisp.

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Tomas Hlavaty]

On Tue 01 Dec 2020 at 17:58, Eli Zaretskii <eliz@gnu.org> wrote:
>> From: Tomas Hlavaty <tom@logand.com>
>> Date: Tue, 01 Dec 2020 09:01:20 +0100
>> 
>> It would be more convenient, if there was a way to specify elisp
>> function to draw an image.  By default, it could just call the existing
>> C code but this would also allow me to specify a different elisp
>> function which would then for example call w3mimgdisplay.
>
> You need to be aware of how this works in Emacs.  In a nutshell,
> there's no function "to draw the image": we call system APIs to do
> that.  By the time we need to draw the image, it is already
> preprocessed into a bitmap/pixmap, so we basically ask the GUI
> subsystem of the OS to bitblt the image to some coordinates on the
> screen which we compute.
>
> The preprocessing of the images into bitmap/pixmap happens when the
> display engine sees a 'display' property that specifies an image.
> At that time we call some image library to convert the image's data
> into bitmap/pixmap; then we cache the result and produce a handle to
> the cache that the drawing routines will use.
>
> So when you ask for the above, you need to fit into this scheme.

Thanks for the great explanation.

I do not have an issue with the way it works currently.

Think of w3mimgdisplay program

- as an alternatine "system API and GUI subsystem of the OS to bitblt
  the image to some coordinates on the screen which we compute".

- as an alternative "some image library to convert the image's data into
  bitmap/pixmap; then we cache the result and produce a handle to the
  cache that the drawing routines will use".

w3mimgdisplay can supply both functions.

w3mimgdisplay program can be run as a kind of "server", where I can say:

- draw this image to these coordinates on the screen

- or additionally, cache image's data and return a handle to the cache,
  which will make redrawing fast.

I think that the only issue for me is that there is no way to implement
an alternative "system API" and "call some image library" in pure elisp.

How much work would it be to enable alternative "system API" and "call
some image library" in pure elisp?

>> It would be more convenient, if an image was represented as elisp data
>> instead of C data.
>
> create-image already knows how to accept image data.

But those must be defined in C, see lookup_image_type.

Or is there a way to define custom image format in pure elisp?

It would be actually enough for me if my custom image was opaque to
emacs (except probably some attributes like width and height) and
w3mimgdisplay handled the rest.

>> iirc there is no way to add new image types without touching C.
>
> No, because you need to know how to convert that into the internal
> format used to draw the image, which is platform dependent.

I don't need to know that.  w3mimgdisplay program does that.  As far as
emacs is concerned, it needs to give to w3mimgdisplay filename to draw
and then use a handle to cache to ask w3mimgdisplay for redraw.

> Why is it a problem to use one of the known image types?  E.g., SVG
> (via the svg.el library).

Because those those are not universally available, e.g. on the console.

Because those might not be suitable.

>> Is there a way to turn off cursor for a buffer?
>
> There's the cursor-type variable.

Thanks.

This does not work on the console.

I suppose I need to use some console specific command/escape sequence.

Is there a way to hook a function when a buffer looses and gains focus?

>> Is there a way run a function after emacs changes something on the
>> screen in a buffer?
>
> That depends by what you mean by "after emacs changes something on the
> screen", since in Emacs redisplay is independent/separate of data
> changes.  Please elaborate.

Example: I am on the console.  I am in dired buffer.  I press my key.
w3mimgdisplay program draws my image on the screen on the specified
coordinates "over" emacs.  All is well (except the blinking cursor which
we can ignore for now).  Now I press another key to display the next
image in the dired buffer.  Emacs redraws some characters, that have
changed.  This disrupts the image drawn earlier and I need to detect,
that I need to tell w3mimgdisplay to redraw the image.  I need to hook
my redraw image function after emacs is done with its drawing.

So far I have to press a key to redraw image.

I tried using timer and redraw after say 1s but that is not so nice.

Is there such hook?  If not, would it be difficult to add it?

Thanks a lot for your help!

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Eli Zaretskii]

> From: Tomas Hlavaty <tom@logand.com>
> Date: Tue, 01 Dec 2020 18:33:56 +0100
> 
> w3mimgdisplay program can be run as a kind of "server", where I can say:
> 
> - draw this image to these coordinates on the screen
> 
> - or additionally, cache image's data and return a handle to the cache,
>   which will make redrawing fast.

How would Emacs communicate with w3mimgdisplay? via some pipe? is that
fast enough to be done in the middle of redisplay?

> I think that the only issue for me is that there is no way to implement
> an alternative "system API" and "call some image library" in pure elisp.

Why is that an issue?  And what does an implementation in pure Lisp
have to do with w3mimgdisplay?

> How much work would it be to enable alternative "system API" and "call
> some image library" in pure elisp?

If they support the same interfaces, very little.  But I'm not sure we
are on the same page yet, so we might be miscommunicating.

> >> It would be more convenient, if an image was represented as elisp data
> >> instead of C data.
> >
> > create-image already knows how to accept image data.
> 
> But those must be defined in C, see lookup_image_type.

You are supposed to submit data in one of the known formats.  Why is
that a problem? why do you care about the exact format of the image
data, and why do you insist to introduce a new type instead of using
existing types?

> Or is there a way to define custom image format in pure elisp?

No.

> It would be actually enough for me if my custom image was opaque to
> emacs (except probably some attributes like width and height) and
> w3mimgdisplay handled the rest.

That's okay, but I still don't see how all that will fit into the
current framework of displaying images.

> >> iirc there is no way to add new image types without touching C.
> >
> > No, because you need to know how to convert that into the internal
> > format used to draw the image, which is platform dependent.
> 
> I don't need to know that.  w3mimgdisplay program does that.

How can w3mimgdisplay know the internal format used by Emacs for
storing images?

> As far as emacs is concerned, it needs to give to w3mimgdisplay
> filename to draw and then use a handle to cache to ask w3mimgdisplay
> for redraw.

I suggest to take a look at x_draw_image_foreground and its
subroutines, this is where we actually put the image on the glass.
Then tell in a bit more detail how to integrate an external program
into that, keeping all the features we support in this regard, like
masks, clipping, slices, etc.

> > Why is it a problem to use one of the known image types?  E.g., SVG
> > (via the svg.el library).
> 
> Because those those are not universally available, e.g. on the console.

On the console you currently cannot have images in Emacs, period.
This is not because of image libraries, this is because the text-mode
display simply cannot currently support such display elements.

> Because those might not be suitable.

"Not suitable" in what ways?

> >> Is there a way to turn off cursor for a buffer?
> >
> > There's the cursor-type variable.
> 
> Thanks.
> 
> This does not work on the console.

(You never said you were talking about a console.)  We currently don't
expose console cursor control to Lisp.  Should be easy to add that, if
needed.

> Is there a way to hook a function when a buffer looses and gains focus?

Buffer cannot have focus.  Do you mean when the buffer's window is the
selected window?  There's window-selection-change-functions.

> >> Is there a way run a function after emacs changes something on the
> >> screen in a buffer?
> >
> > That depends by what you mean by "after emacs changes something on the
> > screen", since in Emacs redisplay is independent/separate of data
> > changes.  Please elaborate.
> 
> Example: I am on the console.  I am in dired buffer.  I press my key.
> w3mimgdisplay program draws my image on the screen on the specified
> coordinates "over" emacs.  All is well (except the blinking cursor which
> we can ignore for now).  Now I press another key to display the next
> image in the dired buffer.  Emacs redraws some characters, that have
> changed.  This disrupts the image drawn earlier and I need to detect,
> that I need to tell w3mimgdisplay to redraw the image.  I need to hook
> my redraw image function after emacs is done with its drawing.

I don't think we have such a hook, no.  But if we had it, using it
like you want would cause annoying flickering, because you'd
constantly redraw the image at very high frequency.

> Is there such hook?  If not, would it be difficult to add it?

It won't be hard to add it, but I'm not sure we want to add such a
hook, see above.

RE: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[arthur miller]

[-- Attachment #1: Type: text/plain, Size: 2562 bytes --]

"> Tomas Hlavaty <tom@logand.com> writes:
>> It would be more convenient, if an image was represented as elisp data
>> instead of C data.
> :-) I don't think you have thought well about it;

Why would you say that?"

Because of the other things you wrote before that citate about drawing images in external process. You have expressed your sentiments about year 2020, efficiency and external processes a day or two ago, and I read similar reasoning here.

If all you wish is just to display an image, sure you can do that with external application, but if you would like to perform something interesting with images, or display something many images, the performance would drop quite fast.

Since you refer to drawing images and mention image data, I assumed you are talking about real image data, i.e. pixels, not handles to lisp objects. With other words I thought you would like to use Lisp to do something interesting as processing images in Emacs on pixel level. When people mention image data that is usually what they mean. But assumptions are always fault on the one that assumes, so I apologize for that.

Otherwise, you could surely easily integrate netpbm, image/openmagic or even libgd (via python or perl wrapper) into Emacs via pure lisp and processes. I don't see what's the problem if that is what you wish.

-------- Originalmeddelande --------
Från: Tomas Hlavaty <tom@logand.com>
Datum: 2020-12-01 17:49 (GMT+01:00)
Till: emacs-devel@gnu.org
Ämne: Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

On Tue 01 Dec 2020 at 16:36, Arthur Miller <arthur.miller@live.com> wrote:
> Tomas Hlavaty <tom@logand.com> writes:
>> It would be more convenient, if an image was represented as elisp data
>> instead of C data.
> :-) I don't think you have thought well about it;

Why would you say that?

> but really nothing forbids you to try to reprsent images as lisp.

What do you mean exactly?

> You can take any de-compressed image and read in raw pixels in as a byte
> buffer and just shuffle around numbers and see how it
> works for you.Image is nothing but a bunch of numbers + some tiny
> metadata on top of it. Take some of netpbm formats and you have "textual
> image" you can manipulate on per-pixel level with lisp.

Image in emacs is represented in C as "struct image".

Image type in emacs is represented in C as "struct image_type".

The issue is that there doesn't seem to be a way to extend those (and/or
define new) in pure elisp.


[-- Attachment #2: Type: text/html, Size: 3451 bytes --]

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Tomas Hlavaty]

On Tue 01 Dec 2020 at 21:41, Eli Zaretskii <eliz@gnu.org> wrote:
> How would Emacs communicate with w3mimgdisplay? via some pipe?

Yes, pipe.

At the moment, in emacs-framebuffer, I just call it to display an image
using call-process-region.  It could be optimized with pipe, but this
has low priority for emacs-framebuffer as it is fast enough for my
needs.

If it was properly integrated with emacs at the level of image.el, then
pipe would be more optimized implementation, the way w3mimgdisplay is
actually meant to be used.

It is meant to be used this way: w3m browser spawns w3mimgdisplay and
talks to it via pipe.  w3m tells w3mimgdisplay to cache some images.
w3mimgdisplay tells w3m sizes of the images, w3m leaves space for the
images on the screen, draws text and then tells w3mimgdisplay to draw
the cached images in the right places on the screen.

> is that fast enough to be done in the middle of redisplay?

I am not exatly sure, what the concern is here.

The images do not travel via pipe.  Only meta-data like filenames,
coordinates and handles.

What exactly needs to be done in the middle of redisplay that is
relevant to drawing images?  Determining image width and height?
Something else like actually drawing the image?

The actual drawing could be postponed after Emacs has drawn its text.

>> I think that the only issue for me is that there is no way to
>> implement an alternative "system API" and "call some image library"
>> in pure elisp.
>
> Why is that an issue?

Because doing it in C requires significantly more effort and my free
time is limited.

> And what does an implementation in pure Lisp have to do with
> w3mimgdisplay?

Nothing.

Except the point above.

>> How much work would it be to enable alternative "system API" and
>> "call some image library" in pure elisp?
>
> If they support the same interfaces, very little.  But I'm not sure we
> are on the same page yet, so we might be miscommunicating.

What are the current interfaces?

>> >> It would be more convenient, if an image was represented as elisp
>> >> data instead of C data.
>> >
>> > create-image already knows how to accept image data.
>> 
>> But those must be defined in C, see lookup_image_type.
>
> You are supposed to submit data in one of the known formats.  Why is
> that a problem?

Because they do not work on the console.

And the reason they do not work on the console is because there is
hardcoded dependency on image drawing libraries which do not work on the
console.

I want Emacs to understand images even on the console.  But I have no
enough free time to do it in C.  If there was elisp level boundary
between image object and something that draws that image object, it
would be much easier.  I could just specify my own thing that draws that
image object without going so low level.  And it would work with any
existing elisp code like image.el transparently.

> why do you care about the exact format of the image data, and why do
> you insist to introduce a new type instead of using existing types?

I do not care about the exact format of the image data.

I do not care about the image data at all.

I care about extending Emacs' understanding of images in elisp at
meta-data level, not image data level.  For example, an image has kind
of type or format like png or jpeg, it has width and height, maybe some
other properties and some kind of opaque data blob.  I would like to be
able to specify and/or override how to draw images in elisp and not
having that hardcoded in C.

Additionally there is no reason why documents like pdf, epub, odt etc
could not be an image (which they basically are).  They have pages but
tiff also has more than one image.  There is also no reason why image
modes could not have similar capabilities to doc-view or pdf-tools.  All
this could be eventually unified.

>> Or is there a way to define custom image format in pure elisp?
>
> No.

That is a shame.

>> It would be actually enough for me if my custom image was opaque to
>> emacs (except probably some attributes like width and height) and
>> w3mimgdisplay handled the rest.
>
> That's okay, but I still don't see how all that will fit into the
> current framework of displaying images.

Maybe I do not need custom image type.

But I need to change the way images are drawn on the console, i.e. not
at all vs draw the image.

But the way image types are implemented currently does not allow me to
do that in elisp.

>> >> iirc there is no way to add new image types without touching C.
>> >
>> > No, because you need to know how to convert that into the internal
>> > format used to draw the image, which is platform dependent.
>> 
>> I don't need to know that.  w3mimgdisplay program does that.
>
> How can w3mimgdisplay know the internal format used by Emacs for
> storing images?

Why would w3mimgdisplay need to know that?

w3mimgdisplay just needs filename and coordinates.

>> As far as emacs is concerned, it needs to give to w3mimgdisplay
>> filename to draw and then use a handle to cache to ask w3mimgdisplay
>> for redraw.
>
> I suggest to take a look at x_draw_image_foreground and its
> subroutines, this is where we actually put the image on the glass.
> Then tell in a bit more detail how to integrate an external program
> into that, keeping all the features we support in this regard, like
> masks, clipping, slices, etc.

Thanks, I'll have a look.

>> > Why is it a problem to use one of the known image types?  E.g., SVG
>> > (via the svg.el library).
>> 
>> Because those those are not universally available, e.g. on the
>> console.
>
> On the console you currently cannot have images in Emacs, period.

Exactly.  And I want this to change.  There is no reason for such
restriction.

> This is not because of image libraries, this is because the text-mode
> display simply cannot currently support such display elements.

This is not an issue at all.  Capabilities of text-mode display are
irrelevant.  w3mimgdisplay is using framebuffer.  I do not know how to
better communicate that.

Try:

   $ w3m https://gnu.org

You will see text-mode HTML browser layout with images on the console.

>> >> Is there a way to turn off cursor for a buffer?
>> >
>> > There's the cursor-type variable.
>> 
>> Thanks.
>> 
>> This does not work on the console.
>
> (You never said you were talking about a console.)  We currently don't
> expose console cursor control to Lisp.  Should be easy to add that, if
> needed.

I see.

cursor-type does not say it does not work on the console.

I think not all values are doable and relevant.  t and nil should be
enough.

Shell I open a bug report for this?

>> Is there a way to hook a function when a buffer looses and gains
>> focus?
>
> Buffer cannot have focus.  Do you mean when the buffer's window is the
> selected window?  There's window-selection-change-functions.

Thanks, I'll look into this.

>> >> Is there a way run a function after emacs changes something on the
>> >> screen in a buffer?
>> >
>> > That depends by what you mean by "after emacs changes something on
>> > the screen", since in Emacs redisplay is independent/separate of
>> > data changes.  Please elaborate.
>> 
>> Example: I am on the console.  I am in dired buffer.  I press my key.
>> w3mimgdisplay program draws my image on the screen on the specified
>> coordinates "over" emacs.  All is well (except the blinking cursor
>> which we can ignore for now).  Now I press another key to display the
>> next image in the dired buffer.  Emacs redraws some characters, that
>> have changed.  This disrupts the image drawn earlier and I need to
>> detect, that I need to tell w3mimgdisplay to redraw the image.  I
>> need to hook my redraw image function after emacs is done with its
>> drawing.
>
> I don't think we have such a hook, no.  But if we had it, using it
> like you want would cause annoying flickering, because you'd
> constantly redraw the image at very high frequency.

I do not think it would cause flickering on the console.  The only thing
that normally "flickers" is the blinking cursor.  Emacs does not draw
anything if nothing changed.

>> Is there such hook?  If not, would it be difficult to add it?
>
> It won't be hard to add it, but I'm not sure we want to add such a
> hook, see above.

That is a shame.  If there was a hook to detect when Emacs has drawn
something on the console, I could plug image redrawing there.

If I wanted to add it, where in the code should I look?

Thanks for your help!

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Akira Kyle]

On Tue, Dec 01, 2020 at 12:44 AM, Tomas Hlavaty <tom@logand.com> 
wrote:

> On Mon 30 Nov 2020 at 10:03, Akira Kyle <akira@akirakyle.com> 
> wrote:
>> and because I see pgtk as the future of Emacs on GNU/Linux 
>> systems as
>> X becomes increasingly obsolete.
>
> I am not sure such future is bright.
>
> In that future, is there no way to do graphics without a widget 
> toolkit?

Sure there is. Under wayland you can just dump pixels into a 
shared buffer to render them to the screen or use EGL to render 
using a GPU. The difference from X is that wayland offers no 
convince functions to actually draw pixels into such a buffer, its 
up to the client to do so. In order to not reinvent the wheel, its 
much more convenient to use a toolkit to draw the elements it's 
good at drawing. 

>> AFAIU the pgtk version, like the ns version it was modeled 
>> after, 
>> doesn't implement everything in the main frame area of Emacs as 
>> widgets, only the tool and menu bars are implemented as proper 
>> widgets.
>
> Also popup menus that pop up on a mouse click are gtk iirc.
>
> And maybe scrollbars?
>
> It may have changed since I turned these distractions off a long 
> time
> ago.

I think this is really ultimately a personal preference.

> I think that they are also fundamentally flawed: How can I 
> search in
> menu?  Maybe it should be a buffer.  New Gtk hides menu under an
> unintuitive button anyway so this could just display searchable 
> Menu
> buffer.

I think it would be possible to implement such menus without GTK 
using just child frames.

>> The main frame area is treated as just one big canvas that 
>> redisplay
>> works its magic on as it would if it were just a TUI.
>
> Maybe that is a good thing.
>
> Widget toolkit dependency is a heavy price to pay for.

I don't think it's a "heavy price to pay" in general. It greatly 
simplifies many applications, which if they were to forego a 
toolkit, would likely have codebases comparable to GTK or 
Emacs. But its true that Emacs is far more decoupled, for better 
or worse, from toolkits than your average GUI app.

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Akira Kyle]

On Tue, Dec 01, 2020 at 01:01 AM, Tomas Hlavaty <tom@logand.com> 
wrote:

> On Sun 29 Nov 2020 at 20:54, Akira Kyle <akira@akirakyle.com> 
> wrote:
>> However I think designing such a feature should keep eventual
>> integration with modules in mind. Especially since in general 
>> graphics
>> is very performance sensitive and so doing some drawing in 
>> elisp just
>> might not cut it and many interesting applications of such a 
>> feature
>> would require interfacing with an external c library.
>
> Not necessarily.
>
> There could also be a specialized program which could handle 
> drawing.
> In fact there already is such a program: w3mimgdisplay and is 
> part of
> the w3m web browser and pager.  It is also used in the file 
> manager
> ranger.  It works both with or without X.  I am using it in
> emacs-framebuffer too to display images and documents in Emacs.
>
> The issues I am facing have nothing to do with modules or 
> libraries.
>
> For example:
>
> It would be more convenient, if there was a way to specify elisp
> function to draw an image.  By default, it could just call the 
> existing
> C code but this would also allow me to specify a different elisp
> function which would then for example call w3mimgdisplay.
>
> It would be more convenient, if an image was represented as 
> elisp data
> instead of C data.  iirc there is no way to add new image types 
> without
> touching C.
>
> Is there a way to turn off cursor for a buffer?  (Blinking 
> cursor
> disrupts the drawn image.)
>
> Is there a way run a function after emacs changes something on 
> the
> screen in a buffer?

I just briefly poked around w3m's codebase and it appears that the 
code paths for X and direct framebuffer are almost entirely 
independent. Its not totally clear what your ultimate goal is with 
respect to w3m's image display and emacs. Do you wish to never 
launch display server such as X or wayland but still have some of 
Emacs graphics support on the linux framebuffer? It seems like 
unless you're going to teach Emacs to entirely draw itself to the 
framebuffer, there will always be glitches like the cursor messing 
up images since TUI Emacs won't be aware of what pixels of the 
framebuffer have been drawn over independently of the linux VT 
displaying lines of Emacs' characters.

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Tomas Hlavaty]

On Wed 02 Dec 2020 at 17:24, Akira Kyle <akira@akirakyle.com> wrote:
> On Tue, Dec 01, 2020 at 12:44 AM, Tomas Hlavaty <tom@logand.com> 
> wrote:
>
>> On Mon 30 Nov 2020 at 10:03, Akira Kyle <akira@akirakyle.com> 
>> wrote:
>>> and because I see pgtk as the future of Emacs on GNU/Linux systems
>>> as X becomes increasingly obsolete.
>>
>> I am not sure such future is bright.
>>
>> In that future, is there no way to do graphics without a widget 
>> toolkit?
>
> Sure there is. Under wayland you can just dump pixels into a 
> shared buffer to render them to the screen or use EGL to render 
> using a GPU. The difference from X is that wayland offers no 
> convince functions to actually draw pixels into such a buffer, its 
> up to the client to do so. In order to not reinvent the wheel, its 
> much more convenient to use a toolkit to draw the elements it's 
> good at drawing.

"just dump pixels" and "no function to actually do that" and "use
toolkit" does seem to contradict the point of being able to do it
without toolkit.

How long will pgtk future last?

Compare that kind of future with for example ansi terminal future.
Which one do you think has longer future ahead?  Why?

>>> AFAIU the pgtk version, like the ns version it was modeled after,
>>> doesn't implement everything in the main frame area of Emacs as
>>> widgets, only the tool and menu bars are implemented as proper
>>> widgets.
>>
>> Also popup menus that pop up on a mouse click are gtk iirc.
>>
>> And maybe scrollbars?
>>
>> It may have changed since I turned these distractions off a long 
>> time
>> ago.
>
> I think this is really ultimately a personal preference.

It was not my personal preference to implement popup menus and
scrollbars as proper widgets.

>> I think that they are also fundamentally flawed: How can I search in
>> menu?  Maybe it should be a buffer.  New Gtk hides menu under an
>> unintuitive button anyway so this could just display searchable Menu
>> buffer.
>
> I think it would be possible to implement such menus without GTK using
> just child frames.

Why not also implement the rest without GTK?

>>> The main frame area is treated as just one big canvas that redisplay
>>> works its magic on as it would if it were just a TUI.
>>
>> Maybe that is a good thing.
>>
>> Widget toolkit dependency is a heavy price to pay for.
>
> I don't think it's a "heavy price to pay" in general. It greatly
> simplifies many applications, which if they were to forego a toolkit,
> would likely have codebases comparable to GTK or Emacs. But its true
> that Emacs is far more decoupled, for better or worse, from toolkits
> than your average GUI app.

Emacs already has to forego a toolkit, doesn't it?  Why?  Because it has
to work portably without any particular toolkit or with all chosen
toolkits.

It does not simplify Emacs I think, on the contrary.

Would not it be better to extend the canvas idea you mentioned above to
allow graphics in general, portable, toolkit independent way?

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Tomas Hlavaty]

On Wed 02 Dec 2020 at 17:37, Akira Kyle <akira@akirakyle.com> wrote:
> Do you wish to never launch display server such as X or wayland but
> still have some of Emacs graphics support on the linux framebuffer?

yes

> It seems like unless you're going to teach Emacs to entirely draw
> itself to the framebuffer, there will always be glitches like the
> cursor messing up images since TUI Emacs won't be aware of what pixels
> of the framebuffer have been drawn over independently of the linux VT
> displaying lines of Emacs' characters.

The cursor could be turned off.

Are there other glitches which cannot be solved?

Try

   $ w3m https://gnu.org

You'll see graphics in text only web browser.  It works even with cursor
on.

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Eli Zaretskii]

> From: Tomas Hlavaty <tom@logand.com>
> Cc: emacs-devel@gnu.org
> Date: Wed, 02 Dec 2020 21:37:08 +0100

It turns out we've been mis-communicating all the time, since I didn't
realize you were interested only in displaying images on TTY frames.
So please forget (almost) everything I said, because most of it is not
relevant.  Below I answer only the questions that are relevant to your
goal.

> At the moment, in emacs-framebuffer, I just call it to display an image
> using call-process-region.  It could be optimized with pipe, but this
> has low priority for emacs-framebuffer as it is fast enough for my
> needs.

IMNSHO, neither call-process-region nor pipes are fast enough to be an
integral part of the Emacs redisplay: they are too slow.  Especially
if you want to be able to display more than one image at the same
time.  To see how the performance is inadequate, turn on flyspell-mode
and scroll through a large text buffer, preferably with lots of
mis-spelled words (or some text full of special terminology the
general-purpose dictionary won't know about).  The effect in your case
will be very similar.

> I am not exatly sure, what the concern is here.
> 
> The images do not travel via pipe.  Only meta-data like filenames,
> coordinates and handles.

Yes, I understood that much.  But that doesn't matter: communicating
with a spell-checker in the flyspell-mode case also sends just a small
number of words in each direction, and yet the slow-down is
considerable, sometimes downright annoying.

> What exactly needs to be done in the middle of redisplay that is
> relevant to drawing images?  Determining image width and height?

Sending the image spec and receiving the dimensions and other image
attributes, yes.  And I presume that the sub-process needs some time
to calculate the answer (because at least some queries require to load
and process an image).

> > You are supposed to submit data in one of the known formats.  Why is
> > that a problem?
> 
> Because they do not work on the console.
> 
> And the reason they do not work on the console is because there is
> hardcoded dependency on image drawing libraries which do not work on the
> console.

No, that's not the main reason.  See below.

> > On the console you currently cannot have images in Emacs, period.
> 
> Exactly.  And I want this to change.  There is no reason for such
> restriction.
> 
> > This is not because of image libraries, this is because the text-mode
> > display simply cannot currently support such display elements.
> 
> This is not an issue at all.  Capabilities of text-mode display are
> irrelevant.  w3mimgdisplay is using framebuffer.  I do not know how to
> better communicate that.

I was talking about the Emacs text-mode display code, sorry for being
unclear.  The Emacs display code which handles the console is not
capable of supporting images, or any display element whose dimensions
are not the same as character cells on a console.  The assumption that
each display element takes exactly one character cell on display is
implicitly present in many places in the Emacs display code that
handles TTYs.  We cannot even display a larger font on a console.
(There is a special handling of wide characters that take two columns
on display -- but still the same height as other characters -- but
that's all.)

To be able to display images on a console within an Emacs text-mode
frame, you need to be able to tell the Emacs display code that a
certain area on display is "taken".  But since images will generally
be larger than a character, in both dimensions, you need the Emacs
text-mode display code to support display elements of arbitrary
dimensions, and that is currently not possible.  Adding that would
need serious changes in the text-mode parts of the Emacs display
engine code, and that is almost entirely in C.

This is the main obstacle we'd need to overcome if Emacs will ever be
able to display images on TTYs.  All the rest is much easier.

> Try:
> 
>    $ w3m https://gnu.org
> 
> You will see text-mode HTML browser layout with images on the console.

I know that text-mode display devices can display graphics, I had it
30 years ago on an IBM PC clone.

> > (You never said you were talking about a console.)  We currently don't
> > expose console cursor control to Lisp.  Should be easy to add that, if
> > needed.
> 
> I see.
> 
> cursor-type does not say it does not work on the console.
> 
> I think not all values are doable and relevant.  t and nil should be
> enough.
> 
> Shell I open a bug report for this?

A feature request, yes.

> > I don't think we have such a hook, no.  But if we had it, using it
> > like you want would cause annoying flickering, because you'd
> > constantly redraw the image at very high frequency.
> 
> I do not think it would cause flickering on the console.  The only thing
> that normally "flickers" is the blinking cursor.  Emacs does not draw
> anything if nothing changed.

Emacs doesn't draw anything because it knows what's on the glass and
what _should_ be on the glass, and compares the two to decide what
should be redrawn.  But you want to draw on parts of the screen behind
Emacs's back, so it won't know whether something changed.  And you
OTOH have no way of knowing where Emacs have redrawn something during
the last redisplay cycle.  So you will have to redraw your images each
time Emacs finishes a redisplay cycle, regardless of whether it
changed anything on the glass in the area of an image.  These
redisplay cycles happen at very high frequency -- usually, Emacs
enters redisplay, quickly decides what needs to be redrawn, and then
does whatever has to be done and exits redisplay.  It tries very hard
not to redraw the parts that haven't changed, and that is why there's
no flickering.  But I don't see how you could prevent the flickering
of those add-on images, because without knowing which parts of the
screen changed, you will have to redraw them very frequently.

> >> Is there such hook?  If not, would it be difficult to add it?
> >
> > It won't be hard to add it, but I'm not sure we want to add such a
> > hook, see above.
> 
> That is a shame.  If there was a hook to detect when Emacs has drawn
> something on the console, I could plug image redrawing there.
> 
> If I wanted to add it, where in the code should I look?

In update_frame and its subroutines.  But my recommendation is not use
such a kludgey approach.

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Eli Zaretskii]

> From: Tomas Hlavaty <tom@logand.com>
> Date: Thu, 03 Dec 2020 09:15:16 +0100
> Cc: emacs-devel@gnu.org
> 
> > It seems like unless you're going to teach Emacs to entirely draw
> > itself to the framebuffer, there will always be glitches like the
> > cursor messing up images since TUI Emacs won't be aware of what pixels
> > of the framebuffer have been drawn over independently of the linux VT
> > displaying lines of Emacs' characters.
> 
> The cursor could be turned off.
> 
> Are there other glitches which cannot be solved?

Even turning the cursor off doesn't guarantee that Emacs will not
trigger overwriting of the images.  For example, sometimes Emacs moves
the cursor by writing newline characters, which could potentially
overwrite the image if the console implements that by writing spaces
or clearing each line to EOL.

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Arthur Miller]

Tomas Hlavaty <tom@logand.com> writes:

> On Wed 02 Dec 2020 at 17:24, Akira Kyle <akira@akirakyle.com> wrote:
>> On Tue, Dec 01, 2020 at 12:44 AM, Tomas Hlavaty <tom@logand.com> 
>> wrote:
>>
>>> On Mon 30 Nov 2020 at 10:03, Akira Kyle <akira@akirakyle.com> 
>>> wrote:
>>>> and because I see pgtk as the future of Emacs on GNU/Linux systems
>>>> as X becomes increasingly obsolete.
>>>
>>> I am not sure such future is bright.
>>>
>>> In that future, is there no way to do graphics without a widget 
>>> toolkit?
>>
>> Sure there is. Under wayland you can just dump pixels into a 
>> shared buffer to render them to the screen or use EGL to render 
>> using a GPU. The difference from X is that wayland offers no 
>> convince functions to actually draw pixels into such a buffer, its 
>> up to the client to do so. In order to not reinvent the wheel, its 
>> much more convenient to use a toolkit to draw the elements it's 
>> good at drawing.
>
> "just dump pixels" and "no function to actually do that" and "use
> toolkit" does seem to contradict the point of being able to do it
> without toolkit.
>
> How long will pgtk future last?

> Compare that kind of future with for example ansi terminal future.
> Which one do you think has longer future ahead?  Why?
I don't think you are askinkg for ANSI terminal; I think you are asking
for Emacs on framebuffer. 

I don't know how hard it is to create it, but it is probably on same
level as concole/terminal backend, X11 backend, Win32, Gtk, etc. I don't
know, I never used framebuffer so I can't really tell; just a sense.

>>>> AFAIU the pgtk version, like the ns version it was modeled after,
>>>> doesn't implement everything in the main frame area of Emacs as
>>>> widgets, only the tool and menu bars are implemented as proper
>>>> widgets.
>>>
>>> Also popup menus that pop up on a mouse click are gtk iirc.
>>>
>> I think it would be possible to implement such menus without GTK using
>> just child frames.
I agree with you on that one. I played with the idea; if you search the
archive you will find some poorman menu I sent in pga bug with focus
with child frames. Emacs already does halv of the jobb; one can display
a text buffer, enable current line highlight and set special text properties
and there is a simulated menu; sort of. 

> Would not it be better to extend the canvas idea you mentioned above to
> allow graphics in general, portable, toolkit independent way?
svg renderer is; as I understand. Maybe I am wrong, but as it seems
Emacs has got a built-in graphics via svg. I always thought Emacs went
through external application to display svg, but I learned recently it
does not.

Other path is to use Vulkan or OpenGL; I think OpenGL is better choice
for Emacs; you can render to a texture in memory, and handle XImage or
XPixmap to Emacs (hopefully - I haven't had time yet to go through and
test Eli's kind walkthrough yet). You could actually render entire emacs
buffers via egl to textures and display them via framebuffer. I don't
think it would be easy, but possible. Would require lots of C coding; if
you have time for that :-).

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Stefan Monnier]

BTW, for the framebuffer case, I think the pgtk code is a step in the
direction of making it possible for Emacs to open up a *graphical* frame
on a framebuffer using something like Cairo-fb.


        Stefan

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Tomas Hlavaty]

On Thu 03 Dec 2020 at 16:39, Eli Zaretskii <eliz@gnu.org> wrote:
>> At the moment, in emacs-framebuffer, I just call it to display an image
>> using call-process-region.  It could be optimized with pipe, but this
>> has low priority for emacs-framebuffer as it is fast enough for my
>> needs.
>
> IMNSHO, neither call-process-region nor pipes are fast enough to be an
> integral part of the Emacs redisplay: they are too slow.  Especially
> if you want to be able to display more than one image at the same
> time.  To see how the performance is inadequate, turn on flyspell-mode
> and scroll through a large text buffer, preferably with lots of
> mis-spelled words (or some text full of special terminology the
> general-purpose dictionary won't know about).  The effect in your case
> will be very similar.
>
>> I am not exatly sure, what the concern is here.
>> 
>> The images do not travel via pipe.  Only meta-data like filenames,
>> coordinates and handles.
>
> Yes, I understood that much.  But that doesn't matter: communicating
> with a spell-checker in the flyspell-mode case also sends just a small
> number of words in each direction, and yet the slow-down is
> considerable, sometimes downright annoying.

Is it an appropriate comparison?

I am sure it is possible to find many examples where Emacs slows down to
crawl even without external process, e.g. a buffer with long lines.
Does it mean I should dismiss Emacs for editing text?

I do not think my use-case is comparable to "scrolling through a large
text buffer, preferably with lots of mis-spelled words (or some text
full of special terminology the general-purpose dictionary won't know
about)".

I do not intent to scroll through a large buffer with lots images.  Even
if I did, there is no evidence that it would suffer the problems in the
above example.  w3m manages just fine.  My use case is something like
image-mode or doc-view or pdf-tools.  emacs-framebuffer already does
that for many image and document types and I have no problems with
speed.

In any case, ability to display images and documents on the console
beats no ability at all.

>> What exactly needs to be done in the middle of redisplay that is
>> relevant to drawing images?  Determining image width and height?
>
> Sending the image spec and receiving the dimensions and other image
> attributes, yes.  And I presume that the sub-process needs some time
> to calculate the answer (because at least some queries require to load
> and process an image).

This is a non-issue.  Images usually do not change those attributes
quickly, if at all.  So this can usually be cached or even calculated
just once.  This is what I do in emacs-framebuffer.

>> > You are supposed to submit data in one of the known formats.  Why is
>> > that a problem?
>> 
>> Because they do not work on the console.
>> 
>> And the reason they do not work on the console is because there is
>> hardcoded dependency on image drawing libraries which do not work on the
>> console.
>
> No, that's not the main reason.  See below.
>
>> > On the console you currently cannot have images in Emacs, period.
>> 
>> Exactly.  And I want this to change.  There is no reason for such
>> restriction.
>> 
>> > This is not because of image libraries, this is because the text-mode
>> > display simply cannot currently support such display elements.
>> 
>> This is not an issue at all.  Capabilities of text-mode display are
>> irrelevant.  w3mimgdisplay is using framebuffer.  I do not know how to
>> better communicate that.
>
> I was talking about the Emacs text-mode display code, sorry for being
> unclear.  The Emacs display code which handles the console is not
> capable of supporting images, or any display element whose dimensions
> are not the same as character cells on a console.  The assumption that
> each display element takes exactly one character cell on display is
> implicitly present in many places in the Emacs display code that
> handles TTYs.  We cannot even display a larger font on a console.
> (There is a special handling of wide characters that take two columns
> on display -- but still the same height as other characters -- but
> that's all.)
>
> To be able to display images on a console within an Emacs text-mode
> frame, you need to be able to tell the Emacs display code that a
> certain area on display is "taken".  But since images will generally
> be larger than a character, in both dimensions, you need the Emacs
> text-mode display code to support display elements of arbitrary
> dimensions, and that is currently not possible.  Adding that would
> need serious changes in the text-mode parts of the Emacs display
> engine code, and that is almost entirely in C.
>
> This is the main obstacle we'd need to overcome if Emacs will ever be
> able to display images on TTYs.  All the rest is much easier.

I understand that.

I do not aim to change that.

Again, my use-case is like that of image-mode, doc-view and pdf-tools.
There is a file, say /tmp/a.png, I open the file in a buffer and the
buffer will show the png image.  Or there is a file, say /tmp/a.pdf, I
open the file in a buffer and the buffer will show the pdf document.

This use-case does not need to deal with the issues outlined above.  It
just needs to know size and position of the buffer, width and height of
characters and screen etc.  emacs-framebuffer already does that and it
is not that difficult, thanks to the hints I already got from you:-)

>> > (You never said you were talking about a console.)  We currently don't
>> > expose console cursor control to Lisp.  Should be easy to add that, if
>> > needed.
>> 
>> I see.
>> 
>> cursor-type does not say it does not work on the console.
>> 
>> I think not all values are doable and relevant.  t and nil should be
>> enough.
>> 
>> Shell I open a bug report for this?
>
> A feature request, yes.

ok, thank you.

>> > I don't think we have such a hook, no.  But if we had it, using it
>> > like you want would cause annoying flickering, because you'd
>> > constantly redraw the image at very high frequency.
>> 
>> I do not think it would cause flickering on the console.  The only thing
>> that normally "flickers" is the blinking cursor.  Emacs does not draw
>> anything if nothing changed.
>
> Emacs doesn't draw anything because it knows what's on the glass and
> what _should_ be on the glass, and compares the two to decide what
> should be redrawn.  But you want to draw on parts of the screen behind
> Emacs's back, so it won't know whether something changed.  And you
> OTOH have no way of knowing where Emacs have redrawn something during
> the last redisplay cycle.  So you will have to redraw your images each
> time Emacs finishes a redisplay cycle, regardless of whether it
> changed anything on the glass in the area of an image.  These
> redisplay cycles happen at very high frequency -- usually, Emacs
> enters redisplay, quickly decides what needs to be redrawn, and then
> does whatever has to be done and exits redisplay.  It tries very hard
> not to redraw the parts that haven't changed, and that is why there's
> no flickering.  But I don't see how you could prevent the flickering
> of those add-on images, because without knowing which parts of the
> screen changed, you will have to redraw them very frequently.

This is not an issue.

You can try and see it easily.

Start emacs on the console.

Start shell in Emacs: M-x shell

Execute: $ cat /dev/urandom >/dev/fb0

Observe what Emacs draws.  I can see blinking cursor.  Part of the
*shell* buffer was drawn and line and column indicators on the
mode-line.  Everything else remains unchanged.

Now ssh from a different computer and run $ cat /dev/urandom >/dev/fb0

Observe what Emacs draws.  I can see blinking cursor.  Everything else
on the screen remains unchanged.  After some time, clock on the
mode-line is drawn.

So if Emacs does not change anything, it does not draw anything on the
console and I do not need to redraw the image to the framebuffer.

The only time I need to redraw the image to the framebuffer is when
emacs changed and drawn something.  But this is a single event, not a
periodic thing.  Because there is no periodic redraw, there is no
flickering.  Unfortunately, there is no hook to run after this single
event.

>> >> Is there such hook?  If not, would it be difficult to add it?
>> >
>> > It won't be hard to add it, but I'm not sure we want to add such a
>> > hook, see above.
>> 
>> That is a shame.  If there was a hook to detect when Emacs has drawn
>> something on the console, I could plug image redrawing there.
>> 
>> If I wanted to add it, where in the code should I look?
>
> In update_frame and its subroutines.  But my recommendation is not use
> such a kludgey approach.

Thanks, I'll have a look.

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Tomas Hlavaty]

On Thu 03 Dec 2020 at 17:13, Eli Zaretskii <eliz@gnu.org> wrote:
>> > It seems like unless you're going to teach Emacs to entirely draw
>> > itself to the framebuffer, there will always be glitches like the
>> > cursor messing up images since TUI Emacs won't be aware of what pixels
>> > of the framebuffer have been drawn over independently of the linux VT
>> > displaying lines of Emacs' characters.
>> 
>> The cursor could be turned off.
>> 
>> Are there other glitches which cannot be solved?
>
> Even turning the cursor off doesn't guarantee that Emacs will not
> trigger overwriting of the images.  For example, sometimes Emacs moves
> the cursor by writing newline characters, which could potentially
> overwrite the image if the console implements that by writing spaces
> or clearing each line to EOL.

But that example is a single event and does not cause glitch or
flickering.

Emacs usually does not move the cursor by writing newline characters
because it feels like it.  And it does not do that frequently.

If I do not do anything, Emacs will not do anything (except updating
clock in the mode-line perhaps, which is outside the buffer with the
image).

Now after that single event, the image should be redrawn but that's it.

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Eli Zaretskii]

> From: Tomas Hlavaty <tom@logand.com>
> Cc: emacs-devel@gnu.org
> Date: Thu, 03 Dec 2020 22:02:47 +0100
> 
> >> The images do not travel via pipe.  Only meta-data like filenames,
> >> coordinates and handles.
> >
> > Yes, I understood that much.  But that doesn't matter: communicating
> > with a spell-checker in the flyspell-mode case also sends just a small
> > number of words in each direction, and yet the slow-down is
> > considerable, sometimes downright annoying.
> 
> Is it an appropriate comparison?

Yes, IME.  It shows what happens when Emacs needs to communicate with
an external subprocess at a high frequency.  Which is what will happen
if your proposed design will be used as a general-purpose feature for
drawing images in Emacs.

> I do not intent to scroll through a large buffer with lots images.  Even
> if I did, there is no evidence that it would suffer the problems in the
> above example.  w3m manages just fine.  My use case is something like
> image-mode or doc-view or pdf-tools.  emacs-framebuffer already does
> that for many image and document types and I have no problems with
> speed.

So you are talking about making Emacs support only your use case?  Not
about a feature that allows any other legitimate use case in Emacs
which could involve images?

> > Sending the image spec and receiving the dimensions and other image
> > attributes, yes.  And I presume that the sub-process needs some time
> > to calculate the answer (because at least some queries require to load
> > and process an image).
> 
> This is a non-issue.  Images usually do not change those attributes
> quickly, if at all.  So this can usually be cached or even calculated
> just once.

That "once" is what I'm talking about.  Imagine going through a
directory of image files one by one.

> Again, my use-case is like that of image-mode, doc-view and pdf-tools.
> There is a file, say /tmp/a.png, I open the file in a buffer and the
> buffer will show the png image.  Or there is a file, say /tmp/a.pdf, I
> open the file in a buffer and the buffer will show the pdf document.

That's a very limited use case.  So much so that I doubt if it's even
justified.  After all, you can already display images on the console,
so why do you insist to show it inside an Emacs window, of all the
places.  People are already complaining about the limitations of
doc-view and similar modes.

> > Emacs doesn't draw anything because it knows what's on the glass and
> > what _should_ be on the glass, and compares the two to decide what
> > should be redrawn.  But you want to draw on parts of the screen behind
> > Emacs's back, so it won't know whether something changed.  And you
> > OTOH have no way of knowing where Emacs have redrawn something during
> > the last redisplay cycle.  So you will have to redraw your images each
> > time Emacs finishes a redisplay cycle, regardless of whether it
> > changed anything on the glass in the area of an image.  These
> > redisplay cycles happen at very high frequency -- usually, Emacs
> > enters redisplay, quickly decides what needs to be redrawn, and then
> > does whatever has to be done and exits redisplay.  It tries very hard
> > not to redraw the parts that haven't changed, and that is why there's
> > no flickering.  But I don't see how you could prevent the flickering
> > of those add-on images, because without knowing which parts of the
> > screen changed, you will have to redraw them very frequently.
> 
> This is not an issue.
> 
> You can try and see it easily.
> 
> Start emacs on the console.
> 
> Start shell in Emacs: M-x shell
> 
> Execute: $ cat /dev/urandom >/dev/fb0
> 
> Observe what Emacs draws.  I can see blinking cursor.  Part of the
> *shell* buffer was drawn and line and column indicators on the
> mode-line.  Everything else remains unchanged.

What you don't see is the frantic activity going on in the display
code behind the scenes.

Look, you can disregard what I'm saying and then later learn this
stuff the hard way.  It's okay, it will be a fine journey and very
educational, I can assure you.  I'm just trying to warn you about the
hidden rocks under the surface.  I will stop now.

Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Eli Zaretskii]

> From: Tomas Hlavaty <tom@logand.com>
> Cc: akira@akirakyle.com, emacs-devel@gnu.org
> Date: Thu, 03 Dec 2020 22:15:08 +0100
> 
> > Even turning the cursor off doesn't guarantee that Emacs will not
> > trigger overwriting of the images.  For example, sometimes Emacs moves
> > the cursor by writing newline characters, which could potentially
> > overwrite the image if the console implements that by writing spaces
> > or clearing each line to EOL.
> 
> But that example is a single event and does not cause glitch or
> flickering.

Why do you assume it's a single event?  Whenever Emacs decides that
moving the cursor from some coordinates to other coordinates should be
done by writing newlines, it will always do that when it needs to.
That could be every redisplay cycle, depending on the situation.

> Emacs usually does not move the cursor by writing newline characters
> because it feels like it.  And it does not do that frequently.

If you build your software on these assumptions, I can assure you it
will break in some use cases.

> If I do not do anything, Emacs will not do anything

That is profoundly incorrect.  First, "I do not do anything" doesn't
mean Emacs doesn't: there are timers running in Emacs all the time,
even if it's "emacs -Q".  More importantly, what looks to you like
"Emacs doesn't do anything" is the result of redisplay cycles that do
quite a lot of processing, before they decide that nothing should be
redrawn on the glass, and if you assume that processing will never
include moving a cursor, you are building your code on shaky grounds.