Multimedia dashboard in GNU Emacs

Multimedia dashboard in GNU Emacs

[Anand Tamariya]

[-- Attachment #1: Type: text/plain, Size: 508 bytes --]

With little enhancements to inbuilt xwidget, you can embed GTK applications
like VLC and gVim as widgets in Emacs buffer. Using this, you can build a
multimedia dashboard for monitoring webcams and other media in a single
place; or take notes while watching video in a separate buffer.

*Video:* https://www.youtube.com/watch?v=kSkZwI78t6g
*Details:*
https://lifeofpenguin.blogspot.com/2021/12/embedding-gtk-applications-via-xembed-2.html

*Code:* https://gitlab.com/atamariya/emacs/-/blob/dev/src/xwidget.c

[-- Attachment #2: Type: text/html, Size: 838 bytes --]

Re: Multimedia dashboard in GNU Emacs

[Lars Ingebrigtsen]

Anand Tamariya <atamariya@gmail.com> writes:

> With little enhancements to inbuilt xwidget, you can embed GTK applications like
> VLC and gVim as widgets in Emacs buffer. Using this, you can build a multimedia
> dashboard for monitoring webcams and other media in a single place; or take
> notes while watching video in a separate buffer.
>
> Video: https://www.youtube.com/watch?v=kSkZwI78t6g
> Details:
> https://lifeofpenguin.blogspot.com/2021/12/embedding-gtk-applications-via-xembed-2.html
>
> Code: https://gitlab.com/atamariya/emacs/-/blob/dev/src/xwidget.c

Looks interesting.  Could you submit this as a patch to Emacs?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Multimedia dashboard in GNU Emacs

[Po Lu]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Looks interesting.  Could you submit this as a patch to Emacs?

That will unfortunately not work with the recent fixes to xwidgets (or
display in multiple Emacs windows in the old implementation), as you
can't render a GtkSocket in an offscreen window.

Re: Multimedia dashboard in GNU Emacs

[Anand Tamariya]

[-- Attachment #1: Type: text/plain, Size: 228 bytes --]

> Looks interesting.  Could you submit this as a patch to Emacs?
>
Not sure if I can. Though it's mostly boilerplate code, I lifted most of
the code from here
http://blog.borovsak.si/2009/07/spawning-processes-using-glib.html .

[-- Attachment #2: Type: text/html, Size: 522 bytes --]

Re: Multimedia dashboard in GNU Emacs

[Anand Tamariya]

[-- Attachment #1.1: Type: text/plain, Size: 365 bytes --]

> That will unfortunately not work with the recent fixes to xwidgets (or
> display in multiple Emacs windows in the old implementation), as you
> can't render a GtkSocket in an offscreen window.
>
Do we have to necessarily use off-screen rendering? Are you referring to
some problem with the scenario as in the attached pic? Can you elaborate?

[image: embed5.png]

[-- Attachment #1.2: Type: text/html, Size: 662 bytes --]

[-- Attachment #2: embed5.png --]
[-- Type: image/png, Size: 265874 bytes --]

Re: Multimedia dashboard in GNU Emacs

[Po Lu]

Anand Tamariya <atamariya@gmail.com> writes:

> Do we have to necessarily use off-screen rendering? Are you referring
> to some problem with the scenario as in the attached pic? Can you
> elaborate?

Yes, a single xwidget has to look and behave identically in every window
it is attached to.

Re: Multimedia dashboard in GNU Emacs

[Lars Ingebrigtsen]

Anand Tamariya <atamariya@gmail.com> writes:

>  Looks interesting.  Could you submit this as a patch to Emacs?
>
> Not sure if I can. Though it's mostly boilerplate code, I lifted most of the code from
> here http://blog.borovsak.si/2009/07/spawning-processes-using-glib.html . 

Perhaps you could ask them if they'd be willing to sign FSF assignment
papers?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Multimedia dashboard in GNU Emacs

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Not sure if I can. Though it's mostly boilerplate code, I lifted most of
  > the code from here
  > http://blog.borovsak.si/2009/07/spawning-processes-using-glib.html .

I can't see anything in that page -- it depends on nonfree JS code --
but in general I can say that copying code that's not profound
is more trouble than using something written from scratch.

Would you like to start from scratch?

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Multimedia dashboard in GNU Emacs

[Po Lu]

Richard Stallman <rms@gnu.org> writes:

>   > Not sure if I can. Though it's mostly boilerplate code, I lifted most of
>   > the code from here
>   > http://blog.borovsak.si/2009/07/spawning-processes-using-glib.html .

BTW, my opinion is that code in Emacs should _never_ use GLib to spawn
processes, because GLib likes to mess around with Emacs's SIGCHLD
handler.  The situation with the WebKit xwidget is confusing enough as
it is, so we really don't need anything else using GLib subprocesses.

Re: Multimedia dashboard in GNU Emacs

[Aiko Kyle]

On Sun, Dec 26, 2021 at 10:07 PM Po Lu <luangruo@yahoo.com> wrote:
>
> Anand Tamariya <atamariya@gmail.com> writes:
>
> > Do we have to necessarily use off-screen rendering? Are you referring
> > to some problem with the scenario as in the attached pic? Can you
> > elaborate?
>
> Yes, a single xwidget has to look and behave identically in every window
> it is attached to.
>

You mean technically it has to given the way its implemented? There
isn't some other reason why an xwidget has to behave this way?

Re: Multimedia dashboard in GNU Emacs

[joakim]

Aiko Kyle <aikokyle@gmail.com> writes:

> On Sun, Dec 26, 2021 at 10:07 PM Po Lu <luangruo@yahoo.com> wrote:
>>
>> Anand Tamariya <atamariya@gmail.com> writes:
>>
>> > Do we have to necessarily use off-screen rendering? Are you referring
>> > to some problem with the scenario as in the attached pic? Can you
>> > elaborate?
>>
>> Yes, a single xwidget has to look and behave identically in every window
>> it is attached to.
>>
>
> You mean technically it has to given the way its implemented? There
> isn't some other reason why an xwidget has to behave this way?

It kind of has to, in order to preserve the way emacs behaves.
That is multiple views of the same buffer, and so on.

In the original xwidget branch there were some alternative behaviours
implemented as proof of concepts, and they worked to some degree, but
breaks the emacs paradigm, so they arent really all that useful IMHO.

There might also be some kind of experiment involving redirection of an
applications main window to an offscreen widget, that might be useful
for xwidget, but I'm not entirely sure.

-- 
Joakim Verona
joakim@verona.se

Re: Multimedia dashboard in GNU Emacs

[Aiko Kyle]

Hi Joakim!

Po and I were previously discussing this in the context of the webkit
xwidget on this thread:
https://lists.gnu.org/archive/html/emacs-devel/2021-12/msg01914.html

On Wed, Dec 29, 2021 at 6:34 AM <joakim@verona.se> wrote:
>
> It kind of has to, in order to preserve the way emacs behaves.
> That is multiple views of the same buffer, and so on.
>
> In the original xwidget branch there were some alternative behaviours
> implemented as proof of concepts, and they worked to some degree, but
> breaks the emacs paradigm, so they arent really all that useful IMHO.
>

I think that the question of how well an xwidget obeys "the emacs
paradigm" depends on the content it's displaying. If it were a button
or some other element in a larger button, I agree that the current
model fits the emacs way well enough. However I think when the xwidget
is the only element in a buffer and takes up a large portion or even
the whole window, then I think the correct paradigm would resemble
something like doc-view, where the two widow's xwidgets can show
different views if they have some notion of scrolling.

> There might also be some kind of experiment involving redirection of an
> applications main window to an offscreen widget, that might be useful
> for xwidget, but I'm not entirely sure.
>

This is the current implementation that Po Lu has fixed various
flickering bugs on master for xwidgets on x. However both the xwidgets
on ns and my emacs module emacs-webkit, handles this in a different
way where the webkitview can only be displayed in one window.

Re: Multimedia dashboard in GNU Emacs

[joakim]

Aiko Kyle <aikokyle@gmail.com> writes:

> Hi Joakim!
>
> Po and I were previously discussing this in the context of the webkit
> xwidget on this thread:
> https://lists.gnu.org/archive/html/emacs-devel/2021-12/msg01914.html
>
> On Wed, Dec 29, 2021 at 6:34 AM <joakim@verona.se> wrote:
>>
>> It kind of has to, in order to preserve the way emacs behaves.
>> That is multiple views of the same buffer, and so on.
>>
>> In the original xwidget branch there were some alternative behaviours
>> implemented as proof of concepts, and they worked to some degree, but
>> breaks the emacs paradigm, so they arent really all that useful IMHO.
>>
>
> I think that the question of how well an xwidget obeys "the emacs
> paradigm" depends on the content it's displaying. If it were a button
> or some other element in a larger button, I agree that the current
> model fits the emacs way well enough. However I think when the xwidget
> is the only element in a buffer and takes up a large portion or even
> the whole window, then I think the correct paradigm would resemble
> something like doc-view, where the two widow's xwidgets can show
> different views if they have some notion of scrolling.

This was one of the methods that was evaluated in the original branch,
that is, every emacs window got its own separate gtk component.

So, if the xwidget was a gtk button, for example, inserted in a buffer,
and there were several emacs window showing the same area, each emacs
window would get a separate gtk button representing the same xwidget.
This worked okay-ish, for buttons. But for sliders it didnt turn out so
well, because one needed special glue code to make each separate gtk
slider be in sync with every other instance. It's doable, but
convoluted. Anyway, I'm sure theres been plenty of progress since the
original experiments, so maybe today it's less convoluted.

>
>> There might also be some kind of experiment involving redirection of an
>> applications main window to an offscreen widget, that might be useful
>> for xwidget, but I'm not entirely sure.
>>
>
> This is the current implementation that Po Lu has fixed various
> flickering bugs on master for xwidgets on x. However both the xwidgets
> on ns and my emacs module emacs-webkit, handles this in a different
> way where the webkitview can only be displayed in one window.
-- 
Joakim Verona
joakim@verona.se

Re: Multimedia dashboard in GNU Emacs

[Aiko Kyle]

On Wed, Dec 29, 2021 at 1:08 PM <joakim@verona.se> wrote:
>
> This was one of the methods that was evaluated in the original branch,
> that is, every emacs window got its own separate gtk component.
>
> So, if the xwidget was a gtk button, for example, inserted in a buffer,
> and there were several emacs window showing the same area, each emacs
> window would get a separate gtk button representing the same xwidget.
> This worked okay-ish, for buttons. But for sliders it didnt turn out so
> well, because one needed special glue code to make each separate gtk
> slider be in sync with every other instance. It's doable, but
> convoluted. Anyway, I'm sure theres been plenty of progress since the
> original experiments, so maybe today it's less convoluted.
>

When I was playing around with this last year I encountered similar
issues with "stateful" gtk widgets. Since I primarily cared about the
webkit widget, and since that's still the only really implemented
widget, I found that forcing there to only ever be one emacs window
displaying it was the best compromise between technical limitations
and what I think of as "the emacs paradigm" for displaying such
content. Although Po seems to disagree with me on that point, hence
his pushing forward with the offscreen rendering technique, although
there are still various bugs to be resolved in that implementation.

Re: Multimedia dashboard in GNU Emacs

[Po Lu]

Aiko Kyle <aikokyle@gmail.com> writes:

> I think that the question of how well an xwidget obeys "the emacs
> paradigm" depends on the content it's displaying. If it were a button
> or some other element in a larger button, I agree that the current
> model fits the emacs way well enough. However I think when the xwidget
> is the only element in a buffer and takes up a large portion or even
> the whole window, then I think the correct paradigm would resemble
> something like doc-view, where the two widow's xwidgets can show
> different views if they have some notion of scrolling.

I think Doc View uses per-window overlays to achieve that, which you can
also do with xwidgets.

> This is the current implementation that Po Lu has fixed various
> flickering bugs on master for xwidgets on x.

That was the old implementation as well, but it used GDK windows instead
of X windows, which could be painted over by Emacs drawing commands, and
caused flickering.

That doesn't make sense with a GtkSocket, because it uses XEmbed
under-the-hood, which basically tells some other program where to place
a window.  A GtkSocket does no drawing by itself.

A better solution would be to redirect a window (given an XID) to some
off-screen pixmap, and to display that pixmap in every window instead.

Re: Multimedia dashboard in GNU Emacs

[Aiko Kyle]

On Wed, Dec 29, 2021 at 6:02 PM Po Lu <luangruo@yahoo.com> wrote:
>
> Aiko Kyle <aikokyle@gmail.com> writes:
>
> > I think that the question of how well an xwidget obeys "the emacs
> > paradigm" depends on the content it's displaying. If it were a button
> > or some other element in a larger button, I agree that the current
> > model fits the emacs way well enough. However I think when the xwidget
> > is the only element in a buffer and takes up a large portion or even
> > the whole window, then I think the correct paradigm would resemble
> > something like doc-view, where the two widow's xwidgets can show
> > different views if they have some notion of scrolling.
>
> I think Doc View uses per-window overlays to achieve that, which you can
> also do with xwidgets.
>

Thanks for letting me know about per-window overlays. I think that may
be a better way to make a full-window xwidget behave closer to "the
emacs paradigm."

However what would that look like with emacs-xwidget? IIUC that would
mean instantiating multiple xwidget webkit instances, one for each
window? Then one would need to figure out how keep the appropriate
state synchronized between them? I think this may prove difficult to
figure out the right UI for since a webpage can end up being pretty
stateful and it would be surprising, for example, if a user entered
text in one webkit widget and it doesn't appear in the others. Or
worse, if one makes modification to state on one page, then kills that
window and now its lost.

Re: Multimedia dashboard in GNU Emacs

[Po Lu]

Aiko Kyle <aikokyle@gmail.com> writes:

> However what would that look like with emacs-xwidget? IIUC that would
> mean instantiating multiple xwidget webkit instances, one for each
> window?

Yes.

> Then one would need to figure out how keep the appropriate
> state synchronized between them? I think this may prove difficult to
> figure out the right UI for since a webpage can end up being pretty
> stateful and it would be surprising, for example, if a user entered
> text in one webkit widget and it doesn't appear in the others. Or
> worse, if one makes modification to state on one page, then kills that
> window and now its lost.

Yes, it's not really practical with GTK+, but it's possible.

Re: Multimedia dashboard in GNU Emacs

[Aiko Kyle]

On Wed, Dec 29, 2021 at 8:29 PM Po Lu <luangruo@yahoo.com> wrote:
>
> Yes, it's not really practical with GTK+, but it's possible.

Doesn't this have to do more with syncronizing two DOMs though? There
isn't much gtk state attached to webkit webviews that would need to be
shared. Fundamentally the emacs window/buffer paradigm doesn't fit
with the browser DOM/webview paradigm. This again goes back to my
opinion that enforcing a one-to-one correspondence between a webkit
xwidget window and buffer is the better UI paradigm for meshing the
two than the offscreen rendering technique.

BTW do you regularly use xwidget webkits in your daily emacs use? I'm
curious as to what your workflow/use case with them might look like?
Understanding that might help me to better understand your opinion on
why the osr technique results in a better UI in this case.

Re: Multimedia dashboard in GNU Emacs

[Po Lu]

Aiko Kyle <aikokyle@gmail.com> writes:

> Doesn't this have to do more with syncronizing two DOMs though? There
> isn't much gtk state attached to webkit webviews that would need to be
> shared.

The idea is to give the view a large enough allocation and to put it
into a GtkScrolledWindow, which can then be scrolled however you want
it.

> Fundamentally the emacs window/buffer paradigm doesn't fit with the
> browser DOM/webview paradigm.  This again goes back to my opinion that
> enforcing a one-to-one correspondence between a webkit xwidget window
> and buffer is the better UI paradigm for meshing the two than the
> offscreen rendering technique.

If an xwidget is only displayed in a single window, it behaves the same
way as in any other application, so I see no reason to prevent it from
being displayed in multiple windows.

It won't hurt for the people who want it the way it is, and the people
who want to keep an xwidget-webkit buffer from being displayed in
multiple windows can simply refrain from opening those other windows.

> I'm curious as to what your workflow/use case with them might look
> like?  Understanding that might help me to better understand your
> opinion on why the osr technique results in a better UI in this case.

I'm not arguing about user experience (I have no opinion on that), only
about what fits in better with the Emacs display architecture.

Re: Multimedia dashboard in GNU Emacs

[Aiko Kyle]

On Wed, Dec 29, 2021 at 9:48 PM Po Lu <luangruo@yahoo.com> wrote:
>
> > Doesn't this have to do more with syncronizing two DOMs though? There
> > isn't much gtk state attached to webkit webviews that would need to be
> > shared.
>
> The idea is to give the view a large enough allocation and to put it
> into a GtkScrolledWindow, which can then be scrolled however you want
> it.
>

As I see. So in this idea, the entire webpage would be rendered in the
case of a xwidget webkit? That seems like it would be costly for
especially large webpages.

> > Fundamentally the emacs window/buffer paradigm doesn't fit with the
> > browser DOM/webview paradigm.  This again goes back to my opinion that
> > enforcing a one-to-one correspondence between a webkit xwidget window
> > and buffer is the better UI paradigm for meshing the two than the
> > offscreen rendering technique.
>
> If an xwidget is only displayed in a single window, it behaves the same
> way as in any other application, so I see no reason to prevent it from
> being displayed in multiple windows.
>
> It won't hurt for the people who want it the way it is, and the people
> who want to keep an xwidget-webkit buffer from being displayed in
> multiple windows can simply refrain from opening those other windows.
>
> > I'm curious as to what your workflow/use case with them might look
> > like?  Understanding that might help me to better understand your
> > opinion on why the osr technique results in a better UI in this case.
>
> I'm not arguing about user experience (I have no opinion on that), only
> about what fits in better with the Emacs display architecture.

Do you then plan on implementing other xwidget types besides the
webkit one? I think with respect to the UI, osr makes more sense for
other widget types like buttons.

I agree the offscreen rendering is more general, however it seems to
me to add significant complexity that would be unnecessary if the UI
for it doesn't require it. Especially since the one-to-one webkit
buffer-window paradigm already has to be accounted for on ns, where
osr isn't available.

Re: Multimedia dashboard in GNU Emacs

[Po Lu]

Aiko Kyle <aikokyle@gmail.com> writes:

> As I see. So in this idea, the entire webpage would be rendered in the
> case of a xwidget webkit? That seems like it would be costly for
> especially large webpages.

WebKit in general tends to choke on large webpages, so I'm not too
worried.

> Do you then plan on implementing other xwidget types besides the
> webkit one? I think with respect to the UI, osr makes more sense for
> other widget types like buttons.

Yes, there will be a GStreamer widget: it will use multiple video sinks
connected to a single tee, allowing for a source to be displayed in
multiple windows without offscreen rendering.

> I agree the offscreen rendering is more general, however it seems to
> me to add significant complexity that would be unnecessary if the UI
> for it doesn't require it. Especially since the one-to-one webkit
> buffer-window paradigm already has to be accounted for on ns, where
> osr isn't available.

I consider that a bug, but I don't know enough about NS development to
fix it.

Re: Multimedia dashboard in GNU Emacs

[Aiko Kyle]

On Thu, Dec 30, 2021 at 5:59 PM Po Lu <luangruo@yahoo.com> wrote:
>
> Aiko Kyle <aikokyle@gmail.com> writes:
>
> > As I see. So in this idea, the entire webpage would be rendered in the
> > case of a xwidget webkit? That seems like it would be costly for
> > especially large webpages.
>
> WebKit in general tends to choke on large webpages, so I'm not too
> worried.
>

Are you actually thinking about implementing this idea?

> > Do you then plan on implementing other xwidget types besides the
> > webkit one? I think with respect to the UI, osr makes more sense for
> > other widget types like buttons.
>
> Yes, there will be a GStreamer widget: it will use multiple video sinks
> connected to a single tee, allowing for a source to be displayed in
> multiple windows without offscreen rendering.
>

I'll be interested to see how that integrates with the existing xwidget code.

> > I agree the offscreen rendering is more general, however it seems to
> > me to add significant complexity that would be unnecessary if the UI
> > for it doesn't require it. Especially since the one-to-one webkit
> > buffer-window paradigm already has to be accounted for on ns, where
> > osr isn't available.
>
> I consider that a bug, but I don't know enough about NS development to
> fix it.

According to the author of nsxwidget.m and their comment at the top of
the file, it's maybe not technically possible. I think rendering to
offscreen surfaces probably breaks assumptions the webkit developers
make, and the situation will likely only get worse as browsers are
increasingly hardware accelerated.

Re: Multimedia dashboard in GNU Emacs

[Po Lu]

Aiko Kyle <aikokyle@gmail.com> writes:

> Are you actually thinking about implementing this idea?

No, but patches welcome.

> I'll be interested to see how that integrates with the existing xwidget code.

The code already exists, and it works very well.  All that's left is to
define the individual pipelines for playing streams in common media
formats.