GStreamer xwidget

GStreamer xwidget

[Po Lu]

[-- Attachment #1: Type: text/plain, Size: 506 bytes --]

I would like to install the following change that introduces a
GStreamer-based media widget.

It depends on GStreamer, gst-plugins-base, and gst-plugins-good and is
thus a compile-time option.

Right now, it can only display a test pattern, but getting that to work
and perform decently was a challenge (for instance, offscreen rendering
is not an option, instead the pipeline is dynamically built based on
available xwidget views, utilizing hardware acceleration and video
decoding if available).

WDYT?


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Implement-a-media-xwidget-based-on-GStreamer.patch --]
[-- Type: text/x-patch, Size: 31895 bytes --]

From 39a034aff24b2b64b028345aeae66c7aae6f8820 Mon Sep 17 00:00:00 2001
From: Po Lu <luangruo@yahoo.com>
Date: Thu, 18 Nov 2021 18:10:45 +0800
Subject: [PATCH] Implement a media xwidget based on GStreamer

* configure.ac: Add option to use GStreamer for media playback.
* doc/lispref/display.texi (Xwidgets): Document new functions
and xwidget type.

* src/Makefile.in (EMACS_CFLAGS): Add GStreamer cflags.
(LIBES): Add GStreamer linker options.

* etc/NEWS: Announce `media' xwidget.
* etc/xterm.c (x_scroll_run): Don't resize cairo surface
if not a GTK xwidget.
(x_term_init): Initialize GStreamer.

* src/xwidget.c (gst_create_window)
(gst_message_cb, unlink_gst_xwidget_view)
(check_gstreamer_dependencies): New functions.

(Fmake_xwidget): Add support for media xwidgets.
(Fxwidget_perform_lispy_event): Return nil if it doesn't make
sense to perform an event.
(Fxwidget_resize): Add support for media widgets.
(xwidget_button, xwidget_motion_or_crossing): Handle non-GTK
xwidgets.
(xwidget_expose): Handle media xwidgets.
(x_draw_xwidget_glyph_string): Set up GStreamer widgets if
appropriate.
(Fxwidget_size_request): Return nil on widgets that can't
have a size request.
(Fdelete_xwidget_view): Clean up after GStreamer.
(Fxwidget_media_pause, Fxwidget_media_play): New functions.

(syms_of_xwidget): New symbols and subrs.
(kill_xwidget): Clean up after GStreamer.

* src/xwidget.h (struct xwidget, struct xwidget_view): Add
GStreamer specific fields for media xwidgets.
(XWIDGET_VIEW_GTK_P): New macro.
---
 configure.ac             |  13 +
 doc/lispref/display.texi |  12 +
 etc/NEWS                 |   7 +
 src/Makefile.in          |   7 +-
 src/xterm.c              |  16 +-
 src/xwidget.c            | 536 +++++++++++++++++++++++++++++++--------
 src/xwidget.h            |  25 ++
 7 files changed, 508 insertions(+), 108 deletions(-)

diff --git a/configure.ac b/configure.ac
index c231c2ceae..ae3f28f8db 100644
--- a/configure.ac
+++ b/configure.ac
@@ -485,6 +485,7 @@ AC_DEFUN
 OPTION_DEFAULT_ON([zlib],[don't compile with zlib decompression support])
 OPTION_DEFAULT_ON([modules],[don't compile with dynamic modules support])
 OPTION_DEFAULT_ON([threads],[don't compile with elisp threading support])
+OPTION_DEFAULT_OFF([gstreamer],[compile with xwidget video playback support using GStreamer])
 OPTION_DEFAULT_OFF([native-compilation],[compile with Emacs Lisp native compiler support])
 OPTION_DEFAULT_OFF([cygwin32-native-compilation],[use native compilation on 32-bit Cygwin])
 
@@ -2834,6 +2835,18 @@ AC_DEFUN
 fi
 AC_SUBST(XWIDGETS_OBJ)
 
+HAVE_GSTREAMER=no
+if test "$with_gstreamer" != "no" && test "${HAVE_X_WINDOWS}" = "yes" \
+   && test "${HAVE_XWIDGETS}" = "yes"; then
+  EMACS_CHECK_MODULES([GSTREAMER], "gstreamer-1.0 gstreamer-video-1.0")
+  if test "$HAVE_GSTREAMER" = "yes"; then
+    AC_DEFINE(HAVE_GSTREAMER, 1, [Define to 1 if using GStreamer for video support in xwidgets.])
+  fi
+fi
+
+AC_SUBST(GSTREAMER_CFLAGS)
+AC_SUBST(GSTREAMER_LIBS)
+
 CFLAGS=$OLD_CFLAGS
 LIBS=$OLD_LIBS
 
diff --git a/doc/lispref/display.texi b/doc/lispref/display.texi
index dd2c6e003f..1b4a3d9a22 100644
--- a/doc/lispref/display.texi
+++ b/doc/lispref/display.texi
@@ -6797,6 +6797,8 @@ Xwidgets
 @table @code
 @item webkit
 The WebKit component.
+@item media
+A media widget that allows to display videos inside Emacs buffers.
 @end table
 
 The @var{width} and @var{height} arguments specify the widget size in
@@ -7004,6 +7006,16 @@ Xwidgets
 be passed as an index to @code{xwidget-webkit-goto-history}.
 @end defun
 
+@defun xwidget-media-pause xwidget
+Suspend playback of @var{xwidget}, a media xwidget.  You can later
+resume playback using @code{xwidget-media-play}.
+@end defun
+
+@defun xwidget-media-pause xwidget
+Start or resume playback of @var{xwidget}, a media xwidget.  You can
+later pause playback using @code{xwidget-media-pause}.
+@end defun
+
 @node Buttons
 @section Buttons
 @cindex buttons in buffers
diff --git a/etc/NEWS b/etc/NEWS
index cee2844be3..8cb5495b4e 100644
--- a/etc/NEWS
+++ b/etc/NEWS
@@ -488,6 +488,13 @@ This is a convenience function to extract the field data from
 
 ** Xwidgets
 
++++
+*** New xwidget type 'media'.
+This xwidget type allows to display video content inside Emacs buffers.
+You must have the GStreamer library installed, along with the plugins
+"xvimagesink", "queue", "videotestsrc" and "tee", and Emacs must be
+built with '--with-gstreamer'.
+
 ---
 *** New user option 'xwidget-webkit-buffer-name-format'.
 Using this option you can control how the xwidget-webkit buffers are
diff --git a/src/Makefile.in b/src/Makefile.in
index 4c5535f8ad..646392c051 100644
--- a/src/Makefile.in
+++ b/src/Makefile.in
@@ -333,6 +333,9 @@ LIBGMP =
 LIBGCCJIT_LIBS = @LIBGCCJIT_LIBS@
 LIBGCCJIT_CFLAGS = @LIBGCCJIT_CFLAGS@
 
+GSTREAMER_LIBS = @GSTREAMER_LIBS@
+GSTREAMER_CFLAGS = @GSTREAMER_CFLAGS@
+
 ## dynlib.o if necessary, else empty
 DYNLIB_OBJ = @DYNLIB_OBJ@
 
@@ -377,7 +380,7 @@ EMACS_CFLAGS=
   $(WEBKIT_CFLAGS) $(WEBP_CFLAGS) $(LCMS2_CFLAGS) \
   $(SETTINGS_CFLAGS) $(FREETYPE_CFLAGS) $(FONTCONFIG_CFLAGS) \
   $(HARFBUZZ_CFLAGS) $(LIBOTF_CFLAGS) $(M17N_FLT_CFLAGS) $(DEPFLAGS) \
-  $(LIBSYSTEMD_CFLAGS) $(JSON_CFLAGS) \
+  $(LIBSYSTEMD_CFLAGS) $(JSON_CFLAGS) $(GSTREAMER_CFLAGS) \
   $(LIBGNUTLS_CFLAGS) $(NOTIFY_CFLAGS) $(CAIRO_CFLAGS) \
   $(WERROR_CFLAGS)
 ALL_CFLAGS = $(EMACS_CFLAGS) $(WARN_CFLAGS) $(CFLAGS)
@@ -524,7 +527,7 @@ LIBES =
    $(FREETYPE_LIBS) $(FONTCONFIG_LIBS) $(HARFBUZZ_LIBS) $(LIBOTF_LIBS) $(M17N_FLT_LIBS) \
    $(LIBGNUTLS_LIBS) $(LIB_PTHREAD) $(GETADDRINFO_A_LIBS) $(LCMS2_LIBS) \
    $(NOTIFY_LIBS) $(LIB_MATH) $(LIBZ) $(LIBMODULES) $(LIBSYSTEMD_LIBS) \
-   $(JSON_LIBS) $(LIBGMP) $(LIBGCCJIT_LIBS)
+   $(JSON_LIBS) $(LIBGMP) $(LIBGCCJIT_LIBS) $(GSTREAMER_LIBS)
 
 ## FORCE it so that admin/unidata can decide whether this file is
 ## up-to-date.  Although since charprop depends on bootstrap-emacs,
diff --git a/src/xterm.c b/src/xterm.c
index 816b6dc5a8..a7116d9d95 100644
--- a/src/xterm.c
+++ b/src/xterm.c
@@ -4471,9 +4471,11 @@ x_scroll_run (struct window *w, struct run *run)
 					 view->y + view->clip_top,
 					 view->clip_right - view->clip_left,
 					 view->clip_bottom - view->clip_top);
-		      cairo_xlib_surface_set_size (view->cr_surface,
-						   view->clip_right - view->clip_left,
-						   view->clip_bottom - view->clip_top);
+
+		      if (XWIDGET_VIEW_GTK_P (view))
+			cairo_xlib_surface_set_size (view->cr_surface,
+						     view->clip_right - view->clip_left,
+						     view->clip_bottom - view->clip_top);
 		    }
 		  xwidget_expose (view);
 		  XFlush (dpy);
@@ -12940,6 +12942,14 @@ #define NUM_ARGV 10
         gtk_init (&argc, &argv2);
         request_sigio ();
 
+#ifdef HAVE_GSTREAMER
+	unrequest_sigio ();
+	/* gst_init might call XOpenDisplay, which fails if a signal
+	   is received.  */
+	gst_init (&argc, &argv2);
+	request_sigio ();
+#endif
+
         g_log_remove_handler ("GLib", id);
 
         xg_initialize ();
diff --git a/src/xwidget.c b/src/xwidget.c
index e1bf40ea43..ffed743412 100644
--- a/src/xwidget.c
+++ b/src/xwidget.c
@@ -40,6 +40,10 @@ Copyright (C) 2011-2021 Free Software Foundation, Inc.
 #include <JavaScriptCore/JavaScript.h>
 #include <cairo.h>
 #include <X11/Xlib.h>
+#ifdef HAVE_GSTREAMER
+#include <gst/video/videooverlay.h>
+#define XG_WINDOW_HANDLE "xg-window-handle"
+#endif
 #elif defined NS_IMPL_COCOA
 #include "nsxwidget.h"
 #endif
@@ -111,6 +115,12 @@ webkit_decide_policy_cb (WebKitWebView *,
 static void find_widget (GtkWidget *t, struct widget_search_data *);
 static void mouse_target_changed (WebKitWebView *, WebKitHitTestResult *, guint,
 				  gpointer);
+#ifdef HAVE_GSTREAMER
+static GstBusSyncReply gst_create_window (GstBus *, GstMessage *, gpointer);
+static gboolean gst_message_cb (GstBus *, GstMessage *, gpointer);
+static bool check_gstreamer_dependencies (void);
+static void unlink_gst_xwidget_view (struct xwidget_view *);
+#endif
 #endif
 
 
@@ -123,6 +133,7 @@ DEFUN ("make-xwidget",
 TYPE is a symbol which can take one of the following values:
 
 - webkit
+- media
 
 RELATED is nil, or an xwidget.  When constructing a WebKit widget, it
 will share the same settings and internal subprocess as RELATED.
@@ -140,9 +151,18 @@ DEFUN ("make-xwidget",
   CHECK_FIXNAT (width);
   CHECK_FIXNAT (height);
 
-  if (!EQ (type, Qwebkit))
+  if (!EQ (type, Qwebkit) && !EQ (type, Qmedia))
     error ("Bad xwidget type");
 
+#ifdef HAVE_GSTREAMER
+  if (EQ (type, Qmedia)
+      && !check_gstreamer_dependencies ())
+    error ("Your Emacs is missing run-time dependencies required for GStreamer support");
+#else
+  if (EQ (type, Qmedia))
+    error ("Your Emacs was not built with GStreamer support");
+#endif
+
   struct xwidget *xw = allocate_xwidget ();
   Lisp_Object val;
   xw->type = type;
@@ -180,103 +200,142 @@ DEFUN ("make-xwidget",
       gtk_window_resize (GTK_WINDOW (xw->widgetwindow_osr), xw->width,
                          xw->height);
 
-      if (EQ (xw->type, Qwebkit))
-        {
-	  WebKitWebView *related_view;
+      xw->xg_p = true;
+      WebKitWebView *related_view;
+
+      if (NILP (related)
+	  || !XWIDGETP (related)
+	  || !EQ (XXWIDGET (related)->type, Qwebkit))
+	{
+	  xw->widget_osr = webkit_web_view_new ();
+
+	  webkit_web_view_load_uri (WEBKIT_WEB_VIEW (xw->widget_osr),
+				    "about:blank");
+	  /* webkitgtk uses GSubprocess which sets sigaction causing
+	     Emacs to not catch SIGCHLD with its usual handle setup in
+	     'catch_child_signal'.  This resets the SIGCHLD sigaction.  */
+	  catch_child_signal ();
+	}
+      else
+	{
+	  related_view = WEBKIT_WEB_VIEW (XXWIDGET (related)->widget_osr);
+	  xw->widget_osr = webkit_web_view_new_with_related_view (related_view);
+	}
+
+      /* Enable the developer extras.  */
+      settings = webkit_web_view_get_settings (WEBKIT_WEB_VIEW (xw->widget_osr));
+      g_object_set (G_OBJECT (settings), "enable-developer-extras", TRUE, NULL);
+
+      if (xw->xg_p)
+	{
+	  gtk_widget_set_size_request (GTK_WIDGET (xw->widget_osr), xw->width,
+				       xw->height);
 
-	  if (NILP (related)
-	      || !XWIDGETP (related)
-	      || !EQ (XXWIDGET (related)->type, Qwebkit))
+	  if (EQ (xw->type, Qwebkit))
 	    {
-	      xw->widget_osr = webkit_web_view_new ();
-
-	      webkit_web_view_load_uri (WEBKIT_WEB_VIEW (xw->widget_osr),
-					"about:blank");
-	      /* webkitgtk uses GSubprocess which sets sigaction causing
-		 Emacs to not catch SIGCHLD with its usual handle setup in
-		 'catch_child_signal'.  This resets the SIGCHLD sigaction.  */
-	      catch_child_signal ();
+	      gtk_container_add (GTK_CONTAINER (xw->widgetwindow_osr),
+				 GTK_WIDGET (WEBKIT_WEB_VIEW (xw->widget_osr)));
 	    }
 	  else
 	    {
-	      related_view = WEBKIT_WEB_VIEW (XXWIDGET (related)->widget_osr);
-	      xw->widget_osr = webkit_web_view_new_with_related_view (related_view);
+	      gtk_container_add (GTK_CONTAINER (xw->widgetwindow_osr),
+				 xw->widget_osr);
 	    }
 
-	  /* Enable the developer extras.  */
-	  settings = webkit_web_view_get_settings (WEBKIT_WEB_VIEW (xw->widget_osr));
-	  g_object_set (G_OBJECT (settings), "enable-developer-extras", TRUE, NULL);
+	  gtk_widget_show (xw->widget_osr);
+	  gtk_widget_show (xw->widgetwindow_osr);
+	  synthesize_focus_in_event (xw->widgetwindow_osr);
+
+
+	  g_signal_connect (G_OBJECT (gtk_widget_get_window (xw->widgetwindow_osr)),
+			    "from-embedder", G_CALLBACK (from_embedder), NULL);
+	  g_signal_connect (G_OBJECT (gtk_widget_get_window (xw->widgetwindow_osr)),
+			    "to-embedder", G_CALLBACK (to_embedder), NULL);
+
+	  /* Store some xwidget data in the gtk widgets for convenient
+	     retrieval in the event handlers.  */
+	  g_object_set_data (G_OBJECT (xw->widget_osr), XG_XWIDGET, xw);
+	  g_object_set_data (G_OBJECT (xw->widgetwindow_osr), XG_XWIDGET, xw);
+
+	  /* signals */
+	  if (EQ (xw->type, Qwebkit))
+	    {
+	      g_signal_connect (G_OBJECT (xw->widget_osr),
+				"load-changed",
+				G_CALLBACK (webkit_view_load_changed_cb), xw);
+
+	      g_signal_connect (G_OBJECT (webkit_context),
+				"download-started",
+				G_CALLBACK (webkit_download_cb), xw);
+
+	      g_signal_connect (G_OBJECT (xw->widget_osr),
+				"decide-policy",
+				G_CALLBACK
+				(webkit_decide_policy_cb),
+				xw);
+
+	      g_signal_connect (G_OBJECT (xw->widget_osr),
+				"mouse-target-changed",
+				G_CALLBACK (mouse_target_changed),
+				xw);
+	      g_signal_connect (G_OBJECT (xw->widget_osr),
+				"create",
+				G_CALLBACK (webkit_create_cb),
+				xw);
+	      g_signal_connect (G_OBJECT (xw->widget_osr),
+				"script-dialog",
+				G_CALLBACK (webkit_script_dialog_cb),
+				NULL);
+	      g_signal_connect (G_OBJECT (xw->widget_osr),
+				"run-file-chooser",
+			    G_CALLBACK (run_file_chooser_cb),
+				NULL);
+	    }
+
+	  g_signal_connect (G_OBJECT (xw->widgetwindow_osr), "damage-event",
+			    G_CALLBACK (offscreen_damage_event), xw);
 	}
 
-      gtk_widget_set_size_request (GTK_WIDGET (xw->widget_osr), xw->width,
-                                   xw->height);
+      unblock_input ();
+    }
+#ifdef HAVE_GSTREAMER
+  else
+    {
+      xw->xg_p = false;
+      xw->gst_pipeline = gst_pipeline_new (NULL);
+      xw->gst_source = gst_element_factory_make ("videotestsrc", NULL);
+      xw->gst_tee = gst_element_factory_make ("tee", NULL);
 
-      if (EQ (xw->type, Qwebkit))
-        {
-          gtk_container_add (GTK_CONTAINER (xw->widgetwindow_osr),
-                             GTK_WIDGET (WEBKIT_WEB_VIEW (xw->widget_osr)));
-        }
-      else
-        {
-          gtk_container_add (GTK_CONTAINER (xw->widgetwindow_osr),
-                             xw->widget_osr);
-        }
+      GstBus *bus = gst_pipeline_get_bus (GST_PIPELINE (xw->gst_pipeline));
+      GstElement *fakesink = gst_element_factory_make ("fakesink", NULL);
+      GstElement *queue = gst_element_factory_make ("queue", NULL);
 
-      gtk_widget_show (xw->widget_osr);
-      gtk_widget_show (xw->widgetwindow_osr);
-      synthesize_focus_in_event (xw->widgetwindow_osr);
+      g_object_set (G_OBJECT (fakesink), "sync", TRUE, NULL);
 
+      gst_bus_set_sync_handler (bus, gst_create_window, xw->gst_pipeline, NULL);
+      gst_bus_add_signal_watch (bus);
+      g_signal_connect (G_OBJECT (bus), "message",
+			G_CALLBACK (gst_message_cb), xw->gst_pipeline);
 
-      g_signal_connect (G_OBJECT (gtk_widget_get_window (xw->widgetwindow_osr)),
-			"from-embedder", G_CALLBACK (from_embedder), NULL);
-      g_signal_connect (G_OBJECT (gtk_widget_get_window (xw->widgetwindow_osr)),
-			"to-embedder", G_CALLBACK (to_embedder), NULL);
+      xw->gst_fakesink = fakesink;
 
-      /* Store some xwidget data in the gtk widgets for convenient
-         retrieval in the event handlers.  */
-      g_object_set_data (G_OBJECT (xw->widget_osr), XG_XWIDGET, xw);
-      g_object_set_data (G_OBJECT (xw->widgetwindow_osr), XG_XWIDGET, xw);
+      gst_object_ref (xw->gst_pipeline);
+      gst_object_ref (xw->gst_source);
+      gst_object_ref (xw->gst_tee);
+      gst_object_ref (xw->gst_fakesink);
 
-      /* signals */
-      if (EQ (xw->type, Qwebkit))
-        {
-          g_signal_connect (G_OBJECT (xw->widget_osr),
-                            "load-changed",
-                            G_CALLBACK (webkit_view_load_changed_cb), xw);
-
-          g_signal_connect (G_OBJECT (webkit_context),
-                            "download-started",
-                            G_CALLBACK (webkit_download_cb), xw);
-
-          g_signal_connect (G_OBJECT (xw->widget_osr),
-                            "decide-policy",
-                            G_CALLBACK
-                            (webkit_decide_policy_cb),
-                            xw);
-
-	  g_signal_connect (G_OBJECT (xw->widget_osr),
-			    "mouse-target-changed",
-			    G_CALLBACK (mouse_target_changed),
-			    xw);
-	  g_signal_connect (G_OBJECT (xw->widget_osr),
-			    "create",
-			    G_CALLBACK (webkit_create_cb),
-			    xw);
-	  g_signal_connect (G_OBJECT (xw->widget_osr),
-			    "script-dialog",
-			    G_CALLBACK (webkit_script_dialog_cb),
-			    NULL);
-	  g_signal_connect (G_OBJECT (xw->widget_osr),
-			    "run-file-chooser",
-			    G_CALLBACK (run_file_chooser_cb),
-			    NULL);
-        }
+      gst_bin_add_many (GST_BIN (xw->gst_pipeline), xw->gst_source,
+			xw->gst_tee, queue, fakesink, NULL);
 
-      g_signal_connect (G_OBJECT (xw->widgetwindow_osr), "damage-event",
-			G_CALLBACK (offscreen_damage_event), xw);
+      gst_element_link_many (xw->gst_tee, queue, fakesink, NULL);
 
-      unblock_input ();
+      if (!gst_element_link (xw->gst_source, xw->gst_tee))
+	emacs_abort ();
+
+      if (!gst_element_set_state (xw->gst_pipeline, GST_STATE_PAUSED))
+	emacs_abort ();
     }
+#endif
 #elif defined NS_IMPL_COCOA
   nsxwidget_init (xw);
 #endif
@@ -335,6 +394,9 @@ DEFUN ("xwidget-perform-lispy-event",
     f = SELECTED_FRAME ();
 
 #ifdef USE_GTK
+  if (!xw->xg_p)
+    return Qnil;
+
   widget = gtk_window_get_focus (GTK_WINDOW (xw->widgetwindow_osr));
 
   if (!widget)
@@ -888,7 +950,8 @@ xwidget_button (struct xwidget_view *view,
 		bool down_p, int x, int y, int button,
 		int modifier_state, Time time)
 {
-  if (NILP (XXWIDGET (view->model)->buffer))
+  if (NILP (XXWIDGET (view->model)->buffer)
+      || !XWIDGET_VIEW_GTK_P (view))
     return;
 
   record_osr_embedder (view);
@@ -947,7 +1010,7 @@ xwidget_motion_or_crossing (struct xwidget_view *view, const XEvent *event)
   int y;
   GtkWidget *target;
 
-  if (NILP (model->buffer))
+  if (NILP (model->buffer) || !model->xg_p)
     return;
 
   xg_event = gdk_event_new (event->type == MotionNotify
@@ -1117,7 +1180,17 @@ xwidget_expose (struct xwidget_view *xv)
 {
   struct xwidget *xw = XXWIDGET (xv->model);
 
-  xv_do_draw (xv, xw);
+  if (xw->xg_p)
+    xv_do_draw (xv, xw);
+#ifdef HAVE_GSTREAMER
+  else if (EQ (xw->type, Qmedia) && XWIDGET_LIVE_P (xw))
+    {
+      if (xv->wdesc != None)
+	XMoveWindow (xv->dpy, xv->internal_window,
+		     -xv->clip_left, -xv->clip_top);
+      gst_video_overlay_expose (GST_VIDEO_OVERLAY (xv->video_sink));
+    }
+#endif
 }
 #endif /* USE_GTK */
 
@@ -1690,12 +1763,81 @@ x_draw_xwidget_glyph_string (struct glyph_string *s)
 				 CopyFromParent, CWEventMask, &a);
       XLowerWindow (xv->dpy, xv->wdesc);
       XDefineCursor (xv->dpy, xv->wdesc, xv->cursor);
-      xv->cr_surface = cairo_xlib_surface_create (xv->dpy,
-						  xv->wdesc,
-						  FRAME_DISPLAY_INFO (s->f)->visual,
-						  clip_right - clip_left,
-						  clip_bottom - clip_top);
-      xv->cr_context = cairo_create (xv->cr_surface);
+
+      if (XWIDGET_VIEW_GTK_P (xv))
+	{
+	  xv->cr_surface = cairo_xlib_surface_create (xv->dpy,
+						      xv->wdesc,
+						      FRAME_DISPLAY_INFO (s->f)->visual,
+						      clip_right - clip_left,
+						      clip_bottom - clip_top);
+	  xv->cr_context = cairo_create (xv->cr_surface);
+	}
+#ifdef HAVE_GSTREAMER
+      else
+	{
+	  XFlush (xv->dpy);
+
+	  if (EQ (xww->type, Qmedia) && XWIDGET_LIVE_P (xww))
+	    {
+	      GstPadTemplate *templ;
+	      GstPad *sinkpad;
+
+	      a.event_mask = ExposureMask;
+	      a.background_pixel = FRAME_BACKGROUND_PIXEL (xv->frame);
+
+	      xv->internal_window = XCreateWindow (xv->dpy, xv->wdesc,
+						   -clip_left, -clip_top,
+						   xww->width, xww->height,
+						   0, CopyFromParent, CopyFromParent,
+						   CopyFromParent,
+						   (CWEventMask | CWBackPixel), &a);
+
+	      XMapWindow (xv->dpy, xv->internal_window);
+	      XFlush (xv->dpy);
+
+	      Fputhash (make_fixnum (xv->internal_window), xvw, x_window_to_xwv_map);
+
+	      templ = gst_element_class_get_pad_template (GST_ELEMENT_GET_CLASS (xww->gst_tee),
+							  "src_%u");
+
+	      xv->teepad = gst_element_request_pad (xww->gst_tee,
+						    templ, NULL, NULL);
+	      xv->video_sink = gst_element_factory_make ("xvimagesink", NULL);
+	      xv->video_queue = gst_element_factory_make ("queue", NULL);
+
+	      gst_object_ref (xv->video_queue);
+	      gst_object_ref (xv->video_sink);
+
+	      g_object_set_data (G_OBJECT (xv->video_sink), XG_WINDOW_HANDLE,
+				 (gpointer) xv->internal_window);
+	      g_object_set (G_OBJECT (xv->video_sink), "display",
+			    FRAME_TERMINAL (xv->frame)->name, NULL);
+
+	      gst_video_overlay_handle_events (GST_VIDEO_OVERLAY (xv->video_sink),
+					       FALSE);
+
+	      gst_bin_add_many (GST_BIN (xww->gst_pipeline), xv->video_queue,
+				xv->video_sink, NULL);
+
+	      if (!gst_element_link (xv->video_queue, xv->video_sink))
+		emacs_abort ();
+
+	      gst_element_sync_state_with_parent (xv->video_queue);
+	      gst_element_sync_state_with_parent (xv->video_sink);
+
+	      sinkpad = gst_element_get_static_pad (xv->video_queue, "sink");
+	      gst_pad_link (xv->teepad, sinkpad);
+	      gst_object_unref (sinkpad);
+
+	      xv->sinkpad = sinkpad;
+
+	      gst_element_sync_state_with_parent (xww->gst_tee);
+
+	      xwidget_expose (xv);
+	    }
+	}
+#endif
       Fputhash (make_fixnum (xv->wdesc), xvw, x_window_to_xwv_map);
 
       moved = false;
@@ -1709,8 +1851,9 @@ x_draw_xwidget_glyph_string (struct glyph_string *s)
       XMoveResizeWindow (xv->dpy, xv->wdesc, x + clip_left, y + clip_top,
 			 clip_right - clip_left, clip_bottom - clip_top);
       XFlush (xv->dpy);
-      cairo_xlib_surface_set_size (xv->cr_surface, clip_right - clip_left,
-				   clip_bottom - clip_top);
+      if (XWIDGET_VIEW_GTK_P (xv))
+	cairo_xlib_surface_set_size (xv->cr_surface, clip_right - clip_left,
+				     clip_bottom - clip_top);
 #elif defined NS_IMPL_COCOA
       nsxwidget_move_view (xv, x + clip_left, y + clip_top);
 #endif
@@ -1740,8 +1883,10 @@ x_draw_xwidget_glyph_string (struct glyph_string *s)
 			     clip_bottom - clip_top);
 	    }
 	  XFlush (xv->dpy);
-	  cairo_xlib_surface_set_size (xv->cr_surface, clip_right - clip_left,
-				       clip_bottom - clip_top);
+
+	  if (xww->xg_p)
+	    cairo_xlib_surface_set_size (xv->cr_surface, clip_right - clip_left,
+					 clip_bottom - clip_top);
 	}
 #elif defined NS_IMPL_COCOA
       nsxwidget_resize_view (xv, clip_right - clip_left,
@@ -1765,7 +1910,10 @@ x_draw_xwidget_glyph_string (struct glyph_string *s)
       if (!xwidget_hidden (xv))
 	{
 #ifdef USE_GTK
-	  gtk_widget_queue_draw (xww->widget_osr);
+	  if (xww->xg_p)
+	    gtk_widget_queue_draw (xww->widget_osr);
+	  else
+	    xwidget_expose (xv);
 #elif defined NS_IMPL_COCOA
 	  nsxwidget_set_needsdisplay (xv);
 #endif
@@ -1788,6 +1936,12 @@ #define CHECK_WEBKIT_WIDGET(xw)				\
   if (NILP (xw->buffer) || !EQ (xw->type, Qwebkit))	\
     error ("Not a WebKit widget")
 
+#ifdef HAVE_GSTREAMER
+#define CHECK_MEDIA_WIDGET(xw)				\
+  if (NILP (xw->buffer) || !EQ (xw->type, Qmedia))	\
+    error ("Not a WebKit widget")
+#endif
+
 /* Macro that checks xwidget hold webkit web view first.  */
 #define WEBKIT_FN_INIT()						\
   CHECK_LIVE_XWIDGET (xwidget);						\
@@ -1989,6 +2143,14 @@ DEFUN ("xwidget-resize", Fxwidget_resize, Sxwidget_resize, 3, 3, 0,
 #ifdef USE_GTK
 	      xv->just_resized = true;
 	      SET_FRAME_GARBAGED (xv->frame);
+#ifdef HAVE_GSTREAMER
+	      if (EQ (xw->type, Qmedia))
+		{
+		  XResizeWindow (xv->dpy, xv->internal_window,
+				 xw->width, xw->height);
+		}
+#endif
+
 #else
 	      wset_redisplay (XWINDOW (xv->w));
 #endif
@@ -2000,7 +2162,7 @@ DEFUN ("xwidget-resize", Fxwidget_resize, Sxwidget_resize, 3, 3, 0,
 
   /* If there is an offscreen widget resize it first.  */
 #ifdef USE_GTK
-  if (xw->widget_osr)
+  if (xw->xg_p && xw->widget_osr)
     {
       gtk_window_resize (GTK_WINDOW (xw->widgetwindow_osr), xw->width,
                          xw->height);
@@ -2030,9 +2192,14 @@ DEFUN ("xwidget-size-request",
 {
   CHECK_LIVE_XWIDGET (xwidget);
 #ifdef USE_GTK
-  GtkRequisition requisition;
-  gtk_widget_size_request (XXWIDGET (xwidget)->widget_osr, &requisition);
-  return list2i (requisition.width, requisition.height);
+  if (XXWIDGET (xwidget)->xg_p)
+    {
+      GtkRequisition requisition;
+      gtk_widget_size_request (XXWIDGET (xwidget)->widget_osr, &requisition);
+      return list2i (requisition.width, requisition.height);
+    }
+
+  return Qnil;
 #elif defined NS_IMPL_COCOA
   return nsxwidget_get_size (XXWIDGET (xwidget));
 #endif
@@ -2115,18 +2282,29 @@ DEFUN ("delete-xwidget-view",
 #ifdef USE_GTK
   struct xwidget *xw = XXWIDGET (xv->model);
   GdkWindow *w;
-
+#ifdef HAVE_GSTREAMER
+  block_input ();
+  if (XWIDGET_LIVE_P (xw) && EQ (xw->type, Qmedia))
+    unlink_gst_xwidget_view (xv);
+  unblock_input ();
+#endif
   if (xv->wdesc != None)
     {
       block_input ();
-      cairo_destroy (xv->cr_context);
-      cairo_surface_destroy (xv->cr_surface);
+      if (XWIDGET_VIEW_GTK_P (xv))
+	{
+	  cairo_destroy (xv->cr_context);
+	  cairo_surface_destroy (xv->cr_surface);
+	}
       XDestroyWindow (xv->dpy, xv->wdesc);
       Fremhash (make_fixnum (xv->wdesc), x_window_to_xwv_map);
+
+      if (XWIDGET_LIVE_P (xw) && EQ (xw->type, Qmedia))
+	Fremhash (make_fixnum (xv->internal_window), x_window_to_xwv_map);
       unblock_input ();
     }
 
-  if (xw->embedder_view == xv && !NILP (xw->buffer))
+  if (xw->xg_p && xw->embedder_view == xv && !NILP (xw->buffer))
     {
       w = gtk_widget_get_window (xw->widgetwindow_osr);
 
@@ -2555,6 +2733,77 @@ DEFUN ("xwidget-webkit-back-forward-list", Fxwidget_webkit_back_forward_list,
 
   return list3 (back, here, forward);
 }
+
+#ifdef HAVE_GSTREAMER
+DEFUN ("xwidget-media-pause", Fxwidget_media_pause, Sxwidget_media_pause,
+       1, 1, 0, doc: /* Pause specified media XWIDGET.
+This causes playback to stop until you resume it using
+`xwidget-media-play'.  */)
+  (Lisp_Object xwidget)
+{
+  struct xwidget *xw;
+  struct xwidget_view *xv;
+  Lisp_Object tem;
+
+  CHECK_LIVE_XWIDGET (xwidget);
+  xw = XXWIDGET (xwidget);
+  CHECK_MEDIA_WIDGET (xw);
+
+  block_input ();
+  gst_element_set_state (xw->gst_pipeline, GST_STATE_PAUSED);
+  gst_element_sync_state_with_parent (xw->gst_tee);
+  gst_element_sync_state_with_parent (xw->gst_fakesink);
+
+  for (tem = internal_xwidget_view_list; CONSP (tem); tem = XCDR (tem))
+    {
+      xv = XXWIDGET_VIEW (XCAR (tem));
+
+      if (EQ (xv->model, xwidget))
+	{
+	  gst_element_sync_state_with_parent (xv->video_queue);
+	  gst_element_sync_state_with_parent (xv->video_sink);
+	}
+    }
+
+  unblock_input ();
+
+  return Qnil;
+}
+
+DEFUN ("xwidget-media-play", Fxwidget_media_play, Sxwidget_media_play,
+       1, 1, 0, doc: /* Resume playback of specified media XWIDGET.  */)
+  (Lisp_Object xwidget)
+{
+  struct xwidget *xw;
+  struct xwidget_view *xv;
+  Lisp_Object tem;
+
+  CHECK_LIVE_XWIDGET (xwidget);
+  xw = XXWIDGET (xwidget);
+  CHECK_MEDIA_WIDGET (xw);
+
+  block_input ();
+  gst_element_set_state (xw->gst_pipeline, GST_STATE_PLAYING);
+  gst_element_sync_state_with_parent (xw->gst_tee);
+  gst_element_sync_state_with_parent (xw->gst_fakesink);
+
+  for (tem = internal_xwidget_view_list; CONSP (tem); tem = XCDR (tem))
+    {
+      xv = XXWIDGET_VIEW (XCAR (tem));
+
+      if (EQ (xv->model, xwidget))
+	{
+	  gst_element_sync_state_with_parent (xv->video_queue);
+	  gst_element_sync_state_with_parent (xv->video_sink);
+	}
+    }
+
+  gst_element_sync_state_with_parent (xw->gst_tee);
+  unblock_input ();
+
+  return Qnil;
+}
+#endif
 #endif
 
 void
@@ -2584,6 +2833,7 @@ syms_of_xwidget (void)
   defsubr (&Sxwidget_webkit_zoom);
   defsubr (&Sxwidget_webkit_execute_script);
   DEFSYM (Qwebkit, "webkit");
+  DEFSYM (Qmedia, "media");
 
   defsubr (&Sxwidget_size_request);
   defsubr (&Sdelete_xwidget_view);
@@ -2600,6 +2850,10 @@ syms_of_xwidget (void)
 #ifdef USE_GTK
   defsubr (&Sxwidget_webkit_load_html);
   defsubr (&Sxwidget_webkit_back_forward_list);
+#ifdef HAVE_GSTREAMER
+  defsubr (&Sxwidget_media_pause);
+  defsubr (&Sxwidget_media_play);
+#endif
 #endif
   defsubr (&Skill_xwidget);
 
@@ -2862,7 +3116,7 @@ kill_xwidget (struct xwidget *xw)
 #ifdef USE_GTK
   xw->buffer = Qnil;
 
-  if (xw->widget_osr && xw->widgetwindow_osr)
+  if (xw->xg_p && xw->widget_osr && xw->widgetwindow_osr)
     {
       gtk_widget_destroy (xw->widget_osr);
       gtk_widget_destroy (xw->widgetwindow_osr);
@@ -2882,6 +3136,26 @@ kill_xwidget (struct xwidget *xw)
 	}
     }
 
+#ifdef HAVE_GSTREAMER
+  Lisp_Object tem;
+
+  if (EQ (xw->type, Qmedia))
+    {
+      for (tem = internal_xwidget_view_list; CONSP (tem);
+	   tem = XCDR (tem))
+	{
+	  struct xwidget_view *xv = XXWIDGET_VIEW (XCAR (tem));
+
+	  if (XXWIDGET (xv->model) == xw)
+	    unlink_gst_xwidget_view (xv);
+	}
+      gst_object_unref (xw->gst_tee);
+      gst_object_unref (xw->gst_pipeline);
+      gst_object_unref (xw->gst_source);
+      gst_object_unref (xw->gst_fakesink);
+    }
+#endif
+
   xw->widget_osr = NULL;
   xw->widgetwindow_osr = NULL;
   xw->find_text = NULL;
@@ -2908,3 +3182,59 @@ kill_buffer_xwidgets (Lisp_Object buffer)
       }
     }
 }
+
+#ifdef HAVE_GSTREAMER
+static GstBusSyncReply
+gst_create_window (GstBus *bus, GstMessage *msg, gpointer user_data)
+{
+  GstObject *elm;
+  Window wdesc;
+
+  if (!gst_is_video_overlay_prepare_window_handle_message (msg))
+    return GST_BUS_PASS;
+
+  elm = GST_MESSAGE_SRC (msg);
+  wdesc = (Window) g_object_get_data (G_OBJECT (elm), XG_WINDOW_HANDLE);
+
+  gst_video_overlay_set_window_handle (GST_VIDEO_OVERLAY (elm), wdesc);
+
+  return GST_BUS_DROP;
+}
+
+static gboolean
+gst_message_cb (GstBus *bus, GstMessage *message, gpointer user_data)
+{
+  return TRUE;
+}
+
+static bool
+check_gstreamer_dependencies (void)
+{
+  GstRegistry *registry = gst_registry_get ();
+
+  return (gst_registry_find_feature (registry, "xvimagesink",
+				     GST_TYPE_ELEMENT_FACTORY)
+	  && gst_registry_find_feature (registry, "queue",
+					GST_TYPE_ELEMENT_FACTORY)
+	  && gst_registry_find_feature (registry, "videotestsrc",
+					GST_TYPE_ELEMENT_FACTORY)
+	  && gst_registry_find_feature (registry, "tee",
+					GST_TYPE_ELEMENT_FACTORY));
+}
+
+static void
+unlink_gst_xwidget_view (struct xwidget_view *xv)
+{
+  struct xwidget *xw = XXWIDGET (xv->model);
+
+  gst_pad_unlink (xv->teepad, xv->sinkpad);
+  gst_element_unlink (xv->video_queue, xv->video_sink);
+
+  gst_element_set_state (xv->video_queue, GST_STATE_NULL);
+  gst_element_set_state (xv->video_sink, GST_STATE_NULL);
+  gst_bin_remove_many (GST_BIN (xw->gst_pipeline), xv->video_sink,
+		       xv->video_queue, NULL);
+  gst_object_unref (xv->video_sink);
+  gst_object_unref (xv->video_queue);
+}
+#endif
diff --git a/src/xwidget.h b/src/xwidget.h
index 78fe865dd8..66f7fb2c4c 100644
--- a/src/xwidget.h
+++ b/src/xwidget.h
@@ -39,6 +39,10 @@ #define XWIDGET_H_INCLUDED
 #import "nsxwidget.h"
 #endif
 
+#ifdef HAVE_GSTREAMER
+#include <gst/gst.h>
+#endif
+
 struct xwidget
 {
   union vectorlike_header header;
@@ -65,6 +69,16 @@ #define XWIDGET_H_INCLUDED
   char *find_text;
 
 #if defined (USE_GTK)
+  /* Whether or not the widget is used to display GTK widgets. */
+  bool xg_p;
+
+#ifdef HAVE_GSTREAMER
+  GstElement *gst_tee;
+  GstElement *gst_pipeline;
+  GstElement *gst_source;
+  GstElement *gst_fakesink;
+#endif
+
   /* For offscreen widgets, unused if not osr.  */
   GtkWidget *widget_osr;
   GtkWidget *widgetwindow_osr;
@@ -112,6 +126,14 @@ #define XWIDGET_H_INCLUDED
   Emacs_Cursor cursor;
   struct frame *frame;
 
+#ifdef HAVE_GSTREAMER
+  Window internal_window;
+  GstPad *teepad;
+  GstPad *sinkpad;
+  GstElement *video_sink;
+  GstElement *video_queue;
+#endif
+
   cairo_surface_t *cr_surface;
   cairo_t *cr_context;
   int just_resized;
@@ -159,6 +181,9 @@ #define XXWIDGET_VIEW(a) (eassert (XWIDGET_VIEW_P (a)), \
 #define CHECK_XWIDGET_VIEW(x) \
   CHECK_TYPE (XWIDGET_VIEW_P (x), Qxwidget_view_p, x)
 
+#define XWIDGET_VIEW_GTK_P(x) \
+  XXWIDGET ((x)->model)->xg_p
+
 #define XG_XWIDGET "emacs_xwidget"
 #define XG_XWIDGET_VIEW "emacs_xwidget_view"
 
-- 
2.31.1

Re: GStreamer xwidget

[T.V Raman]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=gb18030, Size: 579 bytes --]


So this takes away the primary reason why one spends all one's time in
emacs; it's because all information in Emacs is manipulable as text and
more importantly as structured information with a bit of work, eg
specialized modes.

This GStreamer widget that displays text takes Emacs down the deep dark
rathole of "What You See Is All You Have" world down which many tools
have gone, never to return.

I sincerely hope Emacs doesn't go down that rathole  as well.

-- 

Thanks,

--Raman(I Search, I Find, I Misplace, I Research)
7©4 Id: kg:/m/0285kf1  •0Ü8

Re: GStreamer xwidget

[Po Lu]

"T.V Raman" <raman@google.com> writes:

> So this takes away the primary reason why one spends all one's time in
> emacs; it's because all information in Emacs is manipulable as text and
> more importantly as structured information with a bit of work, eg
> specialized modes.

The GStreamer widget doesn't display text, it only displays media.  Just
as we support displaying images and such inside buffers.

If eventually it gains support for subtitles, then they will be sent to
Lisp code as an xwidget event, and not displayed on top of the widget.

So please explain why you think this is different from an image spec,
and more specifically an animated image.

> I sincerely hope Emacs doesn't go down that rathole  as well.

It's not a rathole.

Re: GStreamer xwidget

[Lars Ingebrigtsen]

Po Lu <luangruo@yahoo.com> writes:

> Right now, it can only display a test pattern, but getting that to work
> and perform decently was a challenge (for instance, offscreen rendering
> is not an option,

Hm...  isn't the webkit rendering done offscreen?  And that's able to
play videos just fine?

> instead the pipeline is dynamically built based on
> available xwidget views, utilizing hardware acceleration and video
> decoding if available).
>
> WDYT?

Sounds good to me.

Do you have a code snippet that will display one of these widgets?  Then
I can test the patch here.

Skimming the patch, it looks good to me.  The one extremely minor nit I
have is this superfluous {}:

> +#ifdef HAVE_GSTREAMER
> +	      if (EQ (xw->type, Qmedia))
> +		{
> +		  XResizeWindow (xv->dpy, xv->internal_window,
> +				 xw->width, xw->height);
> +		}

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: GStreamer xwidget

[Po Lu]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Hm...  isn't the webkit rendering done offscreen?  And that's able to
> play videos just fine?

Not on macOS, which is presumably where you tested the video playback,
where a single xwidget can be displayed by only one Emacs window at a
time.  On GNU systems, the offscreen rendering causes some video
playback to consume large amounts of CPU (as it can't utilize hardware
acceleration for video decoding).

And the more immediate problem this tries to solve is video playback not
working at all in most newer versions of WebKitGTK.

> Sounds good to me.
>
> Do you have a code snippet that will display one of these widgets?  Then
> I can test the patch here.

Try this:

  (require 'xwidget)

  (setq xw (make-xwidget 'media "foo" 100 100))
  (insert (propertize "foo" 'display (list 'xwidget :xwidget xw)))
  (xwidget-media-play xw)

> Skimming the patch, it looks good to me.  The one extremely minor nit I
> have is this superfluous {}:

>> +#ifdef HAVE_GSTREAMER
>> +	      if (EQ (xw->type, Qmedia))
>> +		{
>> +		  XResizeWindow (xv->dpy, xv->internal_window,
>> +				 xw->width, xw->height);
>> +		}

It would be ugly to wrap the arguments to call there, and still rely on
implicit braces.  IMO.

But if that's really a problem, I can remove it.

Thanks.

Re: GStreamer xwidget

[Lars Ingebrigtsen]

Po Lu <luangruo@yahoo.com> writes:

> Not on macOS, which is presumably where you tested the video playback,
> where a single xwidget can be displayed by only one Emacs window at a
> time.  On GNU systems, the offscreen rendering causes some video
> playback to consume large amounts of CPU (as it can't utilize hardware
> acceleration for video decoding).

Ah, I see.

> Try this:
>
>   (require 'xwidget)
>
>   (setq xw (make-xwidget 'media "foo" 100 100))
>   (insert (propertize "foo" 'display (list 'xwidget :xwidget xw)))
>   (xwidget-media-play xw)

Thanks.  But I get the following build error:

xwidget.c: In function 'Fdelete_xwidget_view':
xwidget.c:2303:34: error: 'struct xwidget_view' has no member named 'internal_window'
 2303 |         Fremhash (make_fixnum (xv->internal_window), x_window_to_xwv_map);
      |                                  ^~
  make[1]: *** [Makefile:393: xwidget.o] Error  1


-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: GStreamer xwidget

[Po Lu]

[-- Attachment #1: Type: text/plain, Size: 434 bytes --]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Thanks.  But I get the following build error:
>
> xwidget.c: In function 'Fdelete_xwidget_view':
> xwidget.c:2303:34: error: 'struct xwidget_view' has no member named 'internal_window'
>  2303 |         Fremhash (make_fixnum (xv->internal_window), x_window_to_xwv_map);
>       |                                  ^~
>   make[1]: *** [Makefile:393: xwidget.o] Error  1

Try this instead:


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Implement-a-media-xwidget-based-on-GStreamer.patch --]
[-- Type: text/x-patch, Size: 31924 bytes --]

From 310c856151e01f4c82a185eaa56314d15b1e706c Mon Sep 17 00:00:00 2001
From: Po Lu <luangruo@yahoo.com>
Date: Thu, 18 Nov 2021 18:10:45 +0800
Subject: [PATCH] Implement a media xwidget based on GStreamer

* configure.ac: Add option to use GStreamer for media playback.
* doc/lispref/display.texi (Xwidgets): Document new functions
and xwidget type.

* src/Makefile.in (EMACS_CFLAGS): Add GStreamer cflags.
(LIBES): Add GStreamer linker options.

* etc/NEWS: Announce `media' xwidget.
* etc/xterm.c (x_scroll_run): Don't resize cairo surface
if not a GTK xwidget.
(x_term_init): Initialize GStreamer.

* src/xwidget.c (gst_create_window)
(gst_message_cb, unlink_gst_xwidget_view)
(check_gstreamer_dependencies): New functions.

(Fmake_xwidget): Add support for media xwidgets.
(Fxwidget_perform_lispy_event): Return nil if it doesn't make
sense to perform an event.
(Fxwidget_resize): Add support for media widgets.
(xwidget_button, xwidget_motion_or_crossing): Handle non-GTK
xwidgets.
(xwidget_expose): Handle media xwidgets.
(x_draw_xwidget_glyph_string): Set up GStreamer widgets if
appropriate.
(Fxwidget_size_request): Return nil on widgets that can't
have a size request.
(Fdelete_xwidget_view): Clean up after GStreamer.
(Fxwidget_media_pause, Fxwidget_media_play): New functions.

(syms_of_xwidget): New symbols and subrs.
(kill_xwidget): Clean up after GStreamer.

* src/xwidget.h (struct xwidget, struct xwidget_view): Add
GStreamer specific fields for media xwidgets.
(XWIDGET_VIEW_GTK_P): New macro.
---
 configure.ac             |  13 +
 doc/lispref/display.texi |  12 +
 etc/NEWS                 |   7 +
 src/Makefile.in          |   7 +-
 src/xterm.c              |  16 +-
 src/xwidget.c            | 537 +++++++++++++++++++++++++++++++--------
 src/xwidget.h            |  25 ++
 7 files changed, 509 insertions(+), 108 deletions(-)

diff --git a/configure.ac b/configure.ac
index c231c2ceae..ae3f28f8db 100644
--- a/configure.ac
+++ b/configure.ac
@@ -485,6 +485,7 @@ AC_DEFUN
 OPTION_DEFAULT_ON([zlib],[don't compile with zlib decompression support])
 OPTION_DEFAULT_ON([modules],[don't compile with dynamic modules support])
 OPTION_DEFAULT_ON([threads],[don't compile with elisp threading support])
+OPTION_DEFAULT_OFF([gstreamer],[compile with xwidget video playback support using GStreamer])
 OPTION_DEFAULT_OFF([native-compilation],[compile with Emacs Lisp native compiler support])
 OPTION_DEFAULT_OFF([cygwin32-native-compilation],[use native compilation on 32-bit Cygwin])
 
@@ -2834,6 +2835,18 @@ AC_DEFUN
 fi
 AC_SUBST(XWIDGETS_OBJ)
 
+HAVE_GSTREAMER=no
+if test "$with_gstreamer" != "no" && test "${HAVE_X_WINDOWS}" = "yes" \
+   && test "${HAVE_XWIDGETS}" = "yes"; then
+  EMACS_CHECK_MODULES([GSTREAMER], "gstreamer-1.0 gstreamer-video-1.0")
+  if test "$HAVE_GSTREAMER" = "yes"; then
+    AC_DEFINE(HAVE_GSTREAMER, 1, [Define to 1 if using GStreamer for video support in xwidgets.])
+  fi
+fi
+
+AC_SUBST(GSTREAMER_CFLAGS)
+AC_SUBST(GSTREAMER_LIBS)
+
 CFLAGS=$OLD_CFLAGS
 LIBS=$OLD_LIBS
 
diff --git a/doc/lispref/display.texi b/doc/lispref/display.texi
index 8decff6fa8..c35d63f848 100644
--- a/doc/lispref/display.texi
+++ b/doc/lispref/display.texi
@@ -6797,6 +6797,8 @@ Xwidgets
 @table @code
 @item webkit
 The WebKit component.
+@item media
+A media widget that allows to display videos inside Emacs buffers.
 @end table
 
 The @var{width} and @var{height} arguments specify the widget size in
@@ -7004,6 +7006,16 @@ Xwidgets
 be passed as an index to @code{xwidget-webkit-goto-history}.
 @end defun
 
+@defun xwidget-media-pause xwidget
+Suspend playback of @var{xwidget}, a media xwidget.  You can later
+resume playback using @code{xwidget-media-play}.
+@end defun
+
+@defun xwidget-media-pause xwidget
+Start or resume playback of @var{xwidget}, a media xwidget.  You can
+later pause playback using @code{xwidget-media-pause}.
+@end defun
+
 @node Buttons
 @section Buttons
 @cindex buttons in buffers
diff --git a/etc/NEWS b/etc/NEWS
index cee2844be3..8cb5495b4e 100644
--- a/etc/NEWS
+++ b/etc/NEWS
@@ -488,6 +488,13 @@ This is a convenience function to extract the field data from
 
 ** Xwidgets
 
++++
+*** New xwidget type 'media'.
+This xwidget type allows to display video content inside Emacs buffers.
+You must have the GStreamer library installed, along with the plugins
+"xvimagesink", "queue", "videotestsrc" and "tee", and Emacs must be
+built with '--with-gstreamer'.
+
 ---
 *** New user option 'xwidget-webkit-buffer-name-format'.
 Using this option you can control how the xwidget-webkit buffers are
diff --git a/src/Makefile.in b/src/Makefile.in
index 4c5535f8ad..646392c051 100644
--- a/src/Makefile.in
+++ b/src/Makefile.in
@@ -333,6 +333,9 @@ LIBGMP =
 LIBGCCJIT_LIBS = @LIBGCCJIT_LIBS@
 LIBGCCJIT_CFLAGS = @LIBGCCJIT_CFLAGS@
 
+GSTREAMER_LIBS = @GSTREAMER_LIBS@
+GSTREAMER_CFLAGS = @GSTREAMER_CFLAGS@
+
 ## dynlib.o if necessary, else empty
 DYNLIB_OBJ = @DYNLIB_OBJ@
 
@@ -377,7 +380,7 @@ EMACS_CFLAGS=
   $(WEBKIT_CFLAGS) $(WEBP_CFLAGS) $(LCMS2_CFLAGS) \
   $(SETTINGS_CFLAGS) $(FREETYPE_CFLAGS) $(FONTCONFIG_CFLAGS) \
   $(HARFBUZZ_CFLAGS) $(LIBOTF_CFLAGS) $(M17N_FLT_CFLAGS) $(DEPFLAGS) \
-  $(LIBSYSTEMD_CFLAGS) $(JSON_CFLAGS) \
+  $(LIBSYSTEMD_CFLAGS) $(JSON_CFLAGS) $(GSTREAMER_CFLAGS) \
   $(LIBGNUTLS_CFLAGS) $(NOTIFY_CFLAGS) $(CAIRO_CFLAGS) \
   $(WERROR_CFLAGS)
 ALL_CFLAGS = $(EMACS_CFLAGS) $(WARN_CFLAGS) $(CFLAGS)
@@ -524,7 +527,7 @@ LIBES =
    $(FREETYPE_LIBS) $(FONTCONFIG_LIBS) $(HARFBUZZ_LIBS) $(LIBOTF_LIBS) $(M17N_FLT_LIBS) \
    $(LIBGNUTLS_LIBS) $(LIB_PTHREAD) $(GETADDRINFO_A_LIBS) $(LCMS2_LIBS) \
    $(NOTIFY_LIBS) $(LIB_MATH) $(LIBZ) $(LIBMODULES) $(LIBSYSTEMD_LIBS) \
-   $(JSON_LIBS) $(LIBGMP) $(LIBGCCJIT_LIBS)
+   $(JSON_LIBS) $(LIBGMP) $(LIBGCCJIT_LIBS) $(GSTREAMER_LIBS)
 
 ## FORCE it so that admin/unidata can decide whether this file is
 ## up-to-date.  Although since charprop depends on bootstrap-emacs,
diff --git a/src/xterm.c b/src/xterm.c
index 816b6dc5a8..a7116d9d95 100644
--- a/src/xterm.c
+++ b/src/xterm.c
@@ -4471,9 +4471,11 @@ x_scroll_run (struct window *w, struct run *run)
 					 view->y + view->clip_top,
 					 view->clip_right - view->clip_left,
 					 view->clip_bottom - view->clip_top);
-		      cairo_xlib_surface_set_size (view->cr_surface,
-						   view->clip_right - view->clip_left,
-						   view->clip_bottom - view->clip_top);
+
+		      if (XWIDGET_VIEW_GTK_P (view))
+			cairo_xlib_surface_set_size (view->cr_surface,
+						     view->clip_right - view->clip_left,
+						     view->clip_bottom - view->clip_top);
 		    }
 		  xwidget_expose (view);
 		  XFlush (dpy);
@@ -12940,6 +12942,14 @@ #define NUM_ARGV 10
         gtk_init (&argc, &argv2);
         request_sigio ();
 
+#ifdef HAVE_GSTREAMER
+	unrequest_sigio ();
+	/* gst_init might call XOpenDisplay, which fails if a signal
+	   is received.  */
+	gst_init (&argc, &argv2);
+	request_sigio ();
+#endif
+
         g_log_remove_handler ("GLib", id);
 
         xg_initialize ();
diff --git a/src/xwidget.c b/src/xwidget.c
index e1bf40ea43..fa460347a6 100644
--- a/src/xwidget.c
+++ b/src/xwidget.c
@@ -40,6 +40,10 @@ Copyright (C) 2011-2021 Free Software Foundation, Inc.
 #include <JavaScriptCore/JavaScript.h>
 #include <cairo.h>
 #include <X11/Xlib.h>
+#ifdef HAVE_GSTREAMER
+#include <gst/video/videooverlay.h>
+#define XG_WINDOW_HANDLE "xg-window-handle"
+#endif
 #elif defined NS_IMPL_COCOA
 #include "nsxwidget.h"
 #endif
@@ -111,6 +115,12 @@ webkit_decide_policy_cb (WebKitWebView *,
 static void find_widget (GtkWidget *t, struct widget_search_data *);
 static void mouse_target_changed (WebKitWebView *, WebKitHitTestResult *, guint,
 				  gpointer);
+#ifdef HAVE_GSTREAMER
+static GstBusSyncReply gst_create_window (GstBus *, GstMessage *, gpointer);
+static gboolean gst_message_cb (GstBus *, GstMessage *, gpointer);
+static bool check_gstreamer_dependencies (void);
+static void unlink_gst_xwidget_view (struct xwidget_view *);
+#endif
 #endif
 
 
@@ -123,6 +133,7 @@ DEFUN ("make-xwidget",
 TYPE is a symbol which can take one of the following values:
 
 - webkit
+- media
 
 RELATED is nil, or an xwidget.  When constructing a WebKit widget, it
 will share the same settings and internal subprocess as RELATED.
@@ -140,9 +151,18 @@ DEFUN ("make-xwidget",
   CHECK_FIXNAT (width);
   CHECK_FIXNAT (height);
 
-  if (!EQ (type, Qwebkit))
+  if (!EQ (type, Qwebkit) && !EQ (type, Qmedia))
     error ("Bad xwidget type");
 
+#ifdef HAVE_GSTREAMER
+  if (EQ (type, Qmedia)
+      && !check_gstreamer_dependencies ())
+    error ("Your Emacs is missing run-time dependencies required for GStreamer support");
+#else
+  if (EQ (type, Qmedia))
+    error ("Your Emacs was not built with GStreamer support");
+#endif
+
   struct xwidget *xw = allocate_xwidget ();
   Lisp_Object val;
   xw->type = type;
@@ -180,103 +200,142 @@ DEFUN ("make-xwidget",
       gtk_window_resize (GTK_WINDOW (xw->widgetwindow_osr), xw->width,
                          xw->height);
 
-      if (EQ (xw->type, Qwebkit))
-        {
-	  WebKitWebView *related_view;
+      xw->xg_p = true;
+      WebKitWebView *related_view;
+
+      if (NILP (related)
+	  || !XWIDGETP (related)
+	  || !EQ (XXWIDGET (related)->type, Qwebkit))
+	{
+	  xw->widget_osr = webkit_web_view_new ();
+
+	  webkit_web_view_load_uri (WEBKIT_WEB_VIEW (xw->widget_osr),
+				    "about:blank");
+	  /* webkitgtk uses GSubprocess which sets sigaction causing
+	     Emacs to not catch SIGCHLD with its usual handle setup in
+	     'catch_child_signal'.  This resets the SIGCHLD sigaction.  */
+	  catch_child_signal ();
+	}
+      else
+	{
+	  related_view = WEBKIT_WEB_VIEW (XXWIDGET (related)->widget_osr);
+	  xw->widget_osr = webkit_web_view_new_with_related_view (related_view);
+	}
+
+      /* Enable the developer extras.  */
+      settings = webkit_web_view_get_settings (WEBKIT_WEB_VIEW (xw->widget_osr));
+      g_object_set (G_OBJECT (settings), "enable-developer-extras", TRUE, NULL);
+
+      if (xw->xg_p)
+	{
+	  gtk_widget_set_size_request (GTK_WIDGET (xw->widget_osr), xw->width,
+				       xw->height);
 
-	  if (NILP (related)
-	      || !XWIDGETP (related)
-	      || !EQ (XXWIDGET (related)->type, Qwebkit))
+	  if (EQ (xw->type, Qwebkit))
 	    {
-	      xw->widget_osr = webkit_web_view_new ();
-
-	      webkit_web_view_load_uri (WEBKIT_WEB_VIEW (xw->widget_osr),
-					"about:blank");
-	      /* webkitgtk uses GSubprocess which sets sigaction causing
-		 Emacs to not catch SIGCHLD with its usual handle setup in
-		 'catch_child_signal'.  This resets the SIGCHLD sigaction.  */
-	      catch_child_signal ();
+	      gtk_container_add (GTK_CONTAINER (xw->widgetwindow_osr),
+				 GTK_WIDGET (WEBKIT_WEB_VIEW (xw->widget_osr)));
 	    }
 	  else
 	    {
-	      related_view = WEBKIT_WEB_VIEW (XXWIDGET (related)->widget_osr);
-	      xw->widget_osr = webkit_web_view_new_with_related_view (related_view);
+	      gtk_container_add (GTK_CONTAINER (xw->widgetwindow_osr),
+				 xw->widget_osr);
 	    }
 
-	  /* Enable the developer extras.  */
-	  settings = webkit_web_view_get_settings (WEBKIT_WEB_VIEW (xw->widget_osr));
-	  g_object_set (G_OBJECT (settings), "enable-developer-extras", TRUE, NULL);
+	  gtk_widget_show (xw->widget_osr);
+	  gtk_widget_show (xw->widgetwindow_osr);
+	  synthesize_focus_in_event (xw->widgetwindow_osr);
+
+
+	  g_signal_connect (G_OBJECT (gtk_widget_get_window (xw->widgetwindow_osr)),
+			    "from-embedder", G_CALLBACK (from_embedder), NULL);
+	  g_signal_connect (G_OBJECT (gtk_widget_get_window (xw->widgetwindow_osr)),
+			    "to-embedder", G_CALLBACK (to_embedder), NULL);
+
+	  /* Store some xwidget data in the gtk widgets for convenient
+	     retrieval in the event handlers.  */
+	  g_object_set_data (G_OBJECT (xw->widget_osr), XG_XWIDGET, xw);
+	  g_object_set_data (G_OBJECT (xw->widgetwindow_osr), XG_XWIDGET, xw);
+
+	  /* signals */
+	  if (EQ (xw->type, Qwebkit))
+	    {
+	      g_signal_connect (G_OBJECT (xw->widget_osr),
+				"load-changed",
+				G_CALLBACK (webkit_view_load_changed_cb), xw);
+
+	      g_signal_connect (G_OBJECT (webkit_context),
+				"download-started",
+				G_CALLBACK (webkit_download_cb), xw);
+
+	      g_signal_connect (G_OBJECT (xw->widget_osr),
+				"decide-policy",
+				G_CALLBACK
+				(webkit_decide_policy_cb),
+				xw);
+
+	      g_signal_connect (G_OBJECT (xw->widget_osr),
+				"mouse-target-changed",
+				G_CALLBACK (mouse_target_changed),
+				xw);
+	      g_signal_connect (G_OBJECT (xw->widget_osr),
+				"create",
+				G_CALLBACK (webkit_create_cb),
+				xw);
+	      g_signal_connect (G_OBJECT (xw->widget_osr),
+				"script-dialog",
+				G_CALLBACK (webkit_script_dialog_cb),
+				NULL);
+	      g_signal_connect (G_OBJECT (xw->widget_osr),
+				"run-file-chooser",
+			    G_CALLBACK (run_file_chooser_cb),
+				NULL);
+	    }
+
+	  g_signal_connect (G_OBJECT (xw->widgetwindow_osr), "damage-event",
+			    G_CALLBACK (offscreen_damage_event), xw);
 	}
 
-      gtk_widget_set_size_request (GTK_WIDGET (xw->widget_osr), xw->width,
-                                   xw->height);
+      unblock_input ();
+    }
+#ifdef HAVE_GSTREAMER
+  else
+    {
+      xw->xg_p = false;
+      xw->gst_pipeline = gst_pipeline_new (NULL);
+      xw->gst_source = gst_element_factory_make ("videotestsrc", NULL);
+      xw->gst_tee = gst_element_factory_make ("tee", NULL);
 
-      if (EQ (xw->type, Qwebkit))
-        {
-          gtk_container_add (GTK_CONTAINER (xw->widgetwindow_osr),
-                             GTK_WIDGET (WEBKIT_WEB_VIEW (xw->widget_osr)));
-        }
-      else
-        {
-          gtk_container_add (GTK_CONTAINER (xw->widgetwindow_osr),
-                             xw->widget_osr);
-        }
+      GstBus *bus = gst_pipeline_get_bus (GST_PIPELINE (xw->gst_pipeline));
+      GstElement *fakesink = gst_element_factory_make ("fakesink", NULL);
+      GstElement *queue = gst_element_factory_make ("queue", NULL);
 
-      gtk_widget_show (xw->widget_osr);
-      gtk_widget_show (xw->widgetwindow_osr);
-      synthesize_focus_in_event (xw->widgetwindow_osr);
+      g_object_set (G_OBJECT (fakesink), "sync", TRUE, NULL);
 
+      gst_bus_set_sync_handler (bus, gst_create_window, xw->gst_pipeline, NULL);
+      gst_bus_add_signal_watch (bus);
+      g_signal_connect (G_OBJECT (bus), "message",
+			G_CALLBACK (gst_message_cb), xw->gst_pipeline);
 
-      g_signal_connect (G_OBJECT (gtk_widget_get_window (xw->widgetwindow_osr)),
-			"from-embedder", G_CALLBACK (from_embedder), NULL);
-      g_signal_connect (G_OBJECT (gtk_widget_get_window (xw->widgetwindow_osr)),
-			"to-embedder", G_CALLBACK (to_embedder), NULL);
+      xw->gst_fakesink = fakesink;
 
-      /* Store some xwidget data in the gtk widgets for convenient
-         retrieval in the event handlers.  */
-      g_object_set_data (G_OBJECT (xw->widget_osr), XG_XWIDGET, xw);
-      g_object_set_data (G_OBJECT (xw->widgetwindow_osr), XG_XWIDGET, xw);
+      gst_object_ref (xw->gst_pipeline);
+      gst_object_ref (xw->gst_source);
+      gst_object_ref (xw->gst_tee);
+      gst_object_ref (xw->gst_fakesink);
 
-      /* signals */
-      if (EQ (xw->type, Qwebkit))
-        {
-          g_signal_connect (G_OBJECT (xw->widget_osr),
-                            "load-changed",
-                            G_CALLBACK (webkit_view_load_changed_cb), xw);
-
-          g_signal_connect (G_OBJECT (webkit_context),
-                            "download-started",
-                            G_CALLBACK (webkit_download_cb), xw);
-
-          g_signal_connect (G_OBJECT (xw->widget_osr),
-                            "decide-policy",
-                            G_CALLBACK
-                            (webkit_decide_policy_cb),
-                            xw);
-
-	  g_signal_connect (G_OBJECT (xw->widget_osr),
-			    "mouse-target-changed",
-			    G_CALLBACK (mouse_target_changed),
-			    xw);
-	  g_signal_connect (G_OBJECT (xw->widget_osr),
-			    "create",
-			    G_CALLBACK (webkit_create_cb),
-			    xw);
-	  g_signal_connect (G_OBJECT (xw->widget_osr),
-			    "script-dialog",
-			    G_CALLBACK (webkit_script_dialog_cb),
-			    NULL);
-	  g_signal_connect (G_OBJECT (xw->widget_osr),
-			    "run-file-chooser",
-			    G_CALLBACK (run_file_chooser_cb),
-			    NULL);
-        }
+      gst_bin_add_many (GST_BIN (xw->gst_pipeline), xw->gst_source,
+			xw->gst_tee, queue, fakesink, NULL);
 
-      g_signal_connect (G_OBJECT (xw->widgetwindow_osr), "damage-event",
-			G_CALLBACK (offscreen_damage_event), xw);
+      gst_element_link_many (xw->gst_tee, queue, fakesink, NULL);
 
-      unblock_input ();
+      if (!gst_element_link (xw->gst_source, xw->gst_tee))
+	emacs_abort ();
+
+      if (!gst_element_set_state (xw->gst_pipeline, GST_STATE_PAUSED))
+	emacs_abort ();
     }
+#endif
 #elif defined NS_IMPL_COCOA
   nsxwidget_init (xw);
 #endif
@@ -335,6 +394,9 @@ DEFUN ("xwidget-perform-lispy-event",
     f = SELECTED_FRAME ();
 
 #ifdef USE_GTK
+  if (!xw->xg_p)
+    return Qnil;
+
   widget = gtk_window_get_focus (GTK_WINDOW (xw->widgetwindow_osr));
 
   if (!widget)
@@ -888,7 +950,8 @@ xwidget_button (struct xwidget_view *view,
 		bool down_p, int x, int y, int button,
 		int modifier_state, Time time)
 {
-  if (NILP (XXWIDGET (view->model)->buffer))
+  if (NILP (XXWIDGET (view->model)->buffer)
+      || !XWIDGET_VIEW_GTK_P (view))
     return;
 
   record_osr_embedder (view);
@@ -947,7 +1010,7 @@ xwidget_motion_or_crossing (struct xwidget_view *view, const XEvent *event)
   int y;
   GtkWidget *target;
 
-  if (NILP (model->buffer))
+  if (NILP (model->buffer) || !model->xg_p)
     return;
 
   xg_event = gdk_event_new (event->type == MotionNotify
@@ -1117,7 +1180,17 @@ xwidget_expose (struct xwidget_view *xv)
 {
   struct xwidget *xw = XXWIDGET (xv->model);
 
-  xv_do_draw (xv, xw);
+  if (xw->xg_p)
+    xv_do_draw (xv, xw);
+#ifdef HAVE_GSTREAMER
+  else if (EQ (xw->type, Qmedia) && XWIDGET_LIVE_P (xw))
+    {
+      if (xv->wdesc != None)
+	XMoveWindow (xv->dpy, xv->internal_window,
+		     -xv->clip_left, -xv->clip_top);
+      gst_video_overlay_expose (GST_VIDEO_OVERLAY (xv->video_sink));
+    }
+#endif
 }
 #endif /* USE_GTK */
 
@@ -1690,12 +1763,81 @@ x_draw_xwidget_glyph_string (struct glyph_string *s)
 				 CopyFromParent, CWEventMask, &a);
       XLowerWindow (xv->dpy, xv->wdesc);
       XDefineCursor (xv->dpy, xv->wdesc, xv->cursor);
-      xv->cr_surface = cairo_xlib_surface_create (xv->dpy,
-						  xv->wdesc,
-						  FRAME_DISPLAY_INFO (s->f)->visual,
-						  clip_right - clip_left,
-						  clip_bottom - clip_top);
-      xv->cr_context = cairo_create (xv->cr_surface);
+
+      if (XWIDGET_VIEW_GTK_P (xv))
+	{
+	  xv->cr_surface = cairo_xlib_surface_create (xv->dpy,
+						      xv->wdesc,
+						      FRAME_DISPLAY_INFO (s->f)->visual,
+						      clip_right - clip_left,
+						      clip_bottom - clip_top);
+	  xv->cr_context = cairo_create (xv->cr_surface);
+	}
+#ifdef HAVE_GSTREAMER
+      else
+	{
+	  XFlush (xv->dpy);
+
+	  if (EQ (xww->type, Qmedia) && XWIDGET_LIVE_P (xww))
+	    {
+	      GstPadTemplate *templ;
+	      GstPad *sinkpad;
+
+	      a.event_mask = ExposureMask;
+	      a.background_pixel = FRAME_BACKGROUND_PIXEL (xv->frame);
+
+	      xv->internal_window = XCreateWindow (xv->dpy, xv->wdesc,
+						   -clip_left, -clip_top,
+						   xww->width, xww->height,
+						   0, CopyFromParent, CopyFromParent,
+						   CopyFromParent,
+						   (CWEventMask | CWBackPixel), &a);
+
+	      XMapWindow (xv->dpy, xv->internal_window);
+	      XFlush (xv->dpy);
+
+	      Fputhash (make_fixnum (xv->internal_window), xvw, x_window_to_xwv_map);
+
+	      templ = gst_element_class_get_pad_template (GST_ELEMENT_GET_CLASS (xww->gst_tee),
+							  "src_%u");
+
+	      xv->teepad = gst_element_request_pad (xww->gst_tee,
+						    templ, NULL, NULL);
+	      xv->video_sink = gst_element_factory_make ("xvimagesink", NULL);
+	      xv->video_queue = gst_element_factory_make ("queue", NULL);
+
+	      gst_object_ref (xv->video_queue);
+	      gst_object_ref (xv->video_sink);
+
+	      g_object_set_data (G_OBJECT (xv->video_sink), XG_WINDOW_HANDLE,
+				 (gpointer) xv->internal_window);
+	      g_object_set (G_OBJECT (xv->video_sink), "display",
+			    FRAME_TERMINAL (xv->frame)->name, NULL);
+
+	      gst_video_overlay_handle_events (GST_VIDEO_OVERLAY (xv->video_sink),
+					       FALSE);
+
+	      gst_bin_add_many (GST_BIN (xww->gst_pipeline), xv->video_queue,
+				xv->video_sink, NULL);
+
+	      if (!gst_element_link (xv->video_queue, xv->video_sink))
+		emacs_abort ();
+
+	      gst_element_sync_state_with_parent (xv->video_queue);
+	      gst_element_sync_state_with_parent (xv->video_sink);
+
+	      sinkpad = gst_element_get_static_pad (xv->video_queue, "sink");
+	      gst_pad_link (xv->teepad, sinkpad);
+	      gst_object_unref (sinkpad);
+
+	      xv->sinkpad = sinkpad;
+
+	      gst_element_sync_state_with_parent (xww->gst_tee);
+
+	      xwidget_expose (xv);
+	    }
+	}
+#endif
       Fputhash (make_fixnum (xv->wdesc), xvw, x_window_to_xwv_map);
 
       moved = false;
@@ -1709,8 +1851,9 @@ x_draw_xwidget_glyph_string (struct glyph_string *s)
       XMoveResizeWindow (xv->dpy, xv->wdesc, x + clip_left, y + clip_top,
 			 clip_right - clip_left, clip_bottom - clip_top);
       XFlush (xv->dpy);
-      cairo_xlib_surface_set_size (xv->cr_surface, clip_right - clip_left,
-				   clip_bottom - clip_top);
+      if (XWIDGET_VIEW_GTK_P (xv))
+	cairo_xlib_surface_set_size (xv->cr_surface, clip_right - clip_left,
+				     clip_bottom - clip_top);
 #elif defined NS_IMPL_COCOA
       nsxwidget_move_view (xv, x + clip_left, y + clip_top);
 #endif
@@ -1740,8 +1883,10 @@ x_draw_xwidget_glyph_string (struct glyph_string *s)
 			     clip_bottom - clip_top);
 	    }
 	  XFlush (xv->dpy);
-	  cairo_xlib_surface_set_size (xv->cr_surface, clip_right - clip_left,
-				       clip_bottom - clip_top);
+
+	  if (xww->xg_p)
+	    cairo_xlib_surface_set_size (xv->cr_surface, clip_right - clip_left,
+					 clip_bottom - clip_top);
 	}
 #elif defined NS_IMPL_COCOA
       nsxwidget_resize_view (xv, clip_right - clip_left,
@@ -1765,7 +1910,10 @@ x_draw_xwidget_glyph_string (struct glyph_string *s)
       if (!xwidget_hidden (xv))
 	{
 #ifdef USE_GTK
-	  gtk_widget_queue_draw (xww->widget_osr);
+	  if (xww->xg_p)
+	    gtk_widget_queue_draw (xww->widget_osr);
+	  else
+	    xwidget_expose (xv);
 #elif defined NS_IMPL_COCOA
 	  nsxwidget_set_needsdisplay (xv);
 #endif
@@ -1788,6 +1936,12 @@ #define CHECK_WEBKIT_WIDGET(xw)				\
   if (NILP (xw->buffer) || !EQ (xw->type, Qwebkit))	\
     error ("Not a WebKit widget")
 
+#ifdef HAVE_GSTREAMER
+#define CHECK_MEDIA_WIDGET(xw)				\
+  if (NILP (xw->buffer) || !EQ (xw->type, Qmedia))	\
+    error ("Not a WebKit widget")
+#endif
+
 /* Macro that checks xwidget hold webkit web view first.  */
 #define WEBKIT_FN_INIT()						\
   CHECK_LIVE_XWIDGET (xwidget);						\
@@ -1989,6 +2143,14 @@ DEFUN ("xwidget-resize", Fxwidget_resize, Sxwidget_resize, 3, 3, 0,
 #ifdef USE_GTK
 	      xv->just_resized = true;
 	      SET_FRAME_GARBAGED (xv->frame);
+#ifdef HAVE_GSTREAMER
+	      if (EQ (xw->type, Qmedia))
+		{
+		  XResizeWindow (xv->dpy, xv->internal_window,
+				 xw->width, xw->height);
+		}
+#endif
+
 #else
 	      wset_redisplay (XWINDOW (xv->w));
 #endif
@@ -2000,7 +2162,7 @@ DEFUN ("xwidget-resize", Fxwidget_resize, Sxwidget_resize, 3, 3, 0,
 
   /* If there is an offscreen widget resize it first.  */
 #ifdef USE_GTK
-  if (xw->widget_osr)
+  if (xw->xg_p && xw->widget_osr)
     {
       gtk_window_resize (GTK_WINDOW (xw->widgetwindow_osr), xw->width,
                          xw->height);
@@ -2030,9 +2192,14 @@ DEFUN ("xwidget-size-request",
 {
   CHECK_LIVE_XWIDGET (xwidget);
 #ifdef USE_GTK
-  GtkRequisition requisition;
-  gtk_widget_size_request (XXWIDGET (xwidget)->widget_osr, &requisition);
-  return list2i (requisition.width, requisition.height);
+  if (XXWIDGET (xwidget)->xg_p)
+    {
+      GtkRequisition requisition;
+      gtk_widget_size_request (XXWIDGET (xwidget)->widget_osr, &requisition);
+      return list2i (requisition.width, requisition.height);
+    }
+
+  return Qnil;
 #elif defined NS_IMPL_COCOA
   return nsxwidget_get_size (XXWIDGET (xwidget));
 #endif
@@ -2115,18 +2282,30 @@ DEFUN ("delete-xwidget-view",
 #ifdef USE_GTK
   struct xwidget *xw = XXWIDGET (xv->model);
   GdkWindow *w;
-
+#ifdef HAVE_GSTREAMER
+  block_input ();
+  if (XWIDGET_LIVE_P (xw) && EQ (xw->type, Qmedia))
+    unlink_gst_xwidget_view (xv);
+  unblock_input ();
+#endif
   if (xv->wdesc != None)
     {
       block_input ();
-      cairo_destroy (xv->cr_context);
-      cairo_surface_destroy (xv->cr_surface);
+      if (XWIDGET_VIEW_GTK_P (xv))
+	{
+	  cairo_destroy (xv->cr_context);
+	  cairo_surface_destroy (xv->cr_surface);
+	}
       XDestroyWindow (xv->dpy, xv->wdesc);
       Fremhash (make_fixnum (xv->wdesc), x_window_to_xwv_map);
+#ifdef HAVE_GSTREAMER
+      if (XWIDGET_LIVE_P (xw) && EQ (xw->type, Qmedia))
+	Fremhash (make_fixnum (xv->internal_window), x_window_to_xwv_map);
+#endif
       unblock_input ();
     }
 
-  if (xw->embedder_view == xv && !NILP (xw->buffer))
+  if (xw->xg_p && xw->embedder_view == xv && !NILP (xw->buffer))
     {
       w = gtk_widget_get_window (xw->widgetwindow_osr);
 
@@ -2555,6 +2734,77 @@ DEFUN ("xwidget-webkit-back-forward-list", Fxwidget_webkit_back_forward_list,
 
   return list3 (back, here, forward);
 }
+
+#ifdef HAVE_GSTREAMER
+DEFUN ("xwidget-media-pause", Fxwidget_media_pause, Sxwidget_media_pause,
+       1, 1, 0, doc: /* Pause specified media XWIDGET.
+This causes playback to stop until you resume it using
+`xwidget-media-play'.  */)
+  (Lisp_Object xwidget)
+{
+  struct xwidget *xw;
+  struct xwidget_view *xv;
+  Lisp_Object tem;
+
+  CHECK_LIVE_XWIDGET (xwidget);
+  xw = XXWIDGET (xwidget);
+  CHECK_MEDIA_WIDGET (xw);
+
+  block_input ();
+  gst_element_set_state (xw->gst_pipeline, GST_STATE_PAUSED);
+  gst_element_sync_state_with_parent (xw->gst_tee);
+  gst_element_sync_state_with_parent (xw->gst_fakesink);
+
+  for (tem = internal_xwidget_view_list; CONSP (tem); tem = XCDR (tem))
+    {
+      xv = XXWIDGET_VIEW (XCAR (tem));
+
+      if (EQ (xv->model, xwidget))
+	{
+	  gst_element_sync_state_with_parent (xv->video_queue);
+	  gst_element_sync_state_with_parent (xv->video_sink);
+	}
+    }
+
+  unblock_input ();
+
+  return Qnil;
+}
+
+DEFUN ("xwidget-media-play", Fxwidget_media_play, Sxwidget_media_play,
+       1, 1, 0, doc: /* Resume playback of specified media XWIDGET.  */)
+  (Lisp_Object xwidget)
+{
+  struct xwidget *xw;
+  struct xwidget_view *xv;
+  Lisp_Object tem;
+
+  CHECK_LIVE_XWIDGET (xwidget);
+  xw = XXWIDGET (xwidget);
+  CHECK_MEDIA_WIDGET (xw);
+
+  block_input ();
+  gst_element_set_state (xw->gst_pipeline, GST_STATE_PLAYING);
+  gst_element_sync_state_with_parent (xw->gst_tee);
+  gst_element_sync_state_with_parent (xw->gst_fakesink);
+
+  for (tem = internal_xwidget_view_list; CONSP (tem); tem = XCDR (tem))
+    {
+      xv = XXWIDGET_VIEW (XCAR (tem));
+
+      if (EQ (xv->model, xwidget))
+	{
+	  gst_element_sync_state_with_parent (xv->video_queue);
+	  gst_element_sync_state_with_parent (xv->video_sink);
+	}
+    }
+
+  gst_element_sync_state_with_parent (xw->gst_tee);
+  unblock_input ();
+
+  return Qnil;
+}
+#endif
 #endif
 
 void
@@ -2584,6 +2834,7 @@ syms_of_xwidget (void)
   defsubr (&Sxwidget_webkit_zoom);
   defsubr (&Sxwidget_webkit_execute_script);
   DEFSYM (Qwebkit, "webkit");
+  DEFSYM (Qmedia, "media");
 
   defsubr (&Sxwidget_size_request);
   defsubr (&Sdelete_xwidget_view);
@@ -2600,6 +2851,10 @@ syms_of_xwidget (void)
 #ifdef USE_GTK
   defsubr (&Sxwidget_webkit_load_html);
   defsubr (&Sxwidget_webkit_back_forward_list);
+#ifdef HAVE_GSTREAMER
+  defsubr (&Sxwidget_media_pause);
+  defsubr (&Sxwidget_media_play);
+#endif
 #endif
   defsubr (&Skill_xwidget);
 
@@ -2862,7 +3117,7 @@ kill_xwidget (struct xwidget *xw)
 #ifdef USE_GTK
   xw->buffer = Qnil;
 
-  if (xw->widget_osr && xw->widgetwindow_osr)
+  if (xw->xg_p && xw->widget_osr && xw->widgetwindow_osr)
     {
       gtk_widget_destroy (xw->widget_osr);
       gtk_widget_destroy (xw->widgetwindow_osr);
@@ -2882,6 +3137,26 @@ kill_xwidget (struct xwidget *xw)
 	}
     }
 
+#ifdef HAVE_GSTREAMER
+  Lisp_Object tem;
+
+  if (EQ (xw->type, Qmedia))
+    {
+      for (tem = internal_xwidget_view_list; CONSP (tem);
+	   tem = XCDR (tem))
+	{
+	  struct xwidget_view *xv = XXWIDGET_VIEW (XCAR (tem));
+
+	  if (XXWIDGET (xv->model) == xw)
+	    unlink_gst_xwidget_view (xv);
+	}
+      gst_object_unref (xw->gst_tee);
+      gst_object_unref (xw->gst_pipeline);
+      gst_object_unref (xw->gst_source);
+      gst_object_unref (xw->gst_fakesink);
+    }
+#endif
+
   xw->widget_osr = NULL;
   xw->widgetwindow_osr = NULL;
   xw->find_text = NULL;
@@ -2908,3 +3183,59 @@ kill_buffer_xwidgets (Lisp_Object buffer)
       }
     }
 }
+
+#ifdef HAVE_GSTREAMER
+static GstBusSyncReply
+gst_create_window (GstBus *bus, GstMessage *msg, gpointer user_data)
+{
+  GstObject *elm;
+  Window wdesc;
+
+  if (!gst_is_video_overlay_prepare_window_handle_message (msg))
+    return GST_BUS_PASS;
+
+  elm = GST_MESSAGE_SRC (msg);
+  wdesc = (Window) g_object_get_data (G_OBJECT (elm), XG_WINDOW_HANDLE);
+
+  gst_video_overlay_set_window_handle (GST_VIDEO_OVERLAY (elm), wdesc);
+
+  return GST_BUS_DROP;
+}
+
+static gboolean
+gst_message_cb (GstBus *bus, GstMessage *message, gpointer user_data)
+{
+  return TRUE;
+}
+
+static bool
+check_gstreamer_dependencies (void)
+{
+  GstRegistry *registry = gst_registry_get ();
+
+  return (gst_registry_find_feature (registry, "xvimagesink",
+				     GST_TYPE_ELEMENT_FACTORY)
+	  && gst_registry_find_feature (registry, "queue",
+					GST_TYPE_ELEMENT_FACTORY)
+	  && gst_registry_find_feature (registry, "videotestsrc",
+					GST_TYPE_ELEMENT_FACTORY)
+	  && gst_registry_find_feature (registry, "tee",
+					GST_TYPE_ELEMENT_FACTORY));
+}
+
+static void
+unlink_gst_xwidget_view (struct xwidget_view *xv)
+{
+  struct xwidget *xw = XXWIDGET (xv->model);
+
+  gst_pad_unlink (xv->teepad, xv->sinkpad);
+  gst_element_unlink (xv->video_queue, xv->video_sink);
+
+  gst_element_set_state (xv->video_queue, GST_STATE_NULL);
+  gst_element_set_state (xv->video_sink, GST_STATE_NULL);
+  gst_bin_remove_many (GST_BIN (xw->gst_pipeline), xv->video_sink,
+		       xv->video_queue, NULL);
+  gst_object_unref (xv->video_sink);
+  gst_object_unref (xv->video_queue);
+}
+#endif
diff --git a/src/xwidget.h b/src/xwidget.h
index 78fe865dd8..66f7fb2c4c 100644
--- a/src/xwidget.h
+++ b/src/xwidget.h
@@ -39,6 +39,10 @@ #define XWIDGET_H_INCLUDED
 #import "nsxwidget.h"
 #endif
 
+#ifdef HAVE_GSTREAMER
+#include <gst/gst.h>
+#endif
+
 struct xwidget
 {
   union vectorlike_header header;
@@ -65,6 +69,16 @@ #define XWIDGET_H_INCLUDED
   char *find_text;
 
 #if defined (USE_GTK)
+  /* Whether or not the widget is used to display GTK widgets. */
+  bool xg_p;
+
+#ifdef HAVE_GSTREAMER
+  GstElement *gst_tee;
+  GstElement *gst_pipeline;
+  GstElement *gst_source;
+  GstElement *gst_fakesink;
+#endif
+
   /* For offscreen widgets, unused if not osr.  */
   GtkWidget *widget_osr;
   GtkWidget *widgetwindow_osr;
@@ -112,6 +126,14 @@ #define XWIDGET_H_INCLUDED
   Emacs_Cursor cursor;
   struct frame *frame;
 
+#ifdef HAVE_GSTREAMER
+  Window internal_window;
+  GstPad *teepad;
+  GstPad *sinkpad;
+  GstElement *video_sink;
+  GstElement *video_queue;
+#endif
+
   cairo_surface_t *cr_surface;
   cairo_t *cr_context;
   int just_resized;
@@ -159,6 +181,9 @@ #define XXWIDGET_VIEW(a) (eassert (XWIDGET_VIEW_P (a)), \
 #define CHECK_XWIDGET_VIEW(x) \
   CHECK_TYPE (XWIDGET_VIEW_P (x), Qxwidget_view_p, x)
 
+#define XWIDGET_VIEW_GTK_P(x) \
+  XXWIDGET ((x)->model)->xg_p
+
 #define XG_XWIDGET "emacs_xwidget"
 #define XG_XWIDGET_VIEW "emacs_xwidget_view"
 
-- 
2.31.1


[-- Attachment #3: Type: text/plain, Size: 315 bytes --]


But this probably means you don't have the GStreamer development files
installed.

The packages ought to be named "gstreamer1-devel" and
"gstreamer1-plugins-base-devel" or something to that effect.  Make sure
to install GStreamer 1.0, instead of GStreamer 0.10, and configure with
`--with-gstreamer=yes'.

Thanks!

Re: GStreamer xwidget

[Lars Ingebrigtsen]

[-- Attachment #1: Type: text/plain, Size: 582 bytes --]

Po Lu <luangruo@yahoo.com> writes:

> But this probably means you don't have the GStreamer development files
> installed.
>
>
> The packages ought to be named "gstreamer1-devel" and
> "gstreamer1-plugins-base-devel" or something to that effect.  Make sure
> to install GStreamer 1.0, instead of GStreamer 0.10, and configure with
> `--with-gstreamer=yes'.

Ah, right.  For the people at home that wants to reproduce, I did:

sudo apt install libgstreamer-plugins-base1.0-dev libgstreamer1.0-dev
./configure --with-xwidgets --with-gstreamer

and then everything seems to work here:


[-- Attachment #2: Type: image/png, Size: 9502 bytes --]

[-- Attachment #3: Type: text/plain, Size: 105 bytes --]



-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: GStreamer xwidget

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: emacs-devel@gnu.org
> Date: Fri, 19 Nov 2021 14:37:29 +0800
> 
> +@defun xwidget-media-pause xwidget
> +Suspend playback of @var{xwidget}, a media xwidget.  You can later
> +resume playback using @code{xwidget-media-play}.
> +@end defun
> +
> +@defun xwidget-media-pause xwidget
> +Start or resume playback of @var{xwidget}, a media xwidget.  You can
> +later pause playback using @code{xwidget-media-pause}.
> +@end defun

How do you stop the playback?

For that matter, why not have user commands to control playback?

Re: GStreamer xwidget

[Po Lu]

Eli Zaretskii <eliz@gnu.org> writes:

>> +@defun xwidget-media-pause xwidget
>> +Suspend playback of @var{xwidget}, a media xwidget.  You can later
>> +resume playback using @code{xwidget-media-play}.
>> +@end defun
>> +
>> +@defun xwidget-media-pause xwidget
>> +Start or resume playback of @var{xwidget}, a media xwidget.  You can
>> +later pause playback using @code{xwidget-media-pause}.
>> +@end defun

> How do you stop the playback?

That's the same as pausing playback, at least in GStreamer.

> For that matter, why not have user commands to control playback?

That would be for Lisp code using these primitives to implement.  (Just
like image-mode has most of the user commands related to images.)

Thanks.

Re: GStreamer xwidget

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: larsi@gnus.org,  emacs-devel@gnu.org
> Date: Fri, 19 Nov 2021 21:07:54 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> +@defun xwidget-media-pause xwidget
> >> +Suspend playback of @var{xwidget}, a media xwidget.  You can later
> >> +resume playback using @code{xwidget-media-play}.
> >> +@end defun
> >> +
> >> +@defun xwidget-media-pause xwidget
> >> +Start or resume playback of @var{xwidget}, a media xwidget.  You can
> >> +later pause playback using @code{xwidget-media-pause}.
> >> +@end defun
> 
> > How do you stop the playback?
> 
> That's the same as pausing playback, at least in GStreamer.

No, I mean stopping it so that it can never be resumed.

If GStreamer only supports those two actions, I wonder whether we
should have an abstraction layer above it.  But that could be in a
separate patch.

> > For that matter, why not have user commands to control playback?
> 
> That would be for Lisp code using these primitives to implement.  (Just
> like image-mode has most of the user commands related to images.)

Well, I hope it will be.  Right now, even playing an audio in Emacs is
very restrictive: it locks up the main thread until the audio finishes
playing.  That's not appropriate for a modern platform, so I wish we
could have a better UI and UX.

Re: GStreamer xwidget

[Po Lu]

Eli Zaretskii <eliz@gnu.org> writes:

> No, I mean stopping it so that it can never be resumed.

Ah, I understand now.  That would be `xwidget-kill'.

> Well, I hope it will be.  Right now, even playing an audio in Emacs is
> very restrictive: it locks up the main thread until the audio finishes
> playing.  That's not appropriate for a modern platform, so I wish we
> could have a better UI and UX.

GStreamer has an asynchronous interface, where the playback is performed
in separate threads, and events are posted to the main GLib event loop
which is run by the code in `xgselect.c'.

Thanks.

Re: GStreamer xwidget

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: larsi@gnus.org,  emacs-devel@gnu.org
> Date: Fri, 19 Nov 2021 21:33:02 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > No, I mean stopping it so that it can never be resumed.
> 
> Ah, I understand now.  That would be `xwidget-kill'.

Then perhaps mention that in the doc string?

> > Well, I hope it will be.  Right now, even playing an audio in Emacs is
> > very restrictive: it locks up the main thread until the audio finishes
> > playing.  That's not appropriate for a modern platform, so I wish we
> > could have a better UI and UX.
> 
> GStreamer has an asynchronous interface, where the playback is performed
> in separate threads, and events are posted to the main GLib event loop
> which is run by the code in `xgselect.c'.

Sure, I meant to have user commands that take advantage of that.

Re: GStreamer xwidget

[Po Lu]

Po Lu <luangruo@yahoo.com> writes:

> I would like to install the following change that introduces a
> GStreamer-based media widget.
>
> It depends on GStreamer, gst-plugins-base, and gst-plugins-good and is
> thus a compile-time option.

So is it OK if I install this?  Even if it doesn't do much ATM, it still
lays down a lot of important groundwork, including the ability to have
xwidgets that display directly to an X window without GTK.

Thanks.

Re: GStreamer xwidget

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Date: Sat, 20 Nov 2021 13:07:31 +0800
> 
> So is it OK if I install this?  Even if it doesn't do much ATM, it still
> lays down a lot of important groundwork, including the ability to have
> xwidgets that display directly to an X window without GTK.

If you (or someone else) intend to work on building more useful
features on this soon enough, I'd prefer not to install until you've
implemented those more useful features.  If you don't intend to work
on this soon, perhaps a feature branch is in order?  I'm not sure it's
a good idea to install infrastructure that doesn't have enough
functionality to be useful to Lisp programs.

Thanks.

Re: GStreamer xwidget

[Po Lu]

Eli Zaretskii <eliz@gnu.org> writes:

> If you (or someone else) intend to work on building more useful
> features on this soon enough, I'd prefer not to install until you've
> implemented those more useful features.  If you don't intend to work
> on this soon, perhaps a feature branch is in order?  I'm not sure it's
> a good idea to install infrastructure that doesn't have enough
> functionality to be useful to Lisp programs.

I plan to do more work on these features in about a week, so I think
I'll wait until they're finished before installing.

Thanks.

Re: GStreamer xwidget

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > I would like to install the following change that introduces a
  > > GStreamer-based media widget.
  > >
  > > It depends on GStreamer, gst-plugins-base, and gst-plugins-good and is
  > > thus a compile-time option.

  > So is it OK if I install this?

Not now.  Please wait until we finish examining the issue and judge
whether this feature is ok to have in Emacs.


-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: GStreamer xwidget

[Lars Ingebrigtsen]

Richard Stallman <rms@gnu.org> writes:

>   > So is it OK if I install this?
>
> Not now.  Please wait until we finish examining the issue and judge
> whether this feature is ok to have in Emacs.

Po Lu has already answered all your question, and we have already
decided to add GStreamer support to Emacs.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: GStreamer xwidget

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Po Lu has already answered all your question, and we have already
  > decided to add GStreamer support to Emacs.

Don't jump the gun.  There's a possible moral issue here, so please
help me make sure this feature is ok.

It may take a few days to do this, -- but there's no rush.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: GStreamer xwidget

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > I would like to install the following change that introduces a
  > GStreamer-based media widget.

  > It depends on GStreamer, gst-plugins-base, and gst-plugins-good and is
  > thus a compile-time option.

Before deciding whether to install this, we need to understand clearly
what jobs it can do.  What kinds of media does GStreamer stream?
Which formats?  Does GStreamer implement DRM?  Is GStreamer 100% free
ziyou software, with no exceptions?

We also have to think about whether it is good or bad to introduce
this nonmodularity.  What is better about showing videos this way,
rather than just launching vlc in another window?

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: GStreamer xwidget

[Po Lu]

Richard Stallman <rms@gnu.org> writes:

>   > I would like to install the following change that introduces a
>   > GStreamer-based media widget.
>
>   > It depends on GStreamer, gst-plugins-base, and gst-plugins-good and is
>   > thus a compile-time option.

> Before deciding whether to install this, we need to understand clearly
> what jobs it can do.  What kinds of media does GStreamer stream?

The plugins that this change takes advantage of will only stream free
software, as it doesn't depend on gst-plugins-ugly or gst-plugins-bad.

> Which formats?  Does GStreamer implement DRM?

I can't give a definitive list of formats, but the GStreamer plugins
this change will take advantage of only support free formats, consist
entirely of free code, and have a policy against implementing DRM.

> Is GStreamer 100% free ziyou software, with no exceptions?

GStreamer itself is a framework that is 100% free (as in freedom)
software.  There are proprietary plugins that implement DRM, but they
are separate from GStreamer itself and the base and good plugins, and
Emacs will not use them even if present on a user's system.

> We also have to think about whether it is good or bad to introduce
> this nonmodularity.  What is better about showing videos this way,
> rather than just launching vlc in another window?

People seem to want that in Emacs.

Thanks.

Re: GStreamer xwidget

[Lars Ingebrigtsen]

Po Lu <luangruo@yahoo.com> writes:

>> We also have to think about whether it is good or bad to introduce
>> this nonmodularity.  What is better about showing videos this way,
>> rather than just launching vlc in another window?
>
> People seem to want that in Emacs.

Reading web pages that have embedded videos is quite common.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: GStreamer xwidget

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Reading web pages that have embedded videos is quite common.

How _exactly_ does that relate to letting Emacs make GStreamer windows?
Please describe the scenario clearly and spell out the argument.
That way we can tell what conclusion really follows.

We should have this discussion now, _before_ installing any code for
GStreamer, because that's much easier than discussing it afterward.

The most important questions are these,

    Before deciding whether to install this, we need to understand clearly
    what jobs it can do.  What kinds of media does GStreamer stream?
    Which formats?  Does GStreamer implement DRM?  Is GStreamer 100% free
    ziyou software, with no exceptions?

because they will determine whether there is an important reason
to reject making Emacs work closely with GStreamer.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: GStreamer xwidget

[Po Lu]

Richard Stallman <rms@gnu.org> writes:

> The most important questions are these,

>     Before deciding whether to install this, we need to understand clearly
>     what jobs it can do.  What kinds of media does GStreamer stream?
>     Which formats?  Does GStreamer implement DRM?  Is GStreamer 100% free
>     ziyou software, with no exceptions?

> because they will determine whether there is an important reason
> to reject making Emacs work closely with GStreamer.

Perhaps my mail didn't reach you, but I sent you mail answering your
questions earlier.

It is reproduced below for your convenience:

> From: Po Lu <luangruo@yahoo.com>
> Subject: Re: GStreamer xwidget
> To: Richard Stallman <rms@gnu.org>
> Cc: emacs-devel@gnu.org
> Date: Sat, 20 Nov 2021 16:05:07 +0800 (21 hours, 21 minutes, 9 seconds ago)
>
> Richard Stallman <rms@gnu.org> writes:
>
>>   > I would like to install the following change that introduces a
>>   > GStreamer-based media widget.
>>
>>   > It depends on GStreamer, gst-plugins-base, and gst-plugins-good and is
>>   > thus a compile-time option.
>>
>> Before deciding whether to install this, we need to understand clearly
>> what jobs it can do.  What kinds of media does GStreamer stream?
>
> The plugins that this change takes advantage of will only stream free
> software, as it doesn't depend on gst-plugins-ugly or gst-plugins-bad.
>
>> Which formats?  Does GStreamer implement DRM?
>
> I can't give a definitive list of formats, but the GStreamer plugins
> this change will take advantage of only support free formats, consist
> entirely of free code, and have a policy against implementing DRM.
>
>> Is GStreamer 100% free ziyou software, with no exceptions?
>
> GStreamer itself is a framework that is 100% free (as in freedom)
> software.  There are proprietary plugins that implement DRM, but they
> are separate from GStreamer itself and the base and good plugins, and
> Emacs will not use them even if present on a user's system.

Thanks.

Re: GStreamer xwidget

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Perhaps my mail didn't reach you, but I sent you mail answering your
  > questions earlier.

I didn't see it before.  Thanks for repeating it.

Because I know nothing about GStreamer except what I've seen in these
messages, I can't be sure of the implications of these facts.  I don't
have enough context to fit them into.

  > > The plugins that this change takes advantage of will only stream free
  > > software,

That suggests the feature might be ok -- but verifying that calls
for some more discussion.

                as it doesn't depend on gst-plugins-ugly or gst-plugins-bad.

You have mentioned gst-plugins-good, gst-plugins-ugly and
gst-plugins-bad.  Can you tell me how they relate to GStreamer itself?
How does GStreamer interact with them?  Are they linked it by ld?
Is it static linking?

Does the choice of one of those inevitably have to be made when
building Emacs?  Does GStreamer ever use some sort of dynamic linking
to select one of them?

They say "plugins", not "plugin".  Is  there a separate plugin for each
media format?  If so, does selecting gst-plugins-good statically
link _all_ of the free plugins?  If not that, then what?


-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: GStreamer xwidget

[Po Lu]

Richard Stallman <rms@gnu.org> writes:

> Because I know nothing about GStreamer except what I've seen in these
> messages, I can't be sure of the implications of these facts.  I don't
> have enough context to fit them into.

Thanks, please see below.

> That suggests the feature might be ok -- but verifying that calls
> for some more discussion.
>
>                 as it doesn't depend on gst-plugins-ugly or gst-plugins-bad.

> You have mentioned gst-plugins-good, gst-plugins-ugly and
> gst-plugins-bad.  Can you tell me how they relate to GStreamer itself?

They are plugins maintained by the GStreamer developers, but are
separate from GStreamer itself.

> How does GStreamer interact with them?  Are they linked it by ld?
> Is it static linking?

They're not statically linked by GStreamer.  Instead, GStreamer will
dynamically link them depending on what the application requests.

> Does the choice of one of those inevitably have to be made when
> building Emacs?

No, see below.

> Does GStreamer ever use some sort of dynamic linking to select one of
> them?

GStreamer does, but it will not load any plugin Emacs doesn't ask it to.

> They say "plugins", not "plugin".  Is  there a separate plugin for each
> media format?

Yes, that is true.

> If so, does selecting gst-plugins-good statically link _all_ of the
> free plugins?  If not that, then what?

Asking for gst-plugins-base and gst-plugins-good will result in
GStreamer dynamically linking with the free plugins.  As long as Emacs C
code doesn't explictly tell GStreamer to load the ugly or bad plugins,
which are non-free, GStreamer will never load them.

The ugly and bad plugins are not installed with a default GStreamer
distribution either.  AFAIU, most GNU distributions do not have them in
their main package repositories either.

Re: GStreamer xwidget

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > If so, does selecting gst-plugins-good statically link _all_ of the
  > > free plugins?  If not that, then what?

  > Asking for gst-plugins-base and gst-plugins-good will result in
  > GStreamer dynamically linking with the free plugins.  As long as Emacs C
  > code doesn't explictly tell GStreamer to load the ugly or bad plugins,
  > which are non-free, GStreamer will never load them.

This sounds mostly good, but this worries me:

  > > Does the choice of one of those inevitably have to be made when
  > > building Emacs?

  > No, see below.

I don't see any specifics about that point, and it is an important point.

Can you please show me precisely how Emacs would specify which plugins
to load?  What are the specifications of the interface?
What does the code actually look like?  If it uses any macros,
what does it macroexpand into?


-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: GStreamer xwidget

[Po Lu]

Richard Stallman <rms@gnu.org> writes:

> Can you please show me precisely how Emacs would specify which plugins
> to load?  What are the specifications of the interface?  What does the
> code actually look like?  If it uses any macros, what does it
> macroexpand into?

Here's how one would create a GStreamer element (and in doing so load a
plugin):

  GstElement *element = gst_element_factory_make ("factoryname", NULL);

Where "factoryname" is the name of the element factory you want.

GStreamer will then find and load the shared library containing the
plugin that provides an element by that name, and then use it to create
the GstElement.

Obviously, creating a GstElement is insufficient, because for it to be
useful one will inevitably have to connect it to something, and
configure it.

For configuration, GstElements use GObject properties, so if I created
an element of the type "tee", provided by `gst-plugins-base', and I
wanted to configure it to work without any other element being attached
to it, I would set the property named "allow-not-linked" to `TRUE' as
follows:

  GstElement *tee = gst_element_factory_make ("tee", NULL);
  g_object_set (G_OBJECT (tee), "allow-not-linked", TRUE, NULL);

Where `g_object_set' is provided by the GObject library (an object
system for C that is free software, developed by GNOME) and `G_OBJECT'
is a macro that casts `tee' from GstElement to the type named `GObject'
after checking at run-time that the type of `tee' is correct.

Then, I have to put it in a GstBin (which is a container for various
GstElements) like so:

  gst_bin_add (GST_BIN (bin), tee);

Where `gst_bin_add' is a function provided by the GStreamer library,
`bin' is the bin, and `GST_BIN' is a cast macro similar to `G_OBJECT',
but checks that `bin' is of the type `GstBin' instead of `GstObject'.

Afterwards, I will have to "link" (note that this has nothing to do with
linking programs with object files) `tee' to some other element that
displays its output, like such:

  gst_element_link (tee, that_other_element);

Thanks.

Re: GStreamer xwidget

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]


  > Here's how one would create a GStreamer element (and in doing so load a
  > plugin):

  >   GstElement *element = gst_element_factory_make ("factoryname", NULL);

  > Where "factoryname" is the name of the element factory you want.

This is incomplete -- I can't make sense of it.  It talks about things
I never heard of: "elements" and "element factories".

  > GStreamer will then find and load the shared library containing the
  > plugin that provides an element by that name, and then use it to create
  > the GstElement.

What does it mean for the plug-in to "provide" an "element"?

  > Obviously, creating a GstElement is insufficient, because for it to be
  > useful one will inevitably have to connect it to something, and
  > configure it.

What does that mean?

The story "The Gostak and the Doshes", by Miles Breuer, gives an idea
of how lost I feel at this point.

Is the argument "factoryname" the place where one specifies which format is
to be used?

You wrote doublequotes around that name.  What do those doublequotes mean?
Do they mean that the argument is supposed to have a C string as value?
Do they mean that the argument is supposed to be a C string constant?
Something else?

What data type is that argument?  Is it a C string?  Please show me a few
examples of real, valid, possible values for that argument.

What argument value would specify MPEG4?
What argument value would specify Webm?


How does the program control which set of plug-ins are permitted?
For instance, suppose the program wants to use only the "good" plug-ins.
What C code does it use to specify this?

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: GStreamer xwidget

[Po Lu]

Richard Stallman <rms@gnu.org> writes:

> This is incomplete -- I can't make sense of it.  It talks about things
> I never heard of: "elements" and "element factories".

"Elements" are the basic kind of data structure provided by plugins.
They can be linked to other elements, and perform actions on data
streams (such as decoding them into video or displaying the video
on-screen.)

Element factories search for elements by a given name inside plugins,
load plugins that contain said element, and use them to create the
element.

> What does it mean for the plug-in to "provide" an "element"?

It means the code in the element that processes data streams is provided
by the plugin (a shared library).

> Is the argument "factoryname" the place where one specifies which format is
> to be used?

Yes, as data formats are implemented as codec elements.

> You wrote doublequotes around that name.  What do those doublequotes mean?
> Do they mean that the argument is supposed to have a C string as value?

Yes.

> What data type is that argument?  Is it a C string?  Please show me a few
> examples of real, valid, possible values for that argument.

It is a C string.

> What argument value would specify MPEG4?

That would be "mp4mux".

> What argument value would specify Webm?

"webmmux".

> How does the program control which set of plug-ins are permitted?

As GStreamer doesn't load plugins by yourself, you do so by making sure
to only pass good plugin names to the element factory.

> For instance, suppose the program wants to use only the "good" plug-ins.
> What C code does it use to specify this?

As long as you don't explictly load bad plugins, you will be constrained
to the good plug-ins.

Thanks.

Re: GStreamer xwidget

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > How does the program control which set of plug-ins are permitted?

  > As GStreamer doesn't load plugins by yourself, you do so by making sure
  > to only pass good plugin names to the element factory.

We're starting to close in on the crucial point.

Can you show me the code Emacs would use to control which plug-ins are
permitted?  In other words, how does a program "pass plug-in names"
to the element factory?

Does the program need to pass the list of good plug-in names?
If so, how does it get that list, and from where?

I'm trying to do a kind of security analysis of this.  Does passing
the right list of plug-in names depend on the cooperation of other
projects?  If so, which ones, and how would Emacs interact with them?
I want to make sure we are not volunerable to misbehavior on their
part.

If we'd need to trust some people, who are they, and what concretely
would we need to trust them to do?


-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: GStreamer xwidget

[Po Lu]

Richard Stallman <rms@gnu.org> writes:

>   > As GStreamer doesn't load plugins by yourself, you do so by making sure
>   > to only pass good plugin names to the element factory.

> We're starting to close in on the crucial point.

> Can you show me the code Emacs would use to control which plug-ins are
> permitted?  In other words, how does a program "pass plug-in names"
> to the element factory?

In the function call to `gst_element_factory_make'.  We would have to
verify that the first argument, a C string, names a plugin included in
`gst-plugins-base' or `gst-plugins-good'.

> Does the program need to pass the list of good plug-in names?

No, it only has to refrain from passing a string that names a plugin
that is not free software to `gst_element_factory_make'.

> I'm trying to do a kind of security analysis of this.  Does passing
> the right list of plug-in names depend on the cooperation of other
> projects?

The GStreamer developers, who overlap greatly with other Freedesktop.org
projects.

> If so, which ones, and how would Emacs interact with them?  I want to
> make sure we are not volunerable to misbehavior on their part.

We would have to make sure that, when we decide to use a plugin in
Emacs, that it is part of `gst-plugins-base' or `gst-plugins-good'.

> If we'd need to trust some people, who are they, and what concretely
> would we need to trust them to do?

We would have to trust them to place only free plugins in
`gst-plugins-good' and `gst-plugins-base', and to document the plugins
correctly.

Thanks.

Re: GStreamer xwidget

[Alexandre Garreau]

Le Thursday, 25 November 2021, 09:13:06 CET Po Lu a écrit :
> Richard Stallman <rms@gnu.org> writes:
> >   > As GStreamer doesn't load plugins by yourself, you do so by making
> >   > sure
> >   > to only pass good plugin names to the element factory.
> > 
> > We're starting to close in on the crucial point.
> > 
> > Can you show me the code Emacs would use to control which plug-ins are
> > permitted?  In other words, how does a program "pass plug-in names"
> > to the element factory?
> 
> In the function call to `gst_element_factory_make'.  We would have to
> verify that the first argument, a C string, names a plugin included in
> `gst-plugins-base' or `gst-plugins-good'.

> > I'm trying to do a kind of security analysis of this.  Does passing
> > the right list of plug-in names depend on the cooperation of other
> > projects?
> 
> The GStreamer developers, who overlap greatly with other Freedesktop.org
> projects.

> We would have to trust them to place only free plugins in
> `gst-plugins-good' and `gst-plugins-base', and to document the plugins
> correctly.

Wait, it’s not needed to trust anyone.  There aren’t millions of plugins, 
maximum hundreds: it would be perfectly feasible to include the list of 
all of them into emacs.  The question is whether to *delegate* that work 
to GNOME, and the issue would then be when that list changes, what about 
updates, etc.

Btw, why whitelisting good and base, instead of blacklisting bad and ugly?  
Do we want to blacklist any unknown plugins?  GNU software, through 
configure for instance, has been known to ease not only classical hacking, 
by distros and full forks, but also by individual users, private 
configurations, etc. and still have all software interactions working…

Isn’t there a way to tell the license of the plugin inside each of him? if 
I remember well, GCC had some sort of a such thing (something to declare 
explicitely the plugin is under GPLv3 or GPLv3+), and I guess emacs too, 
right? It would be bad, I believe, if any individual programs a gst plugin 
for themselves (to begin, before to publish it), and cannot use it into 
emacs, even though it’s free… and neither a friend of them can, although 
they would be a free license to make the friend free…  it would mean to 
centralize the decision of “what plugins can run” into GNOME’s hand, and 
it’s actually, I believe, some sort of free-software, weak (*because* it’s 
free-software, then modifiable (btw it would be even better if that 
checking was done in lisp, or had some kind of configuration option for 
it: afaiu it’s illegal to distribute software with incompatible licenses, 
but not using one so, so an individual could still use emacs with some 
plugin that’s not juridically free-software because it has no license and 
the author is the only person to possess a copy of that plugin)) DRM…

On the other hand, the main risk here is if a distribution includes bad or 
ugly, then emacs would use proprietary software, not that the user 
programs or download one plugin, and that, by disfortune, the plugin the 
user chose is possibly proprietary, right? because since the user choose, 
well they can just as well choose a free software, and the 4 categories 
(good, base, bad, ugly) established by GNOME are outside of that…

Re: GStreamer xwidget

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > We would have to trust them to place only free plugins in
  > > `gst-plugins-good' and `gst-plugins-base', and to document the plugins
  > > correctly.

  > Wait, it’s not needed to trust anyone.  There aren’t millions of plugins, 
  > maximum hundreds: it would be perfectly feasible to include the list of 
  > all of them into emacs.  The question is whether to *delegate* that work 
  > to GNOME, and the issue would then be when that list changes, what about 
  > updates, etc.

Please state concretely what it is that you're disagreeing about.
Instead of arguing about whether to "trust GNOME", please tell
us what exactly the GStreamer developers did.  Then we can  see
whether that solves the problem.

  > Btw, why whitelisting good and base, instead of blacklisting bad and ugly? 

There are probably lots of nonfree plugins.  Trying to list them would
be a lot of work, and unreliable too.

If you want to use some plug-in that isn't in the good list, you can
edit the code and recompile.  Or patch in the string constant of your
choice with GDB.  Those take a little work, but they are not an obstacle
for a hacker.


-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: GStreamer xwidget

[Dmitry Gutov]

This whole thread is very puzzling.

On 27.11.2021 07:09, Richard Stallman wrote:
>    > > We would have to trust them to place only free plugins in
>    > > `gst-plugins-good' and `gst-plugins-base', and to document the plugins
>    > > correctly.
> 
>    > Wait, it’s not needed to trust anyone.  There aren’t millions of plugins,
>    > maximum hundreds: it would be perfectly feasible to include the list of
>    > all of them into emacs.  The question is whether to*delegate*  that work
>    > to GNOME, and the issue would then be when that list changes, what about
>    > updates, etc.
> 
> Please state concretely what it is that you're disagreeing about.
> Instead of arguing about whether to "trust GNOME", please tell
> us what exactly the GStreamer developers did.  Then we can  see
> whether that solves the problem.

Whatever plugins are available on the user's system, are a result of the 
distro including them (which, by default, means only the "good" ones get 
in), as well as the ones the user installed explicitly. Possibly "bad" 
too (the less well-written ones).

The "licensing problems" with "ugly" plugins are more on the side of 
"not being distributed under LGPL", rather not being free software. Or 
maybe having dependencies like that, because I wasn't able to find an 
example of the former.

More likely, they are in that category because of the video/image 
formats being patent encumbered (so the distros can't easily distribute 
that code). But if the user has installed such a plugin manually, who 
are we to deny them the pleasure of viewing asf, realmedia or whatever?

>    > Btw, why whitelisting good and base, instead of blacklisting bad and ugly?
> 
> There are probably lots of nonfree plugins.  Trying to list them would
> be a lot of work, and unreliable too.

If you were designing the 'shell-command' command in Emacs today, would 
you start with a whitelist of all known free software programs and 
refuse to call anything not from that list?

Re: GStreamer xwidget

[Arthur Miller]

Dmitry Gutov <dgutov@yandex.ru> writes:

> This whole thread is very puzzling.
>
> On 27.11.2021 07:09, Richard Stallman wrote:
>>    > > We would have to trust them to place only free plugins in
>>    > > `gst-plugins-good' and `gst-plugins-base', and to document the plugins
>>    > > correctly.
>>    > Wait, it’s not needed to trust anyone.  There aren’t millions of plugins,
>>    > maximum hundreds: it would be perfectly feasible to include the list of
>>    > all of them into emacs.  The question is whether to*delegate*  that work
>>    > to GNOME, and the issue would then be when that list changes, what about
>>    > updates, etc.
>> Please state concretely what it is that you're disagreeing about.
>> Instead of arguing about whether to "trust GNOME", please tell
>> us what exactly the GStreamer developers did.  Then we can  see
>> whether that solves the problem.
>
> Whatever plugins are available on the user's system, are a result of the distro
> including them (which, by default, means only the "good" ones get in), as well
> as the ones the user installed explicitly. Possibly "bad" too (the less
> well-written ones).

That, but, it still does not mean that proprietary codecs will not be present on
the OS even if a distro does not include them, even on a libre system.

A user can always install proprietary plugin(s), so it realy is up to user
consciusness.

If I remember well, people could just copy prorietary codecs from their Windows
systems and install them in gnu/linux to enable them in gstreamer, so there is
no guarantee that gstreamer won't play those.

I would really like to see gstreamer in Emacs, but I am not sure you can truly
prevent gstreamer from loading proprietary codecs.

https://lwn.net/Articles/217583/

Gstreamer seems to support those in order to support users who legaly purchase
licenses to proprietary formats, so getting this off gstreamer means probably
patching gstreamer and distributing own binary with Emacs. Prohibition should
maybe not be a goal?

> If you were designing the 'shell-command' command in Emacs today, would you
> start with a whitelist of all known free software programs and refuse to call
> anything not from that list?

Indeed. And how much are proprietary codecs different from linking Emacs to all
the system libraries in "unjust" systems, i.e. Windows and MacOS?

I think benefits of Emacs being able to play media outweigh the cons. In the
very end, if someone is bying DRM protected media, well, it is his/her personal
choice. In all these years with computing and DRM I have yet not purchased a
single DRM protected piece of software, beside few games back in time when they
used to come in paperboxes on CDs. I agree DRM is crap, but I also think it is a
personal choice people should make on their own.

Re: GStreamer xwidget

[Po Lu]

Arthur Miller <arthur.miller@live.com> writes:

> If I remember well, people could just copy prorietary codecs from their Windows
> systems and install them in gnu/linux to enable them in gstreamer, so there is
> no guarantee that gstreamer won't play those.

If we do not explictly load them, they will not be loaded.

Re: GStreamer xwidget

[chad]

[-- Attachment #1: Type: text/plain, Size: 598 bytes --]

On Wed, Dec 1, 2021 at 9:54 AM Arthur Miller <arthur.miller@live.com> wrote:

> I would really like to see gstreamer in Emacs, but I am not sure you can
> truly
> prevent gstreamer from loading proprietary codecs.
>

This is primarily true if the code uses the playbin module for gstreamer,
which most people do, and Po Lu is explicitly NOT doing. The manual process
they intend will instead miss some codecs (etc) that are supported on the
system, but it won't unknowingly load proprietary code (well, I guess
modulo some extreme behavior on the system maintainer's part).

Hope that helps,
~Chad

[-- Attachment #2: Type: text/html, Size: 982 bytes --]

Re: GStreamer xwidget

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > If so, which ones, and how would Emacs interact with them?  I want to
  > > make sure we are not volunerable to misbehavior on their part.

  > We would have to make sure that, when we decide to use a plugin in
  > Emacs, that it is part of `gst-plugins-base' or `gst-plugins-good'.


This leads me to more questions?

1. In the code you have currenly written, where does the plug-in name
come from?  In other words, how is it specified?

1a. Do Lisp programs specify it?

1a. Is there code in Emacs that figures out the right plug-in to use?
If so, what method does it use, and based on what information?

2. In the code you have currenly written, what prevents use of
a bad or ugly plug-in?

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: GStreamer xwidget

[Po Lu]

Richard Stallman <rms@gnu.org> writes:

> This leads me to more questions?
>
> 1. In the code you have currenly written, where does the plug-in name
> come from?  In other words, how is it specified?

It's hard coded.

> 1a. Do Lisp programs specify it?

No.

> 1a. Is there code in Emacs that figures out the right plug-in to use?
> If so, what method does it use, and based on what information?

It doesn't dynamically determine plugins at all.  Instead, the name of
each plugin Emacs will use is hard coded inside C code.

> 2. In the code you have currenly written, what prevents use of
> a bad or ugly plug-in?

That nothing in the code will ever have a chance to load such a bad or
ugly plugin, as only the names of good plugins are specified.

Thanks.

Re: GStreamer xwidget

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > 1. In the code you have currenly written, where does the plug-in name
  > > come from?  In other words, how is it specified?

  > It's hard coded.

Would you please show the code which gets the plug-in name and
specifies it?

  > > 1a. Is there code in Emacs that figures out the right plug-in to use?
  > > If so, what method does it use, and based on what information?

  > It doesn't dynamically determine plugins at all.  Instead, the name of
  > each plugin Emacs will use is hard coded inside C code.

That sounds safe.  Would you please show the code which
chooses the plug-in name to use in each case?

It starts to appear that this is ok, but I'd like to see those parts
and be sure.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: GStreamer xwidget

[Po Lu]

Richard Stallman <rms@gnu.org> writes:

> That sounds safe.  Would you please show the code which
> chooses the plug-in name to use in each case?
>
> It starts to appear that this is ok, but I'd like to see those parts
> and be sure.

+static bool
+check_gstreamer_dependencies (void)
+{
+  GstRegistry *registry = gst_registry_get ();
+
+  return (gst_registry_find_feature (registry, "xvimagesink",
+				     GST_TYPE_ELEMENT_FACTORY)
+	  && gst_registry_find_feature (registry, "queue",
+					GST_TYPE_ELEMENT_FACTORY)
+	  && gst_registry_find_feature (registry, "videotestsrc",
+					GST_TYPE_ELEMENT_FACTORY)
+	  && gst_registry_find_feature (registry, "fakesink",
+					GST_TYPE_ELEMENT_FACTORY)
+	  && gst_registry_find_feature (registry, "tee",
+					GST_TYPE_ELEMENT_FACTORY));
+}

Here is where Emacs checks for the presence of the plugins it wants.
All of these plugins are in gst-plugins-base and gst-plugins-good.

+      xw->gst_source = gst_element_factory_make ("videotestsrc", NULL);
+      xw->gst_tee = gst_element_factory_make ("tee", NULL);

+      GstElement *fakesink = gst_element_factory_make ("fakesink", NULL);
+      GstElement *queue = gst_element_factory_make ("queue", NULL);

+	      xv->video_sink = gst_element_factory_make ("xvimagesink", NULL);
+	      xv->video_queue = gst_element_factory_make ("queue", NULL);

And here is where the plugins are created.  Thanks.

Re: GStreamer xwidget

[Yuri Khan]

On Sun, 28 Nov 2021 at 11:43, Po Lu <luangruo@yahoo.com> wrote:

> +         && gst_registry_find_feature (registry, "videotestsrc",
> +                                       GST_TYPE_ELEMENT_FACTORY)

> +      xw->gst_source = gst_element_factory_make ("videotestsrc", NULL);

Here you are checking for and instantiating a known source element
that produces a fixed test image, but for any practical use (e.g.
playing a video from a web page or local file system) you’ll need to
instantiate a demultiplexer that knows how to split the original byte
stream into video, audio, and subtitle streams, and video and audio
decoders that handle the particular compression format. Some of those
could be patent-encumbered (and so end up in ugly) or poorly
maintained (bad). Are you planning to avoid those? If so, how? Also,
will the user be able to override the avoidance (“Yes I know those are
patent-encumbered and I have no problem with that”, or “Yes I know
that decoder might have a security vulnerability and I accept the
risk”)? If so, how?

Re: GStreamer xwidget

[Po Lu]

Yuri Khan <yuri.v.khan@gmail.com> writes:

> Here you are checking for and instantiating a known source element
> that produces a fixed test image, but for any practical use (e.g.
> playing a video from a web page or local file system) you’ll need to
> instantiate a demultiplexer that knows how to split the original byte
> stream into video, audio, and subtitle streams, and video and audio
> decoders that handle the particular compression format.

I will write individual pipelines for most of the common use cases,
taking care to use only plugins that are free software.

I do not plan on using playbins or any other elements that will load
plugins dynamically.

> Also, will the user be able to override the avoidance (“Yes I know
> those are patent-encumbered and I have no problem with that”, or “Yes
> I know that decoder might have a security vulnerability and I accept
> the risk”)? If so, how?

The user will not.  I don't want to make Emacs a tool that is more
convenient when used with proprietary software.

In short, I don't want Emacs to lead people who judge by convenience to
use proprietary software.

Re: GStreamer xwidget

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > +         && gst_registry_find_feature (registry, "videotestsrc",
  > > +                                       GST_TYPE_ELEMENT_FACTORY)

  > > +      xw->gst_source = gst_element_factory_make ("videotestsrc", NULL);
  ...

You seem to know something about this code.  Can you explain the
meaning of it?  What the dats structures are, and what these
operations actually do?

I'm trying to find out how Emacs determines which plug-ins to use
for any given argument supplied by the ueer.  And what do they depend on?
Does a given file format fully determine the plug-ins to use?

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: GStreamer xwidget

[Yuri Khan]

On Mon, 29 Nov 2021 at 10:02, Richard Stallman <rms@gnu.org> wrote:

>   > > +         && gst_registry_find_feature (registry, "videotestsrc",
>   > > +                                       GST_TYPE_ELEMENT_FACTORY)
>
>   > > +      xw->gst_source = gst_element_factory_make ("videotestsrc", NULL);
>   ...
>
> You seem to know something about this code.

Disclaimer: I have not worked specifically with GStreamer, but I come
from a Windows background where I used to work with DirectShow. They
are technically similar.

> Can you explain the
> meaning of it?  What the dats structures are, and what these
> operations actually do?

I think the easiest way to explain media processing frameworks is to
make an analogy with UNIX pipes.

A typical UNIX process has a single input descriptor and two outputs
(stdout and stderr). You combine them in pipelines by connecting an
output of one program to the input of another.

Shell pipelines are typically linear or tree-like.

Media processing frameworks generalize this. Each “filter” or
“element” can have an arbitrary number of inputs and outputs, and they
are typed so the framework can check if the connection you are trying
to make is nonsensical. For example, you cannot meaningfully connect
an audio source to a video sink.

Filters and connections form a directed acyclic graph.

The videotestsrc element is a source. That is, it has no inputs, and
has one video output. For this particular filter, the output is a
fixed test image.

There is a sink filter that can display video. By connecting a
videotestsrc to a sink, you get a minimal complete graph that does
something.

A full graph for playing a video or audio clip will typically contain:

* A file or URL source. This knows the URL of the input file and
outputs a byte stream.

* A splitter or demultiplexer. This takes the byte stream and outputs
separate streams for any video, audio or subtitle tracks included.
Different container formats (e.g. AVI, MP4, Matroska, Ogg) require
different demux filters. (Closest UNIX analogy: zip and tar are
different containers, and use different programs.)
The user may not want all streams rendered — e.g. when watching a film
that contains audio tracks in English and Spanish, they will want just
one.

* Decoders for a subset of streams in the clip. (The user may not want
all streams rendered — e.g. when watching a film that contains audio
tracks in English and Spanish, they will want just one.) Each specific
video or audio compression format requires a different decoder filter.
(Analogy: deflate, xz, and bzip are different compression formats.) A
decoder takes a compressed stream from the demultiplexer and outputs
raw RGB or YCbCr video or PCM audio.

* “Renderers” or “sinks” for the decoded video and audio. These differ
by target video or audio subsystem (e.g. pulseaudio), hardware
acceleration capabilities, etc.


> I'm trying to find out how Emacs determines which plug-ins to use
> for any given argument supplied by the ueer.  And what do they depend on?
> Does a given file format fully determine the plug-ins to use?

A given file format (which is a combination of the container format
and compression formats of individual streams within) constrains the
set of possible rendering graphs.

A system may have multiple filters capable of filling each of the
roles listed above. In that case, the application (e.g. Emacs) can
choose between possibilities.

Frameworks have a way to try to automatically complete a rendering
graph for a source; I do not know all the details on how they resolve
ambiguities.

Re: GStreamer xwidget

[Po Lu]

Yuri Khan <yuri.v.khan@gmail.com> writes:

> Frameworks have a way to try to automatically complete a rendering
> graph for a source; I do not know all the details on how they resolve
> ambiguities.

In GStreamer, the plugin used for that is the playbin plugin.  I refuse
to use that plugin and will instead write individual pipelines for
common playback formats, as the playbin likes to load arbitrary plugins
with no regard for whether or not they are free software.

Re: GStreamer xwidget

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > In GStreamer, the plugin used for that is the playbin plugin.  I refuse
  > to use that plugin and will instead write individual pipelines for
  > common playback formats, as the playbin likes to load arbitrary plugins
  > with no regard for whether or not they are free software.

This sounds like the right approach, basically.

But I have a couple of ideas to suggest, to reduce the work.

1. Does playbin offer any hooks for the calling program to judge the plug-ins
it wants to load, and say "That plug-in is ok" or "That plug-in is no good"?
If it does, using them would be simpler and less work.

2. If playbin does not offer such a hook, we could patch playbin
to offer a suitable hook.  Maybe we could get that patch upstreamed.

3. We could make a copy of the code for playbin under another name,
and add those hooks to it.  We coulod link it staticly with Emacs.

WDYT?

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: GStreamer xwidget

[Po Lu]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > In GStreamer, the plugin used for that is the playbin plugin.  I refuse
>   > to use that plugin and will instead write individual pipelines for
>   > common playback formats, as the playbin likes to load arbitrary plugins
>   > with no regard for whether or not they are free software.
>
> This sounds like the right approach, basically.
>
> But I have a couple of ideas to suggest, to reduce the work.
>
> 1. Does playbin offer any hooks for the calling program to judge the plug-ins
> it wants to load, and say "That plug-in is ok" or "That plug-in is no good"?
> If it does, using them would be simpler and less work.

I don't think so, but if someone else knows better, please chime in :)

> 2. If playbin does not offer such a hook, we could patch playbin
> to offer a suitable hook.  Maybe we could get that patch upstreamed.

Hmm, I will take a look at that.

> 3. We could make a copy of the code for playbin under another name,
> and add those hooks to it.  We coulod link it staticly with Emacs.

That would certainly complicate the Emacs building process a great deal.
Building GStreamer (including GStreamer plugins) is no easy task!

Re: GStreamer xwidget

[Yasushi SHOJI]

Hi Po,

On Tue, Nov 30, 2021 at 10:41 AM Po Lu <luangruo@yahoo.com> wrote:
> > 1. Does playbin offer any hooks for the calling program to judge the plug-ins
> > it wants to load, and say "That plug-in is ok" or "That plug-in is no good"?
> > If it does, using them would be simpler and less work.
>
> I don't think so, but if someone else knows better, please chime in :)

How about "element-setup"
https://gstreamer.freedesktop.org/documentation/playback/playbin.html?gi-language=c#playbin::element-setup

There is "license" field in `GstPlugin`:
https://gstreamer.freedesktop.org/documentation/gstreamer/gstplugin.html?gi-language=c#members

With these, I think you can craft what you want.

Best,
-- 
             yashi

Re: GStreamer xwidget

[Po Lu]

Yasushi SHOJI <yasushi.shoji@gmail.com> writes:

> How about "element-setup"
> https://gstreamer.freedesktop.org/documentation/playback/playbin.html?gi-language=c#playbin::element-setup

Thanks!

> There is "license" field in `GstPlugin`:
> https://gstreamer.freedesktop.org/documentation/gstreamer/gstplugin.html?gi-language=c#members
>
> With these, I think you can craft what you want.

Thanks.  Do you know if the proprietary plugins have their `license'
fields labeled correctly?  What about patent-encumbered plugins?

Re: GStreamer xwidget

[Yasushi SHOJI]

On Tue, Nov 30, 2021 at 6:30 PM Po Lu <luangruo@yahoo.com> wrote:
> Thanks.  Do you know if the proprietary plugins have their `license'
> fields labeled correctly?  What about patent-encumbered plugins?

No I don't.
-- 
             yashi

Re: GStreamer xwidget

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > 1. Does playbin offer any hooks for the calling program to judge the plug-ins
  > > it wants to load, and say "That plug-in is ok" or "That plug-in is no good"?
  > > If it does, using them would be simpler and less work.

  > > 3. We could make a copy of the code for playbin under another name,
  > > and add those hooks to it.  We coulod link it staticly with Emacs.

  > That would certainly complicate the Emacs building process a great deal.
  > Building GStreamer (including GStreamer plugins) is no easy task!

We don't have to include the modified playbin _in GStreamer itself_.
We could make it self-contained by copying in whatever GStreamer
header material it depends on, so it will build easily as part of
Emacs.

However, releasing a modified GStreamer (modified in playbin) is
another alternative.  Free distros could use that version of
GStreamer.

The clean solution is to include those hooks in the main GStreamer
release; then Emacs can simply use them.  But we can't implement that
ourselves.  We can hope that the GStreamer developers will do so, but
until then, we don't have a responsibility to make this clean.  It is
ok if it includes kludges, as long as it doesn't make a lot of work
for us.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: GStreamer xwidget

[Po Lu]

Richard Stallman <rms@gnu.org> writes:

> However, releasing a modified GStreamer (modified in playbin) is
> another alternative.  Free distros could use that version of
> GStreamer.

> The clean solution is to include those hooks in the main GStreamer
> release; then Emacs can simply use them.  But we can't implement that
> ourselves.  We can hope that the GStreamer developers will do so, but
> until then, we don't have a responsibility to make this clean.  It is
> ok if it includes kludges, as long as it doesn't make a lot of work
> for us.

I will certainly look into both of the options above, thanks.

Re: GStreamer xwidget

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

Thanks for the very clear explanation.  Now I have a general overview
of the architecture of GStreamer.  It seems clean.

So each file (more generally, source)`, based on the combination of
container formats and stream formats it contains, requires a
collection of filters which correspond to them, and connections
between them.  Is that right?

So how can we make Emacs look at, and accept or reject, those filters
based on our nontechnical criteria?

Where is the code that figures out the filters and connections
for a given file?  Is that a general facility included in GSreamer?
Or is it part of the proposed change to Emacs?

If it is part of the proposed change to Emacs, could someone please
show me that code, and explain what the parts of it do?

If it is part of GStreamer, could someone please show me the
code in Emacs which calls that part of GStreamer, and explain for
me the arguments passed to it and the values it passes back?




-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: GStreamer xwidget

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > +  return (gst_registry_find_feature (registry, "xvimagesink",
  > +				     GST_TYPE_ELEMENT_FACTORY)
  > +	  && gst_registry_find_feature (registry, "queue",
  > +					GST_TYPE_ELEMENT_FACTORY)
  > +	  && gst_registry_find_feature (registry, "videotestsrc",
  > +					GST_TYPE_ELEMENT_FACTORY)
  > +	  && gst_registry_find_feature (registry, "fakesink",
  > +					GST_TYPE_ELEMENT_FACTORY)
  > +	  && gst_registry_find_feature (registry, "tee",
  > +					GST_TYPE_ELEMENT_FACTORY));

This takes me one step forward.  But you haven't explained what
this code does, and I can't recognize the meaning of anything.

What does gst_type_element_factory do?  It looks up something
in a registry, right?  What does a registry mean?

What is the meaning of the "xvimagesink" feature?
What is the meaning of the "queue" feature?
And so on?

Can you tell me an example that explains what this is doing and why?

  > Here is where Emacs checks for the presence of the plugins it wants.
  > All of these plugins are in gst-plugins-base and gst-plugins-good.

Are you saying that "xvimagesink", "queue", "videotestsrc", "fakesink"
and "tee" are the names of plug-ins?

If so, can you describe what each one does?

For instance, if the user specifies the URL of an mp4 file, which
of those plug-ins will Emacs tell GStreamer to load? 

Are you saying that those five plug-ins are the only five plug-ins
that Emacs will ever tell GStreamer to load?

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: GStreamer xwidget

[Po Lu]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > +  return (gst_registry_find_feature (registry, "xvimagesink",
>   > +				     GST_TYPE_ELEMENT_FACTORY)
>   > +	  && gst_registry_find_feature (registry, "queue",
>   > +					GST_TYPE_ELEMENT_FACTORY)
>   > +	  && gst_registry_find_feature (registry, "videotestsrc",
>   > +					GST_TYPE_ELEMENT_FACTORY)
>   > +	  && gst_registry_find_feature (registry, "fakesink",
>   > +					GST_TYPE_ELEMENT_FACTORY)
>   > +	  && gst_registry_find_feature (registry, "tee",
>   > +					GST_TYPE_ELEMENT_FACTORY));
>
> This takes me one step forward.  But you haven't explained what
> this code does, and I can't recognize the meaning of anything.

> What does gst_type_element_factory do?  It looks up something
> in a registry, right?  What does a registry mean?

Essentially, it searches for a plugin (an element factory) by a name in
the plugin registry, which is the data structure that tells GStreamer
how to look for plugins.

> What is the meaning of the "xvimagesink" feature?
> What is the meaning of the "queue" feature?

See below.

> And so on?
>
> Can you tell me an example that explains what this is doing and why?

> Are you saying that "xvimagesink", "queue", "videotestsrc", "fakesink"
> and "tee" are the names of plug-ins?
>
> If so, can you describe what each one does?

videotestsrc provides a test pattern.  xvimagesink displays the media to
an X window (solely an implementation detail, the window created is
internal to Emacs), fakesink is a plugin used to keep the data flow up
even if no Emacs window is currently displaying the xwidget, "tee"
allows a single video stream to display in multiple windows, and "queue"
provides an in-memory buffer that works in cooperation with a tee to
allow that kind of display.

> For instance, if the user specifies the URL of an mp4 file, which
> of those plug-ins will Emacs tell GStreamer to load? 

That feature is not implemented yet, and it will probably not be, as the
H.265 decoder is either ugly or bad (precisely which I don't remember).

> Are you saying that those five plug-ins are the only five plug-ins
> that Emacs will ever tell GStreamer to load?

Yes, in the future I will add other free plugins once I implement the
feature that allows it to load common free formats.

Thanks.

Re: GStreamer xwidget

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Essentially, it searches for a plugin (an element factory) by a name in
  > the plugin registry, which is the data structure that tells GStreamer
  > how to look for plugins.

What code puts the data into that data stucture?
Is it done by a function in GStreamer that Emacs calls?
If so, what is its name, and where does Emacs call it from?

Can Emacs study the data structure after it is filled in?
Could Emacs delete some of the elements (i.e., put some
plug-ins off limits)?


-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: GStreamer xwidget

[Po Lu]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > Essentially, it searches for a plugin (an element factory) by a name in
>   > the plugin registry, which is the data structure that tells GStreamer
>   > how to look for plugins.
>
> What code puts the data into that data stucture?
> Is it done by a function in GStreamer that Emacs calls?
> If so, what is its name, and where does Emacs call it from?

It is initialized by GStreamer when Emacs calls `gst_init'.  The data
structure is private to GStreamer, and as such, its contents are not
guaranteed to be stable.

> Can Emacs study the data structure after it is filled in?
> Could Emacs delete some of the elements (i.e., put some
> plug-ins off limits)?

Unfortunately, I don't think so.

Thanks.

Re: GStreamer xwidget

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > It is initialized by GStreamer when Emacs calls `gst_init'.  The data
  > structure is private to GStreamer, and as such, its contents are not
  > guaranteed to be stable.

  > > Can Emacs study the data structure after it is filled in?
  > > Could Emacs delete some of the elements (i.e., put some
  > > plug-ins off limits)?

  > Unfortunately, I don't think so.

We can write unmodular code if that's the best solution.
We can have a file in Emacs that compiles using the GStreamer
header files -- perhaps syntactically transformed -- so it can
access this data structure and do what we want it to do.

We can write it so that it would be a reasonable added interface for
GStreamer, and ask them to install it.  But we will have a fallback if
they choose not to.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: GStreamer xwidget

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > The ugly and bad plugins are not installed with a default GStreamer
  > distribution either.  AFAIU, most GNU distributions do not have them in
  > their main package repositories either.

This is not directly concerned with Emacs, but I'm curious.

According to Wikipedia, the "bad" problems of GStreamer have
_technical_ problems; perhaps they need more work.  Do you know
whether all the "bad" plug-ins are ziyou?

As for the "ugly" plug-ins, can you find for me a list of them, what
formats they support, and what the "distribution problem" of each one
is?  Basically, I'd like to know what the obstacles are to their use.

-- 
Dr Richard Stallman (https://stallman.org) Chief GNUisance of the GNU
Project (https://gnu.org) Founder, Free Software Foundation
(https://fsf.org) Internet Hall-of-Famer
(https://internethalloffame.org)

Re: GStreamer xwidget

[Po Lu]

Richard Stallman <rms@gnu.org> writes:

> This is not directly concerned with Emacs, but I'm curious.
>
> According to Wikipedia, the "bad" problems of GStreamer have
> _technical_ problems; perhaps they need more work.  Do you know
> whether all the "bad" plug-ins are ziyou?

I don't know, but I was told by word-of-mouth that the GStreamer
developers do not hold the "bad" plugins to any standard, both technical
and freedom-wise.

As such, they don't guarantee that those plugins are free software.

> As for the "ugly" plug-ins, can you find for me a list of them, what
> formats they support, and what the "distribution problem" of each one
> is?  Basically, I'd like to know what the obstacles are to their use.

Unfortunately, I don't know of a precise list of obstacles, but here's a
list of formats supported by the various plugins:

  - Microsoft ASF (possibly strangled by patents?)
  - DVD subtitles (though I don't know what precisely this is)
  - dvdlpcmdec (this possibly involves breaking DRM, could be illegal
    because of that)
  - RealMedia and RealAudio (patent encumbered, possible license problems)
  - "xingmux" (I couldn't find any information on this one)

Thanks.

Re: GStreamer xwidget

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > I don't know, but I was told by word-of-mouth that the GStreamer
  > developers do not hold the "bad" plugins to any standard, both technical
  > and freedom-wise.

  > As such, they don't guarantee that those plugins are free software.

If some of them are nonfree, we should not mention even the existence
of the category of "bad" plug-ins.

(It is too bad that they inform the users about them at all.)

  > Unfortunately, I don't know of a precise list of obstacles, but here's a
  > list of formats supported by the various plugins:

  >   - Microsoft ASF (possibly strangled by patents?)
  >   - DVD subtitles (though I don't know what precisely this is)
  >   - dvdlpcmdec (this possibly involves breaking DRM, could be illegal
  >     because of that)
  >   - RealMedia and RealAudio (patent encumbered, possible license problems)
  >   - "xingmux" (I couldn't find any information on this one)

Our way of looking at this kind of issue is very different from
theirs.  If a format is patented, we might have various different
responses to that, depending on the details of the situation.  But one
response we would certainly NOT have is, "You must not violate some
divine patent-holder's sacred rights."

If the plug-in violates the DMCA, we would not take the risk of
distributing it ourselves, but aside from that, we see nothing wrong
with its use on that score.  Breaking DRM is not wrong in the
slightest.  The injustice of DRM is in using it to restrict others.

However, if we don't know the situations for these plug-ins, I think
we have to avoid mentioning the ugly plug-ins unless we learned the situation
with each one.  And that's work we don't have an easy way to do.




-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: GStreamer xwidget

[Lars Ingebrigtsen]

Richard Stallman <rms@gnu.org> writes:

>   > Reading web pages that have embedded videos is quite common.
>
> How _exactly_ does that relate to letting Emacs make GStreamer windows?

Windows?  There's no GStreamer windows.

Po Lu's patch is for putting GStreamer widgets into buffers, and this
will allow eww to display the videos embedded in web pages in Emacs.

> Please describe the scenario clearly and spell out the argument.
> That way we can tell what conclusion really follows.

You seem to be demanding a lot of stuff here.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: GStreamer xwidget

[Arthur Miller]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Richard Stallman <rms@gnu.org> writes:
>
>>   > Reading web pages that have embedded videos is quite common.
>>
>> How _exactly_ does that relate to letting Emacs make GStreamer windows?
>
> Windows?  There's no GStreamer windows.
>
> Po Lu's patch is for putting GStreamer widgets into buffers, and this
> will allow eww to display the videos embedded in web pages in Emacs.
>
>> Please describe the scenario clearly and spell out the argument.
>> That way we can tell what conclusion really follows.
>
> You seem to be demanding a lot of stuff here.

A possible application scenario would be also to stream or capture
your web camera; for example for chats or educational material, maybe playing
with image detection with Emacs and Elisp etc.

Re: GStreamer xwidget

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > A possible application scenario would be also to stream or capture
  > your web camera;

That looks like is a niche scenario.  Does the argument for including
this feature rest on that scenario?  If so, it's pretty clear that
this isn't worth the effort of maintaining.

My point here is, niche uses are not enough to affect the issue at
hand.

I am sure the GStreamer feature has more important uses.  I'm not sure
what makes them so important, but if people think they are useful
enough to be worth the effort of maintaining it, I am sure that's true.

The issue that worries me is a potential deeper problem.  At present
it looks like things are probably ok -- see the other email -- but
it's not clear yet.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: GStreamer xwidget

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > How _exactly_ does that relate to letting Emacs make GStreamer windows?

  > Windows?  There's no GStreamer windows.

  > Po Lu's patch is for putting GStreamer widgets into buffers, and this
  > will allow eww to display the videos embedded in web pages in Emacs.

Thank you for telling me this.  I don't know how this patch works, or
how GStreamer works, or how eww works -- that's why I am asking these
questions.

It's not easy to recognize all the questions that need to be asked.
Sometimes I make an assumption without realizing it, such as assuming
that a widget would be handled by telling a window to display it, and
it turns out that's mistaken.

  > > Please describe the scenario clearly and spell out the argument.
  > > That way we can tell what conclusion really follows.

  > You seem to be demanding a lot of stuff here.

I'm asking the questions that will enable me to understand the
situation so I can tell whether this feature fits the goals and values
of the GNU Project.  It is important to recognize situations where a
feature raises doubts, and think carefully about the issue before
installing the feature.

When we neglect to have that discussion in advance, the GNU Project
can drift into a contradictory moral position, and that is very bad.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: GStreamer xwidget

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

I just saw some messages from Saturday that I had overlooked while
falling behind.

  > >   > It depends on GStreamer, gst-plugins-base, and gst-plugins-good and is
  > >   > thus a compile-time option.

  > The plugins that this change takes advantage of will only stream free
  > software, as it doesn't depend on gst-plugins-ugly or gst-plugins-bad.

Can you please show me how Emacs invokes get-plugins-good?
How does that function specify plugins to use?

  > > We also have to think about whether it is good or bad to introduce
  > > this nonmodularity.  What is better about showing videos this way,
  > > rather than just launching vlc in another window?

  > People seem to want that in Emacs.

That's not a very convincing reason.

If there is nothing bad about connecting Emacs to GStreamer, then
there's no reason to oppose it.  But if there is something bad,
it would take a good, strong reason to overcome that counter-reason.

  > Reading web pages that have embedded videos is quite common.

Those videos are very often links to Youtube, which depend on running
nonfree JS code.    If the aim of linking with GStreamer is to make
those embedded Youtube videos work in Emacs, we should not do it!

Unless we can bypass the JS code of Youtube.

We may be able to do that, if we make those embedded video links
automatically talk to invidious proxies rather than to youtube.

Can someone see how to implement that?

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: GStreamer xwidget

[Po Lu]

Richard Stallman <rms@gnu.org> writes:

> Those videos are very often links to Youtube, which depend on running
> nonfree JS code.    If the aim of linking with GStreamer is to make
> those embedded Youtube videos work in Emacs, we should not do it!

> Unless we can bypass the JS code of Youtube.

Don't worry, we will not load the proprietary JavaScript, as shr (which
is the program that motivated me to work on this feature) is incapable
of running any JavaScript at all.

> We may be able to do that, if we make those embedded video links
> automatically talk to invidious proxies rather than to youtube.

> Can someone see how to implement that?

OTOH, that may actually be simple to implement, depending on the
interface exposed by invidious.

There are also programs such as youtube-dl, which allow to view YouTube
videos without running any JavaScript.

Perhaps we could use their interface instead, which other free software
such as mpv already makes use of.

But this is becoming something of a tangent.

Re: GStreamer xwidget

[Alexandre Garreau]

Le merkredo, 1-a de decembro 2021, 8-a horo kaj 7:47 CET Richard Stallman 
a écrit :
> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
> 
> I just saw some messages from Saturday that I had overlooked while
> falling behind.
> 
>   > >   > It depends on GStreamer, gst-plugins-base, and
>   > >   > gst-plugins-good and is
>   > >   > thus a compile-time option.
>   > 
>   > The plugins that this change takes advantage of will only stream
>   > free
>   > software, as it doesn't depend on gst-plugins-ugly or
>   > gst-plugins-bad.
> 
> Can you please show me how Emacs invokes get-plugins-good?
> How does that function specify plugins to use?
> 
>   > > We also have to think about whether it is good or bad to introduce
>   > > this nonmodularity.  What is better about showing videos this way,
>   > > rather than just launching vlc in another window?
>   > 
>   > People seem to want that in Emacs.
> 
> That's not a very convincing reason.

Windowing is confusing, while embeding is more ergonomic, because it’s 
more straightforward to figure out/remember where things are, since they’re 
more related/constrained

>   > Reading web pages that have embedded videos is quite common.
> 
> Those videos are very often links to Youtube, which depend on running
> nonfree JS code.    If the aim of linking with GStreamer is to make
> those embedded Youtube videos work in Emacs, we should not do it!
> 
> Unless we can bypass the JS code of Youtube.
> 
> We may be able to do that, if we make those embedded video links
> automatically talk to invidious proxies rather than to youtube.
> 
> Can someone see how to implement that?

It would be better to mimick mpv or vlc in that respect, and use youtube-
dl or libquvi to get the direct links to the streams, and give that to 
gstreamer.  It would avoid an intermediary (what if the invidious proxy is 
malicious, requires a proprietary captcha, is overloaded, or stops 
working?).  Plus youtube-dl would allow that to work with most streaming 
platforms, not only youtube.  And youtube-dl devs are really active and do 
most of the up-to-date work of reverse engineering to keep those working 
correctly.

Isn’t that how ViewTube used to work?