Touchscreen support

Touchscreen support

[Po Lu via Emacs development discussions.]

[-- Attachment #1: Type: text/plain, Size: 369 bytes --]


I implemented the events required for full touchscreen support for
XInput 2.  These events are generated only for DirectTouch devices,
which means real touchscreens only, no touchpads.

Please take a look and see if the API I'm exposing is useful, and also
easy enough to implement on non-X platforms.

If there are no problems, I would like to install this.

Thanks.

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Add-support-for-touchscreen-events-generated-by-XInp.patch --]
[-- Type: text/x-patch, Size: 17841 bytes --]

From f9df927f3f70c348ceee50eb88c1ddad7eff2c4f Mon Sep 17 00:00:00 2001
From: Po Lu <luangruo@yahoo.com>
Date: Thu, 16 Dec 2021 15:15:28 +0800
Subject: [PATCH] Add support for touchscreen events generated by XInput 2

* doc/lispref/commands.texi (Input Events): Add "Touchscreen
Events" to menu.
(Touchscreen Events): New node.
* etc/NEWS: Announce new event types.

* src/keyboard.c (make_lispy_event): Handle touchscreen events.
(syms_of_keyboard): New symbols `touchscreen-begin',
`touchscreen-end' and `touchscreen-update'.
* src/termhooks.h (enum event_kind): New touchscreen events.

* src/xfns.c (setup_xi_event_mask): Set up touch event mask if
appropriate.
(x_window) [USE_GTK]: Call `setup_xi_event_mask'.

* src/xterm.c (x_free_xi_devices): Free touchpoints.
(x_init_master_valuators): Handle all devices and set master and
touch mode flags appropriately.
(xi_reset_scroll_valuators_for_device_id):
(x_get_scroll_valuator_delta): Ignore slave devices.

(xi_link_touch_point):
(xi_unlink_touch_point):
(xi_find_touch_point): New functions.
(handle_one_xevent): Handle XI touch events.

* src/xterm.h (struct xi_touch_point_t): New structure.
(struct xi_device_t): Add touchpoints field.
---
 doc/lispref/commands.texi |  53 ++++++++++
 etc/NEWS                  |   6 ++
 src/keyboard.c            |  43 ++++++++
 src/termhooks.h           |  13 +++
 src/xfns.c                |  21 +++-
 src/xterm.c               | 210 +++++++++++++++++++++++++++++++++++---
 src/xterm.h               |  11 ++
 7 files changed, 344 insertions(+), 13 deletions(-)

diff --git a/doc/lispref/commands.texi b/doc/lispref/commands.texi
index cc1c216d57..77a5385241 100644
--- a/doc/lispref/commands.texi
+++ b/doc/lispref/commands.texi
@@ -1175,6 +1175,7 @@ Input Events
 * Button-Down Events::          A button was pushed and not yet released.
 * Repeat Events::               Double and triple click (or drag, or down).
 * Motion Events::               Just moving the mouse, not pushing a button.
+* Touchscreen Events::          Tapping and moving fingers on a touchscreen.
 * Focus Events::                Moving the mouse between frames.
 * Xwidget Events::              Events generated by xwidgets.
 * Misc Events::                 Other events the system can generate.
@@ -1835,6 +1836,58 @@ Motion Events
 as the mouse cursor remains pointing to the same glyph in the text.
 @end defvar
 
+@node Touchscreen Events
+@subsection Touchscreen Events
+@cindex touchscreen events
+
+Some window systems provide support for input devices that react to
+the user's finger, and translate those finger movements into points at
+an on-screen position.  These input devices are known as touchscreens,
+and Emacs reports the movements they generate as @dfn{touchscreen
+events}.
+
+Most individual events generated by a touchscreen only have meaning as
+part of a larger sequence of other events: for instance, the simple
+operation of tapping the touchscreen involves the user placing and
+releasing a finger on the touchscreen, and swiping the display to
+scroll it involves placing a finger, moving it many times upwards or
+downwards, and then releasing the finger.
+
+While a simplistic model consisting of one finger is adequate for taps
+and scrolling, most applications support more complicated gestures
+that involve multiple fingers, where the position of each finger is
+represented by a @dfn{touch point}.  For instance, a ``pinch to zoom''
+gesture might consist of the user placing two fingers and moving them
+individually in opposite directions, where the distance between the
+positions of their individual points determine the amount by which to
+zoom the display, and the center of an imaginary line between those
+positions determines where to pan the display after zooming.
+
+@cindex touch point representation
+The low-level touchscreen events described below can be used to
+implement all the touch sequences described above.  In those events,
+each point is represented by a cons of an arbitrary number identifying
+the point and a mouse position list (@pxref{Click Events}) specifying
+the position of the finger when the event occurred.
+
+@table @code
+@cindex @code{touchscreen-begin} event
+@item (touchscreen-begin @var{point})
+This event is sent when @var{point} is created by the user pressing a
+finger against the touchscreen.
+
+@cindex @code{touchscreen-update} event
+@item (touchscreen-update @var{points})
+This event is sent when an point on the touchscreen has changed
+position.  @var{points} is a list of points containing the up-to-date
+positions of each point currently on the touchscreen.
+
+@cindex @code{touchscreen-end} event
+@item (touchscreen-end @var{point-id})
+This event is sent when the point identified by the number
+@var{point-id} is no longer present on the display.
+@end table
+
 @node Focus Events
 @subsection Focus Events
 @cindex focus event
diff --git a/etc/NEWS b/etc/NEWS
index 1d78f1f5c3..6203537475 100644
--- a/etc/NEWS
+++ b/etc/NEWS
@@ -1156,6 +1156,12 @@ This variable is bound to t during the preparation of a "*Help*" buffer.
 month, day, or time.  For example, (date-to-time "2021-12-04") now
 assumes a time of 00:00 instead of signaling an error.
 
++++
+** New events for taking advantages of touchscreen devices.
+The events 'touchscreen-down', 'touchscreen-update' and
+'touchscreen-end' have been added to take better advantage of
+touch-capable display panels.
+
 \f
 * Changes in Emacs 29.1 on Non-Free Operating Systems
 
diff --git a/src/keyboard.c b/src/keyboard.c
index 899c9109c2..ee4b18b012 100644
--- a/src/keyboard.c
+++ b/src/keyboard.c
@@ -6010,6 +6010,46 @@ make_lispy_event (struct input_event *event)
 	return list2 (Qtouch_end, position);
       }
 
+    case TOUCHSCREEN_BEGIN_EVENT:
+    case TOUCHSCREEN_END_EVENT:
+      {
+	Lisp_Object x, y, id, position;
+	struct frame *f = XFRAME (event->frame_or_window);
+
+	id = event->arg;
+	x = event->x;
+	y = event->y;
+
+	position = make_lispy_position (f, x, y, event->timestamp);
+
+	return list2 (((event->kind
+			== TOUCHSCREEN_BEGIN_EVENT)
+		       ? Qtouchscreen_begin
+		       : Qtouchscreen_end),
+		      Fcons (id, position));
+      }
+
+    case TOUCHSCREEN_UPDATE_EVENT:
+      {
+	Lisp_Object x, y, id, position, tem, it, evt;
+	struct frame *f = XFRAME (event->frame_or_window);
+	evt = Qnil;
+
+	for (tem = event->arg; CONSP (tem); tem = XCDR (tem))
+	  {
+	    it = XCAR (tem);
+
+	    x = XCAR (it);
+	    y = XCAR (XCDR (it));
+	    id = XCAR (XCDR (XCDR (it)));
+
+	    position = make_lispy_position (f, x, y, event->timestamp);
+	    evt = Fcons (Fcons (id, position), evt);
+	  }
+
+	return list2 (Qtouchscreen_update, evt);
+      }
+
 #ifdef USE_TOOLKIT_SCROLL_BARS
 
       /* We don't have down and up events if using toolkit scroll bars,
@@ -12257,6 +12297,9 @@ syms_of_keyboard (void)
 	       doc: /* Normal hook run when clearing the echo area.  */);
 #endif
   DEFSYM (Qecho_area_clear_hook, "echo-area-clear-hook");
+  DEFSYM (Qtouchscreen_begin, "touchscreen-begin");
+  DEFSYM (Qtouchscreen_end, "touchscreen-end");
+  DEFSYM (Qtouchscreen_update, "touchscreen-update");
   Fset (Qecho_area_clear_hook, Qnil);
 
   DEFVAR_LISP ("lucid-menu-bar-dirty-flag", Vlucid_menu_bar_dirty_flag,
diff --git a/src/termhooks.h b/src/termhooks.h
index f64c19e039..587d1eef91 100644
--- a/src/termhooks.h
+++ b/src/termhooks.h
@@ -274,6 +274,19 @@ #define EMACS_TERMHOOKS_H
      In the future, this may take into account other multi-touch
      events generated from touchscreens and such.  */
   , TOUCH_END_EVENT
+
+  /* In a TOUCHSCREEN_UPDATE_EVENT, ARG is a list of elements of the
+     form (X Y ID), where X and Y are the coordinates of the
+     touchpoint relative to the top-left corner of the frame, and ID
+     is a unique number identifying the touchpoint.
+
+     In TOUCHSCREEN_BEGIN_EVENT and TOUCHSCREEN_END_EVENT, ARG is the
+     unique ID of the touchpoint, and X and Y are the frame-relative
+     positions of the touchpoint.  */
+
+  , TOUCHSCREEN_UPDATE_EVENT
+  , TOUCHSCREEN_BEGIN_EVENT
+  , TOUCHSCREEN_END_EVENT
 };
 
 /* Bit width of an enum event_kind tag at the start of structs and unions.  */
diff --git a/src/xfns.c b/src/xfns.c
index b5694829ae..dc25d7bfca 100644
--- a/src/xfns.c
+++ b/src/xfns.c
@@ -2916,7 +2916,7 @@ initial_set_up_x_back_buffer (struct frame *f)
   unblock_input ();
 }
 
-#if defined HAVE_XINPUT2 && !defined USE_GTK
+#if defined HAVE_XINPUT2
 static void
 setup_xi_event_mask (struct frame *f)
 {
@@ -2927,6 +2927,9 @@ setup_xi_event_mask (struct frame *f)
   mask.mask = m = alloca (l);
   memset (m, 0, l);
   mask.mask_len = l;
+
+  block_input ();
+#ifndef USE_GTK
   mask.deviceid = XIAllMasterDevices;
 
   XISetMask (m, XI_ButtonPress);
@@ -2945,14 +2948,25 @@ setup_xi_event_mask (struct frame *f)
 		  &mask, 1);
 
   memset (m, 0, l);
+#endif /* !USE_GTK */
+
   mask.deviceid = XIAllDevices;
 
   XISetMask (m, XI_PropertyEvent);
   XISetMask (m, XI_HierarchyChanged);
   XISetMask (m, XI_DeviceChanged);
+#ifdef XI_TouchBegin
+  if (FRAME_DISPLAY_INFO (f)->xi2_version >= 2)
+    {
+      XISetMask (m, XI_TouchBegin);
+      XISetMask (m, XI_TouchUpdate);
+      XISetMask (m, XI_TouchEnd);
+    }
+#endif
   XISelectEvents (FRAME_X_DISPLAY (f),
 		  FRAME_X_WINDOW (f),
 		  &mask, 1);
+  unblock_input ();
 }
 #endif
 
@@ -3249,6 +3263,11 @@ x_window (struct frame *f)
     unblock_input ();
   }
 #endif
+
+#ifdef HAVE_XINPUT2
+  if (FRAME_DISPLAY_INFO (f)->supports_xi2)
+    setup_xi_event_mask (f);
+#endif
 }
 
 #else /*! USE_GTK */
diff --git a/src/xterm.c b/src/xterm.c
index 646985bdc2..c654fe0ce0 100644
--- a/src/xterm.c
+++ b/src/xterm.c
@@ -353,6 +353,8 @@ x_extension_initialize (struct x_display_info *dpyinfo)
 static void
 x_free_xi_devices (struct x_display_info *dpyinfo)
 {
+  struct xi_touch_point_t *tem, *last;
+
   block_input ();
 
   if (dpyinfo->num_devices)
@@ -362,6 +364,14 @@ x_free_xi_devices (struct x_display_info *dpyinfo)
 	  XIUngrabDevice (dpyinfo->display, dpyinfo->devices[i].device_id,
 			  CurrentTime);
 	  xfree (dpyinfo->devices[i].valuators);
+
+	  tem = dpyinfo->devices[i].touchpoints;
+	  while (tem)
+	    {
+	      last = tem;
+	      tem = tem->next;
+	      xfree (last);
+	    }
 	}
 
       xfree (dpyinfo->devices);
@@ -407,7 +417,7 @@ x_init_master_valuators (struct x_display_info *dpyinfo)
   block_input ();
   x_free_xi_devices (dpyinfo);
   infos = XIQueryDevice (dpyinfo->display,
-			 XIAllMasterDevices,
+			 XIAllDevices,
 			 &ndevices);
 
   if (!ndevices)
@@ -432,6 +442,10 @@ x_init_master_valuators (struct x_display_info *dpyinfo)
 	  xi_device->grab = 0;
 	  xi_device->valuators =
 	    xmalloc (sizeof *xi_device->valuators * device->num_classes);
+	  xi_device->touchpoints = NULL;
+	  xi_device->master_p = (device->use == XIMasterKeyboard
+				 || device->use == XIMasterPointer);
+	  xi_device->direct_p = false;
 
 	  for (int c = 0; c < device->num_classes; ++c)
 	    {
@@ -442,22 +456,36 @@ x_init_master_valuators (struct x_display_info *dpyinfo)
 		  {
 		    XIScrollClassInfo *info =
 		      (XIScrollClassInfo *) device->classes[c];
-		    struct xi_scroll_valuator_t *valuator =
-		      &xi_device->valuators[actual_valuator_count++];
-
-		    valuator->horizontal
-		      = (info->scroll_type == XIScrollTypeHorizontal);
-		    valuator->invalid_p = true;
-		    valuator->emacs_value = DBL_MIN;
-		    valuator->increment = info->increment;
-		    valuator->number = info->number;
+		    struct xi_scroll_valuator_t *valuator;
+
+		    if (xi_device->master_p)
+		      {
+			valuator = &xi_device->valuators[actual_valuator_count++];
+			valuator->horizontal
+			  = (info->scroll_type == XIScrollTypeHorizontal);
+			valuator->invalid_p = true;
+			valuator->emacs_value = DBL_MIN;
+			valuator->increment = info->increment;
+			valuator->number = info->number;
+		      }
+
 		    break;
 		  }
+#endif
+#ifdef XITouchClass /* XInput 2.2 */
+		case XITouchClass:
+		  {
+		    XITouchClassInfo *info;
+
+		    info = (XITouchClassInfo *) device->classes[c];
+		    xi_device->direct_p = info->mode == XIDirectTouch;
+		  }
 #endif
 		default:
 		  break;
 		}
 	    }
+
 	  xi_device->scroll_valuator_count = actual_valuator_count;
 	}
     }
@@ -484,7 +512,7 @@ x_get_scroll_valuator_delta (struct x_display_info *dpyinfo, int device_id,
     {
       struct xi_device_t *device = &dpyinfo->devices[i];
 
-      if (device->device_id == device_id)
+      if (device->device_id == device_id && device->master_p)
 	{
 	  for (int j = 0; j < device->scroll_valuator_count; ++j)
 	    {
@@ -534,6 +562,63 @@ xi_device_from_id (struct x_display_info *dpyinfo, int deviceid)
   return NULL;
 }
 
+#ifdef XI_TouchBegin
+
+static void
+xi_link_touch_point (struct xi_device_t *device,
+		     int detail, double x, double y)
+{
+  struct xi_touch_point_t *touchpoint;
+
+  touchpoint = xmalloc (sizeof *touchpoint);
+  touchpoint->next = device->touchpoints;
+  touchpoint->x = x;
+  touchpoint->y = y;
+  touchpoint->number = detail;
+
+  device->touchpoints = touchpoint;
+}
+
+static void
+xi_unlink_touch_point (int detail,
+		       struct xi_device_t *device)
+{
+  struct xi_touch_point_t *last, *tem;
+
+  for (last = NULL, tem = device->touchpoints; tem;
+       last = tem, tem = tem->next)
+    {
+      if (tem->number == detail)
+	{
+	  if (!last)
+	    device->touchpoints = tem->next;
+	  else
+	    last->next = tem->next;
+
+	  xfree (tem);
+	  return;
+	}
+    }
+
+  emacs_abort ();
+}
+
+static struct xi_touch_point_t *
+xi_find_touch_point (struct xi_device_t *device, int detail)
+{
+  struct xi_touch_point_t *point;
+
+  for (point = device->touchpoints; point; point = point->next)
+    {
+      if (point->number == detail)
+	return point;
+    }
+
+  return NULL;
+}
+
+#endif /* XI_TouchBegin */
+
 static void
 xi_grab_or_ungrab_device (struct xi_device_t *device,
 			  struct x_display_info *dpyinfo,
@@ -570,7 +655,7 @@ xi_reset_scroll_valuators_for_device_id (struct x_display_info *dpyinfo, int id)
   struct xi_device_t *device = xi_device_from_id (dpyinfo, id);
   struct xi_scroll_valuator_t *valuator;
 
-  if (!device)
+  if (!device || !device->master_p)
     return;
 
   if (!device->scroll_valuator_count)
@@ -10242,6 +10327,9 @@ handle_one_xevent (struct x_display_info *dpyinfo,
 
 	      device = xi_device_from_id (dpyinfo, xev->deviceid);
 
+	      if (!device || !device->master_p)
+		goto XI_OTHER;
+
 	      bv.button = xev->detail;
 	      bv.type = xev->evtype == XI_ButtonPress ? ButtonPress : ButtonRelease;
 	      bv.x = lrint (xev->event_x);
@@ -10765,6 +10853,104 @@ handle_one_xevent (struct x_display_info *dpyinfo,
 	  case XI_DeviceChanged:
 	    x_init_master_valuators (dpyinfo);
 	    goto XI_OTHER;
+#ifdef XI_TouchBegin
+	  case XI_TouchBegin:
+	    {
+	      struct xi_device_t *device;
+	      device = xi_device_from_id (dpyinfo, xev->deviceid);
+
+	      if (!device)
+		goto XI_OTHER;
+
+	      if (xi_find_touch_point (device, xev->detail))
+		emacs_abort ();
+
+	      xi_link_touch_point (device, xev->detail, xev->event_x,
+				   xev->event_y);
+	      f = x_any_window_to_frame (dpyinfo, xev->event);
+
+	      if (f && device->direct_p)
+		{
+		  inev.ie.kind = TOUCHSCREEN_BEGIN_EVENT;
+		  inev.ie.timestamp = xev->time;
+		  XSETFRAME (inev.ie.frame_or_window, f);
+		  XSETINT (inev.ie.x, lrint (xev->event_x));
+		  XSETINT (inev.ie.y, lrint (xev->event_y));
+		  XSETINT (inev.ie.arg, xev->detail);
+
+		  XIAllowTouchEvents (dpyinfo->display, xev->deviceid,
+				      xev->detail, xev->event, XIAcceptTouch);
+		}
+
+	      goto XI_OTHER;
+	    }
+	  case XI_TouchUpdate:
+	    {
+	      struct xi_device_t *device;
+	      struct xi_touch_point_t *touchpoint;
+	      Lisp_Object arg = Qnil;
+
+	      device = xi_device_from_id (dpyinfo, xev->deviceid);
+
+	      if (!device)
+		goto XI_OTHER;
+
+	      touchpoint = xi_find_touch_point (device, xev->detail);
+
+	      if (!touchpoint)
+		emacs_abort ();
+
+	      touchpoint->x = xev->event_x;
+	      touchpoint->y = xev->event_y;
+
+	      f = x_any_window_to_frame (dpyinfo, xev->event);
+
+	      if (f && device->direct_p)
+		{
+		  inev.ie.kind = TOUCHSCREEN_UPDATE_EVENT;
+		  inev.ie.timestamp = xev->time;
+		  XSETFRAME (inev.ie.frame_or_window, f);
+
+		  for (touchpoint = device->touchpoints;
+		       touchpoint; touchpoint = touchpoint->next)
+		    {
+		      arg = Fcons (list3i (lrint (touchpoint->x),
+					   lrint (touchpoint->y),
+					   lrint (touchpoint->number)),
+				   arg);
+		    }
+
+		  inev.ie.arg = arg;
+		}
+
+	      goto XI_OTHER;
+	    }
+	  case XI_TouchEnd:
+	    {
+	      struct xi_device_t *device;
+
+	      device = xi_device_from_id (dpyinfo, xev->deviceid);
+
+	      if (!device)
+		goto XI_OTHER;
+
+	      xi_unlink_touch_point (xev->detail, device);
+
+	      f = x_any_window_to_frame (dpyinfo, xev->event);
+
+	      if (f && device->direct_p)
+		{
+		  inev.ie.kind = TOUCHSCREEN_END_EVENT;
+		  inev.ie.timestamp = xev->time;
+		  XSETFRAME (inev.ie.frame_or_window, f);
+		  XSETINT (inev.ie.x, lrint (xev->event_x));
+		  XSETINT (inev.ie.y, lrint (xev->event_y));
+		  XSETINT (inev.ie.arg, xev->detail);
+		}
+
+	      goto XI_OTHER;
+	    }
+#endif
 	  default:
 	    goto XI_OTHER;
 	  }
diff --git a/src/xterm.h b/src/xterm.h
index 7abe168bc6..d9ace002d5 100644
--- a/src/xterm.h
+++ b/src/xterm.h
@@ -179,13 +179,24 @@ #define MAX_CLIP_RECTS 2
   int horizontal;
 };
 
+struct xi_touch_point_t
+{
+  struct xi_touch_point_t *next;
+
+  int number;
+  double x, y;
+};
+
 struct xi_device_t
 {
   int device_id;
   int scroll_valuator_count;
   int grab;
+  bool master_p;
+  bool direct_p;
 
   struct xi_scroll_valuator_t *valuators;
+  struct xi_touch_point_t *touchpoints;
 };
 #endif
 
-- 
2.33.1

Re: Touchscreen support

[Lars Ingebrigtsen]

Po Lu via "Emacs development discussions." <emacs-devel@gnu.org> writes:

> I implemented the events required for full touchscreen support for
> XInput 2.  These events are generated only for DirectTouch devices,
> which means real touchscreens only, no touchpads.

That's fantastic -- proper touchscreen support is something I've been
wanting to have in Emacs for at least a decade.

One question:

> Please take a look and see if the API I'm exposing is useful, and also
> easy enough to implement on non-X platforms.
> +The low-level touchscreen events described below can be used to
> +implement all the touch sequences described above.  In those events,
> +each point is represented by a cons of an arbitrary number identifying
> +the point and a mouse position list (@pxref{Click Events}) specifying
> +the position of the finger when the event occurred.

What would the implementation of a (for instance) "three finger swipe
left" event look like based on these primitives?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Touchscreen support

[Po Lu]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Po Lu via "Emacs development discussions." <emacs-devel@gnu.org> writes:
>
>> I implemented the events required for full touchscreen support for
>> XInput 2.  These events are generated only for DirectTouch devices,
>> which means real touchscreens only, no touchpads.
>
> That's fantastic -- proper touchscreen support is something I've been
> wanting to have in Emacs for at least a decade.
>
> One question:
>
>> Please take a look and see if the API I'm exposing is useful, and also
>> easy enough to implement on non-X platforms.
>> +The low-level touchscreen events described below can be used to
>> +implement all the touch sequences described above.  In those events,
>> +each point is represented by a cons of an arbitrary number identifying
>> +the point and a mouse position list (@pxref{Click Events}) specifying
>> +the position of the finger when the event occurred.
>
> What would the implementation of a (for instance) "three finger swipe
> left" event look like based on these primitives?

You would wait for three consecutive `touchscreen-down' events, then
decide if the fingers are in an appropriate location, and then keep
reading for `touchscreen-update' events, calculating their movement.

I can't think of that in any more detail right now, but it should be
similar to how it's implemented on X, whose APIs are paralleled by this
implementation.

Thanks.

Re: Touchscreen support

[Po Lu]

Po Lu via "Emacs development discussions." <emacs-devel@gnu.org> writes:

> If there are no problems, I would like to install this.

I installed this after testing it thoroughly on a simulated touchscreen.
Testing with a real touchscreen will be very much appreciated.

Even though the user commands for taking advantage of touchscreens have
not been implemented, at least GTK builds with XInput 2 don't crash
anymore when receiving touchscreen events.

We will want an abstraction layer (probably in Lisp) later on that
translates the low level events generated by this code into commonly
used gestures.  Users can then bind things as usual to those gestures.

Lars, are you interested in doing this work?  I'd be happy to help if
necessary.

Re: Touchscreen support

[Lars Ingebrigtsen]

Po Lu <luangruo@yahoo.com> writes:

> We will want an abstraction layer (probably in Lisp) later on that
> translates the low level events generated by this code into commonly
> used gestures.  Users can then bind things as usual to those gestures.
>
> Lars, are you interested in doing this work?  I'd be happy to help if
> necessary.

I'm definitely interested in seeing this happen, but I'm afraid I don't
have the time to take on this -- I've got a backlog of stuff that I have
to get done.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Touchscreen support

[Stefan Monnier]

> We will want an abstraction layer (probably in Lisp) later on that
> translates the low level events generated by this code into commonly
> used gestures.  Users can then bind things as usual to those gestures.

What I don't understand is why we need to (re)implement this.  I mean,
obvious all other applications will want/need something similar, so
isn't there some standard library offering the functionality already?


        Stefan

Re: Touchscreen support

[Po Lu]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> What I don't understand is why we need to (re)implement this.  I mean,
> obvious all other applications will want/need something similar, so
> isn't there some standard library offering the functionality already?

There isn't one we can acceptably use on X, and the other platforms
(such as GTK and NS) offer very different meanings of what constitutes a
"gesture" which are difficult to abstract around in a useful manner.

Other cross-platform programs I examined such as Mozilla Firefox also
implement gesture recognition from scratch.

Thanks.

Re: Touchscreen support

[Stefan Monnier]

Po Lu [2021-12-17 20:36:20] wrote:
> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>> What I don't understand is why we need to (re)implement this.  I mean,
>> obvious all other applications will want/need something similar, so
>> isn't there some standard library offering the functionality already?
>
> There isn't one we can acceptably use on X, and the other platforms
> (such as GTK and NS) offer very different meanings of what constitutes a
> "gesture" which are difficult to abstract around in a useful manner.
>
> Other cross-platform programs I examined such as Mozilla Firefox also
> implement gesture recognition from scratch.

Hmm... that sucks.  Hopefully this will improve in the longer term,


        Stefan

Re: Touchscreen support

[Po Lu]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> Hmm... that sucks.  Hopefully this will improve in the longer term,

Sadly, I doubt that will happen, the reason being that the X world is
moving further and further from 'all batteries included' to 'just use a
toolkit', and programs (such as Emacs or Firefox) that directly act as X
clients are getting more and more rare.

In the eyes of the X desktop people and toolkit developers, program
portability between toolkits (or window systems) is also not as
important as it used to be.

As an extreme indicator of the direction things are moving in, it is
practically impossible to directly act as a Wayland client without some
kind of toolkit in between.  The protocol (and its common extensions) is
extremely spartan and leaves most functionality to the imagination of
toolkit developers, including basics such as window decorations,
controls, and even dragging and grabs.

Thanks.

Re: Touchscreen support

[Stefan Monnier]

Po Lu [2021-12-17 21:48:46] wrote:
> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>> Hmm... that sucks.  Hopefully this will improve in the longer term,
>
> Sadly, I doubt that will happen, the reason being that the X world is
> moving further and further from 'all batteries included' to 'just use a
> toolkit', and programs (such as Emacs or Firefox) that directly act as X
> clients are getting more and more rare.

I can't see why we should not use such a toolkit.  We already do for
many other things.


        Stefan

Re: Touchscreen support

[Po Lu]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> I can't see why we should not use such a toolkit.  We already do for
> many other things.

As I said, the "gestures" exposed by the various toolkits mean extremely
different things depending on the toolkit, which are impossible to
abstract around in a useful manner.

The PGTK branch could in theory use GtkGesture, but it would not be very
useful.

Thanks.

Re: Touchscreen support

[Stefan Monnier]

Po Lu [2021-12-18 08:02:38] wrote:
> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>> I can't see why we should not use such a toolkit.  We already do for
>> many other things.
> As I said, the "gestures" exposed by the various toolkits mean extremely
> different things depending on the toolkit, which are impossible to
> abstract around in a useful manner.

That's the part I'm hoping to improve over time.


        Stefan

Re: Touchscreen support

[Stefan Monnier]

> That's the part I'm hoping to improve over time.
                             ^^
                            will

That came out wrong: I do not plan on working on that myself.


        Stefan

Re: Touchscreen support

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > In the eyes of the X desktop people and toolkit developers, program
  > portability between toolkits (or window systems) is also not as
  > important as it used to be.

Could you explain to me what "portability between toolkits" means in
a concrete example?

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Touchscreen support

[Po Lu]

Richard Stallman <rms@gnu.org> writes:

> Could you explain to me what "portability between toolkits" means in a
> concrete example?

For instance, such a portable program will, even if written to use GTK+,
be easily adapted to use GNUstep instead.

Re: Touchscreen support

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > Could you explain to me what "portability between toolkits" means in a
  > > concrete example?

  > For instance, such a portable program will, even if written to use GTK+,
  > be easily adapted to use GNUstep instead.

I see.  Thanks.  That suggests other pertinent questions:

What is causing this to become more difficult?

Are there going to be fewer different toolkits than in the past?

Will any of them be more universal than in the past?


-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Touchscreen support

[Po Lu]

Richard Stallman <rms@gnu.org> writes:

> I see.  Thanks.  That suggests other pertinent questions:
>
> What is causing this to become more difficult?

Probably the proliferation of very different high-level abstractions
among toolkits, which then proceed to discourage applications from using
the lower-level (but more general) alternatives.

> Are there going to be fewer different toolkits than in the past?

Yes, certainly so.

> Will any of them be more universal than in the past?

I don't really think this will be the case.

Re: Touchscreen support

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > What is causing this to become more difficult?

  > Probably the proliferation of very different high-level abstractions
  > among toolkits, which then proceed to discourage applications from using
  > the lower-level (but more general) alternatives.

That sounds like a big problem, but it is described in a very abstract
way.  Can you present a concrete exanple of such a difference in
abstractions?

Would it be possible to invent a higher-level abstraction to unify
these toolkits?

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Touchscreen support

[Po Lu]

Richard Stallman <rms@gnu.org> writes:

> That sounds like a big problem, but it is described in a very abstract
> way.  Can you present a concrete exanple of such a difference in
> abstractions?

For example, programs in GTK+ aren't supposed to create windows or
widgets manually anymore.  Instead, GTK provides a system to create
"applications", which run in the background and respond to requests to
open files or to display a default window, and it also provides an
XML-based format for describing the layout of widgets and windows.

In GTK 4, menus cannot even be modified by applications on-the-fly
anymore.  You are again supposed to use a GTK-specific XML format to
describe the menus of your application in a way that cannot be changed
at run-time.

What used to be a few ifdefs (and some code) away now requires you to
write a different XML file for each toolkit, and to redesign your
application to fit various unnecessary limitations imposed by the
toolkit developers.

> Would it be possible to invent a higher-level abstraction to unify
> these toolkits?

I'm not sure.

Re: Touchscreen support

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > For example, programs in GTK+ aren't supposed to create windows or
  > widgets manually anymore.  Instead, GTK provides a system to create
  > "applications", which run in the background and respond to requests to
  > open files or to display a default window, and it also provides an
  > XML-based format for describing the layout of widgets and windows.

1. You say "anymore", so I wonder what that refers to.  Is this a
change that was made in GTK+ since we started using it in Emacs?  Do
you have a rough idea of when that change was made?

2. Is it possible for Emacs to work with that new method?

3. Surely Emacs is not the only system that handles user-defined
menus.  How do the others work with GTK or with GNOME?

  > In GTK 4, menus cannot even be modified by applications on-the-fly
  > anymore.  You are again supposed to use a GTK-specific XML format to
  > describe the menus of your application in a way that cannot be changed
  > at run-time.

This is amazing.  Applications are not supposed to support dynamic
menus at all???

Is it possible for an application to submit a new XML file
to specify modified menus?

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Touchscreen support

[Po Lu]

Richard Stallman <rms@gnu.org> writes:

>   > For example, programs in GTK+ aren't supposed to create windows or
>   > widgets manually anymore.  Instead, GTK provides a system to create
>   > "applications", which run in the background and respond to requests to
>   > open files or to display a default window, and it also provides an
>   > XML-based format for describing the layout of widgets and windows.

> 1. You say "anymore", so I wonder what that refers to.  Is this a
> change that was made in GTK+ since we started using it in Emacs?  Do
> you have a rough idea of when that change was made?

It was made a few years ago, but I forgot when precisely.

> 2. Is it possible for Emacs to work with that new method?

Yes, but that would require very large changes.  For instance, Lisp code
wouldn't be able to create frames anymore, that task instead being
delegated to GTK+.

> This is amazing.  Applications are not supposed to support dynamic
> menus at all???

Yes.

> 3. Surely Emacs is not the only system that handles user-defined
> menus.  How do the others work with GTK or with GNOME?

The GTK developers have decided that those applications are undesirable,
so they are not supported anymore.

> Is it possible for an application to submit a new XML file to specify
> modified menus?

Yes, but it's a hack that doesn't work very well (i.e. you cannot define
several kinds of radio buttons) and is slow.  This is why I defined a
custom menu bar widget that tries to imitate the look of the built-in
widget in my (WIP) port of Emacs to GTK 4, instead of working around its
deliberate limitations.

Thanks.

Re: Touchscreen support

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > 2. Is it possible for Emacs to work with that new method?

  > Yes, but that would require very large changes.  For instance, Lisp code
  > wouldn't be able to create frames anymore, that task instead being
  > delegated to GTK+.

I am confused by this statement.  The first half seems to talk about
the interface for Lisp code, while the second talks about internals.
Why can't Lisp code direct GTK+ to create a new frame?

There may be a reason, I'm only saying I can't yet see why so.

  > > 3. Surely Emacs is not the only system that handles user-defined
  > > menus.  How do the others work with GTK or with GNOME?

  > The GTK developers have decided that those applications are undesirable,
  > so they are not supported anymore.

Does this mean they have decided not to support Emacs any more?
Have they said anything about this problem?

  >   This is why I defined a
  > custom menu bar widget that tries to imitate the look of the built-in
  > widget in my (WIP) port of Emacs to GTK 4, instead of working around its
  > deliberate limitations.

If that solution remains possible, I guess the problem isn't so bad --
a least, not with X11.

But maybe with the new fashion in toolkits that solution won't exist
any more.  Is that what you're saying?

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Touchscreen support

[Po Lu]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > > 2. Is it possible for Emacs to work with that new method?
>
>   > Yes, but that would require very large changes.  For instance, Lisp code
>   > wouldn't be able to create frames anymore, that task instead being
>   > delegated to GTK+.

> I am confused by this statement.  The first half seems to talk about
> the interface for Lisp code, while the second talks about internals.
> Why can't Lisp code direct GTK+ to create a new frame?

It's not recommended anymore by the GTK+ developers.  Frames created
that way won't fit into the GtkApplication framework.

> Does this mean they have decided not to support Emacs any more?
> Have they said anything about this problem?

They haven't mentioned Emacs specifically, though it won't immediately
affect Emacs, as we use GTK 3 instead of GTK 4.

> If that solution remains possible, I guess the problem isn't so bad --
> a least, not with X11.

It's still an imitation of the built-in GTK menu bar widget.  You can
still notice subtle differences between how it behaves and how the
built-in widget does.

> But maybe with the new fashion in toolkits that solution won't exist
> any more.  Is that what you're saying?

Yes.

Re: Touchscreen support

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > Does this mean they have decided not to support Emacs any more?
  > > Have they said anything about this problem?

  > They haven't mentioned Emacs specifically, though it won't immediately
  > affect Emacs, as we use GTK 3 instead of GTK 4.

Is it to be expected that, at some date, they will say that GTK 3 is no
longer supported and people should downgrade to GTK version 4?

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Touchscreen support

[Po Lu]

Richard Stallman <rms@gnu.org> writes:

> Is it to be expected that, at some date, they will say that GTK 3 is no
> longer supported and people should downgrade to GTK version 4?

If meant "upgrade to GTK version 4", then yes.

Re: Touchscreen support

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > Is it to be expected that, at some date, they will say that GTK 3 is no
  > > longer supported and people should downgrade to GTK version 4?

  > If meant "upgrade to GTK version 4", then yes.

For Emacs, changing to GTK 4 would be a downgrade, so that's what I call it.

Anyway, if you're right about their plans, it means that sticking with
GTK 3 is a short-term solution but not a permanent solution.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Touchscreen support

[Stefan Monnier]

> Anyway, if you're right about their plans, it means that sticking with
> GTK 3 is a short-term solution but not a permanent solution.

Such is life: Emacs outlives pretty much all the systems on which it
builds, which is why it comes with support for various toolkits: each
one of them was "the toolkit to support" at a given time, and then it
became outdated so Emacs had to add support for another one.
Same thing for font libraries, etc...


        Stefan

Re: Touchscreen support

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > Anyway, if you're right about their plans, it means that sticking with
  > > GTK 3 is a short-term solution but not a permanent solution.

  > Such is life: Emacs outlives pretty much all the systems on which it
  > builds, which is why it comes with support for various toolkits: each
  > one of them was "the toolkit to support" at a given time, and then it
  > became outdated so Emacs had to add support for another one.
  > Same thing for font libraries, etc...

I think you're right.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Touchscreen support

[chad]

[-- Attachment #1: Type: text/plain, Size: 2611 bytes --]

On Sun, Dec 26, 2021 at 11:15 PM Richard Stallman <rms@gnu.org> wrote:

>
>   > > Is it to be expected that, at some date, they will say that GTK 3 is
> no
>   > > longer supported and people should downgrade to GTK version 4?
>
>   > If meant "upgrade to GTK version 4", then yes.
>
> For Emacs, changing to GTK 4 would be a downgrade, so that's what I call
> it.
>
> Anyway, if you're right about their plans, it means that sticking with
> GTK 3 is a short-term solution but not a permanent solution.
>

In general, the currently-used toolkits have raised the floor of the "stamp
of approval" abstraction layer. As a (somewhat contrived) example, once
upon a time one might have used xmh to read email. It was an X application
that used the Athena Widgets and also could use the 3D Athena widgets, so
in terms of code, you would see X, XAW, and XA3D calls. These days, one
might instead use a GTK4 or QT mail reader, and those toolkits would only
"promise support" for code that made gtk or qt calls; the programmer isn't
exactly prohibited from using X11 calls, but doing so is not recommended
nor supported, and if the programmer runs into trouble, they are very
likely to be told "you're breaking the abstraction barrier we set; remove
it and then come back with your problem". (This is more or less exactly the
situation that created these lines in xterm.c:)

#ifdef USE_GTK
>       /* A long-standing GTK bug prevents proper disconnect handling
> <https://gitlab.gnome.org/GNOME/gtk/issues/221>.  Once,
> the resulting Glib error message loop filled a user's disk.
> To avoid this, kill Emacs unconditionally on disconnect.  */
>       shut_down_emacs (0, Qnil);
>       fprintf (stderr, "%s\n\
> When compiled with GTK, Emacs cannot recover from X disconnects.\n\
> This is a GTK bug: https://gitlab.gnome.org/GNOME/gtk/issues/221\n\
> For details, see etc/PROBLEMS.\n",
>       error_msg);
>       emacs_abort ();
> #endif /* USE_GTK */


Over time, reaching "past" the abstraction barrier has become harder and
harder as the toolkits have become more "all encompassing", to the point
where it's more semantically accurate to talk about an app being a GTK or
QT (or W32 or mac) app than, say, an X app that uses QT. I don't know if
this owes more to "easy platform consistency" or to "just do everything via
javascript inside a browser", but the end result is pretty much the same.
It probably doesn't help that Wayland is basically a decade old and still
doesn't have a stable standard specification (or really a draft spec beyond
"whatever is currently in the release code").

Hope that helps,
~Chad

[-- Attachment #2: Type: text/html, Size: 3496 bytes --]

Re: Touchscreen support

[Po Lu]

I agree with what you said in the general sense, but you got some
details wrong:

> In general, the currently-used toolkits have raised the floor of the
> "stamp of approval" abstraction layer. As a (somewhat contrived)
> example, once upon a time one might have used xmh to read email. It
> was an X application that used the Athena Widgets and also could use
> the 3D Athena widgets, so in terms of code, you would see X, XAW, and
> XA3D calls.

Inaccurate comparison: the X tookit intrinstics are not intended to
abstract away X, so you're always supposed to (and even encouraged to)
use X to do things.

> These days, one might instead use a GTK4 or QT mail reader, and those
> toolkits would only "promise support" for code that made gtk or qt
> calls; the programmer isn't exactly prohibited from using X11 calls,
> but doing so is not recommended nor supported, and if the programmer
> runs into trouble, they are very likely to be told "you're breaking
> the abstraction barrier we set; remove it and then come back with your
> problem". (This is more or less exactly the situation that created
> these lines in xterm.c:)

>  #ifdef USE_GTK
>        /* A long-standing GTK bug prevents proper disconnect handling
>  <https://gitlab.gnome.org/GNOME/gtk/issues/221>.  Once,
>  the resulting Glib error message loop filled a user's disk.
>  To avoid this, kill Emacs unconditionally on disconnect.  */
>        shut_down_emacs (0, Qnil);
>        fprintf (stderr, "%s\n\
>  When compiled with GTK, Emacs cannot recover from X disconnects.\n\
>  This is a GTK bug: https://gitlab.gnome.org/GNOME/gtk/issues/221\n\
>  For details, see etc/PROBLEMS.\n",
>        error_msg);
>        emacs_abort ();
>  #endif /* USE_GTK */

That is a rather bad example as well: GTK can't handle a display
closing, even if you use the GTK-provided abstraction layer
(GdkDisplayManager, etc) to open and close multiple displays.  It's just
a bug in their code that they have no interest in fixing, since multiple
display support is of no interest to every program but Emacs.

x_wm_set_size_hint in gtkutil.c would be a better example, IMO.

Thanks.

Re: Touchscreen support

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > These days, one
  > might instead use a GTK4 or QT mail reader, and those toolkits would only
  > "promise support" for code that made gtk or qt calls; the programmer isn't
  > exactly prohibited from using X11 calls, but doing so is not recommended
  > nor supported, and if the programmer runs into trouble, they are very
  > likely to be told "you're breaking the abstraction barrier we set; remove
  > it and then come back with your problem".

What happens if one asks them, "How do I solve the problem XYZ without
doing an X11 call?"

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Touchscreen support

[Po Lu]

Richard Stallman <rms@gnu.org> writes:

>   > These days, one
>   > might instead use a GTK4 or QT mail reader, and those toolkits would only
>   > "promise support" for code that made gtk or qt calls; the programmer isn't
>   > exactly prohibited from using X11 calls, but doing so is not recommended
>   > nor supported, and if the programmer runs into trouble, they are very
>   > likely to be told "you're breaking the abstraction barrier we set; remove
>   > it and then come back with your problem".

> What happens if one asks them, "How do I solve the problem XYZ without
> doing an X11 call?"

I don't know how it is with Qt, but the response from the GTK developers
these days is typically "stop doing whatever you did which required you
to solve problem XYZ".

Where "whatever you did" can be reparenting a window onto another
window, moving a window to an absolute coordinate on-screen, warping the
mouse pointer, dynamically updating the menu bar, or closing a display
connection, all of which are today discouraged by the GTK developers.

Thanks.

Re: Touchscreen support

[Lars Ingebrigtsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> Other cross-platform programs I examined such as Mozilla Firefox also
>> implement gesture recognition from scratch.
>
> Hmm... that sucks.  Hopefully this will improve in the longer term,

It's unlikely -- they've instead been removing what little gesture
support that originally existed, leaving it all to the applications to
figure out.  (I had some rudimentary gesture support going in Emacs
about half a decade ago, and it stopped working as they removed those
bits from...  was it Gnome or was it X?  I forget.)

And, yes, it sucks.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Touchscreen support

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Date: Sat, 18 Dec 2021 08:48:12 +0100
> Cc: Po Lu <luangruo@yahoo.com>,
>  "Po Lu via Emacs development discussions." <emacs-devel@gnu.org>
> 
> Stefan Monnier <monnier@iro.umontreal.ca> writes:
> 
> >> Other cross-platform programs I examined such as Mozilla Firefox also
> >> implement gesture recognition from scratch.
> >
> > Hmm... that sucks.  Hopefully this will improve in the longer term,
> 
> It's unlikely -- they've instead been removing what little gesture
> support that originally existed, leaving it all to the applications to
> figure out.  (I had some rudimentary gesture support going in Emacs
> about half a decade ago, and it stopped working as they removed those
> bits from...  was it Gnome or was it X?  I forget.)
> 
> And, yes, it sucks.

FWIW, MS-Windows seems to provide higher-level abstraction for touch
events, see

  https://docs.microsoft.com/en-us/windows/win32/wintouch/windows-touch-gestures-overview

Is the plan to have xterm.c/xfns.c to generate similar higher-level
events based on just the 3 basic low-level events we now have in
xterm.c?

Re: Touchscreen support

[Po Lu]

Eli Zaretskii <eliz@gnu.org> writes:

> FWIW, MS-Windows seems to provide higher-level abstraction for touch
> events, see
>
>   https://docs.microsoft.com/en-us/windows/win32/wintouch/windows-touch-gestures-overview
>
> Is the plan to have xterm.c/xfns.c to generate similar higher-level
> events based on just the 3 basic low-level events we now have in
> xterm.c?

The plan is to have that infrastructure in Lisp, which should be much
more portable (for example, judging by the MS documentation you sent,
there are at least 3 gestures not supported by GTK: left/right selection
drag, flicks, and I think rotate as well.)  MS-Windows does support the
low level reporting of touch events like we have in xterm.c, correct?

Thanks.

Re: Touchscreen support

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: Lars Ingebrigtsen <larsi@gnus.org>,  monnier@iro.umontreal.ca,
>   emacs-devel@gnu.org
> Date: Sun, 19 Dec 2021 08:24:58 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > FWIW, MS-Windows seems to provide higher-level abstraction for touch
> > events, see
> >
> >   https://docs.microsoft.com/en-us/windows/win32/wintouch/windows-touch-gestures-overview
> >
> > Is the plan to have xterm.c/xfns.c to generate similar higher-level
> > events based on just the 3 basic low-level events we now have in
> > xterm.c?
> 
> The plan is to have that infrastructure in Lisp, which should be much
> more portable (for example, judging by the MS documentation you sent,
> there are at least 3 gestures not supported by GTK: left/right selection
> drag, flicks, and I think rotate as well.)

If you do it in Lisp, it will be hard to use on other platforms, where
the higher-level abstractions are reported on the level of the
window-system events.

> MS-Windows does support the low level reporting of touch events like
> we have in xterm.c, correct?

I don't think so, although I'm far from being an expert.  I see it
provides the gesture-begin and gesture-end events, but not
gesture-update; instead, the "cooked" gestures are provided which are
already constructed from lower-level events.  The MS documentation
clearly says:

  Note Most applications should ignore the GID_BEGIN and GID_END
  messages and pass them to DefWindowProc. These messages are used by
  the default gesture handler. Application behavior is undefined when
  the GID_BEGIN and GID_END messages are consumed by a third-party
  application.

So they advise against using the low-level events.

In any case, it makes no sense to me to "decompose" events that are
already composed into meaningful gestures.  We should be able to use
those higher-level abstractions when they are provided.

Re: Touchscreen support

[Po Lu]

Eli Zaretskii <eliz@gnu.org> writes:

> If you do it in Lisp, it will be hard to use on other platforms, where
> the higher-level abstractions are reported on the level of the
> window-system events.

[...]

> I don't think so, although I'm far from being an expert.  I see it
> provides the gesture-begin and gesture-end events, but not
> gesture-update; instead, the "cooked" gestures are provided which are
> already constructed from lower-level events.  The MS documentation
> clearly says:
>
>   Note Most applications should ignore the GID_BEGIN and GID_END
>   messages and pass them to DefWindowProc. These messages are used by
>   the default gesture handler. Application behavior is undefined when
>   the GID_BEGIN and GID_END messages are consumed by a third-party
>   application.
>
> So they advise against using the low-level events.

> In any case, it makes no sense to me to "decompose" events that are
> already composed into meaningful gestures.  We should be able to use
> those higher-level abstractions when they are provided.

The difference is that those higher-level abstractions vary too much for
us to abstract around usefully.  (The GTK one is also very painful, for
unrelated reasons.)  I think that's why Mozilla Firefox and other
cross-platform programs that have to handle touch events rely on their
own machinery for this stuff opposed to the platform-provided one.

We would have to limit ourselves to only handling tap events and only
sending them when the finger is lifted, if we were to rely on the lowest
common denominator of all those different abstractions.

Re: Touchscreen support

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: larsi@gnus.org,  monnier@iro.umontreal.ca,  emacs-devel@gnu.org
> Date: Sun, 19 Dec 2021 17:24:14 +0800
> 
> > In any case, it makes no sense to me to "decompose" events that are
> > already composed into meaningful gestures.  We should be able to use
> > those higher-level abstractions when they are provided.
> 
> The difference is that those higher-level abstractions vary too much for
> us to abstract around usefully.  (The GTK one is also very painful, for
> unrelated reasons.)  I think that's why Mozilla Firefox and other
> cross-platform programs that have to handle touch events rely on their
> own machinery for this stuff opposed to the platform-provided one.

We'd need to try to find some scheme that provides equivalence between
the different abstractions.  I don't understand how could it be too
hard, since the gestures are very similar, if not identical.

Which higher-level gestures did you intend to support?  The MS-Windows
classification has this list:

  . zoom
  . pan
  . rotate
  . two-finger tap
  . press and tap

Which other gestures did you have in mind?

> We would have to limit ourselves to only handling tap events and only
> sending them when the finger is lifted, if we were to rely on the lowest
> common denominator of all those different abstractions.

I see no reason to limit ourselves to the lowest common denominator.

Re: Touchscreen support

[Po Lu]

Eli Zaretskii <eliz@gnu.org> writes:

> We'd need to try to find some scheme that provides equivalence between
> the different abstractions.  I don't understand how could it be too
> hard, since the gestures are very similar, if not identical.
>
> Which higher-level gestures did you intend to support?  The MS-Windows
> classification has this list:
>
>   . zoom
>   . pan

What about being able to change the selection by dragging (not holding
and then dragging) left and right?  That has no equivalent on GTK.

I don't think GTK has a public API for the pan gesture either, even
though it's clearly supported by the built-in widgets.

However, I think your classification is an otherwise reasonable list of
gestures to support; how about we use the code in Lisp for platforms
where low level events are reported (such as PGTK and X-Windows), while
directly sending the high-level events on platforms where those aren't
available?

Thanks.

Re: Touchscreen support

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: larsi@gnus.org,  monnier@iro.umontreal.ca,  emacs-devel@gnu.org
> Date: Sun, 19 Dec 2021 17:38:30 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Which higher-level gestures did you intend to support?  The MS-Windows
> > classification has this list:
> >
> >   . zoom
> >   . pan
> 
> What about being able to change the selection by dragging (not holding
> and then dragging) left and right?  That has no equivalent on GTK.

How is that different from "pan"?

> I don't think GTK has a public API for the pan gesture either, even
> though it's clearly supported by the built-in widgets.

So maybe we will have to construct that from other gestures on those
platforms.

> However, I think your classification is an otherwise reasonable list of
> gestures to support; how about we use the code in Lisp for platforms
> where low level events are reported (such as PGTK and X-Windows), while
> directly sending the high-level events on platforms where those aren't
> available?

I'm not opposed to having parts of this in Lisp, but I do want to see
the events placed in the normal Emacs input queue, which probably
means that Lisp will be called from C.  Not sure if such design will
make sense in practice, though.  But OTOH, having events coming in
from another source, not through the input queue read by
read_socket_hook, is a complication I'd very much prefer to avoid (if
it's even feasible).

Re: Touchscreen support

[Po Lu]

Eli Zaretskii <eliz@gnu.org> writes:

>> I don't think GTK has a public API for the pan gesture either, even
>> though it's clearly supported by the built-in widgets.

> So maybe we will have to construct that from other gestures on those
> platforms.

I will try that, but I don't think it will work out very well.

> I'm not opposed to having parts of this in Lisp, but I do want to see
> the events placed in the normal Emacs input queue, which probably
> means that Lisp will be called from C.  Not sure if such design will
> make sense in practice, though.  But OTOH, having events coming in
> from another source, not through the input queue read by
> read_socket_hook, is a complication I'd very much prefer to avoid (if
> it's even feasible).

We could have an API that allows Lisp to push some kinds of input events
to that queue, which is then read by the various read_socket_hooks.

Re: Touchscreen support

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: larsi@gnus.org,  monnier@iro.umontreal.ca,  emacs-devel@gnu.org
> Date: Sun, 19 Dec 2021 19:42:31 +0800
> 
> > I'm not opposed to having parts of this in Lisp, but I do want to see
> > the events placed in the normal Emacs input queue, which probably
> > means that Lisp will be called from C.  Not sure if such design will
> > make sense in practice, though.  But OTOH, having events coming in
> > from another source, not through the input queue read by
> > read_socket_hook, is a complication I'd very much prefer to avoid (if
> > it's even feasible).
> 
> We could have an API that allows Lisp to push some kinds of input events
> to that queue, which is then read by the various read_socket_hooks.

How would the low-level events, from which you want to construct
higher-level events, get to Lisp in the first place?

Re: Touchscreen support

[Po Lu]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Po Lu <luangruo@yahoo.com>
>> Cc: larsi@gnus.org,  monnier@iro.umontreal.ca,  emacs-devel@gnu.org
>> Date: Sun, 19 Dec 2021 19:42:31 +0800
>> 
>> > I'm not opposed to having parts of this in Lisp, but I do want to see
>> > the events placed in the normal Emacs input queue, which probably
>> > means that Lisp will be called from C.  Not sure if such design will
>> > make sense in practice, though.  But OTOH, having events coming in
>> > from another source, not through the input queue read by
>> > read_socket_hook, is a complication I'd very much prefer to avoid (if
>> > it's even feasible).
>> 
>> We could have an API that allows Lisp to push some kinds of input events
>> to that queue, which is then read by the various read_socket_hooks.

> How would the low-level events, from which you want to construct
> higher-level events, get to Lisp in the first place?

Through the usual event system (though probably as special events in
special-event-map).

Re: Touchscreen support

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: larsi@gnus.org,  monnier@iro.umontreal.ca,  emacs-devel@gnu.org
> Date: Mon, 20 Dec 2021 08:54:17 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> From: Po Lu <luangruo@yahoo.com>
> >> Cc: larsi@gnus.org,  monnier@iro.umontreal.ca,  emacs-devel@gnu.org
> >> Date: Sun, 19 Dec 2021 19:42:31 +0800
> >> 
> >> > I'm not opposed to having parts of this in Lisp, but I do want to see
> >> > the events placed in the normal Emacs input queue, which probably
> >> > means that Lisp will be called from C.  Not sure if such design will
> >> > make sense in practice, though.  But OTOH, having events coming in
> >> > from another source, not through the input queue read by
> >> > read_socket_hook, is a complication I'd very much prefer to avoid (if
> >> > it's even feasible).
> >> 
> >> We could have an API that allows Lisp to push some kinds of input events
> >> to that queue, which is then read by the various read_socket_hooks.
> 
> > How would the low-level events, from which you want to construct
> > higher-level events, get to Lisp in the first place?
> 
> Through the usual event system (though probably as special events in
> special-event-map).

So the events from X will be delivered via read_socket_hook, then the
code which reads these events will call Lisp, and then it will turn
around and feed the synthetic events it produces back into the input
queue?  That's exactly what I prefer to avoid, since there's no way in
the world you will be able to pretend that the synthetic events were
delivered in the same place as the real ones: some additional events
could have arrived meanwhile.

Re: Touchscreen support

[Po Lu]

Eli Zaretskii <eliz@gnu.org> writes:

> So the events from X will be delivered via read_socket_hook, then the
> code which reads these events will call Lisp, and then it will turn
> around and feed the synthetic events it produces back into the input
> queue?  That's exactly what I prefer to avoid, since there's no way in
> the world you will be able to pretend that the synthetic events were
> delivered in the same place as the real ones: some additional events
> could have arrived meanwhile.

Hmm, is there some other mechanism we can use?  Or perhaps we can design
a new mechanism for Lisp code to translate sequences of lower-level
events into high level ones?  (Though I have no idea how that work work.)

Re: Touchscreen support

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: larsi@gnus.org,  monnier@iro.umontreal.ca,  emacs-devel@gnu.org
> Date: Tue, 21 Dec 2021 09:08:22 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > So the events from X will be delivered via read_socket_hook, then the
> > code which reads these events will call Lisp, and then it will turn
> > around and feed the synthetic events it produces back into the input
> > queue?  That's exactly what I prefer to avoid, since there's no way in
> > the world you will be able to pretend that the synthetic events were
> > delivered in the same place as the real ones: some additional events
> > could have arrived meanwhile.
> 
> Hmm, is there some other mechanism we can use?  Or perhaps we can design
> a new mechanism for Lisp code to translate sequences of lower-level
> events into high level ones?

The code which processes low-level events in keyboard.c could call
Lisp, which would return higher-level events, and then you
fall-through to the handling of higher-level events back in
keyboard.c.  Or something to that effect.

But I think we first need an idea about what kind of even translation
is needed, or maybe you already know and can present it?

Re: Touchscreen support

[Po Lu]

Eli Zaretskii <eliz@gnu.org> writes:

> But I think we first need an idea about what kind of even translation
> is needed, or maybe you already know and can present it?

No, I don't know of the specifics yet, but I do plan to work on it soon.

Thanks.

Re: Touchscreen support

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: larsi@gnus.org,  monnier@iro.umontreal.ca,  emacs-devel@gnu.org
> Date: Tue, 21 Dec 2021 20:21:39 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > But I think we first need an idea about what kind of even translation
> > is needed, or maybe you already know and can present it?
> 
> No, I don't know of the specifics yet, but I do plan to work on it soon.

OK, so let's talk when you have enough information to describe the
event translation, then we could discuss the possible implementation
alternatives.

RE: [External] : Re: Touchscreen support

[Drew Adams]

Please excuse the noise, but I wonder if
there's any relation between the touchscreen
support being looked into and the existing
mouse-gestures support in standard library 
`strokes.el'.

I know you're not talking about a mouse, and
I guess you're talking about lower-level
support of touch gestures.  Just wondering
if anything useful could be leveraged from
what already exists in `strokes.el'.

Re: Touchscreen support

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > It's unlikely -- they've instead been removing what little gesture
  > support that originally existed, leaving it all to the applications to
  > figure out.  (I had some rudimentary gesture support going in Emacs
  > about half a decade ago, and it stopped working as they removed those
  > bits from...  was it Gnome or was it X?  I forget.)

Is there a library for this that we should consider using in Emacs?

We should allow a sufficient time to consider the question -- at least
a month.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)