bug#71472: [PATCH] Add pty support by using ConPTY on Windows

bug#71472: [PATCH] Add pty support by using ConPTY on Windows

[Ke Wu]

[-- Attachment #1: Type: text/plain, Size: 619 bytes --]

Tags: patch

Tags: patch

This patch adds pty support by using ConPTY on Windows. The conhost.exe
runs in pty mode and provides pty interface. The package term is also
patched to make it usable on Windows.



In GNU Emacs 30.0.50 (build 4, x86_64-w64-mingw32) of 2024-06-10 built
 on WIN1729
Repository revision: ed122417b98d711bacf5ed24778886bf21d86956
Repository branch: master
Windowing system distributor 'Microsoft Corp.', version 10.0.22631
System Description: Microsoft Windows 10 Pro (v10.0.2009.22631.3672)

Configured using:
 'configure --prefix=/c/Users/oracl/Documents/Programs/emacs-dist
 --without-dbus'


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Add-pty-support-by-using-ConPTY-on-Windows.patch --]
[-- Type: text/patch, Size: 12105 bytes --]

From 99bdacc3d6b9e6ea1ddcb0ae31b44afffe0ff697 Mon Sep 17 00:00:00 2001
From: Ke Wu <ellpih@zohomail.jp>
Date: Mon, 10 Jun 2024 18:09:45 +0800
Subject: [PATCH] Add pty support by using ConPTY on Windows

* src/w32proc.c (w32-make-console-process): Add a function to create
process with pty interface using conhost.exe in pty mode.
* src/w32.c (make_console_with_pipe): Make a process using
conhost.exe. Attach a pipe to conhost.exe for resize. This function is
used by 'w32-make-console-process'.
* src/w32.h (make_console_with_pipe): Make above function definition
an extern function.
* src/sysdep.c (set_window_size): Implement window resize for
conhost.exe.
* src/process.c (set-process-window-size): Implement window resize.
* src/process.c (deactivate_process): Close pipe fd when process
exits. The pipe is attached to conhost.exe in 'make_console_with_pipe'.
* lisp/term.el (term-exec-1): Use 'w32-make-console-process' to create a
process with pty interface.
* lisp/term.el (term-coding-system): Set coding system to UTF-8 for
newly created buffer with terminal process attached since conhost.exe
assumes UTF-8 encoding in ConPTY mode.
---
 lisp/term.el  | 39 +++++++++++++++--------
 src/process.c | 28 +++++++++++++++++
 src/sysdep.c  | 12 +++++++
 src/w32.c     | 87 +++++++++++++++++++++++++++++++++++++++++++++++++++
 src/w32.h     |  2 ++
 src/w32proc.c | 35 +++++++++++++++++++++
 6 files changed, 190 insertions(+), 13 deletions(-)

diff --git a/lisp/term.el b/lisp/term.el
index c15f6cf2e9f..24509037805 100644
--- a/lisp/term.el
+++ b/lisp/term.el
@@ -1746,18 +1746,24 @@ term-exec-1
     (when (term--bash-needs-EMACSp)
       (push (format "EMACS=%s (term:%s)" emacs-version term-protocol-version)
             process-environment))
-    (apply #'start-process name buffer
-           ;; On Android, /bin doesn't exist, and the default shell is
-           ;; found as /system/bin/sh.
-	   (if (eq system-type 'android)
-               "/system/bin/sh"
-             "/bin/sh")
-           "-c"
-	   (format "stty -nl echo rows %d columns %d sane 2>%s;\
+    (if (eq system-type 'windows-nt)
+        ;; Use `w32-make-console-process' to make use of the Windows ConPTY.
+        (apply #'w32-make-console-process
+               (append (list :name name :buffer buffer)
+                       (list :command (cons command switches))
+                       (list :width term-width :height term-height)))
+      (apply #'start-process name buffer
+             ;; On Android, /bin doesn't exist, and the default shell is
+             ;; found as /system/bin/sh.
+	     (if (eq system-type 'android)
+                 "/system/bin/sh"
+               "/bin/sh")
+             "-c"
+	     (format "stty -nl echo rows %d columns %d sane 2>%s;\
 if [ $1 = .. ]; then shift; fi; exec \"$@\""
-		   term-height term-width null-device)
-	   ".."
-	   command switches)))
+		     term-height term-width null-device)
+	     ".."
+	     command switches))))
 
 \f
 ;;; Input history processing in a buffer
@@ -3007,6 +3013,13 @@ term-control-seq-regexp
 (defconst term-control-seq-prefix-regexp
   "[\032\e]")
 
+(defconst term-coding-system
+  (if (eq system-type 'windows-nt)
+      ;; The conhost.exe assumes UTF-8 for ConPTY.
+      'utf-8-dos
+    locale-coding-system)
+  "Coding system for terminal in term.el.")
+
 (defun term-emulate-terminal (proc str)
   (when (buffer-live-p (process-buffer proc))
     (with-current-buffer (process-buffer proc)
@@ -3074,7 +3087,7 @@ term-emulate-terminal
                   (setq decoded-substring
                         (decode-coding-string
                          (substring str i funny)
-                         locale-coding-system t))
+                         term-coding-system t))
                   ;; Check for multibyte characters that ends
                   ;; before end of string, and save it for
                   ;; next time.
@@ -3173,7 +3186,7 @@ term-emulate-terminal
                                         (- ctl-end
                                            (if (eq (aref str (- ctl-end 2)) ?\r)
                                                2 1)))
-                             locale-coding-system t)))
+                             term-coding-system t)))
                   (?\e
                    (pcase (aref str (1+ i))
                      (?\[
diff --git a/src/process.c b/src/process.c
index 9670be64279..b3e56bc8d9c 100644
--- a/src/process.c
+++ b/src/process.c
@@ -1497,12 +1497,32 @@ DEFUN ("set-process-window-size", Fset_process_window_size,
   unsigned short h = check_uinteger_max (height, USHRT_MAX);
   unsigned short w = check_uinteger_max (width, USHRT_MAX);
 
+#ifdef WINDOWSNT
+
+  Lisp_Object sigfd_obj;
+  int sigfd;
+  if (NETCONN_P (process))
+    return Qnil;
+
+  sigfd_obj = plist_get (XPROCESS (process) -> plist, QCfile_handler);
+  if (!FIXNUMP (sigfd_obj))
+    return Qnil;
+  sigfd = XFIXNUM (sigfd_obj);
+  if(sigfd < 0 || set_window_size (sigfd, h, w) < 0)
+    return Qnil;
+  else
+    return Qt;
+
+#else
+
   if (NETCONN_P (process)
       || XPROCESS (process)->infd < 0
       || set_window_size (XPROCESS (process)->infd, h, w) < 0)
     return Qnil;
   else
     return Qt;
+
+#endif
 }
 
 DEFUN ("set-process-inherit-coding-system-flag",
@@ -4783,6 +4803,14 @@ deactivate_process (Lisp_Object proc)
   /* Delete GnuTLS structures in PROC, if any.  */
   emacs_gnutls_deinit (proc);
 #endif /* HAVE_GNUTLS */
+#ifdef WINDOWSNT
+  /* Close write side of the pipe in console process of Windows */
+  Lisp_Object infd = plist_get(p -> plist, QCfile_handler);
+  if (!NILP (infd) && FIXNUMP(infd)) {
+    i = XFIXNUM(infd);
+    emacs_close(i);
+  }
+#endif /* WINDOWSNT */
 
   if (p->read_output_delay > 0)
     {
diff --git a/src/sysdep.c b/src/sysdep.c
index 07237885cb9..144ea4b413b 100644
--- a/src/sysdep.c
+++ b/src/sysdep.c
@@ -1481,8 +1481,20 @@ set_window_size (int fd, int height, int width)
   size.ts_cols = width;
 
   return ioctl (fd, TIOCGSIZE, &size);
+
+#else
+#ifdef WINDOWSNT
+
+  /* Windows console process */
+  unsigned short signal_packet[3];
+  signal_packet[0] = 8u;
+  signal_packet[1] = width;
+  signal_packet[2] = height;
+  return write(fd, signal_packet, sizeof(signal_packet));
+
 #else
   return -1;
+#endif /* not Windows NT */
 #endif /* not SunOS-style */
 #endif /* not BSD-style */
 }
diff --git a/src/w32.c b/src/w32.c
index 6d0b178e978..b5708458a8c 100644
--- a/src/w32.c
+++ b/src/w32.c
@@ -11166,6 +11166,93 @@ register_aux_fd (int infd)
   fd_info[ infd ].flags |= FILE_DONT_CLOSE;
 }
 
+/* Start a console process by wraping the command in conhost.exe.  The process is
+   started by calling make-process.  The variable fds stores the infd and outfd of
+   the pipe serving as the signal pipe of conhost.exe. */
+Lisp_Object
+make_console_with_pipe (ptrdiff_t nargs, Lisp_Object * args, const int * fds)
+{
+
+  Lisp_Object command, contact;
+  Lisp_Object command_new, contact_new;
+  Lisp_Object width, height;
+  Lisp_Object process;
+  unsigned long pipe_outhnd;
+
+  HANDLE parent, newoutfd;
+
+  parent = GetCurrentProcess ();
+
+  /* Make inheritable copies of the fds[0]. */
+  if (!DuplicateHandle (parent,
+                        (HANDLE) _get_osfhandle (fds[0]),
+                        parent,
+                        &newoutfd,
+                        0,
+                        TRUE,
+                        DUPLICATE_SAME_ACCESS))
+    report_file_error ("Duplicating input handle for child", Qnil);
+
+  emacs_close( fds[0] );
+  pipe_outhnd = (unsigned long) newoutfd;
+
+  /* Compose new command based on given parameters. */
+  contact = Flist (nargs, args);
+  command = plist_get (contact, QCcommand);
+  if (NILP (command))
+    return Qnil;
+  if (!CONSP (command))
+    command = list1 (command);
+  width = plist_get (contact, QCwidth);
+  height = plist_get (contact, QCheight);
+
+  command_new = CALLN (Flist,
+                       build_string ("conhost.exe"),
+                       build_string ("--headless"),
+                       build_string ("--feature"),
+                       build_string ("pty"));
+  if (!NILP (width))
+    command_new = CALLN (Fappend,
+                         command_new,
+                         CALLN (Flist,
+                                build_string ("--width"),
+                                CALLN (Fformat, build_string ("%d"), width)));
+
+  if (!NILP (height))
+    command_new = CALLN (Fappend,
+                         command_new,
+                         CALLN (Flist,
+                                build_string ("--height"),
+                                CALLN (Fformat, build_string ("%d"), height)));
+
+  command_new = CALLN (Fappend,
+                       command_new,
+                       CALLN (Flist, build_string ("--signal"),
+                              CALLN (Fformat,
+                                     build_string ("0x%x"),
+                                     make_uint (pipe_outhnd))));
+
+  command_new = CALLN (Fappend,
+                       command_new,
+                       command);
+
+  contact_new = plist_put (contact, QCcommand, command_new);
+  process = CALLN (Fapply, Qmake_process, contact_new);
+
+  CloseHandle (newoutfd);
+  if (NILP (process))
+    emacs_close( fds[1] );
+  else {
+    fd_info[ fds[1] ].cp = fd_info[XPROCESS (process) -> infd].cp;
+    /* Store the signal pipe's out fd in process plist.  Not sure if
+       QCfile_handler is a proper key. */
+    XPROCESS (process) -> plist =
+      plist_put (XPROCESS (process) -> plist,
+                 QCfile_handler, make_uint(fds[1]));
+  }
+  return process;
+}
+
 #ifdef HAVE_GNUTLS
 
 ssize_t
diff --git a/src/w32.h b/src/w32.h
index cf470ae9901..7393c5299f7 100644
--- a/src/w32.h
+++ b/src/w32.h
@@ -230,6 +230,8 @@ #define FILE_DONT_CLOSE         0x1000
 extern int lchmod (char const *, mode_t);
 extern bool symlinks_supported (const char *);
 
+/* Create console process with signal pipe */
+extern Lisp_Object make_console_with_pipe (ptrdiff_t, Lisp_Object *, const int *);
 
 /* Return total and free memory info.  */
 extern int w32_memory_info (unsigned long long *, unsigned long long *,
diff --git a/src/w32proc.c b/src/w32proc.c
index 55ead13647b..efb32c3ed7e 100644
--- a/src/w32proc.c
+++ b/src/w32proc.c
@@ -3726,6 +3726,39 @@ DEFUN ("w32-set-keyboard-layout", Fw32_set_keyboard_layout,
   return Fw32_get_keyboard_layout ();
 }
 
+DEFUN ("w32-make-console-process", Fw32_make_console_process,
+       Sw32_make_console_process, 0, MANY, 0,
+       doc: /* Start a process wrapped in conhost.exe.
+
+This is similar to `make-process', which following extra arguments:
+
+:width WIDTH -- WIDTH is the initial width of the conhost.exe process.
+
+:width HEIGHT -- HEIGHT is the initial height of the conhost.exe process.
+
+The conhost.exe runs in pty mode, which acts like a pty devices in *NIX. With
+following differences:
+
+1. The text going in and out from the stdin and the stdout of conhost.exe are
+always encoded in UTF-8.
+
+2. The conhost.exe requires an extra pipe to send signals which causes the
+console resize.  This is implemented in this function.  */)
+  (ptrdiff_t nargs, Lisp_Object *args)
+{
+
+  Lisp_Object process;
+  int fds[2];
+
+  /* Create signal pipe for this process */
+  if (emacs_pipe (fds) < 0)
+    report_file_error ("Creating signal pipe for console process", Qnil);
+  fd_info[ fds[1] ].hnd = (HANDLE) _get_osfhandle (fds[1]);
+
+  process = make_console_with_pipe(nargs, args, fds);
+  return process;
+}
+
 /* Two variables to interface between get_lcid and the EnumLocales
    callback function below.  */
 #ifndef LOCALE_NAME_MAX_LENGTH
@@ -3949,6 +3982,8 @@ syms_of_ntproc (void)
   defsubr (&Sw32_get_keyboard_layout);
   defsubr (&Sw32_set_keyboard_layout);
 
+  defsubr (&Sw32_make_console_process);
+
   DEFVAR_LISP ("w32-quote-process-args", Vw32_quote_process_args,
 	       doc: /* Non-nil enables quoting of process arguments to ensure correct parsing.
 Because Windows does not directly pass argv arrays to child processes,
-- 
2.45.1

bug#71472: [PATCH] Add pty support by using ConPTY on Windows

[Eli Zaretskii]

> From: Ke Wu <ellpih@zohomail.jp>
> Date: Mon, 10 Jun 2024 18:26:46 +0800
> 
> This patch adds pty support by using ConPTY on Windows. The conhost.exe
> runs in pty mode and provides pty interface. The package term is also
> patched to make it usable on Windows.

Thanks.  It's a pity you didn't discuss this before working on the
code, because ConPTY is not very useful on MS-Windows due to the
limitations of its encoding.  See this issue I opened 3 years ago, and
which they meanwhile closed (I guess they don't plan on fixing this
anytime soon).  See

  https://github.com/microsoft/terminal/issues/9174

If we must use UTF-8 as the only encoding to talk to sub-processes via
ConPTY, that makes the number of applications that can be used this
way very small, since most programs we are used to run as
subprocesses, in particularly ports of GNU software like GCC, GDB,
Grep, Find, and many others, cannot reliably talk to Emacs in UTF-8
encoding on MS-Windows.  And without that, what would we use this
feature for?

So unfortunately, I don't think we should install these patches.

bug#71472: [PATCH] Add pty support by using ConPTY on Windows

[Eli Zaretskii]

[Please use Reply All to reply, to keep the bug tracker CC'ed.]

> Date: Tue, 11 Jun 2024 12:34:48 +0900
> From: Ke Wu <ellpih@zohomail.jp>
> 
>  > If we must use UTF-8 as the only encoding to talk to sub-processes via 
>  > ConPTY, that makes the number of applications that can be used this 
>  > way very small, since most programs we are used to run as 
>  > subprocesses, in particularly ports of GNU software like GCC, GDB, 
>  > Grep, Find, and many others, cannot reliably talk to Emacs in UTF-8 
>  > encoding on MS-Windows.
> 
> The statement is not so accurate. On Emacs side, UTF-8 is assumed due
> to the limitation of ConPTY (it would communicate with the console only in
> UTF-8). However, on the subprocesses side, ConPTY would respect its
> codepage and translate it into UTF-8 when sending to the console. So
> we can make these subprocesses run in the codepage other than
> 65001(UTF-8).

This is inaccurate: ConPTY always assumes the process running on the
other side of the connection uses the system codepage.  If the
subprocess expects some other encoding, ConPTY will not know that, and
Emacs has no way of telling ConPTY to use a different encoding.  This
is the essence of the issue I filed with them, and they basically told
me that what ConPTY does is "by design".

This is not an academic issue: some very important programs we invoke
from Emacs need us to talk to them in encoding different from the
system codepage.  A notable example is Git, which wants UTF-8 (it can
support other encodings, but that is not recommended, and Emacs
doesn't really support that well on Windows).

> I am not very familiar with these GNU software ports :(
> Please let me know if there will be problems with ConPTY translating from
> UTF-8 to other codepages.

See above.  There's no way for Emacs to set that up, except when the
"other codepage" is the system codepage.

>  >  https://github.com/microsoft/terminal/issues/9174 
> 
> I think a possible solution to this issue is to use a wrapper program to
> set the codepage for the applications that do not call `SetConsoleOutputCP`.
> As a proof of concept, the following code snippet uses cmdproxy.exe to
> change the codepage to 1255. Please replace the cmdproxy.exe path in the
> snippet.
> 
> (progn
>   (set-buffer
>    (apply #'make-term
>           "terminal"
>           "C:/Users/oracl/Documents/Programs/emacs-master/nt/cmdproxy.exe"
>           nil
>           '("-c" "chcp 1255 && call cmd")))
>   (term-char-mode)
>   (pop-to-buffer-same-window "*terminal*"))
> 
> The codepage can be verified by either using `chcp` in the newly created cmd process.
> Also, the following hack can be applied to make the created conhost.exe visible.
> Therefore, the codepage can be directly verified by viewing the properties of the
> conhost.exe window. 
> 
> --- a/src/w32.c
> +++ b/src/w32.c
> @@ -11208,7 +11208,7 @@ make_console_with_pipe (ptrdiff_t nargs, Lisp_Object * args, const int * fds)
> 
>    command_new = CALLN (Flist,
>                         build_string ("conhost.exe"),
> -                       build_string ("--headless"),
> +                       /* build_string ("--headless"), */
>                         build_string ("--feature"),
>                         build_string ("pty"));
>    if (!NILP (width)) 
> 
> Therefore, we can have subprocesses run in codepage other than 65001 or the OEM default
> codepage.  And as a console program, Emacs talks in UTF-8.  It may be feasible if we add a
> `:coding` to function `term`, which builds up a wrapper to change the code page before the
> real program starts.

cmdproxy is only used when invoking programs via the shell.  But Emacs
also invokes programs directly (call-process etc.), in which case
cmdproxy (or any other kind of wrapper) will be very problematic at
best, if not impossible.  See below about the complications this
causes wrt quoting of command-line arguments, for example.

Please keep in mind how Emacs arranges to use correct encoding when
invoking other programs: we have data structures
(process-coding-system-alist etc.) which define the correct encoding
by program name, and we also have variables (coding-system-for-read
etc.) that can be bound to override those defaults temporarily.  The
encoding is applied separately to the program's command-line arguments
and to the stuff we write and read to and from the process.  How can
all this work reliably with ConPTY, even if the wrapper trick could
sometimes work?  Specifically:

  . how do we control encoding of command-line arguments? most
    programs running on Windows cannot handle UTF-8 encoded command
    lines
  . what if the encoding we need doesn't have a corresponding Windows
    codepage (which means chcp will not work)?
  . how can we handle the eol-conversion part of the encoding (some
    programs _must_ be fed with Unix EOLs)?

Also please note that using a wrapper adds another layer of
interpreting command-line arguments, which might break some
complicated cases that use fancy quoting of special characters.  Any
wrapper we provide will be compiled with MinGW, so it will use the
MinGW startup code to process quoting.  But the program the wrapper
runs might not be a MinGW program, so it could use different ways of
processing quotes.  The simplest example of such a combination is
cmd.exe itself: its quoting rules are very different from what MinGW
uses.  This will definitely break some cases.  For example, Git uses
the '^' character for special purposes, and some Windows styles of
quoting interpret '^' as a quote character -- this could easily break
Emacs commands that invoke Git.

If someone can figure out how to do all this stuff with ConPTY, then
okay, we could use it.  But it is not a trivial problem, not at all.
The way ConPTY was designed is the way Windows works everywhere else:
it doesn't allow applications to communicate with raw bytestreams
without interpreting; instead, Windows _interprets_ the bytestreams as
characters encoded in the encoding it assumes for the source, and then
converts those characters to the encoding of the destination.  This
basic design principle is built into every part of Windows APIs.  For
example, a program whose 'main' function is declared as accepting
wchar_t (i.e. UTF-16) command-line arguments will magically have the
command-line arguments converted to UTF-16, even if the calling
process uses plain ASCII.  ConPTY uses the same design principles, so
it is inherently unable to pass through raw bytes without interpreting
them.  And without that, we cannot easily implement the way Emacs
expects this stuff to work, because Emacs assumes the encoding to be a
private contract between Emacs and the program it calls, with nothing
in-between interfering.

I hope I explained some of the issues with ConPTY, and why we cannot
install its support without some reasonably reliable solutions for
those problematic aspects.

Thanks.

bug#71472: [PATCH] Add pty support by using ConPTY on Windows

[Ke Wu]

Thank you for your detailed explanation.  The patch itself cannot provide
a satisfying pty support on Windows.  Its original purpose is to provide
support to term.el and other packages which setup a terminal emulator
inside Emacs.  It is not meant to be a replacement of `make-process` on
Windows, but is used as a supplementary method to patch the broken
term.el, which has long been an issue for Windows users.

Thanks.

bug#71472: [PATCH] Add pty support by using ConPTY on Windows

[Eli Zaretskii]

> Date: Tue, 11 Jun 2024 17:24:58 +0900
> From: Ke Wu <ellpih@zohomail.jp>
> Cc: "71472" <71472@debbugs.gnu.org>
> 
> Thank you for your detailed explanation.  The patch itself cannot provide
> a satisfying pty support on Windows.  Its original purpose is to provide
> support to term.el and other packages which setup a terminal emulator
> inside Emacs.  It is not meant to be a replacement of `make-process` on
> Windows, but is used as a supplementary method to patch the broken
> term.el, which has long been an issue for Windows users.

OK, that narrows the applicability and thus the extent of possible
problems.  But still some problems remain: does that work with running
in "M-x term" programs that display fancy/unusual characters?  Did you
try that with Git, for example, showing changes whose authors or text
of the diffs include Unicode characters?  Or what happens if you run
the DIR command of cmd.exe in a directory which has files whose names
aren't supported by the system codepage?  Or what happens if you run a
program whose command-line arguments include characters not supported
by the current system codepage?

IOW, we must understand the limitations of this before we decide
whether this feature is justified.  Installing features with too many
limitations will annoy the users and cause complaints and bug reports.

Thanks.