pipe

pipe

[Daiki Ueno]

[-- Attachment #1: Type: text/plain, Size: 2288 bytes --]

Related to:
https://lists.gnu.org/archive/html/emacs-devel/2014-09/msg00777.html
in which I proposed a generalization of start-process, in a similar way
to open-network-stream and make-network-process.

My motivation behind that was to avoid temporary files in epg.el, by
using file descriptors other than 0 and 1 (as you know, gpg has options
--status-fd, --command-fd, --attribute-fd, etc. for that).  In order to
do that, I thought that it would be inevitable to change the calling
convention of start-process.  However, I hesitate to do such an
intrusive change.  So, here is an alternative approach:

- Add a new process type 'pipe, which represents a bidirectional pipe,
  not associated with a child process when it is created.

- Add a new global variable process-pipe-list, which associates
  additional pipe processes with a child process, when start-process is
  called.  This is needed for setting the FD_CLOEXEC flag on the child
  ends of the pipes so they are not leaked to other processes created
  later, and to delete the pipe processes when the real process is
  deleted.

Here is a basic usage of this:

  (setq pipe1 (make-pipe-process :name "pipe1"))
  ;=> #<process pipe1>
  (process-contact pipe1)
  ;=> (17 20)
  (kill-buffer "pipe1")
  ;=>t
  (process-contact pipe1)
  ;=> (-1 -1)
  (setq pipe1 (make-pipe-process :name "pipe1"))
  ;=> #<process pipe1>
  (process-contact pipe1)
  ;=> (17 20)

  (setq pipe2 (make-pipe-process :name "pipe2"))

  (let ((process-connection-type nil)
        (process-pipe-list (list pipe1 pipe2)))
    (start-process-shell-command
     "process" (current-buffer)
     (format "\
  echo hello to stdout; \
  echo hello to pipe1 >&%d; \
  echo hello to pipe2 >&%d"
             (nth 1 (process-contact pipe1))
             (nth 1 (process-contact pipe2)))))

This delivers the output lines to the respective process buffers:
"pipe1", "pipe2", and the current buffer.

Would this kind of feature be acceptable?  I guess Stefan is not happy
with the new global variable, set with dynamic binding.

Anyway, I'm attaching a couple of patches: one is the implementation,
and thew other is the usage in epg.el.  It's already working with gpg,
while I haven't tested it with gpgsm.  Comments would be appreciated.

Regards,
-- 
Daiki Ueno

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Add-make-pipe-process.patch --]
[-- Type: text/x-patch, Size: 22634 bytes --]

From 37d86df013baa679d06d6c1ce048db74302b6d97 Mon Sep 17 00:00:00 2001
From: Daiki Ueno <ueno@gnu.org>
Date: Mon, 9 Mar 2015 09:50:32 +0900
Subject: [PATCH 1/2] Add make-pipe-process

* lisp.h (emacs_pipe2): New function declaration.
* sysdep.c (emacs_pipe2): New function.
(emacs_pipe): Define as a wrapper around emacs_pipe2.
* process.h (struct Lisp_Process): New member pipe_list.
* process.c (PIPECONN_P): New macro.
(PIPECONN1_P): New macro.
(SUBPROCESS_STDIN, WRITE_TO_SUBPROCESS, READ_FROM_SUBPROCESS,
SUBPROCESS_STDOUT, READ_FROM_EXEC_MONITOR, EXEC_MONITOR_OUTPUT): Move
the enum before the use in...
(Fprocess_contact): ...here.  Handle pipe process specially.
(Fdelete_process, Fprocess_status, Fset_process_buffer)
(Fset_process_filter, Fset_process_sentinel, Fstop_process)
(Fcontinue_process): Handle pipe process specially.
(create_process): Respect Vprocess_pipe_list and
Vprocess_standard_error.
(Fmake_pipe_process): New function.
(deactivate_process): Call Fdelete_process on associated pipe
processes.
(syms_of_process): New variables Vprocess_pipe_list and
Vprocess_standard_error, register Qpipe and Smake_pipe_process.
---
 src/lisp.h    |   1 +
 src/process.c | 344 +++++++++++++++++++++++++++++++++++++++++++++++++++-------
 src/process.h |   3 +
 src/sysdep.c  |  14 ++-
 4 files changed, 320 insertions(+), 42 deletions(-)

diff --git a/src/lisp.h b/src/lisp.h
index b730619..2e161da 100644
--- a/src/lisp.h
+++ b/src/lisp.h
@@ -4378,6 +4378,7 @@ extern void emacs_backtrace (int);
 extern _Noreturn void emacs_abort (void) NO_INLINE;
 extern int emacs_open (const char *, int, int);
 extern int emacs_pipe (int[2]);
+extern int emacs_pipe2 (int[2], int);
 extern int emacs_close (int);
 extern ptrdiff_t emacs_read (int, void *, ptrdiff_t);
 extern ptrdiff_t emacs_write (int, void const *, ptrdiff_t);
diff --git a/src/process.c b/src/process.c
index 1d935ba..4e6c0b9 100644
--- a/src/process.c
+++ b/src/process.c
@@ -189,6 +189,8 @@ process_socket (int domain, int type, int protocol)
 #define NETCONN1_P(p) (EQ (p->type, Qnetwork))
 #define SERIALCONN_P(p) (EQ (XPROCESS (p)->type, Qserial))
 #define SERIALCONN1_P(p) (EQ (p->type, Qserial))
+#define PIPECONN_P(p) (EQ (XPROCESS (p)->type, Qpipe))
+#define PIPECONN1_P(p) (EQ (p->type, Qpipe))
 
 /* Number of events of change of status of a process.  */
 static EMACS_INT process_tick;
@@ -253,6 +255,25 @@ static bool process_output_skip;
 #define process_output_delay_count 0
 #endif
 
+/* Indexes of file descriptors in open_fds.  */
+enum
+  {
+    /* The pipe from Emacs to its subprocess.  */
+    SUBPROCESS_STDIN,
+    WRITE_TO_SUBPROCESS,
+
+    /* The main pipe from the subprocess to Emacs.  */
+    READ_FROM_SUBPROCESS,
+    SUBPROCESS_STDOUT,
+
+    /* The pipe from the subprocess to Emacs that is closed when the
+       subprocess execs.  */
+    READ_FROM_EXEC_MONITOR,
+    EXEC_MONITOR_OUTPUT
+  };
+
+verify (PROCESS_OPEN_FDS == EXEC_MONITOR_OUTPUT + 1);
+
 static void create_process (Lisp_Object, char **, Lisp_Object);
 #ifdef USABLE_SIGIO
 static bool keyboard_bit_set (fd_set *);
@@ -420,6 +441,11 @@ pset_write_queue (struct Lisp_Process *p, Lisp_Object val)
 {
   p->write_queue = val;
 }
+static void
+pset_pipe_list (struct Lisp_Process *p, Lisp_Object val)
+{
+  p->pipe_list = val;
+}
 
 \f
 static Lisp_Object
@@ -739,6 +765,7 @@ make_process (Lisp_Object name)
   pset_name (p, name);
   pset_sentinel (p, Qinternal_default_process_sentinel);
   pset_filter (p, Qinternal_default_process_filter);
+  pset_pipe_list (p, Qnil);
   XSETPROCESS (val, p);
   Vprocess_alist = Fcons (Fcons (name, val), Vprocess_alist);
   return val;
@@ -846,7 +873,7 @@ nil, indicating the current buffer's process.  */)
   p = XPROCESS (process);
 
   p->raw_status_new = 0;
-  if (NETCONN1_P (p) || SERIALCONN1_P (p))
+  if (NETCONN1_P (p) || SERIALCONN1_P (p) || PIPECONN1_P (p))
     {
       pset_status (p, list2 (Qexit, make_number (0)));
       p->tick = ++process_tick;
@@ -912,7 +939,7 @@ nil, indicating the current buffer's process.  */)
   status = p->status;
   if (CONSP (status))
     status = XCAR (status);
-  if (NETCONN1_P (p) || SERIALCONN1_P (p))
+  if (NETCONN1_P (p) || SERIALCONN1_P (p) || PIPECONN1_P (p))
     {
       if (EQ (status, Qexit))
 	status = Qclosed;
@@ -996,7 +1023,7 @@ Return BUFFER.  */)
     CHECK_BUFFER (buffer);
   p = XPROCESS (process);
   pset_buffer (p, buffer);
-  if (NETCONN1_P (p) || SERIALCONN1_P (p))
+  if (NETCONN1_P (p) || SERIALCONN1_P (p) || PIPECONN1_P (p))
     pset_childp (p, Fplist_put (p->childp, QCbuffer, buffer));
   setup_process_coding_systems (process);
   return buffer;
@@ -1072,7 +1099,7 @@ The string argument is normally a multibyte string, except:
     }
 
   pset_filter (p, filter);
-  if (NETCONN1_P (p) || SERIALCONN1_P (p))
+  if (NETCONN1_P (p) || SERIALCONN1_P (p) || PIPECONN1_P (p))
     pset_childp (p, Fplist_put (p->childp, QCfilter, filter));
   setup_process_coding_systems (process);
   return filter;
@@ -1104,7 +1131,7 @@ It gets two arguments: the process, and a string describing the change.  */)
     sentinel = Qinternal_default_process_sentinel;
 
   pset_sentinel (p, sentinel);
-  if (NETCONN1_P (p) || SERIALCONN1_P (p))
+  if (NETCONN1_P (p) || SERIALCONN1_P (p) || PIPECONN1_P (p))
     pset_childp (p, Fplist_put (p->childp, QCsentinel, sentinel));
   return sentinel;
 }
@@ -1192,13 +1219,14 @@ DEFUN ("process-query-on-exit-flag",
 DEFUN ("process-contact", Fprocess_contact, Sprocess_contact,
        1, 2, 0,
        doc: /* Return the contact info of PROCESS; t for a real child.
-For a network or serial connection, the value depends on the optional
-KEY arg.  If KEY is nil, value is a cons cell of the form (HOST
-SERVICE) for a network connection or (PORT SPEED) for a serial
-connection.  If KEY is t, the complete contact information for the
-connection is returned, else the specific value for the keyword KEY is
-returned.  See `make-network-process' or `make-serial-process' for a
-list of keywords.  */)
+For a network or serial connection, or a pipe, the value depends on
+the optional KEY arg.  If KEY is nil, value is a cons cell of the form
+(HOST SERVICE) for a network connection or (PORT SPEED) for a serial
+connection, or (CHILD_INPUT CHILD_OUTPUT) for a pipe.  If KEY is t,
+the complete contact information for the connection is returned, else
+the specific value for the keyword KEY is returned.  See
+`make-network-process' or `make-serial-process' for a list of
+keywords.  */)
   (register Lisp_Object process, Lisp_Object key)
 {
   Lisp_Object contact;
@@ -1213,6 +1241,12 @@ list of keywords.  */)
 			  Fprocess_datagram_address (process));
 #endif
 
+  if (PIPECONN_P (process))
+    {
+      struct Lisp_Process *p = XPROCESS (process);
+      return list2 (make_number (p->open_fd[SUBPROCESS_STDIN]),
+		    make_number (p->open_fd[SUBPROCESS_STDOUT]));
+    }
   if ((!NETCONN_P (process) && !SERIALCONN_P (process)) || EQ (key, Qt))
     return contact;
   if (NILP (key) && NETCONN_P (process))
@@ -1614,42 +1648,40 @@ close_process_fd (int *fd_addr)
     }
 }
 
-/* Indexes of file descriptors in open_fds.  */
-enum
-  {
-    /* The pipe from Emacs to its subprocess.  */
-    SUBPROCESS_STDIN,
-    WRITE_TO_SUBPROCESS,
-
-    /* The main pipe from the subprocess to Emacs.  */
-    READ_FROM_SUBPROCESS,
-    SUBPROCESS_STDOUT,
-
-    /* The pipe from the subprocess to Emacs that is closed when the
-       subprocess execs.  */
-    READ_FROM_EXEC_MONITOR,
-    EXEC_MONITOR_OUTPUT
-  };
-
-verify (PROCESS_OPEN_FDS == EXEC_MONITOR_OUTPUT + 1);
-
 static void
 create_process (Lisp_Object process, char **new_argv, Lisp_Object current_dir)
 {
   struct Lisp_Process *p = XPROCESS (process);
-  int inchannel, outchannel;
+  int inchannel, outchannel, errchannel;
   pid_t pid;
   int vfork_errno;
-  int forkin, forkout;
+  int forkin, forkout, forkerr;
   bool pty_flag = 0;
   char pty_name[PTY_NAME_SIZE];
   Lisp_Object lisp_pty_name = Qnil;
   sigset_t oldset;
 
-  inchannel = outchannel = -1;
+  inchannel = outchannel = errchannel = -1;
 
   if (!NILP (Vprocess_connection_type))
     outchannel = inchannel = allocate_pty (pty_name);
+  else
+    {
+      if (!NILP (Vprocess_standard_error)
+	  && NILP (Fmemq (Vprocess_standard_error, Vprocess_pipe_list)))
+	error ("process-standard-error is not in process-pipe-list");
+      else
+	{
+	  Lisp_Object tem;
+
+	  for (tem = Vprocess_pipe_list; CONSP (tem); tem = XCDR (tem))
+	    {
+	      CHECK_PROCESS (XCAR (tem));
+	      if (!PIPECONN_P (XCAR (tem)))
+		error ("Process is not a pipe process");
+	    }
+	}
+    }
 
   if (inchannel >= 0)
     {
@@ -1666,6 +1698,7 @@ create_process (Lisp_Object process, char **new_argv, Lisp_Object current_dir)
 #else
       forkin = forkout = -1;
 #endif /* not USG, or USG_SUBTTY_WORKS */
+      forkerr = -1;
       pty_flag = 1;
       lisp_pty_name = build_string (pty_name);
     }
@@ -1678,6 +1711,22 @@ create_process (Lisp_Object process, char **new_argv, Lisp_Object current_dir)
       outchannel = p->open_fd[WRITE_TO_SUBPROCESS];
       inchannel = p->open_fd[READ_FROM_SUBPROCESS];
       forkout = p->open_fd[SUBPROCESS_STDOUT];
+
+      if (!NILP (Vprocess_standard_error))
+	{
+	  struct Lisp_Process *pp = XPROCESS (Vprocess_standard_error);
+
+	  forkerr = pp->open_fd[SUBPROCESS_STDOUT];
+	  errchannel = pp->open_fd[READ_FROM_SUBPROCESS];
+
+	  /* FORKERR will be redirected in child_setup.  */
+	  fcntl (forkerr, F_SETFD, FD_CLOEXEC);
+	  /* Close unnecessary file descriptors.  */
+	  close_process_fd (&pp->open_fd[WRITE_TO_SUBPROCESS]);
+	  close_process_fd (&pp->open_fd[SUBPROCESS_STDIN]);
+	}
+
+      pset_pipe_list (p, Fcopy_sequence (Vprocess_pipe_list));
     }
 
 #ifndef WINDOWSNT
@@ -1720,6 +1769,7 @@ create_process (Lisp_Object process, char **new_argv, Lisp_Object current_dir)
     char **volatile new_argv_volatile = new_argv;
     int volatile forkin_volatile = forkin;
     int volatile forkout_volatile = forkout;
+    int volatile forkerr_volatile = forkerr;
     struct Lisp_Process *p_volatile = p;
 
     pid = vfork ();
@@ -1729,6 +1779,7 @@ create_process (Lisp_Object process, char **new_argv, Lisp_Object current_dir)
     new_argv = new_argv_volatile;
     forkin = forkin_volatile;
     forkout = forkout_volatile;
+    forkerr = forkerr_volatile;
     p = p_volatile;
 
     pty_flag = p->pty_flag;
@@ -1739,6 +1790,7 @@ create_process (Lisp_Object process, char **new_argv, Lisp_Object current_dir)
     {
       int xforkin = forkin;
       int xforkout = forkout;
+      int xforkerr = forkerr;
 
       /* Make the pty be the controlling terminal of the process.  */
 #ifdef HAVE_PTYS
@@ -1838,10 +1890,13 @@ create_process (Lisp_Object process, char **new_argv, Lisp_Object current_dir)
 
       if (pty_flag)
 	child_setup_tty (xforkout);
+
+      if (xforkerr < 0)
+	xforkerr = xforkout;
 #ifdef WINDOWSNT
-      pid = child_setup (xforkin, xforkout, xforkout, new_argv, 1, current_dir);
+      pid = child_setup (xforkin, xforkout, xforkerr, new_argv, 1, current_dir);
 #else  /* not WINDOWSNT */
-      child_setup (xforkin, xforkout, xforkout, new_argv, 1, current_dir);
+      child_setup (xforkin, xforkout, xforkerr, new_argv, 1, current_dir);
 #endif /* not WINDOWSNT */
     }
 
@@ -1861,11 +1916,23 @@ create_process (Lisp_Object process, char **new_argv, Lisp_Object current_dir)
   else
     {
       /* vfork succeeded.  */
+      Lisp_Object tem;
 
       /* Close the pipe ends that the child uses, or the child's pty.  */
       close_process_fd (&p->open_fd[SUBPROCESS_STDIN]);
       close_process_fd (&p->open_fd[SUBPROCESS_STDOUT]);
 
+      for (tem = p->pipe_list; CONSP (tem); tem = XCDR (tem))
+	{
+	  struct Lisp_Process *pp = XPROCESS (XCAR (tem));
+
+	  /* FIXME: Why we can't close them immediately?  */
+	  if (pp->open_fd[SUBPROCESS_STDIN] >= 0)
+	    fcntl (pp->open_fd[SUBPROCESS_STDIN], F_SETFD, FD_CLOEXEC);
+	  if (pp->open_fd[SUBPROCESS_STDOUT] >= 0)
+	    fcntl (pp->open_fd[SUBPROCESS_STDOUT], F_SETFD, FD_CLOEXEC);
+	}
+
 #ifdef WINDOWSNT
       register_child (pid, inchannel);
 #endif /* WINDOWSNT */
@@ -1944,6 +2011,186 @@ create_pty (Lisp_Object process)
   p->pid = -2;
 }
 
+DEFUN ("make-pipe-process", Fmake_pipe_process, Smake_pipe_process,
+       0, MANY, 0,
+       doc: /* Create and return a bidirectional pipe process.
+
+In Emacs, pipes are represented by process objects, so input and
+output work as for subprocesses, and `delete-process' closes a pipe.
+However, a pipe process has no process id, it cannot be signaled,
+and the status codes are different from normal processes.
+
+Arguments are specified as keyword/argument pairs.  The following
+arguments are defined:
+
+:name NAME -- NAME is the name of the process.  It is modified if necessary to make it unique.
+
+:buffer BUFFER -- BUFFER is the buffer (or buffer-name) to associate
+with the process.  Process output goes at the end of that buffer,
+unless you specify an output stream or filter function to handle the
+output.  If BUFFER is not given, the value of NAME is used.
+
+:coding CODING -- If CODING is a symbol, it specifies the coding
+system used for both reading and writing for this process.  If CODING
+is a cons (DECODING . ENCODING), DECODING is used for reading, and
+ENCODING is used for writing.
+
+:noquery BOOL -- When exiting Emacs, query the user if BOOL is nil and
+the process is running.  If BOOL is not given, query before exiting.
+
+:stop BOOL -- Start process in the `stopped' state if BOOL is non-nil.
+In the stopped state, a serial process does not accept incoming data,
+but you can send outgoing data.  The stopped state is cleared by
+`continue-process' and set by `stop-process'.
+
+:filter FILTER -- Install FILTER as the process filter.
+
+:sentinel SENTINEL -- Install SENTINEL as the process sentinel.
+
+usage:  (make-pipe-process &rest ARGS)  */)
+  (ptrdiff_t nargs, Lisp_Object *args)
+{
+  Lisp_Object proc, contact;
+  struct Lisp_Process *p;
+  struct gcpro gcpro1;
+  Lisp_Object name, buffer;
+  Lisp_Object tem, val;
+  ptrdiff_t specpdl_count;
+  int inchannel, outchannel;
+
+  if (nargs == 0)
+    return Qnil;
+
+  contact = Flist (nargs, args);
+  GCPRO1 (contact);
+
+  name = Fplist_get (contact, QCname);
+  CHECK_STRING (name);
+  proc = make_process (name);
+  specpdl_count = SPECPDL_INDEX ();
+  record_unwind_protect (remove_process, proc);
+  p = XPROCESS (proc);
+
+  if (emacs_pipe2 (p->open_fd + SUBPROCESS_STDIN, O_BINARY) != 0
+      || emacs_pipe2 (p->open_fd + READ_FROM_SUBPROCESS, O_BINARY) != 0)
+    report_file_error ("Creating pipe", Qnil);
+  outchannel = p->open_fd[WRITE_TO_SUBPROCESS];
+  inchannel = p->open_fd[READ_FROM_SUBPROCESS];
+
+  /* Child doesn't need inchannel/outchannel after exec.  */
+  fcntl (inchannel, F_SETFD, FD_CLOEXEC);
+  fcntl (outchannel, F_SETFD, FD_CLOEXEC);
+
+  fcntl (inchannel, F_SETFL, O_NONBLOCK);
+  fcntl (outchannel, F_SETFL, O_NONBLOCK);
+
+  /* Record this as an active process, with its channels.  */
+  chan_process[inchannel] = proc;
+  p->infd = inchannel;
+  p->outfd = outchannel;
+
+  FD_SET (inchannel, &input_wait_mask);
+  FD_SET (inchannel, &non_keyboard_wait_mask);
+  if (inchannel > max_process_desc)
+    max_process_desc = inchannel;
+
+  buffer = Fplist_get (contact, QCbuffer);
+  if (NILP (buffer))
+    buffer = name;
+  buffer = Fget_buffer_create (buffer);
+  pset_buffer (p, buffer);
+
+  pset_childp (p, contact);
+  pset_plist (p, Fcopy_sequence (Fplist_get (contact, QCplist)));
+  pset_type (p, Qpipe);
+  pset_sentinel (p, Fplist_get (contact, QCsentinel));
+  pset_filter (p, Fplist_get (contact, QCfilter));
+  pset_log (p, Qnil);
+  if (tem = Fplist_get (contact, QCnoquery), !NILP (tem))
+    p->kill_without_query = 1;
+  if (tem = Fplist_get (contact, QCstop), !NILP (tem))
+    pset_command (p, Qt);
+  eassert (! p->pty_flag);
+
+#ifdef ADAPTIVE_READ_BUFFERING
+  p->adaptive_read_buffering
+    = (NILP (Vprocess_adaptive_read_buffering) ? 0
+       : EQ (Vprocess_adaptive_read_buffering, Qt) ? 1 : 2);
+#endif
+
+  /* Make the process marker point into the process buffer (if any).  */
+  if (BUFFERP (buffer))
+    set_marker_both (p->mark, buffer,
+		     BUF_ZV (XBUFFER (buffer)),
+		     BUF_ZV_BYTE (XBUFFER (buffer)));
+
+  tem = Fplist_member (contact, QCcoding);
+  if (!NILP (tem) && (!CONSP (tem) || !CONSP (XCDR (tem))))
+    tem = Qnil;  /* No error message (too late!).  */
+
+  {
+    /* Setup coding systems for communicating with the network stream.  */
+    struct gcpro gcpro1;
+    /* Qt denotes we have not yet called Ffind_operation_coding_system.  */
+    Lisp_Object coding_systems = Qt;
+    Lisp_Object val;
+
+    if (!NILP (tem))
+      {
+	val = XCAR (XCDR (tem));
+	if (CONSP (val))
+	  val = XCAR (val);
+      }
+    else if (!NILP (Vcoding_system_for_read))
+      val = Vcoding_system_for_read;
+    else if ((!NILP (buffer) && NILP (BVAR (XBUFFER (buffer), enable_multibyte_characters)))
+	     || (NILP (buffer) && NILP (BVAR (&buffer_defaults, enable_multibyte_characters))))
+      /* We dare not decode end-of-line format by setting VAL to
+	 Qraw_text, because the existing Emacs Lisp libraries
+	 assume that they receive bare code including a sequence of
+	 CR LF.  */
+      val = Qnil;
+    else
+      {
+	if (CONSP (coding_systems))
+	  val = XCAR (coding_systems);
+	else if (CONSP (Vdefault_process_coding_system))
+	  val = XCAR (Vdefault_process_coding_system);
+	else
+	  val = Qnil;
+      }
+    pset_decode_coding_system (p, val);
+
+    if (!NILP (tem))
+      {
+	val = XCAR (XCDR (tem));
+	if (CONSP (val))
+	  val = XCDR (val);
+      }
+    else if (!NILP (Vcoding_system_for_write))
+      val = Vcoding_system_for_write;
+    else if (NILP (BVAR (current_buffer, enable_multibyte_characters)))
+      val = Qnil;
+    else
+      {
+	if (CONSP (coding_systems))
+	  val = XCDR (coding_systems);
+	else if (CONSP (Vdefault_process_coding_system))
+	  val = XCDR (Vdefault_process_coding_system);
+	else
+	  val = Qnil;
+      }
+    pset_encode_coding_system (p, val);
+  }
+  /* This may signal an error.  */
+  setup_process_coding_systems (proc);
+
+  specpdl_ptr = specpdl + specpdl_count;
+
+  UNGCPRO;
+  return proc;
+}
+
 \f
 /* Convert an internal struct sockaddr to a lisp object (vector or string).
    The address family of sa is not included in the result.  */
@@ -3861,6 +4108,7 @@ deactivate_process (Lisp_Object proc)
   int inchannel;
   struct Lisp_Process *p = XPROCESS (proc);
   int i;
+  Lisp_Object tem;
 
 #ifdef HAVE_GNUTLS
   /* Delete GnuTLS structures in PROC, if any.  */
@@ -3919,6 +4167,9 @@ deactivate_process (Lisp_Object proc)
 	  max_process_desc = i;
 	}
     }
+
+  for (tem = p->pipe_list; CONSP (tem); tem = XCDR (tem))
+    Fdelete_process (XCAR (tem));
 }
 
 \f
@@ -5879,7 +6130,8 @@ If PROCESS is a network or serial process, inhibit handling of incoming
 traffic.  */)
   (Lisp_Object process, Lisp_Object current_group)
 {
-  if (PROCESSP (process) && (NETCONN_P (process) || SERIALCONN_P (process)))
+  if (PROCESSP (process) && (NETCONN_P (process) || SERIALCONN_P (process)
+			     || PIPECONN_P (process)))
     {
       struct Lisp_Process *p;
 
@@ -5908,7 +6160,8 @@ If PROCESS is a network or serial process, resume handling of incoming
 traffic.  */)
   (Lisp_Object process, Lisp_Object current_group)
 {
-  if (PROCESSP (process) && (NETCONN_P (process) || SERIALCONN_P (process)))
+  if (PROCESSP (process) && (NETCONN_P (process) || SERIALCONN_P (process)
+			     || PIPECONN_P (process)))
     {
       struct Lisp_Process *p;
 
@@ -6955,7 +7208,7 @@ kill_buffer_processes (Lisp_Object buffer)
   FOR_EACH_PROCESS (tail, proc)
     if (NILP (buffer) || EQ (XPROCESS (proc)->buffer, buffer))
       {
-	if (NETCONN_P (proc) || SERIALCONN_P (proc))
+	if (NETCONN_P (proc) || SERIALCONN_P (proc) || PIPECONN_P (proc))
 	  Fdelete_process (proc);
 	else if (XPROCESS (proc)->infd >= 0)
 	  process_send_signal (proc, SIGHUP, Qnil, 1);
@@ -7255,6 +7508,7 @@ syms_of_process (void)
   DEFSYM (Qreal, "real");
   DEFSYM (Qnetwork, "network");
   DEFSYM (Qserial, "serial");
+  DEFSYM (Qpipe, "pipe");
   DEFSYM (QCbuffer, ":buffer");
   DEFSYM (QChost, ":host");
   DEFSYM (QCservice, ":service");
@@ -7347,6 +7601,17 @@ The variable takes effect when `start-process' is called.  */);
   Vprocess_adaptive_read_buffering = Qt;
 #endif
 
+  DEFVAR_LISP ("process-pipe-list", Vprocess_pipe_list,
+	       doc: /* List of pipe processes attached to child process.
+The value takes effect when `start-process' is called.  */);
+  Vprocess_pipe_list = Qnil;
+
+  DEFVAR_LISP ("process-standard-error", Vprocess_standard_error,
+	       doc: /* Pipe process attached to standard error output of child process.
+This process must be listed in `process-pipe-list'.
+The value takes effect when `start-process' is called.  */);
+  Vprocess_standard_error = Qnil;
+
   defsubr (&Sprocessp);
   defsubr (&Sget_process);
   defsubr (&Sdelete_process);
@@ -7372,6 +7637,7 @@ The variable takes effect when `start-process' is called.  */);
   defsubr (&Sset_process_plist);
   defsubr (&Sprocess_list);
   defsubr (&Sstart_process);
+  defsubr (&Smake_pipe_process);
   defsubr (&Sserial_process_configure);
   defsubr (&Smake_serial_process);
   defsubr (&Sset_network_process_option);
diff --git a/src/process.h b/src/process.h
index 36979dc..b76ebd2 100644
--- a/src/process.h
+++ b/src/process.h
@@ -105,6 +105,9 @@ struct Lisp_Process
     Lisp_Object gnutls_cred_type;
 #endif
 
+    /* Pipe processes attached to this process.  */
+    Lisp_Object pipe_list;
+
     /* After this point, there are no Lisp_Objects any more.  */
     /* alloc.c assumes that `pid' is the first such non-Lisp slot.  */
 
diff --git a/src/sysdep.c b/src/sysdep.c
index 0a0b0ac..2fe4624 100644
--- a/src/sysdep.c
+++ b/src/sysdep.c
@@ -2257,19 +2257,27 @@ emacs_fopen (char const *file, char const *mode)
 /* Create a pipe for Emacs use.  */
 
 int
-emacs_pipe (int fd[2])
+emacs_pipe2 (int fd[2], int flags)
 {
 #ifdef MSDOS
   return pipe (fd);
 #else  /* !MSDOS */
-  int result = pipe2 (fd, O_BINARY | O_CLOEXEC);
+  return pipe2 (fd, flags);
+#endif	/* !MSDOS */
+}
+
+int
+emacs_pipe (int fd[2])
+{
+  int result = emacs_pipe2 (fd, O_BINARY | O_CLOEXEC);
+#ifndef MSDOS
   if (! O_CLOEXEC && result == 0)
     {
       fcntl (fd[0], F_SETFD, FD_CLOEXEC);
       fcntl (fd[1], F_SETFD, FD_CLOEXEC);
     }
-  return result;
 #endif	/* !MSDOS */
+  return result;
 }
 
 /* Approximate posix_close and POSIX_CLOSE_RESTART well enough for Emacs.
-- 
2.1.0


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #3: 0002-epg-Avoid-using-temporary-files.patch --]
[-- Type: text/x-patch, Size: 19974 bytes --]

From ab2b2ec47b7b7eb22582fb138a134be6d0af84dd Mon Sep 17 00:00:00 2001
From: Daiki Ueno <ueno@gnu.org>
Date: Thu, 12 Mar 2015 16:47:57 +0900
Subject: [PATCH 2/2] epg: Avoid using temporary files

* epg.el (epg-error-output): Abolish.
(epg-context): Remove slot output-file, add slots error-process,
status-process, and command-process.
(epg--start): Separate stdout, stderr, --status-fd output, and
--command-fd output using pipe process.
(epg--process-filter): Define as a process filter for the pipe process
watching the --status-fd output.
(epg--process-sentinel): New sentinel.
(epg-read-output): Use the content of process buffer, instead of
temporary file.
(epg-wait-for-status, epg-wait-for-completion): Read --status-fd
output instead of the process output.
(epg-reset): Kill all buffers associated with the process.
(epg-delete-output-file): Abolish.
(epg-decrypt-file, epg-decrypt-string, epg-verify-file)
(epg-verify-string, epg-sign-file, epg-sign-string)
(epg-encrypt-file, epg-encrypt-string)
(epg-export-keys-to-file): Remove temporary file usage.
---
 lisp/epg.el | 182 +++++++++++++++++++++++++++++-------------------------------
 1 file changed, 88 insertions(+), 94 deletions(-)

diff --git a/lisp/epg.el b/lisp/epg.el
index f665453..97119b1 100644
--- a/lisp/epg.el
+++ b/lisp/epg.el
@@ -40,7 +40,6 @@
 (defvar epg-debug-buffer nil)
 (defvar epg-agent-file nil)
 (defvar epg-agent-mtime nil)
-(defvar epg-error-output nil)
 
 ;; from gnupg/include/cipher.h
 (defconst epg-cipher-algorithm-alist
@@ -209,11 +208,13 @@
   signers
   sig-notations
   process
-  output-file
   result
   operation
   pinentry-mode
-  (error-output ""))
+  (error-output "")
+  error-process
+  status-process
+  command-process)
 
 ;; This is not an alias, just so we can mark it as autoloaded.
 ;;;###autoload
@@ -558,8 +559,19 @@ callback data (if any)."
       (error "%s is already running in this context"
 	     (epg-context-program context)))
   (let* ((agent-info (getenv "GPG_AGENT_INFO"))
+	 (error-buffer (generate-new-buffer " *epg-error*"))
+	 (error-process (make-pipe-process :name "epg-error"
+					   :buffer error-buffer
+					   :noquery t))
+	 (status-buffer (generate-new-buffer " *epg-status*"))
+	 (status-process (make-pipe-process :name "epg-status"
+					    :buffer status-buffer
+					    :noquery t
+					    :filter #'epg--process-filter))
 	 (args (append (list "--no-tty"
-			     "--status-fd" "1"
+			     "--status-fd"
+			     (number-to-string
+			      (nth 1 (process-contact status-process)))
 			     "--yes")
 		       (if (and (not (eq (epg-context-protocol context) 'CMS))
 				(string-match ":" (or agent-info "")))
@@ -570,12 +582,8 @@ callback data (if any)."
 		       (if (epg-context-home-directory context)
 			   (list "--homedir"
 				 (epg-context-home-directory context)))
-		       (unless (eq (epg-context-protocol context) 'CMS)
-			 '("--command-fd" "0"))
 		       (if (epg-context-armor context) '("--armor"))
 		       (if (epg-context-textmode context) '("--textmode"))
-		       (if (epg-context-output-file context)
-			   (list "--output" (epg-context-output-file context)))
 		       (if (epg-context-pinentry-mode context)
 			   (list "--pinentry-mode"
 				 (symbol-name (epg-context-pinentry-mode
@@ -584,12 +592,25 @@ callback data (if any)."
 	 (coding-system-for-write 'binary)
 	 (coding-system-for-read 'binary)
 	 process-connection-type
+	 (process-pipe-list (list error-process status-process))
+	 (process-standard-error error-process)
 	 (process-environment process-environment)
 	 (buffer (generate-new-buffer " *epg*"))
 	 process
 	 terminal-name
 	 agent-file
-	 (agent-mtime '(0 0 0 0)))
+	 (agent-mtime '(0 0 0 0))
+	 command-buffer command-process)
+    (unless (eq (epg-context-protocol context) 'CMS)
+      (setq command-buffer (generate-new-buffer " *epg-command*")
+	    command-process (make-pipe-process :name "epg-command"
+					       :buffer command-buffer
+					       :noquery t))
+      (push command-process process-pipe-list)
+      (setq args (append (list "--command-fd"
+			       (number-to-string
+				(car (process-contact command-process))))
+			 args)))
     ;; Set GPG_TTY and TERM for pinentry-curses.  Note that we can't
     ;; use `terminal-name' here to get the real pty name for the child
     ;; process, though /dev/fd/0" is not portable.
@@ -626,6 +647,9 @@ callback data (if any)."
 			  (mapconcat #'identity args " ")))))
     (with-current-buffer buffer
       (if (fboundp 'set-buffer-multibyte)
+	  (set-buffer-multibyte nil)))
+    (with-current-buffer status-buffer
+      (if (fboundp 'set-buffer-multibyte)
 	  (set-buffer-multibyte nil))
       (make-local-variable 'epg-last-status)
       (setq epg-last-status nil)
@@ -642,13 +666,15 @@ callback data (if any)."
       (make-local-variable 'epg-agent-file)
       (setq epg-agent-file agent-file)
       (make-local-variable 'epg-agent-mtime)
-      (setq epg-agent-mtime agent-mtime)
-      (make-local-variable 'epg-error-output)
-      (setq epg-error-output nil))
+      (setq epg-agent-mtime agent-mtime))
+    (setf (epg-context-error-process context) error-process)
+    (setf (epg-context-status-process context) status-process)
+    (if command-process
+	(setf (epg-context-command-process context) command-process))
     (with-file-modes 448
       (setq process (apply #'start-process "epg" buffer
 			   (epg-context-program context) args)))
-    (set-process-filter process #'epg--process-filter)
+    (set-process-sentinel process #'epg--process-sentinel)
     (setf (epg-context-process context) process)))
 
 (defun epg--process-filter (process input)
@@ -658,8 +684,11 @@ callback data (if any)."
               (setq epg-debug-buffer (generate-new-buffer " *epg-debug*")))
 	(goto-char (point-max))
 	(insert input)))
-  (if (buffer-live-p (process-buffer process))
-      (with-current-buffer (process-buffer process)
+  (when (buffer-live-p (process-buffer process))
+    (let ((context
+	   (with-current-buffer (process-buffer process)
+	     epg-context)))
+      (with-current-buffer (process-buffer (epg-context-status-process context))
         (save-excursion
           (goto-char (point-max))
           (insert input)
@@ -690,34 +719,28 @@ callback data (if any)."
 			(if (and symbol
 				 (fboundp symbol))
 			    (funcall symbol epg-context string)))
-                      (setq epg-last-status (cons status string)))
-		  ;; Record other lines sent to stderr.  This assumes
-		  ;; that the process-filter receives output only from
-		  ;; stderr and the FD specified with --status-fd.
-		  (setq epg-error-output
-			(cons (buffer-substring (point)
-						(line-end-position))
-			      epg-error-output)))
+                      (setq epg-last-status (cons status string))))
                 (forward-line)
-                (setq epg-read-point (point)))))))))
+                (setq epg-read-point (point))))))))))
+
+(defun epg--process-sentinel (_process _status)
+  ;; Do nothing.
+  )
 
 (defun epg-read-output (context)
   "Read the output file CONTEXT and return the content as a string."
-  (with-temp-buffer
-    (if (fboundp 'set-buffer-multibyte)
-	(set-buffer-multibyte nil))
-    (if (file-exists-p (epg-context-output-file context))
-	(let ((coding-system-for-read 'binary))
-	  (insert-file-contents (epg-context-output-file context))
+  (let ((buffer (process-buffer (epg-context-process context))))
+    (if (buffer-live-p buffer)
+	(with-current-buffer buffer
 	  (buffer-string)))))
 
 (defun epg-wait-for-status (context status-list)
   "Wait until one of elements in STATUS-LIST arrives."
-  (with-current-buffer (process-buffer (epg-context-process context))
+  (with-current-buffer (process-buffer (epg-context-status-process context))
     (setq epg-pending-status-list status-list)
     (while (and (eq (process-status (epg-context-process context)) 'run)
 		epg-pending-status-list)
-      (accept-process-output (epg-context-process context) 1))
+      (accept-process-output (epg-context-status-process context) 1 0 1))
     (if epg-pending-status-list
 	(epg-context-set-result-for
 	 context 'error
@@ -727,11 +750,11 @@ callback data (if any)."
 (defun epg-wait-for-completion (context)
   "Wait until the `epg-gpg-program' process completes."
   (while (eq (process-status (epg-context-process context)) 'run)
-    (accept-process-output (epg-context-process context) 1))
+    (accept-process-output (epg-context-status-process context) 1 0 1))
   ;; This line is needed to run the process-filter right now.
   (sleep-for 0.1)
   ;; Restore Emacs frame on text terminal, when pinentry-curses has terminated.
-  (if (with-current-buffer (process-buffer (epg-context-process context))
+  (if (with-current-buffer (process-buffer (epg-context-status-process context))
 	(and epg-agent-file
 	     (> (float-time (or (nth 5 (file-attributes epg-agent-file))
 				'(0 0 0 0)))
@@ -740,24 +763,26 @@ callback data (if any)."
   (epg-context-set-result-for
    context 'error
    (nreverse (epg-context-result-for context 'error)))
-  (with-current-buffer (process-buffer (epg-context-process context))
-    (setf (epg-context-error-output context)
-	(mapconcat #'identity (nreverse epg-error-output) "\n"))))
+  (with-current-buffer (process-buffer (epg-context-error-process context))
+    (setf (epg-context-error-output context) (buffer-string))))
 
 (defun epg-reset (context)
   "Reset the CONTEXT."
-  (if (and (epg-context-process context)
-	   (buffer-live-p (process-buffer (epg-context-process context))))
-      (kill-buffer (process-buffer (epg-context-process context))))
+  (let ((processes
+	 (delq nil
+	       (list (epg-context-process context)
+		     (epg-context-error-process context)
+		     (epg-context-status-process context)
+		     (epg-context-command-process context)))))
+    (dolist (process processes)
+      (if (buffer-live-p (process-buffer process))
+	  (kill-buffer (process-buffer process)))))
   (setf (epg-context-process context) nil)
+  (setf (epg-context-error-process context) nil)
+  (setf (epg-context-status-process context) nil)
+  (setf (epg-context-command-process context) nil)
   (setf (epg-context-edit-callback context) nil))
 
-(defun epg-delete-output-file (context)
-  "Delete the output file of CONTEXT."
-  (if (and (epg-context-output-file context)
-	   (file-exists-p (epg-context-output-file context)))
-      (delete-file (epg-context-output-file context))))
-
 (eval-and-compile
   (if (fboundp 'decode-coding-string)
       (defalias 'epg--decode-coding-string 'decode-coding-string)
@@ -833,7 +858,7 @@ callback data (if any)."
 		    (setq encoded-passphrase-with-new-line
 			  passphrase-with-new-line
 			  passphrase-with-new-line nil))
-		  (process-send-string (epg-context-process context)
+		  (process-send-string (epg-context-command-process context)
 				       encoded-passphrase-with-new-line)))
 	    (quit
 	     (epg-context-set-result-for
@@ -870,8 +895,8 @@ callback data (if any)."
 	(if (funcall (or (intern-soft (concat "epg--prompt-GET_BOOL-" string))
 			 #'epg--prompt-GET_BOOL)
 		     context string)
-	    (process-send-string (epg-context-process context) "y\n")
-	  (process-send-string (epg-context-process context) "n\n"))
+	    (process-send-string (epg-context-command-process context) "y\n")
+	  (process-send-string (epg-context-command-process context) "n\n"))
       (quit
        (epg-context-set-result-for
 	context 'error
@@ -883,7 +908,7 @@ callback data (if any)."
   (let ((entry (assoc string epg-prompt-alist))
 	inhibit-quit)
     (condition-case nil
-	(process-send-string (epg-context-process context)
+	(process-send-string (epg-context-command-process context)
 			     (concat (read-string
 				      (if entry
 					  (cdr entry)
@@ -1469,8 +1494,8 @@ You can then use `write-region' to write new data into the file."
 	  notations)))
 
 (defun epg-cancel (context)
-  (if (buffer-live-p (process-buffer (epg-context-process context)))
-      (with-current-buffer (process-buffer (epg-context-process context))
+  (if (buffer-live-p (process-buffer (epg-context-status-process context)))
+      (with-current-buffer (process-buffer (epg-context-status-process context))
 	(epg-context-set-result-for
 	 epg-context 'error
 	 (cons '(quit)
@@ -1480,18 +1505,23 @@ You can then use `write-region' to write new data into the file."
 
 (defun epg-start-decrypt (context cipher)
   "Initiate a decrypt operation on CIPHER.
-CIPHER must be a file data object.
+CIPHER must be a data object.
 
 If you use this function, you will need to wait for the completion of
 `epg-gpg-program' by using `epg-wait-for-completion' and call
 `epg-reset' to clear a temporary output file.
 If you are unsure, use synchronous version of this function
 `epg-decrypt-file' or `epg-decrypt-string' instead."
-  (unless (epg-data-file cipher)
-    (error "Not a file"))
   (setf (epg-context-operation context) 'decrypt)
   (setf (epg-context-result context) nil)
-  (epg--start context (list "--decrypt" "--" (epg-data-file cipher)))
+  (if (epg-data-file cipher)
+      (epg--start context (list "--decrypt" "--" (epg-data-file cipher)))
+    (epg--start context (list "--decrypt"))
+    (if (eq (process-status (epg-context-process context)) 'run)
+	(process-send-string (epg-context-process context)
+			     (epg-data-string cipher)))
+    (if (eq (process-status (epg-context-process context)) 'run)
+	(process-send-eof (epg-context-process context))))
   ;; `gpgsm' does not read passphrase from stdin, so waiting is not needed.
   (unless (eq (epg-context-protocol context) 'CMS)
     (epg-wait-for-status context '("BEGIN_DECRYPTION"))))
@@ -1510,33 +1540,22 @@ If you are unsure, use synchronous version of this function
 If PLAIN is nil, it returns the result as a string."
   (unwind-protect
       (progn
-	(setf (epg-context-output-file context)
-              (or plain (epg--make-temp-file "epg-output")))
 	(epg-start-decrypt context (epg-make-data-from-file cipher))
 	(epg-wait-for-completion context)
 	(epg--check-error-for-decrypt context)
 	(unless plain
 	  (epg-read-output context)))
-    (unless plain
-      (epg-delete-output-file context))
     (epg-reset context)))
 
 (defun epg-decrypt-string (context cipher)
   "Decrypt a string CIPHER and return the plain text."
-  (let ((input-file (epg--make-temp-file "epg-input"))
-	(coding-system-for-write 'binary))
+  (let ((coding-system-for-write 'binary))
     (unwind-protect
 	(progn
-	  (write-region cipher nil input-file nil 'quiet)
-	  (setf (epg-context-output-file context)
-                (epg--make-temp-file "epg-output"))
-	  (epg-start-decrypt context (epg-make-data-from-file input-file))
+	  (epg-start-decrypt context (epg-make-data-from-string cipher))
 	  (epg-wait-for-completion context)
 	  (epg--check-error-for-decrypt context)
 	  (epg-read-output context))
-      (epg-delete-output-file context)
-      (if (file-exists-p input-file)
-	  (delete-file input-file))
       (epg-reset context))))
 
 (defun epg-start-verify (context signature &optional signed-text)
@@ -1599,8 +1618,6 @@ To check the verification results, use `epg-context-result-for' as follows:
 which will return a list of `epg-signature' object."
   (unwind-protect
       (progn
-        (setf (epg-context-output-file context)
-              (or plain (epg--make-temp-file "epg-output")))
 	(if signed-text
 	    (epg-start-verify context
 			      (epg-make-data-from-file signature)
@@ -1610,8 +1627,6 @@ which will return a list of `epg-signature' object."
 	(epg-wait-for-completion context)
 	(unless plain
 	  (epg-read-output context)))
-    (unless plain
-      (epg-delete-output-file context))
     (epg-reset context)))
 
 (defun epg-verify-string (context signature &optional signed-text)
@@ -1636,8 +1651,6 @@ which will return a list of `epg-signature' object."
 	input-file)
     (unwind-protect
 	(progn
-	  (setf (epg-context-output-file context)
-                (epg--make-temp-file "epg-output"))
 	  (if signed-text
 	      (progn
 		(setq input-file (epg--make-temp-file "epg-signature"))
@@ -1648,7 +1661,6 @@ which will return a list of `epg-signature' object."
 	    (epg-start-verify context (epg-make-data-from-string signature)))
 	  (epg-wait-for-completion context)
 	  (epg-read-output context))
-      (epg-delete-output-file context)
       (if (and input-file
 	       (file-exists-p input-file))
 	  (delete-file input-file))
@@ -1707,8 +1719,6 @@ If it is nil or 'normal, it makes a normal signature.
 Otherwise, it makes a cleartext signature."
   (unwind-protect
       (progn
-        (setf (epg-context-output-file context)
-              (or signature (epg--make-temp-file "epg-output")))
 	(epg-start-sign context (epg-make-data-from-file plain) mode)
 	(epg-wait-for-completion context)
 	(unless (epg-context-result-for context 'sign)
@@ -1717,8 +1727,6 @@ Otherwise, it makes a cleartext signature."
 		    (list "Sign failed" (epg-errors-to-string errors)))))
 	(unless signature
 	  (epg-read-output context)))
-    (unless signature
-      (epg-delete-output-file context))
     (epg-reset context)))
 
 (defun epg-sign-string (context plain &optional mode)
@@ -1737,8 +1745,6 @@ Otherwise, it makes a cleartext signature."
 	(coding-system-for-write 'binary))
     (unwind-protect
 	(progn
-	  (setf (epg-context-output-file context)
-                (epg--make-temp-file "epg-output"))
 	  (if input-file
 	      (write-region plain nil input-file nil 'quiet))
 	  (epg-start-sign context
@@ -1753,7 +1759,6 @@ Otherwise, it makes a cleartext signature."
 		  (signal 'epg-error
 			  (list "Sign failed" (epg-errors-to-string errors))))))
 	  (epg-read-output context))
-      (epg-delete-output-file context)
       (if input-file
 	  (delete-file input-file))
       (epg-reset context))))
@@ -1814,8 +1819,6 @@ If CIPHER is nil, it returns the result as a string.
 If RECIPIENTS is nil, it performs symmetric encryption."
   (unwind-protect
       (progn
-        (setf (epg-context-output-file context)
-              (or cipher (epg--make-temp-file "epg-output")))
 	(epg-start-encrypt context (epg-make-data-from-file plain)
 			   recipients sign always-trust)
 	(epg-wait-for-completion context)
@@ -1829,8 +1832,6 @@ If RECIPIENTS is nil, it performs symmetric encryption."
 		      (list "Encrypt failed" (epg-errors-to-string errors)))))
 	(unless cipher
 	  (epg-read-output context)))
-    (unless cipher
-      (epg-delete-output-file context))
     (epg-reset context)))
 
 (defun epg-encrypt-string (context plain recipients
@@ -1849,8 +1850,6 @@ If RECIPIENTS is nil, it performs symmetric encryption."
 	(coding-system-for-write 'binary))
     (unwind-protect
 	(progn
-	  (setf (epg-context-output-file context)
-                (epg--make-temp-file "epg-output"))
 	  (if input-file
 	      (write-region plain nil input-file nil 'quiet))
 	  (epg-start-encrypt context
@@ -1868,7 +1867,6 @@ If RECIPIENTS is nil, it performs symmetric encryption."
 		(signal 'epg-error
 			(list "Encrypt failed" (epg-errors-to-string errors)))))
 	  (epg-read-output context))
-      (epg-delete-output-file context)
       (if input-file
 	  (delete-file input-file))
       (epg-reset context))))
@@ -1894,8 +1892,6 @@ If you are unsure, use synchronous version of this function
   "Extract public KEYS."
   (unwind-protect
       (progn
-	(setf (epg-context-output-file context)
-              (or file (epg--make-temp-file "epg-output")))
 	(epg-start-export-keys context keys)
 	(epg-wait-for-completion context)
 	(let ((errors (epg-context-result-for context 'error)))
@@ -1905,8 +1901,6 @@ If you are unsure, use synchronous version of this function
 			    (epg-errors-to-string errors)))))
 	(unless file
 	  (epg-read-output context)))
-    (unless file
-      (epg-delete-output-file context))
     (epg-reset context)))
 
 (defun epg-export-keys-to-string (context keys)
-- 
2.1.0

Re: pipe

[Eli Zaretskii]

> From: Daiki Ueno <ueno@gnu.org>
> Date: Fri, 13 Mar 2015 18:01:48 +0900
> 
> Related to:
> https://lists.gnu.org/archive/html/emacs-devel/2014-09/msg00777.html
> in which I proposed a generalization of start-process, in a similar way
> to open-network-stream and make-network-process.

I don't see the additional file descriptors mentioned there, did I
miss something?

> My motivation behind that was to avoid temporary files in epg.el, by
> using file descriptors other than 0 and 1 (as you know, gpg has options
> --status-fd, --command-fd, --attribute-fd, etc. for that).

Can these features of gpg be used on MS-Windows?  IOW, how do you
invoke a subprocess with redirected file descriptors beyond the 3
standard ones, in a way that will work not only on Posix platforms
that support the full 'fork' functionality?

(I was about to ask why not use the existing feature of talking to
localhost via sockets, but then realized that it would suffer from the
same problem of how can we invoke a subprocess with more than just the
standard handles redirected.)

In any case, the Emacs side of this new process type, if it is added,
will "need work" in w32proc.c.

Thanks.

Re: pipe

[Daiki Ueno]

Eli Zaretskii <eliz@gnu.org> writes:

>> Related to:
>> https://lists.gnu.org/archive/html/emacs-devel/2014-09/msg00777.html
>> in which I proposed a generalization of start-process, in a similar way
>> to open-network-stream and make-network-process.
>
> I don't see the additional file descriptors mentioned there, did I
> miss something?

Sorry, I only mentioned it as a euphemism:

  * We could collect stderr output naturally.  'make-subprocess' could
    have a keyword, say :error, to prepare a pipe for stderr when spawning
    a process.

>> My motivation behind that was to avoid temporary files in epg.el, by
>> using file descriptors other than 0 and 1 (as you know, gpg has options
>> --status-fd, --command-fd, --attribute-fd, etc. for that).
>
> Can these features of gpg be used on MS-Windows?  IOW, how do you
> invoke a subprocess with redirected file descriptors beyond the 3
> standard ones, in a way that will work not only on Posix platforms
> that support the full 'fork' functionality?

Actually, I'm not sure, but there was a porting effort of the GPG stack
to Windows and Windows CE, and I think there should be a way to work
that around (added Werner to Cc).

> (I was about to ask why not use the existing feature of talking to
> localhost via sockets, but then realized that it would suffer from the
> same problem of how can we invoke a subprocess with more than just the
> standard handles redirected.)
>
> In any case, the Emacs side of this new process type, if it is added,
> will "need work" in w32proc.c.

Thanks for the info.

Regards,
--
Daiki Ueno

Re: pipe

[Stefan Monnier]

> My motivation behind that was to avoid temporary files in epg.el, by
> using file descriptors other than 0 and 1 (as you know, gpg has options
> --status-fd, --command-fd, --attribute-fd, etc. for that).  In order to
> do that, I thought that it would be inevitable to change the calling
> convention of start-process.  However, I hesitate to do such an
> intrusive change.  So, here is an alternative approach:

Supporting other file descriptors would be great, yes.  But I agree with
Eli: we need to take into account the w32 API in the design, because it
would significantly reduce the usefulness of such a feature if it only
works under POSIX systems.

This said, I know nothing about w32's API in this regard, so maybe your
suggestion could work.

BTW, what happened with your earlier `make-process' suggestion?


        Stefan

Re: pipe

[Daiki Ueno]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> BTW, what happened with your earlier `make-process' suggestion?

I'm still willing to work on that, but just wanted to be sure that my
desired feature (additional pipes) could be implemented atop of it,
maybe by adding a keyword :pipe-list.  Will post rebased patches soon.

Regards,
--
Daiki Ueno

Re: pipe

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Fri, 13 Mar 2015 08:45:17 -0400
> Cc: emacs-devel@gnu.org
> 
> This said, I know nothing about w32's API in this regard, so maybe your
> suggestion could work.

I certainly hope so.  I just said that I didn't know about a way, but
that doesn't mean it doesn't exist.

AFAIK, the only way to pass redirected files beyond the 3 standard
ones to subprocesses on Windows is via file handles (which are
actually pointers in disguise).  Handles to kernel objects are
system-global on Windows, so you can make a handle inheritable, and
then pass its value to the subprocess as a command-line argument; the
subprocess then should convert it to a Posix-style file descriptor by
calling a function.

But for this to work vis-à-vis gpg, the Windows port of gpg should
perform its part of this dance: it should know it accepts handles
(which could be 64-bit wide in 64-bit builds) instead of file
descriptors, and it should convert them to file descriptors before
using them as it would on Unix.

Re: pipe

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 1048 bytes --]

On 03/13/2015 07:54 AM, Eli Zaretskii wrote:
> file handles (which are
> actually pointers in disguise).

Only to the extent that file descriptors are also pointers in disguise.

> Handles to kernel objects are
> system-global on Windows,

They work like file descriptors.

> so you can make a handle inheritable, and
> then pass its value to the subprocess as a command-line argument; the
> subprocess then should convert it to a Posix-style file descriptor by
> calling a function.
> 
> But for this to work vis-à-vis gpg, the Windows port of gpg should
> perform its part of this dance: it should know it accepts handles
> (which could be 64-bit wide in 64-bit builds)

Even on 64-bit builds of Windows, HANDLEs only have 32 significant bits:
see

https://msdn.microsoft.com/en-us/library/windows/desktop/aa384203%28v=vs.85%29.aspx

https://msdn.microsoft.com/en-us/library/windows/desktop/aa384249%28v=vs.85%29.aspx

(Note that we can't know in advance whether a HANDLE will be shared with
a 32-bit application.)


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: pipe

[Eli Zaretskii]

> Date: Fri, 13 Mar 2015 08:28:50 -0700
> From: Daniel Colascione <dancol@dancol.org>
> Cc: ueno@gnu.org, emacs-devel@gnu.org
> 
> Even on 64-bit builds of Windows, HANDLEs only have 32 significant bits:

I don't think we should count on that.

I don't really agree with you about the other points, but I have no
interest in arguing.

Re: pipe

[Werner Koch]

On Fri, 13 Mar 2015 13:29, ueno@gnu.org said:

>> Can these features of gpg be used on MS-Windows?  IOW, how do you
>> invoke a subprocess with redirected file descriptors beyond the 3
>> standard ones, in a way that will work not only on Posix platforms
>> that support the full 'fork' functionality?
>
> Actually, I'm not sure, but there was a porting effort of the GPG stack
> to Windows and Windows CE, and I think there should be a way to work

GnuPG-2 runs very well on Windows for about 10 years.  Windows has pipes
but inheriting only specific handles (OS file descriptors) is a bit
complicated and requires the use of a wrapper process (gpgme-w32spawn,
Glib uses something similar).

The old WindowsCE has only per-process file descriptors.  Thus I had to
write a device driver to implement OS wide file descriptors.  But the
old WindowsCE is more or less dead.


Salam-Shalom,

   Werner

-- 
Die Gedanken sind frei.  Ausnahmen regelt ein Bundesgesetz.

Re: pipe

[Eli Zaretskii]

> From: Werner Koch <wk@gnupg.org>
> Date: Fri, 13 Mar 2015 21:08:21 +0100
> Cc: Eli Zaretskii <eliz@gnu.org>, emacs-devel@gnu.org
> 
> On Fri, 13 Mar 2015 13:29, ueno@gnu.org said:
> 
> >> Can these features of gpg be used on MS-Windows?  IOW, how do you
> >> invoke a subprocess with redirected file descriptors beyond the 3
> >> standard ones, in a way that will work not only on Posix platforms
> >> that support the full 'fork' functionality?
> >
> > Actually, I'm not sure, but there was a porting effort of the GPG stack
> > to Windows and Windows CE, and I think there should be a way to work
> 
> GnuPG-2 runs very well on Windows for about 10 years.  Windows has pipes
> but inheriting only specific handles (OS file descriptors) is a bit
> complicated and requires the use of a wrapper process (gpgme-w32spawn,
> Glib uses something similar).

Thanks, but where do I see the source of that wrapper?  I tried the
gpg Git repo, but I don't think I see it there.  Did I miss something?

And anyway, how does this wrapper solve the problem of invoking gpg
with the --status-fd, --command-fd, and --attribute-fd options?  IOW,
how can the invoking program pass file descriptors on the gpg command
line, and be sure gpg will use the same files/devices as the caller?

You say:

> The old WindowsCE has only per-process file descriptors.  Thus I had to
> write a device driver to implement OS wide file descriptors.

I'm guessing you actually mean "file handles", not "file descriptors"
here.  The latter are small integer numbers, like 0 for standard
input, 1 for standard output, etc. -- these are per-process on any
version of Windows, AFAIK.  So I'm unsure how can a file descriptor
like 6 be passed to gpg -- AFAIK it will end up connected to nothing
on the child side.  (There's an undocumented trick in the MSVC runtime
that allows this to work, but AFAIK that doesn't work with MinGW, only
with the Microsoft compilers.)

Re: pipe

[Werner Koch]

On Sat, 14 Mar 2015 09:54, eliz@gnu.org said:

> Thanks, but where do I see the source of that wrapper?  I tried the
> gpg Git repo, but I don't think I see it there.  Did I miss something?

It is in gpgme, a library interface to gnupg:

git://git.gnupg.org/gpgme.git

The wrapper is src/gpgme_w32spawn.c and the code in gpgme is in
src/w32-io.c (and src/w32-util.c).  However if you also link to glib the
used code is w32-glib-io.c.  

> And anyway, how does this wrapper solve the problem of invoking gpg
> with the --status-fd, --command-fd, and --attribute-fd options?  IOW,
> how can the invoking program pass file descriptors on the gpg command
> line, and be sure gpg will use the same files/devices as the caller?

Use gpgme ;-)

Simply use "--status-fd HANDLE" or if gpg expects a filename use the
option --enable-special-filenames which interprets filename of the form
"-&N" as the handle N.

Under Windows we use a HANDLE and not a file descriptor.  Thus ReadFile
and WriteFile is used.  Using the libc file handles is not a good idea.
The problem in gpg is that this is all mapped to an "int" and thus we
can't easily change this to a 64 bit Windows because a HANDLE is a
pointer and sizeof(void*) > sizeof(int).  There are too many different
objects in Windows with the same function but they are not all the
same.  Using a kernel object like HANDLE is the only clean solution.

> I'm guessing you actually mean "file handles", not "file descriptors"
> here.  The latter are small integer numbers, like 0 for standard

What I mean is that WindowsCE(!) has no global file descriptor table.
There is only the per-process one.  But WindowsCE is entirely different
form Windows.  They only share the the same API with lot of properties
not implemented.

> like 6 be passed to gpg -- AFAIK it will end up connected to nothing
> on the child side.  (There's an undocumented trick in the MSVC runtime

You create a pipe using CreatePipe and make one end inheritable using
DuplicateHandle.  See gpgme/src/w32-io.c - it is a bit complicated due
to the fact that you can't select (WaitForMultipleObjects) on HANDLEs.

IIRC, with Windows 7 there are newer APIs which could be used instead of
the select emulation using I/O threads.

> that allows this to work, but AFAIK that doesn't work with MinGW, only

Nope.  It is all specified by the Win32 API.  If you are interest in
details on this I suggest to read Hart's Windows System Programming.  I
consider this the Windows counterpart to APUE.


Shalom-Salam,

   Werner


-- 
Die Gedanken sind frei.  Ausnahmen regelt ein Bundesgesetz.

Re: pipe

[Eli Zaretskii]

> From: Werner Koch <wk@gnupg.org>
> Cc: ueno@gnu.org,  emacs-devel@gnu.org
> Date: Sat, 14 Mar 2015 12:51:00 +0100
> 
> On Sat, 14 Mar 2015 09:54, eliz@gnu.org said:
> 
> > Thanks, but where do I see the source of that wrapper?  I tried the
> > gpg Git repo, but I don't think I see it there.  Did I miss something?
> 
> It is in gpgme, a library interface to gnupg:
> 
> git://git.gnupg.org/gpgme.git

Thanks.

> > And anyway, how does this wrapper solve the problem of invoking gpg
> > with the --status-fd, --command-fd, and --attribute-fd options?  IOW,
> > how can the invoking program pass file descriptors on the gpg command
> > line, and be sure gpg will use the same files/devices as the caller?
> 
> Use gpgme ;-)
> 
> Simply use "--status-fd HANDLE" or if gpg expects a filename use the
> option --enable-special-filenames which interprets filename of the form
> "-&N" as the handle N.
> 
> Under Windows we use a HANDLE and not a file descriptor.

OK, if handles are used, then the rest is clear.  (Your original
message talked about file descriptors, which is what confused me.)

But that means it would be impossible to invoke gpg from a shell that
can redirect handles beyond the first 3, like some decent port of
Bash, is that correct?  Or is that where the wrapper comes into play,
in some way?

> > that allows this to work, but AFAIK that doesn't work with MinGW, only
> 
> Nope.  It is all specified by the Win32 API.

Again, I was talking about file descriptors; the Win32 API knows
nothing about those.  MS runtime passes the translation table between
handles and file descriptors through undocumented fields of
STARTUPINFO, but using that needs special startup code, which only
MSVC supplies, AFAIK.

> If you are interest in details on this I suggest to read Hart's
> Windows System Programming.  I consider this the Windows counterpart
> to APUE.

Thanks, I have that book in my possession since long ago, and studied
it already (and continue studying it).

Re: pipe

[Werner Koch]

On Sat, 14 Mar 2015 14:42, eliz@gnu.org said:

> But that means it would be impossible to invoke gpg from a shell that
> can redirect handles beyond the first 3, like some decent port of
> Bash, is that correct?  Or is that where the wrapper comes into play,

I don't know about a native Bash port to Windows.  Using Cygwin with
native Windows programs is problematic has you probably know.

> Again, I was talking about file descriptors; the Win32 API knows
> nothing about those.  MS runtime passes the translation table between
> handles and file descriptors through undocumented fields of

I never looked at the MS runtime this since I did the first
cross-compiler toolkit ~20 year ago.  I have not a found away to inherit
only specific handles without the wrapper trick.  How do they do that or
is the problem of inherited handles in the child ignored?


Salam-Shalom,

   Werner

-- 
Die Gedanken sind frei.  Ausnahmen regelt ein Bundesgesetz.

Re: pipe

[Eli Zaretskii]

> From: Werner Koch <wk@gnupg.org>
> Cc: ueno@gnu.org,  emacs-devel@gnu.org
> Date: Sat, 14 Mar 2015 20:28:01 +0100
> 
> On Sat, 14 Mar 2015 14:42, eliz@gnu.org said:
> 
> > But that means it would be impossible to invoke gpg from a shell that
> > can redirect handles beyond the first 3, like some decent port of
> > Bash, is that correct?  Or is that where the wrapper comes into play,
> 
> I don't know about a native Bash port to Windows.  Using Cygwin with
> native Windows programs is problematic has you probably know.

I was thinking about MSYS Bash.  But any port of Bash should do, as
redirecting file handles for a sub-process is trivial.

> > Again, I was talking about file descriptors; the Win32 API knows
> > nothing about those.  MS runtime passes the translation table between
> > handles and file descriptors through undocumented fields of
> 
> I never looked at the MS runtime this since I did the first
> cross-compiler toolkit ~20 year ago.  I have not a found away to inherit
> only specific handles without the wrapper trick.  How do they do that or
> is the problem of inherited handles in the child ignored?

That black magic is described here:

  http://www.catch22.net/tuts/undocumented-createprocess

[PATCH] Generalize start-process with keyword args

[Daiki Ueno]

[-- Attachment #1: Type: text/plain, Size: 583 bytes --]

Daiki Ueno <ueno@gnu.org> writes:

>> BTW, what happened with your earlier `make-process' suggestion?
>
> I'm still willing to work on that, but just wanted to be sure that my
> desired feature (additional pipes) could be implemented atop of it,

So here it is.  This merely adds a new function `make-process' as an
alternative form of `start-process', and use it in the `start-process'
implementation.  No extra feature has been added by now.

All tests succeeded with this, except file-notify-tests, which is
failing without the patch.  Reviews appreciated.

Thanks,
--
Daiki Ueno

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Generalize-start-process-with-keyword-args.patch --]
[-- Type: text/x-patch, Size: 13681 bytes --]

From 100b268e02cba94afa40948c54a7a3def47ce1d3 Mon Sep 17 00:00:00 2001
From: Daiki Ueno <ueno@gnu.org>
Date: Mon, 16 Mar 2015 11:38:05 +0900
Subject: [PATCH] Generalize start-process with keyword args

* src/process.c (Fmake_process): New function.
(Fstart_process): Define as a wrapper around Fmake_process.
(create_process, create_pty): Check p->pty_flag instead of
Vprocess_connection_type.
(syms_of_process): Register QCcommand, QCconnection_type,
and Smake_process.

* doc/lispref/processes.texi (Asynchronous Processes): Mention
`make-process'.
---
 doc/lispref/ChangeLog      |   4 ++
 doc/lispref/processes.texi |  48 +++++++++++++
 src/ChangeLog              |   9 +++
 src/process.c              | 168 +++++++++++++++++++++++++++++++++++++--------
 4 files changed, 199 insertions(+), 30 deletions(-)

diff --git a/doc/lispref/ChangeLog b/doc/lispref/ChangeLog
index 260656c..06d4630 100644
--- a/doc/lispref/ChangeLog
+++ b/doc/lispref/ChangeLog
@@ -1,3 +1,7 @@
+2015-03-16  Daiki Ueno  <ueno@gnu.org>
+
+	* processes.texi (Asynchronous Processes): Mention `make-process'.
+
 2015-03-09  Nicolas Petton <nicolas@petton.fr>
 
 	* sequences.texi (seq-into): Add documentation for the new
diff --git a/doc/lispref/processes.texi b/doc/lispref/processes.texi
index 177cd68..c386696 100644
--- a/doc/lispref/processes.texi
+++ b/doc/lispref/processes.texi
@@ -692,6 +692,54 @@ use the function @code{process-tty-name} (@pxref{Process
 Information}).
 @end defvar
 
+@defun make-process &rest args
+This function is like @code{start-process}, but takes keyword arguments.
+
+The arguments @var{args} are a list of keyword/argument pairs.
+Omitting a keyword is always equivalent to specifying it with value
+@code{nil}, except for @code{:coding} and @code{:connection-type}.
+Here are the meaningful keywords:
+
+@table @asis
+@item :name @var{name}
+Use the string @var{name} as the process name.  It is modified if
+necessary to make it unique.
+
+@item :buffer @var{buffer}
+Use @var{buffer} as the process buffer.
+
+@item :command @var{command}
+Use @var{command} as the command line of the process.  @var{command}
+is a list starting with the program name, followed by strings to give
+to program as arguments.
+
+@item :coding @var{coding}
+Use @var{coding} as the coding system for this process.  To specify
+different coding systems for decoding data from the connection and for
+encoding data sent to it, specify @code{(@var{decoding} .
+@var{encoding})} for @var{coding}.
+
+If you don't specify this keyword at all, the default
+is to determine the coding systems from the data.
+
+@item :connection-type @var{TYPE}
+Initialize the type of device used to communicate with the subprocess.
+
+If you don't specify this keyword at all, the default is the value of
+the @code{process-connection-type} variable.
+
+@item :noquery @var{query-flag}
+Initialize the process query flag to @var{query-flag}.
+@xref{Query Before Exit}.
+
+@item :filter @var{filter}
+Initialize the process filter to @var{filter}.
+
+@item :sentinel @var{sentinel}
+Initialize the process sentinel to @var{sentinel}.
+
+@end defun
+
 @node Deleting Processes
 @section Deleting Processes
 @cindex deleting processes
diff --git a/src/ChangeLog b/src/ChangeLog
index e328afc..d8ac459 100644
--- a/src/ChangeLog
+++ b/src/ChangeLog
@@ -1,3 +1,12 @@
+2015-03-16  Daiki Ueno  <ueno@gnu.org>
+
+	* process.c (Fmake_process): New function.
+	(Fstart_process): Define as a wrapper around Fmake_process.
+	(create_process, create_pty): Check p->pty_flag instead of
+	Vprocess_connection_type.
+	(syms_of_process): Register QCcommand, QCconnection_type, and
+	Smake_process.
+
 2015-03-15  Eli Zaretskii  <eliz@gnu.org>
 
 	* xdisp.c (handle_invisible_prop): Fix up it->position even when
diff --git a/src/process.c b/src/process.c
index 1d935ba..44b7a2d 100644
--- a/src/process.c
+++ b/src/process.c
@@ -1376,13 +1376,73 @@ syntax.
 usage: (start-process NAME BUFFER PROGRAM &rest PROGRAM-ARGS)  */)
   (ptrdiff_t nargs, Lisp_Object *args)
 {
-  Lisp_Object buffer, name, program, proc, current_dir, tem;
+  Lisp_Object plist[6];
+
+  plist[0] = QCname;
+  plist[1] = args[0];
+  plist[2] = QCbuffer;
+  plist[3] = args[1];
+  plist[4] = QCcommand;
+  plist[5] = Flist (nargs - 2, args + 2);
+
+  /* Don't need to GCPRO each element of PLIST, since PLIST will be
+     protected as a whole in Fmake_process.  */
+  return CALLMANY (Fmake_process, plist);
+}
+
+DEFUN ("make-process", Fmake_process, Smake_process, 0, MANY, 0,
+       doc: /* Start a program in a subprocess.  Return the process object for it.
+
+This is similar to `start-process', but arguments are specified as
+keyword/argument pairs.  The following arguments are defined:
+
+:name NAME -- NAME is name for process.  It is modified if necessary
+to make it unique.
+
+:buffer BUFFER -- BUFFER is the buffer (or buffer-name) to associate
+with the process.  Process output goes at end of that buffer, unless
+you specify an output stream or filter function to handle the output.
+BUFFER may be also nil, meaning that this process is not associated
+with any buffer.
+
+:command COMMAND -- COMMAND is a list starting with the program file
+name, followed by strings to give to the program as arguments.
+
+:coding CODING -- If CODING is a symbol, it specifies the coding
+system used for both reading and writing for this process.  If CODING
+is a cons (DECODING . ENCODING), DECODING is used for reading, and
+ENCODING is used for writing.
+
+:noquery BOOL -- When exiting Emacs, query the user if BOOL is nil and
+the process is running.  If BOOL is not given, query before exiting.
+
+:connection-type TYPE -- TYPE is control type of device used to
+communicate with subprocesses.  Values are nil to use a pipe, or t or
+`pty' to use a pty.
+
+:filter FILTER -- Install FILTER as the process filter.
+
+:sentinel SENTINEL -- Install SENTINEL as the process sentinel.
+
+usage: (make-process &rest ARGS)  */)
+  (ptrdiff_t nargs, Lisp_Object *args)
+{
+  Lisp_Object buffer, name, command, program, proc, contact, current_dir, tem;
   unsigned char **new_argv;
+  ptrdiff_t new_argc;
   ptrdiff_t i;
   ptrdiff_t count = SPECPDL_INDEX ();
+  struct gcpro gcpro1;
   USE_SAFE_ALLOCA;
 
-  buffer = args[1];
+  if (nargs == 0)
+    return Qnil;
+
+  /* Save arguments for process-contact and clone-process.  */
+  contact = Flist (nargs, args);
+  GCPRO1 (contact);
+
+  buffer = Fplist_get (contact, QCbuffer);
   if (!NILP (buffer))
     buffer = Fget_buffer_create (buffer);
 
@@ -1402,10 +1462,11 @@ usage: (start-process NAME BUFFER PROGRAM &rest PROGRAM-ARGS)  */)
     UNGCPRO;
   }
 
-  name = args[0];
+  name = Fplist_get (contact, QCname);
   CHECK_STRING (name);
 
-  program = args[2];
+  command = Fplist_get (contact, QCcommand);
+  program = XCAR (command);
 
   if (!NILP (program))
     CHECK_STRING (program);
@@ -1423,7 +1484,16 @@ usage: (start-process NAME BUFFER PROGRAM &rest PROGRAM-ARGS)  */)
   pset_buffer (XPROCESS (proc), buffer);
   pset_sentinel (XPROCESS (proc), Qinternal_default_process_sentinel);
   pset_filter (XPROCESS (proc), Qinternal_default_process_filter);
-  pset_command (XPROCESS (proc), Flist (nargs - 2, args + 2));
+  pset_command (XPROCESS (proc), Fcopy_sequence (command));
+
+  if (tem = Fplist_get (contact, QCnoquery), !NILP (tem))
+    XPROCESS (proc)->kill_without_query = 1;
+
+  tem = Fplist_member (contact, QCconnection_type);
+  if (!NILP (tem))
+    XPROCESS (proc)->pty_flag = !NILP (XCAR (tem));
+  else
+    XPROCESS (proc)->pty_flag = !NILP (Vprocess_connection_type);
 
 #ifdef HAVE_GNUTLS
   /* AKA GNUTLS_INITSTAGE(proc).  */
@@ -1453,15 +1523,33 @@ usage: (start-process NAME BUFFER PROGRAM &rest PROGRAM-ARGS)  */)
     Lisp_Object val, *args2;
     struct gcpro gcpro1, gcpro2;
 
-    val = Vcoding_system_for_read;
+    tem = Fplist_member (contact, QCcoding);
+    if (!NILP (tem) && (!CONSP (tem) || !CONSP (XCDR (tem))))
+      tem = Qnil;
+
+    val = Qnil;
+    if (!NILP (tem))
+      {
+	val = XCAR (XCDR (tem));
+	if (CONSP (val))
+	  val = XCAR (val);
+      }
+    else
+      val = Vcoding_system_for_read;
     if (NILP (val))
       {
-	SAFE_ALLOCA_LISP (args2, nargs + 1);
-	args2[0] = Qstart_process;
-	for (i = 0; i < nargs; i++) args2[i + 1] = args[i];
+	ptrdiff_t nargs2 = 3 + XINT (Flength (command));
+	Lisp_Object tem2;
+	SAFE_ALLOCA_LISP (args2, nargs2);
+	i = 0;
+	args2[i++] = Qstart_process;
+	args2[i++] = name;
+	args2[i++] = buffer;
+	for (tem2 = command; CONSP (tem2); tem2 = XCDR (tem2))
+	  args2[i++] = XCAR (tem2);
 	GCPRO2 (proc, current_dir);
 	if (!NILP (program))
-	  coding_systems = Ffind_operation_coding_system (nargs + 1, args2);
+	  coding_systems = Ffind_operation_coding_system (nargs2, args2);
 	UNGCPRO;
 	if (CONSP (coding_systems))
 	  val = XCAR (coding_systems);
@@ -1470,17 +1558,31 @@ usage: (start-process NAME BUFFER PROGRAM &rest PROGRAM-ARGS)  */)
       }
     pset_decode_coding_system (XPROCESS (proc), val);
 
-    val = Vcoding_system_for_write;
+    val = Qnil;
+    if (!NILP (tem))
+      {
+	val = XCAR (XCDR (tem));
+	if (CONSP (val))
+	  val = XCDR (val);
+      }
+    else
+      val = Vcoding_system_for_write;
     if (NILP (val))
       {
 	if (EQ (coding_systems, Qt))
 	  {
-	    SAFE_ALLOCA_LISP (args2, nargs + 1);
-	    args2[0] = Qstart_process;
-	    for (i = 0; i < nargs; i++) args2[i + 1] = args[i];
+	    ptrdiff_t nargs2 = 3 + XINT (Flength (command));
+	    Lisp_Object tem2;
+	    SAFE_ALLOCA_LISP (args2, nargs2);
+	    i = 0;
+	    args2[i++] = Qstart_process;
+	    args2[i++] = name;
+	    args2[i++] = buffer;
+	    for (tem2 = command; CONSP (tem2); tem2 = XCDR (tem2))
+	      args2[i++] = XCAR (tem2);
 	    GCPRO2 (proc, current_dir);
 	    if (!NILP (program))
-	      coding_systems = Ffind_operation_coding_system (nargs + 1, args2);
+	      coding_systems = Ffind_operation_coding_system (nargs2, args2);
 	    UNGCPRO;
 	  }
 	if (CONSP (coding_systems))
@@ -1512,10 +1614,10 @@ usage: (start-process NAME BUFFER PROGRAM &rest PROGRAM-ARGS)  */)
 	  && !(SCHARS (program) > 1
 	       && IS_DEVICE_SEP (SREF (program, 1))))
 	{
-	  struct gcpro gcpro1, gcpro2, gcpro3, gcpro4;
+	  struct gcpro gcpro1, gcpro2;
 
 	  tem = Qnil;
-	  GCPRO4 (name, program, buffer, current_dir);
+	  GCPRO2 (buffer, current_dir);
 	  openp (Vexec_path, program, Vexec_suffixes, &tem,
 		 make_number (X_OK), false);
 	  UNGCPRO;
@@ -1534,31 +1636,32 @@ usage: (start-process NAME BUFFER PROGRAM &rest PROGRAM-ARGS)  */)
       tem = remove_slash_colon (tem);
 
       {
-	Lisp_Object arg_encoding = Qnil;
+	Lisp_Object arg_encoding = Qnil, tem2;
 	struct gcpro gcpro1;
+
+	tem = Fcons (tem, Fcopy_sequence (XCDR (command)));
 	GCPRO1 (tem);
 
 	/* Encode the file name and put it in NEW_ARGV.
 	   That's where the child will use it to execute the program.  */
-	tem = list1 (ENCODE_FILE (tem));
+	XSETCAR (tem, ENCODE_FILE (XCAR (tem)));
 
 	/* Here we encode arguments by the coding system used for sending
 	   data to the process.  We don't support using different coding
 	   systems for encoding arguments and for encoding data sent to the
 	   process.  */
 
-	for (i = 3; i < nargs; i++)
+	for (tem2 = XCDR (tem); CONSP (tem2); tem2 = XCDR (tem2))
 	  {
-	    tem = Fcons (args[i], tem);
-	    CHECK_STRING (XCAR (tem));
-	    if (STRING_MULTIBYTE (XCAR (tem)))
+	    CHECK_STRING (XCAR (tem2));
+	    if (STRING_MULTIBYTE (XCAR (tem2)))
 	      {
 		if (NILP (arg_encoding))
 		  arg_encoding = (complement_process_encoding_system
 				  (XPROCESS (proc)->encode_coding_system));
-		XSETCAR (tem,
+		XSETCAR (tem2,
 			 code_convert_string_norecord
-			 (XCAR (tem), arg_encoding, 1));
+			 (XCAR (tem2), arg_encoding, 1));
 	      }
 	  }
 
@@ -1567,10 +1670,11 @@ usage: (start-process NAME BUFFER PROGRAM &rest PROGRAM-ARGS)  */)
 
       /* Now that everything is encoded we can collect the strings into
 	 NEW_ARGV.  */
-      SAFE_NALLOCA (new_argv, 1, nargs - 1);
-      new_argv[nargs - 2] = 0;
+      new_argc = XINT (Flength (tem));
+      SAFE_NALLOCA (new_argv, 1, new_argc + 1);
+      new_argv[new_argc] = 0;
 
-      for (i = nargs - 2; i-- != 0; )
+      for (i = 0; i < new_argc; i++)
 	{
 	  new_argv[i] = SDATA (XCAR (tem));
 	  tem = XCDR (tem);
@@ -1581,6 +1685,7 @@ usage: (start-process NAME BUFFER PROGRAM &rest PROGRAM-ARGS)  */)
   else
     create_pty (proc);
 
+  UNGCPRO;
   SAFE_FREE ();
   return unbind_to (count, proc);
 }
@@ -1648,7 +1753,7 @@ create_process (Lisp_Object process, char **new_argv, Lisp_Object current_dir)
 
   inchannel = outchannel = -1;
 
-  if (!NILP (Vprocess_connection_type))
+  if (p->pty_flag)
     outchannel = inchannel = allocate_pty (pty_name);
 
   if (inchannel >= 0)
@@ -1894,7 +1999,7 @@ create_pty (Lisp_Object process)
 {
   struct Lisp_Process *p = XPROCESS (process);
   char pty_name[PTY_NAME_SIZE];
-  int pty_fd = NILP (Vprocess_connection_type) ? -1 : allocate_pty (pty_name);
+  int pty_fd = !p->pty_flag ? -1 : allocate_pty (pty_name);
 
   if (pty_fd >= 0)
     {
@@ -7269,6 +7374,8 @@ syms_of_process (void)
   DEFSYM (QCstop, ":stop");
   DEFSYM (QCoptions, ":options");
   DEFSYM (QCplist, ":plist");
+  DEFSYM (QCcommand, ":command");
+  DEFSYM (QCconnection_type, ":connection-type");
 
   DEFSYM (Qlast_nonmenu_event, "last-nonmenu-event");
 
@@ -7372,6 +7479,7 @@ The variable takes effect when `start-process' is called.  */);
   defsubr (&Sset_process_plist);
   defsubr (&Sprocess_list);
   defsubr (&Sstart_process);
+  defsubr (&Smake_process);
   defsubr (&Sserial_process_configure);
   defsubr (&Smake_serial_process);
   defsubr (&Sset_network_process_option);
-- 
2.1.0

Re: [PATCH] Generalize start-process with keyword args

[Stefan Monnier]

> (Fstart_process): Define as a wrapper around Fmake_process.

Could we move this to Elisp?

> +Omitting a keyword is always equivalent to specifying it with value
> +@code{nil}, except for @code{:coding} and @code{:connection-type}.

Could we eliminate those two exceptions?


        Stefan

Re: Generalize start-process with keyword args

[Andy Moreton]

On Mon 16 Mar 2015, Daiki Ueno wrote:

> Daiki Ueno <ueno@gnu.org> writes:
>
>>> BTW, what happened with your earlier `make-process' suggestion?
>>
>> I'm still willing to work on that, but just wanted to be sure that my
>> desired feature (additional pipes) could be implemented atop of it,
>
> So here it is.  This merely adds a new function `make-process' as an
> alternative form of `start-process', and use it in the `start-process'
> implementation.  No extra feature has been added by now.
>
> All tests succeeded with this, except file-notify-tests, which is
> failing without the patch.  Reviews appreciated.

As you are extending code in this area, perhaps you could consider a
feature request.

start-process assumes that the system is capable of handling shebang
lines in shell scripts and launching them by executing a shell. This is
not supported by Windows, and so does not work in the native Windows
builds of emacs.

It would be useful to have a way to have start-process have an option to
start the new process via a shell, so that existing code that assumes
that it can use start-process with an executable shell script can also
be made to work on Windows.

    AndyM

Re: Generalize start-process with keyword args

[Eli Zaretskii]

> From: Andy Moreton <andrewjmoreton@gmail.com>
> Date: Mon, 16 Mar 2015 19:12:45 +0000
> 
> start-process assumes that the system is capable of handling shebang
> lines in shell scripts and launching them by executing a shell.

Can you point out where does this assumption live in the code?

> It would be useful to have a way to have start-process have an option to
> start the new process via a shell, so that existing code that assumes
> that it can use start-process with an executable shell script can also
> be made to work on Windows.

The Windows shells cannot run Unix shell script, so I'm not sure I
understand how would you like this to work.

Re: Generalize start-process with keyword args

[Andy Moreton]

On Mon 16 Mar 2015, Eli Zaretskii wrote:

>> From: Andy Moreton <andrewjmoreton@gmail.com>
>> Date: Mon, 16 Mar 2015 19:12:45 +0000
>> 
>> start-process assumes that the system is capable of handling shebang
>> lines in shell scripts and launching them by executing a shell.
>
> Can you point out where does this assumption live in the code?

In the callers of start-process, which assume that an executeable shell
script can be used as the command argument.

>> It would be useful to have a way to have start-process have an option to
>> start the new process via a shell, so that existing code that assumes
>> that it can use start-process with an executable shell script can also
>> be made to work on Windows.
>
> The Windows shells cannot run Unix shell script, so I'm not sure I
> understand how would you like this to work.

By (optionally) invoking the command via a shell. This allows
integration of unix style commands inplemented as executable shell
scripts with Win32 emacs. At the moment I handle this on a case by case
basis with advice, but it would be useful to allow a more general
wrapper.

    AndyM

Re: Generalize start-process with keyword args

[Stefan Monnier]

>> Can you point out where does this assumption live in the code?
> In the callers of start-process, which assume that an executeable shell
> script can be used as the command argument.

Could you be more concrete?

And give an example of such a script?


        Stefan

Re: [PATCH] Generalize start-process with keyword args

[Daiki Ueno]

[-- Attachment #1: Type: text/plain, Size: 1078 bytes --]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> (Fstart_process): Define as a wrapper around Fmake_process.
>
> Could we move this to Elisp?

Thanks for the comment.  I forgot that the initial patch did that.
Fixed in the attached patch.

>> +Omitting a keyword is always equivalent to specifying it with value
>> +@code{nil}, except for @code{:coding} and @code{:connection-type}.
>
> Could we eliminate those two exceptions?

I think `:coding' at least needs to be treated specially, because if it
is omitted, other methods to determine the coding system take place
(`coding-system-for-{read,write}' and `find-operation-coding-system').
This is the same behavior as `make-network-process and
`make-serial-process' and maybe it is not worth breaking the
consistency.

`:connection-type' is there because the default value of the counterpart
global variable `process-connection-type' is t (not nil).  So, yes, it
could be eliminated if it is given another name which means a negation,
say `:nopty'.  But I'm not sure if it is intuitive for users.

Regards,
-- 
Daiki Ueno

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Generalize-start-process-with-keyword-args.patch --]
[-- Type: text/x-patch, Size: 16763 bytes --]

From 76fd9ac3d6fa8d95b2cc63da3cec226533962285 Mon Sep 17 00:00:00 2001
From: Daiki Ueno <ueno@gnu.org>
Date: Mon, 16 Mar 2015 11:38:05 +0900
Subject: [PATCH] Generalize start-process with keyword args

* src/process.c (Fmake_process): New function.
(create_process, create_pty): Check p->pty_flag instead of
Vprocess_connection_type.
(syms_of_process): Register QCcommand, QCconnection_type,
and Smake_process.

* lisp/subr.el (start-process): New function, ported from the C
implementation.

* doc/lispref/processes.texi (Asynchronous Processes): Mention
`make-process'.
---
 doc/lispref/ChangeLog      |   4 ++
 doc/lispref/processes.texi |  55 +++++++++++++++
 lisp/ChangeLog             |   5 ++
 lisp/subr.el               |  22 ++++++
 src/ChangeLog              |   8 +++
 src/process.c              | 170 +++++++++++++++++++++++++++++++++------------
 6 files changed, 219 insertions(+), 45 deletions(-)

diff --git a/doc/lispref/ChangeLog b/doc/lispref/ChangeLog
index 260656c..06d4630 100644
--- a/doc/lispref/ChangeLog
+++ b/doc/lispref/ChangeLog
@@ -1,3 +1,7 @@
+2015-03-16  Daiki Ueno  <ueno@gnu.org>
+
+	* processes.texi (Asynchronous Processes): Mention `make-process'.
+
 2015-03-09  Nicolas Petton <nicolas@petton.fr>
 
 	* sequences.texi (seq-into): Add documentation for the new
diff --git a/doc/lispref/processes.texi b/doc/lispref/processes.texi
index 177cd68..42c658d 100644
--- a/doc/lispref/processes.texi
+++ b/doc/lispref/processes.texi
@@ -692,6 +692,61 @@ use the function @code{process-tty-name} (@pxref{Process
 Information}).
 @end defvar
 
+@defun make-process &rest args
+This function is like @code{start-process}, but takes keyword arguments.
+
+The arguments @var{args} are a list of keyword/argument pairs.
+Omitting a keyword is always equivalent to specifying it with value
+@code{nil}, except for @code{:coding} and @code{:connection-type}.
+Here are the meaningful keywords:
+
+@table @asis
+@item :name @var{name}
+Use the string @var{name} as the process name.  It is modified if
+necessary to make it unique.
+
+@item :buffer @var{buffer}
+Use @var{buffer} as the process buffer.
+
+@item :command @var{command}
+Use @var{command} as the command line of the process.  @var{command}
+is a list starting with the program name, followed by strings to give
+to program as arguments.
+
+@item :coding @var{coding}
+Use @var{coding} as the coding system for this process.  To specify
+different coding systems for decoding data from the connection and for
+encoding data sent to it, specify @code{(@var{decoding} .
+@var{encoding})} for @var{coding}.
+
+If you don't specify this keyword at all, the default
+is to determine the coding systems from the data.
+
+@item :connection-type @var{TYPE}
+Initialize the type of device used to communicate with the subprocess.
+
+If you don't specify this keyword at all, the default is the value of
+the @code{process-connection-type} variable.
+
+@item :stop @var{stopped}
+If @var{stopped} is non-@code{nil}, start the network connection or
+server in the ``stopped'' state.
+
+@item :noquery @var{query-flag}
+Initialize the process query flag to @var{query-flag}.
+@xref{Query Before Exit}.
+
+@item :filter @var{filter}
+Initialize the process filter to @var{filter}.
+
+@item :sentinel @var{sentinel}
+Initialize the process sentinel to @var{sentinel}.
+@end table
+
+The original argument list, modified with the actual connection
+information, is available via the @code{process-contact} function.
+@end defun
+
 @node Deleting Processes
 @section Deleting Processes
 @cindex deleting processes
diff --git a/lisp/ChangeLog b/lisp/ChangeLog
index 747a1d6..835712c 100644
--- a/lisp/ChangeLog
+++ b/lisp/ChangeLog
@@ -1,3 +1,8 @@
+2015-03-17  Daiki Ueno  <ueno@gnu.org>
+
+	* subr.el (start-process): New function, ported from the C
+	implementation.
+
 2015-03-15  Michael Albinus  <michael.albinus@gmx.de>
 
 	* net/tramp-adb.el:
diff --git a/lisp/subr.el b/lisp/subr.el
index deadca6..5d763f9 100644
--- a/lisp/subr.el
+++ b/lisp/subr.el
@@ -1901,6 +1901,28 @@ and the file name is displayed in the echo area."
 \f
 ;;;; Process stuff.
 
+(defun start-process (name buffer program &rest program-args)
+  "Start a program in a subprocess.  Return the process object for it.
+NAME is name for process.  It is modified if necessary to make it unique.
+BUFFER is the buffer (or buffer name) to associate with the process.
+
+Process output (both standard output and standard error streams) goes
+at end of BUFFER, unless you specify an output stream or filter
+function to handle the output.  BUFFER may also be nil, meaning that
+this process is not associated with any buffer.
+
+PROGRAM is the program file name.  It is searched for in `exec-path'
+\(which see).  If nil, just associate a pty with the buffer.  Remaining
+arguments are strings to give program as arguments.
+
+If you want to separate standard output from standard error, invoke
+the command through a shell and redirect one of them using the shell
+syntax."
+  (unless (fboundp 'make-process)
+    (error "Emacs was compiled without subprocess support"))
+  (make-process :name name :buffer buffer
+		:command (cons program program-args)))
+
 (defun process-lines (program &rest args)
   "Execute PROGRAM with ARGS, returning its output as a list of lines.
 Signal an error if the program returns with a non-zero exit status."
diff --git a/src/ChangeLog b/src/ChangeLog
index e328afc..2ec576a 100644
--- a/src/ChangeLog
+++ b/src/ChangeLog
@@ -1,3 +1,11 @@
+2015-03-16  Daiki Ueno  <ueno@gnu.org>
+
+	* process.c (Fmake_process): New function.
+	(create_process, create_pty): Check p->pty_flag instead of
+	Vprocess_connection_type.
+	(syms_of_process): Register QCcommand, QCconnection_type, and
+	Smake_process.  Unregister Sstart_process.
+
 2015-03-15  Eli Zaretskii  <eliz@gnu.org>
 
 	* xdisp.c (handle_invisible_prop): Fix up it->position even when
diff --git a/src/process.c b/src/process.c
index 1d935ba..49f5d5c 100644
--- a/src/process.c
+++ b/src/process.c
@@ -1355,34 +1355,65 @@ DEFUN ("process-list", Fprocess_list, Sprocess_list, 0, 0, 0,
 
 static void start_process_unwind (Lisp_Object proc);
 
-DEFUN ("start-process", Fstart_process, Sstart_process, 3, MANY, 0,
+DEFUN ("make-process", Fmake_process, Smake_process, 0, MANY, 0,
        doc: /* Start a program in a subprocess.  Return the process object for it.
-NAME is name for process.  It is modified if necessary to make it unique.
-BUFFER is the buffer (or buffer name) to associate with the process.
 
-Process output (both standard output and standard error streams) goes
-at end of BUFFER, unless you specify an output stream or filter
-function to handle the output.  BUFFER may also be nil, meaning that
-this process is not associated with any buffer.
+This is similar to `start-process', but arguments are specified as
+keyword/argument pairs.  The following arguments are defined:
 
-PROGRAM is the program file name.  It is searched for in `exec-path'
-(which see).  If nil, just associate a pty with the buffer.  Remaining
-arguments are strings to give program as arguments.
+:name NAME -- NAME is name for process.  It is modified if necessary
+to make it unique.
 
-If you want to separate standard output from standard error, invoke
-the command through a shell and redirect one of them using the shell
-syntax.
+:buffer BUFFER -- BUFFER is the buffer (or buffer-name) to associate
+with the process.  Process output goes at end of that buffer, unless
+you specify an output stream or filter function to handle the output.
+BUFFER may be also nil, meaning that this process is not associated
+with any buffer.
 
-usage: (start-process NAME BUFFER PROGRAM &rest PROGRAM-ARGS)  */)
+:command COMMAND -- COMMAND is a list starting with the program file
+name, followed by strings to give to the program as arguments.
+
+:coding CODING -- If CODING is a symbol, it specifies the coding
+system used for both reading and writing for this process.  If CODING
+is a cons (DECODING . ENCODING), DECODING is used for reading, and
+ENCODING is used for writing.
+
+:noquery BOOL -- When exiting Emacs, query the user if BOOL is nil and
+the process is running.  If BOOL is not given, query before exiting.
+
+:stop BOOL -- Start process in the `stopped' state if BOOL non-nil.
+In the stopped state, a server process does not accept new
+connections, and a client process does not handle incoming traffic.
+The stopped state is cleared by `continue-process' and set by
+`stop-process'.
+
+:connection-type TYPE -- TYPE is control type of device used to
+communicate with subprocesses.  Values are nil to use a pipe, or t or
+`pty' to use a pty.
+
+:filter FILTER -- Install FILTER as the process filter.
+
+:sentinel SENTINEL -- Install SENTINEL as the process sentinel.
+
+usage: (make-process &rest ARGS)  */)
   (ptrdiff_t nargs, Lisp_Object *args)
 {
-  Lisp_Object buffer, name, program, proc, current_dir, tem;
+  Lisp_Object buffer, name, command, program, proc, contact, current_dir, tem;
   unsigned char **new_argv;
+  ptrdiff_t new_argc;
   ptrdiff_t i;
   ptrdiff_t count = SPECPDL_INDEX ();
+  struct gcpro gcpro1;
   USE_SAFE_ALLOCA;
 
-  buffer = args[1];
+  if (nargs == 0)
+    return Qnil;
+
+  /* Save arguments for process-contact and clone-process.  */
+  contact = Flist (nargs, args);
+  GCPRO1 (contact);
+
+  buffer = Fplist_get (contact, QCbuffer);
   if (!NILP (buffer))
     buffer = Fget_buffer_create (buffer);
 
@@ -1402,10 +1433,11 @@ usage: (start-process NAME BUFFER PROGRAM &rest PROGRAM-ARGS)  */)
     UNGCPRO;
   }
 
-  name = args[0];
+  name = Fplist_get (contact, QCname);
   CHECK_STRING (name);
 
-  program = args[2];
+  command = Fplist_get (contact, QCcommand);
+  program = XCAR (command);
 
   if (!NILP (program))
     CHECK_STRING (program);
@@ -1423,7 +1455,18 @@ usage: (start-process NAME BUFFER PROGRAM &rest PROGRAM-ARGS)  */)
   pset_buffer (XPROCESS (proc), buffer);
   pset_sentinel (XPROCESS (proc), Qinternal_default_process_sentinel);
   pset_filter (XPROCESS (proc), Qinternal_default_process_filter);
-  pset_command (XPROCESS (proc), Flist (nargs - 2, args + 2));
+  pset_command (XPROCESS (proc), Fcopy_sequence (command));
+
+  if (tem = Fplist_get (contact, QCnoquery), !NILP (tem))
+    XPROCESS (proc)->kill_without_query = 1;
+  if (tem = Fplist_get (contact, QCstop), !NILP (tem))
+    pset_command (XPROCESS (proc), Qt);
+
+  tem = Fplist_member (contact, QCconnection_type);
+  if (!NILP (tem))
+    XPROCESS (proc)->pty_flag = !NILP (XCAR (XCDR (tem)));
+  else
+    XPROCESS (proc)->pty_flag = !NILP (Vprocess_connection_type);
 
 #ifdef HAVE_GNUTLS
   /* AKA GNUTLS_INITSTAGE(proc).  */
@@ -1453,15 +1496,33 @@ usage: (start-process NAME BUFFER PROGRAM &rest PROGRAM-ARGS)  */)
     Lisp_Object val, *args2;
     struct gcpro gcpro1, gcpro2;
 
-    val = Vcoding_system_for_read;
+    tem = Fplist_member (contact, QCcoding);
+    if (!NILP (tem) && (!CONSP (tem) || !CONSP (XCDR (tem))))
+      tem = Qnil;
+
+    val = Qnil;
+    if (!NILP (tem))
+      {
+	val = XCAR (XCDR (tem));
+	if (CONSP (val))
+	  val = XCAR (val);
+      }
+    else
+      val = Vcoding_system_for_read;
     if (NILP (val))
       {
-	SAFE_ALLOCA_LISP (args2, nargs + 1);
-	args2[0] = Qstart_process;
-	for (i = 0; i < nargs; i++) args2[i + 1] = args[i];
+	ptrdiff_t nargs2 = 3 + XINT (Flength (command));
+	Lisp_Object tem2;
+	SAFE_ALLOCA_LISP (args2, nargs2);
+	i = 0;
+	args2[i++] = Qstart_process;
+	args2[i++] = name;
+	args2[i++] = buffer;
+	for (tem2 = command; CONSP (tem2); tem2 = XCDR (tem2))
+	  args2[i++] = XCAR (tem2);
 	GCPRO2 (proc, current_dir);
 	if (!NILP (program))
-	  coding_systems = Ffind_operation_coding_system (nargs + 1, args2);
+	  coding_systems = Ffind_operation_coding_system (nargs2, args2);
 	UNGCPRO;
 	if (CONSP (coding_systems))
 	  val = XCAR (coding_systems);
@@ -1470,17 +1531,31 @@ usage: (start-process NAME BUFFER PROGRAM &rest PROGRAM-ARGS)  */)
       }
     pset_decode_coding_system (XPROCESS (proc), val);
 
-    val = Vcoding_system_for_write;
+    val = Qnil;
+    if (!NILP (tem))
+      {
+	val = XCAR (XCDR (tem));
+	if (CONSP (val))
+	  val = XCDR (val);
+      }
+    else
+      val = Vcoding_system_for_write;
     if (NILP (val))
       {
 	if (EQ (coding_systems, Qt))
 	  {
-	    SAFE_ALLOCA_LISP (args2, nargs + 1);
-	    args2[0] = Qstart_process;
-	    for (i = 0; i < nargs; i++) args2[i + 1] = args[i];
+	    ptrdiff_t nargs2 = 3 + XINT (Flength (command));
+	    Lisp_Object tem2;
+	    SAFE_ALLOCA_LISP (args2, nargs2);
+	    i = 0;
+	    args2[i++] = Qstart_process;
+	    args2[i++] = name;
+	    args2[i++] = buffer;
+	    for (tem2 = command; CONSP (tem2); tem2 = XCDR (tem2))
+	      args2[i++] = XCAR (tem2);
 	    GCPRO2 (proc, current_dir);
 	    if (!NILP (program))
-	      coding_systems = Ffind_operation_coding_system (nargs + 1, args2);
+	      coding_systems = Ffind_operation_coding_system (nargs2, args2);
 	    UNGCPRO;
 	  }
 	if (CONSP (coding_systems))
@@ -1512,10 +1587,10 @@ usage: (start-process NAME BUFFER PROGRAM &rest PROGRAM-ARGS)  */)
 	  && !(SCHARS (program) > 1
 	       && IS_DEVICE_SEP (SREF (program, 1))))
 	{
-	  struct gcpro gcpro1, gcpro2, gcpro3, gcpro4;
+	  struct gcpro gcpro1, gcpro2;
 
 	  tem = Qnil;
-	  GCPRO4 (name, program, buffer, current_dir);
+	  GCPRO2 (buffer, current_dir);
 	  openp (Vexec_path, program, Vexec_suffixes, &tem,
 		 make_number (X_OK), false);
 	  UNGCPRO;
@@ -1534,31 +1609,32 @@ usage: (start-process NAME BUFFER PROGRAM &rest PROGRAM-ARGS)  */)
       tem = remove_slash_colon (tem);
 
       {
-	Lisp_Object arg_encoding = Qnil;
+	Lisp_Object arg_encoding = Qnil, tem2;
 	struct gcpro gcpro1;
+
+	tem = Fcons (tem, Fcopy_sequence (XCDR (command)));
 	GCPRO1 (tem);
 
 	/* Encode the file name and put it in NEW_ARGV.
 	   That's where the child will use it to execute the program.  */
-	tem = list1 (ENCODE_FILE (tem));
+	XSETCAR (tem, ENCODE_FILE (XCAR (tem)));
 
 	/* Here we encode arguments by the coding system used for sending
 	   data to the process.  We don't support using different coding
 	   systems for encoding arguments and for encoding data sent to the
 	   process.  */
 
-	for (i = 3; i < nargs; i++)
+	for (tem2 = XCDR (tem); CONSP (tem2); tem2 = XCDR (tem2))
 	  {
-	    tem = Fcons (args[i], tem);
-	    CHECK_STRING (XCAR (tem));
-	    if (STRING_MULTIBYTE (XCAR (tem)))
+	    CHECK_STRING (XCAR (tem2));
+	    if (STRING_MULTIBYTE (XCAR (tem2)))
 	      {
 		if (NILP (arg_encoding))
 		  arg_encoding = (complement_process_encoding_system
 				  (XPROCESS (proc)->encode_coding_system));
-		XSETCAR (tem,
+		XSETCAR (tem2,
 			 code_convert_string_norecord
-			 (XCAR (tem), arg_encoding, 1));
+			 (XCAR (tem2), arg_encoding, 1));
 	      }
 	  }
 
@@ -1567,10 +1643,11 @@ usage: (start-process NAME BUFFER PROGRAM &rest PROGRAM-ARGS)  */)
 
       /* Now that everything is encoded we can collect the strings into
 	 NEW_ARGV.  */
-      SAFE_NALLOCA (new_argv, 1, nargs - 1);
-      new_argv[nargs - 2] = 0;
+      new_argc = XINT (Flength (tem));
+      SAFE_NALLOCA (new_argv, 1, new_argc + 1);
+      new_argv[new_argc] = 0;
 
-      for (i = nargs - 2; i-- != 0; )
+      for (i = 0; i < new_argc; i++)
 	{
 	  new_argv[i] = SDATA (XCAR (tem));
 	  tem = XCDR (tem);
@@ -1581,6 +1658,7 @@ usage: (start-process NAME BUFFER PROGRAM &rest PROGRAM-ARGS)  */)
   else
     create_pty (proc);
 
+  UNGCPRO;
   SAFE_FREE ();
   return unbind_to (count, proc);
 }
@@ -1648,7 +1726,7 @@ create_process (Lisp_Object process, char **new_argv, Lisp_Object current_dir)
 
   inchannel = outchannel = -1;
 
-  if (!NILP (Vprocess_connection_type))
+  if (p->pty_flag)
     outchannel = inchannel = allocate_pty (pty_name);
 
   if (inchannel >= 0)
@@ -1894,7 +1972,7 @@ create_pty (Lisp_Object process)
 {
   struct Lisp_Process *p = XPROCESS (process);
   char pty_name[PTY_NAME_SIZE];
-  int pty_fd = NILP (Vprocess_connection_type) ? -1 : allocate_pty (pty_name);
+  int pty_fd = !p->pty_flag ? -1 : allocate_pty (pty_name);
 
   if (pty_fd >= 0)
     {
@@ -7269,6 +7347,8 @@ syms_of_process (void)
   DEFSYM (QCstop, ":stop");
   DEFSYM (QCoptions, ":options");
   DEFSYM (QCplist, ":plist");
+  DEFSYM (QCcommand, ":command");
+  DEFSYM (QCconnection_type, ":connection-type");
 
   DEFSYM (Qlast_nonmenu_event, "last-nonmenu-event");
 
@@ -7371,7 +7451,7 @@ The variable takes effect when `start-process' is called.  */);
   defsubr (&Sprocess_plist);
   defsubr (&Sset_process_plist);
   defsubr (&Sprocess_list);
-  defsubr (&Sstart_process);
+  defsubr (&Smake_process);
   defsubr (&Sserial_process_configure);
   defsubr (&Smake_serial_process);
   defsubr (&Sset_network_process_option);
-- 
2.1.0

Re: [PATCH] Generalize start-process with keyword args

[Stefan Monnier]

> I think `:coding' at least needs to be treated specially, because if it
> is omitted, other methods to determine the coding system take place
> (`coding-system-for-{read,write}' and `find-operation-coding-system').

Why can't we make it use the same methods when ":coding nil" is specified?

> `:connection-type' is there because the default value of the counterpart
> global variable `process-connection-type' is t (not nil).  So, yes, it
> could be eliminated if it is given another name which means a negation,
> say `:nopty'.  But I'm not sure if it is intuitive for users.

I always found process-connection-type hard to remember.  I think it
should either have been named process-connection-use-pty, or its values
should have been pty/pipe.  So I think for :connection-type, it would
make sense to require the use of either `pipe' or `pty' (or nil to use
the default rules).


        Stefan

Re: [PATCH] Generalize start-process with keyword args

[Daiki Ueno]

Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

>> I think `:coding' at least needs to be treated specially, because if it
>> is omitted, other methods to determine the coding system take place
>> (`coding-system-for-{read,write}' and `find-operation-coding-system').
>
> Why can't we make it use the same methods when ":coding nil" is specified?

That would mean that the user explicitly specified no-conversion here.

From the comment in setup_coding_system, which is used by
setup_process_coding_systems:

  /* Setup coding context CODING from information about CODING_SYSTEM.
     If CODING_SYSTEM is nil, `no-conversion' is assumed.  If
     CODING_SYSTEM is invalid, signal an error.  */

> I always found process-connection-type hard to remember.  I think it
> should either have been named process-connection-use-pty, or its values
> should have been pty/pipe.  So I think for :connection-type, it would
> make sense to require the use of either `pipe' or `pty' (or nil to use
> the default rules).

Ah, yes, that's a good idea.

Regards,
-- 
Daiki Ueno

Re: Generalize start-process with keyword args

[Eli Zaretskii]

> From: Andy Moreton <andrewjmoreton@gmail.com>
> Date: Mon, 16 Mar 2015 22:27:44 +0000
> 
> On Mon 16 Mar 2015, Eli Zaretskii wrote:
> 
> >> From: Andy Moreton <andrewjmoreton@gmail.com>
> >> Date: Mon, 16 Mar 2015 19:12:45 +0000
> >> 
> >> start-process assumes that the system is capable of handling shebang
> >> lines in shell scripts and launching them by executing a shell.
> >
> > Can you point out where does this assumption live in the code?
> 
> In the callers of start-process

Can you give me an example of such a caller?  (I'm not asking idle
questions here; there are aspects of the problem you describe that
aren't really clear to me, but I prefer finding out the answers by
looking at the code than by asking too many questions.)

> which assume that an executeable shell script can be used as the
> command argument.

How do they decide that a shell script is executable?

> >> It would be useful to have a way to have start-process have an option to
> >> start the new process via a shell, so that existing code that assumes
> >> that it can use start-process with an executable shell script can also
> >> be made to work on Windows.
> >
> > The Windows shells cannot run Unix shell script, so I'm not sure I
> > understand how would you like this to work.
> 
> By (optionally) invoking the command via a shell.

But what shell would be able to interpret such scripts on Windows?
AFAIK, there are no good candidates for that role.

> This allows integration of unix style commands inplemented as
> executable shell scripts with Win32 emacs.

IMO, those shell scripts should be ported to Windows by converting
them to batch files that the stock Windows shell can interpret.

Re: pipe

[Daiki Ueno]

Eli Zaretskii <eliz@gnu.org> writes:

>> > But that means it would be impossible to invoke gpg from a shell that
>> > can redirect handles beyond the first 3, like some decent port of
>> > Bash, is that correct?  Or is that where the wrapper comes into play,
>> 
>> I don't know about a native Bash port to Windows.  Using Cygwin with
>> native Windows programs is problematic has you probably know.
>
> I was thinking about MSYS Bash.  But any port of Bash should do, as
> redirecting file handles for a sub-process is trivial.

Thanks for all the information.  I'm almost lost :-)

Are the following assumptions correct?

- A subprocess on Windows is able to access a handle passed as an
  external representation from the parent, as described in the first
  example of:
  https://msdn.microsoft.com/en-us/library/aa298531%28v=vs.60%29.aspx

- Some shells maintain the mappings between file descriptors and
  handles, and if Emacs runs a shell program with a redirection to
  non-standard file descriptors, it might not work as expected

- Emacs uses Gnulib's pipe2 implementation around _pipe on native
  Windows and the returned file descriptors are actually a handle

If all of those are correct, perhaps we could just mention the fact that
the child ends of a pipe process is a handle on Windows?

In any case, supporting stderr could be a starting point, as it seems to
be a long-standing request:
https://lists.gnu.org/archive/html/emacs-devel/2004-04/msg01051.html
and it wouldn't involve a portability issue.

Regards,
-- 
Daiki Ueno

Re: [PATCH] Generalize start-process with keyword args

[Eli Zaretskii]

> From: Daiki Ueno <ueno@gnu.org>
> Date: Tue, 17 Mar 2015 11:16:49 +0900
> Cc: emacs-devel@gnu.org
> 
> +@item :command @var{command}
> +Use @var{command} as the command line of the process.  @var{command}
> +is a list starting with the program name, followed by strings to give
> +to program as arguments.

I think it'd be good to clearly state here that "the program name" is
actually the name of the program's executable file.

> +@item :coding @var{coding}
> +Use @var{coding} as the coding system for this process.  To specify
> +different coding systems for decoding data from the connection and for
> +encoding data sent to it, specify @code{(@var{decoding} .
> +@var{encoding})} for @var{coding}.

FWIW, I like the doc string's wording better:

  If @var{coding} is a symbol, it specifies the coding system to be
  used for both reading and writing of data from and to the
  connection.  If @var{coding} is a cons cell
  @w{@code{(@var{decoding} . @var{encoding})}}, then @var{decoding}
  will be used for reading and @var{encoding} for writing.

> +If you don't specify this keyword at all, the default
> +is to determine the coding systems from the data.

Please mention 'undecided' explicitly here, since that is more
specific than the more vague "determine from the data".

Thanks.

Re: pipe

[Eli Zaretskii]

> From: Daiki Ueno <ueno@gnu.org>
> Cc: Werner Koch <wk@gnupg.org>,  emacs-devel@gnu.org
> Date: Tue, 17 Mar 2015 16:22:07 +0900
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> > But that means it would be impossible to invoke gpg from a shell that
> >> > can redirect handles beyond the first 3, like some decent port of
> >> > Bash, is that correct?  Or is that where the wrapper comes into play,
> >> 
> >> I don't know about a native Bash port to Windows.  Using Cygwin with
> >> native Windows programs is problematic has you probably know.
> >
> > I was thinking about MSYS Bash.  But any port of Bash should do, as
> > redirecting file handles for a sub-process is trivial.
> 
> Thanks for all the information.  I'm almost lost :-)

Welcome to the club ;-)

> Are the following assumptions correct?
> 
> - A subprocess on Windows is able to access a handle passed as an
>   external representation from the parent, as described in the first
>   example of:
>   https://msdn.microsoft.com/en-us/library/aa298531%28v=vs.60%29.aspx

Yes.  (But what is passed on the command line is not the HANDLE, it's
the file descriptor; the corresponding HANDLE is passed behind the
scenes, AFAIK, using the undocumented fields of the STARTUPINFO
structure.)

Please note that the magic that passes the inherited HANDLEs to the
child process is in the Windows C runtime, as part of the
implementation of the "spawn*" family of functions.  Emacs bypasses
"spawn*" and has its own implementation of 'spawnve', which doesn't
include the equivalent code.  So if we want Emacs to be able to use
pipes that use file descriptors beyond the 3 standard ones, we will
have to add the code which passes the corresponding HANDLEs.

> - Some shells maintain the mappings between file descriptors and
>   handles, and if Emacs runs a shell program with a redirection to
>   non-standard file descriptors, it might not work as expected

I don't understand what you are saying here.  Please elaborate, or
give an example.

What I meant is this:

  . Werner says that the --status-fd etc. options need to have HANDLEs
    as arguments on Windows, not file descriptors.

  . Invoking gpg from Bash with the likes of ">&5" doesn't provide a
    way of finding out the HANDLE that corresponds to the file
    descriptor 5, and so I'm unsure how can the gpg command line be
    constructed using HANDLE values instead of descriptors.

The mapping between file descriptors and the corresponding HANDLEs is
maintained by the C runtime for each process, and there are 2 library
functions to get the HANDLE given a descriptor, and a descriptor for a
handle.  But I'm not aware of a way of getting these mappings from the
shell's prompt.

> - Emacs uses Gnulib's pipe2 implementation around _pipe on native
>   Windows and the returned file descriptors are actually a handle

No, Emacs uses its own implementation of pipe2 (see w32.c), which
returns file descriptors.  (And I think you are mistaken about the
Gnulib implementation: it also returns file descriptors, not HANDLEs,
because the '_pipe' call returns descriptors.)

> If all of those are correct, perhaps we could just mention the fact that
> the child ends of a pipe process is a handle on Windows?

Not all of those are correct, but I'm unsure what the implications
are, see above.

> In any case, supporting stderr could be a starting point, as it seems to
> be a long-standing request:
> https://lists.gnu.org/archive/html/emacs-devel/2004-04/msg01051.html
> and it wouldn't involve a portability issue.

Redirecting the child's stderr is already supported on Windows, so
this is only a matter of having the higher layers DTRT.

Re: [PATCH] Generalize start-process with keyword args

[Stefan Monnier]

>> Why can't we make it use the same methods when ":coding nil" is specified?
> That would mean that the user explicitly specified no-conversion here.

Bad idea.  If she wants `no-conversion' (a bad name, for sure, she
should say `binary' instead), then she should say so explicitly rather
than by using nil.


        Stefan

Re: [PATCH] Generalize start-process with keyword args

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Tue, 17 Mar 2015 11:35:56 -0400
> Cc: emacs-devel@gnu.org
> 
> >> Why can't we make it use the same methods when ":coding nil" is specified?
> > That would mean that the user explicitly specified no-conversion here.
> 
> Bad idea.  If she wants `no-conversion' (a bad name, for sure, she
> should say `binary' instead), then she should say so explicitly rather
> than by using nil.

Unfortunately, that ship sailed long ago.

Re: [PATCH] Generalize start-process with keyword args

[Stefan Monnier]

>> Bad idea.  If she wants `no-conversion' (a bad name, for sure, she
>> should say `binary' instead), then she should say so explicitly rather
>> than by using nil.
> Unfortunately, that ship sailed long ago.

Not for the new `make-process' function.


        Stefan

Re: [PATCH] Generalize start-process with keyword args

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: ueno@gnu.org,  emacs-devel@gnu.org
> Date: Tue, 17 Mar 2015 14:08:53 -0400
> 
> >> Bad idea.  If she wants `no-conversion' (a bad name, for sure, she
> >> should say `binary' instead), then she should say so explicitly rather
> >> than by using nil.
> > Unfortunately, that ship sailed long ago.
> 
> Not for the new `make-process' function.

What, you mean it will implement its own encoding and decoding
machinery?

Re: Generalize start-process with keyword args

[Andy Moreton]

On Tue 17 Mar 2015, Eli Zaretskii wrote:

>> From: Andy Moreton <andrewjmoreton@gmail.com>
>> Date: Mon, 16 Mar 2015 22:27:44 +0000
>> 
>> On Mon 16 Mar 2015, Eli Zaretskii wrote:
>> 
>> >> From: Andy Moreton <andrewjmoreton@gmail.com>
>> >> Date: Mon, 16 Mar 2015 19:12:45 +0000
>> >> 
>> >> start-process assumes that the system is capable of handling shebang
>> >> lines in shell scripts and launching them by executing a shell.
>> >
>> > Can you point out where does this assumption live in the code?
>> 
>> In the callers of start-process
>
> Can you give me an example of such a caller?  (I'm not asking idle
> questions here; there are aspects of the problem you describe that
> aren't really clear to me, but I prefer finding out the answers by
> looking at the code than by asking too many questions.)

Sorry for not being clear.

A simple example is using the cscope package from Cygwin. This installs
a cscope executable, a cscope-indexer script and cscope.el that uses the
cscope indexes for symbol lookup. cscope.el can also invoke the indexer
script to regenerate the indexes (by invoking `cscope-indexing-script'
with arguments via `start-process').

This script can easily be reused without modification, by advising
`start-process to rework its arguments: cahnge PROGRAM to be "bash" and
prepend "-c" to the PROGRAM-ARGS list. This allows cscope.el to invoke
the original script from a Win32 emacs build without needing any changes
to the original package.

Similar issues arise with using other tools (e.g. git, hg) that use
shell scripts or symlinks for some executable tools.

>> By (optionally) invoking the command via a shell.
>
> But what shell would be able to interpret such scripts on Windows?
> AFAIK, there are no good candidates for that role.

A shell from cygwin, msys2 or msys.

>> This allows integration of unix style commands inplemented as
>> executable shell scripts with Win32 emacs.
>
> IMO, those shell scripts should be ported to Windows by converting
> them to batch files that the stock Windows shell can interpret.

Far more work (and more error prone) than simply invoking the existing
script via an appropriate (posix) shell. 

    AndyM

Re: Generalize start-process with keyword args

[Eli Zaretskii]

> From: Andy Moreton <andrewjmoreton@gmail.com>
> Date: Tue, 17 Mar 2015 20:55:28 +0000
> 
> A simple example is using the cscope package from Cygwin. This installs
> a cscope executable, a cscope-indexer script and cscope.el that uses the
> cscope indexes for symbol lookup. cscope.el can also invoke the indexer
> script to regenerate the indexes (by invoking `cscope-indexing-script'
> with arguments via `start-process').
> 
> This script can easily be reused without modification, by advising
> `start-process to rework its arguments: cahnge PROGRAM to be "bash" and
> prepend "-c" to the PROGRAM-ARGS list. This allows cscope.el to invoke
> the original script from a Win32 emacs build without needing any changes
> to the original package.

So how you suggest this to work?  Should such scripts be treated as
executable only if the interpreter they require is on PATH, and
otherwise be treated as non-executable?  Or should we always treat
them as executables, and let them fail if the interpreter is not
available?  Or should we require that a script that says

  #! /usr/local/bin/perl

is run by "/usr/local/bin/perl", and nothing else?

What about scripts that say

  #! /usr/bin/env perl

should we run 'env' for them, too?

And what about MSYS mounting of Windows directories, whereby /bin/bash
resolves to something like C:\MSYS\1.0\bin\bash.exe (of which Emacs
has no idea)?

I see a lot of complications with the semantics of this, and no easy
solutions, since the necessary tools are not available on Windows,
could be installed in any directory, and come with a set of
assumptions that don't necessarily play well with a native Emacs.

> > But what shell would be able to interpret such scripts on Windows?
> > AFAIK, there are no good candidates for that role.
> 
> A shell from cygwin, msys2 or msys.

They are all subtly incompatible with a native Emacs, both in the
file-name syntax and in how they treat I/O.

> >> This allows integration of unix style commands inplemented as
> >> executable shell scripts with Win32 emacs.
> >
> > IMO, those shell scripts should be ported to Windows by converting
> > them to batch files that the stock Windows shell can interpret.
> 
> Far more work (and more error prone) than simply invoking the existing
> script via an appropriate (posix) shell. 

The same can be said about porting any program.  Porting shell scripts
is part of the job, IMO.

Re: [PATCH] Generalize start-process with keyword args

[Stefan Monnier]

> What, you mean it will implement its own encoding and decoding
> machinery?

No, just that it will treat ":coding nil" in the same way as the absence
of a :coding arg (i.e. fallback on some default mechanism).


        Stefan

Re: Generalize start-process with keyword args

[Stefan Monnier]

> A simple example is using the cscope package from Cygwin. This installs
> a cscope executable, a cscope-indexer script and cscope.el that uses the
> cscope indexes for symbol lookup. cscope.el can also invoke the indexer
> script to regenerate the indexes (by invoking `cscope-indexing-script'
> with arguments via `start-process').

> This script can easily be reused without modification, by advising
> `start-process to rework its arguments: cahnge PROGRAM to be "bash" and
> prepend "-c" to the PROGRAM-ARGS list. This allows cscope.el to invoke
> the original script from a Win32 emacs build without needing any changes
> to the original package.

But there is no need for such changes if you use the Cygwin version of
Emacs, right?

The root of the problem is that you're trying to run a Cygwin program
from the W32 build of Emacs, while this Cygwin program was designed for
Cygwin-only (you can't run that script directly from the normal W32
command prompt, for example).

Of course, the W32 build of Emacs could try to accommodate Cygwin
programs a bit better, but it seems like it's probably easier to just
use the Cygwin build of Emacs instead (especially now that there's
a Cygwin-W32 build which uses the native GUI).


        Stefan

Re: Generalize start-process with keyword args

[Andy Moreton]

On Wed 18 Mar 2015, Eli Zaretskii wrote:

>> From: Andy Moreton <andrewjmoreton@gmail.com>
>> Date: Tue, 17 Mar 2015 20:55:28 +0000
>> 
>> A simple example is using the cscope package from Cygwin. This installs
>> a cscope executable, a cscope-indexer script and cscope.el that uses the
>> cscope indexes for symbol lookup. cscope.el can also invoke the indexer
>> script to regenerate the indexes (by invoking `cscope-indexing-script'
>> with arguments via `start-process').
>> 
>> This script can easily be reused without modification, by advising
>> `start-process to rework its arguments: cahnge PROGRAM to be "bash" and
>> prepend "-c" to the PROGRAM-ARGS list. This allows cscope.el to invoke
>> the original script from a Win32 emacs build without needing any changes
>> to the original package.
>
> So how you suggest this to work?  Should such scripts be treated as
> executable only if the interpreter they require is on PATH, and
> otherwise be treated as non-executable?  Or should we always treat
> them as executables, and let them fail if the interpreter is not
> available?  Or should we require that a script that says
>
>   #! /usr/local/bin/perl
>
> is run by "/usr/local/bin/perl", and nothing else?
>
> What about scripts that say
>
>   #! /usr/bin/env perl
>
> should we run 'env' for them, too?
>
> And what about MSYS mounting of Windows directories, whereby /bin/bash
> resolves to something like C:\MSYS\1.0\bin\bash.exe (of which Emacs
> has no idea)?
>
> I see a lot of complications with the semantics of this, and no easy
> solutions, since the necessary tools are not available on Windows,
> could be installed in any directory, and come with a set of
> assumptions that don't necessarily play well with a native Emacs.

I agree that this is awkward. What I envisaged was an (optional) alist
of (REGEXP . (INTERPRETER INTERPRETER-ARGS)) pairs such that if
start-process is onvoked for a PROGRAM that matches REGEXP, then use
INTERPRETER as the PROGRAM argument to start-process, and prepend
INTERPRETER-ARGS to the PROGRAM-ARGS list.

E.g. (start-process name buffer "/path/to/script" "foo" "bar" "baz")

and assuming something like:
(setq start-process-interpreter-alist
      '(("/path/to/script" . ("/path/to/shell" "-c")))

would result in behaviour equivalent to:
  (start-process name buffer "/path/to/shell" "-c" "/path/to/script"
                 "foo" "bar" "baz")

>> > But what shell would be able to interpret such scripts on Windows?
>> > AFAIK, there are no good candidates for that role.
>> 
>> A shell from cygwin, msys2 or msys.
>
> They are all subtly incompatible with a native Emacs, both in the
> file-name syntax and in how they treat I/O.

Agreed. The filenames can be massaged (e.g. cygwin-mount.el), and other
issues can be ameliorated by using a different shell (or arguments) for
different tools.

    AndyM

Re: [PATCH] Generalize start-process with keyword args

[Eli Zaretskii]

> From: Stefan Monnier <monnier@IRO.UMontreal.CA>
> Cc: ueno@gnu.org, emacs-devel@gnu.org
> Date: Tue, 17 Mar 2015 17:36:50 -0400
> 
> > What, you mean it will implement its own encoding and decoding
> > machinery?
> 
> No, just that it will treat ":coding nil" in the same way as the absence
> of a :coding arg (i.e. fallback on some default mechanism).

IMO, it will be terribly confusing to have incompatible treatment of
nil in this one API.

Re: [PATCH] Generalize start-process with keyword args

[Daiki Ueno]

[-- Attachment #1: Type: text/plain, Size: 639 bytes --]

Eli Zaretskii <eliz@gnu.org> writes:

>> No, just that it will treat ":coding nil" in the same way as the absence
>> of a :coding arg (i.e. fallback on some default mechanism).
>
> IMO, it will be terribly confusing to have incompatible treatment of
> nil in this one API.

I tend to agree with Stefan, as I find no documentation about the
implication of no-conversion for ":coding nil" except in the C source
code.  Maybe we could add the exception back if it turns out to be too
confusing.

If there are no other problems, I'm going to use the attaching patch
which addressed the suggestions (thanks for that!).

Regards,
-- 
Daiki Ueno

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Generalize-start-process-with-keyword-args.patch --]
[-- Type: text/x-patch, Size: 18772 bytes --]

From 206196c18652601920017b3a30316ac4205b42dd Mon Sep 17 00:00:00 2001
From: Daiki Ueno <ueno@gnu.org>
Date: Mon, 16 Mar 2015 11:38:05 +0900
Subject: [PATCH] Generalize start-process with keyword args

* src/process.c (Fmake_process): New function.
(create_process, create_pty): Check p->pty_flag instead of
Vprocess_connection_type.
(syms_of_process): Register QCcommand, QCconnection_type, Qpty,
Qpipe, and Smake_process.

* lisp/subr.el (start-process): New function, ported from the C
implementation.

* doc/lispref/processes.texi (Asynchronous Processes): Mention
`make-process'.

* etc/NEWS: Mention `make-process'.
---
 doc/lispref/ChangeLog      |   4 +
 doc/lispref/processes.texi |  57 ++++++++++++++
 etc/ChangeLog              |   4 +
 etc/NEWS                   |   3 +
 lisp/ChangeLog             |   5 ++
 lisp/subr.el               |  22 ++++++
 src/ChangeLog              |   8 ++
 src/process.c              | 187 +++++++++++++++++++++++++++++++++------------
 8 files changed, 241 insertions(+), 49 deletions(-)

diff --git a/doc/lispref/ChangeLog b/doc/lispref/ChangeLog
index 260656c..06d4630 100644
--- a/doc/lispref/ChangeLog
+++ b/doc/lispref/ChangeLog
@@ -1,3 +1,7 @@
+2015-03-16  Daiki Ueno  <ueno@gnu.org>
+
+	* processes.texi (Asynchronous Processes): Mention `make-process'.
+
 2015-03-09  Nicolas Petton <nicolas@petton.fr>
 
 	* sequences.texi (seq-into): Add documentation for the new
diff --git a/doc/lispref/processes.texi b/doc/lispref/processes.texi
index 177cd68..337669d 100644
--- a/doc/lispref/processes.texi
+++ b/doc/lispref/processes.texi
@@ -692,6 +692,63 @@ use the function @code{process-tty-name} (@pxref{Process
 Information}).
 @end defvar
 
+@defun make-process &rest args
+This function is like @code{start-process}, but takes keyword arguments.
+
+The arguments @var{args} are a list of keyword/argument pairs.
+Omitting a keyword is always equivalent to specifying it with value
+@code{nil}.  Here are the meaningful keywords:
+
+@table @asis
+@item :name @var{name}
+Use the string @var{name} as the process name.  It is modified if
+necessary to make it unique.
+
+@item :buffer @var{buffer}
+Use @var{buffer} as the process buffer.
+
+@item :command @var{command}
+Use @var{command} as the command line of the process.  @var{command}
+is a list starting with the program's executable file name, followed
+by strings to give to program as arguments.
+
+@item :coding @var{coding}
+If @var{coding} is a symbol, it specifies the coding system to be
+used for both reading and writing of data from and to the
+connection.  If @var{coding} is a cons cell
+@w{@code{(@var{decoding} . @var{encoding})}}, then @var{decoding}
+will be used for reading and @var{encoding} for writing.
+
+If @var{coding} is @code{nil}, the coding system chosen for decoding
+output is @code{undecided}, meaning deduce the encoding from the
+actual data.
+@xref{Output from Processes}.
+
+@item :connection-type @var{TYPE}
+Initialize the type of device used to communicate with the subprocess.
+Possible values are @code{pty} to use a pty, @code{pipe} to use a
+pipe, or @code{nil} to use the default derived from the value of
+the @code{process-connection-type} variable.
+
+@item :noquery @var{query-flag}
+Initialize the process query flag to @var{query-flag}.
+@xref{Query Before Exit}.
+
+@item :stop @var{stopped}
+If @var{stopped} is non-@code{nil}, start the process in the
+``stopped'' state.
+
+@item :filter @var{filter}
+Initialize the process filter to @var{filter}.
+
+@item :sentinel @var{sentinel}
+Initialize the process sentinel to @var{sentinel}.
+@end table
+
+The original argument list, modified with the actual connection
+information, is available via the @code{process-contact} function.
+@end defun
+
 @node Deleting Processes
 @section Deleting Processes
 @cindex deleting processes
diff --git a/etc/ChangeLog b/etc/ChangeLog
index cd5c540..5d5a47f 100644
--- a/etc/ChangeLog
+++ b/etc/ChangeLog
@@ -1,3 +1,7 @@
+2015-03-18  Daiki Ueno  <ueno@gnu.org>
+
+	* NEWS: Mention `make-process'.
+
 2015-03-03  Kelvin White  <kwhite@gnu.org>
 
 	* NEWS.24: Add section to include ERC changes.
diff --git a/etc/NEWS b/etc/NEWS
index 75d55de..8cfc238 100644
--- a/etc/NEWS
+++ b/etc/NEWS
@@ -623,6 +623,9 @@ word syntax, use `\sw' instead.
 \f
 * Lisp Changes in Emacs 25.1
 
+** New function make-process provides a generalized Lisp interface to
+   subprocess creation.
+
 ** Emacs Lisp now supports generators.
 
 ** New finalizer facility for running code when objects
diff --git a/lisp/ChangeLog b/lisp/ChangeLog
index 747a1d6..835712c 100644
--- a/lisp/ChangeLog
+++ b/lisp/ChangeLog
@@ -1,3 +1,8 @@
+2015-03-17  Daiki Ueno  <ueno@gnu.org>
+
+	* subr.el (start-process): New function, ported from the C
+	implementation.
+
 2015-03-15  Michael Albinus  <michael.albinus@gmx.de>
 
 	* net/tramp-adb.el:
diff --git a/lisp/subr.el b/lisp/subr.el
index deadca6..5d763f9 100644
--- a/lisp/subr.el
+++ b/lisp/subr.el
@@ -1901,6 +1901,28 @@ and the file name is displayed in the echo area."
 \f
 ;;;; Process stuff.
 
+(defun start-process (name buffer program &rest program-args)
+  "Start a program in a subprocess.  Return the process object for it.
+NAME is name for process.  It is modified if necessary to make it unique.
+BUFFER is the buffer (or buffer name) to associate with the process.
+
+Process output (both standard output and standard error streams) goes
+at end of BUFFER, unless you specify an output stream or filter
+function to handle the output.  BUFFER may also be nil, meaning that
+this process is not associated with any buffer.
+
+PROGRAM is the program file name.  It is searched for in `exec-path'
+\(which see).  If nil, just associate a pty with the buffer.  Remaining
+arguments are strings to give program as arguments.
+
+If you want to separate standard output from standard error, invoke
+the command through a shell and redirect one of them using the shell
+syntax."
+  (unless (fboundp 'make-process)
+    (error "Emacs was compiled without subprocess support"))
+  (make-process :name name :buffer buffer
+		:command (cons program program-args)))
+
 (defun process-lines (program &rest args)
   "Execute PROGRAM with ARGS, returning its output as a list of lines.
 Signal an error if the program returns with a non-zero exit status."
diff --git a/src/ChangeLog b/src/ChangeLog
index e328afc..8f9a5ee 100644
--- a/src/ChangeLog
+++ b/src/ChangeLog
@@ -1,3 +1,11 @@
+2015-03-17  Daiki Ueno  <ueno@gnu.org>
+
+	* process.c (Fmake_process): New function.
+	(create_process, create_pty): Check p->pty_flag instead of
+	Vprocess_connection_type.
+	(syms_of_process): Register QCcommand, QCconnection_type, Qpty,
+	Qpipe, and Smake_process.  Unregister Sstart_process.
+
 2015-03-15  Eli Zaretskii  <eliz@gnu.org>
 
 	* xdisp.c (handle_invisible_prop): Fix up it->position even when
diff --git a/src/process.c b/src/process.c
index 1d935ba..6068540 100644
--- a/src/process.c
+++ b/src/process.c
@@ -1355,34 +1355,65 @@ DEFUN ("process-list", Fprocess_list, Sprocess_list, 0, 0, 0,
 
 static void start_process_unwind (Lisp_Object proc);
 
-DEFUN ("start-process", Fstart_process, Sstart_process, 3, MANY, 0,
+DEFUN ("make-process", Fmake_process, Smake_process, 0, MANY, 0,
        doc: /* Start a program in a subprocess.  Return the process object for it.
-NAME is name for process.  It is modified if necessary to make it unique.
-BUFFER is the buffer (or buffer name) to associate with the process.
 
-Process output (both standard output and standard error streams) goes
-at end of BUFFER, unless you specify an output stream or filter
-function to handle the output.  BUFFER may also be nil, meaning that
-this process is not associated with any buffer.
+This is similar to `start-process', but arguments are specified as
+keyword/argument pairs.  The following arguments are defined:
 
-PROGRAM is the program file name.  It is searched for in `exec-path'
-(which see).  If nil, just associate a pty with the buffer.  Remaining
-arguments are strings to give program as arguments.
+:name NAME -- NAME is name for process.  It is modified if necessary
+to make it unique.
+
+:buffer BUFFER -- BUFFER is the buffer (or buffer-name) to associate
+with the process.  Process output goes at end of that buffer, unless
+you specify an output stream or filter function to handle the output.
+BUFFER may be also nil, meaning that this process is not associated
+with any buffer.
+
+:command COMMAND -- COMMAND is a list starting with the program file
+name, followed by strings to give to the program as arguments.
 
-If you want to separate standard output from standard error, invoke
-the command through a shell and redirect one of them using the shell
-syntax.
+:coding CODING -- If CODING is a symbol, it specifies the coding
+system used for both reading and writing for this process.  If CODING
+is a cons (DECODING . ENCODING), DECODING is used for reading, and
+ENCODING is used for writing.
 
-usage: (start-process NAME BUFFER PROGRAM &rest PROGRAM-ARGS)  */)
+:noquery BOOL -- When exiting Emacs, query the user if BOOL is nil and
+the process is running.  If BOOL is not given, query before exiting.
+
+:stop BOOL -- Start process in the `stopped' state if BOOL non-nil.
+In the stopped state, a process does not accept incoming data, but you
+can send outgoing data.  The stopped state is cleared by
+`continue-process' and set by `stop-process'.
+
+:connection-type TYPE -- TYPE is control type of device used to
+communicate with subprocesses.  Values are `pipe' to use a pipe, `pty'
+to use a pty, or nil to use the default specified through
+`process-connection-type'.
+
+:filter FILTER -- Install FILTER as the process filter.
+
+:sentinel SENTINEL -- Install SENTINEL as the process sentinel.
+
+usage: (make-process &rest ARGS)  */)
   (ptrdiff_t nargs, Lisp_Object *args)
 {
-  Lisp_Object buffer, name, program, proc, current_dir, tem;
+  Lisp_Object buffer, name, command, program, proc, contact, current_dir, tem;
   unsigned char **new_argv;
+  ptrdiff_t new_argc;
   ptrdiff_t i;
   ptrdiff_t count = SPECPDL_INDEX ();
+  struct gcpro gcpro1;
   USE_SAFE_ALLOCA;
 
-  buffer = args[1];
+  if (nargs == 0)
+    return Qnil;
+
+  /* Save arguments for process-contact and clone-process.  */
+  contact = Flist (nargs, args);
+  GCPRO1 (contact);
+
+  buffer = Fplist_get (contact, QCbuffer);
   if (!NILP (buffer))
     buffer = Fget_buffer_create (buffer);
 
@@ -1402,10 +1433,11 @@ usage: (start-process NAME BUFFER PROGRAM &rest PROGRAM-ARGS)  */)
     UNGCPRO;
   }
 
-  name = args[0];
+  name = Fplist_get (contact, QCname);
   CHECK_STRING (name);
 
-  program = args[2];
+  command = Fplist_get (contact, QCcommand);
+  program = XCAR (command);
 
   if (!NILP (program))
     CHECK_STRING (program);
@@ -1423,7 +1455,25 @@ usage: (start-process NAME BUFFER PROGRAM &rest PROGRAM-ARGS)  */)
   pset_buffer (XPROCESS (proc), buffer);
   pset_sentinel (XPROCESS (proc), Qinternal_default_process_sentinel);
   pset_filter (XPROCESS (proc), Qinternal_default_process_filter);
-  pset_command (XPROCESS (proc), Flist (nargs - 2, args + 2));
+  pset_command (XPROCESS (proc), Fcopy_sequence (command));
+
+  if (tem = Fplist_get (contact, QCnoquery), !NILP (tem))
+    XPROCESS (proc)->kill_without_query = 1;
+  if (tem = Fplist_get (contact, QCstop), !NILP (tem))
+    pset_command (XPROCESS (proc), Qt);
+
+  tem = Fplist_get (contact, QCconnection_type);
+  if (!NILP (tem))
+    {
+      if (EQ (tem, Qpty))
+	XPROCESS (proc)->pty_flag = 1;
+      else if (EQ (tem, Qpipe))
+	XPROCESS (proc)->pty_flag = 0;
+      else
+	report_file_error ("Unknown connection type", tem);
+    }
+  else
+    XPROCESS (proc)->pty_flag = !NILP (Vprocess_connection_type);
 
 #ifdef HAVE_GNUTLS
   /* AKA GNUTLS_INITSTAGE(proc).  */
@@ -1453,15 +1503,30 @@ usage: (start-process NAME BUFFER PROGRAM &rest PROGRAM-ARGS)  */)
     Lisp_Object val, *args2;
     struct gcpro gcpro1, gcpro2;
 
-    val = Vcoding_system_for_read;
+    tem = Fplist_get (contact, QCcoding);
+    val = Qnil;
+    if (!NILP (tem))
+      {
+	val = tem;
+	if (CONSP (val))
+	  val = XCAR (val);
+      }
+    else
+      val = Vcoding_system_for_read;
     if (NILP (val))
       {
-	SAFE_ALLOCA_LISP (args2, nargs + 1);
-	args2[0] = Qstart_process;
-	for (i = 0; i < nargs; i++) args2[i + 1] = args[i];
+	ptrdiff_t nargs2 = 3 + XINT (Flength (command));
+	Lisp_Object tem2;
+	SAFE_ALLOCA_LISP (args2, nargs2);
+	i = 0;
+	args2[i++] = Qstart_process;
+	args2[i++] = name;
+	args2[i++] = buffer;
+	for (tem2 = command; CONSP (tem2); tem2 = XCDR (tem2))
+	  args2[i++] = XCAR (tem2);
 	GCPRO2 (proc, current_dir);
 	if (!NILP (program))
-	  coding_systems = Ffind_operation_coding_system (nargs + 1, args2);
+	  coding_systems = Ffind_operation_coding_system (nargs2, args2);
 	UNGCPRO;
 	if (CONSP (coding_systems))
 	  val = XCAR (coding_systems);
@@ -1470,17 +1535,31 @@ usage: (start-process NAME BUFFER PROGRAM &rest PROGRAM-ARGS)  */)
       }
     pset_decode_coding_system (XPROCESS (proc), val);
 
-    val = Vcoding_system_for_write;
+    val = Qnil;
+    if (!NILP (tem))
+      {
+	val = tem;
+	if (CONSP (val))
+	  val = XCDR (val);
+      }
+    else
+      val = Vcoding_system_for_write;
     if (NILP (val))
       {
 	if (EQ (coding_systems, Qt))
 	  {
-	    SAFE_ALLOCA_LISP (args2, nargs + 1);
-	    args2[0] = Qstart_process;
-	    for (i = 0; i < nargs; i++) args2[i + 1] = args[i];
+	    ptrdiff_t nargs2 = 3 + XINT (Flength (command));
+	    Lisp_Object tem2;
+	    SAFE_ALLOCA_LISP (args2, nargs2);
+	    i = 0;
+	    args2[i++] = Qstart_process;
+	    args2[i++] = name;
+	    args2[i++] = buffer;
+	    for (tem2 = command; CONSP (tem2); tem2 = XCDR (tem2))
+	      args2[i++] = XCAR (tem2);
 	    GCPRO2 (proc, current_dir);
 	    if (!NILP (program))
-	      coding_systems = Ffind_operation_coding_system (nargs + 1, args2);
+	      coding_systems = Ffind_operation_coding_system (nargs2, args2);
 	    UNGCPRO;
 	  }
 	if (CONSP (coding_systems))
@@ -1512,10 +1591,10 @@ usage: (start-process NAME BUFFER PROGRAM &rest PROGRAM-ARGS)  */)
 	  && !(SCHARS (program) > 1
 	       && IS_DEVICE_SEP (SREF (program, 1))))
 	{
-	  struct gcpro gcpro1, gcpro2, gcpro3, gcpro4;
+	  struct gcpro gcpro1, gcpro2;
 
 	  tem = Qnil;
-	  GCPRO4 (name, program, buffer, current_dir);
+	  GCPRO2 (buffer, current_dir);
 	  openp (Vexec_path, program, Vexec_suffixes, &tem,
 		 make_number (X_OK), false);
 	  UNGCPRO;
@@ -1534,31 +1613,32 @@ usage: (start-process NAME BUFFER PROGRAM &rest PROGRAM-ARGS)  */)
       tem = remove_slash_colon (tem);
 
       {
-	Lisp_Object arg_encoding = Qnil;
+	Lisp_Object arg_encoding = Qnil, tem2;
 	struct gcpro gcpro1;
+
+	tem = Fcons (tem, Fcopy_sequence (XCDR (command)));
 	GCPRO1 (tem);
 
 	/* Encode the file name and put it in NEW_ARGV.
 	   That's where the child will use it to execute the program.  */
-	tem = list1 (ENCODE_FILE (tem));
+	XSETCAR (tem, ENCODE_FILE (XCAR (tem)));
 
 	/* Here we encode arguments by the coding system used for sending
 	   data to the process.  We don't support using different coding
 	   systems for encoding arguments and for encoding data sent to the
 	   process.  */
 
-	for (i = 3; i < nargs; i++)
+	for (tem2 = XCDR (tem); CONSP (tem2); tem2 = XCDR (tem2))
 	  {
-	    tem = Fcons (args[i], tem);
-	    CHECK_STRING (XCAR (tem));
-	    if (STRING_MULTIBYTE (XCAR (tem)))
+	    CHECK_STRING (XCAR (tem2));
+	    if (STRING_MULTIBYTE (XCAR (tem2)))
 	      {
 		if (NILP (arg_encoding))
 		  arg_encoding = (complement_process_encoding_system
 				  (XPROCESS (proc)->encode_coding_system));
-		XSETCAR (tem,
+		XSETCAR (tem2,
 			 code_convert_string_norecord
-			 (XCAR (tem), arg_encoding, 1));
+			 (XCAR (tem2), arg_encoding, 1));
 	      }
 	  }
 
@@ -1567,10 +1647,11 @@ usage: (start-process NAME BUFFER PROGRAM &rest PROGRAM-ARGS)  */)
 
       /* Now that everything is encoded we can collect the strings into
 	 NEW_ARGV.  */
-      SAFE_NALLOCA (new_argv, 1, nargs - 1);
-      new_argv[nargs - 2] = 0;
+      new_argc = XINT (Flength (tem));
+      SAFE_NALLOCA (new_argv, 1, new_argc + 1);
+      new_argv[new_argc] = 0;
 
-      for (i = nargs - 2; i-- != 0; )
+      for (i = 0; i < new_argc; i++)
 	{
 	  new_argv[i] = SDATA (XCAR (tem));
 	  tem = XCDR (tem);
@@ -1581,6 +1662,7 @@ usage: (start-process NAME BUFFER PROGRAM &rest PROGRAM-ARGS)  */)
   else
     create_pty (proc);
 
+  UNGCPRO;
   SAFE_FREE ();
   return unbind_to (count, proc);
 }
@@ -1648,7 +1730,7 @@ create_process (Lisp_Object process, char **new_argv, Lisp_Object current_dir)
 
   inchannel = outchannel = -1;
 
-  if (!NILP (Vprocess_connection_type))
+  if (p->pty_flag)
     outchannel = inchannel = allocate_pty (pty_name);
 
   if (inchannel >= 0)
@@ -1692,6 +1774,8 @@ create_process (Lisp_Object process, char **new_argv, Lisp_Object current_dir)
   chan_process[inchannel] = process;
   p->infd = inchannel;
   p->outfd = outchannel;
+  if (inchannel > max_process_desc)
+    max_process_desc = inchannel;
 
   /* Previously we recorded the tty descriptor used in the subprocess.
      It was only used for getting the foreground tty process, so now
@@ -1701,10 +1785,11 @@ create_process (Lisp_Object process, char **new_argv, Lisp_Object current_dir)
   p->pty_flag = pty_flag;
   pset_status (p, Qrun);
 
-  FD_SET (inchannel, &input_wait_mask);
-  FD_SET (inchannel, &non_keyboard_wait_mask);
-  if (inchannel > max_process_desc)
-    max_process_desc = inchannel;
+  if (!EQ (p->command, Qt))
+    {
+      FD_SET (inchannel, &input_wait_mask);
+      FD_SET (inchannel, &non_keyboard_wait_mask);
+    }
 
   /* This may signal an error.  */
   setup_process_coding_systems (process);
@@ -1894,7 +1979,7 @@ create_pty (Lisp_Object process)
 {
   struct Lisp_Process *p = XPROCESS (process);
   char pty_name[PTY_NAME_SIZE];
-  int pty_fd = NILP (Vprocess_connection_type) ? -1 : allocate_pty (pty_name);
+  int pty_fd = !p->pty_flag ? -1 : allocate_pty (pty_name);
 
   if (pty_fd >= 0)
     {
@@ -7269,6 +7354,10 @@ syms_of_process (void)
   DEFSYM (QCstop, ":stop");
   DEFSYM (QCoptions, ":options");
   DEFSYM (QCplist, ":plist");
+  DEFSYM (QCcommand, ":command");
+  DEFSYM (QCconnection_type, ":connection-type");
+  DEFSYM (Qpty, "pty");
+  DEFSYM (Qpipe, "pipe");
 
   DEFSYM (Qlast_nonmenu_event, "last-nonmenu-event");
 
@@ -7371,7 +7460,7 @@ The variable takes effect when `start-process' is called.  */);
   defsubr (&Sprocess_plist);
   defsubr (&Sset_process_plist);
   defsubr (&Sprocess_list);
-  defsubr (&Sstart_process);
+  defsubr (&Smake_process);
   defsubr (&Sserial_process_configure);
   defsubr (&Smake_serial_process);
   defsubr (&Sset_network_process_option);
-- 
2.1.0

[PATCH] Add facility to collect stderr of async subprocess

[Daiki Ueno]

[-- Attachment #1: Type: text/plain, Size: 1657 bytes --]

Eli Zaretskii <eliz@gnu.org> writes:

>> In any case, supporting stderr could be a starting point, as it seems to
>> be a long-standing request:
>> https://lists.gnu.org/archive/html/emacs-devel/2004-04/msg01051.html
>> and it wouldn't involve a portability issue.
>
> Redirecting the child's stderr is already supported on Windows, so
> this is only a matter of having the higher layers DTRT.

So, let's start with small (but common) things.  The attached patch adds
a new keyword `:stderr' to `make-process'.  The argument can be either a
buffer or a (newly introduced) pipe process.

One could write:

  (make-process :name "test"
                :buffer (current-buffer)
                :command (list (expand-file-name "./test.sh"))
                :stderr "stderr")

to collect the stderr output in the "stderr" buffer, or could write:

  (let ((stderr (make-pipe-process :name "stderr")))
    (make-process :name "test"
                  :buffer (current-buffer)
                  :command (list (expand-file-name "./test.sh"))
                  :stderr stderr)
    (set-process-filter stderr ...))

to collect the stderr output with a process filter.

The patch should apply after the make-process patch:

> From 206196c18652601920017b3a30316ac4205b42dd Mon Sep 17 00:00:00 2001
> From: Daiki Ueno <ueno@gnu.org>
> Date: Mon, 16 Mar 2015 11:38:05 +0900
> Subject: [PATCH] Generalize start-process with keyword args

Comments appreciated.  To be honest, a new process type to collect
stderr might be overkill, but it would make it possible for the current
process I/O functions to support further file descriptors / handles.

Regards,
-- 
Daiki Ueno

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Add-facility-to-collect-stderr-of-async-subprocess.patch --]
[-- Type: text/x-patch, Size: 30077 bytes --]

From 2319faa7a9853b562c2296030de70188aee75f8b Mon Sep 17 00:00:00 2001
From: Daiki Ueno <ueno@gnu.org>
Date: Mon, 9 Mar 2015 09:50:32 +0900
Subject: [PATCH] Add facility to collect stderr of async subprocess

* src/lisp.h (emacs_pipe2): New function declaration.
* src/sysdep.c (emacs_pipe2): New function.
(emacs_pipe): Define as a wrapper around emacs_pipe2.
* src/process.h (struct Lisp_Process): New member stderr.
* src/process.c (PIPECONN_P): New macro.
(PIPECONN1_P): New macro.
(Fdelete_process, Fprocess_status, Fset_process_buffer)
(Fset_process_filter, Fset_process_sentinel, Fstop_process)
(Fcontinue_process): Handle pipe process specially.
(create_process): Respect Vprocess_pipe_list and
Vprocess_standard_error.
(Fmake_pipe_process): New function.
(Fmake_process): Add new keyword argument :stderr.
(deactivate_process): Call Fdelete_process on associated pipe
process.
(wait_reading_process_output): Don't consider a pipe process has
gone when the read end is closed.
(syms_of_process): Register Qpipe and Smake_pipe_process.

* doc/lispref/processes.texi (Asynchronous Processes): Mention
`make-pipe-process' and `:stderr' keyword of `make-process'.

* test/automated/process-tests.el (process-test-stderr-buffer)
(process-test-stderr-filter): New tests.

* etc/NEWS: Mention pipe process.
---
 doc/lispref/ChangeLog           |   5 +
 doc/lispref/processes.texi      |  54 +++++++
 etc/ChangeLog                   |   4 +
 etc/NEWS                        |   4 +
 src/ChangeLog                   |  22 +++
 src/lisp.h                      |   1 +
 src/process.c                   | 303 +++++++++++++++++++++++++++++++++++++---
 src/process.h                   |   3 +
 src/sysdep.c                    |  14 +-
 test/ChangeLog                  |   5 +
 test/automated/process-tests.el |  65 +++++++++
 11 files changed, 455 insertions(+), 25 deletions(-)

diff --git a/doc/lispref/ChangeLog b/doc/lispref/ChangeLog
index 06d4630..777d386 100644
--- a/doc/lispref/ChangeLog
+++ b/doc/lispref/ChangeLog
@@ -1,3 +1,8 @@
+2015-03-17  Daiki Ueno  <ueno@gnu.org>
+
+	* processes.texi (Asynchronous Processes): Mention
+	`make-pipe-process' and `:stderr' keyword of `make-process'.
+
 2015-03-16  Daiki Ueno  <ueno@gnu.org>
 
 	* processes.texi (Asynchronous Processes): Mention `make-process'.
diff --git a/doc/lispref/processes.texi b/doc/lispref/processes.texi
index 337669d..27ed697 100644
--- a/doc/lispref/processes.texi
+++ b/doc/lispref/processes.texi
@@ -743,6 +743,60 @@ Initialize the process filter to @var{filter}.
 
 @item :sentinel @var{sentinel}
 Initialize the process sentinel to @var{sentinel}.
+
+@item :stderr @var{stderr}
+Associate @var{stderr} with the standard error of the process.
+@var{stderr} is either a buffer or a pipe process created with
+@code{make-pipe-process}.
+@end table
+
+The original argument list, modified with the actual connection
+information, is available via the @code{process-contact} function.
+@end defun
+
+@defun make-pipe-process &rest args
+This function creates a bidirectional pipe which can be attached to a
+child process (typically through the @code{:stderr} keyword of
+@code{make-process}).
+
+The arguments @var{args} are a list of keyword/argument pairs.
+Omitting a keyword is always equivalent to specifying it with value
+@code{nil}, except for @code{:coding}.
+Here are the meaningful keywords:
+
+@table @asis
+@item :name @var{name}
+Use the string @var{name} as the process name.  It is modified if
+necessary to make it unique.
+
+@item :buffer @var{buffer}
+Use @var{buffer} as the process buffer.
+
+@item :coding @var{coding}
+If @var{coding} is a symbol, it specifies the coding system to be
+used for both reading and writing of data from and to the
+connection.  If @var{coding} is a cons cell
+@w{@code{(@var{decoding} . @var{encoding})}}, then @var{decoding}
+will be used for reading and @var{encoding} for writing.
+
+If @var{coding} is @code{nil}, the coding system chosen for decoding
+output is @code{undecided}, meaning deduce the encoding from the
+actual data.
+@xref{Output from Processes}.
+
+@item :noquery @var{query-flag}
+Initialize the process query flag to @var{query-flag}.
+@xref{Query Before Exit}.
+
+@item :stop @var{stopped}
+If @var{stopped} is non-@code{nil}, start the process in the
+``stopped'' state.
+
+@item :filter @var{filter}
+Initialize the process filter to @var{filter}.
+
+@item :sentinel @var{sentinel}
+Initialize the process sentinel to @var{sentinel}.
 @end table
 
 The original argument list, modified with the actual connection
diff --git a/etc/ChangeLog b/etc/ChangeLog
index 5d5a47f..d9a3006 100644
--- a/etc/ChangeLog
+++ b/etc/ChangeLog
@@ -1,5 +1,9 @@
 2015-03-18  Daiki Ueno  <ueno@gnu.org>
 
+	* NEWS: Mention pipe process.
+
+2015-03-18  Daiki Ueno  <ueno@gnu.org>
+
 	* NEWS: Mention `make-process'.
 
 2015-03-03  Kelvin White  <kwhite@gnu.org>
diff --git a/etc/NEWS b/etc/NEWS
index 8cfc238..da19b3f 100644
--- a/etc/NEWS
+++ b/etc/NEWS
@@ -623,6 +623,10 @@ word syntax, use `\sw' instead.
 \f
 * Lisp Changes in Emacs 25.1
 
+** New process type `pipe', which can be used in combination with the
+   `:stderr' keyword of make-process to collect standard error output
+   of subprocess.
+
 ** New function make-process provides a generalized Lisp interface to
    subprocess creation.
 
diff --git a/src/ChangeLog b/src/ChangeLog
index 8f9a5ee..421e3c5 100644
--- a/src/ChangeLog
+++ b/src/ChangeLog
@@ -1,5 +1,27 @@
 2015-03-17  Daiki Ueno  <ueno@gnu.org>
 
+	Add facility to collect stderr of async subprocess
+	* lisp.h (emacs_pipe2): New function declaration.
+	* sysdep.c (emacs_pipe2): New function.
+	(emacs_pipe): Define as a wrapper around emacs_pipe2.
+	* process.h (struct Lisp_Process): New member stderr.
+	* process.c (PIPECONN_P): New macro.
+	(PIPECONN1_P): New macro.
+	(Fdelete_process, Fprocess_status, Fset_process_buffer)
+	(Fset_process_filter, Fset_process_sentinel, Fstop_process)
+	(Fcontinue_process): Handle pipe process specially.
+	(create_process): Respect Vprocess_pipe_list and
+	Vprocess_standard_error.
+	(Fmake_pipe_process): New function.
+	(Fmake_process): Add new keyword argument :stderr.
+	(deactivate_process): Call Fdelete_process on associated pipe
+	process.
+	(wait_reading_process_output): Don't consider a pipe process has
+	gone when the read end is closed.
+	(syms_of_process): Register Qpipe and Smake_pipe_process.
+
+2015-03-17  Daiki Ueno  <ueno@gnu.org>
+
 	* process.c (Fmake_process): New function.
 	(create_process, create_pty): Check p->pty_flag instead of
 	Vprocess_connection_type.
diff --git a/src/lisp.h b/src/lisp.h
index b730619..2e161da 100644
--- a/src/lisp.h
+++ b/src/lisp.h
@@ -4378,6 +4378,7 @@ extern void emacs_backtrace (int);
 extern _Noreturn void emacs_abort (void) NO_INLINE;
 extern int emacs_open (const char *, int, int);
 extern int emacs_pipe (int[2]);
+extern int emacs_pipe2 (int[2], int);
 extern int emacs_close (int);
 extern ptrdiff_t emacs_read (int, void *, ptrdiff_t);
 extern ptrdiff_t emacs_write (int, void const *, ptrdiff_t);
diff --git a/src/process.c b/src/process.c
index 6068540..9596ccd 100644
--- a/src/process.c
+++ b/src/process.c
@@ -189,6 +189,8 @@ process_socket (int domain, int type, int protocol)
 #define NETCONN1_P(p) (EQ (p->type, Qnetwork))
 #define SERIALCONN_P(p) (EQ (XPROCESS (p)->type, Qserial))
 #define SERIALCONN1_P(p) (EQ (p->type, Qserial))
+#define PIPECONN_P(p) (EQ (XPROCESS (p)->type, Qpipe))
+#define PIPECONN1_P(p) (EQ (p->type, Qpipe))
 
 /* Number of events of change of status of a process.  */
 static EMACS_INT process_tick;
@@ -420,6 +422,11 @@ pset_write_queue (struct Lisp_Process *p, Lisp_Object val)
 {
   p->write_queue = val;
 }
+static void
+pset_stderr (struct Lisp_Process *p, Lisp_Object val)
+{
+  p->stderr = val;
+}
 
 \f
 static Lisp_Object
@@ -846,7 +853,7 @@ nil, indicating the current buffer's process.  */)
   p = XPROCESS (process);
 
   p->raw_status_new = 0;
-  if (NETCONN1_P (p) || SERIALCONN1_P (p))
+  if (NETCONN1_P (p) || SERIALCONN1_P (p) || PIPECONN1_P (p))
     {
       pset_status (p, list2 (Qexit, make_number (0)));
       p->tick = ++process_tick;
@@ -912,7 +919,7 @@ nil, indicating the current buffer's process.  */)
   status = p->status;
   if (CONSP (status))
     status = XCAR (status);
-  if (NETCONN1_P (p) || SERIALCONN1_P (p))
+  if (NETCONN1_P (p) || SERIALCONN1_P (p) || PIPECONN1_P (p))
     {
       if (EQ (status, Qexit))
 	status = Qclosed;
@@ -996,7 +1003,7 @@ Return BUFFER.  */)
     CHECK_BUFFER (buffer);
   p = XPROCESS (process);
   pset_buffer (p, buffer);
-  if (NETCONN1_P (p) || SERIALCONN1_P (p))
+  if (NETCONN1_P (p) || SERIALCONN1_P (p) || PIPECONN1_P (p))
     pset_childp (p, Fplist_put (p->childp, QCbuffer, buffer));
   setup_process_coding_systems (process);
   return buffer;
@@ -1072,7 +1079,7 @@ The string argument is normally a multibyte string, except:
     }
 
   pset_filter (p, filter);
-  if (NETCONN1_P (p) || SERIALCONN1_P (p))
+  if (NETCONN1_P (p) || SERIALCONN1_P (p) || PIPECONN1_P (p))
     pset_childp (p, Fplist_put (p->childp, QCfilter, filter));
   setup_process_coding_systems (process);
   return filter;
@@ -1104,7 +1111,7 @@ It gets two arguments: the process, and a string describing the change.  */)
     sentinel = Qinternal_default_process_sentinel;
 
   pset_sentinel (p, sentinel);
-  if (NETCONN1_P (p) || SERIALCONN1_P (p))
+  if (NETCONN1_P (p) || SERIALCONN1_P (p) || PIPECONN1_P (p))
     pset_childp (p, Fplist_put (p->childp, QCsentinel, sentinel));
   return sentinel;
 }
@@ -1192,13 +1199,13 @@ DEFUN ("process-query-on-exit-flag",
 DEFUN ("process-contact", Fprocess_contact, Sprocess_contact,
        1, 2, 0,
        doc: /* Return the contact info of PROCESS; t for a real child.
-For a network or serial connection, the value depends on the optional
-KEY arg.  If KEY is nil, value is a cons cell of the form (HOST
-SERVICE) for a network connection or (PORT SPEED) for a serial
-connection.  If KEY is t, the complete contact information for the
-connection is returned, else the specific value for the keyword KEY is
-returned.  See `make-network-process' or `make-serial-process' for a
-list of keywords.  */)
+For a network or serial connection, or a pipe, the value depends on
+the optional KEY arg.  If KEY is nil, value is a cons cell of the form
+\(HOST SERVICE) for a network connection or (PORT SPEED) for a serial
+connection, or (INPUT OUTPUT) for a pipe.  If KEY is t, the complete
+contact information for the connection is returned, else the specific
+value for the keyword KEY is returned.  See `make-network-process' or
+`make-serial-process' for a list of keywords.  */)
   (register Lisp_Object process, Lisp_Object key)
 {
   Lisp_Object contact;
@@ -1395,10 +1402,14 @@ to use a pty, or nil to use the default specified through
 
 :sentinel SENTINEL -- Install SENTINEL as the process sentinel.
 
+:stderr STDERR -- STDERR is either a buffer or a pipe process attached
+to the standard error of subprocess.
+
 usage: (make-process &rest ARGS)  */)
   (ptrdiff_t nargs, Lisp_Object *args)
 {
   Lisp_Object buffer, name, command, program, proc, contact, current_dir, tem;
+  Lisp_Object stderr_proc;
   unsigned char **new_argv;
   ptrdiff_t new_argc;
   ptrdiff_t i;
@@ -1442,6 +1453,27 @@ usage: (make-process &rest ARGS)  */)
   if (!NILP (program))
     CHECK_STRING (program);
 
+  stderr_proc = Fplist_get (contact, QCstderr);
+  if (!NILP (stderr_proc))
+    {
+      if (PROCESSP (stderr_proc))
+	{
+	  if (!PIPECONN_P (stderr_proc))
+	    error ("Process is not a pipe process");
+	}
+      else
+	{
+	  struct gcpro gcpro1, gcpro2;
+	  GCPRO2 (buffer, current_dir);
+	  stderr_proc = CALLN (Fmake_pipe_process,
+			       QCname,
+			       concat2 (name, build_string (" stderr")),
+			       QCbuffer,
+			       Fget_buffer_create (stderr_proc));
+	  UNGCPRO;
+	}
+    }
+
   proc = make_process (name);
   /* If an error occurs and we can't start the process, we want to
      remove it from the process list.  This means that each error
@@ -1475,6 +1507,12 @@ usage: (make-process &rest ARGS)  */)
   else
     XPROCESS (proc)->pty_flag = !NILP (Vprocess_connection_type);
 
+  if (!NILP (stderr_proc))
+    {
+      pset_stderr (XPROCESS (proc), stderr_proc);
+      XPROCESS (proc)->pty_flag = 0;
+    }
+
 #ifdef HAVE_GNUTLS
   /* AKA GNUTLS_INITSTAGE(proc).  */
   XPROCESS (proc)->gnutls_initstage = GNUTLS_STAGE_EMPTY;
@@ -1719,16 +1757,16 @@ static void
 create_process (Lisp_Object process, char **new_argv, Lisp_Object current_dir)
 {
   struct Lisp_Process *p = XPROCESS (process);
-  int inchannel, outchannel;
+  int inchannel, outchannel, errchannel;
   pid_t pid;
   int vfork_errno;
-  int forkin, forkout;
+  int forkin, forkout, forkerr;
   bool pty_flag = 0;
   char pty_name[PTY_NAME_SIZE];
   Lisp_Object lisp_pty_name = Qnil;
   sigset_t oldset;
 
-  inchannel = outchannel = -1;
+  inchannel = outchannel = errchannel = -1;
 
   if (p->pty_flag)
     outchannel = inchannel = allocate_pty (pty_name);
@@ -1748,6 +1786,7 @@ create_process (Lisp_Object process, char **new_argv, Lisp_Object current_dir)
 #else
       forkin = forkout = -1;
 #endif /* not USG, or USG_SUBTTY_WORKS */
+      forkerr = -1;
       pty_flag = 1;
       lisp_pty_name = build_string (pty_name);
     }
@@ -1760,6 +1799,23 @@ create_process (Lisp_Object process, char **new_argv, Lisp_Object current_dir)
       outchannel = p->open_fd[WRITE_TO_SUBPROCESS];
       inchannel = p->open_fd[READ_FROM_SUBPROCESS];
       forkout = p->open_fd[SUBPROCESS_STDOUT];
+
+      if (!NILP (p->stderr))
+	{
+	  struct Lisp_Process *pp = XPROCESS (p->stderr);
+
+	  forkerr = pp->open_fd[SUBPROCESS_STDOUT];
+	  errchannel = pp->open_fd[READ_FROM_SUBPROCESS];
+
+	  /* FORKERR will be redirected in child_setup.  */
+	  fcntl (forkerr, F_SETFD, FD_CLOEXEC);
+
+	  /* Close unnecessary file descriptors.  */
+	  close_process_fd (&pp->open_fd[WRITE_TO_SUBPROCESS]);
+	  close_process_fd (&pp->open_fd[SUBPROCESS_STDIN]);
+	}
+      else
+	forkerr = -1;
     }
 
 #ifndef WINDOWSNT
@@ -1805,6 +1861,7 @@ create_process (Lisp_Object process, char **new_argv, Lisp_Object current_dir)
     char **volatile new_argv_volatile = new_argv;
     int volatile forkin_volatile = forkin;
     int volatile forkout_volatile = forkout;
+    int volatile forkerr_volatile = forkerr;
     struct Lisp_Process *p_volatile = p;
 
     pid = vfork ();
@@ -1814,6 +1871,7 @@ create_process (Lisp_Object process, char **new_argv, Lisp_Object current_dir)
     new_argv = new_argv_volatile;
     forkin = forkin_volatile;
     forkout = forkout_volatile;
+    forkerr = forkerr_volatile;
     p = p_volatile;
 
     pty_flag = p->pty_flag;
@@ -1824,6 +1882,7 @@ create_process (Lisp_Object process, char **new_argv, Lisp_Object current_dir)
     {
       int xforkin = forkin;
       int xforkout = forkout;
+      int xforkerr = forkerr;
 
       /* Make the pty be the controlling terminal of the process.  */
 #ifdef HAVE_PTYS
@@ -1923,10 +1982,13 @@ create_process (Lisp_Object process, char **new_argv, Lisp_Object current_dir)
 
       if (pty_flag)
 	child_setup_tty (xforkout);
+
+      if (xforkerr < 0)
+	xforkerr = xforkout;
 #ifdef WINDOWSNT
-      pid = child_setup (xforkin, xforkout, xforkout, new_argv, 1, current_dir);
+      pid = child_setup (xforkin, xforkout, xforkerr, new_argv, 1, current_dir);
 #else  /* not WINDOWSNT */
-      child_setup (xforkin, xforkout, xforkout, new_argv, 1, current_dir);
+      child_setup (xforkin, xforkout, xforkerr, new_argv, 1, current_dir);
 #endif /* not WINDOWSNT */
     }
 
@@ -1971,6 +2033,11 @@ create_process (Lisp_Object process, char **new_argv, Lisp_Object current_dir)
 	close_process_fd (&p->open_fd[READ_FROM_EXEC_MONITOR]);
       }
 #endif
+      if (!NILP (p->stderr))
+	{
+	  struct Lisp_Process *pp = XPROCESS (p->stderr);
+	  close_process_fd (&pp->open_fd[SUBPROCESS_STDOUT]);
+	}
     }
 }
 
@@ -2029,6 +2096,187 @@ create_pty (Lisp_Object process)
   p->pid = -2;
 }
 
+DEFUN ("make-pipe-process", Fmake_pipe_process, Smake_pipe_process,
+       0, MANY, 0,
+       doc: /* Create and return a bidirectional pipe process.
+
+In Emacs, pipes are represented by process objects, so input and
+output work as for subprocesses, and `delete-process' closes a pipe.
+However, a pipe process has no process id, it cannot be signaled,
+and the status codes are different from normal processes.
+
+Arguments are specified as keyword/argument pairs.  The following
+arguments are defined:
+
+:name NAME -- NAME is the name of the process.  It is modified if necessary to make it unique.
+
+:buffer BUFFER -- BUFFER is the buffer (or buffer-name) to associate
+with the process.  Process output goes at the end of that buffer,
+unless you specify an output stream or filter function to handle the
+output.  If BUFFER is not given, the value of NAME is used.
+
+:coding CODING -- If CODING is a symbol, it specifies the coding
+system used for both reading and writing for this process.  If CODING
+is a cons (DECODING . ENCODING), DECODING is used for reading, and
+ENCODING is used for writing.
+
+:noquery BOOL -- When exiting Emacs, query the user if BOOL is nil and
+the process is running.  If BOOL is not given, query before exiting.
+
+:stop BOOL -- Start process in the `stopped' state if BOOL non-nil.
+In the stopped state, a pipe process does not accept incoming data,
+but you can send outgoing data.  The stopped state is cleared by
+`continue-process' and set by `stop-process'.
+
+:filter FILTER -- Install FILTER as the process filter.
+
+:sentinel SENTINEL -- Install SENTINEL as the process sentinel.
+
+usage:  (make-pipe-process &rest ARGS)  */)
+  (ptrdiff_t nargs, Lisp_Object *args)
+{
+  Lisp_Object proc, contact;
+  struct Lisp_Process *p;
+  struct gcpro gcpro1;
+  Lisp_Object name, buffer;
+  Lisp_Object tem, val;
+  ptrdiff_t specpdl_count;
+  int inchannel, outchannel;
+
+  if (nargs == 0)
+    return Qnil;
+
+  contact = Flist (nargs, args);
+  GCPRO1 (contact);
+
+  name = Fplist_get (contact, QCname);
+  CHECK_STRING (name);
+  proc = make_process (name);
+  specpdl_count = SPECPDL_INDEX ();
+  record_unwind_protect (remove_process, proc);
+  p = XPROCESS (proc);
+
+  if (emacs_pipe2 (p->open_fd + SUBPROCESS_STDIN, O_BINARY) != 0
+      || emacs_pipe2 (p->open_fd + READ_FROM_SUBPROCESS, O_BINARY) != 0)
+    report_file_error ("Creating pipe", Qnil);
+  outchannel = p->open_fd[WRITE_TO_SUBPROCESS];
+  inchannel = p->open_fd[READ_FROM_SUBPROCESS];
+
+  /* Child doesn't need inchannel/outchannel after exec.  */
+  fcntl (inchannel, F_SETFD, FD_CLOEXEC);
+  fcntl (outchannel, F_SETFD, FD_CLOEXEC);
+
+  fcntl (inchannel, F_SETFL, O_NONBLOCK);
+  fcntl (outchannel, F_SETFL, O_NONBLOCK);
+
+  /* Record this as an active process, with its channels.  */
+  chan_process[inchannel] = proc;
+  p->infd = inchannel;
+  p->outfd = outchannel;
+
+  if (inchannel > max_process_desc)
+    max_process_desc = inchannel;
+
+  buffer = Fplist_get (contact, QCbuffer);
+  if (NILP (buffer))
+    buffer = name;
+  buffer = Fget_buffer_create (buffer);
+  pset_buffer (p, buffer);
+
+  pset_childp (p, contact);
+  pset_plist (p, Fcopy_sequence (Fplist_get (contact, QCplist)));
+  pset_type (p, Qpipe);
+  pset_sentinel (p, Fplist_get (contact, QCsentinel));
+  pset_filter (p, Fplist_get (contact, QCfilter));
+  pset_log (p, Qnil);
+  if (tem = Fplist_get (contact, QCnoquery), !NILP (tem))
+    p->kill_without_query = 1;
+  if (tem = Fplist_get (contact, QCstop), !NILP (tem))
+    pset_command (p, Qt);
+  eassert (! p->pty_flag);
+
+  if (!EQ (p->command, Qt))
+    {
+      FD_SET (inchannel, &input_wait_mask);
+      FD_SET (inchannel, &non_keyboard_wait_mask);
+    }
+#ifdef ADAPTIVE_READ_BUFFERING
+  p->adaptive_read_buffering
+    = (NILP (Vprocess_adaptive_read_buffering) ? 0
+       : EQ (Vprocess_adaptive_read_buffering, Qt) ? 1 : 2);
+#endif
+
+  /* Make the process marker point into the process buffer (if any).  */
+  if (BUFFERP (buffer))
+    set_marker_both (p->mark, buffer,
+		     BUF_ZV (XBUFFER (buffer)),
+		     BUF_ZV_BYTE (XBUFFER (buffer)));
+
+  {
+    /* Setup coding systems for communicating with the network stream.  */
+    struct gcpro gcpro1;
+    /* Qt denotes we have not yet called Ffind_operation_coding_system.  */
+    Lisp_Object coding_systems = Qt;
+    Lisp_Object val;
+
+    tem = Fplist_get (contact, QCcoding);
+    val = Qnil;
+    if (!NILP (tem))
+      {
+	val = tem;
+	if (CONSP (val))
+	  val = XCAR (val);
+      }
+    else if (!NILP (Vcoding_system_for_read))
+      val = Vcoding_system_for_read;
+    else if ((!NILP (buffer) && NILP (BVAR (XBUFFER (buffer), enable_multibyte_characters)))
+	     || (NILP (buffer) && NILP (BVAR (&buffer_defaults, enable_multibyte_characters))))
+      /* We dare not decode end-of-line format by setting VAL to
+	 Qraw_text, because the existing Emacs Lisp libraries
+	 assume that they receive bare code including a sequence of
+	 CR LF.  */
+      val = Qnil;
+    else
+      {
+	if (CONSP (coding_systems))
+	  val = XCAR (coding_systems);
+	else if (CONSP (Vdefault_process_coding_system))
+	  val = XCAR (Vdefault_process_coding_system);
+	else
+	  val = Qnil;
+      }
+    pset_decode_coding_system (p, val);
+
+    if (!NILP (tem))
+      {
+	val = tem;
+	if (CONSP (val))
+	  val = XCDR (val);
+      }
+    else if (!NILP (Vcoding_system_for_write))
+      val = Vcoding_system_for_write;
+    else if (NILP (BVAR (current_buffer, enable_multibyte_characters)))
+      val = Qnil;
+    else
+      {
+	if (CONSP (coding_systems))
+	  val = XCDR (coding_systems);
+	else if (CONSP (Vdefault_process_coding_system))
+	  val = XCDR (Vdefault_process_coding_system);
+	else
+	  val = Qnil;
+      }
+    pset_encode_coding_system (p, val);
+  }
+  /* This may signal an error.  */
+  setup_process_coding_systems (proc);
+
+  specpdl_ptr = specpdl + specpdl_count;
+
+  UNGCPRO;
+  return proc;
+}
+
 \f
 /* Convert an internal struct sockaddr to a lisp object (vector or string).
    The address family of sa is not included in the result.  */
@@ -3946,6 +4194,7 @@ deactivate_process (Lisp_Object proc)
   int inchannel;
   struct Lisp_Process *p = XPROCESS (proc);
   int i;
+  Lisp_Object tem;
 
 #ifdef HAVE_GNUTLS
   /* Delete GnuTLS structures in PROC, if any.  */
@@ -4004,6 +4253,9 @@ deactivate_process (Lisp_Object proc)
 	  max_process_desc = i;
 	}
     }
+
+  if (!NILP (p->stderr))
+    Fdelete_process (p->stderr);
 }
 
 \f
@@ -4897,7 +5149,8 @@ wait_reading_process_output (intmax_t time_limit, int nsecs, int read_kbd,
 		 available now and a closed pipe.
 		 With luck, a closed pipe will be accompanied by
 		 subprocess termination and SIGCHLD.  */
-	      else if (nread == 0 && !NETCONN_P (proc) && !SERIALCONN_P (proc))
+	      else if (nread == 0 && !NETCONN_P (proc) && !SERIALCONN_P (proc)
+		       && !PIPECONN_P (proc))
 		;
 #endif
 #ifdef HAVE_PTYS
@@ -4929,7 +5182,8 @@ wait_reading_process_output (intmax_t time_limit, int nsecs, int read_kbd,
 #endif /* HAVE_PTYS */
 	      /* If we can detect process termination, don't consider the
 		 process gone just because its pipe is closed.  */
-	      else if (nread == 0 && !NETCONN_P (proc) && !SERIALCONN_P (proc))
+	      else if (nread == 0 && !NETCONN_P (proc) && !SERIALCONN_P (proc)
+		       && !PIPECONN_P (proc))
 		;
 	      else
 		{
@@ -5964,7 +6218,8 @@ If PROCESS is a network or serial process, inhibit handling of incoming
 traffic.  */)
   (Lisp_Object process, Lisp_Object current_group)
 {
-  if (PROCESSP (process) && (NETCONN_P (process) || SERIALCONN_P (process)))
+  if (PROCESSP (process) && (NETCONN_P (process) || SERIALCONN_P (process)
+			     || PIPECONN_P (process)))
     {
       struct Lisp_Process *p;
 
@@ -5993,7 +6248,8 @@ If PROCESS is a network or serial process, resume handling of incoming
 traffic.  */)
   (Lisp_Object process, Lisp_Object current_group)
 {
-  if (PROCESSP (process) && (NETCONN_P (process) || SERIALCONN_P (process)))
+  if (PROCESSP (process) && (NETCONN_P (process) || SERIALCONN_P (process)
+			     || PIPECONN_P (process)))
     {
       struct Lisp_Process *p;
 
@@ -7040,7 +7296,7 @@ kill_buffer_processes (Lisp_Object buffer)
   FOR_EACH_PROCESS (tail, proc)
     if (NILP (buffer) || EQ (XPROCESS (proc)->buffer, buffer))
       {
-	if (NETCONN_P (proc) || SERIALCONN_P (proc))
+	if (NETCONN_P (proc) || SERIALCONN_P (proc) || PIPECONN_P (proc))
 	  Fdelete_process (proc);
 	else if (XPROCESS (proc)->infd >= 0)
 	  process_send_signal (proc, SIGHUP, Qnil, 1);
@@ -7340,6 +7596,7 @@ syms_of_process (void)
   DEFSYM (Qreal, "real");
   DEFSYM (Qnetwork, "network");
   DEFSYM (Qserial, "serial");
+  DEFSYM (Qpipe, "pipe");
   DEFSYM (QCbuffer, ":buffer");
   DEFSYM (QChost, ":host");
   DEFSYM (QCservice, ":service");
@@ -7356,6 +7613,7 @@ syms_of_process (void)
   DEFSYM (QCplist, ":plist");
   DEFSYM (QCcommand, ":command");
   DEFSYM (QCconnection_type, ":connection-type");
+  DEFSYM (QCstderr, ":stderr");
   DEFSYM (Qpty, "pty");
   DEFSYM (Qpipe, "pipe");
 
@@ -7461,6 +7719,7 @@ The variable takes effect when `start-process' is called.  */);
   defsubr (&Sset_process_plist);
   defsubr (&Sprocess_list);
   defsubr (&Smake_process);
+  defsubr (&Smake_pipe_process);
   defsubr (&Sserial_process_configure);
   defsubr (&Smake_serial_process);
   defsubr (&Sset_network_process_option);
diff --git a/src/process.h b/src/process.h
index 36979dc..92d7b64 100644
--- a/src/process.h
+++ b/src/process.h
@@ -105,6 +105,9 @@ struct Lisp_Process
     Lisp_Object gnutls_cred_type;
 #endif
 
+    /* Pipe process attached to standard error.  */
+    Lisp_Object stderr;
+
     /* After this point, there are no Lisp_Objects any more.  */
     /* alloc.c assumes that `pid' is the first such non-Lisp slot.  */
 
diff --git a/src/sysdep.c b/src/sysdep.c
index 0a0b0ac..2fe4624 100644
--- a/src/sysdep.c
+++ b/src/sysdep.c
@@ -2257,19 +2257,27 @@ emacs_fopen (char const *file, char const *mode)
 /* Create a pipe for Emacs use.  */
 
 int
-emacs_pipe (int fd[2])
+emacs_pipe2 (int fd[2], int flags)
 {
 #ifdef MSDOS
   return pipe (fd);
 #else  /* !MSDOS */
-  int result = pipe2 (fd, O_BINARY | O_CLOEXEC);
+  return pipe2 (fd, flags);
+#endif	/* !MSDOS */
+}
+
+int
+emacs_pipe (int fd[2])
+{
+  int result = emacs_pipe2 (fd, O_BINARY | O_CLOEXEC);
+#ifndef MSDOS
   if (! O_CLOEXEC && result == 0)
     {
       fcntl (fd[0], F_SETFD, FD_CLOEXEC);
       fcntl (fd[1], F_SETFD, FD_CLOEXEC);
     }
-  return result;
 #endif	/* !MSDOS */
+  return result;
 }
 
 /* Approximate posix_close and POSIX_CLOSE_RESTART well enough for Emacs.
diff --git a/test/ChangeLog b/test/ChangeLog
index 6a474e1..0f61e24 100644
--- a/test/ChangeLog
+++ b/test/ChangeLog
@@ -1,3 +1,8 @@
+2015-03-18  Daiki Ueno  <ueno@gnu.org>
+
+	* automated/process-tests.el (process-test-stderr-buffer)
+	(process-test-stderr-filter): New tests.
+
 2015-03-10  Jackson Ray Hamilton  <jackson@jacksonrayhamilton.com>
 
 	* indent/js-indent-init-dynamic.js: Fix spelling error.
diff --git a/test/automated/process-tests.el b/test/automated/process-tests.el
index dabfbc5..3389387 100644
--- a/test/automated/process-tests.el
+++ b/test/automated/process-tests.el
@@ -72,4 +72,69 @@
               (should (string= (buffer-string) "arg1 = \"x &y\", arg2 = \n"))))
         (when batfile (delete-file batfile))))))
 
+(ert-deftest process-test-stderr-buffer ()
+  (let* ((stdout-buffer (generate-new-buffer "*stdout*"))
+	 (stderr-buffer (generate-new-buffer "*stderr*"))
+	 (proc (make-process :name "test" :buffer stdout-buffer
+			     :command (list "bash" "-c"
+					    (concat "echo hello stdout!; "
+						    "echo hello stderr! >&2; "
+						    "exit 20"))
+			     :stderr stderr-buffer))
+	 (sentinel-called nil)
+	 (start-time (float-time)))
+    (while (not (or sentinel-called
+		    (> (- (float-time) start-time)
+		       process-test-sentinel-wait-timeout)))
+      (accept-process-output))
+    (cl-assert (eq (process-status proc) 'exit))
+    (cl-assert (= (process-exit-status proc) 20))
+    (should (with-current-buffer stdout-buffer
+	      (goto-char (point-min))
+	      (looking-at "hello stdout!")))
+    (should (with-current-buffer stderr-buffer
+	      (goto-char (point-min))
+	      (looking-at "hello stderr!")))))
+
+(ert-deftest process-test-stderr-filter ()
+  (let* ((sentinel-called nil)
+	 (stderr-sentinel-called nil)
+	 (stdout-input nil)
+	 (stderr-input nil)
+	 (stdout-buffer (generate-new-buffer "*stdout*"))
+	 (stderr-buffer (generate-new-buffer "*stderr*"))
+	 (stderr-proc (make-pipe-process :name "stderr"
+					 :buffer stderr-buffer))
+	 (proc (make-process :name "test" :buffer stdout-buffer
+			     :command (list "bash" "-c"
+					    (concat "echo hello stdout!; "
+						    "echo hello stderr! >&2; "
+						    "exit 20"))
+			     :stderr stderr-proc))
+	 (start-time (float-time)))
+    (set-process-filter proc (lambda (proc input)
+			       (push input stdout-input)))
+    (set-process-sentinel proc (lambda (proc msg)
+				 (setq sentinel-called t)))
+    (set-process-filter stderr-proc (lambda (proc input)
+				      (push input stderr-input)))
+    (set-process-sentinel stderr-proc (lambda (proc input)
+					(setq stderr-sentinel-called t)))
+    (while (not (or sentinel-called
+		    (> (- (float-time) start-time)
+		       process-test-sentinel-wait-timeout)))
+      (accept-process-output))
+    (cl-assert (eq (process-status proc) 'exit))
+    (cl-assert (= (process-exit-status proc) 20))
+    (should sentinel-called)
+    (should (equal 1 (with-current-buffer stdout-buffer
+		       (point-max))))
+    (should (equal "hello stdout!\n"
+		   (mapconcat #'identity (nreverse stdout-input) "")))
+    (should stderr-sentinel-called)
+    (should (equal 1 (with-current-buffer stderr-buffer
+		       (point-max))))
+    (should (equal "hello stderr!\n"
+		   (mapconcat #'identity (nreverse stderr-input) "")))))
+
 (provide 'process-tests)
-- 
2.1.0

Re: [PATCH] Generalize start-process with keyword args

[Stefan Monnier]

> I tend to agree with Stefan, as I find no documentation about the
> implication of no-conversion for ":coding nil" except in the C source
> code.  Maybe we could add the exception back if it turns out to be too
> confusing.

We should probably also declare that uses of nil in other similar
circumstances is deprecated (I actually don't know where are those
places where nil means "binary").

> +If @var{coding} is @code{nil}, the coding system chosen for decoding
> +output is @code{undecided}, meaning deduce the encoding from the
> +actual data.

Doesn't it also depend on coding-system-for-read and coding-system-for-write?

> +  tem = Fplist_get (contact, QCconnection_type);
> +  if (!NILP (tem))
> +    {
> +      if (EQ (tem, Qpty))
> +	XPROCESS (proc)->pty_flag = 1;
> +      else if (EQ (tem, Qpipe))
> +	XPROCESS (proc)->pty_flag = 0;

Please use `true' and `false' instead of 0 and 1.

> +      else
> +	report_file_error ("Unknown connection type", tem);
> +    }
> +  else
> +    XPROCESS (proc)->pty_flag = !NILP (Vprocess_connection_type);

Any reason why you nested the "if/elseif/else" inside the outer "if"
instead of having a 4-way "if/elseif/elseif/else"?
[ Either way is fine by me, I'm just being curious here.  ]


        Stefan

Re: [PATCH] Generalize start-process with keyword args

[Eli Zaretskii]

> From: Daiki Ueno <ueno@gnu.org>
> Cc: Stefan Monnier <monnier@IRO.UMontreal.CA>,  emacs-devel@gnu.org
> Date: Wed, 18 Mar 2015 15:17:39 +0900
> 
> > IMO, it will be terribly confusing to have incompatible treatment of
> > nil in this one API.
> 
> I tend to agree with Stefan, as I find no documentation about the
> implication of no-conversion for ":coding nil" except in the C source
> code.

Since you mentioned the documentation, you seem to interpret
"confusing" as in "confusing Lisp programmers".  But that's not what I
meant: I meant that this unfortunate convention will confuse Lisp
programs which call this API instead or together with the existing
APIs.  Despite what you might think, this convention _is_ used in
several places in Emacs sources; I've bumped into it in the past.  If
'make-process' wants to be compatible with existing interfaces, it
should use the same conventions.

For example, imagine some wrapper around 'make-process' that generates
the coding systems from some data, like MIME charset or something.
Such programs usually test the validity of the result by calling
coding-system-p (which returns t for nil), and if that returns OK, go
ahead and use the resulting coding system.  Imagine the confusion if
'make-process' rejects such values, or assigns its own semantics to
some of them.

I could support a thorough eradication of this convention from all
related interfaces.  But doing that for a single interface is not TRT,
IMO, as it will increase the memory pressure on the Lisp programmers,
which will now have to remember 2 different conventions about this,
and also which API supports which semantics of nil there.  Moreover,
if we ever want to replace 'start-process' with 'make-process' in some
it the users of the former, we will now have to remember to add code
that, when given nil, passes 'binary'.  That's worse than bad
conventions, IMO.

Re: [PATCH] Add facility to collect stderr of async subprocess

[Eli Zaretskii]

> From: Daiki Ueno <ueno@gnu.org>
> Date: Wed, 18 Mar 2015 16:37:08 +0900
> 
> >> In any case, supporting stderr could be a starting point, as it seems to
> >> be a long-standing request:
> >> https://lists.gnu.org/archive/html/emacs-devel/2004-04/msg01051.html
> >> and it wouldn't involve a portability issue.
> >
> > Redirecting the child's stderr is already supported on Windows, so
> > this is only a matter of having the higher layers DTRT.
> 
> So, let's start with small (but common) things.  The attached patch adds
> a new keyword `:stderr' to `make-process'.  The argument can be either a
> buffer or a (newly introduced) pipe process.

Do you expect this to work on Windows?  If so, then there's at least
some part of "having higher layers DTRT" I failed to adequately
explain ;-)

Redirection of stderr does indeed work on the lowest level of creating
subprocesses on MS-Windows, but we don't expect to read from a pipe
connected to child's stderr, and thus the 'pselect' emulation watches
only one file handle per sub-process.  This needs to be expanded.

In addition, this:

>   (let ((stderr (make-pipe-process :name "stderr")))
>     (make-process :name "test"
>                   :buffer (current-buffer)
>                   :command (list (expand-file-name "./test.sh"))
>                   :stderr stderr)
>     (set-process-filter stderr ...))

introduces a new concept whereby a process object ('stderr' in this
case) can be passed to another subprocess, which AFAIK was not
possible until now, at least not on MS-Windows.

Volunteers are welcome to work on this, of course.

Re: [PATCH] Generalize start-process with keyword args

[Eli Zaretskii]

> From: Stefan Monnier <monnier@IRO.UMontreal.CA>
> Cc: Eli Zaretskii <eliz@gnu.org>, emacs-devel@gnu.org
> Date: Wed, 18 Mar 2015 09:03:19 -0400
> 
> > I tend to agree with Stefan, as I find no documentation about the
> > implication of no-conversion for ":coding nil" except in the C source
> > code.  Maybe we could add the exception back if it turns out to be too
> > confusing.
> 
> We should probably also declare that uses of nil in other similar
> circumstances is deprecated

It won't help much, because AFAIR most or all of those places are not
on the user or Lisp program level, they are deep in the bowels, and
used during initialization AFAIR.

Re: [PATCH] Generalize start-process with keyword args

[Stefan Monnier]

> I could support a thorough eradication of this convention from all
> related interfaces.

Then let's do that.


        Stefan

Re: [PATCH] Generalize start-process with keyword args

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Daiki Ueno <ueno@gnu.org>,  emacs-devel@gnu.org
> Date: Wed, 18 Mar 2015 14:57:18 -0400
> 
> > I could support a thorough eradication of this convention from all
> > related interfaces.
> 
> Then let's do that.

Yes, let's.

Re: [PATCH] Generalize start-process with keyword args

[Daiki Ueno]

Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

>> +If @var{coding} is @code{nil}, the coding system chosen for decoding
>> +output is @code{undecided}, meaning deduce the encoding from the
>> +actual data.
>
> Doesn't it also depend on coding-system-for-read and coding-system-for-write?

Also `find-operation-coding-system' and multibyteness of the buffer.
But do we really want to describe all the possibilities there?

Similarly, since the make-*-process functions share common keywords
(:filter, :sentinel, etc), can we arrange them in a single section and
refer to it from those function documentation?

> Any reason why you nested the "if/elseif/else" inside the outer "if"
> instead of having a 4-way "if/elseif/elseif/else"?
> [ Either way is fine by me, I'm just being curious here.  ]

Maybe I just wanted to have clear distinction between nil and non-nil
cases, since we changed the meaning during the review.  Fixed.

Regards,
-- 
Daiki Ueno

Re: [PATCH] Generalize start-process with keyword args

[Stefan Monnier]

>>> +If @var{coding} is @code{nil}, the coding system chosen for decoding
>>> +output is @code{undecided}, meaning deduce the encoding from the
>>> +actual data.
>> Doesn't it also depend on coding-system-for-read and coding-system-for-write?
> Also `find-operation-coding-system' and multibyteness of the buffer.
> But do we really want to describe all the possibilities there?

No, we probably don't, but we shouldn't state something that is false.
Instead we should be less specific and point to some place that does
give the full story.

> Similarly, since the make-*-process functions share common keywords
> (:filter, :sentinel, etc), can we arrange them in a single section and
> refer to it from those function documentation?

Sure.


        Stefan

Re: Generalize start-process with keyword args

[Eli Zaretskii]

> From: Andy Moreton <andrewjmoreton@gmail.com>
> Date: Tue, 17 Mar 2015 22:04:26 +0000
> 
> > I see a lot of complications with the semantics of this, and no easy
> > solutions, since the necessary tools are not available on Windows,
> > could be installed in any directory, and come with a set of
> > assumptions that don't necessarily play well with a native Emacs.
> 
> I agree that this is awkward. What I envisaged was an (optional) alist
> of (REGEXP . (INTERPRETER INTERPRETER-ARGS)) pairs such that if
> start-process is onvoked for a PROGRAM that matches REGEXP, then use
> INTERPRETER as the PROGRAM argument to start-process, and prepend
> INTERPRETER-ARGS to the PROGRAM-ARGS list.

But that means this alist will have to be updated for each new script
you want to handle, no?  How useful is that?

Re: Generalize start-process with keyword args

[Andy Moreton]

On Thu 19 Mar 2015, Eli Zaretskii wrote:

>> From: Andy Moreton <andrewjmoreton@gmail.com>
>> Date: Tue, 17 Mar 2015 22:04:26 +0000
>> 
>> > I see a lot of complications with the semantics of this, and no easy
>> > solutions, since the necessary tools are not available on Windows,
>> > could be installed in any directory, and come with a set of
>> > assumptions that don't necessarily play well with a native Emacs.
>> 
>> I agree that this is awkward. What I envisaged was an (optional) alist
>> of (REGEXP . (INTERPRETER INTERPRETER-ARGS)) pairs such that if
>> start-process is onvoked for a PROGRAM that matches REGEXP, then use
>> INTERPRETER as the PROGRAM argument to start-process, and prepend
>> INTERPRETER-ARGS to the PROGRAM-ARGS list.
>
> But that means this alist will have to be updated for each new script
> you want to handle, no?  How useful is that?

This issue is clearly a minority concern, so feel free to disregard it.

    AndyM

Re: Generalize start-process with keyword args

[Stefan Monnier]

>>> I agree that this is awkward. What I envisaged was an (optional) alist
>>> of (REGEXP . (INTERPRETER INTERPRETER-ARGS)) pairs such that if
>>> start-process is onvoked for a PROGRAM that matches REGEXP, then use
>>> INTERPRETER as the PROGRAM argument to start-process, and prepend
>>> INTERPRETER-ARGS to the PROGRAM-ARGS list.
>> But that means this alist will have to be updated for each new script
>> you want to handle, no?  How useful is that?
> This issue is clearly a minority concern, so feel free to disregard it.

BTW, in the absence of this alist in Emacs, you can code it up yourself
with a quick advice like (guaranteed 100% untested):

   (defun handle-scripts-in-start-process (args)
     (if (string-match REGEXP (nth 2 args))
         `(,(nth 0 args) (nth 1 args) "bash" "-c" ,@(nthcdr 2 args))
       args))
   (avice-add 'start-process :filter-args #'handle-scripts-in-start-process)


-- Stefan

Re: [PATCH] Generalize start-process with keyword args

[Daiki Ueno]

Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

> No, we probably don't, but we shouldn't state something that is false.
> Instead we should be less specific and point to some place that does
> give the full story.

I see.  I've added an xref to "Default Coding Systems" and pushed the
patch.

>> Similarly, since the make-*-process functions share common keywords
>> (:filter, :sentinel, etc), can we arrange them in a single section and
>> refer to it from those function documentation?
>
> Sure.

Okay; let's do that as a separate commit.

Regards,
--
Daiki Ueno

Re: [PATCH] Add facility to collect stderr of async subprocess

[Daiki Ueno]

Hello Eli,

Eli Zaretskii <eliz@gnu.org> writes:

>> So, let's start with small (but common) things.  The attached patch adds
>> a new keyword `:stderr' to `make-process'.  The argument can be either a
>> buffer or a (newly introduced) pipe process.
>
> Do you expect this to work on Windows?  If so, then there's at least
> some part of "having higher layers DTRT" I failed to adequately
> explain ;-)
>
> Redirection of stderr does indeed work on the lowest level of creating
> subprocesses on MS-Windows, but we don't expect to read from a pipe
> connected to child's stderr, and thus the 'pselect' emulation watches
> only one file handle per sub-process.  This needs to be expanded.

Sorry for my ignorance about Windows and thanks for writing up the
detail comment in commit 22ece83a.  It is very helpful.

> In addition, this:
>
>>   (let ((stderr (make-pipe-process :name "stderr")))
>>     (make-process :name "test"
>>                   :buffer (current-buffer)
>>                   :command (list (expand-file-name "./test.sh"))
>>                   :stderr stderr)
>>     (set-process-filter stderr ...))
>
> introduces a new concept whereby a process object ('stderr' in this
> case) can be passed to another subprocess, which AFAIK was not
> possible until now, at least not on MS-Windows.

I don't get your concern here.  The above code is merely meant to pass
the actual FD of the child end of pipe, through an opaque Lisp object
`stderr'.

> Volunteers are welcome to work on this, of course.

I've just set up MSYS2 and tried the Emacs w64 build instruction[1] (and
also obtained a copy of Hart's book).  Let me try again later.

Footnotes: 
[1]  http://sourceforge.net/p/emacsbinw64/wiki/Build%20guideline%20for%20MSYS2-MinGW-w64%20system/

Regards,
-- 
Daiki Ueno

Re: [PATCH] Add facility to collect stderr of async subprocess

[Eli Zaretskii]

> From: Daiki Ueno <ueno@gnu.org>
> Cc: emacs-devel@gnu.org
> Date: Tue, 31 Mar 2015 16:27:10 +0900
> 
> > In addition, this:
> >
> >>   (let ((stderr (make-pipe-process :name "stderr")))
> >>     (make-process :name "test"
> >>                   :buffer (current-buffer)
> >>                   :command (list (expand-file-name "./test.sh"))
> >>                   :stderr stderr)
> >>     (set-process-filter stderr ...))
> >
> > introduces a new concept whereby a process object ('stderr' in this
> > case) can be passed to another subprocess, which AFAIK was not
> > possible until now, at least not on MS-Windows.
> 
> I don't get your concern here.  The above code is merely meant to pass
> the actual FD of the child end of pipe, through an opaque Lisp object
> `stderr'.

Then perhaps there's no problem.

However, please note that the call to emacs_pipe2 in make-pipe-process
will hit an assertion in w32.c's implementation of pipe2.

> I've just set up MSYS2 and tried the Emacs w64 build instruction[1] (and
> also obtained a copy of Hart's book).  Let me try again later.
> 
> Footnotes: 
> [1]  http://sourceforge.net/p/emacsbinw64/wiki/Build%20guideline%20for%20MSYS2-MinGW-w64%20system/

I'd be happier if you used nt/INSTALL instead, and reported any
changes/additions needed for supporting MSYS2.

Thanks.

Re: [PATCH] Add facility to collect stderr of async subprocess

[Daiki Ueno]

[-- Attachment #1: Type: text/plain, Size: 973 bytes --]

Eli Zaretskii <eliz@gnu.org> writes:

> However, please note that the call to emacs_pipe2 in make-pipe-process
> will hit an assertion in w32.c's implementation of pipe2.

OK, I added a wrapper around pipe2 in w32.c to preserve the assert
condition.  With the attached patch, I confirmed that the included test
passes on both Windows and GNU/Linux.

Other changes from v1 are:

- Change field name 'stderr' to 'stderrproc', since 'stderr' is defined
  as a macro on MinGW.

- Stop explicit deactivation of a pipe process when the attached (real)
  process terminates.  Instead, deativate the pipe process immediately
  when an EOF is detected.

- Add a new function 'pipe_open' analogous to 'serial_open', which has
  different definitions on Windows and on GNU/Linux

I guess that there is still room for improvement (e.g. check if a child
process closed stderr before exit), but I'd like to ask if this approach
is acceptable in the first place.

Regards,
-- 
Daiki Ueno

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: v2-0001-Add-facility-to-collect-stderr-of-async-subproces.patch --]
[-- Type: text/x-diff, Size: 29961 bytes --]

From 554f6f95f3732e7da859006ca1f699533c39a315 Mon Sep 17 00:00:00 2001
From: Daiki Ueno <ueno@gnu.org>
Date: Tue, 7 Apr 2015 17:42:09 +0900
Subject: [PATCH v2] Add facility to collect stderr of async subprocess

* src/sysdep.c (pipe_open) [!WINDOWSNT]: New function.
* src/w32.c (pipe_open): New function.
(sys_pipe2): New static function, renamed from pipe2.
(pipe2): Call sys_pipe2, with an assertion that both O_BINARY and
O_CLOEXEC are set.
* src/process.h (struct Lisp_Process): New member stderrproc.
(pipe_open): New function declaration.
* src/process.c (PIPECONN_P): New macro.
(PIPECONN1_P): New macro.
(Fdelete_process, Fprocess_status, Fset_process_buffer)
(Fset_process_filter, Fset_process_sentinel, Fstop_process)
(Fcontinue_process): Handle pipe process specially.
(create_process): Respect p->stderrproc.
(Fmake_pipe_process): New function.
(Fmake_process): Add new keyword argument :stderr.
(wait_reading_process_output): Specially handle a pipe process when
it gets an EOF.
(syms_of_process): Register Qpipe and Smake_pipe_process.

* doc/lispref/processes.texi (Asynchronous Processes): Mention
`make-pipe-process' and `:stderr' keyword of `make-process'.

* test/automated/process-tests.el (process-test-stderr-buffer)
(process-test-stderr-filter): New tests.

* etc/NEWS: Mention pipe process.
---
 doc/lispref/processes.texi      |  52 +++++++
 etc/NEWS                        |   4 +
 src/process.c                   | 298 +++++++++++++++++++++++++++++++++++++---
 src/process.h                   |   7 +
 src/sysdep.c                    |  36 +++++
 src/w32.c                       |  68 ++++++++-
 test/automated/process-tests.el |  67 +++++++++
 7 files changed, 507 insertions(+), 25 deletions(-)

diff --git a/doc/lispref/processes.texi b/doc/lispref/processes.texi
index 3e9cc50..59bc846 100644
--- a/doc/lispref/processes.texi
+++ b/doc/lispref/processes.texi
@@ -741,6 +741,58 @@ Initialize the process filter to @var{filter}.
 
 @item :sentinel @var{sentinel}
 Initialize the process sentinel to @var{sentinel}.
+
+@item :stderr @var{stderr}
+Associate @var{stderr} with the standard error of the process.
+@var{stderr} is either a buffer or a pipe process created with
+@code{make-pipe-process}.
+@end table
+
+The original argument list, modified with the actual connection
+information, is available via the @code{process-contact} function.
+@end defun
+
+@defun make-pipe-process &rest args
+This function creates a bidirectional pipe which can be attached to a
+child process (typically through the @code{:stderr} keyword of
+@code{make-process}).
+
+The arguments @var{args} are a list of keyword/argument pairs.
+Omitting a keyword is always equivalent to specifying it with value
+@code{nil}, except for @code{:coding}.
+Here are the meaningful keywords:
+
+@table @asis
+@item :name @var{name}
+Use the string @var{name} as the process name.  It is modified if
+necessary to make it unique.
+
+@item :buffer @var{buffer}
+Use @var{buffer} as the process buffer.
+
+@item :coding @var{coding}
+If @var{coding} is a symbol, it specifies the coding system to be
+used for both reading and writing of data from and to the
+connection.  If @var{coding} is a cons cell
+@w{@code{(@var{decoding} . @var{encoding})}}, then @var{decoding}
+will be used for reading and @var{encoding} for writing.
+
+If @var{coding} is @code{nil}, the default rules for finding the
+coding system will apply.  @xref{Default Coding Systems}.
+
+@item :noquery @var{query-flag}
+Initialize the process query flag to @var{query-flag}.
+@xref{Query Before Exit}.
+
+@item :stop @var{stopped}
+If @var{stopped} is non-@code{nil}, start the process in the
+``stopped'' state.
+
+@item :filter @var{filter}
+Initialize the process filter to @var{filter}.
+
+@item :sentinel @var{sentinel}
+Initialize the process sentinel to @var{sentinel}.
 @end table
 
 The original argument list, modified with the actual connection
diff --git a/etc/NEWS b/etc/NEWS
index 0332fc5..cf26bbf 100644
--- a/etc/NEWS
+++ b/etc/NEWS
@@ -671,6 +671,10 @@ word syntax, use `\sw' instead.
 \f
 * Lisp Changes in Emacs 25.1
 
+** New process type `pipe', which can be used in combination with the
+   `:stderr' keyword of make-process to collect standard error output
+   of subprocess.
+
 ** New function `make-process' provides an alternative interface to
 `start-process'.  It allows programs to set process parameters such as
 process filter, sentinel, etc., through keyword arguments (similar to
diff --git a/src/process.c b/src/process.c
index 2800fa5..00a5f4d 100644
--- a/src/process.c
+++ b/src/process.c
@@ -189,6 +189,8 @@ process_socket (int domain, int type, int protocol)
 #define NETCONN1_P(p) (EQ (p->type, Qnetwork))
 #define SERIALCONN_P(p) (EQ (XPROCESS (p)->type, Qserial))
 #define SERIALCONN1_P(p) (EQ (p->type, Qserial))
+#define PIPECONN_P(p) (EQ (XPROCESS (p)->type, Qpipe))
+#define PIPECONN1_P(p) (EQ (p->type, Qpipe))
 
 /* Number of events of change of status of a process.  */
 static EMACS_INT process_tick;
@@ -411,6 +413,11 @@ pset_write_queue (struct Lisp_Process *p, Lisp_Object val)
 {
   p->write_queue = val;
 }
+static void
+pset_stderrproc (struct Lisp_Process *p, Lisp_Object val)
+{
+  p->stderrproc = val;
+}
 
 \f
 static Lisp_Object
@@ -837,7 +844,7 @@ nil, indicating the current buffer's process.  */)
   p = XPROCESS (process);
 
   p->raw_status_new = 0;
-  if (NETCONN1_P (p) || SERIALCONN1_P (p))
+  if (NETCONN1_P (p) || SERIALCONN1_P (p) || PIPECONN1_P (p))
     {
       pset_status (p, list2 (Qexit, make_number (0)));
       p->tick = ++process_tick;
@@ -903,7 +910,7 @@ nil, indicating the current buffer's process.  */)
   status = p->status;
   if (CONSP (status))
     status = XCAR (status);
-  if (NETCONN1_P (p) || SERIALCONN1_P (p))
+  if (NETCONN1_P (p) || SERIALCONN1_P (p) || PIPECONN1_P (p))
     {
       if (EQ (status, Qexit))
 	status = Qclosed;
@@ -987,7 +994,7 @@ Return BUFFER.  */)
     CHECK_BUFFER (buffer);
   p = XPROCESS (process);
   pset_buffer (p, buffer);
-  if (NETCONN1_P (p) || SERIALCONN1_P (p))
+  if (NETCONN1_P (p) || SERIALCONN1_P (p) || PIPECONN1_P (p))
     pset_childp (p, Fplist_put (p->childp, QCbuffer, buffer));
   setup_process_coding_systems (process);
   return buffer;
@@ -1063,7 +1070,7 @@ The string argument is normally a multibyte string, except:
     }
 
   pset_filter (p, filter);
-  if (NETCONN1_P (p) || SERIALCONN1_P (p))
+  if (NETCONN1_P (p) || SERIALCONN1_P (p) || PIPECONN1_P (p))
     pset_childp (p, Fplist_put (p->childp, QCfilter, filter));
   setup_process_coding_systems (process);
   return filter;
@@ -1095,7 +1102,7 @@ It gets two arguments: the process, and a string describing the change.  */)
     sentinel = Qinternal_default_process_sentinel;
 
   pset_sentinel (p, sentinel);
-  if (NETCONN1_P (p) || SERIALCONN1_P (p))
+  if (NETCONN1_P (p) || SERIALCONN1_P (p) || PIPECONN1_P (p))
     pset_childp (p, Fplist_put (p->childp, QCsentinel, sentinel));
   return sentinel;
 }
@@ -1183,13 +1190,13 @@ DEFUN ("process-query-on-exit-flag",
 DEFUN ("process-contact", Fprocess_contact, Sprocess_contact,
        1, 2, 0,
        doc: /* Return the contact info of PROCESS; t for a real child.
-For a network or serial connection, the value depends on the optional
-KEY arg.  If KEY is nil, value is a cons cell of the form (HOST
-SERVICE) for a network connection or (PORT SPEED) for a serial
-connection.  If KEY is t, the complete contact information for the
-connection is returned, else the specific value for the keyword KEY is
-returned.  See `make-network-process' or `make-serial-process' for a
-list of keywords.  */)
+For a network or serial connection, or a pipe, the value depends on
+the optional KEY arg.  If KEY is nil, value is a cons cell of the form
+\(HOST SERVICE) for a network connection or (PORT SPEED) for a serial
+connection, or (INPUT OUTPUT) for a pipe.  If KEY is t, the complete
+contact information for the connection is returned, else the specific
+value for the keyword KEY is returned.  See `make-network-process' or
+`make-serial-process' for a list of keywords.  */)
   (register Lisp_Object process, Lisp_Object key)
 {
   Lisp_Object contact;
@@ -1386,10 +1393,15 @@ to use a pty, or nil to use the default specified through
 
 :sentinel SENTINEL -- Install SENTINEL as the process sentinel.
 
+:stderr STDERR -- STDERR is either a buffer or a pipe process attached
+to the standard error of subprocess.  Specifying this implies
+`:connection-type' is set to `pipe'.
+
 usage: (make-process &rest ARGS)  */)
   (ptrdiff_t nargs, Lisp_Object *args)
 {
   Lisp_Object buffer, name, command, program, proc, contact, current_dir, tem;
+  Lisp_Object xstderr, stderrproc;
   ptrdiff_t count = SPECPDL_INDEX ();
   struct gcpro gcpro1;
   USE_SAFE_ALLOCA;
@@ -1433,6 +1445,27 @@ usage: (make-process &rest ARGS)  */)
   if (!NILP (program))
     CHECK_STRING (program);
 
+  stderrproc = Qnil;
+  xstderr = Fplist_get (contact, QCstderr);
+  if (PROCESSP (xstderr))
+    {
+      if (!PIPECONN_P (xstderr))
+	error ("Process is not a pipe process");
+      stderrproc = xstderr;
+    }
+  else if (!NILP (xstderr))
+    {
+      struct gcpro gcpro1, gcpro2;
+      CHECK_STRING (program);
+      GCPRO2 (buffer, current_dir);
+      stderrproc = CALLN (Fmake_pipe_process,
+			  QCname,
+			  concat2 (name, build_string (" stderr")),
+			  QCbuffer,
+			  Fget_buffer_create (xstderr));
+      UNGCPRO;
+    }
+
   proc = make_process (name);
   /* If an error occurs and we can't start the process, we want to
      remove it from the process list.  This means that each error
@@ -1463,6 +1496,13 @@ usage: (make-process &rest ARGS)  */)
   else
     report_file_error ("Unknown connection type", tem);
 
+  if (!NILP (stderrproc))
+    {
+      pset_stderrproc (XPROCESS (proc), stderrproc);
+
+      XPROCESS (proc)->pty_flag = false;
+    }
+
 #ifdef HAVE_GNUTLS
   /* AKA GNUTLS_INITSTAGE(proc).  */
   XPROCESS (proc)->gnutls_initstage = GNUTLS_STAGE_EMPTY;
@@ -1705,10 +1745,10 @@ static void
 create_process (Lisp_Object process, char **new_argv, Lisp_Object current_dir)
 {
   struct Lisp_Process *p = XPROCESS (process);
-  int inchannel, outchannel;
+  int inchannel, outchannel, errchannel = -1;
   pid_t pid;
   int vfork_errno;
-  int forkin, forkout;
+  int forkin, forkout, forkerr = -1;
   bool pty_flag = 0;
   char pty_name[PTY_NAME_SIZE];
   Lisp_Object lisp_pty_name = Qnil;
@@ -1746,6 +1786,21 @@ create_process (Lisp_Object process, char **new_argv, Lisp_Object current_dir)
       outchannel = p->open_fd[WRITE_TO_SUBPROCESS];
       inchannel = p->open_fd[READ_FROM_SUBPROCESS];
       forkout = p->open_fd[SUBPROCESS_STDOUT];
+
+      if (!NILP (p->stderrproc))
+	{
+	  struct Lisp_Process *pp = XPROCESS (p->stderrproc);
+
+	  forkerr = pp->open_fd[SUBPROCESS_STDOUT];
+	  errchannel = pp->open_fd[READ_FROM_SUBPROCESS];
+
+	  /* FORKERR will be redirected in child_setup.  */
+	  fcntl (forkerr, F_SETFD, FD_CLOEXEC);
+
+	  /* Close unnecessary file descriptors.  */
+	  close_process_fd (&pp->open_fd[WRITE_TO_SUBPROCESS]);
+	  close_process_fd (&pp->open_fd[SUBPROCESS_STDIN]);
+	}
     }
 
 #ifndef WINDOWSNT
@@ -1792,6 +1847,7 @@ create_process (Lisp_Object process, char **new_argv, Lisp_Object current_dir)
     char **volatile new_argv_volatile = new_argv;
     int volatile forkin_volatile = forkin;
     int volatile forkout_volatile = forkout;
+    int volatile forkerr_volatile = forkerr;
     struct Lisp_Process *p_volatile = p;
 
     pid = vfork ();
@@ -1801,6 +1857,7 @@ create_process (Lisp_Object process, char **new_argv, Lisp_Object current_dir)
     new_argv = new_argv_volatile;
     forkin = forkin_volatile;
     forkout = forkout_volatile;
+    forkerr = forkerr_volatile;
     p = p_volatile;
 
     pty_flag = p->pty_flag;
@@ -1811,6 +1868,7 @@ create_process (Lisp_Object process, char **new_argv, Lisp_Object current_dir)
     {
       int xforkin = forkin;
       int xforkout = forkout;
+      int xforkerr = forkerr;
 
       /* Make the pty be the controlling terminal of the process.  */
 #ifdef HAVE_PTYS
@@ -1910,10 +1968,13 @@ create_process (Lisp_Object process, char **new_argv, Lisp_Object current_dir)
 
       if (pty_flag)
 	child_setup_tty (xforkout);
+
+      if (xforkerr < 0)
+	xforkerr = xforkout;
 #ifdef WINDOWSNT
-      pid = child_setup (xforkin, xforkout, xforkout, new_argv, 1, current_dir);
+      pid = child_setup (xforkin, xforkout, xforkerr, new_argv, 1, current_dir);
 #else  /* not WINDOWSNT */
-      child_setup (xforkin, xforkout, xforkout, new_argv, 1, current_dir);
+      child_setup (xforkin, xforkout, xforkerr, new_argv, 1, current_dir);
 #endif /* not WINDOWSNT */
     }
 
@@ -1958,6 +2019,11 @@ create_process (Lisp_Object process, char **new_argv, Lisp_Object current_dir)
 	close_process_fd (&p->open_fd[READ_FROM_EXEC_MONITOR]);
       }
 #endif
+      if (!NILP (p->stderrproc))
+	{
+	  struct Lisp_Process *pp = XPROCESS (p->stderrproc);
+	  close_process_fd (&pp->open_fd[SUBPROCESS_STDOUT]);
+	}
     }
 }
 
@@ -2016,6 +2082,180 @@ create_pty (Lisp_Object process)
   p->pid = -2;
 }
 
+DEFUN ("make-pipe-process", Fmake_pipe_process, Smake_pipe_process,
+       0, MANY, 0,
+       doc: /* Create and return a bidirectional pipe process.
+
+In Emacs, pipes are represented by process objects, so input and
+output work as for subprocesses, and `delete-process' closes a pipe.
+However, a pipe process has no process id, it cannot be signaled,
+and the status codes are different from normal processes.
+
+Arguments are specified as keyword/argument pairs.  The following
+arguments are defined:
+
+:name NAME -- NAME is the name of the process.  It is modified if necessary to make it unique.
+
+:buffer BUFFER -- BUFFER is the buffer (or buffer-name) to associate
+with the process.  Process output goes at the end of that buffer,
+unless you specify an output stream or filter function to handle the
+output.  If BUFFER is not given, the value of NAME is used.
+
+:coding CODING -- If CODING is a symbol, it specifies the coding
+system used for both reading and writing for this process.  If CODING
+is a cons (DECODING . ENCODING), DECODING is used for reading, and
+ENCODING is used for writing.
+
+:noquery BOOL -- When exiting Emacs, query the user if BOOL is nil and
+the process is running.  If BOOL is not given, query before exiting.
+
+:stop BOOL -- Start process in the `stopped' state if BOOL non-nil.
+In the stopped state, a pipe process does not accept incoming data,
+but you can send outgoing data.  The stopped state is cleared by
+`continue-process' and set by `stop-process'.
+
+:filter FILTER -- Install FILTER as the process filter.
+
+:sentinel SENTINEL -- Install SENTINEL as the process sentinel.
+
+usage:  (make-pipe-process &rest ARGS)  */)
+  (ptrdiff_t nargs, Lisp_Object *args)
+{
+  Lisp_Object proc, contact;
+  struct Lisp_Process *p;
+  struct gcpro gcpro1;
+  Lisp_Object name, buffer;
+  Lisp_Object tem, val;
+  ptrdiff_t specpdl_count;
+  int inchannel, outchannel;
+
+  if (nargs == 0)
+    return Qnil;
+
+  contact = Flist (nargs, args);
+  GCPRO1 (contact);
+
+  name = Fplist_get (contact, QCname);
+  CHECK_STRING (name);
+  proc = make_process (name);
+  specpdl_count = SPECPDL_INDEX ();
+  record_unwind_protect (remove_process, proc);
+  p = XPROCESS (proc);
+
+  if (pipe_open (p->open_fd + SUBPROCESS_STDIN,
+		 p->open_fd + READ_FROM_SUBPROCESS) != 0)
+    report_file_error ("Creating pipe", Qnil);
+  outchannel = p->open_fd[WRITE_TO_SUBPROCESS];
+  inchannel = p->open_fd[READ_FROM_SUBPROCESS];
+
+  /* Record this as an active process, with its channels.  */
+  chan_process[inchannel] = proc;
+  p->infd = inchannel;
+  p->outfd = outchannel;
+
+  if (inchannel > max_process_desc)
+    max_process_desc = inchannel;
+
+  buffer = Fplist_get (contact, QCbuffer);
+  if (NILP (buffer))
+    buffer = name;
+  buffer = Fget_buffer_create (buffer);
+  pset_buffer (p, buffer);
+
+  pset_childp (p, contact);
+  pset_plist (p, Fcopy_sequence (Fplist_get (contact, QCplist)));
+  pset_type (p, Qpipe);
+  pset_sentinel (p, Fplist_get (contact, QCsentinel));
+  pset_filter (p, Fplist_get (contact, QCfilter));
+  pset_log (p, Qnil);
+  if (tem = Fplist_get (contact, QCnoquery), !NILP (tem))
+    p->kill_without_query = 1;
+  if (tem = Fplist_get (contact, QCstop), !NILP (tem))
+    pset_command (p, Qt);
+  eassert (! p->pty_flag);
+
+  if (!EQ (p->command, Qt))
+    {
+      FD_SET (inchannel, &input_wait_mask);
+      FD_SET (inchannel, &non_keyboard_wait_mask);
+    }
+#ifdef ADAPTIVE_READ_BUFFERING
+  p->adaptive_read_buffering
+    = (NILP (Vprocess_adaptive_read_buffering) ? 0
+       : EQ (Vprocess_adaptive_read_buffering, Qt) ? 1 : 2);
+#endif
+
+  /* Make the process marker point into the process buffer (if any).  */
+  if (BUFFERP (buffer))
+    set_marker_both (p->mark, buffer,
+		     BUF_ZV (XBUFFER (buffer)),
+		     BUF_ZV_BYTE (XBUFFER (buffer)));
+
+  {
+    /* Setup coding systems for communicating with the network stream.  */
+    struct gcpro gcpro1;
+    /* Qt denotes we have not yet called Ffind_operation_coding_system.  */
+    Lisp_Object coding_systems = Qt;
+    Lisp_Object val;
+
+    tem = Fplist_get (contact, QCcoding);
+    val = Qnil;
+    if (!NILP (tem))
+      {
+	val = tem;
+	if (CONSP (val))
+	  val = XCAR (val);
+      }
+    else if (!NILP (Vcoding_system_for_read))
+      val = Vcoding_system_for_read;
+    else if ((!NILP (buffer) && NILP (BVAR (XBUFFER (buffer), enable_multibyte_characters)))
+	     || (NILP (buffer) && NILP (BVAR (&buffer_defaults, enable_multibyte_characters))))
+      /* We dare not decode end-of-line format by setting VAL to
+	 Qraw_text, because the existing Emacs Lisp libraries
+	 assume that they receive bare code including a sequence of
+	 CR LF.  */
+      val = Qnil;
+    else
+      {
+	if (CONSP (coding_systems))
+	  val = XCAR (coding_systems);
+	else if (CONSP (Vdefault_process_coding_system))
+	  val = XCAR (Vdefault_process_coding_system);
+	else
+	  val = Qnil;
+      }
+    pset_decode_coding_system (p, val);
+
+    if (!NILP (tem))
+      {
+	val = tem;
+	if (CONSP (val))
+	  val = XCDR (val);
+      }
+    else if (!NILP (Vcoding_system_for_write))
+      val = Vcoding_system_for_write;
+    else if (NILP (BVAR (current_buffer, enable_multibyte_characters)))
+      val = Qnil;
+    else
+      {
+	if (CONSP (coding_systems))
+	  val = XCDR (coding_systems);
+	else if (CONSP (Vdefault_process_coding_system))
+	  val = XCDR (Vdefault_process_coding_system);
+	else
+	  val = Qnil;
+      }
+    pset_encode_coding_system (p, val);
+  }
+  /* This may signal an error.  */
+  setup_process_coding_systems (proc);
+
+  specpdl_ptr = specpdl + specpdl_count;
+
+  UNGCPRO;
+  return proc;
+}
+
 \f
 /* Convert an internal struct sockaddr to a lisp object (vector or string).
    The address family of sa is not included in the result.  */
@@ -4884,7 +5124,8 @@ wait_reading_process_output (intmax_t time_limit, int nsecs, int read_kbd,
 		 available now and a closed pipe.
 		 With luck, a closed pipe will be accompanied by
 		 subprocess termination and SIGCHLD.  */
-	      else if (nread == 0 && !NETCONN_P (proc) && !SERIALCONN_P (proc))
+	      else if (nread == 0 && !NETCONN_P (proc) && !SERIALCONN_P (proc)
+		       && !PIPECONN_P (proc))
 		;
 #endif
 #ifdef HAVE_PTYS
@@ -4916,8 +5157,18 @@ wait_reading_process_output (intmax_t time_limit, int nsecs, int read_kbd,
 #endif /* HAVE_PTYS */
 	      /* If we can detect process termination, don't consider the
 		 process gone just because its pipe is closed.  */
-	      else if (nread == 0 && !NETCONN_P (proc) && !SERIALCONN_P (proc))
+	      else if (nread == 0 && !NETCONN_P (proc) && !SERIALCONN_P (proc)
+		       && !PIPECONN_P (proc))
 		;
+	      else if (nread == 0 && PIPECONN_P (proc))
+		{
+		  /* Preserve status of processes already terminated.  */
+		  XPROCESS (proc)->tick = ++process_tick;
+		  deactivate_process (proc);
+		  if (EQ (XPROCESS (proc)->status, Qrun))
+		    pset_status (XPROCESS (proc),
+				 list2 (Qexit, make_number (0)));
+		}
 	      else
 		{
 		  /* Preserve status of processes already terminated.  */
@@ -5954,7 +6205,8 @@ If PROCESS is a network or serial process, inhibit handling of incoming
 traffic.  */)
   (Lisp_Object process, Lisp_Object current_group)
 {
-  if (PROCESSP (process) && (NETCONN_P (process) || SERIALCONN_P (process)))
+  if (PROCESSP (process) && (NETCONN_P (process) || SERIALCONN_P (process)
+			     || PIPECONN_P (process)))
     {
       struct Lisp_Process *p;
 
@@ -5983,7 +6235,8 @@ If PROCESS is a network or serial process, resume handling of incoming
 traffic.  */)
   (Lisp_Object process, Lisp_Object current_group)
 {
-  if (PROCESSP (process) && (NETCONN_P (process) || SERIALCONN_P (process)))
+  if (PROCESSP (process) && (NETCONN_P (process) || SERIALCONN_P (process)
+			     || PIPECONN_P (process)))
     {
       struct Lisp_Process *p;
 
@@ -7030,7 +7283,7 @@ kill_buffer_processes (Lisp_Object buffer)
   FOR_EACH_PROCESS (tail, proc)
     if (NILP (buffer) || EQ (XPROCESS (proc)->buffer, buffer))
       {
-	if (NETCONN_P (proc) || SERIALCONN_P (proc))
+	if (NETCONN_P (proc) || SERIALCONN_P (proc) || PIPECONN_P (proc))
 	  Fdelete_process (proc);
 	else if (XPROCESS (proc)->infd >= 0)
 	  process_send_signal (proc, SIGHUP, Qnil, 1);
@@ -7330,6 +7583,7 @@ syms_of_process (void)
   DEFSYM (Qreal, "real");
   DEFSYM (Qnetwork, "network");
   DEFSYM (Qserial, "serial");
+  DEFSYM (Qpipe, "pipe");
   DEFSYM (QCbuffer, ":buffer");
   DEFSYM (QChost, ":host");
   DEFSYM (QCservice, ":service");
@@ -7346,6 +7600,7 @@ syms_of_process (void)
   DEFSYM (QCplist, ":plist");
   DEFSYM (QCcommand, ":command");
   DEFSYM (QCconnection_type, ":connection-type");
+  DEFSYM (QCstderr, ":stderr");
   DEFSYM (Qpty, "pty");
   DEFSYM (Qpipe, "pipe");
 
@@ -7451,6 +7706,7 @@ The variable takes effect when `start-process' is called.  */);
   defsubr (&Sset_process_plist);
   defsubr (&Sprocess_list);
   defsubr (&Smake_process);
+  defsubr (&Smake_pipe_process);
   defsubr (&Sserial_process_configure);
   defsubr (&Smake_serial_process);
   defsubr (&Sset_network_process_option);
diff --git a/src/process.h b/src/process.h
index 36979dc..43cdc4c 100644
--- a/src/process.h
+++ b/src/process.h
@@ -105,6 +105,9 @@ struct Lisp_Process
     Lisp_Object gnutls_cred_type;
 #endif
 
+    /* Pipe process attached to the standard error of this process.  */
+    Lisp_Object stderrproc;
+
     /* After this point, there are no Lisp_Objects any more.  */
     /* alloc.c assumes that `pid' is the first such non-Lisp slot.  */
 
@@ -215,6 +218,10 @@ extern void record_kill_process (struct Lisp_Process *, Lisp_Object);
 extern Lisp_Object list_system_processes (void);
 extern Lisp_Object system_process_attributes (Lisp_Object);
 
+/* Defined in sysdep.c or w32.c.  */
+
+extern int pipe_open (int[2], int[2]);
+
 /* Defined in process.c.  */
 
 extern void record_deleted_pid (pid_t, Lisp_Object);
diff --git a/src/sysdep.c b/src/sysdep.c
index 0a0b0ac..3281a07 100644
--- a/src/sysdep.c
+++ b/src/sysdep.c
@@ -2272,6 +2272,42 @@ emacs_pipe (int fd[2])
 #endif	/* !MSDOS */
 }
 
+#ifndef WINDOWSNT
+/* For make-pipe-process  */
+int
+pipe_open (int wfd[2], int rfd[2])
+{
+  int result;
+  int inchannel, outchannel;
+#ifdef MSDOS
+  result = pipe (wfd);
+  if (result < 0)
+    return -1;
+  result = pipe (rfd);
+  if (result 0)
+    return -1;
+#else  /* !MSDOS */
+  result = pipe2 (wfd, O_BINARY);
+  if (result < 0)
+    return -1;
+  result = pipe2 (rfd, O_BINARY);
+  if (result < 0)
+    return -1;
+
+  inchannel = rfd[0];
+  outchannel = wfd[1];
+
+  fcntl (inchannel, F_SETFD, FD_CLOEXEC);
+  fcntl (outchannel, F_SETFD, FD_CLOEXEC);
+
+  fcntl (inchannel, F_SETFL, O_NONBLOCK);
+  fcntl (outchannel, F_SETFL, O_NONBLOCK);
+#endif	/* !MSDOS */
+
+  return 0;
+}
+#endif	/* !WINDOWSNT */
+
 /* Approximate posix_close and POSIX_CLOSE_RESTART well enough for Emacs.
    For the background behind this mess, please see Austin Group defect 529
    <http://austingroupbugs.net/view.php?id=529>.  */
diff --git a/src/w32.c b/src/w32.c
index 6f16704..95842f3 100644
--- a/src/w32.c
+++ b/src/w32.c
@@ -255,6 +255,7 @@ extern void *e_malloc (size_t);
 extern int sys_select (int, SELECT_TYPE *, SELECT_TYPE *, SELECT_TYPE *,
 		       struct timespec *, void *);
 extern int sys_dup (int);
+static int sys_pipe2 (int *, int);
 
 
 
@@ -7938,14 +7939,12 @@ sys_dup2 (int src, int dst)
   return rc;
 }
 
-int
-pipe2 (int * phandles, int pipe2_flags)
+static int
+sys_pipe2 (int * phandles, int pipe2_flags)
 {
   int rc;
   unsigned flags;
 
-  eassert (pipe2_flags == (O_BINARY | O_CLOEXEC));
-
   /* make pipe handles non-inheritable; when we spawn a child, we
      replace the relevant handle with an inheritable one.  Also put
      pipes into binary mode; we will do text mode translation ourselves
@@ -7976,6 +7975,14 @@ pipe2 (int * phandles, int pipe2_flags)
   return rc;
 }
 
+int
+pipe2 (int * phandles, int pipe2_flags)
+{
+  eassert (pipe2_flags == (O_BINARY | O_CLOEXEC));
+
+  return sys_pipe2 (phandles, pipe2_flags);
+}
+
 /* Function to do blocking read of one byte, needed to implement
    select.  It is only allowed on communication ports, sockets, or
    pipes. */
@@ -9473,6 +9480,59 @@ serial_configure (struct Lisp_Process *p, Lisp_Object contact)
   pset_childp (p, childp2);
 }
 
+int
+pipe_open (int wfd[2], int rfd[2])
+{
+  int result;
+  int inchannel, outchannel;
+  child_process *cp;
+
+  result = sys_pipe2 (wfd, O_BINARY);
+  if (result < 0)
+    return -1;
+  result = sys_pipe2 (rfd, O_BINARY);
+  if (result < 0)
+    return -1;
+
+  inchannel = rfd[0];
+  outchannel = wfd[1];
+
+  fcntl (inchannel, F_SETFD, FD_CLOEXEC);
+  fcntl (outchannel, F_SETFD, FD_CLOEXEC);
+
+  fcntl (inchannel, F_SETFL, O_NONBLOCK);
+  fcntl (outchannel, F_SETFL, O_NONBLOCK);
+
+  cp = new_child ();
+  if (!cp)
+    error ("Could not create child process");
+  cp->fd = inchannel;
+  cp->status = STATUS_READ_ACKNOWLEDGED;
+
+  if (fd_info[ inchannel ].cp != NULL)
+    {
+      error ("fd_info[fd = %d] is already in use", inchannel);
+    }
+  fd_info[ inchannel ].cp = cp;
+  fd_info[ inchannel ].hnd = (HANDLE) _get_osfhandle (inchannel);
+
+  if (fd_info[ outchannel ].cp != NULL)
+    {
+      error ("fd_info[fd = %d] is already in use", outchannel);
+    }
+  fd_info[ outchannel ].cp = cp;
+  fd_info[ outchannel ].hnd = (HANDLE) _get_osfhandle (outchannel);
+
+  cp->ovl_read.hEvent = CreateEvent (NULL, TRUE, FALSE, NULL);
+  if (cp->ovl_read.hEvent == NULL)
+      error ("Could not create read event");
+  cp->ovl_write.hEvent = CreateEvent (NULL, TRUE, FALSE, NULL);
+  if (cp->ovl_write.hEvent == NULL)
+      error ("Could not create write event");
+
+  return 0;
+}
+
 #ifdef HAVE_GNUTLS
 
 ssize_t
diff --git a/test/automated/process-tests.el b/test/automated/process-tests.el
index dabfbc5..3267999 100644
--- a/test/automated/process-tests.el
+++ b/test/automated/process-tests.el
@@ -72,4 +72,71 @@
               (should (string= (buffer-string) "arg1 = \"x &y\", arg2 = \n"))))
         (when batfile (delete-file batfile))))))
 
+(ert-deftest process-test-stderr-buffer ()
+  (skip-unless (executable-find "bash"))
+  (let* ((stdout-buffer (generate-new-buffer "*stdout*"))
+	 (stderr-buffer (generate-new-buffer "*stderr*"))
+	 (proc (make-process :name "test" :buffer stdout-buffer
+			     :command (list "bash" "-c"
+					    (concat "echo hello stdout!; "
+						    "echo hello stderr! >&2; "
+						    "exit 20"))
+			     :stderr stderr-buffer))
+	 (sentinel-called nil)
+	 (start-time (float-time)))
+    (while (not (or sentinel-called
+		    (> (- (float-time) start-time)
+		       process-test-sentinel-wait-timeout)))
+      (accept-process-output))
+    (cl-assert (eq (process-status proc) 'exit))
+    (cl-assert (= (process-exit-status proc) 20))
+    (should (with-current-buffer stdout-buffer
+	      (goto-char (point-min))
+	      (looking-at "hello stdout!")))
+    (should (with-current-buffer stderr-buffer
+	      (goto-char (point-min))
+	      (looking-at "hello stderr!")))))
+
+(ert-deftest process-test-stderr-filter ()
+  (skip-unless (executable-find "bash"))
+  (let* ((sentinel-called nil)
+	 (stderr-sentinel-called nil)
+	 (stdout-output nil)
+	 (stderr-output nil)
+	 (stdout-buffer (generate-new-buffer "*stdout*"))
+	 (stderr-buffer (generate-new-buffer "*stderr*"))
+	 (stderr-proc (make-pipe-process :name "stderr"
+					 :buffer stderr-buffer))
+	 (proc (make-process :name "test" :buffer stdout-buffer
+			     :command (list "bash" "-c"
+					    (concat "echo hello stdout!; "
+						    "echo hello stderr! >&2; "
+						    "exit 20"))
+			     :stderr stderr-proc))
+	 (start-time (float-time)))
+    (set-process-filter proc (lambda (proc input)
+			       (push input stdout-output)))
+    (set-process-sentinel proc (lambda (proc msg)
+				 (setq sentinel-called t)))
+    (set-process-filter stderr-proc (lambda (proc input)
+				      (push input stderr-output)))
+    (set-process-sentinel stderr-proc (lambda (proc input)
+					(setq stderr-sentinel-called t)))
+    (while (not (or sentinel-called
+		    (> (- (float-time) start-time)
+		       process-test-sentinel-wait-timeout)))
+      (accept-process-output))
+    (cl-assert (eq (process-status proc) 'exit))
+    (cl-assert (= (process-exit-status proc) 20))
+    (should sentinel-called)
+    (should (equal 1 (with-current-buffer stdout-buffer
+		       (point-max))))
+    (should (equal "hello stdout!\n"
+		   (mapconcat #'identity (nreverse stdout-output) "")))
+    (should stderr-sentinel-called)
+    (should (equal 1 (with-current-buffer stderr-buffer
+		       (point-max))))
+    (should (equal "hello stderr!\n"
+		   (mapconcat #'identity (nreverse stderr-output) "")))))
+
 (provide 'process-tests)
-- 
2.1.3

Re: [PATCH] Add facility to collect stderr of async subprocess

[Paul Eggert]

I took a quick look, and in the ordinary POSIXish code there's a race 
condition between the calls to pipe2 and the calls to fcntl with 
FD_CLOEXEC.  If some other thread forks and execs a process after pipe2 
but before fcntl, that process will have access to the pipe ends and 
this could cause problems.  Instead, please call pipe2 (.., O_BINARY | 
O_CLOEXEC) so that the other process won't see the pipe end.  Any child 
process that needs to have the pipe end survive an exec should then call 
fcntl (pipe_end, F_SETFD, 0) before execing.  This is how the existing 
Emacs code works.  So, there should be no need to call fcntl (inchannel, 
F_SETFD, FD_CLOEXEC) and likewise for outchannel.

Re: [PATCH] Add facility to collect stderr of async subprocess

[Daiki Ueno]

[-- Attachment #1: Type: text/plain, Size: 839 bytes --]

Paul Eggert <eggert@cs.ucla.edu> writes:

> I took a quick look, and in the ordinary POSIXish code there's a race
> condition between the calls to pipe2 and the calls to fcntl with
> FD_CLOEXEC.  If some other thread forks and execs a process after
> pipe2 but before fcntl, that process will have access to the pipe ends
> and this could cause problems.  Instead, please call pipe2 (..,
> O_BINARY | O_CLOEXEC) so that the other process won't see the pipe
> end.  Any child process that needs to have the pipe end survive an
> exec should then call fcntl (pipe_end, F_SETFD, 0) before execing.

Thanks.  Actually, the special case of pipe2 flags was not necessary; it
was a wreck when I tried to support FD above 2 (for stderr, the flag is
cleared in child_setup).

I updated the patch and it's now much simpler.

Regards,
-- 
Daiki Ueno

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: v3-0001-Add-facility-to-collect-stderr-of-async-subproces.patch --]
[-- Type: text/x-patch, Size: 27234 bytes --]

From 5924db1deeb49869d82906891e003c80be4bb486 Mon Sep 17 00:00:00 2001
From: Daiki Ueno <ueno@gnu.org>
Date: Tue, 7 Apr 2015 17:42:09 +0900
Subject: [PATCH v3] Add facility to collect stderr of async subprocess

* src/w32.h (register_aux_fd): New function declaration.
* src/w32.c (register_aux_fd): New function.
* src/process.h (struct Lisp_Process): New member stderrproc.
* src/process.c (PIPECONN_P): New macro.
(PIPECONN1_P): New macro.
(Fdelete_process, Fprocess_status, Fset_process_buffer)
(Fset_process_filter, Fset_process_sentinel, Fstop_process)
(Fcontinue_process): Handle pipe process specially.
(create_process): Respect p->stderrproc.
(Fmake_pipe_process): New function.
(Fmake_process): Add new keyword argument :stderr.
(wait_reading_process_output): Specially handle a pipe process when
it gets an EOF.
(syms_of_process): Register Qpipe and Smake_pipe_process.

* doc/lispref/processes.texi (Asynchronous Processes): Mention
`make-pipe-process' and `:stderr' keyword of `make-process'.

* test/automated/process-tests.el (process-test-stderr-buffer)
(process-test-stderr-filter): New tests.

* etc/NEWS: Mention pipe process.
---
 doc/lispref/processes.texi      |  52 +++++++
 etc/NEWS                        |   4 +
 src/process.c                   | 302 +++++++++++++++++++++++++++++++++++++---
 src/process.h                   |   3 +
 src/w32.c                       |  32 +++++
 src/w32.h                       |   1 +
 test/automated/process-tests.el |  67 +++++++++
 7 files changed, 440 insertions(+), 21 deletions(-)

diff --git a/doc/lispref/processes.texi b/doc/lispref/processes.texi
index 3e9cc50..59bc846 100644
--- a/doc/lispref/processes.texi
+++ b/doc/lispref/processes.texi
@@ -741,6 +741,58 @@ Initialize the process filter to @var{filter}.
 
 @item :sentinel @var{sentinel}
 Initialize the process sentinel to @var{sentinel}.
+
+@item :stderr @var{stderr}
+Associate @var{stderr} with the standard error of the process.
+@var{stderr} is either a buffer or a pipe process created with
+@code{make-pipe-process}.
+@end table
+
+The original argument list, modified with the actual connection
+information, is available via the @code{process-contact} function.
+@end defun
+
+@defun make-pipe-process &rest args
+This function creates a bidirectional pipe which can be attached to a
+child process (typically through the @code{:stderr} keyword of
+@code{make-process}).
+
+The arguments @var{args} are a list of keyword/argument pairs.
+Omitting a keyword is always equivalent to specifying it with value
+@code{nil}, except for @code{:coding}.
+Here are the meaningful keywords:
+
+@table @asis
+@item :name @var{name}
+Use the string @var{name} as the process name.  It is modified if
+necessary to make it unique.
+
+@item :buffer @var{buffer}
+Use @var{buffer} as the process buffer.
+
+@item :coding @var{coding}
+If @var{coding} is a symbol, it specifies the coding system to be
+used for both reading and writing of data from and to the
+connection.  If @var{coding} is a cons cell
+@w{@code{(@var{decoding} . @var{encoding})}}, then @var{decoding}
+will be used for reading and @var{encoding} for writing.
+
+If @var{coding} is @code{nil}, the default rules for finding the
+coding system will apply.  @xref{Default Coding Systems}.
+
+@item :noquery @var{query-flag}
+Initialize the process query flag to @var{query-flag}.
+@xref{Query Before Exit}.
+
+@item :stop @var{stopped}
+If @var{stopped} is non-@code{nil}, start the process in the
+``stopped'' state.
+
+@item :filter @var{filter}
+Initialize the process filter to @var{filter}.
+
+@item :sentinel @var{sentinel}
+Initialize the process sentinel to @var{sentinel}.
 @end table
 
 The original argument list, modified with the actual connection
diff --git a/etc/NEWS b/etc/NEWS
index 0332fc5..cf26bbf 100644
--- a/etc/NEWS
+++ b/etc/NEWS
@@ -671,6 +671,10 @@ word syntax, use `\sw' instead.
 \f
 * Lisp Changes in Emacs 25.1
 
+** New process type `pipe', which can be used in combination with the
+   `:stderr' keyword of make-process to collect standard error output
+   of subprocess.
+
 ** New function `make-process' provides an alternative interface to
 `start-process'.  It allows programs to set process parameters such as
 process filter, sentinel, etc., through keyword arguments (similar to
diff --git a/src/process.c b/src/process.c
index 2800fa5..e9bf059 100644
--- a/src/process.c
+++ b/src/process.c
@@ -189,6 +189,8 @@ process_socket (int domain, int type, int protocol)
 #define NETCONN1_P(p) (EQ (p->type, Qnetwork))
 #define SERIALCONN_P(p) (EQ (XPROCESS (p)->type, Qserial))
 #define SERIALCONN1_P(p) (EQ (p->type, Qserial))
+#define PIPECONN_P(p) (EQ (XPROCESS (p)->type, Qpipe))
+#define PIPECONN1_P(p) (EQ (p->type, Qpipe))
 
 /* Number of events of change of status of a process.  */
 static EMACS_INT process_tick;
@@ -411,6 +413,11 @@ pset_write_queue (struct Lisp_Process *p, Lisp_Object val)
 {
   p->write_queue = val;
 }
+static void
+pset_stderrproc (struct Lisp_Process *p, Lisp_Object val)
+{
+  p->stderrproc = val;
+}
 
 \f
 static Lisp_Object
@@ -837,7 +844,7 @@ nil, indicating the current buffer's process.  */)
   p = XPROCESS (process);
 
   p->raw_status_new = 0;
-  if (NETCONN1_P (p) || SERIALCONN1_P (p))
+  if (NETCONN1_P (p) || SERIALCONN1_P (p) || PIPECONN1_P (p))
     {
       pset_status (p, list2 (Qexit, make_number (0)));
       p->tick = ++process_tick;
@@ -903,7 +910,7 @@ nil, indicating the current buffer's process.  */)
   status = p->status;
   if (CONSP (status))
     status = XCAR (status);
-  if (NETCONN1_P (p) || SERIALCONN1_P (p))
+  if (NETCONN1_P (p) || SERIALCONN1_P (p) || PIPECONN1_P (p))
     {
       if (EQ (status, Qexit))
 	status = Qclosed;
@@ -987,7 +994,7 @@ Return BUFFER.  */)
     CHECK_BUFFER (buffer);
   p = XPROCESS (process);
   pset_buffer (p, buffer);
-  if (NETCONN1_P (p) || SERIALCONN1_P (p))
+  if (NETCONN1_P (p) || SERIALCONN1_P (p) || PIPECONN1_P (p))
     pset_childp (p, Fplist_put (p->childp, QCbuffer, buffer));
   setup_process_coding_systems (process);
   return buffer;
@@ -1063,7 +1070,7 @@ The string argument is normally a multibyte string, except:
     }
 
   pset_filter (p, filter);
-  if (NETCONN1_P (p) || SERIALCONN1_P (p))
+  if (NETCONN1_P (p) || SERIALCONN1_P (p) || PIPECONN1_P (p))
     pset_childp (p, Fplist_put (p->childp, QCfilter, filter));
   setup_process_coding_systems (process);
   return filter;
@@ -1095,7 +1102,7 @@ It gets two arguments: the process, and a string describing the change.  */)
     sentinel = Qinternal_default_process_sentinel;
 
   pset_sentinel (p, sentinel);
-  if (NETCONN1_P (p) || SERIALCONN1_P (p))
+  if (NETCONN1_P (p) || SERIALCONN1_P (p) || PIPECONN1_P (p))
     pset_childp (p, Fplist_put (p->childp, QCsentinel, sentinel));
   return sentinel;
 }
@@ -1183,13 +1190,13 @@ DEFUN ("process-query-on-exit-flag",
 DEFUN ("process-contact", Fprocess_contact, Sprocess_contact,
        1, 2, 0,
        doc: /* Return the contact info of PROCESS; t for a real child.
-For a network or serial connection, the value depends on the optional
-KEY arg.  If KEY is nil, value is a cons cell of the form (HOST
-SERVICE) for a network connection or (PORT SPEED) for a serial
-connection.  If KEY is t, the complete contact information for the
-connection is returned, else the specific value for the keyword KEY is
-returned.  See `make-network-process' or `make-serial-process' for a
-list of keywords.  */)
+For a network or serial connection, or a pipe, the value depends on
+the optional KEY arg.  If KEY is nil, value is a cons cell of the form
+\(HOST SERVICE) for a network connection or (PORT SPEED) for a serial
+connection, or (INPUT OUTPUT) for a pipe.  If KEY is t, the complete
+contact information for the connection is returned, else the specific
+value for the keyword KEY is returned.  See `make-network-process' or
+`make-serial-process' for a list of keywords.  */)
   (register Lisp_Object process, Lisp_Object key)
 {
   Lisp_Object contact;
@@ -1386,10 +1393,15 @@ to use a pty, or nil to use the default specified through
 
 :sentinel SENTINEL -- Install SENTINEL as the process sentinel.
 
+:stderr STDERR -- STDERR is either a buffer or a pipe process attached
+to the standard error of subprocess.  Specifying this implies
+`:connection-type' is set to `pipe'.
+
 usage: (make-process &rest ARGS)  */)
   (ptrdiff_t nargs, Lisp_Object *args)
 {
   Lisp_Object buffer, name, command, program, proc, contact, current_dir, tem;
+  Lisp_Object xstderr, stderrproc;
   ptrdiff_t count = SPECPDL_INDEX ();
   struct gcpro gcpro1;
   USE_SAFE_ALLOCA;
@@ -1433,6 +1445,27 @@ usage: (make-process &rest ARGS)  */)
   if (!NILP (program))
     CHECK_STRING (program);
 
+  stderrproc = Qnil;
+  xstderr = Fplist_get (contact, QCstderr);
+  if (PROCESSP (xstderr))
+    {
+      if (!PIPECONN_P (xstderr))
+	error ("Process is not a pipe process");
+      stderrproc = xstderr;
+    }
+  else if (!NILP (xstderr))
+    {
+      struct gcpro gcpro1, gcpro2;
+      CHECK_STRING (program);
+      GCPRO2 (buffer, current_dir);
+      stderrproc = CALLN (Fmake_pipe_process,
+			  QCname,
+			  concat2 (name, build_string (" stderr")),
+			  QCbuffer,
+			  Fget_buffer_create (xstderr));
+      UNGCPRO;
+    }
+
   proc = make_process (name);
   /* If an error occurs and we can't start the process, we want to
      remove it from the process list.  This means that each error
@@ -1463,6 +1496,13 @@ usage: (make-process &rest ARGS)  */)
   else
     report_file_error ("Unknown connection type", tem);
 
+  if (!NILP (stderrproc))
+    {
+      pset_stderrproc (XPROCESS (proc), stderrproc);
+
+      XPROCESS (proc)->pty_flag = false;
+    }
+
 #ifdef HAVE_GNUTLS
   /* AKA GNUTLS_INITSTAGE(proc).  */
   XPROCESS (proc)->gnutls_initstage = GNUTLS_STAGE_EMPTY;
@@ -1705,10 +1745,10 @@ static void
 create_process (Lisp_Object process, char **new_argv, Lisp_Object current_dir)
 {
   struct Lisp_Process *p = XPROCESS (process);
-  int inchannel, outchannel;
+  int inchannel, outchannel, errchannel = -1;
   pid_t pid;
   int vfork_errno;
-  int forkin, forkout;
+  int forkin, forkout, forkerr = -1;
   bool pty_flag = 0;
   char pty_name[PTY_NAME_SIZE];
   Lisp_Object lisp_pty_name = Qnil;
@@ -1746,6 +1786,18 @@ create_process (Lisp_Object process, char **new_argv, Lisp_Object current_dir)
       outchannel = p->open_fd[WRITE_TO_SUBPROCESS];
       inchannel = p->open_fd[READ_FROM_SUBPROCESS];
       forkout = p->open_fd[SUBPROCESS_STDOUT];
+
+      if (!NILP (p->stderrproc))
+	{
+	  struct Lisp_Process *pp = XPROCESS (p->stderrproc);
+
+	  forkerr = pp->open_fd[SUBPROCESS_STDOUT];
+	  errchannel = pp->open_fd[READ_FROM_SUBPROCESS];
+
+	  /* Close unnecessary file descriptors.  */
+	  close_process_fd (&pp->open_fd[WRITE_TO_SUBPROCESS]);
+	  close_process_fd (&pp->open_fd[SUBPROCESS_STDIN]);
+	}
     }
 
 #ifndef WINDOWSNT
@@ -1792,6 +1844,7 @@ create_process (Lisp_Object process, char **new_argv, Lisp_Object current_dir)
     char **volatile new_argv_volatile = new_argv;
     int volatile forkin_volatile = forkin;
     int volatile forkout_volatile = forkout;
+    int volatile forkerr_volatile = forkerr;
     struct Lisp_Process *p_volatile = p;
 
     pid = vfork ();
@@ -1801,6 +1854,7 @@ create_process (Lisp_Object process, char **new_argv, Lisp_Object current_dir)
     new_argv = new_argv_volatile;
     forkin = forkin_volatile;
     forkout = forkout_volatile;
+    forkerr = forkerr_volatile;
     p = p_volatile;
 
     pty_flag = p->pty_flag;
@@ -1811,6 +1865,7 @@ create_process (Lisp_Object process, char **new_argv, Lisp_Object current_dir)
     {
       int xforkin = forkin;
       int xforkout = forkout;
+      int xforkerr = forkerr;
 
       /* Make the pty be the controlling terminal of the process.  */
 #ifdef HAVE_PTYS
@@ -1910,10 +1965,13 @@ create_process (Lisp_Object process, char **new_argv, Lisp_Object current_dir)
 
       if (pty_flag)
 	child_setup_tty (xforkout);
+
+      if (xforkerr < 0)
+	xforkerr = xforkout;
 #ifdef WINDOWSNT
-      pid = child_setup (xforkin, xforkout, xforkout, new_argv, 1, current_dir);
+      pid = child_setup (xforkin, xforkout, xforkerr, new_argv, 1, current_dir);
 #else  /* not WINDOWSNT */
-      child_setup (xforkin, xforkout, xforkout, new_argv, 1, current_dir);
+      child_setup (xforkin, xforkout, xforkerr, new_argv, 1, current_dir);
 #endif /* not WINDOWSNT */
     }
 
@@ -1958,6 +2016,11 @@ create_process (Lisp_Object process, char **new_argv, Lisp_Object current_dir)
 	close_process_fd (&p->open_fd[READ_FROM_EXEC_MONITOR]);
       }
 #endif
+      if (!NILP (p->stderrproc))
+	{
+	  struct Lisp_Process *pp = XPROCESS (p->stderrproc);
+	  close_process_fd (&pp->open_fd[SUBPROCESS_STDOUT]);
+	}
     }
 }
 
@@ -2016,6 +2079,187 @@ create_pty (Lisp_Object process)
   p->pid = -2;
 }
 
+DEFUN ("make-pipe-process", Fmake_pipe_process, Smake_pipe_process,
+       0, MANY, 0,
+       doc: /* Create and return a bidirectional pipe process.
+
+In Emacs, pipes are represented by process objects, so input and
+output work as for subprocesses, and `delete-process' closes a pipe.
+However, a pipe process has no process id, it cannot be signaled,
+and the status codes are different from normal processes.
+
+Arguments are specified as keyword/argument pairs.  The following
+arguments are defined:
+
+:name NAME -- NAME is the name of the process.  It is modified if necessary to make it unique.
+
+:buffer BUFFER -- BUFFER is the buffer (or buffer-name) to associate
+with the process.  Process output goes at the end of that buffer,
+unless you specify an output stream or filter function to handle the
+output.  If BUFFER is not given, the value of NAME is used.
+
+:coding CODING -- If CODING is a symbol, it specifies the coding
+system used for both reading and writing for this process.  If CODING
+is a cons (DECODING . ENCODING), DECODING is used for reading, and
+ENCODING is used for writing.
+
+:noquery BOOL -- When exiting Emacs, query the user if BOOL is nil and
+the process is running.  If BOOL is not given, query before exiting.
+
+:stop BOOL -- Start process in the `stopped' state if BOOL non-nil.
+In the stopped state, a pipe process does not accept incoming data,
+but you can send outgoing data.  The stopped state is cleared by
+`continue-process' and set by `stop-process'.
+
+:filter FILTER -- Install FILTER as the process filter.
+
+:sentinel SENTINEL -- Install SENTINEL as the process sentinel.
+
+usage:  (make-pipe-process &rest ARGS)  */)
+  (ptrdiff_t nargs, Lisp_Object *args)
+{
+  Lisp_Object proc, contact;
+  struct Lisp_Process *p;
+  struct gcpro gcpro1;
+  Lisp_Object name, buffer;
+  Lisp_Object tem, val;
+  ptrdiff_t specpdl_count;
+  int inchannel, outchannel;
+
+  if (nargs == 0)
+    return Qnil;
+
+  contact = Flist (nargs, args);
+  GCPRO1 (contact);
+
+  name = Fplist_get (contact, QCname);
+  CHECK_STRING (name);
+  proc = make_process (name);
+  specpdl_count = SPECPDL_INDEX ();
+  record_unwind_protect (remove_process, proc);
+  p = XPROCESS (proc);
+
+  if (emacs_pipe (p->open_fd + SUBPROCESS_STDIN) != 0
+      || emacs_pipe (p->open_fd + READ_FROM_SUBPROCESS) != 0)
+    report_file_error ("Creating pipe", Qnil);
+  outchannel = p->open_fd[WRITE_TO_SUBPROCESS];
+  inchannel = p->open_fd[READ_FROM_SUBPROCESS];
+
+  fcntl (inchannel, F_SETFL, O_NONBLOCK);
+  fcntl (outchannel, F_SETFL, O_NONBLOCK);
+
+#ifdef WINDOWSNT
+  register_aux_fd (inchannel, outchannel);
+#endif
+
+  /* Record this as an active process, with its channels.  */
+  chan_process[inchannel] = proc;
+  p->infd = inchannel;
+  p->outfd = outchannel;
+
+  if (inchannel > max_process_desc)
+    max_process_desc = inchannel;
+
+  buffer = Fplist_get (contact, QCbuffer);
+  if (NILP (buffer))
+    buffer = name;
+  buffer = Fget_buffer_create (buffer);
+  pset_buffer (p, buffer);
+
+  pset_childp (p, contact);
+  pset_plist (p, Fcopy_sequence (Fplist_get (contact, QCplist)));
+  pset_type (p, Qpipe);
+  pset_sentinel (p, Fplist_get (contact, QCsentinel));
+  pset_filter (p, Fplist_get (contact, QCfilter));
+  pset_log (p, Qnil);
+  if (tem = Fplist_get (contact, QCnoquery), !NILP (tem))
+    p->kill_without_query = 1;
+  if (tem = Fplist_get (contact, QCstop), !NILP (tem))
+    pset_command (p, Qt);
+  eassert (! p->pty_flag);
+
+  if (!EQ (p->command, Qt))
+    {
+      FD_SET (inchannel, &input_wait_mask);
+      FD_SET (inchannel, &non_keyboard_wait_mask);
+    }
+#ifdef ADAPTIVE_READ_BUFFERING
+  p->adaptive_read_buffering
+    = (NILP (Vprocess_adaptive_read_buffering) ? 0
+       : EQ (Vprocess_adaptive_read_buffering, Qt) ? 1 : 2);
+#endif
+
+  /* Make the process marker point into the process buffer (if any).  */
+  if (BUFFERP (buffer))
+    set_marker_both (p->mark, buffer,
+		     BUF_ZV (XBUFFER (buffer)),
+		     BUF_ZV_BYTE (XBUFFER (buffer)));
+
+  {
+    /* Setup coding systems for communicating with the network stream.  */
+    struct gcpro gcpro1;
+    /* Qt denotes we have not yet called Ffind_operation_coding_system.  */
+    Lisp_Object coding_systems = Qt;
+    Lisp_Object val;
+
+    tem = Fplist_get (contact, QCcoding);
+    val = Qnil;
+    if (!NILP (tem))
+      {
+	val = tem;
+	if (CONSP (val))
+	  val = XCAR (val);
+      }
+    else if (!NILP (Vcoding_system_for_read))
+      val = Vcoding_system_for_read;
+    else if ((!NILP (buffer) && NILP (BVAR (XBUFFER (buffer), enable_multibyte_characters)))
+	     || (NILP (buffer) && NILP (BVAR (&buffer_defaults, enable_multibyte_characters))))
+      /* We dare not decode end-of-line format by setting VAL to
+	 Qraw_text, because the existing Emacs Lisp libraries
+	 assume that they receive bare code including a sequence of
+	 CR LF.  */
+      val = Qnil;
+    else
+      {
+	if (CONSP (coding_systems))
+	  val = XCAR (coding_systems);
+	else if (CONSP (Vdefault_process_coding_system))
+	  val = XCAR (Vdefault_process_coding_system);
+	else
+	  val = Qnil;
+      }
+    pset_decode_coding_system (p, val);
+
+    if (!NILP (tem))
+      {
+	val = tem;
+	if (CONSP (val))
+	  val = XCDR (val);
+      }
+    else if (!NILP (Vcoding_system_for_write))
+      val = Vcoding_system_for_write;
+    else if (NILP (BVAR (current_buffer, enable_multibyte_characters)))
+      val = Qnil;
+    else
+      {
+	if (CONSP (coding_systems))
+	  val = XCDR (coding_systems);
+	else if (CONSP (Vdefault_process_coding_system))
+	  val = XCDR (Vdefault_process_coding_system);
+	else
+	  val = Qnil;
+      }
+    pset_encode_coding_system (p, val);
+  }
+  /* This may signal an error.  */
+  setup_process_coding_systems (proc);
+
+  specpdl_ptr = specpdl + specpdl_count;
+
+  UNGCPRO;
+  return proc;
+}
+
 \f
 /* Convert an internal struct sockaddr to a lisp object (vector or string).
    The address family of sa is not included in the result.  */
@@ -4884,7 +5128,8 @@ wait_reading_process_output (intmax_t time_limit, int nsecs, int read_kbd,
 		 available now and a closed pipe.
 		 With luck, a closed pipe will be accompanied by
 		 subprocess termination and SIGCHLD.  */
-	      else if (nread == 0 && !NETCONN_P (proc) && !SERIALCONN_P (proc))
+	      else if (nread == 0 && !NETCONN_P (proc) && !SERIALCONN_P (proc)
+		       && !PIPECONN_P (proc))
 		;
 #endif
 #ifdef HAVE_PTYS
@@ -4916,8 +5161,18 @@ wait_reading_process_output (intmax_t time_limit, int nsecs, int read_kbd,
 #endif /* HAVE_PTYS */
 	      /* If we can detect process termination, don't consider the
 		 process gone just because its pipe is closed.  */
-	      else if (nread == 0 && !NETCONN_P (proc) && !SERIALCONN_P (proc))
+	      else if (nread == 0 && !NETCONN_P (proc) && !SERIALCONN_P (proc)
+		       && !PIPECONN_P (proc))
 		;
+	      else if (nread == 0 && PIPECONN_P (proc))
+		{
+		  /* Preserve status of processes already terminated.  */
+		  XPROCESS (proc)->tick = ++process_tick;
+		  deactivate_process (proc);
+		  if (EQ (XPROCESS (proc)->status, Qrun))
+		    pset_status (XPROCESS (proc),
+				 list2 (Qexit, make_number (0)));
+		}
 	      else
 		{
 		  /* Preserve status of processes already terminated.  */
@@ -5954,7 +6209,8 @@ If PROCESS is a network or serial process, inhibit handling of incoming
 traffic.  */)
   (Lisp_Object process, Lisp_Object current_group)
 {
-  if (PROCESSP (process) && (NETCONN_P (process) || SERIALCONN_P (process)))
+  if (PROCESSP (process) && (NETCONN_P (process) || SERIALCONN_P (process)
+			     || PIPECONN_P (process)))
     {
       struct Lisp_Process *p;
 
@@ -5983,7 +6239,8 @@ If PROCESS is a network or serial process, resume handling of incoming
 traffic.  */)
   (Lisp_Object process, Lisp_Object current_group)
 {
-  if (PROCESSP (process) && (NETCONN_P (process) || SERIALCONN_P (process)))
+  if (PROCESSP (process) && (NETCONN_P (process) || SERIALCONN_P (process)
+			     || PIPECONN_P (process)))
     {
       struct Lisp_Process *p;
 
@@ -7030,7 +7287,7 @@ kill_buffer_processes (Lisp_Object buffer)
   FOR_EACH_PROCESS (tail, proc)
     if (NILP (buffer) || EQ (XPROCESS (proc)->buffer, buffer))
       {
-	if (NETCONN_P (proc) || SERIALCONN_P (proc))
+	if (NETCONN_P (proc) || SERIALCONN_P (proc) || PIPECONN_P (proc))
 	  Fdelete_process (proc);
 	else if (XPROCESS (proc)->infd >= 0)
 	  process_send_signal (proc, SIGHUP, Qnil, 1);
@@ -7330,6 +7587,7 @@ syms_of_process (void)
   DEFSYM (Qreal, "real");
   DEFSYM (Qnetwork, "network");
   DEFSYM (Qserial, "serial");
+  DEFSYM (Qpipe, "pipe");
   DEFSYM (QCbuffer, ":buffer");
   DEFSYM (QChost, ":host");
   DEFSYM (QCservice, ":service");
@@ -7346,6 +7604,7 @@ syms_of_process (void)
   DEFSYM (QCplist, ":plist");
   DEFSYM (QCcommand, ":command");
   DEFSYM (QCconnection_type, ":connection-type");
+  DEFSYM (QCstderr, ":stderr");
   DEFSYM (Qpty, "pty");
   DEFSYM (Qpipe, "pipe");
 
@@ -7451,6 +7710,7 @@ The variable takes effect when `start-process' is called.  */);
   defsubr (&Sset_process_plist);
   defsubr (&Sprocess_list);
   defsubr (&Smake_process);
+  defsubr (&Smake_pipe_process);
   defsubr (&Sserial_process_configure);
   defsubr (&Smake_serial_process);
   defsubr (&Sset_network_process_option);
diff --git a/src/process.h b/src/process.h
index 36979dc..e889055 100644
--- a/src/process.h
+++ b/src/process.h
@@ -105,6 +105,9 @@ struct Lisp_Process
     Lisp_Object gnutls_cred_type;
 #endif
 
+    /* Pipe process attached to the standard error of this process.  */
+    Lisp_Object stderrproc;
+
     /* After this point, there are no Lisp_Objects any more.  */
     /* alloc.c assumes that `pid' is the first such non-Lisp slot.  */
 
diff --git a/src/w32.c b/src/w32.c
index 6f16704..77bf307 100644
--- a/src/w32.c
+++ b/src/w32.c
@@ -9473,6 +9473,38 @@ serial_configure (struct Lisp_Process *p, Lisp_Object contact)
   pset_childp (p, childp2);
 }
 
+/* For make-pipe-process */
+void
+register_aux_fd (int infd, int outfd)
+{
+  cp = new_child ();
+  if (!cp)
+    error ("Could not create child process");
+  cp->fd = infd;
+  cp->status = STATUS_READ_ACKNOWLEDGED;
+
+  if (fd_info[ infd ].cp != NULL)
+    {
+      error ("fd_info[fd = %d] is already in use", infd);
+    }
+  fd_info[ infd ].cp = cp;
+  fd_info[ infd ].hnd = (HANDLE) _get_osfhandle (infd);
+
+  if (fd_info[ outfd ].cp != NULL)
+    {
+      error ("fd_info[fd = %d] is already in use", outfd);
+    }
+  fd_info[ outfd ].cp = cp;
+  fd_info[ outfd ].hnd = (HANDLE) _get_osfhandle (outfd);
+
+  cp->ovl_read.hEvent = CreateEvent (NULL, TRUE, FALSE, NULL);
+  if (cp->ovl_read.hEvent == NULL)
+      error ("Could not create read event");
+  cp->ovl_write.hEvent = CreateEvent (NULL, TRUE, FALSE, NULL);
+  if (cp->ovl_write.hEvent == NULL)
+      error ("Could not create write event");
+}
+
 #ifdef HAVE_GNUTLS
 
 ssize_t
diff --git a/src/w32.h b/src/w32.h
index 9b3521d..32d232e 100644
--- a/src/w32.h
+++ b/src/w32.h
@@ -202,6 +202,7 @@ extern int random (void);
 extern int fchmod (int, mode_t);
 extern int sys_rename_replace (char const *, char const *, BOOL);
 extern int pipe2 (int *, int);
+extern void register_aux_fd (int, int);
 
 extern void set_process_dir (char *);
 extern int sys_spawnve (int, char *, char **, char **);
diff --git a/test/automated/process-tests.el b/test/automated/process-tests.el
index dabfbc5..3267999 100644
--- a/test/automated/process-tests.el
+++ b/test/automated/process-tests.el
@@ -72,4 +72,71 @@
               (should (string= (buffer-string) "arg1 = \"x &y\", arg2 = \n"))))
         (when batfile (delete-file batfile))))))
 
+(ert-deftest process-test-stderr-buffer ()
+  (skip-unless (executable-find "bash"))
+  (let* ((stdout-buffer (generate-new-buffer "*stdout*"))
+	 (stderr-buffer (generate-new-buffer "*stderr*"))
+	 (proc (make-process :name "test" :buffer stdout-buffer
+			     :command (list "bash" "-c"
+					    (concat "echo hello stdout!; "
+						    "echo hello stderr! >&2; "
+						    "exit 20"))
+			     :stderr stderr-buffer))
+	 (sentinel-called nil)
+	 (start-time (float-time)))
+    (while (not (or sentinel-called
+		    (> (- (float-time) start-time)
+		       process-test-sentinel-wait-timeout)))
+      (accept-process-output))
+    (cl-assert (eq (process-status proc) 'exit))
+    (cl-assert (= (process-exit-status proc) 20))
+    (should (with-current-buffer stdout-buffer
+	      (goto-char (point-min))
+	      (looking-at "hello stdout!")))
+    (should (with-current-buffer stderr-buffer
+	      (goto-char (point-min))
+	      (looking-at "hello stderr!")))))
+
+(ert-deftest process-test-stderr-filter ()
+  (skip-unless (executable-find "bash"))
+  (let* ((sentinel-called nil)
+	 (stderr-sentinel-called nil)
+	 (stdout-output nil)
+	 (stderr-output nil)
+	 (stdout-buffer (generate-new-buffer "*stdout*"))
+	 (stderr-buffer (generate-new-buffer "*stderr*"))
+	 (stderr-proc (make-pipe-process :name "stderr"
+					 :buffer stderr-buffer))
+	 (proc (make-process :name "test" :buffer stdout-buffer
+			     :command (list "bash" "-c"
+					    (concat "echo hello stdout!; "
+						    "echo hello stderr! >&2; "
+						    "exit 20"))
+			     :stderr stderr-proc))
+	 (start-time (float-time)))
+    (set-process-filter proc (lambda (proc input)
+			       (push input stdout-output)))
+    (set-process-sentinel proc (lambda (proc msg)
+				 (setq sentinel-called t)))
+    (set-process-filter stderr-proc (lambda (proc input)
+				      (push input stderr-output)))
+    (set-process-sentinel stderr-proc (lambda (proc input)
+					(setq stderr-sentinel-called t)))
+    (while (not (or sentinel-called
+		    (> (- (float-time) start-time)
+		       process-test-sentinel-wait-timeout)))
+      (accept-process-output))
+    (cl-assert (eq (process-status proc) 'exit))
+    (cl-assert (= (process-exit-status proc) 20))
+    (should sentinel-called)
+    (should (equal 1 (with-current-buffer stdout-buffer
+		       (point-max))))
+    (should (equal "hello stdout!\n"
+		   (mapconcat #'identity (nreverse stdout-output) "")))
+    (should stderr-sentinel-called)
+    (should (equal 1 (with-current-buffer stderr-buffer
+		       (point-max))))
+    (should (equal "hello stderr!\n"
+		   (mapconcat #'identity (nreverse stderr-output) "")))))
+
 (provide 'process-tests)
-- 
2.1.0

Re: [PATCH] Add facility to collect stderr of async subprocess

[Eli Zaretskii]

> From: Daiki Ueno <ueno@gnu.org>
> Cc: emacs-devel@gnu.org
> Date: Wed, 08 Apr 2015 09:21:28 +0900
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > However, please note that the call to emacs_pipe2 in make-pipe-process
> > will hit an assertion in w32.c's implementation of pipe2.
> 
> OK, I added a wrapper around pipe2 in w32.c to preserve the assert
> condition.  With the attached patch, I confirmed that the included test
> passes on both Windows and GNU/Linux.

Thanks.

However, I think this is not the right solution.  The assertion is
there because w32.c:pipe2 calls _pipe with two flags, _O_BINARY and
_O_NOINHERIT, which exactly correspond to the flags being tested by
the assertion.  To support different combinations of flags, the flags
passed to _pipe need to be computed dynamically based on the argument
passed to pipe2.  By contrast, with your wrapper, the pipe will always
be closed in the subprocess, which I think is not what we want (does
the test actually test this aspect?), and is at least confusing when
reading the source (a.k.a. "looks like a bug").

> - Add a new function 'pipe_open' analogous to 'serial_open', which has
>   different definitions on Windows and on GNU/Linux

The sysdep.c definition of pipe_open is for all Posix platforms, I
think, not just for GNU/Linux.

Re: [PATCH] Add facility to collect stderr of async subprocess

[Eli Zaretskii]

> From: Daiki Ueno <ueno@gnu.org>
> Date: Wed, 08 Apr 2015 11:55:17 +0900
> Cc: emacs-devel@gnu.org
> 
> I updated the patch and it's now much simpler.

Thanks.  I'm not sure I'm following this bit:

> +  if (emacs_pipe (p->open_fd + SUBPROCESS_STDIN) != 0
> +      || emacs_pipe (p->open_fd + READ_FROM_SUBPROCESS) != 0)
> +    report_file_error ("Creating pipe", Qnil);
> +  outchannel = p->open_fd[WRITE_TO_SUBPROCESS];
> +  inchannel = p->open_fd[READ_FROM_SUBPROCESS];
> +
> +  fcntl (inchannel, F_SETFL, O_NONBLOCK);
> +  fcntl (outchannel, F_SETFL, O_NONBLOCK);
> +
> +#ifdef WINDOWSNT
> +  register_aux_fd (inchannel, outchannel);
> +#endif

The function register_aux_fd records both of its arguments in the
fd_info[] array.  Why do we need to record outchannel?  We never watch
on MS-Windows the descriptors used to write to the subprocesses, and
all the writes to them are made from the man thread.  E.g.,
register_child only registers a single file descriptor.  Why do we
need to record two in this case?  (If you worry about losing it, then
don't: they are both recorded in the process object.)

What am I missing?

Thanks.

Re: [PATCH] Add facility to collect stderr of async subprocess

[Eli Zaretskii]

> Date: Wed, 08 Apr 2015 09:17:21 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: eggert@cs.ucla.edu, emacs-devel@gnu.org
> 
> The function register_aux_fd records both of its arguments in the
> fd_info[] array.  Why do we need to record outchannel?  We never watch
> on MS-Windows the descriptors used to write to the subprocesses, and
> all the writes to them are made from the man thread.  E.g.,
                                           ^^^
"main", of course.  Sorry.

Re: [PATCH] Add facility to collect stderr of async subprocess

[Daiki Ueno]

Eli Zaretskii <eliz@gnu.org> writes:

> The function register_aux_fd records both of its arguments in the
> fd_info[] array.  Why do we need to record outchannel?  We never watch
> on MS-Windows the descriptors used to write to the subprocesses, and
> all the writes to them are made from the man thread.  E.g.,
> register_child only registers a single file descriptor.  Why do we
> need to record two in this case?  (If you worry about losing it, then
> don't: they are both recorded in the process object.)
>
> What am I missing?

No, that's my misunderstanding.  Thanks for pointing it out.  Also,
serial port related stuff in that function (copied from 'serial_open')
was not necessary.  The 'register_aux_fd' function now looks like:

void
register_aux_fd (int fd)
{
  cp = new_child ();
  if (!cp)
    error ("Could not create child process");
  cp->fd = fd;
  cp->status = STATUS_READ_ACKNOWLEDGED;

  if (fd_info[ fd ].cp != NULL)
    {
      error ("fd_info[fd = %d] is already in use", fd);
    }
  fd_info[ fd ].cp = cp;
  fd_info[ fd ].hnd = (HANDLE) _get_osfhandle (fd);
}

Regards,
-- 
Daiki Ueno

Re: [PATCH] Add facility to collect stderr of async subprocess

[Daiki Ueno]

Daiki Ueno <ueno@gnu.org> writes:

> No, that's my misunderstanding.  Thanks for pointing it out.  Also,
> serial port related stuff in that function (copied from 'serial_open')
> was not necessary.  The 'register_aux_fd' function now looks like:

I've pushed it to master.

Thanks,
-- 
Daiki Ueno

Re: [PATCH] Add facility to collect stderr of async subprocess

[Ted Zlatanov]

On Sat, 11 Apr 2015 08:11:42 +0900 Daiki Ueno <ueno@gnu.org> wrote: 

DU> Daiki Ueno <ueno@gnu.org> writes:
>> No, that's my misunderstanding.  Thanks for pointing it out.  Also,
>> serial port related stuff in that function (copied from 'serial_open')
>> was not necessary.  The 'register_aux_fd' function now looks like:

DU> I've pushed it to master.

This and the `make-process' patch are going to be very useful all
around, thank you!

Ted

Re: [PATCH] Add facility to collect stderr of async subprocess

[Tino Calancha]

Daiki Ueno <ueno@gnu.org> writes:

> Daiki Ueno <ueno@gnu.org> writes:
> I've pushed it to master.

How about to allow direct standard-error output to a different buffer
in start-process as well?

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
From 087970e4952476d94c43927ed96b1b960fc316fc Mon Sep 17 00:00:00 2001
From: Tino Calancha <tino.calancha@gmail.com>
Date: Wed, 5 Oct 2016 11:59:25 +0900
Subject: [PATCH] start-process: Allow separate standard-error from
 standard-output

* lisp/subr.el (start-process): Allow buffer being a cons
(STDOUT-BUFFER . STDERR-BUFFER); the car is the buffer-or-name
for the standard outout, the cdr is the buffer-or-name for the
standard error.
Update doc string.
* lisp/simple.el (async-shell-command):
Use start-process-shell-command instead of start-process.
If error-buffer is non-nil, then write standard-error output there.
* etc/TODO (Other features we would like): Drop this task from TODO list.
* doc/lispref/processes.texi (Asynchronous Processes, Output from Processes):
Update manual.
; * etc/NEWS (Lisp Changes in Emacs 26.1): Add news entry.
---
 doc/lispref/processes.texi | 17 ++++++++++++++---
 etc/NEWS                   |  3 +++
 etc/TODO                   |  3 ---
 lisp/simple.el             | 10 +++++++---
 lisp/subr.el               | 20 ++++++++++----------
 5 files changed, 34 insertions(+), 19 deletions(-)

diff --git a/doc/lispref/processes.texi b/doc/lispref/processes.texi
index 87c0b5c..4fbdf47 100644
--- a/doc/lispref/processes.texi
+++ b/doc/lispref/processes.texi
@@ -745,7 +745,11 @@ Asynchronous Processes
 for the new subprocess in Lisp.  The argument @var{name} specifies the
 name for the process object; as with @code{make-process}, it is
 modified if necessary to make it unique.  The buffer
-@var{buffer-or-name} is the buffer to associate with the process.
+@var{buffer-or-name} is the buffer to associate with the process.  It
+might be a cons @w{@code{(@var{stdout-buffer} . @var{stderr-buffer})}}
+as well; in this case @var{stdout-buffer} is the buffer to associate with
+the standard output of the process, and @var{stderr-buffer} is the buffer
+to associate with its error output.
 
 If @var{program} is @code{nil}, Emacs opens a new pseudoterminal (pty)
 and associates its input and output with @var{buffer-or-name}, without
@@ -1359,8 +1363,15 @@ Output from Processes
 Emacs uses a pseudo-TTY (pty) for communication with the subprocess,
 then it is impossible to separate the standard output and standard
 error streams of the subprocess, because a pseudo-TTY has only one
-output channel.  In that case, if you want to keep the output to those
-streams separate, you should redirect one of them to a file---for
+output channel.  In case you want to keep the output to those
+streams separate, you should provide also a buffer for the standard
+error.  Then Emacs doesn't use a pseudo-TTY for communication with
+the subprocess.  The way to do that is to start the subprocess with
+the argument @var{buffer-or-name} being a cons
+@w{@code{(@var{stdout-buffer} . @var{stderr-buffer})}}; in that case,
+the subprocess writes its standard output in @var{stdout-buffer} and
+writes its error output in @var{stderr-buffer}.
+Another way could be redirect one of them to a file---for
 example, by using an appropriate shell command via
 @code{start-process-shell-command} or a similar function.
 
diff --git a/etc/NEWS b/etc/NEWS
index bd94c94..a60d859 100644
--- a/etc/NEWS
+++ b/etc/NEWS
@@ -499,6 +499,9 @@ function 'check-declare-errmsg' has been removed.
 \f
 * Lisp Changes in Emacs 26.1
 
+** Now 'start-process' can direct standard-error output to a different
+buffer.
+
 ** New function undo-amalgamate-change-group to get rid of undo-boundaries
 between two states.
 
diff --git a/etc/TODO b/etc/TODO
index fe0e2ac..6749afd 100644
--- a/etc/TODO
+++ b/etc/TODO
@@ -489,9 +489,6 @@ from the emacsclient process.
   immediately, then replace it later.  So that C-s a with
   input method latin-1-postfix would immediately search for an a.
 
-** Give start-process the ability to direct standard-error
-  output to a different filter.
-
 ** Give desktop.el a feature to switch between different named desktops.
 
 ** Add a cpio mode, more or less like tar mode.
diff --git a/lisp/simple.el b/lisp/simple.el
index 70bd759..4c734f2 100644
--- a/lisp/simple.el
+++ b/lisp/simple.el
@@ -3480,9 +3480,13 @@ shell-command
 		  (display-buffer buffer '(nil (allow-no-window . t)))
                   (shell-command--save-pos-or-erase)
 		  (setq default-directory directory)
-		  (setq proc (start-process "Shell" buffer shell-file-name
-					    shell-command-switch command))
-		  (setq mode-line-process '(":%s"))
+		  (setq proc (start-process-shell-command
+                              "Shell"
+                              (if error-buffer
+                                  `(,buffer . ,error-buffer)
+                                buffer)
+                              command)
+                        mode-line-process '(":%s"))
 		  (require 'shell) (shell-mode)
 		  (set-process-sentinel proc 'shell-command-sentinel)
 		  ;; Use the comint filter for proper handling of carriage motion
diff --git a/lisp/subr.el b/lisp/subr.el
index b143812..03b902e 100644
--- a/lisp/subr.el
+++ b/lisp/subr.el
@@ -1989,24 +1989,24 @@ start-process
 NAME is name for process.  It is modified if necessary to make it unique.
 BUFFER is the buffer (or buffer name) to associate with the process.
 
-Process output (both standard output and standard error streams) goes
+If buffer is a cons (STDOUT-BUFFER . STDERR-BUFFER), then the standard
+output and the standard error appear in different buffers.  Otherwise,
+process output (both standard output and standard error streams) goes
 at end of BUFFER, unless you specify an output stream or filter
 function to handle the output.  BUFFER may also be nil, meaning that
 this process is not associated with any buffer.
 
 PROGRAM is the program file name.  It is searched for in `exec-path'
 \(which see).  If nil, just associate a pty with the buffer.  Remaining
-arguments are strings to give program as arguments.
-
-If you want to separate standard output from standard error, use
-`make-process' or invoke the command through a shell and redirect
-one of them using the shell syntax."
+arguments are strings to give program as arguments."
   (unless (fboundp 'make-process)
     (error "Emacs was compiled without subprocess support"))
-  (apply #'make-process
-	 (append (list :name name :buffer buffer)
-		 (if program
-		     (list :command (cons program program-args))))))
+  (apply #'make-process :name name
+         (append (if (consp buffer)
+                     (list :buffer (car buffer) :stderr (cdr buffer))
+                   (list :buffer buffer))
+                 (if program
+                     (list :command (cons program program-args))))))
 
 (defun process-lines (program &rest args)
   "Execute PROGRAM with ARGS, returning its output as a list of lines.
-- 
2.9.3

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
In GNU Emacs 25.1.50.1 (x86_64-pc-linux-gnu, GTK+ Version 3.22.0)
 of 2016-10-04 built on calancha-pc
Repository revision: 13ba5af7427bdbd022a9d449dc2987d6a96591eb

Re: [PATCH] Add facility to collect stderr of async subprocess

[Eli Zaretskii]

> From: Tino Calancha <tino.calancha@gmail.com>
> Date: Wed, 05 Oct 2016 13:33:17 +0900
> Cc: tino.calancha@gmail.com
> 
> How about to allow direct standard-error output to a different buffer
> in start-process as well?

This proposal has a disadvantage of being confusingly similar to what
call-process provides, except that there the cdr is a file name, not a
buffer.

What advantages do you see to adding this convenience feature, when
make-process already provides it?

Thanks.

Re: [PATCH] Add facility to collect stderr of async subprocess

[Tino Calancha]

On Wed, 5 Oct 2016, Eli Zaretskii wrote:

>> From: Tino Calancha <tino.calancha@gmail.com>
>> Date: Wed, 05 Oct 2016 13:33:17 +0900
>> Cc: tino.calancha@gmail.com
>>
>> How about to allow direct standard-error output to a different buffer
>> in start-process as well?
>
> This proposal has a disadvantage of being confusingly similar to what
> call-process provides, except that there the cdr is a file name, not a
> buffer.
I see as an advantage: it follows similar syntaxis as `call-process', so
it's easier to remember.
The difference 'buffer' <--> 'file' makes clear in the doc strings.

> What advantages do you see to adding this convenience feature, when
> make-process already provides it?
Again, some people may be familiar with (buf-out . buf-err) idiom from
`call-process', and they may be reluctant to use functions using
keyword arguments.  Not my case, though.

Re: [PATCH] Add facility to collect stderr of async subprocess

[Eli Zaretskii]

> From: Tino Calancha <tino.calancha@gmail.com>
> Date: Wed, 5 Oct 2016 16:10:25 +0900 (JST)
> cc: Tino Calancha <tino.calancha@gmail.com>, emacs-devel@gnu.org
> 
> > This proposal has a disadvantage of being confusingly similar to what
> > call-process provides, except that there the cdr is a file name, not a
> > buffer.
> I see as an advantage: it follows similar syntaxis as `call-process', so
> it's easier to remember.

So you see an advantage where I see a disadvantage.  Interesting.

> The difference 'buffer' <--> 'file' makes clear in the doc strings.

It is IME best not to create confusion that requires to read the
documentation in the first place.

> > What advantages do you see to adding this convenience feature, when
> > make-process already provides it?
> Again, some people may be familiar with (buf-out . buf-err) idiom from
> `call-process', and they may be reluctant to use functions using
> keyword arguments.  Not my case, though.

Does anyone else have an opinion on this proposal?

Re: [PATCH] Add facility to collect stderr of async subprocess

[Alain Schneble]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Tino Calancha <tino.calancha@gmail.com>
>> Date: Wed, 05 Oct 2016 13:33:17 +0900
>> Cc: tino.calancha@gmail.com
>> 
>> How about to allow direct standard-error output to a different buffer
>> in start-process as well?
>
> This proposal has a disadvantage of being confusingly similar to what
> call-process provides, except that there the cdr is a file name, not a
                                               ^^^
FWIW, the cAdr is the file name, I think.  DESTINATION in call-process
is a list and not a single cons cell, if I read doc and code correctly.

> buffer.

Alain

Re: [PATCH] Add facility to collect stderr of async subprocess

[Tino Calancha]

On Wed, 5 Oct 2016, Alain Schneble wrote:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>>> From: Tino Calancha <tino.calancha@gmail.com>
>>> Date: Wed, 05 Oct 2016 13:33:17 +0900
>>> Cc: tino.calancha@gmail.com
>>>
>>> How about to allow direct standard-error output to a different buffer
>>> in start-process as well?
>>
>> This proposal has a disadvantage of being confusingly similar to what
>> call-process provides, except that there the cdr is a file name, not a
>                                               ^^^
> FWIW, the cAdr is the file name, I think.  DESTINATION in call-process
> is a list and not a single cons cell, if I read doc and code correctly.
You are right.
May i ask you if you prefer keep the things as they are or add this 
feature?
Thank you.

Re: [PATCH] Add facility to collect stderr of async subprocess

[Michael Albinus]

Eli Zaretskii <eliz@gnu.org> writes:

>> How about to allow direct standard-error output to a different buffer
>> in start-process as well?
>
> This proposal has a disadvantage of being confusingly similar to what
> call-process provides, except that there the cdr is a file name, not a
> buffer.
>
> What advantages do you see to adding this convenience feature, when
> make-process already provides it?

Usually, start-file-process follows the argument list of
start-process. And the handlers for start-file-process do not use
make-process.

Personally, I'm undecided whether we shall add this feature.

> Thanks.

Best regards, Michael.

Re: [PATCH] Add facility to collect stderr of async subprocess

[John Wiegley]

[-- Attachment #1: Type: text/plain, Size: 900 bytes --]

>>>>> "EZ" == Eli Zaretskii <eliz@gnu.org> writes:

>> > What advantages do you see to adding this convenience feature, when >
>> make-process already provides it? Again, some people may be familiar with
>> (buf-out . buf-err) idiom from `call-process', and they may be reluctant to
>> use functions using keyword arguments. Not my case, though.

EZ> Does anyone else have an opinion on this proposal?

Since we have `make-process' for cases like this, I don't see a need to extend
`call-process'. I'd rather we made changes like this either because (a) lots
of people want it or (b) it's blocking someone from doing something -- and not
purely from a desire for consistency. I see such changes as wholly unnecessary
at this time.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 629 bytes --]

Re: [PATCH] Add facility to collect stderr of async subprocess

[Eli Zaretskii]

> From: Michael Albinus <michael.albinus@gmx.de>
> Date: Wed, 05 Oct 2016 13:20:54 +0200
> Cc: emacs-devel@gnu.org, Tino Calancha <tino.calancha@gmail.com>
> 
> Usually, start-file-process follows the argument list of
> start-process. And the handlers for start-file-process do not use
> make-process.

How hard would it be to implement this functionality in
start-file-process?

Re: [PATCH] Add facility to collect stderr of async subprocess

[Tino Calancha]

On Wed, 5 Oct 2016, John Wiegley wrote:

> Since we have `make-process' for cases like this, I don't see a need to extend
> `call-process'. I'd rather we made changes like this either because (a) lots
> of people want it or (b) it's blocking someone from doing something -- and not
> purely from a desire for consistency. I see such changes as wholly unnecessary
> at this time.

Well, it's not just my typical wish of consistency in what
the functions should do... it's also a matter of consistency with
Emacs own TODO list:
We have an entry in the TODO list requesting exactly what i am
suggesting :-)

Question:
can we conclude that following entry in TODO file has being
fulfilled by Daiki work?
-----
-** Give start-process the ability to direct standard-error
-  output to a different filter.
-----

The entry mentions `start-process' which currently cannot do the
stdout, stderr separation.  As pointed out by Michael, 
`start-file-process' either.
If we understand this entry as:
-----
Give Emacs the ability to direct standard-error
output to a different filter on asynchronous processes.
-----

then it's already done and we can drop such entry from TODO file.

Regards,
Tino

Re: [PATCH] Add facility to collect stderr of async subprocess

[Eli Zaretskii]

> From: Tino Calancha <tino.calancha@gmail.com>
> Date: Thu, 6 Oct 2016 12:13:11 +0900 (JST)
> 
> We have an entry in the TODO list requesting exactly what i am
> suggesting :-)

It's an old item (from 2004), written before we had make-process.
Thanks for pointing this out.

> Question:
> can we conclude that following entry in TODO file has being
> fulfilled by Daiki work?

Yes, I think so.

Re: [PATCH] Add facility to collect stderr of async subprocess

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 1180 bytes --]

Eli Zaretskii <eliz@gnu.org> schrieb am Mi., 5. Okt. 2016, 09:38:

> > From: Tino Calancha <tino.calancha@gmail.com>
> > Date: Wed, 5 Oct 2016 16:10:25 +0900 (JST)
> > cc: Tino Calancha <tino.calancha@gmail.com>, emacs-devel@gnu.org
> >
> > > This proposal has a disadvantage of being confusingly similar to what
> > > call-process provides, except that there the cdr is a file name, not a
> > > buffer.
> > I see as an advantage: it follows similar syntaxis as `call-process', so
> > it's easier to remember.
>
> So you see an advantage where I see a disadvantage.  Interesting.
>
> > The difference 'buffer' <--> 'file' makes clear in the doc strings.
>
> It is IME best not to create confusion that requires to read the
> documentation in the first place.
>
> > > What advantages do you see to adding this convenience feature, when
> > > make-process already provides it?
> > Again, some people may be familiar with (buf-out . buf-err) idiom from
> > `call-process', and they may be reluctant to use functions using
> > keyword arguments.  Not my case, though.
>
> Does anyone else have an opinion on this proposal?
>

My opinion would be to mark start-process as obsolete.

>

[-- Attachment #2: Type: text/html, Size: 2412 bytes --]

Re: [PATCH] Add facility to collect stderr of async subprocess

[Tino Calancha]

On Thu, 6 Oct 2016, Eli Zaretskii wrote:

>> From: Tino Calancha <tino.calancha@gmail.com>
>> Date: Thu, 6 Oct 2016 12:13:11 +0900 (JST)
>>
>> We have an entry in the TODO list requesting exactly what i am
>> suggesting :-)
>
> It's an old item (from 2004), written before we had make-process.
> Thanks for pointing this out.
>
>> Question:
>> can we conclude that following entry in TODO file has being
>> fulfilled by Daiki work?
>
> Yes, I think so.
I agree : i also think the idea underneath such TODO entry
matches what Daiki san did.

By the way, i am fine if we don't follow my suggestion.
I just want to remark that my patch repect backward compatibility:
callers passing a buffer will get the traditional behaviour.
It is just in case we pass a list that we will separate
the stderr from stdout.
As a bonus, several other functions will automatically inherit this 
feature: `start-file-process', `start-process-shell-command', and 
`start-file-process-shell-command'.
Personally, i find pleasant if every function creating an asynchronous 
process us able to separate stdout from stderr.

Another minor thing is that my patch fix a bug in async-shell-command:
current implementation don't use the second optional argument 
ERROR-BUFFER.  Of course, that could be fixed using `make-process'.

Re: [PATCH] Add facility to collect stderr of async subprocess

[Michael Albinus]

Eli Zaretskii <eliz@gnu.org> writes:

>> Usually, start-file-process follows the argument list of
>> start-process. And the handlers for start-file-process do not use
>> make-process.
>
> How hard would it be to implement this functionality in
> start-file-process?

Not trivial. I need to start a second remote process, in order to
distinguish between stdout and stderr. But shall be possible.

This is for tramp-adb-handle-start-file-process and
tramp-sh-handle-start-file-process. tramp-smb-handle-start-file-process
won't support this, likely.

Best regards, Michael.

Re: [PATCH] Add facility to collect stderr of async subprocess

[Eli Zaretskii]

> From: Philipp Stephani <p.stephani2@gmail.com>
> Date: Thu, 06 Oct 2016 07:15:42 +0000
> Cc: emacs-devel@gnu.org
> 
> My opinion would be to mark start-process as obsolete.

??? Why?  It's a perfectly okay API, not to mention that it's used in
gazillions of places, including in Emacs itself.

Re: [PATCH] Add facility to collect stderr of async subprocess

[Eli Zaretskii]

> From: Tino Calancha <tino.calancha@gmail.com>
> Date: Thu, 6 Oct 2016 16:25:39 +0900 (JST)
> cc: Tino Calancha <tino.calancha@gmail.com>, jwiegley@gmail.com, 
>     emacs-devel@gnu.org, michael.albinus@gmx.de
> 
> As a bonus, several other functions will automatically inherit this 
> feature: `start-file-process', `start-process-shell-command', and 
> `start-file-process-shell-command'.

As Micheal explained, start-file-process and
start-file-process-shell-command cannot inherit this without some
non-trivial coding for the remote case.

> Personally, i find pleasant if every function creating an asynchronous 
> process us able to separate stdout from stderr.

Actually, the need in this separation is rather rare.  Which is not
surprising, since running commands from a terminal by default delivers
both stdout and stderr to the screen, and the cases where these are
redirected separately are rare.

Re: [PATCH] Add facility to collect stderr of async subprocess

[Tino Calancha]

On Thu, 6 Oct 2016, Eli Zaretskii wrote:

>> From: Tino Calancha <tino.calancha@gmail.com>
>> Date: Thu, 6 Oct 2016 16:25:39 +0900 (JST)
>> cc: Tino Calancha <tino.calancha@gmail.com>, jwiegley@gmail.com,
>>     emacs-devel@gnu.org, michael.albinus@gmx.de
>>
>> As a bonus, several other functions will automatically inherit this
>> feature: `start-file-process', `start-process-shell-command', and
>> `start-file-process-shell-command'.
>
> As Micheal explained, start-file-process and
> start-file-process-shell-command cannot inherit this without some
> non-trivial coding for the remote case.
Fair enough, but it wouldn't be the first time that we get more
locally compared with remote, for instance:
Bug#24394
or
https://lists.gnu.org/archive/html/tramp-devel/2015-12/msg00000.html

>> Personally, i find pleasant if every function creating an asynchronous
>> process us able to separate stdout from stderr.
>
> Actually, the need in this separation is rather rare.  Which is not
> surprising, since running commands from a terminal by default delivers
> both stdout and stderr to the screen, and the cases where these are
> redirected separately are rare.
It depends of what the user is doing.  `shell-command' and 
`async-shell-command' have being offering that since long time ago.

It is quite common redirect stderr from a shell: all shells allow that 
AFAIK.
In Emacs, there is the idiom that everything is a buffer, so redirecting
stderr to a buffer instead of a file (as `call-process') seems the
natural thing.  Or maybe I'm missing something.

Re: [PATCH] Add facility to collect stderr of async subprocess

[Eli Zaretskii]

> From: Tino Calancha <tino.calancha@gmail.com>
> Date: Thu, 6 Oct 2016 17:37:05 +0900 (JST)
> Cc: jwiegley@gmail.com, Emacs developers <emacs-devel@gnu.org>,
> 	michael.albinus@gmx.de, Tino Calancha <tino.calancha@gmail.com>
> 
> > Actually, the need in this separation is rather rare.  Which is not
> > surprising, since running commands from a terminal by default delivers
> > both stdout and stderr to the screen, and the cases where these are
> > redirected separately are rare.
> It depends of what the user is doing.  `shell-command' and 
> `async-shell-command' have being offering that since long time ago.
> 
> It is quite common redirect stderr from a shell: all shells allow that 
> AFAIK.

My point is not about allowing it -- we definitely do.  My point is
about the importance of having this in every API.  I'm saying that
this feature is relatively rarely needed, that's all.  How many times
did you need to redirect stderr separately from stdout in shell
commands you run from the shell prompt?

Re: [PATCH] Add facility to collect stderr of async subprocess

[Tino Calancha]

[-- Attachment #1: Type: text/plain, Size: 1721 bytes --]



On Thu, 6 Oct 2016, Eli Zaretskii wrote:

>> From: Tino Calancha <tino.calancha@gmail.com>
>> Date: Thu, 6 Oct 2016 17:37:05 +0900 (JST)
>> Cc: jwiegley@gmail.com, Emacs developers <emacs-devel@gnu.org>,
>> 	michael.albinus@gmx.de, Tino Calancha <tino.calancha@gmail.com>
>>
>>> Actually, the need in this separation is rather rare.  Which is not
>>> surprising, since running commands from a terminal by default delivers
>>> both stdout and stderr to the screen, and the cases where these are
>>> redirected separately are rare.
>> It depends of what the user is doing.  `shell-command' and
>> `async-shell-command' have being offering that since long time ago.
>>
>> It is quite common redirect stderr from a shell: all shells allow that
>> AFAIK.
>
> My point is not about allowing it -- we definitely do.  My point is
> about the importance of having this in every API.
If Michael is not ging to support it in remote i must agree with you
and say that is not important. 
>I'm saying that
> this feature is relatively rarely needed, that's all.  How many times
> did you need to redirect stderr separately from stdout in shell
> commands you run from the shell prompt?
Honestly, not often.

We might modify the doc strings for shell-comamand and 
async-shell-command, following part:
-----
In Elisp, you will often be better served by calling ‘start-process’
directly, since it offers more control and does not impose the use of a
shell (with its need to quote arguments).
-----

I understand what Stefan means here with more control, but someone may 
argue that s-c and a-s-c allow the possibility to separate stdout from
stderr, which s-p does not.  That sounds like less control, at least
with respect the output.

Re: [PATCH] Add facility to collect stderr of async subprocess

[Michael Albinus]

Tino Calancha <tino.calancha@gmail.com> writes:

>> As Micheal explained, start-file-process and
>> start-file-process-shell-command cannot inherit this without some
>> non-trivial coding for the remote case.
> Fair enough, but it wouldn't be the first time that we get more
> locally compared with remote, for instance:
> Bug#24394
> or
> https://lists.gnu.org/archive/html/tramp-devel/2015-12/msg00000.html

This one is on hold. As soon as we have connection-local variables,
I'll implement a fix.

Btw, I'm checking whether connection-local variables could be
implemented as special case of directory-local variables. But this is
not trivial.

Best regards, Michael.

Re: [PATCH] Add facility to collect stderr of async subprocess

[Michael Albinus]

Tino Calancha <tino.calancha@gmail.com> writes:

>> My point is not about allowing it -- we definitely do.  My point is
>> about the importance of having this in every API.
> If Michael is not ging to support it in remote i must agree with you
> and say that is not important. 

I haven't said I won't do it. But it might take time.

Best regards, Michael.

Re: [PATCH] Add facility to collect stderr of async subprocess

[Tino Calancha]

On Thu, 6 Oct 2016, Michael Albinus wrote:

> Tino Calancha <tino.calancha@gmail.com> writes:
>
>>> My point is not about allowing it -- we definitely do.  My point is
>>> about the importance of having this in every API.
>> If Michael is not ging to support it in remote i must agree with you
>> and say that is not important.
>
> I haven't said I won't do it. But it might take time.
Sorry Michael, i didn't mean that you are against that feature.
By the way, thank you very much for your great work in tramp: it's
simply amazing.