bug#24287: Process filters received stderr output, not documented

bug#24287: Process filters received stderr output, not documented

[Paul A. Steckler]

The documentation for Process Filter Functions mentions that process
filters receive the standard output from their associated processes. It
does not mention that the filters also receive standard error output.


In GNU Emacs 24.5.1 (x86_64-pc-linux-gnu, GTK+ Version 3.18.9)
 of 2016-04-17 on lgw01-04, modified by Debian
System Description:    Linux Mint 18 Sarah

Configured using:
 `configure --build x86_64-linux-gnu --prefix=/usr
 --sharedstatedir=/var/lib --libexecdir=/usr/lib
 --localstatedir=/var/lib --infodir=/usr/share/info
 --mandir=/usr/share/man --with-pop=yes
 --enable-locallisppath=/etc/emacs24:/etc/emacs:/usr/local/share/emacs/24.5/site-lisp:/usr/local/share/emacs/site-lisp:/usr/share/emacs/24.5/site-lisp:/usr/share/emacs/site-lisp
 --build x86_64-linux-gnu --prefix=/usr --sharedstatedir=/var/lib
 --libexecdir=/usr/lib --localstatedir=/var/lib
 --infodir=/usr/share/info --mandir=/usr/share/man --with-pop=yes
 --enable-locallisppath=/etc/emacs24:/etc/emacs:/usr/local/share/emacs/24.5/site-lisp:/usr/local/share/emacs/site-lisp:/usr/share/emacs/24.5/site-lisp:/usr/share/emacs/site-lisp
 --with-x=yes --with-x-toolkit=gtk3 --with-toolkit-scroll-bars
 'CFLAGS=-g -O2 -fstack-protector-strong -Wformat
 -Werror=format-security -Wall' 'CPPFLAGS=-Wdate-time
 -D_FORTIFY_SOURCE=2' 'LDFLAGS=-Wl,-Bsymbolic-functions -Wl,-z,relro''

Important settings:
  value of $LANG: en_US.UTF-8
  locale-coding-system: utf-8-unix

Major mode: Fundamental

Minor modes in effect:
  tooltip-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  buffer-read-only: t
  line-number-mode: t
  transient-mark-mode: t

Recent messages:
Loading /etc/emacs/site-start.d/50cmake-data.el (source)...done
Loading /etc/emacs/site-start.d/50dictionaries-common.el (source)...
Loading debian-ispell...
Loading /var/cache/dictionaries-common/emacsen-ispell-default.el (source)...done
Loading debian-ispell...done
Loading /var/cache/dictionaries-common/emacsen-ispell-dicts.el (source)...done
Loading /etc/emacs/site-start.d/50dictionaries-common.el (source)...done
Loading /etc/emacs/site-start.d/51tuareg-mode.el (source)...done
For information about GNU Emacs and the GNU system, type C-h C-a.
Making completion list...

Load-path shadows:
/usr/share/emacs24/site-lisp/cmake-data/cmake-mode hides
/usr/share/emacs/site-lisp/cmake-mode
/usr/share/emacs/24.5/site-lisp/debian-startup hides
/usr/share/emacs/site-lisp/debian-startup

Features:
(shadow sort gnus-util mail-extr emacsbug message format-spec rfc822 mml
mml-sec mm-decode mm-bodies mm-encode mail-parse rfc2231 mailabbrev
gmm-utils mailheader sendmail rfc2047 rfc2045 ietf-drums mm-util
help-fns mail-prsvr mail-utils help-mode easymenu xterm time-date
package epg-config tooltip electric uniquify ediff-hook vc-hooks
lisp-float-type mwheel x-win x-dnd tool-bar dnd fontset image regexp-opt
fringe tabulated-list newcomment lisp-mode prog-mode register page
menu-bar rfn-eshadow timer select scroll-bar mouse jit-lock font-lock
syntax facemenu font-core frame cham georgian utf-8-lang misc-lang
vietnamese tibetan thai tai-viet lao korean japanese hebrew greek
romanian slovak czech european ethiopic indian cyrillic chinese
case-table epa-hook jka-cmpr-hook help simple abbrev minibuffer nadvice
loaddefs button faces cus-face macroexp files text-properties overlay
sha1 md5 base64 format env code-pages mule custom widget
hashtable-print-readable backquote make-network-process dbusbind
gfilenotify dynamic-setting system-font-setting font-render-setting
move-toolbar gtk x-toolkit x multi-tty emacs)

Memory information:
((conses 16 82441 5872)
 (symbols 48 19084 0)
 (miscs 40 71 124)
 (strings 32 11704 3827)
 (string-bytes 1 311934)
 (vectors 16 7574)
 (vector-slots 8 345765 28255)
 (floats 8 66 515)
 (intervals 56 211 1)
 (buffers 960 13)
 (heap 1024 17669 1772))

bug#24287: Process filters received stderr output, not documented

[Eli Zaretskii]

> From: "Paul A. Steckler" <steck@stecksoft.com>
> Date: Mon, 22 Aug 2016 13:35:25 -0400
> 
> The documentation for Process Filter Functions mentions that process
> filters receive the standard output from their associated processes. It
> does not mention that the filters also receive standard error output.

That is described in the parent section:

     It is impossible to separate the standard output and standard error
  streams of the subprocess, because Emacs normally spawns the subprocess
  inside a pseudo-TTY, and a pseudo-TTY has only one output channel.  If
  you want to keep the output to those streams separate, you should
  redirect one of them to a file—for example, by using an appropriate
  shell command.

bug#24287: Process filters received stderr output, not documented

[Clément Pit--Claudel]

[-- Attachment #1.1: Type: text/plain, Size: 1372 bytes --]

On 2016-08-22 14:23, Eli Zaretskii wrote:
>> From: "Paul A. Steckler" <steck@stecksoft.com>
>> Date: Mon, 22 Aug 2016 13:35:25 -0400
>>
>> The documentation for Process Filter Functions mentions that process
>> filters receive the standard output from their associated processes. It
>> does not mention that the filters also receive standard error output.
> 
> That is described in the parent section:
> 
>      It is impossible to separate the standard output and standard error
>   streams of the subprocess, because Emacs normally spawns the subprocess
>   inside a pseudo-TTY, and a pseudo-TTY has only one output channel.  If
>   you want to keep the output to those streams separate, you should
>   redirect one of them to a file—for example, by using an appropriate
>   shell command.

Still, even the parent section could be confusing:

       The output that a subprocess writes to its standard output stream is
                                                  ^^^^^^^^^^^^^^^
    passed to a function called the “filter function”.  The default filter
    function simply inserts the output into a buffer, which is called the
    associated buffer of the process (*note Process Buffers::).  If the process
    has no buffer then the default filter discards the output.

The highlighted section is not true, is it?

Clément.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

bug#24287: Process filters received stderr output, not documented

[Paul A. Steckler]

That's true, but the documentation for filter functions is misleading.
It specifically mentions the standard output of the associated
process, but neglects to mention the standard error.

Also, one might not see that earlier section when looking at the
documentation for filter functions. I did not see that section, in
fact, when using the manual to help me write my code.

-- Paul

On Mon, Aug 22, 2016 at 2:23 PM, Eli Zaretskii <eliz@gnu.org> wrote:
>> From: "Paul A. Steckler" <steck@stecksoft.com>
>> Date: Mon, 22 Aug 2016 13:35:25 -0400
>>
>> The documentation for Process Filter Functions mentions that process
>> filters receive the standard output from their associated processes. It
>> does not mention that the filters also receive standard error output.
>
> That is described in the parent section:
>
>      It is impossible to separate the standard output and standard error
>   streams of the subprocess, because Emacs normally spawns the subprocess
>   inside a pseudo-TTY, and a pseudo-TTY has only one output channel.  If
>   you want to keep the output to those streams separate, you should
>   redirect one of them to a file—for example, by using an appropriate
>   shell command.

bug#24287: Process filters received stderr output, not documented

[Eli Zaretskii]

> Cc: Eli Zaretskii <eliz@gnu.org>
> From: Clément Pit--Claudel <clement.pit@gmail.com>
> Date: Mon, 22 Aug 2016 14:38:01 -0400
> 
>        The output that a subprocess writes to its standard output stream is
>                                                   ^^^^^^^^^^^^^^^
>     passed to a function called the “filter function”.  The default filter
>     function simply inserts the output into a buffer, which is called the
>     associated buffer of the process (*note Process Buffers::).  If the process
>     has no buffer then the default filter discards the output.
> 
> The highlighted section is not true, is it?

Of course, it's true: the standard output _is_ passed to a filter.

bug#24287: Process filters received stderr output, not documented

[Clément Pit--Claudel]

[-- Attachment #1.1: Type: text/plain, Size: 2708 bytes --]

On 2016-08-22 14:23, Eli Zaretskii wrote:
>> From: "Paul A. Steckler" <steck@stecksoft.com>
>> Date: Mon, 22 Aug 2016 13:35:25 -0400
>>
>> The documentation for Process Filter Functions mentions that process
>> filters receive the standard output from their associated processes. It
>> does not mention that the filters also receive standard error output.
> 
> That is described in the parent section:
> 
>      It is impossible to separate the standard output and standard error
>   streams of the subprocess, because Emacs normally spawns the subprocess
>   inside a pseudo-TTY, and a pseudo-TTY has only one output channel.  If
>   you want to keep the output to those streams separate, you should
>   redirect one of them to a file—for example, by using an appropriate
>   shell command.

Hi Eli,

Am I misunderstanding something?  This section sounds misleading, at least for Emacs 25 (I'm not sure about Emacs 24).  The documentation of make-process (which is new in Emacs 25, right?) says:

     :stderr STDERR
          Associate STDERR with the standard error of the process.
          STDERR is either a buffer or a pipe process created with
          ‘make-pipe-process’.

The docstring of start-process was updated accordingly:

    If you want to separate standard output from standard error, use
    ‘make-process’ or invoke the command through a shell and redirect
    one of them using the shell syntax.

Both of these suggest that the impossibility that the manual mention only applies to the pty case (the manual recommends against using PTYs for non-user-facing processes anyway).  What about something like this:

    It is impossible to separate the standard output and standard error streams
    of a subprocess spawned inside a pseudo-TTY, as pseudo-TTYs have only one
    output channel.  If you want to keep the output to those streams separate,
    you should either use `make-process' with the :stderr argument, or redirect
    one of the streams to a file—for example, by using
    start-process-shell-command with an appropriate redirection.

Additionally, we could update the documentation of make-process: right now, it starts with "This function is like ‘start-process’, but takes keyword arguments.", but from what I understand it can actually do more than start-process. What about this:

     This function is like ‘start-process’, but takes keyword arguments, and can
     separate output to standard output and standard error.

Btw, was there a way in Emacs 24 to separate these streams if the process was run after setting process-connection-type to nil? (I can't think of one).

Cheers,
Clément.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

bug#24287: Process filters received stderr output, not documented

[Eli Zaretskii]

> From: Clément Pit--Claudel <clement.pit@gmail.com>
> Date: Tue, 23 Aug 2016 17:47:16 -0400
> 
> > That is described in the parent section:
> > 
> >      It is impossible to separate the standard output and standard error
> >   streams of the subprocess, because Emacs normally spawns the subprocess
> >   inside a pseudo-TTY, and a pseudo-TTY has only one output channel.  If
> >   you want to keep the output to those streams separate, you should
> >   redirect one of them to a file—for example, by using an appropriate
> >   shell command.
> 
> Hi Eli,
> 
> Am I misunderstanding something?  This section sounds misleading, at least for Emacs 25 (I'm not sure about Emacs 24).  The documentation of make-process (which is new in Emacs 25, right?) says:
> 
>      :stderr STDERR
>           Associate STDERR with the standard error of the process.
>           STDERR is either a buffer or a pipe process created with
>           ‘make-pipe-process’.
> 
> The docstring of start-process was updated accordingly:
> 
>     If you want to separate standard output from standard error, use
>     ‘make-process’ or invoke the command through a shell and redirect
>     one of them using the shell syntax.
> 
> Both of these suggest that the impossibility that the manual mention only applies to the pty case (the manual recommends against using PTYs for non-user-facing processes anyway).  What about something like this:
> 
>     It is impossible to separate the standard output and standard error streams
>     of a subprocess spawned inside a pseudo-TTY, as pseudo-TTYs have only one
>     output channel.  If you want to keep the output to those streams separate,
>     you should either use `make-process' with the :stderr argument, or redirect
>     one of the streams to a file—for example, by using
>     start-process-shell-command with an appropriate redirection.
> 
> Additionally, we could update the documentation of make-process: right now, it starts with "This function is like ‘start-process’, but takes keyword arguments.", but from what I understand it can actually do more than start-process. What about this:
> 
>      This function is like ‘start-process’, but takes keyword arguments, and can
>      separate output to standard output and standard error.

Actually, the situation was much worse than those minor changes could
handle.  The documentation of process-related APIs was not reviewed in
its entirety for quite some time, and as result bit-rotted quite a
lot.  I fixed that.

This is what happens and will continue to happen if people will not
volunteer to review documentation during the pretest, certainly when a
major version is about to be released.  For Emacs 25.1, only a single
individual volunteered, and a single *.texi file was reviewed.  I
expect the accuracy and the overall quality of our documentation to go
downhill if we don't see more volunteers.

> Btw, was there a way in Emacs 24 to separate these streams if the process was run after setting process-connection-type to nil? (I can't think of one).

No, I don't think so.

bug#24287: Process filters received stderr output, not documented

[Clément Pit--Claudel]

[-- Attachment #1.1: Type: text/plain, Size: 303 bytes --]

On 2016-08-24 10:44, Eli Zaretskii wrote:
> Actually, the situation was much worse than those minor changes could
> handle.  The documentation of process-related APIs was not reviewed in
> its entirety for quite some time, and as result bit-rotted quite a
> lot.  I fixed that.

Great, thanks!


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]