bug#35328: 27.0.50; Lisp reference, "Accepting Output": Should describe what happens to standard error

bug#35328: 27.0.50; Lisp reference, "Accepting Output": Should describe what happens to standard error

[Philipp]

The Emacs Lisp manual section "Accepting Output" now documents how to
read all output from a process (thanks!).  However, the section still
lacks some discussion what happens to standard error.  Specifically, it
should answer the following questions:

Does the code
(while (accept-process-output PROC))
also ensure that all of the standard error has been read and passed to
the filter/inserted into the buffer?

If not, how to ensure this?


In GNU Emacs 27.0.50 (build 44, x86_64-apple-darwin18.2.0, NS appkit-1671.20 Version 10.14.3 (Build 18D109))
 of 2019-04-19
Repository revision: 0685baf44b1959eda556aafad94f45354b6c1f72
Repository branch: hash-table-contains
Windowing system distributor 'Apple', version 10.3.1671
System Description:  Mac OS X 10.14.3

Recent messages:
For information about GNU Emacs and the GNU system, type C-h C-a.

Configured using:
 'configure --without-xml2 --with-modules --without-pop --with-mailutils
 --enable-gcc-warnings=warn-only --enable-checking=yes
 --enable-check-lisp-object-type 'CFLAGS=-ggdb3 -O0''

Configured features:
IMAGEMAGICK NOTIFY KQUEUE ACL GNUTLS ZLIB TOOLKIT_SCROLL_BARS NS MODULES
THREADS JSON PDUMPER LCMS2 GMP

Important settings:
  value of $LANG: de_DE.UTF-8
  locale-coding-system: utf-8-unix

Major mode: Lisp Interaction

Minor modes in effect:
  tooltip-mode: t
  global-eldoc-mode: t
  eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  transient-mark-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message rmc dired dired-loaddefs
format-spec rfc822 mml easymenu mml-sec epa epg epg-config gnus-util
rmail rmail-loaddefs text-property-search time-date mm-decode mm-bodies
mm-encode mail-parse rfc2231 mailabbrev gmm-utils mailheader sendmail
rfc2047 rfc2045 ietf-drums mm-util mail-prsvr mail-utils phst skeleton
derived edmacro kmacro pcase ffap thingatpt url-parse auth-source cl-seq
eieio eieio-core cl-macs eieio-loaddefs password-cache json map url-vars
subr-x rx gnutls puny seq byte-opt gv bytecomp byte-compile cconv dbus
xml cl-loaddefs cl-lib elec-pair tooltip eldoc electric uniquify
ediff-hook vc-hooks lisp-float-type mwheel term/ns-win ns-win
ucs-normalize mule-util term/common-win tool-bar dnd fontset image
regexp-opt fringe tabulated-list replace newcomment text-mode elisp-mode
lisp-mode prog-mode register page menu-bar rfn-eshadow isearch timer
select scroll-bar mouse jit-lock font-lock syntax facemenu font-core
term/tty-colors frame cl-generic cham georgian utf-8-lang misc-lang
vietnamese tibetan thai tai-viet lao korean japanese eucjp-ms cp51932
hebrew greek romanian slovak czech european ethiopic indian cyrillic
chinese composite charscript charprop case-table epa-hook jka-cmpr-hook
help simple abbrev obarray minibuffer cl-preloaded nadvice loaddefs
button faces cus-face macroexp files text-properties overlay sha1 md5
base64 format env code-pages mule custom widget hashtable-print-readable
backquote threads kqueue cocoa ns lcms2 multi-tty make-network-process
emacs)

Memory information:
((conses 16 59004 6608)
 (symbols 48 7730 1)
 (strings 32 19869 1366)
 (string-bytes 1 656811)
 (vectors 16 11333)
 (vector-slots 8 141146 10566)
 (floats 8 23 15)
 (intervals 56 198 0)
 (buffers 992 11))

bug#35328: 27.0.50; Lisp reference, "Accepting Output": Should describe what happens to standard error

[Eli Zaretskii]

> From: Philipp <p.stephani2@gmail.com>
> Date: Fri, 19 Apr 2019 21:36:50 +0200
> 
> The Emacs Lisp manual section "Accepting Output" now documents how to
> read all output from a process (thanks!).  However, the section still
> lacks some discussion what happens to standard error.  Specifically, it
> should answer the following questions:
> 
> Does the code
> (while (accept-process-output PROC))
> also ensure that all of the standard error has been read and passed to
> the filter/inserted into the buffer?

I don't think I understand why you say that node doesn't already
answer this question.  Why did you think standard error is special in
this regard?

bug#35328: 27.0.50; Lisp reference, "Accepting Output": Should describe what happens to standard error

[Philipp Stephani]

Am Sa., 20. Apr. 2019 um 09:13 Uhr schrieb Eli Zaretskii <eliz@gnu.org>:
>
> > From: Philipp <p.stephani2@gmail.com>
> > Date: Fri, 19 Apr 2019 21:36:50 +0200
> >
> > The Emacs Lisp manual section "Accepting Output" now documents how to
> > read all output from a process (thanks!).  However, the section still
> > lacks some discussion what happens to standard error.  Specifically, it
> > should answer the following questions:
> >
> > Does the code
> > (while (accept-process-output PROC))
> > also ensure that all of the standard error has been read and passed to
> > the filter/inserted into the buffer?
>
> I don't think I understand why you say that node doesn't already
> answer this question.  Why did you think standard error is special in
> this regard?

The node doesn't describe what happens with standard error, e.g.
whether standard error is part of "output" and how it's handled. In
particular, I can't see how the node would answer the question I
posted.

bug#35328: [PATCH] Improve documentation around standard error pipes (Bug#35328).

[Philipp Stephani]

* doc/lispref/processes.texi (Asynchronous Processes): Document
existence and properties of the standard error process.
(Accepting Output): Document that one has to accept output from the
standard error process separately.
---
 doc/lispref/processes.texi | 24 ++++++++++++++++++++++++
 1 file changed, 24 insertions(+)

diff --git a/doc/lispref/processes.texi b/doc/lispref/processes.texi
index 7eb136af5f..3af8a1d5eb 100644
--- a/doc/lispref/processes.texi
+++ b/doc/lispref/processes.texi
@@ -701,6 +701,18 @@ Asynchronous Processes
 @var{stderr} is @code{nil}, standard error is mixed with standard
 output, and both are sent to @var{buffer} or @var{filter}.
 
+If @var{stderr} is a buffer, Emacs will create a pipe process, the
+@dfn{standard error process}.  This process will have the default
+filter (@pxref{Filter Functions}), sentinel (@pxref{Sentinels}), and
+coding systems (@pxref{Default Coding Systems}).  On the other hand,
+it will use @var{query-flag} as its query-on-exit flag (@pxref{Query
+Before Exit}).  It will be associated with the @var{stderr} buffer
+(@pxref{Process Buffers}) and send its output (which is the standard
+error of the main process) there.
+
+If @var{stderr} is a pipe process, Emacs will use it as standard error
+process for the new process.
+
 @item :file-handler @var{file-handler}
 If @var{file-handler} is non-@code{nil}, then look for a file name
 handler for the current buffer's @code{default-directory}, and invoke
@@ -1882,6 +1894,18 @@ Accepting Output
 (while (accept-process-output process))
 @end example
 
+If you have passed a non-@code{nil} @var{stderr} to
+@code{make-process}, it will have a standard error process.
+@xref{Asynchronous Processes}.  In that case, waiting for process
+output from the main process doesn't wait for output from the standard
+error process.  To make sure that you have received both standard
+output and error from a process, use the following code:
+
+@example
+(while (accept-process-output process))
+(while (accept-process-output stderr-process))
+@end example
+
 @node Processes and Threads
 @subsection Processes and Threads
 @cindex processes, threads
-- 
2.20.1 (Apple Git-117)

bug#35328: 27.0.50; Lisp reference, "Accepting Output": Should describe what happens to standard error

[Eli Zaretskii]

> From: Philipp Stephani <p.stephani2@gmail.com>
> Date: Sun, 21 Apr 2019 15:58:11 +0200
> Cc: 35328@debbugs.gnu.org
> 
> > > Does the code
> > > (while (accept-process-output PROC))
> > > also ensure that all of the standard error has been read and passed to
> > > the filter/inserted into the buffer?
> >
> > I don't think I understand why you say that node doesn't already
> > answer this question.  Why did you think standard error is special in
> > this regard?
> 
> The node doesn't describe what happens with standard error, e.g.
> whether standard error is part of "output" and how it's handled.

Well, it says "output", not "standard output".  I take it to mean both
output streams.  If the text is interpreted to mean both streams, is
it accurate?  If it is, we only need to clarify that both standard
output and standard error are included.

bug#35328: 27.0.50; Lisp reference, "Accepting Output": Should describe what happens to standard error

[Philipp Stephani]

Am So., 21. Apr. 2019 um 21:12 Uhr schrieb Eli Zaretskii <eliz@gnu.org>:
>
> > From: Philipp Stephani <p.stephani2@gmail.com>
> > Date: Sun, 21 Apr 2019 15:58:11 +0200
> > Cc: 35328@debbugs.gnu.org
> >
> > > > Does the code
> > > > (while (accept-process-output PROC))
> > > > also ensure that all of the standard error has been read and passed to
> > > > the filter/inserted into the buffer?
> > >
> > > I don't think I understand why you say that node doesn't already
> > > answer this question.  Why did you think standard error is special in
> > > this regard?
> >
> > The node doesn't describe what happens with standard error, e.g.
> > whether standard error is part of "output" and how it's handled.
>
> Well, it says "output", not "standard output".  I take it to mean both
> output streams.  If the text is interpreted to mean both streams, is
> it accurate?

I don't think so. At least empirically one has to wait for the
standard error process separately to ensure that all standard error is
read. I've sent a patch to extend the documentation to say that, is
that ok?

bug#35328: [PATCH] Improve documentation around standard error pipes (Bug#35328).

[Eli Zaretskii]

> From: Philipp Stephani <p.stephani2@gmail.com>
> Cc: 35328@debbugs.gnu.org,
> 	Philipp Stephani <phst@google.com>
> Date: Sun, 21 Apr 2019 18:11:47 +0200
> 
> +If @var{stderr} is a buffer, Emacs will create a pipe process, the
> +@dfn{standard error process}.  This process will have the default

As a general rule, any use of @dfn should have a corresponding @cindex
entry leading to the text.  So please add one here.

> +If you have passed a non-@code{nil} @var{stderr} to
> +@code{make-process}, it will have a standard error process.
> +@xref{Asynchronous Processes}.  In that case, waiting for process
> +output from the main process doesn't wait for output from the standard
> +error process.  To make sure that you have received both standard
> +output and error from a process, use the following code:

Suggest a minor rewording of the last sentence:

  To make sure you have received both all of standard output and all of
  standard error from a process ...

Otherwise, LGTM, thanks.

bug#35328: 27.0.50; Lisp reference, "Accepting Output": Should describe what happens to standard error

[Eli Zaretskii]

> From: Philipp Stephani <p.stephani2@gmail.com>
> Date: Mon, 22 Apr 2019 11:14:19 +0200
> Cc: 35328@debbugs.gnu.org
> 
> > Well, it says "output", not "standard output".  I take it to mean both
> > output streams.  If the text is interpreted to mean both streams, is
> > it accurate?
> 
> I don't think so. At least empirically one has to wait for the
> standard error process separately to ensure that all standard error is
> read. I've sent a patch to extend the documentation to say that, is
> that ok?

Yes, modulo my minor comments there.

Thanks.

bug#35328: [PATCH] Improve documentation around standard error pipes (Bug#35328).

[Philipp Stephani]

Am Mo., 22. Apr. 2019 um 11:26 Uhr schrieb Eli Zaretskii <eliz@gnu.org>:
>
> > From: Philipp Stephani <p.stephani2@gmail.com>
> > Cc: 35328@debbugs.gnu.org,
> >       Philipp Stephani <phst@google.com>
> > Date: Sun, 21 Apr 2019 18:11:47 +0200
> >
> > +If @var{stderr} is a buffer, Emacs will create a pipe process, the
> > +@dfn{standard error process}.  This process will have the default
>
> As a general rule, any use of @dfn should have a corresponding @cindex
> entry leading to the text.  So please add one here.
>
> > +If you have passed a non-@code{nil} @var{stderr} to
> > +@code{make-process}, it will have a standard error process.
> > +@xref{Asynchronous Processes}.  In that case, waiting for process
> > +output from the main process doesn't wait for output from the standard
> > +error process.  To make sure that you have received both standard
> > +output and error from a process, use the following code:
>
> Suggest a minor rewording of the last sentence:
>
>   To make sure you have received both all of standard output and all of
>   standard error from a process ...
>
> Otherwise, LGTM, thanks.

Thanks, pushed as 3b4e312cfe.