From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Ioannis Kappas Newsgroups: gmane.emacs.bugs Subject: bug#47299: 27.1; emacs --batch on MS-Windows does not immediately display `print'ed lines when invoked outside of the Command Prompt Date: Sun, 21 Mar 2021 21:06:28 +0000 Message-ID: References: Mime-Version: 1.0 Content-Type: multipart/alternative; boundary="00000000000026385905be12536c" Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="24781"; mail-complaints-to="usenet@ciao.gmane.io" To: 47299@debbugs.gnu.org Original-X-From: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Sun Mar 21 22:07:11 2021 Return-path: Envelope-to: geb-bug-gnu-emacs@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1lO5Hz-0006K6-BM for geb-bug-gnu-emacs@m.gmane-mx.org; Sun, 21 Mar 2021 22:07:11 +0100 Original-Received: from localhost ([::1]:52186 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lO5Hy-0005E0-CJ for geb-bug-gnu-emacs@m.gmane-mx.org; Sun, 21 Mar 2021 17:07:10 -0400 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]:42566) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lO5Hq-0005Df-Ny for bug-gnu-emacs@gnu.org; Sun, 21 Mar 2021 17:07:02 -0400 Original-Received: from debbugs.gnu.org ([209.51.188.43]:44006) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1lO5Hq-0003yO-F1 for bug-gnu-emacs@gnu.org; Sun, 21 Mar 2021 17:07:02 -0400 Original-Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1lO5Hq-0001ef-6a for bug-gnu-emacs@gnu.org; Sun, 21 Mar 2021 17:07:02 -0400 X-Loop: help-debbugs@gnu.org In-Reply-To: Resent-From: Ioannis Kappas Original-Sender: "Debbugs-submit" Resent-CC: bug-gnu-emacs@gnu.org Resent-Date: Sun, 21 Mar 2021 21:07:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 47299 X-GNU-PR-Package: emacs Original-Received: via spool by 47299-submit@debbugs.gnu.org id=B47299.16163608076341 (code B ref 47299); Sun, 21 Mar 2021 21:07:02 +0000 Original-Received: (at 47299) by debbugs.gnu.org; 21 Mar 2021 21:06:47 +0000 Original-Received: from localhost ([127.0.0.1]:55552 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lO5Ha-0001eC-JC for submit@debbugs.gnu.org; Sun, 21 Mar 2021 17:06:47 -0400 Original-Received: from mail-oi1-f180.google.com ([209.85.167.180]:35345) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lO5HX-0001dy-4q for 47299@debbugs.gnu.org; Sun, 21 Mar 2021 17:06:45 -0400 Original-Received: by mail-oi1-f180.google.com with SMTP id x2so11035233oiv.2 for <47299@debbugs.gnu.org>; Sun, 21 Mar 2021 14:06:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to; bh=HlBYhqxZSJA7A8x6nYm9An5qcarF64QBeRe51I30aeM=; b=rk6udBrVHUtJ5WFharS5rb072wC7tX4qjI0OiXP97l7c0xvbGPfNiNX0REAw33sl0M KcS7DBX1OSVWj7pjUWxG/ydbB0/PdzwvH9qbThPBgPIe8pq/cbdtQB49s56BsyAgont1 8XWRcYC0rIFNeZEueKEz2Jkv3HTq0VThLn0MTgSFXWsBSSRSzflMJFIQCW9TrFKgMtGx vPAeHswtaIOgmWptP2zcgjL5HndeyTiv/r7nWL3Qmw4UXiP+M8UlRyJ2DApABH4XKoAR ND7gMc64AaHGCN6nKhnfE4QnyJIdakaae0mpGgnK4ic2cJcqGuKUYWGjiPFsnrO4G/AQ H4JQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=HlBYhqxZSJA7A8x6nYm9An5qcarF64QBeRe51I30aeM=; b=o+Oh5r21fvh2r2Dm3Zjv7D8EZ8Oy92jKgNHS3j8dBvPN3K7wRRRF9YfQPtXyEiefCO gu0aayA3Qx7nJq0Q+Aa6wE7zqJTM7g9p2DAF/65nI1gS6xZxsHl/WgIZ3xBZFoMmGk3K HDfVxw3R3j+DTTlP776ccf/6hmh7o549kXDVn6VjCtE9OZASihLbMfS90UijVNErTPK5 XWg0vM38pwECw05Dif89/7hnOsrdnBSk0uOO4JH+QW9Nb9AV/ijrP17f7mKgC3hz/ZGM AKIEjL0287MoWF3fpZCPBVl/NrC0geZ076UpwAiwknsgcpse02ovhEiAhvGsRD7LRuwo XUJw== X-Gm-Message-State: AOAM531qzsOSQOyI2Xx4sd/PJT6Ed+VPfz84lRajEjxIzxHm+wWUixNC yXy2fiUMWeWCYJvPT46bHZRvu433SI3WU2TaToK8oeO3fjM= X-Google-Smtp-Source: ABdhPJxGt/cDtnAZYMyjcSqUW0AlgbeDgMkz7H3wDEITPj6ZTCf6MX8LKPe8zXy0+PwHXeqAwqRZPvkmLuFmhdBQMxo= X-Received: by 2002:aca:4b03:: with SMTP id y3mr8269564oia.78.1616360797254; Sun, 21 Mar 2021 14:06:37 -0700 (PDT) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-gnu-emacs@gnu.org List-Id: "Bug reports for GNU Emacs, the Swiss army knife of text editors" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Original-Sender: "bug-gnu-emacs" Xref: news.gmane.io gmane.emacs.bugs:202808 Archived-At: --00000000000026385905be12536c Content-Type: text/plain; charset="UTF-8" Analysis follows. The `print' family of functions send their output to stdout by default when ran in emacs --batch mode. Similar to bug#46388 (27.1; emacs -batch does not output messages immediately when invoked outside of the command prompt) the issue stems from the expectation that stdout buffering should behave like in POSIX systems, which is not necessarily the case on 'windows-nt. The POSIX standard reads (https://pubs.opengroup.org/onlinepubs/9699919799/functions/stdin.html) under "stderr, stdin, stdout - standard I/O streams": """ ... At program start-up, three streams shall be predefined and need not be opened explicitly: standard input (for reading conventional input), standard output (for writing conventional output), and standard error (for writing diagnostic output). When opened, the standard error stream is not fully buffered; the standard input and standard output streams are fully buffered if and only if the stream can be determined not to refer to an interactive device. ... """ stdout on windows-nt is either always unbuffered (when attached to the console) or fully-buffered (otherwise), while on 'gnu/linux I have experimentally found it to be line-buffered when invoked from the terminal or from another program such as Emacs M-x shell. I consider this line-buffered behavior of 'gnu/linux to fall under the "interactive device" of the standard mentioned above. (When stdout is redirected to a file on 'gnu/linux, I found stdout to be fully-buffered having a 4096 buffer size). (See standard-streams-test report @ https://github.com/ikappaki/standard-streams-test for a tool that was written to investigate the buffering behavior of stderr on MS-Windows, i.e. unbuffered when attached to console, fully buffered otherwise. The same tool was modified to test stdout on a similar manner, which yielded exactly the same result). Thus the difference in behavior as described in the bug report is due to stdout on 'windows-nt being fully buffered, rather than being line buffered as in 'gnu/linux. As such, two likely fixes are presented below, one that flushes stdout on a newline only iff stdout is attached to a pipe (as if it is the "interactive device" of the POSIX standard) but staying fully buffered otherwise (e.g. when output was redirected to a pipe or a socket), while the other fix always flushing stdout on a newline (a much simpler and less involved solution -- similar to the one in bug#46388 suggested by Mr. Eggert -- though not optimized for redirections to files). (Please note that the intention below is to change src/print.c:printchar_to_stream(), which unless I missed something, is only called from the `print' family of functions when Emacs is in non-interactive mode). _flush on newline when pipe_: #+begin_src diff 5 files changed, 40 insertions(+) src/print.c | 2 ++ src/sysdep.c | 21 +++++++++++++++++++++ src/sysstdio.h | 1 + src/w32.c | 13 +++++++++++++ src/w32.h | 3 +++ modified src/print.c @@ -234,6 +234,8 @@ printchar_to_stream (unsigned int ch, FILE *stream) if (ASCII_CHAR_P (ch)) { putc (ch, stream); + if (ch == '\n') + maybe_flush_stdout(); #ifdef WINDOWSNT /* Send the output to a debugger (nothing happens if there isn't one). */ modified src/sysdep.c @@ -2821,6 +2821,27 @@ errwrite (void const *buf, ptrdiff_t nbuf) fwrite_unlocked (buf, 1, nbuf, errstream ()); } +/* On windows, stdout is unbuffered when attached to the console but + fully buffered (4096 bytes) when redirected to a pipe (this buffer + is complementary to the pipe buffer). + + Since Emacs --batch, at least on Windows, does not flush stdout it + means printing to standard output (for example with `princ' and its + variants) will not reach the parent process until at least this + process exits or the stream buffer is full, resulting to a very + poor interaction with the parent, contrary to how 'gnu/linux stdout works. + + We thus provide an interface to these functions to flush stdout + when has been redirected to a pipe. +*/ +void maybe_flush_stdout (void) +{ +#ifdef WINDOWSNT + if (is_stdout_pipe()) + fflush_unlocked(stdout); +#endif /* WINDOWSNT */ +} + /* Close standard output and standard error, reporting any write errors as best we can. This is intended for use with atexit. */ void modified src/sysstdio.h @@ -27,6 +27,7 @@ #define EMACS_SYSSTDIO_H #include "unlocked-io.h" extern FILE *emacs_fopen (char const *, char const *); +extern void maybe_flush_stdout (void); extern void errputc (int); extern void errwrite (void const *, ptrdiff_t); extern void close_output_streams (void); modified src/w32.c @@ -10190,6 +10190,12 @@ term_ntproc (int ignored) term_w32select (); } +static bool _is_stdout_pipe = false; +bool is_stdout_pipe (void) +{ + return _is_stdout_pipe; +} + void init_ntproc (int dumping) { @@ -10268,6 +10274,13 @@ init_ntproc (int dumping) _fdopen (2, "w"); } + HANDLE soh = (HANDLE)_get_osfhandle(_fileno(stdout)); + _is_stdout_pipe = + /* is pipe (anonymous, named or socket)*/ + GetFileType(soh) == FILE_TYPE_PIPE + /* and is definitely not a socket */ + && GetNamedPipeInfo(soh, NULL, NULL, NULL, NULL); + /* unfortunately, atexit depends on implementation of malloc */ /* atexit (term_ntproc); */ if (!dumping) modified src/w32.h @@ -170,6 +170,9 @@ #define FILE_SERIAL 0x0800 extern HANDLE maybe_load_unicows_dll (void); extern void globals_of_w32 (void); +/* return whether standard output is redirected to a pipe. */ +extern bool is_stdout_pipe (void); + extern void term_timers (void); extern void init_timers (void); #+end_src #+begin_src diff 1 file changed, 8 insertions(+) src/print.c | 8 ++++++++ modified src/print.c @@ -235,6 +235,14 @@ printchar_to_stream (unsigned int ch, FILE *stream) { putc (ch, stream); #ifdef WINDOWSNT + /* Flush stdout after outputting a newline since stdout is + fully buffered when redirected to a pipe, contrary to + POSIX when attached to an "interactive device" (see + "stderr, stdin, stdout - standard I/O streams" in IEEE + 1003.1-2017) */ + if (ch == '\n' && stream == stdout) + fflush_unlocked(stdout); + /* Send the output to a debugger (nothing happens if there isn't one). */ if (print_output_debug_flag && stream == stderr) #+end_src Feel free to modify/replace the above two as you find appropriate (especially the excessive comments). Thanks --00000000000026385905be12536c Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Analysis fo= llows.

The `print' family of functions send their output = to stdout by default
when ran in = emacs --batch mode.

Similar to bug#46388 (27.1; emacs -batch = does not output messages
immediat= ely when invoked outside of the command prompt) the issue
= stems from the expectation that stdout buffering s= hould behave like in
POSIX system= s, which is not necessarily the case on 'windows-nt.
<= font face=3D"monospace">
The = POSIX standard reads
under "stderr, stdin, stdout - s= tandard I/O streams":

"""
...
At= program start-up, three streams shall be predefined and need not be=
opened explicitly: standard input (for = reading conventional input),
stan= dard output (for writing conventional output), and standard error
(for writing diagnostic output). When open= ed, the standard error
stream is = not fully buffered; the standard input and standard output
streams are fully buffered if and only if the str= eam can be determined
not to refe= r to an interactive device.
...
"""

s= tdout on windows-nt is either always unbuffered (when attached to the
console) or fully-buffered (otherwise)= , while on 'gnu/linux I have
= experimentally found it to be line-buffered when invoked from the
terminal or from another program such as E= macs M-x=C2=A0shell. I conside= r=C2=A0
this line-bu= ffered behavior of 'gnu/linux to fall under the=C2=A0"interactive=C2=A0
device" of=C2=A0the standard mentioned above.

(When std= out is redirected to a file on 'gnu/linux, I found stdout to
be fully-buffered having a 4096 buffer size= ).

(See standard-streams-test report @
written to investigate the = buffering behavior of stderr on MS-Windows,
i.e. unbuffered when attached to console, fully buffered<= /div>
otherwise. The same tool was modified to= test stdout on a similar
manner,= which yielded exactly the same result).

Thus the difference = in behavior as described in the bug report is due
to stdout on 'windows-nt being fully buffered, rather = than being line
buffered as in &#= 39;gnu/linux.

As such, two likely fixes are presented below, = one that flushes stdout
on a newl= ine only iff stdout is attached to a pipe (as if it is the
"interactive device" of the POSIX stand= ard) but staying fully buffered
o= therwise (e.g. when output was redirected to a pipe or a socket), while
the other fix always flushing stdout= on a newline (a much simpler and less
involved solution -- similar to the one in bug#46388 suggested by Mr.= Eggert
-- though not optimized f= or redirections to files).

(Please note that the intention be= low is to change
src/print.c:prin= tchar_to_stream(), which unless I missed something, is
only called from the `print' family of functions = when Emacs is in
non-interactive = mode).

_flush on newline when pipe_:
#+begin_src diff
5 files changed, 40 insertions(+)
src/print.c=C2=A0 =C2=A0 |=C2=A0 2 ++
src/sysdep.c=C2=A0 =C2=A0| 21 +++++++++++++++++++++
src/sysstdio.h |=C2=A0 1 +
src/w32.c=C2=A0 =C2=A0 =C2=A0 | 13 +++++++++++++
src/w32.h=C2=A0 =C2=A0 =C2=A0 |=C2=A0 = 3 +++

modified=C2=A0 =C2=A0src/print.c
@@ -234,6 +234,8 @@ printchar_to_stream (unsigned int c= h, FILE *stream)
=C2=A0 =C2=A0 = =C2=A0 =C2=A0if (ASCII_CHAR_P (ch))
=C2=A0 {
=C2=A0 =C2=A0 putc= (ch, stream);

<= div>+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 if (ch =3D= =3D '\n')
+=C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 maybe_flush_stdout();
=C2=A0#ifdef WINDOWSNT
=C2=A0 =C2=A0 /* Send the out= put to a debugger (nothing happens if there
=C2=A0 =C2=A0 =C2=A0 =C2= =A0isn't one).=C2=A0 */
modif= ied=C2=A0 =C2=A0src/sysdep.c
@@ -= 2821,6 +2821,27 @@ errwrite (void const *buf, ptrdiff_t nbuf)
<= div>=C2=A0 =C2=A0fwrite_unlocked (buf, 1, nbuf, er= rstream ());
=C2=A0}
=
=C2=A0
+/* On windows, stdout is unbuffered when attached to the console but
+=C2=A0 =C2=A0fully buffered (4096= bytes) when redirected to a pipe (this buffer
+=C2=A0 =C2=A0is complementary to the pipe buffer).
+
+=C2=A0 =C2=A0Since Emacs --batch, at least on Windows, does not flush s= tdout it
+=C2=A0 =C2=A0means prin= ting to standard output (for example with `princ' and its
<= div>+=C2=A0 =C2=A0variants) will not reach the par= ent process until at least this
+= =C2=A0 =C2=A0process exits or the stream buffer is full, resulting to a ver= y
+=C2=A0 =C2=A0poor interaction = with the parent, contrary to how 'gnu/linux stdout works.
<= div>+
+= =C2=A0 =C2=A0We thus provide an interface to these functions to flush stdou= t
+=C2=A0 =C2=A0when has been red= irected to a pipe.
+*/
+void maybe_flush_stdout (void)
+{
+#ifdef WINDOWSNT
+=C2=A0 if (is= _stdout_pipe())
+=C2=A0 =C2=A0 ff= lush_unlocked(stdout);
+#endif /*= WINDOWSNT */
+}
+
=C2= =A0/* Close standard output and standard error, reporting any write<= /div>
=C2=A0 =C2=A0 errors as best we can.=C2= =A0 This is intended for use with atexit.=C2=A0 */
=C2=A0void
modi= fied=C2=A0 =C2=A0src/sysstdio.h
@= @ -27,6 +27,7 @@ #define EMACS_SYSSTDIO_H
=C2=A0#include "unlocked-io.h"
=C2=A0
=C2=A0ext= ern FILE *emacs_fopen (char const *, char const *);
+extern void maybe_flush_stdout (void);
=C2=A0extern void errputc (int);
=C2=A0extern void errwrite (void const *, ptrdiff= _t);
=C2=A0extern void close_outp= ut_streams (void);
modified=C2=A0= =C2=A0src/w32.c
@@ -10190,6 +101= 90,12 @@ term_ntproc (int ignored)
=C2=A0 =C2=A0term_w32select ();
=C2=A0}
=C2=A0
= +static bool _is_stdout_pipe =3D false;
+bool is_stdout_pipe (void)
+{
+= =C2=A0 return _is_stdout_pipe;
+}=
+
=C2=A0void
=C2=A0init= _ntproc (int dumping)
=C2=A0{
@@ -10268,6 +10274,13 @@ init_ntproc= (int dumping)
=C2=A0 =C2=A0 =C2= =A0_fdopen (2, "w");
= =C2=A0 =C2=A0}
=C2=A0
+=C2=A0 HANDLE soh =3D (HANDLE)_get_osfhandl= e(_fileno(stdout));
+=C2=A0 _is_s= tdout_pipe =3D
+=C2=A0 =C2=A0 /* = is pipe (anonymous, named or socket)*/
+=C2=A0 =C2=A0 GetFileType(soh) =3D=3D FILE_TYPE_PIPE
+=C2=A0 =C2=A0 /* and is definitely not a socket= */
+=C2=A0 =C2=A0 && Get= NamedPipeInfo(soh, NULL, NULL, NULL, NULL);
+
=C2=A0 =C2=A0/* unfo= rtunately, atexit depends on implementation of malloc */
<= font face=3D"monospace">=C2=A0 =C2=A0/* atexit (term_ntproc); */
=C2=A0 =C2=A0if (!dumping)
modified=C2=A0 =C2=A0src/w32.h
<= font face=3D"monospace">@@ -170,6 +170,9 @@ #define FILE_SERIAL=C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A00x0800
=C2=A0extern HANDLE maybe_load_unicows_dll (void);
=
=C2=A0extern void globals_of_w32 (void);
=C2=A0
+/* return whether standard output is redirected to a pipe. */<= /font>
+extern bool is_stdout_pipe (void= );
+
=C2=A0extern void term_timers (void);
=C2=A0extern void init_timers (void);
= #+end_src

#+begin_src diff
=C2=A0 1 file changed, 8 insertions(+)
=C2=A0 src/print.c | 8 ++++++++

=C2=A0 modified=C2=A0 =C2=A0src/print.c
=C2=A0 @@ -235,6 +235,14 @@ printchar_to_stream (unsigned = int ch, FILE *stream)
=C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 {
=C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 putc (ch, stream);
=C2=A0 =C2=A0#ifdef WINDOWSNT
=C2=A0 +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 /* Flush stdo= ut after outputting a newline since stdout is
=C2=A0 +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0full= y buffered when redirected to a pipe, contrary to
=C2=A0 +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0PO= SIX when attached to an "interactive device" (see
=C2=A0 +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0"stderr, stdin, stdout - standard I/O streams" in IEEE<= /font>
=C2=A0 +=C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A01003.1-2017) */
=C2=A0 +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 if (ch =3D=3D '\n'= ; && stream =3D=3D stdout)
=C2=A0 +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 fflush_unlocked(stdout)= ;
=C2=A0 +
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 /* Send the o= utput to a debugger (nothing happens if there
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0isn&#= 39;t one).=C2=A0 */
=C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 if (print_output_debug_flag && stream = =3D=3D stderr)

<= div>#+end_src

Feel free to modify/re= place the above two as you find appropriate
(especially the excessive comments).

Thanks=

--00000000000026385905be12536c--