bug#46388: 27.1; emacs -batch does not output messages immediately when invoked outside of the command prompt

[Ioannis Kappas <ioannis.kappas@gmail.com>]

Hi Eli,

thanks for taking the time to look into this bug report!

(my apologies for the long reply that follows)

I understand there are two concerns:

1. stderr could be connected to a file, in which case we do want it to
   be buffered for efficiency.
2. If stderr is forced to unbuffered, this will make it behave
   differently from posix systems.

I had a look around #2 in the posix standard. The section under
stderr, stdin, stdout at
https://pubs.opengroup.org/onlinepubs/9699919799/functions/stdin.html
reads:

"""At program start-up, three streams shall be predefined and need not
be opened explicitly: standard input (for reading conventional input),
standard output (for writing conventional output), and standard error
(for writing diagnostic output). When opened, the standard error
stream is not fully buffered; the standard input and standard output
streams are fully buffered if and only if the stream can be determined
not to refer to an interactive device.  """

Thus, at open (I assume this to mean at program invocation), the posix
spec suggests that stderr is not "fully buffered".

The section under setvbuf under the same spec at
https://pubs.opengroup.org/onlinepubs/009695399/functions/setvbuf.html
differentiates between the three different buffering regimes:


"""
 {_IOFBF} shall cause input/output to be fully buffered.

 {_IOLBF} shall cause input/output to be line buffered.

  {_IONBF} shall cause input/output to be unbuffered.
 """

So, if stderr is not "fully buffered", it can be either "line
buffered" or "unbuffered".

(This does make some sense to me, since stderr is usually for urgent
messages and thus any messages are needed to be readily available to
the recipients. Fully buffered defeats the purpose?).

How does Linux conforms to the posix standard? Having a look at
stderr(3) in the foot note, it reads:

""" Notes

The stream stderr is unbuffered. The stream stdout is line-buffered
when it points to a terminal. Partial lines will not appear until
fflush(3) or exit(3) is called, or a newline is printed. This can
produce unexpected results, especially with debugging output. The
buffering mode of the standard streams (or any other stream) can be
changed using the setbuf(3) or setvbuf(3) call. Note that in case
stdin is associated with a terminal, there may also be input buffering
in the terminal driver, entirely unrelated to stdio
buffering. (Indeed, normally terminal input is line buffered in the
kernel.) This kernel input handling can be modified using calls like
tcsetattr(3); see also stty(1), and termios(3).

"""

Which appears to suggests that stderr always starts as "unbuffered"
when a program starts, irrespective of how it was invoked?  If so,
then it conforms to the posix standard of not being "fully buffered"
at open, by always being "unbuffered" (this behavior has also been
seen in all tests mentioned so far). If this is the case, then the
desired buffering efficiency as mentioned in concern #1, does not
exist in Linux.

I have also conducted an additional test to check Linux behavior when
stderr is redirected to a file (using the same c program that I
mentioned in the original bug report):

: ./fwrite-a-character-to-stderr-and-sleep-for-two-seconds 2> io.txt

while at around the same time in another terminal I read from that file:

: cat io.txt

The result is that cat outputs "t", the character the fwrite program
has written to stderr, i.e. the io.txt file, proving that no buffering
has being utilized by the fwrite process. The result is the same when
I try the same experiment from inside an emacs shell.

Back to the windows behavior at hand, the simple test conducted with
fwrite as the bug report mentioned, has a 2048 bytes associated with
it which I will consider it to be categorized as "fully buffered",
which brings it further away to the posix standard and even more further
away from Linux being on the other end (always unbuffered).

(There is also this thread on stackoverlow quoting the following from
comp.lang.c:

"""

Unix convention is that stdin and stdout are line-buffered when
associated with a terminal, and fully-buffered (aka block-buffered)
otherwise. stderr is always unbuffered.

"""

emphasis on *Unix convention ... stderr .. always unbuffered*, though
it will be difficult to confirm this claim for all *nices)

Thus, if we assume the above interpretation given so far to be
correct, then both of
the concerns can be addressed as such:

For #1, we can use Linux as a benchmark of established
behavior. There is no efficiency in Linux with regards to buffering
when stderr is redirected to files, thus having emacs batch mode on
windows being
on par with Linux should be good enough in this respect.

For #2, emacs batch stderr behavior when invoked from outside the
command line is not compatible with the posix standard, since it uses
"fully buffered" mode, which actually makes it behave very differently
from Linux (which is posix compliant by always having stderr as being
unbuffered). Thus, the suggested patch actually addresses this
concern, rather than invalidating it.

(I understand ConPTY is the long term solution looking for a hacker,
though I still do
suppose the current behavior on windows is odd and needs to be
addressed until that time).

Sincerely





On Tue, Feb 9, 2021 at 5:36 AM Eli Zaretskii <eliz@gnu.org> wrote:
>
> > From: Ioannis Kappas <ioannis.kappas@gmail.com>
> > Date: Mon, 8 Feb 2021 21:42:09 +0000
> >
> >     A solution thus to correct this behavior in emacs -batch on
> >     windows-nt would be to check if it is connected to the console,
> >     and when not, set stderr mode to unbuffered.
>
> Thanks for the analysis and the patch proposal.  However, I don't
> think this is a good idea.  For starters, stderr could be connected to
> a file, in which case we do want it to be buffered.  More generally,
> stderr could be used for something other than outputting urgent
> messages, in which case making it unbuffered will make I/O less
> efficient for no good reason.  And this would make Emacs on Windows
> behave differently from Posix systems, which is also a downside.
>
> I think a much better way forward in this area is to teach Emacs to
> use the Pseudo Consoles introduced in recent Windows versions.  That
> would allow us to support subprocess communications via PTYs on
> MS-Windows, and thus will solve this and other similar issues.
>
> Patches to support PTYs on Windows are welcome.