bug#46388: 27.1; emacs -batch does not output messages immediately when invoked outside of the command prompt

[Ioannis Kappas <ioannis.kappas@gmail.com>]

On Wed, Feb 10, 2021 at 3:57 PM Eli Zaretskii <eliz@gnu.org> wrote:

> >
> >     | # | System     | emacs -batch invoked from | MESSAGE behavior
> >
> >         |
> >     |---+------------+---------------------------+----------------------------------------------------------------------------------------------------|
> >     | 1 | Linux      | bash                      | any MESSAGE is
> > immediately printed, i.e. stderr is unbuffered
> >               |
> >     | 2 | Linux      | emacs eshell/shell etc    | >>
>
> Did you try this 2nd item when the connection type for the subprocess
> is 'pipe'?  Because otherwise we are comparing apples to oranges.

I haven't, how do I do this? I was merely describing the plain user
experience of running emacs -batch from a shell.

for case #2 for example:

1. On Linux, open emacs
2.  M-x eshell
3. at the eshell prompt type the following command:
3.1 emacs -Q --batch --eval="(progn (message \"hi\") (sit-for 5))"
4. observe the result
4.1 "hi" is printed immediately, emacs -batch exits after five seconds.

In my tests, the second command was ran in all five cases, without applying any
other configuration to the parent command prompt/shell/eshell process.
Apologies if this was not clear.

> >     I think I've just realized what you were saying from the
> >     beginning. That the difference in behavior is expected, since
> >     it is the parent process which decides the buffering regime to be
> >     used for the subprocess. Thus in #5, it is emacs on windows that
> >     decided to invoke emacs -batch as a subprocess using pipes, which
> >     has resulted in emacs -batch's stderr to be buffered.
>
> That is correct.  As we don't support PTYs on Windows, we can only use
> pipes for communicating with subprocesses there.
>
> Btw, did you try to play with the value of w32-pipe-buffer-size,
> e.g. setting it to a small value?

No, I haven't experimented with pipes yet, my next plan is to study
how subprocess
are invoked from within emacs and mintty (since both demonstrate the
same behavior).

> >     If the current behavior is indeed the correct expected behavior, how do I
> >     flush text message to stderr (or even stdout) from an emacs
> >     -batch script/eval?
>
> My reading of the code is that we already fflush stderr after emitting
> a message, so this should already happen.  See message_to_stderr.  If
> that still doesn't help, then there's some buffering in the OS (for
> example, in the pipe machinery itself), which we cannot control.

the xdisp.c:message_to_stderr() is the first function i studied with
gdb when I started the investigation. Unless I've missed something,
it does not seem to lead to calling fflush (under windows at least):

1. it will sysdep.c:errwrite() the message
1.1 errwrite() will call sysdep.c:errstream() to get stderr
1.1.1 errstream() *may* call fflush on stderr in some system (via
buferr static variable), but this does not happen under windows-nt at
least.
1.2 will call fwrite to write the message
2. it may call sysdep.c:errputc() to to append a newline, following
exactly the same logic as for errwrite(), but using fputc() instead

/* Log the message M to stderr.  Log an empty line if M is not a string.  */

static void
message_to_stderr (Lisp_Object m)
{
  if (noninteractive_need_newline)
    {
      noninteractive_need_newline = false;
      errputc ('\n');
    }
  if (STRINGP (m))
    {
      Lisp_Object coding_system = Vlocale_coding_system;
      Lisp_Object s;

      if (!NILP (Vcoding_system_for_write))
coding_system = Vcoding_system_for_write;
      if (!NILP (coding_system))
s = code_convert_string_norecord (m, coding_system, true);
      else
s = m;

      errwrite (SDATA (s), SBYTES (s));
    }
  if (STRINGP (m) || !cursor_in_echo_area)
    errputc ('\n');
}


void
errputc (int c)
{
  fputc_unlocked (c, errstream ());
}

void
errwrite (void const *buf, ptrdiff_t nbuf)
{
  fwrite_unlocked (buf, 1, nbuf, errstream ());
}

/* Return the error output stream.  */
static FILE *
errstream (void)
{
  FILE *err = buferr;
  if (!err)
    return stderr;
  fflush_unlocked (stderr);
  return err;
}

in sysdep.c:init_standard_fds()
...
  /* Set buferr if possible on platforms defining _PC_PIPE_BUF, as
     they support the notion of atomic writes to pipes.  */
  #ifdef _PC_PIPE_BUF
    buferr = fdopen (STDERR_FILENO, "w");
    if (buferr)
      setvbuf (buferr, NULL, _IOLBF, 0);
  #endif
...



> There were some changes in this area lately (that's the discussion
> from 2019 I mentioned before): we now try to make a buffered variant
> of stderr, and use that for error messages.  The reason, in a
> nutshell, is that when you build Emacs with "make -jN", several copies
> of the Emacs process can work in parallel, so it was deemed better to
> have their messages be emitted atomically, instead of being
> interspersed with one another, which produces an illegible mess.
> However, that change makes a line-buffered variant of stderr, and on
> Windows line buffering is the same as full buffering.  So maybe we
> would need more changes in that area, for example some variable to
> control this behavior instead of making it unconditional.

Noted, thanks again!