`message' not outputting the newline "atomically"

`message' not outputting the newline "atomically"

[Lars Ingebrigtsen]

When lines are output as a unit (and we're below OS-dependent block
sizes, I guess), we're guaranteed that two processes outputting lines
won't have them stomp on each other.

However, with make -j8:

  INFO     Scraping files for autoloads... 
  INFO     Scraping files for autoloads...20%   GEN      ../../lisp/cedet/semantic/wisent/javat-wy.el

  GEN      ../../lisp/cedet/semantic/wisent/js-wy.el

And it always seems like it's the newline that's output separately?

These messages are output with `message', I think?  Is this something
that could be fixed?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: `message' not outputting the newline "atomically"

[Andreas Schwab]

On Jun 19 2019, Lars Ingebrigtsen <larsi@gnus.org> wrote:

> When lines are output as a unit (and we're below OS-dependent block
> sizes, I guess), we're guaranteed that two processes outputting lines
> won't have them stomp on each other.

Are we?

Andreas.

-- 
Andreas Schwab, SUSE Labs, schwab@suse.de
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

Re: `message' not outputting the newline "atomically"

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Date: Wed, 19 Jun 2019 16:12:47 +0200
> 
> When lines are output as a unit (and we're below OS-dependent block
> sizes, I guess), we're guaranteed that two processes outputting lines
> won't have them stomp on each other.

I agree with Andreas: we aren't.  It's just sheer luck if this happens
(probably because the time window for mixing output is small).

>   INFO     Scraping files for autoloads... 
>   INFO     Scraping files for autoloads...20%   GEN      ../../lisp/cedet/semantic/wisent/javat-wy.el
> 
>   GEN      ../../lisp/cedet/semantic/wisent/js-wy.el

If you are annoyed by this (I am not), use the -O option of Make (and
make sure you have a recent enough version of it to support that).
Caveat: doing that might slow down the build.

Re: `message' not outputting the newline "atomically"

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> I agree with Andreas: we aren't.  It's just sheer luck if this happens
> (probably because the time window for mixing output is small).

No, we are on all Unix-like OS-es I'm familiar with, and certainly on
GNU/Linux: If you output data that's less than PIPE_BUF in length, a
write(2) call will never interleave the data.

If you're using internal buffering of some kind, then you can get all
kinds of funny interleaving, but this is the reason that the output from
Makefiles never step on each others' lines, no matter how many gccs you
run in parallel.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: `message' not outputting the newline "atomically"

[Andreas Schwab]

On Jun 19 2019, Lars Ingebrigtsen <larsi@gnus.org> wrote:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>> I agree with Andreas: we aren't.  It's just sheer luck if this happens
>> (probably because the time window for mixing output is small).
>
> No, we are on all Unix-like OS-es I'm familiar with, and certainly on
> GNU/Linux: If you output data that's less than PIPE_BUF in length, a
> write(2) call will never interleave the data.

Noninteractively, message goes to stderr, which is unbuffered by
default.

Andreas.

-- 
Andreas Schwab, SUSE Labs, schwab@suse.de
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

Re: `message' not outputting the newline "atomically"

[Paul Eggert]

On 6/19/19 9:05 AM, Andreas Schwab wrote:
> Noninteractively, message goes to stderr, which is unbuffered by
> default.

Yes, POSIX says stderr defaults to being not fully buffered, which means 
it might be line-buffered or completely unbuffered or something else 
(glibc is "something else"). This should be fixable, though; I'll see if 
I can come up with something.

Re: `message' not outputting the newline "atomically"

[Eli Zaretskii]

> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Wed, 19 Jun 2019 16:22:32 -0700
> Cc: Eli Zaretskii <eliz@gnu.org>, emacs-devel@gnu.org
> 
> On 6/19/19 9:05 AM, Andreas Schwab wrote:
> > Noninteractively, message goes to stderr, which is unbuffered by
> > default.
> 
> Yes, POSIX says stderr defaults to being not fully buffered, which means 
> it might be line-buffered or completely unbuffered or something else 
> (glibc is "something else"). This should be fixable, though; I'll see if 
> I can come up with something.

What exactly is the problem you would like to fix here?  I don't see
any problem yet.

Re: `message' not outputting the newline "atomically"

[Paul Eggert]

[-- Attachment #1: Type: text/plain, Size: 856 bytes --]

Eli Zaretskii wrote:
> What exactly is the problem you would like to fix here?

$ strace -o /tmp/tr emacs -Q -batch -eval '(message "hello")'
hello
$ grep '^write(2,' /tmp/tr
write(2, "hello", 5)                    = 5
write(2, "\n", 1)                       = 1

There should be just one 'write' system call, not two. Because of this problem, 
if two Emacs processes output the lines "foo" and "bar" to the same file 
descriptor at nearly the same time, the output might be "foobar\n\n" or 
"barfoo\n\n" rather than the more-desirable "foo\nbar\n" or "bar\nfoo\n". This 
can cause the annoying symptoms that Lars reported in 
<https://lists.gnu.org/r/emacs-devel/2019-06/msg00710.html>.

I fixed it on GNU and similar platforms by installing the attached patch into 
Emacs master. I fixed some other minor stdio glitches while I was in the 
neighborhood.

[-- Attachment #2: 0001-Line-buffer-stderr.patch --]
[-- Type: text/x-patch, Size: 901 bytes --]

From 2079e40a3cffbfacc79725c8788d56d05f232222 Mon Sep 17 00:00:00 2001
From: Paul Eggert <eggert@cs.ucla.edu>
Date: Thu, 20 Jun 2019 00:32:17 -0700
Subject: [PATCH] Line-buffer stderr

* src/sysdep.c (init_standard_fds) [!DOS_NT]:
Use setvbuf to ensure stderr is line-buffered.
---
 src/sysdep.c | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/src/sysdep.c b/src/sysdep.c
index bc88e70dcb..3396764d5d 100644
--- a/src/sysdep.c
+++ b/src/sysdep.c
@@ -242,6 +242,12 @@ init_standard_fds (void)
   force_open (STDIN_FILENO, O_WRONLY);
   force_open (STDOUT_FILENO, O_RDONLY);
   force_open (STDERR_FILENO, O_RDONLY);
+
+  /* Line-buffer stderr.  However, leave stderr unbuffered on
+     MS-Windows, where setvbuf treats _IOLBF like _IOFBF.  */
+#ifndef DOS_NT
+  setvbuf (stderr, NULL, _IOLBF, 0);
+#endif
 }
 
 /* Return the current working directory.  The result should be freed
-- 
2.17.1

Re: `message' not outputting the newline "atomically"

[Lars Ingebrigtsen]

Paul Eggert <eggert@cs.ucla.edu> writes:

> I fixed it on GNU and similar platforms by installing the attached
> patch into Emacs master. I fixed some other minor stdio glitches while
> I was in the neighborhood.

Great; thanks!

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: `message' not outputting the newline "atomically"

[Eli Zaretskii]

> Cc: schwab@suse.de, larsi@gnus.org, emacs-devel@gnu.org
> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Thu, 20 Jun 2019 00:47:05 -0700
> 
> I fixed it on GNU and similar platforms by installing the attached patch into 
> Emacs master. I fixed some other minor stdio glitches while I was in the 
> neighborhood.

I don't think I like this change.  stderr is unbuffered for a reason,
and that reason is relevant for Emacs as well: important messages,
especially about errors and problems should not be lost because Emacs
crashes at an un-opportune moment.

The original issue is at worst a minor inconvenience, whereas the
"solution" adversely affects much more serious use cases.  So on
balance I don't think we should install this change.

As I wrote earlier, people who want to avoid mixing messages in
parallel builds can use the GNU Make's -O option.  That does the job
without impacting the rest of Emacs in any way.

Re: `message' not outputting the newline "atomically"

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> As I wrote earlier, people who want to avoid mixing messages in
> parallel builds can use the GNU Make's -O option.  That does the job
> without impacting the rest of Emacs in any way.

This behaviour isn't limited to makefiles -- anybody who uses Emacs to
output data (from scripts and the like) will possibly see these
intermittent glitches.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: `message' not outputting the newline "atomically"

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: Paul Eggert <eggert@cs.ucla.edu>,  schwab@suse.de,  emacs-devel@gnu.org
> Date: Thu, 20 Jun 2019 14:55:48 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > As I wrote earlier, people who want to avoid mixing messages in
> > parallel builds can use the GNU Make's -O option.  That does the job
> > without impacting the rest of Emacs in any way.
> 
> This behaviour isn't limited to makefiles -- anybody who uses Emacs to
> output data (from scripts and the like) will possibly see these
> intermittent glitches.

Only if more than one instance of Emacs is running in parallel, right?
Which is somewhat unusual for scripts.

If someone wants to implement an optional feature that will have the
same effect, I'm sure it would be a welcome addition.  But forcing
this behavior on all of Emacs is IMO not a good idea.

Re: `message' not outputting the newline "atomically"

[Andreas Schwab]

On Jun 20 2019, Lars Ingebrigtsen <larsi@gnus.org> wrote:

> This behaviour isn't limited to makefiles -- anybody who uses Emacs to
> output data (from scripts and the like) will possibly see these
> intermittent glitches.

Anybody who uses Emacs to output data doesn't use message.

Andreas.

-- 
Andreas Schwab, SUSE Labs, schwab@suse.de
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

Re: `message' not outputting the newline "atomically"

[Paul Eggert]

On 6/20/19 5:52 AM, Eli Zaretskii wrote:
> stderr is unbuffered for a reason,

No: POSIX allows stderr to be line-buffered, or to use any other kind of 
buffering that is not fully buffered, and many platforms do just that 
instead of having stderr be unbuffered (which is quite inefficient on 
some platforms). It's true that stderr is unbuffered on MS-Windows, but 
the change does not affect MS-Windows behavior so there should be no 
problem there.

All this change is doing (on non-MS-Windows platforms) is to use line 
buffering on stderr, which is fine for Emacs as it's what Emacs does 
already on some platforms.

Re: `message' not outputting the newline "atomically"

[Eli Zaretskii]

> Cc: schwab@suse.de, larsi@gnus.org, emacs-devel@gnu.org
> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Thu, 20 Jun 2019 09:26:48 -0700
> 
> On 6/20/19 5:52 AM, Eli Zaretskii wrote:
> > stderr is unbuffered for a reason,
> 
> No: POSIX allows stderr to be line-buffered

Allows, yes.  But by default it's unbuffered, right?

> All this change is doing (on non-MS-Windows platforms) is to use line 
> buffering on stderr, which is fine for Emacs as it's what Emacs does 
> already on some platforms.

It's a bad change.  I'm unhappy about it.  OK?

Re: `message' not outputting the newline "atomically"

[Paul Eggert]

On 6/20/19 9:45 AM, Eli Zaretskii wrote:
>>
>> No: POSIX allows stderr to be line-buffered
> Allows, yes.  But by default it's unbuffered, right?

No. POSIX allows the default to be line buffered (or to be buffered in 
other ways, so long as it's not "fully buffered"), and some platforms do 
that. (POSIX also allows the default to be unbuffered, but it doesn't 
require this.)

We really don't want stderr to be unbuffered on GNU/Linux and similar 
platforms; that would slow down execution signficantly.

Re: `message' not outputting the newline "atomically"

[Eli Zaretskii]

> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Thu, 20 Jun 2019 10:41:34 -0700
> Cc: schwab@suse.de, larsi@gnus.org, emacs-devel@gnu.org
> >>
> >> No: POSIX allows stderr to be line-buffered
> > Allows, yes.  But by default it's unbuffered, right?
> 
> No. POSIX allows the default to be line buffered (or to be buffered in 
> other ways, so long as it's not "fully buffered"), and some platforms do 
> that.

But GNU/Linux does start with it unbuffered, or else the code to make
it line-buffered wouldn't be needed, right?

> We really don't want stderr to be unbuffered on GNU/Linux and similar 
> platforms; that would slow down execution signficantly.

I think we do want it to be unbuffered, so that important error
messages are output in their entirety, or as close as possible.  the
main use of stderr is to report errors, not output informative
messages.

Re: `message' not outputting the newline "atomically"

[Paul Eggert]

On 6/20/19 11:06 AM, Eli Zaretskii wrote:
> But GNU/Linux does start with it unbuffered, or else the code to make
> it line-buffered wouldn't be needed, right?

Yes, and it turns out that I was misinformed about what "unbuffered" 
means in GNU/Linux. I thought "unbuffered" (_IONBF) meant completely 
unbuffered, i.e., that individual 'write' system calls need to be done 
for each byte of output. But although POSIX allows this, it does not 
require it; all that's required of _IONBF is that writes must be done 
"as soon as possible", a term POSIX does not define. GCC+glibc takes "as 
soon as possible" to mean that by the time that (say) printf has 
successfully returned, a sequence of 'write' system calls has been done 
that implement the printf output; this is typically more efficient than 
completely unbuffered, since it means (for example) that printf ("x=%d", 
42) can be implemented via write (1, "x=42", 4) rather than via four 
one-byte 'write' calls.

In contrast, GCC+glibc takes line buffering (_IOLBF) to mean that a 
'write' system call is needed whenever printf etc. outputs '\n' (or 
whenever the output buffer is full of course).

POSIX allows stderr to default to any kind of buffering other than 
_IOFBF (fully buffered). This is similar to POSIX's treatment of the 
default buffering for stdin and stdout when they are interactive devices 
(noninteractive stdin/stdout must be fully buffered).

In GCC+glibc, neither _IOLBF  nor _IONBF strictly dominates the other in 
terms of buffering. That is, there are examples like where _IONBF 
buffers more than _IOLBF, and vice versa. It is counterintuitive that 
unbuffered output can buffer more than line-buffered output does, but 
there it is.


> I think we do want it to be unbuffered, so that important error
> messages are output in their entirety, or as close as possible.
Every important Emacs stderr message ends in newline, so there's no 
significant practical difference between _IOLBF and _IONBF on that 
score. And _IOLBF has a practical advantage: it causes stderr output to 
interleave better with the output of other processes. The problem of 
interleaving has bugged me for a while (long before Lars pointed it 
out). In some cases the interleaved output can be pretty hard to 
decipher. Although _IOLBF does not fix all the problems with interleaved 
output from Emacs, it should fix most of the problems I run into.

Re: `message' not outputting the newline "atomically"

[Eli Zaretskii]

> Cc: schwab@suse.de, larsi@gnus.org, emacs-devel@gnu.org
> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Thu, 20 Jun 2019 12:33:13 -0700
> 
> > I think we do want it to be unbuffered, so that important error
> > messages are output in their entirety, or as close as possible.
> Every important Emacs stderr message ends in newline

So now we will have to enforce this all over the place, in order to
ensure that critical messages get through?  Just to placate the
situation with interleaving output from two or more instances of
Emacs?

Like I said, on balance this is a bad change.  It may or may not make
a significant difference on GNU/Linux, but there are other systems out
there.  And even on GNU/Linux, we have no control on what the glibc
developers may decide about this tomorrow, given the lax Posix
requirements.

Re: `message' not outputting the newline "atomically"

[Eli Zaretskii]

> Date: Fri, 21 Jun 2019 08:46:51 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: schwab@suse.de, larsi@gnus.org, emacs-devel@gnu.org
> 
> Like I said, on balance this is a bad change.  It may or may not make
> a significant difference on GNU/Linux, but there are other systems out
> there.  And even on GNU/Linux, we have no control on what the glibc
> developers may decide about this tomorrow, given the lax Posix
> requirements.

One more reason for this to be a bad idea is that people don't expect
stderr to be buffered, and might write code under the assumption that
it isn't.

Re: `message' not outputting the newline "atomically"

[Paul Eggert]

On 6/20/19 10:46 PM, Eli Zaretskii wrote:
>> Every important Emacs stderr message ends in newline
> So now we will have to enforce this all over the place

It's already enforced, in the sense that (a) it's standard practice 
anyway, (b) a stderr diagnostic without a trailing newline is a bug 
since it'll mess up formatting for the next stderr diagnostic, and (c) 
POSIX allows platforms to line-buffer stderr by default and Emacs has 
been working on such platforms for ages anyway.

If it helps, I can audit all uses of stderr in the Emacs source to see 
if any newlines are being omitted. Any such omissions are buggy anyway 
for reasons (a), (b), and (c), and this is true regardless of whether 
Emacs explicitly enables line-buffering for stderr. I would not be 
surprised to find one or two bugs in the hundreds of diagnostics that 
Emacs generates, but now is as good a time as any to fix any bugs that I 
find.


> people don't expect stderr to be buffered
Actually I expected stderr to be line-buffered, a behavior that POSIX 
allows and that I vaguely recall when running GNU code (sorry, don't 
remember what platforms). Evidently things are different in the milieu 
of MS-Windows as it does not support line-buffering; still, Emacs and 
other GNU programs have worked with line-buffered stderr for ages.

Re: `message' not outputting the newline "atomically"

[Eli Zaretskii]

> Cc: schwab@suse.de, larsi@gnus.org, emacs-devel@gnu.org
> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Fri, 21 Jun 2019 17:20:56 -0700
> 
> On 6/20/19 10:46 PM, Eli Zaretskii wrote:
> >> Every important Emacs stderr message ends in newline
> > So now we will have to enforce this all over the place
> 
> It's already enforced, in the sense that (a) it's standard practice 
> anyway, (b) a stderr diagnostic without a trailing newline is a bug 
> since it'll mess up formatting for the next stderr diagnostic, and (c) 
> POSIX allows platforms to line-buffer stderr by default and Emacs has 
> been working on such platforms for ages anyway.

It isn't a bug, because an error message could legitimately be output
piecemeal.

> If it helps, I can audit all uses of stderr in the Emacs source

You will audit the sources from now to eternity?  I don't think
auditing is a valid solution for such problems.

> > people don't expect stderr to be buffered
> Actually I expected stderr to be line-buffered

It's possible that you are an exception.

Re: `message' not outputting the newline "atomically"

[Paul Eggert]

Eli Zaretskii wrote:
> an error message could legitimately be output piecemeal.

All such error messages in Emacs are output with a trailing newline right away, 
so there is no significant problem here. The trailing newline is standard 
practice, both in Emacs and elsewhere.  For example, consider the following code 
(taken from src/pdumper.c):

   static void
   dump_fingerprint (const char *label, unsigned char const *xfingerprint)
   {
     fprintf (stderr, "%s: ", label);
     for (int i = 0; i < 32; ++i)
       fprintf (stderr, "%02x", (unsigned) xfingerprint[i]);
     fprintf (stderr, "\n");
   }

Here, GNU/Linux Emacs would issue 34 'write' system calls if stderr were using 
_IONBF (and the output line could be interleaved with other output lines, 
possibly causing confusion), whereas GNU/Linux Emacs issues just one 'write' 
system call now that stderr uses _IOLBF (thus avoiding the confusion).

I don't see any significant problem with line-buffering here, or anywhere else 
that Emacs outputs to stderr. And I've looked at quite a few uses.

> You will audit the sources from now to eternity?

An audit is not necessary. I volunteered to audit only to allay any concerns 
with my assertion that this change is OK. If these concerns cannot be allayed by 
any such evidence, then of course an audit would be a waste of time.

Re: `message' not outputting the newline "atomically"

[Eli Zaretskii]

> Cc: schwab@suse.de, larsi@gnus.org, emacs-devel@gnu.org
> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Sat, 22 Jun 2019 12:14:55 -0700
> 
>    static void
>    dump_fingerprint (const char *label, unsigned char const *xfingerprint)
>    {
>      fprintf (stderr, "%s: ", label);
>      for (int i = 0; i < 32; ++i)
>        fprintf (stderr, "%02x", (unsigned) xfingerprint[i]);
>      fprintf (stderr, "\n");
>    }
> 
> Here, GNU/Linux Emacs would issue 34 'write' system calls if stderr were using 
> _IONBF (and the output line could be interleaved with other output lines, 
> possibly causing confusion), whereas GNU/Linux Emacs issues just one 'write' 
> system call now that stderr uses _IOLBF (thus avoiding the confusion).
> 
> I don't see any significant problem with line-buffering here, or anywhere else 
> that Emacs outputs to stderr. And I've looked at quite a few uses.

What if Emacs crashes half way through that loop, because 'i' is
computed incorrectly?  With line buffering, we risk not seeing
anything.

OTOH, the number of system calls is not something we should be
bothered with.

So this looks like a net loss to me.

Re: `message' not outputting the newline "atomically"

[Paul Eggert]

Eli Zaretskii wrote:
> What if Emacs crashes half way through that loop

What, due to a hardware error or a buffer overrun or something like that? In 
that case all bets are off: Emacs could output nonsense or truncated output 
regardless of whether it uses one system call or a hundred.

Besides, if we were allowed to hypothesize nonexistent bugs to argue against a 
change to Emacs, then we could easily argue against any change whatsoever. When 
thinking about changes to Emacs, it's more productive to consider real problems 
than imaginary ones.

> the number of system calls is not something we should be
> bothered with

If by "number of system calls" you're referring to performance, then I agree 
that stderr output typically is not a significant performance problem (this 
particular output certainly isn't). However, the motivation for line-buffering 
stderr is correctness, not performance.

Line-buffered stderr does a better job of interleaving diagnostics from parallel 
instances of Emacs, and this is what prompted the fix in question. The 
poorly-interleaved diagnostics have been a practical problem that has bugged me 
for a while, and bugs Lars and I assume others. In contrast, unbuffered stderr's 
only advantages mentioned so far have been theoretical.

Re: `message' not outputting the newline "atomically"

[Lars Ingebrigtsen]

Paul Eggert <eggert@cs.ucla.edu> writes:

> Line-buffered stderr does a better job of interleaving diagnostics
> from parallel instances of Emacs, and this is what prompted the fix in
> question. The poorly-interleaved diagnostics have been a practical
> problem that has bugged me for a while, and bugs Lars and I assume
> others.

Yeah, I'm using Emacses in batch mode extensively for doing various
tasks (even triggering from remotes to control the lighting in my apt),
and I've seen messed-up output in these places before, so I think it's
very nice that it's fixed.

> In contrast, unbuffered stderr's only advantages mentioned so
> far have been theoretical.

I do share Eli's worries here that there are some cases where having
stderr being line-buffered would lead to unforeseen situations, and
altering `message' to add the newline in the string being output would
be a safer solution.

But you said that that required more invasive rewriting of the `message'
machinery, so I think it makes sense at least to try the current
approach for a few months and see whether anything untoward is
uncovered...

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: `message' not outputting the newline "atomically"

[Eli Zaretskii]

> Cc: schwab@suse.de, larsi@gnus.org, emacs-devel@gnu.org
> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Sun, 23 Jun 2019 01:34:05 -0700
> 
> Eli Zaretskii wrote:
> > What if Emacs crashes half way through that loop
> 
> What, due to a hardware error or a buffer overrun or something like that? In 
> that case all bets are off: Emacs could output nonsense or truncated output 
> regardless of whether it uses one system call or a hundred.

Diagnostic output is precious precisely when "all bets are off".  For
example, knowing that the loop did N iterations and died on (N+1)st or
later can give a very important clue about the cause of the crash.

> Besides, if we were allowed to hypothesize nonexistent bugs to argue against a 
> change to Emacs, then we could easily argue against any change whatsoever. When 
> thinking about changes to Emacs, it's more productive to consider real problems 
> than imaginary ones.

Bugs aren't hypotheses, they actually happen.  And stderr is one
facility to diagnose abnormal behavior.  Changing the buffering mode
of a diagnostic stream globally because of a specific use case is
therefore a bad idea.

> Line-buffered stderr does a better job of interleaving diagnostics from parallel 
> instances of Emacs, and this is what prompted the fix in question.

Once again, the primary purpose of stderr is to produce diagnostic
error messages, so the producing nicer output is much less important
than producing as much of the output as possible.

> The poorly-interleaved diagnostics have been a practical problem
> that has bugged me for a while, and bugs Lars and I assume
> others.

That use case could have a more specialized solution, such as explicit
control of buffering from Lisp or a variant of 'message' that writes
to a different stream.  There's no need to juggernaut a
general-purpose facility for the benefit of a specific use case, and
not a very important one at that.

Re: `message' not outputting the newline "atomically"

[Paul Eggert]

[-- Attachment #1: Type: text/plain, Size: 1584 bytes --]

Eli Zaretskii wrote:

> Bugs aren't hypotheses, they actually happen.

I've looked carefully for any of the bugs hypothesized so far, and haven't found 
any. That's not to say they don't exist, but these mythical creatures are so 
rare that in practice it's fair to assert that they are less important than the 
practical bugs that this patch fixed.

If we could use hypothetical bugs as part of the argument, I could easily 
hypothesize bugs where unbuffered stderr produces less-useful output than 
line-buffered stderr does. But that would be going down a rabbit hole. We should 
focus on bugs that actually occur rather than implausible bugs hypothesized for 
the sake of argument.

> the primary purpose of stderr is to produce diagnostic
> error messages

And if these diagnostics are confusing or unintelligible, as with unbuffered 
stderr, this purpose is not being fulfilled.

> That use case could have a more specialized solution

Sure: we could take every bit of Emacs code that produces piecemeal stderr 
output, and rewrite it to generate the output into a temporary memory buffer and 
then output that buffer manually. For example, we could install something like 
the attached patch to address the dump_fingerprint issue. But this sort of 
change would be counterproductive: it would complicate the code and it would not 
make diagnostic output any more reliable (on the contrary - just look at all 
those tricky buffer-size calculations). In contrast, using _IOLBF is a one-line 
fix that fixes the problem without introducing all this complexity and 
unreliability.

[-- Attachment #2: handbuffer.diff --]
[-- Type: text/x-patch, Size: 957 bytes --]

diff --git a/src/pdumper.c b/src/pdumper.c
index c00f8a0af5..c4d48cc575 100644
--- a/src/pdumper.c
+++ b/src/pdumper.c
@@ -326,10 +326,16 @@ dump_reloc_set_offset (struct dump_reloc *reloc, dump_off offset)
 static void
 dump_fingerprint (const char *label, unsigned char const *xfingerprint)
 {
-  fprintf (stderr, "%s: ", label);
+  /* The maximum length of LABEL.  If you change code that calls this
+     function you may need to change this value.  */
+  enum { max_label_length = sizeof "desired fingerprint" - 1 };
+
+  char buf[max_label_length + sizeof ": \n" + 2 * 32];
+  int buflen = sprintf (buf, "%s: ", label);
   for (int i = 0; i < 32; ++i)
-    fprintf (stderr, "%02x", (unsigned) xfingerprint[i]);
-  fprintf (stderr, "\n");
+    buflen += sprintf (buf + buflen, "%02x", (unsigned) xfingerprint[i]);
+  buf[buflen++] = '\n';
+  fwrite (buf, 1, buflen, stderr);
 }
 
 /* Format of an Emacs portable dump file.  All offsets are relative to

Re: `message' not outputting the newline "atomically"

[Eli Zaretskii]

> Cc: schwab@suse.de, larsi@gnus.org, emacs-devel@gnu.org
> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Sun, 23 Jun 2019 10:32:48 -0700
> 
> > the primary purpose of stderr is to produce diagnostic
> > error messages
> 
> And if these diagnostics are confusing or unintelligible, as with unbuffered 
> stderr, this purpose is not being fulfilled.

I see no such confusing messages in all the examples brought so far.

> > That use case could have a more specialized solution
> 
> Sure: we could take every bit of Emacs code that produces piecemeal stderr 
> output, and rewrite it

I pointed out some much less painful solutions.  So this is a straw
man, really.

Would you please just revert that change?  Please?  Because I ask, if
for no other reason?

Re: `message' not outputting the newline "atomically"

[Stefan Monnier]

> What if Emacs crashes half way through that loop, because 'i' is
> computed incorrectly?  With line buffering, we risk not seeing
> anything.

Seeing none of the 32 bytes of fingerprint instead of only seeing the
first N of them seems like a very minor inconvenient.

Buffering can indeed be harmful, but only when significant
computation/time can pass between the `fprintf` and the moment the
corresponding text is "displayed", which is not the case here (nor
anywhere else in Emacs's C code, according to my quick "grep stderr").

> So this looks like a net loss to me.

I don't have a strong opinion on this, but FWIW, I see a net gain and no
significant loss.


        Stefan

Re: `message' not outputting the newline "atomically"

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Sun, 23 Jun 2019 08:53:48 -0400
> 
> > What if Emacs crashes half way through that loop, because 'i' is
> > computed incorrectly?  With line buffering, we risk not seeing
> > anything.
> 
> Seeing none of the 32 bytes of fingerprint instead of only seeing the
> first N of them seems like a very minor inconvenient.

See my other message: seeing the full output could be priceless.
Besides, with the current buffering mode you might not see any of the
bytes at all.  (Not that this is a very important example, but I
didn't invent it.)

> Buffering can indeed be harmful, but only when significant
> computation/time can pass between the `fprintf` and the moment the
> corresponding text is "displayed", which is not the case here (nor
> anywhere else in Emacs's C code, according to my quick "grep stderr").

The time has no influence on this, because no stdio implementation
I've seen flushes buffers based on time since the last output.

> > So this looks like a net loss to me.
> 
> I don't have a strong opinion on this, but FWIW, I see a net gain and no
> significant loss.

I see a net loss.

Re: `message' not outputting the newline "atomically"

[Stefan Monnier]

>> Seeing none of the 32 bytes of fingerprint instead of only seeing the
>> first N of them seems like a very minor inconvenient.
>
> See my other message: seeing the full output could be priceless.

Maybe in some hypothetical scenario.  FWIW this kind of thing never
happened to me, wheres garbled output is something I've seen at least
once a week.

> Besides, with the current buffering mode you might not see any of the
> bytes at all.

Yes, that's what I said.  And I'm perfectly fine with that.

If you think the output of each byte is important, you can add `fflush`
in the loop, of course.

> (Not that this is a very important example, but I didn't invent it.)

In my experience buffered-but-not-printed output is only a problem if
the programs gets to another place in the code, such that the lack of
output leads you to believe the code hasn't yet reached that code when
in fact it's already further.
If it crashes in the middle, the actual output doesn't matter: the
debugger will immediately tell you where the program was.

>> Buffering can indeed be harmful, but only when significant
>> computation/time can pass between the `fprintf` and the moment the
>> corresponding text is "displayed", which is not the case here (nor
>> anywhere else in Emacs's C code, according to my quick "grep stderr").
> The time has no influence on this, because no stdio implementation
> I've seen flushes buffers based on time since the last output.

It's significant when as a user you expected the output to appear
a while ago and it hasn't appeared yet (it leads you to build up
a flawed model in your head of what the program is actually doing).
In my experience, this is the situation where buffering makes debugging
painful.

In the present case this is not an issue because the \n is printed right
after and causes a flush.

>> > So this looks like a net loss to me.
>> I don't have a strong opinion on this, but FWIW, I see a net gain and no
>> significant loss.
> I see a net loss.

Yup, we disagree ;-)


        Stefan

Re: `message' not outputting the newline "atomically"

[Andreas Schwab]

On Jun 21 2019, Paul Eggert <eggert@cs.ucla.edu> wrote:

> it'll mess up formatting for the next stderr diagnostic, and (c) POSIX
> allows platforms to line-buffer stderr by default and Emacs has been
> working on such platforms for ages anyway.

How do you know?

Andreas.

-- 
Andreas Schwab, SUSE Labs, schwab@suse.de
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

Re: `message' not outputting the newline "atomically"

[Paul Eggert]

Andreas Schwab wrote:
> On Jun 21 2019, Paul Eggert <eggert@cs.ucla.edu> wrote:
> 
>> it'll mess up formatting for the next stderr diagnostic, and (c) POSIX
>> allows platforms to line-buffer stderr by default and Emacs has been
>> working on such platforms for ages anyway.
> 
> How do you know?

Trivially, by using a recent Emacs master on GNU/Linux, as this uses 
line-buffered stderr. It works fine.

Re: `message' not outputting the newline "atomically"

[Eli Zaretskii]

> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Sat, 22 Jun 2019 11:53:36 -0700
> Cc: Eli Zaretskii <eliz@gnu.org>, larsi@gnus.org, emacs-devel@gnu.org
> 
> Andreas Schwab wrote:
> > On Jun 21 2019, Paul Eggert <eggert@cs.ucla.edu> wrote:
> > 
> >> it'll mess up formatting for the next stderr diagnostic, and (c) POSIX
> >> allows platforms to line-buffer stderr by default and Emacs has been
> >> working on such platforms for ages anyway.
> > 
> > How do you know?
> 
> Trivially, by using a recent Emacs master on GNU/Linux, as this uses 
> line-buffered stderr. It works fine.

That's not "for ages", though, is it?

Re: `message' not outputting the newline "atomically"

[Paul Eggert]

Eli Zaretskii wrote:
> That's not "for ages", though, is it?

No, it's not. But it does work fine.

Re: `message' not outputting the newline "atomically"

[Andreas Schwab]

On Jun 22 2019, Paul Eggert <eggert@cs.ucla.edu> wrote:

> Andreas Schwab wrote:
>> On Jun 21 2019, Paul Eggert <eggert@cs.ucla.edu> wrote:
>>
>>> it'll mess up formatting for the next stderr diagnostic, and (c) POSIX
>>> allows platforms to line-buffer stderr by default and Emacs has been
>>> working on such platforms for ages anyway.
>>
>> How do you know?
>
> Trivially, by using a recent Emacs master on GNU/Linux, as this uses
> line-buffered stderr. It works fine.

That doesn't prove anything.

Andreas.

-- 
Andreas Schwab, SUSE Labs, schwab@suse.de
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

Re: `message' not outputting the newline "atomically"

[Stefan Monnier]

> $ strace -o /tmp/tr emacs -Q -batch -eval '(message "hello")'
> hello
> $ grep '^write(2,' /tmp/tr
> write(2, "hello", 5)                    = 5
> write(2, "\n", 1)                       = 1
>
> There should be just one 'write' system call, not two.

Rather than fiddle with the line-buffering, how hard would it be to
change our code such that there's a single call to `fprintf` (or
whichever function it is we use in that code) which prints both the
"hello" and the "\n"?


        Stefan

Re: `message' not outputting the newline "atomically"

[Paul Eggert]

On 6/20/19 6:32 AM, Stefan Monnier wrote:
> Rather than fiddle with the line-buffering, how hard would it be to
> change our code such that there's a single call to `fprintf` (or
> whichever function it is we use in that code) which prints both the
> "hello" and the "\n"?

Reasonably hard. That's what I tried to do first, and it's not nearly as 
easy.

I also wrote a patch in which Emacs fully-buffers stderr but calls 
'fflush_unlocked (stderr)' at strategic spots. That will also work and 
is reasonably easy to do, but it's a more-intrusive patch.

Re: `message' not outputting the newline "atomically"

[Daniele Nicolodi]

On 20/06/2019 10:28, Paul Eggert wrote:
> On 6/20/19 6:32 AM, Stefan Monnier wrote:
>> Rather than fiddle with the line-buffering, how hard would it be to
>> change our code such that there's a single call to `fprintf` (or
>> whichever function it is we use in that code) which prints both the
>> "hello" and the "\n"?
> 
> Reasonably hard. That's what I tried to do first, and it's not nearly as 
> easy.

Hi Paul,

maybe I am missing something, but I thought that the issue is only with
output omitted by (message) function call.  If my quick analysis of the
code is correct, those print to stderr only via message_to_stderr() in
xdisp.c. I don't think it would be that hard to make that function emit
the trailing newline along with the message.

What am I missing?

It is funny how message_to_stderr() has an explicit fflush(stderr) at
the end. Eli makes it look like Emacs should force stderr to be
unbuffered, and thus that call is useless.

Cheers,
Dan

Re: `message' not outputting the newline "atomically"

[Paul Eggert]

Daniele Nicolodi wrote:
> I thought that the issue is only with
> output omitted by (message) function call.

Although the original bug report was indeed about 'message', the problem can 
occur with any output sent to stderr, and merely fiddling with 'message' will 
not fix the more-general problem.

> It is funny how message_to_stderr() has an explicit fflush(stderr) at
> the end.

Yes, there is a lot of cargo-culting in the stdio part of the Emacs source code, 
as nobody has taken the time to review all the unnecessary cruft that has 
accumulated over the years; the fflush you mentioned is just part of the 
problem. And although I *did* take the time to do this one little thing right, I 
have been asked to revert the fix because of purely-hypothetical fear of change. 
It is frustrating, indeed.

Eli essentially ordered me to revert the change, so I just now did that. He's 
the maintainer.

Re: `message' not outputting the newline "atomically"

[Lars Ingebrigtsen]

Paul Eggert <eggert@cs.ucla.edu> writes:

> Although the original bug report was indeed about 'message', the
> problem can occur with any output sent to stderr, and merely fiddling
> with 'message' will not fix the more-general problem.

That's true, but the special thing about that message_to_stderr function
is that it first does that fwrite, and then the fputc for the newline,
which is exactly what make this glitch appear.  That can be rewritten
rather trivially to do just one fwrite...

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: `message' not outputting the newline "atomically"

[Paul Eggert]

Lars Ingebrigtsen wrote:
> That's true, but the special thing about that message_to_stderr function
> is that it first does that fwrite, and then the fputc for the newline,
> which is exactly what make this glitch appear.  That can be rewritten
> rather trivially to do just one fwrite...

Yes, that particular function can be fixed, but unfortunately there are several 
other functions like that, and they can't all be fixed so easily. I wrote a 
simple three-line patch to fix them all, but if you'd rather do it the hard way 
and "audit the sources from now to eternity" to make sure the hard way continues 
to be done correctly everywhere in the future, please feel free to do that. Or 
if you want to fix just message_to_stderr but not the other occurrences of the 
problem, please feel free to do that too. Either of these things would improve 
over what is in master now, although they wouldn't be as good as the three-line fix.

Re: `message' not outputting the newline "atomically"

[Lars Ingebrigtsen]

Paul Eggert <eggert@cs.ucla.edu> writes:

> I wrote a simple three-line patch to fix them all, but if
> you'd rather do it the hard way and "audit the sources from now to
> eternity" to make sure the hard way continues to be done correctly
> everywhere in the future, please feel free to do that. Or if you want
> to fix just message_to_stderr but not the other occurrences of the
> problem, please feel free to do that too. Either of these things would
> improve over what is in master now, although they wouldn't be as good
> as the three-line fix.

I agree with you that the line buffering fix was more elegant, but it's
an unnerving change.

Just fixing message_to_stderr, on the other hand, wouldn't scare anybody
and gets us almost all the way there.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: `message' not outputting the newline "atomically"

[Daniele Nicolodi]

On 23/06/2019 14:34, Paul Eggert wrote:
> Daniele Nicolodi wrote:
>> I thought that the issue is only with
>> output omitted by (message) function call.
> 
> Although the original bug report was indeed about 'message', the problem can 
> occur with any output sent to stderr, and merely fiddling with 'message' will 
> not fix the more-general problem.

I am much less familiar with the codebase than you, but I would expect
that there are only a few mechanisms Emacs uses to write to stderr,
excluding debug output.  Anyhow, fixing message_to_stderr is easy and is
going to improve the situation for the most common cases.

Cheers,
Dan

Re: `message' not outputting the newline "atomically"

[Eli Zaretskii]

> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Sun, 23 Jun 2019 13:34:37 -0700
> Cc: emacs-devel@gnu.org
> 
> Eli essentially ordered me to revert the change, so I just now did
> that.

Thank you.

Re: `message' not outputting the newline "atomically"

[HaiJun Zhang]

[-- Attachment #1: Type: text/plain, Size: 1297 bytes --]

What about providing a custom variable to control this? I like the line buffer.

________________________________
From: Emacs-devel <emacs-devel-bounces+netjune=outlook.com@gnu.org> on behalf of Paul Eggert <eggert@cs.ucla.edu>
Sent: Monday, June 24, 2019 4:34
To: Daniele Nicolodi
Cc: emacs-devel@gnu.org
Subject: Re: `message' not outputting the newline "atomically"

Daniele Nicolodi wrote:
> I thought that the issue is only with
> output omitted by (message) function call.

Although the original bug report was indeed about 'message', the problem can
occur with any output sent to stderr, and merely fiddling with 'message' will
not fix the more-general problem.

> It is funny how message_to_stderr() has an explicit fflush(stderr) at
> the end.

Yes, there is a lot of cargo-culting in the stdio part of the Emacs source code,
as nobody has taken the time to review all the unnecessary cruft that has
accumulated over the years; the fflush you mentioned is just part of the
problem. And although I *did* take the time to do this one little thing right, I
have been asked to revert the fix because of purely-hypothetical fear of change.
It is frustrating, indeed.

Eli essentially ordered me to revert the change, so I just now did that. He's
the maintainer.


[-- Attachment #2: Type: text/html, Size: 2235 bytes --]

Re: `message' not outputting the newline "atomically"

[Lars Ingebrigtsen]

So...  does this look OK?

diff --git a/src/xdisp.c b/src/xdisp.c
index 5d70440f1c..0b45ca9e02 100644
--- a/src/xdisp.c
+++ b/src/xdisp.c
@@ -10705,10 +10705,22 @@ message_to_stderr (Lisp_Object m)
       else
 	s = m;
 
-      fwrite (SDATA (s), SBYTES (s), 1, stderr);
+      /* We want to write this out with a single fwrite call so that
+	 output doesn't interleave with other processes writing to
+	 stderr at the same time. */
+      {
+	int length = SBYTES (s);
+	char *string = xmalloc (length + 1);
+	
+	memcpy (string, SSDATA (s), length);
+	*(string + length) = '\n';
+	fwrite (string, length + 1, 1, stderr);
+	xfree (string);
+      }
     }
-  if (!cursor_in_echo_area)
+  else if (!cursor_in_echo_area)
     fputc ('\n', stderr);
+
   fflush (stderr);
 }
 


-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: `message' not outputting the newline "atomically"

[Daniele Nicolodi]

On 24-06-2019 13:48, Lars Ingebrigtsen wrote:
> So...  does this look OK?
> 
> diff --git a/src/xdisp.c b/src/xdisp.c
> index 5d70440f1c..0b45ca9e02 100644
> --- a/src/xdisp.c
> +++ b/src/xdisp.c
> @@ -10705,10 +10705,22 @@ message_to_stderr (Lisp_Object m)
>        else
>  	s = m;
>  
> -      fwrite (SDATA (s), SBYTES (s), 1, stderr);
> +      /* We want to write this out with a single fwrite call so that
> +	 output doesn't interleave with other processes writing to
> +	 stderr at the same time. */
> +      {
> +	int length = SBYTES (s);
> +	char *string = xmalloc (length + 1);
> +	
> +	memcpy (string, SSDATA (s), length);
> +	*(string + length) = '\n';

Isn't this more naturally spelled as below?

string[length] = '\n';

> +	fwrite (string, length + 1, 1, stderr);
> +	xfree (string);
> +      }
>      }
> -  if (!cursor_in_echo_area)
> +  else if (!cursor_in_echo_area)
>      fputc ('\n', stderr);
> +
>    fflush (stderr);

I think the fflush() here can be dropped, stderr is not buffered.

>  }

Cheers,
Dan

Re: `message' not outputting the newline "atomically"

[Lars Ingebrigtsen]

Daniele Nicolodi <daniele@grinta.net> writes:

> Isn't this more naturally spelled as below?
>
> string[length] = '\n';

Yup.

>>    fflush (stderr);
>
> I think the fflush() here can be dropped, stderr is not buffered.

Probably, but Emacs is being built on a large number of systems.  Is
stderr unbuffered on all of them?

It has no performance impact, I'd imagine...

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: `message' not outputting the newline "atomically"

[Paul Eggert]

On 6/24/19 1:17 PM, Lars Ingebrigtsen wrote:
>> I think the fflush() here can be dropped, stderr is not buffered.
> Probably, but Emacs is being built on a large number of systems.  Is
> stderr unbuffered on all of them?

It's safe to drop the fflush, since stderr isn't fully-buffered on any 
system and we should remove cargo-cult code that gets in the way of 
maintenance. But there are more-important problems with the patch.

* The patch also needs a FIXME comment saying it fixes only "message" 
output, not the other uses of stderr in Emacs.

* The patched code has undefined behavior if the string length is 
INT_MAX, and messes up in other ways if the string length exceeds 
INT_MAX. This bug is unlikely but should be fixed.

* The second and third arguments to fwrite should be interchanged. This 
makes a difference on some non-POSIX platforms, and we might as well do 
it right here.

* More important, the patched code shouldn't call xmalloc. Having to 
allocate a buffer as part of an error diagnostic is a recipe for 
trouble. (Suppose the diagnostic is related to being low on memory?) 
Instead, the patched code should just use a fixed-size buffer that is 
guaranteed to exist and be big enough. This is a basic design principle 
for error diagnostics (I vaguely recall Dijkstra did this back in the 
1960s).

* The buffer should contain only PIPE_BUF bytes, since writes larger 
than that can be split by the operating system anyway so any excess size 
is wasted. On POSIX platforms you can use fpathconf to calculate 
PIPE_BUF for stderr. I don't know how to calculate it on MS-Windows 
platforms, but maybe it doesn't matter and you can just pretend it's 
1024 or whatever.

* Larger strings can be be output PIPE_BUF bytes at a time. You'll need 
a loop for this of course, and the loop should do the right thing if one 
of the earlier fwrites fail.

* Better yet, fix the code so that it doesn't need to copy the string 
into an extra buffer at all. Admittedly this will be more work, but it's 
what the code really should be doing anyway.


When I originally looked into this problem, I drafted a patch along the 
lines you're suggesting. But it's a lot of effort to get it right, and 
it's effort that we should be spending elsewhere, not here. This is why 
the three-line patch that I already gave is way better than what you're 
proposing.

Re: `message' not outputting the newline "atomically"

[Lars Ingebrigtsen]

Paul Eggert <eggert@cs.ucla.edu> writes:

> * The patch also needs a FIXME comment saying it fixes only "message"
> output, not the other uses of stderr in Emacs.

Well...  that's kinda unusual.  The commit message may say so, but
having that in the code would be odd.

> * The patched code has undefined behavior if the string length is
> INT_MAX, and messes up in other ways if the string length exceeds
> INT_MAX. This bug is unlikely but should be fixed.

Yup.

> * The second and third arguments to fwrite should be
> interchanged. This makes a difference on some non-POSIX platforms, and
> we might as well do it right here.

I agree, but I just wanted to keep the code as similar as the previous
one as possible.

> * More important, the patched code shouldn't call xmalloc. Having to
> allocate a buffer as part of an error diagnostic is a recipe for
> trouble. (Suppose the diagnostic is related to being low on memory?)
> Instead, the patched code should just use a fixed-size buffer that is
> guaranteed to exist and be big enough. This is a basic design
> principle for error diagnostics (I vaguely recall Dijkstra did this
> back in the 1960s).

It is, but the code also calls code_convert_string, so I thought one
more xmalloc didn't make much difference.

> * The buffer should contain only PIPE_BUF bytes, since writes larger
> than that can be split by the operating system anyway so any excess
> size is wasted. On POSIX platforms you can use fpathconf to calculate
> PIPE_BUF for stderr. I don't know how to calculate it on MS-Windows
> platforms, but maybe it doesn't matter and you can just pretend it's
> 1024 or whatever.

I don't think it's important -- if the buffer is larger than PIPE_BUF,
we still might get multiprocess output interleaved, but I don't think we
care that much.  On modern OS-es it's 4096, and we seldom output things
that are bigger than that via `message', to put it mildly.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: `message' not outputting the newline "atomically"

[Paul Eggert]

On 6/24/19 2:33 PM, Lars Ingebrigtsen wrote:
> the code also calls code_convert_string, so I thought one
> more xmalloc didn't make much difference.

If code_convert_string could cause an infloop when reporting 
low-on-memory messages, then we should fix that infloop too. In the 
meantime we shouldn't make things worse by doing yet another heap 
allocation.

One possibility would be to use an auto (C stack) buffer of size (min 
(MAX_ALLOCA, PIPE_BUF)) bytes if that buffer is large enough; otherwise, 
stick with the current code. This would handle the vast majority of 
cases atomically, and the remaining cases would be no worse off than 
they are now.

Still, it would be better if the code didn't need to copy the string 
into a buffer at all. writev would work for that, since stderr is 
unbuffered. There are other possibilities.

> * The patch also needs a FIXME comment saying it fixes only "message"
> output, not the other uses of stderr in Emacs.
> Well...  that's kinda unusual.  The commit message may say so, but
> having that in the code would be odd.
I think it's worth having that comment here. Someone reading this code 
will reasonably wonder why we're going to all this trouble to 
hand-buffer stderr here, when we don't bother doing it elsewhere. The 
commit log is not a good place to document curious code like that.

Re: `message' not outputting the newline "atomically"

[Paul Eggert]

On 6/24/19 3:03 PM, Paul Eggert wrote:
> writev would work for that, since stderr is unbuffered.

Ooops, not true as stderr might be line-buffered. But you could make 
writev work anyway, by calling fflush before writev.

Re: `message' not outputting the newline "atomically"

[Lars Ingebrigtsen]

Paul Eggert <eggert@cs.ucla.edu> writes:

> On 6/24/19 2:33 PM, Lars Ingebrigtsen wrote:
>> the code also calls code_convert_string, so I thought one
>> more xmalloc didn't make much difference.
>
> If code_convert_string could cause an infloop when reporting
> low-on-memory messages, then we should fix that infloop too. In the
> meantime we shouldn't make things worse by doing yet another heap
> allocation.

But as you've said -- this isn't a general stderr Emacs thing -- this
function is called solely from `message', right?  I tried tracing all
the calls, and that was my conclusion, but I may well have missed
something.

So any very low-level messages about low-on-memory wouldn't be hitting
that code at all, I would have thunk.  Thinked.  Thought.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: `message' not outputting the newline "atomically"

[Lars Ingebrigtsen]

I'm pushing the current version I have now, so we can tinker with it
more collaboratively.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: `message' not outputting the newline "atomically"

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Date: Tue, 25 Jun 2019 00:47:12 +0200
> Cc: Daniele Nicolodi <daniele@grinta.net>, emacs-devel@gnu.org
> 
> I'm pushing the current version I have now, so we can tinker with it
> more collaboratively.

There was no reason to rush this to the repository.  Paul raises valid
concerns, some of which I had as well.  Also, the commit message
describes something that is very different from the code that was
actually pushed.

I think a version with a fixed-size automatic buffer and a loop using
it to write the message in chunks, would be a much cleaner solution
than allocating the buffer off the heap.

There's also a possibility to use the null byte that we
always/normally have in strings after the last byte.  'fwrite' doesn't
need it, so we could replace it with a newline, write the message,
then replace the newline back with the null byte.  This is somewhat
hacky, but it eliminates the need for a buffer and a loop.

Re: `message' not outputting the newline "atomically"

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> There was no reason to rush this to the repository.  Paul raises valid
> concerns, some of which I had as well.  Also, the commit message
> describes something that is very different from the code that was
> actually pushed.

The commit message talked about interleaved outputs, which is exactly
what the patch tries to fix, so I'm not quite sure what you mean here...

> I think a version with a fixed-size automatic buffer and a loop using
> it to write the message in chunks, would be a much cleaner solution
> than allocating the buffer off the heap.

Unless I misunderstood something fundamental here (which is very
possible), this is the function that's run when Emacs is in --batch mode
and some code says `(message ...)'.  I tried to trace the callers to see
that this was the case.  I may not have found them all (users are
message3_nolog and message_with_string), but that does seem to be the
use case.  So this is not an error reporting function, but a bit inside
`message', and I don't think anything that calls `message' could
reasonably expect it to not allocate memory.

> There's also a possibility to use the null byte that we
> always/normally have in strings after the last byte.  'fwrite' doesn't
> need it, so we could replace it with a newline, write the message,
> then replace the newline back with the null byte.  This is somewhat
> hacky, but it eliminates the need for a buffer and a loop.

That's an intriguing idea.  But is there a possibility that the fwrite
would fail in such a way that we'd not get back control in such a way
that we could guarantee that we could replace the newline with a null
byte again?  Especially in a multi-threaded Emacs...

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: `message' not outputting the newline "atomically"

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: eggert@cs.ucla.edu,  daniele@grinta.net,  emacs-devel@gnu.org
> Date: Wed, 26 Jun 2019 11:15:54 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > There was no reason to rush this to the repository.  Paul raises valid
> > concerns, some of which I had as well.  Also, the commit message
> > describes something that is very different from the code that was
> > actually pushed.
> 
> The commit message talked about interleaved outputs, which is exactly
> what the patch tries to fix, so I'm not quite sure what you mean here...

The commit message mentions PIPE_BUF etc., something that was
discussed, but is not in the code.

> > There's also a possibility to use the null byte that we
> > always/normally have in strings after the last byte.  'fwrite' doesn't
> > need it, so we could replace it with a newline, write the message,
> > then replace the newline back with the null byte.  This is somewhat
> > hacky, but it eliminates the need for a buffer and a loop.
> 
> That's an intriguing idea.  But is there a possibility that the fwrite
> would fail in such a way that we'd not get back control in such a way
> that we could guarantee that we could replace the newline with a null
> byte again?

How would that happen?  If fwrite crashes, we are going down in flames
anyway, so no one will have the opportunity to miss that null byte.
Any other failure means fwrite returns with an error code, and we then
put the null byte back as usual.

> Especially in a multi-threaded Emacs...

With the current "concurrency" feature, only one thread can run at a
time, and the thread which called this code cannot be
interrupted/preempted until it returns, because fwrite doesn't cause
us to yield.  In some distant future, when Emacs is really
multi-threaded, all bets are off, because we do such stuff in more
than one place.

Re: `message' not outputting the newline "atomically"

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> The commit message mentions PIPE_BUF etc., something that was
> discussed, but is not in the code.

I attempted to describe the rationale in the commit message, and
PIPE_BUF had to be mentioned (because most people don't seem to know
that there's something magical about writes below that length)...

>> That's an intriguing idea.  But is there a possibility that the fwrite
>> would fail in such a way that we'd not get back control in such a way
>> that we could guarantee that we could replace the newline with a null
>> byte again?
>
> How would that happen?  If fwrite crashes, we are going down in flames
> anyway, so no one will have the opportunity to miss that null byte.
> Any other failure means fwrite returns with an error code, and we then
> put the null byte back as usual.

It's more an instinctual reaction.  :-)   When there's IO, things can go
wrong in ways difficult to imagine, so it's just...  scary.  

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: `message' not outputting the newline "atomically"

[Paul Eggert]

[-- Attachment #1: Type: text/plain, Size: 1342 bytes --]

Eli Zaretskii wrote:
> a version with a fixed-size automatic buffer and a loop using
> it to write the message in chunks, would be a much cleaner solution
> than allocating the buffer off the heap.

Yes, that would be an improvement. I see other issues with the current master. 
It doesn't fix output-line interleaving on AIX or Solaris in places where 
GNU/Linux works OK, as AIX and Solaris fprintf functions are less buffered 
(i.e., they commonly split Emacs output into 'write' calls that are smaller than 
a line), which means that common fprintf output can be interleaved on AIX and 
Solaris when it is not interleaved on GNU/Linux. Also, the current master 
mishandles the rare case where the string has INT_MAX or more bytes, and also 
mishandles line-interleaving if the string contains newlines.

Like Lars, I'm leery about temporarily trashing the representation of a Lisp 
string in order to do I/O. There will likely be other places where we need to do 
this sort of thing, but where the data are not in Lisp strings. It's not that 
hard to efficiently output even a very long Lisp string without modifying it, so 
let's do that.

Proposed patch attached. This defaults to line buffering on glibc as well as on 
AIX and Solaris, as I've verified that glibc buffers appropriately and have used 
it pretty extensively with Emacs.

[-- Attachment #2: 0001-Avoid-interleaving-stderr-lines.patch --]
[-- Type: text/x-patch, Size: 6114 bytes --]

From 9be73e5106262232a722cbe04fb8e387b1d3511d Mon Sep 17 00:00:00 2001
From: Paul Eggert <eggert@cs.ucla.edu>
Date: Wed, 26 Jun 2019 10:24:15 -0700
Subject: [PATCH] Avoid interleaving stderr lines

Do a better job of avoiding interleaved stderr lines.
The previous approach did not work on AIX or Solaris
because they issue multiple writes for common fprintf usages.
Also, it unnecessarily copied data for large strings,
unnecessarily used the heap, and mishandled the rare case
where the string contains INT_MAX or more bytes.
* src/sysdep.c (LINE_BUFFER_STDERR): New macro.
(init_standard_fds): Line-buffer stderr if LINE_BUFFER_STDERR says so.
(write_stderr): New function, that avoids interleaving stderr
lines when possible.
* src/pdumper.c (print_paths_to_root_1):
* src/xdisp.c (message_to_stderr, Ftrace_to_stderr): Use it.
---
 src/lisp.h    |  1 +
 src/pdumper.c |  3 +-
 src/sysdep.c  | 77 +++++++++++++++++++++++++++++++++++++++++++++++++++
 src/xdisp.c   | 15 ++--------
 4 files changed, 81 insertions(+), 15 deletions(-)

diff --git a/src/lisp.h b/src/lisp.h
index 77fc22d118..836b0fcbb2 100644
--- a/src/lisp.h
+++ b/src/lisp.h
@@ -4550,6 +4550,7 @@ maybe_disable_address_randomization (bool dumping, int argc, char **argv)
 extern ptrdiff_t emacs_write_sig (int, void const *, ptrdiff_t);
 extern ptrdiff_t emacs_write_quit (int, void const *, ptrdiff_t);
 extern void emacs_perror (char const *);
+extern void write_stderr (void const *, ptrdiff_t, bool);
 extern int renameat_noreplace (int, char const *, int, char const *);
 extern int str_collate (Lisp_Object, Lisp_Object, Lisp_Object, Lisp_Object);
 extern void syms_of_sysdep (void);
diff --git a/src/pdumper.c b/src/pdumper.c
index c00f8a0af5..68fe4d8ed4 100644
--- a/src/pdumper.c
+++ b/src/pdumper.c
@@ -1405,8 +1405,7 @@ print_paths_to_root_1 (struct dump_context *ctx,
       Lisp_Object repr = Fprin1_to_string (referrer, Qnil);
       for (int i = 0; i < level; ++i)
         fputc (' ', stderr);
-      fwrite (SDATA (repr), 1, SBYTES (repr), stderr);
-      fputc ('\n', stderr);
+      write_stderr (SDATA (repr), SBYTES (repr), true);
       print_paths_to_root_1 (ctx, referrer, level + 1);
     }
 }
diff --git a/src/sysdep.c b/src/sysdep.c
index 4f89e8aba1..54352022eb 100644
--- a/src/sysdep.c
+++ b/src/sysdep.c
@@ -231,6 +231,18 @@ force_open (int fd, int flags)
     }
 }
 
+/* Line-buffered stderr works with glibc.  It is required in AIX and
+   Solaris for short single-line printfs to be atomic.  It does not
+   work with MS-Windows.  When in doubt, assume it does not work.
+   Override the default by compiling with -D LINE_BUFFER_STDERR=[0|1].  */
+#ifndef LINE_BUFFER_STDERR
+# if defined __GLIBC__ || defined _AIX || defined __sun
+#  define LINE_BUFFER_STDERR 1
+# else
+#  define LINE_BUFFER_STDERR 0
+# endif
+#endif
+
 /* Make sure stdin, stdout, and stderr are open to something, so that
    their file descriptors are not hijacked by later system calls.  */
 void
@@ -243,6 +255,9 @@ init_standard_fds (void)
   force_open (STDIN_FILENO, O_WRONLY);
   force_open (STDOUT_FILENO, O_RDONLY);
   force_open (STDERR_FILENO, O_RDONLY);
+
+  if (LINE_BUFFER_STDERR)
+    setvbuf (stderr, NULL, _IOLBF, 0);
 }
 
 /* Return the current working directory.  The result should be freed
@@ -2714,6 +2729,68 @@ emacs_perror (char const *message)
     }
   errno = err;
 }
+
+/* Write BUFFER (of size BUFLEN bytes) to standard error.
+   Follow it with a newline if NLFLAG is true.
+   Avoid interleaving output lines with those of other processes,
+   if the output lines contain fewer than PIPE_BUF bytes.  */
+void
+write_stderr (void const *buffer, ptrdiff_t buflen, bool nlflag)
+{
+#if LINE_BUFFER_STDERR
+  fwrite_unlocked (buffer, 1, buflen, stderr);
+  if (nlflag)
+    fputc_unlocked ('\n', stderr);
+#else
+
+  /* This platform is not known to support line-buffering.
+     Line-buffer by hand.  */
+
+  fflush_unlocked (stderr);
+  if (buflen <= -nlflag)
+    return;
+
+  #ifndef PIPE_BUF
+  enum { PIPE_BUF = 512 };
+  #endif
+  verify (PIPE_BUF <= SSIZE_MAX);
+
+  /* Write chunks each containing as many lines as fit into PIPE_BUF bytes.
+     If a line is too long to fit in PIPE_BUF, write it out piecemeal;
+     it might be interleaved by the OS but this is the best we can do.
+     Append a newline to the last chunk if NLFLAG.  */
+
+  while (true)
+    {
+      char const *buf = buffer;
+      ptrdiff_t n = buflen;
+      bool appendnl = nlflag;
+      if (PIPE_BUF < n + appendnl)
+	{
+	  char const *nladdr = memrchr (buf, '\n', PIPE_BUF);
+	  n = nladdr ? nladdr - buf + 1 : PIPE_BUF;
+	  appendnl = false;
+	}
+
+      verify (PIPE_BUF <= MAX_ALLOCA);
+      char stackbuf[PIPE_BUF];
+      if (appendnl)
+	{
+	  memcpy (stackbuf, buf, n);
+	  stackbuf[n++] = '\n';
+	  buf = stackbuf;
+	}
+
+      n = write (STDERR_FILENO, buf, n);
+      if (n < 0)
+	break;
+      buflen -= n;
+      if (buflen <= -nlflag)
+	break;
+      buffer = buf + n;
+    }
+#endif
+}
 \f
 /* Set the access and modification time stamps of FD (a.k.a. FILE) to be
    ATIME and MTIME, respectively.
diff --git a/src/xdisp.c b/src/xdisp.c
index 9f63ef4b18..4329b48cb6 100644
--- a/src/xdisp.c
+++ b/src/xdisp.c
@@ -10705,18 +10705,7 @@ message_to_stderr (Lisp_Object m)
       else
 	s = m;
 
-      /* We want to write this out with a single fwrite call so that
-	 output doesn't interleave with other processes writing to
-	 stderr at the same time. */
-      {
-	int length = min (INT_MAX, SBYTES (s) + 1);
-	char *string = xmalloc (length);
-
-	memcpy (string, SSDATA (s), length - 1);
-	string[length - 1] = '\n';
-	fwrite (string, 1, length, stderr);
-	xfree (string);
-      }
+      write_stderr (SDATA (s), SBYTES (s), true);
     }
   else if (!cursor_in_echo_area)
     fputc ('\n', stderr);
@@ -19850,7 +19839,7 @@ DEFUN ("trace-to-stderr", Ftrace_to_stderr, Strace_to_stderr, 1, MANY, "",
   (ptrdiff_t nargs, Lisp_Object *args)
 {
   Lisp_Object s = Fformat (nargs, args);
-  fwrite (SDATA (s), 1, SBYTES (s), stderr);
+  write_stderr (SDATA (s), SBYTES (s), false);
   return Qnil;
 }
 
-- 
2.21.0

Re: `message' not outputting the newline "atomically"

[Eli Zaretskii]

> Cc: daniele@grinta.net, emacs-devel@gnu.org
> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Wed, 26 Jun 2019 11:27:45 -0700
> 
> Like Lars, I'm leery about temporarily trashing the representation of a Lisp 
> string in order to do I/O. There will likely be other places where we need to do 
> this sort of thing, but where the data are not in Lisp strings. It's not that 
> hard to efficiently output even a very long Lisp string without modifying it, so 
> let's do that.
> 
> Proposed patch attached. This defaults to line buffering on glibc as well as on 
> AIX and Solaris, as I've verified that glibc buffers appropriately and have used 
> it pretty extensively with Emacs.

I don't understand: this brings back line buffering for stderr,
something that was reverted earlier.  I still object to that.

IOW, my problem is not with the platforms that don't support line
buffering, it's with making stderr line-buffered globally on those
platforms that do.  I thought the loop solution was proposed for those
few places where we want short messages to go out atomically, but
without making stderr line-buffered.

What am I missing?

Re: `message' not outputting the newline "atomically"

[Paul Eggert]

Eli Zaretskii wrote:
> the loop solution was proposed for those
> few places where we want short messages to go out atomically, but
> without making stderr line-buffered.
> 
> What am I missing?

Emacs contains many fprintf calls like this:

   fprintf (stderr, "Using %s\n", term);

Here, TERM is typically a short string. On AIX and Solaris, this fprintf is 
implemented via three 'write' syscalls - one for "Using ", one for the contents 
of TERM, and one for "\n". These outputs will be broken up and hard-to-read on 
AIX and Solaris, if Emacs runs in parallel with other programs also generating 
stderr output. Using line-buffering fixes this.

Essentially, line-buffering is designed to make short output lines atomic, which 
is the problem we are trying to address.

Re: `message' not outputting the newline "atomically"

[Eli Zaretskii]

> Cc: larsi@gnus.org, daniele@grinta.net, emacs-devel@gnu.org
> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Wed, 26 Jun 2019 11:58:26 -0700
> 
> Emacs contains many fprintf calls like this:
> 
>    fprintf (stderr, "Using %s\n", term);
> 
> Here, TERM is typically a short string. On AIX and Solaris, this fprintf is 
> implemented via three 'write' syscalls - one for "Using ", one for the contents 
> of TERM, and one for "\n". These outputs will be broken up and hard-to-read on 
> AIX and Solaris, if Emacs runs in parallel with other programs also generating 
> stderr output. Using line-buffering fixes this.

Can't we also fix that by replacing the above with 'sprintf' followed
by 'write'?  Or some other similar solution?

Re: `message' not outputting the newline "atomically"

[Daniele Nicolodi]

Not that I care much about this, I feel that defending the use case of
debugging Emacs through printf() statements is rooted in believes akin
to religious ones, and religious arguments cannot be won with logic.
However, people keeps putting me in Cc so...

On 26/06/2019 13:11, Eli Zaretskii wrote:
>> Cc: larsi@gnus.org, daniele@grinta.net, emacs-devel@gnu.org
>> From: Paul Eggert <eggert@cs.ucla.edu>
>> Date: Wed, 26 Jun 2019 11:58:26 -0700
>>
>> Emacs contains many fprintf calls like this:
>>
>>    fprintf (stderr, "Using %s\n", term);
>>
>> Here, TERM is typically a short string. On AIX and Solaris, this fprintf is 
>> implemented via three 'write' syscalls - one for "Using ", one for the contents 
>> of TERM, and one for "\n". These outputs will be broken up and hard-to-read on 
>> AIX and Solaris, if Emacs runs in parallel with other programs also generating 
>> stderr output. Using line-buffering fixes this.
> 
> Can't we also fix that by replacing the above with 'sprintf' followed
> by 'write'?

Why is re-implementing line buffering in Emacs any better than using
libc line buffering?  For your main argument in the thread, we would be
loosing crucial debug information if something goes irreparably wrong
between the string preparation and the write().  And given how
error-prone is string manipulation in C, there are much higher chances
that this will happen if we start doing that for every function that
want to print something on stderr.

> Or some other similar solution?

Any solution requires buffering of the output till the first newline.
And you are opposed to buffering.

Are you proposing that we selectively apply buffering in some occasions
but not others?  I am looking forward to the discussion to decide which
functions should use home-grown buffering and which ones should no...

However, if that is the route you are suggesting, it is much easier to
enable line buffering unconditionally ad place fflush() calls where it
matters, than the other way around.

Cheers,
Dan

Re: `message' not outputting the newline "atomically"

[Eli Zaretskii]

> Cc: larsi@gnus.org, emacs-devel@gnu.org
> From: Daniele Nicolodi <daniele@grinta.net>
> Date: Wed, 26 Jun 2019 13:36:54 -0600
> 
> Not that I care much about this, I feel that defending the use case of
> debugging Emacs through printf() statements is rooted in believes akin
> to religious ones, and religious arguments cannot be won with logic.

It isn't religious at all.  We have built-in debugging capabilities
that use stderr, see the trace-redisplay feature as one example.  This
particular feature is very valuable for me, as it happens.

> > Can't we also fix that by replacing the above with 'sprintf' followed
> > by 'write'?
> 
> Why is re-implementing line buffering in Emacs any better than using
> libc line buffering?

Because we can apply that on a per-case basis, whereas setvbuf is
global and irreversible.

> For your main argument in the thread, we would be loosing crucial
> debug information if something goes irreparably wrong between the
> string preparation and the write().  And given how error-prone is
> string manipulation in C, there are much higher chances that this
> will happen if we start doing that for every function that want to
> print something on stderr.

My main argument is actually that the issue this thread is trying to
fix is minor, even insignificant.  But given that some people pressure
to find a solution, I propose various compromises.  As any compromise,
they are somewhat ugly.  My personal preference would be to leave
things as they are.

> Any solution requires buffering of the output till the first newline.
> And you are opposed to buffering.

I'm opposed to buffering stderr globally, yes.  But I can agree to
that in specific cases.  The problem is, setvbuf doesn't allow
selective buffering.

> Are you proposing that we selectively apply buffering in some occasions
> but not others?

YES!

> However, if that is the route you are suggesting, it is much easier to
> enable line buffering unconditionally ad place fflush() calls where it
> matters, than the other way around.

You are suggesting fflush after every character written?  That's
impractical, and we usually use something other than fputc to output
the messages.

Re: `message' not outputting the newline "atomically"

[Paul Eggert]

>> However, if that is the route you are suggesting, it is much easier to
>> enable line buffering unconditionally ad place fflush() calls where it
>> matters, than the other way around.
> You are suggesting fflush after every character written?

I think he was suggesting enabling line buffering (_IOLBF) unconditionally, and 
also placing fflush where we want the output right away even on MS-Windows. The 
fflush would be needed only for MS-Windows, because MS-Windows is the only Emacs 
platform where using _IOLBF does not work (MS-Windows silently treats requests 
to use line buffering as if they requested full buffering, which is not what we 
want for stderr).

If this is the compromise needed to fix this problem then let's do it. _IOLBF is 
a simple and effective solution to this problem on GNU and other POSIXish 
platforms, and we shouldn't let MS-Windows's lack of support for a useful 
feature prevent Emacs from using the feature on non-MS-Windows platforms.

Re: `message' not outputting the newline "atomically"

[Daniele Nicolodi]

On 26/06/2019 20:34, Eli Zaretskii wrote:
>> Cc: larsi@gnus.org, emacs-devel@gnu.org
>> From: Daniele Nicolodi <daniele@grinta.net>
>> Date: Wed, 26 Jun 2019 13:36:54 -0600
>>
>> Not that I care much about this, I feel that defending the use case of
>> debugging Emacs through printf() statements is rooted in believes akin
>> to religious ones, and religious arguments cannot be won with logic.
> 
> It isn't religious at all.  We have built-in debugging capabilities
> that use stderr, see the trace-redisplay feature as one example.  This
> particular feature is very valuable for me, as it happens.

What I meant is that you have a strong believe that there may be cases
in which there is non zero probability that Emacs segfaults after
writing a string a sequence of characters to stderr that does not end in
a newline but at the same time contains crucial (or at least valuable)
information that will help diagnose the problem.  I call this a
religious believe because (as far as I can tell) no example of such
occurrence has been provided and (in an unbounded system) logically
proving the non-existence of something is impossible (as the existence
of God in the universe, thus me calling this a religious believe).  Of
course the Emacs' codebase is bounded, however an exhaustive list of the
supported platforms does not exist, making the possibilities practically
unbounded.  I thus think it is on you to provide a plausible eou xample.

>> Why is re-implementing line buffering in Emacs any better than using
>> libc line buffering?
> 
> Because we can apply that on a per-case basis, whereas setvbuf is
> global and irreversible.

Just add an fflush() after the write()s you care about...

>> Are you proposing that we selectively apply buffering in some occasions
>> but not others?
> 
> YES!
> 
>> However, if that is the route you are suggesting, it is much easier to
>> enable line buffering unconditionally ad place fflush() calls where it
>> matters, than the other way around.
> 
> You are suggesting fflush after every character written?  That's
> impractical, and we usually use something other than fputc to output
> the messages.

I suggest an fflush() after every message that does not terminate in a
newline and that must absolutely end in the terminal if Emacs crashes,
if any exists (see above).

Cheers,
Dan

Re: `message' not outputting the newline "atomically"

[Daniele Nicolodi]

On 26/06/2019 20:34, Eli Zaretskii wrote:
>> Cc: larsi@gnus.org, emacs-devel@gnu.org
>> From: Daniele Nicolodi <daniele@grinta.net>
>> Date: Wed, 26 Jun 2019 13:36:54 -0600
>>
>> Not that I care much about this, I feel that defending the use case of
>> debugging Emacs through printf() statements is rooted in believes akin
>> to religious ones, and religious arguments cannot be won with logic.
> 
> It isn't religious at all.  We have built-in debugging capabilities
> that use stderr, see the trace-redisplay feature as one example.  This
> particular feature is very valuable for me, as it happens.

Because you mention it explicitly, I went and checked how the
trace-redisplay feature works.  I found only one function used by this
feature to outputs something on stderr, debug_method_add(), and this
function uses an fprintf() with a format string that is terminated by a
new line:

  fprintf (stderr, "%p (%s): %s\n", ...)

to output messages to stderr.  I don't think this would be affected in
any way by line buffering stderr.  Can you please be more precise in
what is that makes you worried in relation to trace-redisplay and line
buffered stderr?

Thank you.

Best,
Dan

Re: `message' not outputting the newline "atomically"

[Eli Zaretskii]

> From: Daniele Nicolodi <daniele@grinta.net>
> Date: Mon, 1 Jul 2019 09:41:52 +0200
> 
> Because you mention it explicitly, I went and checked how the
> trace-redisplay feature works.  I found only one function used by this
> feature to outputs something on stderr, debug_method_add(), and this
> function uses an fprintf() with a format string that is terminated by a
> new line:
> 
>   fprintf (stderr, "%p (%s): %s\n", ...)
> 
> to output messages to stderr.

First, you took what I wrote too literally: I said "trace-redisplay",
but didn't mean just the function by that name.  Rather, I meant all
the dump-* debug functions in that area of xdisp.c; sorry for not
being accurate enough.  Though debugging printf's without a newline at
the end are naturally relatively rare, you will find one such printf
even in this particular facility, see dump_glyph.  And if you grep our
sources for "stderr", you will see more of them: I see them in
emacs-module.c, in pdumper.c in xfaces.c, and quite a lot in
regex-emacs.c, to give just a few examples.  Yes, they are the
minority, but they are there.  And even if we go ahead and rewrite the
code to eliminate them (which I would consider a waste of energy),
they will appear again in the future, because sometimes it is just
more natural to write code that way, and avoiding that is a burden for
programmers.

More importantly, I think you are missing the main point here:

> I don't think this would be affected in any way by line buffering
> stderr.

See, this is incorrect.  The problem with line buffering is that you
see nothing at all until the newline is processed.  By contrast, an
unbuffered stream will output the text piecemeal as it is processed,
so if the program is killed half way through printing a line, you will
see more of the line with unbuffered stream than with line buffering,
even if the line ends with a newline.  How much more of that line you
will see depends on the details: the format string (if the output was
generated by the likes of fprintf), the internal implementation
details of the stdio facilities, etc.  But in general you can expect
to see more, and those additional few characters are sometimes
invaluable in debugging a problem, especially when you cannot
reproduce it at will, or not at all (imagine a bug report from another
user).

Emacs is on many platforms a multithreaded program, so it could crash
or be killed at an un-opportune moment even if the data it processes
for the current stderr printout in the main thread is itself
completely valid, let alone if Emacs is in trouble in its main thread
and some of its data is corrupted.  Thus, the situations where the
buffering _will_ affect diagnostics are real, not illusory, and the
desire to avoid losing what little diagnostics we have in these cases
has nothing to do with religion.

Once again, the people who decided to make stderr unbuffered were very
smart.  Over the years, I silently thanked them many times for that
decision, as I needed to diagnose bugs, both mine and those of others.
In high-quality software bugs are quite rare, but when they do happen,
each femto-bit of additional information is precious.  If you don't
share this basic view, if you think the need for having stderr
unbuffered needs some proof, then I guess our experiences and/or the
amounts of gray hair from insufficient diagnostics are very different.

Re: `message' not outputting the newline "atomically"

[Daniele Nicolodi]

On 01/07/2019 08:39, Eli Zaretskii wrote:
> More importantly, I think you are missing the main point here:
> 
>> I don't think this would be affected in any way by line buffering
>> stderr.
> 
> See, this is incorrect.  The problem with line buffering is that you
> see nothing at all until the newline is processed.  By contrast, an
> unbuffered stream will output the text piecemeal as it is processed,
> so if the program is killed half way through printing a line, you will
> see more of the line with unbuffered stream than with line buffering,
> even if the line ends with a newline.  How much more of that line you
> will see depends on the details: the format string (if the output was
> generated by the likes of fprintf), the internal implementation
> details of the stdio facilities, etc.  But in general you can expect
> to see more, and those additional few characters are sometimes
> invaluable in debugging a problem, especially when you cannot
> reproduce it at will, or not at all (imagine a bug report from another
> user).

I think you didn't understand my point, and your selective quoting makes
me thing you are trying to warp what i wrote in your favor. I was
arguing that in the case of a fprintf() with a format string that ends
in a new line there is no way to obtain a partial output, with stderr
unbuffered or line buffered. On any sane implementation of the c library
Either you will get it all or nothing. If you don't think this is the
case I would like to know in what circumstances this does not hold true
and how those apply to Emacs.

Cheers,
Dan

Re: `message' not outputting the newline "atomically"

[Eli Zaretskii]

> From: Daniele Nicolodi <daniele@grinta.net>
> Date: Mon, 1 Jul 2019 19:01:18 +0200
> 
> I think you didn't understand my point, and your selective quoting makes
> me thing you are trying to warp what i wrote in your favor. I was
> arguing that in the case of a fprintf() with a format string that ends
> in a new line there is no way to obtain a partial output, with stderr
> unbuffered or line buffered. On any sane implementation of the c library
> Either you will get it all or nothing.

And I was saying that this is wrong.  With unbuffered stderr, you
_can_ obtain partial output.

Re: `message' not outputting the newline "atomically"

[Daniele Nicolodi]

On 01/07/2019 20:28, Eli Zaretskii wrote:
>> From: Daniele Nicolodi <daniele@grinta.net>
>> Date: Mon, 1 Jul 2019 19:01:18 +0200
>>
>> I think you didn't understand my point, and your selective quoting makes
>> me thing you are trying to warp what i wrote in your favor. I was
>> arguing that in the case of a fprintf() with a format string that ends
>> in a new line there is no way to obtain a partial output, with stderr
>> unbuffered or line buffered. On any sane implementation of the c library
>> Either you will get it all or nothing.
> 
> And I was saying that this is wrong.  With unbuffered stderr, you
> _can_ obtain partial output.

I still disagree. Can you please write a piece of C code that achieves
that, having a fprintf() function call error out, in whichever way you
prefer, such that it's output is partial? I tried to achieve that myself
and I haven't found a way.

Thank you.

Best,
Daniele

Re: `message' not outputting the newline "atomically"

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org
> From: Daniele Nicolodi <daniele@grinta.net>
> Date: Tue, 2 Jul 2019 09:58:34 +0200
> 
> On 01/07/2019 20:28, Eli Zaretskii wrote:
> >> From: Daniele Nicolodi <daniele@grinta.net>
> >> Date: Mon, 1 Jul 2019 19:01:18 +0200
> >>
> >> I think you didn't understand my point, and your selective quoting makes
> >> me thing you are trying to warp what i wrote in your favor. I was
> >> arguing that in the case of a fprintf() with a format string that ends
> >> in a new line there is no way to obtain a partial output, with stderr
> >> unbuffered or line buffered. On any sane implementation of the c library
> >> Either you will get it all or nothing.
> > 
> > And I was saying that this is wrong.  With unbuffered stderr, you
> > _can_ obtain partial output.
> 
> I still disagree.

Then maybe you should have a look at the implementation in the various
libc's out there.  In all those I looked at, fwrite, fprintf,
etc. call fputc or its moral equivalent to emit every single character
they produce; they rely on the lower-level buffering layer to DTRT
regarding buffering.  IOW, none of the functions that write to FILE
stream pay any attention to the newline.  In particular, using fputc
for every character means that in an unbuffered stream each character
will be immediately written out, modulo the low-level device-specific
buffering in the OS.

> Can you please write a piece of C code that achieves that, having a
> fprintf() function call error out, in whichever way you prefer, such
> that it's output is partial? I tried to achieve that myself and I
> haven't found a way.

See one such example below.  The effect might be system-dependent, but
it crashes in the middle of output for me both on GNU/Linux and on
MS-Windows.

(I still don't understand why you insist on an example.  It should be
clear that such situations are possible, even if rare.  We are talking
about a general-purpose diagnostic facility, not something designed
for special-purpose use cases.)

======================================================================
#include <stdio.h>

char foo[] = "Lorem ipsum dolor sit amet, consectetur adipisicing elit, \
sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.";

int
main (void)
{
  fprintf (stderr, "%s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s\n", foo, foo, foo, foo, foo, foo, foo, foo, foo, foo, foo, foo, foo, foo, foo, foo, foo, foo, foo, foo, foo, foo, foo, foo, foo, foo, foo, foo, foo, foo, foo, foo, foo, foo, foo, foo, foo, foo, foo, foo, foo, foo, foo, foo, foo, foo, foo, foo, foo, foo, foo, foo, foo, foo, foo, foo, foo, foo, foo, foo, foo, foo, foo, foo, foo, foo, foo, foo, foo, foo, foo, foo, foo, foo, foo, foo, foo, foo, foo, foo, foo, foo, foo, foo, foo, foo, foo, foo, foo, foo, fo
 o);
  return 0;
}

Re: `message' not outputting the newline "atomically"

[Daniele Nicolodi]

On 02/07/2019 08:47, Eli Zaretskii wrote:
>> Cc: emacs-devel@gnu.org
>> From: Daniele Nicolodi <daniele@grinta.net>
>> Date: Tue, 2 Jul 2019 09:58:34 +0200
>>
>> On 01/07/2019 20:28, Eli Zaretskii wrote:
>>>> From: Daniele Nicolodi <daniele@grinta.net>
>>>> Date: Mon, 1 Jul 2019 19:01:18 +0200
>>>>
>>>> I think you didn't understand my point, and your selective quoting makes
>>>> me thing you are trying to warp what i wrote in your favor. I was
>>>> arguing that in the case of a fprintf() with a format string that ends
>>>> in a new line there is no way to obtain a partial output, with stderr
>>>> unbuffered or line buffered. On any sane implementation of the c library
>>>> Either you will get it all or nothing.
>>>
>>> And I was saying that this is wrong.  With unbuffered stderr, you
>>> _can_ obtain partial output.
>>
>> I still disagree.
> 
> Then maybe you should have a look at the implementation in the various
> libc's out there.  In all those I looked at, fwrite, fprintf,
> etc. call fputc or its moral equivalent to emit every single character
> they produce; they rely on the lower-level buffering layer to DTRT
> regarding buffering.  IOW, none of the functions that write to FILE
> stream pay any attention to the newline.  In particular, using fputc
> for every character means that in an unbuffered stream each character
> will be immediately written out, modulo the low-level device-specific
> buffering in the OS.

I agree.  I would like to stress the "modulo the low-level
device-specific buffering in the OS" part.

>> Can you please write a piece of C code that achieves that, having a
>> fprintf() function call error out, in whichever way you prefer, such
>> that it's output is partial? I tried to achieve that myself and I
>> haven't found a way.
> 
> See one such example below.  The effect might be system-dependent, but
> it crashes in the middle of output for me both on GNU/Linux and on
> MS-Windows.

Great!  Thanks!  Have you verified what is the _minimum_ length of the
output for which you get truncated output vs no output at all?  On my
system it is 1024 characters (and I think that this value is fairly
standard, on the small side if anything).

So the only circumstance in which we would end with no output in the
case of the line buffering, but some output with no buffering is if the
string we are printing is longer than 1024 characters.  This does not
change if the line contains newlines.

Funnily, if the printf() output is less of 1024 characters but contains
newline characters, full lines are printed in case of line buffering,
but nothing is printed in case of no buffering.  It seems thus that if
the median message printed on stderr by Emacs less than 1024 characters,
line buffered stderr is actually allowing Emacs to dump more output on
the terminal than unbuffered stderr  :-)

PS: stdbuf from coreutils is very handy for those tests.

Cheers,
Dan

Re: `message' not outputting the newline "atomically"

[Eli Zaretskii]

> From: Daniele Nicolodi <daniele@grinta.net>
> Date: Tue, 2 Jul 2019 22:56:05 +0200
> 
> > Then maybe you should have a look at the implementation in the various
> > libc's out there.  In all those I looked at, fwrite, fprintf,
> > etc. call fputc or its moral equivalent to emit every single character
> > they produce; they rely on the lower-level buffering layer to DTRT
> > regarding buffering.  IOW, none of the functions that write to FILE
> > stream pay any attention to the newline.  In particular, using fputc
> > for every character means that in an unbuffered stream each character
> > will be immediately written out, modulo the low-level device-specific
> > buffering in the OS.
> 
> I agree.  I would like to stress the "modulo the low-level
> device-specific buffering in the OS" part.

Console devices are usually unbuffered on that level (otherwise, you
couldn't see each character as you type them).

> Great!  Thanks!  Have you verified what is the _minimum_ length of the
> output for which you get truncated output vs no output at all?  On my
> system it is 1024 characters (and I think that this value is fairly
> standard, on the small side if anything).

I don't think the details matter for the issue at hand, because it's
unthinkable to me to rely on such details for GP diagnostic
facilities.  What's important to me is that before seeing this program
in action none of us could predict where exactly it will stop printing.

> So the only circumstance in which we would end with no output in the
> case of the line buffering, but some output with no buffering is if the
> string we are printing is longer than 1024 characters.  This does not
> change if the line contains newlines.

With this buggy program, yes.  But that's just one example, unsuitable
for making general conclusions for the issue at hand.  It just shows
that such problems are possible, and that they happen at places we
cannot easily predict in advance.

> It seems thus that if the median message printed on stderr by Emacs
> less than 1024 characters, line buffered stderr is actually allowing
> Emacs to dump more output on the terminal than unbuffered stderr :-)

You are making very general conclusions from a very specific use case,
something that can easily lead to wrong decisions.

In any case, I think this discussion exhausted itself.  If you still
disagree with the policy of leaving stderr unbuffered, let's agree to
disagree.

Re: `message' not outputting the newline "atomically"

[Daniele Nicolodi]

On 01/07/2019 08:39, Eli Zaretskii wrote:
> Emacs is on many platforms a multithreaded program, so it could crash
> or be killed at an un-opportune moment even if the data it processes
> for the current stderr printout in the main thread is itself
> completely valid, let alone if Emacs is in trouble in its main thread
> and some of its data is corrupted.  Thus, the situations where the
> buffering _will_ affect diagnostics are real, not illusory, and the
> desire to avoid losing what little diagnostics we have in these cases
> has nothing to do with religion.
> 
> Once again, the people who decided to make stderr unbuffered were very
> smart.  Over the years, I silently thanked them many times for that
> decision, as I needed to diagnose bugs, both mine and those of others.
> In high-quality software bugs are quite rare, but when they do happen,
> each femto-bit of additional information is precious.  If you don't
> share this basic view, if you think the need for having stderr
> unbuffered needs some proof, then I guess our experiences and/or the
> amounts of gray hair from insufficient diagnostics are very different.

As I already expressed a couple of times now, this is all hypothetical.
The Emacs bug database is vast, can you please explicitly name just one
bug where there was truncated stderr output?

Thank you.

Best,
Daniele

Re: `message' not outputting the newline "atomically"

[Eli Zaretskii]

> From: Daniele Nicolodi <daniele@grinta.net>
> Date: Mon, 1 Jul 2019 19:03:46 +0200
> 
> As I already expressed a couple of times now, this is all hypothetical.

No, it is not.

> The Emacs bug database is vast, can you please explicitly name just one
> bug where there was truncated stderr output?

I already mentioned in my previous message the places where it could
be truncated.

Re: `message' not outputting the newline "atomically"

[Paul Eggert]

Eli Zaretskii wrote:
> Can't we also fix that by replacing the above with 'sprintf' followed
> by 'write'?

Not easily. Using a fixed-size buffer and truncating the output would violate 
the GNU coding standards; using a PIPE_BUF-size buffer and looping until the 
output was done would be quite difficult to do with sprintf; and allocating the 
buffer on the heap would result in the usual buffer-management hassles such as 
potential leaks and dealing with allocation failures. In short, none of these 
alternatives are nearly as simple and reliable as line-buffered stderr. Even 
fully-buffered stderr (not a solution I favor) would be simpler and more 
reliable than trying to buffer stderr via sprintf+write.

Re: `message' not outputting the newline "atomically"

[Eli Zaretskii]

> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Mon, 24 Jun 2019 15:03:01 -0700
> Cc: Daniele Nicolodi <daniele@grinta.net>, emacs-devel@gnu.org
> 
> On 6/24/19 2:33 PM, Lars Ingebrigtsen wrote:
> > the code also calls code_convert_string, so I thought one
> > more xmalloc didn't make much difference.
> 
> If code_convert_string could cause an infloop when reporting 
> low-on-memory messages, then we should fix that infloop too. In the 
> meantime we shouldn't make things worse by doing yet another heap 
> allocation.

It is very unusual for code_convert_string to allocate memory;
normally it just reuses a buffer that was allocated at startup.

So I think we don't normally have any heap allocations in this use
case -- or didn't have until now.  I agree that we should avoid that.

Re: `message' not outputting the newline "atomically"

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> It is very unusual for code_convert_string to allocate memory;
> normally it just reuses a buffer that was allocated at startup.

Hm!  How does that work?  Is that a buffer that's used for all strings
Emacs allocates or something?  I tried to follow the code, and we seem
to end up in allocate_string_data, which calls malloc...

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: `message' not outputting the newline "atomically"

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: Paul Eggert <eggert@cs.ucla.edu>,  daniele@grinta.net,  emacs-devel@gnu.org
> Date: Wed, 26 Jun 2019 11:21:43 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > It is very unusual for code_convert_string to allocate memory;
> > normally it just reuses a buffer that was allocated at startup.
> 
> Hm!  How does that work?  Is that a buffer that's used for all strings
> Emacs allocates or something?  I tried to follow the code, and we seem
> to end up in allocate_string_data, which calls malloc...

Sorry, I've probably misunderstood what was being discussed.  I thought
you were talking about the conversion itself.  The target string is of
course created via xmalloc.  B ut that could hardly infloop.

Re: `message' not outputting the newline "atomically"

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> Sorry, I've probably misunderstood what was being discussed.  I thought
> you were talking about the conversion itself.  The target string is of
> course created via xmalloc.  B ut that could hardly infloop.

So, that function is a function that will commonly already call xmalloc,
so adding another one doesn't substantially change the guarantees or
behaviour of that function, which was my point (because that's the
`message' path).

However, as Paul points out, there are other uses of stderr, which are
indeed low-level error reporting paths -- but they never call the
function I modified.  Paul wants these to also not interleave in a
multi-process environment, and I think that's a nice goal to have, but I
think we're into somewhat theoretical areas there.

The practical problems with interleaving (that is, the only ones I see
with any regularity) are the ones that stems from using `message' in a
batch Emacs, and the other uses of stderr aren't problems in practice.
(Because they're only used for errors that "shouldn't happen".)

But I'm all for fixing those things if that can be done in a way that
doesn't break anything, and then (of course) the `message' path could
also use that machinery.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: `message' not outputting the newline "atomically"

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: eggert@cs.ucla.edu,  daniele@grinta.net,  emacs-devel@gnu.org
> Date: Thu, 27 Jun 2019 13:03:17 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Sorry, I've probably misunderstood what was being discussed.  I thought
> > you were talking about the conversion itself.  The target string is of
> > course created via xmalloc.  B ut that could hardly infloop.
> 
> So, that function is a function that will commonly already call xmalloc,
> so adding another one doesn't substantially change the guarantees or
> behaviour of that function, which was my point (because that's the
> `message' path).

Each additional call increases the risk you will run out of memory or
hit some calamity.  It isn't an all or nothing situation.

> The practical problems with interleaving (that is, the only ones I see
> with any regularity) are the ones that stems from using `message' in a
> batch Emacs, and the other uses of stderr aren't problems in practice.
> (Because they're only used for errors that "shouldn't happen".)
> 
> But I'm all for fixing those things if that can be done in a way that
> doesn't break anything, and then (of course) the `message' path could
> also use that machinery.

I'm okay with making 'message' write in one go to stderr, but I don't
want to pay the price of having stderr buffered globally.  Several
solutions for this single issue were already proposed, but we keep
rejecting them and returning to line-buffering time and again.  I
think if we want to solve the original problem, that of 'message' in
batch sessions, we should stop looking for a 110% perfect solution,
and certainly not try to solve additional issues while we are at that.

Re: `message' not outputting the newline "atomically"

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> I'm okay with making 'message' write in one go to stderr, but I don't
> want to pay the price of having stderr buffered globally.  Several
> solutions for this single issue were already proposed, but we keep
> rejecting them and returning to line-buffering time and again.  I
> think if we want to solve the original problem, that of 'message' in
> batch sessions, we should stop looking for a 110% perfect solution,
> and certainly not try to solve additional issues while we are at that.

I'm also not very enthusiastic about making stderr line-buffered -- when
exploring Emacs internals, I've sometimes resorted to putting
fprintf(stderr...) calls into the C code to see what's happening, and
while the normal thing to do is to have a \n at the end of those,
that may not be what other people in the same situation always do, and
we'll end up with frustrating people unnecessarily.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: `message' not outputting the newline "atomically"

[Paul Eggert]

[-- Attachment #1: Type: text/plain, Size: 1384 bytes --]

Eli Zaretskii wrote:
> I'm okay with making 'message' write in one go to stderr, but I don't
> want to pay the price of having stderr buffered globally.

I came up with a fix that does all that. Although the fix does not alter the 
buffering of the stderr stream, it causes 'message' and similar functions to 
write in one go to avoid interleaving output. It also fixes the problem on AIX 
and Solaris where a single call to fprintf is implemented by multiple calls to 
'write' even if the diagnostic is short, botching interleaving. It also fixes 
the INT_MAX overflow and memory-allocation issues of the current master's 
implementation of 'message'.

Proposed patches attached. The first one merely refactors and simplifies the 
C-level code, without changing behavior; this simplifies later patches. The 
second patch fixes buffering of most C-level diagnostics in Emacs, so that they 
interleave well with other processes' diagnostics. The third patch similarly 
fixes buffering of most Lisp-level diagnostics in Emacs. The last patch is one 
more simplification of the C code.

I still far prefer the three-line patch I submitted earlier, as it was much 
simpler and the objections to it were theoretical (i.e., not based on actual 
Emacs code). However, the attached patches work nearly as well, and they do so 
while obeying the constraints you imposed on fixing the problem.

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-New-function-errprintf-for-printf-to-stderr.patch --]
[-- Type: text/x-patch; name="0001-New-function-errprintf-for-printf-to-stderr.patch", Size: 79840 bytes --]

From 3739f4b6821435c9feb74b098e95f668d604c768 Mon Sep 17 00:00:00 2001
From: Paul Eggert <eggert@cs.ucla.edu>
Date: Mon, 1 Jul 2019 09:46:11 -0700
Subject: [PATCH 1/4] New function errprintf for printf to stderr
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

This doesn’t change behavior; it just simplifies callers a bit
and makes further changes easier.
* src/alloc.c (mark_memory, test_setjmp, die):
* src/bidi.c (bidi_dump_cached_states):
* src/buffer.c (mmap_free_1, mmap_enlarge, mmap_alloc)
(init_buffer):
* src/charset.c (init_charset):
* src/dispextern.h (TRACE):
* src/dispnew.c (Fdump_redisplay_history)
(init_display_interactive):
* src/emacs-module.c (module_abort):
* src/emacs.c (main, sort_args, Fdump_emacs):
* src/gmalloc.c (mabort):
* src/gtkutil.c (my_log_handler, xg_set_geometry)
(xg_create_widget):
* src/image.c (convert_mono_to_color_image):
* src/lread.c (dir_warning):
* src/nsfont.m (ns_descriptor_to_entity, ns_findfonts)
(nsfont_list_family, nsfont_open, ns_uni_to_glyphs)
(ns_glyph_metrics, ns_dump_glyphstring):
* src/nsmenu.m (ns_update_menubar):
* src/nsterm.h (NSTRACE_MSG_NO_DASHES):
* src/nsterm.m (ns_mouse_position, ns_default, ns_term_init)
(sendEvent:, keyDown:, performDragOperation:, mouseDown:):
* src/pdumper.c (dump_trace, dump_fingerprint)
(print_paths_to_root_1, Fdump_emacs_portable):
* src/print.c (debug_print, safe_debug_print):
* src/regex-emacs.c (DEBUG_PRINT, debug_putchar)
(print_fastmap, print_partial_compiled_pattern)
(print_compiled_pattern, print_double_string, regex_compile):
* src/region-cache.c (pp_cache):
* src/systhread.c (sys_mutex_init, sys_cond_init):
* src/term.c (vfatal):
* src/unexaix.c (CHECK_SCNHDR):
* src/unexelf.c (DEBUG_LOG, unexec):
* src/unexhp9k800.c (read_header):
* src/unexmacosx.c (unexec_error):
* src/widget.c (EmacsFrameInitialize):
* src/xdisp.c (TRACE_MOVE, pos_visible_p, message_to_stderr)
(vmessage, debug_method_add, dump_glyph, dump_glyph_row)
(Fdump_glyph_matrix, dump_glyph_string, expose_window)
(expose_frame):
* src/xfaces.c (Fdump_colors, dump_realized_face)
(Fdump_face, Fshow_face_resources):
* src/xfns.c (print_fontset_result):
* src/xmenu.c (x_menu_show) [XDEBUG]:
* src/xrdb.c (fatal):
* src/xselect.c (TRACE0, TRACE1, TRACE2, TRACE3)
(x_clipboard_manager_error_2):
* src/xsmfns.c (x_session_initialize):
* src/xterm.c (x_trace_wire, x_connection_closed)
(my_log_handler, x_initialize):
Prefer the new functions to using fprintf etc. to stderr.
All files changed to include sysstdio.h instead if stdio.h
if they weren’t doing so already.
* src/dispextern.h (TRACE):
* src/xdisp.c (TRACE_MOVE):
All callers changed to omit stderr, since macro now arranges
for stderr itself.
* src/sysdep.c (errputc, errputs, errprintf, verrprintf):
New functions.
---
 src/alloc.c        |  12 +-
 src/bidi.c         |  24 ++--
 src/buffer.c       |  16 +--
 src/charset.c      |   2 +-
 src/dispnew.c      |  13 +-
 src/emacs-module.c |   8 +-
 src/emacs.c        |  57 ++++----
 src/gmalloc.c      |   2 +-
 src/gtkutil.c      |   9 +-
 src/image.c        |   2 +-
 src/lread.c        |   2 +-
 src/nsfont.m       |  41 +++---
 src/nsmenu.m       |  12 +-
 src/nsterm.h       |  11 +-
 src/nsterm.m       |  27 ++--
 src/pdumper.c      |  34 +++--
 src/print.c        |   8 +-
 src/regex-emacs.c  | 134 +++++++++----------
 src/region-cache.c |  15 +--
 src/sysdep.c       |  29 ++++
 src/sysstdio.h     |   5 +
 src/systhread.c    |   7 +-
 src/term.c         |   6 +-
 src/unexaix.c      |   7 +-
 src/unexelf.c      |   8 +-
 src/unexhp9k800.c  |   2 +-
 src/unexmacosx.c   |   6 +-
 src/widget.c       |   3 +-
 src/xdisp.c        | 324 ++++++++++++++++++++++-----------------------
 src/xfaces.c       |  64 ++++-----
 src/xfns.c         |   6 +-
 src/xmenu.c        |   2 +-
 src/xrdb.c         |   2 +-
 src/xselect.c      |  12 +-
 src/xsmfns.c       |   6 +-
 src/xterm.c        |  15 +--
 36 files changed, 473 insertions(+), 460 deletions(-)

diff --git a/src/alloc.c b/src/alloc.c
index 64aaa8acdf..9610f01c05 100644
--- a/src/alloc.c
+++ b/src/alloc.c
@@ -22,7 +22,6 @@ Copyright (C) 1985-1986, 1988, 1993-1995, 1997-2019 Free Software
 
 #include <errno.h>
 #include <stdint.h>
-#include <stdio.h>
 #include <stdlib.h>
 #include <limits.h>		/* For CHAR_BIT.  */
 #include <signal.h>		/* For SIGABRT, SIGDANGER.  */
@@ -38,6 +37,7 @@ Copyright (C) 1985-1986, 1988, 1993-1995, 1997-2019 Free Software
 #include "ptr-bounds.h"
 #include "puresize.h"
 #include "sheap.h"
+#include "sysstdio.h"
 #include "systime.h"
 #include "character.h"
 #include "buffer.h"
@@ -4717,7 +4717,7 @@ mark_memory (void const *start, void const *end)
        Lisp_Object obj = build_string ("test");
        struct Lisp_String *s = XSTRING (obj);
        garbage_collect ();
-       fprintf (stderr, "test '%s'\n", s->u.s.data);
+       errprintf ("test '%s'\n", s->u.s.data);
        return Qnil;
      }
 
@@ -4813,10 +4813,10 @@ test_setjmp (void)
          taking place, or the setjmp just didn't save the register.  */
 
       if (x == 1)
-	fprintf (stderr, SETJMP_WILL_LIKELY_WORK);
+	errputs (SETJMP_WILL_LIKELY_WORK);
       else
 	{
-	  fprintf (stderr, SETJMP_WILL_NOT_WORK);
+	  errputs (SETJMP_WILL_NOT_WORK);
 	  exit (1);
 	}
     }
@@ -7189,8 +7189,8 @@ DEFUN ("suspicious-object", Fsuspicious_object, Ssuspicious_object, 1, 1, 0,
 void
 die (const char *msg, const char *file, int line)
 {
-  fprintf (stderr, "\r\n%s:%d: Emacs fatal error: assertion failed: %s\r\n",
-	   file, line, msg);
+  errprintf ("\r\n%s:%d: Emacs fatal error: assertion failed: %s\r\n",
+	     file, line, msg);
   terminate_due_to_signal (SIGABRT, INT_MAX);
 }
 
diff --git a/src/bidi.c b/src/bidi.c
index c530d49c10..991894c4e0 100644
--- a/src/bidi.c
+++ b/src/bidi.c
@@ -238,13 +238,13 @@ Copyright (C) 2000-2001, 2004-2005, 2009-2019 Free Software Foundation, Inc.
    necessary.  */
 
 #include <config.h>
-#include <stdio.h>
 
 #include "lisp.h"
 #include "character.h"
 #include "buffer.h"
 #include "dispextern.h"
 #include "region-cache.h"
+#include "sysstdio.h"
 
 static bool bidi_initialized = 0;
 
@@ -3586,24 +3586,22 @@ bidi_dump_cached_states (void)
 
   if (bidi_cache_idx == 0)
     {
-      fprintf (stderr, "The cache is empty.\n");
+      errputs ("The cache is empty.\n");
       return;
     }
-  fprintf (stderr, "Total of  %"pD"d state%s in cache:\n",
-	   bidi_cache_idx, bidi_cache_idx == 1 ? "" : "s");
+  errprintf ("Total of  %"pD"d state%s in cache:\n",
+	     bidi_cache_idx, bidi_cache_idx == 1 ? "" : "s");
 
   for (i = bidi_cache[bidi_cache_idx - 1].charpos; i > 0; i /= 10)
     ndigits++;
-  fputs ("ch  ", stderr);
+  errputs ("ch  ");
   for (i = 0; i < bidi_cache_idx; i++)
-    fprintf (stderr, "%*c", ndigits, bidi_cache[i].ch);
-  fputs ("\n", stderr);
-  fputs ("lvl ", stderr);
+    errprintf ("%*c", ndigits, bidi_cache[i].ch);
+  errputs ("\nlvl ");
   for (i = 0; i < bidi_cache_idx; i++)
-    fprintf (stderr, "%*d", ndigits, bidi_cache[i].resolved_level);
-  fputs ("\n", stderr);
-  fputs ("pos ", stderr);
+    errprintf ("%*d", ndigits, bidi_cache[i].resolved_level);
+  errputs ("\npos ");
   for (i = 0; i < bidi_cache_idx; i++)
-    fprintf (stderr, "%*"pD"d", ndigits, bidi_cache[i].charpos);
-  fputs ("\n", stderr);
+    errprintf ("%*"pD"d", ndigits, bidi_cache[i].charpos);
+  errputc ('\n');
 }
diff --git a/src/buffer.c b/src/buffer.c
index 209e29f0f1..14477d4ae1 100644
--- a/src/buffer.c
+++ b/src/buffer.c
@@ -24,7 +24,6 @@ Copyright (C) 1985-1989, 1993-1995, 1997-2019 Free Software Foundation,
 #include <sys/stat.h>
 #include <sys/param.h>
 #include <errno.h>
-#include <stdio.h>
 #include <stdlib.h>
 #include <unistd.h>
 
@@ -33,6 +32,7 @@ Copyright (C) 1985-1989, 1993-1995, 1997-2019 Free Software Foundation,
 #include "lisp.h"
 #include "intervals.h"
 #include "process.h"
+#include "sysstdio.h"
 #include "systime.h"
 #include "window.h"
 #include "commands.h"
@@ -4781,7 +4781,7 @@ mmap_free_1 (struct mmap_region *r)
     mmap_regions = r->next;
 
   if (munmap (r, r->nbytes_mapped) == -1)
-    fprintf (stderr, "munmap: %s\n", emacs_strerror (errno));
+    errprintf ("munmap: %s\n", emacs_strerror (errno));
 }
 
 
@@ -4800,7 +4800,7 @@ mmap_enlarge (struct mmap_region *r, int npages)
       /* Unmap pages at the end of the region.  */
       nbytes = - npages * mmap_page_size;
       if (munmap (region_end - nbytes, nbytes) == -1)
-	fprintf (stderr, "munmap: %s\n", emacs_strerror (errno));
+	errprintf ("munmap: %s\n", emacs_strerror (errno));
       else
 	{
 	  r->nbytes_mapped -= nbytes;
@@ -4822,14 +4822,14 @@ mmap_enlarge (struct mmap_region *r, int npages)
 	  p = mmap (region_end, nbytes, PROT_READ | PROT_WRITE,
 		    MAP_ANON | MAP_PRIVATE | MAP_FIXED, mmap_fd, 0);
 	  if (p == MAP_FAILED)
-	    ; /* fprintf (stderr, "mmap: %s\n", emacs_strerror (errno)); */
+	    ; /* errprintf ("mmap: %s\n", emacs_strerror (errno)); */
 	  else if (p != region_end)
 	    {
 	      /* Kernels are free to choose a different address.  In
 		 that case, unmap what we've mapped above; we have
 		 no use for it.  */
 	      if (munmap (p, nbytes) == -1)
-		fprintf (stderr, "munmap: %s\n", emacs_strerror (errno));
+		errprintf ("munmap: %s\n", emacs_strerror (errno));
 	    }
 	  else
 	    {
@@ -4867,7 +4867,7 @@ mmap_alloc (void **var, size_t nbytes)
   if (p == MAP_FAILED)
     {
       if (errno != ENOMEM)
-	fprintf (stderr, "mmap: %s\n", emacs_strerror (errno));
+	errprintf ("mmap: %s\n", emacs_strerror (errno));
       p = NULL;
     }
   else
@@ -5378,8 +5378,8 @@ init_buffer (void)
 
   if (!pwd)
     {
-      fprintf (stderr, "Error getting directory: %s\n",
-               emacs_strerror (errno));
+      errprintf ("Error getting directory: %s\n",
+		 emacs_strerror (errno));
       bset_directory (current_buffer, Qnil);
     }
   else
diff --git a/src/charset.c b/src/charset.c
index c0700f972e..f2efc7946a 100644
--- a/src/charset.c
+++ b/src/charset.c
@@ -2293,7 +2293,7 @@ init_charset (void)
       /* This used to be non-fatal (dir_warning), but it should not
          happen, and if it does sooner or later it will cause some
          obscure problem (eg bug#6401), so better abort.  */
-      fprintf (stderr, "Error: charsets directory not found:\n\
+      errprintf ("Error: charsets directory not found:\n\
 %s\n\
 Emacs will not function correctly without the character map files.\n%s\
 Please check your installation!\n",
diff --git a/src/dispnew.c b/src/dispnew.c
index 52a7b6d6ee..28f0c49c0b 100644
--- a/src/dispnew.c
+++ b/src/dispnew.c
@@ -220,7 +220,7 @@ DEFUN ("dump-redisplay-history", Fdump_redisplay_history,
     {
       if (i < 0)
 	i = REDISPLAY_HISTORY_SIZE - 1;
-      fprintf (stderr, "%s\n", redisplay_history[i].trace);
+      errprintf ("%s\n", redisplay_history[i].trace);
     }
 
   return Qnil;
@@ -6153,8 +6153,7 @@ init_display_interactive (void)
 
       if (display_arg && !x_display_ok (display))
 	{
-	  fprintf (stderr, "Display %s unavailable, simulating -nw\n",
-		   display);
+	  errprintf ("Display %s unavailable, simulating -nw\n", display);
 	  inhibit_window_system = 1;
 	}
     }
@@ -6204,12 +6203,14 @@ init_display_interactive (void)
 #endif
   if (!terminal_type)
     {
+      char const *msg = ("Please set the environment variable "
+			 "TERM; see 'tset'.");
 #ifdef HAVE_WINDOW_SYSTEM
       if (! inhibit_window_system)
-	fprintf (stderr, "Please set the environment variable DISPLAY or TERM (see 'tset').\n");
-      else
+	msg = ("Please set the environment variable "
+	       "DISPLAY or TERM (see 'tset').");
 #endif /* HAVE_WINDOW_SYSTEM */
-	fprintf (stderr, "Please set the environment variable TERM; see 'tset'.\n");
+      errprintf ("%s\n", msg);
       exit (1);
     }
 
diff --git a/src/emacs-module.c b/src/emacs-module.c
index c856663d2f..7faad14adc 100644
--- a/src/emacs-module.c
+++ b/src/emacs-module.c
@@ -81,7 +81,6 @@ #define EMACS_MODULE_GMP
 #include <stdarg.h>
 #include <stddef.h>
 #include <stdint.h>
-#include <stdio.h>
 #include <stdlib.h>
 #include <time.h>
 
@@ -91,6 +90,7 @@ #define EMACS_MODULE_GMP
 #include "coding.h"
 #include "keyboard.h"
 #include "syssignal.h"
+#include "sysstdio.h"
 #include "thread.h"
 
 #include <intprops.h>
@@ -1299,12 +1299,12 @@ value_storage_contains_p (const struct emacs_value_storage *storage,
 static AVOID ATTRIBUTE_FORMAT_PRINTF (1, 2)
 module_abort (const char *format, ...)
 {
-  fputs ("Emacs module assertion: ", stderr);
+  errputs ("Emacs module assertion: ");
   va_list args;
   va_start (args, format);
-  vfprintf (stderr, format, args);
+  verrprintf (format, args);
   va_end (args);
-  fputc ('\n', stderr);
+  errputc ('\n');
   fflush (NULL);
   emacs_abort ();
 }
diff --git a/src/emacs.c b/src/emacs.c
index 32bb57e272..4fd45497fc 100644
--- a/src/emacs.c
+++ b/src/emacs.c
@@ -1145,12 +1145,12 @@ main (int argc, char **argv)
 	  tem2 = Fsymbol_value (intern_c_string ("emacs-copyright"));
 	  if (!STRINGP (tem))
 	    {
-	      fprintf (stderr, "Invalid value of 'emacs-version'\n");
+	      errputs ("Invalid value of 'emacs-version'\n");
 	      exit (1);
 	    }
 	  if (!STRINGP (tem2))
 	    {
-	      fprintf (stderr, "Invalid value of 'emacs-copyright'\n");
+	      errputs ("Invalid value of 'emacs-copyright'\n");
 	      exit (1);
 	    }
 	  else
@@ -1192,8 +1192,8 @@ main (int argc, char **argv)
 #endif
       if (chdir (ch_to_dir) != 0)
         {
-          fprintf (stderr, "%s: Can't chdir to %s: %s\n",
-                   argv[0], ch_to_dir, strerror (errno));
+          errprintf ("%s: Can't chdir to %s: %s\n",
+		     argv[0], ch_to_dir, strerror (errno));
           exit (1);
         }
       original_pwd = emacs_wd;
@@ -1317,15 +1317,15 @@ main (int argc, char **argv)
 		  != STDOUT_FILENO))
 	    {
 	      char *errstring = strerror (errno);
-	      fprintf (stderr, "%s: %s: %s\n", argv[0], term, errstring);
+	      errprintf ("%s: %s: %s\n", argv[0], term, errstring);
 	      exit (EXIT_FAILURE);
 	    }
 	  if (! isatty (STDIN_FILENO))
 	    {
-	      fprintf (stderr, "%s: %s: not a tty\n", argv[0], term);
+	      errprintf ("%s: %s: not a tty\n", argv[0], term);
 	      exit (EXIT_FAILURE);
 	    }
-	  fprintf (stderr, "Using %s\n", term);
+	  errprintf ("Using %s\n", term);
 #ifdef HAVE_WINDOW_SYSTEM
 	  inhibit_window_system = true; /* -t => -nw */
 #endif
@@ -1422,7 +1422,7 @@ main (int argc, char **argv)
              before exiting.  */
           if (emacs_pipe (daemon_pipe) != 0)
             {
-              fprintf (stderr, "Cannot pipe!\n");
+              errputs ("Cannot pipe!\n");
               exit (1);
             }
         } /* daemon_type == 2 */
@@ -1432,8 +1432,7 @@ main (int argc, char **argv)
       int systemd_socket = sd_listen_fds (1);
 
       if (systemd_socket > 1)
-        fprintf (stderr,
-		 ("\n"
+	errputs (("\n"
 		  "Warning: systemd passed more than one socket to Emacs.\n"
 		  "Try 'Accept=false' in the Emacs socket unit file.\n"));
       else if (systemd_socket == 1
@@ -1443,7 +1442,7 @@ main (int argc, char **argv)
 #endif /* HAVE_LIBSYSTEMD */
 
 #ifdef USE_GTK
-      fprintf (stderr, "\nWarning: due to a long standing Gtk+ bug\nhttps://gitlab.gnome.org/GNOME/gtk/issues/221\n\
+      errputs ("\nWarning: due to a long standing Gtk+ bug\nhttps://gitlab.gnome.org/GNOME/gtk/issues/221\n\
 Emacs might crash when run in daemon mode and the X11 connection is unexpectedly lost.\n\
 Using an Emacs configured with --with-x-toolkit=lucid does not have this problem.\n");
 #endif /* USE_GTK */
@@ -1477,12 +1476,12 @@ main (int argc, char **argv)
 
               if (retval < 0)
                 {
-                  fprintf (stderr, "Error reading status from child\n");
+                  errputs ("Error reading status from child\n");
                   exit (1);
                 }
               else if (retval == 0)
                 {
-                  fprintf (stderr, "Error: server did not start correctly\n");
+                  errputs ("Error: server did not start correctly\n");
                   exit (1);
                 }
 
@@ -1508,7 +1507,7 @@ main (int argc, char **argv)
 
                 if (! (0 <= fdStrlen && fdStrlen < sizeof fdStr))
                   {
-                    fprintf (stderr, "daemon: child name too long\n");
+                    errputs ("daemon: child name too long\n");
                     exit (EXIT_CANNOT_INVOKE);
                   }
 
@@ -1525,7 +1524,7 @@ main (int argc, char **argv)
             if (!dname_arg || !*dname_arg || strnlen (dname_arg, 71) == 71
 		|| !strchr (dname_arg, '\n'))
           {
-            fprintf (stderr, "emacs daemon: daemon name absent or too long\n");
+            errputs ("emacs daemon: daemon name absent or too long\n");
             exit (EXIT_CANNOT_INVOKE);
           }
             dname_arg2[0] = '\0';
@@ -1546,12 +1545,12 @@ main (int argc, char **argv)
       w32_daemon_event = CreateEvent (NULL, TRUE, FALSE, W32_DAEMON_EVENT);
       if (w32_daemon_event == NULL)
         {
-          fprintf (stderr, "Couldn't create MS-Windows event for daemon: %s\n",
-		   w32_strerror (0));
+          errprintf ("Couldn't create MS-Windows event for daemon: %s\n",
+		     w32_strerror (0));
           exit (1);
         }
 #else /* MSDOS */
-      fprintf (stderr, "This platform does not support daemon mode.\n");
+      errputs ("This platform does not support daemon mode.\n");
       exit (1);
 #endif /* MSDOS */
       if (dname_arg)
@@ -1662,7 +1661,7 @@ main (int argc, char **argv)
                 NULL, &skip_args);
   if (will_dump_p () && module_assertions)
     {
-      fputs ("Module assertions are not supported during dumping\n", stderr);
+      errputs ("Module assertions are not supported during dumping\n");
       exit (1);
     }
   init_module_assertions (module_assertions);
@@ -2329,7 +2328,9 @@ sort_args (int argc, char **argv)
 		{
 		  /* This is an internal error.
 		     Eg if one long option is a prefix of another.  */
-		  fprintf (stderr, "Option '%s' matched multiple standard arguments\n", argv[from]);
+		  errprintf
+		    ("Option '%s' matched multiple standard arguments\n",
+		     argv[from]);
 		}
 	      /* Should we not also warn if there was no match?	 */
 	    }
@@ -2566,14 +2567,14 @@ DEFUN ("dump-emacs", Fdump_emacs, Sdump_emacs, 2, 2, 0,
 
   if (heap_bss_diff > MAX_HEAP_BSS_DIFF)
     {
-      fprintf (stderr, "**************************************************\n");
-      fprintf (stderr, "Warning: Your system has a gap between BSS and the\n");
-      fprintf (stderr, "heap (%"pMu" bytes).  This usually means that exec-shield\n",
-               heap_bss_diff);
-      fprintf (stderr, "or something similar is in effect.  The dump may\n");
-      fprintf (stderr, "fail because of this.  See the section about\n");
-      fprintf (stderr, "exec-shield in etc/PROBLEMS for more information.\n");
-      fprintf (stderr, "**************************************************\n");
+      errprintf (("**************************************************\n"
+		  "Warning: Your system has a gap between BSS and the\n"
+		  "heap (%"pMu" bytes).  This usually means that exec-shield\n"
+		  "or something similar is in effect.  The dump may\n"
+		  "fail because of this.  See the section about\n"
+		  "exec-shield in etc/PROBLEMS for more information.\n"
+		  "**************************************************\n"),
+		 heap_bss_diff);
     }
 # endif
 
diff --git a/src/gmalloc.c b/src/gmalloc.c
index bac3ffb7e5..57ebe12d3c 100644
--- a/src/gmalloc.c
+++ b/src/gmalloc.c
@@ -2012,7 +2012,7 @@ mabort (enum mcheck_status status)
 #ifdef __GNU_LIBRARY__
   __libc_fatal (msg);
 #else
-  fprintf (stderr, "mcheck: %s\n", msg);
+  errprintf ("mcheck: %s\n", msg);
   emacs_abort ();
 #endif
 }
diff --git a/src/gtkutil.c b/src/gtkutil.c
index 1d15aec253..3c8447a0c4 100644
--- a/src/gtkutil.c
+++ b/src/gtkutil.c
@@ -21,13 +21,13 @@ Copyright (C) 2003-2019 Free Software Foundation, Inc.
 
 #ifdef USE_GTK
 #include <float.h>
-#include <stdio.h>
 
 #include <c-ctype.h>
 
 #include "lisp.h"
 #include "dispextern.h"
 #include "frame.h"
+#include "sysstdio.h"
 #include "systime.h"
 #include "xterm.h"
 #include "blockinput.h"
@@ -829,7 +829,7 @@ my_log_handler (const gchar *log_domain, GLogLevelFlags log_level,
 		const gchar *msg, gpointer user_data)
 {
   if (!strstr (msg, "visible children"))
-    fprintf (stderr, "XX %s-WARNING **: %s\n", log_domain, msg);
+    errprintf ("XX %s-WARNING **: %s\n", log_domain, msg);
 }
 #endif
 
@@ -894,7 +894,7 @@ xg_set_geometry (struct frame *f)
 
 	  if (!gtk_window_parse_geometry (GTK_WINDOW (FRAME_GTK_OUTER_WIDGET (f)),
 					  geom_str))
-	    fprintf (stderr, "Failed to parse: '%s'\n", geom_str);
+	    errprintf ("Failed to parse: '%s'\n", geom_str);
 
 	  g_log_remove_handler ("Gtk", id);
 	}
@@ -2850,8 +2850,7 @@ xg_create_widget (const char *type, const char *name, struct frame *f,
     }
   else
     {
-      fprintf (stderr, "bad type in xg_create_widget: %s, doing nothing\n",
-               type);
+      errprintf ("bad type in xg_create_widget: %s, doing nothing\n", type);
     }
 
   return w;
diff --git a/src/image.c b/src/image.c
index d44a9d3dc2..864f73f8ef 100644
--- a/src/image.c
+++ b/src/image.c
@@ -3335,7 +3335,7 @@ convert_mono_to_color_image (struct frame *f, struct image *img,
   DeleteDC (new_img_dc);
   DeleteObject (img->pixmap);
   if (new_pixmap == 0)
-    fprintf (stderr, "Failed to convert image to color.\n");
+    errputs ("Failed to convert image to color.\n");
   else
     img->pixmap = new_pixmap;
 }
diff --git a/src/lread.c b/src/lread.c
index 5fa90cad3f..da8dd774dc 100644
--- a/src/lread.c
+++ b/src/lread.c
@@ -4721,7 +4721,7 @@ dir_warning (char const *use, Lisp_Object dirname)
 {
   static char const format[] = "Warning: %s '%s': %s\n";
   char *diagnostic = emacs_strerror (errno);
-  fprintf (stderr, format, use, SSDATA (ENCODE_SYSTEM (dirname)), diagnostic);
+  errprintf (format, use, SSDATA (ENCODE_SYSTEM (dirname)), diagnostic);
 
   /* Don't log the warning before we've initialized!!  */
   if (initialized)
diff --git a/src/nsfont.m b/src/nsfont.m
index eca97ab86c..0bda357e3a 100644
--- a/src/nsfont.m
+++ b/src/nsfont.m
@@ -212,7 +212,7 @@ static void ns_glyph_metrics (struct nsfont_info *font_info,
 
     if (NSFONT_TRACE)
       {
-	fprintf (stderr, "created font_entity:\n    ");
+	errputs ("created font_entity:\n    ");
 	debug_print (font_entity);
       }
 
@@ -313,7 +313,7 @@ static void ns_glyph_metrics (struct nsfont_info *font_info,
 	    if (*bytes1 == 0)  // *bytes1 & *bytes2 != *bytes2
 		off++;
 	  }
-    // fprintf(stderr, "off = %d\ttot = %d\n", off,tot);
+    // errprintf ("off = %d\ttot = %d\n", off,tot);
     return (float)off / tot < 1.0F - pct;
 }
 
@@ -542,8 +542,8 @@ but also for ascii (which causes unnecessary font substitution).  */
     block_input ();
     if (NSFONT_TRACE)
       {
-	fprintf (stderr, "nsfont: %s for fontspec:\n    ",
-		 (isMatch ? "match" : "list"));
+	errprintf ("nsfont: %s for fontspec:\n    ",
+		   (isMatch ? "match" : "list"));
 	debug_print (font_spec);
       }
 
@@ -596,8 +596,8 @@ but also for ascii (which causes unnecessary font substitution).  */
       return ns_fallback_entity ();
 
     if (NSFONT_TRACE)
-	fprintf (stderr, "    Returning %"pD"d entities.\n",
-		 list_length (list));
+	errprintf ("    Returning %"pD"d entities.\n",
+		   list_length (list));
 
     return list;
 }
@@ -668,8 +668,8 @@ Properties to be considered are same as for list().  */
   /* FIXME: escape the name?  */
 
   if (NSFONT_TRACE)
-    fprintf (stderr, "nsfont: list families returning %"pD"d entries\n",
-	     list_length (list));
+    errprintf ("nsfont: list families returning %"pD"d entries\n",
+	       list_length (list));
 
   unblock_input ();
   return list;
@@ -698,7 +698,7 @@ Properties to be considered are same as for list().  */
 
   if (NSFONT_TRACE)
     {
-      fprintf (stderr, "nsfont: open size %d of fontentity:\n    ", pixel_size);
+      errprintf ("nsfont: open size %d of fontentity:\n    ", pixel_size);
       debug_print (font_entity);
     }
 
@@ -1272,8 +1272,8 @@ is false when (FROM > 0 || TO < S->nchars).  */
   unsigned short *glyphs;
 
   if (NSFONT_TRACE)
-    fprintf (stderr, "%p\tFinding glyphs for glyphs in block %d\n",
-            font_info, block);
+    errprintf ("%p\tFinding glyphs for glyphs in block %d\n",
+	       font_info, block);
 
   block_input ();
 
@@ -1348,8 +1348,8 @@ is false when (FROM > 0 || TO < S->nchars).  */
   struct font_metrics *metrics;
 
   if (NSFONT_TRACE)
-    fprintf (stderr, "%p\tComputing metrics for glyphs in block %d\n",
-            font_info, block);
+    errprintf ("%p\tComputing metrics for glyphs in block %d\n",
+	       font_info, block);
 
 #ifdef NS_IMPL_GNUSTEP
   /* not implemented yet (as of startup 0.18), so punt */
@@ -1472,16 +1472,13 @@ - (void)setIntAttribute: (NSInteger)attributeTag value: (NSInteger)val
 {
   int i;
 
-  fprintf (stderr, "Glyph string len = %d at (%d, %d) overhang (%d, %d),"
-"overlap = %d, bg_filled = %d:",
-           s->nchars, s->x, s->y, s->left_overhang, s->right_overhang,
-           s->row->overlapping_p, s->background_filled_p);
+  errprintf (("Glyph string len = %d at (%d, %d) overhang (%d, %d),"
+	      "overlap = %d, bg_filled = %d:"),
+	     s->nchars, s->x, s->y, s->left_overhang, s->right_overhang,
+	     s->row->overlapping_p, s->background_filled_p);
   for (i =0; i<s->nchars; i++)
-    {
-      int c = s->first_glyph[i].u.ch;
-      fprintf (stderr, "%c", c);
-    }
-  fprintf (stderr, "\n");
+    errputc (s->first_glyph[i].u.ch);
+  errputc ('\n');
 }
 
 static void syms_of_nsfont_for_pdumper (void);
diff --git a/src/nsmenu.m b/src/nsmenu.m
index 817f8cff18..099f20bb0d 100644
--- a/src/nsmenu.m
+++ b/src/nsmenu.m
@@ -119,7 +119,7 @@
   if (f != SELECTED_FRAME ())
       return;
   XSETFRAME (Vmenu_updating_frame, f);
-/*fprintf (stderr, "ns_update_menubar: frame: %p\tdeep: %d\tsub: %p\n", f, deep_p, submenu); */
+/*errprintf ("ns_update_menubar: frame: %p\tdeep: %d\tsub: %p\n", f, deep_p, submenu); */
 
   block_input ();
   pool = [[NSAutoreleasePool alloc] init];
@@ -236,8 +236,8 @@
       if (submenu && n == 0)
         {
           /* should have found a menu for this one but didn't */
-          fprintf (stderr, "ERROR: did not find lisp menu for submenu '%s'.\n",
-                  [[submenu title] UTF8String]);
+          errprintf ("ERROR: did not find lisp menu for submenu '%s'.\n",
+		     [[submenu title] UTF8String]);
 	  discard_menu_items ();
 	  unbind_to (specpdl_count, Qnil);
           [pool release];
@@ -292,7 +292,7 @@
 #if NSMENUPROFILE
               ftime (&tb);
               t += 1000*tb.time+tb.millitm;
-              fprintf (stderr, "NO CHANGE!  CUTTING OUT after %ld msec.\n", t);
+              errprintf ("NO CHANGE!  CUTTING OUT after %ld msec.\n", t);
 #endif
 
               free_menubar_widget_value_tree (first_wv);
@@ -447,7 +447,7 @@
 #if NSMENUPROFILE
   ftime (&tb);
   t += 1000*tb.time+tb.millitm;
-  fprintf (stderr, "Menu update took %ld msec.\n", t);
+  errprintf ("Menu update took %ld msec.\n", t);
 #endif
 
   /* set main menu */
@@ -581,7 +581,7 @@ - (void)menuNeedsUpdate: (NSMenu *)menu
   */
   if (trackingMenu == 0)
     return;
-/*fprintf (stderr, "Updating menu '%s'\n", [[self title] UTF8String]); NSLog (@"%@\n", event); */
+/*errprintf ("Updating menu '%s'\n", [[self title] UTF8String]); NSLog (@"%@\n", event); */
 #ifdef NS_IMPL_GNUSTEP
   /* Don't know how to do this for anything other than Mac OS X 10.5 and later.
      This is wrong, as it might run Lisp code in the event loop.  */
diff --git a/src/nsterm.h b/src/nsterm.h
index 567f462ec6..7e15b9cd34 100644
--- a/src/nsterm.h
+++ b/src/nsterm.h
@@ -23,6 +23,7 @@
 #include "character.h"
 #include "font.h"
 #include "sysselect.h"
+#include "sysstdio.h"
 
 #ifdef HAVE_NS
 #ifdef __OBJC__
@@ -176,11 +177,11 @@ #define NSTRACE_MSG_NO_DASHES(...)                                          \
     {                                                                       \
       if (nstrace_enabled)                                                  \
         {                                                                   \
-          fprintf (stderr, "%-10s:%5d: [%5d]%.*s",                          \
-                   __FILE__, __LINE__, nstrace_num++,                       \
-                   2*nstrace_depth, "  | | | | | | | | | | | | | | | ..");  \
-          fprintf (stderr, __VA_ARGS__);                                    \
-          fprintf (stderr, "\n");                                           \
+	  errprintf ("%-10s:%5d: [%5d]%.*s",				    \
+		     __FILE__, __LINE__, nstrace_num++,			    \
+		     2*nstrace_depth, "  | | | | | | | | | | | | | | | .."); \
+	  errprintf (__VA_ARGS__);					    \
+	  errputc ('\n');						    \
         }                                                                   \
     }                                                                       \
   while(0)
diff --git a/src/nsterm.m b/src/nsterm.m
index bc1c7860aa..91eab1b0d2 100644
--- a/src/nsterm.m
+++ b/src/nsterm.m
@@ -2121,7 +2121,7 @@ so some key presses (TAB) are swallowed by the system.  */
 
   color_table->colors[idx] = color;
   [color retain];
-  /* fprintf(stderr, "color_table: allocated %d\n",idx); */
+  /* errprintf ("color_table: allocated %d\n", idx); */
   return idx;
 }
 
@@ -2466,7 +2466,7 @@ so some key presses (TAB) are swallowed by the system.  */
 
   if (*fp == NULL)
     {
-      fprintf (stderr, "Warning: ns_mouse_position () called with null *fp.\n");
+      errputs ("Warning: ns_mouse_position () called with null *fp.\n");
       return;
     }
 
@@ -5032,8 +5032,8 @@ static Lisp_Object ns_string_to_lispmod (const char *s)
         *result = make_float (f);
       else if (is_modstring && value)
         *result = ns_string_to_lispmod (value);
-      else fprintf (stderr,
-                   "Bad value for default \"%s\": \"%s\"\n", parameter, value);
+      else
+	errprintf ("Bad value for default \"%s\": \"%s\"\n", parameter, value);
     }
 }
 
@@ -5222,8 +5222,7 @@ static Lisp_Object ns_new_font (struct frame *f, Lisp_Object font_object,
     {
       if (emacs_pipe (selfds) != 0)
         {
-          fprintf (stderr, "Failed to create pipe: %s\n",
-                   emacs_strerror (errno));
+          errprintf ("Failed to create pipe: %s\n", emacs_strerror (errno));
           emacs_abort ();
         }
 
@@ -5613,7 +5612,7 @@ - (void)sendEvent: (NSEvent *)theEvent
 
   if (type == NSEventTypeCursorUpdate && window == nil)
     {
-      fprintf (stderr, "Dropping external cursor update event.\n");
+      errputs ("Dropping external cursor update event.\n");
       return;
     }
 
@@ -6288,8 +6287,8 @@ In that case we use UCKeyTranslate (ns_get_shifted_character)
         emacs_event->modifiers ^= parse_solitary_modifier (ns_function_modifier);
 
       if (NS_KEYLOG)
-        fprintf (stderr, "keyDown: code =%x\tfnKey =%x\tflags = %x\tmods = %x\n",
-                 code, fnKeysym, flags, emacs_event->modifiers);
+        errprintf ("keyDown: code =%x\tfnKey =%x\tflags = %x\tmods = %x\n",
+		   code, fnKeysym, flags, emacs_event->modifiers);
 
       /* If it was a function key or had control-like modifiers, pass
          it directly to Emacs.  */
@@ -6333,7 +6332,7 @@ In that case we use UCKeyTranslate (ns_get_shifted_character)
      https://developer.apple.com/library/mac/documentation/Cocoa/Conceptual/EventOverview/HandlingKeyEvents/HandlingKeyEvents.html.  */
 
   if (NS_KEYLOG && !processingCompose)
-    fprintf (stderr, "keyDown: Begin compose sequence.\n");
+    errputs ("keyDown: Begin compose sequence.\n");
 
   /* FIXME: interpretKeyEvents doesn’t seem to send insertText if ⌘ is
      used as shift-like modifier, at least on El Capitan.  Mask it
@@ -8291,7 +8290,7 @@ -(BOOL)performDragOperation: (id <NSDraggingInfo>) sender
     }
   else
     {
-      fprintf (stderr, "Invalid data type in dragging pasteboard\n");
+      errputs ("Invalid data type in dragging pasteboard\n");
       return NO;
     }
 
@@ -9035,8 +9034,8 @@ - (void)mouseDown: (NSEvent *)e
     case NSScrollerKnobSlot:  /* GNUstep-only */
       last_hit_part = scroll_bar_move_ratio; break;
     default:  /* NSScrollerNoPart? */
-      fprintf (stderr, "EmacsScroller-mouseDown: unexpected part %ld\n",
-               (long) part);
+      errprintf ("EmacsScroller-mouseDown: unexpected part %ld\n",
+		 (long) part);
       return;
     }
 
@@ -9319,7 +9318,7 @@ Convert an X font name (XLFD) to an NS font name.
             name[i+1] = c_toupper (name[i+1]);
         }
     }
-  /* fprintf (stderr, "converted '%s' to '%s'\n",xlfd,name); */
+  /* errprintf ("converted '%s' to '%s'\n", xlfd, name); */
   ret = [[NSString stringWithUTF8String: name] UTF8String];
   xfree (name);
   return ret;
diff --git a/src/pdumper.c b/src/pdumper.c
index c00f8a0af5..bfa0dba7f0 100644
--- a/src/pdumper.c
+++ b/src/pdumper.c
@@ -23,7 +23,6 @@
 #include <math.h>
 #include <stdarg.h>
 #include <stdint.h>
-#include <stdio.h>
 #include <stdlib.h>
 #include <sys/mman.h>
 #include <sys/param.h>
@@ -42,6 +41,7 @@
 #include "lisp.h"
 #include "pdumper.h"
 #include "window.h"
+#include "sysstdio.h"
 #include "systime.h"
 #include "thread.h"
 #include "bignum.h"
@@ -158,7 +158,7 @@ dump_trace (const char *fmt, ...)
     {
       va_list args;
       va_start (args, fmt);
-      vfprintf (stderr, fmt, args);
+      verrprintf (fmt, args);
       va_end (args);
     }
 }
@@ -326,10 +326,10 @@ dump_reloc_set_offset (struct dump_reloc *reloc, dump_off offset)
 static void
 dump_fingerprint (const char *label, unsigned char const *xfingerprint)
 {
-  fprintf (stderr, "%s: ", label);
+  errprintf ("%s: ", label);
   for (int i = 0; i < 32; ++i)
-    fprintf (stderr, "%02x", (unsigned) xfingerprint[i]);
-  fprintf (stderr, "\n");
+    errprintf ("%02x", (unsigned) xfingerprint[i]);
+  errputc ('\n');
 }
 
 /* Format of an Emacs portable dump file.  All offsets are relative to
@@ -1403,10 +1403,9 @@ print_paths_to_root_1 (struct dump_context *ctx,
       Lisp_Object referrer = XCAR (referrers);
       referrers = XCDR (referrers);
       Lisp_Object repr = Fprin1_to_string (referrer, Qnil);
-      for (int i = 0; i < level; ++i)
-        fputc (' ', stderr);
+      errprintf ("%*s", level, "");
       fwrite (SDATA (repr), 1, SBYTES (repr), stderr);
-      fputc ('\n', stderr);
+      errputc ('\n');
       print_paths_to_root_1 (ctx, referrer, level + 1);
     }
 }
@@ -4226,16 +4225,15 @@ DEFUN ("dump-emacs-portable",
   dump_seek (ctx, 0);
   dump_write (ctx, &ctx->header, sizeof (ctx->header));
 
-  fprintf (stderr, "Dump complete\n");
-  fprintf (stderr,
-           "Byte counts: header=%lu hot=%lu discardable=%lu cold=%lu\n",
-           (unsigned long) (header_end - header_start),
-           (unsigned long) (hot_end - hot_start),
-           (unsigned long) (discardable_end - ctx->header.discardable_start),
-           (unsigned long) (cold_end - ctx->header.cold_start));
-  fprintf (stderr, "Reloc counts: hot=%u discardable=%u\n",
-           number_hot_relocations,
-           number_discardable_relocations);
+  errputs ("Dump complete\n");
+  errprintf ("Byte counts: header=%lu hot=%lu discardable=%lu cold=%lu\n",
+	     (unsigned long) (header_end - header_start),
+	     (unsigned long) (hot_end - hot_start),
+	     (unsigned long) (discardable_end - ctx->header.discardable_start),
+	     (unsigned long) (cold_end - ctx->header.cold_start));
+  errprintf ("Reloc counts: hot=%u discardable=%u\n",
+	     number_hot_relocations,
+	     number_discardable_relocations);
 
   unblock_input ();
   return unbind_to (count, Qnil);
diff --git a/src/print.c b/src/print.c
index 406abbf4a3..df45cb27ce 100644
--- a/src/print.c
+++ b/src/print.c
@@ -840,7 +840,7 @@ DEFUN ("redirect-debugging-output", Fredirect_debugging_output, Sredirect_debugg
 debug_print (Lisp_Object arg)
 {
   Fprin1 (arg, Qexternal_debugging_output);
-  fprintf (stderr, "\r\n");
+  errputs ("\r\n");
 }
 
 void safe_debug_print (Lisp_Object) EXTERNALLY_VISIBLE;
@@ -854,9 +854,9 @@ safe_debug_print (Lisp_Object arg)
   else
     {
       EMACS_UINT n = XLI (arg);
-      fprintf (stderr, "#<%s_LISP_OBJECT 0x%08"pI"x>\r\n",
-	       !valid ? "INVALID" : "SOME",
-	       n);
+      errprintf ("#<%s_LISP_OBJECT 0x%08"pI"x>\r\n",
+		 !valid ? "INVALID" : "SOME",
+		 n);
     }
 }
 
diff --git a/src/regex-emacs.c b/src/regex-emacs.c
index ac9f91dacb..83850b8b7d 100644
--- a/src/regex-emacs.c
+++ b/src/regex-emacs.c
@@ -442,7 +442,7 @@ #define CHARSET_RANGE_TABLE_END(range_table, count)	\
 
 # define DEBUG_STATEMENT(e) e
 # define DEBUG_PRINT(...)                                       \
-  if (regex_emacs_debug > 0) fprintf (stderr, __VA_ARGS__)
+  if (regex_emacs_debug > 0) errprintf (__VA_ARGS__)
 # define DEBUG_COMPILES_ARGUMENTS
 # define DEBUG_PRINT_COMPILED_PATTERN(p, s, e)				\
   if (regex_emacs_debug > 0) print_partial_compiled_pattern (s, e)
@@ -453,11 +453,11 @@ #define CHARSET_RANGE_TABLE_END(range_table, count)	\
 debug_putchar (int c)
 {
   if (c >= 32 && c <= 126)
-    fputc (c, stderr);
+    errputc (c);
   else
     {
       unsigned int uc = c;
-      fprintf (stderr, "{%02x}", uc);
+      errprintf ("{%02x}", uc);
     }
 }
 
@@ -482,12 +482,12 @@ print_fastmap (char *fastmap)
 	    }
 	  if (was_a_range)
 	    {
-	      fprintf (stderr, "-");
+	      errputs ("-");
 	      debug_putchar (i - 1);
 	    }
 	}
     }
-  fputc ('\n', stderr);
+  errputc ('\n');
 }
 
 
@@ -503,50 +503,50 @@ print_partial_compiled_pattern (re_char *start, re_char *end)
 
   if (start == NULL)
     {
-      fprintf (stderr, "(null)\n");
+      errputs ("(null)\n");
       return;
     }
 
   /* Loop over pattern commands.  */
   while (p < pend)
     {
-      fprintf (stderr, "%td:\t", p - start);
+      errprintf ("%td:\t", p - start);
 
       switch ((re_opcode_t) *p++)
 	{
 	case no_op:
-	  fprintf (stderr, "/no_op");
+	  errputs ("/no_op");
 	  break;
 
 	case succeed:
-	  fprintf (stderr, "/succeed");
+	  errputs ("/succeed");
 	  break;
 
 	case exactn:
 	  mcnt = *p++;
-	  fprintf (stderr, "/exactn/%d", mcnt);
+	  errprintf ("/exactn/%d", mcnt);
 	  do
 	    {
-	      fprintf (stderr, "/");
+	      errputc ('/');
 	      debug_putchar (*p++);
 	    }
 	  while (--mcnt);
 	  break;
 
 	case start_memory:
-	  fprintf (stderr, "/start_memory/%d", *p++);
+	  errprintf ("/start_memory/%d", *p++);
 	  break;
 
 	case stop_memory:
-	  fprintf (stderr, "/stop_memory/%d", *p++);
+	  errprintf ("/stop_memory/%d", *p++);
 	  break;
 
 	case duplicate:
-	  fprintf (stderr, "/duplicate/%d", *p++);
+	  errprintf ("/duplicate/%d", *p++);
 	  break;
 
 	case anychar:
-	  fprintf (stderr, "/anychar");
+	  errputs ("/anychar");
 	  break;
 
 	case charset:
@@ -557,11 +557,11 @@ print_partial_compiled_pattern (re_char *start, re_char *end)
 	    int length = CHARSET_BITMAP_SIZE (p - 1);
 	    bool has_range_table = CHARSET_RANGE_TABLE_EXISTS_P (p - 1);
 
-	    fprintf (stderr, "/charset [%s",
-		     (re_opcode_t) *(p - 1) == charset_not ? "^" : "");
+	    errprintf ("/charset [%s",
+		       (re_opcode_t) *(p - 1) == charset_not ? "^" : "");
 
 	    if (p + *p >= pend)
-	      fprintf (stderr, " !extends past end of pattern! ");
+	      errputs (" !extends past end of pattern! ");
 
 	    for (c = 0; c < 256; c++)
 	      if (c / 8 < length
@@ -570,7 +570,7 @@ print_partial_compiled_pattern (re_char *start, re_char *end)
 		  /* Are we starting a range?  */
 		  if (last + 1 == c && ! in_range)
 		    {
-		      fprintf (stderr, "-");
+		      errputc ('-');
 		      in_range = true;
 		    }
 		  /* Have we broken a range?  */
@@ -589,14 +589,14 @@ print_partial_compiled_pattern (re_char *start, re_char *end)
 	    if (in_range)
 	      debug_putchar (last);
 
-	    fprintf (stderr, "]");
+	    errputc (']');
 
 	    p += 1 + length;
 
 	    if (has_range_table)
 	      {
 		int count;
-		fprintf (stderr, "has-range-table");
+		errputs ("has-range-table");
 
 		/* ??? Should print the range table; for now, just skip it.  */
 		p += 2;		/* skip range table bits */
@@ -607,136 +607,136 @@ print_partial_compiled_pattern (re_char *start, re_char *end)
 	  break;
 
 	case begline:
-	  fprintf (stderr, "/begline");
+	  errputs ("/begline");
 	  break;
 
 	case endline:
-	  fprintf (stderr, "/endline");
+	  errputs ("/endline");
 	  break;
 
 	case on_failure_jump:
 	  EXTRACT_NUMBER_AND_INCR (mcnt, p);
-	  fprintf (stderr, "/on_failure_jump to %td", p + mcnt - start);
+	  errprintf ("/on_failure_jump to %td", p + mcnt - start);
 	  break;
 
 	case on_failure_keep_string_jump:
 	  EXTRACT_NUMBER_AND_INCR (mcnt, p);
-	  fprintf (stderr, "/on_failure_keep_string_jump to %td",
-		   p + mcnt - start);
+	  errprintf ("/on_failure_keep_string_jump to %td",
+		     p + mcnt - start);
 	  break;
 
 	case on_failure_jump_nastyloop:
 	  EXTRACT_NUMBER_AND_INCR (mcnt, p);
-	  fprintf (stderr, "/on_failure_jump_nastyloop to %td",
-		   p + mcnt - start);
+	  errprintf ("/on_failure_jump_nastyloop to %td",
+		     p + mcnt - start);
 	  break;
 
 	case on_failure_jump_loop:
 	  EXTRACT_NUMBER_AND_INCR (mcnt, p);
-	  fprintf (stderr, "/on_failure_jump_loop to %td",
-		   p + mcnt - start);
+	  errprintf ("/on_failure_jump_loop to %td",
+		     p + mcnt - start);
 	  break;
 
 	case on_failure_jump_smart:
 	  EXTRACT_NUMBER_AND_INCR (mcnt, p);
-	  fprintf (stderr, "/on_failure_jump_smart to %td",
-		   p + mcnt - start);
+	  errprintf ("/on_failure_jump_smart to %td",
+		     p + mcnt - start);
 	  break;
 
 	case jump:
 	  EXTRACT_NUMBER_AND_INCR (mcnt, p);
-	  fprintf (stderr, "/jump to %td", p + mcnt - start);
+	  errprintf ("/jump to %td", p + mcnt - start);
 	  break;
 
 	case succeed_n:
 	  EXTRACT_NUMBER_AND_INCR (mcnt, p);
 	  EXTRACT_NUMBER_AND_INCR (mcnt2, p);
-	  fprintf (stderr, "/succeed_n to %td, %d times",
-		   p - 2 + mcnt - start, mcnt2);
+	  errprintf ("/succeed_n to %td, %d times",
+		     p - 2 + mcnt - start, mcnt2);
 	  break;
 
 	case jump_n:
 	  EXTRACT_NUMBER_AND_INCR (mcnt, p);
 	  EXTRACT_NUMBER_AND_INCR (mcnt2, p);
-	  fprintf (stderr, "/jump_n to %td, %d times",
-		   p - 2 + mcnt - start, mcnt2);
+	  errprintf ("/jump_n to %td, %d times",
+		     p - 2 + mcnt - start, mcnt2);
 	  break;
 
 	case set_number_at:
 	  EXTRACT_NUMBER_AND_INCR (mcnt, p);
 	  EXTRACT_NUMBER_AND_INCR (mcnt2, p);
-	  fprintf (stderr, "/set_number_at location %td to %d",
-		   p - 2 + mcnt - start, mcnt2);
+	  errprintf ("/set_number_at location %td to %d",
+		     p - 2 + mcnt - start, mcnt2);
 	  break;
 
 	case wordbound:
-	  fprintf (stderr, "/wordbound");
+	  errputs ("/wordbound");
 	  break;
 
 	case notwordbound:
-	  fprintf (stderr, "/notwordbound");
+	  errputs ("/notwordbound");
 	  break;
 
 	case wordbeg:
-	  fprintf (stderr, "/wordbeg");
+	  errputs ("/wordbeg");
 	  break;
 
 	case wordend:
-	  fprintf (stderr, "/wordend");
+	  errputs ("/wordend");
 	  break;
 
 	case symbeg:
-	  fprintf (stderr, "/symbeg");
+	  errputs ("/symbeg");
 	  break;
 
 	case symend:
-	  fprintf (stderr, "/symend");
+	  errputs ("/symend");
 	  break;
 
 	case syntaxspec:
-	  fprintf (stderr, "/syntaxspec");
+	  errputs ("/syntaxspec");
 	  mcnt = *p++;
-	  fprintf (stderr, "/%d", mcnt);
+	  errprintf ("/%d", mcnt);
 	  break;
 
 	case notsyntaxspec:
-	  fprintf (stderr, "/notsyntaxspec");
+	  errputs ("/notsyntaxspec");
 	  mcnt = *p++;
-	  fprintf (stderr, "/%d", mcnt);
+	  errprintf ("/%d", mcnt);
 	  break;
 
 	case at_dot:
-	  fprintf (stderr, "/at_dot");
+	  errputs ("/at_dot");
 	  break;
 
 	case categoryspec:
-	  fprintf (stderr, "/categoryspec");
+	  errputs ("/categoryspec");
 	  mcnt = *p++;
-	  fprintf (stderr, "/%d", mcnt);
+	  errprintf ("/%d", mcnt);
 	  break;
 
 	case notcategoryspec:
-	  fprintf (stderr, "/notcategoryspec");
+	  errputs ("/notcategoryspec");
 	  mcnt = *p++;
-	  fprintf (stderr, "/%d", mcnt);
+	  errprintf ("/%d", mcnt);
 	  break;
 
 	case begbuf:
-	  fprintf (stderr, "/begbuf");
+	  errputs ("/begbuf");
 	  break;
 
 	case endbuf:
-	  fprintf (stderr, "/endbuf");
+	  errputs ("/endbuf");
 	  break;
 
 	default:
-	  fprintf (stderr, "?%d", *(p-1));
+	  errprintf ("?%d", *(p-1));
 	}
 
-      fprintf (stderr, "\n");
+      errputc ('\n');
     }
 
-  fprintf (stderr, "%td:\tend of pattern.\n", p - start);
+  errprintf ("%td:\tend of pattern.\n", p - start);
 }
 
 
@@ -746,18 +746,18 @@ print_compiled_pattern (struct re_pattern_buffer *bufp)
   re_char *buffer = bufp->buffer;
 
   print_partial_compiled_pattern (buffer, buffer + bufp->used);
-  fprintf (stderr, "%td bytes used/%td bytes allocated.\n",
-           bufp->used, bufp->allocated);
+  errprintf ("%td bytes used/%td bytes allocated.\n",
+	     bufp->used, bufp->allocated);
 
   if (bufp->fastmap_accurate && bufp->fastmap)
     {
-      fprintf (stderr, "fastmap: ");
+      errputs ("fastmap: ");
       print_fastmap (bufp->fastmap);
     }
 
-  fprintf (stderr, "re_nsub: %td\t", bufp->re_nsub);
-  fprintf (stderr, "regs_alloc: %d\t", bufp->regs_allocated);
-  fprintf (stderr, "can_be_null: %d\n", bufp->can_be_null);
+  errprintf ("re_nsub: %td\t", bufp->re_nsub);
+  errprintf ("regs_alloc: %d\t", bufp->regs_allocated);
+  errprintf ("can_be_null: %d\n", bufp->can_be_null);
   fflush (stderr);
   /* Perhaps we should print the translate table?  */
 }
@@ -768,7 +768,7 @@ print_double_string (re_char *where, re_char *string1, ptrdiff_t size1,
 		     re_char *string2, ptrdiff_t size2)
 {
   if (where == NULL)
-    fprintf (stderr, "(null)");
+    errputs ("(null)");
   else
     {
       int i;
@@ -1751,7 +1751,7 @@ regex_compile (re_char *pattern, ptrdiff_t size,
     {
       for (ptrdiff_t debug_count = 0; debug_count < size; debug_count++)
 	debug_putchar (pattern[debug_count]);
-      fputc ('\n', stderr);
+      errputc ('\n');
     }
 #endif
 
diff --git a/src/region-cache.c b/src/region-cache.c
index 57a26f2fa6..51604e815f 100644
--- a/src/region-cache.c
+++ b/src/region-cache.c
@@ -762,19 +762,18 @@ pp_cache (struct region_cache *c)
   ptrdiff_t beg_u = c->buffer_beg + c->beg_unchanged;
   ptrdiff_t end_u = c->buffer_end - c->end_unchanged;
 
-  fprintf (stderr,
-           "basis: %"pD"d..%"pD"d    modified: %"pD"d..%"pD"d\n",
-           c->buffer_beg, c->buffer_end,
-           beg_u, end_u);
+  errprintf ("basis: %"pD"d..%"pD"d    modified: %"pD"d..%"pD"d\n",
+	     c->buffer_beg, c->buffer_end,
+	     beg_u, end_u);
 
   for (ptrdiff_t i = 0; i < c->cache_len; i++)
     {
       ptrdiff_t pos = BOUNDARY_POS (c, i);
 
-      fprintf (stderr, "%c%c%"pD"d : %d\n",
-	       pos < beg_u ? 'v' : pos == beg_u ? '-' : ' ',
-	       pos > end_u ? '^' : pos == end_u ? '-' : ' ',
-	       pos, BOUNDARY_VALUE (c, i));
+      errprintf ("%c%c%"pD"d : %d\n",
+		 pos < beg_u ? 'v' : pos == beg_u ? '-' : ' ',
+		 pos > end_u ? '^' : pos == end_u ? '-' : ' ',
+		 pos, BOUNDARY_VALUE (c, i));
     }
 }
 
diff --git a/src/sysdep.c b/src/sysdep.c
index 4f89e8aba1..beccc3c537 100644
--- a/src/sysdep.c
+++ b/src/sysdep.c
@@ -2767,6 +2767,35 @@ safe_strsignal (int code)
 
   return signame;
 }
+
+/* Output to stderr.  */
+
+void
+errputc (int c)
+{
+  fputc_unlocked (c, stderr);
+}
+
+void
+errputs (char const *str)
+{
+  fputs_unlocked (str, stderr);
+}
+
+void
+errprintf (char const *fmt, ...)
+{
+  va_list ap;
+  va_start (ap, fmt);
+  verrprintf (fmt, ap);
+  va_end (ap);
+}
+
+void
+verrprintf (char const *fmt, va_list ap)
+{
+  vfprintf (stderr, fmt, ap);
+}
 \f
 #ifndef DOS_NT
 /* For make-serial-process  */
diff --git a/src/sysstdio.h b/src/sysstdio.h
index 3ff1d6a572..ebe0845d40 100644
--- a/src/sysstdio.h
+++ b/src/sysstdio.h
@@ -25,6 +25,11 @@ #define EMACS_SYSSTDIO_H
 
 extern FILE *emacs_fopen (char const *, char const *);
 
+extern void errputc (int);
+extern void errputs (char const *);
+extern void errprintf (char const *, ...) ATTRIBUTE_FORMAT_PRINTF (1, 2);
+extern void verrprintf (char const *, va_list) ATTRIBUTE_FORMAT_PRINTF (1, 0);
+
 #if O_BINARY
 # define FOPEN_BINARY "b"
 # define FOPEN_TEXT "t"
diff --git a/src/systhread.c b/src/systhread.c
index 6f4de536fb..ef57c0e8b3 100644
--- a/src/systhread.c
+++ b/src/systhread.c
@@ -18,9 +18,10 @@ Copyright (C) 2012-2019 Free Software Foundation, Inc.
 
 #include <config.h>
 #include <setjmp.h>
-#include <stdio.h>
 #include <string.h>
+
 #include "lisp.h"
+#include "sysstdio.h"
 
 #ifdef HAVE_NS
 #include "nsterm.h"
@@ -122,7 +123,7 @@ sys_mutex_init (sys_mutex_t *mutex)
   /* We could get ENOMEM.  Can't do anything except aborting.  */
   if (error != 0)
     {
-      fprintf (stderr, "\npthread_mutex_init failed: %s\n", strerror (error));
+      errprintf ("\npthread_mutex_init failed: %s\n", strerror (error));
       emacs_abort ();
     }
 #ifdef ENABLE_CHECKING
@@ -152,7 +153,7 @@ sys_cond_init (sys_cond_t *cond)
   /* We could get ENOMEM.  Can't do anything except aborting.  */
   if (error != 0)
     {
-      fprintf (stderr, "\npthread_cond_init failed: %s\n", strerror (error));
+      errprintf ("\npthread_cond_init failed: %s\n", strerror (error));
       emacs_abort ();
     }
 }
diff --git a/src/term.c b/src/term.c
index 8b5a710d80..5b9c010520 100644
--- a/src/term.c
+++ b/src/term.c
@@ -4398,10 +4398,10 @@ init_tty (const char *name, const char *terminal_type, bool must_succeed)
 static void
 vfatal (const char *str, va_list ap)
 {
-  fprintf (stderr, "emacs: ");
-  vfprintf (stderr, str, ap);
+  errputs ("emacs: ");
+  verrprintf (str, ap);
   if (! (str[0] && str[strlen (str) - 1] == '\n'))
-    fprintf (stderr, "\n");
+    errputc ('\n');
   exit (1);
 }
 
diff --git a/src/unexaix.c b/src/unexaix.c
index 349d365383..a273ba1adb 100644
--- a/src/unexaix.c
+++ b/src/unexaix.c
@@ -225,12 +225,11 @@ make_hdr (int new, int a_out,
 #define CHECK_SCNHDR(ptr, name, flags) \
   if (strcmp (s->s_name, name) == 0) { \
     if (s->s_flags != flags) { \
-      fprintf (stderr, "unexec: %lx flags where %x expected in %s section.\n", \
-               (unsigned long)s->s_flags, flags, name);                 \
+      errprintf ("unexec: %lx flags where %x expected in %s section.\n", \
+		 (unsigned long) s->s_flags, flags, name);		\
     } \
     if (ptr) { \
-      fprintf (stderr, "unexec: duplicate section header for section %s.\n", \
-               name);                                                   \
+      errprintf ("unexec: duplicate section header for section %s.\n", name); \
     } \
     ptr = s; \
   }
diff --git a/src/unexelf.c b/src/unexelf.c
index 6d19bf1fb9..d09a5f5aac 100644
--- a/src/unexelf.c
+++ b/src/unexelf.c
@@ -187,7 +187,7 @@ verify ((! TYPE_SIGNED (ElfW (Half))
 	&& TYPE_MAXIMUM (ElfW (Half)) <= PTRDIFF_MAX);
 
 #ifdef UNEXELF_DEBUG
-# define DEBUG_LOG(expr) fprintf (stderr, #expr " 0x%jx\n", (uintmax_t) (expr))
+# define DEBUG_LOG(expr) errprintf (#expr " 0x%jx\n", (uintmax_t) (expr))
 #endif
 
 /* Get the address of a particular section or program header entry,
@@ -344,7 +344,7 @@ unexec (const char *new_name, const char *old_name)
   new_data2_offset = old_bss_offset;
 
 #ifdef UNEXELF_DEBUG
-  fprintf (stderr, "old_bss_index %td\n", old_bss_index);
+  errprintf ("old_bss_index %td\n", old_bss_index);
   DEBUG_LOG (old_bss_addr);
   DEBUG_LOG (old_bss_size);
   DEBUG_LOG (old_bss_offset);
@@ -395,9 +395,9 @@ unexec (const char *new_name, const char *old_name)
 
 #ifdef UNEXELF_DEBUG
   DEBUG_LOG (old_file_h->e_shoff);
-  fprintf (stderr, "Old section count %td\n", (ptrdiff_t) old_file_h->e_shnum);
+  errprintf ("Old section count %td\n", (ptrdiff_t) old_file_h->e_shnum);
   DEBUG_LOG (new_file_h->e_shoff);
-  fprintf (stderr, "New section count %td\n", (ptrdiff_t) new_file_h->e_shnum);
+  errprintf ("New section count %td\n", (ptrdiff_t) new_file_h->e_shnum);
 #endif
 
   /* Fix up program header.  Extend the writable data segment so
diff --git a/src/unexhp9k800.c b/src/unexhp9k800.c
index cbf1835b9e..48d516fa2f 100644
--- a/src/unexhp9k800.c
+++ b/src/unexhp9k800.c
@@ -167,7 +167,7 @@ read_header (int file, struct header *hdr, struct som_exec_auxhdr *auxhdr)
   if (hdr->a_magic != EXEC_MAGIC && hdr->a_magic != SHARE_MAGIC
       &&  hdr->a_magic != DEMAND_MAGIC)
     {
-      fprintf (stderr, "a.out file doesn't have valid magic number\n");
+      errputs ("a.out file doesn't have valid magic number\n");
       exit (1);
     }
 
diff --git a/src/unexmacosx.c b/src/unexmacosx.c
index a94c0cccb6..4c93301369 100644
--- a/src/unexmacosx.c
+++ b/src/unexmacosx.c
@@ -303,9 +303,9 @@ unexec_error (const char *format, ...)
   va_list ap;
 
   va_start (ap, format);
-  fprintf (stderr, "unexec: ");
-  vfprintf (stderr, format, ap);
-  fprintf (stderr, "\n");
+  errputs ("unexec: ");
+  verrprintf (format, ap);
+  errputc ('\n');
   va_end (ap);
   exit (1);
 }
diff --git a/src/widget.c b/src/widget.c
index e662dd3ecd..a150088c93 100644
--- a/src/widget.c
+++ b/src/widget.c
@@ -364,8 +364,7 @@ EmacsFrameInitialize (Widget request, Widget new, ArgList dum1, Cardinal *dum2)
 
   if (!ew->emacs_frame.frame)
     {
-      fprintf (stderr,
-	       "can't create an emacs frame widget without a frame\n");
+      errputs ("can't create an emacs frame widget without a frame\n");
       exit (1);
     }
 
diff --git a/src/xdisp.c b/src/xdisp.c
index c13a950e3a..4dbb183bd5 100644
--- a/src/xdisp.c
+++ b/src/xdisp.c
@@ -302,7 +302,6 @@ Copyright (C) 1985-1988, 1993-1995, 1997-2019 Free Software Foundation,
    buffer_posn_from_coords in dispnew.c for how this is handled.  */
 
 #include <config.h>
-#include <stdio.h>
 #include <stdlib.h>
 #include <limits.h>
 #include <math.h>
@@ -311,6 +310,7 @@ Copyright (C) 1985-1988, 1993-1995, 1997-2019 Free Software Foundation,
 #include "atimer.h"
 #include "composite.h"
 #include "keyboard.h"
+#include "sysstdio.h"
 #include "systime.h"
 #include "frame.h"
 #include "window.h"
@@ -1762,10 +1762,10 @@ pos_visible_p (struct window *w, ptrdiff_t charpos, int *x, int *y,
 #if false
   /* Debugging code.  */
   if (visible_p)
-    fprintf (stderr, "+pv pt=%d vs=%d --> x=%d y=%d rt=%d rb=%d rh=%d vp=%d\n",
-	     charpos, w->vscroll, *x, *y, *rtop, *rbot, *rowh, *vpos);
+    errprintf ("+pv pt=%d vs=%d --> x=%d y=%d rt=%d rb=%d rh=%d vp=%d\n",
+	       charpos, w->vscroll, *x, *y, *rtop, *rbot, *rowh, *vpos);
   else
-    fprintf (stderr, "-pv pt=%d vs=%d\n", charpos, w->vscroll);
+    errprintf ("-pv pt=%d vs=%d\n", charpos, w->vscroll);
 #endif
 
   /* Restore potentially overwritten values.  */
@@ -10713,7 +10713,7 @@ message_to_stderr (Lisp_Object m)
   if (noninteractive_need_newline)
     {
       noninteractive_need_newline = false;
-      fputc ('\n', stderr);
+      errputc ('\n');
     }
   if (STRINGP (m))
     {
@@ -10741,7 +10741,7 @@ message_to_stderr (Lisp_Object m)
       }
     }
   else if (!cursor_in_echo_area)
-    fputc ('\n', stderr);
+    errputc ('\n');
 
   fflush (stderr);
 }
@@ -10888,11 +10888,11 @@ vmessage (const char *m, va_list ap)
       if (m)
 	{
 	  if (noninteractive_need_newline)
-	    fputc ('\n', stderr);
+	    errputc ('\n');
 	  noninteractive_need_newline = false;
-	  vfprintf (stderr, m, ap);
+	  verrprintf (m, ap);
 	  if (!cursor_in_echo_area)
-	    fputc ('\n', stderr);
+	    errputc ('\n');
 	  fflush (stderr);
 	}
     }
@@ -13679,13 +13679,13 @@ debug_method_add (struct window *w, char const *fmt, ...)
   va_end (ap);
 
   if (trace_redisplay_p)
-    fprintf (stderr, "%p (%s): %s\n",
-	     ptr,
-	     ((BUFFERP (w->contents)
-	       && STRINGP (BVAR (XBUFFER (w->contents), name)))
-	      ? SSDATA (BVAR (XBUFFER (w->contents), name))
-	      : "no buffer"),
-	     method + len);
+    errprintf ("%p (%s): %s\n",
+	       ptr,
+	       ((BUFFERP (w->contents)
+		 && STRINGP (BVAR (XBUFFER (w->contents), name)))
+		? SSDATA (BVAR (XBUFFER (w->contents), name))
+		: "no buffer"),
+	       method + len);
 }
 
 #endif /* GLYPH_DEBUG */
@@ -19521,117 +19521,111 @@ dump_glyph (struct glyph_row *row, struct glyph *glyph, int area)
   if (glyph->type == CHAR_GLYPH
       || glyph->type == GLYPHLESS_GLYPH)
     {
-      fprintf (stderr,
-	       "  %5"pD"d     %c %9"pD"d   %c %3d 0x%06x      %c %4d %1.1d%1.1d\n",
-	       glyph - row->glyphs[TEXT_AREA],
-	       (glyph->type == CHAR_GLYPH
-		? 'C'
-		: 'G'),
-	       glyph->charpos,
-	       (BUFFERP (glyph->object)
-		? 'B'
-		: (STRINGP (glyph->object)
-		   ? 'S'
-		   : (NILP (glyph->object)
-		      ? '0'
-		      : '-'))),
-	       glyph->pixel_width,
-	       glyph->u.ch,
-	       (glyph->u.ch < 0x80 && glyph->u.ch >= ' '
-		? (int) glyph->u.ch
-		: '.'),
-	       glyph->face_id,
-	       glyph->left_box_line_p,
-	       glyph->right_box_line_p);
+      errprintf ("  %5"pD"d     %c %9"pD"d   %c %3d 0x%06x      %c %4d %1.1d%1.1d\n",
+		 glyph - row->glyphs[TEXT_AREA],
+		 (glyph->type == CHAR_GLYPH
+		  ? 'C'
+		  : 'G'),
+		 glyph->charpos,
+		 (BUFFERP (glyph->object)
+		  ? 'B'
+		  : (STRINGP (glyph->object)
+		     ? 'S'
+		     : (NILP (glyph->object)
+			? '0'
+			: '-'))),
+		 glyph->pixel_width,
+		 glyph->u.ch,
+		 (glyph->u.ch < 0x80 && glyph->u.ch >= ' '
+		  ? (int) glyph->u.ch
+		  : '.'),
+		 glyph->face_id,
+		 glyph->left_box_line_p,
+		 glyph->right_box_line_p);
     }
   else if (glyph->type == STRETCH_GLYPH)
     {
-      fprintf (stderr,
-	       "  %5"pD"d     %c %9"pD"d   %c %3d 0x%06x      %c %4d %1.1d%1.1d\n",
-	       glyph - row->glyphs[TEXT_AREA],
-	       'S',
-	       glyph->charpos,
-	       (BUFFERP (glyph->object)
-		? 'B'
-		: (STRINGP (glyph->object)
-		   ? 'S'
-		   : (NILP (glyph->object)
-		      ? '0'
-		      : '-'))),
-	       glyph->pixel_width,
-	       0u,
-	       ' ',
-	       glyph->face_id,
-	       glyph->left_box_line_p,
-	       glyph->right_box_line_p);
+      errprintf ("  %5"pD"d     %c %9"pD"d   %c %3d 0x%06x      %c %4d %1.1d%1.1d\n",
+		 glyph - row->glyphs[TEXT_AREA],
+		 'S',
+		 glyph->charpos,
+		 (BUFFERP (glyph->object)
+		  ? 'B'
+		  : (STRINGP (glyph->object)
+		     ? 'S'
+		     : (NILP (glyph->object)
+			? '0'
+			: '-'))),
+		 glyph->pixel_width,
+		 0u,
+		 ' ',
+		 glyph->face_id,
+		 glyph->left_box_line_p,
+		 glyph->right_box_line_p);
     }
   else if (glyph->type == IMAGE_GLYPH)
     {
-      fprintf (stderr,
-	       "  %5"pD"d     %c %9"pD"d   %c %3d 0x%06x      %c %4d %1.1d%1.1d\n",
-	       glyph - row->glyphs[TEXT_AREA],
-	       'I',
-	       glyph->charpos,
-	       (BUFFERP (glyph->object)
-		? 'B'
-		: (STRINGP (glyph->object)
-		   ? 'S'
-		   : (NILP (glyph->object)
-		      ? '0'
-		      : '-'))),
-	       glyph->pixel_width,
-	       (unsigned int) glyph->u.img_id,
-	       '.',
-	       glyph->face_id,
-	       glyph->left_box_line_p,
-	       glyph->right_box_line_p);
+      errprintf ("  %5"pD"d     %c %9"pD"d   %c %3d 0x%06x      %c %4d %1.1d%1.1d\n",
+		 glyph - row->glyphs[TEXT_AREA],
+		 'I',
+		 glyph->charpos,
+		 (BUFFERP (glyph->object)
+		  ? 'B'
+		  : (STRINGP (glyph->object)
+		     ? 'S'
+		     : (NILP (glyph->object)
+			? '0'
+			: '-'))),
+		 glyph->pixel_width,
+		 (unsigned int) glyph->u.img_id,
+		 '.',
+		 glyph->face_id,
+		 glyph->left_box_line_p,
+		 glyph->right_box_line_p);
     }
   else if (glyph->type == COMPOSITE_GLYPH)
     {
-      fprintf (stderr,
-	       "  %5"pD"d     %c %9"pD"d   %c %3d 0x%06x",
-	       glyph - row->glyphs[TEXT_AREA],
-	       '+',
-	       glyph->charpos,
-	       (BUFFERP (glyph->object)
-		? 'B'
-		: (STRINGP (glyph->object)
-		   ? 'S'
-		   : (NILP (glyph->object)
-		      ? '0'
-		      : '-'))),
-	       glyph->pixel_width,
-	       (unsigned int) glyph->u.cmp.id);
+      errprintf ("  %5"pD"d     %c %9"pD"d   %c %3d 0x%06x",
+		 glyph - row->glyphs[TEXT_AREA],
+		 '+',
+		 glyph->charpos,
+		 (BUFFERP (glyph->object)
+		  ? 'B'
+		  : (STRINGP (glyph->object)
+		     ? 'S'
+		     : (NILP (glyph->object)
+			? '0'
+			: '-'))),
+		 glyph->pixel_width,
+		 (unsigned int) glyph->u.cmp.id);
       if (glyph->u.cmp.automatic)
-	fprintf (stderr,
-		 "[%d-%d]",
-		 glyph->slice.cmp.from, glyph->slice.cmp.to);
-      fprintf (stderr, " . %4d %1.1d%1.1d\n",
-	       glyph->face_id,
-	       glyph->left_box_line_p,
-	       glyph->right_box_line_p);
+	errprintf ("[%d-%d]",
+		   glyph->slice.cmp.from, glyph->slice.cmp.to);
+      errprintf (" . %4d %1.1d%1.1d\n",
+		 glyph->face_id,
+		 glyph->left_box_line_p,
+		 glyph->right_box_line_p);
     }
   else if (glyph->type == XWIDGET_GLYPH)
     {
 #ifndef HAVE_XWIDGETS
       eassume (false);
 #else
-      fprintf (stderr,
-	       "  %5td %4c %6td %c %3d %7p %c %4d %1.1d%1.1d\n",
-	       glyph - row->glyphs[TEXT_AREA],
-	       'X',
-	       glyph->charpos,
-	       (BUFFERP (glyph->object)
-		? 'B'
-		: (STRINGP (glyph->object)
-		   ? 'S'
-		   : '-')),
-	       glyph->pixel_width,
-	       glyph->u.xwidget,
-	       '.',
-	       glyph->face_id,
-	       glyph->left_box_line_p,
-	       glyph->right_box_line_p);
+      errprintf ("  %5td %4c %6td %c %3d %7p %c %4d %1.1d%1.1d\n",
+		 glyph - row->glyphs[TEXT_AREA],
+		 'X',
+		 glyph->charpos,
+		 (BUFFERP (glyph->object)
+		  ? 'B'
+		  : (STRINGP (glyph->object)
+		     ? 'S'
+		     : '-')),
+		 glyph->pixel_width,
+		 glyph->u.xwidget,
+		 '.',
+		 glyph->face_id,
+		 glyph->left_box_line_p,
+		 glyph->right_box_line_p);
 #endif
     }
 }
@@ -19647,43 +19641,43 @@ dump_glyph_row (struct glyph_row *row, int vpos, int glyphs)
 {
   if (glyphs != 1)
     {
-      fprintf (stderr, "Row     Start       End Used oE><\\CTZFesm     X    Y    W    H    V    A    P\n");
-      fprintf (stderr, "==============================================================================\n");
+      errputs ("Row     Start       End Used oE><\\CTZFesm     X    Y    W    H    V    A    P\n");
+      errputs ("==============================================================================\n");
 
-      fprintf (stderr, "%3d %9"pD"d %9"pD"d %4d %1.1d%1.1d%1.1d%1.1d\
+      errprintf ("%3d %9"pD"d %9"pD"d %4d %1.1d%1.1d%1.1d%1.1d\
 %1.1d%1.1d%1.1d%1.1d%1.1d%1.1d%1.1d%1.1d  %4d %4d %4d %4d %4d %4d %4d\n",
-	       vpos,
-	       MATRIX_ROW_START_CHARPOS (row),
-	       MATRIX_ROW_END_CHARPOS (row),
-	       row->used[TEXT_AREA],
-	       row->contains_overlapping_glyphs_p,
-	       row->enabled_p,
-	       row->truncated_on_left_p,
-	       row->truncated_on_right_p,
-	       row->continued_p,
-	       MATRIX_ROW_CONTINUATION_LINE_P (row),
-	       MATRIX_ROW_DISPLAYS_TEXT_P (row),
-	       row->ends_at_zv_p,
-	       row->fill_line_p,
-	       row->ends_in_middle_of_char_p,
-	       row->starts_in_middle_of_char_p,
-	       row->mouse_face_p,
-	       row->x,
-	       row->y,
-	       row->pixel_width,
-	       row->height,
-	       row->visible_height,
-	       row->ascent,
-	       row->phys_ascent);
+		 vpos,
+		 MATRIX_ROW_START_CHARPOS (row),
+		 MATRIX_ROW_END_CHARPOS (row),
+		 row->used[TEXT_AREA],
+		 row->contains_overlapping_glyphs_p,
+		 row->enabled_p,
+		 row->truncated_on_left_p,
+		 row->truncated_on_right_p,
+		 row->continued_p,
+		 MATRIX_ROW_CONTINUATION_LINE_P (row),
+		 MATRIX_ROW_DISPLAYS_TEXT_P (row),
+		 row->ends_at_zv_p,
+		 row->fill_line_p,
+		 row->ends_in_middle_of_char_p,
+		 row->starts_in_middle_of_char_p,
+		 row->mouse_face_p,
+		 row->x,
+		 row->y,
+		 row->pixel_width,
+		 row->height,
+		 row->visible_height,
+		 row->ascent,
+		 row->phys_ascent);
       /* The next 3 lines should align to "Start" in the header.  */
-      fprintf (stderr, "    %9"pD"d %9"pD"d\t%5d\n", row->start.overlay_string_index,
-	       row->end.overlay_string_index,
-	       row->continuation_lines_width);
-      fprintf (stderr, "    %9"pD"d %9"pD"d\n",
-	       CHARPOS (row->start.string_pos),
-	       CHARPOS (row->end.string_pos));
-      fprintf (stderr, "    %9d %9d\n", row->start.dpvec_index,
-	       row->end.dpvec_index);
+      errprintf ("    %9"pD"d %9"pD"d\t%5d\n", row->start.overlay_string_index,
+		 row->end.overlay_string_index,
+		 row->continuation_lines_width);
+      errprintf ("    %9"pD"d %9"pD"d\n",
+		 CHARPOS (row->start.string_pos),
+		 CHARPOS (row->end.string_pos));
+      errprintf ("    %9d %9d\n", row->start.dpvec_index,
+		 row->end.dpvec_index);
     }
 
   if (glyphs > 1)
@@ -19700,7 +19694,7 @@ dump_glyph_row (struct glyph_row *row, int vpos, int glyphs)
 	    ++glyph_end;
 
 	  if (glyph < glyph_end)
-	    fprintf (stderr, " Glyph#  Type       Pos   O   W     Code      C Face LR\n");
+	    errputs (" Glyph#  Type       Pos   O   W     Code      C Face LR\n");
 
 	  for (; glyph < glyph_end; ++glyph)
 	    dump_glyph (row, glyph, area);
@@ -19736,7 +19730,7 @@ dump_glyph_row (struct glyph_row *row, int vpos, int glyphs)
 	    }
 
 	  s[i] = '\0';
-	  fprintf (stderr, "%3d: (%d) '%s'\n", vpos, row->enabled_p, s);
+	  errprintf ("%3d: (%d) '%s'\n", vpos, row->enabled_p, s);
 	}
     }
 }
@@ -19756,11 +19750,11 @@ DEFUN ("dump-glyph-matrix", Fdump_glyph_matrix,
   struct window *w = XWINDOW (selected_window);
   struct buffer *buffer = XBUFFER (w->contents);
 
-  fprintf (stderr, "PT = %"pD"d, BEGV = %"pD"d. ZV = %"pD"d\n",
-	   BUF_PT (buffer), BUF_BEGV (buffer), BUF_ZV (buffer));
-  fprintf (stderr, "Cursor x = %d, y = %d, hpos = %d, vpos = %d\n",
-	   w->cursor.x, w->cursor.y, w->cursor.hpos, w->cursor.vpos);
-  fprintf (stderr, "=============================================\n");
+  errprintf ("PT = %"pD"d, BEGV = %"pD"d. ZV = %"pD"d\n",
+	     BUF_PT (buffer), BUF_BEGV (buffer), BUF_ZV (buffer));
+  errprintf ("Cursor x = %d, y = %d, hpos = %d, vpos = %d\n",
+	     w->cursor.x, w->cursor.y, w->cursor.hpos, w->cursor.vpos);
+  errputs ("=============================================\n");
   dump_glyph_matrix (w->current_matrix,
 		     TYPE_RANGED_FIXNUMP (int, glyphs) ? XFIXNUM (glyphs) : 0);
   return Qnil;
@@ -19777,7 +19771,7 @@ DEFUN ("dump-frame-glyph-matrix", Fdump_frame_glyph_matrix,
   if (f->current_matrix)
     dump_glyph_matrix (f->current_matrix, 1);
   else
-    fprintf (stderr, "*** This frame doesn't have a frame glyph matrix ***\n");
+    errputs ("*** This frame doesn't have a frame glyph matrix ***\n");
   return Qnil;
 }
 
@@ -25973,18 +25967,18 @@ get_font_ascent_descent (struct font *font, int *ascent, int *descent)
 void
 dump_glyph_string (struct glyph_string *s)
 {
-  fprintf (stderr, "glyph string\n");
-  fprintf (stderr, "  x, y, w, h = %d, %d, %d, %d\n",
-	   s->x, s->y, s->width, s->height);
-  fprintf (stderr, "  ybase = %d\n", s->ybase);
-  fprintf (stderr, "  hl = %u\n", s->hl);
-  fprintf (stderr, "  left overhang = %d, right = %d\n",
-	   s->left_overhang, s->right_overhang);
-  fprintf (stderr, "  nchars = %d\n", s->nchars);
-  fprintf (stderr, "  extends to end of line = %d\n",
-	   s->extends_to_end_of_line_p);
-  fprintf (stderr, "  font height = %d\n", FONT_HEIGHT (s->font));
-  fprintf (stderr, "  bg width = %d\n", s->background_width);
+  errputs ("glyph string\n");
+  errprintf ("  x, y, w, h = %d, %d, %d, %d\n",
+	     s->x, s->y, s->width, s->height);
+  errprintf ("  ybase = %d\n", s->ybase);
+  errprintf ("  hl = %u\n", s->hl);
+  errprintf ("  left overhang = %d, right = %d\n",
+	     s->left_overhang, s->right_overhang);
+  errprintf ("  nchars = %d\n", s->nchars);
+  errprintf ("  extends to end of line = %d\n",
+	     s->extends_to_end_of_line_p);
+  errprintf ("  font height = %d\n", FONT_HEIGHT (s->font));
+  errprintf ("  bg width = %d\n", s->background_width);
 }
 
 #endif /* GLYPH_DEBUG */
diff --git a/src/xfaces.c b/src/xfaces.c
index d9e66eaf2d..ed3bb0945e 100644
--- a/src/xfaces.c
+++ b/src/xfaces.c
@@ -418,21 +418,15 @@ DEFUN ("dump-colors", Fdump_colors, Sdump_colors, 0, 0, 0,
 {
   int i, n;
 
-  fputc ('\n', stderr);
+  errputc ('\n');
 
   for (i = n = 0; i < ARRAYELTS (color_count); ++i)
     if (color_count[i])
-      {
-	fprintf (stderr, "%3d: %5d", i, color_count[i]);
-	++n;
-	if (n % 5 == 0)
-	  fputc ('\n', stderr);
-	else
-	  fputc ('\t', stderr);
-      }
+      errprintf ("%3d: %5d%c", i, color_count[i],
+		 ++n % 5 == 0 ? '\n' : '\t');
 
   if (n % 5 != 0)
-    fputc ('\n', stderr);
+    errputc ('\n');
   return Qnil;
 }
 
@@ -6397,28 +6391,28 @@ DEFUN ("x-load-color-file", Fx_load_color_file,
 static void
 dump_realized_face (struct face *face)
 {
-  fprintf (stderr, "ID: %d\n", face->id);
+  errprintf ("ID: %d\n", face->id);
 #ifdef HAVE_X_WINDOWS
-  fprintf (stderr, "gc: %p\n", face->gc);
+  errprintf ("gc: %p\n", face->gc);
 #endif
-  fprintf (stderr, "foreground: 0x%lx (%s)\n",
-	   face->foreground,
-	   SDATA (face->lface[LFACE_FOREGROUND_INDEX]));
-  fprintf (stderr, "background: 0x%lx (%s)\n",
-	   face->background,
-	   SDATA (face->lface[LFACE_BACKGROUND_INDEX]));
+  errprintf ("foreground: 0x%lx (%s)\n",
+	     face->foreground,
+	     SDATA (face->lface[LFACE_FOREGROUND_INDEX]));
+  errprintf ("background: 0x%lx (%s)\n",
+	     face->background,
+	     SDATA (face->lface[LFACE_BACKGROUND_INDEX]));
   if (face->font)
-    fprintf (stderr, "font_name: %s (%s)\n",
-	     SDATA (face->font->props[FONT_NAME_INDEX]),
-	     SDATA (face->lface[LFACE_FAMILY_INDEX]));
+    errprintf ("font_name: %s (%s)\n",
+	       SDATA (face->font->props[FONT_NAME_INDEX]),
+	       SDATA (face->lface[LFACE_FAMILY_INDEX]));
 #ifdef HAVE_X_WINDOWS
-  fprintf (stderr, "font = %p\n", face->font);
+  errprintf ("font = %p\n", face->font);
 #endif
-  fprintf (stderr, "fontset: %d\n", face->fontset);
-  fprintf (stderr, "underline: %d (%s)\n",
-	   face->underline_p,
-	   SDATA (Fsymbol_name (face->lface[LFACE_UNDERLINE_INDEX])));
-  fprintf (stderr, "hash: %" PRIuPTR "\n", face->hash);
+  errprintf ("fontset: %d\n", face->fontset);
+  errprintf ("underline: %d (%s)\n",
+	     face->underline_p,
+	     SDATA (Fsymbol_name (face->lface[LFACE_UNDERLINE_INDEX])));
+  errprintf ("hash: %" PRIuPTR "\n", face->hash);
 }
 
 
@@ -6429,14 +6423,14 @@ DEFUN ("dump-face", Fdump_face, Sdump_face, 0, 1, 0, doc: /* */)
     {
       int i;
 
-      fprintf (stderr, "font selection order: ");
+      errputs ("font selection order: ");
       for (i = 0; i < ARRAYELTS (font_sort_order); ++i)
-	fprintf (stderr, "%d ", font_sort_order[i]);
-      fprintf (stderr, "\n");
+	errprintf ("%d ", font_sort_order[i]);
+      errputc ('\n');
 
-      fprintf (stderr, "alternative fonts: ");
+      errputs ("alternative fonts: ");
       debug_print (Vface_alternative_font_family_alist);
-      fprintf (stderr, "\n");
+      errputc ('\n');
 
       for (i = 0; i < FRAME_FACE_CACHE (SELECTED_FRAME ())->used; ++i)
 	Fdump_face (make_fixnum (i));
@@ -6459,9 +6453,9 @@ DEFUN ("show-face-resources", Fshow_face_resources, Sshow_face_resources,
        0, 0, 0, doc: /* */)
   (void)
 {
-  fprintf (stderr, "number of colors = %d\n", ncolors_allocated);
-  fprintf (stderr, "number of pixmaps = %d\n", npixmaps_allocated);
-  fprintf (stderr, "number of GCs = %d\n", ngcs);
+  errprintf ("number of colors = %d\n", ncolors_allocated);
+  errprintf ("number of pixmaps = %d\n", npixmaps_allocated);
+  errprintf ("number of GCs = %d\n", ngcs);
   return Qnil;
 }
 
diff --git a/src/xfns.c b/src/xfns.c
index b8a1914186..24c01ace58 100644
--- a/src/xfns.c
+++ b/src/xfns.c
@@ -2384,13 +2384,13 @@ print_fontset_result (XFontSet xfs, char *name, char **missing_list,
 		      int missing_count)
 {
   if (xfs)
-    fprintf (stderr, "XIC Fontset created: %s\n", name);
+    errprintf ("XIC Fontset created: %s\n", name);
   else
     {
-      fprintf (stderr, "XIC Fontset failed: %s\n", name);
+      errprintf ("XIC Fontset failed: %s\n", name);
       while (missing_count-- > 0)
 	{
-	  fprintf (stderr, "  missing: %s\n", *missing_list);
+	  errprintf ("  missing: %s\n", *missing_list);
 	  missing_list++;
 	}
     }
diff --git a/src/xmenu.c b/src/xmenu.c
index 22d1cc21aa..2f952f363e 100644
--- a/src/xmenu.c
+++ b/src/xmenu.c
@@ -2321,7 +2321,7 @@ x_menu_show (struct frame *f, int x, int y, int menuflags,
     {
     case XM_SUCCESS:
 #ifdef XDEBUG
-      fprintf (stderr, "pane= %d line = %d\n", panes, selidx);
+      errprintf ("pane= %d line = %d\n", panes, selidx);
 #endif
 
       /* Find the item number SELIDX in pane number PANE.  */
diff --git a/src/xrdb.c b/src/xrdb.c
index 9c625e9821..4dbc7cee30 100644
--- a/src/xrdb.c
+++ b/src/xrdb.c
@@ -589,7 +589,7 @@ member (char *elt, List list)
 static void
 fatal (char *msg, char *prog)
 {
-  fprintf (stderr, msg, prog);
+  errprintf (msg, prog);
   exit (1);
 }
 
diff --git a/src/xselect.c b/src/xselect.c
index 5f0bb44cc9..6268e74d8e 100644
--- a/src/xselect.c
+++ b/src/xselect.c
@@ -21,7 +21,6 @@
 
 #include <config.h>
 #include <limits.h>
-#include <stdio.h>      /* termhooks.h needs this */
 
 #ifdef HAVE_SYS_TYPES_H
 #include <sys/types.h>
@@ -33,6 +32,7 @@
 #include "xterm.h"	/* for all of the X includes */
 #include "frame.h"	/* Need this to get the X window of selected_frame */
 #include "blockinput.h"
+#include "sysstdio.h"
 #include "termhooks.h"
 #include "keyboard.h"
 #include "pdumper.h"
@@ -63,13 +63,13 @@
 
 #ifdef TRACE_SELECTION
 #define TRACE0(fmt) \
-  fprintf (stderr, "%"pMd": " fmt "\n", (printmax_t) getpid ())
+  errprintf ("%"pMd": " fmt "\n", (printmax_t) getpid ())
 #define TRACE1(fmt, a0) \
-  fprintf (stderr, "%"pMd": " fmt "\n", (printmax_t) getpid (), a0)
+  errprintf ("%"pMd": " fmt "\n", (printmax_t) getpid (), a0)
 #define TRACE2(fmt, a0, a1) \
-  fprintf (stderr, "%"pMd": " fmt "\n", (printmax_t) getpid (), a0, a1)
+  errprintf ("%"pMd": " fmt "\n", (printmax_t) getpid (), a0, a1)
 #define TRACE3(fmt, a0, a1, a2) \
-  fprintf (stderr, "%"pMd": " fmt "\n", (printmax_t) getpid (), a0, a1, a2)
+  errprintf ("%"pMd": " fmt "\n", (printmax_t) getpid (), a0, a1, a2)
 #else
 #define TRACE0(fmt)		(void) 0
 #define TRACE1(fmt, a0)		(void) 0
@@ -2172,7 +2172,7 @@ x_clipboard_manager_error_1 (Lisp_Object err)
 static Lisp_Object
 x_clipboard_manager_error_2 (Lisp_Object err)
 {
-  fprintf (stderr, "Error saving to X clipboard manager.\n\
+  errprintf ("Error saving to X clipboard manager.\n\
 If the problem persists, set '%s' \
 to nil.\n", "x-select-enable-clipboard-manager");
   return Qnil;
diff --git a/src/xsmfns.c b/src/xsmfns.c
index 1706cddf27..e2e544516c 100644
--- a/src/xsmfns.c
+++ b/src/xsmfns.c
@@ -29,10 +29,10 @@ Copyright (C) 2002-2019 Free Software Foundation, Inc.
 #include <unistd.h>
 #include <sys/param.h>
 #include <errno.h>
-#include <stdio.h>
 
 #include "lisp.h"
 #include "frame.h"
+#include "sysstdio.h"
 #include "termhooks.h"
 #include "xterm.h"
 #include "process.h"
@@ -404,8 +404,8 @@ #define SM_ERRORSTRING_LEN 512
   char *pwd = emacs_get_current_dir_name ();
   if (!pwd)
     {
-      fprintf (stderr, "Disabling session management due to pwd error: %s\n",
-               emacs_strerror (errno));
+      errprintf ("Disabling session management due to pwd error: %s\n",
+		 emacs_strerror (errno));
       return;
     }
   xfree (pwd);
diff --git a/src/xterm.c b/src/xterm.c
index 38bc17de97..7fea1946eb 100644
--- a/src/xterm.c
+++ b/src/xterm.c
@@ -21,12 +21,12 @@ Copyright (C) 1989, 1993-2019 Free Software Foundation, Inc.
 /* Xt features made by Fred Pierresteguy.  */
 
 #include <config.h>
-#include <stdio.h>
 #include <stdlib.h>
 #include <math.h>
 
 #include "lisp.h"
 #include "blockinput.h"
+#include "sysstdio.h"
 
 /* This may include sys/types.h, and that somehow loses
    if this is not done before the other system files.  */
@@ -9876,7 +9876,7 @@ x_fully_uncatch_errors (void)
 static unsigned int x_wire_count;
 x_trace_wire (void)
 {
-  fprintf (stderr, "Lib call: %d\n", ++x_wire_count);
+  errprintf ("Lib call: %d\n", ++x_wire_count);
 }
 #endif
 
@@ -9954,11 +9954,11 @@ x_connection_closed (Display *dpy, const char *error_message, bool ioerror)
 	 the resulting Glib error message loop filled a user's disk.
 	 To avoid this, kill Emacs unconditionally on disconnect.  */
       shut_down_emacs (0, Qnil);
-      fprintf (stderr, "%s\n\
+      errprintf ("%s\n\
 When compiled with GTK, Emacs cannot recover from X disconnects.\n\
 This is a GTK bug: https://gitlab.gnome.org/GNOME/gtk/issues/221\n\
 For details, see etc/PROBLEMS.\n",
-	       error_msg);
+		 error_msg);
       emacs_abort ();
 #endif /* USE_GTK */
 
@@ -9980,7 +9980,7 @@ x_connection_closed (Display *dpy, const char *error_message, bool ioerror)
 
   if (terminal_list == 0)
     {
-      fprintf (stderr, "%s\n", error_msg);
+      errprintf ("%s\n", error_msg);
       Fkill_emacs (make_fixnum (70));
       /* NOTREACHED */
     }
@@ -12465,7 +12465,7 @@ my_log_handler (const gchar *log_domain, GLogLevelFlags log_level,
 		const gchar *msg, gpointer user_data)
 {
   if (!strstr (msg, "g_set_prgname"))
-      fprintf (stderr, "%s-WARNING **: %s\n", log_domain, msg);
+    errprintf ("%s-WARNING **: %s\n", log_domain, msg);
 }
 #endif
 
@@ -13441,8 +13441,7 @@ x_initialize (void)
 #if THREADS_ENABLED
   /* This must be called before any other Xlib routines.  */
   if (XInitThreads () == 0)
-    fprintf (stderr,
-	     "Warning: An error occurred initializing X11 thread support!\n");
+    errputs ("Warning: An error occurred initializing X11 thread support!\n");
 #endif
 
 #ifdef USE_X_TOOLKIT
-- 
2.21.0


[-- Attachment #3: 0002-Avoid-interleaving-stderr-in-most-of-C-code-Emacs.patch --]
[-- Type: text/x-patch, Size: 5955 bytes --]

From 560b8b7cc2ad44f2d2a6e8d43432ed8b28c14e7e Mon Sep 17 00:00:00 2001
From: Paul Eggert <eggert@cs.ucla.edu>
Date: Tue, 2 Jul 2019 23:44:12 -0700
Subject: [PATCH 2/4] Avoid interleaving stderr in most of C-code Emacs

To avoid interleaving stderr messages with those of other processes,
line-buffer C-level diagnostics separately where this is safe,
i.e., when various parts of a diagnostic are output right away.
This change does not affect stderr buffering;
that is, the stderr stream still uses its default buffering,
which is typically unbuffered (although POSIX allows line buffering).
* src/pdumper.c (print_paths_to_root_1):
Use errwrite instead of fwrite to stderr; this avoids interleaving.
* src/sysdep.c (buferr): New static var.
(init_standard_fds) [!DOS_NT]: Initialize it.
(errstream): New function.
(errputc, errputs, verrprintf): Use it.
(errwrite): New function.
* src/xdisp.c (message_to_stderr):
Use errwrite instead of fwrite to stderr; this avoids interleaving
in a simpler and better way than allocating a temporary heap buffer.
* src/xdisp.c (vmessage): Simplify; make it more like message_to_stderr.
---
 src/pdumper.c  |  2 +-
 src/sysdep.c   | 49 ++++++++++++++++++++++++++++++++++++++++++++-----
 src/sysstdio.h |  1 +
 src/xdisp.c    | 27 +++++++--------------------
 4 files changed, 53 insertions(+), 26 deletions(-)

diff --git a/src/pdumper.c b/src/pdumper.c
index bfa0dba7f0..ff092e2e49 100644
--- a/src/pdumper.c
+++ b/src/pdumper.c
@@ -1404,7 +1404,7 @@ print_paths_to_root_1 (struct dump_context *ctx,
       referrers = XCDR (referrers);
       Lisp_Object repr = Fprin1_to_string (referrer, Qnil);
       errprintf ("%*s", level, "");
-      fwrite (SDATA (repr), 1, SBYTES (repr), stderr);
+      errwrite (SDATA (repr), SBYTES (repr));
       errputc ('\n');
       print_paths_to_root_1 (ctx, referrer, level + 1);
     }
diff --git a/src/sysdep.c b/src/sysdep.c
index beccc3c537..2bd7f1afd2 100644
--- a/src/sysdep.c
+++ b/src/sysdep.c
@@ -231,8 +231,13 @@ force_open (int fd, int flags)
     }
 }
 
+/* A stream that is like stderr, except line buffered when possible.
+   It is NULL during startup or if initializing it failed.  */
+static FILE *buferr;
+
 /* Make sure stdin, stdout, and stderr are open to something, so that
-   their file descriptors are not hijacked by later system calls.  */
+   their file descriptors are not hijacked by later system calls.
+   Initialize buferr too.  */
 void
 init_standard_fds (void)
 {
@@ -243,6 +248,15 @@ init_standard_fds (void)
   force_open (STDIN_FILENO, O_WRONLY);
   force_open (STDOUT_FILENO, O_RDONLY);
   force_open (STDERR_FILENO, O_RDONLY);
+
+#ifndef DOS_NT /* _IOLBF does not work on MS-Windows.  */
+  buferr = fdopen (STDERR_FILENO, "w");
+  if (buferr && setvbuf (buferr, NULL, _IOLBF, 0) != 0)
+    {
+      fclose (buferr);
+      buferr = NULL;
+    }
+#endif
 }
 
 /* Return the current working directory.  The result should be freed
@@ -2767,19 +2781,38 @@ safe_strsignal (int code)
 
   return signame;
 }
+\f
+/* Return the error output stream.  */
 
-/* Output to stderr.  */
+static FILE *
+errstream (void)
+{
+  FILE *err = buferr;
+  if (!err)
+    return stderr;
+  fflush_unlocked (stderr);
+  return err;
+}
+
+/* These functions are like fputc, fputs, fprintf, vfprintf, and
+   fwrite, except that they output to stderr and buffer better on
+   platforms that support line buffering.  This avoids interleaving
+   output when Emacs and other processes write to stderr
+   simultaneously, so long as the lines are short enough.  When a
+   single diagnostic is emitted via a sequence of calls of one or more
+   of these functions, the caller should arrange for the last called
+   function to output a newline at the end.  */
 
 void
 errputc (int c)
 {
-  fputc_unlocked (c, stderr);
+  fputc_unlocked (c, errstream ());
 }
 
 void
 errputs (char const *str)
 {
-  fputs_unlocked (str, stderr);
+  fputs_unlocked (str, errstream ());
 }
 
 void
@@ -2794,7 +2827,13 @@ errprintf (char const *fmt, ...)
 void
 verrprintf (char const *fmt, va_list ap)
 {
-  vfprintf (stderr, fmt, ap);
+  vfprintf (errstream (), fmt, ap);
+}
+
+void
+errwrite (void const *buf, ptrdiff_t nbuf)
+{
+  fwrite_unlocked (buf, 1, nbuf, errstream ());
 }
 \f
 #ifndef DOS_NT
diff --git a/src/sysstdio.h b/src/sysstdio.h
index ebe0845d40..87568c86fd 100644
--- a/src/sysstdio.h
+++ b/src/sysstdio.h
@@ -29,6 +29,7 @@ #define EMACS_SYSSTDIO_H
 extern void errputs (char const *);
 extern void errprintf (char const *, ...) ATTRIBUTE_FORMAT_PRINTF (1, 2);
 extern void verrprintf (char const *, va_list) ATTRIBUTE_FORMAT_PRINTF (1, 0);
+extern void errwrite (void const *, ptrdiff_t);
 
 #if O_BINARY
 # define FOPEN_BINARY "b"
diff --git a/src/xdisp.c b/src/xdisp.c
index 4dbb183bd5..12ab7d9c03 100644
--- a/src/xdisp.c
+++ b/src/xdisp.c
@@ -10727,23 +10727,10 @@ message_to_stderr (Lisp_Object m)
       else
 	s = m;
 
-      /* We want to write this out with a single fwrite call so that
-	 output doesn't interleave with other processes writing to
-	 stderr at the same time. */
-      {
-	int length = min (INT_MAX, SBYTES (s) + 1);
-	char *string = xmalloc (length);
-
-	memcpy (string, SSDATA (s), length - 1);
-	string[length - 1] = '\n';
-	fwrite (string, 1, length, stderr);
-	xfree (string);
-      }
+      errwrite (SDATA (s), SBYTES (s));
     }
-  else if (!cursor_in_echo_area)
+  if (STRINGP (m) || !cursor_in_echo_area)
     errputc ('\n');
-
-  fflush (stderr);
 }
 
 /* The non-logging version of message3.
@@ -10888,12 +10875,12 @@ vmessage (const char *m, va_list ap)
       if (m)
 	{
 	  if (noninteractive_need_newline)
-	    errputc ('\n');
-	  noninteractive_need_newline = false;
+	    {
+	      noninteractive_need_newline = false;
+	      errputc ('\n');
+	    }
 	  verrprintf (m, ap);
-	  if (!cursor_in_echo_area)
-	    errputc ('\n');
-	  fflush (stderr);
+	  errputc ('\n');
 	}
     }
   else if (INTERACTIVE)
-- 
2.21.0


[-- Attachment #4: 0003-Avoid-interleaving-stderr-in-most-of-Lisp-code-Emacs.patch --]
[-- Type: text/x-patch, Size: 7488 bytes --]

From 7350126db02071f7a1dbf2406082c395f9d5073d Mon Sep 17 00:00:00 2001
From: Paul Eggert <eggert@cs.ucla.edu>
Date: Tue, 2 Jul 2019 23:44:12 -0700
Subject: [PATCH 3/4] Avoid interleaving stderr in most of Lisp-code Emacs

* lisp/startup.el (command-line):
* src/keyboard.c (Fcommand_error_default_function):
* src/print.c (debug_print):
Use line-buffered-debugging-output.
* src/print.c (Fline_buffered_debugging_output): New function.
(init_print_once): Define it.
* src/sysdep.c (errstream): Now extern.
---
 doc/lispref/streams.texi | 15 +++++++++++++--
 etc/NEWS                 |  6 ++++++
 lisp/startup.el          |  6 +++---
 src/keyboard.c           |  4 ++--
 src/print.c              | 21 +++++++++++++++++----
 src/sysdep.c             |  2 +-
 src/sysstdio.h           |  1 +
 7 files changed, 43 insertions(+), 12 deletions(-)

diff --git a/doc/lispref/streams.texi b/doc/lispref/streams.texi
index 600639f244..761c7d40f0 100644
--- a/doc/lispref/streams.texi
+++ b/doc/lispref/streams.texi
@@ -535,11 +535,21 @@ Output Streams
 @defun external-debugging-output character
 This function can be useful as an output stream when debugging.  It
 writes @var{character} to the standard error stream.
+It uses the system default buffering (either unbuffered or line-buffered).
+@end defun
+
+@anchor{line-buffered-debugging-output}
+@defun line-buffered-debugging-output character
+This function can be useful as an output stream when debugging.
+It writes @var{character} to the standard error stream.
+It uses line buffering if available, as this works better than
+unbuffered when processes other than Emacs are simultaneously writing
+to the error stream.
 
 For example
 @example
 @group
-(print "This is the output" #'external-debugging-output)
+(print "This is the output" #'line-buffered-debugging-output)
 @print{} This is the output
 @result{} "This is the output"
 @end group
@@ -587,7 +597,8 @@ Output Functions
 
   In the functions below, @var{stream} stands for an output stream.
 (See the previous section for a description of output streams.  Also
-@xref{external-debugging-output}, a useful stream value for debugging.)
+@xref{external-debugging-output} and @xref{line-buffered-debugging-output},
+useful stream values for debugging.)
 If @var{stream} is @code{nil} or omitted, it defaults to the value of
 @code{standard-output}.
 
diff --git a/etc/NEWS b/etc/NEWS
index abbece374a..1dfec126ed 100644
--- a/etc/NEWS
+++ b/etc/NEWS
@@ -2031,6 +2031,12 @@ between two strings.
 ** 'print-quoted' now defaults to t, so if you want to see
 '(quote x)' instead of 'x you will have to bind it to nil where applicable.
 
++++
+** New function line-buffered-debugging-output which outputs to the
+standard error stream using line-buffering than available.  This works
+better than external-debugging-output when processes other than Emacs
+are simultaneously writing to the standard error stream.
+
 +++
 ** Numbers formatted via '%o' or '%x' are now formatted as signed integers.
 This avoids problems in calls like '(read (format "#x%x" -1))', and is
diff --git a/lisp/startup.el b/lisp/startup.el
index 7759ed5aed..977591c02a 100644
--- a/lisp/startup.el
+++ b/lisp/startup.el
@@ -1016,7 +1016,7 @@ command-line
     ;; Although in most usage we are going to cryptically abort a moment
     ;; later anyway, due to missing required bidi data files (eg bug#13430).
     (if (null simple-file-name)
-	(let ((standard-output 'external-debugging-output)
+	(let ((standard-output 'line-buffered-debugging-output)
 	      (lispdir (expand-file-name "../lisp" data-directory)))
 	  (princ "Warning: Could not find simple.el or simple.elc")
 	  (terpri)
@@ -1244,8 +1244,8 @@ command-line
                   (get (car error) 'error-message)
                   (mapconcat (lambda (obj) (prin1-to-string obj t))
                              (cdr error) ", "))))
-      'external-debugging-output)
-     (terpri 'external-debugging-output)
+      'line-buffered-debugging-output)
+     (terpri 'line-buffered-debugging-output)
      (setq initial-window-system nil)
      (kill-emacs)))
 
diff --git a/src/keyboard.c b/src/keyboard.c
index 56916e0cb4..c635372d15 100644
--- a/src/keyboard.c
+++ b/src/keyboard.c
@@ -1014,9 +1014,9 @@ DEFUN ("command-error-default-function", Fcommand_error_default_function,
 	   || (!IS_DAEMON && FRAME_INITIAL_P (sf))
 	   || noninteractive)
     {
-      print_error_message (data, Qexternal_debugging_output,
+      print_error_message (data, Qline_buffered_debugging_output,
 			   SSDATA (context), signal);
-      Fterpri (Qexternal_debugging_output, Qnil);
+      Fterpri (Qline_buffered_debugging_output, Qnil);
       Fkill_emacs (make_fixnum (-1));
     }
   else
diff --git a/src/print.c b/src/print.c
index df45cb27ce..012c535644 100644
--- a/src/print.c
+++ b/src/print.c
@@ -771,7 +771,7 @@ DEFUN ("print", Fprint, Sprint, 1, 2, 0,
 }
 
 DEFUN ("external-debugging-output", Fexternal_debugging_output, Sexternal_debugging_output, 1, 1, 0,
-       doc: /* Write CHARACTER to stderr.
+       doc: /* Write CHARACTER to stderr using system-dependent buffering.
 You can call `print' while debugging emacs, and pass it this function
 to make it write to the debugging output.  */)
   (Lisp_Object character)
@@ -781,6 +781,18 @@ DEFUN ("external-debugging-output", Fexternal_debugging_output, Sexternal_debugg
   return character;
 }
 
+DEFUN ("line-buffered-debugging-output", Fline_buffered_debugging_output,
+       Sline_buffered_debugging_output, 1, 1, 0,
+       doc: /* Write CHARACTER to stderr using line-buffering if possible.
+You can call `print' while debugging emacs, and pass it this function
+to make it write to the debugging output.  */)
+  (Lisp_Object character)
+{
+  CHECK_FIXNUM (character);
+  printchar_to_stream (XFIXNUM (character), errstream ());
+  return character;
+}
+
 /* This function is never called.  Its purpose is to prevent
    print_output_debug_flag from being optimized away.  */
 
@@ -839,7 +851,7 @@ DEFUN ("redirect-debugging-output", Fredirect_debugging_output, Sredirect_debugg
 void
 debug_print (Lisp_Object arg)
 {
-  Fprin1 (arg, Qexternal_debugging_output);
+  Fprin1 (arg, Qline_buffered_debugging_output);
   errputs ("\r\n");
 }
 
@@ -2206,11 +2218,12 @@ print_interval (INTERVAL interval, Lisp_Object printcharfun)
 void
 init_print_once (void)
 {
-  /* The subroutine object for external-debugging-output is kept here
-     for the convenience of the debugger.  */
+  /* These subroutine objects are kept here for debugger convenience.  */
   DEFSYM (Qexternal_debugging_output, "external-debugging-output");
+  DEFSYM (Qline_buffered_debugging_output, "line-buffered-debugging-output");
 
   defsubr (&Sexternal_debugging_output);
+  defsubr (&Sline_buffered_debugging_output);
 }
 
 void
diff --git a/src/sysdep.c b/src/sysdep.c
index 2bd7f1afd2..cf7699d43e 100644
--- a/src/sysdep.c
+++ b/src/sysdep.c
@@ -2784,7 +2784,7 @@ safe_strsignal (int code)
 \f
 /* Return the error output stream.  */
 
-static FILE *
+FILE *
 errstream (void)
 {
   FILE *err = buferr;
diff --git a/src/sysstdio.h b/src/sysstdio.h
index 87568c86fd..966a5abd7f 100644
--- a/src/sysstdio.h
+++ b/src/sysstdio.h
@@ -25,6 +25,7 @@ #define EMACS_SYSSTDIO_H
 
 extern FILE *emacs_fopen (char const *, char const *);
 
+extern FILE *errstream (void);
 extern void errputc (int);
 extern void errputs (char const *);
 extern void errprintf (char const *, ...) ATTRIBUTE_FORMAT_PRINTF (1, 2);
-- 
2.21.0


[-- Attachment #5: 0004-Simplify-emacs_perror.patch --]
[-- Type: text/x-patch, Size: 1834 bytes --]

From 6adaa706cbf4da6c45782b5107a6a3fa423c8487 Mon Sep 17 00:00:00 2001
From: Paul Eggert <eggert@cs.ucla.edu>
Date: Tue, 2 Jul 2019 23:44:12 -0700
Subject: [PATCH 4/4] Simplify emacs_perror

* src/sysdep.c (emacs_perror): Simplify by using errprintf.
---
 src/sysdep.c | 19 ++-----------------
 1 file changed, 2 insertions(+), 17 deletions(-)

diff --git a/src/sysdep.c b/src/sysdep.c
index cf7699d43e..640d4908e1 100644
--- a/src/sysdep.c
+++ b/src/sysdep.c
@@ -2701,7 +2701,7 @@ emacs_write_quit (int fd, void const *buf, ptrdiff_t nbyte)
 }
 
 /* Write a diagnostic to standard error that contains MESSAGE and a
-   string derived from errno.  Preserve errno.  Do not buffer stderr.
+   string derived from errno.  Preserve errno.  Flush the output.
    Do not process quits or pending signals if interrupted.  */
 void
 emacs_perror (char const *message)
@@ -2710,22 +2710,7 @@ emacs_perror (char const *message)
   char const *error_string = emacs_strerror (err);
   char const *command = (initial_argv && initial_argv[0]
 			 ? initial_argv[0] : "emacs");
-  /* Write it out all at once, if it's short; this is less likely to
-     be interleaved with other output.  */
-  char buf[BUFSIZ];
-  int nbytes = snprintf (buf, sizeof buf, "%s: %s: %s\n",
-			 command, message, error_string);
-  if (0 <= nbytes && nbytes < BUFSIZ)
-    emacs_write (STDERR_FILENO, buf, nbytes);
-  else
-    {
-      emacs_write (STDERR_FILENO, command, strlen (command));
-      emacs_write (STDERR_FILENO, ": ", 2);
-      emacs_write (STDERR_FILENO, message, strlen (message));
-      emacs_write (STDERR_FILENO, ": ", 2);
-      emacs_write (STDERR_FILENO, error_string, strlen (error_string));
-      emacs_write (STDERR_FILENO, "\n", 1);
-    }
+  errprintf ("%s: %s: %s\n", command, message, error_string);
   errno = err;
 }
 \f
-- 
2.21.0

Re: `message' not outputting the newline "atomically"

[Eli Zaretskii]

> Cc: daniele@grinta.net, emacs-devel@gnu.org
> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Wed, 3 Jul 2019 00:31:18 -0700
> 
> Eli Zaretskii wrote:
> > I'm okay with making 'message' write in one go to stderr, but I don't
> > want to pay the price of having stderr buffered globally.
> 
> I came up with a fix that does all that. Although the fix does not alter the 
> buffering of the stderr stream, it causes 'message' and similar functions to 
> write in one go to avoid interleaving output. It also fixes the problem on AIX 
> and Solaris where a single call to fprintf is implemented by multiple calls to 
> 'write' even if the diagnostic is short, botching interleaving. It also fixes 
> the INT_MAX overflow and memory-allocation issues of the current master's 
> implementation of 'message'.

Whoa!  Is it really worth all that?  Including exposing this to Lisp?

> +#ifndef DOS_NT /* _IOLBF does not work on MS-Windows.  */

This should be WINDOWSNT, as the problem is specific to the
MS-Windows runtime.

Re: `message' not outputting the newline "atomically"

[Eli Zaretskii]

> Date: Wed, 03 Jul 2019 10:41:22 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: larsi@gnus.org, daniele@grinta.net, emacs-devel@gnu.org
> 
> > Cc: daniele@grinta.net, emacs-devel@gnu.org
> > From: Paul Eggert <eggert@cs.ucla.edu>
> > Date: Wed, 3 Jul 2019 00:31:18 -0700
> > 
> > Eli Zaretskii wrote:
> > > I'm okay with making 'message' write in one go to stderr, but I don't
> > > want to pay the price of having stderr buffered globally.
> > 
> > I came up with a fix that does all that. Although the fix does not alter the 
> > buffering of the stderr stream, it causes 'message' and similar functions to 
> > write in one go to avoid interleaving output. It also fixes the problem on AIX 
> > and Solaris where a single call to fprintf is implemented by multiple calls to 
> > 'write' even if the diagnostic is short, botching interleaving. It also fixes 
> > the INT_MAX overflow and memory-allocation issues of the current master's 
> > implementation of 'message'.
> 
> Whoa!  Is it really worth all that?  Including exposing this to Lisp?

Actually, this patch goes way too far.  It makes _all_ (or maybe
almost all) writes to stderr use the new line-buffered stream, which
is the same as making stderr itself line-buffered.

I only agreed to making 'message' write its text in one go.  Let's
stick to that, please.

Re: `message' not outputting the newline "atomically"

[Eli Zaretskii]

> Date: Wed, 03 Jul 2019 10:47:14 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: larsi@gnus.org, daniele@grinta.net, emacs-devel@gnu.org
> 
> Actually, this patch goes way too far.  It makes _all_ (or maybe
> almost all) writes to stderr use the new line-buffered stream, which
> is the same as making stderr itself line-buffered.
> 
> I only agreed to making 'message' write its text in one go.  Let's
> stick to that, please.

Let me clarify: the reason I don't want to make stderr buffered is
that I want all our diagnostic/debug messages to go outside without
buffering as much as possible.  So please let's not consider patches
that switch all those diagnostics to use new code which buffers its
output, because that is exactly why I do NOT want stderr become
buffered.

The 'message' function is another matter, as it frequently used for
purposes other than debugging, and usually from Lisp, not from C.

Re: `message' not outputting the newline "atomically"

[Paul Eggert]

Eli Zaretskii wrote:
> I want all our diagnostic/debug messages to go outside without
> buffering as much as possible.

To do that we would have to outlaw plain fprintf too, as it buffers on GNU/Linux 
and most other platforms. For example:

    char *p = "hello";
    char *q = (char *) 1;
    fprintf (stderr, "p=%s q=%s\n", p, q);

This dumps core on GNU/Linux without producing any output. It would be possible 
to do "better" than that, by writing our own fprintf variant that does not do 
any buffering, and so outputs "p=hello q=" before it dumps core. However, this 
would be counterproductive overkill, just as it's counterproductive overkill to 
insist that stderr must not be line-buffered.

Instead, we should assume that fprintf works as advertised and that Emacs 
doesn't pass bad pointers to fprintf, and we should worry about any buffering 
issues only in the rare cases where we actually have them -- e.g., because a 
stray pointer has corrupted memory, which is something that can screw up *any* 
of these approaches.

I know you prefer unbuffered stderr for debugging, so here's a simple 
compromise: let's leave stderr unbuffered if ENABLE_DEBUG is defined. This would 
let us keep the diagnostic-interleaving fix down to three lines, while still 
having the unbuffered stderr that you prefer when you are debugging something.

 > Whoa!  Is it really worth all that?  Including exposing this to Lisp?

I thought it was worth it, or I wouldn't have written the patches. 'message' is 
not the only place where Emacs issues diagnostics that can interleave, and it 
shouldn't be hard to fix the other places.

Re: `message' not outputting the newline "atomically"

[Eli Zaretskii]

> Cc: larsi@gnus.org, daniele@grinta.net, emacs-devel@gnu.org
> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Wed, 3 Jul 2019 01:45:51 -0700
> 
> Eli Zaretskii wrote:
> > I want all our diagnostic/debug messages to go outside without
> > buffering as much as possible.
> 
> To do that we would have to outlaw plain fprintf too, as it buffers on GNU/Linux 
> and most other platforms. For example:
> 
>     char *p = "hello";
>     char *q = (char *) 1;
>     fprintf (stderr, "p=%s q=%s\n", p, q);
> 
> This dumps core on GNU/Linux without producing any output.

This dumps core because the invalid pointer is dereferenced before
fprintf is called, right?  If so, this is not relevant to the issue at
hand; just because there's a call to fprintf somewhere near the crash
locus doesn't yet mean there's a problem with that fprintf.

I said "as much as possible", and I meant that.  I don't see any
reasons to outlaw fprintf, just because some platforms buffer stderr,
or because Emacs might crash without even starting to print anything,
that'd be too unreasonable.

But I don't want us to invent a whole new family of functions just to
avoid fprintf, like your patch did.

> I know you prefer unbuffered stderr for debugging, so here's a simple 
> compromise: let's leave stderr unbuffered if ENABLE_DEBUG is defined.

Sorry, no: these messages are important in production builds as well
(those which aren't should be under some DEBUG cpp conditional
already).  Even out of people who track the master branch there seem
to be a fair amount of those who don't build with --enable-checking.

>  > Whoa!  Is it really worth all that?  Including exposing this to Lisp?
> 
> I thought it was worth it, or I wouldn't have written the patches. 'message' is 
> not the only place where Emacs issues diagnostics that can interleave, and it 
> shouldn't be hard to fix the other places.

What other places are those?  Let's discuss them one by one, please.

`message' not outputting the newline "atomically"

[Paul Eggert]

[-- Attachment #1: Type: text/plain, Size: 3170 bytes --]

>>      char *p = "hello";
>>      char *q = (char *) 1;
>>      fprintf (stderr, "p=%s q=%s\n", p, q);
>>
>> This dumps core on GNU/Linux without producing any output.
> 
> This dumps core because the invalid pointer is dereferenced before
> fprintf is called, right?

No, the caller does not dereference the bad pointer. And on some 
platforms fprintf outputs the "hello" before dumping core.

> I said "as much as possible", and I meant that.

As the above example shows, it's quite possible to output more than what 
GNU/Linux fprintf does. So, if "as much as possible" is a hard 
requirement, Emacs must stop using fprintf to format diagnostics like 
the above.

Which would be absurd and I'm not seriously proposing it. However, it is 
logically implied by the axiom that diagnostics must be unbuffered as 
much as possible. That axiom is therefore untenable as stated.

> I don't see any
> reasons to outlaw fprintf, just because some platforms buffer stderr,
> or because Emacs might crash without even starting to print anything,
> that'd be too unreasonable.

I agree. Similarly, there is no reason to require Emacs code to use a 
single fprintf to output each diagnostic line. Instead, we should allow 
code like this:

    fprintf (stderr, "emacs debug [");
    for (i = 0; i < 10; i++)
      fprintf (stderr, " %d", a[i]);
    fprintf (stderr, "]\n");

That is, there should not be a need to for code like this to use special 
output functions that are not used for ordinary diagnostics.

> I don't want us to invent a whole new family of functions just to
> avoid fprintf, like your patch did.

I agree with this as well. My patch went to that length only because it 
was the best way I saw to fix the problem under the unreasonable 
constraint that the stderr stream cannot be line-buffered. There are 
other ways to fix the problem, such as avoiding stdio and using only 
'write' for diagnostics, but these other ways are messier. As far as I 
know, nobody has suggested any patch to fix the problem other than the 
three-line patch I originally submitted, or the recent set of 
more-complicated patches.

> these messages are important in production builds as well

Of course they're important! However, the three-line change doesn't lose 
these messages in realistic scenarios involving current Emacs code. I 
proposed its ENABLE_DEBUG variant as a compromise to allay theoretical 
concerns that the three-line change might make debugging harder, not 
because conditionalizing on ENABLE_DEBUG would make a significant 
practical difference.

>> I thought it was worth it, or I wouldn't have written the patches. 'message' is
>> not the only place where Emacs issues diagnostics that can interleave, and it
>> shouldn't be hard to fix the other places.
> 
> What other places are those?  Let's discuss them one by one, please.

To help get that discussion started I am attaching a list of grep hits 
for some of these places in the current master source (commit 
619592df9ed4d54a63f653bf6912ecc44f46dc59). This is only a partial list. 
Although there are lots more where this came from, coming up with a 
complete list would take quite a bit more time.

[-- Attachment #2: fprintf-list.out --]
[-- Type: text/plain, Size: 6965 bytes --]

alloc.c:7192:  fprintf (stderr, "\r\n%s:%d: Emacs fatal error: assertion failed: %s\r\n",
bidi.c:3592:  fprintf (stderr, "Total of  %"pD"d state%s in cache:\n",
bidi.c:3599:    fprintf (stderr, "%*c", ndigits, bidi_cache[i].ch);
bidi.c:3603:    fprintf (stderr, "%*d", ndigits, bidi_cache[i].resolved_level);
bidi.c:3607:    fprintf (stderr, "%*"pD"d", ndigits, bidi_cache[i].charpos);
buffer.c:4784:    fprintf (stderr, "munmap: %s\n", emacs_strerror (errno));
buffer.c:4803:	fprintf (stderr, "munmap: %s\n", emacs_strerror (errno));
buffer.c:4832:		fprintf (stderr, "munmap: %s\n", emacs_strerror (errno));
buffer.c:4870:	fprintf (stderr, "mmap: %s\n", emacs_strerror (errno));
buffer.c:5381:      fprintf (stderr, "Error getting directory: %s\n",
dispnew.c:223:      fprintf (stderr, "%s\n", redisplay_history[i].trace);
dispnew.c:6156:	  fprintf (stderr, "Display %s unavailable, simulating -nw\n",
emacs.c:1195:          fprintf (stderr, "%s: Can't chdir to %s: %s\n",
emacs.c:1320:	      fprintf (stderr, "%s: %s: %s\n", argv[0], term, errstring);
emacs.c:1325:	      fprintf (stderr, "%s: %s: not a tty\n", argv[0], term);
emacs.c:1328:	  fprintf (stderr, "Using %s\n", term);
emacs.c:1549:          fprintf (stderr, "Couldn't create MS-Windows event for daemon: %s\n",
emacs.c:2332:		  fprintf (stderr, "Option '%s' matched multiple standard arguments\n", argv[from]);
emacs-module.c:1305:  vfprintf (stderr, format, args);
gmalloc.c:2015:  fprintf (stderr, "mcheck: %s\n", msg);
gtkutil.c:832:    fprintf (stderr, "XX %s-WARNING **: %s\n", log_domain, msg);
gtkutil.c:897:	    fprintf (stderr, "Failed to parse: '%s'\n", geom_str);
gtkutil.c:2853:      fprintf (stderr, "bad type in xg_create_widget: %s, doing nothing\n",
lread.c:4724:  fprintf (stderr, format, use, SSDATA (ENCODE_SYSTEM (dirname)), diagnostic);
msdos.c:3859:  fprintf (stderr, "%s not yet implemented\r\n", badfunc);
pdumper.c:329:  fprintf (stderr, "%s: ", label);
pdumper.c:331:    fprintf (stderr, "%02x", (unsigned) xfingerprint[i]);
print.c:857:      fprintf (stderr, "#<%s_LISP_OBJECT 0x%08"pI"x>\r\n",
regex-emacs.c:445:  if (regex_emacs_debug > 0) fprintf (stderr, __VA_ARGS__)
regex-emacs.c:460:      fprintf (stderr, "{%02x}", uc);
regex-emacs.c:485:	      fprintf (stderr, "-");
regex-emacs.c:518:	  fprintf (stderr, "/no_op");
regex-emacs.c:522:	  fprintf (stderr, "/succeed");
regex-emacs.c:527:	  fprintf (stderr, "/exactn/%d", mcnt);
regex-emacs.c:530:	      fprintf (stderr, "/");
regex-emacs.c:537:	  fprintf (stderr, "/start_memory/%d", *p++);
regex-emacs.c:541:	  fprintf (stderr, "/stop_memory/%d", *p++);
regex-emacs.c:545:	  fprintf (stderr, "/duplicate/%d", *p++);
regex-emacs.c:549:	  fprintf (stderr, "/anychar");
regex-emacs.c:560:	    fprintf (stderr, "/charset [%s",
regex-emacs.c:564:	      fprintf (stderr, " !extends past end of pattern! ");
regex-emacs.c:573:		      fprintf (stderr, "-");
regex-emacs.c:592:	    fprintf (stderr, "]");
regex-emacs.c:599:		fprintf (stderr, "has-range-table");
regex-emacs.c:610:	  fprintf (stderr, "/begline");
regex-emacs.c:614:	  fprintf (stderr, "/endline");
regex-emacs.c:619:	  fprintf (stderr, "/on_failure_jump to %td", p + mcnt - start);
regex-emacs.c:624:	  fprintf (stderr, "/on_failure_keep_string_jump to %td",
regex-emacs.c:630:	  fprintf (stderr, "/on_failure_jump_nastyloop to %td",
regex-emacs.c:636:	  fprintf (stderr, "/on_failure_jump_loop to %td",
regex-emacs.c:642:	  fprintf (stderr, "/on_failure_jump_smart to %td",
regex-emacs.c:648:	  fprintf (stderr, "/jump to %td", p + mcnt - start);
regex-emacs.c:654:	  fprintf (stderr, "/succeed_n to %td, %d times",
regex-emacs.c:661:	  fprintf (stderr, "/jump_n to %td, %d times",
regex-emacs.c:668:	  fprintf (stderr, "/set_number_at location %td to %d",
regex-emacs.c:673:	  fprintf (stderr, "/wordbound");
regex-emacs.c:677:	  fprintf (stderr, "/notwordbound");
regex-emacs.c:681:	  fprintf (stderr, "/wordbeg");
regex-emacs.c:685:	  fprintf (stderr, "/wordend");
regex-emacs.c:689:	  fprintf (stderr, "/symbeg");
regex-emacs.c:693:	  fprintf (stderr, "/symend");
regex-emacs.c:697:	  fprintf (stderr, "/syntaxspec");
regex-emacs.c:699:	  fprintf (stderr, "/%d", mcnt);
regex-emacs.c:703:	  fprintf (stderr, "/notsyntaxspec");
regex-emacs.c:705:	  fprintf (stderr, "/%d", mcnt);
regex-emacs.c:709:	  fprintf (stderr, "/at_dot");
regex-emacs.c:713:	  fprintf (stderr, "/categoryspec");
regex-emacs.c:715:	  fprintf (stderr, "/%d", mcnt);
regex-emacs.c:719:	  fprintf (stderr, "/notcategoryspec");
regex-emacs.c:721:	  fprintf (stderr, "/%d", mcnt);
regex-emacs.c:725:	  fprintf (stderr, "/begbuf");
regex-emacs.c:729:	  fprintf (stderr, "/endbuf");
regex-emacs.c:733:	  fprintf (stderr, "?%d", *(p-1));
regex-emacs.c:754:      fprintf (stderr, "fastmap: ");
regex-emacs.c:771:    fprintf (stderr, "(null)");
systhread.c:125:      fprintf (stderr, "\npthread_mutex_init failed: %s\n", strerror (error));
systhread.c:155:      fprintf (stderr, "\npthread_cond_init failed: %s\n", strerror (error));
term.c:4401:  fprintf (stderr, "emacs: ");
term.c:4402:  vfprintf (stderr, str, ap);
unexaix.c:228:      fprintf (stderr, "unexec: %lx flags where %x expected in %s section.\n", \
unexaix.c:232:      fprintf (stderr, "unexec: duplicate section header for section %s.\n", \
unexmacosx.c:306:  fprintf (stderr, "unexec: ");
unexmacosx.c:307:  vfprintf (stderr, format, ap);
xdisp.c:10893:	  vfprintf (stderr, m, ap);
xdisp.c:13682:    fprintf (stderr, "%p (%s): %s\n",
xdisp.c:19739:	  fprintf (stderr, "%3d: (%d) '%s'\n", vpos, row->enabled_p, s);
xfaces.c:426:	fprintf (stderr, "%3d: %5d", i, color_count[i]);
xfaces.c:6404:  fprintf (stderr, "foreground: 0x%lx (%s)\n",
xfaces.c:6407:  fprintf (stderr, "background: 0x%lx (%s)\n",
xfaces.c:6411:    fprintf (stderr, "font_name: %s (%s)\n",
xfaces.c:6418:  fprintf (stderr, "underline: %d (%s)\n",
xfaces.c:6432:      fprintf (stderr, "font selection order: ");
xfaces.c:6434:	fprintf (stderr, "%d ", font_sort_order[i]);
xfaces.c:6437:      fprintf (stderr, "alternative fonts: ");
xfns.c:2387:    fprintf (stderr, "XIC Fontset created: %s\n", name);
xfns.c:2390:      fprintf (stderr, "XIC Fontset failed: %s\n", name);
xfns.c:2393:	  fprintf (stderr, "  missing: %s\n", *missing_list);
xrdb.c:592:  fprintf (stderr, msg, prog);
xsmfns.c:407:      fprintf (stderr, "Disabling session management due to pwd error: %s\n",
xterm.c:9957:      fprintf (stderr, "%s\n\
xterm.c:9983:      fprintf (stderr, "%s\n", error_msg);
xterm.c:12468:      fprintf (stderr, "%s-WARNING **: %s\n", log_domain, msg);
nsfont.m:545:	fprintf (stderr, "nsfont: %s for fontspec:\n    ",
nsfont.m:1475:  fprintf (stderr, "Glyph string len = %d at (%d, %d) overhang (%d, %d),"
nsfont.m:1482:      fprintf (stderr, "%c", c);
nsmenu.m:239:          fprintf (stderr, "ERROR: did not find lisp menu for submenu '%s'.\n",
nsterm.m:5035:      else fprintf (stderr,
nsterm.m:5225:          fprintf (stderr, "Failed to create pipe: %s\n",

Re: `message' not outputting the newline "atomically"

[Eli Zaretskii]

> From: Paul Eggert <eggert@cs.ucla.edu>
> Cc: larsi@gnus.org, daniele@grinta.net, emacs-devel@gnu.org
> Date: Wed, 3 Jul 2019 16:08:36 -0700
> 
> >>      char *p = "hello";
> >>      char *q = (char *) 1;
> >>      fprintf (stderr, "p=%s q=%s\n", p, q);
> >>
> >> This dumps core on GNU/Linux without producing any output.
> > 
> > This dumps core because the invalid pointer is dereferenced before
> > fprintf is called, right?
> 
> No, the caller does not dereference the bad pointer.

Ah, you are right.  I've misread the code.

> And on some platforms fprintf outputs the "hello" before dumping
> core.

Then that's all I'd expect from such a printf.

> > I said "as much as possible", and I meant that.
> 
> As the above example shows, it's quite possible to output more than what 
> GNU/Linux fprintf does. So, if "as much as possible" is a hard 
> requirement, Emacs must stop using fprintf to format diagnostics like 
> the above.

I meant as much as possible by just using fprintf.

> Which would be absurd and I'm not seriously proposing it. However, it is 
> logically implied by the axiom that diagnostics must be unbuffered as 
> much as possible. That axiom is therefore untenable as stated.

Yes, which is why I was certain what I wrote will not be interpreted
in this way.

> > I don't see any
> > reasons to outlaw fprintf, just because some platforms buffer stderr,
> > or because Emacs might crash without even starting to print anything,
> > that'd be too unreasonable.
> 
> I agree. Similarly, there is no reason to require Emacs code to use a 
> single fprintf to output each diagnostic line. Instead, we should allow 
> code like this:
> 
>     fprintf (stderr, "emacs debug [");
>     for (i = 0; i < 10; i++)
>       fprintf (stderr, " %d", a[i]);
>     fprintf (stderr, "]\n");

Sure.

> That is, there should not be a need to for code like this to use special 
> output functions that are not used for ordinary diagnostics.

Right.

However, using the above for diagnostic output, or for informative
messages output during an Emacs build, is unrelated to the original
problem raised by Lars.  That problem was with messages from a running
Emacs, outside of the build process (because the build process use
case can be handled by the Make's -O switch), i.e. when Emacs is run
as part of some script.

> >> I thought it was worth it, or I wouldn't have written the patches. 'message' is
> >> not the only place where Emacs issues diagnostics that can interleave, and it
> >> shouldn't be hard to fix the other places.
> > 
> > What other places are those?  Let's discuss them one by one, please.
> 
> To help get that discussion started I am attaching a list of grep hits 
> for some of these places in the current master source (commit 
> 619592df9ed4d54a63f653bf6912ecc44f46dc59). This is only a partial list. 
> Although there are lots more where this came from, coming up with a 
> complete list would take quite a bit more time.

Thanks.  I reviewed the list, and I think only message3_nolog and
vmessage need to be changed.  All the rest are fatal error messages
and debugging diagnostics of sorts, and quite a few of them even
conditioned by some CPP symbol that is only defined in debugging
builds.

One way of fixing message3_nolog and vmessage would be to use in them
that additional line-buffer stream you defined in your patch.

Re: `message' not outputting the newline "atomically"

[Paul Eggert]

[-- Attachment #1: Type: text/plain, Size: 4024 bytes --]

Eli Zaretskii wrote:

> One way of fixing message3_nolog and vmessage would be to use in them
> that additional line-buffer stream you defined in your patch.

That would be easy enough to do; see the attached patch. However, although it's 
good to fix the problem for these two functions, we should also fix the problem 
where it occurs elsewhere. Please see below for more about this.

>> As the above example shows, it's quite possible to output more than what
>> GNU/Linux fprintf does. So, if "as much as possible" is a hard
>> requirement, Emacs must stop using fprintf to format diagnostics like
>> the above.
> 
> I meant as much as possible by just using fprintf.

I expect that's not precisely what you meant, as it's quite possible to just use 
fprintf to output more than what Emacs currently would do if Emacs had a serious 
bug in its error-printing code. For example, we could write our own printf-like 
function that does its actual work via 'fprintf (stderr, "%c" C)' where C is the 
next byte to output. That would just use fprintf for I/O, and on GNU/Linux it 
would output more than what we're doing now, for hypothetical buggy code such as 
the example I gave.

What I *think* you meant is that you want code that outputs diagnostics 
conveniently, via functions like 'fprintf' and 'message' and 'fatal' etc., to 
output as much as possible within the constraints of (a) not significantly 
obstructing convenience and (b) fixing the line-interleaving bugs somehow.

I could be wrong, though.

> I was certain what I wrote will not be interpreted
> in this way.

I'm afraid that you were mistaken there. Even now I'm not sure I understand 
exactly what you meant. It is an area where it's tough to be precise.

> using the above for diagnostic output, or for informative
> messages output during an Emacs build, is unrelated to the original
> problem raised by Lars.  That problem was with messages from a running
> Emacs, outside of the build process (because the build process use
> case can be handled by the Make's -O switch), i.e. when Emacs is run
> as part of some script.

Sorry, I'm not following you here, as Lars's original email 
<https://lists.gnu.org/r/emacs-devel/2019-06/msg00710.html> was about messages 
generated by Emacs running inside the build process.

We cannot rely on 'make -O' to address the problem, as 'make -O' slows down 
development, which is why I typically don't use 'make -O' and don't recommend it 
for building Emacs interactively. 'make -O' can be useful for buildbots where 
there's a batch process that waits for build completion before publishing it. 
However, under an interactive Emacs where I want to act on the first diagnostic 
right away (before the build finishes), 'make -O' is a net minus because it can 
delay diagnostics significantly, and this delay costs more than it's worth.

And even if we assumed 'make -O' sufficed for builds (which it doesn't), we 
can't assume these diagnostics are generated from a single instance of GNU Make. 
It's reasonably common, for example, that from a terminal window I'll run the 
shell command "emacs &", and then do other stuff in that window while Emacs 
occasionally outputs stderr diagnostics. (Although I think this is the sort of 
thing you're alluding to above, I'm not sure.) So we would have an interleaving 
problem anyway even if 'make -O' worked well.

> I reviewed the list, and I think only message3_nolog and
> vmessage need to be changed.  All the rest are fatal error messages
> and debugging diagnostics of sorts, and quite a few of them even
> conditioned by some CPP symbol that is only defined in debugging
> builds.

Unfortunately even fatal error messages and debugging diagnostics can be output 
in circumstances such as the above, and so can be interleaved confusingly. So 
the other items in that list still need fixing. Plus, there are other 
interleaved-I/O problems not in that list; as I mentioned, the list is only 
partial. So the attached patch does not suffice to fix the problem.

[-- Attachment #2: 0001-Avoid-interleaving-stderr-in-a-few-cases.patch --]
[-- Type: text/x-patch, Size: 5942 bytes --]

From b88da6aca8d12d0aee9703e54eee4f8a97c4d297 Mon Sep 17 00:00:00 2001
From: Paul Eggert <eggert@cs.ucla.edu>
Date: Sat, 6 Jul 2019 17:25:24 -0700
Subject: [PATCH] Avoid interleaving stderr in a few cases

* src/sysdep.c (buferr): New static var.
(init_standard_fds) [!DOS_NT]: Initialize it.
(errstream, errputc, verrprintf, errwrite): New functions.
(close_output_streams): Close buferr too.
* src/xdisp.c: Include sysstdio.h instead of stdio.h.
(message_to_stderr, vmessage): Use the new functions
to avoid interleaving stderr.
---
 src/sysdep.c   | 60 +++++++++++++++++++++++++++++++++++++++++++++++---
 src/sysstdio.h |  3 +++
 src/xdisp.c    | 35 +++++++++--------------------
 3 files changed, 71 insertions(+), 27 deletions(-)

diff --git a/src/sysdep.c b/src/sysdep.c
index 48eebb594f..ee17e08482 100644
--- a/src/sysdep.c
+++ b/src/sysdep.c
@@ -232,6 +232,10 @@ force_open (int fd, int flags)
     }
 }
 
+/* A stream that is like stderr, except line buffered when possible.
+   It is NULL during startup or if initializing it failed.  */
+static FILE *buferr;
+
 /* Make sure stdin, stdout, and stderr are open to something, so that
    their file descriptors are not hijacked by later system calls.  */
 void
@@ -244,6 +248,15 @@ init_standard_fds (void)
   force_open (STDIN_FILENO, O_WRONLY);
   force_open (STDOUT_FILENO, O_RDONLY);
   force_open (STDERR_FILENO, O_RDONLY);
+
+#ifndef DOS_NT /* _IOLBF does not work on MS-Windows.  */
+  buferr = fdopen (STDERR_FILENO, "w");
+  if (buferr && setvbuf (buferr, NULL, _IOLBF, 0) != 0)
+    {
+      fclose (buferr);
+      buferr = NULL;
+    }
+#endif
 }
 
 /* Return the current working directory.  The result should be freed
@@ -2769,6 +2782,46 @@ safe_strsignal (int code)
   return signame;
 }
 \f
+/* Output to stderr.  */
+
+/* Return the error output stream.  */
+static FILE *
+errstream (void)
+{
+  FILE *err = buferr;
+  if (!err)
+    return stderr;
+  fflush_unlocked (stderr);
+  return err;
+}
+
+/* These functions are like fputc, vfprintf, and fwrite,
+   except that they output to stderr and buffer better on
+   platforms that support line buffering.  This avoids interleaving
+   output when Emacs and other processes write to stderr
+   simultaneously, so long as the lines are short enough.  When a
+   single diagnostic is emitted via a sequence of calls of one or more
+   of these functions, the caller should arrange for the last called
+   function to output a newline at the end.  */
+
+void
+errputc (int c)
+{
+  fputc_unlocked (c, errstream ());
+}
+
+void
+verrprintf (char const *fmt, va_list ap)
+{
+  vfprintf (errstream (), fmt, ap);
+}
+
+void
+errwrite (void const *buf, ptrdiff_t nbuf)
+{
+  fwrite_unlocked (buf, 1, nbuf, errstream ());
+}
+
 /* Close standard output and standard error, reporting any write
    errors as best we can.  This is intended for use with atexit.  */
 void
@@ -2782,9 +2835,10 @@ close_output_streams (void)
 
   /* Do not close stderr if addresses are being sanitized, as the
      sanitizer might report to stderr after this function is invoked.  */
-  if (ADDRESS_SANITIZER
-      ? fflush_unlocked (stderr) != 0 || ferror (stderr)
-      : close_stream (stderr) != 0)
+  bool err = buferr && (fflush_unlocked (buferr) != 0 || ferror (buferr));
+  if (err | (ADDRESS_SANITIZER
+	     ? fflush_unlocked (stderr) != 0 || ferror (stderr)
+	     : close_stream (stderr) != 0))
     _exit (EXIT_FAILURE);
 }
 \f
diff --git a/src/sysstdio.h b/src/sysstdio.h
index a2364c4e1f..4dc9287e43 100644
--- a/src/sysstdio.h
+++ b/src/sysstdio.h
@@ -24,6 +24,9 @@ #define EMACS_SYSSTDIO_H
 #include <stdio.h>
 
 extern FILE *emacs_fopen (char const *, char const *);
+extern void errputc (int);
+extern void verrprintf (char const *, va_list) ATTRIBUTE_FORMAT_PRINTF (1, 0);
+extern void errwrite (void const *, ptrdiff_t);
 extern void close_output_streams (void);
 
 #if O_BINARY
diff --git a/src/xdisp.c b/src/xdisp.c
index 5fb690e746..85cac3cd29 100644
--- a/src/xdisp.c
+++ b/src/xdisp.c
@@ -302,7 +302,6 @@ Copyright (C) 1985-1988, 1993-1995, 1997-2019 Free Software Foundation,
    buffer_posn_from_coords in dispnew.c for how this is handled.  */
 
 #include <config.h>
-#include <stdio.h>
 #include <stdlib.h>
 #include <limits.h>
 #include <math.h>
@@ -311,6 +310,7 @@ Copyright (C) 1985-1988, 1993-1995, 1997-2019 Free Software Foundation,
 #include "atimer.h"
 #include "composite.h"
 #include "keyboard.h"
+#include "sysstdio.h"
 #include "systime.h"
 #include "frame.h"
 #include "window.h"
@@ -10713,7 +10713,7 @@ message_to_stderr (Lisp_Object m)
   if (noninteractive_need_newline)
     {
       noninteractive_need_newline = false;
-      fputc ('\n', stderr);
+      errputc ('\n');
     }
   if (STRINGP (m))
     {
@@ -10727,23 +10727,10 @@ message_to_stderr (Lisp_Object m)
       else
 	s = m;
 
-      /* We want to write this out with a single fwrite call so that
-	 output doesn't interleave with other processes writing to
-	 stderr at the same time. */
-      {
-	int length = min (INT_MAX, SBYTES (s) + 1);
-	char *string = xmalloc (length);
-
-	memcpy (string, SSDATA (s), length - 1);
-	string[length - 1] = '\n';
-	fwrite (string, 1, length, stderr);
-	xfree (string);
-      }
+      errwrite (SDATA (s), SBYTES (s));
     }
-  else if (!cursor_in_echo_area)
-    fputc ('\n', stderr);
-
-  fflush (stderr);
+  if (STRINGP (m) || !cursor_in_echo_area)
+    errputc ('\n');
 }
 
 /* The non-logging version of message3.
@@ -10888,12 +10875,12 @@ vmessage (const char *m, va_list ap)
       if (m)
 	{
 	  if (noninteractive_need_newline)
-	    fputc ('\n', stderr);
-	  noninteractive_need_newline = false;
-	  vfprintf (stderr, m, ap);
-	  if (!cursor_in_echo_area)
-	    fputc ('\n', stderr);
-	  fflush (stderr);
+	    {
+	      noninteractive_need_newline = false;
+	      errputc ('\n');
+	    }
+	  verrprintf (m, ap);
+	  errputc ('\n');
 	}
     }
   else if (INTERACTIVE)
-- 
2.21.0

Re: `message' not outputting the newline "atomically"

[Eli Zaretskii]

> Cc: larsi@gnus.org, daniele@grinta.net, emacs-devel@gnu.org
> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Sat, 6 Jul 2019 18:16:03 -0700
> 
> What I *think* you meant is that you want code that outputs diagnostics 
> conveniently, via functions like 'fprintf' and 'message' and 'fatal' etc., to 
> output as much as possible within the constraints of (a) not significantly 
> obstructing convenience and (b) fixing the line-interleaving bugs somehow.

Yes, that's what I meant.

> > I was certain what I wrote will not be interpreted
> > in this way.
> 
> I'm afraid that you were mistaken there. Even now I'm not sure I understand 
> exactly what you meant. It is an area where it's tough to be precise.

Sorry, I'm trying.

> > using the above for diagnostic output, or for informative
> > messages output during an Emacs build, is unrelated to the original
> > problem raised by Lars.  That problem was with messages from a running
> > Emacs, outside of the build process (because the build process use
> > case can be handled by the Make's -O switch), i.e. when Emacs is run
> > as part of some script.
> 
> Sorry, I'm not following you here, as Lars's original email 
> <https://lists.gnu.org/r/emacs-devel/2019-06/msg00710.html> was about messages 
> generated by Emacs running inside the build process.
> 
> We cannot rely on 'make -O' to address the problem, as 'make -O' slows down 
> development, which is why I typically don't use 'make -O' and don't recommend it 
> for building Emacs interactively. 'make -O' can be useful for buildbots where 
> there's a batch process that waits for build completion before publishing it. 
> However, under an interactive Emacs where I want to act on the first diagnostic 
> right away (before the build finishes), 'make -O' is a net minus because it can 
> delay diagnostics significantly, and this delay costs more than it's worth.
> 
> And even if we assumed 'make -O' sufficed for builds (which it doesn't), we 
> can't assume these diagnostics are generated from a single instance of GNU Make. 
> It's reasonably common, for example, that from a terminal window I'll run the 
> shell command "emacs &", and then do other stuff in that window while Emacs 
> occasionally outputs stderr diagnostics. (Although I think this is the sort of 
> thing you're alluding to above, I'm not sure.) So we would have an interleaving 
> problem anyway even if 'make -O' worked well.

This long thread began with a message that is output during a build.
Lars didn't like such messages mixed up from several Emacsen running
in parallel, so I proposed -O.  Then Lars said that use of 'message'
is not limited to building Emacs, and we embarked on the rest of the
thread.

With the 'message' case now solved by your patch (which is fine by me,
thanks), we are now back to the messages which are only output during
the build, or are diagnostic messages about abnormal or even fatal
situations.

I don't want us to buffer diagnostics and fatal error messages.  The
other messages, such as the one which outputs the pdumper fingerprint,
could be modified to not mix with others, if you really want, but I
really question the utility of that.

As for "make -O", could it be that you were talking literally about
that, i.e. about "make -O=target"?  Because I meant "make -O=line",
which AFAIR slows down the build only slightly, and provides the
line-level separation that you want to see.  I think using that is a
good compromise.

> +#ifndef DOS_NT /* _IOLBF does not work on MS-Windows.  */

This should be WINDOWSNT, not DOS_NT.

Thanks.

Re: `message' not outputting the newline "atomically"

[Richard Copley]

[-- Attachment #1: Type: text/plain, Size: 649 bytes --]

On Sun, 7 Jul 2019 at 15:52, Eli Zaretskii <eliz@gnu.org> wrote:

> > Cc: larsi@gnus.org, daniele@grinta.net, emacs-devel@gnu.org
> > From: Paul Eggert <eggert@cs.ucla.edu>
> > Date: Sat, 6 Jul 2019 18:16:03 -0700
>


> As for "make -O", could it be that you were talking literally about
> that, i.e. about "make -O=target"?  Because I meant "make -O=line",
> which AFAIR slows down the build only slightly, and provides the
> line-level separation that you want to see.  I think using that is a
> good compromise.
>

"make -O=line" buffers all output from one line of the Makefile.
For typical targets with only one command, it's the same as "-O".

[-- Attachment #2: Type: text/html, Size: 1467 bytes --]

Re: `message' not outputting the newline "atomically"

[Eli Zaretskii]

> From: Richard Copley <rcopley@gmail.com>
> Date: Mon, 8 Jul 2019 23:35:24 +0100
> Cc: Paul Eggert <eggert@cs.ucla.edu>, Lars Magne Ingebrigtsen <larsi@gnus.org>, daniele@grinta.net, 
> 	Emacs Development <emacs-devel@gnu.org>
> 
>  As for "make -O", could it be that you were talking literally about
>  that, i.e. about "make -O=target"?  Because I meant "make -O=line",
>  which AFAIR slows down the build only slightly, and provides the
>  line-level separation that you want to see.  I think using that is a
>  good compromise.
> 
> "make -O=line" buffers all output from one line of the Makefile.
> For typical targets with only one command, it's the same as "-O".

Our targets are not one-line, though.

Re: `message' not outputting the newline "atomically"

[Richard Copley]

[-- Attachment #1: Type: text/plain, Size: 1017 bytes --]

On Tue, 9 Jul 2019 at 03:33, Eli Zaretskii <eliz@gnu.org> wrote:

> > From: Richard Copley <rcopley@gmail.com>
> > Date: Mon, 8 Jul 2019 23:35:24 +0100
> > Cc: Paul Eggert <eggert@cs.ucla.edu>, Lars Magne Ingebrigtsen <
> larsi@gnus.org>, daniele@grinta.net,
> >       Emacs Development <emacs-devel@gnu.org>
> >
> >  As for "make -O", could it be that you were talking literally about
> >  that, i.e. about "make -O=target"?  Because I meant "make -O=line",
> >  which AFAIR slows down the build only slightly, and provides the
> >  line-level separation that you want to see.  I think using that is a
> >  good compromise.
> >
> > "make -O=line" buffers all output from one line of the Makefile.
> > For typical targets with only one command, it's the same as "-O".
>
> Our targets are not one-line, though.
>

Some are, notably .c.o. The point is that your statement that "make -O=line"
"provides the line-level separation that you want to see" is not true. It's
up to
you to decide if that affects your argument.

[-- Attachment #2: Type: text/html, Size: 1909 bytes --]

Re: `message' not outputting the newline "atomically"

[Eli Zaretskii]

> From: Richard Copley <rcopley@gmail.com>
> Date: Tue, 9 Jul 2019 14:45:58 +0100
> Cc: Paul Eggert <eggert@cs.ucla.edu>, Lars Magne Ingebrigtsen <larsi@gnus.org>, daniele@grinta.net, 
> 	Emacs Development <emacs-devel@gnu.org>
> 
>  > "make -O=line" buffers all output from one line of the Makefile.
>  > For typical targets with only one command, it's the same as "-O".
> 
>  Our targets are not one-line, though.
> 
> Some are, notably .c.o.

Which is fine, because people who use this option don't want
diagnostics from different source files mixed up.

> The point is that your statement that "make -O=line"
> "provides the line-level separation that you want to see" is not true. It's up to
> you to decide if that affects your argument.

Whether to use that option is in the eyes of the beholder, of course.
Exactly like any other Make option, like -jN.  It comes for a price,
but whether one is prepared to pay that price is up to each one of us,
and depends on what we want to do.

Re: `message' not outputting the newline "atomically"

[Paul Eggert]

[-- Attachment #1: Type: text/plain, Size: 1773 bytes --]

Eli Zaretskii wrote:

> I proposed -O.  Then Lars said that use of 'message'
> is not limited to building Emacs, and we embarked on the rest of the
> thread.

OK, I see the background better now. However, it's not just 'message', as Emacs 
will output to stderr in a few other places when I run it in the background from 
my terminal window (or whatever), even when Emacs is is not generating debugging 
output or a fatal error message. These other places should also be fixed.

> As for "make -O", could it be that you were talking literally about
> that, i.e. about "make -O=target"?  Because I meant "make -O=line",
> which AFAIR slows down the build only slightly, and provides the
> line-level separation that you want to see

Give the background you mentioned, 'make -O' is a cold trail since it doesn't 
suffice to fix the other problems. That being said, 'make -Oline' does have 
delays that slow me down. For example, when 'make -Oline' compiles src/xdisp.c, 
it doesn't show me any of GCC's output until GCC exits. On my desktop in some 
cases that's a 15-second wait for a diagnostic that I can see in a few 
milliseconds if I don't use -O.

> I don't want us to buffer diagnostics and fatal error messages.  The
> other messages, such as the one which outputs the pdumper fingerprint,
> could be modified to not mix with others

With that constraint in mind, I thought of a fix that's simpler and and more 
straightforward than what I sent you earlier. The idea is to output the other 
messages to stdout, not stderr. That way, they'll avoid the interleaving 
problem, and it'll be easier to separate them from debugging diagnostics and 
fatal error messages. Proposed patch attached; it attempts to identify all the 
places that output these "other messages".

[-- Attachment #2: 0001-Send-informative-messages-to-stdout-not-stderr.patch --]
[-- Type: text/x-patch, Size: 17477 bytes --]

From 91fa47f0d041b866230d2ed421f97ee89f176b00 Mon Sep 17 00:00:00 2001
From: Paul Eggert <eggert@cs.ucla.edu>
Date: Mon, 8 Jul 2019 19:16:28 -0700
Subject: [PATCH] Send informative messages to stdout, not stderr

This helps avoid interleaving info lines with lines from other
processes, since stdout is normally at least line buffered.
* doc/emacs/cmdargs.texi (Initial Options), etc/NEWS: Document this.
* src/dispnew.c (init_display_interactive):
* src/emacs.c (main, Fdump_emacs):
* src/gtkutil.c (my_log_handler, xg_set_geometry)
(xg_create_widget):
* src/image.c (convert_mono_to_color_image):
* src/lread.c (dir_warning):
* src/nsmenu.m (ns_update_menubar):
* src/nsterm.m (ns_mouse_position, ns_default)
(sendEvent:, performDragOperation:, mouseDown:):
* src/pdumper.c (dump_fingerprint, print_paths_to_root_1)
(Fdump_emacs_portable):
* src/xdisp.c (message_to_stdout, vmessage):
* src/xselect.c (x_clipboard_manager_error_2):
* src/xsmfns.c (x_session_initialize):
* src/xterm.c (my_log_handler, x_initialize):
Send info messages to stdout, not stderr.
* src/xdisp.c (message_to_stdout): Rename from message_to_stderr.
All uses changed.  Avoid need to allocate a buffer on the heap.
---
 doc/emacs/cmdargs.texi |  4 ++--
 etc/NEWS               |  5 +++++
 src/dispnew.c          |  3 +--
 src/emacs.c            | 23 +++++++++++------------
 src/gtkutil.c          |  9 +++------
 src/image.c            |  2 +-
 src/lread.c            |  4 ++--
 src/nsmenu.m           |  2 +-
 src/nsterm.m           | 13 ++++++-------
 src/pdumper.c          | 31 +++++++++++++++----------------
 src/xdisp.c            | 42 ++++++++++++++++--------------------------
 src/xselect.c          |  2 +-
 src/xsmfns.c           |  4 ++--
 src/xterm.c            |  4 ++--
 14 files changed, 68 insertions(+), 80 deletions(-)

diff --git a/doc/emacs/cmdargs.texi b/doc/emacs/cmdargs.texi
index 34a5ff5f2c..24319ea2d4 100644
--- a/doc/emacs/cmdargs.texi
+++ b/doc/emacs/cmdargs.texi
@@ -254,8 +254,8 @@ Initial Options
 message in the echo area will print to either the standard output
 stream (@code{stdout}) or the standard error stream (@code{stderr})
 instead.  (To be precise, functions like @code{prin1}, @code{princ}
-and @code{print} print to @code{stdout}, while @code{message} and
-@code{error} print to @code{stderr}.)  Functions that normally read
+@code{print} and @code{message} print to @code{stdout}, while
+@code{error} prints to @code{stderr}.)  Functions that normally read
 keyboard input from the minibuffer take their input from the
 terminal's standard input stream (@code{stdin}) instead.
 
diff --git a/etc/NEWS b/etc/NEWS
index 532babd0fa..8d6bac87a8 100644
--- a/etc/NEWS
+++ b/etc/NEWS
@@ -1732,6 +1732,11 @@ old-style backquotes as new-style, bind the new variable
 'cl-struct-define' whose name clashes with a builtin type (e.g.,
 'integer' or 'hash-table') now signals an error.
 
+** Non-error messages that were formerly sent to standard error are
+now sent to standard output.  For example, the 'message' function now
+outputs to standard output.  This helps prevent these output lines
+from being broken up by the output of other processes.
+
 ** When formatting a floating-point number as an octal or hexadecimal
 integer, Emacs now signals an error if the number is too large for the
 implementation to format.
diff --git a/src/dispnew.c b/src/dispnew.c
index 0131b63767..d354ce2b44 100644
--- a/src/dispnew.c
+++ b/src/dispnew.c
@@ -6154,8 +6154,7 @@ init_display_interactive (void)
 
       if (display_arg && !x_display_ok (display))
 	{
-	  fprintf (stderr, "Display %s unavailable, simulating -nw\n",
-		   display);
+	  printf ("Display %s unavailable, simulating -nw\n", display);
 	  inhibit_window_system = 1;
 	}
     }
diff --git a/src/emacs.c b/src/emacs.c
index 9c93748a0f..8bff66f073 100644
--- a/src/emacs.c
+++ b/src/emacs.c
@@ -1306,7 +1306,7 @@ main (int argc, char **argv)
 	      fprintf (stderr, "%s: %s: not a tty\n", argv[0], term);
 	      exit (EXIT_FAILURE);
 	    }
-	  fprintf (stderr, "Using %s\n", term);
+	  printf ("Using %s\n", term);
 #ifdef HAVE_WINDOW_SYSTEM
 	  inhibit_window_system = true; /* -t => -nw */
 #endif
@@ -1416,7 +1416,7 @@ main (int argc, char **argv)
         fputs (("\n"
 		"Warning: systemd passed more than one socket to Emacs.\n"
 		"Try 'Accept=false' in the Emacs socket unit file.\n"),
-	       stderr);
+	       stdout);
       else if (systemd_socket == 1
 	       && (0 < sd_is_socket (SD_LISTEN_FDS_START,
 				     AF_UNSPEC, SOCK_STREAM, 1)))
@@ -1427,7 +1427,7 @@ main (int argc, char **argv)
       fputs ("\nWarning: due to a long standing Gtk+ bug\nhttps://gitlab.gnome.org/GNOME/gtk/issues/221\n\
 Emacs might crash when run in daemon mode and the X11 connection is unexpectedly lost.\n\
 Using an Emacs configured with --with-x-toolkit=lucid does not have this problem.\n",
-	     stderr);
+	     stdout);
 #endif /* USE_GTK */
 
       if (daemon_type == 2)
@@ -2548,15 +2548,14 @@ DEFUN ("dump-emacs", Fdump_emacs, Sdump_emacs, 2, 2, 0,
 #  define MAX_HEAP_BSS_DIFF (1024 * 1024)
 
   if (heap_bss_diff > MAX_HEAP_BSS_DIFF)
-    fprintf (stderr,
-	     ("**************************************************\n"
-	      "Warning: Your system has a gap between BSS and the\n"
-	      "heap (%"PRIuMAX" bytes). This usually means that exec-shield\n"
-	      "or something similar is in effect.  The dump may\n"
-	      "fail because of this.  See the section about\n"
-	      "exec-shield in etc/PROBLEMS for more information.\n"
-	      "**************************************************\n"),
-	     heap_bss_diff);
+    printf (("**************************************************\n"
+	     "Warning: Your system has a gap between BSS and the\n"
+	     "heap (%"PRIuMAX" bytes). This usually means that exec-shield\n"
+	     "or something similar is in effect.  The dump may\n"
+	     "fail because of this.  See the section about\n"
+	     "exec-shield in etc/PROBLEMS for more information.\n"
+	     "**************************************************\n"),
+	    heap_bss_diff);
     }
 # endif
 
diff --git a/src/gtkutil.c b/src/gtkutil.c
index 1d15aec253..d96c193d08 100644
--- a/src/gtkutil.c
+++ b/src/gtkutil.c
@@ -829,7 +829,7 @@ my_log_handler (const gchar *log_domain, GLogLevelFlags log_level,
 		const gchar *msg, gpointer user_data)
 {
   if (!strstr (msg, "visible children"))
-    fprintf (stderr, "XX %s-WARNING **: %s\n", log_domain, msg);
+    printf ("XX %s-WARNING **: %s\n", log_domain, msg);
 }
 #endif
 
@@ -894,7 +894,7 @@ xg_set_geometry (struct frame *f)
 
 	  if (!gtk_window_parse_geometry (GTK_WINDOW (FRAME_GTK_OUTER_WIDGET (f)),
 					  geom_str))
-	    fprintf (stderr, "Failed to parse: '%s'\n", geom_str);
+	    printf ("Failed to parse: '%s'\n", geom_str);
 
 	  g_log_remove_handler ("Gtk", id);
 	}
@@ -2849,10 +2849,7 @@ xg_create_widget (const char *type, const char *name, struct frame *f,
         }
     }
   else
-    {
-      fprintf (stderr, "bad type in xg_create_widget: %s, doing nothing\n",
-               type);
-    }
+    printf ("bad type in xg_create_widget: %s, doing nothing\n", type);
 
   return w;
 }
diff --git a/src/image.c b/src/image.c
index e898a7364a..6609488758 100644
--- a/src/image.c
+++ b/src/image.c
@@ -3338,7 +3338,7 @@ convert_mono_to_color_image (struct frame *f, struct image *img,
   DeleteDC (new_img_dc);
   DeleteObject (img->pixmap);
   if (new_pixmap == 0)
-    fputs ("Failed to convert image to color.\n", stderr);
+    fputs ("Failed to convert image to color.\n", stdout);
   else
     img->pixmap = new_pixmap;
 }
diff --git a/src/lread.c b/src/lread.c
index e06eafcf6c..f32c56b573 100644
--- a/src/lread.c
+++ b/src/lread.c
@@ -4715,7 +4715,7 @@ init_lread (void)
 
 /* Print a warning that directory intended for use USE and with name
    DIRNAME cannot be accessed.  On entry, errno should correspond to
-   the access failure.  Print the warning on stderr and put it in
+   the access failure.  Print the warning on stdout and put it in
    *Messages*.  */
 
 void
@@ -4723,7 +4723,7 @@ dir_warning (char const *use, Lisp_Object dirname)
 {
   static char const format[] = "Warning: %s '%s': %s\n";
   char *diagnostic = emacs_strerror (errno);
-  fprintf (stderr, format, use, SSDATA (ENCODE_SYSTEM (dirname)), diagnostic);
+  printf (format, use, SSDATA (ENCODE_SYSTEM (dirname)), diagnostic);
 
   /* Don't log the warning before we've initialized!!  */
   if (initialized)
diff --git a/src/nsmenu.m b/src/nsmenu.m
index 817f8cff18..d7274c0723 100644
--- a/src/nsmenu.m
+++ b/src/nsmenu.m
@@ -236,7 +236,7 @@
       if (submenu && n == 0)
         {
           /* should have found a menu for this one but didn't */
-          fprintf (stderr, "ERROR: did not find lisp menu for submenu '%s'.\n",
+	  printf ("ERROR: did not find lisp menu for submenu '%s'.\n",
                   [[submenu title] UTF8String]);
 	  discard_menu_items ();
 	  unbind_to (specpdl_count, Qnil);
diff --git a/src/nsterm.m b/src/nsterm.m
index 02331826d9..b202013577 100644
--- a/src/nsterm.m
+++ b/src/nsterm.m
@@ -2466,7 +2466,7 @@ so some key presses (TAB) are swallowed by the system.  */
 
   if (*fp == NULL)
     {
-      fputs ("Warning: ns_mouse_position () called with null *fp.\n", stderr);
+      fputs ("Warning: ns_mouse_position () called with null *fp.\n", stdout);
       return;
     }
 
@@ -5032,8 +5032,8 @@ static Lisp_Object ns_string_to_lispmod (const char *s)
         *result = make_float (f);
       else if (is_modstring && value)
         *result = ns_string_to_lispmod (value);
-      else fprintf (stderr,
-                   "Bad value for default \"%s\": \"%s\"\n", parameter, value);
+      else
+	printf ("Bad value for default \"%s\": \"%s\"\n", parameter, value);
     }
 }
 
@@ -5613,7 +5613,7 @@ - (void)sendEvent: (NSEvent *)theEvent
 
   if (type == NSEventTypeCursorUpdate && window == nil)
     {
-      fputs ("Dropping external cursor update event.\n", stderr);
+      fputs ("Dropping external cursor update event.\n", stdout);
       return;
     }
 
@@ -8291,7 +8291,7 @@ -(BOOL)performDragOperation: (id <NSDraggingInfo>) sender
     }
   else
     {
-      fputs ("Invalid data type in dragging pasteboard\n", stderr);
+      fputs ("Invalid data type in dragging pasteboard\n", stdout);
       return NO;
     }
 
@@ -9035,8 +9035,7 @@ - (void)mouseDown: (NSEvent *)e
     case NSScrollerKnobSlot:  /* GNUstep-only */
       last_hit_part = scroll_bar_move_ratio; break;
     default:  /* NSScrollerNoPart? */
-      fprintf (stderr, "EmacsScroller-mouseDown: unexpected part %ld\n",
-               (long) part);
+      printf ("EmacsScroller-mouseDown: unexpected part %ld\n", (long) part);
       return;
     }
 
diff --git a/src/pdumper.c b/src/pdumper.c
index 8b630d221b..cd4ec9fbcf 100644
--- a/src/pdumper.c
+++ b/src/pdumper.c
@@ -326,10 +326,10 @@ dump_reloc_set_offset (struct dump_reloc *reloc, dump_off offset)
 static void
 dump_fingerprint (const char *label, unsigned char const *xfingerprint)
 {
-  fprintf (stderr, "%s: ", label);
+  printf ("%s: ", label);
   for (int i = 0; i < 32; ++i)
-    fprintf (stderr, "%02x", (unsigned) xfingerprint[i]);
-  putc ('\n', stderr);
+    printf ("%02x", (unsigned) xfingerprint[i]);
+  putchar ('\n');
 }
 
 /* Format of an Emacs portable dump file.  All offsets are relative to
@@ -1404,9 +1404,9 @@ print_paths_to_root_1 (struct dump_context *ctx,
       referrers = XCDR (referrers);
       Lisp_Object repr = Fprin1_to_string (referrer, Qnil);
       for (int i = 0; i < level; ++i)
-	putc (' ', stderr);
-      fwrite (SDATA (repr), 1, SBYTES (repr), stderr);
-      putc ('\n', stderr);
+	putchar (' ');
+      fwrite (SDATA (repr), 1, SBYTES (repr), stdout);
+      putchar ('\n');
       print_paths_to_root_1 (ctx, referrer, level + 1);
     }
 }
@@ -4226,16 +4226,15 @@ DEFUN ("dump-emacs-portable",
   dump_seek (ctx, 0);
   dump_write (ctx, &ctx->header, sizeof (ctx->header));
 
-  fprintf (stderr,
-	   ("Dump complete\n"
-	    "Byte counts: header=%lu hot=%lu discardable=%lu cold=%lu\n"
-	    "Reloc counts: hot=%u discardable=%u\n"),
-           (unsigned long) (header_end - header_start),
-           (unsigned long) (hot_end - hot_start),
-           (unsigned long) (discardable_end - ctx->header.discardable_start),
-           (unsigned long) (cold_end - ctx->header.cold_start),
-           number_hot_relocations,
-           number_discardable_relocations);
+  printf (("Dump complete\n"
+	   "Byte counts: header=%lu hot=%lu discardable=%lu cold=%lu\n"
+	   "Reloc counts: hot=%u discardable=%u\n"),
+	  (unsigned long) (header_end - header_start),
+	  (unsigned long) (hot_end - hot_start),
+	  (unsigned long) (discardable_end - ctx->header.discardable_start),
+	  (unsigned long) (cold_end - ctx->header.cold_start),
+	  number_hot_relocations,
+	  number_discardable_relocations);
 
   unblock_input ();
   return unbind_to (count, Qnil);
diff --git a/src/xdisp.c b/src/xdisp.c
index 2711e54382..04aad05413 100644
--- a/src/xdisp.c
+++ b/src/xdisp.c
@@ -10706,15 +10706,15 @@ message3 (Lisp_Object m)
     message3_nolog (m);
 }
 
-/* Log the message M to stderr.  Log an empty line if M is not a string.  */
+/* Log the message M to stdout.  Log an empty line if M is not a string.  */
 
 static void
-message_to_stderr (Lisp_Object m)
+message_to_stdout (Lisp_Object m)
 {
   if (noninteractive_need_newline)
     {
       noninteractive_need_newline = false;
-      putc ('\n', stderr);
+      putchar ('\n');
     }
   if (STRINGP (m))
     {
@@ -10728,21 +10728,11 @@ message_to_stderr (Lisp_Object m)
       else
 	s = m;
 
-      /* We want to write this out with a single call so that
-	 output doesn't interleave with other processes writing to
-	 stderr at the same time. */
-      {
-	int length = min (INT_MAX, SBYTES (s) + 1);
-	char *string = xmalloc (length);
-
-	memcpy (string, SSDATA (s), length - 1);
-	string[length - 1] = '\n';
-	fwrite (string, 1, length, stderr);
-	xfree (string);
-      }
+      fwrite_unlocked (SDATA (s), 1, SBYTES (s), stdout);
     }
-  else if (!cursor_in_echo_area)
-    putc ('\n', stderr);
+
+  if (STRINGP (m) || !cursor_in_echo_area)
+    putchar ('\n');
 }
 
 /* The non-logging version of message3.
@@ -10756,7 +10746,7 @@ message3_nolog (Lisp_Object m)
   struct frame *sf = SELECTED_FRAME ();
 
   if (FRAME_INITIAL_P (sf))
-    message_to_stderr (m);
+    message_to_stdout (m);
   /* Error messages get reported properly by cmd_error, so this must be just an
      informative message; if the frame hasn't really been initialized yet, just
      toss it.  */
@@ -10853,7 +10843,7 @@ message_with_string (const char *m, Lisp_Object string, bool log)
       Lisp_Object msg = CALLN (Fformat_message, fmt, string);
 
       if (noninteractive)
-	message_to_stderr (msg);
+	message_to_stdout (msg);
       else
 	{
 	  if (log)
@@ -10873,7 +10863,7 @@ message_with_string (const char *m, Lisp_Object string, bool log)
    any existing message, and let the mini-buffer text show through.
 
    The message must be safe ASCII (because when Emacs is
-   non-interactive the message is sent straight to stderr without
+   non-interactive the message is sent straight to stdout without
    encoding first) and the format must not contain ` or ' (because
    this function does not account for `text-quoting-style').  If your
    message and format do not fit into this category, convert your
@@ -10887,12 +10877,12 @@ vmessage (const char *m, va_list ap)
       if (m)
 	{
 	  if (noninteractive_need_newline)
-	    putc ('\n', stderr);
-	  noninteractive_need_newline = false;
-	  vfprintf (stderr, m, ap);
-	  if (!cursor_in_echo_area)
-	    putc ('\n', stderr);
-	  fflush (stderr);
+	    {
+	      noninteractive_need_newline = false;
+	      putchar ('\n');
+	    }
+	  vprintf (m, ap);
+	  putchar ('\n');
 	}
     }
   else if (INTERACTIVE)
diff --git a/src/xselect.c b/src/xselect.c
index d74f064a1c..14ecfeddb7 100644
--- a/src/xselect.c
+++ b/src/xselect.c
@@ -2175,7 +2175,7 @@ x_clipboard_manager_error_2 (Lisp_Object err)
   fputs (("Error saving to X clipboard manager.\n"
 	  "If the problem persists,"
 	  " set 'x-select-enable-clipboard-manager' to nil.\n"),
-	 stderr);
+	 stdout);
   return Qnil;
 }
 
diff --git a/src/xsmfns.c b/src/xsmfns.c
index 1706cddf27..8dfd60663f 100644
--- a/src/xsmfns.c
+++ b/src/xsmfns.c
@@ -404,8 +404,8 @@ #define SM_ERRORSTRING_LEN 512
   char *pwd = emacs_get_current_dir_name ();
   if (!pwd)
     {
-      fprintf (stderr, "Disabling session management due to pwd error: %s\n",
-               emacs_strerror (errno));
+      printf ("Disabling session management due to pwd error: %s\n",
+	      emacs_strerror (errno));
       return;
     }
   xfree (pwd);
diff --git a/src/xterm.c b/src/xterm.c
index c96aa74a7a..0cc2df6ecb 100644
--- a/src/xterm.c
+++ b/src/xterm.c
@@ -12465,7 +12465,7 @@ my_log_handler (const gchar *log_domain, GLogLevelFlags log_level,
 		const gchar *msg, gpointer user_data)
 {
   if (!strstr (msg, "g_set_prgname"))
-      fprintf (stderr, "%s-WARNING **: %s\n", log_domain, msg);
+    printf ("%s-WARNING **: %s\n", log_domain, msg);
 }
 #endif
 
@@ -13442,7 +13442,7 @@ x_initialize (void)
   /* This must be called before any other Xlib routines.  */
   if (XInitThreads () == 0)
     fputs ("Warning: An error occurred initializing X11 thread support!\n",
-	   stderr);
+	   stdout);
 #endif
 
 #ifdef USE_X_TOOLKIT
-- 
2.21.0

Re: `message' not outputting the newline "atomically"

[Eli Zaretskii]

> Cc: larsi@gnus.org, daniele@grinta.net,
>  Emacs Development <Emacs-devel@gnu.org>
> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Mon, 8 Jul 2019 19:47:26 -0700
> 
> > As for "make -O", could it be that you were talking literally about
> > that, i.e. about "make -O=target"?  Because I meant "make -O=line",
> > which AFAIR slows down the build only slightly, and provides the
> > line-level separation that you want to see
> 
> Give the background you mentioned, 'make -O' is a cold trail since it doesn't 
> suffice to fix the other problems. That being said, 'make -Oline' does have 
> delays that slow me down. For example, when 'make -Oline' compiles src/xdisp.c, 
> it doesn't show me any of GCC's output until GCC exits. On my desktop in some 
> cases that's a 15-second wait for a diagnostic that I can see in a few 
> milliseconds if I don't use -O.

With recent GCC versions emitting several lines of diagnostics for
several related but different files, using -Oline avoids mixing up
diagnostics from different sources, which is something someone may
wish to have.  Sure, it slows things down a little, but free lunches
are no longer available.  Whether this is a price one wants to pay
depends on the situation and on what one needs to do or test.  Like
with any other Make option; for example, I sometimes don't use -jN
because I need to avoid parallelism for some reason or another, and it
certainly slows down the build.

> > I don't want us to buffer diagnostics and fatal error messages.  The
> > other messages, such as the one which outputs the pdumper fingerprint,
> > could be modified to not mix with others
> 
> With that constraint in mind, I thought of a fix that's simpler and and more 
> straightforward than what I sent you earlier. The idea is to output the other 
> messages to stdout, not stderr. That way, they'll avoid the interleaving 
> problem, and it'll be easier to separate them from debugging diagnostics and 
> fatal error messages. Proposed patch attached; it attempts to identify all the 
> places that output these "other messages".

I feel we are going in circles whose radius doesn't get smaller, which
means we will never converge.  We already found an agreed solution for
'message' and 'vmessage', so why step back and re-introduce them into
the problem again?  Switching 'message' to use stdout is something I
don't think we can reasonably consider, as that changes behavior Emacs
had for decades.

As for other messages, I think the vast majority are diagnostics, and
thus using stdout for them is inappropriate, as it will get in the way
of using Emacs in scripts.  Anything that has "warning" or "error" or
"failed" or any similar words in the message is definitely for stderr.

IOW, as I already said, I see only one candidate for stdout here: the
pdumper fingerprint, and even there I'm not 100% sure.  All the other
messages you suggested to output to stdout are diagnostics, and should
stay on stderr.  The exception is 'message' and 'vmessage', where we
already arrived at an acceptable solution which uses a stream that is
line-buffered, but separate from stdout.

Hmm... about 'message', though: what would happen if a script says
2>foo? will the file 'foo' remain empty if we use any stream but
stderr inside message_to_stderr?  If so, maybe even the solution I
thought will be acceptable isn't.

Re: `message' not outputting the newline "atomically"

[Paul Eggert]

Eli Zaretskii wrote:
> We already found an agreed solution for
> 'message' and 'vmessage', so why step back and re-introduce them into
> the problem again?

Because we now have a simpler solution.

> As for other messages, I think the vast majority are diagnostics, and
> thus using stdout for them is inappropriate, as it will get in the way
> of using Emacs in scripts.

No, Emacs uses stdout reasonably extensively for this sort of thing already; see 
'write_stdout' and I can cite several other examples. In practice this has not 
been a problem because nobody cares whether these kinds of Emacs messages go to 
stdout or stderr. One can also see this in the decades-old commentary for the 
implementation of 'message', which says both 'stdout' and 'stderr' (obviously 
some of this commentary is incorrect, and the proposed patch fixes it).

All I'm suggesting is that we move a few messages from Emacs's stderr category 
to its (already-existing) stdout category, to fix the interleaving problem for 
messages that are not fatal diagnostics or for debugging. These messages are the 
ones most likely to cause interleaving problems in practice. The vast majority 
of messages would be unaffected by the proposed patch.

> what would happen if a script says
> 2>foo? will the file 'foo' remain empty if we use any stream but
> stderr inside message_to_stderr?

No, it'll work fine. The OS doesn't care about stdio's buffers; all it cares is 
whether file descriptor 2 is used, which it is.

Re: `message' not outputting the newline "atomically"

[Eli Zaretskii]

> Cc: larsi@gnus.org, daniele@grinta.net, Emacs-devel@gnu.org
> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Tue, 9 Jul 2019 11:12:52 -0700
> 
> Eli Zaretskii wrote:
> > We already found an agreed solution for
> > 'message' and 'vmessage', so why step back and re-introduce them into
> > the problem again?
> 
> Because we now have a simpler solution.

I don't think it's simpler: it causes 'message' write to a different
file descriptor, and thus introduces a backward incompatibility.

> > As for other messages, I think the vast majority are diagnostics, and
> > thus using stdout for them is inappropriate, as it will get in the way
> > of using Emacs in scripts.
> 
> No, Emacs uses stdout reasonably extensively for this sort of thing already; see 
> 'write_stdout' and I can cite several other examples.

write_stdout is used in exactly 10 places, which I wouldn't call
"extensive".  It is used where AFAIU any other file descriptor would
be problematic.  So I don't think I see how this argument should
convince me.

> In practice this has not 
> been a problem because nobody cares whether these kinds of Emacs messages go to 
> stdout or stderr.

I don't understand how could you make these assertions.  What are they
based on?

I believe it is a convention to output diagnostic messages about
abnormal or unexpected events to stderr, not stdout.  Why should Emacs
deviate from this convention?

> > what would happen if a script says
> > 2>foo? will the file 'foo' remain empty if we use any stream but
> > stderr inside message_to_stderr?
> 
> No, it'll work fine. The OS doesn't care about stdio's buffers; all it cares is 
> whether file descriptor 2 is used, which it is.

Oh, I've misremembered that they use the same descriptor, sorry.

So I think your previous proposal for message_to_stderr and vmessage
should be okay to go in.  But not the rest, sorry.

Re: `message' not outputting the newline "atomically"

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

>> In practice this has not 
>> been a problem because nobody cares whether these kinds of Emacs
>> messages go to
>> stdout or stderr.
>
> I don't understand how could you make these assertions.  What are they
> based on?
>
> I believe it is a convention to output diagnostic messages about
> abnormal or unexpected events to stderr, not stdout.  Why should Emacs
> deviate from this convention?

I, for one, want the error messages to go to stderr and informational
messages to go to stdout.  But there currently doesn't seem to be a way
to specify that from Emacs Lisp?  That is, there's no version of message
under --batch that allows you to write to stdout?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: `message' not outputting the newline "atomically"

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: Paul Eggert <eggert@cs.ucla.edu>,  daniele@grinta.net,  Emacs-devel@gnu.org
> Date: Tue, 09 Jul 2019 20:44:54 +0200
> 
> I, for one, want the error messages to go to stderr and informational
> messages to go to stdout.  But there currently doesn't seem to be a way
> to specify that from Emacs Lisp?  That is, there's no version of message
> under --batch that allows you to write to stdout?

We could add one.

`message' not outputting the newline "atomically"

[Paul Eggert]

[-- Attachment #1: Type: text/plain, Size: 2132 bytes --]

Eli Zaretskii wrote:

>> Because we now have a simpler solution.
> 
> I don't think it's simpler: it causes 'message' write to a different
> file descriptor, and thus introduces a backward incompatibility.

I meant simpler to implement and easier to maintain. My email discussed the 
backward compatbility issue, which is negligible.

>> Emacs uses stdout reasonably extensively for this sort of thing already; see
>> 'write_stdout' and I can cite several other examples.
> 
> write_stdout is used in exactly 10 places, which I wouldn't call "extensive".

As I mentioned, write_stdout isn't the only place stdout is used for this sort 
of thing. Another example is w32con_write_glyphs. I'm not saying stdout is used 
as often as stderr; however, stdout is used in a reasonably large set of places, 
and nobody cares. In this sense, Emacs differs from typical Unix-like utilities.

> It is used where AFAIU any other file descriptor would
> be problematic.

In the places where write_stdout is used, stderr is no more problematic than 
stdout is. Neither is guaranteed to be displayed to the user.

> I don't understand how could you make these assertions.  What are they
> based on?

You're asking me to prove a negative? OK, I concede; I can't do that. However, 
my statements were based on experience in using Emacs and in reading its source 
code and bug reports and complaints about it.
> So I think your previous proposal for message_to_stderr and vmessage
> should be okay to go in.  But not the rest, sorry.

OK, I put that proposal into master (with some minor improvements). Also, I 
installed the attached patches to fix the most-annoying remaining stderr 
interleaving problems without changing stderr buffering. This should be enough 
to address the interleaving problems on GNU/Linux, in areas where the OS lets us 
address them.

Unfortunately, GNU Emacs master still suffers from interleaving problems on 
Solaris, AIX, and perhaps some other POSIXish platforms, because their fprintf 
implementations write "%s"-formatted strings separately. Although it's easy to 
fix that, no fix is likely to meet your standards.

[-- Attachment #2: 0001-Avoid-interleaving-stderr-in-dump_fingerprint.patch --]
[-- Type: text/x-patch, Size: 3447 bytes --]

From 1178f98f2c0973dd1f8a66cbb4de20c0d7af3271 Mon Sep 17 00:00:00 2001
From: Paul Eggert <eggert@cs.ucla.edu>
Date: Sat, 13 Jul 2019 10:41:46 -0700
Subject: [PATCH] Avoid interleaving stderr in dump_fingerprint

* src/fns.c (hexbuf_digest): New function, containing most of
the old make_digest_string.
(make_digest_string): Use it.
* src/pdumper.c (dump_fingerprint): Rewrite to use a single
fprintf call, to avoid interleaving on GNU/Linux.
---
 src/fns.c     | 19 ++++++++++++++-----
 src/lisp.h    |  1 +
 src/pdumper.c | 13 +++++++------
 3 files changed, 22 insertions(+), 11 deletions(-)

diff --git a/src/fns.c b/src/fns.c
index 6a7c347728..54dafe07ac 100644
--- a/src/fns.c
+++ b/src/fns.c
@@ -5040,18 +5040,27 @@ returns nil, then (funcall TEST x1 x2) also returns nil.  */)
 #include "sha256.h"
 #include "sha512.h"
 
-static Lisp_Object
-make_digest_string (Lisp_Object digest, int digest_size)
+/* Store into HEXBUF an unterminated hexadecimal character string
+   representing DIGEST, which is binary data of size DIGEST_SIZE bytes.
+   HEXBUF might equal DIGEST.  */
+void
+hexbuf_digest (char *hexbuf, void const *digest, int digest_size)
 {
-  unsigned char *p = SDATA (digest);
+  unsigned char const *p = digest;
 
   for (int i = digest_size - 1; i >= 0; i--)
     {
       static char const hexdigit[16] = "0123456789abcdef";
       int p_i = p[i];
-      p[2 * i] = hexdigit[p_i >> 4];
-      p[2 * i + 1] = hexdigit[p_i & 0xf];
+      hexbuf[2 * i] = hexdigit[p_i >> 4];
+      hexbuf[2 * i + 1] = hexdigit[p_i & 0xf];
     }
+}
+
+static Lisp_Object
+make_digest_string (Lisp_Object digest, int digest_size)
+{
+  hexbuf_digest (SSDATA (digest), SDATA (digest), digest_size);
   return digest;
 }
 
diff --git a/src/lisp.h b/src/lisp.h
index e93a219625..4885e26e3f 100644
--- a/src/lisp.h
+++ b/src/lisp.h
@@ -3586,6 +3586,7 @@ extern ptrdiff_t list_length (Lisp_Object);
 extern EMACS_INT next_almost_prime (EMACS_INT) ATTRIBUTE_CONST;
 extern Lisp_Object larger_vector (Lisp_Object, ptrdiff_t, ptrdiff_t);
 extern bool sweep_weak_table (struct Lisp_Hash_Table *, bool);
+extern void hexbuf_digest (char *, void const *, int);
 extern char *extract_data_from_object (Lisp_Object, ptrdiff_t *, ptrdiff_t *);
 EMACS_UINT hash_string (char const *, ptrdiff_t);
 EMACS_UINT sxhash (Lisp_Object, int);
diff --git a/src/pdumper.c b/src/pdumper.c
index b80757c207..03c00bf27b 100644
--- a/src/pdumper.c
+++ b/src/pdumper.c
@@ -324,12 +324,13 @@ dump_reloc_set_offset (struct dump_reloc *reloc, dump_off offset)
 }
 
 static void
-dump_fingerprint (const char *label, unsigned char const *xfingerprint)
+dump_fingerprint (char const *label,
+		  unsigned char const xfingerprint[sizeof fingerprint])
 {
-  fprintf (stderr, "%s: ", label);
-  for (int i = 0; i < 32; ++i)
-    fprintf (stderr, "%02x", (unsigned) xfingerprint[i]);
-  putc ('\n', stderr);
+  enum { hexbuf_size = 2 * sizeof fingerprint };
+  char hexbuf[hexbuf_size];
+  hexbuf_digest (hexbuf, xfingerprint, sizeof fingerprint);
+  fprintf (stderr, "%s: %.*s\n", label, hexbuf_size, hexbuf);
 }
 
 /* Format of an Emacs portable dump file.  All offsets are relative to
@@ -355,7 +356,7 @@ struct dump_header
   char magic[sizeof (dump_magic)];
 
   /* Associated Emacs binary.  */
-  unsigned char fingerprint[32];
+  unsigned char fingerprint[sizeof fingerprint];
 
   /* Relocation table for the dump file; each entry is a
      struct dump_reloc.  */
-- 
2.17.1


[-- Attachment #3: 0001-Avoid-interleaving-stderr-lines-when-shutting-down.patch --]
[-- Type: text/x-patch, Size: 2656 bytes --]

From 34810ab4f9990a8de1e503fdf6b485d8eeea1601 Mon Sep 17 00:00:00 2001
From: Paul Eggert <eggert@cs.ucla.edu>
Date: Sat, 13 Jul 2019 16:42:18 -0700
Subject: [PATCH] Avoid interleaving stderr lines when shutting down

* src/emacs.c (shut_down_emacs) [!DOS_NT]: Avoid interleaving
to stderr in the usual case, by using a single write and by
appending a newline.
* src/sysdep.c (emacs_backtrace) [HAVE_BACKTRACE_SYMBOLS_FD]:
Omit newline since shut_down_emacs now does that.
---
 src/emacs.c  | 38 ++++++++++++++++++++++----------------
 src/sysdep.c |  2 +-
 2 files changed, 23 insertions(+), 17 deletions(-)

diff --git a/src/emacs.c b/src/emacs.c
index 9c93748a0f..ad661a081b 100644
--- a/src/emacs.c
+++ b/src/emacs.c
@@ -2454,23 +2454,29 @@ shut_down_emacs (int sig, Lisp_Object stuff)
 
   /* If we are controlling the terminal, reset terminal modes.  */
 #ifndef DOS_NT
-  {
-    pid_t pgrp = getpgrp ();
-    pid_t tpgrp = tcgetpgrp (0);
-    if ((tpgrp != -1) && tpgrp == pgrp)
-      {
-	reset_all_sys_modes ();
-	if (sig && sig != SIGTERM)
-	  {
-	    static char const format[] = "Fatal error %d: ";
-	    char buf[sizeof format - 2 + INT_STRLEN_BOUND (int)];
-	    int buflen = sprintf (buf, format, sig);
-	    char const *sig_desc = safe_strsignal (sig);
+  pid_t tpgrp = tcgetpgrp (STDIN_FILENO);
+  if (tpgrp != -1 && tpgrp == getpgrp ())
+    {
+      reset_all_sys_modes ();
+      if (sig && sig != SIGTERM)
+	{
+	  static char const fmt[] = "Fatal error %d: %n%s\n";
+	  char buf[max ((sizeof fmt - sizeof "%d%n%s\n"
+			 + INT_STRLEN_BOUND (int) + 1),
+			min (PIPE_BUF, MAX_ALLOCA))];
+	  char const *sig_desc = safe_strsignal (sig);
+	  int nlen;
+	  int buflen = snprintf (buf, sizeof buf, fmt, sig, &nlen, sig_desc);
+	  if (0 <= buflen && buflen < sizeof buf)
 	    emacs_write (STDERR_FILENO, buf, buflen);
-	    emacs_write (STDERR_FILENO, sig_desc, strlen (sig_desc));
-	  }
-      }
-  }
+	  else
+	    {
+	      emacs_write (STDERR_FILENO, buf, nlen);
+	      emacs_write (STDERR_FILENO, sig_desc, strlen (sig_desc));
+	      emacs_write (STDERR_FILENO, fmt + sizeof fmt - 2, 1);
+	    }
+	}
+    }
 #else
   fflush (stdout);
   reset_all_sys_modes ();
diff --git a/src/sysdep.c b/src/sysdep.c
index 9301405943..f7478253a3 100644
--- a/src/sysdep.c
+++ b/src/sysdep.c
@@ -2436,7 +2436,7 @@ emacs_backtrace (int backtrace_limit)
 
   if (npointers)
     {
-      emacs_write (STDERR_FILENO, "\nBacktrace:\n", 12);
+      emacs_write (STDERR_FILENO, "Backtrace:\n", 11);
       backtrace_symbols_fd (buffer, npointers, STDERR_FILENO);
       if (bounded_limit < npointers)
 	emacs_write (STDERR_FILENO, "...\n", 4);
-- 
2.17.1

Re: `message' not outputting the newline "atomically"

[Eli Zaretskii]

> From: Paul Eggert <eggert@cs.ucla.edu>
> Cc: larsi@gnus.org, daniele@grinta.net, Emacs-devel@gnu.org
> Date: Sat, 13 Jul 2019 17:42:45 -0700
> 
> > So I think your previous proposal for message_to_stderr and vmessage
> > should be okay to go in.  But not the rest, sorry.
> 
> OK, I put that proposal into master (with some minor improvements). Also, I 
> installed the attached patches to fix the most-annoying remaining stderr 
> interleaving problems without changing stderr buffering. This should be enough 
> to address the interleaving problems on GNU/Linux, in areas where the OS lets us 
> address them.

Thank you.

Re: `message' not outputting the newline "atomically"

[Eli Zaretskii]

> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Mon, 24 Jun 2019 14:11:31 -0700
> Cc: Daniele Nicolodi <daniele@grinta.net>, emacs-devel@gnu.org
> 
> * The buffer should contain only PIPE_BUF bytes, since writes larger 
> than that can be split by the operating system anyway so any excess size 
> is wasted. On POSIX platforms you can use fpathconf to calculate 
> PIPE_BUF for stderr. I don't know how to calculate it on MS-Windows 
> platforms, but maybe it doesn't matter and you can just pretend it's 
> 1024 or whatever.

I think the equivalent size on Windows should be between 16 and 64KB
(AFAIK, the recent versions of Windows increased it to the latter
value).  But as we are talking about 'fwrite', isn't a more important
value the buffer size of the FILE stream object?  That one is usually
something like 4KB, I think.