From: "Ludovic Courtès" <ludo@gnu.org>
To: 32980@debbugs.gnu.org
Subject: [bug#32980] [PATCH 0/2] Multiplexed build output from the daemon
Date: Mon, 8 Oct 2018 00:29:45 +0200 [thread overview]
Message-ID: <20181007222945.2077-1-ludo@gnu.org> (raw)
Hello Guix!
This patch set is partly a response to the problem Ricardo raised at:
https://lists.gnu.org/archive/html/guix-devel/2018-09/msg00322.html
and partly a natural followup to the addition of (guix status):
https://issues.guix.info/issue/32837
So far the daemon would send all its output (messages it writes as well as
messages any of its build processes writes) directly as a single stream
to its clients. This had several shortcomings:
1. Clients could not distinguish messages coming from the daemon (such
as build traces or ‘guix substitute’ messages) from messages coming
from build processes. (Notably build processes could “forge” build
traces such as “@ build-started”.)
2. When max-jobs > 1, clients would receive intermingled output from all
the build processes without any way to disentangle it.
3. Build traces written by the daemon were expected to start on a new
line but a builder could write output not terminated by a newline
(e.g., “checking for …” messages from ./configure) and consequently
build traces would not start on a new line and would go unnoticed
by (guix status) and co.
With this change clients can optionally ask for “multiplexed build
output”. When it’s enabled, build output is prefixed by a special
trace, like this:
@ build-output 1234 21
checking for fcntl...
where 1234 is the PID of the build process speaking and 21 is the number
of bytes in the following build output fragment. The PID is first given
in the corresponding “@ build-started” trace.
On the client side, (guix status) is adjusted to produce events like:
(build-log #f MESSAGE)
for a message coming from the daemon, and:
(build-log PID MESSAGE)
for a message coming from PID.
The downside of the protocol is that it creates quite some overhead. For
example, when extracting a tarball, we see things like:
read(13, "gmlo\0\0\0\0", 8) = 8
read(13, "5\0\0\0\0\0\0\0", 8) = 8
read(13, "@ build-output 25935 29\ncoreutils-8.29/m4/fseterr.m4\n", 53) = 53
read(13, "\0\0\0", 3) = 3
That is, a 29-byte message with a 24-byte header (plus the
8 + 8 + 3 = 19 bytes of the underlying protocol; see ‘process-stderr’.)
Another option would be to incorporate multiplexing in the lower-level
binary protocol. However, the client side would need potentially bigger
changes: the single ‘current-build-output-port’ sink would no longer
be a good match.
Thoughts?
Ludo’.
Ludovic Courtès (2):
daemon: Support multiplexed build output.
status: Build upon multiplexed build output.
guix/scripts/build.scm | 3 +
guix/scripts/environment.scm | 1 +
guix/scripts/pack.scm | 1 +
guix/scripts/package.scm | 3 +-
guix/scripts/pull.scm | 1 +
guix/scripts/system.scm | 1 +
guix/status.scm | 165 ++++++++++++++++++++++++--------
guix/store.scm | 15 ++-
nix/libstore/build.cc | 25 +++--
nix/libstore/globals.cc | 2 +
nix/libstore/globals.hh | 9 +-
nix/libstore/worker-protocol.hh | 2 +-
nix/nix-daemon/nix-daemon.cc | 2 +-
tests/status.scm | 47 ++++++++-
tests/store.scm | 63 ++++++++++++
15 files changed, 289 insertions(+), 51 deletions(-)
--
2.19.0
next reply other threads:[~2018-10-07 22:59 UTC|newest]
Thread overview: 4+ messages / expand[flat|nested] mbox.gz Atom feed top
2018-10-07 22:29 Ludovic Courtès [this message]
2018-10-07 22:38 ` [bug#32980] [PATCH 1/2] daemon: Support multiplexed build output Ludovic Courtès
2018-10-07 22:38 ` [bug#32980] [PATCH 2/2] status: Build upon " Ludovic Courtès
2018-10-15 21:27 ` bug#32980: [PATCH 0/2] Multiplexed build output from the daemon Ludovic Courtès
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: https://guix.gnu.org/
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20181007222945.2077-1-ludo@gnu.org \
--to=ludo@gnu.org \
--cc=32980@debbugs.gnu.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://git.savannah.gnu.org/cgit/guix.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).