From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:50679) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1g9I0v-0002yn-RQ for guix-patches@gnu.org; Sun, 07 Oct 2018 18:59:06 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1g9HoA-0007nH-UP for guix-patches@gnu.org; Sun, 07 Oct 2018 18:45:55 -0400 Received: from debbugs.gnu.org ([208.118.235.43]:35463) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1g9HZm-00088S-C4 for guix-patches@gnu.org; Sun, 07 Oct 2018 18:31:02 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1g9HZm-0003l9-8c for guix-patches@gnu.org; Sun, 07 Oct 2018 18:31:02 -0400 Subject: [bug#32980] [PATCH 0/2] Multiplexed build output from the daemon Resent-Message-ID: Received: from eggs.gnu.org ([208.118.235.92]:57047) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1g9HYm-0007S5-5E for guix-patches@gnu.org; Sun, 07 Oct 2018 18:30:01 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1g9HYk-0006mJ-Vd for guix-patches@gnu.org; Sun, 07 Oct 2018 18:29:59 -0400 From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Date: Mon, 8 Oct 2018 00:29:45 +0200 Message-Id: <20181007222945.2077-1-ludo@gnu.org> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+kyle=kyleam.com@gnu.org Sender: "Guix-patches" To: 32980@debbugs.gnu.org Hello Guix! This patch set is partly a response to the problem Ricardo raised at: https://lists.gnu.org/archive/html/guix-devel/2018-09/msg00322.html and partly a natural followup to the addition of (guix status): https://issues.guix.info/issue/32837 So far the daemon would send all its output (messages it writes as well as messages any of its build processes writes) directly as a single stream to its clients. This had several shortcomings: 1. Clients could not distinguish messages coming from the daemon (such as build traces or ‘guix substitute’ messages) from messages coming from build processes. (Notably build processes could “forge” build traces such as “@ build-started”.) 2. When max-jobs > 1, clients would receive intermingled output from all the build processes without any way to disentangle it. 3. Build traces written by the daemon were expected to start on a new line but a builder could write output not terminated by a newline (e.g., “checking for …” messages from ./configure) and consequently build traces would not start on a new line and would go unnoticed by (guix status) and co. With this change clients can optionally ask for “multiplexed build output”. When it’s enabled, build output is prefixed by a special trace, like this: @ build-output 1234 21 checking for fcntl... where 1234 is the PID of the build process speaking and 21 is the number of bytes in the following build output fragment. The PID is first given in the corresponding “@ build-started” trace. On the client side, (guix status) is adjusted to produce events like: (build-log #f MESSAGE) for a message coming from the daemon, and: (build-log PID MESSAGE) for a message coming from PID. The downside of the protocol is that it creates quite some overhead. For example, when extracting a tarball, we see things like: read(13, "gmlo\0\0\0\0", 8) = 8 read(13, "5\0\0\0\0\0\0\0", 8) = 8 read(13, "@ build-output 25935 29\ncoreutils-8.29/m4/fseterr.m4\n", 53) = 53 read(13, "\0\0\0", 3) = 3 That is, a 29-byte message with a 24-byte header (plus the 8 + 8 + 3 = 19 bytes of the underlying protocol; see ‘process-stderr’.) Another option would be to incorporate multiplexing in the lower-level binary protocol. However, the client side would need potentially bigger changes: the single ‘current-build-output-port’ sink would no longer be a good match. Thoughts? Ludo’. Ludovic Courtès (2): daemon: Support multiplexed build output. status: Build upon multiplexed build output. guix/scripts/build.scm | 3 + guix/scripts/environment.scm | 1 + guix/scripts/pack.scm | 1 + guix/scripts/package.scm | 3 +- guix/scripts/pull.scm | 1 + guix/scripts/system.scm | 1 + guix/status.scm | 165 ++++++++++++++++++++++++-------- guix/store.scm | 15 ++- nix/libstore/build.cc | 25 +++-- nix/libstore/globals.cc | 2 + nix/libstore/globals.hh | 9 +- nix/libstore/worker-protocol.hh | 2 +- nix/nix-daemon/nix-daemon.cc | 2 +- tests/status.scm | 47 ++++++++- tests/store.scm | 63 ++++++++++++ 15 files changed, 289 insertions(+), 51 deletions(-) -- 2.19.0