From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp2 ([2001:41d0:8:6d80::]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id mJQBJ5o7Y2DYQgAAgWs5BA (envelope-from ) for ; Tue, 30 Mar 2021 16:54:18 +0200 Received: from aspmx1.migadu.com ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp2 with LMTPS id qP4FIZo7Y2CibwAAB5/wlQ (envelope-from ) for ; Tue, 30 Mar 2021 14:54:18 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id AAE792A291 for ; Tue, 30 Mar 2021 16:54:17 +0200 (CEST) Received: from localhost ([::1]:42040 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lRFl1-0006Ej-OG for larch@yhetil.org; Tue, 30 Mar 2021 10:54:15 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:41414) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lRFd5-0004a5-36 for bug-guix@gnu.org; Tue, 30 Mar 2021 10:46:03 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:41007) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1lRFd4-0000lj-P7 for bug-guix@gnu.org; Tue, 30 Mar 2021 10:46:02 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1lRFd4-0002gL-Jh for bug-guix@gnu.org; Tue, 30 Mar 2021 10:46:02 -0400 X-Loop: help-debbugs@gnu.org Subject: bug#41930: =?UTF-8?Q?=E2=80=98guix_?= =?UTF-8?Q?pull=E2=80=99?= shows raw build log output Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: bug-guix@gnu.org Resent-Date: Tue, 30 Mar 2021 14:46:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 41930 X-GNU-PR-Package: guix X-GNU-PR-Keywords: To: 41930@debbugs.gnu.org Received: via spool by 41930-submit@debbugs.gnu.org id=B41930.16171155097005 (code B ref 41930); Tue, 30 Mar 2021 14:46:02 +0000 Received: (at 41930) by debbugs.gnu.org; 30 Mar 2021 14:45:09 +0000 Received: from localhost ([127.0.0.1]:52549 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lRFcC-0001o0-LA for submit@debbugs.gnu.org; Tue, 30 Mar 2021 10:45:09 -0400 Received: from eggs.gnu.org ([209.51.188.92]:48080) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lRFc7-0001gU-LB for 41930@debbugs.gnu.org; Tue, 30 Mar 2021 10:45:07 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:38537) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lRFc2-0000B5-Dx for 41930@debbugs.gnu.org; Tue, 30 Mar 2021 10:44:58 -0400 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=53998 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1lRFc1-0001nm-Rg for 41930@debbugs.gnu.org; Tue, 30 Mar 2021 10:44:58 -0400 From: Ludovic =?UTF-8?Q?Court=C3=A8s?= References: <87o8pgjwuc.fsf@inria.fr> Date: Tue, 30 Mar 2021 16:44:56 +0200 In-Reply-To: <87o8pgjwuc.fsf@inria.fr> ("Ludovic =?UTF-8?Q?Court=C3=A8s?="'s message of "Thu, 18 Jun 2020 16:02:35 +0200") Message-ID: <87im58g0pj.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.1 (gnu/linux) MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-guix@gnu.org List-Id: Bug reports for GNU Guix List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-guix-bounces+larch=yhetil.org@gnu.org Sender: "bug-Guix" X-Migadu-Flow: FLOW_IN ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1617116057; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version: content-type:content-type:resent-cc:resent-from:resent-sender: resent-message-id:in-reply-to:in-reply-to:references:references: list-id:list-help:list-unsubscribe:list-subscribe:list-post; bh=IBVQ1sBmv+E3wRppZQqxSa9qLvZ9e0vHUrG9d7oBl0M=; b=A1DyKgSdfi0IkQO15pHlJkxD+fXEvpCekiBmvjlwAs/dO6UyOGvfRzr40ibnYvqQCXHbD0 V0jpimBCnQrWn4Td+YKde+z40QLsxUvqNy2VkuGFBuLtHmELdftsc4OFM3cB8DPI4jkN2V vQ0rOudwJbWUFLEWPQSLu1hymY6kYwURXat23CwAeOO77QlgxsZCNsgYZvL63K5eQzWo21 REu33SO1yTkjUSl6TgTGfYGaDhNURW+eIrHdNumUtv8P0qApwVxJ5teU+JCAME78iRjD08 icehUv8tL1AYS5iVn9KZ9wZNmj3FdPYicDSvhwyjCmV+Qlk4V8mGFjFeUSBtpg== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1617116057; a=rsa-sha256; cv=none; b=UrdbgXEnGrSH5x3H2ZnZkuOEDI4QT90pISlwii9MaIKfamKVmhIPQoZ8q9Sto4OMeVrTnV Tg/TE/G/JetdHN/gfMSkRxKdO6SCkjo4DfH1pf8ML0siEc6QVXhe5izebabUDwari2eDCX H/QenBd1WcLqQSahQRW244VdDW+mJplHzGI+eCdQNazBryMtqenkEp7jj+dvrYaaj916wf jmVMlUg8qpvVlfIjhQtmET1hDRjT3jm0JB2GlOdXwxFz5Qcwf+IbLM6j++surs8D/jGQJH 1xJLoN9JInDr4EJ3rLhjZlr2PwqSFP4w3Oz2oO6GdkM1wx4O2BYQOUvvNRRS+g== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=none; dmarc=pass (policy=none) header.from=gnu.org; spf=pass (aspmx1.migadu.com: domain of bug-guix-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=bug-guix-bounces@gnu.org X-Migadu-Spam-Score: -2.92 Authentication-Results: aspmx1.migadu.com; dkim=none; dmarc=pass (policy=none) header.from=gnu.org; spf=pass (aspmx1.migadu.com: domain of bug-guix-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=bug-guix-bounces@gnu.org X-Migadu-Queue-Id: AAE792A291 X-Spam-Score: -2.92 X-Migadu-Scanner: scn0.migadu.com X-TUID: Nf9mrC0+ioO4 --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Hi, Ludovic Court=C3=A8s skribis: > =E2=80=98guix pull=E2=80=99 & co. show raw =E2=80=9Cdetailed log=E2=80=9D= output for things > built/downloaded while building =E2=80=98compute-guix-derivation.drv=E2= =80=99: > > $ guix time-machine -- build =E2=80=A6 > Updating channel 'guix' from Git repository at 'https://git.savannah.gnu.= org/git/guix.git'... [...] > @ substituter-started /gnu/store/frg642g7pxh95cdahar2s884ig82i1xn-nghttp2= -1.41.0-lib substitute > @ substituter-started /gnu/store/v5rgf8v6cjxjbngkzjcznj98dkxj8svg-jansson= -2.12 substitute > @ substituter-started /gnu/store/23d9f16wbv22qin71ac32hql81bzzkab-libev-4= .31 substitute > @ download-started /gnu/store/23d9f16wbv22qin71ac32hql81bzzkab-libev-4.31= https://ci.guix.gnu.org/nar/lzip/23d9f16wbv22qin71ac32hql81bzzkab-libev-4.= 31 124297 > @ download-started /gnu/store/v5rgf8v6cjxjbngkzjcznj98dkxj8svg-jansson-2.= 12 https://ci.guix.gnu.org/nar/lzip/v5rgf8v6cjxjbngkzjcznj98dkxj8svg-jansso= n-2.12 27840 The attached patch fixes that in an unimaginative but efficient fashion: 1. the parent process (which runs =E2=80=98build-self.scm=E2=80=99) accep= ts connections on a named socket; 2. the =E2=80=98compute-guix-derivation=E2=80=99 process connects to that= socket and sends it its raw build output (what we see in the snippet above); 3. the parent process reads that and sends it to its own (current-build-output-port); that port processes those =E2=80=9C@=E2= =80=9D build traces according to the current =E2=80=98--verbosity=E2=80=99=E2=80=94= see (guix status). With this in place, builds or downloads triggered during the evaluation of =E2=80=98compute-guix-derivation=E2=80=99 are reported in a consistent w= ay from a UI viewpoint. There was one remaining glitch: the spinner that =E2=80=98compute-guix-derivation=E2=80=99 prints would show up in the middl= e of the prettified build output. The second patch addresses that. Feedback welcome! Ludo=E2=80=99. --=-=-= Content-Type: text/x-patch Content-Disposition: inline; filename=0001-build-self-Forward-sub-process-build-output-to-curre.patch >From 882c47b51223c6002eaec79466106a29ac84c613 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ludovic=20Court=C3=A8s?= Date: Tue, 30 Mar 2021 16:07:26 +0200 Subject: [PATCH 1/2] build-self: Forward sub-process build output to (current-build-output-port). Fixes . * build-aux/build-self.scm (build-program): Add extra 'build-output' parameter. Interpret it as a socket name and connect to it; use it as the CURRENT-BUILD-OUTPUT-PORT. (proxy): New procedure. (build): Open a named socket. Accept connections and call 'proxy' on it. --- build-aux/build-self.scm | 90 +++++++++++++++++++++++++++++----------- 1 file changed, 65 insertions(+), 25 deletions(-) diff --git a/build-aux/build-self.scm b/build-aux/build-self.scm index dd845d1596..3e057ca5d2 100644 --- a/build-aux/build-self.scm +++ b/build-aux/build-self.scm @@ -336,7 +336,8 @@ interface (FFI) of Guile.") (loop (cdr spin))))) (match (command-line) - ((_ source system version protocol-version) + ((_ source system version protocol-version + build-output) ;; The current input port normally wraps a file ;; descriptor connected to the daemon, or it is ;; connected to /dev/null. In the former case, reuse @@ -349,16 +350,22 @@ interface (FFI) of Guile.") (current-input-port) "w+0") #:version proto) - (open-connection)))) + (open-connection))) + (sock (socket AF_UNIX SOCK_STREAM 0))) (call-with-new-thread (lambda () (spin system))) + ;; Connect to BUILD-OUTPUT and send it the raw + ;; build output. + (connect sock AF_UNIX build-output) + (display (and=> ;; Silence autoload warnings and the likes. (parameterize ((current-warning-port - (%make-void-port "w"))) + (%make-void-port "w")) + (current-build-output-port sock)) (run-with-store store (guix-derivation source version #$guile-version @@ -370,6 +377,20 @@ interface (FFI) of Guile.") derivation-file-name)))))) #:module-path (list source)))) +(define (proxy input output) + "Dump the contents of INPUT to OUTPUT until EOF is reached on INPUT." + (setvbuf input 'block 16384) + (let loop () + (match (select (list input) '() '()) + ((() () ()) + (loop)) + (((_) () ()) + ;; Read from INPUT as much as can be read without blocking. + (let ((bv (get-bytevector-some input))) + (unless (eof-object? bv) + (put-bytevector output bv) + (loop))))))) + (define (call-with-clean-environment thunk) (let ((env (environ))) (dynamic-wind @@ -426,7 +447,14 @@ files." ;; way, we know 'open-pipe*' will not close it on 'exec'. If PORT is ;; not a file port (e.g., it's an SSH channel), then the subprocess's ;; stdin will actually be /dev/null. - (let* ((pipe (with-input-from-port port + (let* ((sock (socket AF_UNIX SOCK_STREAM 0)) + (node (let ((file (string-append (or (getenv "TMPDIR") "/tmp") + "/guix-build-output-" + (number->string (getpid))))) + (bind sock AF_UNIX file) + (listen sock 1) + file)) + (pipe (with-input-from-port port (lambda () ;; Make sure BUILD is not influenced by ;; $GUILE_LOAD_PATH & co. @@ -442,30 +470,42 @@ files." (if (file-port? port) (number->string (logior major minor)) - "none")))))) - (str (get-string-all pipe)) - (status (close-pipe pipe))) - (match str - ((? eof-object?) - (error "build program failed" (list build status))) - ((? derivation-path? drv) - (mbegin %store-monad - (return (newline (current-error-port))) - ((store-lift add-temp-root) drv) - (return (read-derivation-from-file drv)))) - ("#f" - ;; Unsupported PULL-VERSION. - (return #f)) - ((? string? str) - (raise (condition - (&message - (message (format #f "You found a bug: the program '~a' + "none") + node)))))) + ;; Wait for a connection on SOCK and proxy build output so it can be + ;; processed according to the settings currently in effect (build + ;; traces, verbosity level, and so on). + (match (accept sock) + ((port . _) + (close-port sock) + (delete-file node) + (proxy port (current-build-output-port)))) + + ;; Now that the build output connection was closed, read the result, a + ;; derivation file name, from PIPE. + (let ((str (get-string-all pipe)) + (status (close-pipe pipe))) + (match str + ((? eof-object?) + (error "build program failed" (list build status))) + ((? derivation-path? drv) + (mbegin %store-monad + (return (newline (current-error-port))) + ((store-lift add-temp-root) drv) + (return (read-derivation-from-file drv)))) + ("#f" + ;; Unsupported PULL-VERSION. + (return #f)) + ((? string? str) + (raise (condition + (&message + (message (format #f "You found a bug: the program '~a' failed to compute the derivation for Guix (version: ~s; system: ~s; host version: ~s; pull-version: ~s). Please report it by email to <~a>.~%" - (derivation->output-path build) - version system %guix-version pull-version - %guix-bug-report-address))))))))))) + (derivation->output-path build) + version system %guix-version pull-version + %guix-bug-report-address)))))))))))) ;; This file is loaded by 'guix pull'; return it the build procedure. build -- 2.31.0 --=-=-= Content-Type: text/x-patch; charset=utf-8 Content-Disposition: inline; filename=0002-build-self-Take-care-of-the-spinner-in-the-parent-pr.patch Content-Transfer-Encoding: quoted-printable >From 7710dca2453667f61f866be0115d632efa93a3b3 Mon Sep 17 00:00:00 2001 From: =3D?UTF-8?q?Ludovic=3D20Court=3DC3=3DA8s?=3D Date: Tue, 30 Mar 2021 16:35:05 +0200 Subject: [PATCH 2/2] build-self: Take care of the spinner in the parent process. This simplifies code and mostly ensures we don't print a spinner while there's build activity going on. * build-aux/build-self.scm (build-program): Remove 'spin' and 'call-with-new-thread' call from "compute-guix-derivation" body. Remove "Computing Guix derivation" message. (proxy): Pass extra argument to 'select'. Display a spinner when 'select' returns empty lists. (build): Print "Computing Guix derivation" message here. --- build-aux/build-self.scm | 43 ++++++++++++++++------------------------ 1 file changed, 17 insertions(+), 26 deletions(-) diff --git a/build-aux/build-self.scm b/build-aux/build-self.scm index 3e057ca5d2..853a2f328f 100644 --- a/build-aux/build-self.scm +++ b/build-aux/build-self.scm @@ -285,8 +285,7 @@ interface (FFI) of Guile.") #:select? select?)) (gexp->script "compute-guix-derivation" #~(begin - (use-modules (ice-9 match) - (ice-9 threads)) + (use-modules (ice-9 match)) =20 (eval-when (expand load eval) ;; (gnu packages =E2=80=A6) modules are going to b= e looked up @@ -320,21 +319,6 @@ interface (FFI) of Guile.") (guix derivations) (srfi srfi-1)) =20 - (define (spin system) - (define spin - (circular-list "-" "\\" "|" "/" "-" "\\" "|" "/"= )) - - (format (current-error-port) - "Computing Guix derivation for '~a'... " - system) - (when (isatty? (current-error-port)) - (let loop ((spin spin)) - (display (string-append "\b" (car spin)) - (current-error-port)) - (force-output (current-error-port)) - (sleep 1) - (loop (cdr spin))))) - (match (command-line) ((_ source system version protocol-version build-output) @@ -352,10 +336,6 @@ interface (FFI) of Guile.") #:version pro= to) (open-connection))) (sock (socket AF_UNIX SOCK_STREAM 0))) - (call-with-new-thread - (lambda () - (spin system))) - ;; Connect to BUILD-OUTPUT and send it the raw ;; build output. (connect sock AF_UNIX build-output) @@ -378,18 +358,26 @@ interface (FFI) of Guile.") #:module-path (list source)))) =20 (define (proxy input output) - "Dump the contents of INPUT to OUTPUT until EOF is reached on INPUT." + "Dump the contents of INPUT to OUTPUT until EOF is reached on INPUT. +Display a spinner when nothing happens." + (define spin + (circular-list "-" "\\" "|" "/" "-" "\\" "|" "/")) + (setvbuf input 'block 16384) - (let loop () - (match (select (list input) '() '()) + (let loop ((spin spin)) + (match (select (list input) '() '() 1) ((() () ()) - (loop)) + (when (isatty? (current-error-port)) + (display (string-append "\b" (car spin)) + (current-error-port)) + (force-output (current-error-port))) + (loop (cdr spin))) (((_) () ()) ;; Read from INPUT as much as can be read without blocking. (let ((bv (get-bytevector-some input))) (unless (eof-object? bv) (put-bytevector output bv) - (loop))))))) + (loop spin))))))) =20 (define (call-with-clean-environment thunk) (let ((env (environ))) @@ -472,6 +460,9 @@ files." (logior major minor)) "none") node)))))) + (format (current-error-port) "Computing Guix derivation for '~a'..= . " + system) + ;; Wait for a connection on SOCK and proxy build output so it can = be ;; processed according to the settings currently in effect (build ;; traces, verbosity level, and so on). --=20 2.31.0 --=-=-=--