From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp10.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms5.migadu.com with LMTPS id SE+mCNnirWP1QgAAbAwnHQ (envelope-from ) for ; Thu, 29 Dec 2022 19:56:25 +0100 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp10.migadu.com with LMTPS id iGzWB9nirWNHnwAAG6o9tA (envelope-from ) for ; Thu, 29 Dec 2022 19:56:25 +0100 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 6D63A2EF1E for ; Thu, 29 Dec 2022 19:56:24 +0100 (CET) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pAxs7-00084I-IX; Thu, 29 Dec 2022 13:43:19 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pAxrr-0007z5-2V for guix-patches@gnu.org; Thu, 29 Dec 2022 13:43:04 -0500 Received: from debbugs.gnu.org ([209.51.188.43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1pAxrq-0006mJ-Eq for guix-patches@gnu.org; Thu, 29 Dec 2022 13:43:02 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1pAxrq-00088V-AP for guix-patches@gnu.org; Thu, 29 Dec 2022 13:43:02 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#60408] [PATCH wip] guix: Support showing status in parallel. Resent-From: Julien Lepiller Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Thu, 29 Dec 2022 18:43:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: report 60408 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 60408@debbugs.gnu.org X-Debbugs-Original-To: guix-patches@gnu.org Received: via spool by submit@debbugs.gnu.org id=B.167233936931251 (code B ref -1); Thu, 29 Dec 2022 18:43:02 +0000 Received: (at submit) by debbugs.gnu.org; 29 Dec 2022 18:42:49 +0000 Received: from localhost ([127.0.0.1]:32852 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pAxrc-00087y-2s for submit@debbugs.gnu.org; Thu, 29 Dec 2022 13:42:49 -0500 Received: from lists.gnu.org ([209.51.188.17]:59066) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pAxrY-00087p-UU for submit@debbugs.gnu.org; Thu, 29 Dec 2022 13:42:46 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pAxrY-0007w3-6n for guix-patches@gnu.org; Thu, 29 Dec 2022 13:42:44 -0500 Received: from lepiller.eu ([2a00:5884:8208::1]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pAxrU-0006iW-UH for guix-patches@gnu.org; Thu, 29 Dec 2022 13:42:43 -0500 Received: from lepiller.eu (localhost [127.0.0.1]) by lepiller.eu (OpenSMTPD) with ESMTP id da4e5937 for ; Thu, 29 Dec 2022 18:42:31 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed; d=lepiller.eu; h=date:from :to:subject:message-id:mime-version:content-type; s=dkim; bh=11L AcLwX5WV2Sa1J1RtrT3tTg0Ywy/ZXSdMsNszpC0A=; b=iUFRNjPwZscRvbns+HS v3NoJ1xUud0SDPTgUAssKmxlcMRUuCrNKI7heyS6h23uSBbT79DMiZ4nJe+XBW7E Bez1GgrJooYWIpgl7ez1uHR+BRyiH6FwGj0EGNyoVaFxYWA2iw6bG/vvXIzYl3lC /11lhgesgTjiDtbZJAGYyC/EiLmTo6KmYZlP1unSFXTFQUlOkMlBn2GFt4+r+qtS gmjrunoeZ34pXcbmbQecBnCvhrUmXPBjA2BHbap3TrraXGMulACnRRaxL2Dn6UvR 48QVcFXPFYCKWogHHNGwUkg3FhvcWvo9QnmQ8sj5mkbgUsxxzfB8SObqqoxFToOF X0g== Received: by lepiller.eu (OpenSMTPD) with ESMTPSA id a17d1b78 (TLSv1.3:AEAD-AES256-GCM-SHA384:256:NO) for ; Thu, 29 Dec 2022 18:42:30 +0000 (UTC) Date: Thu, 29 Dec 2022 19:42:12 +0100 From: Julien Lepiller Message-ID: <20221229194212.44e5cabf@tachikoma.lepiller.eu> X-Mailer: Claws Mail 4.1.1 (GTK 3.24.30; x86_64-pc-linux-gnu) MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="MP_/3mhdKAe/CGQI_cITvmS0eBI" Received-SPF: pass client-ip=2a00:5884:8208::1; envelope-from=julien@lepiller.eu; helo=lepiller.eu X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+larch=yhetil.org@gnu.org Sender: guix-patches-bounces+larch=yhetil.org@gnu.org X-Migadu-Country: US X-Migadu-Flow: FLOW_IN ARC-Seal: i=1; s=key1; d=yhetil.org; t=1672340184; a=rsa-sha256; cv=none; b=rFgaIVxvgnjOHlXJN6lAGpX9qWnob8uDMKb7HWoaU6wRxvcU8U53p+aSRqlPKSbZ9atfGp lDYnuz6/4prsyVwpj/TyPLKb+Cic1NE5Ww3KytbZ9h3coMI5yhLi//JaRp4v/Wx+ii8hUN 0KCaLlA2/yCCPvp0pvNFz7DBhISTHZJDLDKES4g12XlVA67vbdu2C+Ma2gcjYghBHWakd3 j0ytgFF+VSVeIAztYPdkwaMkHaoEcgz2lYjtZFTejMwD6UQX8nyVLecpt9bGabgFWlyef9 dhty68zHffA48WjcONaUMbWfZvC7/ZCmaQFudb5uTm3nEr4d5Ov/9HlEzF30sg== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=lepiller.eu header.s=dkim header.b=iUFRNjPw; spf=pass (aspmx1.migadu.com: domain of "guix-patches-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-patches-bounces+larch=yhetil.org@gnu.org"; dmarc=fail reason="SPF not aligned (relaxed)" header.from=lepiller.eu (policy=none) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1672340184; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version: content-type:content-type:resent-cc:resent-from:resent-sender: resent-message-id:list-id:list-help:list-unsubscribe:list-subscribe: list-post:dkim-signature; bh=m3gfZySoXBnuURWwYDCB1Q16Y87k1xHqotBxBbPMBeA=; b=hI+6hHcI55WONFHSVLTwOMJJzvUWF2FIioqocOY5KaTrYizrA0o1vXPExAaJgSNLpq/ipq JFo+lRT8p5BBsRYGVSv7BKhuS7CbhVTf18XSnjl+MRNPBpH1mKtM6CeiFBqUefS5BwDyxL 4c6U58PU2gmnMbzH2piVThcXR5vsKlE8jN9XoAMYukmexsAm5Tbg0rPNyL9uu8vjNUbcGe i2XdrMWGJ2x2BQvNfHcisyFAyZX2LCGqMx/K2W9zPCJf0gg7NMKdaTLFpkf3uHuKdIRzT0 tnXGxukORg7mkl+MSY9z7Kk1Aetm+lKnvLFYBoPrrAvHRAC+Ml5u3Z/6/9e9vQ== X-Spam-Score: -1.43 X-Migadu-Queue-Id: 6D63A2EF1E Authentication-Results: aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=lepiller.eu header.s=dkim header.b=iUFRNjPw; spf=pass (aspmx1.migadu.com: domain of "guix-patches-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-patches-bounces+larch=yhetil.org@gnu.org"; dmarc=fail reason="SPF not aligned (relaxed)" header.from=lepiller.eu (policy=none) X-Migadu-Scanner: scn0.migadu.com X-Migadu-Spam-Score: -1.43 X-TUID: /T2qWG/2rK3D --MP_/3mhdKAe/CGQI_cITvmS0eBI Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Content-Disposition: inline Hi Guix! The attached patch allows showing more detailed status in parallel. One of the reasons for doing that is for supporting another of my patches, that allows specifying download tasks and build tasks numbers separately, with a default of 1 for each (so 2 tasks in parallel by default). With verbosity level 2, all messages from all builds are shown (interleaved, obviously), and messages for other levels are the same. When only one build is performed, no changes are visible. When multiple builds are happening in parallel, this patch shows one line per running job (whether build or download), and messages above. This will look like this: module-import-compiled.drv 75% =E2=96=95=E2=96=88=E2=96=88=E2=96=88=E2=96= =88=E2=96=88=E2=96=88=E2=96=88=E2=96=88=E2=96=88=E2=96=88=E2=96=88=E2=96=88= =E2=96=88=E2=96=88=E2=96=88=E2=96=88=E2=96=88=E2=96=88=E2=96=88=E2=96=88=E2= =96=88=E2=96=88=E2=96=88=E2=96=88=E2=96=88 =E2=96=8F openjdk-9.181-jdk 337.6MiB 4.3MiB/s 00:07 =E2=96=95=E2=96=88=E2=96=8C = =E2=96=8F 8.8% If there is no progress lines in the output of a build, it doesn't show a spinner (yet), so no feedback that something is happening, but that's planned for v2. One other issue is probably caused by having lots of events (caused by build log output), that make status lines blink. Another issue is when you ^C, the cursor is on the first status line. My ideas for this patch are: First, it's possible to go back a few lines with an ANSI escape code, so basically print all status lines, then go back to the first line. When an even needs to print a new line, let it do it from the first status line, and print the status lines from below it. Sometimes, an event will contain an incomplete line (for instance, it ends with \r instead of \n), so I want to record the line to prevent it from being overwritten by a status line. It's printed together with the status lines, so we can go back to that line and print the rest of the line when we get more of it. Thoughts, ideas? --MP_/3mhdKAe/CGQI_cITvmS0eBI Content-Type: text/x-patch Content-Transfer-Encoding: quoted-printable Content-Disposition: attachment; filename=0001-guix-Support-showing-status-in-parallel.patch =46rom c40fc712dec93299657e916907bc603d30178327 Mon Sep 17 00:00:00 2001 Message-Id: From: Julien Lepiller Date: Thu, 29 Dec 2022 19:20:34 +0100 Subject: [PATCH] guix: Support showing status in parallel. * guix/status.scm (build-status): Add `last-daemon-line` field. (build): Add `last-line` and `start` fields. (update-build): Record partial lines (not ending with \n) in the last-line field of the new build or status record. (print-build-event): Always print status of all current builds and downloads at the end. Update all status lines. --- guix/status.scm | 358 +++++++++++++++++++++++++++++++++--------------- 1 file changed, 245 insertions(+), 113 deletions(-) diff --git a/guix/status.scm b/guix/status.scm index 2c69f49fb5..5eb3ebc46b 100644 --- a/guix/status.scm +++ b/guix/status.scm @@ -49,6 +49,7 @@ (define-module (guix status) build-status-downloading build-status-builds-completed build-status-downloads-completed + build-status-last-daemon-line =20 build? build @@ -57,6 +58,8 @@ (define-module (guix status) build-log-file build-phase build-completion + build-start + build-last-line =20 download? download @@ -100,11 +103,13 @@ (define-record-type* build-status make= -build-status (builds-completed build-status-builds-completed ;list of (default '())) (downloads-completed build-status-downloads-completed ;list of - (default '()))) + (default '())) + (last-daemon-line build-status-last-daemon-line ;string + (default ""))) =20 ;; On-going or completed build. (define-immutable-record-type - (%build derivation id system log-file phase completion) + (%build derivation id system log-file phase completion start last-line) build? (derivation build-derivation) ;string (.drv file name) (id build-id) ;#f | integer @@ -113,11 +118,17 @@ (define-immutable-record-type (phase build-phase ;#f | symbol set-build-phase) (completion build-completion ;#f | integer (percentage) - set-build-completion)) + set-build-completion) + (start build-start ;