From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp0.migadu.com ([2001:41d0:403:4876::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms8.migadu.com with LMTPS id 0AfaL8fR2GX2CgEAqHPOHw:P1 (envelope-from ) for ; Fri, 23 Feb 2024 18:11:35 +0100 Received: from aspmx1.migadu.com ([2001:41d0:403:4876::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp0.migadu.com with LMTPS id 0AfaL8fR2GX2CgEAqHPOHw (envelope-from ) for ; Fri, 23 Feb 2024 18:11:35 +0100 X-Envelope-To: larch@yhetil.org Authentication-Results: aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=gnu.org header.s=fencepost-gnu-org header.b="PfnZUX/7"; spf=pass (aspmx1.migadu.com: domain of "guix-patches-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-patches-bounces+larch=yhetil.org@gnu.org"; dmarc=pass (policy=none) header.from=gnu.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1708708295; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:resent-cc: resent-from:resent-sender:resent-message-id:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post:dkim-signature; bh=+s4shKIpRGvto9YZ7Yc35Q27K1Cb15aOsObxLoN+bs4=; b=fgT2dwUbaOqaUwHEvwqqa/15gBZVOMO09jMsJnyXDp+n3X8I1RUahKm1fofSle3uH/Tf/J q3AgehwI9dgohbn00AnkFjFFQjmNFjuQBHmp6RySBZ/oVWnByLau+I5IJQw8aXsiixYDxK u6JblyA3H0KkejdOFnP/8EwrlGeK+NoHjtgGPdbUbAcbUwqSlr7JOg62tGPG8V9bAISYOX hgZ3x4SUTv9Nb5SnipIWF/+01IZ6uNEduWRUxpFDL+JkTpVjRUQju04TcStWuXsSXcaHGC p73n3En29Cc7RaaJgKpfI1sj8b7vuR8rz5j4BqD9uySC1BQzt3LwTfkfbbNpPg== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=gnu.org header.s=fencepost-gnu-org header.b="PfnZUX/7"; spf=pass (aspmx1.migadu.com: domain of "guix-patches-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-patches-bounces+larch=yhetil.org@gnu.org"; dmarc=pass (policy=none) header.from=gnu.org ARC-Seal: i=1; s=key1; d=yhetil.org; t=1708708295; a=rsa-sha256; cv=none; b=b83c+pJRKmltcBhpAZozi25DgbNBlngHI9MYT5A5n7KFomdP/YGrVGaiWZsidKObzu+RxW uMH4k1zH4mcp100s6/qvojxe6sHhBP7ndprgXmZIreRhjtVKKiF5PElzQIkErYrFiKzFZY HYH5lioXPg5eY75iNuPOSiTlz7NK5sheWyu5pDrJ/6KHuOxzVT7iYja3THCOdSfviZdpNv habaW78gZh9YPN/vL3ozf1NzLlZ+EYv97AcFAjI7eK3qMlJG9oOU/BtlQ93hd1hopAjH3i OdBjmsVIvLNYGH8CgREdbm6t2e87uV+kmf0/KjWUYZHtal2DGZoI69/oFF9WYg== Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 9476975576 for ; Fri, 23 Feb 2024 18:11:35 +0100 (CET) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rdZ4F-0007mP-Af; Fri, 23 Feb 2024 12:10:35 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rdYpq-0007g9-6Z for guix-patches@gnu.org; Fri, 23 Feb 2024 11:55:42 -0500 Received: from debbugs.gnu.org ([2001:470:142:5::43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1rdYpn-0003u1-TS for guix-patches@gnu.org; Fri, 23 Feb 2024 11:55:40 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1rdYq9-0005SA-UO for guix-patches@gnu.org; Fri, 23 Feb 2024 11:56:01 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#69291] [PATCH 4/5] scripts: substitute: Untangle selecting fast vs small compressions. Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Fri, 23 Feb 2024 16:56:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 69291 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Christopher Baines Cc: Josselin Poiret , Simon Tournier , Mathieu Othacehe , Tobias Geerinckx-Rice , 69291@debbugs.gnu.org, Ricardo Wurmus , Christopher Baines Received: via spool by 69291-submit@debbugs.gnu.org id=B69291.170870731920826 (code B ref 69291); Fri, 23 Feb 2024 16:56:01 +0000 Received: (at 69291) by debbugs.gnu.org; 23 Feb 2024 16:55:19 +0000 Received: from localhost ([127.0.0.1]:53690 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rdYpT-0005Pj-0H for submit@debbugs.gnu.org; Fri, 23 Feb 2024 11:55:19 -0500 Received: from eggs.gnu.org ([209.51.188.92]:34142) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rdYbr-0004c2-2g for 69291@debbugs.gnu.org; Fri, 23 Feb 2024 11:41:16 -0500 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rdYNv-0006y6-RV; Fri, 23 Feb 2024 11:26:51 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:Date:References:In-Reply-To:Subject:To: From; bh=+s4shKIpRGvto9YZ7Yc35Q27K1Cb15aOsObxLoN+bs4=; b=PfnZUX/7w5mn0BmCUHbr rVxFYTQDKXaas76MCe1Vv8A9At820i4EUEVT9WTFYRmzW13/lIPP6BHE4agkbOhjlPW2Nl5iQAfOy pdeeQZrHj/+fzpWYVgNp9tm6tsXlswqYhSJ+GRRDksyeWnGjvf5W3Y5EO8S0zi0o7KaUwG3oBPC30 1JlceB/NnQgT38TCkb9qrNajhaqFIbTmSdYzWF/44pekAP2Rqn/TgT4CW3IluYbavHlrPIeuLgB0+ 0E4/BKKPqwMue5+oSE/C5TyARDmzESk7C/1N1EqbLE1DOQGEVnuiW/ik9JtqNtLuL6ynostymqwRP ir/EQ4aWdbgbuA==; From: Ludovic =?UTF-8?Q?Court=C3=A8s?= In-Reply-To: <2a4b5b57cb676c5a0149296d0251b269b2cebd6e.1708458147.git.mail@cbaines.net> (Christopher Baines's message of "Tue, 20 Feb 2024 19:42:26 +0000") References: <2a4b5b57cb676c5a0149296d0251b269b2cebd6e.1708458147.git.mail@cbaines.net> Date: Fri, 23 Feb 2024 17:26:43 +0100 Message-ID: <87sf1jf8os.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+larch=yhetil.org@gnu.org Sender: guix-patches-bounces+larch=yhetil.org@gnu.org X-Migadu-Flow: FLOW_IN X-Migadu-Country: US X-Migadu-Scanner: mx10.migadu.com X-Migadu-Spam-Score: -2.31 X-Spam-Score: -2.31 X-Migadu-Queue-Id: 9476975576 X-TUID: b7jcg867vFkM Christopher Baines skribis: > Pulling the logic up to the script makes this code more portable and not > reliant on setting a global variable. > > * guix/scripts/substitute.scm (%prefer-fast-decompression?): Rename to=E2= =80=A6 > (%default-prefer-fast-decompression?): this. > (display-narinfo-data): Update accordingly. > (download-nar): Add prefer-fast-decompression? as a keyword argument, rem= ove > code to set! it and return the cpu-usage recorded. > (process-substitution, process-substitution/fallback): Accept and pass th= rough > prefer-fast-decompression? to download-nar. > (guix-substitute): Move the prefer fast decompression switching logic her= e. > > Change-Id: I4e80b457b55bcda8c0ff4ee224dd94a55e1b24fb [...] > -(define %prefer-fast-decompression? > - ;; Whether to prefer fast decompression over good compression ratios. = This > - ;; serves in particular to choose between lzip (high compression ratio= but > - ;; low decompression throughput) and zstd (lower compression ratio but= high > - ;; decompression throughput). > - #f) > +(define %default-prefer-fast-decompression? #f) I would either remove this variable or add a comment describing it (we should do that for all top-level variables). > @@ -604,7 +585,9 @@ (define* (download-nar narinfo destination > (format status-port "hash-mismatch ~a ~a ~a~%" > (hash-algorithm-name algorithm) > (bytevector->nix-base32-string expected) > - (bytevector->nix-base32-string actual))))))) > + (bytevector->nix-base32-string actual)))) > + > + cpu-usage))) [...] > + (let ((cpu-usage > + (process-substitution reply-port store-path destin= ation > + #:cache-urls (substitute-url= s) > + #:acl (current-acl) > + #:deduplicate? deduplicate? > + #:print-build-trace? > + print-build-trace? > + #:prefer-fast-decompression? > + prefer-fast-decompression?))) > + > + ;; Create a hysteresis: depending on CPU usage, favor > + ;; compression methods with faster decompression (like = ztsd) > + ;; or methods with better compression ratios (like lzip= ). > + ;; This stems from the observation that substitution ca= n be > + ;; CPU-bound when high-speed networks are used: > + ;; . > + ;; To simulate "slow" networking or changing conditions= , run: > + ;; sudo tc qdisc add dev eno1 root tbf rate 512kbit lat= ency > + ;; 50ms burst 1540 and then cancel with: sudo tc qdisc = del > + ;; dev eno1 root > + (loop (cond > + ;; Whether to prefer fast decompression over good > + ;; compression ratios. This serves in particula= r to > + ;; choose between lzip (high compression ratio b= ut low > + ;; decompression throughput) and zstd (lower > + ;; compression ratio but high decompression > + ;; throughput). > + ((> cpu-usage .8) #t) > + ((< cpu-usage .2) #f) > + (else prefer-fast-decompression?))))))))) Instead of having =E2=80=98download-nar=E2=80=99 return its CPU usage, whic= h is surprising, maybe should wrap the =E2=80=98process-substitution=E2=80=99 ca= ll in =E2=80=98guix-substitute=E2=80=99 in =E2=80=98with-cpu-usage-monitoring=E2= =80=99 and keep all the logic in =E2=80=98guix-substitute=E2=80=99? Ludo=E2=80=99.