From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp0.migadu.com ([2001:41d0:303:e224::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms1.migadu.com with LMTPS id wLZdMOSRDWbNRgEAqHPOHw:P1 (envelope-from ) for ; Wed, 03 Apr 2024 19:29:08 +0200 Received: from aspmx1.migadu.com ([2001:41d0:303:e224::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp0.migadu.com with LMTPS id wLZdMOSRDWbNRgEAqHPOHw (envelope-from ) for ; Wed, 03 Apr 2024 19:29:08 +0200 X-Envelope-To: larch@yhetil.org Authentication-Results: aspmx1.migadu.com; dkim=none; spf=pass (aspmx1.migadu.com: domain of "guix-patches-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-patches-bounces+larch=yhetil.org@gnu.org"; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1712165348; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:resent-cc:resent-from:resent-sender: resent-message-id:in-reply-to:in-reply-to:references:references: list-id:list-help:list-unsubscribe:list-subscribe:list-post; bh=DLREsayPaQVJIHBLoR65+TSUJiP1GS6XvIbrtkB6Wlg=; b=P20JIIhRXmqRat4uAhlHQT1dMLKEdOcEMFbogOBu8KR6uaEwMstic81e5rvqrmqJIevygi BPKMhuF/Srx4gMe0E3gLGDA1zrJ7DJ3cLnyRVYYmYESzQgE5+iZrWtOWG7DIgGzMYHRYT3 TeqednBNcoxs9/d4rTGAewdQUUy3t7EOCYDcHiZrYDGqXcvSS2/xnKuJ0qa/M/bTWswMk7 IkEafTmDF+AdUDZFm6hE6/v90413MO1iYex/fAHsxa29mBFXIWm4n9OWm0/g8X9rB4TpDY bjaD+PCrzhUAeKI3XFG44K6rLWXulonHm5wysx5LKVe9zU3OswDarmFY3CFxLA== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1712165348; a=rsa-sha256; cv=none; b=mCiiE2EB9R2MSgLE/XtxM6WdOPSc+OswPKXq0gP5D8ShBQeKBL6Q4UwNdNUWMb49IUUkIO Mh5y5yEuG0/WQnGEtsUaaBdDaM3e1Z3ucgH5kpmeWdA2GiaqcO56+M/tETPDakEafWVyIS vD8+Gf3VmTDWxsckWg+tuHrfYbeJFVAHHQUhbj4gX1U03LMFalmpnC+VD2+gTm2oXvAViV L7OAKbVbLtSuNSYp4kDGCMvLctqgv5RNMeSpRTUT5uiA0yudm/av/Pv03RVQR/N5pLyJnj ohgNIPZFzVvj/HpKitZ2+kHZnDacJABQLZAScebfgJlXWyYt1ONnj6ADNUgMPA== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=none; spf=pass (aspmx1.migadu.com: domain of "guix-patches-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-patches-bounces+larch=yhetil.org@gnu.org"; dmarc=none Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 9EC9812972 for ; Wed, 3 Apr 2024 19:29:08 +0200 (CEST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rs4Q1-00008W-UC; Wed, 03 Apr 2024 13:29:02 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rs4Q0-00007C-B5 for guix-patches@gnu.org; Wed, 03 Apr 2024 13:29:00 -0400 Received: from debbugs.gnu.org ([2001:470:142:5::43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1rs4Px-0007iq-Tc for guix-patches@gnu.org; Wed, 03 Apr 2024 13:28:58 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1rs4Q2-0007RL-Ad for guix-patches@gnu.org; Wed, 03 Apr 2024 13:29:02 -0400 X-Loop: help-debbugs@gnu.org Subject: [bug#69291] [PATCH 4/5] scripts: substitute: Untangle selecting fast vs small compressions. Resent-From: Christopher Baines Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Wed, 03 Apr 2024 17:29:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 69291 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Ludovic =?UTF-8?Q?Court=C3=A8s?= Cc: 69291@debbugs.gnu.org Received: via spool by 69291-submit@debbugs.gnu.org id=B69291.171216530328455 (code B ref 69291); Wed, 03 Apr 2024 17:29:02 +0000 Received: (at 69291) by debbugs.gnu.org; 3 Apr 2024 17:28:23 +0000 Received: from localhost ([127.0.0.1]:59441 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rs4PO-0007Ot-Q6 for submit@debbugs.gnu.org; Wed, 03 Apr 2024 13:28:23 -0400 Received: from mira.cbaines.net ([212.71.252.8]:43298) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rs4PN-0007Ol-Bj for 69291@debbugs.gnu.org; Wed, 03 Apr 2024 13:28:22 -0400 Received: from localhost (unknown [212.132.255.10]) by mira.cbaines.net (Postfix) with ESMTPSA id 630B727BBE2; Wed, 3 Apr 2024 18:28:16 +0100 (BST) Received: from felis (localhost.lan [127.0.0.1]) by localhost (OpenSMTPD) with ESMTP id 3e8971ef; Wed, 3 Apr 2024 17:28:16 +0000 (UTC) From: Christopher Baines In-Reply-To: <87sf1jf8os.fsf@gnu.org> ("Ludovic =?UTF-8?Q?Court=C3=A8s?="'s message of "Fri, 23 Feb 2024 17:26:43 +0100") References: <2a4b5b57cb676c5a0149296d0251b269b2cebd6e.1708458147.git.mail@cbaines.net> <87sf1jf8os.fsf@gnu.org> User-Agent: mu4e 1.12.2; emacs 29.3 Date: Wed, 03 Apr 2024 18:28:15 +0100 Message-ID: <87o7aqjqz4.fsf@cbaines.net> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha512; protocol="application/pgp-signature" X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+larch=yhetil.org@gnu.org Sender: guix-patches-bounces+larch=yhetil.org@gnu.org X-Migadu-Flow: FLOW_IN X-Migadu-Country: US X-Migadu-Spam-Score: -8.46 X-Spam-Score: -8.46 X-Migadu-Queue-Id: 9EC9812972 X-Migadu-Scanner: mx12.migadu.com X-TUID: vyQ6LU31Gc3T --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Ludovic Court=C3=A8s writes: > Christopher Baines skribis: > >> Pulling the logic up to the script makes this code more portable and not >> reliant on setting a global variable. >> >> * guix/scripts/substitute.scm (%prefer-fast-decompression?): Rename to= =E2=80=A6 >> (%default-prefer-fast-decompression?): this. >> (display-narinfo-data): Update accordingly. >> (download-nar): Add prefer-fast-decompression? as a keyword argument, re= move >> code to set! it and return the cpu-usage recorded. >> (process-substitution, process-substitution/fallback): Accept and pass t= hrough >> prefer-fast-decompression? to download-nar. >> (guix-substitute): Move the prefer fast decompression switching logic he= re. >> >> Change-Id: I4e80b457b55bcda8c0ff4ee224dd94a55e1b24fb > > [...] > >> -(define %prefer-fast-decompression? >> - ;; Whether to prefer fast decompression over good compression ratios.= This >> - ;; serves in particular to choose between lzip (high compression rati= o but >> - ;; low decompression throughput) and zstd (lower compression ratio bu= t high >> - ;; decompression throughput). >> - #f) >> +(define %default-prefer-fast-decompression? #f) > > I would either remove this variable or add a comment describing it (we > should do that for all top-level variables). I've added a comment now, and I'll sent an updated patch. >> @@ -604,7 +585,9 @@ (define* (download-nar narinfo destination >> (format status-port "hash-mismatch ~a ~a ~a~%" >> (hash-algorithm-name algorithm) >> (bytevector->nix-base32-string expected) >> - (bytevector->nix-base32-string actual))))))) >> + (bytevector->nix-base32-string actual)))) >> + >> + cpu-usage))) > > [...] > >> + (let ((cpu-usage >> + (process-substitution reply-port store-path desti= nation >> + #:cache-urls (substitute-ur= ls) >> + #:acl (current-acl) >> + #:deduplicate? deduplicate? >> + #:print-build-trace? >> + print-build-trace? >> + #:prefer-fast-decompression? >> + prefer-fast-decompression?)= )) >> + >> + ;; Create a hysteresis: depending on CPU usage, favor >> + ;; compression methods with faster decompression (like= ztsd) >> + ;; or methods with better compression ratios (like lzi= p). >> + ;; This stems from the observation that substitution c= an be >> + ;; CPU-bound when high-speed networks are used: >> + ;; . >> + ;; To simulate "slow" networking or changing condition= s, run: >> + ;; sudo tc qdisc add dev eno1 root tbf rate 512kbit la= tency >> + ;; 50ms burst 1540 and then cancel with: sudo tc qdisc= del >> + ;; dev eno1 root >> + (loop (cond >> + ;; Whether to prefer fast decompression over go= od >> + ;; compression ratios. This serves in particul= ar to >> + ;; choose between lzip (high compression ratio = but low >> + ;; decompression throughput) and zstd (lower >> + ;; compression ratio but high decompression >> + ;; throughput). >> + ((> cpu-usage .8) #t) >> + ((< cpu-usage .2) #f) >> + (else prefer-fast-decompression?))))))))) > > > Instead of having =E2=80=98download-nar=E2=80=99 return its CPU usage, wh= ich is > surprising, maybe should wrap the =E2=80=98process-substitution=E2=80=99 = call in > =E2=80=98guix-substitute=E2=80=99 in =E2=80=98with-cpu-usage-monitoring= =E2=80=99 and keep all the logic > in =E2=80=98guix-substitute=E2=80=99? Yeah, that makes sense. I'll send an updated patch shortly. --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQKlBAEBCgCPFiEEPonu50WOcg2XVOCyXiijOwuE9XcFAmYNka9fFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNF ODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcRHG1haWxAY2Jh aW5lcy5uZXQACgkQXiijOwuE9XdEWA//XHWY7mjjHWzznRmKks7rlBVFWjgjW60g 2/UVJp6YGz2DPunTKMcVPPF3+PF47JbuPZbPf4fPli17oh861QKJ30LFXtFRFomo hmOAdW52yGjIZOBBE77wDmxgOTQctzuTDXl+xfdVMHptDt7HmsMmeFWs6mvGnxI+ 1pUM4UjN0rDoEnjgX0pw3SUMlxao2Hfh77oOvgmb6AkVArTctcyOiAKQ6FojVqt8 QKb19eCNPudJhNR0fgnX1CEBQkif8S8xgHzj3nG+YKQ3RRxtexebh9oTBA1Uc9xE apNpwRwOcLaJE2IblAM2CJ5EMHnLDFSh6jyKW2M6R/G7Of8MtWX9xvQQy3hdAc6L KWjQhdyQYWazoOVOmxq/AnAQnAJo+ZaOVraM07ZTPpjBlCnG1vCYrlTRbrFQKosP qSDqz3jn7lPNTXNWP3RNtNBZU94eXf3ODlHmeHv6v12jWVj91qG3Xm4I7kXQTvS9 2/ux3InIitJJs3YopC1TtjJkTSCPSyl28kVEvAeHn6nNQWd0hVJG5Ik8VKi67ePK qjDD923+8WoC5P9lrYfuF6NFmzhxTX2UHIUEtqDauE4m3FQsidrbK8x3/fLR6hGc qPDcWKvNwL8dNv7a2dOY1SZDh6dK5q+/QMSM9jNSCeS6WeyKazVdFtnP4uj3T9xd AuUqofni73E= =iiwd -----END PGP SIGNATURE----- --=-=-=--