From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp10.migadu.com ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms9.migadu.com with LMTPS id MIh2Eg6La2TgEgAASxT56A (envelope-from ) for ; Mon, 22 May 2023 17:32:30 +0200 Received: from aspmx1.migadu.com ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp10.migadu.com with LMTPS id 4DmnEQ6La2Qe0wAAG6o9tA (envelope-from ) for ; Mon, 22 May 2023 17:32:30 +0200 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id B81C83F42F for ; Mon, 22 May 2023 17:32:29 +0200 (CEST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1q17WH-0007Qk-On; Mon, 22 May 2023 11:32:21 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1q17W0-0007ND-Lc for guix-patches@gnu.org; Mon, 22 May 2023 11:32:08 -0400 Received: from debbugs.gnu.org ([209.51.188.43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1q17W0-0001LL-1v; Mon, 22 May 2023 11:32:04 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1q17Vx-0004tl-UH; Mon, 22 May 2023 11:32:01 -0400 X-Loop: help-debbugs@gnu.org Subject: [bug#63646] [PATCH] substitute: If a server's nar URL is 404, try the next one(s). Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: mail@cbaines.net, dev@jpoiret.xyz, ludo@gnu.org, othacehe@gnu.org, rekado@elephly.net, zimon.toutoune@gmail.com, me@tobias.gr, guix-patches@gnu.org Resent-Date: Mon, 22 May 2023 15:32:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: report 63646 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 63646@debbugs.gnu.org Cc: Ludovic =?UTF-8?Q?Court=C3=A8s?= , Christopher Baines , Josselin Poiret , Ludovic =?UTF-8?Q?Court=C3=A8s?= , Mathieu Othacehe , Ricardo Wurmus , Simon Tournier , Tobias Geerinckx-Rice X-Debbugs-Original-To: guix-patches@gnu.org X-Debbugs-Original-Xcc: Christopher Baines , Josselin Poiret , Ludovic =?UTF-8?Q?Court=C3=A8s?= , Mathieu Othacehe , Ricardo Wurmus , Simon Tournier , Tobias Geerinckx-Rice Received: via spool by submit@debbugs.gnu.org id=B.168476946918760 (code B ref -1); Mon, 22 May 2023 15:32:01 +0000 Received: (at submit) by debbugs.gnu.org; 22 May 2023 15:31:09 +0000 Received: from localhost ([127.0.0.1]:35934 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1q17V6-0004sV-Kw for submit@debbugs.gnu.org; Mon, 22 May 2023 11:31:09 -0400 Received: from lists.gnu.org ([209.51.188.17]:43202) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1q17V4-0004sN-F8 for submit@debbugs.gnu.org; Mon, 22 May 2023 11:31:07 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1q17V3-0007JC-CE for guix-patches@gnu.org; Mon, 22 May 2023 11:31:06 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1q17V1-0001E5-PC; Mon, 22 May 2023 11:31:04 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:Date:Subject:To:From:in-reply-to: references; bh=W8FgW+p0K19l9XTbOwJpeMnNIILn3UGIhOSd+7VQ6f4=; b=WMKZnyqJO8dQpq ehNwetFFkQANvEyS2EOB5ibdDFU8hFVRlzuJECOyndywjI6tKiPFz38iMvXaYEM10bPSH9qk5MCAl NHx81MVHbamkLOuZeS8H31hZ0Kxal5VI78bt/mm+Utn8ixgCaGqDbYt/eXHPHiNsy0ylIXR+DwpAA cYV+ft79LqFlcjg8erkwN7psQG2tEDjE4vC/IOX2gre10KDf02wViXvIAPGIkw6VnZtjzI+7RNvgm I+Cbe/ec6RzNB7gEO4CFck3EbH5biPeBLF7EzjYdd3CS3i2i/h8v8iXg876GK45InULqX4aDHxE07 QcSMdPHJsAWhkcceLWOw==; Received: from [193.50.110.247] (helo=gnu.org) by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1q17V0-0003C9-65; Mon, 22 May 2023 11:31:02 -0400 From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Date: Mon, 22 May 2023 17:30:50 +0200 Message-Id: X-Mailer: git-send-email 2.40.1 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+larch=yhetil.org@gnu.org Sender: guix-patches-bounces+larch=yhetil.org@gnu.org X-Migadu-Country: US X-Migadu-Flow: FLOW_IN ARC-Seal: i=1; s=key1; d=yhetil.org; t=1684769549; a=rsa-sha256; cv=none; b=Nln32YELCh7EGOZnPkn1RBSOu/mHm+ASd/J993hXUokirffC2JVhC/5QShqmP9ygAS/HB8 FhkMFzJ8K5OmwIP9To4bB2RghlNYzKB3r2fOpyDT2FOW6gQ1m0YzpyHayGel7rzlN5ANIQ SPkltRMI9rceN7PHVpJ08+2UR2ljNV1Fe9mfDOCLDw1Go5AldMgFEG54jBNoZ5QygXm3JH WSq5sxwIqBDacM/oH9KtpN/Z3q9AsT+LcAmrHYdgm2VFer0Atk5VhVpoNDM4P2/EDOMR6S GJsRLnx2lIbEvWsi2N/u/otV5/L5wAzSkzV3JOf9ec3gSK93b2FE6ndkDHSQJA== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=gnu.org header.s=fencepost-gnu-org header.b=WMKZnyqJ; dmarc=pass (policy=none) header.from=gnu.org; spf=pass (aspmx1.migadu.com: domain of "guix-patches-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-patches-bounces+larch=yhetil.org@gnu.org" ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1684769549; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding:resent-cc: resent-from:resent-sender:resent-message-id:list-id:list-help: list-unsubscribe:list-subscribe:list-post:dkim-signature; bh=W8FgW+p0K19l9XTbOwJpeMnNIILn3UGIhOSd+7VQ6f4=; b=QHC3fqExQ0QMl4m7f5lBn9OINK2dSd22gScXZQEGrKKTZjXwMo8cqSwRYvIcbtRB6zB71n fDtoMERAm5pjvU/LuY/NTNSfDslNmP6D7Eca8twR3/b9sIFR3BFktAo/2xf/Q7gBSdijoW UvUIPkl1KyAsx8lDZgM8rH/NvqULejK1EKWd3xrQY7TcCMoVktz5brKLndVW53gk0QYoS3 jkVml19fpKxmp1e5auuzEzeDQYebpo68kk7DMxbzE1QC+ROG4VghOklBr8NiNf92wzHeR3 B5tpztW+b8O2sF5v4U2vw6fG7+6fCCEDbaTkVLIkR4h7rVdTThLWRyLlNYZb+A== X-Migadu-Spam-Score: -1.97 X-Spam-Score: -1.97 X-Migadu-Queue-Id: B81C83F42F X-Migadu-Scanner: scn0.migadu.com Authentication-Results: aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=gnu.org header.s=fencepost-gnu-org header.b=WMKZnyqJ; dmarc=pass (policy=none) header.from=gnu.org; spf=pass (aspmx1.migadu.com: domain of "guix-patches-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-patches-bounces+larch=yhetil.org@gnu.org" X-TUID: pfSNR7PFlEAR If a substitute server advertises in its narinfo, for example, both a /zstd and a /lzip URL but the /zstd URL is unreachable, try the /lzip URL. Fixes . * guix/narinfo.scm (narinfo-preferred-uris): New procedure. (narinfo-best-uri): Rebase on top of it. * guix/scripts/substitute.scm (download-nar)[try-fetch]: New procedure. Use 'narinfo-preferred-uris' and 'try-fetch' to attempt all the URLs of NARINFO. * tests/substitute.scm (request-substitution): Remove 'parameterize'. Delete DESTINATION. ("substitute, preferred nar URL is 404, other is 200"): New test. --- guix/narinfo.scm | 21 +++++++++++++++++---- guix/scripts/substitute.scm | 33 ++++++++++++++++++++++----------- tests/substitute.scm | 36 +++++++++++++++++++++++++++++++----- 3 files changed, 70 insertions(+), 20 deletions(-) diff --git a/guix/narinfo.scm b/guix/narinfo.scm index 741c7ad406..a149d9a901 100644 --- a/guix/narinfo.scm +++ b/guix/narinfo.scm @@ -54,6 +54,7 @@ (define-module (guix narinfo) narinfo-hash-algorithm+value narinfo-hash->sha256 + narinfo-preferred-uris narinfo-best-uri valid-narinfo? @@ -309,9 +310,11 @@ (define (decompresses-faster? compression1 compression2) ("gzip" (string=? compression2 "lzip")) (_ #f))) -(define* (narinfo-best-uri narinfo #:key fast-decompression?) - "Select the \"best\" URI to download NARINFO's nar, and return three values: -the URI, its compression method (a string), and the compressed file size. +(define* (narinfo-preferred-uris narinfo #:key fast-decompression?) + "Return the sorted list of \"preferred\" nar URIs from NARINFO (preferred +comes first) where each entry is a tuple containing: the URI, its compression +method (a string), and the compressed file size. + When FAST-DECOMPRESSION? is true, prefer substitutes with faster decompression (typically zstd) rather than substitutes with a higher compression ratio (typically lzip)." @@ -343,6 +346,16 @@ (define* (narinfo-best-uri narinfo #:key fast-decompression?) ((uri2 compression2 . _) (decompresses-faster? compression2 compression1)))))) - (match (sort choices (if fast-decompression? (negate speedstring uri))))) - (let ((uri compression file-size - (narinfo-best-uri narinfo - #:fast-decompression? - %prefer-fast-decompression?))) - (unless print-build-trace? - (format (current-error-port) - (G_ "Downloading ~a...~%") (uri->string uri))) + (define (try-fetch choices) + (match choices + (((uri compression file-size) rest ...) + (guard (c ((and (pair? rest) (network-error? c)) + (warning (G_ "download from '~a' failed, trying next URL~%") + (uri->string uri)) + (try-fetch rest))) + (let ((port download-size (fetch uri))) + (unless print-build-trace? + (format (current-error-port) + (G_ "Downloading ~a...~%") (uri->string uri))) + (values port uri compression download-size)))) + (() + (leave (G_ "no valid nar URLs for ~a at ~a~%") + (narinfo-path narinfo) + (narinfo-uri-base narinfo))))) - (let* ((raw download-size - ;; 'guix publish' without '--cache' doesn't specify a - ;; Content-Length, so DOWNLOAD-SIZE is #f in this case. - (fetch uri)) + (let ((choices (narinfo-preferred-uris narinfo + #:fast-decompression? + %prefer-fast-decompression?))) + ;; 'guix publish' without '--cache' doesn't specify a Content-Length, so + ;; DOWNLOAD-SIZE is #f in this case. + (let* ((raw uri compression download-size (try-fetch choices)) (progress (let* ((dl-size (or download-size (and (equal? compression "none") diff --git a/tests/substitute.scm b/tests/substitute.scm index 9032a50268..8df3938b59 100644 --- a/tests/substitute.scm +++ b/tests/substitute.scm @@ -64,11 +64,11 @@ (define-syntax-rule (test-quit name error-rx exp) (define (request-substitution item destination) "Run 'guix substitute --substitute' to fetch ITEM to DESTINATION." - (parameterize ((guix-warning-port (current-error-port))) - (with-input-from-string (string-append "substitute " item " " - destination "\n") - (lambda () - (guix-substitute "--substitute"))))) + (false-if-exception (delete-file destination)) + (with-input-from-string (string-append "substitute " item " " + destination "\n") + (lambda () + (guix-substitute "--substitute")))) (define %public-key ;; This key is known to be in the ACL by default. @@ -613,6 +613,32 @@ (define-syntax-rule (with-narinfo* narinfo directory body ...) (lambda () (false-if-exception (delete-file "substitute-retrieved"))))))))))) +(test-equal "substitute, preferred nar URL is 404, other is 200" + "Substitutable data." + (with-narinfo* (string-append %narinfo "Signature: " (signature-field %narinfo)) + %main-substitute-directory + + (with-http-server `((200 ,(string-append %narinfo "Signature: " + (signature-field %narinfo) + "\n" + "URL: example.nar.lz\n" + "Compression: lzip\n")) + (404 "Sorry, nar.lz is missing!") + (200 ,(call-with-input-file + (string-append %main-substitute-directory + "/example.nar") + get-bytevector-all))) + (dynamic-wind + (const #t) + (lambda () + (parameterize ((substitute-urls (list (%local-url)))) + (request-substitution (string-append (%store-prefix) + "/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa-foo") + "substitute-retrieved")) + (call-with-input-file "substitute-retrieved" get-string-all)) + (lambda () + (false-if-exception (delete-file "substitute-retrieved"))))))) + (test-quit "substitute, narinfo is available but nar is missing" "failed to find alternative substitute" (with-narinfo* base-commit: dff1689bb37e5303868584d3f1d7a33cbcb7f51e -- 2.40.1