From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp12.migadu.com ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms5.migadu.com with LMTPS id 2MuROFZRLWOBGwAAbAwnHQ (envelope-from ) for ; Fri, 23 Sep 2022 08:25:27 +0200 Received: from aspmx1.migadu.com ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp12.migadu.com with LMTPS id UCOVOFZRLWMbjQAAauVa8A (envelope-from ) for ; Fri, 23 Sep 2022 08:25:26 +0200 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 7119928FB3 for ; Fri, 23 Sep 2022 08:25:26 +0200 (CEST) Received: from localhost ([::1]:37912 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1obc7n-0002sn-UV for larch@yhetil.org; Fri, 23 Sep 2022 02:25:24 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:44664) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1obc3a-00023C-OT for bug-guix@gnu.org; Fri, 23 Sep 2022 02:21:02 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:40085) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1obc3a-00081c-E8 for bug-guix@gnu.org; Fri, 23 Sep 2022 02:21:02 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1obc3a-0007J4-8x for bug-guix@gnu.org; Fri, 23 Sep 2022 02:21:02 -0400 X-Loop: help-debbugs@gnu.org Subject: bug#57978: [PATCH 2/2] substitute: Retry downloading when a nar is unavailable. Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: bug-guix@gnu.org Resent-Date: Fri, 23 Sep 2022 06:21:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 57978 X-GNU-PR-Package: guix X-GNU-PR-Keywords: To: 58017@debbugs.gnu.org Cc: Attila Lendvai , 57978@debbugs.gnu.org, Ludovic =?UTF-8?Q?Court=C3=A8s?= , zimoun Received: via spool by 57978-submit@debbugs.gnu.org id=B57978.166391403628018 (code B ref 57978); Fri, 23 Sep 2022 06:21:02 +0000 Received: (at 57978) by debbugs.gnu.org; 23 Sep 2022 06:20:36 +0000 Received: from localhost ([127.0.0.1]:39162 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1obc39-0007Hk-7k for submit@debbugs.gnu.org; Fri, 23 Sep 2022 02:20:35 -0400 Received: from eggs.gnu.org ([209.51.188.92]:49970) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1obc2z-0007Gk-R7; Fri, 23 Sep 2022 02:20:27 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:39110) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1obc2u-0007ve-Jr; Fri, 23 Sep 2022 02:20:20 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:References:In-Reply-To:Date:Subject:To: From; bh=GClk5QRgnyaP1phI++P1FZ/h1HaztCmPzpWsKUV67jc=; b=iLvQS6rmqCSMyQWi1Pwl xKY3lgcPqDsXW9ps0Tv9vJZPpfxZmJKKAo2FGLB7No7Tk7hGawGvWziWRjcHYkEciNrptQ8IQXthW xnljBFF4xfCA8xXrcFD4CkF2iVpIaVBTWHxBriLpq5Mg1Yk4wHjU/lTRyg7dfKybE7oWvydE4enJ/ 8EJVKH7x1hfky1mAVNbsXFw+cJU7RU9zcES829V3hVR2Xek0G6PIYSeZNSiaCaJE/Q1heBaTy7DUe 8AppX1pVLX8bSNgNhg5swti5Gq7/gF6DqG4uYnLwoz+GLYUrddgMAxTnmIDXpXlbsOycHfVziOEuR aiT/ISdyvJ9qxw==; Received: from [89.207.171.75] (port=38064 helo=gnu.org) by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1obc2r-0007Hi-KL; Fri, 23 Sep 2022 02:20:20 -0400 From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Date: Fri, 23 Sep 2022 08:19:57 +0200 Message-Id: <20220923061957.5658-2-ludo@gnu.org> X-Mailer: git-send-email 2.37.3 In-Reply-To: <20220923061957.5658-1-ludo@gnu.org> References: <20220923061957.5658-1-ludo@gnu.org> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-guix@gnu.org List-Id: Bug reports for GNU Guix List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-guix-bounces+larch=yhetil.org@gnu.org Sender: "bug-Guix" X-Migadu-Flow: FLOW_IN X-Migadu-To: larch@yhetil.org X-Migadu-Country: US ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1663914326; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding:resent-cc: resent-from:resent-sender:resent-message-id:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post:dkim-signature; bh=g1pm/wJz1VHl+WcTpawmvP3r4OEfwZoLAn6sI+vY/rY=; b=YyYgB/vclu3vgsWZaDPkbnU8J6WYmPTJUahqxZK6hXa52UnbupviSnU1b42TQlsGHYdhb/ yuoao9VZ95xW+AdbfLZ2vIrL1EUURWs2uWepCU1creDFzeaSYi+e1IE3r12bcbEqCgyXi+ HR6VuJ8HZL9fK2LtcK/yhBVoi/BQ40bagNMsognCCRMPx/K77Kriura94f3/2R6gFSz7X6 9Jbt4yZq1unYbGSBACpENTGFnZP4p6r4wTEENMJpVawiD+DhgHKRxPGG1yRTQbewejQiej sLSr75dLu70+R8ixKz9f4EndylRT0D7tSmnH9CBfzpQN76VkcdqXamk35+DcHQ== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1663914326; a=rsa-sha256; cv=none; b=lTUF5t9wUiRCTN++23qraXwWEOXSKmdfPko1xe8m/xTSp5VJvWCNQsCm/n67PFaCmfinc/ lMfX42h2XTJcNJ2Y9oidzR1/qxevmrz75FCGA7vHL0XeTZTrIiO1zx+CcSv1r7bkJsUbdt wmMtMsZaEU7Vl+nzHs84bASSAGYqAu7g1ZMd0ihvqzYpC8qaCiOVw+wkHpayw2EnMAhGuj 1KDEaoMr2NIY6m6lmV/1qbSBGkzF2Roy8z9MDcjXH8A4kjwSzKub+mN+xWmnR4p9zEY4Wd t5SRokN8rbFzxikBe5dgWhsKdH/j3YhkUBrw3pBAmKxuI9ZPV09ZElpQffvA7Q== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=fail ("body hash did not verify") header.d=gnu.org header.s=fencepost-gnu-org header.b=iLvQS6rm; dmarc=pass (policy=none) header.from=gnu.org; spf=pass (aspmx1.migadu.com: domain of "bug-guix-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="bug-guix-bounces+larch=yhetil.org@gnu.org" X-Migadu-Spam-Score: 2.55 Authentication-Results: aspmx1.migadu.com; dkim=fail ("body hash did not verify") header.d=gnu.org header.s=fencepost-gnu-org header.b=iLvQS6rm; dmarc=pass (policy=none) header.from=gnu.org; spf=pass (aspmx1.migadu.com: domain of "bug-guix-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="bug-guix-bounces+larch=yhetil.org@gnu.org" X-Migadu-Queue-Id: 7119928FB3 X-Spam-Score: 2.55 X-Migadu-Scanner: scn1.migadu.com X-TUID: WoP55J7anRh7 Fixes Reported by Attila Lendvai . Previously, if a narinfo was available but its corresponding nar was missing (for instance because the narinfo was cached and the server became unreachable in the meantime), 'guix substitute --substitute' would try to download the nar from its preferred location and abort when that fails. This change forces one retry with each of the URLs. * guix/scripts/substitute.scm (download-nar): Do not catch 'http-get-error?' exceptions. (system-error?, network-error?, process-substitution/fallback): New procedures. (process-substitution): Call 'process-substitution/fallback' upon 'network-error?'. * tests/substitute.scm ("substitute, first URL has narinfo but lacks nar, second URL unauthorized") ("substitute, first URL has narinfo but nar is 404, both URLs authorized") ("substitute, first URL has narinfo but nar is 404, one URL authorized") ("substitute, narinfo is available but nar is missing"): New tests. --- guix/scripts/substitute.scm | 113 ++++++++++++++++++++++++++++-------- tests/substitute.scm | 113 ++++++++++++++++++++++++++++++++++++ 2 files changed, 203 insertions(+), 23 deletions(-) diff --git a/guix/scripts/substitute.scm b/guix/scripts/substitute.scm index e3b382d0d8..cf59db4315 100755 --- a/guix/scripts/substitute.scm +++ b/guix/scripts/substitute.scm @@ -460,25 +460,20 @@ (define (fetch uri) (let ((port (open-file (uri-path uri) "r0b"))) (values port (stat:size (stat port))))) ((http https) - (guard (c ((http-get-error? c) - (leave (G_ "download from '~a' failed: ~a, ~s~%") - (uri->string (http-get-error-uri c)) - (http-get-error-code c) - (http-get-error-reason c)))) - ;; Test this with: - ;; sudo tc qdisc add dev eth0 root netem delay 1500ms - ;; and then cancel with: - ;; sudo tc qdisc del dev eth0 root - (with-timeout %fetch-timeout - (begin - (warning (G_ "while fetching ~a: server is somewhat slow~%") - (uri->string uri)) - (warning (G_ "try `--no-substitutes' if the problem persists~%"))) - (with-cached-connection uri port - (http-fetch uri #:text? #f - #:port port - #:keep-alive? #t - #:buffered? #f))))) + ;; Test this with: + ;; sudo tc qdisc add dev eth0 root netem delay 1500ms + ;; and then cancel with: + ;; sudo tc qdisc del dev eth0 root + (with-timeout %fetch-timeout + (begin + (warning (G_ "while fetching ~a: server is somewhat slow~%") + (uri->string uri)) + (warning (G_ "try `--no-substitutes' if the problem persists~%"))) + (with-cached-connection uri port + (http-fetch uri #:text? #f + #:port port + #:keep-alive? #t + #:buffered? #f)))) (else (leave (G_ "unsupported substitute URI scheme: ~a~%") (uri->string uri))))) @@ -572,6 +567,68 @@ (define cpu-usage (bytevector->nix-base32-string expected) (bytevector->nix-base32-string actual))))))) +(define system-error? + (let ((kind-and-args? (exception-predicate &exception-with-kind-and-args))) + (lambda (exception) + "Return true if EXCEPTION is a Guile 'system-error exception." + (and (kind-and-args? exception) + (eq? 'system-error (exception-kind exception)))))) + +(define network-error? + (let ((kind-and-args? (exception-predicate &exception-with-kind-and-args))) + (lambda (exception) + "Return true if EXCEPTION denotes a networking error." + (or (and (system-error? exception) + (let ((errno (system-error-errno + (cons 'system-error (exception-args exception))))) + (memv errno (list ECONNRESET ECONNABORTED + ECONNREFUSED EHOSTUNREACH + ENOENT)))) ;for "file://" + (and (kind-and-args? exception) + (memq (exception-kind exception) + '(gnutls-error getaddrinfo-error))) + (and (http-get-error? exception) + (begin + (warning (G_ "download from '~a' failed: ~a, ~s~%") + (uri->string (http-get-error-uri exception)) + (http-get-error-code exception) + (http-get-error-reason exception)) + #t)))))) + +(define* (process-substitution/fallback port narinfo destination + #:key cache-urls acl + deduplicate? print-build-trace?) + "Attempt to substitute NARINFO, which is assumed to be authorized or +equivalent, by trying to download its nar from each entry in CACHE-URLS. + +This can be less efficient than 'lookup-narinfo', which stops at the first +entry that provides a valid narinfo, but it makes sure we eventually find a +way to download the nar." + ;; Note: Keep NARINFO's uri-base in CACHE-URLS: that lets us retry in case + ;; this was a transient issue. + (let loop ((cache-urls cache-urls)) + (match cache-urls + (() + (leave (G_ "failed to find alternative substitute for '~a'~%") + (narinfo-path narinfo))) + ((cache-url rest ...) + (match (lookup-narinfos cache-url + (list (narinfo-path narinfo)) + #:open-connection + open-connection-for-uri/cached) + ((alternate) + (if (or (equivalent-narinfo? narinfo alternate) + (valid-narinfo? alternate acl) + (%allow-unauthenticated-substitutes?)) + (guard (c ((network-error? c) (loop rest))) + (download-nar alternate destination + #:status-port port + #:deduplicate? deduplicate? + #:print-build-trace? print-build-trace?)) + (loop rest))) + (() + (loop rest))))))) + (define* (process-substitution port store-item destination #:key cache-urls acl deduplicate? print-build-trace?) @@ -590,10 +647,20 @@ (define narinfo (leave (G_ "no valid substitute for '~a'~%") store-item)) - (download-nar narinfo destination - #:status-port port - #:deduplicate? deduplicate? - #:print-build-trace? print-build-trace?)) + (guard (c ((network-error? c) + (format (current-error-port) + (G_ "retrying download of '~a' with other substitute URLs...~%") + store-item) + (process-substitution/fallback port narinfo destination + #:cache-urls cache-urls + #:acl acl + #:deduplicate? deduplicate? + #:print-build-trace? + print-build-trace?))) + (download-nar narinfo destination + #:status-port port + #:deduplicate? deduplicate? + #:print-build-trace? print-build-trace?))) ;;; diff --git a/tests/substitute.scm b/tests/substitute.scm index 5315292987..9032a50268 100644 --- a/tests/substitute.scm +++ b/tests/substitute.scm @@ -523,6 +523,119 @@ (define-syntax-rule (with-narinfo* narinfo directory body ...) (lambda () (false-if-exception (delete-file "substitute-retrieved"))))))) +(test-equal "substitute, first URL has narinfo but lacks nar, second URL unauthorized" + "Substitutable data." + (with-narinfo* + (string-append %narinfo "Signature: " + (signature-field + %narinfo + #:public-key %wrong-public-key)) + %alternate-substitute-directory + + (with-narinfo* (string-append %narinfo "Signature: " + (signature-field %narinfo)) + %main-substitute-directory + + (dynamic-wind + (const #t) + (lambda () + ;; Remove this file so that the substitute can only be retrieved + ;; from %ALTERNATE-SUBSTITUTE-DIRECTORY. + (delete-file (string-append %main-substitute-directory + "/example.nar")) + + (parameterize ((substitute-urls + (map (cut string-append "file://" <>) + (list %main-substitute-directory + %alternate-substitute-directory)))) + (request-substitution (string-append (%store-prefix) + "/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa-foo") + "substitute-retrieved")) + (call-with-input-file "substitute-retrieved" get-string-all)) + (lambda () + (false-if-exception (delete-file "substitute-retrieved"))))))) + +(test-equal "substitute, first URL has narinfo but nar is 404, both URLs authorized" + "Substitutable data." + (with-narinfo* + (string-append %narinfo "Signature: " + (signature-field %narinfo)) + %main-substitute-directory + + (with-http-server `((200 ,(string-append %narinfo "Signature: " + (signature-field %narinfo))) + (404 "Sorry, nar is missing!")) + (dynamic-wind + (const #t) + (lambda () + (parameterize ((substitute-urls + (list (%local-url) + (string-append "file://" + %main-substitute-directory)))) + (request-substitution (string-append (%store-prefix) + "/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa-foo") + "substitute-retrieved")) + (call-with-input-file "substitute-retrieved" get-string-all)) + (lambda () + (false-if-exception (delete-file "substitute-retrieved"))))))) + +(test-equal "substitute, first URL has narinfo but nar is 404, one URL authorized" + "Substitutable data." + (with-narinfo* + (string-append %narinfo "Signature: " + (signature-field + %narinfo + #:public-key %wrong-public-key)) + %main-substitute-directory + + (with-http-server `((200 ,(string-append %narinfo "Signature: " + (signature-field + %narinfo + #:public-key %wrong-public-key))) + (404 "Sorry, nar is missing!")) + (let ((url1 (%local-url))) + (parameterize ((%http-server-port 0)) + (with-http-server `((200 ,(string-append %narinfo "Signature: " + (signature-field %narinfo))) + (404 "Sorry, nar is missing!")) + (let ((url2 (%local-url))) + (dynamic-wind + (const #t) + (lambda () + (parameterize ((substitute-urls + (list url1 url2 + (string-append "file://" + %main-substitute-directory)))) + (request-substitution (string-append (%store-prefix) + "/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa-foo") + "substitute-retrieved")) + (call-with-input-file "substitute-retrieved" get-string-all)) + (lambda () + (false-if-exception (delete-file "substitute-retrieved"))))))))))) + +(test-quit "substitute, narinfo is available but nar is missing" + "failed to find alternative substitute" + (with-narinfo* + (string-append %narinfo "Signature: " + (signature-field + %narinfo + #:public-key %wrong-public-key)) + %main-substitute-directory + + (with-http-server `((200 ,(string-append %narinfo "Signature: " + (signature-field %narinfo))) + (404 "Sorry, nar is missing!")) + (parameterize ((substitute-urls + (list (%local-url) + (string-append "file://" + %main-substitute-directory)))) + (delete-file (string-append %main-substitute-directory + "/example.nar")) + (request-substitution (string-append (%store-prefix) + "/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa-foo") + "substitute-retrieved") + (not (file-exists? "substitute-retrieved")))))) + (test-equal "substitute, first narinfo is unsigned and has wrong hash" "Substitutable data." (with-narinfo* (regexp-substitute #f -- 2.37.3