From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp2 ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms11 with LMTPS id 6JEoJnh0WWCbGQAA0tVLHw (envelope-from ) for ; Tue, 23 Mar 2021 04:54:16 +0000 Received: from aspmx1.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp2 with LMTPS id UDjdIXh0WWCtVgAAB5/wlQ (envelope-from ) for ; Tue, 23 Mar 2021 04:54:16 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id C53E21C223 for ; Tue, 23 Mar 2021 05:54:15 +0100 (CET) Received: from localhost ([::1]:52874 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lOZ3W-0003op-4r for larch@yhetil.org; Tue, 23 Mar 2021 00:54:14 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:32956) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lOZ3K-0003o3-Rh for bug-guix@gnu.org; Tue, 23 Mar 2021 00:54:02 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:47716) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1lOZ3K-0006WC-K6 for bug-guix@gnu.org; Tue, 23 Mar 2021 00:54:02 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1lOZ3K-00028O-Ic for bug-guix@gnu.org; Tue, 23 Mar 2021 00:54:02 -0400 X-Loop: help-debbugs@gnu.org Subject: bug#47336: [PATCH 2/2] download: Use Disarchive as a last resort. Resent-From: Timothy Sample Original-Sender: "Debbugs-submit" Resent-CC: bug-guix@gnu.org Resent-Date: Tue, 23 Mar 2021 04:54:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 47336 X-GNU-PR-Package: guix X-GNU-PR-Keywords: To: 47336@debbugs.gnu.org Received: via spool by 47336-submit@debbugs.gnu.org id=B47336.16164752218171 (code B ref 47336); Tue, 23 Mar 2021 04:54:02 +0000 Received: (at 47336) by debbugs.gnu.org; 23 Mar 2021 04:53:41 +0000 Received: from localhost ([127.0.0.1]:59262 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lOZ2y-00027i-R7 for submit@debbugs.gnu.org; Tue, 23 Mar 2021 00:53:41 -0400 Received: from wout3-smtp.messagingengine.com ([64.147.123.19]:51937) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lOZ2i-000276-Gd for 47336@debbugs.gnu.org; Tue, 23 Mar 2021 00:53:39 -0400 Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailout.west.internal (Postfix) with ESMTP id BCA45140F; Tue, 23 Mar 2021 00:53:18 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute1.internal (MEProxy); Tue, 23 Mar 2021 00:53:18 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:content-type :date:from:in-reply-to:message-id:mime-version:references :subject:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm2; bh=Sh8vx4Va/ITkI6SHjW/BifjchdPveiaTGxtGoj7SZ yI=; b=rr9U1Tsdg37OXQSTnqsTREm3ajQAetH/38BeY8pNp0b45p9UqZgJKKqZ3 Ee1wD0JjdU39XHa5sgY4qkYehCHXrzwGf0U6EYRD2vCH5B+fMllBeutBXksK68Xn dpxyKXZX7IdwcEYpPfV6MR2u4gvSLsHiVhxsCXBBltlpAEUoS/y2ywNrpV67Mcar rlUGCaM3+eABsrn9ZtG4SVYCtJRCdhpmsictRc8zp16rdDGtBts8acpmXGj05sRO 3zGPzrB11LEl8n6MWVf5PXNksxAtyKsKRZH78r5h4id1jQC99R8tnZ+AL1BpGmkq M9yGmRv/gqrfkAo3U7euPQ2gnK/6g== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduledrudeghedgjeeiucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucenucfjughrpefhvffufffkofgjfhggtgfgsehtke ertdertdejnecuhfhrohhmpefvihhmohhthhihucfurghmphhlvgcuoehsrghmphhlvght sehnghihrhhordgtohhmqeenucggtffrrghtthgvrhhnpeehtdeuteejkeevhfeftdeltd etjeejhfettefgleeigedtgfeguddtfffgtdevgeenucffohhmrghinhepnhhghihrohdr tghomhenucfkphepjeegrdduudeirddukeeirdeggeenucevlhhushhtvghrufhiiigvpe dtnecurfgrrhgrmhepmhgrihhlfhhrohhmpehsrghmphhlvghtsehnghihrhhordgtohhm X-ME-Proxy: Received: from mrblack.lan (74-116-186-44.qc.dsl.ebox.net [74.116.186.44]) by mail.messagingengine.com (Postfix) with ESMTPA id 257551080054; Tue, 23 Mar 2021 00:53:18 -0400 (EDT) From: Timothy Sample Date: Tue, 23 Mar 2021 00:52:13 -0400 Message-Id: <20210323045213.9419-2-samplet@ngyro.com> X-Mailer: git-send-email 2.31.0 In-Reply-To: <20210323045213.9419-1-samplet@ngyro.com> References: <20210323045213.9419-1-samplet@ngyro.com> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-guix@gnu.org List-Id: Bug reports for GNU Guix List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-guix-bounces+larch=yhetil.org@gnu.org Sender: "bug-Guix" X-Migadu-Flow: FLOW_IN ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1616475256; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:resent-cc: resent-from:resent-sender:resent-message-id:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post:dkim-signature; bh=Sh8vx4Va/ITkI6SHjW/BifjchdPveiaTGxtGoj7SZyI=; b=aYDLm/zxyn14gFqtgb5F+ldwlVDe3psgcc47HfXKF2CfQ76n8f85B0z3W68X0J+lal7FE6 mxPPrKKe0nNq8/TdmT4sQFbTpq7FxM7zezJ/jJnDYUIwIirKz9Tcohxofq+JVAzJpm/kvH YwtlTlxmDEpklYK1SIo6MGYQ3sKJRbGdwW8MXdWwQf6MqTsEZkZDHCC+NBstBpnqvCv7uW aHtpdP9tAbvMPmBPxKkm7WqWnAy3QtAR2ofi99um+/dETLtIlGzXI+jdXhNljiJ3HDhXOP cbOQm9jHAnbAooS8JCnknXoxYnb07DL7KD+GgzIm5EKzDXV1/mG54kE8nhWOOA== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1616475256; a=rsa-sha256; cv=none; b=EJhEjYQcDF+glsEQxcfZf4UyTDNThsomdYi271EOgEpTobUvJd1+J2PYgIgXfuYR/xqF3c ENuKDmK7g/cpHBty8z3FZhq2/u6bDrl56/U5cnB04CIw7gup+YXKeVlUOAZZJSEkt5c+tj pkUAPhuTvp5hvmpbqsaaIjDihLTqqvgHA+aAVn+dxlFZdglqBE/fAX8L++OeXLqCKPg1nG Aq+2JMc9vtew13IYibuZS4jZhWjHSXTOk0wtDFz2yhJebT6oyf0qGvSLMeu9rxzbQDu0GY 2dYshmReTpjNY657BIbzD3NyZgvRStPdBmene2dFRNP6xYu+W6qbRPyrkIC4Xw== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=messagingengine.com header.s=fm2 header.b=rr9U1Tsd; dmarc=none; spf=pass (aspmx1.migadu.com: domain of bug-guix-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=bug-guix-bounces@gnu.org X-Migadu-Spam-Score: -0.42 Authentication-Results: aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=messagingengine.com header.s=fm2 header.b=rr9U1Tsd; dmarc=none; spf=pass (aspmx1.migadu.com: domain of bug-guix-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=bug-guix-bounces@gnu.org X-Migadu-Queue-Id: C53E21C223 X-Spam-Score: -0.42 X-Migadu-Scanner: scn0.migadu.com X-TUID: GqA5KoRo5B/0 * guix/download.scm (%disarchive-mirrors): New variable. (%disarchive-mirror-file): New variable. (built-in-download): Add 'disarchive-mirrors' keyword argument and pass its value along to the 'builtin:download' derivation. (url-fetch): Pass '%disarchive-mirror-file' to 'built-in-download'. * guix/scripts/perform-download.scm (perform-download): Read Disarchive mirrors from the environment and pass them to 'url-fetch'. * guix/build/download.scm (disarchive-fetch/any): New procedure. (url-fetch): Add 'disarchive-mirrors' keyword argument, use it to make a list of URIs, and use the new procedure to fetch the file if all other methods fail. --- guix/build/download.scm | 77 +++++++++++++++++++++++++++---- guix/download.scm | 19 ++++++-- guix/scripts/perform-download.scm | 7 ++- 3 files changed, 89 insertions(+), 14 deletions(-) diff --git a/guix/build/download.scm b/guix/build/download.scm index a22d4064ca..f476d0f8ec 100644 --- a/guix/build/download.scm +++ b/guix/build/download.scm @@ -2,6 +2,7 @@ ;;; Copyright © 2012, 2013, 2014, 2015, 2016, 2017, 2018, 2019, 2020, 2021 Ludovic Courtès ;;; Copyright © 2015 Mark H Weaver ;;; Copyright © 2017 Tobias Geerinckx-Rice +;;; Copyright © 2021 Timothy Sample ;;; ;;; This file is part of GNU Guix. ;;; @@ -23,10 +24,12 @@ #:use-module (web http) #:use-module ((web client) #:hide (open-socket-for-uri)) #:use-module (web response) + #:use-module (guix base16) #:use-module (guix base64) #:use-module (guix ftp-client) #:use-module (guix build utils) #:use-module (guix progress) + #:use-module (guix swh) #:use-module (rnrs io ports) #:use-module (rnrs bytevectors) #:use-module (srfi srfi-1) @@ -626,10 +629,50 @@ Return a list of URIs." (else (list uri)))) +(define* (disarchive-fetch/any uris file + #:key (timeout 10)) + "Fetch a Disarchive specification from any of URIS, assemble it, +and write the output to FILE." + (define (fetch-specification uris) + (any (lambda (uri) + (false-if-exception* + (let-values (((port size) (http-fetch uri + #:verify-certificate? #t + #:timeout timeout))) + (let ((specification (read port))) + (close-port port) + specification)))) + uris)) + + (define (resolve addresses output) + (any (match-lambda + (('swhid swhid) + (match (string-split swhid #\:) + (("swh" "1" "dir" id) + (format #t "Downloading from Software Heritage...~%" file) + (false-if-exception* + (swh-download-directory id output))) + (_ #f))) + (_ #f)) + addresses)) + + (match (and=> (resolve-module '(disarchive) #:ensure #f) + (lambda (disarchive) + (cons (module-ref disarchive '%disarchive-log-port) + (module-ref disarchive 'disarchive-assemble)))) + (#f #f) + ((%disarchive-log-port . disarchive-assemble) + (format #t "Trying to use Disarchive to assemble ~a~%" file) + (match (fetch-specification uris) + (#f #f) + (spec (parameterize ((%disarchive-log-port (current-output-port))) + (disarchive-assemble spec file #:resolver resolve))))))) + (define* (url-fetch url file #:key (timeout 10) (verify-certificate? #t) (mirrors '()) (content-addressed-mirrors '()) + (disarchive-mirrors '()) (hashes '()) print-build-trace?) "Fetch FILE from URL; URL may be either a single string, or a list of @@ -693,6 +736,17 @@ otherwise simply ignore them." hashes)) content-addressed-mirrors)) + (define disarchive-uris + (append-map (lambda (mirror) + (map (match-lambda + ((hash-algo . hash) + (string->uri + (string-append mirror + (symbol->string hash-algo) "/" + (bytevector->base16-string hash))))) + hashes)) + disarchive-mirrors)) + ;; Make this unbuffered so 'progress-report/file' works as expected. 'line ;; means '\n', not '\r', so it's not appropriate here. (setvbuf (current-output-port) 'none) @@ -705,15 +759,18 @@ otherwise simply ignore them." (or (fetch uri file) (try tail))) (() - (format (current-error-port) "failed to download ~s from ~s~%" - file url) - - ;; Remove FILE in case we made an incomplete download, for example due - ;; to ENOSPC. - (catch 'system-error - (lambda () - (delete-file file)) - (const #f)) - #f)))) + ;; If we are looking for a software archive, one last thing we + ;; can try is to use Disarchive to assemble it. + (or (disarchive-fetch/any disarchive-uris file #:timeout timeout) + (begin + (format (current-error-port) "failed to download ~s from ~s~%" + file url) + ;; Remove FILE in case we made an incomplete download, for + ;; example due to ENOSPC. + (catch 'system-error + (lambda () + (delete-file file)) + (const #f)) + #f)))))) ;;; download.scm ends here diff --git a/guix/download.scm b/guix/download.scm index 30f69c0325..72094e7318 100644 --- a/guix/download.scm +++ b/guix/download.scm @@ -406,12 +406,19 @@ (plain-file "content-addressed-mirrors" (object->string %content-addressed-mirrors))) +(define %disarchive-mirrors + '("https://disarchive.ngyro.com/")) + +(define %disarchive-mirror-file + (plain-file "disarchive-mirrors" (object->string %disarchive-mirrors))) + (define built-in-builders* (store-lift built-in-builders)) (define* (built-in-download file-name url #:key system hash-algo hash mirrors content-addressed-mirrors + disarchive-mirrors executable? (guile 'unused)) "Download FILE-NAME from URL using the built-in 'download' builder. When @@ -422,13 +429,16 @@ explicitly depend on Guile, GnuTLS, etc. Instead, the daemon performs the download by itself using its own dependencies." (mlet %store-monad ((mirrors (lower-object mirrors)) (content-addressed-mirrors - (lower-object content-addressed-mirrors))) + (lower-object content-addressed-mirrors)) + (disarchive-mirrors (lower-object disarchive-mirrors))) (raw-derivation file-name "builtin:download" '() #:system system #:hash-algo hash-algo #:hash hash #:recursive? executable? - #:sources (list mirrors content-addressed-mirrors) + #:sources (list mirrors + content-addressed-mirrors + disarchive-mirrors) ;; Honor the user's proxy and locale settings. #:leaked-env-vars '("http_proxy" "https_proxy" @@ -439,6 +449,7 @@ download by itself using its own dependencies." ("mirrors" . ,mirrors) ("content-addressed-mirrors" . ,content-addressed-mirrors) + ("disarchive-mirrors" . ,disarchive-mirrors) ,@(if executable? '(("executable" . "1")) '())) @@ -492,7 +503,9 @@ name in the store." #:executable? executable? #:mirrors %mirror-file #:content-addressed-mirrors - %content-addressed-mirror-file))))) + %content-addressed-mirror-file + #:disarchive-mirrors + %disarchive-mirror-file))))) (define* (url-fetch/executable url hash-algo hash #:optional name diff --git a/guix/scripts/perform-download.scm b/guix/scripts/perform-download.scm index 8d409092ba..6889bcef79 100644 --- a/guix/scripts/perform-download.scm +++ b/guix/scripts/perform-download.scm @@ -54,7 +54,8 @@ actual output is different from that when we're doing a 'bmCheck' or (output* "out") (executable "executable") (mirrors "mirrors") - (content-addressed-mirrors "content-addressed-mirrors")) + (content-addressed-mirrors "content-addressed-mirrors") + (disarchive-mirrors "disarchive-mirrors")) (unless url (leave (G_ "~a: missing URL~%") (derivation-file-name drv))) @@ -79,6 +80,10 @@ actual output is different from that when we're doing a 'bmCheck' or (lambda (port) (eval (read port) %user-module))) '()) + #:disarchive-mirrors + (if disarchive-mirrors + (call-with-input-file disarchive-mirrors read) + '()) #:hashes `((,algo . ,hash)) ;; Since DRV's output hash is known, X.509 certificate -- 2.31.0