From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp1 ([2001:41d0:8:6d80::]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id I35fMc0PX2AaGAAAgWs5BA (envelope-from ) for ; Sat, 27 Mar 2021 11:58:21 +0100 Received: from aspmx1.migadu.com ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp1 with LMTPS id WHkYKs0PX2CSQAAAbx9fmQ (envelope-from ) for ; Sat, 27 Mar 2021 10:58:21 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 296E5243F1 for ; Sat, 27 Mar 2021 11:58:21 +0100 (CET) Received: from localhost ([::1]:33140 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lQ6e4-0006b0-C3 for larch@yhetil.org; Sat, 27 Mar 2021 06:58:20 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:57282) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lQ6dq-0006Z8-IQ for guix-patches@gnu.org; Sat, 27 Mar 2021 06:58:06 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:60161) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1lQ6dm-00078m-BB for guix-patches@gnu.org; Sat, 27 Mar 2021 06:58:04 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1lQ6dm-0002rT-6m for guix-patches@gnu.org; Sat, 27 Mar 2021 06:58:02 -0400 X-Loop: help-debbugs@gnu.org Subject: [bug#47336] Disarchive as a fallback for downloads Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Sat, 27 Mar 2021 10:58:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 47336 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: To: Timothy Sample Cc: 47336@debbugs.gnu.org Received: via spool by 47336-submit@debbugs.gnu.org id=B47336.161684264210939 (code B ref 47336); Sat, 27 Mar 2021 10:58:02 +0000 Received: (at 47336) by debbugs.gnu.org; 27 Mar 2021 10:57:22 +0000 Received: from localhost ([127.0.0.1]:43471 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lQ6d8-0002qN-Ey for submit@debbugs.gnu.org; Sat, 27 Mar 2021 06:57:22 -0400 Received: from eggs.gnu.org ([209.51.188.92]:36022) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lQ6d6-0002q9-Om for 47336@debbugs.gnu.org; Sat, 27 Mar 2021 06:57:21 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:34573) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lQ6d1-0006gL-8A; Sat, 27 Mar 2021 06:57:15 -0400 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=41646 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1lQ6d0-00007a-Mw; Sat, 27 Mar 2021 06:57:15 -0400 From: Ludovic =?UTF-8?Q?Court=C3=A8s?= References: <20210323045213.9419-1-samplet@ngyro.com> <20210323045213.9419-2-samplet@ngyro.com> Date: Sat, 27 Mar 2021 11:57:12 +0100 In-Reply-To: <20210323045213.9419-2-samplet@ngyro.com> (Timothy Sample's message of "Tue, 23 Mar 2021 00:52:13 -0400") Message-ID: <87y2e8zwxj.fsf_-_@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+larch=yhetil.org@gnu.org Sender: "Guix-patches" X-Migadu-Flow: FLOW_IN ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1616842701; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:resent-cc: resent-from:resent-sender:resent-message-id:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post; bh=eDUURCHJC7KWAqGpnJ7v03gctZgVWkhys+yYQhyZP3E=; b=Tab/TAsSmsBgGgR22UP8K1ilZYHyG2ZgcG/PMm6ongbDN8tvq9GSR2z8+1quL988alTxwm SglRrCyvMwO1YNYWtNN21BKgFzajKyNpKsR8Pb3wGFBok6Feit/V15B+q7C5Orez8Jjtcs YkL2oBzjtdrVyF0Ed2+hz3cMqXz43Pvp+a3k4F2ZuQpyslDGQO5AW8QMhwf0XaAFOfw4Sc jRn6S8xvq4o0pBEXqRCaiKxGyW1WdSE3tASnmdTaZjMcQgC4MlFxJeSqwu/Duf2cxYrXiq DMq04WvzDXzZnb/t20IzUCoUmgrxXIo85nVVMt5/2OZ+tDnT3fovzznxglvG3w== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1616842701; a=rsa-sha256; cv=none; b=PGrXj/nEOjUf10HXnVkgcY/T6WXlORA0zt7uji/KS1MlNeEC6z7IqSYbnPwASMznxWTJSz WuXijH8n053Gsw0Xm87oUpUppxrfV1jn6NCZEGwrxYmuzXLypsch3TrEwdmipRn0CTXeMX xYr/RGySoo+Q+DBmPdbUEeOKNZ3nIupOcB4LyhXY6OdiX65CsI2rf6fQy9OmcRAr0uh7yR JyBH3sBd91mmL84oO6zZK0049o6zM3sGAAbIJV05o/qPKYqGREPry4mdrC1GpmbTOQiRW/ fdKtjopY9BPPHClq/4knkSc/IBgMBhD/FdRzr5i16neVhFT6BbilfiG0TeySCg== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=none; dmarc=pass (policy=none) header.from=gnu.org; spf=pass (aspmx1.migadu.com: domain of guix-patches-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-patches-bounces@gnu.org X-Migadu-Spam-Score: -1.42 Authentication-Results: aspmx1.migadu.com; dkim=none; dmarc=pass (policy=none) header.from=gnu.org; spf=pass (aspmx1.migadu.com: domain of guix-patches-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-patches-bounces@gnu.org X-Migadu-Queue-Id: 296E5243F1 X-Spam-Score: -1.42 X-Migadu-Scanner: scn0.migadu.com X-TUID: HLJ9WrrFFxci Hi! Timothy Sample skribis: > * guix/download.scm (%disarchive-mirrors): New variable. > (%disarchive-mirror-file): New variable. > (built-in-download): Add 'disarchive-mirrors' keyword argument and > pass its value along to the 'builtin:download' derivation. > (url-fetch): Pass '%disarchive-mirror-file' to 'built-in-download'. > * guix/scripts/perform-download.scm (perform-download): Read > Disarchive mirrors from the environment and pass them to > 'url-fetch'. > * guix/build/download.scm (disarchive-fetch/any): New procedure. > (url-fetch): Add 'disarchive-mirrors' keyword argument, use it to > make a list of URIs, and use the new procedure to fetch the file if > all other methods fail. [...] > + #:use-module (guix base16) > #:use-module (guix base64) > #:use-module (guix ftp-client) > #:use-module (guix build utils) > #:use-module (guix progress) > + #:use-module (guix swh) Maybe #:autoload them. > +(define* (disarchive-fetch/any uris file > + #:key (timeout 10)) > + "Fetch a Disarchive specification from any of URIS, assemble it, > +and write the output to FILE." > + (define (fetch-specification uris) > + (any (lambda (uri) > + (false-if-exception* > + (let-values (((port size) (http-fetch uri > + #:verify-certificate? = #t > + #:timeout timeout))) Perhaps add #:key (verify-certificate? #t) and have the caller pass it? Currently (guix scripts perform-download) sets it to #f, which is a good idea IMO. > + (match (and=3D> (resolve-module '(disarchive) #:ensure #f) > + (lambda (disarchive) > + (cons (module-ref disarchive '%disarchive-log-port) > + (module-ref disarchive 'disarchive-assemble)))) > + (#f #f) > + ((%disarchive-log-port . disarchive-assemble) > + (format #t "Trying to use Disarchive to assemble ~a~%" file) > + (match (fetch-specification uris) > + (#f #f) > + (spec (parameterize ((%disarchive-log-port (current-output-port))) > + (disarchive-assemble spec file #:resolver resolve))))))) So we would normally arrange so that the =E2=80=98guix=E2=80=99 package dep= ends on Disarchive, such that the above =E2=80=98resolve-module=E2=80=99 call works= when done via =E2=80=98guix perform-download=E2=80=99, right? In the #f case, perhaps we should print something like =E2=80=9CDisarchive = not found, bailing out=E2=80=9D? That=E2=80=99s all I have to say; it looks great to me! That=E2=80=99s quite a milestone, it=E2=80=99d be great to have that in the= upcoming release. Next we can discuss how to populate the Disarchive database and where to do that (or your hosting fees could easily skyrocket :-)). I suppose we could run that in Berlin and/or we could make an argument about using SWH or Inria resources for that. Thanks, Ludo=E2=80=99.