From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp2 ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id quLwEZHtnmBJLAAAgWs5BA (envelope-from ) for ; Fri, 14 May 2021 23:37:21 +0200 Received: from aspmx1.migadu.com ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp2 with LMTPS id 8OYCDZHtnmDnPwAAB5/wlQ (envelope-from ) for ; Fri, 14 May 2021 21:37:21 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id A313F3256C for ; Fri, 14 May 2021 23:37:20 +0200 (CEST) Received: from localhost ([::1]:35468 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lhfUl-0007su-0L for larch@yhetil.org; Fri, 14 May 2021 17:37:19 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:51522) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lhfUT-0007sX-Vk for guix-patches@gnu.org; Fri, 14 May 2021 17:37:02 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:35669) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1lhfUT-0004pW-Oo for guix-patches@gnu.org; Fri, 14 May 2021 17:37:01 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1lhfUT-0004LI-L1 for guix-patches@gnu.org; Fri, 14 May 2021 17:37:01 -0400 X-Loop: help-debbugs@gnu.org Subject: [bug#47336] Disarchive as a fallback for downloads Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Fri, 14 May 2021 21:37:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 47336 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: To: Timothy Sample Cc: 47336@debbugs.gnu.org Received: via spool by 47336-submit@debbugs.gnu.org id=B47336.162102819116650 (code B ref 47336); Fri, 14 May 2021 21:37:01 +0000 Received: (at 47336) by debbugs.gnu.org; 14 May 2021 21:36:31 +0000 Received: from localhost ([127.0.0.1]:47215 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lhfTz-0004KU-2x for submit@debbugs.gnu.org; Fri, 14 May 2021 17:36:31 -0400 Received: from eggs.gnu.org ([209.51.188.92]:57806) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lhfTx-0004KH-5j for 47336@debbugs.gnu.org; Fri, 14 May 2021 17:36:30 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:45334) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lhfTr-0004QY-Si; Fri, 14 May 2021 17:36:23 -0400 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=49598 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lhfTr-0002P0-I8; Fri, 14 May 2021 17:36:23 -0400 From: Ludovic =?UTF-8?Q?Court=C3=A8s?= References: <87eeg6o50b.fsf@ngyro.com> Date: Fri, 14 May 2021 23:36:21 +0200 In-Reply-To: <87eeg6o50b.fsf@ngyro.com> (Timothy Sample's message of "Tue, 23 Mar 2021 00:42:12 -0400") Message-ID: <87im3lknfu.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+larch=yhetil.org@gnu.org Sender: "Guix-patches" X-Migadu-Flow: FLOW_IN ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1621028240; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:resent-cc: resent-from:resent-sender:resent-message-id:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post; bh=ZVbU8u2G0yClSgaHvINfwrvdlrIpo+0jIee2dIfO9uY=; b=tSWaCliX0KeZLIAVYkHYKERbIPqspYjxeyxJ2oiQHAi5KVGUb4AcwKxHKgo6m7AOTdVrY/ UTNEMeSzxKrMPiUFuke/lUBZemh0JCeRhFm+6/7gr6VS+MCt8TNKkqYBIwWSYCWEy1YLD8 ewvZBt0qSum7aAJuQj096TbJLtGrASsYk5bOdOY7yVfMMWL04YiWv2EbCnJ1Li5Ksr6Q8K 0FsKk8Kkmwt24VzQSIOyopWkB7bRNWwjvDd/1K3QZYB4KqxXUleNDD19Pd1dLjyCI0uxZM E3Fzz03u3Ke/FIigXJA8Ht5PHIfXyiebMJnq0hF3VXmRTsdu1sdsIjdrRLJIBA== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1621028240; a=rsa-sha256; cv=none; b=QmtmoIVDBpgKpLunNiw33ksGsLDvubxAO0PLRa1FPRYZCX0OTN/ui9jsR7Uh1c6sKFqplB VEolZFW/K5bTvHrCuvmm7YbFq/YGI80jpYNkUqGicM9S0v5GNcgqf5SVT/gnQvmjWEsB/x R0GpSUnMhnBVAdxH6tlDFL9aJN66WomIRXjAXCH7gNR0g9CHzkBjf3C/2cvCemTRwYV3Ld M2kh+wCkwKNwTcEs8288qebYrTkvAoxlJFzOgcQnAbNX1j2sfCrds5IxTGonULlEua59ur OeeVAnDW0fCYUfyx9VCtr7okCad5yWz0D8EScgkopQc51q/cB2to8r4Aqe9uCQ== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=none; dmarc=pass (policy=none) header.from=gnu.org; spf=pass (aspmx1.migadu.com: domain of guix-patches-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-patches-bounces@gnu.org X-Migadu-Spam-Score: -2.95 Authentication-Results: aspmx1.migadu.com; dkim=none; dmarc=pass (policy=none) header.from=gnu.org; spf=pass (aspmx1.migadu.com: domain of guix-patches-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-patches-bounces@gnu.org X-Migadu-Queue-Id: A313F3256C X-Spam-Score: -2.95 X-Migadu-Scanner: scn0.migadu.com X-TUID: MBF5OXGFzwr1 Hi! Timothy Sample skribis: > This patch series adds Disarchive assembly (backed by SWH lookup) as a > fallback for downloads. > > To try it, make sure you are running the daemon in an environment with > Disarchive available: > > $ ./pre-inst-env guix environment --ad-hoc guile disarchive > # ./pre-inst-env guix-daemon --build-users-group=3Dguixbuild > > Don=E2=80=99t forget to stop your existing Guix Daemon. :) > > You also need to make sure that regular downloads are unavailable. I do > this by adjusting the =E2=80=9Ctry=E2=80=9D loop at the end of =E2=80=9Cu= rl-fetch=E2=80=9D in > =E2=80=9Cguix/build/download.scm=E2=80=9D. I replace the usual list of U= RLs with =E2=80=98()=E2=80=99: > > (let try ((uri (append uri content-addressed-uris))) > (match '() ; uri > ...)) > > Now you can ask Guix for a recent .tar.gz source package: > > $ ./pre-inst-env guix build --no-substitutes -S python-httpretty > > You should see: > > Trying to use Disarchive to assemble /gnu/store/kbcnm57y2q1jvhvd8zw1g= 5vdiwlv19y9-httpretty-1.0.5.tar.gz > Assembling the directory httpretty-1.0.5 > Downloading from Software Heritage... > 7903d608efc89c14afb4d692a3721156e31a43e2/ > 7903d608efc89c14afb4d692a3721156e31a43e2/httpretty-1.0.5/ > 7903d608efc89c14afb4d692a3721156e31a43e2/httpretty-1.0.5/COPYING > [...] > Checking httpretty-1.0.5 digest... ok > Assembling the tarball httpretty-1.0.5.tar > Checking httpretty-1.0.5.tar digest... ok > Assembling the Gzip file httpretty-1.0.5.tar.gz > Checking httpretty-1.0.5.tar.gz digest... ok > Copying result to /gnu/store/kbcnm57y2q1jvhvd8zw1g5vdiwlv19y9-httpret= ty-1.0.5.tar.gz > successfully built /gnu/store/k0b3c7kgzyn1nlyhx192pcbcgbfnhnwa-httpre= tty-1.0.5.tar.gz.drv Commits 67bf61255414115ffae0141df9dd3623bc742bff and 0b1f70d1a792af40aa0d13b3d227fde88f02d061 add the dependency on Disarchive, so this fallback path is now enabled! > There=E2=80=99s lots to talk about though.... > > First, it looks up the metadata on my server. This is fine for a demo, > but not what we want forever. The patch series supports adding several > mirrors for looking up the metadata. In the past, we talked about > putting everything on one or a few of the big Git hosting platforms like > GitHub or Gitlab. That way, it would be easily picked up by SWH and > archived =E2=80=9Cforever=E2=80=9D. Right now, I have Cuirass set up to = build the > metadata, and a little script that moves it from the build server to my > Web server. It would be simple enough to adjust that script to push it > to a remote Git repo. (Of course, the next step is to move this setup > to Guix infrastructure.) Thoughts? We should talk to SWH, giving them the figures you gave earlier in this thread. But yeah, a Git repo looks best to me (it would be useful to keep track of changes, for example if we eventually update metadata to a new format) and it simplifies archival to SWH. Second thing we need to figure out if where to create this database. If you have a Cuirass job already, we should run it on ci.guix. WDYT? > On the code level, there were two things I couldn=E2=80=99t figure out for > myself. > > I made the mirror list just simple strings. AIUI, the client and the > daemon have to agree about the format of the mirror list. Given that > running old daemons is common, changing the format is difficult. Is it > worth it to copy the more flexible interface used by the content > addressed mirrors? If yes, do I have to do the same =E2=80=98module-auto= load!=E2=80=99 > dance to use =E2=80=98bytevector->base16-string=E2=80=99? :) (I probabl= y would have > just copied it, but that part confused me a bit.) I had overlooked this suggestion of yours. Yes, I think it=E2=80=99s best = to copy the SWH scheme. Don=E2=80=99t worry about =E2=80=98module-autoload!= =E2=80=99: nowadays we can safely assume (guix base16) is available. When we change from list-of-strings to list-of-procedures, we=E2=80=99ll ha= ve to adjust the (guix build download) code so that it can deal with both. > I imported some modules from =E2=80=9Cguix/build/download.scm=E2=80=9D (w= ell, just > =E2=80=9Cbase16=E2=80=9D and =E2=80=9Cswh=E2=80=9D). It feels weird to u= se a bunch of host-side modules > from what=E2=80=99s nominally a =E2=80=9Cguix/build=E2=80=9D module. Thi= s is okay because > =E2=80=9Cguix/build/download.scm=E2=80=9D is not /really/ build-side code= . It=E2=80=99s more > like daemon (-ish) code that just happens to live in =E2=80=9Cguix/build= =E2=80=9D, which > is why importing host-side modules is OK... right? Yup. :-) In the end, the whole point is to reuse code on both sides, and that=E2=80=99s what=E2=80=99s being done here. Thanks, Ludo=E2=80=99.