From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp1 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms11 with LMTPS id 3MqfNXibQGCFHwAA0tVLHw (envelope-from ) for ; Thu, 04 Mar 2021 08:34:00 +0000 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp1 with LMTPS id cKbRMHibQGBnPAAAbx9fmQ (envelope-from ) for ; Thu, 04 Mar 2021 08:34:00 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id ED8C81051F for ; Thu, 4 Mar 2021 09:33:59 +0100 (CET) Received: from localhost ([::1]:51268 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lHjQl-0003sF-4V for larch@yhetil.org; Thu, 04 Mar 2021 03:33:59 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:45206) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lHijG-0007S2-LJ for guix-patches@gnu.org; Thu, 04 Mar 2021 02:49:02 -0500 Received: from debbugs.gnu.org ([209.51.188.43]:46032) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1lHijG-0000ph-EG for guix-patches@gnu.org; Thu, 04 Mar 2021 02:49:02 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1lHijG-00026m-A1 for guix-patches@gnu.org; Thu, 04 Mar 2021 02:49:02 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#46800] [PATCH] Allow defining multiple substituters Resent-From: Maxime Devos Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Thu, 04 Mar 2021 07:49:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 46800 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Ludovic =?UTF-8?Q?Court=C3=A8s?= Cc: 46800@debbugs.gnu.org Received: via spool by 46800-submit@debbugs.gnu.org id=B46800.16148441388095 (code B ref 46800); Thu, 04 Mar 2021 07:49:02 +0000 Received: (at 46800) by debbugs.gnu.org; 4 Mar 2021 07:48:58 +0000 Received: from localhost ([127.0.0.1]:57578 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lHijB-00026U-Tz for submit@debbugs.gnu.org; Thu, 04 Mar 2021 02:48:58 -0500 Received: from laurent.telenet-ops.be ([195.130.137.89]:55710) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lHij9-00026K-8m for 46800@debbugs.gnu.org; Thu, 04 Mar 2021 02:48:56 -0500 Received: from ptr-bvsjgyjmffd7q9timvx.18120a2.ip6.access.telenet.be ([IPv6:2a02:1811:8c09:9d00:aaf1:9810:a0b8:a55d]) by laurent.telenet-ops.be with bizsmtp id c7os2400u0mfAB4017othY; Thu, 04 Mar 2021 08:48:53 +0100 Message-ID: <865935bc92c080a9ab79044856b01654fcc4410b.camel@telenet.be> From: Maxime Devos Date: Thu, 04 Mar 2021 08:48:44 +0100 In-Reply-To: <87o8g1thpr.fsf@gnu.org> References: <87o8g1thpr.fsf@gnu.org> Content-Type: multipart/signed; micalg="pgp-sha256"; protocol="application/pgp-signature"; boundary="=-1Z9/wUDbSokEVoujQAn2" User-Agent: Evolution 3.34.2 MIME-Version: 1.0 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=telenet.be; s=r21; t=1614844133; bh=BxYsOB5lmTBpI8n7MzigAuINkxbesZ6X4hiV4MfhxHo=; h=Subject:From:To:Cc:Date:In-Reply-To:References; b=jdGAWl8nEkGneqBpBeTsdTvRQVF7rldkESFhrvH3NuPrGK/wUx6d7MhtlmWuJ2IW6 8cqGDRVPsghTiJdD6Ja5YyqWbiBKyKEHe9GjFw3smaYzJ6DkRmvLFyu4ilXW3Jz+Um fsii2t7qkAYMRr49Tc4hsSqPpduWaWZQ56t/puXRo22UPar0szXK2zGc1LymYRzjOA 7C/tkhmD957W2pxKmukd1yEtBncsQhyNvLH4WrHEC6yUi5CIGVd903iBVDtvK40IxO DiV0rgN0Z2at1eRRUEPAGqsq3NmY9cnVUskYx5417hiu5fUJjCTO0wLEueCknhqL9c C0DOnIvSJtaLg== X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-Mailman-Approved-At: Thu, 04 Mar 2021 03:33:53 -0500 X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+larch=yhetil.org@gnu.org Sender: "Guix-patches" X-Migadu-Flow: FLOW_IN ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1614846840; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:resent-cc:resent-from:resent-sender: resent-message-id:in-reply-to:in-reply-to:references:references: list-id:list-help:list-unsubscribe:list-subscribe:list-post: dkim-signature; bh=BxYsOB5lmTBpI8n7MzigAuINkxbesZ6X4hiV4MfhxHo=; b=nwPBdMAAoFGsCiN8j0D2qkn50swxDt1c2mQERccobENYcwesXsk9cSDL5hG15HMcY54les zTn7o9dGalf4Cni4kQAxadiJQA+wTu0qTfbpAZdIQIP6RNcHBRSuJVxTlzn0spuzy+8azB dAXwfo4Jbr4tLW0UKtE+40lVzE7s6IA/ZpGq3Rsi75s2bM5Y7FZ3CeeQif/q3Ok8RBQI8L eTBd5M9H2niek4krhBKyPWvd3mq1n/A3EMS0XnyMcm0Z1dnMz1RhvBatoT3tRneSGdzKwl J9CqJLbYpASkLuff0YTNoKsRmt8OTmlpqP2rRzXPpuRCp3GTPp8f4t+g9g8vtA== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1614846840; a=rsa-sha256; cv=none; b=KEFQEVOpKWV3A9PnRm5L022hysZ1ze76sUho6VJb1XoNnj+P/hZUv65JdmdEWp4IFJSMvE fw8WPj9EIMnFEk9Xj+aKb90/qLBpIdXKD4WQ/70Ykx4nDZwAhH9IsGiMyggF97tkwKd4tn lOBXTD4Wd0LXrSfLsdPX1zzEZKDLYsyyshqkYY1DtUBY6BErabslsgLNglo7Da1/I89Asc 3DOeAu5NrOmRAsYFrTO3i/RlwcLnuRqQG/yn414Wsu/fkBoBpSalVQWnnRScIyR1XfstXQ FZc0zjOpy4raetQmGrt9SwPSKzLwWMykHm4DLiQRjAUj+e867b5W9anmHuOySw== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=telenet.be header.s=r21 header.b=jdGAWl8n; dmarc=fail reason="SPF not aligned (relaxed)" header.from=telenet.be (policy=none); spf=pass (aspmx1.migadu.com: domain of guix-patches-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-patches-bounces@gnu.org X-Migadu-Spam-Score: -3.36 Authentication-Results: aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=telenet.be header.s=r21 header.b=jdGAWl8n; dmarc=fail reason="SPF not aligned (relaxed)" header.from=telenet.be (policy=none); spf=pass (aspmx1.migadu.com: domain of guix-patches-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-patches-bounces@gnu.org X-Migadu-Queue-Id: ED8C81051F X-Spam-Score: -3.36 X-Migadu-Scanner: scn1.migadu.com X-TUID: ZRSD2Zh9rp8i --=-1Z9/wUDbSokEVoujQAn2 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Tue, 2021-03-02 at 21:37 +0100, Ludovic Court=C3=A8s wrote: > Hi Maxime, >=20 > Maxime Devos skribis: >=20 > > This patch series is my suggestion for allowing > > multiple "substitution methods" or "substituters" > > as I call them. Currently, only a method for HTTP/S > > is defined, though I hope it will be a good basis > > for a common framework for substitutes over GNUnet > > and IPFS. >=20 > Thanks for working on this! >=20 > As discussed on IRC, the daemon used to have support for multiple > substituters, but as a built-in C++ interface, which I removed in > f6919ebdc6b0ce0286814cc6ab0564b1a4c67f5f. Was there any particular reason this support was removed, beyond moving from C++ to Scheme and the absence of any alternative substituters? > The Scheme interface you propose is of course nicer :-), but I=E2=80=99m = still > not sure it=E2=80=99s necessary. For example, in the IPFS prototype at > ;, IPFS support goes hand in hand with > HTTP support: narinfos are retrieved over HTTP and nars can be retrieved > over IPFS, or HTTP. About X going hand-in-hand with Y: Note that fetching narinfos, or fetching the nar itself are separated A method can support both procedures, or just one of them (or none, but that's rather useless.) Users (well, the system administrator) can choose multiple methods, which will be each fetch narinfos after each other & combine the results into one large list (or maybe some other data structure, I don't recall the details), and each substituter will be asked to produce a nar until a substituter succeeds or all have said "sorry, I don't have that nar". (That's different from C++ interface for multiple substituters I think, whe= re the methods are only tried sequentialy, they aren't combined.) In case of IPFS, the idea is that *both* the IPFS and HTTP substituter are enabled, in that order: "--substitute-methods=3Dipfs http". The IPFS subst= ituter won't be able to produce any narinfos by itself, but that's no problem as the HTTP substituter can find some. Then, the IPFS substituter will be ask= ed first to download a substitute, as it's first in the "--substitute-methods"= list. And what if the narinfo doesn't have a IPFS URI, as the substitute server d= oesn't support that? Then "guix substitute" automatically fall-backs to HTTP. Summary: some substitution methods can't do everything on their own, but th= at's ok, as "guix substitute" will just ask them to try what they can and will see i= f some combination of methods works. About =E2=80=98not sure it's necessary=E2=80=99: there presumably will be a= GNUnet substituter at some point. I suppose it would be possible to define all substitute met= hods in (guix scripts substitute), but then you would still end up with a proced= ure that tries all methods (e.g. in wip-ipfs-substitutes, process-substitution = has an "if" structures that tries downloading via IPFS with fall-back to HTTP; = this would become a (cond (ipfs? ipfs-code) (gnunet? gnunet-code) (#t http-code?= )) Note that there's (guix scripts import X) and (guix build-system X). > Likewise with =E2=80=9Cdigests=E2=80=9D: ;. I haven't taken a close look at this yet before (I haven't been around guix development for long). To me, this seems compatible with this patch actual= ly. The HTTP substituter's procedure for downloading the substitute itself (process-substitution/http in my patch) could be split in two, and look at the narinfo to see whether the 'digest' or the usual mechanism should be= used. Alternatively, one could define *two* substituters: the =E2=80=98standard= =E2=80=99 http substituter =E2=80=98http=E2=80=99, and the =E2=80=98http-digest=E2=80=99 substituter t= hat can't fetch narinfo's, but rather is an alternative method for downloading the substitute. The daemon can be= started with "--substitute-methods http-digest http" to prefer downloading via the = =E2=80=98http-digest=E2=80=99 method when possible, but uses =E2=80=98http=E2=80=99 for the narinfos and = as a fallback for when the narinfo does not have a digest. But what if a non-HTTP substituter wants to use digests? Well, I don't kno= w any such substituters (-:. But for the (hypothetical) GNUnet substituter & the wip = IPFS substituter, I don't think they will use the digests code. > Another issue is that it may be that, instead of letting users choose > one method and stick to it, They (at least the system administrator) can choose a list of substituters, see above. > we=E2=80=99d rather let them choose a policy that > can automatically pick the =E2=80=9Cbest=E2=80=9D method, dynamically adj= usting choices. Who's the user here? (a) the system administrator, who configuring the daemon to use a certain list of substituters and defines a default list of substitute uris. (b) the =E2=80=98user=E2=80=99, that doesn't directly have the capability t= o modify the system's guix daemon (or possibly an administrator that wants to to test some things out without the possibility of accidentally messing up the =E2=80=98real=E2=80=99 system). If (b), I think it would be ideal to give the (unprivileged) user the possibility of using their own substituter(s) (under their own capabilities= , not root), albeit at the cost of the guix daemon having to verify the narha= sh & narinfo signature. That could be implemented as a separate patch (though this patch would need to be rebased then). WDYT? Would be useful for developing new substituter= s and testing them, I think. About *automatically* dynamically adjusting choices: would be nice, but how= is this supposed to work? Any ideas? The only thing I could think of is a allowing the user to choose which narinfo to use (e.g. from the list of fou= nd narinfos try to choose a narinfo that has an IPFS URI). Also, for (a) the shepherd service could use a "set-substitute-methods" opt= ion, and perhaps the user (b) could be allowed to select a subset of these subst= itute methods to use when running "guix build PACKAGE" and the like (but only a s= ubset, as "guix substitute" when invoked by the daemon runs as root and therefore = the potential attack surface shouldn't be increased beyond what the administrat= or allows). > All in all, I would prefer to wait until there=E2=80=99s a clear need for= this > abstraction. See above responses. WDYT? Thanks, Maxime. --=-1Z9/wUDbSokEVoujQAn2 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part Content-Transfer-Encoding: 7bit -----BEGIN PGP SIGNATURE----- iI0EABYIADUWIQTB8z7iDFKP233XAR9J4+4iGRcl7gUCYECQ3RccbWF4aW1lZGV2 b3NAdGVsZW5ldC5iZQAKCRBJ4+4iGRcl7ltBAP9iYlOav2/YVN2Zt3CBeW0DASO2 PGooVzqFmZwePUzMqAD+PEzziVN7j7ToALv6bJWH74o61yFFk2WziDK1P/wx3w4= =poFh -----END PGP SIGNATURE----- --=-1Z9/wUDbSokEVoujQAn2--