From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp1 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms11 with LMTPS id rg/jKAy1S2BTKwAA0tVLHw (envelope-from ) for ; Fri, 12 Mar 2021 18:38:04 +0000 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp1 with LMTPS id eDxoJAy1S2AOOgAAbx9fmQ (envelope-from ) for ; Fri, 12 Mar 2021 18:38:04 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 1238215295 for ; Fri, 12 Mar 2021 19:38:04 +0100 (CET) Received: from localhost ([::1]:44630 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lKmfi-0001DI-Lr for larch@yhetil.org; Fri, 12 Mar 2021 13:38:02 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:34122) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lKlkc-0004IS-PT for guix-patches@gnu.org; Fri, 12 Mar 2021 12:39:02 -0500 Received: from debbugs.gnu.org ([209.51.188.43]:46732) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1lKlkc-0006cV-HP for guix-patches@gnu.org; Fri, 12 Mar 2021 12:39:02 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1lKlkc-0001e6-FI for guix-patches@gnu.org; Fri, 12 Mar 2021 12:39:02 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#46800] [PATCH] Allow defining multiple substituters Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Fri, 12 Mar 2021 17:39:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 46800 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Maxime Devos Cc: 46800@debbugs.gnu.org Received: via spool by 46800-submit@debbugs.gnu.org id=B46800.16155706896254 (code B ref 46800); Fri, 12 Mar 2021 17:39:02 +0000 Received: (at 46800) by debbugs.gnu.org; 12 Mar 2021 17:38:09 +0000 Received: from localhost ([127.0.0.1]:58274 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lKljl-0001co-Cp for submit@debbugs.gnu.org; Fri, 12 Mar 2021 12:38:09 -0500 Received: from eggs.gnu.org ([209.51.188.92]:44036) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lKlji-0001cI-GL for 46800@debbugs.gnu.org; Fri, 12 Mar 2021 12:38:08 -0500 Received: from fencepost.gnu.org ([2001:470:142:3::e]:58077) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lKljY-0006CC-LB; Fri, 12 Mar 2021 12:38:00 -0500 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=33926 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1lKljW-0000gs-Ri; Fri, 12 Mar 2021 12:37:56 -0500 From: Ludovic =?UTF-8?Q?Court=C3=A8s?= References: <87o8g1thpr.fsf@gnu.org> <865935bc92c080a9ab79044856b01654fcc4410b.camel@telenet.be> Date: Fri, 12 Mar 2021 18:37:53 +0100 In-Reply-To: <865935bc92c080a9ab79044856b01654fcc4410b.camel@telenet.be> (Maxime Devos's message of "Thu, 04 Mar 2021 08:48:44 +0100") Message-ID: <878s6sqnm6.fsf_-_@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+larch=yhetil.org@gnu.org Sender: "Guix-patches" X-Migadu-Flow: FLOW_IN ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1615574284; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:resent-cc: resent-from:resent-sender:resent-message-id:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post; bh=HHLFiBVWs6UQjdWHmRE3HVjuMeaVGExmdEoyZI9aocg=; b=C7Of0ojpDUEMpySLXgLUVbjVbxGsCFTGFW4UyilicGXr2JjgeHszQs101kUClK1vpR9n0X wjCbTBbRzG/McojAlAZpcFA0+iGYanrAZYocK0fq4uESxYUhbI+UG0R1pzZieTF+h85XTX sRmu+92zkutXOYSz7dCXSTn32OPi+lwUbcKK2MDwjvO73+PAWFiYe+5TKVY6omV8E0vChb /RjPVtvCCOI1TmRC8oogQ84spxK9tooTDMIJA1UNvvyE7km1Z+X4EvKdrEo4YJ3wl+Q3cd YkojftrBItKn1Arsz16Nl3Qjk9MRscwq/2vIXehgPo62UDua/3WCIwprPQc1VA== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1615574284; a=rsa-sha256; cv=none; b=ANSdlTM89OSIiiALByWnJdx1arzPccPFROpGreV0R2YyA+8vkgkN+EXUbrfRqtZWlN5pdq gTRFeXgm4GZmkZIIhoXyuY4N/KQisFydnPOUi42tmoH5b5/fjmDq3XJYBIek+70QZh5ePm HIm23nBYZ2DkWEYsVO2j3fvlbI9PDySBU4QuTyCncULsXJk2DZzy1BnXLVwrctPQ6yYrA5 sR7alRwg1xEhQYCJS+2aL1aVz73vTWZi9/QJTbsTBPMK8LAjmc+Jx5kPuuCOTizUA2kEEy 1eTZijKt5AOkmffLaTiyJU4ZsGq4GZs5xSQ2hqgjK5BWV+zWGWd5Y7N70CsksQ== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=none; dmarc=pass (policy=none) header.from=gnu.org; spf=pass (aspmx1.migadu.com: domain of guix-patches-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-patches-bounces@gnu.org X-Migadu-Spam-Score: -2.89 Authentication-Results: aspmx1.migadu.com; dkim=none; dmarc=pass (policy=none) header.from=gnu.org; spf=pass (aspmx1.migadu.com: domain of guix-patches-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-patches-bounces@gnu.org X-Migadu-Queue-Id: 1238215295 X-Spam-Score: -2.89 X-Migadu-Scanner: scn0.migadu.com X-TUID: YVUlelycE4SS Hi Maxime, Maxime Devos skribis: > On Tue, 2021-03-02 at 21:37 +0100, Ludovic Court=C3=A8s wrote: [...] >> As discussed on IRC, the daemon used to have support for multiple >> substituters, but as a built-in C++ interface, which I removed in >> f6919ebdc6b0ce0286814cc6ab0564b1a4c67f5f. > > Was there any particular reason this support was removed, beyond > moving from C++ to Scheme and the absence of any alternative substituters? These were the main reasons, yes. >> The Scheme interface you propose is of course nicer :-), but I=E2=80=99m= still >> not sure it=E2=80=99s necessary. For example, in the IPFS prototype at >> ;, IPFS support goes hand in hand with >> HTTP support: narinfos are retrieved over HTTP and nars can be retrieved >> over IPFS, or HTTP. > > About X going hand-in-hand with Y: > > Note that fetching narinfos, or fetching the nar itself are separated > A method can support both procedures, or just one of them (or none, > but that's rather useless.) > > Users (well, the system administrator) can choose multiple methods, which > will be each fetch narinfos after each other & combine the > results into > one large list (or maybe some other data structure, I don't recall the > details), and each substituter will be asked to produce > a nar until a substituter > succeeds or all have said "sorry, I don't have that nar". OK. > (That's different from C++ interface for multiple substituters I think, w= here > the methods are only tried sequentialy, they aren't combined.) > > In case of IPFS, the idea is that *both* the IPFS and HTTP substituter are > enabled, in that order: "--substitute-methods=3Dipfs http". The IPFS sub= stituter > won't be able to produce any narinfos by itself, but that's no problem as > the HTTP substituter can find some. Then, the IPFS substituter will be a= sked > first to download a substitute, as it's first in the "--substitute-method= s" list. > > And what if the narinfo doesn't have a IPFS URI, as the substitute server= doesn't > support that? Then "guix substitute" automatically fall-backs to HTTP. > > Summary: some substitution methods can't do everything on their own, but = that's ok, > as "guix substitute" will just ask them to try what they can and will see= if some > combination of methods works. Alright. > About =E2=80=98not sure it's necessary=E2=80=99: there presumably will be= a GNUnet substituter > at some point. I suppose it would be possible to define all substitute m= ethods > in (guix scripts substitute), but then you would still end up with a proc= edure > that tries all methods (e.g. in wip-ipfs-substitutes, process-substitutio= n has > an "if" structures that tries downloading via IPFS with fall-back to HTTP= ; this > would become a (cond (ipfs? ipfs-code) (gnunet? gnunet-code) (#t http-cod= e?)) I guess considerations that are more important to me (and to users, I suppose) now than a few years back are maintainability and robustness. Concretely, I wouldn=E2=80=99t want Guix to offer out of the box 4 methods,= 3 of which perform poorly or are downright buggy. I think it would be more fruitful if, as a project, we would focus on one or two methods or method combinations that we have battle-tested, perform well, and a nice long-term maintenance story, and so on. [...] >> we=E2=80=99d rather let them choose a policy that >> can automatically pick the =E2=80=9Cbest=E2=80=9D method, dynamically ad= justing choices. > > Who's the user here? > (a) the system administrator, who configuring the daemon to use a certain > list of substituters and defines a default list of substitute uris. > (b) the =E2=80=98user=E2=80=99, that doesn't directly have the capability= to modify > the system's guix daemon (or possibly an administrator that wants to > to test some things out without the possibility of accidentally messi= ng > up the =E2=80=98real=E2=80=99 system). I think (b) should be possible, just like users can pass =E2=80=98--substitute-urls=E2=80=99. [...] > About *automatically* dynamically adjusting choices: would be nice, but h= ow is > this supposed to work? Any ideas? The only thing I could think of is a > allowing the user to choose which narinfo to use (e.g. from the list of f= ound > narinfos try to choose a narinfo that has an IPFS URI). I think it=E2=80=99ll have to be fine-tuned once we have several stable substitute methods. After all, we have yet to figure out how to choose between zstd and lzip for the current substitution mechanism; the tradeoffs when very different methods are in use may be more complex! >> All in all, I would prefer to wait until there=E2=80=99s a clear need fo= r this >> abstraction. > > See above responses. I don=E2=80=99t think my concerns are really addressed :-), but at the same= time I think we need a playground for these things so we can actually grow new substitute methods like those you=E2=80=99ve been looking at. Hmmm tri= cky! Ludo=E2=80=99.