On Tue, 2021-03-02 at 21:37 +0100, Ludovic Courtès wrote: > Hi Maxime, > > Maxime Devos skribis: > > > This patch series is my suggestion for allowing > > multiple "substitution methods" or "substituters" > > as I call them. Currently, only a method for HTTP/S > > is defined, though I hope it will be a good basis > > for a common framework for substitutes over GNUnet > > and IPFS. > > Thanks for working on this! > > As discussed on IRC, the daemon used to have support for multiple > substituters, but as a built-in C++ interface, which I removed in > f6919ebdc6b0ce0286814cc6ab0564b1a4c67f5f. Was there any particular reason this support was removed, beyond moving from C++ to Scheme and the absence of any alternative substituters? > The Scheme interface you propose is of course nicer :-), but I’m still > not sure it’s necessary. For example, in the IPFS prototype at > ;, IPFS support goes hand in hand with > HTTP support: narinfos are retrieved over HTTP and nars can be retrieved > over IPFS, or HTTP. About X going hand-in-hand with Y: Note that fetching narinfos, or fetching the nar itself are separated A method can support both procedures, or just one of them (or none, but that's rather useless.) Users (well, the system administrator) can choose multiple methods, which will be each fetch narinfos after each other & combine the results into one large list (or maybe some other data structure, I don't recall the details), and each substituter will be asked to produce a nar until a substituter succeeds or all have said "sorry, I don't have that nar". (That's different from C++ interface for multiple substituters I think, where the methods are only tried sequentialy, they aren't combined.) In case of IPFS, the idea is that *both* the IPFS and HTTP substituter are enabled, in that order: "--substitute-methods=ipfs http". The IPFS substituter won't be able to produce any narinfos by itself, but that's no problem as the HTTP substituter can find some. Then, the IPFS substituter will be asked first to download a substitute, as it's first in the "--substitute-methods" list. And what if the narinfo doesn't have a IPFS URI, as the substitute server doesn't support that? Then "guix substitute" automatically fall-backs to HTTP. Summary: some substitution methods can't do everything on their own, but that's ok, as "guix substitute" will just ask them to try what they can and will see if some combination of methods works. About ‘not sure it's necessary’: there presumably will be a GNUnet substituter at some point. I suppose it would be possible to define all substitute methods in (guix scripts substitute), but then you would still end up with a procedure that tries all methods (e.g. in wip-ipfs-substitutes, process-substitution has an "if" structures that tries downloading via IPFS with fall-back to HTTP; this would become a (cond (ipfs? ipfs-code) (gnunet? gnunet-code) (#t http-code?)) Note that there's (guix scripts import X) and (guix build-system X). > Likewise with “digests”: ;. I haven't taken a close look at this yet before (I haven't been around guix development for long). To me, this seems compatible with this patch actually. The HTTP substituter's procedure for downloading the substitute itself (process-substitution/http in my patch) could be split in two, and look at the narinfo to see whether the 'digest' or the usual mechanism should be used. Alternatively, one could define *two* substituters: the ‘standard’ http substituter ‘http’, and the ‘http-digest’ substituter that can't fetch narinfo's, but rather is an alternative method for downloading the substitute. The daemon can be started with "--substitute-methods http-digest http" to prefer downloading via the ‘http-digest’ method when possible, but uses ‘http’ for the narinfos and as a fallback for when the narinfo does not have a digest. But what if a non-HTTP substituter wants to use digests? Well, I don't know any such substituters (-:. But for the (hypothetical) GNUnet substituter & the wip IPFS substituter, I don't think they will use the digests code. > Another issue is that it may be that, instead of letting users choose > one method and stick to it, They (at least the system administrator) can choose a list of substituters, see above. > we’d rather let them choose a policy that > can automatically pick the “best” method, dynamically adjusting choices. Who's the user here? (a) the system administrator, who configuring the daemon to use a certain list of substituters and defines a default list of substitute uris. (b) the ‘user’, that doesn't directly have the capability to modify the system's guix daemon (or possibly an administrator that wants to to test some things out without the possibility of accidentally messing up the ‘real’ system). If (b), I think it would be ideal to give the (unprivileged) user the possibility of using their own substituter(s) (under their own capabilities, not root), albeit at the cost of the guix daemon having to verify the narhash & narinfo signature. That could be implemented as a separate patch (though this patch would need to be rebased then). WDYT? Would be useful for developing new substituters and testing them, I think. About *automatically* dynamically adjusting choices: would be nice, but how is this supposed to work? Any ideas? The only thing I could think of is a allowing the user to choose which narinfo to use (e.g. from the list of found narinfos try to choose a narinfo that has an IPFS URI). Also, for (a) the shepherd service could use a "set-substitute-methods" option, and perhaps the user (b) could be allowed to select a subset of these substitute methods to use when running "guix build PACKAGE" and the like (but only a subset, as "guix substitute" when invoked by the daemon runs as root and therefore the potential attack surface shouldn't be increased beyond what the administrator allows). > All in all, I would prefer to wait until there’s a clear need for this > abstraction. See above responses. WDYT? Thanks, Maxime.