From: "Ludovic Courtès" <ludo@gnu.org>
To: Maxime Devos <maximedevos@telenet.be>
Cc: 46800@debbugs.gnu.org
Subject: [bug#46800] [PATCH] Allow defining multiple substituters
Date: Fri, 12 Mar 2021 18:37:53 +0100 [thread overview]
Message-ID: <878s6sqnm6.fsf_-_@gnu.org> (raw)
In-Reply-To: <865935bc92c080a9ab79044856b01654fcc4410b.camel@telenet.be> (Maxime Devos's message of "Thu, 04 Mar 2021 08:48:44 +0100")
Hi Maxime,
Maxime Devos <maximedevos@telenet.be> skribis:
> On Tue, 2021-03-02 at 21:37 +0100, Ludovic Courtès wrote:
[...]
>> As discussed on IRC, the daemon used to have support for multiple
>> substituters, but as a built-in C++ interface, which I removed in
>> f6919ebdc6b0ce0286814cc6ab0564b1a4c67f5f.
>
> Was there any particular reason this support was removed, beyond
> moving from C++ to Scheme and the absence of any alternative substituters?
These were the main reasons, yes.
>> The Scheme interface you propose is of course nicer :-), but I’m still
>> not sure it’s necessary. For example, in the IPFS prototype at
>> <https://issues.guix.gnu.org/33899>;, IPFS support goes hand in hand with
>> HTTP support: narinfos are retrieved over HTTP and nars can be retrieved
>> over IPFS, or HTTP.
>
> About X going hand-in-hand with Y:
>
> Note that fetching narinfos, or fetching the nar itself are separated
> A method can support both procedures, or just one of them (or none,
> but that's rather useless.)
>
> Users (well, the system administrator) can choose multiple methods, which
> will be each fetch narinfos after each other & combine the
> results into
> one large list (or maybe some other data structure, I don't recall the
> details), and each substituter will be asked to produce
> a nar until a substituter
> succeeds or all have said "sorry, I don't have that nar".
OK.
> (That's different from C++ interface for multiple substituters I think, where
> the methods are only tried sequentialy, they aren't combined.)
>
> In case of IPFS, the idea is that *both* the IPFS and HTTP substituter are
> enabled, in that order: "--substitute-methods=ipfs http". The IPFS substituter
> won't be able to produce any narinfos by itself, but that's no problem as
> the HTTP substituter can find some. Then, the IPFS substituter will be asked
> first to download a substitute, as it's first in the "--substitute-methods" list.
>
> And what if the narinfo doesn't have a IPFS URI, as the substitute server doesn't
> support that? Then "guix substitute" automatically fall-backs to HTTP.
>
> Summary: some substitution methods can't do everything on their own, but that's ok,
> as "guix substitute" will just ask them to try what they can and will see if some
> combination of methods works.
Alright.
> About ‘not sure it's necessary’: there presumably will be a GNUnet substituter
> at some point. I suppose it would be possible to define all substitute methods
> in (guix scripts substitute), but then you would still end up with a procedure
> that tries all methods (e.g. in wip-ipfs-substitutes, process-substitution has
> an "if" structures that tries downloading via IPFS with fall-back to HTTP; this
> would become a (cond (ipfs? ipfs-code) (gnunet? gnunet-code) (#t http-code?))
I guess considerations that are more important to me (and to users, I
suppose) now than a few years back are maintainability and robustness.
Concretely, I wouldn’t want Guix to offer out of the box 4 methods, 3 of
which perform poorly or are downright buggy. I think it would be more
fruitful if, as a project, we would focus on one or two methods or
method combinations that we have battle-tested, perform well, and a nice
long-term maintenance story, and so on.
[...]
>> we’d rather let them choose a policy that
>> can automatically pick the “best” method, dynamically adjusting choices.
>
> Who's the user here?
> (a) the system administrator, who configuring the daemon to use a certain
> list of substituters and defines a default list of substitute uris.
> (b) the ‘user’, that doesn't directly have the capability to modify
> the system's guix daemon (or possibly an administrator that wants to
> to test some things out without the possibility of accidentally messing
> up the ‘real’ system).
I think (b) should be possible, just like users can pass
‘--substitute-urls’.
[...]
> About *automatically* dynamically adjusting choices: would be nice, but how is
> this supposed to work? Any ideas? The only thing I could think of is a
> allowing the user to choose which narinfo to use (e.g. from the list of found
> narinfos try to choose a narinfo that has an IPFS URI).
I think it’ll have to be fine-tuned once we have several stable
substitute methods. After all, we have yet to figure out how to choose
between zstd and lzip for the current substitution mechanism; the
tradeoffs when very different methods are in use may be more complex!
>> All in all, I would prefer to wait until there’s a clear need for this
>> abstraction.
>
> See above responses.
I don’t think my concerns are really addressed :-), but at the same time
I think we need a playground for these things so we can actually grow
new substitute methods like those you’ve been looking at. Hmmm tricky!
Ludo’.
next prev parent reply other threads:[~2021-03-12 18:38 UTC|newest]
Thread overview: 6+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-02-26 17:41 [bug#46800] [PATCH] Allow defining multiple substituters Maxime Devos
2021-03-02 20:37 ` Ludovic Courtès
2021-03-04 7:48 ` Maxime Devos
2021-03-12 17:37 ` Ludovic Courtès [this message]
2021-03-05 20:05 ` Maxime Devos
2021-06-06 17:52 ` Tony Olagbaiye
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: https://guix.gnu.org/
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=878s6sqnm6.fsf_-_@gnu.org \
--to=ludo@gnu.org \
--cc=46800@debbugs.gnu.org \
--cc=maximedevos@telenet.be \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://git.savannah.gnu.org/cgit/guix.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).