bug#22990: Grafts leads to inefficient substitute info retrieval

bug#22990: Grafts leads to inefficient substitute info retrieval

[Ludovic Courtès]

As of right now (v0.9.0-2007-g66a30a3), ‘graft-derivation’ works either by:

  1. Fetching substitute info about the things being built so that it
     can determine its references, which in turns allows it to determine
     whether they need to be grafted.

  2. Building stuff, as a last resort, so that it can determine its
     references.

Case #1 is hopefully going to be the most common.

The problem with #1 is that when building a profile, we do one
‘package-derivation’ call for each package in the profile, which
translates in one ‘graft-derivation’ call for each relevant package¹,
which translates into one ‘references/substitutes’ call for each.

Concretely, what this means is this:

  $ guix package -u
  substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 100.0%
  substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 100.0%
  substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 100.0%
  substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 100.0%
  substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 100.0%
  substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 100.0%
  substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 100.0%
  […]
  The following files would be downloaded:

Each of the initial “updating list” message corresponds to an HTTP
request for a single narinfo file, which can take around 1 second.

Instead, the ideal thing would be to fetch the narinfo files for all the
relevant packages at once; that way, we’d spawn ‘guix substitute’ only
once, and it would benefit from HTTP pipelining (one round-trip instead
of N.)

To achieve this, I’m thinking of extending gexp code such that gexp
compilers can return a list of applicable grafts.  The ‘package’
compiler would do #:graft? #f and instead let ‘gexp->derivation’ call
‘graft-derivation’.

I’ll give it a try and report back.

Ludo’.

¹ A package is “relevant” if ‘package-grafts’ returns a non-empty list.

bug#22990: Grafts leads to inefficient substitute info retrieval

[Alex Kost]

Ludovic Courtès (2016-03-11 19:52 +0300) wrote:

> As of right now (v0.9.0-2007-g66a30a3), ‘graft-derivation’ works either by:
>
>   1. Fetching substitute info about the things being built so that it
>      can determine its references, which in turns allows it to determine
>      whether they need to be grafted.
>
>   2. Building stuff, as a last resort, so that it can determine its
>      references.

I noticed that #1 is happening even with --no-substitutes option.  Is it
intended?

I've tried this:

$ guix build --dry-run --no-substitutes mutt
substitute: updating list of substitutes from 'http://hydra.gnu.org'... 100.0%
substitute: updating list of substitutes from 'http://hydra.gnu.org'... 100.0%
substitute: updating list of substitutes from 'http://hydra.gnu.org'... 100.0%
substitute: updating list of substitutes from 'http://hydra.gnu.org'... 100.0%
substitute: updating list of substitutes from 'http://hydra.gnu.org'... 100.0%
The following derivations would be built:
   /gnu/store/a2w22xlmfwkgwx4vw11dxc6zrdmww435-mutt-1.5.24.drv
   ...

-- 
Alex

bug#22990: Grafts leads to inefficient substitute info retrieval

[Ludovic Courtès]

Alex Kost <alezost@gmail.com> skribis:

> Ludovic Courtès (2016-03-11 19:52 +0300) wrote:
>
>> As of right now (v0.9.0-2007-g66a30a3), ‘graft-derivation’ works either by:
>>
>>   1. Fetching substitute info about the things being built so that it
>>      can determine its references, which in turns allows it to determine
>>      whether they need to be grafted.
>>
>>   2. Building stuff, as a last resort, so that it can determine its
>>      references.
>
> I noticed that #1 is happening even with --no-substitutes option.  Is it
> intended?

Not really, but I see this is because ‘substitutable-path-info’ (called
from ‘references/substitutes’, called from ‘graft-derivation’) works
regardless of whether substitutes are enabled:

--8<---------------cut here---------------start------------->8---
scheme@(guile-user)> ,use(guix)
scheme@(guile-user)> (define s (open-connection))
scheme@(guile-user)> (set-build-options s #:use-substitutes? #f)
$2 = #t
scheme@(guile-user)> (valid-path? s "/gnu/store/qf2lm7jpiiyygxz8zq0r1ca1fazv6smn-mutt-1.5.24")
$3 = #f
scheme@(guile-user)> (substitutable-path-info s '("/gnu/store/qf2lm7jpiiyygxz8zq0r1ca1fazv6smn-mutt-1.5.24"))
$4 = (#<<substitutable> path: "/gnu/store/qf2lm7jpiiyygxz8zq0r1ca1fazv6smn-mutt-1.5.24" deriver: "/gnu/store/jcl9c3w463xa2g963q5a60rrd97y1g28-mutt-1.5.24.drv" refs: ("/gnu/store/3gmzl5jpk700hqyr8p3kfg0vgcnw8d97-libassuan-2.4.2" "/gnu/store/b02lmk67jq1vcflk2m2bwzc8gmwmndqp-ncurses-6.0" "/gnu/store/d3xdc2w87yw3raafwb9q34gxx4xqci8k-cyrus-sasl-2.1.26" "/gnu/store/pkasxagsa4z4viscfpl6sjszmdmwncl1-gcc-4.9.3-lib" "/gnu/store/qf2lm7jpiiyygxz8zq0r1ca1fazv6smn-mutt-1.5.24" "/gnu/store/qvx4q6lbwi4s3cwr8wqaa7kcva0a5c4b-openssl-1.0.2f" "/gnu/store/sb40mddkia0brc814xkbnhxccfm32q3a-gpgme-1.6.0" "/gnu/store/sgzfawy95pfn7nsw3xvmca58llm5zzbc-glibc-2.22" "/gnu/store/x2p2biyybcb2wac77qz9468asc5fm48i-perl-5.22.1" "/gnu/store/x8dmdlrn5qn0wrbcnngj55y3ab73h0pp-bash-4.3.42" "/gnu/store/zpxg45dq67psrn4wmfk4l635h0si8q63-libgpg-error-1.21") dl-size: 0 nar-size: 6661016>)
--8<---------------cut here---------------end--------------->8---

However, substitutes are not downloaded, so in this regard
--no-substitutes is honored.

Ludo’.

bug#22990: Grafts leads to inefficient substitute info retrieval

[Ludovic Courtès]

ludo@gnu.org (Ludovic Courtès) skribis:

> Concretely, what this means is this:
>
>   $ guix package -u
>   substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 100.0%
>   substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 100.0%
>   substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 100.0%
>   substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 100.0%
>   substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 100.0%
>   substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 100.0%
>   substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 100.0%
>   […]
>   The following files would be downloaded:

This is somewhat mitigated by commits
f09aea1b58b3ef961d3cc712f116fe4617bc8f90 and
264fdedb408ba3620d1e361de6c77e7925025301.

In addition, 026ca50fa4c46a8e280cd51621bbec76b12c0757 slightly improves
‘guix substitute’ performance.

Ludo’.

bug#22990: Grafts leads to inefficient substitute info retrieval

[Mark H Weaver]

ludo@gnu.org (Ludovic Courtès) writes:

> Alex Kost <alezost@gmail.com> skribis:
>
>> Ludovic Courtès (2016-03-11 19:52 +0300) wrote:
>>
>>> As of right now (v0.9.0-2007-g66a30a3), ‘graft-derivation’ works either by:
>>>
>>>   1. Fetching substitute info about the things being built so that it
>>>      can determine its references, which in turns allows it to determine
>>>      whether they need to be grafted.
>>>
>>>   2. Building stuff, as a last resort, so that it can determine its
>>>      references.
>>
>> I noticed that #1 is happening even with --no-substitutes option.  Is it
>> intended?
>
> Not really, but I see this is because ‘substitutable-path-info’ (called
> from ‘references/substitutes’, called from ‘graft-derivation’) works
> regardless of whether substitutes are enabled:
>
> scheme@(guile-user)> ,use(guix)
> scheme@(guile-user)> (define s (open-connection))
> scheme@(guile-user)> (set-build-options s #:use-substitutes? #f)
> $2 = #t
> scheme@(guile-user)> (valid-path? s "/gnu/store/qf2lm7jpiiyygxz8zq0r1ca1fazv6smn-mutt-1.5.24")
> $3 = #f
> scheme@(guile-user)> (substitutable-path-info s '("/gnu/store/qf2lm7jpiiyygxz8zq0r1ca1fazv6smn-mutt-1.5.24"))
> $4 = (#<<substitutable> path: "/gnu/store/qf2lm7jpiiyygxz8zq0r1ca1fazv6smn-mutt-1.5.24" deriver: "/gnu/store/jcl9c3w463xa2g963q5a60rrd97y1g28-mutt-1.5.24.drv" refs: ("/gnu/store/3gmzl5jpk700hqyr8p3kfg0vgcnw8d97-libassuan-2.4.2" "/gnu/store/b02lmk67jq1vcflk2m2bwzc8gmwmndqp-ncurses-6.0" "/gnu/store/d3xdc2w87yw3raafwb9q34gxx4xqci8k-cyrus-sasl-2.1.26" "/gnu/store/pkasxagsa4z4viscfpl6sjszmdmwncl1-gcc-4.9.3-lib" "/gnu/store/qf2lm7jpiiyygxz8zq0r1ca1fazv6smn-mutt-1.5.24" "/gnu/store/qvx4q6lbwi4s3cwr8wqaa7kcva0a5c4b-openssl-1.0.2f" "/gnu/store/sb40mddkia0brc814xkbnhxccfm32q3a-gpgme-1.6.0" "/gnu/store/sgzfawy95pfn7nsw3xvmca58llm5zzbc-glibc-2.22" "/gnu/store/x2p2biyybcb2wac77qz9468asc5fm48i-perl-5.22.1" "/gnu/store/x8dmdlrn5qn0wrbcnngj55y3ab73h0pp-bash-4.3.42" "/gnu/store/zpxg45dq67psrn4wmfk4l635h0si8q63-libgpg-error-1.21") dl-size: 0 nar-size: 6661016>)

Is the information from the substitute server authenticated by checking
hydra's signature against the list of keys in /etc/guix/acls?

The reason I ask is that if the set of runtime dependencies received is
incomplete, it could lead to incorrect grafting, namely that references
to compromised libraries could be retained.

> However, substitutes are not downloaded, so in this regard
> --no-substitutes is honored.

It depends on the intent of --no-substitutes.  If the intent is to avoid
trusting the substitute server, then by relying on the accuracy of the
runtime dependency data from Hydra, we are failing to honor that intent.

That said, I think it's okay to document that --no-substitutes alone is
not sufficient to avoid trusting a substitute server, and that the
proper way to accomplish that is to make sure its key is not in
/etc/guix/acls.

What do you think?

    Thanks,
      Mark

bug#22990: Grafts leads to inefficient substitute info retrieval

[Ludovic Courtès]

Mark H Weaver <mhw@netris.org> skribis:

> ludo@gnu.org (Ludovic Courtès) writes:
>
>> Alex Kost <alezost@gmail.com> skribis:
>>
>>> Ludovic Courtès (2016-03-11 19:52 +0300) wrote:
>>>
>>>> As of right now (v0.9.0-2007-g66a30a3), ‘graft-derivation’ works either by:
>>>>
>>>>   1. Fetching substitute info about the things being built so that it
>>>>      can determine its references, which in turns allows it to determine
>>>>      whether they need to be grafted.
>>>>
>>>>   2. Building stuff, as a last resort, so that it can determine its
>>>>      references.
>>>
>>> I noticed that #1 is happening even with --no-substitutes option.  Is it
>>> intended?
>>
>> Not really, but I see this is because ‘substitutable-path-info’ (called
>> from ‘references/substitutes’, called from ‘graft-derivation’) works
>> regardless of whether substitutes are enabled:
>>
>> scheme@(guile-user)> ,use(guix)
>> scheme@(guile-user)> (define s (open-connection))
>> scheme@(guile-user)> (set-build-options s #:use-substitutes? #f)
>> $2 = #t
>> scheme@(guile-user)> (valid-path? s "/gnu/store/qf2lm7jpiiyygxz8zq0r1ca1fazv6smn-mutt-1.5.24")
>> $3 = #f
>> scheme@(guile-user)> (substitutable-path-info s '("/gnu/store/qf2lm7jpiiyygxz8zq0r1ca1fazv6smn-mutt-1.5.24"))
>> $4 = (#<<substitutable> path: "/gnu/store/qf2lm7jpiiyygxz8zq0r1ca1fazv6smn-mutt-1.5.24" deriver: "/gnu/store/jcl9c3w463xa2g963q5a60rrd97y1g28-mutt-1.5.24.drv" refs: ("/gnu/store/3gmzl5jpk700hqyr8p3kfg0vgcnw8d97-libassuan-2.4.2" "/gnu/store/b02lmk67jq1vcflk2m2bwzc8gmwmndqp-ncurses-6.0" "/gnu/store/d3xdc2w87yw3raafwb9q34gxx4xqci8k-cyrus-sasl-2.1.26" "/gnu/store/pkasxagsa4z4viscfpl6sjszmdmwncl1-gcc-4.9.3-lib" "/gnu/store/qf2lm7jpiiyygxz8zq0r1ca1fazv6smn-mutt-1.5.24" "/gnu/store/qvx4q6lbwi4s3cwr8wqaa7kcva0a5c4b-openssl-1.0.2f" "/gnu/store/sb40mddkia0brc814xkbnhxccfm32q3a-gpgme-1.6.0" "/gnu/store/sgzfawy95pfn7nsw3xvmca58llm5zzbc-glibc-2.22" "/gnu/store/x2p2biyybcb2wac77qz9468asc5fm48i-perl-5.22.1" "/gnu/store/x8dmdlrn5qn0wrbcnngj55y3ab73h0pp-bash-4.3.42" "/gnu/store/zpxg45dq67psrn4wmfk4l635h0si8q63-libgpg-error-1.21") dl-size: 0 nar-size: 6661016>)
>
> Is the information from the substitute server authenticated by checking
> hydra's signature against the list of keys in /etc/guix/acls?

Yes.

>> However, substitutes are not downloaded, so in this regard
>> --no-substitutes is honored.
>
> It depends on the intent of --no-substitutes.  If the intent is to avoid
> trusting the substitute server, then by relying on the accuracy of the
> runtime dependency data from Hydra, we are failing to honor that intent.
>
> That said, I think it's okay to document that --no-substitutes alone is
> not sufficient to avoid trusting a substitute server, and that the
> proper way to accomplish that is to make sure its key is not in
> /etc/guix/acls.

The sysadmin gets to choose which principals are trusted; unprivileged
users can only shrink this set.

The weird thing is that in this case, passing --substitute-urls='' on
the client side would effectively disable substitutes entirely.

> What do you think?

We could augment the doc for --no-substitutes, I guess we should first
document that grafting relies on server-provided info.

Ludo’.

bug#22990: Grafts leads to inefficient substitute info retrieval

[Ludovic Courtès]

ludo@gnu.org (Ludovic Courtès) skribis:

> Concretely, what this means is this:
>
>   $ guix package -u
>   substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 100.0%
>   substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 100.0%
>   substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 100.0%
>   substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 100.0%
>   substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 100.0%
>   substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 100.0%
>   substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 100.0%
>   […]
>   The following files would be downloaded:
>
> Each of the initial “updating list” message corresponds to an HTTP
> request for a single narinfo file, which can take around 1 second.
>
> Instead, the ideal thing would be to fetch the narinfo files for all the
> relevant packages at once; that way, we’d spawn ‘guix substitute’ only
> once, and it would benefit from HTTP pipelining (one round-trip instead
> of N.)
>
> To achieve this, I’m thinking of extending gexp code such that gexp
> compilers can return a list of applicable grafts.  The ‘package’
> compiler would do #:graft? #f and instead let ‘gexp->derivation’ call
> ‘graft-derivation’.

The ‘wip-gexp-grafts’ branch does that.  Namely, it’s possible to know
what grafts would apply to a gexp derivation build, so that one can
first build ungrafted, and then apply the grafts to the results.
So for a profile, we’d first build the profile as is, and only then
would we graft it.

Right now the tip of this branch is a hack such that ‘guix package’:

  1. Builds the original (ungrafted) derivation of the profile;

  2. Manually calls ‘graft-derivation’ on that, passing it the list of
     applicable grafts.

Conceptually it’s what we want to do, but the drawback is that the
caller (here ‘guix package’) goes through a lot of hops to get the list
of grafts and to apply it.

I think we should instead have a way to annotate a derivation with a
list of grafts, as well as a procedure to build that derivations in two
phases (first the original derivation, then the grafts).

To be continued…

Ludo’.

bug#22990: Grafts leads to inefficient substitute info retrieval

[Ludovic Courtès]

ludo@gnu.org (Ludovic Courtès) skribis:

> ludo@gnu.org (Ludovic Courtès) skribis:

[...]

>> To achieve this, I’m thinking of extending gexp code such that gexp
>> compilers can return a list of applicable grafts.  The ‘package’
>> compiler would do #:graft? #f and instead let ‘gexp->derivation’ call
>> ‘graft-derivation’.
>
> The ‘wip-gexp-grafts’ branch does that.  Namely, it’s possible to know
> what grafts would apply to a gexp derivation build, so that one can
> first build ungrafted, and then apply the grafts to the results.
> So for a profile, we’d first build the profile as is, and only then
> would we graft it.
>
> Right now the tip of this branch is a hack such that ‘guix package’:
>
>   1. Builds the original (ungrafted) derivation of the profile;
>
>   2. Manually calls ‘graft-derivation’ on that, passing it the list of
>      applicable grafts.
>
> Conceptually it’s what we want to do, but the drawback is that the
> caller (here ‘guix package’) goes through a lot of hops to get the list
> of grafts and to apply it.
>
> I think we should instead have a way to annotate a derivation with a
> list of grafts, as well as a procedure to build that derivations in two
> phases (first the original derivation, then the grafts).

The current iteration introduces “build continuation”: a derivation can
be annotated with a continuation, and the ‘build-things’ procedure will
loop over continuations and return the final results (that only works
for derivations directly passed as an argument to ‘build-things’.)

Example (where Guile 2.0.12 replaced by 2.0.13):

--8<---------------cut here---------------start------------->8---
$ ./pre-inst-env  guix package -p foo -i guile-json guile-ssh gdb
The following packages will be installed:
   guile-json	0.5.0	/gnu/store/sd8jm2rw7cp3bnrk421kr97ki6sqxnhz-guile-json-0.5.0
   guile-ssh	0.10.2	/gnu/store/w90isin9pnm9ri8w9njxby8h98lfnkzq-guile-ssh-0.10.2
   gdb	7.12	/gnu/store/d2a4lmycc13ssdf47a9h410knlfqqq41-gdb-7.12

applying 6 grafts to /gnu/store/s7w6r9ih52kdzcl7kprf2rq48s69zh98-profile
3 packages in profile
The following environment variable definitions may be needed:
   export PATH="foo/bin${PATH:+:}$PATH"
$ guix gc -R foo | grep guile
/gnu/store/7fisf4frrgsjzmknjbab1dal23wxrp8d-guile-2.0.13
/gnu/store/w90isin9pnm9ri8w9njxby8h98lfnkzq-guile-ssh-0.10.2
/gnu/store/sd8jm2rw7cp3bnrk421kr97ki6sqxnhz-guile-json-0.5.0
--8<---------------cut here---------------end--------------->8---

That seems to be a good model.

Now we must make sure that it works also for ‘guix build foo’ and ‘guix
system build’.

Ludo’.

bug#22990: Grafts leads to inefficient substitute info retrieval

[Ludovic Courtès]

ludo@gnu.org (Ludovic Courtès) skribis:

> ludo@gnu.org (Ludovic Courtès) skribis:
>
>> ludo@gnu.org (Ludovic Courtès) skribis:
>
> [...]
>
>>> To achieve this, I’m thinking of extending gexp code such that gexp
>>> compilers can return a list of applicable grafts.  The ‘package’
>>> compiler would do #:graft? #f and instead let ‘gexp->derivation’ call
>>> ‘graft-derivation’.
>>
>> The ‘wip-gexp-grafts’ branch does that.  Namely, it’s possible to know
>> what grafts would apply to a gexp derivation build, so that one can
>> first build ungrafted, and then apply the grafts to the results.
>> So for a profile, we’d first build the profile as is, and only then
>> would we graft it.
>>
>> Right now the tip of this branch is a hack such that ‘guix package’:
>>
>>   1. Builds the original (ungrafted) derivation of the profile;
>>
>>   2. Manually calls ‘graft-derivation’ on that, passing it the list of
>>      applicable grafts.
>>
>> Conceptually it’s what we want to do, but the drawback is that the
>> caller (here ‘guix package’) goes through a lot of hops to get the list
>> of grafts and to apply it.
>>
>> I think we should instead have a way to annotate a derivation with a
>> list of grafts, as well as a procedure to build that derivations in two
>> phases (first the original derivation, then the grafts).
>
> The current iteration introduces “build continuation”: a derivation can
> be annotated with a continuation, and the ‘build-things’ procedure will
> loop over continuations and return the final results (that only works
> for derivations directly passed as an argument to ‘build-things’.)

On second thought, the whole idea of applying grafts on the final result
(instead of applying grafts at each step like we do now) doesn’t fly.
It works well for things like a profile or the system derivation, but
breaks for less trivial things.

For example, if you’re building a VM image or a binary tarball, you
really need to graft packages early on; trying to graft the VM image or
binary tarball wouldn’t have the desired effect.

I’ve pushed ‘wip-gexp-grafts’ again but it seems to be a dead end.

Ludo’.

bug#22990: Grafts leads to inefficient substitute info retrieval

[David Craven]

> On second thought, the whole idea of applying grafts on the final result
> (instead of applying grafts at each step like we do now) doesn’t fly.
> It works well for things like a profile or the system derivation, but
> breaks for less trivial things.

> For example, if you’re building a VM image or a binary tarball, you
> really need to graft packages early on; trying to graft the VM image or
> binary tarball wouldn’t have the desired effect.

Isn't a system derivation or a profile derivation an intermediate step
to these derivations? Can't there be a flag or something called
#:already-grafted? #t?

bug#22990: Grafts leads to inefficient substitute info retrieval

[Ludovic Courtès]

David Craven <david@craven.ch> skribis:

>> On second thought, the whole idea of applying grafts on the final result
>> (instead of applying grafts at each step like we do now) doesn’t fly.
>> It works well for things like a profile or the system derivation, but
>> breaks for less trivial things.
>
>> For example, if you’re building a VM image or a binary tarball, you
>> really need to graft packages early on; trying to graft the VM image or
>> binary tarball wouldn’t have the desired effect.
>
> Isn't a system derivation or a profile derivation an intermediate step
> to these derivations?

Yes, you’re right.  However the implementation I had come up with relied
on “build continuations”, which only worked for the “top-level”
derivations (those you pass to ‘build-derivations’.)

> Can't there be a flag or something called #:already-grafted? #t?

Yeah, we need something like that.  I need to chew a bit more on this to
find a nice way to achieve that.

Thanks for your feedback, I feel less lonely now.  :-)

Ludo’.

bug#22990: Grafts leads to inefficient substitute info retrieval

[Ludovic Courtès]

Hi younger self!

ludo@gnu.org (Ludovic Courtès) skribis:

> As of right now (v0.9.0-2007-g66a30a3), ‘graft-derivation’ works either by:
>
>   1. Fetching substitute info about the things being built so that it
>      can determine its references, which in turns allows it to determine
>      whether they need to be grafted.
>
>   2. Building stuff, as a last resort, so that it can determine its
>      references.
>
> Case #1 is hopefully going to be the most common.
>
> The problem with #1 is that when building a profile, we do one
> ‘package-derivation’ call for each package in the profile, which
> translates in one ‘graft-derivation’ call for each relevant package¹,
> which translates into one ‘references/substitutes’ call for each.
>
> Concretely, what this means is this:
>
>   $ guix package -u
>   substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 100.0%
>   substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 100.0%
>   substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 100.0%
>   substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 100.0%
>   substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 100.0%
>   substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 100.0%
>   substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 100.0%
>   […]
>   The following files would be downloaded:
>
> Each of the initial “updating list” message corresponds to an HTTP
> request for a single narinfo file, which can take around 1 second.

This is finally fixed with commit
710854304b1ab29332edcb76f3de532e0724c197, as discussed earlier this
week¹.  \o/

The solution is simpler and more orthogonal than what was envisioned
earlier in this thread, which is nice.

I also potentially addresses other uses of “dynamic dependencies”
elsewhere in the code base, such as ‘remote-eval’ calls in ‘guix
deploy’.

Feedback welcome!

Ludo’.

¹ https://lists.gnu.org/archive/html/guix-devel/2020-03/msg00337.html