* bug#51472: substitute servers should be preferred according to their coverage rate
@ 2021-10-29 2:07 Maxim Cournoyer
2021-11-07 15:11 ` Ludovic Courtès
0 siblings, 1 reply; 3+ messages in thread
From: Maxim Cournoyer @ 2021-10-29 2:07 UTC (permalink / raw)
To: 51472
Hello,
When using substitute servers discovery, I've noticed that if one of the
substitute servers doesn't have any substitutes available, it'll keep
getting tried instead of others, leading to a slide-show of substitutes
updates such as:
--8<---------------cut here---------------start------------->8---
normalized load on machine '127.0.0.1' is 0.04
building /gnu/store/ajd0hx104702jpz2ycdwgrnyrv8jsp6d-xorg-server-21.1.0.tar.xz.drv...
process 9195 acquired build slot '/var/guix/offload/127.0.0.1:6666/1'
normalized load on machine '127.0.0.1' is 0.04
building /gnu/store/49rqi3wpvdm5pv6in9pamzdvg0wscrl8-xorgproto-2021.5.drv...
substitute: updating substitutes from 'http://192.168.10.102:80'... 0.0%
substitute: updating substitutes from 'http://192.168.10.102:80'... 0.0%
substitute: updating substitutes from 'http://192.168.10.102:80'... 0.0%
substitute: updating substitutes from 'http://192.168.10.102:80'... 0.0%
substitute: updating substitutes from 'http://192.168.10.102:80'... 0.0%
substitute: updating substitutes from 'http://192.168.10.102:80'... 0.0%
substitute: updating substitutes from 'http://192.168.10.102:80'... 0.0%
substitute: updating substitutes from 'http://192.168.10.102:80'... 0.0%
substitute: updating substitutes from 'http://192.168.10.102:80'... 0.0%
substitute: updating substitutes from 'http://192.168.10.102:80'... 0.0%
substitute: updating substitutes from 'http://192.168.10.102:80'... 0.0%
substitute: updating substitutes from 'http://192.168.10.102:80'... 0.0%
substitute: updating substitutes from 'http://192.168.10.102:80'... 0.0%
substitute: updating substitutes from 'http://192.168.10.102:80'... 0.0%
substitute: updating substitutes from 'http://192.168.10.102:80'... 0.0%
substitute: updating substitutes from 'http://192.168.10.102:80'... 0.0%
substitute: updating substitutes from 'http://192.168.10.102:80'... 0.0%
substitute: updating substitutes from 'http://192.168.10.102:80'... 0.0%
substitute: updating substitutes from 'http://192.168.10.102:80'... 0.0%
substitute: updating substitutes from 'http://192.168.10.102:80'... 0.0%
substitute: updating substitutes from 'http://192.168.10.102:80'... 0.0%
substitute: updating substitutes from 'http://127.0.0.1:8080'... 100.0%
substitute: updating substitutes from 'http://192.168.10.102:80'... 0.0%
substitute: updating substitutes from 'http://192.168.10.102:80'... 0.0%
substitute: updating substitutes from 'http://192.168.10.102:80'... 0.0%
substitute: updating substitutes from 'http://192.168.10.102:80'... 0.0%
substitute: updating substitutes from 'http://192.168.10.102:80'... 0.0%
substitute: updating substitutes from 'http://192.168.10.102:80'... 0.0%
substitute: updating substitutes from 'http://192.168.10.102:80'... 0.0%
substitute: updating substitutes from 'http://192.168.10.102:80'... 0.0%
substitute: updating substitutes from 'http://127.0.0.1:8080'... 100.0%
substitute: updating substitutes from 'http://192.168.10.102:80'... 0.0%
--8<---------------cut here---------------end--------------->8---
We should implement some scheme to prefer querying high-substitute
servers first, instead of wasting time querying servers always failed
queries; this would greatly improve performance when using substitute
discovery for example combined with low coverage.
Thanks!
Maxim
^ permalink raw reply [flat|nested] 3+ messages in thread
* bug#51472: substitute servers should be preferred according to their coverage rate
2021-10-29 2:07 bug#51472: substitute servers should be preferred according to their coverage rate Maxim Cournoyer
@ 2021-11-07 15:11 ` Ludovic Courtès
2022-01-11 3:43 ` Maxim Cournoyer
0 siblings, 1 reply; 3+ messages in thread
From: Ludovic Courtès @ 2021-11-07 15:11 UTC (permalink / raw)
To: Maxim Cournoyer; +Cc: 51472
Hi,
Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:
> When using substitute servers discovery, I've noticed that if one of the
> substitute servers doesn't have any substitutes available, it'll keep
> getting tried instead of others, leading to a slide-show of substitutes
> updates such as:
>
> normalized load on machine '127.0.0.1' is 0.04
> building /gnu/store/ajd0hx104702jpz2ycdwgrnyrv8jsp6d-xorg-server-21.1.0.tar.xz.drv...
> process 9195 acquired build slot '/var/guix/offload/127.0.0.1:6666/1'
> normalized load on machine '127.0.0.1' is 0.04
> building /gnu/store/49rqi3wpvdm5pv6in9pamzdvg0wscrl8-xorgproto-2021.5.drv...
> substitute: updating substitutes from 'http://192.168.10.102:80'... 0.0%
> substitute: updating substitutes from 'http://192.168.10.102:80'... 0.0%
> substitute: updating substitutes from 'http://192.168.10.102:80'... 0.0%
> substitute: updating substitutes from 'http://192.168.10.102:80'... 0.0%
> substitute: updating substitutes from 'http://192.168.10.102:80'... 0.0%
We’d need to check why this particular server is checked repeatedly.
The fact that it displays “0.0%” doesn’t mean that the server lacks
substitutes, but that it does not reply to ‘GET /xyz.narinfo’ requests,
for example because it’s off-line (see
<https://issues.guix.gnu.org/48808>.)
> We should implement some scheme to prefer querying high-substitute
> servers first, instead of wasting time querying servers always failed
> queries; this would greatly improve performance when using substitute
> discovery for example combined with low coverage.
There are several problems with that. First one is that you can’t tell
what substitute coverage is until you’ve actually made those GET
requests. Second one is that substitute coverage varies and it’s not an
absolute measure; for example, if a server provides substitutes for only
0.1% of all the packages, but that’s precisely the 0.1% you care about,
it’s more valuable than the one that has 99% of the packages but lacks
those you want.
There are other issues such as the fact that current semantics is to
respect the order of substitute URLs, which is presumably chosen by the
user according to their own criteria: download speed, bandwidth usage,
etc.
I hope this makes sense!
Ludo’.
^ permalink raw reply [flat|nested] 3+ messages in thread
* bug#51472: substitute servers should be preferred according to their coverage rate
2021-11-07 15:11 ` Ludovic Courtès
@ 2022-01-11 3:43 ` Maxim Cournoyer
0 siblings, 0 replies; 3+ messages in thread
From: Maxim Cournoyer @ 2022-01-11 3:43 UTC (permalink / raw)
To: Ludovic Courtès; +Cc: 51472, GNU Debbugs
merge 48808 51472
thanks
Hello Ludovic,
Ludovic Courtès <ludo@gnu.org> writes:
> Hi,
>
> Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:
>
>> When using substitute servers discovery, I've noticed that if one of the
>> substitute servers doesn't have any substitutes available, it'll keep
>> getting tried instead of others, leading to a slide-show of substitutes
>> updates such as:
>>
>> normalized load on machine '127.0.0.1' is 0.04
>> building /gnu/store/ajd0hx104702jpz2ycdwgrnyrv8jsp6d-xorg-server-21.1.0.tar.xz.drv...
>> process 9195 acquired build slot '/var/guix/offload/127.0.0.1:6666/1'
>> normalized load on machine '127.0.0.1' is 0.04
>> building /gnu/store/49rqi3wpvdm5pv6in9pamzdvg0wscrl8-xorgproto-2021.5.drv...
>> substitute: updating substitutes from 'http://192.168.10.102:80'... 0.0%
>> substitute: updating substitutes from 'http://192.168.10.102:80'... 0.0%
>> substitute: updating substitutes from 'http://192.168.10.102:80'... 0.0%
>> substitute: updating substitutes from 'http://192.168.10.102:80'... 0.0%
>> substitute: updating substitutes from 'http://192.168.10.102:80'... 0.0%
>
> We’d need to check why this particular server is checked repeatedly.
> The fact that it displays “0.0%” doesn’t mean that the server lacks
> substitutes, but that it does not reply to ‘GET /xyz.narinfo’ requests,
> for example because it’s off-line (see
> <https://issues.guix.gnu.org/48808>.)
>
>> We should implement some scheme to prefer querying high-substitute
>> servers first, instead of wasting time querying servers always failed
>> queries; this would greatly improve performance when using substitute
>> discovery for example combined with low coverage.
>
> There are several problems with that. First one is that you can’t tell
> what substitute coverage is until you’ve actually made those GET
> requests. Second one is that substitute coverage varies and it’s not an
> absolute measure; for example, if a server provides substitutes for only
> 0.1% of all the packages, but that’s precisely the 0.1% you care about,
> it’s more valuable than the one that has 99% of the packages but lacks
> those you want.
>
> There are other issues such as the fact that current semantics is to
> respect the order of substitute URLs, which is presumably chosen by the
> user according to their own criteria: download speed, bandwidth usage,
> etc.
>
> I hope this makes sense!
It does! I agree that it'd be tricky to get this right; makes me
realize that my problem is probably due to #48808, and fixing that one
would probably have avoided that bug report :-).
I'm merging this one with 48808.
Thank you!
Maxim
^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2022-01-11 3:44 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-10-29 2:07 bug#51472: substitute servers should be preferred according to their coverage rate Maxim Cournoyer
2021-11-07 15:11 ` Ludovic Courtès
2022-01-11 3:43 ` Maxim Cournoyer
Code repositories for project(s) associated with this public inbox
https://git.savannah.gnu.org/cgit/guix.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).