Test US mirror for bordeaux.guix.gnu.org and slow downloading of substitutes

Test US mirror for bordeaux.guix.gnu.org and slow downloading of substitutes

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 3159 bytes --]

Hey!

So the nar-herder came in to existence at the end of last year (2021)
and while the main use at the time was addressing the lack of storage on
bayfront, I also hoped to improve the situation regarding mirrors for
substitutes.

I'm not in a great situation to test this though, as my usual internet
connection is slow enough that a closer mirror probably won't make much
difference.

So, one thing that I'd be interested in, is hearing from anyone who
thinks they get worse download performance from bordeaux.guix.gnu.org or
ci.guix.gnu.org than they get when downloading other
things. Importantly, it would be good to know roughly where
(geographically) the machine doing the downloading is, and some data to
show the difference.

For example, I'm in the United Kingdom in Europe, and this is the output
from wget downloading a ~200M file from bordeaux.guix.gnu.org:

  → wget https://bordeaux.guix.gnu.org/nar/lzip/078vr3r8mn3yrwzwxw64hmcyshic9p3q-stellarium-0.21.0
  --2022-05-20 16:49:56--  https://bordeaux.guix.gnu.org/nar/lzip/078vr3r8mn3yrwzwxw64hmcyshic9p3q-stellarium-0.21.0
  Resolving bordeaux.guix.gnu.org (bordeaux.guix.gnu.org)... 2a0c:e300::58, 185.233.100.56
  Connecting to bordeaux.guix.gnu.org (bordeaux.guix.gnu.org)|2a0c:e300::58|:443... connected.
  HTTP request sent, awaiting response... 200 OK
  Length: 208615205 (199M) [text/plain]
  Saving to: ‘078vr3r8mn3yrwzwxw64hmcyshic9p3q-stellarium-0.21.0.6’
  
  078vr3r8mn3yrwzwxw64 100%[======================>] 198.95M  4.24MB/s    in 46s     
  
  2022-05-20 16:50:43 (4.31 MB/s) - ‘078vr3r8mn3yrwzwxw64hmcyshic9p3q-stellarium-0.21.0.6’ saved [208615205/208615205]


Also, I've setup a US based (Hetzner data center, east coast) mirror of
bordeaux.guix.gnu.org:

  https://bordeaux-us-east-mirror.cbaines.net/

So, I'd also be interested in seeing how that performs for people, and
how it compares against bordeaux.guix.gnu.org, which is hosted in France
in Europe.

Here's my output from wget:

  → wget https://bordeaux-us-east-mirror.cbaines.net/nar/lzip/078vr3r8mn3yrwzwxw64hmcyshic9p3q-stellarium-0.21.0
  --2022-05-20 16:50:44--  https://bordeaux-us-east-mirror.cbaines.net/nar/lzip/078vr3r8mn3yrwzwxw64hmcyshic9p3q-stellarium-0.21.0
  Resolving bordeaux-us-east-mirror.cbaines.net (bordeaux-us-east-mirror.cbaines.net)... 5.161.49.48
  Connecting to bordeaux-us-east-mirror.cbaines.net (bordeaux-us-east-mirror.cbaines.net)|5.161.49.48|:443... connected.
  HTTP request sent, awaiting response... 200 OK
  Length: 208615205 (199M) [text/plain]
  Saving to: ‘078vr3r8mn3yrwzwxw64hmcyshic9p3q-stellarium-0.21.0.7’
  
  078vr3r8mn3yrwzwxw64 100%[======================>] 198.95M  4.17MB/s    in 47s     
  
  2022-05-20 16:51:32 (4.22 MB/s) - ‘078vr3r8mn3yrwzwxw64hmcyshic9p3q-stellarium-0.21.0.7’ saved [208615205/208615205]


Now, performance is probably a bit more complicated than just download
speed. The US mirror holds the narinfo information locally, which should
enable it to respond to requests more quickly, reducing latency, but
this is a little harder to test.

Thanks,

Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

Re: Test US mirror for bordeaux.guix.gnu.org and slow downloading of substitutes

[Blake Shaw]

Hi Christopher,

Here in Singapore download speed has gone from bad to worse over the
last two weeks, so bad that I was having to --fallback often and wound
up setting up a cuirass instance to continuously build a collection of
manifests I'm working with (which solved the issue). Not at my machine
atm but will wget stellarium and send the results when I am.

Best,
Blake

---
Blake Shaw
Director, SWEATSHOPPE
sweatshoppe.org
---

On Fri, May 20, 2022 at 11:09 PM Christopher Baines <mail@cbaines.net> wrote:
>
> Hey!
>
> So the nar-herder came in to existence at the end of last year (2021)
> and while the main use at the time was addressing the lack of storage on
> bayfront, I also hoped to improve the situation regarding mirrors for
> substitutes.
>
> I'm not in a great situation to test this though, as my usual internet
> connection is slow enough that a closer mirror probably won't make much
> difference.
>
> So, one thing that I'd be interested in, is hearing from anyone who
> thinks they get worse download performance from bordeaux.guix.gnu.org or
> ci.guix.gnu.org than they get when downloading other
> things. Importantly, it would be good to know roughly where
> (geographically) the machine doing the downloading is, and some data to
> show the difference.
>
> For example, I'm in the United Kingdom in Europe, and this is the output
> from wget downloading a ~200M file from bordeaux.guix.gnu.org:
>
>   → wget https://bordeaux.guix.gnu.org/nar/lzip/078vr3r8mn3yrwzwxw64hmcyshic9p3q-stellarium-0.21.0
>   --2022-05-20 16:49:56--  https://bordeaux.guix.gnu.org/nar/lzip/078vr3r8mn3yrwzwxw64hmcyshic9p3q-stellarium-0.21.0
>   Resolving bordeaux.guix.gnu.org (bordeaux.guix.gnu.org)... 2a0c:e300::58, 185.233.100.56
>   Connecting to bordeaux.guix.gnu.org (bordeaux.guix.gnu.org)|2a0c:e300::58|:443... connected.
>   HTTP request sent, awaiting response... 200 OK
>   Length: 208615205 (199M) [text/plain]
>   Saving to: ‘078vr3r8mn3yrwzwxw64hmcyshic9p3q-stellarium-0.21.0.6’
>
>   078vr3r8mn3yrwzwxw64 100%[======================>] 198.95M  4.24MB/s    in 46s
>
>   2022-05-20 16:50:43 (4.31 MB/s) - ‘078vr3r8mn3yrwzwxw64hmcyshic9p3q-stellarium-0.21.0.6’ saved [208615205/208615205]
>
>
> Also, I've setup a US based (Hetzner data center, east coast) mirror of
> bordeaux.guix.gnu.org:
>
>   https://bordeaux-us-east-mirror.cbaines.net/
>
> So, I'd also be interested in seeing how that performs for people, and
> how it compares against bordeaux.guix.gnu.org, which is hosted in France
> in Europe.
>
> Here's my output from wget:
>
>   → wget https://bordeaux-us-east-mirror.cbaines.net/nar/lzip/078vr3r8mn3yrwzwxw64hmcyshic9p3q-stellarium-0.21.0
>   --2022-05-20 16:50:44--  https://bordeaux-us-east-mirror.cbaines.net/nar/lzip/078vr3r8mn3yrwzwxw64hmcyshic9p3q-stellarium-0.21.0
>   Resolving bordeaux-us-east-mirror.cbaines.net (bordeaux-us-east-mirror.cbaines.net)... 5.161.49.48
>   Connecting to bordeaux-us-east-mirror.cbaines.net (bordeaux-us-east-mirror.cbaines.net)|5.161.49.48|:443... connected.
>   HTTP request sent, awaiting response... 200 OK
>   Length: 208615205 (199M) [text/plain]
>   Saving to: ‘078vr3r8mn3yrwzwxw64hmcyshic9p3q-stellarium-0.21.0.7’
>
>   078vr3r8mn3yrwzwxw64 100%[======================>] 198.95M  4.17MB/s    in 47s
>
>   2022-05-20 16:51:32 (4.22 MB/s) - ‘078vr3r8mn3yrwzwxw64hmcyshic9p3q-stellarium-0.21.0.7’ saved [208615205/208615205]
>
>
> Now, performance is probably a bit more complicated than just download
> speed. The US mirror holds the narinfo information locally, which should
> enable it to respond to requests more quickly, reducing latency, but
> this is a little harder to test.
>
> Thanks,
>
> Chris

Re: Test US mirror for bordeaux.guix.gnu.org and slow downloading of substitutes

[Blake Shaw]

Hi Christopher,

So its strange, I do:

wget https://bordeaux.guix.gnu.org/nar/lzip/078vr3r8mn3yrwzwxw64hmcyshic9p3q-stellarium-0.21.0
--2022-05-25 05:32:12--
https://bordeaux.guix.gnu.org/nar/lzip/078vr3r8mn3yrwzwxw64hmcyshic9p3q-stellarium-0.21.0
Resolving bordeaux.guix.gnu.org (bordeaux.guix.gnu.org)...
185.233.100.56, 2a0c:e300::58
Connecting to bordeaux.guix.gnu.org
(bordeaux.guix.gnu.org)|185.233.100.56|:443... connected.

...and then it just remains stalled there. wget appears to work fine
in generat though, just not with guix.

Any idea why this might be happening? Could bordeaux be blocking IPs
in Singapore?

Best,
Blake

---
Blake Shaw
Director, SWEATSHOPPE
sweatshoppe.org
---

On Tue, May 24, 2022 at 5:52 AM Blake Shaw <blake@sweatshoppe.org> wrote:
>
> Hi Christopher,
>
> Here in Singapore download speed has gone from bad to worse over the
> last two weeks, so bad that I was having to --fallback often and wound
> up setting up a cuirass instance to continuously build a collection of
> manifests I'm working with (which solved the issue). Not at my machine
> atm but will wget stellarium and send the results when I am.
>
> Best,
> Blake
>
> ---
> Blake Shaw
> Director, SWEATSHOPPE
> sweatshoppe.org
> ---
>
> On Fri, May 20, 2022 at 11:09 PM Christopher Baines <mail@cbaines.net> wrote:
> >
> > Hey!
> >
> > So the nar-herder came in to existence at the end of last year (2021)
> > and while the main use at the time was addressing the lack of storage on
> > bayfront, I also hoped to improve the situation regarding mirrors for
> > substitutes.
> >
> > I'm not in a great situation to test this though, as my usual internet
> > connection is slow enough that a closer mirror probably won't make much
> > difference.
> >
> > So, one thing that I'd be interested in, is hearing from anyone who
> > thinks they get worse download performance from bordeaux.guix.gnu.org or
> > ci.guix.gnu.org than they get when downloading other
> > things. Importantly, it would be good to know roughly where
> > (geographically) the machine doing the downloading is, and some data to
> > show the difference.
> >
> > For example, I'm in the United Kingdom in Europe, and this is the output
> > from wget downloading a ~200M file from bordeaux.guix.gnu.org:
> >
> >   → wget https://bordeaux.guix.gnu.org/nar/lzip/078vr3r8mn3yrwzwxw64hmcyshic9p3q-stellarium-0.21.0
> >   --2022-05-20 16:49:56--  https://bordeaux.guix.gnu.org/nar/lzip/078vr3r8mn3yrwzwxw64hmcyshic9p3q-stellarium-0.21.0
> >   Resolving bordeaux.guix.gnu.org (bordeaux.guix.gnu.org)... 2a0c:e300::58, 185.233.100.56
> >   Connecting to bordeaux.guix.gnu.org (bordeaux.guix.gnu.org)|2a0c:e300::58|:443... connected.
> >   HTTP request sent, awaiting response... 200 OK
> >   Length: 208615205 (199M) [text/plain]
> >   Saving to: ‘078vr3r8mn3yrwzwxw64hmcyshic9p3q-stellarium-0.21.0.6’
> >
> >   078vr3r8mn3yrwzwxw64 100%[======================>] 198.95M  4.24MB/s    in 46s
> >
> >   2022-05-20 16:50:43 (4.31 MB/s) - ‘078vr3r8mn3yrwzwxw64hmcyshic9p3q-stellarium-0.21.0.6’ saved [208615205/208615205]
> >
> >
> > Also, I've setup a US based (Hetzner data center, east coast) mirror of
> > bordeaux.guix.gnu.org:
> >
> >   https://bordeaux-us-east-mirror.cbaines.net/
> >
> > So, I'd also be interested in seeing how that performs for people, and
> > how it compares against bordeaux.guix.gnu.org, which is hosted in France
> > in Europe.
> >
> > Here's my output from wget:
> >
> >   → wget https://bordeaux-us-east-mirror.cbaines.net/nar/lzip/078vr3r8mn3yrwzwxw64hmcyshic9p3q-stellarium-0.21.0
> >   --2022-05-20 16:50:44--  https://bordeaux-us-east-mirror.cbaines.net/nar/lzip/078vr3r8mn3yrwzwxw64hmcyshic9p3q-stellarium-0.21.0
> >   Resolving bordeaux-us-east-mirror.cbaines.net (bordeaux-us-east-mirror.cbaines.net)... 5.161.49.48
> >   Connecting to bordeaux-us-east-mirror.cbaines.net (bordeaux-us-east-mirror.cbaines.net)|5.161.49.48|:443... connected.
> >   HTTP request sent, awaiting response... 200 OK
> >   Length: 208615205 (199M) [text/plain]
> >   Saving to: ‘078vr3r8mn3yrwzwxw64hmcyshic9p3q-stellarium-0.21.0.7’
> >
> >   078vr3r8mn3yrwzwxw64 100%[======================>] 198.95M  4.17MB/s    in 47s
> >
> >   2022-05-20 16:51:32 (4.22 MB/s) - ‘078vr3r8mn3yrwzwxw64hmcyshic9p3q-stellarium-0.21.0.7’ saved [208615205/208615205]
> >
> >
> > Now, performance is probably a bit more complicated than just download
> > speed. The US mirror holds the narinfo information locally, which should
> > enable it to respond to requests more quickly, reducing latency, but
> > this is a little harder to test.
> >
> > Thanks,
> >
> > Chris

Re: Test US mirror for bordeaux.guix.gnu.org and slow downloading of substitutes

[Felix Lechner]

Hi Guix,

On Tue, May 24, 2022 at 3:40 PM Blake Shaw <blake@sweatshoppe.org> wrote:
>
> wget https://bordeaux.guix.gnu.org/nar/lzip/078vr3r8mn3yrwzwxw64hmcyshic9p3q-stellarium-0.21.0
>
> Could bordeaux be blocking IPs in Singapore?

That command also stalls in Fremont, Calif., in the continental US
(run on Guix).

Kind regards
Felix Lechner

* * *

$ wget https://bordeaux.guix.gnu.org/nar/lzip/078vr3r8mn3yrwzwxw64hmcyshic9p3q-stellarium-0.21.0
--2022-05-24 15:41:57--
https://bordeaux.guix.gnu.org/nar/lzip/078vr3r8mn3yrwzwxw64hmcyshic9p3q-stellarium-0.21.0
Resolving bordeaux.guix.gnu.org (bordeaux.guix.gnu.org)...
2a0c:e300::58, 185.233.100.56 Connecting to bordeaux.guix.gnu.org
(bordeaux.guix.gnu.org)|2a0c:e300::58|:443... connected.

$ wget -4 https://bordeaux.guix.gnu.org/nar/lzip/078vr3r8mn3yrwzwxw64hmcyshic9p3q-stellarium-0.21.0
--2022-05-24 15:44:32--
https://bordeaux.guix.gnu.org/nar/lzip/078vr3r8mn3yrwzwxw64hmcyshic9p3q-stellarium-0.21.0
Resolving bordeaux.guix.gnu.org (bordeaux.guix.gnu.org)...
185.233.100.56 Connecting to bordeaux.guix.gnu.org
(bordeaux.guix.gnu.org)|185.233.100.56|:443... connected.

Re: Test US mirror for bordeaux.guix.gnu.org and slow downloading of substitutes

[Tobias Geerinckx-Rice]

[-- Attachment #1: Type: text/plain, Size: 463 bytes --]

Hi Blake,

Blake Shaw 写道：
> Any idea why this might be happening? Could bordeaux be blocking 
> IPs
> in Singapore?

This is *completely* unrelated.  bordeaux (bayfront) is currently 
running out of RAM and has been doing so, quite heroically, for 
more than an hour now.  Because of that it's currently refusing 
connections[0].

Now is not the time to test bordeaux.  Sorry.

Kind regards,

T G-R

[0]: Including SSH, so no easy fix…

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 247 bytes --]

Re: Test US mirror for bordeaux.guix.gnu.org and slow downloading of substitutes

[Ludovic Courtès]

Hi,

Christopher Baines <mail@cbaines.net> skribis:

> Now, performance is probably a bit more complicated than just download
> speed. The US mirror holds the narinfo information locally, which should
> enable it to respond to requests more quickly, reducing latency, but
> this is a little harder to test.

For those of us who have a fast connection to the server, download time
is CPU-bound; in that case, offering zstd substitutes can make a real
difference:

  https://guix.gnu.org/en/blog/2021/getting-bytes-to-disk-more-quickly/

Ludo’.

Re: Test US mirror for bordeaux.guix.gnu.org and slow downloading of substitutes

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 3106 bytes --]


Christopher Baines <mail@cbaines.net> writes:

> So, one thing that I'd be interested in, is hearing from anyone who
> thinks they get worse download performance from bordeaux.guix.gnu.org or
> ci.guix.gnu.org than they get when downloading other
> things. Importantly, it would be good to know roughly where
> (geographically) the machine doing the downloading is, and some data to
> show the difference.
>
> For example, I'm in the United Kingdom in Europe, and this is the output
> from wget downloading a ~200M file from bordeaux.guix.gnu.org:
>
>   → wget https://bordeaux.guix.gnu.org/nar/lzip/078vr3r8mn3yrwzwxw64hmcyshic9p3q-stellarium-0.21.0
>   --2022-05-20 16:49:56--  https://bordeaux.guix.gnu.org/nar/lzip/078vr3r8mn3yrwzwxw64hmcyshic9p3q-stellarium-0.21.0
>   Resolving bordeaux.guix.gnu.org (bordeaux.guix.gnu.org)... 2a0c:e300::58, 185.233.100.56
>   Connecting to bordeaux.guix.gnu.org (bordeaux.guix.gnu.org)|2a0c:e300::58|:443... connected.
>   HTTP request sent, awaiting response... 200 OK
>   Length: 208615205 (199M) [text/plain]
>   Saving to: ‘078vr3r8mn3yrwzwxw64hmcyshic9p3q-stellarium-0.21.0.6’
>
>   078vr3r8mn3yrwzwxw64 100%[======================>] 198.95M  4.24MB/s    in 46s
>
>   2022-05-20 16:50:43 (4.31 MB/s) - ‘078vr3r8mn3yrwzwxw64hmcyshic9p3q-stellarium-0.21.0.6’ saved [208615205/208615205]
>
>
> Also, I've setup a US based (Hetzner data center, east coast) mirror of
> bordeaux.guix.gnu.org:
>
>   https://bordeaux-us-east-mirror.cbaines.net/
>
> So, I'd also be interested in seeing how that performs for people, and
> how it compares against bordeaux.guix.gnu.org, which is hosted in France
> in Europe.
>
> Here's my output from wget:
>
>   → wget https://bordeaux-us-east-mirror.cbaines.net/nar/lzip/078vr3r8mn3yrwzwxw64hmcyshic9p3q-stellarium-0.21.0
>   --2022-05-20 16:50:44--  https://bordeaux-us-east-mirror.cbaines.net/nar/lzip/078vr3r8mn3yrwzwxw64hmcyshic9p3q-stellarium-0.21.0
>   Resolving bordeaux-us-east-mirror.cbaines.net (bordeaux-us-east-mirror.cbaines.net)... 5.161.49.48
>   Connecting to bordeaux-us-east-mirror.cbaines.net (bordeaux-us-east-mirror.cbaines.net)|5.161.49.48|:443... connected.
>   HTTP request sent, awaiting response... 200 OK
>   Length: 208615205 (199M) [text/plain]
>   Saving to: ‘078vr3r8mn3yrwzwxw64hmcyshic9p3q-stellarium-0.21.0.7’
>
>   078vr3r8mn3yrwzwxw64 100%[======================>] 198.95M  4.17MB/s    in 47s
>
>   2022-05-20 16:51:32 (4.22 MB/s) - ‘078vr3r8mn3yrwzwxw64hmcyshic9p3q-stellarium-0.21.0.7’ saved [208615205/208615205]

Hey,

I'm still interested in hearing about how the mirrors perform for
different people.

I've now setup a test mirror in Singapore, in addition to the one in the
US. Plus, there's bishan, which is hosted in Germany.

So, here are the 4 domains which serve bordeaux.guix.gnu.org nars by
geographical location:

  France:     bordeaux.guix.gnu.org
  Germany:    bishan.guix.gnu.org
  US:         bordeaux-us-east-mirror.cbaines.net
  Singapore:  bordeaux-singapore-mirror.cbaines.net

Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]