bug#46942: ci.guix.gnu.org is slow from my system

bug#46942: ci.guix.gnu.org is slow from my system

[raid5atemyhomework via Bug reports for GNU Guix]

Downloading substitutes from ci.guix.gnu.org is slow from my two Guix-using computers.

One is a pure Guix System install without any channels, the other one is a foreign Guix install with the-channel-that-cannot-be-named.


```
downloading from https://ci.guix.gnu.org/nar/lzip/1bdldr80p39g1mjnh76xw6hmwqrrb8lz-wine64-6.0 ...
 wine64-6.0  54.4MiB                                                                                                                                 10KiB/s 01:55 [                  ]   2.1%
```

This can be very slow, including as slow as 4KiB/s at times. This is fairly awful since sometimes I get this result:

```
downloading from https://ci.guix.gnu.org/nar/lzip/1bdldr80p39g1mjnh76xw6hmwqrrb8lz-wine64-6.0 ...
 wine64-6.0  54.4MiB                                                                                                                                 49KiB/s 06:01 [#####             ]  31.9%guix substitute: error: TLS error in procedure 'read_from_session_record_port': Error decoding the received TLS packet.
substitution of /gnu/store/1bdldr80p39g1mjnh76xw6hmwqrrb8lz-wine64-6.0 failed
guix package: error: some substitutes for the outputs of derivation `/gnu/store/wr9kf2bgcsvwxcmhnl9lf047nr8xcklc-wine64-6.0.drv' failed (usually happens due to networking issues); try `--fallback' to build derivation from source
```

Because of the failed incomplete download, I have to ***restart the download again from 0%***, which is awful, awful UX. It would be really nice if:

1. The guix download process would make an effort to ***retry*** this a few times.
2. Continue the previous download instead of restarting from 0%.

The problem is not on my ISP, or at least not solely on my ISP. Doing a `wget` from `github.com`:

```
$ wget https://github.com/zfsonlinux/zfs/releases/download/zfs-2.0.3/zfs-2.0.3.tar.gz
--2021-03-05 17:39:46--  https://github.com/zfsonlinux/zfs/releases/download/zfs-2.0.3/zfs-2.0.3.tar.gz
Resolving github.com (github.com)... 13.229.188.59
Connecting to github.com (github.com)|13.229.188.59|:443... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: https://github.com/openzfs/zfs/releases/download/zfs-2.0.3/zfs-2.0.3.tar.gz [following]
--2021-03-05 17:39:47--  https://github.com/openzfs/zfs/releases/download/zfs-2.0.3/zfs-2.0.3.tar.gz
Reusing existing connection to github.com:443.
HTTP request sent, awaiting response... 302 Found
Location: https://github-releases.githubusercontent.com/437011/71526a80-6ba3-11eb-893f-4ceb55b479d6?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20210305%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20210305T093947Z&X-Amz-Expires=300&X-Amz-Signature=9454446348219aa0731af009a5232bf5772e3d45a3a34052fa61f64f04c3c979&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=437011&response-content-disposition=attachment%3B%20filename%3Dzfs-2.0.3.tar.gz&response-content-type=application%2Foctet-stream [following]
--2021-03-05 17:39:47--  https://github-releases.githubusercontent.com/437011/71526a80-6ba3-11eb-893f-4ceb55b479d6?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20210305%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20210305T093947Z&X-Amz-Expires=300&X-Amz-Signature=9454446348219aa0731af009a5232bf5772e3d45a3a34052fa61f64f04c3c979&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=437011&response-content-disposition=attachment%3B%20filename%3Dzfs-2.0.3.tar.gz&response-content-type=application%2Foctet-stream
Resolving github-releases.githubusercontent.com (github-releases.githubusercontent.com)... 185.199.108.154, 185.199.109.154, 185.199.110.154, ...
Connecting to github-releases.githubusercontent.com (github-releases.githubusercontent.com)|185.199.108.154|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 13114404 (13M) [application/octet-stream]
Saving to: ‘zfs-2.0.3.tar.gz’

zfs-2.0.3.tar.gz                                100%[=====================================================================================================>]  12.51M  9.34MB/s    in 1.3s

2021-03-05 17:39:49 (9.34 MB/s) - ‘zfs-2.0.3.tar.gz’ saved [13114404/13114404]

```

As can be seen above, I get 9.34MB/s elsewhere, which is better than the >60Mbit/s promised by my ISP.

It's possible that the problem is between my ISP and ci.guix.gnu.org. If I `wget` directly:

```
$ wget https://ci.guix.gnu.org/nar/lzip/1bdldr80p39g1mjnh76xw6hmwqrrb8lz-wine64-6.0
--2021-03-05 18:56:21--  https://ci.guix.gnu.org/nar/lzip/1bdldr80p39g1mjnh76xw6hmwqrrb8lz-wine64-6.0
Resolving ci.guix.gnu.org (ci.guix.gnu.org)... 141.80.181.40
Connecting to ci.guix.gnu.org (ci.guix.gnu.org)|141.80.181.40|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 57048561 (54M) [application/octet-stream]
Saving to: ‘1bdldr80p39g1mjnh76xw6hmwqrrb8lz-wine64-6.0’

1bdldr80p39g1mjnh76xw6hmwqrrb8lz-wine64-6.0       0%[                                                                                                      ] 167.79K  13.4KB/s    eta 69m 20s^C
```

HOWEVER, if I `torify wget`:

```
$ torify wget https://ci.guix.gnu.org/nar/lzip/1bdldr80p39g1mjnh76xw6hmwqrrb8lz-wine64-6.0
--2021-03-05 18:57:00--  https://ci.guix.gnu.org/nar/lzip/1bdldr80p39g1mjnh76xw6hmwqrrb8lz-wine64-6.0
Resolving ci.guix.gnu.org (ci.guix.gnu.org)... 141.80.181.40
Connecting to ci.guix.gnu.org (ci.guix.gnu.org)|141.80.181.40|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 57048561 (54M) [application/octet-stream]
Saving to: ‘1bdldr80p39g1mjnh76xw6hmwqrrb8lz-wine64-6.0.1’

1bdldr80p39g1mjnh76xw6hmwqrrb8lz-wine64-6.0.1    13%[============>                                                                                         ]   7.13M   683KB/s    eta 72s    ^C
```

Which is a big WHAT THE FUCK? Tor should not be 45x faster (683KB/s) than directly from my ISP (13KB/s). I am not using any special workarounds for Tor, like meek or obfs4, which makes me doubt that my ISP is attempting to censor --- if my ISP is the one doing the censoring, it would probably target Tor first before ci.guix.gnu.org, but I can access Tor just fine without special workarounds needed for very oppressive countries.

I've tried this a few times.  Consistently, using `torify wget` is much faster than `wget` alone, I don't have to bang my head in boredom with `torify wget`.  This is not my expectation given what I understand of the Tor network and the public network.

Is `ci.guix.gnu.org` doing some kind of per-IP or per-ISP or other throttling?  This looks very suspicious given the massive speed difference when using Tor and not using Tor.

Doing `torify guix package -m manifest.scm` does not seem to perform the download inside Tor, which makes sense since it's probably `guix-daemon` that is doing the download.

So my questions are:

* WHY IS DIRECT DOWNLOAD FROM MY ISP SLOWER THAN TOR? What can I do to check if it's my ISP that is attempting to censor ci.guix.gnu.org?
* Is there a way to make `guix-daemon` use a Tor proxy?  I have two systems using Guix, one is a Guix System, the other is using a foreign distro, and I'd like to adjust both to use Tor instead since it's faster.
  * If not, is there a way to tell `guix-daemon` that I have these `nar` files and it can put them in its store somehow instead of me waiting for it to ***SLOOOOOOOOWLY*** download it?
* Are there any mirrors I can use for substitutes instead? How do I change my systems (both the Guix System and the foreign distro) to use the mirrors? Is there some configuration option to do this?
* How hard would it be to make the substitute download process at least make an effort to attempt to continue a failed download instead of failing immediately and forcing a restart from 0%? `texlive` is something like 400Mb, it takes hours at 10KiB/s, I once had it fail at >90% which was very painful because it had to restart from 0%, I had to do `guix package -m manifest.scm` in a loop over and over again until it finished download. It would probably cut down on people hammering on the server as well.

Thanks
raid5atemyhomework

bug#46942: ci.guix.gnu.org is slow from my system

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 832 bytes --]


raid5atemyhomework via Bug reports for GNU Guix <bug-guix@gnu.org> writes:

> Downloading substitutes from ci.guix.gnu.org is slow from my two Guix-using computers.
>
> One is a pure Guix System install without any channels, the other one is a foreign Guix install with the-channel-that-cannot-be-named.
>
>
> ```
> downloading from https://ci.guix.gnu.org/nar/lzip/1bdldr80p39g1mjnh76xw6hmwqrrb8lz-wine64-6.0 ...
>  wine64-6.0  54.4MiB                                                                                                                                 10KiB/s 01:55 [                  ]   2.1%
> ```

If you try this same download again, do you get the same speed?

guix publish, which is used on ci.guix.gnu.org can dynamically create
nars if they're not cached, which will probably result in slower
download speeds.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

bug#46942: ci.guix.gnu.org is slow from my system

[raid5atemyhomework via Bug reports for GNU Guix]

Hello Christopher,

> raid5atemyhomework via Bug reports for GNU Guix bug-guix@gnu.org writes:
>
> > Downloading substitutes from ci.guix.gnu.org is slow from my two Guix-using computers.
> > One is a pure Guix System install without any channels, the other one is a foreign Guix install with the-channel-that-cannot-be-named.
> >
> >     downloading from https://ci.guix.gnu.org/nar/lzip/1bdldr80p39g1mjnh76xw6hmwqrrb8lz-wine64-6.0 ...
> >      wine64-6.0  54.4MiB                                                                                                                                 10KiB/s 01:55 [                  ]   2.1%
> >
>
> If you try this same download again, do you get the same speed?
>
> guix publish, which is used on ci.guix.gnu.org can dynamically create
> nars if they're not cached, which will probably result in slower
> download speeds.

Yes, I have been suffering slow download speeds for a few weeks now.  Even with repeated downloads, it is consistently slow.  I only just now tried out using `torify wget`, which surprised me since it was much much faster --- one lucky download even got up to 6MB/s over frikking TOR before I ^Ced it.  I've even tried doing `torify wget` and `wget` in parallel on two separate shells --- the `torify wget` was definitely faster, getting anywhere from 100kB/s to 700kB/s with a few outliers at >1MB/s, while `wget` was consistently below 40kB/s and eventually settling to ~11kB/s after a few seconds.

This holds even for packages I expect to be popular and thus likely to be in-cache with high probability, like `linux-libre`.

Thanks
raid5atemyhomework

bug#46942: ci.guix.gnu.org is slow from my system

[raid5atemyhomework via Bug reports for GNU Guix]

> * Is there a way to make `guix-daemon` use a Tor proxy?  I have two systems using Guix, one is a Guix System, the other is using a foreign distro, and I'd like to adjust both to use Tor instead since it's faster.

I saw that `guix-daemon` respects `http_proxy` and `https_proxy` envvars, but trying it out on my foreign-distro Guix computer, adding `https_proxy=socks5h://127.0.0.1:9050 http_proxy=socks5h://127.0.0.1:9050` to the `systemd` service file doesn't work.

```
guix substitute: error: TLS error in procedure 'handshake': The TLS connection was non-properly terminated.
substitution of /gnu/store/1bdldr80p39g1mjnh76xw6hmwqrrb8lz-wine64-6.0 failed
guix package: error: some substitutes for the outputs of derivation `/gnu/store/wr9kf2bgcsvwxcmhnl9lf047nr8xcklc-wine64-6.0.drv' failed (usually happens due to networking issues); try `--fallback' to build derivation from source
```
Looking at the foreign distro's syslog:

```
Mar  5 19:52:03 developer guix-daemon[145182]: accepted connection from pid 145190, user raid5atemyhomework
Mar  5 19:52:05 developer guix-daemon[145200]: spurious SIGPOLL
Mar  5 19:52:07 developer Tor[1029]: Socks version 67 not recognized. (This port is not an HTTP proxy; did you want to use HTTPTunnelPort?)
```

So it looks to me that `guix-daemon` expects `https_proxy` to be an HTTPS proxy and not a SOCKS5/SOCKS5H proxy. I'll look into Tor's HTTPTunnelPort.

Thanks
raid5atemyhomework

bug#46942: ci.guix.gnu.org is slow from my system

[zimoun]

Hi,

On Fri, 05 Mar 2021 at 11:22, raid5atemyhomework via Bug reports for GNU Guix <bug-guix@gnu.org> wrote:

> This can be very slow, including as slow as 4KiB/s at times.

[...]

> The problem is not on my ISP, or at least not solely on my ISP. Doing a `wget` from `github.com`:

[...]

> As can be seen above, I get 9.34MB/s elsewhere, which is better than the >60Mbit/s promised by my ISP.

Where are you located?  I mean, ci.guix.gnu.org is in Berlin, so if you
are far, say in Alaska or in Puerto Williams[*], you get a poor
downloading rate.  However, github.com probably uses CDN or something
like that, so obviously the rate becomes much better.


*Puerto Williams, the World Southest town with an Internet
 connection. :-)


All the best,
simon

bug#46942: ci.guix.gnu.org is slow from my system

[raid5atemyhomework via Bug reports for GNU Guix]

Hi all,

> > -   Is there a way to make `guix-daemon` use a Tor proxy? I have two systems using Guix, one is a Guix System, the other is using a foreign distro, and I'd like to adjust both to use Tor instead since it's faster.
>
> I saw that`guix-daemon` respects `http_proxy` and `https_proxy` envvars, but trying it out on my foreign-distro Guix computer, adding `https_proxy=socks5h://127.0.0.1:9050 http_proxy=socks5h://127.0.0.1:9050` to the `systemd` service file doesn't work.
>
>     guix substitute: error: TLS error in procedure 'handshake': The TLS connection was non-properly terminated.
>     substitution of /gnu/store/1bdldr80p39g1mjnh76xw6hmwqrrb8lz-wine64-6.0 failed
>     guix package: error: some substitutes for the outputs of derivation `/gnu/store/wr9kf2bgcsvwxcmhnl9lf047nr8xcklc-wine64-6.0.drv' failed (usually happens due to networking issues); try `--fallback' to build derivation from source
>
>
> Looking at the foreign distro's syslog:
>
>     Mar  5 19:52:03 developer guix-daemon[145182]: accepted connection from pid 145190, user raid5atemyhomework
>     Mar  5 19:52:05 developer guix-daemon[145200]: spurious SIGPOLL
>     Mar  5 19:52:07 developer Tor[1029]: Socks version 67 not recognized. (This port is not an HTTP proxy; did you want to use HTTPTunnelPort?)
>
>
> So it looks to me that`guix-daemon` expects `https_proxy` to be an HTTPS proxy and not a SOCKS5/SOCKS5H proxy. I'll look into Tor's HTTPTunnelPort.

On the foreign distro computer, adding an `HTTPTunnelPort 9080` to `/etc/tor/torrc` and then adding `http_proxy=https://127.0.0.1:9080 https_proxy=https://127.0.0.1:9080` to `guix-daemon.service`, then restarting services, seems to work.

```
ownloading from https://ci.guix.gnu.org/nar/lzip/1bdldr80p39g1mjnh76xw6hmwqrrb8lz-wine64-6.0 ...
 wine64-6.0  54.4MiB                                                                                                                                579KiB/s 01:36 [##################] 100.0%

substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
The following derivation will be built:
   /gnu/store/7mr17xka558smr0c76crf9g727ccj76g-profile.drv

3.2 MB will be downloaded
downloading from https://ci.guix.gnu.org/nar/lzip/gs3li4m0ydajm57r0qn1wvsdyfsa68p7-font-gnu-unifont-13.0.06 ...
 font-gnu-unifont-13.0.06  3.0MiB                                                                                                                   515KiB/s 00:06 [##################] 100.0%

```

The above is significantly better than the previous runs where I get 11KiB/s, and matches the speeds I get from `torify wget`.

While it's a good ***workaround*** for my problem instead of me silently weeping at the ridiculous slowness of Guix substitutes, it doesn't solve my root problem:

* SOMETHING between my ISP and ci.guix.gnu.org is throttling access to the substitutes.
  * Given that I have been using my ISP for a year without experiencing such spurious slowdowns, and I have been using ci.guix.gnu.org for the past few months only and have been hit with this slowness in the past month or so, I am more inclined to blame ci.guix.gnu.org, but please tell me how I can find out what is throttling the bandwidth here.  The fact that Tor is ***FASTER*** is very suspicious.

Thanks
raid5atemyhomework

bug#46942: ci.guix.gnu.org is slow from my system

[raid5atemyhomework via Bug reports for GNU Guix]

Hi zimoun,

> Hi,
>
> On Fri, 05 Mar 2021 at 11:22, raid5atemyhomework via Bug reports for GNU Guix bug-guix@gnu.org wrote:
>
> > This can be very slow, including as slow as 4KiB/s at times.
>
> [...]
>
> > The problem is not on my ISP, or at least not solely on my ISP. Doing a `wget` from `github.com`:
>
> [...]
>
> > As can be seen above, I get 9.34MB/s elsewhere, which is better than the >60Mbit/s promised by my ISP.
>
> Where are you located? I mean,ci.guix.gnu.org is in Berlin, so if you
> are far, say in Alaska or in Puerto Williams[*], you get a poor
> downloading rate. However, github.com probably uses CDN or something
> like that, so obviously the rate becomes much better.
>
> *Puerto Williams, the World Southest town with an Internet
> connection. :-)

I would rather not say, since I am a very private person (otherwise I wouldn't be using the "raid5atemyhomework" moniker).

What I **do** find strange is that ***Tor is faster***.  Surely connecting directly from my ISP to ci.guix.gnu.org should be, in principle, faster than connecting to my ISP to a random node, which connects to another random node, which connects to another random node, which finally connects to Berlin, should be ***slower***?

Here are a few `ping`s to some places:

```
$ ping gnu.org
PING gnu.org (209.51.188.148) 56(84) bytes of data.
64 bytes from wildebeest.gnu.org (209.51.188.148): icmp_seq=1 ttl=50 time=301 ms
64 bytes from wildebeest.gnu.org (209.51.188.148): icmp_seq=2 ttl=50 time=306 ms
64 bytes from wildebeest.gnu.org (209.51.188.148): icmp_seq=3 ttl=50 time=335 ms
64 bytes from wildebeest.gnu.org (209.51.188.148): icmp_seq=4 ttl=50 time=257 ms
64 bytes from wildebeest.gnu.org (209.51.188.148): icmp_seq=5 ttl=50 time=234 ms
^C
--- gnu.org ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4226ms
rtt min/avg/max/mdev = 233.804/286.626/335.242/36.261 ms

$ ping www.vikings.net
PING www.vikings.net (168.119.169.112) 56(84) bytes of data.
64 bytes from v0.h.vkgs.io (168.119.169.112): icmp_seq=1 ttl=48 time=190 ms
64 bytes from v0.h.vkgs.io (168.119.169.112): icmp_seq=2 ttl=48 time=190 ms
64 bytes from v0.h.vkgs.io (168.119.169.112): icmp_seq=3 ttl=48 time=220 ms
64 bytes from v0.h.vkgs.io (168.119.169.112): icmp_seq=4 ttl=48 time=241 ms
64 bytes from v0.h.vkgs.io (168.119.169.112): icmp_seq=5 ttl=48 time=264 ms
^C
--- www.vikings.net ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4413ms
rtt min/avg/max/mdev = 189.767/220.941/264.326/29.054 ms

$ ping www.raptorcs.com
PING www.raptorcs.com (23.155.224.44) 56(84) bytes of data.
64 bytes from websvc.rptsys.com (23.155.224.44): icmp_seq=1 ttl=55 time=294 ms
64 bytes from websvc.rptsys.com (23.155.224.44): icmp_seq=2 ttl=55 time=307 ms
64 bytes from websvc.rptsys.com (23.155.224.44): icmp_seq=3 ttl=55 time=227 ms
^C
--- www.raptorcs.com ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2627ms
rtt min/avg/max/mdev = 226.868/275.933/306.748/35.071 ms
```

And here's a few select `wget`s:

```
$ wget https://store.vikings.net/image/cache/catalog/kcmad8-1088x816.jpeg
--2021-03-05 20:50:37--  https://store.vikings.net/image/cache/catalog/kcmad8-1088x816.jpeg
Resolving store.vikings.net (store.vikings.net)... 185.199.141.17
Connecting to store.vikings.net (store.vikings.net)|185.199.141.17|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 246991 (241K) [image/jpeg]
Saving to: ‘kcmad8-1088x816.jpeg’

kcmad8-1088x816.jpeg                            100%[=====================================================================================================>] 241.20K  19.2KB/s    in 13s

2021-03-05 20:50:51 (19.2 KB/s) - ‘kcmad8-1088x816.jpeg’ saved [246991/246991]

$ torify wget https://store.vikings.net/image/cache/catalog/kcmad8-1088x816.jpeg
--2021-03-05 20:51:22--  https://store.vikings.net/image/cache/catalog/kcmad8-1088x816.jpeg
Resolving store.vikings.net (store.vikings.net)... 185.199.141.17
Connecting to store.vikings.net (store.vikings.net)|185.199.141.17|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 246991 (241K) [image/jpeg]
Saving to: ‘kcmad8-1088x816.jpeg.1’

kcmad8-1088x816.jpeg.1                          100%[=====================================================================================================>] 241.20K   293KB/s    in 0.8s

2021-03-05 20:51:25 (293 KB/s) - ‘kcmad8-1088x816.jpeg.1’ saved [246991/246991]

$ wget https://static.fsf.org/nosvn/videos/fight-to-repair/videos/Fight-to-Repair-720p.webm
--2021-03-05 20:54:03--  https://static.fsf.org/nosvn/videos/fight-to-repair/videos/Fight-to-Repair-720p.webm
Resolving static.fsf.org (static.fsf.org)... 209.51.188.233, 2001:470:142:5::233
Connecting to static.fsf.org (static.fsf.org)|209.51.188.233|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 40191351 (38M) [video/webm]
Saving to: ‘Fight-to-Repair-720p.webm’

Fight-to-Repair-720p.webm                         2%[=>                                                                                                    ] 856.00K   146KB/s    eta 4m 32s ^C

$ torify wget https://static.fsf.org/nosvn/videos/fight-to-repair/videos/Fight-to-Repair-720p.webm
--2021-03-05 20:54:13--  https://static.fsf.org/nosvn/videos/fight-to-repair/videos/Fight-to-Repair-720p.webm
Resolving static.fsf.org (static.fsf.org)... 209.51.188.233
Connecting to static.fsf.org (static.fsf.org)|209.51.188.233|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 40191351 (38M) [video/webm]
Saving to: ‘Fight-to-Repair-720p.webm.1’

Fight-to-Repair-720p.webm.1                       6%[=====>                                                                                                ]   2.55M   498KB/s    eta 92s    ^C

$ wget https://static.raptorcs.com/TL2WK2/images/boardlarge.png
--2021-03-05 20:57:19--  https://static.raptorcs.com/TL2WK2/images/boardlarge.png
Resolving static.raptorcs.com (static.raptorcs.com)... 23.155.224.44
Connecting to static.raptorcs.com (static.raptorcs.com)|23.155.224.44|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 5398922 (5.1M) [image/png]
Saving to: ‘boardlarge.png’

boardlarge.png                                  100%[=====================================================================================================>]   5.15M   505KB/s    in 9.4s

2021-03-05 20:57:30 (559 KB/s) - ‘boardlarge.png’ saved [5398922/5398922]

$ torify wget https://static.raptorcs.com/TL2WK2/images/boardlarge.png
--2021-03-05 20:57:32--  https://static.raptorcs.com/TL2WK2/images/boardlarge.png
Resolving static.raptorcs.com (static.raptorcs.com)... 23.155.224.44
Connecting to static.raptorcs.com (static.raptorcs.com)|23.155.224.44|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 5398922 (5.1M) [image/png]
Saving to: ‘boardlarge.png.1’

boardlarge.png.1                                100%[=====================================================================================================>]   5.15M   534KB/s    in 10s

2021-03-05 20:57:45 (515 KB/s) - ‘boardlarge.png.1’ saved [5398922/5398922]

```

I know Vikings is based in Germany, so it looks like my ISP does not like connecting to German sites --- again I am seeing this phenomenon where using Tor is faster than connecting directly, but it certainly now looks a whole lot more like a problem with my ISP.

On the other hand the `fsf.org` site has better speed when connected directly, but still the one via Tor is about 3x faster.

And finally RaptorCS is about the same speed both directly and over Tor.  Again, this is still a surprise as I expect Tor to be slower than direct connection.

Lemme go poke at my ISP.

Thanks
raid5atemyhomework

bug#46942: ci.guix.gnu.org is slow from my system

[raid5atemyhomework via Bug reports for GNU Guix]

Hi all,

In case it's useful, here's my `traceroute`, would this be helpful for Guix?

I erased some internal IP addresses and omit a few bytes off the source country. After hop 25 `traceroute` couldn't find anything anymore. I annotated the IP addresses to country map.

```
$ sudo `which traceroute` 141.80.181.40
traceroute to 141.80.181.40 (141.80.181.40), 64 hops max
  1   <LAN>  3.140ms  0.947ms  0.981ms
  2   <ISP INTERNAL>  2.527ms  1.571ms  1.416ms
  3   <ISP INTERNAL>  4.318ms  4.705ms  3.680ms
  4   <COUNTRY>.4.16  2.605ms  1.856ms  1.963ms
  5   <COUNTRY>.1.205  2.177ms  2.087ms  1.876ms
  6   <COUNTRY>.3.54  29.090ms  30.237ms  28.982ms
  7   203.208.168.129  29.200ms  29.497ms  30.863ms     # Singapore Singapore Telecommunications Pte Ltd
  8   203.208.152.82  29.295ms  29.231ms  29.227ms      # Singapore SingTel Internet Exchange
  9   203.208.182.249  30.769ms  31.872ms  47.615ms     # Singapore Singapore Telecommunications Pte Ltd
 10   203.208.183.133  30.271ms  30.713ms  31.635ms     # Singapore Singapore Telecommunications Pte Ltd
 11   203.208.158.178  363.911ms  301.844ms  204.896ms  # Singapore SingTel Internet Exchange
 12   203.208.172.234  213.664ms  298.038ms  213.798ms  # Singapore Singapore Telecommunications Pte Ltd
 13   *  *  *
 14   *  *  *
 15   *  *  *
 16   *  *  *
 17   *  *  *
 18   64.125.28.36  335.579ms  499.267ms  411.078ms     # Netherlands Zayo Bandwidth
 19   *  *  *
 20   64.125.28.36  349.449ms  347.413ms  432.726ms     # Netherlands Zayo Bandwidth
 21   80.81.193.222  334.576ms  334.150ms  335.696ms    # Germany DE-CIX Management GmbH
 22   64.125.26.173  343.767ms  344.376ms  344.544ms    # Germany Zayo Bandwidth
 23   188.1.238.78  375.499ms  346.705ms  417.986ms     # Germany Verein zur Foerderung eines Deutschen Forschungsnetzes e.V.
 24   141.80.238.11  339.792ms  383.421ms  406.579ms    # Germany Campus Berlin-Buch GmbH
 25   188.1.146.210  410.225ms  334.844ms  335.196ms    # Germany Verein zur Foerderung eines Deutschen Forschungsnetzes e.V.
```

The big increase in ping times is somewhere inside Singapore, which seems like a reasonable assumption that it's what is throttling bandwidth. A quick research shows Singapore Telecommunications Pte Ltd is the same as SingTel Internet Exchange.

Are there any other mirror substitute servers aside from `ci.guix.gnu.org`?

Thanks
raid5atemyhomework

bug#46942: ci.guix.gnu.org is slow from my system

[zimoun]

Hi,

On Fri, 05 Mar 2021 at 14:46, raid5atemyhomework <raid5atemyhomework@protonmail.com> wrote:

> Are there any other mirror substitute servers aside from `ci.guix.gnu.org`?

Well, only one in China I AFAIK.

   <https://mirrors.sjtug.sjtu.edu.cn/guix>

See <https://yhetil.org/guix/87czz24ilu.fsf@riseup.net> for details.


All the best,
simon

bug#46942: ci.guix.gnu.org is slow from my system

[Leo Famulari]

On Fri, Mar 05, 2021 at 11:22:17AM +0000, raid5atemyhomework via Bug reports for GNU Guix wrote:
> Downloading substitutes from ci.guix.gnu.org is slow from my two Guix-using computers.
> 
> One is a pure Guix System install without any channels, the other one is a foreign Guix install with the-channel-that-cannot-be-named.
> 
> 
> ```
> downloading from https://ci.guix.gnu.org/nar/lzip/1bdldr80p39g1mjnh76xw6hmwqrrb8lz-wine64-6.0 ...
>  wine64-6.0  54.4MiB                                                                                                                                 10KiB/s 01:55 [                  ]   2.1%
> ```

Sometimes we get reports like this, of strangely slow downloads from
ci.guix.gnu.org. They don't seem to depend on geography. They sometimes
happen to people who usually download quickly from ci.guix.gnu.org. So
far, we don't have any idea why they happen. But, it's not just you.

You are the first person to file a detailed bug report about it, if I
remember correctly (if!). If you'd like to help debug it, let us know,
but it would probably mean identifying your IP / location.

bug#46942: ci.guix.gnu.org is slow from my system

[Julien Lepiller]

[-- Attachment #1: Type: text/plain, Size: 1517 bytes --]

Actually, location usually has nothing to do with it (unless someone dug a hole right in the middle of optic fibers, but you'd notice). The most important info is AS (basically, which ISP you're using, but some have more than one AS). We don't need your IP, but ISP might be already too revealing for your taste.

Le 5 mars 2021 14:40:10 GMT-05:00, Leo Famulari <leo@famulari.name> a écrit :
>On Fri, Mar 05, 2021 at 11:22:17AM +0000, raid5atemyhomework via Bug
>reports for GNU Guix wrote:
>> Downloading substitutes from ci.guix.gnu.org is slow from my two
>Guix-using computers.
>> 
>> One is a pure Guix System install without any channels, the other one
>is a foreign Guix install with the-channel-that-cannot-be-named.
>> 
>> 
>> ```
>> downloading from
>https://ci.guix.gnu.org/nar/lzip/1bdldr80p39g1mjnh76xw6hmwqrrb8lz-wine64-6.0
>...
>>  wine64-6.0  54.4MiB                                                 
>                              10KiB/s 01:55 [                  ]   2.1%
>> ```
>
>Sometimes we get reports like this, of strangely slow downloads from
>ci.guix.gnu.org. They don't seem to depend on geography. They sometimes
>happen to people who usually download quickly from ci.guix.gnu.org. So
>far, we don't have any idea why they happen. But, it's not just you.
>
>You are the first person to file a detailed bug report about it, if I
>remember correctly (if!). If you'd like to help debug it, let us know,
>but it would probably mean identifying your IP / location.

[-- Attachment #2: Type: text/html, Size: 2037 bytes --]

bug#46942: ci.guix.gnu.org is slow from my system

[Leo Famulari]

Right, I meant that the required information would likely make it
possible to determine this person's location. Or at least make a good
guess.

On Fri, Mar 05, 2021 at 04:53:33PM -0500, Julien Lepiller wrote:
> Actually, location usually has nothing to do with it (unless someone dug a hole right in the middle of optic fibers, but you'd notice). The most important info is AS (basically, which ISP you're using, but some have more than one AS). We don't need your IP, but ISP might be already too revealing for your taste.
> 
> Le 5 mars 2021 14:40:10 GMT-05:00, Leo Famulari <leo@famulari.name> a écrit :
> >On Fri, Mar 05, 2021 at 11:22:17AM +0000, raid5atemyhomework via Bug
> >reports for GNU Guix wrote:
> >> Downloading substitutes from ci.guix.gnu.org is slow from my two
> >Guix-using computers.
> >> 
> >> One is a pure Guix System install without any channels, the other one
> >is a foreign Guix install with the-channel-that-cannot-be-named.
> >> 
> >> 
> >> ```
> >> downloading from
> >https://ci.guix.gnu.org/nar/lzip/1bdldr80p39g1mjnh76xw6hmwqrrb8lz-wine64-6.0
> >...
> >>  wine64-6.0  54.4MiB                                                 
> >                              10KiB/s 01:55 [                  ]   2.1%
> >> ```
> >
> >Sometimes we get reports like this, of strangely slow downloads from
> >ci.guix.gnu.org. They don't seem to depend on geography. They sometimes
> >happen to people who usually download quickly from ci.guix.gnu.org. So
> >far, we don't have any idea why they happen. But, it's not just you.
> >
> >You are the first person to file a detailed bug report about it, if I
> >remember correctly (if!). If you'd like to help debug it, let us know,
> >but it would probably mean identifying your IP / location.

bug#46942: ci.guix.gnu.org is slow from my system

[raid5atemyhomework via Bug reports for GNU Guix]

Hi zimoun,

> Hi,
>
> On Fri, 05 Mar 2021 at 14:46, raid5atemyhomework raid5atemyhomework@protonmail.com wrote:
>
> > Are there any other mirror substitute servers aside from `ci.guix.gnu.org`?
>
> Well, only one in China I AFAIK.
>
> https://mirrors.sjtug.sjtu.edu.cn/guix
>
> Seehttps://yhetil.org/guix/87czz24ilu.fsf@riseup.net for details.


Thank you very much, an experimental `wget` shows I get much higher speeds from this:

```
$ wget https://mirror.sjtu.edu.cn/guix/nar/lzip/1bdldr80p39g1mjnh76xw6hmwqrrb8lz-wine64-6.0
--2021-03-06 06:45:10--  https://mirror.sjtu.edu.cn/guix/nar/lzip/1bdldr80p39g1mjnh76xw6hmwqrrb8lz-wine64-6.0
Resolving mirror.sjtu.edu.cn (mirror.sjtu.edu.cn)... 111.186.58.212, 2001:da8:8000:7100::322:a
Connecting to mirror.sjtu.edu.cn (mirror.sjtu.edu.cn)|111.186.58.212|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 57048561 (54M) [application/octet-stream]
Saving to: ‘1bdldr80p39g1mjnh76xw6hmwqrrb8lz-wine64-6.0.3’

1bdldr80p39g1mjnh76xw6hmwqrrb8lz-wine64-6.0.3   100%[=====================================================================================================>]  54.41M  3.82MB/s    in 17s

2021-03-06 06:45:31 (3.21 MB/s) - ‘1bdldr80p39g1mjnh76xw6hmwqrrb8lz-wine64-6.0.3’ saved [57048561/57048561]

```

I will try it out as the first server in `--substitute-urls`.

Would it not be appropriate to somehow put a list of such substitute servers (even if it is currently only a single mirror) somewhere official, like the Guix manual? I suppose if Peng Mei Yu thinks this mirror would be maintained long term?

Thanks
raid5atemyhomework

bug#46942: ci.guix.gnu.org is slow from my system

[zimoun]

Hi,

On Fri, 05 Mar 2021 at 22:57, raid5atemyhomework <raid5atemyhomework@protonmail.com> wrote:

> Would it not be appropriate to somehow put a list of such substitute
> servers (even if it is currently only a single mirror) somewhere
> official, like the Guix manual? I suppose if Peng Mei Yu thinks this
> mirror would be maintained long term? 

It was something discussed in the mentioned thread.  But no one took the
time to send a patch for the manual or the cookbook.  Feel free to do. :-)

Nice if it improves the situation for you.

Do you consider the problem (bug report) is solved?  In order to close it.


Cheers,
simon

bug#46942: ci.guix.gnu.org is slow from my system

[raid5atemyhomework via Bug reports for GNU Guix]

Hi zimoun,

Thanks, this is a definite improvement and I have set both systems to use it as the first item in `substitute-server`. I'll make a patch for the manual at least, then close this issue once that patch is accepted.

So I was thinking of modifying the installer so at least some page offers up options for mirrors. However, because of the way the installer works, it would have to be done by `(gnu installer services)`, which does not use `modify-services` on the base service list.

What the installer expects is that services will have their own `(service <type> <config>)` entry, without modifying the base service type.

What I *want* to do would be to have an extensible `guix-substitute-url-service-type`.  Unfortunately the existing `guix-service-type` accepts a list of build directories to `chroot`.

So here's a sketch:

* Create a new `guix-daemon-service-type` and move most of the `guix-service-type` code into it.
  * This is extensible; extensions provide a procedure which accepts a `<guix-configuration>` record and outputs a `<guix-configuration>` record.
  * `(compose (cut apply compose <>))`
  * `(extend (lambda (config modifier) (modifier config)))`
* Create a new `guix-build-chroot-service-type` which just extends `guix-daemon-service-type`:
  * `(service-extension guix-daemon-service-type (lambda (build-chroots) (lambda (guix-config) (guix-configuration (inherit guix-config) (chroot-directories (append (guix-configuration-chroot-directories guix-config) build-chroots))))))`
* `(define-deprecated guix-service-type guix-build-chroot-service-type)`
  * I mean seriously why does Guix assume only one configuration field of a base system service is usefully extensible, it seems to me that the general pattern should be that basic system service types should be extensible by a procedure that accepts an existing configuration record and returns a modified configuration record, then just define individual service types for each list-of-foo field of the configuration record to make a convenient way to extend such lists.
* Create a new `guix-substitute-url-service-type` similar to `guix-build-chroot-service-type`, which prepends a list of substitute URLs to the one in the configuration.

Then finally in the installer side:

* in `(gnu installer services)` add something like:
  * `(system-service (name (G_ "https://mirrors.sjtug.sjtu.edu.cn/guix/ (SJTUG, China)")) (type 'substitute-url) (snippet '((simple-service guix-substitute-url-service-type (list "https://mirrors.sjtug.sjtu.edu.cn/guix/")))))`
  * `(system-service (name (G_ "https://ci.guix.gnu.org (Guix, Germany) - no mirror") (type 'substitute-url) (snippet '())))`
* In `(gnu installer newt services)`, add a page for substitute URL mirrors, probably a `run-listbox-selection-page`.
  * `(G_ "You can select a mirror that is nearer to you for faster updating of packages. The main Guix substitute server, https://ci.guix.gnu.org/, will always be set as a fallback, so even if the mirror goes down, you can still upgrade from the main Guix substitute server.")`

Is that something that has any chance of getting accepted into Guix? I'd rather not do any coding unless this has any chance at all of getting merged in.

Thanks
raid5atemyhomework

bug#46942: ci.guix.gnu.org is slow from my system

[zimoun]

Hi,

On Sat, 06 Mar 2021 at 00:23, raid5atemyhomework <raid5atemyhomework@protonmail.com> wrote:
> Hi zimoun,
>
> Thanks, this is a definite improvement and I have set both systems to
> use it as the first item in `substitute-server`. I'll make a patch for
> the manual at least, then close this issue once that patch is
> accepted.

Ok.


> So I was thinking of modifying the installer so at least some page
> offers up options for mirrors. However, because of the way the
> installer works, it would have to be done by `(gnu installer
> services)`, which does not use `modify-services` on the base service
> list. 

[...]

> Is that something that has any chance of getting accepted into Guix?
> I'd rather not do any coding unless this has any chance at all of
> getting merged in. 

From my point of view, I will get a better chance for an answer if you
ask on guix-devel with an appropriate subject, rather in (relatively)
long thread inside a bug report with a subject out this very
question. My 2 cent. ;-)


Cheers,
simon

bug#46942: ci.guix.gnu.org is slow from my system

[raid5atemyhomework via Bug reports for GNU Guix]

Hello all,

Unfortunately, it seems that the SJTU server is somewhat unreliable, I sometimes get random failures in various parts talking to the substitute server, complaining of strange responses from the server:

```
Backtrace:
In guix/ui.scm:
  2164:12 19 (run-guix-command _ . _)
In guix/scripts/substitute.scm:
    652:2 18 (guix-substitute . _)
In unknown file:
          17 (with-continuation-barrier #<procedure thunk ()>)
In ice-9/boot-9.scm:
  1736:10 16 (with-exception-handler _ _ #:unwind? _ # _)
In unknown file:
          15 (apply-smob/0 #<thunk 7f5de06c9cc0>)
In ice-9/boot-9.scm:
  1736:10 14 (with-exception-handler _ _ #:unwind? _ # _)
  1736:10 13 (with-exception-handler _ _ #:unwind? _ # _)
  1731:15 12 (with-exception-handler #<procedure 7f5dddeb8e70 at ic…> …)
In guix/scripts/substitute.scm:
   701:17 11 (_)
    410:7 10 (process-substitution _ "/gnu/store/yfzsz94qy92c7m9w0j…" …)
In ice-9/boot-9.scm:
  1736:10  9 (with-exception-handler _ _ #:unwind? _ # _)
In guix/scripts/substitute.scm:
    419:9  8 (_)
In ice-9/boot-9.scm:
  1731:15  7 (with-exception-handler #<procedure 7f5dddeb8990 at ic…> …)
  1669:16  6 (raise-exception _ #:continuable? _)
  1667:16  5 (raise-exception _ #:continuable? _)
  1669:16  4 (raise-exception _ #:continuable? _)
  1764:13  3 (_ #<&compound-exception components: (#<&error> #<&irri…>)
  1669:16  2 (raise-exception _ #:continuable? _)
  1667:16  1 (raise-exception _ #:continuable? _)
  1669:16  0 (raise-exception _ #:continuable? _)

ice-9/boot-9.scm:1669:16: In procedure raise-exception:
Throw to key `bad-response' with args `("Bad Response-Line: ~s" (""))'.
Backtrace:
In ice-9/boot-9.scm:
  1736:10  4 (with-exception-handler _ _ #:unwind? _ # _)
In unknown file:
           3 (apply-smob/0 #<thunk 7f5de074e520>)
In ice-9/boot-9.scm:
    718:2  2 (call-with-prompt _ _ #<procedure default-prompt-handle…>)
In ice-9/eval.scm:
    619:8  1 (_ #(#(#<directory (guile-user) 7f5de0751c80>)))
In guix/ui.scm:
  2164:12  0 (run-guix-command _ . _)

guix/ui.scm:2164:12: In procedure run-guix-command:
Throw to key `bad-response' with args `("Bad Response-Line: ~s" (""))'.
substitution of /gnu/store/yfzsz94qy92c7m9w0jbll7slc2pcap45-guix-packages-base failed
guix pull: error: corrupt input while restoring archive from #<closed: file 7f0dc2367bd0>
```

I configured `guix-daemon` to have two substitute URLs: SJTUG, and the official Cuirass in Berlin.  My expectation is that if the SJTUG server fails, it should fall back on using the Cuirass server.

So my problems are two-fold:

* Apparently the SJTUG mirror sometimes acts in ways that aren't quite how Guix expects it.  Either SJTUG or Guix has to be fixed so they talk in compatible manners.
* Multiple Substitute URLs are not useful for the mirroring configuration where one server is temporarily unable to properly serve; further servers are not checked.

I'm still not satisfied with this solution overall.  Yes it's fast but it's imperfect.

I recently had to rebuild an OS (because I was dumb; the Guix language for shepherd services can easily lead you deadlocking shepherd itself) and had supreme difficulty reinstalling, because the Berlin server was too slow (would have taken ~10hours downloading everything) while the SJTUG server was unreliable and would consistently break the automated install.  I had to `guix system build --fallback` separately and *then* `guix system init` manually.

Thanks
raid5atemyhomework

bug#46942: ci.guix.gnu.org is slow from my system

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 684 bytes --]

On Mon, 2021-03-15 at 00:13 +0000, raid5atemyhomework via Bug reports for GNU Guix wrote:
> Hello all,
> 
> [...]
> I recently had to rebuild an OS (because I was dumb; the Guix language
> for shepherd services can easily lead you deadlocking shepherd itself)
> and had supreme difficulty reinstalling, [...]

Reinstalling after a messed up configuration file shouldn't be necessary.
At least when using GRUB as bootloader, guix keeps some old (& presumably
not broken) system generations around, that can be selected when booting
from the bootloader.  (I don't recall exactly how the menu is named,
maybe ‘Old system generations of $HOSTNAMES?)

Greetings,
Maxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

bug#46942: ci.guix.gnu.org is slow from my system

[raid5atemyhomework via Bug reports for GNU Guix]

Hi Maxime,

> On Mon, 2021-03-15 at 00:13 +0000, raid5atemyhomework via Bug reports for GNU Guix wrote:
>
> > Hello all,
> > [...]
> > I recently had to rebuild an OS (because I was dumb; the Guix language
> > for shepherd services can easily lead you deadlocking shepherd itself)
> > and had supreme difficulty reinstalling, [...]
>
> Reinstalling after a messed up configuration file shouldn't be necessary.
> At least when using GRUB as bootloader, guix keeps some old (& presumably
> not broken) system generations around, that can be selected when booting
> from the bootloader. (I don't recall exactly how the menu is named,
> maybe ‘Old system generations of $HOSTNAMES?)

Unfortunately I had a long-standing latent bug in my configuration file that triggered on a (persistent on-disk) edge case which would cause the shepherd process to enter an infinite loop (because the shepherd configuration language is Turing-complete enough to allow infinite loops in the first place).  All the remaining generations (since I didn't like keeping more than a dozen, and had recently been excessively tweaking the configuration file) had this bug, so I had no way of reverting to an even older generation that predated the bug.

Thanks
raid5atemyhomework

bug#46942: ci.guix.gnu.org is slow from my system

[raid5atemyhomework via Bug reports for GNU Guix]

> Hi Maxime,
>
> > On Mon, 2021-03-15 at 00:13 +0000, raid5atemyhomework via Bug reports for GNU Guix wrote:
> >
> > > Hello all,
> > > [...]
> > > I recently had to rebuild an OS (because I was dumb; the Guix language
> > > for shepherd services can easily lead you deadlocking shepherd itself)
> > > and had supreme difficulty reinstalling, [...]
> >
> > Reinstalling after a messed up configuration file shouldn't be necessary.
> > At least when using GRUB as bootloader, guix keeps some old (& presumably
> > not broken) system generations around, that can be selected when booting
> > from the bootloader. (I don't recall exactly how the menu is named,
> > maybe ‘Old system generations of $HOSTNAMES?)
>
> Unfortunately I had a long-standing latent bug in my configuration file that triggered on a (persistent on-disk) edge case which would cause the shepherd process to enter an infinite loop (because the shepherd configuration language is Turing-complete enough to allow infinite loops in the first place). All the remaining generations (since I didn't like keeping more than a dozen, and had recently been excessively tweaking the configuration file) had this bug, so I had no way of reverting to an even older generation that predated the bug.


And regardless, this kind of problem shouldn't occur in the first place.

* Instead of running the `start` code in the same process 1 (which is special enough that no amount of `kill -s SIGKILL 1` will work even if you manage to log into a console), `shepherd` should really run it in a separate process and monitor it if it's taking too long and possibly allow the operator to break out of it.  Principle of least power and all that...
  * If you want details: there is a shepherd service A that is a requirement of shepherd service B, however the daemon launched by A needed to reach a particular point in its initialization before B can start talking to it.  B itself will fail to start if A has not reached that point in initialization.  The extra code I added to the `start` of shepherd service A was to wait for that point of initialization before A was considered "started".  It turned out it was buggy in that if the point was not reached in 1 second it would inadvertently enter an incorrect looping logic (ironically, the logic was supposed to exit it after 60 seconds, but I got increment/decrement crossed, meaning it would always loop as long as you never reached -60 seconds, which was impossible....) that ended up being an infinite loop and preventing process 1 from advancing.  And this point was getting delayed when the process launched by A had to do a lot of (important) data on-disk that it needed to process at startup, so it was persistent on-disk data that would need > 1 second to process, thus ensuring that the buggy code would be entered.
* If this was a new computer it would also be just as screwed during installation anyway, you should consider this a fortuitous discovery of a latent bug.
  * New users trying out Guix System that happen to get hit by this bug might very well decide that Guix is not stable enough for them to commit to using.

Thanks
raid5atemyhomework

bug#46942: ci.guix.gnu.org is slow from my system

[raid5atemyhomework via Bug reports for GNU Guix]

Here's another error!

For *this* instance notice the very slow download speed; other downloads got up to 2MiB/s.  If my understanding is correct the SJTUG server is effectively a caching proxy, meaning that the low download speed here probably means that the SJTUG server itself is downloading from the Berlin Cuirass.  As noted before, often an issue with using ci.guix.gnu.org is that for large multi-megabyte substitutes sometimes the Berlin server will just disconnect and EOF early, so I suspect something similar is happening to the SJTUG server.

```
downloading from https://mirror.sjtu.edu.cn/guix/nar/lzip/6fa3h0n957hr2nbd05ygjnk4idjq122q-ffmpeg-4.3.2 ...
 ffmpeg-4.3.2  8.8MiB                                                                                                                                13KiB/s 05:05 [#######           ]  44.2%
Backtrace:
In guix/store/deduplication.scm:
    227:2 19 (dump-file/deduplicate "/gnu/store/6fa3h0n957hr2nbd05y…" …)
In ice-9/ports.scm:
   463:17 18 (call-with-output-file _ _ #:binary _ #:encoding _)
In guix/store/deduplication.scm:
   232:10 17 (_ _)
In guix/serialization.scm:
    261:6 16 (dump _)
   247:20 15 (dump #<input: string 7fb2d8c83cb0> #<output: string 7…> …)
In unknown file:
          14 (get-bytevector-n! #<input: string 7fb2d8c83cb0> # 0 #)
In guix/store/deduplication.scm:
   203:22 13 (read! #vu8(85 110 107 110 111 119 110 32 112 105 99 …) …)
In unknown file:
          12 (get-bytevector-n! #<input: string 7fb2d8ed4d90> # 0 #)
In gcrypt/hash.scm:
   223:13 11 (read! #vu8(85 110 107 110 111 119 110 32 112 105 99 …) …)
In unknown file:
          10 (get-bytevector-n! #<input: string 7fb2d8ed4e70> # 0 #)
In lzlib.scm:
    501:4  9 (lzread! #<lz-decoder 7fb2d8c6bbf0> #<input: string 7f…> …)
In unknown file:
           8 (get-bytevector-n #<input: string 7fb2d8ed4ee0> 65537)
In guix/progress.scm:
   358:30  7 (read! _ _ _)
In unknown file:
           6 (get-bytevector-n! #<input: string 7fb2d8c832a0> # 0 #)
In web/response.scm:
     95:2  5 (read! _ _ _)
In ice-9/boot-9.scm:
  1669:16  4 (raise-exception _ #:continuable? _)
  1669:16  3 (raise-exception _ #:continuable? _)
  1764:13  2 (_ #<&compound-exception components: (#<&error> #<&irri…>)
  1669:16  1 (raise-exception _ #:continuable? _)
  1669:16  0 (raise-exception _ #:continuable? _)

ice-9/boot-9.scm:1669:16: In procedure raise-exception:
Throw to key `bad-response' with args `("EOF while reading response body: ~a bytes of ~a" (4112173 9196598))'.
Backtrace:
In ice-9/boot-9.scm:
  1736:10  4 (with-exception-handler _ _ #:unwind? _ # _)
In unknown file:
           3 (apply-smob/0 #<thunk 7fb2dec76520>)
In ice-9/boot-9.scm:
    718:2  2 (call-with-prompt _ _ #<procedure default-prompt-handle…>)
In ice-9/eval.scm:
    619:8  1 (_ #(#(#<directory (guile-user) 7fb2dec79c80>)))
In guix/ui.scm:
  2164:12  0 (run-guix-command _ . _)

guix/ui.scm:2164:12: In procedure run-guix-command:
Throw to key `bad-response' with args `("EOF while reading response body: ~a bytes of ~a" (4112173 9196598))'.
substitution of /gnu/store/6fa3h0n957hr2nbd05ygjnk4idjq122q-ffmpeg-4.3.2 failed
guix upgrade: error: some substitutes for the outputs of derivation `/gnu/store/9xx9kbbjals2y7llhak65fnnyfh9rkyq-gnome-tweaks-3.34.1.drv' failed (usually happens due to networking issues); try `--fallback' to build derivation from source
```

Thanks
raid5atemyhomework