bug#57391: "error: connect*: Connection timed out" when using 6+ jobs to fetch substitutes

bug#57391: "error: connect*: Connection timed out" when using 6+ jobs to fetch substitutes

[Maxim Cournoyer]

Hello,

On a machine configured with 20 builders:

--8<---------------cut here---------------start------------->8---
        (guix-service-type config =>
                           (guix-configuration
                            (inherit config)
                            (build-accounts 20)
                            (discover? #t)
                            (log-compression 'none)
                            (extra-options '("--max-jobs=20"))))
--8<---------------cut here---------------end--------------->8---


Running 'guix package -m manifest.scm' with hundreds of packages/MiB to
fetch from the substitute servers, it fails like:

--8<---------------cut here---------------start------------->8---
$ guix package -m ~/stow/guix/manifest.scm -M7
The following packages will be installed:
   acpi                      1.7
   adb                       7.1.2_r36
   adwaita-icon-theme        40.1.1
   alsa-utils                1.2.4
   ansible                   4.7.0
   anthy                     9100h
   arc-icon-theme            20161122
   arc-theme                 20210412
[...]
   xournal                   0.4.8.2016
   xrandr                    1.5.1
   xrdb                      1.2.1
   xsetroot                  1.1.2
   yelp                      41.0

substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://bordeaux.guix.gnu.org'... 100.0%
924.3 MB will be downloaded

substitution of /gnu/store/1w54x244cykra3p2bmmxv9jn12q92l0r-gimp-2.10.32-doc complete

[...]

substitution of /gnu/store/adq43ck5b12sx0n9hy4hgay4gk7spy5b-libxml++-2.40.1 complete

substitution of /gnu/store/i379f1vadva2lnk1jh60c03s26m6rj7b-ibus-1.5.24-doc complete

substitution of /gnu/store/9j1hfzgpllgshy6qn82pl8zkkgn4058w-hackneyed-x11-cursors-0.8.2 complete

substitution of /gnu/store/bl9y57k0wg3aqxb8xiwddagj3i4dk036-perl-b-hooks-endofscope-0.24 complete

substitution of /gnu/store/qpc614b2hbzcsjhac38m9av3cqnrwjqi-perl-datetime-1.54 complete

substitution of /gnu/store/r07xvd187my02gg5smlw4pis90x7m3ah-opensp-1.5.2 complete
guix substitute: warning: while fetching https://ci.guix.gnu.org/nar/lzip/q7hiikl3q3dk1r46rlyyzd90yaddydrd-gnucash-docs-4.10: server is somewhat slow
guix substitute: warning: try `--no-substitutes' if the problem persists
guix substitute: error: connect*: Connection timed out
substitution of /gnu/store/q7hiikl3q3dk1r46rlyyzd90yaddydrd-gnucash-docs-4.10 failed
guix package: error: corrupt input while restoring archive from socket
--8<---------------cut here---------------end--------------->8---

It fails like the above when the number of jobs is 6 or higher, but
proceeds without error when the number of jobs is 5 or less.

I do not think the network to be at caused, as everything is cabled and
the link is healthy and fast.

Thanks,

Maxim

bug#57391: "error: connect*: Connection timed out" when using 6+ jobs to fetch substitutes

[Maxime Devos]

[-- Attachment #1.1.1: Type: text/plain, Size: 620 bytes --]

On 24-08-2022 22:42, Maxim Cournoyer wrote:

> It fails like the above when the number of jobs is 6 or higher, but
> proceeds without error when the number of jobs is 5 or less.
>
> I do not think the network to be at caused, as everything is cabled and
> the link is healthy and fast.
>
> Thanks,

I was getting a lot of those errors recently -- I have "--max-jobs=14" 
in the extra-options of guix-configuration.

It's not exactly the same situation though (WiFi, albeit usually not too 
bad ...).

I'll give "-M5" a try in the future, maybe I can reproduce the 5/6 boundary.

Greetings,
Maxime.


[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 929 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 236 bytes --]

bug#57391: "error: connect*: Connection timed out" when using 6+ jobs to fetch substitutes

[Maxime Devos]

[-- Attachment #1.1.1: Type: text/plain, Size: 1066 bytes --]


On 24-08-2022 22:42, Maxim Cournoyer wrote:
> It fails like the above when the number of jobs is 6 or higher, but
> proceeds without error when the number of jobs is 5 or less.
>
> I do not think the network to be at caused, as everything is cabled and
> the link is healthy and fast.

I encounter less "connect*: Connection timed out" after setting -M5, but 
it still sometimes happens:

> /gnu/store/vifx9ajaa65624f412bqqcdk4k83v461-flite-2.2-checkout 
> vervangen...
> guix substitute: waarschuwing: tijdens het binnenhalen van 
> https://ci.guix.gnu.org/nar/lzip/vifx9ajaa65624f412bqqcdk4k83v461-flite-2.2-checkout: 
> de server is een beetje traag
> guix substitute: waarschuwing: probeer ‘--no-substitutes’ als het 
> probleem hardnekkig is
> guix substitute: fout: connect*: Verbinding is verlopen
> vervanging van 
> /gnu/store/vifx9ajaa65624f412bqqcdk4k83v461-flite-2.2-checkout mislukt
> guix build: fout: beschadigde invoer tijdens het terugplaatsen van het 
> archief uit #<closed: file 7efc38785e00>

Greetings,
Maxime.


[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 929 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 236 bytes --]