bug#36754: SSH connections to hydra-slave{1, 2, 3} fail during builds

["Ludovic Courtès" <ludo@gnu.org>]

Hi Mark,

Mark H Weaver <mhw@netris.org> skribis:

> I wrote earlier:
>> I now believe that these failures are related to the newly added armhf
>> build slaves, and that they have nothing to do with the recent changes
>> to our linux-libre packages.
>
> I should mention that the armhf build slaves are on a private network,
> and I use my public-facing internet server to forward TCP connections to
> them, using the following entries in /etc/inetd.conf:
>
> # TCP-level forwards for SSH connections to build machines for the GNU
> # Guix build farm:
> 7275    stream  tcp     nowait  nobody  /bin/nc /bin/nc -w 10 172.19.189.11 7275
> 7276    stream  tcp     nowait  nobody  /bin/nc /bin/nc -w 10 172.19.189.12 7276
> 7274    stream  tcp     nowait  nobody  /bin/nc /bin/nc -w 10 172.19.189.13 7274
>
> It's possible that this arrangement is somehow part of the problem.
> However, note that nothing has changed here in several years, and it
> worked fine on hydra.gnu.org.  The build slaves were running a *very*
> old version of Guix though.  It seems likely that the new Guile-SSH code
> doesn't cope well with this setup.

I noticed that connections to the machines were unstable (using
OpenSSH’s client).  That is, the connection would eventually “hang”,
apparently several times a day.

Currently we have an SSH tunnel set up on berlin to connect to each of
these machines via overdrive1.guixsd.org.  This setup proved to be
robust in the past (we used it to connect to another build machine), so
I suspect something’s wrong on “your” end of the network.  It’s hard to
tell exactly what, though.

Ideas?

If it’s causing build failures, I’m afraid we’ll have to comment out
those machines from berlin’s machines.scm until we’ve figured it out.

Thanks,
Ludo’.