bug#53463: ci.guix.gnu.org not building the 'guix' job

[Ricardo Wurmus <rekado@elephly.net>]

Ludovic Courtès <ludo@gnu.org> writes:

> Hi,
>
> Mathieu Othacehe <othacehe@gnu.org> skribis:
>
>>> Oh!  That indicates that it’s failing to offload to one of the
>>> ‘localhost’ build machines specified in /etc/guix/machines.scm.
>>> Normally there’s an SSH tunnel set up for those, but I guess it broke.
>>>
>>> Perhaps we can update /etc/guix/machines.scm to refer to armhf-linux
>>> machines by their WireGuard IP?
>>
>> Seems like the right thing to do. This bit is also an unstaged change in
>> the berlin maintenance repository, we should commit it. Tobias, could
>> you have a look :) ?
>>
>> +(define powerpc64le
>> +  (list
>> +   ;; A VM donated/hosted by OSUOSL & administered by nckx.
>> +   ;; XXX: SSH tunnel via overdrive1:
>> +   ;; ssh -L 2224:p9.tobias.gr:22 hydra@10.0.0.3
>> +   #;(build-machine
>> +    ;;(name "p9.tobias.gr")
>> +    (name "localhost")
>> +    (port 2224)
>> +    (user "hydra")
>> +    (systems '("powerpc64le-linux"))
>> +    (host-key "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIJEbRxJ6WqnNLYEMNDUKFcdMtyZ9V/6oEfBFSHY8xE6A nckx"))))
>
> IIRC this machine is now running WireGuard, Tobias?  If so, could you
> change this to refer to its WireGuard IP and commit it?
>
>> I also found that other machines were unreachable and commented them:
>>
>>    ;; CPU: 16 ARM Cortex-A72 cores
>>    ;; RAM: 32 GB
>> -  (list (build-machine
>> +  (list #;(build-machine
>>           ;;kreuzberg
>>           (name "10.0.0.9")
>>           (user "hydra")
>
> Ricardo, could you check what’s wrong with kreuzberg?

Oh, the usual…

--8<---------------cut here---------------start------------->8---
root@kreuzberg ~# guix shell wireguard-tools -- wg
interface: wg0
  public key: f9WGJTXp8bozJb0KxePjkOclF5pJUy1AomHWJHy80y4=
  private key: (hidden)
  listening port: 51820

peer: wOIfhHqQ+JQmskRS2qSvNRgZGh33UxFDi8uuSXOltF0=
  endpoint: 141.80.181.40:51820
  allowed ips: 10.0.0.1/32
  latest handshake: 2 days, 2 hours, 11 minutes, 13 seconds ago
  transfer: 292.79 MiB received, 6.05 GiB sent
--8<---------------cut here---------------end--------------->8---

Whenever the build farm is awfully quiet (e.g. because of GC) the
wireguard connection times out.  I usually restart the
cuirass-remote-worker and everything’s fine again.

Today I got some additional SD cards for these machines, so I’m going to
reconfigure them (locally, because of the “guix deploy” bug) and then
move them to the data centre.  Once reconfigured they will keep the
wireguard connection alive all by themselves, so no manual intervention
is necessary.

I didn’t reconfigure them locally because I hoped we would be able to
make time for the “guix deploy” bug, but things turned out differently.

-- 
Ricardo