From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp11.migadu.com ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id cDs7Cvl+AmL8LQAAgWs5BA (envelope-from ) for ; Tue, 08 Feb 2022 15:32:25 +0100 Received: from aspmx1.migadu.com ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp11.migadu.com with LMTPS id mLFUB/l+AmIHjwAA9RJhRA (envelope-from ) for ; Tue, 08 Feb 2022 15:32:25 +0100 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 74E061F1FB for ; Tue, 8 Feb 2022 15:32:22 +0100 (CET) Received: from localhost ([::1]:37504 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nHRXZ-00089r-06 for larch@yhetil.org; Tue, 08 Feb 2022 09:32:21 -0500 Received: from eggs.gnu.org ([209.51.188.92]:52656) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nHQ2c-0004ZW-Hm for bug-guix@gnu.org; Tue, 08 Feb 2022 07:56:18 -0500 Received: from debbugs.gnu.org ([209.51.188.43]:51897) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1nHQ2M-0006jD-3J for bug-guix@gnu.org; Tue, 08 Feb 2022 07:56:17 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1nHQ2L-0002Vh-Tc for bug-guix@gnu.org; Tue, 08 Feb 2022 07:56:01 -0500 X-Loop: help-debbugs@gnu.org Subject: bug#53463: ci.guix.gnu.org not building the 'guix' job Resent-From: Ricardo Wurmus Original-Sender: "Debbugs-submit" Resent-CC: bug-guix@gnu.org Resent-Date: Tue, 08 Feb 2022 12:56:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 53463 X-GNU-PR-Package: guix X-GNU-PR-Keywords: To: Ludovic =?UTF-8?Q?Court=C3=A8s?= Received: via spool by 53463-submit@debbugs.gnu.org id=B53463.16443249549635 (code B ref 53463); Tue, 08 Feb 2022 12:56:01 +0000 Received: (at 53463) by debbugs.gnu.org; 8 Feb 2022 12:55:54 +0000 Received: from localhost ([127.0.0.1]:45794 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nHQ2E-0002VL-2z for submit@debbugs.gnu.org; Tue, 08 Feb 2022 07:55:54 -0500 Received: from sender4-of-o51.zoho.com ([136.143.188.51]:21182) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nHQ2C-0002VC-Rv for 53463@debbugs.gnu.org; Tue, 08 Feb 2022 07:55:53 -0500 ARC-Seal: i=1; a=rsa-sha256; t=1644324944; cv=none; d=zohomail.com; s=zohoarc; b=WMdIji/82VVd5HwtaGY6uh/HHU9xhIY91mPZhNnIda7iz4Ve+BrPWnqh8PMa0WHRIeeFTWmZuUfjDcy8/eWM2VocC2/o6NVjM0kQOgtS6NOrJm8AH+WaBlJdGzMLbnJvEfRz0jTibw69KKdG6sW6y/i9lYV3tsQBCmm24GgMCmY= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1644324944; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:MIME-Version:Message-ID:References:Subject:To; bh=K1IWJhjPN7Ml8sDC9T6p3HFftLtesOYkAyeyNuhMvSI=; b=FzlP+grqkaWZwsSNv7niM25LyL3TjDibFpjjnIXZsMa5lk5uawucSiGgfzoplGQiE9Tgn3mUT5P+iA7evG3bhz87Gg8BoNNvKBQ3ZuB1R/1Edr1+1dVT+zZ+Jn4pk+JnMecNZNF12kJgYrbavIB0SUVAQkI2Pn0gXEy4Ko9GSZs= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass header.i=elephly.net; spf=pass smtp.mailfrom=rekado@elephly.net; dmarc=pass header.from= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1644324944; s=zoho; d=elephly.net; i=rekado@elephly.net; h=References:From:To:Cc:Subject:Date:In-reply-to:Message-ID:MIME-Version:Content-Type:Content-Transfer-Encoding; bh=K1IWJhjPN7Ml8sDC9T6p3HFftLtesOYkAyeyNuhMvSI=; b=GANElifdjM514f7c2C4WTFGuvXVA6J4QbrcXdUfpH9gwFUqoMiB0ynLbeJkhaipS vtAi1ZIGSOTbPGflHVnGtRRW8IcMVyCMogIDCiQJI7ofmillioGQf8DW/B0qB462dXB 4lz7AFJ0Tt7q1ZDCIoPTRD+JPc/roH/N5LhHPUsI= Received: from localhost (p54ad4e3e.dip0.t-ipconnect.de [84.173.78.62]) by mx.zohomail.com with SMTPS id 1644324927003948.9265188630875; Tue, 8 Feb 2022 04:55:27 -0800 (PST) References: <87leyt2jsr.fsf@gnu.org> <87sfsznh2z.fsf@gnu.org> <875ypv3qjo.fsf@gnu.org> <87zgn1aca4.fsf@gnu.org> User-agent: mu4e 1.6.10; emacs 27.2 From: Ricardo Wurmus Date: Tue, 08 Feb 2022 13:52:21 +0100 In-reply-to: <87zgn1aca4.fsf@gnu.org> X-URL: https://elephly.net X-PGP-Key: https://elephly.net/rekado.pubkey X-PGP-Fingerprint: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAC Message-ID: <87ee4dzfeu.fsf@elephly.net> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-ZohoMailClient: External X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-guix@gnu.org List-Id: Bug reports for GNU Guix List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Mathieu Othacehe , 53463@debbugs.gnu.org Errors-To: bug-guix-bounces+larch=yhetil.org@gnu.org Sender: "bug-Guix" X-Migadu-Flow: FLOW_IN X-Migadu-Country: US ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1644330744; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:resent-cc: resent-from:resent-sender:resent-message-id:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post:dkim-signature; bh=K1IWJhjPN7Ml8sDC9T6p3HFftLtesOYkAyeyNuhMvSI=; b=PNvXZI8uYLVVb9iDrbKCgySe3K/uVEyXzuqN7B/fcmCDXN+70L27r1M1P4w5JBCQ8AWoqe ZjzD60wmX5x4Go07P1YzIJV5/EwWEemIXDp1pWIV5ui6mi19Tx8+6G86HIDlKXoS3FRzCa lk8z+iwhZE3tDYV4sle0B1T2dX1se91IKuQN9joPhnvhBYJnoat2ybnEtEfGI2z4PHb5nG aeHfhrOMOmEoBWA/AvA5HMC5HxWfPxCwAIq+5VpyocNqGSlx7y9b1jyJfXG7EpW6dhIPbC DcOLERskff9oPdhsPzPRaU59pXg58ZX84WZHicHVj6nb9rq9NJpeDNVWur5MHg== ARC-Seal: i=2; s=key1; d=yhetil.org; t=1644330744; a=rsa-sha256; cv=fail; b=iEh5P8YG5jzclNY5oLiArNE6myNP+VDIxDRpDJW1L3njv4J9HgrBBskhYJTMMYfPZfbaUU vbIOquiYEX4IGBAUjTgdNyvlk+tLXCrarWIB0BIh6VRe172W4FUcBY4OJKDf45hhzHvsui NLUM98Sae9XHZgZ0nB3JCqjm1uGU7XK4Eu+tXoG+U15RZJnO0erYIr7NbPw7tUeUv38QtR QLaitq9fwIUys7N0DK1NmNaQnu1vOB3cZnbJ/H5dMHAzKdfUjFRXNmhBu+1bCoXc4v3kWH Wyaakhu4cFpO+oNGiJ+AJtJP769EY+UBOKtVRGTbPWbgQ+B/7OeEMDvX4S1Ybg== ARC-Authentication-Results: i=2; aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=elephly.net header.s=zoho header.b=GANElifd; arc=reject ("signature check failed: fail, {[1] = sig:zohomail.com:reject}"); dmarc=none; spf=pass (aspmx1.migadu.com: domain of "bug-guix-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="bug-guix-bounces+larch=yhetil.org@gnu.org" X-Migadu-Spam-Score: -1.13 Authentication-Results: aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=elephly.net header.s=zoho header.b=GANElifd; arc=reject ("signature check failed: fail, {[1] = sig:zohomail.com:reject}"); dmarc=none; spf=pass (aspmx1.migadu.com: domain of "bug-guix-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="bug-guix-bounces+larch=yhetil.org@gnu.org" X-Migadu-Queue-Id: 74E061F1FB X-Spam-Score: -1.13 X-Migadu-Scanner: scn1.migadu.com X-TUID: kKRRErww4k6U Ludovic Court=C3=A8s writes: > Hi, > > Mathieu Othacehe skribis: > >>> Oh! That indicates that it=E2=80=99s failing to offload to one of the >>> =E2=80=98localhost=E2=80=99 build machines specified in /etc/guix/machi= nes.scm. >>> Normally there=E2=80=99s an SSH tunnel set up for those, but I guess it= broke. >>> >>> Perhaps we can update /etc/guix/machines.scm to refer to armhf-linux >>> machines by their WireGuard IP? >> >> Seems like the right thing to do. This bit is also an unstaged change in >> the berlin maintenance repository, we should commit it. Tobias, could >> you have a look :) ? >> >> +(define powerpc64le >> + (list >> + ;; A VM donated/hosted by OSUOSL & administered by nckx. >> + ;; XXX: SSH tunnel via overdrive1: >> + ;; ssh -L 2224:p9.tobias.gr:22 hydra@10.0.0.3 >> + #;(build-machine >> + ;;(name "p9.tobias.gr") >> + (name "localhost") >> + (port 2224) >> + (user "hydra") >> + (systems '("powerpc64le-linux")) >> + (host-key "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIJEbRxJ6WqnNLYEMNDUK= FcdMtyZ9V/6oEfBFSHY8xE6A nckx")))) > > IIRC this machine is now running WireGuard, Tobias? If so, could you > change this to refer to its WireGuard IP and commit it? > >> I also found that other machines were unreachable and commented them: >> >> ;; CPU: 16 ARM Cortex-A72 cores >> ;; RAM: 32 GB >> - (list (build-machine >> + (list #;(build-machine >> ;;kreuzberg >> (name "10.0.0.9") >> (user "hydra") > > Ricardo, could you check what=E2=80=99s wrong with kreuzberg? Oh, the usual=E2=80=A6 --8<---------------cut here---------------start------------->8--- root@kreuzberg ~# guix shell wireguard-tools -- wg interface: wg0 public key: f9WGJTXp8bozJb0KxePjkOclF5pJUy1AomHWJHy80y4=3D private key: (hidden) listening port: 51820 peer: wOIfhHqQ+JQmskRS2qSvNRgZGh33UxFDi8uuSXOltF0=3D endpoint: 141.80.181.40:51820 allowed ips: 10.0.0.1/32 latest handshake: 2 days, 2 hours, 11 minutes, 13 seconds ago transfer: 292.79 MiB received, 6.05 GiB sent --8<---------------cut here---------------end--------------->8--- Whenever the build farm is awfully quiet (e.g. because of GC) the wireguard connection times out. I usually restart the cuirass-remote-worker and everything=E2=80=99s fine again. Today I got some additional SD cards for these machines, so I=E2=80=99m goi= ng to reconfigure them (locally, because of the =E2=80=9Cguix deploy=E2=80=9D bug= ) and then move them to the data centre. Once reconfigured they will keep the wireguard connection alive all by themselves, so no manual intervention is necessary. I didn=E2=80=99t reconfigure them locally because I hoped we would be able = to make time for the =E2=80=9Cguix deploy=E2=80=9D bug, but things turned out = differently. --=20 Ricardo