From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp2 ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id uNwyH50Mu2HaOwAAgWs5BA (envelope-from ) for ; Thu, 16 Dec 2021 10:53:33 +0100 Received: from aspmx1.migadu.com ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp2 with LMTPS id 2JTmGp0Mu2HAbgAAB5/wlQ (envelope-from ) for ; Thu, 16 Dec 2021 09:53:33 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 3F17A20747 for ; Thu, 16 Dec 2021 10:53:33 +0100 (CET) Received: from localhost ([::1]:51022 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mxnS8-00025g-A0 for larch@yhetil.org; Thu, 16 Dec 2021 04:53:32 -0500 Received: from eggs.gnu.org ([209.51.188.92]:51650) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mxnLn-0006dN-Sc; Thu, 16 Dec 2021 04:46:59 -0500 Received: from [2001:470:142:3::e] (port=56768 helo=fencepost.gnu.org) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mxnLn-0001yK-J8; Thu, 16 Dec 2021 04:46:59 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:Date:Subject:To:From:in-reply-to: references; bh=Uzw/VblEO0iqqa4FgwwO6qN5Cwxm03P/MHMxFvUHi3E=; b=KeNcfiUfRzHZIs 22pdQu/d6iDxKa/JD4kMOmtDvePaphw8NHCm+BYPdqVrlY7uDtGmxIZQJIKeNE0tU+0eR4AFlwi5V ANNFQqC7T6p+SembTCgPi6ysnzSUe7ezpLmbbJ+ecC3u46ntC69AmpbP9w7S1i00EUkEARX02r5vG zEEtWSZGx/PoujaNOD2SIPR1HZYvphvu07KnVkNPFdh+HvWEqroTxPr8nOFOOsYsKr0qx9PxoDzpK Sc0oq41WmqOX0TmAkKRMcSqy2zGHbz+2liKkdZyCxbZ+SEhtyfmmHc3WqyX4dViPI8I+4/RjV+OlO ZiezgLUmIaLFLf9l7+tA==; Received: from 91-160-117-201.subs.proxad.net ([91.160.117.201]:51558 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mxnLn-0007QW-HY; Thu, 16 Dec 2021 04:46:59 -0500 From: =?utf-8?Q?Ludovic_Court=C3=A8s?= To: Guix Devel Subject: Solstice infrastructure hackathon X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: 25 Frimaire an 230 de la =?utf-8?Q?R=C3=A9volution?= X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-pc-linux-gnu Date: Thu, 16 Dec 2021 10:46:56 +0100 Message-ID: <87v8zorien.fsf@inria.fr> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: guix-sysadmin Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: "Guix-devel" X-Migadu-Flow: FLOW_IN X-Migadu-Country: US ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1639648413; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:list-id:list-help: list-unsubscribe:list-subscribe:list-post:dkim-signature; bh=Uzw/VblEO0iqqa4FgwwO6qN5Cwxm03P/MHMxFvUHi3E=; b=YC+dpzTmLerNprpstnmA0mY7XClrgtVDyEnKBmP6fJWOWkzJLOcQH55ibFtG499Qm9RVFT 2sugNzXkUJq6PQyP2mAlt8VOgL4DXw1lWxi/byr2hHtp2uV+yZnX6O7Utik9zGG/HJfhJs 4SPT1gGHOU/eUK9FGTai5ZVwumIkX8ehsqiN1qvpFeW0ToNHEwgQp+pQTkRzUyeFv9t0DT 8IFoky+kNT3iWdrLGRDnZQJ/MeAqu3tWnsESqK9FmS6DVgWfMJGyPsLVIU7dAL+XmNev68 HPNpN468UUX8zdMGhkVQTBKcsBH1edHbi4+g5egC+zigplRDBeFe691Vh5gDfA== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1639648413; a=rsa-sha256; cv=none; b=SPWlxYas4HGLwIUCuX6Xnee+gtpZyHwfF7HYx6pynvO4IKzT9U40RA7O6u6El7XlNKL1U4 IWsyUoyBvjJ/ao7d7wz8Jz96DGlKo5d3zHLeqIJOjd8AabD0AJ/2juf3u1OB/2PZ3Ee04J 3HeTaU7tskZkjH7OtwO7f0DfhzZmRvgx7jG1ScWA5NS8TffkfFQlB2x+WY/McBXrfzKz/f 6ZhKo8haPrP1v3G8reFI6HUMLOxhU6nzFyWnH11/smv9VFCFdzuSktMjcVrRbgD0PXXLf5 1N4pyYB3eZ56SQVF3OIrd7B1FO1Po9VGAlvVzMIwShd2K7+EnyOIGjpIzimKKA== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=pass header.d=gnu.org header.s=fencepost-gnu-org header.b=KeNcfiUf; dmarc=pass (policy=none) header.from=gnu.org; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org" X-Migadu-Spam-Score: -7.19 Authentication-Results: aspmx1.migadu.com; dkim=pass header.d=gnu.org header.s=fencepost-gnu-org header.b=KeNcfiUf; dmarc=pass (policy=none) header.from=gnu.org; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org" X-Migadu-Queue-Id: 3F17A20747 X-Spam-Score: -7.19 X-Migadu-Scanner: scn0.migadu.com X-TUID: fHVva2PSXN1Z Hello Guix! This week, the node behind {ci,issues,disarchive}.guix.gnu.org and guix.gnu.org was down twice for a few hours=E2=80=94nothing terrible in the= end, but it reminded us that, even though Guix doesn=E2=80=99t rely on any parti= cular machine, we can definitely feel the inconvenience when it=E2=80=99s down. We were unlucky enough that it happened days after the other build farm, bordeaux.guix.gnu.org, ran out of disk space and had its CI stopped, right before the big merge=E2=80=94so it doesn=E2=80=99t have substitutes f= or current master. While discussing this on IRC the other day, we thought that perhaps it was time to have an infrastructure hackathon. How about Tuesday, Dec. 21st? (Probably with a followup in January.) Here are tasks that were brought up: =E2=80=A2 Set up a backup server for berlin.guix.gnu.org, the head node o= f the ci.guix.gnu.org, possibly moving some services such as the web site there. =E2=80=A2 Add DNS redundancy for guix.gnu.org so it can point to one of t= wo hosts (need to figure out certbot challenges so both machines can update their certificates). =E2=80=A2 Set up status.guix.gnu.org with sysadmin status updates (possib= ly using Prometheus?). =E2=80=A2 Come up with a plan to add disks to the RAID array on bayfront,= the head node of bordeaux.guix.gnu.org. =E2=80=A2 Work on a plan to back up the Disarchive database currently on berlin.guix. =E2=80=A2 Work on a plan to mirror nars from ci.guix and bordeaux.guix, u= sing plain rsync or . =E2=80=A2 Have a documented procedure to set up substitute mirrors, such = as the one in .cn (I can=E2=80=99t find the URL), ideally with plain rsync access. Am I forgetting something? Some of these tasks require root or physical access for the final steps, but most of them are about (1) coming up with a plan, and (2) adjusting the system configuration at . Anyone with sysadmin experience to share and Guix System knowledge can join! We=E2=80=99ll communicate over #guix on irc.libera.chat. Who=E2=80=99s in? :-) Cheers, Ludo=E2=80=99.