From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp11.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id QNahKRkd/WHiHQEAgWs5BA (envelope-from ) for ; Fri, 04 Feb 2022 13:33:29 +0100 Received: from aspmx1.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp11.migadu.com with LMTPS id aMUPJxkd/WFAjgAA9RJhRA (envelope-from ) for ; Fri, 04 Feb 2022 13:33:29 +0100 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 58E3932D6C for ; Fri, 4 Feb 2022 13:33:29 +0100 (CET) Received: from localhost ([::1]:53050 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nFxmK-0000h1-G9 for larch@yhetil.org; Fri, 04 Feb 2022 07:33:28 -0500 Received: from eggs.gnu.org ([209.51.188.92]:43260) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nFxlo-0000fs-B2 for guix-devel@gnu.org; Fri, 04 Feb 2022 07:32:58 -0500 Received: from [2a01:7e00:e000:2f8:fd4d:b5c7:13fb:3d27] (port=59009 helo=mira.cbaines.net) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nFxlh-0004Ac-OO for guix-devel@gnu.org; Fri, 04 Feb 2022 07:32:53 -0500 Received: from localhost (unknown [IPv6:2a02:8010:68c1:0:8ac0:b4c7:f5c8:7caa]) by mira.cbaines.net (Postfix) with ESMTPSA id 4D0F427BBE9 for ; Fri, 4 Feb 2022 12:32:37 +0000 (GMT) Received: from capella (localhost [127.0.0.1]) by localhost (OpenSMTPD) with ESMTP id e75a4e58 for ; Fri, 4 Feb 2022 12:32:36 +0000 (UTC) User-agent: mu4e 1.6.10; emacs 27.2 From: Christopher Baines To: guix-devel@gnu.org Subject: nar-herder design retrospective Date: Fri, 04 Feb 2022 12:00:38 +0000 Message-ID: <87o83mg6cd.fsf@cbaines.net> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha512; protocol="application/pgp-signature" X-Host-Lookup-Failed: Reverse DNS lookup failed for 2a01:7e00:e000:2f8:fd4d:b5c7:13fb:3d27 (failed) Received-SPF: pass client-ip=2a01:7e00:e000:2f8:fd4d:b5c7:13fb:3d27; envelope-from=mail@cbaines.net; helo=mira.cbaines.net X-Spam_score_int: -10 X-Spam_score: -1.1 X-Spam_bar: - X-Spam_report: (-1.1 / 5.0 requ) BAYES_00=-1.9, RDNS_NONE=0.793, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: "Guix-devel" X-Migadu-Flow: FLOW_IN X-Migadu-Country: US ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1643978009; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version: content-type:content-type:list-id:list-help:list-unsubscribe: list-subscribe:list-post; bh=e1U0TpdRbdQrhmFOf2Zd6u6P9cS0Aj5iPKh4kGExBxE=; b=r9PJlHF954hMT7GIaoBp7vJZWpn34oVlkeaGZmc9pDHnKwY9mY/YfFXlb9olwzscWqFEdU GQAh7sbFDyU0II05yfYR+xFni+AulNKjzRXbBbFjdVIzegVHmlMLq1qM0q8i/q/cHNt61u LoJUS480JsZFOzgQ4uHBrJBNp0CGJ7BZphN6+RjvhNm7EjEPbmQXQnDxqBH8NOJy7H0IYC fj6YpFzZAbE0uABc3ucVx3bPVQZthmcEqPq+NRkG1ZtVDV8N8CSeKuxsOgKbDXkFJMSraE wGh0A+mhKfR+MncTX0YF7UZ8WgZT0TLkHOKeRrNS0QI4vixlpfmX9VY2MsDWhg== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1643978009; a=rsa-sha256; cv=none; b=hI8W8dQpmYCScybRizEfeYMK6MEk/0PT/HGCfOi1Yf6NSlQ6LSD0VZxltjw1FHicRlXh2o hc5s/cXYgCGRJlHHLj4vPIj7ZqZ45v9xbkQChBC3ICw2jLeJDdiQQNp/YhGGI/QJqb2679 74JCUIeDq341FCKsCZ++u3VQK8wd4XivCahmcY91MFpuvzUjX0zmgZqM+KrviM6nxxNgM7 mbnkX3vdOAvh1SuMQmXl9HvgBlsrcd2UG0nW59xdqOTnmOGz3tJ1rWzIsANQJvBcCg26+l e+3Kfluj52ChNQK8t97slBCEjHFhz/WBvHSYjgKFhmeNb4zDl0cVVBj+Eg7ewg== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org" X-Migadu-Spam-Score: -5.53 Authentication-Results: aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org" X-Migadu-Queue-Id: 58E3932D6C X-Spam-Score: -5.53 X-Migadu-Scanner: scn0.migadu.com X-TUID: HBYEFGc4rFrQ --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Hi! I rushed the nar-herder [1] in to existence back in December, to address the buildup of nars on bayfront by moving the nars to another machine with more space to store them. 1: https://git.cbaines.net/guix/nar-herder/about/ This was something I was planning for a while though, I sent an email to guix-devel outlining some points of the design and aims around a year ago [2]. 2: https://lists.gnu.org/archive/html/guix-devel/2021-02/msg00104.html The nar-herder is currently in a stable state. It's packaged for guix and there's a service to use it. There are probably some bugs, but I think I've fixed the important initial ones. I covered some information about the deployment of the nar-herder in this email back in December [3] and that led to some really good replies, I'll copy/paste some of the important bits below. 3: https://lists.gnu.org/archive/html/guix-devel/2021-12/msg00140.html https://lists.gnu.org/archive/html/guix-devel/2021-12/msg00196.html: > Regarding nar-herder, I think it=E2=80=99d be nice to have a solution to > mirroring in Guix proper, developed similarly to other components, > because it could be a fairly central tool. > Usually I=E2=80=99m the one asking for blog posts :-), but I=E2=80=99d re= ally like us as > a project to collectively engage on the topic before we publicize this > specific approach. https://lists.gnu.org/archive/html/guix-devel/2021-12/msg00201.html: > Why not extend =E2=80=9Cguix archive=E2=80=9D? >=20 > However, without all the details so my remark is totally naive, I miss > what =E2=80=9Cnar-herder=E2=80=9D is doing that =E2=80=9Cguix archive=E2= =80=9D+rsync+=E2=80=9Cguix publish=E2=80=9D is > not doing =E2=80=93 other said I miss why another SQL database is require= d to > serve stuff from one place to another. I have read README but I did not > get the point. https://lists.gnu.org/archive/html/guix-devel/2021-12/msg00204.html: > I'm quite interested in learning more and potentially trying out the > nar-herder! Some thoughts that I'd like to add to the design space: >=20 > I think it would be great if one of the pastures to which we herd the > nars would be a free and open source software mirror site. In my > experience, these are usually some static web hosting in front of a > large disk with a place to run scripts to sync the content. A database > server may not be available. I'd like to support this use case because I > think it is a great way to build bridges to the communities who run or > gather around these mirrors. >=20 > I'd also like the ability fetch nars directly from the local-to-me > mirror rather than having them be proxied through a far way server. >=20 > One of the things that I really like and find empowering about Guix is > that the developer/system administration tools are as available, easy > to use, and convenient as the every day tooling. To the extent > possible, I think that we should strive to make our syncing/mirroring > solution practical to run for local, small setups, and not require > project-scale infrastructure or coordination between many programs > that are not captured in a Guix service. So, currently the nar-herder can be used to move nars between machines, which is what I wanted the initial implementation to be capable of. This functionality should be sufficient for operating mirrors, although I'm not aware of any being setup yet. I'd also like to be able to get metrics about nar requests, but this isn't supported yet. I'm all for having a solution to mirroring in Guix itself, although I don't have a plan for this. Maybe the nar-herder could just be moved in to Guix, maybe with a different name? Any other ideas? I think moving the nar-herder repository on to Savannah is probably a good thing to do regardless. Thanks, Chris --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQKlBAEBCgCPFiEEPonu50WOcg2XVOCyXiijOwuE9XcFAmH9HOJfFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNF ODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcRHG1haWxAY2Jh aW5lcy5uZXQACgkQXiijOwuE9XcJxA//ZbUVWfwg+V2u7XRg0Kowj3B70CQ7Sy/s u2eDlUhiOlDD0NYh048AtFU1iLkrjg+Cd/GpB8Wzr6DAOSrDoI98oCQea1nm/Nxx RlmDtaCDsz9j5FpeKD+VwEFiokCY6OlrNAurkrYH7qlusPyRxM5iloLgHL/CzMZz z+NJK+871NbQLZYEmZ6spgMqjCj6bO++hDgTU67ViUx5XVa1HQJu0+gbX+2WqQqV CVfaRKDld09u5yaI1bNePM8bZ6U7SaNkVVOJit9Q5h5jW/oEDni+LjqZyFFHYkdR QHRpgZKsqvBiipqihMmeTnrfGa8XB0Uwe5HrR2WG9SE9JFYux0l4yHZZ3wKjiWhk e2YNCU03KOLwM6YMUfuFEtcYxpKB2Q+YjDuu47xejXoApob5acPziE/J3T2zrRXl M0/Vja9D8o+HCAYNDcHQEdW35Sd2MVTG7mfAN1HA5lRDxyoXc0WdyIJ9Hbg86tn0 83Opk01dWGwN1GJYeMrNvRLUXDRDgDa02Pki8YfwNJfmrL/Qdz7iU+Rut9iU5Lfs 3g5WFcimMnMqmQQ3FI1l29V/6i4/IYLD6f0i3ZcIb3jdNuuAbwljbf9ux77HF+8p rbXRpKTs3+DPCmO2q+5uysSPOznvsdGhKuNQXawbU6VnkZg+8NkgyJkIlYqkhqmy RTB7CMqBSQA= =vQbf -----END PGP SIGNATURE----- --=-=-=--