From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp1.migadu.com ([2001:41d0:403:58f0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms8.migadu.com with LMTPS id mOPkOsaqBmYx5gAA62LTzQ:P1 (envelope-from ) for ; Fri, 29 Mar 2024 12:49:27 +0100 Received: from aspmx1.migadu.com ([2001:41d0:403:58f0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp1.migadu.com with LMTPS id mOPkOsaqBmYx5gAA62LTzQ (envelope-from ) for ; Fri, 29 Mar 2024 12:49:27 +0100 X-Envelope-To: larch@yhetil.org Authentication-Results: aspmx1.migadu.com; dkim=none; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org"; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1711712966; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post; bh=RTGDbJnhS2L3rbQ4XFAtAsAQITcyUdB7IANfOYpwTnE=; b=mKM/UoyXgmiq7nvFGs/Y361r/EPCxpIss7LixvOc+5Hgmbe0AjZkH1bpX8QqWSbNJcIrA/ WZPAD8MroGOB6f9jw0Uw62LMhTIizRohZqOqbsNMsKo9pUyztfNOvrUOP27hWNMLRXRHYP ENwtrJOhWKEdODboEhdNZEsdDIMkS/SCqFYq6wqUFitskt0HWZeBkOYHYkL6NL8+EPwTNk CEen9yR+xzRZ7k7G9l97gVX4LXWZgAxXmfJ2N0QZHOXAcY4Y/hdJoX9ic+nRIeVgQHJ9o/ mET+s3kNl0en+dsp1/ujAxk9+pQNu6TpXB2nnUB1Nojesjw/AHaVVRrCoaIDUw== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=none; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org"; dmarc=none ARC-Seal: i=1; s=key1; d=yhetil.org; t=1711712966; a=rsa-sha256; cv=none; b=F3aiNNYFEFssSpqzjN4NoU+AZHGRgUBlV4r4/zaajvI7w6UArUxJk8Oo5nCCwNc2JRZRuC piY5sq89IrLk6oszMw5F4Adntdu1fD1qZVceBTNXghsPyIdvDIm/kpG5i0Ujf6hFf8Ep8u Qlz5TfK0cG3h+zp9TKz9glK/HKf4q4zXtx1IYU8OSkpCaUE1pt6Xf7kDgGoGkQkvhIEvs8 SbTFFW+gOBzgLOhxA/JDg+jud/UJ9DcfmD9athI3ECEtBjuM77piVFq61CdTgCnPWScyT9 j4IWeKDtvB1QDxe1mbghFDI2DFXNQIPgY8T7Mk8MWf8Z6elJpwX+N3MXAuQNRQ== Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 6C3225BA77 for ; Fri, 29 Mar 2024 12:49:26 +0100 (CET) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rqAjA-0002SW-Tw; Fri, 29 Mar 2024 07:48:57 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rqAj8-0002S6-NB for guix-devel@gnu.org; Fri, 29 Mar 2024 07:48:54 -0400 Received: from mira.cbaines.net ([212.71.252.8]) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rqAj6-0006MX-5L; Fri, 29 Mar 2024 07:48:53 -0400 Received: from localhost (unknown [212.132.255.10]) by mira.cbaines.net (Postfix) with ESMTPSA id C4BA427BBE2; Fri, 29 Mar 2024 11:48:48 +0000 (GMT) Received: from felis (localhost.lan [127.0.0.1]) by localhost (OpenSMTPD) with ESMTP id 0e908018; Fri, 29 Mar 2024 11:48:47 +0000 (UTC) References: <877chnfw81.fsf@cbaines.net> <87sf09gwy6.fsf@gnu.org> User-agent: mu4e 1.10.8; emacs 29.1 From: Christopher Baines To: Ludovic =?utf-8?Q?Court=C3=A8s?= Cc: guix-devel@gnu.org Subject: Re: March update on bordeaux.guix.gnu.org Date: Fri, 29 Mar 2024 10:53:27 +0000 In-reply-to: <87sf09gwy6.fsf@gnu.org> Message-ID: <87a5mhs1gi.fsf@cbaines.net> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha512; protocol="application/pgp-signature" Received-SPF: pass client-ip=212.71.252.8; envelope-from=mail@cbaines.net; helo=mira.cbaines.net X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: guix-devel-bounces+larch=yhetil.org@gnu.org X-Migadu-Flow: FLOW_IN X-Migadu-Country: US X-Migadu-Queue-Id: 6C3225BA77 X-Spam-Score: -6.95 X-Migadu-Spam-Score: -6.95 X-Migadu-Scanner: mx11.migadu.com X-TUID: AFECI2tN7feM --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Ludovic Court=C3=A8s writes: > Hi! > > Christopher Baines skribis: > >> Related to this, I've added options to the nar-herder to help change the >> TTL being used, and reduced the TTL for bordeaux.guix.gnu.org to 10 >> minutes (from 180 days) [4]. This will at least mean that in the future, >> the nar-herder on bordeaux will be able to delete zstd compressed nars >> that it's generated more quickly. > > It=E2=80=99s not 10mn right now: > > $ wget --debug -qO/dev/null https://bordeaux.guix.gnu.org/yr39rh6wihd1wv6= gzf7w4w687dwzf3vb.narinfo 2>&1 |grep Cache > Cache-Control: max-age=3D15502374 > > > Or maybe that=E2=80=99s just for newly created nars? The max-age of that narinfo is currently based on the scheduled removal of the zstd compressed nar, which is going to happen quite far in the future. I did think of a number of ways to approach this, and I'm not sure I've settled on the right one yet. Maybe the TTL should be capped at 600, and then drop to 0 as the time to remove the zstd nar approaches? This narinfo for example currently has a max-age of 600: https://bordeaux.guix.gnu.org/ganfjbgy75r31bwzgddpnpswwjrrffvj.narinfo > But then again, that=E2=80=99s the advertised TTL; the real TTL is still > infinite, right? As you probably know, the situation is more complex. The problems caused when the nar-herder started removing zstd compressed nars shows the difference between retention of the nar in some form, and whether a cached narinfo response can be considered fresh or stale. Users might also not notice the availability of zstd nars if they cache responses forever, since currently there will be a lag between the nar becoming available, and a zstd compression being created (although we could generate zstd compressed nars for everything). As described below, I also do want to start removing some nars, and that requires not having an infinite TTL. >> I'm really unsure about the need/usefulness of narinfo caching in >> general, the cost of storing all these narinfos locally is quite high I >> think and I don't really know why it's done. > > It=E2=80=99s a cache. It=E2=80=99s useful to have this cache because in = =E2=80=9Ctypical=E2=80=9D Guix > usage you=E2=80=99re likely to ask repeatedly for the same substitutes. > > Regarding the cost, 3f5e14182931f123c10513a3e1e2abaebfb52279 made things > more reasonable by putting a higher bound on narinfo retention. On my > laptop, I have: > > $ ls -lrt /var/guix/substitute/cache/4refhwxbjmeua2kwg2nmzhv4dg4d3dorpjef= q7kiciw2pfhaf26a/ |wc -l > 11549 > $ du -h /var/guix/substitute/cache/4refhwxbjmeua2kwg2nmzhv4dg4d3dorpjefq7= kiciw2pfhaf26a/=20 > 50M /var/guix/substitute/cache/4refhwxbjmeua2kwg2nmzhv4dg4d3dorpjefq7= kiciw2pfhaf26a/ > > Maybe that=E2=80=99s still excessive and we could further reduce the maxi= mum > caching time. Having played around with this a bit (e.g. hacking guix weather not to cache), I'm a bit sceptical. Given maintaining the cache takes time that could be spent doing network I/O, and does potentially slow disk I/O, I think it would be interesting to try and work out in what situations the cache speeds things up overall, and in what situations it slows things down overall.. >> 6: https://lists.gnu.org/archive/html/guix-devel/2023-05/msg00290.html > > BTW, should we document this mirror somewhere (and also ensure that Guix > Foundation pays the bills), or do you view it more as an experiment for > now? If the project does want to provide mirrors, I think that would be great. From this experiment, I think we have some evidence that there are people using Guix outside of Europe, and in some cases they struggle with the European based infrastructure. It also seems like these mirrors do help, and the monetary cost isn't too high in my view. I think we should probably wait until the project starts managing them before documenting/advertising them more widely though. >> Apart from that, the main thing on my mind for the next year regarding >> bordeaux substitutes specifically is storage space. We're at 90% >> capacity on hatysa (one of the two machines storing all the nars) so >> this will need looking at shortly. I'd also like to finally get removing >> nars that don't relate to the guix master branch happening, as that >> should free up a little bit of space at least. > > Nice (the difficulty, I guess, is that some substitutes that we not > initially for =E2=80=98master=E2=80=99 eventually get used on =E2=80=98ma= ster=E2=80=99). Yep, my plan is to wait some long amount of time (say 6 months) before scheduling things for removal, to check that they haven't started being used by the master branch in this time. We could also add other criteria as well, like tracking which nars are generated by fixed output derivations and never removing them. > On this issue, I think we should learn from fellow NixOS hackers. They > kept substitutes for ~20 years I think and are now in a difficult > position because they cannot afford, financially, to keep that. So one > of the solutions envisioned was to figure out which of these millions of > substitutes were =E2=80=9Cimportant=E2=80=9D (for instance, source code),= which turns > out to be very hard if you don=E2=80=99t have that info already at hand. > > https://discourse.nixos.org/t/nixos-s3-long-term-resolution-phase-1/364= 93 > > Do you think the Data Service or another source of info would let us > make such decisions? > > If we take it to the extreme, we could have a sophisticated retention > policy like: drop all fixed-output derivations known to be available > from disarchive.guix + SWH, drop substitutes for packages that have less > than 100 dependents, etc. I think the Data Service (specifically data.guix.gnu.org) might be really helpful here, as it speeds up being able to work out what a nar or derivation relates to. Additionally, the nar-herder can tag narinfos (associate key=3Dvalue pairs with them), and that's intended to help you manage the nars. So we should probably start tagging the nars with potentially useful information now, so that we can use that data later to make desicions. We're storing 17.5TiB of nars currently, and this increases linearly, so it would be good to understand how this can be broken down. The nar-herder should help here as well, as providing you can download the 11G database, that should contain all the information you need to start digging in to this. wget https://bordeaux.guix.gnu.org/latest-database-dump -O bordeaux.db Thanks, Chris --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQKlBAEBCgCPFiEEPonu50WOcg2XVOCyXiijOwuE9XcFAmYGqp1fFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNF ODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcRHG1haWxAY2Jh aW5lcy5uZXQACgkQXiijOwuE9XflbQ//fnneey0YelncxJIzAp8cTIXnkDdhZf/n fgaQHVBJK78DgZtgotQabD25vBMd87wgXTiziLt+ReJHJStYNeurqmxPx/qLcIWO oPz6kc/hXyqS+zeXsKjxcMXfeWJ75GFZzs+ydJZ7EXa6e4V9qBhOc36ePmWJp9fs xhXwCghkj8L4q+iOgZl4sRc3pS2wgmLqxFYl1JyuE++UFHaolGiHGAQ33tqkHJFe XAYm9ZJvUpC9oT8Ftt5yFQckK12C/wi8b3Bwm7843WDnIrGW6MshvMn72fLbMxvH vdBZWi/qQRYGR//WqzLge8WFq1X+IgGejfkpGRVqOLXyd3UUteBoXAHVav/3tAt2 hGEtck+6c75BTQNRPhpevkX13k1OZPhz8/myMqpN5FuecHZNj3QiUUQkjx/cKxkR dQWTPbrNA97N0HY/tyweKoxA1+YSGFA9p9nS0Ph/wP45dDTjAuPthWV1+DVjmmGp Mry0PXh4pDOB1+ohSXkkZ6oVTZcwXDaEBit4NA5LKg5HaqrEIClWS3eEKJI+y/I9 HDFR3LPnhOYAa9jXZG02tVdibEboKrPX7RWC5oZeVDkhqbHfhZefyEomOgCMlc7t gifI1FFFL7oSpMM0zIZdmINUa5m7XUoeSI2OyXCHBTYovm6dCZ1cHi0D9iSOGC4c H330IvsgQZw= =7/5P -----END PGP SIGNATURE----- --=-=-=--