From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp1.migadu.com ([2001:41d0:403:58f0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms1.migadu.com with LMTPS id sK6MD+lgFWZaxQAA62LTzQ:P1 (envelope-from ) for ; Tue, 09 Apr 2024 17:38:17 +0200 Received: from aspmx1.migadu.com ([2001:41d0:403:58f0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp1.migadu.com with LMTPS id sK6MD+lgFWZaxQAA62LTzQ (envelope-from ) for ; Tue, 09 Apr 2024 17:38:17 +0200 X-Envelope-To: larch@yhetil.org Authentication-Results: aspmx1.migadu.com; dkim=pass header.d=gnu.org header.s=fencepost-gnu-org header.b=D7Zt3XQM; dmarc=pass (policy=none) header.from=gnu.org; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org" ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1712677097; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post:dkim-signature; bh=pK7/9W0EtqTg7uKC7su9q3CfPxgSEJ/Uvvbk3t7GAdQ=; b=JpC75aWMdHYEHot7QlmjtJKjCVsBB3S0TBDOx8cN3ssn5DMbHuLWyxKM9VD3sOA7SdtQEy yBHJkW5fEO/XrPibX58eTTZ4Oz2E/CcwSyS09n04HUaG0YzGptFKH0tA8pLVaSp7gQxVKr cc6HSVtAB27xtzAKzgfJmMYcffOnvIQzs0H+LlxcP/BY2N6L706AihEXg/9CJi53mmGMqd VjD2pLwHYWGyCjMvYDpPZzxsG9kVH1JY6cv7B0uxSgaM5AYHFjW+f6tstKkUQ1BgVlIVOk cm0WSAQy/7GwEkjRAwvcTzq6Y4JsfpWUZBHQgSf8f8jTSdd3d4vMSYN19sDVKg== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=pass header.d=gnu.org header.s=fencepost-gnu-org header.b=D7Zt3XQM; dmarc=pass (policy=none) header.from=gnu.org; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org" ARC-Seal: i=1; s=key1; d=yhetil.org; t=1712677097; a=rsa-sha256; cv=none; b=UINO40E4RGeTMJXxkeqDYSOuIw2kHok5zw8wUWEk5OPWE6rtBuWfVSAajlb92pXw3pdhYf 2kjakS+u6AqcPAPUeYC0zL7+ftjW7J4RUCI8oVr86YV25Esc+muFPiLYm3qngCqCcMpauk 3K3w6qMsRIQFcMj7/QUf1bto9Ts4w6dE/Ccpwq5u6cSPK4BFSLdu2r+CiWxzBQOsQHGa15 eDFW9SAsOM9JDMDA+5Pzi4l6qVe7u8u7XRsPqM6yic0B3nKDfXK8rjYdyBiJE18WegE2Oo OOO5wjSQm1xMdzMENHQwmvZwYgrXxeO93x49MuzA6WSjVQ5WOE2IWpBFn962OQ== Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id AE23E7285A for ; Tue, 09 Apr 2024 17:38:16 +0200 (CEST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ruDXn-0004fQ-SM; Tue, 09 Apr 2024 11:37:55 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1ruDXl-0004fD-K1 for guix-devel@gnu.org; Tue, 09 Apr 2024 11:37:53 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1ruDXl-0005Dq-Ab; Tue, 09 Apr 2024 11:37:53 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:Date:References:In-Reply-To:Subject:To: From; bh=pK7/9W0EtqTg7uKC7su9q3CfPxgSEJ/Uvvbk3t7GAdQ=; b=D7Zt3XQMvWfHlQri0fAA Am3uo20eY1ZidFd7tjaojRD0Hpf+mloqVYHMzeJ/UKTWxcsofi5LlZbSxK48Pzj0m53LWR8vq6067 iG0qbBRHd/qzlOkYSYu8lhObmWznscN34b6Fw8QN+Pl5UE/SKxsV/tD/gpn6UWmYKN+cLRimjVK2Z B2946VLwunVoov5deyhChXsC/Zj0CtyBqt0urfjUCdD06VY67ET6bwfB38nK32ZbeMxAg/ivVf5Gf gWAxom8i/XY2CpH5N6le0rj5nGmCkX4iV1R5qlOwxRFTcYusO0VzXzSHqyUA9uLNfnDq05A9IpWz/ w7ui6T4PQLdrwQ==; From: =?utf-8?Q?Ludovic_Court=C3=A8s?= To: Christopher Baines Cc: guix-devel@gnu.org Subject: Re: March update on bordeaux.guix.gnu.org In-Reply-To: <87a5mhs1gi.fsf@cbaines.net> (Christopher Baines's message of "Fri, 29 Mar 2024 10:53:27 +0000") References: <877chnfw81.fsf@cbaines.net> <87sf09gwy6.fsf@gnu.org> <87a5mhs1gi.fsf@cbaines.net> X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: Primidi 21 Germinal an 232 de la =?utf-8?Q?R=C3=A9vo?= =?utf-8?Q?lution=2C?= jour du Gainier X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-pc-linux-gnu Date: Tue, 09 Apr 2024 17:37:29 +0200 Message-ID: <87edbemts6.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: guix-devel-bounces+larch=yhetil.org@gnu.org X-Migadu-Flow: FLOW_IN X-Migadu-Country: US X-Migadu-Spam-Score: -11.14 X-Migadu-Scanner: mx11.migadu.com X-Spam-Score: -11.14 X-Migadu-Queue-Id: AE23E7285A X-TUID: aLjsq4+d0sni Hello, Christopher Baines skribis: > The max-age of that narinfo is currently based on the scheduled removal > of the zstd compressed nar, which is going to happen quite far in the > future. > > I did think of a number of ways to approach this, and I'm not sure I've > settled on the right one yet. Maybe the TTL should be capped at 600, and > then drop to 0 as the time to remove the zstd nar approaches? Not sure what you mean by removal. To me, it=E2=80=99s the other way aroun= d: when the server advertises a TTL, it must guarantee that the associated nars remain available until that TTL has expired. (It can also keep it longer.) The way =E2=80=98guix publish=E2=80=99 honors that commitment is by keeping= track of the last time each narinfo was published and protecting the narinfo and nars from deletion until the TTL has expired. >> But then again, that=E2=80=99s the advertised TTL; the real TTL is still >> infinite, right? > > As you probably know, the situation is more complex. > > The problems caused when the nar-herder started removing zstd compressed > nars shows the difference between retention of the nar in some form, and > whether a cached narinfo response can be considered fresh or stale. > > Users might also not notice the availability of zstd nars if they cache > responses forever, since currently there will be a lag between the nar > becoming available, and a zstd compression being created (although we > could generate zstd compressed nars for everything). I think we should arrange to never advertise nars that are not actually available, if that=E2=80=99s what you meant. >> It=E2=80=99s a cache. It=E2=80=99s useful to have this cache because in= =E2=80=9Ctypical=E2=80=9D Guix >> usage you=E2=80=99re likely to ask repeatedly for the same substitutes. >> >> Regarding the cost, 3f5e14182931f123c10513a3e1e2abaebfb52279 made things >> more reasonable by putting a higher bound on narinfo retention. On my >> laptop, I have: >> >> $ ls -lrt /var/guix/substitute/cache/4refhwxbjmeua2kwg2nmzhv4dg4d3dorpje= fq7kiciw2pfhaf26a/ |wc -l >> 11549 >> $ du -h /var/guix/substitute/cache/4refhwxbjmeua2kwg2nmzhv4dg4d3dorpjefq= 7kiciw2pfhaf26a/=20 >> 50M /var/guix/substitute/cache/4refhwxbjmeua2kwg2nmzhv4dg4d3dorpjefq= 7kiciw2pfhaf26a/ >> >> Maybe that=E2=80=99s still excessive and we could further reduce the max= imum >> caching time. > > Having played around with this a bit (e.g. hacking guix weather not to > cache), I'm a bit sceptical. Given maintaining the cache takes time that > could be spent doing network I/O, and does potentially slow disk I/O, I > think it would be interesting to try and work out in what situations the > cache speeds things up overall, and in what situations it slows things > down overall.. Yes, we can surely fine-tune that to make sure it remains beneficial. >>> 6: https://lists.gnu.org/archive/html/guix-devel/2023-05/msg00290.html >> >> BTW, should we document this mirror somewhere (and also ensure that Guix >> Foundation pays the bills), or do you view it more as an experiment for >> now? > > If the project does want to provide mirrors, I think that would be > great. From this experiment, I think we have some evidence that there > are people using Guix outside of Europe, and in some cases they struggle > with the European based infrastructure. It also seems like these mirrors > do help, and the monetary cost isn't too high in my view. > > I think we should probably wait until the project starts managing them > before documenting/advertising them more widely though. *We* are =E2=80=9Cthe project=E2=80=9D. :-) By that I mean that we should discuss it with people at the Guix Foundation and with the broader community, but it seems pretty clear to me that there=E2=80=99s interest in having mirrors up and running, especial= ly outside Europe. Then of course we need to be able to scale it according to available funds, but at least the goal itself is clear. [...] >> Do you think the Data Service or another source of info would let us >> make such decisions? >> >> If we take it to the extreme, we could have a sophisticated retention >> policy like: drop all fixed-output derivations known to be available >> from disarchive.guix + SWH, drop substitutes for packages that have less >> than 100 dependents, etc. > > I think the Data Service (specifically data.guix.gnu.org) might be > really helpful here, as it speeds up being able to work out what a nar > or derivation relates to. > > Additionally, the nar-herder can tag narinfos (associate key=3Dvalue pairs > with them), and that's intended to help you manage the nars. So we > should probably start tagging the nars with potentially useful > information now, so that we can use that data later to make desicions. Really good. > We're storing 17.5TiB of nars currently, and this increases linearly, so > it would be good to understand how this can be broken down. The > nar-herder should help here as well, as providing you can download the > 11G database, that should contain all the information you need to start > digging in to this. > > wget https://bordeaux.guix.gnu.org/latest-database-dump -O bordeaux.db Neat, thanks! Ludo=E2=80=99.