From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp12.migadu.com ([2001:41d0:303:e224::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms8.migadu.com with LMTPS id aFzMACZIXmWOVAEAauVa8A:P1 (envelope-from ) for ; Wed, 22 Nov 2023 19:27:50 +0100 Received: from aspmx1.migadu.com ([2001:41d0:303:e224::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp12.migadu.com with LMTPS id aFzMACZIXmWOVAEAauVa8A (envelope-from ) for ; Wed, 22 Nov 2023 19:27:50 +0100 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id D97C95E8F6 for ; Wed, 22 Nov 2023 19:27:49 +0100 (CET) Authentication-Results: aspmx1.migadu.com; dkim=pass header.d=gnu.org header.s=fencepost-gnu-org header.b=ZucDjY7f; dmarc=pass (policy=none) header.from=gnu.org; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org" ARC-Seal: i=1; s=key1; d=yhetil.org; t=1700677669; a=rsa-sha256; cv=none; b=eqWhkE6aMYG/9I0HBYeGWQZKbQHtbtvDciRycMThacB76Z8tRmQka2iprmnsaEyIfrTnWw n+6VMs35mO1QA2UHFyFmK2GwUgeiglVGSH+ivtw4aFAfS5wZoL3oJQHdW9sTL2d6BdvCCt t04SPOrgrc+YIs7ORcDDeFt9px2j34C9QYc0f2IOSrga2A3PN2Z3KE2+GDfdfAmVIQvAfa dVXr4XiJlomzM5yfLiRqjZNDIYFjUgIN8OjPcJcv09Pa3tDKNqYuWMqEcyj4IQXQIbVGd1 jresaoj/WGD0wv9nUjJf3wy886VujwpE5BURnTWITq1zBC6jWL+8bmLo9wlmRg== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=pass header.d=gnu.org header.s=fencepost-gnu-org header.b=ZucDjY7f; dmarc=pass (policy=none) header.from=gnu.org; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org" ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1700677669; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post:dkim-signature; bh=XyhefGD4AyDb3rtaM0H2DuSOKWTxSScbh+KByT50grc=; b=Jn0AkJ6PUBJ9Naa/1fgSdh0/lqqndDqtXNh+m73KwAVoAc7m1aIqe8lRSIz4/Wc+9tPNfE T8/bzyKN4EFNJ/c2icPDchvmzh6sPZNTA89lSx8MOjcRYPnTjSQhGJ0PYUeS07fNiYEc9z NFMMGjsfdIi9nS8D/TidrglKHMOpjnpGLw6a2NaeX4G4C2LLz1ss/6B+eighgiIOeNmyyo vcbJdC1oKxtpcUijH5lIaKa9IcXo3AIIUHtMKaRY3am6xThEdcJscfDXbfUIRZwThrPkoL qMpV5fXe8DINuA4JiNovjllxUSRMsMktpeCzKLfHs9a6mCnKmClI28hNmu+SUQ== Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1r5rwZ-0003cp-T9; Wed, 22 Nov 2023 13:27:23 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1r5rwX-0003cP-Lz for guix-devel@gnu.org; Wed, 22 Nov 2023 13:27:21 -0500 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1r5rwX-0003td-9t; Wed, 22 Nov 2023 13:27:21 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:Date:References:In-Reply-To:Subject:To: From; bh=XyhefGD4AyDb3rtaM0H2DuSOKWTxSScbh+KByT50grc=; b=ZucDjY7fRLmpud9OGtR4 Fel3PYq3xApBW2D+uPyOX0Ba13Nqju673ie8EK5rAzO0oHSlgQPIAuLmw7NZ9Qf/VOarYAItSuQDG +0lZ8yUD4rvzHBhKsbfP/1TvFleeV6Ai4t8aZZMx3LjJQjUuQDovvHkDmmUCB19aai2aQ/c9A57i8 mdLhmiexTnOMcjtaf8hcMFu4I+J8DbJhMpPme2rXojhRQAzmYzSp7rDSCjTxkIk5cA0+BTZ3G/hI+ eT8RhKn+KF/CNCYjeT4yD21LXsfPrjqFhLRtw2kdYllCSQXTsTOlzJNG7/d4wZTYlZOvN7G1u3oDE Um8Lzzby2/fmnw==; From: =?utf-8?Q?Ludovic_Court=C3=A8s?= To: Maxim Cournoyer Cc: Ricardo Wurmus , guix-devel Subject: Re: Building and caching old Guix derivations for a faster time machine In-Reply-To: <877cmfd6tm.fsf@gmail.com> (Maxim Cournoyer's message of "Fri, 17 Nov 2023 23:27:33 -0500") References: <87o7g29c94.fsf@elephly.net> <878r6xu2pe.fsf@gnu.org> <877cmfd6tm.fsf@gmail.com> X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: Duodi 2 Frimaire an 232 de la =?utf-8?Q?R=C3=A9volut?= =?utf-8?Q?ion=2C?= jour du Turneps X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-pc-linux-gnu Date: Wed, 22 Nov 2023 19:27:18 +0100 Message-ID: <87h6ldiqyh.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: guix-devel-bounces+larch=yhetil.org@gnu.org X-Migadu-Flow: FLOW_IN X-Migadu-Country: US X-Migadu-Queue-Id: D97C95E8F6 X-Migadu-Scanner: mx12.migadu.com X-Migadu-Spam-Score: -10.64 X-Spam-Score: -10.64 X-TUID: zRg42cJ9ak9p Hi, Maxim Cournoyer skribis: >> I agree. The =E2=80=98guix publish=E2=80=99 TTL=C2=B9 at ci.guix was in= creased to 180 days >> following in 2021. That=E2=80=99s s= till not >> that much and these days and right now we have 84=C2=A0TiB free at ci.gu= ix. >> >> I guess we can afford increasing the TTL, probably starting with, say, >> 300 days, and monitoring disk usage. >> >> WDYT? > > While the 84 TiB we have at our disposal is indeed lot, I'd rather we > keep the TTL at 180 days, to keep things more manageable for backup/sync > purposes. Our current TTL currently yields 7 TiB of compressed NARs, > which fits nicely into the hydra-guix-129 10 TiB slice available for > local/simple redundancy (it's still on my TODO, missing the copy bit). > > I've been meaning to document an easy mirroring setup for that > /var/cache/guix/publish directory, and having 14 TiB instead of 7 TiB > there would hurt such setups. Maybe we should learn from what Chris has been doing with the Nar-Herder, too. Ideally, the build farm front-end (=E2=80=98berlin=E2=80= =99 in this case) would be merely a cache for recently-built artifacts, and we=E2=80=99d have long-term storage elsewhere where we could keep nars for several years. The important thing being: we need to decouple the build farm from (long-term) nar provision. > Perhaps a compromise we could do is drop yet another compression format? > We carry both Zstd and LZMA for Berlin, which I see little value in; if > we carried only ZSTD archives we could probably continue having < 10 TiB > of NARs for a TTL of 360 days (although having only 3.5 TiB of NARs to > sync around for mirrors would be great too!). > > What do you think? For compatibility reasons=C2=B9 and performance reasons=C2=B2, I would refr= ain from removing lzip or zstd substitutes, at least for =E2=80=9Ccurrent=E2=80= =9D substitutes. For long-term storage though, we could choose to keep lzip only (because it compresses better). Not something we can really do with the current =E2=80=98guix publish=E2=80=99 setup though. Thoughts? Ludo=E2=80=99. =C2=B9 Zstd support was added relatively recently. Older daemons may suppo= rt lzip but not zstd. =C2=B2 https://guix.gnu.org/en/blog/2021/getting-bytes-to-disk-more-quickly/