From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp10.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id AJkqFOqaC2KjagAAgWs5BA (envelope-from ) for ; Tue, 15 Feb 2022 13:22:02 +0100 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp10.migadu.com with LMTPS id SL66DOqaC2KLZwEAG6o9tA (envelope-from ) for ; Tue, 15 Feb 2022 13:22:02 +0100 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 3DFD82CA6F for ; Tue, 15 Feb 2022 13:22:01 +0100 (CET) Received: from localhost ([::1]:51008 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nJwqG-0001GX-Cm for larch@yhetil.org; Tue, 15 Feb 2022 07:22:00 -0500 Received: from eggs.gnu.org ([209.51.188.92]:43934) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nJwpQ-0001F6-8e; Tue, 15 Feb 2022 07:21:10 -0500 Received: from [2001:470:142:3::e] (port=53718 helo=fencepost.gnu.org) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nJwpF-0003nr-9O; Tue, 15 Feb 2022 07:21:07 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:In-Reply-To:Date:References:Subject:To: From; bh=7514Ccqd1vLIMig4gNYk7bZaNxKtFcG+SUmf5+LBTQc=; b=S9pa51qcBQPSNwv3EstZ Wd1T6A2S3ZTRXsq0l9ID6PUYeHWzPOmJT6qk0eXtJNUW6e+NFoYN7n1Fgy80LBlX/LiMTaBBkxYNx Fjg1u+2yXmhQjGuWlXN4zTO+Q3oByLbjCvQ0RF10e2KAr92agAItA0Ld+eCJx/6HOGQlw8FRegTbV J8OWykGOYatwPBr0j9TbJ6Ue7CQTOdKL8XJ7l+TOHP9Xchy9zqQDeuNxNWh5b0PCkOnKdxaYMV/fH NQsrHtmpmc/HEtXr904DqkNWHhEidHwiHZi0QFklI+9cpc85fXGKHJskmRlG8JZhKDKq8ejzRU5gt pSmufklZ6xfx2Q==; Received: from [2a01:cb16:2a:f834:b9c8:468f:a6eb:5320] (port=57920 helo=meije) by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nJwos-0005uB-TY; Tue, 15 Feb 2022 07:20:49 -0500 From: Mathieu Othacehe To: Ludovic =?utf-8?Q?Court=C3=A8s?= Subject: Re: Dropping gzip-compressed substitutes References: <878rum1pph.fsf@gmail.com> <87h791s5fr.fsf@gnu.org> Date: Tue, 15 Feb 2022 13:20:31 +0100 In-Reply-To: <87h791s5fr.fsf@gnu.org> ("Ludovic =?utf-8?Q?Court=C3=A8s=22'?= =?utf-8?Q?s?= message of "Mon, 14 Feb 2022 18:50:48 +0100") Message-ID: <87r184l3sg.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: guix-devel , guix-sysadmin , Maxim Cournoyer Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: "Guix-devel" X-Migadu-Flow: FLOW_IN X-Migadu-Country: US ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1644927722; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post:dkim-signature; bh=7514Ccqd1vLIMig4gNYk7bZaNxKtFcG+SUmf5+LBTQc=; b=ZXt/fIs0oTEEtbqZpnuPjT6k/Y/quVb5Fuxg7dY8044wQRQ6zosy2dADUM0WM1RtDH47Bj rUqgX+WCPEtPUNpVKzVbN1pU325NyNr8UfInewINDK0D1dfWGm8sjcGYiv+XUAV2xgvyir a4W7Lzfu99VAmJy/NGzbJm65jM/y1Z8L3gma/lOu40svDK9yJIRcI/P0cs7O1P8wbg6q7+ j7byAa26wgvXkLpiq2moOci1YP2me6mgO9zk2c/RZLkMzOyZim8JJhObLtdhuNoy7wPTsn hgn/jKoeaWCnkLfHQ1872+S5yqKXFfEien03tC5hVZR2ukX6tng7fwRnJSuqzg== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1644927722; a=rsa-sha256; cv=none; b=PMuMVz0pA8DikbJoqTpqTdVzGLzX4POVtAGn5t9gqugFB4wOg5zg+Ouem9PVithyE5KrDZ k0Sgq0OL+FS9VQZHJxH3CLHQsuReVcwMRB7ZWGwM3bbMwIGVF7Qv5YHQUVU1jMBZLX5phF 4po02Lpo8m586yzyuvmWWJS+akI/YDA/QdnrS6hSKGGEWmRvdSO04yFq7yxf4INjiuXIdo 36mCpQqsKRDE/59mRh0wFRhVgFK87YEEYFhbWgYoiwFPEUuux78/gpAJDD+JaFkal11frA HoIJh5CSzrwBngo1cCTrj6UWfmtZqCHSUZ1+hVo+/YwD+vYuzTAwdvxCbubGfQ== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=pass header.d=gnu.org header.s=fencepost-gnu-org header.b=S9pa51qc; dmarc=pass (policy=none) header.from=gnu.org; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org" X-Migadu-Spam-Score: -5.53 Authentication-Results: aspmx1.migadu.com; dkim=pass header.d=gnu.org header.s=fencepost-gnu-org header.b=S9pa51qc; dmarc=pass (policy=none) header.from=gnu.org; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org" X-Migadu-Queue-Id: 3DFD82CA6F X-Spam-Score: -5.53 X-Migadu-Scanner: scn1.migadu.com X-TUID: YS/0ymJcSfvd Hey Ludo, > As discussed on IRC, I=E2=80=99m skeptical about this because: > > 1. It requires the development and testing of a custom tool that=E2=80= =99s > easy to get wrong=E2=80=94e.g., it removes a gzipped nar for somethi= ng that > had nothing but gzip available, etc. > > 2. That code would have to run with privileges that give it access to > the signing key on berlin. > > 3. Those 6.5=C2=A0TB are an initial constant factor; growth of the stor= age > requirements going forward probably matters more and > will give us more flexibility > on that. While those are valid points, we need to keep in mind that it is important that we manage to move the store to the new SSD array quite quickly to start GCing again. If we cannot manage to remove those gzip nars then, I see only two alternatives: * Host the nars on the HDD array only. * Host the nars elsewhere, on a VPS as you are proposing. > I like Chris Baines=E2=80=99 idea of decoupling nar distribution from nar > building. If we want to keep nars long enough so that =E2=80=98time-mach= ine=E2=80=99 is > usable, then storage requirements will keep growing. > > Perhaps that means we can regularly copy nars =E2=80=9Celsewhere=E2=80=9D= for long-term > storage, using nar-herder, rsync, or whatever. The machine that stores > nars long-term has low requirements compared to the build farm because > we don=E2=80=99t need to trust it for anything other than storage. If th= at > makes things easier (and financially viable), a VPS is good enough. Sure, the VPS would also allow us to have a less European-centric hosting. I did not follow closely the development of the nar-herder. Chris what improvements this tool would bring compared to a rsync based approach? Thanks, Mathieu