From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp1.migadu.com ([2001:41d0:1008:1e59::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms8.migadu.com with LMTPS id +NagJgcCnmVmDgEAkFu2QA (envelope-from ) for ; Wed, 10 Jan 2024 03:33:43 +0100 Received: from aspmx1.migadu.com ([2001:41d0:403:58f0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp1.migadu.com with LMTPS id CFSHIQcCnmWmMwEA62LTzQ (envelope-from ) for ; Wed, 10 Jan 2024 03:33:43 +0100 X-Envelope-To: larch@yhetil.org Authentication-Results: aspmx1.migadu.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=HaNkb8RG; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org"; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1704854023; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version: content-type:content-type:list-id:list-help:list-unsubscribe: list-subscribe:list-post:dkim-signature; bh=YSa5uwhXPgBy2PMVUZR3xZ46N8iy05iAhx439rdshNk=; b=dIbP1hxE1N+R++lkNd/SQFlVn74w1WXIdHi14/jCFkMRHQ2GUJikQNr3khgFPmYiYVmT7w hOdZf/miQDxlkiQTHVv6PlzXye56gVv4Tl1ZCpkMhv0VkYcSg+2t+pkHMHN5YaGWMfg7aE AemncPY//sAj/tXOdEPDaYC86mRB2/jzT0QIo0fs/XN2zmWvnl964UCub869OHKXL/ZBLp VCanSBp2NRI+YkALm6E9PUQ4dSmTQt+XbgAQ3MInOsFFs4ydsX1MQdaGqxvFydyg6knur0 8+amFu9X4KmIwRyOcgA15CKvHuijKHkR6UGi8nmLM22l27acyZIbDOKDM3xa6Q== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1704854023; a=rsa-sha256; cv=none; b=qW3v+2tW+WGhY7HbFM0n3U7kElmZuACi/SMjwVCnjoyAOdSB8s+edL7K9o0jB/q7r9andB o4mp5EU+U2I0oRC+Iwf3ghLvkX1e1CTbR9yo4JcSyobIqEDFE8aTipU5+YPegyTNJmXH15 UFxjvbbK7ie016bGVPmDeXaw9zPDnEb+xR4rU7AcJo9PYmPZ3gbsGtxuDNbEKiwYnHhtFD ul6H9BP0P/eRSGkJF7tFqeThBP/lAsaz17NzcU2GhtC/ek/6z+wOPA/YJjOlweZiIjvSz+ rr128CxFv7Tpz5BALdEkk+PAswzrts5OIj1bqxjJRb5wrBB/vT8mIZs1RuC4zg== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=HaNkb8RG; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org"; dmarc=pass (policy=none) header.from=gmail.com Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 227773D885 for ; Wed, 10 Jan 2024 03:33:43 +0100 (CET) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rNOOh-0002q0-Ua; Tue, 09 Jan 2024 21:32:52 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rNOOe-0002po-IH; Tue, 09 Jan 2024 21:32:48 -0500 Received: from mail-qk1-x72b.google.com ([2607:f8b0:4864:20::72b]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1rNOOc-0008Tc-N2; Tue, 09 Jan 2024 21:32:48 -0500 Received: by mail-qk1-x72b.google.com with SMTP id af79cd13be357-783269124a8so163472185a.1; Tue, 09 Jan 2024 18:32:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1704853959; x=1705458759; darn=gnu.org; h=mime-version:message-id:date:subject:to:from:from:to:cc:subject :date:message-id:reply-to; bh=YSa5uwhXPgBy2PMVUZR3xZ46N8iy05iAhx439rdshNk=; b=HaNkb8RGnxF7OdQLz3V3n4/322ZLPb/9F8IOZbhfK8yau2GNaDUt2rqSYv9xJPw+cI nHPWLanAwrrW2IUnt/I0w3nyO4iDgcFpP8co2ILAEmkkgqUSykNFfE0iicO1JVrrBFWY ZYzNbBgvjBKQGOqJmIKsXg518mYipUkx0YACQcA/sdxL6zP25T2tR8fjj2igRa028xY+ 2YCi9wJ131tiT7R62VM0bXOVQhiO1801fgVN0YyUPOoZa8b3tma0/C8YdCv0mp09h2Wm h9TUuar1SXKmbpWzDyPA5QIYra038Xe6Gl/jP51FqbqtRd35S66V8nICnkMV3KvS62gb LoAg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1704853959; x=1705458759; h=mime-version:message-id:date:subject:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=YSa5uwhXPgBy2PMVUZR3xZ46N8iy05iAhx439rdshNk=; b=bIJ5h0zy3Lvrwikjsoo8Ek15xgX664ifVPy4Jde445S3OA6oksv2FAYMVEApnwmnNu Hg9Nxv2rU8JPyHLciuDNAl2YLHhVuhdqwDNNFHHFas2/d8sHMjwP2olE/OwWqYlFWxNa BIblhMEHjL0XVmbBfpOxFWOQILhndeIINvQmZSk0QtVGjZkaH1BEaYaNp3NOE6tGPUYv tunifcFDo86Kq5XaOPuH5AdRy6Bk4WGYgmOvzaPtdVv+lpqyQr6KH6H/RzlShkJPzHMh uLL9eMTE+4kP0hY1m43N45cZ6t9AZe1eGmGH7mzjvqyZ/KJlX3ZoomSKY6Dwe994VzUc FMYA== X-Gm-Message-State: AOJu0YwaSQHzER2OpfLAuUUTyWutYEnh2chh0njKjzhPrVxzI1orz1lS 88VJiwIUc0ZsRrvLd6DpEXqnNy1XwrszVw== X-Google-Smtp-Source: AGHT+IGFlmGrEEMr/5xZDfZfhz0zGiamabYTcU5KpavMv8gaHUaQ7i8jLI386Blxoewo2d0on0WK6A== X-Received: by 2002:a05:620a:271b:b0:783:2ec6:2fb7 with SMTP id b27-20020a05620a271b00b007832ec62fb7mr513380qkp.26.1704853959304; Tue, 09 Jan 2024 18:32:39 -0800 (PST) Received: from hurd (dsl-205-151-56-231.b2b2c.ca. [205.151.56.231]) by smtp.gmail.com with ESMTPSA id x4-20020a05620a0b4400b0078156dd57a3sm1283668qkg.15.2024.01.09.18.32.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 09 Jan 2024 18:32:38 -0800 (PST) From: Maxim Cournoyer To: guix-devel , guix-sysadmin Subject: Proposition to streamline our NAR collection to just zstd-compressed ones Date: Tue, 09 Jan 2024 21:32:37 -0500 Message-ID: <87y1cxuddm.fsf@gmail.com> MIME-Version: 1.0 Content-Type: text/plain Received-SPF: pass client-ip=2607:f8b0:4864:20::72b; envelope-from=maxim.cournoyer@gmail.com; helo=mail-qk1-x72b.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: guix-devel-bounces+larch=yhetil.org@gnu.org X-Migadu-Flow: FLOW_IN X-Migadu-Country: US X-Migadu-Spam-Score: -8.01 X-Spam-Score: -8.01 X-Migadu-Queue-Id: 227773D885 X-Migadu-Scanner: mx11.migadu.com X-TUID: nutkcPynWcld Hello Guix, and Happy New Year! It's been on my head for quite a bit of time (about 2 years, according to [0]), to streamline our offering of cached nars. Letting go of gzip 2 years ago, along a more aggressive garbage collection policy allowed us to reduce our storage needs by at least 6.5 TiB. I'm proposing to do the same with our lzip compressed nars, to let go of an additional 3.9 TiB: --8<---------------cut here---------------start------------->8--- $ du -sh /var/cache/guix/publish/{lzip,zstd} 3.9T /var/cache/guix/publish/lzip 4.1T /var/cache/guix/publish/zstd $ find /var/cache/guix/publish/lzip -name '*.nar' | wc -l 4484645 $ find /var/cache/guix/publish/zstd -name '*.nar' | wc -l 4461195 --8<---------------cut here---------------end--------------->8--- The above suggests that zstd compressed nars are about 5% larger than the lzip ones, which is not big enough to justify carrying both, in my opinion. In exchange for a little bit more bandwidth, users would have the nars decompressed much faster with less CPU overhead locally. Having our complete nars collection fit in around 4 TiB would also open the door for simple rsync-based mirroring, which I have started working on. What do you think? Should we go ahead and effect the following simple change for the Berlin build farm? --8<---------------cut here---------------start------------->8--- modified hydra/modules/sysadmin/services.scm @@ -683,7 +683,7 @@ to a selected directory.") ;; ;; for the compression ratio/decompression speed ;; tradeoffs. - (compression '(("lzip" 9) ("zstd" 19))) + (compression '(("zstd" 19))) (cache-bypass-threshold cache-bypass-threshold) (workers publish-workers))) --8<---------------cut here---------------end--------------->8--- -- Thanks, Maxim