From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp10.migadu.com ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms9.migadu.com with LMTPS id 2DQ9JMi6d2SDdwAASxT56A (envelope-from ) for ; Wed, 31 May 2023 23:23:20 +0200 Received: from aspmx1.migadu.com ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp10.migadu.com with LMTPS id +NGeI8i6d2SWdAEAG6o9tA (envelope-from ) for ; Wed, 31 May 2023 23:23:20 +0200 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id E40B68246 for ; Wed, 31 May 2023 23:23:19 +0200 (CEST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1q4THe-0001J3-7A; Wed, 31 May 2023 17:23:06 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1q4THc-0001Fp-R4 for guix-patches@gnu.org; Wed, 31 May 2023 17:23:04 -0400 Received: from debbugs.gnu.org ([209.51.188.43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1q4THc-0003Rc-IY; Wed, 31 May 2023 17:23:04 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1q4THa-0004wO-Cc; Wed, 31 May 2023 17:23:02 -0400 X-Loop: help-debbugs@gnu.org Subject: [bug#63817] [PATCH] substitute: Delete cached narinfos more than two-month old. Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: mail@cbaines.net, dev@jpoiret.xyz, ludo@gnu.org, othacehe@gnu.org, rekado@elephly.net, zimon.toutoune@gmail.com, me@tobias.gr, guix-patches@gnu.org Resent-Date: Wed, 31 May 2023 21:23:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: report 63817 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 63817@debbugs.gnu.org Cc: Ludovic =?UTF-8?Q?Court=C3=A8s?= , Christopher Baines , Josselin Poiret , Ludovic =?UTF-8?Q?Court=C3=A8s?= , Mathieu Othacehe , Ricardo Wurmus , Simon Tournier , Tobias Geerinckx-Rice X-Debbugs-Original-To: guix-patches@gnu.org X-Debbugs-Original-Xcc: Christopher Baines , Josselin Poiret , Ludovic =?UTF-8?Q?Court=C3=A8s?= , Mathieu Othacehe , Ricardo Wurmus , Simon Tournier , Tobias Geerinckx-Rice Received: via spool by submit@debbugs.gnu.org id=B.168556814418932 (code B ref -1); Wed, 31 May 2023 21:23:02 +0000 Received: (at submit) by debbugs.gnu.org; 31 May 2023 21:22:24 +0000 Received: from localhost ([127.0.0.1]:36428 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1q4TGx-0004vI-UT for submit@debbugs.gnu.org; Wed, 31 May 2023 17:22:24 -0400 Received: from lists.gnu.org ([209.51.188.17]:58126) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1q4TGs-0004v6-LM for submit@debbugs.gnu.org; Wed, 31 May 2023 17:22:22 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1q4TGs-0000Mr-CG for guix-patches@gnu.org; Wed, 31 May 2023 17:22:18 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1q4TGr-0003Gp-SN; Wed, 31 May 2023 17:22:17 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:Date:Subject:To:From:in-reply-to: references; bh=ZpXyQedbaUf3DpqB29SL3N6KJBA5Nd4oWwakE7qNvHY=; b=HkP0iTmrMQSHbc uQu/2t7GfPYgObk1cj0SsJc5Kg7YXOu869zFd2obSUQWQNhPUlzEHaV0+JfNmS/SkFkzzoEgCGoU5 V8mYBh2vpxmSCilMpxIwXnntfn3fYA6I9XRDCygkZ9V6fnUH0GlyHskOpJ/Jo/EDvL7eHsnjNcc/G JgBMb8KoivfFiP0zYVHihChh0xQrzLS9xDEd4JlIPuamYVMmGa+zE/wEcfrEEy0O0/taxdqfSdl1h Gg7GOhm98+6m1W+wm+ue2emQDmraHVXPoHnn5s9eYUNe7bcp8YdLlU+vdFzGZJxArf85EgxDOizqM GBXKQ2zlYdM5aqj9d7Wg==; Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (helo=gnu.org) by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1q4TGr-0003IW-FP; Wed, 31 May 2023 17:22:17 -0400 From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Date: Wed, 31 May 2023 23:22:01 +0200 Message-Id: X-Mailer: git-send-email 2.40.1 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+larch=yhetil.org@gnu.org Sender: guix-patches-bounces+larch=yhetil.org@gnu.org X-Migadu-Country: US X-Migadu-Flow: FLOW_IN ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1685568200; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:resent-cc: resent-from:resent-sender:resent-message-id:list-id:list-help: list-unsubscribe:list-subscribe:list-post:dkim-signature; bh=ZpXyQedbaUf3DpqB29SL3N6KJBA5Nd4oWwakE7qNvHY=; b=kPdG1OASNcdofyrVlYrjFptP0VixFTPKROU0ubXr9Bj+JPyKDvnmHQ5Jn4BR6Zz3QDDF9O AkK1fQhGFwMnIGn6i9Adr3GQq5TWwBqEvzPLPHGlpm/fPag4xOZA0k9W5PKdKPBZfuixMa fbzbhXm9lWt/lhIe5HkyD6F2nMR31jh99PUyYA+R2kGnn+Col47L/nYkcnDpI3Xq+vHQeB pv0zOj1e93E1K66dLewDHIfBqGbvV1RSJ9+YUDSlt0fZNh5UGGlOiDsqrpeYhCxKsLXE0u ANzNUu+mE0g/Jezl3+qBJgdiT2enDHx1LjgVvlayMAxBUeR8ChjY3dH/npCFtA== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=gnu.org header.s=fencepost-gnu-org header.b=HkP0iTmr; dmarc=pass (policy=none) header.from=gnu.org; spf=pass (aspmx1.migadu.com: domain of "guix-patches-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-patches-bounces+larch=yhetil.org@gnu.org" ARC-Seal: i=1; s=key1; d=yhetil.org; t=1685568200; a=rsa-sha256; cv=none; b=VyYQAkxw0FzlalSaVFNxZbdwq8C0poWPRfxvpSbSMRySDf0lpUuiAms4hbvDtPo5e/rCuP z4JPK+48n13h+F+ft0EgzQTtTSSD37FfaCpeoAMJVrZ0s7sHVarQaWlfEBpPXlMX3BRrUH sV7JV+qagJYRr6UqzdgkW0xnyyma6syzxxkDMwsZKVIlJxkb2VQYLV0rZc6LVDSEXZFv+f LARKZYH/kOwRjFw7EwyH5kQiHTniDBZhDhG7VPoaC0VL66a0ts76YPE2SNaTAdWGt2WdQ/ f/D68hCgjT5heo16Px8KgAerqnL6KGc0H8+pAmMY2jDhNtf7nBC/S2PrXtV74Q== X-Migadu-Scanner: scn1.migadu.com X-Migadu-Spam-Score: -1.55 Authentication-Results: aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=gnu.org header.s=fencepost-gnu-org header.b=HkP0iTmr; dmarc=pass (policy=none) header.from=gnu.org; spf=pass (aspmx1.migadu.com: domain of "guix-patches-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-patches-bounces+larch=yhetil.org@gnu.org" X-Migadu-Queue-Id: E40B68246 X-Spam-Score: -1.55 X-TUID: WNls9u0na9kv This allows 'guix substitute' to shrink the cache a bit more, which saves space and improves performance of cache-cleanup phases since fewer entries need to be traversed. * guix/scripts/substitute.scm (cached-narinfo-expiration-time): Define 'max-ttl' and use it as an upper bound. --- guix/scripts/substitute.scm | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) Hi! You might have noticed that /var/guix/substitute/cache tends to grow. On my laptop I have: --8<---------------cut here---------------start------------->8--- $ du -hs /var/guix/substitute/cache/ 152M /var/guix/substitute/cache/ $ du /var/guix/substitute/cache/ |sort -k1 -n [...] 16196 /var/guix/substitute/cache/kzwjeblndsbkjzmjailrt4bnhguil7tqjmewzcyw22hgajbhfy3q 136216 /var/guix/substitute/cache/4refhwxbjmeua2kwg2nmzhv4dg4d3dorpjefq7kiciw2pfhaf26a 155364 /var/guix/substitute/cache/ --8<---------------cut here---------------end--------------->8--- This full of old entries that I’m unlikely to need: --8<---------------cut here---------------start------------->8--- $ ls -lt /var/guix/substitute/cache/4refhwxbjmeua2kwg2nmzhv4dg4d3dorpjefq7kiciw2pfhaf26a |tail -10 -rw------- 1 root root 1187 Nov 30 18:49 xlxgm0lzg25fsz99qql8wp8y4l0ijvy6 -rw------- 1 root root 3182 Nov 30 18:49 yqdklfzzxx2zm2vvx11pm5my2jzlzwmy -rw------- 1 root root 1180 Nov 30 18:49 18pzv1gjw9mi3bzkvb2ms9am37kd960i -rw------- 1 root root 1484 Nov 30 18:49 qqpc6n1q5jz29zzwf1jn8srq3a5ysy86 -rw------- 1 root root 1060 Nov 30 18:27 9r6w8hrw2bpm2sjp03ynl1aifjkcn9wi -rw------- 1 root root 1056 Nov 30 18:27 ghdwy0xa2h9xi5qh2picb19fhhk0fzhp -rw------- 1 root root 2108 Nov 30 18:27 ixp9xd4cbs774yh5qkvywmjk1606s8i8 -rw------- 1 root root 1052 Nov 30 18:27 laawrywlbx1y76sdribiw8aygcvadili -rw------- 1 root root 1092 Nov 30 18:27 m72pfbhqhbhi7gj8kvy66lr99v7hvk7z -rw------- 1 root root 1070 Nov 30 18:27 ym69k8mzmxbw1ar76nzghl38h4g7g8v4 --8<---------------cut here---------------end--------------->8--- That six-month old because that’s the TTL and ci.guix.gnu.org has been advertising for some time now. Long story short: we’d rather trim the cache more aggressively. Thoughts? Ludo’. diff --git a/guix/scripts/substitute.scm b/guix/scripts/substitute.scm index 3626832dda..92935268b3 100755 --- a/guix/scripts/substitute.scm +++ b/guix/scripts/substitute.scm @@ -167,6 +167,12 @@ (define (lookup-narinfo caches path authorized?) (define (cached-narinfo-expiration-time file) "Return the expiration time for FILE, which is a cached narinfo." + (define max-ttl + ;; Higher bound on the TTL used to avoid keeping around cached narinfos + ;; for too long, which makes the cache bigger and more expensive to + ;; traverse. + (* 2 30 24 60 60)) ;2 months + (catch 'system-error (lambda () (call-with-input-file file @@ -174,10 +180,10 @@ (define (cached-narinfo-expiration-time file) (match (read port) (('narinfo ('version 2) ('cache-uri uri) ('date date) ('ttl ttl) ('value #f)) - (+ date ttl)) + (+ date (min ttl max-ttl))) (('narinfo ('version 2) ('cache-uri uri) ('date date) ('ttl ttl) ('value value)) - (+ date ttl)) + (+ date (min ttl max-ttl))) (x 0))))) (lambda args base-commit: 77f52db416a13e195d090cad4e9e7658feb2e86b -- 2.40.1