From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp1 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id qA4DKDqyZWHjOgAAgWs5BA (envelope-from ) for ; Tue, 12 Oct 2021 18:05:14 +0200 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp1 with LMTPS id qLCDIzqyZWHidwAAbx9fmQ (envelope-from ) for ; Tue, 12 Oct 2021 16:05:14 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 440DD373EC for ; Tue, 12 Oct 2021 18:05:14 +0200 (CEST) Received: from localhost ([::1]:54900 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1maKHB-00006Q-8t for larch@yhetil.org; Tue, 12 Oct 2021 12:05:13 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:57694) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1maKGX-0008VU-Nx for guix-devel@gnu.org; Tue, 12 Oct 2021 12:04:35 -0400 Received: from mail2-relais-roc.national.inria.fr ([192.134.164.83]:42404) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1maKGV-0000YL-C3 for guix-devel@gnu.org; Tue, 12 Oct 2021 12:04:33 -0400 IronPort-HdrOrdr: =?us-ascii?q?A9a23=3A/Ekz0KoQJq0cbcR2smwdZSAaV5o6eYIsimQD?= =?us-ascii?q?101hICG9Afbo8vxG/c5rtyMc5wxxZJhNo7y90cq7MBDhHPxOj7X5VI3KNGLbUQ?= =?us-ascii?q?2TRr2KhrGM/9SPIUDDH6JmtZtdTw=3D=3D?= X-IronPort-AV: E=Sophos;i="5.84,326,1620684000"; d="scan'208";a="533531377" Received: from unknown (HELO ribbon) ([193.50.110.91]) by mail2-relais-roc.national.inria.fr with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 12 Oct 2021 18:04:25 +0200 From: =?utf-8?Q?Ludovic_Court=C3=A8s?= To: zimoun Subject: Substitute retention References: <87a6tdce94.fsf@inria.fr> <87mu4iv0gc.fsf@inria.fr> <87v9c0ap22.fsf_-_@gnu.org> <87wnmsn5lz.fsf_-_@gnu.org> <87bl44vfvg.fsf_-_@gmail.com> <87o880byyz.fsf@inria.fr> <87czoay4sq.fsf@inria.fr> <86h7dmms8c.fsf@gmail.com> X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: 21 =?utf-8?Q?Vend=C3=A9miaire?= an 230 de la =?utf-8?Q?R=C3=A9volution?= X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-pc-linux-gnu Date: Tue, 12 Oct 2021 18:04:25 +0200 In-Reply-To: <86h7dmms8c.fsf@gmail.com> (zimoun's message of "Tue, 12 Oct 2021 12:50:59 +0200") Message-ID: <87y26ytek6.fsf_-_@inria.fr> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Received-SPF: pass client-ip=192.134.164.83; envelope-from=ludovic.courtes@inria.fr; helo=mail2-relais-roc.national.inria.fr X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: guix-devel@gnu.org Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: "Guix-devel" X-Migadu-Flow: FLOW_IN ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1634054714; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post; bh=B4GZujUl+6s6tr+z8TL5aTIjsXgCv6lhV2UOOo7GQ0I=; b=q8juBD+cL70/Q6KBaoQTvwI8H3DZMaqe9Vd+UQnG4AcM8vs4FcJw1N413flVG+IqldH41v xn/nnBiaqnjL2dhYRScdJM1UfU1ES8Hvtf32+LNEYO1+Yt0P2mT2UbWDalWp67sjtqGVEG 8NQgTlu1GSrmOuZ5p2lCN5eQP3oJsIL6L6q2K4eyzbgzCMi+RSV/Nk/JNbylOHnp/6eXJY 6naaNgMlSgYfiwS6phqEYt6LVLqvTIsM53mTyY/CqeLWMznB1ikje9GL3z2QgIekEpBlzw XVm0S+nfwQhsbRHnPgEU2UdMOC543YZTuhvqvK0PEC6+pUOGLq5urQdSsU+Syg== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1634054714; a=rsa-sha256; cv=none; b=XkaKc0jMJoUatXM6ImmNCB31au+I+t9e7Ferfalcp3OO4tCynTgX1nZcbyv41IGzyfNt2B 38QXDYEYq9xCFGMMfBvNDzwJggio3aOLAF8nnPv2bfUzGq9/FaQlhF/bcVKr1698wHR8PX Jw9V2LU/QkAC00zQ0eRIHt/d3x9VkwIUWS7xn1iP4xlkTqql2idq1Or0EC2rmzWgrYhVYV ftzi2Ty/JREOUropPAahxNUQ9x5KXk9jY8/RNbwX5N0M7k7TXt0JCg3mtziv9ksMms0dw3 2iPxMXHAaC+vDP0ORo6EPTrUEiPS8xsx07W9EVcfWWV75DYuGgSZCC6/Rb/jaQ== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of guix-devel-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-devel-bounces@gnu.org X-Migadu-Spam-Score: -2.41 Authentication-Results: aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of guix-devel-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-devel-bounces@gnu.org X-Migadu-Queue-Id: 440DD373EC X-Spam-Score: -2.41 X-Migadu-Scanner: scn0.migadu.com X-TUID: Kl6Fg17kAq8s Hi! (Moving to guix-devel from .) zimoun skribis: >> For the record, the =E2=80=98guix publish=E2=80=99 config on berlin is h= ere: >> >> https://git.savannah.gnu.org/cgit/guix/maintenance.git/tree/hydra/modu= les/sysadmin/services.scm#n485 >> >> If I read that correctly, nars have a TTL of 180 days (this is the time >> a nar is retained after the last time it has been requested, so it=E2=80= =99s a >> lower bound.) [...] > Just for the record, a back to envelope computations. 180 days before > today was April 15th (M-x calendar C-u 180 C-b). It means 6996 commits > (35aaf1fe10 is my current last commit). > > git log --format=3D"%cd" --after=3D2021-04-15 | wc -l > 6996 > > However, these commits are pushed by batch. Roughly, it reads: > > git log --format=3D"%cd" --after=3D2021-04-15 --date=3Dunix \ > | awk 'NR =3D=3D 1{old=3D $1; next}{print old - $1; old =3D $1}' \ > | sort -n | uniq -c | grep -e "0$" | head > 1 -1542620 > 3388 0 > 14 10 > 6 20 > 5 30 > 2 40 > 4 50 > 1 60 > 2 70 > 2 80 > > (Take the =E2=80=99awk=E2=80=99 with care, I am not sure of what I am doi= ng. :-) And, > it is rough because timezone etc.) > > Other said 3388/6996=3D ~50% of commits are pushed at the same time, i.e., > missed by both build farms using 2 different strategies to collect the > thing to build (fetch every 5 minutes or fetch from guix-commits). It > is a quick back to envelope so keep that with some salt. :-) OK. > On that number, after 180 days (6 months), it is hard to evaluate the > rate of the time-machine queries. And from my experience (no number to > back), running time-machine on a commit older than this 180 days implies > to build derivations. Or it is a lucky day. :-) Right. So what can we do to address this issue? I *think* we could use a higher TTL on berlin, and we can try that right away (9 months to being with?). However, there is an upper bound anyway. To make informed decisions on the retention policy, we should monitor storage space on berlin/bayfront to better estimate what can be done. We have Zabbix but it=E2=80=99s not accessible from the outside; maybe we could graph storage space somewhere so people can grab the data and work on those estimates? What if we decide that we need to provide substitutes for 2y old commits? In that case, we need a plan to scale up. That could be renting storage space somewhere. That=E2=80=99s largely non-technical work= that needs attention. There are also technical tweaks that could help: distinguishing between =E2=80=9Cimportant=E2=80=9D substitutes that we want to keep, and less impo= rtant substitutes (how?); identifying =E2=80=9Cequivalence classes=E2=80=9D for b= uilds of a given package; etc. The outcome is unclear and it=E2=80=99ll take time. Thoughts? Ludo=E2=80=99.