From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp10.migadu.com ([2001:41d0:403:478a::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms9.migadu.com with LMTPS id CO0YEJS9/mQJFwEAG6o9tA:P1 (envelope-from ) for ; Mon, 11 Sep 2023 09:11:16 +0200 Received: from aspmx1.migadu.com ([2001:41d0:403:478a::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp10.migadu.com with LMTPS id CO0YEJS9/mQJFwEAG6o9tA (envelope-from ) for ; Mon, 11 Sep 2023 09:11:16 +0200 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 1D98B510AB for ; Mon, 11 Sep 2023 09:11:15 +0200 (CEST) Authentication-Results: aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=riseup.net header.s=squak header.b=p8OnFoG5; spf=pass (aspmx1.migadu.com: domain of "bug-guix-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="bug-guix-bounces+larch=yhetil.org@gnu.org"; dmarc=fail reason="SPF not aligned (relaxed)" header.from=riseup.net (policy=none) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1694416275; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:resent-cc: resent-from:resent-sender:resent-message-id:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post:dkim-signature; bh=rjpviJRQTENoE7MjykTpMTZNK5FRa3WeLdHj8oDvpUM=; b=QGj35INta0ZpLbQyoCEW4zk5DgrB747kZGHH2sELShs2e25GeN0IJ2WhtMtr1qXhiJx9rS YdtmnKP5nd9NLBOnE4RIDqYhJaz40GFcSIEVpHquQpV0enNQ/j8C6ZhNV9ZH0RRlLXihlj WaK7CFKLiwUGW6thP6hHNJlvb/zHyZIAJY5RYcWTDkdyCq+dXNlEvKtymEzlOaTEYv6q+T BuR4BgyRCJRdsVltjlWrWw9Eh8S/uM1PvAtM+Ijf/3R9m7sDRgd8fft21GiPvdFpMvD0PX LtNUodz7wdWFXdcx1TdizD3jPvS8G20faUCVZhdQ90gvMP8uT5/Hpi/kc8NAGQ== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1694416275; a=rsa-sha256; cv=none; b=rAZ7CZIfUtz+4vCHcA7duQj9lNTVjaxqTAXy1Z4dNwUZdWs5i5ntyCQbysMyEBh86w+sRE /DHGTTsCmZ6wfwiTcJvnRxrJYzOJm0r1SpUf+jiGNDXeh4UNTuwmXA4dX/2J/kdxB2qDyz UHmVtRyfvFKUTyHDAlDwCNhJDePvT5nKLqiSIkQlVnKithSuP3bivTFFZjzM+gvMH0YWhJ 09AtdePfFrijWLX+/ViG3ebWGoDfe/DbkJLXMP1v41n2KiAygsGkIvHiAzV7TH87u3HJIt p42tWZxhiaeqz9FK5NW0W5GJwT++8eDNwzJ+XjfgBKKtgZ0pjOvCYRIV69VkjA== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=riseup.net header.s=squak header.b=p8OnFoG5; spf=pass (aspmx1.migadu.com: domain of "bug-guix-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="bug-guix-bounces+larch=yhetil.org@gnu.org"; dmarc=fail reason="SPF not aligned (relaxed)" header.from=riseup.net (policy=none) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qfb4Y-0002aL-69; Mon, 11 Sep 2023 03:11:02 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qfb4V-0002Zw-DV for bug-guix@gnu.org; Mon, 11 Sep 2023 03:10:59 -0400 Received: from debbugs.gnu.org ([2001:470:142:5::43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1qfb4U-0000ww-Ti for bug-guix@gnu.org; Mon, 11 Sep 2023 03:10:59 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1qfb4Y-0006nn-N0 for bug-guix@gnu.org; Mon, 11 Sep 2023 03:11:02 -0400 X-Loop: help-debbugs@gnu.org Subject: bug#65720: Guile-Git-managed checkouts grow way too much Resent-From: Csepp Original-Sender: "Debbugs-submit" Resent-CC: bug-guix@gnu.org Resent-Date: Mon, 11 Sep 2023 07:11:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 65720 X-GNU-PR-Package: guix X-GNU-PR-Keywords: To: Simon Tournier Cc: ludo@gnu.org, 65720@debbugs.gnu.org, dev@jpoiret.xyz X-Debbugs-Original-Cc: Ludovic =?UTF-8?Q?Court=C3=A8s?= , 65720@debbugs.gnu.org, bug-guix@gnu.org, Josselin Poiret Received: via spool by 65720-submit@debbugs.gnu.org id=B65720.169441620826063 (code B ref 65720); Mon, 11 Sep 2023 07:11:02 +0000 Received: (at 65720) by debbugs.gnu.org; 11 Sep 2023 07:10:08 +0000 Received: from localhost ([127.0.0.1]:51995 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1qfb3f-0006mI-S6 for submit@debbugs.gnu.org; Mon, 11 Sep 2023 03:10:08 -0400 Received: from mx1.riseup.net ([198.252.153.129]:55064) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1qfb3e-0006lg-2o for 65720@debbugs.gnu.org; Mon, 11 Sep 2023 03:10:06 -0400 Received: from fews02-sea.riseup.net (fews02-sea-pn.riseup.net [10.0.1.112]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx1.riseup.net (Postfix) with ESMTPS id 4RkdBr30skzDqpZ; Mon, 11 Sep 2023 07:09:56 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=riseup.net; s=squak; t=1694416196; bh=jCBwtrOdu4IjIK1DNoGSobWo/knpWw03ag0O4gDJiso=; h=References:From:To:Cc:Subject:Date:In-reply-to:From; b=p8OnFoG5FSZ4qVyuqU7eoO1NHJb8FeMLpDQu9pEtagI3uq2ku2qWHNqtfOFkZMiSk P9inRFGTA0r8s30MFTbGMkkBqwFCiediO2TsptW5zYwKk0QKIZglisSWlbs+QkSsCG CDxEWNaOUY4BssSxgZaE29mtnumAa53HgUXUmdhU= X-Riseup-User-ID: 558C9CB603439A5DAFC0FDC808E10CF6C3CA907716EB7D430B23A41F042139CA Received: from [127.0.0.1] (localhost [127.0.0.1]) by fews02-sea.riseup.net (Postfix) with ESMTPSA id 4RkdBb6bTLzFrwy; Mon, 11 Sep 2023 07:09:43 +0000 (UTC) References: <87bkejc7go.fsf@inria.fr> <87fs3tobju.fsf@gnu.org> <87tts9uj6x.fsf@jpoiret.xyz> <87msy0ln4m.fsf@gnu.org> <86il8mn7al.fsf@gmail.com> <87jzt0382l.fsf@gnu.org> <86cyyrskmj.fsf@gmail.com> From: Csepp Date: Mon, 11 Sep 2023 09:06:58 +0200 In-reply-to: <86cyyrskmj.fsf@gmail.com> Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-guix@gnu.org List-Id: Bug reports for GNU Guix List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-guix-bounces+larch=yhetil.org@gnu.org Sender: bug-guix-bounces+larch=yhetil.org@gnu.org X-Migadu-Country: US X-Migadu-Flow: FLOW_IN X-Migadu-Scanner: mx0.migadu.com X-Migadu-Spam-Score: 2.59 X-Spam-Score: 2.59 X-Migadu-Queue-Id: 1D98B510AB X-TUID: hYSr9Llolv5N Simon Tournier writes: > Hi, > > On Fri, 08 Sep 2023 at 19:09, Ludovic Court=C3=A8s wrote: > >>>> It would also be pretty bad for closure size: >>>> >>>> --8<---------------cut here---------------start------------->8--- >>>> $ guix size guile-git | tail -1 >>>> total: 106.6 MiB >>>> $ guix size guile-git git-minimal | tail -1 >>>> total: 169.8 MiB >>>> --8<---------------cut here---------------end--------------->8--- >>>> >>>> It=E2=80=99s also not clear concretely how we=E2=80=99d add that depen= dency. Try >>>> invoking =E2=80=98git=E2=80=99 from $PATH and print a warning if it do= esn=E2=80=99t work? >>>> But then, what about applications like Cuirass and hpcguix-web? >>> >>> I think we can rely on something like, >>> >>> guix shell -C git-minimal -- git gc >> >> We=E2=80=99re talking about the implementation of a cache (meant to spee= d up >> operations), that would actually fill said cache plus do a whole bunch >> of expensive operations? Nah. :-) > > I do not think. If I understand correctly, we need to run =E2=80=9Cgit g= c=E2=80=9D at > some point, therefore git-minimal needs to me around. The question is > how and when. > > Well, maybe I am missing what the bug is about. For me, it is about > running =E2=80=98git gc=E2=80=99 for cleaning the Git checkout cache, no? > > > Solution #1. Add git-minimal as inputs. It increases the closure and > the extra load (on average) is about the ratio between the rate of =E2=80= =9Cguix > pull=E2=80=9D and the rate of the git-minimal changes. > > Assuming, that people are running =E2=80=9Cguix pull=E2=80=9D once per we= ek and say =E2=80=9Cgit > gc=E2=80=9D is run after 50 pulls. (These both number are totally arbitr= ary and > based on my personal estimate). > > Data Service [1] tells: > > 2023-07-07 15:45:22 2023-09-08 21:22:08 > 2023-05-11 16:10:48 2023-07-07 14:21:45 > 2023-05-01 16:40:08 2023-05-11 14:36:16 > 2023-04-25 13:34:54 2023-05-01 15:19:55 > 2023-04-25 13:34:54 2023-09-08 21:22:08=20=20=20=20=20=20=20=20 > 2023-03-06 17:22:28 2023-04-25 12:27:33 > 2023-01-17 23:49:19 2023-03-06 16:48:43 > 2022-11-08 13:06:42 2023-01-17 15:11:47 > 2022-10-08 05:14:46 2022-11-08 09:56:31 > 2022-09-06 15:00:08 2022-10-08 04:15:43 > 2022-08-13 22:02:31 2022-09-06 12:58:52 > =E2=80=A6 > > It means that an user will download ~10 times git-minimal for nothing. > > > Solution #2. The one I am proposing. :-) Download git-minimal only > when Guix needs it for running =E2=80=9Cgit gc=E2=80=9D. Yeah, there is = probably a > small overload with some operations. But, I bet this overload is much > smaller than the one of solution #1. > > Well, it depends on the number of times people are updating the cache vs > the rate of change of git-minimal. > > For sure, if one updates 100 times per week the cache, having > git-minimal as inputs is far better. But I do not think that the > regular usage on average. :-) > > That=E2=80=99s why I am proposing to have an option for turning off this = =E2=80=9Cgit > gc=E2=80=9C operation. > > Well, we have lived since years without running =E2=80=98git gc=E2=80=99 = so running it > once per year on average is probably enough to keep the cache size > reasonable. And git-minimal is changing every month. > > > Maybe, there is some solution #3. ;-) > > Cheers, > simon > > > 1: https://data.guix.gnu.org/repository/1/branch/master/package/git-minim= al/output-history Please don't create another situation like with guix system roll-back, where a crucial sysadmin operation doesn't work without network access. Or at least make it configurable, so things that are likely to be needed for future operations are pre-fetched.