From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <bug-guix-bounces+larch=yhetil.org@gnu.org>
Received: from mp10.migadu.com ([2001:41d0:306:2d92::])
	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits))
	by ms8.migadu.com with LMTPS
	id uFLTCPinMmWYIgAAG6o9tA:P1
	(envelope-from <bug-guix-bounces+larch=yhetil.org@gnu.org>)
	for <larch@yhetil.org>; Fri, 20 Oct 2023 18:16:56 +0200
Received: from aspmx1.migadu.com ([2001:41d0:306:2d92::])
	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits))
	by mp10.migadu.com with LMTPS
	id uFLTCPinMmWYIgAAG6o9tA
	(envelope-from <bug-guix-bounces+larch=yhetil.org@gnu.org>)
	for <larch@yhetil.org>; Fri, 20 Oct 2023 18:16:56 +0200
Received: from lists.gnu.org (lists.gnu.org [209.51.188.17])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(No client certificate requested)
	by aspmx1.migadu.com (Postfix) with ESMTPS id 7D07453B7F
	for <larch@yhetil.org>; Fri, 20 Oct 2023 18:16:55 +0200 (CEST)
Authentication-Results: aspmx1.migadu.com;
	dkim=fail ("headers rsa verify failed") header.d=gnu.org header.s=fencepost-gnu-org header.b="jF0iDL/G";
	spf=pass (aspmx1.migadu.com: domain of "bug-guix-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="bug-guix-bounces+larch=yhetil.org@gnu.org";
	dmarc=pass (policy=none) header.from=gnu.org
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org;
	s=key1; t=1697818616;
	h=from:from:sender:sender:reply-to:subject:subject:date:date:
	 message-id:message-id:to:to:cc:cc:mime-version:mime-version:
	 content-type:content-type:
	 content-transfer-encoding:content-transfer-encoding:resent-cc:
	 resent-from:resent-sender:resent-message-id:in-reply-to:in-reply-to:
	 references:references:list-id:list-help:list-unsubscribe:
	 list-subscribe:list-post:dkim-signature;
	bh=guykA3Jje4B/fdh8d1YKTVE8F+9iXXIl3FUKarfygvY=;
	b=O8+M+lTFXWgjYNgXMNFoQlLZ/GftZ/GVuRD7I4rLpt3QScyEnDJZ9i4TKggtcLcnNDWLI0
	6OvTe3j2eumjV58Eq40YNpnwFb5hSFfoEPHGUttW/H91iKenv6DD9giZQ1HlxcOu/BOEJk
	FNg/gvw3mdXXLg/D7Kbbq9Y5ZnyU3wv++T60GdXsF1Q9cfHtm1/Ib047a/8ocEFVgJUOy6
	Z60SrpTM+AouJJ3Anxv4TAOwqxNBLgobaKrtiSLyZ3D7kC5Eif0AADJy9qjVmDi3lmW19b
	hH7VD4kRhYHciIETYchP15MtFg2doZPkrhZzsCc+aW2CAYgLjvY9BIxwzGxX2A==
ARC-Authentication-Results: i=1;
	aspmx1.migadu.com;
	dkim=fail ("headers rsa verify failed") header.d=gnu.org header.s=fencepost-gnu-org header.b="jF0iDL/G";
	spf=pass (aspmx1.migadu.com: domain of "bug-guix-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="bug-guix-bounces+larch=yhetil.org@gnu.org";
	dmarc=pass (policy=none) header.from=gnu.org
ARC-Seal: i=1; s=key1; d=yhetil.org; t=1697818616; a=rsa-sha256; cv=none;
	b=kiyRtxfIR26t22AGAe8Yp8lcatq5RnJcXzy0m4M8OEuSCQxy0RxIeyXsqT/xC8+xnlUMig
	ekD80wD++Tvhl+/u54882+e4g5NdC8iMX5Ui4LCNTFk0kLLYPDnwB1imEg99Hd7k2TLSTa
	9+sSC7syceBL8VjF/GDrxV7ZNWKHDvcDEOeVsBhkIjULI8IQJDA7Gs8tOWPRGbHqvpaqcg
	1RKhw+CVrinCKHlccFnXMXBMeH6rneUdZb32t9godXjNiNyi8/EMthtXpMNjLPbkJb1sMU
	mBe7+lx0/pvdr6WYM9qpoN/2wHjS+fd0GOZtv5Oj0oxQc8nTFMTJVWFzNpFVqw==
Received: from localhost ([::1] helo=lists1p.gnu.org)
	by lists.gnu.org with esmtp (Exim 4.90_1)
	(envelope-from <bug-guix-bounces@gnu.org>)
	id 1qtsAz-0000a1-MF; Fri, 20 Oct 2023 12:16:41 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10])
 by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <Debian-debbugs@debbugs.gnu.org>)
 id 1qtsAx-0000ZY-AD
 for bug-guix@gnu.org; Fri, 20 Oct 2023 12:16:39 -0400
Received: from debbugs.gnu.org ([2001:470:142:5::43])
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128)
 (Exim 4.90_1) (envelope-from <Debian-debbugs@debbugs.gnu.org>)
 id 1qtsAx-0004cj-0s; Fri, 20 Oct 2023 12:16:39 -0400
Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2)
 (envelope-from <Debian-debbugs@debbugs.gnu.org>)
 id 1qtsBK-0004hV-L9; Fri, 20 Oct 2023 12:17:02 -0400
X-Loop: help-debbugs@gnu.org
Subject: bug#65720: [PATCH] git: Shell out to =?UTF-8?Q?=E2=80=98git_?=
 =?UTF-8?Q?gc=E2=80=99?= when necessary.
Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@gnu.org>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces@debbugs.gnu.org>
Resent-CC: guix@cbaines.net, dev@jpoiret.xyz, ludo@gnu.org, othacehe@gnu.org,
 rekado@elephly.net, zimon.toutoune@gmail.com, me@tobias.gr, bug-guix@gnu.org
Resent-Date: Fri, 20 Oct 2023 16:17:02 +0000
Resent-Message-ID: <handler.65720.B65720.169781857017956@debbugs.gnu.org>
Resent-Sender: help-debbugs@gnu.org
X-GNU-PR-Message: followup 65720
X-GNU-PR-Package: guix
X-GNU-PR-Keywords: 
To: guix-patches@gnu.org
Cc: Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@gnu.org>, 65720@debbugs.gnu.org,
 Josselin Poiret <dev@jpoiret.xyz>, Simon Tournier <zimon.toutoune@gmail.com>,
 Christopher Baines <guix@cbaines.net>, Josselin Poiret <dev@jpoiret.xyz>,
 Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@gnu.org>,
 Mathieu Othacehe <othacehe@gnu.org>, Ricardo Wurmus <rekado@elephly.net>,
 Simon Tournier <zimon.toutoune@gmail.com>,
 Tobias Geerinckx-Rice <me@tobias.gr>
X-Debbugs-Original-Xcc: Christopher Baines <guix@cbaines.net>,
 Josselin Poiret <dev@jpoiret.xyz>,
 Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@gnu.org>,
 Mathieu Othacehe <othacehe@gnu.org>, Ricardo Wurmus <rekado@elephly.net>,
 Simon Tournier <zimon.toutoune@gmail.com>,
 Tobias Geerinckx-Rice <me@tobias.gr>
Received: via spool by 65720-submit@debbugs.gnu.org id=B65720.169781857017956
 (code B ref 65720); Fri, 20 Oct 2023 16:17:02 +0000
Received: (at 65720) by debbugs.gnu.org; 20 Oct 2023 16:16:10 +0000
Received: from localhost ([127.0.0.1]:41301 helo=debbugs.gnu.org)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <debbugs-submit-bounces@debbugs.gnu.org>)
 id 1qtsAT-0004fY-BD
 for submit@debbugs.gnu.org; Fri, 20 Oct 2023 12:16:09 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10]:37666)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <ludo@gnu.org>) id 1qtsAP-0004eq-NV
 for 65720@debbugs.gnu.org; Fri, 20 Oct 2023 12:16:07 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e])
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <ludo@gnu.org>)
 id 1qts9r-0004Rf-PD; Fri, 20 Oct 2023 12:15:32 -0400
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org;
 s=fencepost-gnu-org; h=MIME-Version:References:In-Reply-To:Date:Subject:To:
 From; bh=guykA3Jje4B/fdh8d1YKTVE8F+9iXXIl3FUKarfygvY=; b=jF0iDL/GbSiYleynVOTG
 /8r/H3EkXts9gzZHBgd9R8nNRQDQE9k6G5S40965qhyaaKSTNmvZmEP05W378rPNL287CkVM+Ibc7
 7E32AmsdJM3Am6miqMoPX9YkU5o+Q5z+orHS8Uflfk+Q6GcnHJFgV54Bg6XcAw2I1w6n8BhwLD8XH
 Xc6+syD3NXx+2uTgwx7sgIiSqx8Q8TyYiO+e+VCDOrgg9PMq1m8eYZ6M32W5h/fNQYw3CoJwQsjTq
 N7T/JL+bxKLs4dAkgwLr8izbta894kjMEC5GTNjZ2vBkHDJJ991iz+BbSwyZrUfYucrv//2aKtBun
 iam1wsZbcvK/hA==;
From: Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@gnu.org>
Date: Fri, 20 Oct 2023 18:15:12 +0200
Message-ID: <f588bb38b4b9fdaff29dd8af8c62aa3c55902f7c.1697818202.git.ludo@gnu.org>
X-Mailer: git-send-email 2.41.0
In-Reply-To: <87jzswsrlt.fsf@gnu.org>
References: <87jzswsrlt.fsf@gnu.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
X-BeenThere: debbugs-submit@debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
X-BeenThere: bug-guix@gnu.org
List-Id: Bug reports for GNU Guix <bug-guix.gnu.org>
List-Unsubscribe: <https://lists.gnu.org/mailman/options/bug-guix>,
 <mailto:bug-guix-request@gnu.org?subject=unsubscribe>
List-Archive: <https://lists.gnu.org/archive/html/bug-guix>
List-Post: <mailto:bug-guix@gnu.org>
List-Help: <mailto:bug-guix-request@gnu.org?subject=help>
List-Subscribe: <https://lists.gnu.org/mailman/listinfo/bug-guix>,
 <mailto:bug-guix-request@gnu.org?subject=subscribe>
Errors-To: bug-guix-bounces+larch=yhetil.org@gnu.org
Sender: bug-guix-bounces+larch=yhetil.org@gnu.org
X-Migadu-Country: US
X-Migadu-Flow: FLOW_IN
X-Migadu-Spam-Score: -4.32
X-Spam-Score: -4.32
X-Migadu-Queue-Id: 7D07453B7F
X-Migadu-Scanner: mx2.migadu.com
X-TUID: kHT+h95RZw/O

Fixes <https://issues.guix.gnu.org/65720>.

This fixes a bug whereby libgit2-managed checkouts would keep growing as
we fetch.

* guix/git.scm (packs-in-git-repository, maybe-run-git-gc): New
procedures.
(update-cached-checkout): Use it.
---
 guix/git.scm | 39 ++++++++++++++++++++++++++++++++++++---
 1 file changed, 36 insertions(+), 3 deletions(-)

Hi!

This is a radical fix/workaround for the unbounded Git checkout growth
problem, shelling out to ‘git gc’ when it’s likely needed (“too many”
pack files around).

I thought we might be able to implement a ‘git gc’ approximation using
the libgit2 “packbuilder” interface, but I haven’t got around to doing
it: <https://libgit2.org/libgit2/#HEAD/search/pack>.

Once again, shelling out is not my favorite option, but it’s a bug we
should fix sooner rather than later, hence this compromise.

Thoughts?

Ludo’.

diff --git a/guix/git.scm b/guix/git.scm
index b7182305cf..d704b62333 100644
--- a/guix/git.scm
+++ b/guix/git.scm
@@ -1,6 +1,6 @@
 ;;; GNU Guix --- Functional package management for GNU
 ;;; Copyright © 2017, 2020 Mathieu Othacehe <m.othacehe@gmail.com>
-;;; Copyright © 2018-2022 Ludovic Courtès <ludo@gnu.org>
+;;; Copyright © 2018-2023 Ludovic Courtès <ludo@gnu.org>
 ;;; Copyright © 2021 Kyle Meyer <kyle@kyleam.com>
 ;;; Copyright © 2021 Marius Bakke <marius@gnu.org>
 ;;; Copyright © 2022 Maxime Devos <maximedevos@telenet.be>
@@ -29,15 +29,16 @@ (define-module (guix git)
   #:use-module (guix cache)
   #:use-module (gcrypt hash)
   #:use-module ((guix build utils)
-                #:select (mkdir-p delete-file-recursively))
+                #:select (mkdir-p delete-file-recursively invoke/quiet))
   #:use-module (guix store)
   #:use-module (guix utils)
   #:use-module (guix records)
   #:use-module (guix gexp)
   #:autoload   (guix git-download)
   (git-reference-url git-reference-commit git-reference-recursive?)
+  #:autoload   (guix config) (%git)
   #:use-module (guix sets)
-  #:use-module ((guix diagnostics) #:select (leave warning))
+  #:use-module ((guix diagnostics) #:select (leave warning info))
   #:use-module (guix progress)
   #:autoload   (guix swh) (swh-download commit-id?)
   #:use-module (rnrs bytevectors)
@@ -428,6 +429,35 @@ (define (delete-checkout directory)
     (rename-file directory trashed)
     (delete-file-recursively trashed)))
 
+(define (packs-in-git-repository directory)
+  "Return the number of pack files under DIRECTORY, a Git checkout."
+  (catch 'system-error
+    (lambda ()
+      (let ((directory (opendir (in-vicinity directory ".git/objects/pack"))))
+        (let loop ((count 0))
+          (match (readdir directory)
+            ((? eof-object?)
+             (closedir directory)
+             count)
+            (str
+             (loop (if (string-suffix? ".pack" str)
+                       (+ 1 count)
+                       count)))))))
+    (const 0)))
+
+(define (maybe-run-git-gc directory)
+  "Run 'git gc' in DIRECTORY if needed."
+  ;; XXX: As of libgit2 1.3.x (used by Guile-Git), there's no support for GC.
+  ;; Each time a checkout is pulled, a new pack is created, which eventually
+  ;; takes up a lot of space (lots of small, poorly-compressed packs).  As a
+  ;; workaround, shell out to 'git gc' when the number of packs in a
+  ;; repository has become "too large", potentially wasting a lot of space.
+  ;; See <https://issues.guix.gnu.org/65720>.
+  (when (> (packs-in-git-repository directory) 25)
+    (info (G_ "compressing cached Git repository at '~a'...~%")
+          directory)
+    (invoke/quiet %git "-C" directory "gc")))
+
 (define* (update-cached-checkout url
                                  #:key
                                  (ref '())
@@ -515,6 +545,9 @@ (define* (update-cached-checkout url
                    seconds seconds
                    nanoseconds nanoseconds))))
 
+       ;; Run 'git gc' if needed.
+       (maybe-run-git-gc cache-directory)
+
        ;; When CACHE-DIRECTORY is a sub-directory of the default cache
        ;; directory, remove expired checkouts that are next to it.
        (let ((parent (dirname cache-directory)))

base-commit: 6b0a32196982a0a2f4dbb59d35e55833a5545ac6
-- 
2.41.0