From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp10.migadu.com ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms5.migadu.com with LMTPS id qK5ZMTcyfGPn/QAAbAwnHQ (envelope-from ) for ; Tue, 22 Nov 2022 03:21:43 +0100 Received: from aspmx1.migadu.com ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp10.migadu.com with LMTPS id IKFeMDcyfGNaEQEAG6o9tA (envelope-from ) for ; Tue, 22 Nov 2022 03:21:43 +0100 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 4B69819859 for ; Tue, 22 Nov 2022 03:21:43 +0100 (CET) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1oxIuN-0005Vg-L9; Mon, 21 Nov 2022 21:21:11 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1oxIuL-0005VS-PU for guix-devel@gnu.org; Mon, 21 Nov 2022 21:21:09 -0500 Received: from mail-qk1-x732.google.com ([2607:f8b0:4864:20::732]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1oxIuH-0004zk-RZ for guix-devel@gnu.org; Mon, 21 Nov 2022 21:21:09 -0500 Received: by mail-qk1-x732.google.com with SMTP id d7so9387390qkk.3 for ; Mon, 21 Nov 2022 18:21:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=mime-version:message-id:date:subject:to:from:from:to:cc:subject :date:message-id:reply-to; bh=wYXgGjV6Z6ISfTxsML3FWVgkMqvs03G4prrv0jyoxDI=; b=P4eLOLI2rOxmhpVzBmdaH05tLIo5oEDuTA6lRzzSqEfrWo4A3hiZHipyQVU63/sg6Q JxtNJJkImCCQnnFqwD/N0Ly4JUZt20AupnozHXnIeEhyH5K7YD18iDTcxNja3OQURJTp OU8z1n6uZy4iD7/KtZSgTHojzTb3otLaVgxbA3L6zpr/UwjdB29jojgKpiGK9mtjOa1L MD1W7AsoTzD1w3utp+QwQNZ25GirDL/dOC66wPQ/yVTI365yiK7yc2IdYj6fAuxO4gAk fALqkPYl0N1rjjj0QKgZjtv9IOl+a+Vyl1NKSK2fv6nyDXAqoFgz6yBxvlpoYEBGboA6 dUPg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=mime-version:message-id:date:subject:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=wYXgGjV6Z6ISfTxsML3FWVgkMqvs03G4prrv0jyoxDI=; b=lh1Xp+4jwQFqnGxXbfwgtn+Uo0GOqC66mDHvfDawV6N2rbFnFVG+2Sy93Hfa7fiAZg OWu+V1iNMsXy6yvYGmANx50cfhH8rq8X484P0vb7PNty+w75YS8phHO8O6X4R5zFJt3j XVS+i2I+JP984at9qHXbBqUg+2psFghrbA0AqWE3OFPc0xS2GkyE4pPPYE13PLlwGA/n /ZoxU6QnQlERX2Wsm/1lTrkG1L8x2M6XUuXqXETy4ba2hp628brkHwpGk54lE8VhX+UG HJ1FTdlUMyTLJqXKkeEsmjOtkFIvJVKNPUFvZ0i1rhDJzWqARdolk5F5LefGsW806tPR 3J6g== X-Gm-Message-State: ANoB5pm/SSRkQDfCJT4dFbQVL7E/B+EJe1iEfV2MpWm2PjySbvg3iEr3 hktjtexiPnvKOWAej482K5eGX/VIZz4= X-Google-Smtp-Source: AA0mqf7UzHQisjiiXBtwum/IhN8pxR9AF/p//8dEfQMXzhvNsk2lchIDAjVhrJSQKve1uMg72UpLEQ== X-Received: by 2002:a37:8683:0:b0:6fa:121:42f7 with SMTP id i125-20020a378683000000b006fa012142f7mr18933821qkd.283.1669083664359; Mon, 21 Nov 2022 18:21:04 -0800 (PST) Received: from hurd ([2607:fad8:4:3::1003]) by smtp.gmail.com with ESMTPSA id m19-20020a05620a291300b006fa43e139b5sm9280008qkp.59.2022.11.21.18.21.03 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 21 Nov 2022 18:21:03 -0800 (PST) From: Maxim Cournoyer To: guix-devel Subject: RFC: libgit2 is slow/inefficient; switch to git command? Date: Mon, 21 Nov 2022 21:21:02 -0500 Message-ID: <87cz9fpw4x.fsf@gmail.com> MIME-Version: 1.0 Content-Type: text/plain Received-SPF: pass client-ip=2607:f8b0:4864:20::732; envelope-from=maxim.cournoyer@gmail.com; helo=mail-qk1-x732.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: guix-devel-bounces+larch=yhetil.org@gnu.org X-Migadu-Flow: FLOW_IN X-Migadu-Country: US ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1669083703; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version: content-type:content-type:list-id:list-help:list-unsubscribe: list-subscribe:list-post:dkim-signature; bh=wYXgGjV6Z6ISfTxsML3FWVgkMqvs03G4prrv0jyoxDI=; b=aCDcZGsp+Z8lS0Ra3IF7FuclZQwU4sXSk9NKZFtdqwTG8CFN9SDdTekIoGRk+S/3N6vXaE lePGPhrjWjflngr9MtpnRCavRmpVmPGVjK7fFrMdfbK7II/J1YhlQ6TyEUI/FA5qvsYCYD Xr/6+CkHtmUuoELlsPlk1wPlJr7Uv9WHpI7ocnSK+8GByLOLeLmEQRmarBYenzG1m74CQk Pj6hH8Sr5rCHKgZsaw0/+PCotLzUmLnI46mSusEvK40OJ9JHTA738RbhFV6RNtuWLXa6+2 UvGFsrS/QPtqQuQGSy0FeTcy2VrFk8FAIYFuQFUk7iK5uh1nLrfSqjqL8hWmUQ== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1669083703; a=rsa-sha256; cv=none; b=RJpn+Ou+7g8sMbLLtpDmehUQWIw6k9V0S0bu9waSkvft7k8lw/F2+trMV3ypsIgFpj0sZC Wg3iXocMkLD28THl6GJF1fknc+KUPLhzlGk2pU31Zl+GF2UangWaFP9OPOu15CStEjsDMV 0wJBwZOqVaDgE+dYKF1+qvIo+mN1hLKTmt6AxFZViUlk0WsJSregvwR7XKc/UtIgeZswD5 Wc3l/p3PLjEdYyzsX2161dJ7e+VnZgzSTMb8YFdX2OKXkvAX+/T+v1kb2incZcGCUe4Zbx 4flS78YH37jlnCq7IUqRBRQw739K3rPKa5+LZFuTHDvYFQ+88B15eb0qyvWPoA== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=P4eLOLI2; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org" X-Migadu-Spam-Score: -5.81 Authentication-Results: aspmx1.migadu.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=P4eLOLI2; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org" X-Migadu-Queue-Id: 4B69819859 X-Spam-Score: -5.81 X-Migadu-Scanner: scn0.migadu.com X-TUID: dM9DRYebNajQ Hi, While attempting to bisect against the Linux kernel tree, the performance of libgit2 quickly became problematic, to the point where simply cloning the repo became a multiple hours affair, using upward to 3 GiB of RAM for the clone and indexing of the objects (!) Given that: * the git CLI doesn't suffer from such poor performance; * This kind of performance problem has been known for years in libgit2 [0] with no fix in sight; * other projects such as Cargo support using the git CLI and that projects are using it for that reason [1]; Would it make sense to switch to use the git command directly instead of calling into this libgit2 C library that ends up being slower? It would provide a hefty speed-up when using 'guix refresh' or building new packages fetched from git without substitutes, or using 'git-checkout', etc. What do you think? [0] https://github.com/libgit2/libgit2/issues/4674 [1] https://github.com/artichoke/playground/pull/700 -- Thanks, Maxim