From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp10.migadu.com ([2001:41d0:403:4789::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms9.migadu.com with LMTPS id iAgQD+OZ/2TJQwAAG6o9tA:P1 (envelope-from ) for ; Tue, 12 Sep 2023 00:51:15 +0200 Received: from aspmx1.migadu.com ([2001:41d0:403:4789::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp10.migadu.com with LMTPS id iAgQD+OZ/2TJQwAAG6o9tA (envelope-from ) for ; Tue, 12 Sep 2023 00:51:15 +0200 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id EB9D3596CF for ; Tue, 12 Sep 2023 00:51:14 +0200 (CEST) Authentication-Results: aspmx1.migadu.com; dkim=pass header.d=gmail.com header.s=20221208 header.b=SPGL8u3i; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org"; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1694472675; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post:dkim-signature; bh=heDurgAZltfbJ6Qrb53yEnmdFGl0Jtt+h66ClIEisuQ=; b=AZo8yf0lcF19fAEAxskoeAWNYvm9DRrcEF2Ra0LUep0BFkGbhLiJR5ADibZI0yd2+aZ0fC IAj7xnPRAO24Jh3vo6/85JhvxqsJ7pKv896ioXJxQi1Iq40qV13pyiuTj75sRDNnDNQBcy v4fCVTasQRp475HM/nb+y95S9NVhNnkNpVWcLx58KUv6OnZvogdOIDTrRC/HaYVJGx2RYC a+B1sj7PHLZaWLfrc4kLhPuL4YezyF1rRiMKhg5U005BFIQ3ZJcZvFdH0xuLlA0FFgsmJ+ 5mEKNyhDHtOUWJzhkTjxw4OPlb0/EUiZwfFAnMQvioGvSLhBXGy0AggsGCwxRg== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=pass header.d=gmail.com header.s=20221208 header.b=SPGL8u3i; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org"; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=key1; d=yhetil.org; t=1694472675; a=rsa-sha256; cv=none; b=C8QECqqD6RtviW8XwsPuhjd+yTkfEvRZZ9JZtL8xrz6FcsVLQw0M3XFlMyYtvXYLv04Uz9 fs+7glq2hVWzEjLiJZ+N8mDXr6Orzn7PwZ6jMcG//lm+tgMQEgs4uqQ9oydI0lRM9BOwQ8 viI6jIEKsxwb5pJv9x7n1bC2Nc/lyVpK0VPx1EjvOP2jw8ARnySdPsslp/scxMZLvIbEpw AIZ25KwyBVGP9wOu8fb6Eb4w6MvnhhWUFkn8+rRY1g3n/EneU9GtVnV9FD8RpJFua3613a ifpGVOSLdHftf9xed4rsGhfxHXNNOQNCkGOhBb15/jI/kpYQ/boP9KWAdpj2ZA== Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qfpjs-0008Fq-Aq; Mon, 11 Sep 2023 18:50:40 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qfpjq-0008Fh-Du for guix-devel@gnu.org; Mon, 11 Sep 2023 18:50:38 -0400 Received: from mail-wr1-x42e.google.com ([2a00:1450:4864:20::42e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1qfpjn-0000Ge-Pi; Mon, 11 Sep 2023 18:50:38 -0400 Received: by mail-wr1-x42e.google.com with SMTP id ffacd0b85a97d-3159b524c56so678276f8f.1; Mon, 11 Sep 2023 15:50:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1694472633; x=1695077433; darn=gnu.org; h=content-transfer-encoding:mime-version:message-id:date:references :in-reply-to:subject:cc:to:from:from:to:cc:subject:date:message-id :reply-to; bh=heDurgAZltfbJ6Qrb53yEnmdFGl0Jtt+h66ClIEisuQ=; b=SPGL8u3i44/xntoqON7534Jud5oou/c9jyxnHLyTrh4a9ZHQeKZx+P2mYy9B/oatwm mh0Q4eQm3DrDKvQqeEHQ1KSy7tkEdeWRXg7SnlMEbPSHCgVbgkWRLjtU/KHhGXthyRtj XEQ52DKRsU5XTMqrVFM1ZGjm3pN3zKSHjktfVJ70Vwv3FDCvsIR3eaQntdEctDGiupOm w+2dkqvMUPBQhsX3Y7IAhg51xf1zJus5uLp9hlsY9Uximf4TbVdnkhzY9VxDvV+qXt7e ABBYh+qj9TCcNX5RSv0UvMjiX7wnLOa1NoTiNfLsj7BB3mV+uDtoAGbGv2evG1rzPx2y 94Ew== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1694472633; x=1695077433; h=content-transfer-encoding:mime-version:message-id:date:references :in-reply-to:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=heDurgAZltfbJ6Qrb53yEnmdFGl0Jtt+h66ClIEisuQ=; b=ix0nyNcBj8gFNZOi3lCz+JcbGIKyX0M2NKnyM2lILxIcvUuqB8WgXhbxtT8wIxg89V 9w2PXgplkPs+39lZBvh+Es0GLccaPeKF4s8QghG3vcpQUsr0PwxyL+lAkarCEKtlK/gR krnYxYHZLtg8NeXNymsgukdn9E/hrbb1m4giOjlRsnNq5C7qpO12GU+cU81i6T9+xaZL TcLRxZ6L02FpqcLMXZObyzl26IoJsFF3bm1Wnwczptc78D8dFCVbzIvgdJx9jVkr14ch zuiZhDNHnBDp3pa04FuqtsnxCluE6ZWcKmndkQPi+wWgyfitRc64rNVPvO4+hxboR6X3 QKhw== X-Gm-Message-State: AOJu0Yya1/79u2/I1oshr9rfxf5WE+bSny0mX0VRdoXSjUZ3iMReAKA7 mqC2L+uZ5sa6RjFe5/bZWA2pFqv+/ug= X-Google-Smtp-Source: AGHT+IEsPfid9nhgNGFNkbutdvKIzd6ovp3dpGXUnMJnW8EgHUVcEd6UYjc2trtC02jDgtPUXWGc+w== X-Received: by 2002:adf:e60f:0:b0:31a:ed75:75d4 with SMTP id p15-20020adfe60f000000b0031aed7575d4mr8103775wrm.2.1694472632522; Mon, 11 Sep 2023 15:50:32 -0700 (PDT) Received: from lili ([2a01:e0a:59b:9120:65d2:2476:f637:db1e]) by smtp.gmail.com with ESMTPSA id c1-20020a5d4cc1000000b0031accc7228asm11109539wrt.34.2023.09.11.15.50.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 11 Sep 2023 15:50:32 -0700 (PDT) From: Simon Tournier To: Maxim Cournoyer Cc: guix-devel@gnu.org, Ludovic =?utf-8?Q?Court=C3=A8s?= Subject: comparing commit-relation using Scheme+libgit2 vs shellout plumbing Git In-Reply-To: <87msxswoph.fsf@gmail.com> References: <871qf4ivsr.fsf@gmail.com> <87msxswoph.fsf@gmail.com> Date: Tue, 12 Sep 2023 00:48:30 +0200 Message-ID: <865y4gz5q9.fsf@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Received-SPF: pass client-ip=2a00:1450:4864:20::42e; envelope-from=zimon.toutoune@gmail.com; helo=mail-wr1-x42e.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: guix-devel-bounces+larch=yhetil.org@gnu.org X-Migadu-Flow: FLOW_IN X-Migadu-Country: US X-Migadu-Scanner: mx1.migadu.com X-Migadu-Spam-Score: -9.57 X-Spam-Score: -9.57 X-Migadu-Queue-Id: EB9D3596CF X-TUID: vS/R0LGP7/Q5 Hi, On Mon, 11 Sep 2023 at 14:26, Maxim Cournoyer w= rote: > In the grand scheme of things (pun intended), we'd like every > programming to be feasible via nice Scheme APIs, which is what Guile-Git > provides to work with git repositories. The appeal is to have a single > language to rule them all, reducing friction among Guix contributors. > The alternative here is to have an API reduced to invoking system > commands with string arguments, which is less expressive and lacks > elegance. As Maxim noticed in the message that I am proposing to revisit, it seems that libgit2 comes with some performance penalties. As wolf is illustrating in the message: bug#65720: Guile-Git-managed checkouts grow way too much wolf Mon, 11 Sep 2023 16:42:59 +0200 id:ZP8nc1m8rN_34XV-@ws https://issues.guix.gnu.org//65720 https://issues.guix.gnu.org/msgid/ZP8nc1m8rN_34XV-@ws https://yhetil.org/guix/ZP8nc1m8rN_34XV-@ws it might be possible to use an invocation of plain Git command which is much faster in this case. Well, that=E2=80=99s need to be investigated, IM= HO. For instance, instead of the current =E2=80=99commit-relation=E2=80=99 impl= ementation, (define (commit-relation old new) "Return a symbol denoting the relation between OLD and NEW, two c= ommit objects: 'ancestor (meaning that OLD is an ancestor of NEW), 'desce= ndant, or 'unrelated, or 'self (OLD and NEW are the same commit)." (if (eq? old new) 'self (let ((newest (commit-closure new))) (if (set-contains? newest old) 'ancestor (let* ((seen (list->setq (commit-parents new))) (oldest (commit-closure old seen))) (if (set-contains? oldest new) 'descendant 'unrelated)))))) which relies on =E2=80=99commit-closure=E2=80=99, they propose to use a plu= mbing Git commands, as: (define (shelling-commit-relation old new) (let ((h-old (oid->string (commit-id old))) (h-new (oid->string (commit-id new)))) (cond ((eq? old new) 'self) ((zero? (git-C %repo "merge-base" "--is-ancestor" h-old h= -new)) 'ancestor) ((zero? (git-C %repo "merge-base" "--is-ancestor" h-new h= -old)) 'descendant) (else 'unrelated)))) Well, this needs to be checked (read the Git documentation which is probable harder than read some Scheme implementation ;-)) in order to see if these invocations are doing the same. >> I=E2=80=99m quite confident this would be slow > > My version is ~2000x faster compared to (guix git): > > Guix: 1048.620992ms > Git: 0.532143ms On my machine, I get something less spectacular for a history with 1000 commits in between. --8<---------------cut here---------------start------------->8--- scheme@(guix-user)> ,time (commit-relation* 1000th newest) $1 =3D ancestor ;; 0.128948s real time, 0.082921s run time. 0.046578s spent in GC. scheme@(guix-user)> ,time (commit-relation 1000th newest) $2 =3D ancestor ;; 4.588075s real time, 5.521358s run time. 1.404764s spent in GC. --8<---------------cut here---------------end--------------->8--- I did something very similar as wolf is proposing and named it =E2=80=99commit-relation*=E2=80=99. Well, considering the implementation of =E2=80=99commit-relation=E2=80=99, = I think the slowness is expected compared to the plain plumbing Git command. Basically, =E2=80=99commit-closure=E2=80=99 walks the Git history and for s= ure the loop cannot be as efficient as an optimized Git specific implementation. Hum, I think the most annoying is the time spent in GC. Basically, =E2=80=99commit-closure=E2=80=99 is building a set with many visited elemen= ts and that set must be garbage collected. And this GC time is not nothing compared to the whole time, IMHO. I agree with the grand scheme of things and that=E2=80=99s why I started th= is thread. :-) However, for what it is worth, today I am less convinced that manipulating libgit2 is able to provide =E2=80=9Cnot-so-worse=E2=80=9D= performance compared to what plain plumbing Git commands could offer. Cheers, simon