From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp0 ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id 8ElgGmxtO2H6GQAAgWs5BA (envelope-from ) for ; Fri, 10 Sep 2021 16:36:28 +0200 Received: from aspmx1.migadu.com ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp0 with LMTPS id 4G0tFmxtO2E3LgAA1q6Kng (envelope-from ) for ; Fri, 10 Sep 2021 14:36:28 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 91477F93C for ; Fri, 10 Sep 2021 16:36:27 +0200 (CEST) Received: from localhost ([::1]:53006 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mOhdi-000079-Eb for larch@yhetil.org; Fri, 10 Sep 2021 10:36:26 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:43004) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mOhdK-00083V-9n for bug-guix@gnu.org; Fri, 10 Sep 2021 10:36:05 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:55624) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1mOhdK-0007dp-1V for bug-guix@gnu.org; Fri, 10 Sep 2021 10:36:02 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1mOhdJ-0000ep-TG for bug-guix@gnu.org; Fri, 10 Sep 2021 10:36:01 -0400 X-Loop: help-debbugs@gnu.org Subject: bug#44187: [PATCH 0/3] Fall back to Software Heritage (SWH) for Git clones Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: bug-guix@gnu.org Resent-Date: Fri, 10 Sep 2021 14:36:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 44187 X-GNU-PR-Package: guix X-GNU-PR-Keywords: To: 44187@debbugs.gnu.org Received: via spool by 44187-submit@debbugs.gnu.org id=B44187.16312845122449 (code B ref 44187); Fri, 10 Sep 2021 14:36:01 +0000 Received: (at 44187) by debbugs.gnu.org; 10 Sep 2021 14:35:12 +0000 Received: from localhost ([127.0.0.1]:38931 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mOhcR-0000d5-48 for submit@debbugs.gnu.org; Fri, 10 Sep 2021 10:35:12 -0400 Received: from eggs.gnu.org ([209.51.188.92]:53748) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mOhcP-0000cN-8q for 44187@debbugs.gnu.org; Fri, 10 Sep 2021 10:35:06 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:47276) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mOhcJ-0006fG-Ad; Fri, 10 Sep 2021 10:34:59 -0400 Received: from [2001:660:6102:320:e120:2c8f:8909:cdfe] (port=47222 helo=gnu.org) by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1mOhcI-0006Ir-UV; Fri, 10 Sep 2021 10:34:59 -0400 From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Date: Fri, 10 Sep 2021 16:34:12 +0200 Message-Id: <20210910143415.14783-1-ludo@gnu.org> X-Mailer: git-send-email 2.33.0 In-Reply-To: <87pn0dk61v.fsf@gnu.org> References: <87pn0dk61v.fsf@gnu.org> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-guix@gnu.org List-Id: Bug reports for GNU Guix List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-guix-bounces+larch=yhetil.org@gnu.org Sender: "bug-Guix" X-Migadu-Flow: FLOW_IN ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1631284587; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:resent-cc: resent-from:resent-sender:resent-message-id:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post; bh=acD69Dj06rOOzl+x3v2POV6ptxZOuo876EaSo9K63mw=; b=ha9kT7BhI5u2aTMeCIURf7gxhXMqB3xQmXptXdPD2rTFsFc3vu/+Wy5O1nQ1XaO6eNnRM/ veFgKWnZq3unIuHK7CerdkywE3bicBPoRanPuhHPVSnAcGve0VL3FURFKBBG4nIo0D/aYU lsREXzv5ikU2wVZXrFPSYryYg8oZjJdPbGBcKfUoCgtvfxzryTbjxNyvryiwHQlXQOKOL9 rSM/DYfE9vTdXK1+bwFEhNa6Mgrra41bWJKcM/EXNcExkn0xLlAlzOCLPifxeHGFPjLv9z Jh7+HeyVVT8f2rNeiE2isIM1KpeR+Ynhl0dTpHwpah9Z47nPdZ7UDREHMqLahg== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1631284587; a=rsa-sha256; cv=none; b=kS0LkljbdegISOyMlxwG2SFEzTgAnC0AuxJHM7iam5/QzLNC9vWViyNqTUxbf7NbP8PbyP hdjd8uOeh/t41UL/19IgcKsHXjl5hODtyFA5HFLaO6KEfY6ti+UD0SgRJAPU3XWqcFFC10 yCgKtQ6rHx9LRbCEG7zesQYhIxwneH3UX+9KVLb6OHEAbLcXNWFBT8kpd7gvsIeiw47Wp0 KnTluWVwD8ALe9f+BE/KqagM7/LOxkE5N5TVjuEm3JKzASu0ucyFHajzHMibjzgp5UfXXL kymynusrJPvFimwubtf0a8B9m01bIlGtXuvnsCoNLkEB9MUI4xqFJm+tJt5GtA== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=none; dmarc=pass (policy=none) header.from=gnu.org; spf=pass (aspmx1.migadu.com: domain of bug-guix-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=bug-guix-bounces@gnu.org X-Migadu-Spam-Score: -0.41 Authentication-Results: aspmx1.migadu.com; dkim=none; dmarc=pass (policy=none) header.from=gnu.org; spf=pass (aspmx1.migadu.com: domain of bug-guix-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=bug-guix-bounces@gnu.org X-Migadu-Queue-Id: 91477F93C X-Spam-Score: -0.41 X-Migadu-Scanner: scn0.migadu.com X-TUID: AiA7TMw3CAjl Hi! A bit of context: we already had automatic SWH fallback for Git checkouts, which is to say that any origin that uses ‘git-fetch’ would have its checkout transparently fetched from SWH if upstream vanished (this dates back to commit 608d3dca89d73fe7260e97a284a8aeea756a3e11, Nov. 2018). What this patch series provides is SWH fallback for full Git clones (as opposed to flat checkouts). It works for anything that uses (guix git). That includes , used by transformation options: --8<---------------cut here---------------start------------->8--- $ ./pre-inst-env guix build footswitch --with-git-url=footswitch=http://example.org/sdf --with-commit=footswitch=1eabc563ca5692b3e08d84f1f0e6fd2283284469 -n updating checkout of 'http://example.org/sdf'... SWH: found revision 1eabc563ca5692b3e08d84f1f0e6fd2283284469 with directory at 'https://archive.softwareheritage.org/api/1/directory/ad8976564375ee55f645387bbcdf4b66e6582fbf/' swh:1:rev:1eabc563ca5692b3e08d84f1f0e6fd2283284469.git/ swh:1:rev:1eabc563ca5692b3e08d84f1f0e6fd2283284469.git/HEAD swh:1:rev:1eabc563ca5692b3e08d84f1f0e6fd2283284469.git/branches/ swh:1:rev:1eabc563ca5692b3e08d84f1f0e6fd2283284469.git/config swh:1:rev:1eabc563ca5692b3e08d84f1f0e6fd2283284469.git/description swh:1:rev:1eabc563ca5692b3e08d84f1f0e6fd2283284469.git/hooks/ swh:1:rev:1eabc563ca5692b3e08d84f1f0e6fd2283284469.git/hooks/applypatch-msg.sample swh:1:rev:1eabc563ca5692b3e08d84f1f0e6fd2283284469.git/hooks/commit-msg.sample swh:1:rev:1eabc563ca5692b3e08d84f1f0e6fd2283284469.git/hooks/fsmonitor-watchman.sample swh:1:rev:1eabc563ca5692b3e08d84f1f0e6fd2283284469.git/hooks/post-update.sample swh:1:rev:1eabc563ca5692b3e08d84f1f0e6fd2283284469.git/hooks/pre-applypatch.sample swh:1:rev:1eabc563ca5692b3e08d84f1f0e6fd2283284469.git/hooks/pre-commit.sample swh:1:rev:1eabc563ca5692b3e08d84f1f0e6fd2283284469.git/hooks/pre-push.sample swh:1:rev:1eabc563ca5692b3e08d84f1f0e6fd2283284469.git/hooks/pre-rebase.sample swh:1:rev:1eabc563ca5692b3e08d84f1f0e6fd2283284469.git/hooks/pre-receive.sample swh:1:rev:1eabc563ca5692b3e08d84f1f0e6fd2283284469.git/hooks/prepare-commit-msg.sample swh:1:rev:1eabc563ca5692b3e08d84f1f0e6fd2283284469.git/hooks/update.sample swh:1:rev:1eabc563ca5692b3e08d84f1f0e6fd2283284469.git/info/ swh:1:rev:1eabc563ca5692b3e08d84f1f0e6fd2283284469.git/info/exclude swh:1:rev:1eabc563ca5692b3e08d84f1f0e6fd2283284469.git/info/refs swh:1:rev:1eabc563ca5692b3e08d84f1f0e6fd2283284469.git/objects/ swh:1:rev:1eabc563ca5692b3e08d84f1f0e6fd2283284469.git/objects/info/ swh:1:rev:1eabc563ca5692b3e08d84f1f0e6fd2283284469.git/objects/info/packs swh:1:rev:1eabc563ca5692b3e08d84f1f0e6fd2283284469.git/objects/pack/ swh:1:rev:1eabc563ca5692b3e08d84f1f0e6fd2283284469.git/objects/pack/pack-ed28f44a2599fe2d0a5f1b1a84c247c43afd14a1.idx swh:1:rev:1eabc563ca5692b3e08d84f1f0e6fd2283284469.git/objects/pack/pack-ed28f44a2599fe2d0a5f1b1a84c247c43afd14a1.pack swh:1:rev:1eabc563ca5692b3e08d84f1f0e6fd2283284469.git/refs/ swh:1:rev:1eabc563ca5692b3e08d84f1f0e6fd2283284469.git/refs/heads/ swh:1:rev:1eabc563ca5692b3e08d84f1f0e6fd2283284469.git/refs/heads/master swh:1:rev:1eabc563ca5692b3e08d84f1f0e6fd2283284469.git/refs/tags/ retrieved commit 1eabc563ca5692b3e08d84f1f0e6fd2283284469 substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0% substitute: updating substitutes from 'https://bayfront.guix.gnu.org'... 100.0% The following derivation would be built: /gnu/store/39kzsy5kgj5150q6zgckc2hbxp999adw-footswitch-git.1eabc56.drv --8<---------------cut here---------------end--------------->8--- In the example above, we pass a bogus Git URL, but since the target commit is known, (guix git) automatically fetches a bare Git repository from the SWH vault. It also works for channels, which is what zimoun reported here: --8<---------------cut here---------------start------------->8--- $ cat /tmp/chan.scm (list (channel (name 'guix) (url "https://git.savannah.gnu.org/git/guix.git") (commit "f91ae9425bb385b60396a544afe27933896b8fa3") (introduction (make-channel-introduction "9edb3f66fd807b096b48283debdcddccfea34bad" (openpgp-fingerprint "BBB0 2DDF 2CEA F6A8 0D1D E643 A2A0 6DF2 A33A 54FA")))) (channel (name 'guix-past) (url "https://does-not-exist.inria.fr/guix-hpc/guix-past") (commit "77e183dc7ade307ad3409fad4b71f12e266de910") #;(introduction (make-channel-introduction "0c119db2ea86a389769f4d2b9c6f5c41c027e336" (openpgp-fingerprint "3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5"))))) $ ./pre-inst-env guix time-machine -C /tmp/chan.scm -- describe Updating channel 'guix' from Git repository at 'https://git.savannah.gnu.org/git/guix.git'... Updating channel 'guix-past' from Git repository at 'https://does-not-exist.inria.fr/guix-hpc/guix-past'... SWH: found revision 77e183dc7ade307ad3409fad4b71f12e266de910 with directory at 'https://archive.softwareheritage.org/api/1/directory/7c6aa10e1e0fa54199566145c6a453731872b87d/' swh:1:rev:77e183dc7ade307ad3409fad4b71f12e266de910.git/ swh:1:rev:77e183dc7ade307ad3409fad4b71f12e266de910.git/HEAD swh:1:rev:77e183dc7ade307ad3409fad4b71f12e266de910.git/branches/ swh:1:rev:77e183dc7ade307ad3409fad4b71f12e266de910.git/config swh:1:rev:77e183dc7ade307ad3409fad4b71f12e266de910.git/description swh:1:rev:77e183dc7ade307ad3409fad4b71f12e266de910.git/hooks/ swh:1:rev:77e183dc7ade307ad3409fad4b71f12e266de910.git/info/ swh:1:rev:77e183dc7ade307ad3409fad4b71f12e266de910.git/info/exclude swh:1:rev:77e183dc7ade307ad3409fad4b71f12e266de910.git/info/refs swh:1:rev:77e183dc7ade307ad3409fad4b71f12e266de910.git/objects/ swh:1:rev:77e183dc7ade307ad3409fad4b71f12e266de910.git/objects/info/ swh:1:rev:77e183dc7ade307ad3409fad4b71f12e266de910.git/objects/info/packs swh:1:rev:77e183dc7ade307ad3409fad4b71f12e266de910.git/objects/pack/ swh:1:rev:77e183dc7ade307ad3409fad4b71f12e266de910.git/objects/pack/pack-e6c0a4813509178eed735708dd60503353a50b9c.idx swh:1:rev:77e183dc7ade307ad3409fad4b71f12e266de910.git/objects/pack/pack-e6c0a4813509178eed735708dd60503353a50b9c.pack swh:1:rev:77e183dc7ade307ad3409fad4b71f12e266de910.git/refs/ swh:1:rev:77e183dc7ade307ad3409fad4b71f12e266de910.git/refs/heads/ swh:1:rev:77e183dc7ade307ad3409fad4b71f12e266de910.git/refs/heads/master swh:1:rev:77e183dc7ade307ad3409fad4b71f12e266de910.git/refs/tags/ Computing Guix derivation for 'x86_64-linux'... \ C-c C-c --8<---------------cut here---------------end--------------->8--- Here, the ‘guix-past’ channel is transparently cloned from SWH. This is pretty cool, because having the whole repo around is what permits things like downgrade prevention¹ and news support². Finally we can enjoy content-addressability and brittle URLs are becoming a thing of the past!* Limitations ~~~~~~~~~~~~ Yes, there’s a couple of them. First, fallback is implemented only for fresh clones, not for updates. Thus, if I rerun the first example, having now the clone in ~/.cache/guix/checkouts, with a different commit, I get: --8<---------------cut here---------------start------------->8--- $ ./pre-inst-env guix build footswitch --with-git-url=footswitch=http://example.org/sdf --with-commit=footswitch=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa -n updating checkout of 'http://example.org/sdf'... guix build: error: Git failure while fetching http://example.org/sdf: unexpected http status code: 404 --8<---------------cut here---------------end--------------->8--- Second, clones from SWH only contain the one branch that the revision is on. For channels, that means that the ‘keyring’ branch is not fetched, which is why I commented out ‘introduction’ in /tmp/chan.scm above. If I uncomment it, I get: --8<---------------cut here---------------start------------->8--- $ ./pre-inst-env guix time-machine -C /tmp/chan.scm -- describe Updating channel 'guix' from Git repository at 'https://git.savannah.gnu.org/git/guix.git'... Updating channel 'guix-past' from Git repository at 'https://does-not-exist.inria.fr/guix-hpc/guix-past'... guix time-machine: error: Git error: cannot locate remote-tracking branch 'origin/keyring' --8<---------------cut here---------------end--------------->8--- The SWH folks tell me it’ll eventually be possible to map a revision to its containing snapshot(s) via the HTTP API, and to obtain entire snapshots (i.e., the repo and all its branches) from the vault. That’s what we need to fix this issue. *Third, and this answers the asterisk above, we must keep in mind that this is content-addressibility *with SHA1*. Generating a chosen-prefix collision is becoming affordable³, so users absolutely need an additional mechanism to authenticate code they fetched. For origins, we have the content SHA256, so we’re fine. For channels, we have Guix’s authentication mechanism¹, except it’s not available yet via SWH, as I wrote above. For the footswitch example above using ‘--with-commit’, we don’t have any authentication method, but in fact, that’s the situation of Git repositories in general: they can rarely be authenticated. Overall, I think it’s a step in the right direction. Thoughts? Thanks to vlorentz and olasd on #swh-devel for their support! Thanks, Ludo’. ¹ https://guix.gnu.org/en/blog/2020/securing-updates/ ² https://guix.gnu.org/en/blog/2019/spreading-the-news/ ³ https://sha-mbles.github.io/ Ludovic Courtès (3): swh: Support downloads of bare Git repositories. git: 'update-cached-checkout' can fall back to SWH when cloning. git: 'reference-available?' recognizes 'tag-or-commit'. guix/git.scm | 45 +++++++++++++++++++++++++++++++++++++++++++-- guix/swh.scm | 52 ++++++++++++++++++++++++++++++++++++++++------------ 2 files changed, 83 insertions(+), 14 deletions(-) -- 2.33.0