bug#44187: Channel clones lack SWH fallback

[zimoun <zimon.toutoune@gmail.com>]

Hi,

On ven., 10 sept. 2021 at 16:34, Ludovic Courtès <ludo@gnu.org> wrote:

>   Finally we can enjoy content-addressability and brittle URLs
>   are becoming a thing of the past!*

Yeah, it is awesome!

The original URL of the channel was:
<https://github.com/zimoun/channel-example.git>.  And this channel
defines a package where the upstream has also disappeared
<https://github.com/zimoun/hello-example.git>.  Note the URL in the
package definition is not bogus… but using one was already working. :-)

All is saved on SWH, so now all is transparent!  From my point of view,
this is a killer feature for scientific folks. :-)

--8<---------------cut here---------------start------------->8---
$ cat /tmp/channels.scm
(list (channel
        (name 'guix)
        (url "/home/sitour/src/guix/guix")
        (branch "fix-44187")
        (commit
          "cdea76a2fdaf7705583a02081a6468d436b8df05"))
      (channel
        (name 'example)
        (url "https://example.org/foo.git")
        (commit
          "67c9f2143aa6f545419ae913b4ae02af4cd3effc")))

$ ./pre-inst-env guix time-machine -C /tmp/channels.scm --disable-authentication -- build hi
Updating channel 'guix' from Git repository at '/home/sitour/src/guix/guix'...
guix time-machine: warning: channel authentication disabled
Updating channel 'example' from Git repository at 'https://example.org/foo.git'...
SWH: found revision 67c9f2143aa6f545419ae913b4ae02af4cd3effc with directory at 'https://archive.softwareheritage.org/api/1/directory/fe423e88ce277d3fc230c88d408e42b14a3a458c/'
SWH vault: requested bundle cooking, waiting for completion...
swh:1:rev:67c9f2143aa6f545419ae913b4ae02af4cd3effc.git/
swh:1:rev:67c9f2143aa6f545419ae913b4ae02af4cd3effc.git/HEAD
swh:1:rev:67c9f2143aa6f545419ae913b4ae02af4cd3effc.git/branches/
swh:1:rev:67c9f2143aa6f545419ae913b4ae02af4cd3effc.git/config
swh:1:rev:67c9f2143aa6f545419ae913b4ae02af4cd3effc.git/description
swh:1:rev:67c9f2143aa6f545419ae913b4ae02af4cd3effc.git/hooks/
swh:1:rev:67c9f2143aa6f545419ae913b4ae02af4cd3effc.git/info/
swh:1:rev:67c9f2143aa6f545419ae913b4ae02af4cd3effc.git/info/exclude
swh:1:rev:67c9f2143aa6f545419ae913b4ae02af4cd3effc.git/info/refs
swh:1:rev:67c9f2143aa6f545419ae913b4ae02af4cd3effc.git/objects/
swh:1:rev:67c9f2143aa6f545419ae913b4ae02af4cd3effc.git/objects/info/
swh:1:rev:67c9f2143aa6f545419ae913b4ae02af4cd3effc.git/objects/info/packs
swh:1:rev:67c9f2143aa6f545419ae913b4ae02af4cd3effc.git/objects/pack/
swh:1:rev:67c9f2143aa6f545419ae913b4ae02af4cd3effc.git/objects/pack/pack-4e9279a1b64e4dda7bd9d84bb6b50bb1f80def08.idx
swh:1:rev:67c9f2143aa6f545419ae913b4ae02af4cd3effc.git/objects/pack/pack-4e9279a1b64e4dda7bd9d84bb6b50bb1f80def08.pack
swh:1:rev:67c9f2143aa6f545419ae913b4ae02af4cd3effc.git/refs/
swh:1:rev:67c9f2143aa6f545419ae913b4ae02af4cd3effc.git/refs/heads/
swh:1:rev:67c9f2143aa6f545419ae913b4ae02af4cd3effc.git/refs/heads/master
swh:1:rev:67c9f2143aa6f545419ae913b4ae02af4cd3effc.git/refs/tags/
guix time-machine: warning: channel authentication disabled

[...]

Computing Guix derivation for 'x86_64-linux'... -

[...]

construction de /gnu/store/6g9qlysbbk7p4609xrv82j0wzbib1y4r-git-checkout.drv...
guile: warning: failed to install locale
environment variable `PATH' set to `/gnu/store/378zjf2kgajcfd7mfr98jn5xyc5wa3qv-gzip-1.10/bin:/gnu/store/sf3rbvb6iqcphgm1afbplcs72hsywg25-tar-1.32/bin'
hint: Using 'master' as the name for the initial branch. This default branch name
hint: is subject to change. To configure the initial branch name to use in all
hint: of your new repositories, which will suppress this warning, call:
hint:
hint: 	git config --global init.defaultBranch <name>
hint:
hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and
hint: 'development'. The just-created branch can be renamed via this command:
hint:
hint: 	git branch -m <name>
Initialized empty Git repository in /gnu/store/884nsva9r8wkp40kbqyvpj1ad57jc5dd-git-checkout/.git/
fatal: could not read Username for 'https://github.com': No such device or address
Failed to do a shallow fetch; retrying a full fetch...
fatal: could not read Username for 'https://github.com': No such device or address
git-fetch: '/gnu/store/5vai7bfrfkzv22dx13bxpszjrqyi78x6-git-minimal-2.33.0/bin/git fetch origin' failed with exit code 128
Trying content-addressed mirror at berlin.guix.gnu.org...
Trying content-addressed mirror at berlin.guix.gnu.org...
Trying to download from Software Heritage...
SWH: found revision e1eefd033b8a2c4c81babc6fde08ebb116c6abb8 with directory at 'https://archive.softwareheritage.org/api/1/directory/c3e538ed2de412d54c567ed7c8cfc46cbbc35d07/'
swh:1:dir:c3e538ed2de412d54c567ed7c8cfc46cbbc35d07/
swh:1:dir:c3e538ed2de412d54c567ed7c8cfc46cbbc35d07/ABOUT-NLS
swh:1:dir:c3e538ed2de412d54c567ed7c8cfc46cbbc35d07/AUTHORS
swh:1:dir:c3e538ed2de412d54c567ed7c8cfc46cbbc35d07/COPYING

[...]

swh:1:dir:c3e538ed2de412d54c567ed7c8cfc46cbbc35d07/tests/hello-1
swh:1:dir:c3e538ed2de412d54c567ed7c8cfc46cbbc35d07/tests/last-1
swh:1:dir:c3e538ed2de412d54c567ed7c8cfc46cbbc35d07/tests/traditional-1
construction de /gnu/store/6g9qlysbbk7p4609xrv82j0wzbib1y4r-git-checkout.drv réussie
construction de /gnu/store/jx1r7w8xaw768176pjl0j0q1l1529w75-hi-2.10.drv...
starting phase `set-SOURCE-DATE-EPOCH'
phase `set-SOURCE-DATE-EPOCH' succeeded after 0.0 seconds

[...]

construction de /gnu/store/jx1r7w8xaw768176pjl0j0q1l1529w75-hi-2.10.drv réussie
/gnu/store/jn8d031zx4znxy7s5zhj4dbr6xjsfq9v-hi-2.10
--8<---------------cut here---------------end--------------->8---

Well, it still misses the tarball and non-Git fetch method fallback and
the story will be more than awesome! :-)

> Limitations
> ~~~~~~~~~~~~
>
> Yes, there’s a couple of them.

Well, yes some limitations but not so much. ;-)


> First, fallback is implemented only for fresh clones, not for updates.
> Thus, if I rerun the first example, having now the clone in
> ~/.cache/guix/checkouts, with a different commit, I get:

SWH is not a forge but an archive. :-)  Therefore, this update case does
not make sense to me.  I mean,

--8<---------------cut here---------------start------------->8---
$ git -C ~/.cache/guix/checkouts/6k7wvrcpbdsw3pje5b4squybw3jfn3viyrj7gcl7fipa5yjflaza fetch
fatal: dépôt 'http://example.org/sdf/' non trouvé
--8<---------------cut here---------------end--------------->8---

Well, maybe this cache could be removed if the commit is not found
inside this cache and retry to fetch it from SWH.  Obviously, the
downdate case works.

Note that on fresh clone, the error message could be improved:

--8<---------------cut here---------------start------------->8---
$ ./pre-inst-env guix build guix --with-git-url=guix=https://example.org --with-commit=guix=ff613c2b68aac539262822490448e637d8f315ba -n
updating checkout of 'https://example.org'...
guix build: error: Git failure while fetching https://example.org: unexpected http status code: 404
--8<---------------cut here---------------end--------------->8---

where https://example.org is bogus and
ff613c2b68aac539262822490448e637d8f315ba is not yet archived on SWH.  It
could be nice to warn in addition to the 404 that it is not found in
SWH.  WDYT?


> Second, clones from SWH only contain the one branch that the revision
> is on.  For channels, that means that the ‘keyring’ branch is not fetched,
> which is why I commented out ‘introduction’ in /tmp/chan.scm above.

To me, it is not an issue.  Because you reach a commit from the past
knowing the hash.

Aside my opinion, I wanted to know which kind of metadata we get back
from the Git repo, so I tried:

--8<---------------cut here---------------start------------->8---
$ guix build guix --with-git-url=guix=https://example.org --with-commit=guix=c75b30d58f0becb0a5cd6a8bfe69d1063b0d1ada -n
updating checkout of 'https://example.org'...
SWH: found revision c75b30d58f0becb0a5cd6a8bfe69d1063b0d1ada with directory at 'https://archive.softwareheritage.org/api/1/directory/ca2e8a7222b4850c7bea935dff86b9c2a905efd6/'
SWH vault: requested bundle cooking, waiting for completion...
SWH vault: Processing...
[...]
--8<---------------cut here---------------end--------------->8---

then after several hours, I get this:

--8<---------------cut here---------------start------------->8---
SWH vault: failure: Internal Server Error. This incident will be reported.
SWH vault: retrying...
SWH vault: requested bundle cooking, waiting for completion...
SWH vault: Processing...
--8<---------------cut here---------------end--------------->8---

and after more than 12h, the status is still: «SWH vault: Processing...»
and nothing is complete.

About this ’keyring’ branch, somehow it could be as a separated repo, so
why not effectively do it. :-) I mean, get the branch as it is and
mirror this branch in another Git repo saved on SWH; fallback to it if
’keyring’ branch is not there.  I do not know…  Or simply wait that SWH
improves their things. :-)


> *Third, and this answers the asterisk above, we must keep in mind that
> this is content-addressibility *with SHA1*.  Generating a chosen-prefix
> collision is becoming affordable³, so users absolutely need an additional
> mechanism to authenticate code they fetched.
>
> For origins, we have the content SHA256, so we’re fine.  For channels,
> we have Guix’s authentication mechanism¹, except it’s not available yet
> via SWH, as I wrote above.  For the footswitch example above using
> ‘--with-commit’, we don’t have any authentication method, but in fact,
> that’s the situation of Git repositories in general: they can rarely be
> authenticated.

How a chosen-prefix attack could work here?  I understand why the second
preimage attack is an issue.  But I miss how the SHA-1 chosen-prefix attack
could be exploited here to compromise the user, because this hash is provided
by this very same user.


> Ludovic Courtès (3):
>   swh: Support downloads of bare Git repositories.
>   git: 'update-cached-checkout' can fall back to SWH when cloning.
>   git: 'reference-available?' recognizes 'tag-or-commit'.

LGTM!

Cheers,
simon