bug#44187: Channel clones lack SWH fallback

["Ludovic Courtès" <ludo@gnu.org>]

Hello!

zimoun <zimon.toutoune@gmail.com> skribis:

> The original URL of the channel was:
> <https://github.com/zimoun/channel-example.git>.  And this channel
> defines a package where the upstream has also disappeared
> <https://github.com/zimoun/hello-example.git>.  Note the URL in the
> package definition is not bogus… but using one was already working. :-)
>
> All is saved on SWH, so now all is transparent!  From my point of view,
> this is a killer feature for scientific folks. :-)

Yay!  Great that you came up with a nice example to test it on!

>> First, fallback is implemented only for fresh clones, not for updates.
>> Thus, if I rerun the first example, having now the clone in
>> ~/.cache/guix/checkouts, with a different commit, I get:
>
> SWH is not a forge but an archive. :-)  Therefore, this update case does
> not make sense to me.  I mean,
>
> $ git -C ~/.cache/guix/checkouts/6k7wvrcpbdsw3pje5b4squybw3jfn3viyrj7gcl7fipa5yjflaza fetch
> fatal: dépôt 'http://example.org/sdf/' non trouvé

Right, that’s a reasonable limitation.

> Well, maybe this cache could be removed if the commit is not found
> inside this cache and retry to fetch it from SWH.  Obviously, the
> downdate case works.

It’s still useful to keep it cached around in case the user is going to
use it several times in a row.

> Note that on fresh clone, the error message could be improved:
>
> $ ./pre-inst-env guix build guix --with-git-url=guix=https://example.org --with-commit=guix=ff613c2b68aac539262822490448e637d8f315ba -n
> updating checkout of 'https://example.org'...
> guix build: error: Git failure while fetching https://example.org: unexpected http status code: 404
>
>
> where https://example.org is bogus and
> ff613c2b68aac539262822490448e637d8f315ba is not yet archived on SWH.  It
> could be nice to warn in addition to the 404 that it is not found in
> SWH.  WDYT?

Agreed; I’ve made this change (actually ‘swh-download’ prints something
upon failure since commit 60b42bec8413aa9844e625fb1903257f1bc1e55c, but
it looks more like a debugging message.)

> $ guix build guix --with-git-url=guix=https://example.org --with-commit=guix=c75b30d58f0becb0a5cd6a8bfe69d1063b0d1ada -n
> updating checkout of 'https://example.org'...
> SWH: found revision c75b30d58f0becb0a5cd6a8bfe69d1063b0d1ada with directory at 'https://archive.softwareheritage.org/api/1/directory/ca2e8a7222b4850c7bea935dff86b9c2a905efd6/'
> SWH vault: requested bundle cooking, waiting for completion...
> SWH vault: Processing...
> [...]
>
>
> then after several hours, I get this:
>
> SWH vault: failure: Internal Server Error. This incident will be reported.
> SWH vault: retrying...
> SWH vault: requested bundle cooking, waiting for completion...
> SWH vault: Processing...
>
> and after more than 12h, the status is still: «SWH vault: Processing...»
> and nothing is complete.

Did it eventually succeed?  We obviously have no guarantee as to how
long it might take to cook a bundle.

> About this ’keyring’ branch, somehow it could be as a separated repo, so
> why not effectively do it. :-) I mean, get the branch as it is and
> mirror this branch in another Git repo saved on SWH; fallback to it if
> ’keyring’ branch is not there.  I do not know…  Or simply wait that SWH
> improves their things. :-)

Yeah, they’re planning to support it eventually.

>> *Third, and this answers the asterisk above, we must keep in mind that
>> this is content-addressibility *with SHA1*.  Generating a chosen-prefix
>> collision is becoming affordable³, so users absolutely need an additional
>> mechanism to authenticate code they fetched.

[...]

> How a chosen-prefix attack could work here?  I understand why the second
> preimage attack is an issue.  But I miss how the SHA-1 chosen-prefix attack
> could be exploited here to compromise the user, because this hash is provided
> by this very same user.

I think you’re right, it’s rather second-preimage attacks that would be
a serious problem.  My point is: as time passes, assuming that a SHA1
resolves to a single revision on SWH is becoming more and more
questionable.

>>   swh: Support downloads of bare Git repositories.
>>   git: 'update-cached-checkout' can fall back to SWH when cloning.
>>   git: 'reference-available?' recognizes 'tag-or-commit'.

I’ve pushed this after adding the warning as you suggested:

  dce2cf311b * git: 'reference-available?' recognizes 'tag-or-commit'.
  05f44c2d85 * git: 'update-cached-checkout' can fall back to SWH when cloning.
  6ec81c31c0 * swh: Support downloads of bare Git repositories.

Thanks a lot for reviewing and testing on real-world examples!

Ludo’.