bug#69058: Failure when fallback to Disarchive+SWH

bug#69058: Failure when fallback to Disarchive+SWH

[Simon Tournier]

Hi,

Here an example where something is wrong:

 + Diarsarchive specificiation is ok.
   <https://disarchive.guix.gnu.org/sha256/787014b29f0c5793ecc2d93e1109a049ff48ab0c29b851dab34f683ceef6b152>
 + SWH has it
   <https://archive.softwareheritage.org/browse/directory/35e81cfbd7ca261e2ddebf372f38a4d1baa2f931>

But something is failing on Guix side.

--8<---------------cut here---------------start------------->8---
$ guix build /gnu/store/mxsfblzy65v5y4gwrxqkihj7g4w5f5b1-xscreensaver-6.04.tar.gz.drv
/gnu/store/58m53a8ygffj4l5kpqhyxby1ylhri9dr-xscreensaver-6.04.tar.gz

$ guix build /gnu/store/mxsfblzy65v5y4gwrxqkihj7g4w5f5b1-xscreensaver-6.04.tar.gz.drv --check
The following derivation will be built:
  /gnu/store/mxsfblzy65v5y4gwrxqkihj7g4w5f5b1-xscreensaver-6.04.tar.gz.drv
building /gnu/store/mxsfblzy65v5y4gwrxqkihj7g4w5f5b1-xscreensaver-6.04.tar.gz.drv...

Starting download of /gnu/store/123bm8wnld70qnajw5crdw7a7lhbpa0x-xscreensaver-6.04.tar.gz

[...]

Trying to use Disarchive to assemble /gnu/store/123bm8wnld70qnajw5crdw7a7lhbpa0x-xscreensaver-6.04.tar.gz...
Retrieving Disarchive spec from https://disarchive.guix.gnu.org/sha256/787014b29f0c5793ecc2d93e1109a049ff48ab0c29b851dab34f683ceef6b152 ...
Assembling the directory xscreensaver-6.04
Downloading /gnu/store/123bm8wnld70qnajw5crdw7a7lhbpa0x-xscreensaver-6.04.tar.gz from Software Heritage...
SWH: object swh:1:dir:35e81cfbd7ca261e2ddebf372f38a4d1baa2f931 could not be fetched from the vault
Could not resolve directory reference
failed to download "/gnu/store/123bm8wnld70qnajw5crdw7a7lhbpa0x-xscreensaver-6.04.tar.gz" from "https://www.jwz.org/xscreensaver/xscreensaver-6.04.tar.gz"
builder for `/gnu/store/mxsfblzy65v5y4gwrxqkihj7g4w5f5b1-xscreensaver-6.04.tar.gz.drv' failed to produce output path `/gnu/store/58m53a8ygffj4l5kpqhyxby1ylhri9dr-xscreensaver-6.04.tar.gz'
build of /gnu/store/mxsfblzy65v5y4gwrxqkihj7g4w5f5b1-xscreensaver-6.04.tar.gz.drv failed
View build log at '/var/log/guix/drvs/mx/sfblzy65v5y4gwrxqkihj7g4w5f5b1-xscreensaver-6.04.tar.gz.drv.gz'.
guix build: error: build of `/gnu/store/mxsfblzy65v5y4gwrxqkihj7g4w5f5b1-xscreensaver-6.04.tar.gz.drv' failed
--8<---------------cut here---------------end--------------->8---


Let’s do it manually:

 1. fetch content and specification
 2. assemble
 3. compare the assembled with the one from content-addressed-mirrors


1.

--8<---------------cut here---------------start------------->8---
$ wget -O /tmp/spec https://disarchive.guix.gnu.org/sha256/787014b29f0c5793ecc2d93e1109a049ff48ab0c29b851dab34f683ceef6b152
/tmp/spec               [    <=>             ]   3,25M   512KB/s    in 6,6s    

$ wget -O /tmp/content https://archive.softwareheritage.org/api/1/vault/flat/swh:1:dir:35e81cfbd7ca261e2ddebf372f38a4d1baa2f931/raw
/tmp/content        100%[===================>]  24,86M   530KB/s    in 49s     
--8<---------------cut here---------------end--------------->8---


2.

--8<---------------cut here---------------start------------->8---
$ tar xf content
$ disarchive assemble swh:1:dir:35e81cfbd7ca261e2ddebf372f38a4d1baa2f931 spec > xscreensaver-6.04.tar.gz
Assembling the directory xscreensaver-6.04
Checking xscreensaver-6.04 digest... ok
Assembling the tarball xscreensaver-6.04.tar
Checking xscreensaver-6.04.tar digest... ok
Assembling the Gzip file xscreensaver-6.04.tar.gz
Checking xscreensaver-6.04.tar.gz digest... ok
Writing result to output port
--8<---------------cut here---------------end--------------->8---


3.

--8<---------------cut here---------------start------------->8---
$ disarchive assemble swh:1:dir:35e81cfbd7ca261e2ddebf372f38a4d1baa2f931 spec 2>/dev/null | guix hash -
0lmiyvp3qs2gngd53f191jmlizs9l04i2gnrqbn96mqckyr18w3q

$ guix hash $(guix build /gnu/store/mxsfblzy65v5y4gwrxqkihj7g4w5f5b1-xscreensaver-6.04.tar.gz.drv)
0lmiyvp3qs2gngd53f191jmlizs9l04i2gnrqbn96mqckyr18w3q
--8<---------------cut here---------------end--------------->8---

Cheers,
simon

bug#69058: Failure when fallback to Disarchive+SWH

[Simon Tournier]

Hi,

Another example using Git.  It is from v1.1.0.

--8<---------------cut here---------------start------------->8---
$ guix time-machine -q --commit=v1.1.0 -- build -S redkite
guile: warning: failed to install locale
The following derivation will be built:
   /gnu/store/x4hcwpmb3p8llbhjzxlfggpkpqc7n5ng-redkite-0.8.0-checkout.drv
building /gnu/store/x4hcwpmb3p8llbhjzxlfggpkpqc7n5ng-redkite-0.8.0-checkout.drv...
guile: warning: failed to install locale
environment variable `PATH' set to `/gnu/store/i2cdl0hvrml8hjdqplqln8didnvxkgp5-gzip-1.10/bin:/gnu/store/jh17p4sns7dvbizwz58gdh953qpic144-tar-1.32/bin'
Initialized empty Git repository in /gnu/store/y7j54zw1fwhl95534zsjl6rl1rfp7z82-redkite-0.8.0-checkout/.git/
fatal: could not read Username for 'https://gitlab.com': No such device or address
Failed to do a shallow fetch; retrying a full fetch...
fatal: could not read Username for 'https://gitlab.com': No such device or address
git-fetch: '/gnu/store/xzhkhidb2kqwvpv8b8zaqzgd3gcjs9nn-git-minimal-2.26.0/bin/git fetch origin' failed with exit code 128
Trying content-addressed mirror at berlin.guixsd.org...
Trying content-addressed mirror at berlin.guixsd.org...
Trying to download from Software Heritage...
SWH: found revision 68276b9b88491198c33b03c4188dfe7d6e5dbf2a with directory at 'https://archive.softwareheritage.org/api/1/directory/d14c0eb6801bca920d1e1baac7a4b9e2a0589728/'
SWH: directory d14c0eb6801bca920d1e1baac7a4b9e2a0589728 could not be fetched from the vault
builder for `/gnu/store/x4hcwpmb3p8llbhjzxlfggpkpqc7n5ng-redkite-0.8.0-checkout.drv' failed to produce output path `/gnu/store/y7j54zw1fwhl95534zsjl6rl1rfp7z82-redkite-0.8.0-checkout'
build of /gnu/store/x4hcwpmb3p8llbhjzxlfggpkpqc7n5ng-redkite-0.8.0-checkout.drv failed
View build log at '/var/log/guix/drvs/x4/hcwpmb3p8llbhjzxlfggpkpqc7n5ng-redkite-0.8.0-checkout.drv.gz'.
guix build: error: build of `/gnu/store/x4hcwpmb3p8llbhjzxlfggpkpqc7n5ng-redkite-0.8.0-checkout.drv' failed
--8<---------------cut here---------------end--------------->8---


From its definition the package reads:

--8<---------------cut here---------------start------------->8---
(define-public redkite
[...]
       (sha256
        (base32
         "1747w1kg8y9jbl11xi018d85dm38xk7843pz26sh0k5fdv87a10q"))))
--8<---------------cut here---------------end--------------->8---

Then if I manually download it:

--8<---------------cut here---------------start------------->8---
$ wget -O /tmp/content https://archive.softwareheritage.org/api/1/vault/flat/swh:1:dir:d14c0eb6801bca920d1e1baac7a4b9e2a0589728/raw

[...]
/tmp/content        100%[===================>] 129,38K   555KB/s    in 0,2s    
2024-02-11 19:00:33 (555 KB/s) - ‘/tmp/content’ saved [132488/132488]

$ tar xf content
--8<---------------cut here---------------end--------------->8---

And we get the expected value:

--8<---------------cut here---------------start------------->8---
$ guix hash -rx swh:1:dir:d14c0eb6801bca920d1e1baac7a4b9e2a0589728
1747w1kg8y9jbl11xi018d85dm38xk7843pz26sh0k5fdv87a10q
--8<---------------cut here---------------end--------------->8---


Cheers,
simon

bug#69058: Failure when fallback to Disarchive+SWH

[Ludovic Courtès]

Hi,

Simon Tournier <zimon.toutoune@gmail.com> skribis:

> Another example using Git.  It is from v1.1.0.
>
> $ guix time-machine -q --commit=v1.1.0 -- build -S redkite
> guile: warning: failed to install locale
> The following derivation will be built:
>    /gnu/store/x4hcwpmb3p8llbhjzxlfggpkpqc7n5ng-redkite-0.8.0-checkout.drv
> building /gnu/store/x4hcwpmb3p8llbhjzxlfggpkpqc7n5ng-redkite-0.8.0-checkout.drv...
> guile: warning: failed to install locale
> environment variable `PATH' set to `/gnu/store/i2cdl0hvrml8hjdqplqln8didnvxkgp5-gzip-1.10/bin:/gnu/store/jh17p4sns7dvbizwz58gdh953qpic144-tar-1.32/bin'
> Initialized empty Git repository in /gnu/store/y7j54zw1fwhl95534zsjl6rl1rfp7z82-redkite-0.8.0-checkout/.git/
> fatal: could not read Username for 'https://gitlab.com': No such device or address
> Failed to do a shallow fetch; retrying a full fetch...
> fatal: could not read Username for 'https://gitlab.com': No such device or address
> git-fetch: '/gnu/store/xzhkhidb2kqwvpv8b8zaqzgd3gcjs9nn-git-minimal-2.26.0/bin/git fetch origin' failed with exit code 128
> Trying content-addressed mirror at berlin.guixsd.org...
> Trying content-addressed mirror at berlin.guixsd.org...
> Trying to download from Software Heritage...
> SWH: found revision 68276b9b88491198c33b03c4188dfe7d6e5dbf2a with directory at 'https://archive.softwareheritage.org/api/1/directory/d14c0eb6801bca920d1e1baac7a4b9e2a0589728/'
> SWH: directory d14c0eb6801bca920d1e1baac7a4b9e2a0589728 could not be fetched from the vault
> builder for `/gnu/store/x4hcwpmb3p8llbhjzxlfggpkpqc7n5ng-redkite-0.8.0-checkout.drv' failed to produce output path `/gnu/store/y7j54zw1fwhl95534zsjl6rl1rfp7z82-redkite-0.8.0-checkout'
> build of /gnu/store/x4hcwpmb3p8llbhjzxlfggpkpqc7n5ng-redkite-0.8.0-checkout.drv failed
> View build log at '/var/log/guix/drvs/x4/hcwpmb3p8llbhjzxlfggpkpqc7n5ng-redkite-0.8.0-checkout.drv.gz'.
> guix build: error: build of `/gnu/store/x4hcwpmb3p8llbhjzxlfggpkpqc7n5ng-redkite-0.8.0-checkout.drv' failed

I can reproduce it with c4372f7ebfac17236299346fe60c6ca125336b1f:

--8<---------------cut here---------------start------------->8---
scheme@(guile-user)> (swh-download "https://gitlab.com/geontime/redkite.git" "v0.8.0" "/tmp/redkite")
SWH: found revision 68276b9b88491198c33b03c4188dfe7d6e5dbf2a with directory at 'https://archive.softwareheritage.org/api/1/directory/d14c0eb6801bca920d1e1baac7a4b9e2a0589728/'
SWH: object swh:1:dir:d14c0eb6801bca920d1e1baac7a4b9e2a0589728 could not be fetched from the vault
$5 = #f
--8<---------------cut here---------------end--------------->8---

The bug is that the Vault now returns 302 (redirect) in this case, which
is similar to the bug I fixed in <https://issues.guix.gnu.org/68741>.

So I’ll have to push that patch series and add a patch for ‘vault-fetch’
on top of it.

Ludo’.

bug#69058: Failure when fallback to Disarchive+SWH

[Ludovic Courtès]

Ludovic Courtès <ludovic.courtes@inria.fr> skribis:

> I can reproduce it with c4372f7ebfac17236299346fe60c6ca125336b1f:
>
> scheme@(guile-user)> (swh-download "https://gitlab.com/geontime/redkite.git" "v0.8.0" "/tmp/redkite")
> SWH: found revision 68276b9b88491198c33b03c4188dfe7d6e5dbf2a with directory at 'https://archive.softwareheritage.org/api/1/directory/d14c0eb6801bca920d1e1baac7a4b9e2a0589728/'
> SWH: object swh:1:dir:d14c0eb6801bca920d1e1baac7a4b9e2a0589728 could not be fetched from the vault
> $5 = #f
>
> The bug is that the Vault now returns 302 (redirect) in this case, which
> is similar to the bug I fixed in <https://issues.guix.gnu.org/68741>.

In fact it’s the exact same problem.

Fixed in 1610a632d4b3097282d18af27ff3e9e178d7dfcb.

Thanks,
Ludo’.

bug#69058: Failure when fallback to Disarchive+SWH

[Simon Tournier]

Hi Ludo,

On lun., 12 févr. 2024 at 12:24, Ludovic Courtès <ludovic.courtes@inria.fr> wrote:
> Ludovic Courtès <ludovic.courtes@inria.fr> skribis:
>
>> I can reproduce it with c4372f7ebfac17236299346fe60c6ca125336b1f:
>>
>> scheme@(guile-user)> (swh-download "https://gitlab.com/geontime/redkite.git" "v0.8.0" "/tmp/redkite")
>> SWH: found revision 68276b9b88491198c33b03c4188dfe7d6e5dbf2a with directory at 'https://archive.softwareheritage.org/api/1/directory/d14c0eb6801bca920d1e1baac7a4b9e2a0589728/'
>> SWH: object swh:1:dir:d14c0eb6801bca920d1e1baac7a4b9e2a0589728 could not be fetched from the vault
>> $5 = #f
>>
>> The bug is that the Vault now returns 302 (redirect) in this case, which
>> is similar to the bug I fixed in <https://issues.guix.gnu.org/68741>.
>
> In fact it’s the exact same problem.
>
> Fixed in 1610a632d4b3097282d18af27ff3e9e178d7dfcb.

Cool!  Thanks.

However, it means that it stays broken for all commits before this fix,
right?  Somehow, if only SWH is available, then

    guix time-machine -C channels.scm -- shell -m manifest.scm

will fail for all channels older than today.  Right?

For instance, redoing your ReScience paper from 2020 using only SWH as I
did [1] some months ago is now broken (because a change on SWH side).

Well, I am fine to keep this bug closed but for the posterity we need
something like:

    guix swh get -C channels.scm -m manifest.scm

as we discussed elsewhere (off-list IIRC :-)).

1: https://simon.tournier.info/posts/2023-06-23-hackathon-repro.html

Cheers,
simon

bug#69058: Failure when fallback to Disarchive+SWH

[Ludovic Courtès]

Hi,

Simon Tournier <zimon.toutoune@gmail.com> skribis:

>> Fixed in 1610a632d4b3097282d18af27ff3e9e178d7dfcb.
>
> Cool!  Thanks.
>
> However, it means that it stays broken for all commits before this fix,
> right?  Somehow, if only SWH is available, then
>
>     guix time-machine -C channels.scm -- shell -m manifest.scm
>
> will fail for all channels older than today.  Right?

Mostly, yes (“mostly” because derivations using “builtin:git-download”
will automatically be fixed once we’ve updated the ‘guix’ package, but
that only helps if you’re traveling to after September 2023 (commit
13b0cf85eb31e1b1ea674debbbfb0f35a32d1374).

> For instance, redoing your ReScience paper from 2020 using only SWH as I
> did [1] some months ago is now broken (because a change on SWH side).

Indeed.  :-/

That’s a real bummer, but that’s the nature of software: we cannot
retroactively fix bugs.

> Well, I am fine to keep this bug closed but for the posterity we need
> something like:
>
>     guix swh get -C channels.scm -m manifest.scm

Exactly!  We have the demonstration of just how important that is.

Ludo’.

bug#69058: Failure when fallback to Disarchive+SWH

[Simon Tournier]

> That’s a real bummer, but that’s the nature of software: we cannot
> retroactively fix bugs.

The nature of service and not software appears to me more correct. :-)

(Here it is not the retroactive fix of a bug but the introduction of a
non-backward compatible change.)

Cheers,
simon