bug#52591: downloading from software archives as default even if version is wrong

bug#52591: downloading from software archives as default even if version is wrong

[Andy Tai]

I was trying to update a package.   I updated the package definition:

---patch--
diff --git a/gnu/packages/mono.scm b/gnu/packages/mono.scm
index 6a7afbafbf..9eeb46ef16 100644
--- a/gnu/packages/mono.scm
+++ b/gnu/packages/mono.scm
@@ -38,7 +38,7 @@ (define-module (gnu packages mono)
 (define-public mono
   (package
     (name "mono")
-    (version "4.4.1.0")
+    (version "4.8.1")
     (source (origin
               (method url-fetch)
               (uri (string-append
--end of patch--

which at this time shall fail as I have not updated the hash checksum.  If I do

 ./pre-inst-env guix build  -k -K   mono

that actually succeeded but it shall have failed.

This log fragment show what happened:

----log fragment--
The following derivations will be built:
   /gnu/store/cdqqdalqwl3m9bhnfyb0v6nfkzvfj0x1-mono-4.8.1.drv
   /gnu/store/qs8jgc33jpwwmzq7ppdiv49qll6q32sl-mono-4.8.1.tar.xz.drv
   /gnu/store/6aih7gpx3p1dpsbdhkri7mk3qqnqkk9x-mono-4.8.1.tar.bz2.drv
building /gnu/store/6aih7gpx3p1dpsbdhkri7mk3qqnqkk9x-mono-4.8.1.tar.bz2.drv...

Starting download of
/gnu/store/l6309vrm95cwpmcwgjj5xgdddp6vfly5-mono-4.8.1.tar.bz2
From http://download.mono-project.com/sources/mono/mono-4.8.1.tar.bz2...
following redirection to
`https://download.mono-project.com/sources/mono/mono-4.8.1.tar.bz2'...
download failed
"https://download.mono-project.com/sources/mono/mono-4.8.1.tar.bz2"
404 "Not Found"

Starting download of
/gnu/store/l6309vrm95cwpmcwgjj5xgdddp6vfly5-mono-4.8.1.tar.bz2
From https://ci.guix.gnu.org/file/mono-4.8.1.tar.bz2/sha256/0jibyvyv2jy8dq5ij0j00iq3v74r0y90dcjc3dkspc
fbnn37cphn...
download failed
"https://ci.guix.gnu.org/file/mono-4.8.1.tar.bz2/sha256/0jibyvyv2jy8dq5ij0j00iq3v74r0y
90dcjc3dkspcfbnn37cphn" 404 "Not Found"

Starting download of
/gnu/store/l6309vrm95cwpmcwgjj5xgdddp6vfly5-mono-4.8.1.tar.bz2
From https://tarballs.nixos.org/sha256/0jibyvyv2jy8dq5ij0j00iq3v74r0y90dcjc3dkspcfbnn37cphn...
download failed
"https://tarballs.nixos.org/sha256/0jibyvyv2jy8dq5ij0j00iq3v74r0y90dcjc3dkspcfbnn37cph
n" 404 "Not Found"

Starting download of
/gnu/store/l6309vrm95cwpmcwgjj5xgdddp6vfly5-mono-4.8.1.tar.bz2
From https://archive.softwareheritage.org/api/1/content/sha256:165e7686b5cbb1ab671b4cb2069207999c3d700
44002190b6ec84bb1fdf62b4a/raw/...
downloading from
https://archive.softwareheritage.org/api/1/content/sha256:165e7686b5cbb1ab671b4cb2069
207999c3d70044002190b6ec84bb1fdf62b4a/raw/ ...
............
successfully built
/gnu/store/6aih7gpx3p1dpsbdhkri7mk3qqnqkk9x-mono-4.8.1.tar.bz2.drv
building /gnu/store/qs8jgc33jpwwmzq7ppdiv49qll6q32sl-mono-4.8.1.tar.xz.drv...
mono-4.4.1/
mono-4.4.1/test-driver
mono-4.4.1/support/
mono-4.4.1/support/syslog.c
mono-4.4.1/support/supportw.h
....

--end of log fratgment--


as seen above, somehow the old version was downloaded from a cached
copy at softwareheritage archives, and it proceeds to build.   This
should not proceed but fail for wrong checksum.

bug#52591: downloading from software archives as default even if version is wrong

[Maxime Devos]

Andy Tai schreef op vr 17-12-2021 om 15:40 [-0800]:
> @@ -38,7 +38,7 @@ (define-module (gnu packages mono)
>  (define-public mono
>    (package
>      (name "mono")
> -    (version "4.4.1.0")
> +    (version "4.8.1")
>      (source (origin
>                (method url-fetch)
>                (uri (string-append
> --end of patch--

You forgot to update the sha256 hash. You can use
"guix download https://the-website/mono-4.8.1.tar.bz2" to determine the
hash. Or toggle a single character & look at the hash mismatch line.

Does that work for you?

[...]
> as seen above, somehow the old version was downloaded from a cached
> copy at softwareheritage archives, and it proceeds to build.   This
> should not proceed but fail for wrong checksum.

SWH isn't a cache, its an archive that keeps everything forever.

Guix has no way to determine if upstream is doing unspeakable things to
their tarballs (*) and hence needs to use SWH, or if someone forgot to
change the hash on guix. This has been noticed in the past, without
any ideas on how to somehow teach guix to determine the case.

(*) e.g. in-place modification, which changes the hash

Greetings,
Maxime.

bug#52591: downloading from software archives as default even if version is wrong

[Leo Famulari]

On Fri, Dec 17, 2021 at 03:40:45PM -0800, Andy Tai wrote:
> as seen above, somehow the old version was downloaded from a cached
> copy at softwareheritage archives, and it proceeds to build.   This
> should not proceed but fail for wrong checksum.

This can happen with the Nix content-addressed archive fallback that we
use, too.

Basically, when the normal URLs fail, Guix queries these
content-addressed archives by content. That is, by their hash.

The version number is considered to be metadata, not content, and so the
download "succeeds".

Since the source origins are named by their hash, this is the correct
behaviour, although there is some room for error, as you've found.

Do you have any ideas about how to improve things?

We taught Guix to respect the GUIX_DOWNLOAD_FALLBACK_TEST [0]
environment variable, which controls how Guix uses these fallback
archives.

I recommend setting it to "none" if you are doing package development in
order to avoid this pitfall. Otherwise, these archives are helpful for
using Guix, since sources do disappear upstream, URLs change, etc, and
we'd like for old versions of Guix to be usable.

[0]
https://git.savannah.gnu.org/cgit/guix.git/commit/?id=c4a7aa82e25503133a1bd33148d17968c899a5f5
https://git.savannah.gnu.org/cgit/guix.git/commit/?id=bd61d62182bfda4a695757ec66810b28e8e1a6d0

bug#52591: downloading from software archives as default even if version is wrong

[Tobias Geerinckx-Rice via Bug reports for GNU Guix]

[-- Attachment #1: Type: text/plain, Size: 728 bytes --]

Maxime Devos 写道：
> You forgot to update the sha256 hash.

That's it, in a nutshell!

> which at this time shall fail as I have not updated the hash 
> checksum.

No, that's the reason it *succeeds*!

Guix is mostly content-addressed, meaning that the hash identifies 
a file.  If you don't change the hash, you're not asking for a new 
file, by definition.  URLs are like directions (hence: ‘locator’), 
not identifiers.

You can easily generate a bogus-but-valid hash by negating the 
first digit, which (in nix-base32 at least), can only be either 0 
or 1.  Guix will then print the correct hash of whatever it found 
at the URI you specified.

OK to close as NOTABUG?

Kind regards,

T G-R

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 243 bytes --]

bug#52591: downloading from software archives as default even if version is wrong

[raingloom]

On Fri, 17 Dec 2021 15:40:45 -0800
Andy Tai <lichengtai@gmail.com> wrote:

> I was trying to update a package.   I updated the package definition:
> 
> ---patch--
> diff --git a/gnu/packages/mono.scm b/gnu/packages/mono.scm
> index 6a7afbafbf..9eeb46ef16 100644
> --- a/gnu/packages/mono.scm
> +++ b/gnu/packages/mono.scm
> @@ -38,7 +38,7 @@ (define-module (gnu packages mono)
>  (define-public mono
>    (package
>      (name "mono")
> -    (version "4.4.1.0")
> +    (version "4.8.1")
>      (source (origin
>                (method url-fetch)
>                (uri (string-append
> --end of patch--
> 
> which at this time shall fail as I have not updated the hash
> checksum.  If I do
> 
>  ./pre-inst-env guix build  -k -K   mono
> 
> that actually succeeded but it shall have failed.
> 
> This log fragment show what happened:
> 
> ----log fragment--
> The following derivations will be built:
>    /gnu/store/cdqqdalqwl3m9bhnfyb0v6nfkzvfj0x1-mono-4.8.1.drv
>    /gnu/store/qs8jgc33jpwwmzq7ppdiv49qll6q32sl-mono-4.8.1.tar.xz.drv
>    /gnu/store/6aih7gpx3p1dpsbdhkri7mk3qqnqkk9x-mono-4.8.1.tar.bz2.drv
> building
> /gnu/store/6aih7gpx3p1dpsbdhkri7mk3qqnqkk9x-mono-4.8.1.tar.bz2.drv...
> 
> Starting download of
> /gnu/store/l6309vrm95cwpmcwgjj5xgdddp6vfly5-mono-4.8.1.tar.bz2
> From
> http://download.mono-project.com/sources/mono/mono-4.8.1.tar.bz2...
> following redirection to
> `https://download.mono-project.com/sources/mono/mono-4.8.1.tar.bz2'...
> download failed
> "https://download.mono-project.com/sources/mono/mono-4.8.1.tar.bz2"
> 404 "Not Found"
> 
> Starting download of
> /gnu/store/l6309vrm95cwpmcwgjj5xgdddp6vfly5-mono-4.8.1.tar.bz2
> From
> https://ci.guix.gnu.org/file/mono-4.8.1.tar.bz2/sha256/0jibyvyv2jy8dq5ij0j00iq3v74r0y90dcjc3dkspc
> fbnn37cphn... download failed
> "https://ci.guix.gnu.org/file/mono-4.8.1.tar.bz2/sha256/0jibyvyv2jy8dq5ij0j00iq3v74r0y
> 90dcjc3dkspcfbnn37cphn" 404 "Not Found"
> 
> Starting download of
> /gnu/store/l6309vrm95cwpmcwgjj5xgdddp6vfly5-mono-4.8.1.tar.bz2
> From
> https://tarballs.nixos.org/sha256/0jibyvyv2jy8dq5ij0j00iq3v74r0y90dcjc3dkspcfbnn37cphn...
> download failed
> "https://tarballs.nixos.org/sha256/0jibyvyv2jy8dq5ij0j00iq3v74r0y90dcjc3dkspcfbnn37cph
> n" 404 "Not Found"
> 
> Starting download of
> /gnu/store/l6309vrm95cwpmcwgjj5xgdddp6vfly5-mono-4.8.1.tar.bz2
> From
> https://archive.softwareheritage.org/api/1/content/sha256:165e7686b5cbb1ab671b4cb2069207999c3d700
> 44002190b6ec84bb1fdf62b4a/raw/... downloading from
> https://archive.softwareheritage.org/api/1/content/sha256:165e7686b5cbb1ab671b4cb2069
> 207999c3d70044002190b6ec84bb1fdf62b4a/raw/ ...
> ............
> successfully built
> /gnu/store/6aih7gpx3p1dpsbdhkri7mk3qqnqkk9x-mono-4.8.1.tar.bz2.drv
> building
> /gnu/store/qs8jgc33jpwwmzq7ppdiv49qll6q32sl-mono-4.8.1.tar.xz.drv...
> mono-4.4.1/ mono-4.4.1/test-driver
> mono-4.4.1/support/
> mono-4.4.1/support/syslog.c
> mono-4.4.1/support/supportw.h
> ....
> 
> --end of log fratgment--
> 
> 
> as seen above, somehow the old version was downloaded from a cached
> copy at softwareheritage archives, and it proceeds to build.   This
> should not proceed but fail for wrong checksum.
> 
> 
> 

This is somewhat unfortunate, but also kind of expected behaviour.
Fixed output derivations don't have version information as far as I
know and part of the usefulness of cryptographic checksums on them is
that they can be downloaded from anywhere, even when their original
servers go down.
There isn't really any practical way I can see for testing if the
checksum is correct when the URL is wrong. How should Guix know what
file it should download?
I think the best workflow is to `use guix refresh -u` instead of
manually updating the package, but sadly that doesn't always work,
although I think the git based updater was merged recently, which
should improve your chances of success.

bug#52591: downloading from software archives as default even if version is wrong

[Andy Tai]

On Fri, Dec 17, 2021 at 4:10 PM Tobias Geerinckx-Rice <me@tobias.gr> wrote:
> OK to close as NOTABUG?
sure

bug#52591: downloading from software archives as default even if version is wrong

[Leo Famulari]

On Fri, Dec 17, 2021 at 04:23:11PM -0800, Andy Tai wrote:
> On Fri, Dec 17, 2021 at 4:10 PM Tobias Geerinckx-Rice <me@tobias.gr> wrote:
> > OK to close as NOTABUG?
> sure

Wait! :)

Although I agree that this behaviour is not a bug, it's something that
many Guix contributors discover independently and find confusing, or
even distressing.

And we sometimes receive patches that include the wrong source hash or
wrong URL because of similar issues; sometimes these mistakes are even
committed to guix.git.

Maybe there is *something* we can do to improve this experience, if only
a section in the manual or a tweak to the UI.

Any ideas?

bug#52591: downloading from software archives as default even if version is wrong

[Tobias Geerinckx-Rice via Bug reports for GNU Guix]

[-- Attachment #1: Type: text/plain, Size: 222 bytes --]

Leo Famulari 写道：
> Maybe there is *something* we can do to improve this experience, 
> if only
> a section in the manual or a tweak to the UI.

OK!  Let's keep it open for discussion.

Kind regards,

T G-R

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 247 bytes --]