Re: 01/01: gnu: gource: Fix the hashes of mutated GitHub archives.

Re: 01/01: gnu: gource: Fix the hashes of mutated GitHub archives.

[Leo Famulari]

[-- Attachment #1: Type: text/plain, Size: 1606 bytes --]

On Thu, Jan 25, 2018 at 09:17:38AM -0500, Oleg Pykhalov wrote:
> wigust pushed a commit to branch master
> in repository guix.
> 
> commit 45b486984d8ab092cf002cd0b500df4dc62e186b
> Author: Oleg Pykhalov <go.wigust@gmail.com>
> Date:   Thu Jan 25 16:58:35 2018 +0300
> 
>     gnu: gource: Fix the hashes of mutated GitHub archives.
>     
>     * gnu/packages/version-control.scm (gource): Fix hash.

> -                    "https://github.com/acaudwell/Gource/archive/"
> -                    "gource-" version ".tar.gz"))
> +                    "https://github.com/acaudwell/Gource/releases/download"
> +                    "/gource-" version "/gource-" version ".tar.gz"))

Hey, thanks for fixing this up.

The commit message made me think that the hash had changed, but based on
this commit it seems that the URL changed somehow, or was originally
incorrect.

In cases where the hash actually changed, please send a message to
bug-guix so we can investigate publicy.

The automatically created per-tag GitHub snapshots are not guaranteed to
be cached forever by GitHub or recreated deterministically, so their
hashes are subject to change. [0]

Additionally, if a packager uses `guix download` to check the hash of
some file, but uses an incorrect URL in the package definition, Guix
will use the file in /gnu/store and never try the URL. So it's easy to
commit the wrong URL if you use `guix download`. Instead I recommend
downloading the file outside of Guix and using `guix hash`.

[0]
https://github.com/libgit2/libgit2/issues/4343
https://bugs.gnu.org/28659

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: 01/01: gnu: gource: Fix the hashes of mutated GitHub archives.

[Oleg Pykhalov]

[-- Attachment #1: Type: text/plain, Size: 2730 bytes --]

Hello Leo,

Leo Famulari <leo@famulari.name> writes:

> On Thu, Jan 25, 2018 at 09:17:38AM -0500, Oleg Pykhalov wrote:
>> wigust pushed a commit to branch master
>> in repository guix.
>> 
>> commit 45b486984d8ab092cf002cd0b500df4dc62e186b
>> Author: Oleg Pykhalov <go.wigust@gmail.com>
>> Date:   Thu Jan 25 16:58:35 2018 +0300
>> 
>>     gnu: gource: Fix the hashes of mutated GitHub archives.
>>     
>>     * gnu/packages/version-control.scm (gource): Fix hash.
>
>> -                    "https://github.com/acaudwell/Gource/archive/"
>> -                    "gource-" version ".tar.gz"))
>> +                    "https://github.com/acaudwell/Gource/releases/download"
>> +                    "/gource-" version "/gource-" version ".tar.gz"))
>
> Hey, thanks for fixing this up.
>
> The commit message made me think that the hash had changed

I thought about this a little bit differently.  The commit changes URL,
you right.  But because it fixes a wrong hash during build, confused me.

> , but based on this commit it seems that the URL changed somehow, or
> was originally incorrect.

The URL was originally incorrect.

> In cases where the hash actually changed, please send a message to
> bug-guix so we can investigate publicy.

OK.

> The automatically created per-tag GitHub snapshots are not guaranteed to
> be cached forever by GitHub or recreated deterministically, so their
> hashes are subject to change. [0]

OK.  Thank you for the reference.

> Additionally, if a packager uses `guix download` to check the hash of
> some file, but uses an incorrect URL in the package definition, Guix
> will use the file in /gnu/store and never try the URL. So it's easy to
> commit the wrong URL if you use `guix download`. Instead I recommend
> downloading the file outside of Guix and using `guix hash`.

Ah, thank you!  I think because Guix doesn't make a new derivation if
the URL in package recipe was changed.  But it's not clear if you don't
think about that carefully.


Could we have following warnings in the documentation?

  - GitHub archive could lead to non-reproducible source tarball, please
    use a release tarball if it is available.

  - If you use a @code{guix download} command to check the hash of some
    file, but use an incorrect URL in the package definition, Guix will
    use the file in @file{/gnu/store/…pack.tar.gz} and never try the
    URL.  So it's easy to commit the wrong URL if you use @code{guix
    download}. Instead recommended to download the file outside of Guix
    and use a @code{guix hash} command.

> [0]
> https://github.com/libgit2/libgit2/issues/4343
> https://bugs.gnu.org/28659

Thanks,
Oleg.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: 01/01: gnu: gource: Fix the hashes of mutated GitHub archives.

[Leo Famulari]

[-- Attachment #1: Type: text/plain, Size: 2273 bytes --]

On Sun, Jan 28, 2018 at 08:36:42PM +0300, Oleg Pykhalov wrote:
> Leo Famulari <leo@famulari.name> writes:
> > Additionally, if a packager uses `guix download` to check the hash of
> > some file, but uses an incorrect URL in the package definition, Guix
> > will use the file in /gnu/store and never try the URL. So it's easy to
> > commit the wrong URL if you use `guix download`. Instead I recommend
> > downloading the file outside of Guix and using `guix hash`.
> 
> Ah, thank you!  I think because Guix doesn't make a new derivation if
> the URL in package recipe was changed.  But it's not clear if you don't
> think about that carefully.

Yes, this is tricky.

> Could we have following warnings in the documentation?
> 
>   - GitHub archive could lead to non-reproducible source tarball, please
>     use a release tarball if it is available.

The problem of unstable upstream sources is a general problem, not
limited to GitHub. We noticed it recently on GitHub because they host so
many projects, but it happens at other mega-hosters and also with
self-hosted projects.

We use content addressing to make it easier to preserve and find these
sources over time. Guix will look on any substitute servers you are
using, our own content-addressed storage, and the Nix project's
content-addressed storage. The Software Heritage project [0] exists
to address this specific problem, and we'd like to eventually try
fetching sources from them, too.

>   - If you use a @code{guix download} command to check the hash of some
>     file, but use an incorrect URL in the package definition, Guix will
>     use the file in @file{/gnu/store/…pack.tar.gz} and never try the
>     URL.  So it's easy to commit the wrong URL if you use @code{guix
>     download}. Instead recommended to download the file outside of Guix
>     and use a @code{guix hash} command.

Something like this would be helpful, but I'd like to write it more
carefully, and also think about exactly where in the manual it should
go.

To me the obvious choices are 'Invoking guix download' and 'Packaging
Guidelines'.

I'm also wary of filling the manual with warnings and caveats which
could overwhelm the reader...

[0] https://www.softwareheritage.org/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]