git-fetch without a hash

git-fetch without a hash

[Stephen Paul Weber]

[-- Attachment #1: Type: text/plain, Size: 1268 bytes --]

Hello Guix!

It seem that url-fetch will work without a hash (that is, with (sha256 #f)) 
but git-fetch will not.

As near as I can tell this is because git-fetch uses a fixed derivation 
build going via nix/build.cc stuff which contains this line:

	if (i.second.hash == "") fixedOutput = false;

And this results in /etc/resolv.conf not getting written and DNS resolution 
failing.

Now, I *think* by my reading that it is intended for *only* fixed-output 
derivations to have network access, because otherwise the operation would be 
impure?

And then url-fetch is just not actually ending up in a build container and 
so it works even when it isn't fixed-output?

However, there's no real reason that git-fetch *needs* to be fixed-output in 
terms of having a hash pre-defined, at least for local development and other 
purposes.  So is there a way around this?  If having a way around it is not 
desirable should url-fetch consider this an error as well?

Finally, *if* git-fetch should not allow this, the current error message is 
beyond confusing (DNS resolution fails during git fetch and it tries to fall 
back to SWH).  So should git-fetch check for hash of #f and raise at that 
point with a better error message in that case?

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: git-fetch without a hash

[Ludovic Courtès]

Hi,

Stephen Paul Weber <singpolyma@singpolyma.net> skribis:

> It seem that url-fetch will work without a hash (that is, with (sha256
> #f)) but git-fetch will not.

(sha256 #f) is not a documented use case.  :-)

> As near as I can tell this is because git-fetch uses a fixed
> derivation build going via nix/build.cc stuff which contains this
> line:
>
> 	if (i.second.hash == "") fixedOutput = false;
>
> And this results in /etc/resolv.conf not getting written and DNS
> resolution failing.
>
> Now, I *think* by my reading that it is intended for *only*
> fixed-output derivations to have network access, because otherwise the
> operation would be impure?

Yes, exactly.

> And then url-fetch is just not actually ending up in a build container
> and so it works even when it isn't fixed-output?

If an origin has (sha256 #f), it is *not* lowered to a fixed-output
derivation; consequently, its build environment lacks network access,
etc.

> However, there's no real reason that git-fetch *needs* to be
> fixed-output in terms of having a hash pre-defined, at least for local
> development and other purposes.  So is there a way around this?

What’s your use case?

If you want a package to refer to, say, the tip of the “main” branch of
some Git repo, then you can:

  • use ‘--with-branch=PKG=main’, or

  • write (package (source (git-checkout …)) …)

Both are actually equivalent.  ‘git-checkout’ represents a checkout made
on the client side, without knowing in advance what you’ll get out of
this checkout.  (So use with care, too.)

> If having a way around it is not desirable should url-fetch consider
> this an error as well?

I’m not sure; do you have an example where it’s not behaving as
expected?

> Finally, *if* git-fetch should not allow this, the current error
> message is beyond confusing (DNS resolution fails during git fetch and
> it tries to fall back to SWH).  So should git-fetch check for hash of
> #f and raise at that point with a better error message in that case?

Ideally, ‘origin’ would prevent a hash of #f altogether.  However, there
are a couple of weird use cases in (gnu packages …) that need to be
supported.  We should probably consider using a different trick there,
such as using a computed-file instead of an origin; we’ll have to check
what’s feasible.

Thanks,
Ludo’.

Re: git-fetch without a hash

[Stephen Paul Weber]

[-- Attachment #1: Type: text/plain, Size: 712 bytes --]

>> However, there's no real reason that git-fetch *needs* to be
>> fixed-output in terms of having a hash pre-defined, at least for local
>> development and other purposes.  So is there a way around this?
>
>  • write (package (source (git-checkout …)) …)

This works well.  Now I'm curious how to know what can go in the (source) 
field?  Obviously not just (origin)...

>> If having a way around it is not desirable should url-fetch consider
>> this an error as well?
>
>I’m not sure; do you have an example where it’s not behaving as
>expected?

Yes.  When using (sha256 #f) url-fetch still has network access and works to 
download things, which is inconsistent vs other fetchers.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: git-fetch without a hash

[Ludovic Courtès]

Hi!

Stephen Paul Weber <singpolyma@singpolyma.net> skribis:

>>> However, there's no real reason that git-fetch *needs* to be
>>> fixed-output in terms of having a hash pre-defined, at least for local
>>> development and other purposes.  So is there a way around this?
>>
>>  • write (package (source (git-checkout …)) …)
>
> This works well.  Now I'm curious how to know what can go in the
> (source) field?  Obviously not just (origin)...

Any “file-like object” can go there: a package (though that makes little
sense), a “computed-file”, etc.

>>> If having a way around it is not desirable should url-fetch consider
>>> this an error as well?
>>
>>I’m not sure; do you have an example where it’s not behaving as
>>expected?
>
> Yes.  When using (sha256 #f) url-fetch still has network access and
> works to download things, which is inconsistent vs other fetchers.

Hmm indeed:

--8<---------------cut here---------------start------------->8---
scheme@(guile-user)> ,use(guix)
scheme@(guile-user)> ,verbosity 3
scheme@(guile-user)> ,build (origin
			      (method url-fetch)
			      (uri "mirror://gnu/hello/hello-2.12.1.tar.gz")
			      (sha256 #f))
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://bordeaux.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://guix.bordeaux.inria.fr'... 100.0%
building /gnu/store/lz34lhyxhq5wxj87fnd465hmwbhv17bn-hello-2.12.1.tar.gz.drv...

Starting download of /gnu/store/xfc7gsn10j09bi89ldbpfbppfkcldfy9-hello-2.12.1.tar.gz
From https://ftpmirror.gnu.org/gnu/hello/hello-2.12.1.tar.gz...
following redirection to `https://mirror.cyberbits.eu/gnu/hello/hello-2.12.1.tar.gz'...
downloading from https://ftpmirror.gnu.org/gnu/hello/hello-2.12.1.tar.gz ...
 hello-2.12.1.tar.gz  1009KiB                                                                                                                       18.4MiB/s 00:00 [##################] 100.0%
successfully built /gnu/store/lz34lhyxhq5wxj87fnd465hmwbhv17bn-hello-2.12.1.tar.gz.drv
$5 = "/gnu/store/xfc7gsn10j09bi89ldbpfbppfkcldfy9-hello-2.12.1.tar.gz"
--8<---------------cut here---------------end--------------->8---

In this case, the derivation uses the “download” built-in builder:

--8<---------------cut here---------------start------------->8---
scheme@(guile-user)> ,lower (origin
			      (method url-fetch)
			      (uri "mirror://gnu/hello/hello-2.12.1.tar.gz")
			      (sha256 #f))
$6 = #<derivation /gnu/store/lz34lhyxhq5wxj87fnd465hmwbhv17bn-hello-2.12.1.tar.gz.drv => /gnu/store/xfc7gsn10j09bi89ldbpfbppfkcldfy9-hello-2.12.1.tar.gz 7f3ff487c000>
scheme@(guile-user)> (derivation-outputs $6)
$7 = (("out" . #<<derivation-output> path: "/gnu/store/xfc7gsn10j09bi89ldbpfbppfkcldfy9-hello-2.12.1.tar.gz" hash-algo: sha256 hash: #f recursive?: #f>))
scheme@(guile-user)> (fixed-output-derivation? $6)
$8 = #f
--8<---------------cut here---------------end--------------->8---

As it turns out, guix-daemon turns off the chroot for all built-in
builders, fixed-output or not; quoth ‘build.cc’:

    /* Note: built-in builders are *not* running in a chroot environment so
       that we can easily implement them in Guile without having it as a
       derivation input (they are running under a separate build user,
       though).  */
    useChroot = settings.useChroot && !isBuiltin(drv);

I was actually surprised to see this, this wasn’t really intended.  The
good news is that this is safe AFAICS: there’s only one built-in builder
to date, and that’s “download”.

Now I wonder whether we should keep this “feature” or not.  Probably not.

Thoughts?

Besides, we can think about adding more builtins, such as a
“git-download” builtin, which would let us break potential cycles.

Ludo’.

Re: git-fetch without a hash

[Simon Tournier]

Hi,

On Tue, 03 Jan 2023 at 22:34, Ludovic Courtès <ludo@gnu.org> wrote:

>> Yes.  When using (sha256 #f) url-fetch still has network access and
>> works to download things, which is inconsistent vs other fetchers.
>
> Hmm indeed:

On Wed, 21 Dec 2022 at 23:49, Ludovic Courtès <ludo@gnu.org> wrote:

> (sha256 #f) is not a documented use case.  :-)

Maybe my question is naive but what is the use case for this (sha256 #f)
in the first place?  Because maybe it could just error using some
’sanitize’ for the hash record field.


Cheers,
simon

Re: git-fetch without a hash

[Ludovic Courtès]

Simon Tournier <zimon.toutoune@gmail.com> skribis:

> Maybe my question is naive but what is the use case for this (sha256 #f)
> in the first place?  Because maybe it could just error using some
> ’sanitize’ for the hash record field.

There’s a couple of uses: Chromium, IceCat, and Linux-libre (IIRC).

I don’t like that, but I’m not sure what it would take to change these
to <computed-file> or something like that.

Ludo’.

Re: git-fetch without a hash

[Simon Tournier]

Hi,

On Mon, 09 Jan 2023 at 12:16, Ludovic Courtès <ludo@gnu.org> wrote:
> Simon Tournier <zimon.toutoune@gmail.com> skribis:
>
>> Maybe my question is naive but what is the use case for this (sha256 #f)
>> in the first place?  Because maybe it could just error using some
>> ’sanitize’ for the hash record field.
>
> There’s a couple of uses: Chromium, IceCat, and Linux-libre (IIRC).
>
> I don’t like that, but I’m not sure what it would take to change these
> to <computed-file> or something like that.

Well, from (gnu packages linux)

--8<---------------cut here---------------start------------->8---
     (origin
       (method computed-origin-method)
       (file-name (string-append "linux-libre-" version "-guix.tar.xz"))
       (sha256 #f)
--8<---------------cut here---------------end--------------->8---

and from (gnu packages gnuzilla)

--8<---------------cut here---------------start------------->8---
    (origin
      (method computed-origin-method)
      (file-name (string-append "icecat-" %icecat-version ".tar.xz"))
      (sha256 #f)
--8<---------------cut here---------------end--------------->8---

but not from Chromium, if I read correctly.

From my understanding, we could have something like,

      (sha256 (no-hash))

where ’no-hash’ would return a string, say
"0000000000000000000000000000000000000000000000000000" or whatever else
that would satisfy this hypothetical  ’sha256’ sanitizer.


Cheers,
simon

Re: git-fetch without a hash

[bokr]

Hi,

On +2023-01-11 16:34:41 +0100, Simon Tournier wrote:
> Hi,
> 
> On Mon, 09 Jan 2023 at 12:16, Ludovic Courtès <ludo@gnu.org> wrote:
> > Simon Tournier <zimon.toutoune@gmail.com> skribis:
> >
> >> Maybe my question is naive but what is the use case for this (sha256 #f)
> >> in the first place?  Because maybe it could just error using some
> >> ’sanitize’ for the hash record field.
> >
> > There’s a couple of uses: Chromium, IceCat, and Linux-libre (IIRC).
> >
> > I don’t like that, but I’m not sure what it would take to change these
> > to <computed-file> or something like that.
> 
> Well, from (gnu packages linux)
> 
> --8<---------------cut here---------------start------------->8---
>      (origin
>        (method computed-origin-method)
>        (file-name (string-append "linux-libre-" version "-guix.tar.xz"))
>        (sha256 #f)
> --8<---------------cut here---------------end--------------->8---
> 
> and from (gnu packages gnuzilla)
> 
> --8<---------------cut here---------------start------------->8---
>     (origin
>       (method computed-origin-method)
>       (file-name (string-append "icecat-" %icecat-version ".tar.xz"))
>       (sha256 #f)
> --8<---------------cut here---------------end--------------->8---
> 
> but not from Chromium, if I read correctly.
> 
> From my understanding, we could have something like,
> 
>       (sha256 (no-hash))
> 
> where ’no-hash’ would return a string, say
> "0000000000000000000000000000000000000000000000000000" or whatever else
> that would satisfy this hypothetical  ’sha256’ sanitizer.
> 
> 
> Cheers,
> simon
>

For portability to any hash algorithm that returns a hex string,
how about letting them hash a zero-length string (which can never
represent a package tarball or other archive), and using the
resulting strings as no-hash flags?

These strings must be unique for whatever hash algorithm,
so a short table could be used to recognize them as
no-hash indicators.

--8<---------------cut here---------------start------------->8---
$ sha256sum /dev/null
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855  /dev/null
$ sha256sum <(echo -n '')
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855  /dev/fd/63
$ echo -n ''|sha256sum
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855  -
$ echo -n ""|sha256sum
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855  -
--8<---------------cut here---------------end--------------->8---


For other hash values on your system, probably the below will show most.
Translating from hex to various base32 and other-base alphabets
is then trivial, (and open, i.e. permitting new hash representation strings).
--8<---------------cut here---------------start------------->8---
$ ls -1d /usr/bin/*sum|while read hasher;do \
  echo;echo "$hasher:"; "$hasher" /dev/null;done
--8<---------------cut here---------------end--------------->8---

WDYT?
--
Regards,
Bengt Richter

Re: git-fetch without a hash

[Simon Tournier]

Hi,

On dim., 05 févr. 2023 at 18:44, bokr@bokr.com wrote:

>> From my understanding, we could have something like,
>> 
>>       (sha256 (no-hash))
>> 
>> where ’no-hash’ would return a string, say
>> "0000000000000000000000000000000000000000000000000000" or whatever else
>> that would satisfy this hypothetical  ’sha256’ sanitizer.

> For portability to any hash algorithm that returns a hex string,
> how about letting them hash a zero-length string (which can never
> represent a package tarball or other archive), and using the
> resulting strings as no-hash flags?

Instead of “0000000000000000000000000000000000000000000000000000”, you
are proposing “0mdqa9w1p6cmli6976v4wi0sw9r4p5prkj7lzfd1877wk11c9c73”
which is the Guix hash of empty.

> $ sha256sum /dev/null
> e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855  /dev/null

    $ touch foo
    $ guix hash foo
    0mdqa9w1p6cmli6976v4wi0sw9r4p5prkj7lzfd1877wk11c9c73
    $ guix hash foo -f hex -H sha256
    e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

Yeah, probably better. :-)

> These strings must be unique for whatever hash algorithm,
> so a short table could be used to recognize them as
> no-hash indicators.

Well, I do not understand “These strings must be unique for whatever
hash algorithm”.  What do you mean?

Cheers,
simon

Re: git-fetch without a hash

[Ludovic Courtès]

Hi,

Ludovic Courtès <ludo@gnu.org> skribis:

> scheme@(guile-user)> ,lower (origin
> 			      (method url-fetch)
> 			      (uri "mirror://gnu/hello/hello-2.12.1.tar.gz")
> 			      (sha256 #f))
> $6 = #<derivation /gnu/store/lz34lhyxhq5wxj87fnd465hmwbhv17bn-hello-2.12.1.tar.gz.drv => /gnu/store/xfc7gsn10j09bi89ldbpfbppfkcldfy9-hello-2.12.1.tar.gz 7f3ff487c000>
> scheme@(guile-user)> (derivation-outputs $6)
> $7 = (("out" . #<<derivation-output> path: "/gnu/store/xfc7gsn10j09bi89ldbpfbppfkcldfy9-hello-2.12.1.tar.gz" hash-algo: sha256 hash: #f recursive?: #f>))
> scheme@(guile-user)> (fixed-output-derivation? $6)
> $8 = #f
>
> As it turns out, guix-daemon turns off the chroot for all built-in
> builders, fixed-output or not; quoth ‘build.cc’:
>
>     /* Note: built-in builders are *not* running in a chroot environment so
>        that we can easily implement them in Guile without having it as a
>        derivation input (they are running under a separate build user,
>        though).  */
>     useChroot = settings.useChroot && !isBuiltin(drv);
>
> I was actually surprised to see this, this wasn’t really intended.  The
> good news is that this is safe AFAICS: there’s only one built-in builder
> to date, and that’s “download”.

After investigating, I found that ‘guix perform-download’ already has a
check in place:

      (unless (and algo hash)
        (leave (G_ "~a is not a fixed-output derivation~%")
               (derivation-file-name drv)))

The check wasn’t working because ‘hash’ was #vu8() instead of #f.  This
is fixed in 5d24e57a611b43ff68700379338b899f62d198cc.

Once we update the ‘guix’ package, the daemon will no longer accept
‘url-fetch’ downloads with hash = #f.

Thanks again for reporting this oddity!

Ludo’.

Re: git-fetch without a hash

[Stephen Paul Weber]

>Once we update the ‘guix’ package, the daemon will no longer accept
>‘url-fetch’ downloads with hash = #f.

Ok.  Is there something like (git-checkout) we can use that will work 
instead of an (origin) here for url downloads?

Re: git-fetch without a hash

[Ludovic Courtès]

Hi,

Stephen Paul Weber <singpolyma@singpolyma.net> skribis:

>>Once we update the ‘guix’ package, the daemon will no longer accept
>>‘url-fetch’ downloads with hash = #f.
>
> Ok.  Is there something like (git-checkout) we can use that will work
> instead of an (origin) here for url downloads?

There’s <download-file> in (guix transformations), but it’s currently
private.

We could move it to (guix download) and document it.

Thanks,
Ludo’.