* git-fetch without a hash @ 2022-12-21 3:04 Stephen Paul Weber 2022-12-21 22:49 ` Ludovic Courtès 0 siblings, 1 reply; 12+ messages in thread From: Stephen Paul Weber @ 2022-12-21 3:04 UTC (permalink / raw) To: guix-devel; +Cc: 0 [-- Attachment #1: Type: text/plain, Size: 1268 bytes --] Hello Guix! It seem that url-fetch will work without a hash (that is, with (sha256 #f)) but git-fetch will not. As near as I can tell this is because git-fetch uses a fixed derivation build going via nix/build.cc stuff which contains this line: if (i.second.hash == "") fixedOutput = false; And this results in /etc/resolv.conf not getting written and DNS resolution failing. Now, I *think* by my reading that it is intended for *only* fixed-output derivations to have network access, because otherwise the operation would be impure? And then url-fetch is just not actually ending up in a build container and so it works even when it isn't fixed-output? However, there's no real reason that git-fetch *needs* to be fixed-output in terms of having a hash pre-defined, at least for local development and other purposes. So is there a way around this? If having a way around it is not desirable should url-fetch consider this an error as well? Finally, *if* git-fetch should not allow this, the current error message is beyond confusing (DNS resolution fails during git fetch and it tries to fall back to SWH). So should git-fetch check for hash of #f and raise at that point with a better error message in that case? [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 833 bytes --] ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: git-fetch without a hash 2022-12-21 3:04 git-fetch without a hash Stephen Paul Weber @ 2022-12-21 22:49 ` Ludovic Courtès 2023-01-03 19:31 ` Stephen Paul Weber 0 siblings, 1 reply; 12+ messages in thread From: Ludovic Courtès @ 2022-12-21 22:49 UTC (permalink / raw) To: Stephen Paul Weber; +Cc: guix-devel, 0 Hi, Stephen Paul Weber <singpolyma@singpolyma.net> skribis: > It seem that url-fetch will work without a hash (that is, with (sha256 > #f)) but git-fetch will not. (sha256 #f) is not a documented use case. :-) > As near as I can tell this is because git-fetch uses a fixed > derivation build going via nix/build.cc stuff which contains this > line: > > if (i.second.hash == "") fixedOutput = false; > > And this results in /etc/resolv.conf not getting written and DNS > resolution failing. > > Now, I *think* by my reading that it is intended for *only* > fixed-output derivations to have network access, because otherwise the > operation would be impure? Yes, exactly. > And then url-fetch is just not actually ending up in a build container > and so it works even when it isn't fixed-output? If an origin has (sha256 #f), it is *not* lowered to a fixed-output derivation; consequently, its build environment lacks network access, etc. > However, there's no real reason that git-fetch *needs* to be > fixed-output in terms of having a hash pre-defined, at least for local > development and other purposes. So is there a way around this? What’s your use case? If you want a package to refer to, say, the tip of the “main” branch of some Git repo, then you can: • use ‘--with-branch=PKG=main’, or • write (package (source (git-checkout …)) …) Both are actually equivalent. ‘git-checkout’ represents a checkout made on the client side, without knowing in advance what you’ll get out of this checkout. (So use with care, too.) > If having a way around it is not desirable should url-fetch consider > this an error as well? I’m not sure; do you have an example where it’s not behaving as expected? > Finally, *if* git-fetch should not allow this, the current error > message is beyond confusing (DNS resolution fails during git fetch and > it tries to fall back to SWH). So should git-fetch check for hash of > #f and raise at that point with a better error message in that case? Ideally, ‘origin’ would prevent a hash of #f altogether. However, there are a couple of weird use cases in (gnu packages …) that need to be supported. We should probably consider using a different trick there, such as using a computed-file instead of an origin; we’ll have to check what’s feasible. Thanks, Ludo’. ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: git-fetch without a hash 2022-12-21 22:49 ` Ludovic Courtès @ 2023-01-03 19:31 ` Stephen Paul Weber 2023-01-03 21:34 ` Ludovic Courtès 0 siblings, 1 reply; 12+ messages in thread From: Stephen Paul Weber @ 2023-01-03 19:31 UTC (permalink / raw) To: Ludovic Courtès; +Cc: guix-devel, 0 [-- Attachment #1: Type: text/plain, Size: 712 bytes --] >> However, there's no real reason that git-fetch *needs* to be >> fixed-output in terms of having a hash pre-defined, at least for local >> development and other purposes. So is there a way around this? > > • write (package (source (git-checkout …)) …) This works well. Now I'm curious how to know what can go in the (source) field? Obviously not just (origin)... >> If having a way around it is not desirable should url-fetch consider >> this an error as well? > >I’m not sure; do you have an example where it’s not behaving as >expected? Yes. When using (sha256 #f) url-fetch still has network access and works to download things, which is inconsistent vs other fetchers. [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 833 bytes --] ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: git-fetch without a hash 2023-01-03 19:31 ` Stephen Paul Weber @ 2023-01-03 21:34 ` Ludovic Courtès 2023-01-05 10:06 ` Simon Tournier 2023-01-09 17:13 ` Ludovic Courtès 0 siblings, 2 replies; 12+ messages in thread From: Ludovic Courtès @ 2023-01-03 21:34 UTC (permalink / raw) To: Stephen Paul Weber; +Cc: guix-devel, 0 Hi! Stephen Paul Weber <singpolyma@singpolyma.net> skribis: >>> However, there's no real reason that git-fetch *needs* to be >>> fixed-output in terms of having a hash pre-defined, at least for local >>> development and other purposes. So is there a way around this? >> >> • write (package (source (git-checkout …)) …) > > This works well. Now I'm curious how to know what can go in the > (source) field? Obviously not just (origin)... Any “file-like object” can go there: a package (though that makes little sense), a “computed-file”, etc. >>> If having a way around it is not desirable should url-fetch consider >>> this an error as well? >> >>I’m not sure; do you have an example where it’s not behaving as >>expected? > > Yes. When using (sha256 #f) url-fetch still has network access and > works to download things, which is inconsistent vs other fetchers. Hmm indeed: --8<---------------cut here---------------start------------->8--- scheme@(guile-user)> ,use(guix) scheme@(guile-user)> ,verbosity 3 scheme@(guile-user)> ,build (origin (method url-fetch) (uri "mirror://gnu/hello/hello-2.12.1.tar.gz") (sha256 #f)) substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0% substitute: updating substitutes from 'https://bordeaux.guix.gnu.org'... 100.0% substitute: updating substitutes from 'https://guix.bordeaux.inria.fr'... 100.0% building /gnu/store/lz34lhyxhq5wxj87fnd465hmwbhv17bn-hello-2.12.1.tar.gz.drv... Starting download of /gnu/store/xfc7gsn10j09bi89ldbpfbppfkcldfy9-hello-2.12.1.tar.gz From https://ftpmirror.gnu.org/gnu/hello/hello-2.12.1.tar.gz... following redirection to `https://mirror.cyberbits.eu/gnu/hello/hello-2.12.1.tar.gz'... downloading from https://ftpmirror.gnu.org/gnu/hello/hello-2.12.1.tar.gz ... hello-2.12.1.tar.gz 1009KiB 18.4MiB/s 00:00 [##################] 100.0% successfully built /gnu/store/lz34lhyxhq5wxj87fnd465hmwbhv17bn-hello-2.12.1.tar.gz.drv $5 = "/gnu/store/xfc7gsn10j09bi89ldbpfbppfkcldfy9-hello-2.12.1.tar.gz" --8<---------------cut here---------------end--------------->8--- In this case, the derivation uses the “download” built-in builder: --8<---------------cut here---------------start------------->8--- scheme@(guile-user)> ,lower (origin (method url-fetch) (uri "mirror://gnu/hello/hello-2.12.1.tar.gz") (sha256 #f)) $6 = #<derivation /gnu/store/lz34lhyxhq5wxj87fnd465hmwbhv17bn-hello-2.12.1.tar.gz.drv => /gnu/store/xfc7gsn10j09bi89ldbpfbppfkcldfy9-hello-2.12.1.tar.gz 7f3ff487c000> scheme@(guile-user)> (derivation-outputs $6) $7 = (("out" . #<<derivation-output> path: "/gnu/store/xfc7gsn10j09bi89ldbpfbppfkcldfy9-hello-2.12.1.tar.gz" hash-algo: sha256 hash: #f recursive?: #f>)) scheme@(guile-user)> (fixed-output-derivation? $6) $8 = #f --8<---------------cut here---------------end--------------->8--- As it turns out, guix-daemon turns off the chroot for all built-in builders, fixed-output or not; quoth ‘build.cc’: /* Note: built-in builders are *not* running in a chroot environment so that we can easily implement them in Guile without having it as a derivation input (they are running under a separate build user, though). */ useChroot = settings.useChroot && !isBuiltin(drv); I was actually surprised to see this, this wasn’t really intended. The good news is that this is safe AFAICS: there’s only one built-in builder to date, and that’s “download”. Now I wonder whether we should keep this “feature” or not. Probably not. Thoughts? Besides, we can think about adding more builtins, such as a “git-download” builtin, which would let us break potential cycles. Ludo’. ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: git-fetch without a hash 2023-01-03 21:34 ` Ludovic Courtès @ 2023-01-05 10:06 ` Simon Tournier 2023-01-09 11:16 ` Ludovic Courtès 2023-01-09 17:13 ` Ludovic Courtès 1 sibling, 1 reply; 12+ messages in thread From: Simon Tournier @ 2023-01-05 10:06 UTC (permalink / raw) To: Ludovic Courtès, Stephen Paul Weber; +Cc: guix-devel, 0 Hi, On Tue, 03 Jan 2023 at 22:34, Ludovic Courtès <ludo@gnu.org> wrote: >> Yes. When using (sha256 #f) url-fetch still has network access and >> works to download things, which is inconsistent vs other fetchers. > > Hmm indeed: On Wed, 21 Dec 2022 at 23:49, Ludovic Courtès <ludo@gnu.org> wrote: > (sha256 #f) is not a documented use case. :-) Maybe my question is naive but what is the use case for this (sha256 #f) in the first place? Because maybe it could just error using some ’sanitize’ for the hash record field. Cheers, simon ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: git-fetch without a hash 2023-01-05 10:06 ` Simon Tournier @ 2023-01-09 11:16 ` Ludovic Courtès 2023-01-11 15:34 ` Simon Tournier 0 siblings, 1 reply; 12+ messages in thread From: Ludovic Courtès @ 2023-01-09 11:16 UTC (permalink / raw) To: Simon Tournier; +Cc: Stephen Paul Weber, guix-devel, 0 Simon Tournier <zimon.toutoune@gmail.com> skribis: > Maybe my question is naive but what is the use case for this (sha256 #f) > in the first place? Because maybe it could just error using some > ’sanitize’ for the hash record field. There’s a couple of uses: Chromium, IceCat, and Linux-libre (IIRC). I don’t like that, but I’m not sure what it would take to change these to <computed-file> or something like that. Ludo’. ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: git-fetch without a hash 2023-01-09 11:16 ` Ludovic Courtès @ 2023-01-11 15:34 ` Simon Tournier 2023-02-05 17:44 ` bokr 0 siblings, 1 reply; 12+ messages in thread From: Simon Tournier @ 2023-01-11 15:34 UTC (permalink / raw) To: Ludovic Courtès; +Cc: Stephen Paul Weber, guix-devel, 0 Hi, On Mon, 09 Jan 2023 at 12:16, Ludovic Courtès <ludo@gnu.org> wrote: > Simon Tournier <zimon.toutoune@gmail.com> skribis: > >> Maybe my question is naive but what is the use case for this (sha256 #f) >> in the first place? Because maybe it could just error using some >> ’sanitize’ for the hash record field. > > There’s a couple of uses: Chromium, IceCat, and Linux-libre (IIRC). > > I don’t like that, but I’m not sure what it would take to change these > to <computed-file> or something like that. Well, from (gnu packages linux) --8<---------------cut here---------------start------------->8--- (origin (method computed-origin-method) (file-name (string-append "linux-libre-" version "-guix.tar.xz")) (sha256 #f) --8<---------------cut here---------------end--------------->8--- and from (gnu packages gnuzilla) --8<---------------cut here---------------start------------->8--- (origin (method computed-origin-method) (file-name (string-append "icecat-" %icecat-version ".tar.xz")) (sha256 #f) --8<---------------cut here---------------end--------------->8--- but not from Chromium, if I read correctly. From my understanding, we could have something like, (sha256 (no-hash)) where ’no-hash’ would return a string, say "0000000000000000000000000000000000000000000000000000" or whatever else that would satisfy this hypothetical ’sha256’ sanitizer. Cheers, simon ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: git-fetch without a hash 2023-01-11 15:34 ` Simon Tournier @ 2023-02-05 17:44 ` bokr 2023-02-06 17:01 ` Simon Tournier 0 siblings, 1 reply; 12+ messages in thread From: bokr @ 2023-02-05 17:44 UTC (permalink / raw) To: Simon Tournier; +Cc: Ludovic Courtès, Stephen Paul Weber, guix-devel, 0 Hi, On +2023-01-11 16:34:41 +0100, Simon Tournier wrote: > Hi, > > On Mon, 09 Jan 2023 at 12:16, Ludovic Courtès <ludo@gnu.org> wrote: > > Simon Tournier <zimon.toutoune@gmail.com> skribis: > > > >> Maybe my question is naive but what is the use case for this (sha256 #f) > >> in the first place? Because maybe it could just error using some > >> ’sanitize’ for the hash record field. > > > > There’s a couple of uses: Chromium, IceCat, and Linux-libre (IIRC). > > > > I don’t like that, but I’m not sure what it would take to change these > > to <computed-file> or something like that. > > Well, from (gnu packages linux) > > --8<---------------cut here---------------start------------->8--- > (origin > (method computed-origin-method) > (file-name (string-append "linux-libre-" version "-guix.tar.xz")) > (sha256 #f) > --8<---------------cut here---------------end--------------->8--- > > and from (gnu packages gnuzilla) > > --8<---------------cut here---------------start------------->8--- > (origin > (method computed-origin-method) > (file-name (string-append "icecat-" %icecat-version ".tar.xz")) > (sha256 #f) > --8<---------------cut here---------------end--------------->8--- > > but not from Chromium, if I read correctly. > > From my understanding, we could have something like, > > (sha256 (no-hash)) > > where ’no-hash’ would return a string, say > "0000000000000000000000000000000000000000000000000000" or whatever else > that would satisfy this hypothetical ’sha256’ sanitizer. > > > Cheers, > simon > For portability to any hash algorithm that returns a hex string, how about letting them hash a zero-length string (which can never represent a package tarball or other archive), and using the resulting strings as no-hash flags? These strings must be unique for whatever hash algorithm, so a short table could be used to recognize them as no-hash indicators. --8<---------------cut here---------------start------------->8--- $ sha256sum /dev/null e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 /dev/null $ sha256sum <(echo -n '') e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 /dev/fd/63 $ echo -n ''|sha256sum e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 - $ echo -n ""|sha256sum e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 - --8<---------------cut here---------------end--------------->8--- For other hash values on your system, probably the below will show most. Translating from hex to various base32 and other-base alphabets is then trivial, (and open, i.e. permitting new hash representation strings). --8<---------------cut here---------------start------------->8--- $ ls -1d /usr/bin/*sum|while read hasher;do \ echo;echo "$hasher:"; "$hasher" /dev/null;done --8<---------------cut here---------------end--------------->8--- WDYT? -- Regards, Bengt Richter ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: git-fetch without a hash 2023-02-05 17:44 ` bokr @ 2023-02-06 17:01 ` Simon Tournier 0 siblings, 0 replies; 12+ messages in thread From: Simon Tournier @ 2023-02-06 17:01 UTC (permalink / raw) To: bokr; +Cc: Ludovic Courtès, Stephen Paul Weber, guix-devel, 0 Hi, On dim., 05 févr. 2023 at 18:44, bokr@bokr.com wrote: >> From my understanding, we could have something like, >> >> (sha256 (no-hash)) >> >> where ’no-hash’ would return a string, say >> "0000000000000000000000000000000000000000000000000000" or whatever else >> that would satisfy this hypothetical ’sha256’ sanitizer. > For portability to any hash algorithm that returns a hex string, > how about letting them hash a zero-length string (which can never > represent a package tarball or other archive), and using the > resulting strings as no-hash flags? Instead of “0000000000000000000000000000000000000000000000000000”, you are proposing “0mdqa9w1p6cmli6976v4wi0sw9r4p5prkj7lzfd1877wk11c9c73” which is the Guix hash of empty. > $ sha256sum /dev/null > e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 /dev/null $ touch foo $ guix hash foo 0mdqa9w1p6cmli6976v4wi0sw9r4p5prkj7lzfd1877wk11c9c73 $ guix hash foo -f hex -H sha256 e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 Yeah, probably better. :-) > These strings must be unique for whatever hash algorithm, > so a short table could be used to recognize them as > no-hash indicators. Well, I do not understand “These strings must be unique for whatever hash algorithm”. What do you mean? Cheers, simon ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: git-fetch without a hash 2023-01-03 21:34 ` Ludovic Courtès 2023-01-05 10:06 ` Simon Tournier @ 2023-01-09 17:13 ` Ludovic Courtès 2023-01-09 17:35 ` Stephen Paul Weber 1 sibling, 1 reply; 12+ messages in thread From: Ludovic Courtès @ 2023-01-09 17:13 UTC (permalink / raw) To: Stephen Paul Weber; +Cc: guix-devel, 0 Hi, Ludovic Courtès <ludo@gnu.org> skribis: > scheme@(guile-user)> ,lower (origin > (method url-fetch) > (uri "mirror://gnu/hello/hello-2.12.1.tar.gz") > (sha256 #f)) > $6 = #<derivation /gnu/store/lz34lhyxhq5wxj87fnd465hmwbhv17bn-hello-2.12.1.tar.gz.drv => /gnu/store/xfc7gsn10j09bi89ldbpfbppfkcldfy9-hello-2.12.1.tar.gz 7f3ff487c000> > scheme@(guile-user)> (derivation-outputs $6) > $7 = (("out" . #<<derivation-output> path: "/gnu/store/xfc7gsn10j09bi89ldbpfbppfkcldfy9-hello-2.12.1.tar.gz" hash-algo: sha256 hash: #f recursive?: #f>)) > scheme@(guile-user)> (fixed-output-derivation? $6) > $8 = #f > > As it turns out, guix-daemon turns off the chroot for all built-in > builders, fixed-output or not; quoth ‘build.cc’: > > /* Note: built-in builders are *not* running in a chroot environment so > that we can easily implement them in Guile without having it as a > derivation input (they are running under a separate build user, > though). */ > useChroot = settings.useChroot && !isBuiltin(drv); > > I was actually surprised to see this, this wasn’t really intended. The > good news is that this is safe AFAICS: there’s only one built-in builder > to date, and that’s “download”. After investigating, I found that ‘guix perform-download’ already has a check in place: (unless (and algo hash) (leave (G_ "~a is not a fixed-output derivation~%") (derivation-file-name drv))) The check wasn’t working because ‘hash’ was #vu8() instead of #f. This is fixed in 5d24e57a611b43ff68700379338b899f62d198cc. Once we update the ‘guix’ package, the daemon will no longer accept ‘url-fetch’ downloads with hash = #f. Thanks again for reporting this oddity! Ludo’. ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: git-fetch without a hash 2023-01-09 17:13 ` Ludovic Courtès @ 2023-01-09 17:35 ` Stephen Paul Weber 2023-01-17 16:01 ` Ludovic Courtès 0 siblings, 1 reply; 12+ messages in thread From: Stephen Paul Weber @ 2023-01-09 17:35 UTC (permalink / raw) To: Ludovic Courtès; +Cc: guix-devel, 0 >Once we update the ‘guix’ package, the daemon will no longer accept >‘url-fetch’ downloads with hash = #f. Ok. Is there something like (git-checkout) we can use that will work instead of an (origin) here for url downloads? ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: git-fetch without a hash 2023-01-09 17:35 ` Stephen Paul Weber @ 2023-01-17 16:01 ` Ludovic Courtès 0 siblings, 0 replies; 12+ messages in thread From: Ludovic Courtès @ 2023-01-17 16:01 UTC (permalink / raw) To: Stephen Paul Weber; +Cc: guix-devel, 0 Hi, Stephen Paul Weber <singpolyma@singpolyma.net> skribis: >>Once we update the ‘guix’ package, the daemon will no longer accept >>‘url-fetch’ downloads with hash = #f. > > Ok. Is there something like (git-checkout) we can use that will work > instead of an (origin) here for url downloads? There’s <download-file> in (guix transformations), but it’s currently private. We could move it to (guix download) and document it. Thanks, Ludo’. ^ permalink raw reply [flat|nested] 12+ messages in thread
end of thread, other threads:[~2023-02-06 18:21 UTC | newest] Thread overview: 12+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2022-12-21 3:04 git-fetch without a hash Stephen Paul Weber 2022-12-21 22:49 ` Ludovic Courtès 2023-01-03 19:31 ` Stephen Paul Weber 2023-01-03 21:34 ` Ludovic Courtès 2023-01-05 10:06 ` Simon Tournier 2023-01-09 11:16 ` Ludovic Courtès 2023-01-11 15:34 ` Simon Tournier 2023-02-05 17:44 ` bokr 2023-02-06 17:01 ` Simon Tournier 2023-01-09 17:13 ` Ludovic Courtès 2023-01-09 17:35 ` Stephen Paul Weber 2023-01-17 16:01 ` Ludovic Courtès
Code repositories for project(s) associated with this public inbox https://git.savannah.gnu.org/cgit/guix.git This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).