* bug#50597: different fixed-output when using ’--with-source’ transformation
@ 2021-09-15 5:44 zimoun
2021-09-18 16:06 ` Ludovic Courtès
0 siblings, 1 reply; 6+ messages in thread
From: zimoun @ 2021-09-15 5:44 UTC (permalink / raw)
To: 50597
Hi,
If the package source tarball is provided to ’--with-source’, then the
hash differs. I would expect a fixed point.
Let investigate:
--8<---------------cut here---------------start------------->8---
$ guix build hello@2.10 --with-source=hello=https://ftpmirror.gnu.org/gnu/hello/hello-2.10.tar.gz
/gnu/store/zfwhbbknkhxi3yqmp0qgh1l1crljgbm6-hello-2.10
$ guix build hello@2.10
/gnu/store/a462kby1q51ndvxdv3b6p0rsixxrgx1h-hello-2.10
--8<---------------cut here---------------end--------------->8---
Where https://ftpmirror.gnu.org/gnu/hello/hello-2.10.tar.gz is an
official mirror. I expect the same store item.
Why are they different? It is because the derivations are different.
The first call contains the source tarball as an “input”; that’s fine.
The second call contains the source tarball as a derivation; again
that’s fine. The corresponding first and second derivations are:
--8<---------------cut here---------------start------------->8---
Derive
([("out","/gnu/store/zfwhbbknkhxi3yqmp0qgh1l1crljgbm6-hello-2.10","","")]
,[("/gnu/store/101ny738l311p4fm3cas7jgnkzagjv6a-module-import-compiled.drv",["out"])
[...]
,("/gnu/store/z4s27gwacbw8f38andfsh21r8v330dag-xz-5.2.4.drv",["out"])]
,["/gnu/store/8a0wry8cvr405ha8d8bpjyzj5dzghigd-module-import",
"/gnu/store/chariqd6k0sli3s7vcl4q3al0crirz5v-hello-2.10.tar.gz",
"/gnu/store/rndq9g8877l29ha41dvsl3aj1z0gw0ng-hello-2.10-guile-builder"]
[...]
--8<---------------cut here---------------end--------------->8---
and
--8<---------------cut here---------------start------------->8---
Derive
([("out","/gnu/store/a462kby1q51ndvxdv3b6p0rsixxrgx1h-hello-2.10","","")]
,[("/gnu/store/101ny738l311p4fm3cas7jgnkzagjv6a-module-import-compiled.drv",["out"])
,("/gnu/store/1a7xfcqcxj0pqi4f81x1agcxa46v2bbm-hello-2.10.tar.gz.drv",["out"])
[...]
,("/gnu/store/z4s27gwacbw8f38andfsh21r8v330dag-xz-5.2.4.drv",["out"])]
,["/gnu/store/8a0wry8cvr405ha8d8bpjyzj5dzghigd-module-import",
"/gnu/store/kql8b2hbsabcmany4m3hfm3wzdiymliy-hello-2.10-guile-builder"]
[...]
--8<---------------cut here---------------end--------------->8---
Therefore, I understand why the two store items have different hashes.
Although, it should be better that the derivations would be the same;
another story. :-)
What puzzled me are the corresponding ’guile-builder’s:
--8<---------------cut here---------------start------------->8---
(define %build-inputs
(quote
(("source" . "/gnu/store/chariqd6k0sli3s7vcl4q3al0crirz5v-hello-2.10.tar.gz")
--8<---------------cut here---------------end--------------->8---
and
--8<---------------cut here---------------start------------->8---
(define %build-inputs
(quote
(("source" . "/gnu/store/hbdalsf5lpf01x4dcknwx6xbn6n5km6k-hello-2.10.tar.gz")
--8<---------------cut here---------------end--------------->8---
but these 2 items have the same hash:
--8<---------------cut here---------------start------------->8---
$ guix hash /gnu/store/chariqd6k0sli3s7vcl4q3al0crirz5v-hello-2.10.tar.gz
0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i
$ guix hash /gnu/store/hbdalsf5lpf01x4dcknwx6xbn6n5km6k-hello-2.10.tar.gz
0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i
--8<---------------cut here---------------end--------------->8---
Why? It is fixed-output so I was expecting the same thing. And the
same as:
--8<---------------cut here---------------start------------->8---
$ guix download https://ftpmirror.gnu.org/gnu/hello/hello-2.10.tar.gz
Starting download of /tmp/guix-file.qkhxkp
From https://ftpmirror.gnu.org/gnu/hello/hello-2.10.tar.gz...
following redirection to `https://mirrors.sarata.com/gnu/hello/hello-2.10.tar.gz'...
…10.tar.gz 709KiB 549KiB/s 00:01 [##################]
100.0%
/gnu/store/hbdalsf5lpf01x4dcknwx6xbn6n5km6k-hello-2.10.tar.gz
0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i
--8<---------------cut here---------------end--------------->8---
Why the store item of source tarball is it different when applying the
transformation?
Maxime provided a clue [1].
All the best,
simon
1: <https://yhetil.org/guix/bd48bb84243bb6b6a3a7f6e1350767d7c1015238.camel@telenet.be>
^ permalink raw reply [flat|nested] 6+ messages in thread
* bug#50597: different fixed-output when using ’--with-source’ transformation
2021-09-15 5:44 bug#50597: different fixed-output when using ’--with-source’ transformation zimoun
@ 2021-09-18 16:06 ` Ludovic Courtès
2021-09-18 17:35 ` Maxime Devos
2021-09-21 7:40 ` zimoun
0 siblings, 2 replies; 6+ messages in thread
From: Ludovic Courtès @ 2021-09-18 16:06 UTC (permalink / raw)
To: zimoun; +Cc: 50597
Hello!
zimoun <zimon.toutoune@gmail.com> skribis:
> Why the store item of source tarball is it different when applying the
> transformation?
>
> Maxime provided a clue [1].
Maxime is right: ‘--with-source’ passes #:recursive? #t unconditionally,
whereas it’s usually #f for a ‘url-fetch’ kind of URI.
What we could do is make it #t if and only if the URL denotes a local
file and that file is a directory. I think that would achieve what you
want without any observable regression in practice.
WDYT?
Thanks,
Ludo’.
^ permalink raw reply [flat|nested] 6+ messages in thread
* bug#50597: different fixed-output when using ’--with-source’ transformation
2021-09-18 16:06 ` Ludovic Courtès
@ 2021-09-18 17:35 ` Maxime Devos
2021-09-21 7:40 ` zimoun
1 sibling, 0 replies; 6+ messages in thread
From: Maxime Devos @ 2021-09-18 17:35 UTC (permalink / raw)
To: Ludovic Courtès, zimoun; +Cc: 50597
[-- Attachment #1: Type: text/plain, Size: 683 bytes --]
Ludovic Courtès schreef op za 18-09-2021 om 18:06 [+0200]:
> Hello!
>
> zimoun <zimon.toutoune@gmail.com> skribis:
>
> > Why the store item of source tarball is it different when applying the
> > transformation?
> >
> > Maxime provided a clue [1].
>
> Maxime is right: ‘--with-source’ passes #:recursive? #t unconditionally,
> whereas it’s usually #f for a ‘url-fetch’ kind of URI.
>
> What we could do is make it #t if and only if the URL denotes a local
> file and that file is a directory. I think that would achieve what you
> want without any observable regression in practice.
>
> WDYT?
This seems reasonable to me.
Greetings,
Maxime.
[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]
^ permalink raw reply [flat|nested] 6+ messages in thread
* bug#50597: different fixed-output when using ’--with-source’ transformation
2021-09-18 16:06 ` Ludovic Courtès
2021-09-18 17:35 ` Maxime Devos
@ 2021-09-21 7:40 ` zimoun
2021-09-22 10:25 ` Ludovic Courtès
1 sibling, 1 reply; 6+ messages in thread
From: zimoun @ 2021-09-21 7:40 UTC (permalink / raw)
To: Ludovic Courtès; +Cc: 50597
Hi Ludo,
On Sat, 18 Sep 2021 at 18:06, Ludovic Courtès <ludo@gnu.org> wrote:
> zimoun <zimon.toutoune@gmail.com> skribis:
>
>> Why the store item of source tarball is it different when applying the
>> transformation?
>>
>> Maxime provided a clue [1].
>
> Maxime is right: ‘--with-source’ passes #:recursive? #t unconditionally,
> whereas it’s usually #f for a ‘url-fetch’ kind of URI.
Thanks for confirming. Why is it different?
> What we could do is make it #t if and only if the URL denotes a local
> file and that file is a directory. I think that would achieve what you
> want without any observable regression in practice.
What I expect is that 2 (fixed-output) files with the same hash
(0ssi1w…) lives at the same store location…
--8<---------------cut here---------------start------------->8---
$ guix hash /gnu/store/chariqd6k0sli3s7vcl4q3al0crirz5v-hello-2.10.tar.gz
0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i
$ guix hash /gnu/store/hbdalsf5lpf01x4dcknwx6xbn6n5km6k-hello-2.10.tar.gz
0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i
--8<---------------cut here---------------end--------------->8---
…whatever if these (fixed-output) files come from “guix download”, “guix
build -S” or “--with-source=https://” or etc.
The mapping from hash (0ssi1w…) to store location should be 1:1 for
fixed-output, IMHO.
Cheers,
simon
^ permalink raw reply [flat|nested] 6+ messages in thread
* bug#50597: different fixed-output when using ’--with-source’ transformation
2021-09-21 7:40 ` zimoun
@ 2021-09-22 10:25 ` Ludovic Courtès
2021-10-13 9:31 ` zimoun
0 siblings, 1 reply; 6+ messages in thread
From: Ludovic Courtès @ 2021-09-22 10:25 UTC (permalink / raw)
To: zimoun; +Cc: 50597
[-- Attachment #1: Type: text/plain, Size: 1690 bytes --]
Hi,
zimoun <zimon.toutoune@gmail.com> skribis:
> On Sat, 18 Sep 2021 at 18:06, Ludovic Courtès <ludo@gnu.org> wrote:
>> zimoun <zimon.toutoune@gmail.com> skribis:
>>
>>> Why the store item of source tarball is it different when applying the
>>> transformation?
>>>
>>> Maxime provided a clue [1].
>>
>> Maxime is right: ‘--with-source’ passes #:recursive? #t unconditionally,
>> whereas it’s usually #f for a ‘url-fetch’ kind of URI.
>
> Thanks for confirming. Why is it different?
Because we want ‘--with-source’ to also work with directories (see
comment in ‘package-with-source’).
>> What we could do is make it #t if and only if the URL denotes a local
>> file and that file is a directory. I think that would achieve what you
>> want without any observable regression in practice.
>
> What I expect is that 2 (fixed-output) files with the same hash
> (0ssi1w…) lives at the same store location…
>
> $ guix hash /gnu/store/chariqd6k0sli3s7vcl4q3al0crirz5v-hello-2.10.tar.gz
> 0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i
>
> $ guix hash /gnu/store/hbdalsf5lpf01x4dcknwx6xbn6n5km6k-hello-2.10.tar.gz
> 0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i
The difference here is that one is marked as “recursive” and the other
is not.
The patch below does one I suggested earlier: passing #:recursive? #t if
and only if we know the source is a directory.
However it’s not enough to get the same derivation: we’d also need
<downloaded-file> to return a fixed-output derivation rather than a
plain store file name. (See also <https://issues.guix.gnu.org/49696>.)
To be continued…
Ludo’.
[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Type: text/x-patch, Size: 1725 bytes --]
diff --git a/guix/transformations.scm b/guix/transformations.scm
index 5ae1977cb2..e0f5579c28 100644
--- a/guix/transformations.scm
+++ b/guix/transformations.scm
@@ -46,6 +46,7 @@
#:use-module (srfi srfi-37)
#:use-module (ice-9 match)
#:use-module (ice-9 vlist)
+ #:autoload (web uri) (string->uri uri-scheme uri-path)
#:export (options->transformation
manifest-entry-with-transformations
@@ -106,15 +107,25 @@ extensions."
(define* (package-with-source p uri #:optional version)
"Return a package based on P but with its source taken from URI. Extract
the new package's version number from URI."
- (let ((base (tarball-base-name (basename uri))))
+ (let ((base (tarball-base-name (basename uri)))
+ (file (match (string->uri uri)
+ (#f uri)
+ (uri (and (eq? 'file (uri-scheme uri))
+ (uri-path uri))))))
(let-values (((_ version*)
(hyphen-package-name->name+version base)))
(package (inherit p)
(version (or version version*
(package-version p)))
- ;; Use #:recursive? #t to allow for directories.
- (source (downloaded-file uri #t))))))
+ ;; Default to #:recursive? #f to match what 'url-fetch' does,
+ ;; but use #t when URI denotes a directory.
+ (source (let ((recursive?
+ (and file
+ (match (stat file #f)
+ (#f #t)
+ (st (eq? 'directory (stat:type st)))))))
+ (downloaded-file uri recursive?)))))))
\f
;;;
^ permalink raw reply related [flat|nested] 6+ messages in thread
* bug#50597: different fixed-output when using ’--with-source’ transformation
2021-09-22 10:25 ` Ludovic Courtès
@ 2021-10-13 9:31 ` zimoun
0 siblings, 0 replies; 6+ messages in thread
From: zimoun @ 2021-10-13 9:31 UTC (permalink / raw)
To: Ludovic Courtès; +Cc: 50597
Hi,
On Wed, 22 Sep 2021 at 12:25, Ludovic Courtès <ludo@gnu.org> wrote:
> However it’s not enough to get the same derivation: we’d also need
> <downloaded-file> to return a fixed-output derivation rather than a
> plain store file name. (See also
> <https://issues.guix.gnu.org/49696>.)
Thanks for the explanations and the pointer. Indeed, to be continued…
> diff --git a/guix/transformations.scm b/guix/transformations.scm
> index 5ae1977cb2..e0f5579c28 100644
> --- a/guix/transformations.scm
> +++ b/guix/transformations.scm
> @@ -46,6 +46,7 @@
> #:use-module (srfi srfi-37)
> #:use-module (ice-9 match)
> #:use-module (ice-9 vlist)
> + #:autoload (web uri) (string->uri uri-scheme uri-path)
> #:export (options->transformation
> manifest-entry-with-transformations
>
> @@ -106,15 +107,25 @@ extensions."
> (define* (package-with-source p uri #:optional version)
> "Return a package based on P but with its source taken from URI. Extract
> the new package's version number from URI."
> - (let ((base (tarball-base-name (basename uri))))
> + (let ((base (tarball-base-name (basename uri)))
> + (file (match (string->uri uri)
> + (#f uri)
> + (uri (and (eq? 'file (uri-scheme uri))
> + (uri-path uri))))))
> (let-values (((_ version*)
> (hyphen-package-name->name+version base)))
> (package (inherit p)
> (version (or version version*
> (package-version p)))
>
> - ;; Use #:recursive? #t to allow for directories.
> - (source (downloaded-file uri #t))))))
> + ;; Default to #:recursive? #f to match what 'url-fetch' does,
> + ;; but use #t when URI denotes a directory.
> + (source (let ((recursive?
> + (and file
> + (match (stat file #f)
> + (#f #t)
> + (st (eq? 'directory (stat:type st)))))))
> + (downloaded-file uri recursive?)))))))
LGTM!
Cheers,
simon
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2021-10-13 9:48 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-09-15 5:44 bug#50597: different fixed-output when using ’--with-source’ transformation zimoun
2021-09-18 16:06 ` Ludovic Courtès
2021-09-18 17:35 ` Maxime Devos
2021-09-21 7:40 ` zimoun
2021-09-22 10:25 ` Ludovic Courtès
2021-10-13 9:31 ` zimoun
Code repositories for project(s) associated with this public inbox
https://git.savannah.gnu.org/cgit/guix.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).