Non deterministic package

Non deterministic package

[phodina via]

Hi,

I came across a package in the Guix repository which fails to build. The package in question here is rust-brotli-sys-0.3. So the obvious question comes up. How does this happen?

I just ran =guix build rust-brotli-sys= with the following output:

...
phase `reset-gzip-timestamps' succeeded after 0.0 seconds
starting phase `compress-documentation'
phase `compress-documentation' succeeded after 0.0 seconds
guix build: error: derivation `/gnu/store/kf8kwl7yn8jqpwc9p97bw8vs5g5bvas4-rust-brotli-sys-0.3.2.drv' may not be deterministic: output `/gnu/store/yrrdr6rrc1s4b9lf7wz40c2gnv5kvxff-rust-brotli-sys-0.3.2' differs
Also this is my guix channel:

guix:
repository URL: https://git.savannah.gnu.org/git/guix.git
branch: master
commit: b95fb85f593d377864fc1bfb3c118e54d03783be

I tried to run the build with the flag --check and multiple rounds, but got the same result.
So could it be caused by not running =./pre-inst-env guix build --rounds=2 <PKG>= when submitting the patch?

Or when somebody updates some pkg that this pkg depends on and does not run =./pre-inst-env guix --refresh --list-dependent= and then check the dependent pkgs?

Could also sombody please share what kind of CI pipeline is there implemented on applying the patch on the mainter's side?

Thanks
Petr

PS: Yes, I do have more questions regading how Guix works lately. I tried to search the mailing list but haven't found answers to them

Re: Non deterministic package

[Julien Lepiller]

Le 4 août 2021 15:14:16 GMT-04:00, phodina via <help-guix@gnu.org> a écrit :
>Hi,
>
>I came across a package in the Guix repository which fails to build. The package in question here is rust-brotli-sys-0.3. So the obvious question comes up. How does this happen?
>
>I just ran =guix build rust-brotli-sys= with the following output:
>
>...
>phase `reset-gzip-timestamps' succeeded after 0.0 seconds
>starting phase `compress-documentation'
>phase `compress-documentation' succeeded after 0.0 seconds
>guix build: error: derivation `/gnu/store/kf8kwl7yn8jqpwc9p97bw8vs5g5bvas4-rust-brotli-sys-0.3.2.drv' may not be deterministic: output `/gnu/store/yrrdr6rrc1s4b9lf7wz40c2gnv5kvxff-rust-brotli-sys-0.3.2' differs
>Also this is my guix channel:
>
>guix:
>repository URL: https://git.savannah.gnu.org/git/guix.git
>branch: master
>commit: b95fb85f593d377864fc1bfb3c118e54d03783be
>
>I tried to run the build with the flag --check and multiple rounds, but got the same result.
>So could it be caused by not running =./pre-inst-env guix build --rounds=2 <PKG>= when submitting the patch?

You need to use. /pre-inst-env to test your patches, otherwise you'll only run your current guix. For an added package, it will be missing, for an updated package it will not build the newer version, etc.

>
>Or when somebody updates some pkg that this pkg depends on and does not run =./pre-inst-env guix --refresh --list-dependent= and then check the dependent pkgs?

It can also happen when only the current day is recorded in the output: if you use —rounds=2, it is likely run on the same day, so the output will be the same (don't bother though, last time I found such an issue it was in january because only the year was recorded ^^")

You should now use -K to keep the different version and run diffoscope to analyze the differences. Then we should be able to fix the non determinism.

>
>Could also sombody please share what kind of CI pipeline is there implemented on applying the patch on the mainter's side?

The maentainer is responsible for testing the patch (at a minimum that it builds), and sometimes we forget to check it's reproducible. The only automated thing is to check the patch is signed.

>
>Thanks
>Petr
>
>PS: Yes, I do have more questions regading how Guix works lately. I tried to search the mailing list but haven't found answers to them

Re: Non deterministic package

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 6673 bytes --]

I've become involved in the rust packaging effort so I'll jump in with
some specifics.

On Wed, Aug 04, 2021 at 03:50:00PM -0400, Julien Lepiller wrote:
> 
> 
> Le 4 août 2021 15:14:16 GMT-04:00, phodina via <help-guix@gnu.org> a écrit :
> >Hi,
> >
> >I came across a package in the Guix repository which fails to build. The package in question here is rust-brotli-sys-0.3. So the obvious question comes up. How does this happen?
> >
> >I just ran =guix build rust-brotli-sys= with the following output:
> >
> >...
> >phase `reset-gzip-timestamps' succeeded after 0.0 seconds
> >starting phase `compress-documentation'
> >phase `compress-documentation' succeeded after 0.0 seconds
> >guix build: error: derivation `/gnu/store/kf8kwl7yn8jqpwc9p97bw8vs5g5bvas4-rust-brotli-sys-0.3.2.drv' may not be deterministic: output `/gnu/store/yrrdr6rrc1s4b9lf7wz40c2gnv5kvxff-rust-brotli-sys-0.3.2' differs
> >Also this is my guix channel:
> >
> >guix:
> >repository URL: https://git.savannah.gnu.org/git/guix.git
> >branch: master
> >commit: b95fb85f593d377864fc1bfb3c118e54d03783be
> >
> >I tried to run the build with the flag --check and multiple rounds, but got the same result.
> >So could it be caused by not running =./pre-inst-env guix build --rounds=2 <PKG>= when submitting the patch?
> 
> You need to use. /pre-inst-env to test your patches, otherwise you'll only run your current guix. For an added package, it will be missing, for an updated package it will not build the newer version, etc.
> 
> >
> >Or when somebody updates some pkg that this pkg depends on and does not run =./pre-inst-env guix --refresh --list-dependent= and then check the dependent pkgs?
> 
> It can also happen when only the current day is recorded in the output: if you use —rounds=2, it is likely run on the same day, so the output will be the same (don't bother though, last time I found such an issue it was in january because only the year was recorded ^^")
> 
> You should now use -K to keep the different version and run diffoscope to analyze the differences. Then we should be able to fix the non determinism.
> 
> >
> >Could also sombody please share what kind of CI pipeline is there implemented on applying the patch on the mainter's side?
> 
> The maentainer is responsible for testing the patch (at a minimum that it builds), and sometimes we forget to check it's reproducible. The only automated thing is to check the patch is signed.
> 
> >
> >Thanks
> >Petr
> >
> >PS: Yes, I do have more questions regading how Guix works lately. I tried to search the mailing list but haven't found answers to them
> 

I'm not always great with diffoscope, but it turns out this one isn't
too bad to read.

(ins)efraim@3900XT ~ [env]$ diffoscope /gnu/store/yrrdr6rrc1s4b9lf7wz40c2gnv5kvxff-rust-brotli-sys-0.3.2 /gnu/store/yrrdr6rrc1s4b9lf7wz40c2gnv5kvxff-rust-brotli-sys-0.3.2-check
--- /gnu/store/yrrdr6rrc1s4b9lf7wz40c2gnv5kvxff-rust-brotli-sys-0.3.2
+++ /gnu/store/yrrdr6rrc1s4b9lf7wz40c2gnv5kvxff-rust-brotli-sys-0.3.2-check
│   --- /gnu/store/yrrdr6rrc1s4b9lf7wz40c2gnv5kvxff-rust-brotli-sys-0.3.2/share
├── +++ /gnu/store/yrrdr6rrc1s4b9lf7wz40c2gnv5kvxff-rust-brotli-sys-0.3.2-check/share
│ │   --- /gnu/store/yrrdr6rrc1s4b9lf7wz40c2gnv5kvxff-rust-brotli-sys-0.3.2/share/cargo
│ ├── +++ /gnu/store/yrrdr6rrc1s4b9lf7wz40c2gnv5kvxff-rust-brotli-sys-0.3.2-check/share/cargo
│ │ │   --- /gnu/store/yrrdr6rrc1s4b9lf7wz40c2gnv5kvxff-rust-brotli-sys-0.3.2/share/cargo/registry
│ │ ├── +++ /gnu/store/yrrdr6rrc1s4b9lf7wz40c2gnv5kvxff-rust-brotli-sys-0.3.2-check/share/cargo/registry
│ │ │ │   --- /gnu/store/yrrdr6rrc1s4b9lf7wz40c2gnv5kvxff-rust-brotli-sys-0.3.2/share/cargo/registry/brotli-sys-0.3.2.crate
│ │ │ ├── +++ /gnu/store/yrrdr6rrc1s4b9lf7wz40c2gnv5kvxff-rust-brotli-sys-0.3.2-check/share/cargo/registry/brotli-sys-0.3.2.crate
│ │ │ │ ├── brotli-sys-0.3.2.crate-content
│ │ │ │ │ ├── file list
│ │ │ │ │ │ @@ -1,8 +1,8 @@
│ │ │ │ │ │ --rw-r--r--   0        0        0     1123 2021-08-05 07:38:54.000000 brotli-sys-0.3.2/Cargo.toml
│ │ │ │ │ │ +-rw-r--r--   0        0        0     1123 2021-08-05 07:38:41.000000 brotli-sys-0.3.2/Cargo.toml
│ │ │ │ │ │  -rw-r--r--   0      999    30000     1122 1970-01-01 00:00:00.000000 brotli-sys-0.3.2/Cargo.toml.orig
│ │ │ │ │ │  -rw-r--r--   0      999    30000     2168 2017-06-30 05:59:10.000000 brotli-sys-0.3.2/brotli/common/constants.h
│ │ │ │ │ │  -rw-r--r--   0      999    30000   471278 2017-09-19 18:16:36.000000 brotli-sys-0.3.2/brotli/common/dictionary.c
│ │ │ │ │ │  -rw-r--r--   0      999    30000     1499 2017-09-19 18:16:36.000000 brotli-sys-0.3.2/brotli/common/dictionary.h
│ │ │ │ │ │  -rwxr-xr-x   0      999    30000      595 2017-09-19 18:16:36.000000 brotli-sys-0.3.2/brotli/common/version.h
│ │ │ │ │ │  -rw-r--r--   0      999    30000     1259 2017-06-30 05:59:10.000000 brotli-sys-0.3.2/brotli/dec/bit_reader.c
│ │ │ │ │ │  -rw-r--r--   0      999    30000    12006 2017-06-30 05:59:10.000000 brotli-sys-0.3.2/brotli/dec/bit_reader.h
│ │ │ │ ├── stat {}
│ │ │ │ │ @@ -1,8 +1,8 @@
│ │ │ │ │
│ │ │ │ │ -  Size: 406369       Blocks: 800        IO Block: 4096   regular file
│ │ │ │ │ +  Size: 406370       Blocks: 800        IO Block: 4096   regular file
│ │ │ │ │  Links: 1
│ │ │ │ │  Access: (0444/-r--r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
│ │ │ │ │
│ │ │ │ │  Modify: 1970-01-01 00:00:01.000000000 +0000

Here the difference is the timestamp on Cargo.toml. Cargo.toml gets
regenerated during the build process (likely when we run `cargo build`)
and then during the install phase we create a tarball of the source and
put that and an unpacked version in %out/share/cargo as a step toward
switching from #:cargo-inputs to just 'inputs'.

Ideally we'd modify the install phase or add a new phase to reset some
of the timestamps. Unfortunately, modifying the cargo-build-system would
result in the rebuild of ~2500 packages, or ~150, depending on how you
count.

I can see about queueing up something for core-updates-frozen, which
will make its way back to the master branch soonish™.

-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Non deterministic package

[phodina]

> I've become involved in the rust packaging effort so I'll jump in with
>
> some specifics.
>
> On Wed, Aug 04, 2021 at 03:50:00PM -0400, Julien Lepiller wrote:
>
> > Le 4 août 2021 15:14:16 GMT-04:00, phodina via help-guix@gnu.org a écrit :
> >
> > > Hi,
> > >
> > > I came across a package in the Guix repository which fails to build. The package in question here is rust-brotli-sys-0.3. So the obvious question comes up. How does this happen?
> > >
> > > I just ran =guix build rust-brotli-sys= with the following output:
> > >
> > > ...
> > >
> > > phase `reset-gzip-timestamps' succeeded after 0.0 seconds starting phase` compress-documentation'
> > >
> > > phase `compress-documentation' succeeded after 0.0 seconds guix build: error: derivation` /gnu/store/kf8kwl7yn8jqpwc9p97bw8vs5g5bvas4-rust-brotli-sys-0.3.2.drv' may not be deterministic: output `/gnu/store/yrrdr6rrc1s4b9lf7wz40c2gnv5kvxff-rust-brotli-sys-0.3.2' differs
> > >
> > > Also this is my guix channel:
> > >
> > > guix:
> > >
> > > repository URL: https://git.savannah.gnu.org/git/guix.git
> > >
> > > branch: master
> > >
> > > commit: b95fb85f593d377864fc1bfb3c118e54d03783be
> > >
> > > I tried to run the build with the flag --check and multiple rounds, but got the same result.
> > >
> > > So could it be caused by not running =./pre-inst-env guix build --rounds=2 <PKG>= when submitting the patch?
> >
> > You need to use. /pre-inst-env to test your patches, otherwise you'll only run your current guix. For an added package, it will be missing, for an updated package it will not build the newer version, etc.
> >
> > > Or when somebody updates some pkg that this pkg depends on and does not run =./pre-inst-env guix --refresh --list-dependent= and then check the dependent pkgs?
> >
> > It can also happen when only the current day is recorded in the output: if you use —rounds=2, it is likely run on the same day, so the output will be the same (don't bother though, last time I found such an issue it was in january because only the year was recorded ^^")
> >
> > You should now use -K to keep the different version and run diffoscope to analyze the differences. Then we should be able to fix the non determinism.
> >
> > > Could also sombody please share what kind of CI pipeline is there implemented on applying the patch on the mainter's side?
> >
> > The maentainer is responsible for testing the patch (at a minimum that it builds), and sometimes we forget to check it's reproducible. The only automated thing is to check the patch is signed.
> >
> > > Thanks
> > >
> > > Petr
> > >
> > > PS: Yes, I do have more questions regading how Guix works lately. I tried to search the mailing list but haven't found answers to them
>
> I'm not always great with diffoscope, but it turns out this one isn't
>
> too bad to read.
>
> (ins)efraim@3900XT ~ [env]$ diffoscope /gnu/store/yrrdr6rrc1s4b9lf7wz40c2gnv5kvxff-rust-brotli-sys-0.3.2 /gnu/store/yrrdr6rrc1s4b9lf7wz40c2gnv5kvxff-rust-brotli-sys-0.3.2-check
>
> --- /gnu/store/yrrdr6rrc1s4b9lf7wz40c2gnv5kvxff-rust-brotli-sys-0.3.2
>
> +++ /gnu/store/yrrdr6rrc1s4b9lf7wz40c2gnv5kvxff-rust-brotli-sys-0.3.2-check
>
> │ --- /gnu/store/yrrdr6rrc1s4b9lf7wz40c2gnv5kvxff-rust-brotli-sys-0.3.2/share
>
> ├── +++ /gnu/store/yrrdr6rrc1s4b9lf7wz40c2gnv5kvxff-rust-brotli-sys-0.3.2-check/share
>
> │ │ --- /gnu/store/yrrdr6rrc1s4b9lf7wz40c2gnv5kvxff-rust-brotli-sys-0.3.2/share/cargo
>
> │ ├── +++ /gnu/store/yrrdr6rrc1s4b9lf7wz40c2gnv5kvxff-rust-brotli-sys-0.3.2-check/share/cargo
>
> │ │ │ --- /gnu/store/yrrdr6rrc1s4b9lf7wz40c2gnv5kvxff-rust-brotli-sys-0.3.2/share/cargo/registry
>
> │ │ ├── +++ /gnu/store/yrrdr6rrc1s4b9lf7wz40c2gnv5kvxff-rust-brotli-sys-0.3.2-check/share/cargo/registry
>
> │ │ │ │ --- /gnu/store/yrrdr6rrc1s4b9lf7wz40c2gnv5kvxff-rust-brotli-sys-0.3.2/share/cargo/registry/brotli-sys-0.3.2.crate
>
> │ │ │ ├── +++ /gnu/store/yrrdr6rrc1s4b9lf7wz40c2gnv5kvxff-rust-brotli-sys-0.3.2-check/share/cargo/registry/brotli-sys-0.3.2.crate
>
> │ │ │ │ ├── brotli-sys-0.3.2.crate-content
>
> │ │ │ │ │ ├── file list
>
> │ │ │ │ │ │ @@ -1,8 +1,8 @@
>
> │ │ │ │ │ │ --rw-r--r-- 0 0 0 1123 2021-08-05 07:38:54.000000 brotli-sys-0.3.2/Cargo.toml
>
> │ │ │ │ │ │ +-rw-r--r-- 0 0 0 1123 2021-08-05 07:38:41.000000 brotli-sys-0.3.2/Cargo.toml
>
> │ │ │ │ │ │ -rw-r--r-- 0 999 30000 1122 1970-01-01 00:00:00.000000 brotli-sys-0.3.2/Cargo.toml.orig
>
> │ │ │ │ │ │ -rw-r--r-- 0 999 30000 2168 2017-06-30 05:59:10.000000 brotli-sys-0.3.2/brotli/common/constants.h
>
> │ │ │ │ │ │ -rw-r--r-- 0 999 30000 471278 2017-09-19 18:16:36.000000 brotli-sys-0.3.2/brotli/common/dictionary.c
>
> │ │ │ │ │ │ -rw-r--r-- 0 999 30000 1499 2017-09-19 18:16:36.000000 brotli-sys-0.3.2/brotli/common/dictionary.h
>
> │ │ │ │ │ │ -rwxr-xr-x 0 999 30000 595 2017-09-19 18:16:36.000000 brotli-sys-0.3.2/brotli/common/version.h
>
> │ │ │ │ │ │ -rw-r--r-- 0 999 30000 1259 2017-06-30 05:59:10.000000 brotli-sys-0.3.2/brotli/dec/bit_reader.c
>
> │ │ │ │ │ │ -rw-r--r-- 0 999 30000 12006 2017-06-30 05:59:10.000000 brotli-sys-0.3.2/brotli/dec/bit_reader.h
>
> │ │ │ │ ├── stat {}
>
> │ │ │ │ │ @@ -1,8 +1,8 @@
>
> │ │ │ │ │
>
> │ │ │ │ │ - Size: 406369 Blocks: 800 IO Block: 4096 regular file
>
> │ │ │ │ │ + Size: 406370 Blocks: 800 IO Block: 4096 regular file
>
> │ │ │ │ │ Links: 1
>
> │ │ │ │ │ Access: (0444/-r--r--r--) Uid: ( 0/ root) Gid: ( 0/ root)
>
> │ │ │ │ │
>
> │ │ │ │ │ Modify: 1970-01-01 00:00:01.000000000 +0000
>
> Here the difference is the timestamp on Cargo.toml. Cargo.toml gets
>
> regenerated during the build process (likely when we run `cargo build`)
>
> and then during the install phase we create a tarball of the source and
>
> put that and an unpacked version in %out/share/cargo as a step toward
>
> switching from #:cargo-inputs to just 'inputs'.
>
> Ideally we'd modify the install phase or add a new phase to reset some
>
> of the timestamps. Unfortunately, modifying the cargo-build-system would
>
> result in the rebuild of ~2500 packages, or ~150, depending on how you
>
> count.
>
> I can see about queueing up something for core-updates-frozen, which
>
> will make its way back to the master branch soonish™.
>
> -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Efraim Flashner efraim@flashner.co.il אפרים פלשנר
>
> GPG key = A28B F40C 3E55 1372 662D 14F7 41AA E7DC CA3D 8351
>
> Confidentiality cannot be guaranteed on emails sent or received unencrypted

Thanks Julien and Efraim. I didn't know about diffoscope, I see it's described in the manual[1].

I agree that the package build might build if the second build runs the same day. I'll add this tool to my list of checks for submitting new patches.

I use ./pre-inst-env to build and lint the package in question. Not the guix binary in my system itself. So I was little bit surprised that this package didn't build as it was already part of the Guix monorepo.Also this was the first time I encountered such an issue.

[1]: https://guix.gnu.org/manual/en/html_node/Invoking-guix-challenge.html