Re: Non deterministic package

[Efraim Flashner <efraim@flashner.co.il>]

[-- Attachment #1: Type: text/plain, Size: 6673 bytes --]

I've become involved in the rust packaging effort so I'll jump in with
some specifics.

On Wed, Aug 04, 2021 at 03:50:00PM -0400, Julien Lepiller wrote:
> 
> 
> Le 4 août 2021 15:14:16 GMT-04:00, phodina via <help-guix@gnu.org> a écrit :
> >Hi,
> >
> >I came across a package in the Guix repository which fails to build. The package in question here is rust-brotli-sys-0.3. So the obvious question comes up. How does this happen?
> >
> >I just ran =guix build rust-brotli-sys= with the following output:
> >
> >...
> >phase `reset-gzip-timestamps' succeeded after 0.0 seconds
> >starting phase `compress-documentation'
> >phase `compress-documentation' succeeded after 0.0 seconds
> >guix build: error: derivation `/gnu/store/kf8kwl7yn8jqpwc9p97bw8vs5g5bvas4-rust-brotli-sys-0.3.2.drv' may not be deterministic: output `/gnu/store/yrrdr6rrc1s4b9lf7wz40c2gnv5kvxff-rust-brotli-sys-0.3.2' differs
> >Also this is my guix channel:
> >
> >guix:
> >repository URL: https://git.savannah.gnu.org/git/guix.git
> >branch: master
> >commit: b95fb85f593d377864fc1bfb3c118e54d03783be
> >
> >I tried to run the build with the flag --check and multiple rounds, but got the same result.
> >So could it be caused by not running =./pre-inst-env guix build --rounds=2 <PKG>= when submitting the patch?
> 
> You need to use. /pre-inst-env to test your patches, otherwise you'll only run your current guix. For an added package, it will be missing, for an updated package it will not build the newer version, etc.
> 
> >
> >Or when somebody updates some pkg that this pkg depends on and does not run =./pre-inst-env guix --refresh --list-dependent= and then check the dependent pkgs?
> 
> It can also happen when only the current day is recorded in the output: if you use —rounds=2, it is likely run on the same day, so the output will be the same (don't bother though, last time I found such an issue it was in january because only the year was recorded ^^")
> 
> You should now use -K to keep the different version and run diffoscope to analyze the differences. Then we should be able to fix the non determinism.
> 
> >
> >Could also sombody please share what kind of CI pipeline is there implemented on applying the patch on the mainter's side?
> 
> The maentainer is responsible for testing the patch (at a minimum that it builds), and sometimes we forget to check it's reproducible. The only automated thing is to check the patch is signed.
> 
> >
> >Thanks
> >Petr
> >
> >PS: Yes, I do have more questions regading how Guix works lately. I tried to search the mailing list but haven't found answers to them
> 

I'm not always great with diffoscope, but it turns out this one isn't
too bad to read.

(ins)efraim@3900XT ~ [env]$ diffoscope /gnu/store/yrrdr6rrc1s4b9lf7wz40c2gnv5kvxff-rust-brotli-sys-0.3.2 /gnu/store/yrrdr6rrc1s4b9lf7wz40c2gnv5kvxff-rust-brotli-sys-0.3.2-check
--- /gnu/store/yrrdr6rrc1s4b9lf7wz40c2gnv5kvxff-rust-brotli-sys-0.3.2
+++ /gnu/store/yrrdr6rrc1s4b9lf7wz40c2gnv5kvxff-rust-brotli-sys-0.3.2-check
│   --- /gnu/store/yrrdr6rrc1s4b9lf7wz40c2gnv5kvxff-rust-brotli-sys-0.3.2/share
├── +++ /gnu/store/yrrdr6rrc1s4b9lf7wz40c2gnv5kvxff-rust-brotli-sys-0.3.2-check/share
│ │   --- /gnu/store/yrrdr6rrc1s4b9lf7wz40c2gnv5kvxff-rust-brotli-sys-0.3.2/share/cargo
│ ├── +++ /gnu/store/yrrdr6rrc1s4b9lf7wz40c2gnv5kvxff-rust-brotli-sys-0.3.2-check/share/cargo
│ │ │   --- /gnu/store/yrrdr6rrc1s4b9lf7wz40c2gnv5kvxff-rust-brotli-sys-0.3.2/share/cargo/registry
│ │ ├── +++ /gnu/store/yrrdr6rrc1s4b9lf7wz40c2gnv5kvxff-rust-brotli-sys-0.3.2-check/share/cargo/registry
│ │ │ │   --- /gnu/store/yrrdr6rrc1s4b9lf7wz40c2gnv5kvxff-rust-brotli-sys-0.3.2/share/cargo/registry/brotli-sys-0.3.2.crate
│ │ │ ├── +++ /gnu/store/yrrdr6rrc1s4b9lf7wz40c2gnv5kvxff-rust-brotli-sys-0.3.2-check/share/cargo/registry/brotli-sys-0.3.2.crate
│ │ │ │ ├── brotli-sys-0.3.2.crate-content
│ │ │ │ │ ├── file list
│ │ │ │ │ │ @@ -1,8 +1,8 @@
│ │ │ │ │ │ --rw-r--r--   0        0        0     1123 2021-08-05 07:38:54.000000 brotli-sys-0.3.2/Cargo.toml
│ │ │ │ │ │ +-rw-r--r--   0        0        0     1123 2021-08-05 07:38:41.000000 brotli-sys-0.3.2/Cargo.toml
│ │ │ │ │ │  -rw-r--r--   0      999    30000     1122 1970-01-01 00:00:00.000000 brotli-sys-0.3.2/Cargo.toml.orig
│ │ │ │ │ │  -rw-r--r--   0      999    30000     2168 2017-06-30 05:59:10.000000 brotli-sys-0.3.2/brotli/common/constants.h
│ │ │ │ │ │  -rw-r--r--   0      999    30000   471278 2017-09-19 18:16:36.000000 brotli-sys-0.3.2/brotli/common/dictionary.c
│ │ │ │ │ │  -rw-r--r--   0      999    30000     1499 2017-09-19 18:16:36.000000 brotli-sys-0.3.2/brotli/common/dictionary.h
│ │ │ │ │ │  -rwxr-xr-x   0      999    30000      595 2017-09-19 18:16:36.000000 brotli-sys-0.3.2/brotli/common/version.h
│ │ │ │ │ │  -rw-r--r--   0      999    30000     1259 2017-06-30 05:59:10.000000 brotli-sys-0.3.2/brotli/dec/bit_reader.c
│ │ │ │ │ │  -rw-r--r--   0      999    30000    12006 2017-06-30 05:59:10.000000 brotli-sys-0.3.2/brotli/dec/bit_reader.h
│ │ │ │ ├── stat {}
│ │ │ │ │ @@ -1,8 +1,8 @@
│ │ │ │ │
│ │ │ │ │ -  Size: 406369       Blocks: 800        IO Block: 4096   regular file
│ │ │ │ │ +  Size: 406370       Blocks: 800        IO Block: 4096   regular file
│ │ │ │ │  Links: 1
│ │ │ │ │  Access: (0444/-r--r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
│ │ │ │ │
│ │ │ │ │  Modify: 1970-01-01 00:00:01.000000000 +0000

Here the difference is the timestamp on Cargo.toml. Cargo.toml gets
regenerated during the build process (likely when we run `cargo build`)
and then during the install phase we create a tarball of the source and
put that and an unpacked version in %out/share/cargo as a step toward
switching from #:cargo-inputs to just 'inputs'.

Ideally we'd modify the install phase or add a new phase to reset some
of the timestamps. Unfortunately, modifying the cargo-build-system would
result in the rebuild of ~2500 packages, or ~150, depending on how you
count.

I can see about queueing up something for core-updates-frozen, which
will make its way back to the master branch soonish™.

-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]