bug#36777: Guix Inferiors: Curious incorrect derivation output bug

bug#36777: Guix Inferiors: Curious incorrect derivation output bug

[Carl Dong]

Hi all,

I've been working on improving the Guix build support on Bitcoin Core so that
it'll be ready to use for official releases.

One of the things that I wanted to do was to use a combination of channels and
inferiors to have reproducible builds across time. I have most of it set up, but
am running into some trouble. Here are the details:

I have a Guix channel set up at https://github.com/dongcarl/bitcoin-guix, where
I pin the Guix version to 6869b6635afd93872b8f0d9f2db0db4c0d765a86 in the
.guix-channel, and declare all my packages in packages.scm.

I have my manifest.scm here:
https://github.com/dongcarl/bitcoin/blob/2019-06-guix-channels-and-inferiors/contrib/guix/manifest.scm,
and it references the aforementioned Guix channel.

What I expect to happen is that now when I change my default profile's Guix
version by 'guix pull'ing, it won't affect the environment that's generated by
the manifest.scm. I've tested this with differing versions of Guix as my default
profile, and this seems to work with 7304d5623ab5cc35289cb1535cbd0d8a37691fac
and 7f1c69f5d32bee6b8b6b902a9ce445e04aa9d07d being my default profile. However,
I tried an older version b6dc08393e6a8313b88ce422fc3c1e4e9c0efc6f, and got the
following error:

```
guix environment: error: derivation `/gnu/store/r641vpqc9rfjhljf7rzfzwmkmpa642ls-info-dir.drv' has incorrect output `/gnu/store/q9hkdidycz3wq28xxgjq47bzx5s39k52-info-dir', should be `/gnu/store/z5hh2nl0h58b9f6hdxfwm00gjyxfcc3n-info-dir'
```

I remember my previous attempts at integrating channels and inferiors into our
workflow yielded a similar result, so I would like to know if this is a bug, and
how I might go about fixing this so that I can feel comfortable using inferiors
for our builds.

Nevertheless, thank you all for your hard work. The fact that we have inferiors
at all is marvelous!

Cheers,
Carl Dong
contact@carldong.me
"I fight for the users"

bug#36777: Guix Inferiors: Curious incorrect derivation output bug

[Ludovic Courtès]

Hello Carl,

Carl Dong <contact@carldong.me> skribis:

> I have my manifest.scm here:
> https://github.com/dongcarl/bitcoin/blob/2019-06-guix-channels-and-inferiors/contrib/guix/manifest.scm,
> and it references the aforementioned Guix channel.

This one appears to work for me (I interrupted it before it was done
compiling all the toolchains, though.)

> What I expect to happen is that now when I change my default profile's Guix
> version by 'guix pull'ing, it won't affect the environment that's generated by
> the manifest.scm. I've tested this with differing versions of Guix as my default
> profile, and this seems to work with 7304d5623ab5cc35289cb1535cbd0d8a37691fac
> and 7f1c69f5d32bee6b8b6b902a9ce445e04aa9d07d being my default profile. However,
> I tried an older version b6dc08393e6a8313b88ce422fc3c1e4e9c0efc6f, and got the
> following error:

When I put b6dc08393e6a8313b88ce422fc3c1e4e9c0efc6f in .guix-channel and
use that as my channel, it also works fine (well, I commented out the
toolchains as well.)

> ```
> guix environment: error: derivation `/gnu/store/r641vpqc9rfjhljf7rzfzwmkmpa642ls-info-dir.drv' has incorrect output `/gnu/store/q9hkdidycz3wq28xxgjq47bzx5s39k52-info-dir', should be `/gnu/store/z5hh2nl0h58b9f6hdxfwm00gjyxfcc3n-info-dir'
> ```

That definitely looks like a bug.

Could you send the faulty info-dir.drv file?

Thanks,
Ludo’.

bug#36777: Guix Inferiors: Curious incorrect derivation output bug

[Carl Dong]

Hi all,

I did some more digging, and have included a git-bisect log, the -info-dir.drv,
and -info-dir-builder here:

https://gist.github.com/dongcarl/0a305badf20c9b5cfae738147ca416af

Please let me know if I can provide more information.

Cheers,
Carl Dong
contact@carldong.me
"I fight for the users"

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Friday, July 26, 2019 7:50 AM, Ludovic Courtès <ludo@gnu.org> wrote:

> Hello Carl,
>
> Carl Dong contact@carldong.me skribis:
>
> > I have my manifest.scm here:
> > https://github.com/dongcarl/bitcoin/blob/2019-06-guix-channels-and-inferiors/contrib/guix/manifest.scm,
> > and it references the aforementioned Guix channel.
>
> This one appears to work for me (I interrupted it before it was done
> compiling all the toolchains, though.)
>
> > What I expect to happen is that now when I change my default profile's Guix
> > version by 'guix pull'ing, it won't affect the environment that's generated by
> > the manifest.scm. I've tested this with differing versions of Guix as my default
> > profile, and this seems to work with 7304d5623ab5cc35289cb1535cbd0d8a37691fac
> > and 7f1c69f5d32bee6b8b6b902a9ce445e04aa9d07d being my default profile. However,
> > I tried an older version b6dc08393e6a8313b88ce422fc3c1e4e9c0efc6f, and got the
> > following error:
>
> When I put b6dc08393e6a8313b88ce422fc3c1e4e9c0efc6f in .guix-channel and
> use that as my channel, it also works fine (well, I commented out the
> toolchains as well.)
>
> >     guix environment: error: derivation `/gnu/store/r641vpqc9rfjhljf7rzfzwmkmpa642ls-info-dir.drv' has incorrect output `/gnu/store/q9hkdidycz3wq28xxgjq47bzx5s39k52-info-dir', should be `/gnu/store/z5hh2nl0h58b9f6hdxfwm00gjyxfcc3n-info-dir'
> >
>
> That definitely looks like a bug.
>
> Could you send the faulty info-dir.drv file?
>
> Thanks,
> Ludo’.

bug#36777: Guix Inferiors: Curious incorrect derivation output bug

[Ludovic Courtès]

[-- Attachment #1: Type: text/plain, Size: 1236 bytes --]

Hi Carl,

Carl Dong <contact@carldong.me> skribis:

> I did some more digging, and have included a git-bisect log, the -info-dir.drv,
> and -info-dir-builder here:
>
> https://gist.github.com/dongcarl/0a305badf20c9b5cfae738147ca416af

For future reference, the bisect output is:

  # first bad commit: [5cf4b26d52bcea382d98fb4becce89be9ee37b55] derivations: <derivation-input> now aggregates a <derivation>.

and ‘info-dir.drv’ is attached below.

I looked at the offending commit.  I don’t doubt it has the potential to
introduce such a bug :-), but so far I haven’t seen anything fishy as I
inspected it with a fresh eye.

Could you come up with a reduced reproducer for this?  Ideally a small
manifest, the commit of the Guix you’re using (not just that of the
inferior), maybe a manifest that doesn’t even use inferiors, and so on.

Alternately, since the problem is that the output path of the derivation
is incorrectly computed, you could add ‘pk’ calls to print the value of
‘drv-masked’ in (guix derivations) or similar.  That’s not going to be
convenient, though.  :-/

Ludo’.

PS: I’ll be away from keyboard so I won’t be able to look into it until
    some time.


[-- Attachment #2: the derivation --]
[-- Type: text/plain, Size: 5558 bytes --]

Derive([("out","/gnu/store/q9hkdidycz3wq28xxgjq47bzx5s39k52-info-dir","","")],[("/gnu/store/1lyb9v2fnbflccrawnzjw2n87ycaicia-glibc-cross-riscv64-linux-gnu-2.27.drv",["out","static"]),("/gnu/store/3wxvpcvxr5qgs2i1wx0cqyspv3in1kws-linux-libre-headers-cross-riscv64-linux-gnu-4.19.59.drv",["out"]),("/gnu/store/4d4si2aiviy0ldihj7ymjkxrvx6c7rvq-i686-linux-gnu-toolchain-9.1.0.drv",["out"]),("/gnu/store/4imc4bigz499427hqd2yvz38y361vq5k-gcc-cross-arm-linux-gnueabihf-9.1.0.drv",["out"]),("/gnu/store/5i49zz5l4dzh59fi2b8fiq2llw0rkn8s-python-3.7.0.drv",["out"]),("/gnu/store/60x9nffmmdahdfmd0argjkfcvkzcm4c3-linux-libre-headers-cross-x86_64-linux-gnu-4.19.59.drv",["out"]),("/gnu/store/7s7lml5cgign328i6nk13fxz0r2vykng-binutils-cross-x86_64-linux-gnu-2.31.1.drv",["out"]),("/gnu/store/7zc3ms7wxjg9ypg7r4p23ni18f43m6p1-linux-libre-headers-cross-aarch64-linux-gnu-4.19.59.drv",["out"]),("/gnu/store/82r3931cfjm1864mc81wzzlfmb5rlgx1-util-linux-2.32.1.drv",["out"]),("/gnu/store/8wgjmm3cks3xaabhm3m7w25i9wwx50z2-sed-4.5.drv",["out"]),("/gnu/store/9dgy50xx4d9f2pnh4s0pdla7n7j61mjg-findutils-4.6.0.drv",["out"]),("/gnu/store/arvm0v20n93mra1s2la59by3kbm3cbsd-guile-2.2.4.drv",["out"]),("/gnu/store/c0ngi4rm0avjlp3nb8j4yvmdlf598vr2-gzip-1.9.drv",["out"]),("/gnu/store/c0pcf8c405my5g7nsc99gxpq66hzynn8-gcc-cross-riscv64-linux-gnu-9.1.0.drv",["out"]),("/gnu/store/c59fjany683kns33dhb44hfc598654xh-libtool-2.4.6.drv",["out"]),("/gnu/store/d6inhd29h4mp3zfah4dq6mh0hi1bwsd6-gcc-glibc-2.27-toolchain-9.1.0.drv",["out"]),("/gnu/store/dwjr2d9ds00bg8dvmsmljygz6k9y6b5h-tar-1.30.drv",["out"]),("/gnu/store/fgv06c04vkf52s8idc5jm6c10nsa2ph1-bash-4.4.23.drv",["out"]),("/gnu/store/gj3b2r65jfgrz83b6nni44028xciakfy-libfaketime-0.9.7.drv",["out"]),("/gnu/store/iddbw29p9fq2ccygwhyvfc5r7yq0qlyr-grep-3.1.drv",["out"]),("/gnu/store/iv231dbdxg300rk9lqmgkdm855hdxzgr-binutils-cross-aarch64-linux-gnu-2.31.1.drv",["out"]),("/gnu/store/jdzrl9rrdxvyg7y639igda091w1aja1b-module-import-compiled.drv",["out"]),("/gnu/store/k9y9vjpv62nvksm5pgbxfy2ddgkrmwmv-linux-libre-headers-cross-i686-linux-gnu-4.19.59.drv",["out"]),("/gnu/store/kx4f618mcrmkizp5gxyb00fwn1hp4cbw-gcc-cross-aarch64-linux-gnu-9.1.0.drv",["out"]),("/gnu/store/ljjw2dnwpdq73fg3rjh58yw27qalyssc-linux-libre-headers-cross-arm-linux-gnueabihf-4.19.59.drv",["out"]),("/gnu/store/lm5x5sz3s5h4drn0n2l0xrbnqr9xyqzd-gcc-cross-x86_64-linux-gnu-9.1.0.drv",["out"]),("/gnu/store/m3fqaak2rc24ic0aisbl1fdafy14fb45-gcc-cross-i686-linux-gnu-9.1.0.drv",["out"]),("/gnu/store/maiy0a31cqlsls1g139hikwabv7wwvbh-zlib-1.2.11.drv",["out"]),("/gnu/store/n05hlghlgw64ljc8hgbimbajnlapssiw-glibc-cross-aarch64-linux-gnu-2.27.drv",["out","static"]),("/gnu/store/n8a9mj131ddip42cmcci3s12a3y01zj3-texinfo-6.5.drv",["out"]),("/gnu/store/nas9cw7lk7737ha4icmdw5yhxx4j9453-which-2.21.drv",["out"]),("/gnu/store/ngj6zhpajm7693ys419yxmkdpcqwkzg9-tcsh-6.20.00.drv",["out"]),("/gnu/store/nz7vl4xn2s3xwx2f7vlvhl8sp9h4lkgx-glibc-utf8-locales-2.28.drv",["out"]),("/gnu/store/nzq7zmb44hn9wzm997pn0cb0aak7nmxq-binutils-cross-arm-linux-gnueabihf-2.31.1.drv",["out"]),("/gnu/store/ph6l1b0ixbvghvs41dpm06wqf4pw5677-glibc-cross-i686-linux-gnu-2.27.drv",["out","static"]),("/gnu/store/pjnibkx89y2nq3r9wb6dkscdb9rs915z-pkg-config-0.29.2.drv",["out"]),("/gnu/store/pwfrz20z33lhgkvy3y1s9gm2ijwcacrk-coreutils-8.30.drv",["out"]),("/gnu/store/q0l8y15xksvrm6fpn6miivk1dy88ka1a-riscv64-linux-gnu-toolchain-9.1.0.drv",["out"]),("/gnu/store/q1v24z3zfnym7fqbbh4z4jcybaaiwzlv-m4-1.4.18.drv",["out"]),("/gnu/store/q24ic25gr3azaxlabh9pj842kybs8h4m-file-5.33.drv",["out"]),("/gnu/store/qk22gsq1vrkzhrpmmlr2084f4vvlsyg4-make-4.2.1.drv",["out"]),("/gnu/store/qm501xmx0p6h2qyk2jfg6i35n2g2iblw-autoconf-2.69.drv",["out"]),("/gnu/store/qxs84gr9g0vinyq433ym226d54nr668a-diffutils-3.6.drv",["out"]),("/gnu/store/r3ih0hhv74bp4qkg6zfi6wd4bqzlq7df-perl-5.28.0.drv",["out"]),("/gnu/store/rmqarqss73akz9yaad4a02zdvp62ijx2-gzip-1.9.drv",["out"]),("/gnu/store/rwz69w3nnbxvzwnkrrhq1q5fqf1d6rm1-glibc-cross-x86_64-linux-gnu-2.27.drv",["out","static"]),("/gnu/store/s5ajxc1mc2j6njkjs2bj6ssfzb43bvan-arm-linux-gnueabihf-toolchain-9.1.0.drv",["out"]),("/gnu/store/s66bj62iwlh9fndcyv06z997zp6ga7fy-aarch64-linux-gnu-toolchain-9.1.0.drv",["out"]),("/gnu/store/vi1jc3vkpw13nigqzw7iiqqcq9c1xinz-glibc-cross-arm-linux-gnueabihf-2.27.drv",["out","static"]),("/gnu/store/vxh3p0y6v577gzfp016v1xz5xs5lg47m-x86_64-linux-gnu-toolchain-9.1.0.drv",["out"]),("/gnu/store/w1wcggyikprn54y2c2hf1wk9bkn9ryja-automake-1.16.1.drv",["out"]),("/gnu/store/wan77da4mrf8x2fwc0yqndpz6v0fwqja-gawk-4.2.1.drv",["out"]),("/gnu/store/wcnmh5krgmz8a9x59r2gzpd81ybn5b7v-xz-5.2.4.drv",["out"]),("/gnu/store/wgfz82qfkh9vgqlpa5icj3bzqclyg8rj-bzip2-1.0.6.drv",["out"]),("/gnu/store/x9kynl44rpqxd78vbv8j2z58j1vmar0r-binutils-cross-i686-linux-gnu-2.31.1.drv",["out"]),("/gnu/store/xk4qgfc77rd4k4c863fxsyxkxfi9c0ja-binutils-cross-riscv64-linux-gnu-2.31.1.drv",["out"]),("/gnu/store/yvm7rgcskmgsjx6wnxx2gjp71gz6ghjh-patch-2.7.6.drv",["out"])],["/gnu/store/ak3y80w6xjr9fq22bw8d577zkm9qq005-info-dir-builder","/gnu/store/p5q59xqmhwg60y2437fnny76nw7l6gbz-module-import"],"x86_64-linux","/gnu/store/r658y3cgpnf99nxjxqgjiaizx20ac4k0-guile-2.2.4/bin/guile",["--no-auto-compile","-L","/gnu/store/p5q59xqmhwg60y2437fnny76nw7l6gbz-module-import","-C","/gnu/store/syp239hi3xjgxdm8r4hvhhzh67i5ixgl-module-import-compiled","/gnu/store/ak3y80w6xjr9fq22bw8d577zkm9qq005-info-dir-builder"],[("allowSubstitutes","0"),("guix properties","((type . profile-hook) (hook . info-dir))"),("out","/gnu/store/q9hkdidycz3wq28xxgjq47bzx5s39k52-info-dir"),("preferLocalBuild","1")])

bug#36777: Guix Inferiors: Curious incorrect derivation output bug

[Ludovic Courtès]

[-- Attachment #1: Type: text/plain, Size: 516 bytes --]

Ludovic Courtès <ludo@gnu.org> skribis:

> For future reference, the bisect output is:
>
>   # first bad commit: [5cf4b26d52bcea382d98fb4becce89be9ee37b55] derivations: <derivation-input> now aggregates a <derivation>.

Does the patch below have any effect?

My understanding is that it shouldn’t have any effect (because inputs
have already been coalesced at that point), but that’s one place where
5cf4b26d52bcea382d98fb4becce89be9ee37b55 introduced a difference.

Thanks in advance,
Ludo’.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Type: text/x-patch, Size: 1595 bytes --]

diff --git a/guix/derivations.scm b/guix/derivations.scm
index 92d50503ce..eb94fea55e 100644
--- a/guix/derivations.scm
+++ b/guix/derivations.scm
@@ -239,12 +239,17 @@ the store."
   "Return a list of inputs, such that when INPUTS contains the same DRV twice,
 they are coalesced, with their sub-derivations merged.  This is needed because
 Nix itself keeps only one of them."
+  (define (derivation-file-name* obj)
+    (if (derivation? obj)
+        (derivation-file-name obj)
+        obj))
+
   (fold (lambda (input result)
           (match input
-            (($ <derivation-input> (= derivation-file-name path) sub-drvs)
+            (($ <derivation-input> (= derivation-file-name* path) sub-drvs)
              ;; XXX: quadratic
              (match (find (match-lambda
-                            (($ <derivation-input> (= derivation-file-name p)
+                            (($ <derivation-input> (= derivation-file-name* p)
                                                    s)
                              (string=? p path)))
                           result)
@@ -685,7 +690,7 @@ name of each input with that input's hash."
                              (make-derivation-input hash sub-drvs))))
                         inputs)))
        (make-derivation outputs
-                        (sort inputs
+                        (sort (coalesce-duplicate-inputs inputs)
                               (lambda (drv1 drv2)
                                 (string<? (derivation-input-derivation drv1)
                                           (derivation-input-derivation drv2))))

bug#36777: Guix Inferiors: Curious incorrect derivation output bug

[Carl Dong]

All,

Yes! The patch actually fixed the problem when applied on top of 5cf4b26d52bcea382d98fb4becce89be9ee37b55! I was also able to come up with a minimal reproducing manifest:

```
(use-modules (guix inferior) (guix channels)
             (srfi srfi-1))   ;for 'first'

(define channels
  (list (channel
         (name 'guix)
         (url "https://git.savannah.gnu.org/git/guix.git")
         (commit
          "6869b6635afd93872b8f0d9f2db0db4c0d765a86"))))

(define inferior
  ;; An inferior representing the above revision.
  (inferior-for-channels channels))

(packages->manifest
  (map (lambda (x)
         (first (lookup-inferior-packages inferior x)))
       '("gzip")))
```

It seems that the `gzip` package is what's causing this.

Not sure what the next steps are for this, but I'd very much like to understand where this went wrong. Perhaps we could write tests for this so it doesn't happen in the future for releases.

Cheers,
Carl Dong
contact@carldong.me
"I fight for the users"

bug#36777: Guix Inferiors: Curious incorrect derivation output bug

[Ludovic Courtès]

Hi Carl,

Carl Dong <contact@carldong.me> skribis:

> Yes! The patch actually fixed the problem when applied on top of 5cf4b26d52bcea382d98fb4becce89be9ee37b55!

[...]

> Not sure what the next steps are for this, but I'd very much like to understand where this went wrong. Perhaps we could write tests for this so it doesn't happen in the future for releases.

Yup, I understood when this could happen (if multiple inputs of a
derivation are “fixed-output” derivations leading to the same output),
wrote a test for that, and came up with a simpler fix in commit
268896444bed7b958add74b2e1e86ff802c5f5cb.

Let me know if anything is amiss!

Thanks for testing the patch,
Ludo’.