bug#41669: Cross-compiled powerpc64-linux bootstrap-tarballs not reproducible

[Chris Marusich <cmmarusich@gmail.com>]

[-- Attachment #1.1: Type: text/plain, Size: 487 bytes --]

Hi,

Chris Marusich <cmmarusich@gmail.com> writes:

> Going forward, I'm not sure how best to investigate the inputs to find
> out what's causing the differences.  I just know I really, really,
> really don't want to rebuild everything multiple times, since it takes
> hours/days.  If you have any creative ideas for how to speed up the
> investigation, I'm all ears.

I've analyzed the graph of derivations that produce differing output
across machines.  Here's an image of the graph:


[-- Attachment #1.2: Graph of Nonreproducible Derivations (PNG) --]
[-- Type: image/png, Size: 82643 bytes --]

[-- Attachment #1.3: Type: text/plain, Size: 74 bytes --]


And here is the GraphViz source, which you can render however you want:


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1.4: Graph of Nonreproducible Derivations (GraphViz) --]
[-- Type: text/vnd.graphviz, Size: 12491 bytes --]

digraph "Differing Derivations" {

"/gnu/store/07bd5ll0adnyrv1zaz11vz2x1ax447ka-glibc-mesboot-2.16.0.drv" [label = "glibc-mesboot-2.16.0", shape = box, fontname = sans]
"/gnu/store/0zkiqxwm6k637xr5s1690nwllnvybvyw-xz-mesboot-5.0.0.drv" [label = "xz-mesboot-5.0.0", shape = box, fontname = sans]
"/gnu/store/2pziz2j7781mhadl8lcfpzm8anvazb37-binutils-mesboot-2.20.1a.drv" [label = "binutils-mesboot-2.20.1a", shape = box, fontname = sans]
"/gnu/store/4ji6ayrgcyfyxpb583qr5ja4awdlxrdc-bootar-1a.drv" [label = "bootar-1a", shape = box, fontname = sans]
"/gnu/store/5x8a1yib7vdza727vrq4zmp6cmsafy7h-module-import-compiled.drv" [label = "module-import-compiled-1", shape = box, fontname = sans]
"/gnu/store/agday74gvxnd6a7191fw2g473b5v66kx-gcc-mesboot1-4.6.4.drv" [label = "gcc-mesboot1-4.6.4", shape = box, fontname = sans]
"/gnu/store/alvrmh47xqk7glq9wmvrzivfjp2bcvyc-module-import-compiled.drv" [label = "module-import-compiled-2", shape = box, fontname = sans]
"/gnu/store/asnd815v865cvfh2l2dlxmh5y556v3i5-gcc-core-mesboot0-2.95.3.drv" [label = "gcc-core-mesboot0-2.95.3", shape = box, fontname = sans]
"/gnu/store/bjhkfxc5axkjl1jv94q0lwym4n6si6f8-gcc-4.9.4.tar.xz.drv" [label = "gcc-4.9.4.tar.xz", shape = box, fontname = sans]
"/gnu/store/cf3m3ddm8dicrsxba4kjnji5lbyagvbk-gcc-mesboot0-2.95.3.drv" [label = "gcc-mesboot0-2.95.3", shape = box, fontname = sans]
"/gnu/store/d1g7i2w8k1g5jrx4n6xv4q8xnrlmz8f3-module-import-compiled.drv" [label = "module-import-compiled-3", shape = box, fontname = sans]
"/gnu/store/fdmz5blhzfczkpjb9jj6bdbhqlpv3i7l-gcc-7.5.0.drv" [label = "gcc-7.5.0", shape = box, fontname = sans]
"/gnu/store/fs2r7irjx7ppqks3zhsqmxb8lah1a4v0-glibc-mesboot0-2.2.5.drv" [label = "glibc-mesboot0-2.2.5", shape = box, fontname = sans]
"/gnu/store/i5wn3xl6p0zw1vglscgk0bs9dwc6hdh6-gcc-static-5.5.0.drv" [label = "gcc-static-5.5.0", shape = box, fontname = sans]
"/gnu/store/imx7vf2qg44yg9i4gsbn5bgpj3crcyr8-gcc-7.5.0.tar.xz.drv" [label = "gcc-7.5.0.tar.xz", shape = box, fontname = sans]
"/gnu/store/lhhbpfhk2xm8znvhnbrig8dfgd9xc80k-libstdc++-7.5.0.drv" [label = "libstdc++-7.5.0", shape = box, fontname = sans]
"/gnu/store/mrsasf73k1yvdcbn1wyb4ad6dk7ns3vn-binutils-mesboot1-2.14.drv" [label = "binutils-mesboot1-2.14", shape = box, fontname = sans]
"/gnu/store/wxpvfy5g3xjl0kp85cmmy66057p88kln-binutils-cross-boot0-2.34.drv" [label = "binutils-cross-boot0-2.34", shape = box, fontname = sans]
"/gnu/store/y4g2gsdxbk2kmp7lih88kdndi7868dnl-gash-utils-boot-0.1.0.drv" [label = "gash-utils-boot-0.1.0", shape = box, fontname = sans]
"/gnu/store/yz7h0nf33465a32yjpm9rh6w9959h34q-gcc-mesboot-4.9.4.drv" [label = "gcc-mesboot-4.9.4", shape = box, fontname = sans]
"/gnu/store/zf6himkd5rz2ll8ym0c2488bgpnkjkkr-gash-boot-0.2.0.drv" [label = "gash-boot-0.2.0", shape = box, fontname = sans]

"/gnu/store/i5wn3xl6p0zw1vglscgk0bs9dwc6hdh6-gcc-static-5.5.0.drv" -> "/gnu/store/fdmz5blhzfczkpjb9jj6bdbhqlpv3i7l-gcc-7.5.0.drv"
"/gnu/store/fdmz5blhzfczkpjb9jj6bdbhqlpv3i7l-gcc-7.5.0.drv" -> "/gnu/store/0zkiqxwm6k637xr5s1690nwllnvybvyw-xz-mesboot-5.0.0.drv"
"/gnu/store/fdmz5blhzfczkpjb9jj6bdbhqlpv3i7l-gcc-7.5.0.drv" -> "/gnu/store/d1g7i2w8k1g5jrx4n6xv4q8xnrlmz8f3-module-import-compiled.drv"
"/gnu/store/fdmz5blhzfczkpjb9jj6bdbhqlpv3i7l-gcc-7.5.0.drv" -> "/gnu/store/imx7vf2qg44yg9i4gsbn5bgpj3crcyr8-gcc-7.5.0.tar.xz.drv"
"/gnu/store/fdmz5blhzfczkpjb9jj6bdbhqlpv3i7l-gcc-7.5.0.drv" -> "/gnu/store/lhhbpfhk2xm8znvhnbrig8dfgd9xc80k-libstdc++-7.5.0.drv"
"/gnu/store/fdmz5blhzfczkpjb9jj6bdbhqlpv3i7l-gcc-7.5.0.drv" -> "/gnu/store/wxpvfy5g3xjl0kp85cmmy66057p88kln-binutils-cross-boot0-2.34.drv"
"/gnu/store/0zkiqxwm6k637xr5s1690nwllnvybvyw-xz-mesboot-5.0.0.drv" -> "/gnu/store/agday74gvxnd6a7191fw2g473b5v66kx-gcc-mesboot1-4.6.4.drv"
"/gnu/store/0zkiqxwm6k637xr5s1690nwllnvybvyw-xz-mesboot-5.0.0.drv" -> "/gnu/store/d1g7i2w8k1g5jrx4n6xv4q8xnrlmz8f3-module-import-compiled.drv"
"/gnu/store/0zkiqxwm6k637xr5s1690nwllnvybvyw-xz-mesboot-5.0.0.drv" -> "/gnu/store/fs2r7irjx7ppqks3zhsqmxb8lah1a4v0-glibc-mesboot0-2.2.5.drv"
"/gnu/store/0zkiqxwm6k637xr5s1690nwllnvybvyw-xz-mesboot-5.0.0.drv" -> "/gnu/store/mrsasf73k1yvdcbn1wyb4ad6dk7ns3vn-binutils-mesboot1-2.14.drv"
"/gnu/store/agday74gvxnd6a7191fw2g473b5v66kx-gcc-mesboot1-4.6.4.drv" -> "/gnu/store/cf3m3ddm8dicrsxba4kjnji5lbyagvbk-gcc-mesboot0-2.95.3.drv"
"/gnu/store/agday74gvxnd6a7191fw2g473b5v66kx-gcc-mesboot1-4.6.4.drv" -> "/gnu/store/d1g7i2w8k1g5jrx4n6xv4q8xnrlmz8f3-module-import-compiled.drv"
"/gnu/store/agday74gvxnd6a7191fw2g473b5v66kx-gcc-mesboot1-4.6.4.drv" -> "/gnu/store/fs2r7irjx7ppqks3zhsqmxb8lah1a4v0-glibc-mesboot0-2.2.5.drv"
"/gnu/store/agday74gvxnd6a7191fw2g473b5v66kx-gcc-mesboot1-4.6.4.drv" -> "/gnu/store/mrsasf73k1yvdcbn1wyb4ad6dk7ns3vn-binutils-mesboot1-2.14.drv"
"/gnu/store/cf3m3ddm8dicrsxba4kjnji5lbyagvbk-gcc-mesboot0-2.95.3.drv" -> "/gnu/store/4ji6ayrgcyfyxpb583qr5ja4awdlxrdc-bootar-1a.drv"
"/gnu/store/cf3m3ddm8dicrsxba4kjnji5lbyagvbk-gcc-mesboot0-2.95.3.drv" -> "/gnu/store/asnd815v865cvfh2l2dlxmh5y556v3i5-gcc-core-mesboot0-2.95.3.drv"
"/gnu/store/cf3m3ddm8dicrsxba4kjnji5lbyagvbk-gcc-mesboot0-2.95.3.drv" -> "/gnu/store/d1g7i2w8k1g5jrx4n6xv4q8xnrlmz8f3-module-import-compiled.drv"
"/gnu/store/cf3m3ddm8dicrsxba4kjnji5lbyagvbk-gcc-mesboot0-2.95.3.drv" -> "/gnu/store/fs2r7irjx7ppqks3zhsqmxb8lah1a4v0-glibc-mesboot0-2.2.5.drv"
"/gnu/store/cf3m3ddm8dicrsxba4kjnji5lbyagvbk-gcc-mesboot0-2.95.3.drv" -> "/gnu/store/y4g2gsdxbk2kmp7lih88kdndi7868dnl-gash-utils-boot-0.1.0.drv"
"/gnu/store/cf3m3ddm8dicrsxba4kjnji5lbyagvbk-gcc-mesboot0-2.95.3.drv" -> "/gnu/store/zf6himkd5rz2ll8ym0c2488bgpnkjkkr-gash-boot-0.2.0.drv"
"/gnu/store/4ji6ayrgcyfyxpb583qr5ja4awdlxrdc-bootar-1a.drv" -> "/gnu/store/alvrmh47xqk7glq9wmvrzivfjp2bcvyc-module-import-compiled.drv"
"/gnu/store/asnd815v865cvfh2l2dlxmh5y556v3i5-gcc-core-mesboot0-2.95.3.drv" -> "/gnu/store/4ji6ayrgcyfyxpb583qr5ja4awdlxrdc-bootar-1a.drv"
"/gnu/store/asnd815v865cvfh2l2dlxmh5y556v3i5-gcc-core-mesboot0-2.95.3.drv" -> "/gnu/store/d1g7i2w8k1g5jrx4n6xv4q8xnrlmz8f3-module-import-compiled.drv"
"/gnu/store/asnd815v865cvfh2l2dlxmh5y556v3i5-gcc-core-mesboot0-2.95.3.drv" -> "/gnu/store/y4g2gsdxbk2kmp7lih88kdndi7868dnl-gash-utils-boot-0.1.0.drv"
"/gnu/store/asnd815v865cvfh2l2dlxmh5y556v3i5-gcc-core-mesboot0-2.95.3.drv" -> "/gnu/store/zf6himkd5rz2ll8ym0c2488bgpnkjkkr-gash-boot-0.2.0.drv"
"/gnu/store/y4g2gsdxbk2kmp7lih88kdndi7868dnl-gash-utils-boot-0.1.0.drv" -> "/gnu/store/4ji6ayrgcyfyxpb583qr5ja4awdlxrdc-bootar-1a.drv"
"/gnu/store/y4g2gsdxbk2kmp7lih88kdndi7868dnl-gash-utils-boot-0.1.0.drv" -> "/gnu/store/alvrmh47xqk7glq9wmvrzivfjp2bcvyc-module-import-compiled.drv"
"/gnu/store/y4g2gsdxbk2kmp7lih88kdndi7868dnl-gash-utils-boot-0.1.0.drv" -> "/gnu/store/zf6himkd5rz2ll8ym0c2488bgpnkjkkr-gash-boot-0.2.0.drv"
"/gnu/store/zf6himkd5rz2ll8ym0c2488bgpnkjkkr-gash-boot-0.2.0.drv" -> "/gnu/store/4ji6ayrgcyfyxpb583qr5ja4awdlxrdc-bootar-1a.drv"
"/gnu/store/zf6himkd5rz2ll8ym0c2488bgpnkjkkr-gash-boot-0.2.0.drv" -> "/gnu/store/alvrmh47xqk7glq9wmvrzivfjp2bcvyc-module-import-compiled.drv"
"/gnu/store/fs2r7irjx7ppqks3zhsqmxb8lah1a4v0-glibc-mesboot0-2.2.5.drv" -> "/gnu/store/4ji6ayrgcyfyxpb583qr5ja4awdlxrdc-bootar-1a.drv"
"/gnu/store/fs2r7irjx7ppqks3zhsqmxb8lah1a4v0-glibc-mesboot0-2.2.5.drv" -> "/gnu/store/asnd815v865cvfh2l2dlxmh5y556v3i5-gcc-core-mesboot0-2.95.3.drv"
"/gnu/store/fs2r7irjx7ppqks3zhsqmxb8lah1a4v0-glibc-mesboot0-2.2.5.drv" -> "/gnu/store/d1g7i2w8k1g5jrx4n6xv4q8xnrlmz8f3-module-import-compiled.drv"
"/gnu/store/fs2r7irjx7ppqks3zhsqmxb8lah1a4v0-glibc-mesboot0-2.2.5.drv" -> "/gnu/store/y4g2gsdxbk2kmp7lih88kdndi7868dnl-gash-utils-boot-0.1.0.drv"
"/gnu/store/fs2r7irjx7ppqks3zhsqmxb8lah1a4v0-glibc-mesboot0-2.2.5.drv" -> "/gnu/store/zf6himkd5rz2ll8ym0c2488bgpnkjkkr-gash-boot-0.2.0.drv"
"/gnu/store/mrsasf73k1yvdcbn1wyb4ad6dk7ns3vn-binutils-mesboot1-2.14.drv" -> "/gnu/store/4ji6ayrgcyfyxpb583qr5ja4awdlxrdc-bootar-1a.drv"
"/gnu/store/mrsasf73k1yvdcbn1wyb4ad6dk7ns3vn-binutils-mesboot1-2.14.drv" -> "/gnu/store/cf3m3ddm8dicrsxba4kjnji5lbyagvbk-gcc-mesboot0-2.95.3.drv"
"/gnu/store/mrsasf73k1yvdcbn1wyb4ad6dk7ns3vn-binutils-mesboot1-2.14.drv" -> "/gnu/store/d1g7i2w8k1g5jrx4n6xv4q8xnrlmz8f3-module-import-compiled.drv"
"/gnu/store/mrsasf73k1yvdcbn1wyb4ad6dk7ns3vn-binutils-mesboot1-2.14.drv" -> "/gnu/store/fs2r7irjx7ppqks3zhsqmxb8lah1a4v0-glibc-mesboot0-2.2.5.drv"
"/gnu/store/mrsasf73k1yvdcbn1wyb4ad6dk7ns3vn-binutils-mesboot1-2.14.drv" -> "/gnu/store/y4g2gsdxbk2kmp7lih88kdndi7868dnl-gash-utils-boot-0.1.0.drv"
"/gnu/store/mrsasf73k1yvdcbn1wyb4ad6dk7ns3vn-binutils-mesboot1-2.14.drv" -> "/gnu/store/zf6himkd5rz2ll8ym0c2488bgpnkjkkr-gash-boot-0.2.0.drv"
"/gnu/store/imx7vf2qg44yg9i4gsbn5bgpj3crcyr8-gcc-7.5.0.tar.xz.drv" -> "/gnu/store/5x8a1yib7vdza727vrq4zmp6cmsafy7h-module-import-compiled.drv"
"/gnu/store/lhhbpfhk2xm8znvhnbrig8dfgd9xc80k-libstdc++-7.5.0.drv" -> "/gnu/store/0zkiqxwm6k637xr5s1690nwllnvybvyw-xz-mesboot-5.0.0.drv"
"/gnu/store/lhhbpfhk2xm8znvhnbrig8dfgd9xc80k-libstdc++-7.5.0.drv" -> "/gnu/store/d1g7i2w8k1g5jrx4n6xv4q8xnrlmz8f3-module-import-compiled.drv"
"/gnu/store/lhhbpfhk2xm8znvhnbrig8dfgd9xc80k-libstdc++-7.5.0.drv" -> "/gnu/store/imx7vf2qg44yg9i4gsbn5bgpj3crcyr8-gcc-7.5.0.tar.xz.drv"
"/gnu/store/lhhbpfhk2xm8znvhnbrig8dfgd9xc80k-libstdc++-7.5.0.drv" -> "/gnu/store/wxpvfy5g3xjl0kp85cmmy66057p88kln-binutils-cross-boot0-2.34.drv"
"/gnu/store/wxpvfy5g3xjl0kp85cmmy66057p88kln-binutils-cross-boot0-2.34.drv" -> "/gnu/store/07bd5ll0adnyrv1zaz11vz2x1ax447ka-glibc-mesboot-2.16.0.drv"
"/gnu/store/wxpvfy5g3xjl0kp85cmmy66057p88kln-binutils-cross-boot0-2.34.drv" -> "/gnu/store/0zkiqxwm6k637xr5s1690nwllnvybvyw-xz-mesboot-5.0.0.drv"
"/gnu/store/wxpvfy5g3xjl0kp85cmmy66057p88kln-binutils-cross-boot0-2.34.drv" -> "/gnu/store/2pziz2j7781mhadl8lcfpzm8anvazb37-binutils-mesboot-2.20.1a.drv"
"/gnu/store/wxpvfy5g3xjl0kp85cmmy66057p88kln-binutils-cross-boot0-2.34.drv" -> "/gnu/store/d1g7i2w8k1g5jrx4n6xv4q8xnrlmz8f3-module-import-compiled.drv"
"/gnu/store/wxpvfy5g3xjl0kp85cmmy66057p88kln-binutils-cross-boot0-2.34.drv" -> "/gnu/store/yz7h0nf33465a32yjpm9rh6w9959h34q-gcc-mesboot-4.9.4.drv"
"/gnu/store/07bd5ll0adnyrv1zaz11vz2x1ax447ka-glibc-mesboot-2.16.0.drv" -> "/gnu/store/0zkiqxwm6k637xr5s1690nwllnvybvyw-xz-mesboot-5.0.0.drv"
"/gnu/store/07bd5ll0adnyrv1zaz11vz2x1ax447ka-glibc-mesboot-2.16.0.drv" -> "/gnu/store/2pziz2j7781mhadl8lcfpzm8anvazb37-binutils-mesboot-2.20.1a.drv"
"/gnu/store/07bd5ll0adnyrv1zaz11vz2x1ax447ka-glibc-mesboot-2.16.0.drv" -> "/gnu/store/agday74gvxnd6a7191fw2g473b5v66kx-gcc-mesboot1-4.6.4.drv"
"/gnu/store/07bd5ll0adnyrv1zaz11vz2x1ax447ka-glibc-mesboot-2.16.0.drv" -> "/gnu/store/d1g7i2w8k1g5jrx4n6xv4q8xnrlmz8f3-module-import-compiled.drv"
"/gnu/store/07bd5ll0adnyrv1zaz11vz2x1ax447ka-glibc-mesboot-2.16.0.drv" -> "/gnu/store/fs2r7irjx7ppqks3zhsqmxb8lah1a4v0-glibc-mesboot0-2.2.5.drv"
"/gnu/store/2pziz2j7781mhadl8lcfpzm8anvazb37-binutils-mesboot-2.20.1a.drv" -> "/gnu/store/0zkiqxwm6k637xr5s1690nwllnvybvyw-xz-mesboot-5.0.0.drv"
"/gnu/store/2pziz2j7781mhadl8lcfpzm8anvazb37-binutils-mesboot-2.20.1a.drv" -> "/gnu/store/agday74gvxnd6a7191fw2g473b5v66kx-gcc-mesboot1-4.6.4.drv"
"/gnu/store/2pziz2j7781mhadl8lcfpzm8anvazb37-binutils-mesboot-2.20.1a.drv" -> "/gnu/store/d1g7i2w8k1g5jrx4n6xv4q8xnrlmz8f3-module-import-compiled.drv"
"/gnu/store/2pziz2j7781mhadl8lcfpzm8anvazb37-binutils-mesboot-2.20.1a.drv" -> "/gnu/store/fs2r7irjx7ppqks3zhsqmxb8lah1a4v0-glibc-mesboot0-2.2.5.drv"
"/gnu/store/2pziz2j7781mhadl8lcfpzm8anvazb37-binutils-mesboot-2.20.1a.drv" -> "/gnu/store/mrsasf73k1yvdcbn1wyb4ad6dk7ns3vn-binutils-mesboot1-2.14.drv"
"/gnu/store/yz7h0nf33465a32yjpm9rh6w9959h34q-gcc-mesboot-4.9.4.drv" -> "/gnu/store/07bd5ll0adnyrv1zaz11vz2x1ax447ka-glibc-mesboot-2.16.0.drv"
"/gnu/store/yz7h0nf33465a32yjpm9rh6w9959h34q-gcc-mesboot-4.9.4.drv" -> "/gnu/store/0zkiqxwm6k637xr5s1690nwllnvybvyw-xz-mesboot-5.0.0.drv"
"/gnu/store/yz7h0nf33465a32yjpm9rh6w9959h34q-gcc-mesboot-4.9.4.drv" -> "/gnu/store/2pziz2j7781mhadl8lcfpzm8anvazb37-binutils-mesboot-2.20.1a.drv"
"/gnu/store/yz7h0nf33465a32yjpm9rh6w9959h34q-gcc-mesboot-4.9.4.drv" -> "/gnu/store/agday74gvxnd6a7191fw2g473b5v66kx-gcc-mesboot1-4.6.4.drv"
"/gnu/store/yz7h0nf33465a32yjpm9rh6w9959h34q-gcc-mesboot-4.9.4.drv" -> "/gnu/store/bjhkfxc5axkjl1jv94q0lwym4n6si6f8-gcc-4.9.4.tar.xz.drv"
"/gnu/store/yz7h0nf33465a32yjpm9rh6w9959h34q-gcc-mesboot-4.9.4.drv" -> "/gnu/store/d1g7i2w8k1g5jrx4n6xv4q8xnrlmz8f3-module-import-compiled.drv"
"/gnu/store/bjhkfxc5axkjl1jv94q0lwym4n6si6f8-gcc-4.9.4.tar.xz.drv" -> "/gnu/store/5x8a1yib7vdza727vrq4zmp6cmsafy7h-module-import-compiled.drv"
}

[-- Attachment #1.5: Type: text/plain, Size: 244 bytes --]


I generated the GraphViz source by manually massaging the output of the
following script, which walks the graph of derivations that produce an
output that differs across machines, starting with the derivation that
cross-compiles %gcc-static:


[-- Attachment #1.6: Script to Analyze Derivations --]
[-- Type: text/plain, Size: 6343 bytes --]

(use-modules
 (guix ssh)
 (guix packages)
 (guix derivations)
 (guix store)
 (guix inferior)
 (srfi srfi-9)
 (srfi srfi-1)
 (ice-9 match))

;; A very janky script to compare derivation outputs on two machines.
;; The basic idea is to start from a derivation that is known to
;; produce an output that differs on two machines.  We walk the graph
;; of inputs starting from that derivation (an input here is a
;; combination of a derivation and an output, such as "out" or
;; "static").  As we go, we record the derivations that produced a
;; differing output on the two machines.  We stop once there are no
;; more such inputs to walk.
;;
;; It is expected that any derivations under examination have already
;; been built on both machines.  It's probably best to just build the
;; root derivation on both machines before running this script.  Make
;; sure you build without using substitutes, since we're trying to
;; compare reproducibility between machines when built from source.
;;
;; This script was written under the assumption that all machines
;; - the local machine, gs1, and gs2 - are using the same version of
;; Guix.  Run this script with: "guix repl the-script.scm"

(define gs1-eval
  (let* ((session-gs1 (open-ssh-session "gs1"))
         (inferior-gs1 (remote-inferior session-gs1)))
    (lambda (exp)
      (inferior-eval exp inferior-gs1))))

(define gs2-eval
  (let* ((session-gs2 (open-ssh-session "gs2"))
         (inferior-gs2 (remote-inferior session-gs2)))
    (lambda (exp)
      (inferior-eval exp inferior-gs2))))

;; Represent edges in a graph as pairs.

(define (make-edge from to)
  (cons from to))

(define (edge-from edge)
  (car edge))

(define (edge-to edge)
  (cdr edge))

(define (get-inputs drv)
  (pk 'get-inputs drv)
  ;; return the inputs of drv (according to gs1, or gs2, or locally)
  ;; one input is e.g. ("/gnu/store/...foo.drv" ("out")) - a
  ;; two-element list.
  (gs1-eval
   `(map (lambda (input)
           (list (derivation-input-path input)
                 (derivation-input-sub-derivations input)))
         (derivation-inputs (read-derivation-from-file ,drv)))))

;; Evaluate to the recursive SHA-256 hash, as a string, of the output
;; path.
(define (compute-hash-exp file)
  (pk 'compute-hash-exp file)
  `(let-values (((port get-hash)
                 (open-hash-port (lookup-hash-algorithm 'sha256))))
     (write-file ,file port)
     (force-output port)
     (bytevector->nix-base32-string (get-hash))))

(define (get-drv-output-paths drv output-names)
  (pk 'get-drv-output-paths drv output-names)
  ;; from any of gs1 or gs2 or local
  (sort-list
   (gs1-eval
    `(filter-map (match-lambda
                   ((name . output)
                    (if (member name ',output-names)
                        (derivation-output-path output)
                        #f)))
                 (derivation-outputs (read-derivation-from-file ,drv))))
   string<))

(define (get-hashes-gs1 files)
  (pk 'get-hashes-gs1 files)
  (sort-list
   (map (lambda (file)
          ;; There aren't usually many outputs, so connecting once per
          ;; output isn't so bad.
          (gs1-eval (compute-hash-exp file)))
        files)
   string<))

(define (get-hashes-gs2 files)
  (pk 'get-hashes-gs2 files)
  (sort-list
   (map (lambda (file)
          ;; There aren't usually many outputs, so connecting once per
          ;; output isn't so bad.
          (gs2-eval (compute-hash-exp file)))
        files)
   string<))

;; differs for some output.
(define (any-output-differs-on-gs1-and-gs2 drv outputs)
  (pk 'any-output-differs-on-gs1-and-gs2 drv outputs)
  (let* ((drv-output-paths (get-drv-output-paths drv outputs))
         (hashes-gs1 (get-hashes-gs1 drv-output-paths))
         (hashes-gs2 (get-hashes-gs2 drv-output-paths)))
    (any (lambda (hash-gs1 hash-gs2)
           (not (string= hash-gs1 hash-gs2)))         
         hashes-gs1
         hashes-gs2)))

(define (get-bad-drvs inputs)
  (pk 'get-bad-drvs inputs)
  ;; return the drvs that are not identical on gs1 and gs2
  (filter-map (match-lambda
                ((drv outputs)
                 (if (any-output-differs-on-gs1-and-gs2 drv outputs)
                     (begin
                       (display "Differs: ")
                       (display drv)
                       (newline)
                       ;; Without the outputs because the entire drv
                       ;; must be run in order to build any outputs.
                       drv)
                     #f)))
              inputs))

(define visited
  (make-hash-table))

;; from-drv is assumed to be bad.
;; The bad-drvs returned are a list of edges describing the graph of bad derivations.
(define (get-bad-drv-edges from-drv)
  (pk 'get-bad-drv-edges from-drv)
  ;; Don't visit the same node twice, to avoid loops.
  (if (hash-ref visited from-drv)
      '()
      (begin
        (hash-set! visited from-drv #t)
        (let* ((inputs (get-inputs from-drv))
               (bad-drvs (get-bad-drvs inputs))
               (bad-drv-edges (map (lambda (bad-drv)
                                     (make-edge from-drv bad-drv))
                                   bad-drvs)))
          (append bad-drv-edges
                  (append-map (lambda (bad-drv)
                                (get-bad-drv-edges bad-drv))
                              bad-drvs))))))


(pk 'gs1-use-modules
    (gs1-eval
     '(use-modules (guix)
                   (ice-9 match)
                   (srfi srfi-1)
                   (srfi srfi-11)
                   (gcrypt hash)
                   (guix serialization))))

(pk 'gs2-use-modules
    (gs2-eval
     '(use-modules (guix)
                   (ice-9 match)
                   (srfi srfi-1)
                   (srfi srfi-11)
                   (gcrypt hash)
                   (guix serialization))))

;; guix build -d --target=powerpc64-linux-gnu --no-grafts -e '(@@ (gnu packages make-bootstrap) %gcc-static)'
(define root-drv
  "/gnu/store/i5wn3xl6p0zw1vglscgk0bs9dwc6hdh6-gcc-static-5.5.0.drv")

(call-with-output-file "/tmp/myedges"
  (lambda (port)
    (write (get-bad-drv-edges root-drv) port)))

(display "done")
(newline)

[-- Attachment #1.7: Type: text/plain, Size: 7148 bytes --]


This graph shows the derivations that produce differing outputs across
two Guix System machines, when everything is built without substitutes,
walking the graph of inputs starting from the derivation that
cross-compiles %gcc-static for powerpc64-linux-gnu (from x86_64-linux)
using Guix at commit 1ced8379c7641788fa607b19b7a66d18f045362b.  If the
non-reproducibility (across systems) of %gcc-static is caused by a
similarly non-reproducible input, it's probably caused by one or more of
the derivations in this graph.  This graph is significantly smaller than
the graph of all derivations, but it still includes 21 derivations.

Some things I noticed while looking at the graph:

- It's suspicious that the module-import-compiled derivations are all at
  the bottom.  I spot-checked the output of one of the
  module-import-compiled derivations on both machines and found that the
  .go files differed.

- Every derivation except the root (gcc-static-5.5.0-drv) uses
  /gnu/store/0dn61y4n8ig333b23hmc80hvlcy8gdli-guile-bootstrap-2.0.drv as
  an input.  The root uses
  /gnu/store/b5nnbpgkvgdpzgvj67539ylcaqacj90l-guile-3.0.2.drv.

- Strangely,
  /gnu/store/bjhkfxc5axkjl1jv94q0lwym4n6si6f8-gcc-4.9.4.tar.xz.drv is
  non-reproducible, even though it just builds a tarball.  The tarball
  itself does differ on the two systems, but the contents, when
  extracted, are identical.

- Some of the derivations in the graph are reproducible when built
  locally via "guix build --check"; however, some are not reproducible.
  For every derivation, if it was reproducible locally on gs1, it was
  also reproducible on gs2, and if it was not reproducible on gs1, it
  was also not reproducible on gs2.  Here is the full report:

--8<---------------cut here---------------start------------->8---
rm -f /tmp/mylog && for drv in $(</tmp/drvs); do if guix build --check $drv; then echo OK: $drv >> /tmp/mylog; else echo FAILED: $drv >> /tmp/mylog; fi; done 2>&1 | tee /tmp/mylog-toplevel

...

marusich@gs1 ~$ grep 'may not be deterministic' /tmp/mylog-toplevel
guix build: error: derivation `/gnu/store/07bd5ll0adnyrv1zaz11vz2x1ax447ka-glibc-mesboot-2.16.0.drv' may not be deterministic: output `/gnu/store/wgb70ir475v4vhg4bcjzv4h2kaa93qj7-glibc-mesboot-2.16.0' differs
guix build: error: derivation `/gnu/store/0zkiqxwm6k637xr5s1690nwllnvybvyw-xz-mesboot-5.0.0.drv' may not be deterministic: output `/gnu/store/r1zsxj7wlvw1aa1ifv3nyrrjag44pc9s-xz-mesboot-5.0.0' differs
guix build: error: derivation `/gnu/store/2pziz2j7781mhadl8lcfpzm8anvazb37-binutils-mesboot-2.20.1a.drv' may not be deterministic: output `/gnu/store/mza1k0mgr8ab407yzavwwz07ly7221wy-binutils-mesboot-2.20.1a' differs
guix build: error: derivation `/gnu/store/4ji6ayrgcyfyxpb583qr5ja4awdlxrdc-bootar-1a.drv' may not be deterministic: output `/gnu/store/x2iy3ka299sk1x97an783jl4pfbn60pj-bootar-1a' differs
guix build: error: derivation `/gnu/store/agday74gvxnd6a7191fw2g473b5v66kx-gcc-mesboot1-4.6.4.drv' may not be deterministic: output `/gnu/store/q7f07gpd9sq9krc68gxa73vbqmk06y0a-gcc-mesboot1-4.6.4' differs
guix build: error: derivation `/gnu/store/asnd815v865cvfh2l2dlxmh5y556v3i5-gcc-core-mesboot0-2.95.3.drv' may not be deterministic: output `/gnu/store/ri28kdl41bb76qjr4cyarylw7kxpvfxy-gcc-core-mesboot0-2.95.3' differs
guix build: error: derivation `/gnu/store/bjhkfxc5axkjl1jv94q0lwym4n6si6f8-gcc-4.9.4.tar.xz.drv' may not be deterministic: output `/gnu/store/56km255bk1s880yn6rxw4wj3crm2pshk-gcc-4.9.4.tar.xz' differs
guix build: error: derivation `/gnu/store/cf3m3ddm8dicrsxba4kjnji5lbyagvbk-gcc-mesboot0-2.95.3.drv' may not be deterministic: output `/gnu/store/i4p3mp4sghwbjnn6w9dffpswgc31jhm7-gcc-mesboot0-2.95.3' differs
guix build: error: derivation `/gnu/store/fdmz5blhzfczkpjb9jj6bdbhqlpv3i7l-gcc-7.5.0.drv' may not be deterministic: output `/gnu/store/01b4w3m6mp55y531kyi1g8shh722kwqm-gcc-7.5.0-lib' differs
guix build: error: derivation `/gnu/store/fs2r7irjx7ppqks3zhsqmxb8lah1a4v0-glibc-mesboot0-2.2.5.drv' may not be deterministic: output `/gnu/store/6h5a5sn3bx3wlkw8140ksxymwlksm6iz-glibc-mesboot0-2.2.5' differs
guix build: error: derivation `/gnu/store/imx7vf2qg44yg9i4gsbn5bgpj3crcyr8-gcc-7.5.0.tar.xz.drv' may not be deterministic: output `/gnu/store/9qrvpjwydmqfjm81frrdcn0n3gpn0bs3-gcc-7.5.0.tar.xz' differs
guix build: error: derivation `/gnu/store/mrsasf73k1yvdcbn1wyb4ad6dk7ns3vn-binutils-mesboot1-2.14.drv' may not be deterministic: output `/gnu/store/zpqfz6hvg47xml0yrwfy491drfbgg19p-binutils-mesboot1-2.14' differs
guix build: error: derivation `/gnu/store/wxpvfy5g3xjl0kp85cmmy66057p88kln-binutils-cross-boot0-2.34.drv' may not be deterministic: output `/gnu/store/2601fgjs5h9cpvv5nxrcv52wix29ifrd-binutils-cross-boot0-2.34' differs
guix build: error: derivation `/gnu/store/y4g2gsdxbk2kmp7lih88kdndi7868dnl-gash-utils-boot-0.1.0.drv' may not be deterministic: output `/gnu/store/zsz0mj4jdsz4l511d368hwx2d9wrg4my-gash-utils-boot-0.1.0' differs
guix build: error: derivation `/gnu/store/yz7h0nf33465a32yjpm9rh6w9959h34q-gcc-mesboot-4.9.4.drv' may not be deterministic: output `/gnu/store/zz2blvhk3gdqppcwy089p56f57q7zj0q-gcc-mesboot-4.9.4' differs
guix build: error: derivation `/gnu/store/zf6himkd5rz2ll8ym0c2488bgpnkjkkr-gash-boot-0.2.0.drv' may not be deterministic: output `/gnu/store/prkqai3zwh3shlqpll6xyncmmqpj49dd-gash-boot-0.2.0' differs
marusich@gs1 ~$ cat /tmp/mylog
FAILED: /gnu/store/07bd5ll0adnyrv1zaz11vz2x1ax447ka-glibc-mesboot-2.16.0.drv
FAILED: /gnu/store/0zkiqxwm6k637xr5s1690nwllnvybvyw-xz-mesboot-5.0.0.drv
FAILED: /gnu/store/2pziz2j7781mhadl8lcfpzm8anvazb37-binutils-mesboot-2.20.1a.drv
FAILED: /gnu/store/4ji6ayrgcyfyxpb583qr5ja4awdlxrdc-bootar-1a.drv
FAILED: /gnu/store/agday74gvxnd6a7191fw2g473b5v66kx-gcc-mesboot1-4.6.4.drv
FAILED: /gnu/store/asnd815v865cvfh2l2dlxmh5y556v3i5-gcc-core-mesboot0-2.95.3.drv
FAILED: /gnu/store/bjhkfxc5axkjl1jv94q0lwym4n6si6f8-gcc-4.9.4.tar.xz.drv
FAILED: /gnu/store/cf3m3ddm8dicrsxba4kjnji5lbyagvbk-gcc-mesboot0-2.95.3.drv
FAILED: /gnu/store/fdmz5blhzfczkpjb9jj6bdbhqlpv3i7l-gcc-7.5.0.drv
FAILED: /gnu/store/fs2r7irjx7ppqks3zhsqmxb8lah1a4v0-glibc-mesboot0-2.2.5.drv
OK: /gnu/store/i5wn3xl6p0zw1vglscgk0bs9dwc6hdh6-gcc-static-5.5.0.drv
FAILED: /gnu/store/imx7vf2qg44yg9i4gsbn5bgpj3crcyr8-gcc-7.5.0.tar.xz.drv
OK: /gnu/store/lhhbpfhk2xm8znvhnbrig8dfgd9xc80k-libstdc++-7.5.0.drv
FAILED: /gnu/store/mrsasf73k1yvdcbn1wyb4ad6dk7ns3vn-binutils-mesboot1-2.14.drv
FAILED: /gnu/store/wxpvfy5g3xjl0kp85cmmy66057p88kln-binutils-cross-boot0-2.34.drv
FAILED: /gnu/store/y4g2gsdxbk2kmp7lih88kdndi7868dnl-gash-utils-boot-0.1.0.drv
FAILED: /gnu/store/yz7h0nf33465a32yjpm9rh6w9959h34q-gcc-mesboot-4.9.4.drv
FAILED: /gnu/store/zf6himkd5rz2ll8ym0c2488bgpnkjkkr-gash-boot-0.2.0.drv
marusich@gs1 ~$ cat /tmp/mylog | grep ^FAILED | wc -l
16
marusich@gs1 ~$ grep 'may not be deterministic' /tmp/mylog-toplevel | wc -l
16
marusich@gs1 ~$ 
--8<---------------cut here---------------end--------------->8---

Although this does not resovle the cross-system reproducibility issue,
it does seem to narrow down the possibilities.

-- 
Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]