This graph shows the derivations that produce differing outputs across
two Guix System machines, when everything is built without substitutes,
walking the graph of inputs starting from the derivation that
cross-compiles %gcc-static for powerpc64-linux-gnu (from x86_64-linux)
using Guix at commit 1ced8379c7641788fa607b19b7a66d18f045362b.  If the
non-reproducibility (across systems) of %gcc-static is caused by a
similarly non-reproducible input, it's probably caused by one or more of
the derivations in this graph.  This graph is significantly smaller than
the graph of all derivations, but it still includes 21 derivations.

Some things I noticed while looking at the graph:

- It's suspicious that the module-import-compiled derivations are all at
  the bottom.  I spot-checked the output of one of the
  module-import-compiled derivations on both machines and found that the
  .go files differed.

- Every derivation except the root (gcc-static-5.5.0-drv) uses
  /gnu/store/0dn61y4n8ig333b23hmc80hvlcy8gdli-guile-bootstrap-2.0.drv as
  an input.  The root uses
  /gnu/store/b5nnbpgkvgdpzgvj67539ylcaqacj90l-guile-3.0.2.drv.

- Strangely,
  /gnu/store/bjhkfxc5axkjl1jv94q0lwym4n6si6f8-gcc-4.9.4.tar.xz.drv is
  non-reproducible, even though it just builds a tarball.  The tarball
  itself does differ on the two systems, but the contents, when
  extracted, are identical.

- Some of the derivations in the graph are reproducible when built
  locally via "guix build --check"; however, some are not reproducible.
  For every derivation, if it was reproducible locally on gs1, it was
  also reproducible on gs2, and if it was not reproducible on gs1, it
  was also not reproducible on gs2.  Here is the full report:

--8<---------------cut here---------------start------------->8---
rm -f /tmp/mylog && for drv in $(</tmp/drvs); do if guix build --check $drv; then echo OK: $drv >> /tmp/mylog; else echo FAILED: $drv >> /tmp/mylog; fi; done 2>&1 | tee /tmp/mylog-toplevel

...

marusich@gs1 ~$ grep 'may not be deterministic' /tmp/mylog-toplevel
guix build: error: derivation `/gnu/store/07bd5ll0adnyrv1zaz11vz2x1ax447ka-glibc-mesboot-2.16.0.drv' may not be deterministic: output `/gnu/store/wgb70ir475v4vhg4bcjzv4h2kaa93qj7-glibc-mesboot-2.16.0' differs
guix build: error: derivation `/gnu/store/0zkiqxwm6k637xr5s1690nwllnvybvyw-xz-mesboot-5.0.0.drv' may not be deterministic: output `/gnu/store/r1zsxj7wlvw1aa1ifv3nyrrjag44pc9s-xz-mesboot-5.0.0' differs
guix build: error: derivation `/gnu/store/2pziz2j7781mhadl8lcfpzm8anvazb37-binutils-mesboot-2.20.1a.drv' may not be deterministic: output `/gnu/store/mza1k0mgr8ab407yzavwwz07ly7221wy-binutils-mesboot-2.20.1a' differs
guix build: error: derivation `/gnu/store/4ji6ayrgcyfyxpb583qr5ja4awdlxrdc-bootar-1a.drv' may not be deterministic: output `/gnu/store/x2iy3ka299sk1x97an783jl4pfbn60pj-bootar-1a' differs
guix build: error: derivation `/gnu/store/agday74gvxnd6a7191fw2g473b5v66kx-gcc-mesboot1-4.6.4.drv' may not be deterministic: output `/gnu/store/q7f07gpd9sq9krc68gxa73vbqmk06y0a-gcc-mesboot1-4.6.4' differs
guix build: error: derivation `/gnu/store/asnd815v865cvfh2l2dlxmh5y556v3i5-gcc-core-mesboot0-2.95.3.drv' may not be deterministic: output `/gnu/store/ri28kdl41bb76qjr4cyarylw7kxpvfxy-gcc-core-mesboot0-2.95.3' differs
guix build: error: derivation `/gnu/store/bjhkfxc5axkjl1jv94q0lwym4n6si6f8-gcc-4.9.4.tar.xz.drv' may not be deterministic: output `/gnu/store/56km255bk1s880yn6rxw4wj3crm2pshk-gcc-4.9.4.tar.xz' differs
guix build: error: derivation `/gnu/store/cf3m3ddm8dicrsxba4kjnji5lbyagvbk-gcc-mesboot0-2.95.3.drv' may not be deterministic: output `/gnu/store/i4p3mp4sghwbjnn6w9dffpswgc31jhm7-gcc-mesboot0-2.95.3' differs
guix build: error: derivation `/gnu/store/fdmz5blhzfczkpjb9jj6bdbhqlpv3i7l-gcc-7.5.0.drv' may not be deterministic: output `/gnu/store/01b4w3m6mp55y531kyi1g8shh722kwqm-gcc-7.5.0-lib' differs
guix build: error: derivation `/gnu/store/fs2r7irjx7ppqks3zhsqmxb8lah1a4v0-glibc-mesboot0-2.2.5.drv' may not be deterministic: output `/gnu/store/6h5a5sn3bx3wlkw8140ksxymwlksm6iz-glibc-mesboot0-2.2.5' differs
guix build: error: derivation `/gnu/store/imx7vf2qg44yg9i4gsbn5bgpj3crcyr8-gcc-7.5.0.tar.xz.drv' may not be deterministic: output `/gnu/store/9qrvpjwydmqfjm81frrdcn0n3gpn0bs3-gcc-7.5.0.tar.xz' differs
guix build: error: derivation `/gnu/store/mrsasf73k1yvdcbn1wyb4ad6dk7ns3vn-binutils-mesboot1-2.14.drv' may not be deterministic: output `/gnu/store/zpqfz6hvg47xml0yrwfy491drfbgg19p-binutils-mesboot1-2.14' differs
guix build: error: derivation `/gnu/store/wxpvfy5g3xjl0kp85cmmy66057p88kln-binutils-cross-boot0-2.34.drv' may not be deterministic: output `/gnu/store/2601fgjs5h9cpvv5nxrcv52wix29ifrd-binutils-cross-boot0-2.34' differs
guix build: error: derivation `/gnu/store/y4g2gsdxbk2kmp7lih88kdndi7868dnl-gash-utils-boot-0.1.0.drv' may not be deterministic: output `/gnu/store/zsz0mj4jdsz4l511d368hwx2d9wrg4my-gash-utils-boot-0.1.0' differs
guix build: error: derivation `/gnu/store/yz7h0nf33465a32yjpm9rh6w9959h34q-gcc-mesboot-4.9.4.drv' may not be deterministic: output `/gnu/store/zz2blvhk3gdqppcwy089p56f57q7zj0q-gcc-mesboot-4.9.4' differs
guix build: error: derivation `/gnu/store/zf6himkd5rz2ll8ym0c2488bgpnkjkkr-gash-boot-0.2.0.drv' may not be deterministic: output `/gnu/store/prkqai3zwh3shlqpll6xyncmmqpj49dd-gash-boot-0.2.0' differs
marusich@gs1 ~$ cat /tmp/mylog
FAILED: /gnu/store/07bd5ll0adnyrv1zaz11vz2x1ax447ka-glibc-mesboot-2.16.0.drv
FAILED: /gnu/store/0zkiqxwm6k637xr5s1690nwllnvybvyw-xz-mesboot-5.0.0.drv
FAILED: /gnu/store/2pziz2j7781mhadl8lcfpzm8anvazb37-binutils-mesboot-2.20.1a.drv
FAILED: /gnu/store/4ji6ayrgcyfyxpb583qr5ja4awdlxrdc-bootar-1a.drv
FAILED: /gnu/store/agday74gvxnd6a7191fw2g473b5v66kx-gcc-mesboot1-4.6.4.drv
FAILED: /gnu/store/asnd815v865cvfh2l2dlxmh5y556v3i5-gcc-core-mesboot0-2.95.3.drv
FAILED: /gnu/store/bjhkfxc5axkjl1jv94q0lwym4n6si6f8-gcc-4.9.4.tar.xz.drv
FAILED: /gnu/store/cf3m3ddm8dicrsxba4kjnji5lbyagvbk-gcc-mesboot0-2.95.3.drv
FAILED: /gnu/store/fdmz5blhzfczkpjb9jj6bdbhqlpv3i7l-gcc-7.5.0.drv
FAILED: /gnu/store/fs2r7irjx7ppqks3zhsqmxb8lah1a4v0-glibc-mesboot0-2.2.5.drv
OK: /gnu/store/i5wn3xl6p0zw1vglscgk0bs9dwc6hdh6-gcc-static-5.5.0.drv
FAILED: /gnu/store/imx7vf2qg44yg9i4gsbn5bgpj3crcyr8-gcc-7.5.0.tar.xz.drv
OK: /gnu/store/lhhbpfhk2xm8znvhnbrig8dfgd9xc80k-libstdc++-7.5.0.drv
FAILED: /gnu/store/mrsasf73k1yvdcbn1wyb4ad6dk7ns3vn-binutils-mesboot1-2.14.drv
FAILED: /gnu/store/wxpvfy5g3xjl0kp85cmmy66057p88kln-binutils-cross-boot0-2.34.drv
FAILED: /gnu/store/y4g2gsdxbk2kmp7lih88kdndi7868dnl-gash-utils-boot-0.1.0.drv
FAILED: /gnu/store/yz7h0nf33465a32yjpm9rh6w9959h34q-gcc-mesboot-4.9.4.drv
FAILED: /gnu/store/zf6himkd5rz2ll8ym0c2488bgpnkjkkr-gash-boot-0.2.0.drv
marusich@gs1 ~$ cat /tmp/mylog | grep ^FAILED | wc -l
16
marusich@gs1 ~$ grep 'may not be deterministic' /tmp/mylog-toplevel | wc -l
16
marusich@gs1 ~$ 
--8<---------------cut here---------------end--------------->8---

Although this does not resovle the cross-system reproducibility issue,
it does seem to narrow down the possibilities.

-- 
Chris