Hi everyone, Efraim Flashner writes: > Is this a file we actually need during the bootstrap process? Can we > "work around it" by just deleting it? That's a good idea. I tried building %gcc-static with the --disable-libstdcxx configure flag (see attached patch), and that caused the build of %gcc-static itself to become reproducible on Debian and Fedora when using exactly the same inputs. To clarify, I have recently run the following experiments: +------+-------------+---------------------+--------------+----------------+---------------+ | Case | Build | libstdcxx | substitutes? | Inputs | Result | +------+-------------+---------------------+--------------+----------------+---------------+ | | | | | Built fresh on | Output | | 1 | %gcc-static | enabled | yes | Debian, copied | differs: | | | | | | to Fedora | libstdc++.a | +------+-------------+---------------------+--------------+----------------+---------------+ | | | | | Re-used inputs | Toy binary | | 2 | toy program | n/a | yes | from above | does not | | | | | | | differ | +------+-------------+---------------------+--------------+----------------+---------------+ | | | | | Built fresh on | Output does | | 3 | %gcc-static | disabled | yes | Debian, copied | not differ | | | | | | to Fedora | | +------+-------------+---------------------+--------------+----------------+---------------+ | | | | | Built fresh on | Output | | 4 | %gcc-static | disabled | yes | Debian, fresh | differs: | | | | | | on Fedora | various files | +------+-------------+---------------------+--------------+----------------+---------------+ | | | | | Inputs failed | Build failed | | 5 | %gcc-static | disabled | no | to build on | on both | | | | | | both systems | systems | +------+-------------+---------------------+--------------+----------------+---------------+ The "toy program" in case 2 was just this: #include int main() { printf("Hello"); return 0; } When I say I "copied" the inputs from Debian to Fedora, I mean just that. I copied stuff like /gnu and /var/guix from Debian to Fedora (while guix-daemon was stopped, of course), GC'd just %gcc-static on Fedora, and then rebuilt %gcc-static on Fedora so it would use exactly the same inputs as were used on Debian. The most notable new findings are cases 3 and 4. Case 5 made me pretty sad because I spent almost an entire weekend trying to build Guix from source using various binary Guix releases, and in none of the attempts was I successful in running "guix pull" or even just "guix environment --pure guix" without substitutes, which really surprised me. Case 3 confirms Efraim's suggestion: we can fix the libstdc++.a reproducibility problem by simply not building libstdc++.a in the first place. It also confirms that, when built with --disable-libstdcxx, %gcc-static can be built reproducibly on different systems as long as exactly the same inputs are used. Whether or not %gcc-static can be used to successfully bootstrap packages on powerpc64-linux when built in this way remains to be seen. Case 4, unfortunately, demonstrates that there are still other reproducibility issues that we have not yet resolved. Since the only difference between cases 3 and 4 is how the inputs were realized, the differing %gcc-static output must be caused by the inputs somehow. In case 4, the %gcc-static output differed in the following files (the hashes on the left are MD5 hashes): --8<---------------cut here---------------start------------->8--- --- /dev/fd/63 2020-09-25 20:35:33.386554595 -0700 +++ /dev/fd/62 2020-09-25 20:35:33.387554604 -0700 @@ -1,28 +1,28 @@ -c9b0dfcbad566c0b8b88df94bb993312 ./bin/c++ -092823145dc96b9eb81111362f7b4ced ./bin/cpp -c9b0dfcbad566c0b8b88df94bb993312 ./bin/g++ -e4cc43b7790dcd25f31419bad606b36e ./bin/gcc +8f02302b55643f1c711e472a42fea8bd ./bin/c++ +9f1fd993e4f2b796fcc56f0b2f8a47d2 ./bin/cpp +8f02302b55643f1c711e472a42fea8bd ./bin/g++ +583d1b011a7ba009d7385117dd7a33c8 ./bin/gcc f9d94f4bb61f70d14ea4b2ce73c9be9d ./bin/gcc-ar 01fc2184f99c558771aa8f2fe30b373d ./bin/gcc-nm da5356ee09ccda4ca06758d056370f7e ./bin/gcc-ranlib -98645f7b00ba185e713915099853fd37 ./bin/gcov -37dd62589454703ae7f2eaac1668b66e ./bin/gcov-dump -f3dbc7e0c84a40194af3aa2429444e87 ./bin/gcov-tool -c9b0dfcbad566c0b8b88df94bb993312 ./bin/powerpc64-linux-gnu-c++ -c9b0dfcbad566c0b8b88df94bb993312 ./bin/powerpc64-linux-gnu-g++ -e4cc43b7790dcd25f31419bad606b36e ./bin/powerpc64-linux-gnu-gcc -e4cc43b7790dcd25f31419bad606b36e ./bin/powerpc64-linux-gnu-gcc-5.5.0 +a208bedbfca9c7bd6c27d0d42f7096fe ./bin/gcov +43330e8ae00976b4b3427d2f83b0725e ./bin/gcov-dump +9f37da5e96f147d733eb7195350ae5d2 ./bin/gcov-tool +8f02302b55643f1c711e472a42fea8bd ./bin/powerpc64-linux-gnu-c++ +8f02302b55643f1c711e472a42fea8bd ./bin/powerpc64-linux-gnu-g++ +583d1b011a7ba009d7385117dd7a33c8 ./bin/powerpc64-linux-gnu-gcc +583d1b011a7ba009d7385117dd7a33c8 ./bin/powerpc64-linux-gnu-gcc-5.5.0 f9d94f4bb61f70d14ea4b2ce73c9be9d ./bin/powerpc64-linux-gnu-gcc-ar 01fc2184f99c558771aa8f2fe30b373d ./bin/powerpc64-linux-gnu-gcc-nm da5356ee09ccda4ca06758d056370f7e ./bin/powerpc64-linux-gnu-gcc-ranlib -6ed530d13e65c3500b7e7b7cc863afdc ./libexec/gcc/powerpc64-linux-gnu/5.5.0/cc1 -24a83af179ca8849da8c64aa854ec8ed ./libexec/gcc/powerpc64-linux-gnu/5.5.0/cc1plus -0c05b45bb926a06c2b09acdb1db9aad0 ./libexec/gcc/powerpc64-linux-gnu/5.5.0/collect2 +22b72247a5706f090505341263ca1fc2 ./libexec/gcc/powerpc64-linux-gnu/5.5.0/cc1 +3be618d184038dd30011d6aa8198c0be ./libexec/gcc/powerpc64-linux-gnu/5.5.0/cc1plus +2f31e84c01cc087318d0c7f15b6e3f47 ./libexec/gcc/powerpc64-linux-gnu/5.5.0/collect2 417a5b42a26275b2c912db16b9abf73a ./libexec/gcc/powerpc64-linux-gnu/5.5.0/install-tools/fixincl fd6f80ec9089ddf51f9cac26299e45af ./libexec/gcc/powerpc64-linux-gnu/5.5.0/install-tools/fixinc.sh a585abbd6a9cdc474564b54fc72e4efa ./libexec/gcc/powerpc64-linux-gnu/5.5.0/install-tools/mkheaders 5071acceb24c0c0e8a423286205ed54c ./libexec/gcc/powerpc64-linux-gnu/5.5.0/install-tools/mkinstalldirs -4e77b773ac45ce8f82a4d21a34063920 ./libexec/gcc/powerpc64-linux-gnu/5.5.0/lto-wrapper +5267311e0ed8bb5358e5556dfa205ca6 ./libexec/gcc/powerpc64-linux-gnu/5.5.0/lto-wrapper a90ab86f837913280f72beb5310714bc ./lib/gcc/powerpc64-linux-gnu/5.5.0/crtbegin.o 0b38aa831c40b6bc4fb22248746d60d8 ./lib/gcc/powerpc64-linux-gnu/5.5.0/crtbeginS.o 8f62d8795bebd8e87caa3640522ff93b ./lib/gcc/powerpc64-linux-gnu/5.5.0/crtbeginT.o --8<---------------cut here---------------end--------------->8--- These appear to be the same diffs that I reported before - with the notable exception that libstdc++.a now is missing from the list of differing files (hooray!). Going forward, I'm not sure how best to investigate the inputs to find out what's causing the differences. I just know I really, really, really don't want to rebuild everything multiple times, since it takes hours/days. If you have any creative ideas for how to speed up the investigation, I'm all ears. -- Chris