unofficial mirror of bug-guix@gnu.org 
 help / color / mirror / Atom feed
* bug#41669: Cross-compiled powerpc64-linux bootstrap-tarballs not reproducible
@ 2020-06-02 18:59 Chris Marusich
  2020-06-03  9:48 ` Chris Marusich
  0 siblings, 1 reply; 9+ messages in thread
From: Chris Marusich @ 2020-06-02 18:59 UTC (permalink / raw)
  To: 41669

[-- Attachment #1: Type: text/plain, Size: 1810 bytes --]

Hi,

As demonstrated in the following email thread, the powerpc64-linux
bootstrap-tarballs are not reproducible when cross-compiled from an
x86_64-linux system:

https://lists.gnu.org/archive/html/guix-devel/2020-06/msg00003.html

Four people attempted to invoke the following command using Guix commit
8159ce1970d91567468cf1bacac313099a009d2a:

  guix build --no-substitutes --target=powerpc64-linux-gnu bootstrap-tarballs

All of the bootstrap tarballs except for gcc-stripped were reproducible.
However, these attempts produced four different versions of
gcc-stripped-5.5.0-powerpc64-linux-gnu.tar.xz.  At least two of the
attempts were performed on two different x86_64-linux systems.

The derivation that produced the differing output was:

/gnu/store/pygln3lr6qbxcps3kmn3w4bc0d0nlpd3-gcc-stripped-tarball-5.5.0.drv

You can build this derivation in a variety of ways, for example:

  guix build --target=powerpc64-linux-gnu -e '(@ (gnu packages make-bootstrap) %gcc-bootstrap-tarball)'

or just

  guix build /gnu/store/pygln3lr6qbxcps3kmn3w4bc0d0nlpd3-gcc-stripped-tarball-5.5.0.drv

On my x86_64-linux system, twice I tried running "guix build --check" on
this derivation, but each time it surprisingly reported no difference.
Out of paranoia, I tried deleting its output with "guix gc --delete",
and then building it again.  The new output was, indeed, identical to
the old output.  This makes me think that the non-reproducibility is
coming from something outside the immediate build logic of the
gcc-stripped-tarball-5.5.0.drv derivation itself.

I will next try to rebuild everything from scratch again.  I will also
try to get copies of the differing outputs from the people involved in
that email thread, in order to run diffoscope on them.

-- 
Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

^ permalink raw reply	[flat|nested] 9+ messages in thread

* bug#41669: Cross-compiled powerpc64-linux bootstrap-tarballs not reproducible
  2020-06-02 18:59 bug#41669: Cross-compiled powerpc64-linux bootstrap-tarballs not reproducible Chris Marusich
@ 2020-06-03  9:48 ` Chris Marusich
  2020-06-03 20:50   ` Vincent Legoll
                     ` (2 more replies)
  0 siblings, 3 replies; 9+ messages in thread
From: Chris Marusich @ 2020-06-03  9:48 UTC (permalink / raw)
  To: 41669; +Cc: Vincent Legoll, Léo Le Bouter, Maxim Cournoyer

[-- Attachment #1: Type: text/plain, Size: 4040 bytes --]

Hi all,

Chris Marusich <cmmarusich@gmail.com> writes:

> The derivation that produced the differing output was:
>
> /gnu/store/pygln3lr6qbxcps3kmn3w4bc0d0nlpd3-gcc-stripped-tarball-5.5.0.drv
>
> On my x86_64-linux system, twice I tried running "guix build --check" on
> this derivation, but each time it surprisingly reported no difference.

This derivation corresponds to %gcc-bootstrap-tarball from (gnu packages
make-bootstrap), which just creates a tarball of the output of
%gcc-stripped.  Therefore, it's not too surprising that it's
reproducible.  Similarly, %gcc-stripped just strips some store
references from the output of %gcc-static.  The %gcc-static package is
more interesting: it's where we actually build the statically linked
bootstrap GCC (which is then stripped and packed into a tarball as
mentioned above), so I thought that it might be where the
non-determinism is coming from.  However, I haven't yet pinpointed the
problem.

If you examine the derivations and their inputs, you'll find that they
depend upon each other in the following order:

guix build --target=powerpc64-linux-gnu -d -e '(@ (gnu packages make-bootstrap) %gcc-bootstrap-tarball)'
/gnu/store/pygln3lr6qbxcps3kmn3w4bc0d0nlpd3-gcc-stripped-tarball-5.5.0.drv

guix build --target=powerpc64-linux-gnu -d -e '(@@ (gnu packages make-bootstrap) %gcc-stripped)'
/gnu/store/kcv3ja1rfr93hw6ly51878zjhdwpgv7z-gcc-stripped-5.5.0.drv

guix build --target=powerpc64-linux-gnu -d -e '(@@ (gnu packages make-bootstrap) %gcc-static)'
/gnu/store/m9hfwppla8lph0vxa15lfkp81s2bbjjs-gcc-static-5.5.0.drv

In other words, gcc-static-5.5.0.drv is an input of
gcc-stripped-5.5.0.drv, which is an input of
gcc-stripped-tarball-5.5.0.drv.  Above, I've included example guix
commands you can use to obtain each derivation.  Using "guix build
--check", I confirmed that all three of these derivations build
reproducibly on my machine.

I hoped to find more information by invoking "guix build --check" on
every input of gcc-static-5.5.0.drv.  When I tried that, what I found
was that all of its inputs build reproducibly except the following two:

/gnu/store/x32cnfkd50fnxs10xp1jdn24h7ai2gxr-guile-3.0.2.drv
/gnu/store/g9fpkg2qa27mka1znqsvx8vxqyabsj2y-gcc-7.5.0.drv

I haven't investigated guile-3.0.2.drv.  However, gcc-7.5.0.drv felt
more suspicious to me, and it is actually the derivation that builds
gcc-final from (gnu packages commencement), which you can see via:

guix build -d -e '(@@ (gnu packages commencement) gcc-final)'

Using "guix gc", I deleted the outputs of
gcc-stripped-tarball-5.5.0.drv, gcc-stripped-5.5.0.drv,
gcc-static-5.5.0.drv, and gcc-7.5.0.drv.  I then tried building these
four derivations again (without substitutes, the same as before).
Before doing this, I stored the SHA-512 hashes of their output files,
and after the build succeeded, I compared the hashes of the new files
with the previous values.  I found that these derivations' newly rebuilt
outputs were identical to their original values, except for
gcc-7.5.0.drv, which produced some different files.  Most significantly,
this means that gcc-stripped-tarball-5.5.0.drv produced the exact same
tarball for me as it did the first time I built it, even though some of
its inputs are not themselves reproducible.

At present, it seems possible that within the context of a single
machine, gcc-stripped-tarball-5.5.0.drv builds reproducibly, but on a
different machine, it may (reproducibly) build a different output.  I'm
a bit paranoid about making mistakes, so I'll perform another full GC
and then try yet again to build gcc-stripped-tarball-5.5.0.drv in order
to verify whether it truly produces the same output when all (or nearly
all) of its inputs are rebuilt from scratch.

Some people have also shared their differing copies of the binaries on
the email list.  It could be productive to compare the contents with
diffoscope, although I suspect the diff might be too large to be useful.

-- 
Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

^ permalink raw reply	[flat|nested] 9+ messages in thread

* bug#41669: Cross-compiled powerpc64-linux bootstrap-tarballs not reproducible
  2020-06-03  9:48 ` Chris Marusich
@ 2020-06-03 20:50   ` Vincent Legoll
  2020-06-10  6:15   ` Chris Marusich
  2020-09-13  2:53   ` Chris Marusich
  2 siblings, 0 replies; 9+ messages in thread
From: Vincent Legoll @ 2020-06-03 20:50 UTC (permalink / raw)
  To: Chris Marusich, 41669; +Cc: Léo Le Bouter, Maxim Cournoyer

Is that showing the same (or a similar) problem :

https://data.guix-patches.cbaines.net/gnu/store/0lcbxpw1vrca02dzpzw2rxhad7pn4zw7-gcc-objc-5.5.0

?

-- 
Vincent Legoll





^ permalink raw reply	[flat|nested] 9+ messages in thread

* bug#41669: Cross-compiled powerpc64-linux bootstrap-tarballs not reproducible
  2020-06-03  9:48 ` Chris Marusich
  2020-06-03 20:50   ` Vincent Legoll
@ 2020-06-10  6:15   ` Chris Marusich
  2020-06-10 22:20     ` Bengt Richter
  2020-06-11 21:09     ` Jack Hill
  2020-09-13  2:53   ` Chris Marusich
  2 siblings, 2 replies; 9+ messages in thread
From: Chris Marusich @ 2020-06-10  6:15 UTC (permalink / raw)
  To: 41669; +Cc: Vincent Legoll, Léo Le Bouter, Maxim Cournoyer

[-- Attachment #1: Type: text/plain, Size: 26804 bytes --]

Hi Vincent and everyone,

Vincent Legoll <vincent.legoll@gmail.com> writes:

> Is that showing the same (or a similar) problem :
>
> https://data.guix-patches.cbaines.net/gnu/store/0lcbxpw1vrca02dzpzw2rxhad7pn4zw7-gcc-objc-5.5.0
>
> ?

Can you clarify what you mean?  I'm not sure what you're referring to.

Chris Marusich <cmmarusich@gmail.com> writes:

> At present, it seems possible that within the context of a single
> machine, gcc-stripped-tarball-5.5.0.drv builds reproducibly, but on a
> different machine, it may (reproducibly) build a different output.
> I'm a bit paranoid about making mistakes, so I'll perform another full
> GC and then try yet again to build gcc-stripped-tarball-5.5.0.drv in
> order to verify whether it truly produces the same output when all (or
> nearly all) of its inputs are rebuilt from scratch.

I repeated the experiment on the same machine (it took a day or two to
build), and the result was the same: on my machine,
gcc-stripped-tarball-5.5.0.drv builds identical output to what it built
before. To be clear, using Guix 8159ce1970d91567468cf1bacac313099a009d2a
on an x86_64-linux machine, I tried (yet again) the following steps:

- I deleted as much as I could from the store using 'guix gc'.

- I explicitly verified that all outputs for the following derivations
  no longer existed in the store:

  /gnu/store/pygln3lr6qbxcps3kmn3w4bc0d0nlpd3-gcc-stripped-tarball-5.5.0.drv
  /gnu/store/kcv3ja1rfr93hw6ly51878zjhdwpgv7z-gcc-stripped-5.5.0.drv
  /gnu/store/m9hfwppla8lph0vxa15lfkp81s2bbjjs-gcc-static-5.5.0.drv
  /gnu/store/g9fpkg2qa27mka1znqsvx8vxqyabsj2y-gcc-7.5.0.drv

- I then built gcc-stripped-tarball-5.5.0.drv via the following command:

  guix build --no-substitutes --check --target=powerpc64-linux-gnu \
       -e '(@@ (gnu packages make-bootstrap) %gcc-static)'

This rebuilt basically everything, including gcc-7.5.0.drv and the other
derivations mentioned above.  When I checked the built output of the
gcc-stripped-tarball-5.5.0.drv derivation, I found that its SHA-512 hash
was identical to the one I calculated previously, which was:

--8<---------------cut here---------------start------------->8---
8aca7f332a1ba8e3c2225c161a7545b0a04ddd690d164dc97afee9c9ea067b0c49bc155e9f06d285c22e24cdd16d91e59730af5f1dd9efcda13a26bede5948a2  /gnu/store/rsmhiyplmbiqm1qwniiafi4ak76pd61v-gcc-stripped-tarball-5.5.0/gcc-stripped-5.5.0-powerpc64-linux-gnu.tar.xz
--8<---------------cut here---------------end--------------->8---

Anyway, this confirms what we already knew: GCC builds reproducibly on
my machine, but it is different when other people build it on other
machines.  I'm now quite convinced of this fact.

Jack and Vincent shared their build results on the email list.  For
reference, you can get them here:

https://flashner.co.il/~efraim/gcc-stripped-5.5.0-powerpc64-linux-gnu.tar.xz
https://jackhill.us/misc/gcc-stripped-5.5.0-powerpc64-linux-gnu.tar.xz
https://media.marusich.info/gcc-stripped-5.5.0-powerpc64-linux-gnu.tar.xz

I also received a copy of Vincent's build results via private email.  In
short, they all seem to differ in basically the same ways:

--8<---------------cut here---------------start------------->8---
[1] marusich@garuda-lan:~/Documents/notes/guix/ppc/gcc-stripped-tarballs
$ diff -r chris jack
Binary files chris/bin/c++ and jack/bin/c++ differ
Binary files chris/bin/cpp and jack/bin/cpp differ
Binary files chris/bin/g++ and jack/bin/g++ differ
Binary files chris/bin/gcc and jack/bin/gcc differ
Binary files chris/bin/gcov and jack/bin/gcov differ
Binary files chris/bin/gcov-dump and jack/bin/gcov-dump differ
Binary files chris/bin/gcov-tool and jack/bin/gcov-tool differ
Binary files chris/bin/powerpc64-linux-gnu-c++ and jack/bin/powerpc64-linux-gnu-c++ differ
Binary files chris/bin/powerpc64-linux-gnu-g++ and jack/bin/powerpc64-linux-gnu-g++ differ
Binary files chris/bin/powerpc64-linux-gnu-gcc and jack/bin/powerpc64-linux-gnu-gcc differ
Binary files chris/bin/powerpc64-linux-gnu-gcc-5.5.0 and jack/bin/powerpc64-linux-gnu-gcc-5.5.0 differ
Binary files chris/lib/libstdc++.a and jack/lib/libstdc++.a differ
Binary files chris/libexec/gcc/powerpc64-linux-gnu/5.5.0/cc1 and jack/libexec/gcc/powerpc64-linux-gnu/5.5.0/cc1 differ
Binary files chris/libexec/gcc/powerpc64-linux-gnu/5.5.0/cc1plus and jack/libexec/gcc/powerpc64-linux-gnu/5.5.0/cc1plus differ
Binary files chris/libexec/gcc/powerpc64-linux-gnu/5.5.0/collect2 and jack/libexec/gcc/powerpc64-linux-gnu/5.5.0/collect2 differ
Binary files chris/libexec/gcc/powerpc64-linux-gnu/5.5.0/lto-wrapper and jack/libexec/gcc/powerpc64-linux-gnu/5.5.0/lto-wrapper differ
[1] marusich@garuda-lan:~/Documents/notes/guix/ppc/gcc-stripped-tarballs
$ diff -r chris vincent
Binary files chris/bin/c++ and vincent/bin/c++ differ
Binary files chris/bin/cpp and vincent/bin/cpp differ
Binary files chris/bin/g++ and vincent/bin/g++ differ
Binary files chris/bin/gcc and vincent/bin/gcc differ
Binary files chris/bin/gcov and vincent/bin/gcov differ
Binary files chris/bin/gcov-dump and vincent/bin/gcov-dump differ
Binary files chris/bin/gcov-tool and vincent/bin/gcov-tool differ
Binary files chris/bin/powerpc64-linux-gnu-c++ and vincent/bin/powerpc64-linux-gnu-c++ differ
Binary files chris/bin/powerpc64-linux-gnu-g++ and vincent/bin/powerpc64-linux-gnu-g++ differ
Binary files chris/bin/powerpc64-linux-gnu-gcc and vincent/bin/powerpc64-linux-gnu-gcc differ
Binary files chris/bin/powerpc64-linux-gnu-gcc-5.5.0 and vincent/bin/powerpc64-linux-gnu-gcc-5.5.0 differ
Binary files chris/lib/libstdc++.a and vincent/lib/libstdc++.a differ
Binary files chris/libexec/gcc/powerpc64-linux-gnu/5.5.0/cc1 and vincent/libexec/gcc/powerpc64-linux-gnu/5.5.0/cc1 differ
Binary files chris/libexec/gcc/powerpc64-linux-gnu/5.5.0/cc1plus and vincent/libexec/gcc/powerpc64-linux-gnu/5.5.0/cc1plus differ
Binary files chris/libexec/gcc/powerpc64-linux-gnu/5.5.0/collect2 and vincent/libexec/gcc/powerpc64-linux-gnu/5.5.0/collect2 differ
Binary files chris/libexec/gcc/powerpc64-linux-gnu/5.5.0/lto-wrapper and vincent/libexec/gcc/powerpc64-linux-gnu/5.5.0/lto-wrapper differ
[1] marusich@garuda-lan:~/Documents/notes/guix/ppc/gcc-stripped-tarballs
$ diff -r chris efraim
Binary files chris/bin/c++ and efraim/bin/c++ differ
Binary files chris/bin/cpp and efraim/bin/cpp differ
Binary files chris/bin/g++ and efraim/bin/g++ differ
Binary files chris/bin/gcc and efraim/bin/gcc differ
Binary files chris/bin/gcov and efraim/bin/gcov differ
Binary files chris/bin/gcov-dump and efraim/bin/gcov-dump differ
Binary files chris/bin/gcov-tool and efraim/bin/gcov-tool differ
Binary files chris/bin/powerpc64-linux-gnu-c++ and efraim/bin/powerpc64-linux-gnu-c++ differ
Binary files chris/bin/powerpc64-linux-gnu-g++ and efraim/bin/powerpc64-linux-gnu-g++ differ
Binary files chris/bin/powerpc64-linux-gnu-gcc and efraim/bin/powerpc64-linux-gnu-gcc differ
Binary files chris/bin/powerpc64-linux-gnu-gcc-5.5.0 and efraim/bin/powerpc64-linux-gnu-gcc-5.5.0 differ
diff -r chris/lib/gcc/powerpc64-linux-gnu/5.5.0/include-fixed/bits/statx.h efraim/lib/gcc/powerpc64-linux-gnu/5.5.0/include-fixed/bits/statx.h
5c5
< 	"/gnu/store/fa6wj5bxkj5ll1d7292a70knmyl7a0cr-glibc-2.31/include/bits/statx.h"
---
> 	"/gnu/store/cb88z63hyg1icd2kkahiink2p291mhr2-glibc-2.31/include/bits/statx.h"
Binary files chris/lib/libstdc++.a and efraim/lib/libstdc++.a differ
Binary files chris/libexec/gcc/powerpc64-linux-gnu/5.5.0/cc1 and efraim/libexec/gcc/powerpc64-linux-gnu/5.5.0/cc1 differ
Binary files chris/libexec/gcc/powerpc64-linux-gnu/5.5.0/cc1plus and efraim/libexec/gcc/powerpc64-linux-gnu/5.5.0/cc1plus differ
Binary files chris/libexec/gcc/powerpc64-linux-gnu/5.5.0/collect2 and efraim/libexec/gcc/powerpc64-linux-gnu/5.5.0/collect2 differ
Binary files chris/libexec/gcc/powerpc64-linux-gnu/5.5.0/lto-wrapper and efraim/libexec/gcc/powerpc64-linux-gnu/5.5.0/lto-wrapper differ
[1] marusich@garuda-lan:~/Documents/notes/guix/ppc/gcc-stripped-tarballs
$ 
--8<---------------cut here---------------end--------------->8---

Efraim's diff looks a little different in statx.h, even though he used
the same Guix commit as me.  Maybe this is because he cross-compiled on
an aarch64-linux machine, while I cross-compiled on an x86_64-linux
machine.  In the other cases, it looks like the binary files differ in
basically the same ways.  I will share some examples below.

Here is some diffoscope output between my c++ and Efraim's (many other
sections also differed in similarly cryptic ways):

--8<---------------cut here---------------start------------->8---
diffoscope chris/bin/c++ efraim/bin/c++
...
├── /gnu/store/xakj5dgs1729297nv50s84sdmq2jiz64-binutils-2.34/bin/readelf --wide --symbols {}
│ @@ -328,18 +328,18 @@
│     324: 00000000101aece0   152 FUNC    LOCAL  DEFAULT   19 check_free.isra.0
│     325: 00000000101aed28    24 FUNC    LOCAL  DEFAULT   19 fini
│     326: 00000000101b6008     8 OBJECT  LOCAL  DEFAULT   24 static_buf
│     327: 00000000101b6010    32 OBJECT  LOCAL  DEFAULT   24 last_result
│     328: 00000000101b6030     4 OBJECT  LOCAL  DEFAULT   24 key
│     329: 00000000101b6034     4 OBJECT  LOCAL  DEFAULT   24 once
│     330: 00000000101b23b8   104 OBJECT  LOCAL  DEFAULT   22 _dlfcn_hooks
│ -   331: 00000000101a98f8   524 FUNC    LOCAL  DEFAULT   19 _ZN12_GLOBAL__N_14pool4freeEPv.constprop.2
│ -   332: 00000000101a9910   524 FUNC    LOCAL  DEFAULT   19 _ZN12_GLOBAL__N_14pool8allocateEm.constprop.3
│ -   333: 00000000101a99a0   180 FUNC    LOCAL  DEFAULT   19 _GLOBAL__sub_I_eh_alloc.cc
│ -   334: 00000000101b4f68    64 OBJECT  LOCAL  DEFAULT   24 _ZN12_GLOBAL__N_114emergency_poolE
│ +   331: 00000000101a9880   524 FUNC    LOCAL  DEFAULT   19 _ZN12_GLOBAL__N_14pool4freeEPv.constprop.2
│ +   332: 00000000101a9898   524 FUNC    LOCAL  DEFAULT   19 _ZN12_GLOBAL__N_14pool8allocateEm.constprop.3
│ +   333: 00000000101a9928   180 FUNC    LOCAL  DEFAULT   19 _GLOBAL__sub_I_eh_alloc.cc
│ +   334: 00000000101b4f78    64 OBJECT  LOCAL  DEFAULT   24 _ZN12_GLOBAL__N_114emergency_poolE
│     335: 000000001011c5a8    19 OBJECT  LOCAL  DEFAULT    7 _ZZ18ggc_internal_allocmPFvPvEmmE12__FUNCTION__
│     336: 000000001011c5c0    27 OBJECT  LOCAL  DEFAULT    7 _ZZ26ggc_internal_cleared_allocmPFvPvEmmE12__FUNCTION__
│     337: 000000001011c600    21 OBJECT  LOCAL  DEFAULT    7 _ZZ20lang_specific_driverPP17cl_decoded_optionPjPiE12__FUNCTION__
│     338: 00000000101a42d0   228 FUNC    LOCAL  DEFAULT   19 _ZL12elf_platformv
│     339: 00000000101a42e8   236 FUNC    LOCAL  DEFAULT   19 _ZL15elf_dcachebsizev
│     340: 00000000101a4300   164 FUNC    LOCAL  DEFAULT   19 _ZL14describe_cachejjjj.isra.0.constprop.1
│     341: 000000001011c6b8   752 OBJECT  LOCAL  DEFAULT    7 _ZL9asm_names
...
│ @@ -1379,25 +1379,25 @@
│    1375: 00000000101b2448     0 SECTION LOCAL  DEFAULT   23 
│    1376: 00000000101b2b80     0 SECTION LOCAL  DEFAULT   24 
│    1377: 00000000101b6860     0 SECTION LOCAL  DEFAULT   25 
│    1378: 0000000000000000     0 SECTION LOCAL  DEFAULT   26 
│    1379: 0000000000000000     0 SECTION LOCAL  DEFAULT   27 
│    1380: 00000000101a2a28   752 OBJECT  GLOBAL DEFAULT   18 _nl_C_LC_CTYPE
│    1381: 00000000101ace20    56 IFUNC   GLOBAL DEFAULT   19 __libc_strcspn
│ -  1382: 00000000101a9958   100 FUNC    GLOBAL DEFAULT   19 __cxa_free_exception
│ +  1382: 00000000101a98e0   100 FUNC    GLOBAL DEFAULT   19 __cxa_free_exception
│    1383: 00000000101acfe8   104 FUNC    WEAK   DEFAULT   19 stpcpy
│    1384: 00000000101ab7e8   516 FUNC    GLOBAL DEFAULT   19 putchar
│ -  1385: 00000000101a9598    20 FUNC    GLOBAL DEFAULT   19 _ZNKSt9type_info15__is_function_pEv
│ +  1385: 00000000101a9778    20 FUNC    GLOBAL DEFAULT   19 _ZNKSt9type_info15__is_function_pEv
│    1386: 000000001016a3cc    76 OBJECT  GLOBAL DEFAULT    7 _nl_C_LC_CTYPE_class_print
│    1387: 00000000101adfc0   640 FUNC    WEAK   DEFAULT   19 tsearch
│    1388: 00000000101af0a0   176 FUNC    WEAK   DEFAULT   19 clock_gettime
│    1389: 00000000101a4150  4856 FUNC    GLOBAL DEFAULT   19 _ZNK6driver12set_up_specsEv
│    1390: 00000000101a9268    88 FUNC    GLOBAL DEFAULT   19 xmalloc
│    1391: 00000000101b1d68     8 OBJECT  GLOBAL DEFAULT   22 __morecore
│ -  1392: 00000000101a9568    40 FUNC    GLOBAL DEFAULT   19 _ZSt10unexpectedv
│ +  1392: 00000000101a94f0    40 FUNC    GLOBAL DEFAULT   19 _ZSt10unexpectedv
│    1393: 00000000101a8ef0    60 FUNC    GLOBAL DEFAULT   19 lbasename
│    1394: 00000000101aded0    96 FUNC    GLOBAL DEFAULT   19 __getdtablesize
│    1395: 00000000101a4348   752 FUNC    GLOBAL DEFAULT   19 update_path
...
├── /gnu/store/xakj5dgs1729297nv50s84sdmq2jiz64-binutils-2.34/bin/readelf --wide --decompress --hex-dump=.data.rel.ro {}
│ @@ -112,27 +112,27 @@
│    0x101a1718 00000000 10164560 00000009 00000001 ......E`........
│    0x101a1728 00000000 10164570 00000000 10164578 ......Ep......Ex
│    0x101a1738 0000000c 00000002 00000000 101499e8 ................
│    0x101a1748 00000000 10116018 00000001 00000001 ......`.........
│    0x101a1758 00000000 10164588 00000000 10164590 ......E.......E.
│    0x101a1768 00000007 00000001 00000000 10119850 ...............P
│    0x101a1778 00000000 1015e290 00000002 00000002 ................
│ -  0x101a1788 00000000 101641c0 00000000 101653e8 ......A.......S.
│ +  0x101a1788 00000000 101641c0 00000000 101653a8 ......A.......S.
│    0x101a1798 00000001 00000002 00000000 10164598 ..............E.
│    0x101a17a8 00000000 10117df0 00000001 00000002 ......}.........
│    0x101a17b8 00000000 101645a0 00000000 1015e238 ......E........8
│    0x101a17c8 00000002 00000002 00000000 101645a8 ..............E.
│    0x101a17d8 00000000 101631d8 00000001 00000002 ......1.........
│    0x101a17e8 00000000 101645b0 00000000 1015e1e0 ......E.........
│ -  0x101a17f8 00000002 00000002 00000000 101653c0 ..............S.
│ +  0x101a17f8 00000002 00000002 00000000 10165380 ..............S.
│    0x101a1808 00000000 1015e268 00000003 00000003 .......h........
│ -  0x101a1818 00000000 101653c8 00000000 1015e268 ......S........h
│ -  0x101a1828 00000003 00000003 00000000 101653d0 ..............S.
│ +  0x101a1818 00000000 10165388 00000000 1015e268 ......S........h
│ +  0x101a1828 00000003 00000003 00000000 10165390 ..............S.
│    0x101a1838 00000000 1015e268 00000003 00000002 .......h........
│ -  0x101a1848 00000000 101653d8 00000000 1015e268 ......S........h
│ +  0x101a1848 00000000 10165398 00000000 1015e268 ......S........h
│    0x101a1858 00000003 00000002 00000000 10144ea8 ..............N.
│    0x101a1868 00000000 1015e1f0 00000002 00000002 ................
...
--8<---------------cut here---------------end--------------->8---

Here is some diffoscope output betweeen my c++ and Jack's (many other
sections also differed in similarly cryptic ways):

--8<---------------cut here---------------start------------->8---
diffoscope chris/bin/c++ jack/bin/c++
...
├── /gnu/store/xakj5dgs1729297nv50s84sdmq2jiz64-binutils-2.34/bin/readelf --wide --symbols {}
│ @@ -328,17 +328,17 @@
│     324: 00000000101aece0   152 FUNC    LOCAL  DEFAULT   19 check_free.isra.0
│     325: 00000000101aed28    24 FUNC    LOCAL  DEFAULT   19 fini
│     326: 00000000101b6008     8 OBJECT  LOCAL  DEFAULT   24 static_buf
│     327: 00000000101b6010    32 OBJECT  LOCAL  DEFAULT   24 last_result
│     328: 00000000101b6030     4 OBJECT  LOCAL  DEFAULT   24 key
│     329: 00000000101b6034     4 OBJECT  LOCAL  DEFAULT   24 once
│     330: 00000000101b23b8   104 OBJECT  LOCAL  DEFAULT   22 _dlfcn_hooks
│ -   331: 00000000101a98f8   524 FUNC    LOCAL  DEFAULT   19 _ZN12_GLOBAL__N_14pool4freeEPv.constprop.2
│ -   332: 00000000101a9910   524 FUNC    LOCAL  DEFAULT   19 _ZN12_GLOBAL__N_14pool8allocateEm.constprop.3
│ -   333: 00000000101a99a0   180 FUNC    LOCAL  DEFAULT   19 _GLOBAL__sub_I_eh_alloc.cc
│ +   331: 00000000101a94c0   524 FUNC    LOCAL  DEFAULT   19 _ZN12_GLOBAL__N_14pool4freeEPv.constprop.2
│ +   332: 00000000101a94d8   524 FUNC    LOCAL  DEFAULT   19 _ZN12_GLOBAL__N_14pool8allocateEm.constprop.3
│ +   333: 00000000101a9568   180 FUNC    LOCAL  DEFAULT   19 _GLOBAL__sub_I_eh_alloc.cc
│     334: 00000000101b4f68    64 OBJECT  LOCAL  DEFAULT   24 _ZN12_GLOBAL__N_114emergency_poolE
│     335: 000000001011c5a8    19 OBJECT  LOCAL  DEFAULT    7 _ZZ18ggc_internal_allocmPFvPvEmmE12__FUNCTION__
│     336: 000000001011c5c0    27 OBJECT  LOCAL  DEFAULT    7 _ZZ26ggc_internal_cleared_allocmPFvPvEmmE12__FUNCTION__
│     337: 000000001011c600    21 OBJECT  LOCAL  DEFAULT    7 _ZZ20lang_specific_driverPP17cl_decoded_optionPjPiE12__FUNCTION__
│     338: 00000000101a42d0   228 FUNC    LOCAL  DEFAULT   19 _ZL12elf_platformv
│     339: 00000000101a42e8   236 FUNC    LOCAL  DEFAULT   19 _ZL15elf_dcachebsizev
│     340: 00000000101a4300   164 FUNC    LOCAL  DEFAULT   19 _ZL14describe_cachejjjj.isra.0.constprop.1
...
├── /gnu/store/xakj5dgs1729297nv50s84sdmq2jiz64-binutils-2.34/bin/readelf --wide --decompress --hex-dump=.data.rel.ro {}
│ @@ -206,60 +206,60 @@
│    0x101a1cf8 00000000 101a3b08 00000000 10118108 ......;.........
│    0x101a1d08 00000000 101a4120 00000000 10118120 ......A ....... 
│    0x101a1d18 00000000 101a4108 00000000 10115758 ......A.......WX
│    0x101a1d28 00000000 101a3a60 00000000 10118138 ......:`.......8
│    0x101a1d38 00000000 101a3a48 00000000 10115ae8 ......:H......Z.
│    0x101a1d48 00000000 101a3c10 00000000 10118150 ......<........P
│    0x101a1d58 00000000 101a4318 00000000 00000000 ......C.........
│ -  0x101a1d68 00000000 00000000 00000000 101a1dd8 ................
│ -  0x101a1d78 00000000 101650d0 00000000 101a1e70 ......P........p
│ -  0x101a1d88 00000000 00000000 00000000 101a1d70 ...............p
│ -  0x101a1d98 00000000 101a9388 00000000 101a93a0 ................
│ -  0x101a1da8 00000000 101a9370 00000000 101a1dd8 .......p........
│ -  0x101a1db8 00000000 101650f0 00000000 101a1fd8 ......P.........
│ -  0x101a1dc8 00000000 00000000 00000000 101a1db0 ................
│ -  0x101a1dd8 00000000 101a93d0 00000000 101a93e8 ................
│ -  0x101a1de8 00000000 101a9598 00000000 101a9598 ................
│ -  0x101a1df8 00000000 101a9b20 00000000 101a9a90 ....... ........
│ -  0x101a1e08 00000000 101a9430 00000000 101a9418 .......0........
│ -  0x101a1e18 00000000 101a9400 00000000 101a2000 .............. .
│ -  0x101a1e28 00000000 10165118 00000000 00000000 ......Q.........
│ -  0x101a1e38 00000000 101a1e20 00000000 101a9580 ....... ........
│ -  0x101a1e48 00000000 101a95c8 00000000 101a9598 ................
│ -  0x101a1e58 00000000 101a9598 00000000 101a95e0 ................
│ -  0x101a1e68 00000000 101a95b0 00000000 101a2000 .............. .
│ -  0x101a1e78 00000000 10165128 00000000 101a1dd8 ......Q(........
│ -  0x101a1e88 00000000 10165138 00000000 101a1e70 ......Q8.......p
│ -  0x101a1e98 00000000 101a2000 00000000 10165150 ...... .......QP
│ -  0x101a1ea8 00000000 101a2000 00000000 10165170 ...... .......Qp
│ -  0x101a1eb8 00000000 00000000 00000000 101a1e70 ...............p
│ -  0x101a1ec8 00000000 101a96e8 00000000 101a9748 ...............H
│ -  0x101a1ed8 00000000 101a9718 00000000 00000000 ................
│ -  0x101a1ee8 00000000 101a1e80 00000000 101a9700 ................
│ -  0x101a1ef8 00000000 101a9760 00000000 101a9730 .......`.......0
│ -  0x101a1f08 00000000 00000000 00000000 101a1e98 ................
│ -  0x101a1f18 00000000 00000000 00000000 00000000 ................
│ -  0x101a1f28 00000000 101a99b8 00000000 00000000 ................
│ -  0x101a1f38 00000000 101a1ea8 00000000 00000000 ................
│ -  0x101a1f48 00000000 00000000 00000000 101a99b8 ................
│ -  0x101a1f58 00000000 101a1dd8 00000000 101651c0 ..............Q.
│ -  0x101a1f68 00000000 101a1e70 00000000 101a1dd8 .......p........
│ -  0x101a1f78 00000000 101651e8 00000000 101a1e70 ......Q........p
│ -  0x101a1f88 00000000 00000000 00000000 101a1f58 ...............X
│ -  0x101a1f98 00000000 101a9880 00000000 101a98b0 ................
│ -  0x101a1fa8 00000000 101a9850 00000000 00000000 .......P........
│ -  0x101a1fb8 00000000 101a1f70 00000000 101a9898 .......p........
│ -  0x101a1fc8 00000000 101a98c8 00000000 101a9868 ...............h
│ -  0x101a1fd8 00000000 101a1dd8 00000000 101652a8 ..............R.
│ -  0x101a1fe8 00000000 101a1e20 00000000 00000000 ....... ........
│ -  0x101a1ff8 00000000 101a1fd8 00000000 101a9ac0 ................
│ -  0x101a2008 00000000 101a9ad8 00000000 101a9598 ................
│ -  0x101a2018 00000000 101a9598 00000000 101a9b20 ............... 
│ -  0x101a2028 00000000 101a9a90 00000000 101a9b08 ................
│ -  0x101a2038 00000000 101a9af0 00000000 101a9aa8 ................
│ +  0x101a1d68 00000000 00000000 00000000 101a1f40 ...............@
│ +  0x101a1d78 00000000 101650d0 00000000 101a1e30 ......P........0
│ +  0x101a1d88 00000000 101a1f40 00000000 101650f8 .......@......P.
│ +  0x101a1d98 00000000 101a1e30 00000000 00000000 .......0........
│ +  0x101a1da8 00000000 101a1d70 00000000 101a9448 .......p.......H
│ +  0x101a1db8 00000000 101a9478 00000000 101a9418 .......x........
│ +  0x101a1dc8 00000000 00000000 00000000 101a1d88 ................
│ +  0x101a1dd8 00000000 101a9460 00000000 101a9490 .......`........
│ +  0x101a1de8 00000000 101a9430 00000000 101a1f40 .......0.......@
│ +  0x101a1df8 00000000 10165178 00000000 101a1e30 ......Qx.......0
│ +  0x101a1e08 00000000 00000000 00000000 101a1df0 ................
│ +  0x101a1e18 00000000 101a9610 00000000 101a9628 ...............(
│ +  0x101a1e28 00000000 101a95f8 00000000 101a1fb0 ................
│ +  0x101a1e38 00000000 10165198 00000000 101a1f40 ......Q........@
│ +  0x101a1e48 00000000 101651a8 00000000 101a1e30 ......Q........0
│ +  0x101a1e58 00000000 101a1fb0 00000000 101651c0 ..............Q.
│ +  0x101a1e68 00000000 101a1fb0 00000000 101651e0 ..............Q.
│ +  0x101a1e78 00000000 00000000 00000000 101a1e30 ...............0
│ +  0x101a1e88 00000000 101a9760 00000000 101a97c0 .......`........
│ +  0x101a1e98 00000000 101a9790 00000000 00000000 ................
│ +  0x101a1ea8 00000000 101a1e40 00000000 101a9778 .......@.......x
│ +  0x101a1eb8 00000000 101a97d8 00000000 101a97a8 ................
│ +  0x101a1ec8 00000000 00000000 00000000 101a1e58 ...............X
│ +  0x101a1ed8 00000000 00000000 00000000 00000000 ................
│ +  0x101a1ee8 00000000 101a98b0 00000000 00000000 ................
│ +  0x101a1ef8 00000000 101a1e68 00000000 00000000 .......h........
│ +  0x101a1f08 00000000 00000000 00000000 101a98b0 ................
│ +  0x101a1f18 00000000 101a1f40 00000000 10165270 .......@......Rp
│ +  0x101a1f28 00000000 101a1f88 00000000 00000000 ................
│ +  0x101a1f38 00000000 101a1f18 00000000 101a98e0 ................
│ +  0x101a1f48 00000000 101a98f8 00000000 101a9af0 ................
│ +  0x101a1f58 00000000 101a9af0 00000000 101a9ac0 ................
│ +  0x101a1f68 00000000 101a9a30 00000000 101a9940 .......0.......@
│ +  0x101a1f78 00000000 101a9928 00000000 101a9910 .......(........
│ +  0x101a1f88 00000000 101a1f40 00000000 10165298 .......@......R.
│ +  0x101a1f98 00000000 101a1ff8 00000000 00000000 ................
│ +  0x101a1fa8 00000000 101a1f88 00000000 101a9a60 ...............`
│ +  0x101a1fb8 00000000 101a9a78 00000000 101a9af0 .......x........
│ +  0x101a1fc8 00000000 101a9af0 00000000 101a9ac0 ................
│ +  0x101a1fd8 00000000 101a9a30 00000000 101a9aa8 .......0........
│ +  0x101a1fe8 00000000 101a9a90 00000000 101a9a48 ...............H
│ +  0x101a1ff8 00000000 101a1fb0 00000000 101652c0 ..............R.
│ +  0x101a2008 00000000 00000000 00000000 101a1ff8 ................
│ +  0x101a2018 00000000 101a9ad8 00000000 101a9b20 ............... 
│ +  0x101a2028 00000000 101a9af0 00000000 101a9af0 ................
│ +  0x101a2038 00000000 101a9b38 00000000 101a9b08 .......8........
│    0x101a2048 00000000 10165a28 00000000 101aa888 ......Z(........
│    0x101a2058 00000000 00000000 04040404 00000000 ................
│    0x101a2068 00000000 10165a38 00000000 101aa8a0 ......Z8........
│    0x101a2078 00000000 00000000 04040404 00000000 ................
│    0x101a2088 00000000 10165a58 00000000 101aa8b8 ......ZX........
│    0x101a2098 00000000 00000000 04040404 00000000 ................
│    0x101a20a8 00000000 10165a70 00000000 101aa8d0 ......Zp........
--8<---------------cut here---------------end--------------->8---

If I'm reading this correctly, one problem seems to be that our GCC
toolchains are putting symbols at different locations.  This issue (and
maybe others) could be trickling down, causing other aspects of the
binaries to differ (e.g., in length).  Nothing really stands out, but
when we discussed this on IRC, we thought perhaps factors like the
following might contribute to the non-reproducibility:

- Perhaps we are all running different Linux kernel versions?  In some
  cases, the kernel version can unfortunately influence the build
  output, so this might be worth testing.

- Perhaps the GCC Makefiles etc. are doing something non-deterministic?

- Something else?

Avenues of investigation:

- If anything obvious stands out from the diffoscope output, please
  leave a comment.

- Try building with different kernel versions on the same machine, to
  see if they differ.

- If somebody else could please confirm that running the following
  command reports no difference on their own machine (i.e., exit code
  0), that would be good to know, since it would help further solidify
  the theory that on a single machine, the build of gcc-static-5.5.0.drv
  is reproducible, even if it is not reproducible across machines:

  guix build --no-substitutes --check --target=powerpc64-linux-gnu \
       -e '(@@ (gnu packages make-bootstrap) %gcc-static)'

- Try building two different versions of gcc-7.5.0 (maybe by hand?), and
  then use them to build a simple reproduction case and compare results.
  If we're lucky, maybe this will help us understand the problem better.

We'll get there!

-- 
Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

^ permalink raw reply	[flat|nested] 9+ messages in thread

* bug#41669: Cross-compiled powerpc64-linux bootstrap-tarballs not reproducible
  2020-06-10  6:15   ` Chris Marusich
@ 2020-06-10 22:20     ` Bengt Richter
  2020-06-11 21:09     ` Jack Hill
  1 sibling, 0 replies; 9+ messages in thread
From: Bengt Richter @ 2020-06-10 22:20 UTC (permalink / raw)
  To: Chris Marusich; +Cc: 41669, Léo Le Bouter, Maxim Cournoyer, Vincent Legoll

Hi Chris, et al,

On +2020-06-09 23:15:01 -0700, Chris Marusich wrote:
> Hi Vincent and everyone,
> 
> Vincent Legoll <vincent.legoll@gmail.com> writes:
> 
> > Is that showing the same (or a similar) problem :
> >
> > https://data.guix-patches.cbaines.net/gnu/store/0lcbxpw1vrca02dzpzw2rxhad7pn4zw7-gcc-objc-5.5.0
> >
> > ?
> 
> Can you clarify what you mean?  I'm not sure what you're referring to.
> 
> Chris Marusich <cmmarusich@gmail.com> writes:
> 
> > At present, it seems possible that within the context of a single
> > machine, gcc-stripped-tarball-5.5.0.drv builds reproducibly, but on a
> > different machine, it may (reproducibly) build a different output.
> > I'm a bit paranoid about making mistakes, so I'll perform another full
> > GC and then try yet again to build gcc-stripped-tarball-5.5.0.drv in
> > order to verify whether it truly produces the same output when all (or
> > nearly all) of its inputs are rebuilt from scratch.
> 
> I repeated the experiment on the same machine (it took a day or two to
> build), and the result was the same: on my machine,
> gcc-stripped-tarball-5.5.0.drv builds identical output to what it built
> before. To be clear, using Guix 8159ce1970d91567468cf1bacac313099a009d2a
> on an x86_64-linux machine, I tried (yet again) the following steps:
>
[...]

> Efraim's diff looks a little different in statx.h, even though he used
> the same Guix commit as me.  Maybe this is because he cross-compiled on
> an aarch64-linux machine, while I cross-compiled on an x86_64-linux
> machine.  In the other cases, it looks like the binary files differ in
> basically the same ways.  I will share some examples below.
> 
> Here is some diffoscope output between my c++ and Efraim's (many other
> sections also differed in similarly cryptic ways):
> 
[...]

> 
> If I'm reading this correctly, one problem seems to be that our GCC
> toolchains are putting symbols at different locations.  This issue (and
> maybe others) could be trickling down, causing other aspects of the
> binaries to differ (e.g., in length).  Nothing really stands out, but
> when we discussed this on IRC, we thought perhaps factors like the
> following might contribute to the non-reproducibility:
> 
> - Perhaps we are all running different Linux kernel versions?  In some
>   cases, the kernel version can unfortunately influence the build
>   output, so this might be worth testing.
> 
> - Perhaps the GCC Makefiles etc. are doing something non-deterministic?
>

Questions triggered in my mind:

Where are respective machines getting their rules for packing and
aligning structs and unions?

Is any struct or rule/flags source dynamically generated, where different
rules could come from different defaults, or .configs, or even invalid
memoizations jumping domains?

Could pointer arithmetic get done in one domain and the offset be
misused in another? Wrong C preprocessor?

Difference in sort key comparisons for canonicalization of ordering?

Hope that's not all red herrings :)
Sorry for the noise otherwise.

> - Something else?

Hm, some race condition between processes that should be order-independent
but are not.

Then if different hardware components on different systems -- disks, memory,
processors -- cause different but repeatable patterns of waits (convoying?)
you could get repeatable but different builds.

I guess you'd have to figure out which order was really right, and force
the order of processing explicitly to that order, so all systems would
do it that way.

> 
> Avenues of investigation:
> 
> - If anything obvious stands out from the diffoscope output, please
>   leave a comment.
> 
> - Try building with different kernel versions on the same machine, to
>   see if they differ.
> 
> - If somebody else could please confirm that running the following
>   command reports no difference on their own machine (i.e., exit code
>   0), that would be good to know, since it would help further solidify
>   the theory that on a single machine, the build of gcc-static-5.5.0.drv
>   is reproducible, even if it is not reproducible across machines:
> 
>   guix build --no-substitutes --check --target=powerpc64-linux-gnu \
>        -e '(@@ (gnu packages make-bootstrap) %gcc-static)'
> 
> - Try building two different versions of gcc-7.5.0 (maybe by hand?), and
>   then use them to build a simple reproduction case and compare results.
>   If we're lucky, maybe this will help us understand the problem better.
> 
> We'll get there!
> 
> -- 
> Chris

-- 
Regards,
Bengt Richter




^ permalink raw reply	[flat|nested] 9+ messages in thread

* bug#41669: Cross-compiled powerpc64-linux bootstrap-tarballs not reproducible
  2020-06-10  6:15   ` Chris Marusich
  2020-06-10 22:20     ` Bengt Richter
@ 2020-06-11 21:09     ` Jack Hill
  1 sibling, 0 replies; 9+ messages in thread
From: Jack Hill @ 2020-06-11 21:09 UTC (permalink / raw)
  To: Chris Marusich; +Cc: 41669, Léo Le Bouter, Maxim Cournoyer, Vincent Legoll

On Tue, 9 Jun 2020, Chris Marusich wrote:

> - Try building with different kernel versions on the same machine, to
>  see if they differ.

I've done the rebuild after updated from Linux 5.4.41 to 5.4.45, and go 
identical results to my previous build. I am using a different kernel 
configuration than Guix's default kernel, but it was the same between the 
builds.

Best,
Jack




^ permalink raw reply	[flat|nested] 9+ messages in thread

* bug#41669: Cross-compiled powerpc64-linux bootstrap-tarballs not reproducible
  2020-06-03  9:48 ` Chris Marusich
  2020-06-03 20:50   ` Vincent Legoll
  2020-06-10  6:15   ` Chris Marusich
@ 2020-09-13  2:53   ` Chris Marusich
  2020-09-13  6:28     ` Efraim Flashner
  2 siblings, 1 reply; 9+ messages in thread
From: Chris Marusich @ 2020-09-13  2:53 UTC (permalink / raw)
  To: 41669; +Cc: Vincent Legoll, Léo Le Bouter, Maxim Cournoyer

[-- Attachment #1: Type: text/plain, Size: 3320 bytes --]

Hi everyone,

Chris Marusich <cmmarusich@gmail.com> writes:

> If you examine the derivations and their inputs, you'll find that they
> depend upon each other in the following order:
>
> guix build --target=powerpc64-linux-gnu -d -e '(@ (gnu packages make-bootstrap) %gcc-bootstrap-tarball)'
> /gnu/store/pygln3lr6qbxcps3kmn3w4bc0d0nlpd3-gcc-stripped-tarball-5.5.0.drv
>
> guix build --target=powerpc64-linux-gnu -d -e '(@@ (gnu packages make-bootstrap) %gcc-stripped)'
> /gnu/store/kcv3ja1rfr93hw6ly51878zjhdwpgv7z-gcc-stripped-5.5.0.drv
>
> guix build --target=powerpc64-linux-gnu -d -e '(@@ (gnu packages make-bootstrap) %gcc-static)'
> /gnu/store/m9hfwppla8lph0vxa15lfkp81s2bbjjs-gcc-static-5.5.0.drv
>
> In other words, gcc-static-5.5.0.drv is an input of
> gcc-stripped-5.5.0.drv, which is an input of
> gcc-stripped-tarball-5.5.0.drv.  Above, I've included example guix
> commands you can use to obtain each derivation.  Using "guix build
> --check", I confirmed that all three of these derivations build
> reproducibly on my machine.

After further experimentation, I've discovered that %gcc-static, when
built as shown above (without the -d option, of course), produces
different output on Debian than on Fedora.

Specifically, the %gcc-static output contains a file named libstdc++.a.
This file is an archive file.  Although its members are
content-identical in the case of Fedora and Debian, the order of the
members in the archive differs.  Because the exact same inputs were
used, it seems very likely that a difference in the Guix build
environment caused the %gcc-static build logic to order the members of
libstdc++.a differently.

I built %gcc-static using Guix commit
a02b2f8b86c0227eb69aa24b4373aef456365334.  Both Debian and Fedora were
x86_64-linux systems.  I took the following steps to make absolutely
certain that the exact same inputs were used on Debian and Fedora:

- I provisioned two fresh EC2 instances (Debian and Fedora).

- I installed Guix on Debian.

- I did "guix pull" on Debian to get to the aforementioned commit.

- I built %gcc-static on Debian as indicated above.

- I manually copied the Guix store and the Guix database from Debian to
  Fedora.

- I manually fixed up Fedora so it could run Guix (I created the guix
  users, added a systemd unit file, disabled selinux, etc.).

- I manually verified the Guix version and the store contents were
  identical on Fedora and Debian.

- I GC'd %gcc-static (and nothing else) on Fedora.

- I rebuilt %gcc-static on Fedora.

- I compared the Fedora %gcc-static output to the Debian %gcc-static
  output.

The %gcc-static package uses GCC 5.5.0 as its source.  I got a copy of
the GCC 5.5.0 source code, and I looked at it.  However, it's complex.
I can't pinpoint where they actually build the libstdc++.a file.  Can
anyone point me to the code that does this in the GCC 5.5.0 source?  I
expected to find the logic hiding in a makefile or a configure script or
something, but I haven't found it yet.

Since this is an old GCC, it is possible that this was a known
reproducibility bug which has since been fixed.  I haven't looked into
that possibility yet.  If that's the case, though, it would be nice
because we could simply backport a fix.

-- 
Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

^ permalink raw reply	[flat|nested] 9+ messages in thread

* bug#41669: Cross-compiled powerpc64-linux bootstrap-tarballs not reproducible
  2020-09-13  2:53   ` Chris Marusich
@ 2020-09-13  6:28     ` Efraim Flashner
  2020-09-26  6:52       ` Chris Marusich
  0 siblings, 1 reply; 9+ messages in thread
From: Efraim Flashner @ 2020-09-13  6:28 UTC (permalink / raw)
  To: Chris Marusich; +Cc: 41669, Léo Le Bouter, Maxim Cournoyer, Vincent Legoll

[-- Attachment #1: Type: text/plain, Size: 3867 bytes --]

On Sat, Sep 12, 2020 at 07:53:04PM -0700, Chris Marusich wrote:
> Hi everyone,
> 
> Chris Marusich <cmmarusich@gmail.com> writes:
> 
> > If you examine the derivations and their inputs, you'll find that they
> > depend upon each other in the following order:
> >
> > guix build --target=powerpc64-linux-gnu -d -e '(@ (gnu packages make-bootstrap) %gcc-bootstrap-tarball)'
> > /gnu/store/pygln3lr6qbxcps3kmn3w4bc0d0nlpd3-gcc-stripped-tarball-5.5.0.drv
> >
> > guix build --target=powerpc64-linux-gnu -d -e '(@@ (gnu packages make-bootstrap) %gcc-stripped)'
> > /gnu/store/kcv3ja1rfr93hw6ly51878zjhdwpgv7z-gcc-stripped-5.5.0.drv
> >
> > guix build --target=powerpc64-linux-gnu -d -e '(@@ (gnu packages make-bootstrap) %gcc-static)'
> > /gnu/store/m9hfwppla8lph0vxa15lfkp81s2bbjjs-gcc-static-5.5.0.drv
> >
> > In other words, gcc-static-5.5.0.drv is an input of
> > gcc-stripped-5.5.0.drv, which is an input of
> > gcc-stripped-tarball-5.5.0.drv.  Above, I've included example guix
> > commands you can use to obtain each derivation.  Using "guix build
> > --check", I confirmed that all three of these derivations build
> > reproducibly on my machine.
> 
> After further experimentation, I've discovered that %gcc-static, when
> built as shown above (without the -d option, of course), produces
> different output on Debian than on Fedora.
> 
> Specifically, the %gcc-static output contains a file named libstdc++.a.
> This file is an archive file.  Although its members are
> content-identical in the case of Fedora and Debian, the order of the
> members in the archive differs.  Because the exact same inputs were
> used, it seems very likely that a difference in the Guix build
> environment caused the %gcc-static build logic to order the members of
> libstdc++.a differently.
> 
> I built %gcc-static using Guix commit
> a02b2f8b86c0227eb69aa24b4373aef456365334.  Both Debian and Fedora were
> x86_64-linux systems.  I took the following steps to make absolutely
> certain that the exact same inputs were used on Debian and Fedora:
> 
> - I provisioned two fresh EC2 instances (Debian and Fedora).
> 
> - I installed Guix on Debian.
> 
> - I did "guix pull" on Debian to get to the aforementioned commit.
> 
> - I built %gcc-static on Debian as indicated above.
> 
> - I manually copied the Guix store and the Guix database from Debian to
>   Fedora.
> 
> - I manually fixed up Fedora so it could run Guix (I created the guix
>   users, added a systemd unit file, disabled selinux, etc.).
> 
> - I manually verified the Guix version and the store contents were
>   identical on Fedora and Debian.
> 
> - I GC'd %gcc-static (and nothing else) on Fedora.
> 
> - I rebuilt %gcc-static on Fedora.
> 
> - I compared the Fedora %gcc-static output to the Debian %gcc-static
>   output.
> 
> The %gcc-static package uses GCC 5.5.0 as its source.  I got a copy of
> the GCC 5.5.0 source code, and I looked at it.  However, it's complex.
> I can't pinpoint where they actually build the libstdc++.a file.  Can
> anyone point me to the code that does this in the GCC 5.5.0 source?  I
> expected to find the logic hiding in a makefile or a configure script or
> something, but I haven't found it yet.
> 
> Since this is an old GCC, it is possible that this was a known
> reproducibility bug which has since been fixed.  I haven't looked into
> that possibility yet.  If that's the case, though, it would be nice
> because we could simply backport a fix.
> 
> -- 
> Chris

Is this a file we actually need during the bootstrap process? Can we
"work around it" by just deleting it?


-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

^ permalink raw reply	[flat|nested] 9+ messages in thread

* bug#41669: Cross-compiled powerpc64-linux bootstrap-tarballs not reproducible
  2020-09-13  6:28     ` Efraim Flashner
@ 2020-09-26  6:52       ` Chris Marusich
  0 siblings, 0 replies; 9+ messages in thread
From: Chris Marusich @ 2020-09-26  6:52 UTC (permalink / raw)
  To: Efraim Flashner
  Cc: 41669, Léo Le Bouter, Maxim Cournoyer, Vincent Legoll


[-- Attachment #1.1: Type: text/plain, Size: 7951 bytes --]

Hi everyone,

Efraim Flashner <efraim@flashner.co.il> writes:

> Is this a file we actually need during the bootstrap process? Can we
> "work around it" by just deleting it?

That's a good idea.  I tried building %gcc-static with the
--disable-libstdcxx configure flag (see attached patch), and that caused
the build of %gcc-static itself to become reproducible on Debian and
Fedora when using exactly the same inputs.  To clarify, I have recently
run the following experiments:

+------+-------------+---------------------+--------------+----------------+---------------+
| Case |    Build    |      libstdcxx      | substitutes? |     Inputs     |    Result     |
+------+-------------+---------------------+--------------+----------------+---------------+
|      |             |                     |              | Built fresh on |    Output     |
|  1   | %gcc-static |       enabled       |     yes      | Debian, copied |   differs:    |
|      |             |                     |              |   to Fedora    |  libstdc++.a  |
+------+-------------+---------------------+--------------+----------------+---------------+
|      |             |                     |              | Re-used inputs |  Toy binary   |
|  2   | toy program |         n/a         |     yes      |   from above   |   does not    |
|      |             |                     |              |                |    differ     |
+------+-------------+---------------------+--------------+----------------+---------------+
|      |             |                     |              | Built fresh on |  Output does  |
|  3   | %gcc-static |      disabled       |     yes      | Debian, copied |  not differ   |
|      |             |                     |              |   to Fedora    |               |
+------+-------------+---------------------+--------------+----------------+---------------+
|      |             |                     |              | Built fresh on |    Output     |
|  4   | %gcc-static |      disabled       |     yes      | Debian, fresh  |   differs:    |
|      |             |                     |              |   on Fedora    | various files |
+------+-------------+---------------------+--------------+----------------+---------------+
|      |             |                     |              | Inputs failed  | Build failed  |
|  5   | %gcc-static |      disabled       |      no      |  to build on   |    on both    |
|      |             |                     |              |  both systems  |    systems    |
+------+-------------+---------------------+--------------+----------------+---------------+

The "toy program" in case 2 was just this:

  #include <stdio.h>
  int main() {
    printf("Hello");
    return 0;
  }

When I say I "copied" the inputs from Debian to Fedora, I mean just
that.  I copied stuff like /gnu and /var/guix from Debian to Fedora
(while guix-daemon was stopped, of course), GC'd just %gcc-static on
Fedora, and then rebuilt %gcc-static on Fedora so it would use exactly
the same inputs as were used on Debian.

The most notable new findings are cases 3 and 4.  Case 5 made me pretty
sad because I spent almost an entire weekend trying to build Guix from
source using various binary Guix releases, and in none of the attempts
was I successful in running "guix pull" or even just "guix environment
--pure guix" without substitutes, which really surprised me.

Case 3 confirms Efraim's suggestion: we can fix the libstdc++.a
reproducibility problem by simply not building libstdc++.a in the first
place. It also confirms that, when built with --disable-libstdcxx,
%gcc-static can be built reproducibly on different systems as long as
exactly the same inputs are used.  Whether or not %gcc-static can be
used to successfully bootstrap packages on powerpc64-linux when built in
this way remains to be seen.

Case 4, unfortunately, demonstrates that there are still other
reproducibility issues that we have not yet resolved.  Since the only
difference between cases 3 and 4 is how the inputs were realized, the
differing %gcc-static output must be caused by the inputs somehow.

In case 4, the %gcc-static output differed in the following files (the
hashes on the left are MD5 hashes):

--8<---------------cut here---------------start------------->8---
--- /dev/fd/63	2020-09-25 20:35:33.386554595 -0700
+++ /dev/fd/62	2020-09-25 20:35:33.387554604 -0700
@@ -1,28 +1,28 @@
-c9b0dfcbad566c0b8b88df94bb993312  ./bin/c++
-092823145dc96b9eb81111362f7b4ced  ./bin/cpp
-c9b0dfcbad566c0b8b88df94bb993312  ./bin/g++
-e4cc43b7790dcd25f31419bad606b36e  ./bin/gcc
+8f02302b55643f1c711e472a42fea8bd  ./bin/c++
+9f1fd993e4f2b796fcc56f0b2f8a47d2  ./bin/cpp
+8f02302b55643f1c711e472a42fea8bd  ./bin/g++
+583d1b011a7ba009d7385117dd7a33c8  ./bin/gcc
 f9d94f4bb61f70d14ea4b2ce73c9be9d  ./bin/gcc-ar
 01fc2184f99c558771aa8f2fe30b373d  ./bin/gcc-nm
 da5356ee09ccda4ca06758d056370f7e  ./bin/gcc-ranlib
-98645f7b00ba185e713915099853fd37  ./bin/gcov
-37dd62589454703ae7f2eaac1668b66e  ./bin/gcov-dump
-f3dbc7e0c84a40194af3aa2429444e87  ./bin/gcov-tool
-c9b0dfcbad566c0b8b88df94bb993312  ./bin/powerpc64-linux-gnu-c++
-c9b0dfcbad566c0b8b88df94bb993312  ./bin/powerpc64-linux-gnu-g++
-e4cc43b7790dcd25f31419bad606b36e  ./bin/powerpc64-linux-gnu-gcc
-e4cc43b7790dcd25f31419bad606b36e  ./bin/powerpc64-linux-gnu-gcc-5.5.0
+a208bedbfca9c7bd6c27d0d42f7096fe  ./bin/gcov
+43330e8ae00976b4b3427d2f83b0725e  ./bin/gcov-dump
+9f37da5e96f147d733eb7195350ae5d2  ./bin/gcov-tool
+8f02302b55643f1c711e472a42fea8bd  ./bin/powerpc64-linux-gnu-c++
+8f02302b55643f1c711e472a42fea8bd  ./bin/powerpc64-linux-gnu-g++
+583d1b011a7ba009d7385117dd7a33c8  ./bin/powerpc64-linux-gnu-gcc
+583d1b011a7ba009d7385117dd7a33c8  ./bin/powerpc64-linux-gnu-gcc-5.5.0
 f9d94f4bb61f70d14ea4b2ce73c9be9d  ./bin/powerpc64-linux-gnu-gcc-ar
 01fc2184f99c558771aa8f2fe30b373d  ./bin/powerpc64-linux-gnu-gcc-nm
 da5356ee09ccda4ca06758d056370f7e  ./bin/powerpc64-linux-gnu-gcc-ranlib
-6ed530d13e65c3500b7e7b7cc863afdc  ./libexec/gcc/powerpc64-linux-gnu/5.5.0/cc1
-24a83af179ca8849da8c64aa854ec8ed  ./libexec/gcc/powerpc64-linux-gnu/5.5.0/cc1plus
-0c05b45bb926a06c2b09acdb1db9aad0  ./libexec/gcc/powerpc64-linux-gnu/5.5.0/collect2
+22b72247a5706f090505341263ca1fc2  ./libexec/gcc/powerpc64-linux-gnu/5.5.0/cc1
+3be618d184038dd30011d6aa8198c0be  ./libexec/gcc/powerpc64-linux-gnu/5.5.0/cc1plus
+2f31e84c01cc087318d0c7f15b6e3f47  ./libexec/gcc/powerpc64-linux-gnu/5.5.0/collect2
 417a5b42a26275b2c912db16b9abf73a  ./libexec/gcc/powerpc64-linux-gnu/5.5.0/install-tools/fixincl
 fd6f80ec9089ddf51f9cac26299e45af  ./libexec/gcc/powerpc64-linux-gnu/5.5.0/install-tools/fixinc.sh
 a585abbd6a9cdc474564b54fc72e4efa  ./libexec/gcc/powerpc64-linux-gnu/5.5.0/install-tools/mkheaders
 5071acceb24c0c0e8a423286205ed54c  ./libexec/gcc/powerpc64-linux-gnu/5.5.0/install-tools/mkinstalldirs
-4e77b773ac45ce8f82a4d21a34063920  ./libexec/gcc/powerpc64-linux-gnu/5.5.0/lto-wrapper
+5267311e0ed8bb5358e5556dfa205ca6  ./libexec/gcc/powerpc64-linux-gnu/5.5.0/lto-wrapper
 a90ab86f837913280f72beb5310714bc  ./lib/gcc/powerpc64-linux-gnu/5.5.0/crtbegin.o
 0b38aa831c40b6bc4fb22248746d60d8  ./lib/gcc/powerpc64-linux-gnu/5.5.0/crtbeginS.o
 8f62d8795bebd8e87caa3640522ff93b  ./lib/gcc/powerpc64-linux-gnu/5.5.0/crtbeginT.o
--8<---------------cut here---------------end--------------->8---

These appear to be the same diffs that I reported before - with the
notable exception that libstdc++.a now is missing from the list of
differing files (hooray!).

Going forward, I'm not sure how best to investigate the inputs to find
out what's causing the differences.  I just know I really, really,
really don't want to rebuild everything multiple times, since it takes
hours/days.  If you have any creative ideas for how to speed up the
investigation, I'm all ears.

-- 
Chris

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1.2: 0001-gnu-Disable-libstdc-in-bootstrap-GCC.patch --]
[-- Type: text/x-patch, Size: 1230 bytes --]

From e3d1778a86dfd171d59d91eb01417faaf63dfa17 Mon Sep 17 00:00:00 2001
From: Chris Marusich <cmmarusich@gmail.com>
Date: Sat, 19 Sep 2020 14:25:43 -0700
Subject: [PATCH] gnu: Disable libstdc++ in bootstrap GCC.

Fixes part of: <https://bugs.gnu.org/41669>.

* gnu/packages/make-bootstrap.scm (%gcc-static) [#:configure-flags]: Add
--disable-libstdcxx to disable building the libstdc++-v3 directory.
---
 gnu/packages/make-bootstrap.scm | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/gnu/packages/make-bootstrap.scm b/gnu/packages/make-bootstrap.scm
index b2d3e2a326..8632d63c21 100644
--- a/gnu/packages/make-bootstrap.scm
+++ b/gnu/packages/make-bootstrap.scm
@@ -487,6 +487,10 @@ for `sh' in $PATH, and without nscd, and with static NSS modules."
                    ;; Make sure gcc-nm doesn't require liblto_plugin.so.
                    "--disable-lto"
 
+                   ;; In this GCC version, libstdc++.a is not reproducible:
+                   ;; https://debbugs.gnu.org/cgi/bugreport.cgi?bug=41669
+                   "--disable-libstdcxx"
+
                    "--disable-shared"
                    "--disable-plugin"
                    "--disable-libmudflap"
-- 
2.26.2


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

^ permalink raw reply	[flat|nested] 9+ messages in thread

end of thread, other threads:[~2020-09-26  6:54 UTC | newest]

Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-06-02 18:59 bug#41669: Cross-compiled powerpc64-linux bootstrap-tarballs not reproducible Chris Marusich
2020-06-03  9:48 ` Chris Marusich
2020-06-03 20:50   ` Vincent Legoll
2020-06-10  6:15   ` Chris Marusich
2020-06-10 22:20     ` Bengt Richter
2020-06-11 21:09     ` Jack Hill
2020-09-13  2:53   ` Chris Marusich
2020-09-13  6:28     ` Efraim Flashner
2020-09-26  6:52       ` Chris Marusich

unofficial mirror of bug-guix@gnu.org 

This inbox may be cloned and mirrored by anyone:

	git clone --mirror https://yhetil.org/guix-bugs/0 guix-bugs/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 guix-bugs guix-bugs/ https://yhetil.org/guix-bugs \
		bug-guix@gnu.org
	public-inbox-index guix-bugs

Example config snippet for mirrors.
Newsgroups are available over NNTP:
	nntp://news.yhetil.org/yhetil.gnu.guix.bugs
	nntp://news.gmane.io/gmane.comp.gnu.guix.bugs


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git