Re: gcc-ddc

Re: gcc-ddc

[Jan Nieuwenhuizen]

Gábor Boskovits writes:

Hey Gábor!

[cc: guix-devel]

> I'm definietly making progress on this. Now I have a working debug build of gcc.
> Identified the critical symbols, they are:

> static const char *const standard_exec_prefix = STANDARD_EXEC_PREFIX;
> static const char *const standard_libexec_prefix = STANDARD_LIBEXEC_PREFIX;
> static const char *const standard_bindir_prefix = STANDARD_BINDIR_PREFIX;

Oh nice!

> The problem fundamentally is that they are calculated from prefix passed to configure.
> I've checked, that that is the store location.

Right.

> How should we go on with this?
>
> Is it possible to pass other value as prefix, or should we keep prefix as is, and patch the makefile?
> It is set from line 2092 in gcc/Makefile.in by the way.

Good question.  I think we should try patching the Makefile.in.

This is a problem that other distros usually do not see (prefix=/usr).
There is a 3rd option: not using this define in the code, but that might
be theoretical/silly without having looked at it.

If we manage to pass a different --prefix to configure (does this work?

      #:configure-flags= '("--prefix=/usr")

then we still need to make sure that make install uses the prefix we
want by writing our own install phase...it gets ugly here I guess...

Greetings, janneke

-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

Re: gcc-ddc

[Ricardo Wurmus]

Jan Nieuwenhuizen <janneke@gnu.org> writes:

> Gábor Boskovits writes:
>
> Hey Gábor!
>
> [cc: guix-devel]
>
>> I'm definietly making progress on this. Now I have a working debug build of gcc.
>> Identified the critical symbols, they are:
>
>> static const char *const standard_exec_prefix = STANDARD_EXEC_PREFIX;
>> static const char *const standard_libexec_prefix = STANDARD_LIBEXEC_PREFIX;
>> static const char *const standard_bindir_prefix = STANDARD_BINDIR_PREFIX;
>
> Oh nice!
>
>> The problem fundamentally is that they are calculated from prefix passed to configure.
>> I've checked, that that is the store location.
>
> Right.
>
>> How should we go on with this?
>>
>> Is it possible to pass other value as prefix, or should we keep prefix as is, and patch the makefile?
>> It is set from line 2092 in gcc/Makefile.in by the way.
>
> Good question.  I think we should try patching the Makefile.in.

I’m just throwing this in, even though I suspect that it is a terrible
idea: we could replace these symbols with calls to getenv and provide
the values at runtime with a separate wrapper that would be excluded in
the comparison.

--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net

Re: gcc-ddc

[Gábor Boskovits]

[-- Attachment #1: Type: text/plain, Size: 2843 bytes --]

The only problematic one seems to be standard_libexec_prefix, because that
is  used in line 3654 of gcc/gcc.c in a real assignment.
It is also used in line 64 of gcc/gcc-ar.c.

Other uses of all these other symbols could be calculated as compile time
realitve paths, and if we can live with these paths staying in the same
store directory, then it would be ok.

This problematic use pattern is in the from:

x=make_relative_prefix(y,standard_exec_prefix,standard_libexec_prefix);
if(!x) x=standard_libexec_prefix;

Code of make_relative_prefix is in libiberty/make-relative-prefix.c.

Assuming sane values (not nulls, existing program name, valid
GCC_EXEC_PREFIX) we get null in the following cases:
1. GCC_EXEC_PREFIX(or the program name directory
component)==standard_exec_prefix
2. if the path present in standard_exec_prefix and standard_libexec_prefix
has no common directories(starting from the beginning)
3. in case of allocation failure.

We can safely assume that case 2 does not happen, as we at least have
/gnu/store there, I think.
Nothing can be done about case 3, I don't think we get too far in that case
anyway...

So, when this happens we simply have case 1: we are not relocated.

In gcc/gcc.c this pattern is guarded by if(gcc_exec_prefix) basically.(it
is in an else block)
It is not so in gcc/gcc-ar.c.

This is how far I could get with it by now.

2017-11-20 23:14 GMT+01:00 Ricardo Wurmus <rekado@elephly.net>:

>
> Jan Nieuwenhuizen <janneke@gnu.org> writes:
>
> > Gábor Boskovits writes:
> >
> > Hey Gábor!
> >
> > [cc: guix-devel]
> >
> >> I'm definietly making progress on this. Now I have a working debug
> build of gcc.
> >> Identified the critical symbols, they are:
> >
> >> static const char *const standard_exec_prefix = STANDARD_EXEC_PREFIX;
> >> static const char *const standard_libexec_prefix =
> STANDARD_LIBEXEC_PREFIX;
> >> static const char *const standard_bindir_prefix =
> STANDARD_BINDIR_PREFIX;
> >
> > Oh nice!
> >
> >> The problem fundamentally is that they are calculated from prefix
> passed to configure.
> >> I've checked, that that is the store location.
> >
> > Right.
> >
> >> How should we go on with this?
> >>
> >> Is it possible to pass other value as prefix, or should we keep prefix
> as is, and patch the makefile?
> >> It is set from line 2092 in gcc/Makefile.in by the way.
> >
> > Good question.  I think we should try patching the Makefile.in.
>
> I’m just throwing this in, even though I suspect that it is a terrible
> idea: we could replace these symbols with calls to getenv and provide
> the values at runtime with a separate wrapper that would be excluded in
> the comparison.
>
> --
> Ricardo
>
> GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
> https://elephly.net
>
>
>

[-- Attachment #2: Type: text/html, Size: 3710 bytes --]

Re: gcc-ddc

[Gábor Boskovits]

[-- Attachment #1: Type: text/plain, Size: 3221 bytes --]

The wrapper approach eliminated those three, we still have in prefix.c the
prefix used as static initializer. I have to investigate further, but it's
only 300 lines, should be tractable.

2017-11-21 0:16 GMT+01:00 Gábor Boskovits <boskovits@gmail.com>:

> The only problematic one seems to be standard_libexec_prefix, because that
> is  used in line 3654 of gcc/gcc.c in a real assignment.
> It is also used in line 64 of gcc/gcc-ar.c.
>
> Other uses of all these other symbols could be calculated as compile time
> realitve paths, and if we can live with these paths staying in the same
> store directory, then it would be ok.
>
> This problematic use pattern is in the from:
>
> x=make_relative_prefix(y,standard_exec_prefix,standard_libexec_prefix);
> if(!x) x=standard_libexec_prefix;
>
> Code of make_relative_prefix is in libiberty/make-relative-prefix.c.
>
> Assuming sane values (not nulls, existing program name, valid
> GCC_EXEC_PREFIX) we get null in the following cases:
> 1. GCC_EXEC_PREFIX(or the program name directory component)==standard_exec_
> prefix
> 2. if the path present in standard_exec_prefix and standard_libexec_prefix
> has no common directories(starting from the beginning)
> 3. in case of allocation failure.
>
> We can safely assume that case 2 does not happen, as we at least have
> /gnu/store there, I think.
> Nothing can be done about case 3, I don't think we get too far in that
> case anyway...
>
> So, when this happens we simply have case 1: we are not relocated.
>
> In gcc/gcc.c this pattern is guarded by if(gcc_exec_prefix) basically.(it
> is in an else block)
> It is not so in gcc/gcc-ar.c.
>
> This is how far I could get with it by now.
>
> 2017-11-20 23:14 GMT+01:00 Ricardo Wurmus <rekado@elephly.net>:
>
>>
>> Jan Nieuwenhuizen <janneke@gnu.org> writes:
>>
>> > Gábor Boskovits writes:
>> >
>> > Hey Gábor!
>> >
>> > [cc: guix-devel]
>> >
>> >> I'm definietly making progress on this. Now I have a working debug
>> build of gcc.
>> >> Identified the critical symbols, they are:
>> >
>> >> static const char *const standard_exec_prefix = STANDARD_EXEC_PREFIX;
>> >> static const char *const standard_libexec_prefix =
>> STANDARD_LIBEXEC_PREFIX;
>> >> static const char *const standard_bindir_prefix =
>> STANDARD_BINDIR_PREFIX;
>> >
>> > Oh nice!
>> >
>> >> The problem fundamentally is that they are calculated from prefix
>> passed to configure.
>> >> I've checked, that that is the store location.
>> >
>> > Right.
>> >
>> >> How should we go on with this?
>> >>
>> >> Is it possible to pass other value as prefix, or should we keep prefix
>> as is, and patch the makefile?
>> >> It is set from line 2092 in gcc/Makefile.in by the way.
>> >
>> > Good question.  I think we should try patching the Makefile.in.
>>
>> I’m just throwing this in, even though I suspect that it is a terrible
>> idea: we could replace these symbols with calls to getenv and provide
>> the values at runtime with a separate wrapper that would be excluded in
>> the comparison.
>>
>> --
>> Ricardo
>>
>> GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
>> https://elephly.net
>>
>>
>>
>

[-- Attachment #2: Type: text/html, Size: 4338 bytes --]

Re: gcc-ddc

[Gábor Boskovits]

[-- Attachment #1: Type: text/plain, Size: 3135 bytes --]

I just pushed what I have right now. It's on branch gcc-ddc on my github.
Should I post a patch here?

2017-11-21 0:16 GMT+01:00 Gábor Boskovits <boskovits@gmail.com>:

> The only problematic one seems to be standard_libexec_prefix, because that
> is  used in line 3654 of gcc/gcc.c in a real assignment.
> It is also used in line 64 of gcc/gcc-ar.c.
>
> Other uses of all these other symbols could be calculated as compile time
> realitve paths, and if we can live with these paths staying in the same
> store directory, then it would be ok.
>
> This problematic use pattern is in the from:
>
> x=make_relative_prefix(y,standard_exec_prefix,standard_libexec_prefix);
> if(!x) x=standard_libexec_prefix;
>
> Code of make_relative_prefix is in libiberty/make-relative-prefix.c.
>
> Assuming sane values (not nulls, existing program name, valid
> GCC_EXEC_PREFIX) we get null in the following cases:
> 1. GCC_EXEC_PREFIX(or the program name directory component)==standard_exec_
> prefix
> 2. if the path present in standard_exec_prefix and standard_libexec_prefix
> has no common directories(starting from the beginning)
> 3. in case of allocation failure.
>
> We can safely assume that case 2 does not happen, as we at least have
> /gnu/store there, I think.
> Nothing can be done about case 3, I don't think we get too far in that
> case anyway...
>
> So, when this happens we simply have case 1: we are not relocated.
>
> In gcc/gcc.c this pattern is guarded by if(gcc_exec_prefix) basically.(it
> is in an else block)
> It is not so in gcc/gcc-ar.c.
>
> This is how far I could get with it by now.
>
> 2017-11-20 23:14 GMT+01:00 Ricardo Wurmus <rekado@elephly.net>:
>
>>
>> Jan Nieuwenhuizen <janneke@gnu.org> writes:
>>
>> > Gábor Boskovits writes:
>> >
>> > Hey Gábor!
>> >
>> > [cc: guix-devel]
>> >
>> >> I'm definietly making progress on this. Now I have a working debug
>> build of gcc.
>> >> Identified the critical symbols, they are:
>> >
>> >> static const char *const standard_exec_prefix = STANDARD_EXEC_PREFIX;
>> >> static const char *const standard_libexec_prefix =
>> STANDARD_LIBEXEC_PREFIX;
>> >> static const char *const standard_bindir_prefix =
>> STANDARD_BINDIR_PREFIX;
>> >
>> > Oh nice!
>> >
>> >> The problem fundamentally is that they are calculated from prefix
>> passed to configure.
>> >> I've checked, that that is the store location.
>> >
>> > Right.
>> >
>> >> How should we go on with this?
>> >>
>> >> Is it possible to pass other value as prefix, or should we keep prefix
>> as is, and patch the makefile?
>> >> It is set from line 2092 in gcc/Makefile.in by the way.
>> >
>> > Good question.  I think we should try patching the Makefile.in.
>>
>> I’m just throwing this in, even though I suspect that it is a terrible
>> idea: we could replace these symbols with calls to getenv and provide
>> the values at runtime with a separate wrapper that would be excluded in
>> the comparison.
>>
>> --
>> Ricardo
>>
>> GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
>> https://elephly.net
>>
>>
>>
>

[-- Attachment #2: Type: text/html, Size: 4255 bytes --]

Re: gcc-ddc

[Jan Nieuwenhuizen]

Gábor Boskovits writes:

> I just pushed what I have right now. It's on branch gcc-ddc on my github. Should I post a patch here? 

Great!

Yes, that makes commenting on it an easy option.  Also, please mention
the location to clone from.  github is non-free, but cloning from it
is ok.

janneke


>
> 2017-11-21 0:16 GMT+01:00 Gábor Boskovits <boskovits@gmail.com>:
>
>     The only problematic one seems to be standard_libexec_prefix, because that is  used in line 3654 of gcc/
>     gcc.c in a real assignment.
>     It is also used in line 64 of gcc/gcc-ar.c.
>    
>     Other uses of all these other symbols could be calculated as compile time realitve paths, and if we can live
>     with these paths staying in the same store directory, then it would be ok. 
>    
>     This problematic use pattern is in the from:
>    
>     x=make_relative_prefix(y,standard_exec_prefix,standard_libexec_prefix);
>     if(!x) x=standard_libexec_prefix;
>    
>     Code of make_relative_prefix is in libiberty/make-relative-prefix.c.
>    
>     Assuming sane values (not nulls, existing program name, valid GCC_EXEC_PREFIX) we get null in the following
>     cases:
>     1. GCC_EXEC_PREFIX(or the program name directory component)==standard_exec_prefix
>     2. if the path present in standard_exec_prefix and standard_libexec_prefix has no common directories
>     (starting from the beginning)
>     3. in case of allocation failure.
>    
>     We can safely assume that case 2 does not happen, as we at least have /gnu/store there, I think.
>     Nothing can be done about case 3, I don't think we get too far in that case anyway...
>    
>     So, when this happens we simply have case 1: we are not relocated.
>    
>     In gcc/gcc.c this pattern is guarded by if(gcc_exec_prefix) basically.(it is in an else block)
>     It is not so in gcc/gcc-ar.c.
>    
>     This is how far I could get with it by now.
>    
>     2017-11-20 23:14 GMT+01:00 Ricardo Wurmus <rekado@elephly.net>:
>
>         Jan Nieuwenhuizen <janneke@gnu.org> writes:
>        
>         > Gábor Boskovits writes:
>         >
>         > Hey Gábor!
>         >
>         > [cc: guix-devel]
>         >
>         >> I'm definietly making progress on this. Now I have a working debug build of gcc.
>         >> Identified the critical symbols, they are:
>         >
>         >> static const char *const standard_exec_prefix = STANDARD_EXEC_PREFIX;
>         >> static const char *const standard_libexec_prefix = STANDARD_LIBEXEC_PREFIX;
>         >> static const char *const standard_bindir_prefix = STANDARD_BINDIR_PREFIX;
>         >
>         > Oh nice!
>         >
>         >> The problem fundamentally is that they are calculated from prefix passed to configure.
>         >> I've checked, that that is the store location.
>         >
>         > Right.
>         >
>         >> How should we go on with this?
>         >>
>         >> Is it possible to pass other value as prefix, or should we keep prefix as is, and patch the makefile?
>         >> It is set from line 2092 in gcc/Makefile.in by the way.
>         >
>         > Good question.  I think we should try patching the Makefile.in.
>        
>         I’m just throwing this in, even though I suspect that it is a terrible
>         idea: we could replace these symbols with calls to getenv and provide
>         the values at runtime with a separate wrapper that would be excluded in
>         the comparison.
>        
>         --
>         Ricardo
>        
>         GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
>         https://elephly.net
>

-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

Re: gcc-ddc

[Gábor Boskovits]

[-- Attachment #1.1: Type: text/plain, Size: 5107 bytes --]

Hello!

Jan, I can now pass the test you have in the original patch, which
basically builds gcc-4.7.4 twice, and  checks if gcc/bin are identical.
I'm using the getenv approach Ricardo suggested. I've not written a wrapper
yet, the environment variables are set from the build.
As the gcc binary depends on the program name (argv[0]) we should leave it
as is, so the wrapper should be gcc-wrapper, and also gcc-ar-wrapper.
Or something like that, and link the wrappers ot the profile as gcc,
gcc-ar... I think we recently had a discussion on that.

I'd like to know if someone is willing to do this with the wrapper, or at
least can point me to some packages already doing this.

I'd like some help with choosing appropriate names for these environment
variables.

One thing we still miss, is that the gcc-ddc package does not produce
output if the comparison is successful, so we have build failed then too.
It would be great, if we could store the diffoscope output in a file in the
output directory anyway. Would you mind changing it that way?

Now I changed the test to compare the full build trees, and we have several
files, where these same problems arose.
We also have several differences coming from section .gnu_debuglink. I
guess that we don't need that anyway.

Clone location is: https://github.com/Boskovits/guix.git
Branch gcc-ddc.

I guess I will go on with this and try to make the gcc-4.7.4 build fully
reproducible. Once that is done we can try the clang one, and see if some
things are still missing or not.
We have the definition in place for that at least.

I cannot paste the text of the patch because of limitations in my current
mail client, I will work on it, though.
The patch should be applied no top of the one sent by janneke.



2017-11-21 17:48 GMT+01:00 Gábor Boskovits <boskovits@gmail.com>:

> I just pushed what I have right now. It's on branch gcc-ddc on my github.
> Should I post a patch here?
>
> 2017-11-21 0:16 GMT+01:00 Gábor Boskovits <boskovits@gmail.com>:
>
>> The only problematic one seems to be standard_libexec_prefix, because
>> that is  used in line 3654 of gcc/gcc.c in a real assignment.
>> It is also used in line 64 of gcc/gcc-ar.c.
>>
>> Other uses of all these other symbols could be calculated as compile time
>> realitve paths, and if we can live with these paths staying in the same
>> store directory, then it would be ok.
>>
>> This problematic use pattern is in the from:
>>
>> x=make_relative_prefix(y,standard_exec_prefix,standard_libexec_prefix);
>> if(!x) x=standard_libexec_prefix;
>>
>> Code of make_relative_prefix is in libiberty/make-relative-prefix.c.
>>
>> Assuming sane values (not nulls, existing program name, valid
>> GCC_EXEC_PREFIX) we get null in the following cases:
>> 1. GCC_EXEC_PREFIX(or the program name directory
>> component)==standard_exec_prefix
>> 2. if the path present in standard_exec_prefix and
>> standard_libexec_prefix has no common directories(starting from the
>> beginning)
>> 3. in case of allocation failure.
>>
>> We can safely assume that case 2 does not happen, as we at least have
>> /gnu/store there, I think.
>> Nothing can be done about case 3, I don't think we get too far in that
>> case anyway...
>>
>> So, when this happens we simply have case 1: we are not relocated.
>>
>> In gcc/gcc.c this pattern is guarded by if(gcc_exec_prefix) basically.(it
>> is in an else block)
>> It is not so in gcc/gcc-ar.c.
>>
>> This is how far I could get with it by now.
>>
>> 2017-11-20 23:14 GMT+01:00 Ricardo Wurmus <rekado@elephly.net>:
>>
>>>
>>> Jan Nieuwenhuizen <janneke@gnu.org> writes:
>>>
>>> > Gábor Boskovits writes:
>>> >
>>> > Hey Gábor!
>>> >
>>> > [cc: guix-devel]
>>> >
>>> >> I'm definietly making progress on this. Now I have a working debug
>>> build of gcc.
>>> >> Identified the critical symbols, they are:
>>> >
>>> >> static const char *const standard_exec_prefix = STANDARD_EXEC_PREFIX;
>>> >> static const char *const standard_libexec_prefix =
>>> STANDARD_LIBEXEC_PREFIX;
>>> >> static const char *const standard_bindir_prefix =
>>> STANDARD_BINDIR_PREFIX;
>>> >
>>> > Oh nice!
>>> >
>>> >> The problem fundamentally is that they are calculated from prefix
>>> passed to configure.
>>> >> I've checked, that that is the store location.
>>> >
>>> > Right.
>>> >
>>> >> How should we go on with this?
>>> >>
>>> >> Is it possible to pass other value as prefix, or should we keep
>>> prefix as is, and patch the makefile?
>>> >> It is set from line 2092 in gcc/Makefile.in by the way.
>>> >
>>> > Good question.  I think we should try patching the Makefile.in.
>>>
>>> I’m just throwing this in, even though I suspect that it is a terrible
>>> idea: we could replace these symbols with calls to getenv and provide
>>> the values at runtime with a separate wrapper that would be excluded in
>>> the comparison.
>>>
>>> --
>>> Ricardo
>>>
>>> GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
>>> https://elephly.net
>>>
>>>
>>>
>>
>

[-- Attachment #1.2: Type: text/html, Size: 6950 bytes --]

[-- Attachment #2: 0001-Added-changes.patch --]
[-- Type: text/x-patch, Size: 17405 bytes --]

From f4fa6a753c97dcb278c8ce0ba8b19fbb479976bc Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?G=C3=A1bor=20Boskovits?= <boskovits@gmail.com>
Date: Wed, 22 Nov 2017 15:30:21 +0100
Subject: [PATCH] Added changes.
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Squashed commit of the following:

commit 1c0ec69b1f1e019d0be050a7d1094f1768b7e2e4
Author: Gábor Boskovits <boskovits@gmail.com>
Date:   Wed Nov 22 14:40:16 2017 +0100

    Update documentation and test.

commit 8d0b55cb59e5c78799096fa9c49c6505a813f49a
Author: Gábor Boskovits <boskovits@gmail.com>
Date:   Wed Nov 22 09:20:20 2017 +0100

    Update gcc-4-reproducibility-wrapper.patch.

commit f75b44b642dbf7c53c5a9745533ec2be2cb981fa
Author: Gábor Boskovits <boskovits@gmail.com>
Date:   Tue Nov 21 18:30:20 2017 +0100

    Add new environment variable definition to handle gcc/prefix.c.

commit 0f494164aae2ad44d31fa7b21abf6bf8813b0990
Author: Gábor Boskovits <boskovits@gmail.com>
Date:   Tue Nov 21 18:27:58 2017 +0100

    Update wrapper patch to include handling of gcc/prefix.c.

commit 9615bee65f62dbbbe58d3bd83353fe45c435aa05
Author: Gábor Boskovits <boskovits@gmail.com>
Date:   Tue Nov 21 17:37:56 2017 +0100

    Add wrapped versions.

commit 51d5a41f55decbdb8440d1a32f24de6aae8994a9
Author: Gábor Boskovits <boskovits@gmail.com>
Date:   Tue Nov 21 09:11:10 2017 +0100

    Add copyright notice.

commit 82c84eb18d07c542c92459e4ed0b82da289e54f8
Author: Gábor Boskovits <boskovits@gmail.com>
Date:   Tue Nov 21 09:08:59 2017 +0100

    Add gcc-4-reproducibility-wrapper-patch to gnu/local.mk.

commit dcbba610ede784ebef9a201b0f3affb7b3721326
Author: Gábor Boskovits <boskovits@gmail.com>
Date:   Tue Nov 21 09:06:11 2017 +0100

    Add gcc-4-reproducibility-wrapper patch.

commit 5ca720f49181296b8abb51799d6b7818152a75b4
Author: Gábor Boskovits <boskovits@gmail.com>
Date:   Tue Nov 21 09:00:19 2017 +0100

    Gnu add repro-gcc-wrapped.

commit ade9b48e23a8bd63712e41a0cd7cba91587aa628
Author: Gábor Boskovits <boskovits@gmail.com>
Date:   Tue Nov 21 03:00:11 2017 +0100

    Setup environment variables for gcc wrapper.

commit 3eb37f455e399a477df856e7725c04cb47f597bd
Author: Gábor Boskovits <boskovits@gmail.com>
Date:   Tue Nov 21 02:18:18 2017 +0100

    Add nostrip gcc.

commit 22ac376df1e81a98ba7eed07fd0d92dd67f284a9
Author: Gábor Boskovits <boskovits@gmail.com>
Date:   Sun Nov 19 14:15:55 2017 +0100

    Debuggable build trial 1.

commit 5073e1e096496f7a7587c43f0e95e471cec3e0fb
Author: Gábor Boskovits <boskovits@gmail.com>
Date:   Sat Nov 18 19:22:15 2017 +0100

    Fix versions in gcc-ddc description.
---
 gnu/local.mk                                       |   1 +
 gnu/packages/bootstrappable.scm                    | 134 ++++++++++++++-
 .../patches/gcc-4-reproducibility-wrapper.patch    | 190 +++++++++++++++++++++
 3 files changed, 321 insertions(+), 4 deletions(-)
 create mode 100644 gnu/packages/patches/gcc-4-reproducibility-wrapper.patch

diff --git a/gnu/local.mk b/gnu/local.mk
index 72f8fa8e1..d9ce72e18 100644
--- a/gnu/local.mk
+++ b/gnu/local.mk
@@ -647,6 +647,7 @@ dist_patch_DATA =						\
   %D%/packages/patches/gcc-libvtv-runpath.patch			\
   %D%/packages/patches/gcc-strmov-store-file-names.patch	\
   %D%/packages/patches/gcc-4-compile-with-gcc-5.patch		 \
+  %D%/packages/patches/gcc-4-reproducibility-wrapper.patch	\
   %D%/packages/patches/gcc-4.6-gnu-inline.patch			\
   %D%/packages/patches/gcc-4.9.3-mingw-gthr-default.patch	\
   %D%/packages/patches/gcc-5.0-libvtv-runpath.patch		\
diff --git a/gnu/packages/bootstrappable.scm b/gnu/packages/bootstrappable.scm
index cbd7592db..66577ff8d 100644
--- a/gnu/packages/bootstrappable.scm
+++ b/gnu/packages/bootstrappable.scm
@@ -1,5 +1,6 @@
 ;;; GNU Guix --- Functional package management for GNU
 ;;; Copyright © 2017 Jan Nieuwenhuizen <janneke@gnu.org>
+;;; Copyright © 2017 Gábor Boskovits <boskovits@gmail.com>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -225,7 +226,6 @@
                  (("^TOPLEVEL_CONFIGURE_ARGUMENTS=(.*)$" _ rest)
                   "TOPLEVEL_CONFIGURE_ARGUMENTS=\n"))))))))))
 
-
 (define-public repro-gcc-4.7
   (package
     (inherit gcc-4.7-$ORIGIN)
@@ -250,11 +250,82 @@
                        "BUILD_PATH_PREFIX_MAP=~s\n"
                        (getenv "BUILD_PATH_PREFIX_MAP"))))))))))
 
+(define-public repro-gcc-wrapped-4.7
+  (package
+    (inherit gcc-4.7-$ORIGIN)
+    (source
+     (origin
+       (inherit (package-source gcc-4.7-$ORIGIN))
+       (patches (append ((compose origin-patches package-source) gcc-4.7)
+                        (search-patches "gcc-5-reproducibility-drop-profile.patch"
+                                        ;;"gcc-4-compile-with-gcc-5.patch"
+                                        "gcc-4-build-path-prefix-map.patch"
+					"gcc-4-reproducibility-wrapper.patch")))))
+    (name "repro-gcc-wrapped")
+    (version "4.7.4")
+    (arguments
+     (substitute-keyword-arguments (package-arguments gcc-4.7-$ORIGIN)
+       ((#:phases original-phases)
+        `(modify-phases ,original-phases
+           (add-before 'configure 'build-prefix-path
+             (lambda* (#:key inputs #:allow-other-keys)
+               (let ((out (assoc-ref %outputs "out")))
+                 (setenv "PREFIX" out)
+                 (setenv "STANDARD_EXEC_PREFIX" (string-append out "/lib/gcc"))
+                 (setenv "STANDARD_LIBEXEC_PREFIX" (string-append out "/libexec/gcc"))
+                 (setenv "STANDARD_BINDIR_PREFIX" (string-append out "/bin")))
+               (setenv "BUILD_PATH_PREFIX_MAP"
+                       (string-append "gcc" "-" ,version "=" (getcwd)))
+               (format (current-error-port)
+                       "BUILD_PATH_PREFIX_MAP=~s\n"
+                       (getenv "BUILD_PATH_PREFIX_MAP"))))))))))
+
+(define-public repro-gcc-debuggable-4.7
+  (package
+    (inherit repro-gcc-4.7)
+    (name "repro-gcc-debuggable")
+    (arguments
+     (substitute-keyword-arguments (package-arguments repro-gcc-4.7)
+       ((#:make-flags original-flags)
+        `(cons* "BOOT_CFLAGS=-O0 -g3"
+                (delete "BOOT_CFLAGS=-O2 -g0" ,original-flags)))))))
+
+(define-public repro-gcc-wrapped-debuggable-4.7
+  (package
+    (inherit repro-gcc-wrapped-4.7)
+    (name "repro-gcc-wrapped-debuggable")
+    (arguments
+     (substitute-keyword-arguments (package-arguments repro-gcc-wrapped-4.7)
+       ((#:make-flags original-flags)
+        `(cons* "BOOT_CFLAGS=-O0 -g3"
+                (delete "BOOT_CFLAGS=-O2 -g0" ,original-flags)))))))
+
+(define-public repro-gcc-debuggable-nostrip-4.7
+  (package
+    (inherit repro-gcc-debuggable-4.7)
+    (name "repro-gcc-debuggable-nostrip")
+    (arguments
+     (substitute-keyword-arguments (package-arguments repro-gcc-debuggable-4.7)
+        ((#:strip-binaries? _ #f) #f)))))
+
+(define-public repro-gcc-wrapped-debuggable-nostrip-4.7
+  (package
+    (inherit repro-gcc-wrapped-debuggable-4.7)
+    (name "repro-gcc-wrapped-debuggable-nostrip")
+    (arguments
+     (substitute-keyword-arguments (package-arguments repro-gcc-wrapped-debuggable-4.7)
+        ((#:strip-binaries? _ #f) #f)))))
+
 (define-public repr2-gcc-4.7
   (package
     (inherit repro-gcc-4.7)
     (name "repr2-gcc")))
 
+(define-public repr2-gcc-wrapped-4.7
+  (package
+    (inherit repro-gcc-wrapped-4.7)
+    (name "repr2-gcc-wrapped")))
+
 (define-public repro-gcc-7
   (package
     (inherit gcc-4.7-$ORIGIN)
@@ -348,9 +419,64 @@
            (zero? (system* "diffoscope" gcc/bin/gcc clang-gcc/bin/gcc))
            ;;(zero? (system* "diffoscope" gcc clang-gcc))
            ))))
-    (synopsis "test gcc+clang DDC property for gcc-7.2.0")
-    (description "gcc-dcc is a meta-package that depends on repro-gcc-7.2.0
-and on clang-gcc-7.2.0 (the same GCC built with clang).  The builder checks if
+    (synopsis "test gcc+clang DDC property for gcc-4.7.4")
+    (description "gcc-dcc is a meta-package that depends on repro-gcc-4.7.4
+and on clang-gcc-4.7.4 (the same GCC built with clang).  The builder checks if
+both gcc's are bit-for-bit identical and fails if they differ.")
+    (home-page "http://bootstrappable.org")
+    (license gpl3+)))
+
+(define-public gcc-wrapped-ddc-gcc+clang
+  (package
+    (name "gcc-wrapped-ddc")
+    (version "4.7.4")
+    (source #f)
+    (native-inputs `(("clang-gcc" ,repr2-gcc-wrapped-4.7)
+                     ("gcc" ,repro-gcc-wrapped-4.7)
+
+                     ("diffoscope" ,diffoscope)
+                     ("acl" ,acl)       ; For diffoscope
+                     ("binutils" ,binutils)
+                     ("coreutils" ,coreutils)
+                     ("diffutils" ,diffutils)
+                     ("xxd" ,xxd)))
+    (build-system trivial-build-system)
+    (arguments
+     `(#:modules ((guix build utils))
+       #:builder
+       (begin
+         (use-modules (guix build utils))
+         (let* ((diffoscope (assoc-ref %build-inputs "diffoscope"))
+                (acl (assoc-ref %build-inputs "acl"))
+                (binutils (assoc-ref %build-inputs "binutils"))
+                (coreutils (assoc-ref %build-inputs "coreutils"))
+                (diffutils (assoc-ref %build-inputs "diffutils"))
+                (xxd (assoc-ref %build-inputs "xxd"))
+                (gcc (assoc-ref %build-inputs "gcc"))
+                (gcc/bin/gcc (string-append gcc "/bin/gcc"))
+                (clang-gcc (assoc-ref %build-inputs "clang-gcc"))
+                (clang-gcc/bin/gcc (string-append clang-gcc "/bin/gcc")))
+           ;; diffoscope.exc.RequiredToolNotFound: cmp
+           ;; diffoscope.comparators.directory: 'stat' not found! Is PATH wrong?
+           ;; FileNotFoundError: [Errno 2] No such file or directory: 'readelf'
+           ;; diffoscope.comparators.directory: Unable to find 'getfacl'
+           (setenv "PATH" (string-append diffoscope "/bin:"
+
+                                         acl "/acl:"
+                                         binutils "/bin:"
+                                         coreutils "/bin:"
+                                         diffutils "/bin:"
+                                         xxd "/bin:"))
+           ;; ?? xxd not available in path. Falling back to Python hexlify.
+           ;; diffoscope.presenters.formats: Console is unable to print Unicode characters. Set e.g. PYTHONIOENCODING=utf-8
+           (setenv "PYTHONIOENCODING" "utf-8")
+           ;; for starters, only check the gcc binary
+           (zero? (system* "diffoscope" gcc clang-gcc))
+           ;;(zero? (system* "diffoscope" gcc clang-gcc))
+           ))))
+    (synopsis "test gcc+clang DDC property for gcc-4.7.4")
+    (description "gcc-dcc is a meta-package that depends on repro-gcc-wrapped-4.7.4
+and on repr2-gcc-wrapped-4.7.4.  The builder checks if
 both gcc's are bit-for-bit identical and fails if they differ.")
     (home-page "http://bootstrappable.org")
     (license gpl3+)))
diff --git a/gnu/packages/patches/gcc-4-reproducibility-wrapper.patch b/gnu/packages/patches/gcc-4-reproducibility-wrapper.patch
new file mode 100644
index 000000000..5eaba0e25
--- /dev/null
+++ b/gnu/packages/patches/gcc-4-reproducibility-wrapper.patch
@@ -0,0 +1,190 @@
+From 8646998bf1ced61c4fe212eae68a3fb43772d2b9 Mon Sep 17 00:00:00 2001
+From: =?UTF-8?q?G=C3=A1bor=20Boskovits?= <boskovits@gmail.com>
+Date: Wed, 22 Nov 2017 09:18:47 +0100
+Subject: [PATCH] Prepare gcc for wrapper.
+MIME-Version: 1.0
+Content-Type: text/plain; charset=UTF-8
+Content-Transfer-Encoding: 8bit
+
+Squashed commit of the following:
+
+commit b1d80031fed7f094789fd83b02d089376bb4374f
+Author: Gábor Boskovits <boskovits@gmail.com>
+Date:   Wed Nov 22 09:16:57 2017 +0100
+
+    Prepare gcc/prefix.c for wrapper, eliminate remaining PREFIX references.
+
+commit e3749ab19f7a660f9f47abd1e0959c3773df937f
+Merge: 685201b4d 4d2c62dc0
+Author: Gábor Boskovits <boskovits@gmail.com>
+Date:   Tue Nov 21 18:21:13 2017 +0100
+
+    Merge branch 'tmp_squash'
+
+commit 4d2c62dc015407a14175a58d33b5c3a27e0a0755
+Author: Gábor Boskovits <boskovits@gmail.com>
+Date:   Tue Nov 21 18:19:47 2017 +0100
+
+    Prepare gcc/prefix.c for wrapper.
+
+commit 17878706e414e6cccc9400cd324bed5c97a70c64
+Author: Gábor Boskovits <boskovits@gmail.com>
+Date:   Tue Nov 21 09:02:36 2017 +0100
+
+    Add wrapper support.
+
+    Squashed commit of the following:
+
+    commit 685201b4db65a6a32e1b0bb9964d405af99e4f48
+    Author: Gábor Boskovits <boskovits@gmail.com>
+    Date:   Tue Nov 21 02:58:37 2017 +0100
+
+        Prepare gcc-ar.c for the wrapper.
+
+    commit 2dd79c5b79182c2aa500de7a469b21b48dc1995f
+    Author: Gábor Boskovits <boskovits@gmail.com>
+    Date:   Tue Nov 21 02:45:20 2017 +0100
+
+        Prepare source code for the wrapper.
+
+commit 685201b4db65a6a32e1b0bb9964d405af99e4f48
+Author: Gábor Boskovits <boskovits@gmail.com>
+Date:   Tue Nov 21 02:58:37 2017 +0100
+
+    Prepare gcc-ar.c for the wrapper.
+
+commit 2dd79c5b79182c2aa500de7a469b21b48dc1995f
+Author: Gábor Boskovits <boskovits@gmail.com>
+Date:   Tue Nov 21 02:45:20 2017 +0100
+
+    Prepare source code for the wrapper.
+---
+ gcc/gcc-ar.c |  7 +++++--
+ gcc/gcc.c    | 10 +++++++---
+ gcc/prefix.c | 23 +++++++++++++++++------
+ 3 files changed, 29 insertions(+), 11 deletions(-)
+
+diff --git a/gcc/gcc-ar.c b/gcc/gcc-ar.c
+index 5f78378de..4b0352a72 100644
+--- a/gcc/gcc-ar.c
++++ b/gcc/gcc-ar.c
+@@ -26,8 +26,8 @@ along with GCC; see the file COPYING3.  If not see
+ #error "Please set personality"
+ #endif
+ 
+-static const char standard_libexec_prefix[] = STANDARD_LIBEXEC_PREFIX;
+-static const char standard_bin_prefix[] = STANDARD_BINDIR_PREFIX;
++static const char *standard_libexec_prefix;
++static const char *standard_bin_prefix;
+ static const char *const target_machine = TARGET_MACHINE;
+ 
+ static const char dir_separator[] = { DIR_SEPARATOR, 0 };
+@@ -44,6 +44,9 @@ main(int ac, char **av)
+   bool is_ar = !strcmp (PERSONALITY, "ar");
+   int exit_code = FATAL_EXIT_CODE;
+ 
++  standard_libexec_prefix=getenv("STANDARD_LIBEXEC_PREFIX");
++  standard_bin_prefix=getenv("STANDARD_BINDIR_PREFIX");
++
+   exe_name = PERSONALITY;
+ #ifdef CROSS_DIRECTORY_STRUCTURE
+   exe_name = concat (target_machine, "-", exe_name, NULL);
+diff --git a/gcc/gcc.c b/gcc/gcc.c
+index 939dcc873..f75f88eec 100644
+--- a/gcc/gcc.c
++++ b/gcc/gcc.c
+@@ -1125,9 +1125,9 @@ static const char *gcc_libexec_prefix;
+    gcc_exec_prefix is set because, in that case, we know where the
+    compiler has been installed, and use paths relative to that
+    location instead.  */
+-static const char *const standard_exec_prefix = STANDARD_EXEC_PREFIX;
+-static const char *const standard_libexec_prefix = STANDARD_LIBEXEC_PREFIX;
+-static const char *const standard_bindir_prefix = STANDARD_BINDIR_PREFIX;
++static const char *standard_exec_prefix;
++static const char *standard_libexec_prefix;
++static const char *standard_bindir_prefix;
+ static const char *const standard_startfile_prefix = STANDARD_STARTFILE_PREFIX;
+ 
+ /* For native compilers, these are well-known paths containing
+@@ -6154,6 +6154,10 @@ main (int argc, char **argv)
+      on ?: in file-scope variable initializations.  */
+   asm_debug = ASM_DEBUG_SPEC;
+ 
++  standard_exec_prefix=getenv("STANDARD_EXEC_PREFIX");
++  standard_libexec_prefix=getenv("STANDARD_LIBEXEC_PREFIX");
++  standard_bindir_prefix=getenv("STANDARD_BINDIR_PREFIX");
++
+   p = argv[0] + strlen (argv[0]);
+   while (p != argv[0] && !IS_DIR_SEPARATOR (p[-1]))
+     --p;
+diff --git a/gcc/prefix.c b/gcc/prefix.c
+index c7003f8c6..01408e2e5 100644
+--- a/gcc/prefix.c
++++ b/gcc/prefix.c
+@@ -73,7 +73,7 @@ License along with GCC; see the file COPYING3.  If not see
+ #include "prefix.h"
+ #include "common/common-target.h"
+ 
+-static const char *std_prefix = PREFIX;
++static const char *std_prefix;
+ 
+ static const char *get_key_value (char *);
+ static char *translate_name (char *);
+@@ -85,6 +85,17 @@ static char *lookup_key (char *);
+ static HKEY reg_key = (HKEY) INVALID_HANDLE_VALUE;
+ #endif
+ 
++/* Return std_prefix, factory method. */
++
++static const char *
++get_std_prefix (void)
++{
++  if (std_prefix == 0)
++    std_prefix = getenv ("PREFIX");
++
++  return std_prefix;
++}
++
+ /* Given KEY, as above, return its value.  */
+ 
+ static const char *
+@@ -101,7 +112,7 @@ get_key_value (char *key)
+     prefix = getenv (temp = concat (key, "_ROOT", NULL));
+ 
+   if (prefix == 0)
+-    prefix = std_prefix;
++    prefix = get_std_prefix();
+ 
+   free (temp);
+ 
+@@ -207,13 +218,13 @@ translate_name (char *name)
+ 	{
+ 	  prefix = get_key_value (key);
+ 	  if (prefix == 0)
+-	    prefix = std_prefix;
++	    prefix = get_std_prefix();
+ 	}
+       else
+ 	prefix = getenv (key);
+ 
+       if (prefix == 0)
+-	prefix = PREFIX;
++	prefix = getenv("PREFIX");
+ 
+       /* We used to strip trailing DIR_SEPARATORs here, but that can
+ 	 sometimes yield a result with no separator when one was coded
+@@ -248,9 +259,9 @@ char *
+ update_path (const char *path, const char *key)
+ {
+   char *result, *p;
+-  const int len = strlen (std_prefix);
++  const int len = strlen (get_std_prefix());
+ 
+-  if (! filename_ncmp (path, std_prefix, len)
++  if (! filename_ncmp (path, get_std_prefix(), len)
+       && (IS_DIR_SEPARATOR(path[len])
+           || path[len] == '\0')
+       && key != 0)
+-- 
+2.15.0
+
-- 
2.15.0

Re: gcc-ddc

[Jan Nieuwenhuizen]

Gábor Boskovits writes:

Hey Gábor,

> Jan, I can now pass the test you have in the original patch, which
> basically builds gcc-4.7.4 twice, and  checks if gcc/bin are
> identical.

Woohoo!  Amazing work!

> I'm using the getenv approach Ricardo suggested. I've not written a wrapper yet, the environment variables are
> set from the build.
> As the gcc binary depends on the program name (argv[0]) we should leave it as is, so the wrapper should be
> gcc-wrapper, and also gcc-ar-wrapper.
> Or something like that, and link the wrappers ot the profile as gcc, gcc-ar... I think we recently had a
> discussion on that.
>
> I'd like to know if someone is willing to do this with the wrapper, or at least can point me to some packages
> already doing this.
>
> I'd like some help with choosing appropriate names for these environment variables.

Ricardo, ideas?

> One thing we still miss, is that the gcc-ddc package does not produce output if the comparison is successful, so
> we have build failed then too.
> It would be great, if we could store the diffoscope output in a file in the output directory anyway. Would you
> mind changing it that way?

That's a good idea, please do so.

> Now I changed the test to compare the full build trees, and we have several files, where these same problems
> arose.
> We also have several differences coming from section .gnu_debuglink. I guess that we don't need that anyway.
>
> Clone location is: https://github.com/Boskovits/guix.git
> Branch gcc-ddc.

Amazing, that's quite some work...

> I guess I will go on with this and try to make the gcc-4.7.4 build fully reproducible. Once that is done we can
> try the clang one, and see if some things are still missing or not.
> We have the definition in place for that at least.

Sure.

> I cannot paste the text of the patch because of limitations in my current mail client, I will work on it,
> though.
> The patch should be applied no top of the one sent by janneke.

That's fine,

Thanks
janneke

-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

Re: gcc-ddc

[Ricardo Wurmus]

Hi Gábor,

> I'm using the getenv approach Ricardo suggested. I've not written a wrapper
> yet, the environment variables are set from the build.

Maybe that’s sufficient already.  Since the result of this package is
not going to be used as an input to build software it may not actually
need a wrapper.

Instead we can just set the variables in a build phase.

> I'd like some help with choosing appropriate names for these environment
> variables.

I would prefix all of the environment variables with “GUIX_GCC_” just to
avoid conflicts with other legitimate environment variables.  Other than
that the names are fine.

Thanks for investigating this.  I’m impressed with your level of success!

-- 
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net

Re: gcc-ddc

[Gábor Boskovits]

[-- Attachment #1: Type: text/plain, Size: 1374 bytes --]

Hello!

It seems, that one of the source of the reporoducibilty issues with the gcc
build output is that it contains la files with libdir recorded.
Libtool records that in those la files.
I wonder if we have any solution to that already, because it seems to
affect every project using libtool.

If not, do we have any idea to solve this? It would be great if we could
come up with some generic solution to the problem.

2017-11-23 8:14 GMT+01:00 Ricardo Wurmus <rekado@elephly.net>:

>
> Hi Gábor,
>
> > I'm using the getenv approach Ricardo suggested. I've not written a
> wrapper
> > yet, the environment variables are set from the build.
>
> Maybe that’s sufficient already.  Since the result of this package is
> not going to be used as an input to build software it may not actually
> need a wrapper.
>
> Instead we can just set the variables in a build phase.
>
> > I'd like some help with choosing appropriate names for these environment
> > variables.
>
> I would prefix all of the environment variables with “GUIX_GCC_” just to
> avoid conflicts with other legitimate environment variables.  Other than
> that the names are fine.
>
> Thanks for investigating this.  I’m impressed with your level of success!
>
> --
> Ricardo
>
> GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
> https://elephly.net
>
>
>

[-- Attachment #2: Type: text/html, Size: 1970 bytes --]

Re: gcc-ddc

[Gábor Boskovits]

[-- Attachment #1: Type: text/plain, Size: 2231 bytes --]

Hello!

It seems, that I can make really good progress here.
Now the only things that remain:

The libtool .la files record the installation directory, these are textfile
wrappers anyways, so I don't know if we should care about this.
The mkheaders shell srcipt in install-tools record the installation
directory, this is in source form by the way, so I don't know if we should
care about this.

I this two cases the transformation to get the other set of files is simply
to replace the prefix, so we could make a test on that if we want.

The only remainig problem is that the symbol executable_checksum in cc1 and
cc1plus still differ. No other differences remained.

I'm now investigating the checksum issue.

2017-11-23 12:23 GMT+01:00 Gábor Boskovits <boskovits@gmail.com>:

> Hello!
>
> It seems, that one of the source of the reporoducibilty issues with the
> gcc build output is that it contains la files with libdir recorded.
> Libtool records that in those la files.
> I wonder if we have any solution to that already, because it seems to
> affect every project using libtool.
>
> If not, do we have any idea to solve this? It would be great if we could
> come up with some generic solution to the problem.
>
> 2017-11-23 8:14 GMT+01:00 Ricardo Wurmus <rekado@elephly.net>:
>
>>
>> Hi Gábor,
>>
>> > I'm using the getenv approach Ricardo suggested. I've not written a
>> wrapper
>> > yet, the environment variables are set from the build.
>>
>> Maybe that’s sufficient already.  Since the result of this package is
>> not going to be used as an input to build software it may not actually
>> need a wrapper.
>>
>> Instead we can just set the variables in a build phase.
>>
>> > I'd like some help with choosing appropriate names for these environment
>> > variables.
>>
>> I would prefix all of the environment variables with “GUIX_GCC_” just to
>> avoid conflicts with other legitimate environment variables.  Other than
>> that the names are fine.
>>
>> Thanks for investigating this.  I’m impressed with your level of success!
>>
>> --
>> Ricardo
>>
>> GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
>> https://elephly.net
>>
>>
>>
>

[-- Attachment #2: Type: text/html, Size: 3267 bytes --]

Re: gcc-ddc

[Gábor Boskovits]

[-- Attachment #1: Type: text/plain, Size: 2586 bytes --]

Oops, I missed one. we have  libstdc++.so.6.0.17-gdb.py, which is similar
to the first two cases above, records installation diretory, in source
form, I don't think we have to fix that either.

2017-11-29 10:43 GMT+01:00 Gábor Boskovits <boskovits@gmail.com>:

> Hello!
>
> It seems, that I can make really good progress here.
> Now the only things that remain:
>
> The libtool .la files record the installation directory, these are
> textfile wrappers anyways, so I don't know if we should care about this.
> The mkheaders shell srcipt in install-tools record the installation
> directory, this is in source form by the way, so I don't know if we should
> care about this.
>
> I this two cases the transformation to get the other set of files is
> simply to replace the prefix, so we could make a test on that if we want.
>
> The only remainig problem is that the symbol executable_checksum in cc1
> and cc1plus still differ. No other differences remained.
>
> I'm now investigating the checksum issue.
>
> 2017-11-23 12:23 GMT+01:00 Gábor Boskovits <boskovits@gmail.com>:
>
>> Hello!
>>
>> It seems, that one of the source of the reporoducibilty issues with the
>> gcc build output is that it contains la files with libdir recorded.
>> Libtool records that in those la files.
>> I wonder if we have any solution to that already, because it seems to
>> affect every project using libtool.
>>
>> If not, do we have any idea to solve this? It would be great if we could
>> come up with some generic solution to the problem.
>>
>> 2017-11-23 8:14 GMT+01:00 Ricardo Wurmus <rekado@elephly.net>:
>>
>>>
>>> Hi Gábor,
>>>
>>> > I'm using the getenv approach Ricardo suggested. I've not written a
>>> wrapper
>>> > yet, the environment variables are set from the build.
>>>
>>> Maybe that’s sufficient already.  Since the result of this package is
>>> not going to be used as an input to build software it may not actually
>>> need a wrapper.
>>>
>>> Instead we can just set the variables in a build phase.
>>>
>>> > I'd like some help with choosing appropriate names for these
>>> environment
>>> > variables.
>>>
>>> I would prefix all of the environment variables with “GUIX_GCC_” just to
>>> avoid conflicts with other legitimate environment variables.  Other than
>>> that the names are fine.
>>>
>>> Thanks for investigating this.  I’m impressed with your level of success!
>>>
>>> --
>>> Ricardo
>>>
>>> GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
>>> https://elephly.net
>>>
>>>
>>>
>>
>

[-- Attachment #2: Type: text/html, Size: 4003 bytes --]

Re: gcc-ddc

[Jan Nieuwenhuizen]

Gábor Boskovits writes:

> It seems, that I can make really good progress here.
> Now the only things that remain:

Great!

> The libtool .la files record the installation directory, these are textfile wrappers anyways, so I don't know if
> we should care about this.

How about asking the libtool developers?

> The mkheaders shell srcipt in install-tools record the installation directory, this is in source form by the
> way, so I don't know if we should care about this.

In a way similar to the libtool wrappers: the build is still
reproducible in a way; just harder to check with Guix.  Having
everything below /gnu/store/deadbeef-gcc-4.7.4 identical could be a
feature, but possibly something left for later.

> The only remainig problem is that the symbol executable_checksum in cc1 and cc1plus still differ. No other
> differences remained.

OK!

> I'm now investigating the checksum issue.

Great to hear your progress
janneke
-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

Re: gcc-ddc

[Gábor Boskovits]

[-- Attachment #1: Type: text/plain, Size: 1385 bytes --]

It seems, that the libtool file differences leak into the checksum.
I will try to contact developers on how to bypass that issue.


2017-11-29 16:57 GMT+01:00 Jan Nieuwenhuizen <janneke@gnu.org>:

> Gábor Boskovits writes:
>
> > It seems, that I can make really good progress here.
> > Now the only things that remain:
>
> Great!
>
> > The libtool .la files record the installation directory, these are
> textfile wrappers anyways, so I don't know if
> > we should care about this.
>
> How about asking the libtool developers?
>
> > The mkheaders shell srcipt in install-tools record the installation
> directory, this is in source form by the
> > way, so I don't know if we should care about this.
>
> In a way similar to the libtool wrappers: the build is still
> reproducible in a way; just harder to check with Guix.  Having
> everything below /gnu/store/deadbeef-gcc-4.7.4 identical could be a
> feature, but possibly something left for later.
>
> > The only remainig problem is that the symbol executable_checksum in cc1
> and cc1plus still differ. No other
> > differences remained.
>
> OK!
>
> > I'm now investigating the checksum issue.
>
> Great to hear your progress
> janneke
> --
> Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
> Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com
>

[-- Attachment #2: Type: text/html, Size: 2205 bytes --]

Re: gcc-ddc

[Gábor Boskovits]

[-- Attachment #1: Type: text/plain, Size: 2489 bytes --]

Aside from these libtool files we can now say, that this ddc project
succeeded.
I've contacted the libtool developers if we can extend the wrapper approach
to the .la files.
It seems, that in some older version of libtool those were just sourced as
shell script, but
I don't know if now they do something more fancy with it or not...
Anyways, if it's just shell script, then the environment variable approach
can also work out there.
The only problem seems, that I should do the substitution before
checksumming the compiler.
I think I can inject something into the makefile, or use a patched vesion
of libtool.

A patched libtool could be a better option, so other ddc projects can use
it.
I guess I can do something like add an environment variable
GUIX_INSTALL_DIRECTORY, or something like that...
Any maybe name this version libtool-for-ddc.
It should be noted in the package documentation, that this package is not
recommended for general use.

WDYT?


2017-11-30 15:32 GMT+01:00 Gábor Boskovits <boskovits@gmail.com>:

> It seems, that the libtool file differences leak into the checksum.
> I will try to contact developers on how to bypass that issue.
>
>
> 2017-11-29 16:57 GMT+01:00 Jan Nieuwenhuizen <janneke@gnu.org>:
>
>> Gábor Boskovits writes:
>>
>> > It seems, that I can make really good progress here.
>> > Now the only things that remain:
>>
>> Great!
>>
>> > The libtool .la files record the installation directory, these are
>> textfile wrappers anyways, so I don't know if
>> > we should care about this.
>>
>> How about asking the libtool developers?
>>
>> > The mkheaders shell srcipt in install-tools record the installation
>> directory, this is in source form by the
>> > way, so I don't know if we should care about this.
>>
>> In a way similar to the libtool wrappers: the build is still
>> reproducible in a way; just harder to check with Guix.  Having
>> everything below /gnu/store/deadbeef-gcc-4.7.4 identical could be a
>> feature, but possibly something left for later.
>>
>> > The only remainig problem is that the symbol executable_checksum in cc1
>> and cc1plus still differ. No other
>> > differences remained.
>>
>> OK!
>>
>> > I'm now investigating the checksum issue.
>>
>> Great to hear your progress
>> janneke
>> --
>> Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
>> Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com
>>
>
>

[-- Attachment #2: Type: text/html, Size: 3748 bytes --]

Re: gcc-ddc

[Jan Nieuwenhuizen]

Gábor Boskovits writes:

> Aside from these libtool files we can now say, that this ddc project succeeded.

Wait... The libtool's .la files are now the only files that show any
difference, even when gcc is compiled into it's own prefix?  That's amazing!!!

> I've contacted the libtool developers if we can extend the wrapper approach to the .la files.

Great!

> It seems, that in some older version of libtool those were just sourced as shell script, but
> I don't know if now they do something more fancy with it or not...
> Anyways, if it's just shell script, then the environment variable approach can also work out there.
> The only problem seems, that I should do the substitution before checksumming the compiler.
> I think I can inject something into the makefile, or use a patched vesion of libtool.
>
> A patched libtool could be a better option, so other ddc projects can use it.

Indeed.

> I guess I can do something like add an environment variable GUIX_INSTALL_DIRECTORY, or something like that...

What's different about GUIX_INSTALL_DIRECTORY than the usual: PREFIX?

> Any maybe name this version libtool-for-ddc.
> It should be noted in the package documentation, that this package is not recommended for general use.

Are you proposing to patch libtool?  If so, even if patching it for
[gcc-]dcc only would be a great hack for now.  Please go forward with
that idea at least until libtool devs help us with a full solution.

janneke

-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

Re: gcc-ddc

[Gábor Boskovits]

[-- Attachment #1: Type: text/plain, Size: 2214 bytes --]

Ok, I will do that.

I can eliminate differences in the wrappers easily, then I will look into
the libtool patch.
It seems, that the la files are causing the checksum difference in cc1 and
cc1plus, because if I remove libraries from the checksum, then they agree.
I also checked the same things when I compile with clang. The differences
are the same, so we can do this. The rest is just "cosmetics".


2017-12-02 15:48 GMT+01:00 Jan Nieuwenhuizen <janneke@gnu.org>:

> Gábor Boskovits writes:
>
> > Aside from these libtool files we can now say, that this ddc project
> succeeded.
>
> Wait... The libtool's .la files are now the only files that show any
> difference, even when gcc is compiled into it's own prefix?  That's
> amazing!!!
>
> > I've contacted the libtool developers if we can extend the wrapper
> approach to the .la files.
>
> Great!
>
> > It seems, that in some older version of libtool those were just sourced
> as shell script, but
> > I don't know if now they do something more fancy with it or not...
> > Anyways, if it's just shell script, then the environment variable
> approach can also work out there.
> > The only problem seems, that I should do the substitution before
> checksumming the compiler.
> > I think I can inject something into the makefile, or use a patched
> vesion of libtool.
> >
> > A patched libtool could be a better option, so other ddc projects can
> use it.
>
> Indeed.
>
> > I guess I can do something like add an environment variable
> GUIX_INSTALL_DIRECTORY, or something like that...
>
> What's different about GUIX_INSTALL_DIRECTORY than the usual: PREFIX?
>
> > Any maybe name this version libtool-for-ddc.
> > It should be noted in the package documentation, that this package is
> not recommended for general use.
>
> Are you proposing to patch libtool?  If so, even if patching it for
> [gcc-]dcc only would be a great hack for now.  Please go forward with
> that idea at least until libtool devs help us with a full solution.
>
> janneke
>
> --
> Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
> Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com
>

[-- Attachment #2: Type: text/html, Size: 3090 bytes --]

Re: gcc-ddc

[Ricardo Wurmus]

>> I guess I can do something like add an environment variable GUIX_INSTALL_DIRECTORY, or something like that...
>
> What's different about GUIX_INSTALL_DIRECTORY than the usual: PREFIX?

The value of that variable would not be embedded in the binary.  It is
looked up at runtime, so the binary only contains the literal
“GUIX_INSTALL_DIRECTORY”.

--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net

Re: gcc-ddc

[Gábor Boskovits]

[-- Attachment #1: Type: text/plain, Size: 1095 bytes --]

Hello!

Before FOSDEM I decided to take another look at this.
I have really good news, gcc 4.7.4 ddc on x86_64 is now reproducible using
clang.
Code can be found at: https://github.com/Boskovits/guix/tree/gcc-ddc.

How could we go on with this?

Now it is very experimental, and not conforming to policies.
It is also architecture dependent.

I don't think that this should go into upstream guix, because I don't
believe it is useful for the general public. How should we get it into a
place more generally available?


2017-12-02 23:44 GMT+01:00 Ricardo Wurmus <rekado@elephly.net>:

>
> >> I guess I can do something like add an environment variable
> GUIX_INSTALL_DIRECTORY, or something like that...
> >
> > What's different about GUIX_INSTALL_DIRECTORY than the usual: PREFIX?
>
> The value of that variable would not be embedded in the binary.  It is
> looked up at runtime, so the binary only contains the literal
> “GUIX_INSTALL_DIRECTORY”.
>
> --
> Ricardo
>
> GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
> https://elephly.net
>
>
>

[-- Attachment #2: Type: text/html, Size: 1766 bytes --]

Re: gcc-ddc

[Ludovic Courtès]

Hello!

Gábor Boskovits <boskovits@gmail.com> skribis:

> Before FOSDEM I decided to take another look at this.
> I have really good news, gcc 4.7.4 ddc on x86_64 is now reproducible using
> clang.
> Code can be found at: https://github.com/Boskovits/guix/tree/gcc-ddc.

Woow, quite an achievement!

To be clear, this provides packages to perform DDC of GCC 4.7, right?

Did you choose 4.7 due to the fact that it’s the last C-only version?

> How could we go on with this?
>
> Now it is very experimental, and not conforming to policies.
> It is also architecture dependent.
>
> I don't think that this should go into upstream guix, because I don't
> believe it is useful for the general public. How should we get it into a
> place more generally available?

Perhaps there are pieces that (reproducibility fixes, etc.) that are
generally useful and could be merged directly.

As for the DDC packages, I don’t know.  It would probably require some
cleanup and discussion, but I think it would make sense to add it in
Guix proper.  That would allow Guix users to test their compiler through
DDC, which is not negligible.

Thanks,
Ludo’.

Re: gcc-ddc

[Ricardo Wurmus]

Ludovic Courtès <ludo@gnu.org> writes:

> Did you choose 4.7 due to the fact that it’s the last C-only version?

At the reproducible builds summit we talked about “checkpoints” that we
want to achieve, i.e. points in the build graph that one can arrive at
one way or another (in this case via clang or via GCC) but with the same
hash, so that one could confidently cut the graph at that point and
graft it onto a different root.

GCC 4.7 is one of those points; it is a desirable location in the
bootstrap graph as it doesn’t depend on a C++ compiler.

> As for the DDC packages, I don’t know.  It would probably require some
> cleanup and discussion, but I think it would make sense to add it in
> Guix proper.  That would allow Guix users to test their compiler through
> DDC, which is not negligible.

I agree.

Thanks for this great effort Gábor!

--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net