bug#20272: Support reproducible builds

[ludo@gnu.org (Ludovic Courtès)]

Mark H Weaver <mhw@netris.org> skribis:

> ludo@gnu.org (Ludovic Courtès) writes:
>
>> Currently .go files embed randomly-generated symbols stemming from
>> ‘syntax-session-id’, which prevents reproducible builds (see
>> <https://lists.gnu.org/archive/html/guix-devel/2013-09/msg00159.html>.)
>>
>> One way to fix it would be to allow users to specify a random seed used
>> when generating session ids, and to make that available as a
>> command-line option to ‘guild compile’.  (GCC does something similar
>> with its ‘-frandom-seed’ option.)
>
> We could add this, but it is not analogous to the -frandom-seed option
> where it is okay to give it the same value everywhere.  Users would need
> to ensure that distinct session-ids are used for every invocation of
> Guile.

With GCC the common idiom is to use ‘-frandom-seed=$source_file’.

However, it would be best if ‘guild compile’ would choose the seed
deterministically by default somehow, because we cannot expect all users
to add the new flag and use properly.

What about having ‘guild compile’ use the canonical file name of the
source being compiled (or a hash thereof) as the seed?

> More precisely, users of this feature would need to observe the
> following restriction, or else unspecified behavior may result:
>
>   If   A.go is generated by a Guile session with session-id A, and
>        B.go is generated by a Guile session with session-id B, and
>   they are both loaded into a Guile session with session-id C, then
>   A, B, and C must all be distinct session-ids.

Right.

I wonder if we could detect collisions.  Ideally each .go could record
its session ID, but that’s probably not feasible in 2.0.

> One more thing: even with a deterministic session-id, the multi-threaded
> compiling of *.go files recently added to Guix will lead to
> non-deterministic outputs.  I'm not sure how to make this work
> deterministically with concurrency.  Even if we switched to a fixed
> number of concurrent processes with one thread each, when process A
> attempts to load a .go file that is produced by process B, it is not
> deterministic whether it will be there.  If it is there, the macros in
> that .go file will contain B's session-id, and if not, they will contain
> A's session-id.

Hmm, OK.  Well, let’s keep this use case aside for now.

>> Probably, ‘syntax-session-id’ would have to be a SRFI-39 parameter
>
> This might adversely affect the efficiency of our macro expander on
> platforms with slow thread local variables, and I'm not sure what it
> would buy us.  If the idea is that it would allow us to build things in
> multiple threads, I think that won't work anyway, for the reasons given
> above.

I was just thinking that it would be a convenient interface for (scripts
compile) to specify the session ID to use.

I think the first thing to do is to change ‘fresh_syntax_session_id’ so
that they can use a user-specified seed, when available, instead of
‘scm_i_random_bytes_from_platform’.

WDYT?

Thanks for working on it!  :-)

Ludo’.