bug#57593: Racket 8.6 is not reproducible

bug#57593: Racket 8.6 is not reproducible

[Ludovic Courtès]

Hey,

Just noticed that Racket does not build in a reproducible fashion:

--8<---------------cut here---------------start------------->8---
$ guix describe
Generation 227  Sep 04 2022 23:39:52    (current)
  guix aae98c2
    repository URL: https://git.savannah.gnu.org/git/guix.git
    branch: master
    commit: aae98c297214f87eb45302863adb021078c41a6f
$ guix challenge racket | head -40
/gnu/store/av255rh362283i1zaiq9rz4rpli69j59-racket-8.6 contents differ:
  local hash: 1w4dnkvpbbrgfasyq8x1cbqw58jzqsny17ms5l1fb1h6iid38bs1
  https://ci.guix.gnu.org/nar/lzip/av255rh362283i1zaiq9rz4rpli69j59-racket-8.6: 1lnxklizpnc599w7n2svb1jaw595wranm9aagd2928fcbiaavbr6
  differing files:
    /lib/racket/pkgs/2d-doc/scribblings/compiled/2d_scrbl.dep
    /lib/racket/pkgs/2d-doc/scribblings/compiled/2d_scrbl.zo
    /lib/racket/pkgs/algol60/compiled/algol60_rkt.dep
    /lib/racket/pkgs/algol60/compiled/algol60_scrbl.dep
    /lib/racket/pkgs/algol60/compiled/algol60_scrbl.zo
    /lib/racket/pkgs/algol60/compiled/cfg-parser_rkt.dep
    /lib/racket/pkgs/algol60/compiled/compile_rkt.dep
    /lib/racket/pkgs/algol60/compiled/parse_rkt.dep
    /lib/racket/pkgs/algol60/compiled/parse_rkt.zo
    /lib/racket/pkgs/algol60/compiled/simplify_rkt.dep
    /lib/racket/pkgs/algol60/compiled/tool_rkt.dep
    /lib/racket/pkgs/algol60/lang/compiled/reader_rkt.dep
    /lib/racket/pkgs/algol60/tests/compiled/export_rkt.dep
    /lib/racket/pkgs/algol60/tests/compiled/export_rkt.zo
    /lib/racket/pkgs/algol60/tests/compiled/syncheck-test_rkt.dep
    /lib/racket/pkgs/algol60/tests/compiled/test_rkt.dep
    /lib/racket/pkgs/compatibility-doc/mzlib/scribblings/compiled/awk_scrbl.dep
    /lib/racket/pkgs/compatibility-doc/mzlib/scribblings/compiled/awk_scrbl.zo
    /lib/racket/pkgs/compatibility-doc/mzlib/scribblings/compiled/cmdline_scrbl.dep
    /lib/racket/pkgs/compatibility-doc/mzlib/scribblings/compiled/cmdline_scrbl.zo
    /lib/racket/pkgs/compatibility-doc/mzlib/scribblings/compiled/cml_scrbl.dep
    /lib/racket/pkgs/compatibility-doc/mzlib/scribblings/compiled/cml_scrbl.zo
    /lib/racket/pkgs/compatibility-doc/mzlib/scribblings/compiled/common_rkt.dep
    /lib/racket/pkgs/compatibility-doc/mzlib/scribblings/compiled/common_rkt.zo
    /lib/racket/pkgs/compatibility-doc/mzlib/scribblings/compiled/compat_scrbl.dep
    /lib/racket/pkgs/compatibility-doc/mzlib/scribblings/compiled/compat_scrbl.zo
    /lib/racket/pkgs/compatibility-doc/mzlib/scribblings/compiled/compile_scrbl.dep
    /lib/racket/pkgs/compatibility-doc/mzlib/scribblings/compiled/compile_scrbl.zo
    /lib/racket/pkgs/compatibility-doc/mzlib/scribblings/compiled/contract-label_rkt.dep
    /lib/racket/pkgs/compatibility-doc/mzlib/scribblings/compiled/contract_scrbl.dep
    /lib/racket/pkgs/compatibility-doc/mzlib/scribblings/compiled/contract_scrbl.zo
    /lib/racket/pkgs/compatibility-doc/mzlib/scribblings/compiled/etc_scrbl.dep
    /lib/racket/pkgs/compatibility-doc/mzlib/scribblings/compiled/etc_scrbl.zo
    /lib/racket/pkgs/compatibility-doc/mzlib/scribblings/compiled/file_scrbl.dep
    /lib/racket/pkgs/compatibility-doc/mzlib/scribblings/compiled/file_scrbl.zo
    /lib/racket/pkgs/compatibility-doc/mzlib/scribblings/compiled/for_scrbl.dep
    /lib/racket/pkgs/compatibility-doc/mzlib/scribblings/compiled/for_scrbl.zo
    /lib/racket/pkgs/compatibility-doc/mzlib/scribblings/compiled/include_scrbl.dep
    /lib/racket/pkgs/compatibility-doc/mzlib/scribblings/compiled/include_scrbl.zo
--8<---------------cut here---------------end--------------->8---

Ludo’.

bug#57593: Racket 8.6 is not reproducible

[Philip McGrath]

Hi,

On Mon, Sep 5, 2022, at 6:04 AM, Ludovic Courtès wrote:
> Hey,
>
> Just noticed that Racket does not build in a reproducible fashion:
>
> --8<---------------cut here---------------start------------->8---
> $ guix describe
> Generation 227  Sep 04 2022 23:39:52    (current)
>   guix aae98c2
>     repository URL: https://git.savannah.gnu.org/git/guix.git
>     branch: master
>     commit: aae98c297214f87eb45302863adb021078c41a6f
> $ guix challenge racket | head -40
> /gnu/store/av255rh362283i1zaiq9rz4rpli69j59-racket-8.6 contents differ:
>   local hash: 1w4dnkvpbbrgfasyq8x1cbqw58jzqsny17ms5l1fb1h6iid38bs1
>   
> https://ci.guix.gnu.org/nar/lzip/av255rh362283i1zaiq9rz4rpli69j59-racket-8.6: 
> 1lnxklizpnc599w7n2svb1jaw595wranm9aagd2928fcbiaavbr6
>   differing files:
>     /lib/racket/pkgs/2d-doc/scribblings/compiled/2d_scrbl.dep
>     /lib/racket/pkgs/2d-doc/scribblings/compiled/2d_scrbl.zo

Unfortunately, this is a known issue upstream.

One part of the problem is the way that Chez Scheme uses "type 4" (random) UUIDs to implement 'gensym'. There's some upstream discussion at <https://github.com/cisco/ChezScheme/issues/585>, including an idea from me about "type 5" UUIDs as part of a possible solution.

That may well be the only issue with the core compiler for Racket CS. I know effort has been taken to make Racket's front-end deterministic, and Kent Dybvig says that:

> Chez Scheme compiles are reproducible modulo the globally unique
> names of generated symbols (gensyms), assuming macros don't do
> things like introduce time stamps or random expressions into their output.
> Because of differences in gensym names, the files produced by multiple
> runs are not byte-by-byte identical but should compare equal via
> the system primitive $fasl-file-equal? See, for example, the use
> of $fasl-file-equal? in the s/Mf-base checkboot target.

In the output you sent, the ".zo" files contain Chez machine code with a Racket-specific wrapper, and the ".dep" files contain hashes of dependencies (so they differ when the ".zo" files they track differ).

Debian reports at <https://tests.reproducible-builds.org/debian/rb-pkg/unstable/amd64/racket.html>, discussing Racket 8.2+dfsg1-2:

> Build path captured in html documentation. docindex.sqlite
> contains content in arbitrary order. strings in binaries in
> arbitrary order.

Those issues don't appear to affect Guix: some might be caused by Debian conflating "build" and "install" for Racket packages, and, in any case, they seem related to the "scribble" and/or "racket-index" packages, not the core compiler.

IIRC, there were some reproducibility issues with the Racket BC bytecode compiler. If we can build reproducibly with CS but not BC, it might work out to disable compilation for the bootstrapping BC compilers, or maybe to configure them to use the machine-independent format (basically, a serialized form of what the front-end would hand off to the back-end), at the cost of some performance.

I hope to work on these issues, but I plan to work on a 'racket-build-system' first. Among other things, I hope it will be easier to investigate issues when Racket packages can be built independently.

-Philip