Reproducible rust builds

Reproducible rust builds

[Nikolai Merinov]

Hi,

I made some investigations about reproducible builds of rust packages
(at least of my laptop with x86_64 arch) and I got next results:

1. I failed to make rust 1.25.0 reproducible with LLVM 6.0.1, but this
release reproducible if we build it with LLVM 3.9.1. I also tried to
build rust 1.25.0 with internal LLVM and got same reproducibility issue.
2. Rust 1.26.2 reproducible with LLVM 6.0.1 regardless of which LLVM
used for rust 1.25.0. I failed to find any specific changes that allow
this version to be built in reproducible manner with LLVM 6.0.1.
3. There is a same situation with 1.27.2 as with rust 1.25.0:
Reproducible with LLVM 3.9.1, not reproducible with LLVM 6.0.1. Build
with LLVM 3.9.1 step on reproducibility issue in rustdoc, but there is
patch for this issue.
4. Rust 1.28.0 and 1.29.1 releases builds well with LLVM 6.0.1
5. From very first release we missed cargo dependency on libssh2 and
libgit2 and this dependencies was build internally by rust.

Code of reproducible packages for 1.25.0, 1.27.2, 1.28.0, and 1.29.1
rust releases you can find in
https://github.com/mnd/guix-mnd-pkgs/blob/master/mnd/packages/rust.scm

Can you recommend preferable changes for rust packages in GuixSD? I
suggest to make next changes:

1. Switch back to llvm 3.9.1 in rust 1.25.0, 1.26.2, 1.27.2 rust
packages. NOTE: External LLVM 6.0 in rust 1.26.0 and newer allow to use
#[cfg(target-feature)] to check processor features. Prior to 1.26.0
release this feature was supported only with bundled llvm.
2. Use llvm 6.0.1 for rust 1.28.0 and newer.
3. Add external libssh2 and libgit2 dependencies to rust packages.

Open question: To which release should I add dependency on libgit2 and
libssh2? It can be added to rust-1.19 still it was missed from very
beginning or to rust-1.25, because we should change rust-1.25 package in
any case.

Regards,
Nikolai

Re: Reproducible rust builds

[Joe Hillenbrand]

[-- Attachment #1: Type: text/plain, Size: 1267 bytes --]

Hi Nikolai,

I've been trying to make rust reproducible as well.

I starting with version 1.19, but I think you're right to just skip
anything older than 1.25 as it looks like a bunch of reproducibility fixes were
added to cargo and rust around 1.22.

What kind of differences are you getting from between the builds?
For me, it has been liblibc. I've attached the output of the following command:

$ diffoscope --exclude-directory-metadata
/gnu/store/hv2wb30jgzhhgfvlhrfpk3qcwr38igg4-rust-1.22.1
/gnu/store/hv2wb30jgzhhgfvlhrfpk3qcwr38igg4-rust-1.22.1-check --html
diffoscope-rust.html

On Fri, Oct 12, 2018 at 3:44 PM Nikolai Merinov
<nikolai.merinov@member.fsf.org> wrote:
> 2. Rust 1.26.2 reproducible with LLVM 6.0.1 regardless of which LLVM
> used for rust 1.25.0. I failed to find any specific changes that allow
> this version to be built in reproducible manner with LLVM 6.0.1.

How sure are you that 1.26.2 is reproducible? I noticed that sometimes it will
work after 2 rounds and sometimes it won't.

I've pulled your repo and started a build with 4 rounds just to check.
It will probably take a couple of days.

You might also try adding the attached patch I grabbed from Debian.
I haven't gotten it to work yet, but maybe it will help you.

Thanks,
-Joe

[-- Attachment #2: diffoscope-rust.html --]
[-- Type: text/html, Size: 163851 bytes --]

[-- Attachment #3: rust-no-split-dwarf.patch --]
[-- Type: text/x-patch, Size: 652 bytes --]

Description: Don't split dwarf debug for a fully-reproducible build
Author: Ximin Luo <infinity0@debian.org>
Bug: https://github.com/rust-lang/rust/issues/34902

--- a/src/librustc_llvm/build.rs
+++ b/src/librustc_llvm/build.rs
@@ -140,6 +140,11 @@
     let mut cfg = cc::Build::new();
     cfg.warnings(false);
     for flag in cxxflags.split_whitespace() {
+        // Split-dwarf gives unreproducible DW_AT_GNU_dwo_id so don't do it
+        if flag == "-gsplit-dwarf" {
+            continue;
+        }
+
         // Ignore flags like `-m64` when we're doing a cross build
         if is_crossed && flag.starts_with("-m") {
             continue;

Re: Reproducible rust builds

[Nikolai Merinov]

Hi Joe,

I think tomorrow I'll prepare patches according to my findings and then
I'll try to find issues in 1.22 release.

Joe Hillenbrand <joehillen@gmail.com> writes:

> Hi Nikolai,
>
> I've been trying to make rust reproducible as well.
>
> I starting with version 1.19, but I think you're right to just skip
> anything older than 1.25 as it looks like a bunch of reproducibility fixes were
> added to cargo and rust around 1.22.

I concentrated on 1.25 and newer. I worked on this packages in August
and before September's Danny changes initial version was 1.23.

>
> What kind of differences are you getting from between the builds?

1.25 and 1.27 suffer from this type of non-reprodicibility
https://github.com/rust-lang/rust/issues/50556
There is no suggestions how to solve this issue and I have no idea how
to continue with it. As far as I understand we step on non-determenistic
behavior of llvm 6, so I suggest to use llvm 3.9 for this versions.

1.27, 1.28, 1.29 have next issue
https://github.com/rust-lang/rust/issues/24473
It can be fixed with
rust-mdbook-Support-reproducible-builds-by-forcing-window.search.patch
in my packages.

> For me, it has been liblibc. I've attached the output of the following command:
>
> $ diffoscope --exclude-directory-metadata
> /gnu/store/hv2wb30jgzhhgfvlhrfpk3qcwr38igg4-rust-1.22.1
> /gnu/store/hv2wb30jgzhhgfvlhrfpk3qcwr38igg4-rust-1.22.1-check --html
> diffoscope-rust.html
>
> On Fri, Oct 12, 2018 at 3:44 PM Nikolai Merinov
> <nikolai.merinov@member.fsf.org> wrote:
>> 2. Rust 1.26.2 reproducible with LLVM 6.0.1 regardless of which LLVM
>> used for rust 1.25.0. I failed to find any specific changes that allow
>> this version to be built in reproducible manner with LLVM 6.0.1.
>
> How sure are you that 1.26.2 is reproducible? I noticed that sometimes it will
> work after 2 rounds and sometimes it won't.
>

I tried several times and never catched issue after two rounds. Which
files was different in your builds?

> I've pulled your repo and started a build with 4 rounds just to check.
> It will probably take a couple of days.
>
> You might also try adding the attached patch I grabbed from Debian.
> I haven't gotten it to work yet, but maybe it will help you.

This patch is useless for Guix: This patch required when "llvm-config -cxxflags"
returns "-gsplit-dwarf" key. This behavior specific to debian patches to
llvm.

>
> Thanks,
> -Joe

Re: Reproducible rust builds

[Danny Milosavljevic]

[-- Attachment #1: Type: text/plain, Size: 1854 bytes --]

Hi Nikolai,
Hi Joe,

On Tue, 16 Oct 2018 02:05:45 +0500
Nikolai Merinov <nikolai.merinov@member.fsf.org> wrote:

> I concentrated on 1.25 and newer.

Be advised that the wip-rust branch contains a (finished) full source bootstrap
for rust which starts with rust 1.19.  We should coordinate when we merge what
to master so we don't make the reproducibility problems more difficult to track
down.

I'd prefer to first get the rust reproducibility on master under control and
only later merge wip-rust.

> 1.25 and 1.27 suffer from this type of non-reprodicibility
> https://github.com/rust-lang/rust/issues/50556

I confirm that problem.

>1.25 and 1.27 suffer from this type of non-reprodicibility
>https://github.com/rust-lang/rust/issues/50556
>There is no suggestions how to solve this issue and I have no idea how
>to continue with it. As far as I understand we step on non-determenistic
>behavior of llvm 6, so I suggest to use llvm 3.9 for this versions.

I agree, especially if the problem is fixed in later Rust versions
and if they build reproducibly with newer LLVM versions.

> 1.27, 1.28, 1.29 have next issue
> https://github.com/rust-lang/rust/issues/24473
> It can be fixed with
> rust-mdbook-Support-reproducible-builds-by-forcing-window.search.patch
> in my packages.

That looks like a simple patch.  Nice!

> > For me, it has been liblibc. I've attached the output of the following command:
> >
> > $ diffoscope --exclude-directory-metadata
> > /gnu/store/hv2wb30jgzhhgfvlhrfpk3qcwr38igg4-rust-1.22.1
> > /gnu/store/hv2wb30jgzhhgfvlhrfpk3qcwr38igg4-rust-1.22.1-check --html
> > diffoscope-rust.html

Yeah, there are lots of libc differences for me too between the rounds.

It's cool that upstream is quite motivated to fix reproducibility problems, so
let's keep reporting issues there.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Reproducible rust builds

[Joe Hillenbrand]

On Mon, Oct 15, 2018 at 1:21 PM Joe Hillenbrand <joehillen@gmail.com> wrote:
>
> I've pulled your repo and started a build with 4 rounds just to check.
> It will probably take a couple of days.
>

FYI, the build completed without issue.