From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:59851) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gDbsN-0000dP-CM for guix-patches@gnu.org; Fri, 19 Oct 2018 17:00:09 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gDbsJ-0000kw-Fx for guix-patches@gnu.org; Fri, 19 Oct 2018 17:00:07 -0400 Received: from debbugs.gnu.org ([208.118.235.43]:55574) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1gDbsJ-0000kN-5w for guix-patches@gnu.org; Fri, 19 Oct 2018 17:00:03 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1gDbsJ-0003kV-3t for guix-patches@gnu.org; Fri, 19 Oct 2018 17:00:03 -0400 Subject: [bug#33066] [PATCHv3] gnu: rust: workaround rust 1.25-27 reproducibility issues Resent-Message-ID: From: Nikolai Merinov References: <877eihu5hx.fsf@member.fsf.org> <875zy1u5ec.fsf@member.fsf.org> <20181017131406.01c3b011@scratchpost.org> <87y3atsouq.fsf@member.fsf.org> Date: Sat, 20 Oct 2018 01:58:43 +0500 In-Reply-To: <87y3atsouq.fsf@member.fsf.org> (Nikolai Merinov's message of "Sat, 20 Oct 2018 00:44:45 +0500") Message-ID: <87sh11slfg.fsf_-_@member.fsf.org> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+kyle=kyleam.com@gnu.org Sender: "Guix-patches" To: Danny Milosavljevic Cc: 33066@debbugs.gnu.org --=-=-= Content-Type: text/plain Hi, Sorry, ignore my previous mail. There was wrong patch export. Re-attached patches. --=-=-= Content-Type: text/x-patch Content-Disposition: attachment; filename=0001-gnu-rust-workaround-rust-1.25-27-reproducibility-iss.patch >From 6e593ffb8c1d9ecf4a1ed5e131a5c352a76fe986 Mon Sep 17 00:00:00 2001 From: Nikolai Merinov Date: Tue, 16 Oct 2018 02:32:11 +0500 Subject: [PATCH 1/2] gnu: rust: workaround rust 1.25-27 reproducibility issues To: guix-patches@gnu.org * gnu/packages/patches/rust-mdbook-support-reproducible-builds-by-forcing-window.search.patch: New file. * gnu/local.mk (dist_patch_DATA): Add new patch file. * gnu/packages/rust.scm (rust-1.19): Use system libssh2 library during cargo build. Note: libgit2 still bundled, because cargo tests assume specific libgit2 minor release. (rust-1.23): Inherit native-inputs from previous package. (rust-1.25): Switch back to llvm 3.9.1 as workaround for https://github.com/rust-lang/rust/issues/50556 issue. (rust-1.27): Apply changes from https://github.com/rust-lang-nursery/mdBook/pull/692 to make generation of "searchindex.js" files reproducible. Disable cargo test that required llvm 6. --- gnu/local.mk | 1 + ...ible-builds-by-forcing-window.search.patch | 28 +++++++++++++ gnu/packages/rust.scm | 41 ++++++++----------- 3 files changed, 47 insertions(+), 23 deletions(-) create mode 100644 gnu/packages/patches/rust-mdbook-support-reproducible-builds-by-forcing-window.search.patch diff --git a/gnu/local.mk b/gnu/local.mk index 1fa2eaba3..8695ce472 100644 --- a/gnu/local.mk +++ b/gnu/local.mk @@ -1130,6 +1130,7 @@ dist_patch_DATA = \ %D%/packages/patches/rust-1.25-accept-more-detailed-gdb-lines.patch \ %D%/packages/patches/rust-bootstrap-stage0-test.patch \ %D%/packages/patches/rust-coresimd-doctest.patch \ + %D%/packages/patches/rust-mdbook-support-reproducible-builds-by-forcing-window.search.patch \ %D%/packages/patches/rxvt-unicode-escape-sequences.patch \ %D%/packages/patches/scheme48-tests.patch \ %D%/packages/patches/scotch-test-threading.patch \ diff --git a/gnu/packages/patches/rust-mdbook-support-reproducible-builds-by-forcing-window.search.patch b/gnu/packages/patches/rust-mdbook-support-reproducible-builds-by-forcing-window.search.patch new file mode 100644 index 000000000..b4ae3a4b8 --- /dev/null +++ b/gnu/packages/patches/rust-mdbook-support-reproducible-builds-by-forcing-window.search.patch @@ -0,0 +1,28 @@ +From b9ca108fcae2b738ca3f0c88c84ae5dc5a6f843f Mon Sep 17 00:00:00 2001 +From: Tim Ryan +Date: Mon, 14 May 2018 06:22:21 -0400 +Subject: [PATCH] Support reproducible builds by forcing window.search to use + stable key ordering. (#692) +See +--- + src/vendor/mdbook/src/renderer/html_handlebars/search.rs | 4 ++++ + 1 file changed, 4 insertions(+) + +diff --git a/src/vendor/mdbook/src/renderer/html_handlebars/search.rs b/src/vendor/mdbook/src/renderer/html_handlebars/search.rs +index d49772f8b..1ee66a511 100644 +--- a/src/vendor/mdbook/src/renderer/html_handlebars/search.rs ++++ b/src/vendor/mdbook/src/renderer/html_handlebars/search.rs +@@ -205,6 +205,10 @@ fn write_to_js(index: Index, search_config: &Search) -> Result { + searchoptions, + index, + }; ++ ++ // By converting to serde_json::Value as an intermediary, we use a ++ // BTreeMap internally and can force a stable ordering of map keys. ++ let json_contents = serde_json::to_value(&json_contents)?; + let json_contents = serde_json::to_string(&json_contents)?; + + Ok(format!("window.search = {};", json_contents)) +-- +2.19.0 + diff --git a/gnu/packages/rust.scm b/gnu/packages/rust.scm index 40633854a..cd2a8fb9a 100644 --- a/gnu/packages/rust.scm +++ b/gnu/packages/rust.scm @@ -480,7 +480,8 @@ test = { path = \"../libtest\" } `(("jemalloc" ,jemalloc-4.5.0) ("llvm" ,llvm-3.9.1) ("openssl" ,openssl) - ("libcurl" ,curl))) ; For "cargo" + ("libssh2" ,libssh2) ; For "cargo" + ("libcurl" ,curl))) ; For "cargo" ;; rustc invokes gcc, so we need to set its search paths accordingly. ;; Note: duplicate its value here to cope with circular dependencies among @@ -649,18 +650,11 @@ jemalloc = \"" jemalloc "/lib/libjemalloc_pic.a" "\" (name "rust") (version "1.23.0") (source (rust-source version "14fb8vhjzsxlbi6yrn1r6fl5dlbdd1m92dn5zj5gmzfwf4w9ar3l")) + ;; Use rust-bootstrap@1.22 package to build rust 1.23 (native-inputs - `(("bison" ,bison) ; For the tests - ("cmake" ,cmake) - ("flex" ,flex) ; For the tests - ("gdb" ,gdb) ; For the tests - ("git" ,git) - ("procps" ,procps) ; For the tests - ("python-2" ,python-2) - ("rustc-bootstrap" ,rust-bootstrap) - ("cargo-bootstrap" ,rust-bootstrap "cargo") - ("pkg-config" ,pkg-config) ; For "cargo" - ("which" ,which))) + (alist-replace "cargo-bootstrap" (list rust-bootstrap "cargo") + (alist-replace "rustc-bootstrap" (list rust-bootstrap) + (package-native-inputs rust-1.20)))) (arguments (substitute-keyword-arguments (package-arguments rust-1.20) ((#:phases phases) @@ -692,6 +686,9 @@ jemalloc = \"" jemalloc "/lib/libjemalloc_pic.a" "\" (("fn test_loading_cosine") "#[ignore]\nfn test_loading_cosine")) #t))))))))) +;;; Rust 1.25 release support work with llvm 6, but builds with llvm 6 is +;;; not determenistic due to https://github.com/rust-lang/rust/issues/50556 +;;; Keep using llvm 3.9.1 until builds become determenistic (define-public rust-1.25 (let ((base-rust (rust-bootstrapped-package rust-1.24 "1.25.0" @@ -699,10 +696,6 @@ jemalloc = \"" jemalloc "/lib/libjemalloc_pic.a" "\" #:patches '("rust-1.25-accept-more-detailed-gdb-lines.patch")))) (package (inherit base-rust) - (inputs - ;; Use LLVM 6.0 - (alist-replace "llvm" (list llvm) - (package-inputs base-rust))) (arguments (substitute-keyword-arguments (package-arguments base-rust) ((#:phases phases) @@ -713,14 +706,9 @@ jemalloc = \"" jemalloc "/lib/libjemalloc_pic.a" "\" ;; This test wants to update the crate index. (("fn no_index_update") "#[ignore]\nfn no_index_update")) #t)) - (add-after 'configure 'enable-codegen-tests - (lambda _ - (substitute* "config.toml" - (("codegen-tests = false") "")) - #t)) ;; FIXME: Re-enable this test if it's indeed supposed to work. ;; See . - (add-after 'enable-codegen-tests 'disable-nil-enum-test + (add-after 'configure 'disable-nil-enum-test (lambda _ (substitute* "src/test/debuginfo/nil-enum.rs" (("ignore-lldb") "ignore-gdb")) @@ -795,7 +783,8 @@ jemalloc = \"" jemalloc "/lib/libjemalloc_pic.a" "\" #:patches '("rust-coresimd-doctest.patch" "rust-bootstrap-stage0-test.patch" - "rust-1.25-accept-more-detailed-gdb-lines.patch")))) + "rust-1.25-accept-more-detailed-gdb-lines.patch" + "rust-mdbook-support-reproducible-builds-by-forcing-window.search.patch")))) (package (inherit base-rust) (arguments @@ -808,4 +797,10 @@ jemalloc = \"" jemalloc "/lib/libjemalloc_pic.a" "\" ;; `prefix' directory should exist before `install' call (mkdir-p (assoc-ref outputs "out")) (mkdir-p (assoc-ref outputs "cargo")) + #t)) + (add-after 'patch-cargo-tests 'disable-thinlto-test + (lambda* _ + ;; thinlto required llvm 6.0 for work + (substitute* "src/tools/cargo/tests/testsuite/path.rs" + (("fn thin_lto_works") "#[ignore]\nfn thin_lto_works")) #t))))))))) -- 2.19.0 --=-=-= Content-Type: text/x-patch Content-Disposition: attachment; filename=0002-gnu-rust-add-rust-1.28-and-rust-1.29-packages.patch >From 4eb82de9971c13323e7b0195f9f178282039ba35 Mon Sep 17 00:00:00 2001 From: Nikolai Merinov Date: Tue, 16 Oct 2018 02:37:41 +0500 Subject: [PATCH 2/2] gnu: rust: add rust 1.28 and rust 1.29 packages To: guix-patches@gnu.org * gnu/packages/rust.scm (rust-1.27): rename "rust" to "rust-1.27" (rust-1.28): Use llvm-6.0 finally. Disable test that broken on x86_64 machines without "avx". Enable codegen tests. Enable cargo test for thinlto. (rust): Package for rust 1.29.2. --- gnu/packages/rust.scm | 46 ++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 45 insertions(+), 1 deletion(-) diff --git a/gnu/packages/rust.scm b/gnu/packages/rust.scm index cd2a8fb9a..aafa3704b 100644 --- a/gnu/packages/rust.scm +++ b/gnu/packages/rust.scm @@ -776,7 +776,7 @@ jemalloc = \"" jemalloc "/lib/libjemalloc_pic.a" "\" (("fn no_index_update") "#[ignore]\nfn no_index_update")) #t))))))))) -(define-public rust +(define-public rust-1.27 (let ((base-rust (rust-bootstrapped-package rust-1.26 "1.27.2" "0pg1s37bhx9zqbynxyydq5j6q7kij9vxkcv8maz0m25prm88r0cs" @@ -804,3 +804,47 @@ jemalloc = \"" jemalloc "/lib/libjemalloc_pic.a" "\" (substitute* "src/tools/cargo/tests/testsuite/path.rs" (("fn thin_lto_works") "#[ignore]\nfn thin_lto_works")) #t))))))))) + +(define-public rust-1.28 + (let ((base-rust + (rust-bootstrapped-package rust-1.27 "1.28.0" + "11k4rn77bca2rikykkk9fmprrgjswd4x4kaq7fia08vgkir82nhx" + #:patches + '("rust-coresimd-doctest.patch" + "rust-bootstrap-stage0-test.patch" + "rust-1.25-accept-more-detailed-gdb-lines.patch" + "rust-mdbook-support-reproducible-builds-by-forcing-window.search.patch")))) + (package + (inherit base-rust) + (inputs + ;; Use LLVM 6.0 + (alist-replace "llvm" (list llvm) + (package-inputs base-rust))) + (arguments + (substitute-keyword-arguments (package-arguments base-rust) + ((#:phases phases) + `(modify-phases ,phases + (add-after 'configure 'enable-codegen-tests + ;; codegen tests should pass with llvm 6 + (lambda* _ + (substitute* "config.toml" + (("codegen-tests = false") "")) + #t)) + (add-after 'patch-tests 'disable-amd64-avx-test + ;; This test will fail on x86_64 machines without avx + (lambda* _ + (substitute* "src/test/run-pass/issue-44056.rs" + (("only-x86_64") "ignore-test")) + #t)) + ;; thinlto test should pass with llvm 6 + (delete 'disable-thinlto-test)))))))) + +(define-public rust + (let ((base-rust + (rust-bootstrapped-package rust-1.28 "1.29.2" + "1jb787080z754caa2w3w1amsygs4qlzj9rs1vy64firfmabfg22h" + #:patches + '("rust-1.25-accept-more-detailed-gdb-lines.patch" + "rust-mdbook-support-reproducible-builds-by-forcing-window.search.patch")))) + (package + (inherit base-rust)))) -- 2.19.0 --=-=-= Content-Type: text/plain Regards, Nikolai Nikolai Merinov writes: > Hi Danny, > > I fixed patches according to your comments. Also I updated rust 1.29.1 > to 1.29.2 and fixed test issue in 1.27: as I mentioned before I tested > reproducibility of each package previously only in private my private > repo where I disabled `check` phase for 1.25 and 1.27. I have pretty > slow machine, but currently I already checked reproducibility with > suggested patches for 1.23-1.26 releases. 1.27-1.29 in progress. > > Both updated patches attached. > > Danny Milosavljevic writes: > >> Hi Nikolai, >> >> On Tue, 16 Oct 2018 02:32:11 +0500 >> Nikolai Merinov wrote: >> >>> * gnu/packages/patches/rust-mdbook-Support-reproducible-builds-by-forcing-window.search.patch: >> >> Nitpick: No big "S" (file names are easier to find if they are all lower case). >> > Fixed. > >>> patch that make "searchindex.js" reproducible in rust 1.27 and newer. >> >> "New file". > Fixed. > >> >>> * gnu/local.mk (dist_patch_DATA): Add new patch file. >>> * gnu/packages/rust.scm (rust-1.19): Use system libssh2 library >> >> Hmm, I'm not sure about doing this in the same commit. >> Is it also related to reproducibility? > > Looks like it should not, but still I started investigation with > non-reproducible "libssh2" and "libgit2" rust libraries I made libssh2 > related changes at very beginning and never tested wihtout it. I not > sure that it's good idea to remove it now. > >> >>> during cargo build. Note: libgit2 still builded as part of cargo build, >>> because cargo tests assume specific libgit2 minor release. >> >> What does this mean? Does it mean "bundled"? > Yes. Rust used bundled sources for "libgit2-sys" rust library (used by cargo). > >> >>> (rust-1.23): inherit native-inputs from previous package. >> >> Ok. >> >>> (rust-1.25): switch back to llvm 3.9.1 as workaround for >>> https://github.com/rust-lang/rust/issues/50556 issue. >> >> Please add the reasoning as a comment inside the source code instead. > Added comment for rust-1.25 package. > >> >>> (rust-1.27): apply patch to make "searchindex.js" files reproducible. >> >> Maybe add "[source]". > I repharase this comment and added source URL to patch file itself. > >> >>> - (add-after 'configure 'enable-codegen-tests >>> - (lambda _ >>> - (substitute* "config.toml" >>> - (("codegen-tests = false") "")) >>> - #t)) >> >> I think I had reproducibility problems when enabling codegen tests and >> parallel tests. Is that not the case anymore? > Neither me nor Joe Hillenbrand in > https://lists.gnu.org/archive/html/guix-devel/2018-10/msg00292.html got > reproducibility issues with this tests. >> >>> ;; FIXME: Re-enable this test if it's indeed supposed to work. >>> ;; See . >> >> Note to myself: I think the issue comments indicate that the newer gdb output >> is better - so we should create a patch similar to >> rust-1.25-accept-more-detailed-gdb-lines.patch to accept the newer output. --=-=-=--