From: Nikolai Merinov <nikolai.merinov@member.fsf.org>
To: Danny Milosavljevic <dannym@scratchpost.org>
Cc: 33066@debbugs.gnu.org
Subject: [bug#33066] [PATCHv3] gnu: rust: workaround rust 1.25-27 reproducibility issues
Date: Sat, 20 Oct 2018 01:58:43 +0500 [thread overview]
Message-ID: <87sh11slfg.fsf_-_@member.fsf.org> (raw)
In-Reply-To: <87y3atsouq.fsf@member.fsf.org> (Nikolai Merinov's message of "Sat, 20 Oct 2018 00:44:45 +0500")
[-- Attachment #1: Type: text/plain, Size: 89 bytes --]
Hi,
Sorry, ignore my previous mail. There was wrong patch
export. Re-attached patches.
[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-gnu-rust-workaround-rust-1.25-27-reproducibility-iss.patch --]
[-- Type: text/x-patch, Size: 8174 bytes --]
From 6e593ffb8c1d9ecf4a1ed5e131a5c352a76fe986 Mon Sep 17 00:00:00 2001
From: Nikolai Merinov <nikolai.merinov@member.fsf.org>
Date: Tue, 16 Oct 2018 02:32:11 +0500
Subject: [PATCH 1/2] gnu: rust: workaround rust 1.25-27 reproducibility issues
To: guix-patches@gnu.org
* gnu/packages/patches/rust-mdbook-support-reproducible-builds-by-forcing-window.search.patch:
New file.
* gnu/local.mk (dist_patch_DATA): Add new patch file.
* gnu/packages/rust.scm (rust-1.19): Use system libssh2 library
during cargo build. Note: libgit2 still bundled, because cargo
tests assume specific libgit2 minor release.
(rust-1.23): Inherit native-inputs from previous package.
(rust-1.25): Switch back to llvm 3.9.1 as workaround for
https://github.com/rust-lang/rust/issues/50556 issue.
(rust-1.27): Apply changes from
https://github.com/rust-lang-nursery/mdBook/pull/692 to make
generation of "searchindex.js" files reproducible. Disable cargo
test that required llvm 6.
---
gnu/local.mk | 1 +
...ible-builds-by-forcing-window.search.patch | 28 +++++++++++++
gnu/packages/rust.scm | 41 ++++++++-----------
3 files changed, 47 insertions(+), 23 deletions(-)
create mode 100644 gnu/packages/patches/rust-mdbook-support-reproducible-builds-by-forcing-window.search.patch
diff --git a/gnu/local.mk b/gnu/local.mk
index 1fa2eaba3..8695ce472 100644
--- a/gnu/local.mk
+++ b/gnu/local.mk
@@ -1130,6 +1130,7 @@ dist_patch_DATA = \
%D%/packages/patches/rust-1.25-accept-more-detailed-gdb-lines.patch \
%D%/packages/patches/rust-bootstrap-stage0-test.patch \
%D%/packages/patches/rust-coresimd-doctest.patch \
+ %D%/packages/patches/rust-mdbook-support-reproducible-builds-by-forcing-window.search.patch \
%D%/packages/patches/rxvt-unicode-escape-sequences.patch \
%D%/packages/patches/scheme48-tests.patch \
%D%/packages/patches/scotch-test-threading.patch \
diff --git a/gnu/packages/patches/rust-mdbook-support-reproducible-builds-by-forcing-window.search.patch b/gnu/packages/patches/rust-mdbook-support-reproducible-builds-by-forcing-window.search.patch
new file mode 100644
index 000000000..b4ae3a4b8
--- /dev/null
+++ b/gnu/packages/patches/rust-mdbook-support-reproducible-builds-by-forcing-window.search.patch
@@ -0,0 +1,28 @@
+From b9ca108fcae2b738ca3f0c88c84ae5dc5a6f843f Mon Sep 17 00:00:00 2001
+From: Tim Ryan <id@timryan.org>
+Date: Mon, 14 May 2018 06:22:21 -0400
+Subject: [PATCH] Support reproducible builds by forcing window.search to use
+ stable key ordering. (#692)
+See <https://github.com/rust-lang-nursery/mdBook/pull/692>
+---
+ src/vendor/mdbook/src/renderer/html_handlebars/search.rs | 4 ++++
+ 1 file changed, 4 insertions(+)
+
+diff --git a/src/vendor/mdbook/src/renderer/html_handlebars/search.rs b/src/vendor/mdbook/src/renderer/html_handlebars/search.rs
+index d49772f8b..1ee66a511 100644
+--- a/src/vendor/mdbook/src/renderer/html_handlebars/search.rs
++++ b/src/vendor/mdbook/src/renderer/html_handlebars/search.rs
+@@ -205,6 +205,10 @@ fn write_to_js(index: Index, search_config: &Search) -> Result<String> {
+ searchoptions,
+ index,
+ };
++
++ // By converting to serde_json::Value as an intermediary, we use a
++ // BTreeMap internally and can force a stable ordering of map keys.
++ let json_contents = serde_json::to_value(&json_contents)?;
+ let json_contents = serde_json::to_string(&json_contents)?;
+
+ Ok(format!("window.search = {};", json_contents))
+--
+2.19.0
+
diff --git a/gnu/packages/rust.scm b/gnu/packages/rust.scm
index 40633854a..cd2a8fb9a 100644
--- a/gnu/packages/rust.scm
+++ b/gnu/packages/rust.scm
@@ -480,7 +480,8 @@ test = { path = \"../libtest\" }
`(("jemalloc" ,jemalloc-4.5.0)
("llvm" ,llvm-3.9.1)
("openssl" ,openssl)
- ("libcurl" ,curl))) ; For "cargo"
+ ("libssh2" ,libssh2) ; For "cargo"
+ ("libcurl" ,curl))) ; For "cargo"
;; rustc invokes gcc, so we need to set its search paths accordingly.
;; Note: duplicate its value here to cope with circular dependencies among
@@ -649,18 +650,11 @@ jemalloc = \"" jemalloc "/lib/libjemalloc_pic.a" "\"
(name "rust")
(version "1.23.0")
(source (rust-source version "14fb8vhjzsxlbi6yrn1r6fl5dlbdd1m92dn5zj5gmzfwf4w9ar3l"))
+ ;; Use rust-bootstrap@1.22 package to build rust 1.23
(native-inputs
- `(("bison" ,bison) ; For the tests
- ("cmake" ,cmake)
- ("flex" ,flex) ; For the tests
- ("gdb" ,gdb) ; For the tests
- ("git" ,git)
- ("procps" ,procps) ; For the tests
- ("python-2" ,python-2)
- ("rustc-bootstrap" ,rust-bootstrap)
- ("cargo-bootstrap" ,rust-bootstrap "cargo")
- ("pkg-config" ,pkg-config) ; For "cargo"
- ("which" ,which)))
+ (alist-replace "cargo-bootstrap" (list rust-bootstrap "cargo")
+ (alist-replace "rustc-bootstrap" (list rust-bootstrap)
+ (package-native-inputs rust-1.20))))
(arguments
(substitute-keyword-arguments (package-arguments rust-1.20)
((#:phases phases)
@@ -692,6 +686,9 @@ jemalloc = \"" jemalloc "/lib/libjemalloc_pic.a" "\"
(("fn test_loading_cosine") "#[ignore]\nfn test_loading_cosine"))
#t)))))))))
+;;; Rust 1.25 release support work with llvm 6, but builds with llvm 6 is
+;;; not determenistic due to https://github.com/rust-lang/rust/issues/50556
+;;; Keep using llvm 3.9.1 until builds become determenistic
(define-public rust-1.25
(let ((base-rust
(rust-bootstrapped-package rust-1.24 "1.25.0"
@@ -699,10 +696,6 @@ jemalloc = \"" jemalloc "/lib/libjemalloc_pic.a" "\"
#:patches '("rust-1.25-accept-more-detailed-gdb-lines.patch"))))
(package
(inherit base-rust)
- (inputs
- ;; Use LLVM 6.0
- (alist-replace "llvm" (list llvm)
- (package-inputs base-rust)))
(arguments
(substitute-keyword-arguments (package-arguments base-rust)
((#:phases phases)
@@ -713,14 +706,9 @@ jemalloc = \"" jemalloc "/lib/libjemalloc_pic.a" "\"
;; This test wants to update the crate index.
(("fn no_index_update") "#[ignore]\nfn no_index_update"))
#t))
- (add-after 'configure 'enable-codegen-tests
- (lambda _
- (substitute* "config.toml"
- (("codegen-tests = false") ""))
- #t))
;; FIXME: Re-enable this test if it's indeed supposed to work.
;; See <https://github.com/rust-lang/rust/issues/54178>.
- (add-after 'enable-codegen-tests 'disable-nil-enum-test
+ (add-after 'configure 'disable-nil-enum-test
(lambda _
(substitute* "src/test/debuginfo/nil-enum.rs"
(("ignore-lldb") "ignore-gdb"))
@@ -795,7 +783,8 @@ jemalloc = \"" jemalloc "/lib/libjemalloc_pic.a" "\"
#:patches
'("rust-coresimd-doctest.patch"
"rust-bootstrap-stage0-test.patch"
- "rust-1.25-accept-more-detailed-gdb-lines.patch"))))
+ "rust-1.25-accept-more-detailed-gdb-lines.patch"
+ "rust-mdbook-support-reproducible-builds-by-forcing-window.search.patch"))))
(package
(inherit base-rust)
(arguments
@@ -808,4 +797,10 @@ jemalloc = \"" jemalloc "/lib/libjemalloc_pic.a" "\"
;; `prefix' directory should exist before `install' call
(mkdir-p (assoc-ref outputs "out"))
(mkdir-p (assoc-ref outputs "cargo"))
+ #t))
+ (add-after 'patch-cargo-tests 'disable-thinlto-test
+ (lambda* _
+ ;; thinlto required llvm 6.0 for work
+ (substitute* "src/tools/cargo/tests/testsuite/path.rs"
+ (("fn thin_lto_works") "#[ignore]\nfn thin_lto_works"))
#t)))))))))
--
2.19.0
[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #3: 0002-gnu-rust-add-rust-1.28-and-rust-1.29-packages.patch --]
[-- Type: text/x-patch, Size: 3512 bytes --]
From 4eb82de9971c13323e7b0195f9f178282039ba35 Mon Sep 17 00:00:00 2001
From: Nikolai Merinov <nikolai.merinov@member.fsf.org>
Date: Tue, 16 Oct 2018 02:37:41 +0500
Subject: [PATCH 2/2] gnu: rust: add rust 1.28 and rust 1.29 packages
To: guix-patches@gnu.org
* gnu/packages/rust.scm (rust-1.27): rename "rust" to "rust-1.27"
(rust-1.28): Use llvm-6.0 finally. Disable test that broken on x86_64 machines
without "avx". Enable codegen tests. Enable cargo test for thinlto.
(rust): Package for rust 1.29.2.
---
gnu/packages/rust.scm | 46 ++++++++++++++++++++++++++++++++++++++++++-
1 file changed, 45 insertions(+), 1 deletion(-)
diff --git a/gnu/packages/rust.scm b/gnu/packages/rust.scm
index cd2a8fb9a..aafa3704b 100644
--- a/gnu/packages/rust.scm
+++ b/gnu/packages/rust.scm
@@ -776,7 +776,7 @@ jemalloc = \"" jemalloc "/lib/libjemalloc_pic.a" "\"
(("fn no_index_update") "#[ignore]\nfn no_index_update"))
#t)))))))))
-(define-public rust
+(define-public rust-1.27
(let ((base-rust
(rust-bootstrapped-package rust-1.26 "1.27.2"
"0pg1s37bhx9zqbynxyydq5j6q7kij9vxkcv8maz0m25prm88r0cs"
@@ -804,3 +804,47 @@ jemalloc = \"" jemalloc "/lib/libjemalloc_pic.a" "\"
(substitute* "src/tools/cargo/tests/testsuite/path.rs"
(("fn thin_lto_works") "#[ignore]\nfn thin_lto_works"))
#t)))))))))
+
+(define-public rust-1.28
+ (let ((base-rust
+ (rust-bootstrapped-package rust-1.27 "1.28.0"
+ "11k4rn77bca2rikykkk9fmprrgjswd4x4kaq7fia08vgkir82nhx"
+ #:patches
+ '("rust-coresimd-doctest.patch"
+ "rust-bootstrap-stage0-test.patch"
+ "rust-1.25-accept-more-detailed-gdb-lines.patch"
+ "rust-mdbook-support-reproducible-builds-by-forcing-window.search.patch"))))
+ (package
+ (inherit base-rust)
+ (inputs
+ ;; Use LLVM 6.0
+ (alist-replace "llvm" (list llvm)
+ (package-inputs base-rust)))
+ (arguments
+ (substitute-keyword-arguments (package-arguments base-rust)
+ ((#:phases phases)
+ `(modify-phases ,phases
+ (add-after 'configure 'enable-codegen-tests
+ ;; codegen tests should pass with llvm 6
+ (lambda* _
+ (substitute* "config.toml"
+ (("codegen-tests = false") ""))
+ #t))
+ (add-after 'patch-tests 'disable-amd64-avx-test
+ ;; This test will fail on x86_64 machines without avx
+ (lambda* _
+ (substitute* "src/test/run-pass/issue-44056.rs"
+ (("only-x86_64") "ignore-test"))
+ #t))
+ ;; thinlto test should pass with llvm 6
+ (delete 'disable-thinlto-test))))))))
+
+(define-public rust
+ (let ((base-rust
+ (rust-bootstrapped-package rust-1.28 "1.29.2"
+ "1jb787080z754caa2w3w1amsygs4qlzj9rs1vy64firfmabfg22h"
+ #:patches
+ '("rust-1.25-accept-more-detailed-gdb-lines.patch"
+ "rust-mdbook-support-reproducible-builds-by-forcing-window.search.patch"))))
+ (package
+ (inherit base-rust))))
--
2.19.0
[-- Attachment #4: Type: text/plain, Size: 3171 bytes --]
Regards,
Nikolai
Nikolai Merinov <nikolai.merinov@member.fsf.org> writes:
> Hi Danny,
>
> I fixed patches according to your comments. Also I updated rust 1.29.1
> to 1.29.2 and fixed test issue in 1.27: as I mentioned before I tested
> reproducibility of each package previously only in private my private
> repo where I disabled `check` phase for 1.25 and 1.27. I have pretty
> slow machine, but currently I already checked reproducibility with
> suggested patches for 1.23-1.26 releases. 1.27-1.29 in progress.
>
> Both updated patches attached.
>
> Danny Milosavljevic <dannym@scratchpost.org> writes:
>
>> Hi Nikolai,
>>
>> On Tue, 16 Oct 2018 02:32:11 +0500
>> Nikolai Merinov <nikolai.merinov@member.fsf.org> wrote:
>>
>>> * gnu/packages/patches/rust-mdbook-Support-reproducible-builds-by-forcing-window.search.patch:
>>
>> Nitpick: No big "S" (file names are easier to find if they are all lower case).
>>
> Fixed.
>
>>> patch that make "searchindex.js" reproducible in rust 1.27 and newer.
>>
>> "New file".
> Fixed.
>
>>
>>> * gnu/local.mk (dist_patch_DATA): Add new patch file.
>>> * gnu/packages/rust.scm (rust-1.19): Use system libssh2 library
>>
>> Hmm, I'm not sure about doing this in the same commit.
>> Is it also related to reproducibility?
>
> Looks like it should not, but still I started investigation with
> non-reproducible "libssh2" and "libgit2" rust libraries I made libssh2
> related changes at very beginning and never tested wihtout it. I not
> sure that it's good idea to remove it now.
>
>>
>>> during cargo build. Note: libgit2 still builded as part of cargo build,
>>> because cargo tests assume specific libgit2 minor release.
>>
>> What does this mean? Does it mean "bundled"?
> Yes. Rust used bundled sources for "libgit2-sys" rust library (used by cargo).
>
>>
>>> (rust-1.23): inherit native-inputs from previous package.
>>
>> Ok.
>>
>>> (rust-1.25): switch back to llvm 3.9.1 as workaround for
>>> https://github.com/rust-lang/rust/issues/50556 issue.
>>
>> Please add the reasoning as a comment inside the source code instead.
> Added comment for rust-1.25 package.
>
>>
>>> (rust-1.27): apply patch to make "searchindex.js" files reproducible.
>>
>> Maybe add "[source]".
> I repharase this comment and added source URL to patch file itself.
>
>>
>>> - (add-after 'configure 'enable-codegen-tests
>>> - (lambda _
>>> - (substitute* "config.toml"
>>> - (("codegen-tests = false") ""))
>>> - #t))
>>
>> I think I had reproducibility problems when enabling codegen tests and
>> parallel tests. Is that not the case anymore?
> Neither me nor Joe Hillenbrand in
> https://lists.gnu.org/archive/html/guix-devel/2018-10/msg00292.html got
> reproducibility issues with this tests.
>>
>>> ;; FIXME: Re-enable this test if it's indeed supposed to work.
>>> ;; See <https://github.com/rust-lang/rust/issues/54178>.
>>
>> Note to myself: I think the issue comments indicate that the newer gdb output
>> is better - so we should create a patch similar to
>> rust-1.25-accept-more-detailed-gdb-lines.patch to accept the newer output.
next prev parent reply other threads:[~2018-10-19 21:00 UTC|newest]
Thread overview: 14+ messages / expand[flat|nested] mbox.gz Atom feed top
2018-10-16 18:00 [bug#33066] [PATCH 0/2] Workaround rust reproducibility issues Nikolai Merinov
2018-10-15 21:32 ` [bug#33066] [PATCH 1/2] gnu: rust: workaround rust 1.25-27 " Nikolai Merinov
2018-10-17 11:14 ` Danny Milosavljevic
2018-10-19 19:44 ` Nikolai Merinov
2018-10-19 20:58 ` Nikolai Merinov [this message]
2018-10-21 7:28 ` [bug#33066] [PATCHv3] " Nikolai Merinov
2018-10-24 19:43 ` Danny Milosavljevic
2018-10-25 5:44 ` Danny Milosavljevic
2018-10-26 11:40 ` Nikolai Merinov
2018-10-26 16:12 ` Danny Milosavljevic
2021-02-05 15:00 ` bug#33066: [PATCH 0/2] Workaround rust " Maxim Cournoyer
2018-10-15 21:37 ` [bug#33066] [PATCH 2/2] gnu: rust: add rust 1.28 and rust 1.29 packages Nikolai Merinov
2018-10-17 11:15 ` Danny Milosavljevic
2018-10-17 14:14 ` Joe Hillenbrand
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=87sh11slfg.fsf_-_@member.fsf.org \
--to=nikolai.merinov@member.fsf.org \
--cc=33066@debbugs.gnu.org \
--cc=dannym@scratchpost.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this external index
https://git.savannah.gnu.org/cgit/guix.git
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.