bug#74142: [rust-team] rust-bootstrap fails parallel build on x86-64

bug#74142: [rust-team] rust-bootstrap fails parallel build on x86-64

[Greg Hogan]

[-- Attachment #1: Type: text/plain, Size: 4821 bytes --]

derivation: 83qbfg0xxsfr5lsc81m5a770c7c7b55k-rust-1.54.0.drv
commit: 6e50b0c56a8cc767bd3acb26638f78c450bde718

Build log attached, tail portion below. Has consistently failed to
build unless cores=1.

The package definition has "#:parallel-build? ,(target-x86-64?)" but
the noted build issue ("race conditions between various dependent
crates") is also seen on x86-64.

It is only the 1.54 build with mrustc that fails in this manner, the
following rust packages in the chain build fine with parallelism
enabled.

Greg

--8<---------------cut here---------------start------------->8---
(36/175) BUILDING autocfg v1.0.1
> /tmp/guix-build-rust-1.54.0.drv-0/mrustc/bin/mrustc rustc-1.54.0-src/vendor/autocfg/src/lib.rs -o output/rustc-build/libautocfg-1_0_1.rlib --crate-name autocfg --crate-type rlib -C emit-depfile=output/rustc-build/libautocfg-1_0_1.rlib.d --crate-tag 1_0_1 --cfg debug_assertions -O -L output -L output/rustc-build
> /tmp/guix-build-rust-1.54.0.drv-0/mrustc/output/rustc-build/build_bitflags-1_2_1_run
> /tmp/guix-build-rust-1.54.0.drv-0/rustc-1.54.0-src/vendor/bitflags/output/rustc-build/build_memchr-2_4_0_H6_run
Unable to run process
'/tmp/guix-build-rust-1.54.0.drv-0/rustc-1.54.0-src/vendor/bitflags/output/rustc-build/build_memchr-2_4_0_H6_run'
- No such file or directory
Calling /tmp/guix-build-rust-1.54.0.drv-0/rustc-1.54.0-src/vendor/bitflags/output/rustc-build/build_memchr-2_4_0_H6_run
failed (see /tmp/guix-build-rust-1.54.0.drv-0/mrustc/output/rustc-build/build_memchr-2_4_0_H6.txt_failed.txt
for stdout)
(37/175) BUILDING bitflags v1.2.1
> /tmp/guix-build-rust-1.54.0.drv-0/mrustc/bin/mrustc rustc-1.54.0-src/vendor/bitflags/src/lib.rs -o output/rustc-build/libbitflags-1_2_1.rlib --crate-name bitflags --crate-type rlib -C emit-depfile=output/rustc-build/libbitflags-1_2_1.rlib.d --crate-tag 1_2_1 --cfg debug_assertions -O -L output -L output/rustc-build --cfg bitflags_const_fn
Unable to run process
'/tmp/guix-build-rust-1.54.0.drv-0/mrustc/bin/mrustc' - No such file
or directory
> /tmp/guix-build-rust-1.54.0.drv-0/rustc-1.54.0-src/vendor/memchr/output/rustc-build/build_ryu-1_0_5_run
Unable to run process
'/tmp/guix-build-rust-1.54.0.drv-0/rustc-1.54.0-src/vendor/memchr/output/rustc-build/build_ryu-1_0_5_run'
- No such file or directory
Calling /tmp/guix-build-rust-1.54.0.drv-0/rustc-1.54.0-src/vendor/memchr/output/rustc-build/build_ryu-1_0_5_run
failed (see /tmp/guix-build-rust-1.54.0.drv-0/mrustc/output/rustc-build/build_ryu-1_0_5.txt_failed.txt
for stdout)
> /tmp/guix-build-rust-1.54.0.drv-0/rustc-1.54.0-src/vendor/memchr/output/rustc-build/build_snap-1_0_5_run
Unable to run process
'/tmp/guix-build-rust-1.54.0.drv-0/rustc-1.54.0-src/vendor/memchr/output/rustc-build/build_snap-1_0_5_run'
- No such file or directory
Calling /tmp/guix-build-rust-1.54.0.drv-0/rustc-1.54.0-src/vendor/memchr/output/rustc-build/build_snap-1_0_5_run
failed (see /tmp/guix-build-rust-1.54.0.drv-0/mrustc/output/rustc-build/build_snap-1_0_5.txt_failed.txt
for stdout)
> /tmp/guix-build-rust-1.54.0.drv-0/rustc-1.54.0-src/vendor/memchr/output/rustc-build/build_maybe-uninit-2_0_0_run
Unable to run process
'/tmp/guix-build-rust-1.54.0.drv-0/rustc-1.54.0-src/vendor/memchr/output/rustc-build/build_maybe-uninit-2_0_0_run'
- No such file or directory
Calling /tmp/guix-build-rust-1.54.0.drv-0/rustc-1.54.0-src/vendor/memchr/output/rustc-build/build_maybe-uninit-2_0_0_run
failed (see /tmp/guix-build-rust-1.54.0.drv-0/mrustc/output/rustc-build/build_maybe-uninit-2_0_0.txt_failed.txt
for stdout)
> /tmp/guix-build-rust-1.54.0.drv-0/rustc-1.54.0-src/vendor/memchr/output/rustc-build/build_typenum-1_12_0_run
Unable to run process
'/tmp/guix-build-rust-1.54.0.drv-0/rustc-1.54.0-src/vendor/memchr/output/rustc-build/build_typenum-1_12_0_run'
- No such file or directory
Calling /tmp/guix-build-rust-1.54.0.drv-0/rustc-1.54.0-src/vendor/memchr/output/rustc-build/build_typenum-1_12_0_run
failed (see /tmp/guix-build-rust-1.54.0.drv-0/mrustc/output/rustc-build/build_typenum-1_12_0.txt_failed.txt
for stdout)
BUILD FAILED
make: *** [minicargo.mk:228: output/rustc] Error 1
error: in phase 'build': uncaught exception:
%exception #<&invoke-error program: "make" arguments: ("-f"
"minicargo.mk" "output/rustc" "RUSTC_TARGET=x86_64-unknown-linux-gnu"
"RUSTC_VERSION=1.54.0" "MRUSTC_TARGET_VER=1.54" "CFLAGS=-std=gnu11"
"OUTDIR_SUF=") exit-status: 2 term-signal: #f stop-signal: #f>
phase `build' failed after 617.7 seconds
command "make" "-f" "minicargo.mk" "output/rustc"
"RUSTC_TARGET=x86_64-unknown-linux-gnu" "RUSTC_VERSION=1.54.0"
"MRUSTC_TARGET_VER=1.54" "CFLAGS=-std=gnu11" "OUTDIR_SUF=" failed with
status 2
build process 18 exited with status 256
--8<---------------cut here---------------end--------------->8---

[-- Attachment #2: qbfg0xxsfr5lsc81m5a770c7c7b55k-rust-1.54.0.drv.gz --]
[-- Type: application/gzip, Size: 390369 bytes --]

bug#74142: [PATCH rust-team] gnu: rust-bootstrap: Disable parallel builds.

[Greg Hogan]

* gnu/packages/rust.scm (rust-bootstrap): Disable parallel builds for
all architectures.

Change-Id: Icafbbacd30200fad457b6a70730acd466524485b
---
 gnu/packages/rust.scm | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/gnu/packages/rust.scm b/gnu/packages/rust.scm
index cb58a4b4b3..4f74fc7571 100644
--- a/gnu/packages/rust.scm
+++ b/gnu/packages/rust.scm
@@ -208,9 +208,9 @@ (define-public rust-bootstrap
        ;; that doesn't seem to cause issues.
        #:validate-runpath? #f
        ;; Most of the build is single-threaded. This also improves the
-       ;; build time on machines with "only" 8GB of RAM.
-       ;; ppc64le regularly sees race conditions between various dependant crates.
-       #:parallel-build? ,(target-x86-64?)
+       ;; build time on machines with "only" 8GB of RAM and prevents
+       ;; regularly seen race conditions between various dependant crates.
+       #:parallel-build? #f
        #:make-flags
        (list ,(string-append "RUSTC_TARGET="
                              (platform-rust-target

base-commit: 6e50b0c56a8cc767bd3acb26638f78c450bde718
-- 
2.46.0

bug#74142: [rust-team] rust-bootstrap fails parallel build on x86-64

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 1276 bytes --]

On Thu, Oct 31, 2024 at 03:13:40PM -0400, Greg Hogan wrote:
> derivation: 83qbfg0xxsfr5lsc81m5a770c7c7b55k-rust-1.54.0.drv
> commit: 6e50b0c56a8cc767bd3acb26638f78c450bde718
> 
> Build log attached, tail portion below. Has consistently failed to
> build unless cores=1.
> 
> The package definition has "#:parallel-build? ,(target-x86-64?)" but
> the noted build issue ("race conditions between various dependent
> crates") is also seen on x86-64.
> 
> It is only the 1.54 build with mrustc that fails in this manner, the
> following rust packages in the chain build fine with parallelism
> enabled.
> 
> Greg

I see the note there says something about ppc64le having a race
condition too. My machine has "only" 24 cores, so I didn't run into
anything, but I see you're building with 32 cores. IIRC the ppc64le
machine was building with at least 32 cores also.

Can you try building rust-1.54 with only 24 cores and see if that works
for you? If it does then I think we can switch the (job-count) variable
to (min 24 (parallel-job-count))

-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

bug#74142: [rust-team] rust-bootstrap fails parallel build on x86-64

[Greg Hogan]

[-- Attachment #1: Type: text/plain, Size: 1270 bytes --]

On Sun, Nov 3, 2024 at 2:49 AM Efraim Flashner <efraim@flashner.co.il> wrote:
>
> On Thu, Oct 31, 2024 at 03:13:40PM -0400, Greg Hogan wrote:
> > derivation: 83qbfg0xxsfr5lsc81m5a770c7c7b55k-rust-1.54.0.drv
> > commit: 6e50b0c56a8cc767bd3acb26638f78c450bde718
> >
> > Build log attached, tail portion below. Has consistently failed to
> > build unless cores=1.
> >
> > The package definition has "#:parallel-build? ,(target-x86-64?)" but
> > the noted build issue ("race conditions between various dependent
> > crates") is also seen on x86-64.
> >
> > It is only the 1.54 build with mrustc that fails in this manner, the
> > following rust packages in the chain build fine with parallelism
> > enabled.
> >
> > Greg
>
> I see the note there says something about ppc64le having a race
> condition too. My machine has "only" 24 cores, so I didn't run into
> anything, but I see you're building with 32 cores. IIRC the ppc64le
> machine was building with at least 32 cores also.
>
> Can you try building rust-1.54 with only 24 cores and see if that works
> for you? If it does then I think we can switch the (job-count) variable
> to (min 24 (parallel-job-count))

Log attached from what looks to be a similar failure on an 8 core instance.

[-- Attachment #2: qbfg0xxsfr5lsc81m5a770c7c7b55k-rust-1.54.0.drv.gz --]
[-- Type: application/gzip, Size: 387279 bytes --]

bug#74142: [rust-team] rust-bootstrap fails parallel build on x86-64

[Maxim Cournoyer]

Hi,

Greg Hogan <code@greghogan.com> writes:

> On Sun, Nov 3, 2024 at 2:49 AM Efraim Flashner <efraim@flashner.co.il> wrote:
>>
>> On Thu, Oct 31, 2024 at 03:13:40PM -0400, Greg Hogan wrote:
>> > derivation: 83qbfg0xxsfr5lsc81m5a770c7c7b55k-rust-1.54.0.drv
>> > commit: 6e50b0c56a8cc767bd3acb26638f78c450bde718
>> >
>> > Build log attached, tail portion below. Has consistently failed to
>> > build unless cores=1.
>> >
>> > The package definition has "#:parallel-build? ,(target-x86-64?)" but
>> > the noted build issue ("race conditions between various dependent
>> > crates") is also seen on x86-64.
>> >
>> > It is only the 1.54 build with mrustc that fails in this manner, the
>> > following rust packages in the chain build fine with parallelism
>> > enabled.
>> >
>> > Greg
>>
>> I see the note there says something about ppc64le having a race
>> condition too. My machine has "only" 24 cores, so I didn't run into
>> anything, but I see you're building with 32 cores. IIRC the ppc64le
>> machine was building with at least 32 cores also.
>>
>> Can you try building rust-1.54 with only 24 cores and see if that works
>> for you? If it does then I think we can switch the (job-count) variable
>> to (min 24 (parallel-job-count))
>
> Log attached from what looks to be a similar failure on an 8 core instance.

I encountered the same problem on a 32 "cores" (aka threads) machine.
I've reported it here [0].

[0]  https://github.com/thepowersgang/mrustc/issues/343

-- 
Thanks,
Maxim

bug#74142: [rust-team] rust-bootstrap fails parallel build on x86-64

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 3640 bytes --]

On Sat, Dec 07, 2024 at 10:45:32PM +0900, Maxim Cournoyer wrote:
> Hi,
> 
> Greg Hogan <code@greghogan.com> writes:
> 
> > On Sun, Nov 3, 2024 at 2:49 AM Efraim Flashner <efraim@flashner.co.il> wrote:
> >>
> >> On Thu, Oct 31, 2024 at 03:13:40PM -0400, Greg Hogan wrote:
> >> > derivation: 83qbfg0xxsfr5lsc81m5a770c7c7b55k-rust-1.54.0.drv
> >> > commit: 6e50b0c56a8cc767bd3acb26638f78c450bde718
> >> >
> >> > Build log attached, tail portion below. Has consistently failed to
> >> > build unless cores=1.
> >> >
> >> > The package definition has "#:parallel-build? ,(target-x86-64?)" but
> >> > the noted build issue ("race conditions between various dependent
> >> > crates") is also seen on x86-64.
> >> >
> >> > It is only the 1.54 build with mrustc that fails in this manner, the
> >> > following rust packages in the chain build fine with parallelism
> >> > enabled.
> >> >
> >> > Greg
> >>
> >> I see the note there says something about ppc64le having a race
> >> condition too. My machine has "only" 24 cores, so I didn't run into
> >> anything, but I see you're building with 32 cores. IIRC the ppc64le
> >> machine was building with at least 32 cores also.
> >>
> >> Can you try building rust-1.54 with only 24 cores and see if that works
> >> for you? If it does then I think we can switch the (job-count) variable
> >> to (min 24 (parallel-job-count))
> >
> > Log attached from what looks to be a similar failure on an 8 core instance.
> 
> I encountered the same problem on a 32 "cores" (aka threads) machine.
> I've reported it here [0].
> 
> [0]  https://github.com/thepowersgang/mrustc/issues/343

I followed up with Mutabah (the main author) on IRC and a bit on the bug
tracker, and their suggestion is to use a later commit from mrustc.  I
used the latest commit from the master branch but there I ran into
issues on aarch64 not building correctly. From there they basically said
that they were focused on getting 1.74 bootstrapped, and they weren't
going to be circling back to the master branch until 1.74 was merged.

I went and tried out the rust_1.74 branch which is nearing completion
and that one built without any issues on x86_64 but failed for different
reasons on aarch64, powerpc64le and riscv64.

Someone from Gentoo was also asking around, they're interested in using
mrustc to bootstrap rust-1.74 and continuing on from there, so it's
likely we'll get some help going forward with architecture support.

I think in the meantime while it's not ready I'll change the
parallel-builds? flag to false for everyone, and then I'll continue
working on getting the bootstrap to work on the other architectures. The
good news is it only takes about 15 minutes to fail on aarch64 and
powerpc64le, so those should go pretty quickly through each iteration. I
might need to try some by-hand builds on riscv64; that one was closer to
2 hours.

Mutabah also said that their primary target is x86_64 and they don't
have the hardware to test other architectures, so its more hit-or-miss
there, and they rely on patches from others to get those ports working
as well as they do.  I don't love the answer, but I don't blame them.

I'll probably focus on aarch64 first since I can offload that easily and
I don't need to use a remote GUIX_DAEMON_SOCKET to get it to build, but
unfortunately it looks like we're sticking with 1.54 for a while.

-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

bug#74142: [rust-team] rust-bootstrap fails parallel build on x86-64

[Maxim Cournoyer]

Hi Efraim,

Efraim Flashner <efraim@flashner.co.il> writes:

[...]

>> [0]  https://github.com/thepowersgang/mrustc/issues/343
>
> I followed up with Mutabah (the main author) on IRC and a bit on the bug
> tracker, and their suggestion is to use a later commit from mrustc.  I
> used the latest commit from the master branch but there I ran into
> issues on aarch64 not building correctly. From there they basically said
> that they were focused on getting 1.74 bootstrapped, and they weren't
> going to be circling back to the master branch until 1.74 was merged.
>
> I went and tried out the rust_1.74 branch which is nearing completion
> and that one built without any issues on x86_64 but failed for different
> reasons on aarch64, powerpc64le and riscv64.
>
> Someone from Gentoo was also asking around, they're interested in using
> mrustc to bootstrap rust-1.74 and continuing on from there, so it's
> likely we'll get some help going forward with architecture support.
>
> I think in the meantime while it's not ready I'll change the
> parallel-builds? flag to false for everyone, and then I'll continue
> working on getting the bootstrap to work on the other architectures.

Sounds good!  Thanks for putting some efforts into shortening the
bootstrap... it currently takes more 3 hours to achieve on a powerful
machine.

-- 
Thanks,
Maxim

bug#74142: [rust-team] rust-bootstrap fails parallel build on x86-64

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 358 bytes --]

I don't remember if I closed this but I've updated the mrustc snapshot
and now parallel builds are supported by default, on all architectures.

-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

bug#74142: [rust-team] rust-bootstrap fails parallel build on x86-64

[Maxim Cournoyer]

Hi Efraim,

Efraim Flashner <efraim@flashner.co.il> writes:

> I don't remember if I closed this but I've updated the mrustc snapshot
> and now parallel builds are supported by default, on all architectures.

Excellent!

-- 
Thanks,
Maxim