unofficial mirror of guix-devel@gnu.org 
 help / color / mirror / code / Atom feed
* How long does it take to run the full rustc bootstrap chain?
@ 2022-10-15  7:33 Félix Baylac Jacqué
  2022-10-19 19:52 ` Efraim Flashner
                   ` (2 more replies)
  0 siblings, 3 replies; 12+ messages in thread
From: Félix Baylac Jacqué @ 2022-10-15  7:33 UTC (permalink / raw)
  To: guix-devel

Hey Guix,

I'd be curious to know how long it takes to run the full rustc bootstrap
chain on the Guix build farm. I'm sadly not sure how to approach this
problem.

Is there a way to extract this information from Cuirass or the Guix data
service?

Félix


^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: How long does it take to run the full rustc bootstrap chain?
  2022-10-15  7:33 How long does it take to run the full rustc bootstrap chain? Félix Baylac Jacqué
@ 2022-10-19 19:52 ` Efraim Flashner
  2022-10-20  9:59 ` Ludovic Courtès
  2022-10-22 13:48 ` Maxim Cournoyer
  2 siblings, 0 replies; 12+ messages in thread
From: Efraim Flashner @ 2022-10-19 19:52 UTC (permalink / raw)
  To: Félix Baylac Jacqué; +Cc: guix-devel

[-- Attachment #1: Type: text/plain, Size: 1316 bytes --]

On Sat, Oct 15, 2022 at 09:33:23AM +0200, Félix Baylac Jacqué wrote:
> Hey Guix,
> 
> I'd be curious to know how long it takes to run the full rustc bootstrap
> chain on the Guix build farm. I'm sadly not sure how to approach this
> problem.
> 
> Is there a way to extract this information from Cuirass or the Guix data
> service?

If I were to go about it I would try to get the build log from each
package:
guix build --no-grafts -e '(@@ (gnu packages rust) rust-bootstrap)' \
    --log-file

and then I'd add up the build and test phases to get a rough estimate.

For an approximate answer, the build farm runs 4 builds per machine,
with 8 cores per build (all of this is in the guix-maintenance repo I
think). When I build from mrustc to rust-1.60 on my machine with 24
cores it currently takes about 2 hours for rust-bootstrap (1.54) and
then about 30-45 minutes each other version.

I don't remember the bug number offhand, but there's work being done to
speed up the process. The last time I ran though the test build I built
from rust-1.55 to librsvg in 171 minutes.

-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: How long does it take to run the full rustc bootstrap chain?
  2022-10-15  7:33 How long does it take to run the full rustc bootstrap chain? Félix Baylac Jacqué
  2022-10-19 19:52 ` Efraim Flashner
@ 2022-10-20  9:59 ` Ludovic Courtès
  2022-10-20 20:11   ` Efraim Flashner
  2022-10-22 13:48 ` Maxim Cournoyer
  2 siblings, 1 reply; 12+ messages in thread
From: Ludovic Courtès @ 2022-10-20  9:59 UTC (permalink / raw)
  To: Félix Baylac Jacqué; +Cc: guix-devel

Hi Félix,

Félix Baylac Jacqué <felix@alternativebit.fr> skribis:

> I'd be curious to know how long it takes to run the full rustc bootstrap
> chain on the Guix build farm. I'm sadly not sure how to approach this
> problem.

I believe Efraim, Maxim, and probably a few other people have first-hand
experience building the whole chain.  Any estimate, people?

> Is there a way to extract this information from Cuirass or the Guix data
> service?

Maybe not from the Guix Data Service, but most likely from Cuirass and
the Guix Build Coordinator.

For example, <https://ci.guix.gnu.org/build/1505621/details> shows the
duration, which could can also get with:

  wget -qO- https://ci.guix.gnu.org/build/1505621 |jq

(The (guix ci) module needs an update to let you do access that info
right from Scheme.)

Now the /search interface doesn’t have a JSON equivalent, but you can
find the build ID using something along these lines:

--8<---------------cut here---------------start------------->8---
scheme@(guile-user)> ,use(guix ci)
scheme@(guile-user)> (latest-evaluations "https://ci.guix.gnu.org" 10 #:spec "master")
$14 = (#<<evaluation> id: 739193 spec: "master" complete?: #t checkouts: (#<<checkout> commit: "4716cea6256523a8ecf90a426d675bfb7620f3e4" channel: "guix">)> #<<evaluation> id: 738364 spec: "master" complete?: #t checkouts: (#<<checkout> commit: "16d4ded6302c0650978203d0df83614896c453e8" channel: "guix">)> #<<evaluation> id: 737995 spec: "master" complete?: #t checkouts: (#<<checkout> commit: "e61660c78f1190c578dd6f202bc5529cbdcff84e" channel: "guix">)> #<<evaluation> id: 737588 spec: "master" complete?: #t checkouts: (#<<checkout> commit: "88746cd80bc56212ae7922c0fa1cd9a18e44c3bb" channel: "guix">)> #<<evaluation> id: 737529 spec: "master" complete?: #t checkouts: (#<<checkout> commit: "ac553ba68e535810085dd838e48e4fa6ac553e67" channel: "guix">)> #<<evaluation> id: 736841 spec: "master" complete?: #t checkouts: (#<<checkout> commit: "7ecd85eacbfa5b7379f563b83807d3e5258cf700" channel: "guix">)> #<<evaluation> id: 734789 spec: "master" complete?: #t checkouts: (#<<checkout> commit: "3bb145b6e2a8c84e7739ead9ae76dc4d42bb9850" channel: "guix">)> #<<evaluation> id: 734747 spec: "master" complete?: #t checkouts: (#<<checkout> commit: "033cbd11a837dbc7602799f15d691221653e1996" channel: "guix">)> #<<evaluation> id: 734733 spec: "master" complete?: #t checkouts: (#<<checkout> commit: "c9eac0c3553703385c997a70741348ae5d34dc68" channel: "guix">)> #<<evaluation> id: 734718 spec: "master" complete?: #t checkouts: (#<<checkout> commit: "2589997fab07bcebe6d87fd227852389ab5b1962" channel: "guix">)>)
scheme@(guile-user)> (define jobs (evaluation-jobs "https://ci.guix.gnu.org" (evaluation-id (car $14))))
scheme@(guile-user)> (length jobs)
$15 = 76649
scheme@(guile-user)> (car jobs)
$16 = #<<job> build-id: 1627000 status: scheduled name: "0ad.aarch64-linux">
scheme@(guile-user)> ,use(srfi srfi-1)
scheme@(guile-user)> (find (lambda (job) (string=? "rust.x86_64-linux" (job-name job))) jobs)
$17 = #<<job> build-id: 1275492 status: succeeded name: "rust.x86_64-linux">
scheme@(guile-user)> (job-build "https://ci.guix.gnu.org" $17)
$18 = #<<build> id: 1275492 derivation: "/gnu/store/dmzhvql66a1n31j2xpygfxpw576wpkwb-rust-1.60.0.drv" evaluation: 579940 system: "x86_64-linux" status: succeeded timestamp: 1662284654 start-time: #<date nanosecond: 0 second: 14 minute: 44 hour: 11 day: 4 month: 9 year: 2022 zone-offset: 7200> stop-time: #<date nanosecond: 0 second: 14 minute: 44 hour: 11 day: 4 month: 9 year: 2022 zone-offset: 7200> products: ()>
--8<---------------cut here---------------end--------------->8---

That still doesn’t answer your question, because you’re interested in
the build time of the full Rust bootstrap chain, but that’s a start!

HTH,
Ludo’.


^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: How long does it take to run the full rustc bootstrap chain?
  2022-10-20  9:59 ` Ludovic Courtès
@ 2022-10-20 20:11   ` Efraim Flashner
  2022-10-22  8:57     ` Félix Baylac Jacqué
  0 siblings, 1 reply; 12+ messages in thread
From: Efraim Flashner @ 2022-10-20 20:11 UTC (permalink / raw)
  To: Ludovic Courtès; +Cc: Félix Baylac Jacqué, guix-devel

[-- Attachment #1: Type: text/plain, Size: 4557 bytes --]

On Thu, Oct 20, 2022 at 11:59:34AM +0200, Ludovic Courtès wrote:
> Hi Félix,
> 
> Félix Baylac Jacqué <felix@alternativebit.fr> skribis:
> 
> > I'd be curious to know how long it takes to run the full rustc bootstrap
> > chain on the Guix build farm. I'm sadly not sure how to approach this
> > problem.
> 
> I believe Efraim, Maxim, and probably a few other people have first-hand
> experience building the whole chain.  Any estimate, people?

I have a number! I just pushed a couple of patches to staging to fix the
bootstrap on aarch64, and to decrease the build time and resource usage,
and the numbers are in!

https://ci.guix.gnu.org/build/1632586/details

11863 seconds from downloading mrustc to building rust-1.60, for 198
minutes, or 3.3 hours.

> > Is there a way to extract this information from Cuirass or the Guix data
> > service?
> 
> Maybe not from the Guix Data Service, but most likely from Cuirass and
> the Guix Build Coordinator.
> 
> For example, <https://ci.guix.gnu.org/build/1505621/details> shows the
> duration, which could can also get with:
> 
>   wget -qO- https://ci.guix.gnu.org/build/1505621 |jq
> 
> (The (guix ci) module needs an update to let you do access that info
> right from Scheme.)
> 
> Now the /search interface doesn’t have a JSON equivalent, but you can
> find the build ID using something along these lines:
> 
> --8<---------------cut here---------------start------------->8---
> scheme@(guile-user)> ,use(guix ci)
> scheme@(guile-user)> (latest-evaluations "https://ci.guix.gnu.org" 10 #:spec "master")
> $14 = (#<<evaluation> id: 739193 spec: "master" complete?: #t checkouts: (#<<checkout> commit: "4716cea6256523a8ecf90a426d675bfb7620f3e4" channel: "guix">)> #<<evaluation> id: 738364 spec: "master" complete?: #t checkouts: (#<<checkout> commit: "16d4ded6302c0650978203d0df83614896c453e8" channel: "guix">)> #<<evaluation> id: 737995 spec: "master" complete?: #t checkouts: (#<<checkout> commit: "e61660c78f1190c578dd6f202bc5529cbdcff84e" channel: "guix">)> #<<evaluation> id: 737588 spec: "master" complete?: #t checkouts: (#<<checkout> commit: "88746cd80bc56212ae7922c0fa1cd9a18e44c3bb" channel: "guix">)> #<<evaluation> id: 737529 spec: "master" complete?: #t checkouts: (#<<checkout> commit: "ac553ba68e535810085dd838e48e4fa6ac553e67" channel: "guix">)> #<<evaluation> id: 736841 spec: "master" complete?: #t checkouts: (#<<checkout> commit: "7ecd85eacbfa5b7379f563b83807d3e5258cf700" channel: "guix">)> #<<evaluation> id: 734789 spec: "master" complete?: #t checkouts: (#<<checkout> commit: "3bb145b6e2a8c84e7739ead9ae76dc4d42bb9850" channel: "guix">)> #<<evaluation> id: 734747 spec: "master" complete?: #t checkouts: (#<<checkout> commit: "033cbd11a837dbc7602799f15d691221653e1996" channel: "guix">)> #<<evaluation> id: 734733 spec: "master" complete?: #t checkouts: (#<<checkout> commit: "c9eac0c3553703385c997a70741348ae5d34dc68" channel: "guix">)> #<<evaluation> id: 734718 spec: "master" complete?: #t checkouts: (#<<checkout> commit: "2589997fab07bcebe6d87fd227852389ab5b1962" channel: "guix">)>)
> scheme@(guile-user)> (define jobs (evaluation-jobs "https://ci.guix.gnu.org" (evaluation-id (car $14))))
> scheme@(guile-user)> (length jobs)
> $15 = 76649
> scheme@(guile-user)> (car jobs)
> $16 = #<<job> build-id: 1627000 status: scheduled name: "0ad.aarch64-linux">
> scheme@(guile-user)> ,use(srfi srfi-1)
> scheme@(guile-user)> (find (lambda (job) (string=? "rust.x86_64-linux" (job-name job))) jobs)
> $17 = #<<job> build-id: 1275492 status: succeeded name: "rust.x86_64-linux">
> scheme@(guile-user)> (job-build "https://ci.guix.gnu.org" $17)
> $18 = #<<build> id: 1275492 derivation: "/gnu/store/dmzhvql66a1n31j2xpygfxpw576wpkwb-rust-1.60.0.drv" evaluation: 579940 system: "x86_64-linux" status: succeeded timestamp: 1662284654 start-time: #<date nanosecond: 0 second: 14 minute: 44 hour: 11 day: 4 month: 9 year: 2022 zone-offset: 7200> stop-time: #<date nanosecond: 0 second: 14 minute: 44 hour: 11 day: 4 month: 9 year: 2022 zone-offset: 7200> products: ()>
> --8<---------------cut here---------------end--------------->8---
> 
> That still doesn’t answer your question, because you’re interested in
> the build time of the full Rust bootstrap chain, but that’s a start!
> 
> HTH,
> Ludo’.
> 

-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: How long does it take to run the full rustc bootstrap chain?
  2022-10-20 20:11   ` Efraim Flashner
@ 2022-10-22  8:57     ` Félix Baylac Jacqué
  0 siblings, 0 replies; 12+ messages in thread
From: Félix Baylac Jacqué @ 2022-10-22  8:57 UTC (permalink / raw)
  To: Efraim Flashner, Ludovic Courtès; +Cc: guix-devel

Hi Efraim, Ludovic,

Ah! Great! I did not realize the full bootstrapping was happening in a
single Cuirass build job :)

3.3 hours doesn't seem so bad in the grand scheme of things in the end.
I was expecting things to be worse than that. Thanks a lot for the help.

Félix


^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: How long does it take to run the full rustc bootstrap chain?
  2022-10-15  7:33 How long does it take to run the full rustc bootstrap chain? Félix Baylac Jacqué
  2022-10-19 19:52 ` Efraim Flashner
  2022-10-20  9:59 ` Ludovic Courtès
@ 2022-10-22 13:48 ` Maxim Cournoyer
  2022-10-26 19:37   ` bokr
  2 siblings, 1 reply; 12+ messages in thread
From: Maxim Cournoyer @ 2022-10-22 13:48 UTC (permalink / raw)
  To: Félix Baylac Jacqué; +Cc: guix-devel

Hi,

Félix Baylac Jacqué <felix@alternativebit.fr> writes:

> Hey Guix,
>
> I'd be curious to know how long it takes to run the full rustc bootstrap
> chain on the Guix build farm. I'm sadly not sure how to approach this
> problem.
>
> Is there a way to extract this information from Cuirass or the Guix data
> service?
>
> Félix

It used to be 16 hours on a Ryzen 3900x machine, then it got halved to 8
hours with the work to bootstrap from 1.39, and recently we're
bootstrapping from 1.54, so it must have been greatly reduced again.

Looking at (gnu packages rust), the mrustc-based bootstrap starts with
1.54.0.  This one is expensive, probably around 1 h 30 or more on a
Ryzen 3900x CPU (24 logical CPUs).

The intermediate builds are typically around 15-20 minutes on that
machines, with the last one taking a bit more (30 minutes), so the
current bootstrap on such a machine should take about:

1.54.0: 1h30m
1.55.0 - 1.60.0: 6 X 20 min = 1h20m
1.60.0: final build with tests and extra tools: 30 min

The total should be around 3 h 20 on a fast modern x86_64 machine.  I
suppose the time for berlin to build it takes about this.

HTH!

-- 
Thanks,
Maxim


^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: How long does it take to run the full rustc bootstrap chain?
  2022-10-22 13:48 ` Maxim Cournoyer
@ 2022-10-26 19:37   ` bokr
  2022-10-27 14:35     ` Maxim Cournoyer
  2022-11-06  9:08     ` Efraim Flashner
  0 siblings, 2 replies; 12+ messages in thread
From: bokr @ 2022-10-26 19:37 UTC (permalink / raw)
  To: Maxim Cournoyer; +Cc: Félix Baylac Jacqué, guix-devel

Hi,

On +2022-10-22 09:48:50 -0400, Maxim Cournoyer wrote:
> Hi,
> 
> Félix Baylac Jacqué <felix@alternativebit.fr> writes:
> 
> > Hey Guix,
> >
> > I'd be curious to know how long it takes to run the full rustc bootstrap
> > chain on the Guix build farm. I'm sadly not sure how to approach this
> > problem.
> >
> > Is there a way to extract this information from Cuirass or the Guix data
> > service?
> >
> > Félix
> 
> It used to be 16 hours on a Ryzen 3900x machine, then it got halved to 8
> hours with the work to bootstrap from 1.39, and recently we're
> bootstrapping from 1.54, so it must have been greatly reduced again.
> 
> Looking at (gnu packages rust), the mrustc-based bootstrap starts with
> 1.54.0.  This one is expensive, probably around 1 h 30 or more on a
> Ryzen 3900x CPU (24 logical CPUs).
> 
> The intermediate builds are typically around 15-20 minutes on that
> machines, with the last one taking a bit more (30 minutes), so the
> current bootstrap on such a machine should take about:
> 
> 1.54.0: 1h30m
> 1.55.0 - 1.60.0: 6 X 20 min = 1h20m
> 1.60.0: final build with tests and extra tools: 30 min
> 
> The total should be around 3 h 20 on a fast modern x86_64 machine.  I
> suppose the time for berlin to build it takes about this.
> 
> HTH!
> 
> -- 
> Thanks,
> Maxim
> 

I'm curious what
--8<---------------cut here---------------start------------->8---
$ lsblk -o size,model,type,tran,vendor,name|grep -Ei 'ssd|model';echo;lspci |grep -i nvme
--8<---------------cut here---------------end--------------->8---
on your relevant machines would show.

I opted for the best SSD available for my purism librem13v4 at the time,
and was really happy with seems like 10x faster than the SATA SSD in my older
but still i7 x86_64 previous laptop. Prob really 4-5x faster.

So above combo command line now gives me
--8<---------------cut here---------------start------------->8---
SIZE MODEL                          TYPE  TRAN   VENDOR   NAME
465.8G Samsung SSD 970 EVO Plus 500GB disk  nvme            nvme0n1

01:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981
$ 
--8<---------------cut here---------------end--------------->8---

What /is/has been/ on your machines? Could your improved times be part from SSD/controller changes?

There's really a huge difference  between SATA and 4-lane pci
(where both ends can handle it, which may require fw update or not be available)
Obviously 4 lanes is also going to be faster than one.

--
Regards,
Bengt Richter


^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: How long does it take to run the full rustc bootstrap chain?
  2022-10-26 19:37   ` bokr
@ 2022-10-27 14:35     ` Maxim Cournoyer
  2022-10-28 12:00       ` Has guix system ever run a hardware survey? Joshua Branson
  2022-10-31 19:02       ` How long does it take to run the full rustc bootstrap chain? Bengt Richter
  2022-11-06  9:08     ` Efraim Flashner
  1 sibling, 2 replies; 12+ messages in thread
From: Maxim Cournoyer @ 2022-10-27 14:35 UTC (permalink / raw)
  To: bokr; +Cc: Félix Baylac Jacqué, guix-devel

Hi,

bokr@bokr.com writes:

[...]

> I'm curious what
>
> $ lsblk -o size,model,type,tran,vendor,name|grep -Ei 'ssd|model';echo;lspci |grep -i nvme
>
> on your relevant machines would show.
>
> I opted for the best SSD available for my purism librem13v4 at the time,
> and was really happy with seems like 10x faster than the SATA SSD in my older
> but still i7 x86_64 previous laptop. Prob really 4-5x faster.
>
> So above combo command line now gives me
>
> SIZE MODEL                          TYPE  TRAN   VENDOR   NAME
> 465.8G Samsung SSD 970 EVO Plus 500GB disk  nvme            nvme0n1
>
> 01:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981
> $ 
>
> What /is/has been/ on your machines? Could your improved times be part from SSD/controller changes?
>
> There's really a huge difference  between SATA and 4-lane pci
> (where both ends can handle it, which may require fw update or not be available)
> Obviously 4 lanes is also going to be faster than one.

--8<---------------cut here---------------start------------->8---
$ lsblk -o size,model,type,tran,vendor,name|grep -Ei 'ssd|model';echo;lspci |grep -i nvme
  SIZE MODEL                      TYPE  TRAN   VENDOR   NAME
465.8G Samsung SSD 860 EVO 500GB  disk  sata   ATA      sda
931.5G Samsung SSD 840 EVO 1TB    disk  sata   ATA      sdc
--8<---------------cut here---------------end--------------->8---

Building Rust is mostly CPU dependent; I think fast single thread
performance is key as not that much happen in parallel, IIRC.  The 3900X
is a 12 cores (24 logical) beast.

-- 
Thanks,
Maxim


^ permalink raw reply	[flat|nested] 12+ messages in thread

* Has guix system ever run a hardware survey?
  2022-10-27 14:35     ` Maxim Cournoyer
@ 2022-10-28 12:00       ` Joshua Branson
  2022-10-31 19:02       ` How long does it take to run the full rustc bootstrap chain? Bengt Richter
  1 sibling, 0 replies; 12+ messages in thread
From: Joshua Branson @ 2022-10-28 12:00 UTC (permalink / raw)
  To: Maxim Cournoyer; +Cc: bokr, Félix Baylac Jacqué, guix-devel

Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:

> Hi,
>
> bokr@bokr.com writes:
>
> [...]
>
>>
>> I opted for the best SSD available for my purism librem13v4 at the time,
>> and was really happy with seems like 10x faster than the SATA SSD in my older
>> but still i7 x86_64 previous laptop. Prob really 4-5x faster.
>>
>
> Building Rust is mostly CPU dependent; I think fast single thread
> performance is key as not that much happen in parallel, IIRC.  The 3900X
> is a 12 cores (24 logical) beast.
>

This makes me wonder what sort of data we'd get if we asked guix system
users what sort of machine they use guix system on.

I've got an os-booted Thinkpad T400.  My core 2 duo seems pretty slow
compared to what ya'll have.  :)


^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: How long does it take to run the full rustc bootstrap chain?
  2022-10-27 14:35     ` Maxim Cournoyer
  2022-10-28 12:00       ` Has guix system ever run a hardware survey? Joshua Branson
@ 2022-10-31 19:02       ` Bengt Richter
  2022-11-01 12:49         ` Maxim Cournoyer
  1 sibling, 1 reply; 12+ messages in thread
From: Bengt Richter @ 2022-10-31 19:02 UTC (permalink / raw)
  To: Maxim Cournoyer; +Cc: Félix Baylac Jacqué, guix-devel

Hi again, thanks for your reply...

On +2022-10-27 10:35:02 -0400, Maxim Cournoyer wrote:
> Hi,
> 

(Oops, pasting back alternative I thought would be faster)
> > So above combo command line now gives me
> > --8<---------------cut here---------------start------------->8---
> > SIZE MODEL                          TYPE  TRAN   VENDOR   NAME
> > 465.8G Samsung SSD 970 EVO Plus 500GB disk  nvme            nvme0n1
> > 
> > 01:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981
> > $ 
> > --8<---------------cut here---------------end--------------->8---

[...]

--8<---------------cut here---------------start------------->8---
> $ lsblk -o size,model,type,tran,vendor,name|grep -Ei 'ssd|model';echo;lspci |grep -i nvme
>   SIZE MODEL                      TYPE  TRAN   VENDOR   NAME
> 465.8G Samsung SSD 860 EVO 500GB  disk  sata   ATA      sda
> 931.5G Samsung SSD 840 EVO 1TB    disk  sata   ATA      sdc
> --8<---------------cut here---------------end--------------->8---
> 
> Building Rust is mostly CPU dependent; I think fast single thread
> performance is key as not that much happen in parallel, IIRC.  The 3900X
> is a 12 cores (24 logical) beast.
>

Hm, just TRAN sata, no nvme, so it's going to be slow, but
what is the effect on what you timed?

Is there an easy way to get a measure of how many GB went
through those SATA channels during what you timed? That
would give an idea of what faster phusical disk memory
access would do for you. If many people are waiting longer
that they like, maybe they would chip in to fund an upgrade,
to feed that 12(24)-core "beast" :-)

I'd bet it is waiting a lot, if not more than computing :)
--
Regards,
Bengt Richter




^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: How long does it take to run the full rustc bootstrap chain?
  2022-10-31 19:02       ` How long does it take to run the full rustc bootstrap chain? Bengt Richter
@ 2022-11-01 12:49         ` Maxim Cournoyer
  0 siblings, 0 replies; 12+ messages in thread
From: Maxim Cournoyer @ 2022-11-01 12:49 UTC (permalink / raw)
  To: Bengt Richter; +Cc: Félix Baylac Jacqué, guix-devel

Hi Bengt,

Bengt Richter <bokr@bokr.com> writes:

[...]

>> Building Rust is mostly CPU dependent; I think fast single thread
>> performance is key as not that much happen in parallel, IIRC.  The 3900X
>> is a 12 cores (24 logical) beast.
>>
>
> Hm, just TRAN sata, no nvme, so it's going to be slow, but
> what is the effect on what you timed?
>
> Is there an easy way to get a measure of how many GB went
> through those SATA channels during what you timed? That
> would give an idea of what faster phusical disk memory
> access would do for you. If many people are waiting longer
> that they like, maybe they would chip in to fund an upgrade,
> to feed that 12(24)-core "beast" :-)

I really doubt it's IO bound, from experience building it multiple
times.  But I don't have precise data; if you are interested in learning
more, I encourage you to run something like 'guix build --check
--no-grafts rust' or 'guix build --check --no-grafts -e '(@@ (gnu
packages rust) rust-1.55) on your machine monitor the load via atop or
similar tools.

-- 
Thanks,
Maxim


^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: How long does it take to run the full rustc bootstrap chain?
  2022-10-26 19:37   ` bokr
  2022-10-27 14:35     ` Maxim Cournoyer
@ 2022-11-06  9:08     ` Efraim Flashner
  1 sibling, 0 replies; 12+ messages in thread
From: Efraim Flashner @ 2022-11-06  9:08 UTC (permalink / raw)
  To: bokr; +Cc: Maxim Cournoyer, Félix Baylac Jacqué, guix-devel

[-- Attachment #1: Type: text/plain, Size: 3165 bytes --]

On Wed, Oct 26, 2022 at 09:37:32PM +0200, bokr@bokr.com wrote:
> Hi,
> 
> On +2022-10-22 09:48:50 -0400, Maxim Cournoyer wrote:
> > Hi,
> > 
> > Félix Baylac Jacqué <felix@alternativebit.fr> writes:
> > 
> > > Hey Guix,
> > >
> > > I'd be curious to know how long it takes to run the full rustc bootstrap
> > > chain on the Guix build farm. I'm sadly not sure how to approach this
> > > problem.
> > >
> > > Is there a way to extract this information from Cuirass or the Guix data
> > > service?
> > >
> > > Félix
> > 
> > It used to be 16 hours on a Ryzen 3900x machine, then it got halved to 8
> > hours with the work to bootstrap from 1.39, and recently we're
> > bootstrapping from 1.54, so it must have been greatly reduced again.
> > 
> > Looking at (gnu packages rust), the mrustc-based bootstrap starts with
> > 1.54.0.  This one is expensive, probably around 1 h 30 or more on a
> > Ryzen 3900x CPU (24 logical CPUs).
> > 
> > The intermediate builds are typically around 15-20 minutes on that
> > machines, with the last one taking a bit more (30 minutes), so the
> > current bootstrap on such a machine should take about:
> > 
> > 1.54.0: 1h30m
> > 1.55.0 - 1.60.0: 6 X 20 min = 1h20m
> > 1.60.0: final build with tests and extra tools: 30 min
> > 
> > The total should be around 3 h 20 on a fast modern x86_64 machine.  I
> > suppose the time for berlin to build it takes about this.
> > 
> > HTH!
> > 
> > -- 
> > Thanks,
> > Maxim
> > 
> 
> I'm curious what
> --8<---------------cut here---------------start------------->8---
> $ lsblk -o size,model,type,tran,vendor,name|grep -Ei 'ssd|model';echo;lspci |grep -i nvme
> --8<---------------cut here---------------end--------------->8---
> on your relevant machines would show.
> 
> I opted for the best SSD available for my purism librem13v4 at the time,
> and was really happy with seems like 10x faster than the SATA SSD in my older
> but still i7 x86_64 previous laptop. Prob really 4-5x faster.
> 
> So above combo command line now gives me
> --8<---------------cut here---------------start------------->8---
> SIZE MODEL                          TYPE  TRAN   VENDOR   NAME
> 465.8G Samsung SSD 970 EVO Plus 500GB disk  nvme            nvme0n1
> 
> 01:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981
> $ 
> --8<---------------cut here---------------end--------------->8---
> 
> What /is/has been/ on your machines? Could your improved times be part from SSD/controller changes?
> 
> There's really a huge difference  between SATA and 4-lane pci
> (where both ends can handle it, which may require fw update or not be available)
> Obviously 4 lanes is also going to be faster than one.

  SIZE MODEL        TYPE TRAN   VENDOR NAME
931.5G NVME SSD 1TB disk nvme          nvme0n1

01:00.0 Non-Volatile memory controller: Silicon Motion, Inc. Device 2263 (rev 03)

-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

^ permalink raw reply	[flat|nested] 12+ messages in thread

end of thread, other threads:[~2022-11-06  9:08 UTC | newest]

Thread overview: 12+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-10-15  7:33 How long does it take to run the full rustc bootstrap chain? Félix Baylac Jacqué
2022-10-19 19:52 ` Efraim Flashner
2022-10-20  9:59 ` Ludovic Courtès
2022-10-20 20:11   ` Efraim Flashner
2022-10-22  8:57     ` Félix Baylac Jacqué
2022-10-22 13:48 ` Maxim Cournoyer
2022-10-26 19:37   ` bokr
2022-10-27 14:35     ` Maxim Cournoyer
2022-10-28 12:00       ` Has guix system ever run a hardware survey? Joshua Branson
2022-10-31 19:02       ` How long does it take to run the full rustc bootstrap chain? Bengt Richter
2022-11-01 12:49         ` Maxim Cournoyer
2022-11-06  9:08     ` Efraim Flashner

Code repositories for project(s) associated with this public inbox

	https://git.savannah.gnu.org/cgit/guix.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).