unofficial mirror of bug-guix@gnu.org 
 help / color / mirror / Atom feed
* bug#35139: Rust builds systematically time out
@ 2019-04-04  8:59 Ludovic Courtès
  2019-04-04  9:28 ` Pierre Langlois
                   ` (3 more replies)
  0 siblings, 4 replies; 10+ messages in thread
From: Ludovic Courtès @ 2019-04-04  8:59 UTC (permalink / raw)
  To: bug-Guix; +Cc: Ivan Petkov

Hello,

On berlin, Rust 1.24.1 builds systematically exceed the timeout:

--8<---------------cut here---------------start------------->8---
Building stage1 compiler artifacts (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
   Compiling arena v0.0.0 (file:///tmp/guix-build-rust-1.24.1.drv-0/rustc-1.24.1-src/src/libarena)
   Compiling rustc_driver v0.0.0 (file:///tmp/guix-build-rust-1.24.1.drv-0/rustc-1.24.1-src/src/librustc_driver)

[...]

   Compiling rls-data v0.14.0
   Compiling rustc_data_structures v0.0.0 (file:///tmp/guix-build-rust-1.24.1.drv-0/rustc-1.24.1-src/src/librustc_data_structures)
   Compiling flate2 v1.0.1
   Compiling syntax_pos v0.0.0 (file:///tmp/guix-build-rust-1.24.1.drv-0/rustc-1.24.1-src/src/libsyntax_pos)
   Compiling rustc_errors v0.0.0 (file:///tmp/guix-build-rust-1.24.1.drv-0/rustc-1.24.1-src/src/librustc_errors)
   Compiling backtrace v0.3.4
guix offload: error: timeout expired while offloading '/gnu/store/61bd22d9mg3xl260jwddisiahh3kmanj-rust-1.24.1.drv'
--8<---------------cut here---------------end--------------->8---

Strangely, the build lasts ~9000 seconds (2.5 hours) on the front-end
node of berlin¹, and the timeout for guix-daemon on berlin is 6h (see
guix-maintenance.git) while the max-silent-time is 1h.

The build nodes may be slower than the front-end, but still, it seems
unlikely that it would take more than 6h there.  (That could happen if
the test suite, which lasts 2.1h, were “embarrassingly parallel”, but
we’re running tests with ‘-j1’.)

To summarize, there are two problems:

  1. Rust takes too long to build.  What can we do about it?  Enable
     parallel builds?

  2. Offloaded builds seem to time out prematurely or something.

Thoughts?

Ludo’.

¹ See <https://ci.guix.info/log/rkrnm3rr7g6fhr17160vn1mz5rdzh9lv-rust-1.24.1>
  for timings.

^ permalink raw reply	[flat|nested] 10+ messages in thread

* bug#35139: Rust builds systematically time out
  2019-04-04  8:59 bug#35139: Rust builds systematically time out Ludovic Courtès
@ 2019-04-04  9:28 ` Pierre Langlois
  2019-04-04 15:47   ` Ivan Petkov
  2019-04-05 21:18 ` mikadoZero
                   ` (2 subsequent siblings)
  3 siblings, 1 reply; 10+ messages in thread
From: Pierre Langlois @ 2019-04-04  9:28 UTC (permalink / raw)
  To: 35139; +Cc: Ivan Petkov

Hello!

Ludovic Courtès writes:

> Hello,
>
> On berlin, Rust 1.24.1 builds systematically exceed the timeout:
>
> --8<---------------cut here---------------start------------->8---
> Building stage1 compiler artifacts (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
>    Compiling arena v0.0.0 (file:///tmp/guix-build-rust-1.24.1.drv-0/rustc-1.24.1-src/src/libarena)
>    Compiling rustc_driver v0.0.0 (file:///tmp/guix-build-rust-1.24.1.drv-0/rustc-1.24.1-src/src/librustc_driver)
>
> [...]
>
>    Compiling rls-data v0.14.0
>    Compiling rustc_data_structures v0.0.0 (file:///tmp/guix-build-rust-1.24.1.drv-0/rustc-1.24.1-src/src/librustc_data_structures)
>    Compiling flate2 v1.0.1
>    Compiling syntax_pos v0.0.0 (file:///tmp/guix-build-rust-1.24.1.drv-0/rustc-1.24.1-src/src/libsyntax_pos)
>    Compiling rustc_errors v0.0.0 (file:///tmp/guix-build-rust-1.24.1.drv-0/rustc-1.24.1-src/src/librustc_errors)
>    Compiling backtrace v0.3.4
> guix offload: error: timeout expired while offloading '/gnu/store/61bd22d9mg3xl260jwddisiahh3kmanj-rust-1.24.1.drv'
> --8<---------------cut here---------------end--------------->8---
>
> Strangely, the build lasts ~9000 seconds (2.5 hours) on the front-end
> node of berlin¹, and the timeout for guix-daemon on berlin is 6h (see
> guix-maintenance.git) while the max-silent-time is 1h.
>
> The build nodes may be slower than the front-end, but still, it seems
> unlikely that it would take more than 6h there.  (That could happen if
> the test suite, which lasts 2.1h, were “embarrassingly parallel”, but
> we’re running tests with ‘-j1’.)
>
> To summarize, there are two problems:
>
>   1. Rust takes too long to build.  What can we do about it?  Enable
>      parallel builds?

One thing I suggested in the past was to remove the check phase *only*
for rust packages used for bootstrapping. This way we still run the
tests for the final rust but not at every step in the chain.

Although, I wonder if we're more likely to miss a bug if we do this, I'm
not sure.

For reference: https://lists.gnu.org/archive/html/guix-patches/2018-11/msg00453.html

Thanks,
Pierre

^ permalink raw reply	[flat|nested] 10+ messages in thread

* bug#35139: Rust builds systematically time out
  2019-04-04  9:28 ` Pierre Langlois
@ 2019-04-04 15:47   ` Ivan Petkov
  2019-04-04 16:06     ` Ludovic Courtès
  0 siblings, 1 reply; 10+ messages in thread
From: Ivan Petkov @ 2019-04-04 15:47 UTC (permalink / raw)
  To: Pierre Langlois, Ludovic Courtès; +Cc: 35139

[-- Attachment #1: Type: text/plain, Size: 1553 bytes --]


> On Apr 4, 2019, at 1:59 AM, Ludovic Courtès <ludo@gnu.org> wrote:
> 
> The build nodes may be slower than the front-end, but still, it seems
> unlikely that it would take more than 6h there.  (That could happen if
> the test suite, which lasts 2.1h, were “embarrassingly parallel”, but
> we’re running tests with ‘-j1’.)
> 
> To summarize, there are two problems:
> 
>  1. Rust takes too long to build.  What can we do about it?  Enable
>     parallel builds?

Rust tests are designed to run in parallel, as long as you have enough
RAM, file descriptors, etc. available on the machine for the amount of
concurrency being used. The compiler test suite is largely just compiling
files, so the most important resource is probably available RAM/swap.

> On Apr 4, 2019, at 2:28 AM, Pierre Langlois <pierre.langlois@gmx.com> wrote:
> 
> One thing I suggested in the past was to remove the check phase *only*
> for rust packages used for bootstrapping. This way we still run the
> tests for the final rust but not at every step in the chain.
> 
> Although, I wonder if we're more likely to miss a bug if we do this, I'm
> not sure.

Although that definitely will speed the bootstrap chain, I’m concerned that
if a dependency package ever gets updated and breaks things we wouldn’t
know without running the test suite.

Maybe if the bootstrapped versions don’t ever change skipping the check
phase will be safe, but I think we should try running parallel tests first
and see how far that gets us.

—Ivan

[-- Attachment #2: Type: text/html, Size: 6427 bytes --]

^ permalink raw reply	[flat|nested] 10+ messages in thread

* bug#35139: Rust builds systematically time out
  2019-04-04 15:47   ` Ivan Petkov
@ 2019-04-04 16:06     ` Ludovic Courtès
  2019-04-04 17:37       ` Ivan Petkov
  2020-12-18 10:29       ` Mathieu Othacehe
  0 siblings, 2 replies; 10+ messages in thread
From: Ludovic Courtès @ 2019-04-04 16:06 UTC (permalink / raw)
  To: Ivan Petkov; +Cc: 35139

Ivan Petkov <ivanppetkov@gmail.com> skribis:

>> On Apr 4, 2019, at 1:59 AM, Ludovic Courtès <ludo@gnu.org> wrote:
>> 
>> The build nodes may be slower than the front-end, but still, it seems
>> unlikely that it would take more than 6h there.  (That could happen if
>> the test suite, which lasts 2.1h, were “embarrassingly parallel”, but
>> we’re running tests with ‘-j1’.)
>> 
>> To summarize, there are two problems:
>> 
>>  1. Rust takes too long to build.  What can we do about it?  Enable
>>     parallel builds?
>
> Rust tests are designed to run in parallel, as long as you have enough
> RAM, file descriptors, etc. available on the machine for the amount of
> concurrency being used. The compiler test suite is largely just compiling
> files, so the most important resource is probably available RAM/swap.

Perhaps we could start with:

  "-j" (number->string (min (parallel-job-count) 2))

?

> Maybe if the bootstrapped versions don’t ever change skipping the check
> phase will be safe, but I think we should try running parallel tests first
> and see how far that gets us.

Sounds like a good start.

So the only reason we’re running tests sequentially is because of memory
usage concerns?

Thanks,
Ludo’.

^ permalink raw reply	[flat|nested] 10+ messages in thread

* bug#35139: Rust builds systematically time out
  2019-04-04 16:06     ` Ludovic Courtès
@ 2019-04-04 17:37       ` Ivan Petkov
  2020-12-18 10:29       ` Mathieu Othacehe
  1 sibling, 0 replies; 10+ messages in thread
From: Ivan Petkov @ 2019-04-04 17:37 UTC (permalink / raw)
  To: Ludovic Courtès; +Cc: 35139

[-- Attachment #1: Type: text/plain, Size: 1605 bytes --]

Danny’s got a patch for turning on parallel tests in #35126

Not sure why the previous tests were running sequentially, but there is a comment somewhere saying it’s to avoid EAGAIN errors.

--Ivan

> On Apr 4, 2019, at 9:06 AM, Ludovic Courtès <ludo@gnu.org> wrote:
> 
> Ivan Petkov <ivanppetkov@gmail.com> skribis:
> 
>>> On Apr 4, 2019, at 1:59 AM, Ludovic Courtès <ludo@gnu.org> wrote:
>>> 
>>> The build nodes may be slower than the front-end, but still, it seems
>>> unlikely that it would take more than 6h there.  (That could happen if
>>> the test suite, which lasts 2.1h, were “embarrassingly parallel”, but
>>> we’re running tests with ‘-j1’.)
>>> 
>>> To summarize, there are two problems:
>>> 
>>> 1. Rust takes too long to build.  What can we do about it?  Enable
>>>    parallel builds?
>> 
>> Rust tests are designed to run in parallel, as long as you have enough
>> RAM, file descriptors, etc. available on the machine for the amount of
>> concurrency being used. The compiler test suite is largely just compiling
>> files, so the most important resource is probably available RAM/swap.
> 
> Perhaps we could start with:
> 
>  "-j" (number->string (min (parallel-job-count) 2))
> 
> ?
> 
>> Maybe if the bootstrapped versions don’t ever change skipping the check
>> phase will be safe, but I think we should try running parallel tests first
>> and see how far that gets us.
> 
> Sounds like a good start.
> 
> So the only reason we’re running tests sequentially is because of memory
> usage concerns?
> 
> Thanks,
> Ludo’.

[-- Attachment #2: Type: text/html, Size: 3903 bytes --]

^ permalink raw reply	[flat|nested] 10+ messages in thread

* bug#35139: Rust builds systematically time out
  2019-04-04  8:59 bug#35139: Rust builds systematically time out Ludovic Courtès
  2019-04-04  9:28 ` Pierre Langlois
@ 2019-04-05 21:18 ` mikadoZero
  2020-03-30  5:42 ` John Soo
  2021-11-21  6:09 ` Maxim Cournoyer
  3 siblings, 0 replies; 10+ messages in thread
From: mikadoZero @ 2019-04-05 21:18 UTC (permalink / raw)
  To: Ludovic Courtès; +Cc: bug-Guix, Ivan Petkov

When I try to install rust I get similar behavior.  It does not finish
building.  The longest I have let it try for was around 12 hours. That
was is a on a machine with 1GB RAM and 10GB SWAP.

Ludovic Courtès writes:

> Hello,
>
> On berlin, Rust 1.24.1 builds systematically exceed the timeout:
>
> --8<---------------cut here---------------start------------->8---
> Building stage1 compiler artifacts (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
>    Compiling arena v0.0.0 (file:///tmp/guix-build-rust-1.24.1.drv-0/rustc-1.24.1-src/src/libarena)
>    Compiling rustc_driver v0.0.0 (file:///tmp/guix-build-rust-1.24.1.drv-0/rustc-1.24.1-src/src/librustc_driver)
>
> [...]
>
>    Compiling rls-data v0.14.0
>    Compiling rustc_data_structures v0.0.0 (file:///tmp/guix-build-rust-1.24.1.drv-0/rustc-1.24.1-src/src/librustc_data_structures)
>    Compiling flate2 v1.0.1
>    Compiling syntax_pos v0.0.0 (file:///tmp/guix-build-rust-1.24.1.drv-0/rustc-1.24.1-src/src/libsyntax_pos)
>    Compiling rustc_errors v0.0.0 (file:///tmp/guix-build-rust-1.24.1.drv-0/rustc-1.24.1-src/src/librustc_errors)
>    Compiling backtrace v0.3.4
> guix offload: error: timeout expired while offloading '/gnu/store/61bd22d9mg3xl260jwddisiahh3kmanj-rust-1.24.1.drv'
> --8<---------------cut here---------------end--------------->8---
>
> Strangely, the build lasts ~9000 seconds (2.5 hours) on the front-end
> node of berlin¹, and the timeout for guix-daemon on berlin is 6h (see
> guix-maintenance.git) while the max-silent-time is 1h.
>
> The build nodes may be slower than the front-end, but still, it seems
> unlikely that it would take more than 6h there.  (That could happen if
> the test suite, which lasts 2.1h, were “embarrassingly parallel”, but
> we’re running tests with ‘-j1’.)
>
> To summarize, there are two problems:
>
>   1. Rust takes too long to build.  What can we do about it?  Enable
>      parallel builds?
>
>   2. Offloaded builds seem to time out prematurely or something.
>
> Thoughts?
>
> Ludo’.
>
> ¹ See <https://ci.guix.info/log/rkrnm3rr7g6fhr17160vn1mz5rdzh9lv-rust-1.24.1>
>   for timings.

^ permalink raw reply	[flat|nested] 10+ messages in thread

* bug#35139: Rust builds systematically time out
  2019-04-04  8:59 bug#35139: Rust builds systematically time out Ludovic Courtès
  2019-04-04  9:28 ` Pierre Langlois
  2019-04-05 21:18 ` mikadoZero
@ 2020-03-30  5:42 ` John Soo
  2021-11-21  6:09 ` Maxim Cournoyer
  3 siblings, 0 replies; 10+ messages in thread
From: John Soo @ 2020-03-30  5:42 UTC (permalink / raw)
  To: 35139; +Cc: Pierre Neidhardt

Hi everyone,

Is this still happening? It looks like rust-1.24.1 is completing successfully on both ci servers.

- John

^ permalink raw reply	[flat|nested] 10+ messages in thread

* bug#35139: Rust builds systematically time out
  2019-04-04 16:06     ` Ludovic Courtès
  2019-04-04 17:37       ` Ivan Petkov
@ 2020-12-18 10:29       ` Mathieu Othacehe
  2020-12-18 10:45         ` zimoun
  1 sibling, 1 reply; 10+ messages in thread
From: Mathieu Othacehe @ 2020-12-18 10:29 UTC (permalink / raw)
  To: Ludovic Courtès; +Cc: 35139, Ivan Petkov


Hello,

>>>  1. Rust takes too long to build.  What can we do about it?  Enable
>>>     parallel builds?

I've noticed that Rust packages are also built with "-j1". Evaluations
such as: https://ci.guix.gnu.org/eval/19873 are causing rebuilds of many
Rust packages, hence monopolizing the build farm for hours.

Would it be possible to enable parallel building for Rust packages as
suggested by Ludo in this thread?

Thanks,

Mathieu




^ permalink raw reply	[flat|nested] 10+ messages in thread

* bug#35139: Rust builds systematically time out
  2020-12-18 10:29       ` Mathieu Othacehe
@ 2020-12-18 10:45         ` zimoun
  0 siblings, 0 replies; 10+ messages in thread
From: zimoun @ 2020-12-18 10:45 UTC (permalink / raw)
  To: Mathieu Othacehe, Ludovic Courtès; +Cc: 35139, Ivan Petkov

Hi Mathieu,

On Fri, 18 Dec 2020 at 11:29, Mathieu Othacehe <othacehe@gnu.org> wrote:
> Hello,
>
>>>>  1. Rust takes too long to build.  What can we do about it?  Enable
>>>>     parallel builds?
>
> I've noticed that Rust packages are also built with "-j1". Evaluations
> such as: https://ci.guix.gnu.org/eval/19873 are causing rebuilds of many
> Rust packages, hence monopolizing the build farm for hours.
>
> Would it be possible to enable parallel building for Rust packages as
> suggested by Ludo in this thread?

Does the parallel builds build reproductibly?

All the best,
simon




^ permalink raw reply	[flat|nested] 10+ messages in thread

* bug#35139: Rust builds systematically time out
  2019-04-04  8:59 bug#35139: Rust builds systematically time out Ludovic Courtès
                   ` (2 preceding siblings ...)
  2020-03-30  5:42 ` John Soo
@ 2021-11-21  6:09 ` Maxim Cournoyer
  3 siblings, 0 replies; 10+ messages in thread
From: Maxim Cournoyer @ 2021-11-21  6:09 UTC (permalink / raw)
  To: Ludovic Courtès; +Cc: 35139-done, Ivan Petkov

Hello,

Ludovic Courtès <ludo@gnu.org> writes:

> Hello,
>
> On berlin, Rust 1.24.1 builds systematically exceed the timeout:
>
> Building stage1 compiler artifacts (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
>    Compiling arena v0.0.0 (file:///tmp/guix-build-rust-1.24.1.drv-0/rustc-1.24.1-src/src/libarena)
>    Compiling rustc_driver v0.0.0 (file:///tmp/guix-build-rust-1.24.1.drv-0/rustc-1.24.1-src/src/librustc_driver)
>
> [...]
>
>    Compiling rls-data v0.14.0
>    Compiling rustc_data_structures v0.0.0 (file:///tmp/guix-build-rust-1.24.1.drv-0/rustc-1.24.1-src/src/librustc_data_structures)
>    Compiling flate2 v1.0.1
>    Compiling syntax_pos v0.0.0 (file:///tmp/guix-build-rust-1.24.1.drv-0/rustc-1.24.1-src/src/libsyntax_pos)
>    Compiling rustc_errors v0.0.0 (file:///tmp/guix-build-rust-1.24.1.drv-0/rustc-1.24.1-src/src/librustc_errors)
>    Compiling backtrace v0.3.4
> guix offload: error: timeout expired while offloading '/gnu/store/61bd22d9mg3xl260jwddisiahh3kmanj-rust-1.24.1.drv'
>
> Strangely, the build lasts ~9000 seconds (2.5 hours) on the front-end
> node of berlin¹, and the timeout for guix-daemon on berlin is 6h (see
> guix-maintenance.git) while the max-silent-time is 1h.

With the recent improvement in the Rust bootstrap toolchains, I'm
considering this fixed.

If there are still timeouts, Cuirass is now supposed to honor the
'timeout' property.

Closing!

Maxim




^ permalink raw reply	[flat|nested] 10+ messages in thread

end of thread, other threads:[~2021-11-21  6:10 UTC | newest]

Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-04-04  8:59 bug#35139: Rust builds systematically time out Ludovic Courtès
2019-04-04  9:28 ` Pierre Langlois
2019-04-04 15:47   ` Ivan Petkov
2019-04-04 16:06     ` Ludovic Courtès
2019-04-04 17:37       ` Ivan Petkov
2020-12-18 10:29       ` Mathieu Othacehe
2020-12-18 10:45         ` zimoun
2019-04-05 21:18 ` mikadoZero
2020-03-30  5:42 ` John Soo
2021-11-21  6:09 ` Maxim Cournoyer

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).