Discussion: How to package rust crates now and in future?

Discussion: How to package rust crates now and in future?

[Hartmut Goebel]

[-- Attachment #1: Type: text/plain, Size: 3760 bytes --]

Hi,

I suggest to discuss this with a broader community.

*TIL: Shall rust packages be packaged with #:skip-build #t? Shall tests 
be run for all crates?*
*My vote: skip-build #t, no tests*

Background: I just submitted some patches for some crates, setting 
#:skip-build for those I touched or added. My rational:

1) Building crate "libraries" is of no use. Rust still has no notion of 
"libraries", neither shared not static. it does not even provide any 
means to use "object"-files from another package. All crates will be 
build again and again for each package using it. And you will notice 
that the output of most crates will be almost empty (only exception: if 
the crate build a program).

2) This is what the crates importer does: It sets skip-build for all 
packages it imports as dependencies. It also does not add the 
crate-build-dependencies for these packages. (Please note that while I 
made the crates importer to honor semver versions, this has already been 
prepared in other patches and was not argued about.


Am 17.12.20 um 21:08 schrieb Efraim Flashner:
> I'm in favor of building the packages anyway, it serves as a check that
> the inputs are actually correct.

When I started packaging crates, I did this too. But then I learned, 
that others do not. So we should define how this should be handled in 
the future - and adjust the importer accordingly.

This might not be possible - as there is another issue in the Rust 
ecosystem: The language is still moving fast.

3) If some packages requires rustc 1.46, while our default rustc is 
still 1.40, we need to add rustc-1.46 as an input to this package and to 
many of it's dependents. (AFAIR the package will even depend on *both* 
version then.) Now if we move on to rustc 1.50, extra care has to be 
taken to remove these dependencies.

Even worse: All packages depending on such a package on will also depend 
on rustc 1.46, and all changes to rustc 1.46 will trigger a rebuild - 
without *any* use.

4) Since (2) building rust packages costs *a lot* of resources: time, 
memory and electrical power. As an example, building sequoia takes about 
20 Minutes on my machine. Most of the time is spend compiling 
dependencies of dependencies. And all these dependencies of dependencies 
will be compiled over and over again.

5) *If* we decide to build dependencies, we should restrict this to 
*one* build. This means: either not run the tests or only do a test 
build. The reason is: When running the tests, all the code, including 
the dependencies, is compiled again with some "test" flag set. This will 
add yet another huge amount of time, memory and electrical power.

To give you some figure: A release and test build for sequoia takes 
about 45 minutes on my machine, requiring 9 GB of space in /tmp. So this 
is double the time if the release build only.

I can't imaging how many hours it would take to rebuild sequoia is one 
of the lower level dependencies changes - which is quite often the case 
in rust.

6) This not only effect berlin, but also every user out there requiring 
a rebuild for some reason. This will lead to a very, very bad user 
experience - practically kicking out users with less powerful equipment.

7) Given the rushing climate crisis, we MUST NOT waste this gigantic 
amount of electrical power. We are in a position of huge impact. If we 
decide to save power, hundreds of Guix users will save power (and 
money). If we decide to waste power, this will multiply by the number of 
Guix users.

*It's our responsibility to protect the earth!**Yes to #:skip-build #t.*

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |


[-- Attachment #2: Type: text/html, Size: 5235 bytes --]

Re: Discussion: How to package rust crates now and in future?

[John Soo]

[-- Attachment #1: Type: text/plain, Size: 357 bytes --]

     Hey Hartmut,  

  
I’m not sure which way I fall here. I think probably keepijg ci on for most crates makes sense if we can work instead towards real shared libraries.
  

  
In any case, I would like to propose a working group for rust. Perhaps we can meet monthly in jitsi or elsewhere.
  

  
What do you think?
  

  
- John
     

[-- Attachment #2: Type: text/html, Size: 429 bytes --]

Re: Discussion: How to package rust crates now and in future?

[Pjotr Prins]

Hi Hartmut,

We are using Rust from Guix and just a few comments/thoughts:

On Fri, Dec 18, 2020 at 07:08:24PM +0100, Hartmut Goebel wrote:
>    Hi,
> 
>    I suggest to discuss this with a broader community.
> 
>    TIL: Shall rust packages be packaged with #:skip-build #t? Shall tests
>    be run for all crates?
> 
>    My vote: skip-build #t, no tests
> 
>    Background: I just submitted some patches for some crates, setting
>    #:skip-build for those I touched or added. My rational:
> 
>    1) Building crate "libraries" is of no use. Rust still has no notion of
>    "libraries", neither shared not static. it does not even provide any
>    means to use "object"-files from another package. All crates will be
>    build again and again for each package using it. And you will notice
>    that the output of most crates will be almost empty (only exception: if
>    the crate build a program).

You probably know this, but for the benefit of others: Rust builds
static binaries from source. That is their 'philosophy', see

  https://rust-cli.github.io/book/tutorial/packaging.html

This goes against the grain of Unix shared libraries. Building all
crates is what Rust does.  Correct me if I am wrong.

They are talking about librification of the language
https://rust-lang.github.io/compiler-team/minutes/design-meeting/2020-03-12-shared-library-for-types/
which may lead to a wider idea of libraries.

>    2) This is what the crates importer does: It sets skip-build for all
>    packages it imports as dependencies. It also does not add the
>    crate-build-dependencies for these packages. (Please note that while I
>    made the crates importer to honor semver versions, this has already
>    been prepared in other patches and was not argued about.
> 
>    Am 17.12.20 um 21:08 schrieb Efraim Flashner:
> 
> I'm in favor of building the packages anyway, it serves as a check that
> the inputs are actually correct.
> 
>    When I started packaging crates, I did this too. But then I learned,
>    that others do not. So we should define how this should be handled in
>    the future - and adjust the importer accordingly.
> 
>    This might not be possible - as there is another issue in the  Rust
>    ecosystem: The language is still moving fast.
> 
>    3) If some packages requires rustc 1.46, while our default rustc is
>    still 1.40, we need to add rustc-1.46 as an input to this package and
>    to many of it's dependents. (AFAIR the package will even depend on
>    *both* version then.) Now if we move on to rustc 1.50, extra care has
>    to be taken to remove these dependencies.
> 
>    Even worse: All packages depending on such a package on will also
>    depend on rustc 1.46, and all changes to rustc 1.46 will trigger a
>    rebuild - without *any* use.

The language may still be changing - but I think it has gotten to the
point that they can't change too much without hurting large software
projects. It probably is a bad idea to mix different compilers to
build crates in one software stack. Maybe I am misreading your idea,
but we either have a 1.40 build or a 1.46 build.

>    4) Since (2) building rust packages costs *a lot* of resources: time,
>    memory and electrical power. As an example, building sequoia takes
>    about 20 Minutes on my machine. Most of the time is spend compiling
>    dependencies of dependencies. And all these dependencies of
>    dependencies will be compiled over and over again.
> 
>    5) *If* we decide to build dependencies, we should restrict this to
>    *one* build. This means: either not run the tests or only do a test
>    build. The reason is: When running the tests, all the code, including
>    the dependencies, is compiled again with some "test" flag set. This
>    will add yet another huge amount of time, memory and electrical power.
> 
>    To give you some figure: A release and test build for sequoia takes
>    about 45 minutes on my machine, requiring 9 GB of space in /tmp. So
>    this is double the time if the release build only.
> 
>    I can't imaging how many hours it would take to rebuild sequoia is one
>    of the lower level dependencies changes - which is quite often the case
>    in rust.
> 
>    6) This not only effect berlin, but also every user out there requiring
>    a rebuild for some reason. This will lead to a very, very bad user
>    experience - practically kicking out users with less powerful
>    equipment.

The Rust user experience is that Rust builds all crates or installs a
single (static) binary. Even for Guix developers, the default is to
use cargo because of the insane number of dependencies it typically
pulls in. We package dependencies when we want to deploy, not to
develop. 

The user here is a guix user, of course. So between compiler versions
we rebuild the stack and then distribute binaries. No rebuilds get
triggered normally. I think the use case you are referring to is users
as developers (right?) who are actually used to waiting for cargo. I
wait for cargo when I start a new Guix development environment.

Still, it is a relief that cargo does not distribute binaries. They
have not gotten there yet :)

>    7) Given the rushing climate crisis, we MUST NOT waste this gigantic
>    amount of electrical power. We are in a position of huge impact. If we
>    decide to save power, hundreds of Guix users will save power (and
>    money). If we decide to waste power, this will multiply by the number
>    of Guix users.
> 
>    It's our responsibility to protect the earth! Yes to #:skip-build #t.

It is. I said so years ago for the insane time we wait for tests to
pass with many packages. 

I think a super fast build/deploy system for Rust would be fantastic
and an *advertisement* for Rust developers to opt for Guix instead of
them rolling out their own solutions. So, if we can come up with a
solution that is not contrarian to the philosophy of Guix I am all for
that!! 

Pj.

Re: Discussion: How to package rust crates now and in future?

[Hartmut Goebel]

Hi Pjotr,

Am 19.12.20 um 07:42 schrieb Pjotr Prins:
> They are talking about librification of the language
> https://rust-lang.github.io/compiler-team/minutes/design-meeting/2020-03-12-shared-library-for-types/
> which may lead to a wider idea of libraries.

For your needs static libraries would suffice. Anyway: Unfortunately
this feature did not make i to the 2021 roadmap, AFAICS.

I'd be happy enable building crates when rust has some notion of
library. Until then I strongly propose to not build intermediate creates
per default.

> The language may still be changing - but I think it has gotten to the
> point that they can't change too much without hurting large software
> projects. It probably is a bad idea to mix different compilers to
> build crates in one software stack. Maybe I am misreading your idea,
> but we either have a 1.40 build or a 1.46 build.

Unfortunately this is not just an idea, but a sad reality :-( As of
today, the rust used by cargo-build-system is still 1.45, while some
crates (e.g. sequoia) require 1.46 language features. So in fact we
already/currently have a mixed stack. Maybe this is just since nobody
did update default rust to rust 1.46?!

Anyhow, my argument about rebuilding without any use still holds: When
upgrading rust, all crates will be be rebuild (which is part of guix's
concept) and the results are thrown away right away - which is a wast of
time and electrical power.


>>    6) This not only effect berlin, but also every user out there requiring
>>    a rebuild for some reason. This will lead to a very, very bad user
>>    experience - practically kicking out users with less powerful
>>    equipment.
> The Rust user experience is that Rust builds all crates or installs a
> single (static) binary. Even for Guix developers, the default is to
> use cargo because of the insane number of dependencies it typically
> pulls in. We package dependencies when we want to deploy, not to
> develop. 

I assume you are referring "Rust user" to someone using rust for
developing some rust software? I'm referring "user" to someone using
Guix for managing their system.

> The user here is a guix user, of course. So between compiler versions
> we rebuild the stack and then distribute binaries. No rebuilds get
> triggered normally. I think the use case you are referring to is users
> as developers (right?) who are actually used to waiting for cargo. I
> wait for cargo when I start a new Guix development environment.

FMPOV there is not much difference between a user and a developer here:
If substitutes are not available, a user - and this includes developers
- needs to build the packages. And even as a developer I don't want to
build the whole stack myself, but use substitutes - like I want to use
pre-built libraries for other languages.

My point is another one: While for programming languages supporting
(static) libraries the number of compiles is O(n), for rust – assuming
#:skip-build #f – it is O(n*(num_average_dependencies+1)) - since each
crate needs to compile all of its dependencies, even if 20 other creates
did this already.

To give you some numbers: sequoia 0.20.0, while directly depending on
only 55 crates, pulls in more then 430 crates (in words: four hundred
and thirty) and compiles about 280 (in words: two hundred and eighty),
including some duplicates compiles. Assuming each package depends on one
other crate (which is a low assumption), this will be 2*n=860 compiles.
Assuming each compile takes 30 seconds, and half of the packages need
rebuild, this is more the 3,5 hours - right away into the trashcan.


-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |

Re: Discussion: How to package rust crates now and in future?

[Hartmut Goebel]

Am 18.12.20 um 19:26 schrieb John Soo:
> I think probably keepijg ci on for most crates makes sense if we can
> work instead towards real shared libraries.

This discussion (FMPOV) is about how handle crates until rust supports
(static or shared) libraries.

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |