‘cargo-build-system’ makes everything slow

‘cargo-build-system’ makes everything slow

[Ludovic Courtès]

[-- Attachment #1: Type: text/plain, Size: 2329 bytes --]

Hi,

I was profiling the ‘dependents’ procedure use by the new ‘guix build
--dependents’ option in (guix scripts build).  Like ‘guix refresh -l’,
it lowers all the packages to a bag, constructs the graph, and traverses
all the nodes backward.  And it’s slow (~25s).

The profile is clear (test program below):

--8<---------------cut here---------------start------------->8---
100.0% with-exception-handler at ice-9/boot-9.scm:1689:4
  100.0% thunk at guix/store.scm:686:4
    100.0% run-with-store at guix/store.scm:2196:0
      100.0% #x7f1cb0ccbeb0 at guix/scripts/build.scm:596:6
        100.0% #x7f1c9c353ef0 at guix/graph.scm:98:7
          99.6% #x7f1cb1b61e00 at guix/store.scm:1925:0
            99.6% loop at guix/store.scm:1925:0
              99.6% add-edge at guix/graph.scm:92:7
                99.6% #x7f1c9c3340bc at guix/scripts/graph.scm:187:10
                  96.3% #x15d3b3bc
                    95.8% bag-node-edges at guix/scripts/graph.scm:159:0
                      93.1% package->bag at guix/packages.scm:1755:0
                        92.5% thunk at guix/packages.scm:1763:4
                          89.3% lower at guix/build-system/cargo.scm:305:0
                            33.5% vhash-assq at ice-9/vlist.scm:539:0
                              9.4% #x15d3cf98
                            21.0% #x15d3de90
                            3.6% vhash-cons at ice-9/vlist.scm:449:0
                              0.9% #x15d3cf98
                              0.1% #x15d3b430
                                0.1% #x15d3daf4
                                  0.1% reap-pipes at ice-9/popen.scm:183:0
                            2.5% filter-map at srfi/srfi-1.scm:681:0
                              2.0% #x7f1cb0a997f4 at guix/build-system/cargo.scm:298:4
--8<---------------cut here---------------end--------------->8---

89% of the time is spent lowering ‘cargo-build-system’ packages,
specifically in ‘expand-crate-sources’, which computes more or less the
same thing over and over again.  (It also returns more than just
transitive sources of Crates, AFAICS.)

Problem is little can be done about it, at least not without triggering
a world rebuild.

But even then, it would help a lot to just not do that.

Thoughts?

Ludo’.


[-- Attachment #2: test program --]
[-- Type: text/plain, Size: 422 bytes --]

(use-modules (statprof)
             (ice-9 time)
             (guix)
             ((gnu packages) #:select (all-packages))
             ((guix scripts build) #:select (dependents))
             (gnu packages version-control))

(time (->bool (all-packages)))                    ;warm up

(with-store store
  (statprof
   (lambda ()
     (dependents store (list libgit2) 1))
   #:full-stacks? #t
   #:display-style 'tree))

Re: ‘cargo-build-system’ makes everything slow

[Ricardo Wurmus]

Hi,

Perhaps this would be a good time to revive the antioxidant-build-system
by Maxime Devos.  I see on https://issues.guix.gnu.org/64904 that
Nicolas Graves had been working on bringing it back to live, but there
are no patches.

-- 
Ricardo

Re: ‘cargo-build-system’ makes everything slow

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 1215 bytes --]

On Mon, Dec 02, 2024 at 08:24:29PM +0100, Ricardo Wurmus wrote:
> Hi,
> 
> Perhaps this would be a good time to revive the antioxidant-build-system
> by Maxime Devos.  I see on https://issues.guix.gnu.org/64904 that
> Nicolas Graves had been working on bringing it back to live, but there
> are no patches.

I still have a copy of the code on my machine but unfortunately it no
longer builds due to the constant churn of rust packages.

One thing I remember explicitly about it was that building end packages
was faster than the current method, and that was before taking into
account reusing build artifacts.

https://notabug.org/maximed/cargoless-rust-experiments

Another idea which I'm not in love with is what Debian does. They grab
all of the sources into one build environment and then build everything.
It simplifies the dependency management of the sources but for us it
would make it so that we can't touch anything in rust without causing a
full rebuild of everything.

-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

How to build Rust packages

[Ludovic Courtès]

Hello,

Efraim Flashner <efraim@flashner.co.il> skribis:

> I still have a copy of the code on my machine but unfortunately it no
> longer builds due to the constant churn of rust packages.
>
> One thing I remember explicitly about it was that building end packages
> was faster than the current method, and that was before taking into
> account reusing build artifacts.
>
> https://notabug.org/maximed/cargoless-rust-experiments

Neat.

> Another idea which I'm not in love with is what Debian does. They grab
> all of the sources into one build environment and then build everything.
> It simplifies the dependency management of the sources but for us it
> would make it so that we can't touch anything in rust without causing a
> full rebuild of everything.

I believe this is also what Nixpkgs does, as discussed in this thread:

  https://toot.aquilenet.fr/@civodul/113532478383900515

I’m not a fan either.  But I think one of the main criteria here should
be long-term maintainability, which is influenced by internal design
issues and by how we design our relation with the external packaging
tool.

By internal issues I mean things like #:cargo-inputs instead of regular
inputs, which makes the whole thing hard to maintain and causes
friction.  (See <https://issues.guix.gnu.org/53127>.)

As for the relation with Cargo and crates.io, the question is should we
map packages one-to-one?  Is it worth it?  If the answer is yes, do we
have the tools to maintain it in the long run.

Ludo’.

Re: How to build Rust packages

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 3838 bytes --]

On Thu, Dec 05, 2024 at 11:13:07AM +0100, Ludovic Courtès wrote:
> Hello,
> 
> Efraim Flashner <efraim@flashner.co.il> skribis:
> 
> > I still have a copy of the code on my machine but unfortunately it no
> > longer builds due to the constant churn of rust packages.
> >
> > One thing I remember explicitly about it was that building end packages
> > was faster than the current method, and that was before taking into
> > account reusing build artifacts.
> >
> > https://notabug.org/maximed/cargoless-rust-experiments
> 
> Neat.
> 
> > Another idea which I'm not in love with is what Debian does. They grab
> > all of the sources into one build environment and then build everything.
> > It simplifies the dependency management of the sources but for us it
> > would make it so that we can't touch anything in rust without causing a
> > full rebuild of everything.
> 
> I believe this is also what Nixpkgs does, as discussed in this thread:
> 
>   https://toot.aquilenet.fr/@civodul/113532478383900515

I'm pretty sure they parse the Cargo.lock file and download the crates
at build time.

> I’m not a fan either.  But I think one of the main criteria here should
> be long-term maintainability, which is influenced by internal design
> issues and by how we design our relation with the external packaging
> tool.
> 
> By internal issues I mean things like #:cargo-inputs instead of regular
> inputs, which makes the whole thing hard to maintain and causes
> friction.  (See <https://issues.guix.gnu.org/53127>.)
> 
> As for the relation with Cargo and crates.io, the question is should we
> map packages one-to-one?  Is it worth it?  If the answer is yes, do we
> have the tools to maintain it in the long run.

As it stands now the package name is effectively prepending 'rust-' and
switching any underscores to dashes.  Most of the actual packaging work
is making sure the cargo-inputs from patches correctly match the
versions in Cargo.toml, checking the metadata (license, home-page,
synopsis/description), and seeing if any code needs to be removed (such
as from *-sys packages).  If there are any "real" packages then they
normally don't have the rust- prefix.

I don't want to go and parse Cargo.lock, automagically generate packages
based on that, and then download those as cargo-inputs for packages. Not
only does that potentially pull in old versions of libraries which may
have necessary updates or patches, it doesn't check them for license
data or vendored C libraries.

I also don't want to keep a collection of "difficult" crates that need a
human touch and have everything else be autogenerated at package build
time.

I am jealous of the cran updater and all the work Rekado has put into
making it work well, and I know I need to actually fix a bunch of stuff
with the crates.  An updater and also the etc/committer.scm file.  There
are too many crates to actually package them all, so that wouldn't be
something workable to automatically package all of them.

I have a script that goes through the crates and lists how many
dependencies there are per file, and I have used it in the past to
remove unused crates.  I have also come back and added them back in when
something else needed them.

My workflow is I work on 20-50 crates at once, and when they all build
correctly I then break them into the appropriate number of commits.

I'm not sure where to go from here.  I don't even remember if the
antioxidant build system correctly shows the dependency path between
crates, which IMO is one of the big things missing now.

-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: How to build Rust packages

[indieterminacy]

I should point out that I am packaging Scryer-Prolog, which uses Rust in 
its underbelly.

As it stands, the divergences of crate versioning means that my package 
definition is nearly 60k LOC and Ive had to validate over 1.1k packages 
already.
Naturally, some of these are duplicates of Guix package definitions, as 
well as updates.
Nethertheless, I have a considerable bounty to unfurl once I reach 
maturity with this initiative.

It will be a job in itself to provide the actual patches and push them 
to you so there will be some lags (weeks even).
I should brush up on patch workflow in Emacs-Magit for flow.

Naturally, there are some edge cases regarding what Im packaging but Ive 
been trying to minimize attention.
Once I hit a wall should I query at the Guix-Help ML or for such a large 
package environment should I use this ML?

I suppose a link to a scm file on a git-forge (with commit) is apt 
rather than providing a file?

Oh, my TXR parsing of Guix packages is ticking along which I am doing 
this project!
I reckon it can be adapted nicely for a comparative method between 
different config files.

On 2024-12-08 09:20, Efraim Flashner wrote:
> On Thu, Dec 05, 2024 at 11:13:07AM +0100, Ludovic Courtès wrote:
>> Hello,
>> 
>> Efraim Flashner <efraim@flashner.co.il> skribis:
>> 
>> > I still have a copy of the code on my machine but unfortunately it no
>> > longer builds due to the constant churn of rust packages.
>> >
>> > One thing I remember explicitly about it was that building end packages
>> > was faster than the current method, and that was before taking into
>> > account reusing build artifacts.
>> >
>> > https://notabug.org/maximed/cargoless-rust-experiments
>> 
>> Neat.
>> 
>> > Another idea which I'm not in love with is what Debian does. They grab
>> > all of the sources into one build environment and then build everything.
>> > It simplifies the dependency management of the sources but for us it
>> > would make it so that we can't touch anything in rust without causing a
>> > full rebuild of everything.
>> 
>> I believe this is also what Nixpkgs does, as discussed in this thread:
>> 
>>   https://toot.aquilenet.fr/@civodul/113532478383900515
> 
> I'm pretty sure they parse the Cargo.lock file and download the crates
> at build time.
> 
>> I’m not a fan either.  But I think one of the main criteria here 
>> should
>> be long-term maintainability, which is influenced by internal design
>> issues and by how we design our relation with the external packaging
>> tool.
>> 
>> By internal issues I mean things like #:cargo-inputs instead of 
>> regular
>> inputs, which makes the whole thing hard to maintain and causes
>> friction.  (See <https://issues.guix.gnu.org/53127>.)
>> 
>> As for the relation with Cargo and crates.io, the question is should 
>> we
>> map packages one-to-one?  Is it worth it?  If the answer is yes, do we
>> have the tools to maintain it in the long run.
> 
> As it stands now the package name is effectively prepending 'rust-' and
> switching any underscores to dashes.  Most of the actual packaging work
> is making sure the cargo-inputs from patches correctly match the
> versions in Cargo.toml, checking the metadata (license, home-page,
> synopsis/description), and seeing if any code needs to be removed (such
> as from *-sys packages).  If there are any "real" packages then they
> normally don't have the rust- prefix.
> 
> I don't want to go and parse Cargo.lock, automagically generate 
> packages
> based on that, and then download those as cargo-inputs for packages. 
> Not
> only does that potentially pull in old versions of libraries which may
> have necessary updates or patches, it doesn't check them for license
> data or vendored C libraries.
> 
> I also don't want to keep a collection of "difficult" crates that need 
> a
> human touch and have everything else be autogenerated at package build
> time.
> 
> I am jealous of the cran updater and all the work Rekado has put into
> making it work well, and I know I need to actually fix a bunch of stuff
> with the crates.  An updater and also the etc/committer.scm file.  
> There
> are too many crates to actually package them all, so that wouldn't be
> something workable to automatically package all of them.
> 
> I have a script that goes through the crates and lists how many
> dependencies there are per file, and I have used it in the past to
> remove unused crates.  I have also come back and added them back in 
> when
> something else needed them.
> 
> My workflow is I work on 20-50 crates at once, and when they all build
> correctly I then break them into the appropriate number of commits.
> 
> I'm not sure where to go from here.  I don't even remember if the
> antioxidant build system correctly shows the dependency path between
> crates, which IMO is one of the big things missing now.

Re: How to build Rust packages

[Ludovic Courtès]

Hello,

Efraim Flashner <efraim@flashner.co.il> skribis:

> I don't want to go and parse Cargo.lock, automagically generate packages
> based on that, and then download those as cargo-inputs for packages. Not
> only does that potentially pull in old versions of libraries which may
> have necessary updates or patches, it doesn't check them for license
> data or vendored C libraries.

Yes, that makes sense to me.  A corollary is probably that Guix itself
should only focus on a subset of available Crates, with a line drawn
somewhere.

This needs to be spelled out so contributors and users know what to
expect.

> I am jealous of the cran updater and all the work Rekado has put into
> making it work well, and I know I need to actually fix a bunch of stuff
> with the crates.  An updater and also the etc/committer.scm file.  There
> are too many crates to actually package them all, so that wouldn't be
> something workable to automatically package all of them.
>
> I have a script that goes through the crates and lists how many
> dependencies there are per file, and I have used it in the past to
> remove unused crates.  I have also come back and added them back in when
> something else needed them.

The problem is that currently tools that work for the rest of Guix don’t
work for Crate packages.

There’s the proposal at <https://issues.guix.gnu.org/53127>, there’s
also antioxydant, and there’s your script and other tools people might
have.  I think this needs to be consolidated so that things remain under
control.

Ludo’.

Re: ‘cargo-build-system’ makes everything slow

[Liliana Marie Prikler]

Hi Guix,

Am Donnerstag, dem 05.12.2024 um 09:06 +0200 schrieb Efraim Flashner:
> On Mon, Dec 02, 2024 at 08:24:29PM +0100, Ricardo Wurmus wrote:
> > Perhaps this would be a good time to revive the antioxidant-build-
> > system
> […]
> I still have a copy of the code on my machine but unfortunately it no
> longer builds due to the constant churn of rust packages.
> 
> […]
> 
> Another idea which I'm not in love with is what Debian does. […]
I would like to throw a third option into the Mix: Meson.  It not only
supports Rust[1], but also has a wrapper for the cargo build system[2],
which while experimental should help us translating all the cargo bits
to Meson ones.  At least the former is actively being used in Mesa[3],
so it works as a proof of concept – for the latter, it appears that
Mesa uses hand-written meson build files.

Cheers

[1] https://mesonbuild.com/Rust.html
[2] https://mesonbuild.com/Wrap-dependency-system-manual.html#cargo-wraps
[3] https://gitlab.freedesktop.org/mesa/mesa/-/blob/24.3/src/compiler/rust/meson.build

Re: ‘cargo-build-system’ makes everything slow

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 3671 bytes --]

On Mon, Dec 02, 2024 at 05:44:16PM +0100, Ludovic Courtès wrote:
> Hi,
> 
> I was profiling the ‘dependents’ procedure use by the new ‘guix build
> --dependents’ option in (guix scripts build).  Like ‘guix refresh -l’,
> it lowers all the packages to a bag, constructs the graph, and traverses
> all the nodes backward.  And it’s slow (~25s).
> 
> The profile is clear (test program below):
> 
> --8<---------------cut here---------------start------------->8---
> 100.0% with-exception-handler at ice-9/boot-9.scm:1689:4
>   100.0% thunk at guix/store.scm:686:4
>     100.0% run-with-store at guix/store.scm:2196:0
>       100.0% #x7f1cb0ccbeb0 at guix/scripts/build.scm:596:6
>         100.0% #x7f1c9c353ef0 at guix/graph.scm:98:7
>           99.6% #x7f1cb1b61e00 at guix/store.scm:1925:0
>             99.6% loop at guix/store.scm:1925:0
>               99.6% add-edge at guix/graph.scm:92:7
>                 99.6% #x7f1c9c3340bc at guix/scripts/graph.scm:187:10
>                   96.3% #x15d3b3bc
>                     95.8% bag-node-edges at guix/scripts/graph.scm:159:0
>                       93.1% package->bag at guix/packages.scm:1755:0
>                         92.5% thunk at guix/packages.scm:1763:4
>                           89.3% lower at guix/build-system/cargo.scm:305:0
>                             33.5% vhash-assq at ice-9/vlist.scm:539:0
>                               9.4% #x15d3cf98
>                             21.0% #x15d3de90
>                             3.6% vhash-cons at ice-9/vlist.scm:449:0
>                               0.9% #x15d3cf98
>                               0.1% #x15d3b430
>                                 0.1% #x15d3daf4
>                                   0.1% reap-pipes at ice-9/popen.scm:183:0
>                             2.5% filter-map at srfi/srfi-1.scm:681:0
>                               2.0% #x7f1cb0a997f4 at guix/build-system/cargo.scm:298:4
> --8<---------------cut here---------------end--------------->8---
> 
> 89% of the time is spent lowering ‘cargo-build-system’ packages,
> specifically in ‘expand-crate-sources’, which computes more or less the
> same thing over and over again.  (It also returns more than just
> transitive sources of Crates, AFAICS.)
> 
> Problem is little can be done about it, at least not without triggering
> a world rebuild.
> 
> But even then, it would help a lot to just not do that.
> 
> Thoughts?
> 
> Ludo’.

If its a world rebuild through the cargo-build-system its not so bad, CI
and QA build it out pretty quickly.  If it's everything, then yeah,
that's going to be slow.

Can we cache it after the first time like with guix shell?  Or a
hash-table of some sort? (I'm just using big words, I don't remember the
specifics of how the different hashing and caching bits work). Chris
Baines sped up the time to compute derivations for cross-compiling with
something with the make-rust-sysroot.

> (use-modules (statprof)
>              (ice-9 time)
>              (guix)
>              ((gnu packages) #:select (all-packages))
>              ((guix scripts build) #:select (dependents))
>              (gnu packages version-control))
> 
> (time (->bool (all-packages)))                    ;warm up
> 
> (with-store store
>   (statprof
>    (lambda ()
>      (dependents store (list libgit2) 1))
>    #:full-stacks? #t
>    #:display-style 'tree))


-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]