Re: Performance of computing cross derivations

[Efraim Flashner <efraim@flashner.co.il>]

[-- Attachment #1.1: Type: text/plain, Size: 2800 bytes --]

On Thu, Jan 11, 2024 at 01:26:35PM +0000, Christopher Baines wrote:
> 
> Ludovic Courtès <ludo@gnu.org> writes:
> 
> > Christopher Baines <mail@cbaines.net> skribis:
> >
> >> I think you're right, while I send some other changes in #68266, I think
> >> it's this change around make-rust-sysroot that has pretty much all the
> >> effects on performance.
> >>
> >> I think the tens of thousands of duplicated packages from cross-base
> >> that I was looking at are almost entirely coming from
> >> make-rust-sysroot. As Ludo mentions in [1], maybe this has something to
> >> do with use of cross- procedures in native-inputs, although I'm not sure
> >> that moving those calls out of native-inputs is a correct thing to do.
> >>
> >> I don't know what the correct approach here is, but I think something
> >> needs doing here to address the performance regression.
> >
> > I probably missed it in the thread: what commit caused the regression,
> > and how can I test any changes?  I’m willing to help but I missed some
> > of the context.
> 
> It's not a pure performance regression, more that in it's current form,
> rust cross derivations are very expensive to compute. It's been this way
> since cross-compiling was enabled in [1].
> 
> 1: https://git.savannah.gnu.org/cgit/guix.git/patch/?id=e604972d9c697302691aeb22e9c50c933a1a3c72
> 
> I've been looking at data service slowness in processing revisions over
> the last few weeks, and I think it's mostly down to this. Looking at the
> revision prior to the change [2], computing all the derivations took
> around 3 hours, which is ages, but still quick compared to the nearly 9
> hours it took after this change [3].
> 
> 2: https://data.guix.gnu.org/revision/58bbb38c5bd2e42aab9e9408d8c9d8da3409f178
> 3: https://data.guix.gnu.org/revision/c9e1a72cc27925484635ae01bc4de28bf232689d
> 
> Obviously having more derivations is good and that usually means more
> work for the data service, but in this case it seems like things can be
> sped up quite a bit.
> 
> For testing locally, I've been computing all the derivations for
> i586-pc-gnu, but Efraim also posted a concise command to look at
> computing some cross derivations for a subset of rust packages [4].
> 
> 4: https://lists.gnu.org/archive/html/guix-devel/2024-01/msg00053.html

list-all-cargo-build-system-packages is actually a script I have locally
that I should probably put in the etc/teams/rust folder.  I've attached
it in case anyone wants to try it out, or see the speed-up of computing
the cross-derivations.

-- 
Efraim Flashner   <efraim@flashner.co.il>   רנשלפ םירפא
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #1.2: list-all-cargo-build-system-packages --]
[-- Type: text/plain, Size: 253 bytes --]

guile -c '(use-modules (gnu packages)(guix packages)(guix build-system)) (display (fold-packages (lambda (package lst) (if (eq? (build-system-name (package-build-system package)) (quote cargo)) (cons package lst) lst)) (list)))' | tr ' ' '\n' | grep \@

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]