* Performance improvements
@ 2019-10-14 21:39 Ludovic Courtès
2019-10-15 6:35 ` Pierre Neidhardt
` (4 more replies)
0 siblings, 5 replies; 10+ messages in thread
From: Ludovic Courtès @ 2019-10-14 21:39 UTC (permalink / raw)
To: Guix-devel
Hello Guix!
I just wanted to share that recent commits have improved the performance
of ‘package-derivation’ and related operations quite a bit:
8f417ed280 gnu: commencement: Further optimize the package object graph.
f618134e4c build-system/gnu: 'package-with-explicit-inputs' uses 'package-mapping'.
dab669e075 gnu: ld-wrapper: Memoize.
099dbc4fd3 gnu: Improve memoization of 'package-with-bootstrap-guile'.
99b73d0f0c gnu: commencement: Reduce the graph of package objects.
9a45a24f7f gnu: Remove unnecessary uses of 'package-with-bootstrap-guile'.
What these commits do is that they greatly reduce the graph of <package>
objects built in ‘commencement.scm’. As a result, there’s much less
work to do, less code to run, less stuff to memoize, all that. :-)
Before these changes, if you’d run, say:
guix graph -e '(@@ (gnu packages commencement) gnu-make-final)'
you’d see the <package> graph was huge and had a weird shape. Indeed,
many parts were duplicated as a consequence of graph rewriting. That’s
not news, but it became more visible with the reduced binary seed
boostrap, which adds quite a few nodes and edges.
The commit logs have more details, but the take-away is:
--8<---------------cut here---------------start------------->8---
$ time /var/guix/profiles/per-user/ludo/current-guix-109-link/bin/guix build libreoffice -nd
/gnu/store/7whsss0gn7h4dqvz627sq3i4cb1qlc1v-libreoffice-6.1.5.2.drv
real 0m3.238s
user 0m3.693s
sys 0m0.047s
$ time guix build libreoffice -nd
/gnu/store/8drmbhsrayr2j5lkvrwq37rg8g06hgsw-libreoffice-6.1.5.2.drv
real 0m2.142s
user 0m2.323s
sys 0m0.082s
$ guix describe
Generacio 110 Oct 14 2019 08:43:33 (nuna)
guix bd04fe8
repository URL: https://git.savannah.gnu.org/git/guix.git
branch: master
commit: bd04fe878627a14533d908ccdf5b906050d6e0a4
--8<---------------cut here---------------end--------------->8---
I think we need to aim for 1s, but that’s already a good step.
Ludo’.
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Performance improvements
2019-10-14 21:39 Performance improvements Ludovic Courtès
@ 2019-10-15 6:35 ` Pierre Neidhardt
2019-10-15 7:24 ` Jan Nieuwenhuizen
` (3 subsequent siblings)
4 siblings, 0 replies; 10+ messages in thread
From: Pierre Neidhardt @ 2019-10-15 6:35 UTC (permalink / raw)
To: Ludovic Courtès, Guix-devel
[-- Attachment #1: Type: text/plain, Size: 106 bytes --]
This is fantastic! Thanks for much for working on this!
--
Pierre Neidhardt
https://ambrevar.xyz/
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Performance improvements
2019-10-14 21:39 Performance improvements Ludovic Courtès
2019-10-15 6:35 ` Pierre Neidhardt
@ 2019-10-15 7:24 ` Jan Nieuwenhuizen
2019-10-19 20:17 ` Ludovic Courtès
2019-10-15 8:21 ` Dimakakos Dimos
` (2 subsequent siblings)
4 siblings, 1 reply; 10+ messages in thread
From: Jan Nieuwenhuizen @ 2019-10-15 7:24 UTC (permalink / raw)
To: Ludovic Courtès; +Cc: Guix-devel
Ludovic Courtès writes:
> I just wanted to share that recent commits have improved the performance
> of ‘package-derivation’ and related operations quite a bit:
>
> 8f417ed280 gnu: commencement: Further optimize the package object graph.
> f618134e4c build-system/gnu: 'package-with-explicit-inputs' uses 'package-mapping'.
> dab669e075 gnu: ld-wrapper: Memoize.
> 099dbc4fd3 gnu: Improve memoization of 'package-with-bootstrap-guile'.
> 99b73d0f0c gnu: commencement: Reduce the graph of package objects.
> 9a45a24f7f gnu: Remove unnecessary uses of 'package-with-bootstrap-guile'.
Ah, that's great! We get graph images that are actually usable, and
this makes more space for me to slightly worsen things again with my
Scheme-only bootstrap experiments, coming real soon now ;-)
Greetings,
janneke
--
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Performance improvements
2019-10-15 7:24 ` Jan Nieuwenhuizen
@ 2019-10-19 20:17 ` Ludovic Courtès
0 siblings, 0 replies; 10+ messages in thread
From: Ludovic Courtès @ 2019-10-19 20:17 UTC (permalink / raw)
To: Jan Nieuwenhuizen; +Cc: Guix-devel
Jan Nieuwenhuizen <janneke@gnu.org> skribis:
> Ludovic Courtès writes:
>
>> I just wanted to share that recent commits have improved the performance
>> of ‘package-derivation’ and related operations quite a bit:
>>
>> 8f417ed280 gnu: commencement: Further optimize the package object graph.
>> f618134e4c build-system/gnu: 'package-with-explicit-inputs' uses 'package-mapping'.
>> dab669e075 gnu: ld-wrapper: Memoize.
>> 099dbc4fd3 gnu: Improve memoization of 'package-with-bootstrap-guile'.
>> 99b73d0f0c gnu: commencement: Reduce the graph of package objects.
>> 9a45a24f7f gnu: Remove unnecessary uses of 'package-with-bootstrap-guile'.
>
> Ah, that's great! We get graph images that are actually usable,
Yes, that’s a pleasant side effect. :-)
> and this makes more space for me to slightly worsen things again with
> my Scheme-only bootstrap experiments, coming real soon now ;-)
Yay!
Ludo’.
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Performance improvements
2019-10-14 21:39 Performance improvements Ludovic Courtès
2019-10-15 6:35 ` Pierre Neidhardt
2019-10-15 7:24 ` Jan Nieuwenhuizen
@ 2019-10-15 8:21 ` Dimakakos Dimos
2019-10-15 18:21 ` Efraim Flashner
2019-10-27 23:06 ` Ludovic Courtès
4 siblings, 0 replies; 10+ messages in thread
From: Dimakakos Dimos @ 2019-10-15 8:21 UTC (permalink / raw)
To: guix-devel
That is really great! Thanks for your hard work!
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Performance improvements
2019-10-14 21:39 Performance improvements Ludovic Courtès
` (2 preceding siblings ...)
2019-10-15 8:21 ` Dimakakos Dimos
@ 2019-10-15 18:21 ` Efraim Flashner
2019-10-19 20:17 ` Ludovic Courtès
2019-10-27 23:06 ` Ludovic Courtès
4 siblings, 1 reply; 10+ messages in thread
From: Efraim Flashner @ 2019-10-15 18:21 UTC (permalink / raw)
To: Ludovic Courtès; +Cc: Guix-devel
[-- Attachment #1: Type: text/plain, Size: 227 bytes --]
Blog post?
--
Efraim Flashner <efraim@flashner.co.il> אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D 14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Performance improvements
2019-10-14 21:39 Performance improvements Ludovic Courtès
` (3 preceding siblings ...)
2019-10-15 18:21 ` Efraim Flashner
@ 2019-10-27 23:06 ` Ludovic Courtès
2019-10-28 7:44 ` Pierre Neidhardt
2019-10-28 7:56 ` Efraim Flashner
4 siblings, 2 replies; 10+ messages in thread
From: Ludovic Courtès @ 2019-10-27 23:06 UTC (permalink / raw)
To: Guix-devel
Hi!
Some more improvements come from these commits:
ac19950507 gnu: bootstrap: Cache the 'bootstrap-executable' origins.
5166d02730 gnu: bootstrap: 'bootstrap-origin' preserves eq?-ness when no changes are made.
d727a9343d derivations: Don't memoize 'derivation->bytevector'.
f5fca9a82c gexp: Cache the module to derivation mappings.
c57e417eff store: Allow objects in the cache to be inserted and search for with 'equal?'.
f58b45350b gexp: Add 'imported+compiled-modules'.
f726f6f802 derivations: 'build-expression->derivation' caches its module derivations.
cf7648f882 derivations: Introduce 'imported+compiled-modules'.
This caches the module-to-derivation mapping, which wasn’t cached until
now.
Overall the goal is to have the high-level caches work nicely (the
“object cache” in particular), rendering the low-level caches (such as
the ‘add-data-to-store’ cache) potentially useless. This is beneficial
because a cache hit on the object cache saves more CPU time than a cache
hit on a low-level cache.
Ludovic Courtès <ludo@gnu.org> skribis:
> $ time guix build libreoffice -nd
> /gnu/store/8drmbhsrayr2j5lkvrwq37rg8g06hgsw-libreoffice-6.1.5.2.drv
>
> real 0m2.142s
> user 0m2.323s
> sys 0m0.082s
> $ guix describe
> Generacio 110 Oct 14 2019 08:43:33 (nuna)
> guix bd04fe8
> repository URL: https://git.savannah.gnu.org/git/guix.git
> branch: master
> commit: bd04fe878627a14533d908ccdf5b906050d6e0a4
And now:
--8<---------------cut here---------------start------------->8---
$ time /tmp/new/bin/guix build libreoffice -nd
/gnu/store/ibzap0v1c367h0j2mdd2jmlsfx9jplfb-libreoffice-6.1.5.2.drv
real 0m1.658s
user 0m1.699s
sys 0m0.073s
$ /tmp/new/bin/guix describe
Generacio 1 Oct 28 2019 00:03:05 (nuna)
guix 271b3c9
repository URL: https://git.savannah.gnu.org/git/guix.git
branch: master
commit: 271b3c9d16f8880f4a27b52a615cdec611322fae
--8<---------------cut here---------------end--------------->8---
Ludo’.
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Performance improvements
2019-10-27 23:06 ` Ludovic Courtès
@ 2019-10-28 7:44 ` Pierre Neidhardt
2019-10-28 7:56 ` Efraim Flashner
1 sibling, 0 replies; 10+ messages in thread
From: Pierre Neidhardt @ 2019-10-28 7:44 UTC (permalink / raw)
To: Ludovic Courtès, Guix-devel
[-- Attachment #1: Type: text/plain, Size: 81 bytes --]
Only 700ms to go! :)
Congrats!
--
Pierre Neidhardt
https://ambrevar.xyz/
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Performance improvements
2019-10-27 23:06 ` Ludovic Courtès
2019-10-28 7:44 ` Pierre Neidhardt
@ 2019-10-28 7:56 ` Efraim Flashner
1 sibling, 0 replies; 10+ messages in thread
From: Efraim Flashner @ 2019-10-28 7:56 UTC (permalink / raw)
To: Ludovic Courtès; +Cc: Guix-devel
[-- Attachment #1: Type: text/plain, Size: 3376 bytes --]
On Mon, Oct 28, 2019 at 12:06:25AM +0100, Ludovic Courtès wrote:
> Hi!
>
> Some more improvements come from these commits:
>
> ac19950507 gnu: bootstrap: Cache the 'bootstrap-executable' origins.
> 5166d02730 gnu: bootstrap: 'bootstrap-origin' preserves eq?-ness when no changes are made.
> d727a9343d derivations: Don't memoize 'derivation->bytevector'.
> f5fca9a82c gexp: Cache the module to derivation mappings.
> c57e417eff store: Allow objects in the cache to be inserted and search for with 'equal?'.
> f58b45350b gexp: Add 'imported+compiled-modules'.
> f726f6f802 derivations: 'build-expression->derivation' caches its module derivations.
> cf7648f882 derivations: Introduce 'imported+compiled-modules'.
>
> This caches the module-to-derivation mapping, which wasn’t cached until
> now.
>
> Overall the goal is to have the high-level caches work nicely (the
> “object cache” in particular), rendering the low-level caches (such as
> the ‘add-data-to-store’ cache) potentially useless. This is beneficial
> because a cache hit on the object cache saves more CPU time than a cache
> hit on a low-level cache.
>
> Ludovic Courtès <ludo@gnu.org> skribis:
>
> > $ time guix build libreoffice -nd
> > /gnu/store/8drmbhsrayr2j5lkvrwq37rg8g06hgsw-libreoffice-6.1.5.2.drv
> >
> > real 0m2.142s
> > user 0m2.323s
> > sys 0m0.082s
> > $ guix describe
> > Generacio 110 Oct 14 2019 08:43:33 (nuna)
> > guix bd04fe8
> > repository URL: https://git.savannah.gnu.org/git/guix.git
> > branch: master
> > commit: bd04fe878627a14533d908ccdf5b906050d6e0a4
>
> And now:
>
> --8<---------------cut here---------------start------------->8---
> $ time /tmp/new/bin/guix build libreoffice -nd
> /gnu/store/ibzap0v1c367h0j2mdd2jmlsfx9jplfb-libreoffice-6.1.5.2.drv
>
> real 0m1.658s
> user 0m1.699s
> sys 0m0.073s
> $ /tmp/new/bin/guix describe
> Generacio 1 Oct 28 2019 00:03:05 (nuna)
> guix 271b3c9
> repository URL: https://git.savannah.gnu.org/git/guix.git
> branch: master
> commit: 271b3c9d16f8880f4a27b52a615cdec611322fae
> --8<---------------cut here---------------end--------------->8---
>
Results are even better on a slower machine:
$ time guix build libreoffice -nd
/gnu/store/ibzap0v1c367h0j2mdd2jmlsfx9jplfb-libreoffice-6.1.5.2.drv
real 0m5.408s
user 0m5.186s
sys 0m0.299s
$ time ./pre-inst-env guix build libreoffice -nd
/gnu/store/ibzap0v1c367h0j2mdd2jmlsfx9jplfb-libreoffice-6.1.5.2.drv
real 0m3.836s
user 0m3.519s
sys 0m0.235s
$ guix describe
Generation 12 Oct 27 2019 11:53:42 (current)
ietf 8c6079e
repository URL: https://gitlab.com/efraim/guix-ietf.git
branch: master
commit: 8c6079e10fdf640ce2cdafa3c1ea24961650f451
guix 24adcba
repository URL: https://git.savannah.gnu.org/git/guix.git
branch: master
commit: 24adcbab0ad903c59793dbe141950fff5058b8a2
$ ./pre-inst-env guix describe
Git checkout:
repository: /home/efraim/workspace/guix
branch: master
commit: 64e3fe04b32e2eab1144d5b0c9278eab81056bd6
--
Efraim Flashner <efraim@flashner.co.il> אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D 14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2019-10-28 7:56 UTC | newest]
Thread overview: 10+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2019-10-14 21:39 Performance improvements Ludovic Courtès
2019-10-15 6:35 ` Pierre Neidhardt
2019-10-15 7:24 ` Jan Nieuwenhuizen
2019-10-19 20:17 ` Ludovic Courtès
2019-10-15 8:21 ` Dimakakos Dimos
2019-10-15 18:21 ` Efraim Flashner
2019-10-19 20:17 ` Ludovic Courtès
2019-10-27 23:06 ` Ludovic Courtès
2019-10-28 7:44 ` Pierre Neidhardt
2019-10-28 7:56 ` Efraim Flashner
Code repositories for project(s) associated with this external index
https://git.savannah.gnu.org/cgit/guix.git
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.