Hello Guix! Over the last few days I’ve been head-down working on ‘wip-build-systems-gexp’, the mythical branch that brings gexps to build systems and packages, so we can say goodbye to ‘build-expression->derivation’. And… it’s quite a ride! I rebased it on ‘core-updates’, which was a bit tedious (it hadn’t been touched in 3 years or so). There are still test failures and build systems not yet converted, but that’s the easy part, although both are time-consuming. The more difficult part is performance. On current ‘core-updates’ I get: --8<---------------cut here---------------start------------->8--- $ GUIX_PROFILING=gc time ./pre-inst-env guix build qemu -d --no-grafts /gnu/store/z27l6plrxr5wm7818xhj9mdll99jcqz3-qemu-5.1.0.drv Garbage collection statistics: heap size: 80.52 MiB allocated: 191.14 MiB GC times: 15 time spent in GC: 0.66 seconds (32% of user time) 2.05user 0.12system 0:01.93elapsed 112%CPU (0avgtext+0avgdata 242872maxresident)k 0inputs+0outputs (0major+34228minor)pagefaults 0swaps --8<---------------cut here---------------end--------------->8--- but on ‘wip-build-systems-gexp’ I get: --8<---------------cut here---------------start------------->8--- $ GUIX_PROFILING=gc time ./pre-inst-env guix build qemu -d --no-grafts /gnu/store/5n44l8cmrmkr747nsqbxpm4764jdsl3l-qemu-5.1.0.drv Garbage collection statistics: heap size: 80.52 MiB allocated: 249.89 MiB GC times: 16 time spent in GC: 0.75 seconds (34% of user time) 2.22user 0.13system 0:02.06elapsed 114%CPU (0avgtext+0avgdata 243532maxresident)k 0inputs+0outputs (0major+34304minor)pagefaults 0swaps --8<---------------cut here---------------end--------------->8--- In short, 30% more garbage allocated and an 8% slowdown. I’ve made progress identifying and mitigating what seems to be the main cause of this (the fact that it’s possible to ungexp a list and that list will be scanned in its entirety in search of file-like objects—convenient but expensive) but as you can see, there’s still a lot to do. Statprof and gcprof are not super helpful here; I feel a need for better tools here, or better metrics. Anyway, that’s the situation. If you have ideas or if you’re in a profiling & optimization mood, now’s the time to unleash your creativity. :-) Ludo’.
Hi!
Ludovic Courtès <ludo@gnu.org> skribis:
> but on ‘wip-build-systems-gexp’ I get:
>
> $ GUIX_PROFILING=gc time ./pre-inst-env guix build qemu -d --no-grafts
> /gnu/store/5n44l8cmrmkr747nsqbxpm4764jdsl3l-qemu-5.1.0.drv
> Garbage collection statistics:
> heap size: 80.52 MiB
> allocated: 249.89 MiB
> GC times: 16
> time spent in GC: 0.75 seconds (34% of user time)
> 2.22user 0.13system 0:02.06elapsed 114%CPU (0avgtext+0avgdata 243532maxresident)k
> 0inputs+0outputs (0major+34304minor)pagefaults 0swaps
>
> In short, 30% more garbage allocated and an 8% slowdown.
In the meantime, librsvg on core-updates switched to the version that
depends on Rust. As a result, the above command, on core-updates, went
to ~2.25s on my laptop (+10–20%!).
Would be nice to investigate. One issue is that ‘cargo-build-system’
traverses packages. The ‘crate-closure’ call in there certainly adds a
fair bit of overhead. Could we avoid it?
Thanks,
Ludo’.
Hi!
Ludovic Courtès <ludo@gnu.org> skribis:
> I’ve made progress identifying and mitigating what seems to be the main
> cause of this (the fact that it’s possible to ungexp a list and that
> list will be scanned in its entirety in search of file-like
> objects—convenient but expensive) but as you can see, there’s still a
> lot to do.
I cherry-picked to ‘master’ the gexp optimizations I made on
‘wip-build-systems-gexp’. The good news is that it pays off on system
compilations, going from (commit 3c548c3e0eade12d3e86b1201dbd95863ca64ea7):
--8<---------------cut here---------------start------------->8---
$ GUIX_PROFILING="gc object-cache" ./pre-inst-env guix system build -d --no-grafts gnu/system/examples/desktop.tmpl
/gnu/store/g6kd2blajq6yja80516m85zv07d4ysb3-system.drv
Store object cache:
fresh caches: 2
lookups: 119282
hits: 116225 (97.4%)
Garbage collection statistics:
heap size: 119.43 MiB
allocated: 649.07 MiB
GC times: 26
time spent in GC: 2.15 seconds (37% of user time)
--8<---------------cut here---------------end--------------->8---
to (commit 0a912b3a434c9ff9515f9036feae4d3a7075fb2f):
--8<---------------cut here---------------start------------->8---
$ GUIX_PROFILING="gc object-cache" ./pre-inst-env guix system build -d --no-grafts gnu/system/examples/desktop.tmpl
/gnu/store/f63c8rk8rwrfr5cl4s74hnx25l3z1hrp-system.drv
Store object cache:
fresh caches: 2
lookups: 119285
hits: 116227 (97.4%)
cache size: 3058 entries
Garbage collection statistics:
heap size: 111.43 MiB
allocated: 608.01 MiB
GC times: 25
time spent in GC: 2.00 seconds (37% of user time)
--8<---------------cut here---------------end--------------->8---
Reduced heap allocations (-6%), reduce heap size (-7%), and reduced run
time (ditto). Getting better!
Ludo’.
Hello! Ludovic Courtès <ludo@gnu.org> skribis: > Over the last few days I’ve been head-down working on > ‘wip-build-systems-gexp’, the mythical branch that brings gexps to build > systems and packages, so we can say goodbye to > ‘build-expression->derivation’. And… it’s quite a ride! The current tip of ‘wip-build-systems-gexp’ Just Works; it’s being built, it can build ‘guix’ and cross-build things like ‘sed’: https://data.guix-patches.cbaines.net/repository/2/branch/wip-build-systems-gexp https://ci.guix.gnu.org/jobset/wip-build-systems-gexp (though Cuirass currently has unrelated problems) In terms of performance, there’s still a ~10% slowdown when computing derivations compared to the ‘core-updates’ revision the branch is based on. Here’s what I’d like to do in the coming days, if that doesn’t interfere with what others have in mind for the upcoming release: • Monitor build failures due to typos/thinkos made while adjusting build systems; • Merge on ‘core-updates’. Then there are optimizations to work on, but that can take a bit longer. In particular, in ‘gexp->derivation’, allow file-like objects to be specified as environment variable values. In turn, use that so that, say, ‘gnu-build-system’ has a single builder for all its packages and just calls ‘getenv’ to get the value of its various parameters, similar to what (guix git-download) does. That said, if people think it’s too late in the cycle, we could just as well keep it for the next cycle. Thoughts? Thanks, Ludo’.
[-- Attachment #1: Type: text/plain, Size: 2092 bytes --] If we aim at merging c-u before next release, we should probably wait next cycle, as this introduces quite a lot of changes, and packages might break in subtle ways. 10% increase for computing derivations is not great :/ it already takes a long time to do that on my arm system ^^". I wonder how we could improve that? Le 10 mars 2021 06:09:23 GMT-05:00, "Ludovic Courtès" <ludo@gnu.org> a écrit : >Hello! > >Ludovic Courtès <ludo@gnu.org> skribis: > >> Over the last few days I’ve been head-down working on >> ‘wip-build-systems-gexp’, the mythical branch that brings gexps to >build >> systems and packages, so we can say goodbye to >> ‘build-expression->derivation’. And… it’s quite a ride! > >The current tip of ‘wip-build-systems-gexp’ Just Works; it’s being >built, >it can build ‘guix’ and cross-build things like ‘sed’: > >https://data.guix-patches.cbaines.net/repository/2/branch/wip-build-systems-gexp > > https://ci.guix.gnu.org/jobset/wip-build-systems-gexp (though Cuirass > currently has unrelated problems) > >In terms of performance, there’s still a ~10% slowdown when computing >derivations compared to the ‘core-updates’ revision the branch is based >on. > >Here’s what I’d like to do in the coming days, if that doesn’t >interfere >with what others have in mind for the upcoming release: > > • Monitor build failures due to typos/thinkos made while adjusting > build systems; > > • Merge on ‘core-updates’. > >Then there are optimizations to work on, but that can take a bit >longer. >In particular, in ‘gexp->derivation’, allow file-like objects to be >specified as environment variable values. In turn, use that so that, >say, ‘gnu-build-system’ has a single builder for all its packages and >just calls ‘getenv’ to get the value of its various parameters, similar >to what (guix git-download) does. > >That said, if people think it’s too late in the cycle, we could just as >well keep it for the next cycle. > >Thoughts? > >Thanks, >Ludo’. [-- Attachment #2: Type: text/html, Size: 2683 bytes --]
Hi Ludo, On Wed, 10 Mar 2021 at 12:09, Ludovic Courtès <ludo@gnu.org> wrote: > The current tip of ‘wip-build-systems-gexp’ Just Works; it’s being built, > it can build ‘guix’ and cross-build things like ‘sed’: > > https://data.guix-patches.cbaines.net/repository/2/branch/wip-build-systems-gexp > > https://ci.guix.gnu.org/jobset/wip-build-systems-gexp (though Cuirass > currently has unrelated problems) Cool! > In terms of performance, there’s still a ~10% slowdown when computing > derivations compared to the ‘core-updates’ revision the branch is based > on. Less cool! :-) > Here’s what I’d like to do in the coming days, if that doesn’t interfere > with what others have in mind for the upcoming release: > > • Monitor build failures due to typos/thinkos made while adjusting > build systems; > > • Merge on ‘core-updates’. [...] > That said, if people think it’s too late in the cycle, we could just as > well keep it for the next cycle. I do not think it interferes with the release since for now and except a big change, the plan is to release without the core-updates merge. Well, that’s my understanding of the previous discussion. Cheers, simon
On Wed, Mar 10, 2021 at 03:12:42PM +0100, zimoun wrote:
> I do not think it interferes with the release since for now and except a
> big change, the plan is to release without the core-updates merge.
> Well, that’s my understanding of the previous discussion.
That's my understand as well.
Hello! Ludovic Courtès <ludo@gnu.org> skribis: > Ludovic Courtès <ludo@gnu.org> skribis: > >> Over the last few days I’ve been head-down working on >> ‘wip-build-systems-gexp’, the mythical branch that brings gexps to build >> systems and packages, so we can say goodbye to >> ‘build-expression->derivation’. And… it’s quite a ride! > > The current tip of ‘wip-build-systems-gexp’ Just Works; it’s being built, > it can build ‘guix’ and cross-build things like ‘sed’: > > https://data.guix-patches.cbaines.net/repository/2/branch/wip-build-systems-gexp > > https://ci.guix.gnu.org/jobset/wip-build-systems-gexp (though Cuirass > currently has unrelated problems) It’s building and well! > In terms of performance, there’s still a ~10% slowdown when computing > derivations compared to the ‘core-updates’ revision the branch is based > on. I made some improvements yesterday (reducing object cache lookups and the number of entries therein), but we’re still in the 10% ballpark. WIP branch: --8<---------------cut here---------------start------------->8--- $ git log |head -5 commit 082df93be3472e0f38970634260af8c432420b35 Author: Ludovic Courtès <ludo@gnu.org> Date: Mon Mar 8 13:59:23 2021 +0100 gnu: docbook-xsl: Move 'use-modules' form to the top level. $ time GUIX_PROFILING=gc ./pre-inst-env guix build libreoffice --no-grafts -d /gnu/store/fsrbbi8vfrwwdz2dlyzpfvvnky03nczz-libreoffice-6.4.7.2.drv Garbage collection statistics: heap size: 87.18 MiB allocated: 254.25 MiB GC times: 16 time spent in GC: 0.74 seconds (31% of user time) real 0m2.225s user 0m2.415s sys 0m0.087s --8<---------------cut here---------------end--------------->8--- Compared to ‘core-updates’: --8<---------------cut here---------------start------------->8--- $ git log |head -5 commit b35581bd63d929e83d18f42b067f63efc867353c Author: Efraim Flashner <efraim@flashner.co.il> Date: Sun Mar 21 09:42:06 2021 +0200 gnu: openjpeg: Update to 2.4.0. $ time GUIX_PROFILING=gc ./pre-inst-env guix build libreoffice --no-grafts -d /gnu/store/irdhm6jx30bgdxvgb0an1mn223rzshkg-libreoffice-6.4.7.2.drv Garbage collection statistics: heap size: 79.18 MiB allocated: 216.51 MiB GC times: 16 time spent in GC: 0.74 seconds (33% of user time) real 0m2.094s user 0m2.277s sys 0m0.106s --8<---------------cut here---------------end--------------->8--- > Here’s what I’d like to do in the coming days, if that doesn’t interfere > with what others have in mind for the upcoming release: > > • Monitor build failures due to typos/thinkos made while adjusting > build systems; > > • Merge on ‘core-updates’. I’ll go ahead with that if there are no objections. > Then there are optimizations to work on, but that can take a bit longer. > In particular, in ‘gexp->derivation’, allow file-like objects to be > specified as environment variable values. In turn, use that so that, > say, ‘gnu-build-system’ has a single builder for all its packages and > just calls ‘getenv’ to get the value of its various parameters, similar > to what (guix git-download) does. I’m also starting work in this area. Ludo’.
Ludovic Courtès <ludo@gnu.org> writes: > Hello! > > Ludovic Courtès <ludo@gnu.org> skribis: > >> Ludovic Courtès <ludo@gnu.org> skribis: >> >>> Over the last few days I’ve been head-down working on >>> ‘wip-build-systems-gexp’, the mythical branch that brings gexps to build >>> systems and packages, so we can say goodbye to >>> ‘build-expression->derivation’. And… it’s quite a ride! >> >> The current tip of ‘wip-build-systems-gexp’ Just Works; it’s being built, >> it can build ‘guix’ and cross-build things like ‘sed’: >> >> https://data.guix-patches.cbaines.net/repository/2/branch/wip-build-systems-gexp >> >> https://ci.guix.gnu.org/jobset/wip-build-systems-gexp (though Cuirass >> currently has unrelated problems) > > It’s building and well! Excellent! It’s great to see that you brought this idea across all the hurdles you encountered in previous attempts. Inspiring! >> Here’s what I’d like to do in the coming days, if that doesn’t interfere >> with what others have in mind for the upcoming release: >> >> • Monitor build failures due to typos/thinkos made while adjusting >> build systems; >> >> • Merge on ‘core-updates’. > > I’ll go ahead with that if there are no objections. This sounds reasonable to me. “core-updates” will need some work to push it into shape for the next merge, but I don’t think merging a well-monitored branch into “core-updates” would make the job considerably harder. Thank you! -- Ricardo
Hi Ludo! Ludovic Courtès <ludo@gnu.org> writes: > Hello! > > Ludovic Courtès <ludo@gnu.org> skribis: > >> Ludovic Courtès <ludo@gnu.org> skribis: >> >>> Over the last few days I’ve been head-down working on >>> ‘wip-build-systems-gexp’, the mythical branch that brings gexps to build >>> systems and packages, so we can say goodbye to >>> ‘build-expression->derivation’. And… it’s quite a ride! >> >> The current tip of ‘wip-build-systems-gexp’ Just Works; it’s being built, >> it can build ‘guix’ and cross-build things like ‘sed’: >> >> https://data.guix-patches.cbaines.net/repository/2/branch/wip-build-systems-gexp >> >> https://ci.guix.gnu.org/jobset/wip-build-systems-gexp (though Cuirass >> currently has unrelated problems) > > It’s building and well! > >> In terms of performance, there’s still a ~10% slowdown when computing >> derivations compared to the ‘core-updates’ revision the branch is based >> on. > > I made some improvements yesterday (reducing object cache lookups and > the number of entries therein), but we’re still in the 10% ballpark. > WIP branch: > > $ git log |head -5 > commit 082df93be3472e0f38970634260af8c432420b35 > Author: Ludovic Courtès <ludo@gnu.org> > Date: Mon Mar 8 13:59:23 2021 +0100 > > gnu: docbook-xsl: Move 'use-modules' form to the top level. > $ time GUIX_PROFILING=gc ./pre-inst-env guix build libreoffice --no-grafts -d > /gnu/store/fsrbbi8vfrwwdz2dlyzpfvvnky03nczz-libreoffice-6.4.7.2.drv > Garbage collection statistics: > heap size: 87.18 MiB > allocated: 254.25 MiB > GC times: 16 > time spent in GC: 0.74 seconds (31% of user time) > > real 0m2.225s > user 0m2.415s > sys 0m0.087s > > > Compared to ‘core-updates’: > > $ git log |head -5 > commit b35581bd63d929e83d18f42b067f63efc867353c > Author: Efraim Flashner <efraim@flashner.co.il> > Date: Sun Mar 21 09:42:06 2021 +0200 > > gnu: openjpeg: Update to 2.4.0. > $ time GUIX_PROFILING=gc ./pre-inst-env guix build libreoffice --no-grafts -d > /gnu/store/irdhm6jx30bgdxvgb0an1mn223rzshkg-libreoffice-6.4.7.2.drv > Garbage collection statistics: > heap size: 79.18 MiB > allocated: 216.51 MiB > GC times: 16 > time spent in GC: 0.74 seconds (33% of user time) > > real 0m2.094s > user 0m2.277s > sys 0m0.106s This looks reasonable! >> Here’s what I’d like to do in the coming days, if that doesn’t interfere >> with what others have in mind for the upcoming release: >> >> • Monitor build failures due to typos/thinkos made while adjusting >> build systems; >> >> • Merge on ‘core-updates’. > > I’ll go ahead with that if there are no objections. Sounds good! Thanks for picking up this work! :-) Maxim
Hi Maxim and all, So this is it, 2021 *is* the year of build systems on gexps! I’ve rebased it and pushed on ‘core-updates’ with 2eafeb2f3d661061bc412c3f27c90202e4532532 as the tip. The build farm has a bit of work to do now it seems. Hmm actually the evaluation at <https://ci.guix.gnu.org/jobset/core-updates> just failed with a yellow cross, not sure what that means. Anyway, I’ll be monitoring in the coming days, but do ping me when you encounter problems! It’s crazy that it took 6 years (six!) to get past the finish line. The initial reason for not merging it was that I was worried about performance regressions; there’s still work to do in this area, as I wrote earlier, but it’s looking good. Another reason is that I was afraid of unlocking features that would break assumptions we make about packages—for instance, that all the inputs are specified in the various inputs fields, and that these fields are all we need to care about when doing graph rewriting. (I now think this fear was unwarranted.) And then, rebasing the branch became harder every time, there were more and more build systems that needed to be converted… and there were always more pressing issues to work on. What’s the lesson here? I don’t know! Had it been done six years ago, it would have had to go through a number of iterations: ‘with-imported-modules’ and ‘with-extensions’ didn’t exist, the template mechanism in (guix monads) didn’t exist, self-referential records (‘this-package’, etc.) weren’t there. It would have worked though, and perhaps we’d have explored other parts of the design space, who knows. Ludo’.
Hi Ludo, Ludovic Courtès <ludo@gnu.org> writes: > So this is it, 2021 *is* the year of build systems on gexps! Wow! Congratulations! I am very happy to see this finally come together. > It’s crazy that it took 6 years (six!) to get past the finish line. I think I’ve only known what a gexp is for three years... and that’s about how long I’ve been excited by this feature. Thanks for sticking with it for six years! -- Tim