Will 2021 be the year of build systems on gexps?

Will 2021 be the year of build systems on gexps?

[Ludovic Courtès]

Hello Guix!

Over the last few days I’ve been head-down working on
‘wip-build-systems-gexp’, the mythical branch that brings gexps to build
systems and packages, so we can say goodbye to
‘build-expression->derivation’.  And… it’s quite a ride!

I rebased it on ‘core-updates’, which was a bit tedious (it hadn’t been
touched in 3 years or so).  There are still test failures and build
systems not yet converted, but that’s the easy part, although both are
time-consuming.

The more difficult part is performance.  On current ‘core-updates’ I get:

--8<---------------cut here---------------start------------->8---
$ GUIX_PROFILING=gc time ./pre-inst-env guix build qemu -d --no-grafts
/gnu/store/z27l6plrxr5wm7818xhj9mdll99jcqz3-qemu-5.1.0.drv
Garbage collection statistics:
  heap size:        80.52 MiB
  allocated:        191.14 MiB
  GC times:         15
  time spent in GC: 0.66 seconds (32% of user time)
2.05user 0.12system 0:01.93elapsed 112%CPU (0avgtext+0avgdata 242872maxresident)k
0inputs+0outputs (0major+34228minor)pagefaults 0swaps
--8<---------------cut here---------------end--------------->8---

but on ‘wip-build-systems-gexp’ I get:

--8<---------------cut here---------------start------------->8---
$ GUIX_PROFILING=gc time ./pre-inst-env guix build qemu -d --no-grafts
/gnu/store/5n44l8cmrmkr747nsqbxpm4764jdsl3l-qemu-5.1.0.drv
Garbage collection statistics:
  heap size:        80.52 MiB
  allocated:        249.89 MiB
  GC times:         16
  time spent in GC: 0.75 seconds (34% of user time)
2.22user 0.13system 0:02.06elapsed 114%CPU (0avgtext+0avgdata 243532maxresident)k
0inputs+0outputs (0major+34304minor)pagefaults 0swaps
--8<---------------cut here---------------end--------------->8---

In short, 30% more garbage allocated and an 8% slowdown.

I’ve made progress identifying and mitigating what seems to be the main
cause of this (the fact that it’s possible to ungexp a list and that
list will be scanned in its entirety in search of file-like
objects—convenient but expensive) but as you can see, there’s still a
lot to do.

Statprof and gcprof are not super helpful here; I feel a need for better
tools here, or better metrics.

Anyway, that’s the situation.  If you have ideas or if you’re in a
profiling & optimization mood, now’s the time to unleash your
creativity.  :-)

Ludo’.

Rust packages and performance impact on Guix

[Ludovic Courtès]

Hi!

Ludovic Courtès <ludo@gnu.org> skribis:

> but on ‘wip-build-systems-gexp’ I get:
>
> $ GUIX_PROFILING=gc time ./pre-inst-env guix build qemu -d --no-grafts
> /gnu/store/5n44l8cmrmkr747nsqbxpm4764jdsl3l-qemu-5.1.0.drv
> Garbage collection statistics:
>   heap size:        80.52 MiB
>   allocated:        249.89 MiB
>   GC times:         16
>   time spent in GC: 0.75 seconds (34% of user time)
> 2.22user 0.13system 0:02.06elapsed 114%CPU (0avgtext+0avgdata 243532maxresident)k
> 0inputs+0outputs (0major+34304minor)pagefaults 0swaps
>
> In short, 30% more garbage allocated and an 8% slowdown.

In the meantime, librsvg on core-updates switched to the version that
depends on Rust.  As a result, the above command, on core-updates, went
to ~2.25s on my laptop (+10–20%!).

Would be nice to investigate.  One issue is that ‘cargo-build-system’
traverses packages.  The ‘crate-closure’ call in there certainly adds a
fair bit of overhead.  Could we avoid it?

Thanks,
Ludo’.

Re: Will 2021 be the year of build systems on gexps?

[Ludovic Courtès]

Hi!

Ludovic Courtès <ludo@gnu.org> skribis:

> I’ve made progress identifying and mitigating what seems to be the main
> cause of this (the fact that it’s possible to ungexp a list and that
> list will be scanned in its entirety in search of file-like
> objects—convenient but expensive) but as you can see, there’s still a
> lot to do.

I cherry-picked to ‘master’ the gexp optimizations I made on
‘wip-build-systems-gexp’.  The good news is that it pays off on system
compilations, going from (commit 3c548c3e0eade12d3e86b1201dbd95863ca64ea7):

--8<---------------cut here---------------start------------->8---
$ GUIX_PROFILING="gc object-cache" ./pre-inst-env guix system build -d --no-grafts gnu/system/examples/desktop.tmpl
/gnu/store/g6kd2blajq6yja80516m85zv07d4ysb3-system.drv
Store object cache:
  fresh caches:     2
  lookups:      119282
  hits:         116225 (97.4%)
Garbage collection statistics:
  heap size:        119.43 MiB
  allocated:        649.07 MiB
  GC times:         26
  time spent in GC: 2.15 seconds (37% of user time)
--8<---------------cut here---------------end--------------->8---

to (commit 0a912b3a434c9ff9515f9036feae4d3a7075fb2f):

--8<---------------cut here---------------start------------->8---
$ GUIX_PROFILING="gc object-cache" ./pre-inst-env guix system build -d --no-grafts gnu/system/examples/desktop.tmpl
/gnu/store/f63c8rk8rwrfr5cl4s74hnx25l3z1hrp-system.drv
Store object cache:
  fresh caches:     2
  lookups:      119285
  hits:         116227 (97.4%)
  cache size:    3058 entries
Garbage collection statistics:
  heap size:        111.43 MiB
  allocated:        608.01 MiB
  GC times:         25
  time spent in GC: 2.00 seconds (37% of user time)
--8<---------------cut here---------------end--------------->8---

Reduced heap allocations (-6%), reduce heap size (-7%), and reduced run
time (ditto).  Getting better!

Ludo’.