unofficial mirror of guix-devel@gnu.org 
 help / color / mirror / Atom feed
* Will 2021 be the year of build systems on gexps?
@ 2021-02-19 16:01 Ludovic Courtès
  2021-02-23 10:31 ` Rust packages and performance impact on Guix Ludovic Courtès
  2021-02-23 14:28 ` Will 2021 be the year of build systems on gexps? Ludovic Courtès
  0 siblings, 2 replies; 3+ messages in thread
From: Ludovic Courtès @ 2021-02-19 16:01 UTC (permalink / raw)
  To: Guix-devel

Hello Guix!

Over the last few days I’ve been head-down working on
‘wip-build-systems-gexp’, the mythical branch that brings gexps to build
systems and packages, so we can say goodbye to
‘build-expression->derivation’.  And… it’s quite a ride!

I rebased it on ‘core-updates’, which was a bit tedious (it hadn’t been
touched in 3 years or so).  There are still test failures and build
systems not yet converted, but that’s the easy part, although both are
time-consuming.

The more difficult part is performance.  On current ‘core-updates’ I get:

--8<---------------cut here---------------start------------->8---
$ GUIX_PROFILING=gc time ./pre-inst-env guix build qemu -d --no-grafts
/gnu/store/z27l6plrxr5wm7818xhj9mdll99jcqz3-qemu-5.1.0.drv
Garbage collection statistics:
  heap size:        80.52 MiB
  allocated:        191.14 MiB
  GC times:         15
  time spent in GC: 0.66 seconds (32% of user time)
2.05user 0.12system 0:01.93elapsed 112%CPU (0avgtext+0avgdata 242872maxresident)k
0inputs+0outputs (0major+34228minor)pagefaults 0swaps
--8<---------------cut here---------------end--------------->8---

but on ‘wip-build-systems-gexp’ I get:

--8<---------------cut here---------------start------------->8---
$ GUIX_PROFILING=gc time ./pre-inst-env guix build qemu -d --no-grafts
/gnu/store/5n44l8cmrmkr747nsqbxpm4764jdsl3l-qemu-5.1.0.drv
Garbage collection statistics:
  heap size:        80.52 MiB
  allocated:        249.89 MiB
  GC times:         16
  time spent in GC: 0.75 seconds (34% of user time)
2.22user 0.13system 0:02.06elapsed 114%CPU (0avgtext+0avgdata 243532maxresident)k
0inputs+0outputs (0major+34304minor)pagefaults 0swaps
--8<---------------cut here---------------end--------------->8---

In short, 30% more garbage allocated and an 8% slowdown.

I’ve made progress identifying and mitigating what seems to be the main
cause of this (the fact that it’s possible to ungexp a list and that
list will be scanned in its entirety in search of file-like
objects—convenient but expensive) but as you can see, there’s still a
lot to do.

Statprof and gcprof are not super helpful here; I feel a need for better
tools here, or better metrics.

Anyway, that’s the situation.  If you have ideas or if you’re in a
profiling & optimization mood, now’s the time to unleash your
creativity.  :-)

Ludo’.


^ permalink raw reply	[flat|nested] 3+ messages in thread

* Rust packages and performance impact on Guix
  2021-02-19 16:01 Will 2021 be the year of build systems on gexps? Ludovic Courtès
@ 2021-02-23 10:31 ` Ludovic Courtès
  2021-02-23 14:28 ` Will 2021 be the year of build systems on gexps? Ludovic Courtès
  1 sibling, 0 replies; 3+ messages in thread
From: Ludovic Courtès @ 2021-02-23 10:31 UTC (permalink / raw)
  To: Guix-devel

Hi!

Ludovic Courtès <ludo@gnu.org> skribis:

> but on ‘wip-build-systems-gexp’ I get:
>
> $ GUIX_PROFILING=gc time ./pre-inst-env guix build qemu -d --no-grafts
> /gnu/store/5n44l8cmrmkr747nsqbxpm4764jdsl3l-qemu-5.1.0.drv
> Garbage collection statistics:
>   heap size:        80.52 MiB
>   allocated:        249.89 MiB
>   GC times:         16
>   time spent in GC: 0.75 seconds (34% of user time)
> 2.22user 0.13system 0:02.06elapsed 114%CPU (0avgtext+0avgdata 243532maxresident)k
> 0inputs+0outputs (0major+34304minor)pagefaults 0swaps
>
> In short, 30% more garbage allocated and an 8% slowdown.

In the meantime, librsvg on core-updates switched to the version that
depends on Rust.  As a result, the above command, on core-updates, went
to ~2.25s on my laptop (+10–20%!).

Would be nice to investigate.  One issue is that ‘cargo-build-system’
traverses packages.  The ‘crate-closure’ call in there certainly adds a
fair bit of overhead.  Could we avoid it?

Thanks,
Ludo’.


^ permalink raw reply	[flat|nested] 3+ messages in thread

* Re: Will 2021 be the year of build systems on gexps?
  2021-02-19 16:01 Will 2021 be the year of build systems on gexps? Ludovic Courtès
  2021-02-23 10:31 ` Rust packages and performance impact on Guix Ludovic Courtès
@ 2021-02-23 14:28 ` Ludovic Courtès
  1 sibling, 0 replies; 3+ messages in thread
From: Ludovic Courtès @ 2021-02-23 14:28 UTC (permalink / raw)
  To: Guix-devel

Hi!

Ludovic Courtès <ludo@gnu.org> skribis:

> I’ve made progress identifying and mitigating what seems to be the main
> cause of this (the fact that it’s possible to ungexp a list and that
> list will be scanned in its entirety in search of file-like
> objects—convenient but expensive) but as you can see, there’s still a
> lot to do.

I cherry-picked to ‘master’ the gexp optimizations I made on
‘wip-build-systems-gexp’.  The good news is that it pays off on system
compilations, going from (commit 3c548c3e0eade12d3e86b1201dbd95863ca64ea7):

--8<---------------cut here---------------start------------->8---
$ GUIX_PROFILING="gc object-cache" ./pre-inst-env guix system build -d --no-grafts gnu/system/examples/desktop.tmpl
/gnu/store/g6kd2blajq6yja80516m85zv07d4ysb3-system.drv
Store object cache:
  fresh caches:     2
  lookups:      119282
  hits:         116225 (97.4%)
Garbage collection statistics:
  heap size:        119.43 MiB
  allocated:        649.07 MiB
  GC times:         26
  time spent in GC: 2.15 seconds (37% of user time)
--8<---------------cut here---------------end--------------->8---

to (commit 0a912b3a434c9ff9515f9036feae4d3a7075fb2f):

--8<---------------cut here---------------start------------->8---
$ GUIX_PROFILING="gc object-cache" ./pre-inst-env guix system build -d --no-grafts gnu/system/examples/desktop.tmpl
/gnu/store/f63c8rk8rwrfr5cl4s74hnx25l3z1hrp-system.drv
Store object cache:
  fresh caches:     2
  lookups:      119285
  hits:         116227 (97.4%)
  cache size:    3058 entries
Garbage collection statistics:
  heap size:        111.43 MiB
  allocated:        608.01 MiB
  GC times:         25
  time spent in GC: 2.00 seconds (37% of user time)
--8<---------------cut here---------------end--------------->8---

Reduced heap allocations (-6%), reduce heap size (-7%), and reduced run
time (ditto).  Getting better!

Ludo’.


^ permalink raw reply	[flat|nested] 3+ messages in thread

end of thread, other threads:[~2021-02-23 14:32 UTC | newest]

Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-02-19 16:01 Will 2021 be the year of build systems on gexps? Ludovic Courtès
2021-02-23 10:31 ` Rust packages and performance impact on Guix Ludovic Courtès
2021-02-23 14:28 ` Will 2021 be the year of build systems on gexps? Ludovic Courtès

unofficial mirror of guix-devel@gnu.org 

This inbox may be cloned and mirrored by anyone:

	git clone --mirror https://yhetil.org/guix-devel/0 guix-devel/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 guix-devel guix-devel/ https://yhetil.org/guix-devel \
		guix-devel@gnu.org
	public-inbox-index guix-devel

Example config snippet for mirrors.
Newsgroups are available over NNTP:
	nntp://news.yhetil.org/yhetil.gnu.guix.devel
	nntp://news.gmane.io/gmane.comp.gnu.guix.devel


AGPL code for this site: git clone http://ou63pmih66umazou.onion/public-inbox.git