Will 2021 be the year of build systems on gexps?

Will 2021 be the year of build systems on gexps?

[Ludovic Courtès]

Hello Guix!

Over the last few days I’ve been head-down working on
‘wip-build-systems-gexp’, the mythical branch that brings gexps to build
systems and packages, so we can say goodbye to
‘build-expression->derivation’.  And… it’s quite a ride!

I rebased it on ‘core-updates’, which was a bit tedious (it hadn’t been
touched in 3 years or so).  There are still test failures and build
systems not yet converted, but that’s the easy part, although both are
time-consuming.

The more difficult part is performance.  On current ‘core-updates’ I get:

--8<---------------cut here---------------start------------->8---
$ GUIX_PROFILING=gc time ./pre-inst-env guix build qemu -d --no-grafts
/gnu/store/z27l6plrxr5wm7818xhj9mdll99jcqz3-qemu-5.1.0.drv
Garbage collection statistics:
  heap size:        80.52 MiB
  allocated:        191.14 MiB
  GC times:         15
  time spent in GC: 0.66 seconds (32% of user time)
2.05user 0.12system 0:01.93elapsed 112%CPU (0avgtext+0avgdata 242872maxresident)k
0inputs+0outputs (0major+34228minor)pagefaults 0swaps
--8<---------------cut here---------------end--------------->8---

but on ‘wip-build-systems-gexp’ I get:

--8<---------------cut here---------------start------------->8---
$ GUIX_PROFILING=gc time ./pre-inst-env guix build qemu -d --no-grafts
/gnu/store/5n44l8cmrmkr747nsqbxpm4764jdsl3l-qemu-5.1.0.drv
Garbage collection statistics:
  heap size:        80.52 MiB
  allocated:        249.89 MiB
  GC times:         16
  time spent in GC: 0.75 seconds (34% of user time)
2.22user 0.13system 0:02.06elapsed 114%CPU (0avgtext+0avgdata 243532maxresident)k
0inputs+0outputs (0major+34304minor)pagefaults 0swaps
--8<---------------cut here---------------end--------------->8---

In short, 30% more garbage allocated and an 8% slowdown.

I’ve made progress identifying and mitigating what seems to be the main
cause of this (the fact that it’s possible to ungexp a list and that
list will be scanned in its entirety in search of file-like
objects—convenient but expensive) but as you can see, there’s still a
lot to do.

Statprof and gcprof are not super helpful here; I feel a need for better
tools here, or better metrics.

Anyway, that’s the situation.  If you have ideas or if you’re in a
profiling & optimization mood, now’s the time to unleash your
creativity.  :-)

Ludo’.

Rust packages and performance impact on Guix

[Ludovic Courtès]

Hi!

Ludovic Courtès <ludo@gnu.org> skribis:

> but on ‘wip-build-systems-gexp’ I get:
>
> $ GUIX_PROFILING=gc time ./pre-inst-env guix build qemu -d --no-grafts
> /gnu/store/5n44l8cmrmkr747nsqbxpm4764jdsl3l-qemu-5.1.0.drv
> Garbage collection statistics:
>   heap size:        80.52 MiB
>   allocated:        249.89 MiB
>   GC times:         16
>   time spent in GC: 0.75 seconds (34% of user time)
> 2.22user 0.13system 0:02.06elapsed 114%CPU (0avgtext+0avgdata 243532maxresident)k
> 0inputs+0outputs (0major+34304minor)pagefaults 0swaps
>
> In short, 30% more garbage allocated and an 8% slowdown.

In the meantime, librsvg on core-updates switched to the version that
depends on Rust.  As a result, the above command, on core-updates, went
to ~2.25s on my laptop (+10–20%!).

Would be nice to investigate.  One issue is that ‘cargo-build-system’
traverses packages.  The ‘crate-closure’ call in there certainly adds a
fair bit of overhead.  Could we avoid it?

Thanks,
Ludo’.

Re: Will 2021 be the year of build systems on gexps?

[Ludovic Courtès]

Hi!

Ludovic Courtès <ludo@gnu.org> skribis:

> I’ve made progress identifying and mitigating what seems to be the main
> cause of this (the fact that it’s possible to ungexp a list and that
> list will be scanned in its entirety in search of file-like
> objects—convenient but expensive) but as you can see, there’s still a
> lot to do.

I cherry-picked to ‘master’ the gexp optimizations I made on
‘wip-build-systems-gexp’.  The good news is that it pays off on system
compilations, going from (commit 3c548c3e0eade12d3e86b1201dbd95863ca64ea7):

--8<---------------cut here---------------start------------->8---
$ GUIX_PROFILING="gc object-cache" ./pre-inst-env guix system build -d --no-grafts gnu/system/examples/desktop.tmpl
/gnu/store/g6kd2blajq6yja80516m85zv07d4ysb3-system.drv
Store object cache:
  fresh caches:     2
  lookups:      119282
  hits:         116225 (97.4%)
Garbage collection statistics:
  heap size:        119.43 MiB
  allocated:        649.07 MiB
  GC times:         26
  time spent in GC: 2.15 seconds (37% of user time)
--8<---------------cut here---------------end--------------->8---

to (commit 0a912b3a434c9ff9515f9036feae4d3a7075fb2f):

--8<---------------cut here---------------start------------->8---
$ GUIX_PROFILING="gc object-cache" ./pre-inst-env guix system build -d --no-grafts gnu/system/examples/desktop.tmpl
/gnu/store/f63c8rk8rwrfr5cl4s74hnx25l3z1hrp-system.drv
Store object cache:
  fresh caches:     2
  lookups:      119285
  hits:         116227 (97.4%)
  cache size:    3058 entries
Garbage collection statistics:
  heap size:        111.43 MiB
  allocated:        608.01 MiB
  GC times:         25
  time spent in GC: 2.00 seconds (37% of user time)
--8<---------------cut here---------------end--------------->8---

Reduced heap allocations (-6%), reduce heap size (-7%), and reduced run
time (ditto).  Getting better!

Ludo’.

Re: Will 2021 be the year of build systems on gexps?

[Ludovic Courtès]

Hello!

Ludovic Courtès <ludo@gnu.org> skribis:

> Over the last few days I’ve been head-down working on
> ‘wip-build-systems-gexp’, the mythical branch that brings gexps to build
> systems and packages, so we can say goodbye to
> ‘build-expression->derivation’.  And… it’s quite a ride!

The current tip of ‘wip-build-systems-gexp’ Just Works; it’s being built,
it can build ‘guix’ and cross-build things like ‘sed’:

  https://data.guix-patches.cbaines.net/repository/2/branch/wip-build-systems-gexp

  https://ci.guix.gnu.org/jobset/wip-build-systems-gexp (though Cuirass
  currently has unrelated problems)

In terms of performance, there’s still a ~10% slowdown when computing
derivations compared to the ‘core-updates’ revision the branch is based
on.

Here’s what I’d like to do in the coming days, if that doesn’t interfere
with what others have in mind for the upcoming release:

  • Monitor build failures due to typos/thinkos made while adjusting
    build systems;

  • Merge on ‘core-updates’.

Then there are optimizations to work on, but that can take a bit longer.
In particular, in ‘gexp->derivation’, allow file-like objects to be
specified as environment variable values.  In turn, use that so that,
say, ‘gnu-build-system’ has a single builder for all its packages and
just calls ‘getenv’ to get the value of its various parameters, similar
to what (guix git-download) does.

That said, if people think it’s too late in the cycle, we could just as
well keep it for the next cycle.

Thoughts?

Thanks,
Ludo’.

Re: Will 2021 be the year of build systems on gexps?

[Julien Lepiller]

[-- Attachment #1: Type: text/plain, Size: 2092 bytes --]

If we aim at merging c-u before next release, we should probably wait next cycle, as this introduces quite a lot of changes, and packages might break in subtle ways.

10% increase for computing derivations is not great :/ it already takes a long time to do that on my arm system ^^". I wonder how we could improve that?

Le 10 mars 2021 06:09:23 GMT-05:00, "Ludovic Courtès" <ludo@gnu.org> a écrit :
>Hello!
>
>Ludovic Courtès <ludo@gnu.org> skribis:
>
>> Over the last few days I’ve been head-down working on
>> ‘wip-build-systems-gexp’, the mythical branch that brings gexps to
>build
>> systems and packages, so we can say goodbye to
>> ‘build-expression->derivation’.  And… it’s quite a ride!
>
>The current tip of ‘wip-build-systems-gexp’ Just Works; it’s being
>built,
>it can build ‘guix’ and cross-build things like ‘sed’:
>
>https://data.guix-patches.cbaines.net/repository/2/branch/wip-build-systems-gexp
>
>  https://ci.guix.gnu.org/jobset/wip-build-systems-gexp (though Cuirass
>  currently has unrelated problems)
>
>In terms of performance, there’s still a ~10% slowdown when computing
>derivations compared to the ‘core-updates’ revision the branch is based
>on.
>
>Here’s what I’d like to do in the coming days, if that doesn’t
>interfere
>with what others have in mind for the upcoming release:
>
>  • Monitor build failures due to typos/thinkos made while adjusting
>    build systems;
>
>  • Merge on ‘core-updates’.
>
>Then there are optimizations to work on, but that can take a bit
>longer.
>In particular, in ‘gexp->derivation’, allow file-like objects to be
>specified as environment variable values.  In turn, use that so that,
>say, ‘gnu-build-system’ has a single builder for all its packages and
>just calls ‘getenv’ to get the value of its various parameters, similar
>to what (guix git-download) does.
>
>That said, if people think it’s too late in the cycle, we could just as
>well keep it for the next cycle.
>
>Thoughts?
>
>Thanks,
>Ludo’.

[-- Attachment #2: Type: text/html, Size: 2683 bytes --]

Re: Will 2021 be the year of build systems on gexps?

[zimoun]

Hi Ludo,

On Wed, 10 Mar 2021 at 12:09, Ludovic Courtès <ludo@gnu.org> wrote:

> The current tip of ‘wip-build-systems-gexp’ Just Works; it’s being built,
> it can build ‘guix’ and cross-build things like ‘sed’:
>
>   https://data.guix-patches.cbaines.net/repository/2/branch/wip-build-systems-gexp
>
>   https://ci.guix.gnu.org/jobset/wip-build-systems-gexp (though Cuirass
>   currently has unrelated problems)

Cool!

> In terms of performance, there’s still a ~10% slowdown when computing
> derivations compared to the ‘core-updates’ revision the branch is based
> on.

Less cool! :-)


> Here’s what I’d like to do in the coming days, if that doesn’t interfere
> with what others have in mind for the upcoming release:
>
>   • Monitor build failures due to typos/thinkos made while adjusting
>     build systems;
>
>   • Merge on ‘core-updates’.

[...]

> That said, if people think it’s too late in the cycle, we could just as
> well keep it for the next cycle.

I do not think it interferes with the release since for now and except a
big change, the plan is to release without the core-updates merge.
Well, that’s my understanding of the previous discussion.


Cheers,
simon

Re: Will 2021 be the year of build systems on gexps?

[Leo Famulari]

On Wed, Mar 10, 2021 at 03:12:42PM +0100, zimoun wrote:
> I do not think it interferes with the release since for now and except a
> big change, the plan is to release without the core-updates merge.
> Well, that’s my understanding of the previous discussion.

That's my understand as well.

Re: Will 2021 be the year of build systems on gexps?

[Ludovic Courtès]

Hello!

Ludovic Courtès <ludo@gnu.org> skribis:

> Ludovic Courtès <ludo@gnu.org> skribis:
>
>> Over the last few days I’ve been head-down working on
>> ‘wip-build-systems-gexp’, the mythical branch that brings gexps to build
>> systems and packages, so we can say goodbye to
>> ‘build-expression->derivation’.  And… it’s quite a ride!
>
> The current tip of ‘wip-build-systems-gexp’ Just Works; it’s being built,
> it can build ‘guix’ and cross-build things like ‘sed’:
>
>   https://data.guix-patches.cbaines.net/repository/2/branch/wip-build-systems-gexp
>
>   https://ci.guix.gnu.org/jobset/wip-build-systems-gexp (though Cuirass
>   currently has unrelated problems)

It’s building and well!

> In terms of performance, there’s still a ~10% slowdown when computing
> derivations compared to the ‘core-updates’ revision the branch is based
> on.

I made some improvements yesterday (reducing object cache lookups and
the number of entries therein), but we’re still in the 10% ballpark.
WIP branch:

--8<---------------cut here---------------start------------->8---
$ git log |head -5
commit 082df93be3472e0f38970634260af8c432420b35
Author: Ludovic Courtès <ludo@gnu.org>
Date:   Mon Mar 8 13:59:23 2021 +0100

    gnu: docbook-xsl: Move 'use-modules' form to the top level.
$ time GUIX_PROFILING=gc ./pre-inst-env guix build libreoffice --no-grafts -d
/gnu/store/fsrbbi8vfrwwdz2dlyzpfvvnky03nczz-libreoffice-6.4.7.2.drv
Garbage collection statistics:
  heap size:        87.18 MiB
  allocated:        254.25 MiB
  GC times:         16
  time spent in GC: 0.74 seconds (31% of user time)

real	0m2.225s
user	0m2.415s
sys	0m0.087s
--8<---------------cut here---------------end--------------->8---

Compared to ‘core-updates’:

--8<---------------cut here---------------start------------->8---
$ git log |head -5
commit b35581bd63d929e83d18f42b067f63efc867353c
Author: Efraim Flashner <efraim@flashner.co.il>
Date:   Sun Mar 21 09:42:06 2021 +0200

    gnu: openjpeg: Update to 2.4.0.
$ time GUIX_PROFILING=gc ./pre-inst-env guix build libreoffice --no-grafts -d
/gnu/store/irdhm6jx30bgdxvgb0an1mn223rzshkg-libreoffice-6.4.7.2.drv
Garbage collection statistics:
  heap size:        79.18 MiB
  allocated:        216.51 MiB
  GC times:         16
  time spent in GC: 0.74 seconds (33% of user time)

real	0m2.094s
user	0m2.277s
sys	0m0.106s
--8<---------------cut here---------------end--------------->8---

> Here’s what I’d like to do in the coming days, if that doesn’t interfere
> with what others have in mind for the upcoming release:
>
>   • Monitor build failures due to typos/thinkos made while adjusting
>     build systems;
>
>   • Merge on ‘core-updates’.

I’ll go ahead with that if there are no objections.

> Then there are optimizations to work on, but that can take a bit longer.
> In particular, in ‘gexp->derivation’, allow file-like objects to be
> specified as environment variable values.  In turn, use that so that,
> say, ‘gnu-build-system’ has a single builder for all its packages and
> just calls ‘getenv’ to get the value of its various parameters, similar
> to what (guix git-download) does.

I’m also starting work in this area.

Ludo’.

Re: Will 2021 be the year of build systems on gexps?

[Ricardo Wurmus]

Ludovic Courtès <ludo@gnu.org> writes:

> Hello!
>
> Ludovic Courtès <ludo@gnu.org> skribis:
>
>> Ludovic Courtès <ludo@gnu.org> skribis:
>>
>>> Over the last few days I’ve been head-down working on
>>> ‘wip-build-systems-gexp’, the mythical branch that brings gexps to build
>>> systems and packages, so we can say goodbye to
>>> ‘build-expression->derivation’.  And… it’s quite a ride!
>>
>> The current tip of ‘wip-build-systems-gexp’ Just Works; it’s being built,
>> it can build ‘guix’ and cross-build things like ‘sed’:
>>
>>   https://data.guix-patches.cbaines.net/repository/2/branch/wip-build-systems-gexp
>>
>>   https://ci.guix.gnu.org/jobset/wip-build-systems-gexp (though Cuirass
>>   currently has unrelated problems)
>
> It’s building and well!

Excellent!  It’s great to see that you brought this idea across all
the hurdles you encountered in previous attempts.  Inspiring!

>> Here’s what I’d like to do in the coming days, if that doesn’t interfere
>> with what others have in mind for the upcoming release:
>>
>>   • Monitor build failures due to typos/thinkos made while adjusting
>>     build systems;
>>
>>   • Merge on ‘core-updates’.
>
> I’ll go ahead with that if there are no objections.

This sounds reasonable to me.  “core-updates” will need some work to
push it into shape for the next merge, but I don’t think merging a
well-monitored branch into “core-updates” would make the job
considerably harder.

Thank you!

-- 
Ricardo

Re: Will 2021 be the year of build systems on gexps?

[Maxim Cournoyer]

Hi Ludo!

Ludovic Courtès <ludo@gnu.org> writes:

> Hello!
>
> Ludovic Courtès <ludo@gnu.org> skribis:
>
>> Ludovic Courtès <ludo@gnu.org> skribis:
>>
>>> Over the last few days I’ve been head-down working on
>>> ‘wip-build-systems-gexp’, the mythical branch that brings gexps to build
>>> systems and packages, so we can say goodbye to
>>> ‘build-expression->derivation’.  And… it’s quite a ride!
>>
>> The current tip of ‘wip-build-systems-gexp’ Just Works; it’s being built,
>> it can build ‘guix’ and cross-build things like ‘sed’:
>>
>>   https://data.guix-patches.cbaines.net/repository/2/branch/wip-build-systems-gexp
>>
>>   https://ci.guix.gnu.org/jobset/wip-build-systems-gexp (though Cuirass
>>   currently has unrelated problems)
>
> It’s building and well!
>
>> In terms of performance, there’s still a ~10% slowdown when computing
>> derivations compared to the ‘core-updates’ revision the branch is based
>> on.
>
> I made some improvements yesterday (reducing object cache lookups and
> the number of entries therein), but we’re still in the 10% ballpark.
> WIP branch:
>
> $ git log |head -5
> commit 082df93be3472e0f38970634260af8c432420b35
> Author: Ludovic Courtès <ludo@gnu.org>
> Date:   Mon Mar 8 13:59:23 2021 +0100
>
>     gnu: docbook-xsl: Move 'use-modules' form to the top level.
> $ time GUIX_PROFILING=gc ./pre-inst-env guix build libreoffice --no-grafts -d
> /gnu/store/fsrbbi8vfrwwdz2dlyzpfvvnky03nczz-libreoffice-6.4.7.2.drv
> Garbage collection statistics:
>   heap size:        87.18 MiB
>   allocated:        254.25 MiB
>   GC times:         16
>   time spent in GC: 0.74 seconds (31% of user time)
>
> real	0m2.225s
> user	0m2.415s
> sys	0m0.087s
>
>
> Compared to ‘core-updates’:
>
> $ git log |head -5
> commit b35581bd63d929e83d18f42b067f63efc867353c
> Author: Efraim Flashner <efraim@flashner.co.il>
> Date:   Sun Mar 21 09:42:06 2021 +0200
>
>     gnu: openjpeg: Update to 2.4.0.
> $ time GUIX_PROFILING=gc ./pre-inst-env guix build libreoffice --no-grafts -d
> /gnu/store/irdhm6jx30bgdxvgb0an1mn223rzshkg-libreoffice-6.4.7.2.drv
> Garbage collection statistics:
>   heap size:        79.18 MiB
>   allocated:        216.51 MiB
>   GC times:         16
>   time spent in GC: 0.74 seconds (33% of user time)
>
> real	0m2.094s
> user	0m2.277s
> sys	0m0.106s

This looks reasonable!

>> Here’s what I’d like to do in the coming days, if that doesn’t interfere
>> with what others have in mind for the upcoming release:
>>
>>   • Monitor build failures due to typos/thinkos made while adjusting
>>     build systems;
>>
>>   • Merge on ‘core-updates’.
>
> I’ll go ahead with that if there are no objections.

Sounds good!  Thanks for picking up this work! :-)

Maxim

Re: Will 2021 be the year of build systems on gexps?

[Ludovic Courtès]

Hi Maxim and all,

So this is it, 2021 *is* the year of build systems on gexps!

I’ve rebased it and pushed on ‘core-updates’ with
2eafeb2f3d661061bc412c3f27c90202e4532532 as the tip.

The build farm has a bit of work to do now it seems.  Hmm actually the
evaluation at <https://ci.guix.gnu.org/jobset/core-updates> just failed
with a yellow cross, not sure what that means.

Anyway, I’ll be monitoring in the coming days, but do ping me when you
encounter problems!

It’s crazy that it took 6 years (six!) to get past the finish line.  The
initial reason for not merging it was that I was worried about
performance regressions; there’s still work to do in this area, as I
wrote earlier, but it’s looking good.  Another reason is that I was
afraid of unlocking features that would break assumptions we make about
packages—for instance, that all the inputs are specified in the various
inputs fields, and that these fields are all we need to care about when
doing graph rewriting.  (I now think this fear was unwarranted.)

And then, rebasing the branch became harder every time, there were more
and more build systems that needed to be converted… and there were
always more pressing issues to work on.

What’s the lesson here?  I don’t know!  Had it been done six years ago,
it would have had to go through a number of iterations:
‘with-imported-modules’ and ‘with-extensions’ didn’t exist, the template
mechanism in (guix monads) didn’t exist, self-referential records
(‘this-package’, etc.) weren’t there.  It would have worked though, and
perhaps we’d have explored other parts of the design space, who knows.

Ludo’.

Re: Will 2021 be the year of build systems on gexps?

[Timothy Sample]

Hi Ludo,

Ludovic Courtès <ludo@gnu.org> writes:

> So this is it, 2021 *is* the year of build systems on gexps!

Wow!  Congratulations!  I am very happy to see this finally come
together.

> It’s crazy that it took 6 years (six!) to get past the finish line.

I think I’ve only known what a gexp is for three years... and that’s
about how long I’ve been excited by this feature.

Thanks for sticking with it for six years!


-- Tim