bug#27284: Memory leak in 'guix pull' or 'make' in guix source

[ludo@gnu.org (Ludovic Courtès)]

Howdy,

Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:

> ludo@gnu.org (Ludovic Courtès) writes:

[...]

>> This has been discussed informally many times, and here’s a summary of
>> the ideas I’m aware of:
>>
>>   1. Build Guix as separate derivations: the first derivation builds the
>
> [...]
>
>>
>>   2. Build all of Guix like the ‘guix’ package does, and hope that we
>>      can get a substitute.
>
> This seems like the best option to me. It strikes me as odd that Guix
> isn't distributed as its own substitutable package that can be
> efficiently downloaded and unpacked to the store, rather than the
> current intensive build-it-all process.

Yeah.

>>      Bootstrapping issue: to do that, we first need compute the
>>      derivation of this new ‘guix’ package.  Thus, we at least need to
>>      build the closure of (guix packages), which should take a minute or
>>      so, after which we can compute the derivation, which could take a
>>      couple of minutes maybe.
>>      The problem is that building all of Guix (including running the
>>      test suite) takes some time, potentially more than the interval
>>      between two subsequent pushes to the repo.  Thus, it’s quite likely
>>      that the build farm would always be lagging behind.
>
> Do we really have to build it *all*, every time? There are two
> optimizations which I can think of:
>
> 1. Only rebuild the files that changed.

That’s more or less what #1 would give us, though not necessarily at the
finest granularity.

> 2. Remove the --no-auto-compile arg from (guix scripts guix).
>
> I'm not sure why this is desirable. Is it because otherwise the cached
> compiled copies would be susceptible to ABI breakages?

Yes, that’s one thing.

Another thing is that on the first run of ‘guix package’ following ‘guix
pull’, you’d end up compiling everything, which is not much better than
compiling them upfront.

There’s also the option of not compiling (gnu packages *) and instead
evaluating them, but currently this is too costly in terms of memory and
CPU.

> Even with such optimizations it could still take more time to build Guix
> than the rate at which commits hit master. Maybe a cron job that could
> build the latest Guix every, say, 5 or 10 minutes (only when new commits
> appeared since the last time it was built)? We would then distribute the
> latest built Guix package when users would issue a 'guix pull'. There
> could be a 'guix pull' option to force building the package from the
> latest sources, for those who prefer the current behavior.

Yes, what I was suggesting was to build Guix at each push, and
automatically tag successful builds, such that ‘guix pull’ can pull up
to the latest successfully-built tag.  (Though we’d need to be careful:
if the auto-tagging machinery stops working, we still want users to get
the latest Guix so they have security updates.)

Thanks for chiming in!

Ludo’.