bug#27284: Memory leak in 'guix pull' or 'make' in guix source

[Maxim Cournoyer <maxim.cournoyer@gmail.com>]

ludo@gnu.org (Ludovic Courtès) writes:

> Hello Guix!
>
> A heads-up to share the cognitive burden related to this topic.  :-)
>
> So, we have two problems: compilation time, and memory consumption.  I
> *think* I’ve identified one of the major causes for both in Guile,
> though it’s too early to say exactly how much this will impact resource
> consumption for a full Guix compilation.  See
> <https://lists.gnu.org/archive/html/guile-devel/2017-09/msg00031.html>
> for details.
>
> When that is fixed, we’ll still have a performance problem: building all
> of Guix will still take more time than we’d like, and it won’t get
> better as we add new files.  So we need to address this.
>
> This has been discussed informally many times, and here’s a summary of
> the ideas I’m aware of:
>
>   1. Build Guix as separate derivations: the first derivation builds the

[...]

>
>   2. Build all of Guix like the ‘guix’ package does, and hope that we
>      can get a substitute.

This seems like the best option to me. It strikes me as odd that Guix
isn't distributed as its own substitutable package that can be
efficiently downloaded and unpacked to the store, rather than the
current intensive build-it-all process.

>      Bootstrapping issue: to do that, we first need compute the
>      derivation of this new ‘guix’ package.  Thus, we at least need to
>      build the closure of (guix packages), which should take a minute or
>      so, after which we can compute the derivation, which could take a
>      couple of minutes maybe.
>      The problem is that building all of Guix (including running the
>      test suite) takes some time, potentially more than the interval
>      between two subsequent pushes to the repo.  Thus, it’s quite likely
>      that the build farm would always be lagging behind.

Do we really have to build it *all*, every time? There are two
optimizations which I can think of:

1. Only rebuild the files that changed.

I guess the challenge here is for the build to be deterministic. It
would be important to be able to detect when any of the inputs part of
the guix closure changed; when such a thing occurs we'd have to fully
rebuild Guix. Otherwise, for a constant closure, it should be safe to
simply 'git pull' and run 'make' to rebuild only the modified files,
which could speed up things a bit. A smarter Guile that would only
partially recompile the changed/referenced parts of massive files
instead of completely (think python.scm) would also improve things a lot
(this would also be a must when working on packages directly in the
tree! Updating the hash of a Python package shouldn't require minutes of
compilation).

2. Remove the --no-auto-compile arg from (guix scripts guix).

I'm not sure why this is desirable. Is it because otherwise the cached
compiled copies would be susceptible to ABI breakages? This could be
catch and handled by the build script, in which case it would retry with
'--fresh-auto-compile'. IIUC, these cached files would never end in the the
package to be distributed anyway, but having them would greatly speed up
computing the closure of guix.

Even with such optimizations it could still take more time to build Guix
than the rate at which commits hit master. Maybe a cron job that could
build the latest Guix every, say, 5 or 10 minutes (only when new commits
appeared since the last time it was built)? We would then distribute the
latest built Guix package when users would issue a 'guix pull'. There
could be a 'guix pull' option to force building the package from the
latest sources, for those who prefer the current behavior.

I think most users would happily trade having to entirely compile Guix
from scratch everytime they use 'guix pull' with being behind 5 minutes
worth of commits from master's tip!

Maxim