bug#27284: Memory leak in 'guix pull' or 'make' in guix source

[ludo@gnu.org (Ludovic Courtès)]

Heya,

ludo@gnu.org (Ludovic Courtès) skribis:

> So, we have two problems: compilation time, and memory consumption.  I
> *think* I’ve identified one of the major causes for both in Guile,
> though it’s too early to say exactly how much this will impact resource
> consumption for a full Guix compilation.  See
> <https://lists.gnu.org/archive/html/guile-devel/2017-09/msg00031.html>
> for details.

The latest news is that (1) the weak-table fix in Guile¹ helps in
general but does not significantly reduce memory consumption when
compiling gnu/packages/python.scm (which requires ~1.5G on x86_64), and
(2) we’ve identified the compiler’s slot allocator as a major cause of
memory consumption and CPU cost when compiling large files like
python.scm:

  https://lists.gnu.org/archive/html/guile-devel/2017-10/msg00035.html

Andy may work on a simpler slot allocator for -O0 and/or large functions
that doesn’t exhibit this pathological behavior.

In the meantime, our best workaround to reduce memory consumption is… to
split large files into smaller ones.  Per M-x guix-locations, the
candidates are:

  gnu/packages/python.scm                                   986
  gnu/packages/perl.scm                                     401
  gnu/packages/haskell.scm                                  348
  gnu/packages/bioinformatics.scm                           274
  gnu/packages/statistics.scm                               222
  gnu/packages/emacs.scm                                    213
  gnu/packages/xorg.scm                                     196
  gnu/packages/web.scm                                      188
  gnu/packages/gnome.scm                                    162
  gnu/packages/ruby.scm                                     159
  gnu/packages/java.scm                                     149
  gnu/packages/ocaml.scm                                    126
  gnu/packages/tex.scm                                      107
  gnu/packages/linux.scm                                    106

I think we could focus on the first two or three files.  FTR, compiling
bioinformatics.scm peaks at ~500 MiB resident on x86_64.

Ricardo, WDYT?

If we do this, do we split arbitrarily?  Like the second half of
python.scm goes to python-cont.scm (provided there are no cross
top-level references)?  Or do you have a better idea?

Ludo’.

¹ https://lists.gnu.org/archive/html/guile-devel/2017-10/msg00051.html