bug#65456: [PATCH 0/2] Split guix build into more steps for 32bit hosts.

["Ludovic Courtès" <ludo@gnu.org>]

Hi!

Janneke Nieuwenhuizen <janneke@gnu.org> skribis:

>>From ad94f06620e53fcc1495a2e2479dfc627177047c Mon Sep 17 00:00:00 2001
> Message-ID: <ad94f06620e53fcc1495a2e2479dfc627177047c.1692783678.git.janneke@gnu.org>
> From: Janneke Nieuwenhuizen <janneke@gnu.org>
> Date: Thu, 22 Jun 2023 08:30:25 +0200
> Subject: [PATCH v4] self: Build directories in chunks of max 25 files at a
>  time.
>
> Similar to split build of make-go in Makefile.am, this breaks-up building
> directories into chunks of max 25 files.  Also force garbage collection.

The big difference with ‘make-go’ is that ‘make-go’ spawns a new process
for each chunk of files: each process starts with an empty heap, which
is not the case here as we reuse the same process.

However, (guix self) is already splitting gnu/packages/*.scm in two
pieces: ‘guix-packages-base’ and ‘guix-packages’.  The former is the
closure of (gnu packages base), and the latter contains the remaining
files.  Unfortunately this is uneven:

--8<---------------cut here---------------start------------->8---
$ readlink -f $(type -P guix)
/gnu/store/12p5axbr4gjrghlrqa4ikmhsxwq2wgw3-guix-command
$ guix gc -R /gnu/store/12p5axbr4gjrghlrqa4ikmhsxwq2wgw3-guix-command|grep packages-base
/gnu/store/ivprgy9b2lv8wmkm10wkypf7k24cdifb-guix-packages-base
/gnu/store/05pjlcfcfa0k9y833nnxxxjcn5mqr8zj-guix-packages-base-source
/gnu/store/gnxjbyfwfmb216krz2x0cf1z5k1lla9x-guix-packages-base-modules
$ find /gnu/store/ivprgy9b2lv8wmkm10wkypf7k24cdifb-guix-packages-base  -type f |wc -l
361
$ guix gc -R /gnu/store/12p5axbr4gjrghlrqa4ikmhsxwq2wgw3-guix-command|grep packages$
/gnu/store/8cda50hsayydrlw0qrhcy8q4dr9f1avx-guix-locale-guix-packages
ludo@ribbon ~/src/guix [env]$ find /gnu/store/8cda50hsayydrlw0qrhcy8q4dr9f1avx-guix-locale-guix-packages | wc -l
64
$ guix describe
Generation 271  Aug 20 2023 23:48:59    (current)
  guix a0f5885
    repository URL: https://git.savannah.gnu.org/git/guix.git
    branch: master
    commit: a0f5885fefd93a3859b6e4b82b18a6db9faeee05
--8<---------------cut here---------------end--------------->8---

Maxime Devos looked into this a while back:

  https://issues.guix.gnu.org/54539

> * guix/self.scm (compiled-modules)[process-directory]: Split building of
> directories into chunks of max 25 files.

[...]

>            (define (process-directory directory files output)
> -            ;; Hide compilation warnings.
> -            (parameterize ((current-warning-port (%make-void-port "w")))
> -              (compile-files directory #$output files
> -                             #:workers (parallel-job-count)
> -                             #:report-load report-load
> -                             #:report-compilation report-compilation)))
> +            (let* ((size 25)                      ;compile max 25 files a time
> +                   (chunks (unfold
> +                            (lambda (seed) (< (length seed) size)) ;p
> +                            (cute take <> size)                    ;f
> +                            (cute drop <> size)                    ;g
> +                            files                                  ;seed
> +                            list)))                                ;tail
> +              (for-each
> +               (lambda (chunck)

s/chunck/chunk/

Can you confirm that this reduces memory usage observably?  One way to
check that would be to print (gc-stats) from ‘process-directory’, with
and without the change.  Could you give it a try?

Intuitively, I don’t see why it would eat less memory; maybe peak memory
usage is lower because we do less at once?

Also, I think we should remove the explicit (gc) call: it should not be
necessary, and if we depend on that, something’s wrong.

Anyhow, thanks for tackling this issue!

Ludo’.