bug#63852: ‘guix-package-cache.drv’ eats all your memory with ‘guix-cran’

["Ludovic Courtès" <ludo@gnu.org>]

[-- Attachment #1: Type: text/plain, Size: 5306 bytes --]

Ludovic Courtès <ludovic.courtes@inria.fr> skribis:

> It goes OK until building /gnu/store/6rfaqfq693vda59a55asc4wjjg52ilns-guix-package-cache.drv, which
> consumes memory until it gets OOM-killed.

The culprit is easily found (here on just the 22K packages of ‘guix’, so
less than half of what you get with ‘guix-cran’):

--8<---------------cut here---------------start------------->8---
$ guix repl
GNU Guile 3.0.9
Copyright (C) 1995-2023 Free Software Foundation, Inc.

Guile comes with ABSOLUTELY NO WARRANTY; for details type `,show w'.
This program is free software, and you are welcome to redistribute it
under certain conditions; type `,show c' for details.

Enter `,help' for help.
scheme@(guix-user)> ,m(gnu packages)
scheme@(gnu packages)> ,use(statprof)
scheme@(gnu packages)> (gcprof (lambda () (generate-package-cache "/tmp/t")))
%     cumulative   self             
time   seconds     seconds  procedure
 50.00     13.77     13.77  ice-9/boot-9.scm:2498:4
  6.94      1.91      1.91  display
  5.56      1.53      1.53  object->string
  2.78      0.76      0.76  reverse
  2.78      0.76      0.76  gensym
  2.78      0.76      0.76  ice-9/boot-9.scm:2217:0:%load-announce
  1.39      6.88      0.38  gnu/packages.scm:388:4:expand-cache
  1.39      0.76      0.38  srfi/srfi-1.scm:1028:0:lset-intersection
  1.39      0.38      0.38  number->string
  1.39      0.38      0.38  gnu/packages/crates-io.scm:17783:2:arguments
  1.39      0.38      0.38  open-output-string
  1.39      0.38      0.38  gnu/packages/crates-io.scm:18450:2:arguments
  1.39      0.38      0.38  gnu/packages/crates-io.scm:36208:2:arguments
  1.39      0.38      0.38  guix/packages.scm:1320:8:mproc
  1.39      0.38      0.38  gnu/packages/crates-io.scm:25101:2:arguments
  1.39      0.38      0.38  gnu/packages/crates-io.scm:29972:2:arguments
  1.39      0.38      0.38  gnu/packages/crates-io.scm:883:2:arguments
  1.39      0.38      0.38  ice-9/vlist.scm:449:0:vhash-cons
  1.39      0.38      0.38  append
  1.39      0.38      0.38  string-append
  1.39      0.38      0.38  system/vm/assembler.scm:2175:8
  1.39      0.38      0.38  guix/build/syscalls.scm:1168:0:read-dirent-header/linux
  1.39      0.38      0.38  srfi/srfi-1.scm:1033:17
  1.39      0.38      0.38  gnu/packages/crates-io.scm:9336:2:arguments
  1.39      0.38      0.38  gnu/packages/crates-io.scm:64009:2:arguments
  1.39      0.38      0.38  guix/build-system/cargo.scm:229:4
  1.39      0.38      0.38  ice-9/boot-9.scm:2759:0:module-make-local-var!
  0.00   7328.64      0.00  system/vm/assembler.scm:1258:0:intern-constant
  0.00   1423.97      0.00  ice-9/boot-9.scm:220:5:map1
  0.00    201.90      0.00  ice-9/threads.scm:388:4

[...]

---
Sample count: 72
Total time: 27.531145665 seconds (14.056589389 seconds in GC)
--8<---------------cut here---------------end--------------->8---

The assembler allocates a huge constant table in ‘intern-constant’
(‘generate-package-cache’ produces one literal list containing one small
vector per package, each of which contains strings and other constants).

The attached file emulates what ‘generate-package-cache’ does.  Each
vector in the list looks like this:

  #(0 "xyzxyz-0" "1.0.0" (gnu packages xyz0) xyzxyz-0 ("out") #t #f "gnu/packages/xyz0" 0 1)

The program terminates with roughly a 250 MiB heap for 22K entries—about
10 MiB per vector.

The heap profile on completion looks like this:

--8<---------------cut here---------------start------------->8---
  %   type                               self    avg obj size
 57.5 bytevector                          8,205,632 455868.4
 23.9 vector                              3,413,857   549.3
  7.7 struct                              1,093,840    40.8
  5.3 pair                                  753,376    16.0
  2.7 stringbuf                             385,856    51.1
  1.1 symbol                                151,424    32.0
  0.4 program                                59,424    47.3
  0.3 string                                 49,152    32.0
  0.2 heap-number                            23,520    32.0
  0.2 variable                               21,696    30.3
  0.1 smob                                   21,056    32.0
  0.1 vm-continuation                        20,864    32.0
  0.1 atomic-box                             20,736    32.0
  0.1 weak-table                             19,584    31.8
  0.1 unknown                                19,040    32.0
  0.0 pointer                                 2,000    16.0
  0.0 hash-table                                896    32.0
  0.0 primitive                                 144    16.0
  0.0 weak-vector                                96    16.0
  0.0 frame                                      96    32.0
  0.0 primitive-generic                          64    32.0
  0.0 keyword                                    16    16.0
sampled heap: 13.60166 MiB (heap size: 233.16016 MiB)
--8<---------------cut here---------------end--------------->8---

It’s likely that the ‘buf’ field of <asm> is what’s taking space

On that topic, see also:

  https://lists.gnu.org/archive/html/guile-devel/2023-01/msg00013.html

Ludo’.


[-- Attachment #2: the reproducer --]
[-- Type: text/plain, Size: 1673 bytes --]

;; https://issues.guix.gnu.org/63852

(use-modules (system base compile)
             (language tree-il)
             (srfi srfi-1)
             (srfi srfi-26)
             (ice-9 match)
             (ice-9 time)
             (statprof))

(define lst
  (unfold (cut > <> 22000)
          (lambda (i)
            ;; Mimic 'generate-package-cache'.
            (define name
              (string-append "xyzxyz-" (number->string i 8)))
            (vector i name
                    (string-append "1.0."
                                   (number->string (modulo i 300)))
                    (list 'gnu 'packages
                          (string->symbol
                           (string-append "xyz" (number->string i))))
                    (string->symbol name)
                    '("out")
                    #t
                    #f
                    (string-append "gnu/packages/xyz"
                                   (number->string (modulo i 300)))
                    (modulo i 1000)
                    1))
          1+
          0))

(pk 'go! (gc-stats))
(let ((exp (match (or (getenv "VARIANT") "0")
             ("0" `',lst)
             ("1" `(list ,@lst))
             ("2" (make-const #f lst))            ;avoid psyntax overhead
             ("3" (make-primcall #f 'list
                                 (map (lambda (vector)
                                        (make-const #f vector))
                                      lst))))))
  (compile exp #:to 'bytecode
           #:from (if (pair? exp)
                      'scheme
                      'tree-il)
           #:optimization-level 0
           #:opts '(#:to-file? #t)))
(pk (gc-stats))