* bug#63852: ‘guix-package-cache.drv’ eats all your memory with ‘guix-cran’
@ 2023-06-02 16:22 Ludovic Courtès
2023-06-03 14:05 ` Ludovic Courtès
0 siblings, 1 reply; 5+ messages in thread
From: Ludovic Courtès @ 2023-06-02 16:22 UTC (permalink / raw)
To: 63852; +Cc: Lars-Dominik Braun
Consider this:
--8<---------------cut here---------------start------------->8---
$ cat /tmp/cran.scm
(cons (channel
(name 'guix-cran)
(url "https://github.com/guix-science/guix-cran.git"))
%default-channels)
$ guix time-machine -C /tmp/cran.scm -- describe
Updating channel 'guix-cran' from Git repository at 'https://github.com/guix-science/guix-cran.git'...
Updating channel 'guix' from Git repository at 'https://git.savannah.gnu.org/git/guix.git'...
[...]
The following derivations will be built:
/gnu/store/90qiphp9048zgvi96lj8zqg8zhwi4js0-profile.drv
/gnu/store/2lv56bl8ghyf2kvfkwj5iy3mrlfik4la-inferior-script.scm.drv
/gnu/store/vnjf8gf6yf5gjm1syvz7yyfn71mmvy27-profile.drv
/gnu/store/8x49757i1s0r6igrah8ramn9zc1v577b-guix-69dfdb7bd.drv
/gnu/store/ll5x27npc3znyqv0rkjh945x9cr41k9q-guix-command.drv
/gnu/store/m03hp1n62bv8gkid53g8igizbw012gai-guix-module-union.drv
/gnu/store/pw8k2jwwyj2nwvqjcd1w3xjvz2n5zpbg-guix-69dfdb7bd-modules.drv
/gnu/store/a6si5zgzsxyha2466v4q45hqimg31mjg-guix-config-modules.drv
/gnu/store/3pm8aivzg3hfv95zlffxjrxvf8hfbg5n-guix-config-source.drv
/gnu/store/rffsvls17s21v2h6gw7agah5vczwi3rm-config.scm.drv
/gnu/store/v61jlksbgfslh27y8dbaj1jcajxz5c8z-guix-config.drv
/gnu/store/z6r6xdcq9dyxlk8ln8cq1ka2gspxp83i-guix-daemon.drv
/gnu/store/fshds3i961v1p3x8f7awh55lix0f7vkl-guix-cran.drv
/gnu/store/xsnp3znb0h630kipjxd198qfd3jr2ww5-inferior-script.scm.drv
building /gnu/store/xsnp3znb0h630kipjxd198qfd3jr2ww5-inferior-script.scm.drv...
building /gnu/store/rffsvls17s21v2h6gw7agah5vczwi3rm-config.scm.drv...
building /gnu/store/3pm8aivzg3hfv95zlffxjrxvf8hfbg5n-guix-config-source.drv...
building /gnu/store/v61jlksbgfslh27y8dbaj1jcajxz5c8z-guix-config.drv...
building /gnu/store/a6si5zgzsxyha2466v4q45hqimg31mjg-guix-config-modules.drv...
building /gnu/store/pw8k2jwwyj2nwvqjcd1w3xjvz2n5zpbg-guix-69dfdb7bd-modules.drv...
building /gnu/store/m03hp1n62bv8gkid53g8igizbw012gai-guix-module-union.drv...
building /gnu/store/ll5x27npc3znyqv0rkjh945x9cr41k9q-guix-command.drv...
building /gnu/store/z6r6xdcq9dyxlk8ln8cq1ka2gspxp83i-guix-daemon.drv...
building /gnu/store/8x49757i1s0r6igrah8ramn9zc1v577b-guix-69dfdb7bd.drv...
building /gnu/store/fshds3i961v1p3x8f7awh55lix0f7vkl-guix-cran.drv...
building CA certificate bundle...
listing Emacs sub-directories...
building fonts directory...
building directory of Info manuals...
building profile with 2 packages...
building /gnu/store/2lv56bl8ghyf2kvfkwj5iy3mrlfik4la-inferior-script.scm.drv...
building package cache...
--8<---------------cut here---------------end--------------->8---
It goes OK until building /gnu/store/6rfaqfq693vda59a55asc4wjjg52ilns-guix-package-cache.drv, which
consumes memory until it gets OOM-killed.
It used to work fine a few months ago.
Ludo’.
^ permalink raw reply [flat|nested] 5+ messages in thread
* bug#63852: ‘guix-package-cache.drv’ eats all your memory with ‘guix-cran’
2023-06-02 16:22 bug#63852: ‘guix-package-cache.drv’ eats all your memory with ‘guix-cran’ Ludovic Courtès
@ 2023-06-03 14:05 ` Ludovic Courtès
2023-06-05 16:34 ` Ludovic Courtès
0 siblings, 1 reply; 5+ messages in thread
From: Ludovic Courtès @ 2023-06-03 14:05 UTC (permalink / raw)
To: 63852; +Cc: Lars-Dominik Braun
[-- Attachment #1: Type: text/plain, Size: 5306 bytes --]
Ludovic Courtès <ludovic.courtes@inria.fr> skribis:
> It goes OK until building /gnu/store/6rfaqfq693vda59a55asc4wjjg52ilns-guix-package-cache.drv, which
> consumes memory until it gets OOM-killed.
The culprit is easily found (here on just the 22K packages of ‘guix’, so
less than half of what you get with ‘guix-cran’):
--8<---------------cut here---------------start------------->8---
$ guix repl
GNU Guile 3.0.9
Copyright (C) 1995-2023 Free Software Foundation, Inc.
Guile comes with ABSOLUTELY NO WARRANTY; for details type `,show w'.
This program is free software, and you are welcome to redistribute it
under certain conditions; type `,show c' for details.
Enter `,help' for help.
scheme@(guix-user)> ,m(gnu packages)
scheme@(gnu packages)> ,use(statprof)
scheme@(gnu packages)> (gcprof (lambda () (generate-package-cache "/tmp/t")))
% cumulative self
time seconds seconds procedure
50.00 13.77 13.77 ice-9/boot-9.scm:2498:4
6.94 1.91 1.91 display
5.56 1.53 1.53 object->string
2.78 0.76 0.76 reverse
2.78 0.76 0.76 gensym
2.78 0.76 0.76 ice-9/boot-9.scm:2217:0:%load-announce
1.39 6.88 0.38 gnu/packages.scm:388:4:expand-cache
1.39 0.76 0.38 srfi/srfi-1.scm:1028:0:lset-intersection
1.39 0.38 0.38 number->string
1.39 0.38 0.38 gnu/packages/crates-io.scm:17783:2:arguments
1.39 0.38 0.38 open-output-string
1.39 0.38 0.38 gnu/packages/crates-io.scm:18450:2:arguments
1.39 0.38 0.38 gnu/packages/crates-io.scm:36208:2:arguments
1.39 0.38 0.38 guix/packages.scm:1320:8:mproc
1.39 0.38 0.38 gnu/packages/crates-io.scm:25101:2:arguments
1.39 0.38 0.38 gnu/packages/crates-io.scm:29972:2:arguments
1.39 0.38 0.38 gnu/packages/crates-io.scm:883:2:arguments
1.39 0.38 0.38 ice-9/vlist.scm:449:0:vhash-cons
1.39 0.38 0.38 append
1.39 0.38 0.38 string-append
1.39 0.38 0.38 system/vm/assembler.scm:2175:8
1.39 0.38 0.38 guix/build/syscalls.scm:1168:0:read-dirent-header/linux
1.39 0.38 0.38 srfi/srfi-1.scm:1033:17
1.39 0.38 0.38 gnu/packages/crates-io.scm:9336:2:arguments
1.39 0.38 0.38 gnu/packages/crates-io.scm:64009:2:arguments
1.39 0.38 0.38 guix/build-system/cargo.scm:229:4
1.39 0.38 0.38 ice-9/boot-9.scm:2759:0:module-make-local-var!
0.00 7328.64 0.00 system/vm/assembler.scm:1258:0:intern-constant
0.00 1423.97 0.00 ice-9/boot-9.scm:220:5:map1
0.00 201.90 0.00 ice-9/threads.scm:388:4
[...]
---
Sample count: 72
Total time: 27.531145665 seconds (14.056589389 seconds in GC)
--8<---------------cut here---------------end--------------->8---
The assembler allocates a huge constant table in ‘intern-constant’
(‘generate-package-cache’ produces one literal list containing one small
vector per package, each of which contains strings and other constants).
The attached file emulates what ‘generate-package-cache’ does. Each
vector in the list looks like this:
#(0 "xyzxyz-0" "1.0.0" (gnu packages xyz0) xyzxyz-0 ("out") #t #f "gnu/packages/xyz0" 0 1)
The program terminates with roughly a 250 MiB heap for 22K entries—about
10 MiB per vector.
The heap profile on completion looks like this:
--8<---------------cut here---------------start------------->8---
% type self avg obj size
57.5 bytevector 8,205,632 455868.4
23.9 vector 3,413,857 549.3
7.7 struct 1,093,840 40.8
5.3 pair 753,376 16.0
2.7 stringbuf 385,856 51.1
1.1 symbol 151,424 32.0
0.4 program 59,424 47.3
0.3 string 49,152 32.0
0.2 heap-number 23,520 32.0
0.2 variable 21,696 30.3
0.1 smob 21,056 32.0
0.1 vm-continuation 20,864 32.0
0.1 atomic-box 20,736 32.0
0.1 weak-table 19,584 31.8
0.1 unknown 19,040 32.0
0.0 pointer 2,000 16.0
0.0 hash-table 896 32.0
0.0 primitive 144 16.0
0.0 weak-vector 96 16.0
0.0 frame 96 32.0
0.0 primitive-generic 64 32.0
0.0 keyword 16 16.0
sampled heap: 13.60166 MiB (heap size: 233.16016 MiB)
--8<---------------cut here---------------end--------------->8---
It’s likely that the ‘buf’ field of <asm> is what’s taking space
On that topic, see also:
https://lists.gnu.org/archive/html/guile-devel/2023-01/msg00013.html
Ludo’.
[-- Attachment #2: the reproducer --]
[-- Type: text/plain, Size: 1673 bytes --]
;; https://issues.guix.gnu.org/63852
(use-modules (system base compile)
(language tree-il)
(srfi srfi-1)
(srfi srfi-26)
(ice-9 match)
(ice-9 time)
(statprof))
(define lst
(unfold (cut > <> 22000)
(lambda (i)
;; Mimic 'generate-package-cache'.
(define name
(string-append "xyzxyz-" (number->string i 8)))
(vector i name
(string-append "1.0."
(number->string (modulo i 300)))
(list 'gnu 'packages
(string->symbol
(string-append "xyz" (number->string i))))
(string->symbol name)
'("out")
#t
#f
(string-append "gnu/packages/xyz"
(number->string (modulo i 300)))
(modulo i 1000)
1))
1+
0))
(pk 'go! (gc-stats))
(let ((exp (match (or (getenv "VARIANT") "0")
("0" `',lst)
("1" `(list ,@lst))
("2" (make-const #f lst)) ;avoid psyntax overhead
("3" (make-primcall #f 'list
(map (lambda (vector)
(make-const #f vector))
lst))))))
(compile exp #:to 'bytecode
#:from (if (pair? exp)
'scheme
'tree-il)
#:optimization-level 0
#:opts '(#:to-file? #t)))
(pk (gc-stats))
^ permalink raw reply [flat|nested] 5+ messages in thread
* bug#63852: ‘guix-package-cache.drv’ eats all your memory with ‘guix-cran’
2023-06-03 14:05 ` Ludovic Courtès
@ 2023-06-05 16:34 ` Ludovic Courtès
2023-06-05 21:58 ` Ludovic Courtès
2023-06-15 13:13 ` Ludovic Courtès
0 siblings, 2 replies; 5+ messages in thread
From: Ludovic Courtès @ 2023-06-05 16:34 UTC (permalink / raw)
To: 63852; +Cc: Lars-Dominik Braun
Ludovic Courtès <ludo@gnu.org> skribis:
> Ludovic Courtès <ludovic.courtes@inria.fr> skribis:
>
>> It goes OK until building /gnu/store/6rfaqfq693vda59a55asc4wjjg52ilns-guix-package-cache.drv, which
>> consumes memory until it gets OOM-killed.
>
> The culprit is easily found (here on just the 22K packages of ‘guix’, so
> less than half of what you get with ‘guix-cran’):
(Beware of “easily” when the guy doesn’t know what he’s talking about.)
Turns out the problem is more pedestrian: ‘r-future-tests’ depends on
itself.
--8<---------------cut here---------------start------------->8---
$ guix import cran future.tests
Starting download of /tmp/guix-file.TOBjcf
From http://cran.r-project.org/src/contrib/future.tests_0.7.0.tar.gz...
…_0.7.0.tar.gz 186KiB 1.2MiB/s 00:00 ▕██████████████████▏ 100.0%
(package
(name "r-future-tests")
(version "0.7.0")
(source (origin
(method url-fetch)
(uri (cran-uri "future.tests" version))
(sha256
(base32
"10g1w99xqr7l0cn27642aphqcvfidgpas38f84r815yy6k1ryrlx"))))
(properties `((upstream-name . "future.tests")))
(build-system r-build-system)
(propagated-inputs (list r-cli r-crayon r-future r-prettyunits r-sessioninfo))
(native-inputs (list r-future-tests))
(home-page "https://future.tests.futureverse.org")
(synopsis "Test Suite for 'Future API' Backends")
(description
"Backends implementing the Future API, as defined by the future package, should
use the tests provided by this package to validate that they meet the minimal
requirements of the Future API. The tests can be performed easily from within R
or from outside of R from the command line making it straightforward to include
them in package tests and in Continuous Integration (CI) pipelines.")
$ guix describe
Generation 265 Jun 04 2023 23:48:31 (current)
guix eed55a6
repository URL: https://git.savannah.gnu.org/git/guix.git
branch: master
commit: eed55a6544d5bda2245ec853e5fa4b28e1865bea
--8<---------------cut here---------------end--------------->8---
To be fixed in the importer.
Ludo’.
^ permalink raw reply [flat|nested] 5+ messages in thread
* bug#63852: ‘guix-package-cache.drv’ eats all your memory with ‘guix-cran’
2023-06-05 16:34 ` Ludovic Courtès
@ 2023-06-05 21:58 ` Ludovic Courtès
2023-06-15 13:13 ` Ludovic Courtès
1 sibling, 0 replies; 5+ messages in thread
From: Ludovic Courtès @ 2023-06-05 21:58 UTC (permalink / raw)
To: 63852; +Cc: Lars-Dominik Braun
Ludovic Courtès <ludo@gnu.org> skribis:
> Ludovic Courtès <ludo@gnu.org> skribis:
>
>> Ludovic Courtès <ludovic.courtes@inria.fr> skribis:
>>
>>> It goes OK until building /gnu/store/6rfaqfq693vda59a55asc4wjjg52ilns-guix-package-cache.drv, which
>>> consumes memory until it gets OOM-killed.
>>
>> The culprit is easily found (here on just the 22K packages of ‘guix’, so
>> less than half of what you get with ‘guix-cran’):
>
> (Beware of “easily” when the guy doesn’t know what he’s talking about.)
>
> Turns out the problem is more pedestrian: ‘r-future-tests’ depends on
> itself.
Here’s a patch to prevent that from happening again:
https://issues.guix.gnu.org/63917
(We also need to fix the importer.)
Ludo’.
^ permalink raw reply [flat|nested] 5+ messages in thread
* bug#63852: ‘guix-package-cache.drv’ eats all your memory with ‘guix-cran’
2023-06-05 16:34 ` Ludovic Courtès
2023-06-05 21:58 ` Ludovic Courtès
@ 2023-06-15 13:13 ` Ludovic Courtès
1 sibling, 0 replies; 5+ messages in thread
From: Ludovic Courtès @ 2023-06-15 13:13 UTC (permalink / raw)
To: 63852-done; +Cc: Lars-Dominik Braun
Ludovic Courtès <ludo@gnu.org> skribis:
> Ludovic Courtès <ludo@gnu.org> skribis:
>
>> Ludovic Courtès <ludovic.courtes@inria.fr> skribis:
>>
>>> It goes OK until building /gnu/store/6rfaqfq693vda59a55asc4wjjg52ilns-guix-package-cache.drv, which
>>> consumes memory until it gets OOM-killed.
>>
>> The culprit is easily found (here on just the 22K packages of ‘guix’, so
>> less than half of what you get with ‘guix-cran’):
>
> (Beware of “easily” when the guy doesn’t know what he’s talking about.)
>
> Turns out the problem is more pedestrian: ‘r-future-tests’ depends on
> itself.
With commit e4259d4e9e3251e4c4b45d1cce4008ac32b504c8, such cycles are
detected early on.
Closing this bug.
We should still improve the memory footprint of the assembler though.
Ludo’.
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2023-06-15 13:14 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2023-06-02 16:22 bug#63852: ‘guix-package-cache.drv’ eats all your memory with ‘guix-cran’ Ludovic Courtès
2023-06-03 14:05 ` Ludovic Courtès
2023-06-05 16:34 ` Ludovic Courtès
2023-06-05 21:58 ` Ludovic Courtès
2023-06-15 13:13 ` Ludovic Courtès
Code repositories for project(s) associated with this public inbox
https://git.savannah.gnu.org/cgit/guix.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).