bug#74381: High memory usage during guix pull (i686-linux, guile jit)

bug#74381: High memory usage during guix pull (i686-linux, guile jit)

[Dariqq]

Hi,

Using 'guix pull' on i686-linux regularly fails when building 
'guix-packages-base':

E.g this recent run on ci.guix.gnu.org: 
https://ci.guix.gnu.org/build/6472396/details

Log from me trying to pull 3e8d3d80f41e016cdfe80e488a78c2351c94fef8:

Warning: Failed to expand heap by 8192 KiB
GC Warning: Failed to expand heap by 8192 KiB
GC Warning: Failed to expand heap by 8192 KiB
GC Warning: Failed to expand heap by 8192 KiB
GC Warning: Failed to expand heap by 8192 KiB
GC Warning: Failed to expand heap by 8192 KiB
GC Warning: Failed to expand heap by 8192 KiB
GC Warning: Failed to expand heap by 8192 KiB
GC Warning: Failed to expand heap by 8192 KiB
GC Warning: Failed to expand heap by 8192 KiB
GC Warning: Failed to expand heap by 8192 KiB
GC Warning: Failed to expand heap by 8192 KiB
GC Warning: Failed to expand heap by 64 KiB
GC Warning: Out of Memory! Heap size: 3436 MiB. Returning NULL!
Warning: Unwind-only out of memory exception; skipping pre-unwind handler.
Warning: Unwind-only out of memory exception; skipping pre-unwind handler.
[ 19/ 20] compiling...	 90.0% of 10 filesallocating JIT code buffer 
failed: Cannot allocate memory
JIT failed due to resource exhaustion
disabling automatic JIT compilation
allocate_stack failed: Cannot allocate memory
Warning: Unwind-only stack overflow exception; skipping pre-unwind handler.
[...]

The error suggests this might be related to guile jit compiling things

Disabling it by adding GUILE_JIT_THRESHOLD=-1 to the guix-daemon 
environment got things working for me again.

bug#74381: High memory usage during guix pull (i686-linux, guile jit)

[Dariqq]

It turns out disabling guile jit does not work reliably and even without 
it guix pull may oom building guix-packages-base using 3GB+ of RAM.

My workaround for now is to run 'guix pull -s i686-linux -p 
/tmp/whatever' on an x86_64 machine (to have more RAM available) with 
guile jit disabled multiple times until it succeeds (not exactly sure 
why that works, is something getting cached somewhere?) and use that 
machine to serve substitutes.

After mentioning this on irc today janneke mentioned this commit 
https://git.savannah.gnu.org/cgit/guix.git/commit/?id=15c5f1a2c20b21de0f19f42db1ccab4c42117ebb 
which reduces the number of files per chunk from 25 to 10 to get guix 
pull working on the 32 bit hurd.

I tried reducing that even further to 5 in my local checkout and the max 
RAM usage went to (a little bit more reasonable) 1.5-2 GB (both on i686 
natively and when emulating with -s i686 with and without guile jit). 
This would be still not enough for my little machine but might be enough 
for the substitute servers to build it reliably.


I also wanted to try this in a i586-gnu childhurd but guix pull 
immediately segfaulted.

bug#74381: High memory usage during guix pull (i686-linux, guile jit)

[Dariqq]

Yesterday I had a very weird situation,
I tried to pull f365d6c6365b823bae2ae06d53088088ee07fba7 which failed on 
i686 natively with the 'Failed to expand Heap by 8192 KiB' warning.

Rerun things on an x86_64 machine with -s i686-linux and building 
packages-base succededd with the guile process memory maxing out at 3.5 
GB  (+ another 500 MB for the guix process)

However when I then pulled on the i686 machine substituting the 
packages-base guix was then left in a halfbroken state claiming that 
rust-criterion-0.5 is unbound which led to an error for almost anything 
useful(i think determining librsvg-for-system).

I checked the guix (gnu packages crates-check) of that generation output 
and the scm file seemed file, so something must have gone horrible wrong 
when that one got (jit) - compiled.

Unfortunately I had to gc the evidence to continue.

I then rolled back, deleted and gced the broken generation and repulled 
(now 666f484fcfb03d801dc5c78a15501e01eb78567d) this time disabling 
guile-jit on the x86-64 machine as well, which drastically reduced 
memory consumption. After pulling that one on i686 aswell I had no 
further issues