unofficial mirror of bug-guix@gnu.org 
 help / color / mirror / code / Atom feed
* bug#48963: "guix build guix" halts after exhausting memory
@ 2021-06-11 22:23 Julien Lepiller
  2021-06-11 22:30 ` Julien Lepiller
  2021-06-16 20:01 ` Ludovic Courtès
  0 siblings, 2 replies; 10+ messages in thread
From: Julien Lepiller @ 2021-06-11 22:23 UTC (permalink / raw)
  To: 48963

Hi Guix!

I tried updating my system on my armhf board (2GB of RAM), but during
"guix system reconfigure", guix tries to build itself (the guix package
from (gnu packages package-management)). This package uses too much
memory to build, and I start getting GC warnings like so:

GC Warning: Out of memory - trying to allocate requested amount (552
bytes)...
GC Warning: Header allocation failed: dropping block
GC Warning: Out of Memory! Heap size: 2571 MiB. Returning NULL!
Warning: Unwind-only out of memory exception; skipping pre-unwind
handler.

after some more progress and a lot more warnings, the build stops (CPU
is at 2%), but the memory is not freed. The build stayed stuck for a
long time until I decided to cancel it.

As a work-around, I tried using the guix-daemon package instead (by
changing the guix field in guix-configuration), but "guix system" still
wants to build the guix package anyway. I tried finding usages of the
guix package (grepping for "package-management" yields a relatively
short list of files, and none of them seems to be used by my config,
except (gnu services base) which always uses the guix from the
configuration). Why?

Can we instead break the build (at the Makefile level) into multiple
smaller chunks, that require less memory, in the same way (guix self)
works?




^ permalink raw reply	[flat|nested] 10+ messages in thread

* bug#48963: "guix build guix" halts after exhausting memory
  2021-06-11 22:23 bug#48963: "guix build guix" halts after exhausting memory Julien Lepiller
@ 2021-06-11 22:30 ` Julien Lepiller
  2021-06-16 20:01 ` Ludovic Courtès
  1 sibling, 0 replies; 10+ messages in thread
From: Julien Lepiller @ 2021-06-11 22:30 UTC (permalink / raw)
  To: 48963

Le Sat, 12 Jun 2021 00:23:30 +0200,
Julien Lepiller <julien@lepiller.eu> a écrit :

> Hi Guix!
> 
> I tried updating my system on my armhf board (2GB of RAM), but during
> "guix system reconfigure", guix tries to build itself (the guix
> package from (gnu packages package-management)). This package uses
> too much memory to build, and I start getting GC warnings like so:
> 
> GC Warning: Out of memory - trying to allocate requested amount (552
> bytes)...
> GC Warning: Header allocation failed: dropping block
> GC Warning: Out of Memory! Heap size: 2571 MiB. Returning NULL!
> Warning: Unwind-only out of memory exception; skipping pre-unwind
> handler.
> 
> after some more progress and a lot more warnings, the build stops (CPU
> is at 2%), but the memory is not freed. The build stayed stuck for a
> long time until I decided to cancel it.
> 
> As a work-around, I tried using the guix-daemon package instead (by
> changing the guix field in guix-configuration), but "guix system"
> still wants to build the guix package anyway. I tried finding usages
> of the guix package (grepping for "package-management" yields a
> relatively short list of files, and none of them seems to be used by
> my config, except (gnu services base) which always uses the guix from
> the configuration). Why?
> 
> Can we instead break the build (at the Makefile level) into multiple
> smaller chunks, that require less memory, in the same way (guix self)
> works?
> 
> 
> 

also note it's not an OOM issue at the system level, because guile
doesn't seem to care I have 8GB of free swap it could use. Top reports
1.8GB of resident memory being used by the process. I tried to stop all
services at the time in the hope it would save some memory and help,
but guile just kept swallowing all that free memory I gave it.




^ permalink raw reply	[flat|nested] 10+ messages in thread

* bug#48963: "guix build guix" halts after exhausting memory
  2021-06-11 22:23 bug#48963: "guix build guix" halts after exhausting memory Julien Lepiller
  2021-06-11 22:30 ` Julien Lepiller
@ 2021-06-16 20:01 ` Ludovic Courtès
  2021-06-17 13:58   ` Julien Lepiller
  2021-06-23 21:43   ` Ludovic Courtès
  1 sibling, 2 replies; 10+ messages in thread
From: Ludovic Courtès @ 2021-06-16 20:01 UTC (permalink / raw)
  To: Julien Lepiller; +Cc: 48963

[-- Attachment #1: Type: text/plain, Size: 1323 bytes --]

Hi,

Julien Lepiller <julien@lepiller.eu> skribis:

> I tried updating my system on my armhf board (2GB of RAM), but during
> "guix system reconfigure", guix tries to build itself (the guix package
> from (gnu packages package-management)). This package uses too much
> memory to build, and I start getting GC warnings like so:
>
> GC Warning: Out of memory - trying to allocate requested amount (552
> bytes)...
> GC Warning: Header allocation failed: dropping block
> GC Warning: Out of Memory! Heap size: 2571 MiB. Returning NULL!
> Warning: Unwind-only out of memory exception; skipping pre-unwind
> handler.
>
> after some more progress and a lot more warnings, the build stops (CPU
> is at 2%), but the memory is not freed. The build stayed stuck for a
> long time until I decided to cancel it.

This is ridiculous.  :-/

> Can we instead break the build (at the Makefile level) into multiple
> smaller chunks, that require less memory, in the same way (guix self)
> works?

Yes, that’s a good idea.

Could you check the extent to which the attached patch helps on this
machine?

It doesn’t split as much as (guix self) does, only in three pieces, but
hopefully that helps a bit.  A side effect is that progress report is
now off, but we can fix it later.

Thanks,
Ludo’.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Type: text/x-patch, Size: 1425 bytes --]

diff --git a/Makefile.am b/Makefile.am
index aa21b5383b..758d8b9b8a 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -663,7 +663,11 @@ CLEANFILES =					\
 # the whole thing.  Likewise, set 'XDG_CACHE_HOME' to avoid loading possibly
 # stale files from ~/.cache/guile/ccache.
 %.go: make-go ; @:
-make-go: $(MODULES) guix/config.scm $(dist_noinst_DATA)
+make-go: make-core-go make-packages-go make-system-go
+
+define guile-compilation-rule 
+
+$(1): $(2)
 	$(AM_V_at)echo "Compiling Scheme modules..." ;			\
 	unset GUILE_LOAD_COMPILED_PATH ;				\
 	XDG_CACHE_HOME=/nowhere						\
@@ -671,7 +675,19 @@ make-go: $(MODULES) guix/config.scm $(dist_noinst_DATA)
 	$(top_builddir)/pre-inst-env					\
 	$(GUILE) -L "$(top_builddir)" -L "$(top_srcdir)"		\
 	  --no-auto-compile 						\
-	  -s "$(top_srcdir)"/build-aux/compile-all.scm $^
+	  -s "$(top_srcdir)"/build-aux/compile-all.scm $$(filter %.scm,$$^)
+
+.PHONY: $(1)
+
+endef
+
+MODULES_CORE     = $(filter guix/%,$(MODULES))
+MODULES_PACKAGES = $(filter gnu/packages/%,$(MODULES))
+MODULES_SYSTEM   = $(filter-out gnu/packages/%,$(filter gnu/%,$(MODULES)))
+
+$(eval $(call guile-compilation-rule,make-core-go,$(MODULES_CORE) guix/config.scm $(dist_noinst_DATA)))
+$(eval $(call guile-compilation-rule,make-packages-go,$(MODULES_PACKAGES) make-core-go))
+$(eval $(call guile-compilation-rule,make-system-go,$(MODULES_SYSTEM) make-packages-go make-core-go))
 
 SUFFIXES = .go
 

^ permalink raw reply related	[flat|nested] 10+ messages in thread

* bug#48963: "guix build guix" halts after exhausting memory
  2021-06-16 20:01 ` Ludovic Courtès
@ 2021-06-17 13:58   ` Julien Lepiller
  2021-06-17 22:17     ` Julien Lepiller
  2021-06-23 21:43   ` Ludovic Courtès
  1 sibling, 1 reply; 10+ messages in thread
From: Julien Lepiller @ 2021-06-17 13:58 UTC (permalink / raw)
  To: Ludovic Courtès; +Cc: 48963

[-- Attachment #1: Type: text/plain, Size: 1806 bytes --]

Hi Ludo,

I tried your patch by creating a variant of the guix package. My first attempt was a failure because it's still using almost all my memory and I forgot to enable my swap. In the second attempt, the build phase succeeded, but the build failed during the test phase (test/inferior). I've started a new build hoping it's a non-deterministic failure.

Le 16 juin 2021 16:01:20 GMT-04:00, "Ludovic Courtès" <ludo@gnu.org> a écrit :
>Hi,
>
>Julien Lepiller <julien@lepiller.eu> skribis:
>
>> I tried updating my system on my armhf board (2GB of RAM), but during
>> "guix system reconfigure", guix tries to build itself (the guix
>package
>> from (gnu packages package-management)). This package uses too much
>> memory to build, and I start getting GC warnings like so:
>>
>> GC Warning: Out of memory - trying to allocate requested amount (552
>> bytes)...
>> GC Warning: Header allocation failed: dropping block
>> GC Warning: Out of Memory! Heap size: 2571 MiB. Returning NULL!
>> Warning: Unwind-only out of memory exception; skipping pre-unwind
>> handler.
>>
>> after some more progress and a lot more warnings, the build stops
>(CPU
>> is at 2%), but the memory is not freed. The build stayed stuck for a
>> long time until I decided to cancel it.
>
>This is ridiculous.  :-/
>
>> Can we instead break the build (at the Makefile level) into multiple
>> smaller chunks, that require less memory, in the same way (guix self)
>> works?
>
>Yes, that’s a good idea.
>
>Could you check the extent to which the attached patch helps on this
>machine?
>
>It doesn’t split as much as (guix self) does, only in three pieces, but
>hopefully that helps a bit.  A side effect is that progress report is
>now off, but we can fix it later.
>
>Thanks,
>Ludo’.

[-- Attachment #2: Type: text/html, Size: 2342 bytes --]

^ permalink raw reply	[flat|nested] 10+ messages in thread

* bug#48963: "guix build guix" halts after exhausting memory
  2021-06-17 13:58   ` Julien Lepiller
@ 2021-06-17 22:17     ` Julien Lepiller
  2021-06-18  9:51       ` Ludovic Courtès
  0 siblings, 1 reply; 10+ messages in thread
From: Julien Lepiller @ 2021-06-17 22:17 UTC (permalink / raw)
  To: 48963

Le Thu, 17 Jun 2021 09:58:09 -0400,
Julien Lepiller <julien@lepiller.eu> a écrit :

> Hi Ludo,
> 
> I tried your patch by creating a variant of the guix package. My
> first attempt was a failure because it's still using almost all my
> memory and I forgot to enable my swap. In the second attempt, the
> build phase succeeded, but the build failed during the test phase
> (test/inferior). I've started a new build hoping it's a
> non-deterministic failure.

I'm afraid after testing 2 more times that the test failure is
happening consistently, at least of armhf.




^ permalink raw reply	[flat|nested] 10+ messages in thread

* bug#48963: "guix build guix" halts after exhausting memory
  2021-06-17 22:17     ` Julien Lepiller
@ 2021-06-18  9:51       ` Ludovic Courtès
  2021-06-18 10:52         ` Julien Lepiller
  0 siblings, 1 reply; 10+ messages in thread
From: Ludovic Courtès @ 2021-06-18  9:51 UTC (permalink / raw)
  To: Julien Lepiller; +Cc: 48963

Hi,

Julien Lepiller <julien@lepiller.eu> skribis:

> Le Thu, 17 Jun 2021 09:58:09 -0400,
> Julien Lepiller <julien@lepiller.eu> a écrit :
>
>> Hi Ludo,
>> 
>> I tried your patch by creating a variant of the guix package. My
>> first attempt was a failure because it's still using almost all my
>> memory and I forgot to enable my swap. In the second attempt, the
>> build phase succeeded, but the build failed during the test phase
>> (test/inferior). I've started a new build hoping it's a
>> non-deterministic failure.
>
> I'm afraid after testing 2 more times that the test failure is
> happening consistently, at least of armhf.

(The _build_ failure, right?)

Have you been able to see how this affects max RSS?

Does it go any further than without the patch?

Thanks,
Ludo’.




^ permalink raw reply	[flat|nested] 10+ messages in thread

* bug#48963: "guix build guix" halts after exhausting memory
  2021-06-18  9:51       ` Ludovic Courtès
@ 2021-06-18 10:52         ` Julien Lepiller
  2021-06-20 21:00           ` Ludovic Courtès
  0 siblings, 1 reply; 10+ messages in thread
From: Julien Lepiller @ 2021-06-18 10:52 UTC (permalink / raw)
  To: Ludovic Courtès; +Cc: 48963

[-- Attachment #1: Type: text/plain, Size: 1035 bytes --]

No, the _test_ failure is consistent. The build itself now always completes, which is much better that before.

Le 18 juin 2021 05:51:14 GMT-04:00, "Ludovic Courtès" <ludo@gnu.org> a écrit :
>Hi,
>
>Julien Lepiller <julien@lepiller.eu> skribis:
>
>> Le Thu, 17 Jun 2021 09:58:09 -0400,
>> Julien Lepiller <julien@lepiller.eu> a écrit :
>>
>>> Hi Ludo,
>>> 
>>> I tried your patch by creating a variant of the guix package. My
>>> first attempt was a failure because it's still using almost all my
>>> memory and I forgot to enable my swap. In the second attempt, the
>>> build phase succeeded, but the build failed during the test phase
>>> (test/inferior). I've started a new build hoping it's a
>>> non-deterministic failure.
>>
>> I'm afraid after testing 2 more times that the test failure is
>> happening consistently, at least of armhf.
>
>(The _build_ failure, right?)
>
>Have you been able to see how this affects max RSS?
>
>Does it go any further than without the patch?
>
>Thanks,
>Ludo’.

[-- Attachment #2: Type: text/html, Size: 1554 bytes --]

^ permalink raw reply	[flat|nested] 10+ messages in thread

* bug#48963: "guix build guix" halts after exhausting memory
  2021-06-18 10:52         ` Julien Lepiller
@ 2021-06-20 21:00           ` Ludovic Courtès
  2021-06-21 11:05             ` Julien Lepiller
  0 siblings, 1 reply; 10+ messages in thread
From: Ludovic Courtès @ 2021-06-20 21:00 UTC (permalink / raw)
  To: Julien Lepiller; +Cc: 48963

Hi!

Julien Lepiller <julien@lepiller.eu> skribis:

> No, the _test_ failure is consistent. The build itself now always completes, which is much better that before.

Ah, that’s great!  I can work on a variant of this patch that shows
correct completion numbers.

Could you share ‘tests/inferiors.log’ if that’s the one that’s failing?
Is it also failing for you on x86_64?

Thanks,
Ludo’.




^ permalink raw reply	[flat|nested] 10+ messages in thread

* bug#48963: "guix build guix" halts after exhausting memory
  2021-06-20 21:00           ` Ludovic Courtès
@ 2021-06-21 11:05             ` Julien Lepiller
  0 siblings, 0 replies; 10+ messages in thread
From: Julien Lepiller @ 2021-06-21 11:05 UTC (permalink / raw)
  To: Ludovic Courtès; +Cc: 48963

Le Sun, 20 Jun 2021 23:00:53 +0200,
Ludovic Courtès <ludo@gnu.org> a écrit :

> Hi!
> 
> Julien Lepiller <julien@lepiller.eu> skribis:
> 
> > No, the _test_ failure is consistent. The build itself now always
> > completes, which is much better that before.  
> 
> Ah, that’s great!  I can work on a variant of this patch that shows
> correct completion numbers.
> 
> Could you share ‘tests/inferiors.log’ if that’s the one that’s
> failing? Is it also failing for you on x86_64?
> 
> Thanks,
> Ludo’.

Here is the content of `tests/inferiors.log`. The failure is the same
on armhf and x86_64:

Backtrace:
           2 (primitive-load-path "tests/inferior.scm")
In ice-9/eval.scm:
   626:19  1 (_ #<directory (test-inferior) 7ffff5d7ad20>)
In unknown file:
           0 (dirname #f)

ERROR: In procedure dirname:
In procedure scm_to_utf8_stringn: Wrong type argument in position 1
(expecting string): #f




^ permalink raw reply	[flat|nested] 10+ messages in thread

* bug#48963: "guix build guix" halts after exhausting memory
  2021-06-16 20:01 ` Ludovic Courtès
  2021-06-17 13:58   ` Julien Lepiller
@ 2021-06-23 21:43   ` Ludovic Courtès
  1 sibling, 0 replies; 10+ messages in thread
From: Ludovic Courtès @ 2021-06-23 21:43 UTC (permalink / raw)
  To: Julien Lepiller; +Cc: 48963-done

Hi,

Ludovic Courtès <ludo@gnu.org> skribis:

> Julien Lepiller <julien@lepiller.eu> skribis:

>> Can we instead break the build (at the Makefile level) into multiple
>> smaller chunks, that require less memory, in the same way (guix self)
>> works?
>
> Yes, that’s a good idea.
>
> Could you check the extent to which the attached patch helps on this
> machine?
>
> It doesn’t split as much as (guix self) does, only in three pieces, but
> hopefully that helps a bit.  A side effect is that progress report is
> now off, but we can fix it later.

I pushed a variant of this patch, resorting to an ugly hack so that
compile-all.scm can still estimate progress.  The estimate is correct if
you’re building all the .go files, not so much if you just do, say:

  make
  rm guix/cpio.go && make

because guix/cpio.scm is in the first module set, so it’ll think it’s
starting from scratch and display “0%”.

Fixing it is left as an exercise to the reader.  :-)

It seemed more important to me to fix the memory exhaustion issue first.

Thanks,
Ludo’.




^ permalink raw reply	[flat|nested] 10+ messages in thread

end of thread, other threads:[~2021-06-23 21:45 UTC | newest]

Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-06-11 22:23 bug#48963: "guix build guix" halts after exhausting memory Julien Lepiller
2021-06-11 22:30 ` Julien Lepiller
2021-06-16 20:01 ` Ludovic Courtès
2021-06-17 13:58   ` Julien Lepiller
2021-06-17 22:17     ` Julien Lepiller
2021-06-18  9:51       ` Ludovic Courtès
2021-06-18 10:52         ` Julien Lepiller
2021-06-20 21:00           ` Ludovic Courtès
2021-06-21 11:05             ` Julien Lepiller
2021-06-23 21:43   ` Ludovic Courtès

Code repositories for project(s) associated with this public inbox

	https://git.savannah.gnu.org/cgit/guix.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).