bug#65446: [cuirass] guix gc throws: "error: build daemon out of memory"

bug#65446: [cuirass] guix gc throws: "error: build daemon out of memory"

[Maxim Cournoyer]

Hi,

Since a few days on Berlin, the following appears in /var/log/mcron.log:

--8<---------------cut here---------------start------------->8---
2023-08-22 04:00:29 28215 /gnu/store/4lmpqf8dnk86avq5xa2l6bznxnmrz27x-guix-1.4.0-7.44bbfc2/bin/guix gc: guix gc: error: build daemon out of memory
--8<---------------cut here---------------end--------------->8---

It's not just a fluke, according to the kernel's dmesg:

--8<---------------cut here---------------start------------->8---
[7190243.475312] __vm_enough_memory: pid: 28300, comm: guix-daemon, not enough memory for the allocation
--8<---------------cut here---------------end--------------->8---

Once it appears, it can be triggered simply by running 'guix gc'.  The
problem can be worked around by restarting the Guix daemon, with:

--8<---------------cut here---------------start------------->8---
herd restart guix-daemon
--8<---------------cut here---------------end--------------->8---

Any ideas what could cause such a condition?

The machine hasn't been rebooted in 83 days, perhaps we should start
with that.

-- 
Thanks,
Maxim

bug#65446: [berlin] guix gc throws: "error: build daemon out of memory"

[Ludovic Courtès]

Hi!

Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:

> Since a few days on Berlin, the following appears in /var/log/mcron.log:
>
> 2023-08-22 04:00:29 28215 /gnu/store/4lmpqf8dnk86avq5xa2l6bznxnmrz27x-guix-1.4.0-7.44bbfc2/bin/guix gc: guix gc: error: build daemon out of memory
>
>
> It's not just a fluke, according to the kernel's dmesg:
>
> [7190243.475312] __vm_enough_memory: pid: 28300, comm: guix-daemon, not enough memory for the allocation
>
>
> Once it appears, it can be triggered simply by running 'guix gc'.  The
> problem can be worked around by restarting the Guix daemon, with:
>
> herd restart guix-daemon
>
> Any ideas what could cause such a condition?

No idea!

I ran “guix gc -C0” minutes ago, which reproduced the problem;
/var/log/guix-daemon.log reads this:

--8<---------------cut here---------------start------------->8---
2023-09-02 21:40:31 accepted connection from pid 12853, user ludo
2023-09-02 21:40:50 unexpected build daemon error: std::bad_alloc
--8<---------------cut here---------------end--------------->8---

and the kernel writes what you wrote above:

--8<---------------cut here---------------start------------->8---
Sep  2 21:40:50 localhost vmunix: [8204269.198144] __vm_enough_memory: pid: 12873, comm: guix-daemon, not enough memory for the allocation
Sep  2 21:40:50 localhost vmunix: [8204269.207396] __vm_enough_memory: pid: 12873, comm: guix-daemon, not enough memory for the allocation
Sep  2 21:40:50 localhost vmunix: [8204269.216639] __vm_enough_memory: pid: 12873, comm: guix-daemon, not enough memory for the allocation
--8<---------------cut here---------------end--------------->8---

My reading of the Linux code is ‘__vm_enough_memory’ is used by Linux
Security Modules (LSM), so perhaps something changed in our kernel
config, be it compile-time or run-time?  (Cc: Leo just in case.)

This is with Linux-libre 6.3.4-gnu.

Ludo’.

bug#65446: [berlin] guix gc throws: "error: build daemon out of memory"

[Collin J. Doering via Bug reports for GNU Guix]

[-- Attachment #1: Type: text/plain, Size: 333 bytes --]

Perhaps worth noting that I've seen this numerous times on cuirass.genenetwork.org.

Usually restarting guix-daemon will allow 'guix gc' to complete without error. I see dmesg errors that match what Ludo provided.

Kind regards

-- 
Collin J. Doering

http://rekahsoft.ca
http://blog.rekahsoft.ca
http://git.rekahsoft.ca

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 255 bytes --]

bug#65446: [berlin] guix gc throws: "error: build daemon out of memory"

[Maxim Cournoyer]

Hello,

"Collin J. Doering" via Bug reports for GNU Guix <bug-guix@gnu.org>
writes:

> Perhaps worth noting that I've seen this numerous times on cuirass.genenetwork.org.
>
> Usually restarting guix-daemon will allow 'guix gc' to complete
> without error. I see dmesg errors that match what Ludo provided.

Thanks for tipping in; it's a useful thing to know the bug is
reproducible in other environments.

-- 
Thanks,
Maxim