guix --container is RAM hungry

guix --container is RAM hungry

[Edouard Klein]

Dear Guixers,

I'm a huge fan of guix --container, and I created a system to use those
by default for network services. But the VPS these services run on has
only 2GB of RAM, and I just realized that a container, by default,
requires at least 200MB.

Try it:
guix shell time which -- bash -c "$(which time) -v guix shell
--container"

Then Ctrl-D, and look for this line:
	Maximum resident set size (kbytes): 291300

291MB of RAM to run bash.


By contrast, removing the --container option from above yields:
	Maximum resident set size (kbytes): 64496

64MB, still a lot, but I can live with that.

I tried various calls to unshare, but got no significant increase of
RAM.

From an outsider's point of view, the --container option is a wrapper
for unshare, and the dependency resolution is done by guix shell with
or without the --container option, so I don't understand where the RAM
explosion come from.
I mean no implication that '--container' is simple or trivial to
implement, I just wrestled with namespaces for a few weeks and I know
they're a pain, to stay polite. I'm thankful for the tool and would like
to use it more, but I can't in its current state.


Any ideas ?

Thanks in advance,

Edouard.

Re: guix --container is RAM hungry

[raingloom]

On 2024-03-21 22:44, Edouard Klein wrote:
> Dear Guixers,
> 
> I'm a huge fan of guix --container, and I created a system to use those
> by default for network services. But the VPS these services run on has
> only 2GB of RAM, and I just realized that a container, by default,
> requires at least 200MB.
> 
> Try it:
> guix shell time which -- bash -c "$(which time) -v guix shell
> --container"
> 
> Then Ctrl-D, and look for this line:
> 	Maximum resident set size (kbytes): 291300
> 
> 291MB of RAM to run bash.
> 
> 
> By contrast, removing the --container option from above yields:
> 	Maximum resident set size (kbytes): 64496
> 
> 64MB, still a lot, but I can live with that.
> 
> I tried various calls to unshare, but got no significant increase of
> RAM.
> 
> From an outsider's point of view, the --container option is a wrapper
> for unshare, and the dependency resolution is done by guix shell with
> or without the --container option, so I don't understand where the RAM
> explosion come from.
> I mean no implication that '--container' is simple or trivial to
> implement, I just wrestled with namespaces for a few weeks and I know
> they're a pain, to stay polite. I'm thankful for the tool and would like
> to use it more, but I can't in its current state.
> 
> 
> Any ideas ?
> 
> Thanks in advance,
> 
> Edouard.

My first guess is that the Guile process is hanging around waiting for
the process it launched, instead of exec-ing into it directly.

Re: guix --container is RAM hungry

[Ricardo Wurmus]

raingloom@riseup.net writes:

> On 2024-03-21 22:44, Edouard Klein wrote:
>> Dear Guixers,
>> 
>> I'm a huge fan of guix --container, and I created a system to use those
>> by default for network services. But the VPS these services run on has
>> only 2GB of RAM, and I just realized that a container, by default,
>> requires at least 200MB.
>> 
>> Try it:
>> guix shell time which -- bash -c "$(which time) -v guix shell
>> --container"
>> 
>> Then Ctrl-D, and look for this line:
>> 	Maximum resident set size (kbytes): 291300
>> 
>> 291MB of RAM to run bash.
>> 
>> 
>> By contrast, removing the --container option from above yields:
>> 	Maximum resident set size (kbytes): 64496
>> 
>> 64MB, still a lot, but I can live with that.
[...]
> My first guess is that the Guile process is hanging around waiting for
> the process it launched, instead of exec-ing into it directly.

In (@@ (guix scripts environment) launch-environment) we use execlp;
launch-environment is used by both launch-environment/fork as well as
launch-environment/container.  There is, however, an awful lot of
forking going on.

launch-environment/container uses call-with-container, which ends in
using launch-environment.  call-with-container forks, and
launch-environment also forks.

The Guile process that was started when launching Guix probably doesn't
get replaced in this case, as raingloom guessed.

-- 
Ricardo

Re: guix --container is RAM hungry

[Ludovic Courtès]

Hi Edouard,

Edouard Klein <edou@rdklein.fr> skribis:

> I'm a huge fan of guix --container, and I created a system to use those
> by default for network services. But the VPS these services run on has
> only 2GB of RAM, and I just realized that a container, by default,
> requires at least 200MB.

Ouch, confirmed:

--8<---------------cut here---------------start------------->8---
$ \time -v guix shell -C coreutils -- uname 2>&1 |grep 'Maximum resident'
        Maximum resident set size (kbytes): 283048
$ \time -v guix shell coreutils -- uname 2>&1 |grep 'Maximum resident'
        Maximum resident set size (kbytes): 56588
$ guix describe
Generation 297  Mar 24 2024 23:12:25    (current)
  guix 28bc0e8
    repository URL: https://git.savannah.gnu.org/git/guix.git
    branch: master
    commit: 28bc0e870b4d48b8e3e773382bb0e999df2e3611
--8<---------------cut here---------------end--------------->8---

As raingloom and Ricardo wrote, there’s a Guile process that keeps
waiting.

The ‘-C’ variant has to load more modules and do more work (compute
derivations), so it’s not surprising that it takes more memory than the
other variant (on a cache hit it has nothing to do).

Merely loading (guix scripts shell) consumes a lot, but GC says it’s
“only” using 7 MiB out of this:

--8<---------------cut here---------------start------------->8---
$ \time -f %M guile  -c '(use-modules (guix scripts shell)) (pk (quotient (assoc-ref (gc-stats) (quote heap-size)) 1024))'

;;; (7204)
38272
--8<---------------cut here---------------end--------------->8---

So what we’re seeing here is probably the mappings made by the loader
(libguile/loader.c).

Needs more investigation…

Ludo’.

Re: guix --container is RAM hungry

[Maxim Cournoyer]

Hi Ludovic,

Ludovic Courtès <ludo@gnu.org> writes:

> Hi Edouard,
>
> Edouard Klein <edou@rdklein.fr> skribis:
>
>> I'm a huge fan of guix --container, and I created a system to use those
>> by default for network services. But the VPS these services run on has
>> only 2GB of RAM, and I just realized that a container, by default,
>> requires at least 200MB.
>
> Ouch, confirmed:
>
> $ \time -v guix shell -C coreutils -- uname 2>&1 |grep 'Maximum resident'
>         Maximum resident set size (kbytes): 283048
> $ \time -v guix shell coreutils -- uname 2>&1 |grep 'Maximum resident'
>         Maximum resident set size (kbytes): 56588
> $ guix describe
> Generation 297  Mar 24 2024 23:12:25    (current)
>   guix 28bc0e8
>     repository URL: https://git.savannah.gnu.org/git/guix.git
>     branch: master
>     commit: 28bc0e870b4d48b8e3e773382bb0e999df2e3611
>
>
> As raingloom and Ricardo wrote, there’s a Guile process that keeps
> waiting.

Is there a technical reason for this?  Couldn't we replace the
current Guix process with 'exec', as hinted by Edouard?

If possible, that'd be the most direct way to avoid any of the memory
cost incurred by Guile/Guix.

-- 
Thanks,
Maxim

Re: guix --container is RAM hungry

[Edouard Klein]

Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:

> Hi Ludovic,
>
> Ludovic Courtès <ludo@gnu.org> writes:
>
>> Hi Edouard,
>>
>> Edouard Klein <edou@rdklein.fr> skribis:
>>
>>> I'm a huge fan of guix --container, and I created a system to use those
>>> by default for network services. But the VPS these services run on has
>>> only 2GB of RAM, and I just realized that a container, by default,
>>> requires at least 200MB.
>>
>> Ouch, confirmed:
>>
>> $ \time -v guix shell -C coreutils -- uname 2>&1 |grep 'Maximum resident'
>>         Maximum resident set size (kbytes): 283048
>> $ \time -v guix shell coreutils -- uname 2>&1 |grep 'Maximum resident'
>>         Maximum resident set size (kbytes): 56588
>> $ guix describe
>> Generation 297  Mar 24 2024 23:12:25    (current)
>>   guix 28bc0e8
>>     repository URL: https://git.savannah.gnu.org/git/guix.git
>>     branch: master
>>     commit: 28bc0e870b4d48b8e3e773382bb0e999df2e3611
>>
>>
>> As raingloom and Ricardo wrote, there’s a Guile process that keeps
>> waiting.
>
> Is there a technical reason for this?  Couldn't we replace the
> current Guix process with 'exec', as hinted by Edouard?
>

Raingloom did, I just reported the problem without investigating the
cause.

>
> If possible, that'd be the most direct way to avoid any of the memory
> cost incurred by Guile/Guix.

I stand ready to test any WIP patch, I'll take a look as well to see if
I can guess where to replace a fork with an exec. Ricardo's breakdown of
the calls will be helpful.

Thank you all for having taken the report into account.

Re: guix --container is RAM hungry

[Ludovic Courtès]

Hi,

Edouard Klein <edou@rdklein.fr> skribis:

> Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:
>
>> Hi Ludovic,
>>
>> Ludovic Courtès <ludo@gnu.org> writes:
>>
>>> Hi Edouard,
>>>
>>> Edouard Klein <edou@rdklein.fr> skribis:
>>>
>>>> I'm a huge fan of guix --container, and I created a system to use those
>>>> by default for network services. But the VPS these services run on has
>>>> only 2GB of RAM, and I just realized that a container, by default,
>>>> requires at least 200MB.
>>>
>>> Ouch, confirmed:
>>>
>>> $ \time -v guix shell -C coreutils -- uname 2>&1 |grep 'Maximum resident'
>>>         Maximum resident set size (kbytes): 283048
>>> $ \time -v guix shell coreutils -- uname 2>&1 |grep 'Maximum resident'
>>>         Maximum resident set size (kbytes): 56588
>>> $ guix describe
>>> Generation 297  Mar 24 2024 23:12:25    (current)
>>>   guix 28bc0e8
>>>     repository URL: https://git.savannah.gnu.org/git/guix.git
>>>     branch: master
>>>     commit: 28bc0e870b4d48b8e3e773382bb0e999df2e3611
>>>
>>>
>>> As raingloom and Ricardo wrote, there’s a Guile process that keeps
>>> waiting.
>>
>> Is there a technical reason for this?  Couldn't we replace the
>> current Guix process with 'exec', as hinted by Edouard?
>>
>
> Raingloom did, I just reported the problem without investigating the
> cause.

Did what?

I don’t think execing is possible in this case, unless we let the user’s
process run as PID 1 inside its namespace (which isn’t recommended).
See #:child-is-pid1? in ‘call-with-container’.

I submitted a patch set to help a bit with that and related issues:

  https://issues.guix.gnu.org/70132

(I shouldn’t have said in the cover letter that it’s a boring patch
series: I got zero comments.  :-))

Thanks,
Ludo’.