unofficial mirror of bug-guix@gnu.org 
 help / color / mirror / code / Atom feed
* bug#30948: [PATCH core-updates] guix: Reap finished child processes in build containers.
       [not found]         ` <874lkxoanq.fsf@gnu.org>
@ 2022-11-24 16:44           ` Maxim Cournoyer
  2022-11-26 15:11             ` Ludovic Courtès
  0 siblings, 1 reply; 6+ messages in thread
From: Maxim Cournoyer @ 2022-11-24 16:44 UTC (permalink / raw)
  To: Ludovic Courtès; +Cc: 30948, Carlo Zancanaro

Hi,

ludo@gnu.org (Ludovic Courtès) writes:

> Hello,
>
> Carlo Zancanaro <carlo@zancanaro.id.au> skribis:
>
>> On Fri, Mar 30 2018, Ludovic Courtès wrote:
>>>> From what I can understand it's one of pid 1's responsiblities to
>>>> reap child processes, so I would expect this to be set up for every
>>>> builder, before the builder is run.
>>>
>>> True, but for derivations it’s also “optional” because eventually
>>> guix-daemon terminates all its child processes.
>>
>> As long as the build process doesn't rely on behaviour that, strictly
>> speaking, it should be allowed to rely on. It's not an issue of
>> resource leaking, it's an issue of correctness.
>
> Right.
>
>>>> Given it's not specific to the gnu-build-system, I don't think it
>>>> really fits there.

For what it's worth, I agree.  The evaluation container should have the
correct signal handling configured for *any* code about to be evaluated,
not just when on demand, if we want to fix this fully in a way that
won't come back to haunt us in some edge case.

>>> Yes, but note that it would be inherited by all the build systems.
>>
>> Except for trivial-build-system, which is probably fine. I still don't
>> think it fits in a specific build system, given it's a behaviour that
>> transcends the specific action happening within the container.
>>
>> Putting it in gnu-build-system will solve the problem in all realistic
>> cases, so that's probably fine. It's still subtly incorrect, but will
>> only be a problem if something using the trivial build system relies
>> on pid 1 to reap a process, or if we make a new build system not
>> deriving from gnu-build-system (which seems unlikely, but not
>> impossible).
>
> I agree, every Guile process running as PID 1 should reap processes.

Agreed too.

> My view is just that this mechanism belongs in “user code”, not in the
> low-level mechanisms such as ‘build-expression->derivation’ and
> ‘gexp->derivation’.  It’s a matter of separation of concerns.

Why?  On my Guix System, such signal handling is handled by Shepherd, if
I'm not mistaken.  As I user, I can trust the foundation to be sane,
rather than having to provide the bits to make it so myself.

> Of course we don’t want to duplicate that code every time, but the way
> we should factorize it, IMO, is by putting it in a “normal” module that
> people will use.
>
> Putting it in gnu-build-system is an admittedly hacky but easy way to
> have it widely shared.

I think we can do better than hacky here :-)

-- 
Thanks,
Maxim




^ permalink raw reply	[flat|nested] 6+ messages in thread

* bug#30948: [PATCH core-updates] guix: Reap finished child processes in build containers.
  2022-11-24 16:44           ` bug#30948: [PATCH core-updates] guix: Reap finished child processes in build containers Maxim Cournoyer
@ 2022-11-26 15:11             ` Ludovic Courtès
  2022-11-27  3:00               ` Maxim Cournoyer
  0 siblings, 1 reply; 6+ messages in thread
From: Ludovic Courtès @ 2022-11-26 15:11 UTC (permalink / raw)
  To: Maxim Cournoyer; +Cc: 30948, Carlo Zancanaro

Hi,

Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:

>> My view is just that this mechanism belongs in “user code”, not in the
>> low-level mechanisms such as ‘build-expression->derivation’ and
>> ‘gexp->derivation’.  It’s a matter of separation of concerns.
>
> Why?  On my Guix System, such signal handling is handled by Shepherd, if
> I'm not mistaken.  As I user, I can trust the foundation to be sane,
> rather than having to provide the bits to make it so myself.
>
>> Of course we don’t want to duplicate that code every time, but the way
>> we should factorize it, IMO, is by putting it in a “normal” module that
>> people will use.
>>
>> Putting it in gnu-build-system is an admittedly hacky but easy way to
>> have it widely shared.
>
> I think we can do better than hacky here :-)

I think the real issue here is semantic clarity when it comes to
derivation inputs.

If I write:

  (gexp->derivation "foo" #~(mkdir #$output))

I can be sure that my derivation depends on nothing but (default-guile).
This is important for tests, but also to make sure we can use this
primitive everywhere—if it pulled in the Shepherd, I wouldn’t be able to
use to build glibc, because there’d be a cycle.

In that sense, having child-reaping code in gnu-build-system.scm, just
like in (guix least-authority), doesn’t seem unreasonable to me.

That said, I’m open to other proposals so please unleash your
creativity!  :-)

We’re touching core components though so this will require discussion.

Ludo’.




^ permalink raw reply	[flat|nested] 6+ messages in thread

* bug#30948: [PATCH core-updates] guix: Reap finished child processes in build containers.
  2022-11-26 15:11             ` Ludovic Courtès
@ 2022-11-27  3:00               ` Maxim Cournoyer
  2022-11-28 15:04                 ` Ludovic Courtès
  0 siblings, 1 reply; 6+ messages in thread
From: Maxim Cournoyer @ 2022-11-27  3:00 UTC (permalink / raw)
  To: Ludovic Courtès; +Cc: 30948, Carlo Zancanaro

Hi,

Ludovic Courtès <ludo@gnu.org> writes:

> Hi,
>
> Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:
>
>>> My view is just that this mechanism belongs in “user code”, not in the
>>> low-level mechanisms such as ‘build-expression->derivation’ and
>>> ‘gexp->derivation’.  It’s a matter of separation of concerns.
>>
>> Why?  On my Guix System, such signal handling is handled by Shepherd, if
>> I'm not mistaken.  As I user, I can trust the foundation to be sane,
>> rather than having to provide the bits to make it so myself.
>>
>>> Of course we don’t want to duplicate that code every time, but the way
>>> we should factorize it, IMO, is by putting it in a “normal” module that
>>> people will use.
>>>
>>> Putting it in gnu-build-system is an admittedly hacky but easy way to
>>> have it widely shared.
>>
>> I think we can do better than hacky here :-)
>
> I think the real issue here is semantic clarity when it comes to
> derivation inputs.
>
> If I write:
>
>   (gexp->derivation "foo" #~(mkdir #$output))
>
> I can be sure that my derivation depends on nothing but (default-guile).
> This is important for tests, but also to make sure we can use this
> primitive everywhere—if it pulled in the Shepherd, I wouldn’t be able to
> use to build glibc, because there’d be a cycle.

I was not suggesting to pull in extra dependencies such as Shepherd, but
to weave the to-be-added signal handling logic at a much lower level.
One idea could be to arrange so that the correct signal handlers always
get installed for any Guile code running in the build side (I'm not sure
how, but perhaps by adjusting the gexp "compiler"?).

The handlers could be defined in (guix build signal-handling) or
similar.  Users wouldn't need to explicitly import the module and
install its signal handlers, that'd be taken care of automatically, all
the time.

Does that sound feasible?

-- 
Thanks,
Maxim




^ permalink raw reply	[flat|nested] 6+ messages in thread

* bug#30948: [PATCH core-updates] guix: Reap finished child processes in build containers.
  2022-11-27  3:00               ` Maxim Cournoyer
@ 2022-11-28 15:04                 ` Ludovic Courtès
  2022-11-28 20:10                   ` Maxim Cournoyer
  2022-11-29  2:07                   ` Maxim Cournoyer
  0 siblings, 2 replies; 6+ messages in thread
From: Ludovic Courtès @ 2022-11-28 15:04 UTC (permalink / raw)
  To: Maxim Cournoyer; +Cc: 30948, Carlo Zancanaro

Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:

> Ludovic Courtès <ludo@gnu.org> writes:

[...]

>> If I write:
>>
>>   (gexp->derivation "foo" #~(mkdir #$output))
>>
>> I can be sure that my derivation depends on nothing but (default-guile).
>> This is important for tests, but also to make sure we can use this
>> primitive everywhere—if it pulled in the Shepherd, I wouldn’t be able to
>> use to build glibc, because there’d be a cycle.
>
> I was not suggesting to pull in extra dependencies such as Shepherd, but
> to weave the to-be-added signal handling logic at a much lower level.
> One idea could be to arrange so that the correct signal handlers always
> get installed for any Guile code running in the build side (I'm not sure
> how, but perhaps by adjusting the gexp "compiler"?).
>
> The handlers could be defined in (guix build signal-handling) or
> similar.  Users wouldn't need to explicitly import the module and
> install its signal handlers, that'd be taken care of automatically, all
> the time.
>
> Does that sound feasible?

Not like this: the imported-modules derivation for (guix build
signal-handling) would be a dependency in themselves.

Ludo’.




^ permalink raw reply	[flat|nested] 6+ messages in thread

* bug#30948: [PATCH core-updates] guix: Reap finished child processes in build containers.
  2022-11-28 15:04                 ` Ludovic Courtès
@ 2022-11-28 20:10                   ` Maxim Cournoyer
  2022-11-29  2:07                   ` Maxim Cournoyer
  1 sibling, 0 replies; 6+ messages in thread
From: Maxim Cournoyer @ 2022-11-28 20:10 UTC (permalink / raw)
  To: Ludovic Courtès; +Cc: 30948, Carlo Zancanaro

Hi,

Ludovic Courtès <ludo@gnu.org> writes:

> Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:
>
>> Ludovic Courtès <ludo@gnu.org> writes:
>
> [...]
>
>>> If I write:
>>>
>>>   (gexp->derivation "foo" #~(mkdir #$output))
>>>
>>> I can be sure that my derivation depends on nothing but (default-guile).
>>> This is important for tests, but also to make sure we can use this
>>> primitive everywhere—if it pulled in the Shepherd, I wouldn’t be able to
>>> use to build glibc, because there’d be a cycle.
>>
>> I was not suggesting to pull in extra dependencies such as Shepherd, but
>> to weave the to-be-added signal handling logic at a much lower level.
>> One idea could be to arrange so that the correct signal handlers always
>> get installed for any Guile code running in the build side (I'm not sure
>> how, but perhaps by adjusting the gexp "compiler"?).
>>
>> The handlers could be defined in (guix build signal-handling) or
>> similar.  Users wouldn't need to explicitly import the module and
>> install its signal handlers, that'd be taken care of automatically, all
>> the time.
>>
>> Does that sound feasible?
>
> Not like this: the imported-modules derivation for (guix build
> signal-handling) would be a dependency in themselves.

Can we make it an implicit dependency, since we want it to *always* be
used?

It'd be useless/annoying boilerplate otherwise.

-- 
Thanks,
Maxim




^ permalink raw reply	[flat|nested] 6+ messages in thread

* bug#30948: [PATCH core-updates] guix: Reap finished child processes in build containers.
  2022-11-28 15:04                 ` Ludovic Courtès
  2022-11-28 20:10                   ` Maxim Cournoyer
@ 2022-11-29  2:07                   ` Maxim Cournoyer
  1 sibling, 0 replies; 6+ messages in thread
From: Maxim Cournoyer @ 2022-11-29  2:07 UTC (permalink / raw)
  To: Ludovic Courtès; +Cc: 30948, Carlo Zancanaro

Hi,

Ludovic Courtès <ludo@gnu.org> writes:

> Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:
>
>> Ludovic Courtès <ludo@gnu.org> writes:
>
> [...]
>
>>> If I write:
>>>
>>>   (gexp->derivation "foo" #~(mkdir #$output))
>>>
>>> I can be sure that my derivation depends on nothing but (default-guile).
>>> This is important for tests, but also to make sure we can use this
>>> primitive everywhere—if it pulled in the Shepherd, I wouldn’t be able to
>>> use to build glibc, because there’d be a cycle.
>>
>> I was not suggesting to pull in extra dependencies such as Shepherd, but
>> to weave the to-be-added signal handling logic at a much lower level.
>> One idea could be to arrange so that the correct signal handlers always
>> get installed for any Guile code running in the build side (I'm not sure
>> how, but perhaps by adjusting the gexp "compiler"?).
>>
>> The handlers could be defined in (guix build signal-handling) or
>> similar.  Users wouldn't need to explicitly import the module and
>> install its signal handlers, that'd be taken care of automatically, all
>> the time.
>>
>> Does that sound feasible?
>
> Not like this: the imported-modules derivation for (guix build
> signal-handling) would be a dependency in themselves.

I see a couple of options for the lowest place to inject the minimal
signal handling of a PID.

1. In Guile itself.  We could make it detect when it's running as PID 1
and then set up the required signal handling.  This is apparently what
Bash does, a peculiarity exploited by NixOS (they launch their builder
scripts via Bash, which is PID 1 and takes care of reaping the dead
processes)

2. In a Guile wrapper.  Instead of running Guile directly in the
container, guix-daemon would run it through a wrapper that acts as PID 1.
This would make it a tool comparable to dumb-init [0] or tini [1],
except written in Scheme.

[0] https://github.com/Yelp/dumb-init/
[1] https://github.com/krallin/tini

If we implement 1, it'd make Guile potentially useful as a wrapper
itself to launch scripts in containerized environment (the same as
tini), and it alleviates any integration overhead for us, so I find it
attractive.

What do you think?

For further reading, see [2], which I found interesting.

[2]  https://medium.com/hackernoon/my-process-became-pid-1-and-now-signals-behave-strangely-b05c52cc551c

-- 
Thanks,
Maxim




^ permalink raw reply	[flat|nested] 6+ messages in thread

end of thread, other threads:[~2022-11-29  2:08 UTC | newest]

Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
     [not found] <87muyvulwt.fsf@zancanaro.id.au>
     [not found] ` <87bmf6ve6u.fsf@gnu.org>
     [not found]   ` <87sh8id1mg.fsf@zancanaro.id.au>
     [not found]     ` <87vadeou54.fsf@gnu.org>
     [not found]       ` <87o9j5x1d4.fsf@zancanaro.id.au>
     [not found]         ` <874lkxoanq.fsf@gnu.org>
2022-11-24 16:44           ` bug#30948: [PATCH core-updates] guix: Reap finished child processes in build containers Maxim Cournoyer
2022-11-26 15:11             ` Ludovic Courtès
2022-11-27  3:00               ` Maxim Cournoyer
2022-11-28 15:04                 ` Ludovic Courtès
2022-11-28 20:10                   ` Maxim Cournoyer
2022-11-29  2:07                   ` Maxim Cournoyer

Code repositories for project(s) associated with this public inbox

	https://git.savannah.gnu.org/cgit/guix.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).