RFC: subcommand to pause/resume builds

RFC: subcommand to pause/resume builds

[John Soo]

Hi Guix!

I was looking to pause a long build today and asked on IRC how to
accomplish pause/resume.  It seems this is possible already with the
following:

kill --signal SIGSTOP|SIGCONT {pids-of-build-process-tree}

There is already a command to list the processes associated to guix
commands: guix processes.  Perhaps pause/resume can be a subcommand or
set of flags to guix processes. The following is the first thing that
comes to mind:

guix processes --pause package-name ... --resume package-name ...

What do you think?

Thanks!

- John

Re: RFC: subcommand to pause/resume builds

[Ludovic Courtès]

Hi,

John Soo <jsoo1@asu.edu> skribis:

> I was looking to pause a long build today and asked on IRC how to
> accomplish pause/resume.  It seems this is possible already with the
> following:
>
> kill --signal SIGSTOP|SIGCONT {pids-of-build-process-tree}
>
> There is already a command to list the processes associated to guix
> commands: guix processes.  Perhaps pause/resume can be a subcommand or
> set of flags to guix processes. The following is the first thing that
> comes to mind:
>
> guix processes --pause package-name ... --resume package-name ...
>
> What do you think?

First, note that the daemon is unaware of “packages”, it only knows
about “derivations”.

Second, ‘guix processes’ is nice but it uses low-level heuristics to
determine what daemon sessions are open, what their clients are, and
what they’re building; it resorts to heuristics because the daemon as it
stands doesn’t have a way to communicate its current state.  It works
well in practice, but still I wouldn’t go too far building around it.

Last, you’d need to send SIGTSTP to the whole process group of the
build, like so (I think, haven’t tried):

  sudo kill -TSTP -123

where 123 is the “SessionPID” shown by ‘guix processes’.  However, doing
so may affect build results: processes in the build environment might
handle SIGTSTP specially, which can have side effects.  It’s an
observable action.

Conclusion: I don’t think we can implement this reliably.

HTH!

Ludo’.

Re: RFC: subcommand to pause/resume builds

[John Soo]

Hello!

I want to preface all this by saying this is not a huge priority to me.
pause/resume would just be a nice quality of life improvement.

Ludovic Courtès <ludo@gnu.org> writes:

> First, note that the daemon is unaware of “packages”, it only knows
> about “derivations”.

Agreed. I was thinking mostly about the best interface for a user. But
if derivations are too low-level I can see not being able to use the
package name.

> Second, ‘guix processes’ is nice but it uses low-level heuristics to
> determine what daemon sessions are open, what their clients are, and
> what they’re building; it resorts to heuristics because the daemon as it
> stands doesn’t have a way to communicate its current state.  It works
> well in practice, but still I wouldn’t go too far building around it.
>
> Last, you’d need to send SIGTSTP to the whole process group of the
> build, like so (I think, haven’t tried):
>
>   sudo kill -TSTP -123

Ah right, I got that advice on IRC also but I don't actually know how to
do it. Thanks!

> where 123 is the “SessionPID” shown by ‘guix processes’.  However, doing
> so may affect build results: processes in the build environment might
> handle SIGTSTP specially, which can have side effects.  It’s an
> observable action.

I thought of the side effects after sending the email. Makes sense to
me. Does that mean it is not worth including?  Given that the source and
state of a build do not change while paused, if a build tool changes its
outputs after paused/resumed is it still reproducible?

The other part that would make it difficult is that maybe different
build-systems respond differently to different signals. A simpler to
implement alternative might just be to provide a way to send an
arbitrary signal to the process tree. Given some builds might respond
un-reproducibly to it, a warning could be provided.

In any case, such a subcommand not a high priority to me as a user,
though sometimes I long to pause guix build ungoogled-chromium.

Thanks again,

John

Re: RFC: subcommand to pause/resume builds

[Tobias Geerinckx-Rice]

[-- Attachment #1: Type: text/plain, Size: 1137 bytes --]

Ludo',

Ludovic Courtès 写道：
> First, note that the daemon is unaware of “packages”, it only 
> knows
> about “derivations”.

Derivations have a (file) name, which can be matched with a regex 
allowing one to, say, ‘pause libreoffice’.  It works in practice. 
I do this often & it's *extremely* convenient.

Sometimes even necessary, because I have a habit of starting too 
many builds ;-)  It's nice not to lose 6h of work.

Not every handy hack needs to be upstreamed though.  ‘It's fugly’ 
is a strong argument in Guixland, and I like that.

However, this is FUD:

> Last, you’d need to send SIGTSTP to the whole process group of 
> the
> build, like so (I think, haven’t tried):
>
>   sudo kill -TSTP -123
>
> where 123 is the “SessionPID” shown by ‘guix processes’. 
> However, doing
> so may affect build results: processes in the build environment 
> might
> handle SIGTSTP specially, which can have side effects.  It’s an
> observable action.

Kind regards,

T G-R, now thinking about acronyms like ‘CRIU’ and what a 
next-level hack would look like...

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 247 bytes --]

Re: RFC: subcommand to pause/resume builds

[John Soo]

Hello Tobias :),

Tobias Geerinckx-Rice <me@tobias.gr> writes:

> Ludo',
>
> Ludovic Courtès 写道：
>> First, note that the daemon is unaware of “packages”, it only knows
>> about “derivations”.
>
> Derivations have a (file) name, which can be matched with a regex
> allowing one to, say, ‘pause libreoffice’.  It works in practice.
> I do this often & it's *extremely* convenient.

This feels close to little sed/awk pipelines.  Which is not to be
entirely dismissive. I like the compositionality of these tools.  In
fact I mentioned earlier that it might be good to send arbitrary
signals. But why not let kill (shell or scheme) do that?  All we would
need is to filter and format pids in a composable way (on the scheme
side and the shell side). That has the benefits of remaining agnostic on
side effects in builds (let the user decide what they are comfortable
with) and being more composable.

Maybe flags like this would be enough:

guix processes --session=<derivation-regex> ...

to get something like

5555
1212
343434
...

> Sometimes even necessary, because I have a habit of starting too many
> builds ;-)  It's nice not to lose 6h of work.

Indeed. This already saved me hours after learning it yesterday.

> Not every handy hack needs to be upstreamed though.  ‘It's fugly’ is a
> strong argument in Guixland, and I like that.

Same.

> T G-R, now thinking about acronyms like ‘CRIU’ and what a next-level
> hack would look like...

Yeah seems like functional build systems would ideally be
order-independent, pause/resumable, and the build steps time-travelable
themselves. But that's a cool aside for now.

Thanks for the tips!

John

Re: RFC: subcommand to pause/resume builds

[Tobias Geerinckx-Rice]

[-- Attachment #1: Type: text/plain, Size: 831 bytes --]

After playing around with the daemon I agree that it plays a 
little too loose with processes to make this upstream material ATM 
(groups? shepherd? *shrug*).

To close:

Tobias Geerinckx-Rice 写道：
> However, this is FUD:
>
>> Last, you’d need to send SIGTSTP to the whole process group of 
>> the
>> build, like so (I think, haven’t tried):

Re-reading this I wonder if you misread SIGSTOP as SIGTSTP?  The 
two are not related.  SIGSTOP cannot be handled or blocked.

It's possible to *detect*, like one can detect almost anything 
because, but you'd be going out of your way to do so.  I still 
think the concern is oddly specific & theoretical.

Build processes *could* be using rowhammer to deliberately crash 
if you're browsing Reddit, too, and who could blame them.

Kind regards,

T G-R

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 247 bytes --]

Re: RFC: subcommand to pause/resume builds

[John Soo]

After further review, I realize that guix processes already formats
for recutils. I never think to reach for that tool, but it seems
good.

Re: RFC: subcommand to pause/resume builds

[John Soo]

After even further investigation,

Does guix processes output the desired rec format? It seems hard to
select the child process pid:

ChildProcess: 16923: guile --no-auto-compile -L /gnu/store/8a0wry8cvr405ha8d8bpjyzj5dzghigd-module-import /gnu/store/mh1fkn1d9c9mg6hihxvjngxmn3qjmp38-ungoogled-chromium-86.0.4240.111-0.c34a56d-guile-builder
ChildProcess: 31859: /gnu/store/ah16zr8mmfkqy23rr7jy5a842ca1q9h1-guile-3.0.4/bin/guile \ /gnu/store/xi3jc6476vyv2vmcsfa1xkpqbxp1apk6-guix-1.1.0-27.1c21468/bin/.guix-real substitute --query
ChildProcess: 31869: /gnu/store/ah16zr8mmfkqy23rr7jy5a842ca1q9h1-guile-3.0.4/bin/guile \ /gnu/store/xi3jc6476vyv2vmcsfa1xkpqbxp1apk6-guix-1.1.0-27.1c21468/bin/.guix-real offload x86_64-linux 0 1 0

After perusing info recutils some, I expected the output to be more
like:

ChildProcessPID: 16923
ChildProcessCommand: guile --no-auto-compile -L /gnu/store/8a0wry8cvr405ha8d8bpjyzj5dzghigd-module-import /gnu/store/mh1fkn1d9c9mg6hihxvjngxmn3qjmp38-ungoogled-chromium-86.0.4240.111-0.c34a56d-guile-builder

ChildProcessPID: 31859
ChildProcessCommand: /gnu/store/ah16zr8mmfkqy23rr7jy5a842ca1q9h1-guile-3.0.4/bin/guile \ /gnu/store/xi3jc6476vyv2vmcsfa1xkpqbxp1apk6-guix-1.1.0-27.1c21468/bin/.guix-real substitute --query

ChildProcessPID: 31869
ChildProcessCommand: /gnu/store/ah16zr8mmfkqy23rr7jy5a842ca1q9h1-guile-3.0.4/bin/guile \ /gnu/store/xi3jc6476vyv2vmcsfa1xkpqbxp1apk6-guix-1.1.0-27.1c21468/bin/.guix-real offload x86_64-linux 0 1 0

But even that does not capture that the child processes all belong to
the session.

I must be missing something about the rec format or querying.

Any clues?

Thanks,

John

Re: RFC: subcommand to pause/resume builds

[Bengt Richter]

Hi all,

On +2020-11-03 14:53:07 +0100, Ludovic Courtès wrote:
> Hi,
> 
> John Soo <jsoo1@asu.edu> skribis:
> 
> > I was looking to pause a long build today and asked on IRC how to
> > accomplish pause/resume.  It seems this is possible already with the
> > following:
> >
> > kill --signal SIGSTOP|SIGCONT {pids-of-build-process-tree}
> >
> > There is already a command to list the processes associated to guix
> > commands: guix processes.  Perhaps pause/resume can be a subcommand or
> > set of flags to guix processes. The following is the first thing that
> > comes to mind:
> >
> > guix processes --pause package-name ... --resume package-name ...
> >
> > What do you think?
> 
> First, note that the daemon is unaware of “packages”, it only knows
> about “derivations”.
>

What if you turned the problem inside out, and made the derivation
volunteer to be paused (at sensible places in its progress)?

I.e., if you defined part of a given derivation "hash-prefixed-derivation-file-name-in-question.drv"
to do a check for the existence of

    /var/guix/daemon-ctl/hash-prefixed-derivation-file-name-in-question.drv.sfx
    (.sfx appended to signify special effects :) 

ISTM that could open the door for some easy hacks, (and probably some dangers to watch for :)
E.g. a proof of concept might be just to sleep 6 seconds (say) and repeat sleep/check
until the file disappears.

IWG this should not change anything for non-volunteering derivations other than the load-relief
of not running the sleeping process(es).

Then the person wanting to pause the derivation "hash-prefixed-derivation-file-name-in-question.drv"
could do so simply by
    touch  /var/guix/daemon-ctl/hash-prefixed-derivation-file-name-in-question.drv.sfx
and deleting it when wanting to allow it to continue.

Later, if that works, .sfx files could have content, for as yet unimagined purposes ;)

[...]
> 
> Conclusion: I don’t think we can implement this reliably.
>

IDK from the outside, but inside-out, WDYT?

> HTH!
> 
> Ludo’.
> 

-- 
Regards,
Bengt Richter

RFC: subcommand to pause/resume builds

[John Soo]

[-- Attachment #1: Type: text/plain, Size: 384 bytes --]

     
 

 
 
 
Hello Guix, 

 
I just sent a patch to normalize the output of processes with bug number 44460. The allows me to compose recutils with kill to get the desired effect of pausing all process trees for the things I want without any convoluted implementation.    I think I would be satisfied with such a patch and nothing else.
 

 
Thanks again!
 

 
John
 
 
 
 

 
     

[-- Attachment #2: Type: text/html, Size: 557 bytes --]

Re: RFC: subcommand to pause/resume builds

[Ludovic Courtès]

Hi,

John Soo <jsoo1@asu.edu> skribis:

> This feels close to little sed/awk pipelines.  Which is not to be
> entirely dismissive. I like the compositionality of these tools.  In
> fact I mentioned earlier that it might be good to send arbitrary
> signals. But why not let kill (shell or scheme) do that?  All we would
> need is to filter and format pids in a composable way (on the scheme
> side and the shell side). That has the benefits of remaining agnostic on
> side effects in builds (let the user decide what they are comfortable
> with) and being more composable.
>
> Maybe flags like this would be enough:
>
> guix processes --session=<derivation-regex> ...
>
> to get something like
>
> 5555
> 1212
> 343434
> ...

You can filter by piping ‘guix processes’ output through ‘recsel’:

--8<---------------cut here---------------start------------->8---
$ sudo guix processes | recsel -p SessionPID,LockHeld -e 'LockHeld ~ "chromium"'
SessionPID: 31410
LockHeld: /gnu/store/kdsp1pjj6znaxzs3d0vfwdcddc436g7f-ungoogled-chromium-86.0.4240.183-0.b68e17f.lock

SessionPID: 3455
LockHeld: /gnu/store/bhy3c5damrpzx7hdp8bam1lk2rk7789r-ungoogled-chromium-86.0.4240.183-0.b68e17f.lock
--8<---------------cut here---------------end--------------->8---

HTH,
Ludo’.

Re: RFC: subcommand to pause/resume builds

[Ludovic Courtès]

Hi,

John Soo <jsoo1@asu.edu> skribis:

> After even further investigation,
>
> Does guix processes output the desired rec format? It seems hard to
> select the child process pid:
>
> ChildProcess: 16923: guile --no-auto-compile -L /gnu/store/8a0wry8cvr405ha8d8bpjyzj5dzghigd-module-import /gnu/store/mh1fkn1d9c9mg6hihxvjngxmn3qjmp38-ungoogled-chromium-86.0.4240.111-0.c34a56d-guile-builder
> ChildProcess: 31859: /gnu/store/ah16zr8mmfkqy23rr7jy5a842ca1q9h1-guile-3.0.4/bin/guile \ /gnu/store/xi3jc6476vyv2vmcsfa1xkpqbxp1apk6-guix-1.1.0-27.1c21468/bin/.guix-real substitute --query
> ChildProcess: 31869: /gnu/store/ah16zr8mmfkqy23rr7jy5a842ca1q9h1-guile-3.0.4/bin/guile \ /gnu/store/xi3jc6476vyv2vmcsfa1xkpqbxp1apk6-guix-1.1.0-27.1c21468/bin/.guix-real offload x86_64-linux 0 1 0
>
> After perusing info recutils some, I expected the output to be more
> like:
>
> ChildProcessPID: 16923
> ChildProcessCommand: guile --no-auto-compile -L /gnu/store/8a0wry8cvr405ha8d8bpjyzj5dzghigd-module-import /gnu/store/mh1fkn1d9c9mg6hihxvjngxmn3qjmp38-ungoogled-chromium-86.0.4240.111-0.c34a56d-guile-builder

Yes, that would be nicer.

However, note that if you fiddle with child processes, your warranty is
void!  :-)

You’d be interfering with build processes, thus potentially changing
their result in an uncontrolled way.

Ludo’.

Re: RFC: subcommand to pause/resume builds

[Mark H Weaver]

Hi Ludovic,

Ludovic Courtès <ludo@gnu.org> writes:

> Last, you’d need to send SIGTSTP to the whole process group of the
> build, like so (I think, haven’t tried):
>
>   sudo kill -TSTP -123
>
> where 123 is the “SessionPID” shown by ‘guix processes’.  However, doing
> so may affect build results: processes in the build environment might
> handle SIGTSTP specially, which can have side effects.  It’s an
> observable action.

What's the rationale for using SIGTSTP instead of SIGSTOP here?

FWIW, on a few occasions I've paused builds by sending SIGSTOP to the
relevant process group, and later SIGCONT, and it has worked for me.  As
I recall, I've done this while building rust, webkitgtk, and icecat.

However, I suspect that if I paused a build while running tests, the
test suite might ultimately fail due to a "timeout".

> Conclusion: I don’t think we can implement this reliably.

Agreed.

     Thanks,
       Mark

Re: RFC: subcommand to pause/resume builds

[John Soo]

Hello everyone,

Ludovic Courtès <ludo@gnu.org> writes:

>> After perusing info recutils some, I expected the output to be more
>> like:
>>
>> ChildProcessPID: 16923
>> ChildProcessCommand: guile --no-auto-compile -L
>> /gnu/store/8a0wry8cvr405ha8d8bpjyzj5dzghigd-module-import
>> /gnu/store/mh1fkn1d9c9mg6hihxvjngxmn3qjmp38-ungoogled-chromium-86.0.4240.111-0.c34a56d-guile-builder
>
> Yes, that would be nicer.

I sent in patch https://issues.guix.gnu.org/issue/44460 to address
it. With that I don't see any reason to consider a subcommand.


> However, note that if you fiddle with child processes, your warranty is
> void!  :-)

Definitely agreed.

> You’d be interfering with build processes, thus potentially changing
> their result in an uncontrolled way.

Like Mark said, I have seen tests potentially time out, and I have to agree.

I am done discussing a subcommand. I'll shift attention to #44460.

Thanks for listening to little old me :).

Have a nice weekend,

John

Re: RFC: subcommand to pause/resume builds

[Ludovic Courtès]

Hi Mark,

Mark H Weaver <mhw@netris.org> skribis:

> Ludovic Courtès <ludo@gnu.org> writes:
>
>> Last, you’d need to send SIGTSTP to the whole process group of the
>> build, like so (I think, haven’t tried):
>>
>>   sudo kill -TSTP -123
>>
>> where 123 is the “SessionPID” shown by ‘guix processes’.  However, doing
>> so may affect build results: processes in the build environment might
>> handle SIGTSTP specially, which can have side effects.  It’s an
>> observable action.
>
> What's the rationale for using SIGTSTP instead of SIGSTOP here?

No good reason; SIGSTOP is safer because it cannot be handled (info
"(libc) Job Control Signals"), and thus my argument above is moot.

Thanks for the correction!

Ludo’.