Default autogroup niceness of Guix build daemon

Default autogroup niceness of Guix build daemon

[J. R. Haigh (re. Guix)]

Hi all,
	I've been using Guix on Debian 9 Stretch for a few months, now. Besides various build failures, one of the worst issues is hogging of the CPU, making the rest of my system almost unusable for several hours (or sometimes even days), depending on the builds. If builds don't finish over night then I have had to SIGSTOP the builds in order to use my laptop, and SIGCONT them when idle again.
	Today, however, I've found a partial solution, after realising why setting process niceness to 19 does not fix the problem on my system; instead of the process niceness, changing the /autogroup/ niceness to 19 by issuïng the following command:

$ (for P in $(ps --group="guixbuild" --format="pid="); do F="/proc/$P/autogroup"; echo "$F"; cat "$F" && sudo tee "$F" <<<19; done)
/proc/30487/autogroup
/autogroup-10675 nice 0
19
/proc/30537/autogroup
/autogroup-10679 nice 0
19

…completely removes the lag caused by any processes in the ‘guixbuild’ usergroup, and allows my system to be usable for interactive work simultaneously with the build!!! This is great, but it only lasts as long as the build, and subsequent builds default to autogroup niceness 0, which is really not very nice!! I'd rather not run the above command in a loop, hence the question…
	Is there a way to declaratively set the default autogroup niceness of Guix's build daemon?

Regards,
James R. Haigh.
P.s.: Although I've been on this list for just under a year, this is my first email to the list – hi everyone! Maybe see you again at FOSDEM 2020.
-- 
6 days, 10 hours, and 30 minutes left until FOSDEM 2020 (Saturday)!
7 days, 10 hours, and 30 minutes left until FOSDEM 2020 (Sunday)!
Wealth doesn't bring happiness, but poverty brings sadness.
https://wiki.FSFE.org/Fellows/JRHaigh
Sent from Debian with Claws Mail, using email subaddressing as an alternative to error-prone heuristical spam filtering.

Re: Default autogroup niceness of Guix build daemon

[Giovanni Biscuolo]

[-- Attachment #1: Type: text/plain, Size: 778 bytes --]

Hi James,

"J. R. Haigh (re. Guix)" <JRHaigh+ML.GNU.Guix@Runbox.com> writes:

> Hi all,
> 	I've been using Guix on Debian 9 Stretch

[...]

> 	Is there a way to declaratively set the default autogroup
> 	niceness of Guix's build daemon?

Since Debian 9 users systemd, should be possible by configuring a limit
in the systemd service unit file [1]; I've never tried but try adding
"LimitNICE=19" in the [Service] stanza

Documentation on that parameter here:
https://www.freedesktop.org/software/systemd/man/systemd.exec.html#Process%20Properties

Remember to "systemctl daemon-reload" after editing one the systemd
service unit file

HTH! Gio'

[1] /etc/systemd/system/guix-daemon.service

[...]

-- 
Giovanni Biscuolo

Xelera IT Infrastructures

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Default autogroup niceness of Guix build daemon

[J. R. Haigh (re. Guix)]

Hi Giovanni,

At 2020-01-27Mon18:37:53+01, Giovanni Biscuolo sent:
> […]
> Since Debian 9 [uses] systemd, should be possible by configuring a limit in the systemd service unit file [1]; I've never tried but try adding "LimitNICE=19" in the [Service] stanza.

Thanks for the suggestion. I tried it and it seemed to only affect process niceness, not autogroup niceness. Htop reported process niceness values of 19 on the relevant processes, and I already knew that setting these manually from Htop does not fix the lag. Autogroup niceness still defaults to 0:

$ (for P in $(ps --group="guixbuild" --format="pid="); do F="/proc/$P/autogroup"; echo "$F"; cat "$F"; done)
/proc/26741/autogroup
/autogroup-12141 nice 0

> Documentation on that parameter here:
> https://www.freedesktop.org/software/systemd/man/systemd.exec.html#Process%20Properties

Thank you. There does not seem to be anything on autogroup niceness there, which is apparently deliberate, unfortunately. I also tried ‘LimitNICE=+19’ and ‘CPUSchedulingPolicy=idle’ but neither prevented Guix builds from hogging the CPU. As well as reloading the SystemD configuration, I also tried interrupting the build and starting it again, but the lag returned as soon as the build got going again.
	As a temporary workaround, I tried disabling CFS's autogrouping feature with:

sudo tee /proc/sys/kernel/sched_autogroup_enabled <<<0

…but it did not seem to do anything, despite Htop reporting that the relevant processes have a process niceness of 19. It seems that autogrouping is still enabled, despite the attempt to disable it, because the command in my previous email (repeated below) to imperatively change the autogroup niceness (to 19 and then back to 0) continues to have unmistakable effect on interactivity. As all other attempts so far have had no noticeable effect, that command remains to be the only workaround so far:

$ (for P in $(ps --group="guixbuild" --format="pid="); do F="/proc/$P/autogroup"; echo "$F"; cat "$F" && sudo tee "$F" <<<19; done)
/proc/26741/autogroup
/autogroup-12141 nice 0
[sudo] password for JRHaigh: 
19
$ (for P in $(ps --group="guixbuild" --format="pid="); do F="/proc/$P/autogroup"; echo "$F"; cat "$F" && sudo tee "$F" <<<0; done)  ##
/proc/26741/autogroup
/autogroup-12141 nice 19
0
$ (for P in $(ps --group="guixbuild" --format="pid="); do F="/proc/$P/autogroup"; echo "$F"; cat "$F" && sudo tee "$F" <<<19; done)
/proc/26741/autogroup
/autogroup-12141 nice 0
19

These cammands very clearly made my system responsive, laggy, and then responsive again, which was totally unexpected seeing as I had supposedly disabled autogrouping. Given that there are reasons why autogrouping was added and has been default on most/many distros for many years, it would be best if the solution worked with autogrouping enabled anyway, so I don't really want to waste time investigating why autogrouping does not seem to get disabled.
	Are there any modifications that could be made to the Guix daemon to fix this? Seeing as SystemD apparently does not support autogroup niceness, perhaps the next best alternative is to fix the CPUSchedulingPolicy setting. I'm guessing that it only affects the Guix daemon and does not cascade to its builders. Fixing the CPUSchedulingPolicy setting might provide a decent, declarative solution for build deprioritisation that could be used instead of niceness.

Regards,
James R. Haigh.
-- 
4 days, 5 hours, and 4 minutes left until FOSDEM 2020 (Saturday)!
5 days, 5 hours, and 4 minutes left until FOSDEM 2020 (Sunday)!
Wealth doesn't bring happiness, but poverty brings sadness.
https://wiki.FSFE.org/Fellows/JRHaigh
Sent from Debian with Claws Mail, using email subaddressing as an alternative to error-prone heuristical spam filtering.