guix-daemon hitting systemd TasksMax

guix-daemon hitting systemd TasksMax

[Joram Schrijver]

Hi,

In version 228, systemd introduced a unit attribute `TasksMax`[1] that
limits the number of tasks that can run in a unit. It has effect on
kernel versions 4.3 and up, kernel threads and userspace process count
towards the limit, and the default value is 512.

Running `guix pull` without changing any settings crashed for me,
because building/testing texinfo hit this task limit. Increasing it to
1024 using `systemctl set-property guix-daemon.service TasksMax=1024`
fixed the problem.

Because I expect guix-daemon to hit this limit regularly, it might be
wise to set a higher `TasksMax` in the service file itself, perhaps even
`infinity`. Doing this would, sadly, cause older systemd versions
shipped by various distributions to complain about the existence of the
attribute, so perhaps a mention in the manual would instead be enough.


[1]:
https://www.freedesktop.org/software/systemd/man/systemd.resource-control.html#TasksAccounting=

--
  Joram

Re: guix-daemon hitting systemd TasksMax

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 1488 bytes --]

On Fri, Apr 15, 2016 at 01:37:32AM +0200, Joram Schrijver wrote:
> Hi,
> 
> In version 228, systemd introduced a unit attribute `TasksMax`[1] that
> limits the number of tasks that can run in a unit. It has effect on
> kernel versions 4.3 and up, kernel threads and userspace process count
> towards the limit, and the default value is 512.
> 
> Running `guix pull` without changing any settings crashed for me,
> because building/testing texinfo hit this task limit. Increasing it to
> 1024 using `systemctl set-property guix-daemon.service TasksMax=1024`
> fixed the problem.
> 
> Because I expect guix-daemon to hit this limit regularly, it might be
> wise to set a higher `TasksMax` in the service file itself, perhaps even
> `infinity`. Doing this would, sadly, cause older systemd versions
> shipped by various distributions to complain about the existence of the
> attribute, so perhaps a mention in the manual would instead be enough.
> 
> 
> [1]:
> https://www.freedesktop.org/software/systemd/man/systemd.resource-control.html#TasksAccounting=
> 
> --
>   Joram
> 

If it complains but doesn't cause it to error out it might be worth it
to add it to etc/guix-daemon.service in the guix source tree as well as
mention it in the manual.

-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: guix-daemon hitting systemd TasksMax

[Leo Famulari]

On Fri, Apr 15, 2016 at 01:37:32AM +0200, Joram Schrijver wrote:
> Hi,
> 
> In version 228, systemd introduced a unit attribute `TasksMax`[1] that
> limits the number of tasks that can run in a unit. It has effect on
> kernel versions 4.3 and up, kernel threads and userspace process count
> towards the limit, and the default value is 512.
> 
> Running `guix pull` without changing any settings crashed for me,
> because building/testing texinfo hit this task limit. Increasing it to
> 1024 using `systemctl set-property guix-daemon.service TasksMax=1024`
> fixed the problem.

Interesting. What operating system are you using? I'm running systemd
229 on Debian testing and I don't have this issue, nor have I altered
this setting.

It would be useful to know what the difference is between our systemd
installations.  I don't see anything related to TasksMax in
systemd-229's NEWS:

https://github.com/systemd/systemd/blob/master/NEWS

Re: guix-daemon hitting systemd TasksMax

[Joram Schrijver]

On Sat, Apr 16, 2016, at 03:29 AM, Leo Famulari wrote:
> On Fri, Apr 15, 2016 at 01:37:32AM +0200, Joram Schrijver wrote:
> > Hi,
> > 
> > In version 228, systemd introduced a unit attribute `TasksMax`[1] that
> > limits the number of tasks that can run in a unit. It has effect on
> > kernel versions 4.3 and up, kernel threads and userspace process count
> > towards the limit, and the default value is 512.
> > 
> > Running `guix pull` without changing any settings crashed for me,
> > because building/testing texinfo hit this task limit. Increasing it to
> > 1024 using `systemctl set-property guix-daemon.service TasksMax=1024`
> > fixed the problem.
> 
> Interesting. What operating system are you using? I'm running systemd
> 229 on Debian testing and I don't have this issue, nor have I altered
> this setting.
> 
> It would be useful to know what the difference is between our systemd
> installations.  I don't see anything related to TasksMax in
> systemd-229's NEWS:
> 
> https://github.com/systemd/systemd/blob/master/NEWS

I am running Arch Linux, also with systemd 229. It doesn't look like
Debian applies any patches to systemd, so our installations should
pretty much be the same.

Perhaps the limit is only reached during the build process of some
specific packages?

--
  Joram

Re: guix-daemon hitting systemd TasksMax

[Leo Famulari]

On Sat, Apr 16, 2016 at 03:12:15PM +0200, Joram Schrijver wrote:
> I am running Arch Linux, also with systemd 229. It doesn't look like
> Debian applies any patches to systemd, so our installations should
> pretty much be the same.
> 
> Perhaps the limit is only reached during the build process of some
> specific packages?

`guix pull` usually doesn't build any packages, but sometimes it does
need to. My (limited) understanding is that it sometimes needs to build
(or download) updated dependencies of Guix itself.

Do you think the number of "Tasks" would increase with more cores? I
only have 4 cores on my machines. Maybe if you had more, the builders
spawned by your guix-daemon would run more tasks as a consequence of
using all the cores.

I think there are other Guix people on Debian testing. I wonder if they
will hit this issue, too.

Re: guix-daemon hitting systemd TasksMax

[Ludovic Courtès]

Leo Famulari <leo@famulari.name> skribis:

> On Sat, Apr 16, 2016 at 03:12:15PM +0200, Joram Schrijver wrote:
>> I am running Arch Linux, also with systemd 229. It doesn't look like
>> Debian applies any patches to systemd, so our installations should
>> pretty much be the same.
>> 
>> Perhaps the limit is only reached during the build process of some
>> specific packages?
>
> `guix pull` usually doesn't build any packages, but sometimes it does
> need to. My (limited) understanding is that it sometimes needs to build
> (or download) updated dependencies of Guix itself.
>
> Do you think the number of "Tasks" would increase with more cores? I
> only have 4 cores on my machines. Maybe if you had more, the builders
> spawned by your guix-daemon would run more tasks as a consequence of
> using all the cores.

Right, ‘guix pull’ entails a build of Guix using as many cores as
possible, with one process per core (and I think each Guile has two
threads, one of which is the signal delivery thread.)

So, if the limit on the total number of threads + processes is 512 and
you have 256 cores or more (lucky you!), you may hit the limit.

Ludo’.

Re: guix-daemon hitting systemd TasksMax

[Leo Famulari]

On Sun, Apr 17, 2016 at 12:46:16AM +0200, Ludovic Courtès wrote:
> Leo Famulari <leo@famulari.name> skribis:
> 
> > On Sat, Apr 16, 2016 at 03:12:15PM +0200, Joram Schrijver wrote:
> >> I am running Arch Linux, also with systemd 229. It doesn't look like
> >> Debian applies any patches to systemd, so our installations should
> >> pretty much be the same.
> >> 
> >> Perhaps the limit is only reached during the build process of some
> >> specific packages?
> >
> > `guix pull` usually doesn't build any packages, but sometimes it does
> > need to. My (limited) understanding is that it sometimes needs to build
> > (or download) updated dependencies of Guix itself.
> >
> > Do you think the number of "Tasks" would increase with more cores? I
> > only have 4 cores on my machines. Maybe if you had more, the builders
> > spawned by your guix-daemon would run more tasks as a consequence of
> > using all the cores.
> 
> Right, ‘guix pull’ entails a build of Guix using as many cores as
> possible, with one process per core (and I think each Guile has two
> threads, one of which is the signal delivery thread.)
> 
> So, if the limit on the total number of threads + processes is 512 and
> you have 256 cores or more (lucky you!), you may hit the limit.

Joram, are you this lucky? ;)

Re: guix-daemon hitting systemd TasksMax

[Joram Schrijver]

Heh, no I'm not. I have to do it with 4 cores + hyperthreading. My first
`guix pull` did have to build many things though, spawning numerous
threads in some cases. I eye-sampled the thread counts during one try
and they were certainly in the hundreds.

I installed Guix from the AUR[1], so I was running the most recent
release. Substitutes were enabled, but almost everything had to be built
instead of downloaded.

[1]: https://aur.archlinux.org/packages/guix/

--
  Joram

On Mon, Apr 18, 2016, at 10:33 PM, Leo Famulari wrote:
> On Sun, Apr 17, 2016 at 12:46:16AM +0200, Ludovic Courtès wrote:
> > Leo Famulari <leo@famulari.name> skribis:
> > 
> > > On Sat, Apr 16, 2016 at 03:12:15PM +0200, Joram Schrijver wrote:
> > >> I am running Arch Linux, also with systemd 229. It doesn't look like
> > >> Debian applies any patches to systemd, so our installations should
> > >> pretty much be the same.
> > >> 
> > >> Perhaps the limit is only reached during the build process of some
> > >> specific packages?
> > >
> > > `guix pull` usually doesn't build any packages, but sometimes it does
> > > need to. My (limited) understanding is that it sometimes needs to build
> > > (or download) updated dependencies of Guix itself.
> > >
> > > Do you think the number of "Tasks" would increase with more cores? I
> > > only have 4 cores on my machines. Maybe if you had more, the builders
> > > spawned by your guix-daemon would run more tasks as a consequence of
> > > using all the cores.
> > 
> > Right, ‘guix pull’ entails a build of Guix using as many cores as
> > possible, with one process per core (and I think each Guile has two
> > threads, one of which is the signal delivery thread.)
> > 
> > So, if the limit on the total number of threads + processes is 512 and
> > you have 256 cores or more (lucky you!), you may hit the limit.
> 
> Joram, are you this lucky? ;)

Re: guix-daemon hitting systemd TasksMax

[Ludovic Courtès]

[-- Attachment #1: Type: text/plain, Size: 476 bytes --]

Joram Schrijver <i+guix-devel@joram.io> skribis:

> Heh, no I'm not. I have to do it with 4 cores + hyperthreading. My first
> `guix pull` did have to build many things though, spawning numerous
> threads in some cases. I eye-sampled the thread counts during one try
> and they were certainly in the hundreds.

Oh, then it could be that it’s a “make -j8” (assuming 2 hyperthreads per
core) for one of the packages that exceeded the limit.

What about this:


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Type: text/x-patch, Size: 366 bytes --]

diff --git a/etc/guix-daemon.service.in b/etc/guix-daemon.service.in
index 28ef000..c79f867 100644
--- a/etc/guix-daemon.service.in
+++ b/etc/guix-daemon.service.in
@@ -10,6 +10,7 @@ ExecStart=@bindir@/guix-daemon --build-users-group=guixbuild
 RemainAfterExit=yes
 StandardOutput=syslog
 StandardError=syslog
+TasksMax=1024
 
 [Install]
 WantedBy=multi-user.target

[-- Attachment #3: Type: text/plain, Size: 50 bytes --]


(Is it the right syntax?)

Thanks,
Ludo’.

Re: guix-daemon hitting systemd TasksMax

[Leo Famulari]

On Tue, Apr 19, 2016 at 05:57:05PM +0200, Ludovic Courtès wrote:
> What about this:
> 

> diff --git a/etc/guix-daemon.service.in b/etc/guix-daemon.service.in
> index 28ef000..c79f867 100644
> --- a/etc/guix-daemon.service.in
> +++ b/etc/guix-daemon.service.in
> @@ -10,6 +10,7 @@ ExecStart=@bindir@/guix-daemon --build-users-group=guixbuild
>  RemainAfterExit=yes
>  StandardOutput=syslog
>  StandardError=syslog
> +TasksMax=1024
>  
>  [Install]
>  WantedBy=multi-user.target

> 
> (Is it the right syntax?)

It works for me on systemd-229 on Debian.

Re: guix-daemon hitting systemd TasksMax

[Ludovic Courtès]

Leo Famulari <leo@famulari.name> skribis:

> On Tue, Apr 19, 2016 at 05:57:05PM +0200, Ludovic Courtès wrote:
>> What about this:
>> 
>
>> diff --git a/etc/guix-daemon.service.in b/etc/guix-daemon.service.in
>> index 28ef000..c79f867 100644
>> --- a/etc/guix-daemon.service.in
>> +++ b/etc/guix-daemon.service.in
>> @@ -10,6 +10,7 @@ ExecStart=@bindir@/guix-daemon --build-users-group=guixbuild
>>  RemainAfterExit=yes
>>  StandardOutput=syslog
>>  StandardError=syslog
>> +TasksMax=1024
>>  
>>  [Install]
>>  WantedBy=multi-user.target
>
>> 
>> (Is it the right syntax?)
>
> It works for me on systemd-229 on Debian.

Thanks, pushed as f489ce3c9343e3ba9927645d11fcb91bb69f0fa7.

Ludo’.