all messages for Guix-related lists mirrored at yhetil.org
 help / color / mirror / code / Atom feed
* guix-daemon hitting systemd TasksMax
@ 2016-04-14 23:37 Joram Schrijver
  2016-04-15 14:35 ` Efraim Flashner
  2016-04-16  1:29 ` Leo Famulari
  0 siblings, 2 replies; 11+ messages in thread
From: Joram Schrijver @ 2016-04-14 23:37 UTC (permalink / raw)
  To: guix-devel

Hi,

In version 228, systemd introduced a unit attribute `TasksMax`[1] that
limits the number of tasks that can run in a unit. It has effect on
kernel versions 4.3 and up, kernel threads and userspace process count
towards the limit, and the default value is 512.

Running `guix pull` without changing any settings crashed for me,
because building/testing texinfo hit this task limit. Increasing it to
1024 using `systemctl set-property guix-daemon.service TasksMax=1024`
fixed the problem.

Because I expect guix-daemon to hit this limit regularly, it might be
wise to set a higher `TasksMax` in the service file itself, perhaps even
`infinity`. Doing this would, sadly, cause older systemd versions
shipped by various distributions to complain about the existence of the
attribute, so perhaps a mention in the manual would instead be enough.


[1]:
https://www.freedesktop.org/software/systemd/man/systemd.resource-control.html#TasksAccounting=

--
  Joram

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: guix-daemon hitting systemd TasksMax
  2016-04-14 23:37 guix-daemon hitting systemd TasksMax Joram Schrijver
@ 2016-04-15 14:35 ` Efraim Flashner
  2016-04-16  1:29 ` Leo Famulari
  1 sibling, 0 replies; 11+ messages in thread
From: Efraim Flashner @ 2016-04-15 14:35 UTC (permalink / raw)
  To: Joram Schrijver; +Cc: guix-devel

[-- Attachment #1: Type: text/plain, Size: 1488 bytes --]

On Fri, Apr 15, 2016 at 01:37:32AM +0200, Joram Schrijver wrote:
> Hi,
> 
> In version 228, systemd introduced a unit attribute `TasksMax`[1] that
> limits the number of tasks that can run in a unit. It has effect on
> kernel versions 4.3 and up, kernel threads and userspace process count
> towards the limit, and the default value is 512.
> 
> Running `guix pull` without changing any settings crashed for me,
> because building/testing texinfo hit this task limit. Increasing it to
> 1024 using `systemctl set-property guix-daemon.service TasksMax=1024`
> fixed the problem.
> 
> Because I expect guix-daemon to hit this limit regularly, it might be
> wise to set a higher `TasksMax` in the service file itself, perhaps even
> `infinity`. Doing this would, sadly, cause older systemd versions
> shipped by various distributions to complain about the existence of the
> attribute, so perhaps a mention in the manual would instead be enough.
> 
> 
> [1]:
> https://www.freedesktop.org/software/systemd/man/systemd.resource-control.html#TasksAccounting=
> 
> --
>   Joram
> 

If it complains but doesn't cause it to error out it might be worth it
to add it to etc/guix-daemon.service in the guix source tree as well as
mention it in the manual.

-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: guix-daemon hitting systemd TasksMax
  2016-04-14 23:37 guix-daemon hitting systemd TasksMax Joram Schrijver
  2016-04-15 14:35 ` Efraim Flashner
@ 2016-04-16  1:29 ` Leo Famulari
  2016-04-16 13:12   ` Joram Schrijver
  1 sibling, 1 reply; 11+ messages in thread
From: Leo Famulari @ 2016-04-16  1:29 UTC (permalink / raw)
  To: Joram Schrijver; +Cc: guix-devel

On Fri, Apr 15, 2016 at 01:37:32AM +0200, Joram Schrijver wrote:
> Hi,
> 
> In version 228, systemd introduced a unit attribute `TasksMax`[1] that
> limits the number of tasks that can run in a unit. It has effect on
> kernel versions 4.3 and up, kernel threads and userspace process count
> towards the limit, and the default value is 512.
> 
> Running `guix pull` without changing any settings crashed for me,
> because building/testing texinfo hit this task limit. Increasing it to
> 1024 using `systemctl set-property guix-daemon.service TasksMax=1024`
> fixed the problem.

Interesting. What operating system are you using? I'm running systemd
229 on Debian testing and I don't have this issue, nor have I altered
this setting.

It would be useful to know what the difference is between our systemd
installations.  I don't see anything related to TasksMax in
systemd-229's NEWS:

https://github.com/systemd/systemd/blob/master/NEWS

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: guix-daemon hitting systemd TasksMax
  2016-04-16  1:29 ` Leo Famulari
@ 2016-04-16 13:12   ` Joram Schrijver
  2016-04-16 16:14     ` Leo Famulari
  0 siblings, 1 reply; 11+ messages in thread
From: Joram Schrijver @ 2016-04-16 13:12 UTC (permalink / raw)
  To: Leo Famulari; +Cc: guix-devel

On Sat, Apr 16, 2016, at 03:29 AM, Leo Famulari wrote:
> On Fri, Apr 15, 2016 at 01:37:32AM +0200, Joram Schrijver wrote:
> > Hi,
> > 
> > In version 228, systemd introduced a unit attribute `TasksMax`[1] that
> > limits the number of tasks that can run in a unit. It has effect on
> > kernel versions 4.3 and up, kernel threads and userspace process count
> > towards the limit, and the default value is 512.
> > 
> > Running `guix pull` without changing any settings crashed for me,
> > because building/testing texinfo hit this task limit. Increasing it to
> > 1024 using `systemctl set-property guix-daemon.service TasksMax=1024`
> > fixed the problem.
> 
> Interesting. What operating system are you using? I'm running systemd
> 229 on Debian testing and I don't have this issue, nor have I altered
> this setting.
> 
> It would be useful to know what the difference is between our systemd
> installations.  I don't see anything related to TasksMax in
> systemd-229's NEWS:
> 
> https://github.com/systemd/systemd/blob/master/NEWS

I am running Arch Linux, also with systemd 229. It doesn't look like
Debian applies any patches to systemd, so our installations should
pretty much be the same.

Perhaps the limit is only reached during the build process of some
specific packages?

--
  Joram

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: guix-daemon hitting systemd TasksMax
  2016-04-16 13:12   ` Joram Schrijver
@ 2016-04-16 16:14     ` Leo Famulari
  2016-04-16 22:46       ` Ludovic Courtès
  0 siblings, 1 reply; 11+ messages in thread
From: Leo Famulari @ 2016-04-16 16:14 UTC (permalink / raw)
  To: Joram Schrijver; +Cc: guix-devel

On Sat, Apr 16, 2016 at 03:12:15PM +0200, Joram Schrijver wrote:
> I am running Arch Linux, also with systemd 229. It doesn't look like
> Debian applies any patches to systemd, so our installations should
> pretty much be the same.
> 
> Perhaps the limit is only reached during the build process of some
> specific packages?

`guix pull` usually doesn't build any packages, but sometimes it does
need to. My (limited) understanding is that it sometimes needs to build
(or download) updated dependencies of Guix itself.

Do you think the number of "Tasks" would increase with more cores? I
only have 4 cores on my machines. Maybe if you had more, the builders
spawned by your guix-daemon would run more tasks as a consequence of
using all the cores.

I think there are other Guix people on Debian testing. I wonder if they
will hit this issue, too.

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: guix-daemon hitting systemd TasksMax
  2016-04-16 16:14     ` Leo Famulari
@ 2016-04-16 22:46       ` Ludovic Courtès
  2016-04-18 20:33         ` Leo Famulari
  0 siblings, 1 reply; 11+ messages in thread
From: Ludovic Courtès @ 2016-04-16 22:46 UTC (permalink / raw)
  To: Leo Famulari; +Cc: guix-devel

Leo Famulari <leo@famulari.name> skribis:

> On Sat, Apr 16, 2016 at 03:12:15PM +0200, Joram Schrijver wrote:
>> I am running Arch Linux, also with systemd 229. It doesn't look like
>> Debian applies any patches to systemd, so our installations should
>> pretty much be the same.
>> 
>> Perhaps the limit is only reached during the build process of some
>> specific packages?
>
> `guix pull` usually doesn't build any packages, but sometimes it does
> need to. My (limited) understanding is that it sometimes needs to build
> (or download) updated dependencies of Guix itself.
>
> Do you think the number of "Tasks" would increase with more cores? I
> only have 4 cores on my machines. Maybe if you had more, the builders
> spawned by your guix-daemon would run more tasks as a consequence of
> using all the cores.

Right, ‘guix pull’ entails a build of Guix using as many cores as
possible, with one process per core (and I think each Guile has two
threads, one of which is the signal delivery thread.)

So, if the limit on the total number of threads + processes is 512 and
you have 256 cores or more (lucky you!), you may hit the limit.

Ludo’.

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: guix-daemon hitting systemd TasksMax
  2016-04-16 22:46       ` Ludovic Courtès
@ 2016-04-18 20:33         ` Leo Famulari
  2016-04-18 20:42           ` Joram Schrijver
  0 siblings, 1 reply; 11+ messages in thread
From: Leo Famulari @ 2016-04-18 20:33 UTC (permalink / raw)
  To: Ludovic Courtès; +Cc: guix-devel

On Sun, Apr 17, 2016 at 12:46:16AM +0200, Ludovic Courtès wrote:
> Leo Famulari <leo@famulari.name> skribis:
> 
> > On Sat, Apr 16, 2016 at 03:12:15PM +0200, Joram Schrijver wrote:
> >> I am running Arch Linux, also with systemd 229. It doesn't look like
> >> Debian applies any patches to systemd, so our installations should
> >> pretty much be the same.
> >> 
> >> Perhaps the limit is only reached during the build process of some
> >> specific packages?
> >
> > `guix pull` usually doesn't build any packages, but sometimes it does
> > need to. My (limited) understanding is that it sometimes needs to build
> > (or download) updated dependencies of Guix itself.
> >
> > Do you think the number of "Tasks" would increase with more cores? I
> > only have 4 cores on my machines. Maybe if you had more, the builders
> > spawned by your guix-daemon would run more tasks as a consequence of
> > using all the cores.
> 
> Right, ‘guix pull’ entails a build of Guix using as many cores as
> possible, with one process per core (and I think each Guile has two
> threads, one of which is the signal delivery thread.)
> 
> So, if the limit on the total number of threads + processes is 512 and
> you have 256 cores or more (lucky you!), you may hit the limit.

Joram, are you this lucky? ;)

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: guix-daemon hitting systemd TasksMax
  2016-04-18 20:33         ` Leo Famulari
@ 2016-04-18 20:42           ` Joram Schrijver
  2016-04-19 15:57             ` Ludovic Courtès
  0 siblings, 1 reply; 11+ messages in thread
From: Joram Schrijver @ 2016-04-18 20:42 UTC (permalink / raw)
  To: Leo Famulari, Ludovic Courtès; +Cc: guix-devel

Heh, no I'm not. I have to do it with 4 cores + hyperthreading. My first
`guix pull` did have to build many things though, spawning numerous
threads in some cases. I eye-sampled the thread counts during one try
and they were certainly in the hundreds.

I installed Guix from the AUR[1], so I was running the most recent
release. Substitutes were enabled, but almost everything had to be built
instead of downloaded.

[1]: https://aur.archlinux.org/packages/guix/

--
  Joram

On Mon, Apr 18, 2016, at 10:33 PM, Leo Famulari wrote:
> On Sun, Apr 17, 2016 at 12:46:16AM +0200, Ludovic Courtès wrote:
> > Leo Famulari <leo@famulari.name> skribis:
> > 
> > > On Sat, Apr 16, 2016 at 03:12:15PM +0200, Joram Schrijver wrote:
> > >> I am running Arch Linux, also with systemd 229. It doesn't look like
> > >> Debian applies any patches to systemd, so our installations should
> > >> pretty much be the same.
> > >> 
> > >> Perhaps the limit is only reached during the build process of some
> > >> specific packages?
> > >
> > > `guix pull` usually doesn't build any packages, but sometimes it does
> > > need to. My (limited) understanding is that it sometimes needs to build
> > > (or download) updated dependencies of Guix itself.
> > >
> > > Do you think the number of "Tasks" would increase with more cores? I
> > > only have 4 cores on my machines. Maybe if you had more, the builders
> > > spawned by your guix-daemon would run more tasks as a consequence of
> > > using all the cores.
> > 
> > Right, ‘guix pull’ entails a build of Guix using as many cores as
> > possible, with one process per core (and I think each Guile has two
> > threads, one of which is the signal delivery thread.)
> > 
> > So, if the limit on the total number of threads + processes is 512 and
> > you have 256 cores or more (lucky you!), you may hit the limit.
> 
> Joram, are you this lucky? ;)

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: guix-daemon hitting systemd TasksMax
  2016-04-18 20:42           ` Joram Schrijver
@ 2016-04-19 15:57             ` Ludovic Courtès
  2016-04-20  1:20               ` Leo Famulari
  0 siblings, 1 reply; 11+ messages in thread
From: Ludovic Courtès @ 2016-04-19 15:57 UTC (permalink / raw)
  To: Joram Schrijver; +Cc: guix-devel

[-- Attachment #1: Type: text/plain, Size: 476 bytes --]

Joram Schrijver <i+guix-devel@joram.io> skribis:

> Heh, no I'm not. I have to do it with 4 cores + hyperthreading. My first
> `guix pull` did have to build many things though, spawning numerous
> threads in some cases. I eye-sampled the thread counts during one try
> and they were certainly in the hundreds.

Oh, then it could be that it’s a “make -j8” (assuming 2 hyperthreads per
core) for one of the packages that exceeded the limit.

What about this:


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Type: text/x-patch, Size: 366 bytes --]

diff --git a/etc/guix-daemon.service.in b/etc/guix-daemon.service.in
index 28ef000..c79f867 100644
--- a/etc/guix-daemon.service.in
+++ b/etc/guix-daemon.service.in
@@ -10,6 +10,7 @@ ExecStart=@bindir@/guix-daemon --build-users-group=guixbuild
 RemainAfterExit=yes
 StandardOutput=syslog
 StandardError=syslog
+TasksMax=1024
 
 [Install]
 WantedBy=multi-user.target

[-- Attachment #3: Type: text/plain, Size: 50 bytes --]


(Is it the right syntax?)

Thanks,
Ludo’.

^ permalink raw reply related	[flat|nested] 11+ messages in thread

* Re: guix-daemon hitting systemd TasksMax
  2016-04-19 15:57             ` Ludovic Courtès
@ 2016-04-20  1:20               ` Leo Famulari
  2016-04-25 21:37                 ` Ludovic Courtès
  0 siblings, 1 reply; 11+ messages in thread
From: Leo Famulari @ 2016-04-20  1:20 UTC (permalink / raw)
  To: Ludovic Courtès; +Cc: guix-devel

On Tue, Apr 19, 2016 at 05:57:05PM +0200, Ludovic Courtès wrote:
> What about this:
> 

> diff --git a/etc/guix-daemon.service.in b/etc/guix-daemon.service.in
> index 28ef000..c79f867 100644
> --- a/etc/guix-daemon.service.in
> +++ b/etc/guix-daemon.service.in
> @@ -10,6 +10,7 @@ ExecStart=@bindir@/guix-daemon --build-users-group=guixbuild
>  RemainAfterExit=yes
>  StandardOutput=syslog
>  StandardError=syslog
> +TasksMax=1024
>  
>  [Install]
>  WantedBy=multi-user.target

> 
> (Is it the right syntax?)

It works for me on systemd-229 on Debian.

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: guix-daemon hitting systemd TasksMax
  2016-04-20  1:20               ` Leo Famulari
@ 2016-04-25 21:37                 ` Ludovic Courtès
  0 siblings, 0 replies; 11+ messages in thread
From: Ludovic Courtès @ 2016-04-25 21:37 UTC (permalink / raw)
  To: Leo Famulari; +Cc: guix-devel

Leo Famulari <leo@famulari.name> skribis:

> On Tue, Apr 19, 2016 at 05:57:05PM +0200, Ludovic Courtès wrote:
>> What about this:
>> 
>
>> diff --git a/etc/guix-daemon.service.in b/etc/guix-daemon.service.in
>> index 28ef000..c79f867 100644
>> --- a/etc/guix-daemon.service.in
>> +++ b/etc/guix-daemon.service.in
>> @@ -10,6 +10,7 @@ ExecStart=@bindir@/guix-daemon --build-users-group=guixbuild
>>  RemainAfterExit=yes
>>  StandardOutput=syslog
>>  StandardError=syslog
>> +TasksMax=1024
>>  
>>  [Install]
>>  WantedBy=multi-user.target
>
>> 
>> (Is it the right syntax?)
>
> It works for me on systemd-229 on Debian.

Thanks, pushed as f489ce3c9343e3ba9927645d11fcb91bb69f0fa7.

Ludo’.

^ permalink raw reply	[flat|nested] 11+ messages in thread

end of thread, other threads:[~2016-04-25 21:37 UTC | newest]

Thread overview: 11+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2016-04-14 23:37 guix-daemon hitting systemd TasksMax Joram Schrijver
2016-04-15 14:35 ` Efraim Flashner
2016-04-16  1:29 ` Leo Famulari
2016-04-16 13:12   ` Joram Schrijver
2016-04-16 16:14     ` Leo Famulari
2016-04-16 22:46       ` Ludovic Courtès
2016-04-18 20:33         ` Leo Famulari
2016-04-18 20:42           ` Joram Schrijver
2016-04-19 15:57             ` Ludovic Courtès
2016-04-20  1:20               ` Leo Famulari
2016-04-25 21:37                 ` Ludovic Courtès

Code repositories for project(s) associated with this external index

	https://git.savannah.gnu.org/cgit/guix.git

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.