* guix-daemon hitting systemd TasksMax @ 2016-04-14 23:37 Joram Schrijver 2016-04-15 14:35 ` Efraim Flashner 2016-04-16 1:29 ` Leo Famulari 0 siblings, 2 replies; 11+ messages in thread From: Joram Schrijver @ 2016-04-14 23:37 UTC (permalink / raw) To: guix-devel Hi, In version 228, systemd introduced a unit attribute `TasksMax`[1] that limits the number of tasks that can run in a unit. It has effect on kernel versions 4.3 and up, kernel threads and userspace process count towards the limit, and the default value is 512. Running `guix pull` without changing any settings crashed for me, because building/testing texinfo hit this task limit. Increasing it to 1024 using `systemctl set-property guix-daemon.service TasksMax=1024` fixed the problem. Because I expect guix-daemon to hit this limit regularly, it might be wise to set a higher `TasksMax` in the service file itself, perhaps even `infinity`. Doing this would, sadly, cause older systemd versions shipped by various distributions to complain about the existence of the attribute, so perhaps a mention in the manual would instead be enough. [1]: https://www.freedesktop.org/software/systemd/man/systemd.resource-control.html#TasksAccounting= -- Joram ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: guix-daemon hitting systemd TasksMax 2016-04-14 23:37 guix-daemon hitting systemd TasksMax Joram Schrijver @ 2016-04-15 14:35 ` Efraim Flashner 2016-04-16 1:29 ` Leo Famulari 1 sibling, 0 replies; 11+ messages in thread From: Efraim Flashner @ 2016-04-15 14:35 UTC (permalink / raw) To: Joram Schrijver; +Cc: guix-devel [-- Attachment #1: Type: text/plain, Size: 1488 bytes --] On Fri, Apr 15, 2016 at 01:37:32AM +0200, Joram Schrijver wrote: > Hi, > > In version 228, systemd introduced a unit attribute `TasksMax`[1] that > limits the number of tasks that can run in a unit. It has effect on > kernel versions 4.3 and up, kernel threads and userspace process count > towards the limit, and the default value is 512. > > Running `guix pull` without changing any settings crashed for me, > because building/testing texinfo hit this task limit. Increasing it to > 1024 using `systemctl set-property guix-daemon.service TasksMax=1024` > fixed the problem. > > Because I expect guix-daemon to hit this limit regularly, it might be > wise to set a higher `TasksMax` in the service file itself, perhaps even > `infinity`. Doing this would, sadly, cause older systemd versions > shipped by various distributions to complain about the existence of the > attribute, so perhaps a mention in the manual would instead be enough. > > > [1]: > https://www.freedesktop.org/software/systemd/man/systemd.resource-control.html#TasksAccounting= > > -- > Joram > If it complains but doesn't cause it to error out it might be worth it to add it to etc/guix-daemon.service in the guix source tree as well as mention it in the manual. -- Efraim Flashner <efraim@flashner.co.il> אפרים פלשנר GPG key = A28B F40C 3E55 1372 662D 14F7 41AA E7DC CA3D 8351 Confidentiality cannot be guaranteed on emails sent or received unencrypted [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 819 bytes --] ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: guix-daemon hitting systemd TasksMax 2016-04-14 23:37 guix-daemon hitting systemd TasksMax Joram Schrijver 2016-04-15 14:35 ` Efraim Flashner @ 2016-04-16 1:29 ` Leo Famulari 2016-04-16 13:12 ` Joram Schrijver 1 sibling, 1 reply; 11+ messages in thread From: Leo Famulari @ 2016-04-16 1:29 UTC (permalink / raw) To: Joram Schrijver; +Cc: guix-devel On Fri, Apr 15, 2016 at 01:37:32AM +0200, Joram Schrijver wrote: > Hi, > > In version 228, systemd introduced a unit attribute `TasksMax`[1] that > limits the number of tasks that can run in a unit. It has effect on > kernel versions 4.3 and up, kernel threads and userspace process count > towards the limit, and the default value is 512. > > Running `guix pull` without changing any settings crashed for me, > because building/testing texinfo hit this task limit. Increasing it to > 1024 using `systemctl set-property guix-daemon.service TasksMax=1024` > fixed the problem. Interesting. What operating system are you using? I'm running systemd 229 on Debian testing and I don't have this issue, nor have I altered this setting. It would be useful to know what the difference is between our systemd installations. I don't see anything related to TasksMax in systemd-229's NEWS: https://github.com/systemd/systemd/blob/master/NEWS ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: guix-daemon hitting systemd TasksMax 2016-04-16 1:29 ` Leo Famulari @ 2016-04-16 13:12 ` Joram Schrijver 2016-04-16 16:14 ` Leo Famulari 0 siblings, 1 reply; 11+ messages in thread From: Joram Schrijver @ 2016-04-16 13:12 UTC (permalink / raw) To: Leo Famulari; +Cc: guix-devel On Sat, Apr 16, 2016, at 03:29 AM, Leo Famulari wrote: > On Fri, Apr 15, 2016 at 01:37:32AM +0200, Joram Schrijver wrote: > > Hi, > > > > In version 228, systemd introduced a unit attribute `TasksMax`[1] that > > limits the number of tasks that can run in a unit. It has effect on > > kernel versions 4.3 and up, kernel threads and userspace process count > > towards the limit, and the default value is 512. > > > > Running `guix pull` without changing any settings crashed for me, > > because building/testing texinfo hit this task limit. Increasing it to > > 1024 using `systemctl set-property guix-daemon.service TasksMax=1024` > > fixed the problem. > > Interesting. What operating system are you using? I'm running systemd > 229 on Debian testing and I don't have this issue, nor have I altered > this setting. > > It would be useful to know what the difference is between our systemd > installations. I don't see anything related to TasksMax in > systemd-229's NEWS: > > https://github.com/systemd/systemd/blob/master/NEWS I am running Arch Linux, also with systemd 229. It doesn't look like Debian applies any patches to systemd, so our installations should pretty much be the same. Perhaps the limit is only reached during the build process of some specific packages? -- Joram ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: guix-daemon hitting systemd TasksMax 2016-04-16 13:12 ` Joram Schrijver @ 2016-04-16 16:14 ` Leo Famulari 2016-04-16 22:46 ` Ludovic Courtès 0 siblings, 1 reply; 11+ messages in thread From: Leo Famulari @ 2016-04-16 16:14 UTC (permalink / raw) To: Joram Schrijver; +Cc: guix-devel On Sat, Apr 16, 2016 at 03:12:15PM +0200, Joram Schrijver wrote: > I am running Arch Linux, also with systemd 229. It doesn't look like > Debian applies any patches to systemd, so our installations should > pretty much be the same. > > Perhaps the limit is only reached during the build process of some > specific packages? `guix pull` usually doesn't build any packages, but sometimes it does need to. My (limited) understanding is that it sometimes needs to build (or download) updated dependencies of Guix itself. Do you think the number of "Tasks" would increase with more cores? I only have 4 cores on my machines. Maybe if you had more, the builders spawned by your guix-daemon would run more tasks as a consequence of using all the cores. I think there are other Guix people on Debian testing. I wonder if they will hit this issue, too. ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: guix-daemon hitting systemd TasksMax 2016-04-16 16:14 ` Leo Famulari @ 2016-04-16 22:46 ` Ludovic Courtès 2016-04-18 20:33 ` Leo Famulari 0 siblings, 1 reply; 11+ messages in thread From: Ludovic Courtès @ 2016-04-16 22:46 UTC (permalink / raw) To: Leo Famulari; +Cc: guix-devel Leo Famulari <leo@famulari.name> skribis: > On Sat, Apr 16, 2016 at 03:12:15PM +0200, Joram Schrijver wrote: >> I am running Arch Linux, also with systemd 229. It doesn't look like >> Debian applies any patches to systemd, so our installations should >> pretty much be the same. >> >> Perhaps the limit is only reached during the build process of some >> specific packages? > > `guix pull` usually doesn't build any packages, but sometimes it does > need to. My (limited) understanding is that it sometimes needs to build > (or download) updated dependencies of Guix itself. > > Do you think the number of "Tasks" would increase with more cores? I > only have 4 cores on my machines. Maybe if you had more, the builders > spawned by your guix-daemon would run more tasks as a consequence of > using all the cores. Right, ‘guix pull’ entails a build of Guix using as many cores as possible, with one process per core (and I think each Guile has two threads, one of which is the signal delivery thread.) So, if the limit on the total number of threads + processes is 512 and you have 256 cores or more (lucky you!), you may hit the limit. Ludo’. ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: guix-daemon hitting systemd TasksMax 2016-04-16 22:46 ` Ludovic Courtès @ 2016-04-18 20:33 ` Leo Famulari 2016-04-18 20:42 ` Joram Schrijver 0 siblings, 1 reply; 11+ messages in thread From: Leo Famulari @ 2016-04-18 20:33 UTC (permalink / raw) To: Ludovic Courtès; +Cc: guix-devel On Sun, Apr 17, 2016 at 12:46:16AM +0200, Ludovic Courtès wrote: > Leo Famulari <leo@famulari.name> skribis: > > > On Sat, Apr 16, 2016 at 03:12:15PM +0200, Joram Schrijver wrote: > >> I am running Arch Linux, also with systemd 229. It doesn't look like > >> Debian applies any patches to systemd, so our installations should > >> pretty much be the same. > >> > >> Perhaps the limit is only reached during the build process of some > >> specific packages? > > > > `guix pull` usually doesn't build any packages, but sometimes it does > > need to. My (limited) understanding is that it sometimes needs to build > > (or download) updated dependencies of Guix itself. > > > > Do you think the number of "Tasks" would increase with more cores? I > > only have 4 cores on my machines. Maybe if you had more, the builders > > spawned by your guix-daemon would run more tasks as a consequence of > > using all the cores. > > Right, ‘guix pull’ entails a build of Guix using as many cores as > possible, with one process per core (and I think each Guile has two > threads, one of which is the signal delivery thread.) > > So, if the limit on the total number of threads + processes is 512 and > you have 256 cores or more (lucky you!), you may hit the limit. Joram, are you this lucky? ;) ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: guix-daemon hitting systemd TasksMax 2016-04-18 20:33 ` Leo Famulari @ 2016-04-18 20:42 ` Joram Schrijver 2016-04-19 15:57 ` Ludovic Courtès 0 siblings, 1 reply; 11+ messages in thread From: Joram Schrijver @ 2016-04-18 20:42 UTC (permalink / raw) To: Leo Famulari, Ludovic Courtès; +Cc: guix-devel Heh, no I'm not. I have to do it with 4 cores + hyperthreading. My first `guix pull` did have to build many things though, spawning numerous threads in some cases. I eye-sampled the thread counts during one try and they were certainly in the hundreds. I installed Guix from the AUR[1], so I was running the most recent release. Substitutes were enabled, but almost everything had to be built instead of downloaded. [1]: https://aur.archlinux.org/packages/guix/ -- Joram On Mon, Apr 18, 2016, at 10:33 PM, Leo Famulari wrote: > On Sun, Apr 17, 2016 at 12:46:16AM +0200, Ludovic Courtès wrote: > > Leo Famulari <leo@famulari.name> skribis: > > > > > On Sat, Apr 16, 2016 at 03:12:15PM +0200, Joram Schrijver wrote: > > >> I am running Arch Linux, also with systemd 229. It doesn't look like > > >> Debian applies any patches to systemd, so our installations should > > >> pretty much be the same. > > >> > > >> Perhaps the limit is only reached during the build process of some > > >> specific packages? > > > > > > `guix pull` usually doesn't build any packages, but sometimes it does > > > need to. My (limited) understanding is that it sometimes needs to build > > > (or download) updated dependencies of Guix itself. > > > > > > Do you think the number of "Tasks" would increase with more cores? I > > > only have 4 cores on my machines. Maybe if you had more, the builders > > > spawned by your guix-daemon would run more tasks as a consequence of > > > using all the cores. > > > > Right, ‘guix pull’ entails a build of Guix using as many cores as > > possible, with one process per core (and I think each Guile has two > > threads, one of which is the signal delivery thread.) > > > > So, if the limit on the total number of threads + processes is 512 and > > you have 256 cores or more (lucky you!), you may hit the limit. > > Joram, are you this lucky? ;) ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: guix-daemon hitting systemd TasksMax 2016-04-18 20:42 ` Joram Schrijver @ 2016-04-19 15:57 ` Ludovic Courtès 2016-04-20 1:20 ` Leo Famulari 0 siblings, 1 reply; 11+ messages in thread From: Ludovic Courtès @ 2016-04-19 15:57 UTC (permalink / raw) To: Joram Schrijver; +Cc: guix-devel [-- Attachment #1: Type: text/plain, Size: 476 bytes --] Joram Schrijver <i+guix-devel@joram.io> skribis: > Heh, no I'm not. I have to do it with 4 cores + hyperthreading. My first > `guix pull` did have to build many things though, spawning numerous > threads in some cases. I eye-sampled the thread counts during one try > and they were certainly in the hundreds. Oh, then it could be that it’s a “make -j8” (assuming 2 hyperthreads per core) for one of the packages that exceeded the limit. What about this: [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #2: Type: text/x-patch, Size: 366 bytes --] diff --git a/etc/guix-daemon.service.in b/etc/guix-daemon.service.in index 28ef000..c79f867 100644 --- a/etc/guix-daemon.service.in +++ b/etc/guix-daemon.service.in @@ -10,6 +10,7 @@ ExecStart=@bindir@/guix-daemon --build-users-group=guixbuild RemainAfterExit=yes StandardOutput=syslog StandardError=syslog +TasksMax=1024 [Install] WantedBy=multi-user.target [-- Attachment #3: Type: text/plain, Size: 50 bytes --] (Is it the right syntax?) Thanks, Ludo’. ^ permalink raw reply related [flat|nested] 11+ messages in thread
* Re: guix-daemon hitting systemd TasksMax 2016-04-19 15:57 ` Ludovic Courtès @ 2016-04-20 1:20 ` Leo Famulari 2016-04-25 21:37 ` Ludovic Courtès 0 siblings, 1 reply; 11+ messages in thread From: Leo Famulari @ 2016-04-20 1:20 UTC (permalink / raw) To: Ludovic Courtès; +Cc: guix-devel On Tue, Apr 19, 2016 at 05:57:05PM +0200, Ludovic Courtès wrote: > What about this: > > diff --git a/etc/guix-daemon.service.in b/etc/guix-daemon.service.in > index 28ef000..c79f867 100644 > --- a/etc/guix-daemon.service.in > +++ b/etc/guix-daemon.service.in > @@ -10,6 +10,7 @@ ExecStart=@bindir@/guix-daemon --build-users-group=guixbuild > RemainAfterExit=yes > StandardOutput=syslog > StandardError=syslog > +TasksMax=1024 > > [Install] > WantedBy=multi-user.target > > (Is it the right syntax?) It works for me on systemd-229 on Debian. ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: guix-daemon hitting systemd TasksMax 2016-04-20 1:20 ` Leo Famulari @ 2016-04-25 21:37 ` Ludovic Courtès 0 siblings, 0 replies; 11+ messages in thread From: Ludovic Courtès @ 2016-04-25 21:37 UTC (permalink / raw) To: Leo Famulari; +Cc: guix-devel Leo Famulari <leo@famulari.name> skribis: > On Tue, Apr 19, 2016 at 05:57:05PM +0200, Ludovic Courtès wrote: >> What about this: >> > >> diff --git a/etc/guix-daemon.service.in b/etc/guix-daemon.service.in >> index 28ef000..c79f867 100644 >> --- a/etc/guix-daemon.service.in >> +++ b/etc/guix-daemon.service.in >> @@ -10,6 +10,7 @@ ExecStart=@bindir@/guix-daemon --build-users-group=guixbuild >> RemainAfterExit=yes >> StandardOutput=syslog >> StandardError=syslog >> +TasksMax=1024 >> >> [Install] >> WantedBy=multi-user.target > >> >> (Is it the right syntax?) > > It works for me on systemd-229 on Debian. Thanks, pushed as f489ce3c9343e3ba9927645d11fcb91bb69f0fa7. Ludo’. ^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2016-04-25 21:37 UTC | newest] Thread overview: 11+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2016-04-14 23:37 guix-daemon hitting systemd TasksMax Joram Schrijver 2016-04-15 14:35 ` Efraim Flashner 2016-04-16 1:29 ` Leo Famulari 2016-04-16 13:12 ` Joram Schrijver 2016-04-16 16:14 ` Leo Famulari 2016-04-16 22:46 ` Ludovic Courtès 2016-04-18 20:33 ` Leo Famulari 2016-04-18 20:42 ` Joram Schrijver 2016-04-19 15:57 ` Ludovic Courtès 2016-04-20 1:20 ` Leo Famulari 2016-04-25 21:37 ` Ludovic Courtès
Code repositories for project(s) associated with this public inbox https://git.savannah.gnu.org/cgit/guix.git This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).