bug#30644: Cuirass runs out of build users

[Efraim Flashner <efraim@flashner.co.il>]

[-- Attachment #1: Type: text/plain, Size: 3627 bytes --]

On Wed, Feb 28, 2018 at 10:07:14AM +0100, Andreas Enge wrote:
> Hello,
> 
> in trying out the new bayfront machine, I raised the number of concurrent
> build jobs to 10; the number of build accounts is programmatically set to
> twice this number.
> 
> Looking at the build directories in /tmp, 9 build jobs were started on
> February 26 at around 21:50, a 10th one at 22:24.
> 
> Later, I have been getting messages such as these:
> 2018-02-27T18:31:13 building 14491 derivations in batches of 200
> 2018-02-27T18:31:13 building batch of 200 jobs (0/14491)
> 2018-02-27T18:31:13 build failed: '/gnu/store/3qy3s1dy3nnb108ifz9yiscs687g0xkh-gfortran-5.5.0.drv'
> 2018-02-27T18:31:13 build failed: '/gnu/store/1zdxbjmcw1qnmzh2qcbafsrbr8i2j1x3-glib-2.54.2.drv'
> 2018-02-27T18:31:14 build failed: '/gnu/store/w56wpx4jar88hfabipvf6gd6hz81p6wk-python2-gevent-1.2.2.drv'
> 2018-02-27T18:31:14 build failed: '/gnu/store/i2zmgbsi599pxskqh15s38fpms359jp7-gcc-cross-sans-libc-arm-linux-gnueabihf-5.5.0.drv'
> 2018-02-27T18:31:14 build failed: '/gnu/store/qd59gqza3ly4rk2xjf40q2539rjgzh5r-gcc-cross-sans-libc-powerpc-linux-gnu-5.5.0.drv'
> 2018-02-27T18:31:14 build failed: '/gnu/store/3qy3s1dy3nnb108ifz9yiscs687g0xkh-gfortran-5.5.0.drv'
> 2018-02-27T18:31:14 build failed: '/gnu/store/1zdxbjmcw1qnmzh2qcbafsrbr8i2j1x3-glib-2.54.2.drv'
> 2018-02-27T18:31:15 build started: '/gnu/store/08pjxws1pd04syqf28d3hhsfrb946bpc-dzen-0.9.5-1.488ab66.drv'
> 2018-02-27T18:31:15 build failed: '/gnu/store/3qy3s1dy3nnb108ifz9yiscs687g0xkh-gfortran-5.5.0.drv'
> 2018-02-27T18:31:15 build failed: '/gnu/store/1zdxbjmcw1qnmzh2qcbafsrbr8i2j1x3-glib-2.54.2.drv'
> 2018-02-27T18:31:15 build started: '/gnu/store/01r4xm1nqixcxjbbn6xakpg3m5wkr12v-po4a-0.47.drv'
> 2018-02-27T18:31:15 build failed: '/gnu/store/d545d4jzw0mfa61d9ml3vygchr93xp51-python-gevent-1.2.2.drv'
> 2018-02-27T18:31:15 build started: '/gnu/store/02chd3k5qc8p8fc46j4w955cfld77cw1-spinsim-0.75-1.66915a7ad.drv'
> 2018-02-27T18:31:15 build started: '/gnu/store/81b2340psschvvk7ybhnz6nx9jkpdlwa-Net-SMTP-SSL-1.04.tar.gz.drv'
> 2018-02-27T18:31:15 build started: '/gnu/store/rkkacywdj7yya3wf51i141x5khzld4zl-clx-0.0.0-1.tar.xz.drv'
> 2018-02-27T18:31:16 build started: '/gnu/store/286gkv0z6g9fb4frcx6yp2fg9v6iwk3b-ecl-alexandria-0.0.0-1.926a066.drv'
> 2018-02-27T18:31:16 build started: '/gnu/store/hmy3ds71gbv1mspc9p5yvvhz5pibs8wl-elixir-1.5.3.tar.xz.drv'
> 2018-02-27T18:31:16 build failed: '/gnu/store/1zdxbjmcw1qnmzh2qcbafsrbr8i2j1x3-glib-2.54.2.drv'
> 2018-02-27T18:31:16 build started: '/gnu/store/y0p2myl0al8x3rlbxphgyjk83zl3kmb5-idris-mode-0.9.19.tar.drv'
> 2018-02-27T18:31:16 batch of builds (partially) failed:all build users are currently in use; consider creating additional users and adding them to the `guixbuild' group (status: 1)
> 
> So it looks as if build jobs get stalled, but somehow do not free their
> build users.
> 
> Hm, the directories in /tmp are created by guixbuilder01 to guixbuilder10.
> 10 is the default value, but it should have been overwritten.
> 
> And in /etc/passwd, I see 20 users. But apparently, they are not used!
> 
> Andreas
> 

When I use a shell script to build all the packages, if a package hangs
and gets eaten by the timout-limit then it holds the build user and
space in /tmp until all 10 builders are taken and building stops or all
the builds finish and then it releases them.

-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]