From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: <bug-guix-bounces+larch=yhetil.org@gnu.org> Received: from mp10.migadu.com ([2001:41d0:306:2d92::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms8.migadu.com with LMTPS id QD/4N7zcNmUHigAAG6o9tA:P1 (envelope-from <bug-guix-bounces+larch=yhetil.org@gnu.org>) for <larch@yhetil.org>; Mon, 23 Oct 2023 22:51:09 +0200 Received: from aspmx1.migadu.com ([2001:41d0:306:2d92::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp10.migadu.com with LMTPS id QD/4N7zcNmUHigAAG6o9tA (envelope-from <bug-guix-bounces+larch=yhetil.org@gnu.org>) for <larch@yhetil.org>; Mon, 23 Oct 2023 22:51:09 +0200 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 87A063737E for <larch@yhetil.org>; Mon, 23 Oct 2023 22:51:08 +0200 (CEST) Authentication-Results: aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=gnu.org header.s=fencepost-gnu-org header.b=VvV+siyx; spf=pass (aspmx1.migadu.com: domain of "bug-guix-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="bug-guix-bounces+larch=yhetil.org@gnu.org"; dmarc=pass (policy=none) header.from=gnu.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1698094268; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:resent-cc: resent-from:resent-sender:resent-message-id:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post:dkim-signature; bh=5xkj3ex/rOWjQRuEPvvx2ebD1zC6rv2TRoeAn0/bTWU=; b=f9sd2WZigv+JD4jFZTDqJfiWTgB4oAzKon5HeXYFvDyPRAzzFb1pu1cDFCvnq/5a8OTjle Lipn8BTjZkyxohhkOANroEW2lqJEA2xF3ui4dNWqYMoZLb/zU0AHqICmnNBHq4NruGaSsN m9f6vNyveyuysNMrc6svhkpT2lBHEJKV4aHeiJW+SXfW8w4Tpga3tqnQAsobzB2MTHKZjL KqZJ7uVlge/6nea4Vrx8WDIxNRQ5xQxnkXuZIOqrmYs+zLvM3SRpUgtv6PtEPwmq/afVZH GbBun1yf+JT7YQrVyA7bqaqZ20RIyNAy/E6epuxtj+/w6wkVUMxIUHS4QsPvlA== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=gnu.org header.s=fencepost-gnu-org header.b=VvV+siyx; spf=pass (aspmx1.migadu.com: domain of "bug-guix-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="bug-guix-bounces+larch=yhetil.org@gnu.org"; dmarc=pass (policy=none) header.from=gnu.org ARC-Seal: i=1; s=key1; d=yhetil.org; t=1698094268; a=rsa-sha256; cv=none; b=Pe0tzCqPxLT4Hokl0YphQC9TJ37uhPDESKVK3WWvUglHedEGWgb+FfvIlIWey/u6L4AAYt KJ39gX9UvxvyDyonCGCT9vs47GWiGBqMm+LkM+McunYBk7soLFEMZ6VRvpj5jxXlZyftWb /TY2WcsX0HJpp6qCVWR6nJ2MtquKihho6YRktg3st3CMIJh02aWAZtVWI+4Nrkqo/RhwaW DH44HALNECemKnamP7PLAxtIcAvo1A2CUZ4Y5nidXl77TE6f9M/fxTVn7TQF1NWp/2RSso pNj+VSG1xy8EI8w9DZHWbqoB3lWqiB0sGnamm7oO1e+qsmVK5UybGRt80ezexg== Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from <bug-guix-bounces@gnu.org>) id 1qv1si-0008Ie-0b; Mon, 23 Oct 2023 16:50:36 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from <Debian-debbugs@debbugs.gnu.org>) id 1qv1sf-0008I7-TP for bug-guix@gnu.org; Mon, 23 Oct 2023 16:50:33 -0400 Received: from debbugs.gnu.org ([2001:470:142:5::43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from <Debian-debbugs@debbugs.gnu.org>) id 1qv1sf-0004aF-Kl for bug-guix@gnu.org; Mon, 23 Oct 2023 16:50:33 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from <Debian-debbugs@debbugs.gnu.org>) id 1qv1t8-0001FS-IF for bug-guix@gnu.org; Mon, 23 Oct 2023 16:51:02 -0400 X-Loop: help-debbugs@gnu.org Subject: bug#66692: [cuirass] Spurious failed builds following patching u-boot sources Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@gnu.org> Original-Sender: "Debbugs-submit" <debbugs-submit-bounces@debbugs.gnu.org> Resent-CC: bug-guix@gnu.org Resent-Date: Mon, 23 Oct 2023 20:51:02 +0000 Resent-Message-ID: <handler.66692.B66692.16980942384761@debbugs.gnu.org> Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 66692 X-GNU-PR-Package: guix X-GNU-PR-Keywords: To: Maxim Cournoyer <maxim.cournoyer@gmail.com> Cc: 66692@debbugs.gnu.org Received: via spool by 66692-submit@debbugs.gnu.org id=B66692.16980942384761 (code B ref 66692); Mon, 23 Oct 2023 20:51:02 +0000 Received: (at 66692) by debbugs.gnu.org; 23 Oct 2023 20:50:38 +0000 Received: from localhost ([127.0.0.1]:51985 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces@debbugs.gnu.org>) id 1qv1sj-0001Ei-7P for submit@debbugs.gnu.org; Mon, 23 Oct 2023 16:50:38 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:60262) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <ludo@gnu.org>) id 1qv1sf-0001EO-Tn for 66692@debbugs.gnu.org; Mon, 23 Oct 2023 16:50:35 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from <ludo@gnu.org>) id 1qv1s7-0004Hb-Cz; Mon, 23 Oct 2023 16:49:59 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:Date:References:In-Reply-To:Subject:To: From; bh=5xkj3ex/rOWjQRuEPvvx2ebD1zC6rv2TRoeAn0/bTWU=; b=VvV+siyxZA91Kp+i0xIj re97fM9uDqEIqyOK+OMCwDWYUrExFljCsty8ldo+XLX4YLaVJqjVzYRrBhGSQNx72v0mKVSMLqKCS IMngTk1M0DgMEwlziAoFP5OL2NLWGo1NAanAlp+3ZEh+bTNvbsHu4SAhCoYEFc+hNHuMtKvkAX7TS GGFYtsOw26hmRcZZaSIEouWEwIK/3Vn/1w413ex4yQaUVY0mWS2GWVaWsfPXCmL64SkTpB+L99sRF 3/O/c4VP53+1t8iiojdidlSZHBLc1Yo0SR4DjC4PBZ+nfqQFhHZPY9R7mN10vbKu23HaSR23NmS4A 5ctozqRzeBkGuw==; From: Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@gnu.org> In-Reply-To: <87ttqi82rh.fsf@gmail.com> (Maxim Cournoyer's message of "Sun, 22 Oct 2023 22:57:06 -0400") References: <87ttqi82rh.fsf@gmail.com> Date: Mon, 23 Oct 2023 22:49:56 +0200 Message-ID: <87pm15dpxn.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-guix@gnu.org List-Id: Bug reports for GNU Guix <bug-guix.gnu.org> List-Unsubscribe: <https://lists.gnu.org/mailman/options/bug-guix>, <mailto:bug-guix-request@gnu.org?subject=unsubscribe> List-Archive: <https://lists.gnu.org/archive/html/bug-guix> List-Post: <mailto:bug-guix@gnu.org> List-Help: <mailto:bug-guix-request@gnu.org?subject=help> List-Subscribe: <https://lists.gnu.org/mailman/listinfo/bug-guix>, <mailto:bug-guix-request@gnu.org?subject=subscribe> Errors-To: bug-guix-bounces+larch=yhetil.org@gnu.org Sender: bug-guix-bounces+larch=yhetil.org@gnu.org X-Migadu-Country: US X-Migadu-Flow: FLOW_IN X-Migadu-Spam-Score: -3.80 X-Spam-Score: -3.80 X-Migadu-Queue-Id: 87A063737E X-Migadu-Scanner: mx2.migadu.com X-TUID: 8w9vWwiLMjFT Hi, Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis: > The build failures are not captured in the logs, which appear truncated, > e.g. from <https://ci.guix.gnu.org/build/2294052/details> (log empty -- > builder: sjd-p9, worker N38vG7OW)=20 On sjd-p9, I get: --8<---------------cut here---------------start------------->8--- root@sjd-p9:~# journalctl -u cuirass-remote-worker.service |grep -C2 s167lh= 0w0azfxx83ri8k0mn2gj57i5rl Oct 23 00:06:21 sjd-p9 cuirass[531]: N38vG7OW: derivation /gnu/store/kcv6hf= qz94f1vdw24viw7d9ih6v9zi1j-libkmahjongg-23.04.3.drv build succeeded. Oct 23 00:06:21 sjd-p9 cuirass[531]: N38vG7OW: request work. Oct 23 00:06:21 sjd-p9 cuirass[531]: N38vG7OW: building derivation `/gnu/st= ore/s167lh0w0azfxx83ri8k0mn2gj57i5rl-u-boot-rpi-4-32b-2023.07.02.drv' (syst= em: powerpc64le-linux) Oct 23 00:06:32 sjd-p9 cuirass[531]: C2q41fts: ping tcp://10.0.0.1:5555. Oct 23 00:06:32 sjd-p9 cuirass[531]: BrnR2Sju: ping tcp://10.0.0.1:5555. -- Oct 23 00:34:16 sjd-p9 cuirass[531]: vA6cYbb5: request work. Oct 23 00:34:16 sjd-p9 cuirass[531]: vA6cYbb5: building derivation `/gnu/st= ore/lj4h1g1zq235l6i8mjh88s1ylvj7ksxd-u-boot-pine64-plus-2023.07.02.drv' (sy= stem: powerpc64le-linux) Oct 23 00:34:29 sjd-p9 cuirass[531]: N38vG7OW: derivation /gnu/store/s167lh= 0w0azfxx83ri8k0mn2gj57i5rl-u-boot-rpi-4-32b-2023.07.02.drv build failed. Oct 23 00:34:29 sjd-p9 cuirass[531]: N38vG7OW: request work. Oct 23 00:34:29 sjd-p9 cuirass[531]: N38vG7OW: building derivation `/gnu/st= ore/dy8g5pi1gv8dza1av9957ic6zzj4l88j-u-boot-tools-2023.07.02.drv' (system: = powerpc64le-linux) root@sjd-p9:~# guix build --log-file /gnu/store/s167lh0w0azfxx83ri8k0mn2gj5= 7i5rl-u-boot-rpi-4-32b-2023.07.02.drv https://ci.guix.gnu.org/log/0yzfwl2b017gsnf60wvr7dnn2g26wrj7-u-boot-rpi-4-3= 2b-2023.07.02 --8<---------------cut here---------------end--------------->8--- So it looks as though the build was attempted and failed, but there=E2=80= =99s no local build log in /var/log/guix/drvs, meaning that, most likely, no build process was even started. The log of guix-daemon provide few hints: --8<---------------cut here---------------start------------->8--- root@sjd-p9:~# journalctl -u guix-daemon |grep 'Oct 23 00:' [...] Oct 23 00:06:21 sjd-p9 guix-daemon[534]: accepted connection from pid 531, = user root Oct 23 00:06:21 sjd-p9 guix-daemon[534]: accepted connection from pid 531, = user root Oct 23 00:06:55 sjd-p9 guix-daemon[534]: accepted connection from pid 531, = user root Oct 23 00:07:51 sjd-p9 guix-daemon[534]: accepted connection from pid 531, = user root Oct 23 00:16:25 sjd-p9 guix-daemon[534]: accepted connection from pid 531, = user root Oct 23 00:17:15 sjd-p9 guix-daemon[534]: accepted connection from pid 531, = user root Oct 23 00:33:42 sjd-p9 guix-daemon[2097426]: SIGPOLL Oct 23 00:33:42 sjd-p9 guix-daemon[534]: accepted connection from pid 531, = user root Oct 23 00:33:49 sjd-p9 guix-daemon[2097426]: unexpected build daemon error:= interrupted by the user Oct 23 00:34:16 sjd-p9 guix-daemon[534]: accepted connection from pid 531, = user root Oct 23 00:34:16 sjd-p9 guix-daemon[1990649]: SIGPOLL Oct 23 00:34:23 sjd-p9 guix-daemon[1990649]: unexpected build daemon error:= interrupted by the user Oct 23 00:34:29 sjd-p9 guix-daemon[2052532]: SIGPOLL Oct 23 00:34:29 sjd-p9 guix-daemon[534]: accepted connection from pid 531, = user root Oct 23 00:34:30 sjd-p9 guix-daemon[2052532]: unexpected build daemon error:= interrupted by the user Oct 23 00:34:38 sjd-p9 guix-daemon[1996946]: SIGPOLL Oct 23 00:34:38 sjd-p9 guix-daemon[534]: accepted connection from pid 531, = user root Oct 23 00:34:38 sjd-p9 guix-daemon[1996946]: unexpected build daemon error:= interrupted by the user Oct 23 00:34:45 sjd-p9 guix-daemon[2248556]: SIGPOLL Oct 23 00:34:45 sjd-p9 guix-daemon[534]: accepted connection from pid 531, = user root Oct 23 00:34:45 sjd-p9 guix-daemon[2248556]: unexpected build daemon error:= interrupted by the user Oct 23 00:35:33 sjd-p9 guix-daemon[534]: accepted connection from pid 531, = user root Oct 23 00:36:46 sjd-p9 guix-daemon[534]: accepted connection from pid 531, = user root --8<---------------cut here---------------end--------------->8--- One possibility would have been that =E2=80=98guix gc=E2=80=99 was running,= preventing the build process from starting (=E2=80=9Cbig GC lock=E2=80=9D taken); howe= ver, if this were the case, we=E2=80=99d see a connection from a different process in the guix-daemon log. > or from <https://ci.guix.gnu.org/build/2294218/details> (builder: > hydra-guix-107, worker 4CWieLpQ) which ends with: Here we get: --8<---------------cut here---------------start------------->8--- root@hydra-guix-107 ~# grep -C2 jy463789qf42dn4zk5i5jzry2lf82ypy /var/log/c= uirass-remote-worker.log 2023-10-23 00:17:34 vDKeRcHy: no available build. 2023-10-23 00:17:46 4CWieLpQ: request work. 2023-10-23 00:17:46 4CWieLpQ: building derivation `/gnu/store/jy463789qf42d= n4zk5i5jzry2lf82ypy-u-boot-cubietruck-2023.07.02.drv' (system: x86_64-linux) 2023-10-23 00:18:03 zARGXQMa: request work. 2023-10-23 00:18:03 zARGXQMa: no available build. -- 2023-10-23 00:27:35 vDKeRcHy: request work. 2023-10-23 00:27:35 vDKeRcHy: no available build. 2023-10-23 00:27:42 4CWieLpQ: derivation /gnu/store/jy463789qf42dn4zk5i5jzr= y2lf82ypy-u-boot-cubietruck-2023.07.02.drv build failed. 2023-10-23 00:27:42 4CWieLpQ: request work. 2023-10-23 00:27:42 4CWieLpQ: no available build. root@hydra-guix-107 ~# ls /var/log/guix/drvs/*/*u-boot-cubietruck-2023* ls: cannot access '/var/log/guix/drvs/*/*u-boot-cubietruck-2023*': No such = file or directory --8<---------------cut here---------------end--------------->8--- And likewise: --8<---------------cut here---------------start------------->8--- root@hydra-guix-107 ~# grep -C2 '10-23 00:27:' /var/log/guix-daemon.log 2023-10-23 00:17:46 accepted connection from pid 64022, user root 2023-10-23 00:19:34 accepted connection from pid 64022, user root 2023-10-23 00:27:42 SIGPOLL 2023-10-23 00:27:45 unexpected build daemon error: interrupted by the user 2023-10-23 00:33:33 accepted connection from pid 18016, user hydra 2023-10-23 00:33:34 accepted connection from pid 18035, user hydra --8<---------------cut here---------------end--------------->8--- But hey, as a matter of fact, =E2=80=98cuirass remote-server=E2=80=99 on be= rlin was restarted at around that time, as can be seen in /var/log/cuirass-remote-server.log: --8<---------------cut here---------------start------------->8--- 2023-10-23 00:26:06 0 items queued for eventual download 2023-10-23 00:26:55 period update: 0 resumable, 0 failed builds. 2023-10-23 00:26:58 error: failed to start worker/database proxy: Invalid a= rgument 2023-10-23 00:26:59 increased maximum number of open files from 1024 to 2048 2023-10-23 00:26:59 listening for build logs on port 5556 2023-10-23 00:26:59 deleting old build logs from '/var/cache/cuirass/remote= /'... 2023-10-23 00:28:49 selected 1115 build logs to remove 2023-10-23 00:28:49 WARNING: (cuirass scripts remote-server): imported modu= le (fibers) overrides core binding `sleep' 2023-10-23 00:28:49 starting fetch worker with up to 8 concurrent downloads 2023-10-23 00:28:49 build failed: '/gnu/store/qlmhcyk9ac30464v99i2ys0irs7jz= 3lg-u-boot-am335x-evm-2023.07.02.drv' 2023-10-23 00:28:49 build failed: '/gnu/store/dl3far2xa4paq6nl6pp9iklpxrwg1= wy2-ungoogled-chromium-wayland-112.0.5615.165-1.drv' 2023-10-23 00:28:50 build failed: '/gnu/store/y9fwzzgvyhvrmf0avihm6944j8rvp= g6m-u-boot-mx6cuboxi-2023.07.02.drv' 2023-10-23 00:28:50 build failed: '/gnu/store/5bhl5q86j6bkqx8zybq29khndhf05= 0jm-u-boot-rpi-2-2023.07.02.drv' 2023-10-23 00:28:50 build failed: '/gnu/store/16d5vq3dlzy1f4s7vazlfg6358bl8= dlc-u-boot-qemu-arm-2023.07.02.drv' 2023-10-23 00:28:51 build failed: '/gnu/store/i88vczhcnlz462cmpls8qgx5n3ssw= 7rc-u-boot-qemu-arm-2023.07.02.drv' 2023-10-23 00:28:51 build failed: '/gnu/store/jy463789qf42dn4zk5i5jzry2lf82= ypy-u-boot-cubietruck-2023.07.02.drv' 2023-10-23 00:28:51 build failed: '/gnu/store/i1ri2nkh1bcfzynmggbjg0rjlyi12= 1y8-u-boot-rpi-3-32b-2023.07.02.drv' 2023-10-23 00:28:51 build failed: '/gnu/store/q08xw052r484c1c1nj31sx0fg5yf2= py9-openjdk-18.0.2.drv' 2023-10-23 00:28:52 build failed: '/gnu/store/5j4pp2b5dx2ykw4na2wbmiwp84g0h= i4i-u-boot-am335x-boneblack-2023.07.02.drv' 2023-10-23 00:28:52 build failed: '/gnu/store/nqp61id4bavvcsib307rw5234myl7= ykr-u-boot-rpi-3-32b-efi-2023.07.02.drv' 2023-10-23 00:28:52 build failed: '/gnu/store/9h7x080cn44ik79mrzycbg2x0dvvg= igc-u-boot-a20-olinuxino-lime-2023.07.02.drv' 2023-10-23 00:28:52 build failed: '/gnu/store/mkwy8ywa6mrw5zdxwr88skyw94afg= aqv-u-boot-novena-2023.07.02.drv' 2023-10-23 00:28:53 build failed: '/gnu/store/lfjp146k0z0w2yqd3y6m2mv09fq01= 948-u-boot-sifive-unmatched-2023.07.02.drv' 2023-10-23 00:28:53 build failed: '/gnu/store/425ykg0ci4i612ryc7z79as94vipp= 9rb-u-boot-rpi-2-efi-2023.07.02.drv' 2023-10-23 00:28:57 build failed: '/gnu/store/7fyq24jsmvclnapjgf1li43w6dmm5= n06-u-boot-mx6cuboxi-2023.07.02.drv' [=E2=80=A6] 2023-10-23 00:34:29 build failed: '/gnu/store/s167lh0w0azfxx83ri8k0mn2gj57i= 5rl-u-boot-rpi-4-32b-2023.07.02.drv' --8<---------------cut here---------------end--------------->8--- /var/log/messages shows that =E2=80=98remote-server=E2=80=99 failed and was= respawned (the failure manifests as =E2=80=9Cfailed to start worker/database proxy=E2= =80=9D in the cuirass-remote-server.log excerpt above): --8<---------------cut here---------------start------------->8--- Oct 23 00:24:33 localhost vmunix: [12534110.756389] BTRFS error (device sde= 2): parent transid verify failed on logical 30457856 mirror 2 wanted 61709 = found 11 Oct 23 00:24:33 localhost vmunix: [12534110.768173] BTRFS error (device sde= 2): parent transid verify failed on logical 30457856 mirror 1 wanted 61709 = found 11 Oct 23 00:24:34 localhost vmunix: [12534110.779454] BTRFS error (device sde= 2): parent transid verify failed on logical 30457856 mirror 2 wanted 61709 = found 11 Oct 23 00:24:59 localhost ntpd[1953]: Soliciting pool server 81.169.208.5 Oct 23 00:26:04 localhost ntpd[1953]: Soliciting pool server 141.98.136.83 Oct 23 00:26:58 localhost shepherd[1]: Service cuirass-remote-server (PID 9= 2414) exited with 1.=20 Oct 23 00:26:58 localhost shepherd[1]: Respawning cuirass-remote-server.=20 Oct 23 00:26:58 localhost shepherd[1]: Starting service user-homes...=20 Oct 23 00:26:58 localhost shepherd[1]: Service user-homes has been started.= =20 Oct 23 00:26:58 localhost shepherd[1]: Service user-homes started.=20 Oct 23 00:26:58 localhost shepherd[1]: Service user-homes running with valu= e #t.=20 Oct 23 00:26:58 localhost shepherd[1]: Starting service postgres-roles...=20 Oct 23 00:26:58 localhost shepherd[1]: Service postgres-roles has been star= ted.=20 Oct 23 00:26:58 localhost shepherd[1]: Service postgres-roles started.=20 Oct 23 00:26:58 localhost shepherd[1]: Service postgres-roles running with = value #t.=20 Oct 23 00:26:58 localhost shepherd[1]: Starting service cuirass-remote-serv= er...=20 Oct 23 00:26:58 localhost shepherd[1]: Service cuirass-remote-server has be= en started.=20 Oct 23 00:26:58 localhost shepherd[1]: Service cuirass-remote-server starte= d.=20 Oct 23 00:26:58 localhost shepherd[1]: Service cuirass-remote-server runnin= g with value 63052.=20 Oct 23 00:26:59 localhost vmunix: [12534255.969754] validate_extent_buffer:= 58 callbacks suppressed Oct 23 00:26:59 localhost vmunix: [12534255.969764] BTRFS error (device sde= 2): parent transid verify failed on logical 30457856 mirror 1 wanted 61709 = found 11 --8<---------------cut here---------------end--------------->8--- To summarize: 1. =E2=80=98cuirass remote-server=E2=80=99 failed for some unknown reason= and was respawned. 2. All the ongoing and about-to-be-started builds carried out by worker processes at that point were marked as failed. We can improve on both parts. To be continued=E2=80=A6 Ludo=E2=80=99.