From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <bug-guix-bounces+larch=yhetil.org@gnu.org>
Received: from mp10.migadu.com ([2001:41d0:306:2d92::])
	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits))
	by ms8.migadu.com with LMTPS
	id QD/4N7zcNmUHigAAG6o9tA:P1
	(envelope-from <bug-guix-bounces+larch=yhetil.org@gnu.org>)
	for <larch@yhetil.org>; Mon, 23 Oct 2023 22:51:09 +0200
Received: from aspmx1.migadu.com ([2001:41d0:306:2d92::])
	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits))
	by mp10.migadu.com with LMTPS
	id QD/4N7zcNmUHigAAG6o9tA
	(envelope-from <bug-guix-bounces+larch=yhetil.org@gnu.org>)
	for <larch@yhetil.org>; Mon, 23 Oct 2023 22:51:09 +0200
Received: from lists.gnu.org (lists.gnu.org [209.51.188.17])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(No client certificate requested)
	by aspmx1.migadu.com (Postfix) with ESMTPS id 87A063737E
	for <larch@yhetil.org>; Mon, 23 Oct 2023 22:51:08 +0200 (CEST)
Authentication-Results: aspmx1.migadu.com;
	dkim=fail ("headers rsa verify failed") header.d=gnu.org header.s=fencepost-gnu-org header.b=VvV+siyx;
	spf=pass (aspmx1.migadu.com: domain of "bug-guix-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="bug-guix-bounces+larch=yhetil.org@gnu.org";
	dmarc=pass (policy=none) header.from=gnu.org
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org;
	s=key1; t=1698094268;
	h=from:from:sender:sender:reply-to:subject:subject:date:date:
	 message-id:message-id:to:to:cc:cc:mime-version:mime-version:
	 content-type:content-type:
	 content-transfer-encoding:content-transfer-encoding:resent-cc:
	 resent-from:resent-sender:resent-message-id:in-reply-to:in-reply-to:
	 references:references:list-id:list-help:list-unsubscribe:
	 list-subscribe:list-post:dkim-signature;
	bh=5xkj3ex/rOWjQRuEPvvx2ebD1zC6rv2TRoeAn0/bTWU=;
	b=f9sd2WZigv+JD4jFZTDqJfiWTgB4oAzKon5HeXYFvDyPRAzzFb1pu1cDFCvnq/5a8OTjle
	Lipn8BTjZkyxohhkOANroEW2lqJEA2xF3ui4dNWqYMoZLb/zU0AHqICmnNBHq4NruGaSsN
	m9f6vNyveyuysNMrc6svhkpT2lBHEJKV4aHeiJW+SXfW8w4Tpga3tqnQAsobzB2MTHKZjL
	KqZJ7uVlge/6nea4Vrx8WDIxNRQ5xQxnkXuZIOqrmYs+zLvM3SRpUgtv6PtEPwmq/afVZH
	GbBun1yf+JT7YQrVyA7bqaqZ20RIyNAy/E6epuxtj+/w6wkVUMxIUHS4QsPvlA==
ARC-Authentication-Results: i=1;
	aspmx1.migadu.com;
	dkim=fail ("headers rsa verify failed") header.d=gnu.org header.s=fencepost-gnu-org header.b=VvV+siyx;
	spf=pass (aspmx1.migadu.com: domain of "bug-guix-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="bug-guix-bounces+larch=yhetil.org@gnu.org";
	dmarc=pass (policy=none) header.from=gnu.org
ARC-Seal: i=1; s=key1; d=yhetil.org; t=1698094268; a=rsa-sha256; cv=none;
	b=Pe0tzCqPxLT4Hokl0YphQC9TJ37uhPDESKVK3WWvUglHedEGWgb+FfvIlIWey/u6L4AAYt
	KJ39gX9UvxvyDyonCGCT9vs47GWiGBqMm+LkM+McunYBk7soLFEMZ6VRvpj5jxXlZyftWb
	/TY2WcsX0HJpp6qCVWR6nJ2MtquKihho6YRktg3st3CMIJh02aWAZtVWI+4Nrkqo/RhwaW
	DH44HALNECemKnamP7PLAxtIcAvo1A2CUZ4Y5nidXl77TE6f9M/fxTVn7TQF1NWp/2RSso
	pNj+VSG1xy8EI8w9DZHWbqoB3lWqiB0sGnamm7oO1e+qsmVK5UybGRt80ezexg==
Received: from localhost ([::1] helo=lists1p.gnu.org)
	by lists.gnu.org with esmtp (Exim 4.90_1)
	(envelope-from <bug-guix-bounces@gnu.org>)
	id 1qv1si-0008Ie-0b; Mon, 23 Oct 2023 16:50:36 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10])
 by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <Debian-debbugs@debbugs.gnu.org>)
 id 1qv1sf-0008I7-TP
 for bug-guix@gnu.org; Mon, 23 Oct 2023 16:50:33 -0400
Received: from debbugs.gnu.org ([2001:470:142:5::43])
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128)
 (Exim 4.90_1) (envelope-from <Debian-debbugs@debbugs.gnu.org>)
 id 1qv1sf-0004aF-Kl
 for bug-guix@gnu.org; Mon, 23 Oct 2023 16:50:33 -0400
Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2)
 (envelope-from <Debian-debbugs@debbugs.gnu.org>) id 1qv1t8-0001FS-IF
 for bug-guix@gnu.org; Mon, 23 Oct 2023 16:51:02 -0400
X-Loop: help-debbugs@gnu.org
Subject: bug#66692: [cuirass] Spurious failed builds following patching u-boot
 sources
Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@gnu.org>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces@debbugs.gnu.org>
Resent-CC: bug-guix@gnu.org
Resent-Date: Mon, 23 Oct 2023 20:51:02 +0000
Resent-Message-ID: <handler.66692.B66692.16980942384761@debbugs.gnu.org>
Resent-Sender: help-debbugs@gnu.org
X-GNU-PR-Message: followup 66692
X-GNU-PR-Package: guix
X-GNU-PR-Keywords: 
To: Maxim Cournoyer <maxim.cournoyer@gmail.com>
Cc: 66692@debbugs.gnu.org
Received: via spool by 66692-submit@debbugs.gnu.org id=B66692.16980942384761
 (code B ref 66692); Mon, 23 Oct 2023 20:51:02 +0000
Received: (at 66692) by debbugs.gnu.org; 23 Oct 2023 20:50:38 +0000
Received: from localhost ([127.0.0.1]:51985 helo=debbugs.gnu.org)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <debbugs-submit-bounces@debbugs.gnu.org>)
 id 1qv1sj-0001Ei-7P
 for submit@debbugs.gnu.org; Mon, 23 Oct 2023 16:50:38 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10]:60262)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <ludo@gnu.org>) id 1qv1sf-0001EO-Tn
 for 66692@debbugs.gnu.org; Mon, 23 Oct 2023 16:50:35 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e])
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <ludo@gnu.org>)
 id 1qv1s7-0004Hb-Cz; Mon, 23 Oct 2023 16:49:59 -0400
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org;
 s=fencepost-gnu-org; h=MIME-Version:Date:References:In-Reply-To:Subject:To:
 From; bh=5xkj3ex/rOWjQRuEPvvx2ebD1zC6rv2TRoeAn0/bTWU=; b=VvV+siyxZA91Kp+i0xIj
 re97fM9uDqEIqyOK+OMCwDWYUrExFljCsty8ldo+XLX4YLaVJqjVzYRrBhGSQNx72v0mKVSMLqKCS
 IMngTk1M0DgMEwlziAoFP5OL2NLWGo1NAanAlp+3ZEh+bTNvbsHu4SAhCoYEFc+hNHuMtKvkAX7TS
 GGFYtsOw26hmRcZZaSIEouWEwIK/3Vn/1w413ex4yQaUVY0mWS2GWVaWsfPXCmL64SkTpB+L99sRF
 3/O/c4VP53+1t8iiojdidlSZHBLc1Yo0SR4DjC4PBZ+nfqQFhHZPY9R7mN10vbKu23HaSR23NmS4A
 5ctozqRzeBkGuw==;
From: Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@gnu.org>
In-Reply-To: <87ttqi82rh.fsf@gmail.com> (Maxim Cournoyer's message of "Sun, 22
 Oct 2023 22:57:06 -0400")
References: <87ttqi82rh.fsf@gmail.com>
Date: Mon, 23 Oct 2023 22:49:56 +0200
Message-ID: <87pm15dpxn.fsf@gnu.org>
User-Agent: Gnus/5.13 (Gnus v5.13)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-BeenThere: debbugs-submit@debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
X-BeenThere: bug-guix@gnu.org
List-Id: Bug reports for GNU Guix <bug-guix.gnu.org>
List-Unsubscribe: <https://lists.gnu.org/mailman/options/bug-guix>,
 <mailto:bug-guix-request@gnu.org?subject=unsubscribe>
List-Archive: <https://lists.gnu.org/archive/html/bug-guix>
List-Post: <mailto:bug-guix@gnu.org>
List-Help: <mailto:bug-guix-request@gnu.org?subject=help>
List-Subscribe: <https://lists.gnu.org/mailman/listinfo/bug-guix>,
 <mailto:bug-guix-request@gnu.org?subject=subscribe>
Errors-To: bug-guix-bounces+larch=yhetil.org@gnu.org
Sender: bug-guix-bounces+larch=yhetil.org@gnu.org
X-Migadu-Country: US
X-Migadu-Flow: FLOW_IN
X-Migadu-Spam-Score: -3.80
X-Spam-Score: -3.80
X-Migadu-Queue-Id: 87A063737E
X-Migadu-Scanner: mx2.migadu.com
X-TUID: 8w9vWwiLMjFT

Hi,

Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:

> The build failures are not captured in the logs, which appear truncated,
> e.g. from <https://ci.guix.gnu.org/build/2294052/details> (log empty --
> builder: sjd-p9, worker N38vG7OW)=20

On sjd-p9, I get:

--8<---------------cut here---------------start------------->8---
root@sjd-p9:~# journalctl -u cuirass-remote-worker.service |grep -C2 s167lh=
0w0azfxx83ri8k0mn2gj57i5rl
Oct 23 00:06:21 sjd-p9 cuirass[531]: N38vG7OW: derivation /gnu/store/kcv6hf=
qz94f1vdw24viw7d9ih6v9zi1j-libkmahjongg-23.04.3.drv build succeeded.
Oct 23 00:06:21 sjd-p9 cuirass[531]: N38vG7OW: request work.
Oct 23 00:06:21 sjd-p9 cuirass[531]: N38vG7OW: building derivation `/gnu/st=
ore/s167lh0w0azfxx83ri8k0mn2gj57i5rl-u-boot-rpi-4-32b-2023.07.02.drv' (syst=
em: powerpc64le-linux)
Oct 23 00:06:32 sjd-p9 cuirass[531]: C2q41fts: ping tcp://10.0.0.1:5555.
Oct 23 00:06:32 sjd-p9 cuirass[531]: BrnR2Sju: ping tcp://10.0.0.1:5555.
--
Oct 23 00:34:16 sjd-p9 cuirass[531]: vA6cYbb5: request work.
Oct 23 00:34:16 sjd-p9 cuirass[531]: vA6cYbb5: building derivation `/gnu/st=
ore/lj4h1g1zq235l6i8mjh88s1ylvj7ksxd-u-boot-pine64-plus-2023.07.02.drv' (sy=
stem: powerpc64le-linux)
Oct 23 00:34:29 sjd-p9 cuirass[531]: N38vG7OW: derivation /gnu/store/s167lh=
0w0azfxx83ri8k0mn2gj57i5rl-u-boot-rpi-4-32b-2023.07.02.drv build failed.
Oct 23 00:34:29 sjd-p9 cuirass[531]: N38vG7OW: request work.
Oct 23 00:34:29 sjd-p9 cuirass[531]: N38vG7OW: building derivation `/gnu/st=
ore/dy8g5pi1gv8dza1av9957ic6zzj4l88j-u-boot-tools-2023.07.02.drv' (system: =
powerpc64le-linux)
root@sjd-p9:~# guix build --log-file /gnu/store/s167lh0w0azfxx83ri8k0mn2gj5=
7i5rl-u-boot-rpi-4-32b-2023.07.02.drv
https://ci.guix.gnu.org/log/0yzfwl2b017gsnf60wvr7dnn2g26wrj7-u-boot-rpi-4-3=
2b-2023.07.02
--8<---------------cut here---------------end--------------->8---

So it looks as though the build was attempted and failed, but there=E2=80=
=99s no
local build log in /var/log/guix/drvs, meaning that, most likely, no
build process was even started.

The log of guix-daemon provide few hints:

--8<---------------cut here---------------start------------->8---
root@sjd-p9:~# journalctl -u guix-daemon |grep 'Oct 23 00:'

[...]

Oct 23 00:06:21 sjd-p9 guix-daemon[534]: accepted connection from pid 531, =
user root
Oct 23 00:06:21 sjd-p9 guix-daemon[534]: accepted connection from pid 531, =
user root
Oct 23 00:06:55 sjd-p9 guix-daemon[534]: accepted connection from pid 531, =
user root
Oct 23 00:07:51 sjd-p9 guix-daemon[534]: accepted connection from pid 531, =
user root
Oct 23 00:16:25 sjd-p9 guix-daemon[534]: accepted connection from pid 531, =
user root
Oct 23 00:17:15 sjd-p9 guix-daemon[534]: accepted connection from pid 531, =
user root
Oct 23 00:33:42 sjd-p9 guix-daemon[2097426]: SIGPOLL
Oct 23 00:33:42 sjd-p9 guix-daemon[534]: accepted connection from pid 531, =
user root
Oct 23 00:33:49 sjd-p9 guix-daemon[2097426]: unexpected build daemon error:=
 interrupted by the user
Oct 23 00:34:16 sjd-p9 guix-daemon[534]: accepted connection from pid 531, =
user root
Oct 23 00:34:16 sjd-p9 guix-daemon[1990649]: SIGPOLL
Oct 23 00:34:23 sjd-p9 guix-daemon[1990649]: unexpected build daemon error:=
 interrupted by the user
Oct 23 00:34:29 sjd-p9 guix-daemon[2052532]: SIGPOLL
Oct 23 00:34:29 sjd-p9 guix-daemon[534]: accepted connection from pid 531, =
user root
Oct 23 00:34:30 sjd-p9 guix-daemon[2052532]: unexpected build daemon error:=
 interrupted by the user
Oct 23 00:34:38 sjd-p9 guix-daemon[1996946]: SIGPOLL
Oct 23 00:34:38 sjd-p9 guix-daemon[534]: accepted connection from pid 531, =
user root
Oct 23 00:34:38 sjd-p9 guix-daemon[1996946]: unexpected build daemon error:=
 interrupted by the user
Oct 23 00:34:45 sjd-p9 guix-daemon[2248556]: SIGPOLL
Oct 23 00:34:45 sjd-p9 guix-daemon[534]: accepted connection from pid 531, =
user root
Oct 23 00:34:45 sjd-p9 guix-daemon[2248556]: unexpected build daemon error:=
 interrupted by the user
Oct 23 00:35:33 sjd-p9 guix-daemon[534]: accepted connection from pid 531, =
user root
Oct 23 00:36:46 sjd-p9 guix-daemon[534]: accepted connection from pid 531, =
user root
--8<---------------cut here---------------end--------------->8---

One possibility would have been that =E2=80=98guix gc=E2=80=99 was running,=
 preventing
the build process from starting (=E2=80=9Cbig GC lock=E2=80=9D taken); howe=
ver, if this
were the case, we=E2=80=99d see a connection from a different process in the
guix-daemon log.

> or from <https://ci.guix.gnu.org/build/2294218/details> (builder:
> hydra-guix-107, worker 4CWieLpQ) which ends with:

Here we get:

--8<---------------cut here---------------start------------->8---
root@hydra-guix-107 ~# grep -C2 jy463789qf42dn4zk5i5jzry2lf82ypy /var/log/c=
uirass-remote-worker.log
2023-10-23 00:17:34 vDKeRcHy: no available build.
2023-10-23 00:17:46 4CWieLpQ: request work.
2023-10-23 00:17:46 4CWieLpQ: building derivation `/gnu/store/jy463789qf42d=
n4zk5i5jzry2lf82ypy-u-boot-cubietruck-2023.07.02.drv' (system: x86_64-linux)
2023-10-23 00:18:03 zARGXQMa: request work.
2023-10-23 00:18:03 zARGXQMa: no available build.
--
2023-10-23 00:27:35 vDKeRcHy: request work.
2023-10-23 00:27:35 vDKeRcHy: no available build.
2023-10-23 00:27:42 4CWieLpQ: derivation /gnu/store/jy463789qf42dn4zk5i5jzr=
y2lf82ypy-u-boot-cubietruck-2023.07.02.drv build failed.
2023-10-23 00:27:42 4CWieLpQ: request work.
2023-10-23 00:27:42 4CWieLpQ: no available build.
root@hydra-guix-107 ~# ls /var/log/guix/drvs/*/*u-boot-cubietruck-2023*
ls: cannot access '/var/log/guix/drvs/*/*u-boot-cubietruck-2023*': No such =
file or directory
--8<---------------cut here---------------end--------------->8---

And likewise:

--8<---------------cut here---------------start------------->8---
root@hydra-guix-107 ~# grep -C2 '10-23 00:27:' /var/log/guix-daemon.log
2023-10-23 00:17:46 accepted connection from pid 64022, user root
2023-10-23 00:19:34 accepted connection from pid 64022, user root
2023-10-23 00:27:42 SIGPOLL
2023-10-23 00:27:45 unexpected build daemon error: interrupted by the user
2023-10-23 00:33:33 accepted connection from pid 18016, user hydra
2023-10-23 00:33:34 accepted connection from pid 18035, user hydra
--8<---------------cut here---------------end--------------->8---

But hey, as a matter of fact, =E2=80=98cuirass remote-server=E2=80=99 on be=
rlin was
restarted at around that time, as can be seen in
/var/log/cuirass-remote-server.log:

--8<---------------cut here---------------start------------->8---
2023-10-23 00:26:06 0 items queued for eventual download
2023-10-23 00:26:55 period update: 0 resumable, 0 failed builds.
2023-10-23 00:26:58 error: failed to start worker/database proxy: Invalid a=
rgument
2023-10-23 00:26:59 increased maximum number of open files from 1024 to 2048
2023-10-23 00:26:59 listening for build logs on port 5556
2023-10-23 00:26:59 deleting old build logs from '/var/cache/cuirass/remote=
/'...
2023-10-23 00:28:49 selected 1115 build logs to remove
2023-10-23 00:28:49 WARNING: (cuirass scripts remote-server): imported modu=
le (fibers) overrides core binding `sleep'
2023-10-23 00:28:49 starting fetch worker with up to 8 concurrent downloads
2023-10-23 00:28:49 build failed: '/gnu/store/qlmhcyk9ac30464v99i2ys0irs7jz=
3lg-u-boot-am335x-evm-2023.07.02.drv'
2023-10-23 00:28:49 build failed: '/gnu/store/dl3far2xa4paq6nl6pp9iklpxrwg1=
wy2-ungoogled-chromium-wayland-112.0.5615.165-1.drv'
2023-10-23 00:28:50 build failed: '/gnu/store/y9fwzzgvyhvrmf0avihm6944j8rvp=
g6m-u-boot-mx6cuboxi-2023.07.02.drv'
2023-10-23 00:28:50 build failed: '/gnu/store/5bhl5q86j6bkqx8zybq29khndhf05=
0jm-u-boot-rpi-2-2023.07.02.drv'
2023-10-23 00:28:50 build failed: '/gnu/store/16d5vq3dlzy1f4s7vazlfg6358bl8=
dlc-u-boot-qemu-arm-2023.07.02.drv'
2023-10-23 00:28:51 build failed: '/gnu/store/i88vczhcnlz462cmpls8qgx5n3ssw=
7rc-u-boot-qemu-arm-2023.07.02.drv'
2023-10-23 00:28:51 build failed: '/gnu/store/jy463789qf42dn4zk5i5jzry2lf82=
ypy-u-boot-cubietruck-2023.07.02.drv'
2023-10-23 00:28:51 build failed: '/gnu/store/i1ri2nkh1bcfzynmggbjg0rjlyi12=
1y8-u-boot-rpi-3-32b-2023.07.02.drv'
2023-10-23 00:28:51 build failed: '/gnu/store/q08xw052r484c1c1nj31sx0fg5yf2=
py9-openjdk-18.0.2.drv'
2023-10-23 00:28:52 build failed: '/gnu/store/5j4pp2b5dx2ykw4na2wbmiwp84g0h=
i4i-u-boot-am335x-boneblack-2023.07.02.drv'
2023-10-23 00:28:52 build failed: '/gnu/store/nqp61id4bavvcsib307rw5234myl7=
ykr-u-boot-rpi-3-32b-efi-2023.07.02.drv'
2023-10-23 00:28:52 build failed: '/gnu/store/9h7x080cn44ik79mrzycbg2x0dvvg=
igc-u-boot-a20-olinuxino-lime-2023.07.02.drv'
2023-10-23 00:28:52 build failed: '/gnu/store/mkwy8ywa6mrw5zdxwr88skyw94afg=
aqv-u-boot-novena-2023.07.02.drv'
2023-10-23 00:28:53 build failed: '/gnu/store/lfjp146k0z0w2yqd3y6m2mv09fq01=
948-u-boot-sifive-unmatched-2023.07.02.drv'
2023-10-23 00:28:53 build failed: '/gnu/store/425ykg0ci4i612ryc7z79as94vipp=
9rb-u-boot-rpi-2-efi-2023.07.02.drv'
2023-10-23 00:28:57 build failed: '/gnu/store/7fyq24jsmvclnapjgf1li43w6dmm5=
n06-u-boot-mx6cuboxi-2023.07.02.drv'
[=E2=80=A6]
2023-10-23 00:34:29 build failed: '/gnu/store/s167lh0w0azfxx83ri8k0mn2gj57i=
5rl-u-boot-rpi-4-32b-2023.07.02.drv'
--8<---------------cut here---------------end--------------->8---

/var/log/messages shows that =E2=80=98remote-server=E2=80=99 failed and was=
 respawned
(the failure manifests as =E2=80=9Cfailed to start worker/database proxy=E2=
=80=9D in the
cuirass-remote-server.log excerpt above):

--8<---------------cut here---------------start------------->8---
Oct 23 00:24:33 localhost vmunix: [12534110.756389] BTRFS error (device sde=
2): parent transid verify failed on logical 30457856 mirror 2 wanted 61709 =
found 11
Oct 23 00:24:33 localhost vmunix: [12534110.768173] BTRFS error (device sde=
2): parent transid verify failed on logical 30457856 mirror 1 wanted 61709 =
found 11
Oct 23 00:24:34 localhost vmunix: [12534110.779454] BTRFS error (device sde=
2): parent transid verify failed on logical 30457856 mirror 2 wanted 61709 =
found 11
Oct 23 00:24:59 localhost ntpd[1953]: Soliciting pool server 81.169.208.5
Oct 23 00:26:04 localhost ntpd[1953]: Soliciting pool server 141.98.136.83
Oct 23 00:26:58 localhost shepherd[1]: Service cuirass-remote-server (PID 9=
2414) exited with 1.=20
Oct 23 00:26:58 localhost shepherd[1]: Respawning cuirass-remote-server.=20
Oct 23 00:26:58 localhost shepherd[1]: Starting service user-homes...=20
Oct 23 00:26:58 localhost shepherd[1]: Service user-homes has been started.=
=20
Oct 23 00:26:58 localhost shepherd[1]: Service user-homes started.=20
Oct 23 00:26:58 localhost shepherd[1]: Service user-homes running with valu=
e #t.=20
Oct 23 00:26:58 localhost shepherd[1]: Starting service postgres-roles...=20
Oct 23 00:26:58 localhost shepherd[1]: Service postgres-roles has been star=
ted.=20
Oct 23 00:26:58 localhost shepherd[1]: Service postgres-roles started.=20
Oct 23 00:26:58 localhost shepherd[1]: Service postgres-roles running with =
value #t.=20
Oct 23 00:26:58 localhost shepherd[1]: Starting service cuirass-remote-serv=
er...=20
Oct 23 00:26:58 localhost shepherd[1]: Service cuirass-remote-server has be=
en started.=20
Oct 23 00:26:58 localhost shepherd[1]: Service cuirass-remote-server starte=
d.=20
Oct 23 00:26:58 localhost shepherd[1]: Service cuirass-remote-server runnin=
g with value 63052.=20
Oct 23 00:26:59 localhost vmunix: [12534255.969754] validate_extent_buffer:=
 58 callbacks suppressed
Oct 23 00:26:59 localhost vmunix: [12534255.969764] BTRFS error (device sde=
2): parent transid verify failed on logical 30457856 mirror 1 wanted 61709 =
found 11
--8<---------------cut here---------------end--------------->8---

To summarize:

  1. =E2=80=98cuirass remote-server=E2=80=99 failed for some unknown reason=
 and was
     respawned.

  2. All the ongoing and about-to-be-started builds carried out by
     worker processes at that point were marked as failed.

We can improve on both parts.

To be continued=E2=80=A6

Ludo=E2=80=99.