From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp12.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms5.migadu.com with LMTPS id 2MpVB0EXfmOXPgAAbAwnHQ (envelope-from ) for ; Wed, 23 Nov 2022 13:51:13 +0100 Received: from aspmx1.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp12.migadu.com with LMTPS id 8CEyB0EXfmMaPAAAauVa8A (envelope-from ) for ; Wed, 23 Nov 2022 13:51:13 +0100 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id ABB64EE8B for ; Wed, 23 Nov 2022 13:51:12 +0100 (CET) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1oxpDU-0008Qt-Fh; Wed, 23 Nov 2022 07:51:04 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1oxpDS-0008QM-Qx for bug-guix@gnu.org; Wed, 23 Nov 2022 07:51:02 -0500 Received: from debbugs.gnu.org ([209.51.188.43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1oxpDS-0002rN-IH; Wed, 23 Nov 2022 07:51:02 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1oxpDS-0004Qs-Bs; Wed, 23 Nov 2022 07:51:02 -0500 X-Loop: help-debbugs@gnu.org Subject: bug#59514: Stuck builds in Cuirass Resent-From: Marius Bakke Original-Sender: "Debbugs-submit" Resent-CC: othacehe@gnu.org, bug-guix@gnu.org Resent-Date: Wed, 23 Nov 2022 12:51:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: report 59514 X-GNU-PR-Package: guix X-GNU-PR-Keywords: To: 59514@debbugs.gnu.org Cc: othacehe@gnu.org X-Debbugs-Original-To: bug-guix@gnu.org X-Debbugs-Original-Xcc: othacehe@gnu.org Received: via spool by submit@debbugs.gnu.org id=B.166920784917003 (code B ref -1); Wed, 23 Nov 2022 12:51:02 +0000 Received: (at submit) by debbugs.gnu.org; 23 Nov 2022 12:50:49 +0000 Received: from localhost ([127.0.0.1]:54096 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1oxpDF-0004QB-2M for submit@debbugs.gnu.org; Wed, 23 Nov 2022 07:50:49 -0500 Received: from lists.gnu.org ([209.51.188.17]:57328) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1oxpDD-0004Q2-6X for submit@debbugs.gnu.org; Wed, 23 Nov 2022 07:50:47 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1oxpDC-0008JQ-Vr for bug-guix@gnu.org; Wed, 23 Nov 2022 07:50:47 -0500 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1oxpDC-0002oK-Mg for bug-guix@gnu.org; Wed, 23 Nov 2022 07:50:46 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:Date:Subject:To:From:in-reply-to: references; bh=NE0AyU4pYv3tPCGsqeFSDfBxMlu7vrwo1AToliItGqs=; b=RMeg/hb1g6Tfej uCNvPuLMXyhbPSny21x4QmN3Got/nwDbMQ+FdaXp1Qx/7lpMDL3Ayqv3BjEkImnh9Y+2WCHUkD5dV jJniwEuTgwTFpdcV301tUzgkhuiU3hderTx6kVccqjxj+fHICU+Cy5qUWRTaxj4qcOBcb9j79XQkC HbvQXv5hyIenR3DrGP4OwXKdq13C/EkC6ntx4AC79wo2hS/Dx8SkeBjZNrkEiIiquYmNrJ8rvQqWB BcL8XI+XDsBJYHdM1tI5HqqdbxJDskr0774coORy2tSrEfFbJ20eB0m8a7WPvQFtLOkV/mC8d3uRW NQ86oTf7cMYlEavVoMRg==; Received: from [2a02:2121:302:f5f9:52eb:71ff:fe49:3a13] (helo=localhost) by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1oxpDB-000194-9n for bug-guix@gnu.org; Wed, 23 Nov 2022 07:50:46 -0500 From: Marius Bakke Date: Wed, 23 Nov 2022 13:50:35 +0100 Message-ID: <87tu2pzvfo.fsf@gnu.org> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="==-=-="; micalg=pgp-sha512; protocol="application/pgp-signature" X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-guix@gnu.org List-Id: Bug reports for GNU Guix List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-guix-bounces+larch=yhetil.org@gnu.org Sender: bug-guix-bounces+larch=yhetil.org@gnu.org X-Migadu-Flow: FLOW_IN X-Migadu-Country: US ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1669207872; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:resent-cc:resent-from:resent-sender: resent-message-id:list-id:list-help:list-unsubscribe:list-subscribe: list-post:dkim-signature; bh=NE0AyU4pYv3tPCGsqeFSDfBxMlu7vrwo1AToliItGqs=; b=WCvOeG47swPLs3jZZ6SgHuW/AJ0TJDSBMghUK8bTBaWwKZkC2EJR9dO6m2BUm+SsNAIxFd czstsa4LxvUQwrPw14735EVXpDghJVYLvf+yymA08PcwjLGeSVHt0XwSNfcvWn2CRS3WqW 4lMjdOuK0rVz9jQtyhMaBSS8O6DRCsDx9XDygaSknPvX4/+kDg2qrqUdkU33hPpwKKty2q oKr3B0k2D0Ssnj42Rle+GycrZXvhKTmAKAvAMtU41DvgOO6Ii4PZVKPBPm/RViMTfQxntR WYvSsV5Zbx21PPmBZmxcI3Q7BLXCjol5+iUmjmHo9maktbJedZeCYO6NrkYxZg== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1669207872; a=rsa-sha256; cv=none; b=NRIUXi6ta2joIsK5nCP7y8BoXyMUh+XLqAtc6DLJUEJOVxsLOdN/ak29E+Zw+h5Stx+Ymq 2Meieaw4t9Tf9x3moP/jvP4NY4tDdGs+iQ1d/44UEWpukRm4PmIVADJKUVFEBwFfZiuew8 KKEAYGdqb0NT4WCBdAUynO7qgsDsdkrKmH+MCNrVXy+1VN3UR+1lBiHoa2nn0nixGXZbau J9R8t6hwK108hcsWLVHeTdRhsOm0JQdUffMOofiBdWmuTnjR4pYasb5sOxugYHqHZMwAEB ICtk1kUdsFTz9mvFiG4zUpqkvWjFQPMHrT3ceeYT6NA5+ZVy9bkkOArSiXG+HA== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=gnu.org header.s=fencepost-gnu-org header.b="RMeg/hb1"; dmarc=pass (policy=none) header.from=gnu.org; spf=pass (aspmx1.migadu.com: domain of "bug-guix-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="bug-guix-bounces+larch=yhetil.org@gnu.org" X-Migadu-Spam-Score: -4.49 Authentication-Results: aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=gnu.org header.s=fencepost-gnu-org header.b="RMeg/hb1"; dmarc=pass (policy=none) header.from=gnu.org; spf=pass (aspmx1.migadu.com: domain of "bug-guix-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="bug-guix-bounces+larch=yhetil.org@gnu.org" X-Migadu-Queue-Id: ABB64EE8B X-Spam-Score: -4.49 X-Migadu-Scanner: scn1.migadu.com X-TUID: rWzb+YX9DXR/ --==-=-= Content-Type: multipart/mixed; boundary="=-=-=" --=-=-= Content-Type: text/plain Hi, Cuirass has a tendency to not notice when a build is finished, leaving it in a "running" state. The phenomenon can be observed by going to and look at builds that are running for a suspiciously long time. Typically the build log will indicate that it has finished, yet Cuirass is patiently waiting...and not scheduling further builds. Restarting the builds typically get things going again. I wrote a nasty script to automatically restart builds that are running for >1 hour, but it's not a sustainable solution: --=-=-= Content-Type: text/plain Content-Disposition: attachment; filename=restart-old-builds.py Content-Transfer-Encoding: quoted-printable #!/usr/bin/env python3 # Restart stuck builds.... TODO fix cuirass properly. import requests from bs4 import BeautifulSoup import re builds_page =3D "https://ci.guix.gnu.org/status" builds_html =3D requests.get(builds_page).text soup =3D BeautifulSoup(builds_html, "html5lib") main =3D soup.find('main', {'id': 'content'}) table =3D main.find('table') result =3D {} for row in table.find_all('tr'): data =3D row.find_all('td') if len(data) > 0: build_id =3D row.find('a').contents[0] name =3D data[0].contents[0] age =3D data[1].contents[0] system =3D data[2].contents[0] log =3D data[3] result[build_id] =3D {'name': name, 'age': age, 'system': system} age_re =3D re.compile("(\d+) (\w+) ago") restart =3D [] for id in result.keys(): age =3D result[id]['age'] match =3D age_re.match(result[id]['age']) if match is not None: # "seconds ago" digits =3D match.group(1) time_unit =3D match.group(2) if time_unit =3D=3D "hours": restart.append(id) elif time_unit =3D=3D "minutes" and int(digits) > 60: restart.append(id) certificate_file =3D "/home/marius/tmp/mbakke.cert.pem" certificate_key =3D "/home/marius/tmp/mbakke.key.pem" import time print(f"Found {len(restart)} stuck builds..!") for id in restart: print(f"Going to restart {result[id]['name']} ({id}, running since {res= ult[id]['age']})...") requests.get(f"https://ci.guix.gnu.org/admin/build/{id}/restart", cert=3D(certificate_file, certificate_key)) time.sleep(3) --=-=-=-- --==-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iIUEARYKAC0WIQRNTknu3zbaMQ2ddzTocYulkRQQdwUCY34XGw8cbWFyaXVzQGdu dS5vcmcACgkQ6HGLpZEUEHcYCQD/WbYxZ+Mi1I4kYSCKqRmuVrucf7oVXlZwAyFT KHhbOrQA/jUT3vZCpeiiSPWyxedXqYOBllkcvQXgmT3tj4RPcZMH =pDj4 -----END PGP SIGNATURE----- --==-=-=--