From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp12.migadu.com ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms5.migadu.com with LMTPS id kC5AJKGLrmIOKQAAbAwnHQ (envelope-from ) for ; Sun, 19 Jun 2022 04:36:17 +0200 Received: from aspmx1.migadu.com ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp12.migadu.com with LMTPS id MMvqI6GLrmKO5wAAauVa8A (envelope-from ) for ; Sun, 19 Jun 2022 04:36:17 +0200 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 4039A28087 for ; Sun, 19 Jun 2022 04:36:17 +0200 (CEST) Received: from localhost ([::1]:33052 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1o2kMF-0004Db-B4 for larch@yhetil.org; Sat, 18 Jun 2022 22:08:11 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:53082) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1o2kM6-0004DS-A1 for bug-guix@gnu.org; Sat, 18 Jun 2022 22:08:02 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:56492) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1o2kM6-0007Zy-0s for bug-guix@gnu.org; Sat, 18 Jun 2022 22:08:02 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1o2kM5-0002Tc-MS for bug-guix@gnu.org; Sat, 18 Jun 2022 22:08:01 -0400 X-Loop: help-debbugs@gnu.org Subject: bug#55848: [cuirass] workers stalled Resent-From: Tom Fitzhenry Original-Sender: "Debbugs-submit" Resent-CC: bug-guix@gnu.org Resent-Date: Sun, 19 Jun 2022 02:08:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 55848 X-GNU-PR-Package: guix X-GNU-PR-Keywords: To: Mathieu Othacehe Cc: 55848@debbugs.gnu.org Received: via spool by 55848-submit@debbugs.gnu.org id=B55848.16556044699501 (code B ref 55848); Sun, 19 Jun 2022 02:08:01 +0000 Received: (at 55848) by debbugs.gnu.org; 19 Jun 2022 02:07:49 +0000 Received: from localhost ([127.0.0.1]:50389 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1o2kLt-0002TB-G4 for submit@debbugs.gnu.org; Sat, 18 Jun 2022 22:07:49 -0400 Received: from wout1-smtp.messagingengine.com ([64.147.123.24]:58281) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1o2kLr-0002Sy-Ti for 55848@debbugs.gnu.org; Sat, 18 Jun 2022 22:07:48 -0400 Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailout.west.internal (Postfix) with ESMTP id 299CF32007F0; Sat, 18 Jun 2022 22:07:42 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute3.internal (MEProxy); Sat, 18 Jun 2022 22:07:42 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= tom-fitzhenry.me.uk; h=cc:cc:content-type:date:date:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:sender:subject:subject:to:to; s=fm2; t=1655604461; x= 1655690861; bh=wJi4EkR5njy+MkwPSF51IeYlqZu/qhJwlUW1v1eKalk=; b=D xmDWqkt9esdyxEA+Ypo6eLgzheGgImoq4p7iGlMcgqIgcURGJmYzykQsmhPJGmzy aJ/b9nvAZtjveFNevqiqDGvF/bUD1AHhztAHcwbDYUcHXSxbJUeyYpUQT5dvqVRi Uwla2NLeXjlhaYGE4y2qvs0mOY2xOJ2sKeRT1nejnaxRTmuuRnYkOxqFCGK4OYpg DwdFGsSZ3+2c7Lz1YUMJK4grY45+oq1SQYQJ2nlcpJpqfcz3JqLjoQllw9x3AWxE DD2ZlrBDbhsOWBpK+lun0KlIwGAk2wyXD70EnQ52T7ufToeHy36Hmr5Z4ztKJyYn bKmslBgSD7u9WSqQdaZhw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:date:date:feedback-id :feedback-id:from:from:in-reply-to:in-reply-to:message-id :mime-version:references:reply-to:sender:subject:subject:to:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; t=1655604461; x=1655690861; bh=wJi4EkR5njy+MkwPSF51IeYlqZu/ qhJwlUW1v1eKalk=; b=TCvH698K1GMUevFszyeCue6Wk6F1re4t+Sqh6dKGsgy7 sxmR2RY8uQH4ByF5DOK8QGCYqhlI7n0lwsc4+M2EzAiSsQqBLgICW6Jpz8badbMD bM7AAJmV/gOQTp8IBA64XEpRJtuixaD7HCz5TvNmRUuGrdKpGtKg08WhGtrxjzTd Kfd406MhynHqrl1uqWVbJIRTZZXKzFXPVCN/Nz0fJLqzK8v5+ofu2/M0ivNpEzpL gHGjSakLoNXvh+2scoJ5s9XrpHc4DV9I4OLT6tdamy4dxhSbIqMBLqwA1IRFnkIE tsu8ZCHYWjHRO9NkTJdWacrvBZz8IWsoYrA6+8scSQ== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvfedruddvkedgheefucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvvefujghffgffkfggtgesthdtredttdertdenucfhrhhomhepvfhomhcu hfhithiihhgvnhhrhicuoehtohhmsehtohhmqdhfihhtiihhvghnrhihrdhmvgdruhhkqe enucggtffrrghtthgvrhhnpeelkeejffdujeetgeejhfeugeejvdffkeeujefhveegkefg jeeiffettedvueefjeenucffohhmrghinhepghhnuhdrohhrghenucevlhhushhtvghruf hiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpehtohhmsehtohhmqdhfihhtiihh vghnrhihrdhmvgdruhhk X-ME-Proxy: Feedback-ID: iefc945ae:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Sat, 18 Jun 2022 22:07:39 -0400 (EDT) From: Tom Fitzhenry In-Reply-To: <87h74v2mu7.fsf@gnu.org> (Mathieu Othacehe's message of "Wed, 08 Jun 2022 17:31:44 +0200") References: <87h74v2mu7.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.1 (gnu/linux) Date: Sun, 19 Jun 2022 12:07:32 +1000 Message-ID: <875ykxcsnv.fsf@tom-fitzhenry.me.uk> MIME-Version: 1.0 Content-Type: text/plain X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-guix@gnu.org List-Id: Bug reports for GNU Guix List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-guix-bounces+larch=yhetil.org@gnu.org Sender: "bug-Guix" X-Migadu-Flow: FLOW_IN X-Migadu-To: larch@yhetil.org X-Migadu-Country: US ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1655606177; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:resent-cc:resent-from:resent-sender: resent-message-id:in-reply-to:in-reply-to:references:references: list-id:list-help:list-unsubscribe:list-subscribe:list-post: dkim-signature; bh=wJi4EkR5njy+MkwPSF51IeYlqZu/qhJwlUW1v1eKalk=; b=GRCRJ1QzFHtzjaDf4sgE6vJWMev2bmyJRD/NvNaAVUpf4v63ikI+HI2hI8nfpBVdKoykCI FdWYu2IQArSR0oLyI72O+6BTac2kO8ovHa+dn9kZdI8YeC012RG4q5PkQflVxy7lXYfgzS RSPZ6mmmKJ2bRnhNik9wdquRorw0yxJ8XElR2TWvHeTiU4v6R1HDpcZ4EcGXK4CMC5alkY ov+5s3UliAzkFkbiP3F0uJVCLlxFFELdE97tpsZavA29SEofJy0zl/+DwBRCBZ5dSQ57Y8 tRgK3d0+MINQjSFjq27XUEBACTo2JiLB/tOXhVLFIQQKgJpW0JW+H9M9qjSc3w== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1655606177; a=rsa-sha256; cv=none; b=CvPsduC28MKD67LAgmo8bJy/kD4dmnJK+TRhksxkkD055c96Bz5PrTupmZ2nwL9QW+moJ6 suAzRKi0uN00RKzM/R9MbFDXs2VWEtyl/sPxxKaYe6qbHb9WEopUDFbq7KmQxsLS9Un7+1 bWcVmkgWLBC1l7qnblITPNOYj1vm7Z0VO+FIaOpFbaMbCCc+mFpVymTA1h/QU9zOWaevma KenqFhOHbPIRG9Z26ssMFsOLmLFbSlf1+veTnldIWENl4/88dre6Q4GzO5j308gaFHDRP3 TTG3i6d2gTFXMQZ8WJlID0T0RkjxndIWRoanufvagwLbwGyy8LETqpPlNNNRAw== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=tom-fitzhenry.me.uk header.s=fm2 header.b="D xmDWqk"; dkim=fail ("headers rsa verify failed") header.d=messagingengine.com header.s=fm2 header.b=TCvH698K; dmarc=fail reason="SPF not aligned (relaxed)" header.from=tom-fitzhenry.me.uk (policy=none); spf=pass (aspmx1.migadu.com: domain of "bug-guix-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="bug-guix-bounces+larch=yhetil.org@gnu.org" X-Migadu-Spam-Score: 6.51 Authentication-Results: aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=tom-fitzhenry.me.uk header.s=fm2 header.b="D xmDWqk"; dkim=fail ("headers rsa verify failed") header.d=messagingengine.com header.s=fm2 header.b=TCvH698K; dmarc=fail reason="SPF not aligned (relaxed)" header.from=tom-fitzhenry.me.uk (policy=none); spf=pass (aspmx1.migadu.com: domain of "bug-guix-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="bug-guix-bounces+larch=yhetil.org@gnu.org" X-Migadu-Queue-Id: 4039A28087 X-Spam-Score: 6.51 X-Migadu-Scanner: scn0.migadu.com X-TUID: 3LAeOr4prd5L Mathieu Othacehe writes: Substitutes for aarch64 are a lot healthier now. Thanks Ludovic! * kreuzberg is now successfully building and has been for a while. * ci.guix.gnu.has has 41% of substitutes (a low percentage, but likely a high percentage of toolchains). 0 jobs are queued, presumably because Curiass believes its up-to-date. This should increase over time, as packages are updated. * bordeaux has 83.8% of substitutes. A few issues remain for aarch64: * grunewald and kreuzberg are not on . Perhaps they were taken down while the substitute ratio was low to avoid each worker independently recompiling expensive toolchains? * rust@1.39.0 (and thus all of Rust) is missing from ci and bordeaux. I had expected this would have been working. I'll take a look and raise a separate issue. --8<---------------cut here---------------start------------->8--- $ ./pre-inst-env guix weather -s aarch64-linux -c2000 computing 15514 package derivations for aarch64-linux... looking for 16265 store items on https://ci.guix.gnu.org... https://ci.guix.gnu.org 41.0% substitutes available (6668 out of 16265) at least 34188.1 MiB of nars (compressed) 45362.5 MiB on disk (uncompressed) 0.015 seconds per request (144.9 seconds in total) 66.2 requests per second 0.0% (0 out of 9597) of the missing items are queued at least 1000 queued builds aarch64-linux: 110 (11.0%) powerpc64le-linux: 890 (89.0%) build rate: 36.81 builds per hour aarch64-linux: 17.23 builds per hour x86_64-linux: 14.25 builds per hour powerpc64le-linux: 1.01 builds per hour i686-linux: 4.83 builds per hour 1871 packages are missing from 'https://ci.guix.gnu.org' for 'aarch64-linux', among which: 3479 rust@1.39.0 /gnu/store/xxlgndidxvhdd391k35vcmviixq5d9b0-rust-1.39.0-cargo /gnu/store/cfy1p8q4bwwy1i01cjfssfry21kpljz3-rust-1.39.0 2111 cairomm@1.14.2 /gnu/store/bxknxn3nbmmvavf537k0pggrynhrgsaf-cairomm-1.14.2-doc /gnu/store/3sn66mgr29v73zpp93c2v09a0rj87l3w-cairomm-1.14.2 2101 texlive-latex-pgf@59745 /gnu/store/l6jr7v8ygn3ybj4gxcwskf8ifsjcj6x1-texlive-latex-pgf-59745 looking for 16265 store items on https://bordeaux.guix.gnu.org... https://bordeaux.guix.gnu.org 83.8% substitutes available (13624 out of 16265) 35138.6 MiB of nars (compressed) 109501.6 MiB on disk (uncompressed) 0.060 seconds per request (699.4 seconds in total) 16.7 requests per second (continuous integration information unavailable) 579 packages are missing from 'https://bordeaux.guix.gnu.org' for 'aarch64-linux', among which: 3479 rust@1.39.0 /gnu/store/xxlgndidxvhdd391k35vcmviixq5d9b0-rust-1.39.0-cargo /gnu/store/cfy1p8q4bwwy1i01cjfssfry21kpljz3-rust-1.39.0 --8<---------------cut here---------------end--------------->8--- > Hello, > > The aarch64 workers were all idle whereas 70k builds were > available. Once restarted, they started building again. > > The problem might be that when the server is unavailable for a while the > worker connections expire and cannot be resumed once the server is > available again. > > Thanks, > > Mathieu