From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp0 ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id ECbLNb0UumHYwwAAgWs5BA (envelope-from ) for ; Wed, 15 Dec 2021 17:15:57 +0100 Received: from aspmx1.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp0 with LMTPS id SP54Mb0UumHUfQAA1q6Kng (envelope-from ) for ; Wed, 15 Dec 2021 16:15:57 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 7992113FF7 for ; Wed, 15 Dec 2021 17:15:57 +0100 (CET) Received: from localhost ([::1]:43056 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mxWwe-00077t-JH for larch@yhetil.org; Wed, 15 Dec 2021 11:15:56 -0500 Received: from eggs.gnu.org ([209.51.188.92]:53558) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mxWvz-000759-PK for guix-devel@gnu.org; Wed, 15 Dec 2021 11:15:17 -0500 Received: from [2001:470:142:3::e] (port=55878 helo=fencepost.gnu.org) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mxWvx-0003HC-Uj for guix-devel@gnu.org; Wed, 15 Dec 2021 11:15:15 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:Date:Subject:To:From:in-reply-to: references; bh=ACJyRyetQKaeMXJC7nnxw6gH3JfdwYbtCO7UqzWs9LU=; b=GavhYcKAZ+IAAP jLBPN2VrVZolcGyU+4Y9iNAB+OaEqqVZyt9qv0czm/4LvR7J4eQWwxEHMUa1SR9OnY6ghnWhKUhFd dJnqsc4Yl0t+TaVg7tpi+9N3evmbXjtE8L+U2RlpTHdU0sPrNjETSF0OgOBB+UQCHOe+yLHmfafg9 413KRtO9BhOcDqIbnTyXnI3b56DWpGAeGCQ/3QJQEhSqADKdQjcR8hk1jeRrzmw0zkWl/a6QMqPtE v+NjaXAjSrUG+NINz+YVhlPUyqDVVKXm4is/DAM3Wf6wJrmON+kdUCwJdcgK/T4nbJJKvtVgc9RJp NjNY78IUWgjDjc4xy5VQ==; Received: from lfbn-lyo-1-236-245.w86-202.abo.wanadoo.fr ([86.202.80.245]:48394 helo=meije) by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mxWvv-00064T-OJ for guix-devel@gnu.org; Wed, 15 Dec 2021 11:15:14 -0500 From: Mathieu Othacehe To: guix-devel@gnu.org Subject: CI status Date: Wed, 15 Dec 2021 17:15:08 +0100 Message-ID: <87zgp1j14j.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: "Guix-devel" X-Migadu-Flow: FLOW_IN X-Migadu-Country: US ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1639584957; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version: content-type:content-type:list-id:list-help:list-unsubscribe: list-subscribe:list-post:dkim-signature; bh=ACJyRyetQKaeMXJC7nnxw6gH3JfdwYbtCO7UqzWs9LU=; b=V0GNOKSL+/5MtKCBImIQaLKkZGRxzLq4VvqGY23X/iJrLkjP50AxT+gW5F7JeDlSW3QYro o4YyQ+fjnzQEb6/BKeu/6cTcPRK+CTPPIoP6XV/4A4iEuabWlS8pCuifSfO7Pea/MZiwYo ZJQfqmbGcfgMh92AAeKusICfHD6yfoZVkMkrWdEVsKT7ARgeWmD3EpjC9dXKBsvhLrEGuE nixfxtSdj64x4RGafAFKgGVZ6h48HnR4OHNVTAQa88Hzat0vm7ZOJRRwEPqfgTQpUcjHIJ Sq5IeowqSj2WnOcBMukhrTvMaIEkrrjh+Dh+OpxPrQfUUqa4jWCHH1SHsYrtew== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1639584957; a=rsa-sha256; cv=none; b=sQUiHUFs5iQ8QF6sZqxRMOYgL0LSGJMbOrmOP7obYjo+xMUJU450WFK9RhkbjUReqxP7Sn 0pUIvgOj2asMVhnXRZqQu9NUA8R3X/rk5uYOQJ+wXYE6xgYtRpkglXMgy1un2LSDT92loQ cOlevBCfT7AgztLY0je9Koj6bVYfwCQTZ1vuttPBfaui4vSPk51DIRN02jHXJUDiA/qsdU cpxB3iIsOZXj2pLnaWpPdRMGeKUAeosA9NYw1p/8RTS7sWRNULBxLc3pKT0qzxihBm/Jc0 TRLSZlfLCAs+Y+EOrOyf9Vc+fhl7oQp04t+nBJ6DYMDspi54F41nqu1QJpSkOg== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=pass header.d=gnu.org header.s=fencepost-gnu-org header.b=GavhYcKA; dmarc=pass (policy=none) header.from=gnu.org; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org" X-Migadu-Spam-Score: -9.68 Authentication-Results: aspmx1.migadu.com; dkim=pass header.d=gnu.org header.s=fencepost-gnu-org header.b=GavhYcKA; dmarc=pass (policy=none) header.from=gnu.org; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org" X-Migadu-Queue-Id: 7992113FF7 X-Spam-Score: -9.68 X-Migadu-Scanner: scn0.migadu.com X-TUID: 1llZTkmWFXoc Hello, You must have noticed that the CI is currently struggling a bit. Here is a small recap of the situation. * The IO operations on Berlin are mysteriously slow. Removing files from /gnu/store/trash is taking ages. This is reported here: https://issues.guix.gnu.org/51787. We have to kill the garbage collect frequently to keep things going. The bad side is obviously that we can't do that forever, as we only have 9.3T and decreasing, while we aim to stay at 10T available. * The PostgreSQL database behind ci.guix.gnu.org also became super slow and I decided to drop it. I don't know if there's a connection with the above point. I'm missing the appropriate tools/knowledge to monitor the IO & file-system performances. * The php package isn't building anymore, reported here: https://issues.guix.gnu.org/52513. This means that we cannot reconfigure zabbix. I removed it from the berlin configuration temporarily. * The cuirass-remote-server Avahi service is no longer visible when running "avahi-browse -a". I strongly suspect that this is related to the static-networking update, even if I don't have a proof for now. This means that the remote-workers using Avahi for discovering (hydra-guix-*) machines can no longer connect. The ci.guix.gnu.org/workers list is thus quite empty. * Facing those problems, I tried to rollback to a previous system generation, but this is bringing even more issues, as for instance the older Cuirass package, is struggling with the new database structure and other niceties. I think out best course of action is to stick to master and fix the above problems. Thanks, Mathieu