From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp1 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id YDf1LIfnv2EcZwEAgWs5BA (envelope-from ) for ; Mon, 20 Dec 2021 03:16:39 +0100 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp1 with LMTPS id 2Jt8KIfnv2EZSgAAbx9fmQ (envelope-from ) for ; Mon, 20 Dec 2021 02:16:39 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 77BF9D9EB for ; Mon, 20 Dec 2021 03:16:39 +0100 (CET) Received: from localhost ([::1]:42146 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mz8EA-0001CJ-Kg for larch@yhetil.org; Sun, 19 Dec 2021 21:16:38 -0500 Received: from eggs.gnu.org ([209.51.188.92]:51322) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mz8Dw-00018W-Iw for guix-devel@gnu.org; Sun, 19 Dec 2021 21:16:24 -0500 Received: from [2607:f8b0:4864:20::834] (port=41785 helo=mail-qt1-x834.google.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1mz8Du-00060l-Uy; Sun, 19 Dec 2021 21:16:24 -0500 Received: by mail-qt1-x834.google.com with SMTP id v22so8447085qtx.8; Sun, 19 Dec 2021 18:16:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:references:date:in-reply-to:message-id :user-agent:mime-version; bh=u0KVxRWCL6jgOZw0DnceKNOT4P23EnlW3jGlkPlVzlU=; b=YWzoFL2/dlobxY+t1qqm005gZZbSVST8+dLVYXrH9Y2eb8JkrP+9ET7jtlPUJDUDmY Pa4zxxoglPmsFeR+HzHMQ1HgRur0qCr6ipR5Wh9zk8gYQaCaMgZKdLG2LEeMEA7ObrgU uVGeEfOTO4yR9ETedJxkvqXHeLx3uptKXOe6eRvMc1qyyy7M68aoDzV+ywFQ23jjgFNx HKW9myH20nExY+fLwK2s4rDYVYBR7dyZ3ZSR+7An4o95R4wmbTA2MaZLyjWa44jbonpD z01YlAVWspxKgwIkqGCM77xGIGyWtwnjZn8kECfs+Qtbp0ApN8NyW4KbAQ5FlBMrejPJ ExFA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:references:date:in-reply-to :message-id:user-agent:mime-version; bh=u0KVxRWCL6jgOZw0DnceKNOT4P23EnlW3jGlkPlVzlU=; b=mbHEqR63RkmhC+iR2tuzS11uqunoXTJExVUMFUJ9kJeeqKHX9iHl9OqJMsIKduhAj7 +aQSsYnwFkFXybcTS5BQOkCpywHl8cs/MNW4eN4bakSzVSuswL4ZziqZY7NmfSTizEP6 HnH/A8Vmz2DGHgsCd/dNl5DW9lCpjk/5kqLJSPcVZNC/pA5SKpBVCa2YYH7hQMjDkNqt M+TdKCLwEbWeij+8roxS4ATbv52N2PY+vUFdrBs0o5PSrxaCOy7vKyV23OVae7RbSXNo 4uehngUmow+0Iu8YWssrsXDiegEJfR4Os+UG9QB4iFtOZVoo709SlGbjI7b8SWtYlfQN Yw3Q== X-Gm-Message-State: AOAM5313VVBAAyZdiBIwFy1rNGBenIv0g9Dw7FTa7L99dvhNha4pQrlI X1UGOzajPmilWEfXiBE8ZVKAmFmvfcw= X-Google-Smtp-Source: ABdhPJyGonT16mo8PcZDPGosAPMfS5t554yuegGtpoD0swgW4OA6moQHjIPo1SCz5XxkAqXigR1TPg== X-Received: by 2002:ac8:5e08:: with SMTP id h8mr10973820qtx.204.1639966581443; Sun, 19 Dec 2021 18:16:21 -0800 (PST) Received: from hurd (dsl-10-133-19.b2b2c.ca. [72.10.133.19]) by smtp.gmail.com with ESMTPSA id y12sm10440708qko.36.2021.12.19.18.16.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 19 Dec 2021 18:16:21 -0800 (PST) From: Maxim Cournoyer To: Mathieu Othacehe Subject: Re: CI status References: <87zgp1j14j.fsf@gnu.org> Date: Sun, 19 Dec 2021 21:16:20 -0500 In-Reply-To: <87zgp1j14j.fsf@gnu.org> (Mathieu Othacehe's message of "Wed, 15 Dec 2021 17:15:08 +0100") Message-ID: <87r1a82f7v.fsf@gmail.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-Host-Lookup-Failed: Reverse DNS lookup failed for 2607:f8b0:4864:20::834 (failed) Received-SPF: pass client-ip=2607:f8b0:4864:20::834; envelope-from=maxim.cournoyer@gmail.com; helo=mail-qt1-x834.google.com X-Spam_score_int: -12 X-Spam_score: -1.3 X-Spam_bar: - X-Spam_report: (-1.3 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, RDNS_NONE=0.793, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: guix-devel@gnu.org Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: "Guix-devel" X-Migadu-Flow: FLOW_IN X-Migadu-Country: US ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1639966599; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post:dkim-signature; bh=u0KVxRWCL6jgOZw0DnceKNOT4P23EnlW3jGlkPlVzlU=; b=efBiqhraFzYfzkw0cfvgjaQhiKZYC3/Cn7WBaCUxVSUlTKfyRU0WdQS9okZTkd/oM+HIYO SNka23v+k0GCQvqeAIuteLjzHIJGOdTb1iJ9Ha/LbJNdbY3kMlk1bu9q0yg92DRUi8xti4 SObRyqoX6MTBekBu3BHWBd/mLL0K0LILDLxvtD9ruZiDcqsDEvK+4c8y6psFnAfUd/tqeT MXJn7vesqJWX0NDjPhP7ioiQDaaW9TzVbg/7ZFVW85veuhJ06MRCT5gTy4mQAAyRwnAdJ7 E0f2IhtU4dTsQMfzkKw7SsLXmcFmaO2ss4wvvhrvrb+u8S0I5qWX+pZdrELE3g== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1639966599; a=rsa-sha256; cv=none; b=dlUgx6XIhGjZdKNAmEBKXo9HOPuvRht4b1wB1Z6tQ/ayvuGuWRszuU/G59Jx6GQX88KRmi bSB3pYZnza3w4kiTBMxdS0NbfpUbV3EnU/1NkiPHULTjdS7OjDNxqvsa+TSJEm7SqWP/p/ 8RNBCBEbyHlfSvjRYpW4clZSlvfamre5cZBprPmP4KWd482hmj2KwSp5T0rjLakzh9LPtH eMVdCKcIb3IxIJd+bRbtlMoTH4wWNkEbR2w2MrAl6D617UrjPg8TQQ98JAzizBbamVnNzZ 9vrD+DB4iR+y9RPIWl/HpNWPsJp2kv4UT5Orwm5zqVX7YUTKN8l2O0cTp8ISZw== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=pass header.d=gmail.com header.s=20210112 header.b="YWzoFL2/"; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org" X-Migadu-Spam-Score: -9.22 Authentication-Results: aspmx1.migadu.com; dkim=pass header.d=gmail.com header.s=20210112 header.b="YWzoFL2/"; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org" X-Migadu-Queue-Id: 77BF9D9EB X-Spam-Score: -9.22 X-Migadu-Scanner: scn0.migadu.com X-TUID: WAueYw/t+WQR Hi Mathieu, Mathieu Othacehe writes: > Hello, > > You must have noticed that the CI is currently struggling a bit. Here is > a small recap of the situation. > > * The IO operations on Berlin are mysteriously slow. Removing files from > /gnu/store/trash is taking ages. This is reported here: > https://issues.guix.gnu.org/51787. > > We have to kill the garbage collect frequently to keep things > going. The bad side is obviously that we can't do that forever, as we > only have 9.3T and decreasing, while we aim to stay at 10T available. > > * The PostgreSQL database behind ci.guix.gnu.org also became super slow > and I decided to drop it. I don't know if there's a connection with > the above point. I'm missing the appropriate tools/knowledge to > monitor the IO & file-system performances. > > * The php package isn't building anymore, reported here: > https://issues.guix.gnu.org/52513. This means that we cannot > reconfigure zabbix. I removed it from the berlin configuration > temporarily. > > * The cuirass-remote-server Avahi service is no longer visible when > running "avahi-browse -a". I strongly suspect that this is related to > the static-networking update, even if I don't have a proof for > now. This means that the remote-workers using Avahi for discovering > (hydra-guix-*) machines can no longer connect. The > ci.guix.gnu.org/workers list is thus quite empty. > > * Facing those problems, I tried to rollback to a previous system > generation, but this is bringing even more issues, as for instance the > older Cuirass package, is struggling with the new database structure and > other niceties. I think out best course of action is to stick to > master and fix the above problems. Ooof, thanks a lot with reporting (and fixing) the above problems on top of baby sitting Cuirass while things stabilize... I'll try to keep an eye on Berlin IO activity to see if there are any offender consuming an abnormal amount of IO. Maxim