From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp12.migadu.com ([2001:41d0:403:4789::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms9.migadu.com with LMTPS id Y7EDLegt5GSKjgAASxT56A (envelope-from ) for ; Tue, 22 Aug 2023 05:39:20 +0200 Received: from aspmx1.migadu.com ([2001:41d0:403:4789::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp12.migadu.com with LMTPS id +HlNLOgt5GRebQEAauVa8A (envelope-from ) for ; Tue, 22 Aug 2023 05:39:20 +0200 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 1A34D57161 for ; Tue, 22 Aug 2023 05:39:20 +0200 (CEST) Authentication-Results: aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=gmail.com header.s=20221208 header.b=B0L3LRwd; spf=pass (aspmx1.migadu.com: domain of "bug-guix-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="bug-guix-bounces+larch=yhetil.org@gnu.org"; dmarc=fail reason="SPF not aligned (relaxed)" header.from=gmail.com (policy=none) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1692675560; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:resent-cc: resent-from:resent-sender:resent-message-id:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post:dkim-signature; bh=ZRD1hTZ8AGpwGrr9aAi/kGzPIafQBRxxssm+cstrHlU=; b=VEiu3KrAiTFdiWkFtIxRR3TF4jdfdhu4PQ25simRefE9cd69fQWmKDAAHv0sG4EnXkK3T7 N0wo1n+k+cPmEG7nBfZNBK2eJhmkgNM7SHuhDCIzD9N2V0wxU73k73XC8idHR3ol0P2z2f tObJJzlPuH1Z0WI4q3zw4+7oEK9K5St/PvmGd1TgwfQ5Q9+rS5xDlTJyzohldvtzALpt75 UkfbfE7Ap+pZF1WbARaJJ6R1Qqq0UkDhrCtjAdM+I81yD5qir5kK1QaNAyJ8fijD43e3Qn G8Uleu+lff0AKKBiBgWAuB6fVxDVhe5YGbq6mfozP9HgeGh9tc4PoHnVPZmPlg== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1692675560; a=rsa-sha256; cv=none; b=N7ioKYnKWVYWHuF2CR7JkFRWuIgrz0Z/zmdFVNCpwKBCWgUhst0RC4CYj/odmticCmdhrv CuynXPwLtRdg1jn/oP+mY1yJ2GuTXxvUEopJW2/jWZOVqmSs0FkWrRPcBcIei7FOHb8RjM VpIurzBVOfLIbmOuO0T3ICkQNVTZwWOnSpKJoLpZRR+Fng0Y8V1JnLGBGYY2izgbom11Xh 9RxQ1t8Lebn4sNBQiwYm908xRv/Y7jgaJs9VyrCFwNyiJbGgQzymM7hwwjw+Dxf2Jv4GL3 VzvEMPnrGqWKd9z4AZWwa81OM53woXp7/khOVG2pBC4e8eCyYgDW1wh1SpI1fA== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=gmail.com header.s=20221208 header.b=B0L3LRwd; spf=pass (aspmx1.migadu.com: domain of "bug-guix-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="bug-guix-bounces+larch=yhetil.org@gnu.org"; dmarc=fail reason="SPF not aligned (relaxed)" header.from=gmail.com (policy=none) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qYIEP-0000Ox-Ao; Mon, 21 Aug 2023 23:39:01 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qYIEO-0000Oo-Hw for bug-guix@gnu.org; Mon, 21 Aug 2023 23:39:00 -0400 Received: from debbugs.gnu.org ([2001:470:142:5::43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1qYIEO-0005UD-AM for bug-guix@gnu.org; Mon, 21 Aug 2023 23:39:00 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1qYIEQ-00087m-G2 for bug-guix@gnu.org; Mon, 21 Aug 2023 23:39:02 -0400 X-Loop: help-debbugs@gnu.org Subject: bug#54447: cuirass: missing derivation error Resent-From: Maxim Cournoyer Original-Sender: "Debbugs-submit" Resent-CC: bug-guix@gnu.org Resent-Date: Tue, 22 Aug 2023 03:39:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 54447 X-GNU-PR-Package: guix X-GNU-PR-Keywords: To: Mathieu Othacehe Cc: 54447@debbugs.gnu.org Received: via spool by 54447-submit@debbugs.gnu.org id=B54447.169267553531215 (code B ref 54447); Tue, 22 Aug 2023 03:39:02 +0000 Received: (at 54447) by debbugs.gnu.org; 22 Aug 2023 03:38:55 +0000 Received: from localhost ([127.0.0.1]:58222 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1qYIEI-00087O-Fs for submit@debbugs.gnu.org; Mon, 21 Aug 2023 23:38:54 -0400 Received: from mail-qk1-x734.google.com ([2607:f8b0:4864:20::734]:46537) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1qYIEG-00087B-3r for 54447@debbugs.gnu.org; Mon, 21 Aug 2023 23:38:53 -0400 Received: by mail-qk1-x734.google.com with SMTP id af79cd13be357-76d7bcefe8bso264400585a.1 for <54447@debbugs.gnu.org>; Mon, 21 Aug 2023 20:38:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1692675523; x=1693280323; h=content-transfer-encoding:mime-version:user-agent:message-id :in-reply-to:date:references:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=ZRD1hTZ8AGpwGrr9aAi/kGzPIafQBRxxssm+cstrHlU=; b=B0L3LRwdSp8U+nC+twjbno0+vIHr7s+GQQIuHhzA0IK6sArxwJgpgyfjaGFs46hHWj FbcyInPA78BfOSz6hPip5yBxVUwJcFXrHuAfMC8/MHk043Uafdb8/nPmlZUAtW/aqHmn JfAO4Te5DKpyFuWg71O0FafNbIWEuRN086qYMAqhrRrYTWN3+l2Mxdd6SBHlAILSafJH Fu0P4bjLBoDlNiDou1GIARsrMOnO4kuRdNTM8IBDmAU0tXEpH+DYRdWtEC3AeC5TPgia cCJBgPZXV+mH4QYF29rY7UDDd62V1MSXgREfpDPhDvX0stbMphSDCz0CZq1tqec0s9+Y 4wqg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1692675523; x=1693280323; h=content-transfer-encoding:mime-version:user-agent:message-id :in-reply-to:date:references:subject:cc:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=ZRD1hTZ8AGpwGrr9aAi/kGzPIafQBRxxssm+cstrHlU=; b=UbYAZFPebBOJ5pSsy7TmYijz8B/pPSDYTkmftspyKYlCIaQCzBVa4K7oLaiLdhIqfR tDUmUB+aKKJVnfLZygJ2vyqZXF93Si0KI2Bo0d0tv8+DIJ7dMnrQc37sfUtx3qtps2Zw ws63uQHrRcvkeoCCBCsnvQSLOMAiooV2EmZoas75c572CN+p0s7YP+/BBCAR2dfxyuML Zi4M4Wgtg0lVvMZydPVTxWqPW4R7DqBh5NDrLGcF0RpcSUlThBf/ttBTgK8jiog02IkK NXPf1scpnHCEbgp3d2T1vVolVOJKrvFO5Nxzap0nUBKGoxe40HdhptLrI0Kb1RYQK6jU IJbw== X-Gm-Message-State: AOJu0YyU5jpIa/8kECMhwuFIlN6YsBS34hHeHr0s9T21yjkxa4C+OKpP DDEeuUdttbKAXpXgRjraOSrOMmrWmdk= X-Google-Smtp-Source: AGHT+IHXDDgoxmWk4jpQreMim3eLRTHhXjhiejoEJSK1S1+8KhmKDpVZOMIVx74NjQCqQEAm/stG/Q== X-Received: by 2002:a05:620a:b58:b0:76d:a70f:1bb4 with SMTP id x24-20020a05620a0b5800b0076da70f1bb4mr4424389qkg.66.1692675523571; Mon, 21 Aug 2023 20:38:43 -0700 (PDT) Received: from hurd (dsl-10-148-105.b2b2c.ca. [72.10.148.105]) by smtp.gmail.com with ESMTPSA id c1-20020a05620a200100b0076ca401d8c7sm2953675qka.111.2023.08.21.20.38.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 21 Aug 2023 20:38:42 -0700 (PDT) From: Maxim Cournoyer References: <877d8r4etz.fsf@gnu.org> Date: Mon, 21 Aug 2023 23:38:41 -0400 In-Reply-To: <877d8r4etz.fsf@gnu.org> (Mathieu Othacehe's message of "Fri, 18 Mar 2022 13:36:56 +0100") Message-ID: <87ttsrbvb2.fsf@gmail.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-guix@gnu.org List-Id: Bug reports for GNU Guix List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-guix-bounces+larch=yhetil.org@gnu.org Sender: bug-guix-bounces+larch=yhetil.org@gnu.org X-Migadu-Flow: FLOW_IN X-Migadu-Country: US X-Migadu-Spam-Score: -5.11 X-Spam-Score: -5.11 X-Migadu-Queue-Id: 1A34D57161 X-Migadu-Scanner: mx1.migadu.com X-TUID: bdOdD1Fp21Px Hello, Mathieu Othacehe writes: > Hello, > > A lot of builds, among them ~20 system tests[1], are failing with: > "cannot build missing derivation > ?/gnu/store/hs6kp1lqgymhyp3jndc0dsp0pn4psgv0-gui-installed-desktop-os-enc= rypted.drv?" > errors. > > Those derivations are present on the CI head node. This means that the > errors occur during substitution. This is most likely caused by some > issue with the publish server, because: > > - The publish server serves a 404 error. We should get rid once and for > all of this 404 thing, pushing something like: > https://issues.guix.gnu.org/50040. > > or > > - The publish server is not fast enough and hits an Nginx timeout that > closes the communication. > > Any other cause I could be missing? Looking at multiple of recent 'cannot build missing derivation' build failures on Cuirass, I see for example: --8<---------------cut here---------------start------------->8--- substitute:=20 substitute: [Kupdating substitutes from 'http://141.80.167.131'... 0.0% substitute: [Kcould not fetch http://141.80.167.131/rhgrs3ac6h64siz0krqh2ia= 8kkn3h6ym.narinfo 504 substitute: updating substitutes from 'http://141.80.167.131'... 100.0% cannot build missing derivation ?/gnu/store/rhgrs3ac6h64siz0krqh2ia8kkn3h6y= m-python-asdf-standard-1.0.3.drv? --8<---------------cut here---------------end--------------->8--- So it seems the error originated from guix-publish being too heavily under load to produce a timely reply, and the nginx proxy issued a 504 (timeout) error response. Looking into /var/log/guix-publish.log for a corresponding entry, I found: --8<---------------cut here---------------start------------->8--- 2023-08-21 23:59:35 GET /rhgrs3ac6h64siz0krqh2ia8kkn3h6ym.narinfo 2023-08-21 23:59:35 In web/server/http.scm: 2023-08-21 23:59:35 159:7 2 (http-write #< socket: # =E2=80=A6) 2023-08-21 23:59:35 In unknown file: 2023-08-21 23:59:35 1 (put-bytevector #= #vu8(83 # =E2=80=A6) =E2=80=A6) 2023-08-21 23:59:35 In ice-9/boot-9.scm: 2023-08-21 23:59:35 1685:16 0 (raise-exception _ #:continuable? _) 2023-08-21 23:59:35 In procedure fport_write: Broken pipe --8<---------------cut here---------------end--------------->8--- So the connection was apparently severed (?), resulting in the "broken pipe" error. Here's a different one: --8<---------------cut here---------------start------------->8--- substitute:=20 substitute: [Kupdating substitutes from 'http://141.80.167.131'... 0.0% substitute: [Kcould not fetch http://141.80.167.131/p2lfyvbxicjqsm4qp6368bx= 76gp0g948.narinfo 504 substitute: updating substitutes from 'http://141.80.167.131'... 100.0% cannot build missing derivation ?/gnu/store/p2lfyvbxicjqsm4qp6368bx76gp0g94= 8-python-astropy-healpix-0.7.drv? --8<---------------cut here---------------end--------------->8--- it occurred around the same time, and the failing mode was the same, per guix-publish.log: --8<---------------cut here---------------start------------->8--- 2023-08-21 23:59:35 GET /p2lfyvbxicjqsm4qp6368bx76gp0g948.narinfo 2023-08-21 23:59:35 In web/server/http.scm: 2023-08-21 23:59:35 159:7 2 (http-write #< socket: # =E2=80=A6) 2023-08-21 23:59:35 In unknown file: 2023-08-21 23:59:35 1 (put-bytevector #= #vu8(83 # =E2=80=A6) =E2=80=A6) 2023-08-21 23:59:35 In ice-9/boot-9.scm: 2023-08-21 23:59:35 1685:16 0 (raise-exception _ #:continuable? _) 2023-08-21 23:59:35 In procedure fport_write: Broken pipe --8<---------------cut here---------------end--------------->8--- I wonder if these could be related to the DDoS protection discovered on the Berlin network. I'll keep looking for other, potentially different occurrences. --=20 Thanks, Maxim