From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp1.migadu.com ([2001:41d0:303:e16b::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms1.migadu.com with LMTPS id iB3CEIEgG2Y6fQAA62LTzQ:P1 (envelope-from ) for ; Sun, 14 Apr 2024 02:17:05 +0200 Received: from aspmx1.migadu.com ([2001:41d0:303:e16b::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp1.migadu.com with LMTPS id iB3CEIEgG2Y6fQAA62LTzQ (envelope-from ) for ; Sun, 14 Apr 2024 02:17:05 +0200 X-Envelope-To: larch@yhetil.org Authentication-Results: aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=protonmail.com header.s=protonmail3 header.b=tW7fVttC; spf=pass (aspmx1.migadu.com: domain of "bug-guix-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="bug-guix-bounces+larch=yhetil.org@gnu.org"; dmarc=pass (policy=none) header.from=gnu.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1713053825; h=from:from:sender:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:resent-cc: resent-from:resent-sender:resent-message-id:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post:dkim-signature; bh=abTz5za9O36tpuSSb9MFhXTE+Ft24TMU+q2/Ysj4KJc=; b=ZRAXEZZuNPFIn8vQdmv2YOQjHcFljcEZM7DWrOByqPo8jXMpYfApU3EaZH/XK79kYjAH2A xLm7Q9v0lV1Ib45BMyeeMbIOjQeynp7mwVc9LMgm+oIHPVz5M9QLlOk1ZfLbBw8W2LI8pj rH03M5ecnhJkuJ6mLoR+4wv0X80e+LZAKWVacqw9E7kjjmwGv5xr40dcUUexR6tyiNqenj z/7tWtJhDmFkxWAzu2uQt7enu3qNSfR3CM6XWIOYkHsllnV7M12i3DQ5gmHEcSc9kvADYL 6Stx0AAJwmU5JoBYLbfM5d8EY9szRVpK+sCBLxnb0JrzTCUNougUpD84wgO8Ag== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1713053825; a=rsa-sha256; cv=none; b=EDlAXdb9/lKwtPrzzf0jXiTAqiTdIAOPAomCcjDdgFi1EJmSGX22rCr4eLuYpQHeHwhUWK 9bBoyiZIs2eOnwRbvzMeLhDNn0lv9T40gilSR/kZhALeQ4gTnGhCaBlOv0ZF1hnFaBO0it xtHvrKnvUJWgUVXHgyycBq+ov6x7m/YzlhYLoKpEFhKgpysYciaabtT0HgkEhfBMVYiKJf FNT9mlyrHwSsl4CfC5qPefSYwulSx391VoKQDszIpIlFX6jAKT5OKhUQ9HxUKfby8zCg03 1MF9TsLqr8hwV8n2xZdAhdo3ABwHQVyFF1WMCG7z3rgvb7hvHaAdvB0E1s0bQg== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=protonmail.com header.s=protonmail3 header.b=tW7fVttC; spf=pass (aspmx1.migadu.com: domain of "bug-guix-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="bug-guix-bounces+larch=yhetil.org@gnu.org"; dmarc=pass (policy=none) header.from=gnu.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id E6CB263B20 for ; Sun, 14 Apr 2024 02:17:04 +0200 (CEST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rvnYE-00033p-UR; Sat, 13 Apr 2024 20:16:54 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rvnYD-00033g-Jz for bug-guix@gnu.org; Sat, 13 Apr 2024 20:16:53 -0400 Received: from debbugs.gnu.org ([2001:470:142:5::43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1rvnYD-0006yw-9k for bug-guix@gnu.org; Sat, 13 Apr 2024 20:16:53 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1rvnYN-0006Wp-ST for bug-guix@gnu.org; Sat, 13 Apr 2024 20:17:03 -0400 X-Loop: help-debbugs@gnu.org Subject: bug#54447: cuirass: missing derivation error References: <877d8r4etz.fsf@gnu.org> In-Reply-To: <877d8r4etz.fsf@gnu.org> Resent-From: John Kehayias Original-Sender: "Debbugs-submit" Resent-CC: bug-guix@gnu.org Resent-Date: Sun, 14 Apr 2024 00:17:03 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 54447 X-GNU-PR-Package: guix X-GNU-PR-Keywords: To: Ludovic =?UTF-8?Q?Court=C3=A8s?= Cc: 54447@debbugs.gnu.org, guix-sysadmin , Maxim Cournoyer , Mathieu Othacehe Received: via spool by 54447-submit@debbugs.gnu.org id=B54447.171305377824673 (code B ref 54447); Sun, 14 Apr 2024 00:17:03 +0000 Received: (at 54447) by debbugs.gnu.org; 14 Apr 2024 00:16:18 +0000 Received: from localhost ([127.0.0.1]:33997 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rvnXb-0006PR-QD for submit@debbugs.gnu.org; Sat, 13 Apr 2024 20:16:17 -0400 Received: from mail-4316.protonmail.ch ([185.70.43.16]:25221) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rvnXU-0006Mn-24 for 54447@debbugs.gnu.org; Sat, 13 Apr 2024 20:16:13 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=protonmail.com; s=protonmail3; t=1713053751; x=1713312951; bh=abTz5za9O36tpuSSb9MFhXTE+Ft24TMU+q2/Ysj4KJc=; h=Date:To:From:Cc:Subject:Message-ID:Feedback-ID:From:To:Cc:Date: Subject:Reply-To:Feedback-ID:Message-ID:BIMI-Selector; b=tW7fVttCg2BP6T9zbGTd96P4b9ZaInTZpyQup4JFajUjbBnE/slGtgbcd8XtgNQ9I Exb6TWKh+TRJ3Twf3p3Dma9RMCN1uSXM9mYvdLbwFFZ80fVpBqTyZThR6CTDWtCZNS YSUO8D0uUsHBqdDMat/W3y+fLmH49h2RbNh53hxEQVS7QhqFmSuRBq7J0PmdKMFqsA aemqKaNL9FNYuQz9hknRK/iu4p6CfEA1E+nDduchuuAlo/N4s44S7c3yK1TSqGj+3D H8mx164l0hUnRLyEDeB0/AmIn90sbOxsdyln2X0FHuDYrRI15ggMU8xfAYxAeA0a/N C5HwvL0X95Zeg== Date: Sun, 14 Apr 2024 00:15:45 +0000 Message-ID: <87mspwbxzn.fsf@protonmail.com> Feedback-ID: 7805494:user:proton MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-guix@gnu.org List-Id: Bug reports for GNU Guix List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-to: John Kehayias From: John Kehayias via Bug reports for GNU Guix Errors-To: bug-guix-bounces+larch=yhetil.org@gnu.org Sender: bug-guix-bounces+larch=yhetil.org@gnu.org X-Migadu-Country: US X-Migadu-Flow: FLOW_IN X-Migadu-Spam-Score: -5.38 X-Spam-Score: -5.38 X-Migadu-Queue-Id: E6CB263B20 X-Migadu-Scanner: mx13.migadu.com X-TUID: bgyW8CfeCCMu Hi all, On Thu, Apr 04, 2024 at 11:33 PM, Ludovic Court=C3=A8s wrote: > Hello! > > News from the everlasting bug! > > cannot build missing derivation > =E2=80=98/gnu/store/dfgc46q3l8wlnymv49a1wjnxypin8p0y-plink-1.07.drv= =E2=80=99 > > (From .) > > Why was it missing this time? /var/log/nginx/error.log: > > 2024/04/04 17:15:03 [error] 98751#0: *152293778 upstream timed out (110: = Connection timed out) while reading response header from upstream, client: = 141.80.167.169, server: ci.guix.gnu.org, request: "GET /dfgc46q3l8wlnymv49a= 1wjnxypin8p0y.narinfo HTTP/1.1", upstream: "http://127.0.0.1:3000/dfgc46q3l= 8wlnymv49a1wjnxypin8p0y.narinfo", host: "141.80.167.131" > > > Oops! (There are dozens of upstream timeouts logged on that minute.) > > /var/log/guix-publish.log: > > 2024-04-04 17:14:51 GET /nar/lzip/pz39bkq7pd1hgy5rwiynqa33gyjvpgs5-python= -pygments-2.12.0 > 2024-04-04 17:14:51 GET /z2xxwwxswdd4b8c8iwmxhqnqbp5nwz09.narinfo > 2024-04-04 17:14:51 GET /lgyck285bsxzwrnh3x5ix5dwzd3n3wga.narinfo > 2024-04-04 17:14:51 GET /nar/zstd/jxkglr445f215m2faqz1i2lgmbans4rf-texliv= e-amsmath-66594-doc > 2024-04-04 17:15:33 GET /qg5cxb869i42jn7x2dm6k5l41ikkz21w.narinfo > 2024-04-04 17:15:33 GET /nar/zstd/i2hp3q2pfhsyl0al7z38am7cqpddi4qr-texliv= e-capt-of-66594-doc > 2024-04-04 17:15:33 GET /hh0gdbljj3cjdnjbr88kfm21mhys5sy7.narinfo > 2024-04-04 17:15:33 GET /dfgc46q3l8wlnymv49a1wjnxypin8p0y.narinfo > 2024-04-04 17:15:33 GET /yj63wifalfr6sla42h7mkqg011qrl5d0.narinfo > 2024-04-04 17:15:33 GET /h2s2g2adxbnd67g34mnjnpcr6p3nhr69.narinfo > 2024-04-04 17:15:33 -> GET /h2s2g2adxbnd67g34mnjnpcr6p3nhr69.narinfo: 404 > 2024-04-04 17:15:33 GET /nar/lzip/6zxlrw15b9dsv73s7v5fqabl7iv5v5il-python= -exceptiongroup-1.1.1 > 2024-04-04 17:15:33 GET /nar/zstd/pychjd114abscbqlzcr3s7myf1497vw2-julia-= compilersupportlibraries-jll-0.4.0%2B1 > > =E2=80=98guix publish=E2=80=99 replied, but 40s too late (nginx has > =E2=80=9Cproxy_connect_timeout 10s;=E2=80=9D for .narinfo URLs=C2=B9). > > Notice the 40s pause time between 17:14:51 and 17:15:33. Stop-the-world > GC? Unlikely, because =E2=80=98guix publish=E2=80=99 had been running fo= r ~3h, so even > with a leak=C2=B2, it=E2=80=99s hard to believe GC could take this long. > > Ludo=E2=80=99. > > =C2=B9 > https://git.savannah.gnu.org/cgit/guix/maintenance.git/tree/hydra/nginx/b= erlin.scm#n103 > =C2=B2 https://issues.guix.gnu.org/69596 I don't have any insight, but if anyone wants to see this in action at a large scale, take look at pretty much any red dot on https://ci.guix.gnu.org/eval/1238471/dashboard?system=3Di686-linux >From my quick look all the CL and texlive failures were all missing derivation. I've tried restarting a bunch to get i686 coverage going, so hopefully some will disappear. But I can't/won't manually restart the thousands(?) of failed builds. I didn't see such issues on x86_64, while other architectures take a really long time to build on Berlin so I haven't looked. I don't know if this is helpful, but thought I would chime in if anyone wants potentially a bunch of data. And if there are good ideas to recover (just restart all builds?) that would be great so mesa-updates will be build on i686 since otherwise it looks good. Thanks! John