From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp1.migadu.com ([2001:41d0:403:58f0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms13.migadu.com with LMTPS id 4MtALKVOx2ZgWwAA62LTzQ:P1 (envelope-from ) for ; Thu, 22 Aug 2024 14:43:49 +0000 Received: from aspmx1.migadu.com ([2001:41d0:403:58f0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp1.migadu.com with LMTPS id 4MtALKVOx2ZgWwAA62LTzQ (envelope-from ) for ; Thu, 22 Aug 2024 16:43:49 +0200 X-Envelope-To: larch@yhetil.org Authentication-Results: aspmx1.migadu.com; dkim=pass header.d=debbugs.gnu.org header.s=debbugs-gnu-org header.b=olWkNFYV; dkim=fail ("headers rsa verify failed") header.d=gnu.org header.s=fencepost-gnu-org header.b=kYwkklOh; dmarc=pass (policy=none) header.from=gnu.org; spf=pass (aspmx1.migadu.com: domain of "bug-guix-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="bug-guix-bounces+larch=yhetil.org@gnu.org" ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1724337829; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:resent-cc: resent-from:resent-sender:resent-message-id:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post:dkim-signature; bh=ik7WZkMhi7HBGwOLH3EPM0Ie3i7z4JOF6QyyN5adrSo=; b=V4h/vXyUoW56IsZ+bL2IHxqIfDfAFvP0M5CZQfqHcPWZwSlYnInpwZl1wRtI3PWhikOXru US6EyIOdwgp8VojerRTz9u9WolLoqT5LkiovEG7408liWO8Yw+EZs6MVaK2i9AfNhMDWu4 jpiXUuvNs7XLANV7Qwp0dpWBK/x+BvQczLZJ2LkWE6QTZr0gWW186jBKRkYosdea3gJrut lVhA9SNFMyKuokFCosttRdyoc2hOttAUSVWZI7mtBmOs+uPgBxmTMWOfTNMDk1jwR0JUmS DlfWlN93W/fO2Wfot8oyUhLC50i5fHK2YygWF3re15fCsRPNSy/X8+P2J8/aVg== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1724337829; a=rsa-sha256; cv=none; b=CNIxlYme4f3MhI4QFhG6O4Eq5YRqa5G7LFG48hmB7cDd2C61L34u/gAH9Yf0FtxjjwrPZH O3EVHWwNtbhSnCJQ3UQLT3Tg7vWMRv6oGzN7RJNWqZ7Fq6XXTu1JZsxHTqVEmtfWeGXWlO 3Fy2z3M7GcKVwnrRRRcaQCOL9M7xleXlhzYEu/pYn2iCqO8Xcbw4oWVOaQm5huSbEoi3nH p2vG1J5QqhtjvUUe1I1DqEk2WQHS2uzNvinHS6znRrOxwleN2IoiZ62SITgGp+SWAzcLag khgfX+BamxyJNETZ4hKNjD8aNN9CCbpnnZsOrjTK1R4Dnup/Ng8ERuhpcJDM5g== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=pass header.d=debbugs.gnu.org header.s=debbugs-gnu-org header.b=olWkNFYV; dkim=fail ("headers rsa verify failed") header.d=gnu.org header.s=fencepost-gnu-org header.b=kYwkklOh; dmarc=pass (policy=none) header.from=gnu.org; spf=pass (aspmx1.migadu.com: domain of "bug-guix-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="bug-guix-bounces+larch=yhetil.org@gnu.org" Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 43A7962415 for ; Thu, 22 Aug 2024 16:43:48 +0200 (CEST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1sh91z-0001wd-60; Thu, 22 Aug 2024 10:43:19 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sh91x-0001wU-7s for bug-guix@gnu.org; Thu, 22 Aug 2024 10:43:17 -0400 Received: from debbugs.gnu.org ([2001:470:142:5::43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1sh91w-0001GG-UP for bug-guix@gnu.org; Thu, 22 Aug 2024 10:43:16 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=debbugs.gnu.org; s=debbugs-gnu-org; h=MIME-Version:Date:References:In-Reply-To:From:To:Subject; bh=ik7WZkMhi7HBGwOLH3EPM0Ie3i7z4JOF6QyyN5adrSo=; b=olWkNFYVM+RgCQNYKR5umhaUdFOZcw6HxrE8nrwdNoniZU0lb4JLXvpV+DetKRV3PWHRufPXKOahnk5Wf/EYg4DIN6FPZEUn6rLXi/8oxVZkH/syPaWTRe6FqehzCSIqL6DZHzX2D2yZIAiM9RbwiLID1NngP2V4M3gDAvKEIw73NZU3zKnJ5KJ9uNDCQC/BnFoIome0jmyJrZzJ7Um0xVcXlDdm8/EWvJkdpKsBLuB0MSkt7EaZqPbvvwe0BEwbTXeNS+YESMJSaxRZNVF4MnuszCUNDpZX38K6JvDXiavmh4mghKzVU/r9YsUC3WgxPQdxQIPAd0K/vrKkqgrzUg==; Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1sh92g-0004hX-HG for bug-guix@gnu.org; Thu, 22 Aug 2024 10:44:02 -0400 X-Loop: help-debbugs@gnu.org Subject: bug#72722: [cuirass] Failure to write build log leads to build failure Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: bug-guix@gnu.org Resent-Date: Thu, 22 Aug 2024 14:44:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 72722 X-GNU-PR-Package: guix X-GNU-PR-Keywords: To: 72722@debbugs.gnu.org Received: via spool by 72722-submit@debbugs.gnu.org id=B72722.172433779918003 (code B ref 72722); Thu, 22 Aug 2024 14:44:02 +0000 Received: (at 72722) by debbugs.gnu.org; 22 Aug 2024 14:43:19 +0000 Received: from localhost ([127.0.0.1]:38184 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1sh91y-0004gH-Oz for submit@debbugs.gnu.org; Thu, 22 Aug 2024 10:43:19 -0400 Received: from eggs.gnu.org ([209.51.188.92]:50052) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1sh91u-0004fz-R3 for 72722@debbugs.gnu.org; Thu, 22 Aug 2024 10:43:17 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sh915-0001D9-8l for 72722@debbugs.gnu.org; Thu, 22 Aug 2024 10:42:23 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:Date:References:In-Reply-To:Subject:To: From; bh=ik7WZkMhi7HBGwOLH3EPM0Ie3i7z4JOF6QyyN5adrSo=; b=kYwkklOh9uobaZsNHgke 3znJjxkevmBydbC4RNSjyHqBKVaiU93X5SOEAxw3mFhVl89B5syRpPK7UHPDi+oNOnWyW8Bg72G6m nNLxOn5KXprE00tZN9Y+krFMBhO4JsIvW9AwMwBfFSIFxqJOj04PS9ypfYi0WN3lpQfFgpQSwx8OQ qImSuEJPnEf8HnbnrLNBxmi/CFTNFyAeND8obRHkGNKtBhRkBmuWprqVkSTmYa4rIr8xpZ2S/SBg6 Sps/tlv3C+f1//rSGfzCkjW9luitbuDEXammeTu8DzhSQ5dOWg81De1RU/u/IGiWE8RpSVPEFhLyw HwAaT26f2nBfhg==; From: Ludovic =?UTF-8?Q?Court=C3=A8s?= In-Reply-To: <878qwsuodx.fsf@inria.fr> ("Ludovic =?UTF-8?Q?Court=C3=A8s?="'s message of "Tue, 20 Aug 2024 00:41:14 +0200") References: <878qwsuodx.fsf@inria.fr> Date: Thu, 22 Aug 2024 16:42:20 +0200 Message-ID: <87ttfcliur.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-guix@gnu.org List-Id: Bug reports for GNU Guix List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-guix-bounces+larch=yhetil.org@gnu.org Sender: bug-guix-bounces+larch=yhetil.org@gnu.org X-Migadu-Flow: FLOW_IN X-Migadu-Country: US X-Migadu-Spam-Score: -6.08 X-Spam-Score: -6.08 X-Migadu-Queue-Id: 43A7962415 X-Migadu-Scanner: mx11.migadu.com X-TUID: gmqWYKbhxj6Y Hi, Ludovic Court=C3=A8s skribis: > We occasionally see failed builds with truncated logs on ci.guix. These > happens in situations where =E2=80=98cuirass remote-worker=E2=80=99 gets = EPIPE as it > sends the build log to =E2=80=98remote-server=E2=80=99: > > 2024-08-19 19:54:52 @ substituter-started /gnu/store/sv3z77cgg2788hrl87w3= 5bfmyhkmkv54-libomp-16.0.6.drv substitute > 2024-08-19 19:54:52 Downloading http://141.80.167.131/nar/lzip/sv3z77cgg2= 788hrl87w35bfmyhkmkv54-libomp-16.0.6.drv... > 2024-08-19 19:54:52=20 > 2024-08-19 19:54:52 ESC[K libomp-16.0.6.drv 1.= 8MiB/s 00:00 | 1KiB transferred > 2024-08-19 19:54:52 ESC[K libomp-16.0.6.drv 94= 2KiB/s 00:00 | 1KiB transferred > 2024-08-19 19:54:52=20 > 2024-08-19 19:54:52 @ substituter-succeeded /gnu/store/sv3z77cgg2788hrl87= w35bfmyhkmkv54-libomp-16.0.6.drv > 2024-08-19 19:55:04 warning: zlib error in 'gzwrite' while sending log to= 141.80.167.131: 0 > 2024-08-19 19:55:04 error: gdPO1dI1: unexpected error while building '/gn= u/store/sv3z77cgg2788hrl87w35bfmyhkmkv54-libomp-16.0.6.drv': #<&compound-ex= ception components: (#<&external-error> #<&origin origin: "fport_write"> #<= &message message: "~A"> #<&irritants irritants: ("Broken pipe")> #<&excepti= on-with-kind-and-args kind: system-error args: ("fport_write" "~A" ("Broken= pipe") (32))>)> But hey, why does =E2=80=98gzwrite=E2=80=99 fail in the first place? I noticed that this usually happened when dumping big logs (several MiBs) very quickly (typically the unpack phase of a large package like LLVM producing lots of data very quickly.) As it turns out, =E2=80=98send-log=E2=80=99 opens its socket with SOCK_NONB= LOCK, and then passes it to zlib, which writes to it in =E2=80=98gzwrite=E2=80=99. B= ut zlib is not equipped to deal with EAGAIN: it just errors out, with =E2=80=98gzwrite= =E2=80=99 returning Z_ERRNO, hence the bug above. I was able to confirm this hypothesis by running: echo '(log-server (version 0))' | nc -l -p 5000 -v | \ (sleep 10; echo starting >&2; wc -c) and then, from a REPL: scheme@(cuirass remote)> (send-log "127.0.0.1" 5000 "foo.drv" (open-input= -file "llvm.log")) 2024-08-22T16:35:37 warning: zlib error in 'gzwrite' while sending log to= 127.0.0.1: -1 : Resource temporarily unavailable $30 =3D #f QED. (Here I used Guile-zlib 0.2.1 with a small modification to =E2=80=98remote.scm=E2=80=99 so it displays the error message after Z_ERRNO= =3D -1.) Ludo=E2=80=99.