From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp11.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms5.migadu.com with LMTPS id OCJlExVUeWKBCAEAbAwnHQ (envelope-from ) for ; Mon, 09 May 2022 19:49:09 +0200 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp11.migadu.com with LMTPS id SJuBExVUeWKBGwEA9RJhRA (envelope-from ) for ; Mon, 09 May 2022 19:49:09 +0200 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id D3E83D758 for ; Mon, 9 May 2022 19:49:08 +0200 (CEST) Received: from localhost ([::1]:57112 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1no7VL-0002Mo-OO for larch@yhetil.org; Mon, 09 May 2022 13:49:07 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:47454) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1no7SM-00085X-Ay for guix-patches@gnu.org; Mon, 09 May 2022 13:46:02 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:37207) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1no7SM-0002bv-0O for guix-patches@gnu.org; Mon, 09 May 2022 13:46:02 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1no7SL-000626-SJ for guix-patches@gnu.org; Mon, 09 May 2022 13:46:01 -0400 X-Loop: help-debbugs@gnu.org Subject: [bug#55242] [PATCH 07/10] guix: import: go: More resilience wrt network errors; add logging. Resent-From: Maxime Devos Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Mon, 09 May 2022 17:46:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 55242 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Attila Lendvai Cc: 55242@debbugs.gnu.org Received: via spool by 55242-submit@debbugs.gnu.org id=B55242.165211831723130 (code B ref 55242); Mon, 09 May 2022 17:46:01 +0000 Received: (at 55242) by debbugs.gnu.org; 9 May 2022 17:45:17 +0000 Received: from localhost ([127.0.0.1]:59337 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1no7Rd-000610-EB for submit@debbugs.gnu.org; Mon, 09 May 2022 13:45:17 -0400 Received: from andre.telenet-ops.be ([195.130.132.53]:42950) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1no7Ra-00060m-To for 55242@debbugs.gnu.org; Mon, 09 May 2022 13:45:16 -0400 Received: from ptr-bvsjgyhxw7psv60dyze.18120a2.ip6.access.telenet.be ([IPv6:2a02:1811:8c09:9d00:3c5f:2eff:feb0:ba5a]) by andre.telenet-ops.be with bizsmtp id UhlC2700U4UW6Th01hlDnV; Mon, 09 May 2022 19:45:13 +0200 Message-ID: <91eb2af9b86d2491c9e9f9010b9e3cb2d05dd253.camel@telenet.be> From: Maxime Devos Date: Mon, 09 May 2022 19:45:07 +0200 In-Reply-To: References: <20220503114301.9524-2-attila@lendvai.name> <20220503114301.9524-7-attila@lendvai.name> <7974a00256e3ee153507f77d6b0a405473e9bbc4.camel@telenet.be> <76510182d505978ad41f5bff88bf7eee7c289558.camel@telenet.be> Content-Type: multipart/signed; micalg="pgp-sha512"; protocol="application/pgp-signature"; boundary="=-43xsJYWm2329njOGkNDs" User-Agent: Evolution 3.38.3-1 MIME-Version: 1.0 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=telenet.be; s=r22; t=1652118313; bh=Yh4IDVxEmjtr8D7fkSbuH6hxSamTYKjjkr6AR1h+ENA=; h=Subject:From:To:Cc:Date:In-Reply-To:References; b=auWwrbnMpyfWt6kW6CI3om5PGdG9gbv6ulai59vYD2+rMCRPpbYYiAV3MA6pTWnNc GOR8LIl5RRYRJgA/TuAmXYXp2PszyfDqFlKiie6cR5vLNmlH/crGol9lbDvhjUWmcI O0wB11YCC4PEY/XEse5SuUkTSVV250UmyjwMwSmKSQ3Tq2cQpwTk34N9hZAXj95KOB L94OEMkY/dpoKOOvWBZiHo9Y14Z1g9vJXjmP0rxRXtGHRWgCLJKbtw7YIEJvFbAZa2 MzTF0qdh6hFJIO++PVz9HjFgQK7GOWq+flAf+7G4hGx5ImsSoVZ3qCXsPGby+u3Wr+ 5WwE9MvMJUcAg== X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+larch=yhetil.org@gnu.org Sender: "Guix-patches" X-Migadu-Flow: FLOW_IN X-Migadu-To: larch@yhetil.org X-Migadu-Country: US ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1652118549; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:resent-cc:resent-from:resent-sender: resent-message-id:in-reply-to:in-reply-to:references:references: list-id:list-help:list-unsubscribe:list-subscribe:list-post: dkim-signature; bh=Yh4IDVxEmjtr8D7fkSbuH6hxSamTYKjjkr6AR1h+ENA=; b=dPnTUgYvTc/hk4X4eBLfa0LoSn2gq5QLAJLdYLbnQiAn+ehycF2bFesHoJ9YaOX6uqL8nI T0v+XH6IsLd607BeQU0Hci9Cjj6v77oqEsFUlS2qc8W4LrqUsza8WDKi765sv2m8REIUn5 vi/Ii5zWpeoB9mhziuF1RGo0CUv/d1EQ/+xqUxw0eEtbAhek4D2MxsZSr0X32QTq/fE/bH Lyz0pT6UFS2+jKhrmdUaNe+B6SWc8+/6IdkxIJNGXi8SAV1vsk5VOr5GaVdWBXS7tR8sxA i6pIU3D3/WoYZ5Dv8R9ZpTWb9+rtWPZcVTy/FuN4scjMxCWzclae5y66VSCduA== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1652118549; a=rsa-sha256; cv=none; b=UlGRINMFBu+DNinXN8znMqfMbwUfjfBBGNa0nelk3wsTdl/GKeUfmHIO3CoZk2patVguvM 3phypdI6tC9J81k9Gog/e955rlQHhIHX4fsDtI4fDXI4KT36VIBM0rXKWVRYFU1txuQakY eeDEWHI93CQ/4ahSE1fPmElsthbbqDJV6dbbZtzUmth72FSTEeUwty7RVZAKuwyo0k3wRx V02DdKteUzWNXMJjpN6paxgOnTRoDY6vOQwHY0gDkjafWVemKYBI/fvQ0LJ+H5RpCfKF4r tHj3YUMPxTX9FXpD+XIUE6SDoVvmFs7F6uslGJIs74jYxyUZC2oVWFWmLBcm9A== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=telenet.be header.s=r22 header.b=auWwrbnM; dmarc=fail reason="SPF not aligned (relaxed)" header.from=telenet.be (policy=none); spf=pass (aspmx1.migadu.com: domain of "guix-patches-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-patches-bounces+larch=yhetil.org@gnu.org" X-Migadu-Spam-Score: 3.90 Authentication-Results: aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=telenet.be header.s=r22 header.b=auWwrbnM; dmarc=fail reason="SPF not aligned (relaxed)" header.from=telenet.be (policy=none); spf=pass (aspmx1.migadu.com: domain of "guix-patches-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-patches-bounces+larch=yhetil.org@gnu.org" X-Migadu-Queue-Id: D3E83D758 X-Spam-Score: 3.90 X-Migadu-Scanner: scn1.migadu.com X-TUID: R8wulcfrYEBh --=-43xsJYWm2329njOGkNDs Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Attila Lendvai schreef op ma 09-05-2022 om 12:34 [+0000]: > [...] does it make enough sense to justify my proposed behavior? (For a positive note, skip to the last paragraph!) If we do something like this (*) (this =3D catching errors for dependencies and skipping those packages), I don't want this to be something Go-specific. Could this be done in 'recursive-import' or something like that instead, such that all importers benefit and Go isn't some weird special case? Also, given that we appear to have a number of people effectively maintaining the Go importer ("git log guix/import.go.scm" mentions a few =E2=80=98Maxim Cournoyer=E2=80=99, =E2=80=98Ludovic Court=C3=A8s=E2=80= =99 =E2=80=98Xinglu Chen=E2=80=99 and =E2=80=98Sarah Morgensen=E2=80=99, and more recently maybe you?) and I don't see any open = old Go importer bugs/patches on issues.guix.gnu.org, I don't think we have to worry about > realistically? because there are years old issues, and long bitrotten > patches in every project's issue tracker, Guix included. in this case. > short of hard data, this can be argued forever. my intuition is > clearly pointing towards 2), because i doubt that the choking point > of improving the importer is due to not having enough examples on > which it breaks. If you know of some examples on which the Go importer broke, could you add them to tests/go, to make sure it won't break again later (regression), in case of future changes? Anyway, let's take a look at for which kind of things I think of in case of =E2=80=98buggy importer=E2=80=99: 1. (Maybe [PATCH 04/10]?) The parser for a file format cannot parse=C2=A0 , so it throws an exception. (I consider this more a =E2=80=98limitation=E2=80=99 than a =E2=80=98bug= =E2=80=99.) For this kind of thing, maybe we can just throw a =E2=80=98oops I don't understand the file format=E2=80=99 exception, report an appropriate err= or message (package name + maybe line number, but no need for a backtrace), skip that package and continue. That way, the Go importer will be to some degree robust to future changes. That's already done in this patch series, except for the =E2=80=98no nee= d for a backtrace=E2=80=99 part and this patch series catching all excepti= ons instead of just networking errors, =E2=80=98I don't understand the file format=E2=80=99 errors, ... 2. The Go importer misinterprets some kind of field or such. These kind of things don't (typically?) cause errors, so skipping. (Still a bug though!) 3. The Go importer cannot produce a package definition because it doesn't support . Similar (1). 4. The Go importer has some kind of type error or such. This case looks like an actual bug to me, where I think that catching the resulting exceptions is just sweeping things under the rug, implicitly teaching other people that it's ok to just ignore problems without making any attempt of fixing them or reporting them. Vaguely similar to =E2=80=98broken windows=E2=80=99 (). Sure, for a single particular user and short term this makes the importer a bit less useful, but knowing bugs is a prerequisite for fixing them, and long term, by actually knowing about the bugs and fixing them, more users will benefit. That said, like "guix build" has a "--keep-going" option to temporarily ignore to some degree packages that fail to build, maybe "guix import" (when using --recursive) can have a "--keep-going" option. Greetings, Maxime. --=-43xsJYWm2329njOGkNDs Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part Content-Transfer-Encoding: 7bit -----BEGIN PGP SIGNATURE----- iI0EABYKADUWIQTB8z7iDFKP233XAR9J4+4iGRcl7gUCYnlTIxccbWF4aW1lZGV2 b3NAdGVsZW5ldC5iZQAKCRBJ4+4iGRcl7hqmAP9/mbFb5ISVnEYRAFT1pTTsYOBZ 37FeGSrC4Vavb0A3AAD9G+Zo2FvqafGGH6HQwbRqQjJgNADS5dmoO0zgkMcAKQs= =9ani -----END PGP SIGNATURE----- --=-43xsJYWm2329njOGkNDs--