From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp12.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms9.migadu.com with LMTPS id tujnG5+0U2SohAAASxT56A (envelope-from ) for ; Thu, 04 May 2023 15:35:27 +0200 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp12.migadu.com with LMTPS id KMyqGp+0U2Q40gAAauVa8A (envelope-from ) for ; Thu, 04 May 2023 15:35:27 +0200 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 3B1123C04D for ; Thu, 4 May 2023 15:35:27 +0200 (CEST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1puZ6u-00014t-A4; Thu, 04 May 2023 09:35:04 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1puZ3z-0008Oe-Dv for guix-patches@gnu.org; Thu, 04 May 2023 09:32:03 -0400 Received: from debbugs.gnu.org ([209.51.188.43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1puZ3z-0005wu-4N for guix-patches@gnu.org; Thu, 04 May 2023 09:32:03 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1puZ3y-0003qS-7l for guix-patches@gnu.org; Thu, 04 May 2023 09:32:02 -0400 X-Loop: help-debbugs@gnu.org Subject: [bug#63263] [PATCH] gexp: Stop generating unreadable builder scripts. Resent-From: Christopher Baines Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Thu, 04 May 2023 13:32:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 63263 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Ludovic =?UTF-8?Q?Court=C3=A8s?= Cc: 63263@debbugs.gnu.org Received: via spool by 63263-submit@debbugs.gnu.org id=B63263.168320709714729 (code B ref 63263); Thu, 04 May 2023 13:32:02 +0000 Received: (at 63263) by debbugs.gnu.org; 4 May 2023 13:31:37 +0000 Received: from localhost ([127.0.0.1]:48832 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1puZ3Y-0003pU-Hm for submit@debbugs.gnu.org; Thu, 04 May 2023 09:31:36 -0400 Received: from mira.cbaines.net ([212.71.252.8]:42490) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1puZ3U-0003pD-GV for 63263@debbugs.gnu.org; Thu, 04 May 2023 09:31:34 -0400 Received: from localhost (unknown [IPv6:2a02:8010:68c1:0:54d1:d5d4:280e:f699]) by mira.cbaines.net (Postfix) with ESMTPSA id 6102A27BBE2; Thu, 4 May 2023 14:31:31 +0100 (BST) Received: from felis (localhost [127.0.0.1]) by localhost (OpenSMTPD) with ESMTP id 0deae971; Thu, 4 May 2023 13:31:30 +0000 (UTC) References: <20230504112448.22462-1-mail@cbaines.net> <87zg6kqn50.fsf@gnu.org> User-agent: mu4e 1.8.13; emacs 28.2 From: Christopher Baines Date: Thu, 04 May 2023 13:57:38 +0100 In-reply-to: <87zg6kqn50.fsf@gnu.org> Message-ID: <875y98nryn.fsf@cbaines.net> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha512; protocol="application/pgp-signature" X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+larch=yhetil.org@gnu.org Sender: guix-patches-bounces+larch=yhetil.org@gnu.org X-Migadu-Country: US X-Migadu-Flow: FLOW_IN ARC-Seal: i=1; s=key1; d=yhetil.org; t=1683207327; a=rsa-sha256; cv=none; b=Vz4kyn7HNbwfC1KViqSECLkerfPa0m+RtpzkFvOzhLfp5i34fVQkym6ay/pHjpvTU5HMX5 xNyRd6ZAdwpObvLO1kIQ8TBX1FTEolvJHEb7JH2Q03gpKat7Ch1+emnLyRuyLNW/n4bVFH jlxS4g349sWGLsvP3SYZCwynNLG9Zp2Ni+knQXKWSro/zw5F6SW0WPmqQlCO4MebAo3GdV 9OveubMUhLGVqauc+IXNWqvQu4th0lPGKv9l115EsfnRp/BSznjH1Nn44is93BrNBgqG20 y+uwTnHY6zncEW5WblL/Ydpy+f0YuyIj/09uSfnwOs6u01dL2v48etMvBi74fQ== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of "guix-patches-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-patches-bounces+larch=yhetil.org@gnu.org" ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1683207327; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:resent-cc:resent-from:resent-sender: resent-message-id:in-reply-to:in-reply-to:references:references: list-id:list-help:list-unsubscribe:list-subscribe:list-post; bh=+aSTowpPtbkYQ78LvScg3AlS61eOUYt9TEzBn5asMmw=; b=Y4orHhVlClcJUChLWFDKeoIMlnQMy1Rj9p/vsAQtxcQDB5baWeLMPCCVe4cahfBQVXTNBI XwWtiI0n6cxD09hfmIjto69UnaTuGS1ufXBX5hJsJ/0z8M3hxrUxSvbp2FNf0k54oN3w4X zFug8bKfZvCVdmCs4LPoxMA3kmB4N8hZ7TZe35nL4Xx/O5Uhe6zqsnnzSRtdQAqWPsM7EU RfPBsANL+FTtlCXVZFNE9yuZ1O2qL3zmzupOkGls5UKsajrph263h7Rftyo7vZqVFmlEuM LYhGz1n6SSajmCP0Ek0o8Y+747wlUaZClfHoLnjJ4e0kW/4L+cKS7zIo8NSwFg== X-Migadu-Spam-Score: -3.01 X-Spam-Score: -3.01 X-Migadu-Queue-Id: 3B1123C04D X-Migadu-Scanner: scn0.migadu.com Authentication-Results: aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of "guix-patches-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-patches-bounces+larch=yhetil.org@gnu.org" X-TUID: tb67k2lPKWp/ --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Ludovic Court=C3=A8s writes: > Hi, > > Christopher Baines skribis: > >> In Guile, it's possible to produce output from write that can't be read,= and >> this applies to the code staged through g-expressions for derivations. = This >> commit detects this early when the derivation is being created, rather t= han >> leaving the error to happen when the derivation is built. >> >> This is important as it means that tools like guix lint will indicate th= at >> there's a problem, hopefully reducing the number of broken derivations in >> Guix. >> >> * guix/gexp.scm (gexp->derivation): Check that the builder script can be= read. > > Calling =E2=80=98read=E2=80=99 on every generated sexp is definitely not = something we > should do, performance-wise. > > Commit 24ab804ce11fe12ff49cd144a3d9c4bfcf55b41c addressed that to some > extent. It works in examples like this: > > scheme@(guile-user)> ,lower (computed-file "foo" #~(list #$(current-modul= e))) > While executing meta-command: > ERROR: > 1. &gexp-input-error: # > > > =E2=80=A6 where =E2=80=98current-module=E2=80=99 returns a non-serializab= le object. > > I think the problem you=E2=80=99re trying to address that we frequently > encounter is old-style packages that end up splicing gexps inside sexps, > as in: > > (package > ;; =E2=80=A6 > (arguments `(#:phases (modify-phases whatever ,#~doh!)))) > > Is that right? I think so, I can't remember if I've seen any other ways that this happens. > The problem here is that =E2=80=98sexp->gexp=E2=80=99, which was added pr= ecisely as an > optimization for build systems=C2=B9, does not check the sexp it=E2=80=99= s given. > Example: > > scheme@(guile-user)> ,lower (computed-file "foo" (sexp->gexp `(list a b c= ,(current-module)))) > $19 =3D # 7f26baf56be0> > scheme@(guile-user)> (sexp->gexp `(list a b c ,(current-module))) > $20 =3D #) 7f26bb= f2f090> > > Oops! > > It would be tempting to change =E2=80=98sexp->gexp=E2=80=99 to traverse t= he sexp in > search of non-serializable things=E2=80=A6 but that=E2=80=99d defeat the = whole point of > =E2=80=98sexp->gexp=E2=80=99. > > How about a linter instead, with the understanding that use of sexps in > packages is vanishing? Perhaps coupled with a =E2=80=98guix style=E2=80= =99 automatic > rewriter. A linter might be helpful, but I'm not sure it'll help that much. I think it's quite a lofty expectation for the linter to be run on packages that are edited, let alone on the packages affected by those changes (which is what's needed to catch this problem), so adding a linter will mean we get lint warnings, but we'll still be living with these broken derivations. The builds for affected derivations fail immediately, and it's pretty obvious from the log that the builder is unreadable, so it should already be possible to spot this problem from looking at the effect of package changes on builds, so I think the main way a linter will help is that it would provide a way to find out what derivations are broken in this way, without attempting to build all of them. I guess my perspective on this is more from the operation of the guix data service, which is carefully computing and storing all of these broken derivations (and there's a lot, like 10,000+ per revision at the moment, since they change every time you compute them). This then propagates down to the build coordinator as well, since there's builds being submitted for all these broken derivations. I have considered trying to detect these breakages in the data service, but I'm not sure how to do it while removing the possibility of false positives. --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQKlBAEBCgCPFiEEPonu50WOcg2XVOCyXiijOwuE9XcFAmRTs7BfFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNF ODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcRHG1haWxAY2Jh aW5lcy5uZXQACgkQXiijOwuE9Xeobg/+K8wwXAXEmjgLeHr60iMUPHfH4Nts6zJZ 4lQLG+kMNKtARuc8xQP/KxIwkuUkbb1gCwOcER+HDw6DSwAk4MtFzfEG7Ai7H5w2 XPB79I7FeKxithGaX11qowb3Obu10tMOc5WS8dkusXCLUOlqxVekGXn+DMu/Pfhi Wxnmm6GoUOuzVACUvowE5Cz9QVCB1P8/PWo7lqLOrNmfMpxFATVyBbA1ijSeNhM3 wLFHTTsLMnmZv7+CkEh/tKzXGWZ0AMOFq1HDmWhSPejgSTKtM5pQyySBCTvx0kd0 bYCRviuEGXuneiBhV2AkauoetM9muT/Q69pxTIpDh9x30/7+bib8azVSnmnwkIiQ q8sSK4wBOIqiZCGdDQBEqucEr+6po02MBwp7A5OArAETblkVUIg0wqiFkL3bgPgl C7goR/p3BUfIslEcquHG+7hgFqk0pdZeLHibvzTKS/zBYfuzMt2tUb3PUkzqGZfV 8VMD268voHcLmDEMflCecx5bt90WY869e5xK0kDebWR69Otupq1LweNfJI8h5CcK VsxlTbhCy0dYWZPwPIL4VfdGkjGdmcggeTVNjCJK4V9LUrJkiOc+nYoM2XtJKP2Z YKNYDHhyP3uD7/5EAURkC0dq++jRSKhP2yj1q0V0FLzDDZwIOJ253SjlcRzt4oL+ RRb6Mn9eNHA= =j0e4 -----END PGP SIGNATURE----- --=-=-=--