From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp10.migadu.com ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id UKyfNrNxEmJZCQAAgWs5BA (envelope-from ) for ; Sun, 20 Feb 2022 17:52:03 +0100 Received: from aspmx1.migadu.com ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp10.migadu.com with LMTPS id dp5ZL7NxEmJvNQAAG6o9tA (envelope-from ) for ; Sun, 20 Feb 2022 17:52:03 +0100 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 62E112B68F for ; Sun, 20 Feb 2022 17:52:03 +0100 (CET) Received: from localhost ([::1]:43602 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nLpRK-00084k-DL for larch@yhetil.org; Sun, 20 Feb 2022 11:52:02 -0500 Received: from eggs.gnu.org ([209.51.188.92]:47970) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nLpNR-00053I-C9 for guix-devel@gnu.org; Sun, 20 Feb 2022 11:48:01 -0500 Received: from new3-smtp.messagingengine.com ([66.111.4.229]:36255) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nLpNO-0007tn-Hn; Sun, 20 Feb 2022 11:48:01 -0500 Received: from compute4.internal (compute4.nyi.internal [10.202.2.44]) by mailnew.nyi.internal (Postfix) with ESMTP id 8799758018C; Sun, 20 Feb 2022 11:47:53 -0500 (EST) Received: from mailfrontend1 ([10.202.2.162]) by compute4.internal (MEProxy); Sun, 20 Feb 2022 11:47:53 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= philipmcgrath.com; h=cc:cc:content-type:date:date:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:sender:subject:subject:to:to; s=fm1; bh=HOc7g8EuD8eszX dfNRXPtZn5xv7ySDPsegzP58wRp2w=; b=jDwHxhv342B2m+4d+t3B0M8cyoldIQ 6b2+266SAuI/ob90xM8E8JK5RO3jm6iqNqyHQ4NmNSvLkHRjZsarn8UKmGGf9GCc 2xLmzjXJJhcPqlcl8tzO+uaqyWJfiM5bqLpP8toMXF+Nphz4XPldUtS/07EIeSbH Q41VleHatmJ67RFe2mG3tVjys+sRQLFDvwwysqzBnDNTCs0C3UL3+cT9TJSGVZn3 LSxjimiH2i0ULJB8sIKEjBe4aKnE9ASFb8oOlbGjD7jR870VEJKWHVodyO0tU3Oc w/GIYbcP1WHWhE+ueicftMqQ4iSck8RYz49Yp/FGFEKBa44XcvIbzflA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:date:date:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:sender:subject:subject:to:to:x-me-proxy:x-me-proxy :x-me-sender:x-me-sender:x-sasl-enc; s=fm2; bh=HOc7g8EuD8eszXdfN RXPtZn5xv7ySDPsegzP58wRp2w=; b=olGHY/W330qvrPQYE5c9fdCxm+9g0DXY1 6Cf9msTSY3g8mJ3CEWWUc64g0uK97dGZZfnh/fqB5h4nIjjn01/vtoqGAwPzU1gg mv6ovDHKfMuY6r1TFoPkQ6Luco8+ThJMTdg3v/tTJRv1MrACXDz/5BK0hq+NITYC OL+8nk6ddZVw2xK/SVmrYUiMUTWvSlGHXakybEGEjBnTvd2EmXqHOddWwb37/Por o1L9qzxcvHXJsgp916OIOccE36A01UYRZUaOKNrQlG5EeDKf0NDSSRJvu7NkFhYb iSvyvftGB6ftblEruMGrFpHY1KOvGxh3ECxyOxVvhmk1x349v2sNw== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvvddrkeeggdelgecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc fjughrpefhvffufffkjghfgggtsehgtderredttdejnecuhfhrohhmpefrhhhilhhiphcu ofgtifhrrghthhcuoehphhhilhhiphesphhhihhlihhpmhgtghhrrghthhdrtghomheqne cuggftrfgrthhtvghrnhepudffhfdttdffhfekhfevieeggfdtieetkeffuddtleehuddt tdeitdeuueehgfdvnecuffhomhgrihhnpehgnhhurdhorhhgpdhuthgrhhdrvgguuhdprh grtghkvghtqdhlrghnghdrohhrghdprheirhhsrdhorhhgpdhnvghurdgvughunecuvehl uhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepphhhihhlihhpse hphhhilhhiphhmtghgrhgrthhhrdgtohhm X-ME-Proxy: Received: by mail.messagingengine.com (Postfix) with ESMTPA; Sun, 20 Feb 2022 11:47:51 -0500 (EST) From: Philip McGrath To: Ekaitz Zarraga , Ricardo Wurmus , guix-devel@gnu.org Subject: Re: Faster "guix pull" by incremental compilation and non-circular modules? Date: Sun, 20 Feb 2022 11:47:45 -0500 Message-ID: <6012789.Rgyoke53jH@bastet> In-Reply-To: <753ba5897ed397b5e95175cd139137975245945b.camel@telenet.be> References: <2067ba1e606855eace261fd0b0ae9721b369bbd5.camel@telenet.be> <753ba5897ed397b5e95175cd139137975245945b.camel@telenet.be> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="nextPart8620947.KgQKAJ9IsF"; micalg="pgp-sha512"; protocol="application/pgp-signature" Received-SPF: pass client-ip=66.111.4.229; envelope-from=philip@philipmcgrath.com; helo=new3-smtp.messagingengine.com X-Spam_score_int: -15 X-Spam_score: -1.6 X-Spam_bar: - X-Spam_report: (-1.6 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01, URI_DOTEDU=1.246 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: guix-devel@gnu.org, Liliana Marie Prikler Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: "Guix-devel" X-Migadu-Flow: FLOW_IN X-Migadu-Country: US ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1645375923; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post:dkim-signature; bh=HOc7g8EuD8eszXdfNRXPtZn5xv7ySDPsegzP58wRp2w=; b=TJTe6OW2e0mYFRk1MtfpVaFIFoATZFRXhJmvkZgTC1ArSwLv00laFCEU6n4eu6aoPyepJE +1WG++aBNe5lZ2GQmJeZtEIAIChYViZx1GLG/XQlIDCSC6R51OmfzrNEF7z0T7YJUgm0Bd z4Buptk7fSFwIibyEcK/FEeaH89NAq6Ns9sE4cBxyeC8jb9CectIeFSIfJe/C74M2keqeS TH9qlo+TRQoFnBOLaQUwNkDWyBBIF+u1tFqnwdBDL+T1j4aEvk8hMrlWCKm37hMDGHJpLy kyXoat5qqQM255O85QTaTkABrlF9ViKD99gRaoTfrei1LcqEI9Uyr/+bOhF/3w== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1645375923; a=rsa-sha256; cv=none; b=kY+D3ITz+hy1AeOsIR5xmiMAJ5lDQzNQRRFkna63Da7dofkD43z9q/p+SFIOT63ptC6rqL 7S6mDMAR+rDjuZJrMzY0YSrA0nDD7YV5oekZmLuBLapLjpxEGcruUpUGjEm+mYz2Ion59r UpL0wdYayULy6CsVKpxFGorYFmJ4JHwEiCZbyLdh8+5X6c6E+Dcetdk3uOMswxtTaEyseC L7/T3ykeVNKq1Vv1vpvptf1j8LTtjHDxQtxnRZaAXtszl3ae4c21VGyvnagUevsJoT0xt6 whidQsdPOovCBnqspr2E2M5CJG7/8UaKEADIv+MP01G9v8nSt/fvw3Xs1zQtxg== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=philipmcgrath.com header.s=fm1 header.b=jDwHxhv3; dkim=fail ("headers rsa verify failed") header.d=messagingengine.com header.s=fm2 header.b="olGHY/W3"; dmarc=none; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org" X-Migadu-Spam-Score: -3.73 Authentication-Results: aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=philipmcgrath.com header.s=fm1 header.b=jDwHxhv3; dkim=fail ("headers rsa verify failed") header.d=messagingengine.com header.s=fm2 header.b="olGHY/W3"; dmarc=none; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org" X-Migadu-Queue-Id: 62E112B68F X-Spam-Score: -3.73 X-Migadu-Scanner: scn0.migadu.com X-TUID: qmJtZkSOFQBT --nextPart8620947.KgQKAJ9IsF Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="UTF-8"; protected-headers="v1" From: Philip McGrath To: Ekaitz Zarraga , Ricardo Wurmus , guix-devel@gnu.org Cc: guix-devel@gnu.org, Maxime Devos , Ludovic =?ISO-8859-1?Q?Court=E8s?= , Liliana Marie Prikler Subject: Re: Faster "guix pull" by incremental compilation and non-circular modules? Date: Sun, 20 Feb 2022 11:47:45 -0500 Message-ID: <6012789.Rgyoke53jH@bastet> In-Reply-To: <753ba5897ed397b5e95175cd139137975245945b.camel@telenet.be> References: <2067ba1e606855eace261fd0b0ae9721b369bbd5.camel@telenet.be> <753ba5897ed397b5e95175cd139137975245945b.camel@telenet.be> Hi, On Sunday, February 20, 2022 7:19:07 AM EST Maxime Devos wrote: > Ekaitz Zarraga schreef op zo 20-02-2022 om 11:34 [+0000]: > > Making a Guix pull is unpredictable too... >=20 > An idea for making "guix pull" faster: >=20 > 1. Make package imports non-circular, breaking up package modules > in parts when necessary. >=20 > 2. Instead of building all of Guix as a single derivation, > create a DAG of derivations. More concretely: >=20 > First read the *.scm files to determine which module imports > which modules. Then to compile, say, (gnu packages acl), > a derivation taking gnu/packages/acl.scm and its dependencies > gnu/packages/attr.go, gnu/packages/base.go, ... is made > compiling gnu/packages/acl.scm to a gnu/packages/acl.go. >=20 > Then to build all of Guix, 'union-build' or 'file-union' is used. >=20 > The benefit is that if, say, gnu/packages/gnunet.scm is changed, then > the old gnu/packages/acl.go will be reused because its derivation > doesn't depend on gnu/packages/gnunet.scm (*). >=20 > The need for non-circular imports can be avoided by computing the > strongly-connected components and compiling all the modules in a > component as a single unit. >=20 > Greetings, > Maxime. >=20 > (*) Actually I didn't verify if acl.scm depends on gnunet.scm or not. I was just (or maybe am still?) dealing with some issues caused by cyclic imports of package modules while updating Racket to 8.4: see in particular , as well as #66, #112, and #113 in t= he same thread. As I mentioned there, I am most familiar with the semantics of Racket modules,[1][2][3][4][5][6] and secondarily with R6RS libraries.[7] I find the semantics of Guile's cyclic module imports very confusing, and I don't know of any documentation for what is and isn't supported other than advice from Ludo=E2=80=99 in =E2= =80=94is there any? Racket's module system fundamentally requires that module imports for a DAG (see footnote 1 in =C2=A7 2.2 of [1]), which is necessary to provide =E2=80= =9CThe Separate Compilation Guarantee=E2=80=9D[1][6]: > Any effects of the instantiation of the module=E2=80=99s phase 1 due to c= ompilation > on the Racket runtime system are discarded. This is an extremely useful property, especially if you care about reproducibility. I know that R6RS does not provide The Separate Compilation Guarantee, since =C2=A7 7.2 [8] explicitly says: > An implementation may distinguish instances/visits of a library for > different phases or to use an instance/visit at any phase as an > instance/visit at any other phase. An implementation may further expand > each library form with distinct visits of libraries in any phase and/or > instances of libraries in phases above 0. An implementation may create > instances/visits of more libraries at more phases than required to satisfy > references. When an identifier appears as an expression in a phase that is > inconsistent with the identifier's level, then an implementation may raise > an exception either at expand time or run time, or it may allow the > reference. Thus, a library whose meaning depends on whether the instances > of a library are distinguished or shared across phases or library > expansions may be unportable. I haven't found anything in R6RS that explicitly addresses cyclic library imports, but I think this passage from =C2=A7 7.2 [8]: > If any of a library's definitions are referenced at phase 0 in the expand= ed > form of a program, then an instance of the referenced library is created > for phase 0 before the program's definitions and expressions are evaluate= d. > This rule applies transitively: if the expanded form of one library > references at phase 0 an identifier from another library, then before the > referencing library is instantiated at phase n, the referenced library mu= st > be instantiated at phase n. When an identifier is referenced at any phase= n > greater than 0, in contrast, then the defining library is instantiated at > phase n at some unspecified time before the reference is evaluated. > Similarly, when a macro keyword is referenced at phase n during the > expansion of a library, then the defining library is visited at phase n at > some unspecified time before the reference is evaluated. combined with the specification in =C2=A7 7.1 [9] that: > The expressions of variable definitions are evaluated from left to right,= as > if in an implicit letrec*, and the body expressions are also evaluated fr= om > left to right after the expressions of the variable definitions. A fresh > location is created for each exported variable and initialized to the val= ue > of its local counterpart. The effect of returning twice to the continuati= on > of the last body expression is unspecified. means that they are prohibited. If library (a) and library (b) were each to import each other and each to refer to the other's definitions at phase 0, then library (a) must be instantiated *before* library (b), but library (b) must also be instantiated *before* library (a), so an attempt to instantiate either library would diverge. Of course, Guile can, as an extension, assign meaning to cyclic module imports, but I question whether Guile would be wise to do so, and whether Guix would be wise to rely on it. It was before my time, but I know the design of Racket's module system was influenced by the experience of programming with Rackets "units"[10][11], which do support mutually recursi= ve references. Units are great when you really do need them, but the view of those who experienced them was that acyclic modules are a better choice as the fundamental mechanism of code organization. While you can make a compil= er or build system handle cyclic dependencies (albeit without all the nice properties of the Racket and R6RS module systems), humans tend to find them very confusing, especially when they are not explicit. =2DPhilip [1]: Matthew Flatt, =E2=80=9CComposable and Compilable Macros: You Want it = When?=E2=80=9D (ICFP 2002), https://www.cs.utah.edu/plt/publications/macromod.pdf [2]: Matthew Flatt, "Submodules in Racket: You Want it When, Again?" (GCPE 2013), https://www.cs.utah.edu/plt/publications/gpce13-f-color.pdf [3]: https://docs.racket-lang.org/reference/syntax-model.html#%28part._mod-= parse%29 [4]: https://docs.racket-lang.org/reference/eval-model.html#%28part._module= =2Deval-model%29 [5]: https://docs.racket-lang.org/reference/eval-model.html#%28part._module= =2Dphase%29 [6]: https://docs.racket-lang.org/reference/eval-model.html#%28part._separa= te-compilation%29 [7]: http://www.r6rs.org/final/html/r6rs/r6rs-Z-H-10.html#node_chap_7 [8]: http://www.r6rs.org/final/html/r6rs/r6rs-Z-H-10.html#node_sec_7.2 [9]: http://www.r6rs.org/final/html/r6rs/r6rs-Z-H-10.html#node_sec_7.1 [10]: Flatt & Felleisen, "Units: Cool Modules for HOT Languages" (PLDI 1998= ), http://www.ccs.neu.edu/scheme/pubs/pldi98-ff.ps.gz [11]: https://docs.racket-lang.org/reference/mzlib_unit.html --nextPart8620947.KgQKAJ9IsF Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part. Content-Transfer-Encoding: 7Bit -----BEGIN PGP SIGNATURE----- iQIzBAABCgAdFiEE9GWrrNY3rqwUFVXPygNjjfo/HHoFAmIScLEACgkQygNjjfo/ HHqlwQ/9FWqtCKweNwREA3CefG7gL3updFS/p0Ksyz8KpRHZ7RhhjCLarIS/uTDz pgmRcKfgTz6c1bq8/Zts4xVuy2q2vYK8yCJq2+Mh7stv46lHedfzXBAYuQMY8/oA MgqJuGAmaLNAeEPqJaZ51k2LnIy7E41qasSQqUN0mFrmx8pfVUrdGfD60iswKeW8 8/1OFlQf2xtG/Hs0vxKLL997O5Qy9/YqT85kXGzcbb7tFd3MfAOz4gwrWYtpgFMN xN1HdraAny3udBmjxhcNKG8SYtl0bGCTIbZ/1DPf6TqOtepdm3k6l3jfnPhG8GtJ xHEobau5B7lhF9dMpia8PccMcQxajILHjtf9BGPzd653Mg5Owc3eXGZFr4PFM4jd QKLZqScAvJnHL/0czCt65NMAsPrXzEpCPlsMP/ybap5IGsSUIFJDafIE3tpmGCi3 bYykPtcBbm6xgNApxaRfyPANvIOxw+VxLV8YRro8EzJJkmwOq8KkQ0hWrlmfDKck Bs3GB3wPxnP7rEEPmxHEcsoCaU/H+HjuZUiE6eGhqpvRQ/THZIOJswfQuYrvAT/u NYOt4/eUKs7Z/fUFOx7BUBZVOObw+bfRX24xD2YEw6EdYtP3VqGZkQ3S9p4Inzwn oeeJfNRSYcXAvDu8ueHqGLeEY2pPTp96xxSxgf5QmJGvwvixLv4= =Ec1z -----END PGP SIGNATURE----- --nextPart8620947.KgQKAJ9IsF--