From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp12.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id 0GeEESRxQGIKVgEAgWs5BA (envelope-from ) for ; Sun, 27 Mar 2022 16:13:56 +0200 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp12.migadu.com with LMTPS id QB8pDiRxQGIcGQAAauVa8A (envelope-from ) for ; Sun, 27 Mar 2022 16:13:56 +0200 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id A305910808 for ; Sun, 27 Mar 2022 16:13:55 +0200 (CEST) Received: from localhost ([::1]:59490 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nYTeU-000852-Oz for larch@yhetil.org; Sun, 27 Mar 2022 10:13:54 -0400 Received: from eggs.gnu.org ([209.51.188.92]:52354) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nYTdJ-0007m2-BH for guix-devel@gnu.org; Sun, 27 Mar 2022 10:12:42 -0400 Received: from new4-smtp.messagingengine.com ([66.111.4.230]:48849) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nYTdE-00037p-MQ; Sun, 27 Mar 2022 10:12:38 -0400 Received: from compute5.internal (compute5.nyi.internal [10.202.2.45]) by mailnew.nyi.internal (Postfix) with ESMTP id 0869A5808F0; Sun, 27 Mar 2022 10:12:35 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute5.internal (MEProxy); Sun, 27 Mar 2022 10:12:35 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= philipmcgrath.com; h=cc:cc:content-type:date:date:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:sender:subject:subject:to:to; s=fm1; bh=6NHp1arfoZ5ugv j0unPCvZGnC5xtnqwEqL20cT+Kzy0=; b=H3Tf5kxmbET2jRtcSMeQNASAIce1ug xFJtOWG0ZjrwPWCJH5gDGDJz4lhzAGHRZ++k/nUa3DW1QFhAz3CMApqS1iUFIuxr ifPOiGxglNz41k8V6KwYuHTQ3299EQOC3OIatzLBJsHzTfXB5Rx9Nx+jOw8UVlRP u6OK9WJnV064G/pM6p6XiZ9N7YEKDT8Krn8PxJOPKrTC/BqSg7QaH4IAlN0aqivr Y1B5dnS7JTqQcO+qvUXOvbp+WGzP0n1vfYQjfSeEOR6jXaovzyoMomEer+16+FKc Mq4YFmjrRJ56qSi9vNm3m0lDqsJvsfPdRzkwRpsu17wgkpyQ9P66G+hA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:date:date:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:sender:subject:subject:to:to:x-me-proxy:x-me-proxy :x-me-sender:x-me-sender:x-sasl-enc; s=fm3; bh=6NHp1arfoZ5ugvj0u nPCvZGnC5xtnqwEqL20cT+Kzy0=; b=kmfE4Focl13IHmgyGwIvQ9nEbYBUeOASM mBjgo1UQLD9BaWegLLj84MlFi5JYqULGHczu2KEJ/5Ux/CC5XoX8SQ3bztsbGHhw tOxRTMsru3m1bi8+44UbqkcRNp3+6cRkYUw7WqRlKAjc4OjLbzsHgIiBfqYCgFYJ ZF73m03KMeNG/i96rrccE94BG4TM5E94gWs5hD0IzrtlPnfj+oZtyOZevrPLxqEg cgxKwTIpSucTlKVuXL7ILEipNdf73kvYWdi5z0a/SAAAlsBPQfeq9t0PFhET6z0C cREsbB6Uml53IrQb9GoP2zPXLQBX00KOeVxbdvbzQ7Sorg3zCJu+w== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvvddrudehhedgjeehucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvufffkfgjfhggtgesghdtreertddtjeenucfhrhhomheprfhhihhlihhp ucfotgfirhgrthhhuceophhhihhlihhpsehphhhilhhiphhmtghgrhgrthhhrdgtohhmqe enucggtffrrghtthgvrhhnpedvhfduleehuefhgfdvkeevveeufffgueeivedvffduffef fedvieehkedvveffveenucffohhmrghinhepghhnuhdrohhrghdprhgrtghkvghtqdhlrg hnghdrohhrghdpphhsuhdrvgguuhenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgr mhepmhgrihhlfhhrohhmpehphhhilhhiphesphhhihhlihhpmhgtghhrrghthhdrtghomh X-ME-Proxy: Received: by mail.messagingengine.com (Postfix) with ESMTPA; Sun, 27 Mar 2022 10:12:33 -0400 (EDT) From: Philip McGrath To: Maxime Devos , Ekaitz Zarraga , Ricardo Wurmus , guix-devel@gnu.org Subject: Re: Semantics of circular imports Date: Sun, 27 Mar 2022 10:12:25 -0400 Message-ID: <3493949.dX3coAEF3O@bastet> In-Reply-To: References: <2067ba1e606855eace261fd0b0ae9721b369bbd5.camel@telenet.be> <753ba5897ed397b5e95175cd139137975245945b.camel@telenet.be> <6012789.Rgyoke53jH@bastet> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="nextPart2466355.TVEiu29ZbQ"; micalg="pgp-sha512"; protocol="application/pgp-signature" Received-SPF: pass client-ip=66.111.4.230; envelope-from=philip@philipmcgrath.com; helo=new4-smtp.messagingengine.com X-Spam_score_int: -15 X-Spam_score: -1.6 X-Spam_bar: - X-Spam_report: (-1.6 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01, URI_DOTEDU=1.246 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Liliana Marie Prikler Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: "Guix-devel" X-Migadu-Flow: FLOW_IN X-Migadu-To: larch@yhetil.org X-Migadu-Country: US ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1648390435; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post:dkim-signature; bh=6NHp1arfoZ5ugvj0unPCvZGnC5xtnqwEqL20cT+Kzy0=; b=I3Qb4Jjm0milM9KuIqxPU/AOd/oKYNIqOIWXedt8kLXMbdhZRQpJcRDbbufscStmP4FSsY P7BA/Qrp520gLTzD/zPdq3EpQ/t4ED/kWE/P9ba0kw/S+kG0Yv/P9vQMluGmrMbwxE+3ap /BWU1d9NFviK9eia9aJQgu2J2RM+bxPVpTGikPn4fKPaENqPJ75TJsi+f5mDKnqtOZoQo6 sb1GED/M2vwuj7gsoOYtBdyKIPAGHBIGeM6vQWkHjQ3O20y8714Z9vbsRUxLvb5i9RXNcI B+u1wRuF2UtPMl44bM/00zX+2dVXNtmSmVMCnto3S0E/fquLBCVM/Kf4MMQ6mw== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1648390435; a=rsa-sha256; cv=none; b=sv9pTHl0CXa5OrxTf2nG/somgEXvl5KaTYsiAHuz6/6on1kUPDrpTkMC1FJrqhx1ztPVEO iUBS/wqkFTKdC8F3HbPyv6Ge0f46PkX39TIEEaao6ll5vlM+XWVs1UOrhFli81Akgu/wg1 F/WICerZgY4e5tHr2EqklI2gR5vV2ku0xrcn7ejYRS/bI6ufZrqMYSIIEWbDYmBtyLXkgS k1CzPvb9qlxtmXsemsTP0H763+lk5LOZlTtcDCYwkeCiKGssxguAZKU31Vn8Ixfkbj0XNw a0TTnJGzD8DhmEAhLTV+EDODMr8Vcg1354Iv7hPakJTugamitA52+R/7tW02cA== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=philipmcgrath.com header.s=fm1 header.b=H3Tf5kxm; dkim=fail ("headers rsa verify failed") header.d=messagingengine.com header.s=fm3 header.b=kmfE4Foc; dmarc=none; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org" X-Migadu-Spam-Score: -1.37 Authentication-Results: aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=philipmcgrath.com header.s=fm1 header.b=H3Tf5kxm; dkim=fail ("headers rsa verify failed") header.d=messagingengine.com header.s=fm3 header.b=kmfE4Foc; dmarc=none; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org" X-Migadu-Queue-Id: A305910808 X-Spam-Score: -1.37 X-Migadu-Scanner: scn0.migadu.com X-TUID: fX24dDQ3BL/g --nextPart2466355.TVEiu29ZbQ Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="UTF-8"; protected-headers="v1" From: Philip McGrath Subject: Re: Semantics of circular imports Date: Sun, 27 Mar 2022 10:12:25 -0400 Message-ID: <3493949.dX3coAEF3O@bastet> Hi, (Apparently I wrote this a month ago but left it sitting in "Drafts" ...) On 2/20/22 12:47, Maxime Devos wrote: > Philip McGrath schreef op zo 20-02-2022 om 11:47 [-0500]: >> I was just (or maybe am still?) dealing with some issues caused by cyclic >> imports of package modules while updating Racket to 8.4: see in particul= ar >> , as well as #66, #112, and #113 i= n the >> same thread. >> [...] >> I find the semantics of Guile's cyclic module imports very confusing, >> and I don't know of any documentation for what is and isn't supported >> other than advice from Ludo=E2=80=99 in > =E2=80=94is there any? >=20 > (The following explanation ignores syntax transformers, #:select and > #:autoload.) >=20 > Basically, a module consists of a hash table and a thunk > initialising the hash table. A few situations to demonstrate: >=20 > ;; Non-cyclic > (define-module (foo) > #:export (foo)) > (define foo 0) >=20 > (define-module (bar) > #:export (bar) > #:use-module (foo)) > (define bar (+ 1 foo)) >=20 > The thunk of 'foo' would be >=20 > (lambda () > (set-module-value! '(foo) 'foo 0)) >=20 > and the thunk of 'bar' would be >=20 > (lambda () > ;; This calls the thunk of 'foo' if it hasn't yet been called > ;; before. > (initialise-module '(foo)) > (set-module-value! '(bar) 'bar (+ 1 (module-value '(foo) 'foo 0)))) >=20 > ;; Cyclic, non-problematic > (define-module (foo) > #:export (foo) > #:use-module (bar)) > (define foo 0) > (define (calculate-foobar) > (+ 1 (calculate-bar)) >=20 > (define-module (bar) > #:export (calculate-bar) > #:use-module (foo)) > (define (calculate-bar) (+ 1 foo)) >=20 > The initialisation thunk of 'foo' would be >=20 > (lambda () > (initialise-module '(bar)) ; L1 > (set-module-value! '(foo) 0) ; L2 > (set-module-value! '(foo) 'calculate-foobar ; L3 > (lambda () (+ 1 ((module-value '(bar) 'calculate-bar)))))) ; L4 >=20 > and the thunk of 'bar' is: >=20 > (lambda () > (initialise-module '(foo)) ; L6 > (set-module-value! '(bar) 'calculate-bar ; L7 > (lambda () (+ 1 (module-value '(foo) 'foo))))) ; L8 >=20 > Now let's see what happens if the module (bar) is loaded: >=20 > ; Initialising '(bar)' > (initialise-module '(foo)) ; L6 > ;; (foo) has not yet begun initialisation, so run the thunk: > -> (initialise-module '(bar)) ; L1 > ;; (bar) is being initialised, so don't do anything here > -> (set-module-value! '(foo) 0) ; L2 > -> (set-module-value! '(foo) 'calculate-foobar ; L3 > (lambda () (+1 ((module-value '(bar) 'calculate-bar)))) ; L4 >=20 > The hash table of '(bar)' does not yet contain 'calculate-bar', > but that's not a problem because the procedure 'calculate-foobar' > is not yet run. > (set-module-value! '(bar) '(calculate-bar) ; L7 > (lambda () (+ 1 (module-value '(foo) 'foo)))) ; L8 > ;; The hash table of '(foo)' contains 'foo', so no problem! > ;; Alternatively, even if '(foo)' did not contain 'foo', > ;; the procedure '(calculate-bar)' is not yet run, so no problem! Oh, wow. I definitely had not realized that, *even inside a declarative=20 module*, a reference to a variable with no statically visible definition=20 would semantically be a dynamic lookup in a mutable environment at=20 runtime (rather than a compile-time error), though I do see now that=20 `info guile declarative` does indeed say that marking a module as=20 declarative "applies only to the subset of top-level definitions that=20 are themselves declarative: those that are defined within the=20 compilation unit, and not assigned (=E2=80=98set!=E2=80=99) or redefined wi= thin the=20 compilation unit." Does this mean that Guile treats all imported bindings as non-declarative? This seems like a big barrier to cross-module inlining, though IIUC Guile currently doesn't do much of that by default (maybe for this reason). The use of "top-level" to refer to definitions within a module is=20 somewhat confusing to me. I usually understand "top-level" to refer to=20 the kind of interactive REPL environment for which R6RS leaves the=20 semantics unspecified. Racket uses "module-level variable" and "module=20 context" in contrast to "top-level variable" and "top-level context" to make this distinction.[1][2][3] (There are also R6RS "top-level=20 programs", but I wouldn't think of those unless made very clear from=20 context.) (Also, what is a "compilation unit" in Guile? Is it ever something other=20 than a single module corresponding to a single file (potentially using=20 `include` to incorporate other files at expand time?) >=20 > ;; Done! >=20 > Now for a problematic import cycle: >=20 > (define-module (foo) > #:export (foo) > #:use-module (bar)) > (define foo (+ 1 bar)) >=20 > (define-module (bar) > #:export (bar) > #:use-module (foo)) > (define bar (+ 1 foo)) >=20 > Now let's reason what happens when we try importing 'bar'. > The init thunk of '(foo)' is: >=20 > (lambda () > (initialise-module '(bar)) ; L1 > (set-module-value! '(foo) 'foo (+ 1 (module-value '(bar) 'bar)))) ; = L2 >=20 > and the thunk of '(bar)': >=20 > (lambda () > (initialise-module '(foo)) ; L3 > (set-module-value! '(bar) 'bar (+ 1 (module-value '(foo) 'foo)))) ; = L4 >=20 > Now let's see what happens if 'bar' is loaded: >=20 > ; Initialising (bar) > (initialise-module! '(foo)) ; L3 > ;; (foo) has not yet begun initialisation, so run the thunk: > -> (initialise-module '(bar)) ; L1 > ;; (bar) is already initialising, so don't do anything > -> (set-module-value! '(foo) 'foo (+ 1 bar))) >=20 > Oops, the variable foo of the module (foo) has not yet been defined, > so an 'unbound-variable' exception is raised! In the context of Racket or R6RS modules, where the semantics are=20 essentially those of `letrec*`, I'm used to distinguishing "unbound" variab= les from "undefined" variables, two types of errors, though informally "defined" is often used more loosely:=20 1. Every variable reference must refer to a lexically visible binding, or it is an "unbound" variable error. Inherently, this can *always* be detected statically at compile time. 2. A variable may not be accessed until it has been initialized. (More precisely, we could discuss a variable's "location", which I see is part of Guile's model.) This is the essence of the `letrec`/`letrec*` restriction: while the "fixing letrec" papers[4][5] show that useful static analysis can be done, this is fundamentally a dynamic property. Violating the restriction is an "undefined" variable error at runtime. >=20 > I hope that illustrates a little when and when not cyclic imports work! This helped quite a bit with avoiding problems in practice---thanks! I think there are two aspects I still feel uncertain about: =46irst, it is not clear to me why there was a dependency between (gnu packages chez) and (gnu packages racket) prior to the change that prompted me to ask this. Given what Ludo=E2=80=99 wrote in : > > Do you have any advice on what would be good practice? >=20 > For package modules, the main things are: > 1. Don=E2=80=99t use #:select or #:autoload for (gnu packages =E2=80=A6= ) modules in a > (gnu packages =E2=80=A6) module. > 2. At the top level of a module, only refer to variables within that > module. it sounds like maybe there's some kind of implicit cycle involving---most? all?---(gnu packages =E2=80=A6) modules. Is that true? Second, Guile's semantics being what they are, is it considered good practi= ce to take advantage of this dynamic variable resolution and tolerance for cyc= lic modules? My personal taste (you may not be surprised to hear) is to prefer more static semantics and handling cycles explicitly when they are truly necessary. But if this is a pervasive idiom in Guile, I can try to write wi= th less of a Racket "accent". Thanks again for this detailed explanation! =2DPhilip [1]:=20 https://docs.racket-lang.org/reference/eval-model.html#%28part._vars-and-lo= cs%29 [2]:=20 https://docs.racket-lang.org/reference/eval-model.html#%28part._module-eval= =2Dmodel%29 [3]:=20 https://docs.racket-lang.org/reference/syntax-model.html#%28part._expand-co= ntext-model%29 [4]: https://citeseerx.ist.psu.edu/viewdoc/download?doi=3D10.1.1.441.8816&rep=3D= rep1&type=3Dpdf [5]: https://citeseerx.ist.psu.edu/viewdoc/download?doi=3D10.1.1.309.420&rep=3Dr= ep1&type=3Dpdf --nextPart2466355.TVEiu29ZbQ Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part. Content-Transfer-Encoding: 7Bit -----BEGIN PGP SIGNATURE----- iQIzBAABCgAdFiEE9GWrrNY3rqwUFVXPygNjjfo/HHoFAmJAcMkACgkQygNjjfo/ HHrjfw/+PZ3YxzCNYYAKgAQ2S1K98G0iQiF3SqcpIbxxS/UJ5CZSwt4+902wn3Q+ t7szoeBUtl3rzUUFUjQX9yBZ1f4Ho837lj2ow2FLLOdoqcMsXUDnXf1Uaq9IjylZ h/DEVhtoMxh9GrJxfpR+TPJ5tyq6U4WZuYPKANYWLSp+/r6NnCVEx8bVHPaYv6xv NDJRuaouaH+zHAM6WcGM5yy7PzRD4lhdYlaGgAw9zRsdlAiiL8dKsSU84p21nLwX GdoNiaa6XS8qvZeoIsCgEWIbgAFMMn84nb8UDv1FvmTBajaXLplXGdblYzyRJcY8 78u9M2PJBTL3i8/QyBmMgCZ2l+f+8xaDH7Ki7SfwEmcs1DztTNO1QeJjEJjhlHwg so6rp+Uzq2NMfUuZPEdkO76ThgmSVbHLpRNvLOVUa2u7HgirXoKZ1AcQwlkzctJ2 B292vtawqpreik5tFOc2Zr5SMwOGcvtPjCy9DPLQs8wRcLv5pkKfkTPPDjbzOxJu e3PYs9Ppf/VBc0zxxpu1plrV6yF15IXS8G5fpzHerzf1cTglcQhtl8RfK+1gfsf1 rA37EDOIe7j1GEQIY700K+oJ+xUY/t+Ulk8MfkCi3eRPLNDh9UHu4VCbOnIFMIbJ OF4dAut9+DriTN5p9OYAYvTsBSzvmxaussSq0H6cU0RqZOAVSws= =BKNG -----END PGP SIGNATURE----- --nextPart2466355.TVEiu29ZbQ--