From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp0.migadu.com ([2001:41d0:303:e16b::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms8.migadu.com with LMTPS id 2Mv9JgIf7WWLjwAAqHPOHw:P1 (envelope-from ) for ; Sun, 10 Mar 2024 03:46:26 +0100 Received: from aspmx1.migadu.com ([2001:41d0:303:e16b::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp0.migadu.com with LMTPS id 2Mv9JgIf7WWLjwAAqHPOHw (envelope-from ) for ; Sun, 10 Mar 2024 03:46:26 +0100 X-Envelope-To: larch@yhetil.org Authentication-Results: aspmx1.migadu.com; dkim=pass header.d=retrospec.tv header.s=fm3 header.b=NJNksc9X; dkim=pass header.d=messagingengine.com header.s=fm1 header.b="S 8twdXM"; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org"; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1710038786; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post:dkim-signature; bh=IHNCvYYka6vLOjdZQWAKaqcsAaFBlqFYb7PqHOF2yng=; b=C70UWS3WIN6iEkKYAMR2OGLmySPLb0SqHzEWVnZp8I12vToGn0tNErkIZQ30eU5fWZkoPg eiBfJnnoT2fwUqWyZOLJfBIH2/6E5u2M/mBjUwOhWtBu8O1QUHGhaMl2nQcj81p2OrprzP zz8uu39nRqxuDLyp5c4DNiruamLJxk9tef0fFLWK+kH51yhvG/ms3PlQMKux9ckmEZPey+ Acbtaj/oqfRT6dU/xRwqjgw+n/yHdZUEgdCYvvMLMR61sPwMWMJXj3oJJRMx4qowHLCiN8 N4rPrjW1pKjZ3xJ4hwJkpR86DqP4y7tIbQVPB9AE5teqnDZ6mCPXPHkRcsvPig== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=pass header.d=retrospec.tv header.s=fm3 header.b=NJNksc9X; dkim=pass header.d=messagingengine.com header.s=fm1 header.b="S 8twdXM"; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org"; dmarc=none ARC-Seal: i=1; s=key1; d=yhetil.org; t=1710038786; a=rsa-sha256; cv=none; b=nnMnv61jzPSabtjNoVI4McX18ho2b3a4DxxoqjOoTdLy5XBZ41EECjCxasjkMdVWavL8R5 VWgMruz+qRKJn/dodZfxrmoawvsq96tWaNuQ7FXy24meJgC5LV+z1r4KCBmj8hlGeS+D8x 8JBqLZtorB8g2tI/GCaQbWkXFQRRJXoqLaVPndaxKai/PaUoTLK5aaRaoJbHuNbCe2okZ0 SMHdxVJSqGWqEQBrspaL/mtZ/b88GGkMaCW7scHoUjEg2j3LntWRJ4TQ8udImRNb5lcitA asEuC+/0Iqalsak553XxAt3ISXntwB8VA9JYhmGNIjUfYjkPBtD6prQI+K9W1w== Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 424CF131DB for ; Sun, 10 Mar 2024 03:46:26 +0100 (CET) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rj8oL-00014X-VB; Sat, 09 Mar 2024 21:21:13 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rj8oJ-00014N-1E for guix-devel@gnu.org; Sat, 09 Mar 2024 21:21:11 -0500 Received: from wfout5-smtp.messagingengine.com ([64.147.123.148]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rj8oF-0004NV-0r for guix-devel@gnu.org; Sat, 09 Mar 2024 21:21:10 -0500 Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailfout.west.internal (Postfix) with ESMTP id F170E1C00083; Sat, 9 Mar 2024 21:20:59 -0500 (EST) Received: from mailfrontend1 ([10.202.2.162]) by compute1.internal (MEProxy); Sat, 09 Mar 2024 21:21:00 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=retrospec.tv; h= cc:cc:content-transfer-encoding:content-type:content-type:date :date:from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:subject:subject:to:to; s=fm3; t=1710037259; x=1710123659; bh=IHNCvYYka6vLOjdZQWAKaqcsAaFBlqFYb7PqHOF2yng=; b= NJNksc9XcXz6pIjQRNr4aFgFvJHmmiQQHSg69NGheyOzV51u0FBqfm7MZdTpNvNM LRteMlSLBNfWvTabEDlyItL9qGXdSphXU45s4YhfG1T6+suF1af9EqMboGr745Kw ad+rkdne4L2lBAXnaHw3NegUfTSLbOy21NbtEG2RXgEta6Wn/NA3OBXkrgOHYDL/ Wuwp+EB3eLIue3RJ+2QcNyzS88yeZvgb+WF1l6/CbG9BWlOJgl+Oe83FvdZEcfXC O6WomsUo77+0kVRw+nFQsa+5EcR1XHFzYnjPNHKep8f+0nyGRlXbJuQitDZTfjTl 1cmAlcNFflLJiafSjjA2gA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:content-type:date:date:feedback-id:feedback-id :from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:subject:subject:to:to:x-me-proxy:x-me-proxy :x-me-sender:x-me-sender:x-sasl-enc; s=fm1; t=1710037259; x= 1710123659; bh=IHNCvYYka6vLOjdZQWAKaqcsAaFBlqFYb7PqHOF2yng=; b=S 8twdXMlYDrLhBi5qU7kYeZfoFk/ONyh6+80PsSjic7PjIVE6S+OZWwPrEDbMFjJ7 SBKB4uMpjSDTP1SiMZOzpKm3C0QFoZhAJAUAVrlvPtxGXDH/iADsyaIMNiNzCqz+ wzI8DGmmQ+eolpJUjNy2OQYZdeapSixbXStiHPnRKrQH+eHapKCj14cKLsBdY4ud 44X24Vltwe8n7sSEGaWTXkLqvtL4Ya4v76O/F9BpKtn6L/rsDVIpcrpj75oUTtfi yO70vOgU9kiP3xzL+Nv9YnnknDH/sNXNog3XMIssmLL784RZfFArK2RUZv6ekWdT haoIPHoqwKvb6B1AJu1TA== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvledrieekgdeghecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc fjughrpehffgfhvfevufffjgfkgggtgfesthhqredttderjeenucfhrhhomhepkfgrnhcu gfhurhgvuceoihgrnhesrhgvthhrohhsphgvtgdrthhvqeenucggtffrrghtthgvrhhnpe fhgfehgfduhfekjedugfeffffftdfhieeuueehkeellefgiefgteekueethfeftdenucff ohhmrghinheprghtlhgrshhsihgrnhdrnhgvthdpghhithhhuhgsrdgtohhmpdgtlhhojh hurhgvvhgvrhhsvgdrohhrghdptghlohhjuhhrvgdrohhrghdpghhnuhdrohhrghenucev lhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpehirghnsehrvg htrhhoshhpvggtrdhtvh X-ME-Proxy: Feedback-ID: id9014242:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Sat, 9 Mar 2024 21:20:58 -0500 (EST) References: User-agent: mu4e 1.8.13; emacs 28.2 From: Ian Eure To: Steve George Cc: r0man , Reilly Siegel , Maxime Devos , guix-devel@gnu.org Subject: Re: Proposal to turn off AOT in clojure-build-system Date: Sat, 09 Mar 2024 14:27:56 -0800 In-reply-to: Message-ID: <878r2qx1x2.fsf@meson> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: quoted-printable Received-SPF: pass client-ip=64.147.123.148; envelope-from=ian@retrospec.tv; helo=wfout5-smtp.messagingengine.com X-Spam_score_int: -11 X-Spam_score: -1.2 X-Spam_bar: - X-Spam_report: (-1.2 / 5.0 requ) BAYES_00=-1.9, DATE_IN_PAST_03_06=1.592, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: guix-devel-bounces+larch=yhetil.org@gnu.org X-Migadu-Flow: FLOW_IN X-Migadu-Country: US X-Migadu-Spam-Score: -8.85 X-Spam-Score: -8.85 X-Migadu-Queue-Id: 424CF131DB X-Migadu-Scanner: mx13.migadu.com X-TUID: C6SQg8MON+zR Hello, I=E2=80=99ve been following along with this discussion, as well as a=20 discussion on Clojureverse, and thought it might be helpful to=20 pull together some threads and design decisions around Clojure=E2=80=99s=20 behavior. Clojure is designed to ship libraries as source artifacts, not=20 bytecode ("pretty much all other Clojure libraries ... are all=20 source code by design[1]."; "Clojure is ... a source-first=20 language[2]"), and the view of the community is that shipping AOT=20 artifacts "is an anti-pattern[1]." Clojure library JARs are more=20 akin to source tarballs than binaries. The original design and=20 intent of Clojure=E2=80=99s AOT compiler is to compile "just a few=20 things... for the interop case" or "Everything... For the=20 'Application delivery', 'Syntax check', and 'reflection warnings'=20 cases[3]." Clojure=E2=80=99s compiler is transitive and "does not support separate=20 compilation"[3], meaning when a namespace is compiled, anything it=20 uses is compiled and emitted with it. This is the crux of why=20 mixing AOT and non-AOT code is troublesome: it causes dependency=20 diamonds, where the AOT=E2=80=99d library contains a duplicate, older=20 version of code used elsewhere in the project. The Clojure reference on compiling[4] gives some reasons you might=20 want to AOT: "To deliver your application without source," "To=20 speed up application startup," "To generate named classes for use=20 by Java," "To create an application that does not need runtime=20 bytecode generation and custom classloaders." Note that there=E2=80=99s=20 no mention of compiling libraries for any reason; only=20 applications. When AOT is used "for the interop case," it=E2=80=99s typical to AOT only=20 those namespaces[5], not the entire library. Shipping AOT-compiled Clojure libraries has caused real and very=20 weird and hard-to-debug problems in the past: https://clojure.atlassian.net/browse/CLJ-1886?focusedCommentId=3D15290 https://github.com/clj-commons/byte-streams/issues/68 and=20 https://clojure.atlassian.net/browse/CLJ-1741 Clojure doesn=E2=80=99t have guarantees around ABI stability[6][7]. To=20 date, most ABI changes have been additive, but there are no=20 guarantees that the ABI will be compatible from any one version of=20 Clojure to any other. The understanding of the Clojure community=20 is that the design of the current compiler can=E2=80=99t offer a stable=20 ABI[8] at all. Because nobody in the Clojure community AOTs=20 intermediate (that is, library) code, this hasn=E2=80=99t been a problem=20 and is unlikely to change. "Clojure tries very hard to provide source compatibility but not=20 bytecode compatibility across versions[9]." Correctly handling the ABI concerns =E2=80=94 which Guix currently does=20 not do =E2=80=94 would result in a combinatorial explosion of Clojure=20 packages should multiple versions of Clojure ever be available in=20 Guix at the same time. For example, if someone wanted to package=20 Clojure 1.12.0-alpha9, you=E2=80=99d need to duplicate every package=20 taking Clojure as an input so they use the correct version. While=20 ABI breakage has been rare thus far, it seems likely that it=E2=80=99ll=20 occur at some point; perhaps if Clojure reaches version 2.0.0. If=20 Guix disables AOT for Clojure libraries, we have source=20 compatibility, and the AOT/ABI problems are moot. Clojure=E2=80=99s compiler is non-deterministic[10]: the same compiler can= =20 will produce different bytecode for the same input across multiple=20 runs. I=E2=80=99m not sure if this is a problem for Guix at this point in= =20 time, but it seems out of line with Guix expectations for=20 compilation generally. Opinions follow: If we=E2=80=99re taking votes, mine is to *not* AOT Clojure libraries,=20 both for the technical reasons laid out in, and also for the=20 social reason of not violating the principle of least surprise. I=20 understand that Guix and Clojure have very different approaches,=20 and some balance must be struck. However, the lack of ABI=20 guarantees, the compiler=E2=80=99s behavior, the promise of source=20 compatibility, and matching the expectation of the audience these=20 tools are meant for all convince me that disabling AOT is the=20 right course here. AOT=E2=80=99ing Clojure applications (which means, more or less, "the=20 Clojure tooling") is desirable, and should be maintained. =E2=80=94 Ian [1]:=20 https://clojureverse.org/t/should-linux-distributions-ship-clojure-byte-com= piled-aot-or-not/10595/8 [2]:=20 https://clojureverse.org/t/should-linux-distributions-ship-clojure-byte-com= piled-aot-or-not/10595/30 [3]: https://clojure.org/reference/compilation [4]:=20 https://archive.clojure.org/design-wiki/display/design/Transitive%2BAOT%2BC= ompilation.html [5]: https://clojure.org/guides/deps_and_cli#aot_compilation [6]:=20 https://clojureverse.org/t/should-linux-distributions-ship-clojure-byte-com= piled-aot-or-not/10595/30 [7]:=20 https://gist.github.com/hiredman/c5710ad9247c6da12a99ff6c26dd442e [8]:=20 https://clojureverse.org/t/should-linux-distributions-ship-clojure-byte-com= piled-aot-or-not/10595/4 [9]:=20 https://clojureverse.org/t/should-linux-distributions-ship-clojure-byte-com= piled-aot-or-not/10595/18 [10]:=20 https://ask.clojure.org/index.php/12249/bytecode-not-100-deterministic-give= n-identical-inputs Steve George writes: > Hi, > > Guix's clojure-build-system turns on AOT compilation by=20 > default. I would like to advocate that 'as a distributor' we=20 > should *not* ship Clojure code AOT'd, so we should change the=20 > default. > > This has been discussed previously. In #56604 r0man noted that=20 > AOT compilation should not be on by default [0], Reilly makes=20 > the same point in #53765 [1]. > > Maxime makes the point that where a compiler is available it=20 > should be used [2] and that if it doesn't work it's a bug: > > "if a Clojure library misbehaves when AOT-compiled, without=20 > additional context, that seems like a bug in the Clojure=20 > library to me (or the AOT-compilation code). > > The perspective in the Clojure community is quite different from=20 > Guix's on a number of fronts. There's not much discussion about=20 > offline builds, reproducibility or code coming from=20 > Distributions. The internalised perspective is that you use the=20 > build tools to download libraries directly from Clojars (a Maven=20 > repo) and developers create a final uberjar for production usage=20 > Consequently, there is no specific statement saying=20 > 'Distributors should not AOT libraries' that I can point=20 > to. But, I would like to draw attention to this thread on=20 > Clojureverse as the best source I could find: > > Alex Miller is the main community manager for Clojure, and is a=20 > maintainer of the core libraries, so his perspective is key. He=20 > notes that, AOT code is tied to *specific versions of Clojure*: > > "AOT'ed code is that it is inherently the product of a=20 > particular version of tthe Clojure compiler ... I would=20 > recommend NOT AOT compiling libraries" [4] > > In the same thread thheller, who is the maintainer of the most=20 > popular ClojureScript tooling, notes you cannot mix AOT and=20 > non-AOT libraries [5]: > > "you cannot just ship your library AOT compiles as it would=20 > also contain clojure.core. Clojure AOT current ... can not=20 > load clj files from .class files. So AOT produces the class=20 > files and will fail if one of the dependent classes is=20 > missing although the .clj file is present" > > I believe this means that with AOT code on, any user who=20 > installs a different version of Clojure from the one that we=20 > used to AOT the libraries *may* have problems. And, that we=20 > can't have trees where some part is AOT'd but a dependency is=20 > not. Finally, there is no expectation in the Clojure community=20 > that this is a bug, consequently it will not be=20 > fixed. Therefore, we should change to default to AOT off. > > What do people think, does this make sense? > > Thanks, > > Steve / Futurile > > [0] https://debbugs.gnu.org/cgi/bugreport.cgi?bug=3D56604#5 > [1] https://debbugs.gnu.org/cgi/bugreport.cgi?bug=3D53765#290 > [2] https://debbugs.gnu.org/cgi/bugreport.cgi?bug=3D53765#293 > [4]=20 > https://clojureverse.org/t/deploying-aot-compiled-libraries/2545/6 > [5]=20 > https://clojureverse.org/t/deploying-aot-compiled-libraries/2545/3 > [5]=20 > https://gist.github.com/hiredman/c5710ad9247c6da12a99ff6c26dd442e