From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp1.migadu.com ([2001:41d0:403:4876::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms8.migadu.com with LMTPS id 6K7ZIMpq02VlDgEA62LTzQ:P1 (envelope-from ) for ; Mon, 19 Feb 2024 15:50:50 +0100 Received: from aspmx1.migadu.com ([2001:41d0:403:4876::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp1.migadu.com with LMTPS id 6K7ZIMpq02VlDgEA62LTzQ (envelope-from ) for ; Mon, 19 Feb 2024 15:50:50 +0100 X-Envelope-To: larch@yhetil.org Authentication-Results: aspmx1.migadu.com; dkim=pass header.d=telenet.be header.s=r24 header.b=NFmPlqEP; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org"; dmarc=pass (policy=none) header.from=telenet.be ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1708354250; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post:dkim-signature; bh=Ytuq3KWjFXToyKvkxVY2WSGL1pHi72yy4mGELkBh3Ww=; b=n3Zb97A0SHlSHCx1/83yqGegI53xDzAth7fOXJc0iVIsclwrZOJIN/S1lcZ1og5a0h0c2L mbf4QpSo53D9w3T1RIAQ8vrQzgWGQgqmlcwQ7GslaZ2rGxihVAR8X6gGVVmFu/67IBaSvH NfM1UFLWLRuLel6kLOIV+KoB/B9lVrypShN5/M7anx7lkLbbMDuVbN+5oW40GE+G3VSMy6 pCSlm0PAXCcx45A4Or0qx0+oMTU6eIYbZV90Z18Fyvh3rDhYlzDxxKChPTE/x88bFvbCjL E1Nt4IzFpX2W05byemNxb44FtnWCutLB3NqAJPHpMItQt8+CdLeT92M4RBFxXQ== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=pass header.d=telenet.be header.s=r24 header.b=NFmPlqEP; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org"; dmarc=pass (policy=none) header.from=telenet.be ARC-Seal: i=1; s=key1; d=yhetil.org; t=1708354250; a=rsa-sha256; cv=none; b=TGi1EssVT0K3aYU7ujcdEewPQqT0YvGh4h65+R2PAWuALVCE3DWWT8Sl2i13ceLqiW4oth 5XzfIeh3yxOmttCa+UYuOAlVJGPeYAd+ttvXqvlWitbpfKGx3zi1J+1FAWoqCKI5c1rPEy BuU/+wg8O/MTZWSF07kbA9VVN0z8nxUr52vVr5H949vntH0Y+uknn4mfKVVGUpuLybl2Ug W7D5pqVlyYJkS6B2av75BCJNty7GNfun71SX/cipP3iJNvMKI0ECrTaNr5/8kWa1tmUMLD VwhBKbWtxqR1SYZmgtVoMeeea57hvn6oDfdjis5LKzn98WZXzNv/9STX/gVeQw== Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 03AC4CE27 for ; Mon, 19 Feb 2024 15:50:50 +0100 (CET) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rc4y0-0001bF-1E; Mon, 19 Feb 2024 09:50:00 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rc4xw-0001VA-Oz for guix-devel@gnu.org; Mon, 19 Feb 2024 09:49:56 -0500 Received: from baptiste.telenet-ops.be ([2a02:1800:120:4::f00:13]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1rc4xt-0007EI-78 for guix-devel@gnu.org; Mon, 19 Feb 2024 09:49:56 -0500 Received: from [IPv6:2a02:1808:86:2214:8190:adfc:998:6dab] ([IPv6:2a02:1808:86:2214:8190:adfc:998:6dab]) by baptiste.telenet-ops.be with bizsmtp id p2kj2B0094wMGJ4012kk71; Mon, 19 Feb 2024 15:44:44 +0100 Message-ID: <20240219154444.p2kj2B0094wMGJ4012kk71@baptiste.telenet-ops.be> MIME-Version: 1.0 To: Steve George , "guix-devel@gnu.org" Cc: r0man , Reilly Siegel From: Maxime Devos Subject: RE: Proposal to turn off AOT in clojure-build-system Date: Mon, 19 Feb 2024 15:44:44 +0100 Importance: normal X-Priority: 3 In-Reply-To: References: Content-Type: multipart/alternative; boundary="_D04EECC6-1F16-4F1B-B438-2384465A95FE_" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=telenet.be; s=r24; t=1708353884; bh=lBL1JE2s2MV5R1UMV3lefLnVac9m+Ld7Hs0FHW908nE=; h=To:Cc:From:Subject:Date:In-Reply-To:References; b=NFmPlqEPyh93Xq6BZbHwifW0r2XwLbTTKW2AIlM3m406ePJEhKjbRTe8ZjhPnhnWc P67ZD8X5ie3L08hqAA+TGazUNbj0w5WapGjnkbFHJCAKqoSvrYP96Yg2zFkVQ3aNFC 7pJ2i/acXZHimRqmgKrwo5DPp9EqEVSNMA0m9hCXsMKAz1THrMNX4i6KFVzP/07K5h eOuaxqsoRi50eVqCyt1m+aJY/c2T8V0V1HV8yNrl04cHKUXkPu1v6Lmo7rSWr94bg0 CrxMoX06d8otp2o2st6yQpqKm5XrlbGHYmYPCFR6LLy5CQFcGRpn/t0ym2zbfqIzFN JyefkF5B8Ao1w== Received-SPF: pass client-ip=2a02:1800:120:4::f00:13; envelope-from=maximedevos@telenet.be; helo=baptiste.telenet-ops.be X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_LOW=-0.7, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: guix-devel-bounces+larch=yhetil.org@gnu.org X-Migadu-Flow: FLOW_IN X-Migadu-Country: US X-Migadu-Scanner: mx10.migadu.com X-Migadu-Spam-Score: -7.97 X-Spam-Score: -7.97 X-Migadu-Queue-Id: 03AC4CE27 X-TUID: 8h3hSzHDBlF9 --_D04EECC6-1F16-4F1B-B438-2384465A95FE_ Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" ((As I said about a month ago, in theory I should have access to a proper e= -mail program again a week or two in the past.)) >Hi, >Guix's clojure-build-system turns on AOT compilation by default. I would l= ike to advocate that 'as a distributor' we should *not* ship Clojure code A= OT'd, so we should change the default. >This has been discussed previously. In #56604 r0man noted that AOT compila= tion should not be on by default [0], Reilly makes the same point in #53765= [1]. >Maxime makes the point that where a compiler is available it should be use= d [2] and that if it doesn't work it's a bug: >> "if a Clojure library misbehaves when AOT-compiled, without additional = context, that seems like a bug in the Clojure library to me (or the AOT-com= pilation code). >The perspective in the Clojure community is quite different from Guix's on= a number of fronts. There's not much discussion about offline builds This (not the has nothing to do with AOT. > reproducibility If some libraries have irreproducible binaries, then just don=E2=80=99t AOT= those libraries (and, if the irreproducibility is in macros (and maybe inl= inable generated code, if that=E2=80=99s something the AOT does), the relev= ant dependents of those libraries as well). An AOT problem for a few libraries is not a reason to turn off AOT by defau= lt, it is a reason to turn off AOT for those few libraries in particular. > or code coming from Distributions.=20 You are contradicting this by the next paragraph, in which you mention the = distribution =E2=80=9CClojars=E2=80=9D. A rather specialised and lacking di= stribution, but a distribution nonetheless. >The internalised perspective is that you use the build tools to download l= ibraries directly from Clojars (a Maven repo) That seems quite similar to Guix, where you use the build tools =E2=80=9Cgu= ix build=E2=80=9D / =E2=80=9Cguix install=E2=80=9D / ... to download librar= ies (by default, when available) directly from ci.guix.gnu.org (I forgot th= e terminology, but you could compare it to Clojars I guess). > and developers create a final uberjar for production usage Except for the delusion of grandeur (AFAIK, in non-Java English, =C3=BCber = only appears in the word =C3=9Cbermensch) (maybe whoever coined the term di= dn=E2=80=99t really mean it but eergh), this is also familiar, see =E2=80= =9Cguix system vm=E2=80=9D for large-scale things, and the command for crea= ting relocatable packs (I forgot the exact name and command) on a smaller s= cale. > Consequently, there is no specific statement saying 'Distributors should = not AOT libraries' that I can point to. In this bit about differences in perspective, I haven=E2=80=99t seen any me= ntion of AOT, hence the =E2=80=9CConsequently=E2=80=9D does not follow. The= part that=E2=80=99s missing here is that (IIUC) in Clojure, it is somewhat= conventional to stuff the compiled .class files in a superior Aryan JAR in= stead =E2=80=93 the inferior UnderJARs you get from the =E2=80=9Cguix insta= ll clj-whatever=E2=80=9D equivalent would only contain non-compiled .clj (a= nd data files, whatever). > But, I would like to draw attention to this thread on Clojureverse as the= best source I could find: >Alex Miller is the main community manager for Clojure, and is a maintainer= of the core libraries, so his perspective is key. He notes that, AOT code = is tied to *specific versions of Clojure*: > > "AOT'ed code is that it is inherently the product of a particular versio= n of tthe Clojure compiler ... I would recommend NOT AOT compiling librarie= s" [4] This reasoning does not follow =E2=80=93 yes, it is tied to the Clojure ver= sion, so what? Guix automatically rebuilds dependents when the dependency (= in this case, the Clojure compiler) changes. I guess the maintainer has something like Maven etc. in mind, where many ma= intainers, each with different distribution, Java version, etc. upload bina= ries, but that=E2=80=99s not the case in Guix. (To a lesser extent, this is= not the case in, say, Debian as well.) >In the same thread thheller, who is the maintainer of the most popular Clo= jureScript tooling, notes you cannot mix AOT and non-AOT libraries [5]: "you cannot just ship your library AOT compiles as it would also contai= n clojure.core. Clojure AOT current ... can not load clj files from .class = files. So AOT produces the class files and will fail if one of the dependen= t classes is missing although the .clj file is present" This argument makes no sense at all. Of course it can=E2=80=99t load CLJ fi= les from .class files =E2=80=93 Clojure AOT is a compiler, not a decompiler= , cf. how GCC can=E2=80=99t =E2=80=98load=E2=80=99 C files from .o / ELF / = ..., and why would it in the first place? And of course you can=E2=80=99t run or compile an application if a part of = the binaries / a part of the source code is missing. This is the same as wi= th C & GCC, (Guile) Scheme & =E2=80=9Cguild compile=E2=80=9D, etc.. Yet, G= CC and Guile somehow have a functioning AOT (if we count Guile=E2=80=99s by= tecode as AOT, which we can because we are counting .class files as =E2=80= =9Ccompiled=E2=80=9D). (As a side-remark, I think a typo was made here: dependent classes -> depe= ndency classes (i.e. classes of dependencies).) Everything after the first sentence appears to be true, but the conclusion = of the first sentence does not follow. >I believe this means that with AOT code on, any user who installs a differ= ent version of Clojure from the one that we used to AOT the libraries *may*= have problems. Unlike, say, Maven, this situation simply does not happen in Guix, because = we don=E2=80=99t just download binaries and call it a day (except for some = bootstrapping stuff, but that=E2=80=99s not relevant for Clojure AOT), beca= use we have functioning recompilation of dependents, because of shebang pat= ching, because binaries that are to be invoked should not rely on the ambie= nt CLASSPATH / LD_LIBRARY_PATH / etc. and, if, the underlying binaries do r= ely on that, they are wrapped (see wrap-program) to set them (or, at least,= they should be, you might find some bugs in this department if you go look= ing). Even if they aren=E2=80=99t wrapped, then in that case the dependencies are= propagated-inputs, and you can only have a single version of a propagated = package in the same environment (barring stacking environment shenanigans, = but then you are looking for it and/or you can just report a bug about how = the binaries should be wrapped/classpath should be patched in/...). And, that we can't have trees where some part is AOT'd but a dependency is = not. Finally, there is no expectation in the Clojure community that this is= a bug, consequently it will not be fixed. Therefore, we should change to d= efault to AOT off. >What do people think, does this make sense? No. Here is an argument for disabling AOT that I think makes sense: >The main problem is that AOT will compile your code __and all its dependen= cies__. This redundancy is wasteful, albeit otherwise harmless. (adapted from https://clojureverse.org/t/deploying-aot-compiled-libraries/2= 545/3, but with a different conclusion.) Going by https://clojureverse.org/t/deploying-aot-compiled-libraries/2545/4= and its response, this appears to be a surmountable problem however. Presumably there is a way to select _which_ code is compiled (instead of co= mpiling the code and its dependencies) =E2=80=93 IIRC, this is done already= by clojure-build-system? >[0] https://debbugs.gnu.org/cgi/bugreport.cgi?bug=3D56604#5 >[1] https://debbugs.gnu.org/cgi/bugreport.cgi?bug=3D53765#290 >[2] https://debbugs.gnu.org/cgi/bugreport.cgi?bug=3D53765#293 >[4] https://clojureverse.org/t/deploying-aot-compiled-libraries/2545/6 >[5] https://clojureverse.org/t/deploying-aot-compiled-libraries/2545/3 >[5] https://gist.github.com/hiredman/c5710ad9247c6da12a99ff6c26dd442e --_D04EECC6-1F16-4F1B-B438-2384465A95FE_ Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset="utf-8"

((As I said about a month ago, in theory I should have access= to a proper e-mail program again a week or two in the past.))

 

>Hi,=

 

>Guix's clojure-b= uild-system turns on AOT compilation by default. I would like to advocate t= hat 'as a distributor' we should *not* ship Clojure code AOT'd, so we shoul= d change the default.

 

>This has been discussed previously. In #56604 r0man noted that AOT co= mpilation should not be on by default [0], Reilly makes the same point in #= 53765 [1].

 

>Maxime= makes the point that where a compiler is available it should be used [2] a= nd that if it doesn't work it's a bug:

 

<= span lang=3Den-BE>>>=C2=A0 "if a Clojure library misbehaves when= AOT-compiled, without additional context, that seems like a bug in the Clo= jure library to me (or the AOT-compilation code).

 

>The perspective in the Clojure community i= s quite different from Guix's on a number of fronts. There's not much discu= ssion about offline builds

 

This (not the has nothing to do with AOT.

 

> reproducibility

 

If some libraries have irreproducible binaries, = then just don=E2=80=99t AOT those libraries (and, if the irreproducibility = is in macros (and maybe inlinable generated code, if that=E2=80=99s somethi= ng the AOT does), the relevant dependents of those libraries as well).=

 <= /span>

An AOT problem for a few = libraries is not a reason to turn off AOT by default, it is a reason to tur= n off AOT for those few libraries in particular.

 

> or code coming from Distributions.

 

You are contradicting this = by the next paragraph, in which you mention the distribution =E2=80=9CCloja= rs=E2=80=9D. A rather specialised and lacking distribution, but a distribut= ion nonetheless.

 

>= The internalised perspective is that you use the build tools to download li= braries directly from Clojars (a Maven repo)

 

That seems quite similar to Guix, where you use th= e build tools =E2=80=9Cguix build=E2=80=9D / =E2=80=9Cguix install=E2=80=9D= / ... to download libraries (by default, when available) directly from ci.= guix.gnu.org (I forgot the terminology, but you could compare it to Clojars= I guess).

 

> and d= evelopers create a final uberjar for production usage

=

 

Except for the delusion of grandeur (AFAIK= , in non-Java English, =C3=BCber only appears in the word =C3=9Cbermensch) = (maybe whoever coined the term didn=E2=80=99t really mean it but eergh), th= is is also familiar, see =E2=80=9Cguix system vm=E2=80=9D for large-scale t= hings, and the command for creating relocatable packs (I forgot the exact n= ame and command) on a smaller scale.

 

> Consequently, there is no specific statement saying 'D= istributors should not AOT libraries' that I can point to.

 

In this bit about differences in pers= pective, I haven=E2=80=99t seen any mention of AOT, hence the =E2=80=9CCons= equently=E2=80=9D does not follow. The part that=E2=80=99s missing here is = that (IIUC) in Clojure, it is somewhat conventional to stuff the compiled .= class files in a superior Aryan JAR instead =E2=80=93 the inferior UnderJAR= s you get from the =E2=80=9Cguix install clj-whatever=E2=80=9D equivalent w= ould only contain non-compiled .clj (and data files, whatever).<= /span>

 <= /p>

> But, I would like to draw a= ttention to this thread on Clojureverse as the best source I could find:

>Alex Miller = is the main community manager for Clojure, and is a maintainer of the core = libraries, so his perspective is key. He notes that, AOT code is tied to *s= pecific versions of Clojure*:

> 

>=C2=A0 "AOT'ed code is that it is inherently the prod= uct of a particular version of tthe Clojure compiler ... I would recommend = NOT AOT compiling libraries" [4]

 

This reasoning does not follow =E2=80=93 yes, it is tied t= o the Clojure version, so what? Guix automatically rebuilds dependents when= the dependency (in this case, the Clojure compiler) changes.

 

I guess the maintainer has somethi= ng like Maven etc. in mind, where many maintainers, each with different dis= tribution, Java version, etc. upload binaries, but that=E2=80=99s not the c= ase in Guix. (To a lesser extent, this is not the case in, say, Debian as w= ell.)

&nb= sp;

>In the same= thread thheller, who is the maintainer of the most popular ClojureScript t= ooling,=C2=A0 notes you cannot mix AOT and non-AOT libraries [5]:

 

=C2=A0=C2=A0=C2=A0 "you c= annot just ship your library AOT compiles as it would also contain clojure.= core. Clojure AOT current ... can not load clj files from .class files. So = AOT produces the class files and will fail if one of the dependent classes = is missing although the .clj file is present"

 

This argument makes no sense at all. Of cou= rse it can=E2=80=99t load CLJ files from .class files =E2=80=93 Clojure AOT= is a compiler, not a decompiler, cf. how GCC can=E2=80=99t =E2=80=98load= =E2=80=99 C files from .o / ELF / ..., and why would it in the first place?=

 

And of course you ca= n=E2=80=99t run or compile an application if a part of the binaries / a par= t of the source code is missing. This is the same as with C & GCC, (Gui= le) Scheme & =E2=80=9Cguild compile=E2=80=9D, etc..=C2=A0 Yet, GCC and = Guile somehow have a functioning AOT (if we count Guile=E2=80=99s bytecode = as AOT, which we can because we are counting .class files as =E2=80=9Ccompi= led=E2=80=9D).

 

(As a = side-remark, I think a typo was made here:=C2=A0 dependent classes -> de= pendency classes (i.e. classes of dependencies).)

 

Everything after the first sentence appears to= be true, but the conclusion of the first sentence does not follow.

 

>I believe this means tha= t with AOT code on, any user who installs a different version of Clojure fr= om the one that we used to AOT the libraries *may* have problems.

 

Unlike, say, Maven, this situa= tion simply does not happen in Guix, because we don=E2=80=99t just download= binaries and call it a day (except for some bootstrapping stuff, but that= =E2=80=99s not relevant for Clojure AOT), because we have functioning recom= pilation of dependents, because of shebang patching, because binaries that = are to be invoked should not rely on the ambient CLASSPATH / LD_LIBRARY_PAT= H / etc. and, if, the underlying binaries do rely on that, they are wrapped= (see wrap-program) to set them (or, at least, they should be, you might fi= nd some bugs in this department if you go looking).

 

Even if they aren=E2=80=99t wrapped, then i= n that case the dependencies are propagated-inputs, and you can only have a= single version of a propagated package in the same environment (barring st= acking environment shenanigans, but then you are looking for it and/or you = can just report a bug about how the binaries should be wrapped/classpath sh= ould be patched in/...).

 

And, that we can't have trees where some part is AOT'd but a dependency= is not. Finally, there is no expectation in the Clojure community that thi= s is a bug, consequently it will not be fixed. Therefore, we should change = to default to AOT off.

 

>What do people think, does this make sense?

 

No.

=  

Here is an argument for disabling AOT that I think makes sense:=

 

>The main problem is that AOT will compile your code __and all = its dependencies__. This redundancy is wasteful, albeit otherwise harmless.=

 

(adapted from ht= tps://clojureverse.org/t/deploying-aot-compiled-libraries/2545/3, but w= ith a different conclusion.)

 

Going by https://clojureverse.org/t/deploying-aot-compiled-lib= raries/2545/4 and its response, this appears to be a surmountable probl= em however.

 

Presumabl= y there is a way to select _which_ code is compiled (instead of comp= iling the code and its dependencies) =E2=80=93 IIRC, this is done already b= y clojure-build-system?

 

>[0] https://debbugs.gnu.org/cgi/bugreport.cgi?bug=3D56604#5

>[1] https://debbug= s.gnu.org/cgi/bugreport.cgi?bug=3D53765#290

>[2] https://debbugs.gnu.org/cgi/bugrepo= rt.cgi?bug=3D53765#293

>[4] https://clojureverse.org/t/deploying-aot-compiled-librarie= s/2545/6

>[= 5] https://clojureverse.org/t/deploying-aot-compiled-libraries/2545/3<= /o:p>

>[5] https://gis= t.github.com/hiredman/c5710ad9247c6da12a99ff6c26dd442e

 

= = --_D04EECC6-1F16-4F1B-B438-2384465A95FE_--