Proposal to turn off AOT in clojure-build-system

Proposal to turn off AOT in clojure-build-system

[Steve George]

Hi,

Guix's clojure-build-system turns on AOT compilation by default. I would like to advocate that 'as a distributor' we should *not* ship Clojure code AOT'd, so we should change the default.

This has been discussed previously. In #56604 r0man noted that AOT compilation should not be on by default [0], Reilly makes the same point in #53765 [1].

Maxime makes the point that where a compiler is available it should be used [2] and that if it doesn't work it's a bug:

  "if a Clojure library misbehaves when AOT-compiled, without additional context, that seems like a bug in the Clojure library to me (or the AOT-compilation code).

The perspective in the Clojure community is quite different from Guix's on a number of fronts. There's not much discussion about offline builds, reproducibility or code coming from Distributions. The internalised perspective is that you use the build tools to download libraries directly from Clojars (a Maven repo) and developers create a final uberjar for production usage Consequently, there is no specific statement saying 'Distributors should not AOT libraries' that I can point to. But, I would like to draw attention to this thread on Clojureverse as the best source I could find:

Alex Miller is the main community manager for Clojure, and is a maintainer of the core libraries, so his perspective is key. He notes that, AOT code is tied to *specific versions of Clojure*:

  "AOT'ed code is that it is inherently the product of a particular version of tthe Clojure compiler ... I would recommend NOT AOT compiling libraries" [4]

In the same thread thheller, who is the maintainer of the most popular ClojureScript tooling,  notes you cannot mix AOT and non-AOT libraries [5]:

    "you cannot just ship your library AOT compiles as it would also contain clojure.core. Clojure AOT current ... can not load clj files from .class files. So AOT produces the class files and will fail if one of the dependent classes is missing although the .clj file is present"

I believe this means that with AOT code on, any user who installs a different version of Clojure from the one that we used to AOT the libraries *may* have problems. And, that we can't have trees where some part is AOT'd but a dependency is not. Finally, there is no expectation in the Clojure community that this is a bug, consequently it will not be fixed. Therefore, we should change to default to AOT off.

What do people think, does this make sense?

Thanks,

Steve / Futurile

[0] https://debbugs.gnu.org/cgi/bugreport.cgi?bug=56604#5
[1] https://debbugs.gnu.org/cgi/bugreport.cgi?bug=53765#290
[2] https://debbugs.gnu.org/cgi/bugreport.cgi?bug=53765#293
[4] https://clojureverse.org/t/deploying-aot-compiled-libraries/2545/6
[5] https://clojureverse.org/t/deploying-aot-compiled-libraries/2545/3
[5] https://gist.github.com/hiredman/c5710ad9247c6da12a99ff6c26dd442e

RE: Proposal to turn off AOT in clojure-build-system

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 7677 bytes --]

((As I said about a month ago, in theory I should have access to a proper e-mail program again a week or two in the past.))

>Hi,

>Guix's clojure-build-system turns on AOT compilation by default. I would like to advocate that 'as a distributor' we should *not* ship Clojure code AOT'd, so we should change the default.

>This has been discussed previously. In #56604 r0man noted that AOT compilation should not be on by default [0], Reilly makes the same point in #53765 [1].

>Maxime makes the point that where a compiler is available it should be used [2] and that if it doesn't work it's a bug:

>>  "if a Clojure library misbehaves when AOT-compiled, without additional context, that seems like a bug in the Clojure library to me (or the AOT-compilation code).

>The perspective in the Clojure community is quite different from Guix's on a number of fronts. There's not much discussion about offline builds

This (not the has nothing to do with AOT.

> reproducibility

If some libraries have irreproducible binaries, then just don’t AOT those libraries (and, if the irreproducibility is in macros (and maybe inlinable generated code, if that’s something the AOT does), the relevant dependents of those libraries as well).

An AOT problem for a few libraries is not a reason to turn off AOT by default, it is a reason to turn off AOT for those few libraries in particular.

> or code coming from Distributions. 

You are contradicting this by the next paragraph, in which you mention the distribution “Clojars”. A rather specialised and lacking distribution, but a distribution nonetheless.

>The internalised perspective is that you use the build tools to download libraries directly from Clojars (a Maven repo)

That seems quite similar to Guix, where you use the build tools “guix build” / “guix install” / ... to download libraries (by default, when available) directly from ci.guix.gnu.org (I forgot the terminology, but you could compare it to Clojars I guess).

> and developers create a final uberjar for production usage

Except for the delusion of grandeur (AFAIK, in non-Java English, über only appears in the word Übermensch) (maybe whoever coined the term didn’t really mean it but eergh), this is also familiar, see “guix system vm” for large-scale things, and the command for creating relocatable packs (I forgot the exact name and command) on a smaller scale.

> Consequently, there is no specific statement saying 'Distributors should not AOT libraries' that I can point to.

In this bit about differences in perspective, I haven’t seen any mention of AOT, hence the “Consequently” does not follow. The part that’s missing here is that (IIUC) in Clojure, it is somewhat conventional to stuff the compiled .class files in a superior Aryan JAR instead – the inferior UnderJARs you get from the “guix install clj-whatever” equivalent would only contain non-compiled .clj (and data files, whatever).

> But, I would like to draw attention to this thread on Clojureverse as the best source I could find:
>Alex Miller is the main community manager for Clojure, and is a maintainer of the core libraries, so his perspective is key. He notes that, AOT code is tied to *specific versions of Clojure*:
>
>  "AOT'ed code is that it is inherently the product of a particular version of tthe Clojure compiler ... I would recommend NOT AOT compiling libraries" [4]

This reasoning does not follow – yes, it is tied to the Clojure version, so what? Guix automatically rebuilds dependents when the dependency (in this case, the Clojure compiler) changes.

I guess the maintainer has something like Maven etc. in mind, where many maintainers, each with different distribution, Java version, etc. upload binaries, but that’s not the case in Guix. (To a lesser extent, this is not the case in, say, Debian as well.)

>In the same thread thheller, who is the maintainer of the most popular ClojureScript tooling,  notes you cannot mix AOT and non-AOT libraries [5]:

    "you cannot just ship your library AOT compiles as it would also contain clojure.core. Clojure AOT current ... can not load clj files from .class files. So AOT produces the class files and will fail if one of the dependent classes is missing although the .clj file is present"

This argument makes no sense at all. Of course it can’t load CLJ files from .class files – Clojure AOT is a compiler, not a decompiler, cf. how GCC can’t ‘load’ C files from .o / ELF / ..., and why would it in the first place?

And of course you can’t run or compile an application if a part of the binaries / a part of the source code is missing. This is the same as with C & GCC, (Guile) Scheme & “guild compile”, etc..  Yet, GCC and Guile somehow have a functioning AOT (if we count Guile’s bytecode as AOT, which we can because we are counting .class files as “compiled”).

(As a side-remark, I think a typo was made here:  dependent classes -> dependency classes (i.e. classes of dependencies).)

Everything after the first sentence appears to be true, but the conclusion of the first sentence does not follow.

>I believe this means that with AOT code on, any user who installs a different version of Clojure from the one that we used to AOT the libraries *may* have problems.

Unlike, say, Maven, this situation simply does not happen in Guix, because we don’t just download binaries and call it a day (except for some bootstrapping stuff, but that’s not relevant for Clojure AOT), because we have functioning recompilation of dependents, because of shebang patching, because binaries that are to be invoked should not rely on the ambient CLASSPATH / LD_LIBRARY_PATH / etc. and, if, the underlying binaries do rely on that, they are wrapped (see wrap-program) to set them (or, at least, they should be, you might find some bugs in this department if you go looking).

Even if they aren’t wrapped, then in that case the dependencies are propagated-inputs, and you can only have a single version of a propagated package in the same environment (barring stacking environment shenanigans, but then you are looking for it and/or you can just report a bug about how the binaries should be wrapped/classpath should be patched in/...).

And, that we can't have trees where some part is AOT'd but a dependency is not. Finally, there is no expectation in the Clojure community that this is a bug, consequently it will not be fixed. Therefore, we should change to default to AOT off.

>What do people think, does this make sense?

No.

Here is an argument for disabling AOT that I think makes sense:

>The main problem is that AOT will compile your code __and all its dependencies__. This redundancy is wasteful, albeit otherwise harmless.

(adapted from https://clojureverse.org/t/deploying-aot-compiled-libraries/2545/3, but with a different conclusion.)

Going by https://clojureverse.org/t/deploying-aot-compiled-libraries/2545/4 and its response, this appears to be a surmountable problem however.

Presumably there is a way to select _which_ code is compiled (instead of compiling the code and its dependencies) – IIRC, this is done already by clojure-build-system?

>[0] https://debbugs.gnu.org/cgi/bugreport.cgi?bug=56604#5
>[1] https://debbugs.gnu.org/cgi/bugreport.cgi?bug=53765#290
>[2] https://debbugs.gnu.org/cgi/bugreport.cgi?bug=53765#293
>[4] https://clojureverse.org/t/deploying-aot-compiled-libraries/2545/6
>[5] https://clojureverse.org/t/deploying-aot-compiled-libraries/2545/3
>[5] https://gist.github.com/hiredman/c5710ad9247c6da12a99ff6c26dd442e


[-- Attachment #2: Type: text/html, Size: 14407 bytes --]

Re: Proposal to turn off AOT in clojure-build-system

[Steve George]

Hi Maxime,

On 19 Feb, Maxime Devos wrote:
(...) 
> > Consequently, there is no specific statement saying 'Distributors should not AOT libraries' that I can point to.
> 
> In this bit about differences in perspective, I haven’t seen any mention of AOT, hence the “Consequently” does not follow. The part that’s missing here is that (IIUC) in Clojure, it is somewhat conventional to stuff the compiled .class files in a superior Aryan JAR instead – the inferior UnderJARs you get from the “guix install clj-whatever” equivalent would only contain non-compiled .clj (and data files, whatever).
> 
> > But, I would like to draw attention to this thread on Clojureverse as the best source I could find:
> >Alex Miller is the main community manager for Clojure, and is a maintainer of the core libraries, so his perspective is key. He notes that, AOT code is tied to *specific versions of Clojure*:
> >
> >  "AOT'ed code is that it is inherently the product of a particular version of tthe Clojure compiler ... I would recommend NOT AOT compiling libraries" [4]
> 
> This reasoning does not follow – yes, it is tied to the Clojure version, so what? Guix automatically rebuilds dependents when the dependency (in this case, the Clojure compiler) changes.
(...)

I think this preceding sentence is the heart of different assumptions between us.

The Clojure packages we haave are for developers writing applications (libraries and tools). The ecosystem has very few end-user applications [0]. As a Clojure developer I can use the Clojure tools from Guix, and a few libraries. We (and all the other distributions) have a miniscule portion of the Clojure/Java library universe [1]. This leads to the following usage scenarios:

1. A developer installs Clojure from Guix, and uses libraries from outside Guix.
They can install the JVM/Clojure and some common tools (like clj-tools-cli). They will use libraries from elsewhere, including their own. AOT compilation is a problem because of the issue of mixed AOT and non-AOT.

2. A developer installs a Clojure from outside Guix, uses libraries from inside Guix
This will cause problems because the Guix Clojure libraries will have been AOT'd by a different version of the compiler. It's also fairly common to install/use parallel versions of Clojure/jvm due to different deployment needs, this is likely to cause difficult to find bugs.

I can see the sense of compiling to byte code if it's an end-user application. In that case we'd want to make the start-up as fast as possible. Your comments seem to have this use-case in mind.

But, today there aren't any such end-user Clojure applications in Guix.

> >I believe this means that with AOT code on, any user who installs a different version of Clojure from the one that we used to AOT the libraries *may* have problems.
> 
> Unlike, say, Maven, this situation simply does not happen in Guix, because we don’t just download binaries and call it a day (except for some bootstrapping stuff, but that’s not relevant for Clojure AOT), because we have functioning recompilation of dependents, because of shebang patching, because binaries that are to be invoked should not rely on the ambient CLASSPATH / LD_LIBRARY_PATH / etc. and, if, the underlying binaries do rely on that, they are wrapped (see wrap-program) to set them (or, at least, they should be, you might find some bugs in this department if you go looking).
> 
> Even if they aren’t wrapped, then in that case the dependencies are propagated-inputs, and you can only have a single version of a propagated package in the same environment (barring stacking environment shenanigans, but then you are looking for it and/or you can just report a bug about how the binaries should be wrapped/classpath should be patched in/...).
> 

In this paragraph you're assumption is that a Guix user is only using libraries from within Guix. Hopefully, I've shown why this assumption is unlikely above.

You also mentioned Debian, and @e.hashiman [2] said that Clojure libraries are not AOT'd on Debian, while applications are. From what I can find there are 130 packages in Debian with the word Clojure in them [3]. I looked at a selection and it seems true that Debian does not AOT libraries (and I can't find any Clojure 'apps'). For completeness I also checked what Clojars, the main distribution archive for Clojure developers, does:

- https://sources.debian.org/src/core-match-clojure/1.0.0-1/project.clj/
    - Lein-based project - has to have :aot keyword - distributes as source files
    - Clojure source files (.clj) in Debian
    - Clojure source files in Clojars
    - Byte compiled files in GUIX
    - installed and inspected with jar -tvf

- https://sources.debian.org/src/data-csv-clojure/1.0.0-1/deps.edn/
    - tools.deps project - has to have a specific aot alias - distributes as source files
    - Clojure source files (.clj) in the Debian provided jar
    - Clojure source files in the Clojars provided jar
    - Byte compiled files in Guix
    - installed and inspected with jar -tvf

I tried looking at both Arch and Nix but neither of them package any Clojure libraries at all.

Does this help clarify why I'm asking to change the default?

Thanks,

Steve

[0] https://fr.surveymonkey.com/stories/SM-_2BH3b49f_2FXEkUlrb_2BJSThxg_3D_3D/ , there are tools for programmers like clj and babashka
[1] We have 12 Clojure packages, Debian has 180 and according to Maven central (https://mvnrepository.com/repos) there are ~2,000 repos, Clojars shows 3105 projects.
[2] https://mastodon.social/@ehashman@cloudisland.nz/111960714219364737
[3] https://sources.debian.org/search/clojure/
[4] https://www.debian.org/doc/packaging-manuals/java-policy/ch02.html#policy-libraries

RE: Proposal to turn off AOT in clojure-build-system

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 8720 bytes --]

>> [...] 
>> > But, I would like to draw attention to this thread on Clojureverse as the best source I could find:
>> >Alex Miller is the main community manager for Clojure, and is a maintainer of the core libraries, so his perspective is key. He notes that, AOT code is tied to *specific versions of Clojure*:
>> >
>> >  "AOT'ed code is that it is inherently the product of a particular version of tthe Clojure compiler ... I would recommend NOT AOT compiling libraries" [4]
>> 
>> This reasoning does not follow – yes, it is tied to the Clojure version, so what? Guix automatically rebuilds dependents when the dependency (in this case, the Clojure compiler) changes.
(...)

>I think this preceding sentence is the heart of different assumptions between us.

>The Clojure packages we haave are for developers writing applications (libraries and tools). The ecosystem has very few end-user applications [0]. As a Clojure developer I can use the Clojure tools from Guix, and a few libraries. We (and all the other distributions) have a miniscule portion of the Clojure/Java library universe [1]. This leads to the following usage scenarios:

>1. A developer installs Clojure from Guix, and uses libraries from outside Guix.
They can install the JVM/Clojure and some common tools (like clj-tools-cli). They will use libraries from elsewhere, including their own. AOT compilation is a problem because of the issue of mixed AOT and non-AOT.

>2. A developer installs a Clojure from outside Guix, uses libraries from inside Guix
This will cause problems because the Guix Clojure libraries will have been AOT'd by a different version of the compiler. It's also fairly common to install/use parallel versions of Clojure/jvm due to different deployment needs, this is likely to cause difficult to find bugs.

My answer to (1) and (2) is:

(a) About “install Clojure from outside Guix + use libraries inside Guix”:

 If these libraries are AOT:

Don’t do that, then. If you mix-and-match binaries (in this case, .class files) different distributions, you are free to do so, but when (not if, when) things break, you get to keep the pieces.

If these libraries are just the .clj files (not AOT) (which as I understand it is the standard situation): doesn’t seem a problem to me (see point (b)).

Adding source from other places is one thing (probably ok), adding binaries is another (probably not ok, especially if unstable ABIs (see Clojure compiler) and mismatched versions (see hypotheses of 2.) are involved.

(b) What is this “the issue of mixed AOT and non-AOT”? Do you have a source on this? Besides Clojure supposedly, I haven’t ever heard of such problems for any language – for example, there is no such issue with Guile and AFAIK not for Python. I haven’t heard of any such issues for the Common Lisp implementations either (though I haven’t checked), so this doesn’t seem like a “Clojure doesn’t do hygienic macros” issue.

(c) Guix isn’t forcing anyone to use AOT’d libraries. If people really want to assist in murdering the climate (or its a situation where total cost of non-AOT is lower than AOT), they can unfortunately (*) simply do so applying a recursive transformation that adds #:aot? #false flags everywhere or whatever the exact argument is (**).

Given that this transformation has some legitimate use cases, this transformation could even be a pre-canned procedure + transformation included in Guix itself

(*) ‘unfortunately’ only applies to the first case. For the case in parentheses, replace by ‘fortunately’.
(**) IIRC is wasn’t #:aot? but some other name, but that’s not really the point here.

(d) If a deployment need multiple versions of Clojure, then just fix your deployment to make everything work with the latest version of Clojure. Or, if you for some reason don’t do that, just use a (recursive) transformation to change the version of the Clojure compiler used to match the Clojure that will be used in the particular deployment.

(e) You can simply add missing packages to Guix as the need arises.

>I can see the sense of compiling to byte code if it's an end-user application. In that case we'd want to make the start-up as fast as possible. Your comments seem to have this use-case in mind.

>But, today there aren't any such end-user Clojure applications in Guix.

That’s a shame. Hopefully that will change some day.
Also, this is false, “clojure-tools” exists – developers are people too (I don’t care about the “_end_-user” distinction – surely developers want fast start-up as well).

Also, I _don’t_ have that use case in mind – I have efficiency in general in mind, efficiency of the start-up is only a part of that.

The point is: most likely you will want the application to have AOT code, and that library has dependencies. Furthermore, likely many of these dependencies are dependencies of other applications as well.

Instead of redoing the compilation of N shared dependencies for each M applications  (total work: ~N*M), by not participating in these Clojure conventions, the total work of compiling dependencies can be reduced to ~N).

Let’s avoid joining the madness that is Go’s build system and Cargo(*).

(*) Rust is fine for compilation, it’s cargo that is the problem (at least, Cargo as used by Guix) – antioxidant (https://notabug.org/maximed/cargoless-rust-experiments) avoids the sheer inefficiency of Cargo.

>> >I believe this means that with AOT code on, any user who installs a different version of Clojure from the one that we used to AOT the libraries *may* have problems.
>> 
>> Unlike, say, Maven, this situation simply does not happen in Guix, because we don’t just download binaries and call it a day (except for some bootstrapping stuff, but that’s not relevant for Clojure AOT), because we have functioning recompilation of dependents, because of shebang patching, because binaries that are to be invoked should not rely on the ambient CLASSPATH / LD_LIBRARY_PATH / etc. and, if, the underlying binaries do rely on that, they are wrapped (see wrap-program) to set them (or, at least, they should be, you might find some bugs in this department if you go looking).
>> 
>> Even if they aren’t wrapped, then in that case the dependencies are propagated-inputs, and you can only have a single version of a propagated package in the same environment (barring stacking environment shenanigans, but then you are looking for it and/or you can just report a bug about how the binaries should be wrapped/classpath should be patched in/...).
> 

>In this paragraph you're assumption is that a Guix user is only using libraries from within Guix. Hopefully, I've shown why this assumption is unlikely above.

You haven’t. You have shown that other situations exist, not that this situation is unlikely. To demonstrate likelihood / unlikelyhood, you need some statistics (or a very thorough argument on why people wouldn’t just make a package for missing dependencies – it’s very simple, especially if you aren’t sending them for inclusion.)

>You also mentioned Debian, and @e.hashiman [2] said that Clojure libraries are not AOT'd on Debian, while applications are. From what I can find there are 130 packages in Debian with the word Clojure in them [3]. I looked at a selection and it seems true that Debian does not AOT libraries (and I can't find any Clojure 'apps'). For completeness I also checked what Clojars, the main distribution archive for Clojure developers, does:
> [...]

OK ... and? Other people doing things differently doesn’t mean those different approaches are better. If Debian mentions somewhere the reason _why_ it doesn’t AOT libraries, that reason can be investigated and perhaps carried over to Guix, but it’s the ‘why’ that’s important, not the ‘it doesn’t’.

> Does this help clarify why I'm asking to change the default?

Yes. It appears you are unfamiliar with the “transformations” functionality of Guix, which allows for easy rewriting of packages to whatever Clojure version you want, eliminating compatibility concerns or even turning off AOT completely.

(This requires the relevant versions of Clojure to be packaged in Guix, but that shouldn’t be too much of a problem.)

It also appears you are unfamiliar with Guix importers, which easily make package definitions (somewhat low quality w.r.t. descriptions, with lack of verification of freeness, etc., but usually otherwise fully functional). I don’t know if an importer for Clojure exists but it shouldn’t be too difficult I think.

Best regards,
Maxime Devos.

[-- Attachment #2: Type: text/html, Size: 23213 bytes --]

Re: Proposal to turn off AOT in clojure-build-system

[Andreas Enge]

Am Thu, Feb 22, 2024 at 03:57:41PM +0100 schrieb Maxime Devos:
> Yes. It appears you are unfamiliar with (...)
> It also appears you are unfamiliar with (...)

May I suggest to not make assumptions about what other people are familiar
with or not? There is no point in claiming that others are less knowledge-
able than you; they may know as much or even more than you, and still come
to different conclusions. (And even if people were unfamiliar with
something, I would object to this haughty tone and suggest a more pleasant
way of making suggestions.)

For instance concerning the topic at hand, knowing that users may transform
packages as they wish to me seems to be independent of which default choice
we should make for the distribution.

Andreas

Re: Proposal to turn off AOT in clojure-build-system

[Felix Lechner via Development of GNU Guix and the GNU System distribution.]

Hey,

On Thu, Feb 22 2024, Andreas Enge wrote:

> Am Thu, Feb 22, 2024 at 03:57:41PM +0100 schrieb Maxime Devos:
>> Yes. It appears you are unfamiliar with (...)
>> It also appears you are unfamiliar with (...)
>
> May I suggest to not make assumptions about what other people are familiar
> with or not? There is no point in claiming that others are less knowledge-
> able than you; they may know as much or even more than you

Asking questions can avoid any such conclusions. It's my favorite
thing. Here, I might have asked: "Do you know about...?"

Goodwill toward all,
Felix

RE: Proposal to turn off AOT in clojure-build-system

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 638 bytes --]

>On Thu, Feb 22 2024, Andreas Enge wrote:
>> Am Thu, Feb 22, 2024 at 03:57:41PM +0100 schrieb Maxime Devos:
>>> Yes. It appears you are unfamiliar with (...)
>>> It also appears you are unfamiliar with (...)
>>
>> May I suggest to not make assumptions about what other people are familiar
>> with or not? There is no point in claiming that others are less knowledge-
>> able than you; they may know as much or even more than you
>
>Asking questions can avoid any such conclusions. It's my favorite
>thing. Here, I might have asked: "Do you know about...?"

OK. I don’t think I’ll do that.

Best regards,
Maxime Devos.

[-- Attachment #2: Type: text/html, Size: 2106 bytes --]

Re: Proposal to turn off AOT in clojure-build-system

[Maxim Cournoyer]

Hi Maxime!

Maxime Devos <maximedevos@telenet.be> writes:

>>On Thu, Feb 22 2024, Andreas Enge wrote:
>>> Am Thu, Feb 22, 2024 at 03:57:41PM +0100 schrieb Maxime Devos:
>>>> Yes. It appears you are unfamiliar with (...)
>>>> It also appears you are unfamiliar with (...)
>>>
>>> May I suggest to not make assumptions about what other people are familiar
>>> with or not? There is no point in claiming that others are less knowledge-
>>> able than you; they may know as much or even more than you
>>
>>Asking questions can avoid any such conclusions. It's my favorite
>>thing. Here, I might have asked: "Do you know about...?"
>
> OK. I don’t think I’ll do that.

Your sharp replies came off as harsh, which is unfortunate because you
seem to bring points of technical merits.  Please smooth out your
interactions/communication style, to ensure it remains pleasant for
everyone involved, taking these code of conduct items into account:

* Demonstrating empathy and kindness toward other people
* Being respectful of differing opinions, viewpoints, and experiences

-- 
Thanks,
Maxim

RE: Proposal to turn off AOT in clojure-build-system

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 3678 bytes --]

>Am Thu, Feb 22, 2024 at 03:57:41PM +0100 schrieb Maxime Devos:
>> Yes. It appears you are unfamiliar with (...)
>> It also appears you are unfamiliar with (...)

>May I suggest to not make assumptions about what other people are familiar
with or not? There is no point in claiming that others are less knowledge-
able than you; they may know as much or even more than you, and still come
to different conclusions. (And even if people were unfamiliar with
something, I would object to this haughty tone and suggest a more pleasant
way of making suggestions.)

This is hypocritical, you are (somewhat implicitly) making assumptions on my (un)familiarity with good manners and (somewhat implicitly) claiming that I am less knowledges than you on good manners. Furthermore, you aren’t actually suggesting a more pleasant to formulate the message of the part you quoted.

Also, I did not simply _assume_ that Steve was unfamiliar with transformation, I _concluded_ that Steve was likely unfamiliar by what they didn’t mention in their e-mails. Furthermore, the mere “likelihood” is included in the paragraph you quoted as part of the word “appears” – I did not claim that Steve is unfamiliar, I only claimed that it appeared to be the case, which is not the same thing. 

After all, perhaps Steve does know that such transformation exist but personally concluded them to not be a proper solution for some reason. In that case, now people know there is a disagreement on the role of transformations w.r.t. AOT and perhaps the source of the disagreement can be resolved, furthering knowledge and bring us closer to deciding on what the proper AOT default would be.

You mention that other people might know more, but there is also a flipside to this – sometimes people know _less_. In this case, perhaps Steve simply did not know about transformations and the proposed use of transformations in combination with enabling AOT by default would be agreeable to Steve and as such perhaps an answer to the decision whether to enable AOT by default.

As such, simply _assuming_ that Steve knew of transformations would be wrong, so I had to mention the option of transformations.

I did not simply claim that Steve is less knowledgable than me, at most it could be said that I (implicitly) claimed that Steve is less knowledged than me on the relation between transformations and Clojure AOT problems, but even then, I included a qualifier “It appears that”, not “It is the case that”.

The beginning “It appears you are unfamiliar with [...]” is simply a perfectly cromulent beginning of a sentence, not some assertion of superiority.

>For instance concerning the topic at hand, knowing that users may transform
packages as they wish to me seems to be independent of which default choice
we should make for the distribution.

It is not independent, see my previous e-mail where I explained how the existence of transformations turns some problems mentioned w.r.t. enabling AOT into non-problems.

If you have a disagreement with that explanation, please actually say what the disagreement is instead of only saying that you disagree. The former might bring us closer to some collective decision on what the proper default behaviour is for Guix, the latter doesn’t, is almost useless and makes it look like you didn’t bother to read the ‘(...)’ part.

While I suppose it is technically possible you have read the part ‘(...)’, given your response I simply don’t believe you did and hence I consider it fair to conclude that you are not familiar with the contents of the (...) part.

Best regards,
Maxime Devos

[-- Attachment #2: Type: text/html, Size: 5728 bytes --]

RE: Proposal to turn off AOT in clojure-build-system

[Felix Lechner via Development of GNU Guix and the GNU System distribution.]

Hey Maxime,

On Thu, Feb 22 2024, Maxime Devos wrote:

> familiarity with good manners

Sometimes it's better to admit defeat. Maybe you find this apology [1]
helpful, although it was imperfect.

Kind regards
Felix

[1] https://lists.gnu.org/archive/html/guix-devel/2022-10/msg00141.html

Re: Proposal to turn off AOT in clojure-build-system

[Steve George]

Hi,

I don't think I'm making any progress convincing you, and I'm not enjoying the interaction so I'm going to take a few days off from this thread.

I've tried to ask the Clojure community for a definitive expression of what they think Linux distributions should do with byte-compiled libraries. We'll see if there's a response, this is a public forum so feel free to chime in and express clearly your perpective:

  https://clojureverse.org/t/should-linux-distributions-ship-clojure-byte-compiled-aot-or-not/10595/1

Thanks,

Steve / Futurile

On 22 Feb, Maxime Devos wrote:
> >> [...] 
> >> > But, I would like to draw attention to this thread on Clojureverse as the best source I could find:
> >> >Alex Miller is the main community manager for Clojure, and is a maintainer of the core libraries, so his perspective is key. He notes that, AOT code is tied to *specific versions of Clojure*:
> >> >
> >> >  "AOT'ed code is that it is inherently the product of a particular version of tthe Clojure compiler ... I would recommend NOT AOT compiling libraries" [4]
> >> 
> >> This reasoning does not follow – yes, it is tied to the Clojure version, so what? Guix automatically rebuilds dependents when the dependency (in this case, the Clojure compiler) changes.
> (...)
> 
> >I think this preceding sentence is the heart of different assumptions between us.
> 
> >The Clojure packages we haave are for developers writing applications (libraries and tools). The ecosystem has very few end-user applications [0]. As a Clojure developer I can use the Clojure tools from Guix, and a few libraries. We (and all the other distributions) have a miniscule portion of the Clojure/Java library universe [1]. This leads to the following usage scenarios:
> 
> >1. A developer installs Clojure from Guix, and uses libraries from outside Guix.
> They can install the JVM/Clojure and some common tools (like clj-tools-cli). They will use libraries from elsewhere, including their own. AOT compilation is a problem because of the issue of mixed AOT and non-AOT.
> 
> >2. A developer installs a Clojure from outside Guix, uses libraries from inside Guix
> This will cause problems because the Guix Clojure libraries will have been AOT'd by a different version of the compiler. It's also fairly common to install/use parallel versions of Clojure/jvm due to different deployment needs, this is likely to cause difficult to find bugs.
> 
> My answer to (1) and (2) is:
> 
> (a) About “install Clojure from outside Guix + use libraries inside Guix”:
> 
>  If these libraries are AOT:
> 
> Don’t do that, then. If you mix-and-match binaries (in this case, .class files) different distributions, you are free to do so, but when (not if, when) things break, you get to keep the pieces.
> 
> If these libraries are just the .clj files (not AOT) (which as I understand it is the standard situation): doesn’t seem a problem to me (see point (b)).
> 
> Adding source from other places is one thing (probably ok), adding binaries is another (probably not ok, especially if unstable ABIs (see Clojure compiler) and mismatched versions (see hypotheses of 2.) are involved.
> 
> (b) What is this “the issue of mixed AOT and non-AOT”? Do you have a source on this? Besides Clojure supposedly, I haven’t ever heard of such problems for any language – for example, there is no such issue with Guile and AFAIK not for Python. I haven’t heard of any such issues for the Common Lisp implementations either (though I haven’t checked), so this doesn’t seem like a “Clojure doesn’t do hygienic macros” issue.
> 
> (c) Guix isn’t forcing anyone to use AOT’d libraries. If people really want to assist in murdering the climate (or its a situation where total cost of non-AOT is lower than AOT), they can unfortunately (*) simply do so applying a recursive transformation that adds #:aot? #false flags everywhere or whatever the exact argument is (**).
> 
> Given that this transformation has some legitimate use cases, this transformation could even be a pre-canned procedure + transformation included in Guix itself
> 
> (*) ‘unfortunately’ only applies to the first case. For the case in parentheses, replace by ‘fortunately’.
> (**) IIRC is wasn’t #:aot? but some other name, but that’s not really the point here.
> 
> (d) If a deployment need multiple versions of Clojure, then just fix your deployment to make everything work with the latest version of Clojure. Or, if you for some reason don’t do that, just use a (recursive) transformation to change the version of the Clojure compiler used to match the Clojure that will be used in the particular deployment.
> 
> (e) You can simply add missing packages to Guix as the need arises.
> 
> >I can see the sense of compiling to byte code if it's an end-user application. In that case we'd want to make the start-up as fast as possible. Your comments seem to have this use-case in mind.
> 
> >But, today there aren't any such end-user Clojure applications in Guix.
> 
> That’s a shame. Hopefully that will change some day.
> Also, this is false, “clojure-tools” exists – developers are people too (I don’t care about the “_end_-user” distinction – surely developers want fast start-up as well).
> 
> Also, I _don’t_ have that use case in mind – I have efficiency in general in mind, efficiency of the start-up is only a part of that.
> 
> The point is: most likely you will want the application to have AOT code, and that library has dependencies. Furthermore, likely many of these dependencies are dependencies of other applications as well.
> 
> Instead of redoing the compilation of N shared dependencies for each M applications  (total work: ~N*M), by not participating in these Clojure conventions, the total work of compiling dependencies can be reduced to ~N).
> 
> Let’s avoid joining the madness that is Go’s build system and Cargo(*).
> 
> (*) Rust is fine for compilation, it’s cargo that is the problem (at least, Cargo as used by Guix) – antioxidant (https://notabug.org/maximed/cargoless-rust-experiments) avoids the sheer inefficiency of Cargo.
> 
> >> >I believe this means that with AOT code on, any user who installs a different version of Clojure from the one that we used to AOT the libraries *may* have problems.
> >> 
> >> Unlike, say, Maven, this situation simply does not happen in Guix, because we don’t just download binaries and call it a day (except for some bootstrapping stuff, but that’s not relevant for Clojure AOT), because we have functioning recompilation of dependents, because of shebang patching, because binaries that are to be invoked should not rely on the ambient CLASSPATH / LD_LIBRARY_PATH / etc. and, if, the underlying binaries do rely on that, they are wrapped (see wrap-program) to set them (or, at least, they should be, you might find some bugs in this department if you go looking).
> >> 
> >> Even if they aren’t wrapped, then in that case the dependencies are propagated-inputs, and you can only have a single version of a propagated package in the same environment (barring stacking environment shenanigans, but then you are looking for it and/or you can just report a bug about how the binaries should be wrapped/classpath should be patched in/...).
> > 
> 
> >In this paragraph you're assumption is that a Guix user is only using libraries from within Guix. Hopefully, I've shown why this assumption is unlikely above.
> 
> You haven’t. You have shown that other situations exist, not that this situation is unlikely. To demonstrate likelihood / unlikelyhood, you need some statistics (or a very thorough argument on why people wouldn’t just make a package for missing dependencies – it’s very simple, especially if you aren’t sending them for inclusion.)
> 
> >You also mentioned Debian, and @e.hashiman [2] said that Clojure libraries are not AOT'd on Debian, while applications are. From what I can find there are 130 packages in Debian with the word Clojure in them [3]. I looked at a selection and it seems true that Debian does not AOT libraries (and I can't find any Clojure 'apps'). For completeness I also checked what Clojars, the main distribution archive for Clojure developers, does:
> > [...]
> 
> OK ... and? Other people doing things differently doesn’t mean those different approaches are better. If Debian mentions somewhere the reason _why_ it doesn’t AOT libraries, that reason can be investigated and perhaps carried over to Guix, but it’s the ‘why’ that’s important, not the ‘it doesn’t’.
> 
> > Does this help clarify why I'm asking to change the default?
> 
> Yes. It appears you are unfamiliar with the “transformations” functionality of Guix, which allows for easy rewriting of packages to whatever Clojure version you want, eliminating compatibility concerns or even turning off AOT completely.
> 
> (This requires the relevant versions of Clojure to be packaged in Guix, but that shouldn’t be too much of a problem.)
> 
> It also appears you are unfamiliar with Guix importers, which easily make package definitions (somewhat low quality w.r.t. descriptions, with lack of verification of freeness, etc., but usually otherwise fully functional). I don’t know if an importer for Clojure exists but it shouldn’t be too difficult I think.
> 
> Best regards,
> Maxime Devos.

Re: Proposal to turn off AOT in clojure-build-system

[Ryan Sundberg]

As a daily Clojure programmer using Guix the default 'off' makes sense. The only situation where AOT compilation is useful is in the final runnable application programs, and even then, its often incompatible with AOT compilation in subtle ways. Having libraries aot compiled not only adds incompatibility (including which java the library was built for, which creates a "least common denominator problem which JDK you can use for the final build) but also wastes time and energy.

Identifying AOT compilation bugs can also be extremely frustrating to package maintainers as the underlying cause is usually not obvious. In my experience using AOT is the exception rather than the rule; it is a nice optimization when practical for release engineering, but typically the juice is not worth the squeeze.

Sincerely,
Ryan Sundberg

Feb 19, 2024 3:48:54 AM Steve George <steve@futurile.net>:

> Hi,
> 
> Guix's clojure-build-system turns on AOT compilation by default. I would like to advocate that 'as a distributor' we should *not* ship Clojure code AOT'd, so we should change the default.
> 
> This has been discussed previously. In #56604 r0man noted that AOT compilation should not be on by default [0], Reilly makes the same point in #53765 [1].
> 
> Maxime makes the point that where a compiler is available it should be used [2] and that if it doesn't work it's a bug:
> 
>   "if a Clojure library misbehaves when AOT-compiled, without additional context, that seems like a bug in the Clojure library to me (or the AOT-compilation code).
> 
> The perspective in the Clojure community is quite different from Guix's on a number of fronts. There's not much discussion about offline builds, reproducibility or code coming from Distributions. The internalised perspective is that you use the build tools to download libraries directly from Clojars (a Maven repo) and developers create a final uberjar for production usage Consequently, there is no specific statement saying 'Distributors should not AOT libraries' that I can point to. But, I would like to draw attention to this thread on Clojureverse as the best source I could find:
> 
> Alex Miller is the main community manager for Clojure, and is a maintainer of the core libraries, so his perspective is key. He notes that, AOT code is tied to *specific versions of Clojure*:
> 
>   "AOT'ed code is that it is inherently the product of a particular version of tthe Clojure compiler ... I would recommend NOT AOT compiling libraries" [4]
> 
> In the same thread thheller, who is the maintainer of the most popular ClojureScript tooling,  notes you cannot mix AOT and non-AOT libraries [5]:
> 
>     "you cannot just ship your library AOT compiles as it would also contain clojure.core. Clojure AOT current ... can not load clj files from .class files. So AOT produces the class files and will fail if one of the dependent classes is missing although the .clj file is present"
> 
> I believe this means that with AOT code on, any user who installs a different version of Clojure from the one that we used to AOT the libraries *may* have problems. And, that we can't have trees where some part is AOT'd but a dependency is not. Finally, there is no expectation in the Clojure community that this is a bug, consequently it will not be fixed. Therefore, we should change to default to AOT off.
> 
> What do people think, does this make sense?
> 
> Thanks,
> 
> Steve / Futurile
> 
> [0] https://debbugs.gnu.org/cgi/bugreport.cgi?bug=56604#5
> [1] https://debbugs.gnu.org/cgi/bugreport.cgi?bug=53765#290
> [2] https://debbugs.gnu.org/cgi/bugreport.cgi?bug=53765#293
> [4] https://clojureverse.org/t/deploying-aot-compiled-libraries/2545/6
> [5] https://clojureverse.org/t/deploying-aot-compiled-libraries/2545/3
> [5] https://gist.github.com/hiredman/c5710ad9247c6da12a99ff6c26dd442e

Re: Proposal to turn off AOT in clojure-build-system

[Carlo Zancanaro]

As someone who has worked as a professional Clojure programmer, I would
like to add my voice in support of this:

On Mon, Feb 19 2024, Ryan Sundberg wrote:
> ... In my experience using AOT is the exception rather than the rule;
> it is a nice optimization when practical for release engineering, but
> typically the juice is not worth the squeeze.

That is: the norm in the Clojure community is *not* using AOT, so there
isn't any specific statement against it. It's just assumed that you
won't use AOT unless you have a reason to do so.

Reading the clojure.org page about compilation[1] with this in mind, we
see four possible (although admittedly not exhaustive) reasons to
provide AOT compilation:
 - to deliver your application without source
 - to speed up application startup
 - to generated named classes
 - to create an application that does not need runtime bytecode
   generation and custom classloaders

I'm not convinced any of these apply across the board in Guix.
Potentially for some libraries and applications it does, and so should
be applied in those specific cases.

The next statement on the page, after those reasons to use AOT is: "The
Clojure compilation model preserves as much as possible the dynamic
nature of Clojure, in spite of the code-reloading limitations of Java."
Or, to read it another way: AOT compiled Clojure is not as dynamic as
on-the-fly compiled Clojure. This isn't a bug in Clojure, this is how it
is designed to function.

It could be argued that it is a bug in the individual libraries to not
work with AOT, but I think it's inappropriate for Guix to attempt to
impose "AOT is the right way to do things" on Clojure libraries. That is
not the norm in that community.

As a personal datapoint: I spent three and a half years working on
mission-critical Clojure libraries and applications, and I have never
used AOT compilation in production.

Carlo

[1]: https://clojure.org/reference/compilation

Re: Proposal to turn off AOT in clojure-build-system

[宋文武]

Steve George <steve@futurile.net> writes:

> Hi,
>
> Guix's clojure-build-system turns on AOT compilation by default. I would like to advocate that 'as a distributor' we should *not* ship Clojure code AOT'd, so we should change the default.
>
> This has been discussed previously. In #56604 r0man noted that AOT compilation should not be on by default [0], Reilly makes the same point in #53765 [1].

Hello, we have both source and binary packages for Common Lisp (eg:
cl-ppcre and sbcl-ppcre), the source packages are for development and
binary packages for build applications.  I think we can do the same for
Clojure, and yes package them in source package (without AOT) by
default, and add binary packages when needed by actual applications.

I'm not using clojure myself, but the use cases for development and
applications distributation are both important for Guix, and manage
development environments better is a big sell point.

Re: Proposal to turn off AOT in clojure-build-system

[Ian Eure]

Hello,

I’ve been following along with this discussion, as well as a 
discussion on Clojureverse, and thought it might be helpful to 
pull together some threads and design decisions around Clojure’s 
behavior.

Clojure is designed to ship libraries as source artifacts, not 
bytecode ("pretty much all other Clojure libraries ... are all 
source code by design[1]."; "Clojure is ... a source-first 
language[2]"), and the view of the community is that shipping AOT 
artifacts "is an anti-pattern[1]."   Clojure library JARs are more 
akin to source tarballs than binaries.  The original design and 
intent of Clojure’s AOT compiler is to compile "just a few 
things... for the interop case" or "Everything... For the 
'Application delivery', 'Syntax check', and 'reflection warnings' 
cases[3]."

Clojure’s compiler is transitive and "does not support separate 
compilation"[3], meaning when a namespace is compiled, anything it 
uses is compiled and emitted with it.  This is the crux of why 
mixing AOT and non-AOT code is troublesome: it causes dependency 
diamonds, where the AOT’d library contains a duplicate, older 
version of code used elsewhere in the project.

The Clojure reference on compiling[4] gives some reasons you might 
want to AOT: "To deliver your application without source," "To 
speed up application startup," "To generate named classes for use 
by Java," "To create an application that does not need runtime 
bytecode generation and custom classloaders."  Note that there’s 
no mention of compiling libraries for any reason; only 
applications.

When AOT is used "for the interop case," it’s typical to AOT only 
those namespaces[5], not the entire library.

Shipping AOT-compiled Clojure libraries has caused real and very 
weird and hard-to-debug problems in the past:

    https://clojure.atlassian.net/browse/CLJ-1886?focusedCommentId=15290
    https://github.com/clj-commons/byte-streams/issues/68 and 
    https://clojure.atlassian.net/browse/CLJ-1741

Clojure doesn’t have guarantees around ABI stability[6][7].  To 
date, most ABI changes have been additive, but there are no 
guarantees that the ABI will be compatible from any one version of 
Clojure to any other.  The understanding of the Clojure community 
is that the design of the current compiler can’t offer a stable 
ABI[8] at all.  Because nobody in the Clojure community AOTs 
intermediate (that is, library) code, this hasn’t been a problem 
and is unlikely to change.

"Clojure tries very hard to provide source compatibility but not 
bytecode compatibility across versions[9]."

Correctly handling the ABI concerns — which Guix currently does 
not do — would result in a combinatorial explosion of Clojure 
packages should multiple versions of Clojure ever be available in 
Guix at the same time.  For example, if someone wanted to package 
Clojure 1.12.0-alpha9, you’d need to duplicate every package 
taking Clojure as an input so they use the correct version.  While 
ABI breakage has been rare thus far, it seems likely that it’ll 
occur at some point; perhaps if Clojure reaches version 2.0.0.  If 
Guix disables AOT for Clojure libraries, we have source 
compatibility, and the AOT/ABI problems are moot.

Clojure’s compiler is non-deterministic[10]: the same compiler can 
will produce different bytecode for the same input across multiple 
runs.  I’m not sure if this is a problem for Guix at this point in 
time, but it seems out of line with Guix expectations for 
compilation generally.


Opinions follow:

If we’re taking votes, mine is to *not* AOT Clojure libraries, 
both for the technical reasons laid out in, and also for the 
social reason of not violating the principle of least surprise.  I 
understand that Guix and Clojure have very different approaches, 
and some balance must be struck.  However, the lack of ABI 
guarantees, the compiler’s behavior, the promise of source 
compatibility, and matching the expectation of the audience these 
tools are meant for all convince me that disabling AOT is the 
right course here.

AOT’ing Clojure applications (which means, more or less, "the 
Clojure tooling") is desirable, and should be maintained.

  — Ian

[1]: 
https://clojureverse.org/t/should-linux-distributions-ship-clojure-byte-compiled-aot-or-not/10595/8
[2]: 
https://clojureverse.org/t/should-linux-distributions-ship-clojure-byte-compiled-aot-or-not/10595/30
[3]: https://clojure.org/reference/compilation
[4]: 
https://archive.clojure.org/design-wiki/display/design/Transitive%2BAOT%2BCompilation.html
[5]: https://clojure.org/guides/deps_and_cli#aot_compilation
[6]: 
https://clojureverse.org/t/should-linux-distributions-ship-clojure-byte-compiled-aot-or-not/10595/30
[7]: 
https://gist.github.com/hiredman/c5710ad9247c6da12a99ff6c26dd442e
[8]: 
https://clojureverse.org/t/should-linux-distributions-ship-clojure-byte-compiled-aot-or-not/10595/4
[9]: 
https://clojureverse.org/t/should-linux-distributions-ship-clojure-byte-compiled-aot-or-not/10595/18
[10]: 
https://ask.clojure.org/index.php/12249/bytecode-not-100-deterministic-given-identical-inputs

Steve George <steve@futurile.net> writes:

> Hi,
>
> Guix's clojure-build-system turns on AOT compilation by 
> default. I would like to advocate that 'as a distributor' we 
> should *not* ship Clojure code AOT'd, so we should change the 
> default.
>
> This has been discussed previously. In #56604 r0man noted that 
> AOT compilation should not be on by default [0], Reilly makes 
> the same point in #53765 [1].
>
> Maxime makes the point that where a compiler is available it 
> should be used [2] and that if it doesn't work it's a bug:
>
>   "if a Clojure library misbehaves when AOT-compiled, without 
>   additional context, that seems like a bug in the Clojure 
>   library to me (or the AOT-compilation code).
>
> The perspective in the Clojure community is quite different from 
> Guix's on a number of fronts. There's not much discussion about 
> offline builds, reproducibility or code coming from 
> Distributions. The internalised perspective is that you use the 
> build tools to download libraries directly from Clojars (a Maven 
> repo) and developers create a final uberjar for production usage 
> Consequently, there is no specific statement saying 
> 'Distributors should not AOT libraries' that I can point 
> to. But, I would like to draw attention to this thread on 
> Clojureverse as the best source I could find:
>
> Alex Miller is the main community manager for Clojure, and is a 
> maintainer of the core libraries, so his perspective is key. He 
> notes that, AOT code is tied to *specific versions of Clojure*:
>
>   "AOT'ed code is that it is inherently the product of a 
>   particular version of tthe Clojure compiler ... I would 
>   recommend NOT AOT compiling libraries" [4]
>
> In the same thread thheller, who is the maintainer of the most 
> popular ClojureScript tooling,  notes you cannot mix AOT and 
> non-AOT libraries [5]:
>
>     "you cannot just ship your library AOT compiles as it would 
>     also contain clojure.core. Clojure AOT current ... can not 
>     load clj files from .class files. So AOT produces the class 
>     files and will fail if one of the dependent classes is 
>     missing although the .clj file is present"
>
> I believe this means that with AOT code on, any user who 
> installs a different version of Clojure from the one that we 
> used to AOT the libraries *may* have problems. And, that we 
> can't have trees where some part is AOT'd but a dependency is 
> not. Finally, there is no expectation in the Clojure community 
> that this is a bug, consequently it will not be 
> fixed. Therefore, we should change to default to AOT off.
>
> What do people think, does this make sense?
>
> Thanks,
>
> Steve / Futurile
>
> [0] https://debbugs.gnu.org/cgi/bugreport.cgi?bug=56604#5
> [1] https://debbugs.gnu.org/cgi/bugreport.cgi?bug=53765#290
> [2] https://debbugs.gnu.org/cgi/bugreport.cgi?bug=53765#293
> [4] 
> https://clojureverse.org/t/deploying-aot-compiled-libraries/2545/6
> [5] 
> https://clojureverse.org/t/deploying-aot-compiled-libraries/2545/3
> [5] 
> https://gist.github.com/hiredman/c5710ad9247c6da12a99ff6c26dd442e

Re: Proposal to turn off AOT in clojure-build-system

[Jean-Pierre De Jesus Diaz]

Hello,

>Correctly handling the ABI concerns — which Guix currently does
>not do — would result in a combinatorial explosion of Clojure
>packages should multiple versions of Clojure ever be available in
>Guix at the same time.

I think this is partly true and also a problem of other languages in
Guix too but it is solvable IMO, one does need indeed to duplicate
each and every package for the specific Clojure version used but
that can be solved by writing a procedure with PACKAGE-MAPPING
that changes `#:clojure' argument recursively for the package inputs,
just like it is done with PACKAGE-WITH-PYTHON2, thus using one line
of code to change a lot of packages.

I don't know much of Clojure to have an opinion on it but just saying
how that problem is and should be solved because it could also affect
packages for a potential new Rust build system since ABI compatibility
is a problem too.