Re: Proposal to turn off AOT in clojure-build-system

[Ian Eure <ian@retrospec.tv>]

Hello,

I’ve been following along with this discussion, as well as a 
discussion on Clojureverse, and thought it might be helpful to 
pull together some threads and design decisions around Clojure’s 
behavior.

Clojure is designed to ship libraries as source artifacts, not 
bytecode ("pretty much all other Clojure libraries ... are all 
source code by design[1]."; "Clojure is ... a source-first 
language[2]"), and the view of the community is that shipping AOT 
artifacts "is an anti-pattern[1]."   Clojure library JARs are more 
akin to source tarballs than binaries.  The original design and 
intent of Clojure’s AOT compiler is to compile "just a few 
things... for the interop case" or "Everything... For the 
'Application delivery', 'Syntax check', and 'reflection warnings' 
cases[3]."

Clojure’s compiler is transitive and "does not support separate 
compilation"[3], meaning when a namespace is compiled, anything it 
uses is compiled and emitted with it.  This is the crux of why 
mixing AOT and non-AOT code is troublesome: it causes dependency 
diamonds, where the AOT’d library contains a duplicate, older 
version of code used elsewhere in the project.

The Clojure reference on compiling[4] gives some reasons you might 
want to AOT: "To deliver your application without source," "To 
speed up application startup," "To generate named classes for use 
by Java," "To create an application that does not need runtime 
bytecode generation and custom classloaders."  Note that there’s 
no mention of compiling libraries for any reason; only 
applications.

When AOT is used "for the interop case," it’s typical to AOT only 
those namespaces[5], not the entire library.

Shipping AOT-compiled Clojure libraries has caused real and very 
weird and hard-to-debug problems in the past:

    https://clojure.atlassian.net/browse/CLJ-1886?focusedCommentId=15290
    https://github.com/clj-commons/byte-streams/issues/68 and 
    https://clojure.atlassian.net/browse/CLJ-1741

Clojure doesn’t have guarantees around ABI stability[6][7].  To 
date, most ABI changes have been additive, but there are no 
guarantees that the ABI will be compatible from any one version of 
Clojure to any other.  The understanding of the Clojure community 
is that the design of the current compiler can’t offer a stable 
ABI[8] at all.  Because nobody in the Clojure community AOTs 
intermediate (that is, library) code, this hasn’t been a problem 
and is unlikely to change.

"Clojure tries very hard to provide source compatibility but not 
bytecode compatibility across versions[9]."

Correctly handling the ABI concerns — which Guix currently does 
not do — would result in a combinatorial explosion of Clojure 
packages should multiple versions of Clojure ever be available in 
Guix at the same time.  For example, if someone wanted to package 
Clojure 1.12.0-alpha9, you’d need to duplicate every package 
taking Clojure as an input so they use the correct version.  While 
ABI breakage has been rare thus far, it seems likely that it’ll 
occur at some point; perhaps if Clojure reaches version 2.0.0.  If 
Guix disables AOT for Clojure libraries, we have source 
compatibility, and the AOT/ABI problems are moot.

Clojure’s compiler is non-deterministic[10]: the same compiler can 
will produce different bytecode for the same input across multiple 
runs.  I’m not sure if this is a problem for Guix at this point in 
time, but it seems out of line with Guix expectations for 
compilation generally.


Opinions follow:

If we’re taking votes, mine is to *not* AOT Clojure libraries, 
both for the technical reasons laid out in, and also for the 
social reason of not violating the principle of least surprise.  I 
understand that Guix and Clojure have very different approaches, 
and some balance must be struck.  However, the lack of ABI 
guarantees, the compiler’s behavior, the promise of source 
compatibility, and matching the expectation of the audience these 
tools are meant for all convince me that disabling AOT is the 
right course here.

AOT’ing Clojure applications (which means, more or less, "the 
Clojure tooling") is desirable, and should be maintained.

  — Ian

[1]: 
https://clojureverse.org/t/should-linux-distributions-ship-clojure-byte-compiled-aot-or-not/10595/8
[2]: 
https://clojureverse.org/t/should-linux-distributions-ship-clojure-byte-compiled-aot-or-not/10595/30
[3]: https://clojure.org/reference/compilation
[4]: 
https://archive.clojure.org/design-wiki/display/design/Transitive%2BAOT%2BCompilation.html
[5]: https://clojure.org/guides/deps_and_cli#aot_compilation
[6]: 
https://clojureverse.org/t/should-linux-distributions-ship-clojure-byte-compiled-aot-or-not/10595/30
[7]: 
https://gist.github.com/hiredman/c5710ad9247c6da12a99ff6c26dd442e
[8]: 
https://clojureverse.org/t/should-linux-distributions-ship-clojure-byte-compiled-aot-or-not/10595/4
[9]: 
https://clojureverse.org/t/should-linux-distributions-ship-clojure-byte-compiled-aot-or-not/10595/18
[10]: 
https://ask.clojure.org/index.php/12249/bytecode-not-100-deterministic-given-identical-inputs

Steve George <steve@futurile.net> writes:

> Hi,
>
> Guix's clojure-build-system turns on AOT compilation by 
> default. I would like to advocate that 'as a distributor' we 
> should *not* ship Clojure code AOT'd, so we should change the 
> default.
>
> This has been discussed previously. In #56604 r0man noted that 
> AOT compilation should not be on by default [0], Reilly makes 
> the same point in #53765 [1].
>
> Maxime makes the point that where a compiler is available it 
> should be used [2] and that if it doesn't work it's a bug:
>
>   "if a Clojure library misbehaves when AOT-compiled, without 
>   additional context, that seems like a bug in the Clojure 
>   library to me (or the AOT-compilation code).
>
> The perspective in the Clojure community is quite different from 
> Guix's on a number of fronts. There's not much discussion about 
> offline builds, reproducibility or code coming from 
> Distributions. The internalised perspective is that you use the 
> build tools to download libraries directly from Clojars (a Maven 
> repo) and developers create a final uberjar for production usage 
> Consequently, there is no specific statement saying 
> 'Distributors should not AOT libraries' that I can point 
> to. But, I would like to draw attention to this thread on 
> Clojureverse as the best source I could find:
>
> Alex Miller is the main community manager for Clojure, and is a 
> maintainer of the core libraries, so his perspective is key. He 
> notes that, AOT code is tied to *specific versions of Clojure*:
>
>   "AOT'ed code is that it is inherently the product of a 
>   particular version of tthe Clojure compiler ... I would 
>   recommend NOT AOT compiling libraries" [4]
>
> In the same thread thheller, who is the maintainer of the most 
> popular ClojureScript tooling,  notes you cannot mix AOT and 
> non-AOT libraries [5]:
>
>     "you cannot just ship your library AOT compiles as it would 
>     also contain clojure.core. Clojure AOT current ... can not 
>     load clj files from .class files. So AOT produces the class 
>     files and will fail if one of the dependent classes is 
>     missing although the .clj file is present"
>
> I believe this means that with AOT code on, any user who 
> installs a different version of Clojure from the one that we 
> used to AOT the libraries *may* have problems. And, that we 
> can't have trees where some part is AOT'd but a dependency is 
> not. Finally, there is no expectation in the Clojure community 
> that this is a bug, consequently it will not be 
> fixed. Therefore, we should change to default to AOT off.
>
> What do people think, does this make sense?
>
> Thanks,
>
> Steve / Futurile
>
> [0] https://debbugs.gnu.org/cgi/bugreport.cgi?bug=56604#5
> [1] https://debbugs.gnu.org/cgi/bugreport.cgi?bug=53765#290
> [2] https://debbugs.gnu.org/cgi/bugreport.cgi?bug=53765#293
> [4] 
> https://clojureverse.org/t/deploying-aot-compiled-libraries/2545/6
> [5] 
> https://clojureverse.org/t/deploying-aot-compiled-libraries/2545/3
> [5] 
> https://gist.github.com/hiredman/c5710ad9247c6da12a99ff6c26dd442e