RE: Proposal to turn off AOT in clojure-build-system

[Maxime Devos <maximedevos@telenet.be>]

[-- Attachment #1: Type: text/plain, Size: 7677 bytes --]

((As I said about a month ago, in theory I should have access to a proper e-mail program again a week or two in the past.))

>Hi,

>Guix's clojure-build-system turns on AOT compilation by default. I would like to advocate that 'as a distributor' we should *not* ship Clojure code AOT'd, so we should change the default.

>This has been discussed previously. In #56604 r0man noted that AOT compilation should not be on by default [0], Reilly makes the same point in #53765 [1].

>Maxime makes the point that where a compiler is available it should be used [2] and that if it doesn't work it's a bug:

>>  "if a Clojure library misbehaves when AOT-compiled, without additional context, that seems like a bug in the Clojure library to me (or the AOT-compilation code).

>The perspective in the Clojure community is quite different from Guix's on a number of fronts. There's not much discussion about offline builds

This (not the has nothing to do with AOT.

> reproducibility

If some libraries have irreproducible binaries, then just don’t AOT those libraries (and, if the irreproducibility is in macros (and maybe inlinable generated code, if that’s something the AOT does), the relevant dependents of those libraries as well).

An AOT problem for a few libraries is not a reason to turn off AOT by default, it is a reason to turn off AOT for those few libraries in particular.

> or code coming from Distributions. 

You are contradicting this by the next paragraph, in which you mention the distribution “Clojars”. A rather specialised and lacking distribution, but a distribution nonetheless.

>The internalised perspective is that you use the build tools to download libraries directly from Clojars (a Maven repo)

That seems quite similar to Guix, where you use the build tools “guix build” / “guix install” / ... to download libraries (by default, when available) directly from ci.guix.gnu.org (I forgot the terminology, but you could compare it to Clojars I guess).

> and developers create a final uberjar for production usage

Except for the delusion of grandeur (AFAIK, in non-Java English, über only appears in the word Übermensch) (maybe whoever coined the term didn’t really mean it but eergh), this is also familiar, see “guix system vm” for large-scale things, and the command for creating relocatable packs (I forgot the exact name and command) on a smaller scale.

> Consequently, there is no specific statement saying 'Distributors should not AOT libraries' that I can point to.

In this bit about differences in perspective, I haven’t seen any mention of AOT, hence the “Consequently” does not follow. The part that’s missing here is that (IIUC) in Clojure, it is somewhat conventional to stuff the compiled .class files in a superior Aryan JAR instead – the inferior UnderJARs you get from the “guix install clj-whatever” equivalent would only contain non-compiled .clj (and data files, whatever).

> But, I would like to draw attention to this thread on Clojureverse as the best source I could find:
>Alex Miller is the main community manager for Clojure, and is a maintainer of the core libraries, so his perspective is key. He notes that, AOT code is tied to *specific versions of Clojure*:
>
>  "AOT'ed code is that it is inherently the product of a particular version of tthe Clojure compiler ... I would recommend NOT AOT compiling libraries" [4]

This reasoning does not follow – yes, it is tied to the Clojure version, so what? Guix automatically rebuilds dependents when the dependency (in this case, the Clojure compiler) changes.

I guess the maintainer has something like Maven etc. in mind, where many maintainers, each with different distribution, Java version, etc. upload binaries, but that’s not the case in Guix. (To a lesser extent, this is not the case in, say, Debian as well.)

>In the same thread thheller, who is the maintainer of the most popular ClojureScript tooling,  notes you cannot mix AOT and non-AOT libraries [5]:

    "you cannot just ship your library AOT compiles as it would also contain clojure.core. Clojure AOT current ... can not load clj files from .class files. So AOT produces the class files and will fail if one of the dependent classes is missing although the .clj file is present"

This argument makes no sense at all. Of course it can’t load CLJ files from .class files – Clojure AOT is a compiler, not a decompiler, cf. how GCC can’t ‘load’ C files from .o / ELF / ..., and why would it in the first place?

And of course you can’t run or compile an application if a part of the binaries / a part of the source code is missing. This is the same as with C & GCC, (Guile) Scheme & “guild compile”, etc..  Yet, GCC and Guile somehow have a functioning AOT (if we count Guile’s bytecode as AOT, which we can because we are counting .class files as “compiled”).

(As a side-remark, I think a typo was made here:  dependent classes -> dependency classes (i.e. classes of dependencies).)

Everything after the first sentence appears to be true, but the conclusion of the first sentence does not follow.

>I believe this means that with AOT code on, any user who installs a different version of Clojure from the one that we used to AOT the libraries *may* have problems.

Unlike, say, Maven, this situation simply does not happen in Guix, because we don’t just download binaries and call it a day (except for some bootstrapping stuff, but that’s not relevant for Clojure AOT), because we have functioning recompilation of dependents, because of shebang patching, because binaries that are to be invoked should not rely on the ambient CLASSPATH / LD_LIBRARY_PATH / etc. and, if, the underlying binaries do rely on that, they are wrapped (see wrap-program) to set them (or, at least, they should be, you might find some bugs in this department if you go looking).

Even if they aren’t wrapped, then in that case the dependencies are propagated-inputs, and you can only have a single version of a propagated package in the same environment (barring stacking environment shenanigans, but then you are looking for it and/or you can just report a bug about how the binaries should be wrapped/classpath should be patched in/...).

And, that we can't have trees where some part is AOT'd but a dependency is not. Finally, there is no expectation in the Clojure community that this is a bug, consequently it will not be fixed. Therefore, we should change to default to AOT off.

>What do people think, does this make sense?

No.

Here is an argument for disabling AOT that I think makes sense:

>The main problem is that AOT will compile your code __and all its dependencies__. This redundancy is wasteful, albeit otherwise harmless.

(adapted from https://clojureverse.org/t/deploying-aot-compiled-libraries/2545/3, but with a different conclusion.)

Going by https://clojureverse.org/t/deploying-aot-compiled-libraries/2545/4 and its response, this appears to be a surmountable problem however.

Presumably there is a way to select _which_ code is compiled (instead of compiling the code and its dependencies) – IIRC, this is done already by clojure-build-system?

>[0] https://debbugs.gnu.org/cgi/bugreport.cgi?bug=56604#5
>[1] https://debbugs.gnu.org/cgi/bugreport.cgi?bug=53765#290
>[2] https://debbugs.gnu.org/cgi/bugreport.cgi?bug=53765#293
>[4] https://clojureverse.org/t/deploying-aot-compiled-libraries/2545/6
>[5] https://clojureverse.org/t/deploying-aot-compiled-libraries/2545/3
>[5] https://gist.github.com/hiredman/c5710ad9247c6da12a99ff6c26dd442e


[-- Attachment #2: Type: text/html, Size: 14407 bytes --]