Re: Proposal to turn off AOT in clojure-build-system

[Steve George <steve@futurile.net>]

Hi Maxime,

On 19 Feb, Maxime Devos wrote:
(...) 
> > Consequently, there is no specific statement saying 'Distributors should not AOT libraries' that I can point to.
> 
> In this bit about differences in perspective, I haven’t seen any mention of AOT, hence the “Consequently” does not follow. The part that’s missing here is that (IIUC) in Clojure, it is somewhat conventional to stuff the compiled .class files in a superior Aryan JAR instead – the inferior UnderJARs you get from the “guix install clj-whatever” equivalent would only contain non-compiled .clj (and data files, whatever).
> 
> > But, I would like to draw attention to this thread on Clojureverse as the best source I could find:
> >Alex Miller is the main community manager for Clojure, and is a maintainer of the core libraries, so his perspective is key. He notes that, AOT code is tied to *specific versions of Clojure*:
> >
> >  "AOT'ed code is that it is inherently the product of a particular version of tthe Clojure compiler ... I would recommend NOT AOT compiling libraries" [4]
> 
> This reasoning does not follow – yes, it is tied to the Clojure version, so what? Guix automatically rebuilds dependents when the dependency (in this case, the Clojure compiler) changes.
(...)

I think this preceding sentence is the heart of different assumptions between us.

The Clojure packages we haave are for developers writing applications (libraries and tools). The ecosystem has very few end-user applications [0]. As a Clojure developer I can use the Clojure tools from Guix, and a few libraries. We (and all the other distributions) have a miniscule portion of the Clojure/Java library universe [1]. This leads to the following usage scenarios:

1. A developer installs Clojure from Guix, and uses libraries from outside Guix.
They can install the JVM/Clojure and some common tools (like clj-tools-cli). They will use libraries from elsewhere, including their own. AOT compilation is a problem because of the issue of mixed AOT and non-AOT.

2. A developer installs a Clojure from outside Guix, uses libraries from inside Guix
This will cause problems because the Guix Clojure libraries will have been AOT'd by a different version of the compiler. It's also fairly common to install/use parallel versions of Clojure/jvm due to different deployment needs, this is likely to cause difficult to find bugs.

I can see the sense of compiling to byte code if it's an end-user application. In that case we'd want to make the start-up as fast as possible. Your comments seem to have this use-case in mind.

But, today there aren't any such end-user Clojure applications in Guix.

> >I believe this means that with AOT code on, any user who installs a different version of Clojure from the one that we used to AOT the libraries *may* have problems.
> 
> Unlike, say, Maven, this situation simply does not happen in Guix, because we don’t just download binaries and call it a day (except for some bootstrapping stuff, but that’s not relevant for Clojure AOT), because we have functioning recompilation of dependents, because of shebang patching, because binaries that are to be invoked should not rely on the ambient CLASSPATH / LD_LIBRARY_PATH / etc. and, if, the underlying binaries do rely on that, they are wrapped (see wrap-program) to set them (or, at least, they should be, you might find some bugs in this department if you go looking).
> 
> Even if they aren’t wrapped, then in that case the dependencies are propagated-inputs, and you can only have a single version of a propagated package in the same environment (barring stacking environment shenanigans, but then you are looking for it and/or you can just report a bug about how the binaries should be wrapped/classpath should be patched in/...).
> 

In this paragraph you're assumption is that a Guix user is only using libraries from within Guix. Hopefully, I've shown why this assumption is unlikely above.

You also mentioned Debian, and @e.hashiman [2] said that Clojure libraries are not AOT'd on Debian, while applications are. From what I can find there are 130 packages in Debian with the word Clojure in them [3]. I looked at a selection and it seems true that Debian does not AOT libraries (and I can't find any Clojure 'apps'). For completeness I also checked what Clojars, the main distribution archive for Clojure developers, does:

- https://sources.debian.org/src/core-match-clojure/1.0.0-1/project.clj/
    - Lein-based project - has to have :aot keyword - distributes as source files
    - Clojure source files (.clj) in Debian
    - Clojure source files in Clojars
    - Byte compiled files in GUIX
    - installed and inspected with jar -tvf

- https://sources.debian.org/src/data-csv-clojure/1.0.0-1/deps.edn/
    - tools.deps project - has to have a specific aot alias - distributes as source files
    - Clojure source files (.clj) in the Debian provided jar
    - Clojure source files in the Clojars provided jar
    - Byte compiled files in Guix
    - installed and inspected with jar -tvf

I tried looking at both Arch and Nix but neither of them package any Clojure libraries at all.

Does this help clarify why I'm asking to change the default?

Thanks,

Steve

[0] https://fr.surveymonkey.com/stories/SM-_2BH3b49f_2FXEkUlrb_2BJSThxg_3D_3D/ , there are tools for programmers like clj and babashka
[1] We have 12 Clojure packages, Debian has 180 and according to Maven central (https://mvnrepository.com/repos) there are ~2,000 repos, Clojars shows 3105 projects.
[2] https://mastodon.social/@ehashman@cloudisland.nz/111960714219364737
[3] https://sources.debian.org/search/clojure/
[4] https://www.debian.org/doc/packaging-manuals/java-policy/ch02.html#policy-libraries