Re: Proposal to turn off AOT in clojure-build-system

[Steve George <steve@futurile.net>]

Hi,

I don't think I'm making any progress convincing you, and I'm not enjoying the interaction so I'm going to take a few days off from this thread.

I've tried to ask the Clojure community for a definitive expression of what they think Linux distributions should do with byte-compiled libraries. We'll see if there's a response, this is a public forum so feel free to chime in and express clearly your perpective:

  https://clojureverse.org/t/should-linux-distributions-ship-clojure-byte-compiled-aot-or-not/10595/1

Thanks,

Steve / Futurile

On 22 Feb, Maxime Devos wrote:
> >> [...] 
> >> > But, I would like to draw attention to this thread on Clojureverse as the best source I could find:
> >> >Alex Miller is the main community manager for Clojure, and is a maintainer of the core libraries, so his perspective is key. He notes that, AOT code is tied to *specific versions of Clojure*:
> >> >
> >> >  "AOT'ed code is that it is inherently the product of a particular version of tthe Clojure compiler ... I would recommend NOT AOT compiling libraries" [4]
> >> 
> >> This reasoning does not follow – yes, it is tied to the Clojure version, so what? Guix automatically rebuilds dependents when the dependency (in this case, the Clojure compiler) changes.
> (...)
> 
> >I think this preceding sentence is the heart of different assumptions between us.
> 
> >The Clojure packages we haave are for developers writing applications (libraries and tools). The ecosystem has very few end-user applications [0]. As a Clojure developer I can use the Clojure tools from Guix, and a few libraries. We (and all the other distributions) have a miniscule portion of the Clojure/Java library universe [1]. This leads to the following usage scenarios:
> 
> >1. A developer installs Clojure from Guix, and uses libraries from outside Guix.
> They can install the JVM/Clojure and some common tools (like clj-tools-cli). They will use libraries from elsewhere, including their own. AOT compilation is a problem because of the issue of mixed AOT and non-AOT.
> 
> >2. A developer installs a Clojure from outside Guix, uses libraries from inside Guix
> This will cause problems because the Guix Clojure libraries will have been AOT'd by a different version of the compiler. It's also fairly common to install/use parallel versions of Clojure/jvm due to different deployment needs, this is likely to cause difficult to find bugs.
> 
> My answer to (1) and (2) is:
> 
> (a) About “install Clojure from outside Guix + use libraries inside Guix”:
> 
>  If these libraries are AOT:
> 
> Don’t do that, then. If you mix-and-match binaries (in this case, .class files) different distributions, you are free to do so, but when (not if, when) things break, you get to keep the pieces.
> 
> If these libraries are just the .clj files (not AOT) (which as I understand it is the standard situation): doesn’t seem a problem to me (see point (b)).
> 
> Adding source from other places is one thing (probably ok), adding binaries is another (probably not ok, especially if unstable ABIs (see Clojure compiler) and mismatched versions (see hypotheses of 2.) are involved.
> 
> (b) What is this “the issue of mixed AOT and non-AOT”? Do you have a source on this? Besides Clojure supposedly, I haven’t ever heard of such problems for any language – for example, there is no such issue with Guile and AFAIK not for Python. I haven’t heard of any such issues for the Common Lisp implementations either (though I haven’t checked), so this doesn’t seem like a “Clojure doesn’t do hygienic macros” issue.
> 
> (c) Guix isn’t forcing anyone to use AOT’d libraries. If people really want to assist in murdering the climate (or its a situation where total cost of non-AOT is lower than AOT), they can unfortunately (*) simply do so applying a recursive transformation that adds #:aot? #false flags everywhere or whatever the exact argument is (**).
> 
> Given that this transformation has some legitimate use cases, this transformation could even be a pre-canned procedure + transformation included in Guix itself
> 
> (*) ‘unfortunately’ only applies to the first case. For the case in parentheses, replace by ‘fortunately’.
> (**) IIRC is wasn’t #:aot? but some other name, but that’s not really the point here.
> 
> (d) If a deployment need multiple versions of Clojure, then just fix your deployment to make everything work with the latest version of Clojure. Or, if you for some reason don’t do that, just use a (recursive) transformation to change the version of the Clojure compiler used to match the Clojure that will be used in the particular deployment.
> 
> (e) You can simply add missing packages to Guix as the need arises.
> 
> >I can see the sense of compiling to byte code if it's an end-user application. In that case we'd want to make the start-up as fast as possible. Your comments seem to have this use-case in mind.
> 
> >But, today there aren't any such end-user Clojure applications in Guix.
> 
> That’s a shame. Hopefully that will change some day.
> Also, this is false, “clojure-tools” exists – developers are people too (I don’t care about the “_end_-user” distinction – surely developers want fast start-up as well).
> 
> Also, I _don’t_ have that use case in mind – I have efficiency in general in mind, efficiency of the start-up is only a part of that.
> 
> The point is: most likely you will want the application to have AOT code, and that library has dependencies. Furthermore, likely many of these dependencies are dependencies of other applications as well.
> 
> Instead of redoing the compilation of N shared dependencies for each M applications  (total work: ~N*M), by not participating in these Clojure conventions, the total work of compiling dependencies can be reduced to ~N).
> 
> Let’s avoid joining the madness that is Go’s build system and Cargo(*).
> 
> (*) Rust is fine for compilation, it’s cargo that is the problem (at least, Cargo as used by Guix) – antioxidant (https://notabug.org/maximed/cargoless-rust-experiments) avoids the sheer inefficiency of Cargo.
> 
> >> >I believe this means that with AOT code on, any user who installs a different version of Clojure from the one that we used to AOT the libraries *may* have problems.
> >> 
> >> Unlike, say, Maven, this situation simply does not happen in Guix, because we don’t just download binaries and call it a day (except for some bootstrapping stuff, but that’s not relevant for Clojure AOT), because we have functioning recompilation of dependents, because of shebang patching, because binaries that are to be invoked should not rely on the ambient CLASSPATH / LD_LIBRARY_PATH / etc. and, if, the underlying binaries do rely on that, they are wrapped (see wrap-program) to set them (or, at least, they should be, you might find some bugs in this department if you go looking).
> >> 
> >> Even if they aren’t wrapped, then in that case the dependencies are propagated-inputs, and you can only have a single version of a propagated package in the same environment (barring stacking environment shenanigans, but then you are looking for it and/or you can just report a bug about how the binaries should be wrapped/classpath should be patched in/...).
> > 
> 
> >In this paragraph you're assumption is that a Guix user is only using libraries from within Guix. Hopefully, I've shown why this assumption is unlikely above.
> 
> You haven’t. You have shown that other situations exist, not that this situation is unlikely. To demonstrate likelihood / unlikelyhood, you need some statistics (or a very thorough argument on why people wouldn’t just make a package for missing dependencies – it’s very simple, especially if you aren’t sending them for inclusion.)
> 
> >You also mentioned Debian, and @e.hashiman [2] said that Clojure libraries are not AOT'd on Debian, while applications are. From what I can find there are 130 packages in Debian with the word Clojure in them [3]. I looked at a selection and it seems true that Debian does not AOT libraries (and I can't find any Clojure 'apps'). For completeness I also checked what Clojars, the main distribution archive for Clojure developers, does:
> > [...]
> 
> OK ... and? Other people doing things differently doesn’t mean those different approaches are better. If Debian mentions somewhere the reason _why_ it doesn’t AOT libraries, that reason can be investigated and perhaps carried over to Guix, but it’s the ‘why’ that’s important, not the ‘it doesn’t’.
> 
> > Does this help clarify why I'm asking to change the default?
> 
> Yes. It appears you are unfamiliar with the “transformations” functionality of Guix, which allows for easy rewriting of packages to whatever Clojure version you want, eliminating compatibility concerns or even turning off AOT completely.
> 
> (This requires the relevant versions of Clojure to be packaged in Guix, but that shouldn’t be too much of a problem.)
> 
> It also appears you are unfamiliar with Guix importers, which easily make package definitions (somewhat low quality w.r.t. descriptions, with lack of verification of freeness, etc., but usually otherwise fully functional). I don’t know if an importer for Clojure exists but it shouldn’t be too difficult I think.
> 
> Best regards,
> Maxime Devos.