Guidance for packaging Java programs

Guidance for packaging Java programs

[Bruno Victal]

Hi all,

I'm interested in seeing BiglyBT (an awesome BitTorrent client) [1],
jing (a RELAX NG validator) [2], Saxon-HE (a free implementation of
XSLT 2.0 + 3.0) [3] and Mindustry (an addicting tower-defense
game) [4] packaged for Guix and I'd like to ask for some guidance
with this subject.

Though I'm not familiar with the language, this seldomly reveals to
be a problem if the project uses GNU Autotools, Meson, CMake, etc.
but I'm completely lost when it comes to Java. It's hard to understand
what's happening, what are the dependencies, how the whole thing is
assembled together or if I'm even staring at something supposed to be
built at all. The lack of importers doesn't help either :(

The impression I get is that overall the Guix Java subsystem/support
seems to be quite barren which I suspect is in part due to the
rather arcane build process involved. (the manual doesn't have much
information on the topic as well)

Does anyone have any insight in what would it take to get the
aforementioned applications properly packaged in Guix? (i.e. without
repackaging the binary from upstream)
I'd also be grateful if someone could point me to a primer on
“how to go from a (java) source checkout into a built app that can
be used”.


[1]: <https://github.com/BiglySoftware/BiglyBT>
[2]: <https://github.com/relaxng/jing-trang>
[3]: <https://github.com/Saxonica/Saxon-HE/>
[4]: <https://mindustrygame.github.io/>

-- 
Furthermore, I consider that nonfree software must be eradicated.

Cheers,
Bruno.

Re: Guidance for packaging Java programs

[Julien Lepiller]

Hi Bruno,

I'm glad to see some interest in Java :)

First, Java has its own build systems, somewhat like autotools/cmake/meson etc. In the past, ant was the most popular. It's somewhat similar to Make, as you specify build targets in a build.xml file. This case is well supported and we have a ant-build-system for that.

The first difficulty is that ant is-just like make, so it won't manage dependencies, and even not specify any. In the Java world, they either use the same solution as C, in that they don't care and it's your job to bring in dependencies until the build stops breaking (although it might sound bad, this is the good way of doing things for us), or they ship the binary of their dependencies and force you to use that. This is bad for us, since we want to build everything fsom source. It's just a bit more work, but nothing we can patch or snippet away usually.

Nowadays, ant is loosing "market share" in favor of Maven and Gradle. We have a working though probably too primitive maven-build-system which can build a maven project as long as you provide dependencies and plugins.

They are both also package managers, so they usually specify dependencies, which could be useful for an importer, but they often lack source information and there's a lot of bloat in these files (many unneeded dependencies stay there forever…).

Gradle is similar to Maven but it requires quite a lot of work still. The main issue is that it requires Kotlin, the programming language. Since Guix tries to bootstrap its compilers, we need a bootstrap of Kotlin. I tried to do that and after a lot of pain, managed to cleanly bootstrap a kotlin-1.0. I still need to work on adding more to the chain and get more recent versions, but that's pure crazyness at this point.

BiglyBT doesn't seem to have too mazy dependencies at first glance, maybe a better target, though it still requires some cleanup. Not sure what swt is, but we would need to build it ourselves.

Jing looks like it can use ant, but it also looks like it has a ton of lependencies. You could get lost in a rabbithole packaging that one :)

I've never seen anything like saxon-he sources. It looks like a bunch of zip files that contain java files with no build system. The ant-build-system might be able to build that, though no idea how much dependencies there could be.

I'd also like to get Mindustry at some point, but I don't see that happening any time soon : it uses gradle, and with custom plugins too, which means we can't even cheat and use something other than gradle :/

If you feel like helping, maybe an importer would be a good first step :). You should be able to get info from Maven Central (get the pom files, they are XML files and we have a module somewhere to handle them (guix build maven pom) I think). It should work for maven and gradle packages at least. Mapping maven name to guix names might also be challenging, but we can solve with an upstream-name property.

Here are some more resources if you want to learn more about Guix and Java:

I gave a talk about bootstrapping Maven in 2020:

https://xana.lepiller.eu/guix-days-2020/guix-days-2020-julien-lepiller-bootstrapping-maven-480.mp4

There's also my Kotlin bootstrap chain:

https://framagit.org/tyreunom/guix-android/-/blob/master/android/packages/kotlin.scm

There's also a bootstrap of sbt (Scala Build Tool) from Scala, but Scala is not bootstrapped/able in that repo.

To summarize, the main pain points are: lots of dependencies that can quickly go out of hand, source buildability is not great, as many ship their own versions of their dependencies (we need to snippet pre-built jars away, and find a workaround when the build fails to find them), and bootstrappability, as there can be quite a lot of dependency circles.

HTH, and don't hesitate if you want to learn more!

Le 13 juillet 2023 19:00:07 GMT+02:00, Bruno Victal <mirai@makinata.eu> a écrit :
>Hi all,
>
>I'm interested in seeing BiglyBT (an awesome BitTorrent client) [1],
>jing (a RELAX NG validator) [2], Saxon-HE (a free implementation of
>XSLT 2.0 + 3.0) [3] and Mindustry (an addicting tower-defense
>game) [4] packaged for Guix and I'd like to ask for some guidance
>with this subject.
>
>Though I'm not familiar with the language, this seldomly reveals to
>be a problem if the project uses GNU Autotools, Meson, CMake, etc.
>but I'm completely lost when it comes to Java. It's hard to understand
>what's happening, what are the dependencies, how the whole thing is
>assembled together or if I'm even staring at something supposed to be
>built at all. The lack of importers doesn't help either :(
>
>The impression I get is that overall the Guix Java subsystem/support
>seems to be quite barren which I suspect is in part due to the
>rather arcane build process involved. (the manual doesn't have much
>information on the topic as well)
>
>Does anyone have any insight in what would it take to get the
>aforementioned applications properly packaged in Guix? (i.e. without
>repackaging the binary from upstream)
>I'd also be grateful if someone could point me to a primer on
>“how to go from a (java) source checkout into a built app that can
>be used”.
>
>
>[1]: <https://github.com/BiglySoftware/BiglyBT>
>[2]: <https://github.com/relaxng/jing-trang>
>[3]: <https://github.com/Saxonica/Saxon-HE/>
>[4]: <https://mindustrygame.github.io/>
>

Re: Guidance for packaging Java programs

[Bruno Victal]

Hi Julien,

On 2023-07-13 18:48, Julien Lepiller wrote:
> I've never seen anything like saxon-he sources. It looks like a bunch of zip files that contain java files with no build system. The ant-build-system might be able to build that, though no idea how much dependencies there could be.

It looks like the Saxon-HE sources are actually “generated” from this repository:
<https://saxonica.plan.io/projects/saxonmirrorhe/repository>

> If you feel like helping, maybe an importer would be a good first step :). You should be able to get info from Maven Central (get the pom files, they are XML files and we have a module somewhere to handle them (guix build maven pom) I think). It should work for maven and gradle packages at least. Mapping maven name to guix names might also be challenging, but we can solve with an upstream-name property.[…]

> To summarize, the main pain points are: lots of dependencies that can quickly go out of hand, source buildability is not great, as many ship their own versions of their dependencies (we need to snippet pre-built jars away, and find a workaround when the build fails to find them), and bootstrappability, as there can be quite a lot of dependency circles.

I wonder if this is doable for the java uninitiated.
(Why bother with Java when you can write in Guile? :D)
An importer first would certainly be a step in the right direction, yes.
(handcrafting package definitions for dependencies and dependencies of
said dependencies is no fun)


-- 
Furthermore, I consider that nonfree software must be eradicated.

Cheers,
Bruno.

Re: Guidance for packaging Java programs

[Julien Lepiller]

Le 13 juillet 2023 21:35:37 GMT+02:00, Bruno Victal <mirai@makinata.eu> a écrit :
>Hi Julien,
>
>On 2023-07-13 18:48, Julien Lepiller wrote:
>> I've never seen anything like saxon-he sources. It looks like a bunch of zip files that contain java files with no build system. The ant-build-system might be able to build that, though no idea how much dependencies there could be.
>
>It looks like the Saxon-HE sources are actually “generated” from this repository:
><https://saxonica.plan.io/projects/saxonmirrorhe/repository>

I see. We have two problems it seems: it uses gradle, for which there's no build system, and it also requires an older version of itself for some preprocessing. Might get tricky…

>
>> If you feel like helping, maybe an importer would be a good first step :). You should be able to get info from Maven Central (get the pom files, they are XML files and we have a module somewhere to handle them (guix build maven pom) I think). It should work for maven and gradle packages at least. Mapping maven name to guix names might also be challenging, but we can solve with an upstream-name property.[…]
>
>> To summarize, the main pain points are: lots of dependencies that can quickly go out of hand, source buildability is not great, as many ship their own versions of their dependencies (we need to snippet pre-built jars away, and find a workaround when the build fails to find them), and bootstrappability, as there can be quite a lot of dependency circles.
>
>I wonder if this is doable for the java uninitiated.
>(Why bother with Java when you can write in Guile? :D)
>An importer first would certainly be a step in the right direction, yes.
>(handcrafting package definitions for dependencies and dependencies of
>said dependencies is no fun)

Well, I don't write a lot of Java either :)

Re: Guidance for packaging Java programs

[Dr. Arne Babenhauserheide]

[-- Attachment #1: Type: text/plain, Size: 825 bytes --]


Julien Lepiller <julien@lepiller.eu> writes:

> Le 13 juillet 2023 21:35:37 GMT+02:00, Bruno Victal <mirai@makinata.eu> a écrit :
>>> If you feel like helping, maybe an importer would be a good first
>>> step :). You should be able to get info from Maven Central (get the
>>> pom files, they are XML files and we have a module somewhere to
>>> handle them (guix build maven pom) I think). It should work for
>>> maven and gradle packages at least. Mapping maven name to guix
>>> names might also be challenging, but we can solve with an
>>> upstream-name property.[…]

An importer for maven-packages would be great!

I have a project where I gave up after the third dependency needed more
dependencies.

Best wishes,
Arne
-- 
Unpolitisch sein
heißt politisch sein,
ohne es zu merken.
draketo.de

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 1125 bytes --]

Re: Guidance for packaging Java programs

[Bruno Victal]

Hi Julien,

On 2023-07-13 18:48, Julien Lepiller wrote:
> If you feel like helping, maybe an importer would be a good first step :). You should be able to get info from Maven Central (get the pom files, they are XML files and we have a module somewhere to handle them (guix build maven pom) I think). It should work for maven and gradle packages at least. Mapping maven name to guix names might also be challenging, but we can solve with an upstream-name property.
> 
> Here are some more resources if you want to learn more about Guix and Java:
> 
> I gave a talk about bootstrapping Maven in 2020:
> 
> https://xana.lepiller.eu/guix-days-2020/guix-days-2020-julien-lepiller-bootstrapping-maven-480.mp4

Thanks! This was very helpful in providing an overview of the system.

Can we say that the maven in Guix has already been “bootstrapped” or is there something still missing?
Given the large and non-trivial nature of just the “bootstrap” part, would it make sense to split
gnu/packages/maven.scm into maven-bootstrap.scm and maven-<anything else>.scm files to keep things manageable?

The maven-importer situation is quite unfortunate due to the often absent source location, a quick glance
here suggests that the best we might hope for is an automation-½ approach: generate “skeleton” package
definitions and manually fill-in the source & co.

Some additional maven related questions:

* What is 'sisu.sh' & 'components.sh' and when/what are they used for?

* Is it common for maven plugins to end up with package definitions
that make heavy use of inherit? (a quick glance at gnu/packages/maven.scm
gives the impression that it all looks very fragile due to the matryoshka-like
inheritance rabbit-holes)


-- 
Furthermore, I consider that nonfree software must be eradicated.

Cheers,
Bruno.

Re: Guidance for packaging Java programs

[Julien Lepiller]

Le 28 juillet 2023 04:09:42 GMT+02:00, Bruno Victal <mirai@makinata.eu> a écrit :
>Hi Julien,
>
>On 2023-07-13 18:48, Julien Lepiller wrote:
>> If you feel like helping, maybe an importer would be a good first step :). You should be able to get info from Maven Central (get the pom files, they are XML files and we have a module somewhere to handle them (guix build maven pom) I think). It should work for maven and gradle packages at least. Mapping maven name to guix names might also be challenging, but we can solve with an upstream-name property.
>> 
>> Here are some more resources if you want to learn more about Guix and Java:
>> 
>> I gave a talk about bootstrapping Maven in 2020:
>> 
>> https://xana.lepiller.eu/guix-days-2020/guix-days-2020-julien-lepiller-bootstrapping-maven-480.mp4
>
>Thanks! This was very helpful in providing an overview of the system.
>
>Can we say that the maven in Guix has already been “bootstrapped” or is there something still missing?
>Given the large and non-trivial nature of just the “bootstrap” part, would it make sense to split
>gnu/packages/maven.scm into maven-bootstrap.scm and maven-<anything else>.scm files to keep things manageable?

Maven itself is bootstrapped, along with the essential plugins. We're still missing many plugins. As long as we don't have maven-plugin-plugin, we need to build them "manually" in the same way we build other other plugins. It could be an interesting target for improving the situation.

>
>The maven-importer situation is quite unfortunate due to the often absent source location, a quick glance
>here suggests that the best we might hope for is an automation-½ approach: generate “skeleton” package
>definitions and manually fill-in the source & co.
>
>Some additional maven related questions:
>
>* What is 'sisu.sh' & 'components.sh' and when/what are they used for?

They are scripts to automatically generate some files that are required at runtime for "dependency injection". Normally, this generation is done by a maven plugin, but in the bootstrap, we can't use maven, so we managed to replace that with a small script.

>
>* Is it common for maven plugins to end up with package definitions
>that make heavy use of inherit? (a quick glance at gnu/packages/maven.scm
>gives the impression that it all looks very fragile due to the matryoshka-like
>inheritance rabbit-holes)

There's inheritance everywhere in java and maven packages. There shouldn't be that many layers though. Usually, there's a package definition and its bootstrap version or other components from the same source inherit from it.

If you find a way to simplify all that, it's more than welcome :)