Adding the maven-build-system

Adding the maven-build-system

[Julien Lepiller]

Hi Guix!

great news! I have successfully built a hello world package with a
fully bootstrapped maven-build-system!  This email will summarize a bit
what I did and how I got there.  For the impatient, my work is
currently at https://framagit.org/tyreunom/maven-build-channel. You can
build maven-test which only depends on the maven build system, which in
turn depends only on packages in maven-boostrap.scm and
maven-plugins-boostrap.scm.

Before I send a patch series (~200 patches if i count well), I would
like to describe what I want to do and get a bit of feedback.  I feel
that this work is good enough to go to guix, but it might bring it to
an unconfortable state where many packages will need to be converted to
the new build system(s).  I'd like to hear your opinion!



First of all, a bit of background on maven.  It wants to download
dependencies described in the pom.xml file in a local repository.  The
maven-build-system passes the -o (--offline) flag to ensure nothing is
downloaded.  Instead, the build fails immediately if maven determines
that it will need to download something that is not locally present.
Passing the -Dmaven.home flag also ensures we also control the location
of the maven repository when building.

Each maven package has a groupId, an artifactId and a version, and the
repository structure reflects that: from the maven.home directory, a
jar artifact will be located at:
.m2/repository/the/group/id/the-artifact-id/version/the-artifact-id-version.jar

Additionally, maven relies on the presence of a .pom file in that
repository, in the same location:
.m2/repository/the/group/id/the-artifact-id/version/the-artifact-id-version.pom

When building, maven will look at the pom file to get a list of
dependencies (listed in dependencies, and in plugins).  Without the pom
file, some dependencies might be missing while building, running the
tests, etc. However, dependencies in pom files also come with a
version, and maven will refuse to load any other version, even if the
specified version is not present, but a more recent version is present.

The first patch of the series I propose will introduce
https://framagit.org/tyreunom/maven-build-channel/-/blob/master/maven/build/maven/pom.scm,
which contains a parser for the pom.xml as well as a procedure to
override dependency and (optionally) plugin version numbers by versions
present in the guix build environment. This override depends on the
presence of the dependencies in the environment, at a well-known
location (a one-to-one mapping with a maven repository).  This patch
will also introduce two new procedures in
https://framagit.org/tyreunom/maven-build-channel/-/blob/master/maven/build/java-utils.scm:
install-from-pom which applies the transformation to the passed pom,
and installs a jar file (either the one specified in #:jar-name, or the
only jar file present in the build directory) along with the pom file
in a directory structure that reflects maven repository structures:
lib/m2/the/group/id/the-artifact-id/version/the-artifact-id-version.{jar,pom}.

Subsequent patches will move existing package definitions from java.scm
to maven-bootstrap.scm (including junit and some other packages with
more than that purpose), until we have moved maven itself.  During that
move, some packages will be updated (for instance junit) and the junit
bootstrap is completed (I remove the dependency on maven and osgi
completely because we don't build the osgi bundle and maven plugin
properly anyway).  Additional packages will be included (see packages
whose name end in "-pom") to add so-called "parent-poms" referenced in
packages' pom.xml.  I'm not entirely sure if that is required (we could
maybe remove the parent information, but it doesn't seem to be always
the case upstream).  The install-pom-file procedure in java-utils.scm
is tasked with that job: it will install the pom file in the expected
directory structure, in lib/m2.

Since maven itself cannot build anything, it requires some plugins to
do the work (maven is an artifact resolver, but it's not a build tool
by itself).  In order to build the hello-world package, I need at least
these plugins: the maven-resources-plugin (to collect
src/main/java/resources), the maven-compiler-plugin (to actuall build
the packages, it creates .class files in target/build), the
maven-jar-plugin (to create the jar file), the maven-surefire-plugin
(to run the tests using junit) and the maven-install-plugin (to install
the plugin in the final location).  Additionally, the
maven-enforcer-plugin seems to be used by many, so I built it, but it's
probably not required; I'll see if I can omit it for now.

Subsequent patches will add packages in maven-plugins-bootstrap.scm.
These are mostly new packages that will still use the ant-build-system.
They are dependencies of the plugins.  Plugins themeselves are built
there and require a special phase (generate-plugin.xml) to generate yet
another XML file.  In order to recognize its plugins, maven will look
inside the jar file and look for META-INF/maven/plugin.xml.  This file
describes the plugin, its possible options, default values, required
classes for injection, etc.  This is normally produced by the
maven-plugin-plugin which is itself a plugin.  Making it work outside
of maven was too difficult for me, so I decided to ignore it and
re-implement its functionality in guile.  This is the purpose of the
two files
https://framagit.org/tyreunom/maven-build-channel/-/blob/master/maven/build/maven/plugin.scm
and
https://framagit.org/tyreunom/maven-build-channel/-/blob/master/maven/build/maven/java.scm.

java.scm contains a very basic parser for java sources.  Its focus is
to find imports, classes, and top-level constructs in classes,
including annotations.  Annotations are used to indicate default
values, read-only status, required status, injection classes and hints,
etc.  plugin.scm contains a plugin.xml generator that reads the output
of the parser to generate the proper sxml structure for a plugin mojo.
java-utils.scm includes a generate-plugin.xml procedure that uses it to
generate the full plugin.xml and install it in the jar file.
Hopefully, this can be used on the maven-plugin-plugin and other
required plugins we will encounter in the future.

Once the plugins are built, we have all we need to build a maven
package.  The last patch of the series will be adding the
maven-build-system which does the following:

* It searches all its inputs for the lib/m2 directory and creates a
  maven-home/.m2/repository directory from them.
* It overrides the versions of dependencies and plugins in "pom.xml"
  and any other referenced pom in the repository (modules).
* It runs "maven package" with that repository, in offline mode
* It runs "maven test"
* It runs "maven install" to install in that repository
* It collects every file newly installed in that repository and copies
  them to the final location, in $out/lib/.m2
* It removes unreproducible files (because of timestamps) that are not
  needed for our purposes.

It also has the following interesting arguments:

* #:exclude which will instruct the fix-pom-file procedure to remove any
dependency or plugin with some groupid and artifactid (currently
#:maven-plugins-exclude, to be renamed).
* #:maven-plugins a list of maven plugins, default to
  maven-{resources,compiler,jar,install,surefire}-plugin.

Excluding some plugins is useful, because maven will refuse to work if
they are specified but not present.  Plugins such as the
maven-release-plugin are not useful inside the guix build environment.



Sorry for the very long explanation, this is probably worth a blog post
after we merge the changes in guix :)

Opinions? Objections to that plan?

Re: Adding the maven-build-system

[Jack Hill]

On Sat, 4 Apr 2020, Julien Lepiller wrote:

> Hi Guix!
>
> great news! I have successfully built a hello world package with a
> fully bootstrapped maven-build-system!  This email will summarize a bit
> what I did and how I got there.  For the impatient, my work is
> currently at https://framagit.org/tyreunom/maven-build-channel. You can
> build maven-test which only depends on the maven build system, which in
> turn depends only on packages in maven-boostrap.scm and
> maven-plugins-boostrap.scm.
>
> Before I send a patch series (~200 patches if i count well), I would
> like to describe what I want to do and get a bit of feedback.  I feel
> that this work is good enough to go to guix, but it might bring it to
> an unconfortable state where many packages will need to be converted to
> the new build system(s).  I'd like to hear your opinion!

Julien,

I don't have any useful comments at this time, but I wanted to 
congratulate you on this progress. I know that it must have been a lot of 
work. Thank you for the code and the excellent description. I look 
forward to trying the build system soon.

Best,
Jack

Re: Adding the maven-build-system

[Leo Famulari]

On Sat, Apr 04, 2020 at 05:52:37PM +0200, Julien Lepiller wrote:
> Before I send a patch series (~200 patches if i count well),

It will be more convenient for reviewers if you pushed a
wip-maven-build-system branch to Savannah.

Re: Adding the maven-build-system

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 259 bytes --]

Hi Julien,

thanks for this great summary, looking forward to it!

By the way, Leandro was planning to work on Clojars, so having a
functioning maven build system might be helpful to that task.

Cheers!

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Adding the maven-build-system

[Leandro Doctors]

On Sun, 5 Apr 2020 at 06:20, Pierre Neidhardt <mail@ambrevar.xyz> wrote:
> Hi Julien,
> thanks for this great summary, looking forward to it!

Same here, Julien :-)


> By the way, Leandro was planning to work on Clojars, so having a
> functioning maven build system might be helpful to that task.

Thanks for CCing me, Pierre :-)

Yes, assuming I get accepted for GSoC to work to add Clojars support
to Guix, Julien's work will be indeed helpful, as it would help me
evaluating whether The Central Repository (TCR, the new name of Maven
Central) is "Guixable enough" or not.
(That being said, the relative importance of TCR support for my
project would depend on the way I end up achieving Clojure CLI +
deps.edn support...)

As I mention in my proposal, an important aspect of TCR relates with
which types of package metadata it supports (or not). AFAIK, TCR does
not support explicit licensing and source repository metadata, very
important for Guix's values of freedom and reproducibility...

If full TCR support were to be somehow officially added to Guix, my
hypothesis (yet to be proven or refuted :) is that it should involve
mechanisms to deal with those limitations... Of course, all this does
not mean TCR support could not be eventually implemented as a
non-official Guix channel...
http://guix.gnu.org/manual/en/html_node/Channels.html#Channels

Best,
Leandro