From mboxrd@z Thu Jan 1 00:00:00 1970 From: Julien Lepiller Subject: Adding the maven-build-system Date: Sat, 4 Apr 2020 17:52:37 +0200 Message-ID: <20200404175237.0d28e239@tachikoma.lepiller.eu> Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Return-path: Received: from eggs.gnu.org ([2001:470:142:3::10]:49636) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jKl6R-0003MH-0b for guix-devel@gnu.org; Sat, 04 Apr 2020 11:53:01 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1jKl6P-00064q-54 for guix-devel@gnu.org; Sat, 04 Apr 2020 11:52:58 -0400 Received: from lepiller.eu ([2a00:5884:8208::1]:38666) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1jKl6O-0005xb-HI for guix-devel@gnu.org; Sat, 04 Apr 2020 11:52:57 -0400 Received: from lepiller.eu (localhost [127.0.0.1]) by lepiller.eu (OpenSMTPD) with ESMTP id 1eb5dcf0 for ; Sat, 4 Apr 2020 15:52:49 +0000 (UTC) Received: by lepiller.eu (OpenSMTPD) with ESMTPSA id f2591aca (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO) for ; Sat, 4 Apr 2020 15:52:49 +0000 (UTC) List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+gcggd-guix-devel=m.gmane-mx.org@gnu.org Sender: "Guix-devel" To: guix-devel@gnu.org Hi Guix! great news! I have successfully built a hello world package with a fully bootstrapped maven-build-system! This email will summarize a bit what I did and how I got there. For the impatient, my work is currently at https://framagit.org/tyreunom/maven-build-channel. You can build maven-test which only depends on the maven build system, which in turn depends only on packages in maven-boostrap.scm and maven-plugins-boostrap.scm. Before I send a patch series (~200 patches if i count well), I would like to describe what I want to do and get a bit of feedback. I feel that this work is good enough to go to guix, but it might bring it to an unconfortable state where many packages will need to be converted to the new build system(s). I'd like to hear your opinion! First of all, a bit of background on maven. It wants to download dependencies described in the pom.xml file in a local repository. The maven-build-system passes the -o (--offline) flag to ensure nothing is downloaded. Instead, the build fails immediately if maven determines that it will need to download something that is not locally present. Passing the -Dmaven.home flag also ensures we also control the location of the maven repository when building. Each maven package has a groupId, an artifactId and a version, and the repository structure reflects that: from the maven.home directory, a jar artifact will be located at: .m2/repository/the/group/id/the-artifact-id/version/the-artifact-id-version.jar Additionally, maven relies on the presence of a .pom file in that repository, in the same location: .m2/repository/the/group/id/the-artifact-id/version/the-artifact-id-version.pom When building, maven will look at the pom file to get a list of dependencies (listed in dependencies, and in plugins). Without the pom file, some dependencies might be missing while building, running the tests, etc. However, dependencies in pom files also come with a version, and maven will refuse to load any other version, even if the specified version is not present, but a more recent version is present. The first patch of the series I propose will introduce https://framagit.org/tyreunom/maven-build-channel/-/blob/master/maven/build/maven/pom.scm, which contains a parser for the pom.xml as well as a procedure to override dependency and (optionally) plugin version numbers by versions present in the guix build environment. This override depends on the presence of the dependencies in the environment, at a well-known location (a one-to-one mapping with a maven repository). This patch will also introduce two new procedures in https://framagit.org/tyreunom/maven-build-channel/-/blob/master/maven/build/java-utils.scm: install-from-pom which applies the transformation to the passed pom, and installs a jar file (either the one specified in #:jar-name, or the only jar file present in the build directory) along with the pom file in a directory structure that reflects maven repository structures: lib/m2/the/group/id/the-artifact-id/version/the-artifact-id-version.{jar,pom}. Subsequent patches will move existing package definitions from java.scm to maven-bootstrap.scm (including junit and some other packages with more than that purpose), until we have moved maven itself. During that move, some packages will be updated (for instance junit) and the junit bootstrap is completed (I remove the dependency on maven and osgi completely because we don't build the osgi bundle and maven plugin properly anyway). Additional packages will be included (see packages whose name end in "-pom") to add so-called "parent-poms" referenced in packages' pom.xml. I'm not entirely sure if that is required (we could maybe remove the parent information, but it doesn't seem to be always the case upstream). The install-pom-file procedure in java-utils.scm is tasked with that job: it will install the pom file in the expected directory structure, in lib/m2. Since maven itself cannot build anything, it requires some plugins to do the work (maven is an artifact resolver, but it's not a build tool by itself). In order to build the hello-world package, I need at least these plugins: the maven-resources-plugin (to collect src/main/java/resources), the maven-compiler-plugin (to actuall build the packages, it creates .class files in target/build), the maven-jar-plugin (to create the jar file), the maven-surefire-plugin (to run the tests using junit) and the maven-install-plugin (to install the plugin in the final location). Additionally, the maven-enforcer-plugin seems to be used by many, so I built it, but it's probably not required; I'll see if I can omit it for now. Subsequent patches will add packages in maven-plugins-bootstrap.scm. These are mostly new packages that will still use the ant-build-system. They are dependencies of the plugins. Plugins themeselves are built there and require a special phase (generate-plugin.xml) to generate yet another XML file. In order to recognize its plugins, maven will look inside the jar file and look for META-INF/maven/plugin.xml. This file describes the plugin, its possible options, default values, required classes for injection, etc. This is normally produced by the maven-plugin-plugin which is itself a plugin. Making it work outside of maven was too difficult for me, so I decided to ignore it and re-implement its functionality in guile. This is the purpose of the two files https://framagit.org/tyreunom/maven-build-channel/-/blob/master/maven/build/maven/plugin.scm and https://framagit.org/tyreunom/maven-build-channel/-/blob/master/maven/build/maven/java.scm. java.scm contains a very basic parser for java sources. Its focus is to find imports, classes, and top-level constructs in classes, including annotations. Annotations are used to indicate default values, read-only status, required status, injection classes and hints, etc. plugin.scm contains a plugin.xml generator that reads the output of the parser to generate the proper sxml structure for a plugin mojo. java-utils.scm includes a generate-plugin.xml procedure that uses it to generate the full plugin.xml and install it in the jar file. Hopefully, this can be used on the maven-plugin-plugin and other required plugins we will encounter in the future. Once the plugins are built, we have all we need to build a maven package. The last patch of the series will be adding the maven-build-system which does the following: * It searches all its inputs for the lib/m2 directory and creates a maven-home/.m2/repository directory from them. * It overrides the versions of dependencies and plugins in "pom.xml" and any other referenced pom in the repository (modules). * It runs "maven package" with that repository, in offline mode * It runs "maven test" * It runs "maven install" to install in that repository * It collects every file newly installed in that repository and copies them to the final location, in $out/lib/.m2 * It removes unreproducible files (because of timestamps) that are not needed for our purposes. It also has the following interesting arguments: * #:exclude which will instruct the fix-pom-file procedure to remove any dependency or plugin with some groupid and artifactid (currently #:maven-plugins-exclude, to be renamed). * #:maven-plugins a list of maven plugins, default to maven-{resources,compiler,jar,install,surefire}-plugin. Excluding some plugins is useful, because maven will refuse to work if they are specified but not present. Plugins such as the maven-release-plugin are not useful inside the guix build environment. Sorry for the very long explanation, this is probably worth a blog post after we merge the changes in guix :) Opinions? Objections to that plan?