unofficial mirror of guix-devel@gnu.org 
 help / color / mirror / code / Atom feed
From: Julien Lepiller <julien@lepiller.eu>
To: Hartmut Goebel <h.goebel@crazy-compilers.com>,
	guix-devel <guix-devel@gnu.org>
Subject: Re: maven build system - new insights
Date: Thu, 31 Jan 2019 18:06:37 +0100	[thread overview]
Message-ID: <71F627FD-4E2E-4E8D-B73E-433368CE3B6B@lepiller.eu> (raw)
In-Reply-To: <f9e3108a-50be-6254-fd28-ef7d764cfebc@crazy-compilers.com>

Le 31 janvier 2019 11:00:42 GMT+01:00, Hartmut Goebel <h.goebel@crazy-compilers.com> a écrit :
>H Julien,
>
>as promised I've spend some time on a maven build system (as always
>when
>regarding maven it was much more time than expected). I did not yet
>create a build-system but want to share my insights so we can decide
>how
>to proceed further. If I'm wrong on some points, please feel free to
>correct this. Everything below is based on my investigations the last
>days.
>
>TL;DR: Let's use XMvn [11] for our maven build-system. What do you
>think?
>
>  * The only place where maven actually loads artifacts from is the
>    local repository, which is a single directory (default:
>    $HOME/.m2/repository). If an artifact is not found in the local
>    repository, it is downloaded from some "remote" repositories and
>    cached in the local repository. I have not found any way to change
>this behavior. See Analysis of repositories behavior below for details.
>  * The local repository path can be changed by passing
>    "-Dmaven.repo.local=..." to maven.
>  * The files in the repository need to be structured like this:
>o
>.../org/apache/maven/maven-plugin-descriptor/2.2.1/maven-plugin-descriptor-2.2.1.jar
>
>This implies a maven-build-system would need either to merge all
>artifacts into a "local repository" or to set up a lot of "mirrors"
>(one
>for each input, this will copy all artifacts into the "local
>repository"). Both is complicated.
>
>Thus I suggest to leverage Xmvn [11] for building (see Distro-Analysis
>below for details). Xmvn searches for configurations (and thus
>repository definitions) in e.g $XDG_DATA_DIRS plus it requires only ca.
>32 jar-files (compared to ca. 48 for maven). Thus we could not only
>solve the repository issue but also (hopefully) reduce the number of
>packages we need to bootstrap.
>
>*Distro-Analysis*
>
>Here is what other GNU/Linux distributions do:
>
>  * Debian stores the files in a system repository structured like
>    above, see [9]. (Thought this seem not to be done for all packages,
>    |e.g.|||[6] and [7] do not include ||these maven-repo files.)
>  * Debian provides a script "mvn-debian" [8] which seem to tweak maven
>    to use this system repository (by passing
>    -Dmaven.repo.local=$MAVEN_REPO").
>      o This is still a single directory. For guix this would require
>        all inputs to be merged into a single directory :-(
>  * Debian also has some build-helper "mh_make" [10] which does some
>    more magic I did not investigate by running some Debian-specific
>    java classes for collecting dependencies.
>
>  * Fedora uses xmvn [11] which "extends" maven to maintain a
>    system-wide repository of artifacts.
>  * There is also a "/Dependency-reduced version of XMvn" /[12],
>    incapable of using remote repositories. This could be helpful for
>    bootstrapping.
>      o A sort analysis says: 32 instead of 48 jar-files are required.
>        Thus we might be able to eliminate 1/3 of the dependence for
>        bootstrapping maven.
>  * The directory structure does not follow the repo-structure, but
>    lloks like this:
>      o /usr/share/java/maven-plugin-tools/maven-plugin-plugin.jar
>   /usr/share/maven-metadata/maven-plugin-tools-maven-plugin-plugin.xml
>       /usr/share/maven-poms/maven-plugin-tools/maven-plugin-plugin.pom
>  *
>
>  * xmvn searches for config-files, which are merged in
>      o $PWD - which would allow us to create one on the fly based on
>        all inputs
>      o $XDG_CONFIG_DIRS and $XDG_DATA_DIRS - which would allow to use
>        native search-path specification
>
>*Analysis of repositories behavior:*
>
>  * The local repository can consist of a single directory only, I did
>   not find any hint about how to specify several directories. Esp. the
>   "Configuring Maven Guide" [1] and [5] state a single directory only.
>    This makes sense somehow since the local repo is where downloaded
>    files are cached.
>  * "Remote" repositories
>      o can be specified in settings.xml (and require to pass a profile
>        name when running, e.g. mvn -Pmyprofile) or the pom.xml. Maybe
>        also in .mvn/{settings,extension).xml, I did not test this.
>      o repositories are processed *after* the central repository
>        (source: [2])
>      o Even if the repository is specified with file://... the
>        artifacts will be copied to the local repo (source: [2], also
>        observed). [3] does not show any option to suppress download.
>      o When using "--offline", all remote repositories are disabled,
>        even if specified with file://...
>  * Mirrors do not help
>      o Can be specified in settings.xml without requiring a profile
>      o Even if the mirror is specified with file://... the artifacts
>     will be copied to the local repo (observed, and somehow logically)
>  * Some more background about repositories can be found at [4].
>  * I also strace'd and ltrace'd several runs of maven to investigate
> whether there any other env-vars or directories are accessed. Nothing.
>
>[1] https://maven.apache.org/guides/mini/guide-configuring-maven.html
>[2] https://www.tutorialspoint.com/maven/maven_repositories.htm
>[3]
>https://maven.apache.org/ref/3.6.0/maven-settings/settings.html#class_releases
>[4]
>https://blog.packagecloud.io/eng/2017/03/09/how-does-a-maven-repository-work/
>[5] https://maven.apache.org/configure.html
>[6]
>https://packages.debian.org/stretch/all/libjsyntaxpane-java/filelist
>[7]
>https://packages.debian.org/stretch/all/libknopflerfish-osgi-framework-java/filelist
>[8] https://wiki.debian.org/Java/MavenBuilder
>[9]
>https://packages.debian.org/stretch/all/libmaven3-core-java/filelist
>[10] https://salsa.debian.org/java-team/maven-debian-helper
>[11] https://fedora-java.github.io/xmvn/
>[12] https://apps.fedoraproject.org/packages/xmvn-minimal/overview/

Thank you for this very detailed information! We've just had a session on the subject at the guix days in Brussels.

We found that we need more plugins to be able to build maven projects, and that requires some metadata in the jar file. This metadata is generated by maven-plugin-tools-generators, and maybe we can write a thin java wrapper around it, or use sxml to generate the file.

If we are going to use mvn and not xmvn, we could change the ant-build-system to generate the repository structure you describe and have a phase to build a union of these with symlinks. Hopefully, maven will follow the symlinks. It seems we don't need more info in the jar files themselves, but we need to install the pom.xml file alongside the jar itself. Additionnaly, we need "parent poms" because they contain properties and values used by our packages and maven will want to read those.

Xmvn looks very cool, but that's still 32 more packages to build properly and generate metadata for, so it's not clear if that's an advantage when we already have a maven package? I'm also a bit worried that it miqht not work properly, as it's an equivalent of 3.0.0, but we have a newer version of maven.

We will also need to override dependency and plugin versions for maven to find guix' versions, so we'll need a parser and generator for pom.xml files.

WDYT?

  reply	other threads:[~2019-01-31 17:07 UTC|newest]

Thread overview: 6+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-01-31 10:00 maven build system - new insights Hartmut Goebel
2019-01-31 17:06 ` Julien Lepiller [this message]
2019-01-31 22:48   ` Hartmut Goebel
2019-01-31 23:33     ` Ricardo Wurmus
2019-02-05 12:00       ` Hartmut Goebel
2019-01-31 23:30 ` Ricardo Wurmus

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: https://guix.gnu.org/

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=71F627FD-4E2E-4E8D-B73E-433368CE3B6B@lepiller.eu \
    --to=julien@lepiller.eu \
    --cc=guix-devel@gnu.org \
    --cc=h.goebel@crazy-compilers.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
Code repositories for project(s) associated with this public inbox

	https://git.savannah.gnu.org/cgit/guix.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).