maven build system - new insights

maven build system - new insights

[Hartmut Goebel]

[-- Attachment #1: Type: text/plain, Size: 5985 bytes --]

H Julien,

as promised I've spend some time on a maven build system (as always when
regarding maven it was much more time than expected). I did not yet
create a build-system but want to share my insights so we can decide how
to proceed further. If I'm wrong on some points, please feel free to
correct this. Everything below is based on my investigations the last days.

TL;DR: Let's use XMvn [11] for our maven build-system. What do you think?

  * The only place where maven actually loads artifacts from is the
    local repository, which is a single directory (default:
    $HOME/.m2/repository). If an artifact is not found in the local
    repository, it is downloaded from some "remote" repositories and
    cached in the local repository. I have not found any way to change
    this behavior. See Analysis of repositories behavior below for details.
  * The local repository path can be changed by passing
    "-Dmaven.repo.local=..." to maven.
  * The files in the repository need to be structured like this:
      o .../org/apache/maven/maven-plugin-descriptor/2.2.1/maven-plugin-descriptor-2.2.1.jar

This implies a maven-build-system would need either to merge all
artifacts into a "local repository" or to set up a lot of "mirrors" (one
for each input, this will copy all artifacts into the "local
repository"). Both is complicated.

Thus I suggest to leverage Xmvn [11] for building (see Distro-Analysis
below for details). Xmvn searches for configurations (and thus
repository definitions) in e.g $XDG_DATA_DIRS plus it requires only ca.
32 jar-files (compared to ca. 48 for maven). Thus we could not only
solve the repository issue but also (hopefully) reduce the number of
packages we need to bootstrap.

*Distro-Analysis*

Here is what other GNU/Linux distributions do:

  * Debian stores the files in a system repository structured like
    above, see [9]. (Thought this seem not to be done for all packages,
    |e.g.|||[6] and [7] do not include ||these maven-repo files.)
  * Debian provides a script "mvn-debian" [8] which seem to tweak maven
    to use this system repository (by passing
    -Dmaven.repo.local=$MAVEN_REPO").
      o This is still a single directory. For guix this would require
        all inputs to be merged into a single directory :-(
  * Debian also has some build-helper "mh_make" [10] which does some
    more magic I did not investigate by running some Debian-specific
    java classes for collecting dependencies.

  * Fedora uses xmvn [11] which "extends" maven to maintain a
    system-wide repository of artifacts.
  * There is also a "/Dependency-reduced version of XMvn" /[12],
    incapable of using remote repositories. This could be helpful for
    bootstrapping.
      o A sort analysis says: 32 instead of 48 jar-files are required.
        Thus we might be able to eliminate 1/3 of the dependence for
        bootstrapping maven.
  * The directory structure does not follow the repo-structure, but
    lloks like this:
      o /usr/share/java/maven-plugin-tools/maven-plugin-plugin.jar
        /usr/share/maven-metadata/maven-plugin-tools-maven-plugin-plugin.xml
        /usr/share/maven-poms/maven-plugin-tools/maven-plugin-plugin.pom
  *

  * xmvn searches for config-files, which are merged in
      o $PWD - which would allow us to create one on the fly based on
        all inputs
      o $XDG_CONFIG_DIRS and $XDG_DATA_DIRS - which would allow to use
        native search-path specification

*Analysis of repositories behavior:*

  * The local repository can consist of a single directory only, I did
    not find any hint about how to specify several directories. Esp. the
    "Configuring Maven Guide" [1] and [5] state a single directory only.
    This makes sense somehow since the local repo is where downloaded
    files are cached.
  * "Remote" repositories
      o can be specified in settings.xml (and require to pass a profile
        name when running, e.g. mvn -Pmyprofile) or the pom.xml. Maybe
        also in .mvn/{settings,extension).xml, I did not test this.
      o repositories are processed *after* the central repository
        (source: [2])
      o Even if the repository is specified with file://... the
        artifacts will be copied to the local repo (source: [2], also
        observed). [3] does not show any option to suppress download.
      o When using "--offline", all remote repositories are disabled,
        even if specified with file://...
  * Mirrors do not help
      o Can be specified in settings.xml without requiring a profile
      o Even if the mirror is specified with file://... the artifacts
        will be copied to the local repo (observed, and somehow logically)
  * Some more background about repositories can be found at [4].
  * I also strace'd and ltrace'd several runs of maven to investigate
    whether there any other env-vars or directories are accessed. Nothing.

[1] https://maven.apache.org/guides/mini/guide-configuring-maven.html
[2] https://www.tutorialspoint.com/maven/maven_repositories.htm
[3]
https://maven.apache.org/ref/3.6.0/maven-settings/settings.html#class_releases
[4]
https://blog.packagecloud.io/eng/2017/03/09/how-does-a-maven-repository-work/
[5] https://maven.apache.org/configure.html
[6] https://packages.debian.org/stretch/all/libjsyntaxpane-java/filelist
[7]
https://packages.debian.org/stretch/all/libknopflerfish-osgi-framework-java/filelist
[8] https://wiki.debian.org/Java/MavenBuilder
[9] https://packages.debian.org/stretch/all/libmaven3-core-java/filelist
[10] https://salsa.debian.org/java-team/maven-debian-helper
[11] https://fedora-java.github.io/xmvn/
[12] https://apps.fedoraproject.org/packages/xmvn-minimal/overview/

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |


[-- Attachment #2: Type: text/html, Size: 9382 bytes --]

Re: maven build system - new insights

[Julien Lepiller]

Le 31 janvier 2019 11:00:42 GMT+01:00, Hartmut Goebel <h.goebel@crazy-compilers.com> a écrit :
>H Julien,
>
>as promised I've spend some time on a maven build system (as always
>when
>regarding maven it was much more time than expected). I did not yet
>create a build-system but want to share my insights so we can decide
>how
>to proceed further. If I'm wrong on some points, please feel free to
>correct this. Everything below is based on my investigations the last
>days.
>
>TL;DR: Let's use XMvn [11] for our maven build-system. What do you
>think?
>
>  * The only place where maven actually loads artifacts from is the
>    local repository, which is a single directory (default:
>    $HOME/.m2/repository). If an artifact is not found in the local
>    repository, it is downloaded from some "remote" repositories and
>    cached in the local repository. I have not found any way to change
>this behavior. See Analysis of repositories behavior below for details.
>  * The local repository path can be changed by passing
>    "-Dmaven.repo.local=..." to maven.
>  * The files in the repository need to be structured like this:
>o
>.../org/apache/maven/maven-plugin-descriptor/2.2.1/maven-plugin-descriptor-2.2.1.jar
>
>This implies a maven-build-system would need either to merge all
>artifacts into a "local repository" or to set up a lot of "mirrors"
>(one
>for each input, this will copy all artifacts into the "local
>repository"). Both is complicated.
>
>Thus I suggest to leverage Xmvn [11] for building (see Distro-Analysis
>below for details). Xmvn searches for configurations (and thus
>repository definitions) in e.g $XDG_DATA_DIRS plus it requires only ca.
>32 jar-files (compared to ca. 48 for maven). Thus we could not only
>solve the repository issue but also (hopefully) reduce the number of
>packages we need to bootstrap.
>
>*Distro-Analysis*
>
>Here is what other GNU/Linux distributions do:
>
>  * Debian stores the files in a system repository structured like
>    above, see [9]. (Thought this seem not to be done for all packages,
>    |e.g.|||[6] and [7] do not include ||these maven-repo files.)
>  * Debian provides a script "mvn-debian" [8] which seem to tweak maven
>    to use this system repository (by passing
>    -Dmaven.repo.local=$MAVEN_REPO").
>      o This is still a single directory. For guix this would require
>        all inputs to be merged into a single directory :-(
>  * Debian also has some build-helper "mh_make" [10] which does some
>    more magic I did not investigate by running some Debian-specific
>    java classes for collecting dependencies.
>
>  * Fedora uses xmvn [11] which "extends" maven to maintain a
>    system-wide repository of artifacts.
>  * There is also a "/Dependency-reduced version of XMvn" /[12],
>    incapable of using remote repositories. This could be helpful for
>    bootstrapping.
>      o A sort analysis says: 32 instead of 48 jar-files are required.
>        Thus we might be able to eliminate 1/3 of the dependence for
>        bootstrapping maven.
>  * The directory structure does not follow the repo-structure, but
>    lloks like this:
>      o /usr/share/java/maven-plugin-tools/maven-plugin-plugin.jar
>   /usr/share/maven-metadata/maven-plugin-tools-maven-plugin-plugin.xml
>       /usr/share/maven-poms/maven-plugin-tools/maven-plugin-plugin.pom
>  *
>
>  * xmvn searches for config-files, which are merged in
>      o $PWD - which would allow us to create one on the fly based on
>        all inputs
>      o $XDG_CONFIG_DIRS and $XDG_DATA_DIRS - which would allow to use
>        native search-path specification
>
>*Analysis of repositories behavior:*
>
>  * The local repository can consist of a single directory only, I did
>   not find any hint about how to specify several directories. Esp. the
>   "Configuring Maven Guide" [1] and [5] state a single directory only.
>    This makes sense somehow since the local repo is where downloaded
>    files are cached.
>  * "Remote" repositories
>      o can be specified in settings.xml (and require to pass a profile
>        name when running, e.g. mvn -Pmyprofile) or the pom.xml. Maybe
>        also in .mvn/{settings,extension).xml, I did not test this.
>      o repositories are processed *after* the central repository
>        (source: [2])
>      o Even if the repository is specified with file://... the
>        artifacts will be copied to the local repo (source: [2], also
>        observed). [3] does not show any option to suppress download.
>      o When using "--offline", all remote repositories are disabled,
>        even if specified with file://...
>  * Mirrors do not help
>      o Can be specified in settings.xml without requiring a profile
>      o Even if the mirror is specified with file://... the artifacts
>     will be copied to the local repo (observed, and somehow logically)
>  * Some more background about repositories can be found at [4].
>  * I also strace'd and ltrace'd several runs of maven to investigate
> whether there any other env-vars or directories are accessed. Nothing.
>
>[1] https://maven.apache.org/guides/mini/guide-configuring-maven.html
>[2] https://www.tutorialspoint.com/maven/maven_repositories.htm
>[3]
>https://maven.apache.org/ref/3.6.0/maven-settings/settings.html#class_releases
>[4]
>https://blog.packagecloud.io/eng/2017/03/09/how-does-a-maven-repository-work/
>[5] https://maven.apache.org/configure.html
>[6]
>https://packages.debian.org/stretch/all/libjsyntaxpane-java/filelist
>[7]
>https://packages.debian.org/stretch/all/libknopflerfish-osgi-framework-java/filelist
>[8] https://wiki.debian.org/Java/MavenBuilder
>[9]
>https://packages.debian.org/stretch/all/libmaven3-core-java/filelist
>[10] https://salsa.debian.org/java-team/maven-debian-helper
>[11] https://fedora-java.github.io/xmvn/
>[12] https://apps.fedoraproject.org/packages/xmvn-minimal/overview/

Thank you for this very detailed information! We've just had a session on the subject at the guix days in Brussels.

We found that we need more plugins to be able to build maven projects, and that requires some metadata in the jar file. This metadata is generated by maven-plugin-tools-generators, and maybe we can write a thin java wrapper around it, or use sxml to generate the file.

If we are going to use mvn and not xmvn, we could change the ant-build-system to generate the repository structure you describe and have a phase to build a union of these with symlinks. Hopefully, maven will follow the symlinks. It seems we don't need more info in the jar files themselves, but we need to install the pom.xml file alongside the jar itself. Additionnaly, we need "parent poms" because they contain properties and values used by our packages and maven will want to read those.

Xmvn looks very cool, but that's still 32 more packages to build properly and generate metadata for, so it's not clear if that's an advantage when we already have a maven package? I'm also a bit worried that it miqht not work properly, as it's an equivalent of 3.0.0, but we have a newer version of maven.

We will also need to override dependency and plugin versions for maven to find guix' versions, so we'll need a parser and generator for pom.xml files.

WDYT?

Re: maven build system - new insights

[Hartmut Goebel]

Hi,

Am 31.01.19 um 18:06 schrieb Julien Lepiller:

Here are my 2cent:

> We found that we need more plugins to be able to build maven projects, and that requires some metadata in the jar file. This metadata is generated by maven-plugin-tools-generators, and maybe we can write a thin java wrapper around it, or use sxml to generate the file.

Regarding the thin java layer it might be interesting to have a look at
Debians build helpers which might contains some of what we need.
Unfortunately it is not well documented (at least i did not find it
quickly), but asking the Debian Java packagers might be helpful.


> If we are going to use mvn and not xmvn, we could change the ant-build-system to generate the repository structure you describe

There seems to be a maven task for doing so: install:install-file [1].


> and have a phase to build a union of these with symlinks. Hopefully, maven will follow the symlinks.

The drawback of this approach is: This only works within the
build-system. Since within a normal environment or profile, there will
be no such unison repo. Thus for every day's work, users will still
download artifacts from the Internet and will not be able to use those
already installed on the system.

One of the points I like at Xmvn is that is does not try to reach out to
the internet. And users should be able to simply replace mvn by xmvn to
ensure only "well known" packages are used.


> It seems we don't need more info in the jar files themselves, but we need to install the pom.xml file alongside the jar itself. Additionnaly, we need "parent poms" because they contain properties and values used by our packages and maven will want to read those.

For the jar an pom-files, install:install-file could help [1]. For the
parent poms I have no idea, beside the Debians build helpers having some
"DebianDependencies" class.


> Xmvn looks very cool, but that's still 32 more packages to build properly and generate metadata for, so it's not clear if that's an advantage when we already have a maven package? I'm also a bit worried that it miqht not work properly, as it's an equivalent of 3.0.0, but we have a newer version of maven.

Oh, I missed one relevant point: these 32 packages are also dependencies
of maven. Thus they are already available. :-) It just that we most
probable could throw out 16 packages from the manual bootstrap. (There
are 3 additional xmvn packages, though.)

The version 3.0.0 is the Xmvn version which seems to be unrelated to maven.

Prior to going down the Xmvn-road we should collect a list of questions
to ask the developers. E.g. [2] say: "Some parts of XMvn require
Gradle". If we need these parts for bootstrapping this would be a
show-stopper.

> We will also need to override dependency and plugin versions for maven to find guix' versions, so we'll need a parser and generator for pom.xml files.

Maybe some xml-search-replace would suffice? E.g. implemented using
SXPATH [3]


[1] https://maven.apache.org/guides/mini/guide-3rd-party-jars-local.html
[2] https://github.com/fedora-java/xmvn/blob/master/README.md
[3] https://www.gnu.org/software/guile/manual/html_node/SXPath.html

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |

Re: maven build system - new insights

[Ricardo Wurmus]

Hartmut Goebel <h.goebel@crazy-compilers.com> writes:

>   * Debian stores the files in a system repository structured like
>     above, see [9]. (Thought this seem not to be done for all packages,
>     |e.g.|||[6] and [7] do not include ||these maven-repo files.)
>   * Debian provides a script "mvn-debian" [8] which seem to tweak maven
>     to use this system repository (by passing
>     -Dmaven.repo.local=$MAVEN_REPO").
>       o This is still a single directory. For guix this would require
>         all inputs to be merged into a single directory :-(

We can do this dynamically with “union-build” from “(guix build
union)”.  This doesn’t need to be considered an obstacle.

--
Ricardo

Re: maven build system - new insights

[Ricardo Wurmus]

Hartmut Goebel <h.goebel@crazy-compilers.com> writes:

>> and have a phase to build a union of these with symlinks. Hopefully,
>> maven will follow the symlinks.
>
> The drawback of this approach is: This only works within the
> build-system. Since within a normal environment or profile, there will
> be no such unison repo. Thus for every day's work, users will still
> download artifacts from the Internet and will not be able to use those
> already installed on the system.

A profile *is* a union of all installed packages.  If that’s not enough
and more configuration is required we could do that in a profile hook.

--
Ricardo

Re: maven build system - new insights

[Hartmut Goebel]

Am 01.02.19 um 00:33 schrieb Ricardo Wurmus:
> Hartmut Goebel <h.goebel@crazy-compilers.com> writes:
>> The drawback of this approach is: This only works within the
>> build-system. Since within a normal environment or profile, there will
>> be no such unison repo. Thus for every day's work, users will still
>> download artifacts from the Internet and will not be able to use those
>> already installed on the system.
> A profile *is* a union of all installed packages.  If that’s not enough
> and more configuration is required we could do that in a profile hook.

I didn't know about the profile hooks. Nevertheless:

I want to emphasis that maven does *not* support some kind of "system
repository", as I explained in my original post. Even if you mimic a
"system repository", this will always *copy* the artifacts to
$HOME/.m2/repository. Debian uses some *extensions* to make maven use
some "system repository" - at least this is what I assume. So this is
much like xmvn, with a difference that end-users can not use it easily.

Beside this: I suggest first of all to design the expected behavior,
that reach out for a solution. Who is the one driving this? (I'm not, I
can't program guile :-)

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |