Help needed from Java developer to finish maven

Help needed from Java developer to finish maven

[Hartmut Goebel]

Hi,

I'm seeking for help from some skilled Java developer. I nearly finished
bootstrapping maven: I made it to start up, but now fails with this
error message:

org.apache.maven.InternalErrorException: Internal error:
com.google.inject.ProvisionException: Unable to provision, see the
following errors:

1) null returned by binding at org.eclipse.sisu.wire.LocatorWiring
 but parameter 1 of
org.eclipse.aether.transport.wagon.WagonTransporterFactory.<init>() is
not @Nullable
  while locating org.eclipse.aether.transport.wagon.WagonConfigurator
    for parameter 1 at
org.eclipse.aether.transport.wagon.WagonTransporterFactory.<init>(Unknown Source)
  while locating org.eclipse.aether.transport.wagon.WagonTransporterFactory
  while locating java.lang.Object annotated with *

I assume this means some meta-date file is missing in one of the jars
(like plexus/components.xml or sisu/javax.inject.Named), but I was not
able to find any missing file or data. may.

Any tipps?

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |

Re: Help needed from Java developer to finish maven

[Björn Höfling]

On Fri, 21 Apr 2017 12:46:40 +0200
Hartmut Goebel <h.goebel@crazy-compilers.com> wrote:

> Hi,
> 
> I'm seeking for help from some skilled Java developer. I nearly
> finished bootstrapping maven: I made it to start up, but now fails
> with this error message:
> 
> org.apache.maven.InternalErrorException: Internal error:
> com.google.inject.ProvisionException: Unable to provision, see the
> following errors:
> 
> 1) null returned by binding at org.eclipse.sisu.wire.LocatorWiring
>  but parameter 1 of
> org.eclipse.aether.transport.wagon.WagonTransporterFactory.<init>() is
> not @Nullable
>   while locating org.eclipse.aether.transport.wagon.WagonConfigurator
>     for parameter 1 at
> org.eclipse.aether.transport.wagon.WagonTransporterFactory.<init>(Unknown
> Source) while locating
> org.eclipse.aether.transport.wagon.WagonTransporterFactory while
> locating java.lang.Object annotated with *
> 
> I assume this means some meta-date file is missing in one of the jars
> (like plexus/components.xml or sisu/javax.inject.Named), but I was not
> able to find any missing file or data. may.
> 
> Any tipps?
> 

1. Is this already the output of "mvn -X" and/or "mvn -e"?
2. Some posts suggest that this could be caused because of wrong
version of dependencies? I.e. API and thus number of parameters changed:

https://issues.apache.org/jira/browse/MNG-5919
http://maven.40175.n5.nabble.com/Maven-wagon-http-Issue-td5806465.html

3. Do you have a WIP branch and instructions on how to reproduce this?


Björn

Re: Help needed from Java developer to finish maven

[Hartmut Goebel]

Hi Björn,

I'm looking forward to your findings!

> 1. Is this already the output of "mvn -X" and/or "mvn -e"?

Yes, this is the output of mvn -X (basically, in fact I'm running "java
… org.apache.maven.cli.MavenCli -X").

> 3. Do you have a WIP branch and instructions on how to reproduce this?

You can find the branch "WIP-maven" at
https://gitlab.com/htgoebel/guix.git. To reproduce simply run

git clone https://gitlab.com/htgoebel/guix.git
cd guix
git checkout WIP-maven
./pre-inst-env guix build maven-for-bootstrap

> 2. Some posts suggest that this could be caused because of wrong
> version of dependencies? I.e. API and thus number of parameters changed:

Inspired by you suggestion, I tested whether the versions I defined in
guix would build when bootstrapping with "maven-ant-task". *maven
bootstraps with this versions* (see below).

In the repo you can find the file "30-compare-jars.ods", which shows the
versions in different test-scenarios:

Col 1: Versions and dependencies as listed in maven's DEPENDENCIES file.
Col 2: Versions in the CLASSPATH when running maven-ant-task. Files not
listed in Col 1 are marked yellow.
Col 4: Versions in CLASSPATH used in guix WIP-maven. Version differing
from Col 2 are marked blue.
Col 3: Versions for running maven-ant-task, adopted to the versions in
WIP-maven.

Notably the versions used in guix are the same versions as used in Col 3
and Col 3 passes. You may want to double-check this.

To reproduce this, run:

git clone https://gitlab.com/htgoebel/guix.git
cd guix
git checkout WIP-maven
X="$PWD"
cd /tmp
tar xzf /gnu/store/ydp9px7*-apache-maven-3.3.9-src.tar.gz
patch < "$X/maven-with-guix-versions.patch"
cd apache-maven-3.3.9
M2_HOME=/tmp/maven-test-build ant all

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |

Re: Help needed from Java developer to finish maven

[Hilco Wijbenga]

Hi Hartmut,

On 21 April 2017 at 03:46, Hartmut Goebel <h.goebel@crazy-compilers.com> wrote:
> I'm seeking for help from some skilled Java developer. I nearly finished
> bootstrapping maven: I made it to start up, but now fails with this
> error message:
>
> org.apache.maven.InternalErrorException: Internal error:
> com.google.inject.ProvisionException: Unable to provision, see the
> following errors:
>
> 1) null returned by binding at org.eclipse.sisu.wire.LocatorWiring
>  but parameter 1 of
> org.eclipse.aether.transport.wagon.WagonTransporterFactory.<init>() is
> not @Nullable
>   while locating org.eclipse.aether.transport.wagon.WagonConfigurator
>     for parameter 1 at
> org.eclipse.aether.transport.wagon.WagonTransporterFactory.<init>(Unknown Source)
>   while locating org.eclipse.aether.transport.wagon.WagonTransporterFactory
>   while locating java.lang.Object annotated with *
>
> I assume this means some meta-date file is missing in one of the jars
> (like plexus/components.xml or sisu/javax.inject.Named), but I was not
> able to find any missing file or data. may.
>
> Any tipps?

(I've been lurking on this ML, I know very little about Guix or Guile.
I had hoped to use Guix for my own packages in my $HOME but that seems
difficult.)

When the Maven devs want to cut a new release they run Maven's
thousands of unit and integration tests before offering it to the
community for a test drive. Only after all that is successful, is the
new Maven release announced to the world. I had a look at
maven-with-guix-versions.patch and I notice that you are changing
various version numbers and replacing some JARs with other JARs. Why
would you do that? Why do you expect the end result to work well? Or
at all? How would anyone be able to trust this patched Maven? (BTW,
Logback is an SLF4J implementation so your comment there makes no
sense.)

You obviously put a lot of effort into this so you must have had a
good reason to make those changes. I'm guessing that you have a very
different goal in mind than I think you do. :-) So what is that goal?
What are you trying to achieve? I'll try to take a few guesses with
comments/suggestions.

1. Offer Maven as a Guix package so one can easily install Maven and
be productive as a Java dev
* For this scenario you don't want to change Maven. You need to be
able to trust your build tool. You need to be able to get support from
the Maven community which is not going to happen if your Maven is
patched and behaving abnormally.
* I would suggest to take the tarball available at
http://maven.apache.org/download.cgi and extract it "into" the Guix
store. It comes with a bunch of JARs that you could replace with links
to wherever they are in the Guix store. Each could be a separate
package. There are also a couple of OS specific files for Jansi which
you can probably treat the same as JARs (i.e. replace with a link to
somewhere else). With that and $MAVEN_HOME/bin on $PATH, Maven should
work. You now have one issue left: the $MAVEN_HOME/lib/ext directory.
It should be possible for a dev to put Maven extensions in that
directory (Maven will pick them up automatically when starting up). I
don't know how one would handle that in Guix.

2. Offer JARs as Guix packages
* Don't try to build the JAR, simply download it from Maven Central
(for example) and make it available in the Guix store.
* The POM (or one of its parents) has information like author(s),
license, and web site. If it's not available there, use a default like
http://www.gnu.org if the web site is missing.

3. Offer JARs as Guix packages by building them from source
* I think this is what you are aiming for? I would suggest installing
Maven as in (1) and then invoking the build with
"-Dmaven.repo.local=/some/tmp/location" (or with "--settings
/some/tmp/location/guix-settings.xml"). Ignoring all direct and
transitive dependencies of this one JAR, you're done. (After the
build, you can remove /some/tmp/location.) This should actually get
you pretty far because tools that use JARs would have an explicit list
of necessary JARs. (Maven is a good example, the lib directory
contains everything that is needed.)

I hope this is of use to you. If you have more questions, ask away.

Cheers,
Hilco

Re: Help needed from Java developer to finish maven

[Hartmut Goebel]

Hi Hilco,

for what I want to achieve, you need to understand Guix's philosophy:
One of the major points is to have as few components as possible
pre-build be external entities. Because only then Guix can ensure that
the component is build from exactly the known source and not manipulated.

And this means, that neither adding JARs from Maven Central to the store
nor putting the maven tar-ball into the store are admissible options for
Guix. I know other distributions do, like NixOS, but Guix will not.

Sadly maven does not support building from source. Even the maven
"source" includes a jar-file (maven-ant-task), which's job is to
download JAR files from maven central. So I have a take a lot of effort
building all (minimum) requirements, manually recreating e.g. meta-data
files which maven creates, and that like. Most of these packages rely on
maven for buiilding - which is not yet available.

My actual gaol is to have some Java applications I need in Guix – but
the Guix way :-) And this requires to be able to build JARs which are
build using maven, and for this I need to be able to bootstrap maven
from source.

Am 23.04.2017 um 20:43 schrieb Hilco Wijbenga: I had a look at
> maven-with-guix-versions.patch and I notice that you are changing
> various version numbers and replacing some JARs with other JARs. Why
> would you do that? Why do you expect the end result to work well? Or
> at all? How would anyone be able to trust this patched Maven?
I assume you are referring so the patch I posted month ago, which is is
outdated now. For the current status, please have a look at the branch
"WIP-maven" at https://gitlab.com/htgoebel/guix.git. My question refer
to this status.

This branch also uses some different versions, but the tar-ball maven
builds and runs fine when using these versions. See
https://lists.gnu.org/archive/html/guix-devel/2017-04/msg00536.html for
details on how I came to this conclusion.

> I hope this is of use to you. If you have more questions, ask away.

If you have an idea what could cause the error I posted, please give fixing it a try. You can find the Guix description at the branch "WIP-maven" at https://gitlab.com/htgoebel/guix.git.

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |

Re: Help needed from Java developer to finish maven

[Hilco Wijbenga]

Hi Hartmut,

On 23 April 2017 at 13:34, Hartmut Goebel <h.goebel@crazy-compilers.com> wrote:
> for what I want to achieve, you need to understand Guix's philosophy:
> One of the major points is to have as few components as possible
> pre-build be external entities. Because only then Guix can ensure that
> the component is build from exactly the known source and not manipulated.

Sure, that makes sense ... up to a point.

First, if it's about trust ("am I really running a stock,
unmanipulated Maven") then there are the checksums and signatures on
the download page which exist for exactly that purpose. If you feel
you cannot rely on those then why trust the SCM that holds the Maven
code? You'd have to become intimately familiar with the entire Maven
code base to be able to state that it wasn't manipulated. In short, if
you can't trust the Maven devs then you should not use Maven.

Second, as an end user this makes it even harder to trust the end
result. I now have to trust both the Maven devs *and* the Guix devs
who worked on the packages. Not only that, but I actually *know* that
Maven has been changed/patched so I can't even check it against the
published checksum/signature. And that's assuming the Guix devs (who
are usually neither Java nor Maven experts) did not make any honest
mistakes. And, again, if my Guix Maven somehow misbehaves I cannot ask
for support from the Maven community because I'm not running a stock
Maven.

It would make much more sense to download the same tarball from, say,
3 different mirrors and confirm that they all have the same
checksum/signature as published on the Maven download page. Then get
Maven's source code (for the same version) and build it with the
binary Maven you just downloaded. Check that the
Maven-built-from-source's checksum and signature match the published
checksum/signature, and voila! An ordinary user would not go through
all that trouble so now you are actually adding value. Moreover, this
*objectively* proves that Maven was not manipulated. I don't have to
trust you not to do anything nefarious nor do I have to take it on
faith that you didn't make any mistakes.

(By the way, I run Gentoo GNU/Linux so I am very familiar with
building from source. For me (and Gentoo) it's more about control and
performance than anything else, though.)

> And this means, that neither adding JARs from Maven Central to the store
> nor putting the maven tar-ball into the store are admissible options for
> Guix. I know other distributions do, like NixOS, but Guix will not.

I can appreciate not wanting to add "random" JARs from Maven Central,
although getting published on Maven Central actually requires going
through a manual process that involves proof of ownership (of the
project). But, still, building such a JAR from source (once Maven is
available) takes little effort and so is worth it. Especially, because
most projects do not publish a checksum and signature. So here
building from source adds value.

> Sadly maven does not support building from source. Even the maven
> "source" includes a jar-file (maven-ant-task), which's job is to
> download JAR files from maven central. So I have a take a lot of effort
> building all (minimum) requirements, manually recreating e.g. meta-data
> files which maven creates, and that like. Most of these packages rely on
> maven for buiilding - which is not yet available.

:-) Well, no, of course not. Maven isn't at the pre alpha state
anymore. So in the spirit of "eat your own dogfood", Maven builds with
Maven.

> My actual gaol is to have some Java applications I need in Guix – but
> the Guix way :-) And this requires to be able to build JARs which are
> build using maven, and for this I need to be able to bootstrap maven
> from source.

Right, that makes perfect sense except for the part where you jump
through hoops and make unexpected changes to Maven just to be able to
build it in a highly unorthodox way. ;-) :-)

> Am 23.04.2017 um 20:43 schrieb Hilco Wijbenga: I had a look at
>> maven-with-guix-versions.patch and I notice that you are changing
>> various version numbers and replacing some JARs with other JARs. Why
>> would you do that? Why do you expect the end result to work well? Or
>> at all? How would anyone be able to trust this patched Maven?
> I assume you are referring so the patch I posted month ago, which is is
> outdated now. For the current status, please have a look at the branch
> "WIP-maven" at https://gitlab.com/htgoebel/guix.git. My question refer
> to this status.
>
> This branch also uses some different versions, but the tar-ball maven
> builds and runs fine when using these versions. See
> https://lists.gnu.org/archive/html/guix-devel/2017-04/msg00536.html for
> details on how I came to this conclusion.

Yes, that's where I looked. That's why I checked the patch file.

>> I hope this is of use to you. If you have more questions, ask away.
>
> If you have an idea what could cause the error I posted, please give fixing it a try. You can find the Guix description at the branch "WIP-maven" at https://gitlab.com/htgoebel/guix.git.

According to msg00536 you patched Maven to use different JARs and
versions. Obviously, that creates a problem so don't do that. Stick
with the JARs/versions that Maven was built with and all will be well.
Going by bin/mvn, you should probably invoke Maven using
org.codehaus.plexus.classworlds.launcher.Launcher instead of
org.apache.maven.cli.MavenCli. But you would be much better off if you
dropped this whole approach as I mentioned above. Even if you somehow
made it work, it would be unreliable.

Cheers,
Hilco

Re: Help needed from Java developer to finish maven

[Catonano]

[-- Attachment #1: Type: text/plain, Size: 1063 bytes --]

2017-04-23 14:36 GMT+02:00 Hartmut Goebel <h.goebel@crazy-compilers.com>:

> Hi Björn,
>
> I'm looking forward to your findings!
>
> > 1. Is this already the output of "mvn -X" and/or "mvn -e"?
>
> Yes, this is the output of mvn -X (basically, in fact I'm running "java
> … org.apache.maven.cli.MavenCli -X").
>
> > 3. Do you have a WIP branch and instructions on how to reproduce this?
>
> You can find the branch "WIP-maven" at
> https://gitlab.com/htgoebel/guix.git. To reproduce simply run
>
> git clone https://gitlab.com/htgoebel/guix.git
> cd guix
> git checkout WIP-maven
> ./pre-inst-env guix build maven-for-bootstrap
>

I tried to build your branch just out of curiosity and the configure of
Guix fails because

configure: error: The Guile bindings of GnuTLS are missing; please install
them.

I'm running this inside a

guix environment guix

so I assume all the deps should be available

I probably won't be able to help you anyway, I was just curious to see the
error you're running into.

Thanks in advance

[-- Attachment #2: Type: text/html, Size: 1800 bytes --]

Guile bindings of GnuTLS are missing (was: Help needed from Java developer to finish maven)

[Hartmut Goebel]

Am 24.04.2017 um 14:41 schrieb Catonano:
> I tried to build your branch just out of curiosity and the configure
> of Guix fails because
>
> configure: error: The Guile bindings of GnuTLS are missing; please
> install them.
>
> I'm running this inside a
>
> guix environment guix
>
> so I assume all the deps should be available

Yes, this is one of the issues bugging me to quite often. Within the
environment you'll also need to run:

export SSL_CERT_DIR="$GUIX_ENVIRONMENT/etc/ssl/certs"
export SSL_CERT_FILE="$GUIX_ENVIRONMENT/etc/ssl/certs/ca-certificates.crt"
export GIT_SSL_CAINFO="$SSL_CERT_FILE"

(This is why I wrote myself a script for setting up a guix environment.)

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |

Re: Guile bindings of GnuTLS are missing (was: Help needed from Java developer to finish maven)

[Catonano]

[-- Attachment #1: Type: text/plain, Size: 623 bytes --]

2017-04-24 14:58 GMT+02:00 Hartmut Goebel <h.goebel@crazy-compilers.com>:

>
> Yes, this is one of the issues bugging me to quite often.


I'm not sure I understand. This doesn't happen with the guix checkout that
I usually build

>
> Within the
> environment you'll also need to run:
>
> export SSL_CERT_DIR="$GUIX_ENVIRONMENT/etc/ssl/certs"
> export SSL_CERT_FILE="$GUIX_ENVIRONMENT/etc/ssl/certs/ca-certificates.crt"
> export GIT_SSL_CAINFO="$SSL_CERT_FILE"
>
> (This is why I wrote myself a script for setting up a guix environment.)


Good to know
Should the guile-gnutls issue be overcome, I will try this too
Thanks

[-- Attachment #2: Type: text/html, Size: 1215 bytes --]

Trust and reproducible build (was: Help needed from Java developer to finish maven)

[Hartmut Goebel]

Am 24.04.2017 um 03:20 schrieb Hilco Wijbenga:
> First, if it's about trust ("am I really running a stock,
> unmanipulated Maven") then there are the checksums and signatures on
> the download page which exist for exactly that purpose.

While  signatures are very useful in many cases, in this case they are
not. A signature on a JAR file only proofs that the JAR file was build
by some person or organization. It does not proof that the content of
the JAR file is exactly what was produced when compiling the source.

> If you feel
> you cannot rely on those then why trust the SCM that holds the Maven
> code? 

The source can be reviews, the JAR-file content not. (One could
disassemble and reverse engineer it, but for this you need to trust
another tool: the disassembler. And then you still have to review the code.)

> Second, as an end user this makes it even harder to trust the end
> result. I now have to trust both the Maven devs *and* the Guix devs
> who worked on the packages.

Now this is where reproducible build steps in: You don't need to trust
the guix developers. Package descriptions are quite easy to review for
code manipulations (even ugly and long ones like the one for java and
the one for maven). So you can get the maven source-code from some
different source, easily verify the one you have is unchanged and build
it. And you will get the same checksums as anybody else building maven
via guix (on the same platform an guix version). Yes, you still have to
trust the guix developers to use a valif version of the C-compiler (and
a few other tools), but this is *much* less to trust compared a other
operation system distributions (Redhat, Fedora, Debian, Windows, OS X,
Solaris), where you have no chance to verify the code.

> (By the way, I run Gentoo GNU/Linux so I am very familiar with
> building from source. For me (and Gentoo) it's more about control and
> performance than anything else, though.)

Soi your aim is different than Guix's aim :-)

> But, still, building such a JAR from source (once Maven is
> available) takes little effort and so is worth it.

You are absolutely right. But "once Maven is available" is the key to
all of this.

> :-) Well, no, of course not. Maven isn't at the pre alpha state
> anymore. So in the spirit of "eat your own dogfood", Maven builds with
> Maven. 

Umm, well,, .... It would help a lot if there would be some "minimal
maven" which could be used for bootstrapping. But even a "minimal Maven"
need tons of other packages

> Right, that makes perfect sense except for the part where you jump
> through hoops and make unexpected changes to Maven just to be able to
> build it in a highly unorthodox way. ;-) :-)

Be ensured that I have no plan at all to change maven. in any way. I'm
not going to make changes to Maven. I simply try bootstrapping it from
source. If there would be some "official" way, I would use it. What I'm
doing is simply adption the build.xml in guix. (Hmm, thinking on this,
maybe I can trip down my description to manipulate the build.xml. But I
don't think this is of much use since I most probably could only reuse
two simple tasks.)

Please keep in mind that Maven is not a compiler. its just a build tool.
So its easy to compare the build results from the guix package
description with whatever Maven builds when it is bootstrapping itself.
And I'm confident that as soon as I make maven to bootstrap itself from
source, it will be correct.

> According to msg00536 you patched Maven to use different JARs and
> versions. Obviously, that creates a problem so don't do that. Stick
> with the JARs/versions that Maven was built with and all will be well.

Please try to reproduce what I've done. The error message I posted is in
now way related to any version change, since when using the tar-ball I
could change the version and still bootstrap maven.As opposed to the
guix description, where maven fails to run quite early.

> But you would be much better off if you dropped this whole approach as
> I mentioned above. Even if you somehow made it work, it would be
> unreliable.

It would be much better if you'd help finding the cause of the error
message instead of spending time arguing about the way :-) No offence
meant, but adding precompiled JARS from maven central is not an option.

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |