bug#31721: “generate-jar-indices” phase fails on many packages

bug#31721: “generate-jar-indices” phase fails on many packages

[Ricardo Wurmus]

The “generate-jar-indices” phase fails on a couple of packages, such as

  java-picard (all variants)
  dropseq-tools
  java-cisd-jhdf5

The errors are either about duplicates as in java-picard-2.10.3:

    duplicate entry: htsjdk/samtools/AbstractBAMFileIndex$1.class

or about missing files as in dropseq-tools:

    java.io.FileNotFoundException: /gnu/store/q76y0ximcziplgfpbn26kbw4h3s14f33-dropseq-tools-1.13/share/java/lib/biojava-alignment.jar

I have removed the phase in these packages, but that’s not ideal.

Can we make this phase more robust?

-- 
Ricardo

Re: “generate-jar-indices” phase fails on many packages

[Danny Milosavljevic]

[-- Attachment #1: Type: text/plain, Size: 1472 bytes --]

Hi Ricardo,

did you get a bug# for this?  I can't see it on https://debbugs.gnu.org/cgi/pkgreport.cgi?pkg=guix ...

> The errors are either about duplicates as in java-picard-2.10.3:
> 
>     duplicate entry: htsjdk/samtools/AbstractBAMFileIndex$1.class

That's... bad.  What if those files differed?  Which one would be used at
runtime?

> or about missing files as in dropseq-tools:
> 
>     java.io.FileNotFoundException: /gnu/store/q76y0ximcziplgfpbn26kbw4h3s14f33-dropseq-tools-1.13/share/java/lib/biojava-alignment.jar

That's also bad.  How does it run at all if its dependencies are missing?

> I have removed the phase in these packages, but that’s not ideal.

I think that's suboptimal in the above cases since those error messages report
real problems, or in the first case at least pointing to a potential problem.

> Can we make this phase more robust?

Right now, it just invokes

  jar -i

with the finished jar file.  That's as simple as it gets - and I specifically
did only that minimal implementation for this cycle in order to only fix our GC
problem with java packages.

The only thing I can think of that would improve things long term:

The phase can add relative paths to all the dependencies to META-INF/MANIFEST.MF
before invoking "jar -i".
Then "jar -i" will index those - and all java packages can use regular inputs
instead of propagated inputs.  I've tested that locally already - and it
works fine.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

bug#31721: “generate-jar-indices” phase fails on many packages

[Ricardo Wurmus]

Danny Milosavljevic <dannym@scratchpost.org> writes:

>> The errors are either about duplicates as in java-picard-2.10.3:
>> 
>>     duplicate entry: htsjdk/samtools/AbstractBAMFileIndex$1.class
>
> That's... bad.  What if those files differed?  Which one would be used at
> runtime?

I wonder where they come from.  Is this from a dependency or are there
actually two such classes?

>> or about missing files as in dropseq-tools:
>> 
>>     java.io.FileNotFoundException: /gnu/store/q76y0ximcziplgfpbn26kbw4h3s14f33-dropseq-tools-1.13/share/java/lib/biojava-alignment.jar
>
> That's also bad.  How does it run at all if its dependencies are missing?

To satisfy the dropseq-tools build system the dependencies a symlinked
to the build directory.  I suspect that the error is something to do
with that.

> The only thing I can think of that would improve things long term:
>
> The phase can add relative paths to all the dependencies to META-INF/MANIFEST.MF
> before invoking "jar -i".
> Then "jar -i" will index those - and all java packages can use regular inputs
> instead of propagated inputs.  I've tested that locally already - and it
> works fine.

Is it correct that this would ensure that in case of duplicate class
names this would pick the class from the current package?

-- 
Ricardo

bug#31721: “generate-jar-indices” phase fails on many packages

[Gábor Boskovits]

[-- Attachment #1: Type: text/plain, Size: 1699 bytes --]

2018-06-07 9:20 GMT+02:00 Ricardo Wurmus <ricardo.wurmus@mdc-berlin.de>:

>
> Danny Milosavljevic <dannym@scratchpost.org> writes:
>
> >> The errors are either about duplicates as in java-picard-2.10.3:
> >>
> >>     duplicate entry: htsjdk/samtools/AbstractBAMFileIndex$1.class
> >
> > That's... bad.  What if those files differed?  Which one would be used at
> > runtime?
>
> I wonder where they come from.  Is this from a dependency or are there
> actually two such classes?
>
>
Is it possible that these come from a package, where a renamed dependency is
used in the original build system, thus avoiding the name duplication? I
will have a
look at this package source... Will report back if I can confirm this
assumption.



> >> or about missing files as in dropseq-tools:
> >>
> >>     java.io.FileNotFoundException: /gnu/store/
> q76y0ximcziplgfpbn26kbw4h3s14f33-dropseq-tools-1.13/share/
> java/lib/biojava-alignment.jar
> >
> > That's also bad.  How does it run at all if its dependencies are missing?
>
> To satisfy the dropseq-tools build system the dependencies a symlinked
> to the build directory.  I suspect that the error is something to do
> with that.
>
> > The only thing I can think of that would improve things long term:
> >
> > The phase can add relative paths to all the dependencies to
> META-INF/MANIFEST.MF
> > before invoking "jar -i".
> > Then "jar -i" will index those - and all java packages can use regular
> inputs
> > instead of propagated inputs.  I've tested that locally already - and it
> > works fine.
>
> Is it correct that this would ensure that in case of duplicate class
> names this would pick the class from the current package?
>
> --
> Ricardo
>
>
>
>

[-- Attachment #2: Type: text/html, Size: 2611 bytes --]

bug#31721: “generate-jar-indices” phase fails on many packages

[Gábor Boskovits]

[-- Attachment #1: Type: text/plain, Size: 2127 bytes --]

2018-06-07 10:27 GMT+02:00 Gábor Boskovits <boskovits@gmail.com>:

> 2018-06-07 9:20 GMT+02:00 Ricardo Wurmus <ricardo.wurmus@mdc-berlin.de>:
>
>>
>> Danny Milosavljevic <dannym@scratchpost.org> writes:
>>
>> >> The errors are either about duplicates as in java-picard-2.10.3:
>> >>
>> >>     duplicate entry: htsjdk/samtools/AbstractBAMFileIndex$1.class
>> >
>> > That's... bad.  What if those files differed?  Which one would be used
>> at
>> > runtime?
>>
>> I wonder where they come from.  Is this from a dependency or are there
>> actually two such classes?
>>
>>
> Is it possible that these come from a package, where a renamed dependency
> is
> used in the original build system, thus avoiding the name duplication? I
> will have a
> look at this package source... Will report back if I can confirm this
> assumption.
>
> This is what is in build.gradle:
dependencies {
compile('com.intel.gkl:gkl:0.8.5') {
exclude module: 'htsjdk'
}
 So we should exclude htsjdk module from gkl when building java-picard.
Do we have a method to do that easily?


>
>> >> or about missing files as in dropseq-tools:
>> >>
>> >>     java.io.FileNotFoundException: /gnu/store/q76y0ximcziplgfpbn2
>> 6kbw4h3s14f33-dropseq-tools-1.13/share/java/lib/biojava-alignment.jar
>> >
>> > That's also bad.  How does it run at all if its dependencies are
>> missing?
>>
>> To satisfy the dropseq-tools build system the dependencies a symlinked
>> to the build directory.  I suspect that the error is something to do
>> with that.
>>
>> > The only thing I can think of that would improve things long term:
>> >
>> > The phase can add relative paths to all the dependencies to
>> META-INF/MANIFEST.MF
>> > before invoking "jar -i".
>> > Then "jar -i" will index those - and all java packages can use regular
>> inputs
>> > instead of propagated inputs.  I've tested that locally already - and it
>> > works fine.
>>
>> Is it correct that this would ensure that in case of duplicate class
>> names this would pick the class from the current package?
>>
>> --
>> Ricardo
>>
>>
>>
>>
>

[-- Attachment #2: Type: text/html, Size: 7325 bytes --]