bug#58591: Java packages do not appear to keep a reference to their inputs

bug#58591: Java packages do not appear to keep a reference to their inputs

[Maxim Cournoyer]

Hello,

I'm not a Java expert, but this appears to me problematic:

--8<---------------cut here---------------start------------->8---
$ guix build java-commons-dbcp
/gnu/store/jghsa6fmh9vjcsmj7wwilk3w6iblvh32-java-commons-dbcp-2.6.0

$ guix gc -R /gnu/store/jghsa6fmh9vjcsmj7wwilk3w6iblvh32-java-commons-dbcp-2.6.0
/gnu/store/jghsa6fmh9vjcsmj7wwilk3w6iblvh32-java-commons-dbcp-2.6.0
--8<---------------cut here---------------end--------------->8---

Digging a bit more, peeking into the .jar file, which is a ZIP archive:

--8<---------------cut here---------------start------------->8---
$ unzip /gnu/store/jghsa6fmh9vjcsmj7wwilk3w6iblvh32-java-commons-dbcp-2.6.0/\
share/java/java-commons-dbcp.jar -d /tmp/java-commons-dbcp.jar

$ grep -rin CLASSPATH /tmp/java-commons-dbcp.jar
$ grep -rin /gnu/store /tmp/java-commons-dbcp.jar
/tmp/java-commons-dbcp.jar/META-INF/INDEX.LIST:3:/gnu/store/jghsa6fmh9vjcsmj7wwilk3w6iblvh32-java-commons-dbcp-2.6.0/share/java/java-commons-dbcp.jar

$ cat /tmp/java-commons-dbcp.jar/META-INF/INDEX.LIST
JarIndex-Version: 1.0

/gnu/store/jghsa6fmh9vjcsmj7wwilk3w6iblvh32-java-commons-dbcp-2.6.0/share/java/java-commons-dbcp.jar
org
org/apache
org/apache/commons
org/apache/commons/dbcp2
org/apache/commons/dbcp2/cpdsadapter
org/apache/commons/dbcp2/datasources
org/apache/commons/dbcp2/managed
--8<---------------cut here---------------end--------------->8---

Still, no traces of the other libraries such as 'java-commons-pool'
which should be referenced.

I assume this means grafts doesn't currently work for Java libraries.

-- 
Thanks,
Maxim

bug#58591: Java packages do not appear to keep a reference to their inputs

[Julien Lepiller]

[-- Attachment #1: Type: text/plain, Size: 1964 bytes --]

You're right, java package don't retain references to there input, that's why we propagate required dependencies (mh… sometimes). I don't know how they could reference dependencies directly.

Le 17 octobre 2022 23:04:47 GMT+02:00, Maxim Cournoyer <maxim.cournoyer@gmail.com> a écrit :
>Hello,
>
>I'm not a Java expert, but this appears to me problematic:
>
>--8<---------------cut here---------------start------------->8---
>$ guix build java-commons-dbcp
>/gnu/store/jghsa6fmh9vjcsmj7wwilk3w6iblvh32-java-commons-dbcp-2.6.0
>
>$ guix gc -R /gnu/store/jghsa6fmh9vjcsmj7wwilk3w6iblvh32-java-commons-dbcp-2.6.0
>/gnu/store/jghsa6fmh9vjcsmj7wwilk3w6iblvh32-java-commons-dbcp-2.6.0
>--8<---------------cut here---------------end--------------->8---
>
>Digging a bit more, peeking into the .jar file, which is a ZIP archive:
>
>--8<---------------cut here---------------start------------->8---
>$ unzip /gnu/store/jghsa6fmh9vjcsmj7wwilk3w6iblvh32-java-commons-dbcp-2.6.0/\
>share/java/java-commons-dbcp.jar -d /tmp/java-commons-dbcp.jar
>
>$ grep -rin CLASSPATH /tmp/java-commons-dbcp.jar
>$ grep -rin /gnu/store /tmp/java-commons-dbcp.jar
>/tmp/java-commons-dbcp.jar/META-INF/INDEX.LIST:3:/gnu/store/jghsa6fmh9vjcsmj7wwilk3w6iblvh32-java-commons-dbcp-2.6.0/share/java/java-commons-dbcp.jar
>
>$ cat /tmp/java-commons-dbcp.jar/META-INF/INDEX.LIST
>JarIndex-Version: 1.0
>
>/gnu/store/jghsa6fmh9vjcsmj7wwilk3w6iblvh32-java-commons-dbcp-2.6.0/share/java/java-commons-dbcp.jar
>org
>org/apache
>org/apache/commons
>org/apache/commons/dbcp2
>org/apache/commons/dbcp2/cpdsadapter
>org/apache/commons/dbcp2/datasources
>org/apache/commons/dbcp2/managed
>--8<---------------cut here---------------end--------------->8---
>
>Still, no traces of the other libraries such as 'java-commons-pool'
>which should be referenced.
>
>I assume this means grafts doesn't currently work for Java libraries.
>
>-- 
>Thanks,
>Maxim
>
>
>

[-- Attachment #2: Type: text/html, Size: 2329 bytes --]

bug#58591: Java packages do not appear to keep a reference to their inputs

[Mark H Weaver]

Julien Lepiller <julien@lepiller.eu> writes:

> You're right, java package don't retain references to there input,
> that's why we propagate required dependencies (mh… sometimes). I don't
> know how they could reference dependencies directly.

A better workaround would be to add a phase that installs file(s) in the
output(s) that contain references to the required store items.  They
could simply be text files with one line per reference.  That would at
least protect the dependencies from the garbage collector.

The remaining unsolved problem is, of course, grafting.

       Mark

-- 
Disinformation flourishes because many people care deeply about injustice
but very few check the facts.  Ask me about <https://stallmansupport.org>.

bug#58591: Java packages do not appear to keep a reference to their inputs

[Maxim Cournoyer]

Hi Julien,

Julien Lepiller <julien@lepiller.eu> writes:

> You're right, java package don't retain references to there input,
> that's why we propagate required dependencies (mh… sometimes). I don't
> know how they could reference dependencies directly.

Could we, along with installing Java classes as directories instead of
.jar archive files [0] at a more specific prefix, define a search path
specification that'd set CLASSPATH?  Currently I don't see anything
setting CLASSPATH outside of the build systems, so even if we propagate
Java things, I don't see how it'd find them in a profile.

[0]  Not exactly sure how that's done yet, but it's mentioned here: https://docs.oracle.com/javase/8/docs/technotes/tools/windows/classpath.html

--
Thanks,
Maxim

bug#58591: Java packages do not appear to keep a reference to their inputs

[Maxime Devos]

[-- Attachment #1.1.1: Type: text/plain, Size: 461 bytes --]

On 18-10-2022 04:45, Maxim Cournoyer wrote:
> [...] setting CLASSPATH outside of the build systems, so even if we propagate
> Java things, I don't see how it'd find them in a profile.

FWIW, when I used java things in Guix, I manually did 
CLASSPATH=$GUIX_ENVIRONMENT/... or the CLI equivalent (some option 
argument of 'java').

Some more automatisation, e.g. in the form of search paths as you 
propose, would be nice though.

Greetings,
Maxime.

[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 929 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 236 bytes --]

bug#58591: Java packages do not appear to keep a reference to their inputs

[Liliana Marie Prikler]

Am Montag, dem 17.10.2022 um 22:45 -0400 schrieb Maxim Cournoyer:
> Hi Julien,
> 
> Julien Lepiller <julien@lepiller.eu> writes:
> 
> > You're right, java package don't retain references to there input,
> > that's why we propagate required dependencies (mh… sometimes). I
> > don't
> > know how they could reference dependencies directly.
> 
> Could we, along with installing Java classes as directories instead
> of .jar archive files [0] at a more specific prefix, define a search
> path specification that'd set CLASSPATH?  Currently I don't see
> anything setting CLASSPATH outside of the build systems, so even if
> we propagate Java things, I don't see how it'd find them in a
> profile.
I'd recommend writing an xml file like 

  <path id="${java-package-name}.classpath">
    <pathelement location="${output-jar}" />
    <pathelement path="${input1.classpath}" />
    ... 
    <pathelement path="${inputn.classpath}" /> 
  </path>

to a well-known location.  Then we could reuse those files in ant-
build-system.

Cheers

bug#58591: Java packages do not appear to keep a reference to their inputs

[Maxim Cournoyer]

Hello,

Liliana Marie Prikler <liliana.prikler@ist.tugraz.at> writes:

> Am Montag, dem 17.10.2022 um 22:45 -0400 schrieb Maxim Cournoyer:
>> Hi Julien,
>> 
>> Julien Lepiller <julien@lepiller.eu> writes:
>> 
>> > You're right, java package don't retain references to there input,
>> > that's why we propagate required dependencies (mh… sometimes). I
>> > don't
>> > know how they could reference dependencies directly.
>> 
>> Could we, along with installing Java classes as directories instead
>> of .jar archive files [0] at a more specific prefix, define a search
>> path specification that'd set CLASSPATH?  Currently I don't see
>> anything setting CLASSPATH outside of the build systems, so even if
>> we propagate Java things, I don't see how it'd find them in a
>> profile.
> I'd recommend writing an xml file like 
>
>   <path id="${java-package-name}.classpath">
>     <pathelement location="${output-jar}" />
>     <pathelement path="${input1.classpath}" />
>     ... 
>     <pathelement path="${inputn.classpath}" /> 
>   </path>
>
> to a well-known location.  Then we could reuse those files in ant-
> build-system.

A nice read is [0], which mentions the existence of a 'Class-Path' main
attribute that can go in the manifest file.  If using unpacked jars
works the same as .jars (which are just zip files) for Java, then we
could not only have dependency correctly referenced and loaded via
'Class-Path', but also the grafting mechanism would work, since the
paths would appear in clear (not obfuscated due to zip compression).

Our current usage of JarIndex doesn't suite the bill it was intended
for; this is a performance trick to index all the .jars of a .jar pack;
it'll only list its dependencies if they are packed in the same jar,
which is not what we do or want as a distribution.

[0]  https://docs.oracle.com/en/java/javase/19/docs/specs/jar/jar.html

-- 
Thanks,
Maxim

bug#58591: Java packages do not appear to keep a reference to their inputs

[Tobias Geerinckx-Rice via Bug reports for GNU Guix]

[-- Attachment #1: Type: text/plain, Size: 237 bytes --]

Hi Maxim,

Maxim Cournoyer 写道：
> not obfuscated due to zip compression

Groan.  Which package(s) compress .jars?

(I found a few in -checkouts, which is its own potential thing, 
but that aside.)

Kind regards,

T G-R

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 247 bytes --]

bug#58591: Java packages do not appear to keep a reference to their inputs

[Maxim Cournoyer]

Hi,

Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:

[...]

> A nice read is [0], which mentions the existence of a 'Class-Path' main
> attribute that can go in the manifest file.  If using unpacked jars
> works the same as .jars (which are just zip files) for Java, then we
> could not only have dependency correctly referenced and loaded via
> 'Class-Path', but also the grafting mechanism would work, since the
> paths would appear in clear (not obfuscated due to zip compression).

Ugh, Class-Path only accepts relative path, not absolute paths:

   The location of the JAR file or directory represented by this entry
   is contained within the containing directory of the context JAR. Use
   of "../" to navigate to the parent directory is not permitted, except
   for the case when the context JAR is loaded from the file system.

Perhaps we could patch Java so that it's loader is more adapted for our
use case, or extend its manifest with a Guix-specific Guix-Class-Path
section that'd allow for absolute paths.

> [0] https://docs.oracle.com/en/java/javase/19/docs/specs/jar/jar.html

-- 
Thanks,
Maxim

bug#58591: Java packages do not appear to keep a reference to their inputs

[Tobias Geerinckx-Rice via Bug reports for GNU Guix]

[-- Attachment #1: Type: text/plain, Size: 184 bytes --]

Tobias Geerinckx-Rice via Bug reports for GNU Guix 写道：
> Groan.  Which package(s) compress .jars?

OK, found one: openjdk@16.0.1's /lib/jrt-fs.jar.

Kind regards,

T G-R

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 247 bytes --]

bug#58591: Java packages do not appear to keep a reference to their inputs

[Maxim Cournoyer]

Hi Tobias!

Tobias Geerinckx-Rice <me@tobias.gr> writes:

> Hi Maxim,
>
> Maxim Cournoyer 写道：
>> not obfuscated due to zip compression
>
> Groan.  Which package(s) compress .jars?

Oh, aren't they all?  I hadn't realized .jar compression was optional.
I believe our ant-build-system produces compressed jars; in fact it uses
'zip' directly to pack them.

-- 
Thanks,
Maxim

bug#58591: Java packages do not appear to keep a reference to their inputs

[Maxim Cournoyer]

Hello,

Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:

> Tobias Geerinckx-Rice <me@tobias.gr> writes:

[...]

>> Groan.  Which package(s) compress .jars?
>
> Oh, aren't they all?  I hadn't realized .jar compression was optional.

Actually, reading [0] again, it seems a JAR *is* a zip archive, so
cannot be either compressed or uncompressed.

[0]  https://docs.oracle.com/en/java/javase/19/docs/specs/jar/jar.html

-- 
Thanks,
Maxim

bug#58591: Java packages do not appear to keep a reference to their inputs

[Julien Lepiller]

[-- Attachment #1: Type: text/plain, Size: 1442 bytes --]

Hi, replying to a few emails at once.

The ant-build-system uses zip -0 to produce an uncompressed archive. By default, jar produces a compressed one, so there's a repack phase for that:
 http://git.savannah.nongnu.org/cgit/guix.git/tree/guix/build/ant-build-system.scm#n226

Embedding the classpath in the manifest is possible but would not have the expected effect. That's because a line in the manifest cannot exceed 72 bytes (see "line length" in https://docs.oracle.com/javase/8/docs/technotes/guides/jar/jar.html#Notes_on_Manifest_and_Signature_Files), so the classpath will look like:

Class-Path: ../../../1234567891011
 1213141516/share/java/foo.jar

Although java would read that fine, the grafter will not see it, nor be able to graft foo in a meaningful manner: java would still use the ungrafted version even if another file references foo.

Le 18 octobre 2022 16:56:01 GMT+02:00, Maxim Cournoyer <maxim.cournoyer@gmail.com> a écrit :
>Hello,
>
>Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:
>
>> Tobias Geerinckx-Rice <me@tobias.gr> writes:
>
>[...]
>
>>> Groan.  Which package(s) compress .jars?
>>
>> Oh, aren't they all?  I hadn't realized .jar compression was optional.
>
>Actually, reading [0] again, it seems a JAR *is* a zip archive, so
>cannot be either compressed or uncompressed.
>
>[0]  https://docs.oracle.com/en/java/javase/19/docs/specs/jar/jar.html
>
>-- 
>Thanks,
>Maxim

[-- Attachment #2: Type: text/html, Size: 2478 bytes --]

bug#58591: Java packages do not appear to keep a reference to their inputs

[Maxim Cournoyer]

Hello,

Julien Lepiller <julien@lepiller.eu> writes:

> Hi, replying to a few emails at once.
>
> The ant-build-system uses zip -0 to produce an uncompressed
> archive. By default, jar produces a compressed one, so there's a
> repack phase for that:
>  http://git.savannah.nongnu.org/cgit/guix.git/tree/guix/build/ant-build-system.scm#n226

Ah, I had missed the -0 == uncompressed part.  Thank you.

> Embedding the classpath in the manifest is possible but would not have
> the expected effect. That's because a line in the manifest cannot
> exceed 72 bytes (see "line length" in
> https://docs.oracle.com/javase/8/docs/technotes/guides/jar/jar.html#Notes_on_Manifest_and_Signature_Files),
> so the classpath will look like:
>
> Class-Path: ../../../1234567891011
>  1213141516/share/java/foo.jar

Although it looks like the 72 bytes line width limitation may has to do
with binary data:

   Binary data of any form is represented as base64. Continuations are
   required for binary data which causes line length to exceed 72
   bytes. Examples of binary data are digests and signatures.

Worth a try in my opinion (I'm giving it a shot as I write this).

Thanks for the explanations!

Maxim