Re: JARs and reference scanning

[Chris Marusich <cmmarusich@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 3370 bytes --]

Hartmut Goebel <h.goebel@crazy-compilers.com> writes:

> Am 24.04.2017 um 00:57 schrieb Chris Marusich:
>> Chris Marusich <cmmarusich@gmail.com> writes:
>>
>>> Speaking of JAR files, shouldn't we try to avoid them entirely?  My
>>> understanding is that they are compressed files, which means the Guix
>>> daemon won't be able to scan them for references.  I don't know if it's
>>> easy to use Maven to build a project without putting the build output
>>> into a JAR file, though.
>> For the record, I looked into this a little more.  I was mistaken: JAR
>> files are not necessarily compressed.  In fact, it seems [1][2] that it
>> should always be possible to make un-compressed JARs.  So, perhaps the
>> Guix daemon will not have trouble scanning JARs for references, after
>> all.  (Whether or not any references will actually be retained in the
>> JARs produced by Maven is another question; I don't know the answer.)
>
> I have to admit that I do not know at all how the reference scanning and
> dependency-tracking in the store works.

As I understand it, the mechanism used by the Guix daemon (and the Nix
daemon) for scanning references doesn't work when the output of a
derivation is scrambled in some way (e.g. compressed).  Therefore, if we
use JAR files, they should not be compressed.

> Regarding the jar-files ny understanding is:
>
> - JAR-files are Zip-files with additional data (as Ricardo already stated)
> - Zip-files can be used to store data without compressions (I assume
> this is what you are refering to).
> - My experience is that the contents of the JAR file can also be
> *unpacked* into the file-system, so not archives would be needed. Some
> Java guy might know better. I'm not sure it this is desired at all, though.
> - My understanding is that Java normally does not have any reference
> from one package (or jar-file) to another one. There is no such thing
> like "rpath" but is more like Python or Perl where the garbage collector
> AFAIK can not track references either.

I don't think it's necessary to unpack them, as long as we can ensure
they are not compressed.

> - According to [3, 2] the MANIFEST.INF file *may* specify a Class-Path
> containing the relative paths to other Jar-files. If this would help
> we *could* add references here, but the entry-length is limited to
> 65353 bytes, so we might hit this limit with the long paths of the
> store.  - Fedora forbids to use this Class-Path entry in MANIFEST
> files [1].  - If it helps the garbage collector, we could add some
> ".dependencies" file alongside each Java package. But we don't do this
> for Python or Perl, either.

That's an interesting idea.  This would certainly make it clear which
Java packages depend on which other packages (e.g., for garbage
collection purposes), but this wouldn't by itself help Java (or Python,
or Perl, etc.) actually find the classes.  Some other stuff (propagation
of inputs like we do with Python libraries, a Guix-managed CLASSPATH, a
custom ClassLoader, etc.) would probably be needed to ensure that Java
would be able to find the classes.

Thank you for your work on this.  It's easy for me to just give
opinions, but you've actually done the real work.  If I can find more
time, I will try to look at what you've submitted to the list and help.

-- 
Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]