bug#29700: Java packages need to retain references

bug#29700: Java packages need to retain references

[Ricardo Wurmus]

I installed the “axoloti-patcher” package today after “guix gc” and
noticed that not all dependencies are available at runtime.  Upon
running the “Axoloti” command I got this error:

    Exception in thread "AWT-EventQueue-0" java.lang.UnsatisfiedLinkError: Can't load library: /gnu/store/nrx5nmwynmsl7aqxdpgwpsm8ff9b0h4d-libusb4java-0-1.396d642a5/lib/libusb4java.so

The store item does not exist.

The problem here is that jar archives are compressed and thus any
references they contain are thus hidden from the scanner.

I don’t know if it is feasible to disable compression for jar files, or
if we should install a “.guix-dependencies” file that contains plain
text references.

I’ll investigate disabling compression first.  Looks like we can do this
in the strip-jar-timestamps phase where we use “zip” to repack the jar.
Adding “-0” to the arguments should be enough to make this work.

I’ll send a patch soon.

--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net

bug#29700: [PATCH] guix: ant-build-system: Do not compress jars.

[Ricardo Wurmus]

Fixes <https://bugs.gnu.org/29700>.

* guix/build/ant-build-system.scm (strip-jar-timestamps): Do not compress jar
when repacking.
---
 guix/build/ant-build-system.scm | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/guix/build/ant-build-system.scm b/guix/build/ant-build-system.scm
index a440daf05..6ce813a00 100644
--- a/guix/build/ant-build-system.scm
+++ b/guix/build/ant-build-system.scm
@@ -197,9 +197,12 @@ repack them.  This is necessary to ensure that archives are reproducible."
            ;; first.
            (with-directory-excursion dir
              (let* ((files (find-files "." ".*" #:directories? #t))
+                    ;; To ensure that the reference scanner can detect all
+                    ;; store references in the jars we disable compression
+                    ;; with the "-0" option.
                     (command (if (file-exists? manifest)
-                                 `("zip" "-X" ,jar ,manifest ,@files)
-                                 `("zip" "-X" ,jar ,@files))))
+                                 `("zip" "-0" "-X" ,jar ,manifest ,@files)
+                                 `("zip" "-0" "-X" ,jar ,@files))))
                (unless (zero? (apply system* command))
                  (error "'zip' failed"))))
            (utime jar 0 0)
-- 
2.15.0

bug#29700: [PATCH] guix: ant-build-system: Do not compress jars.

[Danny Milosavljevic]

LGTM!

bug#29700: Java packages need to retain references

[Ricardo Wurmus]

Hi Gábor,

The manifest file would not be sufficient.  Take the example of
“java-usb4java”.  It provides a file “share/java/usb4java.jar”, which
contains this manifest:

--8<---------------cut here---------------start------------->8---
Manifest-Version: 1.0
Created-By: 1.7.0_161 (Oracle Corporation)

--8<---------------cut here---------------end--------------->8---

When you open the Loader.class file that is included in the archive you
will notice that it contains this string:

    /gnu/store/iwjqaqr98daqbgwswqi5fi4w891bqh7d-libusb4java-0-1.396d642a5/lib/libusb4java.so

This reference is currently dropped because it cannot be detected in a
compressed archive.  The reference is not contained in the manifest, and
at build time we could only guess which references should be added to
the manifest.

Unless we extend the reference scanner to deal with compressed files I
think that disabling compression in jars is the best approach.

I’ll push this to master.

--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net

bug#29700: Java packages need to retain references

[Gábor Boskovits]

[-- Attachment #1: Type: text/plain, Size: 1649 bytes --]

Do we know the impact of disabling compression?
I don't think it is really a big problem.
You are right with this, I'm still a bit stuck in another mindset, where we
usually know, that we have only one version of a package...
If we later decide that compression is desired we can still go with a text
file with references.
Anyway, do you think that extending the reference scanner would be a good
idea?


2017-12-14 23:59 GMT+01:00 Ricardo Wurmus <rekado@elephly.net>:

> Hi Gábor,
>
> The manifest file would not be sufficient.  Take the example of
> “java-usb4java”.  It provides a file “share/java/usb4java.jar”, which
> contains this manifest:
>
> --8<---------------cut here---------------start------------->8---
> Manifest-Version: 1.0
> Created-By: 1.7.0_161 (Oracle Corporation)
>
> --8<---------------cut here---------------end--------------->8---
>
> When you open the Loader.class file that is included in the archive you
> will notice that it contains this string:
>
>     /gnu/store/iwjqaqr98daqbgwswqi5fi4w891bqh7d-libusb4java-0-1.396d642a5/
> lib/libusb4java.so
>
> This reference is currently dropped because it cannot be detected in a
> compressed archive.  The reference is not contained in the manifest, and
> at build time we could only guess which references should be added to
> the manifest.
>
> Unless we extend the reference scanner to deal with compressed files I
> think that disabling compression in jars is the best approach.
>
> I’ll push this to master.
>
> --
> Ricardo
>
> GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
> https://elephly.net
>
>
>

[-- Attachment #2: Type: text/html, Size: 2223 bytes --]

bug#29700: Java packages need to retain references

[Ricardo Wurmus]

Hi Gábor,

> If we later decide that compression is desired we can still go with a text
> file with references.

This wouldn’t always be feasible, because it would require *manual*
reference tracking.  The packager would need to know what references
need to be retained and then ensure that they are part of that text
file.

> Anyway, do you think that extending the reference scanner would be a good
> idea?

We had the same problem with gzipped R files and zipped Python things.
For the R files a more advanced reference scanner would not work,
because the files just include a gzipped section; it’s not the whole
file that’s simply gzipped.  Detecting sections is too hard.

But for Python and Jars it might just be enough.

[PS: please don’t top-post.]

--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net

bug#29700: Java packages need to retain references

[Ludovic Courtès]

Ricardo Wurmus <rekado@elephly.net> skribis:

> I installed the “axoloti-patcher” package today after “guix gc” and
> noticed that not all dependencies are available at runtime.  Upon
> running the “Axoloti” command I got this error:
>
>     Exception in thread "AWT-EventQueue-0" java.lang.UnsatisfiedLinkError: Can't load library: /gnu/store/nrx5nmwynmsl7aqxdpgwpsm8ff9b0h4d-libusb4java-0-1.396d642a5/lib/libusb4java.so
>
> The store item does not exist.
>
> The problem here is that jar archives are compressed and thus any
> references they contain are thus hidden from the scanner.
>
> I don’t know if it is feasible to disable compression for jar files, or
> if we should install a “.guix-dependencies” file that contains plain
> text references.

Disabling compression was the right choice because that also allows
grafts to work correctly.

Ludo’.

bug#29705: Java packages need to retain references

[Ricardo Wurmus]

I installed the “axoloti-patcher” package today after “guix gc” and
noticed that not all dependencies are available at runtime.  Upon
running the “Axoloti” command I got this error:

    Exception in thread "AWT-EventQueue-0" java.lang.UnsatisfiedLinkError: Can't load library: /gnu/store/nrx5nmwynmsl7aqxdpgwpsm8ff9b0h4d-libusb4java-0-1.396d642a5/lib/libusb4java.so

The store item does not exist.

The problem here is that jar archives are compressed and thus any
references they contain are thus hidden from the scanner.

I don’t know if it is feasible to disable compression for jar files, or
if we should install a “.guix-dependencies” file that contains plain
text references.

I’ll investigate disabling compression first.

--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net

bug#29705: Java packages need to retain references

[Gábor Boskovits]

[-- Attachment #1: Type: text/plain, Size: 1093 bytes --]

Would it be enough if we added the uncompressed manifest?
Is there any drawback to that?


2017-12-13 23:28 GMT+01:00 Ricardo Wurmus <rekado@fhain.coderdojo.berlin>:

> I installed the “axoloti-patcher” package today after “guix gc” and
> noticed that not all dependencies are available at runtime.  Upon
> running the “Axoloti” command I got this error:
>
>     Exception in thread "AWT-EventQueue-0" java.lang.UnsatisfiedLinkError:
> Can't load library: /gnu/store/nrx5nmwynmsl7aqxdpgwpsm8ff9b0h
> 4d-libusb4java-0-1.396d642a5/lib/libusb4java.so
>
> The store item does not exist.
>
> The problem here is that jar archives are compressed and thus any
> references they contain are thus hidden from the scanner.
>
> I don’t know if it is feasible to disable compression for jar files, or
> if we should install a “.guix-dependencies” file that contains plain
> text references.
>
> I’ll investigate disabling compression first.
>
> --
> Ricardo
>
> GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
> https://elephly.net
>
>
>
>
>

[-- Attachment #2: Type: text/html, Size: 1573 bytes --]

bug#29700: Java packages need to retain references

[Ricardo Wurmus]

Gábor Boskovits <boskovits@gmail.com> writes:

> Would it be enough if we added the uncompressed manifest?
> Is there any drawback to that?

I will test that.  I’m not sure if the manifest is really sufficient.

--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net