java: How to patch a jar-sources

java: How to patch a jar-sources

[Björn Höfling]

[-- Attachment #1: Type: text/plain, Size: 2544 bytes --]

Hi Guix,

I have a problem combining ".jar" sources with patches.

Here is my sources definition:

(define-public java-ecj
  (package (inherit java-ecj-3)
           (version "4.6.3")
           (source (origin
                     (method url-fetch)
                     (uri (string-append "http://archive.eclipse.org/eclipse/downloads/drops4/R-"
                                         version
                                         "-201703010400/ecjsrc-"
                                         version
                                         ".jar"))
                     (patches
                      (search-patches "java-ecj-buildxml-fix-manifest.patch"
                                      "java-ecj-include-props.patch"))
                     (sha256
                      (base32
                       "11cfgsdgznja1pvlxkjbqykxd7pcd5655vkm7s44xmahmap15gpl"))))
[...]

This worked pretty well until I added the patches:

./pre-inst-env  guix build java-ecj -K
;;; note: source file /home/bjoern/guix/wt/qt/gnu/packages/java.scm
;;;       newer than compiled /home/bjoern/guix/wt/qt/gnu/packages/java.go
building /gnu/store/wppz7bix1r4mhddjz6k4alfnxkm919gq-ecjsrc-4.6.3.tar.xz.drv...
/gnu/store/bl3pxxj6frg0dww8pj5dvh2d1akwvj47-tar-1.30/bin/tar: This does not look like a tar archive
/gnu/store/bl3pxxj6frg0dww8pj5dvh2d1akwvj47-tar-1.30/bin/tar: Skipping to next header
/gnu/store/bl3pxxj6frg0dww8pj5dvh2d1akwvj47-tar-1.30/bin/tar: Exiting with failure status due to previous errors
Backtrace:
           2 (primitive-load "/gnu/store/lg0dlhwf4mfz82697bvfdwi7d86?")
In ice-9/eval.scm:
    619:8  1 (_ #f)
In guix/build/utils.scm:
    616:6  0 (invoke _ . _)

guix/build/utils.scm:616:6: In procedure invoke:
Throw to key `srfi-34' with args `(#<condition &invoke-error [program: "/gnu/store/bl3pxxj6frg0dww8pj5dvh2d1akwvj47-tar-1.30/bin/tar" arguments: ("xvf" "/gnu/store/1vf8lxrnkgxw0zl8i2m8rf7jxcjg9f5b-ecjsrc-4.6.3.jar") exit-status: 2 term-signal: #f stop-signal: #f] 56dd40>)'.
note: keeping build directory `/tmp/guix-build-ecjsrc-4.6.3.tar.xz.drv-1'
builder for `/gnu/store/wppz7bix1r4mhddjz6k4alfnxkm919gq-ecjsrc-4.6.3.tar.xz.drv' failed with exit code 1
build of /gnu/store/wppz7bix1r4mhddjz6k4alfnxkm919gq-ecjsrc-4.6.3.tar.xz.drv failed


It looks like guix now wants to extract the jar-sources with "tar", no
longer with "jar". How can I tell Guix this is a jar-file to extract
from? Do I need to do that manually in a snippet? Any examples?

Thanks,

Björn


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: java: How to patch a jar-sources

[Gábor Boskovits]

[-- Attachment #1: Type: text/plain, Size: 3025 bytes --]

Hello Björn,



Björn Höfling <bjoern.hoefling@bjoernhoefling.de> ezt írta (időpont: 2019.
ápr. 15., H 8:03):

> Hi Guix,
>
> I have a problem combining ".jar" sources with patches.
>
> Here is my sources definition:
>
> (define-public java-ecj
>   (package (inherit java-ecj-3)
>            (version "4.6.3")
>            (source (origin
>                      (method url-fetch)
>                      (uri (string-append "
> http://archive.eclipse.org/eclipse/downloads/drops4/R-"
>                                          version
>                                          "-201703010400/ecjsrc-"
>                                          version
>                                          ".jar"))
>                      (patches
>                       (search-patches
> "java-ecj-buildxml-fix-manifest.patch"
>                                       "java-ecj-include-props.patch"))
>                      (sha256
>                       (base32
>
>  "11cfgsdgznja1pvlxkjbqykxd7pcd5655vkm7s44xmahmap15gpl"))))
> [...]
>
> This worked pretty well until I added the patches:
>
> ./pre-inst-env  guix build java-ecj -K
> ;;; note: source file /home/bjoern/guix/wt/qt/gnu/packages/java.scm
> ;;;       newer than compiled /home/bjoern/guix/wt/qt/gnu/packages/java.go
> building
> /gnu/store/wppz7bix1r4mhddjz6k4alfnxkm919gq-ecjsrc-4.6.3.tar.xz.drv...
> /gnu/store/bl3pxxj6frg0dww8pj5dvh2d1akwvj47-tar-1.30/bin/tar: This does
> not look like a tar archive
> /gnu/store/bl3pxxj6frg0dww8pj5dvh2d1akwvj47-tar-1.30/bin/tar: Skipping to
> next header
> /gnu/store/bl3pxxj6frg0dww8pj5dvh2d1akwvj47-tar-1.30/bin/tar: Exiting with
> failure status due to previous errors
> Backtrace:
>            2 (primitive-load "/gnu/store/lg0dlhwf4mfz82697bvfdwi7d86?")
> In ice-9/eval.scm:
>     619:8  1 (_ #f)
> In guix/build/utils.scm:
>     616:6  0 (invoke _ . _)
>
> guix/build/utils.scm:616:6: In procedure invoke:
> Throw to key `srfi-34' with args `(#<condition &invoke-error [program:
> "/gnu/store/bl3pxxj6frg0dww8pj5dvh2d1akwvj47-tar-1.30/bin/tar" arguments:
> ("xvf" "/gnu/store/1vf8lxrnkgxw0zl8i2m8rf7jxcjg9f5b-ecjsrc-4.6.3.jar")
> exit-status: 2 term-signal: #f stop-signal: #f] 56dd40>)'.
> note: keeping build directory `/tmp/guix-build-ecjsrc-4.6.3.tar.xz.drv-1'
> builder for
> `/gnu/store/wppz7bix1r4mhddjz6k4alfnxkm919gq-ecjsrc-4.6.3.tar.xz.drv'
> failed with exit code 1
> build of
> /gnu/store/wppz7bix1r4mhddjz6k4alfnxkm919gq-ecjsrc-4.6.3.tar.xz.drv failed
>
>
> It looks like guix now wants to extract the jar-sources with "tar", no
> longer with "jar". How can I tell Guix this is a jar-file to extract
> from? Do I need to do that manually in a snippet? Any examples?
>
I have not actually done this, as jar files are usually not sources. Do we
have the source this file was generated from?
Also what we do in the build system is use zip and unzip for extracting jar
files.

>
> Thanks,
>
> Björn
>
Best regards,
g_bor

>

[-- Attachment #2: Type: text/html, Size: 4321 bytes --]

Re: java: How to patch a jar-sources

[Björn Höfling]

[-- Attachment #1: Type: text/plain, Size: 2462 bytes --]

On Mon, 15 Apr 2019 14:26:41 +0200
Gábor Boskovits <boskovits@gmail.com> wrote:

> Hello Björn,
> 
> 
> 
> Björn Höfling <bjoern.hoefling@bjoernhoefling.de> ezt írta (időpont:
> 2019. ápr. 15., H 8:03):
> 
> > Hi Guix,
> >
> > I have a problem combining ".jar" sources with patches.
> >
> > Here is my sources definition:
> >
> > (define-public java-ecj
> >   (package (inherit java-ecj-3)
> >            (version "4.6.3")
> >            (source (origin
> >                      (method url-fetch)
> >                      (uri (string-append "
> > http://archive.eclipse.org/eclipse/downloads/drops4/R-"
> >                                          version
> >                                          "-201703010400/ecjsrc-"
> >                                          version
> >                                          ".jar"))
[..]

> > It looks like guix now wants to extract the jar-sources with "tar",
> > no longer with "jar". How can I tell Guix this is a jar-file to
> > extract from? Do I need to do that manually in a snippet? Any
> > examples? 
> I have not actually done this, as jar files are usually not sources.
> Do we have the source this file was generated from?
> Also what we do in the build system is use zip and unzip for
> extracting jar files.

jar-files can contain sources. The main intended use for this is to show
the sources to the developer in the IDE (Eclipse), but we can also use
it to build from.

As an example, look at the java-jgit package. It downloads from Maven
central a ...-sources.jar.

In my example the ecjsrc-...jar also contains .java source files, the
compiled version with .class files is named without the "src" in the
name.

Going with zip is an option, this might be a work-around. But actually,
I don't want to do everything by hand:

I thought in general you can put "whatever" (supported) file-extension
you like into the "source" field and by extension of the URI it will
select the proper unpacker (i.e. using "tar xvf" for .tar files, "tar
xzvf" for .tar.gz", "unzip" for .zip, etc).

This seams to work when just using the ".jar" URI, without patches (cmp
the java-jgit example). Only after I added the patches, it was trying
to use "tar x..", which of cause will fail on the .jar file. If noone
has a direct solution, maybe it's time for me to get my fingers dirty
and look into the source-extracting code.

Thanks,

Björn

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: java: How to patch a jar-sources

[Danny Milosavljevic]

[-- Attachment #1: Type: text/plain, Size: 118 bytes --]

Hi Björn,

for the record, the same happened to me but I got distracted by a new shiny thing before I looked why.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: java: How to patch a jar-sources

[Danny Milosavljevic]

[-- Attachment #1: Type: text/plain, Size: 558 bytes --]

(define* (unpack #:key source #:allow-other-keys)
  "Unpack the jar archive SOURCE.  When SOURCE is not a jar archive fall back
to the default GNU unpack strategy."
  (if (string-suffix? ".jar" source)
      (begin
        (mkdir "src")
        (with-directory-excursion "src"
          (invoke "jar" "-xf" source))
        #t)
      ;; Use GNU unpack strategy for things that aren't jar archives.
      ((assq-ref gnu:%standard-phases 'unpack) #:source source)))

Hmm... maybe the patched source gets a random name that doesn't end in ".jar"


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: java: How to patch a jar-sources

[Björn Höfling]

[-- Attachment #1: Type: text/plain, Size: 4774 bytes --]

On Mon, 15 Apr 2019 17:19:53 +0200
Danny Milosavljevic <dannym@scratchpost.org> wrote:

> (define* (unpack #:key source #:allow-other-keys)
>   "Unpack the jar archive SOURCE.  When SOURCE is not a jar archive
> fall back to the default GNU unpack strategy."
>   (if (string-suffix? ".jar" source)
>       (begin
>         (mkdir "src")
>         (with-directory-excursion "src"
>           (invoke "jar" "-xf" source))
>         #t)
>       ;; Use GNU unpack strategy for things that aren't jar archives.
>       ((assq-ref gnu:%standard-phases 'unpack) #:source source)))

This is only for the case without patches.
 
> Hmm... maybe the patched source gets a random name that doesn't end
> in ".jar"

The problem is before: It occurs during unpacking: The .jar file is
getting handled with "tar xvf". I looked into the source-derivation and
found the pieces. I added this line:

diff --git a/guix/packages.scm b/guix/packages.scm
index c94a651f27..412dfcc04c 100644
--- a/guix/packages.scm
+++ b/guix/packages.scm
@@ -488,12 +488,13 @@ specifies modules in scope when evaluating SNIPPET."
   (define decompression-type
     (cond ((string-suffix? "gz" source-file-name)  "gzip")
           ((string-suffix? "Z" source-file-name)  "gzip")
           ((string-suffix? "bz2" source-file-name) "bzip2")
           ((string-suffix? "lz" source-file-name)  "lzip")
           ((string-suffix? "zip" source-file-name) "unzip")
+          ((string-suffix? "jar" source-file-name) "unzip")
           (else "xz")))


This brings me a step further: The .jar-file will now be extracted.
Unfortunately, we have this in line number 593:

            (let ((directory (first-file ".")))
              (format (current-error-port)
                      "source is under '~a'~%" directory)
              (chdir directory)

That means: It is expected that the tarball/zipfile/archive contains a
single directory under which the sources are contained. That seams to
be the case for all .tar.* and .zip archives used within Guix packages.

To verify that, I (randomly) picked the fcgi package and repacked the
tarball, such that sources are directly in the root folder. I then get
this expected error:

[..]
Win32/logdump.dsp
source is under 'acinclude.m4'
Backtrace:
           2 (primitive-load "/gnu/store/phlnb6vy2gqjn75pivpsajyf9mq?")
In ice-9/eval.scm:
    619:8  1 (_ #(#<directory (guile-user) 5ce140> "acinclude.m4"))
In unknown file:
           0 (chdir "acinclude.m4")

ERROR: In procedure chdir:
In procedure chdir: Not a directory


Unfortunately, .jar-source-files ARE organized in such a way that
sources are organized directly in the root folder of the jar. I
currently don't see a quick AND nice way to fix this.

Wait, I have this idea:

diff --git a/guix/packages.scm b/guix/packages.scm
index c94a651f27..ffd06de358 100644
--- a/guix/packages.scm
+++ b/guix/packages.scm
@@ -491,6 +491,7 @@ specifies modules in scope when evaluating SNIPPET."
           ((string-suffix? "bz2" source-file-name) "bzip2")
           ((string-suffix? "lz" source-file-name)  "lzip")
           ((string-suffix? "zip" source-file-name) "unzip")
+          ((string-suffix? "jar" source-file-name) "jar")
           (else "xz")))
 
   (define original-file-name
@@ -584,12 +585,16 @@ specifies modules in scope when evaluating SNIPPET."
                        (directory (string-drop base (+ 1 dash))))
                   (mkdir directory)
                   (copy-recursively #+source directory))
-                #+(if (string=? decompression-type "unzip")
+                #+(if (or
+                       (string=? decompression-type "unzip")
+                       (string=? decompression-type "jar"))
                       #~(invoke "unzip" #+source)
                       #~(invoke (string-append #+tar "/bin/tar")
                                 "xvf" #+source)))
 
-            (let ((directory (first-file ".")))
+            (let ((directory (if (string=? decompression-type "jar")
+                                 "."
+                                 (first-file "."))))
               (format (current-error-port)
                       "source is under '~a'~%" directory)
               (chdir directory)

But it fails with:

In unknown file:
   ?: 5 [primitive-load "/gnu/store/kg3pa52ydp3qjy41wgl0jcx3a98m82x9-guile-2.2.4.tar.xz-builder"]
In ice-9/eval.scm:
 411: 4 [eval # ()]
 399: 3 [eval # ()]
 387: 2 [eval # ()]
 393: 1 [eval #<memoized decompression-type> ()]
In unknown file:
   ?: 0 [memoize-variable-access! # #]

ERROR: In procedure memoize-variable-access!:
ERROR: Unbound variable: decompression-type

What's wrong here?

Björn

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: java: How to patch a jar-sources

[Gábor Boskovits]

Hello Björn,

Björn Höfling <bjoern.hoefling@bjoernhoefling.de> ezt írta (időpont:
2019. ápr. 16., K, 1:49):
>
> On Mon, 15 Apr 2019 17:19:53 +0200
> Danny Milosavljevic <dannym@scratchpost.org> wrote:
>
> > (define* (unpack #:key source #:allow-other-keys)
> >   "Unpack the jar archive SOURCE.  When SOURCE is not a jar archive
> > fall back to the default GNU unpack strategy."
> >   (if (string-suffix? ".jar" source)
> >       (begin
> >         (mkdir "src")
> >         (with-directory-excursion "src"
> >           (invoke "jar" "-xf" source))
> >         #t)
> >       ;; Use GNU unpack strategy for things that aren't jar archives.
> >       ((assq-ref gnu:%standard-phases 'unpack) #:source source)))
>
> This is only for the case without patches.
>
> > Hmm... maybe the patched source gets a random name that doesn't end
> > in ".jar"
>
> The problem is before: It occurs during unpacking: The .jar file is
> getting handled with "tar xvf". I looked into the source-derivation and
> found the pieces. I added this line:
>
> diff --git a/guix/packages.scm b/guix/packages.scm
> index c94a651f27..412dfcc04c 100644
> --- a/guix/packages.scm
> +++ b/guix/packages.scm
> @@ -488,12 +488,13 @@ specifies modules in scope when evaluating SNIPPET."
>    (define decompression-type
>      (cond ((string-suffix? "gz" source-file-name)  "gzip")
>            ((string-suffix? "Z" source-file-name)  "gzip")
>            ((string-suffix? "bz2" source-file-name) "bzip2")
>            ((string-suffix? "lz" source-file-name)  "lzip")
>            ((string-suffix? "zip" source-file-name) "unzip")
> +          ((string-suffix? "jar" source-file-name) "unzip")
>            (else "xz")))
>
>
> This brings me a step further: The .jar-file will now be extracted.
> Unfortunately, we have this in line number 593:
>
>             (let ((directory (first-file ".")))
>               (format (current-error-port)
>                       "source is under '~a'~%" directory)
>               (chdir directory)
>
> That means: It is expected that the tarball/zipfile/archive contains a
> single directory under which the sources are contained. That seams to
> be the case for all .tar.* and .zip archives used within Guix packages.
>
> To verify that, I (randomly) picked the fcgi package and repacked the
> tarball, such that sources are directly in the root folder. I then get
> this expected error:
>
> [..]
> Win32/logdump.dsp
> source is under 'acinclude.m4'
> Backtrace:
>            2 (primitive-load "/gnu/store/phlnb6vy2gqjn75pivpsajyf9mq?")
> In ice-9/eval.scm:
>     619:8  1 (_ #(#<directory (guile-user) 5ce140> "acinclude.m4"))
> In unknown file:
>            0 (chdir "acinclude.m4")
>
> ERROR: In procedure chdir:
> In procedure chdir: Not a directory
>
>
> Unfortunately, .jar-source-files ARE organized in such a way that
> sources are organized directly in the root folder of the jar. I
> currently don't see a quick AND nice way to fix this.
>
> Wait, I have this idea:
>
> diff --git a/guix/packages.scm b/guix/packages.scm
> index c94a651f27..ffd06de358 100644
> --- a/guix/packages.scm
> +++ b/guix/packages.scm
> @@ -491,6 +491,7 @@ specifies modules in scope when evaluating SNIPPET."
>            ((string-suffix? "bz2" source-file-name) "bzip2")
>            ((string-suffix? "lz" source-file-name)  "lzip")
>            ((string-suffix? "zip" source-file-name) "unzip")
> +          ((string-suffix? "jar" source-file-name) "jar")
>            (else "xz")))
>
>    (define original-file-name
> @@ -584,12 +585,16 @@ specifies modules in scope when evaluating SNIPPET."
>                         (directory (string-drop base (+ 1 dash))))
>                    (mkdir directory)
>                    (copy-recursively #+source directory))
> -                #+(if (string=? decompression-type "unzip")
> +                #+(if (or
> +                       (string=? decompression-type "unzip")
> +                       (string=? decompression-type "jar"))
>                        #~(invoke "unzip" #+source)
>                        #~(invoke (string-append #+tar "/bin/tar")
>                                  "xvf" #+source)))
>
> -            (let ((directory (first-file ".")))
> +            (let ((directory (if (string=? decompression-type "jar")
> +                                 "."
> +                                 (first-file "."))))
>                (format (current-error-port)
>                        "source is under '~a'~%" directory)
>                (chdir directory)
>
> But it fails with:
>
> In unknown file:
>    ?: 5 [primitive-load "/gnu/store/kg3pa52ydp3qjy41wgl0jcx3a98m82x9-guile-2.2.4.tar.xz-builder"]
> In ice-9/eval.scm:
>  411: 4 [eval # ()]
>  399: 3 [eval # ()]
>  387: 2 [eval # ()]
>  393: 1 [eval #<memoized decompression-type> ()]
> In unknown file:
>    ?: 0 [memoize-variable-access! # #]
>
> ERROR: In procedure memoize-variable-access!:
> ERROR: Unbound variable: decompression-type
>
> What's wrong here?
>

I don't see the whole code here, but I  guess that the last time use
decompression-type in a section under
different quoting rules. Is that possible?

Also, you might have a look at url-fetch/tarbomb, as something similar
must be done there.

> Björn

Best regards,
g_bor

Re: java: How to patch a jar-sources

[Björn Höfling]

[-- Attachment #1: Type: text/plain, Size: 680 bytes --]

On Tue, 16 Apr 2019 11:50:45 +0200
Gábor Boskovits <boskovits@gmail.com> wrote:

> Hello Björn,
> 
> Björn Höfling <bjoern.hoefling@bjoernhoefling.de> ezt írta (időpont:
> 2019. ápr. 16., K, 1:49):

> > ERROR: In procedure memoize-variable-access!:
> > ERROR: Unbound variable: decompression-type
> >
> > What's wrong here?
> >  
> 
> I don't see the whole code here, but I  guess that the last time use
> decompression-type in a section under
> different quoting rules. Is that possible?

Exactly. I was missing a "#+" (=ungexp-native). It currently rebuilds
the bootstrap path, if that goes on well, I will provide the patch.

Thanks all,

Björn

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: java: How to patch a jar-sources

[Björn Höfling]

[-- Attachment #1: Type: text/plain, Size: 85 bytes --]

Patch sent out as:

https://debbugs.gnu.org/cgi/bugreport.cgi?bug=35312

Björn

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]