"jar" tool and and reproducible builds - how?

"jar" tool and and reproducible builds - how?

[Danny Milosavljevic]

[-- Attachment #1.1: Type: text/plain, Size: 748 bytes --]

Hi,

I'm trying to package nesc which has some small tools written in Java.
It compiles them using "javac" and then packs the result using "jar".

If I want

  guix build --rounds=2 nesc

to pass, how do I best proceed?

I tried

             (substitute* "tools/Makefile.in"
              ((" jar cf \\.\\./nesc.jar")
               " zip -0 -X ../nesc.jar META-INF/MANIFEST.MF "))

but that doesn't seem to be enough.

WIP attached.

Furthermore, I think it would be nice to patch our "jar" tool in icedtea:jdk
so that it provides an option to disable timestamps (not put the timestamp
field into the generated ZIP file in the first place).  I've thus unpacked
icedtea but I can't find where the source code to "jar" is.

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1.2: nesc.scm --]
[-- Type: text/x-scheme, Size: 1664 bytes --]

(define-module (wip nesc)
  #:use-module ((guix licenses) #:prefix license:)
  #:use-module (guix build-system gnu)
  #:use-module (guix download)
  #:use-module (guix packages)
  #:use-module (gnu packages compression)
  #:use-module (gnu packages java)
  #:use-module (gnu packages perl))

(define-public nesc
  (package
    (name "nesc")
    (version "1.3.4")
    (source (origin
              (method url-fetch)
              (uri (string-append "mirror://sourceforge/nescc/nescc/v" version
                                  "/nesc-" version ".tar.gz"))
              (sha256
               (base32
                "0n4mjnm6q96s8pkdq2n9z6cxf5xpy2386cfn30wm4if9gdwhc3w7"))))
    (build-system gnu-build-system)
    (native-inputs
     `(("icedtea" ,icedtea "jdk") ; for tools/java
       ("perl" ,perl)
       ("zip" ,zip)))
    (arguments
     `(#:phases
       (modify-phases %standard-phases
         (add-after 'unpack 'make-jar-reproducible
           (lambda _
             ;; Make JAR file "lib/ncc/nesc.jar" reproducible.
             ;(substitute* "tools/Makefile.in"
             ; ((" jar") " zip -0 -X META-INF/MANIFEST.MF "))
             (substitute* "tools/Makefile.in"
              ((" jar cf \\.\\./nesc.jar")
               " zip -0 -X ../nesc.jar META-INF/MANIFEST.MF "))
             #t)))))
    (synopsis "nesc compiler")
    (description "This package provides a C-like programming language with
small extensions to C, notably a module system (including interfaces),
concurrency and platform-independent networking.")
    (home-page "http://nescc.sourceforge.net/")
    (license license:gpl2)))

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: "jar" tool and and reproducible builds - how?

[Gábor Boskovits]

[-- Attachment #1: Type: text/plain, Size: 1192 bytes --]

Hello Danny,

Danny Milosavljevic <dannym@scratchpost.org> ezt írta (időpont: 2019. nov.
4., H, 18:18):

> Hi,
>
> I'm trying to package nesc which has some small tools written in Java.
> It compiles them using "javac" and then packs the result using "jar".
>
> If I want
>
>   guix build --rounds=2 nesc
>
> to pass, how do I best proceed?
>
> I tried
>
>              (substitute* "tools/Makefile.in"
>               ((" jar cf \\.\\./nesc.jar")
>                " zip -0 -X ../nesc.jar META-INF/MANIFEST.MF "))
>
> but that doesn't seem to be enough.
>
>
What diff remains there? There might be more jars or jmods inside...
Most of the trickery should be in ant build system.


> WIP attached.
>
> Furthermore, I think it would be nice to patch our "jar" tool in
> icedtea:jdk
> so that it provides an option to disable timestamps (not put the timestamp
> field into the generated ZIP file in the first place).  I've thus unpacked
> icedtea but I can't find where the source code to "jar" is.
>

I don't know right now, but have a look later.

Best regards,
g_bor
-- 
OpenPGP Key Fingerprint: 7988:3B9F:7D6A:4DBF:3719:0367:2506:A96C:CF63:0B21

[-- Attachment #2: Type: text/html, Size: 1988 bytes --]

Re: "jar" tool and and reproducible builds - how?

[Danny Milosavljevic]

[-- Attachment #1: Type: text/plain, Size: 689 bytes --]

Hi Gábor,

> What diff remains there? There might be more jars or jmods inside...
> Most of the trickery should be in ant build system.

This package does not use ant-build-system or any other java build system since
there are only very few Java tools in nesc (I'm thinking of disabling those,
too).

diffoscope states only the following, via "zipinfo -v" on "lib/ncc/nesc.jar":

 -  file last modified on (DOS date/time):          2019 Nov 4 17:10:26
 +  file last modified on (DOS date/time):          2019 Nov 4 17:10:56

It's repeated a lot of times.

No other files but "lib/ncc/nesc.jar" are affected.

> I don't know right now, but have a look later.

Thanks!

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: "jar" tool and and reproducible builds - how?

[Gábor Boskovits]

[-- Attachment #1: Type: text/plain, Size: 1272 bytes --]

Hello,

Danny Milosavljevic <dannym@scratchpost.org> ezt írta (időpont: 2019. nov.
5., K, 10:49):

> Hi Gábor,
>
> > What diff remains there? There might be more jars or jmods inside...
> > Most of the trickery should be in ant build system.
>
> This package does not use ant-build-system or any other java build system
> since
> there are only very few Java tools in nesc (I'm thinking of disabling
> those,
> too).
>
> diffoscope states only the following, via "zipinfo -v" on
> "lib/ncc/nesc.jar":
>
>  -  file last modified on (DOS date/time):          2019 Nov 4 17:10:26
>  +  file last modified on (DOS date/time):          2019 Nov 4 17:10:56
>

This looks like a jar timestamps issue. Adding a phase similar to
strip-jar-timestamps in ant-build-system should work.
It extracts the jar to a temporary directory, resets file timestamps, and
then repacks it, and resets the timestamp on the archive.
Could you check if that works for you?


>
> It's repeated a lot of times.
>
> No other files but "lib/ncc/nesc.jar" are affected.
> apply git send email
> > I don't know right now, but have a look later.
>
> Thanks!
>

Best regards,
g_bor
-- 
OpenPGP Key Fingerprint: 7988:3B9F:7D6A:4DBF:3719:0367:2506:A96C:CF63:0B21

[-- Attachment #2: Type: text/html, Size: 1980 bytes --]

Re: "jar" tool and and reproducible builds - how?

[Danny Milosavljevic]

[-- Attachment #1: Type: text/plain, Size: 1381 bytes --]

Hi Gábor,

> This looks like a jar timestamps issue. Adding a phase similar to
> strip-jar-timestamps in ant-build-system should work.
> It extracts the jar to a temporary directory, resets file timestamps, and
> then repacks it, and resets the timestamp on the archive.
> Could you check if that works for you?

Yes, the following works and makes it reproducible:

    (arguments
     `(#:imported-modules
       ((guix build utils)
        (guix build ant-build-system)
        (guix build gremlin)
        (guix build syscalls)
        (guix elf)
        (guix build gnu-build-system))
       #:phases
       (modify-phases %standard-phases
         (add-after 'install 'strip-jar-timestamps
           (assoc-ref (@ (guix build ant-build-system) %standard-phases)
                      'strip-jar-timestamps)))))

But I'd prefer if our "jar" tool did that on its own if SOURCE_DATE_EPOCH
is set--I don't think the code above is maintainable.

The following code does not work (doesn't find ant-build-system):

    (arguments
     `(#:modules
       ((guix build gnu-build-system)
        ((guix build ant-build-system) #:prefix ant:)
        (guix build utils))
       #:phases
       (modify-phases %standard-phases
         (add-after 'install 'strip-jar-timestamps
           (assoc-ref ant:%standard-phases 'strip-jar-timestamps)))))

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: "jar" tool and and reproducible builds - how?

[Gábor Boskovits]

[-- Attachment #1: Type: text/plain, Size: 1808 bytes --]

Hello,

Danny Milosavljevic <dannym@scratchpost.org> ezt írta (időpont: 2019. nov.
5., Ke 16:09):

> Hi Gábor,
>
> > This looks like a jar timestamps issue. Adding a phase similar to
> > strip-jar-timestamps in ant-build-system should work.
> > It extracts the jar to a temporary directory, resets file timestamps, and
> > then repacks it, and resets the timestamp on the archive.
> > Could you check if that works for you?
>
> Yes, the following works and makes it reproducible:
>
>     (arguments
>      `(#:imported-modules
>        ((guix build utils)
>         (guix build ant-build-system)
>         (guix build gremlin)
>         (guix build syscalls)
>         (guix elf)
>         (guix build gnu-build-system))
>        #:phases
>        (modify-phases %standard-phases
>          (add-after 'install 'strip-jar-timestamps
>            (assoc-ref (@ (guix build ant-build-system) %standard-phases)
>                       'strip-jar-timestamps)))))
>
That is great.

>
> But I'd prefer if our "jar" tool did that on its own if SOURCE_DATE_EPOCH
> is set--I don't think the code above is maintainable.
>
Agreed. Could you open a wishlist bug for that, so it is not forgotten.

>
> The following code does not work (doesn't find ant-build-system):
>
>     (arguments
>      `(#:modules
>        ((guix build gnu-build-system)
>         ((guix build ant-build-system) #:prefix ant:)
>         (guix build utils))
>        #:phases
>        (modify-phases %standard-phases
>          (add-after 'install 'strip-jar-timestamps
>            (assoc-ref ant:%standard-phases 'strip-jar-timestamps)))))
>
I believe this strip-jar-timestamps code should be moved to the java utils
module, so that it can be easier to reuse...

Best regards,g_bor

>

[-- Attachment #2: Type: text/html, Size: 2909 bytes --]

Re: "jar" tool and and reproducible builds - how?

[Hartmut Goebel]

Am 04.11.19 um 18:18 schrieb Danny Milosavljevic:
> I tried
>
>              (substitute* "tools/Makefile.in"
>               ((" jar cf \\.\\./nesc.jar")
>                " zip -0 -X ../nesc.jar META-INF/MANIFEST.MF "))
>
> but that doesn't seem to be enough.

The ant-build-system contains phases dealing with this. Have a look at
guix/buid/ant-build-system.scm, near the bottom. Noteble this runs
"strip-jar-timestamps twice!


-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |