bug#30104: ant-bootstrap@1.7.1 does not build reproducibly on core-updates

bug#30104: ant-bootstrap@1.7.1 does not build reproducibly on core-updates

[Chris Marusich]

[-- Attachment #1: Type: text/plain, Size: 1275 bytes --]

Hi,

At commit 1b321229f4653c5daa873813e24910789c0b2918 (i.e., the current
tip of the core-updates branch), ant-bootstrap@1.7.1 does not build
reproducibly.  This package is defined in gnu/packages/java.scm, but it
is not exported (i.e., it is used privately within the module).  Note
that according to 'guix refresh', currently 215 packages depend on this
package.

To verify that it doesn't build reproducibly, run this command:

  guix build --rounds=2 -e '(@@ (gnu packages java) ant-bootstrap)'

Here's the last bit of output I got:

--8<---------------cut here---------------start------------->8---
output ‘/gnu/store/cl2pxkxrasazj7hb7ykjh895vfj76jfy-ant-bootstrap-1.7.1’ of ‘/gnu/store/9p8yw4cjcb2529gz9l6x4d559xb1qvql-ant-bootstrap-1.7.1.drv’ differs from previous round
@ build-failed /gnu/store/9p8yw4cjcb2529gz9l6x4d559xb1qvql-ant-bootstrap-1.7.1.drv - 1 output ‘/gnu/store/cl2pxkxrasazj7hb7ykjh895vfj76jfy-ant-bootstrap-1.7.1’ of ‘/gnu/store/9p8yw4cjcb2529gz9l6x4d559xb1qvql-ant-bootstrap-1.7.1.drv’ differs from previous round
guix build: error: build failed: build of `/gnu/store/9p8yw4cjcb2529gz9l6x4d559xb1qvql-ant-bootstrap-1.7.1.drv' failed
--8<---------------cut here---------------end--------------->8---

-- 
Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

bug#30104: ant-bootstrap@1.7.1 does not build reproducibly on core-updates

[Gábor Boskovits]

[-- Attachment #1: Type: text/plain, Size: 1610 bytes --]

Could you please attach a diffoscope output to this?

I've found this in the notes database, does not sound too promising:
https://tests.reproducible-builds.org/debian/rb-pkg/unstable/amd64/ant.html.

2018-01-14 5:16 GMT+01:00 Chris Marusich <cmmarusich@gmail.com>:

> Hi,
>
> At commit 1b321229f4653c5daa873813e24910789c0b2918 (i.e., the current
> tip of the core-updates branch), ant-bootstrap@1.7.1 does not build
> reproducibly.  This package is defined in gnu/packages/java.scm, but it
> is not exported (i.e., it is used privately within the module).  Note
> that according to 'guix refresh', currently 215 packages depend on this
> package.
>
> To verify that it doesn't build reproducibly, run this command:
>
>   guix build --rounds=2 -e '(@@ (gnu packages java) ant-bootstrap)'
>
> Here's the last bit of output I got:
>
> --8<---------------cut here---------------start------------->8---
> output ‘/gnu/store/cl2pxkxrasazj7hb7ykjh895vfj76jfy-ant-bootstrap-1.7.1’
> of ‘/gnu/store/9p8yw4cjcb2529gz9l6x4d559xb1qvql-ant-bootstrap-1.7.1.drv’
> differs from previous round
> @ build-failed /gnu/store/9p8yw4cjcb2529gz9l6x4d559xb1qvql-ant-bootstrap-1.7.1.drv
> - 1 output ‘/gnu/store/cl2pxkxrasazj7hb7ykjh895vfj76jfy-ant-bootstrap-1.7.1’
> of ‘/gnu/store/9p8yw4cjcb2529gz9l6x4d559xb1qvql-ant-bootstrap-1.7.1.drv’
> differs from previous round
> guix build: error: build failed: build of `/gnu/store/
> 9p8yw4cjcb2529gz9l6x4d559xb1qvql-ant-bootstrap-1.7.1.drv' failed
> --8<---------------cut here---------------end--------------->8---
>
> --
> Chris
>

[-- Attachment #2: Type: text/html, Size: 2235 bytes --]

bug#30104: ant-bootstrap@1.7.1 does not build reproducibly on core-updates

[Ricardo Wurmus]

Gábor Boskovits <boskovits@gmail.com> writes:

> Could you please attach a diffoscope output to this?
>
> I've found this in the notes database, does not sound too promising:
> https://tests.reproducible-builds.org/debian/rb-pkg/unstable/amd64/ant.html.

We are not using GCJ.

-- 
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net

bug#30104: Difference seems to be caused by timestamp

[Gábor Boskovits]

[-- Attachment #1: Type: text/plain, Size: 229 bytes --]

This is due to the modification times of files included in the jar.
No other difference is listed in the diffoscope output.

How do we deal with this in other packages?
That fix should be migrated to apply for this package also.

[-- Attachment #2: Type: text/html, Size: 295 bytes --]

bug#30104: Difference seems to be caused by timestamp

[Gábor Boskovits]

[-- Attachment #1: Type: text/plain, Size: 454 bytes --]

Ok, we have this in guix/build/ant-builde-system.scm,
it is called strip-jar-timestamps.

This should be done also on ant-bootstrap.

2018-01-18 9:58 GMT+01:00 Gábor Boskovits <boskovits@gmail.com>:

> This is due to the modification times of files included in the jar.
> No other difference is listed in the diffoscope output.
>
> How do we deal with this in other packages?
> That fix should be migrated to apply for this package also.
>

[-- Attachment #2: Type: text/html, Size: 829 bytes --]

bug#30104: [PATCH] gnu: ant-bootstrap: Make build reproducible.

[Gábor Boskovits]

* gnu/packages/java.scm (ant-bootstrap)[arguments]<#:phases>[strip-jar-timestamps]:
New phase to reset jar file timestamps.
<#:imported-modules>: Import guix build syscalls, gnu-build-system-modules.
<#:modules>: Import guix build: gnu-build-system, utils, syscalls.
[native-inputs]: Add unzip, zip.
---
 gnu/packages/java.scm | 43 +++++++++++++++++++++++++++++++++++++++++--
 1 file changed, 41 insertions(+), 2 deletions(-)

diff --git a/gnu/packages/java.scm b/gnu/packages/java.scm
index 678cbee18..4fdce65af 100644
--- a/gnu/packages/java.scm
+++ b/gnu/packages/java.scm
@@ -212,7 +212,13 @@ JNI.")
                 "1cg0lga887qz5iizh6mlkxp01lciymrhmp7wzxpl6zpnldxmzrjx"))))
     (build-system gnu-build-system)
     (arguments
-     `(#:tests? #f ; no "check" target
+     `(#:imported-modules ((guix build syscalls)
+                           ,@%gnu-build-system-modules)
+       #:modules ((srfi srfi-1)
+                  (guix build gnu-build-system)
+                  (guix build utils)
+                  (guix build syscalls))
+       #:tests? #f ; no "check" target
        #:phases
        (modify-phases %standard-phases
          (delete 'bootstrap)
@@ -254,10 +260,43 @@ JNI.")
              (zero? (system* "bash" "bootstrap.sh"
                              (string-append "-Ddist.dir="
                                             (assoc-ref %outputs "out"))))))
+         (add-after 'build 'strip-jar-timestamps ;based on ant-build-system
+           (lambda* (#:key outputs #:allow-other-keys)
+             (define (repack-archive jar)
+               (let* ((dir (mkdtemp! "jar-contents.XXXXXX"))
+                      (manifest (string-append dir "/META-INF/MANIFESTS.MF")))
+                 (and (with-directory-excursion dir
+                        (invoke "unzip" jar))
+                      (delete-file jar)
+                      ;; XXX: copied from (gnu build install)
+                      (for-each (lambda (file)
+                                  (let ((s (lstat file)))
+                                    (unless (eq? (stat:type s) 'symlink)
+                                      (utime file  0 0 0 0))))
+                                (find-files dir #:directories? #t))
+                      ;; It is important that the manifes appears first.
+                      (with-directory-excursion dir
+                        (let* ((files (find-files "." ".*" #:directories? #t))
+                               ;; To ensure that the reference scanner can
+                               ;; detect all store references in the jars
+                               ;; we disable compression with the "-0" option.
+                               (command (if (file-exists? manifest)
+                                            `("zip" "-0" "-X" ,jar ,manifest
+                                              ,@files)
+                                            `("zip" "-0" "-X" ,jar ,@files))))
+                          (apply invoke command)))
+                      (utime jar 0 0)
+                      #t)))
+             (every repack-archive
+                    (find-files
+                     (string-append (assoc-ref %outputs "out") "/lib")
+                     "\\.jar$"))))
          (delete 'install))))
     (native-inputs
      `(("jikes" ,jikes)
-       ("jamvm" ,jamvm-1-bootstrap)))
+       ("jamvm" ,jamvm-1-bootstrap)
+       ("unzip" ,unzip)
+       ("zip", zip)))
     (home-page "http://ant.apache.org")
     (synopsis "Build tool for Java")
     (description
-- 
2.17.1

bug#30104: [PATCH] gnu: ant-bootstrap: Make build reproducible.

[Ludovic Courtès]

Hello!

Gábor Boskovits <boskovits@gmail.com> skribis:

> * gnu/packages/java.scm (ant-bootstrap)[arguments]<#:phases>[strip-jar-timestamps]:
> New phase to reset jar file timestamps.
> <#:imported-modules>: Import guix build syscalls, gnu-build-system-modules.
> <#:modules>: Import guix build: gnu-build-system, utils, syscalls.
> [native-inputs]: Add unzip, zip.

Great that you’re tackling this issue!

I have very minor comments:

> +         (add-after 'build 'strip-jar-timestamps ;based on ant-build-system
> +           (lambda* (#:key outputs #:allow-other-keys)
> +             (define (repack-archive jar)
> +               (let* ((dir (mkdtemp! "jar-contents.XXXXXX"))
> +                      (manifest (string-append dir "/META-INF/MANIFESTS.MF")))
> +                 (and (with-directory-excursion dir
> +                        (invoke "unzip" jar))
> +                      (delete-file jar)

‘invoke’ throws an exception upon failure, and the return value of
‘delete-file’ is unspecified, so you shouldn’t rely on it.

Thus, it should be written without ‘and’:

  (with-directory-excursion
    (invoke "unzip" jar))
  (delete-file jar)
  …

> +                      ;; It is important that the manifes appears first.
                                                            ^
Typo.

> +                      (utime jar 0 0)

I think this is unnecessary because guix-daemon resets timestamps on all
the file upon build completion.

> +             (every repack-archive
> +                    (find-files
> +                     (string-append (assoc-ref %outputs "out") "/lib")
> +                     "\\.jar$"))))

Use ‘for-each’ instead of ‘every’ (because the return value of
‘repack-archive’ is undefined/unused), and add a trailing #t to denote
success.

If Ricardo and Julien don’t have anything to add, I think you can go
ahead and push with these changes.

Thank you!

Ludo’.

bug#30104: [PATCH] gnu: ant-bootstrap: Make build reproducible.

[Ricardo Wurmus]

Ludovic Courtès <ludo@gnu.org> writes:

> Hello!
>
> Gábor Boskovits <boskovits@gmail.com> skribis:
>
>> * gnu/packages/java.scm (ant-bootstrap)[arguments]<#:phases>[strip-jar-timestamps]:
>> New phase to reset jar file timestamps.
>> <#:imported-modules>: Import guix build syscalls, gnu-build-system-modules.
>> <#:modules>: Import guix build: gnu-build-system, utils, syscalls.
>> [native-inputs]: Add unzip, zip.
>
> Great that you’re tackling this issue!

I concur.  Thanks Gábor!

> If Ricardo and Julien don’t have anything to add, I think you can go
> ahead and push with these changes.

I have nothing to add.  It looks good to me with these changes.

Thanks!

-- 
Ricardo

bug#30104: ant-bootstrap does not build reproducibly

[Gábor Boskovits]

[-- Attachment #1: Type: text/plain, Size: 67 bytes --]

Fixed on staging, commit cded3a759356ff66b7df668bcdbdfa0daf96f4c5.

[-- Attachment #2: Type: text/html, Size: 90 bytes --]