[PATCH] Improve ant-build-system.

[PATCH] Improve ant-build-system.

[Ricardo Wurmus]

[-- Attachment #1: Type: text/plain, Size: 668 bytes --]

Hi Guix,

our ant-build-system currently throws away the jar manifest, because it
was difficult to control its timestamp.  Without a manifest many jars
cannot be used, though.  Oops!

The manifest and the directory it is in always get the current time,
which is very annoying.  Since controlling the timestamp is not possible
with “jar”, this patch repacks the jar archive with “zip”.  The only
thing we need to take care of is to ensure that the manifest file
appears first.

With these changes the jars actually work properly.

Thanks to Roel for reporting these issues to me privately and assisting
in debugging all of this!

~~ Ricardo


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-build-system-ant-Add-zip.patch --]
[-- Type: text/x-patch, Size: 1756 bytes --]

From 0839fc0c10f6ace360ff88a48951340b5b25e376 Mon Sep 17 00:00:00 2001
From: Ricardo Wurmus <ricardo.wurmus@mdc-berlin.de>
Date: Thu, 24 Mar 2016 15:33:21 +0100
Subject: [PATCH 1/2] build-system/ant: Add zip.

* guix/build-system/ant.scm (default-zip): New variable.
(lower): Add zip to native inputs.
---
 guix/build-system/ant.scm | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)

diff --git a/guix/build-system/ant.scm b/guix/build-system/ant.scm
index d3054e5..f333aa5 100644
--- a/guix/build-system/ant.scm
+++ b/guix/build-system/ant.scm
@@ -54,15 +54,22 @@
   (let ((jdk-mod (resolve-interface '(gnu packages java))))
     (module-ref jdk-mod 'ant)))
 
+(define (default-zip)
+  "Return the default ZIP package."
+  ;; Lazily resolve the binding to avoid a circular dependency.
+  (let ((zip-mod (resolve-interface '(gnu packages zip))))
+    (module-ref zip-mod 'zip)))
+
 (define* (lower name
                 #:key source inputs native-inputs outputs system target
                 (jdk (default-jdk))
                 (ant (default-ant))
+                (zip (default-zip))
                 #:allow-other-keys
                 #:rest arguments)
   "Return a bag for NAME."
   (define private-keywords
-    '(#:source #:target #:jdk #:ant #:inputs #:native-inputs))
+    '(#:source #:target #:jdk #:ant #:zip #:inputs #:native-inputs))
 
   (and (not target)                               ;XXX: no cross-compilation
        (bag
@@ -77,6 +84,7 @@
                         ,@(standard-packages)))
          (build-inputs `(("jdk" ,jdk "jdk")
                          ("ant" ,ant)
+                         ("zip" ,zip)
                          ,@native-inputs))
          (outputs outputs)
          (build ant-build)
-- 
2.1.0


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #3: 0002-ant-build-system-Keep-jar-manifest.patch --]
[-- Type: text/x-patch, Size: 3719 bytes --]

From 7a3ac4d2eb7c7e9f3a96927f6c68c2a182082b6a Mon Sep 17 00:00:00 2001
From: Ricardo Wurmus <ricardo.wurmus@mdc-berlin.de>
Date: Thu, 24 Mar 2016 15:39:45 +0100
Subject: [PATCH 2/2] ant-build-system: Keep jar manifest.

* guix/build/ant-build-system.scm (default-build.xml): Generate default
manifest.
(strip-jar-timestamps): Repack jar archive with zip.
---
 guix/build/ant-build-system.scm | 29 +++++++++++++++++------------
 1 file changed, 17 insertions(+), 12 deletions(-)

diff --git a/guix/build/ant-build-system.scm b/guix/build/ant-build-system.scm
index d302b94..27277af 100644
--- a/guix/build/ant-build-system.scm
+++ b/guix/build/ant-build-system.scm
@@ -65,13 +65,8 @@
                  (target (@ (name "jar")
                             (depends "compile"))
                          (mkdir (@ (dir "${jar.dir}")))
-                         ;; We cannot use the simpler "jar" task here, because
-                         ;; there is no way to disable generation of a
-                         ;; manifest.  We do not include a generated manifest
-                         ;; to ensure determinism, because we cannot easily
-                         ;; reset the ctime/mtime before creating the archive.
                          (exec (@ (executable "jar"))
-                               (arg (@ (line ,(string-append "-Mcf ${jar.dir}/" jar-name
+                               (arg (@ (line ,(string-append "-cf ${jar.dir}/" jar-name
                                                              " -C ${classes.dir} ."))))))
 
                  (target (@ (name "install"))
@@ -105,16 +100,15 @@ INPUTS."
   (zero? (apply system* `("ant" ,build-target ,@make-flags))))
 
 (define* (strip-jar-timestamps #:key outputs
-                 #:allow-other-keys)
+                               #:allow-other-keys)
   "Unpack all jar archives, reset the timestamp of all contained files, and
 repack them.  This is necessary to ensure that archives are reproducible."
   (define (repack-archive jar)
     (format #t "repacking ~a\n" jar)
-    (let ((dir (mkdtemp! "jar-contents.XXXXXX")))
+    (let* ((dir (mkdtemp! "jar-contents.XXXXXX"))
+           (manifest (string-append dir "/META-INF/MANIFEST.MF")))
       (and (with-directory-excursion dir
              (zero? (system* "jar" "xf" jar)))
-           ;; The manifest file contains timestamps
-           (for-each delete-file (find-files dir "MANIFEST.MF"))
            (delete-file jar)
            ;; XXX: copied from (gnu build install)
            (for-each (lambda (file)
@@ -122,8 +116,19 @@ repack them.  This is necessary to ensure that archives are reproducible."
                          (unless (eq? (stat:type s) 'symlink)
                            (utime file 0 0 0 0))))
                      (find-files dir #:directories? #t))
-           (unless (zero? (system* "jar" "-Mcf" jar "-C" dir "."))
-             (error "'jar' failed"))
+
+           ;; The jar tool will always set the timestamp on the manifest file
+           ;; and the containing directory to the current time, even when we
+           ;; reuse an existing manifest file.  To avoid this we use "zip"
+           ;; instead of "jar".  It is important that the manifest appears
+           ;; first.
+           (with-directory-excursion dir
+             (let* ((files (find-files "." ".*" #:directories? #t))
+                    (command (if (file-exists? manifest)
+                                 `("zip" "-X" ,jar ,manifest ,@files)
+                                 `("zip" "-X" ,jar ,@files))))
+               (unless (zero? (apply system* command))
+                 (error "'zip' failed"))))
            (utime jar 0 0)
            #t)))
 
-- 
2.1.0

Re: [PATCH] Improve ant-build-system.

[Leo Famulari]

On Thu, Mar 24, 2016 at 03:50:38PM +0100, Ricardo Wurmus wrote:
> Hi Guix,
> 
> our ant-build-system currently throws away the jar manifest, because it
> was difficult to control its timestamp.  Without a manifest many jars
> cannot be used, though.  Oops!
> 
> The manifest and the directory it is in always get the current time,
> which is very annoying.  Since controlling the timestamp is not possible
> with “jar”, this patch repacks the jar archive with “zip”.  The only
> thing we need to take care of is to ensure that the manifest file
> appears first.

So, the timestamp issue is avoided by using "zip"? Just curious, I would
want to block this change regardless.

> 
> With these changes the jars actually work properly.
> 
> Thanks to Roel for reporting these issues to me privately and assisting
> in debugging all of this!
> 
> ~~ Ricardo
> 

> From 0839fc0c10f6ace360ff88a48951340b5b25e376 Mon Sep 17 00:00:00 2001
> From: Ricardo Wurmus <ricardo.wurmus@mdc-berlin.de>
> Date: Thu, 24 Mar 2016 15:33:21 +0100
> Subject: [PATCH 1/2] build-system/ant: Add zip.
> 
> * guix/build-system/ant.scm (default-zip): New variable.
> (lower): Add zip to native inputs.
> ---
>  guix/build-system/ant.scm | 10 +++++++++-
>  1 file changed, 9 insertions(+), 1 deletion(-)
> 
> diff --git a/guix/build-system/ant.scm b/guix/build-system/ant.scm
> index d3054e5..f333aa5 100644
> --- a/guix/build-system/ant.scm
> +++ b/guix/build-system/ant.scm
> @@ -54,15 +54,22 @@
>    (let ((jdk-mod (resolve-interface '(gnu packages java))))
>      (module-ref jdk-mod 'ant)))
>  
> +(define (default-zip)
> +  "Return the default ZIP package."
> +  ;; Lazily resolve the binding to avoid a circular dependency.
> +  (let ((zip-mod (resolve-interface '(gnu packages zip))))
> +    (module-ref zip-mod 'zip)))
> +
>  (define* (lower name
>                  #:key source inputs native-inputs outputs system target
>                  (jdk (default-jdk))
>                  (ant (default-ant))
> +                (zip (default-zip))
>                  #:allow-other-keys
>                  #:rest arguments)
>    "Return a bag for NAME."
>    (define private-keywords
> -    '(#:source #:target #:jdk #:ant #:inputs #:native-inputs))
> +    '(#:source #:target #:jdk #:ant #:zip #:inputs #:native-inputs))
>  
>    (and (not target)                               ;XXX: no cross-compilation
>         (bag
> @@ -77,6 +84,7 @@
>                          ,@(standard-packages)))
>           (build-inputs `(("jdk" ,jdk "jdk")
>                           ("ant" ,ant)
> +                         ("zip" ,zip)
>                           ,@native-inputs))
>           (outputs outputs)
>           (build ant-build)
> -- 
> 2.1.0
> 

> From 7a3ac4d2eb7c7e9f3a96927f6c68c2a182082b6a Mon Sep 17 00:00:00 2001
> From: Ricardo Wurmus <ricardo.wurmus@mdc-berlin.de>
> Date: Thu, 24 Mar 2016 15:39:45 +0100
> Subject: [PATCH 2/2] ant-build-system: Keep jar manifest.
> 
> * guix/build/ant-build-system.scm (default-build.xml): Generate default
> manifest.
> (strip-jar-timestamps): Repack jar archive with zip.
> ---
>  guix/build/ant-build-system.scm | 29 +++++++++++++++++------------
>  1 file changed, 17 insertions(+), 12 deletions(-)
> 
> diff --git a/guix/build/ant-build-system.scm b/guix/build/ant-build-system.scm
> index d302b94..27277af 100644
> --- a/guix/build/ant-build-system.scm
> +++ b/guix/build/ant-build-system.scm
> @@ -65,13 +65,8 @@
>                   (target (@ (name "jar")
>                              (depends "compile"))
>                           (mkdir (@ (dir "${jar.dir}")))
> -                         ;; We cannot use the simpler "jar" task here, because
> -                         ;; there is no way to disable generation of a
> -                         ;; manifest.  We do not include a generated manifest
> -                         ;; to ensure determinism, because we cannot easily
> -                         ;; reset the ctime/mtime before creating the archive.
>                           (exec (@ (executable "jar"))
> -                               (arg (@ (line ,(string-append "-Mcf ${jar.dir}/" jar-name
> +                               (arg (@ (line ,(string-append "-cf ${jar.dir}/" jar-name
>                                                               " -C ${classes.dir} ."))))))
>  
>                   (target (@ (name "install"))
> @@ -105,16 +100,15 @@ INPUTS."
>    (zero? (apply system* `("ant" ,build-target ,@make-flags))))
>  
>  (define* (strip-jar-timestamps #:key outputs
> -                 #:allow-other-keys)
> +                               #:allow-other-keys)
>    "Unpack all jar archives, reset the timestamp of all contained files, and
>  repack them.  This is necessary to ensure that archives are reproducible."
>    (define (repack-archive jar)
>      (format #t "repacking ~a\n" jar)
> -    (let ((dir (mkdtemp! "jar-contents.XXXXXX")))
> +    (let* ((dir (mkdtemp! "jar-contents.XXXXXX"))
> +           (manifest (string-append dir "/META-INF/MANIFEST.MF")))
>        (and (with-directory-excursion dir
>               (zero? (system* "jar" "xf" jar)))
> -           ;; The manifest file contains timestamps
> -           (for-each delete-file (find-files dir "MANIFEST.MF"))
>             (delete-file jar)
>             ;; XXX: copied from (gnu build install)
>             (for-each (lambda (file)
> @@ -122,8 +116,19 @@ repack them.  This is necessary to ensure that archives are reproducible."
>                           (unless (eq? (stat:type s) 'symlink)
>                             (utime file 0 0 0 0))))
>                       (find-files dir #:directories? #t))
> -           (unless (zero? (system* "jar" "-Mcf" jar "-C" dir "."))
> -             (error "'jar' failed"))
> +
> +           ;; The jar tool will always set the timestamp on the manifest file
> +           ;; and the containing directory to the current time, even when we
> +           ;; reuse an existing manifest file.  To avoid this we use "zip"
> +           ;; instead of "jar".  It is important that the manifest appears
> +           ;; first.
> +           (with-directory-excursion dir
> +             (let* ((files (find-files "." ".*" #:directories? #t))
> +                    (command (if (file-exists? manifest)
> +                                 `("zip" "-X" ,jar ,manifest ,@files)
> +                                 `("zip" "-X" ,jar ,@files))))
> +               (unless (zero? (apply system* command))
> +                 (error "'zip' failed"))))
>             (utime jar 0 0)
>             #t)))
>  
> -- 
> 2.1.0
> 

Re: [PATCH] Improve ant-build-system.

[Ricardo Wurmus]

Leo Famulari <leo@famulari.name> writes:

> On Thu, Mar 24, 2016 at 03:50:38PM +0100, Ricardo Wurmus wrote:
>> Hi Guix,
>> 
>> our ant-build-system currently throws away the jar manifest, because it
>> was difficult to control its timestamp.  Without a manifest many jars
>> cannot be used, though.  Oops!
>> 
>> The manifest and the directory it is in always get the current time,
>> which is very annoying.  Since controlling the timestamp is not possible
>> with “jar”, this patch repacks the jar archive with “zip”.  The only
>> thing we need to take care of is to ensure that the manifest file
>> appears first.
>
> So, the timestamp issue is avoided by using "zip"? Just curious, I would
> want to block this change regardless.

Correct.  “jar” doesn’t just create an archive, it also sets the
timestamp of the manifest.  “zip” doesn’t care about manifest timestamps
and will just take the files as they are.  Since we reset timestamps of
all files in the archive this means that the manifest will have the same
timestamp as any other file in the archive.  This makes the jar archive
reproducible.

~~ Ricardo

Re: [PATCH] Improve ant-build-system.

[Leo Famulari]

On Thu, Mar 24, 2016 at 04:57:11PM +0100, Ricardo Wurmus wrote:
> 
> Leo Famulari <leo@famulari.name> writes:
> 
> > On Thu, Mar 24, 2016 at 03:50:38PM +0100, Ricardo Wurmus wrote:
> >> Hi Guix,
> >> 
> >> our ant-build-system currently throws away the jar manifest, because it
> >> was difficult to control its timestamp.  Without a manifest many jars
> >> cannot be used, though.  Oops!
> >> 
> >> The manifest and the directory it is in always get the current time,
> >> which is very annoying.  Since controlling the timestamp is not possible
> >> with “jar”, this patch repacks the jar archive with “zip”.  The only
> >> thing we need to take care of is to ensure that the manifest file
> >> appears first.
> >
> > So, the timestamp issue is avoided by using "zip"? Just curious, I would
> > want to block this change regardless.

I meant to write "wouldn't".

> 
> Correct.  “jar” doesn’t just create an archive, it also sets the
> timestamp of the manifest.  “zip” doesn’t care about manifest timestamps
> and will just take the files as they are.  Since we reset timestamps of
> all files in the archive this means that the manifest will have the same
> timestamp as any other file in the archive.  This makes the jar archive
> reproducible.

Cool, I'm glad we have this option!

> 
> ~~ Ricardo

Re: [PATCH] Improve ant-build-system.

[Ricardo Wurmus]

Ricardo Wurmus <ricardo.wurmus@mdc-berlin.de> writes:

> our ant-build-system currently throws away the jar manifest, because it
> was difficult to control its timestamp.  Without a manifest many jars
> cannot be used, though.  Oops!
>
> The manifest and the directory it is in always get the current time,
> which is very annoying.  Since controlling the timestamp is not possible
> with “jar”, this patch repacks the jar archive with “zip”.  The only
> thing we need to take care of is to ensure that the manifest file
> appears first.
>
> With these changes the jars actually work properly.
>
> Thanks to Roel for reporting these issues to me privately and assisting
> in debugging all of this!

I pushed these commits as 9941e0812eca a couple of hours ago.

~~ Ricardo