[bug#56604] [PATCH 0/8] Update Clojure to 1.11.1.

[Roman Scherer <roman.scherer@burningswell.com>]

[-- Attachment #1.1: Type: text/plain, Size: 1047 bytes --]


Hi Ludo,

here's the promised patch to follow up with the code duplication I
introduced in my previous patch.

I tested this by compiling Clojure which uses the Ant build system, and
by compiling clojure-tools-deps-alpha which uses the Clojure build
system. Both of the build systems now call the repack-jar function.

Could you have a look at it please?

A related question:

When I run the following commands after modifying the build systems they
run for quite some time, because they were compiling a ton (the jdk,
jetty) of things.

./pre-inst-env guix build clojure
./pre-inst-env guix build clojure-tools

I guess this is expected, since a change in a build system might affect
all packages being built with it. But I was wondering if there is a way
to force only building the packages specified on the command line. Does
such a thing exists?

I was wondering what is the most efficient way to quickly iterate on
changes to a build system, without recompiling the whole world for that
build system. How would you do that?

Thanks, Roman.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1.2: Add repack-jar and use it in Ant & Clojure build systems --]
[-- Type: text/x-diff, Size: 6839 bytes --]

From 756bfd3458ded38e1041ebb255c6b6ffe737732d Mon Sep 17 00:00:00 2001
From: Roman Scherer <roman@burningswell.com>
Date: Mon, 15 Aug 2022 15:29:25 +0000
Subject: [PATCH] build-system: Add repack-jar and use it in Ant & Clojure
 build systems

* guix/build/ant-build-system.scm: Add repack-jar and use it in strip-jar-timestamps
* guix/build/clojure-build-system.scm: Use repack-jar in reset-class-timestamps
---
 guix/build/ant-build-system.scm     | 26 ++++++++------
 guix/build/clojure-build-system.scm | 55 +++++------------------------
 2 files changed, 24 insertions(+), 57 deletions(-)

diff --git a/guix/build/ant-build-system.scm b/guix/build/ant-build-system.scm
index fae1b47ec5..63bdee4651 100644
--- a/guix/build/ant-build-system.scm
+++ b/guix/build/ant-build-system.scm
@@ -195,10 +195,9 @@ (define (generate-index jar)
             outputs)
   #t)
 
-(define* (strip-jar-timestamps #:key outputs
-                               #:allow-other-keys)
-  "Unpack all jar archives, reset the timestamp of all contained files, and
-repack them.  This is necessary to ensure that archives are reproducible."
+(define-public (repack-jar outputs repack-fn)
+  "Unpack all jar archives, invoke repack-fn for each JAR with the directory
+it has been unpacked to, and pack them again."
   (define (repack-archive jar)
     (format #t "repacking ~a\n" jar)
     (let* ((dir (mkdtemp! "jar-contents.XXXXXX"))
@@ -206,13 +205,7 @@ (define (repack-archive jar)
       (with-directory-excursion dir
         (invoke "jar" "xf" jar))
       (delete-file jar)
-      ;; XXX: copied from (gnu build install)
-      (for-each (lambda (file)
-                  (let ((s (lstat file)))
-                    (unless (eq? (stat:type s) 'symlink)
-                      (utime file 0 0 0 0))))
-                (find-files dir #:directories? #t))
-
+      (repack-fn dir)
       ;; The jar tool will always set the timestamp on the manifest file
       ;; and the containing directory to the current time, even when we
       ;; reuse an existing manifest file.  To avoid this we use "zip"
@@ -237,6 +230,17 @@ (define (repack-archive jar)
             outputs)
   #t)
 
+(define* (strip-jar-timestamps #:key outputs
+                               #:allow-other-keys)
+  "Unpack all jar archives, reset the timestamp of all contained files, and
+repack them.  This is necessary to ensure that archives are reproducible."
+  (repack-jar outputs (lambda (dir)
+                        (for-each (lambda (file)
+                                    (let ((s (lstat file)))
+                                      (unless (eq? (stat:type s) 'symlink)
+                                        (utime file 0 0 0 0))))
+                                  (find-files dir #:directories? #t)))))
+
 (define* (check #:key target (make-flags '()) (tests? (not target))
                 (test-target "check")
                 #:allow-other-keys)
diff --git a/guix/build/clojure-build-system.scm b/guix/build/clojure-build-system.scm
index cacbefb386..b82ebc30fe 100644
--- a/guix/build/clojure-build-system.scm
+++ b/guix/build/clojure-build-system.scm
@@ -19,7 +19,7 @@
 (define-module (guix build clojure-build-system)
   #:use-module ((guix build ant-build-system)
                 #:select ((%standard-phases . %standard-phases@ant)
-                          ant-build))
+                          ant-build repack-jar))
   #:use-module (guix build clojure-utils)
   #:use-module (guix build java-utils)
   #:use-module (guix build syscalls)
@@ -112,54 +112,17 @@ (define* (check #:key
                   jar-names)))
   #t)
 
-(define (regular-jar-file? file stat)
-  "Predicate returning true if FILE is ending on '.jar'
-and STAT indicates it is a regular file."
-    (and (string-suffix? ".jar" file)
-         (eq? 'regular (stat:type stat))))
-
-;; XXX: The only difference compared to 'strip-jar-timestamps' in
-;; ant-build-system.scm is the date.  TODO: Adjust and factorize.
 (define* (reset-class-timestamps #:key outputs #:allow-other-keys)
   "Unpack all jar archives, reset the timestamp of all contained class files,
 and repack them.  This is necessary to ensure that archives are reproducible."
-  (define (repack-archive jar)
-    (format #t "resetting class timestamps and repacking ~a\n" jar)
-
-    ;; Note: .class files need to be strictly newer than source files,
-    ;; otherwise the Clojure compiler will recompile sources.
-    (let* ((early-1980 315619200) ; 1980-01-02 UTC
-           (dir (mkdtemp! "jar-contents.XXXXXX"))
-           (manifest (string-append dir "/META-INF/MANIFEST.MF")))
-      (with-directory-excursion dir
-        (invoke "jar" "xf" jar))
-      (delete-file jar)
-      (for-each (lambda (file)
-                  (let ((s (lstat file)))
-                    (unless (eq? (stat:type s) 'symlink)
-                      (when (string-match "^(.*)\\.class$" file)
-                        (utime file early-1980 early-1980)))))
-                (find-files dir #:directories? #t))
-      ;; The jar tool will always set the timestamp on the manifest file
-      ;; and the containing directory to the current time, even when we
-      ;; reuse an existing manifest file.  To avoid this we use "zip"
-      ;; instead of "jar".  It is important that the manifest appears
-      ;; first.
-      (with-directory-excursion dir
-        (let* ((files (find-files "." ".*" #:directories? #t))
-               ;; To ensure that the reference scanner can detect all
-               ;; store references in the jars we disable compression
-               ;; with the "-0" option.
-               (command (if (file-exists? manifest)
-                            `("zip" "-0" "-X" ,jar ,manifest ,@files)
-                            `("zip" "-0" "-X" ,jar ,@files))))
-          (apply invoke command)))
-      (utime jar 0 0)))
-  (for-each (match-lambda
-              ((output . directory)
-               (for-each repack-archive
-                         (find-files directory regular-jar-file?))))
-            outputs))
+  (repack-jar outputs (lambda (dir)
+                        (for-each (lambda (file)
+                                    (let ((s (lstat file))
+                                          (early-1980 315619200)) ; 1980-01-02 UTC
+                                      (unless (eq? (stat:type s) 'symlink)
+                                        (when (string-match "^(.*)\\.class$" file)
+                                          (utime file early-1980 early-1980)))))
+                                  (find-files dir #:directories? #t)))))
 
 (define-with-docs install
   "Standard 'install' phase for clojure-build-system."
-- 
2.37.1


[-- Attachment #1.3: Type: text/plain, Size: 2438 bytes --]


Ludovic Courtès <ludo@gnu.org> writes:

> Hi,
>
> r0man <roman@burningswell.com> skribis:
>
>> This phase makes sure the timestamp of compiled class files is set to a later
>> point in time than the timestamp of the corresponding Clojure source files. If
>> the timestamps of the class and source files are the same, the Clojure
>> compiler will compile the sources again which can lead to issues. This problem
>> has been discussed here [1]. The suggested solution was to keep/adjust the
>> timestamps of the class files.
>
> Sounds reasonable.  It’s a bummer though that the whole phase is pasted
> from ant-build-system.scm, the only difference being the timestamps
> (1980 instead of 1970).
>
> I added a TODO comment in clojure-build-system.scm when applying the
> patch.  Could you follow up with a patch to factorize that?
>
>> Btw, I was a bit surprised that in Guix Clojure packages are AOT compiled. The
>> general wisdom in the Clojure community seems to be to avoid AOT compilation
>> when distributing libraries, and only AOT compiling Uberjars for final
>> deployment. Due to issues like I mentioned in clojure-instaparse.
>>
>> Are we sure that AOT compiling all Clojure source files by default is a good
>> idea, instead of just compiling user declared namespaces which Leiningen and
>> friends are doing? WDYT?
>
> Not much, but as you might have seen in ./etc/teams.scm, the project is
> finally being structured as teams.  There’s an opportunity for you to
> start a Clojure team and to take the lead!  :-)
>
> As a first step, I’d recommend getting in touch with people who have
> worked on ‘clojure-build-system’ and packaged things in the past.
>
>>   gnu: clojure-tools-cli: Update to 1.0.206.
>>   gnu: clojure-tools-gitlibs: Update to 2.4.181.
>>   gnu: clojure-tools-deps-alpha: Update to 0.14.1212.
>>   gnu: clojure-tools: Update to 1.11.1.1149.
>>   gnu: clojure: Update to 1.11.1.
>>   gnu: clojure-algo-generic: Fix test failing under AOT in Clojure 1.11.1.
>>   gnu: clojure-core-match: Update to 1.0.0.
>>   gnu: clojure-instaparse: Update to 1.4.12 (disabled AOT).
>
> I adjusted all the commit logs to follow our conventions; please
> consider doing this next time:
>
>   https://guix.gnu.org/manual/devel/en/html_node/Submitting-Patches.html
>
> The instaparse patch missed the hash update so I did that too.
>
> Thanks!
>
> Ludo’.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 528 bytes --]