[bug#56604] [PATCH 0/8] Update Clojure to 1.11.1.

[bug#56604] [PATCH 0/8] Update Clojure to 1.11.1.

[r0man]

[-- Attachment #1: Type: text/plain, Size: 1875 bytes --]

Hello Guix,

This patch series updates Clojure and it's packages to 1.11.1.

It also adds the 'reset-class-timestamps phase to the Clojure build system.

This phase makes sure the timestamp of compiled class files is set to a later
point in time than the timestamp of the corresponding Clojure source files. If
the timestamps of the class and source files are the same, the Clojure
compiler will compile the sources again which can lead to issues. This problem
has been discussed here [1]. The suggested solution was to keep/adjust the
timestamps of the class files.

[1] https://www.mail-archive.com/clojure@googlegroups.com/msg99928.html

Could you please review this?

In the patch for clojure-instaparse, I disabled the AOT compilation because of
a known issue. Is that ok?

Btw, I was a bit surprised that in Guix Clojure packages are AOT compiled. The
general wisdom in the Clojure community seems to be to avoid AOT compilation
when distributing libraries, and only AOT compiling Uberjars for final
deployment. Due to issues like I mentioned in clojure-instaparse.

Are we sure that AOT compiling all Clojure source files by default is a good
idea, instead of just compiling user declared namespaces which Leiningen and
friends are doing? WDYT?

Thanks Roman.

r0man (8):
  gnu: clojure-tools-cli: Update to 1.0.206.
  gnu: clojure-tools-gitlibs: Update to 2.4.181.
  gnu: clojure-tools-deps-alpha: Update to 0.14.1212.
  gnu: clojure-tools: Update to 1.11.1.1149.
  gnu: clojure: Update to 1.11.1.
  gnu: clojure-algo-generic: Fix test failing under AOT in Clojure 1.11.1.
  gnu: clojure-core-match: Update to 1.0.0.
  gnu: clojure-instaparse: Update to 1.4.12 (disabled AOT).

 gnu/packages/clojure.scm            | 185 +++++++++++++++++++---------
 guix/build/clojure-build-system.scm |  53 +++++++-
 2 files changed, 178 insertions(+), 60 deletions(-)

--
2.36.1


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 519 bytes --]

[bug#56604] [PATCH 1/8] gnu: clojure-tools-cli: Update to 1.0.206.---

[r0man]

[-- Attachment #1: Type: text/plain, Size: 1056 bytes --]

---
 gnu/packages/clojure.scm | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/gnu/packages/clojure.scm b/gnu/packages/clojure.scm
index e6eb749501..0a30e1752a 100644
--- a/gnu/packages/clojure.scm
+++ b/gnu/packages/clojure.scm
@@ -457,7 +457,7 @@ (define-public clojure-tools-macro
 (define-public clojure-tools-cli
   (package
     (name "clojure-tools-cli")
-    (version "0.4.2")
+    (version "1.0.206")
     (home-page "https://github.com/clojure/tools.cli")
     (source (origin
               (method git-fetch)
@@ -466,7 +466,7 @@ (define-public clojure-tools-cli
                     (commit (string-append "tools.cli-" version))))
               (file-name (git-file-name name version))
               (sha256
-               (base32 "1yqlm8lwbcjm0dp032z7vzc4bdlmc4jixznvf4adsqhvqw85hvj2"))))
+               (base32 "1hvk5zacl3fr8lfcbfgckaicqjx697j0kzw2x5hwj8j5xlr8ri2r"))))
     (build-system clojure-build-system)
     (arguments
      '(#:source-dirs '("src/main/clojure/")
-- 
2.36.1



[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 519 bytes --]

[bug#56604] [PATCH 2/8] gnu: clojure-tools-gitlibs: Update to 2.4.181.---

[r0man]

[-- Attachment #1: Type: text/plain, Size: 1019 bytes --]

---
 gnu/packages/clojure.scm | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/gnu/packages/clojure.scm b/gnu/packages/clojure.scm
index 0a30e1752a..494ac25fc4 100644
--- a/gnu/packages/clojure.scm
+++ b/gnu/packages/clojure.scm
@@ -543,7 +543,7 @@ (define-public clojure-tools-deps-alpha
 (define-public clojure-tools-gitlibs
   (package
     (name "clojure-tools-gitlibs")
-    (version "2.4.172")
+    (version "2.4.181")
     (home-page "https://github.com/clojure/tools.gitlibs")
     (source (origin
               (method git-fetch)
@@ -553,7 +553,7 @@ (define-public clojure-tools-gitlibs
               (file-name (git-file-name name version))
               (sha256
                (base32
-                "19d83wigmigy66f5i61a3ishg7zrlv8jngj8m9brsh9v38a0fwdr"))))
+                "1d8c79f33axghadwqa955yjfsxa6fgl8jq4nfll2zrp6sjw0597k"))))
     (build-system clojure-build-system)
     (arguments
      '(#:source-dirs '("src/main/clojure")
-- 
2.36.1



[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 519 bytes --]

[bug#56604] [PATCH 3/8] gnu: clojure-tools-deps-alpha: Update to 0.14.1212.---

[r0man]

[-- Attachment #1: Type: text/plain, Size: 1049 bytes --]

---
 gnu/packages/clojure.scm | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/gnu/packages/clojure.scm b/gnu/packages/clojure.scm
index 494ac25fc4..d33ee4f708 100644
--- a/gnu/packages/clojure.scm
+++ b/gnu/packages/clojure.scm
@@ -481,7 +481,7 @@ (define-public clojure-tools-cli
 (define-public clojure-tools-deps-alpha
   (package
     (name "clojure-tools-deps-alpha")
-    (version "0.12.1104")
+    (version "0.14.1212")
     (home-page "https://github.com/clojure/tools.deps.alpha")
     (source (origin
               (method git-fetch)
@@ -491,7 +491,7 @@ (define-public clojure-tools-deps-alpha
               (file-name (git-file-name name version))
               (sha256
                (base32
-                "174m83n5m3arai2vbg434zjibbsr3r4pp7lz3adja8lxq7g21r80"))))
+                "0r8qfihii6cf95kl86x6zfldnm7wlkgda2qmq3340j1x03v244dd"))))
     (build-system clojure-build-system)
     (arguments
      `(#:source-dirs '("src/main/clojure" "src/main/resources")
-- 
2.36.1



[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 519 bytes --]

[bug#56604] [PATCH 4/8] gnu: clojure-tools: Update to 1.11.1.1149.---

[r0man]

[-- Attachment #1: Type: text/plain, Size: 974 bytes --]

---
 gnu/packages/clojure.scm | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/gnu/packages/clojure.scm b/gnu/packages/clojure.scm
index d33ee4f708..fa201f3110 100644
--- a/gnu/packages/clojure.scm
+++ b/gnu/packages/clojure.scm
@@ -153,14 +153,14 @@ (define (extract-library name)
 (define-public clojure-tools
   (package
     (name "clojure-tools")
-    (version "1.10.3.1040")
+    (version "1.11.1.1149")
     (source
      (origin
        (method url-fetch)
        (uri (string-append "https://download.clojure.org/install/clojure-tools-"
                            version
                            ".tar.gz"))
-       (sha256 (base32 "0xvr9nmk9q789vp32zmmzj4macv8v7y9ivnfd6lf7i8vxgg6hvgv"))
+       (sha256 (base32 "196wl87br8zg3npfwpr5c1q0knxd1810vzgb0b6h195hyjf6i210"))
        ;; Remove AOT compiled JAR.  The other JAR only contains uncompiled
        ;; Clojure source code.
        (snippet
-- 
2.36.1



[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 519 bytes --]

[bug#56604] [PATCH 5/8] gnu: clojure: Update to 1.11.1.This patch updates Clojure to 1.11.1. It also adds the 'reset-class-timestamps

[r0man]

[-- Attachment #1: Type: text/plain, Size: 12365 bytes --]

This patch updates Clojure to 1.11.1. It also adds the 'reset-class-timestamps
phase to the Clojure build system.

This phase makes sure the timestamp of compiled class files is set to a later
point in time than the timestamp of the corresponding Clojure source files. If
the timestamps of the class and source files are the same, the Clojure
compiler will compile the sources again which can lead to issues. This problem
has been discussed here [1]. The suggested solution was to keep/adjust the
timestamps of the class files.

[1] https://www.mail-archive.com/clojure@googlegroups.com/msg99928.html
---
 gnu/packages/clojure.scm            | 93 ++++++++++++++++++++++-------
 guix/build/clojure-build-system.scm | 53 +++++++++++++++-
 2 files changed, 123 insertions(+), 23 deletions(-)

diff --git a/gnu/packages/clojure.scm b/gnu/packages/clojure.scm
index fa201f3110..d51923acdb 100644
--- a/gnu/packages/clojure.scm
+++ b/gnu/packages/clojure.scm
@@ -43,36 +43,44 @@ (define-public clojure
          ;; The libraries below are needed to run the tests.
          (libraries
           `(("core-specs-alpha-src"
-             ,(lib "core.specs.alpha/archive/core.specs.alpha-"
-                   "0.1.24"
-                   "0v2a0svf1ar2y42ajxwsjr7zmm5j7pp2zwrd2jh3k7xzd1p9x1fv"))
+             ,(lib "core.specs.alpha/archive/v"
+                   "0.2.62"
+                   "0v6nhghsigpzm8y7dykfm318q5dvk5l8sykmn1hr0qgs1jsjqh9j"))
             ("data-generators-src"
              ,(lib "data.generators/archive/data.generators-"
-                   "0.1.2"
-                   "0kki093jp4ckwxzfnw8ylflrfqs8b1i1wi9iapmwcsy328dmgzp1"))
+                   "1.0.0"
+                   "0s3hf1njvs68b8igasikvzagzqxl0gbri7w2qhzsypkhfh60v2cp"))
+            ("java-classpath-src"
+             ,(lib "java.classpath/archive/java.classpath-"
+                   "1.0.0"
+                   "178zajjsc9phk5l61r8w9hcpk0wgc9a811pl7kjgvn7rg4l7fh7j"))
             ("spec-alpha-src"
-             ,(lib "spec.alpha/archive/spec.alpha-"
-                   "0.1.143"
-                   "00alf0347licdn773w2jarpllyrbl52qz4d8mw61anjksacxylzz"))
+             ,(lib "spec.alpha/archive/v"
+                   "0.3.218"
+                   "0h5nd9xlind1a2vmllr2yfhnirgj2pm5dndgqzrly78l5iwcc3wa"))
             ("test-check-src"
-             ,(lib "test.check/archive/test.check-"
-                   "0.9.0"
-                   "0p0mnyhr442bzkz0s4k5ra3i6l5lc7kp6ajaqkkyh4c2k5yck1md"))
+             ,(lib "test.check/archive/v"
+                   "1.1.1"
+                   "0kx8l79mhpnn94rpsgc7nac7gb222g7a47mzrycj8crfc54wf0c1"))
             ("test-generative-src"
              ,(lib "test.generative/archive/test.generative-"
-                   "0.5.2"
-                   "1pjafy1i7yblc7ixmcpfq1lfbyf3jaljvkgrajn70sws9xs7a9f8"))
+                   "1.0.0"
+                   "0yy2vc38s4j5n94jdcjx1v7l2gdq0lywam31id1jh07sx37lv5il"))
             ("tools-namespace-src"
              ,(lib "tools.namespace/archive/tools.namespace-"
-                   "0.2.11"
-                   "10baak8v0hnwz2hr33bavshm7y49mmn9zsyyms1dwjz45p5ymhy0"))))
+                   "1.0.0"
+                   "1ifpk93m33rj2xm1qnnninlsdvm1liqmsp9igr63pjjwwwjw1cnn"))
+            ("tools-reader-src"
+             ,(lib "tools.reader/archive/tools.reader-"
+                   "1.3.2"
+                   "1n4dhg61iyypnjbxmihhqjb7lfpc0lzfvlk4jd8w0yr6za414f3a"))))
          (library-names (match libraries
                           (((library-name _) ...)
                            library-name))))
 
     (package
       (name "clojure")
-      (version "1.10.0")
+      (version "1.11.1")
       (source (let ((name+version (string-append name "-" version)))
                 (origin
                   (method git-fetch)
@@ -81,18 +89,23 @@ (define-public clojure
                         (commit name+version)))
                   (file-name (string-append name+version "-checkout"))
                   (sha256
-                   (base32 "1kcyv2836acs27vi75hvf3r773ahv2nlh9b3j9xa9m9sdanz1h83")))))
+                   (base32 "1xbab21rm9zvhmw1i2h5lqm7612vrdkxprq0rgb2i3sbgsxcdsn4")))))
       (build-system ant-build-system)
       (inputs
        `(("jre" ,icedtea)))
       (arguments
        `(#:imported-modules ((guix build clojure-utils)
+                             (guix build clojure-build-system)
                              (guix build guile-build-system)
                              ,@%ant-build-system-modules)
          #:modules ((guix build ant-build-system)
+                    ((guix build clojure-build-system) #:prefix clj:)
                     (guix build clojure-utils)
                     (guix build java-utils)
                     (guix build utils)
+                    (guix build syscalls)
+                    (ice-9 match)
+                    (ice-9 regex)
                     (srfi srfi-26))
          #:test-target "test"
          #:phases
@@ -106,22 +119,60 @@ (define (extract-library name)
                            "--extract"
                            "--verbose"
                            "--file" (assoc-ref inputs name)
-                           "--strip-components=1"))
-                 (copy-recursively (string-append name "/src/main/clojure/")
-                                   "src/clj/"))
+                           "--strip-components=1")))
                (for-each extract-library ',library-names)
+               (copy-recursively "core-specs-alpha-src/src/main/clojure"
+                                 "src/clj/")
+               (copy-recursively "spec-alpha-src/src/main/clojure"
+                                 "src/clj/")
                #t))
            (add-after 'unpack-library-sources 'fix-manifest-classpath
              (lambda _
                (substitute* "build.xml"
                  (("<attribute name=\"Class-Path\" value=\".\"/>") ""))
                #t))
+           (add-after 'unpack-library-sources 'clojure-spec-skip-macros
+             ;; Disable spec macro instrumentation when compiling clojure.spec
+             ;; See: https://clojure.atlassian.net/browse/CLJ-2254
+             (lambda _
+               (substitute* "build.xml"
+                 (("<sysproperty key=\"java.awt.headless\" value=\"true\"/>")
+                  ,(string-join
+                    '("<sysproperty key=\"java.awt.headless\" value=\"true\"/>"
+                      "<sysproperty key=\"clojure.spec.skip-macros\" value=\"true\"/>\n")
+                    "\n")))
+               #t))
+           (add-after 'unpack-library-sources 'clojure-spec-compile
+             ;; Compile and include clojure.spec.alpha & clojure.core.specs.alpha
+             (lambda _
+               (substitute* "build.xml"
+                 (("<arg value=\"clojure.math\"/>")
+                  ,(string-join
+                    '("<arg value=\"clojure.math\"/>"
+                      "<arg value=\"clojure.spec.alpha\"/>"
+                      "<arg value=\"clojure.spec.gen.alpha\"/>"
+                      "<arg value=\"clojure.spec.test.alpha\"/>"
+                      "<arg value=\"clojure.core.specs.alpha\"/>"))))
+               #t))
+           (add-before 'build 'maven-classpath-properties
+             (lambda _
+               (define (make-classpath libraries)
+                 (string-join (map (lambda (library)
+                                     (string-append library "/src/main/clojure"))
+                                   libraries) ":"))
+               (with-output-to-file "maven-classpath.properties"
+                 (lambda ()
+                   (let ((classpath (make-classpath ',library-names)))
+                     (display (string-append "maven.compile.classpath=" classpath "\n"))
+                     (display (string-append "maven.test.classpath=" classpath "\n")))))
+               #t))
            (add-after 'build 'build-javadoc ant-build-javadoc)
            (replace 'install (install-jars "./"))
            (add-after 'install-license-files 'install-doc
              (cut install-doc #:doc-dirs '("doc/clojure/") <...>))
            (add-after 'install-doc 'install-javadoc
-             (install-javadoc "target/javadoc/")))))
+             (install-javadoc "target/javadoc/"))
+           (add-after 'reset-gzip-timestamps 'reset-class-timestamps clj:reset-class-timestamps))))
       (native-inputs libraries)
       (home-page "https://clojure.org/")
       (synopsis "Lisp dialect running on the JVM")
diff --git a/guix/build/clojure-build-system.scm b/guix/build/clojure-build-system.scm
index 7d494078ea..82440e3b7a 100644
--- a/guix/build/clojure-build-system.scm
+++ b/guix/build/clojure-build-system.scm
@@ -22,11 +22,15 @@ (define-module (guix build clojure-build-system)
                           ant-build))
   #:use-module (guix build clojure-utils)
   #:use-module (guix build java-utils)
+  #:use-module (guix build syscalls)
   #:use-module (guix build utils)
+  #:use-module (ice-9 match)
+  #:use-module (ice-9 regex)
   #:use-module (srfi srfi-1)
   #:use-module (srfi srfi-26)
   #:export (%standard-phases
-            clojure-build))
+            clojure-build
+            reset-class-timestamps))
 
 ;; Commentary:
 ;;
@@ -108,6 +112,50 @@ (define* (check #:key
                   jar-names)))
   #t)
 
+(define (regular-jar-file-predicate file stat)
+  "Predicate returning true if FILE is ending on '.jar'
+and STAT indicates it is a regular file."
+    (and ((file-name-predicate "\\.jar$") file stat)
+         (eq? 'regular (stat:type stat))))
+
+(define* (reset-class-timestamps #:key outputs #:allow-other-keys)
+  "Unpack all jar archives, reset the timestamp of all contained class files,
+and repack them.  This is necessary to ensure that archives are reproducible."
+  (define (repack-archive jar)
+    (format #t "resetting class timestamps and repacking ~a\n" jar)
+    (let* ((early-1980 315619200) ; 1980-01-02 UTC
+           (dir (mkdtemp! "jar-contents.XXXXXX"))
+           (manifest (string-append dir "/META-INF/MANIFEST.MF")))
+      (with-directory-excursion dir
+        (invoke "jar" "xf" jar))
+      (delete-file jar)
+      (for-each (lambda (file)
+                  (let ((s (lstat file)))
+                    (unless (eq? (stat:type s) 'symlink)
+                      (when (string-match "^(.*)\\.class$" file)
+                        (utime file early-1980 early-1980)))))
+                (find-files dir #:directories? #t))
+      ;; The jar tool will always set the timestamp on the manifest file
+      ;; and the containing directory to the current time, even when we
+      ;; reuse an existing manifest file.  To avoid this we use "zip"
+      ;; instead of "jar".  It is important that the manifest appears
+      ;; first.
+      (with-directory-excursion dir
+        (let* ((files (find-files "." ".*" #:directories? #t))
+               ;; To ensure that the reference scanner can detect all
+               ;; store references in the jars we disable compression
+               ;; with the "-0" option.
+               (command (if (file-exists? manifest)
+                            `("zip" "-0" "-X" ,jar ,manifest ,@files)
+                            `("zip" "-0" "-X" ,jar ,@files))))
+          (apply invoke command)))
+      (utime jar 0 0)))
+  (for-each (match-lambda
+              ((output . directory)
+               (for-each repack-archive
+                         (find-files directory regular-jar-file-predicate))))
+            outputs))
+
 (define-with-docs install
   "Standard 'install' phase for clojure-build-system."
   (install-jars "./"))
@@ -119,7 +167,8 @@ (define-with-docs %standard-phases
     (replace 'build build)
     (replace 'check check)
     (replace 'install install)
-    (add-after 'install-license-files 'install-doc install-doc)))
+    (add-after 'install-license-files 'install-doc install-doc)
+    (add-after 'reset-gzip-timestamps 'reset-class-timestamps reset-class-timestamps)))
 
 (define* (clojure-build #:key
                         inputs
-- 
2.36.1



[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 519 bytes --]

[bug#56604] [PATCH 6/8] gnu: clojure-algo-generic: Fix test failing under AOT in Clojure 1.11.1.

[r0man]

[-- Attachment #1: Type: text/plain, Size: 1038 bytes --]

---
 gnu/packages/clojure.scm | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)

diff --git a/gnu/packages/clojure.scm b/gnu/packages/clojure.scm
index d51923acdb..36bb27013c 100644
--- a/gnu/packages/clojure.scm
+++ b/gnu/packages/clojure.scm
@@ -274,7 +274,15 @@ (define-public clojure-algo-generic
     (arguments
      '(#:source-dirs '("src/main/clojure/")
        #:test-dirs '("src/test/clojure/")
-       #:doc-dirs '()))
+       #:doc-dirs '()
+       #:phases
+       (modify-phases %standard-phases
+         (add-after 'unpack 'fix-import
+           (lambda _
+             (substitute*
+                 "src/main/clojure/clojure/algo/generic/math_functions.clj"
+               (("clojure.algo.generic.math-functions")
+                "clojure.algo.generic.math-functions\n(:refer-clojure :exclude [abs])")))))))
     (synopsis "Generic versions of common functions")
     (description
      "Generic versions of commonly used functions, implemented as multimethods
-- 
2.36.1



[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 519 bytes --]

[bug#56604] [PATCH 7/8] gnu: clojure-core-match: Update to 1.0.0.---

[r0man]

[-- Attachment #1: Type: text/plain, Size: 2527 bytes --]

---
 gnu/packages/clojure.scm | 46 +++++++++++++++++++---------------------
 1 file changed, 22 insertions(+), 24 deletions(-)

diff --git a/gnu/packages/clojure.scm b/gnu/packages/clojure.scm
index 36bb27013c..0c2bc1f1be 100644
--- a/gnu/packages/clojure.scm
+++ b/gnu/packages/clojure.scm
@@ -319,31 +319,29 @@ (define-public clojure-algo-monads
     (license license:epl1.0)))
 
 (define-public clojure-core-match
-  (let ((commit "1837ffbd4a150e8f3953b2d9ed5cf4a4ad3720a7")
-        (revision "1")) ; this is the 1st commit buildable with clojure 1.9
-    (package
-      (name "clojure-core-match")
-      (version (git-version "0.3.0-alpha5" revision commit))
-      (source (origin
-                (method git-fetch)
-                (uri (git-reference
-                      (url "https://github.com/clojure/core.match")
-                      (commit commit)))
-                (file-name (git-file-name name version))
-                (sha256
-                 (base32
-                  "04bdlp5dgkrqzrz0lw3mfwmygj2218qnm1cz3dkb9wy4m0238s4d"))))
-      (build-system clojure-build-system)
-      (arguments
-       '(#:source-dirs '("src/main/clojure")
-         #:test-dirs '("src/test/clojure")
-         #:doc-dirs '()))
-      (synopsis "Optimized pattern matching for Clojure")
-      (description
-       "An optimized pattern matching library for Clojure.
+  (package
+    (name "clojure-core-match")
+    (version "1.0.0")
+    (source (origin
+              (method git-fetch)
+              (uri (git-reference
+                    (url "https://github.com/clojure/core.match")
+                    (commit (string-append "core.match-" version))))
+              (file-name (git-file-name name version))
+              (sha256
+               (base32
+                "0ajpxjv4yc282cm0jw8819fay2j6jqp9nfy69k7vll09q7vqsd22"))))
+    (build-system clojure-build-system)
+    (arguments
+     '(#:source-dirs '("src/main/clojure")
+       #:test-dirs '("src/test/clojure")
+       #:doc-dirs '()))
+    (synopsis "Optimized pattern matching for Clojure")
+    (description
+     "An optimized pattern matching library for Clojure.
 It supports Clojure 1.5.1 and later as well as ClojureScript.")
-      (home-page "https://github.com/clojure/core.match")
-      (license license:epl1.0))))
+    (home-page "https://github.com/clojure/core.match")
+    (license license:epl1.0)))
 
 (define-public clojure-data-codec
   (package
-- 
2.36.1



[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 519 bytes --]

[bug#56604] [PATCH 8/8] gnu: clojure-instaparse: Update to 1.4.12 (disabled AOT).This patch updates clojure-instaparse to 1.4.12. Due to the following AOT

[r0man]

[-- Attachment #1: Type: text/plain, Size: 2625 bytes --]

This patch updates clojure-instaparse to 1.4.12. Due to the following AOT
related error I disabled AOT compilation for this package.

```
starting phase `build'
Execution error (IllegalArgumentException) at instaparse.auto-flatten-seq/fn$G (auto_flatten_seq.cljc:7).
No implementation of method: :conj-flat of protocol: #'instaparse.auto-flatten-seq/ConjFlat found for class: instaparse.auto_flatten_seq.AutoFlattenSeq
```

This seems to be a known issue with AOT compilation. The issue has been
discussed [1] and seems to be still an open issue.

[1] https://github.com/Engelberg/instaparse/issues/85
---
 gnu/packages/clojure.scm | 20 ++++++++++++++++----
 1 file changed, 16 insertions(+), 4 deletions(-)

diff --git a/gnu/packages/clojure.scm b/gnu/packages/clojure.scm
index 0c2bc1f1be..5332dd499a 100644
--- a/gnu/packages/clojure.scm
+++ b/gnu/packages/clojure.scm
@@ -410,8 +410,7 @@ (define-public clojure-data-xml
     (license license:epl1.0)))
 
 (define-public clojure-instaparse
-  (let ((commit "dcfffad5b065e750f0f5835f017cdd8188b8ca2e")
-        (version "1.4.9")) ; upstream forget to tag this release
+  (let ((version "1.4.12"))
     (package
       (name "clojure-instaparse")
       (version version)
@@ -419,14 +418,27 @@ (define-public clojure-instaparse
                 (method git-fetch)
                 (uri (git-reference
                       (url "https://github.com/Engelberg/instaparse")
-                      (commit commit)))
+                      (commit (string-append "v" version))))
                 (file-name (git-file-name name version))
                 (sha256
                  (base32
                   "002mrgin4z3dqy88r1lak7smd0m7x8d22vmliw0m6w6mh5pa17lk"))))
       (build-system clojure-build-system)
       (arguments
-       '(#:doc-dirs '("docs/")))
+       '(;; Disabled AOT, because of failing test: No implementation of
+         ;; method: :conj-flat of protocol:
+         ;; #'instaparse.auto-flatten-seq/ConjFlat found for class:
+         ;; instaparse.auto_flatten_seq.AutoFlattenSeq
+         #:aot-exclude '(#:all)
+         #:doc-dirs '("docs/")
+         #:phases
+         (modify-phases %standard-phases
+           (add-after 'unpack 'fix-import
+             (lambda _
+               (substitute*
+                   "test/instaparse/defparser_test.cljc"
+                 (("AssertionError")
+                  "Exception")))))))
       (synopsis "No grammar left behind")
       (description
        "Instaparse aims to be the simplest way to build parsers in Clojure.
-- 
2.36.1



[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 519 bytes --]

[bug#56604] Acknowledgement ([PATCH 0/8] Update Clojure to 1.11.1.)

[Roman Scherer]

[-- Attachment #1: Type: text/plain, Size: 58 bytes --]

Sorry, somehow I messed up the the subject lines a bit :/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 528 bytes --]

[bug#56604] [PATCH 0/8] Update Clojure to 1.11.1.

[Maxime Devos]

> This phase makes sure the timestamp of compiled class files is set to
a later
> point in time than the timestamp of the corresponding Clojure source
files. If

Please check that this doesn't create reproducibility issues if you
haven't
already (I haven't looked at the related patches.)

r0man schreef op za 16-07-2022 om 20:17 [+0200]:
> Are we sure that AOT compiling all Clojure source files by default is
> a good
> idea, instead of just compiling user declared namespaces which
> Leiningen and
> friends are doing? WDYT?

I think it's a good idea for the same reason that incremental
compilation is good
and the current Rust packaging is bad,  but I realise this is not
supported in general
for Clojure so we might often need to disable it.

Anyway, possibly related previous e-mails:

https://issues.guix.gnu.org/53765#47 and other AOT mails


> +       '(;; Disabled AOT, because of failing test: No implementation
> of
> +         ;; method: :conj-flat of protocol:
> +         ;; #'instaparse.auto-flatten-seq/ConjFlat found for class:
> +         ;; instaparse.auto_flatten_seq.AutoFlattenSeq

It had been suggested that disabling AOT for test files sometimes is
sufficient, is this the case here?

Greetongs,
Maxime.

bug#56604: [PATCH 0/8] Update Clojure to 1.11.1.

[Ludovic Courtès]

Hi,

r0man <roman@burningswell.com> skribis:

> This phase makes sure the timestamp of compiled class files is set to a later
> point in time than the timestamp of the corresponding Clojure source files. If
> the timestamps of the class and source files are the same, the Clojure
> compiler will compile the sources again which can lead to issues. This problem
> has been discussed here [1]. The suggested solution was to keep/adjust the
> timestamps of the class files.

Sounds reasonable.  It’s a bummer though that the whole phase is pasted
from ant-build-system.scm, the only difference being the timestamps
(1980 instead of 1970).

I added a TODO comment in clojure-build-system.scm when applying the
patch.  Could you follow up with a patch to factorize that?

> Btw, I was a bit surprised that in Guix Clojure packages are AOT compiled. The
> general wisdom in the Clojure community seems to be to avoid AOT compilation
> when distributing libraries, and only AOT compiling Uberjars for final
> deployment. Due to issues like I mentioned in clojure-instaparse.
>
> Are we sure that AOT compiling all Clojure source files by default is a good
> idea, instead of just compiling user declared namespaces which Leiningen and
> friends are doing? WDYT?

Not much, but as you might have seen in ./etc/teams.scm, the project is
finally being structured as teams.  There’s an opportunity for you to
start a Clojure team and to take the lead!  :-)

As a first step, I’d recommend getting in touch with people who have
worked on ‘clojure-build-system’ and packaged things in the past.

>   gnu: clojure-tools-cli: Update to 1.0.206.
>   gnu: clojure-tools-gitlibs: Update to 2.4.181.
>   gnu: clojure-tools-deps-alpha: Update to 0.14.1212.
>   gnu: clojure-tools: Update to 1.11.1.1149.
>   gnu: clojure: Update to 1.11.1.
>   gnu: clojure-algo-generic: Fix test failing under AOT in Clojure 1.11.1.
>   gnu: clojure-core-match: Update to 1.0.0.
>   gnu: clojure-instaparse: Update to 1.4.12 (disabled AOT).

I adjusted all the commit logs to follow our conventions; please
consider doing this next time:

  https://guix.gnu.org/manual/devel/en/html_node/Submitting-Patches.html

The instaparse patch missed the hash update so I did that too.

Thanks!

Ludo’.

[bug#56604] closed (Re: bug#56604: [PATCH 0/8] Update Clojure to 1.11.1.)

[Roman Scherer]

[-- Attachment #1: Type: text/plain, Size: 5718 bytes --]

Hi Maxim and Ludo,

thanks for your replies. Sorry that I was not answering earlier. The heat
killed my laptop and I am having trouble to get set up again.

I follow up when I have my new laptop.

Thanks

On Sat, Jul 23, 2022, 00:12 GNU bug Tracking System <help-debbugs@gnu.org>
wrote:

> Your bug report
>
> #56604: [PATCH 0/8] Update Clojure to 1.11.1.
>
> which was filed against the guix-patches package, has been closed.
>
> The explanation is attached below, along with your original report.
> If you require more details, please reply to 56604@debbugs.gnu.org.
>
> --
> 56604: https://debbugs.gnu.org/cgi/bugreport.cgi?bug=56604
> GNU Bug Tracking System
> Contact help-debbugs@gnu.org with problems
>
>
>
> ---------- Forwarded message ----------
> From: "Ludovic Courtès" <ludo@gnu.org>
> To: r0man <roman@burningswell.com>
> Cc: 56604-done@debbugs.gnu.org
> Bcc:
> Date: Sat, 23 Jul 2022 00:11:05 +0200
> Subject: Re: bug#56604: [PATCH 0/8] Update Clojure to 1.11.1.
> Hi,
>
> r0man <roman@burningswell.com> skribis:
>
> > This phase makes sure the timestamp of compiled class files is set to a
> later
> > point in time than the timestamp of the corresponding Clojure source
> files. If
> > the timestamps of the class and source files are the same, the Clojure
> > compiler will compile the sources again which can lead to issues. This
> problem
> > has been discussed here [1]. The suggested solution was to keep/adjust
> the
> > timestamps of the class files.
>
> Sounds reasonable.  It’s a bummer though that the whole phase is pasted
> from ant-build-system.scm, the only difference being the timestamps
> (1980 instead of 1970).
>
> I added a TODO comment in clojure-build-system.scm when applying the
> patch.  Could you follow up with a patch to factorize that?
>
> > Btw, I was a bit surprised that in Guix Clojure packages are AOT
> compiled. The
> > general wisdom in the Clojure community seems to be to avoid AOT
> compilation
> > when distributing libraries, and only AOT compiling Uberjars for final
> > deployment. Due to issues like I mentioned in clojure-instaparse.
> >
> > Are we sure that AOT compiling all Clojure source files by default is a
> good
> > idea, instead of just compiling user declared namespaces which Leiningen
> and
> > friends are doing? WDYT?
>
> Not much, but as you might have seen in ./etc/teams.scm, the project is
> finally being structured as teams.  There’s an opportunity for you to
> start a Clojure team and to take the lead!  :-)
>
> As a first step, I’d recommend getting in touch with people who have
> worked on ‘clojure-build-system’ and packaged things in the past.
>
> >   gnu: clojure-tools-cli: Update to 1.0.206.
> >   gnu: clojure-tools-gitlibs: Update to 2.4.181.
> >   gnu: clojure-tools-deps-alpha: Update to 0.14.1212.
> >   gnu: clojure-tools: Update to 1.11.1.1149.
> >   gnu: clojure: Update to 1.11.1.
> >   gnu: clojure-algo-generic: Fix test failing under AOT in Clojure
> 1.11.1.
> >   gnu: clojure-core-match: Update to 1.0.0.
> >   gnu: clojure-instaparse: Update to 1.4.12 (disabled AOT).
>
> I adjusted all the commit logs to follow our conventions; please
> consider doing this next time:
>
>   https://guix.gnu.org/manual/devel/en/html_node/Submitting-Patches.html
>
> The instaparse patch missed the hash update so I did that too.
>
> Thanks!
>
> Ludo’.
>
>
>
>
> ---------- Forwarded message ----------
> From: r0man <roman@burningswell.com>
> To: guix-patches@gnu.org
> Cc:
> Bcc:
> Date: Sat, 16 Jul 2022 20:17:34 +0200
> Subject: [PATCH 0/8] Update Clojure to 1.11.1.
> Hello Guix,
>
> This patch series updates Clojure and it's packages to 1.11.1.
>
> It also adds the 'reset-class-timestamps phase to the Clojure build system.
>
> This phase makes sure the timestamp of compiled class files is set to a
> later
> point in time than the timestamp of the corresponding Clojure source
> files. If
> the timestamps of the class and source files are the same, the Clojure
> compiler will compile the sources again which can lead to issues. This
> problem
> has been discussed here [1]. The suggested solution was to keep/adjust the
> timestamps of the class files.
>
> [1] https://www.mail-archive.com/clojure@googlegroups.com/msg99928.html
>
> Could you please review this?
>
> In the patch for clojure-instaparse, I disabled the AOT compilation
> because of
> a known issue. Is that ok?
>
> Btw, I was a bit surprised that in Guix Clojure packages are AOT compiled.
> The
> general wisdom in the Clojure community seems to be to avoid AOT
> compilation
> when distributing libraries, and only AOT compiling Uberjars for final
> deployment. Due to issues like I mentioned in clojure-instaparse.
>
> Are we sure that AOT compiling all Clojure source files by default is a
> good
> idea, instead of just compiling user declared namespaces which Leiningen
> and
> friends are doing? WDYT?
>
> Thanks Roman.
>
> r0man (8):
>   gnu: clojure-tools-cli: Update to 1.0.206.
>   gnu: clojure-tools-gitlibs: Update to 2.4.181.
>   gnu: clojure-tools-deps-alpha: Update to 0.14.1212.
>   gnu: clojure-tools: Update to 1.11.1.1149.
>   gnu: clojure: Update to 1.11.1.
>   gnu: clojure-algo-generic: Fix test failing under AOT in Clojure 1.11.1.
>   gnu: clojure-core-match: Update to 1.0.0.
>   gnu: clojure-instaparse: Update to 1.4.12 (disabled AOT).
>
>  gnu/packages/clojure.scm            | 185 +++++++++++++++++++---------
>  guix/build/clojure-build-system.scm |  53 +++++++-
>  2 files changed, 178 insertions(+), 60 deletions(-)
>
> --
> 2.36.1
>
>

[-- Attachment #2: Type: text/html, Size: 7452 bytes --]

[bug#56604] [PATCH 0/8] Update Clojure to 1.11.1.

[Roman Scherer]

[-- Attachment #1.1: Type: text/plain, Size: 1047 bytes --]


Hi Ludo,

here's the promised patch to follow up with the code duplication I
introduced in my previous patch.

I tested this by compiling Clojure which uses the Ant build system, and
by compiling clojure-tools-deps-alpha which uses the Clojure build
system. Both of the build systems now call the repack-jar function.

Could you have a look at it please?

A related question:

When I run the following commands after modifying the build systems they
run for quite some time, because they were compiling a ton (the jdk,
jetty) of things.

./pre-inst-env guix build clojure
./pre-inst-env guix build clojure-tools

I guess this is expected, since a change in a build system might affect
all packages being built with it. But I was wondering if there is a way
to force only building the packages specified on the command line. Does
such a thing exists?

I was wondering what is the most efficient way to quickly iterate on
changes to a build system, without recompiling the whole world for that
build system. How would you do that?

Thanks, Roman.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1.2: Add repack-jar and use it in Ant & Clojure build systems --]
[-- Type: text/x-diff, Size: 6839 bytes --]

From 756bfd3458ded38e1041ebb255c6b6ffe737732d Mon Sep 17 00:00:00 2001
From: Roman Scherer <roman@burningswell.com>
Date: Mon, 15 Aug 2022 15:29:25 +0000
Subject: [PATCH] build-system: Add repack-jar and use it in Ant & Clojure
 build systems

* guix/build/ant-build-system.scm: Add repack-jar and use it in strip-jar-timestamps
* guix/build/clojure-build-system.scm: Use repack-jar in reset-class-timestamps
---
 guix/build/ant-build-system.scm     | 26 ++++++++------
 guix/build/clojure-build-system.scm | 55 +++++------------------------
 2 files changed, 24 insertions(+), 57 deletions(-)

diff --git a/guix/build/ant-build-system.scm b/guix/build/ant-build-system.scm
index fae1b47ec5..63bdee4651 100644
--- a/guix/build/ant-build-system.scm
+++ b/guix/build/ant-build-system.scm
@@ -195,10 +195,9 @@ (define (generate-index jar)
             outputs)
   #t)
 
-(define* (strip-jar-timestamps #:key outputs
-                               #:allow-other-keys)
-  "Unpack all jar archives, reset the timestamp of all contained files, and
-repack them.  This is necessary to ensure that archives are reproducible."
+(define-public (repack-jar outputs repack-fn)
+  "Unpack all jar archives, invoke repack-fn for each JAR with the directory
+it has been unpacked to, and pack them again."
   (define (repack-archive jar)
     (format #t "repacking ~a\n" jar)
     (let* ((dir (mkdtemp! "jar-contents.XXXXXX"))
@@ -206,13 +205,7 @@ (define (repack-archive jar)
       (with-directory-excursion dir
         (invoke "jar" "xf" jar))
       (delete-file jar)
-      ;; XXX: copied from (gnu build install)
-      (for-each (lambda (file)
-                  (let ((s (lstat file)))
-                    (unless (eq? (stat:type s) 'symlink)
-                      (utime file 0 0 0 0))))
-                (find-files dir #:directories? #t))
-
+      (repack-fn dir)
       ;; The jar tool will always set the timestamp on the manifest file
       ;; and the containing directory to the current time, even when we
       ;; reuse an existing manifest file.  To avoid this we use "zip"
@@ -237,6 +230,17 @@ (define (repack-archive jar)
             outputs)
   #t)
 
+(define* (strip-jar-timestamps #:key outputs
+                               #:allow-other-keys)
+  "Unpack all jar archives, reset the timestamp of all contained files, and
+repack them.  This is necessary to ensure that archives are reproducible."
+  (repack-jar outputs (lambda (dir)
+                        (for-each (lambda (file)
+                                    (let ((s (lstat file)))
+                                      (unless (eq? (stat:type s) 'symlink)
+                                        (utime file 0 0 0 0))))
+                                  (find-files dir #:directories? #t)))))
+
 (define* (check #:key target (make-flags '()) (tests? (not target))
                 (test-target "check")
                 #:allow-other-keys)
diff --git a/guix/build/clojure-build-system.scm b/guix/build/clojure-build-system.scm
index cacbefb386..b82ebc30fe 100644
--- a/guix/build/clojure-build-system.scm
+++ b/guix/build/clojure-build-system.scm
@@ -19,7 +19,7 @@
 (define-module (guix build clojure-build-system)
   #:use-module ((guix build ant-build-system)
                 #:select ((%standard-phases . %standard-phases@ant)
-                          ant-build))
+                          ant-build repack-jar))
   #:use-module (guix build clojure-utils)
   #:use-module (guix build java-utils)
   #:use-module (guix build syscalls)
@@ -112,54 +112,17 @@ (define* (check #:key
                   jar-names)))
   #t)
 
-(define (regular-jar-file? file stat)
-  "Predicate returning true if FILE is ending on '.jar'
-and STAT indicates it is a regular file."
-    (and (string-suffix? ".jar" file)
-         (eq? 'regular (stat:type stat))))
-
-;; XXX: The only difference compared to 'strip-jar-timestamps' in
-;; ant-build-system.scm is the date.  TODO: Adjust and factorize.
 (define* (reset-class-timestamps #:key outputs #:allow-other-keys)
   "Unpack all jar archives, reset the timestamp of all contained class files,
 and repack them.  This is necessary to ensure that archives are reproducible."
-  (define (repack-archive jar)
-    (format #t "resetting class timestamps and repacking ~a\n" jar)
-
-    ;; Note: .class files need to be strictly newer than source files,
-    ;; otherwise the Clojure compiler will recompile sources.
-    (let* ((early-1980 315619200) ; 1980-01-02 UTC
-           (dir (mkdtemp! "jar-contents.XXXXXX"))
-           (manifest (string-append dir "/META-INF/MANIFEST.MF")))
-      (with-directory-excursion dir
-        (invoke "jar" "xf" jar))
-      (delete-file jar)
-      (for-each (lambda (file)
-                  (let ((s (lstat file)))
-                    (unless (eq? (stat:type s) 'symlink)
-                      (when (string-match "^(.*)\\.class$" file)
-                        (utime file early-1980 early-1980)))))
-                (find-files dir #:directories? #t))
-      ;; The jar tool will always set the timestamp on the manifest file
-      ;; and the containing directory to the current time, even when we
-      ;; reuse an existing manifest file.  To avoid this we use "zip"
-      ;; instead of "jar".  It is important that the manifest appears
-      ;; first.
-      (with-directory-excursion dir
-        (let* ((files (find-files "." ".*" #:directories? #t))
-               ;; To ensure that the reference scanner can detect all
-               ;; store references in the jars we disable compression
-               ;; with the "-0" option.
-               (command (if (file-exists? manifest)
-                            `("zip" "-0" "-X" ,jar ,manifest ,@files)
-                            `("zip" "-0" "-X" ,jar ,@files))))
-          (apply invoke command)))
-      (utime jar 0 0)))
-  (for-each (match-lambda
-              ((output . directory)
-               (for-each repack-archive
-                         (find-files directory regular-jar-file?))))
-            outputs))
+  (repack-jar outputs (lambda (dir)
+                        (for-each (lambda (file)
+                                    (let ((s (lstat file))
+                                          (early-1980 315619200)) ; 1980-01-02 UTC
+                                      (unless (eq? (stat:type s) 'symlink)
+                                        (when (string-match "^(.*)\\.class$" file)
+                                          (utime file early-1980 early-1980)))))
+                                  (find-files dir #:directories? #t)))))
 
 (define-with-docs install
   "Standard 'install' phase for clojure-build-system."
-- 
2.37.1


[-- Attachment #1.3: Type: text/plain, Size: 2438 bytes --]


Ludovic Courtès <ludo@gnu.org> writes:

> Hi,
>
> r0man <roman@burningswell.com> skribis:
>
>> This phase makes sure the timestamp of compiled class files is set to a later
>> point in time than the timestamp of the corresponding Clojure source files. If
>> the timestamps of the class and source files are the same, the Clojure
>> compiler will compile the sources again which can lead to issues. This problem
>> has been discussed here [1]. The suggested solution was to keep/adjust the
>> timestamps of the class files.
>
> Sounds reasonable.  It’s a bummer though that the whole phase is pasted
> from ant-build-system.scm, the only difference being the timestamps
> (1980 instead of 1970).
>
> I added a TODO comment in clojure-build-system.scm when applying the
> patch.  Could you follow up with a patch to factorize that?
>
>> Btw, I was a bit surprised that in Guix Clojure packages are AOT compiled. The
>> general wisdom in the Clojure community seems to be to avoid AOT compilation
>> when distributing libraries, and only AOT compiling Uberjars for final
>> deployment. Due to issues like I mentioned in clojure-instaparse.
>>
>> Are we sure that AOT compiling all Clojure source files by default is a good
>> idea, instead of just compiling user declared namespaces which Leiningen and
>> friends are doing? WDYT?
>
> Not much, but as you might have seen in ./etc/teams.scm, the project is
> finally being structured as teams.  There’s an opportunity for you to
> start a Clojure team and to take the lead!  :-)
>
> As a first step, I’d recommend getting in touch with people who have
> worked on ‘clojure-build-system’ and packaged things in the past.
>
>>   gnu: clojure-tools-cli: Update to 1.0.206.
>>   gnu: clojure-tools-gitlibs: Update to 2.4.181.
>>   gnu: clojure-tools-deps-alpha: Update to 0.14.1212.
>>   gnu: clojure-tools: Update to 1.11.1.1149.
>>   gnu: clojure: Update to 1.11.1.
>>   gnu: clojure-algo-generic: Fix test failing under AOT in Clojure 1.11.1.
>>   gnu: clojure-core-match: Update to 1.0.0.
>>   gnu: clojure-instaparse: Update to 1.4.12 (disabled AOT).
>
> I adjusted all the commit logs to follow our conventions; please
> consider doing this next time:
>
>   https://guix.gnu.org/manual/devel/en/html_node/Submitting-Patches.html
>
> The instaparse patch missed the hash update so I did that too.
>
> Thanks!
>
> Ludo’.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 528 bytes --]

[bug#56604] [PATCH 0/8] Update Clojure to 1.11.1.

[Ludovic Courtès]

Hi Roman,

(Catching up after vacation…)

Roman Scherer <roman.scherer@burningswell.com> skribis:

> here's the promised patch to follow up with the code duplication I
> introduced in my previous patch.

Awesome.

> When I run the following commands after modifying the build systems they
> run for quite some time, because they were compiling a ton (the jdk,
> jetty) of things.
>
> ./pre-inst-env guix build clojure
> ./pre-inst-env guix build clojure-tools
>
> I guess this is expected, since a change in a build system might affect
> all packages being built with it. But I was wondering if there is a way
> to force only building the packages specified on the command line. Does
> such a thing exists?

No, it doesn’t exist, because that would be building something
different.  In this case, building everything that depends on
‘ant-build-system.scm’ is unavoidable.

> I was wondering what is the most efficient way to quickly iterate on
> changes to a build system, without recompiling the whole world for that
> build system. How would you do that?

There’s no ideal solution as you’ll have to recompile the world anyway.

When changing build systems, I’d usually stare at my changes for quite
some time first, to make sure I don’t have to rebuild the world on the
next day because of a typo.  :-)

Then, for small local changes, I’d build just the bottom of the
dependency graph to check for breakage (in this case, making sure the
‘strip-jar-timestamps’ phase still works as intended).  Then we can let
ci.guix build the whole thing afterwards, and make sure nothing goes
wrong.

> From 756bfd3458ded38e1041ebb255c6b6ffe737732d Mon Sep 17 00:00:00 2001
> From: Roman Scherer <roman@burningswell.com>
> Date: Mon, 15 Aug 2022 15:29:25 +0000
> Subject: [PATCH] build-system: Add repack-jar and use it in Ant & Clojure
>  build systems
>
> * guix/build/ant-build-system.scm: Add repack-jar and use it in strip-jar-timestamps
> * guix/build/clojure-build-system.scm: Use repack-jar in reset-class-timestamps

Please use the ChangeLog format, specifying procedure/variable names and
their actual changes.

[...]

> +(define-public (repack-jar outputs repack-fn)
> +  "Unpack all jar archives, invoke repack-fn for each JAR with the directory
> +it has been unpacked to, and pack them again."

Instead of ‘define-public’, I’d move ‘repack-jar’ to #:export at the
top, like the other procedures.

But…

> @@ -206,13 +205,7 @@ (define (repack-archive jar)
>        (with-directory-excursion dir
>          (invoke "jar" "xf" jar))
>        (delete-file jar)
> -      ;; XXX: copied from (gnu build install)
> -      (for-each (lambda (file)
> -                  (let ((s (lstat file)))
> -                    (unless (eq? (stat:type s) 'symlink)
> -                      (utime file 0 0 0 0))))
> -                (find-files dir #:directories? #t))
> -
> +      (repack-fn dir)

Looking at the code, the only difference between the two repack
functions is the timestamp, right?  In that case, I’d lean towards
adding a #:jar-timestamp parameter to ‘strip-jar-timestamps’ (rather than
this ‘repack-fn’ argument) that’d be passed to ‘utime’.  The default for
#:jar-timestamp would be '(0 0 0 0); for Clojure we’d set the default
#:jar-timestamp in (guix build-system clojure) to:

  #:jar-timestamp (list early-1980 early-1980)

WDYT?

Thanks,
Ludo’.

[bug#56604] [PATCH 0/8] Update Clojure to 1.11.1.

[Maxime Devos]

[-- Attachment #1.1.1.1: Type: text/plain, Size: 1271 bytes --]


On 01-09-2022 11:09, Ludovic Courtès wrote:
>> I guess this is expected, since a change in a build system might affect
>> all packages being built with it. But I was wondering if there is a way
>> to force only building the packages specified on the command line. Does
>> such a thing exists?
> No, it doesn’t exist, because that would be building something
> different.  In this case, building everything that depends on
> ‘ant-build-system.scm’ is unavoidable.
>
>> I was wondering what is the most efficient way to quickly iterate on
>> changes to a build system, without recompiling the whole world for that
>> build system. How would you do that?
> There’s no ideal solution as you’ll have to recompile the world anyway.

It is possible, actually, to test it for a single package first 
(although without interaction with the command line). In the package you 
would like to test things with, use '=>' in #:imported-modules:

((this module)
  ((that module) => (local-file "customised-that-module.scm")))

and keep the original that/module.scm unmodified.

At least, something like that is done in 'build-program' in 
build-aux/build-self.scm -- I haven't tried it out for something like 
this myself.

Greetings,
Maxime


[-- Attachment #1.1.1.2: Type: text/html, Size: 2010 bytes --]

[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 929 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 236 bytes --]

[bug#56604] [PATCH 0/8] Update Clojure to 1.11.1.

[Roman Scherer]

[-- Attachment #1.1: Type: text/plain, Size: 927 bytes --]


Hi Ludo,

thanks for taking a look. Here's a new patch. I changed the patch to not
use define-public and I'm exporting the repack-jar function now as per
your suggestion. I also updated the commit message to (hopefully) match
the changelog style. Is that correct now?

I did not add the #:jar-timestamp parameter because there is one more
difference. The strip-jar-timestamps function in the Ant build system
changes the timestamp of all files in the JAR. The
reset-class-timestamps function in the Clojure build system sets the
timestamp of only the class files.

I could add another parameter called "extension" to the
strip-jar-timestamps function if you prefer that. It would default to
"*", matching all filenames and use it with "*.class" from the Clojure
build system. I have the feeling the repack-jar function is more
flexible as is, but I'm open to doing this change or any other
suggestion you have.

Wdyt?

Roman


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1.2: 0001-build-system-Remove-code-duplication-in-Ant-Clojure-.patch --]
[-- Type: text/x-diff, Size: 7459 bytes --]

From c3e06172044264ff871cbe8637236415bfd6077f Mon Sep 17 00:00:00 2001
From: Roman Scherer <roman@burningswell.com>
Date: Mon, 15 Aug 2022 15:29:25 +0000
Subject: [PATCH] build-system: Remove code duplication in Ant & Clojure build
 systems

* guix/build/ant-build-system.scm (repack-jar): Add repack-jar function to
unpack, modify, and repack a JAR file.
* guix/build/ant-build-system.scm (strip-jar-timestamps): Use the repack-jar
function to set the timestamps of all files in the JAR file to 1980-01-01.
* guix/build/clojure-build-system.scm (reset-class-timestamps): Use the
repack-jar function from the Ant build system to set the timestamps of all
class files in the JAR file to 1980-01-02 to prevent the Clojure compiler from
re-compiling Clojure source files.
---
 guix/build/ant-build-system.scm     | 29 ++++++++-------
 guix/build/clojure-build-system.scm | 55 +++++------------------------
 2 files changed, 26 insertions(+), 58 deletions(-)

diff --git a/guix/build/ant-build-system.scm b/guix/build/ant-build-system.scm
index fae1b47ec5..dec261cb82 100644
--- a/guix/build/ant-build-system.scm
+++ b/guix/build/ant-build-system.scm
@@ -27,7 +27,8 @@ (define-module (guix build ant-build-system)
   #:use-module (srfi srfi-1)
   #:use-module (srfi srfi-26)
   #:export (%standard-phases
-            ant-build))
+            ant-build
+            repack-jar))
 
 ;; Commentary:
 ;;
@@ -195,10 +196,9 @@ (define (generate-index jar)
             outputs)
   #t)
 
-(define* (strip-jar-timestamps #:key outputs
-                               #:allow-other-keys)
-  "Unpack all jar archives, reset the timestamp of all contained files, and
-repack them.  This is necessary to ensure that archives are reproducible."
+(define (repack-jar outputs repack-fn)
+  "Unpack all jar archives, invoke repack-fn for each JAR with the directory
+it has been unpacked to, and pack them again."
   (define (repack-archive jar)
     (format #t "repacking ~a\n" jar)
     (let* ((dir (mkdtemp! "jar-contents.XXXXXX"))
@@ -206,13 +206,7 @@ (define (repack-archive jar)
       (with-directory-excursion dir
         (invoke "jar" "xf" jar))
       (delete-file jar)
-      ;; XXX: copied from (gnu build install)
-      (for-each (lambda (file)
-                  (let ((s (lstat file)))
-                    (unless (eq? (stat:type s) 'symlink)
-                      (utime file 0 0 0 0))))
-                (find-files dir #:directories? #t))
-
+      (repack-fn dir)
       ;; The jar tool will always set the timestamp on the manifest file
       ;; and the containing directory to the current time, even when we
       ;; reuse an existing manifest file.  To avoid this we use "zip"
@@ -237,6 +231,17 @@ (define (repack-archive jar)
             outputs)
   #t)
 
+(define* (strip-jar-timestamps #:key outputs
+                               #:allow-other-keys)
+  "Unpack all jar archives, reset the timestamp of all contained files, and
+repack them.  This is necessary to ensure that archives are reproducible."
+  (repack-jar outputs (lambda (dir)
+                        (for-each (lambda (file)
+                                    (let ((s (lstat file)))
+                                      (unless (eq? (stat:type s) 'symlink)
+                                        (utime file 0 0 0 0))))
+                                  (find-files dir #:directories? #t)))))
+
 (define* (check #:key target (make-flags '()) (tests? (not target))
                 (test-target "check")
                 #:allow-other-keys)
diff --git a/guix/build/clojure-build-system.scm b/guix/build/clojure-build-system.scm
index cacbefb386..b82ebc30fe 100644
--- a/guix/build/clojure-build-system.scm
+++ b/guix/build/clojure-build-system.scm
@@ -19,7 +19,7 @@
 (define-module (guix build clojure-build-system)
   #:use-module ((guix build ant-build-system)
                 #:select ((%standard-phases . %standard-phases@ant)
-                          ant-build))
+                          ant-build repack-jar))
   #:use-module (guix build clojure-utils)
   #:use-module (guix build java-utils)
   #:use-module (guix build syscalls)
@@ -112,54 +112,17 @@ (define* (check #:key
                   jar-names)))
   #t)
 
-(define (regular-jar-file? file stat)
-  "Predicate returning true if FILE is ending on '.jar'
-and STAT indicates it is a regular file."
-    (and (string-suffix? ".jar" file)
-         (eq? 'regular (stat:type stat))))
-
-;; XXX: The only difference compared to 'strip-jar-timestamps' in
-;; ant-build-system.scm is the date.  TODO: Adjust and factorize.
 (define* (reset-class-timestamps #:key outputs #:allow-other-keys)
   "Unpack all jar archives, reset the timestamp of all contained class files,
 and repack them.  This is necessary to ensure that archives are reproducible."
-  (define (repack-archive jar)
-    (format #t "resetting class timestamps and repacking ~a\n" jar)
-
-    ;; Note: .class files need to be strictly newer than source files,
-    ;; otherwise the Clojure compiler will recompile sources.
-    (let* ((early-1980 315619200) ; 1980-01-02 UTC
-           (dir (mkdtemp! "jar-contents.XXXXXX"))
-           (manifest (string-append dir "/META-INF/MANIFEST.MF")))
-      (with-directory-excursion dir
-        (invoke "jar" "xf" jar))
-      (delete-file jar)
-      (for-each (lambda (file)
-                  (let ((s (lstat file)))
-                    (unless (eq? (stat:type s) 'symlink)
-                      (when (string-match "^(.*)\\.class$" file)
-                        (utime file early-1980 early-1980)))))
-                (find-files dir #:directories? #t))
-      ;; The jar tool will always set the timestamp on the manifest file
-      ;; and the containing directory to the current time, even when we
-      ;; reuse an existing manifest file.  To avoid this we use "zip"
-      ;; instead of "jar".  It is important that the manifest appears
-      ;; first.
-      (with-directory-excursion dir
-        (let* ((files (find-files "." ".*" #:directories? #t))
-               ;; To ensure that the reference scanner can detect all
-               ;; store references in the jars we disable compression
-               ;; with the "-0" option.
-               (command (if (file-exists? manifest)
-                            `("zip" "-0" "-X" ,jar ,manifest ,@files)
-                            `("zip" "-0" "-X" ,jar ,@files))))
-          (apply invoke command)))
-      (utime jar 0 0)))
-  (for-each (match-lambda
-              ((output . directory)
-               (for-each repack-archive
-                         (find-files directory regular-jar-file?))))
-            outputs))
+  (repack-jar outputs (lambda (dir)
+                        (for-each (lambda (file)
+                                    (let ((s (lstat file))
+                                          (early-1980 315619200)) ; 1980-01-02 UTC
+                                      (unless (eq? (stat:type s) 'symlink)
+                                        (when (string-match "^(.*)\\.class$" file)
+                                          (utime file early-1980 early-1980)))))
+                                  (find-files dir #:directories? #t)))))
 
 (define-with-docs install
   "Standard 'install' phase for clojure-build-system."
-- 
2.37.2


[-- Attachment #1.3: Type: text/plain, Size: 3667 bytes --]


Ludovic Courtès <ludo@gnu.org> writes:

> Hi Roman,
>
> (Catching up after vacation…)
>
> Roman Scherer <roman.scherer@burningswell.com> skribis:
>
>> here's the promised patch to follow up with the code duplication I
>> introduced in my previous patch.
>
> Awesome.
>
>> When I run the following commands after modifying the build systems they
>> run for quite some time, because they were compiling a ton (the jdk,
>> jetty) of things.
>>
>> ./pre-inst-env guix build clojure
>> ./pre-inst-env guix build clojure-tools
>>
>> I guess this is expected, since a change in a build system might affect
>> all packages being built with it. But I was wondering if there is a way
>> to force only building the packages specified on the command line. Does
>> such a thing exists?
>
> No, it doesn’t exist, because that would be building something
> different.  In this case, building everything that depends on
> ‘ant-build-system.scm’ is unavoidable.
>
>> I was wondering what is the most efficient way to quickly iterate on
>> changes to a build system, without recompiling the whole world for that
>> build system. How would you do that?
>
> There’s no ideal solution as you’ll have to recompile the world anyway.
>
> When changing build systems, I’d usually stare at my changes for quite
> some time first, to make sure I don’t have to rebuild the world on the
> next day because of a typo.  :-)
>
> Then, for small local changes, I’d build just the bottom of the
> dependency graph to check for breakage (in this case, making sure the
> ‘strip-jar-timestamps’ phase still works as intended).  Then we can let
> ci.guix build the whole thing afterwards, and make sure nothing goes
> wrong.
>
>> From 756bfd3458ded38e1041ebb255c6b6ffe737732d Mon Sep 17 00:00:00 2001
>> From: Roman Scherer <roman@burningswell.com>
>> Date: Mon, 15 Aug 2022 15:29:25 +0000
>> Subject: [PATCH] build-system: Add repack-jar and use it in Ant & Clojure
>>  build systems
>>
>> * guix/build/ant-build-system.scm: Add repack-jar and use it in strip-jar-timestamps
>> * guix/build/clojure-build-system.scm: Use repack-jar in reset-class-timestamps
>
> Please use the ChangeLog format, specifying procedure/variable names and
> their actual changes.
>
> [...]
>
>> +(define-public (repack-jar outputs repack-fn)
>> +  "Unpack all jar archives, invoke repack-fn for each JAR with the directory
>> +it has been unpacked to, and pack them again."
>
> Instead of ‘define-public’, I’d move ‘repack-jar’ to #:export at the
> top, like the other procedures.
>
> But…
>
>> @@ -206,13 +205,7 @@ (define (repack-archive jar)
>>        (with-directory-excursion dir
>>          (invoke "jar" "xf" jar))
>>        (delete-file jar)
>> -      ;; XXX: copied from (gnu build install)
>> -      (for-each (lambda (file)
>> -                  (let ((s (lstat file)))
>> -                    (unless (eq? (stat:type s) 'symlink)
>> -                      (utime file 0 0 0 0))))
>> -                (find-files dir #:directories? #t))
>> -
>> +      (repack-fn dir)
>
> Looking at the code, the only difference between the two repack
> functions is the timestamp, right?  In that case, I’d lean towards
> adding a #:jar-timestamp parameter to ‘strip-jar-timestamps’ (rather than
> this ‘repack-fn’ argument) that’d be passed to ‘utime’.  The default for
> #:jar-timestamp would be '(0 0 0 0); for Clojure we’d set the default
> #:jar-timestamp in (guix build-system clojure) to:
>
>   #:jar-timestamp (list early-1980 early-1980)
>
> WDYT?
>
> Thanks,
> Ludo’.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 528 bytes --]

[bug#56604] [PATCH 0/8] Update Clojure to 1.11.1.

[Roman Scherer]

[-- Attachment #1: Type: text/plain, Size: 1636 bytes --]


Hi Maxime,

I just rebuilt the world again (to be sure it works), but will give this
a try the next time. Thanks for the tip!

Roman

Maxime Devos <maximedevos@telenet.be> writes:

> [[PGP Signed Part:Undecided]]
> On 01-09-2022 11:09, Ludovic Courtès wrote:
>
>  I guess this is expected, since a change in a build system might affect
> all packages being built with it. But I was wondering if there is a way
> to force only building the packages specified on the command line. Does
> such a thing exists?
>
> No, it doesn’t exist, because that would be building something
> different.  In this case, building everything that depends on
> ‘ant-build-system.scm’ is unavoidable.
>
>  I was wondering what is the most efficient way to quickly iterate on
> changes to a build system, without recompiling the whole world for that
> build system. How would you do that?
>
> There’s no ideal solution as you’ll have to recompile the world anyway.
>
> It is possible, actually, to test it for a single package first (although without interaction with the command line). In the
> package you would like to test things with, use '=>' in #:imported-modules:
>
> ((this module)
>  ((that module) => (local-file "customised-that-module.scm")))
>
> and keep the original that/module.scm unmodified.
>
> At least, something like that is done in 'build-program' in build-aux/build-self.scm -- I haven't tried it out for something
> like this myself.
>
> Greetings,
> Maxime
>
> [4. OpenPGP public key --- application/pgp-keys; OpenPGP_0x49E3EE22191725EE.asc]...
>
> [[End of PGP Signed Part]]

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 528 bytes --]