From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp12.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms5.migadu.com with LMTPS id CHkCCbdr+mLiHgAAbAwnHQ (envelope-from ) for ; Mon, 15 Aug 2022 17:52:23 +0200 Received: from aspmx1.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp12.migadu.com with LMTPS id 8EsfCbdr+mJpUwAAauVa8A (envelope-from ) for ; Mon, 15 Aug 2022 17:52:23 +0200 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id B35F13A6E7 for ; Mon, 15 Aug 2022 17:52:22 +0200 (CEST) Received: from localhost ([::1]:34806 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1oNcO5-0005mk-OT for larch@yhetil.org; Mon, 15 Aug 2022 11:52:21 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:53914) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1oNcKu-0004VO-KM for guix-patches@gnu.org; Mon, 15 Aug 2022 11:49:06 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:54153) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1oNcKs-0000T5-IT for guix-patches@gnu.org; Mon, 15 Aug 2022 11:49:03 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1oNcKs-0007Zt-3v for guix-patches@gnu.org; Mon, 15 Aug 2022 11:49:02 -0400 X-Loop: help-debbugs@gnu.org Subject: [bug#56604] [PATCH 0/8] Update Clojure to 1.11.1. Resent-From: Roman Scherer Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Mon, 15 Aug 2022 15:49:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 56604 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Ludovic =?UTF-8?Q?Court=C3=A8s?= Cc: 56604-done@debbugs.gnu.org Received: via spool by 56604-done@debbugs.gnu.org id=D56604.166057853629118 (code D ref 56604); Mon, 15 Aug 2022 15:49:02 +0000 Received: (at 56604-done) by debbugs.gnu.org; 15 Aug 2022 15:48:56 +0000 Received: from localhost ([127.0.0.1]:43902 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1oNcKm-0007ZZ-3E for submit@debbugs.gnu.org; Mon, 15 Aug 2022 11:48:56 -0400 Received: from mail-wr1-f44.google.com ([209.85.221.44]:43677) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1oNcKk-0007ZL-3B for 56604-done@debbugs.gnu.org; Mon, 15 Aug 2022 11:48:54 -0400 Received: by mail-wr1-f44.google.com with SMTP id n4so9492622wrp.10 for <56604-done@debbugs.gnu.org>; Mon, 15 Aug 2022 08:48:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=burningswell-com.20210112.gappssmtp.com; s=20210112; h=mime-version:message-id:in-reply-to:date:subject:cc:to:from :user-agent:references:from:to:cc; bh=h74UlrG8AkaArG59KK2tF2L3UmfXS9gip0vpFTWMv9M=; b=bc+PumEiYr21O8rVaC0cqxLZLqEUKBfLw59nb70YncCgqmBnkx+tod0IKD55dhk5m3 5dzmx6O0QUShwjNzZCf8YoPp4LxQHovVzFjXDWxph39Enay3sr16WT+Mu+BXL/aAgN1V baoFPGb4MnLVrZvXSG9iaTpO+U3rsAJtAQknuMJH7PMqcTRYUy8QMjsqPvDPOEPOdiq+ 4MC4a20q9pkdzBOWIJUPG86h6vLqDh1gURUYXJViMYNEuiBm6EinZBPlvgb62gTqEuDz ziZwbVrJqlqUsmYyDMMsy/lMMuHZ8PW/QlPm4bDMVf9lGIrkwb3tDgOOTy5LtEa4YM2c oEww== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=mime-version:message-id:in-reply-to:date:subject:cc:to:from :user-agent:references:x-gm-message-state:from:to:cc; bh=h74UlrG8AkaArG59KK2tF2L3UmfXS9gip0vpFTWMv9M=; b=2fuErp9wYmhUu7kl38RopRjHUmnATalt8oXJoLxzTzIEElELAViaVdqXGzmO4LVazU 99B7almINzTf2XHKq31ED46tUJturMtLLTcnuyaNAk8EyZN2rxh+FVax/QwnIrLLowZH AnqmqqjELjGIiEFPEGZ9f9as5WQdABSVrGDmRRaD3DJgwml/DE86S/J9YtAfpr2vmuPE 73+PG9TirOadcLAo9lfnGXufJpJjdJAQ7Nz+fuNYOEmc0XfswgjtFR60eOBJbUSGFDpN +HIQ84w/ZCf2ZC1ufNV5+6pl0L3RN4p0RQH2/ugDPg/7awYkRyh0kXAoPiI5/G6/vFJs 6ung== X-Gm-Message-State: ACgBeo3hTv3xrdixGuIh1GLQ/U1V3CdZeD9YYx3a51hGhksXDK227mXZ 9K2dB5jsVf3pgKlPngDrJeX1EcJysXp4hNfI X-Google-Smtp-Source: AA6agR6v0Isbc0TtIhesvCDrSXdfqhJIneDBlj92OQeqwLDrPDvIFHwVnW+Wfldlum7CnXxqx9A4bQ== X-Received: by 2002:a05:6000:18af:b0:222:c48d:9064 with SMTP id b15-20020a05600018af00b00222c48d9064mr9433891wri.18.1660578527780; Mon, 15 Aug 2022 08:48:47 -0700 (PDT) Received: from precision (tmo-119-192.customers.d1-online.com. [80.187.119.192]) by smtp.gmail.com with ESMTPSA id j3-20020a5d6043000000b0021d6dad334bsm7745313wrt.4.2022.08.15.08.48.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 Aug 2022 08:48:47 -0700 (PDT) References: <87fsisg5mu.fsf@gnu.org> User-agent: mu4e 1.8.7; emacs 28.1 From: Roman Scherer Date: Mon, 15 Aug 2022 15:36:32 +0000 In-reply-to: <87fsisg5mu.fsf@gnu.org> Message-ID: <87zgg55wyq.fsf@burningswell.com> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="==-=-="; micalg=pgp-sha256; protocol="application/pgp-signature" X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+larch=yhetil.org@gnu.org Sender: "Guix-patches" X-Migadu-Flow: FLOW_IN X-Migadu-To: larch@yhetil.org X-Migadu-Country: US ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1660578742; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:resent-cc:resent-from:resent-sender: resent-message-id:in-reply-to:in-reply-to:references:references: list-id:list-help:list-unsubscribe:list-subscribe:list-post: dkim-signature; bh=h74UlrG8AkaArG59KK2tF2L3UmfXS9gip0vpFTWMv9M=; b=PafqrE3OoBbYPtuNwyVr/hdruDq2jJ7WTEQYys3oK5ben6uayG38OBIThfGV6ltzwjSe0G Kg0XUnbUmVIlZ2PZprOmFRBsLJQ3d2427QoMwnjPb0XIFpBr66/r5F2hkVHoSPqOmNoS13 TgV72QlACfOcufzu54P6IBe41gUFeH3JCMdKo+wnmomnDtxtFYrq8DzYKCYdywLfx8FG5D yhLNrt/vKhMcyRSQwFwG0WukJM2GhiGssvu6yyYE8/w3eWakw1bK3BMQRX0ChNqNeRuLyl K4wuaAeHmJ2NCNgSb6CTh01GtPjZlZKR0OsgyEVDqGnZ8PiVCEOmjLLDTJNeiw== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1660578742; a=rsa-sha256; cv=none; b=Vk2j62zLoVh5T0MmvWtnOlzAFMQGJmm8AeYzEHKvzer1Xx/haf8E0ZOCZxj0ht5ojafN+n ELLOUnOEGVtpjOvyyv/hRxv2dyLx9+3Zwd5iwRAfX4QLjd3F13WWIqrcX78kEek8vIxUpO mzZ4K9CBlkJN7xlbsGuV464JWdWw6vguxF1m7LBme3/xHiTRpctsHluF5Jy7uTG43hxx64 yZbvRvgHBmPKQAJ4kk6suj6aLN7FmnEkyNjzgmKpYEzEMqp18oYVleiTCliSKXVNZNP2SQ TJRUzp50UXyuqwf4bczLY2HoVbKlrqSwx4OizJxkVknZcAtS4gcfpUenHP9c+g== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=burningswell-com.20210112.gappssmtp.com header.s=20210112 header.b=bc+PumEi; dmarc=none; spf=pass (aspmx1.migadu.com: domain of "guix-patches-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-patches-bounces+larch=yhetil.org@gnu.org" X-Migadu-Spam-Score: 0.54 Authentication-Results: aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=burningswell-com.20210112.gappssmtp.com header.s=20210112 header.b=bc+PumEi; dmarc=none; spf=pass (aspmx1.migadu.com: domain of "guix-patches-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-patches-bounces+larch=yhetil.org@gnu.org" X-Migadu-Queue-Id: B35F13A6E7 X-Spam-Score: 0.54 X-Migadu-Scanner: scn1.migadu.com X-TUID: PqSIjV5lyN0p --==-=-= Content-Type: multipart/mixed; boundary="=-=-=" --=-=-= Content-Type: text/plain Hi Ludo, here's the promised patch to follow up with the code duplication I introduced in my previous patch. I tested this by compiling Clojure which uses the Ant build system, and by compiling clojure-tools-deps-alpha which uses the Clojure build system. Both of the build systems now call the repack-jar function. Could you have a look at it please? A related question: When I run the following commands after modifying the build systems they run for quite some time, because they were compiling a ton (the jdk, jetty) of things. ./pre-inst-env guix build clojure ./pre-inst-env guix build clojure-tools I guess this is expected, since a change in a build system might affect all packages being built with it. But I was wondering if there is a way to force only building the packages specified on the command line. Does such a thing exists? I was wondering what is the most efficient way to quickly iterate on changes to a build system, without recompiling the whole world for that build system. How would you do that? Thanks, Roman. --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=0001-build-system-Add-repack-jar-and-use-it-in-Ant-Clojur.patch Content-Transfer-Encoding: quoted-printable Content-Description: Add repack-jar and use it in Ant & Clojure build systems From=20756bfd3458ded38e1041ebb255c6b6ffe737732d Mon Sep 17 00:00:00 2001 From: Roman Scherer Date: Mon, 15 Aug 2022 15:29:25 +0000 Subject: [PATCH] build-system: Add repack-jar and use it in Ant & Clojure build systems * guix/build/ant-build-system.scm: Add repack-jar and use it in strip-jar-t= imestamps * guix/build/clojure-build-system.scm: Use repack-jar in reset-class-timest= amps =2D-- guix/build/ant-build-system.scm | 26 ++++++++------ guix/build/clojure-build-system.scm | 55 +++++------------------------ 2 files changed, 24 insertions(+), 57 deletions(-) diff --git a/guix/build/ant-build-system.scm b/guix/build/ant-build-system.= scm index fae1b47ec5..63bdee4651 100644 =2D-- a/guix/build/ant-build-system.scm +++ b/guix/build/ant-build-system.scm @@ -195,10 +195,9 @@ (define (generate-index jar) outputs) #t) =20 =2D(define* (strip-jar-timestamps #:key outputs =2D #:allow-other-keys) =2D "Unpack all jar archives, reset the timestamp of all contained files, = and =2Drepack them. This is necessary to ensure that archives are reproducible= ." +(define-public (repack-jar outputs repack-fn) + "Unpack all jar archives, invoke repack-fn for each JAR with the directo= ry +it has been unpacked to, and pack them again." (define (repack-archive jar) (format #t "repacking ~a\n" jar) (let* ((dir (mkdtemp! "jar-contents.XXXXXX")) @@ -206,13 +205,7 @@ (define (repack-archive jar) (with-directory-excursion dir (invoke "jar" "xf" jar)) (delete-file jar) =2D ;; XXX: copied from (gnu build install) =2D (for-each (lambda (file) =2D (let ((s (lstat file))) =2D (unless (eq? (stat:type s) 'symlink) =2D (utime file 0 0 0 0)))) =2D (find-files dir #:directories? #t)) =2D + (repack-fn dir) ;; The jar tool will always set the timestamp on the manifest file ;; and the containing directory to the current time, even when we ;; reuse an existing manifest file. To avoid this we use "zip" @@ -237,6 +230,17 @@ (define (repack-archive jar) outputs) #t) =20 +(define* (strip-jar-timestamps #:key outputs + #:allow-other-keys) + "Unpack all jar archives, reset the timestamp of all contained files, and +repack them. This is necessary to ensure that archives are reproducible." + (repack-jar outputs (lambda (dir) + (for-each (lambda (file) + (let ((s (lstat file))) + (unless (eq? (stat:type s) 'symlink) + (utime file 0 0 0 0)))) + (find-files dir #:directories? #t))))) + (define* (check #:key target (make-flags '()) (tests? (not target)) (test-target "check") #:allow-other-keys) diff --git a/guix/build/clojure-build-system.scm b/guix/build/clojure-build= -system.scm index cacbefb386..b82ebc30fe 100644 =2D-- a/guix/build/clojure-build-system.scm +++ b/guix/build/clojure-build-system.scm @@ -19,7 +19,7 @@ (define-module (guix build clojure-build-system) #:use-module ((guix build ant-build-system) #:select ((%standard-phases . %standard-phases@ant) =2D ant-build)) + ant-build repack-jar)) #:use-module (guix build clojure-utils) #:use-module (guix build java-utils) #:use-module (guix build syscalls) @@ -112,54 +112,17 @@ (define* (check #:key jar-names))) #t) =20 =2D(define (regular-jar-file? file stat) =2D "Predicate returning true if FILE is ending on '.jar' =2Dand STAT indicates it is a regular file." =2D (and (string-suffix? ".jar" file) =2D (eq? 'regular (stat:type stat)))) =2D =2D;; XXX: The only difference compared to 'strip-jar-timestamps' in =2D;; ant-build-system.scm is the date. TODO: Adjust and factorize. (define* (reset-class-timestamps #:key outputs #:allow-other-keys) "Unpack all jar archives, reset the timestamp of all contained class fil= es, and repack them. This is necessary to ensure that archives are reproducib= le." =2D (define (repack-archive jar) =2D (format #t "resetting class timestamps and repacking ~a\n" jar) =2D =2D ;; Note: .class files need to be strictly newer than source files, =2D ;; otherwise the Clojure compiler will recompile sources. =2D (let* ((early-1980 315619200) ; 1980-01-02 UTC =2D (dir (mkdtemp! "jar-contents.XXXXXX")) =2D (manifest (string-append dir "/META-INF/MANIFEST.MF"))) =2D (with-directory-excursion dir =2D (invoke "jar" "xf" jar)) =2D (delete-file jar) =2D (for-each (lambda (file) =2D (let ((s (lstat file))) =2D (unless (eq? (stat:type s) 'symlink) =2D (when (string-match "^(.*)\\.class$" file) =2D (utime file early-1980 early-1980))))) =2D (find-files dir #:directories? #t)) =2D ;; The jar tool will always set the timestamp on the manifest file =2D ;; and the containing directory to the current time, even when we =2D ;; reuse an existing manifest file. To avoid this we use "zip" =2D ;; instead of "jar". It is important that the manifest appears =2D ;; first. =2D (with-directory-excursion dir =2D (let* ((files (find-files "." ".*" #:directories? #t)) =2D ;; To ensure that the reference scanner can detect all =2D ;; store references in the jars we disable compression =2D ;; with the "-0" option. =2D (command (if (file-exists? manifest) =2D `("zip" "-0" "-X" ,jar ,manifest ,@files) =2D `("zip" "-0" "-X" ,jar ,@files)))) =2D (apply invoke command))) =2D (utime jar 0 0))) =2D (for-each (match-lambda =2D ((output . directory) =2D (for-each repack-archive =2D (find-files directory regular-jar-file?)))) =2D outputs)) + (repack-jar outputs (lambda (dir) + (for-each (lambda (file) + (let ((s (lstat file)) + (early-1980 315619200)) ; 1980-0= 1-02 UTC + (unless (eq? (stat:type s) 'symlink) + (when (string-match "^(.*)\\.class= $" file) + (utime file early-1980 early-198= 0))))) + (find-files dir #:directories? #t))))) =20 (define-with-docs install "Standard 'install' phase for clojure-build-system." =2D-=20 2.37.1 --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Ludovic Court=C3=A8s writes: > Hi, > > r0man skribis: > >> This phase makes sure the timestamp of compiled class files is set to a = later >> point in time than the timestamp of the corresponding Clojure source fil= es. If >> the timestamps of the class and source files are the same, the Clojure >> compiler will compile the sources again which can lead to issues. This p= roblem >> has been discussed here [1]. The suggested solution was to keep/adjust t= he >> timestamps of the class files. > > Sounds reasonable. It=E2=80=99s a bummer though that the whole phase is = pasted > from ant-build-system.scm, the only difference being the timestamps > (1980 instead of 1970). > > I added a TODO comment in clojure-build-system.scm when applying the > patch. Could you follow up with a patch to factorize that? > >> Btw, I was a bit surprised that in Guix Clojure packages are AOT compile= d. The >> general wisdom in the Clojure community seems to be to avoid AOT compila= tion >> when distributing libraries, and only AOT compiling Uberjars for final >> deployment. Due to issues like I mentioned in clojure-instaparse. >> >> Are we sure that AOT compiling all Clojure source files by default is a = good >> idea, instead of just compiling user declared namespaces which Leiningen= and >> friends are doing? WDYT? > > Not much, but as you might have seen in ./etc/teams.scm, the project is > finally being structured as teams. There=E2=80=99s an opportunity for yo= u to > start a Clojure team and to take the lead! :-) > > As a first step, I=E2=80=99d recommend getting in touch with people who h= ave > worked on =E2=80=98clojure-build-system=E2=80=99 and packaged things in t= he past. > >> gnu: clojure-tools-cli: Update to 1.0.206. >> gnu: clojure-tools-gitlibs: Update to 2.4.181. >> gnu: clojure-tools-deps-alpha: Update to 0.14.1212. >> gnu: clojure-tools: Update to 1.11.1.1149. >> gnu: clojure: Update to 1.11.1. >> gnu: clojure-algo-generic: Fix test failing under AOT in Clojure 1.11.= 1. >> gnu: clojure-core-match: Update to 1.0.0. >> gnu: clojure-instaparse: Update to 1.4.12 (disabled AOT). > > I adjusted all the commit logs to follow our conventions; please > consider doing this next time: > > https://guix.gnu.org/manual/devel/en/html_node/Submitting-Patches.html > > The instaparse patch missed the hash update so I did that too. > > Thanks! > > Ludo=E2=80=99. --=-=-=-- --==-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQFTBAEBCAA9FiEE0iajOdjfRIFd3gygPdpSUn0qwZkFAmL6at0fHHJvbWFuLnNj aGVyZXJAYnVybmluZ3N3ZWxsLmNvbQAKCRA92lJSfSrBmTSbCAChA5Zmz5Uw95Tn y2/F76iHia9IGhAbK7KACAi7IRkNXu22aDvsFtOMaAn3jso+VeciA7JuqDk0NfUp SOfxnPDhQEvt+GP5rpFcz3tXdmWKwM53IRmV1iCYALpgc2fQQIohfMHJRGtcsZA6 ea1gSmZD81jCElpJO3P6by4IqfBGFZ+VshQise2lbsmIOM1Li/ny7XCqiTzh/h7+ Ol1tnDcBPorVA3z8Q39FaJG08+9spvSqvAkIn871MHxeU4Hvn6MsfuM2h5WCJBHY +2+s68dYz35jK7aZacdlFrulucpstApkCzXu/qRj6/yoU5dnje2bBtfvHURONGY7 Bd7aKeIB =k8Hv -----END PGP SIGNATURE----- --==-=-=--