From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp1 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms11 with LMTPS id SJZEBSM79l98AwAA0tVLHw (envelope-from ) for ; Wed, 06 Jan 2021 22:35:15 +0000 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp1 with LMTPS id IAYGASM79l86bgAAbx9fmQ (envelope-from ) for ; Wed, 06 Jan 2021 22:35:15 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 3DBC2940149 for ; Wed, 6 Jan 2021 22:35:14 +0000 (UTC) Received: from localhost ([::1]:54484 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kxHOZ-0000kC-HR for larch@yhetil.org; Wed, 06 Jan 2021 17:35:11 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:40528) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kxHOQ-0000k4-Dn for bug-guix@gnu.org; Wed, 06 Jan 2021 17:35:02 -0500 Received: from debbugs.gnu.org ([209.51.188.43]:34263) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1kxHOQ-0007iL-6a for bug-guix@gnu.org; Wed, 06 Jan 2021 17:35:02 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1kxHOQ-0005fk-2e for bug-guix@gnu.org; Wed, 06 Jan 2021 17:35:02 -0500 X-Loop: help-debbugs@gnu.org Subject: bug#45675: Zip-based archives store timestamps Resent-From: Miguel =?UTF-8?Q?=C3=81ngel?= Arruga Vivas Original-Sender: "Debbugs-submit" Resent-CC: bug-guix@gnu.org Resent-Date: Wed, 06 Jan 2021 22:35:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 45675 X-GNU-PR-Package: guix X-GNU-PR-Keywords: To: Julien Lepiller Received: via spool by 45675-submit@debbugs.gnu.org id=B45675.160997245321742 (code B ref 45675); Wed, 06 Jan 2021 22:35:02 +0000 Received: (at 45675) by debbugs.gnu.org; 6 Jan 2021 22:34:13 +0000 Received: from localhost ([127.0.0.1]:45809 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kxHNd-0005ea-01 for submit@debbugs.gnu.org; Wed, 06 Jan 2021 17:34:13 -0500 Received: from mail-wr1-f46.google.com ([209.85.221.46]:41696) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kxHNb-0005eO-L4 for 45675@debbugs.gnu.org; Wed, 06 Jan 2021 17:34:12 -0500 Received: by mail-wr1-f46.google.com with SMTP id a12so3782061wrv.8 for <45675@debbugs.gnu.org>; Wed, 06 Jan 2021 14:34:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:references:date:in-reply-to:message-id :user-agent:mime-version; bh=fCdFDDC/xpCRoDsCgcClRvhiStXO6uM3h7Y5XXg0FZQ=; b=k5ipH7yAQ9Hf5DtrhuP6bey380IZdf5FK490WQDt60bB0JTJoWi3y2tCKpF7Rewrdf VOpZNwAB4RDDQthr+CTRgOIL3FFOlNvzLG+Wy86mFv29rt99QDANKinXTOoxnXOW4WsS BohiI6H+t4AxHSmM2ihMnsZvJNdrkTu1jRxmat2D0ormwNTsyBbfxtQmVQ4p767j4+tV ozFYWdiv2gM/xcAt2NHnq2ZYt+AhawestyYfL3Z2htdlcVmJuigSBEFsFFNNpWWflSZG H/y3CGY4R7HuaIGUo9ERXsIxX4ZwzLaOtAmw8bckUsbvE1MSiIBtDtCiQxSTzBuedcMg yWiA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:references:date:in-reply-to :message-id:user-agent:mime-version; bh=fCdFDDC/xpCRoDsCgcClRvhiStXO6uM3h7Y5XXg0FZQ=; b=FGuvR0fvTFwF0WLYF9GQigrQmUvNlgkfnPOtX/F4aKPcKVGFJuKvrq5QECXN/PJJQU 2EH3aUqdRY/z38U1G4G7yDEYlGSZxu968NwNB8U8uFLEPni5LlG9qzDW1b38y91ANE8z SHHNVg7MYTy2dgQ+jTiao34JaJrcP/Kc0CLsLbDrxdZRx03eyQoLr8MDFgDFmaCJXgTp CevY68xy8YS8by+DBwMnG5dd9sEUosPnA1za+FWoXBGOGNXHSx9M38ri9fxEP6wLaxjT RMSEryI8AtKaA3Oc5FejmGHY9ZXKw4WyiWqubr8PSV+nyST7G832/Cvsa3oOIIIyt/mB ns4w== X-Gm-Message-State: AOAM5301iBbB7Z5FtvOb/x39seUyySfq5A+YzBK3yFBBIBkDII9iPDNo MzJop2vC8lyp3E8uIehimFQarIr8Zp0= X-Google-Smtp-Source: ABdhPJwBuuxZdaqh0tdVbWRWOT6DyAb3HWo5SnYQYTrpMQpZCGvZkR3OlBh0CszOwT6eoqk/Mg4RdA== X-Received: by 2002:a5d:4a06:: with SMTP id m6mr6376151wrq.189.1609972445699; Wed, 06 Jan 2021 14:34:05 -0800 (PST) Received: from unfall (36.193.158.146.dynamic.jazztel.es. [146.158.193.36]) by smtp.gmail.com with ESMTPSA id u26sm4616264wmm.24.2021.01.06.14.34.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 06 Jan 2021 14:34:04 -0800 (PST) From: Miguel =?UTF-8?Q?=C3=81ngel?= Arruga Vivas References: <877dor34xe.fsf@gmail.com> Date: Wed, 06 Jan 2021 23:34:01 +0100 In-Reply-To: (Julien Lepiller's message of "Tue, 05 Jan 2021 10:17:53 -0500") Message-ID: <87a6tl1yqu.fsf@gmail.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.1 (gnu/linux) MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-guix@gnu.org List-Id: Bug reports for GNU Guix List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: 45675@debbugs.gnu.org Errors-To: bug-guix-bounces+larch=yhetil.org@gnu.org Sender: "bug-Guix" X-Migadu-Flow: FLOW_IN X-Migadu-Spam-Score: -1.24 Authentication-Results: aspmx1.migadu.com; dkim=fail (headers rsa verify failed) header.d=gmail.com header.s=20161025 header.b=k5ipH7yA; dmarc=fail reason="SPF not aligned (relaxed)" header.from=gmail.com (policy=none); spf=pass (aspmx1.migadu.com: domain of bug-guix-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=bug-guix-bounces@gnu.org X-Migadu-Queue-Id: 3DBC2940149 X-Spam-Score: -1.24 X-Migadu-Scanner: scn0.migadu.com X-TUID: WHgSPstRNd3P --=-=-= Content-Type: text/plain Hi, Julien Lepiller writes: > For java packages, we have a strip-jar-timestamps phase in the ant-build-system. Thanks for the pointer. Do you think could be worth to extract that into (guix build utils) as the attached patch (WIP) does? It rebuilds the world and replaces all of "old usages", so I'm still waiting to reach ant-bootstrap... Happy hacking! Miguel --=-=-= Content-Type: text/x-patch Content-Disposition: attachment; filename=0001-wip-build-utils-Extract-reset-zip-timestamp-and-use-.patch Content-Description: wip.patch >From dd2e78badad805cff8be940411994533aed8b059 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Miguel=20=C3=81ngel=20Arruga=20Vivas?= Date: Wed, 6 Jan 2021 23:29:36 +0100 Subject: [PATCH] wip-build-utils: Extract reset-zip-timestamp and use it everywhere. --- gnu/packages/java.scm | 73 ++++++++------------------------- guix/build/ant-build-system.scm | 32 ++++----------- guix/build/utils.scm | 48 ++++++++++++++++++++++ 3 files changed, 73 insertions(+), 80 deletions(-) diff --git a/gnu/packages/java.scm b/gnu/packages/java.scm index 758f8f1859..82d18bf62a 100644 --- a/gnu/packages/java.scm +++ b/gnu/packages/java.scm @@ -411,28 +411,11 @@ JNI.") (add-after 'build 'strip-jar-timestamps ;based on ant-build-system (lambda* (#:key outputs #:allow-other-keys) (define (repack-archive jar) - (let* ((dir (mkdtemp! "jar-contents.XXXXXX")) - (manifest (string-append dir "/META-INF/MANIFESTS.MF"))) - (with-directory-excursion dir - (invoke "unzip" jar)) - (delete-file jar) - ;; XXX: copied from (gnu build install) - (for-each (lambda (file) - (let ((s (lstat file))) - (unless (eq? (stat:type s) 'symlink) - (utime file 0 0 0 0)))) - (find-files dir #:directories? #t)) - ;; It is important that the manifest appears first. - (with-directory-excursion dir - (let* ((files (find-files "." ".*" #:directories? #t)) - ;; To ensure that the reference scanner can - ;; detect all store references in the jars - ;; we disable compression with the "-0" option. - (command (if (file-exists? manifest) - `("zip" "-0" "-X" ,jar ,manifest - ,@files) - `("zip" "-0" "-X" ,jar ,@files)))) - (apply invoke command))))) + (let ((mktempdir (lambda () + (mkdtemp! "jar-contents.XXXXXX")))) + (reset-zip-timestamp jar mktempdir + #:first-file "/META-INF/MANIFEST.MF" + #:compression-level "-0"))) (for-each repack-archive (find-files (string-append (assoc-ref %outputs "out") "/lib") @@ -1962,21 +1945,10 @@ new Date();")) (add-after 'install 'strip-zip-timestamps (lambda* (#:key outputs #:allow-other-keys) (use-modules (guix build syscalls)) - (for-each (lambda (zip) - (let ((dir (mkdtemp! "zip-contents.XXXXXX"))) - (with-directory-excursion dir - (invoke "unzip" zip)) - (delete-file zip) - (for-each (lambda (file) - (let ((s (lstat file))) - (unless (eq? (stat:type s) 'symlink) - (format #t "reset ~a~%" file) - (utime file 0 0 0 0)))) - (find-files dir #:directories? #t)) - (with-directory-excursion dir - (let ((files (find-files "." ".*" #:directories? #t))) - (apply invoke "zip" "-0" "-X" zip files))))) - (find-files (assoc-ref outputs "doc") ".*.zip$")) + (let ((mktempdir (lambda () (mkdtemp! "zip-contents.XXXXXX")))) + (for-each (lambda (zip) + (reset-zip-timestamp zip mktempdir)) + (find-files (assoc-ref outputs "doc") ".*.zip$"))) #t))))) (inputs `(("alsa-lib" ,alsa-lib) @@ -2197,25 +2169,14 @@ new Date();")) (use-modules (guix build syscalls) (ice-9 binary-ports) (rnrs bytevectors)) - (letrec ((repack-archive - (lambda (archive) - (let ((dir (mkdtemp! "zip-contents.XXXXXX"))) - (with-directory-excursion dir - (invoke "unzip" archive)) - (delete-file archive) - (for-each (compose repack-archive canonicalize-path) - (find-files dir "(ct.sym|.*.jar)$")) - (let ((reset-file-timestamp - (lambda (file) - (let ((s (lstat file))) - (unless (eq? (stat:type s) 'symlink) - (format #t "reset ~a~%" file) - (utime file 0 0 0 0)))))) - (for-each reset-file-timestamp - (find-files dir #:directories? #t))) - (with-directory-excursion dir - (let ((files (find-files "." ".*" #:directories? #t))) - (apply invoke "zip" "-0" "-X" archive files))))))) + (let* ((mktempdir (lambda () + (mkdtemp! "zip-contents.XXXXXX"))) + (repack-archive + (lambda (archive) + (reset-zip-timestamp archive mktempdir + #:compression-level "-0" + #:recursion-regexp + "(ct.sym|.*.jar)$")))) (for-each repack-archive (find-files (assoc-ref outputs "doc") ".*.zip$")) (for-each repack-archive diff --git a/guix/build/ant-build-system.scm b/guix/build/ant-build-system.scm index fae1b47ec5..d6c8b71abc 100644 --- a/guix/build/ant-build-system.scm +++ b/guix/build/ant-build-system.scm @@ -201,35 +201,19 @@ dependencies of this jar file." repack them. This is necessary to ensure that archives are reproducible." (define (repack-archive jar) (format #t "repacking ~a\n" jar) - (let* ((dir (mkdtemp! "jar-contents.XXXXXX")) - (manifest (string-append dir "/META-INF/MANIFEST.MF"))) - (with-directory-excursion dir - (invoke "jar" "xf" jar)) - (delete-file jar) - ;; XXX: copied from (gnu build install) - (for-each (lambda (file) - (let ((s (lstat file))) - (unless (eq? (stat:type s) 'symlink) - (utime file 0 0 0 0)))) - (find-files dir #:directories? #t)) - + (let ((manifest "/META-INF/MANIFEST.MF") + (mktmpdir (lambda () (mkdtemp! "jar-contents.XXXXXX")))) ;; The jar tool will always set the timestamp on the manifest file ;; and the containing directory to the current time, even when we ;; reuse an existing manifest file. To avoid this we use "zip" ;; instead of "jar". It is important that the manifest appears ;; first. - (with-directory-excursion dir - (let* ((files (find-files "." ".*" #:directories? #t)) - ;; To ensure that the reference scanner can detect all - ;; store references in the jars we disable compression - ;; with the "-0" option. - (command (if (file-exists? manifest) - `("zip" "-0" "-X" ,jar ,manifest ,@files) - `("zip" "-0" "-X" ,jar ,@files)))) - (apply invoke command))) - (utime jar 0 0) - #t)) - + (reset-zip-timestamp jar mktmpdir + #:first-file manifest + ;; To ensure that the reference scanner can detect + ;; all store references in the jars we disable + ;; compression with the "-0" option. + #:compression-level "-0"))) (for-each (match-lambda ((output . directory) (for-each repack-archive diff --git a/guix/build/utils.scm b/guix/build/utils.scm index 419c10195b..3f82d87732 100644 --- a/guix/build/utils.scm +++ b/guix/build/utils.scm @@ -56,7 +56,9 @@ elf-file? ar-file? gzip-file? + zip-file? reset-gzip-timestamp + reset-zip-timestamp with-directory-excursion mkdir-p install-file @@ -282,6 +284,52 @@ preserve FILE's modification time." (lambda () (chdir init))))) +(define %zip-magic-bytes + ;; Magic bytes of zip file. Beware, it's a small header so there could be + ;; false positives. + #vu8(#x50 #x4b)) + +(define zip-file? + (file-header-match %zip-magic-bytes)) + +(define* (reset-zip-timestamp zip-file tmp-dir-generator + #:key (first-file #f) + (compression-level "-6") + (recursion-regexp #f)) + "Reset the timestamps inside ZIP-FILE, regenerating it with the +COMPRESSION-LEVEL provided, and optionally placing FIRST-FILE at the +beginning of the archive when it exists. + +TMP-DIR-GENERATOR must return a different directory each time it is called +when RECURSION-REGEXP is provided." + (let* ((dir (tmp-dir-generator)) + (first-file (string-append dir first-file))) + (with-directory-excursion dir + (invoke "unzip" zip-file)) + (delete-file zip-file) + (when recursion-regexp + (for-each (lambda (file) + (reset-zip-timestamp (canonicalize-path file) + tmp-dir-generator + #:first-file first-file + #:compression-level compression-level + #:recursion-regexp recursion-regexp)) + (find-files dir recursion-regexp))) + (for-each (lambda (file) + (let ((s (lstat file))) + (unless (eq? (stat:type s) 'symlink) + (utime file 0 0 0 0)))) + (find-files dir #:directories? #t)) + + (with-directory-excursion dir + (let* ((files (find-files "." ".*" #:directories? #t)) + (call-zip `("zip" ,compression-level "-X" ,zip-file)) + (command (if (file-exists? first-file) + `(,@call-zip ,first-file ,@files) + `(,@call-zip ,@files)))) + (apply invoke command))) + (utime zip-file 0 0))) + (define (mkdir-p dir) "Create directory DIR and all its ancestors." (define absolute? -- 2.30.0 --=-=-=--