From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp0 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id uIDEHzAN1GAoRQAAgWs5BA (envelope-from ) for ; Thu, 24 Jun 2021 06:42:24 +0200 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp0 with LMTPS id wAnpGjAN1GC2SwAA1q6Kng (envelope-from ) for ; Thu, 24 Jun 2021 04:42:24 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id CB32E226C6 for ; Thu, 24 Jun 2021 06:42:23 +0200 (CEST) Received: from localhost ([::1]:49124 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lwHC2-0005ym-P4 for larch@yhetil.org; Thu, 24 Jun 2021 00:42:22 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:56966) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lwHBi-0005xU-JY for guix-patches@gnu.org; Thu, 24 Jun 2021 00:42:02 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:58867) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1lwHBi-0002wG-Cj for guix-patches@gnu.org; Thu, 24 Jun 2021 00:42:02 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1lwHBi-0002Qg-BO for guix-patches@gnu.org; Thu, 24 Jun 2021 00:42:02 -0400 X-Loop: help-debbugs@gnu.org Subject: [bug#49149] [PATCH v2 2/7] pack: Factorize base tar options. Resent-From: Maxim Cournoyer Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Thu, 24 Jun 2021 04:42:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 49149 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 49149@debbugs.gnu.org Cc: Maxim Cournoyer Received: via spool by 49149-submit@debbugs.gnu.org id=B49149.16245096849240 (code B ref 49149); Thu, 24 Jun 2021 04:42:02 +0000 Received: (at 49149) by debbugs.gnu.org; 24 Jun 2021 04:41:24 +0000 Received: from localhost ([127.0.0.1]:42168 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lwHB5-0002Ow-Py for submit@debbugs.gnu.org; Thu, 24 Jun 2021 00:41:24 -0400 Received: from mail-qt1-f181.google.com ([209.85.160.181]:45823) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lwHAz-0002Nw-W0 for 49149@debbugs.gnu.org; Thu, 24 Jun 2021 00:41:18 -0400 Received: by mail-qt1-f181.google.com with SMTP id r7so3887191qta.12 for <49149@debbugs.gnu.org>; Wed, 23 Jun 2021 21:41:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=TbM2uZIlEXZ3guTcWXscZGuwMTdXKJAYrWq+ax3jwSI=; b=Pfxj4MrjBO8ocHNkIHIUaUndb58R76vQ27EU/Akse30jK2uQXKPU+JV5p9a+Z2LPwY /wq5Hw6FRm1qg3Ebu7SHsx33+dm8aT+7LZ2qHFRuAJmr7Rl7sCjwXANYSMmY8jv57RFa XHKfobDfc7KPsbcQvyz7/EycwG8FpT2dZdiXHAjkNCfb7ILuzzat9fNamYOrYrPYqdt1 j/o41NKy5MZ+bsN0gZTLbSY+NzXTcLz3uJpZqqYfkQXHhazA0M3zpFL/djb/LoDDS392 hJDfzML/0ah/pFUL7D5UruLvrqvQymTiCCw3VwbpSd73pDSi5JLAm3lOFEAQEUaOqRQy ++sA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=TbM2uZIlEXZ3guTcWXscZGuwMTdXKJAYrWq+ax3jwSI=; b=Yt/yBut/BDP1MyD7ak6IH6NE9hNKiiccXaoVd19s+3GHKXKBF219Q9YmeCHmL+q2X1 lLffuQbGqtjv/6HFmQ9b+YMYjaVxj7wlNERs3fsk401AuaCMJKEGxMM5Zqj+d/wyuYhp 1jXsydDuKqwOuwQF0WsDA5ukOr/nQ4rA3e47luyTm8xlYG8uy/d+7sxuxt1rH/xmYNHd SnPYgyhS6GAO2+aY09HGhLm3xL3xnfxra5DZ3vM+sZugX05vcfLrfDmeaUPHhPROEL8g 8/Fduf0/bcQVpHa8I4ecI8/koAHUNmd+ZcL2JDHymY/3OQysdBkNtSh+2nAOn03D7+cP 0chQ== X-Gm-Message-State: AOAM530UIXzJFmKECcNl3/EqAmA5KPEKays0g7J+bz9qXdKMxrNreG8N qhyXVkcqnqTd90eKXfFsLU4P9+0G3PGtgA== X-Google-Smtp-Source: ABdhPJywDy3CJG8zQmNJZ9COqjZdXA2vqKN+R10u3ZAK+/2sa0l1YcCVp7ye/8P1yCzGWG5aNUecxQ== X-Received: by 2002:ac8:4646:: with SMTP id f6mr3180511qto.114.1624509672244; Wed, 23 Jun 2021 21:41:12 -0700 (PDT) Received: from localhost.localdomain ([207.35.95.2]) by smtp.gmail.com with ESMTPSA id q199sm1603880qka.112.2021.06.23.21.41.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 23 Jun 2021 21:41:11 -0700 (PDT) From: Maxim Cournoyer Date: Thu, 24 Jun 2021 00:40:44 -0400 Message-Id: <20210624044049.17906-2-maxim.cournoyer@gmail.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20210624044049.17906-1-maxim.cournoyer@gmail.com> References: <20210624044049.17906-1-maxim.cournoyer@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+larch=yhetil.org@gnu.org Sender: "Guix-patches" X-Migadu-Flow: FLOW_IN ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1624509744; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:resent-cc: resent-from:resent-sender:resent-message-id:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post:dkim-signature; bh=TbM2uZIlEXZ3guTcWXscZGuwMTdXKJAYrWq+ax3jwSI=; b=LVTsMtsYdPnbxeLXu0dpuR9SoaN5CpjL/pTrjH16KXe4fDnNKlrv6VIieDEjYCQPwloYMX ZuNnkyMHBTJ5L9Y6ITTPZHMfz9WMbmfHbEo1R05vsveNg5kibjsIURpLyFth08Q6G7m+56 sDYJ779z4jZVMBh+NAKXq0QmqbrYsDwOFsYOpxHywQsK4wOLqYuSLMvQOCKDxXQieoY7zD ArnVvWpJ6VmrRhH4mSY2UNL+bdXSkD5IrVA5Z7wr/9wcOAcn2Dw3zlGsJZZFMzzgWtCqpJ kr9zAcWvh9Z6AEW3Dzw6OMYta6T6rrLlRas86Mine4j6eayEhrdVa9kjZtxS8w== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1624509744; a=rsa-sha256; cv=none; b=Rn9oWAp0lHJfE04FhgzVgX0/1YeL26VV6VAqm+Uw1tBXWSSbjzopZ/nzu1UvfdLTiK2YnG awSyqI/0d3LEFRdWfPaHIGQO0WSVMY64vpzp+mdJDyOtcrDaL1Q/8UvaR+WpPC4yOmXQRw b8O+TKH42r/XDl7ufO7iwWxOPhWic4lEsqay8bW0QmXJLwinSzIz9eut3MZrIXjBMr1npu Mm9H8VA7xiApZ3Ty071EHXP/chaix4LCjoDW8+p+Gdr6rEb6dVSK+XFu23jc5O9yAHzwid GQA8Yh5vXFkvUwgzIi1Obbzr3OaipFHpKqTscz1E6/GEyf/wUyOQDq28GMjsMw== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=gmail.com header.s=20161025 header.b=Pfxj4Mrj; dmarc=fail reason="SPF not aligned (relaxed)" header.from=gmail.com (policy=none); spf=pass (aspmx1.migadu.com: domain of guix-patches-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-patches-bounces@gnu.org X-Migadu-Spam-Score: -1.33 Authentication-Results: aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=gmail.com header.s=20161025 header.b=Pfxj4Mrj; dmarc=fail reason="SPF not aligned (relaxed)" header.from=gmail.com (policy=none); spf=pass (aspmx1.migadu.com: domain of guix-patches-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-patches-bounces@gnu.org X-Migadu-Queue-Id: CB32E226C6 X-Spam-Score: -1.33 X-Migadu-Scanner: scn0.migadu.com X-TUID: SDrVLmKZFCm9 * guix/docker.scm (%tar-determinism-options): Move to a new module and rename to `tar-base-options'. Adjust references accordingly. * guix/build/pack.scm: New file. * Makefile.am (MODULES): Register it. * guix/scripts/pack.scm (self-contained-tarball/builder): Use it. --- Makefile.am | 1 + guix/build/pack.scm | 52 +++++++++++++++++++++++++++ guix/docker.scm | 20 ++--------- guix/scripts/pack.scm | 81 +++++++++++++++++-------------------------- 4 files changed, 87 insertions(+), 67 deletions(-) create mode 100644 guix/build/pack.scm diff --git a/Makefile.am b/Makefile.am index 7bb5de007e..15ac03ebd9 100644 --- a/Makefile.am +++ b/Makefile.am @@ -220,6 +220,7 @@ MODULES = \ guix/build/linux-module-build-system.scm \ guix/build/store-copy.scm \ guix/build/json.scm \ + guix/build/pack.scm \ guix/build/utils.scm \ guix/build/union.scm \ guix/build/profiles.scm \ diff --git a/guix/build/pack.scm b/guix/build/pack.scm new file mode 100644 index 0000000000..05c7a3c594 --- /dev/null +++ b/guix/build/pack.scm @@ -0,0 +1,52 @@ +;;; GNU Guix --- Functional package management for GNU +;;; Copyright © 2021 Maxim Cournoyer +;;; +;;; This file is part of GNU Guix. +;;; +;;; GNU Guix is free software; you can redistribute it and/or modify it +;;; under the terms of the GNU General Public License as published by +;;; the Free Software Foundation; either version 3 of the License, or (at +;;; your option) any later version. +;;; +;;; GNU Guix is distributed in the hope that it will be useful, but +;;; WITHOUT ANY WARRANTY; without even the implied warranty of +;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +;;; GNU General Public License for more details. +;;; +;;; You should have received a copy of the GNU General Public License +;;; along with GNU Guix. If not, see . + +(define-module (guix build pack) + #:use-module (guix build utils) + #:export (tar-base-options)) + +(define* (tar-base-options #:key tar compressor) + "Return the base GNU tar options required to produce deterministic archives +deterministically. When TAR, a GNU tar command file name, is provided, the +`--sort' option is used only if supported. When COMPRESSOR, a command such as +'(\"gzip\" \"-9n\"), is provided, the compressor is explicitly specified via +the `-I' option." + (define (tar-supports-sort? tar) + (zero? (system* tar "cf" "/dev/null" "--files-from=/dev/null" + "--sort=name"))) + + `(,@(if compressor + (list "-I" (string-join compressor)) + '()) + ;; The --sort option was added to GNU tar in version 1.28, released + ;; 2014-07-28. For testing, we use the bootstrap tar, which is older + ;; and doesn't support it. + ,@(if (and=> tar tar-supports-sort?) + '("--sort=name") + '()) + ;; Use GNU format so there's no file name length limitation. + "--format=gnu" + "--mtime=@1" + "--owner=root:0" + "--group=root:0" + ;; The 'nlink' of the store item files leads tar to store hard links + ;; instead of actual copies. However, the 'nlink' count depends on + ;; deduplication in the store; it's an "implicit input" to the build + ;; process. Use '--hard-dereference' to eliminate it. + "--hard-dereference" + "--check-links")) diff --git a/guix/docker.scm b/guix/docker.scm index 889aaeacb5..bd952e45ec 100644 --- a/guix/docker.scm +++ b/guix/docker.scm @@ -21,6 +21,7 @@ (define-module (guix docker) #:use-module (gcrypt hash) #:use-module (guix base16) + #:use-module (guix build pack) #:use-module ((guix build utils) #:select (mkdir-p delete-file-recursively @@ -110,18 +111,6 @@ Return a version of TAG that follows these rules." (rootfs . ((type . "layers") (diff_ids . #(,(layer-diff-id layer))))))) -(define %tar-determinism-options - ;; GNU tar options to produce archives deterministically. - '("--sort=name" "--mtime=@1" - "--owner=root:0" "--group=root:0" - - ;; When 'build-docker-image' is passed store items, the 'nlink' of the - ;; files therein leads tar to store hard links instead of actual copies. - ;; However, the 'nlink' count depends on deduplication in the store; it's - ;; an "implicit input" to the build process. '--hard-dereference' - ;; eliminates it. - "--hard-dereference")) - (define directive-file ;; Return the file or directory created by a 'evaluate-populate-directive' ;; directive. @@ -238,7 +227,7 @@ SRFI-19 time-utc object, as the creation time in metadata." (apply invoke "tar" "-cf" "../layer.tar" `(,@transformation-options - ,@%tar-determinism-options + ,@(tar-base-options) ,@paths ,@(scandir "." (lambda (file) @@ -273,9 +262,6 @@ SRFI-19 time-utc object, as the creation time in metadata." (scm->json (repositories prefix id repository))))) (apply invoke "tar" "-cf" image "-C" directory - `(,@%tar-determinism-options - ,@(if compressor - (list "-I" (string-join compressor)) - '()) + `(,@(tar-base-options #:compressor compressor) ".")) (delete-file-recursively directory))) diff --git a/guix/scripts/pack.scm b/guix/scripts/pack.scm index ac477850e6..d11f498925 100644 --- a/guix/scripts/pack.scm +++ b/guix/scripts/pack.scm @@ -205,12 +205,14 @@ dependencies are registered." (not (equal? '(guix store deduplication) module)))) (with-imported-modules (source-module-closure - `((guix build utils) + `((guix build pack) + (guix build utils) (guix build union) (gnu build install)) #:select? import-module?) #~(begin - (use-modules (guix build utils) + (use-modules (guix build pack) + (guix build utils) ((guix build union) #:select (relative-file-name)) (gnu build install) (srfi srfi-1) @@ -240,19 +242,10 @@ dependencies are registered." ;; Fully-qualified symlinks. (append-map symlink->directives '#$symlinks)) - ;; The --sort option was added to GNU tar in version 1.28, released - ;; 2014-07-28. For testing, we use the bootstrap tar, which is - ;; older and doesn't support it. - (define tar-supports-sort? - (zero? (system* (string-append #+archiver "/bin/tar") - "cf" "/dev/null" "--files-from=/dev/null" - "--sort=name"))) - ;; Make sure non-ASCII file names are properly handled. #+set-utf8-locale - ;; Add 'tar' to the search path. - (setenv "PATH" #+(file-append archiver "/bin")) + (define tar #+(file-append archiver "/bin/tar")) ;; Note: there is not much to gain here with deduplication and there ;; is the overhead of the '.links' directory, so turn it off. @@ -269,45 +262,33 @@ dependencies are registered." (for-each (cut evaluate-populate-directive <> %root) directives) - ;; Create the tarball. Use GNU format so there's no file name - ;; length limitation. + ;; Create the tarball. (with-directory-excursion %root - (apply invoke "tar" - #+@(if (compressor-command compressor) - #~("-I" - (string-join - '#+(compressor-command compressor))) - #~()) - "--format=gnu" - ;; Avoid non-determinism in the archive. - ;; Use mtime = 1, not zero, because that is what the daemon - ;; does for files in the store (see the 'mtimeStore' constant - ;; in local-store.cc.) - (if tar-supports-sort? "--sort=name" "--mtime=@1") - "--owner=root:0" - "--group=root:0" - "--check-links" - "-cvf" #$output - ;; Avoid adding / and /var to the tarball, so - ;; that the ownership and permissions of those - ;; directories will not be overwritten when - ;; extracting the archive. Do not include /root - ;; because the root account might have a - ;; different home directory. - #$@(if localstatedir? - '("./var/guix") - '()) - - (string-append "." (%store-directory)) - - (delete-duplicates - (filter-map (match-lambda - (('directory directory) - (string-append "." directory)) - ((source '-> _) - (string-append "." source)) - (_ #f)) - directives))))))) + (apply invoke tar + `(,@(tar-base-options + #:tar tar + #:compressor '#+(and=> compressor compressor-command)) + "-cvf" ,#$output + ;; Avoid adding / and /var to the tarball, so + ;; that the ownership and permissions of those + ;; directories will not be overwritten when + ;; extracting the archive. Do not include /root + ;; because the root account might have a + ;; different home directory. + ,#$@(if localstatedir? + '("./var/guix") + '()) + + ,(string-append "." (%store-directory)) + + ,@(delete-duplicates + (filter-map (match-lambda + (('directory directory) + (string-append "." directory)) + ((source '-> _) + (string-append "." source)) + (_ #f)) + directives)))))))) (define* (self-contained-tarball name profile #:key target -- 2.32.0