From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp1 ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id ULiGJf0t0GAnRQAAgWs5BA (envelope-from ) for ; Mon, 21 Jun 2021 08:13:17 +0200 Received: from aspmx1.migadu.com ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp1 with LMTPS id iHblIP0t0GCXUgAAbx9fmQ (envelope-from ) for ; Mon, 21 Jun 2021 06:13:17 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id E574B26CC8 for ; Mon, 21 Jun 2021 08:13:16 +0200 (CEST) Received: from localhost ([::1]:38446 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lvDBL-0007ua-V2 for larch@yhetil.org; Mon, 21 Jun 2021 02:13:15 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:47924) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lvDB8-0007u9-Vi for guix-patches@gnu.org; Mon, 21 Jun 2021 02:13:02 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:51892) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1lvDB8-0001ra-Hu for guix-patches@gnu.org; Mon, 21 Jun 2021 02:13:02 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1lvDB8-0006cI-Be for guix-patches@gnu.org; Mon, 21 Jun 2021 02:13:02 -0400 X-Loop: help-debbugs@gnu.org Subject: [bug#49149] [PATCH 1/7] pack: Extract builder code from self-contained-tarball. References: <20210621061039.31557-1-maxim.cournoyer@gmail.com> In-Reply-To: <20210621061039.31557-1-maxim.cournoyer@gmail.com> Resent-From: Maxim Cournoyer Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Mon, 21 Jun 2021 06:13:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 49149 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 49149@debbugs.gnu.org Cc: Maxim Cournoyer Received: via spool by 49149-submit@debbugs.gnu.org id=B49149.162425594925297 (code B ref 49149); Mon, 21 Jun 2021 06:13:02 +0000 Received: (at 49149) by debbugs.gnu.org; 21 Jun 2021 06:12:29 +0000 Received: from localhost ([127.0.0.1]:35184 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lvDAa-0006Zx-K8 for submit@debbugs.gnu.org; Mon, 21 Jun 2021 02:12:29 -0400 Received: from mail-qk1-f177.google.com ([209.85.222.177]:41699) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lvDAZ-0006Zk-5c for 49149@debbugs.gnu.org; Mon, 21 Jun 2021 02:12:27 -0400 Received: by mail-qk1-f177.google.com with SMTP id 22so6580452qkv.8 for <49149@debbugs.gnu.org>; Sun, 20 Jun 2021 23:12:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=nK+XZhiFPUZF48SwwcEJOQPOaSHkTwT9yOejmrHORIk=; b=Mc+L4hmSt+uB/hrshFoqOiD3TqO+oSEvXW/8hVSjpNvBEcXwbxvQqpyMwY7qDvDq7c laXqBLmkRrZ6xAQdpFBBktb6NccbBokNP9bLWw3RUoYtiZUjEzZQDWA5IQxsb2I0u8TM NuOabm0KajiJLpTddnIEfHDpGQcNmInSsXYn/oSkSVF9RLICT73x3xJuaIGsjmXEpLgY 5PpD9V/NiPxn5zQpWb8a07Gi8CnZjq3OcDxNCJKfRbu1/CfeNq4p2bHDbPSM1Wft9vaF 2rzWtHJdwT2PB4zXl95HFsizCmSEXBWZOQGwNwZ47TF8nSOfDrtFjck01o76T3PLk/L2 YSYg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=nK+XZhiFPUZF48SwwcEJOQPOaSHkTwT9yOejmrHORIk=; b=lGqtK8vpYBiHs2USjAA3Xu8D3j0dMmmJI4XY2xEJavz6f1xTikYsK9NDLp4vOqn1Q7 VupMNMKPhaOlcUlJ6/Q1bThvwtSVj1PmetPqUrTVDViIIWzmlbrZKHf4c6E6j33yfZF1 52ZJribbCwOL1OrqgA214PzDC0D1rcIyuK4e0hPEiZb9SO8tYSYpj4VqKivp89+xl2Gq CoeOwE0VM0RjpblCOFhS/PmkahqibtF82yi1NRsYVnr+RZO8SJL1hF1PvhKiuNDIwu1k BhJLpHuHPRo3sjXTQ5/jyRZxzYlegOHGQBUQ8wpYqz6fhIrmImWKiBwlu+Gb3iLSUx/J UY9Q== X-Gm-Message-State: AOAM532eB/BdE0NwDHlfVqYQOf6cTym6WmnroNGFIvlmbSfmLUfDKe1d /AdX+Xj1sUpZX/V8fE0ic8EsVogFtmwzhA== X-Google-Smtp-Source: ABdhPJzBl8oxZ0tsoN9oIDp8kGyrqOXE52cFFinhjwwAkToF7DEMkeln88AB5Y5sKJyBU5cITlur9A== X-Received: by 2002:a05:620a:218e:: with SMTP id g14mr21312418qka.290.1624255941431; Sun, 20 Jun 2021 23:12:21 -0700 (PDT) Received: from localhost.localdomain (dsl-148-219.b2b2c.ca. [66.158.148.219]) by smtp.gmail.com with ESMTPSA id i11sm8478663qke.74.2021.06.20.23.12.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 20 Jun 2021 23:12:21 -0700 (PDT) From: Maxim Cournoyer Date: Mon, 21 Jun 2021 02:11:58 -0400 Message-Id: <20210621061205.31878-1-maxim.cournoyer@gmail.com> X-Mailer: git-send-email 2.32.0 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+larch=yhetil.org@gnu.org Sender: "Guix-patches" X-Migadu-Flow: FLOW_IN ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1624255997; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:resent-cc: resent-from:resent-sender:resent-message-id:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post:dkim-signature; bh=kdBQA0TSysst8Tloj0y+XBA/sAwpUaMZBZgIxmAveW8=; b=kqw8K188Xzhl84UkDWfcHXlEOUnE9piP6f3wSZ/+o2ue+GtCL+cCXYNsUORvnfvmDH2sIf i7J4ZfJxnKfUgCR1bMBw+QB/79NpSNApl/LZ2kfNofMzwvSJgYciDh8Z1FHXnULqhylvio iCVCEUA7pPWb6OD3T0HbJdo6qhOoDTpAVC8DiFnNwUufVcYFu6tzHTPId9qJy8W0Vp9XOM NCVoQk/e344X+YkjiJWiaOJpc9EiHaffpHIzDJO7Cb9tHN7zuqcFe/VkEOAoxy6T93dvni Vi+rdTsO7TIEem84LSwZtJGU3NZqSs5rgDc6vKJKRM698s/TRu3C+RflntOtVg== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1624255997; a=rsa-sha256; cv=none; b=ddrqeOaMTWiQkWYAp+uRNzIoRpkAQgvTUomjz48qLSgKTeZT3L42M3ZBMMiACtrmmvhbCT 0w3mjCO0WZtBmRii9fjewCuTP1BhGrsSwo3qhENWdSeNYvBxvwKW5N05RTVn3rMSGNZO6y sjwcE5js5cmHQ8z3DDdu7xRlPHa78blQDM4AwD44PJ2QS91yUzWgFbuJ7s2hUx3s33oFqV IY+XOdLbJD7eD3OU36jpgvH3TNgWqCueHcxodaaYEhJD/EO82XvpgUSfy0710EjFb0Fp93 1PhS9PqfABbsiuBkOLJZ0tXm9JM05xbSMCTKvnjU7S0H/gkhRiy3czVt1Y4SAg== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=fail ("body hash did not verify") header.d=gmail.com header.s=20161025 header.b=Mc+L4hmS; dmarc=fail reason="SPF not aligned (relaxed)" header.from=gmail.com (policy=none); spf=pass (aspmx1.migadu.com: domain of guix-patches-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-patches-bounces@gnu.org X-Migadu-Spam-Score: -1.32 Authentication-Results: aspmx1.migadu.com; dkim=fail ("body hash did not verify") header.d=gmail.com header.s=20161025 header.b=Mc+L4hmS; dmarc=fail reason="SPF not aligned (relaxed)" header.from=gmail.com (policy=none); spf=pass (aspmx1.migadu.com: domain of guix-patches-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-patches-bounces@gnu.org X-Migadu-Queue-Id: E574B26CC8 X-Spam-Score: -1.32 X-Migadu-Scanner: scn1.migadu.com X-TUID: WfE0pC9fSO03 This is made to allow reusing it for the debian-archive pack format, added in a subsequent commit. * guix/scripts/pack.scm (self-contained-tarball/builder): New procedure, containing the build code extracted from self-contained-tarball. (self-contained-tarball): Use the above procedure. --- guix/scripts/pack.scm | 270 ++++++++++++++++++++++-------------------- 1 file changed, 141 insertions(+), 129 deletions(-) diff --git a/guix/scripts/pack.scm b/guix/scripts/pack.scm index 8cb4e6d2cc..ac477850e6 100644 --- a/guix/scripts/pack.scm +++ b/guix/scripts/pack.scm @@ -172,22 +172,17 @@ dependencies are registered." (computed-file "store-database" build #:options `(#:references-graphs ,(zip labels items)))) -(define* (self-contained-tarball name profile - #:key target - (profile-name "guix-profile") - deduplicate? - entry-point - (compressor (first %compressors)) - localstatedir? - (symlinks '()) - (archiver tar)) - "Return a self-contained tarball containing a store initialized with the -closure of PROFILE, a derivation. The tarball contains /gnu/store; if -LOCALSTATEDIR? is true, it also contains /var/guix, including /var/guix/db -with a properly initialized store database. - -SYMLINKS must be a list of (SOURCE -> TARGET) tuples denoting symlinks to be -added to the pack." + +;;; +;;; Tarball format. +;;; +(define* (self-contained-tarball/builder profile + #:key (profile-name "guix-profile") + (compressor (first %compressors)) + localstatedir? + (symlinks '()) + (archiver tar)) + "Return the G-Expression of the builder used for self-contained-tarball." (define database (and localstatedir? (file-append (store-database (list profile)) @@ -209,125 +204,142 @@ added to the pack." (and (not-config? module) (not (equal? '(guix store deduplication) module)))) - (define build - (with-imported-modules (source-module-closure - `((guix build utils) - (guix build union) - (gnu build install)) - #:select? import-module?) - #~(begin - (use-modules (guix build utils) - ((guix build union) #:select (relative-file-name)) - (gnu build install) - (srfi srfi-1) - (srfi srfi-26) - (ice-9 match)) + (with-imported-modules (source-module-closure + `((guix build utils) + (guix build union) + (gnu build install)) + #:select? import-module?) + #~(begin + (use-modules (guix build utils) + ((guix build union) #:select (relative-file-name)) + (gnu build install) + (srfi srfi-1) + (srfi srfi-26) + (ice-9 match)) + + (define %root "root") + + (define symlink->directives + ;; Return "populate directives" to make the given symlink and its + ;; parent directories. + (match-lambda + ((source '-> target) + (let ((target (string-append #$profile "/" target)) + (parent (dirname source))) + ;; Never add a 'directory' directive for "/" so as to + ;; preserve its ownnership when extracting the archive (see + ;; below), and also because this would lead to adding the + ;; same entries twice in the tarball. + `(,@(if (string=? parent "/") + '() + `((directory ,parent))) + (,source + -> ,(relative-file-name parent target))))))) + + (define directives + ;; Fully-qualified symlinks. + (append-map symlink->directives '#$symlinks)) + + ;; The --sort option was added to GNU tar in version 1.28, released + ;; 2014-07-28. For testing, we use the bootstrap tar, which is + ;; older and doesn't support it. + (define tar-supports-sort? + (zero? (system* (string-append #+archiver "/bin/tar") + "cf" "/dev/null" "--files-from=/dev/null" + "--sort=name"))) + + ;; Make sure non-ASCII file names are properly handled. + #+set-utf8-locale + + ;; Add 'tar' to the search path. + (setenv "PATH" #+(file-append archiver "/bin")) + + ;; Note: there is not much to gain here with deduplication and there + ;; is the overhead of the '.links' directory, so turn it off. + ;; Furthermore GNU tar < 1.30 sometimes fails to extract tarballs + ;; with hard links: + ;; . + (populate-single-profile-directory %root + #:profile #$profile + #:profile-name #$profile-name + #:closure "profile" + #:database #+database) + + ;; Create SYMLINKS. + (for-each (cut evaluate-populate-directive <> %root) + directives) + + ;; Create the tarball. Use GNU format so there's no file name + ;; length limitation. + (with-directory-excursion %root + (apply invoke "tar" + #+@(if (compressor-command compressor) + #~("-I" + (string-join + '#+(compressor-command compressor))) + #~()) + "--format=gnu" + ;; Avoid non-determinism in the archive. + ;; Use mtime = 1, not zero, because that is what the daemon + ;; does for files in the store (see the 'mtimeStore' constant + ;; in local-store.cc.) + (if tar-supports-sort? "--sort=name" "--mtime=@1") + "--owner=root:0" + "--group=root:0" + "--check-links" + "-cvf" #$output + ;; Avoid adding / and /var to the tarball, so + ;; that the ownership and permissions of those + ;; directories will not be overwritten when + ;; extracting the archive. Do not include /root + ;; because the root account might have a + ;; different home directory. + #$@(if localstatedir? + '("./var/guix") + '()) + + (string-append "." (%store-directory)) + + (delete-duplicates + (filter-map (match-lambda + (('directory directory) + (string-append "." directory)) + ((source '-> _) + (string-append "." source)) + (_ #f)) + directives))))))) - (define %root "root") - - (define symlink->directives - ;; Return "populate directives" to make the given symlink and its - ;; parent directories. - (match-lambda - ((source '-> target) - (let ((target (string-append #$profile "/" target)) - (parent (dirname source))) - ;; Never add a 'directory' directive for "/" so as to - ;; preserve its ownnership when extracting the archive (see - ;; below), and also because this would lead to adding the - ;; same entries twice in the tarball. - `(,@(if (string=? parent "/") - '() - `((directory ,parent))) - (,source - -> ,(relative-file-name parent target))))))) - - (define directives - ;; Fully-qualified symlinks. - (append-map symlink->directives '#$symlinks)) - - ;; The --sort option was added to GNU tar in version 1.28, released - ;; 2014-07-28. For testing, we use the bootstrap tar, which is - ;; older and doesn't support it. - (define tar-supports-sort? - (zero? (system* (string-append #+archiver "/bin/tar") - "cf" "/dev/null" "--files-from=/dev/null" - "--sort=name"))) - - ;; Make sure non-ASCII file names are properly handled. - #+set-utf8-locale - - ;; Add 'tar' to the search path. - (setenv "PATH" #+(file-append archiver "/bin")) - - ;; Note: there is not much to gain here with deduplication and there - ;; is the overhead of the '.links' directory, so turn it off. - ;; Furthermore GNU tar < 1.30 sometimes fails to extract tarballs - ;; with hard links: - ;; . - (populate-single-profile-directory %root - #:profile #$profile - #:profile-name #$profile-name - #:closure "profile" - #:database #+database) - - ;; Create SYMLINKS. - (for-each (cut evaluate-populate-directive <> %root) - directives) - - ;; Create the tarball. Use GNU format so there's no file name - ;; length limitation. - (with-directory-excursion %root - (exit - (zero? (apply system* "tar" - #+@(if (compressor-command compressor) - #~("-I" - (string-join - '#+(compressor-command compressor))) - #~()) - "--format=gnu" - - ;; Avoid non-determinism in the archive. Use - ;; mtime = 1, not zero, because that is what the - ;; daemon does for files in the store (see the - ;; 'mtimeStore' constant in local-store.cc.) - (if tar-supports-sort? "--sort=name" "--mtime=@1") - "--mtime=@1" ;for files in /var/guix - "--owner=root:0" - "--group=root:0" - - "--check-links" - "-cvf" #$output - ;; Avoid adding / and /var to the tarball, so - ;; that the ownership and permissions of those - ;; directories will not be overwritten when - ;; extracting the archive. Do not include /root - ;; because the root account might have a - ;; different home directory. - #$@(if localstatedir? - '("./var/guix") - '()) - - (string-append "." (%store-directory)) - - (delete-duplicates - (filter-map (match-lambda - (('directory directory) - (string-append "." directory)) - ((source '-> _) - (string-append "." source)) - (_ #f)) - directives))))))))) +(define* (self-contained-tarball name profile + #:key target + (profile-name "guix-profile") + deduplicate? + entry-point + (compressor (first %compressors)) + localstatedir? + (symlinks '()) + (archiver tar)) + "Return a self-contained tarball containing a store initialized with the +closure of PROFILE, a derivation. The tarball contains /gnu/store; if +LOCALSTATEDIR? is true, it also contains /var/guix, including /var/guix/db +with a properly initialized store database. +SYMLINKS must be a list of (SOURCE -> TARGET) tuples denoting symlinks to be +added to the pack." (when entry-point (warning (G_ "entry point not supported in the '~a' format~%") 'tarball)) - (gexp->derivation (string-append name ".tar" - (compressor-extension compressor)) - build - #:target target - #:references-graphs `(("profile" ,profile)))) + (gexp->derivation + (string-append name ".tar" + (compressor-extension compressor)) + (self-contained-tarball/builder profile + #:profile-name profile-name + #:compressor compressor + #:localstatedir? localstatedir? + #:symlinks symlinks + #:archiver archiver) + #:target target + #:references-graphs `(("profile" ,profile)))) (define (singularity-environment-file profile) "Return a shell script that defines the environment variables corresponding -- 2.32.0