From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp12.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms9.migadu.com with LMTPS id yBb8OEILBmT5KgAASxT56A (envelope-from ) for ; Mon, 06 Mar 2023 16:48:19 +0100 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp12.migadu.com with LMTPS id GKoEOUILBmQsSgEAauVa8A (envelope-from ) for ; Mon, 06 Mar 2023 16:48:18 +0100 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id A5BA211E8C for ; Mon, 6 Mar 2023 16:48:18 +0100 (CET) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pZD4J-0002ul-9P; Mon, 06 Mar 2023 10:48:07 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pZD4E-0002qp-Ex for guix-patches@gnu.org; Mon, 06 Mar 2023 10:48:03 -0500 Received: from debbugs.gnu.org ([209.51.188.43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1pZD4E-0004mC-5Y for guix-patches@gnu.org; Mon, 06 Mar 2023 10:48:02 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1pZD4E-00042Y-1K for guix-patches@gnu.org; Mon, 06 Mar 2023 10:48:02 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#61949] [PATCH] pack: Move common build code to (guix build pack). Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Mon, 06 Mar 2023 15:48:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 61949 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Maxim Cournoyer Cc: 61949@debbugs.gnu.org Received: via spool by 61949-submit@debbugs.gnu.org id=B61949.167811765915413 (code B ref 61949); Mon, 06 Mar 2023 15:48:02 +0000 Received: (at 61949) by debbugs.gnu.org; 6 Mar 2023 15:47:39 +0000 Received: from localhost ([127.0.0.1]:43382 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pZD3q-00040U-SX for submit@debbugs.gnu.org; Mon, 06 Mar 2023 10:47:39 -0500 Received: from eggs.gnu.org ([209.51.188.92]:49904) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pZD3p-00040I-Hp for 61949@debbugs.gnu.org; Mon, 06 Mar 2023 10:47:38 -0500 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pZD3k-0004k2-Aa; Mon, 06 Mar 2023 10:47:32 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:In-Reply-To:Date:References:Subject:To: From; bh=4kI8vykYLKbGOgffmC6bFZfVADVdHiQiitIygaVLndg=; b=l/APIsMOa54bOizf78Y3 8Hbx7zcQHx2jsXwjL9ask9PQCqgcdiYayyrRok+e/Sq5Qp/CSFqn7dIfJUhlnAdfrvPLdJEVBmF8t 9aM00uu50SqZYMlM79NQjej4KlX8O/uhdHGgCR2Tc43nXordsjeFp0qx0vzYkoSP0mSj120y3XFN9 DkoXg3NImJTMRbJjaqWA3q3YZ/sd+fVUOdNdIVM7l1r19qK+eEtGi8ukRK4jf7ddPy8Y/FB9vJw+Q 3XSxH2RVerILRp+eaXM9nwOh8wfDMXMGe3geAM+4Jw5F2ZsFEsM9koSXPljubQhmbRALAc6nE9514 sT3CPgh/ThnEKQ==; Received: from [2001:660:6102:320:e120:2c8f:8909:cdfe] (helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pZD3j-00020z-Ti; Mon, 06 Mar 2023 10:47:32 -0500 From: Ludovic =?UTF-8?Q?Court=C3=A8s?= References: <20230304031523.24102-1-maxim.cournoyer@gmail.com> X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: Sextidi 16 =?UTF-8?Q?Vent=C3=B4se?= an 231 de la =?UTF-8?Q?R=C3=A9volution,?= jour de =?UTF-8?Q?l'=C3=89pinard?= X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-pc-linux-gnu Date: Mon, 06 Mar 2023 16:47:28 +0100 In-Reply-To: <20230304031523.24102-1-maxim.cournoyer@gmail.com> (Maxim Cournoyer's message of "Fri, 3 Mar 2023 22:15:23 -0500") Message-ID: <87sfehanlb.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+larch=yhetil.org@gnu.org Sender: guix-patches-bounces+larch=yhetil.org@gnu.org X-TUID: KZdEGWSzoL05 Hi, Maxim Cournoyer skribis: > The rationale is to reduce the number of derivations built per pack to id= eally > one, to minimize storage requirements. The number of derivations had gon= e up > with 68380db4 ("pack: Extract populate-profile-root from > self-contained-tarball/builder.") as a side effect to improving code reus= e. > > * guix/scripts/pack.scm (guix): Add commentary comment. > (populate-profile-root, self-contained-tarball/builder): Extract to... > * guix/build/pack.scm (populate-profile-root!): ... this, and... > (build-self-contained-tarball): ... that, adjusting for use on the build = side. > (assert-utf8-locale): New procedure. > (self-contained-tarball, debian-archive, rpm-archive): Adjust accordingly. Thanks for working on it! [...] > +;;; Copyright =C2=A9 2021, 2023 Maxim Cournoyer This may be inaccurate given that some of the code here predates this file. > ;;; along with GNU Guix. If not, see . >=20=20 > +;;; Commentary: > + > +;;; This module contains build-side common procedures used by the host-s= ide > +;;; (guix scripts pack) module, mostly to allow for code reuse. Due to = making > +;;; use of the (guix build store-copy) module, it transitively requires = the > +;;; sqlite and gcrypt extensions to be available. > + > +;;; Code: > + > (define-module (guix build pack) Commentary/code should come after =E2=80=98define-module=E2=80=99. > +(define (assert-utf8-locale) > + "Verify the current process is using the en_US.utf8 locale." > + (unless (string=3D? "unset for tests" (getenv "GUIX_LOCPATH")) > + (unless (false-if-exception (setlocale LC_ALL "en_US.utf8")) > + (error "environment not configured for en_US.utf8 locale")))) > + > +(define* (populate-profile-root! profile > + #:key (profile-name "guix-profile") > + localstatedir? > + store-database > + deduplicate? > + (symlinks '())) Please leave out the bang from the name. The convention in Scheme is to suffix a name with bang when it modifies the object(s) it=E2=80=99s given; that=E2=80=99s not the case here (see also =E2=80=98mkdir=E2=80=99, =E2=80= =98open-output-file=E2=80=99, etc.). > + "Populate the root profile directory with SYMLINKS and a Guix database= , when > +LOCALSTATEDIR? is set, and a pre-computed STORE-DATABASE is provided. T= he > +directory is created as \"root\" in the current working directory. When > +DEDUPLICATE? is true, deduplicate the store items, which relies on hard > +links. It needs to run in an environment where " > + (when localstatedir? > + (unless store-database > + (error "missing STORE-DATABASE argument"))) > + > + (define symlink->directives Please move the =E2=80=98when=E2=80=99 expression after all defines so that= this code can be interpreted by Guile 2.0, which in turn will allow us to run tests on =E2=80=98guile-bootstrap=E2=80=99. > +(define* (build-self-contained-tarball profile > + tarball-file-name > + #:key (profile-name "guix-profile= ") > + target > + localstatedir? > + store-database > + deduplicate? > + symlinks > + compressor-command > + archiver) > + "Create a self-contained tarball TARBALL-FILE-NAME from PROFILE, optio= nally > +compressing it with COMPRESSOR-COMMAND, the complete command-line string= to > +use for the compressor." > + (assert-utf8-locale) > + > + (populate-profile-root! profile > + #:profile-name profile-name > + #:localstatedir? localstatedir? > + #:store-database store-database > + #:deduplicate? deduplicate? > + #:symlinks symlinks) > + > + (define tar (string-append archiver "/bin/tar")) Likewise, move defines before statements. Also, I would just assume =E2=80=9Ctar=E2=80=9D is in $PATH. That=E2=80=99= s the assumption generally made for things that need to shell out to various commands, such as (gnu build file-systems), (guix docker), etc. > ;;; along with GNU Guix. If not, see . >=20=20 > +;;; Commentary: > + > +;;; This module implements the 'guix pack' command and the various suppo= rted > +;;; formats. Where feasible, the builders of the packs should be implem= ented > +;;; as single derivations to minimize storage requirements. > + > +;;; Code: Likewise needs to be moved down. :-) > -(test-assertm "self-contained-tarball" %store > - (mlet* %store-monad > - ((profile -> (profile > - (content (packages->manifest (list %bootstrap-guile)= )) > - (hooks '()) > - (locales? #f))) > - (tarball (self-contained-tarball "pack" profile > - #:symlinks '(("/bin/Guile" > - -> "bin/guile")) > - #:compressor %gzip-compressor > - #:archiver %tar-bootstrap)) [...] > ;; The following test needs guile-sqlite3, libgcrypt, etc. as a conseque= nce of > ;; commit c45477d2a1a651485feede20fe0f3d15aec48b39 and related changes. = Thus, > ;; run it on the user's store, if it's available, on the grounds that th= ese > ;; dependencies may be already there, or we can get substitutes or build= them > ;; quite inexpensively; see . > - > (with-external-store store > + (unless store (test-skip 1)) > + (test-assertm "self-contained-tarball" store We should avoid moving this tests here. The goal is to keep as many tests as possible under the =E2=80=9Cnormal mode=E2=80=9D (outside =E2=80=98with-external-store=E2=80=99) because they are exercised more freq= uently. I went to great lengths to make it possible here, so we should strive to preserve that property. (Note that I haven=E2=80=99t tried running the code and tests yet.) Could you send a v2? Thanks, Ludo=E2=80=99.