From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp2 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms11 with LMTPS id 0JihMbZsrV6kQAAA0tVLHw (envelope-from ) for ; Sat, 02 May 2020 12:51:02 +0000 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp2 with LMTPS id mP19BcBsrV7KFwAAB5/wlQ (envelope-from ) for ; Sat, 02 May 2020 12:51:12 +0000 Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:470:142::17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 3A5C29418BA for ; Sat, 2 May 2020 12:51:11 +0000 (UTC) Received: from localhost ([::1]:52716 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jUrbr-0001V3-EE for larch@yhetil.org; Sat, 02 May 2020 08:51:11 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:36234) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jUrbj-0001So-GZ for guix-patches@gnu.org; Sat, 02 May 2020 08:51:04 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.90_1) (envelope-from ) id 1jUrbi-0000Mz-FO for guix-patches@gnu.org; Sat, 02 May 2020 08:51:03 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:39942) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1jUrbi-0000MI-1h for guix-patches@gnu.org; Sat, 02 May 2020 08:51:02 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1jUrbh-0001d5-VW for guix-patches@gnu.org; Sat, 02 May 2020 08:51:02 -0400 X-Loop: help-debbugs@gnu.org Subject: [bug#40955] [PATCH 4/5] image: Add a new API. Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Sat, 02 May 2020 12:51:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 40955 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Mathieu Othacehe Cc: 40955@debbugs.gnu.org Received: via spool by 40955-submit@debbugs.gnu.org id=B40955.15884238396232 (code B ref 40955); Sat, 02 May 2020 12:51:01 +0000 Received: (at 40955) by debbugs.gnu.org; 2 May 2020 12:50:39 +0000 Received: from localhost ([127.0.0.1]:51488 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jUrbL-0001cS-AE for submit@debbugs.gnu.org; Sat, 02 May 2020 08:50:39 -0400 Received: from eggs.gnu.org ([209.51.188.92]:33054) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jUrbH-0001cB-Td for 40955@debbugs.gnu.org; Sat, 02 May 2020 08:50:38 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:56496) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jUrbC-0008Qw-I5; Sat, 02 May 2020 08:50:30 -0400 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=45096 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1jUrbC-0004Qw-30; Sat, 02 May 2020 08:50:30 -0400 From: Ludovic =?UTF-8?Q?Court=C3=A8s?= References: <20200429084756.25072-1-m.othacehe@gmail.com> <20200429084756.25072-4-m.othacehe@gmail.com> Date: Sat, 02 May 2020 14:50:28 +0200 In-Reply-To: <20200429084756.25072-4-m.othacehe@gmail.com> (Mathieu Othacehe's message of "Wed, 29 Apr 2020 10:47:55 +0200") Message-ID: <87wo5u8prf.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.3 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-Received-From: 209.51.188.43 X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+larch=yhetil.org@gnu.org Sender: "Guix-patches" X-Scanner: scn0 X-Spam-Score: -1.01 Authentication-Results: aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of guix-patches-bounces@gnu.org designates 2001:470:142::17 as permitted sender) smtp.mailfrom=guix-patches-bounces@gnu.org X-Scan-Result: default: False [-1.01 / 13.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; GENERIC_REPUTATION(0.00)[-0.49492288291993]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2001:470:142::/48:c]; IP_REPUTATION_HAM(0.00)[asn: 22989(0.15), country: US(-0.00), ip: 2001:470:142::17(-0.49)]; DWL_DNSWL_FAIL(0.00)[2001:470:142::17:server fail]; MX_GOOD(-0.50)[cached: eggs.gnu.org]; RCPT_COUNT_TWO(0.00)[2]; MAILLIST(-0.20)[mailman]; FREEMAIL_TO(0.00)[gmail.com]; RCVD_IN_DNSWL_FAIL(0.00)[2001:470:142::17:server fail]; RCVD_TLS_LAST(0.00)[]; R_DKIM_NA(0.00)[]; ASN(0.00)[asn:22989, ipnet:2001:470:142::/48, country:US]; MID_RHS_MATCH_FROM(0.00)[]; TAGGED_FROM(0.00)[larch=yhetil.org]; ARC_NA(0.00)[]; FORGED_RECIPIENTS_MAILLIST(0.00)[]; FROM_NEQ_ENVFROM(0.00)[ludo@gnu.org,guix-patches-bounces@gnu.org]; FROM_HAS_DN(0.00)[]; TAGGED_RCPT(0.00)[]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[gnu.org]; HAS_LIST_UNSUB(-0.01)[]; MIME_TRACE(0.00)[0:+]; RCVD_COUNT_SEVEN(0.00)[10]; FORGED_SENDER_MAILLIST(0.00)[] X-TUID: FlmJ85eBBdst Mathieu Othacehe skribis: > Raw disk-images and ISO9660 images are created in a Qemu virtual machine.= This > is quite fragile, very slow, and almost unusable without KVM. > > For all these reasons, add support for host image generation. This implie= s the > use new image generation mechanisms. > > - Raw disk images: images of partitions are created using tools such as m= ke2fs > and mkdosfs depending on the partition file-system type. The partition > images are then assembled into a final image using genimage. > > - ISO9660 images: the ISO root directory is populated within the store. G= NU > xorriso is then called on that directory, in the exact same way as this= is > done in (gnu build vm) module. > > Those mechanisms are built upon the new (gnu image) module. > > * gnu/image.scm: New file. > * gnu/system/image.scm: New file. > * gnu/build/image: New file. > * gnu/local.mk: Add them. > * gnu/system/vm.scm (system-disk-image): Rename to system-disk-image-in-v= m. > * gnu/ci.scm (qemu-jobs): Adapt to new API. > * gnu/tests/install.scm (run-install): Ditto. > * guix/scripts/system.scm (system-derivation-for-action): Ditto. Yay! > +++ b/gnu/build/image.scm Maybe we need to preserve some of the copyright lines of (gnu build vm). > +(define (root-size root) > + "Given the ROOT directory, evalute and return its size. As this doesn'= t take > +the partition metadata size into account, take a 25% margin." > + (* 1.25 (file-size root))) Perhaps =E2=80=98estimated-partition-size=E2=80=99 would be a better name? Nitpick: two spaces after end-of-sentence periods. :-) > +(define* (make-ext4-image partition target root > + #:key (owner 0)) Would it make sense to separate #:owner-uid and #:owner-gid? It does mean that we can only create images where all the files have the same UID/GID. Looking at (gnu build install), there=E2=80=99s one case where it might be problematic: the store=E2=80=99s GID is supposed to match the =E2=80=98guix= builder=E2=80=99 group. But the good news is that the daemon does this: if (chown(chrootStoreDir.c_str(), 0, buildUser.getGID()) =3D=3D -1) throw SysError(format("cannot change ownership of =E2=80=98%1%=E2= =80=99") % chrootStoreDir); So we can just remove the UID/GID from the directives that are in (gnu build install). > +(define* (genimage config target) > + "Use genimage to generate in TARGET directory, the image described in = the > +given CONFIG file." > + ;; genimage needs a 'root' directory. > + (mkdir "root") > + (invoke "genimage" "--config" config > + "--outputpath" target)) I had missed that bit, so we still need genimage in the end? > +(define (register-bootcfg-root target bootcfg) > + "On file system TARGET, register BOOTCFG as a GC root." > + (let ((directory (string-append target "/var/guix/gcroots"))) > + (mkdir-p directory) > + (symlink bootcfg (string-append directory "/bootcfg")))) Maybe just =E2=80=98register-gc-root=E2=80=99? > +(define* (register-closure prefix closure > + #:key > + (deduplicate? #t) (reset-timestamps? #t) > + (schema (sql-schema))) > + "Register CLOSURE in PREFIX, where PREFIX is the directory name of the > +target store and CLOSURE is the name of a file containing a reference gr= aph as > +produced by #:references-graphs.. As a side effect, if RESET-TIMESTAMPS= ? is > +true, reset timestamps on store files and, if DEDUPLICATE? is true, > +deduplicates files common to CLOSURE and the rest of PREFIX." > + (let ((items (call-with-input-file closure read-reference-graph))) > + (register-items items > + #:prefix prefix > + #:deduplicate? deduplicate? > + #:reset-timestamps? reset-timestamps? > + #:registration-time %epoch > + #:schema schema))) This is duplicated from (guix build vm). Should we instead factorize it in (guix build store-copy)? > +(define-module (gnu image) > + #:use-module (guix records) > + #:use-module (ice-9 match) (ice-9 match) can be removed. > + #:use-module ((srfi srfi-1) #:prefix scm:) I=E2=80=99d suggest either =E2=80=98srfi-1:=E2=80=99 as the prefix or, bett= er, hide whichever binding is causing a name clash. > +(define-syntax-rule (with-imported-modules* exp ...) > + (with-extensions gcrypt-sqlite3&co > + (with-imported-modules `(,@(source-module-closure > + '((gnu build vm) > + (gnu build image) > + (guix store database)) > + #:select? not-config?) > + ((guix config) =3D> ,(make-config.scm))) > + #~(begin > + (use-modules (gnu build vm) > + (gnu build image) > + (guix store database) > + (guix build utils)) > + exp ...)))) Probably a better name would be =E2=80=98gexp*=E2=80=99 (or =E2=80=98image-= gexp=E2=80=99) to make it clear that it builds a gexp. > +(define* (system-disk-image image > + #:key > + (name "disk-image") > + bootcfg > + bootloader > + register-closures? > + (inputs '())) [...] > + (define (image->genimage-cfg image) > + "Return as a file-like object, the genimage configuration file descr= ibing > +the given IMAGE." > + (define (format->image-type format) > + "Return the genimage format corresponding to FORMAT. For now, only= the > +hdimage format (raw disk-image) is supported." Use comments instead of docstrings for inner procedures in all this file (docstrings could not be accessed anyway). Also, two spaces after end-of-sentence periods. :-) > + (case format > + ((disk-image) "hdimage") > + (else > + (error > + (format #f "Unsupported image type ~a~%." format))))) For host-side code, better use =E2=80=9Cproper=E2=80=9D error reporting tha= t can be gracefully dealt with. So at least: (raise (condition (&message (message (format #f (G_ =E2=80=A6) =E2=80=A6)= )))) and/or a specific error condition type (well, we can leave that for later). > + (gexp->derivation name > + #~(symlink > + (string-append #$image-dir "/" #$genimage-name) > + #$output) > + #:substitutable? substitutable?))) Can we use =E2=80=98computed-file=E2=80=99 as well instead of =E2=80=98gexp= ->derivation=E2=80=99? That way, the API is entirely non-monadic and hopefully easier to use. It does mean that we need to adjust (gnu ci), (gnu tests =E2=80=A6), and (g= uix scripts system), but maybe that=E2=80=99s OK. Anyhow, the switch to non-monadic style could be made later, but not too late so we can still consider the API as not-quite-public. :-) > + (gexp->derivation name builder > + #:references-graphs inputs > + #:substitutable? substitutable?))) Same here. > +(define* (make-system-image image) > + "Return the derivation of IMAGE. It can be a raw disk-image or an ISO9= 660 > +image, depending on IMAGE format." > + (let* ((image-os (image-operating-system image)) > + (format (image-format image)) > + (file-systems-to-keep > + (scm:remove > + (lambda (fs) > + (string=3D? (file-system-mount-point fs) "/")) > + (operating-system-file-systems image-os))) > + (root-file-system-type (image->root-file-system image)) > + (substitutable? (image-substitutable? image)) > + (volatile-root? (image-volatile-root? image)) To improve readability, maybe you can use inner =E2=80=98define=E2=80=99 fo= r these things. > + (os (operating-system > + (inherit image-os) > + (initrd (lambda (file-systems . rest) > + (apply (operating-system-initrd image-os) > + file-systems > + #:volatile-root? volatile-root? > + rest))) > + (bootloader (if (eq? format 'iso9660) > + (bootloader-configuration > + (inherit > + (operating-system-bootloader image-os)) > + (bootloader grub-mkrescue-bootloader)) > + (operating-system-bootloader image-os))) > + (file-systems (cons (file-system > + (mount-point "/") > + (device "/dev/placeholder") > + (type root-file-system-type)) > + file-systems-to-keep)))) Perhaps define an auxiliary =E2=80=98operating-system-for-image=E2=80=99 pr= ocedure, just like we have =E2=80=98virtualized-operating-system=E2=80=99 & co. That=E2=80=99s all! Thanks, Ludo=E2=80=99.