[bug#57643] [PATCH 0/3] Document the image API.

[bug#57643] [PATCH 0/3] Document the image API.

[Mathieu Othacehe]

Hello,

I recently polished a bit the image API and added a new system test
(gnu tests image) that is validating the image creation process
itself with Guile-Parted.

This series makes the operating-system field of the <image> record
mandatory and adds two new chapters to the documentation: "Platforms"
and "System Images".

Thanks,

Mathieu

Mathieu Othacehe (3):
  image: Make the operating-system field mandatory.
  doc: Add a "Platforms" chapter.
  doc: Add a "System Images" chapter.

 doc/guix.texi              | 601 ++++++++++++++++++++++++++++++++++++-
 gnu/image.scm              |   3 +-
 gnu/system/image.scm       |  41 ++-
 gnu/system/images/hurd.scm |   2 +-
 4 files changed, 638 insertions(+), 9 deletions(-)

-- 
2.37.2

[bug#57643] [PATCH 1/3] image: Make the operating-system field mandatory.

[Mathieu Othacehe]

Make the operating-system field mandatory as creating an image without it
makes no sense. Introduce a new macro, image-without-os for the specific cases
where the image is only created to be inherited from afterwards.

* gnu/image.scm (<image>)[operating-system]: Make it mandatory.
* gnu/system/image.scm (image-without-os): New macro.
(efi-disk-image, efi32-disk-image, iso9660-image, docker-image,
raw-with-offset-disk-image): Use it.
* gnu/system/images/hurd.scm (hurd-disk-image): Ditto.
---
 gnu/image.scm              |  3 +--
 gnu/system/image.scm       | 41 +++++++++++++++++++++++++++++++++-----
 gnu/system/images/hurd.scm |  2 +-
 3 files changed, 38 insertions(+), 8 deletions(-)

diff --git a/gnu/image.scm b/gnu/image.scm
index 4a0068934e..68784deb12 100644
--- a/gnu/image.scm
+++ b/gnu/image.scm
@@ -170,8 +170,7 @@ (define-record-type* <image>
   (size               image-size  ;size in bytes as integer
                       (default 'guess)
                       (sanitize validate-size))
-  (operating-system   image-operating-system  ;<operating-system>
-                      (default #f))
+  (operating-system   image-operating-system)  ;<operating-system>
   (partition-table-type image-partition-table-type ; 'mbr or 'gpt
                       (default 'mbr)
                       (sanitize validate-partition-table-type))
diff --git a/gnu/system/image.scm b/gnu/system/image.scm
index a04363a130..709c3ab6ff 100644
--- a/gnu/system/image.scm
+++ b/gnu/system/image.scm
@@ -65,6 +65,7 @@ (define-module (gnu system image)
   #:use-module (ice-9 match)
   #:export (root-offset
             root-label
+            image-without-os
 
             esp-partition
             esp32-partition
@@ -102,6 +103,36 @@ (define root-offset (* 512 2048))
 ;; Generic root partition label.
 (define root-label "Guix_image")
 
+(define-syntax image-without-os
+  (lambda (x)
+    "Return an image record with the mandatory operating-system field set to
+#false.  This is useful when creating an image record that will serve as a
+parent image record."
+
+    (define (maybe-cons field acc)
+      ;; Return the given ACC list if FIELD is 'operating-system or the
+      ;; concatenation of FIELD to ACC otherwise.
+      (syntax-case field ()
+        ((f v)
+         (if (eq? (syntax->datum #'f) 'operating-system)
+             acc
+             (cons field acc)))))
+
+    (syntax-case x (image)
+      ;; Remove the operating-system field from the defined fields and then
+      ;; force it to #false.
+      ((_ fields ...)
+       (let loop ((fields #'(fields ...))
+                  (acc   '()))
+         (syntax-case fields ()
+           ((last)
+            #`(image
+               ;; Force it to #false.
+               (operating-system #false)
+               #,@(maybe-cons #'last acc)))
+           ((field rest ...)
+            (loop #'(rest ...) (maybe-cons #'field acc)))))))))
+
 (define esp-partition
   (partition
    (size (* 40 (expt 2 20)))
@@ -127,17 +158,17 @@ (define root-partition
    (initializer (gexp initialize-root-partition))))
 
 (define efi-disk-image
-  (image
+  (image-without-os
    (format 'disk-image)
    (partitions (list esp-partition root-partition))))
 
 (define efi32-disk-image
-  (image
+  (image-without-os
    (format 'disk-image)
    (partitions (list esp32-partition root-partition))))
 
 (define iso9660-image
-  (image
+  (image-without-os
    (format 'iso9660)
    (partitions
     (list (partition
@@ -146,11 +177,11 @@ (define iso9660-image
            (flags '(boot)))))))
 
 (define docker-image
-  (image
+  (image-without-os
    (format 'docker)))
 
 (define* (raw-with-offset-disk-image #:optional (offset root-offset))
-  (image
+  (image-without-os
    (format 'disk-image)
    (partitions
     (list (partition
diff --git a/gnu/system/images/hurd.scm b/gnu/system/images/hurd.scm
index 6da09b855a..2c64117c08 100644
--- a/gnu/system/images/hurd.scm
+++ b/gnu/system/images/hurd.scm
@@ -74,7 +74,7 @@ (define hurd-initialize-root-partition
                            #:wal-mode? #f)))))
 
 (define hurd-disk-image
-  (image
+  (image-without-os
    (format 'disk-image)
    (platform hurd)
    (partitions
-- 
2.37.2

[bug#57643] [PATCH 2/3] doc: Add a "Platforms" chapter.

[Mathieu Othacehe]

* doc/guix.texi ("Platforms"): New chapter.
("Porting"): Link it.
---
 doc/guix.texi | 103 +++++++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 102 insertions(+), 1 deletion(-)

diff --git a/doc/guix.texi b/doc/guix.texi
index 20abfee772..a24278e431 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -182,6 +182,7 @@ Weblate} (@pxref{Translating Guix}).
 * System Configuration::        Configuring the operating system.
 * Home Configuration::          Configuring the home environment.
 * Documentation::               Browsing software user manuals.
+* Platforms::                   Defining platforms.
 * Installing Debugging Files::  Feeding the debugger.
 * Using TeX and LaTeX::         Typesetting.
 * Security Updates::            Deploying security fixes quickly.
@@ -405,6 +406,11 @@ Defining Services
 * Shepherd Services::           A particular type of service.
 * Complex Configurations::      Defining bindings for complex configurations.
 
+Platforms
+
+* platform Reference::          Detail of platform declarations.
+* Supported Platforms::         Description of the supported platforms.
+
 Installing Debugging Files
 
 * Separate Debug Info::         Installing 'debug' outputs.
@@ -41140,6 +41146,101 @@ reader,, info-stnd, Stand-alone GNU Info}) and its Emacs counterpart
 bindings to navigate manuals.  @xref{Getting Started,,, info, Info: An
 Introduction}, for an introduction to Info navigation.
 
+@node Platforms
+@chapter Platforms
+
+The packages and systems built by Guix are intended, like most computer
+programs, to run on a CPU with a specific instruction set.  Those
+programs are often also targeting a specific kernel and system library.
+Those constraints are captured by Guix in @code{platform} records.
+
+@menu
+* platform Reference::          Detail of platform declarations.
+* Supported Platforms::         Description of the supported platforms.
+@end menu
+
+@node platform Reference
+@section @code{platform} Reference
+
+@deftp {Data Type} platform
+This is the data type representing a platform.
+
+@table @asis
+@item @code{target}
+The 'target' field must be a valid
+@uref{https://www.gnu.org/software/autoconf/manual/autoconf-2.68/html_node/Specifying-Target-Triplets.html,
+GNU triplet}, as a string.  It can be for instance,
+@code{"aarch64-linux-gnu"} and is used for cross-compilation purposes
+(@pxref{Cross-Compilation}).
+
+@item @code{system}
+The name of the corresponding system as defined in the @code{(gnu
+packages bootstrap)} module.  It can be for instance
+@code{"aarch64-linux"} or @code{"armhf-linux"}.  It is used to emulate a
+different host architecture, for instance @code{"i686-linux"} on
+@code{"x86_64-linux"}, or @code{"armhf-linux"} on @code{"x86_64-linux"},
+using the QEMU binfmt transparent emulation mechanism (@pxref{Native
+Builds}).
+
+@item @code{linux-architecture} (default: @code{#false})
+This optional string field is only relevant if the kernel is Linux.  In
+that case, it corresponds to the ARCH variable used when building Linux,
+@code{"mips"} for instance.
+
+@item @code{glibc-dynamic-linker}
+This field is the name of Glibc's dynamic linker for the corresponding
+system, as a string.  It can be @code{"/lib/ld-linux-armhf.so.3"}.
+
+@end table
+@end deftp
+
+@node Supported Platforms
+@section Supported Platforms
+
+@defvr {Scheme Variable} armv7-linux
+Platform targeting ARM v7 CPUs running GNU/Linux.
+@end defvr
+
+@defvr {Scheme Variable} aarch64-linux
+Platform targeting ARM v8 CPUs running GNU/Linux.
+@end defvr
+
+@defvr {Scheme Variable} mips64-linux
+Platform targeting MIPS 64 bits little endian CPUs running GNU/Linux.
+@end defvr
+
+@defvr {Scheme Variable} powerpc-linux
+Platform targeting PowerPC 32 bits CPUs running GNU/Linux.
+@end defvr
+
+@defvr {Scheme Variable} powerpc64le-linux
+Platform targeting PowerPC 64 bits little endian CPUs running GNU/Linux.
+@end defvr
+
+@defvr {Scheme Variable} riscv64-linux
+Platform targeting RISC-V 64 bits CPUs running GNU/Linux.
+@end defvr
+
+@defvr {Scheme Variable} i686-linux
+Platform targeting x86 CPUs running GNU/Linux.
+@end defvr
+
+@defvr {Scheme Variable} x86_64-linux
+Platform targeting x86 64 bits CPUs running GNU/Linux.
+@end defvr
+
+@defvr {Scheme Variable} i686-mingw
+Platform targeting x86 CPUs running WIN32.
+@end defvr
+
+@defvr {Scheme Variable} x86_64-mingw
+Platform targeting x86 64 bits CPUs running WIN32.
+@end defvr
+
+@defvr {Scheme Variable} hurd
+Platform targeting x86 CPUs running GNU/Hurd.
+@end defvr
+
 @node Installing Debugging Files
 @chapter Installing Debugging Files
 
@@ -41879,7 +41980,7 @@ connection between a GNU triplet (@pxref{Specifying Target Triplets, GNU
 configuration triplets,, autoconf, Autoconf}), the equivalent
 @var{system} in Nix notation, the name of the
 @var{glibc-dynamic-linker}, and the corresponding Linux architecture
-name if applicable.
+name if applicable (@pxref{Platforms}).
 
 Once the bootstrap tarball are built, the @code{(gnu packages
 bootstrap)} module needs to be updated to refer to these binaries on the
-- 
2.37.2

[bug#57643] [PATCH 2/3] doc: Add a "Platforms" chapter.

[zimoun]

Hi Mathieu,

On Wed, 07 Sep 2022 at 14:46, Mathieu Othacehe <othacehe@gnu.org> wrote:

> +* platform Reference::          Detail of platform declarations.
    -^
Why not P?

> +@menu
> +* platform Reference::          Detail of platform declarations.

Idem.

> +@item @code{system}
> +The name of the corresponding system as defined in the @code{(gnu
> +packages bootstrap)} module.

The other items are a sentence.  Here I am missing the verb.


LTGM.


Cheers,
simon

[bug#57643] [PATCH 0/3] Document the image API.

[Ludovic Courtès]

zimoun <zimon.toutoune@gmail.com> skribis:

> On Wed, 07 Sep 2022 at 14:46, Mathieu Othacehe <othacehe@gnu.org> wrote:
>
>> +* platform Reference::          Detail of platform declarations.
>     -^
> Why not P?

Because it’s the reference of the ‘platform’ data type.  :-)

Ludo’.

[bug#57643] [PATCH 0/3] Document the image API.

[Ludovic Courtès]

Mathieu Othacehe <othacehe@gnu.org> skribis:

> +The packages and systems built by Guix are intended, like most computer
> +programs, to run on a CPU with a specific instruction set.  Those

s/instruction set/instruction set, and under a specific operating system/

> +@node platform Reference
> +@section @code{platform} Reference
> +
> +@deftp {Data Type} platform

Please add a couple of lines above, like “The @code{platform} data type
describes a @dfn{platform}: an @acronym{ISA, instruction set
architecture}, combined with an operating system and possibly additional
system-wide settings such as the @acronym{ABI, application binary
interface}.”

> +This is the data type representing a platform.
> +
> +@table @asis
> +@item @code{target}
> +The 'target' field must be a valid
> +@uref{https://www.gnu.org/software/autoconf/manual/autoconf-2.68/html_node/Specifying-Target-Triplets.html,
> +GNU triplet}, as a string.

Rather:

  This field specifies the platform's GNU triplet as a string
  (@pxref{Specifying Target Triplets, GNU configuration triplets,,
  autoconf, Autoconf}).

> It can be for instance,
> +@code{"aarch64-linux-gnu"} and is used for cross-compilation purposes
> +(@pxref{Cross-Compilation}).
> +
> +@item @code{system}
> +The name of the corresponding system as defined in the @code{(gnu
> +packages bootstrap)} module.

Maybe:

  This string is the system type as it is known to Guix and passed,
  for instance, to the @option{--system} option of most commands.

  It usually has the form @code{"@var{cpu}-@var{kernel}"}, where
  @var{cpu} is the target CPU and @var{kernel} the target operating
  system kernel.

(I don’t think the (gnu packages bootstrap) is all that important when
explaining this.)

> It can be for instance
> +@code{"aarch64-linux"} or @code{"armhf-linux"}.

OK.

> It is used to emulate a
> +different host architecture, for instance @code{"i686-linux"} on
> +@code{"x86_64-linux"}, or @code{"armhf-linux"} on @code{"x86_64-linux"},
> +using the QEMU binfmt transparent emulation mechanism (@pxref{Native
> +Builds}).

If “It” denotes “the ‘system’ field”, then this is incorrect.  :-)
I’m also unsure this is the right place to discuss emulation.

I would either drop this part or make it like “You will encounter system
types when you perform native builds (@pxref{Native Builds}).” and leave
it at that.  WDYT?

> +@item @code{linux-architecture} (default: @code{#false})
> +This optional string field is only relevant if the kernel is Linux.  In
> +that case, it corresponds to the ARCH variable used when building Linux,
> +@code{"mips"} for instance.
> +
> +@item @code{glibc-dynamic-linker}
> +This field is the name of Glibc's dynamic linker for the corresponding
> +system, as a string.  It can be @code{"/lib/ld-linux-armhf.so.3"}.

Instead of “Glibc”, I suggest writing “the GNU C Library”, “the C
library”, or “glibc” as a last resort.

> +@node Supported Platforms
> +@section Supported Platforms
> +
> +@defvr {Scheme Variable} armv7-linux

Add a couple of lines above, like “The XXX module exports the following
variable, each of which is bound to a @code{platform} record.”

> +Platform targeting ARM v7 CPUs running GNU/Linux.
> +@end defvr
> +
> +@defvr {Scheme Variable} aarch64-linux
> +Platform targeting ARM v8 CPUs running GNU/Linux.
> +@end defvr
> +
> +@defvr {Scheme Variable} mips64-linux
> +Platform targeting MIPS 64 bits little endian CPUs running GNU/Linux.

General note: write “a 64-bit CPU” (hyphen, singular).

> +@defvr {Scheme Variable} i686-linux
> +Platform targeting x86 CPUs running GNU/Linux.

x/x86/Intel/?

> +@defvr {Scheme Variable} x86_64-linux
> +Platform targeting x86 64 bits CPUs running GNU/Linux.

Or x86_64?

> +@defvr {Scheme Variable} i686-mingw
> +Platform targeting x86 CPUs running WIN32.
> +@end defvr
> +
> +@defvr {Scheme Variable} x86_64-mingw
> +Platform targeting x86 64 bits CPUs running WIN32.

s/running WIN32/running Windows, with run-time support from MinGW/

> +@defvr {Scheme Variable} hurd
> +Platform targeting x86 CPUs running GNU/Hurd.

Why is not called ‘i586-gnu’?

Maybe you can write “GNU/Hurd (also referred to as ``GNU'')” to clarify
the name.

Ludo’.

[bug#57643] [PATCH 0/3] Document the image API.

[Maxime Devos]

[-- Attachment #1.1.1: Type: text/plain, Size: 2118 bytes --]

On 24-09-2022 12:10, Ludovic Courtès wrote:
>> +@defvr {Scheme Variable} i686-linux
>> +Platform targeting x86 CPUs running GNU/Linux.
> x/x86/Intel/?
>

(1) i686 is not the same as x86 -- not all x86 are 32-bit (Intel 80286 
is 16-bit) and not all x86_32 are i686 (example: i586)

(2) I don't think i486 / i586 / i686 is relevant for the variable name, 
though I suppose we could mention in the description that Guix assumes 
i686 or later (or otherwise compatible).

(3) x86 and Intel are not equivalent -- Intel has 64-bit architectures 
too: x86-64 (shared with AMD) and IA-64 (shared with HP).

Proposal: rename the variable to x86-32-linux.  Likewise for the hurd.

> +@defvr {Scheme Variable} powerpc-linux
> +Platform targeting PowerPC 32 bits CPUs running GNU/Linux.
> +@end defvr
> +
> +@defvr {Scheme Variable} powerpc64le-linux
> +Platform targeting PowerPC 64 bits little endian CPUs running GNU/Linux.
> +@end defvr

Why is the endian mentioned for powerpc64le-linux and not powerpc (in 
the description and in the variable name)?  This looks inconsistent. 
(From what I've read, PowerPC has both a little-endian and a big-endian 
mode.)

> +@defvr {Scheme Variable} i686-mingw
> +Platform targeting x86 CPUs running WIN32.
> +@end defvr
> +
> +@defvr {Scheme Variable} x86_64-mingw
> +Platform targeting x86 64 bits CPUs running WIN32.
> +@end defvr

I don't think mentioning the version of the Windows API is relevant 
information, similar to how we the version of glibc or linux is not 
mentioned for the -linux variables.  I think it would be clearer to 
mention Microsoft Windows directly instead (and easier to search for).

>> +@defvr {Scheme Variable} mips64-linux
>> +Platform targeting MIPS 64 bits little endian CPUs running GNU/Linux.

'little endian' is an adjective here, so I think 'little-endian' would 
be appropriate here.

>> +@defvr {Scheme Variable} hurd
>> +Platform targeting x86 CPUs running GNU/Hurd.
> 
> Why is not called ‘i586-gnu’?

How about x86-32-gnu, for the same reasons as i686-linux?

Greetings,
Maxime.

[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 929 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 236 bytes --]

[bug#57643] [PATCH 0/3] Document the image API.

[Mathieu Othacehe]

Hello Maxime,

> Proposal: rename the variable to x86-32-linux.  Likewise for the hurd.

It makes sense to me. However, we would also have to rename accordingly
the "system" counterpart in (gnu packages bootstrap). Something to keep
in mind.

> Why is the endian mentioned for powerpc64le-linux and not powerpc (in the
> description and in the variable name)?  This looks inconsistent. (From what
> I've read, PowerPC has both a little-endian and a big-endian mode.)

Fixed.

> 'little endian' is an adjective here, so I think 'little-endian' would be
> appropriate here.

Fixed.

>>> +@defvr {Scheme Variable} hurd
>>> +Platform targeting x86 CPUs running GNU/Hurd.
>> Why is not called ‘i586-gnu’?
>
> How about x86-32-gnu, for the same reasons as i686-linux?

Also looks fine but has to be done at a larger scale like for the
x86-32-linux renaming.

Thanks for reviewing,

Mathieu

[bug#57643] [PATCH 0/3] Document the image API.

[Ludovic Courtès]

Hi,

Maxime Devos <maximedevos@telenet.be> skribis:

> Proposal: rename the variable to x86-32-linux.  Likewise for the hurd.

While we’re discussing the color of the bikeshed :-), I’d like to point
out that “x86_32” or “x86-32” is not a thing.  The official name is
either IA32 or, more specifically, i686, etc.  I’m in favor of sticking
to official (nick)names consistently.

Ludo’.

[bug#57643] [PATCH 0/3] Document the image API.

[Maxime Devos]

[-- Attachment #1.1.1: Type: text/plain, Size: 1713 bytes --]


On 24-09-2022 18:33, Ludovic Courtès wrote> Hi,
> 
> Maxime Devos<maximedevos@telenet.be>  skribis:
> 
>> Proposal: rename the variable to x86-32-linux.  Likewise for the hurd.
> While we’re discussing the color of the bikeshed :-), I’d like to point
> out that “x86_32” or “x86-32” is not a thing.

It is a thing if we let it be a thing.  It also already is a thing: 
target-x86-32? exists, "x86-32" finds some relevant search results 
(though it can be confused with another meaning of "x86-32" -- an ABI 
where pointers are 32-bit but all of the x86-64 instruction set remains 
available, so far from an ideal naming.)

> The official name is
> either IA32 or, more specifically, i686, etc.

In my experience, IA-32 is not a thing except in Intel documents and 
various irregular exceptions, however official it might be ... which 
seems similar to x86-32.

> I’m in favor of sticking
> to official (nick)names consistently.

Going by 
https://www.intel.com/content/www/us/en/developer/articles/technical/intel-sdm.html 
, the official name is IA-32, not IA32.

IA-32 sounds nice to me though, we could make that a thing in Guix, 
though for consistency 'target-x86-32?' would need to be renamed to 
'target-ia32?' (I don't think the original casing and hyphenation is 
important for procedure names).

I don't see the point of going for i686 -- AFAIK, Guix might as well 
have chosen i586 as a minimal supported version, and if it weren't for 
32-bit seemingly being phased out, there might be a i786 eventually and 
Guix might eventually require i786 -- the mention of a particular 
microarchitecture doesn't seem relevant to me.

Greetings,
MAxime.

[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 929 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 236 bytes --]

[bug#57643] [PATCH 0/3] Document the image API.

[Ludovic Courtès]

Maxime Devos <maximedevos@telenet.be> skribis:

> Going by
> https://www.intel.com/content/www/us/en/developer/articles/technical/intel-sdm.html
> , the official name is IA-32, not IA32.
>
> IA-32 sounds nice to me though, we could make that a thing in Guix,
> though for consistency 'target-x86-32?' would need to be renamed to
> 'target-ia32?' (I don't think the original casing and hyphenation is
> important for procedure names).

We could make that change, yes.

(I think this is secondary compared to the change you proposed to make,
which is to rename ‘i686-linux’ (the variable) to something else.  I
think the variable name must match the system type, and the system type
is non-negotiable because it’s what lets Guix distinguish between
derivations for two different hardware platforms.)

Ludo’.

[bug#57643] [PATCH 0/3] Document the image API.

[Mathieu Othacehe]

Hey,

> Why is not called ‘i586-gnu’?
>
> Maybe you can write “GNU/Hurd (also referred to as ``GNU'')” to clarify
> the name.

I added a preliminary patch doing this rename.

Thanks,

Mathieu

[bug#57643] [PATCH 3/3] doc: Add a "System Images" chapter.

[Mathieu Othacehe]

* doc/guix.texi ("System Images"): New chapter.
---
 doc/guix.texi | 498 ++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 498 insertions(+)

diff --git a/doc/guix.texi b/doc/guix.texi
index a24278e431..6a5824d4ab 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -183,6 +183,7 @@ Weblate} (@pxref{Translating Guix}).
 * Home Configuration::          Configuring the home environment.
 * Documentation::               Browsing software user manuals.
 * Platforms::                   Defining platforms.
+* System Images::               Creating system images.
 * Installing Debugging Files::  Feeding the debugger.
 * Using TeX and LaTeX::         Typesetting.
 * Security Updates::            Deploying security fixes quickly.
@@ -411,6 +412,13 @@ Platforms
 * platform Reference::          Detail of platform declarations.
 * Supported Platforms::         Description of the supported platforms.
 
+System Images
+
+* image Reference::             Detail of image declarations.
+* Instantiate an Image::        How to instantiate an image record.
+* image-type Reference::        Detail of image types declaration.
+* Image Modules::               Definition of image modules.
+
 Installing Debugging Files
 
 * Separate Debug Info::         Installing 'debug' outputs.
@@ -41241,6 +41249,496 @@ Platform targeting x86 64 bits CPUs running WIN32.
 Platform targeting x86 CPUs running GNU/Hurd.
 @end defvr
 
+@node System Images
+@chapter Creating System Images.
+
+@cindex system images
+When it comes to installing Guix System for the first time on a new
+machine, you can basically proceed in three different ways.  The first
+one is to use an existing operating system on the machine to run the
+@command{guix system init} command (@pxref{Invoking guix system}).  The
+second one, is to produce an installation image (@pxref{Building the
+Installation Image}). This is a bootable system which role is to
+eventually run @command{guix system init}.  Finally, the third option
+would be to produce an image that is a direct instantiation of the
+system you wish to run.  That image can then be copied on a bootable
+device such as an USB drive or a memory card.  The target machine would
+then directly boot from it, without any kind of installation procedure.
+
+The @command{guix system image} command is able to turn an operating
+system definition into a bootable image.  This command supports
+different image types, such as @code{efi-raw}, @code{iso9660} and
+@code{docker}.  Any modern @code{x86_64} machine will probably be able
+to boot from an @code{iso9660} image.  However, there are a few machines
+out there that require specific image types.  Those machines, in general
+using @code{ARM} processors, may expect specific partitions at specific
+offsets.
+
+This chapter explains how to define customized system images and how to
+turn them into actual bootable images.
+
+@menu
+* image Reference::        Detail of image declarations.
+* Instantiate an Image::   How to instantiate an image record.
+* image-type Reference::   Detail of image types declaration.
+* Image Modules::          Definition of image modules.
+@end menu
+
+@node image Reference
+@section @code{image} Reference
+
+The @code{image} record, described right after, allows you to define a
+customized bootable system image.
+
+@deftp {Data Type} image
+This is the data type representing a system image.
+
+@table @asis
+@item @code{name} (default: @code{#false})
+The image name as a symbol, @code{'my-iso9660} for instance.  The name
+is optional and it defaults to @code{#false}.
+
+@item @code{format}
+The image format as a symbol.  The following formats are supported:
+
+@itemize
+@item @code{disk-image}, a raw disk image composed of one or multiple
+partitions.
+
+@item @code{compressed-qcow2}, a compressed qcow2 image composed of
+one or multiple partitions.
+
+@item @code{docker}, a Docker image.
+
+@item @code{iso9660}, an ISO-9660 image.
+
+@end itemize
+
+@item @code{platform} (default: @code{#false})
+The @code{platform} record the image is targeting (@pxref{Platforms}),
+@code{aarch64-linux} for instance.  By default, this field is set to
+@code{#false} and the image will target the host platform.
+
+@item @code{size} (default: @code{'guess})
+The image size in bytes or @code{'guess}.  The @code{'guess} symbol,
+which is the default, means that the image size will be inferred based
+on the image content.
+
+@item @code{operating-system}
+The image's @code{operating-system} record that is instanciated.
+
+@item @code{partition-table-type} (default: @code{'mbr})
+The image partition table type as a symbol.  Possible values are
+@code{'mbr} and @code{'gpt}.  It default to @code{'mbr}.
+
+@item @code{partitions} (default: @code{'()})
+The image partitions as a list of @code{partition} records
+(@pxref{partition Reference}).
+
+@item @code{compression?} (default: @code{#true})
+Whether the image content should be compressed, as a boolean.  It
+defaults to @code{#true} and only applies to @code{'iso9660} image
+formats.
+
+@item @code{volatile-root?} (default: @code{#true})
+Whether the image root partition should be made volatile, as a boolean.
+
+This is achieved by using a RAM backed file system (overlayfs) that is
+mounted on top of the root partition by the initrd.  It defaults to
+@code{#true}.  When set to @code{#false}, the image root partition is
+mounted as read-write partition by the initrd.
+
+@item @code{shared-store?} (default: @code{#false})
+Whether the image's store should be shared with the host system, as a
+boolean.  This can be useful when creating images dedicated to virtual
+machines.  When set to @code{#false}, which is the default, the image's
+@code{operating-system} closure is copied to the image.  Otherwise, when
+set to @code{#true}, it is assumed that the host store will be made
+available at boot, using a @code{9p} mount for instance.
+
+@item @code{shared-network?} (default: @code{#false})
+Whether to use the host network interfaces within the image, as a
+boolean.  This is only used for the @code{'docker} image format.  It
+defaults to @code{#false}.
+
+@item @code{substitutable?} (default: @code{#true})
+Whether the image derivation should be substitutable, as a boolean.  It
+defaults to @code{true}.
+
+@end table
+@end deftp
+
+@node partition Reference
+@subsection @code{partition} Reference
+
+In @code{image} record may contain some partitions.
+
+@deftp {Data Type} partition
+This is the data type representing an image partition.
+
+@table @asis
+@item @code{size} (default: @code{'guess})
+The partition size in bytes or @code{'guess}.  The @code{'guess} symbol,
+which is the default, means that the partition size will be inferred
+based on the partition content.
+
+@item @code{offset} (default: @code{0})
+The partition's start offset in bytes, relative to the image start or
+the previous partition end.  It defaults to @code{0} which means that
+there is no offset applied.
+
+@item @code{file-system} (default: @code{"ext4"})
+The partition file system as a string, defaulting to @code{"ext4"}.  The
+supported values are @code{"vfat"}, @code{"fat16"}, @code{"fat32"} and
+@code{"ext4"}.
+
+@item @code{file-system-options} (default: @code{'()})
+The partition file system creation options that should be passed to the
+partition creation tool, as a list of strings.  This is only supported
+when creating @code{"ext4"} partitions.
+
+See the @code{"extended-options"} man page section of the
+@code{"mke2fs"} tool for a more complete reference.
+
+@item @code{label}
+The partition label as a mandatory string, @code{"my-root"} for
+instance.
+
+@item @code{uuid} (default: @code{#false})
+The partition UUID as an @code{uuid} record (@pxref{File Systems}).  By
+default it is @code{#false}, which means that the partition creation
+tool will attribute a random UUID to the partition.
+
+@item @code{flags} (default: @code{'()})
+The partition flags as a list of symbols.  Possible values are
+@code{'boot} and @code{'esp}.  The @code{'boot} flags should be set if
+you want to boot from this partition.  Exactly one partition should have
+this flag set, usually the root one. The @code{'esp} flag identifies a
+UEFI System Partition.
+
+@item @code{initializer} (default: @code{#false})
+The partition initializer procedure as a gexp.  This procedure is called
+to populate a partition.  If no initializer is passed, the
+@code{initialize-root-partition} procedure from the @code{(gnu build
+image)} module is used.
+
+@end table
+@end deftp
+
+@node Instantiate an Image
+@section Instantiate an Image
+
+Let's say you would like to create an MBR image with three distinct
+partitions:
+
+@itemize
+@item ESP vfat partition of 40MiB at offset 1024KiB.
+@item DATA ext4 partition of 50MiB containing a dummy data file.
+@item ROOT ext4 bootable partition containing the @code{%simple-os}
+operating-system.
+@end itemize
+
+You would then write the following image definition in a
+@code{my-image.scm} file for instance.
+
+@lisp
+(use-modules (gnu)
+             (gnu image)
+             (gnu tests)
+             (gnu system image)
+             (guix gexp))
+
+(define MiB (expt 2 20))
+
+(image
+ (format 'disk-image)
+ (operating-system %simple-os)
+ (partitions
+  (list
+   (partition
+    (size (* 40 MiB))
+    (offset (* 1024 1024))
+    (label "GNU-ESP")
+    (file-system "vfat")
+    (flags '(esp))
+    (initializer (gexp initialize-efi-partition)))
+   (partition
+    (size (* 50 MiB))
+    (label "DATA")
+    (file-system "ext4")
+    (initializer #~(lambda* (root . rest)
+                     (mkdir root)
+                     (call-with-output-file
+                         (string-append root "/data")
+                       (lambda (port)
+                         (format port "my-data"))))))
+   (partition
+    (size 'guess)
+    (label root-label)
+    (file-system "ext4")
+    (flags '(boot))
+    (initializer (gexp initialize-root-partition))))))
+@end lisp
+
+Note that the first and third partitions use generic initializers
+procedures, initialize-efi-partition and initialize-root-partition
+respectively.  The initialize-efi-partition installs a GRUB EFI loader
+that is loading the GRUB bootloader located in the root partition.  The
+initialize-root-partition instantiates a complete system as defined by
+the @code{%simple-os} operating-system.
+
+You can now run:
+
+@example
+guix system image my-image.scm
+@end example
+
+to instantiate the @code{image} definition.  That produces a disk image
+which has the expected structure:
+
+@example
+$ parted $(guix system image my-image.scm) print
+@dots{}
+Model:  (file)
+Disk /gnu/store/yhylv1bp5b2ypb97pd3bbhz6jk5nbhxw-disk-image: 1714MB
+Sector size (logical/physical): 512B/512B
+Partition Table: msdos
+Disk Flags:
+
+Number  Start   End     Size    Type     File system  Flags
+ 1      1049kB  43.0MB  41.9MB  primary  fat16        esp
+ 2      43.0MB  95.4MB  52.4MB  primary  ext4
+ 3      95.4MB  1714MB  1619MB  primary  ext4         boot
+@end example
+
+The size of the @code{boot} partition has been inferred to @code{1619MB}
+so that it is large enough to host the @code{%simple-os}
+operating-system.
+
+You can also use existing @code{image} record definitions and inherit
+from them to simplify the @code{image} definition.  The @code{(gnu
+system image)} module provides the following @code{image} definitions
+variables.
+
+@defvr {Scheme Variable} efi-disk-image
+A MBR disk-image composed of two partitions: a 64 bits ESP partition and
+a ROOT boot partition.  This image can be used on most @code{x86_64} and
+@code{i686} machines, supporting BIOS or UEFI booting.
+@end defvr
+
+@defvr {Scheme Variable} efi32-disk-image
+Same as @code{efi-disk-image} but with a 32 bits EFI partition.
+@end defvr
+
+@defvr {Scheme Variable} iso9660-image
+An ISO-9660 image composed of a single bootable partition.  This image
+can also be used on most @code{x86_64} and @code{i686} machines.
+@end defvr
+
+@defvr {Scheme Variable} docker-image
+A Docker image that can be used to spawn a Docker container.
+@end defvr
+
+Using the @code{efi-disk-image} we can simplify our previous
+@code{image} declaration this way:
+
+@example
+(use-modules (gnu)
+             (gnu image)
+             (gnu tests)
+             (gnu system image)
+             (guix gexp)
+             (ice-9 match))
+
+(define MiB (expt 2 20))
+
+(define data
+  (partition
+   (size (* 50 MiB))
+   (label "DATA")
+   (file-system "ext4")
+   (initializer #~(lambda* (root . rest)
+                    (mkdir root)
+                    (call-with-output-file
+                        (string-append root "/data")
+                      (lambda (port)
+                        (format port "my-data")))))))
+
+(image
+ (inherit efi-disk-image)
+ (operating-system %simple-os)
+ (partitions
+  (match (image-partitions efi-disk-image)
+    ((esp root)
+     (list esp data root)))))
+@end example
+
+This will give the exact same @code{image} instantiation but the
+@code{image} declaration is simpler.
+
+@node image-type Reference
+@section image-type Reference
+
+The @command{guix system image} command can, as we saw above, take a
+file containing an @code{image} declaration as argument and produce an
+actual disk image from it.  The same command can also handle a file
+containing an @code{operating-system} declaration as argument.  In that
+case, how is the @code{operating-system} turned into an image?
+
+That's where the @code{image-type} record intervenes.  This record
+defines how to transform an @code{operating-system} record into an
+@code{image} record.
+
+@deftp {Data Type} image-type
+This is the data type representing an image-type.
+
+@table @asis
+@item @code{name}
+The image-type name as a mandatory symbol, @code{'efi32-raw} for
+instance.
+
+@item @code{constructor}
+The image-type constructor, as a mandatory procedure that takes an
+@code{operating-system} record as argument and returns an @code{image}
+record.
+
+@end table
+@end deftp
+
+There are several @code{image-type} records provided by the @code{(gnu
+system image)} and the @code{(gnu system images @dots{})} modules.
+
+@defvr {Scheme Variable} efi-raw-image-type
+Build an image based on the @code{efi-disk-image} image.
+@end defvr
+
+@defvr {Scheme Variable} efi32-raw-image-type
+Build an image based on the @code{efi32-disk-image} image.
+@end defvr
+
+@defvr {Scheme Variable} qcow2-image-type
+Build an image based on the @code{efi-disk-image} image but with the
+@code{compressed-qcow2} image format.
+@end defvr
+
+@defvr {Scheme Variable} iso-image-type
+Build a compressed image based on the @code{iso9660-image} image.
+@end defvr
+
+@defvr {Scheme Variable} uncompressed-iso-image-type
+Build an image based on the @code{iso9660-image} image but with the
+@code{compression?} field set to @code{#false}.
+@end defvr
+
+@defvr {Scheme Variable} docker-image-type
+Build an image based on the @code{docker-image} image.
+@end defvr
+
+@defvr {Scheme Variable} raw-with-offset-image-type
+Build an MBR image with a single partition starting at a @code{1024KiB}
+offset.  This is useful to leave some room to install a bootloader in
+the post-MBR gap.
+@end defvr
+
+@defvr {Scheme Variable} pinebook-pro-image-type
+Build an image that is targeting the Pinebook Pro machine.  The MBR
+image contains a single partition starting at a @code{9MiB} offset.  The
+@code{u-boot-pinebook-pro-rk3399-bootloader} bootloader will be
+installed in this gap.
+@end defvr
+
+@defvr {Scheme Variable} rock64-image-type
+Build an image that is targeting the Rock64 machine.  The MBR image
+contains a single partition starting at a @code{16MiB} offset.  The
+@code{u-boot-rock64-rk3328-bootloader} bootloader will be installed in
+this gap.
+@end defvr
+
+@defvr {Scheme Variable} novena-image-type
+Build an image that is targeting the Novena machine.  It has the same
+characteristics as @code{raw-with-offset-image-type}.
+@end defvr
+
+@defvr {Scheme Variable} pine64-image-type
+Build an image that is targeting the Pine64 machine.  It has the same
+characteristics as @code{raw-with-offset-image-type}.
+@end defvr
+
+@defvr {Scheme Variable} hurd-image-type
+Build an image that is targeting a @code{i386} machine running the Hurd
+kernel.  The MBR image contains a single ext2 partitions with specific
+@code{file-system-options} flags.
+@end defvr
+
+@defvr {Scheme Variable} hurd-qcow2-image-type
+Build an image similar to the one built by the @code{hurd-image-type}
+but with the @code{format} set to @code{'compressed-qcow2}.
+@end defvr
+
+So, if we get back to the @code{guix system image} command taking an
+@code{operating-system} declaration as argument.  By default, the
+@code{efi-raw-image-type} is used to turn the provided
+@code{operating-system} into an actual bootable image.
+
+To use a different @code{image-type}, the @code{--image-type} option can
+be used.  The @code{--list-image-types} option will list all the
+supported image types.  It turns out to be a textual listing of all the
+@code{image-types} variables described just above (@pxref{Invoking guix
+system}).
+
+@node Image Modules
+@section Image Modules
+
+Let's take the example of the Pine64, an ARM based machine.  To be able
+to produce an image targeting this board, we need the following
+elements:
+
+@itemize
+@item An @code{operating-system} record containing at least
+an appropriate kernel (@code{linux-libre-arm64-generic}) and bootloader
+@code{u-boot-pine64-lts-bootloader}) for the Pine64.
+
+@item Possibly, an @code{image-type} record providing a way to
+turn an @code{operating-system} record to an @code{image} record
+suitable for the Pine64.
+
+@item An actual @code{image} that can be instantiated with the
+@command{guix system image} command.
+
+@end itemize
+
+The @code{(gnu system images pine64)} module provides all those
+elements: @code{pine64-barebones-os}, @code{pine64-image-type} and
+@code{pine64-barebones-raw-image} respectively.
+
+The module returns the @code{pine64-barebones-raw-image} in order for
+users to be able to run:
+
+@example
+guix system image gnu/system/images/pine64.scm
+@end example
+
+Now, thanks to the @code{pine64-image-type} record declaring the
+@code{'pine64-raw} @code{image-type}, one could also prepare a
+@code{my-pine.scm} file with the following content:
+
+@example
+(use-modules (gnu system images pine64))
+(operating-system
+  (inherit pine64-barebones-os)
+  (timezone "Europe/Athens"))
+@end example
+
+to customize the @code{pine64-barebones-os}, and run:
+
+@example
+$ guix system image --image-type=pine64-raw my-pine.scm
+@end example
+
+Note that there are other modules in the @code{gnu/system/images}
+directory targeting @code{Novena}, @code{Pine64}, @code{PinebookPro} and
+@code{Rock64} machines.
+
 @node Installing Debugging Files
 @chapter Installing Debugging Files
 
-- 
2.37.2

[bug#57643] [PATCH 3/3] doc: Add a "System Images" chapter.

[zimoun]

Hi,

On Wed, 07 Sep 2022 at 14:46, Mathieu Othacehe <othacehe@gnu.org> wrote:

> +You can also use existing @code{image} record definitions and inherit
> +from them to simplify the @code{image} definition.  The @code{(gnu
> +system image)} module provides the following @code{image} definitions
> +variables.

extra ’s’ in ’definition’, no?


LTGM.

Cheers,
simon

[bug#57643] [PATCH 0/3] Document the image API.

[Ludovic Courtès]

Mathieu Othacehe <othacehe@gnu.org> skribis:

> * doc/guix.texi ("System Images"): New chapter.

[...]

> +@node Instantiate an Image
> +@section Instantiate an Image
> +
> +Let's say you would like to create an MBR image with three distinct
> +partitions:
> +
> +@itemize
> +@item ESP vfat partition of 40MiB at offset 1024KiB.

Maybe:

  @item The @acronym{ESP, EFI System Partition}, a partition of
  40@tie{}MiB at offset 1024@tie{}KiB with a vfat file system.

> +@item DATA ext4 partition of 50MiB containing a dummy data file.

  @item an ext4 partition of … data file, and labeled ``data''.

> +@item ROOT ext4 bootable partition containing the @code{%simple-os}

Likewise.

This section looks great!

[bug#57643] [PATCH 1/3] image: Make the operating-system field mandatory.

[Liliana Marie Prikler]

Hi,

Am Mittwoch, dem 07.09.2022 um 14:46 +0200 schrieb Mathieu Othacehe:
> Make the operating-system field mandatory as creating an image
> without it
> makes no sense. Introduce a new macro, image-without-os for the
> specific cases
> where the image is only created to be inherited from afterwards.
> 
> * gnu/image.scm (<image>)[operating-system]: Make it mandatory.
> * gnu/system/image.scm (image-without-os): New macro.
> (efi-disk-image, efi32-disk-image, iso9660-image, docker-image,
> raw-with-offset-disk-image): Use it.
> * gnu/system/images/hurd.scm (hurd-disk-image): Ditto.
IMHO adding a new macro is a bit much.  Can't we simply add an empty
image from which the others are derived or use (operating-system #f)
literally?  (Alternatively to #f you might prefer "none" as syntactic
sugar.)

I do agree with making operating-system mandatory, though.

Cheers

[bug#57643] [PATCH 0/3] Document the image API.

[Ludovic Courtès]

Hi,

Liliana Marie Prikler <liliana.prikler@gmail.com> skribis:

> Am Mittwoch, dem 07.09.2022 um 14:46 +0200 schrieb Mathieu Othacehe:
>> Make the operating-system field mandatory as creating an image
>> without it
>> makes no sense. Introduce a new macro, image-without-os for the
>> specific cases
>> where the image is only created to be inherited from afterwards.
>> 
>> * gnu/image.scm (<image>)[operating-system]: Make it mandatory.
>> * gnu/system/image.scm (image-without-os): New macro.
>> (efi-disk-image, efi32-disk-image, iso9660-image, docker-image,
>> raw-with-offset-disk-image): Use it.
>> * gnu/system/images/hurd.scm (hurd-disk-image): Ditto.
> IMHO adding a new macro is a bit much.  Can't we simply add an empty
> image from which the others are derived or use (operating-system #f)
> literally?  (Alternatively to #f you might prefer "none" as syntactic
> sugar.)

I sympathize with what you write, though I don’t have a strong opinion.

> I do agree with making operating-system mandatory, though.

Same here!

Ludo’.

[bug#57643] [PATCH 1/3] image: Make the operating-system field mandatory.

[Maxime Devos]

[-- Attachment #1.1.1: Type: text/plain, Size: 1681 bytes --]



On 07-09-2022 14:46, Mathieu Othacehe wrote:
> +(define-syntax image-without-os
> +  (lambda (x)
> +    "Return an image record with the mandatory operating-system field set to
> +#false.  This is useful when creating an image record that will serve as a
> +parent image record > +
> +    (define (maybe-cons field acc)
> +      ;; Return the given ACC list if FIELD is 'operating-system or the
> +      ;; concatenation of FIELD to ACC otherwise.
> +      (syntax-case field ()
> +        ((f v)
> +         (if (eq? (syntax->datum #'f) 'operating-system)
> +             acc
> +             (cons field acc)))) > +
> +    (syntax-case x (image)
> +      ;; Remove the operating-system field from the defined fields and then
> +      ;; force it to #false.
> +      ((_ fields ...)
> +       (let loop ((fields #'(fields ...))
> +                  (acc   '()))
> +         (syntax-case fields ()
> +           ((last)
> +            #`(image
> +               ;; Force it to #false.
> +               (operating-system #false)
> +               #,@(maybe-cons #'last acc)))
> +           ((field rest ...)
> +            (loop #'(rest ...) (maybe-cons #'field acc)))))))))

The complexity of this 'without os' macro seems to come from accepting 
an 'os' and then throwing it away.  However, when there is an 'os', you 
might as well use 'image' directly, without 'image-without-os', so I 
think this macro can be simplified to:

(define-syntax-rule (image-without-os . settings)
   "docstring"
   (image (operating-system #false) . settings))

(IIUC, '(guix records)' will detect duplicate definitions of fields.)

Greetings,
Maxime.

[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 929 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 236 bytes --]

[bug#57643] [PATCH 1/3] image: Make the operating-system field mandatory.

[Mathieu Othacehe]

Hey,

> The complexity of this 'without os' macro seems to come from accepting an 'os'
> and then throwing it away.  However, when there is an 'os', you might as well
> use 'image' directly, without 'image-without-os', so I think this macro can be
> simplified to:

Yeah this was mostly a good pretext to have fun with macros, I agree
your alternative makes more sense. I used your proposed approach instead.

> (IIUC, '(guix records)' will detect duplicate definitions of fields.)

It does.

Thanks,

Mathieu

[bug#57643] [PATCH 0/3] Document the image API.

[Ludovic Courtès]

Hi Mathieu,

Mathieu Othacehe <othacehe@gnu.org> skribis:

> I recently polished a bit the image API and added a new system test
> (gnu tests image) that is validating the image creation process
> itself with Guile-Parted.
>
> This series makes the operating-system field of the <image> record
> mandatory and adds two new chapters to the documentation: "Platforms"
> and "System Images".

Great piece of work!  That will make things more approachable, and one
can hope it will lead folks in the embedded or devops space to look more
closely as what this has to offer.

I sent mostly cosmetic comments.  It looks pretty much ready to go.

Thank you!

Ludo’.

[bug#57643] [PATCH 0/3] Document the image API.

[Mathieu Othacehe]

Hey Ludo,

> Great piece of work!  That will make things more approachable, and one
> can hope it will lead folks in the embedded or devops space to look more
> closely as what this has to offer.
>
> I sent mostly cosmetic comments.  It looks pretty much ready to go.

Many thanks for the in-depth review, as always. I fixed all your
comments. I also hope it will help people dive into the image creation
stuff.

The two big remaining areas left to be a credible alternative to
buildroot/Yocto and friends are in my opinion:

1. Be able to generate smaller images (without Guix/Guile), with
other init systems (initV, android init language), busybox, minimal
libc and so on.

2. Be able to cross-compile more packages (qt, gnome, ...).

Some work in perspective!

Mathieu

bug#57643: [PATCH 0/3] Document the image API.

[Mathieu Othacehe]

Hey,

> I sent mostly cosmetic comments.  It looks pretty much ready to go.

Pushed, thanks to all reviewers!

Mathieu