Guix System automated installation

Guix System automated installation

[Giovanni Biscuolo]

[-- Attachment #1: Type: text/plain, Size: 4647 bytes --]

Hello Ian,

I'm a little late to this discussion, sorry.

I'm adding guix-devel since it would be nice if some Guix developer have
something to add on this matter, for this reason I'm leaving all
previous messages intact

Csepp <raingloom@riseup.net> writes:

> Ian Eure <ian@retrospec.tv> writes:
>
>> Hello,
>>
>> On Debian, you can create a preseed file containing answers to all the questions
>> you’re prompted for during installation, and build a new install image which
>> includes it.  When booted, this installer skips any steps which have been
>> preconfigured, which allows for either fully automated installation, or partly
>> automated (prompt for hostname and root password, but otherwise automatic).
>>
>> Does Guix have a way to do something like this?  The declarative config is more
>> or less the equivalent of the Debian preseed file, but I don’t see anything that
>> lets you build an image that’ll install a configuration.

When using the guided installation (info "(guix) Guided Graphical
Installation"), right before the actual installation on target (guix
system init...) you can edit the operating-system configuration file:
isn't it something similar to what you are looking for?

Please consider that a preseed file is very limited compared to a
full-fledged operating-system declaration since the latter contains the
declaration for *all* OS configuration, not just the installed packages.

Alternatively, you can use the (info "(guix) Manual Installation") and
copy a pre-configured (preseed? :-) ) operating-system file, but you
have to be very careful (see (info "(guix) Proceeding with the
Installation").

>> I see there’s `guix deploy’, but that requires an already-installed GuixSD to
>> work, which isn’t helpful for getting it installed in the first place.
>>
>> Thanks,
>>
>>  — Ian

I'm also interested in a way to fully automate the installation [1] of
Guix System hosts and I've developed a small bash script to help me (see
below).

The idea is to use the script to install a very basic Guix System on the
machine and then use "guix deploy" (or deploy "manually") for a
full-fledged configuration.

My initial motivation was (and sill is the main) to allow me to install
Guix Systems on rented hosts (dedicates or VPS) provided by vendors that
do not have Guix System in the list of operating systems users can
install on their machines: in this case users can boot machines in
rescue mode (AFAIU all hosters provide a rescue system) and insall Guix
System in a similar way as described in (info "(guix-cookbook) Running
Guix on a Linode Server") or (info "(guix-cookbook) Running Guix on a
Kimsufi Server")

You can find the script here:
https://gitlab.com/softwareworkers/swws/-/blob/master/infrastructure/hosts/cornouiller/bootstrap-guix.sh?ref_type=heads
(that is the last "version" I used, for now I write a script for every
machine I need... I still have to make this script generic putting all 
needed config variables in an external file)

Please consider it's still in early development, although I've already
tested it both locally and with real rented machines, both bare metal
and VPS.

After some tests I realized that with few tests I could use such a
script both on a rescue system and when installing using the Guix
Installer ISO, selecting a full manual installation, see (info
"(guix) Manual Installation"), and then running the script.

> guix system image is maybe closer, but it doesn’t automate everything that the
> installer does.
> But the installer can be used as a Scheme library, at least in theory.  The way
> I would approach the problem is by creating a Shepherd service that runs at boot
> from the live booted ISO.

I would really Love So Much™ to avoid writing imperative bash scripts
and just write Scheme code to be able to do a "full automatic" Guix
System install, using a workflow like this one:

1. guix system prepare --include preseed.scm disk-layout.scm /mnt

where disk-layout.scm is a declarative gexp used to partition, format
and mount all needed filesystems

the resulting config.scm would be an operating-system declaration with
included the contents of preseed.scm (packages and services
declarations)

2. guix system init config.scm /mnt (already working now)

...unfortunately I'm (still?!?) not able to contribute such code :-(


Happy hacking! Gio'



[1] that means: with almost zero needed intervention by the user... the
user just needs to _design_ the installation.

-- 
Giovanni Biscuolo

Xelera IT Infrastructures

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 849 bytes --]

Re: Guix System automated installation

[Ian Eure]

Hi Giovanni,

Giovanni Biscuolo <g@xelera.eu> writes:

> [[PGP Signed Part:Undecided]]
> Hello Ian,
>
> I'm a little late to this discussion, sorry.
>
> I'm adding guix-devel since it would be nice if some Guix 
> developer have
> something to add on this matter, for this reason I'm leaving all
> previous messages intact
>
> Csepp <raingloom@riseup.net> writes:
>
>> Ian Eure <ian@retrospec.tv> writes:
>>
>>> Hello,
>>>
>>> On Debian, you can create a preseed file containing answers to 
>>> all the questions
>>> you’re prompted for during installation, and build a new 
>>> install image which
>>> includes it.  When booted, this installer skips any steps 
>>> which have been
>>> preconfigured, which allows for either fully automated 
>>> installation, or partly
>>> automated (prompt for hostname and root password, but 
>>> otherwise automatic).
>>>
>>> Does Guix have a way to do something like this?  The 
>>> declarative config is more
>>> or less the equivalent of the Debian preseed file, but I don’t 
>>> see anything that
>>> lets you build an image that’ll install a configuration.
>
> When using the guided installation (info "(guix) Guided 
> Graphical
> Installation"), right before the actual installation on target 
> (guix
> system init...) you can edit the operating-system configuration 
> file:
> isn't it something similar to what you are looking for?
>
> Please consider that a preseed file is very limited compared to 
> a
> full-fledged operating-system declaration since the latter 
> contains the
> declaration for *all* OS configuration, not just the installed 
> packages.
>

I appreciate where you’re coming from, I also like the one-file 
system configuration, but this is inaccurate.  Guix’s 
operating-system doesn’t encompass the full scope of configuration 
necessary to install and run an OS; Debian’s preseed has 
significantly more functionality than just specifying the 
installed packages.  Right now, Debian’s system allows you to do 
things which Guix does not.

Preseed files contain values that get set in debconf, Debian’s 
system-wide configuration mechanism, so they can both configure 
the resulting system as well as the install process itself.  This 
means you can use a preseed file to tell the installer to 
partition disks, set up LUKS-encrypted volumes (and specify one or 
more passwords for them), format those with filesystems, install 
the set of packages you want, and configure them -- though 
debconf’s package configuration is more limited, generally, than 
Guix provides[1].  With Debian, I can create a custom installer 
image with a preseed file, boot it, and without touching a single 
other thing, it’ll install and configure the target machine, and 
reboot into it.  That boot-and-it-just-works experience is what I 
want from Guix.

For things that can’t be declared in operating-system, like disk 
partitioning and filesystem layout, the installer performs those 
tasks imperatively, then generates a system config with those 
device files and/or UUIDs populated, then initializes the system. 
There’s no facility for specifying disk partitioning or *creating* 
filesystems in the system config -- it can only be pointed at ones 
which have been created already.


>> guix system image is maybe closer, but it doesn’t automate 
>> everything that the
>> installer does.
>> But the installer can be used as a Scheme library, at least in 
>> theory.  The way
>> I would approach the problem is by creating a Shepherd service 
>> that runs at boot
>> from the live booted ISO.
>
> I would really Love So Much™ to avoid writing imperative bash 
> scripts
> and just write Scheme code to be able to do a "full automatic" 
> Guix
> System install, using a workflow like this one:
>
> 1. guix system prepare --include preseed.scm disk-layout.scm 
> /mnt
>
> where disk-layout.scm is a declarative gexp used to partition, 
> format
> and mount all needed filesystems
>
> the resulting config.scm would be an operating-system 
> declaration with
> included the contents of preseed.scm (packages and services
> declarations)
>
> 2. guix system init config.scm /mnt (already working now)
>
> ...unfortunately I'm (still?!?) not able to contribute such code 
> :-(
>

I don’t think there’s any need for a preseed.scm file, and I’m not 
sure what would be in that, but I think this is close to the right 
track.  Either operating-system should be extended to support 
things like disk partitioning, and effect those changes at 
reconfigure time (with suitable safeguards to avoid wrecking 
existing installs), or the operating-system config could get 
embedded in another struct which contains that, similar to the 
(image ...) config for `guix system image'.  I think there are 
some interesting possibilities here: you could change your 
partition layout and have Guix resize them / create new ones for 
you.

  — Ian

[1]: A workaround for this is to create packages which configure 
the system how you want, then include them on the installer image 
/ list them in the packages to be installed.  Not ideal, but you 
can.

declarative partition and format with Guix (was Re: Guix System automated installation)

[Giovanni Biscuolo]

[-- Attachment #1: Type: text/plain, Size: 4585 bytes --]

Hello Ian,

Ian Eure <ian@retrospec.tv> writes:

> Giovanni Biscuolo <g@xelera.eu> writes:

[...]

>> Please consider that a preseed file is very limited compared to 
>> a
>> full-fledged operating-system declaration since the latter 
>> contains the
>> declaration for *all* OS configuration, not just the installed 
>> packages.
>
> I appreciate where you’re coming from, I also like the one-file 
> system configuration, but this is inaccurate.

Yes you are right, I completely misrepresented the functionality of the
Debian preseed utility, sorry! (...and I used that in a remote past)

[...]

> installed packages.  Right now, Debian’s system allows you to do 
> things which Guix does not.

[...]

> means you can use a preseed file to tell the installer to 
> partition disks, set up LUKS-encrypted volumes (and specify one or 
> more passwords for them), format those with filesystems

Yes, this is what is missing from the Guix installer system

> With Debian, I can create a custom installer image with a preseed
> file, boot it, and without touching a single other thing, it’ll
> install and configure the target machine, and reboot into it.  That
> boot-and-it-just-works experience is what I want from Guix.

I understand that it's just a workaround but you can achieve this
boot-and-it-just-works (if there isn't bugs in the script/preseed)
experience with a simple bash script to automate "manual installation"

I wrote it in bash because I'm not able to write it in Guile and/or
extend the "guix system" command to be able to manage the missing bits,
but that is a solution (more a workaround now)

[...]

> There’s no facility for specifying disk partitioning or *creating* 
> filesystems in the system config -- it can only be pointed at ones 
> which have been created already.

Yes: those facilities are missing, we (still?) cannot do that
declaratively... let's do that imperatively, automatically :-)

[...]

>> I would really Love So Much™ to avoid writing imperative bash 
>> scripts
>> and just write Scheme code to be able to do a "full automatic" 
>> Guix
>> System install, using a workflow like this one:
>>
>> 1. guix system prepare --include preseed.scm disk-layout.scm 
>> /mnt
>>
>> where disk-layout.scm is a declarative gexp used to partition, 
>> format
>> and mount all needed filesystems
>>
>> the resulting config.scm would be an operating-system 
>> declaration with
>> included the contents of preseed.scm (packages and services
>> declarations)
>>
>> 2. guix system init config.scm /mnt (already working now)
>>
>> ...unfortunately I'm (still?!?) not able to contribute such code 
>> :-(
>>
>
> I don’t think there’s any need for a preseed.scm file, and I’m not 
> sure what would be in that,

preseed.scm is "just" the part of "operating-system" declaration without
the (bootloader [...]), (file-systems [...]) and (swap-devices [...])
declaration, that is automatically generated by "guix system prepare"
based on disk-layout.scm

> but I think this is close to the right track.  Either operating-system
> should be extended to support things like disk partitioning, and
> effect those changes at reconfigure time (with suitable safeguards to
> avoid wrecking existing installs),

I would prefer not, such "reconfigurations" should be done "out of band"
and not "in band", IMHO

> or the operating-system config could get 
> embedded in another struct which contains that, similar to the 
> (image ...) config for `guix system image'.  I think there are 
> some interesting possibilities here: you could change your 
> partition layout and have Guix resize them

Root (/) partition resizing must be done with root unmounted, no?

Also, since the resize (shrink?) of filesystem is a very sensitive
operation, I'd exclude that from "normal" operations done via "guix
system reconfigure"... it's more "guix system prepare..." with one or
more disk partitions (i.e. /home) resized/shrinked or kept as is,
_without_ file system (re)formatting.

One interesting thing that could be done at "guix system prepare" time
is to restore one or more filesystem content from a (possibly remote)
backup, useful in a disaster recovery scenario.

> / create new ones for you.

[...]

Meanwhile: WDYT to work togheter on a simple _configurable_ bash script
to help users automate the very first installation of a Guix System and
try to upstream it?

Thanks! Gio'

-- 
Giovanni Biscuolo

Xelera IT Infrastructures

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 849 bytes --]

Re: declarative partition and format with Guix (was Re: Guix System automated installation)

[Giovanni Biscuolo]

[-- Attachment #1: Type: text/plain, Size: 1209 bytes --]

Giovanni Biscuolo <g@xelera.eu> writes:

[...]

>> but I think this is close to the right track.  Either operating-system
>> should be extended to support things like disk partitioning,

the library for doing this with Guile is guile-parted (packaged in
Guix); it's used by the Guix Installer:

https://git.savannah.gnu.org/cgit/guix.git/tree/gnu/installer/parted.scm

AFAIU this (parted.scm above) is the starting point (the Guix library)
that can be used to develop a program that automates the disk
partitioning and filesystem creation based on a gexp (disk-layout.scm ?)
declaration.

>> and effect those changes at reconfigure time (with suitable
>> safeguards to avoid wrecking existing installs),
>
> I would prefer not, such "reconfigurations" should be done "out of band"
> and not "in band", IMHO

Side note: there is a recent discussion on a "Resize Filesystem Service"
at this thread
id:ZR0P278MB0268910B4FE39A48112CE740C186A@ZR0P278MB0268.CHEP278.PROD.OUTLOOK.COM
[1]

[...]

Happy hacking! Gio'



[1] https://yhetil.org/guix/ZR0P278MB0268910B4FE39A48112CE740C186A@ZR0P278MB0268.CHEP278.PROD.OUTLOOK.COM/

-- 
Giovanni Biscuolo

Xelera IT Infrastructures

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 849 bytes --]