Seeking guidance regarding system roll-back and switch-generation

Seeking guidance regarding system roll-back and switch-generation

[Chris Marusich]

[-- Attachment #1: Type: text/plain, Size: 5509 bytes --]

Hi Ludo,

I have questions about the design of the mechanism by which GuixSD
switches from one operating system configuration to another.  I'm
emailing you directly because you have the most commits in this area of
the project, so I think you might be in the best position to answer.

I've noticed that the GuixSD mechanism is different from the NixOS
mechanism.  In particular, NixOS uses an "install-grub" script (which is
specific to each system generation) to install grub, but GuixSD does
not.  Is this difference intentional?

Below, I summarize the differences between the NixOS mechanism and the
GuixSD mechanism, and I describe two possible solutions for augmenting
the GuixSD mechanism to support system roll-back and switch-generation.
What solution do you think is best?

COMPARISON OF NIXOS AND GUIXSD MECHANISMS

In NixOS, to upgrade the system, one invokes 'nixos-rebuild switch'.
This performs the following steps in order: (1) build the system
configuration defined in /etc/nixos/configuration.nix, (2) flip the
system profile symlink, and (3) invoke the newly built system's
switch-to-configuration script, which (4) invokes the newly built
system's install-grub.sh script (which is a wrapper around the system's
install-grub.pl script), and (5) activates the new system.

The switch-to-configuration, install-grub.sh, and install-grub.pl
scripts are scripts that have been installed within each system
generation.  Whenever a new system configuration is built, these three
scripts are generated and installed within that system generation (e.g.,
at /nix/var/nix/profiles/system-5-link/bin/switch-to-configuration).
Because nix builds them at the same time it builds the system, nix knows
precisely which paths in the store these scripts need to do their jobs
(grub, kernel, initrd, etc.), and nix hard-codes these paths into the
generated scripts.  Thus, in general these scripts are unique for each
system generation, and each script knows precisely (and only) how to do
its job for its particular system generation.

In GuixSD, to upgrade the system, one invokes 'guix system reconfigure
config.scm'.  This performs the following steps in order: (1) build the
system configuration defined in config.scm, (2) flip the system profile
symlink, (3) run the new system's activation script, (4) install grub.

The GuixSD mechanism differs from the NixOS mechanism in a few ways.
The biggest difference is that GuixSD does not use a
"switch-to-configuration" script (although GuixSD does have a system
activation script, which activates the system but does not install the
bootloader).  In NixOS, all activities involving a system configuration
switch - upgrade the system, roll back the system, switch the system to
an arbitrary, existing generation - use this script to install grub
and/or activate services.  Because the scripts are generated at build
time and hard-coded with the paths to things like grub, the
'nixos-rebuild' command does not need to concern itself with finding all
the right things; to install the right grub and activate the right
services for a particular system generation, the 'nixos-rebuild' command
just needs to invoke the switch-to-generation script for that
generation.  This means that to perform rollback, the 'nixos-rebuild'
command does not need to know what the original operating system
configuration file was.

SOLUTION 1: USE A SWITCH-TO-CONFIGURATION SCRIPT

The current mechanism for installing grub and activating services in
GuixSD requires the presence of an operating system configuration file.
This makes it difficult to roll back or switch configurations, since we
do not currently store the operating system configuration files for
previous system configurations.  One way to solve this problem is to
follow the NixOS example and generate a similar
"switch-to-configuration" script at system build time.  Perhaps it could
be a gexp or something.  This would be a non-trivial change to the "guix
system" logic, but once we make that change, the implementation of
system roll-back and switch-generation would probably be pretty
straightforward.  This would also make GuixSD's design more similar to
the NixOS design, which might encourage cross-pollination between these
two similar projects.

SOLUTION 2: STORE THE OPERATING SYSTEM CONFIGURATION FILE

However, it may not be necessary to generate a "switch-to-configuration"
script at all.  Instead, what if we just stored the operating system
configuration file in a well known location within each system
generation?  This might enable us to re-use the existing "guix system"
code to implement roll-back and switch-generation.  This might also have
other nice benefits.  For example, it would leave a record of what the
previous system configurations were, and we could add a command like
"guix system diff" that could diff the operating system configuration
files for any two generations.  This might also eliminate the need to
store an activation script.  However, it could introduce a wrinkle: if
GuixSD does not understand how to read a particular system generation's
operating system configuration file (e.g., because the generation was
made by a version of GuixSD from the far past or far future which uses
an incompatible operating system file structure), then it would not be
able to switch to that generation.

Please let me know which solution you think is best.

Thank you,
-- 
Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]

Re: Seeking guidance regarding system roll-back and switch-generation

[Tobias Geerinckx-Rice]

Ahoy Chris,

Just one aside, for context:

(And to softly nudge people into the right™ direction of activation
scripts, of course. ;-)

On 2016-07-17 11:22, Chris Marusich wrote:
> However, it may not be necessary to generate a 
> "switch-to-configuration"
> script at all.  Instead, what if we just stored the operating system
> configuration file in a well known location within each system
> generation?

NixOS offers this option. I've forgot the name, because its utility is
so limited, and its main use is to confuse people on IRC.

All it does is copy /etc/nixos/configuration.nix to the store. That's
it. All your imports are now broken, if they pointed to a relative path
that doesn't exist in the store, or impure, if they pointed to an 
absolute
path that wasn't immutable. This breaks all but the most trivial (or
unmaintainable) set-ups. Even the NixOS installer creates a multi-file
configuration by default.

That no-one has bothered to script a solution probably means something.

Kind regards,

T G-R

Re: Seeking guidance regarding system roll-back and switch-generation

[Ludovic Courtès]

Hello!

Tobias Geerinckx-Rice <me@tobias.gr> skribis:

> On 2016-07-17 11:22, Chris Marusich wrote:
>> However, it may not be necessary to generate a
>> "switch-to-configuration"
>> script at all.  Instead, what if we just stored the operating system
>> configuration file in a well known location within each system
>> generation?
>
> NixOS offers this option. I've forgot the name, because its utility is
> so limited, and its main use is to confuse people on IRC.
>
> All it does is copy /etc/nixos/configuration.nix to the store. That's
> it. All your imports are now broken, if they pointed to a relative path
> that doesn't exist in the store, or impure, if they pointed to an
> absolute
> path that wasn't immutable. This breaks all but the most trivial (or
> unmaintainable) set-ups. Even the NixOS installer creates a multi-file
> configuration by default.

I was about to make the same comment.  To put it differently, the
configuration.{scm,nix} file is not self-contained; its semantics are
determined by the rest of Guix/Nix{pkgs,OS}.  As such, it’s not very
helpful to keep it alone.

Thanks for your feedback,
Ludo’.

Re: Seeking guidance regarding system roll-back and switch-generation

[Ludovic Courtès]

Hi Chris,

Chris Marusich <cmmarusich@gmail.com> skribis:

> I've noticed that the GuixSD mechanism is different from the NixOS
> mechanism.  In particular, NixOS uses an "install-grub" script (which is
> specific to each system generation) to install grub, but GuixSD does
> not.  Is this difference intentional?

Looking at
<https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/system/boot/loader/grub/install-grub.pl>,
part of it seems to be concerned with the generation of grub.cfg, which
is what (gnu system grub) does.

It also does a couple more things, such as providing proper EFI support,
and avoiding reinstalling GRUB when possible (whereas ‘guix system
reconfigure’ currently reruns ‘grub-install’ each time, even when it’s
not strictly needed.)

So I don’t think it’s very different, after all.  Or am I missing
something?

> COMPARISON OF NIXOS AND GUIXSD MECHANISMS

[...]

> The GuixSD mechanism differs from the NixOS mechanism in a few ways.
> The biggest difference is that GuixSD does not use a
> "switch-to-configuration" script (although GuixSD does have a system
> activation script, which activates the system but does not install the
> bootloader).  In NixOS, all activities involving a system configuration
> switch - upgrade the system, roll back the system, switch the system to
> an arbitrary, existing generation - use this script to install grub
> and/or activate services.  Because the scripts are generated at build
> time and hard-coded with the paths to things like grub, the
> 'nixos-rebuild' command does not need to concern itself with finding all
> the right things; to install the right grub and activate the right
> services for a particular system generation, the 'nixos-rebuild' command
> just needs to invoke the switch-to-generation script for that
> generation.  This means that to perform rollback, the 'nixos-rebuild'
> command does not need to know what the original operating system
> configuration file was.

Interesting, I forgot (or ignored!) these details about NixOS.  :-)

> SOLUTION 1: USE A SWITCH-TO-CONFIGURATION SCRIPT
>
> The current mechanism for installing grub and activating services in
> GuixSD requires the presence of an operating system configuration file.
> This makes it difficult to roll back or switch configurations, since we
> do not currently store the operating system configuration files for
> previous system configurations.  One way to solve this problem is to
> follow the NixOS example and generate a similar
> "switch-to-configuration" script at system build time.  Perhaps it could
> be a gexp or something.

Switching to a generations primarily means: (1) running the target’s
activation script, (2) updating Shepherd services, and (3) updating
grub.cfg.

Of these (1) and (3) are currently easy to do on GuixSD.  (Right? :-))

(2) is more difficult.  It’s already difficult when switching to a *new*
generation because we have to arrange to change the state of the
currently-running PID 1 to get closer to its target state.

It’s even more difficult when rolling back to a previous generation
because, as we discussed, we currently don’t have any representation of
the previous generation’s list of Shepherd services.

When we discussed it previously, I said that we could add a purely
declarative representation of Shepherd services in the output of ‘guix
system build’ (just like we currently have a ‘parameters’ file.)  A
‘switch-to-configuration’ script would essentially be an executable
variant of that representation.

However, I think I prefer the declarative approach (sexps to describe
services) over the procedural approach (a ‘switch-to-configuration’
script), because it leaves more flexibility to the ‘guix system’ command
and to the user, and also decouples things a bit more.

Does that make sense?  WDYT?

> SOLUTION 2: STORE THE OPERATING SYSTEM CONFIGURATION FILE

Won’t work, as Tobias notes.  :-)

Thank you for the detailed analysis!

Ludo’.

Re: Seeking guidance regarding system roll-back and switch-generation

[Chris Marusich]

[-- Attachment #1: Type: text/plain, Size: 5637 bytes --]

Hi Ludo,

Thank you very much for taking the time to respond!  Now I have a better
idea of how to proceed.

ludo@gnu.org (Ludovic Courtès) writes:

> Chris Marusich <cmmarusich@gmail.com> skribis:
>
>> I've noticed that the GuixSD mechanism is different from the NixOS
>> mechanism.  In particular, NixOS uses an "install-grub" script (which is
>> specific to each system generation) to install grub, but GuixSD does
>> not.  Is this difference intentional?
>
> Looking at
> <https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/system/boot/loader/grub/install-grub.pl>,
> part of it seems to be concerned with the generation of grub.cfg, which
> is what (gnu system grub) does.
>
> It also does a couple more things, such as providing proper EFI support,
> and avoiding reinstalling GRUB when possible (whereas ‘guix system
> reconfigure’ currently reruns ‘grub-install’ each time, even when it’s
> not strictly needed.)
>
> So I don’t think it’s very different, after all.  Or am I missing
> something?

Yes, they both install grub, but the mechanism seems different.  NixOS
generates a script for each system generation, which installs exactly
the right grub for that generation.  Unless I'm mistaken, GuixSD does
not do that.  Instead, the existing GuixSD mechanism seems to require an
operating system configuration file.  GuixSD uses that to determine the
correct grub, which it then installs.

>> SOLUTION 1: USE A SWITCH-TO-CONFIGURATION SCRIPT
>>
>> The current mechanism for installing grub and activating services in
>> GuixSD requires the presence of an operating system configuration file.
>> This makes it difficult to roll back or switch configurations, since we
>> do not currently store the operating system configuration files for
>> previous system configurations.  One way to solve this problem is to
>> follow the NixOS example and generate a similar
>> "switch-to-configuration" script at system build time.  Perhaps it could
>> be a gexp or something.
>
> Switching to a generations primarily means: (1) running the target’s
> activation script, (2) updating Shepherd services, and (3) updating
> grub.cfg.

In addition to (3), don't we also need to install the grub software
itself?  For Libreboot (or coreboot), I believe that the "payload grub"
[1] is installed in flash ROM via a mechanism that is outside the scope
of GuixSD.  However, for other systems, it's still necessary to install
the bootloader to some drive, e.g., /dev/sda, right?

[1] https://www.coreboot.org/Payloads

> Of these (1) and (3) are currently easy to do on GuixSD.  (Right? :-))

For both (1) and (3), the current GuixSD mechanism appears to require an
operating system configuration file.  Since roll-back and
switch-generation will not have such a file available, it seems the
existing mechanism won't work without some changes.

> (2) is more difficult.  It’s already difficult when switching to a *new*
> generation because we have to arrange to change the state of the
> currently-running PID 1 to get closer to its target state.
>
> It’s even more difficult when rolling back to a previous generation
> because, as we discussed, we currently don’t have any representation of
> the previous generation’s list of Shepherd services.
>
> When we discussed it previously, I said that we could add a purely
> declarative representation of Shepherd services in the output of ‘guix
> system build’ (just like we currently have a ‘parameters’ file.)  A
> ‘switch-to-configuration’ script would essentially be an executable
> variant of that representation.
>
> However, I think I prefer the declarative approach (sexps to describe
> services) over the procedural approach (a ‘switch-to-configuration’
> script), because it leaves more flexibility to the ‘guix system’ command
> and to the user, and also decouples things a bit more.
>
> Does that make sense?  WDYT?

Yes, I think that makes sense.  I need to learn more about GuixSD's
activation mechanism, but it makes sense that a procedural definition
(e.g., like in NixOS's switch-to-configuration script) is equivalent to
a declarative definition.  I think a declarative definition of services
would be fine.  Concretely speaking, is there a specific format that is
convenient to use for such a "declarative definition of services"?

What about the grub installation?  As mentioned earlier, NixOS
hard-codes stuff like the path to the kernel image and initrd into the
install-grub.pl script.  Are you suggesting that I store any necessary
information for installing grub in some kind of a declarative definition
in the output of "guix system build", similar to how we'd like to do it
for activation?  Or are you suggesting that I simply parse the output of
"guix system build" for well-known paths (e.g.,
/var/guix/profiles/system-3-link/initrd)?

Specifically, to complete the first milestone (flip symlinks and
regenerate grub.cfg), I need to determine the paths to certain things
like the kernel and the initrd for an arbitrary system generation.  So,
I'm not sure if I should simply parse that information from the file
system, or if I should try to use the store to obtain that information.
What are your thoughts?

>> SOLUTION 2: STORE THE OPERATING SYSTEM CONFIGURATION FILE
>
> Won’t work, as Tobias notes.  :-)
>
> Thank you for the detailed analysis!

OK, I won't do this.  Thank you for clarifying why it is a bad idea!
It's always good to save time not implementing stuff we don't need.

-- 
Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]

Re: Seeking guidance regarding system roll-back and switch-generation

[Chris Marusich]

[-- Attachment #1: Type: text/plain, Size: 702 bytes --]

Chris Marusich <cmmarusich@gmail.com> writes:

> Specifically, to complete the first milestone (flip symlinks and
> regenerate grub.cfg), I need to determine the paths to certain things
> like the kernel and the initrd for an arbitrary system generation.  So,
> I'm not sure if I should simply parse that information from the file
> system, or if I should try to use the store to obtain that information.
> What are your thoughts?

I did not realize that we already store information like this in a
declarative form in the "parameters" file (e.g.,
/var/guix/profiles/system/parameters).  Now that I know that such a
thing exists, I will look into it to see how I can use it.

-- 
Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]

Re: Seeking guidance regarding system roll-back and switch-generation

[Ludovic Courtès]

[-- Attachment #1: Type: text/plain, Size: 2534 bytes --]

Hi!

Chris Marusich <cmmarusich@gmail.com> skribis:

> Thank you very much for taking the time to respond!  Now I have a better
> idea of how to proceed.

Sorry for the delay!  That’s what you get for asking difficult
questions.  ;-)

> ludo@gnu.org (Ludovic Courtès) writes:
>
>> Chris Marusich <cmmarusich@gmail.com> skribis:
>>
>>> I've noticed that the GuixSD mechanism is different from the NixOS
>>> mechanism.  In particular, NixOS uses an "install-grub" script (which is
>>> specific to each system generation) to install grub, but GuixSD does
>>> not.  Is this difference intentional?
>>
>> Looking at
>> <https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/system/boot/loader/grub/install-grub.pl>,
>> part of it seems to be concerned with the generation of grub.cfg, which
>> is what (gnu system grub) does.
>>
>> It also does a couple more things, such as providing proper EFI support,
>> and avoiding reinstalling GRUB when possible (whereas ‘guix system
>> reconfigure’ currently reruns ‘grub-install’ each time, even when it’s
>> not strictly needed.)
>>
>> So I don’t think it’s very different, after all.  Or am I missing
>> something?
>
> Yes, they both install grub, but the mechanism seems different.  NixOS
> generates a script for each system generation, which installs exactly
> the right grub for that generation.  Unless I'm mistaken, GuixSD does
> not do that.  Instead, the existing GuixSD mechanism seems to require an
> operating system configuration file.  GuixSD uses that to determine the
> correct grub, which it then installs.

Sure, but it seems to be equivalent in the end.

>> Switching to a generations primarily means: (1) running the target’s
>> activation script, (2) updating Shepherd services, and (3) updating
>> grub.cfg.
>
> In addition to (3), don't we also need to install the grub software
> itself?

In general yes (there are cases where this is not needed, but let’s
ignore them.)

>> Of these (1) and (3) are currently easy to do on GuixSD.  (Right? :-))
>
> For both (1) and (3), the current GuixSD mechanism appears to require an
> operating system configuration file.

The output of ‘guix system build’ contains the ‘parameters’ file, which
is enough to generate grub.cfg (see ‘previous-grub-entries’ in (guix
scripts system)).

However, the activation script is indeed missing.  We can add it to the
output of ‘guix system build’ by extending ‘system-service-type’:


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Type: text/x-patch, Size: 951 bytes --]

diff --git a/gnu/services.scm b/gnu/services.scm
index 5479bfa..fc3e17e 100644
--- a/gnu/services.scm
+++ b/gnu/services.scm
@@ -352,11 +352,18 @@ ACTIVATION-SCRIPT-TYPE."
 
 (define (second-argument a b) b)
 
+(define (gexps->activation-system-entry gexps)
+  "Return a directory entry to add to the result of the 'system' derivation."
+  (mlet %store-monad ((script (activation-script gexps)))
+    (return `(("activate" ,script)))))
+
 (define activation-service-type
   (service-type (name 'activate)
                 (extensions
                  (list (service-extension boot-service-type
-                                          gexps->activation-gexp)))
+                                          gexps->activation-gexp)
+                       (service-extension system-service-type
+                                          gexps->activation-system-entry)))
                 (compose append)
                 (extend second-argument)))
 

[-- Attachment #3: Type: text/plain, Size: 795 bytes --]


This way we have direct access to each generation’s activation script
and we should be fine with (3).

WDYT?

> Yes, I think that makes sense.  I need to learn more about GuixSD's
> activation mechanism, but it makes sense that a procedural definition
> (e.g., like in NixOS's switch-to-configuration script) is equivalent to
> a declarative definition.  I think a declarative definition of services
> would be fine.  Concretely speaking, is there a specific format that is
> convenient to use for such a "declarative definition of services"?

I would suggest an sexp as discussed at:

  https://lists.gnu.org/archive/html/guix-devel/2016-06/msg00173.html

> What about the grub installation?

The ‘parameters’ file should be enough, as you wrote.

HTH!

Ludo’.

Re: Seeking guidance regarding system roll-back and switch-generation

[Chris Marusich]

[-- Attachment #1: Type: text/plain, Size: 2346 bytes --]

Hi Ludo,

Thank you for the concrete suggestions.  I've looked into using the
parameters file.  It contains almost all the info I need, but I think
some is still missing.

> The output of ‘guix system build’ contains the ‘parameters’ file, which
> is enough to generate grub.cfg (see ‘previous-grub-entries’ in (guix
> scripts system)).

What if someone specifies extra "menu-entries" in their operating system
configuration file?  (as described in "(guix) GRUB Configuration")?
Those extra entries don't appear to be stored in the parameters file.

It would be very convenient if we could just store the entire
<grub-configuration> in the built system output (e.g., the parameters
file).  Is that possible?

> However, the activation script is indeed missing.  We can add it to the
> output of ‘guix system build’ by extending ‘system-service-type’:
>
>
> diff --git a/gnu/services.scm b/gnu/services.scm
> index 5479bfa..fc3e17e 100644
> --- a/gnu/services.scm
> +++ b/gnu/services.scm
> @@ -352,11 +352,18 @@ ACTIVATION-SCRIPT-TYPE."
>  
>  (define (second-argument a b) b)
>  
> +(define (gexps->activation-system-entry gexps)
> +  "Return a directory entry to add to the result of the 'system' derivation."
> +  (mlet %store-monad ((script (activation-script gexps)))
> +    (return `(("activate" ,script)))))
> +
>  (define activation-service-type
>    (service-type (name 'activate)
>                  (extensions
>                   (list (service-extension boot-service-type
> -                                          gexps->activation-gexp)))
> +                                          gexps->activation-gexp)
> +                       (service-extension system-service-type
> +                                          gexps->activation-system-entry)))
>                  (compose append)
>                  (extend second-argument)))
>  
>
>
> This way we have direct access to each generation’s activation script
> and we should be fine with (3).
>
> WDYT?

I'm afraid I don't yet know enough about gexps and the activation
process to give an informed opinion on that suggestion.  After I finish
the first milestone (switch symlinks and rebuild grub.cfg), I'll study
those topics in more detail and revisit your proposal.

Thank you,

-- 
Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]

Re: Seeking guidance regarding system roll-back and switch-generation

[Ludovic Courtès]

Hi Chris,

Chris Marusich <cmmarusich@gmail.com> skribis:

>> The output of ‘guix system build’ contains the ‘parameters’ file, which
>> is enough to generate grub.cfg (see ‘previous-grub-entries’ in (guix
>> scripts system)).
>
> What if someone specifies extra "menu-entries" in their operating system
> configuration file?  (as described in "(guix) GRUB Configuration")?
> Those extra entries don't appear to be stored in the parameters file.

They’re not stored there, but that’s fine: since there’s only one
grub.cfg, we just store the menu entries that are defined in
‘operating-system’ at the moment ‘guix system reconfigure’ is run.  This
part is “imperative” in nature.

> It would be very convenient if we could just store the entire
> <grub-configuration> in the built system output (e.g., the parameters
> file).  Is that possible?

I don’t think it’s useful because what matters is the
<grub-configuration> that we see in the new operating system.

What do you think?

>> However, the activation script is indeed missing.  We can add it to the
>> output of ‘guix system build’ by extending ‘system-service-type’:

[...]

>> This way we have direct access to each generation’s activation script
>> and we should be fine with (3).
>>
>> WDYT?
>
> I'm afraid I don't yet know enough about gexps and the activation
> process to give an informed opinion on that suggestion.  After I finish
> the first milestone (switch symlinks and rebuild grub.cfg), I'll study
> those topics in more detail and revisit your proposal.

Sure.

Thank you!

Ludo’.