Re: guix system switch-generation doesn't

Re: guix system switch-generation doesn't

[Robert Vollmert]

Could we get some input on this bug?

Maybe I’m misunderstanding something, but it seems that a core guix
feature (atomic rollbacks) doesn’t work…

> On 30. Jul 2019, at 12:00, Robert Vollmert <rob@vllmrt.net> wrote:
> 
> What I see:
> 
> 1. edit ~/pzprnode/pzprnode
> 
> rob@garp ~/pzprnode$ git diff
> diff --git a/pzprnode b/pzprnode
> index 612e6a8..d8ef0ea 100755
> --- a/pzprnode
> +++ b/pzprnode
> @@ -190,5 +190,6 @@ const server = http.createServer((req, res) => {
> });
> 
> server.listen(port, hostname, () => {
> +       console.log("updated version");
>       console.log(`Server running at http://${hostname}:${port}/`);
> });
> 
> 2. sudo guix system reconfigure -L ~/garp-config ~/garp-config/config.scm
> 3. sudo herd restart pzprnode
> 4. less /var/log/messages
> 
> Jul 30 11:46:57 localhost shepherd[1]: Service pzprnode has been stopped. 
> Jul 30 11:46:57 localhost shepherd[1]: Service pzprnode has been started. 
> Jul 30 11:46:58 localhost pzprnode[4954]: updated version 
> Jul 30 11:46:58 localhost pzprnode[4954]: Server running at http://127.0.0.1:3456/ 
> 
> 5. sudo guix system list-generations
> 
> Generation 151	Jul 30 2019 10:37:06
> file name: /var/guix/profiles/system-151-link
> canonical file name: /gnu/store/jis33accsfpa068aps0a9mrycmjzfm4m-system
> label: GNU with Linux-Libre 5.2.1
> bootloader: grub
> root device: label: "guix-root"
> kernel: /gnu/store/fp6wsvn10h1is0wkz8l2sbzjmjbzi7vr-linux-libre-5.2.1/bzImage
> Generation 152	Jul 30 2019 11:43:13	(current)
> file name: /var/guix/profiles/system-152-link
> canonical file name: /gnu/store/3z3wmaj0399kihqc372y91nzcjxc1myl-system
> label: GNU with Linux-Libre 5.2.1
> bootloader: grub
> root device: label: "guix-root"
> kernel: /gnu/store/fp6wsvn10h1is0wkz8l2sbzjmjbzi7vr-linux-libre-5.2.1/bzImage
> 
> 6. sudo guix system switch-generation 151
> 
> substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
> The following derivation will be built:
>  /gnu/store/qvxbl3gm6406dbbkm8jigmpc8zi42lfw-grub.cfg.drv
> building /gnu/store/qvxbl3gm6406dbbkm8jigmpc8zi42lfw-grub.cfg.drv...
> switched from generation 152 to 151
> 
> 7. sudo herd restart pzprnode
> 8. less /var/log/messages
> 
> Jul 30 11:48:02 localhost shepherd[1]: Service pzprnode has been stopped. 
> Jul 30 11:48:02 localhost shepherd[1]: Service pzprnode has been started. 
> Jul 30 11:48:03 localhost pzprnode[4994]: updated version 
> Jul 30 11:48:03 localhost pzprnode[4994]: Server running at http://127.0.0.1:3456/ 
> 
> The line with “updated version” should not be there.
> 
> Presumably, this is due to switch-generations not calling upgrade-shepherd-services.
> 

bug#36855: guix system switch-generation doesn't

[Christopher Lemmer Webber]

Could you look at bug #36878 and commit 1db6f137d... as of latest
master, is this fixed?

Robert Vollmert writes:

> Could we get some input on this bug?
>
> Maybe I’m misunderstanding something, but it seems that a core guix
> feature (atomic rollbacks) doesn’t work…
>
>> On 30. Jul 2019, at 12:00, Robert Vollmert <rob@vllmrt.net> wrote:
>> 
>> What I see:
>> 
>> 1. edit ~/pzprnode/pzprnode
>> 
>> rob@garp ~/pzprnode$ git diff
>> diff --git a/pzprnode b/pzprnode
>> index 612e6a8..d8ef0ea 100755
>> --- a/pzprnode
>> +++ b/pzprnode
>> @@ -190,5 +190,6 @@ const server = http.createServer((req, res) => {
>> });
>> 
>> server.listen(port, hostname, () => {
>> +       console.log("updated version");
>>       console.log(`Server running at http://${hostname}:${port}/`);
>> });
>> 
>> 2. sudo guix system reconfigure -L ~/garp-config ~/garp-config/config.scm
>> 3. sudo herd restart pzprnode
>> 4. less /var/log/messages
>> 
>> Jul 30 11:46:57 localhost shepherd[1]: Service pzprnode has been stopped. 
>> Jul 30 11:46:57 localhost shepherd[1]: Service pzprnode has been started. 
>> Jul 30 11:46:58 localhost pzprnode[4954]: updated version 
>> Jul 30 11:46:58 localhost pzprnode[4954]: Server running at http://127.0.0.1:3456/ 
>> 
>> 5. sudo guix system list-generations
>> 
>> Generation 151	Jul 30 2019 10:37:06
>> file name: /var/guix/profiles/system-151-link
>> canonical file name: /gnu/store/jis33accsfpa068aps0a9mrycmjzfm4m-system
>> label: GNU with Linux-Libre 5.2.1
>> bootloader: grub
>> root device: label: "guix-root"
>> kernel: /gnu/store/fp6wsvn10h1is0wkz8l2sbzjmjbzi7vr-linux-libre-5.2.1/bzImage
>> Generation 152	Jul 30 2019 11:43:13	(current)
>> file name: /var/guix/profiles/system-152-link
>> canonical file name: /gnu/store/3z3wmaj0399kihqc372y91nzcjxc1myl-system
>> label: GNU with Linux-Libre 5.2.1
>> bootloader: grub
>> root device: label: "guix-root"
>> kernel: /gnu/store/fp6wsvn10h1is0wkz8l2sbzjmjbzi7vr-linux-libre-5.2.1/bzImage
>> 
>> 6. sudo guix system switch-generation 151
>> 
>> substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
>> The following derivation will be built:
>>  /gnu/store/qvxbl3gm6406dbbkm8jigmpc8zi42lfw-grub.cfg.drv
>> building /gnu/store/qvxbl3gm6406dbbkm8jigmpc8zi42lfw-grub.cfg.drv...
>> switched from generation 152 to 151
>> 
>> 7. sudo herd restart pzprnode
>> 8. less /var/log/messages
>> 
>> Jul 30 11:48:02 localhost shepherd[1]: Service pzprnode has been stopped. 
>> Jul 30 11:48:02 localhost shepherd[1]: Service pzprnode has been started. 
>> Jul 30 11:48:03 localhost pzprnode[4994]: updated version 
>> Jul 30 11:48:03 localhost pzprnode[4994]: Server running at http://127.0.0.1:3456/ 
>> 
>> The line with “updated version” should not be there.
>> 
>> Presumably, this is due to switch-generations not calling upgrade-shepherd-services.
>> 

Re: bug#36855: guix system switch-generation doesn't

[Jakob L. Kreuze]

[-- Attachment #1: Type: text/plain, Size: 453 bytes --]

Hi Chris,

Christopher Lemmer Webber <cwebber@dustycloud.org> writes:

> Could you look at bug #36878 and commit 1db6f137d... as of latest
> master, is this fixed?

Unfortunately, I don't think that 1db6f137d fixes this. The issue is a
bit more structural as 'switch-to-system-generation' doesn't call out to
'upgrade-shepherd-services'. I'm not sure if this was an intentional
decision or not (perhaps we can ask Ludo when he returns).

Regards,
Jakob

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: bug#36855: guix system switch-generation doesn't

[Chris Marusich]

[-- Attachment #1: Type: text/plain, Size: 3048 bytes --]

Hi Jakob,

zerodaysfordays@sdf.lonestar.org (Jakob L. Kreuze) writes:

> 'switch-to-system-generation' doesn't call out to
> 'upgrade-shepherd-services'. I'm not sure if this was an intentional
> decision or not

It is intentional, but only because there is currently no way to call
upgrade-shepherd-services when switching system generations.

Consider the procedure upgrade-shepherd-services: you must pass it an
<operating-system> record.  When you are flipping from one generation to
another, how do you get the <operating-system> record that was used for
the generation you're switching to?  Guix doesn't currently store the
operating system configuration file or its <operating-system> record
anywhere, so we can't call upgrade-shepherd-services.

This was discussed in 2016 and we agreed we need to persist some
information to enable us to handle Shepherd services correctly.  This is
what Ludo suggested at the time:

https://lists.gnu.org/archive/html/guix-devel/2016-06/msg00173.html

"Maybe we could store in the system output (result of ‘guix system
build’) an sexp representation of (part of) our <shepherd-service>
records:

  (shepherd-service
    (provisions (x y z))
    (requirements (a b c))
    (start-script "/gnu/store/…-start-foo.scm")
    (stop-script "/gnu/store/…-stop-foo.scm")
    …)

Then ‘upgrade-shepherd-services’ could start from this simplified
representation instead of using the full-blown <shepherd-service>
objects, and thus could work both when instantiating a new generation
and when rolling back."

Until that happens, you'll always have to reboot to complete the
switch.

FYI, last I checked (about 3 years ago), in NixOS they took a slightly
different approach: instead of storing state describing the previous
system generation and relying on the current system's logic to correctly
parse it and use it to revert the system to a prior configuration, they
just dump everything into a self-contained script that knows how to
update the entire system to one specific configuration.  That approach
is nice in some ways because switching generations is dead simple - you
just run the switching script belonging to the generation you want to
switch to - but it also has downsides.

For example, if the target generation is old enough compared to the
current system, then the target generation's old switching script might
not understand how to deal with the current system correctly.  Instead,
if you only store the bare minimum of state required to take the right
actions, and you implement the meat of the logic in the current Guix
installation, you are more likely to be able to switch generations even
to very old ones where the world was very different, since the current
Guix can be taught how to deal gracefully with the old world.  But it
seems more complicated.  It's all about trade-offs.

That said, I've never gone back to implement this.  If you want to give
it a try, you're more than welcome to do so!

-- 
Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: bug#36855: guix system switch-generation doesn't

[Robert Vollmert]

On 8. Aug 2019, at 18:40, Chris Marusich <cmmarusich@gmail.com> wrote:
> zerodaysfordays@sdf.lonestar.org (Jakob L. Kreuze) writes:
> 
>> 'switch-to-system-generation' doesn't call out to
>> 'upgrade-shepherd-services'. I'm not sure if this was an intentional
>> decision or not
> 
> It is intentional, but only because there is currently no way to call
> upgrade-shepherd-services when switching system generations.

How does shepherd work on a non-guix system? Can’t be it be configured
like other daemons to read its configuration from a file, e.g. from

   /run/current-system/etc/shepherd.conf

and be told via signal to reload its configuration from disk?

…

(I feel a bit cheated right now. This behaviour makes Guix System entirely
unsuitable for server use. It shouldn’t be advertised as supporting
transactional upgrades and rollbacks if those require a reboot.)

Re: bug#36855: guix system switch-generation doesn't

[Chris Marusich]

[-- Attachment #1: Type: text/plain, Size: 2055 bytes --]

Hi Robert,

Robert Vollmert <rob@vllmrt.net> writes:

> On 8. Aug 2019, at 18:40, Chris Marusich <cmmarusich@gmail.com> wrote:
>> zerodaysfordays@sdf.lonestar.org (Jakob L. Kreuze) writes:
>> 
>>> 'switch-to-system-generation' doesn't call out to
>>> 'upgrade-shepherd-services'. I'm not sure if this was an intentional
>>> decision or not
>> 
>> It is intentional, but only because there is currently no way to call
>> upgrade-shepherd-services when switching system generations.
>
> How does shepherd work on a non-guix system? Can’t be it be configured
> like other daemons to read its configuration from a file, e.g. from
>
>    /run/current-system/etc/shepherd.conf
>
> and be told via signal to reload its configuration from disk?

Maybe!  In the email thread I linked, Ludo talked about storing a
description of the Shepherd services in the system generation for future
reference.  Maybe we could store it in a place like this, and maybe
Shepherd already has mechanisms for reloading configurations like this.
I don't intend to work on this because I need to focus on other things
right now, but I would be happy if someone took up this work!

> (I feel a bit cheated right now. This behaviour makes Guix System entirely
> unsuitable for server use. It shouldn’t be advertised as supporting
> transactional upgrades and rollbacks if those require a reboot.)

I agree that Guix should update as many Shepherd services as it can when
switching generations.  However, I don't think it's inaccurate to say
that Guix supports transactional upgrades and rollbacks.  When you
invoke "guix system switch-generation", the system profile symlink is
flipped atomically, so you get an atomic update from one version of the
system to another.  Software running in the system never sees an
inconsistent view of the system.  Contrast this with nearly any other
mutable GNU/Linux system, in which files are more or less sprayed into
the existing file system with no guarantee of consistency or atomicity.

-- 
Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: bug#36855: guix system switch-generation doesn't

[Ludovic Courtès]

Hey Chris & Jakob,

Chris Marusich <cmmarusich@gmail.com> skribis:

> zerodaysfordays@sdf.lonestar.org (Jakob L. Kreuze) writes:
>
>> 'switch-to-system-generation' doesn't call out to
>> 'upgrade-shepherd-services'. I'm not sure if this was an intentional
>> decision or not
>
> It is intentional, but only because there is currently no way to call
> upgrade-shepherd-services when switching system generations.

[...]

> FYI, last I checked (about 3 years ago), in NixOS they took a slightly
> different approach: instead of storing state describing the previous
> system generation and relying on the current system's logic to correctly
> parse it and use it to revert the system to a prior configuration, they
> just dump everything into a self-contained script that knows how to
> update the entire system to one specific configuration.  That approach
> is nice in some ways because switching generations is dead simple - you
> just run the switching script belonging to the generation you want to
> switch to - but it also has downsides.

Jakob, now that we generate scripts for the effectful bits of system
reconfiguration (one of these bits being service upgrades), couldn’t we
take it one step further and store those scripts in the “system”
derivation so we can run them eventually, notably upon
‘switch-generation’?

> For example, if the target generation is old enough compared to the
> current system, then the target generation's old switching script might
> not understand how to deal with the current system correctly.  Instead,
> if you only store the bare minimum of state required to take the right
> actions, and you implement the meat of the logic in the current Guix
> installation, you are more likely to be able to switch generations even
> to very old ones where the world was very different, since the current
> Guix can be taught how to deal gracefully with the old world.  But it
> seems more complicated.  It's all about trade-offs.

Indeed.  The important thing to me is that from the GRUB menu you can
really switch to any generation.  I’ve actually never used
‘switch-generations’ on my laptop, but technically, I feel like storing
the “switch-to-system” script would be the easiest way.

Thanks,
Ludo’.

Re: bug#36855: guix system switch-generation doesn't

[Mark H Weaver]

Hi,

Ludovic Courtès <ludo@gnu.org> writes:

> Jakob, now that we generate scripts for the effectful bits of system
> reconfiguration (one of these bits being service upgrades), couldn’t we
> take it one step further and store those scripts in the “system”
> derivation so we can run them eventually, notably upon
> ‘switch-generation’?

As a bonus, this approach might solve another issue I've observed: on my
Guix system, where I build everything locally, several derivations are
built *during* activation.  Based on the terminal output, I get the
impression that the system is compiling things while the system in an
intermediate state, when some of the activation steps have been done,
but not all of them.

As I recall, the derivations built during activation are limited to
compiled modules for Guile, but it still sometimes takes on the order of
a minute or two on my laptop to complete the "activating system" steps.
This seems suboptimal.

The next time I update my system, I'll try to remember to keep a
transcript of this, so that I can be more specific.

      Best,
       Mark

bug#36855: guix system switch-generation doesn't

[Mark H Weaver]

Hello again,

Mark H Weaver <mhw@netris.org> writes:

> Ludovic Courtès <ludo@gnu.org> writes:
>
>> Jakob, now that we generate scripts for the effectful bits of system
>> reconfiguration (one of these bits being service upgrades), couldn’t we
>> take it one step further and store those scripts in the “system”
>> derivation so we can run them eventually, notably upon
>> ‘switch-generation’?
>
> As a bonus, this approach might solve another issue I've observed: on my
> Guix system, where I build everything locally, several derivations are
> built *during* activation.  Based on the terminal output, I get the
> impression that the system is compiling things while the system in an
> intermediate state, when some of the activation steps have been done,
> but not all of them.
>
> As I recall, the derivations built during activation are limited to
> compiled modules for Guile, but it still sometimes takes on the order of
> a minute or two on my laptop to complete the "activating system" steps.
> This seems suboptimal.
>
> The next time I update my system, I'll try to remember to keep a
> transcript of this, so that I can be more specific.

Here's a transcript:

--8<---------------cut here---------------start------------->8---
activating system...
building /gnu/store/fbp6bbxw9cf617fmk57sddrz7zfsfw5p-module-import-compiled.drv...
building /gnu/store/wfi6hnr9ggal0s1d32xx5wbl5k5wqlvx-switch-to-system.scm.drv...
making '/gnu/store/mjzk53ia3bajn08lscpyzz5apcw3r70g-system' the current system...
setting up setuid programs in '/run/setuid-programs'...
populating /etc from /gnu/store/l7r1has973n26hfqrs6vxbi94xzgh360-etc...
building /gnu/store/h2fqcxv3xx14lkdhyphm3lawkayw7sdl-module-import-compiled.drv...
building /gnu/store/dar9smjyxmri6v6cchnmp5mpyiimyx64-install-bootloader.scm.drv...
guix system: bootloader successfully installed on '/dev/sda'
building /gnu/store/vkk3h5p799lfpmf6msdhrzlq0wqvk3zq-module-import-compiled.drv...
building /gnu/store/hn8sr8p13gg2mf379xawscabckp03fkb-upgrade-shepherd-services.scm.drv...
shepherd: Evaluating user expression (let* ((services (map primitive-load (?))) # ?) ?).
guix system: warning: only 3.9% of free space available on /gnu/store
hint: Consider deleting old profile generations and collecting garbage, along these lines:

     guix gc --delete-generations=1m

--8<---------------cut here---------------end--------------->8---

       Mark

bug#36855: guix system switch-generation doesn't

[Jakob L. Kreuze]

[-- Attachment #1: Type: text/plain, Size: 644 bytes --]

Hi Ludo,

Ludovic Courtès <ludo@gnu.org> writes:

> Jakob, now that we generate scripts for the effectful bits of system
> reconfiguration (one of these bits being service upgrades), couldn’t
> we take it one step further and store those scripts in the “system”
> derivation so we can run them eventually, notably upon
> ‘switch-generation’?

We'd need to find a way of serializing at least the relationships
between services, but I think it's possible (albeit quite involved). I
do really like the idea, though. That way, the system generation would
fully encompass the desired state of the system.

Regards,
Jakob

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: bug#36855: guix system switch-generation doesn't

[Jakob L. Kreuze]

[-- Attachment #1: Type: text/plain, Size: 2296 bytes --]

Hi Mark,

Mark H Weaver <mhw@netris.org> writes:

> Hello again,
>
> Mark H Weaver <mhw@netris.org> writes:
>
>> As a bonus, this approach might solve another issue I've observed: on my
>> Guix system, where I build everything locally, several derivations are
>> built *during* activation.  Based on the terminal output, I get the
>> impression that the system is compiling things while the system in an
>> intermediate state, when some of the activation steps have been done,
>> but not all of them.
>>
>> As I recall, the derivations built during activation are limited to
>> compiled modules for Guile, but it still sometimes takes on the order of
>> a minute or two on my laptop to complete the "activating system" steps.
>> This seems suboptimal.
>>
>> The next time I update my system, I'll try to remember to keep a
>> transcript of this, so that I can be more specific.
>
> Here's a transcript:
>
> activating system...
> building /gnu/store/fbp6bbxw9cf617fmk57sddrz7zfsfw5p-module-import-compiled.drv...
> building /gnu/store/wfi6hnr9ggal0s1d32xx5wbl5k5wqlvx-switch-to-system.scm.drv...
> making '/gnu/store/mjzk53ia3bajn08lscpyzz5apcw3r70g-system' the current system...
> setting up setuid programs in '/run/setuid-programs'...
> populating /etc from /gnu/store/l7r1has973n26hfqrs6vxbi94xzgh360-etc...
> building /gnu/store/h2fqcxv3xx14lkdhyphm3lawkayw7sdl-module-import-compiled.drv...
> building /gnu/store/dar9smjyxmri6v6cchnmp5mpyiimyx64-install-bootloader.scm.drv...
> guix system: bootloader successfully installed on '/dev/sda'
> building /gnu/store/vkk3h5p799lfpmf6msdhrzlq0wqvk3zq-module-import-compiled.drv...
> building /gnu/store/hn8sr8p13gg2mf379xawscabckp03fkb-upgrade-shepherd-services.scm.drv...
> shepherd: Evaluating user expression (let* ((services (map primitive-load (?))) # ?) ?).
> guix system: warning: only 3.9% of free space available on /gnu/store
> hint: Consider deleting old profile generations and collecting garbage, along these lines:
>
>      guix gc --delete-generations=1m
>
>        Mark

Thanks for the input; I wasn't aware that the activation process was
taking so long for some people. One of Ludovic's suggestions was to
create a single derivation, rather than three, to speed up system
activation. I'll look into this further.

Regards,
Jakob

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: bug#36855: guix system switch-generation doesn't

[Mark H Weaver]

Hi Jakob,

zerodaysfordays@sdf.lonestar.org (Jakob L. Kreuze) writes:

> Thanks for the input; I wasn't aware that the activation process was
> taking so long for some people. One of Ludovic's suggestions was to
> create a single derivation, rather than three, to speed up system
> activation. I'll look into this further.

To be clear, I don't care how long it takes to build these derivations.
However, I think they should all be built before starting to activate
the system.  Does that make sense?

On a side note, what would happen if one of those builds failed?  Would
the system activation be left half-done?

     Thanks,
       Mark

bug#36855: guix system switch-generation doesn't

[Jakob L. Kreuze]

[-- Attachment #1: Type: text/plain, Size: 1008 bytes --]

Hi Mark,

Mark H Weaver <mhw@netris.org> writes:

> Hi Jakob,
>
> zerodaysfordays@sdf.lonestar.org (Jakob L. Kreuze) writes:
>
>> Thanks for the input; I wasn't aware that the activation process was
>> taking so long for some people. One of Ludovic's suggestions was to
>> create a single derivation, rather than three, to speed up system
>> activation. I'll look into this further.
>
> To be clear, I don't care how long it takes to build these
> derivations. However, I think they should all be built before starting
> to activate the system. Does that make sense?

Yes, and I agree that it would be ideal to have the derivations built
prior to system activation -- that way, the activation scripts could be
included in 'show-what-to-build*'.

> On a side note, what would happen if one of those builds failed? Would
> the system activation be left half-done?

Yes. You raise another very good reason for why system activation should
be carried out by a single, atomic activation script :)

Regards,
Jakob

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]