watchdog triggered auto-rollback

watchdog triggered auto-rollback

[raingloom]

Since I've been experimenting with a foolproof unikernel based static
website deployment lately, I realized I should write down this idea I've
been chewing for a while:

It would be very nice to have automatic system rollbacks when certain
things break.
One example is broken SSH config that makes a machine unreachable.
Local testing is useful, but like in the SSH example, some issues only
become apparent when you are deploying to the production environment.

Would others find this useful?  Where in the stack would this be solved?
 Could we, for example, catch an issue in the init system and still
perform a rollback?  Or if not a full rollback, then at least a reboot
into the previous config?  (And if that is also broken, then the one
before, etc, etc)

Obviously there are a lot of edge cases and potential bugs in this
mechanism as well.  Sticking with the SSH example, rolling back to a
version that was kept around where the authorized keys are different
would also make the machine unreachable via SSH.

Re: watchdog triggered auto-rollback

[Richard Sent]

raingloom@riseup.net writes:

> Would others find this useful?  Where in the stack would this be solved?
>  Could we, for example, catch an issue in the init system and still
> perform a rollback?  Or if not a full rollback, then at least a reboot
> into the previous config?  (And if that is also broken, then the one
> before, etc, etc)
>
> Obviously there are a lot of edge cases and potential bugs in this
> mechanism as well.  Sticking with the SSH example, rolling back to a
> version that was kept around where the authorized keys are different
> would also make the machine unreachable via SSH.

I would definitely find this useful, particularly when combined with
unattended-upgrade-service. I don't know the best way to handle an init
system failure.

Perhaps a starting point would be a one-shot
conditional-rollback-service with a "shepherd-requirements" field and a
"test" field that takes an file-like object. This service would execute
that file, write the output to some log file, and trigger a rollback if
an error is signaled.

Presumably this service should only trigger on boot, not reconfigure so
we don't risk running the test with old services. I don't believe Guix
has a mechanism yet to say "Yes, this service is new, and I /do/ want
Shepherd to auto-start it, but not on reconfigure". This shouldn't be
too hard to add though.

-- 
Take it easy,
Richard Sent
Making my computer weirder one commit at a time.

watchdog triggered auto-rollback

[Nathan Dehnel]

>Would others find this useful?
I would 100% use this.

>Where in the stack would this be solved?
I think there's two places for rollbacks with two different purposes

GRUB: https://www.gnu.org/software/grub/manual/grub/html_node/fallback.html
GRUB supports falling back to another boot entry if the machine fails
to boot. This could be integrated with guix so GRUB falls back to a
previous guix system generation. This covers the case of "we can't
start a watchdog service because the system won't boot".

SSH watchdog: a shepherd service that tests SSH connectivity, and then
executes "guix system roll-back && reboot". SSH access is a rough
approximation for "the system is working", as kernel, init, and all
manner of networking services, DHCP, DNS, VPN, etc. must work for SSH
to work. And if SSH works then it provides a means for a user to fix
their system anyways.

Re: watchdog triggered auto-rollback

[Richard Sent]

Nathan Dehnel <ncdehnel@gmail.com> writes:

> GRUB: https://www.gnu.org/software/grub/manual/grub/html_node/fallback.html
> GRUB supports falling back to another boot entry if the machine fails
> to boot. This could be integrated with guix so GRUB falls back to a
> previous guix system generation. This covers the case of "we can't
> start a watchdog service because the system won't boot".

How does GRUB determine a boot failed? Does it have to be something
drastic like "kernel failed to mount the initrd" or can it catch more
complex errors?

I believe that if the initrd fails during startup it will abort into an
interactive Guile REPL. This might hurt GRUB's ability to detect
something went wrong since the kernel would still be running. A similar
case may apply if Shepherd gets stuck during system initialization.

-- 
Take it easy,
Richard Sent
Making my computer weirder one commit at a time.

Re: watchdog triggered auto-rollback

[Attila Lendvai]

> I believe that if the initrd fails during startup it will abort into an
> interactive Guile REPL. This might hurt GRUB's ability to detect
> something went wrong since the kernel would still be running.


i'm afraid that's not the case currently:

%guile-static-stripped crashes with a sigsegv (i.e. the guile used in the initrd (?))
https://issues.guix.gnu.org/71211

-- 
• attila lendvai
• PGP: 963F 5D5F 45C7 DFCD 0A39
--
“Self-education is, I firmly believe, the only kind of education there is.”
	— Isaac Asimov (1920–1992)

Re: watchdog triggered auto-rollback

[Richard Sent]

> i'm afraid that's not the case currently:
> 
> %guile-static-stripped crashes with a sigsegv (i.e. the guile used in the initrd (?))
> https://issues.guix.gnu.org/71211

Interesting. Is this a recent bug? When I was trying to bring up Guix on the VisionFive 2 I was being dropped into a Guile REPL when the initrd failed to find the root partition.

Re: watchdog triggered auto-rollback

[Attila Lendvai]

> > i'm afraid that's not the case currently:
> > 
> > %guile-static-stripped crashes with a sigsegv (i.e. the guile used in the initrd (?))
> > https://issues.guix.gnu.org/71211
> 
>
> Interesting. Is this a recent bug? When I was trying to bring up
> Guix on the VisionFive 2 I was being dropped into a Guile REPL when
> the initrd failed to find the root partition.


well, the reproducer "works" on a recent x86_64, but i originally noticed this long ago (maybe a year even). back then i investigated an early crash in the boot, and reached %GUILE-STATIC-STRIPPED, and made a TODO note to further investigate. then i forgot most of what happened, and recently i opened a bug report based on my note.

since then EXPRESSION->INITRD may have changed, because it now uses %GUILE-STATIC-INITRD. but it's created with the same MAKE-GUILE-STATIC-STRIPPED that produces the faulty %GUILE-STATIC-STRIPPED, so...

in short: the reproducer crashes both %GUILE-STATIC-STRIPPED and %GUILE-STATIC-INITRD on x86_64, and i believe that it crashes the same in the early phase of the boot when it tries to enter the debugger.

-- 
• attila lendvai
• PGP: 963F 5D5F 45C7 DFCD 0A39
--
Freedom cannot be given… it can only be taken away.