Guix deploy --keep-going equivalent for machine connection failures

Guix deploy --keep-going equivalent for machine connection failures

[Richard Sent]

Hi all,

When running guix deploy with a list of machines, the command will fail 
if any of the machines are unreachable. I use Guix deploy from a central 
machine to update the devices on my LAN, including laptops, which aren't 
guaranteed to be on. I'm also not terribly worried about keeping every 
single device updated in perfect synchronicity.

I think it would be useful to have a CLI flag that would continue the 
deployment even if we fail to connect to one of the machines.

At present this can be worked around by commenting out entries on the 
list, but this requires
a) Already knowing what machine is offline
b) Remembering to uncomment it later when the machine goes back online
c) Generally feels "ugly" in a way that most Guix commands don't.
d) Makes version control a pain

The default behavior should stay as is, I imagine, since one of the main 
uses of guix deploy is deploying an interconnected cluster of machines 
that probably would not appreciate being partially upgraded.

Another option worth considered is adding a `'can-connection-fail?' 
(default: #f)` to either `machine` or `machine-ssh-configuration`. 
(Probably machine since it would also allow for digitalocean 
deployments.) This would let me mix a list of "core machines" that I 
definitely want upgraded in a deployment (NAS, etc), while still having 
secondary machines that I'm not particularly concerned about upgrading.

When running a deployment, there are effectively 3 options for how I 
could want the deployment to go for machines X, Y, and Z:
a) X, Y, and Z must all be deployed together      <--- Supported
b) X must be deployed, but Y and/or Z can fail    <--- 
can-connection-fail? setting
c) X, Y, and/or Z can fail                        <--- 
--keep-going-but-for-machines-I-don't-know-what-to-call-this

Anyone have thoughts on this?

Thanks,
Richard Sent

Re: Guix deploy --keep-going equivalent for machine connection failures

[Carlo Zancanaro]

On Mon, Jan 15 2024, Richard Sent wrote:
> At present this can be worked around by commenting out entries on the
> list, but this requires
> a) Already knowing what machine is offline
> b) Remembering to uncomment it later when the machine goes back online
> c) Generally feels "ugly" in a way that most Guix commands don't.
> d) Makes version control a pain

This is generally what I've done, but I see this as a manual form of
something that could easily be added the machines.scm file itself.

For instance, you could define a helper function and use it to filter
the list of machines, like so:

  (define (reachable-host? machine)
    (let ((configuration (machine-configuration machine)))
      (cond
       ((machine-ssh-configuration? configuration)
        (zero? (system* "ssh" (machine-ssh-configuration-host-name configuration)
                        "--" "true")))
       (else
        #f))))

  (filter
   reachable-host?
   (list
    (machine ...)
    (machine ...)))

Using this sort of approach can implement whatever strategy you want, as
long as you can determine the machines which you want to deploy to
up-front (i.e. this can't implement a strategy that handles failure
partway through deployment).

Carlo

Re: Guix deploy --keep-going equivalent for machine connection failures

[Csepp]

[-- Attachment #1: Type: text/plain, Size: 1510 bytes --]

Carlo Zancanaro <carlo@zancanaro.id.au> writes:

> On Mon, Jan 15 2024, Richard Sent wrote:
>> At present this can be worked around by commenting out entries on the
>> list, but this requires
>> a) Already knowing what machine is offline
>> b) Remembering to uncomment it later when the machine goes back online
>> c) Generally feels “ugly” in a way that most Guix commands don’t.
>> d) Makes version control a pain
>
> This is generally what I’ve done, but I see this as a manual form of
> something that could easily be added the machines.scm file itself.
>
> For instance, you could define a helper function and use it to filter
> the list of machines, like so:
>
>   (define (reachable-host? machine)
>     (let ((configuration (machine-configuration machine)))
>       (cond
>        ((machine-ssh-configuration? configuration)
>         (zero? (system* “ssh” (machine-ssh-configuration-host-name configuration)
>                         “–” “true”)))
>        (else
>         #f))))
>
>   (filter
>    reachable-host?
>    (list
>     (machine …)
>     (machine …)))
>
> Using this sort of approach can implement whatever strategy you want, as
> long as you can determine the machines which you want to deploy to
> up-front (i.e. this can’t implement a strategy that handles failure
> partway through deployment).
>
> Carlo

One potential problem with this is that it might try to reconnect too often and wait too long for each timeout.

Re: Guix deploy --keep-going equivalent for machine connection failures

[Attila Lendvai]

> Another option worth considered is adding a `'can-connection-fail?' (default: #f)` to either `machine` or `machine-ssh-configuration`.

i'd call it `ignore-connection-failure?`, or if we want to ignore all problems for a machine, then `ignore-failure?`.

--keep-going could set the default value for the session, and the machine specific variable would override it.

as for the implementation, i'd use continuable exceptions of a specific type, and then from scheme code users could install handlers that ignore the situation.

avoiding shell scripts these days is a good idea after all.

-- 
• attila lendvai
• PGP: 963F 5D5F 45C7 DFCD 0A39
--
“Democracy and liberty are not the same. Democracy is little more than mob rule, while liberty refers to the sovereignty of the individual.”
	— Walter E. Williams (1936–)