[bug#33508] [PATCH] gnu: Add ability to restart services on system reconfigure

[Carlo Zancanaro <carlo@zancanaro.id.au>]

Hey Ludo’,

Sorry for not responding to this email for so long. I've been 
trying to think through some of the issues around this, and I'm 
not confident that I have thought through the issues well enough 
to actually decide on a good course of action, beyond what I have 
already written. I'll respond to a few specific things in your 
message, but I don't even know what a good solution would look 
like, let alone how to build it.

On Mon, Dec 10 2018, Ludovic Courtès wrote:
> In what sense is guix-daemon “always safe to restart”?  It’s 
> actually a difficult question for me.

I agree it's tricky. I had mostly intended that as an example, 
because I used guix-daemon for my testing, but ...

> You could argue that its child guix-daemon processes will remain 
> live when we restart it, meaning that client connections remain 
> active and valid.  I believe this is indeed the case, though it 
> would be worth double-checking.

... this is what I was thinking. I'm fairly sure this is the case, 
given my observations while I was testing these patches.

> Now, if safe-to-restart means that we automatically invoke the 
> “restart” action on guix-daemon, that means that anything that 
> depends on it (‘guix-publish’, ‘cuirass’, ‘hpcguix-web’, etc.) 
> would be restarted as well (even though I *think* we don’t have 
> to in this case.)  But these may not be safe to restart: for 
> example, on may want ‘guix-publish’ to run uninterrupted.

At the moment we have no way to capture this, particularly in the 
Shepherd. There's no way to restart a service without restarting 
dependent services, but I particularly want to pick up on the 
"uninterrupted" by talking about nginx below.

> ...

> sshd, nginx, and maybe guix-daemon can more or less be 
> live-upgraded, meaning that (1) existing connections are 
> preserved but future connections will talk to the new daemon, 
> and as a corollary, (2) dependent services do not need to be 
> stopped & restarted.

I did some research into nginx, and it turns out that it is 
possible to upgrade nginx with zero-downtime by running two 
daemons simultaneously listening on the same port(s), then 
shutting down the old daemon after the new one has successfully 
started[1]. This allows for an "uninterrupted" upgrade, but I'm 
not confident that I would be able to implement it within our 
current framework.

In all, I haven't done anything with this in the last month. I've 
thought about it a few times, but it just feels a bit 
overwhelming.

Carlo

[1]: https://nginx.org/en/docs/control.html#upgrade