bug#53580: shepherd's architecture

[Brian Cully via Bug reports for GNU Guix <bug-guix@gnu.org>]

Attila Lendvai <attila@lendvai.name> writes:

> it doesn't seem to be an insurmontable task to make sure that 
> guile
> can safely unlink a module from its heap, check if there are any
> references into the module to be dropped, and then reload this 
> module
> from disk.
>
> the already runing fibers would keep the required code in the 
> heap
> until after they are stopped/restarted. then the module would 
> get GC'd
> eventually.
>
> this would help solve the problem that a reconfigured service 
> may have
> a completely different start/stop code. and by taking some 
> careful
> shortcuts we may be able to make reloading work without having 
> to stop
> the service process in question.

Erlang has had hot code reloading for decades, built around the 
needs of 100% uptime systems. The problem is more complex than it 
often appears to people who are used to how lisps traditionally do 
it. I strongly recommend reading up on Erlang's migration 
system. Briefly: you can't just swap out function definitions, 
because they rely on non-function state which needs to be migrated 
along with the function itself, and you can't do it whenever you 
want, because external actors may be relying on a view of the 
internal state. To accomplish this, Erlang has a lot of machinery, 
and it fits in to the core design of the language and runtime 
which would be extremely difficult to port over to non-Erlang 
languages. Doing it in Scheme is probably possible in an academic 
sense, but not in a practical one.

OTOH, Lisp Flavoured Erlang exists if you want that syntax. There 
would definitely be advantages to writing an init (and, indeed, 
any service that needs 100% uptime) on top of the Erlang virtual 
machine. But going the other way, by porting Erlang's 
functionality into Scheme, is going to be a wash.

> in this setup most of the complexity and the evolution of the 
> shepherd
> codebase would happen in the runner, and the other two parts 
> could be
> kept minimal and would rarely need to change (and thus require a
> reboot).

Accepting that dramatic enough changes to PID 1 are going to 
require a reboot seems reasonable to me. They should be even more 
rare than kernel updates, and we accept rebooting there already.

-bjc