GNU Shepherd 0.10.0rc1 available for testing!

GNU Shepherd 0.10.0rc1 available for testing!

[Ludovic Courtès]

[-- Attachment #1: Type: text/plain, Size: 9357 bytes --]

Hello Guix!

I am pleased to announce that the first release candidate of version
0.10.0 of the GNU Shepherd is available for testing!

If you’re using Guix System or Guix Home, check out the instructions
below to give it a try.  Please report success or failure!  If you had
problems with previous versions, please check if they’re still there.

  code: https://alpha.gnu.org/gnu/shepherd/shepherd-0.10.0rc1.tar.gz
  signature: https://alpha.gnu.org/gnu/shepherd/shepherd-0.10.0rc1.tar.gz.sig
  sha256: 1x3lxsi6xhhds4pq30c3shydmhiidkf1wl2l7mxkpklmlycnbqgg

In your operating system configuration (and similarly for your
‘home-environment’), make the following changes:

  (use-modules (gnu packages admin))

  (define shepherd-next
    (package
      (inherit shepherd)
      (version "0.10.0rc1")
      (source (origin
                (method url-fetch)
                (uri (string-append
                      "https://alpha.gnu.org/gnu/shepherd/shepherd-"
                      version ".tar.gz"))
                (sha256
                 (base32
                  "1x3lxsi6xhhds4pq30c3shydmhiidkf1wl2l7mxkpklmlycnbqgg"))))))

  (operating-system
    ;; …
    (essential-services
     (modify-services (operating-system-default-essential-services
                       this-operating-system)
       (shepherd-root-service-type
        config => (shepherd-configuration
                   (shepherd shepherd-next))))))

You can then reconfigure, reboot, and enjoy!

You can also help with translation.  An updated PO template should soon
be available at the Translation Project:

  https://translationproject.org/domain/shepherd.html

My goal is for 0.10.x to be the last series before 1.0.  The list of
changes is quite long—see the ‘NEWS’ excerpt below.

Feedback more than welcome!

Ludo’.


* Changes in 0.10.0 (yet to be released)

** Distinguish ‘starting’ and ‘stopping’ intermediate service statuses

In previous version, a service would be either “running” or “stopped”.  The
intermediate states “starting” and “stopping” are now properly captured and
you can see them when running ‘herd status’.

** ‘start’ and ‘stop’ block when service is already being started/stopped
  <https://issues.guix.gnu.org/54786#4>

With previous version, a client running ‘herd start SERVICE’ while SERVICE is
already being started would cause shepherd to attempt to start a second
instance of that service, ultimately resulting in confusion, disappointment,
and frustration.

This is no longer the case: when a service is already being started/stopped,
additional invocation of ‘herd start’ or ‘herd stop’ now block until the
service is running/stopped.

** ‘shepherd’ starts services in parallel

Services started with ‘start-in-the-background’ and more generally service
dependencies get started in parallel.  This can reduce startup times in case
of a “wide” service dependency graph with some services that take a while to
start.

** ‘shepherd’ keeps track of failures and status change times

For each service, shepherd maintains an event log including the time of recent
status changes as well as the time of startup failures, if any.  The ‘herd
status SERVICE’ command now shows the time when the service entered its
current status and whether it failed to start; ‘herd status’ also prominently
lists services that failed to start.

** New ‘herd log’ command

Related to the previous item, the new ‘herd log’ command displays an aggregate
of the service event logs, showing the time at which each service changed
statuses.

** New ‘herd graph’ command

The new ‘herd graph’ command emits a Graphviz/Dot representation of the
service dependency graph, which can be viewed for example with ‘xdot’:

  herd graph | xdot -

Guix System users get similar information with ‘guix system shepherd-graph’
(and likewise for Guix Home).  The difference here is that this reflects the
current system status, showing transient services, services that failed to
start, and so on.

** ‘herd’ output is colorized

At long last!  We hope you’ll enjoy a little bit of coloring to highlight
important bits in the output of various commands.

** New services shipped: ‘monitoring’ and ‘repl’

The Shepherd now ships with optional services—see “Service Collection” in the
manual.  The ‘monitoring’ service logs resource usage of the ‘shepherd’
process itself.  The ‘repl’ service runs a read-eval-print loop (REPL) in the
‘shepherd’ so you can hack it live—enjoy it, but handle it with care!

** Socket-actived, systemd-style services can now be started eagerly

The ‘make-systemd-constructor’ procedure has a new #:lazy-start? parameter.
It defaults to #true, meaning that the process is started lazily, on the first
connection to one of its sockets, as was the case in 0.9.x.  Passing
#:lazy-start? #false instructs shepherd to instead start the process eagerly,
as soon as the listening sockets are ready.

This is useful for services that require socket activation as a startup
synchronization mechanism, yet are expected to run as soon as possible.  An
example is ‘guix publish --advertise’: it should be started eagerly so it can
start advertising itself via Avahi.

** Each registered name maps to exactly one service

There used to be a fuzzy notion of “conflicting services”, when a given
service name could potentially refer to more than one service.  This has
proved to be confusing more than anything else; now, each registered service
name refers to exactly one service.  The interface related to that feature,
such as the ‘conflicts-with’ method, is done.

** For systemd and inetd services, retry ‘bind’ upon EADDRINUSE
   <https://issues.guix.gnu.org/58485#13>

Services started with ‘make-systemd-constructor’ and ‘make-inetd-constructor’
will now retry several times when ‘bind’ returns EADDRINUSE (“Address already
in use”) for their listening socket(s).

** ‘system’ and ‘make-system-constructor’ are now non-blocking
   <https://issues.guix.gnu.org/61803>

In versions up to 0.9.3, calling Guile’s ‘system’ procedure (which is what
‘make-system-constructor’ does) would block the ‘shepherd’ process until the
shell spawned by ‘system’ has terminated.  This is no longer the case.

** GOOPS interface is deprecated

When it was created in 2002, the Shepherd (née dmd) embraced GOOPS, Guile’s
object-oriented programming system, then a brand new and promising approach
for 21st century programs.  In hindsight, while there were a couple of classes
and a bunch of methods, the code base was not really making much use of GOOPS.
The current maintainer deemed it unnecessary and encouraging a programming
style at odds with the shiny horizon of purely functional, actor-style
programming.

The GOOPS interface is still available in 0.10.0; for example, you can still
write ~(make <service> #:provides …)~ in your configuration file.  However,
GOOPS support will be removed in the next major series, most likely labeled
1.0.

A new interface has been defined.  Check out the “Legacy GOOPS Interface”
section of the manual for more information, and email guix-devel@gnu.org if
you have any questions or concerns.

** Interfaces removed and changed

Several obscure or undocumented interfaces were removed:

  - support for the ‘unknown’ service;
  - support for “persistency” (sic);
  - the ‘cd’ action of the ‘root’ service;
  - the ‘launch-service’ procedure of (shepherd service).

New deprecations:

  - ‘make-actions’ is deprecated in favor of ‘actions’;
  - calling ‘register-services’ with an arbitrary number of arguments is now
    deprecated; you should now call it with a single argument, the list of
    services to register.

** Major internal overhaul

As you can guess from the list of user-visible changes above, the Shepherd has
undergone a major internal overhaul.  The 0.9.x series introduced the use of
Fibers, Guile’s lightweight concurrent facility; shepherd took advantage of it
notably with the introduction of systemd-style and inetd-style services.  This
new stable series takes it further.

In particular, each <service> record has an associated fiber called the
“service controller”.  Following the actor model, each of these fibers reacts
to messages it receives, be they event notification—e.g., process
termination—or user requests—e.g., querying the service status, requesting
that the service be stopped.  Other noteworthy actors include the “process
monitor” and the “service registry”.

This has allowed us to address a number of race conditions while also leading
to clearer code with linear flows that one can more easily reason about.
Overall, it makes the code base much more pleasant to work with and certainly
easier to hack than other implementations mired in the “callback hell”.

Documentation has been overhauled as well to reflect all these changes.  Check
out the new subsections under “Services” for more information.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 853 bytes --]

Re: GNU Shepherd 0.10.0rc1 available for testing!

[Pjotr Prins]

Awesome work! Note that you can run shepherd in user land too - that
is a great feature.

Pj.

On Fri, Apr 28, 2023 at 03:37:12PM +0200, Ludovic Courtès wrote:
> Hello Guix!
> 
> I am pleased to announce that the first release candidate of version
> 0.10.0 of the GNU Shepherd is available for testing!
> 
> If you’re using Guix System or Guix Home, check out the instructions
> below to give it a try.  Please report success or failure!  If you had
> problems with previous versions, please check if they’re still there.
> 
>   code: https://alpha.gnu.org/gnu/shepherd/shepherd-0.10.0rc1.tar.gz
>   signature: https://alpha.gnu.org/gnu/shepherd/shepherd-0.10.0rc1.tar.gz.sig
>   sha256: 1x3lxsi6xhhds4pq30c3shydmhiidkf1wl2l7mxkpklmlycnbqgg
> 
> In your operating system configuration (and similarly for your
> ‘home-environment’), make the following changes:
> 
>   (use-modules (gnu packages admin))
> 
>   (define shepherd-next
>     (package
>       (inherit shepherd)
>       (version "0.10.0rc1")
>       (source (origin
>                 (method url-fetch)
>                 (uri (string-append
>                       "https://alpha.gnu.org/gnu/shepherd/shepherd-"
>                       version ".tar.gz"))
>                 (sha256
>                  (base32
>                   "1x3lxsi6xhhds4pq30c3shydmhiidkf1wl2l7mxkpklmlycnbqgg"))))))
> 
>   (operating-system
>     ;; …
>     (essential-services
>      (modify-services (operating-system-default-essential-services
>                        this-operating-system)
>        (shepherd-root-service-type
>         config => (shepherd-configuration
>                    (shepherd shepherd-next))))))
> 
> You can then reconfigure, reboot, and enjoy!
> 
> You can also help with translation.  An updated PO template should soon
> be available at the Translation Project:
> 
>   https://translationproject.org/domain/shepherd.html
> 
> My goal is for 0.10.x to be the last series before 1.0.  The list of
> changes is quite long—see the ‘NEWS’ excerpt below.
> 
> Feedback more than welcome!
> 
> Ludo’.
> 
> 
> * Changes in 0.10.0 (yet to be released)
> 
> ** Distinguish ‘starting’ and ‘stopping’ intermediate service statuses
> 
> In previous version, a service would be either “running” or “stopped”.  The
> intermediate states “starting” and “stopping” are now properly captured and
> you can see them when running ‘herd status’.
> 
> ** ‘start’ and ‘stop’ block when service is already being started/stopped
>   <https://issues.guix.gnu.org/54786#4>
> 
> With previous version, a client running ‘herd start SERVICE’ while SERVICE is
> already being started would cause shepherd to attempt to start a second
> instance of that service, ultimately resulting in confusion, disappointment,
> and frustration.
> 
> This is no longer the case: when a service is already being started/stopped,
> additional invocation of ‘herd start’ or ‘herd stop’ now block until the
> service is running/stopped.
> 
> ** ‘shepherd’ starts services in parallel
> 
> Services started with ‘start-in-the-background’ and more generally service
> dependencies get started in parallel.  This can reduce startup times in case
> of a “wide” service dependency graph with some services that take a while to
> start.
> 
> ** ‘shepherd’ keeps track of failures and status change times
> 
> For each service, shepherd maintains an event log including the time of recent
> status changes as well as the time of startup failures, if any.  The ‘herd
> status SERVICE’ command now shows the time when the service entered its
> current status and whether it failed to start; ‘herd status’ also prominently
> lists services that failed to start.
> 
> ** New ‘herd log’ command
> 
> Related to the previous item, the new ‘herd log’ command displays an aggregate
> of the service event logs, showing the time at which each service changed
> statuses.
> 
> ** New ‘herd graph’ command
> 
> The new ‘herd graph’ command emits a Graphviz/Dot representation of the
> service dependency graph, which can be viewed for example with ‘xdot’:
> 
>   herd graph | xdot -
> 
> Guix System users get similar information with ‘guix system shepherd-graph’
> (and likewise for Guix Home).  The difference here is that this reflects the
> current system status, showing transient services, services that failed to
> start, and so on.
> 
> ** ‘herd’ output is colorized
> 
> At long last!  We hope you’ll enjoy a little bit of coloring to highlight
> important bits in the output of various commands.
> 
> ** New services shipped: ‘monitoring’ and ‘repl’
> 
> The Shepherd now ships with optional services—see “Service Collection” in the
> manual.  The ‘monitoring’ service logs resource usage of the ‘shepherd’
> process itself.  The ‘repl’ service runs a read-eval-print loop (REPL) in the
> ‘shepherd’ so you can hack it live—enjoy it, but handle it with care!
> 
> ** Socket-actived, systemd-style services can now be started eagerly
> 
> The ‘make-systemd-constructor’ procedure has a new #:lazy-start? parameter.
> It defaults to #true, meaning that the process is started lazily, on the first
> connection to one of its sockets, as was the case in 0.9.x.  Passing
> #:lazy-start? #false instructs shepherd to instead start the process eagerly,
> as soon as the listening sockets are ready.
> 
> This is useful for services that require socket activation as a startup
> synchronization mechanism, yet are expected to run as soon as possible.  An
> example is ‘guix publish --advertise’: it should be started eagerly so it can
> start advertising itself via Avahi.
> 
> ** Each registered name maps to exactly one service
> 
> There used to be a fuzzy notion of “conflicting services”, when a given
> service name could potentially refer to more than one service.  This has
> proved to be confusing more than anything else; now, each registered service
> name refers to exactly one service.  The interface related to that feature,
> such as the ‘conflicts-with’ method, is done.
> 
> ** For systemd and inetd services, retry ‘bind’ upon EADDRINUSE
>    <https://issues.guix.gnu.org/58485#13>
> 
> Services started with ‘make-systemd-constructor’ and ‘make-inetd-constructor’
> will now retry several times when ‘bind’ returns EADDRINUSE (“Address already
> in use”) for their listening socket(s).
> 
> ** ‘system’ and ‘make-system-constructor’ are now non-blocking
>    <https://issues.guix.gnu.org/61803>
> 
> In versions up to 0.9.3, calling Guile’s ‘system’ procedure (which is what
> ‘make-system-constructor’ does) would block the ‘shepherd’ process until the
> shell spawned by ‘system’ has terminated.  This is no longer the case.
> 
> ** GOOPS interface is deprecated
> 
> When it was created in 2002, the Shepherd (née dmd) embraced GOOPS, Guile’s
> object-oriented programming system, then a brand new and promising approach
> for 21st century programs.  In hindsight, while there were a couple of classes
> and a bunch of methods, the code base was not really making much use of GOOPS.
> The current maintainer deemed it unnecessary and encouraging a programming
> style at odds with the shiny horizon of purely functional, actor-style
> programming.
> 
> The GOOPS interface is still available in 0.10.0; for example, you can still
> write ~(make <service> #:provides …)~ in your configuration file.  However,
> GOOPS support will be removed in the next major series, most likely labeled
> 1.0.
> 
> A new interface has been defined.  Check out the “Legacy GOOPS Interface”
> section of the manual for more information, and email guix-devel@gnu.org if
> you have any questions or concerns.
> 
> ** Interfaces removed and changed
> 
> Several obscure or undocumented interfaces were removed:
> 
>   - support for the ‘unknown’ service;
>   - support for “persistency” (sic);
>   - the ‘cd’ action of the ‘root’ service;
>   - the ‘launch-service’ procedure of (shepherd service).
> 
> New deprecations:
> 
>   - ‘make-actions’ is deprecated in favor of ‘actions’;
>   - calling ‘register-services’ with an arbitrary number of arguments is now
>     deprecated; you should now call it with a single argument, the list of
>     services to register.
> 
> ** Major internal overhaul
> 
> As you can guess from the list of user-visible changes above, the Shepherd has
> undergone a major internal overhaul.  The 0.9.x series introduced the use of
> Fibers, Guile’s lightweight concurrent facility; shepherd took advantage of it
> notably with the introduction of systemd-style and inetd-style services.  This
> new stable series takes it further.
> 
> In particular, each <service> record has an associated fiber called the
> “service controller”.  Following the actor model, each of these fibers reacts
> to messages it receives, be they event notification—e.g., process
> termination—or user requests—e.g., querying the service status, requesting
> that the service be stopped.  Other noteworthy actors include the “process
> monitor” and the “service registry”.
> 
> This has allowed us to address a number of race conditions while also leading
> to clearer code with linear flows that one can more easily reason about.
> Overall, it makes the code base much more pleasant to work with and certainly
> easier to hack than other implementations mired in the “callback hell”.
> 
> Documentation has been overhauled as well to reflect all these changes.  Check
> out the new subsections under “Services” for more information.

Re: GNU Shepherd 0.10.0rc1 available for testing!

[pelzflorian (Florian Pelz)]

Thank you shepherds!  Also the explanations in the news are great.
Finally I understand why GOOPS is not desirable in shepherd.

For Guix Home:

Ludovic Courtès <ludo@gnu.org> writes:
> In your operating system configuration (and similarly for your
> ‘home-environment’), make the following changes:

For Guix Home, it works for me to put this in home-environment; no need
to fiddle with home-environment-essential-services.

(home-environment
  …
  (services
   (list (service home-shepherd-service-type
                  (home-shepherd-configuration
                   (shepherd shepherd-next)))
         …


Regards,
Florian

Re: GNU Shepherd 0.10.0rc1 available for testing!

[pelzflorian (Florian Pelz)]

Wow, shepherd 0.10.0rc1 put an end to occasional hangups when rebooting
my slow Beebox server!  Thanks once more.

Regards,
Florian

Re: GNU Shepherd 0.10.0rc1 available for testing!

[Tanguy LE CARROUR]

Hi,


Quoting pelzflorian (Florian Pelz) (2023-04-29 20:29:52)
> Thank you shepherds!  Also the explanations in the news are great.
> Finally I understand why GOOPS is not desirable in shepherd.
> 
> For Guix Home:
> 
> Ludovic Courtès <ludo@gnu.org> writes:
> > In your operating system configuration (and similarly for your
> > ‘home-environment’), make the following changes:
> 
> For Guix Home, it works for me to put this in home-environment; no need
> to fiddle with home-environment-essential-services.
> 
> (home-environment
>   …
>   (services
>    (list (service home-shepherd-service-type
>                   (home-shepherd-configuration
>                    (shepherd shepherd-next)))
>          …

I did that and… I ended up with the following build error:

```
$ guix home reconfigure tanguy.home.scm
# […]
building /gnu/store/s30bi2zc3m98imlp21ncmzrs3dyi5k3k-shepherd-0.10.0rc1.drv...
| 'configure' phasebuilder for `/gnu/store/s30bi2zc3m98imlp21ncmzrs3dyi5k3k-shepherd-0.10.0rc1.drv' failed with exit code 1
build of /gnu/store/s30bi2zc3m98imlp21ncmzrs3dyi5k3k-shepherd-0.10.0rc1.drv failed
View build log at '/var/log/guix/drvs/s3/0bi2zc3m98imlp21ncmzrs3dyi5k3k-shepherd-0.10.0rc1.drv.gz'.
cannot build derivation `/gnu/store/78i3qkl8y247w5w9fpjk69vq5jmscha8-activate.drv': 1 dependencies couldn't be built
cannot build derivation `/gnu/store/pv7pfl76i1rnsdzjxsrv1rkqab999nls-on-first-login.drv': 1 dependencies couldn't be built
cannot build derivation `/gnu/store/384414cysz4pfrjdsvbgxl15ki6sah7n-profile.drv': 1 dependencies couldn't be built
cannot build derivation `/gnu/store/92ipayml42pmhwmp0x6cwms2fs58xhfq-home.drv': 1 dependencies couldn't be built
guix home: error: build of `/gnu/store/92ipayml42pmhwmp0x6cwms2fs58xhfq-home.drv' failed

$ gunzip -c /var/log/guix/drvs/s3/0bi2zc3m98imlp21ncmzrs3dyi5k3k-shepherd-0.10.0rc1.drv.gz
# […]
checking if (fibers) is available... no
configure: error: Fibers is missing; please install it.
error: in phase 'configure': uncaught exception:
%exception #<&invoke-error program: "/gnu/store/4y5m9lb8k3qkb1y9m02sw9w9a6hacd16-bash-minimal-5.1.8/bin/bash" arguments: ("./configure" "CONFIG_SHELL=/gnu/store/4y5m9lb8k3qkb1y9m02sw9w9a6hacd16-bash-minimal-5.1.8/bin/bash" "SHELL=/gnu/store/4y5m9lb8k3qkb1y9m02sw9w9a6hacd16-bash-minimal-5.1.8/bin/bash" "--prefix=/gnu/store/p4njzm47a1svbfcc845w7sbiwxy5xgm0-shepherd-0.10.0rc1" "--enable-fast-install" "--build=x86_64-unknown-linux-gnu" "--localstatedir=/var") exit-status: 1 term-signal: #f stop-signal: #f>
phase `configure' failed after 3.7 seconds
command "/gnu/store/4y5m9lb8k3qkb1y9m02sw9w9a6hacd16-bash-minimal-5.1.8/bin/bash" "./configure" "CONFIG_SHELL=/gnu/store/4y5m9lb8k3qkb1y9m02sw9w9a6hacd16-bash-minimal-5.1.8/bin/bash" "SHELL=/gnu/store/4y5m9lb8k3qkb1y9m02sw9w9a6hacd16-bash-minimal-5.1.8/bin/bash" "--prefix=/gnu/store/p4njzm47a1svbfcc845w7sbiwxy5xgm0-shepherd-0.10.0rc1" "--enable-fast-install" "--build=x86_64-unknown-linux-gnu" "--localstatedir=/var" failed with status 1
```

Have I missed something?


-- 
Tanguy

Re: GNU Shepherd 0.10.0rc1 available for testing!

[pelzflorian (Florian Pelz)]

Hi Tanguy,

Tanguy LE CARROUR <tanguy@bioneland.org> writes:
> I did that and… I ended up with the following build error:
> […]
> checking if (fibers) is available... no
> configure: error: Fibers is missing; please install it.

Could it be that you have not run “guix pull” recently
or that a wrong Guix program gets called (you can check what it printed
by “which guix”; perhaps at the end of your .bashrc you have something
configured differently from what Guix expects)?

Regards,
Florian

Re: GNU Shepherd 0.10.0rc1 available for testing!

[Tanguy LE CARROUR]

Quoting pelzflorian (Florian Pelz) (2023-05-03 14:51:41)
> Tanguy LE CARROUR <tanguy@bioneland.org> writes:
> > I did that and… I ended up with the following build error:
> > […]
> > checking if (fibers) is available... no
> > configure: error: Fibers is missing; please install it.
> 
> Could it be that you have not run “guix pull” recently

Oh… this must be it!
As soon as I noticed that `poerty` was broken, I rolled back
and postponed the upgrade.

My bad!

Thanks.

-- 
Tanguy

Re: GNU Shepherd 0.10.0rc1 available for testing!

[Ludovic Courtès]

Hi!

Ludovic Courtès <ludo@gnu.org> skribis:

> I am pleased to announce that the first release candidate of version
> 0.10.0 of the GNU Shepherd is available for testing!

An update: a dozen commits have been pushed since rc1, a few of them
fixing logic issues and others making tests more robust.

There remains a bug on AArch64 that manifests as Shepherd test failures,
but it’s a Fibers or Guile bug:

  https://github.com/wingo/fibers/issues/83

Turning off JIT by setting ‘GUILE_JIT_THRESHOLD=-1’ is a simple way to
work around it though, so maybe we’ll just do that for this release.

Anyway I think we’ll have an rc2 before the release.

Thanks to everyone who tested it already!

Ludo’.