GNU Shepherd 0.10.3 released

GNU Shepherd 0.10.3 released

[Ludovic Courtès]

[-- Attachment #1: Type: text/plain, Size: 6665 bytes --]

We are pleased to announce the GNU Shepherd version 0.10.3, a bug-fix
release of the new 0.10.x series, representing 51 commits over 6 months.

The 0.10.x series is a major overhaul towards 1.0, addressing shortcomings
and providing new features that help comprehend system state.


• About

  The GNU Shepherd is a service manager written in Guile that looks
  after the herd of daemons running on the system.  It can be used as an
  “init” system (PID 1) and also by unprivileged users to manage
  per-user daemons—e.g., tor, privoxy, mcron.  It supports several
  daemon startup mechanisms, including inetd and systemd-style socket
  activation.  The GNU Shepherd is configured in Guile Scheme and can be
  extended in the same language.  It builds on a simple memory-safe and
  callback-free programming model.

  The GNU Shepherd is developed jointly with the GNU Guix project; it is
  used as the init system of Guix, GNU’s advanced GNU/Linux distribution.

  https://www.gnu.org/software/shepherd/


• Download

  Here are the compressed sources and a GPG detached signature:
    https://ftp.gnu.org/gnu/shepherd/shepherd-0.10.3.tar.gz
    https://ftp.gnu.org/gnu/shepherd/shepherd-0.10.3.tar.gz.sig

  Use a mirror for higher download bandwidth:
    https://ftpmirror.gnu.org/shepherd/shepherd-0.10.3.tar.gz
    https://ftpmirror.gnu.org/shepherd/shepherd-0.10.3.tar.gz.sig

  Here are the SHA1 and SHA256 checksums:

  aac0f3f11245fca5b13119dc8e25d366fc53df23  shepherd-0.10.3.tar.gz
  jsb+vrAwu1LoTGI/ECnKccLCElFRVhxZFF2qbjuFr+8  shepherd-0.10.3.tar.gz

  The SHA256 checksum is base64 encoded, instead of the
  hexadecimal encoding that most checksum tools default to.

  Use a .sig file to verify that the corresponding file (without the
  .sig suffix) is intact.  First, be sure to download both the .sig file
  and the corresponding tarball.  Then, run a command like this:

    gpg --verify shepherd-0.10.3.tar.gz.sig

  The signature should match the fingerprint of the following key:

    pub   rsa4096 2014-08-11 [SC]
          3CE4 6455 8A84 FDC6 9DB4  0CFB 090B 1199 3D9A EBB5
    uid           [ unknown] Ludovic Courtès <ludo@gnu.org>
    uid           [ unknown] Ludovic Courtès <ludo@chbouib.org>
    uid           [ unknown] Ludovic Courtès (Inria) <ludovic.courtes@inria.fr>

  If that command fails because you don't have the required public key,
  or that public key has expired, try the following commands to retrieve
  or refresh it, and then rerun the 'gpg --verify' command.

    gpg --recv-keys 3CE464558A84FDC69DB40CFB090B11993D9AEBB5

  As a last resort to find the key, you can try the official GNU
  keyring:

    wget -q https://ftp.gnu.org/gnu/gnu-keyring.gpg
    gpg --keyring gnu-keyring.gpg --verify shepherd-0.10.3.tar.gz.sig


  This release was bootstrapped with the following tools:
    Autoconf 2.71
    Automake 1.16.5
    Makeinfo 7.0.3
    Help2man 1.49.2


• Changes since version 0.10.2 (excerpt from the NEWS file)

  ** Fix a bug that could lead shepherd to hang after loading replacements
     (<https://issues.guix.gnu.org/67839>)

  After loading replacements with ‘herd load’ or ‘guix system reconfigure’,
  shepherd could eventually hang.  Specifically, the replaced service would no
  longer respond to messages, so it would be impossible to start it, to stop it,
  or to unload it.  This is now fixed.

  ** Fix ownership and permissions on Unix-domain sockets
     (<https://issues.guix.gnu.org/67867>)

  When using an AF_UNIX endpoint with systemd- and inetd-style services, the
  socket file itself would remain owned by root (when shepherd is running as
  root) with permissions 755.  This is now fixed, with ownership set according
  to #:socket-owner and #:socket-group of the endpoint, and permissions on the
  socket set to 666.

  Likewise, #:socket-directory-permissions was previously ignored when the
  socket’s directory already existed prior to creating the endpoint, potentially
  leading to unexpectedly wide access to the socket.  This is now fixed.

  ** New #:respawn-delay parameter to ‘service’
     (<https://issues.guix.gnu.org/64665>)

  This specifies a delay before a service is respawned.  Its default value is
  given by ‘default-respawn-delay’ and defaults to 100ms.  Until now, services
  were respawned immediately.

  ** Non-blocking ‘sleep’ replacement provided

  Until now, user code could call (@ (guile) sleep), the core Guile binding for
  ‘sleep’, instead of ‘sleep’ as provided by (fibers).  The former would have
  caused ‘shepherd’ to actually sleep for that time, instead of performing other
  on-going tasks.  ‘sleep’ is now replaced by (@ (fibers) sleep) to avoid that.

  ** Ensure termination of services that failed to produce a PID file

  When a service started with the #:pid-file argument of
  ‘make-forkexec-constructor’ or similar would fail to produce its PID file, the
  process that was spawned would be sent SIGTERM.  Now, it is additionally sent
  SIGKILL after ‘default-process-termination-grace-period’ has expired, as is
  the case when using ‘make-kill-destructor’.

  ** Do not accidentally wait for Linux kernel thread completion
     (<https://issues.guix.gnu.org/67132>)

  In cases a PID file contained a bogus PID or one that’s only valid in a
  separate PID namespace, shepherd could end up waiting for the termination of
  what’s actually a Linux kernel thread, such as PID 2 (“kthreadd”).  This
  situation is now recognized and avoided.

  ** Fix portability issues to GNU/Hurd

  Previous versions in the 0.10.x and 0.9.x series did not work on GNU/Hurd.
  This is now fixed, although some features are still implemented in a
  suboptimal way.

  ** Fix cross-compilation to non-Linux operating systems

  Since 0.10.2, when cross-compiling to systems such as GNU/Hurd, the value of
  ‘SFD_CLOEXEC’ and ‘SFD_NONBLOCK’ in (shepherd system) would be left undefined,
  leading to a type error when starting shepherd.  This is now fixed.

  ** Updated translations: de, ro, sr, sv, uk

  This release is fully translated to German, Romanian, Swedish, and Ukrainian;
  partial translations are available for a dozen of languages.  To help make the
  Shepherd speak your language, check out the Translation Project:

    https://translationproject.org/domain/shepherd.html


Please report bugs to bug-guix@gnu.org.
Join guix-devel@gnu.org for discussions.

Ludovic, on behalf of the Shepherd herd.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 853 bytes --]

Re: GNU Shepherd 0.10.3 released

[Tomas Volf]

[-- Attachment #1: Type: text/plain, Size: 965 bytes --]

On 2024-01-07 15:08:59 +0100, Ludovic Courtès wrote:
> We are pleased to announce the GNU Shepherd version 0.10.3, a bug-fix
> release of the new 0.10.x series, representing 51 commits over 6 months.

Congratulations on the release :)

>
>   ** Do not accidentally wait for Linux kernel thread completion
>      (<https://issues.guix.gnu.org/67132>)
>
>   In cases a PID file contained a bogus PID or one that’s only valid in a
>   separate PID namespace, shepherd could end up waiting for the termination of
>   what’s actually a Linux kernel thread, such as PID 2 (“kthreadd”).  This
>   situation is now recognized and avoided.

This is great, I will not have to remember to run `modprobe -r mt7921e' before
each shutdown anymore.  I hope.  Looking forward to getting it in the Guix :)

Have a nice 2024,
Tomas Volf

--
There are only two hard things in Computer Science:
cache invalidation, naming things and off-by-one errors.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: GNU Shepherd 0.10.3 released

[Ludovic Courtès]

Tomas Volf <~@wolfsden.cz> skribis:

> On 2024-01-07 15:08:59 +0100, Ludovic Courtès wrote:

[...]

>>   ** Do not accidentally wait for Linux kernel thread completion
>>      (<https://issues.guix.gnu.org/67132>)
>>
>>   In cases a PID file contained a bogus PID or one that’s only valid in a
>>   separate PID namespace, shepherd could end up waiting for the termination of
>>   what’s actually a Linux kernel thread, such as PID 2 (“kthreadd”).  This
>>   situation is now recognized and avoided.
>
> This is great, I will not have to remember to run `modprobe -r mt7921e' before
> each shutdown anymore.  I hope.  Looking forward to getting it in the Guix :)

D’oh, why did you have to do that?  How did Shepherd end up with “wrong” PID?

I hope this release fixes it!

Ludo’.

Re: GNU Shepherd 0.10.3 released

[Tomas Volf]

[-- Attachment #1: Type: text/plain, Size: 3298 bytes --]

On 2024-01-10 00:34:48 +0100, Ludovic Courtès wrote:
> Tomas Volf <~@wolfsden.cz> skribis:
> 
> > On 2024-01-07 15:08:59 +0100, Ludovic Courtès wrote:
> 
> [...]
> 
> >>   ** Do not accidentally wait for Linux kernel thread completion
> >>      (<https://issues.guix.gnu.org/67132>)
> >>
> >>   In cases a PID file contained a bogus PID or one that’s only valid in a
> >>   separate PID namespace, shepherd could end up waiting for the termination of
> >>   what’s actually a Linux kernel thread, such as PID 2 (“kthreadd”).  This
> >>   situation is now recognized and avoided.
> >
> > This is great, I will not have to remember to run `modprobe -r mt7921e' before
> > each shutdown anymore.  I hope.  Looking forward to getting it in the Guix :)
> 
> D’oh, why did you have to do that?

Otherwise the shepherd would be stuck on shutdown waiting for process named

    [mt76-tx phy0]

to terminate with messages along the lines of:

    shepherd[1]: waiting for process termination (processes left: (1 678))

It is a kernel thread as far as I can tell (based on
https://stackoverflow.com/a/12231039):

    $ cd /proc/678
    $ cat cmdline
    $ readlink exe; echo $?
    1

Removing the module mt7921e stops the thread, so shepherd does not wait for it.

> How did Shepherd end up with “wrong” PID?

That I do not know.  It is visible in `ps' output, so I assume shepherd picked
it up on its own somehow.

> 
> I hope this release fixes it!

As far as I can tell, the 0.10.3 was already added into guix:

    $ ps 1 | cat
      PID TTY      STAT   TIME COMMAND
        1 ?        Sl     0:01 /gnu/store/bhynhk0c6ssq3fqqc59fvhxjzwywsjbb-guile-3.0.9/bin/guile --no-auto-compile /gnu/store/06mz0yjkghi7r6d7lmhvv7gryipljhdd-shepherd-0.10.3/bin/shepherd --config /gnu/store/klkqq2y65k141rlipq4ls0w2rlhds12h-shepherd.conf

So I have to say it sadly did not resolve this issue.  I am unsure why though.
I am not familiar with Shepherd's code base, but quick look at the git log
suggested that procedure (@@ (shepherd service) pseudo-process?) is the relevant
one.  When I try it from a REPL, it returns #t.

    $ guix shell guile shepherd guile-fibers -- guile
    GNU Guile 3.0.9
    Copyright (C) 1995-2023 Free Software Foundation, Inc.
    
    Guile comes with ABSOLUTELY NO WARRANTY; for details type `,show w'.
    This program is free software, and you are welcome to redistribute it
    under certain conditions; type `,show c' for details.
    
    Enter `,help' for help.
    scheme@(guile-user)> ,use (shepherd service)
    scheme@(guile-user)> ((@@ (shepherd service) pseudo-process?) 688)
    $1 = #t

So it *should* work?  However the issue is caused by non-free WiFi driver on a
corrupted kernel, so I am not sure if it is even problem that needs to be
solved...  I would (obviously) like to see it resolved, but I probably cannot
even bug report it, since it requires non-free hardware and software to
reproduce.

Tomas

PS: It is interesting that `guix shell guile shepherd' is not enough, the
    guile-fibers have to be explicitly specified as well.  Is that expected?

-- 
There are only two hard things in Computer Science:
cache invalidation, naming things and off-by-one errors.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: GNU Shepherd 0.10.3 released

[Tomas Volf]

[-- Attachment #1: Type: text/plain, Size: 326 bytes --]

PS:

On 2024-01-10 17:38:17 +0100, Tomas Volf wrote:
>     scheme@(guile-user)> ((@@ (shepherd service) pseudo-process?) 688)

The pid is different than above, because this was after a reboot.

Tomas

-- 
There are only two hard things in Computer Science:
cache invalidation, naming things and off-by-one errors.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: GNU Shepherd 0.10.3 released

[Ludovic Courtès]

Hello,

Tomas Volf <~@wolfsden.cz> skribis:

> Otherwise the shepherd would be stuck on shutdown waiting for process named
>
>     [mt76-tx phy0]
>
> to terminate with messages along the lines of:
>
>     shepherd[1]: waiting for process termination (processes left: (1 678))
>
> It is a kernel thread as far as I can tell (based on
> https://stackoverflow.com/a/12231039):
>
>     $ cd /proc/678
>     $ cat cmdline
>     $ readlink exe; echo $?
>     1
>
> Removing the module mt7921e stops the thread, so shepherd does not wait for it.

Ooooh.

Then I’m afraid this bug isn’t fixed yet because that code (“waiting for
process termination”) is currently in Guix, not in Shepherd.

However, ‘processes’, which is what is used here and which is defined in
(guix build syscalls), already checks for kernel threads, though it
does it differently than what I implemented in shepherd:

  (define (kernel? pid)
    "Return #t if PID designates a \"kernel thread\" rather than a normal
  user-land process."
    (let ((stat (call-with-input-file (format #f "/proc/~a/stat" pid)
                  (compose string-tokenize read-string))))
      ;; See proc.txt in Linux's documentation for the list of fields.
      (match stat
        ((pid tcomm state ppid pgrp sid tty_nr tty_pgrp flags min_flt
              cmin_flt maj_flt cmaj_flt utime stime cutime cstime
              priority nice num_thread it_real_value start_time
              vsize rss rsslim
              (= string->number start_code) (= string->number end_code) _ ...)
         ;; Got this obscure trick from sysvinit's 'killall5' program.
         (and (zero? start_code) (zero? end_code))))))

It would be great if you could check whether this approach works for
you.

(I had completely forgotten about this code.  Funny thing is this one
was inspired by sysvinit, whereas the one in Shepherd was inspired by
systemd.  A sign of times!)

Ludo’.

Re: GNU Shepherd 0.10.3 released

[Tomas Volf]

[-- Attachment #1: Type: text/plain, Size: 2505 bytes --]

On 2024-01-11 13:41:39 +0100, Ludovic Courtès wrote:
> Tomas Volf <~@wolfsden.cz> skribis:
> 
> > Otherwise the shepherd would be stuck on shutdown waiting for process named
> >
> >     [mt76-tx phy0]
> >
> > to terminate with messages along the lines of:
> >
> >     shepherd[1]: waiting for process termination (processes left: (1 678))
> >
> > It is a kernel thread as far as I can tell (based on
> > https://stackoverflow.com/a/12231039):
> >
> >     $ cd /proc/678
> >     $ cat cmdline
> >     $ readlink exe; echo $?
> >     1
> >
> > Removing the module mt7921e stops the thread, so shepherd does not wait for it.
> 
> Ooooh.
> 
> Then I’m afraid this bug isn’t fixed yet because that code (“waiting for
> process termination”) is currently in Guix, not in Shepherd.
> 
> However, ‘processes’, which is what is used here and which is defined in
> (guix build syscalls), already checks for kernel threads, though it
> does it differently than what I implemented in shepherd:
> 
>   (define (kernel? pid)
>     "Return #t if PID designates a \"kernel thread\" rather than a normal
>   user-land process."
>     (let ((stat (call-with-input-file (format #f "/proc/~a/stat" pid)
>                   (compose string-tokenize read-string))))
>       ;; See proc.txt in Linux's documentation for the list of fields.
>       (match stat
>         ((pid tcomm state ppid pgrp sid tty_nr tty_pgrp flags min_flt
>               cmin_flt maj_flt cmaj_flt utime stime cutime cstime
>               priority nice num_thread it_real_value start_time
>               vsize rss rsslim
>               (= string->number start_code) (= string->number end_code) _ ...)
>          ;; Got this obscure trick from sysvinit's 'killall5' program.
>          (and (zero? start_code) (zero? end_code))))))
> 
> It would be great if you could check whether this approach works for
> you.

Ah, that code indeed returns #f for the pid in question:

    scheme@(guix-user)> ((@@ (guix build syscalls) kernel?) 688)
    $1 = #f

The stat file:

    $ cat /proc/688/stat
    688 (mt76-tx phy0) S 2 0 0 0 -1 2129984 0 0 0 0 0 0 0 0 -2 0 1 0 964 0 0 18446744073709551615 0 0 0 0 0 0 0 2147483647 0 0 0 0 17 5 1 1 0 0 0 0 0 0 0 0 0 0 0

So the start_code is not zero (I would guess it is -1).  I have no idea what
that means though.

Tomas

-- 
There are only two hard things in Computer Science:
cache invalidation, naming things and off-by-one errors.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: GNU Shepherd 0.10.3 released

[Ludovic Courtès]

Hi,

Tomas Volf <~@wolfsden.cz> skribis:

> Ah, that code indeed returns #f for the pid in question:
>
>     scheme@(guix-user)> ((@@ (guix build syscalls) kernel?) 688)
>     $1 = #f
>
> The stat file:
>
>     $ cat /proc/688/stat
>     688 (mt76-tx phy0) S 2 0 0 0 -1 2129984 0 0 0 0 0 0 0 0 -2 0 1 0 964 0 0 18446744073709551615 0 0 0 0 0 0 0 2147483647 0 0 0 0 17 5 1 1 0 0 0 0 0 0 0 0 0 0 0
>
> So the start_code is not zero (I would guess it is -1).  I have no idea what
> that means though.

What about this method (from shepherd)?

--8<---------------cut here---------------start------------->8---
(define (linux-process-flags pid)
  "Return the process flags of @var{pid} (or'd @code{PF_} constants), assuming
the Linux /proc file system is mounted; raise a @code{system-error} exception
otherwise."
  (call-with-input-file (string-append "/proc/" (number->string pid)
                                       "/stat")
    (lambda (port)
      (define line
        (get-string-all port))

      ;; Parse like systemd's 'is_kernel_thread' function.
      (let ((offset (string-index line #\))))     ;offset past 'tcomm' field
        (match (and offset
                    (string-tokenize (string-drop line (+ offset 1))))
          ((state ppid pgrp sid tty-nr tty-pgrp flags . _)
           (or (string->number flags) 0))
          (_
           0))))))

;; Per-process flag defined in <linux/sched.h>.
(define PF_KTHREAD #x00200000)                    ;I am a kernel thread

(define (linux-kernel-thread? pid)
  "Return true if @var{pid} is a Linux kernel thread."
  (= PF_KTHREAD (logand (linux-process-flags pid) PF_KTHREAD)))
--8<---------------cut here---------------end--------------->8---

If it works better, we can use that in syscalls.scm as well.

Ludo’.

PS: Having an entry in bug-guix would ensure we don’t lose track of this.

Re: GNU Shepherd 0.10.3 released

[Wilko Meyer]

Hi Ludo,

Ludovic Courtès <ludo@gnu.org> writes:

> We are pleased to announce the GNU Shepherd version 0.10.3, a bug-fix
> release of the new 0.10.x series, representing 51 commits over 6 months.

Congratulations on the release & thanks to everyone that have worked on
this for their work!

>   ** New #:respawn-delay parameter to ‘service’
>      (<https://issues.guix.gnu.org/64665>)
>
>   This specifies a delay before a service is respawned.  Its default value is
>   given by ‘default-respawn-delay’ and defaults to 100ms.  Until now, services
>   were respawned immediately.

I've a couple of services defined that benefits from defining a respawn
delay, thanks for this!

>   ** Fix portability issues to GNU/Hurd
>
>   Previous versions in the 0.10.x and 0.9.x series did not work on GNU/Hurd.
>   This is now fixed, although some features are still implemented in a
>   suboptimal way.

This also sounds great, always happy to read news about the hurd.

-- 
Kind regards,

Wilko Meyer
w@wmeyer.eu