The Shepherd on Fibers

The Shepherd on Fibers

[Ludovic Courtès]

Hello Guix!

I have pushed a ‘wip-fibers’ branch of the Shepherd:

  https://git.savannah.gnu.org/cgit/shepherd.git/log/?h=wip-fibers

The goal is to make shepherd (the daemon) use Fibers¹ for concurrency.

Right now, actions in shepherd are all serialized: starting services,
waiting for their PID files, accepting client connections and serving
them (when running the ‘herd’ command), etc.  This slows down startup,
prevents things like running two clients at the same time, or serving a
client while waiting for a PID file.  In other words, it sucks.

Using Fibers, we can introduce concurrency with few changes to the code;
we can even do new things writing code that looks sequential.  This is
what the branch does.

To illustrate that, it introduces a new, incredible feature: logging!
As you know, shepherd had (euphemism ahead) limited support for logging,
essentially in the form of #:log-file, which would redirect a daemon’s
stdout/stderr to a file, and also in the form of “let’s hope the daemon
talks to syslogd”.  Here I was able to trivially add logging, such that
anything spawned by ‘fork+exec-command’ is logged, with timestamps and
all (the logger is a fiber that calls ‘read-line’ in a loop, which is
automagically non-blocking—delightful!).  The next fun step (besides
logging) will be adding inetd-style and/or systemd-style “socket
activation”.

Fibers is used in a single-threaded fashion, which is the main
constraint for shepherd since it forks.  That also means that fibers
cannot be preempted, so it’s fully cooperative scheduling.

There’s one catch: Fibers is currently Linux-only.  The good news is
that work has been done to port it to other kernels via libevent².
Until it is merged, we could keep using the Shepherd 0.8 on GNU/Hurd.

I’ve done some Guix System testing in VMs and didn’t notice any major
issues.  I’d like to merge that branch in ‘master’ and to eventually
release it as 0.9.0 (with or without socket activation, we’ll see.)
Hopefully, we could be running it within a couple of weeks.

Thoughts?

Ludo’.

¹ https://github.com/wingo/fibers
² https://github.com/wingo/fibers/pull/53

Re: The Shepherd on Fibers

[Maxim Cournoyer]

Hi Ludo!

Ludovic Courtès <ludo@gnu.org> writes:

> Hello Guix!
>
> I have pushed a ‘wip-fibers’ branch of the Shepherd:
>
>   https://git.savannah.gnu.org/cgit/shepherd.git/log/?h=wip-fibers
>
> The goal is to make shepherd (the daemon) use Fibers¹ for concurrency.

Very exciting work!  I can't wait to have my 5 min boot reduced (mostly
busy-waiting on the child-hurd VM to start) ;-).

> To illustrate that, it introduces a new, incredible feature: logging!
> As you know, shepherd had (euphemism ahead) limited support for logging,
> essentially in the form of #:log-file, which would redirect a daemon’s
> stdout/stderr to a file, and also in the form of “let’s hope the daemon
> talks to syslogd”.  Here I was able to trivially add logging, such that
> anything spawned by ‘fork+exec-command’ is logged, with timestamps and
> all (the logger is a fiber that calls ‘read-line’ in a loop, which is
> automagically non-blocking—delightful!).  The next fun step (besides
> logging) will be adding inetd-style and/or systemd-style “socket
> activation”.

Interesting; in a (unmerged) patch I had made to mcron, I had extended
the select loop in a way that it'd detect available output and use a
suspendable port to accomplish this; perhaps Fibers would be a
cleaner/more straightforward way to accomplish this.  Also, I was
particularly happy about how flexible the timestamp/metadata prefixed to
each line could be configured by a user; you may want to steal ideas
from it! [0]

[0]  https://lists.gnu.org/archive/html/bug-mcron/2021-08/msg00008.html

> Fibers is used in a single-threaded fashion, which is the main
> constraint for shepherd since it forks.  That also means that fibers
> cannot be preempted, so it’s fully cooperative scheduling.

I see.  I was puzzle by this configure.ac check:

--8<---------------cut here---------------start------------->8---
+dnl Check for extra dependencies.
+GUILE_MODULE_AVAILABLE([have_fibers], [(fibers)])
+if test "x$have_fibers" != "xyes"; then
+  AC_MSG_ERROR([Fibers is missing; please install it.])
+fi
+
+dnl Make sure Fibers does not create POSIX threads: since shepherd
+dnl forks, it must be single-threaded.
+AC_CACHE_CHECK([whether Fibers might create POSIX threads],
+  [ac_cv_fibers_creates_pthreads],
+  [GUILE_CHECK([retval],
+    [(use-modules (fibers))
+     (set! (@ (ice-9 threads) call-with-new-thread)
+       (lambda _ (throw 'new-thread!)))
+     (run-fibers (lambda () (spawn-fiber (lambda () 1)))
+                 #:parallelism 1 #:hz 0)])
+   if test "$retval" = 0; then
+     ac_cv_fibers_creates_pthreads="no"
+   else
+     ac_cv_fibers_creates_pthreads="yes"
+   fi])
+if test "x$ac_cv_fibers_creates_pthreads" = "xyes"; then
+  AC_MSG_ERROR([Fibers creates POSIX threads behind our back; aborting.])
+fi
--8<---------------cut here---------------end--------------->8---

In which scenario would Fibers create unwanted POSIX threads?  Is it
that older versions of Fibers had that problem?  If so, the message
could be more explicit (Fibers too old, you need Fibers >= N).

> There’s one catch: Fibers is currently Linux-only.  The good news is
> that work has been done to port it to other kernels via libevent².
> Until it is merged, we could keep using the Shepherd 0.8 on GNU/Hurd.

Sounds promising.

> I’ve done some Guix System testing in VMs and didn’t notice any major
> issues.  I’d like to merge that branch in ‘master’ and to eventually
> release it as 0.9.0 (with or without socket activation, we’ll see.)
> Hopefully, we could be running it within a couple of weeks.
>
>
> Thoughts?

Nice!

Here's a very naive review (I'm not yet comfortable with the continuation
barriers and others Fibers specifics):

1. In fae59fb * build: Capture the source and object directories of
Fibers.

Why must we even capture the Fibers source/compiled objects directories?
Can't we let Guile and GUILE_LOAD_PATH (or the FHS) handle this for us?
Messing with the load path system dynamically directly is usually a code
smell, in my book.

2. nitpick; in "service: 'read-pid-file' no longer blocks":

--8<---------------cut here---------------start------------->8---
-;; Copyright (C) 2013, 2014, 2015, 2016, 2017, 2018, 2019, 2020, 2021 Ludovic Courtès <ludo@gnu.org>
+;; Copyright (C) 2013-2022 Ludovic Courtès <ludo@gnu.org>
--8<---------------cut here---------------end--------------->8---

While I agree this reads better, I've recently stumbled in info
'(maintain) Copyright Notices' on:

      You can use a range (‘2008-2010’) instead of listing individual years
   (‘2008, 2009, 2010’) if and only if: 1) every year in the range,
   inclusive, really is a “copyrightable” year that would be listed
   individually; _and_ 2) you make an explicit statement in a ‘README’ file
   about this usage.

So you'd want to add an explicit statement in the README about it.

3. I was a bit skeptical about the ability to configure the encoding of
the log file; in which situation would switching it to something else
than UTF-8 be useful?

Thanks!

I look forward to see this in mainline.

Maxim

Re: The Shepherd on Fibers

[Ludovic Courtès]

Hi Maxim!

Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:

>> The goal is to make shepherd (the daemon) use Fibers¹ for concurrency.
>
> Very exciting work!  I can't wait to have my 5 min boot reduced (mostly
> busy-waiting on the child-hurd VM to start) ;-).

Same here!  The childhurd service will need some help as it’s currently
written in a way that would block.

> Interesting; in a (unmerged) patch I had made to mcron, I had extended
> the select loop in a way that it'd detect available output and use a
> suspendable port to accomplish this; perhaps Fibers would be a
> cleaner/more straightforward way to accomplish this.  Also, I was
> particularly happy about how flexible the timestamp/metadata prefixed to
> each line could be configured by a user; you may want to steal ideas
> from it! [0]
>
> [0]  https://lists.gnu.org/archive/html/bug-mcron/2021-08/msg00008.html

Oh nice!  I hope that patch will eventually make it into mcron (or we
could use our own version!).

For the timestamp format there’s currently a parameter, which is not
exposed, and defaults to syslog style.

> I see.  I was puzzle by this configure.ac check:
>
> +dnl Check for extra dependencies.
> +GUILE_MODULE_AVAILABLE([have_fibers], [(fibers)])
> +if test "x$have_fibers" != "xyes"; then
> +  AC_MSG_ERROR([Fibers is missing; please install it.])
> +fi
> +
> +dnl Make sure Fibers does not create POSIX threads: since shepherd
> +dnl forks, it must be single-threaded.
> +AC_CACHE_CHECK([whether Fibers might create POSIX threads],
> +  [ac_cv_fibers_creates_pthreads],
> +  [GUILE_CHECK([retval],
> +    [(use-modules (fibers))
> +     (set! (@ (ice-9 threads) call-with-new-thread)
> +       (lambda _ (throw 'new-thread!)))
> +     (run-fibers (lambda () (spawn-fiber (lambda () 1)))
> +                 #:parallelism 1 #:hz 0)])
> +   if test "$retval" = 0; then
> +     ac_cv_fibers_creates_pthreads="no"
> +   else
> +     ac_cv_fibers_creates_pthreads="yes"
> +   fi])
> +if test "x$ac_cv_fibers_creates_pthreads" = "xyes"; then
> +  AC_MSG_ERROR([Fibers creates POSIX threads behind our back; aborting.])
> +fi
>
>
> In which scenario would Fibers create unwanted POSIX threads?

It shouldn’t happen with #:parallelism 1 #:hz 0.  Here I’m just being
super defensive: it’s PID 1 after all, and if future Fibers versions
break that assumption for some reason, I’d rather detect it early.

> 1. In fae59fb * build: Capture the source and object directories of
> Fibers.
>
> Why must we even capture the Fibers source/compiled objects directories?

We don’t have to, but if we don’t do it here, then the ‘shepherd’
package definition will have to use ‘wrap-program’ to do it for us.
Since there’s just one dependency, I thought we might as well do it
upstream.

> 2. nitpick; in "service: 'read-pid-file' no longer blocks":
>
> -;; Copyright (C) 2013, 2014, 2015, 2016, 2017, 2018, 2019, 2020, 2021 Ludovic Courtès <ludo@gnu.org>
> +;; Copyright (C) 2013-2022 Ludovic Courtès <ludo@gnu.org>
>
> While I agree this reads better, I've recently stumbled in info
> '(maintain) Copyright Notices' on:
>
>       You can use a range (‘2008-2010’) instead of listing individual years
>    (‘2008, 2009, 2010’) if and only if: 1) every year in the range,
>    inclusive, really is a “copyrightable” year that would be listed
>    individually; _and_ 2) you make an explicit statement in a ‘README’ file
>    about this usage.
>
> So you'd want to add an explicit statement in the README about it.

Yes, will do (it’s typical FSF pedantry; I remember the time it took
before this anecdotal-looking paragraph would end up in that document,
as if software freedom was at stake :-)).

> 3. I was a bit skeptical about the ability to configure the encoding of
> the log file; in which situation would switching it to something else
> than UTF-8 be useful?

I agree; the log files produced by shepherd are always UTF-8.

What’s configurable is the encoding of the data produced by daemons.
It’s probably rarely useful, but if you have a daemon that writes, say,
ISO-8859-1 strings, you can say so.  That way, the output of that daemon
is properly decoded as ISO-8859-1; the log file produced by shepherd
remains UTF-8 no matter what.

Thanks for taking a look!

Since that initial message I added support for inetd-style service
startup.  Among the services I use, it could be use for sshd (OpenSSH),
bitlbee, and dicod.  More on that later…

Ludo’.

Re: The Shepherd on Fibers

[pelzflorian (Florian Pelz)]

Hi Ludo!  That you add fibers to Shepherd is great news.

On Wed, Mar 23, 2022 at 11:36:30PM +0100, Ludovic Courtès wrote:
> Fibers is used in a single-threaded fashion, which is the main
> constraint for shepherd since it forks.  That also means that fibers
> cannot be preempted, so it’s fully cooperative scheduling.

To understand what you mean, I needed to read shepherd commit
934204fbd158aa9fbd42914b89aa960f99eef125

+      ;; Run Fibers in such a way that it does not create any POSIX thread,
+      ;; because POSIX threads and 'fork' cannot be used together

So shepherd service authors still cannot write blocking code but need
to yield?


> I’ve done some Guix System testing in VMs and didn’t notice any major
> issues.  I’d like to merge that branch in ‘master’ and to eventually
> release it as 0.9.0 (with or without socket activation, we’ll see.)
> Hopefully, we could be running it within a couple of weeks.

I wanted to test too how much time it takes to boot to GDM on my slow
2010 dual-core Macbook, but I can’t quite figure out each shepherd-
service failing to build.  How did you test the new shepherd in Guix?

florian@florianmacbook ~/git/guix [env]$ gzip -cd /var/log/guix/drvs/f3/g9drzlg8vkzp6z81dcdsnfnaqa2anx-shepherd-avahi-daemon.go.drv.gz 
Backtrace:
          16 (primitive-load "/gnu/store/4qk3x8dl8hxnwry637kq4ahh2z2?")
In ice-9/eval.scm:
    619:8 15 (_ #(#<directory (guile-user) 7ffff6fdec80> #<module ?>))
    159:9 14 (_ #(#<directory (guile-user) 7ffff6fdec80> #<module ?>))
In ice-9/boot-9.scm:
  3326:17 13 (resolve-interface (shepherd service) #:select _ #:hide ?)
In ice-9/threads.scm:
    390:8 12 (_ _)
In ice-9/boot-9.scm:
  3252:13 11 (_)
In ice-9/threads.scm:
    390:8 10 (_ _)
In ice-9/boot-9.scm:
  3536:20  9 (_)
   2835:4  8 (save-module-excursion #<procedure 7ffff5f4ea50 at ice-?>)
  3556:26  7 (_)
In unknown file:
           6 (primitive-load-path "shepherd/service" #<procedure 7ff?>)
In shepherd/service.scm:
     26:0  5 (_)
In ice-9/boot-9.scm:
   3409:4  4 (define-module* _ #:filename _ #:pure _ #:version _ # _ ?)
  2594:24  3 (call-with-deferred-observers #<procedure 7ffff5f3b410 ?>)
  3422:24  2 (_)
   222:17  1 (map1 (((fibers)) ((fibers scheduler) #:select (#)) # ?))
   3329:6  0 (resolve-interface (fibers) #:select _ #:hide _ #:prefix ?)

ice-9/boot-9.scm:3329:6: In procedure resolve-interface:
no code for module (fibers)



Anyway, testing my boot time is not so important.  I’d probably have
to change something in gnu/services/shepherd.scm.  Or I should rebuild
shepherd in a less crude way.

Feel free to ignore me, though for completeness, here’s what I did:

* Pull shepherd wip-fibers commit
  abbdd1e8b0f8bd4b3952efc5614f560aab9a79bf.

* I changed its Makefile.am and added -O1 to '$(GUILD) compile'
  because Guix normally does likewise in an origin snippet.

* Then `guix shell -D shepherd autoconf automake guile-fibers help2man
  texinfo -- sh -c 'autoreconf -vif && ./configure
  --prefix=/home/florian/git/shepherd/out && make && make install'`.

* Then to Guix I added to gnu/packages/admin.scm:

(define-public new-shepherd
  (package
    (name "shepherd")
    (version "0.9.0")
    (source (local-file "/home/florian/git/shepherd/out" #:recursive? #t))
    (build-system copy-build-system)
    (propagated-inputs (list guile-fibers))
    (arguments
       `(#:install-plan '(("./" ""))))
    …)) ;; and synopsis, description, license, homepage

and at the top #:use-module (gnu packages guile-xyz) to add
guile-fibers to new-shepherd’s propagated inputs.  Also #:use-module
(guix build-system copy).

* Lastly I grafted

(define-public shepherd
  (package
    (name "shepherd")
    (replacement new-shepherd)
    …

* Compiled Guix and reconfigured.  Building
  /gnu/store/f3g9drzlg8vkzp6z81dcdsnfnaqa2anx-shepherd-avahi-daemon.go.drv
  failed.

The same error happens when using new-shepherd as shepherd and
rebuilding the GNOME world.

Regards,
Florian

Re: The Shepherd on Fibers

[Zhu Zihao]

[-- Attachment #1: Type: text/plain, Size: 407 bytes --]

> So shepherd service authors still cannot write blocking code but need
> to yield?

IMO, service should not block service, if they have something important
to do before running a program, why not fork first and do it in the
child-process (maybe wrap the actual startup program in Guile script)?
-- 
Retrieve my PGP public key:

  gpg --recv-keys D47A9C8B2AE3905B563D9135BE42B352A9F6821F

Zihao

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 255 bytes --]

Re: The Shepherd on Fibers

[Zhu Zihao]

[-- Attachment #1: Type: text/plain, Size: 572 bytes --]


Zhu Zihao <all_but_last@163.com> writes:

> [[PGP Signed Part:Undecided]]
>> So shepherd service authors still cannot write blocking code but need
>> to yield?
>
> IMO, service should not block service, if they have something important
> to do before running a program, why not fork first and do it in the
> child-process (maybe wrap the actual startup program in Guile script)?

service should not block service
                         ^^^^^^^ daemon
-- 
Retrieve my PGP public key:

  gpg --recv-keys D47A9C8B2AE3905B563D9135BE42B352A9F6821F

Zihao

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 255 bytes --]

Re: The Shepherd on Fibers

[pelzflorian (Florian Pelz)]

On Sat, Mar 26, 2022 at 07:09:12PM +0800, Zhu Zihao wrote:
> > So shepherd service authors still cannot write blocking code but need
> > to yield?
> 
> IMO, service should not block service, if they have something important
> to do before running a program, why not fork first and do it in the
> child-process (maybe wrap the actual startup program in Guile script)?

In other words, all stays the same as without fibers?

Regards,
Florian

Re: The Shepherd on Fibers

[Zhu Zihao]

[-- Attachment #1: Type: text/plain, Size: 1138 bytes --]


IIUC, After fork and execute the program, shepherd will wait for the pid
file to appear. This is a block operation.

Shepherd on Fiber make this operation non-blocking (because the pid file
is created by Linux kernel, Shepherd just need to wait for it), every
service activation runs on its own fiber. after fork+exec, shepherd
suspend current fiber, andspawn another fiber to start another service
immediately.

But finally all fibers are co-operatively scheduled on 1 thread.

"pelzflorian (Florian Pelz)" <pelzflorian@pelzflorian.de> writes:

> On Sat, Mar 26, 2022 at 07:09:12PM +0800, Zhu Zihao wrote:
>> > So shepherd service authors still cannot write blocking code but need
>> > to yield?
>> 
>> IMO, service should not block service, if they have something important
>> to do before running a program, why not fork first and do it in the
>> child-process (maybe wrap the actual startup program in Guile script)?
>
> In other words, all stays the same as without fibers?
>
> Regards,
> Florian


-- 
Retrieve my PGP public key:

  gpg --recv-keys D47A9C8B2AE3905B563D9135BE42B352A9F6821F

Zihao

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 255 bytes --]

Re: The Shepherd on Fibers

[pelzflorian (Florian Pelz)]

On Sat, Mar 26, 2022 at 07:27:00PM +0800, Zhu Zihao wrote:
> 
> IIUC, After fork and execute the program, shepherd will wait for the pid
> file to appear. This is a block operation.
> 
> Shepherd on Fiber make this operation non-blocking (because the pid file
> is created by Linux kernel, Shepherd just need to wait for it), every
> service activation runs on its own fiber. after fork+exec, shepherd
> suspend current fiber, andspawn another fiber to start another service
> immediately.
> 
> But finally all fibers are co-operatively scheduled on 1 thread.

Thank you Zhu for explaining!  I had not understood waiting for the
PID file is the issue solved by fibers.

Regards,
Florian

Re: The Shepherd on Fibers

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 690 bytes --]

pelzflorian (Florian Pelz) schreef op za 26-03-2022 om 12:06 [+0100]:
> * Lastly I grafted
> 
> (define-public shepherd
>   (package
>     (name "shepherd")
>     (replacement new-shepherd)
>     …

Grafting is not necessary.  See (guix)Shepherd Services:


       ;; Use own Shepherd package.
       (essential-services
        (modify-services (operating-system-default-essential-services
                          this-operating-system)
          (shepherd-root-service-type config => (shepherd-configuration
                                                 (inherit config)
                                                 (shepherd my-
shepherd))))))


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

Re: The Shepherd on Fibers

[pelzflorian (Florian Pelz)]

On Sat, Mar 26, 2022 at 12:59:20PM +0100, Maxime Devos wrote:
> Grafting is not necessary.  See (guix)Shepherd Services:
> 
> 
>        ;; Use own Shepherd package.
>        (essential-services
>         (modify-services (operating-system-default-essential-services
>                           this-operating-system)
>           (shepherd-root-service-type config => (shepherd-configuration
>                                                  (inherit config)
>                                                  (shepherd my-
> shepherd))))))
> 

Oh yes, I had not tried that.  Thank you Maxime.  Still with unchanged
original shepherd package, I get exactly the same
shepherd-avahi-daemon.go.drv.gz build failure iff I set (shepherd
new-shepherd).  Therefore I ask how Ludo or perhaps you tested it.
Perhaps you have a proper new-shepherd package definition.

Regards,
Florian

Re: The Shepherd on Fibers

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 1175 bytes --]

pelzflorian (Florian Pelz) schreef op za 26-03-2022 om 17:52 [+0100]:
> Oh yes, I had not tried that.  Thank you Maxime.  Still with unchanged
> original shepherd package, I get exactly the same
> shepherd-avahi-daemon.go.drv.gz build failure iff I set (shepherd
> new-shepherd).  Therefore I ask how Ludo or perhaps you tested it.
> Perhaps you have a proper new-shepherd package definition.

I have not tested the wip-fibers variant of Shepherd with Guix.
However, looking at scm->go (the procudure resonsible for compiling)
in (gnu services shepherd):

    (with-extensions (list shepherd) [...])

, I'm wondering if the 'propagated-inputs' of 'shepherd' are respected
here.  Looking at (guix gexp), I don't see any mention of propagation
anywhere, so I guess not (unverified).  Maybe that's why (fibers) could
not be found?

For completeness, you could try inheriting from 'shepherd' and using
git-reference/git-checkout instead of the copy-build-system

  (package
    (inherit shepherd)
    (source (git-checkout (url "/home/florian/...") (commit ...))))

which would be closer to how shepherd is usually compiled.

Greetings,
Maxime.


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

Re: The Shepherd on Fibers

[Ludovic Courtès]

Hi!

"pelzflorian (Florian Pelz)" <pelzflorian@pelzflorian.de> skribis:

> On Wed, Mar 23, 2022 at 11:36:30PM +0100, Ludovic Courtès wrote:
>> Fibers is used in a single-threaded fashion, which is the main
>> constraint for shepherd since it forks.  That also means that fibers
>> cannot be preempted, so it’s fully cooperative scheduling.
>
> To understand what you mean, I needed to read shepherd commit
> 934204fbd158aa9fbd42914b89aa960f99eef125
>
> +      ;; Run Fibers in such a way that it does not create any POSIX thread,
> +      ;; because POSIX threads and 'fork' cannot be used together
>
> So shepherd service authors still cannot write blocking code but need
> to yield?

Yes.  Technically though, “yielding” is transparent: any time you do I/O
(‘get-bytevector-n’, etc.) or call ‘sleep’ (specifically the one
provided by Fibers), your code potentially yields, letting Fibers
schedule some other activity.

What that means in practice is that starting services, waiting for PID
files, serving clients—all this happens concurrently.

It *is* possible to write service code that would block everything, but
with little or no effort, you can make it cooperate with the scheduler.

HTH,
Ludo’.

Re: The Shepherd on Fibers

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 617 bytes --]

Ludovic Courtès schreef op wo 23-03-2022 om 23:36 [+0100]:
> Fibers is used in a single-threaded fashion, which is the main
> constraint for shepherd since it forks.  That also means that fibers
> cannot be preempted, so it’s fully cooperative scheduling.

When hz!=0, guile-fibers uses with-interrupts/thread-cputime which uses
threads.  However, when (provided? 'threads) is false, it uses
'with-interrupts/sigprof', which uses signal handlers and SIGPROF
instead.  So with some small tweaks to guile-fibers, it should be
feasible to have singly-threaded, preemptive scheduling.

Greetings,
Maxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

Re: The Shepherd on Fibers

[Ludovic Courtès]

Maxime Devos <maximedevos@telenet.be> skribis:

> Ludovic Courtès schreef op wo 23-03-2022 om 23:36 [+0100]:
>> Fibers is used in a single-threaded fashion, which is the main
>> constraint for shepherd since it forks.  That also means that fibers
>> cannot be preempted, so it’s fully cooperative scheduling.
>
> When hz!=0, guile-fibers uses with-interrupts/thread-cputime which uses
> threads.  However, when (provided? 'threads) is false, it uses
> 'with-interrupts/sigprof', which uses signal handlers and SIGPROF
> instead.  So with some small tweaks to guile-fibers, it should be
> feasible to have singly-threaded, preemptive scheduling.

Right.  Good to know we can take that route if it turns out to be
necessary.

Ludo’.

Re: The Shepherd on Fibers

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 239 bytes --]

Ludovic Courtès schreef op wo 23-03-2022 om 23:36 [+0100]:

> [...]
> Thoughts?

To help translators, I would use positional arguments in

+ (l10n "Accepted connection on ~a from ~:[~a~;~*local process~].")

Greetings,
Maxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

Re: The Shepherd on Fibers

[Ludovic Courtès]

Maxime Devos <maximedevos@telenet.be> skribis:

> To help translators, I would use positional arguments in
>
> + (l10n "Accepted connection on ~a from ~:[~a~;~*local process~].")

Hmm so how would you write this?

Re: The Shepherd on Fibers

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 1031 bytes --]

Ludovic Courtès schreef op di 29-03-2022 om 14:48 [+0200]:
> Maxime Devos <maximedevos@telenet.be> skribis:
> 
> > To help translators, I would use positional arguments in
> > 
> > + (l10n "Accepted connection on ~a from ~:[~a~;~*local process~].")
> 
> Hmm so how would you write this?

Here's an example on how to implement positional arguments in terms of
(ice-9 format) argument jumping:

(format #t "third argument: ~2@*~a, second argument: ~1@*~a, first
argument ~0@*~a!~%" 1 2 3)

(format #t "Accepted connection on ~0@*~a from ~1@*~:[~0@*~a~;local
process~]." "foo" #f "bar")

(format #t "From ~1@*~:[~0@*~a~;local process~] accepted connection on
~0@*~a." "foo" #f "bar")

Or simpler, without format string conditionals, which seems a bit
easier to translate without having to know how (ice-9 format) works:

(if local-process?
    (format #t "Accepted connection on ~a from local process." "foo")
    (format #t "Accepted connection on ~0@*~a from ~1@*~a." "foo" "bar"))

Greetings,
Maxime

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

Re: The Shepherd on Fibers

[Ludovic Courtès]

Maxime Devos <maximedevos@telenet.be> skribis:

>     (format #t "Accepted connection on ~0@*~a from ~1@*~a." "foo" "bar"))

Hmm that doesn’t seem to work:

--8<---------------cut here---------------start------------->8---
scheme@(guile-user)> ,use(ice-9 format)
scheme@(guile-user)> (format #f "Accepted connection on ~@0*~a from ~@1*~a." 1 2)

FORMAT: error with call: (format #f "Accepted connection on ~@0<===*~a from ~@1*~a." ===>1 2 )
        misplaced modifier
FORMAT: INTERNAL ERROR IN FORMAT-ERROR!
        destination: #f
        format string: "Accepted connection on ~@0*~a from ~@1*~a."
        format args: (1 2)
        error args:  (#f "error in format" () #f)
ice-9/boot-9.scm:1685:16: In procedure raise-exception:
error in format
--8<---------------cut here---------------end--------------->8---

Keeping it as future work…

Ludo’.

Re: The Shepherd on Fibers

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 562 bytes --]

Ludovic Courtès schreef op wo 30-03-2022 om 17:14 [+0200]:
> >      (format #t "Accepted connection on ~0@*~a from ~1@*~a." "foo"
> > "bar"))
> 
> Hmm that doesn’t seem to work:
> 
> --8<---------------cut here---------------start------------->8---
> scheme@(guile-user)> ,use(ice-9 format)
> scheme@(guile-user)> (format #f "Accepted connection on ~@0*~a from
> ~@1*~a." 1 2)

You are writing ~@0*~a. However, the number must be before the @:
~0@*~a. If the number is moved between the ~ and @, then it works for
me.

Greetings,
Maxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

Re: The Shepherd on Fibers

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 258 bytes --]

Ludovic Courtès schreef op wo 23-03-2022 om 23:36 [+0100]:
> Thoughts?

What if 'exec-command' in 'fork+exec-command' fails?  Is it still the
case that the exception bubbles up, eventually causing the socket file
to be deleted?

Greetings,
Maxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

Re: The Shepherd on Fibers

[Ludovic Courtès]

Maxime Devos <maximedevos@telenet.be> skribis:

> What if 'exec-command' in 'fork+exec-command' fails?  Is it still the
> case that the exception bubbles up, eventually causing the socket file
> to be deleted?

Which socket file?

(In practice ‘exec-command’ would only fail if the program cannot be
found.)

Ludo’.

Re: The Shepherd on Fibers

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 188 bytes --]

Ludovic Courtès schreef op di 29-03-2022 om 14:50 [+0200]:
> Which socket file?

The socket created by 'open-server-socket', used by 'herd stop' and the
like.

Greetings,
Maxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

Re: The Shepherd on Fibers

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 656 bytes --]

Ludovic Courtès schreef op di 29-03-2022 om 14:50 [+0200]:
> Maxime Devos <maximedevos@telenet.be> skribis:
> 
> > What if 'exec-command' in 'fork+exec-command' fails?  Is it still the
> > case that the exception bubbles up, eventually causing the socket file
> > to be deleted?
> 
> Which socket file?
> 
> (In practice ‘exec-command’ would only fail if the program cannot be
> found.)

Attila Lendvai has encountered the socket file to be deleted before,
this seemed to have been the cause and I have seen a few bug reports
(about tor?) that might have had the same cause.  I don't have a bug
number at hand.

Greetings,
Maxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

Re: The Shepherd on Fibers

[Attila Lendvai]

> > Which socket file?
> >
> > (In practice ‘exec-command’ would only fail if the program cannot be
> > found.)
>
> Attila Lendvai has encountered the socket file to be deleted before,
> this seemed to have been the cause and I have seen a few bug reports
> (about tor?) that might have had the same cause. I don't have a bug
> number at hand.

my first ever service has non-trivial work to do in the start GEXP,
and unsurprisingly, it has often ended up throwing an exception while
i subled along my learning curve. it's not caught by shepherd right
around the call to the start GEXP, and reaches a handler up higher
that ends up deleting the socket (and IIRC skipping the start of rest
of the services).

it's on my TODO to harden the error handling in shepherd once the
shepherd-for-guix patch is in, or something equivalent that shortens
the edit-compile-test cycle of shepherd.

but obviously, i don't mind if it gets fixed meanwhile by the open
source fairies... :)

--
• attila lendvai
• PGP: 963F 5D5F 45C7 DFCD 0A39
--
“If we could read the secret history of our enemies, we should find in each man's life sorrow and suffering enough to disarm all hostility.”
	— Henry Wadsworth Longfellow (1807–1901)

Re: The Shepherd on Fibers

[Ludovic Courtès]

Hi,

Attila Lendvai <attila@lendvai.name> skribis:

> my first ever service has non-trivial work to do in the start GEXP,
> and unsurprisingly, it has often ended up throwing an exception while
> i subled along my learning curve. it's not caught by shepherd right
> around the call to the start GEXP, and reaches a handler up higher
> that ends up deleting the socket (and IIRC skipping the start of rest
> of the services).

Ah ha!  I guess that can happen if the service is started right from the
config file; it cannot happen if the service is started via ‘herd start’
(because all the ‘start’ exceptions are caught and reported early), or
if it’s started from the config file but in a separate fiber with
‘start-in-the-background’.

Ludo’.

Re: The Shepherd on Fibers

[adriano]

Il giorno mar, 29/03/2022 alle 15.29 +0000, Attila Lendvai ha scritto:
> > > Which socket file?
> > > 
> > > (In practice ‘exec-command’ would only fail if the program cannot
> > > be
> > > found.)
> > 
> > Attila Lendvai has encountered the socket file to be deleted
> > before,
> > this seemed to have been the cause and I have seen a few bug
> > reports
> > (about tor?) that might have had the same cause. I don't have a bug
> > number at hand.
> 
> my first ever service has non-trivial work to do in the start GEXP,
> and unsurprisingly, it has often ended up throwing an exception while
> i subled along my learning curve. it's not caught by shepherd right
> around the call to the start GEXP, and reaches a handler up higher
> that ends up deleting the socket (and IIRC skipping the start of rest
> of the services).
> 
> it's on my TODO to harden the error handling in shepherd once the
> shepherd-for-guix patch is in, or something equivalent that shortens
> the edit-compile-test cycle of shepherd.

What's the shepherd-for-guix patch ? What does it bring to Shepherd ?

Re: The Shepherd on Fibers

[Attila Lendvai]

> What's the shepherd-for-guix patch ? What does it bring to Shepherd ?

https://issues.guix.gnu.org/54216

it basically introduces a shepherd-next package. it's one way to
greatly shorten the edit-compile-test cycle of shepherd within a guix
system vm.

in the current setup, changing the shepherd package leads to hours
long recompilation due to the dependencies.

--
• attila lendvai
• PGP: 963F 5D5F 45C7 DFCD 0A39
--
“Goodbyes are only for those who love with their eyes. Because for those who love with heart and soul there is no such thing as separation.”
	— Rumi (1207–1273)

Re: The Shepherd on Fibers

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 1512 bytes --]

Ludovic Courtès schreef op wo 23-03-2022 om 23:36 [+0100]:
Thoughts?

From service.scm:

> +  (define (sleep* n)
> +    ;; In general we want to use (@ (fibers) sleep) to yield to the scheduler.
> +    ;; However, this code might be non-suspendable--e.g., if the user calls
> +    ;; the 'start' method right from their config file, which is loaded with
> +    ;; 'primitive-load', which is a continuation barrier.  Thus, this variant
> +    ;; checks whether it can suspend and picks the right 'sleep'.
> +    (if (yield-current-task)
> +        (begin
> +          (set! sleep* (@ (fibers) sleep))
> +          (sleep n))
> +        (begin
> +          (set! sleep* (@ (guile) sleep))
> +          ((@ (guile) sleep) n))))

Instead of working around this limitation of Fiber's 'sleep',
why not modify Fiber's sleep to detect if it is from a suspendable
context or not, and depending on that, use Guile's sleep/usleep or
the perform-operation+sleep-operation?

Additionally, the set! can be avoided here in favour of a loop variable:

(define start ...)
(define (sleep n)
  ;; In general [...]
  (if (yield-current-task)
      (begin ((@ (fibers) sleep) n) ((@ (fibers) sleep) n))
      (begin ((@ (guile) sleep) n) ((@ (guile) sleep) n))))
(let loop ((sleep sleep))
  (define (try-again)
     ...
     
     (loop (sleep 1)))
  (catch ...))

FWIW, the delay can be decreased here, if Guile's sleep is replaced
by an appropriate use of 'usleep'.

Greetings,
Maxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

Re: The Shepherd on Fibers

[Ludovic Courtès]

Maxime Devos <maximedevos@telenet.be> skribis:

> Ludovic Courtès schreef op wo 23-03-2022 om 23:36 [+0100]:
> Thoughts?
>
> From service.scm:
>
>> +  (define (sleep* n)
>> +    ;; In general we want to use (@ (fibers) sleep) to yield to the scheduler.
>> +    ;; However, this code might be non-suspendable--e.g., if the user calls
>> +    ;; the 'start' method right from their config file, which is loaded with
>> +    ;; 'primitive-load', which is a continuation barrier.  Thus, this variant
>> +    ;; checks whether it can suspend and picks the right 'sleep'.
>> +    (if (yield-current-task)
>> +        (begin
>> +          (set! sleep* (@ (fibers) sleep))
>> +          (sleep n))
>> +        (begin
>> +          (set! sleep* (@ (guile) sleep))
>> +          ((@ (guile) sleep) n))))
>
> Instead of working around this limitation of Fiber's 'sleep',
> why not modify Fiber's sleep to detect if it is from a suspendable
> context or not, and depending on that, use Guile's sleep/usleep or
> the perform-operation+sleep-operation?

The short answer is that I want to get that done and not rely on an
unreleased version of Fibers.  The hack above uses the public interface;
it’s not pretty, but I think it’s okay.

Longer-term, maybe we could change Fibers’ ‘sleep’ as you suggest.
Whether it’s a good idea is debatable: one could argue that Fibers’
‘sleep’ is meant to be called from a fiber (suspendable).

Ludo’.

Re: The Shepherd on Fibers

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 274 bytes --]

Ludovic Courtès schreef op wo 23-03-2022 om 23:36 [+0100]:
> Thoughts?

To avoid accidentally using threads, perhaps Shepherd could do

(set! call-with-new-thread
  (lambda ()
    (error "multi-threading is not supported in Shepherd")))

?

Greetings,
Maxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

Re: The Shepherd on Fibers

[Philip McGrath]

Hi,

On 3/23/22 18:36, Ludovic Courtès wrote:
> Hello Guix!
> 
> I have pushed a ‘wip-fibers’ branch of the Shepherd:
> 
>    https://git.savannah.gnu.org/cgit/shepherd.git/log/?h=wip-fibers
> 
> The goal is to make shepherd (the daemon) use Fibers¹ for concurrency.
> 

Exciting!

As I wrote in 
<https://lists.gnu.org/archive/html/guix-devel/2021-05/msg00283.html>, 
"I haven't programmed with it in Guile at all, but, from my experience 
using Racket's Concurrent ML system, I think it is the right foundation 
for a supervisor daemon."

I still haven't programmed with Guile Fibers, and looking briefly at the 
diff of the "wip-fibers" branch was the first time I'd looked at the 
Shepherd's implementation, but I had a couple thoughts.

First,

> Fibers is used in a single-threaded fashion, which is the main
> constraint for shepherd since it forks.  That also means that fibers
> cannot be preempted, so it’s fully cooperative scheduling.
> 

Would it be feasible for shepherd *not* to fork? Or only to fork in a 
way that cooperates with fibers?

Obviously forking is pretty ubiquitous, but I the 2019 paper "A fork() 
in the road"[1] fairly convincing in its argument that

> fork has lost its classic simplicity; it today impacts all the
> other operating system abstractions with which it was once
> orthogonal. Moreover, a fundamental challenge with fork is
> that, since it conflates the process and the address space in
> which it runs, fork is hostile to user-mode implementation
> of OS functionality, breaking everything from buffered IO
> to kernel-bypass networking. Perhaps most problematically,
> fork doesn’t compose—every layer of a system from the kernel
> to the smallest user-mode library must support it.

I consider a Concurrent ML system a "user-mode implementation of OS 
functionality": indeed, an early version of Racket's thread system 
(where thread = fiber) is one of the examples in Flatt, Findler, 
Krishnamurthi, & Felleisen, "Programming Languages as Operating Systems 
(or Revenge of the Son of the Lisp Machine)" (ICFP 1999)[2].

More concretely, preemption is a big benefit of fibers.

Racket's approach also addresses this part:

> There’s one catch: Fibers is currently Linux-only.  The good news is
> that work has been done to port it to other kernels via libevent².
> Until it is merged, we could keep using the Shepherd 0.8 on GNU/Hurd.
> 

Racket has a cross-platform C library, "rktio"[3], for accessing 
os-specific functionality. It was refactored into its current form in 
2017 as an early step toward Racket-on-Chez: while it happens to provide 
exactly the functionality Racket needs, it no longer is specific to 
Racket or any particular implementation thereof. That includes 
everything needed to implement the Concurrent ML system and nonblocking 
IO on a variety of OSes and kernels.

In particular, it implements—IIUC primarily in 
"rktio_process.c"—abstractions (over `fork` or alternatives) to start a 
new process running something, with environment, stdin, stdout, and 
stderror wired up ports in the sense of 
`current-{input,output,error}-port`, and use the Concurrent ML system to 
monitor its exit status, send it signals, etc. The Racket-level API is 
documented at [4].

I spend as little time as possible with these C-level sorts of issues, 
and I particularly don't know about special considerations for PID 1, 
but I hope there would be some way to meet Shepherd's needs without 
interfering with Fibers.

Second, I'm a little uneasy about `unwind-protect`:

```
+(define-syntax-rule (unwind-protect body ... conclude)
+  "Evaluate BODY... and return its result(s), but always evaluate CONCLUDE
+before leaving, even if an exception is raised.
+
+This is *not* implemented with 'dynamic-wind' in order to play well with
+delimited continuations and fibers."
+  (let ((conclusion (lambda () conclude)))
+    (catch #t
+      (lambda ()
+        (call-with-values
+            (lambda ()
+              body ...)
+          (lambda results
+            (conclusion)
+            (apply values results))))
+      (lambda args
+        (conclusion)
+        (apply throw args)))))
```

though maybe it's used only internally and doesn't have to deal with the 
general case: I'm not an expert by any means, but my understanding (from 
a Racket mailing list thread at [5], continued at [6]) is that dealing 
with the general case may be something of an open research question.

As an example of the sort of problem, what if the `body`s evaluate 
normally, but an exception is raised in the dynamic extent of 
`(conclusion)`, causing `(conclusion)` to run again? One possible 
mitigation would be to `set!` a local variable before the normal 
`(conclusion)` and check it in the exception handler.

More generally, of course, `body ...` could escape by some mechanism 
other than a raising an exception.

If you were writing Racket, I would recommend 
`(call-with-continuation-barrier (λ () body ...))`—logically, `body ...` 
isn't re-entrant anyway—but I see from [7] that Guile's continuation 
barriers are barriers to the fibers' context switches.

The key difference between Guile and Racket is that Guile's Fibers are 
just a library, without any special privileges, and use the same control 
operators that are available to user code. I think that may be unique 
among the Concurrent ML implementations I'm aware of.

Why is that a problem? Well, in many ways I think it's quite cool! But a 
problematic aspect is that (from the discussion at [5]):


> On 11/30/19 10:23, Matthew Flatt wrote:
>> You're trying to implement `amb` where a client can mix `amb` and
>> `dynamic-wind` and get sensible behavior, right?
>> 
>> The `dynamic-wind` operation certainly interferes with building new
>> control forms. Racket threads and other control forms that need to
>> interact a certain way with `dynamic-wind` end up being built in at the
>> same level as `dynamic-wind`.
>> 
>> At Sat, 30 Nov 2019 06:15:16 -0600, Alexis King wrote:
>>> Is there any way to do that with Racket’s continuation machinery,
>> 
>> There's not a safe way. In many cases, Racket lets you write new things
>> that have the power of built-in through unsafe APIs --- and it turns
>> out that there are unadvertised procedures (provided by the primitive
>> `#%unsafe` module) for this particular case:
>> 
>> [...]
>> 
>> At Sat, 30 Nov 2019 06:15:16 -0600, Alexis King wrote:
>>> Also, is this kind of thing discussed anywhere in the literature?
>> 
>> I don't know of any published work on this topic (so let me know if you
>> find something!). As you probably have seen already, our ICFP'07 paper
>> just points out that `dynamic-wind` causes problems, but doesn't try to
>> hold `dynamic-wind` itself responsible for those problems.
>> 
>> An opinion and some pointers to newsgroup discussions:
>> 
>>    http://okmij.org/ftp/continuations/against-callcc.html#dynamic_wind
>> 

On 11/30/19 21:38, Alexis King wrote:
 > Yes, most of the relevant discussions I’ve found have been
 > about Lisp’s `unwind-protect` and how it relates to Scheme’s
 > `dynamic-wind`. It’s alluded to in Matthias’s original POPL ’88
 > paper and mentioned explicitly in the following LASC paper, but
 > neither address this particular issue. I also found Dorai
 > Sitaram’s “Unwind-protect in portable Scheme”[1] and the
 > related commentary by Kent Pitman[2] and Will Clinger[3], but
 > though obviously related, both Sitaram and Clinger seem to
 > imagine a system where the author of the program defines all
 > control operations, so there isn’t as much of a problem. I’ve
 > also been trying to find if any of these issues are discussed
 > in any algebraic effects literature, but I haven’t found
 > anything there yet, either.
 >
 > [1]: http://www.ccs.neu.edu/~dorai/uwcallcc/uwcallcc.html
 > [2]: 
http://www.nhplace.com/kent/PFAQ/unwind-protect-vs-continuations-overview.html
 > [3]: http://www.ccs.neu.edu/home/will/UWESC/uwesc.sch
 >

Maybe it would be enough for this case for Fibers to provide variants of 
`dynamic-wind` and/or `call-with-continuation-barrier` that cooperate 
with the Fibers implementation. I don't know if that would be possible 
of not—in addition to my limited familiarity with Fibers, I have not 
read all of the related literature that Alexis, Matthew, and Matthias 
Felleisen discuss in [5] and [6]—again, I am not an expert!

I'm not sure how useful any of that is, but take it for whatever it's 
worth. My overall impression is that the Shepherd on Fibers is a big 
step in the right direction. Congrats on the great work!

-Philip

[1]: 
https://www.microsoft.com/en-us/research/uploads/prod/2019/04/fork-hotos19.pdf
[2]: https://www2.ccs.neu.edu/racket/pubs/icfp99-ffkf.pdf
[3]: https://github.com/racket/racket/tree/master/racket/src/rktio
[4]: https://docs.racket-lang.org/reference/subprocess.html
[5]: https://groups.google.com/g/racket-users/c/AAeEp_llxaY/m/uRviksPGAgAJ
[6]: https://groups.google.com/g/racket-dev/c/PyetqHSjtAA/m/slBdazdqAwAJ
[7]: https://github.com/wingo/fibers/wiki/Manual#32-barriers

Re: The Shepherd on Fibers

[Philip McGrath]

On 3/28/22 20:14, Philip McGrath wrote:
> I'm not sure how useful any of that is, but take it for whatever it's 
> worth. My overall impression is that the Shepherd on Fibers is a big 
> step in the right direction. Congrats on the great work!
> 
> -Philip
> 

P.S.: If you can get access to a copy of John Reppy's book "Concurrent 
Programming in ML" (Cambridge University Press, 1999, reprinted with 
corrections in 2007), chapter 7 presents an extended example of a 
software build system. In addition to the other interesting features of 
the problem domain, it provides an example of controlling external 
programs in Concurrent ML style.

Re: The Shepherd on Fibers

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 1906 bytes --]

Philip McGrath schreef op ma 28-03-2022 om 20:14 [-0400]:
> Maybe it would be enough for this case for Fibers to provide variants of 
> `dynamic-wind` and/or `call-with-continuation-barrier` that cooperate 
> with the Fibers implementation. I don't know if that would be possible 
> of not—in addition to my limited familiarity with Fibers, I have not 
> read all of the related literature that Alexis, Matthew, and Matthias 
> Felleisen discuss in [5] and [6]—again, I am not an expert!

Fibers' context switching is based on continuations.  By definition,
‘call-with-continuation-barrier’ must create a continuation barrier
(and as a consequence, interfere with Fibers' context switching).

It can be important to let 'call-with-continuation-barrier' (*)
actually create a continuation barrier, even when fibers is used, e.g.
to not create deadlocks (or livelocks or whatever, I don't know the
terminology) when POSIX mutexes are used.  As such, as-is, I don't
think so.

Unless Guile could add some kind of '#:key "foobar"' argument to 'call-
wih-continuation-barrier' and 'call-with-prompt/abort-to-prompt' -- the
idea is that, by default, a continuation barrier cannot be passed,
unless a matching key is supplied.  So you could have a continuation
barrier that allows context switching, but not continuing with raise-
continuable or amb.

The unwind-protect defined at
<https://www.ccs.neu.edu/home/will/UWESC/uwesc.sch> looks interesting,
although it needs to be generalised to remove the global variables (for
multi-threading) and it needs to be verified whether it interacts well
with exeption handling.

Greetings,
Maxime.

(*) Actually, 'call-with-continuation-barrier' and 'dynamic-wind' are
already a bit of a lie, since the kernel ignores them when context
switching ... Maybe continuation barriers and {un,re}winding is a
matter of perspective?

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

Re: The Shepherd on Fibers

[Philip McGrath]

Hi,

On 3/29/22 05:36, Maxime Devos wrote:
> Philip McGrath schreef op ma 28-03-2022 om 20:14 [-0400]:
>> Maybe it would be enough for this case for Fibers to provide 
>> variants of `dynamic-wind` and/or `call-with-continuation-barrier` 
>> that cooperate with the Fibers implementation. I don't know if
>> that would be possible of not—in addition to my limited familiarity
>> with Fibers, I have not read all of the related literature that
>> Alexis, Matthew, and Matthias Felleisen discuss in [5] and
>> [6]—again, I am not an expert!
> 
> Fibers' context switching is based on continuations.  By definition,
>  ‘call-with-continuation-barrier’ must create a continuation barrier
>  (and as a consequence, interfere with Fibers' context switching).
> 
> It can be important to let 'call-with-continuation-barrier' (*) 
> actually create a continuation barrier, even when fibers is used, 
> e.g. to not create deadlocks (or livelocks or whatever, I don't know 
> the terminology) when POSIX mutexes are used.  As such, as-is, I 
> don't think so.
> 
> [...]
> 
> (*) Actually, 'call-with-continuation-barrier' and 'dynamic-wind' are
> already a bit of a lie, since the kernel ignores them when context
> switching ... Maybe continuation barriers and {un,re}winding is a
> matter of perspective?

Yes, that's the crux of the problem.

All of the references I know of are discussed in mailing list threads
[1] and [2], plus the more recent Flatt & Dybvig, "Compiler and Runtime
Support for Continuation Marks" (PLDI 2020) [3], which discusses the
Racket-on-Chez implementation of delimited control. In [1], Matthew
Flatt wrote:

> For an implementation that relies on a representation choice instead 
> of tracing or analysis, Racket CS's implementation of delimited 
> control is the state of the art --- mostly by building on Chez 
> Scheme's implementation of continuations.

Again, I am very far from an expert, but I'll try to distill the
relevant parts here.

To quote from the abstract and introduction of "Adding Delimited and
Composable Control to a Production Programming Environment" [4]
(internal citations omitted),

> Operators for delimiting control and for capturing composable 
> continuations litter the landscape of theoretical programming 
> language research.  Numerous papers explain their advantages, how
> the operators explain each other (or don’t), and other aspects of
> the operators’ existence.  Production programming languages, however,
> do not support these operators, partly because their relationship to 
> existing and demonstrably useful constructs—such as exceptions and 
> dynamic binding—remains relatively unexplored.
> 
> [...]
> 
> Due to this semantic interference, simulations of delimited control 
> do not immediately yield production-quality implementations.  For 
> example, a Scheme library can use `call/cc` to simulate delimited 
> continuations, but other libraries that use `call/cc` directly or 
> that use `dynamic-wind` can interfere with the simulation.
> 
> Over the past year, we have integrated a full set of delimited- 
> control operators within PLT Scheme, ensuring that all of them 
> interact properly with the rest of the Scheme programming language
> as well as pre-existing extensions in PLT Scheme. Specifically, PLT 
> Scheme’s prompts and composable continuations have a well-defined
> and useful interaction with `call/cc`, `dynamic-wind`, dynamic
> binding via continuation marks, and exceptions.

Racket's Concurrent ML subsystem also falls into that category.

The result of this paper is `racket/control` library[5].

To take Racket CS as an example, Chez Scheme doesn't provide delimited
continuations or "green" threads/fibers, but it does provide an
efficient and powerful implementation of continuations (even including
support for `equal?`!). The Racket CS runtime system implementation uses
Chez's `call/cc`, `dynamic-wind`, etc. to implement Racket's primitive
control operators (from the built-in module '#%kernel). Then, a larger
set of control operators can be implemented as a library in terms of the
primitives.

But, as the above paper says, this means that Chez's `call/cc`,
`dynamic-wind`, etc. are *unsafe* from the perspective of Racket's
control primitives. From the docs for Racket's `ffi/unsafe/vm` library [6]:

> Beware of directly calling a Chez Scheme primitive that uses Chez 
> Scheme parameters or `dynamic-wind` internally. Note that `eval`, in 
> particular, is such a primitive. The problem is that Chez Scheme’s 
> `dynamic-wind` does not automatically cooperate with Racket’s 
> continuations or threads. To call such primitives, use the 
> `call-with-system-wind primitive`, which takes a procedure of no 
> arguments to run in a context that bridges Chez Scheme’s
> `dynamic-wind` and Racket continuations and threads.

Anything that has the potential to block Racket's scheduler (as opposed 
to a fiber/thread), like POSIX mutexes, is likewise unsafe and should be 
encapsulated in a safe abstraction. For more on this, see the docs for 
`ffi/unsafe/atomic`[7], `ffi/unsafe/schedule`[8], `ffi/unsafe/port`[9], 
`ffi/unsafe/os-thread`[10], `ffi/unsafe/global`[11], and the 
`#:atomic?`, `#:async-apply`, `#:lock-name`, `#:in-original-plce?`, 
`#:blocking?`, `#:callback-exns?`, and `#:save-errno` arguments to 
`_cprocedure`[12], plus the section titled "Threads, Threads, Atomicity, 
Atomicity, and Atomicity" (yes, there are that many layers!) in the file 
"racket/src/cs/README.txt" [13] from the main Racket Git repository.

The key difference with Guile's Fibers is that it uses continuations at 
the same layer of abstraction available to other Guile code.

By "variants of `dynamic-wind` and/or `call-with-continuation-barrier` 
that cooperate with the Fibers implementation", I meant maybe Fibers 
could provide something like 
`call-with-continuation-barrier/except-for-fibers` and the Shepherd 
could use it. But I don't know enough about the implementation details 
to know if that would actually be a viable option.

It seems like `dynamic-wind` and reliable resource finalization are 
particular problems in this respect. Yesterday I started looking at the 
technical report Alexis King mentioned in [1], “Algebraic Effect 
Handlers with Resources and Deep Finalization”[14]. (Apparently there is 
a deep correspondence between algebraic effect handlers and delimited 
control.) In § 8.7 "Dynamic Wind", it says (internal citations omitted):

> The Scheme language always supported delimited continuations and
> they have also struggled with initialization- and finalization
> for continuations that resumed more than once. The
> `unwind-protect` in Scheme is like a `finally` clause, while
> `dynamic-wind` is like `initially`/finally with a pre- and
> postlude. Sitaram describes how the standard dynamic-wind is not
> good enough in general:
> 
>> While this may seem like a natural extension of the first-order
>> `unwind-protect` to a higher-order control scenario, it does not
>> tackle the pragmatic need that `unwind-protect` addresses,
>> namely, the need to ensure that a kind of “clean-up” happens only
>> for those jumps that significantly exit the block, and not for
>> those that are a minor excursion.  The crux is identifying which
>> of these two categories a jump falls into.
> 
> Interestingly, this is exactly what is addressed by algebraic
> effects where “significant exit”s are operations that do not
> resume, while “minor excursions” are regular operations that
> resume with a result.

(That Sitaram quote is from [15].)

This sounds more promising than anything else I've heard of! Enough to 
make me want to finish reading that paper :)

However, I know extremely little about algebraic effects, and I have no 
idea how this might translate into delimited control in the Scheme 
tradition, much less Guile or Fibers specifically. I think Alexis might 
be the most likely person to be able to answer that question.

-Philip

[1]: https://groups.google.com/g/racket-users/c/AAeEp_llxaY/m/uRviksPGAgAJ
[2]: https://groups.google.com/g/racket-dev/c/PyetqHSjtAA/m/slBdazdqAwAJ
[3]: https://www.cs.utah.edu/plt/publications/pldi20-fd.pdf
[4]: https://www.cs.utah.edu/plt/publications/icfp07-fyff.pdf
[5]: https://docs.racket-lang.org/reference/cont.html
[6]: https://docs.racket-lang.org/foreign/vm.html
[7]: https://docs.racket-lang.org/foreign/Atomic_Execution.html
[8]: https://docs.racket-lang.org/foreign/Thread_Scheduling.html
[9]: https://docs.racket-lang.org/foreign/Ports.html
[10]: https://docs.racket-lang.org/foreign/Operating_System_Threads.html
[11]: https://docs.racket-lang.org/foreign/unsafe-global.html
[12]: https://docs.racket-lang.org/foreign/foreign_procedures.html
[13]: https://github.com/racket/racket/blob/master/racket/src/cs/README.txt
[14]: 
https://www.microsoft.com/en-us/research/uploads/prod/2018/04/resource-v1.pdf
[15]: 
https://web.archive.org/web/20200223020813/http://www.ccs.neu.edu/~dorai/uwcallcc/uwcallcc.html

Re: The Shepherd on Fibers

[Ludovic Courtès]

Philip McGrath <philip@philipmcgrath.com> skribis:

> But, as the above paper says, this means that Chez's `call/cc`,
> `dynamic-wind`, etc. are *unsafe* from the perspective of Racket's
> control primitives. From the docs for Racket's `ffi/unsafe/vm` library [6]:

I think that’s the crux of the problem and widely recognized:
‘dynamic-wind’ and ‘call/cc’ should be avoided.  This was also mentioned
in the context of Rees’ 1996 “Security Kernel” paper¹.

Ludo’.

¹ http://mumble.net/~jar/pubs/secureos/

Re: The Shepherd on Fibers

[Zhu Zihao]

[-- Attachment #1: Type: text/plain, Size: 5459 bytes --]


Philip McGrath <philip@philipmcgrath.com> writes:

> Would it be feasible for shepherd *not* to fork? Or only to fork in a way that
> cooperates with fibers?

IMO the problem is not fork, you can't do anything useful to create
subprocess without fork on *NIX systems.

fork is allowed to execute in multi-thread environment. It just don't
care about other threads, it's safe and recommended if you only call
execv after fork.

The problem is, Shepherd is too flexible and people can do so many
things other than fork+exec in the startup of Shepherd service (unlike
systemd, only create subprocess is allowed). So Shepherd has to be
conservative.

> Second, I'm a little uneasy about `unwind-protect`:
>
> ```
> +(define-syntax-rule (unwind-protect body ... conclude)
> +  "Evaluate BODY... and return its result(s), but always evaluate CONCLUDE
> +before leaving, even if an exception is raised.
> +
> +This is *not* implemented with 'dynamic-wind' in order to play well with
> +delimited continuations and fibers."
> +  (let ((conclusion (lambda () conclude)))
> +    (catch #t
> +      (lambda ()
> +        (call-with-values
> +            (lambda ()
> +              body ...)
> +          (lambda results
> +            (conclusion)
> +            (apply values results))))
> +      (lambda args
> +        (conclusion)
> +        (apply throw args)))))
> ```
>
> though maybe it's used only internally and doesn't have to deal with the general
> case: I'm not an expert by any means, but my understanding (from a Racket
> mailing list thread at [5], continued at [6]) is that dealing with the general
> case may be something of an open research question.
>
> As an example of the sort of problem, what if the `body`s evaluate normally, but
> an exception is raised in the dynamic extent of `(conclusion)`, causing
> `(conclusion)` to run again? One possible mitigation would be to `set!` a local
> variable before the normal `(conclusion)` and check it in the exception handler.
>
> More generally, of course, `body ...` could escape by some mechanism other than
> a raising an exception.
>
> If you were writing Racket, I would recommend `(call-with-continuation-barrier
> (λ () body ...))`—logically, `body ...` isn't re-entrant anyway—but I see from
> [7] that Guile's continuation barriers are barriers to the fibers' context
> switches.

> The key difference between Guile and Racket is that Guile's Fibers are just a
> library, without any special privileges, and use the same control operators that
> are available to user code. I think that may be unique among the Concurrent ML
> implementations I'm aware of.
>
> Why is that a problem? Well, in many ways I think it's quite cool! But a
> problematic aspect is that (from the discussion at [5]):
>
>
>> On 11/30/19 10:23, Matthew Flatt wrote:
>>> You're trying to implement `amb` where a client can mix `amb` and
>>> `dynamic-wind` and get sensible behavior, right?
>>> The `dynamic-wind` operation certainly interferes with building new
>>> control forms. Racket threads and other control forms that need to
>>> interact a certain way with `dynamic-wind` end up being built in at the
>>> same level as `dynamic-wind`.
>>> At Sat, 30 Nov 2019 06:15:16 -0600, Alexis King wrote:
>>>> Is there any way to do that with Racket’s continuation machinery,
>>> There's not a safe way. In many cases, Racket lets you write new things
>>> that have the power of built-in through unsafe APIs --- and it turns
>>> out that there are unadvertised procedures (provided by the primitive
>>> `#%unsafe` module) for this particular case:
>>> [...]
>>> At Sat, 30 Nov 2019 06:15:16 -0600, Alexis King wrote:
>>>> Also, is this kind of thing discussed anywhere in the literature?
>>> I don't know of any published work on this topic (so let me know if you
>>> find something!). As you probably have seen already, our ICFP'07 paper
>>> just points out that `dynamic-wind` causes problems, but doesn't try to
>>> hold `dynamic-wind` itself responsible for those problems.
>>> An opinion and some pointers to newsgroup discussions:
>>>    http://okmij.org/ftp/continuations/against-callcc.html#dynamic_wind
>>>

The reentrant feature of call/cc costs many but worth nothing. And people
have to create dynamic-wind to fix this ugly issue.

Schemers now create a new proposal, SRFI-226. In this proposal, call/cc
is reimplemented in delimited control operator.

> Maybe it would be enough for this case for Fibers to provide variants of
> `dynamic-wind` and/or `call-with-continuation-barrier` that cooperate 
> with the Fibers implementation. I don't know if that would be possible of not—in
> addition to my limited familiarity with Fibers, I have not read all of the
> related literature that Alexis, Matthew, and Matthias Felleisen discuss in [5]
> and [6]—again, I am not an expert!
>
> I'm not sure how useful any of that is, but take it for whatever it's worth. My
> overall impression is that the Shepherd on Fibers is a big step in the right
> direction. Congrats on the great work!

If Shepherd can use something like G-exp, we can force user to do its
customized job in subprocess via code staging. And restrict the shepherd
only do process creation in the startup of service. 
-- 
Retrieve my PGP public key:

  gpg --recv-keys D47A9C8B2AE3905B563D9135BE42B352A9F6821F

Zihao

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 255 bytes --]

Re: The Shepherd on Fibers

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 779 bytes --]

Zhu Zihao schreef op di 29-03-2022 om 18:13 [+0800]:
> IMO the problem is not fork, you can't do anything useful to create
> subprocess without fork on *NIX systems.
> 
> fork is allowed to execute in multi-thread environment. It just don't
> care about other threads, it's safe and recommended if you only call
> execv after fork.
> 
> The problem is, Shepherd is too flexible and people can do so many
> things other than fork+exec in the startup of Shepherd service
> (unlike
> systemd, only create subprocess is allowed). So Shepherd has to be
> conservative.
> 

Being conservative is not sufficient.  What if, in-between 'fork' and
'exec', some memory is allocated (and hence, in typical 'malloc'
implementations, locking happens)?

Greetings,
Maxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

Re: The Shepherd on Fibers

[Ludovic Courtès]

Hi!

Philip McGrath <philip@philipmcgrath.com> skribis:

>> Fibers is used in a single-threaded fashion, which is the main
>> constraint for shepherd since it forks.  That also means that fibers
>> cannot be preempted, so it’s fully cooperative scheduling.
>> 
>
> Would it be feasible for shepherd *not* to fork?

Sure, but that would make it a daemon supervisor that doesn’t launch any
daemon, which would make it less useful.  :-)

> Or only to fork in a way that cooperates with fibers?

A process that has multiple POSIX threads cannot fork reliably; that’s
the core of the problem (it’s a problem of POSIX and ‘fork’, not
specific to Guile and Fibers).

> Obviously forking is pretty ubiquitous, but I the 2019 paper "A fork()
> in the road"[1] fairly convincing in its argument that

Yes, I read it before.  We could have bindings for ‘posix_spawn’ but
it’s kind of ugly and limited.

> More concretely, preemption is a big benefit of fibers.

I’m not arguing against preemption, I like it.  :-)

However, as currently implemented in Fibers, we cannot rely on it,
though Maxime mentioned that it could use signals instead of a separate
thread, which would work.  We can always revisit that later.  I making a
first step here, and being cautious, but we can and should unleash our
creativity once we have the basics in place.

> Racket has a cross-platform C library, "rktio"[3], for accessing
> os-specific functionality. It was refactored into its current form in 
> 2017 as an early step toward Racket-on-Chez: while it happens to
> provide exactly the functionality Racket needs, it no longer is
> specific to Racket or any particular implementation thereof. That
> includes everything needed to implement the Concurrent ML system and
> nonblocking IO on a variety of OSes and kernels.
>
> In particular, it implements—IIUC primarily in
> "rktio_process.c"—abstractions (over `fork` or alternatives) to start
> a new process running something, with environment, stdin, stdout, and 
> stderror wired up ports in the sense of
> `current-{input,output,error}-port`, and use the Concurrent ML system
> to monitor its exit status, send it signals, etc. The Racket-level API
> is documented at [4].

Interesting.  I’m not sure this is directly translatable to Guile.
However, libevent support for Fibers should address this specific point,
so I’m not really concerned.

> Second, I'm a little uneasy about `unwind-protect`:

Your criticism is valid, though here it has a single user and it’s “good
enough” for that case I think.

Thanks for your feedback!

Ludo’.

Re: The Shepherd on Fibers

[Ludovic Courtès]

Hi!

Ludovic Courtès <ludo@gnu.org> skribis:

> I have pushed a ‘wip-fibers’ branch of the Shepherd:
>
>   https://git.savannah.gnu.org/cgit/shepherd.git/log/?h=wip-fibers
>
> The goal is to make shepherd (the daemon) use Fibers¹ for concurrency.

The latest news are:

  1. Support for inetd-style services¹, where shepherd listens and
     accepts client connections, starts the daemon and hands it the
     connection socket—e.g., ‘sshd -i’.

     Each daemon process is handled as a “transient service”, a new type
     of service that is automatically unregistered when it’s done.  You
     can see those when running ‘herd status’; in the OpenSSH example,
     you’d see ‘sshd-1’, ‘sshd-2’, etc. corresponding to individual SSH
     client connections.

  2. Support for systemd-style “socket activation”², where shepherd
     listens, spawns the daemon when the first request arrives, and
     hands it the *listening* socket.

     From the user viewpoint, this one is similar to the existing
     ‘make-forkexec-constructor’ type of service.  The main difference
     is that its “running value” as shown by ‘herd status SERVICE’ is
     initially a socket and then becomes a PID.

My plan is to merge ‘wip-fibers’ in ‘master’, publish a release
candidate, publish a ‘wip-’ branch of Guix that we can use for testing,
and hopefully release the Shepherd 0.9.0 within a week or two.

Ludo’.

¹ https://www.gnu.org/software/inetutils/manual/html_node/inetd-invocation.html
² https://www.freedesktop.org/software/systemd/man/daemon.html#Socket-Based%20Activation

The Shepherd on Fibers

[Brendan Tildesley]

I think you are doing amazing work! I hope socket activation can be added
because it may be helpful for launching the user services for pipewire, but I've
never really understood how it actually works.

I would like to replace pulseaudio with pipewire as the default in
%desktop-services, the only hurdle is how to launch the user daemons in all the
different desktop configurations one might use. Other distros use systemd's
socket activation to magically launch pipewire.

Otherwise XDG autostarts or some kind of guix home service could launch it?

The Shepherd on Fibers

[Nathan Dehnel]

Hooray!