Hack the (init) system!

Hack the (init) system!

[Ludovic Courtès]

[-- Attachment #1: Type: text/plain, Size: 140 bytes --]

Howdy Guix!

We’ve all been talking about it for some time: a REPL server in dmd!

This can be done by changing zero lines in dmd:


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Type: text/x-patch, Size: 1879 bytes --]

diff --git a/gnu/services/base.scm b/gnu/services/base.scm
index 888e446..f39a0a4 100644
--- a/gnu/services/base.scm
+++ b/gnu/services/base.scm
@@ -46,6 +46,7 @@
             device-mapping-service
             swap-service
             user-processes-service
+            dmd-repl-service
             host-name-service
             console-keymap-service
             console-font-service
@@ -287,6 +288,34 @@ stopped before 'kill' is called."
                        #f))
              (respawn? #f)))))
 
+
+(define* (dmd-repl-service #:optional (file-name "/var/run/dmd/repl")
+                           #:key (dmd dmd))
+  "Return a service that opens a REPL for dmd.  Authorized users can connect
+to the REPL at @var{file-name}, which points to a Unix-domain socket.  This
+may be done from the command using @command{socat unix-connect:@var{file-name}
+stdio}, or from Emacs using @code{M-x geiser-connect-local}."
+  (with-monad %store-monad
+    (return (service
+             (documentation "Run a REPL service inside dmd.")
+             (provision '(dmd-repl))
+             (requirement '(root-file-system))
+             (start #~(begin
+                        (use-modules (system repl server))
+
+                        (lambda* (#:optional (file-name #$file-name))
+                          (let ((socket (make-unix-domain-server-socket
+                                         #:path file-name)))
+                            (spawn-server socket)
+                            #t))))
+             (stop #~(begin
+                       (use-modules (system repl server))
+
+                       (lambda _
+                         (stop-server-and-clients!)
+                         #f)))
+             (auto-start? #f)))))
+
 (define (host-name-service name)
   "Return a service that sets the host name to @var{name}."
   (with-monad %store-monad

[-- Attachment #3: Type: text/plain, Size: 580 bytes --]


Then you can do:

  deco start dmd-repl
  socat unix-connect:/var/run/dmd/repl stdio

and tinker from there.  New ways to cra^W experiment with your system!

I’m tempted to just commit that.  There are shortcomings: (1) the REPL
server runs in a thread and threads + fork don’t go together well
(although in practice dmd only does fork followed by exec, so it’s OK),
and (2) for some reason ‘stop-server-and-clients!’ seems to leave open
sockets behind it, so if you restart the REPL on the same socket, it
fails with EADDRINUSE.

Thoughts?

Ludo’.

Re: Hack the (init) system!

[Thompson, David]

On Thu, Sep 3, 2015 at 5:02 PM, Ludovic Courtès <ludo@gnu.org> wrote:
> Howdy Guix!
>
> We’ve all been talking about it for some time: a REPL server in dmd!
>
> This can be done by changing zero lines in dmd:
>
>
>
> Then you can do:
>
>   deco start dmd-repl
>   socat unix-connect:/var/run/dmd/repl stdio
>
> and tinker from there.  New ways to cra^W experiment with your system!
>
> I’m tempted to just commit that.  There are shortcomings: (1) the REPL
> server runs in a thread and threads + fork don’t go together well
> (although in practice dmd only does fork followed by exec, so it’s OK),
> and (2) for some reason ‘stop-server-and-clients!’ seems to leave open
> sockets behind it, so if you restart the REPL on the same socket, it
> fails with EADDRINUSE.
>
> Thoughts?

Awesome little hack.  Thanks!  I probably won't use this for PID 1,
but I will use something similar for my userspace dmd process.

Now, I think this solves the immediate need of being able to live hack
dmd, but I'd like to note an additional shortcoming: Thread
synchronization isssues.  As you know, there's potential for the main
dmd thread and a REPL thread to write to the same memory and blow
things up.  In practice, this would be unlikely, but it shouldn't even
be a possibility.  To accommodate programs that run in an event loop
(though I know dmd doesn't truly have this yet), Mark and I developed
the (system repl coop-server) module available in Guile 2.0.11.  This
"cooperative" REPL server can be integrated into an event loop and
guarantee that expressions are only evaluated in the context of a
single thread, in this case the main thread.  I think that this should
be the approach taken in the not-so-long term, which will require
modifying dmd itself.  I have been using the cooperative REPL server
for my game engine since Guile 2.0.11 was released and I've been able
to comfortably live hack games without fear.  Problem (1) still exists
because threads are  used for each client, but the real magic happens
via coroutines.  Not sure if we can reasonably get rid of threads
altogether and uses a non-blocking I/O model.  Might be worth looking
into, but I'm getting off-topic.

Now that we can live hack dmd, we'll need some things to help make it
pleasant.  Most of the time I am tweaking service definitions until
they are just right.  Currently, that means calling a procedure to
unload the version that exists, and then registering a new one.  I'd
like to reduce that to a single step to tighten the feedback loop.
What do you think about adding a 'register-services*' procedure, or
maybe a 'define-service' form, that first unregisters the old service
before registering the new one?

If we can achieve a safe live hacking environment, then we'll have one
hell of an init system, I say.  Thanks!

- Dave

Re: Hack the (init) system!

[Ludovic Courtès]

"Thompson, David" <dthompson2@worcester.edu> skribis:

> Now, I think this solves the immediate need of being able to live hack
> dmd, but I'd like to note an additional shortcoming: Thread
> synchronization isssues.  As you know, there's potential for the main
> dmd thread and a REPL thread to write to the same memory and blow
> things up.  In practice, this would be unlikely, but it shouldn't even
> be a possibility.  To accommodate programs that run in an event loop
> (though I know dmd doesn't truly have this yet), Mark and I developed
> the (system repl coop-server) module available in Guile 2.0.11.  This
> "cooperative" REPL server can be integrated into an event loop and
> guarantee that expressions are only evaluated in the context of a
> single thread, in this case the main thread.  I think that this should
> be the approach taken in the not-so-long term, which will require
> modifying dmd itself.

I agree that the cooperative REPL server is the way to go.  I just
couldn’t resist the temptation to hack that thing.  ;-)

It would be ideal if instead of having built-in support for the REPL
server, dmd instead provided a way for services to return file
descriptors to monitor and if they could be notified of I/O events on
those file descriptors.  That way, the REPL server could be a normal
REPL service.

> Now that we can live hack dmd, we'll need some things to help make it
> pleasant.  Most of the time I am tweaking service definitions until
> they are just right.  Currently, that means calling a procedure to
> unload the version that exists, and then registering a new one.  I'd
> like to reduce that to a single step to tighten the feedback loop.
> What do you think about adding a 'register-services*' procedure, or
> maybe a 'define-service' form, that first unregisters the old service
> before registering the new one?

Sounds good to me.

Thanks,
Ludo’.

Re: Hack the (init) system!

[Thompson, David]

On Fri, Sep 4, 2015 at 8:05 AM, Ludovic Courtès <ludo@gnu.org> wrote:
> "Thompson, David" <dthompson2@worcester.edu> skribis:
>
>> Now, I think this solves the immediate need of being able to live hack
>> dmd, but I'd like to note an additional shortcoming: Thread
>> synchronization isssues.  As you know, there's potential for the main
>> dmd thread and a REPL thread to write to the same memory and blow
>> things up.  In practice, this would be unlikely, but it shouldn't even
>> be a possibility.  To accommodate programs that run in an event loop
>> (though I know dmd doesn't truly have this yet), Mark and I developed
>> the (system repl coop-server) module available in Guile 2.0.11.  This
>> "cooperative" REPL server can be integrated into an event loop and
>> guarantee that expressions are only evaluated in the context of a
>> single thread, in this case the main thread.  I think that this should
>> be the approach taken in the not-so-long term, which will require
>> modifying dmd itself.
>
> I agree that the cooperative REPL server is the way to go.  I just
> couldn’t resist the temptation to hack that thing.  ;-)
>
> It would be ideal if instead of having built-in support for the REPL
> server, dmd instead provided a way for services to return file
> descriptors to monitor and if they could be notified of I/O events on
> those file descriptors.  That way, the REPL server could be a normal
> REPL service.

That sounds like a nice generalization.

>> Now that we can live hack dmd, we'll need some things to help make it
>> pleasant.  Most of the time I am tweaking service definitions until
>> they are just right.  Currently, that means calling a procedure to
>> unload the version that exists, and then registering a new one.  I'd
>> like to reduce that to a single step to tighten the feedback loop.
>> What do you think about adding a 'register-services*' procedure, or
>> maybe a 'define-service' form, that first unregisters the old service
>> before registering the new one?
>
> Sounds good to me.

Okay, I will prepare some patches.  Thanks.

- Dave

Re: Hack the (init) system!

[Thompson, David]

Hi there, me again.

On Thu, Sep 3, 2015 at 5:02 PM, Ludovic Courtès <ludo@gnu.org> wrote:
> Howdy Guix!
>
> We’ve all been talking about it for some time: a REPL server in dmd!
>
> This can be done by changing zero lines in dmd:
>
>
>
> Then you can do:
>
>   deco start dmd-repl
>   socat unix-connect:/var/run/dmd/repl stdio

I tried to use 'geiser-connect-local' to connect to the REPL and it
seems that Geiser just breaks when using a UNIX domain socket.  Can
anyone else reproduce this?

- Dave

Re: Hack the (init) system!

[Mark H Weaver]

ludo@gnu.org (Ludovic Courtès) writes:

> We’ve all been talking about it for some time: a REPL server in dmd!
>
> This can be done by changing zero lines in dmd:

Cool hack!

> I’m tempted to just commit that.  There are shortcomings: (1) the REPL
> server runs in a thread and threads + fork don’t go together well
> (although in practice dmd only does fork followed by exec, so it’s OK),

Unfortunately, it's only okay if the code between fork and exec in the
child process is carefully written to execute only "async-signal-safe"
operations.

  http://pubs.opengroup.org/onlinepubs/9699919799/functions/fork.html

I don't think we can ensure this unless the fork, exec, and everything
in between is carefully written in C.  This is not currently the case in
dmd.

So, I think we have two choices:

1. Avoid threads in dmd, i.e. either refrain from adding this REPL
   server feature, or re-implement it in a way that avoids threads.

2. Avoid 'primitive-fork' in dmd, which means reimplementing
   'fork+exec-command' in C; reimplementing the code where we currently
   use 'primitive-fork' within various guix service definitions; and
   documenting that users should never use 'primitive-fork' in their
   services.  If we choose this route, we should probably disable
   'primitive-fork' somehow, or at least have it issue a stern warning.

I don't think that we should add a set of features to dmd that will make
it fundamentally unreliable in a way that cannot be fixed.

What do you think?

      Mark

Re: Hack the (init) system!

[Ludovic Courtès]

Mark H Weaver <mhw@netris.org> skribis:

> ludo@gnu.org (Ludovic Courtès) writes:

[...]

>> I’m tempted to just commit that.  There are shortcomings: (1) the REPL
>> server runs in a thread and threads + fork don’t go together well
>> (although in practice dmd only does fork followed by exec, so it’s OK),
>
> Unfortunately, it's only okay if the code between fork and exec in the
> child process is carefully written to execute only "async-signal-safe"
> operations.

Right (I was not suggesting that the hack is robus, rather that it’s
kinda OK “in practice”, notably because I would only start it once all
the ‘fork’ calls have been made.)

> So, I think we have two choices:
>
> 1. Avoid threads in dmd, i.e. either refrain from adding this REPL
>    server feature, or re-implement it in a way that avoids threads.
>
> 2. Avoid 'primitive-fork' in dmd, which means reimplementing
>    'fork+exec-command' in C; reimplementing the code where we currently
>    use 'primitive-fork' within various guix service definitions; and
>    documenting that users should never use 'primitive-fork' in their
>    services.  If we choose this route, we should probably disable
>    'primitive-fork' somehow, or at least have it issue a stern warning.
>
> I don't think that we should add a set of features to dmd that will make
> it fundamentally unreliable in a way that cannot be fixed.
>
> What do you think?

Agreed, of course.  I think #1 is the way to go.

Thanks,
Ludo’.

Re: Hack the (init) system!

[Christopher Allan Webber]

Ludovic Courtès writes:

>> So, I think we have two choices:
>>
>> 1. Avoid threads in dmd, i.e. either refrain from adding this REPL
>>    server feature, or re-implement it in a way that avoids threads.
>>
>> 2. Avoid 'primitive-fork' in dmd, which means reimplementing
>>    'fork+exec-command' in C; reimplementing the code where we currently
>>    use 'primitive-fork' within various guix service definitions; and
>>    documenting that users should never use 'primitive-fork' in their
>>    services.  If we choose this route, we should probably disable
>>    'primitive-fork' somehow, or at least have it issue a stern warning.
>>
>> I don't think that we should add a set of features to dmd that will make
>> it fundamentally unreliable in a way that cannot be fixed.
>>
>> What do you think?
>
> Agreed, of course.  I think #1 is the way to go.
>
> Thanks,
> Ludo’.

I wonder if it's about time that Guile get something along the lines of
a well-supported, general event loop system?  There's value in having a
mainline way to do it; Python has AsyncIO which is gaining a lot of
traction... the main advantage being that having an officially supported
cooperative event loop (and as a bonus, an official way of working with
it nicely with coroutines) permits writing a lot of code in a
non-blocking way that's compatible... you can use this non-blocking
database wrapper with that non-blocking websocket implementation, for
example.

This is pretty off-topic from Guix though I guess... maybe it's a
conversation for guile-user, or guile-devel.  But I'm curious if others
have been thinking along these lines.

 - Chris

Re: Hack the (init) system!

[Ludovic Courtès]

Christopher Allan Webber <cwebber@dustycloud.org> skribis:

> I wonder if it's about time that Guile get something along the lines of
> a well-supported, general event loop system?

Sure.  My take on this is that we want FRP, as implemented in Sly, or a
synchronous reactive API like HIPHOP¹.  I wouldn’t want Guile to invite
users to use a JavaScriptish asynchronous framework with callbacks and
IoC all over the place.

Then of course that’s a bit of work, including work to integrate that
with ports.

Ludo’.

¹ http://www.inf.fu-berlin.de/lehre/SS13/Sem-Prog/material/HIPHOP.pdf

Re: Hack the (init) system!

[Christopher Allan Webber]

Ludovic Courtès writes:

> Christopher Allan Webber <cwebber@dustycloud.org> skribis:
>
>> I wonder if it's about time that Guile get something along the lines of
>> a well-supported, general event loop system?
>
> Sure.  My take on this is that we want FRP, as implemented in Sly, or a
> synchronous reactive API like HIPHOP¹.  I wouldn’t want Guile to invite
> users to use a JavaScriptish asynchronous framework with callbacks and
> IoC all over the place.
>
> Then of course that’s a bit of work, including work to integrate that
> with ports.
>
> Ludo’.
>
> ¹ http://www.inf.fu-berlin.de/lehre/SS13/Sem-Prog/material/HIPHOP.pdf

HIPHOP looks interesting!  And Sly is of course great, I'd love to see
more Guile programs exploring that design.  I haven't thought about what
a web application that was Sly-like would look like.

Just a side note: I don't know much about what IoC (presumably Inversion
of Control) translates to in practice, though I can say that asyncio
avoids callback hell pretty well through clever use of coroutines... it
reduces some of thes stresses that node.js brings.  I wonder if clever
use of delimited continuations can allow us to also make things nice.

 - Chris

Re: Hack the (init) system!

[Andy Wingo]

On Fri 25 Sep 2015 23:04, Christopher Allan Webber <cwebber@dustycloud.org> writes:

> I wonder if it's about time that Guile get something along the lines of
> a well-supported, general event loop system?

I think Guile needs user-space threads, implemented on top of delimited
continuations.  With threads, you don't need to invert control in your
program.  Racket takes this approach as well.

To get there we need to expose port buffers to Scheme, mark all file
descriptors as nonblocking, and cause EWOULDBLOCK to suspend the current
green thread.  I had a prototype working a while back on the
wip-ethreads branch, but you don't want to introduce a separate port
type -- really you want to have this work on all ports, so that's the
hacking that's needed.

I'll try to get out a 2.2 prerelease in the next week or so --
everything is up to date now, finally, I just have a pending patch to
make the stack grow down instead of up so we can use native CALL
instructions in some future.  Works in progress...

A

Re: Hack the (init) system!

[Taylan Ulrich Bayırlı/Kammer]

Andy Wingo <wingo@igalia.com> writes:

> On Fri 25 Sep 2015 23:04, Christopher Allan Webber <cwebber@dustycloud.org> writes:
>
> [...]
>
> I'll try to get out a 2.2 prerelease in the next week or so --
> everything is up to date now, finally, I just have a pending patch to
> make the stack grow down instead of up so we can use native CALL
> instructions in some future.  Works in progress...

Exciting news! :-)

Thanks so much for all your work on Guile.


Taylan

Re: Hack the (init) system!

[Thompson, David]

On Mon, Sep 28, 2015 at 5:13 AM, Andy Wingo <wingo@igalia.com> wrote:
> On Fri 25 Sep 2015 23:04, Christopher Allan Webber <cwebber@dustycloud.org> writes:
>
>> I wonder if it's about time that Guile get something along the lines of
>> a well-supported, general event loop system?
>
> I think Guile needs user-space threads, implemented on top of delimited
> continuations.  With threads, you don't need to invert control in your
> program.  Racket takes this approach as well.
>
> To get there we need to expose port buffers to Scheme, mark all file
> descriptors as nonblocking, and cause EWOULDBLOCK to suspend the current
> green thread.  I had a prototype working a while back on the
> wip-ethreads branch, but you don't want to introduce a separate port
> type -- really you want to have this work on all ports, so that's the
> hacking that's needed.

"Green threads" and "user-space threads" are coroutines, yes?  That
would be a very nice feature.

> I'll try to get out a 2.2 prerelease in the next week or so --
> everything is up to date now, finally, I just have a pending patch to
> make the stack grow down instead of up so we can use native CALL
> instructions in some future.  Works in progress...

Exciting! :)

- Dave

Re: Hack the (init) system!

[Christopher Allan Webber]

Andy Wingo writes:

> On Fri 25 Sep 2015 23:04, Christopher Allan Webber <cwebber@dustycloud.org> writes:
>
>> I wonder if it's about time that Guile get something along the lines of
>> a well-supported, general event loop system?
>
> I think Guile needs user-space threads, implemented on top of delimited
> continuations.  With threads, you don't need to invert control in your
> program.  Racket takes this approach as well.
>
> To get there we need to expose port buffers to Scheme, mark all file
> descriptors as nonblocking, and cause EWOULDBLOCK to suspend the current
> green thread.  I had a prototype working a while back on the
> wip-ethreads branch, but you don't want to introduce a separate port
> type -- really you want to have this work on all ports, so that's the
> hacking that's needed.
>
> I'll try to get out a 2.2 prerelease in the next week or so --
> everything is up to date now, finally, I just have a pending patch to
> make the stack grow down instead of up so we can use native CALL
> instructions in some future.  Works in progress...
>
> A

Sounds great!  I'm looking forward to this and Guile 2.2 :)