Implementing the guix-dameon in Guile

Implementing the guix-dameon in Guile

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 2316 bytes --]

Hey!

I think this has been talked about for a while [1], but I want to make it
happen. Currently the guix-daemon is still similar to the nix-daemon
that it was forked from, and is implemented in C++. I think that a Guile
implementation of the guix-daemon will simplify Guix and better support
hacking on and around the daemon to add new features and move Guix
forward.

1: https://git.savannah.gnu.org/cgit/guix/maintenance.git/tree/doc/ROADMAP.org#n71

My plan is to focus on this over the next year. I left my previous day
job quite a few months ago now to take a bit of a break, that's the main
reason I've been able to spend more time trying to push forward some of
the QA stuff. With some monetary support from NLNet [2], I'm planning to
continue this break and focus for the next year on getting a Guile
implementation of the guix-daemon written and adopted.

2: https://nlnet.nl/project/GuixDaemon-Guile/

Rewrites are risky because you only get the value right at the end,
therefore the priority is to get a minimal but viable implementation in
Guile that can be switched to, and not to get distracted on adding or
improving functionality unnecessarily. That is better done once the new
implementation has been adopted.

While I think there's a substantial amount of work to do, progress
towards a Guile guix-daemon has already been made. There was a least one
GSoC project which did make progress, and there's Guile implementations
of some of the functionality in Guix already.

Still though, I'd like to hear what people think about which direction
the implementation should go, and what features they'd like to see. Even
if those are not essential to make the Guile implementation viable, it
still might inform the direction to take.

The Guile rewrite of the guix-dameon was on my mind over 3 years ago
when I was thinking about the build coordinator [3]. As part of writing
the build coordinator, I got experienced using SQLite and the Guile
bindings and that'll come in very useful. The build coordinator also
uses fibers, and I'll probably look to use fibers as well in the
guix-daemon. I think the work Ludo has done on the shepherd has made
this possible.

3: https://lists.gnu.org/archive/html/guix-devel/2020-04/msg00323.html

Let me know if you've got any comments or questions!

Thanks,

Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

Re: Implementing the guix-dameon in Guile

[Felix Lechner via Development of GNU Guix and the GNU System distribution.]

Hi Chris,

On Wed, Sep 13, 2023 at 9:29 AM Christopher Baines <mail@cbaines.net> wrote:
>
> Rewrites are risky because you only get the value right at the end,
> therefore the priority is to get a minimal but viable implementation in
> Guile that can be switched to

Sounds exciting! I would like to help with testing.

The transformation toward a Guile daemon is a point of consistency and
pride for the project and therefore unlikely to be second-guessed or
reverted. My recommendation is to replace the daemon gradually—working
from (apply system* (command-line) downward—and mainline your
incremental changes as quickly as possible.

Thanks for doing this!

Kind regards,
Felix

Re: Implementing the guix-dameon in Guile

[Maxim Cournoyer]

Hi!

Felix Lechner via "Development of GNU Guix and the GNU System
distribution." <guix-devel@gnu.org> writes:

> Hi Chris,
>
> On Wed, Sep 13, 2023 at 9:29 AM Christopher Baines <mail@cbaines.net> wrote:
>>
>> Rewrites are risky because you only get the value right at the end,
>> therefore the priority is to get a minimal but viable implementation in
>> Guile that can be switched to
>
> Sounds exciting! I would like to help with testing.

Indeed, congrats for securing the NLnet founding (and thanks to them!).

> The transformation toward a Guile daemon is a point of consistency and
> pride for the project and therefore unlikely to be second-guessed or
> reverted. My recommendation is to replace the daemon gradually—working
> from (apply system* (command-line) downward—and mainline your
> incremental changes as quickly as possible.

I had some musing about the daemon recently; I was thinking libguile
could be added to our old C++ daemon, which could then replace its
functions piece-wise with Scheme implemented ones?

At the end, we'd have almost everything in Scheme, at which point we
could rewrite the main loop to Scheme and say farewell to the old
daemon.

Would that approach makes any sense?  I know of at least one C++ project
integrating libguile, and that's the jami-daemon, so it could perhaps
provide some clues as to how to proceed to do so.

Thanks for the great initiative!

-- 
Thanks,
Maxim

Re: Implementing the guix-dameon in Guile

[Vagrant Cascadian]

[-- Attachment #1: Type: text/plain, Size: 1738 bytes --]

On 2023-09-13, Christopher Baines wrote:
> I think this has been talked about for a while [1], but I want to make it
> happen. Currently the guix-daemon is still similar to the nix-daemon
> that it was forked from, and is implemented in C++. I think that a Guile
> implementation of the guix-daemon will simplify Guix and better support
> hacking on and around the daemon to add new features and move Guix
> forward.
>
> 1: https://git.savannah.gnu.org/cgit/guix/maintenance.git/tree/doc/ROADMAP.org#n71
...
> Let me know if you've got any comments or questions!

Sounds great!

My only real concern, as someone maintaining guix packages in Debian, is
to make sure that we do not break compatibility with being able to use
an older daemon, as Debian stable/bookworm is still at guix 1.4.x and it
would be nice to not have to force people to manually upgrade the daemon
(e.g. and even if a newer version lands in a future Debian stable
release, in general it will stuck using that version for some years as
well).

I have noticed occasional issues with the Debian packages of guix having
compatibility issues when newer versions of guile-git/libgit2,
guile-ssh/libssh2, etc. get introduced, and wonder if the same would
hold true of a daemon?

In Guix, by design you wouldn't really notice these sorts of problems as
it is always generally built with the current version, but Debian does
rely on ABI compatibility for package upgrades... I might be able to
keep better track of these types of issues in Debian, although various
guile-* modules that depend on C libraries seem to avoid the normal
detection mechanisms to trigger rebuilds in Debian.

That is a bit of a tangent, but it reminded me about that issue...


live well,
  vagrant

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

Re: Implementing the guix-dameon in Guile

[Simon Tournier]

Hi,

Really cool!  Congrats Chris! :-)

On Wed, 13 Sep 2023 at 13:13, Maxim Cournoyer <maxim.cournoyer@gmail.com> wrote:

> I had some musing about the daemon recently; I was thinking libguile
> could be added to our old C++ daemon, which could then replace its
> functions piece-wise with Scheme implemented ones?

Maxim, you are proposing to keep the C++ and gradually replace some
parts by implementing these parts in Guile and add a C layer via
libguile to call the Guile code from the C++, right?

If I remember correctly from the last discussion on Guile guix-daemon, I
proposed the converse.  However, the C layer of Maxim’s proposal seems
much easier. ;-)

Some gradual replacements seem a good strategy for the feedback loop.
However, the final design would be more or less the same as the current
one.  Although, I do not know if another design would be possible. :-)

Chris, all the best for this challenging project!  It’s nice that you
have a grant and I am convinced all the ingredients are here for making
this project a success. :-)

Cheers,
simon

Re: Implementing the guix-dameon in Guile

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 1179 bytes --]


Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:

>> The transformation toward a Guile daemon is a point of consistency and
>> pride for the project and therefore unlikely to be second-guessed or
>> reverted. My recommendation is to replace the daemon gradually—working
>> from (apply system* (command-line) downward—and mainline your
>> incremental changes as quickly as possible.
>
> I had some musing about the daemon recently; I was thinking libguile
> could be added to our old C++ daemon, which could then replace its
> functions piece-wise with Scheme implemented ones?
>
> At the end, we'd have almost everything in Scheme, at which point we
> could rewrite the main loop to Scheme and say farewell to the old
> daemon.
>
> Would that approach makes any sense?  I know of at least one C++ project
> integrating libguile, and that's the jami-daemon, so it could perhaps
> provide some clues as to how to proceed to do so.

I think it's a viable approach, but probably not one for me. Since I
don't really write C++, I think that trying to replace parts of the C++
code and keep things working would take more work than just writing
Guile.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

Re: Implementing the guix-dameon in Guile

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 2216 bytes --]


Vagrant Cascadian <vagrant@debian.org> writes:

> On 2023-09-13, Christopher Baines wrote:
>> I think this has been talked about for a while [1], but I want to make it
>> happen. Currently the guix-daemon is still similar to the nix-daemon
>> that it was forked from, and is implemented in C++. I think that a Guile
>> implementation of the guix-daemon will simplify Guix and better support
>> hacking on and around the daemon to add new features and move Guix
>> forward.
>>
>> 1: https://git.savannah.gnu.org/cgit/guix/maintenance.git/tree/doc/ROADMAP.org#n71
> ...
>> Let me know if you've got any comments or questions!
>
> Sounds great!
>
> My only real concern, as someone maintaining guix packages in Debian, is
> to make sure that we do not break compatibility with being able to use
> an older daemon, as Debian stable/bookworm is still at guix 1.4.x and it
> would be nice to not have to force people to manually upgrade the daemon
> (e.g. and even if a newer version lands in a future Debian stable
> release, in general it will stuck using that version for some years as
> well).
>
> I have noticed occasional issues with the Debian packages of guix having
> compatibility issues when newer versions of guile-git/libgit2,
> guile-ssh/libssh2, etc. get introduced, and wonder if the same would
> hold true of a daemon?
>
> In Guix, by design you wouldn't really notice these sorts of problems as
> it is always generally built with the current version, but Debian does
> rely on ABI compatibility for package upgrades... I might be able to
> keep better track of these types of issues in Debian, although various
> guile-* modules that depend on C libraries seem to avoid the normal
> detection mechanisms to trigger rebuilds in Debian.
>
> That is a bit of a tangent, but it reminded me about that issue...

I think compatibility is a priority, and although I haven't looked too
much in to the details yet, I think it's quite realistic.

I think it's very important for Guix to keep compatibility with older
daemons, and on the daemon side, I'd want to see the Guile
implementation work with older Guix's, as well as it being possible to
switch back and forth between the implementations without issue.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

Re: Implementing the guix-dameon in Guile

[Josselin Poiret]

[-- Attachment #1: Type: text/plain, Size: 2404 bytes --]

Hi Chris,

Christopher Baines <mail@cbaines.net> writes:

> Hey!
>
> I think this has been talked about for a while [1], but I want to make it
> happen. Currently the guix-daemon is still similar to the nix-daemon
> that it was forked from, and is implemented in C++. I think that a Guile
> implementation of the guix-daemon will simplify Guix and better support
> hacking on and around the daemon to add new features and move Guix
> forward.
>
> 1: https://git.savannah.gnu.org/cgit/guix/maintenance.git/tree/doc/ROADMAP.org#n71
>
> My plan is to focus on this over the next year. I left my previous day
> job quite a few months ago now to take a bit of a break, that's the main
> reason I've been able to spend more time trying to push forward some of
> the QA stuff. With some monetary support from NLNet [2], I'm planning to
> continue this break and focus for the next year on getting a Guile
> implementation of the guix-daemon written and adopted.

Great and ambitious project! I am also interested in it and am willing
to work on it (although at a much slower pace than you might).

> 2: https://nlnet.nl/project/GuixDaemon-Guile/
>
> Rewrites are risky because you only get the value right at the end,
> therefore the priority is to get a minimal but viable implementation in
> Guile that can be switched to, and not to get distracted on adding or
> improving functionality unnecessarily. That is better done once the new
> implementation has been adopted.
>
> While I think there's a substantial amount of work to do, progress
> towards a Guile guix-daemon has already been made. There was a least one
> GSoC project which did make progress, and there's Guile implementations
> of some of the functionality in Guix already.
>
> Still though, I'd like to hear what people think about which direction
> the implementation should go, and what features they'd like to see. Even
> if those are not essential to make the Guile implementation viable, it
> still might inform the direction to take.

I think the #1 feature for me would be to have it completely
unpriviledged using mount namespaces, so that you could still build
software without needing to run the daemon on the system.  You won't be
able to run the built software without using namespaces as well though,
but that still a step in the right direction imo.

WDYT?
-- 
Josselin Poiret

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 682 bytes --]

Re: Implementing the guix-dameon in Guile

[Ludovic Courtès]

Hi Christopher,

Christopher Baines <mail@cbaines.net> skribis:

> My plan is to focus on this over the next year. I left my previous day
> job quite a few months ago now to take a bit of a break, that's the main
> reason I've been able to spend more time trying to push forward some of
> the QA stuff. With some monetary support from NLNet [2], I'm planning to
> continue this break and focus for the next year on getting a Guile
> implementation of the guix-daemon written and adopted.
>
> 2: https://nlnet.nl/project/GuixDaemon-Guile/

Yay, this is great news!! But also: thank you for your long-term
commitment, it’s an invaluable contribution to the project.

> Rewrites are risky because you only get the value right at the end,
> therefore the priority is to get a minimal but viable implementation in
> Guile that can be switched to, and not to get distracted on adding or
> improving functionality unnecessarily. That is better done once the new
> implementation has been adopted.

In the past I wondered, as Maxim wrote, whether we could move
incrementally—after all, a fair bit of “the daemon” is already in Scheme
(there’s substitute support and other helps, plus (guix store …) etc.)
I’m not sure that’s feasible or desirable though.

My take today :-) is that ‘wip-guile-daemon’ is a great starting point.
We could aim towards having a minimal ‘guix daemon’ (space) command that
would coexist with ‘guix-daemon’ and that people could try out soonish
(I think Caleb got it to build derivations back then).  Eventually less
adventurous people will use it, and at some point it’ll be sufficiently
mature that we can default to ‘guix daemon’ instead of ‘guix-daemon’.

Technically, I think it should be a single-threaded Fibers program,
building on the experience we got from the Coordinator, shepherd, etc.
That should allow us to do everything in one process (in contrast, the
C++ implementation forks for every incoming connection, which then
significantly complicates the implementation of locking, etc.)

I imagine the daemon could be structured as a set of actors (it’s really
my thing these days ;-)), with an eye on facilitating code sharing and
interaction with the Coordinator, Cuirass, and all that.

I hope this makes sense.  Time will tell!

Ludo’.

Re: Implementing the guix-dameon in Guile

[Ludovic Courtès]

Christopher Baines <mail@cbaines.net> skribis:

> I think compatibility is a priority, and although I haven't looked too
> much in to the details yet, I think it's quite realistic.
>
> I think it's very important for Guix to keep compatibility with older
> daemons, and on the daemon side, I'd want to see the Guile
> implementation work with older Guix's, as well as it being possible to
> switch back and forth between the implementations without issue.

+1.  The derivation format and what goes in the build environment (on
GNU/Linux) are set in stone.

The RPC protocol could evolve if needed, preserving backward
compatibility.

Ludo’.

Re: Implementing the guix-dameon in Guile

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 2157 bytes --]


Ludovic Courtès <ludo@gnu.org> writes:

>> Rewrites are risky because you only get the value right at the end,
>> therefore the priority is to get a minimal but viable implementation in
>> Guile that can be switched to, and not to get distracted on adding or
>> improving functionality unnecessarily. That is better done once the new
>> implementation has been adopted.
>
> In the past I wondered, as Maxim wrote, whether we could move
> incrementally—after all, a fair bit of “the daemon” is already in Scheme
> (there’s substitute support and other helps, plus (guix store …) etc.)
> I’m not sure that’s feasible or desirable though.
>
> My take today :-) is that ‘wip-guile-daemon’ is a great starting point.
> We could aim towards having a minimal ‘guix daemon’ (space) command that
> would coexist with ‘guix-daemon’ and that people could try out soonish
> (I think Caleb got it to build derivations back then).  Eventually less
> adventurous people will use it, and at some point it’ll be sufficiently
> mature that we can default to ‘guix daemon’ instead of ‘guix-daemon’.

Yep, my hope is that this can happen over the next year.

> Technically, I think it should be a single-threaded Fibers program,
> building on the experience we got from the Coordinator, shepherd, etc.
> That should allow us to do everything in one process (in contrast, the
> C++ implementation forks for every incoming connection, which then
> significantly complicates the implementation of locking, etc.)

Yep, that sounds good to me.

> I imagine the daemon could be structured as a set of actors (it’s really
> my thing these days ;-)), with an eye on facilitating code sharing and
> interaction with the Coordinator, Cuirass, and all that.

I'm not very familiar with actors, but I guess that's similar to having
a bunch of cooperating fibers which handle different things.

And maybe not in the short term, but if the Guile daemon implementation
is flexible and extensible enough, it should be possible to replace the
build coordinator with some extensions in and around the guix daemon.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

Re: Implementing the guix-dameon in Guile

[Ludovic Courtès]

Christopher Baines <mail@cbaines.net> skribis:

> I'm not very familiar with actors, but I guess that's similar to having
> a bunch of cooperating fibers which handle different things.

Yes.  Specifically, actors here are a fiber together with a channel;
each actor looks like:

  (let loop ((state …))
    (match (get-message channel)
      ;; handle request and call ‘loop’
      ))

The actor keeps serving requests it gets on its channel.

It makes quite a difference when you start framing it this way.  When I
first fiberized Cuirass years ago, I’d just use ‘spawn-fiber’ here and
there where concurrency was needed.  Thinking in terms of actors makes
the code clearer, allows you to think about “concurrency units”, respect
separation of concerns, and so on.  (Our Spritely comrades would explain
this much better!)

Ludo’.

Re: Implementing the guix-dameon in Guile

[Caleb Ristvedt via Development of GNU Guix and the GNU System distribution.]

[-- Attachment #1: Type: text/plain, Size: 9620 bytes --]

My old university email address started demanding a phone number to
"verify the security of my account", which was pretty funny considering
it never had a phone number to begin with, so I'm locked out of
that.  Same GPG public key, though.

Christopher Baines <mail@cbaines.net> writes:

> Hey!
>
> I think this has been talked about for a while [1], but I want to make it
> happen. Currently the guix-daemon is still similar to the nix-daemon
> that it was forked from, and is implemented in C++. I think that a Guile
> implementation of the guix-daemon will simplify Guix and better support
> hacking on and around the daemon to add new features and move Guix
> forward.

I'd like to help with this if at all possible.

> Still though, I'd like to hear what people think about which direction
> the implementation should go, and what features they'd like to see. Even
> if those are not essential to make the Guile implementation viable, it
> still might inform the direction to take.
Okay, brain dump time:

I think that using fibers has a lot of potential, but there are
obstacles that need to be worked around.  In the single-threaded case,
we risk a big slowdown if multiple clients are active at once, since
we're doing what used to be done by n processes with one single thread.
It would be especially noticeable during big disk reads and writes,
since those basically ignore O_NONBLOCK, and most procedures that act on
entire files at once would therefore probably not hit many yield points.
The worst situation would be where multiple worker fibers are attempting
to do reference scanning at the same time.  Posix asynchronous disk IO
could be used, but glibc currently implements it... using a thread pool.
There is the RWF_NOWAIT flag to preadv2, though it's only available on
newer linuxes and has bugs in 5.9 and 5.10.

Additionally, being single-threaded means use of F_SETLKW is a no-go, so
you're stuck with polling there.  Granted, that's not such a big issue if
in 99% of use cases there is only one process doing the locking, so it
can all be managed internally.

Speaking of file locks, the risk of accidental clobbering of locks jumps
way up once it's all moved in to one process, and IIRC we already have
bugs with accidental clobbering of locks.  You can get a half-decent
interface by doing what sqlite does, which is a combination of
intra-process locking and holding on to open file descriptors until all
locks on the underlying file are released.  There are some subtle
pathological cases there that are a lot more likely in the guix daemon
than in sqlite, though.  For example, suppose you open a file twice to
get ports p1 and p2, acquire read locks on both of them, then close p1,
then open the file again to get p3, acquire a read lock on it, close p2,
get p4, acquire a read lock on it, close p3, get p5... and so on.  This
will cause unbounded file descriptor usage, and eventually you'll run
out.  There is no workaround in this model other than "hope that usage
pattern doesn't come up much".  Additionally, you need to ensure that
every close of a potentially-locked file goes through a special
close-wrapper.

I'm actually in the middle of working on a solution for this that
involves a separate locker process that gets passed file descriptors to
lock via a unix socket.

Speaking of file descriptors, running the entire daemon in one process
is going to mean much higher pressure on file descriptor resource usage.
IIRC, while building a derivation, the closure of its inputs needs to be
locked, and that means a file descriptor for each and every store item
in its input closure, simultaneously.  The separate locker process would
make it possible to retain those locks while not having them open in the
main process.

Another issue that will need to be addressed, whether single-threaded or
not, is the use of memoization caches in various places.  These aren't
weak hash tables, so they are both not-thread-safe and will retain
strong references to both the cached results and the arguments used to
obtain them for as long as the procedure it is based on remains.  In a
long-running server process, this is less than ideal.  One approach
could be to put a bound on how large they can grow, with some eviction
policy for deciding what gets discarded first.  If memoization is used
to ensure pointer equality as a matter of correctness, though, that
probably won't work well.  The simplest solution would probably be to
change them to use weak hash tables, though perhaps with an option
available to bring back non-weak hash tables on the client side.

In the multithreaded case, fork() and clone() become concerns, since
they can no longer be safely run from guile.  One way around this would
be to use posix_spawn to produce a single-threaded guile process, then
have that do the fork or clone as necessary.  The fork case shouldn't
actually be necessary, though, as the child process can just exec
directly.  In the clone case, CLONE_PARENT can be used to make the
resulting process a child of the original, main process, though I don't
know how portable that is to hurd (really, I don't know how namespace
setup works in general on hurd).  Instead of doing this
spawn-two-processes-to-spawn-one routine every time we want to set up a
container, we could create a spawner-helper process once and just keep
it around.  If we can do that before any threads are created, we don't
even need posix_spawn (though it is nice to have around, and I do have
bindings to it).  I remember reading that that's what the apache web
server did.

This would however mean some places would need to use interfaces like
"eval-with-container" instead of "call-with-container", which is
somewhat less convenient.  But code staging shouldn't be a terribly
foreign concept to most guixers.

Another concern is child process management; a blocking waitpid will of
course block the calling thread, so something like a SIGCHLD handler or
a dedicated reaper thread would be needed in order to simulate a
blocking waitpid.  Personally I think it would be a good idea to go with
something like Shepherd's process monitor approach, but with some
changes.  First, move child reaping into the process monitor itself, so
that all the SIGCHLD handler does is send a notification to the process
monitor (and it should do this via a condition variable, not a channel,
so that it doesn't block, since asyncs run on whatever fiber happens to
be current on that thread's scheduler at the time, and this means it is
possible for a signal's handler to be run from within the process
monitor fiber).  Second, wrap all process-spawning procedures such that
they now return <process> objects instead of PIDs.  A <process> object
contains a PID, a condition variable signaled when the process is
terminated, and a slot for holding the exit status.  Immediately before
spawning a process, send a message to the process monitor temporarily
disabling reaping, then spawn the process, create the <process> object,
and register it with the process monitor, resuming reaping at the same
time.  Then a waitpid replacement can very easily operate on these
process objects.

Sqlite is yet another concern.  I haven't yet looked at how you've
handled this in the build coordinator, but I'm curious.  Any blocking
interface it has, such as a busy handler, isn't going to work very well.
We could wrap the sqlite procedures with ones that retry with
exponential backoff (which is what "PRAGMA busy_timeout = ..." does
internally).  That would work, though not optimally.  I'm not sure of a
better way, though - https://www.sqlite.org/c3ref/unlock_notify.html
looks sort of right, but the documentation says that's just for "shared
cache mode".  It seems a bit of a shame to keep guessing at when the
database "might" next be available when the other entities accessing the
database may well be in the very same process and so could just give us
an earliest-possible checking point directly.

Those are all the design concerns that I had off the top of my head, I
might recall some more later on.  Personally I think it would be prudent
to design as if for multiple threads.

On the subject of features, I would like it if downloaders (fixed-output
derivations) had access to /gnu/store/.links so that they can easily
look up whether a file with a given hash already exists, and copy it
over if so.  Often when writing a package definition I'll run "guix
download" to get the hash, and that will put it in the store as a side
effect, but then when it comes time to build the package it will
re-download it all over again because the file name is different.
Technically that should already be achievable just by tweaking
chroot-dirs and the downloaders.  It would also be nice if the same
concept could be applied to directories, such as git repositories -
perhaps a /gnu/store/.dirlinks with symbolic links?  Of course, those
wouldn't be used for deduplication, just for easy access for
fixed-output derivations.  A similar approach could also include e.g. a
mounted DVD with The GNU Source Distribution on it.

Oh, another feature that just occurred to me - the ability to
automatically retry a failed derivation with fewer builds in parallel,
or depending on system load or available memory.  It's quite annoying to
run a guix command that is supposed to take multiple days, and have it
end up taking more than a week because it fails several times each day -
often bringing down several hours worth of progress with it due to very
large derivations - because of some small package's flaky tests that
fail under high load.

- reepca

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 515 bytes --]

Re: Implementing the guix-dameon in Guile

[Mekeor Melire]

2023-09-13 16:36 mail@cbaines.net:

> I think this has been talked about for a while [1], but I want 
> to make it happen. Currently the guix-daemon is still similar to 
> the nix-daemon that it was forked from, and is implemented in 
> C++. I think that a Guile implementation of the guix-daemon will 
> simplify Guix and better support hacking on and around the 
> daemon to add new features and move Guix forward.

Yippie! I'm really excited about this; and convinced that it'll greatly impact and improve the Guix project.

> I'd like to hear what people think about which direction the 
> implementation should go, and what features they'd like to see.

Here's a feature-request (in the long-run). But I'm not sure if I understand correctly that a new, improved Guix-daemon enables us to implement this. Sorry if I'm wrong. I'd like to be able to step through the build-process of a package phase-by-phase and see what changed after each phase by looking at the file-tree.

Re: Implementing the guix-dameon in Guile

[Mathieu Othacehe]

Hey Chris,

Congrats on getting the grant! I'm sure it will lead to a lot of
positive things. I tried it myself a couple years ago, for
the exact same subject, without success. 

> I imagine the daemon could be structured as a set of actors (it’s really
> my thing these days ;-)), with an eye on facilitating code sharing and
> interaction with the Coordinator, Cuirass, and all that.

I think this point is really important here. If the offload mechanism of
the existing guix-daemon was scaling correctly, with a better API, we
would probably never had to develop the Cuirass remote worker and the
Build Coordinator mechanisms.

Perhaps, part of the Build Coordinator code could be one day integrated
inside the new guix-daemon so that we can only use one offload mechanism
for all use cases. Users of the current "guix offload" as well as CI
tools could rely upon a unique offload mechanism.

The three other points I had in mind when proposing the subject to NLNet
were to:

- Fix the sqlite contention issues that have been observed on Berlin and
  other machines with huge local databases.

- Remove the GC lock: I think it has been done on recent nix-daemon.

- Add support for unprivileged daemon.

I don't know if it fits your plans but thought it could be interesting
to share that.

Good luck,

Mathieu

Re: Implementing the guix-dameon in Guile

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 1443 bytes --]


Josselin Poiret <dev@jpoiret.xyz> writes:

>> 2: https://nlnet.nl/project/GuixDaemon-Guile/
>>
>> Rewrites are risky because you only get the value right at the end,
>> therefore the priority is to get a minimal but viable implementation in
>> Guile that can be switched to, and not to get distracted on adding or
>> improving functionality unnecessarily. That is better done once the new
>> implementation has been adopted.
>>
>> While I think there's a substantial amount of work to do, progress
>> towards a Guile guix-daemon has already been made. There was a least one
>> GSoC project which did make progress, and there's Guile implementations
>> of some of the functionality in Guix already.
>>
>> Still though, I'd like to hear what people think about which direction
>> the implementation should go, and what features they'd like to see. Even
>> if those are not essential to make the Guile implementation viable, it
>> still might inform the direction to take.
>
> I think the #1 feature for me would be to have it completely
> unpriviledged using mount namespaces, so that you could still build
> software without needing to run the daemon on the system.  You won't be
> able to run the built software without using namespaces as well though,
> but that still a step in the right direction imo.
>
> WDYT?

Thanks for the suggestion :)

I'm not quite sure what this would involve, but it sounds like it might
not be that hard to do.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

Re: Implementing the guix-dameon in Guile

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 10118 bytes --]


Caleb Ristvedt <caleb@russelstein.xyz> writes:

>> Still though, I'd like to hear what people think about which direction
>> the implementation should go, and what features they'd like to see. Even
>> if those are not essential to make the Guile implementation viable, it
>> still might inform the direction to take.
> Okay, brain dump time:
>
> I think that using fibers has a lot of potential, but there are
> obstacles that need to be worked around.  In the single-threaded case,
> we risk a big slowdown if multiple clients are active at once, since
> we're doing what used to be done by n processes with one single thread.
> It would be especially noticeable during big disk reads and writes,
> since those basically ignore O_NONBLOCK, and most procedures that act on
> entire files at once would therefore probably not hit many yield points.
> The worst situation would be where multiple worker fibers are attempting
> to do reference scanning at the same time.  Posix asynchronous disk IO
> could be used, but glibc currently implements it... using a thread pool.
> There is the RWF_NOWAIT flag to preadv2, though it's only available on
> newer linuxes and has bugs in 5.9 and 5.10.
>
> Additionally, being single-threaded means use of F_SETLKW is a no-go, so
> you're stuck with polling there.  Granted, that's not such a big issue if
> in 99% of use cases there is only one process doing the locking, so it
> can all be managed internally.

I think a thread pool is essential for using SQLite, so we're already
going to have thread pools interacting with fibers, so if more cases
come up where this is needed, then I think it's an option.

> Speaking of file locks, the risk of accidental clobbering of locks jumps
> way up once it's all moved in to one process, and IIRC we already have
> bugs with accidental clobbering of locks.  You can get a half-decent
> interface by doing what sqlite does, which is a combination of
> intra-process locking and holding on to open file descriptors until all
> locks on the underlying file are released.  There are some subtle
> pathological cases there that are a lot more likely in the guix daemon
> than in sqlite, though.  For example, suppose you open a file twice to
> get ports p1 and p2, acquire read locks on both of them, then close p1,
> then open the file again to get p3, acquire a read lock on it, close p2,
> get p4, acquire a read lock on it, close p3, get p5... and so on.  This
> will cause unbounded file descriptor usage, and eventually you'll run
> out.  There is no workaround in this model other than "hope that usage
> pattern doesn't come up much".  Additionally, you need to ensure that
> every close of a potentially-locked file goes through a special
> close-wrapper.
>
> I'm actually in the middle of working on a solution for this that
> involves a separate locker process that gets passed file descriptors to
> lock via a unix socket.
>
> Speaking of file descriptors, running the entire daemon in one process
> is going to mean much higher pressure on file descriptor resource usage.
> IIRC, while building a derivation, the closure of its inputs needs to be
> locked, and that means a file descriptor for each and every store item
> in its input closure, simultaneously.  The separate locker process would
> make it possible to retain those locks while not having them open in the
> main process.

Maybe that will still need to happen, however I think it might be
possible to replace the IPC through locking files with inter-fiber
communication inside of the dameon.

Backwards compatibility is the priority though, so this can only happen
in cases where that's unaffected.

> In the multithreaded case, fork() and clone() become concerns, since
> they can no longer be safely run from guile.  One way around this would
> be to use posix_spawn to produce a single-threaded guile process, then
> have that do the fork or clone as necessary.  The fork case shouldn't
> actually be necessary, though, as the child process can just exec
> directly.  In the clone case, CLONE_PARENT can be used to make the
> resulting process a child of the original, main process, though I don't
> know how portable that is to hurd (really, I don't know how namespace
> setup works in general on hurd).  Instead of doing this
> spawn-two-processes-to-spawn-one routine every time we want to set up a
> container, we could create a spawner-helper process once and just keep
> it around.  If we can do that before any threads are created, we don't
> even need posix_spawn (though it is nice to have around, and I do have
> bindings to it).  I remember reading that that's what the apache web
> server did.
>
> This would however mean some places would need to use interfaces like
> "eval-with-container" instead of "call-with-container", which is
> somewhat less convenient.  But code staging shouldn't be a terribly
> foreign concept to most guixers.
>
> Another concern is child process management; a blocking waitpid will of
> course block the calling thread, so something like a SIGCHLD handler or
> a dedicated reaper thread would be needed in order to simulate a
> blocking waitpid.  Personally I think it would be a good idea to go with
> something like Shepherd's process monitor approach, but with some
> changes.  First, move child reaping into the process monitor itself, so
> that all the SIGCHLD handler does is send a notification to the process
> monitor (and it should do this via a condition variable, not a channel,
> so that it doesn't block, since asyncs run on whatever fiber happens to
> be current on that thread's scheduler at the time, and this means it is
> possible for a signal's handler to be run from within the process
> monitor fiber).  Second, wrap all process-spawning procedures such that
> they now return <process> objects instead of PIDs.  A <process> object
> contains a PID, a condition variable signaled when the process is
> terminated, and a slot for holding the exit status.  Immediately before
> spawning a process, send a message to the process monitor temporarily
> disabling reaping, then spawn the process, create the <process> object,
> and register it with the process monitor, resuming reaping at the same
> time.  Then a waitpid replacement can very easily operate on these
> process objects.

I think the work on the shepherd has probably made things much easier in
this area, it is probably going to be tricky though as this isn't
something I've done much with before.

> Sqlite is yet another concern.  I haven't yet looked at how you've
> handled this in the build coordinator, but I'm curious.  Any blocking
> interface it has, such as a busy handler, isn't going to work very well.
> We could wrap the sqlite procedures with ones that retry with
> exponential backoff (which is what "PRAGMA busy_timeout = ..." does
> internally).  That would work, though not optimally.  I'm not sure of a
> better way, though - https://www.sqlite.org/c3ref/unlock_notify.html
> looks sort of right, but the documentation says that's just for "shared
> cache mode".  It seems a bit of a shame to keep guessing at when the
> database "might" next be available when the other entities accessing the
> database may well be in the very same process and so could just give us
> an earliest-possible checking point directly.

As I say above, I think the way to use SQLite is a thread pool. So you
have a channel to talk to the thread pool, you put-message on it with
some query/lambda and a response channel, then get-message on the
response channel. The thread pool thread takes care of reading and
writing to the channel, as well as calling in to the SQLite code to
actually execute the query.

> Those are all the design concerns that I had off the top of my head, I
> might recall some more later on.  Personally I think it would be prudent
> to design as if for multiple threads.
>
> On the subject of features, I would like it if downloaders (fixed-output
> derivations) had access to /gnu/store/.links so that they can easily
> look up whether a file with a given hash already exists, and copy it
> over if so.  Often when writing a package definition I'll run "guix
> download" to get the hash, and that will put it in the store as a side
> effect, but then when it comes time to build the package it will
> re-download it all over again because the file name is different.
> Technically that should already be achievable just by tweaking
> chroot-dirs and the downloaders.  It would also be nice if the same
> concept could be applied to directories, such as git repositories -
> perhaps a /gnu/store/.dirlinks with symbolic links?  Of course, those
> wouldn't be used for deduplication, just for easy access for
> fixed-output derivations.  A similar approach could also include e.g. a
> mounted DVD with The GNU Source Distribution on it.

I guess this would be nice, although you'd want to not couple fixed
output derivations with part of the daemon implementation. Maybe the
daemon could just look at the fixed output derivation though and work
out that it can do the work by grabbing the data and arranging it
properly, and avoid running the builder altogether.

> Oh, another feature that just occurred to me - the ability to
> automatically retry a failed derivation with fewer builds in parallel,
> or depending on system load or available memory.  It's quite annoying to
> run a guix command that is supposed to take multiple days, and have it
> end up taking more than a week because it fails several times each day -
> often bringing down several hours worth of progress with it due to very
> large derivations - because of some small package's flaky tests that
> fail under high load.

I think there's a whole load of interesting features in this area. The
build coordinator has a way of doing automatic retries, on the agents it
also has some tuned delays for starting new builds based on the system
load. There's more I think that could be done in terms of packing builds
on to a system to make good use of the hardware, whilst avoiding
resource based failures.

Thanks for your long email!

Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

Re: Implementing the guix-dameon in Guile

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 1522 bytes --]


Mekeor Melire <mekeor@posteo.de> writes:

> 2023-09-13 16:36 mail@cbaines.net:
>
>> I think this has been talked about for a while [1], but I want to
>> make it happen. Currently the guix-daemon is still similar to the
>> nix-daemon that it was forked from, and is implemented in C++. I
>> think that a Guile implementation of the guix-daemon will simplify
>> Guix and better support hacking on and around the daemon to add new
>> features and move Guix forward.
>
> Yippie! I'm really excited about this; and convinced that it'll
> greatly impact and improve the Guix project.
>
>> I'd like to hear what people think about which direction the
>> implementation should go, and what features they'd like to see.
>
> Here's a feature-request (in the long-run). But I'm not sure if I
> understand correctly that a new, improved Guix-daemon enables us to
> implement this. Sorry if I'm wrong. I'd like to be able to step
> through the build-process of a package phase-by-phase and see what
> changed after each phase by looking at the file-tree.

This definitely isn't in the minimal scope, but it's something that I
think having the Guile daemon will maybe make easier to do.

Given the phases are running inside the builder script, I think you'd
need to have that cooperate somehow, and with the help of the daemon
connect somehow to the user so that they can pause and resume the build
process, and inspect the internal state of the builder. The daemon could
also provide access to process/filesystem information as well.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

Re: Implementing the guix-dameon in Guile

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 1742 bytes --]


Mathieu Othacehe <othacehe@gnu.org> writes:

>> I imagine the daemon could be structured as a set of actors (it’s really
>> my thing these days ;-)), with an eye on facilitating code sharing and
>> interaction with the Coordinator, Cuirass, and all that.
>
> I think this point is really important here. If the offload mechanism of
> the existing guix-daemon was scaling correctly, with a better API, we
> would probably never had to develop the Cuirass remote worker and the
> Build Coordinator mechanisms.

Indeed.

> Perhaps, part of the Build Coordinator code could be one day integrated
> inside the new guix-daemon so that we can only use one offload mechanism
> for all use cases. Users of the current "guix offload" as well as CI
> tools could rely upon a unique offload mechanism.

Yeah, I think the ideal situation would be to have a daemon that's
flexible and extensible enough to allow this, without being too
complicated or fragile.

> The three other points I had in mind when proposing the subject to NLNet
> were to:
>
> - Fix the sqlite contention issues that have been observed on Berlin and
>   other machines with huge local databases.

Yeah, I think there's some simple things that can be done to improve the
use of SQLite, and then some instrumentation can be added so that any
performance issues can be better understood.

> - Remove the GC lock: I think it has been done on recent nix-daemon.

I'm hoping that with some careful design of how to do garbage
collection, this should be possible.

> - Add support for unprivileged daemon.

I'm not quite sure what is required for this, but maybe it would be
possible without too much effort.

Thanks for your input!

Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

Re: Implementing the guix-dameon in Guile

[Thompson, David]

Hey Christopher and Ludovic,

First, congrats on the NLnet funding! I've wanted to see this project
happen for years so I'm excited to see it finally happening.

On Thu, Sep 14, 2023 at 9:30 AM Ludovic Courtès <ludo@gnu.org> wrote:
>
> Hi Christopher,
>
> Christopher Baines <mail@cbaines.net> skribis:
>
> > My plan is to focus on this over the next year. I left my previous day
> > job quite a few months ago now to take a bit of a break, that's the main
> > reason I've been able to spend more time trying to push forward some of
> > the QA stuff. With some monetary support from NLNet [2], I'm planning to
> > continue this break and focus for the next year on getting a Guile
> > implementation of the guix-daemon written and adopted.
> >
> > 2: https://nlnet.nl/project/GuixDaemon-Guile/
>
> Yay, this is great news!! But also: thank you for your long-term
> commitment, it’s an invaluable contribution to the project.
>
> ...
>
> I imagine the daemon could be structured as a set of actors (it’s really
> my thing these days ;-)), with an eye on facilitating code sharing and
> interaction with the Coordinator, Cuirass, and all that.

Sounds like an excellent idea to us Spritely folks, too! :)

Christopher, if you ever need help with actor model related things,
come talk to us in #spritely on libera.chat.

- Dave

Re: Implementing the guix-dameon in Guile

[Yarl via Development of GNU Guix and the GNU System distribution.]

Hello,

I saw this and the blog post about it just now. This is very
interesting!
I have started working on this on my own, in the hope of eventually
sharing something useful. However, I am a guile (and guix) novice and I
don't have plenty of time and my job is completely unrelated to
computers. That is/was a mean for me to learn more about guix and guile,
by the way.
I thought that because it was in the roadmap in low priority
I could take my time.
Anyway I am glad someone is working on this. What I have done so far is
probably not useful at all because it's not much, very far from complete
and probably a lot messy. Whatever, if I could help in any way please
tell. Do you work on a specific branch or some external repository? Do
you plan to work with others at some point or do you want to want rough
it on your own for now?

I look forward to hearing from you. Thanks!