Re: [GSoC] Porting GuixSD to GNU Hurd draft

[ludo@gnu.org (Ludovic Courtès)]

Hello!

Manolis Ragkousis <manolis837@gmail.com> skribis:

> Although I have uploaded and shared my draft to the GSoC website, I am
> also resending it to the lists as per Ludovic's instruction.

Yeah, the GSoC website makes it possible to provide a link to a text
file, so let’s do that instead of using their SaaSS.

> 2. The Project
>
> The project consists of four main stages
>
> 1. Modify Guix so it will be able to create and mount the file-system needed to boot into a system with Hurd at its core. 
> 2. Modify Guix so it can produce a working image, while isolating any cases of Linux assumptions.
> 3. Successfully boot into one such system using GNU Shepherd with pid 1.
> 4. Modify the new Guix system to take advantage of Hurd specific mechanisms.
>
>
> Currently the tools Guix uses to interact with the filesystem exist inside the (guix build syscalls) module. 
> This module provides bindings to libc's syscall wrappers, which are only available on a GNU/Linux system. 
> In order to offer the same functionality on a GNU/Hurd system we must first write Guile bindings for the 
> relevant Hurd functions, like 'file_set_translator' in hurd/fs.defs
> for example.

Note that technically the ‘file_set_translator’ function is in libc:

--8<---------------cut here---------------start------------->8---
scheme@(guile-user)> (dynamic-func "file_set_translator" (dynamic-link))
$1 = #<pointer 0x1675660>
--8<---------------cut here---------------end--------------->8---

> This module will be called (guix build hurd). This allows us to
> re-implement the functionality of the 'settrans' command, as described
> here[1], in Scheme and use that in place of 'mount()', where
> applicable.

Good!  (We might as well call it (hurd …) in fact: (hurd fs) would
correspond to fs.defs, (hurd io) for io.defs, and so on.)

> Additionally, we need to make sure that all modules in (guix build *) and (gnu build *) can offer the same 
> functionalities on a GNU/Hurd system.  For example (gnu build vm) relies on QEMU's '-kernel' command-line 
> option, which is Linux-specific.  On the other hand, in the case of modules like (gnu build linux-boot) or 
> (gnu build linux-initrd), which by design are Linux-specific, we will need to provide a separate module with 
> equivalent functionality. (gnu system *) modules will require changes as well, in order to be able to use 
> modifications that will happen in the (guix build *) and (gnu build *) modules. 

I think it’s important to think about:

  1. How functionality equivalent to linux-{initrd,boot} will be
     implemented on the Hurd.

     In my experience the equivalent thing is simpler on the Hurd,
     esp. if relying on passive translators (see daemons/runsystem.sh in
     the Hurd), though here we’ll probably explicitly activate
     translators at boot time.

  2. How to structure GuixSD code in a way that abstracts the underlying
     OS kernel.

     For instance, we could have a (guix build os) module providing an
     API that works for both kernels, probably close to the Linux one,
     and that would dispatch to the right implementation, the Linux or
     Hurd one.

Both of these are quite a bit of design and implementation work that we
should not underestimate.

> Finally, once GuixSD is successfully ported, we can cater to the last stage of taking advantage of Hurd specific 
> mechanisms. 
> This includes but is not limited to:
> 1)Replacing (gnu build linux-container) with (gnu build subhurd) when on a GNU/Hurd system. Subhurds can offer 
> isolation similar to Linux containers as described here[3].

This is really super optional IMO.  This module is only used by ‘guix
environment --container’, which is an optional feature.

> 2)Modify the guix-daemon to run without root privileges by utilizing the Hurd architecture. The daemon needs root 
> privileges in order to setup chrooted environments for the builds.  In the Hurd this can be done by setting up a 
> translator as described here[4]. 

This is more important, and already non-trivial work.  If libc provided
‘mount’ with support for MS_BIND (which I think it should), it would
help a bit, but there’s still the problem of the other Linux name spaces
that are used (the CLONE_NEW* clone(2) flags.)

Thus it may make more sense to write this functionality in guix-daemon
using directly the Hurd interfaces.  Separate PID name spaces, UID name
spaces, mount name spaces, etc. are fairly natural on the Hurd: it’s
“just” a matter of giving the child process ports to separate proc,
auth, etc. translators.

In itself this is also a bit of work.  I wonder what the Hurd folks
think about priorities here?

> 3)Guix uses symlink trees for user profiles. Instead we can use stowfs[5]. Stowfs creates a traditional Unix directory 
> structure from all the files in the individual package directories.

Fun but optional.  ;-)

> 3. Estimated Project Timeline
>
> Before March 31
> Finish merging the wip-hurd branch to upstream.

Note that this task depends on, ahem, other people as well.  ;-)

> Write a (guix build hurd) module which will contain Guile bindings to the RPC stubs like hurd/fs.defs or hurd/exec.defs .
>
> April 1 - April 15
> Package missing dependencies (Hurd libs).
> Re-implement 'settrans' in scheme.

Don’t just straight into low-level “details.”  I’d recommend
familiarizing yourself with GuixSD on GNU/Linux, fiddling with bits of
code and so on (you can try safely in a VM with ‘guix system vm’), and
then, familiarizing yourself with the GNU/Hurd startup process (looking
at daemons/runsystem.sh already gives a good idea.)

> April 16 - May 1
> At this point we will have the tools needed to build a Hurd based file-system. (Milestone 1)

“Hurd-based file system”?

> May 2 - May 22
> Create (gnu build hurd-boot) and (gnu build hurd-initrd).
> Start working on describing the GNU/Hurd system in (gnu system).

I know Debian at some point added initrd support to GNU Mach for some
reason, but fundamentally, the whole point of multiboot is to provide a
solution more flexible than initrds.  So, hopefully, no initrds here.
Since there’s no initrd, there’s also no ‘switch_root’ dance (on
Linux-based system the initrd is the initial root file system, so you
need to switch to the real root file system as soon as you’re able to
mount it), which simplifies things.

Justus, Samuel, WDYT?

> May 23 - 12 June
> Modify (gnu system *) modules as needed.
> All the modules (guix build *) and (gnu build *) will be working as expected by now.
> Try building a GuixSD image. (Milestone 2)

This is the crux of the matter.  :-)

Before starting, it would be nice to have a rough idea of how GuixSD
startup will work on the Hurd, and what changes this entails.

For debugging purposes, it would be very helpful to say the least to
have a working ‘guix system vm’: it would allow you to test your changes
very quickly, without rebooting and so on.

This in itself requires some thought: currently (guix system vm) relies
on QEMU’s ‘-kernel’ option to boot the kernel Linux.  For GNU/Hurd, we
instead need to boot a complete image with GRUB, like ‘guix system
vm-image’ does.  You’ll have to investigate how to port this.

> 13 June - 23 June
> Solve any problems with booting into the system and running GNU Shepherd.
>
> 24 June - 9 July
> Have a fully working GNU/Hurd system. (Milestone 3)
> Make sure all the services/packages run correctly on the new GuixSD/Hurd system and solve any issues.
>
> 10 July - 8 August
> Start working on getting Hurd specific mechanisms integrated to Guix.

What do you mean?

> 9 August - 23 August
> By now all the objectives will have been achieved.
> Merge patches, code refactoring, documentation.
> Deliver a working GuixSD system image with Hurd as the kernel.

Victory!  :-)

I think this is super cool, and also super ambitious!  I’d expect that
we’d be able to reach milestone #2 if everything goes well (and assuming
we don’t try to address isolation in guix-daemon), but milestone #3 is
most likely for later.

What do people think?

The main question is whether you should implement build isolation in
guix-daemon, in which case that would leave little time for the GuixSD
parts.  I think I would rather let you focus on the GuixSD stuff as you
wrote, but I’d like to hear what the Hurd folks think.

Justus, Samuel: could you add yourselves as official co-mentors on
Google’s web site, if not already done?  :-)

Thanks, Manolis!

Ludo’.