Re: [GSoC] Porting GuixSD to GNU Hurd draft

[Manolis Ragkousis <manolis837@gmail.com>]

Hello Justus, Hello Ludo

On 03/23/16 18:55, Justus Winter wrote:
> Quoting Ludovic Courtès (2016-03-23 14:40:38)
>>> 2. The Project
>>>
>>> The project consists of four main stages
>>>
>>> 1. Modify Guix so it will be able to create and mount the file-system needed to boot into a system with Hurd at its core. 
>>> 2. Modify Guix so it can produce a working image, while isolating any cases of Linux assumptions.
>>> 3. Successfully boot into one such system using GNU Shepherd with pid 1.
>>> 4. Modify the new Guix system to take advantage of Hurd specific mechanisms.
> 
> For me, 4. is the most important bit, so we can build packages in
> isolation.

I think our priority should be to first get GuixSD working and then
concentrate to achieving isolation.  From what I understand none of the
two is trivial but in the long run the ability to spawn GNU/Hurd system
vms on the fly will make it easier to work on it.

>>> Currently the tools Guix uses to interact with the filesystem exist inside the (guix build syscalls) module. 
>>> This module provides bindings to libc's syscall wrappers, which are only available on a GNU/Linux system. 
>>> In order to offer the same functionality on a GNU/Hurd system we must first write Guile bindings for the 
>>> relevant Hurd functions, like 'file_set_translator' in hurd/fs.defs
>>> for example.
>>
>> Note that technically the ‘file_set_translator’ function is in libc:
>>
>> --8<---------------cut here---------------start------------->8---
>> scheme@(guile-user)> (dynamic-func "file_set_translator" (dynamic-link))
>> $1 = #<pointer 0x1675660>
>> --8<---------------cut here---------------end--------------->8---
>>
>>> This module will be called (guix build hurd). This allows us to
>>> re-implement the functionality of the 'settrans' command, as described
>>> here[1], in Scheme and use that in place of 'mount()', where
>>> applicable.
> 
> Right.  In fact, what settrans (or mount) does isn't that hard to
> reproduce, though I wouldn't mind moving the c implementation to
> libhurdutil or the like and binding to that.

Then I think it will be wiser to move the c implementation of settrans
to libhurdutil and binding to that.  Then we will only need a (guix
build hurd) module which will include that binding.  I will update the
proposal if you agree.

>> I think it’s important to think about:
>>
>>   1. How functionality equivalent to linux-{initrd,boot} will be
>>      implemented on the Hurd.
>>
>>      In my experience the equivalent thing is simpler on the Hurd,
>>      esp. if relying on passive translators (see daemons/runsystem.sh in
>>      the Hurd), though here we’ll probably explicitly activate
>>      translators at boot time.
> 
> Currently, there is not really a reason to have an initrd-like
> solution on the Hurd.  Yes, one has to start the default pager
> explicitly, but that's about it.

Then it seems we only need a hurd-boot module which will take care of
activating translators and booting the system.

>   2. How to structure GuixSD code in a way that abstracts the
> underlying OS kernel.
>
>      For instance, we could have a (guix build os) module providing an
>      API that works for both kernels, probably close to the Linux one,
>      and that would dispatch to the right implementation, the Linux or
>      Hurd one.

This module sounds like a great idea. I will add it to the proposal.


>>> Finally, once GuixSD is successfully ported, we can cater to the last stage of taking advantage of Hurd specific 
>>> mechanisms. 
>>> This includes but is not limited to:
>>> 1)Replacing (gnu build linux-container) with (gnu build subhurd) when on a GNU/Hurd system. Subhurds can offer 
>>> isolation similar to Linux containers as described here[3].
>>
>> This is really super optional IMO.  This module is only used by ‘guix
>> environment --container’, which is an optional feature.
> 
> Yes, subhurds are more on the experimental side imho.

So Adding subhurd support -> if there is time.
> 
>>> 2)Modify the guix-daemon to run without root privileges by utilizing the Hurd architecture. The daemon needs root 
>>> privileges in order to setup chrooted environments for the builds.  In the Hurd this can be done by setting up a 
>>> translator as described here[4]. 
>>
>> This is more important, and already non-trivial work.  If libc provided
>> ‘mount’ with support for MS_BIND (which I think it should), it would
>> help a bit, but there’s still the problem of the other Linux name spaces
>> that are used (the CLONE_NEW* clone(2) flags.)
>>
>> Thus it may make more sense to write this functionality in guix-daemon
>> using directly the Hurd interfaces.  Separate PID name spaces, UID name
>> spaces, mount name spaces, etc. are fairly natural on the Hurd: it’s
>> “just” a matter of giving the child process ports to separate proc,
>> auth, etc. translators.
>>
>> In itself this is also a bit of work.  I wonder what the Hurd folks
>> think about priorities here?
> 
> I'd go for specializing guix-daemon over trying too hard to implement
> Linux-compatible interfaces in the libc.  I consider the
> filesystem-isolation part easy, UID isolation relatively easy, PID
> isolation is a bit tricky.  Currently one cannot simply start another
> proc server and expect it to work as it needs privileged kernel
> interfaces.  I created an RPC to allow nested proc servers for
> unprivileged subhurds.  That should do the trick, it is merely a
> matter of making it actually work I guess.

As I said above I believe that first getting GuixSD to work will make it
easier to work on this and test any changes.

>>> 3)Guix uses symlink trees for user profiles. Instead we can use stowfs[5]. Stowfs creates a traditional Unix directory 
>>> structure from all the files in the individual package directories.
>>
>> Fun but optional.  ;-)
> 
> Agreed.

Stowfs support -> if there is time.

>> Before March 31
>> Finish merging the wip-hurd branch to upstream.
>
> Note that this task depends on, ahem, other people as well.  ;-)

Well one can only hope :-)

>>> May 2 - May 22
>>> Create (gnu build hurd-boot) and (gnu build hurd-initrd).
>>> Start working on describing the GNU/Hurd system in (gnu system).
>>
>> I know Debian at some point added initrd support to GNU Mach for some
>> reason, but fundamentally, the whole point of multiboot is to provide a
>> solution more flexible than initrds.  So, hopefully, no initrds here.
>> Since there’s no initrd, there’s also no ‘switch_root’ dance (on
>> Linux-based system the initrd is the initial root file system, so you
>> need to switch to the real root file system as soon as you’re able to
>> mount it), which simplifies things.
>>
>> Justus, Samuel, WDYT?
> 
> Debian has the initrd for the live cds, but that is a bit hacky.  I
> don't believe we need an initrd at this point.

(gnu build hurd-initrd) -> dropped

>>> May 23 - 12 June
>>> Modify (gnu system *) modules as needed.
>>> All the modules (guix build *) and (gnu build *) will be working as expected by now.
>>> Try building a GuixSD image. (Milestone 2)
>>
>> This is the crux of the matter.  :-)
>>
>> Before starting, it would be nice to have a rough idea of how GuixSD
>> startup will work on the Hurd, and what changes this entails.

Currently I am working on familiarizing myself with both how Hurd and
GuixSD booting works. I am already using GuixSD as a system and I am
going though the source code so I can better understand what is going on.
>>
>> For debugging purposes, it would be very helpful to say the least to
>> have a working ‘guix system vm’: it would allow you to test your changes
>> very quickly, without rebooting and so on.
>>
>> This in itself requires some thought: currently (guix system vm) relies
>> on QEMU’s ‘-kernel’ option to boot the kernel Linux.  For GNU/Hurd, we
>> instead need to boot a complete image with GRUB, like ‘guix system
>> vm-image’ does.  You’ll have to investigate how to port this.
> 
> qemu can boot multiboot operating systems.

Justus is correct here. I found this
https://www.gnu.org/software/hurd/hurd/running/qemu.html#index9h1 which
explains how to make qemu start with gnumach.  This way guix system vm
can work, just with the proper modifications.

>>> Deliver a working GuixSD system image with Hurd as the kernel.
> 
> Hurd is not a kernel.

I will rephrase it correctly.  I am sorry again.

>> The main question is whether you should implement build isolation in
>> guix-daemon, in which case that would leave little time for the GuixSD
>> parts.  I think I would rather let you focus on the GuixSD stuff as you
>> wrote, but I’d like to hear what the Hurd folks think.
> 
> I consider isolation more important.

I understand that isolation is more important but I think first getting
GuixSD will be more beneficial in the long run.  If everything goes
well, even though we may not be able to fully implement isolation in the
duration of this gsoc, we can bootstrap the process so we continue after
the end of the summer.

WDYT?

Thank you :-)
Manolis