[GSOC 2020] network-boot-service

[GSOC 2020] network-boot-service

[Brice Waegeneire]

Hello Danny, Gábor,

Sorry for the very late update on the status of this GSOC about network
booting. At the moment I have working network boot service which I'm 
using
to boot various baremetal machines, I'm currently working on adding NFS
support to the initrd.


To support the widest hardware and boot options possible I went with 
iPXE
as a chainloader. Meaning that any machine doing a PXE boot (or with
builtin iPXE with restricted feature set) will load the iPXE bootloader
first, which will then properly load the initrd.

For the DHCP/TFTP servers I choose dnsmasq instead of isc-dhcp-server 
with
a separate TFTP server mainly because it support ProxyDHCP mode which is
required for one of the most used case.

Those technical choice where instructed from the LTSP[1] project and 
some
Guixers advice, Vagrant and Giovani to name a few.


Speaking of uses cases for the network boot service, I see three of 
them.
Configuration wise, the most straight-forward is as an authoritative 
DHCP
server where all of the interfaces of the server provide the only DHCP
server. I'm guessing it won't be used a lot yet since it imply that the
machine running Guix is a router which quite rare ATM since our 
networking
configuration are limited ATM. Probably the most popular use case will 
be
as a ProxyDHCP, in a network where there is already a non-PXE 
authoritative
DHCP server, the authoritative server provides IP addresses where our
dnsmasq server only send PXE entries and provides images through TFTP. 
The
last one is as a interface specific DHCP server, where dnsmasq attach to
one interface to avoid messing around on an already configured network, 
a
NAT can be put in place to allow client to access Internet. That's the 
one
I'm currently using to develop and test the network boot features.

The new network-boot-service allows all of those use cases for clients
doing PXE boot or UEFI HTTP Boot, arbitrary images to boot from can be
specified or extended from an other service.


Currently I'm working on the initrd part to add NFS mounting capability 
to
it. At this point I'm blocked by building a lightweight staticly built
'nfs-utils' package to be included in the initrd. It's current total 
size
219.2 MiB, I manage to reduced it to 162.2 MiB which is still one order 
of
magnitude larger than my initrd at 19 MiB.

My issue building a static 'nfs-utils' is that it can't find
'getrpcbynumber{,_r}' “configure: error: Neither getrpcbynumber_r nor
getrpcbynumber are available”. This function should be provided by the 
libc
or by libtirpc if it's not that first one. The problem is that 
'libtirpc'
don't build it's own 'getrpcbynumber' because it find one in libc but
nfs-utils can't find it...

Some other distros are using the kernel parameter 'nfsroot'[2], but we
probably don't want to use it since it can't be used together with an
initrd and it also mean we need built-in modules for NFS and network 
card
driver in the kernel.

I tried to workaround NFS mounting by copying the image from HTTPS using
(web client) but it's not really elegant since the image has to be 
loaded
in RAM and even simple Guix images can be relatively large.


After that the intrd work is done I will need to add support for PXE to
qemu so that network boot functionality and the network-boot-service can 
be
tested in Guix; I'll try doing so with OVMF.

You can find the part of my work which is committed in the
'wip-network-boot' at https://git.sr.ht/~bricewge/guix.

[1]: https://ltsp.org/
[2]: 
https://www.kernel.org/doc/Documentation/filesystems/nfs/nfsroot.txt

Have a good day,
- Brice

Re: [GSOC 2020] network-boot-service

[Danny Milosavljevic]

[-- Attachment #1: Type: text/plain, Size: 925 bytes --]

Hi Brice,

On Thu, 02 Jul 2020 10:11:28 +0000
Brice Waegeneire <brice@waegenei.re> wrote:
> My issue building a static 'nfs-utils' is that it can't find
> 'getrpcbynumber{,_r}' “configure: error: Neither getrpcbynumber_r nor
> getrpcbynumber are available”. This function should be provided by the 
> libc
> or by libtirpc if it's not that first one. The problem is that 
> 'libtirpc'
> don't build it's own 'getrpcbynumber' because it find one in libc but
> nfs-utils can't find it...

Since glibc:static doesn't seem to contain it
(/gnu/store/rj4il3jjqg23cm3a0h17yzq3b5wa8llm-glibc-2.31-static/lib$
 objdump -t *.a |grep -i getrpcby returns nothing), we could make our
own variant of libtirpc that does enable their version.

> You can find the part of my work which is committed in the
> 'wip-network-boot' at https://git.sr.ht/~bricewge/guix.

Thanks!

I've checked it out, looks good so far!

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [GSOC 2020] network-boot-service

[Vagrant Cascadian]

[-- Attachment #1: Type: text/plain, Size: 2135 bytes --]

On 2020-07-02, Brice Waegeneire wrote:
> To support the widest hardware and boot options possible I went with 
> iPXE
> as a chainloader. Meaning that any machine doing a PXE boot (or with
> builtin iPXE with restricted feature set) will load the iPXE bootloader
> first, which will then properly load the initrd.

iPXE is really cool! The main downside is I don't think the support for
non-x86 architectures is there, but would be happy to find out
otherwise.


> Currently I'm working on the initrd part to add NFS mounting capability 
> to
> it. At this point I'm blocked by building a lightweight staticly built
> 'nfs-utils' package to be included in the initrd. It's current total 
> size
> 219.2 MiB, I manage to reduced it to 162.2 MiB which is still one order 
> of
> magnitude larger than my initrd at 19 MiB.
>
> My issue building a static 'nfs-utils' is that it can't find
> 'getrpcbynumber{,_r}' “configure: error: Neither getrpcbynumber_r nor
> getrpcbynumber are available”. This function should be provided by the 
> libc
> or by libtirpc if it's not that first one. The problem is that 
> 'libtirpc'
> don't build it's own 'getrpcbynumber' because it find one in libc but
> nfs-utils can't find it...

Have you looked into using klibc (or maybe busybox) instead? They might
be smaller and have enough capability to mount NFS.


> Some other distros are using the kernel parameter 'nfsroot'[2], but we
> probably don't want to use it since it can't be used together with an
> initrd and it also mean we need built-in modules for NFS and network 
> card
> driver in the kernel.

Debian at least implements support for the nfsroot kernel parameter by
emulating the behavior in the initrd, using kernel modules instead of
built-ins. Seems like Guix could do the same?

If you're trying to use user-space NFS, I suspect you'll run into a lot
of issues... Debian at least dropped support for parts of the user-space
NFS stack years ago as it had huge numbers of bugs and little activity
upstream.


Great to hear about progress on this work!


live well,
  vagrant

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

Re: [GSOC 2020] network-boot-service

[Brice Waegeneire]

Hello Vagrant,

On 2020-07-02 16:34, Vagrant Cascadian wrote:
> On 2020-07-02, Brice Waegeneire wrote:
>> To support the widest hardware and boot options possible I went with
>> iPXE
>> as a chainloader. Meaning that any machine doing a PXE boot (or with
>> builtin iPXE with restricted feature set) will load the iPXE 
>> bootloader
>> first, which will then properly load the initrd.
> 
> iPXE is really cool! The main downside is I don't think the support for
> non-x86 architectures is there, but would be happy to find out
> otherwise.
> 

It looks like it does support arm 32 and 64 bits EFI:
https://ipxe.org/appnote/buildtargets. Anyway dnsmasq can be configured 
to
provide PXE entries by on the client's architecture or feature it 
support.
For example using network-boot-service, a iPXE client without http 
support
will chainload Guix's iPXE which has http support.

>> Currently I'm working on the initrd part to add NFS mounting 
>> capability
>> to
>> it. At this point I'm blocked by building a lightweight staticly built
>> 'nfs-utils' package to be included in the initrd. It's current total
>> size
>> 219.2 MiB, I manage to reduced it to 162.2 MiB which is still one 
>> order
>> of
>> magnitude larger than my initrd at 19 MiB.
>> 
>> My issue building a static 'nfs-utils' is that it can't find
>> 'getrpcbynumber{,_r}' “configure: error: Neither getrpcbynumber_r nor
>> getrpcbynumber are available”. This function should be provided by the
>> libc
>> or by libtirpc if it's not that first one. The problem is that
>> 'libtirpc'
>> don't build it's own 'getrpcbynumber' because it find one in libc but
>> nfs-utils can't find it...
> 
> Have you looked into using klibc (or maybe busybox) instead? They might
> be smaller and have enough capability to mount NFS.

Non I did not, thanks for the tip. Busybox is smaller that was I
expecting, 1.0 MiB by itself, but it doesn't seems it would follow the
Guix way of doing most of the work in Guile (and using boot-to-Guile™)
and it's unique binary forbid us to extract only the NFS part. As for
klibc, it seems to be what NixOS[1] is using and TIL that it also the
the original source of Arch's mkinitcpio-nfs-utils which is 
unmaintained.
So klibc may work here, I'll try packaging it.

>> Some other distros are using the kernel parameter 'nfsroot'[2], but we
>> probably don't want to use it since it can't be used together with an
>> initrd and it also mean we need built-in modules for NFS and network
>> card
>> driver in the kernel.
> 
> Debian at least implements support for the nfsroot kernel parameter by
> emulating the behavior in the initrd, using kernel modules instead of
> built-ins. Seems like Guix could do the same?

Do you happen to know which package implement it in Debian? Yes, there
will be a need to pass parameters to the initrd so we better use the
common syntax.

> If you're trying to use user-space NFS, I suspect you'll run into a lot
> of issues... Debian at least dropped support for parts of the 
> user-space
> NFS stack years ago as it had huge numbers of bugs and little activity
> upstream.

What do you mean by “user-space NFS”, be it nfs-utils, busybox or klibc,
they all are user-space tools to used to mount NFS shares?
Do you have some reference about this, I would like to learn more about
it. It's true that I did not find anyone trying to build a static
nfs-utils...

> Great to hear about progress on this work!
> 
> 
> live well,
>   vagrant

[1]: 
https://github.com/NixOS/nixpkgs/commit/d25c1a8fdc383b8997f6e7b4e1479875df1f06b2
[2]: 
https://github.com/NixOS/nixpkgs/blob/84cf00f98031e93f389f1eb93c4a7374a33cc0a9/pkgs/os-specific/linux/mkinitcpio-nfs-utils/default.nix

- Brice