Build machine sysadmin support

Build machine sysadmin support

[Ludovic Courtès]

[-- Attachment #1: Type: text/plain, Size: 1829 bytes --]

Hello!

After my talk yesterday at the GHM, Nacho and I talked a bit and Nacho
kindly offered to help make chapters.gnu.org the first build machine
that would use GuixSD.

So I quickly hacked up together, in guix-maintenance.git, a couple of
modules that allow us to give the high-level view of a build machine,
such that the GuixSD configuration of one machine is just:

--8<---------------cut here---------------start------------->8---
(use-modules (sysadmin people)
             (sysadmin build-machines)
             (guix))

(define %sysadmins
  ;; The fine folks!
  (list (sysadmin (name "ludo")
                  (full-name "Ludovic Courtès")
                  (lsh-public-key (local-file "keys/lsh/ludo.pub")))
        (sysadmin (name "hydra")                  ;fake sysadmin
                  (full-name "Hydra User")
                  (restricted? #t)
                  (lsh-public-key
                   (local-file "keys/lsh/hydra.gnu.org.pub")))))

(define %authorized-guix-keys
  ;; List of authorized 'guix archive' keys.
  (list (local-file "keys/guix/hydra.gnu.org-export.pub")))

;; The actual machine.
(build-machine-os "chapters" %sysadmins
                  #:authorized-guix-keys %authorized-guix-keys)
--8<---------------cut here---------------end--------------->8---

… and we get a GuixSD config with the relevant accounts created, and
with the right lsh and Guix keys authorized.

I’ll see with Nacho if we can deploy it on chapters.gnu.org, and from
there we can incrementally migrate our build machines to GuixSD!  \o/

Also, all the sysadmin work will happen in a bunch of Scheme files,
which means that everyone can now contribute to sysadmin, without
necessarily having shell access to the machines.  No excuses!  :-)

Ludo’, live from the GHM.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]

Re: Build machine sysadmin support

[David Craven]

> Also, all the sysadmin work will happen in a bunch of Scheme files,
> which means that everyone can now contribute to sysadmin, without
> necessarily having shell access to the machines.  No excuses!  :-)

Is there a reason why this isn't in gnu/system/examples? I think that
this is a common use case and would be nice to have it there, and
enable it in gnu/system/install.scm as an out-of-the-box experience.

Re: Build machine sysadmin support

[David Craven]

guix install probably needs support for handling local-file's. This is
what I added as a template to gnu/system/examples. Apart from easily
deploying build-machines, I think this is also useful for getting guix
build --system=armhf-linux working.

;; GuixSD configuration file for the build machines.
;; Copyright © 2016 Ludovic Courtès <ludo@gnu.org>
;; Released under the GNU GPLv3 or any later version.

(use-modules (gnu system modules build-machines)
             (gnu system modules people)
             (guix))

(define %sysadmins
  (list (sysadmin (name "Alice")
                  (full-name "Bob's sister Alice")
                  (lsh-public-key (local-file
                                   (string-append (getenv "HOME")
                                                  "/.ssh/id_ed25519.pub"))))))

(define %authorized-guix-keys
  ;; List of authorized 'guix archive' keys.
  (list (local-file "../../../hydra.gnu.org.pub")
        (local-file "/etc/guix/signing-key.pub")))

;; The actual machine.
(build-machine-os "build-machine" "Europe/Paris" %sysadmins
                  #:authorized-guix-keys %authorized-guix-keys)

Re: Build machine sysadmin support

[David Craven]

So guix doesn't work inside a vm - because /gnu/store is mounted
readonly. To be able to offload arm builds once guix system vm
--system=armhf-linux works, /gnu/store needs to be writeable.

Can multiple daemons concurrently use the same store?

or alternatively, can we move the

(unless (zero? (system* unionfs "-o"

"cow,allow_other,use_ino,suid,dev,max_files=65536"
                                "/rw-root=RW:/real-root=RO"
                                "/root"))
          (error "unionfs failed"))

to after the 9p filesystems have been mounted, so that the unionfs
allows writing to /gnu/store too?

Re: Build machine sysadmin support

[David Craven]

On Sun, Aug 21, 2016 at 7:38 PM, David Craven <david@craven.ch> wrote:
> So guix doesn't work inside a vm - because /gnu/store is mounted
> readonly. To be able to offload arm builds once guix system vm
> --system=armhf-linux works, /gnu/store needs to be writeable.
>
> Can multiple daemons concurrently use the same store?
>
> or alternatively, can we move the
>
> (unless (zero? (system* unionfs "-o"
>
> "cow,allow_other,use_ino,suid,dev,max_files=65536"
>                                 "/rw-root=RW:/real-root=RO"
>                                 "/root"))
>           (error "unionfs failed"))
>
> to after the 9p filesystems have been mounted, so that the unionfs
> allows writing to /gnu/store too?

I'd appreciate some ideas on how to best tackle this problem
(offloading builds to a guix system vm instance) =)

‘guix system vm’ and shared store

[Ludovic Courtès]

Hi,

David Craven <david@craven.ch> skribis:

> So guix doesn't work inside a vm - because /gnu/store is mounted
> readonly.

Right, and also because ‘guix system vm’ creates a VM that shares its
store with the host system.

> To be able to offload arm builds once guix system vm
> --system=armhf-linux works, /gnu/store needs to be writeable.
>
> Can multiple daemons concurrently use the same store?

No.

> or alternatively, can we move the
>
> (unless (zero? (system* unionfs "-o"
>
> "cow,allow_other,use_ino,suid,dev,max_files=65536"
>                                 "/rw-root=RW:/real-root=RO"
>                                 "/root"))
>           (error "unionfs failed"))
>
> to after the 9p filesystems have been mounted, so that the unionfs
> allows writing to /gnu/store too?

Yes, something like that: we need to make /gnu/store a union of the
host’s /gnu/store and a writable tmpfs, and we need to make sure
/var/guix/db/db.sqlite is populated appropriately (images produced by
‘guix system vm-image’ do the latter.)

HTH!

Ludo’.

Offloading to a VM

[Ludovic Courtès]

Hello!

David Craven <david@craven.ch> skribis:

> On Sun, Aug 21, 2016 at 7:38 PM, David Craven <david@craven.ch> wrote:
>> So guix doesn't work inside a vm - because /gnu/store is mounted
>> readonly. To be able to offload arm builds once guix system vm
>> --system=armhf-linux works, /gnu/store needs to be writeable.
>>
>> Can multiple daemons concurrently use the same store?
>>
>> or alternatively, can we move the
>>
>> (unless (zero? (system* unionfs "-o"
>>
>> "cow,allow_other,use_ino,suid,dev,max_files=65536"
>>                                 "/rw-root=RW:/real-root=RO"
>>                                 "/root"))
>>           (error "unionfs failed"))
>>
>> to after the 9p filesystems have been mounted, so that the unionfs
>> allows writing to /gnu/store too?
>
> I'd appreciate some ideas on how to best tackle this problem
> (offloading builds to a guix system vm instance) =)

Ooh, since I’m reading messages in the wrong order, I missed the
original intent.

If you want to offload to VMs of a different system type, you could of
course have a full blown VM with an SSH daemon running, and have the
main machine offload to that one (the VM would need to have its own
store and its own disk, though.)

However, a more efficient way to do that would be to use the “build
hook” mechanism of the daemon (this is how ‘guix offload’ gets invoked
by guix-daemon.)  That way, a full VM would not be needed; in fact, we
could probably use QEMU’s ability to run ELF programs directly:

--8<---------------cut here---------------start------------->8---
$ qemu-arm $(guix build coreutils -s armhf-linux |grep -v debug)/bin/uname -a
Linux pluto 4.7.0-gnu #1 SMP 1 armv7l GNU/Linux
$ uname -a
Linux pluto 4.7.0-gnu #1 SMP 1 x86_64 GNU/Linux
--8<---------------cut here---------------end--------------->8---

The idea was briefly discussed at
<http://debbugs.gnu.org/cgi/bugreport.cgi?bug=20239>.

Re: Build machine sysadmin support

[Ludovic Courtès]

David Craven <david@craven.ch> skribis:

>> Also, all the sysadmin work will happen in a bunch of Scheme files,
>> which means that everyone can now contribute to sysadmin, without
>> necessarily having shell access to the machines.  No excuses!  :-)
>
> Is there a reason why this isn't in gnu/system/examples?

It’s mostly that it’s a quick hack that is specific to this build
machine use case.

But yeah, it may be that a generalized form of these modules could end
up in Guix proper, either as examples or as an API.  I think we need to
gain experience with it before we can decide what to do.

Ludo’.

Re: Offloading to a VM

[David Craven]

> --8<---------------cut here---------------start------------->8---
> $ qemu-arm $(guix build coreutils -s armhf-linux |grep -v debug)/bin/uname -a
> Linux pluto 4.7.0-gnu #1 SMP 1 armv7l GNU/Linux
> $ uname -a
> Linux pluto 4.7.0-gnu #1 SMP 1 x86_64 GNU/Linux
> --8<---------------cut here---------------end--------------->8---

If I understand correctly this only works when there is a substitute
already. If it's a new package for example that would fail with "not
an armhf system" right?

Re: Offloading to a VM

[Ludovic Courtès]

David Craven <david@craven.ch> skribis:

>> --8<---------------cut here---------------start------------->8---
>> $ qemu-arm $(guix build coreutils -s armhf-linux |grep -v debug)/bin/uname -a
>> Linux pluto 4.7.0-gnu #1 SMP 1 armv7l GNU/Linux
>> $ uname -a
>> Linux pluto 4.7.0-gnu #1 SMP 1 x86_64 GNU/Linux
>> --8<---------------cut here---------------end--------------->8---
>
> If I understand correctly this only works when there is a substitute
> already. If it's a new package for example that would fail with "not
> an armhf system" right?

The “build hook” is the mechanism by which the daemon asks an external
program whether it can build a derivation for some system.

Our (guix scripts offload) is such a program; it replies positively if
there’s a machine that matches the target system.

What I was suggesting is to have another build hook that would reply
positively as long as the target system can be emulated by QEMU.

Does that makes sense?

Thanks,
Ludo’.

Re: Offloading to a VM

[David Craven]

Yep, thanks! =)

Re: Offloading to a VM

[David Craven]

> The “build hook” is the mechanism by which the daemon asks an external
> program whether it can build a derivation for some system.

> Our (guix scripts offload) is such a program; it replies positively if
> there’s a machine that matches the target system.

> What I was suggesting is to have another build hook that would reply
> positively as long as the target system can be emulated by QEMU.

So I thought about this some more. guix offload uses guix build to
build a derivation which again uses a different nix-daemon to build
it. I'm not sure that guix offload can help us here. If it is possible
to build a derivation using only qemu-arm instead of qemu-system-arm -
which I'm still not convinced - I think it's the nix-daemon that would
have to support this. In build.cc it sets up a chroot and then
executes the builder. I think it's that builder that needs to be run
with qemu-arm. But I don't think it's going to be as easy as that =)

Does this make sense?

Re: Offloading to a VM

[David Craven]

The other problem I forgot to mention is that many architectures that
qemu supports don't support a "kernel emulation" but only a "system
emulation". In practice the system emulation isn't that much slower
than kernel emulation, because emulating instructions is a lot easier
than emulating syscalls... [needs quote]

Re: Offloading to a VM

[Ludovic Courtès]

David Craven <david@craven.ch> skribis:

>> The “build hook” is the mechanism by which the daemon asks an external
>> program whether it can build a derivation for some system.
>
>> Our (guix scripts offload) is such a program; it replies positively if
>> there’s a machine that matches the target system.
>
>> What I was suggesting is to have another build hook that would reply
>> positively as long as the target system can be emulated by QEMU.
>
> So I thought about this some more. guix offload uses guix build to
> build a derivation which again uses a different nix-daemon to build
> it. I'm not sure that guix offload can help us here.

Above, I was not suggesting to use ‘guix offload’.  Instead, I was
suggesting to write a new “build hook” that would run qemu-arm,
mentioning (guix scripts offload) as an example of a build hook.

> If it is possible to build a derivation using only qemu-arm instead of
> qemu-system-arm - which I'm still not convinced - I think it's the
> nix-daemon that would have to support this. In build.cc it sets up a
> chroot and then executes the builder. I think it's that builder that
> needs to be run with qemu-arm. But I don't think it's going to be as
> easy as that =)

guix-daemon can be extended via this “build hook” mechanism.  What I’m
proposing is to extend guix-daemon via an external build hook, which
means that guix-daemon itself can remain unchanged.

A derivation is essentially the file name of a “builder” along with its
argv and the needed dependencies.  For a given derivation, the hook
would roughly run:

   qemu-arm /gnu/store/…-builder arg1 arg2 …

Hope this clarifies things!

Ludo’.

Re: Offloading to a VM

[Ludovic Courtès]

David Craven <david@craven.ch> skribis:

> The other problem I forgot to mention is that many architectures that
> qemu supports don't support a "kernel emulation" but only a "system
> emulation".

I think we can leave this concern aside as a first approximation:

--8<---------------cut here---------------start------------->8---
$ (cd ~/.guix-profile/bin/ ; ls -1 qemu-* |grep -v system-)
qemu-aarch64
qemu-alpha
qemu-arm
qemu-armeb
qemu-cris
qemu-ga
qemu-i386
qemu-img
qemu-io
qemu-m68k
qemu-microblaze
qemu-microblazeel
qemu-mips
qemu-mips64
qemu-mips64el
qemu-mipsel
qemu-mipsn32
qemu-mipsn32el
qemu-nbd
qemu-or32
qemu-ppc
qemu-ppc64
qemu-ppc64abi32
qemu-ppc64le
qemu-s390x
qemu-sh4
qemu-sh4eb
qemu-sparc
qemu-sparc32plus
qemu-sparc64
qemu-tilegx
qemu-unicore32
qemu-x86_64
--8<---------------cut here---------------end--------------->8---

:-)

Ludo’.

Re: Offloading to a VM

[David Craven]

> I think we can leave this concern aside as a first approximation:

Ah interesting. The only architecture other than arm I'm interested in
is riscv. The riscv qemu port doesn't include a syscall emulation. But
I guess the riscv ecosystem still has a long way to go...

> A derivation is essentially the file name of a “builder” along with its
> argv and the needed dependencies.  For a given derivation, the hook
> would roughly run:

>   qemu-arm /gnu/store/…-builder arg1 arg2 …

> Hope this clarifies things!

Yes this does clarify things. Thanks!