Re: guixsd in lxd container

[Eddy Pronk <epronk@muftor.com>]

On Mon, Jun 19, 2017 at 9:41 PM, Ludovic Courtès <ludo@gnu.org> wrote:
>> Shall I open a bug for this so it can be tracked?
>
> Before, I’d like to make sure we have a good understanding of what we
> need.
>
> My first question would be, do we really need to have a mechanism other
> than ‘guix system container’?  I guess that if the host system runs LXD,
> it’s better to use it than to spawn the script that ‘guix system
> container’ generates.
>
> In that case, what about adding an LXD backend to ‘guix system
> container’?  AIUI LXD has a REST API¹; by doing a POST on /1.0/images,
> we should be able to register our container image, though it’s not clear
> to me what image format is expected.  If we can figure out these
> details, it might not be that hard to implement.

Lxd expects a root file system and a traditional 'init' process.
For this experiment I'm using a bash script to play the role of /sbin/init.

I'll need to spend some time to see what 'guix system container' generates.
I'm very new to guix, so that will be my homework for my spare time
this week. :-)

This weekend I set up a VM on Google cloud with lxd.

Below the log of all the steps I did for this experiment.

(I can give anyone who wants to experiment in this environment access.
Just send me you ssh public key.)


My recipe so far:

$ wget https://alpha.gnu.org/gnu/guix/guixsd-vm-image-0.13.0.x86_64-linux.xz

$ xz -d guixsd-vm-image-0.13.0.x86_64-linux.xz

$ qemu-img convert guixsd-vm-image-0.13.0.x86_64-linux image.raw

We need the sector size and start sector for the right offset:
$ fdisk image.raw

Sector size (logical/physical): 512 bytes / 512 bytes

Device     Boot   Start     End Sectors Size Id Type
image.raw1 *       2048 4093952 4091905   2G 83 Linux
image.raw2      4093953 4175873   81921  40M ef EFI (FAT-12/16/32)

Create a loopback device with an offset poiting to the Linux partition:

$ sudo losetup /dev/loop0 image.raw -o $((2048 * 512))

Mount it. Now we have to content of the vm image on /mnt.
$ sudo mount /dev/loop0 /mnt

I hope this preserves links, timestamps in the right way.
$ sudo tar cpf ./rootfs.tar -C /mnt/ .
tar: ./dev/log: socket ignored

=== metadata.yaml ===
architecture: "x86_64"
creation_date: 1424284563
properties:
  description: "GuixSD Intel 64bit"
  os: "guixsd"
  release: "0.0"
===

lxc imports an image from 2 tarballs:
$ tar cf metadata.tar metadata.yaml
$ lxc image import metadata.tar rootfs.tar --alias guixsd-vm

epronk@instance-1:~$ lxc image list
+-----------+--------------+--------+--------------------+--------+----------+------------------------------+
|   ALIAS   | FINGERPRINT  | PUBLIC |    DESCRIPTION     |  ARCH  |
SIZE   |         UPLOAD DATE          |
+-----------+--------------+--------+--------------------+--------+----------+------------------------------+
| guixsd-vm | c9eeb3dfcee7 | no     | GuixSD Intel 64bit | x86_64 |
883.92MB | Jun 17, 2017 at 5:43am (UTC) |
+-----------+--------------+--------+--------------------+--------+----------+------------------------------+

Create container called guixsd from guixsd-vm image:

$ lxc launch guixsd-vm guixsd

barf... no /sbin/init yet.

epronk@instance-1:~$ lxc list
+--------+---------+------+------+------------+-----------+
|  NAME  |  STATE  | IPV4 | IPV6 |    TYPE    | SNAPSHOTS |
+--------+---------+------+------+------------+-----------+
| guixsd | STOPPED |      |      | PERSISTENT | 0         |
+--------+---------+------+------+------------+-----------+

epronk@instance-1:~$ sudo ls /var/lib/lxd/containers/guixsd/rootfs
bin  boot  dev         etc  gnu  home    lost+found  mnt  root  run  tmp  var

epronk@instance-1:~$ sudo find /var/lib/lxd/containers/guixsd/rootfs -name sleep
/var/lib/lxd/containers/guixsd/rootfs/gnu/store/xniak294s1x03zssfj2xzvfkcny1gn0x-profile/bin/sleep
(other entries omitted)

I don't know how to see the output of /sbin/init. For now all
/sbin/init does is sleep.
We start shepherd manually in later steps.

=== /sbin/init ===
#!/gnu/store/xniak294s1x03zssfj2xzvfkcny1gn0x-profile/bin/bash
/gnu/store/xniak294s1x03zssfj2xzvfkcny1gn0x-profile/bin/sleep 99999
===

$ sudo mkdir /var/lib/lxd/containers/guixsd/rootfs/sbin
$ sudo cp init /var/lib/lxd/containers/guixsd/rootfs/sbin/init

$ lxc start guixsd
$ lxc list
+--------+---------+------+------+------------+-----------+
|  NAME  |  STATE  | IPV4 | IPV6 |    TYPE    | SNAPSHOTS |
+--------+---------+------+------+------------+-----------+
| guixsd | RUNNING |      |      | PERSISTENT | 0         |
+--------+---------+------+------+------------+-----------+

Now that the container is in a running state I can attach bash as a
process to it:
epronk@instance-1:~$ lxc exec guixsd
/gnu/store/xniak294s1x03zssfj2xzvfkcny1gn0x-profile/bin/bash
bash-4.4#

'start' is blueprint for /sbin/init.

=== /sbin/start/ ===
export HOME=/
export TERM=linux

export PATH=/gnu/store/crvb68g89b479n4h44r8l42hy39axhg2-shadow-4.4/sbin/
export GUIX_NEW_SYSTEM=/gnu/store/4pr317614r1ff1bi6vd1q0jjdca5h78s-system
cd $HOME
/gnu/store/zk41gmzbibvpx9dpsm5gs8p0liz8shy0-guile-2.0.14/bin/guile
--no-auto-compile $GUIX_NEW_SYSTEM/boot
===

Run start script via bash, so we can see stderr and stdout from host OS.

$ lxc exec guixsd --
/gnu/store/xniak294s1x03zssfj2xzvfkcny1gn0x-profile/bin/bash -c
"/sbin/start 2>&1"

Error #1:

   ?: 2 [primitive-load
"/gnu/store/ysvjgjb9ph1vg0m4y67lfrj06wc5gdx4-activate-service"]
   In ice-9/boot-9.scm:
    893: 1 [call-with-output-file
"/sys/module/firmware_class/parameters/path" ...]
    In unknown file:
       ?: 0 [open-file "/sys/module/firmware_class/parameters/path"
"w" #:encoding #f]

$ sudo chmod +w
/var/lib/lxd/containers/guixsd/rootfs/gnu/store/ysvjgjb9ph1vg0m4y67lfrj06wc5gdx4-activate-service

Comment out with ;;
$ sudo emacs /var/lib/lxd/containers/guixsd/rootfs/gnu/store/ysvjgjb9ph1vg0m4y67lfrj06wc5gdx4-activate-service

Error #2:

?: 3 [primitive-load
"/gnu/store/nz2wixyg218l9j56vb21w0whnvdrnmh5-activate-service"]
In ice-9/eval.scm:
 432: 2 [eval # ()]
 In ice-9/boot-9.scm:
  893: 1 [call-with-output-file "/proc/sys/kernel/modprobe" ...]
  In unknown file:
     ?: 0 [open-file "/proc/sys/kernel/modprobe" "w" #:encoding #f]


Commented out expression in
/gnu/store/nz2wixyg218l9j56vb21w0whnvdrnmh5-activate-service

After fixing these #1 #2, shepherd starts, but reports services that
didn't start.

The first service that reports an issue is file-systems.

Next we tried to start some services manually:

epronk@instance-1:~$ lxc exec guixsd
/gnu/store/xniak294s1x03zssfj2xzvfkcny1gn0x-profile/bin/bash
bash-4.4#

root@gnu ~# herd start guix-daemon
herd start guix-daemon herd: exception caught while executing 'start'
on service 'file-system-/gnu/store': ERROR:
In procedure mount: mount "/gnu/store" on "///gnu/store": Permission denied

This is how far I got.

I hope this give some idea of what the image looks like and what I
tried to start it.


Cheers,
Eddy