From mboxrd@z Thu Jan 1 00:00:00 1970 From: Eddy Pronk Subject: Re: guixsd in lxd container Date: Tue, 20 Jun 2017 22:35:19 +1000 Message-ID: References: <87h8zo3jzk.fsf@gnu.org> <8737b6qnhp.fsf@gnu.org> <874lvcyzl9.fsf@gnu.org> Mime-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:43524) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dNINS-0000Na-7A for guix-devel@gnu.org; Tue, 20 Jun 2017 08:35:28 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dNINN-0000Xn-KD for guix-devel@gnu.org; Tue, 20 Jun 2017 08:35:26 -0400 Received: from mail-oi0-x236.google.com ([2607:f8b0:4003:c06::236]:34086) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1dNINN-0000XW-BS for guix-devel@gnu.org; Tue, 20 Jun 2017 08:35:21 -0400 Received: by mail-oi0-x236.google.com with SMTP id b6so71317466oia.1 for ; Tue, 20 Jun 2017 05:35:20 -0700 (PDT) In-Reply-To: <874lvcyzl9.fsf@gnu.org> List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org Sender: "Guix-devel" To: =?UTF-8?Q?Ludovic_Court=C3=A8s?= Cc: guix-devel@gnu.org, David Thompson On Mon, Jun 19, 2017 at 9:41 PM, Ludovic Court=C3=A8s wrote: >> Shall I open a bug for this so it can be tracked? > > Before, I=E2=80=99d like to make sure we have a good understanding of wha= t we > need. > > My first question would be, do we really need to have a mechanism other > than =E2=80=98guix system container=E2=80=99? I guess that if the host s= ystem runs LXD, > it=E2=80=99s better to use it than to spawn the script that =E2=80=98guix= system > container=E2=80=99 generates. > > In that case, what about adding an LXD backend to =E2=80=98guix system > container=E2=80=99? AIUI LXD has a REST API=C2=B9; by doing a POST on /1= .0/images, > we should be able to register our container image, though it=E2=80=99s no= t clear > to me what image format is expected. If we can figure out these > details, it might not be that hard to implement. Lxd expects a root file system and a traditional 'init' process. For this experiment I'm using a bash script to play the role of /sbin/init. I'll need to spend some time to see what 'guix system container' generates. I'm very new to guix, so that will be my homework for my spare time this week. :-) This weekend I set up a VM on Google cloud with lxd. Below the log of all the steps I did for this experiment. (I can give anyone who wants to experiment in this environment access. Just send me you ssh public key.) My recipe so far: $ wget https://alpha.gnu.org/gnu/guix/guixsd-vm-image-0.13.0.x86_64-linux.x= z $ xz -d guixsd-vm-image-0.13.0.x86_64-linux.xz $ qemu-img convert guixsd-vm-image-0.13.0.x86_64-linux image.raw We need the sector size and start sector for the right offset: $ fdisk image.raw Sector size (logical/physical): 512 bytes / 512 bytes Device Boot Start End Sectors Size Id Type image.raw1 * 2048 4093952 4091905 2G 83 Linux image.raw2 4093953 4175873 81921 40M ef EFI (FAT-12/16/32) Create a loopback device with an offset poiting to the Linux partition: $ sudo losetup /dev/loop0 image.raw -o $((2048 * 512)) Mount it. Now we have to content of the vm image on /mnt. $ sudo mount /dev/loop0 /mnt I hope this preserves links, timestamps in the right way. $ sudo tar cpf ./rootfs.tar -C /mnt/ . tar: ./dev/log: socket ignored =3D=3D=3D metadata.yaml =3D=3D=3D architecture: "x86_64" creation_date: 1424284563 properties: description: "GuixSD Intel 64bit" os: "guixsd" release: "0.0" =3D=3D=3D lxc imports an image from 2 tarballs: $ tar cf metadata.tar metadata.yaml $ lxc image import metadata.tar rootfs.tar --alias guixsd-vm epronk@instance-1:~$ lxc image list +-----------+--------------+--------+--------------------+--------+--------= --+------------------------------+ | ALIAS | FINGERPRINT | PUBLIC | DESCRIPTION | ARCH | SIZE | UPLOAD DATE | +-----------+--------------+--------+--------------------+--------+--------= --+------------------------------+ | guixsd-vm | c9eeb3dfcee7 | no | GuixSD Intel 64bit | x86_64 | 883.92MB | Jun 17, 2017 at 5:43am (UTC) | +-----------+--------------+--------+--------------------+--------+--------= --+------------------------------+ Create container called guixsd from guixsd-vm image: $ lxc launch guixsd-vm guixsd barf... no /sbin/init yet. epronk@instance-1:~$ lxc list +--------+---------+------+------+------------+-----------+ | NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS | +--------+---------+------+------+------------+-----------+ | guixsd | STOPPED | | | PERSISTENT | 0 | +--------+---------+------+------+------------+-----------+ epronk@instance-1:~$ sudo ls /var/lib/lxd/containers/guixsd/rootfs bin boot dev etc gnu home lost+found mnt root run tmp v= ar epronk@instance-1:~$ sudo find /var/lib/lxd/containers/guixsd/rootfs -name = sleep /var/lib/lxd/containers/guixsd/rootfs/gnu/store/xniak294s1x03zssfj2xzvfkcny= 1gn0x-profile/bin/sleep (other entries omitted) I don't know how to see the output of /sbin/init. For now all /sbin/init does is sleep. We start shepherd manually in later steps. =3D=3D=3D /sbin/init =3D=3D=3D #!/gnu/store/xniak294s1x03zssfj2xzvfkcny1gn0x-profile/bin/bash /gnu/store/xniak294s1x03zssfj2xzvfkcny1gn0x-profile/bin/sleep 99999 =3D=3D=3D $ sudo mkdir /var/lib/lxd/containers/guixsd/rootfs/sbin $ sudo cp init /var/lib/lxd/containers/guixsd/rootfs/sbin/init $ lxc start guixsd $ lxc list +--------+---------+------+------+------------+-----------+ | NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS | +--------+---------+------+------+------------+-----------+ | guixsd | RUNNING | | | PERSISTENT | 0 | +--------+---------+------+------+------------+-----------+ Now that the container is in a running state I can attach bash as a process to it: epronk@instance-1:~$ lxc exec guixsd /gnu/store/xniak294s1x03zssfj2xzvfkcny1gn0x-profile/bin/bash bash-4.4# 'start' is blueprint for /sbin/init. =3D=3D=3D /sbin/start/ =3D=3D=3D export HOME=3D/ export TERM=3Dlinux export PATH=3D/gnu/store/crvb68g89b479n4h44r8l42hy39axhg2-shadow-4.4/sbin/ export GUIX_NEW_SYSTEM=3D/gnu/store/4pr317614r1ff1bi6vd1q0jjdca5h78s-system cd $HOME /gnu/store/zk41gmzbibvpx9dpsm5gs8p0liz8shy0-guile-2.0.14/bin/guile --no-auto-compile $GUIX_NEW_SYSTEM/boot =3D=3D=3D Run start script via bash, so we can see stderr and stdout from host OS. $ lxc exec guixsd -- /gnu/store/xniak294s1x03zssfj2xzvfkcny1gn0x-profile/bin/bash -c "/sbin/start 2>&1" Error #1: ?: 2 [primitive-load "/gnu/store/ysvjgjb9ph1vg0m4y67lfrj06wc5gdx4-activate-service"] In ice-9/boot-9.scm: 893: 1 [call-with-output-file "/sys/module/firmware_class/parameters/path" ...] In unknown file: ?: 0 [open-file "/sys/module/firmware_class/parameters/path" "w" #:encoding #f] $ sudo chmod +w /var/lib/lxd/containers/guixsd/rootfs/gnu/store/ysvjgjb9ph1vg0m4y67lfrj06wc= 5gdx4-activate-service Comment out with ;; $ sudo emacs /var/lib/lxd/containers/guixsd/rootfs/gnu/store/ysvjgjb9ph1vg0= m4y67lfrj06wc5gdx4-activate-service Error #2: ?: 3 [primitive-load "/gnu/store/nz2wixyg218l9j56vb21w0whnvdrnmh5-activate-service"] In ice-9/eval.scm: 432: 2 [eval # ()] In ice-9/boot-9.scm: 893: 1 [call-with-output-file "/proc/sys/kernel/modprobe" ...] In unknown file: ?: 0 [open-file "/proc/sys/kernel/modprobe" "w" #:encoding #f] Commented out expression in /gnu/store/nz2wixyg218l9j56vb21w0whnvdrnmh5-activate-service After fixing these #1 #2, shepherd starts, but reports services that didn't start. The first service that reports an issue is file-systems. Next we tried to start some services manually: epronk@instance-1:~$ lxc exec guixsd /gnu/store/xniak294s1x03zssfj2xzvfkcny1gn0x-profile/bin/bash bash-4.4# root@gnu ~# herd start guix-daemon herd start guix-daemon herd: exception caught while executing 'start' on service 'file-system-/gnu/store': ERROR: In procedure mount: mount "/gnu/store" on "///gnu/store": Permission denied This is how far I got. I hope this give some idea of what the image looks like and what I tried to start it. Cheers, Eddy