Re: shepherd service works on host but fails inside system container

[Vladilen Kozin <vladilen.kozin@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 4236 bytes --]

I now have a hypothesis as to what's happening. Could someone confirm or
disprove and maybe suggest a solution or point at existing workarounds.

Host and container will share the exact same kernel that's unsurprisingly
already running, so the above has nothing to do with kernel modules or
settings. Fwiw I figured a way to find where kernel modules reside by doing:

$ sudo dmesg | grep -i "kernel command line"
which shows where current system is inside the store and relevant
/lib/modules will be under it. We could then
--expose=/gnu/store/hash-system/lib/modules=/lib/modules if we wanted to.

Real problem, IIUC, is with capabilities. Notion of "container" can be
misleading and evokes thoughts of "vm" when in practice its just a process
with some isolation applied to it. So, presently I'm guessing container
Shepherd maybe PID 1 inside its isolated environment, but from the host pow
it is just a process and one that unlike our host's shepherd may lack
certain capabilities and privileges to e.g. create new devices or load
kernel modules on request, etc. In the sense of
https://man7.org/linux/man-pages/man7/capabilities.7.html maybe?

Am I on the right track? But then, how does one test services like that
that may require ability to modify devices etc? Have we "outgrown"
container and ought to `guix system vm` for such services? Or is there a
way to bless container shepherd with necessary capabilities? If not from
`guix system container` command line, then perhaps dropping down to the
underlying programmatic interface i.e. whatever `guix system container`
ends up calling to containerize a system?

Thanks


On Wed, 22 Mar 2023 at 10:20, Vladilen Kozin <vladilen.kozin@gmail.com>
wrote:

> Hello guix.
>
> I put together a tailscale system service that's meant to start a
> tailscale daemon managed by the system shepherd, that is to say that my
> `tailscaled-service-type` specifies `(service-extension
> shepherd-root-service-type tailscaled-shepherd-service)`, where
> `tailscaled-shepherd-service` creates a `shepherd-service` with (provision
> '(tailscaled)) and (requirement '(networking)).
>
> I tested it by lowering to store via `shepherd-service-file` and then
> loading the generated script via `sudo herd load root ...`. This works fine
> and the daemon starts without a problem.
>
> Next, I try to spawn tailscaled as part of my OS definition:
> (services (cons* (service tailscaled-service-type
> (tailscaled-configuration)) %base-services))
> ;; tried %desktop-services too
>
> To test, we create a container:
> sudo guix system -K -L /home/vlad/Code/fullmeta-guix/channel container
> os.scm --network --expose=/dev/net=/dev
>
> Earlier runs had it complaining that /dev/net/tun was missing, so I
> exposed that. Dunno if that's how I'm supposed to handle this. Now,
> /var/log/messages show:
>
> Mar 22 09:38:48 twgter shepherd[1]: [tailscaled] 2023/03/22 09:38:48 Linux
> kernel version: 5.18.10
> Mar 22 09:38:48 twgter shepherd[1]: [tailscaled] 2023/03/22 09:38:48 is
> CONFIG_TUN enabled in your kernel? `modprobe tun` failed with:
> Mar 22 09:38:48 twgter shepherd[1]: [tailscaled] 2023/03/22 09:38:48
> wgengine.NewUserspaceEngine(tun "tailscale0") error:
> tstun.New("tailscale0"): operation not permitted
>
> I feel like maybe I'm missing some kernel modules, but I would've expected
> host and container to share the kernel, so I dunno. In fact, when I
> randomly attempted adding (kernel-arguments (cons* "CONFIG_TUN=m"
> %default-kernel-arguments)) to my os definition, resulting script hash came
> out the same, which tells me, containers don't even look at these kernel
> params when generating a script.
>
> Any guesses as to why this works under host but not inside container?
>
> Relatedly, does anyone have a nicer workflow they use to define and test
> shepherd services? Such containerization was the next step in testing the
> service and would've been ok were it not for the above failure, but the
> initial indirection with lowering to store, then `sudo herd load root ...`
> is a bit too involved and "indirect" for my liking as well - anyone has an
> improved way of developing shepherd services?
>
> Thanks!
> --
> Best regards
> Vlad Kozin
>


-- 
Best regards
Vlad Kozin

[-- Attachment #2: Type: text/html, Size: 7616 bytes --]