From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp11.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms9.migadu.com with LMTPS id gATqJ4k5HGTZlwAASxT56A (envelope-from ) for ; Thu, 23 Mar 2023 12:35:37 +0100 Received: from aspmx1.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp11.migadu.com with LMTPS id 4CrhJ4k5HGS8FAEA9RJhRA (envelope-from ) for ; Thu, 23 Mar 2023 12:35:37 +0100 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 4F10B377A0 for ; Thu, 23 Mar 2023 12:35:37 +0100 (CET) Authentication-Results: aspmx1.migadu.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=f7IkfQa+; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org"; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1679571337; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version: content-type:content-type:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post:dkim-signature; bh=WiVqfdSjDtq4MkDmLJKjyIcFDTWzwqqJatjEwhWi1iY=; b=BNz8Akj5f8aBNwothkZgJ+oXjBYY8kpjw1l3bqOnGOYiaNeNzd2MNcV2DMTd3tgMKdJiVb qAfZclxC5sqOauQBziScbQgU5vtiMROr29wdIr0zybe7cnDJ2newBUYNVNGqNvIsYzLJIq mtautwyHOQ5IbHqF+2nNqVNr/7JhppekPjlOS/a/pP3IKWPOo7DnZnOADmNny+0Ed2dTfi 12xgVVleWkAJDr6dULaCR76FOJ592FeEzr+PsYymbdejhAn7P2Zy9nwygs/phNDz/hks98 7smIzMqQOtHeWb9PtjDwHJvQP4gFCDwHEl+jSER9F38t27sMXMJPEZd8bUQ8Fw== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1679571337; a=rsa-sha256; cv=none; b=aJJ5Zw4snecNOULjhkwJ/u+DT5YD+Am/J0BH0Vzitol8fZc+CHNafTbkNs6n+VVp6lZva+ mt4DST5dQXEhKEbR/LG9Kpelq2LKVLZjtX220e56GMP9cUJSAlQ1hfI8ojtshj6W5aruVG HD6b3cKk1TH4UhGsosfoxGBxQ9L95/n0Roxa1gR1B6LfBB+yNFAV9iJRvl3Eb7mnugjSKY 1WdbjHltrNYmfs+lguhktdWeOe58O9jOIQT9obfzBNVkfP00TO6bXPebcM/GdjyOs8fLag jON1seWh/6xeWp8rfY+fjwqB6b1snzRECnglsz6xrnknA1xsFnXBNTQ/wlfYHg== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=f7IkfQa+; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org"; dmarc=pass (policy=none) header.from=gmail.com Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pfJD3-0006Lw-1G; Thu, 23 Mar 2023 07:34:21 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pexMF-0006Xe-MM for guix-devel@gnu.org; Wed, 22 Mar 2023 08:14:23 -0400 Received: from mail-ed1-x52a.google.com ([2a00:1450:4864:20::52a]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1pexMC-0001Cc-6L for guix-devel@gnu.org; Wed, 22 Mar 2023 08:14:23 -0400 Received: by mail-ed1-x52a.google.com with SMTP id r11so72009087edd.5 for ; Wed, 22 Mar 2023 05:14:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; t=1679487258; h=to:subject:message-id:date:from:in-reply-to:references:mime-version :from:to:cc:subject:date:message-id:reply-to; bh=WiVqfdSjDtq4MkDmLJKjyIcFDTWzwqqJatjEwhWi1iY=; b=f7IkfQa+V+/IbgTjPYh5xkDM5Zr7Lg+7KY5XSoDHG11lh6jwaJT5EqvWSC2uX4Q0db 6GW7qGWsPKiKvSrqhaLgwMlZaV6oaQW/5HZO2UitMJnjyQ2bKg+fTmNSA778h1XIHD25 RwZEQzxoHQFnveYG4yVHkFEqXYZ+avCPDJIT4MTqMDT5F6AZXVJJ4VIb2yDKpgiO3VVx SILRIb9D8XSY7rRBB/5r/y4TnJEJuFTNeOUNrEnbJMIKyMXIp6bDrvTNguCFwfFMhDts hA2lyu3MM6g26ivHvYvJbeNSulIVIf7BCtzkqHuf6j8CgHxpR26fHa27THOtl17HT4sW 3Gow== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1679487258; h=to:subject:message-id:date:from:in-reply-to:references:mime-version :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=WiVqfdSjDtq4MkDmLJKjyIcFDTWzwqqJatjEwhWi1iY=; b=Bq+X9mlkNiXjOvbDvAf4FfjvbBXg/MNBvQSIyVaKo4tsASEmk3/47M79TdE/Clobqr aClYeLYkFD4NXHKqpss37ys9GBJG4Hdv4F/IGN1i/q76z0S1g+hUC7IRUZEJVyPpvOIe 3vtN1va0saS1U1SgmSbJJdtbPfr9UfbCy8YF3n2GyJumuky6Jy8l3VDuiAtg5ZmKQ/eB Yj3Uk55bL4hNE7Ix9BmOrQ9485aG/Xn5xS27il7MK1WesZPZlUYfjKvD4XL9dc2qNtke o6QSGKoLvKNgSwUuZFqqsSGPRT82mSRKEztPU9fum+2sjDCQeI7AcgFS32dLsEQxsGYk fgKg== X-Gm-Message-State: AO0yUKW5gTeDft7LMoEXO34AjpoqsiyrfW5+7LJVZLCdPAHFWH91g759 kHbfVcGT4wcviNrx9Hg+7ADvm9J5gGw85gwAh71RvwERl7A= X-Google-Smtp-Source: AK7set+dJdSo6Q4ra2U+uo8ROJG/BF/Z7K/KzU1S7qO+EpdgKQCfVuGJ4g+iJqQkmbaeC960pEog4l/tCRtDMgmyrQg= X-Received: by 2002:a17:906:c793:b0:92f:97a3:9f21 with SMTP id cw19-20020a170906c79300b0092f97a39f21mr2984052ejb.13.1679487258501; Wed, 22 Mar 2023 05:14:18 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Vladilen Kozin Date: Wed, 22 Mar 2023 12:14:07 +0000 Message-ID: Subject: Re: shepherd service works on host but fails inside system container To: guix-devel@gnu.org Content-Type: multipart/alternative; boundary="00000000000072a6f505f77c199c" Received-SPF: pass client-ip=2a00:1450:4864:20::52a; envelope-from=vladilen.kozin@gmail.com; helo=mail-ed1-x52a.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-Mailman-Approved-At: Thu, 23 Mar 2023 07:34:17 -0400 X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: X-Migadu-Queue-Id: 4F10B377A0 X-Spam-Score: -6.48 X-Migadu-Spam-Score: -6.48 X-Migadu-Scanner: scn0.migadu.com List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: guix-devel-bounces+larch=yhetil.org@gnu.org X-Migadu-Country: US X-Migadu-Flow: FLOW_IN X-TUID: 320HBf3/EJs+ --00000000000072a6f505f77c199c Content-Type: text/plain; charset="UTF-8" I now have a hypothesis as to what's happening. Could someone confirm or disprove and maybe suggest a solution or point at existing workarounds. Host and container will share the exact same kernel that's unsurprisingly already running, so the above has nothing to do with kernel modules or settings. Fwiw I figured a way to find where kernel modules reside by doing: $ sudo dmesg | grep -i "kernel command line" which shows where current system is inside the store and relevant /lib/modules will be under it. We could then --expose=/gnu/store/hash-system/lib/modules=/lib/modules if we wanted to. Real problem, IIUC, is with capabilities. Notion of "container" can be misleading and evokes thoughts of "vm" when in practice its just a process with some isolation applied to it. So, presently I'm guessing container Shepherd maybe PID 1 inside its isolated environment, but from the host pow it is just a process and one that unlike our host's shepherd may lack certain capabilities and privileges to e.g. create new devices or load kernel modules on request, etc. In the sense of https://man7.org/linux/man-pages/man7/capabilities.7.html maybe? Am I on the right track? But then, how does one test services like that that may require ability to modify devices etc? Have we "outgrown" container and ought to `guix system vm` for such services? Or is there a way to bless container shepherd with necessary capabilities? If not from `guix system container` command line, then perhaps dropping down to the underlying programmatic interface i.e. whatever `guix system container` ends up calling to containerize a system? Thanks On Wed, 22 Mar 2023 at 10:20, Vladilen Kozin wrote: > Hello guix. > > I put together a tailscale system service that's meant to start a > tailscale daemon managed by the system shepherd, that is to say that my > `tailscaled-service-type` specifies `(service-extension > shepherd-root-service-type tailscaled-shepherd-service)`, where > `tailscaled-shepherd-service` creates a `shepherd-service` with (provision > '(tailscaled)) and (requirement '(networking)). > > I tested it by lowering to store via `shepherd-service-file` and then > loading the generated script via `sudo herd load root ...`. This works fine > and the daemon starts without a problem. > > Next, I try to spawn tailscaled as part of my OS definition: > (services (cons* (service tailscaled-service-type > (tailscaled-configuration)) %base-services)) > ;; tried %desktop-services too > > To test, we create a container: > sudo guix system -K -L /home/vlad/Code/fullmeta-guix/channel container > os.scm --network --expose=/dev/net=/dev > > Earlier runs had it complaining that /dev/net/tun was missing, so I > exposed that. Dunno if that's how I'm supposed to handle this. Now, > /var/log/messages show: > > Mar 22 09:38:48 twgter shepherd[1]: [tailscaled] 2023/03/22 09:38:48 Linux > kernel version: 5.18.10 > Mar 22 09:38:48 twgter shepherd[1]: [tailscaled] 2023/03/22 09:38:48 is > CONFIG_TUN enabled in your kernel? `modprobe tun` failed with: > Mar 22 09:38:48 twgter shepherd[1]: [tailscaled] 2023/03/22 09:38:48 > wgengine.NewUserspaceEngine(tun "tailscale0") error: > tstun.New("tailscale0"): operation not permitted > > I feel like maybe I'm missing some kernel modules, but I would've expected > host and container to share the kernel, so I dunno. In fact, when I > randomly attempted adding (kernel-arguments (cons* "CONFIG_TUN=m" > %default-kernel-arguments)) to my os definition, resulting script hash came > out the same, which tells me, containers don't even look at these kernel > params when generating a script. > > Any guesses as to why this works under host but not inside container? > > Relatedly, does anyone have a nicer workflow they use to define and test > shepherd services? Such containerization was the next step in testing the > service and would've been ok were it not for the above failure, but the > initial indirection with lowering to store, then `sudo herd load root ...` > is a bit too involved and "indirect" for my liking as well - anyone has an > improved way of developing shepherd services? > > Thanks! > -- > Best regards > Vlad Kozin > -- Best regards Vlad Kozin --00000000000072a6f505f77c199c Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
I now have a hypothesis as to what's happening. Could someone= confirm or disprove and maybe suggest a solution or point at existing work= arounds.

Host and container will share the exact same kernel that's un= surprisingly already running, so the above has nothing to do with kernel mo= dules or settings. Fwiw I figured a way to find where kernel modules reside= by doing:

$ sudo dmesg | grep -i "kernel command line"
which show= s where current system is inside the store and relevant /lib/modules will b= e under it. We could then --expose=3D/gnu/store/hash-system/lib/modules=3D/= lib/modules if we wanted to.

Real problem, IIUC, is with capabilities. Not= ion of "container" can be misleading and evokes thoughts of "= ;vm" when in practice its just a process with some isolation applied t= o it. So, presently I'm guessing container Shepherd maybe PID 1 inside = its isolated environment, but from the host pow it is just a process and on= e that unlike our host's shepherd may lack certain capabilities and pri= vileges to e.g. create new devices or load kernel modules on request, etc. = In the sense of=C2=A0https://man7.org/linux/man-pages/man7/capabilities.7.html maybe?

Am I on the right track? But then, how does one test services li= ke that that may require ability to modify devices etc? Have we "outgr= own" container and ought to `guix system vm` for such services? Or is = there a way to bless container shepherd with necessary capabilities? If not= from `guix system container` command line, then perhaps dropping down to t= he underlying programmatic interface i.e. whatever `guix system container` = ends up calling to containerize a system?=C2=A0

Thanks


<= div class=3D"gmail_quote">
On Wed, 22 = Mar 2023 at 10:20, Vladilen Kozin <vladilen.kozin@gmail.com> wrote:
Hello guix.

I put together a = tailscale system service that's meant to start a tailscale daemon manag= ed by the system shepherd, that is to say that my `tailscaled-service-type`= specifies `(service-extension shepherd-root-service-type tailscaled-shephe= rd-service)`, where `tailscaled-shepherd-service` creates a `shepherd-servi= ce` with=C2=A0(provision '(tailscaled)) and (requirement '(networki= ng)).

I tested it by lowering to store via `shepherd-service-file` and the= n loading the generated script via `sudo herd load root ...`. This works fi= ne and the daemon starts without a problem.

Next, I try to spawn tailscale= d as part of my OS definition:
(services (cons*=C2=A0(service tailscaled-serv= ice-type (tailscaled-configuration)) %base-services))
;; tried %desktop-servi= ces too

To test, we create a container:
sudo guix system -K -L /home/vlad/Co= de/fullmeta-guix/channel container os.scm --network --expose=3D/dev/net=3D/= dev

Earlier runs had it complaining that /dev/net/tun was missing, so = I exposed that. Dunno if that's how I'm supposed to handle this. No= w, /var/log/messages show:

Mar 22 09:38:48 twgter shepherd[1]: [tailscaled= ] 2023/03/22 09:38:48 Linux kernel version: 5.18.10
Mar 22 09:38:48 twgt= er shepherd[1]: [tailscaled] 2023/03/22 09:38:48 is CONFIG_TUN enabled in y= our kernel? `modprobe tun` failed with:
Mar 22 09:38:48 twgter shepherd[= 1]: [tailscaled] 2023/03/22 09:38:48 wgengine.NewUserspaceEngine(tun "= tailscale0") error: tstun.New("tailscale0"): operation not p= ermitted

I feel like maybe I'm missing some kerne= l modules, but I would've expected host and container to share the kern= el, so I dunno. In fact, when I randomly attempted adding=C2=A0(kernel-argu= ments (cons* "CONFIG_TUN=3Dm" %default-kernel-arguments)) to my o= s definition, resulting script hash came out the same, which tells me, cont= ainers don't even look at these kernel params when generating a script.=
<= br>
Any guesses as to why this works under host but not inside container?

Relatedly, does anyone have a nicer workflow they use to define and tes= t shepherd services? Such containerization was the next step in testing the= service and would've been ok were it not for the above failure, but th= e initial indirection with lowering to store, then `sudo herd load root ...= ` is a bit too involved and "indirect" for my liking as well - an= yone has an improved way of developing shepherd services?

Thanks!
<= span>--
Best regards
Vlad Kozin


--
Be= st regards
Vlad Kozin
--00000000000072a6f505f77c199c--