From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp12.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms9.migadu.com with LMTPS id a9XzMYg5HGTZlwAASxT56A (envelope-from ) for ; Thu, 23 Mar 2023 12:35:36 +0100 Received: from aspmx1.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp12.migadu.com with LMTPS id YJWGLog5HGQiKwAAauVa8A (envelope-from ) for ; Thu, 23 Mar 2023 12:35:36 +0100 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 5CE93378A0 for ; Thu, 23 Mar 2023 12:35:36 +0100 (CET) Authentication-Results: aspmx1.migadu.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=k5sdkreC; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org"; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1679571336; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version: content-type:content-type:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post:dkim-signature; bh=2Um9sEmHwv7d+YG6U6cEHEGvyW1xSfVMIIMQeqgIUms=; b=km2SmI+zpiWPHREAB71kV3rC9A1CcKdZgGAx3I93rDrkb0DDyCgRpbryErhHbgxAjze5z4 ojDVoUMzif+rfRozFDWox3DzYYx7IjnXIqCAxHBdiiyBuC9/7W2XWPJ/xEf/DcTpUpfSFr M8I+qhZ1FdzF6Nu9Ob43J+gwIbRFIZyRXcWAgGlhJWbbD+5WIFoO2ma4to4g8H2OV5jOfC wL6qYeNLQxVzqSVNrM5mn522n3pYWtiozwf2UBaC9aju4QyLZGIUwKSCO22F6z8v28+VDE PNmc7BrdoNPf41j5k2mp7ImPFCiaYzf5L1I70QULQQ/FHAXoVX0cIxpOAD9e0w== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1679571336; a=rsa-sha256; cv=none; b=hlF/AIOROlUEdr6oFdn4l0vwlcv+W17MhFk3XGwuqa4pSO3BT2L296FWqSV/wheVgATrgH C4xjN7qiAhXNVW1179yTB4VyCzp16S6peBe42bSkaUuoOoDkOiUwphu5ItO0nG5FTh8roP tm4ybWUoe7UisBIq0a8lbqoWZWb4KZrTXPAyE5Pd2g1eH36rZNTCD5cWyjvwLRhbR3e9cc 7CwdxNC6PMgweLjq3NtFSsHZ6n5Gms5IRK1jdP3ME34omXiXls6AnW8XZ8RZrWhwz5Z5ae 1VhMl6EZe4fAH/pk0FLNBQG/0c0Uh/XVYi1xZJHU82McC6mB07CBVeJW/wYfAA== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=k5sdkreC; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org"; dmarc=pass (policy=none) header.from=gmail.com Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pfJD2-0006Lf-M3; Thu, 23 Mar 2023 07:34:20 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1peyFU-0003Uc-HN for guix-devel@gnu.org; Wed, 22 Mar 2023 09:11:28 -0400 Received: from mail-ed1-x52f.google.com ([2a00:1450:4864:20::52f]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1peyFR-0007SH-Vh for guix-devel@gnu.org; Wed, 22 Mar 2023 09:11:28 -0400 Received: by mail-ed1-x52f.google.com with SMTP id y4so72814252edo.2 for ; Wed, 22 Mar 2023 06:11:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; t=1679490684; h=to:subject:message-id:date:from:in-reply-to:references:mime-version :from:to:cc:subject:date:message-id:reply-to; bh=2Um9sEmHwv7d+YG6U6cEHEGvyW1xSfVMIIMQeqgIUms=; b=k5sdkreC1IHxliZY2q+BKKwpaCq45oMkNGtxElzgaFvJmDDndh809IqYIU9MXQ7XV9 bwH7sNpjsnoLE0ZWSNI04hY5QogTOlgOcX27H6yukPMhtMr+MJqhAMK2ic1xoFoatede wYedJHGmORKpH1vdX/6pUtDAVkMLVLKMbQXGL2cCpBwwu04kVvkDF8uEGWFh5JKgHXR4 c5f1AMilW1yLw6dMXRFaeY4Fcr8oddLXJtbBjmQdLiD4Dfq0vBMsIbqy4Qgk+X3MO0pB CBkD7XzFuvXe2onzuyqj75I8P+OY1hnVnlfQSytUaoLc07NLRz3O5s7UKQkVGIQAeVCv 58DQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1679490684; h=to:subject:message-id:date:from:in-reply-to:references:mime-version :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=2Um9sEmHwv7d+YG6U6cEHEGvyW1xSfVMIIMQeqgIUms=; b=R4bClhXJo6k3D9neLWHxZhs5FGmjsUqT9olHmXJbFCv1DYqwhf7peGOJt8RnF881W8 jop35Grz1K9FbNNzG0x1B3SAazdSQl2TUc2iNSC6qo/6usV7j1Qe8loY9Fjb3bkboRpB i6LAhc+Rj3qYFX5TugZ37Bzq8jqC6KbEghfBBRHxX7TOo1oTZFdF74HL/JMf2869sjhH w5YRi+VSVgfbtywCAK3h13Q/UT3zAn5/M/9dl3VbXOz/ABmObaM+lf73CGcTeyGcPvkG KWJ+9+PUep1P10aOsg1cFL7P84DVFX7TAN5gwot6kf3YqdMktKlKDC1OY4lVnEqHW4/4 rqCg== X-Gm-Message-State: AO0yUKVUAzdVnTziQ8hCZMnHUP2mfNbfwQDxefqgAtSYiR/WitaO4rHq bXkK7cxBNNosp05fRAUrEY7YycHpoAwTKcJZYhT9RdeD0F8= X-Google-Smtp-Source: AK7set+YXY4VjC32Q74qQmPH98M8Yf5ftnIqf+Ahq2oqCvy6lCcjXQWBBgIz7TobbGY2ktAsfPO34VNovf6ly8hjzAE= X-Received: by 2002:a17:906:2709:b0:930:310:abef with SMTP id z9-20020a170906270900b009300310abefmr1191743ejc.3.1679490683978; Wed, 22 Mar 2023 06:11:23 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Vladilen Kozin Date: Wed, 22 Mar 2023 13:11:12 +0000 Message-ID: Subject: Re: shepherd service works on host but fails inside system container To: guix-devel@gnu.org Content-Type: multipart/alternative; boundary="0000000000009f4df405f77ce5b6" Received-SPF: pass client-ip=2a00:1450:4864:20::52f; envelope-from=vladilen.kozin@gmail.com; helo=mail-ed1-x52f.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-Mailman-Approved-At: Thu, 23 Mar 2023 07:34:17 -0400 X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: X-Migadu-Queue-Id: 5CE93378A0 X-Spam-Score: -6.48 X-Migadu-Spam-Score: -6.48 X-Migadu-Scanner: scn0.migadu.com List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: guix-devel-bounces+larch=yhetil.org@gnu.org X-Migadu-Country: US X-Migadu-Flow: FLOW_IN X-TUID: rQXE/MsXQFWj --0000000000009f4df405f77ce5b6 Content-Type: text/plain; charset="UTF-8" I guess capabilities aren't handled by container creation code yet: https://github.com/guix-mirror/guix/blob/master/gnu/build/linux-container.scm#L262 IIUC this would also effect the idea of isolating system services as described in https://guix.gnu.org/en/blog/2017/running-system-services-in-containers/. It may require deeper understanding of Linux caps, namescase and how it all handled by Guix code, I suppose. Not something I'll be able to pull off without hand-holding from core devs, sigh. On Wed, 22 Mar 2023 at 12:14, Vladilen Kozin wrote: > I now have a hypothesis as to what's happening. Could someone confirm or > disprove and maybe suggest a solution or point at existing workarounds. > > Host and container will share the exact same kernel that's unsurprisingly > already running, so the above has nothing to do with kernel modules or > settings. Fwiw I figured a way to find where kernel modules reside by doing: > > $ sudo dmesg | grep -i "kernel command line" > which shows where current system is inside the store and relevant > /lib/modules will be under it. We could then > --expose=/gnu/store/hash-system/lib/modules=/lib/modules if we wanted to. > > Real problem, IIUC, is with capabilities. Notion of "container" can be > misleading and evokes thoughts of "vm" when in practice its just a process > with some isolation applied to it. So, presently I'm guessing container > Shepherd maybe PID 1 inside its isolated environment, but from the host pow > it is just a process and one that unlike our host's shepherd may lack > certain capabilities and privileges to e.g. create new devices or load > kernel modules on request, etc. In the sense of > https://man7.org/linux/man-pages/man7/capabilities.7.html maybe? > > Am I on the right track? But then, how does one test services like that > that may require ability to modify devices etc? Have we "outgrown" > container and ought to `guix system vm` for such services? Or is there a > way to bless container shepherd with necessary capabilities? If not from > `guix system container` command line, then perhaps dropping down to the > underlying programmatic interface i.e. whatever `guix system container` > ends up calling to containerize a system? > > Thanks > > > On Wed, 22 Mar 2023 at 10:20, Vladilen Kozin > wrote: > >> Hello guix. >> >> I put together a tailscale system service that's meant to start a >> tailscale daemon managed by the system shepherd, that is to say that my >> `tailscaled-service-type` specifies `(service-extension >> shepherd-root-service-type tailscaled-shepherd-service)`, where >> `tailscaled-shepherd-service` creates a `shepherd-service` with (provision >> '(tailscaled)) and (requirement '(networking)). >> >> I tested it by lowering to store via `shepherd-service-file` and then >> loading the generated script via `sudo herd load root ...`. This works fine >> and the daemon starts without a problem. >> >> Next, I try to spawn tailscaled as part of my OS definition: >> (services (cons* (service tailscaled-service-type >> (tailscaled-configuration)) %base-services)) >> ;; tried %desktop-services too >> >> To test, we create a container: >> sudo guix system -K -L /home/vlad/Code/fullmeta-guix/channel container >> os.scm --network --expose=/dev/net=/dev >> >> Earlier runs had it complaining that /dev/net/tun was missing, so I >> exposed that. Dunno if that's how I'm supposed to handle this. Now, >> /var/log/messages show: >> >> Mar 22 09:38:48 twgter shepherd[1]: [tailscaled] 2023/03/22 09:38:48 >> Linux kernel version: 5.18.10 >> Mar 22 09:38:48 twgter shepherd[1]: [tailscaled] 2023/03/22 09:38:48 is >> CONFIG_TUN enabled in your kernel? `modprobe tun` failed with: >> Mar 22 09:38:48 twgter shepherd[1]: [tailscaled] 2023/03/22 09:38:48 >> wgengine.NewUserspaceEngine(tun "tailscale0") error: >> tstun.New("tailscale0"): operation not permitted >> >> I feel like maybe I'm missing some kernel modules, but I would've >> expected host and container to share the kernel, so I dunno. In fact, when >> I randomly attempted adding (kernel-arguments (cons* "CONFIG_TUN=m" >> %default-kernel-arguments)) to my os definition, resulting script hash came >> out the same, which tells me, containers don't even look at these kernel >> params when generating a script. >> >> Any guesses as to why this works under host but not inside container? >> >> Relatedly, does anyone have a nicer workflow they use to define and test >> shepherd services? Such containerization was the next step in testing the >> service and would've been ok were it not for the above failure, but the >> initial indirection with lowering to store, then `sudo herd load root ...` >> is a bit too involved and "indirect" for my liking as well - anyone has an >> improved way of developing shepherd services? >> >> Thanks! >> -- >> Best regards >> Vlad Kozin >> > > > -- > Best regards > Vlad Kozin > -- Best regards Vlad Kozin --0000000000009f4df405f77ce5b6 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
I guess capabilities aren't handled by container creation cod= e yet:
https://github.com/guix-mirror/guix/blob/master/gn= u/build/linux-container.scm#L262=C2=A0

IIUC this would also effect the= idea of isolating system services as described in=C2=A0https://g= uix.gnu.org/en/blog/2017/running-system-services-in-containers/.
<= div class=3D"gmail_default" style=3D"font-family:arial,sans-serif">
It ma= y require deeper understanding of Linux caps, namescase and how it all hand= led by Guix code, I suppose. Not something I'll be able to pull off wit= hout hand-holding from core devs, sigh.

On Wed, 22 Mar 2023 at 12:14, = Vladilen Kozin <vladilen.koz= in@gmail.com> wrote:
I now have a hypothesis as to what's happening. C= ould someone confirm or disprove and maybe suggest a solution or point at e= xisting workarounds.

Host and container will share the exact same kernel t= hat's unsurprisingly already running, so the above has nothing to do wi= th kernel modules or settings. Fwiw I figured a way to find where kernel mo= dules reside by doing:

$ sudo dmesg | grep -i "kernel command line&qu= ot;
which shows where current system is inside the store and relevant /lib/mo= dules will be under it. We could then --expose=3D/gnu/store/hash-system/lib= /modules=3D/lib/modules if we wanted to.

Real problem, IIUC, is with capab= ilities. Notion of "container" can be misleading and evokes thoug= hts of "vm" when in practice its just a process with some isolati= on applied to it. So, presently I'm guessing container Shepherd maybe P= ID 1 inside its isolated environment, but from the host pow it is just a pr= ocess and one that unlike our host's shepherd may lack certain capabili= ties and privileges to e.g. create new devices or load kernel modules on re= quest, etc. In the sense of=C2=A0https://man7.org/linux/man-pa= ges/man7/capabilities.7.html maybe?

Am I on the right track? But then,= how does one test services like that that may require ability to modify de= vices etc? Have we "outgrown" container and ought to `guix system= vm` for such services? Or is there a way to bless container shepherd with = necessary capabilities? If not from `guix system container` command line, t= hen perhaps dropping down to the underlying programmatic interface i.e. wha= tever `guix system container` ends up calling to containerize a system?=C2= =A0

Thanks


On Wed, 22 Mar 2023 at 10:20, Vladilen Kozin <vladilen.kozin@gmai= l.com> wrote:
Hello guix.

I put together a tailscale system service that&= #39;s meant to start a tailscale daemon managed by the system shepherd, tha= t is to say that my `tailscaled-service-type` specifies `(service-extension= shepherd-root-service-type tailscaled-shepherd-service)`, where `tailscale= d-shepherd-service` creates a `shepherd-service` with=C2=A0(provision '= (tailscaled)) and (requirement '(networking)).

I tested it by lowering= to store via `shepherd-service-file` and then loading the generated script= via `sudo herd load root ...`. This works fine and the daemon starts witho= ut a problem.

Next, I try to spawn tailscaled as part of my OS definition:=
(= services (cons*=C2=A0(service tailscaled-service-type (tailscaled-configura= tion)) %base-services))
;; tried %desktop-services too

To test, we create a = container:
sudo guix system -K -L /home/vlad/Code/fullmeta-guix/channel conta= iner os.scm --network --expose=3D/dev/net=3D/dev

Earlier runs had it c= omplaining that /dev/net/tun was missing, so I exposed that. Dunno if that&= #39;s how I'm supposed to handle this. Now, /var/log/messages show:

<= /div>
Ma= r 22 09:38:48 twgter shepherd[1]: [tailscaled] 2023/03/22 09:38:48 Linux ke= rnel version: 5.18.10
Mar 22 09:38:48 twgter shepherd[1]: [tailscaled] 2= 023/03/22 09:38:48 is CONFIG_TUN enabled in your kernel? `modprobe tun` fai= led with:
Mar 22 09:38:48 twgter shepherd[1]: [tailscaled] 2023/03/22 09= :38:48 wgengine.NewUserspaceEngine(tun "tailscale0") error: tstun= .New("tailscale0"): operation not permitted

I feel like maybe I'm missing some kernel modules, but I would've = expected host and container to share the kernel, so I dunno. In fact, when = I randomly attempted adding=C2=A0(kernel-arguments (cons* "CONFIG_TUN= =3Dm" %default-kernel-arguments)) to my os definition, resulting scrip= t hash came out the same, which tells me, containers don't even look at= these kernel params when generating a script.

Any guesses as to why this = works under host but not inside container?

Relatedly, does anyone ha= ve a nicer workflow they use to define and test shepherd services? Such con= tainerization was the next step in testing the service and would've bee= n ok were it not for the above failure, but the initial indirection with lo= wering to store, then `sudo herd load root ...` is a bit too involved and &= quot;indirect" for my liking as well - anyone has an improved way of d= eveloping shepherd services?

Thanks!
--
Best regards
Vlad Kozin


--
Best regards
Vlad Kozin


--
Be= st regards
Vlad Kozin
--0000000000009f4df405f77ce5b6--