From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp11.migadu.com ([2001:41d0:306:2d92::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms9.migadu.com with LMTPS id wHteLkqDHWUABQEA9RJhRA:P1 (envelope-from ) for ; Wed, 04 Oct 2023 17:22:50 +0200 Received: from aspmx1.migadu.com ([2001:41d0:306:2d92::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp11.migadu.com with LMTPS id wHteLkqDHWUABQEA9RJhRA (envelope-from ) for ; Wed, 04 Oct 2023 17:22:50 +0200 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 4D88C3ADBB for ; Wed, 4 Oct 2023 17:22:50 +0200 (CEST) Authentication-Results: aspmx1.migadu.com; dkim=pass header.d=gnu.org header.s=fencepost-gnu-org header.b=J0NdC0Hh; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org"; dmarc=pass (policy=none) header.from=gnu.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1696432970; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post:dkim-signature; bh=AoG5PrR1mw8XlQJ9VD9hN+Dr2B3WxZf+BGotyzMO5tg=; b=WLj4HeXIWO+RbeAi1Mg8k9CeNrPMu0DCWfoYxrfcZKhdGLDOiyoh6/XDMk+NiM9lB+5W5V bdMpNW2hBgEnIbgfSZa0WGYiWgft3vDXmaZ0nBTHKcGmGZD32KAPJ0iveNJmWqti+ds6oV sWFnKeLaPTqqFWii1xgXuOpcFDq2h22wapmwpga4OEK01FukXjdV1X4BbLVVOY7F7S3HOQ fM+H3RFwJE1kt2wab8G+CKAqDQFykw1rPWCw6fhrlEdSCSyMR7nGE2B7GggupoG4m1KIaz UHjyFm/fzJGsDF+un+yeWMN42WxQE4Q7iVI2BCYRxl3lN9I7oKCev0TRYi2tHg== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=pass header.d=gnu.org header.s=fencepost-gnu-org header.b=J0NdC0Hh; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org"; dmarc=pass (policy=none) header.from=gnu.org ARC-Seal: i=1; s=key1; d=yhetil.org; t=1696432970; a=rsa-sha256; cv=none; b=BBs2vTbQtU8vT03fu2Qay8oGFIJmquyHCvCTj/i0DEIlvDKt5s7W31YqdqBtKx2+0artoG c1POFtuEczVIkMbnFnGQRM5I5CUanZax0hAo5zaUnLb5AOAgIJYhdzjXycBBjTxogk/LBn ZErWO+Lls1GJ3TdKYiFI2H7U0tQGsr6YaJ1lVMdh33B46XGb+EsMlgSaRUocmWsfFZNmMC uGCZDWOuD8A51gbJI/bNfsIhBsQaznq1UzLV5AxBqRI912q2EHd45WPvnbzv0Qj4X+Q9vR FF3JddVRhLSwwlhe3kBeCrlBYe9pNCFs4YxMGEcr0t5FXIO6uGCvFkODg7YXWw== Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qo3hR-0007m9-2D; Wed, 04 Oct 2023 11:22:09 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qo3hP-0007lg-4G for guix-devel@gnu.org; Wed, 04 Oct 2023 11:22:07 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qo3hO-0006eg-Gn; Wed, 04 Oct 2023 11:22:06 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:Date:References:In-Reply-To:Subject:To: From; bh=AoG5PrR1mw8XlQJ9VD9hN+Dr2B3WxZf+BGotyzMO5tg=; b=J0NdC0Hh1eLsAxcaLw4m 3dYSuRZgio/8nT1Du8vfh4bojscZKl9TXIrc+h9Tgd59MFrHAOYWcij1Y3tRtoo2QnUA1NF2bLbrG 9ql5Ey8nrAF8w+Lo+vxAV/Awo+N6Zj/hwAFgaMZOQJvyof9aaHOVD2zLTP4qjzLRV89qknvbVxFkq u7x1E+AC2kamdoBMoWvHfIarjsS7YQ8qQE7TGMcSynXhOWR5BAhUzLsnvpttyQq5texdgzRvLzAHv Lr0cC3OPXuJPf8icbFbrvbFfb6cbDnX31DqLzjm0uQsR1/fYiMWb6c6HmBdiYHcoAlObc0qotIkAT 0y/ElgdF0YVvSA==; From: =?utf-8?Q?Ludovic_Court=C3=A8s?= To: Tao Hansen Cc: guix-devel@gnu.org Subject: Re: Improving cgroups for fun and Kubernetes In-Reply-To: <87zg1bplu4.fsf@riseup.net> (Tao Hansen's message of "Sun, 24 Sep 2023 18:39:15 +0200") References: <87zg1bplu4.fsf@riseup.net> X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: Tridi 13 =?utf-8?Q?Vend=C3=A9miaire?= an 232 de la =?utf-8?Q?R=C3=A9volution=2C?= jour du Potiron X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-pc-linux-gnu Date: Wed, 04 Oct 2023 17:21:49 +0200 Message-ID: <87bkdee7le.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: guix-devel-bounces+larch=yhetil.org@gnu.org X-Migadu-Flow: FLOW_IN X-Migadu-Country: US X-Migadu-Spam-Score: -10.93 X-Migadu-Scanner: mx2.migadu.com X-Migadu-Queue-Id: 4D88C3ADBB X-Spam-Score: -10.93 X-TUID: 1ML9OoT6EFU4 Hi Tao, Tao Hansen skribis: > This is my second posting to the mailing list but the first using Gnus > and smtmpmail. If I've formatted anything poorly, don't hesitate to let > me know. Looks perfect to me. :-) > I've been spending a silly amount of time trying to get a local flavor > of Kubernetes running on Guix System. I wanted to share my experience > and also solicit feedback from Guix's developers on how to improve the > cgroups implementation such that those who follow me will have an easier > time of it. I=E2=80=99ve never used Kubernetes, but I=E2=80=99m confident you=E2=80=99r= e not the only interested in using it on Guix System! [...] > The second problem is kind and minikube are both expecting Delegate=3Dyes > to be set, which is a systemd function that allows these tools to set > cgroups limits. The limits it's expecting to control are cpu, cpuset, > memory and pids. We can force these privileges like so, echo "+cpu > +cpuset +memory +pids" >> /sys/fs/cgroup/cgroup.subtree_control How about having a Shepherd service that does writes to that =E2=80=98cgroup.subtree_control=E2=80=99 file as you write above? > To fix the first problem we can run > > g=3Dusers && sudo chgrp -R ${g} /sys/fs/cgroup/ > u=3D$USER && sudo chown -R ${u}: /sys/fs/cgroup What does Debian do? Perhaps there=E2=80=99s a =E2=80=9Ccgroup=E2=80=9D gr= oup (in /etc/groups) that users who want to user podman need to belong to, similar to the =E2=80=98kvm=E2=80=99 group for those who want to access /dev/kvm? Or maybe we should create a sub-tree specifically for podman usage? At any rate, we could again have a Shepherd service that sets ownership on the relevant file tree. > Once we've addressed the first and second problem, the rest is > relatively easy: we need to make iptables (and iptables' modules so just > the package isn't enough: we need Guix's service) available. We need to > set a range of user IDs and group IDs for Podman to make use of > rootlessly, and finally we need to set a container policy otherwise Podman > can't pull any image from anywhere. All of those can be done from inside > our Guix System configuration file. Right, we should populate /etc/subuid by default (I tried to use subordinate UIDs in the past, by invoking =E2=80=98newuidmap=E2=80=99, but = never managed to get it to work.) > Here's what that Guix System configuration looks like: > > ;; Rootless Podman requires the next 4 services > ;; we're using the iptables service purely to make its resources > ;; available to minikube and kind > > (service iptables-service-type > (iptables-configuration > (ipv4-rules (plain-file "iptables.rules" "*filter > :INPUT ACCEPT > :FORWARD ACCEPT > :OUTPUT ACCEPT > COMMIT > ")) > (ipv6-rules (plain-file "ip6tables.rules" "*filter > :INPUT ACCEPT > :FORWARD ACCEPT > :OUTPUT ACCEPT > COMMIT > ")))) > (simple-service 'etc-subuid etc-service-type > (list `("subuid" ,(plain-file "subuid" > (string-append "root:0:65536\n" username ":100000:65536\n"))))) > (simple-service 'etc-subgid etc-service-type > (list `("subgid" ,(plain-file "subgid" > (string-append "root:0:65536\n" username ":100000:65536\n"))))) > (service pam-limits-service-type > (list > (pam-limits-entry "*" 'both 'nofile 100000))) > (simple-service 'etc-container-policy etc-service-type > (list `("containers/policy.json", (plain-file > "policy.json" "{\"default\": [{\"type\": > \"insecureAcceptAnything\"}]}")))) > %my-services Looks great! We should probably consider /etc/{subuid,subgid} support separately, but otherwise it looks like you already have the start of a =E2=80=98rootless-podman-service-type=E2=80=99 (or similar). Thanks, Ludo=E2=80=99.