From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp2.migadu.com ([2001:41d0:303:e16b::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms1.migadu.com with LMTPS id iBC2AWFhNWZE8gAAe85BDQ:P1 (envelope-from ) for ; Sat, 04 May 2024 00:12:49 +0200 Received: from aspmx1.migadu.com ([2001:41d0:303:e16b::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp2.migadu.com with LMTPS id iBC2AWFhNWZE8gAAe85BDQ (envelope-from ) for ; Sat, 04 May 2024 00:12:49 +0200 X-Envelope-To: larch@yhetil.org Authentication-Results: aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=autistici.org header.s=stigmate header.b=Oj804NXg; spf=pass (aspmx1.migadu.com: domain of "guix-patches-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-patches-bounces+larch=yhetil.org@gnu.org"; dmarc=pass (policy=none) header.from=gnu.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1714774368; h=from:from:sender:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding:resent-cc: resent-from:resent-sender:resent-message-id:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post:dkim-signature; bh=HH2mMLsliNtIIe8z875yGhYONaeqILotnGkQ2QwHgsI=; b=Ts8M4H1XoQF/37FRTwElD4ywSzsvBOzdcySXQMgE1OdetJ2mzrG0uoi2YSPK7deyUBSVXF Jq9CBkpPoo4dT3fZn8CoAVs4pnkYvTKskpQlx4jC4PNclG0kfSgV8V7z9+Zwn/BKucmGw5 WNRK0ButSf0LbF7kFOpn10IWccalRwQ6VywpJfcXJP5euTrbofqZz6fRJeEbVA+6aQ0kxo Rqegiknycc6F650pkWp58VxuCifVqmV8MBcq1LKdKVp1xNsq9RAmZHbBkegIANLS2hZkO2 w6a3ttU4YqOEA1kPi1JkwYDQ8u0sSw/iKBowrSBJhzXf34CT16LKKyHdg2dMPA== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=autistici.org header.s=stigmate header.b=Oj804NXg; spf=pass (aspmx1.migadu.com: domain of "guix-patches-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-patches-bounces+larch=yhetil.org@gnu.org"; dmarc=pass (policy=none) header.from=gnu.org ARC-Seal: i=1; s=key1; d=yhetil.org; t=1714774368; a=rsa-sha256; cv=none; b=awbTDwGa9BwsHYOcqRypEbiZ1BPP7QQXnvMg6XQbWNPs6ZeGCT/gkx8nOSkxOHvjyA5ZoM iXnlad+qD+NAEyqvEwc/LkAQ4F9LWZdneQRjuCcCIVSS66tzcU2vFkhLByf2fR8HN7mYEL aTi1HmHTgCJE0XNpKK2eV3AXJWfuF5JffZ5scUw1vF7xbMnc+kG9+ChgiyA1xwPe2NpS3l wi54+LiM5PW/z5DEfM5APtUMQouGUu5YqvvI66KgS0o8Rea100lX0m6xcqXhc8I1UqVebo pH9rMG8p4Jt9FDNaZ3eG2bP7R7VdbG/JDY7tpG/zpySzZ69wMONVtMvpHKyk2w== Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 9CB8B10D36 for ; Sat, 4 May 2024 00:12:48 +0200 (CEST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1s3190-0007cU-75; Fri, 03 May 2024 18:12:42 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1s318x-0007bA-Rv for guix-patches@gnu.org; Fri, 03 May 2024 18:12:40 -0400 Received: from debbugs.gnu.org ([2001:470:142:5::43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1s318x-0006ga-IX for guix-patches@gnu.org; Fri, 03 May 2024 18:12:39 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1s319K-0005aX-5V for guix-patches@gnu.org; Fri, 03 May 2024 18:13:02 -0400 X-Loop: help-debbugs@gnu.org Subject: [bug#67613] [PATCH v3 4/5] gnu: docker: Allow passing tarballs for images in oci-container-configuration. Resent-From: Giacomo Leidi Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Fri, 03 May 2024 22:13:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 67613 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: To: 67613@debbugs.gnu.org Cc: Giacomo Leidi Received: via spool by 67613-submit@debbugs.gnu.org id=B67613.171477432521431 (code B ref 67613); Fri, 03 May 2024 22:13:02 +0000 Received: (at 67613) by debbugs.gnu.org; 3 May 2024 22:12:05 +0000 Received: from localhost ([127.0.0.1]:49149 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1s318O-0005ZS-GT for submit@debbugs.gnu.org; Fri, 03 May 2024 18:12:05 -0400 Received: from confino.investici.org ([2a11:7980:1::2:0]:32315) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1s318L-0005Yp-ON for 67613@debbugs.gnu.org; Fri, 03 May 2024 18:12:02 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=autistici.org; s=stigmate; t=1714774298; bh=HH2mMLsliNtIIe8z875yGhYONaeqILotnGkQ2QwHgsI=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Oj804NXgGvTkxRBoZcmU5CK0BFSZ89Fpcy1dOL8RFKu28/f5HyXQh7vHNCL+ozejA KrLAA9zs4OWF//TQuGJv77Mg2Mhlt67bvZKTzUtHuhrNLolSer+dWny4rJE7sG6mQm pNjDxPAP+0nVA19GZ7Utf81aSiwAgxLoL5gS1jPo= Received: from mx1.investici.org (unknown [127.0.0.1]) by confino.investici.org (Postfix) with ESMTP id 4VWQ4p48mNz11CB; Fri, 3 May 2024 22:11:38 +0000 (UTC) Received: from [93.190.126.19] (mx1.investici.org [93.190.126.19]) (Authenticated sender: goodoldpaul@autistici.org) by localhost (Postfix) with ESMTPSA id 4VWQ4p3hD6z11BS; Fri, 3 May 2024 22:11:38 +0000 (UTC) Date: Sat, 4 May 2024 00:11:16 +0200 Message-ID: <0c4d11411cd8efcea9c3252146cbc69c9155f2e9.1714774277.git.goodoldpaul@autistici.org> X-Mailer: git-send-email 2.41.0 In-Reply-To: <306393d4c3b28f242fb735d793ef127427f3d072.1714774276.git.goodoldpaul@autistici.org> References: <306393d4c3b28f242fb735d793ef127427f3d072.1714774276.git.goodoldpaul@autistici.org> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-to: Giacomo Leidi X-ACL-Warn: , Giacomo Leidi via Guix-patches From: Giacomo Leidi via Guix-patches via Errors-To: guix-patches-bounces+larch=yhetil.org@gnu.org Sender: guix-patches-bounces+larch=yhetil.org@gnu.org X-Migadu-Flow: FLOW_IN X-Migadu-Country: US X-Migadu-Spam-Score: -6.59 X-Spam-Score: -6.59 X-Migadu-Queue-Id: 9CB8B10D36 X-Migadu-Scanner: mx13.migadu.com X-TUID: HAd+uh1r18sS This commit allows for loading an OCI image tarball before running an OCI backed Shepherd service. It does so by adding a one shot Shepherd service to the dependencies of the OCI backed service that at boot runs docker load on the tarball. * gnu/services/docker.scm (oci-image): New record; (lower-oci-image): new variable, lower it; (string-or-oci-image?): sanitize it; (oci-container-configuration)[image]: allow also for oci-image records; (oci-container-shepherd-service): use it; (%oci-image-loader): new variable. Change-Id: Ie504f479ea0d47f74b0ec5df9085673ffd3f639d --- doc/guix.texi | 70 +++++++++++- gnu/services/docker.scm | 244 ++++++++++++++++++++++++++++++++++++---- 2 files changed, 286 insertions(+), 28 deletions(-) diff --git a/doc/guix.texi b/doc/guix.texi index 2b9cc5602c..451bee5615 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -40390,6 +40390,17 @@ Miscellaneous Services @lisp (service oci-container-service-type (list + (oci-container-configuration + (image + (oci-image + (repository "guile") + (tag "3") + (value (specifications->manifest '("guile"))) + (pack-options '(#:symlinks (("/bin/guile" -> "bin/guile")) + #:max-layers 2)))) + (entrypoint "/bin/guile") + (command + '("-c" "(display \"hello!\n\")"))) (oci-container-configuration (image "prom/prometheus") (network "host") @@ -40466,9 +40477,10 @@ Miscellaneous Services @uref{https://docs.docker.com/engine/reference/commandline/run/#env,upstream} documentation for semantics. -@item @code{image} (type: string) -The image used to build the container. Images are resolved by the -Docker Engine, and follow the usual format +@item @code{image} (type: string-or-oci-image) +The image used to build the container. It can be a string or an +@code{oci-image} record. Strings are resolved by the Docker Engine, and +follow the usual format @code{myregistry.local:5000/testing/test-image:tag}. @item @code{provision} (default: @code{""}) (type: string) @@ -40529,6 +40541,58 @@ Miscellaneous Services @end deftp +@c %end of fragment + +@c %start of fragment + +@deftp {Data Type} oci-image +Available @code{oci-image} fields are: + +@table @asis +@item @code{repository} (type: string) +A string like @code{myregistry.local:5000/testing/test-image} that names +the OCI image. + +@item @code{tag} (default: @code{"latest"}) (type: string) +A string representing the OCI image tag. Defaults to @code{latest}. + +@item @code{value} (type: oci-lowerable-image) +A @code{manifest} or @code{operating-system} record that will be lowered +into an OCI compatible tarball. Otherwise this field's value can be a +gexp or a file-like object that evaluates to an OCI compatible tarball. + +@item @code{pack-options} (default: @code{'()}) (type: list) +An optional set of keyword arguments that will be passed to the +@code{docker-image} procedure from @code{guix scripts pack}. They can +be used to replicate @command{guix pack} behavior: + +@lisp +(oci-image + (repository "guile") + (tag "3") + (value + (specifications->manifest '("guile"))) + (pack-options '(#:symlinks (("/bin/guile" -> "bin/guile")) + #:max-layers 2))) +@end lisp + +If the @code{value} field is an @code{operating-system} record, this field's +value will be ignored. + +@item @code{system} (default: @code{""}) (type: string) +Attempt to build for a given system, e.g. "i686-linux" + +@item @code{target} (default: @code{""}) (type: string) +Attempt to cross-build for a given triple, e.g. "aarch64-linux-gnu" + +@item @code{grafts?} (default: @code{#f}) (type: boolean) +Whether to allow grafting or not in the pack build. + +@end table + +@end deftp + + @c %end of fragment @cindex Audit diff --git a/gnu/services/docker.scm b/gnu/services/docker.scm index a5b1614fa9..7aff8dcc5f 100644 --- a/gnu/services/docker.scm +++ b/gnu/services/docker.scm @@ -23,11 +23,14 @@ ;;; along with GNU Guix. If not, see . (define-module (gnu services docker) + #:use-module (gnu image) #:use-module (gnu services) #:use-module (gnu services configuration) #:use-module (gnu services base) #:use-module (gnu services dbus) #:use-module (gnu services shepherd) + #:use-module (gnu system) + #:use-module (gnu system image) #:use-module (gnu system setuid) #:use-module (gnu system shadow) #:use-module (gnu packages admin) ;shadow @@ -37,7 +40,11 @@ (define-module (gnu services docker) #:use-module (guix diagnostics) #:use-module (guix gexp) #:use-module (guix i18n) + #:use-module (guix monads) #:use-module (guix packages) + #:use-module (guix profiles) + #:use-module ((guix scripts pack) #:prefix pack:) + #:use-module (guix store) #:use-module (srfi srfi-1) #:use-module (ice-9 format) #:use-module (ice-9 match) @@ -45,6 +52,16 @@ (define-module (gnu services docker) #:export (docker-configuration docker-service-type singularity-service-type + oci-image + oci-image? + oci-image-fields + oci-image-repository + oci-image-tag + oci-image-value + oci-image-pack-options + oci-image-target + oci-image-system + oci-image-grafts? oci-container-configuration oci-container-configuration? oci-container-configuration-fields @@ -52,9 +69,11 @@ (define-module (gnu services docker) oci-container-configuration-group oci-container-configuration-command oci-container-configuration-entrypoint + oci-container-configuration-host-environment oci-container-configuration-environment oci-container-configuration-image oci-container-configuration-provision + oci-container-configuration-requirement oci-container-configuration-network oci-container-configuration-ports oci-container-configuration-volumes @@ -62,7 +81,8 @@ (define-module (gnu services docker) oci-container-configuration-workdir oci-container-configuration-extra-arguments oci-container-service-type - oci-container-shepherd-service)) + oci-container-shepherd-service + %oci-container-accounts)) (define-maybe file-like) @@ -320,11 +340,68 @@ (define (oci-sanitize-extra-arguments value) but ~a was found") el)))) value)) +(define (oci-image-reference image) + (if (string? image) + image + (string-append (oci-image-repository image) + ":" (oci-image-tag image)))) + +(define (oci-lowerable-image? image) + (or (manifest? image) + (operating-system? image) + (gexp? image) + (file-like? image))) + +(define (string-or-oci-image? image) + (or (string? image) + (oci-image? image))) + (define list-of-symbols? (list-of symbol?)) (define-maybe/no-serialization string) +(define-configuration/no-serialization oci-image + (repository + (string) + "A string like @code{myregistry.local:5000/testing/test-image} that names +the OCI image.") + (tag + (string "latest") + "A string representing the OCI image tag. Defaults to @code{latest}.") + (value + (oci-lowerable-image) + "A @code{manifest} or @code{operating-system} record that will be lowered +into an OCI compatible tarball. Otherwise this field's value can be a gexp +or a file-like object that evaluates to an OCI compatible tarball.") + (pack-options + (list '()) + "An optional set of keyword arguments that will be passed to the +@code{docker-image} procedure from @code{guix scripts pack}. They can be used +to replicate @command{guix pack} behavior: + +@lisp +(oci-image + (repository \"guile\") + (tag \"3\") + (manifest (specifications->manifest '(\"guile\"))) + (pack-options + '(#:symlinks ((\"/bin/guile\" -> \"bin/guile\")) + #:max-layers 2))) +@end lisp + +If the @code{value} field is an @code{operating-system} record, this field's +value will be ignored.") + (system + (maybe-string) + "Attempt to build for a given system, e.g. \"i686-linux\"") + (target + (maybe-string) + "Attempt to cross-build for a given triple, e.g. \"aarch64-linux-gnu\"") + (grafts? + (boolean #f) + "Whether to allow grafting or not in the pack build.")) + (define-configuration/no-serialization oci-container-configuration (user (string "oci-container") @@ -372,8 +449,9 @@ (define-configuration/no-serialization oci-container-configuration documentation for semantics." (sanitizer oci-sanitize-environment)) (image - (string) - "The image used to build the container. Images are resolved by the Docker + (string-or-oci-image) + "The image used to build the container. It can be a string or an +@code{oci-image} record. Strings are resolved by the Docker Engine, and follow the usual format @code{myregistry.local:5000/testing/test-image:tag}.") (provision @@ -470,14 +548,122 @@ (define oci-container-configuration->options (list "-v" spec)) (oci-container-configuration-volumes config)))))))) +(define* (get-keyword-value args keyword #:key (default #f)) + (let ((kv (memq keyword args))) + (if (and kv (>= (length kv) 2)) + (cadr kv) + default))) + +(define (lower-operating-system os target system) + (mlet* %store-monad + ((tarball + (lower-object + (system-image (os->image os #:type docker-image-type)) + system + #:target target))) + (return tarball))) + +(define (lower-manifest name image target system) + (define value (oci-image-value image)) + (define options (oci-image-pack-options image)) + (define image-reference + (oci-image-reference image)) + (define image-tag + (let* ((extra-options + (get-keyword-value options #:extra-options)) + (image-tag-option + (and extra-options + (get-keyword-value extra-options #:image-tag)))) + (if image-tag-option + '() + `(#:extra-options (#:image-tag ,image-reference))))) + + (mlet* %store-monad + ((_ (set-grafting + (oci-image-grafts? image))) + (guile (set-guile-for-build (default-guile))) + (profile + (profile-derivation value + #:target target + #:system system + #:hooks '() + #:locales? #f)) + (tarball (apply pack:docker-image + `(,name ,profile + ,@options + ,@image-tag + #:localstatedir? #t)))) + (return tarball))) + +(define (lower-oci-image name image) + (define value (oci-image-value image)) + (define image-target (oci-image-target image)) + (define image-system (oci-image-system image)) + (define target + (if (maybe-value-set? image-target) + image-target + (%current-target-system))) + (define system + (if (maybe-value-set? image-system) + image-system + (%current-system))) + (with-store store + (run-with-store store + (match value + ((? manifest? value) + (lower-manifest name image target system)) + ((? operating-system? value) + (lower-operating-system value target system)) + ((or (? gexp? value) + (? file-like? value)) + value) + (_ + (raise + (formatted-message + (G_ "oci-image value must contain only manifest, +operating-system, gexp or file-like records but ~a was found") + value)))) + #:target target + #:system system))) + +(define (%oci-image-loader name image tag) + (let ((docker (file-append docker-cli "/bin/docker")) + (tarball (lower-oci-image name image))) + (with-imported-modules '((guix build utils)) + (program-file (format #f "~a-image-loader" name) + #~(begin + (use-modules (guix build utils) + (ice-9 popen) + (ice-9 rdelim)) + + (format #t "Loading image for ~a from ~a...~%" #$name #$tarball) + (define line + (read-line + (open-input-pipe + (string-append #$docker " load -i " #$tarball)))) + + (unless (or (eof-object? line) + (string-null? line)) + (format #t "~a~%" line) + (let ((repository&tag + (string-drop line + (string-length + "Loaded image: ")))) + + (invoke #$docker "tag" repository&tag #$tag) + (format #t "Tagged ~a with ~a...~%" #$tarball #$tag)))))))) + (define (oci-container-shepherd-service config) (define (guess-name name image) (if (maybe-value-set? name) name (string-append "docker-" - (basename (car (string-split image #\:)))))) + (basename + (if (string? image) + (first (string-split image #\:)) + (oci-image-repository image)))))) - (let* ((docker-command (file-append docker-cli "/bin/docker")) + (let* ((docker (file-append docker-cli "/bin/docker")) (user (oci-container-configuration-user config)) (group (oci-container-configuration-group config)) (host-environment @@ -486,6 +672,7 @@ (define (oci-container-shepherd-service config) (provision (oci-container-configuration-provision config)) (requirement (oci-container-configuration-requirement config)) (image (oci-container-configuration-image config)) + (image-reference (oci-image-reference image)) (options (oci-container-configuration->options config)) (name (guess-name provision image)) (extra-arguments @@ -496,30 +683,37 @@ (define (oci-container-shepherd-service config) (respawn? #f) (documentation (string-append - "Docker backed Shepherd service for image: " image)) + "Docker backed Shepherd service for " + (if (oci-image? image) name image) ".")) (start - #~(make-forkexec-constructor - ;; docker run [OPTIONS] IMAGE [COMMAND] [ARG...] - (list #$docker-command "run" "--rm" - "--name" #$name - #$@options #$@extra-arguments #$image #$@command) - #:user #$user - #:group #$group - #:environment-variables - (list #$@host-environment))) + #~(lambda () + (when #$(oci-image? image) + (invoke #$(%oci-image-loader + name image image-reference))) + (fork+exec-command + ;; docker run [OPTIONS] IMAGE [COMMAND] [ARG...] + (list #$docker "run" "--rm" "--name" #$name + #$@options #$@extra-arguments + #$image-reference #$@command) + #:user #$user + #:group #$group + #:environment-variables + (list #$@host-environment)))) (stop #~(lambda _ - (invoke #$docker-command "rm" "-f" #$name))) + (invoke #$docker "rm" "-f" #$name))) (actions - (list - (shepherd-action - (name 'pull) - (documentation - (format #f "Pull ~a's image (~a)." - name image)) - (procedure - #~(lambda _ - (invoke #$docker-command "pull" #$image))))))))) + (if (oci-image? image) + '() + (list + (shepherd-action + (name 'pull) + (documentation + (format #f "Pull ~a's image (~a)." + name image)) + (procedure + #~(lambda _ + (invoke #$docker "pull" #$image)))))))))) (define %oci-container-accounts (list (user-account -- 2.41.0