From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp2.migadu.com ([2001:41d0:700:3204::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms8.migadu.com with LMTPS id sK35KrE4imVk7AAAkFu2QA (envelope-from ) for ; Tue, 26 Dec 2023 03:21:37 +0100 Received: from aspmx1.migadu.com ([2001:41d0:303:e224::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp2.migadu.com with LMTPS id GEDnJLE4imUdrAAAe85BDQ (envelope-from ) for ; Tue, 26 Dec 2023 03:21:37 +0100 X-Envelope-To: larch@yhetil.org Authentication-Results: aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=gmail.com header.s=20230601 header.b=YWO1WIja; spf=pass (aspmx1.migadu.com: domain of "guix-patches-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-patches-bounces+larch=yhetil.org@gnu.org"; dmarc=fail reason="SPF not aligned (relaxed)" header.from=gmail.com (policy=none) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1703557297; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:resent-cc: resent-from:resent-sender:resent-message-id:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post:dkim-signature; bh=B4CW8S5mwrxE9wU3Tigfd4N1Fl8MimAsHBmgn6Zu100=; b=p2oEKtBwIQUWZ4d5/Bmz/xbejEwhGuoTPjM0Or6E+Ojk224LOKx7SU2nl72DSKWGoAsNC2 RIKsUBerOe/79krqQKX9v7Ol+0vocFgtWVCSIvomKs2MIGN2t+uz+kddKDuog64idMVWpf VrPm7hi6ha7J7F1DojI7XI7/btlQvWC2wtfVom314D9FbFVrqGGNP944tbJFTu/VQlLsS1 AnaFqQZJ9QM84c1EQM3oyNe6SuCsPtTDelpPhydZYfs0PXgb4wK9lelpn8Rt8ymHai35/2 oVw1dWO2b6UFULMW74b726BUY6z5Qj1DG+3hbTnuQh57nKNX8Ku0aujaB9SwwA== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1703557297; a=rsa-sha256; cv=none; b=pMJft8/W2xu/6BaM0HkcU6upkxsYm25/jmnueikXdHqH5FQF7qSvmoMRUsVNzU94A7h/Yz i5ZliEjo2p0hBGGKMmOOxpoYN/9albpbm72tYk6xr0IF/7Wsa83/HqCRRVU6QS0YD/vXOz OSv7pWZHm7TiIcuJAR+NP1CIQp8g/HzzCHnFnjpczmzeHefKwApOPej08l1FpoPm/PoW6Q RMiL0eZFjBrm9Ib6PeyR8+bOXAHzXTmsHa936Y/9ayQyuQ6rn7VJ7xflO868Yd5YVi6Pcv GjaQwML6qsMyXwa8+FNqtynLQ9tlv3pE2HPRiH71VcnzzMyjxJlWJAftC/YmTg== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=gmail.com header.s=20230601 header.b=YWO1WIja; spf=pass (aspmx1.migadu.com: domain of "guix-patches-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-patches-bounces+larch=yhetil.org@gnu.org"; dmarc=fail reason="SPF not aligned (relaxed)" header.from=gmail.com (policy=none) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 41A1837015 for ; Tue, 26 Dec 2023 03:21:37 +0100 (CET) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rHx40-0006za-7P; Mon, 25 Dec 2023 21:21:00 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rHx3z-0006z0-GZ for guix-patches@gnu.org; Mon, 25 Dec 2023 21:20:59 -0500 Received: from debbugs.gnu.org ([2001:470:142:5::43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1rHx3z-0007i0-7m; Mon, 25 Dec 2023 21:20:59 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1rHx43-0000mw-Jz; Mon, 25 Dec 2023 21:21:03 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#62153] [PATCH 3/5] guix: docker: Build layered images. Resent-From: Oleg Pykhalov Original-Sender: "Debbugs-submit" Resent-CC: guix@cbaines.net, dev@jpoiret.xyz, ludo@gnu.org, othacehe@gnu.org, rekado@elephly.net, zimon.toutoune@gmail.com, me@tobias.gr, guix-patches@gnu.org Resent-Date: Tue, 26 Dec 2023 02:21:03 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 62153 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 62153@debbugs.gnu.org Cc: Oleg Pykhalov , Christopher Baines , Josselin Poiret , Ludovic =?UTF-8?Q?Court=C3=A8s?= , Mathieu Othacehe , Ricardo Wurmus , Simon Tournier , Tobias Geerinckx-Rice X-Debbugs-Original-Xcc: Christopher Baines , Josselin Poiret , Ludovic =?UTF-8?Q?Court=C3=A8s?= , Mathieu Othacehe , Ricardo Wurmus , Simon Tournier , Tobias Geerinckx-Rice Received: via spool by 62153-submit@debbugs.gnu.org id=B62153.17035572092925 (code B ref 62153); Tue, 26 Dec 2023 02:21:03 +0000 Received: (at 62153) by debbugs.gnu.org; 26 Dec 2023 02:20:09 +0000 Received: from localhost ([127.0.0.1]:55616 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rHx3A-0000l0-JC for submit@debbugs.gnu.org; Mon, 25 Dec 2023 21:20:09 -0500 Received: from mail-lf1-x130.google.com ([2a00:1450:4864:20::130]:38233) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rHx33-0000jm-7P for 62153@debbugs.gnu.org; Mon, 25 Dec 2023 21:20:04 -0500 Received: by mail-lf1-x130.google.com with SMTP id 2adb3069b0e04-50e79299da9so148826e87.1 for <62153@debbugs.gnu.org>; Mon, 25 Dec 2023 18:19:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1703557188; x=1704161988; darn=debbugs.gnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=B4CW8S5mwrxE9wU3Tigfd4N1Fl8MimAsHBmgn6Zu100=; b=YWO1WIjaxNOwYz/0PB1myKwtlkcq4rjfgKUQuCy7jbuXSyZE/C6UFC5gtGwwVQFVjR KVJwbW3rl1/8hR1gPm5kIinkZzI856WNWDgU8ED792RQlqnYjc83sKyoD3P8LSG5QmCr Wm6NdAjcxUa+q/djP6qLqyQVzEFPEBZ4IWRevumhkM9FFjmFLC46dgVyV3Qmg9oUS6Bz 2jqIezLM3J8Y/o86cgERmmWuCwfRP9vncWOqz3ek8FzGinrQ7NkJL2A/u4UOYIJGkycX rqz4HF1YMmw8Q9CPO5s1MH5EzTbKI4lWIxXBqb2MvzmbXs6R9VwbRQWbj6b1jofwZoJk Isxw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1703557188; x=1704161988; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=B4CW8S5mwrxE9wU3Tigfd4N1Fl8MimAsHBmgn6Zu100=; b=uo320T4sX4zO+VKPyx0/Ai14oo92cO9XaqQPlY64mXWDnGqDxQq0F7fGgj3vPzB1nP CBWsF2dElBiYh7Zl6z/c7ykq27tQUoq47Kdf8owWlNd+JshDgl1Sm/l1EBKVXB9gYbXq 2QJU6R8J4RTzGp0T45UKl2MPzhEYrZrKOBqJ4MRTRPIEBW93cPiKTcj4i++FFEYehTti caUeKqfX6JxLVGpsxQ1zWhkehCdI/ryzKo9+QeO27vki2QaVDIvDsIEIt1j5mCPRky0h SbDZ4+cwHVdmATTI1hCnXn1IASKwFqrp47hv2NNAQ8i1NgLZQ7DezaezUyEaFfX7O1T3 OgKA== X-Gm-Message-State: AOJu0YyhtaPwtqsQHmf+/2wRWKw3viDFI8a61MwckZP0KRBJLC2xHg9g fKUBn8RhXqBblF6owgCC49B4deZFmhk= X-Google-Smtp-Source: AGHT+IEmtJ9jElPwdlzxbbQk89wdTmtszxClF2esPBA6hju6UpZDj9fvWN+r2Gmem9IEnA/2kFIw3Q== X-Received: by 2002:a2e:9dd2:0:b0:2ca:143:a52f with SMTP id x18-20020a2e9dd2000000b002ca0143a52fmr5071742ljj.2.1703557187638; Mon, 25 Dec 2023 18:19:47 -0800 (PST) Received: from guixsd.wugi.info ([88.201.161.72]) by smtp.gmail.com with ESMTPSA id e11-20020a05651c038b00b002cca609ef8bsm1633436ljp.111.2023.12.25.18.19.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 25 Dec 2023 18:19:46 -0800 (PST) From: Oleg Pykhalov Date: Tue, 26 Dec 2023 05:18:55 +0300 Message-ID: <49f8906ba06af461e17d9badcbbf3967f1a8be3b.1703556298.git.go.wigust@gmail.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: References: MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+larch=yhetil.org@gnu.org Sender: guix-patches-bounces+larch=yhetil.org@gnu.org X-Migadu-Flow: FLOW_IN X-Migadu-Country: US X-Migadu-Spam-Score: -3.64 X-Spam-Score: -3.64 X-Migadu-Queue-Id: 41A1837015 X-Migadu-Scanner: mx12.migadu.com X-TUID: BV4zZxNYqzyd * guix/docker.scm (%docker-image-max-layers): New variable. (size-sorted-store-items, create-empty-tar): New procedures. (config, manifest, build-docker-image): Build layered images. Change-Id: I4c8846bff0a3ceccb77e6bdf95d4942e5c3efe41 --- guix/docker.scm | 212 +++++++++++++++++++++++++++++++++++++----------- 1 file changed, 166 insertions(+), 46 deletions(-) diff --git a/guix/docker.scm b/guix/docker.scm index 5e6460f43f..5deca2afdb 100644 --- a/guix/docker.scm +++ b/guix/docker.scm @@ -3,6 +3,7 @@ ;;; Copyright © 2017, 2018, 2019, 2021 Ludovic Courtès ;;; Copyright © 2018 Chris Marusich ;;; Copyright © 2021 Maxim Cournoyer +;;; Copyright © 2023 Oleg Pykhalov ;;; ;;; This file is part of GNU Guix. ;;; @@ -29,16 +30,27 @@ (define-module (guix docker) with-directory-excursion invoke)) #:use-module (gnu build install) + #:use-module ((guix build store-copy) + #:select (file-size)) #:use-module (json) ;guile-json #:use-module (srfi srfi-1) #:use-module (srfi srfi-19) #:use-module (srfi srfi-26) + #:use-module (srfi srfi-71) #:use-module ((texinfo string-utils) #:select (escape-special-chars)) #:use-module (rnrs bytevectors) #:use-module (ice-9 ftw) #:use-module (ice-9 match) - #:export (build-docker-image)) + #:export (%docker-image-max-layers + build-docker-image)) + +;; The maximum number of layers allowed in a Docker image is typically around +;; 128, although it may vary depending on the Docker daemon. However, we +;; recommend setting the limit to 100 to ensure sufficient room for future +;; extensions. +(define %docker-image-max-layers + #f) ;; Generate a 256-bit identifier in hexadecimal encoding for the Docker image. (define docker-id @@ -92,12 +104,12 @@ (define (canonicalize-repository-name name) (make-string (- min-length l) padding-character))) (_ normalized-name)))) -(define* (manifest path id #:optional (tag "guix")) +(define* (manifest path layers #:optional (tag "guix")) "Generate a simple image manifest." (let ((tag (canonicalize-repository-name tag))) `#(((Config . "config.json") (RepoTags . #(,(string-append tag ":latest"))) - (Layers . #(,(string-append id "/layer.tar"))))))) + (Layers . ,(list->vector layers)))))) ;; According to the specifications this is required for backwards ;; compatibility. It duplicates information provided by the manifest. @@ -106,8 +118,8 @@ (define* (repositories path id #:optional (tag "guix")) `((,(canonicalize-repository-name tag) . ((latest . ,id))))) ;; See https://github.com/opencontainers/image-spec/blob/master/config.md -(define* (config layer time arch #:key entry-point (environment '())) - "Generate a minimal image configuration for the given LAYER file." +(define* (config layers-diff-ids time arch #:key entry-point (environment '())) + "Generate a minimal image configuration for the given LAYERS files." ;; "architecture" must be values matching "platform.arch" in the ;; runtime-spec at ;; https://github.com/opencontainers/runtime-spec/blob/v1.0.0-rc2/config.md#platform @@ -125,7 +137,7 @@ (define* (config layer time arch #:key entry-point (environment '())) (container_config . #nil) (os . "linux") (rootfs . ((type . "layers") - (diff_ids . #(,(layer-diff-id layer))))))) + (diff_ids . ,(list->vector layers-diff-ids)))))) (define directive-file ;; Return the file or directory created by a 'evaluate-populate-directive' @@ -136,6 +148,26 @@ (define directive-file (('directory name _ ...) (string-trim name #\/)))) +(define (size-sorted-store-items items max-layers) + "Split list of ITEMS at %MAX-LAYERS and sort by disk usage." + (let* ((items-length (length items)) + (head tail + (split-at + (map (match-lambda ((size . item) item)) + (sort (map (lambda (item) + (cons (file-size item) item)) + items) + (lambda (item1 item2) + (< (match item2 ((size . _) size)) + (match item1 ((size . _) size)))))) + (if (>= items-length max-layers) + (- max-layers 2) + (1- items-length))))) + (list head tail))) + +(define (create-empty-tar file) + (invoke "tar" "-cf" file "--files-from" "/dev/null")) + (define* (build-docker-image image paths prefix #:key (repository "guix") @@ -146,11 +178,13 @@ (define* (build-docker-image image paths prefix entry-point (environment '()) compressor - (creation-time (current-time time-utc))) - "Write to IMAGE a Docker image archive containing the given PATHS. PREFIX -must be a store path that is a prefix of any store paths in PATHS. REPOSITORY -is a descriptive name that will show up in \"REPOSITORY\" column of the output -of \"docker images\". + (creation-time (current-time time-utc)) + max-layers + root-system) + "Write to IMAGE a layerer Docker image archive containing the given PATHS. +PREFIX must be a store path that is a prefix of any store paths in PATHS. +REPOSITORY is a descriptive name that will show up in \"REPOSITORY\" column of +the output of \"docker images\". When DATABASE is true, copy it to /var/guix/db in the image and create /var/guix/gcroots and friends. @@ -172,7 +206,14 @@ (define* (build-docker-image image paths prefix SYSTEM is a GNU triplet (or prefix thereof) of the system the binaries in PATHS are for; it is used to produce metadata in the image. Use COMPRESSOR, a command such as '(\"gzip\" \"-9n\"), to compress IMAGE. Use CREATION-TIME, a -SRFI-19 time-utc object, as the creation time in metadata." +SRFI-19 time-utc object, as the creation time in metadata. + +When MAX-LAYERS is not false build layered image, providing a Docker +image with many of the store paths being on their own layer to improve sharing +between images. + +ROOT-SYSTEM is a directory with a provisioned root file system, which will be +added to image as a layer." (define (sanitize path-fragment) (escape-special-chars ;; GNU tar strips the leading slash off of absolute paths before applying @@ -203,6 +244,59 @@ (define* (build-docker-image image paths prefix (if (eq? '() transformations) '() `("--transform" ,(transformations->expression transformations)))) + (define (seal-layer) + ;; Add 'layer.tar' to 'image.tar' under the right name. Return its hash. + (let* ((file-hash (layer-diff-id "layer.tar")) + (file-name (string-append file-hash "/layer.tar"))) + (mkdir file-hash) + (rename-file "layer.tar" file-name) + (invoke "tar" "-rf" "image.tar" file-name) + (delete-file file-name) + file-hash)) + (define layers-hashes + ;; Generate a tarball that includes container image layers as tarballs, + ;; along with a manifest.json file describing the layer and config file + ;; locations. + (match-lambda + (((head ...) (tail ...) id) + (create-empty-tar "image.tar") + (let* ((head-layers + (map + (lambda (file) + (invoke "tar" "cf" "layer.tar" file) + (seal-layer)) + head)) + (tail-layer + (begin + (create-empty-tar "layer.tar") + (for-each (lambda (file) + (invoke "tar" "-rf" "layer.tar" file)) + tail) + (let* ((file-hash (layer-diff-id "layer.tar")) + (file-name (string-append file-hash "/layer.tar"))) + (mkdir file-hash) + (rename-file "layer.tar" file-name) + (invoke "tar" "-rf" "image.tar" file-name) + (delete-file file-name) + file-hash))) + (customization-layer + (let* ((file-id (string-append id "/layer.tar")) + (file-hash (layer-diff-id file-id)) + (file-name (string-append file-hash "/layer.tar"))) + (mkdir file-hash) + (rename-file file-id file-name) + (invoke "tar" "-rf" "image.tar" file-name) + file-hash)) + (all-layers + (append head-layers (list tail-layer customization-layer)))) + (with-output-to-file "manifest.json" + (lambda () + (scm->json (manifest prefix + (map (cut string-append <> "/layer.tar") + all-layers) + repository)))) + (invoke "tar" "-rf" "image.tar" "manifest.json") + all-layers)))) (let* ((directory "/tmp/docker-image") ;temporary working directory (id (docker-id prefix)) (time (date->string (time-utc->date creation-time) "~4")) @@ -229,26 +323,39 @@ (define* (build-docker-image image paths prefix (with-output-to-file "json" (lambda () (scm->json (image-description id time)))) - ;; Create a directory for the non-store files that need to go into the - ;; archive. - (mkdir "extra") + (if root-system + (let ((directory (getcwd))) + (with-directory-excursion root-system + (apply invoke "tar" + "-cf" (string-append directory "/layer.tar") + `(,@transformation-options + ,@(tar-base-options) + ,@(scandir "." + (lambda (file) + (not (member file '("." ".."))))))))) + (begin + ;; Create a directory for the non-store files that need to go + ;; into the archive. + (mkdir "extra") - (with-directory-excursion "extra" - ;; Create non-store files. - (for-each (cut evaluate-populate-directive <> "./") - extra-files) + (with-directory-excursion "extra" + ;; Create non-store files. + (for-each (cut evaluate-populate-directive <> "./") + extra-files) - (when database - ;; Initialize /var/guix, assuming PREFIX points to a profile. - (install-database-and-gc-roots "." database prefix)) + (when database + ;; Initialize /var/guix, assuming PREFIX points to a + ;; profile. + (install-database-and-gc-roots "." database prefix)) - (apply invoke "tar" "-cf" "../layer.tar" - `(,@transformation-options - ,@(tar-base-options) - ,@paths - ,@(scandir "." - (lambda (file) - (not (member file '("." "..")))))))) + (apply invoke "tar" "-cf" "../layer.tar" + `(,@transformation-options + ,@(tar-base-options) + ,@(if max-layers '() paths) + ,@(scandir "." + (lambda (file) + (not (member file '("." "..")))))))) + (delete-file-recursively "extra"))) ;; It is possible for "/" to show up in the archive, especially when ;; applying transformations. For example, the transformation @@ -261,24 +368,37 @@ (define* (build-docker-image image paths prefix ;; error messages. (with-error-to-port (%make-void-port "w") (lambda () - (system* "tar" "--delete" "/" "-f" "layer.tar"))) - - (delete-file-recursively "extra")) + (system* "tar" "--delete" "/" "-f" "layer.tar")))) (with-output-to-file "config.json" (lambda () - (scm->json (config (string-append id "/layer.tar") - time arch - #:environment environment - #:entry-point entry-point)))) - (with-output-to-file "manifest.json" - (lambda () - (scm->json (manifest prefix id repository)))) - (with-output-to-file "repositories" - (lambda () - (scm->json (repositories prefix id repository))))) - - (apply invoke "tar" "-cf" image "-C" directory - `(,@(tar-base-options #:compressor compressor) - ".")) + (scm->json + (config (if max-layers + (layers-hashes + (append (size-sorted-store-items paths max-layers) + (list id))) + (list (layer-diff-id (string-append id "/layer.tar")))) + time arch + #:environment environment + #:entry-point entry-point)))) + (if max-layers + (begin + (invoke "tar" "-rf" "image.tar" "config.json") + (if compressor + (begin + (apply invoke `(,@compressor "image.tar")) + (copy-file "image.tar.gz" image)) + (copy-file "image.tar" image))) + (begin + (with-output-to-file "manifest.json" + (lambda () + (scm->json (manifest prefix + (list (string-append id "/layer.tar")) + repository)))) + (with-output-to-file "repositories" + (lambda () + (scm->json (repositories prefix id repository)))) + (apply invoke "tar" "-cf" image + `(,@(tar-base-options #:compressor compressor) + "."))))) (delete-file-recursively directory))) -- 2.41.0