From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp12.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms9.migadu.com with LMTPS id ICclL6yRD2T2SgEASxT56A (envelope-from ) for ; Mon, 13 Mar 2023 22:12:12 +0100 Received: from aspmx1.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp12.migadu.com with LMTPS id +MEvL6yRD2QvdQEAauVa8A (envelope-from ) for ; Mon, 13 Mar 2023 22:12:12 +0100 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 785183EEE0 for ; Mon, 13 Mar 2023 22:12:12 +0100 (CET) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pbpSe-0004tJ-QY; Mon, 13 Mar 2023 17:12:04 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pbpSd-0004tA-JA for guix-patches@gnu.org; Mon, 13 Mar 2023 17:12:03 -0400 Received: from debbugs.gnu.org ([209.51.188.43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1pbpSd-0007C0-3G for guix-patches@gnu.org; Mon, 13 Mar 2023 17:12:03 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1pbpSc-0005ri-V8 for guix-patches@gnu.org; Mon, 13 Mar 2023 17:12:02 -0400 X-Loop: help-debbugs@gnu.org Subject: [bug#62153] [PATCH 1/2] guix: docker: Build layered image. Resent-From: Oleg Pykhalov Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Mon, 13 Mar 2023 21:12:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 62153 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Simon Tournier Cc: 62153@debbugs.gnu.org Received: via spool by 62153-submit@debbugs.gnu.org id=B62153.167874186722459 (code B ref 62153); Mon, 13 Mar 2023 21:12:02 +0000 Received: (at 62153) by debbugs.gnu.org; 13 Mar 2023 21:11:07 +0000 Received: from localhost ([127.0.0.1]:35713 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pbpRi-0005q9-PH for submit@debbugs.gnu.org; Mon, 13 Mar 2023 17:11:07 -0400 Received: from mail-lj1-f170.google.com ([209.85.208.170]:39707) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pbpRg-0005pe-Dm for 62153@debbugs.gnu.org; Mon, 13 Mar 2023 17:11:05 -0400 Received: by mail-lj1-f170.google.com with SMTP id b13so14031430ljf.6 for <62153@debbugs.gnu.org>; Mon, 13 Mar 2023 14:11:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; t=1678741858; h=mime-version:user-agent:message-id:in-reply-to:date:references :subject:cc:to:from:from:to:cc:subject:date:message-id:reply-to; bh=4c7j0i2EFC6jpMuSPWTkkV+r7WutTPeyJ8Svn9EyFio=; b=STvNpsN6Yhr0ULBDdTC+iHNoUuV5FOUPZtwc2Fx7O4e7TGTlHorknzlL4newlTkcS8 3JmcGmB6ouxAYS9QMsmpCOktFsrIftjr+DV+Y6K0w8Nkj1TSp0h98bCbwmxvIpzEL2Qf +sBEaM1kjc8d64kMLpIoRbwOOhWkeeCzqbLz5D6Yr8CHuKglygffFrfUFmFpXCtsmNer ex3+3MtZh3aF9KIrV+cp8xuMLZ8oVKOBoh5yXJWuODA0oadFrNnJzfkpAPA9Q53rtwYD M7YtiMq5nmSdwYMnKkRaJscuH7zJXh6Gvxu9afIy8JSMPl/eASxFP6mYp8G7w73Qdkui GGiw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1678741858; h=mime-version:user-agent:message-id:in-reply-to:date:references :subject:cc:to:from:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=4c7j0i2EFC6jpMuSPWTkkV+r7WutTPeyJ8Svn9EyFio=; b=ngULmHTPCxOBua/MlaeIhsDCLL+xg9TWA/rAGoK9u9shRuVoqspqgrny76mqPc2PMJ HAHlY4stnHQ9uLPKkQs3z9Ix+eQQvG7CBZxHKWBGHDV9SY8R1eBRkSlccdgXLDBZQWto Jdf1Vmr6CsOkz5nFW783a1LJb9E+CvTIf6n5DR4833J2OB6xCoDwWE5Hn7GgmNv8RznO hiLlzu0s8Oj2Z4h0daByLYJMKP6xlHxiN5ItlhIqlPvKYd4amHEmvikFxYHuE+x/8Sqx MkunPo075URMFiHwjPfr1AyZGuPo1X9RNh6yZt85jclgIfQaGDRTTwO2JN8xF3NdDVtm RZYg== X-Gm-Message-State: AO0yUKX24thZo3nei4yph+RXZjl+5JI4KHWwhIiyldpBjdBzE2ybCyWk SnWqv4ECmKPwBnM97fO/GmE= X-Google-Smtp-Source: AK7set9di6UqWrIYphHLNYnfDooguPpk4joROlhw8Ztq8X7KGT+rLkImLSrluTHlajhViTf7ehqgow== X-Received: by 2002:a05:651c:1a2c:b0:295:a08c:12a1 with SMTP id by44-20020a05651c1a2c00b00295a08c12a1mr5160313ljb.0.1678741858001; Mon, 13 Mar 2023 14:10:58 -0700 (PDT) Received: from localhost ([88.201.161.72]) by smtp.gmail.com with ESMTPSA id t6-20020a2e9c46000000b00297dad1a2b0sm143309ljj.103.2023.03.13.14.10.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 13 Mar 2023 14:10:57 -0700 (PDT) From: Oleg Pykhalov References: <20230313003012.14325-1-go.wigust@gmail.com> <20230313003310.17129-1-go.wigust@gmail.com> <87r0tsk85r.fsf@gmail.com> Date: Tue, 14 Mar 2023 00:10:56 +0300 In-Reply-To: <87r0tsk85r.fsf@gmail.com> (Simon Tournier's message of "Mon, 13 Mar 2023 16:01:04 +0100") Message-ID: <87bkkw2w7z.fsf@gmail.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.2 (gnu/linux) MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha512; protocol="application/pgp-signature" X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+larch=yhetil.org@gnu.org Sender: guix-patches-bounces+larch=yhetil.org@gnu.org X-Migadu-Country: US X-Migadu-Flow: FLOW_IN ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=gmail.com header.s=20210112 header.b=STvNpsN6; spf=pass (aspmx1.migadu.com: domain of "guix-patches-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-patches-bounces+larch=yhetil.org@gnu.org"; dmarc=fail reason="SPF not aligned (relaxed)" header.from=gmail.com (policy=none) ARC-Seal: i=1; s=key1; d=yhetil.org; t=1678741932; a=rsa-sha256; cv=none; b=hQZGNqtkhmWwH/QvWPN2yveJuR9ujMf2LD7h9Y1BxilDS0rIbPoMYiY0U7IPGl83cSkAYp /GOaUV2Wx8cSZwGa4Vxd/Aoo8DobmnTpHpZXDo2n9NiiLA5iRRBsifl/A9nXRacQvDFXIx Gxd/DNyuLHLYxEs+Ua56hAxxUFWePY+sAQbKVQK4Rg7vwAxEF0GDLraiA9XeZ90Y9dHC6A kfvewQ+m3OCytcolKTYl1B+TCq9bIIAgiB6KN2lIkRS31wfHbxHlBqIcgQGfeCbWTGT4DZ /8RmNOpo9yh5/HcT9Li+RXgeqV/CrNEwxg5oxv1VIwUX20gYyJ27PwOYv3mkgg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1678741932; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:resent-cc:resent-from:resent-sender: resent-message-id:in-reply-to:in-reply-to:references:references: list-id:list-help:list-unsubscribe:list-subscribe:list-post: dkim-signature; bh=4c7j0i2EFC6jpMuSPWTkkV+r7WutTPeyJ8Svn9EyFio=; b=r9iQ2ZSBm7rbE2fnlHePdbXLgCf7UBS44kRR5Tm/QezUyVzXBQFb+4ethsEX1ascEH+OPF 0hRBNW8naT0/t7O//5Pp8Tg/DJ4yGFNWqk3TJvmtWxgXwdm0XNtWrQ1yccZ+SuyYDbcN2j l4Yr4IkukOwAqvEebmfGwAryszOvknOtQGnGAsFIk6AMGoXBo+mxkcDdoHSENM+f82Y1WX OT0H8gjaERyuNj+n2Gjh0YPhMnJ90pd4OPdhHOZskhBtyd1TKrB7H8LEUFogHszeQ35lYu OqYLV+U215RTbeFXfk2sQ7haz+bwkXCmYHcm0iNAa9qx6jghPH1Jepgl+KAd/g== X-Migadu-Spam-Score: -4.22 X-Spam-Score: -4.22 X-Migadu-Queue-Id: 785183EEE0 X-Migadu-Scanner: scn1.migadu.com Authentication-Results: aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=gmail.com header.s=20210112 header.b=STvNpsN6; spf=pass (aspmx1.migadu.com: domain of "guix-patches-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-patches-bounces+larch=yhetil.org@gnu.org"; dmarc=fail reason="SPF not aligned (relaxed)" header.from=gmail.com (policy=none) X-TUID: uQ485+iWzGbt --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Hi Simon, Thank you for the review. Simon Tournier writes: > On lun., 13 mars 2023 at 03:33, Oleg Pykhalov wrote: > >> diff --git a/gnu/packages/aux-files/python/stream-layered-image.py b/gnu= /packages/aux-files/python/stream-layered-image.py >> new file mode 100644 >> index 0000000000..9ad2168c2d >> --- /dev/null >> +++ b/gnu/packages/aux-files/python/stream-layered-image.py >> @@ -0,0 +1,391 @@ >> +""" >> +This script generates a Docker image from a set of store paths. Uses >> +Docker Image Specification v1.2 as reference [1]. > > Instead of Python, would it possible to implement in Guile? I mean, > does Python have something that is missing in Guile? > > The facility for manipulating Tar? Something else? I think nothing else. As I understand Python implemented Tar inside the language itself in 2500 lines of code by manipulating binary data. /gnu/store/...-python-3.9.9/lib/python3.9/tarfile.py Technically it's probably possible to use tar utility with --append flag instead of opening a new file and streaming to it as the Python script does. To be honest I would like not to write it in this way if the Python script does not block current patch for merge. Also I don't see myself writing Tar implementation in Guile, yet. ;-) The Nix project uses this script heavily to build layered images, so it should be robust in terms of up to date to current Tar and Python implementations. > Because then, if I understand correctly=E2=80=A6 > >> diff --git a/guix/docker.scm b/guix/docker.scm >> index 5e6460f43f..f1adad26dc 100644 >> --- a/guix/docker.scm >> +++ b/guix/docker.scm > > [...] > >> + (if stream-layered-image >> + (let ((input (open-pipe* OPEN_READ "python3" >> + stream-layered-image >> + "config.json"))) > > =E2=80=A6it requires to drag Python for building/packing layered Docker. Correct. > Well, I have not really look yet to the Python script which does most of > the job. Do you use a similar strategy as [1]? > > And I remember something in that direction by Chris but I am unable to > find back the patch. )-: > > 1: https://grahamc.com/blog/nix-and-layered-docker-images/ Not similar. My patch implements a very simple sorting by size, no complex sorting by reference popularity as in [1], which is probably implemented in the following file github.com/NixOS/nixpkgs/pkgs/build-support/references-by-popularity/clo= sure-graph.py From=20https://grahamc.com/blog/nix-and-layered-docker-images/ article: > How Docker really represents an Image > > Docker=E2=80=99s layers are content addressable and aren=E2=80=99t requir= ed to > explicitly reference a parent layer. This means a layer for > readline-7.0p5 doesn=E2=80=99t have to mention that it has any relationsh= ip to > ncurses-6.1 or glibc-2.27 at all. > > Instead each image has a manifest which defines the order: > > { > "Layers": [ > "bash-interactive-4.4-p23", > "bash-4.4p23", > "readline-7.0p5", > ... > ] > } > > If you have only built Docker images using a Dockerfile, then you > would expect the way we flatten our graph to be critically > important. If we sometimes picked readline-7.0p5 to come first and > other times picked bash-4.4p23 then we may never make cache hits. > > However since the Image defines the order, we don=E2=80=99t have to solve= this > impossible problem: we can order the layers in any way we want and the > layer cache will always hit. In case of sorting by size, bigest layers will be on top of a container image, which will produce a cache hit for bigest directories in the GNU store during images transfer with same layers. I would like to say this sorting could binifit more than sorting by popularity during transfer but let's assume I didn't write it. ;-) The following example shows common layers between images, which will be not tranfered if you load image inside Docker as well as pull and push: ./pre-inst-env guix pack -f docker-layered --entry-point=3Dbin/bash -S = /bin=3Dbin bash hello and ./pre-inst-env guix pack -f docker-layered --entry-point=3Dbin/bash -S = /bin=3Dbin bash hello emacs share 6 layers in total =2D-8<---------------cut here---------------start------------->8--- $ f() { docker image inspect "$1" | jq --raw-output '.[0].RootFS.Layers[] |= .' | sort ; } $ comm -1 -2 --total <(f sha256:fb43b32380a5e6a867410721f4ce2917db14d4ae943= c433983afbaf84416c421) <(f sha256:0ce4a11973d1071aeec5441db228d6148dfd09fea= 3ae77b731c750ebfcc2fe1d) sha256:3b3daa2a00f1acd12eeb16698bf1caeb6ba6c436e3dbca6259c3a9c622664e00 sha256:5c2be7469293854257221cb6aa8aa4af1e10e2c550935390dbcfeede3d3fbacd sha256:60317981d94928659389f299e4b86703e5ded420a53537d67627952187fbd3f9 sha256:6d7c8ce5441d4c4c74e0ecff6c203a7b265b37137cca3b0a0ccf10526cfaa6e2 sha256:c2ded2ffe3f46fa7a64a62e0fc6b9d28cb7d4f8d9c64d5a52d137a508cba11fc sha256:fbcad85d7d3c25bd2aa6d95bb3bf3d02c499ee3b3e443ddd3e5b679c2b33c139 5 94 6 total =2D-8<---------------cut here---------------end--------------->8--- Regards, Oleg. --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQJIBAEBCgAyFiEEcjhxI46s62NFSFhXFn+OpQAa+pwFAmQPkWAUHGdvLndpZ3Vz dEBnbWFpbC5jb20ACgkQFn+OpQAa+pzFZBAApU6Atd2czoZgSRir9SG0/V/l7yva zKst6JvXL77QxVU+e0QqVcY/o8rJg9hIGH+cQaOmlxmbYXJExEI1Go+tngk61OgQ dsMrdBBPkaXvqzd7PKQRUtCn221CGfoMMfZGEzCUyQFTYgo6+K7s76Ep6++lODEW nV/nPa/gqKhg6IZ2G+NeuNKeHsFN8YH/U3Si8myvyLSt0B/ZZEb/8eMumhl3lt61 4GXPPhlFXvp/8VebhMbvzN7TUGWl4z8uMpVExYdDjG3BxkcJBqlEfGfJPQlWc18E +8G93/JCFYhtE/ae0d/qCpVs8k7CLRrJEkllQFARau+e0GxNggdA08mACmuxZqTF u1t52FVvg+cgIO5XKN0x0HyR0t+Rv6gkZPA1b4EXF6t8XRCkdh5VXZr/v3Wvwea2 2wbTeX2vDeca3coyCxDNRjka+FgcLrcxxVBZNc33/76sgglXjovQI4g4LDsklkV9 pSa8StEt4/lCSPXRUQYy0g/nUO4Lk2QaIVx7FSefqCaS+EwCz6NkMGi8esKrtHuw l1gFvYhMFtHvQjzbO50edR1QyLgbMTFbyJdFhHJOEI0rAeh4AIde+R/FfXbPFZR4 RCPwCij2+Ng0O8lrd9uQcdo92oms6FGHLN+ndgYm38hzMQdI+ecMpAQtlR09efsW WBhi5/HWNf3372o= =r2xS -----END PGP SIGNATURE----- --=-=-=--