From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp0 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms11 with LMTPS id wB8+KE8z017nBQAA0tVLHw (envelope-from ) for ; Sun, 31 May 2020 04:32:15 +0000 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp0 with LMTPS id YI9GJE8z014DNQAA1q6Kng (envelope-from ) for ; Sun, 31 May 2020 04:32:15 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 0EED39403E9 for ; Sun, 31 May 2020 04:32:14 +0000 (UTC) Received: from localhost ([::1]:34734 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jfFds-0000P2-Et for larch@yhetil.org; Sun, 31 May 2020 00:32:12 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:46372) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jfFdi-0000Mj-A5 for help-guix@gnu.org; Sun, 31 May 2020 00:32:02 -0400 Received: from mail-pg1-x530.google.com ([2607:f8b0:4864:20::530]:35824) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1jfFdg-0006ip-FH for help-guix@gnu.org; Sun, 31 May 2020 00:32:01 -0400 Received: by mail-pg1-x530.google.com with SMTP id o6so1967048pgh.2 for ; Sat, 30 May 2020 21:31:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:references:date:in-reply-to:message-id :user-agent:mime-version; bh=PaBiLm1Z0LTMAY1TyeM3CmgNr8qrfRSNXBgP+I4eCzE=; b=YoAqCs/VPeqx/Z7HHexGAMs9sJfz7qT3wOG0aKe7dw/qrok1PtffuGLTAh4LgWulWq rhRtam59OYHKC1Jw7TM+Tvffxo4HoOYRGBaAPtBPK7jkRbLzxMIK9nCJ+2t+Wb0llly4 UE41wpgyfRNDfOZFHc/FNzBHdjidaaMzzw6LzUwDv60FXxl/6EvlMBGc/STUV5rONGZl KO7nHf6UI/D3UGLMSGrfN2brM2ddUatjxux+2wx1dne2qQ70DbcPuJc8XTNV3Xt/rRDT btH9LtqIc936t0STXslm/uihWw2C3HuUafQHBPdNKWYsoZHSwqqK+qWMSBxYbs2jt7hQ /lyQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:references:date:in-reply-to :message-id:user-agent:mime-version; bh=PaBiLm1Z0LTMAY1TyeM3CmgNr8qrfRSNXBgP+I4eCzE=; b=Gs6SWzE07mfx61LWYj0K8uuPMqTUb6gacms81H1EmlVIkCF1azwY+HjcH/aczPH/5J s3pi7nMQwiAMVAD8U5OCOS52FwYM7mXF4jQKzqgzW/QxCngpsW5Kbqtm2DRA2teTYUCL Em4COJ9BHnexmR5P4GlJ1SErIjPCcMht78cTJGo0Pu+wL8H3Ft2ig1J1wKIcu2Y+1G6d AcunlEhAA4VgHU7n+0WHkeKiGxT2VUR5v9/jX5iCENaJVvqzUA9OZuEQU9Vmf8++Igqu 7qkcQAI4+BAs8KFTThKwfp7/DRV+BqsafzCsJXu/NMWEy4BN7PpBIFdlGf3tLot9yhSC mC4Q== X-Gm-Message-State: AOAM5316rJoYg0J9PJA7i6F5aQh5q0/xNVm9W5pxHkgx8irlHI8OR63v /n9cqDx1rpixSErtTnAwnTemySpI4tY= X-Google-Smtp-Source: ABdhPJzYT2p3cjNR0OBcXSJATVbUjSvAv1uLZdpg4wZ0VdJC3wQNjc8M+HTx/CXnOPD6awonZc5ZKA== X-Received: by 2002:aa7:85da:: with SMTP id z26mr14653086pfn.25.1590899517026; Sat, 30 May 2020 21:31:57 -0700 (PDT) Received: from garuda-lan (c-73-97-103-127.hsd1.wa.comcast.net. [73.97.103.127]) by smtp.gmail.com with ESMTPSA id go1sm3391230pjb.26.2020.05.30.21.31.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 30 May 2020 21:31:55 -0700 (PDT) From: Chris Marusich To: Stephen Scheck Subject: Re: Guix Docker image inflation References: <87h7vyxqrz.fsf@gmail.com> Date: Sat, 30 May 2020 21:31:51 -0700 In-Reply-To: (Stephen Scheck's message of "Sat, 30 May 2020 13:02:02 -0400") Message-ID: <87367glo7c.fsf@gmail.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.3 (gnu/linux) MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha256; protocol="application/pgp-signature" Received-SPF: pass client-ip=2607:f8b0:4864:20::530; envelope-from=cmmarusich@gmail.com; helo=mail-pg1-x530.google.com X-detected-operating-system: by eggs.gnu.org: No matching host in p0f cache. That's all we know. X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_PASS=-0.001 autolearn=_AUTOLEARN X-Spam_action: no action X-BeenThere: help-guix@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: help-guix Errors-To: help-guix-bounces+larch=yhetil.org@gnu.org Sender: "Help-Guix" X-Scanner: scn0 Authentication-Results: aspmx1.migadu.com; dkim=pass header.d=gmail.com header.s=20161025 header.b=YoAqCs/V; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (aspmx1.migadu.com: domain of help-guix-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=help-guix-bounces@gnu.org X-Spam-Score: -1.31 X-TUID: L2uVrhY0IizZ --=-=-= Content-Type: text/plain Content-Transfer-Encoding: quoted-printable Hi Stephen, Stephen Scheck writes: > Layers certainly add some image size overhead, but I don't think that > is the culprit here. > Also, layers are helpful in the case of someone pulling down daily > Guix Docker images on a frequent basis, because then only the new, > ideally small layers need to be downloaded, whereas if you rebase for > every image build, you'd have to download the entire image every day. That is true, but suppose I have the following 3 images: =2D Image A: A base image created in January 2020. =2D Image B: Based on A, and I ran "guix pull" in February 2020. =2D Image C: Based on A, and I ran "guix pull" in June 2020. I would guess that the size difference between A and B is approximately the same as the difference between A and C. It'll be different, of course, but generally the size difference between A and C should not grow linearly with time, since "guix pull" is only going to install at most the total closure of things necessary to build and run Guix, which doesn't increase much in size as time goes on. However, when you daisy-chain the images every day, the image size will grow linearly with time because the contents of all the previous layers is carried forward. > My build script issues several `docker exec ` > sequences, followed by a `docker commit `. Intermediate > changes to the container file system prior to the commit do not > generate layers, only the net changes after the commit. There are two problems here. One is that the image size grows without bound. The other is that guix-daemon is failing to GC store items in the Docker container. Although they are both concerning, the latter is not the cause of the former. If you install new store items (e.g., via "guix pull"), make them dead, and then GC them, all in the same container before running "docker commit", then I agree: those GC'd store items would not persist in a layer anywhere. However, I don't think that's what's happening here. Sure, there might be a few store items like this, but in practice, there will be many store items from the previous image which began live but became dead when you ran "guix pull" and deleted your old profile generations. It is those store items that are adding the most space to your image. Besides store items, I noticed two other things about your images: =2D The contents of /var is growing slowly without bound, but it isn't nearly as bad as the contents of /gnu/store. This is probably due to log files; consider pruning them. =2D Your script runs "docker commit" while guix-daemon (and other programs) are still running. To ensure the guix-daemon's database (or other things) does not become corrupt, consider terminating all processes before committing the new image. > FYI, Guix itself can build Docker images from scratch - no base image >> required! It can even build a Docker image of a full-blown Guix System >> from scratch. Sorry if you already knew that - I just wanted to point >> it out in case you didn't! >> > > Yes, thanks, I know - if you read through the thread you'll see that I ma= ke > reference to `guix system docker-image [...]`. I apologize for not reading your thread more closely to begin with. I took a closer looks, and I think I can explain what is going on now. Please check the bug report and reply there if anything is unclear. =2D-=20 Chris --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEEy/WXVcvn5+/vGD+x3UCaFdgiRp0FAl7TMzcACgkQ3UCaFdgi Rp1joBAAqOI/LuAvrMbkxsqSL/+ghIuAbUNh9a10mPx2cuEJMkyTADrjMO6l5k8n CZ9ZMwu2mqmx6oc/5qQY72M7sQCrM1ZQkH40UcnfYIK0OJQUJK6dpBn7hnicZtXG HOmbAVoc6fkjaZF50uTnY8/pHywuBV4KayhgSNb6KyHayCIu+AUbgInl+Tm6FQqA QiTKWiWObIgfauCT2Cg65MKh8TmKWxYgFkLF0zezblXYAHJnEB67SmhnIee+dD/Y YuhvpGbTI6M3Yyk48gJnd7soFZf6pp7+iIgeZNcm67iJhhRfasGyGF/VLmnjk765 xTeEJCeon4ujNWXmd4d0cMsRbWyAccpWc3ryPsIl/JgBUOZw5bDMLRU5SSII+F9Q yL4GKGimQ1dDiq7tg5rIiRPfME9DFa3r9xdMJiWChWZDDpYP3leO2jwiCSGswAkF VimP1+5rlalUCstQQeR9EefMjFHJZDNKhFdltOveNTkR8Ur3+Sa6RpEPycHQDlau AhfgpUrNj6fkVzTFR0zklVUfv5itUBqHewfmx67FM9CCpvMlYJEXKcUUVVMxuluU 5GvbELF62hsIlTtB2ZNrwv70wSe+tC2NqDF1KzvQgQbzieXdu+UHHS7BSz63cHxk jbZD8pfZtkrqooAY/dMWC8l5UC2Zb42i/7Mn5FhMCsE8w+s+ZL4= =su7Z -----END PGP SIGNATURE----- --=-=-=--