From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp0 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms11 with LMTPS id WBGEDbmR0l4VCAAA0tVLHw (envelope-from ) for ; Sat, 30 May 2020 17:02:49 +0000 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp0 with LMTPS id cKptCbmR0l6sYQAA1q6Kng (envelope-from ) for ; Sat, 30 May 2020 17:02:49 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id B026694042C for ; Sat, 30 May 2020 17:02:48 +0000 (UTC) Received: from localhost ([::1]:48642 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jf4sf-0000ab-VT for larch@yhetil.org; Sat, 30 May 2020 13:02:45 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:53140) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jf4sE-0000aN-Pz for help-guix@gnu.org; Sat, 30 May 2020 13:02:18 -0400 Received: from mail-lf1-x135.google.com ([2a00:1450:4864:20::135]:36813) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1jf4sD-0006cW-DN for help-guix@gnu.org; Sat, 30 May 2020 13:02:18 -0400 Received: by mail-lf1-x135.google.com with SMTP id c21so1535049lfb.3 for ; Sat, 30 May 2020 10:02:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=mNuPBH1JvoKL+XKQ2npd+jqcciBRmxVCEBT/i1m5NFY=; b=atq1oYjWDkWvLMfY2XttqqUl+5CW4ToxnpBioa8HqJvJEz8bRbZ1xHAid/1Tcbccfw kII6M8ZRmFjUtjWqRE5taPbyyd/mRKy6z3p6vFkiEmgO6PRdP9ejpYqNfun3/rMpRK71 iwZrrOqUQ5CgzoqOk4iWb86pBbMd5SQLRksESQzyJ+BiEuNHbtUMGiczpd6ELI27A6tA 3YPIf8DiLe6i8hr+g7tE9VaAXztNrETS9a+a0LpwdbOHd8zdx5zYJJbPr6vk5WlaVHeD qY/PQ5JCUIhQcDFPAO1VVoPPn9fmDOaNhyoTi64OyaFMUjipm/wRfPYY/XJUNRDZVNnV N1Eg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=mNuPBH1JvoKL+XKQ2npd+jqcciBRmxVCEBT/i1m5NFY=; b=LiZt+i3GAcd2AUJhuoS4J4gZlDApIIC4C4DPtOjF0UrpwJN5No+1awdHRQ539rKCsM 9nUZSktiDvAPCmDDrSfWg25CkictmdwlHm7jd5VNKfhDmmJldKVab1cDHxCAzPC5rsxN YrBnjsw8HgzmW65ZuSZYnK8Uw0wXrB+nGixQsaohz2flxxSsBCdikVLZA8VnvS8WDVvI 7x2SXVWNkN7j9V07pzfVW3gT8u85eTaP4x7be9YxDYLg7v2MaECst5PkXwmlBwHajNfc YY1CK0n/o4vIT/wFD0TYFjcliFvKhJfRxMqh37OQOMQMTUIl2dSUnnICASY0Nlyd1gRS QVZA== X-Gm-Message-State: AOAM533jD3g226mknXJrlpQWSEPEOWebBJXP0STpykgQUkm59b343I12 Oo+nL74xsj18AIiTCWKeOAy1SgbTVWQ4n4PTZo4= X-Google-Smtp-Source: ABdhPJxff43814NWbq0wOK92MAm1qnlfMw4xaWHaWj8wK9E3P7EHYX/o3x04yddREzznkcNWhzzGOouCvZylbSupLQg= X-Received: by 2002:a19:fc0a:: with SMTP id a10mr7309172lfi.176.1590858134722; Sat, 30 May 2020 10:02:14 -0700 (PDT) MIME-Version: 1.0 References: <87h7vyxqrz.fsf@gmail.com> In-Reply-To: <87h7vyxqrz.fsf@gmail.com> From: Stephen Scheck Date: Sat, 30 May 2020 13:02:02 -0400 Message-ID: Subject: Re: Guix Docker image inflation To: Chris Marusich Received-SPF: pass client-ip=2a00:1450:4864:20::135; envelope-from=singularsyntax@gmail.com; helo=mail-lf1-x135.google.com X-detected-operating-system: by eggs.gnu.org: No matching host in p0f cache. That's all we know. X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_PASS=-0.001 autolearn=_AUTOLEARN X-Spam_action: no action Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: help-guix@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: help-guix Errors-To: help-guix-bounces+larch=yhetil.org@gnu.org Sender: "Help-Guix" X-Scanner: scn0 Authentication-Results: aspmx1.migadu.com; dkim=fail (body hash did not verify) header.d=gmail.com header.s=20161025 header.b=atq1oYjW; dmarc=fail reason="SPF not aligned (relaxed)" header.from=gmail.com (policy=none); spf=pass (aspmx1.migadu.com: domain of help-guix-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=help-guix-bounces@gnu.org X-Spam-Score: 0.09 X-TUID: yJ4bFnAV+gZ1 On Fri, May 29, 2020 at 7:31 PM Chris Marusich wrote: > > Could it be that you are accumulating layers without bound? > > > https://developers.redhat.com/blog/2016/03/09/more-about-docker-images-size/ > > Since Docker images are built up of immutable layers, if you build your > image from an existing base image, I'm not sure that it's possible to > produce a new image that is smaller than the base image. Basically, > even if you run "guix gc" to remove dead store items, they will still > exist on a prior layer, so the size of the new image won't decrease. > And since you're installing new things, the size will actually increase. > If you repeat this process by using the new image as an input for yet > another build, I think you will accumulate layers and storage space > without bound. > Layers certainly add some image size overhead, but I don't think that is the culprit here. And producing a smaller image isn't really the goal, it's just to keep image growth reasonable between each incremental guix pull. Dead store items would only exist on previous layers if they make it there in the first place. As has been demonstrated on previous posts in the thread, I believe the problem is some guix bug which prevents deletion of garbage-collected store items. What is reasonable growth? That is hard to answer, but I would expect it be roughly proportional to the growth of a guix installation over time in a non-Docker environment, taking some constant amount of layer overhead as a given. I don't really know what `guix pull` does, but I think it's something along these lines: 1) the global package index is brought up-to-date; 2) Any packages which are installed in the profile doing the pull are upgraded to newer versions if they've been updated. So day-to-day, particularly in the case where there have been no updates to packages installed in the profile, size growth should be very small. Periodic "rebasing" of incremental Docker images might still be helpful from time to time using one of the layer squashing tools out there, but I don't think it should be necessary on a daily basis. Also, layers are helpful in the case of someone pulling down daily Guix Docker images on a frequent basis, because then only the new, ideally small layers need to be downloaded, whereas if you rebase for every image build, you'd have to download the entire image every day. The boundless layer accumulation you point out shouldn't be a problem with the way that I'm building the images. When you do a `RUN ` inside a Dockerfile, it is essentially doing `docker exec ` followed by `docker commit `. It is the commit step which produces a new layer. You can think of a RUN command inside a Dockerfile as kind of a single-step transaction, which incorporates the net file system changes into the image. My build script issues several `docker exec ` sequences, followed by a `docker commit `. Intermediate changes to the container file system prior to the commit do not generate layers, only the net changes after the commit. You can convince yourself of this by doing something like the following: docker run docker exec dd if=/dev/urandom of=/RANDOM-DATA bs=1048576 count=1024 docker commit docker exec rm /RANDOM-DATA docker commit You'll end up with two new images - the first one should be about 1 GB larger than the base image, the second one the same size. FYI, Guix itself can build Docker images from scratch - no base image > required! It can even build a Docker image of a full-blown Guix System > from scratch. Sorry if you already knew that - I just wanted to point > it out in case you didn't! > Yes, thanks, I know - if you read through the thread you'll see that I make reference to `guix system docker-image [...]`. -SS