From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp0 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms11 with LMTPS id wNd6C7b8014jCQAA0tVLHw (envelope-from ) for ; Sun, 31 May 2020 18:51:34 +0000 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp0 with LMTPS id ELZQB7b8017NdAAA1q6Kng (envelope-from ) for ; Sun, 31 May 2020 18:51:34 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id CEDA9940145 for ; Sun, 31 May 2020 18:51:33 +0000 (UTC) Received: from localhost ([::1]:59632 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jfT3U-0002RA-QH for larch@yhetil.org; Sun, 31 May 2020 14:51:32 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:42692) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jfT3M-0002Qk-Fs for help-guix@gnu.org; Sun, 31 May 2020 14:51:24 -0400 Received: from mail-qv1-xf2a.google.com ([2607:f8b0:4864:20::f2a]:46592) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1jfT3L-0004GP-Ha for help-guix@gnu.org; Sun, 31 May 2020 14:51:24 -0400 Received: by mail-qv1-xf2a.google.com with SMTP id cv17so61871qvb.13 for ; Sun, 31 May 2020 11:51:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=YpAzTGoXTQ1DoG+DoaPFYhuXuSeoCa+eU9LBuyH4qjE=; b=vgnyYnDLAWAZ4umcSaBoMpHXCPr76rOTc6taVv5MgVkTbU1+b4Avq19qzkJvKYdjxD QCgMkS4OrXW0VLX/DS0a3u3Q4Cnypc7tHzRrax+EaQ8Xdg7MM3eujKP9Lujdw/tJCkN6 f5PR6a+m/u8mcjnyVnejN8fz1nfSTEKUCZFRK1PBF7FW3r8VF1UC2N7G6+j5icoYvd9e dyDvuQPh0BNWdA7N1fGVMgG6nQ76fpVay4f1o6P6s0Qv9oovq1p5SzfMEs/t/uTTlTuG tiSDU9lQlfBDkKSwFgl5wwcgoQ9UAKM0jXNH92p6ze+y+h2T4MFuApVK392h3iGuK2Vn mq9A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=YpAzTGoXTQ1DoG+DoaPFYhuXuSeoCa+eU9LBuyH4qjE=; b=hW/Jb6uP9cSRzjkKi4k52YdGLkkIrBlvU/FVvRLO7bLwYEegv+YY96QCA8e1TwF5Kw ONtSh74ipd8rUx8WkEXjBZRL57bfnt2C/yTtc8pUeItTcq7Za4kCvP9pIADbUZpsv0Fk l/E5aWEhxGMdRmEdX5IOghwoLPpFrw8JHuc+KEalLGMbKH67b3JQ/A3CLI+GW4zQg+4A EpwBEur1B0EP8BuzgitXUyeps4TfYfeLWIvU+oU1EiWAQkQisXu8o0OwscLKTXytaXLd 0EKycc7/jmZ9gZJJeju0sBW5b7nIOLMHM2+0yKyKRH4ndJe1RiIBVgZwYgXyX9AMr5zx 6Flw== X-Gm-Message-State: AOAM531vIvkN1XzNPzkjMn2maZQTQGpBe2yuDTiddeWOPFZgPHTzo86H xvsqO5AogR3acY+HlpPdKorBY89O1/vfj6n3hyU= X-Google-Smtp-Source: ABdhPJw7NWON9x1amUTyjir0dp1lXKhnTrCoOoXkVLeTATkqO3zMzv494emnHrAUlfVjYRxPG4lb8MpJNkKHL8VFUho= X-Received: by 2002:ad4:54aa:: with SMTP id r10mr10621644qvy.6.1590951082410; Sun, 31 May 2020 11:51:22 -0700 (PDT) MIME-Version: 1.0 References: <87h7vyxqrz.fsf@gmail.com> In-Reply-To: From: zimoun Date: Sun, 31 May 2020 20:51:10 +0200 Message-ID: Subject: Re: Guix Docker image inflation To: Stephen Scheck Content-Type: text/plain; charset="UTF-8" Received-SPF: pass client-ip=2607:f8b0:4864:20::f2a; envelope-from=zimon.toutoune@gmail.com; helo=mail-qv1-xf2a.google.com X-detected-operating-system: by eggs.gnu.org: No matching host in p0f cache. That's all we know. X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_PASS=-0.001 autolearn=_AUTOLEARN X-Spam_action: no action X-BeenThere: help-guix@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: help-guix Errors-To: help-guix-bounces+larch=yhetil.org@gnu.org Sender: "Help-Guix" X-Scanner: scn0 Authentication-Results: aspmx1.migadu.com; dkim=fail (rsa verify failed) header.d=gmail.com header.s=20161025 header.b=vgnyYnDL; dmarc=fail reason="SPF not aligned (relaxed)" header.from=gmail.com (policy=none); spf=pass (aspmx1.migadu.com: domain of help-guix-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=help-guix-bounces@gnu.org X-Spam-Score: 1.59 X-TUID: yF9cDEjeyjMR Dear Stephen, again :-) On Sun, 31 May 2020 at 20:30, Stephen Scheck wrote: >> No, it is how Docker is designed. Maybe the terminology "layer" is >> not the Docker one but when the images are chained, one cannot remove >> the data of the previous layer of the total image. > > I'm not disagreeing with that, but IF any of the store files resulting from `guix pull` > are ephemeral (i.e. intermediate build results not anchored to a profile) AND guix > GC worked inside the container, my approach might still work - yes there would be > image and layers growth but it might be small enough not to care between periodic image > rebases. But I'm starting to doubt that, or at least it is difficult to quantify with the > GC issues. Currently, if I read correctly, your images are chained with something like, --8<---------------cut here---------------start------------->8--- GUIX_PATH=/root/.config/guix/current/bin $GUIX_PATH/guix pull --branch=$CI_COMMIT_REF_NAME--fallback /root/.config/guix/current/bin/guix gc --delete-generations /root/.config/guix/current/bin/guix gc --collect-garbage /root/.config/guix/current/bin/guix gc --optimize docker commit --8<---------------cut here---------------end--------------->8--- and instead you should do something like --8<---------------cut here---------------start------------->8--- GUIX_PATH=/root/.config/guix/current/bin $GUIX_PATH/guix pull --branch=$CI_COMMIT_REF_NAME--fallback /root/.config/guix/current/bin/guix pull -d /root/.config/guix/current/bin/guix package -d /root/.config/guix/current/bin/guix gc docker commit docker export | docker import --8<---------------cut here---------------end--------------->8--- Maybe the explosion of size would be slower. If you do, please report here the number after say 12 generations; I am really interesting. ;-) >> Because if you run Guix outside an Docker container, you will not have >> the issue. The main issue is how the Docker "filesystem" is designed. > > Actually, there might be another way around this, still avoiding the need for a custom Runner, > for example mounting /var/guix and /gnu/store into the container instead of belonging to it. If > done that way, layer accumulation wouldn't be an issue, and maybe GC between layers neither. Yes, it is one solution. All the question seems to be: - what is the purpose of such Docker image? Which usage? - what infrastructure do you have at hand to build the Docker images? Thank you for raising all this Docker image production question. :-) All the best, simon