From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp2 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms11 with LMTPS id oGieOic5016LPQAA0tVLHw (envelope-from ) for ; Sun, 31 May 2020 04:57:11 +0000 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp2 with LMTPS id N711Nic5016ZIAAAB5/wlQ (envelope-from ) for ; Sun, 31 May 2020 04:57:11 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 64DFF94030E for ; Sun, 31 May 2020 04:57:11 +0000 (UTC) Received: from localhost ([::1]:39486 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jfG20-0005aI-TT for larch@yhetil.org; Sun, 31 May 2020 00:57:08 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:48814) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jfG1u-0005Uu-1C for bug-guix@gnu.org; Sun, 31 May 2020 00:57:02 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:48004) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1jfG1t-00048N-Ok for bug-guix@gnu.org; Sun, 31 May 2020 00:57:01 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1jfG1t-00057k-NL for bug-guix@gnu.org; Sun, 31 May 2020 00:57:01 -0400 X-Loop: help-debbugs@gnu.org Subject: bug#41607: Deleted store items are not actually deleted Resent-From: Chris Marusich Original-Sender: "Debbugs-submit" Resent-CC: bug-guix@gnu.org Resent-Date: Sun, 31 May 2020 04:57:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 41607 X-GNU-PR-Package: guix X-GNU-PR-Keywords: To: Leo Famulari Received: via spool by 41607-submit@debbugs.gnu.org id=B41607.159090101219681 (code B ref 41607); Sun, 31 May 2020 04:57:01 +0000 Received: (at 41607) by debbugs.gnu.org; 31 May 2020 04:56:52 +0000 Received: from localhost ([127.0.0.1]:59550 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jfG1k-00057N-El for submit@debbugs.gnu.org; Sun, 31 May 2020 00:56:52 -0400 Received: from mail-pf1-f171.google.com ([209.85.210.171]:32993) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jfG1h-000578-RA for 41607@debbugs.gnu.org; Sun, 31 May 2020 00:56:51 -0400 Received: by mail-pf1-f171.google.com with SMTP id n15so1911790pfd.0 for <41607@debbugs.gnu.org>; Sat, 30 May 2020 21:56:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:in-reply-to:references:user-agent:date :message-id:mime-version; bh=FXyPjJoiOr86lhWv0bOtjpLFSsZvqTgS4SMhzSnsYhY=; b=tOSxs12WJnPKgUIc4WmDjI1zeVrLVG6EbHy0PhGTtIsGwxmLYkKE9LSy8ta4CvL+uQ 2yZALjoLoeByk94DFFkXQJsZ3aLIjOH6RaJsA2VrijoPJ1QZza9X60kz7ETJ65CKhEgs jtYHS2w9G38AN0dhY6550TpycOCqgvi6m5RYY35S70BntBWkKIj/U0lwcMbMTFSQIhMG QqRPWoDHbg52njePzD2g38CfckGpQEfs2grYQ1f1ftuUcaDw1Hrg/wxMtW7C2btB6oQz Ox1wcYcfe0LQZxNums2RLUYtNKapyE+EVPQpF3Uw4y5CdSBe4C/LVafVmuxoNviDV4GL oZ6A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:in-reply-to:references :user-agent:date:message-id:mime-version; bh=FXyPjJoiOr86lhWv0bOtjpLFSsZvqTgS4SMhzSnsYhY=; b=fFKMG+BaFqHfgEX+v7JVfePhuLV1QFmaCKa8fsiPkvZbgDkLfMJcoRWYvhhpOJ2RBr HEGFuco0FKRGVhHoWJ/cY2ft0NwiATICvovmgmQN0ncXf8x2lUYaeKK+I3r8kZxtzFpO 6ITbb1iVH/ALCvEehy1+8taYdzQfECAqKv+PhnUQ8iAWiQryofsCq91u30lH7wyDKgRL rgWU+9ebFs3U+tDDdmArkHxWuZKKzkmaQZ+UJxCibwuQr3EYB1bhoOXGxLx7I9+JbE5e UzrZ6rtnnjsTXghPeJ7b5tL8w7A3KngXfNiSib3xHQjT/BA6XRiLrmyplxiEubMtgsRs yotA== X-Gm-Message-State: AOAM531v1XMTDFtngYznybxjzELtk1P5mBlqQTeyYMRfpiKJqCRwyARW H1C/Chtmg/6jZDuLRWjwLwufwuLuybs= X-Google-Smtp-Source: ABdhPJyuRwXbinv9rbZwcP5pr3OTutOV1q3674IBoiBJwIaVweIRRGBZDEtGql247o1syAd0xR/BHg== X-Received: by 2002:a62:1885:: with SMTP id 127mr15352586pfy.258.1590901003201; Sat, 30 May 2020 21:56:43 -0700 (PDT) Received: from garuda-lan (c-73-97-103-127.hsd1.wa.comcast.net. [73.97.103.127]) by smtp.gmail.com with ESMTPSA id p190sm11072267pfp.207.2020.05.30.21.56.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 30 May 2020 21:56:42 -0700 (PDT) From: Chris Marusich In-Reply-To: <20200529190942.GA8440@jasmine.lan> (Leo Famulari's message of "Fri, 29 May 2020 15:09:42 -0400") References: <20200528181043.GC23745@jasmine.lan> <20200529170820.GA30828@jasmine.lan> <20200529180245.GA3754@jasmine.lan> <20200529190942.GA8440@jasmine.lan> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.3 (gnu/linux) Date: Sat, 30 May 2020 21:56:38 -0700 Message-ID: <87r1v0k8hl.fsf@gmail.com> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha256; protocol="application/pgp-signature" X-Spam-Score: 0.0 (/) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-Spam-Score: -1.0 (-) X-BeenThere: bug-guix@gnu.org List-Id: Bug reports for GNU Guix List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: 41607@debbugs.gnu.org, Stephen Scheck Errors-To: bug-guix-bounces+larch=yhetil.org@gnu.org Sender: "bug-Guix" X-Scanner: scn0 Authentication-Results: aspmx1.migadu.com; dkim=fail (rsa verify failed) header.d=gmail.com header.s=20161025 header.b=tOSxs12W; dmarc=fail reason="SPF not aligned (relaxed)" header.from=gmail.com (policy=none); spf=pass (aspmx1.migadu.com: domain of bug-guix-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=bug-guix-bounces@gnu.org X-Spam-Score: 0.49 X-TUID: CxbY569E+4vX --=-=-= Content-Type: text/plain Content-Transfer-Encoding: quoted-printable Hi Leo, Stephen, and others, I originally wrote a very detailed email describing my investigation of this issue and the results. However, I accidentally deleted it, so please bear with me as I write a rough summary instead. Leo Famulari writes: >>>From the discussion "Guix Docker image inflation" [0] on help-guix: > > On Fri, May 29, 2020 at 02:29:53PM -0400, Stephen Scheck wrote: >> # Now try to delete it... >> root@localhost /gnu/store# guix gc --delete >> /gnu/store/x7ns2xcp8lfg24zq7gr3y8ffczn1nsxp-guix-d79c917f2-modules >> finding garbage collector roots... >> [0 MiB] deleting >> '/gnu/store/x7ns2xcp8lfg24zq7gr3y8ffczn1nsxp-guix-d79c917f2-modules' >> deleting `/gnu/store/trash' >> deleting unused links... >> note: currently hard linking saves 1181.36 MiB >>=20 >> # Still there... >> root@localhost /gnu/store# du -hs >> /gnu/store/x7ns2xcp8lfg24zq7gr3y8ffczn1nsxp-guix-d79c917f2-modules >> 210M /gnu/store/x7ns2xcp8lfg24zq7gr3y8ffczn1nsxp-guix-d79c917f2-modules > > Okay, something is definitely not right. > > [0] https://lists.gnu.org/archive/html/help-guix/2020-05/msg00235.html There are two problems. One is that Stephen's images are getting larger every day. The other is that Guix is failing to GC dead store items in the Docker container. The latter does not cause the former. The reason Guix is failing to GC dead items in the Docker container is because those dead items are not on the "top layer", so Docker returns an EXDEV error: https://docs.docker.com/storage/storagedriver/overlayfs-driver/ "Renaming directories: Calling rename(2) for a directory is allowed only when both the source and the destination path are on the top layer. Otherwise, it returns EXDEV error ('cross-device link not permitted'). Your application needs to be designed to handle EXDEV and fall back to a 'copy and unlink' strategy." You can observe this by running guix-daemon with strace in the container, and watching what happens when you try to delete one of the offending store items (make sure it is a directory). For example: =2D-8<---------------cut here---------------start------------->8--- 685 rename("/gnu/store/xib50iqk3w1gw9l770mad59m9bi3bcpc-manual-database",= "/gnu/store/trash/xib50iqk3w1gw9l770mad59m9bi3bcpc-manual-database") =3D -= 1 EXDEV (Invalid cross-device link) =2D-8<---------------cut here---------------end--------------->8--- In most cases, when guix-daemon GC's a dead directory, it does this (see: nix/libstore/gc.cc): =2D Create a trash directory (usually /gnu/store/trash) =2D Move dead directories into the trash directory. =2D Delete the trash directory. The trash directory is on the "top layer" because it gets created in the running container. However, in practice many store items from lower layers are made dead when Stephen's script runs "guix pull" and deletes the old profiles. If any of those store items were directories, guix-daemon will fail to GC them because of an XDEV error. If this is confusing to you, I suggest you experiment with Docker a little bit, and look closely at the steps that Stephen's script is running. I outlined this in the email I accidentally deleted, but I'm a little too tired to reproduce it all a second time. I hope you'll understand. Should Guix do anything about this? We could change guix-daemon to take correct action in the face of an XDEV error. We could also improve the logging, since currently it silently swallows the XDEV error. However, even if we make those changes and Guix is able to GC the dead store items, it would not prevent Stephen's images from growing in size without bound. There would still be many store items that came from a prior image (i.e., a lower layer), became dead after running "guix pull" and deleting old profile generations, and still exist in the prior layer, even though they are not visible in the running container. This is due to Docker's design, in which the visible file system is the result of stitching together all the layers with overlayfs. To work around the issue, Stephen can build the images from the same base image, rather than daisy-chaining new images from old ones. That way, they would not accumulate layers without bound. =2D-=20 Chris --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEEy/WXVcvn5+/vGD+x3UCaFdgiRp0FAl7TOQYACgkQ3UCaFdgi Rp33Ug/9ECuQ7z0lIJi4RJrDuJynnVbEEXGRN0FLUP55HoWULXGvd/cWxKC1Cr5F NRd598QP+40EYy5SVzNrxXUXTRqmQDSUYbIobUINSwcn+q/HjxwLbTnp6v2fijtM aaSc2YoCFzZaPT6MUjb8CB2Spgf/DF9XbwBmpJZdTK84w0wyDb2KEWRrO1bxq7c2 9tdDv3jylah3+Z7k/VnzDyGvJxqrqSsUWdRvJy4QL9THXaU5EP4zRCt8eHGGgaRs SeyjhbYGGCqh2emQOQptEw3AlFnginulO2P0g4dmRx4SaACMnfVfcopB1jGxPNHv YQXEn2o1/AnFxYhUMzycrXhT2pgjHMVrLBWihaCdZxYP7SSyS1UgHZEUfPknYA7H xhoybpHzVDgECiN3MID/s2mTeFZEofianvDy+Wt1/mSkWWwiRS9Eg75UWvWvrcAi 5VD/WhaOpPtdS+UuqM6geLmRRaA2G+aAU9vxtzdZ9uB12iM4Jio/vVWIbfiHofFg acYkYIwibQAKj3K2mfbl/MGyTLL9zc0TIlmLVDOGx4wpOdNeczSkorf3n0Un61mS 1iuA2aThBVn/bdTQRzNEVy9nqN2nbe5oCXesv1NG76lANNC30c4iFLeqsPRipxAM z+nFAWDAWJzviGglcQnyIy4S97T8n4Kq8RYY9AJrQEZhmT7pY24= =dn7X -----END PGP SIGNATURE----- --=-=-=--