From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp1 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms11 with LMTPS id kvu2J4R72l7TTAAA0tVLHw (envelope-from ) for ; Fri, 05 Jun 2020 17:06:12 +0000 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp1 with LMTPS id EKfpIoR72l5rUgAAbx9fmQ (envelope-from ) for ; Fri, 05 Jun 2020 17:06:12 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id EC885940308 for ; Fri, 5 Jun 2020 17:06:11 +0000 (UTC) Received: from localhost ([::1]:33316 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jhFnG-0007YX-OW for larch@yhetil.org; Fri, 05 Jun 2020 13:06:10 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:43280) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jhFn8-0007WB-EU for bug-guix@gnu.org; Fri, 05 Jun 2020 13:06:02 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:38211) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1jhFn8-0005Kb-5n for bug-guix@gnu.org; Fri, 05 Jun 2020 13:06:02 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1jhFn8-0002o6-13 for bug-guix@gnu.org; Fri, 05 Jun 2020 13:06:02 -0400 X-Loop: help-debbugs@gnu.org Subject: bug#41607: Deleted store items are not actually deleted Resent-From: Stephen Scheck Original-Sender: "Debbugs-submit" Resent-CC: bug-guix@gnu.org Resent-Date: Fri, 05 Jun 2020 17:06:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 41607 X-GNU-PR-Package: guix X-GNU-PR-Keywords: To: Christopher Marusich Received: via spool by 41607-submit@debbugs.gnu.org id=B41607.159137675310776 (code B ref 41607); Fri, 05 Jun 2020 17:06:01 +0000 Received: (at 41607) by debbugs.gnu.org; 5 Jun 2020 17:05:53 +0000 Received: from localhost ([127.0.0.1]:49757 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jhFmy-0002nk-Ms for submit@debbugs.gnu.org; Fri, 05 Jun 2020 13:05:53 -0400 Received: from mail-lj1-f182.google.com ([209.85.208.182]:34544) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jhEqk-0001H9-0F for 41607@debbugs.gnu.org; Fri, 05 Jun 2020 12:05:42 -0400 Received: by mail-lj1-f182.google.com with SMTP id b6so12433428ljj.1 for <41607@debbugs.gnu.org>; Fri, 05 Jun 2020 09:05:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=Oh9RFZQoHBhwugyE4gUSlVEeSdqnE8ZS/lBEJVsRRr4=; b=bmjF/Ngsf/wVCb6cfz5F/UfpO/iZKBrZ51IDGL2O2noZTgOx9koS+cgUJsR9BnQHyG 9VNQQ+9ktxtZls5qxL/x4BiwbI/hk7/CM6q7W3Hp+czlqSqX/ws5RlLZ5sWEC3jFU5Id 6IyfbASwR/evAE86k6ut1ggZMp4T5++kUI3XrB9T99dhhxMDZhVr8Accd2lHql/eaADK Ibmgws+/UrthPNUh8eJiWdvYE7F2mh7ahkAv0kvD3bIl2FhoayOujGR+d8juKXWM8YAL c822seOA9oWNNB3SmQQj1Dtfq6k7gzwZ7w7Zf16NIWsNaSY4znLjhF4cLhQbSQ8uweZH XK9w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=Oh9RFZQoHBhwugyE4gUSlVEeSdqnE8ZS/lBEJVsRRr4=; b=rO1hHcJfbRJTy+UN2IveuOw1m61DImNYugWCmPWLym6d8AXcdr7o2rPCfxRWsT/5Xq tUvq5QFu7R5NNHSYg0b4AlBu8JQtuP9aPLcvs5try07/8ZUvlex+J6L18gzDhhnX8c5A p3d6ou6tSTwgXJ2Mxf9qTlaQMjumrc3dgoYQwdIxn64CgmOik0fkNMzMdNYpPicaecsm 64UKTCNysgzoArGBQ/Q1BLeC2j15kBsU2KgS6FPNhvnFc37TXde1kXnPFaFRyNgBR7WT FDqOz9i3t+ULtYKGS4mAlOoNs5KMXy6nK85bx1qBLO2kpKI9qWX2YQ1L8OlfCJw5VIl5 i0/w== X-Gm-Message-State: AOAM531nSKmiyC36VPDgMSaD7Uv06+FUcnUosMpGOgGcp4ZeovO6Cgtj VD3iy0MNI0Uxk4i9QL8LwHZB9pT7yYvCHkjoxYU= X-Google-Smtp-Source: ABdhPJznoS979wYwcbFaBMUHEtIwi/t8pmAeCW9ZoQkylgBzTrDwdyZwxyTE40fEo/O8cHggoJe7WdCHLs3DFrV1zPI= X-Received: by 2002:a05:651c:118f:: with SMTP id w15mr5168823ljo.211.1591373135883; Fri, 05 Jun 2020 09:05:35 -0700 (PDT) MIME-Version: 1.0 References: <20200528181043.GC23745@jasmine.lan> <20200529170820.GA30828@jasmine.lan> <20200529180245.GA3754@jasmine.lan> <20200529190942.GA8440@jasmine.lan> <87r1v0k8hl.fsf@gmail.com> <87eeqvcaau.fsf@gnu.org> <87r1uu1x9e.fsf@gmail.com> <87pnad6eor.fsf@gmail.com> In-Reply-To: <87pnad6eor.fsf@gmail.com> From: Stephen Scheck Date: Fri, 5 Jun 2020 12:05:24 -0400 Message-ID: Content-Type: multipart/alternative; boundary="00000000000078324705a7586e03" X-Spam-Score: 0.0 (/) X-Mailman-Approved-At: Fri, 05 Jun 2020 13:05:51 -0400 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-Spam-Score: -1.0 (-) X-BeenThere: bug-guix@gnu.org List-Id: Bug reports for GNU Guix List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: 41607@debbugs.gnu.org Errors-To: bug-guix-bounces+larch=yhetil.org@gnu.org Sender: "bug-Guix" X-Scanner: scn0 Authentication-Results: aspmx1.migadu.com; dkim=fail (rsa verify failed) header.d=gmail.com header.s=20161025 header.b=bmjF/Ngs; dmarc=fail reason="SPF not aligned (relaxed)" header.from=gmail.com (policy=none); spf=pass (aspmx1.migadu.com: domain of bug-guix-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=bug-guix-bounces@gnu.org X-Spam-Score: 1.09 X-TUID: 9s8M34shj0m4 --00000000000078324705a7586e03 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Very cool - thanks Chris! In the meantime, I've updated my build script to externalize the Guix environment from the Docker container. So far the daily builds are staying nice and small at ~197MB and one layer. The images and updated script are here if anybody is curious: https://hub.docker.com/r/singularsyntax/guix/tags https://gitlab.com/singularsyntax-docker-hub/guix/-/blob/master/.gitlab-ci.= yml GitLab allows you to cache files between job stages and even full pipeline runs. I first tried to cache /var/guix and /gnu/store and mount them inside a container in which to perform `guix pull` etc. However, it seems that handling hard links on externally mounted file systems from within a container is problematic. I think passing `--disable-deduplication` to guix-daemon might resolve it, but I couldn't figure out where/how to change the Shepherd configuration to do that. So instead, I just copy the directories into and out of the container at start and end of the process. It seems to work. Mounting would probably speed up the process quite a bit if I could make it work. Cheers, -Steve On Fri, Jun 5, 2020 at 5:32 AM Christopher Marusich wrote: > Chris Marusich writes: > > > Ludovic Court=C3=A8s writes: > > > >>> Should Guix do anything about this? We could change guix-daemon to > take > >>> correct action in the face of an XDEV error. We could also improve t= he > >>> logging, since currently it silently swallows the XDEV error. > >> > >> I guess we could delete recursively right away upon EXDEV. It should = be > >> just two lines of code, right? > > > > I'll try making the change and report back. Yes, there are other cases > > where we immediately delete without moving into the trash directory > > (e.g., when the trash directory fails to be created), so it seems OK. > > Here is a patch. Turns out it's was just a one line change! If nobody > has any further feedback on it, I'll go ahead and merge it to the master > branch in the next couple days. > > I tested it in one of the Docker containers provided by Stephen which > exhibited the problem. I built the new Guix inside the Docker container > and verified that a path which was previously unable to be GC'd due to > the EXDEV error, was now able to be successfully GC'd. > > My understanding is that the only reason why the guix-daemon attempts to > move dead directories to the trash directory is to save time on > deleting, since large directories could take a while to fully delete. > If there is any reason why it might be unsafe to delete the directories > directly in case of EXDEV (I cannot think of any), please let me know. > > -- > Chris > --00000000000078324705a7586e03 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Very cool - thanks Chris!

In= the meantime, I've updated my build script to externalize the Guix env= ironment from the Docker container.
So far the daily builds are s= taying nice and small at ~197MB and one layer. The images and updated scrip= t are
here if anybody is curious:

<= div>
GitLab allows you to cache files between job stages and even = full pipeline runs. I first tried to cache /var/guix
and /gnu/store and = mount them inside a container in which to perform `guix pull` etc. However,= it seems
that handling hard links on externally mounted file systems fr= om within a container is problematic. I think
passing `--disable-de= duplication` to guix-daemon might resolve it, but I couldn't figure out= where/how to
change the Shepherd configuration to do that. So in= stead, I just copy the directories into and out of the
container = at start and end of the process. It seems to work. Mounting would probably = speed up the process
quite a bit if I could make it work.

Cheers,
-Steve

On Fri, Jun 5, 2020 at = 5:32 AM Christopher Marusich <cm= marusich@gmail.com> wrote:
Chris Marusich <cmmarusich@gmail.com> writes:

> Ludovic Court=C3=A8s <ludo@gnu.org> writes:
>
>>> Should Guix do anything about this?=C2=A0 We could change guix= -daemon to take
>>> correct action in the face of an XDEV error.=C2=A0 We could al= so improve the
>>> logging, since currently it silently swallows the XDEV error.<= br> >>
>> I guess we could delete recursively right away upon EXDEV.=C2=A0 I= t should be
>> just two lines of code, right?
>
> I'll try making the change and report back.=C2=A0 Yes, there are o= ther cases
> where we immediately delete without moving into the trash directory > (e.g., when the trash directory fails to be created), so it seems OK.<= br>
Here is a patch.=C2=A0 Turns out it's was just a one line change!=C2=A0= If nobody
has any further feedback on it, I'll go ahead and merge it to the maste= r
branch in the next couple days.

I tested it in one of the Docker containers provided by Stephen which
exhibited the problem.=C2=A0 I built the new Guix inside the Docker contain= er
and verified that a path which was previously unable to be GC'd due to<= br> the EXDEV error, was now able to be successfully GC'd.

My understanding is that the only reason why the guix-daemon attempts to move dead directories to the trash directory is to save time on
deleting, since large directories could take a while to fully delete.
If there is any reason why it might be unsafe to delete the directories
directly in case of EXDEV (I cannot think of any), please let me know.

--
Chris
--00000000000078324705a7586e03--