Re: “guix gc”, auto gcroots, cluster deployments

[Roel Janssen <roel@gnu.org>]

On Mon, 2021-05-10 at 11:59 +0200, Ricardo Wurmus wrote:
> Hi Guix,
> 
> On my cluster installation I ran “guix gc --list-dead” out of 
> curiosity.  When finding roots, the daemon also removes what it 
> considers stale links.  On my cluster installation not all links 
> always resolve, because the target files reside on remote file 
> systems.  These remote locations are not readable by the root user 
> on the server where guix-daemon runs (ignoring local root 
> privileges is pretty common for NFS servers), so they cannot 
> possibly be resolved.
> 
> So the daemon ends up deleting all these links from 
> /var/guix/gcroots/auto/.
> 
> This may not be too bad on its own, but it also means that the 
> next time around “guix gc” would consider the eventual target 
> profiles to be garbage.
> 
> There are two problems here:
> 
> 1) I don’t think “guix gc --list-dead” (or “--list-live”, or more 
> generally “findRoots” in nix/libstore/gc.cc) should delete 
> anything.  It should just list and not clean up.
> 

I agree it would be better if --list-* options don't remove store items
and/or database entries.

> 2) For cluster installations with remote file systems perhaps 
> there’s something else we can do to record gcroots.  We now have 
> this excursion into unreadable space because we use a symlink, but 
> the start ($localstatedir/gcroots/auto) and endpoints 
> (/gnu/store/…) are both accessible by the daemon.  Since these 
> intermediate locations are tied to user accounts, could we not 
> store them in a per-user directory?
> 
> This problem does not exist for user profiles, because the link in 
> unreadable home directories is not all that important; it merely 
> points to $localstatedir, which is always readable by the daemon. 
> Perhaps we could do the same for temporary roots and let *users* 
> decide when to let go of them by giving them a command to erase 
> the important links in $localstatedir.
> 
> So instead of having a link from 
> /gnu/var/guix/gcroots/auto/8ypp8dmwnydgbsgjcms2wyb32mng0wri to 
> /home/me/projects/mrg1_chipseq/.guix-profile-1-link pointing to 
> /gnu/store/ap0vrfxjdj57iqdapg8q83l4f7aylqzm-profile, we would 
> record 
> /var/guix/profiles/per-user/me/auto/8ypp8dmwnydgbsgjcms2wyb32mng0wri 
> pointing to /gnu/store/ap0vrfxjdj57iqdapg8q83l4f7aylqzm-profile, 
> and then point /home/me/projects/mrg1_chipseq/.guix-profile-1-link 
> at that.  Yes, removing 
> /home/me/projects/mrg1_chipseq/.guix-profile-1-link would no 
> longer free up the profile for garbage collection, but removing 
> $(readlink /home/me/projects/mrg1_chipseq/.guix-profile-1-link) 
> would.
> 
> This change would pretty much solve the problem for cluster 
> deployments, which otherwise leads to “guix gc” avoidance.
> 
> What do you think?
> 

We are facing a similar issue on our cluster deployment. In our case,
directories are only readable to certain users but not root.  So the
garbage collector can't read the symlinks, and therefore cannot
determine whether a profile is still in use or not.

So this seems like a good idea to me.

Would it be possible to add an option to retrospectively apply this
transformation?  Maybe that could work somewhat like this:

$ ls -lha
... /home/me/projects/mrg1_chipseq/.guix-profile-1-link ->
/gnu/store/ap0vrfxjdj57iqdapg8q83l4f7aylqzm-profile


$ ls -lh $localstatedir/gcroots/auto
$ guix gc --synchronize-profiles-to-gcroots
$ ls -lh $localstatedir/gcroots/auto
... 8ypp8dmwnydgbsgjcms2wyb32mng0wri ->
/gnu/store/ap0vrfxjdj57iqdapg8q83l4f7aylqzm-profile

$ rm /home/me/projects/mrg1_chipseq/.guix-profile-1-link 

$ guix gc --synchronize-profiles-to-gcroots
$ ls -lh $localstatedir/gcroots/auto

Kind regards,
Roel Janssen