“guix gc”, auto gcroots, cluster deployments

“guix gc”, auto gcroots, cluster deployments

[Ricardo Wurmus]

Hi Guix,

On my cluster installation I ran “guix gc --list-dead” out of 
curiosity.  When finding roots, the daemon also removes what it 
considers stale links.  On my cluster installation not all links 
always resolve, because the target files reside on remote file 
systems.  These remote locations are not readable by the root user 
on the server where guix-daemon runs (ignoring local root 
privileges is pretty common for NFS servers), so they cannot 
possibly be resolved.

So the daemon ends up deleting all these links from 
/var/guix/gcroots/auto/.

This may not be too bad on its own, but it also means that the 
next time around “guix gc” would consider the eventual target 
profiles to be garbage.

There are two problems here:

1) I don’t think “guix gc --list-dead” (or “--list-live”, or more 
generally “findRoots” in nix/libstore/gc.cc) should delete 
anything.  It should just list and not clean up.

2) For cluster installations with remote file systems perhaps 
there’s something else we can do to record gcroots.  We now have 
this excursion into unreadable space because we use a symlink, but 
the start ($localstatedir/gcroots/auto) and endpoints 
(/gnu/store/…) are both accessible by the daemon.  Since these 
intermediate locations are tied to user accounts, could we not 
store them in a per-user directory?

This problem does not exist for user profiles, because the link in 
unreadable home directories is not all that important; it merely 
points to $localstatedir, which is always readable by the daemon. 
Perhaps we could do the same for temporary roots and let *users* 
decide when to let go of them by giving them a command to erase 
the important links in $localstatedir.

So instead of having a link from 
/gnu/var/guix/gcroots/auto/8ypp8dmwnydgbsgjcms2wyb32mng0wri to 
/home/me/projects/mrg1_chipseq/.guix-profile-1-link pointing to 
/gnu/store/ap0vrfxjdj57iqdapg8q83l4f7aylqzm-profile, we would 
record 
/var/guix/profiles/per-user/me/auto/8ypp8dmwnydgbsgjcms2wyb32mng0wri 
pointing to /gnu/store/ap0vrfxjdj57iqdapg8q83l4f7aylqzm-profile, 
and then point /home/me/projects/mrg1_chipseq/.guix-profile-1-link 
at that.  Yes, removing 
/home/me/projects/mrg1_chipseq/.guix-profile-1-link would no 
longer free up the profile for garbage collection, but removing 
$(readlink /home/me/projects/mrg1_chipseq/.guix-profile-1-link) 
would.

This change would pretty much solve the problem for cluster 
deployments, which otherwise leads to “guix gc” avoidance.

What do you think?

-- 
Ricardo

Re: “guix gc”, auto gcroots, cluster deployments

[Roel Janssen]

On Mon, 2021-05-10 at 11:59 +0200, Ricardo Wurmus wrote:
> Hi Guix,
> 
> On my cluster installation I ran “guix gc --list-dead” out of 
> curiosity.  When finding roots, the daemon also removes what it 
> considers stale links.  On my cluster installation not all links 
> always resolve, because the target files reside on remote file 
> systems.  These remote locations are not readable by the root user 
> on the server where guix-daemon runs (ignoring local root 
> privileges is pretty common for NFS servers), so they cannot 
> possibly be resolved.
> 
> So the daemon ends up deleting all these links from 
> /var/guix/gcroots/auto/.
> 
> This may not be too bad on its own, but it also means that the 
> next time around “guix gc” would consider the eventual target 
> profiles to be garbage.
> 
> There are two problems here:
> 
> 1) I don’t think “guix gc --list-dead” (or “--list-live”, or more 
> generally “findRoots” in nix/libstore/gc.cc) should delete 
> anything.  It should just list and not clean up.
> 

I agree it would be better if --list-* options don't remove store items
and/or database entries.

> 2) For cluster installations with remote file systems perhaps 
> there’s something else we can do to record gcroots.  We now have 
> this excursion into unreadable space because we use a symlink, but 
> the start ($localstatedir/gcroots/auto) and endpoints 
> (/gnu/store/…) are both accessible by the daemon.  Since these 
> intermediate locations are tied to user accounts, could we not 
> store them in a per-user directory?
> 
> This problem does not exist for user profiles, because the link in 
> unreadable home directories is not all that important; it merely 
> points to $localstatedir, which is always readable by the daemon. 
> Perhaps we could do the same for temporary roots and let *users* 
> decide when to let go of them by giving them a command to erase 
> the important links in $localstatedir.
> 
> So instead of having a link from 
> /gnu/var/guix/gcroots/auto/8ypp8dmwnydgbsgjcms2wyb32mng0wri to 
> /home/me/projects/mrg1_chipseq/.guix-profile-1-link pointing to 
> /gnu/store/ap0vrfxjdj57iqdapg8q83l4f7aylqzm-profile, we would 
> record 
> /var/guix/profiles/per-user/me/auto/8ypp8dmwnydgbsgjcms2wyb32mng0wri 
> pointing to /gnu/store/ap0vrfxjdj57iqdapg8q83l4f7aylqzm-profile, 
> and then point /home/me/projects/mrg1_chipseq/.guix-profile-1-link 
> at that.  Yes, removing 
> /home/me/projects/mrg1_chipseq/.guix-profile-1-link would no 
> longer free up the profile for garbage collection, but removing 
> $(readlink /home/me/projects/mrg1_chipseq/.guix-profile-1-link) 
> would.
> 
> This change would pretty much solve the problem for cluster 
> deployments, which otherwise leads to “guix gc” avoidance.
> 
> What do you think?
> 

We are facing a similar issue on our cluster deployment. In our case,
directories are only readable to certain users but not root.  So the
garbage collector can't read the symlinks, and therefore cannot
determine whether a profile is still in use or not.

So this seems like a good idea to me.

Would it be possible to add an option to retrospectively apply this
transformation?  Maybe that could work somewhat like this:

$ ls -lha
... /home/me/projects/mrg1_chipseq/.guix-profile-1-link ->
/gnu/store/ap0vrfxjdj57iqdapg8q83l4f7aylqzm-profile


$ ls -lh $localstatedir/gcroots/auto
$ guix gc --synchronize-profiles-to-gcroots
$ ls -lh $localstatedir/gcroots/auto
... 8ypp8dmwnydgbsgjcms2wyb32mng0wri ->
/gnu/store/ap0vrfxjdj57iqdapg8q83l4f7aylqzm-profile

$ rm /home/me/projects/mrg1_chipseq/.guix-profile-1-link 

$ guix gc --synchronize-profiles-to-gcroots
$ ls -lh $localstatedir/gcroots/auto

Kind regards,
Roel Janssen

Re: “guix gc”, auto gcroots, cluster deployments

[Ricardo Wurmus]

Hi Roel,

thanks for your feedback!

> Would it be possible to add an option to retrospectively apply 
> this
> transformation?  Maybe that could work somewhat like this:
>
> $ ls -lha
> ... /home/me/projects/mrg1_chipseq/.guix-profile-1-link ->
> /gnu/store/ap0vrfxjdj57iqdapg8q83l4f7aylqzm-profile

This wouldn’t work, because we can’t read 
/home/me/projects/mrg1_chipseq/.guix-profile-1-link centrally.  In 
this particular case only the user “me” could resolve the link and 
thus migrate the link.  (I would do this semi-manually by 
impersonating the users to read their links.)

~ ~ ~

Another related problem I found is that the names of the links in 
unreadable space may have names that only make sense on the system 
where they were created.  For example, on the server “beast” we 
may mount the cluster home directory as “/clusterhome/me”, whereas 
on cluster nodes it would be mounted as “/home/me”.  When I have 
Guix record a gcroot while working on “beast” I would get a link 
that is no longer valid when I work on the cluster.

-- 
Ricardo

Re: “guix gc”, auto gcroots, cluster deployments

[Roel Janssen]

On Mon, 2021-05-10 at 13:59 +0200, Ricardo Wurmus wrote:
> 
> Hi Roel,
> 
> thanks for your feedback!
> 
> > Would it be possible to add an option to retrospectively apply 
> > this
> > transformation?  Maybe that could work somewhat like this:
> > 
> > $ ls -lha
> > ... /home/me/projects/mrg1_chipseq/.guix-profile-1-link ->
> > /gnu/store/ap0vrfxjdj57iqdapg8q83l4f7aylqzm-profile
> 
> This wouldn’t work, because we can’t read 
> /home/me/projects/mrg1_chipseq/.guix-profile-1-link centrally.  In 
> this particular case only the user “me” could resolve the link and 
> thus migrate the link.  (I would do this semi-manually by 
> impersonating the users to read their links.)
> 

Indeed we cannot resolve /home/me/projects/mrg1_chipseq/.guix-profile-
1-link as another user than "me".  So "me" would have to run the
hypothetical "guix gc --synchronize-profiles-to-gcroots".

From my point of view, that would be fine.  We can simply ask the users
on the cluster to execute that command once.

This could work if we have a mechanism to determine how complete the
garbage-collection picture for "root" is, and only continue GCing when
the picture is complete.  A starting point for this is 
$localstatedir/gcroots/auto which provides an overview of all profiles
that have been created. Perhaps we can reuse the filename of the
symlink.  So in the example given in the previous e-mail we could add
another link like so, but in a different directory:
8ypp8dmwnydgbsgjcms2wyb32mng0wri -> /gnu/store/...-profile

For which this already existed in $localstatedir/gcroots/auto:
8ypp8dmwnydgbsgjcms2wyb32mng0wri ->
/home/me/projects/mrg1_chipseq/.guix-profile-1-link

And then a one-time transition can be made by looking up 
8ypp8dmwnydgbsgjcms2wyb32mng0wri in both places.

Maybe I'm making things needlessly complex, but I would really like a
solution that can be applied to an existing cluster setup. :)

So, your initial proposal sounds good to me already. How it could be
made more useful for the retrospective matching of unreachable symlinks
to /gnu/store paths can be considered a separate issue to solve.

> ~ ~ ~
> 
> Another related problem I found is that the names of the links in 
> unreadable space may have names that only make sense on the system 
> where they were created.  For example, on the server “beast” we 
> may mount the cluster home directory as “/clusterhome/me”, whereas 
> on cluster nodes it would be mounted as “/home/me”.  When I have 
> Guix record a gcroot while working on “beast” I would get a link 
> that is no longer valid when I work on the cluster.
> 

Yes, I agree that this is another reason to implement your suggested
change. :)

Kind regards,
Roel Janssen

Re: “guix gc”, auto gcroots, cluster deployments

[Ricardo Wurmus]

Hi Roel,

> Maybe I'm making things needlessly complex, but I would really 
> like a
> solution that can be applied to an existing cluster setup. :)

Oh, I’d also like something automated that works for an existing 
cluster setup (like mine that prompted this conversation), but in 
my imagination this does not involve users running commands that 
would become part of the permanent Guix command line footprint.

Instead I imagine a sysadmin would have enough privileges to 
impersonate the users and rewrite the links for them.

Full automation is not possible in my case anyway, because I can’t 
necessarily tell who owns the target file (it’s not just user home 
directories but also some other obscurely named shared 
directories), nor can I figure out automatically on which server 
the link was created (when I see a link like “/local/foo/bar”, 
which is a server-local directory that is not exported on any 
other system).


Even though the two of us seem to agree that this change is 
necessary, it does result in a regression: for systems where none 
of this is a problem (e.g. single-user laptops) it’s rather 
inconvenient to have all these temporary gcroots accumulate. 
Prior to my proposed change they would be freed up for garbage 
collection as soon as the link was removed.

(I also wonder if the implementation of “guix gc 
--delete-generations” would need adjusting; I haven’t looked yet.)

Is there something we can do to have our cake and eat it too?

-- 
Ricardo

Re: “guix gc”, auto gcroots, cluster deployments

[Ludovic Courtès]

Hello!

Ricardo Wurmus <rekado@elephly.net> skribis:

>> Would it be possible to add an option to retrospectively apply this
>> transformation?  Maybe that could work somewhat like this:
>>
>> $ ls -lha
>> ... /home/me/projects/mrg1_chipseq/.guix-profile-1-link ->
>> /gnu/store/ap0vrfxjdj57iqdapg8q83l4f7aylqzm-profile
>
> This wouldn’t work, because we can’t read
> /home/me/projects/mrg1_chipseq/.guix-profile-1-link centrally.  In 
> this particular case only the user “me” could resolve the link and
> thus migrate the link.  (I would do this semi-manually by 
> impersonating the users to read their links.)
>
> ~ ~ ~
>
> Another related problem I found is that the names of the links in
> unreadable space may have names that only make sense on the system 
> where they were created.  For example, on the server “beast” we may
> mount the cluster home directory as “/clusterhome/me”, whereas on
> cluster nodes it would be mounted as “/home/me”.  When I have Guix
> record a gcroot while working on “beast” I would get a link that is no
> longer valid when I work on the cluster.

To me these are “documented limitations”.  User home directories must be
visible to the head node where guix-daemon runs, or GC won’t see all the
roots.  It seems hard to avoid.  Though an admin could decide that only
~/.guix-profile matters, and in that case the head node only needs to be
able to access /var/guix/profiles (a restriction that may be acceptable
in practice: users need to restrict themselves to ~/.guix-profile and
‘guix environment’).

But yeah, I agree these are gotchas and it’s easy for a cluster admin to
shoot themself in the foot.

Thoughts?

Ludo’.

Re: “guix gc”, auto gcroots, cluster deployments

[Sébastien Lerique]

Hi Ricardo, all,

On 10 May 2021 at 18:59, Ricardo Wurmus <rekado@elephly.net> 
wrote:

> On my cluster installation I ran “guix gc --list-dead” out of 
> curiosity.  When
> finding roots, the daemon also removes what it considers stale 
> links.  On my
> cluster installation not all links always resolve, because the 
> target files
> reside on remote file systems.  These remote locations are not 
> readable by the
> root user on the server where guix-daemon runs (ignoring local 
> root
> privileges is pretty common for NFS servers), so they cannot 
> possibly be
> resolved.

Please excuse my hijacking this thread: I have been trying out 
configurations to get Guix working on the cluster I have access 
to, without root access for the daemon, and one problem I ran into 
is a curious bug when the store is on an NFS share. See 
https://lists.gnu.org/archive/html/guix-science/2021-03/msg00010.html 
(Case 3).

In your setup, is the store on an NFS share? If so, did you do 
anything special to get that working?

Best,
Sébastien

Guix on NFS

[Ricardo Wurmus]

Hi Sébastien,

> Please excuse my hijacking this thread: I have been trying out
> configurations to get Guix working on the cluster I have access 
> to, without root access for the daemon, and one problem I ran 
> into is
> a curious bug when the store is on an NFS share. See
> https://lists.gnu.org/archive/html/guix-science/2021-03/msg00010.html 
> (Case 3).

This sounds like localstatedir is mismatched.

> In your setup, is the store on an NFS share? If so, did you do
> anything special to get that working?

It is on an NFS share, but I’m about to change that for 
performance reasons.

-- 
Ricardo

Re: “guix gc”, auto gcroots, cluster deployments

[Ludovic Courtès]

Hi!

Ricardo Wurmus <rekado@elephly.net> skribis:

> There are two problems here:
>
> 1) I don’t think “guix gc --list-dead” (or “--list-live”, or more
> generally “findRoots” in nix/libstore/gc.cc) should delete 
> anything.  It should just list and not clean up.

Maybe ‘findRoots’ could populate the list of stale roots and it’d be up
to the caller to decide whether to delete them or not?

> 2) For cluster installations with remote file systems perhaps there’s
> something else we can do to record gcroots.  We now have this
> excursion into unreadable space because we use a symlink, but the
> start ($localstatedir/gcroots/auto) and endpoints (/gnu/store/…) are
> both accessible by the daemon.  Since these intermediate locations are
> tied to user accounts, could we not store them in a per-user
> directory?
>
> This problem does not exist for user profiles, because the link in
> unreadable home directories is not all that important; it merely 
> points to $localstatedir, which is always readable by the
> daemon. Perhaps we could do the same for temporary roots and let
> *users* decide when to let go of them by giving them a command to
> erase the important links in $localstatedir.
>
> So instead of having a link from
> /gnu/var/guix/gcroots/auto/8ypp8dmwnydgbsgjcms2wyb32mng0wri to 
> /home/me/projects/mrg1_chipseq/.guix-profile-1-link pointing to
> /gnu/store/ap0vrfxjdj57iqdapg8q83l4f7aylqzm-profile, we would 
> record
> /var/guix/profiles/per-user/me/auto/8ypp8dmwnydgbsgjcms2wyb32mng0wri 
> pointing to /gnu/store/ap0vrfxjdj57iqdapg8q83l4f7aylqzm-profile, and
> then point /home/me/projects/mrg1_chipseq/.guix-profile-1-link at
> that.  Yes, removing
> /home/me/projects/mrg1_chipseq/.guix-profile-1-link would no 
> longer free up the profile for garbage collection, but removing
> $(readlink /home/me/projects/mrg1_chipseq/.guix-profile-1-link) 
> would.

Yes, but how would per-user/me/auto/* be cleaned up?

It seems to be that as long as the final symlink can live the home
directories, the GC needs to have access to those home directories to
determine whether that symlink is still a GC root.

Tricky!

Ludo’.

Re: “guix gc”, auto gcroots, cluster deployments

[Ricardo Wurmus]

Ludovic Courtès <ludovic.courtes@inria.fr> writes:

> Hi!
>
> Ricardo Wurmus <rekado@elephly.net> skribis:
>
>> There are two problems here:
>>
>> 1) I don’t think “guix gc --list-dead” (or “--list-live”, or 
>> more
>> generally “findRoots” in nix/libstore/gc.cc) should delete 
>> anything.  It should just list and not clean up.
>
> Maybe ‘findRoots’ could populate the list of stale roots and 
> it’d be up
> to the caller to decide whether to delete them or not?

Yes, this would be better.  It already does this for links whose 
targets exist but cannot be read.

>> 2) For cluster installations with remote file systems perhaps 
>> there’s
>> something else we can do to record gcroots.  We now have this
>> excursion into unreadable space because we use a symlink, but 
>> the
>> start ($localstatedir/gcroots/auto) and endpoints 
>> (/gnu/store/…) are
>> both accessible by the daemon.  Since these intermediate 
>> locations are
>> tied to user accounts, could we not store them in a per-user
>> directory?
>>
>> This problem does not exist for user profiles, because the link 
>> in
>> unreadable home directories is not all that important; it 
>> merely 
>> points to $localstatedir, which is always readable by the
>> daemon. Perhaps we could do the same for temporary roots and 
>> let
>> *users* decide when to let go of them by giving them a command 
>> to
>> erase the important links in $localstatedir.
>>
>> So instead of having a link from
>> /gnu/var/guix/gcroots/auto/8ypp8dmwnydgbsgjcms2wyb32mng0wri to 
>> /home/me/projects/mrg1_chipseq/.guix-profile-1-link pointing to
>> /gnu/store/ap0vrfxjdj57iqdapg8q83l4f7aylqzm-profile, we would 
>> record
>> /var/guix/profiles/per-user/me/auto/8ypp8dmwnydgbsgjcms2wyb32mng0wri 
>> pointing to 
>> /gnu/store/ap0vrfxjdj57iqdapg8q83l4f7aylqzm-profile, and
>> then point /home/me/projects/mrg1_chipseq/.guix-profile-1-link 
>> at
>> that.  Yes, removing
>> /home/me/projects/mrg1_chipseq/.guix-profile-1-link would no 
>> longer free up the profile for garbage collection, but removing
>> $(readlink /home/me/projects/mrg1_chipseq/.guix-profile-1-link) 
>> would.
>
> Yes, but how would per-user/me/auto/* be cleaned up?

Yeah, that’s an open question.

I get the appeal of having these things be cleaned up 
automatically when the link disappears, but if we added this extra 
layer of indirection for cluster deployments this would become 
manual.

Can we make this configurable perhaps…?  On my cluster 
installation I’d rather have a cron job to erase the stuff in 
per-user/me/auto/* on my own terms, than to have “guix gc” fail to 
resolve links and consider it all garbage.

-- 
Ricardo

Re: “guix gc”, auto gcroots, cluster deployments

[Ludovic Courtès]

Hi,

Ricardo Wurmus <rekado@elephly.net> skribis:

> Ludovic Courtès <ludovic.courtes@inria.fr> writes:
>
>> Hi!
>>
>> Ricardo Wurmus <rekado@elephly.net> skribis:
>>
>>> There are two problems here:
>>>
>>> 1) I don’t think “guix gc --list-dead” (or “--list-live”, or more
>>> generally “findRoots” in nix/libstore/gc.cc) should delete
>>> anything.  It should just list and not clean up.
>>
>> Maybe ‘findRoots’ could populate the list of stale roots and it’d be
>> up
>> to the caller to decide whether to delete them or not?
>
> Yes, this would be better.  It already does this for links whose
> targets exist but cannot be read.

OK.

>>> 2) For cluster installations with remote file systems perhaps
>>> there’s
>>> something else we can do to record gcroots.  We now have this

[...]

>>> […] we would
>>> record
>>> /var/guix/profiles/per-user/me/auto/8ypp8dmwnydgbsgjcms2wyb32mng0wri
>>> pointing to 
>>> /gnu/store/ap0vrfxjdj57iqdapg8q83l4f7aylqzm-profile, and
>>> then point /home/me/projects/mrg1_chipseq/.guix-profile-1-link at
>>> that.  Yes, removing
>>> /home/me/projects/mrg1_chipseq/.guix-profile-1-link would no longer
>>> free up the profile for garbage collection, but removing
>>> $(readlink /home/me/projects/mrg1_chipseq/.guix-profile-1-link)
>>> would.
>>
>> Yes, but how would per-user/me/auto/* be cleaned up?
>
> Yeah, that’s an open question.
>
> I get the appeal of having these things be cleaned up automatically
> when the link disappears, but if we added this extra layer of
> indirection for cluster deployments this would become manual.
>
> Can we make this configurable perhaps…?  On my cluster installation
> I’d rather have a cron job to erase the stuff in per-user/me/auto/* on
> my own terms, than to have “guix gc” fail to resolve links and
> consider it all garbage.

Sure, why not.  Do you have configuration options in mind?

I have troubles wrapping my head around this problem.  Should we split
it into smaller chunks in bug-guix?

Thanks,
Ludo’.