About /var/guix/profiles and guix pull generations

About /var/guix/profiles and guix pull generations

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 1087 bytes --]

Hi!

I had a look at /var/guix/profiles and noticed that there were some forgotten links
there.  For instance, of some custom profiles I didn't use and all.

After removing those links (sudo rm ...), guix gc could free a lot of
space!

I also noticed that "guix pull" generations were accumulating, and that
might eat up some disk space as well.  So I ran a

--8<---------------cut here---------------start------------->8---
$ guix package -p ~/.config/guix/current -d1m
--8<---------------cut here---------------end--------------->8---

to free all guix checkouts older than a month.

I wonder if we should not document this more.  Maybe have "guix" display
extensive information about the profiles in /var/guix/profiles?
When the guix store runs low on disk space, Guix currently suggests to
run "guix gc" and all.  I think we should suggest the above command and
to check for other GC rotos in /var/guix/profiles.

Thoughts?

Finally, do I need /var/guix/profiles/per-user/root at all?  Is it safe to
remove?

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: About /var/guix/profiles and guix pull generations

[Chris Marusich]

[-- Attachment #1: Type: text/plain, Size: 1422 bytes --]

Pierre Neidhardt <mail@ambrevar.xyz> writes:

> I wonder if we should not document this more.  Maybe have "guix" display
> extensive information about the profiles in /var/guix/profiles?
> When the guix store runs low on disk space, Guix currently suggests to
> run "guix gc" and all.  I think we should suggest the above command and
> to check for other GC rotos in /var/guix/profiles.

I think if Guix has germane things to say, it might as well say them.
Maybe you could submit a patch?

Note that in addition to the profiles found in /var/guix/profiles, there
are other GC roots to consider.  For example, if you create a profile
via "guix package -i coreutils -p /tmp/profile", you'll find a symlink
in /var/guix/gcroots/auto/ which points to it.

Presumably, a user would keep track of the profiles they have installed.
However, it can get confusing, so if Guix can help point things out,
that might be useful.

> Finally, do I need /var/guix/profiles/per-user/root at all?  Is it safe to
> remove?

Does anything use the software installed in it?  If not, then you can
probably get rid of it.  I don't think the root profile is "special"; I
think it's just like any other user's profile.  But if you don't know
whether it's being used, then perhaps it's safest just to leave it be.

This is especially true if, for example, there is no software installed
in the profile.

-- 
Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: About /var/guix/profiles and guix pull generations

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 73 bytes --]

I'll send a patch then.

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: About /var/guix/profiles and guix pull generations

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 82 bytes --]

Thanks for your feedback, Chris!

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: About /var/guix/profiles and guix pull generations

[Pierre Neidhardt]

[-- Attachment #1.1: Type: text/plain, Size: 127 bytes --]

And here is my proposed patch.  It's such a little thing, I'm posting here for
context, I'll merge if no one is opposed to it.

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1.2: 0001-guix-scripts-Mention-Guix-checkouts-gcroots-auto-whe.patch --]
[-- Type: text/x-patch, Size: 1511 bytes --]

From 56e88c25d7b327cd7aaee295b334280417f28aba Mon Sep 17 00:00:00 2001
From: Pierre Neidhardt <mail@ambrevar.xyz>
Date: Sun, 9 Dec 2018 14:33:17 +0100
Subject: [PATCH] guix scripts: Mention Guix checkouts + gcroots/auto when
 warning about disk space.

* guix/scripts.scm (warn-about-disk-space): Mention Guix checkouts and
gcroots/auto.
---
 guix/scripts.scm | 16 +++++++++++-----
 1 file changed, 11 insertions(+), 5 deletions(-)

diff --git a/guix/scripts.scm b/guix/scripts.scm
index 5e20ecd92..e85da99a3 100644
--- a/guix/scripts.scm
+++ b/guix/scripts.scm
@@ -212,13 +212,19 @@ available."
                (* ratio 100) (%store-prefix))
       (if profile
           (display-hint (format #f (G_ "Consider deleting old profile
-generations and collecting garbage, along these lines:
+generations, deleting old Guix checkouts and collecting garbage, along these
+lines:
 
 @example
-guix package -p ~s --delete-generations=1m
-guix gc
-@end example\n")
-                                profile))
+guix package --profile=~s --delete-generations=1m
+guix package --profile=~s --delete-generations=1m
+guix gc --free-space=5G
+@end example
+
+You might also want to delete old non-default profiles in
+/var/guix/gcroots/auto.")
+                                profile
+                                (string-append (config-directory #:ensure? #f) "/current")))
           (display-hint (G_ "Consider running @command{guix gc} to free
 space."))))))
 
-- 
2.19.2


[-- Attachment #1.3: Type: text/plain, Size: 48 bytes --]


-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: About /var/guix/profiles and guix pull generations

[Chris Marusich]

[-- Attachment #1: Type: text/plain, Size: 3283 bytes --]

Hi Pierre,

Awesome, thank you for the patch!

Pierre Neidhardt <mail@ambrevar.xyz> writes:

> -generations and collecting garbage, along these lines:
> +generations, deleting old Guix checkouts and collecting garbage, along these
> +lines:

I'm not sure what you mean by "Guix checkouts".  Do you mean a Git
checkout of Guix's Git repository?  If so, why would deleting it help?

>  @example
> -guix package -p ~s --delete-generations=1m
> -guix gc
> -@end example\n")
> -                                profile))
> +guix package --profile=~s --delete-generations=1m
> +guix package --profile=~s --delete-generations=1m
> +guix gc --free-space=5G
> +@end example

This part seems reasonable to me.  We might also mention that others
users may have default profiles of their own.  Something like:

"If there are multiple users of Guix on this system, you might want to
repeat these commands for each user."

> +You might also want to delete old non-default profiles in
> +/var/guix/gcroots/auto.")

This is a little tricky.  This might make people think they should
delete the links directly from /var/guix/gcroots/auto.  For example, if
you create a profile at /tmp/myprofile via

  guix package --profile=/tmp/myprofile -i coreutils

you will get the following symlink structure:

--8<---------------cut here---------------start------------->8---
[0] marusich@garuda.local:~/guix
$ ls -al /tmp/myprofile*
lrwxrwxrwx 1 marusich users 16 Dec  9 13:56 /tmp/myprofile -> myprofile-1-link
lrwxrwxrwx 1 marusich users 51 Dec  9 13:56 /tmp/myprofile-1-link -> /gnu/store/l4vnx2fpi0b090bg7prjbq5hzlb3skpg-profile
[0] marusich@garuda.local:~/guix
$ ls -al /var/guix/gcroots/auto | grep myprofile
lrwxrwxrwx 1 root root   21 Dec  9 13:56 f2j55vpd15brhsqdwjzshwcxid1g8q9q -> /tmp/myprofile-1-link
[0] marusich@garuda.local:~/guix
$ 
--8<---------------cut here---------------end--------------->8---

If you delete /var/guix/gcroots/auto/f2j55vpd15brhsqdwjzshwcxid1g8q9q,
liveness no longer flows through /tmp/myprofile-1-link because you've
removed the GC root, and the guix-daemon won't know about the profile in
/tmp.  However, the profile is still there!  You can still use coreutils
via /tmp/myprofile-1-link, even though coreutils might get garbage
collected.  That isn't good.

In the case of a non-default profile like /tmp/myprofile, it's safe to
remove /tmp/myprofile and /tmp/myprofile-1-link directly (e.g., via "rm
/tmp/myprofile*").  When guix-daemon next runs GC, it will notice that
/var/guix/gcroots/auto/f2j55vpd15brhsqdwjzshwcxid1g8q9q is now broken,
and it will remove the broken symlink.

Maybe we could change

"You might also want to delete old non-default profiles in
/var/guix/gcroots/auto."

to

"You might also want to delete old non-default profiles pointed to by
the symlinks in /var/guix/gcroots/auto (broken symlinks in this
directory will be automatically removed)."

What do you think?  I know I tend to write more verbose error messages,
but I just don't want to encourage people to shoot themselves in the
foot.  If you can think of a better way to convey the message, that'd be
great.  Maybe we should just say "don't delete the symlinks in the auto
directory"?

-- 
Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: About /var/guix/profiles and guix pull generations

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 856 bytes --]


> I'm not sure what you mean by "Guix checkouts".  Do you mean a Git
> checkout of Guix's Git repository?  If so, why would deleting it help?

Well, I meant what I wrote in the example below

> guix package --profile=~s --delete-generations=1m
> ...
> (string-append (config-directory #:ensure? #f) "/current")

It deletes the old ~/.config/guix/ profiles.  If I'm not wrong, each of those
profiles consume a couple of 100MBs.

So if "checkout" is confusing, what about "Guix copy" instead?

See "(guix) Invoking guix pull" in the manual for terminology.

> "You might also want to delete old non-default profiles pointed to by
> the symlinks in /var/guix/gcroots/auto (broken symlinks in this
> directory will be automatically removed)."

I like it better!  I'll use your suggestion.

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: About /var/guix/profiles and guix pull generations

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 55 bytes --]

Ping?

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: About /var/guix/profiles and guix pull generations

[Chris Marusich]

[-- Attachment #1: Type: text/plain, Size: 1458 bytes --]

Hi Pierre,

Thank you for your patience!

Pierre Neidhardt <mail@ambrevar.xyz> writes:

>> I'm not sure what you mean by "Guix checkouts".  Do you mean a Git
>> checkout of Guix's Git repository?  If so, why would deleting it help?
>
> Well, I meant what I wrote in the example below

OK.  Since that example deletes profile generations, Can we just say
"profile generations" instead of "checkouts"?  The latter makes me think
of a Git repository checkout.  Maybe the phrase "cleaning up old
profiles" would be good enough, since we put a clear example right after
the sentence.  I wouldn't mind either way, as long as we avoid using the
term "checkout" to refer to profiles and their generations.

>> guix package --profile=~s --delete-generations=1m
>> ...
>> (string-append (config-directory #:ensure? #f) "/current")
>
> It deletes the old ~/.config/guix/ profiles.  If I'm not wrong, each of those
> profiles consume a couple of 100MBs.
>
> So if "checkout" is confusing, what about "Guix copy" instead?
>
> See "(guix) Invoking guix pull" in the manual for terminology.
>
>> "You might also want to delete old non-default profiles pointed to by
>> the symlinks in /var/guix/gcroots/auto (broken symlinks in this
>> directory will be automatically removed)."
>
> I like it better!  I'll use your suggestion.

OK!  Could you submit a new patch?

Thank you for helping to improve the "user experience"!

-- 
Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: About /var/guix/profiles and guix pull generations

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 729 bytes --]


> OK.  Since that example deletes profile generations, Can we just say
> "profile generations" instead of "checkouts"?  The latter makes me think
> of a Git repository checkout.  Maybe the phrase "cleaning up old
> profiles" would be good enough, since we put a clear example right after
> the sentence.  I wouldn't mind either way, as long as we avoid using the
> term "checkout" to refer to profiles and their generations.

You might have misunderstood the example.  There are two calls to
=--delete-generations=, which delete _both_ the user profile generations and the
Guix checkout/build/copy generations.

Thanks for spending time on reviewing this!

Cheers!

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: About /var/guix/profiles and guix pull generations

[Chris Marusich]

[-- Attachment #1: Type: text/plain, Size: 1334 bytes --]

Pierre Neidhardt <mail@ambrevar.xyz> writes:

>> OK.  Since that example deletes profile generations, Can we just say
>> "profile generations" instead of "checkouts"?  The latter makes me think
>> of a Git repository checkout.  Maybe the phrase "cleaning up old
>> profiles" would be good enough, since we put a clear example right after
>> the sentence.  I wouldn't mind either way, as long as we avoid using the
>> term "checkout" to refer to profiles and their generations.
>
> You might have misunderstood the example.  There are two calls to
> =--delete-generations=, which delete _both_ the user profile generations and the
> Guix checkout/build/copy generations.

When you say the Guix checkout, you're referring to the contents of
~/.config/guix/current, right?  It's been a little while since I peeked
at that, but I believe that profile and its generations contain not only
the currently installed Guix (in pre-compiled form), but also any
channels the user has installed (also in pre-compiled form).  Since it
is not simply a Git checkout of either Guix or the channels, I'm not
sure that "checkout" is the right term.  But if you still think that
"checkout" is easier to understand, then I would be okay with that.  I
feel like I am already bike-shedding, and I do not want to do that.

-- 
Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: About /var/guix/profiles and guix pull generations

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 132 bytes --]

Agreed "checkout" is not very precise, but can anyone come up with a better
term?

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: About /var/guix/profiles and guix pull generations

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 466 bytes --]


> I think we're still waiting for Pierre to send his latest patch.  If the
> text really is too long, perhaps we can provide concrete examples, a
> sentence explaining what they do, and a link to the manual, where we put
> the slightly more verbose explanation.  I think the text so far is OK,
> but I also have not seen the final version yet.  Pierre, what do you
> think?

I'll re-send the patch just now.

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: About /var/guix/profiles and guix pull generations

[Pierre Neidhardt]

[-- Attachment #1.1: Type: text/plain, Size: 127 bytes --]

Here it is, simply including Chris initial review.
I've sticked to "checkout", unless someone can come up with a better term.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1.2: 0001-guix-scripts-Mention-Guix-checkouts-gcroots-auto-whe.patch --]
[-- Type: text/x-patch, Size: 1607 bytes --]

From c4d5ded8e03e76b67daba2ea6050d7d3f6c9c1d8 Mon Sep 17 00:00:00 2001
From: Pierre Neidhardt <mail@ambrevar.xyz>
Date: Sun, 9 Dec 2018 14:33:17 +0100
Subject: [PATCH] guix scripts: Mention Guix checkouts + gcroots/auto when
 warning about disk space.

* guix/scripts.scm (warn-about-disk-space): Mention Guix checkouts and
gcroots/auto.
---
 guix/scripts.scm | 17 ++++++++++++-----
 1 file changed, 12 insertions(+), 5 deletions(-)

diff --git a/guix/scripts.scm b/guix/scripts.scm
index 5e20ecd92..565a0a65c 100644
--- a/guix/scripts.scm
+++ b/guix/scripts.scm
@@ -212,13 +212,20 @@ available."
                (* ratio 100) (%store-prefix))
       (if profile
           (display-hint (format #f (G_ "Consider deleting old profile
-generations and collecting garbage, along these lines:
+generations, deleting old Guix checkouts and collecting garbage, along these
+lines:
 
 @example
-guix package -p ~s --delete-generations=1m
-guix gc
-@end example\n")
-                                profile))
+guix package --profile=~s --delete-generations=1m
+guix package --profile=~s --delete-generations=1m
+guix gc --free-space=5G
+@end example
+
+You might also want to delete old non-default profiles pointed to by
+the symlinks in /var/guix/gcroots/auto (broken symlinks in this
+directory will be automatically removed).")
+                                profile
+                                (string-append (config-directory #:ensure? #f) "/current")))
           (display-hint (G_ "Consider running @command{guix gc} to free
 space."))))))
 
-- 
2.19.2


[-- Attachment #1.3: Type: text/plain, Size: 48 bytes --]


-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: About /var/guix/profiles and guix pull generations

[swedebugia]

Hi :)

On 2018-12-19 17:49, Pierre Neidhardt wrote:
snip


>             (display-hint (format #f (G_ "Consider deleting old profile
> -generations and collecting garbage, along these lines:
> +generations, deleting old Guix checkouts and collecting garbage, along these
> +lines:
>   
>   @example
> -guix package -p ~s --delete-generations=1m
> -guix gc
> -@end example\n")
> -                                profile))
> +guix package --profile=~s --delete-generations=1m
> +guix package --profile=~s --delete-generations=1m

Why duplicate lines here?

> +guix gc --free-space=5G
> +@end example
> +
> +You might also want to delete old non-default profiles pointed to by
> +the symlinks in /var/guix/gcroots/auto (broken symlinks in this
> +directory will be automatically removed).")
> +                                profile
> +                                (string-append (config-directory #:ensure? #f) "/current")))

I think this is quite long and I don't understand your phrase about 
broken symlinks.

Could you write a new section "Freeing up space in GuixSD" in the manual 
instead and refer to that here instead?

-- 
Cheers Swedebugia

Re: About /var/guix/profiles and guix pull generations

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 1239 bytes --]


> Why duplicate lines here?

They are not duplicated, the arguments are

> +                                profile
> +                                (string-append (config-directory #:ensure? #f) "/current")))

so one is for the user profile, one is for the Guix "checkout".

> : +You might also want to delete old non-default profiles pointed to by
> : +the symlinks in /var/guix/gcroots/auto (broken symlinks in this
> : +directory will be automatically removed).")
> : +                                profile
> : +                                (string-append (config-directory #:ensure? #f) "/current")))
> 
> I think this is quite long and I don't understand your phrase about broken
> symlinks.

In /var/guix/gcroots/auto, you'll find symlinks to custom profiles,
e.g. ~/.my-guix-profile.
If you remove ~/.my-guix-profile, then the symlink in /var/guix/gcroots/auto
will be broken and automatically removed by Guix.  Makes sense?

Can you suggest a better phrasing?

> Could you write a new section "Freeing up space in GuixSD" in the manual instead
> and refer to that here instead?

That could be a good solution indeed.  What do you people think?

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: About /var/guix/profiles and guix pull generations

[swedebugia]

On 2018-12-25 19:49, Pierre Neidhardt wrote:
> 
>> Why duplicate lines here?
> 
> They are not duplicated, the arguments are
> 
>> +                                profile
>> +                                (string-append (config-directory #:ensure? #f) "/current")))
> 
> so one is for the user profile, one is for the Guix "checkout".
> 
>> : +You might also want to delete old non-default profiles pointed to by
>> : +the symlinks in /var/guix/gcroots/auto (broken symlinks in this
>> : +directory will be automatically removed).")
>> : +                                profile
>> : +                                (string-append (config-directory #:ensure? #f) "/current")))
>>
>> I think this is quite long and I don't understand your phrase about broken
>> symlinks.
> 
> In /var/guix/gcroots/auto, you'll find symlinks to custom profiles,
> e.g. ~/.my-guix-profile.
> If you remove ~/.my-guix-profile, then the symlink in /var/guix/gcroots/auto
> will be broken and automatically removed by Guix.  Makes sense?
> 
> Can you suggest a better phrasing?
> 
>> Could you write a new section "Freeing up space in GuixSD" in the manual instead
>> and refer to that here instead?
> 
> That could be a good solution indeed.  What do you people think?

I already sent a patch for the manual on Marks suggestion regarding free 
space. See 
https://lists.gnu.org/archive/html/guix-patches/2018-11/msg00243.html

You are welcome to adopt it if you think it can make it into the manual 
that way. At that time I did not know about these stale checkouts you 
are referring to. I think guix should generally clean up after itself 
and be very clear about what factors cause it to consume space.

-- 
Cheers Swedebugia

Re: About /var/guix/profiles and guix pull generations

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 273 bytes --]

I'm realizing that this patch was forgotten and never merged :)

I think we can edit it to include

guix pull --delete-generations

and

guix system delete-generations

I'll resend a patch to guix-patches@gnu.org.

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

GC hints

[Ludovic Courtès]

Hi!

Pierre Neidhardt <mail@ambrevar.xyz> skribis:

>        (if profile
>            (display-hint (format #f (G_ "Consider deleting old profile
> -generations and collecting garbage, along these lines:
> +generations, deleting old Guix checkouts and collecting garbage, along these
> +lines:
>  
>  @example
> -guix package -p ~s --delete-generations=1m
> -guix gc
> -@end example\n")
> -                                profile))
> +guix package --profile=~s --delete-generations=1m
> +guix package --profile=~s --delete-generations=1m
> +guix gc --free-space=5G
> +@end example
> +
> +You might also want to delete old non-default profiles in
> +/var/guix/gcroots/auto.")
> +                                profile
> +                                (string-append (config-directory #:ensure? #f) "/current")))

I agree that this would be more accurate, but isn’t there also a risk
with making hints too long?  I’m split between the idea of providing
concise hints that can be pasted and get things done, and the idea or
not trying to be a substitute for the manual.

Actually, I was also wondering whether we should provide a configurable
mechanism that would, by default, automatically delete old GC roots and
maybe even run the GC automatically when needed—similar to what Git
does.

Thoughts?

Ludo’.

Re: GC hints

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 777 bytes --]


> I agree that this would be more accurate, but isn’t there also a risk
> with making hints too long?  I’m split between the idea of providing
> concise hints that can be pasted and get things done, and the idea or
> not trying to be a substitute for the manual.

A middle ground between long contextual messages and the manual would be to
print out a link to the manual! :)

> Actually, I was also wondering whether we should provide a configurable
> mechanism that would, by default, automatically delete old GC roots and
> maybe even run the GC automatically when needed—similar to what Git
> does.

But can Guix delete GC roots automatically at all?  What if the user wants to
keep those old profiles?

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: GC hints

[Ricardo Wurmus]

Ludovic Courtès <ludo@gnu.org> writes:

> Actually, I was also wondering whether we should provide a configurable
> mechanism that would, by default, automatically delete old GC roots and
> maybe even run the GC automatically when needed—similar to what Git
> does.

While this would be convenient and probably an improvement I’d like to
mention that in the HPC use case this might be problematic as profiles
might be located on file shares that are not accessible by the node
running the daemon.  It would be good if the GC would err on the side of
caution in those cases.  (It currently removes links to unreachable
profiles.)

--
Ricardo

Re: GC hints

[Chris Marusich]

[-- Attachment #1: Type: text/plain, Size: 1801 bytes --]

Ludovic Courtès <ludo@gnu.org> writes:

> I agree that this would be more accurate, but isn’t there also a risk
> with making hints too long?  I’m split between the idea of providing
> concise hints that can be pasted and get things done, and the idea or
> not trying to be a substitute for the manual.

I agree the manual should be authoritative, but I also think that if
Guix can provide contextual information, it's great when it does so.
The "did you forget to add (use-module..." hints you added a while back
are a great example of this.  The shorter the better, of course.

I think we're still waiting for Pierre to send his latest patch.  If the
text really is too long, perhaps we can provide concrete examples, a
sentence explaining what they do, and a link to the manual, where we put
the slightly more verbose explanation.  I think the text so far is OK,
but I also have not seen the final version yet.  Pierre, what do you
think?

> Actually, I was also wondering whether we should provide a
>configurable
> mechanism that would, by default, automatically delete old GC roots and
> maybe even run the GC automatically when needed—similar to what Git
> does.

I think it's a great idea to allow a user or system administrator to
configure a policy for automatic GC root clean-up.  I'm not sure about
ad-hoc roots, but for profiles, it would be great to be able to say
something like "Keep the last 2 generations, and delete any others that
are older than 2 weeks".  Perhaps we could enforce the policy any time
that a user runs code that modifies the profiles, such as "guix package"
commands.

Of course, that's a little more than what is being proposed in this
email thread, so it would probably be best to work on that separately.

-- 
Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: GC hints

[Mark H Weaver]

Hi Ludovic,

Ludovic Courtès <ludo@gnu.org> writes:

> Actually, I was also wondering whether we should provide a configurable
> mechanism that would, by default, automatically delete old GC roots and
> maybe even run the GC automatically when needed—similar to what Git
> does.
>
> Thoughts?

I think it's reasonable to automatically run GC by default, but I would
strongly advise against deleting GC roots automatically by default
without the user's knowledge and consent.

It's certainly true that git performs GC automatically, but does it
automatically delete GC roots by default?  I've never seen it do this,
and I would be surprised and angry if it did.

I consider Guix GC roots to be potentially valuable user data,
regardless of age.  For example, an old GC root might be valuable
because it was used to perform an experiment that should be repeatable,
or because it is known to work reliably for a given job, and newer
versions have not yet been tested.

I, for one, expect my old profiles, system generations, and other GC
roots to be kept unless I explicitly delete them, and I suspect I'm not
alone.  If I hadn't been paying close attention to Guix development, and
later discovered that Guix had deleted my GC roots without my consent, I
would be surprised and angry.

For these reasons, I think we should not automatically delete GC roots
without the user's knowledge and consent.  User consent could be
inferred by a configuration change, or perhaps obtained interactively by
printing a warning and asking the user to give their explicit consent.

What do you think?

    Thanks,
      Mark

Re: GC hints

[Alex Kost]

Mark H Weaver (2018-12-20 07:12 -0500) wrote:

> Hi Ludovic,
>
> Ludovic Courtès <ludo@gnu.org> writes:
>
>> Actually, I was also wondering whether we should provide a configurable
>> mechanism that would, by default, automatically delete old GC roots and
>> maybe even run the GC automatically when needed—similar to what Git
>> does.
>>
>> Thoughts?
>
> I think it's reasonable to automatically run GC by default, but I would
> strongly advise against deleting GC roots automatically by default
> without the user's knowledge and consent.
>
> It's certainly true that git performs GC automatically, but does it
> automatically delete GC roots by default?  I've never seen it do this,
> and I would be surprised and angry if it did.
>
> I consider Guix GC roots to be potentially valuable user data,
> regardless of age.  For example, an old GC root might be valuable
> because it was used to perform an experiment that should be repeatable,
> or because it is known to work reliably for a given job, and newer
> versions have not yet been tested.
>
> I, for one, expect my old profiles, system generations, and other GC
> roots to be kept unless I explicitly delete them, and I suspect I'm not
> alone.  If I hadn't been paying close attention to Guix development, and
> later discovered that Guix had deleted my GC roots without my consent, I
> would be surprised and angry.

You are not alone!  I completely agree with your points.

-- 
Alex

Re: GC hints

[Chris Marusich]

[-- Attachment #1: Type: text/plain, Size: 1056 bytes --]

Mark H Weaver <mhw@netris.org> writes:

> Hi Ludovic,
>
> Ludovic Courtès <ludo@gnu.org> writes:
>
>> Actually, I was also wondering whether we should provide a configurable
>> mechanism that would, by default, automatically delete old GC roots and
>> maybe even run the GC automatically when needed—similar to what Git
>> does.
>>
>> Thoughts?
>
> I think it's reasonable to automatically run GC by default, but I would
> strongly advise against deleting GC roots automatically by default
> without the user's knowledge and consent.

Just to be clear, I agree with you, Mark.  Guix shouldn't delete GC
roots automatically by default.  I think we were just saying that it
might be nice if a user could configure Guix to automatically delete GC
roots according to some policy (e.g., retain the last 2, and delete any
others older than 1 month).  Guix would only delete the GC roots
according to the policy that the user has set, and if no policy has been
set, the default would be not to delete any of the GC roots.

-- 
Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: GC hints

[swedebugia]

[-- Attachment #1.1: Type: text/html, Size: 1867 bytes --]

[-- Attachment #1.2: Type: text/plain, Size: 1340 bytes --]

Chris Marusich <cmmarusich@gmail.com> skrev: (21 december 2018 09:30:30 CET)
>Mark H Weaver <mhw@netris.org> writes:
>
>> Hi Ludovic,
>>
>> Ludovic Courtès <ludo@gnu.org> writes:
>>
>>> Actually, I was also wondering whether we should provide a
>configurable
>>> mechanism that would, by default, automatically delete old GC roots
>and
>>> maybe even run the GC automatically when needed—similar to what Git
>>> does.
>>>
>>> Thoughts?
>>
>> I think it's reasonable to automatically run GC by default, but I
>would
>> strongly advise against deleting GC roots automatically by default
>> without the user's knowledge and consent.
>
>Just to be clear, I agree with you, Mark.  Guix shouldn't delete GC
>roots automatically by default.  I think we were just saying that it
>might be nice if a user could configure Guix to automatically delete GC
>roots according to some policy (e.g., retain the last 2, and delete any
>others older than 1 month).  Guix would only delete the GC roots
>according to the policy that the user has set, and if no policy has
>been
>set, the default would be not to delete any of the GC roots.
>
>-- 
>Chris

I agree with you all 😃
This should not be the default, but be included in all the config templates with a clarifying comment.
-- 
Sent from my p≡p for Android.

[-- Attachment #2: pEpkey.asc --]
[-- Type: application/pgp-keys, Size: 3825 bytes --]

Re: GC hints

[Mark H Weaver]

Chris Marusich <cmmarusich@gmail.com> writes:

> Mark H Weaver <mhw@netris.org> writes:
>
>> Hi Ludovic,
>>
>> Ludovic Courtès <ludo@gnu.org> writes:
>>
>>> Actually, I was also wondering whether we should provide a configurable
>>> mechanism that would, by default, automatically delete old GC roots and
>>> maybe even run the GC automatically when needed—similar to what Git
>>> does.
>>>
>>> Thoughts?
>>
>> I think it's reasonable to automatically run GC by default, but I would
>> strongly advise against deleting GC roots automatically by default
>> without the user's knowledge and consent.
>
> Just to be clear, I agree with you, Mark.  Guix shouldn't delete GC
> roots automatically by default.  I think we were just saying that it
> might be nice if a user could configure Guix to automatically delete GC
> roots according to some policy (e.g., retain the last 2, and delete any
> others older than 1 month).  Guix would only delete the GC roots
> according to the policy that the user has set, and if no policy has been
> set, the default would be not to delete any of the GC roots.

As long as it's not the default behavior, I think this would be a nice
feature to have.

       Thanks!
         Mark

Re: GC hints

[swedebugia]

On 2018-12-23 16:58, Mark H Weaver wrote:
> Chris Marusich <cmmarusich@gmail.com> writes:
> 
>> Mark H Weaver <mhw@netris.org> writes:
>>
>>> Hi Ludovic,
>>>
>>> Ludovic Courtès <ludo@gnu.org> writes:
>>>
>>>> Actually, I was also wondering whether we should provide a configurable
>>>> mechanism that would, by default, automatically delete old GC roots and
>>>> maybe even run the GC automatically when needed—similar to what Git
>>>> does.
>>>>
>>>> Thoughts?
>>>
>>> I think it's reasonable to automatically run GC by default, but I would
>>> strongly advise against deleting GC roots automatically by default
>>> without the user's knowledge and consent.
>>
>> Just to be clear, I agree with you, Mark.  Guix shouldn't delete GC
>> roots automatically by default.  I think we were just saying that it
>> might be nice if a user could configure Guix to automatically delete GC
>> roots according to some policy (e.g., retain the last 2, and delete any
>> others older than 1 month).  Guix would only delete the GC roots
>> according to the policy that the user has set, and if no policy has been
>> set, the default would be not to delete any of the GC roots.
> 
> As long as it's not the default behavior, I think this would be a nice
> feature to have.
> 
>         Thanks!
>           Mark
> 

Pierre, could you send an updated patch?

-- 
Cheers Swedebugia

Re: GC hints

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 146 bytes --]

I've sent it already (December 19).

I'll apply it in the coming days if no one objects to it.

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: GC hints

[Ludovic Courtès]

Hi,

Mark H Weaver <mhw@netris.org> skribis:

> Chris Marusich <cmmarusich@gmail.com> writes:
>
>> Mark H Weaver <mhw@netris.org> writes:
>>
>>> Hi Ludovic,
>>>
>>> Ludovic Courtès <ludo@gnu.org> writes:
>>>
>>>> Actually, I was also wondering whether we should provide a configurable
>>>> mechanism that would, by default, automatically delete old GC roots and
>>>> maybe even run the GC automatically when needed—similar to what Git
>>>> does.
>>>>
>>>> Thoughts?
>>>
>>> I think it's reasonable to automatically run GC by default, but I would
>>> strongly advise against deleting GC roots automatically by default
>>> without the user's knowledge and consent.
>>
>> Just to be clear, I agree with you, Mark.  Guix shouldn't delete GC
>> roots automatically by default.  I think we were just saying that it
>> might be nice if a user could configure Guix to automatically delete GC
>> roots according to some policy (e.g., retain the last 2, and delete any
>> others older than 1 month).  Guix would only delete the GC roots
>> according to the policy that the user has set, and if no policy has been
>> set, the default would be not to delete any of the GC roots.
>
> As long as it's not the default behavior, I think this would be a nice
> feature to have.

Yes, that would sound like a reasonable default to me.

Thinking more about it, we could imagine the equivalent of Git packs:
for old generations, we’d remove the GC root itself, but we’d store
metadata that would allow us to rebuild the thing.

That’s not always possible in general, but in some cases it is: for
‘guix pull’ profiles, we’d just need to store the channel commits and
URLs, and that’s all it’d take to rebuild them.  We could arrange for
‘guix pull -l’ to traverse these seamlessly.

For ~/.guix-profile we could perhaps keep a GC root to the profile’s
derivation but not the profile itself.  That way you could still rebuild
things afterwards.

Anyway, that’s very much science fiction at this point.

Ludo’.