guix gc doesn't seem to clean old guix revision

guix gc doesn't seem to clean old guix revision

[YOANN P]

Hi guix,

I was testing the creation of guix docker image to be able to have multiple versions :

- 1.0.1 ( based on binary installation )
- latest ( based on the last commit from guix git with git pull )
- commit_hash ( specific commit )

And what i observed is it seems that 'guix gc' doesn't clean old guix revision.

I use alpine as a base image.
After installing the 1.0.1 binary, 'docker ps --size' report '0B (virtual 404MB)'
After a 'guix pull' and 'guix gc --delete-generations --optimize',  'docker ps --size' report 1.07GB (virtual 1.47GB)

How to clean old guix revision and let only the current one ?

$ docker exec -ti guix-test sh -c 'guix --version'
guix (GNU Guix) 7b40d59114e1462d6d8140f325a66b12e91db667
Copyright (C) 2019 the Guix authors
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

$ docker exec -ti guix-test sh -c 'guix pull --list-generations'
Generation 1	Nov 19 2019 11:30:28	(current)
  guix 7b40d59
    repository URL: https://git.savannah.gnu.org/git/guix.git
    branch: master
    commit: 7b40d59114e1462d6d8140f325a66b12e91db667

Regards,

RE: guix gc doesn't seem to clean old guix revision

[YOANN P]

After re-reading the doc for 'guix gc', i now know why.

It seems that guix pull doesn't change the symlink in /var/guix/profiles/per-user/root/current-guix and is list in gc roots

/var/guix/profiles/per-user/root/current-guix --> current-guix-1-link --> /gnu/store/rrhs9c8k0vbpirqplfzh2bf1xczr3zan-profile --> version 1.0.1
/var/guix/profiles/default/current-guix --> current-guix-1-link --> /gnu/store/vhfc20q1f7yj63h21sn7bvdpqkr3h25m-profile --> version 7b40d59114e1462d6d8140f325a66b12e91db667

Not sure if it is related but i also observed that guix pull changed my /root/.config/guix/current symlink

Prior to guix pull:
/root/.config/guix/current --> /var/guix/profiles/per-user/root/current-guix
After guix pull:
/root/.config/guix/current --> /var/guix/profiles/default/current-guix

Regards,

RE: guix gc doesn't seem to clean old guix revision

[Julien Lepiller]

Le 20 novembre 2019 13:00:23 GMT+01:00, YOANN P <yoann_mac_donald@hotmail.com> a écrit :
>After re-reading the doc for 'guix gc', i now know why.
>
>It seems that guix pull doesn't change the symlink in
>/var/guix/profiles/per-user/root/current-guix and is list in gc roots
>
>/var/guix/profiles/per-user/root/current-guix --> current-guix-1-link
>--> /gnu/store/rrhs9c8k0vbpirqplfzh2bf1xczr3zan-profile --> version
>1.0.1
>/var/guix/profiles/default/current-guix --> current-guix-1-link -->
>/gnu/store/vhfc20q1f7yj63h21sn7bvdpqkr3h25m-profile --> version
>7b40d59114e1462d6d8140f325a66b12e91db667
>
>Not sure if it is related but i also observed that guix pull changed my
>/root/.config/guix/current symlink
>
>Prior to guix pull:
>/root/.config/guix/current -->
>/var/guix/profiles/per-user/root/current-guix
>After guix pull:
>/root/.config/guix/current --> /var/guix/profiles/default/current-guix
>
>Regards,

Is default a username or the value of $USER maybe?

Re: guix gc doesn't seem to clean old guix revision

[John Soo]

Hi,

To delete old generations, use `guix system delete-generations`, for packages, use `guix package` with flag `delete-generations`.  Hope that helps. 

As an aside, your question might find more traction on the help-guix@gnu.org list instead; guix-devel is the development list.

- John

RE: guix gc doesn't seem to clean old guix revision

[YOANN P]

[-- Attachment #1: Type: text/plain, Size: 859 bytes --]

> Is default a username or the value of $USER maybe?

@julien : default is not the name of any user, only root and guix-build users are present under my container

> To delete old generations, use `guix system delete-generations`, for packages, use `guix package` with flag `delete-generations`.  Hope that helps.

Already done "package --delete-generations" but "guix pull" is different than "guix package" and didn't install anything with my root user, only installed guix binary, then guix pull,then guix gc.

> As an aside, your question might find more traction on the help-guix@gnu.org list instead; guix-devel is the development list.

Or maybe guix-bug because the fact that the change of the symlink is not done is the reason for 'gc --list-root' to still have 'guix 1.0.1' in use

Thanks,

Regards

________________________________

[-- Attachment #2: Type: text/html, Size: 3221 bytes --]

Re: guix gc doesn't seem to clean old guix revision

[zimoun]

Hi,


On Tue, 19 Nov 2019 at 13:34, YOANN P <yoann_mac_donald@hotmail.com> wrote:

> I was testing the creation of guix docker image to be able to have multiple versions :

Please could you report here how you generate your Docker image? Using:

  guix pack -f docker

or something else?


Other said, how can I reproduce the behaviour you are observing?


Thank you.

All the best,
simon

RE: guix gc doesn't seem to clean old guix revision

[YOANN P]

> Other said, how can I reproduce the behaviour you are observing?

# Download requirements

$ mkdir -p /tmp/guix-docker
$ wget -O /tmp/guix-docker/Dockerfile https://raw.githubusercontent.com/rockandska/.dotfiles/084b3c6924eeb985cb40edfc90e3827b10ba61e2/.local/opt/dotfiles-bootstrap/tests/Dockerfiles/Dockerfile
$ wget -O /tmp/guix-docker/build.sh https://raw.githubusercontent.com/rockandska/.dotfiles/084b3c6924eeb985cb40edfc90e3827b10ba61e2/.local/opt/dotfiles-bootstrap/tests/Dockerfiles/hooks/build
$ chmod +x /tmp/guix-docker/build.sh
$ cd /tmp/guix-docker

# it will create two docker image : guix:1.0.1 and guix:latest

/tmp/guix-docker $ ./build.sh

# start two container with the guix daemon as main process

/tmp/guix-docker $ docker run -d -ti --name guix-test-1.0.1 guix:1.0.1
/tmp/guix-docker $ docker run -d -ti --name guix-test-latest guix:latest

# install some useful tools

/tmp/guix-docker $ docker exec -i guix-test-1.0.1 apk add util-linux
/tmp/guix-docker $ docker exec -i guix-test-latest apk add util-linux

# compare some things
# sh automatically load /root/.guix-profile/etc/profile and is the reason we don't call guix directly

/tmp/guix-docker $ sdiff <(docker exec -i guix-test-1.0.1 sh <<<"env") <(docker exec -i guix-test-latest sh <<<"env")
/tmp/guix-docker $ sdiff <(docker exec -i guix-test-1.0.1 sh <<<"guix --version") <(docker exec -i guix-test-latest sh <<<"guix --version")
/tmp/guix-docker $ sdiff <(docker exec -i guix-test-1.0.1 sh <<<"guix-daemon --version") <(docker exec -i guix-test-latest sh <<<"guix-daemon --version")
/tmp/guix-docker $ sdiff <(docker exec -i guix-test-1.0.1 sh <<<"namei /root/.config/guix/current") <(docker exec -i guix-test-latest sh <<<"namei /root/.config/guix/current")
/tmp/guix-docker $ sdiff <(docker exec -i guix-test-1.0.1 sh <<<"guix gc --list-roots") <(docker exec -i guix-test-latest sh <<<"guix gc --list-roots")

Regards,

Re: guix gc doesn't seem to clean old guix revision

[Ludovic Courtès]

Hi Yoann,

YOANN P <yoann_mac_donald@hotmail.com> skribis:

> Not sure if it is related but i also observed that guix pull changed my /root/.config/guix/current symlink
>
> Prior to guix pull:
> /root/.config/guix/current --> /var/guix/profiles/per-user/root/current-guix
> After guix pull:
> /root/.config/guix/current --> /var/guix/profiles/default/current-guix

Like Julien, I find it surprising: where does that “default” come from?
Did you mean “/per-user/default” or really “/default”?  If the latter,
something’s really fishy.  :-)

Ludo’.

RE: guix gc doesn't seem to clean old guix revision

[YOANN P]

> Like Julien, I find it surprising: where does that “default” come from?
> Did you mean “/per-user/default” or really “/default”?  If the latter,
> something’s really fishy.  :-)
> Ludo’.

Hi ludo,

I provided a reproducible case with how i arrive at this "weird" state in
a previous email.
But to answer your question, this is the difference after just invoking 
'guix pull'.

$ sdiff -s <(docker exec -i guix-test-1.0.1 sh <<<"namei /root/.config/guix/current") <(docker exec -i guix-test-latest sh <<<"namei /root/.config/guix/current")
 l current -> /var/guix/profiles/per-user/root/current-guix    |	 l current -> /var/guix/profiles/default/current-guix
   d per-user						                            |	   d default
   d root						      <
     l current-guix-1-link -> /gnu/store/rrhs9c8k0vbpirqplfzh  |	     l current-guix-1-link -> /gnu/store/ls30zsp76w3nxkggkpb1
       d rrhs9c8k0vbpirqplfzh2bf1xczr3zan-profile	             |	       d ls30zsp76w3nxkggkpb1s6sdj5v2zzfl-profile

I don't know from where this 'default' come ^^

Regards, 

Re: guix gc doesn't seem to clean old guix revision

[zimoun]

Hi,

> And what i observed is it seems that 'guix gc' doesn't clean old guix revision.

It is because when pulling you do not replace but you create another
profile. Therefore, garbage collect does not apply as you expect. It
appears clearly by running "guix gc --list-roots" on your guix:latest
Docker image.

I was thinking that the size would be only the double instead of the
triple. But who knows. :-)

So now the question is: why does your configuration generate another
profile when "guix pull"ing? I have quickly investigated a bit and I
do not know right now. Maybe something is hidden with Alpine or other.


To be clear, "guix pull" claims the creation of this new profile.
Inside the Docker of the 1.0.1 image, I get:

--8<---------------cut here---------------start------------->8---
which guix
/usr/local/bin/guix

ls -l /usr/local/bin
total 0
lrwxrwxrwx    1 root     root            54 Nov 26 17:58 guix ->
/var/guix/profiles/per-user/root/current-guix/bin/guix

guix pull
Migrating profile generations to '/var/guix/profiles/default'...
--8<---------------cut here---------------end--------------->8---

I was expecting that `which guix` returns /root/.config/guix/current/bin. Hum?


Hope that helps.

All the best,
simon

Re: guix gc doesn't seem to clean old guix revision

[zimoun]

Hi,

On Tue, 26 Nov 2019 at 20:26, zimoun <zimon.toutoune@gmail.com> wrote:

>
> guix pull
> Migrating profile generations to '/var/guix/profiles/default'...
> --8<---------------cut here---------------end--------------->8---

It is similar to Bug#36785 [1].

As said, it is not a bug of "guix pull" but a bug of the
configuration. Adding the root user to your Dockerfile should fix the
issue you encounter.


Hope that helps.

simon

[1] https://debbugs.gnu.org/cgi/bugreport.cgi?bug=36785

RE: guix gc doesn't seem to clean old guix revision

[YOANN P]

> It is similar to Bug#36785 [1].

> As said, it is not a bug of "guix pull" but a bug of the
> configuration. Adding the root user to your Dockerfile should fix the
> issue you encounter.

The fact that guix require $USER to be set is IMO a kind of bug and is
documented anywhere.
It will be less prone to errors to relay for $USER on something like 
`whoami` and for $HOME expanding `~$USER`.

That say, finally, after setting $USER, and after restart the container before
doing `guix gc`, it seems far better: 1.0.1 ( 404Mo ) , latest ( 570Mo ).

So, not sure if it is normal, but it seems there is an overhead of 166Mo 
between the binary version and after doing 'guix pull' and 'guix gc'.

Best regards,

Re: guix gc doesn't seem to clean old guix revision

[zimoun]

Dear,


On Fri, 29 Nov 2019 at 16:14, YOANN P <yoann_mac_donald@hotmail.com> wrote:

> > As said, it is not a bug of "guix pull" but a bug of the
> > configuration. Adding the root user to your Dockerfile should fix the
> > issue you encounter.
>
> The fact that guix require $USER to be set is IMO a kind of bug and is
> documented anywhere.

I do not understand: running without any $USER is kind of very special
edge case.
In your configuration file [1], you create everything with the
(non-existent) root user and you are doing as they exists, i.e, ENV
PATH, "ln -s", etc. then "guix pull" and because there is no user,
Guix creates the default profile.
I do not see where there is a bug. Guix does not break and it is a
misconfiguration.
IMO, the bug comes from your configuration file. The fix is: create an
user or correctly symlink the profile
(/var/guix/profiles/default/current-guix instead of
/var/guix/profiles/per-user/root/current-guix).


However, I totally agree with you that it lacks documentation about
this user case. Where do you suggest to add such information? A
warning note in the binary installation section [2]? Something else?


[1] https://raw.githubusercontent.com/rockandska/.dotfiles/084b3c6924eeb985cb40edfc90e3827b10ba61e2/.local/opt/dotfiles-bootstrap/tests/Dockerfiles/Dockerfile
[2] https://guix.gnu.org/manual/en/html_node/Binary-Installation.html#Binary-Installation


> It will be less prone to errors to relay for $USER on something like
> `whoami` and for $HOME expanding `~$USER`.

What do you mean?
If there is no user, then there is no user. :-)


> So, not sure if it is normal, but it seems there is an overhead of 166Mo
> between the binary version and after doing 'guix pull' and 'guix gc'.

Interesting. It should deserve more investigation why. :-)


All the best,
simon

RE: guix gc doesn't seem to clean old guix revision

[YOANN P]

[-- Attachment #1: Type: text/plain, Size: 2843 bytes --]

> > > As said, it is not a bug of "guix pull" but a bug of the
> > > configuration. Adding the root user to your Dockerfile should fix the
> > > issue you encounter.
> >
> > The fact that guix require $USER to be set is IMO a kind of bug and is
> > documented anywhere.

> I do not understand: running without any $USER is kind of very special
> edge case.

It is not special at all, $USER is only set when using a login shell.
So if you script some guix commands (who doesn't use a login shell),
it broke as it broke with my tests.

> In your configuration file [1], you create everything with the
> (non-existent) root user and you are doing as they exists, i.e, ENV
> PATH, "ln -s", etc. then "guix pull" and because there is no user,
> Guix creates the default profile.
> I do not see where there is a bug. Guix does not break and it is a
> misconfiguration.
> IMO, the bug comes from your configuration file. The fix is: create an
> user or correctly symlink the profile
> (/var/guix/profiles/default/current-guix instead of
> /var/guix/profiles/per-user/root/current-guix).

The user already exists, the only problem I had was that my commands
was running with a non-login login shell, so $USER was not set and had to force the
set of $USER in a wrapper to allow guix to works without linux shell.

> However, I totally agree with you that it lacks documentation about
> this user case. Where do you suggest to add such information? A
> warning note in the binary installation section [2]? Something else?

It is not an installation problem but a usage problem, so every commands
who need $USER/$HOME should have a warning about the need to have
those variables set.

> [1] https://raw.githubusercontent.com/rockandska/.dotfiles/084b3c6924eeb985cb40edfc90e3827b10ba61e2/.local/opt/dotfiles-bootstrap/tests/Dockerfiles/Dockerfile
> [2] https://guix.gnu.org/manual/en/html_node/Binary-Installation.html#Binary-Installation


> > It will be less prone to errors to relay for $USER on something like
> > `whoami` and for $HOME expanding `~$USER`.

> What do you mean?
> If there is no user, then there is no user. :-)

There is a user, this is just that the shell used is not a login shell.
Since $USER/$HOME are only set when using a login shell, it is
better to use `whoami` in place of $USER and expand ~$(whoami)
in place of $HOME in scripts to be sure to have the right values even
when using a non-login shell or to avoid errors if $USER was override.

https://groups.google.com/forum/#!topic/comp.unix.shell/PLZPzNx3F_4

> > So, not sure if it is normal, but it seems there is an overhead of 166Mo
> > between the binary version and after doing 'guix pull' and 'guix gc'.

> Interesting. It should deserve more investigation why. :-)


> All the best,
> simon

Regards

[-- Attachment #2: Type: text/html, Size: 4861 bytes --]

Re: guix gc doesn't seem to clean old guix revision

[zimoun]

Hi,

On Mon, 2 Dec 2019 at 21:55, YOANN P <yoann_mac_donald@hotmail.com> wrote:

> > I do not understand: running without any $USER is kind of very special
> > edge case.
>
> It is not special at all, $USER is only set when using a login shell.
> So if you script some guix commands (who doesn't use a login shell),
> it broke as it broke with my tests.

You have right about $USER. But it is not so common because the case
has not been encountered too much already. ;-)


> There is a user, this is just that the shell used is not a login shell.
> Since $USER/$HOME are only set when using a login shell, it is
> better to use `whoami` in place of $USER and expand ~$(whoami)
> in place of $HOME in scripts to be sure to have the right values even
> when using a non-login shell or to avoid errors if $USER was override.

You are proposing to avoid the call to '(or (getenv "USER") (getenv
"LOGNAME"))' in guix/profiles.scm and guix/store.scm and instead to
rely on the command "whoami" from the coreutils package, right?
Other said, instead of falling in the "default" case in
%profile-directory, call the whoami command, right?

Is whoami exposed in Guile?


Thank you for the explanations.

All the best,
simon

Re: guix gc doesn't seem to clean old guix revision

[Ludovic Courtès]

[-- Attachment #1: Type: text/plain, Size: 477 bytes --]

Hi Yoann,

YOANN P <yoann_mac_donald@hotmail.com> skribis:

>> It is similar to Bug#36785 [1].
>
>> As said, it is not a bug of "guix pull" but a bug of the
>> configuration. Adding the root user to your Dockerfile should fix the
>> issue you encounter.
>
> The fact that guix require $USER to be set is IMO a kind of bug and is
> documented anywhere.

I had not understood that.  I guess the patch below fixes it, I’ll push
it shortly.

Thanks,
Ludo’.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Type: text/x-patch, Size: 642 bytes --]

diff --git a/guix/profiles.scm b/guix/profiles.scm
index f5e5cc33d6..616605151e 100644
--- a/guix/profiles.scm
+++ b/guix/profiles.scm
@@ -1727,7 +1727,9 @@ because the NUMBER is zero.)"
 (define %profile-directory
   (string-append %state-directory "/profiles/"
                  (or (and=> (or (getenv "USER")
-                                (getenv "LOGNAME"))
+                                (getenv "LOGNAME")
+                                (false-if-exception
+                                 (passwd:name (getpwuid (getuid)))))
                             (cut string-append "per-user/" <>))
                      "default")))
 

Re: guix gc doesn't seem to clean old guix revision

[zimoun]

Hi Ludo,

On Tue, 10 Dec 2019 at 17:19, Ludovic Courtès <ludo@gnu.org> wrote:

> YOANN P <yoann_mac_donald@hotmail.com> skribis:

> >> As said, it is not a bug of "guix pull" but a bug of the
> >> configuration. Adding the root user to your Dockerfile should fix the
> >> issue you encounter.
> >
> > The fact that guix require $USER to be set is IMO a kind of bug and is
> > documented anywhere.
>
> I had not understood that.  I guess the patch below fixes it, I’ll push
> it shortly.

Bug or not a bug? that is the question. ;-)

If I understand well, the "default" case will never happen now.
And does the addition not include (getenv "USER") and LOGNAME?


Cheers,
simon

RE: guix gc doesn't seem to clean old guix revision

[YOANN P]

> Hi Yoann,

Hi Ludo,

> I had not understood that.  I guess the patch below fixes it, I’ll push
> t shortly.

> Thanks,
> Ludo’.

Look great, but IMO, your addition, "(passwd:name (getpwuid (getuid))))",
should be the only way to get the username since $USER and $LOGNAME
could be overridden or unset.

I don't always do it in my own scripts, but it is better to do not relay on
env vars when you want something to be bulletproof.

Since there is "passwd:dir pw", $HOME could be replaced with it too.

Thanks,
Yoann

Re: guix gc doesn't seem to clean old guix revision

[Ludovic Courtès]

Hi,

YOANN P <yoann_mac_donald@hotmail.com> skribis:

>> Hi Yoann,
>
> Hi Ludo,
>
>> I had not understood that.  I guess the patch below fixes it, I’ll push
>> t shortly.
>
>> Thanks,
>> Ludo’.
>
> Look great, but IMO, your addition, "(passwd:name (getpwuid (getuid))))",
> should be the only way to get the username since $USER and $LOGNAME
> could be overridden or unset.

It’s precisely because they can be overridden that they should take
precedence.  :-)

Ludo’.