Uninstalling a "unknown" package installs new stuff -> totally unexpected

Uninstalling a "unknown" package installs new stuff -> totally unexpected

[Hartmut Goebel]

Hi,

TL;DR: Removing packages makes to guix behave *totally* unexpected. guix
should at least output some explanation to reduce the confusion.

I've build a vm-image using my development environment ("./pre-inst-env
guix system vm-image …"). Now within the VM I have

- guix 0.13.0-8.357ab93,

- kdelibs4support 5.39.0,

- plasma-nm 5.11.2.

The installed guix differs from the one used to build the vm-image
(modified commit 9ae29fc0e): The later defines some version of guix
(0.13.0-8.357ab93) which will go into the vm-image. This means, the
installed guix does not know the new package plasma-nm and for
kdelibs4support is known 5.37 (while 5.39 is installed).

Now when I try to uninstall plasma-nm (within the vm), guix starts
downloading stuff. Esp. it starts with "binutils-bootstrap-0".

# guix package --profile=/run/current-system/profile --remove plasma-nm

I would have expected guix to simply remove the links for the package
from the profile. Even for me a an somewhat experienced user this is
*totally* unexpected. IMHO is crazy, since the system is running already
and FMPOV there is *nothing* todo but removing some links from the profile.

I would have had some (even little) acceptance for this behavior, *if*
I'd removed kdelibs4support, since kdelibs4support has some other
referrers, which *might* have some other effects. But for plasma-nm,
this is unexpected.

Whatever is happening here, guix should at least output some explanation
to reduce the confusion.

# guix gc --referrers /gnu/store/…-plasma-nm-5.11.2
/gnu/store/…-profile
/gnu/store/…-plasma-nm-5.11.2

# guix gc --referrers /gnu/store/…-kdelibs4support-5.39.0/
/gnu/store/…-plasma-desktop-5.11.2
/gnu/store/…-kdelibs4support-5.39.0
/gnu/store/…-profile
/gnu/store/…-khotkeys-5.11.2
/gnu/store/…-kdeplasma-addons-5.11.2
/gnu/store/…-plasma-nm-5.11.2
/gnu/store/…-kmenuedit-5.11.2
/gnu/store/…-plasma-workspace-5.11.2

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |

Re: Uninstalling a "unknown" package installs new stuff -> totally unexpected

[Chris Marusich]

[-- Attachment #1: Type: text/plain, Size: 6681 bytes --]

Hartmut Goebel <h.goebel@crazy-compilers.com> writes:

> I've build a vm-image using my development environment ("./pre-inst-env
> guix system vm-image …"). Now within the VM I have
>
> - guix 0.13.0-8.357ab93,
>
> - kdelibs4support 5.39.0,
>
> - plasma-nm 5.11.2.
>
> The installed guix differs from the one used to build the vm-image
> (modified commit 9ae29fc0e): The later defines some version of guix
> (0.13.0-8.357ab93) which will go into the vm-image. This means, the
> installed guix does not know the new package plasma-nm and for
> kdelibs4support is known 5.37 (while 5.39 is installed).

Indeed, this is true.  You might find the following short email thread
interesting - it explains why this happens:

https://lists.gnu.org/archive/html/help-guix/2017-01/msg00010.html

Maybe you can work around this by using the "current-guix" variable that
Ludo mentioned?  Although, also note that he mentions it's a bit of a
hack.

Anything that makes the version of Guix in the VM "closer" to the
version of Guix outside the VM will probably help reduce build times.

> Now when I try to uninstall plasma-nm (within the vm), guix starts
> downloading stuff. Esp. it starts with "binutils-bootstrap-0".
>
> # guix package --profile=/run/current-system/profile --remove plasma-nm

Yes, that happens because, as you know, the version of Guix installed in
the VM (call it G_vm) is older than the version of Guix that was used to
build the VM image (call it G_host).  I wonder if using "current-guix",
as mentioned above, would cause G_vm and G_host to be identical.  If so,
that might fix your problem.  In theory, anything that allows G_vm to
approximate G_host (e.g., running "guix pull" in the VM) will probably
help make the build go faster.

It is surprising that G_vm is older than G_host, but it is not
surprising that, given that G_vm is older than G_host, G_vm will need to
build many things in order to remove a single package.  Suppose you want
to remove a package p_1 from a profile in the VM that contains another
package p_2.  The version of p_2 that is installed in the VM was built
by G_host.  To remove p_1 from a profile means to build a new profile in
which p_1 is absent.  To build a new profile in the VM which lacks p_1,
G_vm must build p_2.  Assuming that the definition of p_2 has not
changed since the last profile was built, and assuming that none of the
definitions of p_2's dependencies has changed, when G_vm builds p_2 for
the new profile, the build will succeed quickly because the output of
p_2 is already a valid path in G_vm's store.  However, if the definition
of p_2 or any of its dependencies has changed, then G_vm will need to
build all of them in order to build p_2 for the new profile.  All of
this is necessary in order to build the next profile generation without
p_1.  The reason that you are seeing so many builds is probably because
either many things changed between G_vm and G_host, or something deep in
the dependency tree changed between G_vm and G_host.  Either way, G_host
did not build all the things that G_vm now needs in order to build a new
profile without p_1, so G_vm must build all of those missing things.
This is a natural result when G_vm is older than G_host.

> I would have expected guix to simply remove the links for the package
> from the profile.

Your expectation is totally understandable.  However, due to the way in
which Guix builds a profile, removing symlinks is not as simple as it
sounds.  When building a profile generation, it is sometimes necessary
to resolve symlink conflicts when two packages both provide the same
file - one file is chosen for inclusion in the profile, and the other is
left out.  How should we "simply remove the link" in this case?  It is
simpler and more functional (in the purely functional sense) to just
build an entirely new profile in which the removed package is not
present to begin with, and that is what Guix does.

It may be technically possible to implement a different profile
generation mechanism which allows the removal of a package by
"un-symlinking" it from a previous profile generation.  However, if we
did that, it would mean that a profile generation might now depend on a
profile generation that came before it.  That is not good because it
would make it harder to reproducibly build any given profile.  It would
be better not to make profile generations depend on the state of the
profile generations that came before it.  Since the problem here is that
it took much longer to build the new profile than you expected because
G_vm was older than G_host, I suspect that we can resolve the problem by
taking steps to bring G_vm "closer" to G_host, such as using
"current-guix" as mentioned above, or running "guix pull" in the VM.

> Whatever is happening here, guix should at least output some explanation
> to reduce the confusion.

Guix does say it's going to build a ton of things (I think).  But right
after that, Guix spits out tons of build output.  I rarely get a chance
to inspect it closely.  And once the build begins, it's really hard to
tell what's going on or how long it will last.  I have often wished for
a higher level view of what's going on.  What sort of changes do you
think will help here?

I personally think it would be nice if I could somehow see the number of
derivations that have been built so far, as a percentage of the total
derivations that need to be built.  I'm not sure if that's possible with
the way the guix-daemon schedules builds, but it would be nicer than a
stream of build output for 8 hours (or longer...).  Something like this
(the numbers don't really add up, but hopefully it conveys the idea):

Successfully built derivations: 37
Failed derivations: 2
Derivations waiting to be built: 33
Total size of substitutable derivations: 1.7 GiB
Currently building 3 derivations:
    37 seconds elapsed: /gnu/store/...-foo.drv
    1 minute 7 seconds elapsed: /gnu/store/...-bar.drv
    9 seconds elapsed: /gnu/store/...-baz.drv
Time elapsed (total): 2 hours 2 minutes 27 seconds
Average time per derivation: 7 minutes 41 seconds
Estimated time remaining: 4 hours 11 minutes 40 seconds

If a derivation fails, we can get the log with "guix build --log-file",
so I'm not sure why we need to spew output to the terminal during the
build by default.  When you're hacking on a package or something, it's
convenient to see the failure output immediately, but when all you want
to do is manage your profile and build failures are not expected, the
build logs are just distracting.

-- 
Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Uninstalling a "unknown" package installs new stuff -> totally unexpected

[Chris Marusich]

[-- Attachment #1: Type: text/plain, Size: 1734 bytes --]

Chris Marusich <cmmarusich@gmail.com> writes:

> It may be technically possible to implement a different profile
> generation mechanism which allows the removal of a package by
> "un-symlinking" it from a previous profile generation.  However, if we
> did that, it would mean that a profile generation might now depend on a
> profile generation that came before it.

I'd like to correct something I said in my last reply.

I just remembered that we have a declarative description of what is
installed in a profile at $PROFILE/manifest.  In my last email, I may
have incorrectly described what happens during the removal of a package.
Judging by the code in guix/profiles.scm, it looks like we actually do
re-use the existing store items that are referenced by the previous
profile generation.  Packages won't be upgraded unless you explicitly
ask for it (e.g., with the "-u" option).  So, when G_vm removes p_1 from
the profile (i.e., when G_vm builds a new profile in which p_1 is not
present), G_vm will in fact re-use the same package versions in the new
generation that were installed in the previous generation.  When I
suggested that the builds you were seeing occurred because G_vm would
want to build newer versions of all the packages, I believe was mistaken
- I apologize for any confusion I may have caused.

That said, I still think it's true that if G_vm is very different than
G_host, it is likely that G_vm will wind up doing a lot of builds for
things that G_host has "already" built (but which aren't installed in
the profile itself).  Guix uses derivations to do a lot of stuff under
the covers, so I think this is likely to occur when G_vm and G_host are
very different.

-- 
Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]