bug#53224: Cookbook recipe about profile collisions

bug#53224: Cookbook recipe about profile collisions

[Leo Famulari]

Recently, Matt pointed out that profile collisions can be confusing and
difficult to resolve:

https://lists.gnu.org/archive/html/guix-devel/2022-01/msg00115.html

Specifically see this blog post:
http://excalamus.com/2021-10-06-guix-debug.html

I think we should add a Cookbook chapter on the subject.

This particular error state seems unavoidable given the size of the
distro and the number of packages that use propagation.

So, let's help users understand it and give them the knowledge to even
help resolve such collisions as contributors.

Speaking as somebody who understands intimately what a profile is, what
propagated inputs are, and how they can interact to cause "profile
contains conflicting entries", I am sympathetic.

Although I know exactly how to resolve such errors, I have also
explained the situation to new Guix users several times on IRC.

Many people come to Guix, install some packages, use them, then want to
upgrade them, and they have never learned what a profile is or how it is
the fundamental mechanism by which Guix implements package management.

It also seems to me that plenty of people would prefer not to ask for
help, but rather go searching online and in the documentation, which
does not include the string "contains conflicting entries".

I envision a Cookbook chapter that explains what a profile is, what
propagation does, what profile generations are, and then how they all
combine to cause this error state. And an example of a resolution,
whether by adjusting a package definition, uninstalling a package, etc.

There are some older Guix presentation videos that I think do a great
job of explaining profiles and generations, including slides with good
illustrations of the subject. It would be nice to try using that style
of illustration, as I found it clear and illuminating when I first
learned about Guix.

bug#53224: Cookbook recipe about profile collisions

[Ludovic Courtès]

Hi,

Leo Famulari <leo@famulari.name> skribis:

> Recently, Matt pointed out that profile collisions can be confusing and
> difficult to resolve:
>
> https://lists.gnu.org/archive/html/guix-devel/2022-01/msg00115.html

I don’t see the words “profile” and “collision” here; maybe that was
upthread?

> Specifically see this blog post:
> http://excalamus.com/2021-10-06-guix-debug.html
>
> I think we should add a Cookbook chapter on the subject.
>
> This particular error state seems unavoidable given the size of the
> distro and the number of packages that use propagation.
>
> So, let's help users understand it and give them the knowledge to even
> help resolve such collisions as contributors.
>
> Speaking as somebody who understands intimately what a profile is, what
> propagated inputs are, and how they can interact to cause "profile
> contains conflicting entries", I am sympathetic.
>
> Although I know exactly how to resolve such errors, I have also
> explained the situation to new Guix users several times on IRC.

Yeah, we need to get a good understanding of when that happens, and
what’s confusing.

Currently, on profile collisions, the error message shows where the
collision originates and a hint on how to work around it.  Perhaps the
hint is sometimes wrong (in which cases?), or perhaps it’s too terse?
Can it be improved?

> Many people come to Guix, install some packages, use them, then want to
> upgrade them, and they have never learned what a profile is or how it is
> the fundamental mechanism by which Guix implements package management.
>
> It also seems to me that plenty of people would prefer not to ask for
> help, but rather go searching online and in the documentation, which
> does not include the string "contains conflicting entries".
>
> I envision a Cookbook chapter that explains what a profile is, what
> propagation does, what profile generations are, and then how they all
> combine to cause this error state. And an example of a resolution,
> whether by adjusting a package definition, uninstalling a package, etc.
>
> There are some older Guix presentation videos that I think do a great
> job of explaining profiles and generations, including slides with good
> illustrations of the subject. It would be nice to try using that style
> of illustration, as I found it clear and illuminating when I first
> learned about Guix.

The definition of what a profile is is another topic.  Currently the
term “profile” is defined in “Getting Started”:

  https://guix.gnu.org/manual/en/html_node/Getting-Started.html#index-profile

It’s very much defined in passing though.  How can we improve on that?

In some early talks we had illustrations of the symlink forest of a
profile borrowed from Eelco Dolstra’s own talks on the matter; see for
instance p. 17 of <https://guix.gnu.org/guix-fosdem-20140201.pdf>.  I
stopped using it because I think those symlinks are an implementation
detail and it doesn’t help to focus on symlinks (and hashes and all
that) when giving an overview of the tool.  Now, perhaps that
illustration could be useful in the manual.

WDYT?

Thanks,
Ludo’.

bug#53224: Cookbook recipe about profile collisions

[zimoun]

Hi,

On Fri, 14 Jan 2022 at 10:02, Ludovic Courtès <ludo@gnu.org> wrote:

> Currently, on profile collisions, the error message shows where the
> collision originates and a hint on how to work around it.  Perhaps the
> hint is sometimes wrong (in which cases?), or perhaps it’s too terse?
> Can it be improved?

I do not know how it could be improved since it is often case per
case.  For instance, recently we had a collision for python-numpy and
the fix was to introduce python-numpy-next.  That's said, to me the
origin of the collision is often enough for resolving.  However,
indeed the user must have a clear understanding of Guix profile and
propagation.  Therefore, an entry to the Cookbook explaining what does
it mean and where to look appears to me a good idea.


> The definition of what a profile is is another topic.  Currently the
> term “profile” is defined in “Getting Started”:
>
>   https://guix.gnu.org/manual/en/html_node/Getting-Started.html#index-profile
>
> It’s very much defined in passing though.  How can we improve on that?
>
> In some early talks we had illustrations of the symlink forest of a
> profile borrowed from Eelco Dolstra’s own talks on the matter; see for
> instance p. 17 of <https://guix.gnu.org/guix-fosdem-20140201.pdf>.  I
> stopped using it because I think those symlinks are an implementation
> detail and it doesn’t help to focus on symlinks (and hashes and all
> that) when giving an overview of the tool.  Now, perhaps that
> illustration could be useful in the manual.

Well, I do not want to drift the initial message.  I am just keeping
the rolling ball. :-)

In the tutorial I am preparing for JRES, I am doing this way, as a
dialogue (below).  Basically, it is how I am somehow explaining to the
few users I have here, with various background.  This cannot go to the
manual but maybe it could be useful or worth for the Cookbook.

To me, what is an implementation detail is all the dance with /var/,
the generations, the garbage collection, etc.  But the fact that a
profile is symlink pointing a directory with the structure of /usr/
and containing all the environment variables is not an implementation
detail.  It appears to me a core concept to switch from traditional
distro with only one /usr/ to Guix and how to compose many /usr/.

Alice: Where are my packages?
Bob: Installing a package ends with this hint ...snippet...
Alice: Ah, I am confused
Bob: Well, $HOME/.guix-profile is thus the default profile
Alice: What does it mean?
Bob: Look at guix package --search-paths -p $HOME/.guix-profile
Alice: I see, but what is this $HOME/.guix-profile
Bob: Look at: file $HOME/.guix-profile
Alice: Ah, right.  My conclusion is:
  - a profile contains the environment variables
  - a profile is a symlink
Bob: Correct.
Alice: But a symlink pointing to where?
Bob: Look at: readlink -f $HOME/.guix-profile
Alice: It points to a store item, but what does it mean?
Bob: Look at: file $(readlink -f $HOME/.guix-profile)
Alice: Ah right.  My conclusion is:
  - a profile points to a directory item in the store
Bob: Correct.
Alice: What is inside this directory
Bob: Look at: ls $(readlink -f $HOME/.guix-profile)
Alice: Oh, it looks the same hierarchy as the usual and well-know /usr/
Bob: Correct.
Alice: What does it happen when I type python3 I just installed
Bob: Look at: which python3
Alice: Ah right.  Is the binary really in the directory item above?
Bob: Look at: readlink $(which python3)
Alice: Oh right.  A profile points to store item which points to other
store items.
Bob: Correct.
Alice: I guess these other store items have also an /usr/ hierarchy, right?
Bob: Correct.  Do you know how to check that?
Alice: ls $(readlink $(which python3))
Bob: Correct.
Alice: I summarize.  A profile is:
  - a symlink
  - which points to a store item
  - which points to many other store items
  - and all have the structure hierarchy of /usr/
Woo, it is a forest of symlinks!
... image ...


Cheers,
simon

bug#53224: Cookbook recipe about profile collisions

[Leo Famulari]

On Fri, Jan 14, 2022 at 09:47:52AM +0100, Ludovic Courtès wrote:
> > Recently, Matt pointed out that profile collisions can be confusing and
> > difficult to resolve:
> >
> > https://lists.gnu.org/archive/html/guix-devel/2022-01/msg00115.html
> 
> I don’t see the words “profile” and “collision” here; maybe that was
> upthread?

I pointed to this blog post, which was linked from that email:

> > Specifically see this blog post:
> > http://excalamus.com/2021-10-06-guix-debug.html

It's the story of an attempt to learn what the "conflicting entries"
message means and how to resolve it.

> Currently, on profile collisions, the error message shows where the
> collision originates and a hint on how to work around it.  Perhaps the
> hint is sometimes wrong (in which cases?), or perhaps it’s too terse?
> Can it be improved?

The hint is perfectly clear, when you understand "profiles" and
"propagation".

It's like I said later in my message: People use Guix without knowing
very much about it.

That's surprising to me, because I started using Guix *because* I
learned about profiles and how they are used: it's one of the core
innovations of Guix / Nix, to implement unprivileged package management.

But nevertheless, people are using Guix without understanding how it
works. And it's hard to learn about Guix when you are dealing with a
profile collision that you don't understand at all. For many people,
that is not a good moment to start learning. On top of that, profile
collisions are an error state that we can't fix as a bug: we have to
give users the knowledge to resolve the collisions themselves.

For example, the blog post that I linked to ends with "Lessons Learned",
which includes this:

------
What is a profile?

A profile is a directory of symlinks located at ~/.guix-profile.

Propagated inputs can cause conflicts

Propagated inputs are treated somehow differently. It's still not 100%
clear how or why, but it's good to know they're a potential source of
errors.
------

This person spent a lot of time trying to understand the situation and
writing the blog post, but their understanding is still rather weak.

That's why I propose a Cookbook chapter that specifically addresses
profile collisions, to help new users go from "oh no, an error message"
to "aha!"

> The definition of what a profile is is another topic.  Currently the
> term “profile” is defined in “Getting Started”:
> 
>   https://guix.gnu.org/manual/en/html_node/Getting-Started.html#index-profile
> 
> It’s very much defined in passing though.  How can we improve on that?

I think this part of the manual is fine and can stay as it is.

> In some early talks we had illustrations of the symlink forest of a
> profile borrowed from Eelco Dolstra’s own talks on the matter; see for
> instance p. 17 of <https://guix.gnu.org/guix-fosdem-20140201.pdf>.  I
> stopped using it because I think those symlinks are an implementation
> detail and it doesn’t help to focus on symlinks (and hashes and all
> that) when giving an overview of the tool.  Now, perhaps that
> illustration could be useful in the manual.
> 
> WDYT?

The illustrations of the symlink forest were *extremely* helpful to me
when learning how Guix implements unprivileged package management. I
think that later talks from the 2015-2017 era refined the illustrations
to be even more clear.

I'm sorry if the use case for my proposal is still unclear. I can work
on the Cookbook chapter myself.

bug#53224: Cookbook recipe about profile collisions

[Ludovic Courtès]

Hi Leo,

Leo Famulari <leo@famulari.name> skribis:

> I pointed to this blog post, which was linked from that email:
>
>> > Specifically see this blog post:
>> > http://excalamus.com/2021-10-06-guix-debug.html
>
> It's the story of an attempt to learn what the "conflicting entries"
> message means and how to resolve it.

Oh sorry, I had overlooked this.  It makes perfect sense now, and this
blog post is useful in understand what it feels like to encounter that
error message for the first time.

>> Currently, on profile collisions, the error message shows where the
>> collision originates and a hint on how to work around it.  Perhaps the
>> hint is sometimes wrong (in which cases?), or perhaps it’s too terse?
>> Can it be improved?
>
> The hint is perfectly clear, when you understand "profiles" and
> "propagation".
>
> It's like I said later in my message: People use Guix without knowing
> very much about it.
>
> That's surprising to me, because I started using Guix *because* I
> learned about profiles and how they are used: it's one of the core
> innovations of Guix / Nix, to implement unprivileged package management.
>
> But nevertheless, people are using Guix without understanding how it
> works. And it's hard to learn about Guix when you are dealing with a
> profile collision that you don't understand at all. For many people,
> that is not a good moment to start learning. On top of that, profile
> collisions are an error state that we can't fix as a bug: we have to
> give users the knowledge to resolve the collisions themselves.

Yeah.  So, given that “profile” is already defined in “Getting Started”,
I think we can’t do a lot more in the manual.

However, as a way of helping users incrementally discover concepts they
need to understand, we could use one-time hints (sorta like the “Did you
know?” boxes in GUIs.)  ‘guix shell’ already uses a one-time hint
inviting users to run ‘--check’.

We could imagine that the first run of ‘guix install’ & co. would print
a message explaining that your packages’ files are now in
~/.guix-profile, and that it’s what we call a “profile”.

Would that make sense?

> For example, the blog post that I linked to ends with "Lessons Learned",
> which includes this:
>
> ------
> What is a profile?
>
> A profile is a directory of symlinks located at ~/.guix-profile.
>
> Propagated inputs can cause conflicts
>
> Propagated inputs are treated somehow differently. It's still not 100%
> clear how or why, but it's good to know they're a potential source of
> errors.
> ------
>
> This person spent a lot of time trying to understand the situation and
> writing the blog post, but their understanding is still rather weak.

To be fair, the person didn’t look for “profile” in the manual.  I’m not
blaming—mentioning it to clarify what it is we’re trying to fix.

> That's why I propose a Cookbook chapter that specifically addresses
> profile collisions, to help new users go from "oh no, an error message"
> to "aha!"

Makes sense!

>> In some early talks we had illustrations of the symlink forest of a
>> profile borrowed from Eelco Dolstra’s own talks on the matter; see for
>> instance p. 17 of <https://guix.gnu.org/guix-fosdem-20140201.pdf>.  I

[...]

> The illustrations of the symlink forest were *extremely* helpful to me
> when learning how Guix implements unprivileged package management. I
> think that later talks from the 2015-2017 era refined the illustrations
> to be even more clear.

OK, let’s see if we can include an illustration like that under
“Managing Software the Guix Way” or something like that.

> I'm sorry if the use case for my proposal is still unclear. I can work
> on the Cookbook chapter myself.

Great, thanks.

Ludo’.