bug#53224: Cookbook recipe about profile collisions

bug#53224: Cookbook recipe about profile collisions

[Leo Famulari]

Recently, Matt pointed out that profile collisions can be confusing and
difficult to resolve:

https://lists.gnu.org/archive/html/guix-devel/2022-01/msg00115.html

Specifically see this blog post:
http://excalamus.com/2021-10-06-guix-debug.html

I think we should add a Cookbook chapter on the subject.

This particular error state seems unavoidable given the size of the
distro and the number of packages that use propagation.

So, let's help users understand it and give them the knowledge to even
help resolve such collisions as contributors.

Speaking as somebody who understands intimately what a profile is, what
propagated inputs are, and how they can interact to cause "profile
contains conflicting entries", I am sympathetic.

Although I know exactly how to resolve such errors, I have also
explained the situation to new Guix users several times on IRC.

Many people come to Guix, install some packages, use them, then want to
upgrade them, and they have never learned what a profile is or how it is
the fundamental mechanism by which Guix implements package management.

It also seems to me that plenty of people would prefer not to ask for
help, but rather go searching online and in the documentation, which
does not include the string "contains conflicting entries".

I envision a Cookbook chapter that explains what a profile is, what
propagation does, what profile generations are, and then how they all
combine to cause this error state. And an example of a resolution,
whether by adjusting a package definition, uninstalling a package, etc.

There are some older Guix presentation videos that I think do a great
job of explaining profiles and generations, including slides with good
illustrations of the subject. It would be nice to try using that style
of illustration, as I found it clear and illuminating when I first
learned about Guix.

bug#53224: Cookbook recipe about profile collisions

[Ludovic Courtès]

Hi,

Leo Famulari <leo@famulari.name> skribis:

> Recently, Matt pointed out that profile collisions can be confusing and
> difficult to resolve:
>
> https://lists.gnu.org/archive/html/guix-devel/2022-01/msg00115.html

I don’t see the words “profile” and “collision” here; maybe that was
upthread?

> Specifically see this blog post:
> http://excalamus.com/2021-10-06-guix-debug.html
>
> I think we should add a Cookbook chapter on the subject.
>
> This particular error state seems unavoidable given the size of the
> distro and the number of packages that use propagation.
>
> So, let's help users understand it and give them the knowledge to even
> help resolve such collisions as contributors.
>
> Speaking as somebody who understands intimately what a profile is, what
> propagated inputs are, and how they can interact to cause "profile
> contains conflicting entries", I am sympathetic.
>
> Although I know exactly how to resolve such errors, I have also
> explained the situation to new Guix users several times on IRC.

Yeah, we need to get a good understanding of when that happens, and
what’s confusing.

Currently, on profile collisions, the error message shows where the
collision originates and a hint on how to work around it.  Perhaps the
hint is sometimes wrong (in which cases?), or perhaps it’s too terse?
Can it be improved?

> Many people come to Guix, install some packages, use them, then want to
> upgrade them, and they have never learned what a profile is or how it is
> the fundamental mechanism by which Guix implements package management.
>
> It also seems to me that plenty of people would prefer not to ask for
> help, but rather go searching online and in the documentation, which
> does not include the string "contains conflicting entries".
>
> I envision a Cookbook chapter that explains what a profile is, what
> propagation does, what profile generations are, and then how they all
> combine to cause this error state. And an example of a resolution,
> whether by adjusting a package definition, uninstalling a package, etc.
>
> There are some older Guix presentation videos that I think do a great
> job of explaining profiles and generations, including slides with good
> illustrations of the subject. It would be nice to try using that style
> of illustration, as I found it clear and illuminating when I first
> learned about Guix.

The definition of what a profile is is another topic.  Currently the
term “profile” is defined in “Getting Started”:

  https://guix.gnu.org/manual/en/html_node/Getting-Started.html#index-profile

It’s very much defined in passing though.  How can we improve on that?

In some early talks we had illustrations of the symlink forest of a
profile borrowed from Eelco Dolstra’s own talks on the matter; see for
instance p. 17 of <https://guix.gnu.org/guix-fosdem-20140201.pdf>.  I
stopped using it because I think those symlinks are an implementation
detail and it doesn’t help to focus on symlinks (and hashes and all
that) when giving an overview of the tool.  Now, perhaps that
illustration could be useful in the manual.

WDYT?

Thanks,
Ludo’.

bug#53224: Cookbook recipe about profile collisions

[zimoun]

Hi,

On Fri, 14 Jan 2022 at 10:02, Ludovic Courtès <ludo@gnu.org> wrote:

> Currently, on profile collisions, the error message shows where the
> collision originates and a hint on how to work around it.  Perhaps the
> hint is sometimes wrong (in which cases?), or perhaps it’s too terse?
> Can it be improved?

I do not know how it could be improved since it is often case per
case.  For instance, recently we had a collision for python-numpy and
the fix was to introduce python-numpy-next.  That's said, to me the
origin of the collision is often enough for resolving.  However,
indeed the user must have a clear understanding of Guix profile and
propagation.  Therefore, an entry to the Cookbook explaining what does
it mean and where to look appears to me a good idea.


> The definition of what a profile is is another topic.  Currently the
> term “profile” is defined in “Getting Started”:
>
>   https://guix.gnu.org/manual/en/html_node/Getting-Started.html#index-profile
>
> It’s very much defined in passing though.  How can we improve on that?
>
> In some early talks we had illustrations of the symlink forest of a
> profile borrowed from Eelco Dolstra’s own talks on the matter; see for
> instance p. 17 of <https://guix.gnu.org/guix-fosdem-20140201.pdf>.  I
> stopped using it because I think those symlinks are an implementation
> detail and it doesn’t help to focus on symlinks (and hashes and all
> that) when giving an overview of the tool.  Now, perhaps that
> illustration could be useful in the manual.

Well, I do not want to drift the initial message.  I am just keeping
the rolling ball. :-)

In the tutorial I am preparing for JRES, I am doing this way, as a
dialogue (below).  Basically, it is how I am somehow explaining to the
few users I have here, with various background.  This cannot go to the
manual but maybe it could be useful or worth for the Cookbook.

To me, what is an implementation detail is all the dance with /var/,
the generations, the garbage collection, etc.  But the fact that a
profile is symlink pointing a directory with the structure of /usr/
and containing all the environment variables is not an implementation
detail.  It appears to me a core concept to switch from traditional
distro with only one /usr/ to Guix and how to compose many /usr/.

Alice: Where are my packages?
Bob: Installing a package ends with this hint ...snippet...
Alice: Ah, I am confused
Bob: Well, $HOME/.guix-profile is thus the default profile
Alice: What does it mean?
Bob: Look at guix package --search-paths -p $HOME/.guix-profile
Alice: I see, but what is this $HOME/.guix-profile
Bob: Look at: file $HOME/.guix-profile
Alice: Ah, right.  My conclusion is:
  - a profile contains the environment variables
  - a profile is a symlink
Bob: Correct.
Alice: But a symlink pointing to where?
Bob: Look at: readlink -f $HOME/.guix-profile
Alice: It points to a store item, but what does it mean?
Bob: Look at: file $(readlink -f $HOME/.guix-profile)
Alice: Ah right.  My conclusion is:
  - a profile points to a directory item in the store
Bob: Correct.
Alice: What is inside this directory
Bob: Look at: ls $(readlink -f $HOME/.guix-profile)
Alice: Oh, it looks the same hierarchy as the usual and well-know /usr/
Bob: Correct.
Alice: What does it happen when I type python3 I just installed
Bob: Look at: which python3
Alice: Ah right.  Is the binary really in the directory item above?
Bob: Look at: readlink $(which python3)
Alice: Oh right.  A profile points to store item which points to other
store items.
Bob: Correct.
Alice: I guess these other store items have also an /usr/ hierarchy, right?
Bob: Correct.  Do you know how to check that?
Alice: ls $(readlink $(which python3))
Bob: Correct.
Alice: I summarize.  A profile is:
  - a symlink
  - which points to a store item
  - which points to many other store items
  - and all have the structure hierarchy of /usr/
Woo, it is a forest of symlinks!
... image ...


Cheers,
simon

bug#53224: Cookbook recipe about profile collisions

[Leo Famulari]

On Fri, Jan 14, 2022 at 09:47:52AM +0100, Ludovic Courtès wrote:
> > Recently, Matt pointed out that profile collisions can be confusing and
> > difficult to resolve:
> >
> > https://lists.gnu.org/archive/html/guix-devel/2022-01/msg00115.html
> 
> I don’t see the words “profile” and “collision” here; maybe that was
> upthread?

I pointed to this blog post, which was linked from that email:

> > Specifically see this blog post:
> > http://excalamus.com/2021-10-06-guix-debug.html

It's the story of an attempt to learn what the "conflicting entries"
message means and how to resolve it.

> Currently, on profile collisions, the error message shows where the
> collision originates and a hint on how to work around it.  Perhaps the
> hint is sometimes wrong (in which cases?), or perhaps it’s too terse?
> Can it be improved?

The hint is perfectly clear, when you understand "profiles" and
"propagation".

It's like I said later in my message: People use Guix without knowing
very much about it.

That's surprising to me, because I started using Guix *because* I
learned about profiles and how they are used: it's one of the core
innovations of Guix / Nix, to implement unprivileged package management.

But nevertheless, people are using Guix without understanding how it
works. And it's hard to learn about Guix when you are dealing with a
profile collision that you don't understand at all. For many people,
that is not a good moment to start learning. On top of that, profile
collisions are an error state that we can't fix as a bug: we have to
give users the knowledge to resolve the collisions themselves.

For example, the blog post that I linked to ends with "Lessons Learned",
which includes this:

------
What is a profile?

A profile is a directory of symlinks located at ~/.guix-profile.

Propagated inputs can cause conflicts

Propagated inputs are treated somehow differently. It's still not 100%
clear how or why, but it's good to know they're a potential source of
errors.
------

This person spent a lot of time trying to understand the situation and
writing the blog post, but their understanding is still rather weak.

That's why I propose a Cookbook chapter that specifically addresses
profile collisions, to help new users go from "oh no, an error message"
to "aha!"

> The definition of what a profile is is another topic.  Currently the
> term “profile” is defined in “Getting Started”:
> 
>   https://guix.gnu.org/manual/en/html_node/Getting-Started.html#index-profile
> 
> It’s very much defined in passing though.  How can we improve on that?

I think this part of the manual is fine and can stay as it is.

> In some early talks we had illustrations of the symlink forest of a
> profile borrowed from Eelco Dolstra’s own talks on the matter; see for
> instance p. 17 of <https://guix.gnu.org/guix-fosdem-20140201.pdf>.  I
> stopped using it because I think those symlinks are an implementation
> detail and it doesn’t help to focus on symlinks (and hashes and all
> that) when giving an overview of the tool.  Now, perhaps that
> illustration could be useful in the manual.
> 
> WDYT?

The illustrations of the symlink forest were *extremely* helpful to me
when learning how Guix implements unprivileged package management. I
think that later talks from the 2015-2017 era refined the illustrations
to be even more clear.

I'm sorry if the use case for my proposal is still unclear. I can work
on the Cookbook chapter myself.

bug#53224: Cookbook recipe about profile collisions

[Ludovic Courtès]

Hi Leo,

Leo Famulari <leo@famulari.name> skribis:

> I pointed to this blog post, which was linked from that email:
>
>> > Specifically see this blog post:
>> > http://excalamus.com/2021-10-06-guix-debug.html
>
> It's the story of an attempt to learn what the "conflicting entries"
> message means and how to resolve it.

Oh sorry, I had overlooked this.  It makes perfect sense now, and this
blog post is useful in understand what it feels like to encounter that
error message for the first time.

>> Currently, on profile collisions, the error message shows where the
>> collision originates and a hint on how to work around it.  Perhaps the
>> hint is sometimes wrong (in which cases?), or perhaps it’s too terse?
>> Can it be improved?
>
> The hint is perfectly clear, when you understand "profiles" and
> "propagation".
>
> It's like I said later in my message: People use Guix without knowing
> very much about it.
>
> That's surprising to me, because I started using Guix *because* I
> learned about profiles and how they are used: it's one of the core
> innovations of Guix / Nix, to implement unprivileged package management.
>
> But nevertheless, people are using Guix without understanding how it
> works. And it's hard to learn about Guix when you are dealing with a
> profile collision that you don't understand at all. For many people,
> that is not a good moment to start learning. On top of that, profile
> collisions are an error state that we can't fix as a bug: we have to
> give users the knowledge to resolve the collisions themselves.

Yeah.  So, given that “profile” is already defined in “Getting Started”,
I think we can’t do a lot more in the manual.

However, as a way of helping users incrementally discover concepts they
need to understand, we could use one-time hints (sorta like the “Did you
know?” boxes in GUIs.)  ‘guix shell’ already uses a one-time hint
inviting users to run ‘--check’.

We could imagine that the first run of ‘guix install’ & co. would print
a message explaining that your packages’ files are now in
~/.guix-profile, and that it’s what we call a “profile”.

Would that make sense?

> For example, the blog post that I linked to ends with "Lessons Learned",
> which includes this:
>
> ------
> What is a profile?
>
> A profile is a directory of symlinks located at ~/.guix-profile.
>
> Propagated inputs can cause conflicts
>
> Propagated inputs are treated somehow differently. It's still not 100%
> clear how or why, but it's good to know they're a potential source of
> errors.
> ------
>
> This person spent a lot of time trying to understand the situation and
> writing the blog post, but their understanding is still rather weak.

To be fair, the person didn’t look for “profile” in the manual.  I’m not
blaming—mentioning it to clarify what it is we’re trying to fix.

> That's why I propose a Cookbook chapter that specifically addresses
> profile collisions, to help new users go from "oh no, an error message"
> to "aha!"

Makes sense!

>> In some early talks we had illustrations of the symlink forest of a
>> profile borrowed from Eelco Dolstra’s own talks on the matter; see for
>> instance p. 17 of <https://guix.gnu.org/guix-fosdem-20140201.pdf>.  I

[...]

> The illustrations of the symlink forest were *extremely* helpful to me
> when learning how Guix implements unprivileged package management. I
> think that later talks from the 2015-2017 era refined the illustrations
> to be even more clear.

OK, let’s see if we can include an illustration like that under
“Managing Software the Guix Way” or something like that.

> I'm sorry if the use case for my proposal is still unclear. I can work
> on the Cookbook chapter myself.

Great, thanks.

Ludo’.

Profile definition, was Re: bug#53224: Cookbook recipe about profile collisions

[Matt]

 ---- On Mon, 17 Jan 2022 09:16:28 -0500 Ludovic Courtès <ludo@gnu.org> wrote ----

 > > This person spent a lot of time trying to understand the situation and
 > > writing the blog post, but their understanding is still rather weak.
 > 
 > To be fair, the person didn’t look for “profile” in the manual.  I’m not
 > blaming—mentioning it to clarify what it is we’re trying to fix.

With all respect, I *did* look at "profile" in the manual. I spent a lot of time looking and trying to understand things.  I hear you say you're not trying to blame me and I trust that's not your intent.  However, what was said *does* blame me.  It says what I did and did not do, independent of reality: you are not me and you were not present.  Unfortunately, what was said carries all sorts of judgments and implications (ouch!) which, opposite to your intent, is not fair. 

I see you want clarification on what we're trying to fix. May I suggest instead asking, "What problem are we trying to solve?"

I see several problems beyond what I've already said.  However, I'll try to stick to just one that I encountered with the documentation.  Leo is certainly working toward something specific which I suspect is different from what I see. I'll let them speak for themselves.

I'm going to assume that you probably wanted to ask something like, "It looks to me like the manual adequately explains profiles.  But, since I'm the main architect of this system, maybe I have knowledge that someone new doesn't have. Person who is confused, are you able to say where you're confused?"

I would reply:

There are several places I've been confused. Let me give you a specific example.

The manual has at least four places that "profile" is defined:

1.  [[https://guix.gnu.org/en/manual/en/html_node/Getting-Started.html#Getting-Started][(guix) Getting Started]] says,

#+begin_quote
"A profile is a directory containing installed packages"
#+end_quote

2.  [[https://guix.gnu.org/en/manual/en/html_node/Invoking-guix-package.html#Invoking-guix-package][(guix) Invoking guix package]] says,

#+begin_quote
"a directory of installed packages"
#+end_quote

and 3, yet in the guix package options:

#+begin_quote
'-p PROFILE'
     Use PROFILE instead of the user's default profile.

     PROFILE must be the name of a file that will be created upon
     completion.  Concretely, PROFILE will be a mere symbolic link
     ("symlink") pointing to the actual profile where packages are
     installed:

          #+begin_example
          $ guix install hello -p ~/code/my-profile
          ...
          $ ~/code/my-profile/bin/hello
          Hello, world!
          #+end_example
#+end_quote

Elsewhere in (guix) Features is a 4th which says,

#+begin_quote
users have their own “profile”, which points to the packages that they actually want to use
#+end_quote

So, is the profile a directory or a pointer file (e.g. symlink)?  I tend to think of a directory as a container, like a manilla folder that contains papers.  To me, something that points is a file that contains a path to another location.  I see that I used the word "contains" to describe both file and directory, so maybe that's a sign to me I'm missing something there.

Regardless, I hope you can see that it's not always clear whether a profile is a directory or a file.  Yes, on Unix-like systems, directories are files. But Guix throws an error if you call =guix package -p= with a directory:

: guix package: error: rename-file: Is a directory

If you follow the symlinks, the profile is indeed a directory; it is a directory in the store.  But the way users interact with profiles is

          GUIX_PROFILE="$HOME/.guix-profile"
          . "$GUIX_PROFILE/etc/profile"

which is a file. And there are a bunch of symlinks in general. Those appear to be implementation details. But I think it's reasonable to say the abstraction isn't airtight and that, as a user, I have to interact with the implementation details at some level. Certainly at the documentation level. 

Leo is 100% correct that my understanding is still rather weak. I'll do my best despite that to help make the documentation better.

Re: Profile definition, was Re: bug#53224: Cookbook recipe about profile collisions

[Leo Famulari]

On Mon, Jan 17, 2022 at 12:56:20PM -0500, Matt wrote:
> Leo is 100% correct that my understanding is still rather weak. I'll do my best despite that to help make the documentation better.

I hope you will not feel too bad about that. Remember, everyone begins
by not knowing anything. Your situation is not unique at all. Rather,
your energy for improving the documentation for yourself and others is
exemplary, and will improve Guix in the long run.

Overall, this highlights a case where there is tension between when an
implementation detail doesn't matter, and when it does. That is, the
ideal situation is that the implementation details of profiles do not
matter. However, when there are "profile collisions", it does matter.

Re: Profile definition, was Re: bug#53224: Cookbook recipe about profile collisions

[Ludovic Courtès]

Hi,

Leo Famulari <leo@famulari.name> skribis:

> On Mon, Jan 17, 2022 at 12:56:20PM -0500, Matt wrote:
>> Leo is 100% correct that my understanding is still rather weak. I'll do my best despite that to help make the documentation better.
>
> I hope you will not feel too bad about that. Remember, everyone begins
> by not knowing anything. Your situation is not unique at all. Rather,
> your energy for improving the documentation for yourself and others is
> exemplary, and will improve Guix in the long run.

I agree, and I apologize for throwing my own guess of what you did and
did not do, Matt.

Ludo’.

Re: Profile definition, was Re: bug#53224: Cookbook recipe about profile collisions

[Matt]

 ---- On Tue, 18 Jan 2022 10:36:20 -0500 Ludovic Courtès <ludo@gnu.org> wrote ----
 > Hi,
 > 
 > Leo Famulari <leo@famulari.name> skribis:
 > 
 > > On Mon, Jan 17, 2022 at 12:56:20PM -0500, Matt wrote:
 > >> Leo is 100% correct that my understanding is still rather weak. I'll do my best despite that to help make the documentation better.
 > >
 > > I hope you will not feel too bad about that. Remember, everyone begins
 > > by not knowing anything. Your situation is not unique at all. Rather,
 > > your energy for improving the documentation for yourself and others is
 > > exemplary, and will improve Guix in the long run.
 > 
 > I agree, and I apologize for throwing my own guess of what you did and
 > did not do, Matt.
 
Hey, thanks.  I accept your apology and really appreciate it.  Recent interactions had left me feeling unheard and wondering whether I was wasting my time with the Guix community.  I appreciate being seen. I'll continue working with Leo and jgart on documentation.