What exactly guix pull does and why is there a seperate profile at ~/.config/guix/current ?

What exactly guix pull does and why is there a seperate profile at ~/.config/guix/current ?

[Doron Behar]

Hey all,

I'm using Arch Linux and I'd like to get my hands dirty with Guix as an
alternative (but better) package manager. Unfortunately, I'm very
confused as for the role of `guix pull` and `guix package -u` and the
'current guix profile' symlinked to ~/.config/guix/current .

In 'traditional` package managers, there's the concept of a "repository"
which seems to correspond to all the scheme code written in
gnu/packages/*.scm under the Guix Git repository.

Usually, the repositories (or more precisely, the package definitions)
are managed separately from the source code of the package manager
itself, and very reasonably IMO. Since the source code to retrieve
packages / reading definitions and building them shouldn't be linked to
changes such as packages added or updated in the repositories.

According to the documentation, `guix pull` 'updates the distribution
along with the Guix tools'. Does it mean that every time something is
updated in the actual source code of the remote Guix repository, when I
run `guix pull` afterwards my computer builds `guix` and `guix-daemon`
from the ground up? Does it actually run `./configure && make && make
check` behind the scenes on the source files? If so, that's insanely CPU
wasteful and definitely unnecessary.

This wouldn't have been that bad if Guile would have had a compiler
cache like C and some other languages have (see [ccache][1] and
[sccache][2]).

Additionally, I'm not sure I understand the relation between `guix pull`
and `guix package -u`. Let's say I'm running `guix pull` and some
packages have been updated / added.  Do I have to use the guix binary in
~/.config/guix/current/bin/ in order to install them? Speaking of which,
what is the purpose of this profile anyway? Why doesn't `guix pull` just
updates a database of all package definitions and then suggests to run
`guix package -u` to do all the rest - including updating `guix` it self
in the default profile **if needed**?

One more unclear thing for me: If I run `guix pull`, I always get the
message that I should add ~/.config/guix/current/bin/ to my $PATH so I'd
use the latest `guix`. Does it mean that with using `guix` from
~/.guix-profile/bin I don't really get any updates?

I hope the design decisions of the way things work now are genius as the
rest of the ecosystem and that I've failed to see that. Anyway, above
all, I'm not sure as for what `etc/profile` files to `source` so my $PATH
and other environmental variables will always be up to date with the
current state..

Thanks for any help, Doron.

[1]: https://ccache.samba.org
[2]: https://github.com/mozilla/sccache

Re: What exactly guix pull does and why is there a seperate profile at ~/.config/guix/current ?

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 6017 bytes --]


Doron Behar <doron.behar@gmail.com> writes:

> I'm using Arch Linux and I'd like to get my hands dirty with Guix as an
> alternative (but better) package manager. Unfortunately, I'm very
> confused as for the role of `guix pull` and `guix package -u` and the
> 'current guix profile' symlinked to ~/.config/guix/current .
>
> In 'traditional` package managers, there's the concept of a "repository"
> which seems to correspond to all the scheme code written in
> gnu/packages/*.scm under the Guix Git repository.

Yep, Guix doesn't fit the typical repository definition. There's a
concept of "channels" in Guix (and also Nix I hear) [1].

1: https://www.gnu.org/software/guix/manual/en/html_node/Channels.html

But this relates more to how you access a "repository" of Guix packages.

> Usually, the repositories (or more precisely, the package definitions)
> are managed separately from the source code of the package manager
> itself, and very reasonably IMO. Since the source code to retrieve
> packages / reading definitions and building them shouldn't be linked to
> changes such as packages added or updated in the repositories.
>
> According to the documentation, `guix pull` 'updates the distribution
> along with the Guix tools'. Does it mean that every time something is
> updated in the actual source code of the remote Guix repository, when I
> run `guix pull` afterwards my computer builds `guix` and `guix-daemon`
> from the ground up? Does it actually run `./configure && make && make
> check` behind the scenes on the source files? If so, that's insanely CPU
> wasteful and definitely unnecessary.

The "source form" for Guix package definitions are snippets of Guile
code. Given this, I think it's not as unreasonable to follow the normal
processes for building software when getting an updated copy of the
package definitions...

I don't have a great understanding of it, but running guix pull doesn't
just correspond to a single derivation, the task is broken down in to
multiple parts, some of which might already be done. There's a more
technical description here [2].

2: https://issues.guix.info/issue/27284#43

Running guix pull isn't quite the same as running ./configure && make &&
make check. In particular, I don't think it uses ./configure, or runs
the Guix testsuite.

> This wouldn't have been that bad if Guile would have had a compiler
> cache like C and some other languages have (see [ccache][1] and
> [sccache][2]).

While I'm not sure that level of caching would help, or even be possible
due to the isolation of the builds, it is perfectly possible that some
of the derivations computed by guix pull are already built, and don't
need building. Or that a substitute server you've authorised has built
them, and you can just download the data.

> Additionally, I'm not sure I understand the relation between `guix pull`
> and `guix package -u`. Let's say I'm running `guix pull` and some
> packages have been updated / added.  Do I have to use the guix binary in
> ~/.config/guix/current/bin/ in order to install them? Speaking of which,
> what is the purpose of this profile anyway? Why doesn't `guix pull` just
> updates a database of all package definitions and then suggests to run
> `guix package -u` to do all the rest - including updating `guix` it self
> in the default profile **if needed**?

I'm not quite sure if you _have_ to use the guix binary from
~/.config/guix/current/bin, but I think that it's probably a good idea.

I think .guix-profile and .config/guix/current are separate things, as
they're quite different. .guix-profile is a profile constructed from
packages, it has a manifest at .guix-profile/manifest. Whereas
.config/guix/current I think might be constructed by building all the
channels you have configured? Because the guix pull process involves
multiple derivations for different parts, I don't think it's represented
internally as a package (there is also a package definition for guix,
but that's not what's being built when doing guix pull).

I think there's also advantages for treating them separately, especially
being able to control the generations separately, so you can roll back
the .config/guix/current generation without affecting the packages you
have installed for example.

> One more unclear thing for me: If I run `guix pull`, I always get the
> message that I should add ~/.config/guix/current/bin/ to my $PATH so I'd
> use the latest `guix`. Does it mean that with using `guix` from
> ~/.guix-profile/bin I don't really get any updates?

I'm not quite sure, but I'd suggest doing that as otherwise you may be
using an older version of Guix in some respects. The binary installation
docs now say to do this:

  GUIX_PROFILE="`echo ~root`/.config/guix/current" ; \
  source $GUIX_PROFILE/etc/profile

So I'd put that in the relevant initialisation files.

> I hope the design decisions of the way things work now are genius as the
> rest of the ecosystem and that I've failed to see that. Anyway, above
> all, I'm not sure as for what `etc/profile` files to `source` so my $PATH
> and other environmental variables will always be up to date with the
> current state..

So, I don't actually have much experience using Guix on "foreign"
distributions any longer, but on a Guix System, this is included in
/etc/profile.

  # Arrange so that ~/.config/guix/current comes first.
  for profile in "$HOME/.guix-profile" "$HOME/.config/guix/current"
  do
    if [ -f "$profile/etc/profile" ]
    then
      # Load the user profile's settings.
      GUIX_PROFILE="$profile" ; \
      . "$profile/etc/profile"
    else
      # At least define this one so that basic things just work
      # when the user installs their first package.
      export PATH="$profile/bin:$PATH"
    fi
  done

So, I think it's good to source both the etc/profile files inside both
~/.guix-profile and ~/.config/guix/current, and you should source the
~/.config/guix/current one last, as it prepends the profile directory to
the $PATH.

Hope that helps,

Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 962 bytes --]

Re: What exactly guix pull does and why is there a seperate profile at ~/.config/guix/current ?

[Doron Behar]

Thanks for giving your attention Christopher. Most of what you wrote was
very helpful for me to gain an understanding of the ecosystem. I have a
few more doubts though that perhaps could be figured out with help of
someone with better technical background.

On Tue, May 14, 2019 at 08:30:44AM +0100, Christopher Baines wrote:
> > In 'traditional` package managers, there's the concept of a
> > "repository" which seems to correspond to all the scheme code
> > written in gnu/packages/*.scm under the Guix Git repository.
> 
> Yep, Guix doesn't fit the typical repository definition. There's a
> concept of "channels" in Guix (and also Nix I hear) [1].
> 
> 1: https://www.gnu.org/software/guix/manual/en/html_node/Channels.html
> 
> But this relates more to how you access a "repository" of Guix packages.

I'm aware of the idea of channels and I tried to configure that but I've
encountered several issues. I'll post these in a separate thread.

> > According to the documentation, `guix pull` 'updates the distribution
> > along with the Guix tools'. Does it mean that every time something is
> > updated in the actual source code of the remote Guix repository, when I
> > run `guix pull` afterwards my computer builds `guix` and `guix-daemon`
> > from the ground up? Does it actually run `./configure && make && make
> > check` behind the scenes on the source files? If so, that's insanely CPU
> > wasteful and definitely unnecessary.
> 
> The "source form" for Guix package definitions are snippets of Guile
> code. Given this, I think it's not as unreasonable to follow the normal
> processes for building software when getting an updated copy of the
> package definitions...
> 
> I don't have a great understanding of it, but running guix pull doesn't
> just correspond to a single derivation, the task is broken down in to
> multiple parts, some of which might already be done. There's a more
> technical description here [2].
> 
> 2: https://issues.guix.info/issue/27284#43
> 
> Running guix pull isn't quite the same as running ./configure && make &&
> make check. In particular, I don't think it uses ./configure, or runs
> the Guix testsuite.

I've read that 'issue' webpage and it seems the introduced feature
certainly has given a useful improvement but I'm not sure I understand
how relevant it is..

I do understand though that since most of Guix is written in Guile,
packages' definitions are definitely linked to the code that states how
to build them. I guess that means that most it will have to be
recompiled after every update in the repository.

It seems that way from the following stdout of `guix pull` I got today:

https://gist.github.com/doronbehar/fe19099502beecf4fb5c4523d8890862

As you can see, it clearly states multiple times the following:

    building /gnu/store/....drv
    building /gnu/store/....drv
    building /gnu/store/....drv
    ...

And it is also reflected when looking at the CPU usage of this command.
I think there's defiantly room for improving the efficiency of this so
standard and common process..

Note that I'm saying that under the impression that I've setup
substitutes correctly - I've authorized substitutes according to the
documentation and `guix build emacs --dry-run` prints "downloaded" and
not "will be built".

Thinking about it from a Git development work flow perspective, maybe
the Guix project should declare that no other thing then package
additions / version updates should be merged to the master branch and
that all core development should be done on a `develop` branch. This
would hopefully have the effect of having `guix pull` only build
`guix-packages.drv` every time and not everything else as well.

> I'm not quite sure if you _have_ to use the guix binary from
> ~/.config/guix/current/bin, but I think that it's probably a good idea.
> 
> I think .guix-profile and .config/guix/current are separate things, as
> they're quite different. .guix-profile is a profile constructed from
> packages, it has a manifest at .guix-profile/manifest. Whereas
> .config/guix/current I think might be constructed by building all the
> channels you have configured? Because the guix pull process involves
> multiple derivations for different parts, I don't think it's represented
> internally as a package (there is also a package definition for guix,
> but that's not what's being built when doing guix pull).

Thanks for your honesty, I've proven you are correct by comparing the
output of the following commands:

$ ~/.config/guix/current/bin/guix package -A
$ ~/.guix-profile/bin/guix package -A

There were new package and version updates..

> 
> I think there's also advantages for treating them separately, especially
> being able to control the generations separately, so you can roll back
> the .config/guix/current generation without affecting the packages you
> have installed for example.

I'm not sure I agree. After all, `guix pull && guix package -u` also
updates guix in ~/.guix-profile so it means that after that, both of
these `guix` commands in both profiles correspond to the same Guix.. I
hope someone could convince me I'm wrong.

> 
> So, I don't actually have much experience using Guix on "foreign"
> distributions any longer, but on a Guix System, this is included in
> /etc/profile.
> 
>   # Arrange so that ~/.config/guix/current comes first.
>   for profile in "$HOME/.guix-profile" "$HOME/.config/guix/current"
>   do
>     if [ -f "$profile/etc/profile" ]
>     then
>       # Load the user profile's settings.
>       GUIX_PROFILE="$profile" ; \
>       . "$profile/etc/profile"
>     else
>       # At least define this one so that basic things just work
>       # when the user installs their first package.
>       export PATH="$profile/bin:$PATH"
>     fi
>   done
> 

Very helpful indeed, I'll use that in my initialization files.

Re: What exactly guix pull does and why is there a seperate profile at ~/.config/guix/current ?

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 5513 bytes --]


Doron Behar <doron.behar@gmail.com> writes:

>> > According to the documentation, `guix pull` 'updates the distribution
>> > along with the Guix tools'. Does it mean that every time something is
>> > updated in the actual source code of the remote Guix repository, when I
>> > run `guix pull` afterwards my computer builds `guix` and `guix-daemon`
>> > from the ground up? Does it actually run `./configure && make && make
>> > check` behind the scenes on the source files? If so, that's insanely CPU
>> > wasteful and definitely unnecessary.
>>
>> The "source form" for Guix package definitions are snippets of Guile
>> code. Given this, I think it's not as unreasonable to follow the normal
>> processes for building software when getting an updated copy of the
>> package definitions...
>>
>> I don't have a great understanding of it, but running guix pull doesn't
>> just correspond to a single derivation, the task is broken down in to
>> multiple parts, some of which might already be done. There's a more
>> technical description here [2].
>>
>> 2: https://issues.guix.info/issue/27284#43
>>
>> Running guix pull isn't quite the same as running ./configure && make &&
>> make check. In particular, I don't think it uses ./configure, or runs
>> the Guix testsuite.
>
> I've read that 'issue' webpage and it seems the introduced feature
> certainly has given a useful improvement but I'm not sure I understand
> how relevant it is..
>
> I do understand though that since most of Guix is written in Guile,
> packages' definitions are definitely linked to the code that states how
> to build them. I guess that means that most it will have to be
> recompiled after every update in the repository.
>
> It seems that way from the following stdout of `guix pull` I got today:
>
> https://gist.github.com/doronbehar/fe19099502beecf4fb5c4523d8890862
>
> As you can see, it clearly states multiple times the following:
>
>     building /gnu/store/....drv
>     building /gnu/store/....drv
>     building /gnu/store/....drv
>     ...
>
> And it is also reflected when looking at the CPU usage of this command.
> I think there's defiantly room for improving the efficiency of this so
> standard and common process..
>
> Note that I'm saying that under the impression that I've setup
> substitutes correctly - I've authorized substitutes according to the
> documentation and `guix build emacs --dry-run` prints "downloaded" and
> not "will be built".
>
> Thinking about it from a Git development work flow perspective, maybe
> the Guix project should declare that no other thing then package
> additions / version updates should be merged to the master branch and
> that all core development should be done on a `develop` branch. This
> would hopefully have the effect of having `guix pull` only build
> `guix-packages.drv` every time and not everything else as well.

Yeah, I also think there's room for improvement. Just to add more data,
you can see the relevant builds for guix pull on ci.guix.gnu.org here
[1].

As far as I can see, ae5de93 which you pulled was pushed to master at
~05:47, and the build finished at ~12:59 [3]. Assuming this is correct,
that's ~7 hours. I don't think it actually took this long (hopefully),
but it probably wasn't started immediately.

1: http://ci.guix.gnu.org/jobset/guix-modular-master
2: http://ci.guix.gnu.org/eval/5333
3: http://ci.guix.gnu.org/build/1321772

As far as I'm aware, this is more of a software issue with the current
build farm, that could possibly be addressed with some
prioritisation. It probably wouldn't be perfect, but I'd hope if it was
just a bit faster, less people would be impacted by guix pull taking a
while.

>> I think there's also advantages for treating them separately, especially
>> being able to control the generations separately, so you can roll back
>> the .config/guix/current generation without affecting the packages you
>> have installed for example.
>
> I'm not sure I agree. After all, `guix pull && guix package -u` also
> updates guix in ~/.guix-profile so it means that after that, both of
> these `guix` commands in both profiles correspond to the same Guix.. I
> hope someone could convince me I'm wrong.

Maybe I can. Firstly, you probably don't want to have the Guix package
in your profile. If you've got it installed, I'd suggest removing it.

The package for guix in guix normally referrs to a particular commit,
which is updated occasionally. It's in the package version, so you can
check this:

  → guix package --show=guix
  name: guix
  version: 1.0.0-1.326dcbf

This package is mainly used by other packages which have a dependency on
Guix, for example Cuirass, which is the software behind ci.guix.gnu.org.

If I build the guix package, and then run guix describe, I'll get an
error:

  → /gnu/store/cwlghngrh03igf8cfsp2mf49c2l9fnf5-guix-1.0.0-1.326dcbf/bin/guix describe
  guix describe: error: failed to determine origin
  hint: Perhaps this `guix' command was not obtained with `guix pull'? Its version string is 1.0.0-1.326dcbf.

However, if I run guix describe for the store item generated by guix
pull, I get the expected result.

  → ~/.config/guix/current/bin/guix describe
  Generation 72	May 06 2019 11:27:52	(current)
    guix fc7d632
      repository URL: https://git.savannah.gnu.org/git/guix.git
      branch: master
      commit: fc7d632829e93bd1c6d2db6f99ce22766ff9c67a

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 962 bytes --]