bug#22629: ‘guix pull’ and external dependencies

bug#22629: ‘guix pull’ and external dependencies

[Ludovic Courtès]

Hi!

Efraim Flashner <efraim@flashner.co.il> skribis:

> If I understand it correctly, as part of `guix pull' we get the latest
> package definitions, but `guix' and `guix-daemon' are at the
> guix-snapshot version, aka 0.11.0-4. If instead `guix-daemon' was from
> the tip of master then it'd be at the equivalant of running
> './pre-inst-env guix-daemon --build-users...', which would have all
> these changes.

What ‘guix pull’ does is fetch the latest code, build *the Scheme
subset* of that code, and install it in ~/.config/guix/latest.

Thus, it gives you the latest package recipes as well as the latest
‘guix’ sub-commands.

What it does not give you is:

  1. The latest C++ code (guix-daemon, guix-register).
  2. The latest locales.
  3. The latest elisp code.
  4. The latest dependencies (the Guile that appears in the shebang of
     the ‘guix’ command, zlib, Guile-SSH, Guile-JSON, etc.)

It worked OK when Guix was self-contained, and assuming people would
update guix-daemon through other ways (‘guix system reconfigure’ on
GuixSD).  But now we see that these shortcomings are starting to bite.

So I think what we need to do is for “guix pull-ng” to build and install
a complete ‘guix’ package, and to manage it pretty much like other
packages is managed, except not in the user’s main profile (because that
could lead to undesirable behavior, where upgrading Guix creates a new
generation, or, in theory, unrecoverable problems, where you cannot
roll back because previous generations use an old Guix that does not
understand the new manifest format.)

The difficulty is that ./configure && make && make install in Guix takes
some time, and we probably wouldn’t want to do that on each ‘guix pull’
invocation (esp. with Guile 2.2’s compilation times.)

So we may have to provide substitutes of Guix itself, and arrange so
that ‘guix pull’ pulls up to a tag for which we have substitutes.

Ideas welcome!  See <https://bugs.gnu.org/22629>.

Ludo’.

bug#22629: ‘guix pull’ and external dependencies

[Chris Marusich]

[-- Attachment #1: Type: text/plain, Size: 3659 bytes --]

Hi Ludo`,

ludo@gnu.org (Ludovic Courtès) writes:

> So I think what we need to do is for “guix pull-ng” to build and install
> a complete ‘guix’ package, and to manage it pretty much like other
> packages is managed, 

I think that's very reasonable.  It seems more intuitive than the
current way 'guix pull' works.  I suspect that managing the installed
version of Guix via the same Guix mechanisms that we use to manage any
other package might be the best, most intuitive solution.

Would it simplify the problem if we packaged the "Guix client stuff",
the "Guix daemon stuff", and maybe the "Guix package definition stuff"
separately?  Then a user could just install the "Guix client stuff"
package if she wanted to upgrade the Guix client tools, or the "Guix
package definition stuff" package if she wanted to get the latest
package definitions.

> except not in the user’s main profile (because that could lead to
> undesirable behavior, 

If we don't store Guix in the user's main profile, where would it go?  A
system profile (like in GuixSD)?  What if another user wants to run a
different version of Guix?  It might be nice to let them do that.

It's not clear to me why it's riskier to store Guix in a profile rather
than outside the profile (but still in the store via the
$HOME/.config/guix/latest symlink), which is what 'guix pull' does now.
You seem to think it's riskier; I'm curious to know more about why.

> where upgrading Guix creates a new generation, 

Why should upgrading Guix NOT create a new generation?  I thought that a
new profile generation would be created any time you upgrade a package,
and I thought that was a good thing because it facilitates easy,
transactional roll-back.  Perhaps I'm missing something.

> or, in theory, unrecoverable problems, where you cannot roll back
> because previous generations use an old Guix that does not understand
> the new manifest format.)

Why would a change in manifest format be unrecoverable?  It looks like
each profile generation contains a manifest file.  Assuming that the new
Guix functions well enough to perform roll back, couldn't we just roll
back to the previous profile generation, where we would have both (1)
the old profile's manifest file, and (2) the previous Guix, which
understands that format?  Since rolling back a profile is basically just
a symlink flip, I think the new Guix could probably do that even if it
didn't understand the old manifest format.

> The difficulty is that ./configure && make && make install in Guix takes
> some time, and we probably wouldn’t want to do that on each ‘guix pull’
> invocation (esp. with Guile 2.2’s compilation times.)
>
> So we may have to provide substitutes of Guix itself, and arrange so
> that ‘guix pull’ pulls up to a tag for which we have substitutes.

What if we had a special package version of Guix (e.g., "v0.11.0") which
we kept up to date via some mechanism?  Maybe something as simple as a
Git hook could help increase the likelihood of that version being
substitutable.  For example, we could have a Git hook that prevents
someone from checking in a change if the latest Git tag does not
correspond to a Guix package version.  Maybe we can do better.

I actually think it would be a good thing if we can run "guix pull"
without substitutes available.  But it should use a substitute by
default, and "build from source" should be a fallback mechanism that the
user has to explicitly request, just like when installing new packages.
That would help avoid unexpectedly long "guix pull" invocations.

-- 
Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]

bug#22629: ‘guix pull’ and external dependencies

[Ludovic Courtès]

Hello,

Chris Marusich <cmmarusich@gmail.com> skribis:

> ludo@gnu.org (Ludovic Courtès) writes:
>
>> So I think what we need to do is for “guix pull-ng” to build and install
>> a complete ‘guix’ package, and to manage it pretty much like other
>> packages is managed, 
>
> I think that's very reasonable.  It seems more intuitive than the
> current way 'guix pull' works.  I suspect that managing the installed
> version of Guix via the same Guix mechanisms that we use to manage any
> other package might be the best, most intuitive solution.
>
> Would it simplify the problem if we packaged the "Guix client stuff",
> the "Guix daemon stuff", and maybe the "Guix package definition stuff"
> separately?  Then a user could just install the "Guix client stuff"
> package if she wanted to upgrade the Guix client tools, or the "Guix
> package definition stuff" package if she wanted to get the latest
> package definitions.

It would be bad to separate package definitions from the rest because
they are very much intertwined: package definitions depend on the
definition of ‘package’, on build system implementations, and so on.

We could have guix-sans-daemon though, if that helps (which I suspect is
not the case).

>> except not in the user’s main profile (because that could lead to
>> undesirable behavior, 
>
> If we don't store Guix in the user's main profile, where would it go?

In “some sort of a profile” in ~/.config/guix/latest or similar.

> It's not clear to me why it's riskier to store Guix in a profile rather
> than outside the profile (but still in the store via the
> $HOME/.config/guix/latest symlink), which is what 'guix pull' does now.
> You seem to think it's riskier; I'm curious to know more about why.

There’s the manifest format change issue I mentioned, or the inability
to roll back if you install a broken Guix.

>> where upgrading Guix creates a new generation, 
>
> Why should upgrading Guix NOT create a new generation?  I thought that a
> new profile generation would be created any time you upgrade a package,
> and I thought that was a good thing because it facilitates easy,
> transactional roll-back.  Perhaps I'm missing something.

I’m suggesting that upgrading Guix creates a new generation (so we agree
here), just not in the user’s profile.

>> or, in theory, unrecoverable problems, where you cannot roll back
>> because previous generations use an old Guix that does not understand
>> the new manifest format.)
>
> Why would a change in manifest format be unrecoverable?  It looks like
> each profile generation contains a manifest file.  Assuming that the new
> Guix functions well enough to perform roll back, couldn't we just roll
> back to the previous profile generation, where we would have both (1)
> the old profile's manifest file, and (2) the previous Guix, which
> understands that format?  Since rolling back a profile is basically just
> a symlink flip, I think the new Guix could probably do that even if it
> didn't understand the old manifest format.

Yeah, I think you’re right.  :-)

In general, I think my concern is more that we cannot promise that
downgrading Guix will work, considering the potential for on-disk format
changes.  It’s a bit theoretical, but not entirely sci-fi either.

>> The difficulty is that ./configure && make && make install in Guix takes
>> some time, and we probably wouldn’t want to do that on each ‘guix pull’
>> invocation (esp. with Guile 2.2’s compilation times.)
>>
>> So we may have to provide substitutes of Guix itself, and arrange so
>> that ‘guix pull’ pulls up to a tag for which we have substitutes.
>
> What if we had a special package version of Guix (e.g., "v0.11.0") which
> we kept up to date via some mechanism?  Maybe something as simple as a
> Git hook could help increase the likelihood of that version being
> substitutable.  For example, we could have a Git hook that prevents
> someone from checking in a change if the latest Git tag does not
> correspond to a Guix package version.  Maybe we can do better.

Right, we could do something like that.  There are still non-zero
chances that someone running ‘guix pull’ at an arbitrary point in time
will have to build locally, which is not great.

> I actually think it would be a good thing if we can run "guix pull"
> without substitutes available.  But it should use a substitute by
> default, and "build from source" should be a fallback mechanism that the
> user has to explicitly request, just like when installing new packages.
> That would help avoid unexpectedly long "guix pull" invocations.

Yes, using substitutes or falling back to source builds is always the
default.

Thanks for your feedback!

Ludo’.