Offloading to use Guile-SSH instead of lsh

Offloading to use Guile-SSH instead of lsh

[Ludovic Courtès]

Hello!

The ‘wip-guile-ssh’ branch is finally getting into shape.  As soon as
a new Guile-SSH release is out, I’ll merge it on master.

The goal of this branch is to use Guile-SSH (libssh bindings) instead of
invoking ‘lsh’ when offloading builds (see
<https://www.gnu.org/software/guix/manual/html_node/Daemon-Offload-Setup.html>
for background.)

The new code is shorter(!), more efficient, more elegant, and hopefully
more reliable.  The efficiency gains come from the fact that we can
reuse SSH sessions in an intelligent way.  The elegance has to do with
the fact that we have direct access to the SSH API, as opposed to
invoking ‘lsh’ and checking its exit status.

Improved reliability (well, modulo new bugs ;-)) comes from the fact
that we open a direct connection to the remote guix-daemon for the
duration of the offload, which allows us to better control GC root
registration—we can use the ‘add-temp-root’ RPC directly instead of
writing indirect root files in $HOME.

Before we put it in production on hydra.gnu.org :-), I’d welcome more
testing!  To do that you need a current Guile-SSH (with the
‘channel-send-eof!’  procedure), and the ‘build-machine’ form is
slightly different than before: it has a new ‘host-key’ field to contain
the OpenSSH-formatted host key of the machine.  I’ve updated the doc to
reflect that.

Feedback welcome!

Ludo’.

Re: Offloading to use Guile-SSH instead of lsh

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 1882 bytes --]

On Sat, Nov 05, 2016 at 04:41:41PM +0100, Ludovic Courtès wrote:
> Hello!
> 
> The ‘wip-guile-ssh’ branch is finally getting into shape.  As soon as
> a new Guile-SSH release is out, I’ll merge it on master.
> 
> The goal of this branch is to use Guile-SSH (libssh bindings) instead of
> invoking ‘lsh’ when offloading builds (see
> <https://www.gnu.org/software/guix/manual/html_node/Daemon-Offload-Setup.html>
> for background.)
> 
> The new code is shorter(!), more efficient, more elegant, and hopefully
> more reliable.  The efficiency gains come from the fact that we can
> reuse SSH sessions in an intelligent way.  The elegance has to do with
> the fact that we have direct access to the SSH API, as opposed to
> invoking ‘lsh’ and checking its exit status.
> 
> Improved reliability (well, modulo new bugs ;-)) comes from the fact
> that we open a direct connection to the remote guix-daemon for the
> duration of the offload, which allows us to better control GC root
> registration—we can use the ‘add-temp-root’ RPC directly instead of
> writing indirect root files in $HOME.
> 
> Before we put it in production on hydra.gnu.org :-), I’d welcome more
> testing!  To do that you need a current Guile-SSH (with the
> ‘channel-send-eof!’  procedure), and the ‘build-machine’ form is
> slightly different than before: it has a new ‘host-key’ field to contain
> the OpenSSH-formatted host key of the machine.  I’ve updated the doc to
> reflect that.
> 
> Feedback welcome!
> 
> Ludo’.
> 

So to clarify, I need to switch just the master to the wip-guile-ssh
branch, or also the slaves?

-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 801 bytes --]

Re: Offloading to use Guile-SSH instead of lsh

[Ludovic Courtès]

Efraim Flashner <efraim@flashner.co.il> skribis:

> So to clarify, I need to switch just the master to the wip-guile-ssh
> branch, or also the slaves?

Just the master node.

The requirements for build machines are unchanged: they need to have
‘guile’ in $PATH and have guix-daemon running.

Ludo’.

Re: Offloading to use Guile-SSH instead of lsh

[Ludovic Courtès]

Hi!

ludo@gnu.org (Ludovic Courtès) skribis:

> The ‘wip-guile-ssh’ branch is finally getting into shape.  As soon as
> a new Guile-SSH release is out, I’ll merge it on master.

Done!

It seems to work well in my own tests, but more feedback is welcome.

Ludo’.

Re: Offloading to use Guile-SSH instead of lsh

[Leo Famulari]

On Fri, Nov 25, 2016 at 11:50:22PM +0100, Ludovic Courtès wrote:
> Hi!
> 
> ludo@gnu.org (Ludovic Courtès) skribis:
> 
> > The ‘wip-guile-ssh’ branch is finally getting into shape.  As soon as
> > a new Guile-SSH release is out, I’ll merge it on master.
> 
> Done!
> 
> It seems to work well in my own tests, but more feedback is welcome.

It looks like there is a spurious module import of (ssh key) with the
latest code:

$ guix pull                                         
                          
Starting download of /tmp/guix-file.rr3z8q
From http://git.savannah.gnu.org/cgit/guix.git/snapshot/master.tar.gz...
 ….tar.gz                                   2.7MiB/s 00:04 | 10.7MiB transferred
unpacking '/gnu/store/7agpxhdimc0gbyjmryfjnnpd1ahzdx4q-guix-latest.tar.gz'...
The following derivation will be built:
   /gnu/store/z958zcva82idmanswyw7zg00zyack221-guix-latest.drv
building path(s) `/gnu/store/q3kckrsgqdnrka24aavbbk423y6zhlkj-guix-latest'
copying and compiling to '/gnu/store/q3kckrsgqdnrka24aavbbk423y6zhlkj-guix-latest'...
loading...	 20.5% of 547 filesBacktrace:
In ice-9/boot-9.scm:
1727: 19 [%start-stack load-stack #<procedure 8fdb60 at ice-9/boot-9.scm:4047:10 ()>]
1732: 18 [#<procedure 8fe6f0 ()>]
In unknown file:
   ?: 17 [primitive-load "/gnu/store/waghfcxyknvhd4ribwz6cxq8b5437y9d-guix-latest-builder"]
In ./guix/build/pull.scm:
  57: 16 [build-guix "/gnu/store/q3kckrsgqdnrka24aavbbk423y6zhlkj-guix-latest" ...]
 123: 15 [#<procedure c0f280 at ./guix/build/pull.scm:57:2 ()>]
In ice-9/boot-9.scm:
2867: 14 [resolve-interface (guix scripts offload) #:select ...]
2792: 13 [#<procedure 852d80 at ice-9/boot-9.scm:2780:4 (name #:optional autoload version #:key ensure)> # ...]
3068: 12 [try-module-autoload (guix scripts offload) #f]
2404: 11 [save-module-excursion #<procedure cf0c90 at ice-9/boot-9.scm:3069:17 ()>]
3088: 10 [#<procedure cf0c90 at ice-9/boot-9.scm:3069:17 ()>]
In unknown file:
   ?: 9 [primitive-load-path "guix/scripts/offload" ...]
In ice-9/eval.scm:
 505: 8 [#<procedure 79c6e0 at ice-9/eval.scm:499:4 (exp)> (define-module # # ...)]
In ice-9/psyntax.scm:
1106: 7 [expand-top-sequence ((define-module # # # ...)) () ((top)) ...]
 989: 6 [scan ((define-module (guix scripts offload) #:use-module ...)) () ...]
 279: 5 [scan ((#(syntax-object let # ...) (#) (# #) ...)) () ...]
In ice-9/eval.scm:
 411: 4 [eval # ()]
In ice-9/boot-9.scm:
2954: 3 [define-module* (guix scripts offload) #:filename ...]
2929: 2 [resolve-imports (((ssh key)) ((ssh auth)) ((ssh session)) ...)]
2870: 1 [resolve-interface (ssh key) #:select ...]
In unknown file:
   ?: 0 [scm-error misc-error #f "~A ~S" ("no code for module" (ssh key)) #f]

ERROR: In procedure scm-error:
ERROR: no code for module (ssh key)
builder for `/gnu/store/z958zcva82idmanswyw7zg00zyack221-guix-latest.drv' failed with exit code 1
guix pull: error: build failed: build of `/gnu/store/z958zcva82idmanswyw7zg00zyack221-guix-latest.drv' failed

Re: Offloading to use Guile-SSH instead of lsh

[宋文武]

Leo Famulari <leo@famulari.name> writes:

> On Fri, Nov 25, 2016 at 11:50:22PM +0100, Ludovic Courtès wrote:
>> Hi!
>> 
>> ludo@gnu.org (Ludovic Courtès) skribis:
>> 
>> > The ‘wip-guile-ssh’ branch is finally getting into shape.  As soon as
>> > a new Guile-SSH release is out, I’ll merge it on master.
>> 
>> Done!
>> 
>> It seems to work well in my own tests, but more feedback is welcome.
>
> It looks like there is a spurious module import of (ssh key) with the
> latest code:
>
> $ guix pull                                         
>                           
> ...
> 2870: 1 [resolve-interface (ssh key) #:select ...]
> In unknown file:
>    ?: 0 [scm-error misc-error #f "~A ~S" ("no code for module" (ssh key)) #f]
>
> ERROR: In procedure scm-error:
> ERROR: no code for module (ssh key)
> builder for `/gnu/store/z958zcva82idmanswyw7zg00zyack221-guix-latest.drv' failed with exit code 1
> guix pull: error: build failed: build of `/gnu/store/z958zcva82idmanswyw7zg00zyack221-guix-latest.drv' failed

I push a commit 5aed7f10f to add guile-ssh to the load-path of the pull
builder (the commit message refers a wrong commit, sorry) to fix it.

But as ng0 report in #guix, the 0.11 install media is still broken.
Since I add guile-ssh's go files to load-compiled-path, which are
broken before commit 92b7258.

Should I remove guile-ssh from %load-compiled-path in
build-aux/build-self.scm?  (assuming the offload script build fine with
guile-ssh-0.9.0 of the guix-0.11)

Re: Offloading to use Guile-SSH instead of lsh

[Ludovic Courtès]

iyzsong@member.fsf.org (宋文武) skribis:

> Leo Famulari <leo@famulari.name> writes:
>
>> On Fri, Nov 25, 2016 at 11:50:22PM +0100, Ludovic Courtès wrote:
>>> Hi!
>>> 
>>> ludo@gnu.org (Ludovic Courtès) skribis:
>>> 
>>> > The ‘wip-guile-ssh’ branch is finally getting into shape.  As soon as
>>> > a new Guile-SSH release is out, I’ll merge it on master.
>>> 
>>> Done!
>>> 
>>> It seems to work well in my own tests, but more feedback is welcome.
>>
>> It looks like there is a spurious module import of (ssh key) with the
>> latest code:
>>
>> $ guix pull                                         
>>                           
>> ...
>> 2870: 1 [resolve-interface (ssh key) #:select ...]
>> In unknown file:
>>    ?: 0 [scm-error misc-error #f "~A ~S" ("no code for module" (ssh key)) #f]
>>
>> ERROR: In procedure scm-error:
>> ERROR: no code for module (ssh key)
>> builder for `/gnu/store/z958zcva82idmanswyw7zg00zyack221-guix-latest.drv' failed with exit code 1
>> guix pull: error: build failed: build of `/gnu/store/z958zcva82idmanswyw7zg00zyack221-guix-latest.drv' failed
>
> I push a commit 5aed7f10f to add guile-ssh to the load-path of the pull
> builder (the commit message refers a wrong commit, sorry) to fix it.
>
> But as ng0 report in #guix, the 0.11 install media is still broken.
> Since I add guile-ssh's go files to load-compiled-path, which are
> broken before commit 92b7258.
>
> Should I remove guile-ssh from %load-compiled-path in
> build-aux/build-self.scm?  (assuming the offload script build fine with
> guile-ssh-0.9.0 of the guix-0.11)

Bah, what a mess!

‘guix offload’ needs Guile-SSH 0.10.2, but ‘guix offload’ is only
invoked from guix-daemon, and guix-daemon is not updated using ‘guix
pull’ anyway.

I committed a workaround as aa28ecc40af91d7cdff2fb3ab4ad86ad10d43ab7.

Fundamentally, it illustrates that the current strategy of ‘guix pull’
of relying on user-provided packages is flawed: we can get a broken
Guile-SSH, or one with the wrong version, or none at all, and that
really sucks.

Updating Guix should be like updating a normal package: you get the
package itself and all its dependencies that are known to work.

Ludo’.

Re: Offloading to use Guile-SSH instead of lsh

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 3015 bytes --]

On Sun, Nov 27, 2016 at 11:10:53PM +0100, Ludovic Courtès wrote:
> iyzsong@member.fsf.org (宋文武) skribis:
> 
> > Leo Famulari <leo@famulari.name> writes:
> >
> >> On Fri, Nov 25, 2016 at 11:50:22PM +0100, Ludovic Courtès wrote:
> >>> Hi!
> >>> 
> >>> ludo@gnu.org (Ludovic Courtès) skribis:
> >>> 
> >>> > The ‘wip-guile-ssh’ branch is finally getting into shape.  As soon as
> >>> > a new Guile-SSH release is out, I’ll merge it on master.
> >>> 
> >>> Done!
> >>> 
> >>> It seems to work well in my own tests, but more feedback is welcome.
> >>
> >> It looks like there is a spurious module import of (ssh key) with the
> >> latest code:
> >>
> >> $ guix pull                                         
> >>                           
> >> ...
> >> 2870: 1 [resolve-interface (ssh key) #:select ...]
> >> In unknown file:
> >>    ?: 0 [scm-error misc-error #f "~A ~S" ("no code for module" (ssh key)) #f]
> >>
> >> ERROR: In procedure scm-error:
> >> ERROR: no code for module (ssh key)
> >> builder for `/gnu/store/z958zcva82idmanswyw7zg00zyack221-guix-latest.drv' failed with exit code 1
> >> guix pull: error: build failed: build of `/gnu/store/z958zcva82idmanswyw7zg00zyack221-guix-latest.drv' failed
> >
> > I push a commit 5aed7f10f to add guile-ssh to the load-path of the pull
> > builder (the commit message refers a wrong commit, sorry) to fix it.
> >
> > But as ng0 report in #guix, the 0.11 install media is still broken.
> > Since I add guile-ssh's go files to load-compiled-path, which are
> > broken before commit 92b7258.
> >
> > Should I remove guile-ssh from %load-compiled-path in
> > build-aux/build-self.scm?  (assuming the offload script build fine with
> > guile-ssh-0.9.0 of the guix-0.11)
> 
> Bah, what a mess!
> 
> ‘guix offload’ needs Guile-SSH 0.10.2, but ‘guix offload’ is only
> invoked from guix-daemon, and guix-daemon is not updated using ‘guix
> pull’ anyway.
> 
> I committed a workaround as aa28ecc40af91d7cdff2fb3ab4ad86ad10d43ab7.
> 
> Fundamentally, it illustrates that the current strategy of ‘guix pull’
> of relying on user-provided packages is flawed: we can get a broken
> Guile-SSH, or one with the wrong version, or none at all, and that
> really sucks.
> 
> Updating Guix should be like updating a normal package: you get the
> package itself and all its dependencies that are known to work.
> 
> Ludo’.
> 

If I understand it correctly, as part of `guix pull' we get the latest
package definitions, but `guix' and `guix-daemon' are at the
guix-snapshot version, aka 0.11.0-4. If instead `guix-daemon' was from
the tip of master then it'd be at the equivalant of running
'./pre-inst-env guix-daemon --build-users...', which would have all
these changes.

-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

bug#22629: ‘guix pull’ and external dependencies

[Ludovic Courtès]

Hi!

Efraim Flashner <efraim@flashner.co.il> skribis:

> If I understand it correctly, as part of `guix pull' we get the latest
> package definitions, but `guix' and `guix-daemon' are at the
> guix-snapshot version, aka 0.11.0-4. If instead `guix-daemon' was from
> the tip of master then it'd be at the equivalant of running
> './pre-inst-env guix-daemon --build-users...', which would have all
> these changes.

What ‘guix pull’ does is fetch the latest code, build *the Scheme
subset* of that code, and install it in ~/.config/guix/latest.

Thus, it gives you the latest package recipes as well as the latest
‘guix’ sub-commands.

What it does not give you is:

  1. The latest C++ code (guix-daemon, guix-register).
  2. The latest locales.
  3. The latest elisp code.
  4. The latest dependencies (the Guile that appears in the shebang of
     the ‘guix’ command, zlib, Guile-SSH, Guile-JSON, etc.)

It worked OK when Guix was self-contained, and assuming people would
update guix-daemon through other ways (‘guix system reconfigure’ on
GuixSD).  But now we see that these shortcomings are starting to bite.

So I think what we need to do is for “guix pull-ng” to build and install
a complete ‘guix’ package, and to manage it pretty much like other
packages is managed, except not in the user’s main profile (because that
could lead to undesirable behavior, where upgrading Guix creates a new
generation, or, in theory, unrecoverable problems, where you cannot
roll back because previous generations use an old Guix that does not
understand the new manifest format.)

The difficulty is that ./configure && make && make install in Guix takes
some time, and we probably wouldn’t want to do that on each ‘guix pull’
invocation (esp. with Guile 2.2’s compilation times.)

So we may have to provide substitutes of Guix itself, and arrange so
that ‘guix pull’ pulls up to a tag for which we have substitutes.

Ideas welcome!  See <https://bugs.gnu.org/22629>.

Ludo’.

‘guix pull’ and external dependencies

[Ludovic Courtès]

Hi!

Efraim Flashner <efraim@flashner.co.il> skribis:

> If I understand it correctly, as part of `guix pull' we get the latest
> package definitions, but `guix' and `guix-daemon' are at the
> guix-snapshot version, aka 0.11.0-4. If instead `guix-daemon' was from
> the tip of master then it'd be at the equivalant of running
> './pre-inst-env guix-daemon --build-users...', which would have all
> these changes.

What ‘guix pull’ does is fetch the latest code, build *the Scheme
subset* of that code, and install it in ~/.config/guix/latest.

Thus, it gives you the latest package recipes as well as the latest
‘guix’ sub-commands.

What it does not give you is:

  1. The latest C++ code (guix-daemon, guix-register).
  2. The latest locales.
  3. The latest elisp code.
  4. The latest dependencies (the Guile that appears in the shebang of
     the ‘guix’ command, zlib, Guile-SSH, Guile-JSON, etc.)

It worked OK when Guix was self-contained, and assuming people would
update guix-daemon through other ways (‘guix system reconfigure’ on
GuixSD).  But now we see that these shortcomings are starting to bite.

So I think what we need to do is for “guix pull-ng” to build and install
a complete ‘guix’ package, and to manage it pretty much like other
packages is managed, except not in the user’s main profile (because that
could lead to undesirable behavior, where upgrading Guix creates a new
generation, or, in theory, unrecoverable problems, where you cannot
roll back because previous generations use an old Guix that does not
understand the new manifest format.)

The difficulty is that ./configure && make && make install in Guix takes
some time, and we probably wouldn’t want to do that on each ‘guix pull’
invocation (esp. with Guile 2.2’s compilation times.)

So we may have to provide substitutes of Guix itself, and arrange so
that ‘guix pull’ pulls up to a tag for which we have substitutes.

Ideas welcome!  See <https://bugs.gnu.org/22629>.

Ludo’.

bug#22629: ‘guix pull’ and external dependencies

[Chris Marusich]

[-- Attachment #1: Type: text/plain, Size: 3659 bytes --]

Hi Ludo`,

ludo@gnu.org (Ludovic Courtès) writes:

> So I think what we need to do is for “guix pull-ng” to build and install
> a complete ‘guix’ package, and to manage it pretty much like other
> packages is managed, 

I think that's very reasonable.  It seems more intuitive than the
current way 'guix pull' works.  I suspect that managing the installed
version of Guix via the same Guix mechanisms that we use to manage any
other package might be the best, most intuitive solution.

Would it simplify the problem if we packaged the "Guix client stuff",
the "Guix daemon stuff", and maybe the "Guix package definition stuff"
separately?  Then a user could just install the "Guix client stuff"
package if she wanted to upgrade the Guix client tools, or the "Guix
package definition stuff" package if she wanted to get the latest
package definitions.

> except not in the user’s main profile (because that could lead to
> undesirable behavior, 

If we don't store Guix in the user's main profile, where would it go?  A
system profile (like in GuixSD)?  What if another user wants to run a
different version of Guix?  It might be nice to let them do that.

It's not clear to me why it's riskier to store Guix in a profile rather
than outside the profile (but still in the store via the
$HOME/.config/guix/latest symlink), which is what 'guix pull' does now.
You seem to think it's riskier; I'm curious to know more about why.

> where upgrading Guix creates a new generation, 

Why should upgrading Guix NOT create a new generation?  I thought that a
new profile generation would be created any time you upgrade a package,
and I thought that was a good thing because it facilitates easy,
transactional roll-back.  Perhaps I'm missing something.

> or, in theory, unrecoverable problems, where you cannot roll back
> because previous generations use an old Guix that does not understand
> the new manifest format.)

Why would a change in manifest format be unrecoverable?  It looks like
each profile generation contains a manifest file.  Assuming that the new
Guix functions well enough to perform roll back, couldn't we just roll
back to the previous profile generation, where we would have both (1)
the old profile's manifest file, and (2) the previous Guix, which
understands that format?  Since rolling back a profile is basically just
a symlink flip, I think the new Guix could probably do that even if it
didn't understand the old manifest format.

> The difficulty is that ./configure && make && make install in Guix takes
> some time, and we probably wouldn’t want to do that on each ‘guix pull’
> invocation (esp. with Guile 2.2’s compilation times.)
>
> So we may have to provide substitutes of Guix itself, and arrange so
> that ‘guix pull’ pulls up to a tag for which we have substitutes.

What if we had a special package version of Guix (e.g., "v0.11.0") which
we kept up to date via some mechanism?  Maybe something as simple as a
Git hook could help increase the likelihood of that version being
substitutable.  For example, we could have a Git hook that prevents
someone from checking in a change if the latest Git tag does not
correspond to a Guix package version.  Maybe we can do better.

I actually think it would be a good thing if we can run "guix pull"
without substitutes available.  But it should use a substitute by
default, and "build from source" should be a fallback mechanism that the
user has to explicitly request, just like when installing new packages.
That would help avoid unexpectedly long "guix pull" invocations.

-- 
Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]

Re: ‘guix pull’ and external dependencies

[Chris Marusich]

[-- Attachment #1: Type: text/plain, Size: 3659 bytes --]

Hi Ludo`,

ludo@gnu.org (Ludovic Courtès) writes:

> So I think what we need to do is for “guix pull-ng” to build and install
> a complete ‘guix’ package, and to manage it pretty much like other
> packages is managed, 

I think that's very reasonable.  It seems more intuitive than the
current way 'guix pull' works.  I suspect that managing the installed
version of Guix via the same Guix mechanisms that we use to manage any
other package might be the best, most intuitive solution.

Would it simplify the problem if we packaged the "Guix client stuff",
the "Guix daemon stuff", and maybe the "Guix package definition stuff"
separately?  Then a user could just install the "Guix client stuff"
package if she wanted to upgrade the Guix client tools, or the "Guix
package definition stuff" package if she wanted to get the latest
package definitions.

> except not in the user’s main profile (because that could lead to
> undesirable behavior, 

If we don't store Guix in the user's main profile, where would it go?  A
system profile (like in GuixSD)?  What if another user wants to run a
different version of Guix?  It might be nice to let them do that.

It's not clear to me why it's riskier to store Guix in a profile rather
than outside the profile (but still in the store via the
$HOME/.config/guix/latest symlink), which is what 'guix pull' does now.
You seem to think it's riskier; I'm curious to know more about why.

> where upgrading Guix creates a new generation, 

Why should upgrading Guix NOT create a new generation?  I thought that a
new profile generation would be created any time you upgrade a package,
and I thought that was a good thing because it facilitates easy,
transactional roll-back.  Perhaps I'm missing something.

> or, in theory, unrecoverable problems, where you cannot roll back
> because previous generations use an old Guix that does not understand
> the new manifest format.)

Why would a change in manifest format be unrecoverable?  It looks like
each profile generation contains a manifest file.  Assuming that the new
Guix functions well enough to perform roll back, couldn't we just roll
back to the previous profile generation, where we would have both (1)
the old profile's manifest file, and (2) the previous Guix, which
understands that format?  Since rolling back a profile is basically just
a symlink flip, I think the new Guix could probably do that even if it
didn't understand the old manifest format.

> The difficulty is that ./configure && make && make install in Guix takes
> some time, and we probably wouldn’t want to do that on each ‘guix pull’
> invocation (esp. with Guile 2.2’s compilation times.)
>
> So we may have to provide substitutes of Guix itself, and arrange so
> that ‘guix pull’ pulls up to a tag for which we have substitutes.

What if we had a special package version of Guix (e.g., "v0.11.0") which
we kept up to date via some mechanism?  Maybe something as simple as a
Git hook could help increase the likelihood of that version being
substitutable.  For example, we could have a Git hook that prevents
someone from checking in a change if the latest Git tag does not
correspond to a Guix package version.  Maybe we can do better.

I actually think it would be a good thing if we can run "guix pull"
without substitutes available.  But it should use a substitute by
default, and "build from source" should be a fallback mechanism that the
user has to explicitly request, just like when installing new packages.
That would help avoid unexpectedly long "guix pull" invocations.

-- 
Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]

bug#22629: ‘guix pull’ and external dependencies

[Ludovic Courtès]

Hello,

Chris Marusich <cmmarusich@gmail.com> skribis:

> ludo@gnu.org (Ludovic Courtès) writes:
>
>> So I think what we need to do is for “guix pull-ng” to build and install
>> a complete ‘guix’ package, and to manage it pretty much like other
>> packages is managed, 
>
> I think that's very reasonable.  It seems more intuitive than the
> current way 'guix pull' works.  I suspect that managing the installed
> version of Guix via the same Guix mechanisms that we use to manage any
> other package might be the best, most intuitive solution.
>
> Would it simplify the problem if we packaged the "Guix client stuff",
> the "Guix daemon stuff", and maybe the "Guix package definition stuff"
> separately?  Then a user could just install the "Guix client stuff"
> package if she wanted to upgrade the Guix client tools, or the "Guix
> package definition stuff" package if she wanted to get the latest
> package definitions.

It would be bad to separate package definitions from the rest because
they are very much intertwined: package definitions depend on the
definition of ‘package’, on build system implementations, and so on.

We could have guix-sans-daemon though, if that helps (which I suspect is
not the case).

>> except not in the user’s main profile (because that could lead to
>> undesirable behavior, 
>
> If we don't store Guix in the user's main profile, where would it go?

In “some sort of a profile” in ~/.config/guix/latest or similar.

> It's not clear to me why it's riskier to store Guix in a profile rather
> than outside the profile (but still in the store via the
> $HOME/.config/guix/latest symlink), which is what 'guix pull' does now.
> You seem to think it's riskier; I'm curious to know more about why.

There’s the manifest format change issue I mentioned, or the inability
to roll back if you install a broken Guix.

>> where upgrading Guix creates a new generation, 
>
> Why should upgrading Guix NOT create a new generation?  I thought that a
> new profile generation would be created any time you upgrade a package,
> and I thought that was a good thing because it facilitates easy,
> transactional roll-back.  Perhaps I'm missing something.

I’m suggesting that upgrading Guix creates a new generation (so we agree
here), just not in the user’s profile.

>> or, in theory, unrecoverable problems, where you cannot roll back
>> because previous generations use an old Guix that does not understand
>> the new manifest format.)
>
> Why would a change in manifest format be unrecoverable?  It looks like
> each profile generation contains a manifest file.  Assuming that the new
> Guix functions well enough to perform roll back, couldn't we just roll
> back to the previous profile generation, where we would have both (1)
> the old profile's manifest file, and (2) the previous Guix, which
> understands that format?  Since rolling back a profile is basically just
> a symlink flip, I think the new Guix could probably do that even if it
> didn't understand the old manifest format.

Yeah, I think you’re right.  :-)

In general, I think my concern is more that we cannot promise that
downgrading Guix will work, considering the potential for on-disk format
changes.  It’s a bit theoretical, but not entirely sci-fi either.

>> The difficulty is that ./configure && make && make install in Guix takes
>> some time, and we probably wouldn’t want to do that on each ‘guix pull’
>> invocation (esp. with Guile 2.2’s compilation times.)
>>
>> So we may have to provide substitutes of Guix itself, and arrange so
>> that ‘guix pull’ pulls up to a tag for which we have substitutes.
>
> What if we had a special package version of Guix (e.g., "v0.11.0") which
> we kept up to date via some mechanism?  Maybe something as simple as a
> Git hook could help increase the likelihood of that version being
> substitutable.  For example, we could have a Git hook that prevents
> someone from checking in a change if the latest Git tag does not
> correspond to a Guix package version.  Maybe we can do better.

Right, we could do something like that.  There are still non-zero
chances that someone running ‘guix pull’ at an arbitrary point in time
will have to build locally, which is not great.

> I actually think it would be a good thing if we can run "guix pull"
> without substitutes available.  But it should use a substitute by
> default, and "build from source" should be a fallback mechanism that the
> user has to explicitly request, just like when installing new packages.
> That would help avoid unexpectedly long "guix pull" invocations.

Yes, using substitutes or falling back to source builds is always the
default.

Thanks for your feedback!

Ludo’.

Re: ‘guix pull’ and external dependencies

[Ludovic Courtès]

Hello,

Chris Marusich <cmmarusich@gmail.com> skribis:

> ludo@gnu.org (Ludovic Courtès) writes:
>
>> So I think what we need to do is for “guix pull-ng” to build and install
>> a complete ‘guix’ package, and to manage it pretty much like other
>> packages is managed, 
>
> I think that's very reasonable.  It seems more intuitive than the
> current way 'guix pull' works.  I suspect that managing the installed
> version of Guix via the same Guix mechanisms that we use to manage any
> other package might be the best, most intuitive solution.
>
> Would it simplify the problem if we packaged the "Guix client stuff",
> the "Guix daemon stuff", and maybe the "Guix package definition stuff"
> separately?  Then a user could just install the "Guix client stuff"
> package if she wanted to upgrade the Guix client tools, or the "Guix
> package definition stuff" package if she wanted to get the latest
> package definitions.

It would be bad to separate package definitions from the rest because
they are very much intertwined: package definitions depend on the
definition of ‘package’, on build system implementations, and so on.

We could have guix-sans-daemon though, if that helps (which I suspect is
not the case).

>> except not in the user’s main profile (because that could lead to
>> undesirable behavior, 
>
> If we don't store Guix in the user's main profile, where would it go?

In “some sort of a profile” in ~/.config/guix/latest or similar.

> It's not clear to me why it's riskier to store Guix in a profile rather
> than outside the profile (but still in the store via the
> $HOME/.config/guix/latest symlink), which is what 'guix pull' does now.
> You seem to think it's riskier; I'm curious to know more about why.

There’s the manifest format change issue I mentioned, or the inability
to roll back if you install a broken Guix.

>> where upgrading Guix creates a new generation, 
>
> Why should upgrading Guix NOT create a new generation?  I thought that a
> new profile generation would be created any time you upgrade a package,
> and I thought that was a good thing because it facilitates easy,
> transactional roll-back.  Perhaps I'm missing something.

I’m suggesting that upgrading Guix creates a new generation (so we agree
here), just not in the user’s profile.

>> or, in theory, unrecoverable problems, where you cannot roll back
>> because previous generations use an old Guix that does not understand
>> the new manifest format.)
>
> Why would a change in manifest format be unrecoverable?  It looks like
> each profile generation contains a manifest file.  Assuming that the new
> Guix functions well enough to perform roll back, couldn't we just roll
> back to the previous profile generation, where we would have both (1)
> the old profile's manifest file, and (2) the previous Guix, which
> understands that format?  Since rolling back a profile is basically just
> a symlink flip, I think the new Guix could probably do that even if it
> didn't understand the old manifest format.

Yeah, I think you’re right.  :-)

In general, I think my concern is more that we cannot promise that
downgrading Guix will work, considering the potential for on-disk format
changes.  It’s a bit theoretical, but not entirely sci-fi either.

>> The difficulty is that ./configure && make && make install in Guix takes
>> some time, and we probably wouldn’t want to do that on each ‘guix pull’
>> invocation (esp. with Guile 2.2’s compilation times.)
>>
>> So we may have to provide substitutes of Guix itself, and arrange so
>> that ‘guix pull’ pulls up to a tag for which we have substitutes.
>
> What if we had a special package version of Guix (e.g., "v0.11.0") which
> we kept up to date via some mechanism?  Maybe something as simple as a
> Git hook could help increase the likelihood of that version being
> substitutable.  For example, we could have a Git hook that prevents
> someone from checking in a change if the latest Git tag does not
> correspond to a Guix package version.  Maybe we can do better.

Right, we could do something like that.  There are still non-zero
chances that someone running ‘guix pull’ at an arbitrary point in time
will have to build locally, which is not great.

> I actually think it would be a good thing if we can run "guix pull"
> without substitutes available.  But it should use a substitute by
> default, and "build from source" should be a fallback mechanism that the
> user has to explicitly request, just like when installing new packages.
> That would help avoid unexpectedly long "guix pull" invocations.

Yes, using substitutes or falling back to source builds is always the
default.

Thanks for your feedback!

Ludo’.

Re: Offloading to use Guile-SSH instead of lsh

[Ludovic Courtès]

ludo@gnu.org (Ludovic Courtès) skribis:

> The ‘wip-guile-ssh’ branch is finally getting into shape.  As soon as
> a new Guile-SSH release is out, I’ll merge it on master.

[...]

> Before we put it in production on hydra.gnu.org :-)

It’s now used on hydra.gnu.org and seems to work well, though we’ll have
to see if new bugs pop up.

There were few hickups as evidenced by the commits in that area
yesterday ;-), but nothing serious.

I added a new sub-command to test whether offloading is properly set up
and that helped a lot:

--8<---------------cut here---------------start------------->8---
~# guix offload test ~hydra/maintenance/hydra/machines.scm.test
guix offload: testing 1 build machines defined in '/home/hydra/maintenance/hydra/machines.scm.test'...
guix offload: 'hydra-slave1.netris.org' is running guile (GNU Guile) 2.0.11
guix offload: Guix is usable on 'hydra-slave1.netris.org' (test returned "/gnu/store/883yjkl46dxw9mzykykmbs0yzwyxm17z-test")
sending 1 store item to 'hydra-slave1.netris.org'...
exporting path `/gnu/store/x9ab32kjqldyqlcb96n5i6ipiwfz5npc-export-test'
guix offload: 'hydra-slave1.netris.org' successfully imported '/gnu/store/x9ab32kjqldyqlcb96n5i6ipiwfz5npc-export-test'
guix offload: successfully imported '/gnu/store/z289jx28vzwfahvqa8h622ljzfja2hv4-import-test' from 'hydra-slave1.netris.org'
--8<---------------cut here---------------end--------------->8---

Ludo’.