Malconnected machines & guix pull

Malconnected machines & guix pull

[Marco van Zwetselaar]

All,

Is there a way to do a 'guix pull' from an internet-disconnected machine?

I can tell it to fetch master.tar.gz from file:

$ guix pull --url=file:///tmp/master.tar.gz
unpacking 
'/gnu/store/d6x5rcb58fqfsyhr08v65bii1pfm7jz1-guix-latest.tar.gz'...
The following derivations will be built:
    /gnu/store/z6lxp83lkk6ba719i4c0x3rv087kgjkk-guix-latest.drv
    /gnu/store/vx592ns1hk7xs038c1h5kjnzlb7cxzyw-module-import.drv
/gnu/store/4p141cjhz9wfkgi3gi5nk11d5f0jpplg-module-import-compiled.drv
The following files will be downloaded:
    /gnu/store/fmxxkrpwajcnb9cyncgh4f4z6ybknl1g-guile-2.0.11
    ...

But then off it goes again looking for substitutes from hydra. Guix pull 
offers neither --no-substitutes nor --substitute-urls.

In fact, what I am really looking for is a --substitute-urls option for 
'guix pull', or a way to separate the pull and the subsequent build, so 
that I can make it connect to a 'guix publish'-ing server.

Context: internet connectivity at my research institute (in Kilimanjaro, 
Tanzania) is often such that I need to restart a guix pull four or five 
times. Multiply that by 6 machines and you see why I'm good friends with 
guix publish ;-)

Cheers
Marco

Re: Malconnected machines & guix pull

[Ricardo Wurmus]

Marco van Zwetselaar <zwets@zwets.com> writes:

> In fact, what I am really looking for is a --substitute-urls option for 
> 'guix pull', or a way to separate the pull and the subsequent build, so 
> that I can make it connect to a 'guix publish'-ing server.

Instead of “guix pull” may I suggest using git instead?  Especially,
when you have multiple machines on which Guix should run, it may make
sense to just clone the Guix git repository once from the Internet,
clone it to the other machines via local network, and then update once
in a while via “git pull” (not “guix pull”).

You can create a link “~/.config/guix/latest” pointing to, say,
“~/dev/guix” (if that’s the directory holding the cloned repository).

When you update with “git pull” you’ll only have to download recent
changes, not the whole Guix code.

Does this make sense?

~~ Ricardo

Re: Malconnected machines & guix pull

[Pjotr Prins]

On Thu, Mar 10, 2016 at 04:44:30PM +0100, Ricardo Wurmus wrote:
> 
> Marco van Zwetselaar <zwets@zwets.com> writes:
> 
> > In fact, what I am really looking for is a --substitute-urls option for 
> > 'guix pull', or a way to separate the pull and the subsequent build, so 
> > that I can make it connect to a 'guix publish'-ing server.
> 
> Instead of “guix pull” may I suggest using git instead?  Especially,
> when you have multiple machines on which Guix should run, it may make
> sense to just clone the Guix git repository once from the Internet,
> clone it to the other machines via local network, and then update once
> in a while via “git pull” (not “guix pull”).
> 
> You can create a link “~/.config/guix/latest” pointing to, say,
> “~/dev/guix” (if that’s the directory holding the cloned repository).
> 
> When you update with “git pull” you’ll only have to download recent
> changes, not the whole Guix code.
> 
> Does this make sense?
> 
> ~~ Ricardo

This has the additional benefit of 'fixating' the dependencies and
having some control over status of packages. This is why we have a
gn-stable checkout of guix which is shared among deployments. And it
can save a lot of downloads. 

Pj.
-- 

[PATCH] doc: Add footnote about pointing ~/.config/guix/latest at user's checkout

[Christopher Allan Webber]

[-- Attachment #1: Type: text/plain, Size: 434 bytes --]

Ricardo Wurmus writes:

> You can create a link “~/.config/guix/latest” pointing to, say,
> “~/dev/guix” (if that’s the directory holding the cloned repository).
>
> When you update with “git pull” you’ll only have to download recent
> changes, not the whole Guix code.

This is a nice feature that few people know about.  I figured it might
be nice to document it.  Here's a patch that attempts to do so!


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-doc-Add-footnote-about-pointing-.config-guix-latest-.patch --]
[-- Type: text/x-patch, Size: 1413 bytes --]

From 96a7b39ae17f5628bdebec95630af722e1b11561 Mon Sep 17 00:00:00 2001
From: Christopher Allan Webber <cwebber@dustycloud.org>
Date: Thu, 10 Mar 2016 09:52:18 -0800
Subject: [PATCH] doc: Add footnote about pointing ~/.config/guix/latest at
 user's checkout

* doc/contributing.texi (Running Guix Before It Is Installed): New footnote.
---
 doc/contributing.texi | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/doc/contributing.texi b/doc/contributing.texi
index 3dbd3db..2e13ee3 100644
--- a/doc/contributing.texi
+++ b/doc/contributing.texi
@@ -140,7 +140,13 @@ necessary to support this, including @env{PATH} and @env{GUILE_LOAD_PATH}.
 Note that @command{./pre-inst-env guix pull} does @emph{not} upgrade the
 local source tree; it simply updates the @file{~/.config/guix/latest}
 symlink (@pxref{Invoking guix pull}).  Run @command{git pull} instead if
-you want to upgrade your local source tree.
+you want to upgrade your local source tree.@footnote{If you would like
+to set up @command{guix} to use your git checkout, you can point the
+@file{~/.config/guix/latest} symlink to your git checkout directory.
+If you are the sole user of your system, you may also consider pointing
+the @file{/root/.config/guix/latest} symlink to point to
+@file{~/.config/guix/latest}; this way it will always use the same
+@command{guix} your user does.}
 
 
 @node The Perfect Setup
-- 
2.6.3

Re: Malconnected machines & guix pull

[Ricardo Wurmus]

Pjotr Prins <pjotr.public12@thebird.nl> writes:

> On Thu, Mar 10, 2016 at 04:44:30PM +0100, Ricardo Wurmus wrote:
>> 
>> Marco van Zwetselaar <zwets@zwets.com> writes:
>> 
>> > In fact, what I am really looking for is a --substitute-urls option for 
>> > 'guix pull', or a way to separate the pull and the subsequent build, so 
>> > that I can make it connect to a 'guix publish'-ing server.
>> 
>> Instead of “guix pull” may I suggest using git instead?  Especially,
>> when you have multiple machines on which Guix should run, it may make
>> sense to just clone the Guix git repository once from the Internet,
>> clone it to the other machines via local network, and then update once
>> in a while via “git pull” (not “guix pull”).
>> 
>> You can create a link “~/.config/guix/latest” pointing to, say,
>> “~/dev/guix” (if that’s the directory holding the cloned repository).
>> 
>> When you update with “git pull” you’ll only have to download recent
>> changes, not the whole Guix code.
>> 
>> Does this make sense?
>> 
>> ~~ Ricardo
>
> This has the additional benefit of 'fixating' the dependencies and
> having some control over status of packages. This is why we have a
> gn-stable checkout of guix which is shared among deployments. And it
> can save a lot of downloads. 

Yeah, this achieves what in traditional package management scenarios is
done by a repository mirror.  In combination with local build hosts you
can ensure that all packages at the current checkout are available as
binary substitutes.

~~ Ricardo

Re: [PATCH] doc: Add footnote about pointing ~/.config/guix/latest at user's checkout

[Ricardo Wurmus]

Christopher Allan Webber <cwebber@dustycloud.org> writes:

> Ricardo Wurmus writes:
>
>> You can create a link “~/.config/guix/latest” pointing to, say,
>> “~/dev/guix” (if that’s the directory holding the cloned repository).
>>
>> When you update with “git pull” you’ll only have to download recent
>> changes, not the whole Guix code.
>
> This is a nice feature that few people know about.  I figured it might
> be nice to document it.  Here's a patch that attempts to do so!

> From 96a7b39ae17f5628bdebec95630af722e1b11561 Mon Sep 17 00:00:00 2001
> From: Christopher Allan Webber <cwebber@dustycloud.org>
> Date: Thu, 10 Mar 2016 09:52:18 -0800
> Subject: [PATCH] doc: Add footnote about pointing ~/.config/guix/latest at
>  user's checkout

> * doc/contributing.texi (Running Guix Before It Is Installed): New footnote.
> ---
>  doc/contributing.texi | 8 +++++++-
>  1 file changed, 7 insertions(+), 1 deletion(-)

> diff --git a/doc/contributing.texi b/doc/contributing.texi
> index 3dbd3db..2e13ee3 100644
> --- a/doc/contributing.texi
> +++ b/doc/contributing.texi
> @@ -140,7 +140,13 @@ necessary to support this, including @env{PATH} and @env{GUILE_LOAD_PATH}.
>  Note that @command{./pre-inst-env guix pull} does @emph{not} upgrade the
>  local source tree; it simply updates the @file{~/.config/guix/latest}
>  symlink (@pxref{Invoking guix pull}).  Run @command{git pull} instead if
> -you want to upgrade your local source tree.
> +you want to upgrade your local source tree.@footnote{If you would like
> +to set up @command{guix} to use your git checkout, you can point the
> +@file{~/.config/guix/latest} symlink to your git checkout directory.
> +If you are the sole user of your system, you may also consider pointing
> +the @file{/root/.config/guix/latest} symlink to point to
> +@file{~/.config/guix/latest}; this way it will always use the same
> +@command{guix} your user does.}

In the last sentence, do you miss an “as”?

  “this way it will always use the same @command{guix} [as] your user
  does.”

Otherwise looks good to me.  Thanks!

~~ Ricardo

Re: [PATCH] doc: Add footnote about pointing ~/.config/guix/latest at user's checkout

[Ludovic Courtès]

Christopher Allan Webber <cwebber@dustycloud.org> skribis:

> From 96a7b39ae17f5628bdebec95630af722e1b11561 Mon Sep 17 00:00:00 2001
> From: Christopher Allan Webber <cwebber@dustycloud.org>
> Date: Thu, 10 Mar 2016 09:52:18 -0800
> Subject: [PATCH] doc: Add footnote about pointing ~/.config/guix/latest at
>  user's checkout
>
> * doc/contributing.texi (Running Guix Before It Is Installed): New footnote.
> ---
>  doc/contributing.texi | 8 +++++++-
>  1 file changed, 7 insertions(+), 1 deletion(-)
>
> diff --git a/doc/contributing.texi b/doc/contributing.texi
> index 3dbd3db..2e13ee3 100644
> --- a/doc/contributing.texi
> +++ b/doc/contributing.texi
> @@ -140,7 +140,13 @@ necessary to support this, including @env{PATH} and @env{GUILE_LOAD_PATH}.
>  Note that @command{./pre-inst-env guix pull} does @emph{not} upgrade the
>  local source tree; it simply updates the @file{~/.config/guix/latest}
>  symlink (@pxref{Invoking guix pull}).  Run @command{git pull} instead if
> -you want to upgrade your local source tree.
> +you want to upgrade your local source tree.@footnote{If you would like
> +to set up @command{guix} to use your git checkout, you can point the
> +@file{~/.config/guix/latest} symlink to your git checkout directory.

OK with s/git/Git/, thanks!

Ludo'.

Re: [PATCH] doc: Add footnote about pointing ~/.config/guix/latest at user's checkout

[Christopher Allan Webber]

Ricardo Wurmus writes:

> In the last sentence, do you miss an “as”?
>
>   “this way it will always use the same @command{guix} [as] your user
>   does.”
>
> Otherwise looks good to me.  Thanks!
>
> ~~ Ricardo

Okay I integrated both your and Ludo's suggestions and pushed.  Thanks!

Re: Malconnected machines & guix pull

[Marco van Zwetselaar]

On 10/03/16 18:44, Ricardo Wurmus wrote:

> Instead of “guix pull” may I suggest using git instead?  Especially,
> when you have multiple machines on which Guix should run, it may make
> sense to just clone the Guix git repository once from the Internet,
> clone it to the other machines via local network, and then update once
> in a while via “git pull” (not “guix pull”).
>
> You can create a link “~/.config/guix/latest” pointing to, say,
> “~/dev/guix” (if that’s the directory holding the cloned repository).
>
> When you update with “git pull” you’ll only have to download recent
> changes, not the whole Guix code.
>
> Does this make sense?

It's a good start for sure, thx!  However, I now get loads of

;;; note: source file /home/zwets/.config/guix/latest/gnu/packages/{...}.scm
;;;       newer than compiled /gnu/store/3lg5c1nidbj0kjdz5b63hn3vp29kzf0s-guix-0.9.0.c3f29bc/share/guile/site/2.0/gnu/packages/{...}.go

whenever I run guix.  Do I need to build the pulled source,
then point my guix command elsewhere?

My `which guix` is /usr/local/bin/guix pointing at
/var/guix/profiles/per-user/root/guix-profile/bin/guix,
the way the installation instructions suggest.

Cheers
Marco

Re: Malconnected machines & guix pull

[Ricardo Wurmus]

Marco van Zwetselaar <zwets@zwets.com> writes:

> On 10/03/16 18:44, Ricardo Wurmus wrote:
>
>> Instead of “guix pull” may I suggest using git instead?  Especially,
>> when you have multiple machines on which Guix should run, it may make
>> sense to just clone the Guix git repository once from the Internet,
>> clone it to the other machines via local network, and then update once
>> in a while via “git pull” (not “guix pull”).
>>
>> You can create a link “~/.config/guix/latest” pointing to, say,
>> “~/dev/guix” (if that’s the directory holding the cloned repository).
>>
>> When you update with “git pull” you’ll only have to download recent
>> changes, not the whole Guix code.
>>
>> Does this make sense?
>
> It's a good start for sure, thx!  However, I now get loads of
>
> ;;; note: source file /home/zwets/.config/guix/latest/gnu/packages/{...}.scm
> ;;;       newer than compiled /gnu/store/3lg5c1nidbj0kjdz5b63hn3vp29kzf0s-guix-0.9.0.c3f29bc/share/guile/site/2.0/gnu/packages/{...}.go
>
> whenever I run guix.  Do I need to build the pulled source,
> then point my guix command elsewhere?

Whenever I pull the latest version from git I also run “make” to compile
the modules.  The warnings will disappear then and things will be
faster.

~~ Ricardo