Why does `system reconfigure` need to `pull`?

Why does `system reconfigure` need to `pull`?

[45mg]

Hi Guix,

So I've just installed Guix System on my main machine, and right off the
bat, I want to thank everyone here for spending their time on this
amazing project. While there have been a few pain points (expect some
(minor) patches, hopefully!), the feeling of finally getting `guix
system init` to work and then instantly rebooting into the /exact same
system/ that I built in a VM over weeks is downright magical.

I'm writing to discuss one of the aforementioned pain points. It's one
of those things that seems insignificant at first, but I fear it may
impact my ability to keep using Guix. Let me explain.

Whenever I run `guix system reconfigure`, the first output I see from
the command is this:

Updating channel 'guix' from Git repository at 'https://git.savannah.gnu.org/git/guix.git'...

It then says it's fetching and indexing objects, authenticating N new
commits, etc. As far as I can tell, this stage is equivalent to `guix
pull`. It sometimes repeats this several times during the course of the
`reconfigure`.

I didn't think much of this at first, although I thought it was odd that
the manual did not mention a 'pull' stage [1].

However, I quickly realised that *this makes it impossible to
reconfigure without an internet connection*. The command will exit
unsuccessfully when it fails to pull [2].

This means that I cannot tweak my system *at all* without an internet
connection. Even to do something that doesn't require Guix to download
anything - like add a new user, or tweak my `bootloader-configuration`,
or fix my `network-manager-configuration` so I can connect to the
internet (see the chicken-and-egg issue there?), I must be connected to
the internet.

For me, this is a huge step backwards from conventional GNU/Linux
distributions in terms of usability. My internet connection tends to be
extremely flaky, with frequent outages for days or more. This will mean
I won't be able to make changes to my system at all during this period.

Now, I really want to believe that there's a way around this; that I
haven't read the docs enough, and there's some option or command to
reconfigure my system without pulling new commits. But I can't seem to
find any such thing. (`guix time-machine` has the same problem [3].)

Is there really no way to reconfigure my system without an internet
connection?

P.S. I tried to delve into the source to understand how `reconfigure`
works internally, but I was not able to understand much from
guix/scripts.scm and the associated files... which means I likely won't
be able to address this myself any time soon. Which concerns me all the
more, because while I have a stable connection for the next week or so,
I don't know how I'll continue using my system once this is no longer
the case...

---

[1] From '(guix) Invoking guix system':
‘reconfigure’
Build the operating system described in FILE, activate it, and
switch to it.

[2]
Updating channel 'guix' from Git repository at 'https://git.savannah.gnu.org/git/guix.git'...
Backtrace:
In guix/gexp.scm:
    926:4 19 (_ _)
In guix/store.scm:
  2095:12 18 (_ #<store-connection 256.100 7f389f90d2d0>)
  1412:11 17 (map/accumulate-builds #<store-connection 256.100 7f38…> …)
   1330:8 16 (call-with-build-handler #<procedure 7f388b8988a0 at g…> …)
  2210:25 15 (run-with-store #<store-connection 256.100 7f389f90d2d0> …)
In guix/gexp.scm:
   931:13 14 (_ _)
In guix/store.scm:
   2038:8 13 (_ _)
In guix/gexp.scm:
   300:22 12 (_ _)
In guix/store.scm:
   2038:8 11 (_ _)
In guix/gexp.scm:
   300:22 10 (_ _)
In guix/store.scm:
   2038:8  9 (_ _)
In guix/packages.scm:
  2050:11  8 (_ _)
In guix/build-system/channel.scm:
     43:2  7 (_ _)
In guix/store.scm:
  2082:38  6 (_ #<store-connection 256.100 7f389f90d2d0>)
In guix/channels.scm:
   559:23  5 (latest-channel-instances #<store-connection 256.100 7…> …)
   422:18  4 (latest-channel-instance #<store-connection 256.100 7f…> …)
In guix/git.scm:
    586:7  3 (update-cached-checkout _ #:connection-timeout _ # _ # _ …)
In git/bindings.scm:
     77:2  2 (raise-git-error _)
In ice-9/boot-9.scm:
  1685:16  1 (raise-exception _ #:continuable? _)
  1685:16  0 (raise-exception _ #:continuable? _)

ice-9/boot-9.scm:1685:16: In procedure raise-exception:
Git error: failed to resolve address for git.savannah.gnu.org: Name or service not known

[3]
$ guix describe -f channels > channels.scm
$ sudo guix time-machine -C channels.scm -- system reconfigure -L /etc/guix-system/ /etc/guix-system/config.scm
Updating channel 'guix' from Git repository at 'https://git.savannah.gnu.org/git/guix.git'...

Re: Why does `system reconfigure` need to `pull`?

[Ian Eure]

Hi 45mg,

45mg <45mg.writes@gmail.com> writes:

> Updating channel 'guix' from Git repository at
> 'https://git.savannah.gnu.org/git/guix.git'...
>
> It then says it's fetching and indexing objects, authenticating 
> N
> new commits, etc. As far as I can tell, this stage is equivalent 
> to
> `guix pull`. It sometimes repeats this several times during the
> course of the `reconfigure`.
>
> I didn't think much of this at first, although I thought it was 
> odd
> that the manual did not mention a 'pull' stage [1].
>

I’ve also noticed this, and wondered why it always pulls 
first. Even
if you have an internet connection, the pull makes the reconfigure
slower than it could be.

> Now, I really want to believe that there's a way around this; 
> that I
> haven't read the docs enough, and there's some option or command 
> to
> reconfigure my system without pulling new commits. But I can't 
> seem
> to find any such thing. (`guix time-machine` has the same 
> problem
> [3].)
>
> Is there really no way to reconfigure my system without an 
> internet
> connection?
>

I dug around in the --help output, but I didn’t see anything that
looked like it’d skip the pull.  It’d definitely be nice to have 
an
--offline or --no-pull option when reconfiguring.

The simplest path forward for you is to replace your Guix channels
with local clones of them, so `guix pull' uses your filesystem 
instead
of the network.  You can then `git pull' in your clone to get new
commits when you like, and `guix pull' after that to update Guix 
to
use them.

I’ve heard of Guix getting used in places without much 
connectivity
before, though I don’t know how it was accomplished. Maybe someone 
can
chime in with their workflow.

 -- Ian

Re: Why does `system reconfigure` need to `pull`?

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 995 bytes --]

45mg <45mg.writes@gmail.com> writes:

> So I've just installed Guix System on my main machine, and right off the
> bat, I want to thank everyone here for spending their time on this
> amazing project. While there have been a few pain points (expect some
> (minor) patches, hopefully!), the feeling of finally getting `guix
> system init` to work and then instantly rebooting into the /exact same
> system/ that I built in a VM over weeks is downright magical.
>
> I'm writing to discuss one of the aforementioned pain points. It's one
> of those things that seems insignificant at first, but I fear it may
> impact my ability to keep using Guix. Let me explain.
>
> Whenever I run `guix system reconfigure`, the first output I see from
> the command is this:
>
> Updating channel 'guix' from Git repository at 'https://git.savannah.gnu.org/git/guix.git'...

Do you have (current-guix) in your configuration?

The behaviour of (current-guix) can be changed, but I think it might use
the network.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

Re: Why does `system reconfigure` need to `pull`?

[45mg]

Hello again Guix,

Thanks, everyone, for the replies to my inital message. Reading through
them, I developed a hunch that that the issue might only occur when you
configure additional channels in your system configuration, and I think
I've been able to verify this.

Rutherther suggested that I share my configuration. I figured that my
sprawling multi-file configuration would present far more noise than
useful information - so, with much effort, I installed Guix into a VM,
then built up a minimal configuration from the initial template until I
could reproduce the issue.

As I suspected, the issue does not show up until you add additional
channels in your configuration file - in the case of the configuration
below, that means adding the `my-channels` definition and the
`modify-services` form. When you reconfigure after doing that, you
should see it updating the channel repositories as I originally
described.

--8<---------------cut here---------------start------------->8---
(use-modules (gnu)
             (gnu packages package-management)
             (guix channels))
(use-service-modules cups desktop networking ssh xorg)

(define my-channels
  (append
   (list (channel
          (name 'guix-science)
          (url "https://codeberg.org/guix-science/guix-science.git")
          (introduction
           (make-channel-introduction
            "b1fe5aaff3ab48e798a4cce02f0212bc91f423dc"
            (openpgp-fingerprint
             "CA4F 8CF4 37D7 478F DA05  5FD4 4213 7701 1A37 8446")))))
   %default-channels))

(operating-system
  (locale "en_US.utf8")
  (timezone "America/New_York")
  (keyboard-layout (keyboard-layout "us"))
  (host-name "guix-1")

  (users (cons* (user-account
                 (name "me")
                 (comment "Me")
                 (group "users")
                 (home-directory "/home/me")
                 (supplementary-groups '("wheel" "netdev" "audio" "video")))
                %base-user-accounts))

  (packages (append (specifications->packages (list "tmux" "neovim"))
                    %base-packages))

  (services
   (append (list
            (service openssh-service-type)
            (service network-manager-service-type)
            (service wpa-supplicant-service-type)
            (service ntp-service-type))
           (modify-services %base-services
             (guix-service-type
              config => (guix-configuration
                         (inherit config)
                         (channels my-channels)
                         (guix (guix-for-channels my-channels)))))))

  (bootloader (bootloader-configuration
               (bootloader grub-bootloader)
               (targets (list "/dev/sda"))
               (keyboard-layout keyboard-layout)))
  (swap-devices (list (swap-space
                        (target (uuid
                                 "87ca086c-59c8-4cd5-8b69-355afbee886a")))))

  (file-systems (cons* (file-system
                         (mount-point "/")
                         (device (uuid
                                  "2d3eae58-a22e-4bfa-9929-0bcfc47c163a"
                                  'ext4))
                         (type "ext4")) %base-file-systems)))
--8<---------------cut here---------------end--------------->8---

Re: Why does `system reconfigure` need to `pull`?

[Tomas Volf]

[-- Attachment #1: Type: text/plain, Size: 934 bytes --]


Hello,

45mg <45mg.writes@gmail.com> writes:

> [..]
>
> As I suspected, the issue does not show up until you add additional
> channels in your configuration file - in the case of the configuration
> below, that means adding the `my-channels` definition and the
> `modify-services` form. When you reconfigure after doing that, you
> should see it updating the channel repositories as I originally
> described.
>
> [..]
>                          (guix (guix-for-channels my-channels)))))))

`guix-for-channels' does not cache the computation, that is a known
defect.  There are workarounds to make it cache, but they all suck in
one way or another.

For single-user system, you can just not use it, and access the
additional channels from your guix-pulled profile.

Have a nice day,
Tomas

-- 
There are only two hard things in Computer Science:
cache invalidation, naming things and off-by-one errors.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 853 bytes --]

Docs: Clarify use-cases of `guix-for-channels`?

[45mg]

Hi Tomas,

I finally understand that it's my use of `guix-for-channels` that's
making it pull when I reconfigure. So, my issue is solved by removing
the `(guix)` field from the `(guix-configuration)` form in the
configuration I posted.

All I was trying to do was specify additional channels in my system
configuration, without having to write anything in
~/.config/guix/channels.scm (I prefer for everything related to my
system configuration to be tracked by git in a single root-owned
directory). It looks like I only need the `(channels)` field of
`(guix-configuration)` for that. Is that correct?

This would have been obvious to me if I had understood what
`guix-for-channels` is actually /for/ from the start. The only reason I
ended up using it in the first place is that I was reading through
'(guix) Channels' to understand how to configure additional ones, and
then I saw this in '(guix) Customizing the System-Wide Guix':

> If you’re running Guix System or building system images with it, maybe
> you will want to customize the system-wide ‘guix’ it
> provides—specifically, ‘/run/current-system/profile/bin/guix’.  For
> example, you might want to provide additional channels or to pin its
> revision.
> 
>    This can be done using the ‘guix-for-channels’ procedure, which
> returns a package for the given channels, and using it as part of your
> operating system configuration, as in this example:
> 
>      (use-modules (gnu packages package-management)
>                   (guix channels))
> 
>      (define my-channels
>        ;; Channels that should be available to
>        ;; /run/current-system/profile/bin/guix.
>        (append
>         (list (channel
>                (name 'guix-science)
>                (url "https://github.com/guix-science/guix-science")
>                (branch "master")))
>         %default-channels))
> 
>      (operating-system
>        ;; ...
>        (services
>          ;; Change the package used by 'guix-service-type'.
>          (modify-services %base-services
>            (guix-service-type
>             config => (guix-configuration
>                        (inherit config)
>                        (channels my-channels)
>                        (guix (guix-for-channels my-channels)))))))
> 
>    The resulting operating system will have both the ‘guix’ and the
> ‘guix-science’ channels visible by default.  The ‘channels’ field of
> ‘guix-configuration’ above further ensures that
> ‘/etc/guix/channels.scm’, which is used by ‘guix pull’, specifies the
> same set of channels (*note ‘channels’ field of ‘guix-configuration’:
> guix-configuration-channels.).

If I had really been paying close attention, I would have noticed
phrasing like '/maybe/ you will want', and realised that using
`guix-for-channels` is /optional/. But since this subsection is under
the 'Channels' section of the manual, which explains the basics of using
channels - and does not provide any other snippet explaining how to
configure channels in the system configuration file - I assumed that the
code snippet quoted above is THE Official Way™ to configure additional
channels directly in the system configuration, without using
~/.config/guix/channels.scm; and copied it near-verbatim.

I think this could be clarified a little, so that other people who are
in a hurry to set things up for the first time don't make the same
mistake as me, and then think that Guix needs an internet connection
just to reconfigure the system.

I could write a patch to the manual for this. Here's what I'm proposing
to do:

- Add a snippet to '(guix) Specifying Additional Channels', explaining
  how to configure additional channels within the system configuration
  file (as per the second paragraph of this message).

- In '(guix) Customizing the System-Wide Guix', explain more clearly
  where `guix-for-channels` would be useful. As I understand it, the
  main use-case is for fresh system installations where running `guix
  pull` would be a hassle, such as on an installer ISO. Are there other
  significant use-cases? Any other reasons why one might want to use it
  on their main system, despite the hassle of it pulling every time and
  not caching the derivation? (We should also document that limitation,
  unless it's likely to change in the near future.)

Re: Docs: Clarify use-cases of `guix-for-channels`?

[Tomas Volf]

[-- Attachment #1: Type: text/plain, Size: 5906 bytes --]

Hi,

45mg <45mg.writes@gmail.com> writes:

> Hi Tomas,
>
> I finally understand that it's my use of `guix-for-channels` that's
> making it pull when I reconfigure. So, my issue is solved by removing
> the `(guix)` field from the `(guix-configuration)` form in the
> configuration I posted.
>
> All I was trying to do was specify additional channels in my system
> configuration, without having to write anything in
> ~/.config/guix/channels.scm (I prefer for everything related to my
> system configuration to be tracked by git in a single root-owned
> directory). It looks like I only need the `(channels)` field of
> `(guix-configuration)` for that. Is that correct?

Yes, I believe so.  Setting the (channels) field will create
/etc/guix/channels.scm file, which, in absence of
~/.config/guix/channels.scm, will be used.

>
> This would have been obvious to me if I had understood what
> `guix-for-channels` is actually /for/ from the start. The only reason I
> ended up using it in the first place is that I was reading through
> '(guix) Channels' to understand how to configure additional ones, and
> then I saw this in '(guix) Customizing the System-Wide Guix':
>
>> If you’re running Guix System or building system images with it, maybe
>> you will want to customize the system-wide ‘guix’ it
>> provides—specifically, ‘/run/current-system/profile/bin/guix’.  For
>> example, you might want to provide additional channels or to pin its
>> revision.
>> 
>>    This can be done using the ‘guix-for-channels’ procedure, which
>> returns a package for the given channels, and using it as part of your
>> operating system configuration, as in this example:
>> 
>>      (use-modules (gnu packages package-management)
>>                   (guix channels))
>> 
>>      (define my-channels
>>        ;; Channels that should be available to
>>        ;; /run/current-system/profile/bin/guix.
>>        (append
>>         (list (channel
>>                (name 'guix-science)
>>                (url "https://github.com/guix-science/guix-science")
>>                (branch "master")))
>>         %default-channels))
>> 
>>      (operating-system
>>        ;; ...
>>        (services
>>          ;; Change the package used by 'guix-service-type'.
>>          (modify-services %base-services
>>            (guix-service-type
>>             config => (guix-configuration
>>                        (inherit config)
>>                        (channels my-channels)
>>                        (guix (guix-for-channels my-channels)))))))
>> 
>>    The resulting operating system will have both the ‘guix’ and the
>> ‘guix-science’ channels visible by default.  The ‘channels’ field of
>> ‘guix-configuration’ above further ensures that
>> ‘/etc/guix/channels.scm’, which is used by ‘guix pull’, specifies the
>> same set of channels (*note ‘channels’ field of ‘guix-configuration’:
>> guix-configuration-channels.).
>
> If I had really been paying close attention, I would have noticed
> phrasing like '/maybe/ you will want', and realised that using
> `guix-for-channels` is /optional/. But since this subsection is under
> the 'Channels' section of the manual, which explains the basics of using
> channels - and does not provide any other snippet explaining how to
> configure channels in the system configuration file - I assumed that the
> code snippet quoted above is THE Official Way™ to configure additional
> channels directly in the system configuration, without using
> ~/.config/guix/channels.scm; and copied it near-verbatim.
>
> I think this could be clarified a little, so that other people who are
> in a hurry to set things up for the first time don't make the same
> mistake as me, and then think that Guix needs an internet connection
> just to reconfigure the system.
>
> I could write a patch to the manual for this. Here's what I'm proposing
> to do:
>
> - Add a snippet to '(guix) Specifying Additional Channels', explaining
>   how to configure additional channels within the system configuration
>   file (as per the second paragraph of this message).

The (channels) field is fairly new, so it is well possible some
documentation might be missing.  The suggested addition might be useful,
but the thing is, it is not really required.  Usually you would use the
~/.config/guix/channels.scm, since guix pull is per user.  So it should
be clearly stated that (channels) are to be used if you intend to set
default channels for every user on the machine.

>
> - In '(guix) Customizing the System-Wide Guix', explain more clearly
>   where `guix-for-channels` would be useful. As I understand it, the
>   main use-case is for fresh system installations where running `guix
>   pull` would be a hassle, such as on an installer ISO. Are there other
>   significant use-cases?

Sure there are, guix pull is fairly slow operation, so any system that
does not have guix pull done as part of normal workflow might benefit
from setting the (guix) field to a custom guix (produced for example by
guix-for-channels).  I set the (guix) field for all systems except my
laptop.

>   Any other reasons why one might want to use it on their main system,
>   despite the hassle of it pulling every time and not caching the
>   derivation?

There is a workaround[0], however I would not recommend using it on any
system that actually guix-pulls.

>   (We should also document that limitation, unless it's likely to
>   change in the near future.)

However I agree the current limitation should be documented (and so
should be bug #74396).

Tomas

0: https://wolfsden.cz/blog/post/what-goes-into-guix-shaped-hole.html

-- 
There are only two hard things in Computer Science:
cache invalidation, naming things and off-by-one errors.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 853 bytes --]