bug#24993: Documentation: 'guix pull' needed during SD installation

bug#24993: Documentation: 'guix pull' needed during SD installation

[Christopher Howard]

In the online installation docs at
<https://www.gnu.org/software/guix/manual/html_node/Proceeding-with-the-Installation.html#Proceeding-with-the-Installation>,
please insert the command 'guix pull' before the "guix system init"
command. Otherwise installation can fail due to missing binary packages,
which creates confusion for newbie installers like myself. Thank you!

-- 
Christopher Howard, Computer Assistant
Alaska Satellite Internet
3239 La Ree Way, Fairbanks, AK 99709
907-451-0088 or 888-396-5623 (toll free)
fax: 888-260-3584
mailto:christopher@alaskasi.com
http://www.alaskasatelliteinternet.com

bug#24993: Documentation: 'guix pull' needed during SD installation

[Leo Famulari]

On Tue, Nov 22, 2016 at 01:25:57PM -0900, Christopher Howard wrote:

Welcome Christopher!

> In the online installation docs at
> <https://www.gnu.org/software/guix/manual/html_node/Proceeding-with-the-Installation.html#Proceeding-with-the-Installation>,
> please insert the command 'guix pull' before the "guix system init"
> command. Otherwise installation can fail due to missing binary packages,
> which creates confusion for newbie installers like myself. Thank you!

Can you give more detail about the problems you ran into? We want `guix
system init` to "just work".

In general, if some binary fails to be substituted [0], adding
--fallback to the command should cause the package to be built from
source, working around the issue.

I think that running `guix pull` before `guix system init` is not the
right choice for newbies, although there are some trade-offs [1].

Most new users will be installing from the binary GuixSD installer
offered on our web site. This installer is built from a Git commit that
we have tested to ensure that everything should work. We strive to keep
our master branch "deployable", but of course we make mistakes
sometimes. `guix pull` draws from the HEAD commit of the master branch,
and so new users might end up with a broken system as their first
experience.  GuixSD is extremely robust once installed — you can always
roll-back — so it's a good idea to initialize from a known good commit
and then update.

Additionally, it's likely that we will not yet have built binary
substitutes for the most recent changes on the master branch, so new
users will end up building some things from source if they use `guix
pull`.

[0] Guix is a hybrid of build-from-source and binary package management
systems. It's really a build-from-source distro, but we offer so-called
"binary substitutes", and they are transparently substituted for a
source build if they are available and the user has authorized our
substitutes server. But, if Guix expects a package to be substituted
and something goes wrong, the action will fail. Using --fallback works
around this and builds from source even when the substitution fails.

[1] I think the main drawback of installing from the release tag without
`guix pull` is that the initial generation of the GuixSD system will be
lacking important upstream bug fixes. I think users should `guix pull &&
guix system reconfigure` immediately after initializing the system. Guix
actually nags users to do this:

http://git.savannah.gnu.org/cgit/guix.git/tree/guix/scripts/system.scm?id=08b3e4a97066c9baaf39e3df7c2dd9c39e693ead#n577

bug#24993: Documentation: 'guix pull' needed during SD installation

[Christopher Howard]

Hi. What I remember happening was that on some package I got an error
something like "could not get package such-and-such, may be caused by
network errors" which wasn't fixed by running the command again. Then I
ran with --fallback as recommended, but then it seemed to be building
the entire gnome desktop from source. After a few hours of this it was
still compiling low-level gnome packages, so I gave up, and the folks at
#guix irc suggested guix pull.

To be honest, I can't remember which package it was. I'll try again with
a fresh installation.

On 11/22/2016 07:46 PM, Leo Famulari wrote:
> On Tue, Nov 22, 2016 at 01:25:57PM -0900, Christopher Howard wrote:
> 
> Welcome Christopher!
> 
>> In the online installation docs at
>> <https://www.gnu.org/software/guix/manual/html_node/Proceeding-with-the-Installation.html#Proceeding-with-the-Installation>,
>> please insert the command 'guix pull' before the "guix system init"
>> command. Otherwise installation can fail due to missing binary packages,
>> which creates confusion for newbie installers like myself. Thank you!
> 
> Can you give more detail about the problems you ran into? We want `guix
> system init` to "just work".
> 
> In general, if some binary fails to be substituted [0], adding
> --fallback to the command should cause the package to be built from
> source, working around the issue.
> 
> I think that running `guix pull` before `guix system init` is not the
> right choice for newbies, although there are some trade-offs [1].
> 
> Most new users will be installing from the binary GuixSD installer
> offered on our web site. This installer is built from a Git commit that
> we have tested to ensure that everything should work. We strive to keep
> our master branch "deployable", but of course we make mistakes
> sometimes. `guix pull` draws from the HEAD commit of the master branch,
> and so new users might end up with a broken system as their first
> experience.  GuixSD is extremely robust once installed — you can always
> roll-back — so it's a good idea to initialize from a known good commit
> and then update.
> 
> Additionally, it's likely that we will not yet have built binary
> substitutes for the most recent changes on the master branch, so new
> users will end up building some things from source if they use `guix
> pull`.
> 
> [0] Guix is a hybrid of build-from-source and binary package management
> systems. It's really a build-from-source distro, but we offer so-called
> "binary substitutes", and they are transparently substituted for a
> source build if they are available and the user has authorized our
> substitutes server. But, if Guix expects a package to be substituted
> and something goes wrong, the action will fail. Using --fallback works
> around this and builds from source even when the substitution fails.
> 
> [1] I think the main drawback of installing from the release tag without
> `guix pull` is that the initial generation of the GuixSD system will be
> lacking important upstream bug fixes. I think users should `guix pull &&
> guix system reconfigure` immediately after initializing the system. Guix
> actually nags users to do this:
> 
> http://git.savannah.gnu.org/cgit/guix.git/tree/guix/scripts/system.scm?id=08b3e4a97066c9baaf39e3df7c2dd9c39e693ead#n577
> 

-- 
Christopher Howard, Computer Assistant
Alaska Satellite Internet
3239 La Ree Way, Fairbanks, AK 99709
907-451-0088 or 888-396-5623 (toll free)
fax: 888-260-3584
mailto:christopher@alaskasi.com
http://www.alaskasatelliteinternet.com

bug#24993: Documentation: 'guix pull' needed during SD installation

[Christopher Howard]

The error is:

"
guix substitute: error: download from
'https://mirror.hydra.gnu.org/nar/zrwslk773<...snip...>e2fsck-static-1.42.13'
failed 410, "Gone"
killing process 859
guix system: error: build failed: some substitutes for the outputs of
derivation `/gnu/store/kjzga<...snip...>-e2fsck-static-1.42..13.drv'
failed (usually happens due to networking issues); try `--fallback' to
build derivation from source
"

Sorry for snipping the file names, but I have to type it all in manually
to this email.

On 11/23/2016 07:46 AM, Christopher Howard wrote:
> Hi. What I remember happening was that on some package I got an error
> something like "could not get package such-and-such, may be caused by
> network errors" which wasn't fixed by running the command again. Then I
> ran with --fallback as recommended, but then it seemed to be building
> the entire gnome desktop from source. After a few hours of this it was
> still compiling low-level gnome packages, so I gave up, and the folks at
> #guix irc suggested guix pull.
> 
> To be honest, I can't remember which package it was. I'll try again with
> a fresh installation.
> 
> On 11/22/2016 07:46 PM, Leo Famulari wrote:
>> On Tue, Nov 22, 2016 at 01:25:57PM -0900, Christopher Howard wrote:
>>
>> Welcome Christopher!
>>
>>> In the online installation docs at
>>> <https://www.gnu.org/software/guix/manual/html_node/Proceeding-with-the-Installation.html#Proceeding-with-the-Installation>,
>>> please insert the command 'guix pull' before the "guix system init"
>>> command. Otherwise installation can fail due to missing binary packages,
>>> which creates confusion for newbie installers like myself. Thank you!
>>
>> Can you give more detail about the problems you ran into? We want `guix
>> system init` to "just work".
>>
>> In general, if some binary fails to be substituted [0], adding
>> --fallback to the command should cause the package to be built from
>> source, working around the issue.
>>
>> I think that running `guix pull` before `guix system init` is not the
>> right choice for newbies, although there are some trade-offs [1].
>>
>> Most new users will be installing from the binary GuixSD installer
>> offered on our web site. This installer is built from a Git commit that
>> we have tested to ensure that everything should work. We strive to keep
>> our master branch "deployable", but of course we make mistakes
>> sometimes. `guix pull` draws from the HEAD commit of the master branch,
>> and so new users might end up with a broken system as their first
>> experience.  GuixSD is extremely robust once installed — you can always
>> roll-back — so it's a good idea to initialize from a known good commit
>> and then update.
>>
>> Additionally, it's likely that we will not yet have built binary
>> substitutes for the most recent changes on the master branch, so new
>> users will end up building some things from source if they use `guix
>> pull`.
>>
>> [0] Guix is a hybrid of build-from-source and binary package management
>> systems. It's really a build-from-source distro, but we offer so-called
>> "binary substitutes", and they are transparently substituted for a
>> source build if they are available and the user has authorized our
>> substitutes server. But, if Guix expects a package to be substituted
>> and something goes wrong, the action will fail. Using --fallback works
>> around this and builds from source even when the substitution fails.
>>
>> [1] I think the main drawback of installing from the release tag without
>> `guix pull` is that the initial generation of the GuixSD system will be
>> lacking important upstream bug fixes. I think users should `guix pull &&
>> guix system reconfigure` immediately after initializing the system. Guix
>> actually nags users to do this:
>>
>> http://git.savannah.gnu.org/cgit/guix.git/tree/guix/scripts/system.scm?id=08b3e4a97066c9baaf39e3df7c2dd9c39e693ead#n577
>>
> 

-- 
Christopher Howard, Computer Assistant
Alaska Satellite Internet
3239 La Ree Way, Fairbanks, AK 99709
907-451-0088 or 888-396-5623 (toll free)
fax: 888-260-3584
mailto:christopher@alaskasi.com
http://www.alaskasatelliteinternet.com

bug#24993: Documentation: 'guix pull' needed during SD installation

[Leo Famulari]

On Wed, Nov 23, 2016 at 08:25:27AM -0900, Christopher Howard wrote:
> The error is:
> 
> "
> guix substitute: error: download from
> 'https://mirror.hydra.gnu.org/nar/zrwslk773<...snip...>e2fsck-static-1.42.13'
> failed 410, "Gone"
> killing process 859
> guix system: error: build failed: some substitutes for the outputs of
> derivation `/gnu/store/kjzga<...snip...>-e2fsck-static-1.42..13.drv'
> failed (usually happens due to networking issues); try `--fallback' to
> build derivation from source
> "

Based on that, it looks like we garbage collected the substitutes for
the 0.11.0 release to make space for new ones, which is too bad, but
hopefully not a permament limitation. We have a new machine for our
build farm front-end currently being installed and tested, and that
server should have a great deal more storage.

You should try passing --substitute-urls=https://hydra.gnu.org in case
the issue is with the mirror.

Otherwise, I think your options for using the 0.11.0 installer are to
use --fallback and build the missing things from source, or use `guix
pull` as was suggested on IRC. I expect the latter option to require
less building.

bug#24993: Documentation: 'guix pull' needed during SD installation

[Christopher Howard]

I was able to make more progress with the --substitute-urls=... option
you mentioned. However, later, when the system is building the
gnupg-2.1.13 drv (I did not pass --fallback, but it still builds stuff)
one of the 36 check tests fails ("tofu.test"), causing the build to fail.

On 11/23/2016 11:29 AM, Leo Famulari wrote:
> On Wed, Nov 23, 2016 at 08:25:27AM -0900, Christopher Howard wrote:
>> The error is:
>>
>> "
>> guix substitute: error: download from
>> 'https://mirror.hydra.gnu.org/nar/zrwslk773<...snip...>e2fsck-static-1.42.13'
>> failed 410, "Gone"
>> killing process 859
>> guix system: error: build failed: some substitutes for the outputs of
>> derivation `/gnu/store/kjzga<...snip...>-e2fsck-static-1.42..13.drv'
>> failed (usually happens due to networking issues); try `--fallback' to
>> build derivation from source
>> "
> 


-- 
Christopher Howard, Computer Assistant
Alaska Satellite Internet
3239 La Ree Way, Fairbanks, AK 99709
907-451-0088 or 888-396-5623 (toll free)
fax: 888-260-3584
mailto:christopher@alaskasi.com
http://www.alaskasatelliteinternet.com

bug#24993: System installer grows brittle with time

[Leo Famulari]

On Mon, Nov 28, 2016 at 08:45:39AM -0900, Christopher Howard wrote:
> I was able to make more progress with the --substitute-urls=... option
> you mentioned. However, later, when the system is building the
> gnupg-2.1.13 drv (I did not pass --fallback, but it still builds stuff)
> one of the 36 check tests fails ("tofu.test"), causing the build to fail.

It will build stuff if it can't find a substitute (not an error).
'--fallback' is only required when substitution fails (an error).

That particular test failure was a bug in GnuPG's test suite that we
worked around:

http://git.savannah.gnu.org/cgit/guix.git/commit/?id=d404a6f9711c8dcc1cc6cf55d8c07901aa450192

Code with an expiration date is very annoying!

It sounds like you will need to use `guix pull`.

What do others think? Should we mention `guix pull` in the installation
documentation?

I'm skeptical for reasons described upthread. I think the real bug is
that the installation image becomes brittle as time passes (so I changed
the subject of my reply).

Does anyone have ideas to mitigate this? Can we tweak the mirrors? Will
this become less pressing when we have more storage space and can store
substitutes for a longer period?

bug#24993: System installer grows brittle with time

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 2351 bytes --]

On Mon, Nov 28, 2016 at 09:33:36PM -0500, Leo Famulari wrote:
> On Mon, Nov 28, 2016 at 08:45:39AM -0900, Christopher Howard wrote:
> > I was able to make more progress with the --substitute-urls=... option
> > you mentioned. However, later, when the system is building the
> > gnupg-2.1.13 drv (I did not pass --fallback, but it still builds stuff)
> > one of the 36 check tests fails ("tofu.test"), causing the build to fail.
> 
> It will build stuff if it can't find a substitute (not an error).
> '--fallback' is only required when substitution fails (an error).
> 
> That particular test failure was a bug in GnuPG's test suite that we
> worked around:
> 
> http://git.savannah.gnu.org/cgit/guix.git/commit/?id=d404a6f9711c8dcc1cc6cf55d8c07901aa450192
> 
> Code with an expiration date is very annoying!
> 
> It sounds like you will need to use `guix pull`.
> 
> What do others think? Should we mention `guix pull` in the installation
> documentation?
> 
> I'm skeptical for reasons described upthread. I think the real bug is
> that the installation image becomes brittle as time passes (so I changed
> the subject of my reply).
> 
> Does anyone have ideas to mitigate this? Can we tweak the mirrors? Will
> this become less pressing when we have more storage space and can store
> substitutes for a longer period?
> 
> 

Currently when people pop into IRC saying they're having trouble with
their install, the suggestion is "run guix pull, use one of the
'default' configurations and then reconfigure after you're finally up
and running, and my favorite, no one really knows why ratpoison is the
fallback WM even when it's not listed in the OS config."

I think we should add 'guix pull' as a regular part of the install
process, and we should add more example configs so people can mix and
match them with fewer issues by virtue of having more examples to
compare against. It would also give us examples to point to for 'how to
use encrypted partitions', 'how to not use wicd', 'how to use a
swapfile', 'i want my kid's account's shell to be 'bambam' or
'gcompris', how do I do that?'

-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

bug#24993: System installer grows brittle with time

[David Craven]

I have a nice one too. How to run guixsd on any hardware :)

(use-modules (gnu)
             (gnu system nss)
             (guix build-system trivial)
             (guix download)
             (guix git-download)
             (guix packages))
(use-service-modules base networking)
(use-package-modules certs linux)

(define-public linux-firmware
  (let ((commit "21e216f591bed97ec1adcda0e8b4b28ddf065433"))
    (package
      (name "linux-firmware")
      (version (string-append "2016.11.27-" (string-take commit 7)))
      (source (origin
                (method git-fetch)
                (uri (git-reference
                        (url (string-append
                              "https://github.com/wkennington/linux-firmware"))
                        (commit commit)))
                (sha256
                 (base32
                  "0iiayy00zr3fk8xfi8i9vrbpg7r59n5vfzx9x7cpx6gjaja8czxa"))))
      (build-system trivial-build-system)
      (arguments
       `(#:modules ((guix build utils))
         #:builder
         (begin
           (use-modules (guix build utils))

           (let* ((source   (assoc-ref %build-inputs "source"))
                  (out      (assoc-ref %outputs "out"))
                  (firmware (string-append out "/lib/firmware")))
             (mkdir-p firmware)
             (copy-recursively source firmware)))))
      (home-page "https://github.com/wkennington/linux-firmware")
      (synopsis "Linux firmware")
      (description "Linux firmware.")
      (license #f))))

(define-public linux
  (package
    (inherit linux-libre)
    (version "4.8.11")
    (source (origin
              (method url-fetch)
              (uri (string-append
                    "https://cdn.kernel.org/pub/linux/kernel/v4.x/"
                    "linux-" version ".tar.xz"))
              (sha256
               (base32
                "03w90vfjfcya38mcp1njasa5c67za203sgp9n3w52gms13s443yc"))))))

(operating-system
  (host-name "system76")
  (timezone "Europe/Zurich")
  (locale "en_US.UTF-8")

  (kernel linux)
  (firmware (list linux-firmware))

  (bootloader (grub-configuration (device "/dev/sda")))

  (file-systems (cons (file-system
                        (device "my-root")
                        (title 'label)
                        (mount-point "/")
                        (type "ext4"))
                      %base-file-systems))

  (users (cons (user-account
                (name "dvc")
                (comment "David Craven")
                (group "users")
                (supplementary-groups '("wheel" "netdev"
                                        "audio" "video"))
                (home-directory "/home/dvc"))
               %base-user-accounts))

  (packages (cons* nss-certs
                   %base-packages))

  (services (cons* (dhcp-client-service)
                   %base-services))

  ;; Allow resolution of '.local' host names with mDNS.
  (name-service-switch %mdns-host-lookup-nss))

bug#24993: System installer grows brittle with time

[Ludovic Courtès]

Leo Famulari <leo@famulari.name> skribis:

> On Mon, Nov 28, 2016 at 08:45:39AM -0900, Christopher Howard wrote:
>> I was able to make more progress with the --substitute-urls=... option
>> you mentioned. However, later, when the system is building the
>> gnupg-2.1.13 drv (I did not pass --fallback, but it still builds stuff)
>> one of the 36 check tests fails ("tofu.test"), causing the build to fail.
>
> It will build stuff if it can't find a substitute (not an error).
> '--fallback' is only required when substitution fails (an error).
>
> That particular test failure was a bug in GnuPG's test suite that we
> worked around:
>
> http://git.savannah.gnu.org/cgit/guix.git/commit/?id=d404a6f9711c8dcc1cc6cf55d8c07901aa450192
>
> Code with an expiration date is very annoying!
>
> It sounds like you will need to use `guix pull`.
>
> What do others think? Should we mention `guix pull` in the installation
> documentation?

I’m not a fan of this, for the reasons you gave.

> I'm skeptical for reasons described upthread. I think the real bug is
> that the installation image becomes brittle as time passes (so I changed
> the subject of my reply).

Right.

A solution that some suggested before (but for other reasons) would be
to include more packages on the installation image.

The image is currently slightly below 1G, but we could add binaries for
GTK+, Python, and a few other relevant packages.  We’d need to find out
what makes sense and how much extra space it would take.

> this become less pressing when we have more storage space and can store
> substitutes for a longer period?

True.

The mirror has room to store things for a long period of time, but
there’s a subtle problem with non-deterministic builds: the mirror might
cache a narinfo and the corresponding nar, then the narinfo leaves the
cache, then a new narinfo is fetched from hydra.gnu.org, and at this
point the mirror has a narinfo advertising a hash that doesn’t match
that of its nar.

Of course the solution to this is reproducible builds, but the fact is
that we still have a bunch of non-reproducible packages.

I recently lowered narinfo caching time in the mirror because of that:

  http://git.savannah.gnu.org/cgit/guix/maintenance.git/commit/?id=caaeb7bea3515e7ef45e33e5e75674f7b72c2f06

… but that’s not great since it means the mirror can lose narinfos
quickly, even though it has enough room to store them.

Ideas?

Ludo’.

bug#24993: System installer grows brittle with time

[Ludovic Courtès]

David Craven <david@craven.ch> skribis:

> I have a nice one too. How to run guixsd on any hardware :)

This is off-topic, and it is not what this project is about, as you
know (well of course, we all want it to run on any hardware, just not at
any cost!).

Ludo’.

bug#24993: System installer grows brittle as substitutes for the release disappear

[Ludovic Courtès]

Heya,

Reviving an old bug…

Leo Famulari <leo@famulari.name> skribis:

> Based on that, it looks like we garbage collected the substitutes for
> the 0.11.0 release to make space for new ones, which is too bad, but
> hopefully not a permament limitation. We have a new machine for our
> build farm front-end currently being installed and tested, and that
> server should have a great deal more storage.

Commit 4e854b1814a9216ae7cc90aef4d82fd989a519c3 added the whole
bare-bones OS as a dependency of the installation image.  Thus,
everything bare-bones depends on is included in the image.

This should ensure that at least key store items are already available
in the image.

Infrastructure-wise, the situation has also improved I think, and is
getting better with berlin.guixsd.org.

I think we can close this bug.  Let’s reopen it if I turned out to be
too optimistic.

Ludo’.

bug#24993: System installer grows brittle as substitutes for the release disappear

[Leo Famulari]

[-- Attachment #1: Type: text/plain, Size: 1080 bytes --]

On Sun, Nov 12, 2017 at 09:30:14PM +0100, Ludovic Courtès wrote:
> Heya,
> 
> Reviving an old bug…
> 
> Leo Famulari <leo@famulari.name> skribis:
> 
> > Based on that, it looks like we garbage collected the substitutes for
> > the 0.11.0 release to make space for new ones, which is too bad, but
> > hopefully not a permament limitation. We have a new machine for our
> > build farm front-end currently being installed and tested, and that
> > server should have a great deal more storage.
> 
> Commit 4e854b1814a9216ae7cc90aef4d82fd989a519c3 added the whole
> bare-bones OS as a dependency of the installation image.  Thus,
> everything bare-bones depends on is included in the image.

Nice! I missed this commit when it was pushed.

> This should ensure that at least key store items are already available
> in the image.
> 
> Infrastructure-wise, the situation has also improved I think, and is
> getting better with berlin.guixsd.org.
> 
> I think we can close this bug.  Let’s reopen it if I turned out to be
> too optimistic.

Good plan :)

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]