ISO image available for testing!

ISO image available for testing!

[Ludovic Courtès]

[-- Attachment #1: Type: text/plain, Size: 560 bytes --]

Hello there!

I’ve uploaded a GuixSD installation ISO image for testing:

  http://web.fdn.fr/~lcourtes/software/guix/guixsd-install-0.13.0.8d7f1d736.x86_64-linux.iso.xz
  SHA256: 7e04017494c419bb6ca654cff05dc7e5ba4fdfd3c2812fba9ba6d929b319e93b

  http://web.fdn.fr/~lcourtes/software/guix/guixsd-install-0.13.0.8d7f1d736.x86_64-linux.iso.xz.sig

It was built with “guix system disk-image -t iso9660
gnu/system/install.scm” on commit 8d7f1d736.

Note that it’s 1.1G uncompressed so it won’t fit on a CD.

Feedback welcome!

Ludo’.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: ISO image available for testing!

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 1417 bytes --]


Ludovic Courtès writes:

> Hello there!
>
> I’ve uploaded a GuixSD installation ISO image for testing:
>
>   http://web.fdn.fr/~lcourtes/software/guix/guixsd-install-0.13.0.8d7f1d736.x86_64-linux.iso.xz
>   SHA256: 7e04017494c419bb6ca654cff05dc7e5ba4fdfd3c2812fba9ba6d929b319e93b
>
>   http://web.fdn.fr/~lcourtes/software/guix/guixsd-install-0.13.0.8d7f1d736.x86_64-linux.iso.xz.sig
>
> It was built with “guix system disk-image -t iso9660
> gnu/system/install.scm” on commit 8d7f1d736.
>
> Note that it’s 1.1G uncompressed so it won’t fit on a CD.
>
> Feedback welcome!

Hey,

I've attempted to use this to install GuixSD on a Bytemark
VM. Unfortunately I haven't succeeded yet. I managed to get as far as
running guix system init, but when I did, it started downloading the
bootstrap binaries, and then building binutils.

When I run guix system build, or guix build with the --dry-run options,
it says that it will download, rather than building, but it doesn't.

I also made a few other notes during the installation process:

 - The documentation mentions ipconfig to get the wired interface up,
   but it isn't available in the image. I had to run ip link set eth0
   up.
 - The documentation says that the ssh server needs to be started, but
   it was running without me starting it.

Any tips on debugging the building from source issue?

Thanks,

Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 962 bytes --]

Re: ISO image available for testing!

[Ludovic Courtès]

Hello,

Christopher Baines <mail@cbaines.net> skribis:

> I've attempted to use this to install GuixSD on a Bytemark
> VM. Unfortunately I haven't succeeded yet. I managed to get as far as
> running guix system init, but when I did, it started downloading the
> bootstrap binaries, and then building binutils.
>
> When I run guix system build, or guix build with the --dry-run options,
> it says that it will download, rather than building, but it doesn't.

I’m pretty sure the building-from-source comes from grafts where the
package replacements, for some reason, are not getting built by Hydra.
I thought this had been fixed by
b574cee361bdabf21f5506252b6a183dcfcd028d, but clearly it hasn’t.

I’ll investigate tomorrow.

> I also made a few other notes during the installation process:
>
>  - The documentation mentions ipconfig to get the wired interface up,
>    but it isn't available in the image. I had to run ip link set eth0
>    up.

It mentions “ifconfig” (with an “f”) and that one is present.  Or am I
missing something?

>  - The documentation says that the ssh server needs to be started, but
>    it was running without me starting it.

Indeed.  I’ve pushed a fix as aab322d909c0b4abec132ef7aff31c31a1208841
in the new ‘version-0.14.0’ branch (note that it changes the ABI of (gnu
services ssh)).

Thanks,
Ludo’.

Re: ISO image available for testing!

[D4n1]

I'll test it and report here.

Thanks, 

Daniel Pimentel (d4n1)On Dec 5, 2017 8:47 PM, ludo@gnu.org wrote:
>
> Hi Chris, 
>
> Christopher Baines <mail@cbaines.net> skribis: 
>
> > I've attempted to use this to install GuixSD on a Bytemark 
> > VM. Unfortunately I haven't succeeded yet. I managed to get as far as 
> > running guix system init, but when I did, it started downloading the 
> > bootstrap binaries, and then building binutils. 
> > 
> > When I run guix system build, or guix build with the --dry-run options, 
> > it says that it will download, rather than building, but it doesn't. 
>
> It turned out to be issues related to grafts and to what Hydra builds, 
> fixed with these commits: 
>
> 3e442f85f * gnu: ghostscript-with-cups: Turn into a public variable. 
> 91c9b5d01 * packages: 'package-grafts' trims native inputs. 
> ff0e0041f * packages: 'fold-bag-dependencies' honors nativeness in recursive calls. 
> f00b85ff8 * gnu: commencement: Do not graft early bootstrap packages. 
>
> The Binutils issue is fixed by f00b85ff8. 
>
> Commit 91c9b5d01 notably fixes the “expat issue”: coreutils had expat in 
> its dependency graph, via gettext.  Thus, the expat graft was picked up 
> as a candidate graft.  However, expat itself was subject to the glibc 
> graft, and since there was no substitute for this particular expat, we’d 
> have to build it first, just to throw it away later on because coreutils 
> does not refer to it at run time. 
>
> Long story short: we were flagging native inputs as potential sources of 
> grafts even though, by definition, native inputs are not referred to at 
> run time. 
>
> The last commit ensures that Hydra builds the replacement for 
> ‘ghostscript-with-cups’. 
>
> What’s tricky is that one doesn’t notice these issues unless starting 
> from a fresh store. 
>
> I’ve uploaded an updated ISO image, which I used to test substitute 
> availability and grafts.  If you have time in the coming hours, feedback 
> welcome: 
>
>   http://web.fdn.fr/~lcourtes/software/guix/guixsd-install-0.13.0.91c9b5d01.x86_64-linux.iso.xz 
>   http://web.fdn.fr/~lcourtes/software/guix/guixsd-install-0.13.0.91c9b5d01.x86_64-linux.iso.xz.sig 
>
> Ludo’. 
>

Re: ISO image available for testing!

[D4n1]

Thanks Ludo! But there isn't a i686 ISO :( I'll try install on my x60s. But I'll try on may another PC using x86_64 platform.

Thanks,

Daniel Pimentel (d4n1)On Dec 5, 2017 8:47 PM, ludo@gnu.org wrote:
>
> Hi Chris, 
>
> Christopher Baines <mail@cbaines.net> skribis: 
>
> > I've attempted to use this to install GuixSD on a Bytemark 
> > VM. Unfortunately I haven't succeeded yet. I managed to get as far as 
> > running guix system init, but when I did, it started downloading the 
> > bootstrap binaries, and then building binutils. 
> > 
> > When I run guix system build, or guix build with the --dry-run options, 
> > it says that it will download, rather than building, but it doesn't. 
>
> It turned out to be issues related to grafts and to what Hydra builds, 
> fixed with these commits: 
>
> 3e442f85f * gnu: ghostscript-with-cups: Turn into a public variable. 
> 91c9b5d01 * packages: 'package-grafts' trims native inputs. 
> ff0e0041f * packages: 'fold-bag-dependencies' honors nativeness in recursive calls. 
> f00b85ff8 * gnu: commencement: Do not graft early bootstrap packages. 
>
> The Binutils issue is fixed by f00b85ff8. 
>
> Commit 91c9b5d01 notably fixes the “expat issue”: coreutils had expat in 
> its dependency graph, via gettext.  Thus, the expat graft was picked up 
> as a candidate graft.  However, expat itself was subject to the glibc 
> graft, and since there was no substitute for this particular expat, we’d 
> have to build it first, just to throw it away later on because coreutils 
> does not refer to it at run time. 
>
> Long story short: we were flagging native inputs as potential sources of 
> grafts even though, by definition, native inputs are not referred to at 
> run time. 
>
> The last commit ensures that Hydra builds the replacement for 
> ‘ghostscript-with-cups’. 
>
> What’s tricky is that one doesn’t notice these issues unless starting 
> from a fresh store. 
>
> I’ve uploaded an updated ISO image, which I used to test substitute 
> availability and grafts.  If you have time in the coming hours, feedback 
> welcome: 
>
>   http://web.fdn.fr/~lcourtes/software/guix/guixsd-install-0.13.0.91c9b5d01.x86_64-linux.iso.xz 
>   http://web.fdn.fr/~lcourtes/software/guix/guixsd-install-0.13.0.91c9b5d01.x86_64-linux.iso.xz.sig 
>
> Ludo’. 
>

Re: ISO image available for testing!

[Ludovic Courtès]

Hi Chris,

Christopher Baines <mail@cbaines.net> skribis:

> I've attempted to use this to install GuixSD on a Bytemark
> VM. Unfortunately I haven't succeeded yet. I managed to get as far as
> running guix system init, but when I did, it started downloading the
> bootstrap binaries, and then building binutils.
>
> When I run guix system build, or guix build with the --dry-run options,
> it says that it will download, rather than building, but it doesn't.

It turned out to be issues related to grafts and to what Hydra builds,
fixed with these commits:

3e442f85f * gnu: ghostscript-with-cups: Turn into a public variable.
91c9b5d01 * packages: 'package-grafts' trims native inputs.
ff0e0041f * packages: 'fold-bag-dependencies' honors nativeness in recursive calls.
f00b85ff8 * gnu: commencement: Do not graft early bootstrap packages.

The Binutils issue is fixed by f00b85ff8.

Commit 91c9b5d01 notably fixes the “expat issue”: coreutils had expat in
its dependency graph, via gettext.  Thus, the expat graft was picked up
as a candidate graft.  However, expat itself was subject to the glibc
graft, and since there was no substitute for this particular expat, we’d
have to build it first, just to throw it away later on because coreutils
does not refer to it at run time.

Long story short: we were flagging native inputs as potential sources of
grafts even though, by definition, native inputs are not referred to at
run time.

The last commit ensures that Hydra builds the replacement for
‘ghostscript-with-cups’.

What’s tricky is that one doesn’t notice these issues unless starting
from a fresh store.

I’ve uploaded an updated ISO image, which I used to test substitute
availability and grafts.  If you have time in the coming hours, feedback
welcome:

  http://web.fdn.fr/~lcourtes/software/guix/guixsd-install-0.13.0.91c9b5d01.x86_64-linux.iso.xz
  http://web.fdn.fr/~lcourtes/software/guix/guixsd-install-0.13.0.91c9b5d01.x86_64-linux.iso.xz.sig

Ludo’.

Re: ISO image available for testing!

[Mark H Weaver]

Hi Ludovic,

ludo@gnu.org (Ludovic Courtès) writes:
> 91c9b5d01 * packages: 'package-grafts' trims native inputs.

[...]

> Long story short: we were flagging native inputs as potential sources of
> grafts even though, by definition, native inputs are not referred to at
> run time.

I agree that this *should* never happen, but I see little reason for
confidence that it never happens in actual fact.

What would happen if a reference to a native-input *was* present in the
build outputs?  The reason I ask is that, for security reasons, it's
obviously very important to reliably avoid using ungrafted software at
run time.

I'm concerned that this recent change could cause minor
nearly-undetectable packaging mistakes to become major security holes.

One solution would be to explicitly check build outputs for references
to native-inputs, and to force a build failure in that case.

What do you think?

    Regards,
      Mark

Re: ISO image available for testing!

[Ben Woodcroft]

On 06/12/17 10:52, Mark H Weaver wrote:
> Hi Ludovic,
>
> ludo@gnu.org (Ludovic Courtès) writes:
>> 91c9b5d01 * packages: 'package-grafts' trims native inputs.
> [...]
>
>> Long story short: we were flagging native inputs as potential sources of
>> grafts even though, by definition, native inputs are not referred to at
>> run time.
> I agree that this *should* never happen, but I see little reason for
> confidence that it never happens in actual fact.
>
> What would happen if a reference to a native-input *was* present in the
> build outputs?  The reason I ask is that, for security reasons, it's
> obviously very important to reliably avoid using ungrafted software at
> run time.
>
> I'm concerned that this recent change could cause minor
> nearly-undetectable packaging mistakes to become major security holes.
>
> One solution would be to explicitly check build outputs for references
> to native-inputs, and to force a build failure in that case.
I believe there are a number of cases of this that happen when binaries 
are wrapped with paths derived from getenv, e.g. this phase in bamm:

          (add-after 'install 'wrap-executable
            (lambda* (#:key outputs #:allow-other-keys)
             (let* ((out  (assoc-ref outputs "out"))
                    (path (getenv "PATH")))
               (wrap-program (string-append out "/bin/bamm")
                 `("PATH" ":" prefix (,path))))
             #t))

It would be good to stop using getenv for this, for the reasons 
described here and others e.g. non-reproducibility, unnecessary 
dependencies etc.

Is there some easy way to "getenv" excluding unwanted components of an 
environment variable?

Thanks, ben

Re: ISO image available for testing!

[Mark H Weaver]

I wrote:
> One solution would be to explicitly check build outputs for references
> to native-inputs, and to force a build failure in that case.

More precisely, we could force a build failure when any build output
contains a reference to a component (store path) in the following set:

  requisites(native-inputs) - requisites(inputs)

       Mark

Re: ISO image available for testing!

[Ludovic Courtès]

Hi Mark,

Mark H Weaver <mhw@netris.org> skribis:

> ludo@gnu.org (Ludovic Courtès) writes:
>> 91c9b5d01 * packages: 'package-grafts' trims native inputs.
>
> [...]
>
>> Long story short: we were flagging native inputs as potential sources of
>> grafts even though, by definition, native inputs are not referred to at
>> run time.
>
> I agree that this *should* never happen, but I see little reason for
> confidence that it never happens in actual fact.
>
> What would happen if a reference to a native-input *was* present in the
> build outputs?  The reason I ask is that, for security reasons, it's
> obviously very important to reliably avoid using ungrafted software at
> run time.
>
> I'm concerned that this recent change could cause minor
> nearly-undetectable packaging mistakes to become major security holes.

Given the examples that Tobias and Ben were quick to find, I’m afraid
you’re right and I was overconfident.  I’m reverting the change.

> One solution would be to explicitly check build outputs for references
> to native-inputs, and to force a build failure in that case.

We could do that, though I suppose a lot of packages would break.

Thanks to the quick reply,
Ludo’.

Re: ISO image available for testing!

[Tobias Geerinckx-Rice]

Ludovic Courtès wrote on 06/12/17 at 09:14:
>> One solution would be to explicitly check build outputs for references
>> to native-inputs, and to force a build failure in that case.

I started a rudimentary PoC for this last week, after realising how bad
the situation is in ghc-* alone. We'll see what comes of it.

> We could do that, though I suppose a lot of packages would break.

We *should* do it, though. Just not before the pending release.

Kind regards,

T G-R

Re: ISO image available for testing!

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 2953 bytes --]


Ludovic Courtès writes:

> Hi Chris,
>
> Christopher Baines <mail@cbaines.net> skribis:
>
>> I've attempted to use this to install GuixSD on a Bytemark
>> VM. Unfortunately I haven't succeeded yet. I managed to get as far as
>> running guix system init, but when I did, it started downloading the
>> bootstrap binaries, and then building binutils.
>>
>> When I run guix system build, or guix build with the --dry-run options,
>> it says that it will download, rather than building, but it doesn't.
>
> It turned out to be issues related to grafts and to what Hydra builds,
> fixed with these commits:
>
> 3e442f85f * gnu: ghostscript-with-cups: Turn into a public variable.
> 91c9b5d01 * packages: 'package-grafts' trims native inputs.
> ff0e0041f * packages: 'fold-bag-dependencies' honors nativeness in recursive calls.
> f00b85ff8 * gnu: commencement: Do not graft early bootstrap packages.
>
> The Binutils issue is fixed by f00b85ff8.
>
> Commit 91c9b5d01 notably fixes the “expat issue”: coreutils had expat in
> its dependency graph, via gettext.  Thus, the expat graft was picked up
> as a candidate graft.  However, expat itself was subject to the glibc
> graft, and since there was no substitute for this particular expat, we’d
> have to build it first, just to throw it away later on because coreutils
> does not refer to it at run time.
>
> Long story short: we were flagging native inputs as potential sources of
> grafts even though, by definition, native inputs are not referred to at
> run time.
>
> The last commit ensures that Hydra builds the replacement for
> ‘ghostscript-with-cups’.
>
> What’s tricky is that one doesn’t notice these issues unless starting
> from a fresh store.
>
> I’ve uploaded an updated ISO image, which I used to test substitute
> availability and grafts.  If you have time in the coming hours, feedback
> welcome:

Thanks for fixing this Ludo, and congratulations on the release. I'm
glad to say that I've now managed to install GuixSD using the 0.14.0
x86_64 ISO, however I did encounter some difficulties.

I tried a few times with both the ISO you replied with here, and the
released ISO, but each time the virtual machine I was installing on to
appeared to restart while guix system init was running. It's difficult
to get more information, but the last messages I got out of guix system
init relate to grafting and collisions.

This evening when I tried again I passed --no-grafts to guix system
init, and this time it successfully finished the
installation. Interestingly, this is also what is actually tested by the
iso-image-installer system test, as it sets the GUIX_BUILD_OPTIONS
environment variable.

This isn't conclusive, but I'd be very interested to hear from anyone
that has had similar issues, or successes in using the ISO installer,
both with and without the --no-grafts option.

Thanks again,

Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 962 bytes --]

Re: ISO image available for testing!

[Ludovic Courtès]

Hello Chris,

Christopher Baines <mail@cbaines.net> skribis:

> Thanks for fixing this Ludo, and congratulations on the release. I'm
> glad to say that I've now managed to install GuixSD using the 0.14.0
> x86_64 ISO, however I did encounter some difficulties.
>
> I tried a few times with both the ISO you replied with here, and the
> released ISO, but each time the virtual machine I was installing on to
> appeared to restart while guix system init was running. It's difficult
> to get more information, but the last messages I got out of guix system
> init relate to grafting and collisions.

I wonder how the VM can restart.  Could it be an out-of-memory
condition?  Though even that should not lead to a reboot.

Could you see if you can get more details from the VM?

Thanks,
Ludo’.