Re: Cross-building GuixSD (and maybe using pre-built toolchains)

Re: Cross-building GuixSD (and maybe using pre-built toolchains)

[Paul Boddie]

Hello,

(Quoting a discussion on the development list, which proved rather too active 
for me to follow. Sending to this list at Ludo's suggestion.)

> Paul Boddie <paul@boddie.org.uk> skribis:
> > I was looking at the documentation, which is rather more helpful than
> > most distributions when it comes to describing how the actual
> > distribution archives are built, and I figured a few things out about
> > guix-daemon. However, I'm doing all my work in a chroot - I don't want
> > to "make install" and populate various locations on my main system - and
> > this doesn't seem to be entirely compatible with the recommended way the
> > daemon is used. Specifically, the daemon does not seem to be able to
> > build anything unless I specify the -- disable-chroot flag.
> 
> How does guix-daemon fail exactly?  If needed, could you strace it to
> get more details on what’s going on?

I could try and take a look at this.

> > Are there any recommended methods of running guix-daemon in a chroot and
> > have it create new chroots, or do I have to use some kind of
> > virtualisation or container technology? Is any kind of
> > fakeroot/fakechroot mechanism supported?
> 
> I think few people do this, on the grounds that Guix is rather
> non-intrusive: it stores all its stuff in /gnu/store and /var/guix.  So
> if you want to get rid of it, all you have to do is delete those two
> directories.  The rest of the system is untouched.
> 
> Now, if you really want it, I can’t think of any reason why guix-daemon
> wouldn’t run in a chroot.  It currently requires root privileges,
> precisely so that it can set up a chroot, separate name spaces, and so
> on, but that could work in a chroot too.

OK. I tend to run things in chroots for basic protection against things 
deciding to install stuff in places I would rather keep "clean", and to use 
different distro versions and packages. I guess I could instead have separate 
partitions for /usr and other things and make them read-only, and so on, but 
I'm not sure many people do this these days.

Then again, I do see people wanting to run Guix on other distros, so perhaps I 
should just accept that it doesn't need to run inside a chroot itself.

> > One thing that looks very promising is the ability to cross-bootstrap the
> > system, and I did manage to get that started for a new architecture
> > (mipsel) by performing some modifications suggested in the documentation
> > [1], running guix-daemon with the --disable-chroot flag, and then
> > running this:
> > 
> > guix build --target=mipsel-linux-gnu bootstrap-tarballs
> > 
> > However, not wanting to leave my computer switched on all night, I
> > stopped the build after a while. I can understand the need to bootstrap
> > things like toolchains, but given that I am running Debian which has
> > cross-toolchains for mipsel, I wondered if I could short-circuit this
> > process by employing pre- built toolchains. Would this be possible?
> 
> No.  The approach we’re taking is to have a self-contained distro, and
> that’s where we get all the fruits of the “functional paradigm”
> described in the manual.
> 
> It’s surely an annoyance when bootstrapping an architecture for the
> first time.  Once it’s in place though, it’s OK.  You could have build
> machines for that architecture publishing binaries to users (via ‘guix
> publish’).

I was really only looking for a short cut here, I admit. And if you have a 
robust way of building toolchains, that's actually a great thing, because it 
can be difficult to do repeatably (tracking the evolution of the software, 
dealing with platform issues), and people still download random binary 
toolchains for such purposes even today because it is a scary thought to build 
their own from source.

I noticed that Arch Linux and its relative Parabola GNU/Linux support the 
target machine using distcc as a client and having distcc servers cross-
compile code for the target architecture. That seems to require the 
coordinating host to be running Arch/Parabola, which then means that some 
bootstrapping needs to be done already. Could a similar thing be done with 
GuixSD?

I do understand the "self-contained" motivation: it's exactly what Debian and 
other distros insist upon, although I'd argue that it's also what holds such 
distros back from properly supporting cross-compilation. (I did follow the 
Emdebian project for a while, and it just seemed like very punishing work that 
tended to interest very few people, sadly.)

> > Finally, it seems to be the case that the workflow involves building a
> > bootstrapped minimal system and then natively building packages. Or have
> > I misunderstood the process?
> 
> Correct, this is the process to port the distro to a new platform.

Understood.

> > Is it possible to cross-build all of the packages in the distribution?
> 
> That’s also an option: you can simply cross-build the packages that you
> need and copy them to the target machine.
> 
> We make sure core packages can be cross-built to the architectures we
> support (currently x86_64, i686, armhf, and mips64el).  However, there’s
> no guarantee that cross-building works for other targets, or that
> non-core packages cross-build at all.  We’d definitely welcome patches
> in this area, though.  :-)

Right. So you'd cross-build things like the kernel, libc, gcc, binutils, maybe 
some other things (the bootstrapping discussed above). Then bring up the 
target machine and hopefully have enough to start building packages somehow. 
Does that sound correct?

> HTH!

Yes, I think it does. Thanks for the response and sorry once again for not 
replying before now!

Regards,

Paul

Re: Cross-building GuixSD (and maybe using pre-built toolchains)

[Ludovic Courtès]

Hi Paul,

Paul Boddie <paul@boddie.org.uk> skribis:

>> > Are there any recommended methods of running guix-daemon in a chroot and
>> > have it create new chroots, or do I have to use some kind of
>> > virtualisation or container technology? Is any kind of
>> > fakeroot/fakechroot mechanism supported?
>> 
>> I think few people do this, on the grounds that Guix is rather
>> non-intrusive: it stores all its stuff in /gnu/store and /var/guix.  So
>> if you want to get rid of it, all you have to do is delete those two
>> directories.  The rest of the system is untouched.
>> 
>> Now, if you really want it, I can’t think of any reason why guix-daemon
>> wouldn’t run in a chroot.  It currently requires root privileges,
>> precisely so that it can set up a chroot, separate name spaces, and so
>> on, but that could work in a chroot too.
>
> OK. I tend to run things in chroots for basic protection against things 
> deciding to install stuff in places I would rather keep "clean", and to use 
> different distro versions and packages.

Precisely: Guix only ever touches /gnu/store, /var/guix, and optionally
/etc/guix; nothing else, I swear.  ;-)

> I noticed that Arch Linux and its relative Parabola GNU/Linux support the 
> target machine using distcc as a client and having distcc servers cross-
> compile code for the target architecture. That seems to require the 
> coordinating host to be running Arch/Parabola, which then means that some 
> bootstrapping needs to be done already. Could a similar thing be done with 
> GuixSD?

Yes, using offloading:

  https://www.gnu.org/software/guix/manual/html_node/Daemon-Offload-Setup.html

If your master node is x86_64 and its machines.scm lists a mips64el
machine as in the example above, then “guix build foo -s mips64el-linux”
automatically offloads to that mips machine.  (This is the mechanism our
hydra.gnu.org build farm uses.)

>> That’s also an option: you can simply cross-build the packages that you
>> need and copy them to the target machine.
>> 
>> We make sure core packages can be cross-built to the architectures we
>> support (currently x86_64, i686, armhf, and mips64el).  However, there’s
>> no guarantee that cross-building works for other targets, or that
>> non-core packages cross-build at all.  We’d definitely welcome patches
>> in this area, though.  :-)
>
> Right. So you'd cross-build things like the kernel, libc, gcc, binutils, maybe 
> some other things (the bootstrapping discussed above). Then bring up the 
> target machine and hopefully have enough to start building packages somehow. 
> Does that sound correct?

When bootstrapping a new architecture, we cross-build the relevant
“bootstrap binaries”, which are statically-linked tools such as GCC,
Guile, as well as libc, and from there we can start building natively:

  https://www.gnu.org/software/guix/manual/html_node/Porting.html

Ludo’.

Re: Cross-building GuixSD (and maybe using pre-built toolchains)

[Paul Boddie]

On Wednesday 31. August 2016 22.04.35 Ludovic Courtès wrote:
> 
> Paul Boddie <paul@boddie.org.uk> skribis:
> > 
> > OK. I tend to run things in chroots for basic protection against things
> > deciding to install stuff in places I would rather keep "clean", and to
> > use different distro versions and packages.
> 
> Precisely: Guix only ever touches /gnu/store, /var/guix, and optionally
> /etc/guix; nothing else, I swear.  ;-)

Right! Well, I may have another try outside a chroot and take your word for 
it. ;-)

[...]

> > I noticed that Arch Linux and its relative Parabola GNU/Linux support the
> > target machine using distcc as a client and having distcc servers cross-
> > compile code for the target architecture. That seems to require the
> > coordinating host to be running Arch/Parabola, which then means that some
> > bootstrapping needs to be done already. Could a similar thing be done
> > with GuixSD?
> 
> Yes, using offloading:
> 
>  
> https://www.gnu.org/software/guix/manual/html_node/Daemon-Offload-Setup.html
> 
> If your master node is x86_64 and its machines.scm lists a mips64el
> machine as in the example above, then “guix build foo -s mips64el-linux”
> automatically offloads to that mips machine.  (This is the mechanism our
> hydra.gnu.org build farm uses.)

But this seems like the opposite of the approach in Arch/Parabola: there, the 
target machine (mips64el) coordinates the build, with the farm machines 
(x86_64) cross-compiling the sources. The benefit is that architectures with 
less powerful hardware can have their packages built for them, but this would 
presumably be contrary to the principles employed by Guix, where native builds 
are mandatory for viability and, I guess, reproducibility reasons.

[...]

> When bootstrapping a new architecture, we cross-build the relevant
> “bootstrap binaries”, which are statically-linked tools such as GCC,
> Guile, as well as libc, and from there we can start building natively:
> 
>   https://www.gnu.org/software/guix/manual/html_node/Porting.html

Understood, thanks!

Paul

Re: Cross-building GuixSD (and maybe using pre-built toolchains)

[Ludovic Courtès]

Paul Boddie <paul@boddie.org.uk> skribis:

> On Wednesday 31. August 2016 22.04.35 Ludovic Courtès wrote:

[...]

>> > I noticed that Arch Linux and its relative Parabola GNU/Linux support the
>> > target machine using distcc as a client and having distcc servers cross-
>> > compile code for the target architecture. That seems to require the
>> > coordinating host to be running Arch/Parabola, which then means that some
>> > bootstrapping needs to be done already. Could a similar thing be done
>> > with GuixSD?
>> 
>> Yes, using offloading:
>> 
>>  
>> https://www.gnu.org/software/guix/manual/html_node/Daemon-Offload-Setup.html
>> 
>> If your master node is x86_64 and its machines.scm lists a mips64el
>> machine as in the example above, then “guix build foo -s mips64el-linux”
>> automatically offloads to that mips machine.  (This is the mechanism our
>> hydra.gnu.org build farm uses.)
>
> But this seems like the opposite of the approach in Arch/Parabola: there, the 
> target machine (mips64el) coordinates the build, with the farm machines 
> (x86_64) cross-compiling the sources. The benefit is that architectures with 
> less powerful hardware can have their packages built for them, but this would 
> presumably be contrary to the principles employed by Guix, where native builds 
> are mandatory for viability and, I guess, reproducibility reasons.

I should mention that native builds are also subject to offloading.  So
if your main machine is x86_64 and has a machines.scm file that lists
other x86_64 machines, then native builds can be offloaded to them.

If you cross-compile from x86_64, it’s a native build that can be
offloaded to another x86_64 machine.

Hope this clarifies things!

Ludo’.

Re: Cross-building GuixSD (and maybe using pre-built toolchains)

[Paul Boddie]

On Thursday 1. September 2016 10.10.15 Ludovic Courtès wrote:
> Paul Boddie <paul@boddie.org.uk> skribis:
> > 

[Offloading]

> > But this seems like the opposite of the approach in Arch/Parabola: there,
> > the target machine (mips64el) coordinates the build, with the farm
> > machines (x86_64) cross-compiling the sources. The benefit is that
> > architectures with less powerful hardware can have their packages built
> > for them, but this would presumably be contrary to the principles
> > employed by Guix, where native builds are mandatory for viability and, I
> > guess, reproducibility reasons.
> 
> I should mention that native builds are also subject to offloading.  So
> if your main machine is x86_64 and has a machines.scm file that lists
> other x86_64 machines, then native builds can be offloaded to them.
> 
> If you cross-compile from x86_64, it’s a native build that can be
> offloaded to another x86_64 machine.
> 
> Hope this clarifies things!

Perhaps. Here come the stupid questions: ;-)

When you write "cross-compile from x86_64" do you mean run a cross-compiler on 
an x86_64 host that targets another architecture? (I'm guessing "yes".)

From previous questions, it appears that any compiler must have been built 
specifically for Guix to use (and cannot be those those pre-built toolchains 
provided by, say, Debian), so would these cross-compilers be the ones normally 
used to generate the "bootstrap binaries" (that would normally be used to then 
build packages on the target system)?

Or do I first bootstrap Guix on my (x86_64, in this example) "build system", 
then generate special cross-compilers for my target architecture (that are 
then exposed via offloading)?

And can I provide a foreign architecture build daemon on the same machine as 
the one performing the build?

Sorry if this is all obvious or if I'm not assuming that what is possible is 
actually possible! I'm used to discovering limitations with build systems. (To 
me it is remarkable that given all the portable things like shell scripts that 
do a lot of the lifting in build processes, many build systems have an 
unreasonable affinity to the build host's architecture.)

Thanks for all the help!

Paul

Re: Cross-building GuixSD (and maybe using pre-built toolchains)

[Ludovic Courtès]

Paul Boddie <paul@boddie.org.uk> skribis:

> When you write "cross-compile from x86_64" do you mean run a cross-compiler on 
> an x86_64 host that targets another architecture? (I'm guessing "yes".)

Yes.

> From previous questions, it appears that any compiler must have been built 
> specifically for Guix to use (and cannot be those those pre-built toolchains 
> provided by, say, Debian), so would these cross-compilers be the ones normally 
> used to generate the "bootstrap binaries" (that would normally be used to then 
> build packages on the target system)?

Yes, Guix packages only depend on other Guix packages; there cannot be
dependencies on external packages.

> Or do I first bootstrap Guix on my (x86_64, in this example) "build system", 
> then generate special cross-compilers for my target architecture (that are 
> then exposed via offloading)?

Offloading and cross-compilation are unrelated.  Offloading is just a
mechanism that allows distributed builds.

Guix supports x86_64-linux-gnu, so, IOW, the distro is already
“bootstrapped” for that platform.  There’s nothing special to be done.

I’m trying to find ways to rephrase this, but I can’t think of other
ways than what the “Bootstrapping” and “Porting” sections of the manuals
say.  :-)

> And can I provide a foreign architecture build daemon on the same machine as 
> the one performing the build?

No.

> Sorry if this is all obvious or if I'm not assuming that what is possible is 
> actually possible! I'm used to discovering limitations with build systems. (To 
> me it is remarkable that given all the portable things like shell scripts that 
> do a lot of the lifting in build processes, many build systems have an 
> unreasonable affinity to the build host's architecture.)
>
> Thanks for all the help!

No problem!  I realize it can be difficult to convey the big picture of
Guix here to someone already familiar with other build systems.

Ludo’.

Re: Cross-building GuixSD (and maybe using pre-built toolchains)

[David Craven]

> There’s an untested hack to illustrate:

Thanks! I think I got it kind of working - but we'll have to see after
there's a armhf kernel available.

When adding cross-compilation support to the linux-libre package I
noticed that native-inputs are in #:key native-inputs when
cross-compiling but not when compiling natively. I think this is a
problem because we can't just use #:key native-inputs instead of #:key
inputs when adding support for cross-compilation.

WDYT?

Re: Cross-building GuixSD (and maybe using pre-built toolchains)

[Ludovic Courtès]

Hi,

David Craven <david@craven.ch> skribis:

> When adding cross-compilation support to the linux-libre package I
> noticed that native-inputs are in #:key native-inputs when
> cross-compiling but not when compiling natively. I think this is a
> problem because we can't just use #:key native-inputs instead of #:key
> inputs when adding support for cross-compilation.
>
> WDYT?

It’s on purpose, and packages rarely (if even) need to refer to
#:native-inputs; only a couple of phases in gnu-build-system.scm refer
to #:native-inputs.

In which context do you need to refer to #:native-inputs?

Ludo’.

Re: Cross-building GuixSD (and maybe using pre-built toolchains)

[David Craven]

> It’s on purpose, and packages rarely (if even) need to refer to
> #:native-inputs; only a couple of phases in gnu-build-system.scm refer
> to #:native-inputs.
>
> In which context do you need to refer to #:native-inputs?

Cross-compiling the linux kernel.

> ;; Apply the neat patch.
> (system* "patch" "-p1" "--force"
>                "-i" (assoc-ref inputs "patch/freedo+gnu"))

and

> (mit    (assoc-ref inputs "module-init-tools")))

fail for no "real" reason. I could add a (if target native-inputs
inputs) but it seems ugly and think it might be a common problem?

Re: Cross-building GuixSD (and maybe using pre-built toolchains)

[Ludovic Courtès]

David Craven <david@craven.ch> skribis:

>> It’s on purpose, and packages rarely (if even) need to refer to
>> #:native-inputs; only a couple of phases in gnu-build-system.scm refer
>> to #:native-inputs.
>>
>> In which context do you need to refer to #:native-inputs?
>
> Cross-compiling the linux kernel.
>
>> ;; Apply the neat patch.
>> (system* "patch" "-p1" "--force"
>>                "-i" (assoc-ref inputs "patch/freedo+gnu"))

This looks like a remnant of the past; nowadays it would work to add
‘%boot-logo’ to ‘patches’ in ‘origin’.  We should do that.

> and
>
>> (mit    (assoc-ref inputs "module-init-tools")))
>
> fail for no "real" reason. I could add a (if target native-inputs
> inputs) but it seems ugly and think it might be a common problem?

Hmm, I can see two occurrences of

  (assoc-ref (or native-inputs inputs) "bash")

in gnu/packages/*.scm.

Not ideal, but I’d suggest sticking to that for now.  :-)

Ludo’.

Re: Cross-building GuixSD (and maybe using pre-built toolchains)

[David Craven]

>  (assoc-ref (or native-inputs inputs) "bash")

Yes, that's not too bad. Thanks!

I'll wait for someone to comment the patches I've already submitted to
the mailing list (to see if they are cosher :) before making the
changes.

FYI: I rebuilt linux-libre-headers but the tar test-suite kept
crashing my computer, don't know what that's about. I don't think it's
related to my changes...

Re: Cross-building GuixSD (and maybe using pre-built toolchains)

[David Craven]

Hi,

So one issue that happens when cross building is for things that have
the #:no-substitutes #t enabled. This is mainly found in service
derivations for generating configuration files. I guess if you aren't
using substitutes at all and want to cross-compile everything, it will
take a while till you hit this problem. I have not found a solution to
this problem yet, but I think it should be possible to get
gexp->script and gexp->file to ignore the #:system and #:target, since
I don't think there's a reason why configuration files can't be
cross-built.

David

Re: Cross-building GuixSD (and maybe using pre-built toolchains)

[Ludovic Courtès]

Hi David,

David Craven <david@craven.ch> skribis:

> So one issue that happens when cross building is for things that have
> the #:no-substitutes #t enabled. This is mainly found in service
> derivations for generating configuration files.

I think there’s currently no easy way to cross build a full GuixSD
anyway, like ‘guix system build --target=mips64el-linux-gnu’ (the
--target flag doesn’t exist yet).  Or did you experiment in this area?

I’d be curious.  It’s designed to allow this (in particular because
gexps are cross-compilation-aware), but I’ve never tried.

Ludo’.

Re: Cross-building GuixSD (and maybe using pre-built toolchains)

[David Craven]

Hi Ludo,

>> So one issue that happens when cross building is for things that have
>> the #:no-substitutes #t enabled. This is mainly found in service
>> derivations for generating configuration files.
>
> I think there’s currently no easy way to cross build a full GuixSD
> anyway, like ‘guix system build --target=mips64el-linux-gnu’ (the
> --target flag doesn’t exist yet).  Or did you experiment in this area?
>
> I’d be curious.  It’s designed to allow this (in particular because
> gexps are cross-compilation-aware), but I’ve never tried.

I'm currently not trying to cross-compile everything. The
--system=mips64el-linux gets us pretty far already. The missing parts
are only the ones where no substitutes are available.

What I've done so far is remove grub from the system/vm.scm file which
is added as a hard dependency (even when run with --no-grub). That
gets guix system build arm-minimal.scm --system=armhf-linux to
download all the substitutable dependencies but then fails at the end.

The other thing I've tried is guix system vm arm-minimal.scm
--system=armhf-linux. This fails when it tries to fetch
gstreamer-plugins-bad. This is required to build spice-gtk-viewer
which is required to run the tests for spice. This looks like qemu is
added as an input somewhere when it should be a native-input.

Other than these two issues I don't think that getting guixsd arm
running in qemu is too far off. This of course doesn't help
bootstrapping a new architecture, but it's nice for developers to be
able to test stuff on arm or mips.

Thanks,
David

Re: Cross-building GuixSD (and maybe using pre-built toolchains)

[David Craven]

>> I think there’s currently no easy way to cross build a full GuixSD
>> anyway, like ‘guix system build --target=mips64el-linux-gnu’ (the
>> --target flag doesn’t exist yet).  Or did you experiment in this area?

I'm operating under the assumption that when using the --target flag
packages aren't substituteable (since cross-compiling a package will
likely not yield the same result as natively compiling it) in which
case the --target flag would only be useful for either bootstrapping
or optimizing guixsd for an embedded system or something like that.

Re: Cross-building GuixSD (and maybe using pre-built toolchains)

[Ludovic Courtès]

Hi David,

David Craven <david@craven.ch> skribis:

>>> So one issue that happens when cross building is for things that have
>>> the #:no-substitutes #t enabled. This is mainly found in service
>>> derivations for generating configuration files.
>>
>> I think there’s currently no easy way to cross build a full GuixSD
>> anyway, like ‘guix system build --target=mips64el-linux-gnu’ (the
>> --target flag doesn’t exist yet).  Or did you experiment in this area?
>>
>> I’d be curious.  It’s designed to allow this (in particular because
>> gexps are cross-compilation-aware), but I’ve never tried.
>
> I'm currently not trying to cross-compile everything. The
> --system=mips64el-linux gets us pretty far already. The missing parts
> are only the ones where no substitutes are available.

Aaah OK.  That’s not cross-compilation though, so it works if and only
if substitutes are available or builds can be offloaded to a machine of
that architecture.  Otherwise it cannot really work.

Now, in most cases, the tiny Guile derivations created via
‘gexp->derivation’ et al. in GuixSD (e.g., the derivation that builds
the initrd) could use, say, an x86_64 Guile, even if building for
mips64el.  However, it would be difficult to take advantage of this
AFAICS.

Ludo’.

Re: Cross-building GuixSD (and maybe using pre-built toolchains)

[David Craven]

> Now, in most cases, the tiny Guile derivations created via
> ‘gexp->derivation’ et al. in GuixSD (e.g., the derivation that builds
> the initrd) could use, say, an x86_64 Guile, even if building for
> mips64el.  However, it would be difficult to take advantage of this
> AFAICS.

So maybe I am taking the wrong approach. If an hypothetical --target
flag would exist, could it use natively built substitutes?

Re: Cross-building GuixSD (and maybe using pre-built toolchains)

[Ludovic Courtès]

David Craven <david@craven.ch> skribis:

>> Now, in most cases, the tiny Guile derivations created via
>> ‘gexp->derivation’ et al. in GuixSD (e.g., the derivation that builds
>> the initrd) could use, say, an x86_64 Guile, even if building for
>> mips64el.  However, it would be difficult to take advantage of this
>> AFAICS.
>
> So maybe I am taking the wrong approach. If an hypothetical --target
> flag would exist, could it use natively built substitutes?

It could use substitutes for cross-compiled things, just like “guix
build coreutils --target=mips64el-linux-gnu” can use substitutes.

Ludo’.

Re: Cross-building GuixSD (and maybe using pre-built toolchains)

[Ludovic Courtès]

[-- Attachment #1: Type: text/plain, Size: 1418 bytes --]

ludo@gnu.org (Ludovic Courtès) skribis:

> David Craven <david@craven.ch> skribis:
>
>>>> So one issue that happens when cross building is for things that have
>>>> the #:no-substitutes #t enabled. This is mainly found in service
>>>> derivations for generating configuration files.
>>>
>>> I think there’s currently no easy way to cross build a full GuixSD
>>> anyway, like ‘guix system build --target=mips64el-linux-gnu’ (the
>>> --target flag doesn’t exist yet).  Or did you experiment in this area?
>>>
>>> I’d be curious.  It’s designed to allow this (in particular because
>>> gexps are cross-compilation-aware), but I’ve never tried.
>>
>> I'm currently not trying to cross-compile everything. The
>> --system=mips64el-linux gets us pretty far already. The missing parts
>> are only the ones where no substitutes are available.
>
> Aaah OK.  That’s not cross-compilation though, so it works if and only
> if substitutes are available or builds can be offloaded to a machine of
> that architecture.  Otherwise it cannot really work.
>
> Now, in most cases, the tiny Guile derivations created via
> ‘gexp->derivation’ et al. in GuixSD (e.g., the derivation that builds
> the initrd) could use, say, an x86_64 Guile, even if building for
> mips64el.  However, it would be difficult to take advantage of this
> AFAICS.

There’s an untested hack to illustrate:


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Type: text/x-patch, Size: 2531 bytes --]

diff --git a/guix/gexp.scm b/guix/gexp.scm
index 302879f..53239ac 100644
--- a/guix/gexp.scm
+++ b/guix/gexp.scm
@@ -523,6 +523,8 @@ The other arguments are as for 'derivation'."
             (cons file-name thing)))
          graphs))
 
+  (define gsystem "x86_64-linux")
+
   (mlet* %store-monad (;; The following binding forces '%current-system' and
                        ;; '%current-target-system' to be looked up at >>=
                        ;; time.
@@ -546,13 +548,13 @@ The other arguments are as for 'derivation'."
                                             (object->string sexp)))
                        (modules  (if (pair? %modules)
                                      (imported-modules %modules
-                                                       #:system system
+                                                       #:system gsystem
                                                        #:module-path module-path
                                                        #:guile guile-for-build)
                                      (return #f)))
                        (compiled (if (pair? %modules)
                                      (compiled-modules %modules
-                                                       #:system system
+                                                       #:system gsystem
                                                        #:module-path module-path
                                                        #:guile guile-for-build)
                                      (return #f)))
@@ -587,7 +589,7 @@ The other arguments are as for 'derivation'."
                         ,builder)
                       #:outputs outputs
                       #:env-vars env-vars
-                      #:system system
+                      #:system gsystem
                       #:inputs `((,guile)
                                  (,builder)
                                  ,@(if modules
diff --git a/guix/packages.scm b/guix/packages.scm
index 3646b9b..fe23d35 100644
--- a/guix/packages.scm
+++ b/guix/packages.scm
@@ -1104,7 +1104,7 @@ symbolic output name, such as \"out\".  Note that this procedure calls
   "This monadic procedure changes the Guile currently used to run the build
 code of derivations to GUILE, a package object."
   (lambda (store)
-    (let ((guile (package-derivation store guile)))
+    (let ((guile (package-derivation store guile "x86_64-linux")))
       (values (%guile-for-build guile) store))))
 
 (define* (package-file package

[-- Attachment #3: Type: text/plain, Size: 12 bytes --]


Ludo’.

Cross-building GuixSD (and maybe using pre-built toolchains)

[Paul Boddie]

Hello,

I've been looking at FSF-endorsed Free Software distributions and found 
GuixSD, and I have a few questions about bootstrapping the distribution for 
another architecture.

I was looking at the documentation, which is rather more helpful than most 
distributions when it comes to describing how the actual distribution archives 
are built, and I figured a few things out about guix-daemon. However, I'm 
doing all my work in a chroot - I don't want to "make install" and populate 
various locations on my main system - and this doesn't seem to be entirely 
compatible with the recommended way the daemon is used. Specifically, the 
daemon does not seem to be able to build anything unless I specify the --
disable-chroot flag.

Are there any recommended methods of running guix-daemon in a chroot and have 
it create new chroots, or do I have to use some kind of virtualisation or 
container technology? Is any kind of fakeroot/fakechroot mechanism supported?

One thing that looks very promising is the ability to cross-bootstrap the 
system, and I did manage to get that started for a new architecture (mipsel) 
by performing some modifications suggested in the documentation [1], running 
guix-daemon with the --disable-chroot flag, and then running this:

guix build --target=mipsel-linux-gnu bootstrap-tarballs

However, not wanting to leave my computer switched on all night, I stopped the 
build after a while. I can understand the need to bootstrap things like 
toolchains, but given that I am running Debian which has cross-toolchains for 
mipsel, I wondered if I could short-circuit this process by employing pre-
built toolchains. Would this be possible?

Finally, it seems to be the case that the workflow involves building a 
bootstrapped minimal system and then natively building packages. Or have I 
misunderstood the process? Is it possible to cross-build all of the packages 
in the distribution?

I hope I haven't missed anything that tells me the answers to all of these 
questions.

Thanks in advance for any answers you may have!

Paul

[1] https://www.gnu.org/software/guix/manual/html_node/Porting.html#Porting

Re: Cross-building GuixSD (and maybe using pre-built toolchains)

[Ludovic Courtès]

Hello,

Paul Boddie <paul@boddie.org.uk> skribis:

> I was looking at the documentation, which is rather more helpful than most 
> distributions when it comes to describing how the actual distribution archives 
> are built, and I figured a few things out about guix-daemon. However, I'm 
> doing all my work in a chroot - I don't want to "make install" and populate 
> various locations on my main system - and this doesn't seem to be entirely 
> compatible with the recommended way the daemon is used. Specifically, the 
> daemon does not seem to be able to build anything unless I specify the --
> disable-chroot flag.

How does guix-daemon fail exactly?  If needed, could you strace it to
get more details on what’s going on?

> Are there any recommended methods of running guix-daemon in a chroot and have 
> it create new chroots, or do I have to use some kind of virtualisation or 
> container technology? Is any kind of fakeroot/fakechroot mechanism supported?

I think few people do this, on the grounds that Guix is rather
non-intrusive: it stores all its stuff in /gnu/store and /var/guix.  So
if you want to get rid of it, all you have to do is delete those two
directories.  The rest of the system is untouched.

Now, if you really want it, I can’t think of any reason why guix-daemon
wouldn’t run in a chroot.  It currently requires root privileges,
precisely so that it can set up a chroot, separate name spaces, and so
on, but that could work in a chroot too.

> One thing that looks very promising is the ability to cross-bootstrap the 
> system, and I did manage to get that started for a new architecture (mipsel) 
> by performing some modifications suggested in the documentation [1], running 
> guix-daemon with the --disable-chroot flag, and then running this:
>
> guix build --target=mipsel-linux-gnu bootstrap-tarballs
>
> However, not wanting to leave my computer switched on all night, I stopped the 
> build after a while. I can understand the need to bootstrap things like 
> toolchains, but given that I am running Debian which has cross-toolchains for 
> mipsel, I wondered if I could short-circuit this process by employing pre-
> built toolchains. Would this be possible?

No.  The approach we’re taking is to have a self-contained distro, and
that’s where we get all the fruits of the “functional paradigm”
described in the manual.

It’s surely an annoyance when bootstrapping an architecture for the
first time.  Once it’s in place though, it’s OK.  You could have build
machines for that architecture publishing binaries to users (via ‘guix
publish’).

> Finally, it seems to be the case that the workflow involves building a 
> bootstrapped minimal system and then natively building packages. Or have I 
> misunderstood the process?

Correct, this is the process to port the distro to a new platform.

> Is it possible to cross-build all of the packages in the distribution?

That’s also an option: you can simply cross-build the packages that you
need and copy them to the target machine.

We make sure core packages can be cross-built to the architectures we
support (currently x86_64, i686, armhf, and mips64el).  However, there’s
no guarantee that cross-building works for other targets, or that
non-core packages cross-build at all.  We’d definitely welcome patches
in this area, though.  :-)

HTH!

Ludo’.

Re: Cross-building GuixSD (and maybe using pre-built toolchains)

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 1772 bytes --]

On Sun, Jul 03, 2016 at 04:51:27PM +0200, Paul Boddie wrote:
> Hello,
> 
...
>
> One thing that looks very promising is the ability to cross-bootstrap the 
> system, and I did manage to get that started for a new architecture (mipsel) 
> by performing some modifications suggested in the documentation [1], running 
> guix-daemon with the --disable-chroot flag, and then running this:
> 
> guix build --target=mipsel-linux-gnu bootstrap-tarballs
> 
> However, not wanting to leave my computer switched on all night, I stopped the 
> build after a while. I can understand the need to bootstrap things like 
> toolchains, but given that I am running Debian which has cross-toolchains for 
> mipsel, I wondered if I could short-circuit this process by employing pre-
> built toolchains. Would this be possible?

I built a set of bootstrap-tarballs on my very underpowered laptop
which took about 15 hours. The longest single package was gcc-4.9, which
took me about 5 hours. On a more powerful machine it should go much
faster.

> Finally, it seems to be the case that the workflow involves building a 
> bootstrapped minimal system and then natively building packages. Or have I 
> misunderstood the process? Is it possible to cross-build all of the packages 
> in the distribution?
> 
> I hope I haven't missed anything that tells me the answers to all of these 
> questions.
> 
> Thanks in advance for any answers you may have!
> 
> Paul
> 
> [1] https://www.gnu.org/software/guix/manual/html_node/Porting.html#Porting
> 

-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]