Running guix on nixos

Running guix on nixos

[Jeff Mickey]

There is a host at work that is running nixos. I thought "hey, I'll try
doing that whole shared daemon thing".

I have not succeeded yet. The approach has been to use the same
nix-daemon, as I saw the hydra.gnu.org instance does something similar.

I ran in the following shell:

nix-shell -p coreutils findutils automake autogen gettext guile \
  pkgconfig gcc libgcrypt sqlite bzip2 graphviz gnutls help2man

And I configured with the following configure line:

./configure \
  --prefix=/home/codemac/guix \
  --with-libgcrypt-prefix=/nix/store/mrr...-libgcrypt-1.6.4 \
  --disable-daemon \
  --with-store-dir=/nix/store \
  --localstatedir=/nix/var

This built successfully but runs into the following problems:

$ guix pull
guix pull: error: failed to connect to `/nix/var/guix/daemon-socket/socket': No such file or directory

Ran sudo ln -s /nix/var/nix /nix/var/guix to get around that, but I'd
like it to look properly in the statedir when I do
--disable-daemon, so if there is a better way to do it I'm all ears.

In attempting to guix pull again, it fails to build gcc complaining:

/nix/store/xw7...-binutils-cross-boot0-2.25.1/bin/x86_64-guix-linux-gnu-ld: cannot find -lstdc++

Is there a way to insert the boostrap binaries into the nix store? I'm
assuming that's what's happening?

Any help would be appreciated, I apologize if this is not the proper
venue.

  //  codemac

Re: Running guix on nixos

[Ludovic Courtès]

Jeff Mickey <j@codemac.net> skribis:

> There is a host at work that is running nixos. I thought "hey, I'll try
> doing that whole shared daemon thing".

:-)

> I have not succeeded yet. The approach has been to use the same
> nix-daemon, as I saw the hydra.gnu.org instance does something similar.

hydra.gnu.org did that long ago, but nowadays it uses guix-daemon and
/gnu/store.  Hydra (the software) calls out to Nix command-line tools,
which talk to guix-daemon.

> $ guix pull
> guix pull: error: failed to connect to `/nix/var/guix/daemon-socket/socket': No such file or directory
>
> Ran sudo ln -s /nix/var/nix /nix/var/guix to get around that, but I'd
> like it to look properly in the statedir when I do
> --disable-daemon, so if there is a better way to do it I'm all ears.

I don’t think there’s a better way.

> In attempting to guix pull again, it fails to build gcc complaining:
>
> /nix/store/xw7...-binutils-cross-boot0-2.25.1/bin/x86_64-guix-linux-gnu-ld: cannot find -lstdc++

Hmm, weird.  Could you try:

  guix build -K hello

and send the excerpt of the build log that’s failing?

Also, make sure nix-daemon uses pristine chroot builds.  It used to be
the case that chroot builds were not used by default, and that when
doing chroot builds, the chroot would be augmented with at least /bin
taken from the host system, which brings in /bin/sh but also a number of
related problems:

  https://lists.gnu.org/archive/html/bug-guix/2013-01/msg00041.html

HTH,
Ludo’.

Re: Running guix on nixos

[Jeff Mickey]

* Ludovic Courtès <ludo@gnu.org> [2016-01-21 01:02]:
>   guix build -K hello

This fails with the exact same error, it can't build gcc.

g++   -g -DIN_GCC    -fno-exceptions -fno-rtti -fasynchronous-unwind-tables -W -Wall -Wno-narrowing -Wwrite-strings -Wcast-qual -Wno-format -Wmissing-format-attribute -pedantic -Wno-long-long -Wno-variadic-macros -Wno-overlength-strings -fno-common  -DHAVE_CONFIG_H -DGENERATOR_FILE -Wl,-rpath=/nix/store/dbdj046q2ybl01ld56ywr0rkpy91g0xz-glibc-2.22/lib -Wl,-dynamic-linker -Wl,/nix/store/dbdj046q2ybl01ld56ywr0rkpy91g0xz-glibc-2.22/lib/ld-linux-x86-64.so.2 -L/nix/store/df8qyymanigrki2rgcmwbpkygg7c5n2g-libstdc++-4.9.3/lib -o build/genhooks \
    build/genhooks.o build/errors.o ../build-x86_64-unknown-linux-gnu/libiberty/libiberty.a
g++   -g -DIN_GCC    -fno-exceptions -fno-rtti -fasynchronous-unwind-tables -W -Wall -Wno-narrowing -Wwrite-strings -Wcast-qual -Wno-format -Wmissing-format-attribute -pedantic -Wno-long-long -Wno-variadic-macros -Wno-overlength-strings -fno-common  -DHAVE_CONFIG_H -DGENERATOR_FILE -Wl,-rpath=/nix/store/dbdj046q2ybl01ld56ywr0rkpy91g0xz-glibc-2.22/lib -Wl,-dynamic-linker -Wl,/nix/store/dbdj046q2ybl01ld56ywr0rkpy91g0xz-glibc-2.22/lib/ld-linux-x86-64.so.2 -L/nix/store/df8qyymanigrki2rgcmwbpkygg7c5n2g-libstdc++-4.9.3/lib -o build/genchecksum \
    build/genchecksum.o ../build-x86_64-unknown-linux-gnu/libiberty/libiberty.a
/nix/store/xw7xz3cr8nhw24m6cchiha8g6lslwv1j-binutils-cross-boot0-2.25.1/bin/x86_64-guix-linux-gnu-ld: cannot find -lstdc++
collect2: error: ld returned 1 exit status
Makefile:2532: recipe for target 'build/genhooks' failed
make[3]: *** [build/genhooks] Error 1
make[3]: *** Waiting for unfinished jobs....
/nix/store/xw7xz3cr8nhw24m6cchiha8g6lslwv1j-binutils-cross-boot0-2.25.1/bin/x86_64-guix-linux-gnu-ld: cannot find -lstdc++
collect2: error: ld returned 1 exit status
Makefile:2532: recipe for target 'build/genchecksum' failed


If I look in
/nix/store/df8qyymanigrki2rgcmwbpkygg7c5n2g-libstdc++-4.9.3/lib I see
the following:

$ ls /nix/store/df8qyymanigrki2rgcmwbpkygg7c5n2g-libstdc++-4.9.3/lib
$ ls /nix/store/df8qyymanigrki2rgcmwbpkygg7c5n2g-libstdc++-4.9.3/
include  lib  lib64  share
$ ls /nix/store/df8qyymanigrki2rgcmwbpkygg7c5n2g-libstdc++-4.9.3/lib64
libstdc++.a  libstdc++.a-gdb.py  libstdc++.la  libsupc++.a  libsupc++.la
$

So.. it looks like the lib64 part isn't getting propagated correctly in
the guix build.

> Also, make sure nix-daemon uses pristine chroot builds.

[...]

Yes, had to add nix.useChroot to true in my
/etc/nixos/configuration.nix. God I miss some s-exp.

Thanks for your help! Really appreciate it.

  //  codemac

Re: Running guix on nixos

[Ludovic Courtès]

Jeff Mickey <j@codemac.net> skribis:

> * Ludovic Courtès <ludo@gnu.org> [2016-01-21 01:02]:
>>   guix build -K hello
>
> This fails with the exact same error, it can't build gcc.
>
> g++   -g -DIN_GCC    -fno-exceptions -fno-rtti -fasynchronous-unwind-tables -W -Wall -Wno-narrowing -Wwrite-strings -Wcast-qual -Wno-format -Wmissing-format-attribute -pedantic -Wno-long-long -Wno-variadic-macros -Wno-overlength-strings -fno-common  -DHAVE_CONFIG_H -DGENERATOR_FILE -Wl,-rpath=/nix/store/dbdj046q2ybl01ld56ywr0rkpy91g0xz-glibc-2.22/lib -Wl,-dynamic-linker -Wl,/nix/store/dbdj046q2ybl01ld56ywr0rkpy91g0xz-glibc-2.22/lib/ld-linux-x86-64.so.2 -L/nix/store/df8qyymanigrki2rgcmwbpkygg7c5n2g-libstdc++-4.9.3/lib -o build/genhooks \
>     build/genhooks.o build/errors.o ../build-x86_64-unknown-linux-gnu/libiberty/libiberty.a
> g++   -g -DIN_GCC    -fno-exceptions -fno-rtti -fasynchronous-unwind-tables -W -Wall -Wno-narrowing -Wwrite-strings -Wcast-qual -Wno-format -Wmissing-format-attribute -pedantic -Wno-long-long -Wno-variadic-macros -Wno-overlength-strings -fno-common  -DHAVE_CONFIG_H -DGENERATOR_FILE -Wl,-rpath=/nix/store/dbdj046q2ybl01ld56ywr0rkpy91g0xz-glibc-2.22/lib -Wl,-dynamic-linker -Wl,/nix/store/dbdj046q2ybl01ld56ywr0rkpy91g0xz-glibc-2.22/lib/ld-linux-x86-64.so.2 -L/nix/store/df8qyymanigrki2rgcmwbpkygg7c5n2g-libstdc++-4.9.3/lib -o build/genchecksum \
>     build/genchecksum.o ../build-x86_64-unknown-linux-gnu/libiberty/libiberty.a
> /nix/store/xw7xz3cr8nhw24m6cchiha8g6lslwv1j-binutils-cross-boot0-2.25.1/bin/x86_64-guix-linux-gnu-ld: cannot find -lstdc++
> collect2: error: ld returned 1 exit status
> Makefile:2532: recipe for target 'build/genhooks' failed
> make[3]: *** [build/genhooks] Error 1
> make[3]: *** Waiting for unfinished jobs....
> /nix/store/xw7xz3cr8nhw24m6cchiha8g6lslwv1j-binutils-cross-boot0-2.25.1/bin/x86_64-guix-linux-gnu-ld: cannot find -lstdc++
> collect2: error: ld returned 1 exit status
> Makefile:2532: recipe for target 'build/genchecksum' failed
>
>
> If I look in
> /nix/store/df8qyymanigrki2rgcmwbpkygg7c5n2g-libstdc++-4.9.3/lib I see
> the following:
>
> $ ls /nix/store/df8qyymanigrki2rgcmwbpkygg7c5n2g-libstdc++-4.9.3/lib
> $ ls /nix/store/df8qyymanigrki2rgcmwbpkygg7c5n2g-libstdc++-4.9.3/
> include  lib  lib64  share
> $ ls /nix/store/df8qyymanigrki2rgcmwbpkygg7c5n2g-libstdc++-4.9.3/lib64
> libstdc++.a  libstdc++.a-gdb.py  libstdc++.la  libsupc++.a  libsupc++.la

I think there’s an impurity leading the build system to use lib64/
instead of lib/.  Compare with what we get with “real” Guix:

--8<---------------cut here---------------start------------->8---
$ ls $(guix build -e '(@@ (gnu packages commencement) libstdc++)')
include/  lib/  share/
--8<---------------cut here---------------end--------------->8---

>> Also, make sure nix-daemon uses pristine chroot builds.
>
> [...]
>
> Yes, had to add nix.useChroot to true in my
> /etc/nixos/configuration.nix. God I miss some s-exp.

Heh.

Could it be that the above libstdc++ was built before you had enabled
chroot builds?  Are you sure the chroot doesn’t contain /bin or anything
like that?

(Are you sure you don’t want to use guix-daemon?  :-))

Ludo’.

Re: Running guix on nixos

[Jeff Mickey]

* Jeff Mickey <j@codemac.net> [2016-01-21 14:24]:
> Yes, had to add nix.useChroot to true in my
> /etc/nixos/configuration.nix. God I miss some s-exp.

To continue the saga, this did not help the lib64 problem :/

Will try a few more things today, would love to get guix as a package on
nixos that "just works (tm)".

  //  mickey

Re: Running guix on nixos

[Jeff Mickey]

* Ludovic Courtès <ludo@gnu.org> [2016-01-22 04:57]:
> I think there’s an impurity leading the build system to use lib64/
> instead of lib/.  Compare with what we get with “real” Guix:
>
> --8<---------------cut here---------------start------------->8---
> $ ls $(guix build -e '(@@ (gnu packages commencement) libstdc++)')
> include/  lib/  share/
> --8<---------------cut here---------------end--------------->8---
>
> Could it be that the above libstdc++ was built before you had enabled
> chroot builds?  Are you sure the chroot doesn’t contain /bin or anything
> like that?

$ ls $(guix build -e '(@@ (gnu packages commencement) libstdc++)')
include  lib  lib64  share

Aha! So I ran nix-garbage-collect (which deleted all the things),
rebuilt andd.... still the same :(

$ ls $(guix build -e '(@@ (gnu packages commencement) libstdc++)')
include  lib  lib64  share

> (Are you sure you don’t want to use guix-daemon?  :-))

What I would like as an end result is my host to be reproducible from a
`nixos-rebuild switch` command line, and then user's have access to
guix's wonderfulness. I'm realizing now also that if guix build's all of
it's own stuff anyways, then sharing the same daemon is just a win for
simplicity of runtime (only one daemon), but not much simplicity on disk
(still two profiles everywhere, almost an entirely separate graph on
disk).

So, I guess now I need to learn more about nix's configuration file to
get this set up in /gnu properly for users in a nixos way. An odd goal,
I admit.

Dankon por via helpo, Ludo!

  //  codemac

Re: Running guix on nixos

[Ludovic Courtès]

Saluton!

Jeff Mickey <j@codemac.net> skribis:

> * Ludovic Courtès <ludo@gnu.org> [2016-01-22 04:57]:
>> I think there’s an impurity leading the build system to use lib64/
>> instead of lib/.  Compare with what we get with “real” Guix:
>>
>> --8<---------------cut here---------------start------------->8---
>> $ ls $(guix build -e '(@@ (gnu packages commencement) libstdc++)')
>> include/  lib/  share/
>> --8<---------------cut here---------------end--------------->8---
>>
>> Could it be that the above libstdc++ was built before you had enabled
>> chroot builds?  Are you sure the chroot doesn’t contain /bin or anything
>> like that?
>
> $ ls $(guix build -e '(@@ (gnu packages commencement) libstdc++)')
> include  lib  lib64  share
>
> Aha! So I ran nix-garbage-collect (which deleted all the things),
> rebuilt andd.... still the same :(

It would be interesting (on an intellectual level ;-)) to find out what
piece of libstdc++’s build system is responsible for choosing lib/
vs. lib64/ and what makes it choose something different.  I’m guessing
there’s necessarily an impurity in the build environment that explains
this.

Could you post the build log of this libstdc++ somewhere?  (Use ‘guix
build --log-file -e '(@@ (gnu packages commencement) libstdc++)'’.)
Then we can compare it to
<http://hydra.gnu.org/log/3wicw2xrq38hpn4x28lcqsvbyqzrmwka-libstdc++-4.9.3>.

> What I would like as an end result is my host to be reproducible from a
> `nixos-rebuild switch` command line, and then user's have access to
> guix's wonderfulness. I'm realizing now also that if guix build's all of
> it's own stuff anyways, then sharing the same daemon is just a win for
> simplicity of runtime (only one daemon), but not much simplicity on disk
> (still two profiles everywhere, almost an entirely separate graph on
> disk).

Yes, you would be using two completely separate distros anyway.  The
advantages of using a single store, though, is that you would get
deduplication across the two distros, and running ‘guix gc’ or
‘nix-collect-garbage’ would affect the whole store.

Ludo’.

Re: Running guix on nixos

[Jeff Mickey]

* Ludovic Courtès <ludo@gnu.org> [2016-01-23 12:48]:
> It would be interesting (on an intellectual level ;-)) to find out what
> piece of libstdc++’s build system is responsible for choosing lib/
> vs. lib64/ and what makes it choose something different.  I’m guessing
> there’s necessarily an impurity in the build environment that explains
> this.

Never figured this out, seems it was the chroot issue, and maybe not
running garbage collection when I thought I was. Sorry I don't have any
logs.

> Yes, you would be using two completely separate distros anyway.  The
> advantages of using a single store, though, is that you would get
> deduplication across the two distros, and running ‘guix gc’ or
> ‘nix-collect-garbage’ would affect the whole store.

I successfully got both the guix-daemon and the nix-daemon running on
the same machine through nix's configuration syntax.

Unfortunately, there is something wrong with how it uses acl, as it
warns me every time and then re-builds the world. Just installing guix
this way as it stands took 10+ hours of compute on an 8 core xeon with
32 gigs of ram.

So.. I need to figure out the substitutes issue (which mark_weaver on
irc rightly pointed out requires the second daemon in /gnu to have all
the package derivations match), and I think maybe my sysconfdir gets set
incorrectly.

Now if only I had fully free hardware so I could hack on guixsd more
easily.. but other than that, I'm very close! Redankon!

  //  codemac

Re: Running guix on nixos

[Ludovic Courtès]

Jeff Mickey <j@codemac.net> skribis:

> * Ludovic Courtès <ludo@gnu.org> [2016-01-23 12:48]:
>> It would be interesting (on an intellectual level ;-)) to find out what
>> piece of libstdc++’s build system is responsible for choosing lib/
>> vs. lib64/ and what makes it choose something different.  I’m guessing
>> there’s necessarily an impurity in the build environment that explains
>> this.
>
> Never figured this out, seems it was the chroot issue, and maybe not
> running garbage collection when I thought I was.

You mean it works now?

> Sorry I don't have any logs.

Logs are automatically kept in /var/guix or /nix/var/something (see
‘guix build --log-file’), so it may still be possible to retrieve them.

>> Yes, you would be using two completely separate distros anyway.  The
>> advantages of using a single store, though, is that you would get
>> deduplication across the two distros, and running ‘guix gc’ or
>> ‘nix-collect-garbage’ would affect the whole store.
>
> I successfully got both the guix-daemon and the nix-daemon running on
> the same machine through nix's configuration syntax.
>
> Unfortunately, there is something wrong with how it uses acl, as it
> warns me every time and then re-builds the world. Just installing guix
> this way as it stands took 10+ hours of compute on an 8 core xeon with
> 32 gigs of ram.

What’s the warning?  That /etc/guix/acl is empty?
See <https://www.gnu.org/software/guix/manual/html_node/Substitutes.html> 
if that is the case.

Ĝis,
Ludo’.

Re: Running guix on nixos

[Jeff Mickey]

* Ludovic Courtès <ludo@gnu.org> [2016-01-25 00:06]:
> Logs are automatically kept in /var/guix or /nix/var/something (see
> ‘guix build --log-file’), so it may still be possible to retrieve them.

Hm, that command returns empty, and I don't see any logs anywhere. I
wasn't deing detailed enough while working on this, I apologize for
missing the opportunity to debug it.

>> Unfortunately, there is something wrong with how it uses acl, as it
>> warns me every time and then re-builds the world.
>
> What’s the warning?  That /etc/guix/acl is empty?

It says:

substitute: guix substitute: warning: ACL for archive imports seems to be uninitialized, substitutes may be unavailable

And I got an idea while replying to your mail. NixOS sets the
NIX_CONF_DIR environment variable for users, and it looks like the guix
command respects it and places the acl file in /etc/nix/acl.

The guix-daemon however is being launched from systemd, and it looks
like nix doesn't set up those same environment variables for services?

$ sudo systemctl show-environment
LANG=en_US.UTF-8
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin

Well, fek.

guix-daemon defaults to using whatever guix was compiled with, and as
this is the binary installation method, so it was probably compiled with
/etc/guix.

I apologize, an environmental mess up.

It seems that running guix on top of NixOS has much more interesting
edge cases compared to my previous arch+guix and debian+guix installs
due to the overlap in functionality of the distros. Those others were a
breeze compared to this.

Is there anywhere I can document putting guix on NixOS from binary
installation? It'd be nice to have a per-distro "how to install guix"
instructions for each's corner case.

Thanks again for your help! And I have got to brush up on my
Esperanto. (Redankon? Dankon ree? Dankon denove? Cxu "cimo"?)

  //  codemac

Re: Running guix on nixos

[Leo Famulari]

On Mon, Jan 25, 2016 at 11:29:36AM -0800, Jeff Mickey wrote:
> * Ludovic Courtès <ludo@gnu.org> [2016-01-25 00:06]:
> > Logs are automatically kept in /var/guix or /nix/var/something (see
> > ‘guix build --log-file’), so it may still be possible to retrieve them.
> 
> Hm, that command returns empty, and I don't see any logs anywhere. I
> wasn't deing detailed enough while working on this, I apologize for
> missing the opportunity to debug it.

It will return nothing unless you give it a package as an argument:
$ guix build --log-file hello
/var/log/guix/drvs/17/n7nhssy98idxcs4zgs7aprfd3f7w7z-hello-2.10.drv.bz2

If the package was built remotely then it will return a path to the
remote log.
$ guix build --log-file webkitgtk
http://hydra.gnu.org/log/zzpb2i9926w9y9kwc2kvkf1hibmz2izj-webkitgtk-2.10.4

> 
> >> Unfortunately, there is something wrong with how it uses acl, as it
> >> warns me every time and then re-builds the world.
> >
> > What’s the warning?  That /etc/guix/acl is empty?
> 
> It says:
> 
> substitute: guix substitute: warning: ACL for archive imports seems to be uninitialized, substitutes may be unavailable
> 
> And I got an idea while replying to your mail. NixOS sets the
> NIX_CONF_DIR environment variable for users, and it looks like the guix
> command respects it and places the acl file in /etc/nix/acl.
> 
> The guix-daemon however is being launched from systemd, and it looks
> like nix doesn't set up those same environment variables for services?
> 
> $ sudo systemctl show-environment
> LANG=en_US.UTF-8
> PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
> 
> Well, fek.
> 
> guix-daemon defaults to using whatever guix was compiled with, and as
> this is the binary installation method, so it was probably compiled with
> /etc/guix.
> 
> I apologize, an environmental mess up.
> 
> It seems that running guix on top of NixOS has much more interesting
> edge cases compared to my previous arch+guix and debian+guix installs
> due to the overlap in functionality of the distros. Those others were a
> breeze compared to this.
> 
> Is there anywhere I can document putting guix on NixOS from binary
> installation? It'd be nice to have a per-distro "how to install guix"
> instructions for each's corner case.
> 
> Thanks again for your help! And I have got to brush up on my
> Esperanto. (Redankon? Dankon ree? Dankon denove? Cxu "cimo"?)
> 
>   //  codemac
> 

Re: Running guix on nixos

[Ludovic Courtès]

Jeff Mickey <j@codemac.net> skribis:

> And I got an idea while replying to your mail. NixOS sets the
> NIX_CONF_DIR environment variable for users, and it looks like the guix
> command respects it and places the acl file in /etc/nix/acl.

Indeed.

> The guix-daemon however is being launched from systemd, and it looks
> like nix doesn't set up those same environment variables for services?
>
> $ sudo systemctl show-environment
> LANG=en_US.UTF-8
> PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
>
> Well, fek.
>
> guix-daemon defaults to using whatever guix was compiled with, and as
> this is the binary installation method, so it was probably compiled with
> /etc/guix.
>
> I apologize, an environmental mess up.
>
> It seems that running guix on top of NixOS has much more interesting
> edge cases compared to my previous arch+guix and debian+guix installs
> due to the overlap in functionality of the distros. Those others were a
> breeze compared to this.

Well, it seems that the main issue is that Guix honors the same
environment variables as Nix, which causes confusion.

We should probably fix it.

> Is there anywhere I can document putting guix on NixOS from binary
> installation? It'd be nice to have a per-distro "how to install guix"
> instructions for each's corner case.

I’d rather avoid per-distro sections.  The “Application Setup” section
is meant to contain all the hints for using Guix on a “foreign distro”,
and those hints should be pretty much the same on all distros.

So I think the problems you found with NixOS indicate that we must fix
the environment variable collisions.

Ĝoja kodado!  :-)

Ludo’.