Daemon update

Daemon update

[Ludovic Courtès]

Hello!

I have pushed a big update of the daemon code.  We were using a copy of
the Nix code from May 2014, and there have been worthwhile changes in
the meantime (thanks for the Nix folks!).

There have also been undesirable changes, such as the gratuitous switch
to multithreading, which entails various issues (see
<http://lists.gnu.org/archive/html/guix-devel/2014-12/msg00363.html>.)

Thus, in our ‘nix’ branch, I have selected a the best of those changes
;-).  The main difference is that I didn’t take said
multithreading-related commits, as well as a couple of less significant,
Nix-specific changes.

The new changes include some refactorization and cosmetic changes (a
bunch of C++11 lambdas here, a bit of ‘auto’ there.)  Interesting
changes include:

  • The possibility to use regular files as GC roots, where the file
    name denotes a store item; this reduces I/O compared to using
    symlinks.

  • Slightly reduced I/O for GC.

  • New ‘disallowedReferences’ and ‘disallowedRequisites’ attributes for
    derivations (akin to our #:allowed-references.)

  • New ‘optimiseStore’ RPC.

  • ASLR disabled in the build environment.

Please report any issues with the daemon!

Thanks,
Ludo’.

Re: Daemon update

[Alex Vorobiev]

Hi,

Ludovic Courtès <ludo <at> gnu.org> writes:
> bunch of C++11 lambdas here, a bit of ‘auto’ there.)  

Does it mean it can't be installed on systems with older c++ compilers? I 
am on RHEL 6.5 which has gcc-4.4.7 and ironically one of my main reasons to 
try guix was to be able to install newer versions of various applications 
so that they coexisted with the ones which were shipped with the system and 
didn't break anything. My current solution is to build everything myself 
and use GNU stow to keep it separate.

Thanks,
Alex

Re: Daemon update

[Taylan Ulrich Bayırlı/Kammer]

Alex Vorobiev <alexander.vorobiev@gmail.com> writes:

> Ludovic Courtès <ludo <at> gnu.org> writes:
>> bunch of C++11 lambdas here, a bit of ‘auto’ there.)  
>
> Does it mean it can't be installed on systems with older c++
> compilers?

FYI Guix can be "bootstrapped" very easily with the new binary tarballs
without needing to compile anything.

See the master branch's manual for some pitfalls that made it into the
0.8.2 release.  (Namely, extracting the binary tarball might change the
ownership and permissions of /, /root, and /var, if you don't pass
--skip-old-files to tar.)

After extracting the tarball there will be a /root/.guix-profile with
guix installed, and any user can call:

  /gnu/store/...-guix-.../bin/guix package -i guix

to create their ~/.guix-profile with guix installed.

After that, git checkouts of guix can be built via:

  guix environment guix --pure -E 'autoreconf -vif'
  guix environment guix --pure -E \
    "./configure --with-libgcrypt-prefix=$(guix build libgcrypt | head -1)"
  guix environment guix --pure -E make

and for later re-compilations only the last line is necessary, and
sometimes the first.  'make install' should probably be avoided.

Hope that helps,
Taylan

Re: Daemon update

[Ludovic Courtès]

Alex Vorobiev <alexander.vorobiev@gmail.com> skribis:

> Does it mean it can't be installed on systems with older c++
> compilers?

On these systems, I would suggest installing from the binary tarball.

Not having to build Guile, libgcrypt, and everything on such a system is
already a good incentive to use the binary tarball.

How does that sound?

Thanks,
Ludo’.

Re: Daemon update

[Alexander Vorobiev]

[-- Attachment #1: Type: text/plain, Size: 987 bytes --]

Ludo', Taylan,

Doesn't binary distribution have hardcoded /gnu/* paths? I can't use those
unfortunately. We have a standard configuration of RHEL 6.5 installed on
hundreds of servers and any modifications of the root directory (and all
other "standard" directories) layout are out of question. Would it be too
hard to add an environment variable(s) pointing to a Guix's store and cache
directories so that the binary build of the daemon doesn't depend on the
hardcoded values?
Thanks,
Alex

On Thu, May 21, 2015 at 3:24 AM, Ludovic Courtès <ludo@gnu.org> wrote:

> Alex Vorobiev <alexander.vorobiev@gmail.com> skribis:
>
> > Does it mean it can't be installed on systems with older c++
> > compilers?
>
> On these systems, I would suggest installing from the binary tarball.
>
> Not having to build Guile, libgcrypt, and everything on such a system is
> already a good incentive to use the binary tarball.
>
> How does that sound?
>
> Thanks,
> Ludo’.
>

[-- Attachment #2: Type: text/html, Size: 1452 bytes --]

Re: Daemon update

[Taylan Ulrich Bayırlı/Kammer]

Alexander Vorobiev <alexander.vorobiev@gmail.com> writes:

> Doesn't binary distribution have hardcoded /gnu/* paths? I can't use
> those unfortunately. We have a standard configuration of RHEL 6.5
> installed on hundreds of servers and any modifications of the root
> directory (and all other "standard" directories) layout are out of
> question. Would it be too hard to add an environment variable(s)
> pointing to a Guix's store and cache directories so that the binary
> build of the daemon doesn't depend on the hardcoded values?

I'm afraid it's not the daemon itself for which changing the store path
is problematic (although it's not supported at run-time AFAIK, only at
configure-time via --with-store-dir=), it's that any built packages in
/gnu/store are likely to have "hardcoded" absolute references to other
/gnu/store files.

I think this can't really be avoided either, because some pieces of
software have configuration files and such which only support plain file
paths without any dynamic parameterization, meaning the recipe for the
package has to patch said files to contain absolute references to
/gnu/store files when they have to refer to any files from their
dependencies.

For example, when sources of package A contain a dlopen() call to open a
library from package B, then A's source is patched to turn the file name
argument to dlopen() into an absolute path to package B's library file
in /gnu/store.  That is because dlopen() does not support changes to its
search path in a portable manner.  (Maybe one day we will hack dlopen to
fix this, but it's just one example.)

This would mean that to use a non-standard store directory, you will
have to build Guix packages locally, as well as Guix itself.  Unless I'm
missing something.


You could still use the binary tarballs to install Guix on a machine
where you are permitted to, so you can easily compile another Guix via
the method described at the bottom here:

https://lists.gnu.org/archive/html/guix-devel/2015-05/msg00360.html


I'm afraid this doesn't help very much, but it's all I've got. :-\

Taylan

Re: Daemon update

[Ludovic Courtès]

Alexander Vorobiev <alexander.vorobiev@gmail.com> skribis:

> Doesn't binary distribution have hardcoded /gnu/* paths?

Yes it does.

> I can't use those unfortunately. We have a standard configuration of
> RHEL 6.5 installed on hundreds of servers and any modifications of the
> root directory (and all other "standard" directories) layout are out
> of question.

OK.  Note that using a store other than /gnu/store prevents you from
using substitutes from hydra.gnu.org (just mentioning it if that is of
interest to you.)

> Would it be too hard to add an environment variable(s) pointing to a
> Guix's store and cache directories so that the binary build of the
> daemon doesn't depend on the hardcoded values?  Thanks, Alex

There are two issues here:

  1. File names hard-coded in the RUNPATH and other places of the
     ‘guix-daemon’ binary of the tarball.

  2. The store directory that ‘guix-daemon’ targets.

#2 can actually be changed at run-time via the (purposefully
undocumented) ‘NIX_STORE_DIR’ environment variable (see ‘test-env.in’ in
the source tree.)

#1 means that if you want to run the ‘guix-daemon’ binary from the
binary tarball, you’ll have to somehow satisfy its requirements.  You
might be able to unpack the tarball in an arbitrary directory, and then
define ‘LD_LIBRARY_PATH’ so that ‘guix-daemon’ finds all the .so files
that it needs and are provided by the binary tarball (libstdc++.so,
libgcrypt.so, libsqlite3.so, libc.so, etc.)

Once ‘guix-daemon’ is running, you could run ‘guix environment guix’,
and from there you could build Guix from source.

But that is really hacky, and perhaps not particularly easier than
building a newer GCC on your machine.

Alternately, you could build a suitable binary tarball on another
machine: just install and configure Guix with the options that you need,
and run ‘make guix-binary.x86_64-linux.tar.xz’.

WDYT?

Ludo’.

Re: Daemon update

[Alexander Vorobiev]

[-- Attachment #1: Type: text/plain, Size: 4494 bytes --]

Ok, I have just tried to build the binary tarball on a VM where I
reproduced all the paths I want (basically, instead of /gnu I want
/shared/shape_tier3/common/local/guix) and which has c++11 compliant gcc --
that also failed. I pulled today's git and ran guix-daemon
--no-substitutes. The error seems to be "unable to fork: Operation not
permitted". I am running guix-daemon as myself and I have all permissions
to the /.../local/guix):

GUIX_PACKAGE_PATH= \
./pre-inst-env "/usr/bin/guile"                 \
  "./build-aux/make-binary-tarball.scm" "x86_64-linux"
"guix-binary.x86_64-linux.tar.xz"

.........


|   recursively deleting path
`/shared/shape_tier3/common/local/guix/store/m38laca1idmvx8m03dl6b5qmn8xhv9c3-guile-bootstrap-2.0.drv.chroot'
|   lock released on
`/shared/shape_tier3/common/local/guix/store/pgs6vqy03wg2m5mn2ahvi4sarx89vnls-guile-bootstrap-2.0.lock'
|   building of
`/shared/shape_tier3/common/local/guix/store/m38laca1idmvx8m03dl6b5qmn8xhv9c3-guile-bootstrap-2.0.drv':
goal destroyed
Backtrace:
In ice-9/boot-9.scm:
 157: 11 [catch #t #<catch-closure 198d7e0> ...]
In unknown file:
   ?: 10 [apply-smob/1 #<catch-closure 198d7e0>]
In ice-9/boot-9.scm:
  63: 9 [call-with-prompt prompt0 ...]
In ice-9/eval.scm:
 432: 8 [eval # #]
In ice-9/boot-9.scm:
2401: 7 [save-module-excursion #<procedure 19aa9c0 at
ice-9/boot-9.scm:4045:3 ()>]
4052: 6 [#<procedure 19aa9c0 at ice-9/boot-9.scm:4045:3 ()>]
In unknown file:
   ?: 5 [load-compiled/vm
"/home/alex/.cache/guile/ccache/2.0-LE-8-2.0/home/alex/src/guix/build-aux/make-binary-tarball.scm.go"]
In build-aux/make-binary-tarball.scm:
  38: 4 [#<procedure 19bd840 ()>]
In guix/store.scm:
 986: 3 [run-with-store # ...]
In build-aux/make-binary-tarball.scm:
  41: 2 [#<procedure 315e840 at build-aux/make-binary-tarball.scm:40:7
(state)> #]
In guix/store.scm:
 933: 1 [#<procedure 3aac870 at guix/store.scm:932:28 (store)> #]
 624: 0 [build-things # #]

guix/store.scm:624:0: In procedure build-things:
guix/store.scm:624:0: Throw to key `srfi-34' with args `(#<condition
&nix-protocol-error [message: "unable to fork: Operation not permitted"
status: 1] 3d65f60>)'.
Makefile:4331: recipe for target 'guix-binary.x86_64-linux.tar.xz' failed
make: [guix-binary.x86_64-linux.tar.xz] Error 1 (ignored)

Thanks,
Alex

On Fri, May 22, 2015 at 8:45 AM, Ludovic Courtès <ludo@gnu.org> wrote:

> Alexander Vorobiev <alexander.vorobiev@gmail.com> skribis:
>
> > Doesn't binary distribution have hardcoded /gnu/* paths?
>
> Yes it does.
>
> > I can't use those unfortunately. We have a standard configuration of
> > RHEL 6.5 installed on hundreds of servers and any modifications of the
> > root directory (and all other "standard" directories) layout are out
> > of question.
>
> OK.  Note that using a store other than /gnu/store prevents you from
> using substitutes from hydra.gnu.org (just mentioning it if that is of
> interest to you.)
>
> > Would it be too hard to add an environment variable(s) pointing to a
> > Guix's store and cache directories so that the binary build of the
> > daemon doesn't depend on the hardcoded values?  Thanks, Alex
>
> There are two issues here:
>
>   1. File names hard-coded in the RUNPATH and other places of the
>      ‘guix-daemon’ binary of the tarball.
>
>   2. The store directory that ‘guix-daemon’ targets.
>
> #2 can actually be changed at run-time via the (purposefully
> undocumented) ‘NIX_STORE_DIR’ environment variable (see ‘test-env.in’ in
> the source tree.)
>
> #1 means that if you want to run the ‘guix-daemon’ binary from the
> binary tarball, you’ll have to somehow satisfy its requirements.  You
> might be able to unpack the tarball in an arbitrary directory, and then
> define ‘LD_LIBRARY_PATH’ so that ‘guix-daemon’ finds all the .so files
> that it needs and are provided by the binary tarball (libstdc++.so,
> libgcrypt.so, libsqlite3.so, libc.so, etc.)
>
> Once ‘guix-daemon’ is running, you could run ‘guix environment guix’,
> and from there you could build Guix from source.
>
> But that is really hacky, and perhaps not particularly easier than
> building a newer GCC on your machine.
>
> Alternately, you could build a suitable binary tarball on another
> machine: just install and configure Guix with the options that you need,
> and run ‘make guix-binary.x86_64-linux.tar.xz’.
>
> WDYT?
>
> Ludo’.
>

[-- Attachment #2: Type: text/html, Size: 5778 bytes --]

Re: Daemon update

[Ludovic Courtès]

Alexander Vorobiev <alexander.vorobiev@gmail.com> skribis:

> Ok, I have just tried to build the binary tarball on a VM where I
> reproduced all the paths I want (basically, instead of /gnu I want
> /shared/shape_tier3/common/local/guix) and which has c++11 compliant gcc --
> that also failed.

What failed exactly?

Note that ‘make guix-binary.x86_64-linux.tar.xz’ produces a tarball that
uses the exact same store directory as the host Guix.  So if you want to
produce a binary tarball with a custom store directory, you first need
to build Guix --with-store-dir=/custom/store.

If, in addition, you want a custom $localstatedir, then at least the
‘guix’ recipe in (gnu packages package-management) must be adjusted, as
well as the “/var” in ‘self-contained-tarball’ in (gnu system install).

> I pulled today's git and ran guix-daemon --no-substitutes. The error
> seems to be "unable to fork: Operation not permitted". I am running
> guix-daemon as myself

When running guix-daemon unprivileged, you need to pass --disable-chroot
(EACCES is returned by clone(2) here since we ask for separate name
spaces, which only root or someone with CAP_SYS_ADMIN can do.)

It’s better to use run it as root though (info "(guix) Invoking
guix-daemon").

Thanks,
Ludo’.

Re: Daemon update

[Alexander Vorobiev]

[-- Attachment #1: Type: text/plain, Size: 3964 bytes --]

Hi,
I modified the files (to use my paths for the cache and store) and ran
guix-daemon as root. Now it got much, much further! But still failed, it
seems while building perl. Here is the end of the log file:

In unknown file:
   ?: 5 [load-compiled/vm
"/home/alex/.cache/guile/ccache/2.0-LE-8-2.0/home/alex/src/guix/build-aux/make-binary-tarball.scm.go"]
In build-aux/make-binary-tarball.scm:
  38: 4 [#<procedure 21c0840 ()>]
In guix/store.scm:
 986: 3 [run-with-store # ...]
In build-aux/make-binary-tarball.scm:
  41: 2 [#<procedure 3961960 at build-aux/make-binary-tarball.scm:40:7
(state)> #]
In guix/store.scm:
 933: 1 [#<procedure 4ed3f30 at guix/store.scm:932:28 (store)> #]
 624: 0 [build-things # #]

guix/store.scm:624:0: In procedure build-things:
guix/store.scm:624:0: Throw to key `srfi-34' with args `(#<condition
&nix-protocol-error [message: "|   |   |   bind mounting `/dev/full' to
`/shared/shape_tier3/commo/local/guix/store/828rlmpkg4bi5i28lif2a1af7g4s62yq-perl-5.16.1.drv.chroot/dev/full'"
status: 1] 4b4cae0>)'.
Makefile:4331: recipe for target 'guix-binary.x86_64-linux.tar.xz' failed
make: [guix-binary.x86_64-linux.tar.xz] Error 1 (ignored)

/dev/full seems to be where is it supposed to be.

I also tried to use --disable-chroot but it also failed while building
perl. The first error message seems to be during perl's configure run:

Checking your choice of C compiler and flags for coherency...
I've tried to compile and run the following simple program:

#include <stdio.h>
int main() { printf("Ok\n"); return(0); }

I used the command:

        gcc -o try -O2 -fno-strict-aliasing -pipe -fstack-protector
-I/shared/shape_tier3/common/local/guix/store/idc1i3zmd0i04h3n9ii9pz1k1wi7z16z-glibc-bootstrap-0/include
-fstack-protector
-L/shared/shape_tier3/common/local/guix/store/idc1i3zmd0i04h3n9ii9pz1k1wi7z16z-glibc-bootstrap-0/lib
try.c -lnsl -lgdbm -ldb -ldl -lm -lcrypt -lutil -lc -lgdbm_compat
         ./try

and I got the following output:

/shared/shape_tier3/common/local/guix/store/714jwgzxf49qw2ysv8p9zz80b1q8bv5n-binutils-bootstrap-0/bin/ld:
cannot find -lgdbm
/shared/shape_tier3/common/local/guix/store/714jwgzxf49qw2ysv8p9zz80b1q8bv5n-binutils-bootstrap-0/bin/ld:
cannot find -ldb
collect2: error: ld returned 1 exit status
I can't compile the test program.
(The supplied flags or libraries might be incorrect.)

You have a BIG problem.  Shall I abort Configure [y]
Ok.  Stopping Configure.


Thanks,
Alex


On Wed, May 27, 2015 at 10:18 AM, Ludovic Courtès <ludo@gnu.org> wrote:

> Alexander Vorobiev <alexander.vorobiev@gmail.com> skribis:
>
> > Ok, I have just tried to build the binary tarball on a VM where I
> > reproduced all the paths I want (basically, instead of /gnu I want
> > /shared/shape_tier3/common/local/guix) and which has c++11 compliant gcc
> --
> > that also failed.
>
> What failed exactly?
>
> Note that ‘make guix-binary.x86_64-linux.tar.xz’ produces a tarball that
> uses the exact same store directory as the host Guix.  So if you want to
> produce a binary tarball with a custom store directory, you first need
> to build Guix --with-store-dir=/custom/store.
>
> If, in addition, you want a custom $localstatedir, then at least the
> ‘guix’ recipe in (gnu packages package-management) must be adjusted, as
> well as the “/var” in ‘self-contained-tarball’ in (gnu system install).
>
> > I pulled today's git and ran guix-daemon --no-substitutes. The error
> > seems to be "unable to fork: Operation not permitted". I am running
> > guix-daemon as myself
>
> When running guix-daemon unprivileged, you need to pass --disable-chroot
> (EACCES is returned by clone(2) here since we ask for separate name
> spaces, which only root or someone with CAP_SYS_ADMIN can do.)
>
> It’s better to use run it as root though (info "(guix) Invoking
> guix-daemon").
>
> Thanks,
> Ludo’.
>

[-- Attachment #2: Type: text/html, Size: 5087 bytes --]

Re: Daemon update

[Ludovic Courtès]

Alexander Vorobiev <alexander.vorobiev@gmail.com> skribis:

> I modified the files (to use my paths for the cache and store) and ran
> guix-daemon as root. Now it got much, much further! But still failed, it
> seems while building perl. Here is the end of the log file:
>
> In unknown file:
>    ?: 5 [load-compiled/vm
> "/home/alex/.cache/guile/ccache/2.0-LE-8-2.0/home/alex/src/guix/build-aux/make-binary-tarball.scm.go"]
> In build-aux/make-binary-tarball.scm:
>   38: 4 [#<procedure 21c0840 ()>]
> In guix/store.scm:
>  986: 3 [run-with-store # ...]
> In build-aux/make-binary-tarball.scm:
>   41: 2 [#<procedure 3961960 at build-aux/make-binary-tarball.scm:40:7
> (state)> #]
> In guix/store.scm:
>  933: 1 [#<procedure 4ed3f30 at guix/store.scm:932:28 (store)> #]
>  624: 0 [build-things # #]
>
> guix/store.scm:624:0: In procedure build-things:
> guix/store.scm:624:0: Throw to key `srfi-34' with args `(#<condition
> &nix-protocol-error [message: "|   |   |   bind mounting `/dev/full' to
> `/shared/shape_tier3/commo/local/guix/store/828rlmpkg4bi5i28lif2a1af7g4s62yq-perl-5.16.1.drv.chroot/dev/full'"
> status: 1] 4b4cae0>)'.
> Makefile:4331: recipe for target 'guix-binary.x86_64-linux.tar.xz' failed

You seem to be running the daemon with --debug no?  What surprises me is
that the above is not an error message but a debugging message.  The
actual error seems to be elsewhere.

> I also tried to use --disable-chroot but it also failed while building
> perl. The first error message seems to be during perl's configure run:

If you can run the daemon as root, then do run it as root, but without
--disable-chroot; as the manual states, --disable-chroot is a last
resort, and it’s only needed if you cannot run the daemon as root.

In the case of Perl, I wouldn’t be surprised if its Configure script
looks at stuff from the host distro in /usr/lib and so on, which can
lead to a failure down the path.

Ludo’.

Re: Daemon update

[Alexander Vorobiev]

[-- Attachment #1: Type: text/plain, Size: 3763 bytes --]

I have some good progress finally. I started from scratch and pulled the
latest from git. I am now running guix-daemon as root with only one option
--no-substitutes. The make guix-binary* ran for hours and built a lot of
stuff (bash, gcc, perl, etc) but stumbled upon openldap which doesn't seem
to be available at the url it wants to use:

building path(s)
`/shared/shape_tier3/common/local/guix/store/50l6s098sy4ig2irjm1g4c8y261nxzfc-openldap-2.4.40.tgz'
starting download of
`/shared/shape_tier3/common/local/guix/store/50l6s098sy4ig2irjm1g4c8y261nxzfc-openldap-2.4.40.tgz'
from `
ftp://sunsite.cnlab-switch.ch/mirror/OpenLDAP/openldap-release/openldap-2.4.40.tgz'.
..
ERROR: In procedure getaddrinfo: Name or service not known
failed to download
"/shared/shape_tier3/common/local/guix/store/50l6s098sy4ig2irjm1g4c8y261nxzfc-openldap-2.4.40.tgz"
from "
ftp://sunsite.cnlab-switch.ch/mirror/OpenLDAP/openldap-release/openldap-2.4.40.tgz
"

I tried modifying gnu/packages/openldap.scm to use correct url (
ftp://mirror.switch.ch/mirror/OpenLDAP/openldap-release/...) but that
didn't change anything since (I am guessing) the .drv files in store
weren't rebuilt and still point to the old url. How do I fix that without
rebuilding everything that it created so far?

As a side note, would it be possible to modify the files you suggested (gnu
packages package-management and gnu system instal) for the custom local
state dir in such a way (through configure? scheme variable?) that it
wouldn't be hard-coded? I am guessing any update to guix would cause my
changes to be lost?

Thanks,
Alex

On Wed, May 27, 2015 at 3:51 PM, Ludovic Courtès <ludo@gnu.org> wrote:

> Alexander Vorobiev <alexander.vorobiev@gmail.com> skribis:
>
> > I modified the files (to use my paths for the cache and store) and ran
> > guix-daemon as root. Now it got much, much further! But still failed, it
> > seems while building perl. Here is the end of the log file:
> >
> > In unknown file:
> >    ?: 5 [load-compiled/vm
> >
> "/home/alex/.cache/guile/ccache/2.0-LE-8-2.0/home/alex/src/guix/build-aux/make-binary-tarball.scm.go"]
> > In build-aux/make-binary-tarball.scm:
> >   38: 4 [#<procedure 21c0840 ()>]
> > In guix/store.scm:
> >  986: 3 [run-with-store # ...]
> > In build-aux/make-binary-tarball.scm:
> >   41: 2 [#<procedure 3961960 at build-aux/make-binary-tarball.scm:40:7
> > (state)> #]
> > In guix/store.scm:
> >  933: 1 [#<procedure 4ed3f30 at guix/store.scm:932:28 (store)> #]
> >  624: 0 [build-things # #]
> >
> > guix/store.scm:624:0: In procedure build-things:
> > guix/store.scm:624:0: Throw to key `srfi-34' with args `(#<condition
> > &nix-protocol-error [message: "|   |   |   bind mounting `/dev/full' to
> >
> `/shared/shape_tier3/commo/local/guix/store/828rlmpkg4bi5i28lif2a1af7g4s62yq-perl-5.16.1.drv.chroot/dev/full'"
> > status: 1] 4b4cae0>)'.
> > Makefile:4331: recipe for target 'guix-binary.x86_64-linux.tar.xz' failed
>
> You seem to be running the daemon with --debug no?  What surprises me is
> that the above is not an error message but a debugging message.  The
> actual error seems to be elsewhere.
>
> > I also tried to use --disable-chroot but it also failed while building
> > perl. The first error message seems to be during perl's configure run:
>
> If you can run the daemon as root, then do run it as root, but without
> --disable-chroot; as the manual states, --disable-chroot is a last
> resort, and it’s only needed if you cannot run the daemon as root.
>
> In the case of Perl, I wouldn’t be surprised if its Configure script
> looks at stuff from the host distro in /usr/lib and so on, which can
> lead to a failure down the path.
>
> Ludo’.
>

[-- Attachment #2: Type: text/html, Size: 4830 bytes --]

Re: Daemon update

[Ludovic Courtès]

Alexander Vorobiev <alexander.vorobiev@gmail.com> skribis:

> I have some good progress finally. I started from scratch and pulled the
> latest from git. I am now running guix-daemon as root with only one option
> --no-substitutes. The make guix-binary* ran for hours and built a lot of
> stuff (bash, gcc, perl, etc) but stumbled upon openldap which doesn't seem
> to be available at the url it wants to use:

Indeed, I’ve updated the OpenLDAP URLs.

> I tried modifying gnu/packages/openldap.scm to use correct url (
> ftp://mirror.switch.ch/mirror/OpenLDAP/openldap-release/...) but that
> didn't change anything since (I am guessing) the .drv files in store
> weren't rebuilt and still point to the old url. How do I fix that without
> rebuilding everything that it created so far?

Changing the file does have an effect, but perhaps you modified the
wrong file?  At any rate, if you git pull now and try again it should
work (and it won’t rebuild everything.)

> As a side note, would it be possible to modify the files you suggested (gnu
> packages package-management and gnu system instal) for the custom local
> state dir in such a way (through configure? scheme variable?) that it
> wouldn't be hard-coded? I am guessing any update to guix would cause my
> changes to be lost?

Since you’re working from a git checkout (right?), you can always
arrange to have your changes applied.

Thanks,
Ludo’.

Re: Daemon update

[Alexander Vorobiev]

[-- Attachment #1: Type: text/plain, Size: 1856 bytes --]

Pulled, restarted. The next stop is tcsh. The tarball at ftp.astron.com s
gone, replaced with more recent version...
One observation: apparently coreutils refuses to be built as root so I had
to create a build user/group to give to guix-daemon.

Thanks,
Alex

On Fri, May 29, 2015 at 3:57 PM, Ludovic Courtès <ludo@gnu.org> wrote:

> Alexander Vorobiev <alexander.vorobiev@gmail.com> skribis:
>
> > I have some good progress finally. I started from scratch and pulled the
> > latest from git. I am now running guix-daemon as root with only one
> option
> > --no-substitutes. The make guix-binary* ran for hours and built a lot of
> > stuff (bash, gcc, perl, etc) but stumbled upon openldap which doesn't
> seem
> > to be available at the url it wants to use:
>
> Indeed, I’ve updated the OpenLDAP URLs.
>
> > I tried modifying gnu/packages/openldap.scm to use correct url (
> > ftp://mirror.switch.ch/mirror/OpenLDAP/openldap-release/...) but that
> > didn't change anything since (I am guessing) the .drv files in store
> > weren't rebuilt and still point to the old url. How do I fix that without
> > rebuilding everything that it created so far?
>
> Changing the file does have an effect, but perhaps you modified the
> wrong file?  At any rate, if you git pull now and try again it should
> work (and it won’t rebuild everything.)
>
> > As a side note, would it be possible to modify the files you suggested
> (gnu
> > packages package-management and gnu system instal) for the custom local
> > state dir in such a way (through configure? scheme variable?) that it
> > wouldn't be hard-coded? I am guessing any update to guix would cause my
> > changes to be lost?
>
> Since you’re working from a git checkout (right?), you can always
> arrange to have your changes applied.
>
> Thanks,
> Ludo’.
>

[-- Attachment #2: Type: text/html, Size: 2547 bytes --]

Re: Daemon update

[Ludovic Courtès]

Alexander Vorobiev <alexander.vorobiev@gmail.com> skribis:

> Pulled, restarted. The next stop is tcsh. The tarball at ftp.astron.com s
> gone, replaced with more recent version...

Fixed, thanks.

> One observation: apparently coreutils refuses to be built as root so I had
> to create a build user/group to give to guix-daemon.

Right, and both the manual and guix-daemon warn you against running
guix-daemon as root without --build-users-group.  :-)

Ludo’.

Re: Daemon update

[Alexander Vorobiev]

[-- Attachment #1: Type: text/plain, Size: 1641 bytes --]

That fixed tcsh, thanks. Here is the next stop - glib (libgio), seems to be
failing its unit tests:

....
PASS: defaultvalue 80 /Default Values/GZlibCompressor
PASS: defaultvalue 81 /Default Values/GZlibDecompressor
tap-driver.sh: internal error getting exit status
tap-driver.sh: fatal: I/O or internal error
Makefile:3751: recipe for target 'defaultvalue.log' failed
make[8]: *** [defaultvalue.log] Error 1
make[8]: Leaving directory
'/tmp/nix-build-glib-2.44.0.drv-0/glib-2.44.0/gio/tests'
Makefile:3653: recipe for target 'check-TESTS' failed
make[7]: *** [check-TESTS] Error 2
make[7]: Leaving directory
'/tmp/nix-build-glib-2.44.0.drv-0/glib-2.44.0/gio/tests'
Makefile:4353: recipe for target 'check-am' failed
make[6]: *** [check-am] Error 2
make[6]: Leaving directory
'/tmp/nix-build-glib-2.44.0.drv-0/glib-2.44.0/gio/tests'
Makefile:3440: recipe for target 'check-recursive' failed
make[5]: *** [check-recursive] Error 1
make[5]: Leaving directory
'/tmp/nix-build-glib-2.44.0.drv-0/glib-2.44.0/gio/tests'

Thanks,
Alex

On Sun, May 31, 2015 at 2:14 PM, Ludovic Courtès <ludo@gnu.org> wrote:

> Alexander Vorobiev <alexander.vorobiev@gmail.com> skribis:
>
> > Pulled, restarted. The next stop is tcsh. The tarball at ftp.astron.com
> s
> > gone, replaced with more recent version...
>
> Fixed, thanks.
>
> > One observation: apparently coreutils refuses to be built as root so I
> had
> > to create a build user/group to give to guix-daemon.
>
> Right, and both the manual and guix-daemon warn you against running
> guix-daemon as root without --build-users-group.  :-)
>
> Ludo’.
>

[-- Attachment #2: Type: text/html, Size: 2380 bytes --]

Re: Daemon update

[Ludovic Courtès]

Alexander Vorobiev <alexander.vorobiev@gmail.com> skribis:

> That fixed tcsh, thanks. Here is the next stop - glib (libgio), seems to be
> failing its unit tests:
>
> ....
> PASS: defaultvalue 80 /Default Values/GZlibCompressor
> PASS: defaultvalue 81 /Default Values/GZlibDecompressor
> tap-driver.sh: internal error getting exit status
> tap-driver.sh: fatal: I/O or internal error
> Makefile:3751: recipe for target 'defaultvalue.log' failed
> make[8]: *** [defaultvalue.log] Error 1
> make[8]: Leaving directory
> '/tmp/nix-build-glib-2.44.0.drv-0/glib-2.44.0/gio/tests'

Is that deterministic?  I.e., does it happen if you run the build again?
We haven’t experienced it on hydra.gnu.org.

Thanks,
Ludo’.

Re: Daemon update

[Alexander Vorobiev]

[-- Attachment #1: Type: text/plain, Size: 1057 bytes --]

Yes it is, happens every time I run it (make guix-binary....). Could it be
custom store and/or local cache? Or wrong version of something in the
system I am using (latest Arch linux)?

Thanks,
Alex

On Mon, Jun 1, 2015 at 2:58 PM, Ludovic Courtès <ludo@gnu.org> wrote:

> Alexander Vorobiev <alexander.vorobiev@gmail.com> skribis:
>
> > That fixed tcsh, thanks. Here is the next stop - glib (libgio), seems to
> be
> > failing its unit tests:
> >
> > ....
> > PASS: defaultvalue 80 /Default Values/GZlibCompressor
> > PASS: defaultvalue 81 /Default Values/GZlibDecompressor
> > tap-driver.sh: internal error getting exit status
> > tap-driver.sh: fatal: I/O or internal error
> > Makefile:3751: recipe for target 'defaultvalue.log' failed
> > make[8]: *** [defaultvalue.log] Error 1
> > make[8]: Leaving directory
> > '/tmp/nix-build-glib-2.44.0.drv-0/glib-2.44.0/gio/tests'
>
> Is that deterministic?  I.e., does it happen if you run the build again?
> We haven’t experienced it on hydra.gnu.org.
>
> Thanks,
> Ludo’.
>

[-- Attachment #2: Type: text/html, Size: 1588 bytes --]

Re: Daemon update

[Alexander Vorobiev]

[-- Attachment #1: Type: text/plain, Size: 1638 bytes --]

At first I saw the exact same error in this list:

http://lists.gnu.org/archive/html/bug-guix/2014-12/msg00002.html

I then pulled again, rebooted the VM just in case, reinstalled guix, and
now I am seeing an error similar to the second error from that email:

ERROR: gapplication - missing test plan
ERROR: gapplication - exited with status 139 (terminated by signal 11?)

That email had status 136 and signal 6.

Thanks,
Alex

On Mon, Jun 1, 2015 at 3:23 PM, Alexander Vorobiev <
alexander.vorobiev@gmail.com> wrote:

> Yes it is, happens every time I run it (make guix-binary....). Could it be
> custom store and/or local cache? Or wrong version of something in the
> system I am using (latest Arch linux)?
>
> Thanks,
> Alex
>
> On Mon, Jun 1, 2015 at 2:58 PM, Ludovic Courtès <ludo@gnu.org> wrote:
>
>> Alexander Vorobiev <alexander.vorobiev@gmail.com> skribis:
>>
>> > That fixed tcsh, thanks. Here is the next stop - glib (libgio), seems
>> to be
>> > failing its unit tests:
>> >
>> > ....
>> > PASS: defaultvalue 80 /Default Values/GZlibCompressor
>> > PASS: defaultvalue 81 /Default Values/GZlibDecompressor
>> > tap-driver.sh: internal error getting exit status
>> > tap-driver.sh: fatal: I/O or internal error
>> > Makefile:3751: recipe for target 'defaultvalue.log' failed
>> > make[8]: *** [defaultvalue.log] Error 1
>> > make[8]: Leaving directory
>> > '/tmp/nix-build-glib-2.44.0.drv-0/glib-2.44.0/gio/tests'
>>
>> Is that deterministic?  I.e., does it happen if you run the build again?
>> We haven’t experienced it on hydra.gnu.org.
>>
>> Thanks,
>> Ludo’.
>>
>
>

[-- Attachment #2: Type: text/html, Size: 2699 bytes --]

Re: Daemon update

[Ludovic Courtès]

Alexander Vorobiev <alexander.vorobiev@gmail.com> skribis:

> At first I saw the exact same error in this list:
>
> http://lists.gnu.org/archive/html/bug-guix/2014-12/msg00002.html

Oh.

> I then pulled again, rebooted the VM just in case, reinstalled guix, and

It’s not necessary to reinstall Guix, it shouldn’t make any difference.

> now I am seeing an error similar to the second error from that email:
>
> ERROR: gapplication - missing test plan
> ERROR: gapplication - exited with status 139 (terminated by signal 11?)
>
> That email had status 136 and signal 6.

Could you paste more context from the build log, so we can see which
test is failing?  (Use ‘guix build --log-file glib’ to access the log.)

Ludo’.

Re: Daemon update

[Alexander Vorobiev]

[-- Attachment #1: Type: text/plain, Size: 1756 bytes --]

Yes, sorry for too short a snippet:

PASS: gapplication 1 /gapplication/no-dbus
PASS: gapplication 2 /gapplication/no-appid
PASS: gapplication 3 /gapplication/properties
PASS: gapplication 4 /gapplication/app-id
ERROR: gapplication - missing test plan
ERROR: gapplication - exited with status 139 (terminated by signal 11?)
SKIP: gdbus-auth
PASS: gdbus-bz627724 1 /gdbus/bz627724
PASS: gdbus-close-pending 1 /gdbus/close-pending
PASS: gdbus-close-pending 2 /gdbus/unref-pending

and finally:

============================================================================
Testsuite summary for glib 2.44.0
============================================================================
# TOTAL: 673
# PASS:  632
# SKIP:  39
# XFAIL: 0
# FAIL:  0
# XPASS: 0
# ERROR: 2


by the way, I had to

touch doc/guix-environment.1

which was missing.

Thanks,
Alex

On Wed, Jun 3, 2015 at 3:24 AM, Ludovic Courtès <ludo@gnu.org> wrote:

> Alexander Vorobiev <alexander.vorobiev@gmail.com> skribis:
>
> > At first I saw the exact same error in this list:
> >
> > http://lists.gnu.org/archive/html/bug-guix/2014-12/msg00002.html
>
> Oh.
>
> > I then pulled again, rebooted the VM just in case, reinstalled guix, and
>
> It’s not necessary to reinstall Guix, it shouldn’t make any difference.
>
> > now I am seeing an error similar to the second error from that email:
> >
> > ERROR: gapplication - missing test plan
> > ERROR: gapplication - exited with status 139 (terminated by signal 11?)
> >
> > That email had status 136 and signal 6.
>
> Could you paste more context from the build log, so we can see which
> test is failing?  (Use ‘guix build --log-file glib’ to access the log.)
>
> Ludo’.
>

[-- Attachment #2: Type: text/html, Size: 2717 bytes --]

Daemon update

[Ludovic Courtès]

Commit 1303a4a fixes a use-after-free in the daemon that manifested when
compiling it with GCC 5.1 (string destructors were called before
‘execve’ calls that used the underlying ‘char *’ pointers.)

The fix is a backport of an upstream commit.  I plan to update our copy
of the Nix daemon code eventually, but it seemed wiser to do that after
the release, hence this backport.

Please report any issues.

Thanks,
Ludo’.