Core-updates: Downloads do not work

Core-updates: Downloads do not work

[Andreas Enge]

[-- Attachment #1: Type: text/plain, Size: 1762 bytes --]

Hello,

in the core-updates branch, tarballs are not downloaded automatically any 
more. Here is the result of "guix-build hello" (which tries to rebuild 
everything, as expected):

...
starting download of `/nix/store/c09pc1yvdai4ylb5i7x242aj66dy3f07-
glibc-2.17.tar.xz' from 
`http://ftpmirror.gnu.org/glibc/glibc-2.17.tar.xz'...
ERROR: In procedure getaddrinfo: Name or service not known
starting download of `/nix/store/c09pc1yvdai4ylb5i7x242aj66dy3f07-
glibc-2.17.tar.xz' from `ftp://ftp.cs.tu-
berlin.de/pub/gnu/glibc/glibc-2.17.tar.xz'...
failed to resolve `ftp.cs.tu-berlin.de': Name or service not known
ERROR: In procedure struct_vtable: Wrong type argument in position 1 
(expecting pair): #f
starting download of `/nix/store/c09pc1yvdai4ylb5i7x242aj66dy3f07-
glibc-2.17.tar.xz' from 
`ftp://ftp.chg.ru/pub/gnu/glibc/glibc-2.17.tar.xz'...
failed to resolve `ftp.chg.ru': Name or service not known
ERROR: In procedure struct_vtable: Wrong type argument in position 1 
(expecting pair): #f
starting download of `/nix/store/c09pc1yvdai4ylb5i7x242aj66dy3f07-
glibc-2.17.tar.xz' from 
`ftp://ftp.funet.fi/pub/mirrors/ftp.gnu.org/gnu/glibc/glibc-2.17.tar.xz'...
failed to resolve `ftp.funet.fi': Name or service not known
ERROR: In procedure struct_vtable: Wrong type argument in position 1 
(expecting pair): #f
starting download of `/nix/store/c09pc1yvdai4ylb5i7x242aj66dy3f07-
glibc-2.17.tar.xz' from 
`http://ftp.gnu.org/pub/gnu/glibc/glibc-2.17.tar.xz'...
ERROR: In procedure getaddrinfo: Name or service not known
failed to download "/nix/store/c09pc1yvdai4ylb5i7x242aj66dy3f07-
glibc-2.17.tar.xz" from "mirror://gnu/glibc/glibc-2.17.tar.xz"
...

On the other hand,
   guix-download http://ftpmirror.gnu.org/glibc/glibc-2.17.tar.xz
succeeds.

Andreas

[-- Attachment #2: Type: text/html, Size: 5982 bytes --]

Re: Core-updates: Downloads do not work

[Ludovic Courtès]

Andreas Enge <andreas@enge.fr> skribis:

> in the core-updates branch, tarballs are not downloaded automatically any 
> more. Here is the result of "guix-build hello" (which tries to rebuild 
> everything, as expected):

I can’t reproduce it:

--8<---------------cut here---------------start------------->8---
$ ./pre-inst-env guix-build -S -e '(@ (distro packages base) glibc-final)'
the following derivations will be built:
    /nix/store/r132zr05rnbwl56xlzaqypsh85cd8cfv-glibc-2.17.tar.xz.drv
    /nix/store/lasdjqypfkcfqkf4ryvycz3b0x239c71-module-import.drv
    /nix/store/n0pdl6b6f8qykbh9mhdgjj56mb24d5k2-module-import-compiled.drv

[...]

starting download of `/nix/store/c09pc1yvdai4ylb5i7x242aj66dy3f07-glibc-2.17.tar.xz' from `http://ftpmirror.gnu.org/glibc/glibc-2.17.tar.xz'...
following redirection to `http://mirror.ibcp.fr/pub/gnu/glibc/glibc-2.17.tar.xz'...
--8<---------------cut here---------------end--------------->8---

Most likely this is because nscd is not running, or not listening to the
right socket:

--8<---------------cut here---------------start------------->8---
$ strace -o ,,s /nix/store/k8qmk5n2zsrndvzqs4bq7x9jyyxf5ndr-guile-bootstrap-2.0/bin/guile -c '(pk (getaddrinfo "ftp.gnu.org" "80" AI_NUMERICSERV))'

;;; ((#(1024 2 1 6 #(2 3497454356 80) #f) #(1024 2 2 17 #(2 3497454356 80) #f) #(1024 2 3 0 #(2 3497454356 80) #f) #(1024 10 1 6 #(10 42541952298791455573290623124440612875 80 0 0) #f) #(1024 10 2 17 #(10 42541952298791455573290623124440612875 80 0 0) #f) #(1024 10 3 0 #(10 42541952298791455573290623124440612875 80 0 0) #f)))

$ grep nscd ,,s
connect(5, {sa_family=AF_FILE, path="/var/run/nscd/socket"}, 110) = 0
connect(5, {sa_family=AF_FILE, path="/var/run/nscd/socket"}, 110) = 0
--8<---------------cut here---------------end--------------->8---

Can you check that?

TIA,
Ludo’.

Re: Core-updates: Downloads do not work

[Andreas Enge]

[-- Attachment #1: Type: text/plain, Size: 1704 bytes --]

Am Donnerstag, 3. Januar 2013 schrieb Ludovic Courtès:
> Most likely this is because nscd is not running, or not listening to the
> right socket:
> 
> --8<---------------cut here---------------start------------->8---
> $ strace -o ,,s
> /nix/store/k8qmk5n2zsrndvzqs4bq7x9jyyxf5ndr-guile-bootstrap-2.0/bin/gui
> le -c '(pk (getaddrinfo "ftp.gnu.org" "80" AI_NUMERICSERV))'
> 
> ;;; ((#(1024 2 1 6 #(2 3497454356 80) #f) #(1024 2 2 17 #(2 3497454356
> 80) #f) #(1024 2 3 0 #(2 3497454356 80) #f) #(1024 10 1 6 #(10
> 42541952298791455573290623124440612875 80 0 0) #f) #(1024 10 2 17 #(10
> 42541952298791455573290623124440612875 80 0 0) #f) #(1024 10 3 0 #(10
> 42541952298791455573290623124440612875 80 0 0) #f)))
> 
> $ grep nscd ,,s
> connect(5, {sa_family=AF_FILE, path="/var/run/nscd/socket"}, 110) = 0
> connect(5, {sa_family=AF_FILE, path="/var/run/nscd/socket"}, 110) = 0
> --8<---------------cut here---------------end--------------->8---
> 
> Can you check that?

Here is what I get:
connect(5, {sa_family=AF_FILE, path="/var/run/nscd/socket"}, 110) = -1 
ENOENT (No such file or directory)
connect(5, {sa_family=AF_FILE, path="/var/run/nscd/socket"}, 110) = -1 
ENOENT (No such file or directory)

Under debian, there is a separate package "nscd" with the following 
description:
Package: nscd
Source: eglibc
Description-en: Embedded GNU C Library: Name Service Cache Daemon
 A daemon which handles passwd, group and host lookups
 for running programs and caches the results for the next
 query. You should install this package only if you use
 slow services like LDAP, NIS or NIS+.

I am running neither LDAP nor NIS*, so it is not installed.

Andreas

[-- Attachment #2: Type: text/html, Size: 7742 bytes --]

Re: Core-updates: Downloads do not work

[Ludovic Courtès]

Andreas Enge <andreas@enge.fr> skribis:

> Here is what I get:
> connect(5, {sa_family=AF_FILE, path="/var/run/nscd/socket"}, 110) = -1 
> ENOENT (No such file or directory)
> connect(5, {sa_family=AF_FILE, path="/var/run/nscd/socket"}, 110) = -1 
> ENOENT (No such file or directory)

OK, this confirms my intuition.

> Under debian, there is a separate package "nscd" with the following 
> description:
> Package: nscd
> Source: eglibc
> Description-en: Embedded GNU C Library: Name Service Cache Daemon
>  A daemon which handles passwd, group and host lookups
>  for running programs and caches the results for the next
>  query. You should install this package only if you use
>  slow services like LDAP, NIS or NIS+.
>
> I am running neither LDAP nor NIS*, so it is not installed.

It works for name lookups in general, including DNS lookups.  It’s
useful to always enable it.

Thanks,
Ludo’.

Re: Core-updates: Downloads do not work

[Andreas Enge]

[-- Attachment #1: Type: text/plain, Size: 465 bytes --]

Am Donnerstag, 3. Januar 2013 schrieb Ludovic Courtès:
> It works for name lookups in general, including DNS lookups.  It’s
> useful to always enable it.

So should this be checked by ./configure?

I am still not sure why this is needed; the "host" command does work and I 
can surf the web, so there seems to be some name lookup service available 
(even though maybe without the caching advertised for nscd). Also, it does 
work in master.

Andreas

[-- Attachment #2: Type: text/html, Size: 2274 bytes --]

Re: Core-updates: Downloads do not work

[Ludovic Courtès]

Andreas Enge <andreas@enge.fr> skribis:

> Am Donnerstag, 3. Januar 2013 schrieb Ludovic Courtès:
>> It works for name lookups in general, including DNS lookups.  It’s
>> useful to always enable it.
>
> So should this be checked by ./configure?

It wouldn’t be enough: nscd could be running at configure time, and not later.

> I am still not sure why this is needed; the "host" command does work and I 
> can surf the web, so there seems to be some name lookup service available 
> (even though maybe without the caching advertised for nscd).

The ‘host’ command includes its own DNS client, IIRC (it does not use
getaddrinfo and similar libc services.)

The C library’s getaddrinfo and other lookup functions can of course be
used without nscd, which is what your applications were using.

What differs here is that the bootstrap Guile is statically linked, and
getaddrinfo and similar functions are not fully functional in statically
linked binaries.

Hmm, that said, we should probably build the bootstrap libc with
--disable-nscd to avoid this problem altogether.  I’ll try that, but
rebuilding everything is gonna take ages again...

> Also, it does work in master.

That’s more surprising.

Can you try:

  strace -o ,,s /nix/store/7ys18z66aih9dwzlwkj2jynk05a3xsds-guile-bootstrap-2.0/bin/guile -c '(pk (getaddrinfo "ftp.gnu.org" "80" AI_NUMERICSERV))'
  grep nscd ,,s

The above path is that of the bootstrap Guile in master.

Thanks,
Ludo’.

Re: Core-updates: Downloads do not work

[Andreas Enge]

[-- Attachment #1: Type: text/plain, Size: 2329 bytes --]

Am Donnerstag, 3. Januar 2013 schrieb Ludovic Courtès:
> Andreas Enge <andreas@enge.fr> skribis:
> > So should this be checked by ./configure?
> It wouldn’t be enough: nscd could be running at configure time, and not
> later.

True, but it would be better than nothing - as long as it is not packaged 
for distributions, the code will most probably run on the machine where it 
has been compiled.

> The C library’s getaddrinfo and other lookup functions can of course be
> used without nscd, which is what your applications were using.
> 
> What differs here is that the bootstrap Guile is statically linked, and
> getaddrinfo and similar functions are not fully functional in statically
> linked binaries.
> 
> Hmm, that said, we should probably build the bootstrap libc with
> --disable-nscd to avoid this problem altogether.  I’ll try that, but
> rebuilding everything is gonna take ages again...

Well, if it solves the problem, it is worth the time!

> Can you try:
>   strace -o ,,s
> /nix/store/7ys18z66aih9dwzlwkj2jynk05a3xsds-guile-bootstrap-2.0/bin/gui
> le -c '(pk (getaddrinfo "ftp.gnu.org" "80" AI_NUMERICSERV))' grep nscd
> ,,s
> The above path is that of the bootstrap Guile in master.

The same problem:
privat@debian:/tmp$ strace -o ,,s 
/nix/store/7ys18z66aih9dwzlwkj2jynk05a3xsds-guile-bootstrap-2.0/bin/guile -
c '(pk (getaddrinfo "ftp.gnu.org" "80" AI_NUMERICSERV))'

;;; ((#(1024 2 1 6 #(2 3497454356 80) #f) #(1024 2 2 17 #(2 3497454356 80) 
#f) #(1024 2 3 0 #(2 3497454356 80) #f) #(1024 10 1 6 #(10 
42541952298791455573290623124440612875 80 0 0) #f) #(1024 10 2 17 #(10 
42541952298791455573290623124440612875 80 0 0) #f) #(1024 10 3 0 #(10 
42541952298791455573290623124440612875 80 0 0) #f)))

privat@debian:/tmp$ grep nscd ,,s
connect(5, {sa_family=AF_FILE, path="/var/run/nscd/socket"}, 110) = -1 
ENOENT (No such file or directory)
connect(5, {sa_family=AF_FILE, path="/var/run/nscd/socket"}, 110) = -1 
ENOENT (No such file or directory)

Admittedly, I recently downloaded new tarballs mainly for my own new 
packages, and since I needed the hash, I did so using guix-download. So 
maybe it does not work any more since guix includes the build code from 
nix? It definitely did work at some point in time for my first "hello".

Andreas

[-- Attachment #2: Type: text/html, Size: 8696 bytes --]

Re: Core-updates: Downloads do not work

[Ludovic Courtès]

Hi!

Andreas Enge <andreas@enge.fr> skribis:

> Am Donnerstag, 3. Januar 2013 schrieb Ludovic Courtès:

[...]

>> The C library’s getaddrinfo and other lookup functions can of course be
>> used without nscd, which is what your applications were using.
>> 
>> What differs here is that the bootstrap Guile is statically linked, and
>> getaddrinfo and similar functions are not fully functional in statically
>> linked binaries.
>> 
>> Hmm, that said, we should probably build the bootstrap libc with
>> --disable-nscd to avoid this problem altogether.  I’ll try that, but
>> rebuilding everything is gonna take ages again...
>
> Well, if it solves the problem, it is worth the time!

I’m working on it, and it seems to solve the problem.

>> Can you try:
>>   strace -o ,,s
>> /nix/store/7ys18z66aih9dwzlwkj2jynk05a3xsds-guile-bootstrap-2.0/bin/gui
>> le -c '(pk (getaddrinfo "ftp.gnu.org" "80" AI_NUMERICSERV))' grep nscd
>> ,,s
>> The above path is that of the bootstrap Guile in master.
>
> The same problem:

Good, I wouldn’t have been able to explain a different behavior.  ;-)

> Admittedly, I recently downloaded new tarballs mainly for my own new 
> packages, and since I needed the hash, I did so using guix-download. So 
> maybe it does not work any more since guix includes the build code from 
> nix?

Not sure what you mean here.  It could be that you had tarballs already
in the store, and thus didn’t notice the problem.

> It definitely did work at some point in time for my first "hello".

Well, dunno.  The nscd is enabled by default in glibc builds, so it
could be that it was there once upon a time...

Thanks,
Ludo’.

Re: Core-updates: Downloads do not work

[Ludovic Courtès]

Hi!

Andreas Enge <andreas@enge.fr> skribis:

> Am Donnerstag, 3. Januar 2013 schrieb Ludovic Courtès:

[...]

>> The C library’s getaddrinfo and other lookup functions can of course be
>> used without nscd, which is what your applications were using.
>> 
>> What differs here is that the bootstrap Guile is statically linked, and
>> getaddrinfo and similar functions are not fully functional in statically
>> linked binaries.
>> 
>> Hmm, that said, we should probably build the bootstrap libc with
>> --disable-nscd to avoid this problem altogether.  I’ll try that, but
>> rebuilding everything is gonna take ages again...
>
> Well, if it solves the problem, it is worth the time!

I’ve rebuilt and uploaded all the bootstrap binaries with a libc that
doesn’t use the nscd and has static NSS modules.  With commit 79580eb of
core-updates, these new binaries are being used.

Let me know how it works!  :-)

Ludo’.

Re: Core-updates: Downloads do not work

[Andreas Enge]

[-- Attachment #1: Type: text/plain, Size: 3320 bytes --]

Am Sonntag, 6. Januar 2013 schrieb Ludovic Courtès:
> I’ve rebuilt and uploaded all the bootstrap binaries with a libc that
> doesn’t use the nscd and has static NSS modules.  With commit 79580eb of
> core-updates, these new binaries are being used.
> 
> Let me know how it works!  :-)

Well, I do not understand what is happening. To check whether the problem 
was solved, I wanted to verify that it was still present before the update, 
and tried to build a new package (ddrescue). What shall I say, it was 
downloaded correctly before updating my git repository!

After updating, things still do not work, which makes sense, since the new 
bootstrap binaries are not yet downloaded:

@ build-started /nix/store/blsxf7m8h6rddhq6ajksjhq27fzfi90v-
gcc-4.7.2.tar.xz.drv /nix/store/9vijlrv06dvli9m2fw9ha3r3mswz0qas-
gcc-4.7.2.tar.xz x86_64-linux /usr/local/guix-
git/var/log/nix/drvs/blsxf7m8h6rddhq6ajksjhq27fzfi90v-
gcc-4.7.2.tar.xz.drv.bz2
@ build-started /nix/store/ks8lzbbnw97n9rxbyf7fxhgsl2alkjc3-static-
binaries.tar.xz.drv /nix/store/n5wq53xj7fyzbf4w8whz0q8ffh6yy8b9-static-
binaries.tar.xz x86_64-linux /usr/local/guix-
git/var/log/nix/drvs/ks8lzbbnw97n9rxbyf7fxhgsl2alkjc3-static-
binaries.tar.xz.drv.bz2
@ build-started /nix/store/zma6787ffbaj9dy8vxjirdnsrs5k8mrx-
binutils-2.22.tar.xz.drv /nix/store/bvhn746hnl96xwm88g3xs3wqv015zbha-
binutils-2.22.tar.xz x86_64-linux /usr/local/guix-
git/var/log/nix/drvs/zma6787ffbaj9dy8vxjirdnsrs5k8mrx-
binutils-2.22.tar.xz.drv.bz2
@ build-started /nix/store/w5d3ha447cb8zhrxh64y8q14izcp1m8q-
glibc-2.17.tar.xz.drv /nix/store/q61dk4k2zcnvxvha8k71j1a46lyr89vn-
glibc-2.17.tar.xz x86_64-linux /usr/local/guix-
git/var/log/nix/drvs/w5d3ha447cb8zhrxh64y8q14izcp1m8q-
glibc-2.17.tar.xz.drv.bz2
starting download of `/nix/store/9vijlrv06dvli9m2fw9ha3r3mswz0qas-
gcc-4.7.2.tar.xz' from 
`http://www.fdn.fr/~lcourtes/software/guix/packages/x86_64-
linux/20130105/gcc-4.7.2.tar.xz'...
ERROR: In procedure getaddrinfo: Name or service not known
failed to download "/nix/store/9vijlrv06dvli9m2fw9ha3r3mswz0qas-
gcc-4.7.2.tar.xz" from 
"http://www.fdn.fr/~lcourtes/software/guix/packages/x86_64-
linux/20130105/gcc-4.7.2.tar.xz"
starting download of `/nix/store/q61dk4k2zcnvxvha8k71j1a46lyr89vn-
glibc-2.17.tar.xz' from 
`http://www.fdn.fr/~lcourtes/software/guix/packages/x86_64-
linux/20130105/glibc-2.17.tar.xz'...
ERROR: In procedure getaddrinfo: Name or service not known
failed to download "/nix/store/q61dk4k2zcnvxvha8k71j1a46lyr89vn-
glibc-2.17.tar.xz" from 
"http://www.fdn.fr/~lcourtes/software/guix/packages/x86_64-
linux/20130105/glibc-2.17.tar.xz"
builder for `/nix/store/blsxf7m8h6rddhq6ajksjhq27fzfi90v-
gcc-4.7.2.tar.xz.drv' failed with exit code 1
@ build-failed /nix/store/blsxf7m8h6rddhq6ajksjhq27fzfi90v-
gcc-4.7.2.tar.xz.drv /nix/store/9vijlrv06dvli9m2fw9ha3r3mswz0qas-
gcc-4.7.2.tar.xz 1 builder for 
`/nix/store/blsxf7m8h6rddhq6ajksjhq27fzfi90v-gcc-4.7.2.tar.xz.drv' failed 
with exit code 1

However, I can execute
   guix-download http://www.fdn.fr/~lcourtes/software/guix/packages/x86_64-
linux/20130105/gcc-4.7.2.tar.xz
(as well as for all other packages), and then start a build.

How come that "guix-download" does work, but the same download fails 
through "guix-build"?

Andreas

[-- Attachment #2: Type: text/html, Size: 8260 bytes --]

Re: Core-updates: Downloads do not work

[Ludovic Courtès]

Hi Andreas!

Andreas Enge <andreas@enge.fr> skribis:

> After updating, things still do not work, which makes sense, since the new 
> bootstrap binaries are not yet downloaded:

Ah, you need to:

  $ rm distro/packages/bootstrap/x86_64-linux/guile-*.tar.xz
  $ make

The latter will download the current guile-2.0.7.tar.xz bootstrap
binary, and check its hash (the hash of that tarball is only checked
upon downloading, by build-aux/download.scm.)

> However, I can execute
>    guix-download http://www.fdn.fr/~lcourtes/software/guix/packages/x86_64-
> linux/20130105/gcc-4.7.2.tar.xz
> (as well as for all other packages), and then start a build.
>
> How come that "guix-download" does work, but the same download fails 
> through "guix-build"?

That’s because guix-download uses your Guile, which is a full-fledged
Guile linked against a feature-complete libc, whereas the old bootstrap
Guile was statically linked against a libc whose getaddrinfo depended on
a running nscd to work.

Does that clarify a bit?  :-)

Thanks,
Ludo’.

Re: Core-updates: Downloads do not work

[Andreas Enge]

[-- Attachment #1: Type: text/plain, Size: 525 bytes --]

Am Montag, 7. Januar 2013 schrieb Ludovic Courtès:
> > bootstrap binaries are not yet downloaded:
> Ah, you need to:
>   $ rm distro/packages/bootstrap/x86_64-linux/guile-*.tar.xz
>   $ make
> ...
> Does that clarify a bit?  :-)

Thanks, it clarifies a lot. To check whether things work, can I simply 
delete the "hello" tarball in the nix store? I once deleted a compiled 
package in the nix store, and that caused an inconsistency with the package 
database in guix-installation-directory/var/... .

Andreas

[-- Attachment #2: Type: text/html, Size: 2617 bytes --]

Re: Core-updates: Downloads do not work

[Ludovic Courtès]

Hi!

Andreas Enge <andreas@enge.fr> skribis:

> Thanks, it clarifies a lot. To check whether things work, can I simply 
> delete the "hello" tarball in the nix store? I once deleted a compiled 
> package in the nix store, and that caused an inconsistency with the package 
> database in guix-installation-directory/var/... .

Anything you delete must be deleted with ‘guix-gc --delete’ (or
‘nix-store --delete’), which also updates the database that’s under
/nix/var/nix/db/.

Thanks,
Ludo’.

Re: Core-updates: Downloads do not work

[Andreas Enge]

[-- Attachment #1: Type: text/plain, Size: 134 bytes --]

Am Sonntag, 6. Januar 2013 schrieb Ludovic Courtès:
> Let me know how it works!  :-)

It does work now. Thanks a lot!

Andreas

[-- Attachment #2: Type: text/html, Size: 1499 bytes --]

Re: Core-updates: Downloads do not work

[Ludovic Courtès]

Andreas Enge <andreas@enge.fr> skribis:

> Am Sonntag, 6. Januar 2013 schrieb Ludovic Courtès:
>> Let me know how it works!  :-)
>
> It does work now. Thanks a lot!

Cool, thank you!

I think we should merge ‘core-updates’ real soon, if there are no
objections, and clear a few rough edges before 0.1.

Ludo’.