bug#22209: name resolution failures in installer image

bug#22209: name resolution failures in installer image

[Florian Paul Schmidt]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256


Hi,

this occurs over and over to me and I saw other people's reports on
IRC, too:

After booting the installer image, doing the ifconfig/dhclient dance,
preparing the target fs, mounting it and starting the guix system init
invocation at some point in time, though not predictably so, but
usually after a couple of minutes (about 10 or so) name resolution
will start to fail causing guix system init to fail.

killall dhclient; dhclient <your_interface_name_here>

and a subsequent invocation to guix system init will make the
installation continue and usually succeed.

On the first few occasions it seemed to me that my local network could
be at fault, but it's _almost_ reliably reproducable in qemu,
virtualbox and even on bare metal and is not limited to myself (check
the channel logs with some good search queries).

So it might be worth looking into.

The installed system does not have this problem after a reboot.

Regards,
Flo

- -- 
https://fps.io
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQEcBAEBCAAGBQJWdQ+YAAoJEA5f4Coltk8ZgdoIAJ9rDLDhTztcfs4fHWBF+mcf
U6SYAX36nc7R+TiT7FCWQVxcp38bG5LrTInmP4C/ojyPHEYZcMQLQg2CoT4/q6Ys
kjMUKvBgFqMsjeZzvo9oXP1VDWygC9YeCg55BGPzpEW5x3oQHcVH0zw8acNrlZ6A
I9sVoQnaEDKhsNECBwPDMNujqjik6TilvddEl0AS7HYCbyNgPy+dR97G5z+gRt8w
Wrio0ua5QeyAW5kFCLX7LdDa44OBPoiAlLyWa9dXgFUOORgh88z/dtCPbV0vRNJc
FKpVmAzcovximrPGgDXs9DzERaSkROM05GGRb/uYFfs5GdoGja5YwjXcoWaHUzo=
=9+XX
-----END PGP SIGNATURE-----

bug#22209: name resolution failures in installer image

[Leo Famulari]

On Sat, Dec 19, 2015 at 09:04:41AM +0100, Florian Paul Schmidt wrote:
> After booting the installer image, doing the ifconfig/dhclient dance,
> preparing the target fs, mounting it and starting the guix system init
> invocation at some point in time, though not predictably so, but
> usually after a couple of minutes (about 10 or so) name resolution
> will start to fail causing guix system init to fail.

A little more information:

I was installing on i686 bare metal from the current USB installation
image.

I installed vim in the installation environment, and then when I
tried to run `guix system init...` I couldn't resolve hydra.gnu.org,
google.com, and a few other sites. I could resolve the domain
famulari.name, but not another domain that points to the same machine.

I installed dig and could successfully query the nameserver described in
/etc/resolv.conf for hydra.gnu.org.

I never have name resolution problems on this network with my other
machines.

I decided to edit /etc/hosts so I could continue with the installation.
If anyone has any ideas about how to debug this, I'm happy to help.

bug#22209: name resolution failures in installer image

[Ricardo Wurmus]

Leo Famulari <leo@famulari.name> writes:

> On Sat, Dec 19, 2015 at 09:04:41AM +0100, Florian Paul Schmidt wrote:
>> After booting the installer image, doing the ifconfig/dhclient dance,
>> preparing the target fs, mounting it and starting the guix system init
>> invocation at some point in time, though not predictably so, but
>> usually after a couple of minutes (about 10 or so) name resolution
>> will start to fail causing guix system init to fail.
>
> A little more information:
>
> I was installing on i686 bare metal from the current USB installation
> image.
>
> I installed vim in the installation environment, and then when I
> tried to run `guix system init...` I couldn't resolve hydra.gnu.org,
> google.com, and a few other sites. I could resolve the domain
> famulari.name, but not another domain that points to the same machine.
>
> I installed dig and could successfully query the nameserver described in
> /etc/resolv.conf for hydra.gnu.org.

Could this be related to cached resolution failures in nscd?  Have you
tried restarting the nscd service?

~~ Ricardo

bug#22209: name resolution failures in installer image

[Ludovic Courtès]

Ricardo Wurmus <rekado@elephly.net> skribis:

> Leo Famulari <leo@famulari.name> writes:
>
>> On Sat, Dec 19, 2015 at 09:04:41AM +0100, Florian Paul Schmidt wrote:
>>> After booting the installer image, doing the ifconfig/dhclient dance,
>>> preparing the target fs, mounting it and starting the guix system init
>>> invocation at some point in time, though not predictably so, but
>>> usually after a couple of minutes (about 10 or so) name resolution
>>> will start to fail causing guix system init to fail.
>>
>> A little more information:
>>
>> I was installing on i686 bare metal from the current USB installation
>> image.
>>
>> I installed vim in the installation environment, and then when I
>> tried to run `guix system init...` I couldn't resolve hydra.gnu.org,
>> google.com, and a few other sites. I could resolve the domain
>> famulari.name, but not another domain that points to the same machine.
>>
>> I installed dig and could successfully query the nameserver described in
>> /etc/resolv.conf for hydra.gnu.org.
>
> Could this be related to cached resolution failures in nscd?  Have you
> tried restarting the nscd service?

That sounds likely.

Nscd caches negative lookups, but normally only for 20 seconds (see
%nscd-minimal-caches in (gnu system install)).  So if you look up
hydra.gnu.org before the network is set up, it’ll remember for 20s that
this is a lookup failure, even if you’ve set up the network in the
meantime.

However, nscd is reportedly behaving strangely:
<http://bugs.gnu.org/20816>.

Do you initially get successful lookups for hydra.gnu.org, and
eventually lookup failures?

Ludo’.

bug#22209: name resolution failures in installer image

[Florian Paul Schmidt]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 19.12.2015 19:09, Ludovic Courtès wrote:

> However, nscd is reportedly behaving strangely: 
> <http://bugs.gnu.org/20816>.
> 
> Do you initially get successful lookups for hydra.gnu.org, and 
> eventually lookup failures?

Yes, that's at least in my case precisely what happens. It works for
the first few packages during system init, and then at some point in
time fails.

I'll try to reproduce and then restart the nscd to see if it helps
(when I find time to do it)..

Flo


- -- 
https://fps.io
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQEcBAEBCAAGBQJWdn8QAAoJEA5f4Coltk8ZrSIH/3YKmBO2ZDCqxxZGWYJDRgQc
Q74OFkSrw56ov29uSU+Xs63njem19PLxyECLoICex2tOMOuELpowKt0ee8hS/7PT
eYyRlMiUfIk4WC2qpFGQtfURDOmsAiWCGP2LzBO1RqhgU5CtgnChH5Z7ZVmlNloy
EqVKZ0tpquagf1k6FkMsk65fY6gPa2D+2Ova1SkwtEiIRwMDMNkguadHXu9LShKu
9fSOnP7BXwAZfPC6hPyrB8tjUWtr8PUlzYNsLsW6PTqtRNry5QSSa2I4wsx9Gjse
HQ3NeRjp3RPvOx/LwP5+nGaTsUN8R8aPSX3eMumpBfzrImLqANExhcOvkFtkL6U=
=R54m
-----END PGP SIGNATURE-----

bug#22209: name resolution failures in installer image

[Ludovic Courtès]

ludo@gnu.org (Ludovic Courtès) skribis:

> Ricardo Wurmus <rekado@elephly.net> skribis:

[...]

>> Could this be related to cached resolution failures in nscd?  Have you
>> tried restarting the nscd service?
>
> That sounds likely.
>
> Nscd caches negative lookups, but normally only for 20 seconds (see
> %nscd-minimal-caches in (gnu system install)).  So if you look up
> hydra.gnu.org before the network is set up, it’ll remember for 20s that
> this is a lookup failure, even if you’ve set up the network in the
> meantime.

Commit c96ba2c disables caching of lookup failures altogether.

Could you try again to reproduce it?  (I think Florian wrote in the
initial bug report that this was reproducible in QEMU, so using
‘guix system vm gnu/system/install.scm’ might suffice.)

Alternately we could remove nscd altogether, but I think it’s still
somewhat useful to cache successful lookups.

TIA,
Ludo’.

bug#22209: name resolution failures in installer image

[Ludovic Courtès]

ludo@gnu.org (Ludovic Courtès) skribis:

> ludo@gnu.org (Ludovic Courtès) skribis:
>
>> Ricardo Wurmus <rekado@elephly.net> skribis:
>
> [...]
>
>>> Could this be related to cached resolution failures in nscd?  Have you
>>> tried restarting the nscd service?
>>
>> That sounds likely.
>>
>> Nscd caches negative lookups, but normally only for 20 seconds (see
>> %nscd-minimal-caches in (gnu system install)).  So if you look up
>> hydra.gnu.org before the network is set up, it’ll remember for 20s that
>> this is a lookup failure, even if you’ve set up the network in the
>> meantime.
>
> Commit c96ba2c disables caching of lookup failures altogether.

0.10.0 uses this fix and we haven’t heard from name resolution failures
in the installer image, so I’m closing it.

Ludo’.