Ganeti with Guix tutorial from the Guix blog post

Ganeti with Guix tutorial from the Guix blog post

[david larsson]

Hi!

First of all - thanks for a great post about using Ganeti with Guix!

I need some help with the networking setup part since I am stuck at the 
end stages of the blog post tutorial - I am unable to run "gnt-instance 
console <someVM>". So, I have things setup enough to create these VM's 
successfully but I can't ping the hosts or connect to them using 
gnt-instance console <someVM>. Any ideas what the issue might be?

One thing I noticed was that the arp -n output are giving me 
"(incomplete)" listings in the "HWaddress" column (arp from the 
net-tools package), which IMU means that ethernet/layer2 frames are not 
passed around correctly - usually meaning that bridges aren't setup 
properly, right? This applies to the 192.168.1.200 lan address and the 
ip address assigned to the VM hosts which I manually set to 
192.168.1.210 instead of ip=pool as in the example. My local network is 
setup to use 192.168.1.0/24 addresses.

I think it would be great if blog posts like these had comments enabled, 
so that people trying to follow a tutorial would be able to discuss and 
help eachother directly on the blog post page.

Any suggestions?

Best regards,
David

Re: Ganeti with Guix tutorial from the Guix blog post

[Marius Bakke]

[-- Attachment #1: Type: text/plain, Size: 2242 bytes --]

david larsson <david.larsson@selfhosted.xyz> writes:

> Hi!
>
> First of all - thanks for a great post about using Ganeti with Guix!

Thanks.  :-)

> I need some help with the networking setup part since I am stuck at the 
> end stages of the blog post tutorial - I am unable to run "gnt-instance 
> console <someVM>". So, I have things setup enough to create these VM's 
> successfully but I can't ping the hosts or connect to them using 
> gnt-instance console <someVM>. Any ideas what the issue might be?

If the serial console is not responding (i.e. pressing RET does not make
a login screen appear), it is likely the instance has failed to boot.  A
typical cause is lack of bootloader.  In that case the instance will use
100% of a core reading the same virtual disk sector over and over...

You can configure a SPICE server with "gnt-instance modify -H
kvm:spice_bind=0.0.0.0" and connect remotely with 'spicy' (from
spice-gtk) to the host IP and the allocated instance port (gnt-instance
info foo | grep port).  Then you should be able to see what QEMU is up
to.

Clues may also be found in
/var/log/ganeti/os/add_$provider_$instance_$date.

> One thing I noticed was that the arp -n output are giving me 
> "(incomplete)" listings in the "HWaddress" column (arp from the 
> net-tools package), which IMU means that ethernet/layer2 frames are not 
> passed around correctly - usually meaning that bridges aren't setup 
> properly, right? This applies to the 192.168.1.200 lan address and the 
> ip address assigned to the VM hosts which I manually set to 
> 192.168.1.210 instead of ip=pool as in the example. My local network is 
> setup to use 192.168.1.0/24 addresses.

It could be useful to see the relevant system configuration, as well as
output of 'ovs-vsctl show' and 'gnt-instance info the-instance' (and
maybe also 'gnt-network info').

> I think it would be great if blog posts like these had comments enabled, 
> so that people trying to follow a tutorial would be able to discuss and 
> help eachother directly on the blog post page.

That is an interesting suggestion.  The blog is entirely static, but
perhaps we could link in a mailing list or something (no joke!).  :-)

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Ganeti with Guix tutorial from the Guix blog post

[david larsson]

On 2020-09-26 16:23, Marius Bakke wrote:
> david larsson <david.larsson@selfhosted.xyz> writes:
> 
>> Hi!
>> 
>> First of all - thanks for a great post about using Ganeti with Guix!
> 
> Thanks.  :-)
> 
>> I need some help with the networking setup part since I am stuck at 
>> the
>> end stages of the blog post tutorial - I am unable to run 
>> "gnt-instance
>> console <someVM>". So, I have things setup enough to create these VM's
>> successfully but I can't ping the hosts or connect to them using
>> gnt-instance console <someVM>. Any ideas what the issue might be?
> 
> If the serial console is not responding (i.e. pressing RET does not 
> make
> a login screen appear), it is likely the instance has failed to boot.  
> A
> typical cause is lack of bootloader.  In that case the instance will 
> use
> 100% of a core reading the same virtual disk sector over and over...

This happens for the busterVM1 that I created as in the tutorial. Do you 
have any suggestion on how to verify that no bootloader is installed and 
how to install it?

By the way, is it possible to use drbd for a Guix instance VM?

> 
> You can configure a SPICE server with "gnt-instance modify -H
> kvm:spice_bind=0.0.0.0" and connect remotely with 'spicy' (from
> spice-gtk) to the host IP and the allocated instance port (gnt-instance
> info foo | grep port).  Then you should be able to see what QEMU is up
> to.

I did this and indeed busterVM1 has just stopped on the Seabios boot 
screen. Grub is not loaded.

> 
> Clues may also be found in
> /var/log/ganeti/os/add_$provider_$instance_$date.

I only guix+default files in that folder, nothing for debootstrap+buster

> 
>> One thing I noticed was that the arp -n output are giving me
>> "(incomplete)" listings in the "HWaddress" column (arp from the
>> net-tools package), which IMU means that ethernet/layer2 frames are 
>> not
>> passed around correctly - usually meaning that bridges aren't setup
>> properly, right? This applies to the 192.168.1.200 lan address and the
>> ip address assigned to the VM hosts which I manually set to
>> 192.168.1.210 instead of ip=pool as in the example. My local network 
>> is
>> setup to use 192.168.1.0/24 addresses.
> 
> It could be useful to see the relevant system configuration, as well as
> output of 'ovs-vsctl show' and 'gnt-instance info the-instance' (and
> maybe also 'gnt-network info').

I worked it out, and I can now ssh into the guix1 VM, like 
this(presenting it here for posterity if it helps someone):

   - This I had already done: interface name was enp0s25 instead of eth0 
in my case (ip link show)

   - mac-address needed fixed in my case since the 2 cluster members had 
the same address, and then restart the networking service. (ip link set 
enp0s25 address ab:cd:ef:ab:cd:ef && herd restart networking)

   - I also had to open ports in my router between the cluster nodes - 
port 22 tcp for SSH and for the ganeti stuff the ports 
1811/tcp,1814/udp,1815/tcp,80/tcp,5080/tcp.

   - I made sure to generate the root user's SSH key before creating the 
guix+default cluster node to have ssh access to the node itself.

   - I had to manually create the path /srv/ganeti/file-storage

   - I had some issue creating the guix+default cluster node with recent 
commits (possibly a bug), and had to resort to an old commit close in 
time to the blog post - I picked -O 
"commit=7e6df0e99305c800a24b9217776f8b75dad3bab9" from July 19th which 
worked.

Thanks for your help so far!

> 
>> I think it would be great if blog posts like these had comments 
>> enabled,
>> so that people trying to follow a tutorial would be able to discuss 
>> and
>> help eachother directly on the blog post page.
> 
> That is an interesting suggestion.  The blog is entirely static, but
> perhaps we could link in a mailing list or something (no joke!).  :-)

That sounds nice.

Re: Ganeti with Guix tutorial from the Guix blog post

[Marius Bakke]

[-- Attachment #1: Type: text/plain, Size: 865 bytes --]

Hello!

I worked with David off-list on this, and it eventually turned out that
the problems were hardware-related.

The cluster consisted of two Librebooted x200 machines.  One that
happened to support KVM, and one that did not.  The Libreboot wiki entry
on the x200 states that there is only one known microcode revision that
works with KVM:

  https://libreboot.org/docs/hardware/x200.html#compatibility_noblobs

Interestingly, the failing machine did have the mentioned 1067a
microcode version, whereas the other (functioning) node had 10676.

The problem with 1067a manifests with QEMU starting, but not loading the
guest OS.  Live migrating a running VM to the 1067a node *did* work,
adding to the confusion.  Just "offline migration" and starting new VMs
did not.

I'm happy to report that the cluster works fine after replacing the
dysfunctional node.  :-)

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]