Lance Albertson

I’ve been asked quite a bit about how I do our network setup with Ganeti. I admit that it did take me a bit to figure out a sane way to do it in Gentoo. Unfortunately (at least in baselayout-1.x) bringing up VLANs with bridge interfaces in Gentoo is rather a pain. What I’m about to describe is basically a hack and there’s probably a better way to do this. I hope it gets improved in baselayout-2.x but I haven’t had a chance to take a look. Please feel free to add comments on what you feel will work better.

The key problem I ran into was dealing with starting up the vlan interfaces first, then starting up the bridged interfaces in the correct order. Here’s a peek at the network config on one of our Ganeti hosts on Gentoo:

# bring up bridge interfaces manually after eth0 is up
postup() {
    local vlans="42 113"
    if [ "${IFACE}" = "eth0" ] ; then
        for vlan in $vlans ; do
            /etc/init.d/net.br${vlan} start
            if [ "${vlan}" = "113" ] ; then
                # make sure the bridges get going first
                sleep 10
            fi
        done
    fi
}
# bring down bridge interfaces first
predown() {
    local vlans="42 113"
    if [ "${IFACE}" = "eth0" ] ; then
        for vlan in $vlans ; do
            /etc/init.d/net.br${vlan} stop
        done
    fi
}

# Setup trunked VLANs
vlans_eth0="42 113"
config_eth0=( "null" )
vconfig_eth0=( "set_name_type VLAN_PLUS_VID_NO_PAD" )
config_vlan42=( "null" )
config_vlan113=( "null" )

# Bring up primary IP on eth0 via the bridged interface
bridge_br42="vlan42"
config_br42=( "10.18.0.150 netmask 255.255.254.0" )
routes_br42=( "default gw 10.18.0.1" )

# Setup bridged VLAN interfaces
bridge_br113="vlan113"
config_br113=( "null" )

# Backend drbd network
config_eth1=( "192.168.19.136 netmask 255.255.255.0" )

The latter portion of the config its fairly normal. I setup eth0 to null, set the VLAN’s to null, then I add settings to the bridge interfaces. In our case we have the IP for the node itself on br42. The rest of the VLAN’s are just set to null. Finally we setup the backend secondary IP.

The first part of the config is the “fun stuff”. In order for this to work you need to only add net.eth0 and net.eth1 to the default enabled level. The post_up() function will start the bridge interfaces after eth0 has started and iterates through the list of vlans/bridges. Since I’m using the bridge interface as the primary host connection, I added a simple sleep at the end to let it see the traffic first.

That’s it! A fun hack that seems to work. I would love to hear feedback on this :)

Installing Ganeti is a relatively simple process on Gentoo. This post will go over the basics on getting it running on Gentoo. Its based primarily on a wiki page at the OSUOSL so check it out for more detailed instructions. I also recommend you read the upstream docs on Ganeti prior to installing it on your own. It will cover a lot more topics in detail and this post is intended just as a diff from that doc.

I should note that I have only installed Ganeti with KVM and have not tested it with Xen on Gentoo. I appreciate feedback if you have installed and used Xen with Ganeti on Gentoo. I’m also the current package maintainer for Ganeti and the related packages in Gentoo such as:

• app-emulation/ganeti (primary package)
• app-emulation/ganeti-htools (automatic allocation)
• app-emulation/ganeti-instance-debootstrap (instance creation)

The first step is to install a base Gentoo system using the standard profile. You can use a hardened profile however if you intend to use ganeti-htools, it requires haskell which seems to have issues in hardened.

Configuring DNS

Ganeti requires the following names to resolve before you can set it up.

• A master name for the cluster, this IP must be available (ganeti.example.org)
• A name for each node or Dom0 (node1.example.org)
• A name for each instance or virtual machine (instance1.example.org)

Kernel

DRBD is optional in Ganeti so you can skip this step if you’re not planning on using it. DRBD was recently included in the mainline kernel in 2.6.33 however Gentoo’s DRBD packages do not currently reflect that. I hope to get that changed soon but for now you have two options.

1. Install gentoo-sources, drbd, and drbd-kernel
2. Install gentoo-sources & enable drbd, install drbd without deps

For simplicity, I’ll describe option #2 above below. Check out the wiki page for #1.

DRBD requires you have the following option enabled. Make sure you’ve rebooted using a kernel with these options above before you continue.

Device Drivers --->
    <*> Connector - unified userspace <-> kernelspace linker

We recommend that you keyword both sys-cluster/drbd and sys-cluster/drbd-kernel so that you pull in the latest 8.3.x version.

echo "sys-cluster/drbd" >> /etc/portage/package.keywords
echo "sys-cluster/drbd-kernel" >> /etc/portage/package.keywords

Install DRBD.

emerge drbd

Ganeti uses DRBD in a unique way and requires the module to be loaded with specific settings. Add the autoload settings and load the module.

echo "drbd minor_count=255 usermode_helper=/bin/true" >> /etc/modules.autoload.d/kernel-2.6
modprobe drbd

If you forget this step, you will get an error similar to the one mentioned in this email thread.

Install Ganeti

Set the appropriate USE flags. In this case we will be using kvm with drbd.

echo "app-emulation/ganeti kvm drbd" >> /etc/portage/package.use

Install Ganeti (you might need to keyword other dependencies)

emerge ganeti

Configure Networking

There’s currently two methods for setting up networking: bridged or routed. I picked the bridged method mainly because I’m familiar with the setup and it seemed to be the simplest.

Ideally you should have a public network that will be used for communicating with the nodes and instances from the outside, and a backend private network that will be used by ganeti for DRBD, migrations, etc. Assuming your public IP (which node1.example.org should resolve to) is 10.1.0.11 and your backend IP is 192.168.1.11, you should edit /etc/conf.d/net to look something like this:

bridge_br0="eth0"
config_eth0=( "null" )

config_br0=( "10.1.0.11 netmask 255.255.254.0" )
routes_br0=( "default gw 10.1.0.1" )

# make sure eth0 is up before configuring br0
depend_br0() {
        need net.eth0
}

config_eth1=( "192.168.1.11 netmask 255.255.255.0" )

You can have a more complicated networking setup using VLAN tagging and bridging but I’ll go over that in another blog post.

Set the Hostname

Ganeti is picky about hostnames, and requires that the output of hostname be fully qualified. So make sure /etc/conf.d/hostname uses the FQDN and looks like this:

HOSTNAME="node1.example.org"

NOT like this:

HOSTNAME="node1"

Configure LVM

It is recommended that you edit this line in /etc/lvm/lvm.conf

filter = [ "r|/dev/nbd.*|", "a/.*/", "r|/dev/drbd[0-9]+|" ]

The important part is the

r|/dev/drbd[0-9]+|

entry, which will prevent LVM from scanning drbd devices.

Now, go ahead and create an LVM volume group with the disks you plan to use for instance storage. The default name that Ganeti prefers is xenvg but we recommend you choose something more useful for your infrastructure (we use ganeti).

pvcreate /dev/sda3
vgcreate ganeti /dev/sda3

Initialize the Cluster

Now we can initialize the cluster on the first node. The command below will do the following:

• Set br0 as the primary interface for Ganeti communication
• Set 192.168.1.11 as the DRBD ip for the node
• Enable KVM
• Set the default bridged interface for instances to br0
• Set the default KVM settings to 2 vcpus & 512M RAM
• Set the default kernel path to /boot/guest/vmlinuz-x86_64
• Set the master DNS name is ganeti.example.org

gnt-cluster init --master-netdev=br0 \
  -g ganeti \
  -s 192.168.1.11 \
  --enabled-hypervisors=kvm \
  -N link=br0 \
  -B vcpus=2,memory=512M \
  -H kvm:kernel_path=/boot/guest/vmlinuz-x86_64
  ganeti.example.org

Now you have a ganeti cluster! Lets verify everything is setup correctly.

$ gnt-cluster verify
Sun May 16 22:43:00 2010 * Verifying global settings
Sun May 16 22:43:00 2010 * Gathering data (1 nodes)
Sun May 16 22:43:02 2010 * Verifying node status
Sun May 16 22:43:02 2010 * Verifying instance status
Sun May 16 22:43:02 2010 * Verifying orphan volumes
Sun May 16 22:43:02 2010 * Verifying remaining instances
Sun May 16 22:43:02 2010 * Verifying N+1 Memory redundancy
Sun May 16 22:43:02 2010 * Other Notes
Sun May 16 22:43:02 2010 * Hooks Results

Yay!

SSH Keys

Ganeti uses ssh to run some tasks but not for all tasks. During the initialization, it generated a new ssh key for the root user and installs it in /root/.ssh/authorized_keys. In our case, we manage that file with cfengine, so to work around it we copy the key as /root/.ssh/authorized_keys2 which ssh will automatically pick up.

Adding nother node

To add an additional node, you duplicate the setup steps above skipping initializing the cluster. Instead run the following command:

gnt-node add -s <node drbd_ip> <node hostname>

Next steps…

The next steps is actually deploying new virtual machines using Ganeti. I wrote a new instance creation script called ganeti-instance-image which uses disk images for deployment. I’m currently working on a new project website with detailed documentation and a blog post about it as well. We’re able to deploy new virtual machines (such as Ubuntu, Centos, or Gentoo) in under 30 seconds using this method!

As some of you may have noticed, I usually carry around a tiny little black laptop when I’m at Beer and Blog or at a coffee shop. Since people ask about it so much, I decided to write a few blog posts about it with this being the first.

What is it? Its an EeePC 1000 from ASUS (the same guys that make motherboards). Its a new type of laptop that is generally referred to as a netbook mainly because it lacks some of the features that a normal laptop has such as a cdrom, being heavy, a multi-core power sucking CPU, and a huge screen. Outside of that, its like any other laptop with a few other enhancements. In my case, this EeePC has these features which I love about it.

• Long battery life, claims one day computing (6+hrs)
• Solid State Drives (SSD) – no moving parts & better battery life!
• Multi-touch track pad
• Very usable keyboard (92% from a normal laptop)
• Lightweight (2lbs 15oz)
• Fast 1.6 Ghz Intel Atom processor
• 802.11n Wi-Fi and Blue-tooth
• 1.3MP web cam

I’ve used my trusty 15″ PowerBook for many years and I still use it, but its become more of a hassle to carry around with me. I’m a UNIX Admin, so I don’t need much on my laptop (xterm, Firefox, Thunderbird, and pidgin) to get work done and the EeePC was a perfect fit. The first models that ASUS releases for the EeePC didn’t appeal to me since they had a screen and keyboard that was too small for me to deal with. A good friend of mine (who also has the habit of being an enabler for me :P) purchased one for himself and kept raging about it. I was actually considering getting an upgrade for my PowerBook but didn’t want to pay $2K for a new MacBook, so instead I decided to get this EeePC for  around $500 (its down to $430 now).

Asus gives you two options for Operating Systems on these laptops: Windows XP or Linux (Xandros). Since I’m a Gentoo developer, I decided to give it a try on this laptop. I encountered a few issues of course, but overall I have all the hardware working. The only annoying thing so far is the wireless driver not being included in the mainline kernel, but that’s going to change soon I hope. There is an open source driver but its a little finicky to deal with on networks using any form of security passwords. But it does work!

In the upcoming posts, I plan to write about the following:

• How I installed Gentoo on it
• Gentoo tweaks I use on it
• Window management
• Firefox tweaks (yes, you need them!)
• How netbooks are helping promote Linux to the masses

I love my EeePC and you should get one too! :)