Linux Virtual Server on Gentoo for Email

Introduction

Setting up LVS on Gentoo is pretty simple. My setup is described below. I use LVS-NAT instead of VS-DR (Direct routing). VS-DR is faster than LVS-NAT, but I'm doing it this way for a few reasons.

• I want to firewall the cluster off from the rest of the world.
• The nodes comminicate with the NFS and accounts servers on a private network. I do this to help speed up the connections between the nodes and the NFS server where the mail store is located.
• NFS has some security issues that I want to minimize and isolating it to a private network.
• I netboot my nodes and I really don't feel like doing DHCP over a public network.

Note: This document does not describe the setup of the mail nodes. That will be left for another document.

Note: I run a load balanced cluster for email. You should be able to adapt these docs for many other services.

Note: I'm going to assume you have a working installation of Gentoo on a box with two NICs. You might be able to do this on one NIC but that is left as an excersize for the reader.

And here we go .... :-)

Add IPVS to the Kernel

The first thing we're going to do is upgrade to the 2.6.x kernel. You don't have to do this, I suppose, but I did.

emerge -k gentoo-dev-sources

Now add IPVS to the kernel.

cd /usr/src/linux
make menuconfig

Here are the options that I set specific to LVS. Feel free to adjust as needed.

Code maturity level options  --->
 [*] Prompt for development and/or incomplete code/drivers
Device Drivers  --->
 Networking support  --->
 Networking options  --->
  [*] Network packet filtering (replaces ipchains)  --->
  [ ]   Network packet filtering debugging
  IP: Virtual Server Configuration  --->
    IP virtual server support (EXPERIMENTAL)
   [ ]   IP virtual server debugging
   (12)  IPVS connection table size (the Nth power of 2)
   ---   IPVS transport protocol load balancing support
   [*]   TCP load balancing support
   [*]   UDP load balancing support
   [ ]   ESP load balancing support
   [ ]   AH load balancing support
   ---   IPVS scheduler
      round-robin scheduling
      weighted round-robin scheduling
      least-connection scheduling
      weighted least-connection scheduling
      locality-based least-connection scheduling
      locality-based least-connection with replication scheduling
      destination hashing scheduling
      source hashing scheduling
      shortest expected delay scheduling
      never queue scheduling
   ---   IPVS application helper
      FTP protocol helper

Now build and install your new kernel.

make && make modules_install
mount /boot
cp arch/i386/boot/bzImage /boot/kernel-version
cp System.map /boot/System.map-version
cp .config /boot/config-version
umount /boot

Boot to your new kernel and make sure everything works.

Install Needed Ports

ipvsadm

Because we're using the latest kernel we need to be sure that we have the latest version of ipvsadm.

emerge -k >=ipvsadm-1.24

or

cd /usr/portage/sys-cluster/ipvsadm
emerge -k ipvsadm-1.24.ebuild

iptables

You may be able to skip this step. I setup NAT through iptables before I setup the LVS-NAT. It's not a bad idea to have iptables on the gateway/firewall, though.

emerge -k iptables

Install hearbeat

heartbeat is what will provide the high-availability features needed to automatically remove nodes from the rotation when it goes out of service for whatever reason. It also allows me to setup a redundant LB to take over from the main one if it goes down.

USE='ldirectord' emerge -k heartbeat

Note: You can add ldirectord to your USE line in /etc/make.conf.

Configuring LVS

Now the real fun begins. All of the configs are in /etc/ha.d so we'll start there.

Configuring HA

We'll first copy the example configs to /etc/ha.d.

cd /usr/share/doc/heartbeat-version
cp ha.cf haresources /etc/ha.d

ha.cf

Here is my ha.cf with the comments removed.

logfacility     local0
bcast eth1
node    hydra cerberus

hydra is my primary LB and cerberus is the secondary. The names need to match the output of uname -n.

haresources

Here is my haresources, again, with the comments removed. wan_ip is the WAN address of the cluster and lan_ip is the LAN address. The nodes will use lan_ip as their gateway address.

hydra   wan_ip/24/eth0 ldirectord
hydra   lan_ip/24/eth1 ldirectord

Note: This file should be the same on all LVS servers in this group.

authkeys

authkeys controls access to the LVS group. My LBs are talking to each other on a private network so security isn't a big an issue here. If you are trying to setup VS-DR, you will want to use something a bit more secure that this.

auth 1
1 crc

ldirectord.cf

ldirectord.cf controls the load balancer itself. This is where you set which nodes handle which services and the weights for the nodes.

logfile="local0"

virtual = wan_ip:25
        real = node1:25 masq 1000
        real = node2:25 masq 1000
        real = node3:25 masq 1000
        real = node4:25 masq  667
        checktimeout = 10
        scheduler = sed
        protocol = tcp

virtual = wan_ip:110
        real = node1:110 masq 1000
        real = node2:110 masq 1000
        real = node3:110 masq 1000
        real = node4:110 masq  667
        scheduler = sed
        protocol = tcp

virtual = wan_ip:143
        real = node1:143 masq 1000
        real = node2:143 masq 1000
        real = node3:143 masq 1000
        real = node4:143 masq 1000
        scheduler = sed
        protocol = tcp

virtual = wan_ip:80
        real = node1:80 masq   10
        real = node2:80 masq   10
        real = node3:80 masq   10
        real = node4:80 masq   10
        real = node5:80 masq 1000
        scheduler = sed
        persistent = 300
        protocol = tcp
        request = "/testpage.html"
        receive = "This server seems to be up."

See ipvsadm(8) and ldirectord(8) for details on what these options mean.

I want to bring the last block to your attention. That is the setup for WebMail. The persistent option is required to keep users going to the web server when the log in. That is needed to preserve session information.

Turn on IP Forwarding

We need to turn on IP forwarding in /etc/conf.d/iptables or the NAT won't work.

ENABLE_FORWARDING_IPv4="yes"

Setup iptables

Speaking of NAT, I added a NAT rule to iptables. I'm still not 100% sure that it was needed, but it doesn't seem to hurt.

iptables -A POSTROUTING -s lan_net/255.255.255.0 -j MASQUERADE
/etc/init.d/iptables save

Turning on Services

Everything is configured, let's turn things on so that we test.

/etc/init.d/iptables start
/etc/init.d/heartbeat start
/etc/init.d/ldirector start

Use ipvsadm to view the status of the load balancer.

# ipvsadm
IP Virtual Server version 1.2.0 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  wan_ip:smtp sed
  -> mail4:smtp                   Masq    667    22         99        
  -> mail3:smtp                   Masq    1000   34         130       
  -> mail2:smtp                   Masq    1000   28         193       
  -> mail1:smtp                   Masq    1000   28         104       
TCP  wan_ip:www sed persistent 300
  -> mail5:www                    Masq    1000   6          18        
  -> mail4:www                    Masq    10     0          0         
  -> mail3:www                    Masq    10     0          0         
  -> mail2:www                    Masq    10     0          0         
  -> mail1:www                    Masq    10     0          0         
TCP  wan_ip:pop3 sed
  -> mail4:pop3                   Masq    667    2          46        
  -> mail3:pop3                   Masq    1000   3          54        
  -> mail2:pop3                   Masq    1000   3          21        
  -> mail1:pop3                   Masq    1000   2          43        
TCP  wan_ip:imap2 sed
  -> mail4:imap2                  Masq    1000   2          0         
  -> mail3:imap2                  Masq    1000   1          1         
  -> mail2:imap2                  Masq    1000   1          0         
  -> mail1:imap2                  Masq    1000   0          3       

Testing

Now that everything is up, it's time to make sure it works. Use a client or telnet to the wan_ip from outside the cluster. You should be able to see your connection in the server logs.

Making it Permanent

At this point you should have a working system and it's now time to make sure all the services we need will be started when the machine reboots.

rc-update add iptables default
rc-update add heartbeat default

And that, as they say, is that.