lvs+keepalived搭建

环境：基于lvs/dr模式 通过wlc 加权轮叫 访问web服务器
              web1 192.168.4.7
              web2 192.168.4.8
              lvs-master 192.168.4.5
              lvs-backup 192.168.4.6
              VIP 192.168.4.10

配置：lvs-master
            [root@example keepalived]# cat keepalived.conf
! Configuration File for keepalived

global_defs {
   notification_email {
     [e-mail]
     [e-mail]
     [e-mail]
   }
   notification_email_from [e-mail]
   smtp_server127.0.0.1
   smtp_connect_timeout 30
   router_id LVS_DEVEL
}

vrrp_instance VI_1 {
    state MASTER
    interface eth1
    virtual_router_id 51
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.4.10
    }
}
virtual_server 192.168.4.10 80 {
      delay_loop 6
      lb_algo wlc
      lb_kind DR
      persistence_timeout 60
      protocol TCP
      real_server 192.168.4.7 80 {
          weight 3
          TCP_CHECK {
             connect_timeout 10
             nb_get_retry 3
             delay_before_retry 3
             connect_port 80
          }
      }
       real_server 192.168.4.8 80 {
          weight 3
          TCP_CHECK {
             connect_timeout 10
             nb_get_retry 3
             delay_before_retry 3
             connect_port 80
          }
      }
}

        lvs-backup
! Configuration File for keepalived

global_defs {
   notification_email {
     [e-mail]
   }
   notification_email_from [e-mail]
   smtp_server127.0.0.1
   smtp_connect_timeout 30
   router_id LVS_DEVEL
}

vrrp_instance VI_1 {
    state BACKUP
    interface eth2
    virtual_router_id 51
    priority  99
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.4.10
    }
}
virtual_server 192.168.4.10 80 {
      delay_loop 6
      lb_algo wlc
      lb_kind DR
      persistence_timeout 60
      protocol TCP
      real_server 192.168.4.7 80 {
          weight 4
          TCP_CHECK {
             connect_timeout 10
             nb_get_retry 3
             delay_before_retry 3
             connect_port 80
          }
      }
       real_server 192.168.4.8 80 {
          weight 6
          TCP_CHECK {
             connect_timeout 10
             nb_get_retry 3
             delay_before_retry 3
             connect_port 80
          }
      }
}

web配置
         [root@example2 home]# cat vip.sh
#!/bin/bash
# description: Config realserver lo and apply noarp
VIP=192.168.4.10
source  /etc/rc.d/init.d/functions
 
case "$1" in
start)
       /sbin/ifconfig lo:0 $VIP  broadcast $VIP netmask 255.255.255.255 up
       echo "1" >/proc/sys/net/ipv4/conf/lo/arp_ignore
       echo "2" >/proc/sys/net/ipv4/conf/lo/arp_announce
       echo "1" >/proc/sys/net/ipv4/conf/all/arp_ignore
       echo "2" >/proc/sys/net/ipv4/conf/all/arp_announce
       echo "VIPservice Start OK"
 
       ;;
stop)
       /sbin/ifconfig lo:0 down
       echo "0" >/proc/sys/net/ipv4/conf/lo/arp_ignore
       echo "0" >/proc/sys/net/ipv4/conf/lo/arp_announce
       echo "0" >/proc/sys/net/ipv4/conf/all/arp_ignore
       echo "0" >/proc/sys/net/ipv4/conf/all/arp_announce
       echo "VIPservice Stoped"
       ;;
*)
       echo "Usage: $0 {start|stop}"
       exit 1
esac
 
exit 0
修改web的index.html
echo "web1"  > /var/www/html/index.html
echo "web2"  > /var/www/html/index.html

开始测试，启动web服务器的vip.sh 
sh  vip.sh start 用ifconfig查看就出一个lo：0网卡信息
[root@example2 home]# ifconfig
eth2      Link encap:Ethernet  HWaddr 00:0C:29:04:2F:68  
          inet addr:192.168.4.7  Bcast:192.168.4.255  Mask:255.255.255.0
          inet6 addr: fe80::20c:29ff:fe04:2f68/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:3357 errors:0 dropped:0 overruns:0 frame:0
          TX packets:538 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:217741 (212.6 KiB)  TX bytes:45405 (44.3 KiB)
          Interrupt:18 Base address:0x2000

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)

lo:0      Link encap:Local Loopback  
          inet addr:192.168.4.10  Mask:255.255.255.255
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
启动master和backup的keepalived service  keepalived start

执行 ipvsadm -lnc

通过访问192.168.4.10可以看到页面信息
web1
关掉master的服务，再去请求

master /var/log/messages日志


May 20 19:09:08 localhost Keepalived[19251]: Starting Keepalived v1.2.12 (04/30,2014)
May 20 19:09:08 localhost Keepalived[19252]: Starting Healthcheck child process, pid=19254
May 20 19:09:08 localhost Keepalived[19252]: Starting VRRP child process, pid=19255
May 20 19:09:08 localhost Keepalived_vrrp[19255]: Netlink reflector reports IP 192.168.4.6 added
May 20 19:09:08 localhost Keepalived_vrrp[19255]: Netlink reflector reports IP 192.168.218.5 added
May 20 19:09:08 localhost Keepalived_vrrp[19255]: Netlink reflector reports IP fe80::20c:29ff:fe25:11b2 added
May 20 19:09:08 localhost Keepalived_vrrp[19255]: Netlink reflector reports IP fe80::20c:29ff:fe25:11a8 added
May 20 19:09:08 localhost Keepalived_vrrp[19255]: Registering Kernel netlink reflector
May 20 19:09:08 localhost Keepalived_vrrp[19255]: Registering Kernel netlink command channel
May 20 19:09:08 localhost Keepalived_vrrp[19255]: Registering gratuitous ARP shared channel
May 20 19:09:08 localhost Keepalived_healthcheckers[19254]: Netlink reflector reports IP 192.168.4.6 added
May 20 19:09:08 localhost Keepalived_healthcheckers[19254]: Netlink reflector reports IP 192.168.218.5 added
May 20 19:09:08 localhost Keepalived_healthcheckers[19254]: Netlink reflector reports IP fe80::20c:29ff:fe25:11b2 added
May 20 19:09:08 localhost Keepalived_healthcheckers[19254]: Netlink reflector reports IP fe80::20c:29ff:fe25:11a8 added
May 20 19:09:08 localhost Keepalived_healthcheckers[19254]: Registering Kernel netlink reflector
May 20 19:09:08 localhost Keepalived_healthcheckers[19254]: Registering Kernel netlink command channel
May 20 19:09:08 localhost Keepalived_vrrp[19255]: Opening file '/etc/keepalived/keepalived.conf'.
May 20 19:09:08 localhost Keepalived_healthcheckers[19254]: Opening file '/etc/keepalived/keepalived.conf'.
May 20 19:09:08 localhost Keepalived_healthcheckers[19254]: Configuration is using : 14019 Bytes
May 20 19:09:08 localhost Keepalived_vrrp[19255]: Configuration is using : 37628 Bytes
May 20 19:09:08 localhost Keepalived_vrrp[19255]: Using LinkWatch kernel netlink reflector...
May 20 19:09:08 localhost Keepalived_vrrp[19255]: VRRP sockpool: [ifindex(2), proto(112), unicast(0), fd(10,11)]
May 20 19:09:08 localhost Keepalived_healthcheckers[19254]: Using LinkWatch kernel netlink reflector...
May 20 19:09:08 localhost Keepalived_healthcheckers[19254]: Activating healthchecker for service [192.168.4.7]:80
May 20 19:09:08 localhost Keepalived_healthcheckers[19254]: Activating healthchecker for service [192.168.4.8]:80
May 20 19:09:09 localhost Keepalived_vrrp[19255]: VRRP_Instance(VI_1) Transition to MASTER STATE
May 20 19:09:10 localhost Keepalived_vrrp[19255]: VRRP_Instance(VI_1) Entering MASTER STATE
May 20 19:09:10 localhost Keepalived_vrrp[19255]: VRRP_Instance(VI_1) setting protocol VIPs.
May 20 19:09:10 localhost Keepalived_vrrp[19255]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth1 for 192.168.4.10
May 20 19:09:10 localhost Keepalived_healthcheckers[19254]: Netlink reflector reports IP 192.168.4.10 added
May 20 19:09:15 localhost Keepalived_vrrp[19255]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth1 for 192.168.4.10
May 20 19:18:08 localhost kernel: IPVS: __ip_vs_del_service: enter
May 20 19:18:08 localhost Keepalived[19252]: Stopping Keepalived v1.2.12 (04/30,2014)
May 20 19:18:08 localhost Keepalived_healthcheckers[19254]: Removing service [192.168.4.7]:80 from VS [192.168.4.10]:80
May 20 19:18:08 localhost Keepalived_healthcheckers[19254]: Removing service [192.168.4.8]:80 from VS [192.168.4.10]:80
May 20 19:18:08 localhost Keepalived_vrrp[19255]: VRRP_Instance(VI_1) sending 0 priority
May 20 19:18:08 localhost Keepalived_vrrp[19255]: VRRP_Instance(VI_1) removing protocol VIPs.

backup的messages日志

May 20 21:26:43 example4 Keepalived: Starting Keepalived v1.1.15 (05/19,2014)
May 20 21:26:43 example4 Keepalived: Starting Healthcheck child process, pid=1948
May 20 21:26:43 example4 Keepalived: Starting VRRP child process, pid=1949
May 20 21:26:43 example4 Keepalived_healthcheckers: Using MII-BMSR NIC polling thread...
May 20 21:26:43 example4 Keepalived_vrrp: Using MII-BMSR NIC polling thread...
May 20 21:26:43 example4 Keepalived_healthcheckers: Netlink reflector reports IP 192.168.4.5 added
May 20 21:26:43 example4 Keepalived_healthcheckers: Registering Kernel netlink reflector
May 20 21:26:43 example4 Keepalived_healthcheckers: Registering Kernel netlink command channel
May 20 21:26:43 example4 Keepalived_vrrp: Netlink reflector reports IP 192.168.4.5 added
May 20 21:26:43 example4 Keepalived_vrrp: Registering Kernel netlink reflector
May 20 21:26:43 example4 Keepalived_vrrp: Registering Kernel netlink command channel
May 20 21:26:43 example4 Keepalived_vrrp: Registering gratutious ARP shared channel
May 20 21:27:03 example4 Keepalived_healthcheckers: Opening file '/etc/keepalived/keepalived.conf'.
May 20 21:27:03 example4 Keepalived_vrrp: Opening file '/etc/keepalived/keepalived.conf'.
May 20 21:27:03 example4 Keepalived_healthcheckers: Configuration is using : 11844 Bytes
May 20 21:27:03 example4 Keepalived_vrrp: Configuration is using : 36231 Bytes
May 20 21:27:03 example4 Keepalived_vrrp: VRRP_Instance(VI_1) Entering BACKUP STATE
May 20 21:27:03 example4 Keepalived_vrrp: VRRP sockpool: [ifindex(2), proto(112), fd(8,9)]
May 20 21:27:03 example4 Keepalived_healthcheckers: Activating healtchecker for service [192.168.4.7:80]
May 20 21:27:03 example4 Keepalived_healthcheckers: Activating healtchecker for service [192.168.4.8:80]
May 20 21:27:03 example4 kernel: IPVS: [wlc] scheduler registered.
May 20 21:27:49 example4 dhclient[1197]: DHCPREQUEST on eth2 to 192.168.4.254 port 67 (xid=0x76964cd)
May 20 21:27:49 example4 dhclient[1197]: DHCPACK from 192.168.4.254 (xid=0x76964cd)
May 20 21:27:49 example4 NetworkManager[1147]: (eth2): DHCPv4 state changed renew -> renew
May 20 21:27:49 example4 NetworkManager[1147]:    address 192.168.4.5
May 20 21:27:49 example4 NetworkManager[1147]:    prefix 24 (255.255.255.0)
May 20 21:27:49 example4 NetworkManager[1147]:    gateway 192.168.4.2
May 20 21:27:49 example4 NetworkManager[1147]:    nameserver '192.168.4.2'
May 20 21:27:49 example4 NetworkManager[1147]:    domain name 'localdomain'
May 20 21:27:49 example4 dhclient[1197]: bound to 192.168.4.5 -- renewal in 856 seconds.
May 20 21:35:51 example4 Keepalived_vrrp: VRRP_Instance(VI_1) Transition to MASTER STATE
May 20 21:35:52 example4 Keepalived_vrrp: VRRP_Instance(VI_1) Entering MASTER STATE
May 20 21:35:52 example4 Keepalived_vrrp: VRRP_Instance(VI_1) setting protocol VIPs.
May 20 21:35:52 example4 Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth2 for 192.168.4.10
May 20 21:35:52 example4 Keepalived_healthcheckers: Netlink reflector reports IP 192.168.4.10 added
May 20 21:35:52 example4 Keepalived_vrrp: Netlink: skipping nl_cmd msg...
May 20 21:35:57 example4 Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth2 for 192.168.4.10

容易出错的地方，在配置keepalived.conf文件的时候 一定要看一下自己对外服务的到底是哪个网卡eth0还是eth1
再就是第一次配置nat模式的时候 ，原理理解不清楚，把web1和web2的网关指向了他们真实的网关，发现怎么也请求不到
[root@example4 keepalived]# ipvsadm
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  192.168.4.10:http wlc persistent 60
  -> 192.168.4.8:http             Route   6      0          0         
  -> 192.168.4.7:http             Route   4      0          0    
当时InActConn有数据，但是就是请求不到，后来才知道原来是web的网关许配置lvs的realip
lvs的activeconn是个一直让我很迷惑的东东.每次看到这个数巨大而真实机上的活动连接数并不是很高的时候,我都忍不住网上搜索一番,虽然大多时候总是无功而返,但是渐渐的总结出来了以下理论.
      ActiveConn是活动连接数,也就是tcp连接状态的ESTABLISHED;InActConn是指除了ESTABLISHED以外的,所有的其 它状态的tcp连接.那既然这样,为什么从lvs里看的ActiveConn会比在真实机上通过netstats看到的ESTABLISHED高很多呢? 问得好!这也是笔者一直迷惑而渐渐清晰的一个问题.原来lvs自身也有一个默认超时时间.可以用ipvsadm -L --timeout查看,默认是900 120 300,分别是TCP TCPFIN UDP的时间.也就是说一条tcp的连接经过lvs后,lvs会把这台记录保存15分钟,而不管这条连接是不是已经失效!所以如果你的服务器在15分钟以 内有大量的并发请求连进来的时候,你就会看到这个数值直线上升.
      其实很多时候,我们看lvs的这个连接数是想知道现在的每台机器的真实连接数吧?怎么样做到这一点呢?其实知道现在的ActiveConn是怎样产生的, 做到这一点就简单了.举个例子:比如你的lvs是用来负载网站,用的模式是dr,后台的web server用的nginx.这时候一条请求过来,在程序没有问题的情况下,一条连接最多也就五秒就断开了.这时候你可以这样设置:ipvsadm --set 5 10 300.设置tcp连接只保持5秒中.如果现在ActiveConn很高你会发现这个数值会很快降下来,直到降到和你用nginx的status看当前连 接数的时候差不多.你可以继续增加或者减小5这个数值,直到真实机的status连接数和lvs里的ActiveConn一致.