利用LVS+Keepalived 实现高性能高可用负载均衡服务器

利用LVS+Keepalived 实现高性能高可用负载均衡服务器

     作者：feiyang     blog：pkfeiyang.cublog.cn

 

, , , ,

背景:
        随着你的网站业务量的增长你网站的服务器压力越来越大？需要负载均衡方案！商业的硬件如F5又太贵，你们又是创业型互联公司如何有效节约成本，节省不必要的浪费？同时实现商业硬件一样的高性能高可用的功能？有什么好的负载均衡可伸张可扩展的方案吗？答案是肯定的！有！我们利用LVS+Keepalived基于完整开源软件的架构可以为你提供一个负载均衡及高可用的服务器。

LVS+Keepalived 介绍
LVS
LVS是Linux Virtual Server的简写，意即Linux虚拟服务器，是一个虚拟的服务器集群系统。本项目在1998年5月由章文嵩博士成立，是中国国内最早出现的自由软件项目之一。目前有三种IP负载均衡技术（VS/NAT、VS/TUN和VS/DR）；
十种调度算法（rrr|wrr|lc|wlc|lblc|lblcr|dh|sh|sed|nq）。

Keepalvied
               Keepalived在这里主要用作RealServer的健康状态检查以及LoadBalance主机和BackUP主机之间failover的实现
P信息列表：
名称        IP

1         LVS-DR-Master        192.168.188.138   （lvs本身的真实ip）    

2         LVS-DR-BACKUP        192.168.188.139     

3         LVS-DR-VIP             192.168.188.131       (虚拟ip)

4         WEB1-Realserver        10.10.10.133 （web的真实ip）     

5         WEB2-Realserver        10.10.10.136       

6         GateWay               192.168.188.1

复制代码

三. 安装LVS和Keepalvied软件包
   1. 下载相关软件包

7              #mkdir /usr/local/src/lvs

8              #cd /usr/local/src/lvs

9           wget

wget

 yum install -y kernel-devel  如果没找内核目录时要进行安装

2. 安装LVS和Keepalived

10      #lsmod |grep ip_vs

11              #uname -r

12              2.6.18-164.el5

13              # ln -s /usr/src/kernels/2.6.18-194.11.3.el5-i686/ /usr/src/linux  （不能连接时 缺包 要装一个kernel-devel）

14       

15              #tar zxvf ipvsadm-1.24.tar.gz

16              #cd ipvsadm-1.24

17              #make && make install

18              #find / -name ipvsadm  # 查看ipvsadm的位置

19       

20              #tar zxvf keepalived-1.1.20.tar.gz

21              #cd keepalived-1.1.20

22              #./configure  && make && make install

23              #find / -name keepalived  # 查看keepalived位置               

24         

25              #cp /usr/local/etc/rc.d/init.d/keepalived /etc/rc.d/init.d/

26              #cp /usr/local/etc/sysconfig/keepalived /etc/sysconfig/

27              #mkdir /etc/keepalived

28              #cp /usr/local/etc/keepalived/keepalived.conf /etc/keepalived/

29              #cp /usr/local/sbin/keepalived /usr/sbin/

30              #service keepalived start|stop     #做成系统启动服务方便管理

复制代码

四. 配置LVS实现负载均衡
  1． LVS-DR，配置LVS脚本实现负载均衡
注: 此脚本在于演示方便大家理解lvs,在keepalived方案中不要启动此脚本，所以看后面有注掉，很多朋友对此有问题。关于LVS的keepalvied的HA方案，完全由keepalived.conf一个文件搞定，特此声明！

1              [root@daniel ~]# vi /usr/local/sbin/lvs-dr-sh

2          

3         #!/bin/bash

4         GW=192.168.188.1

5         # website director vip.

6         SNS_VIP=192.168.188.131

7         SNS_RIP1=192.168.188.133

8         SNS_RIP2=192.168.188.136

9         /etc/rc.d/init.d/functions

10       

11      logger $0 called with $1

12       

13      case "$1" in

14       

15      start)

16                 # set squid vip

17                 /sbin/ipvsadm --set 30 5 60

18                 /sbin/route add -host $SNS_VIP dev eth0:0

19                 /sbin/ipvsadm -A -t $SNS_VIP:80 -s wrr -p 3

20                 /sbin/ipvsadm -a -t $SNS_VIP:80 -r $SNS_RIP1:80 -g -w 1

21                 /sbin/ipvsadm -a -t $SNS_VIP:80 -r $SNS_RIP2:80 -g -w 1

22                 touch /var/lock/subsys/ipvsadm >/dev/null 2>&1

23       

24             ;;

25      stop)

26                 /sbin/ipvsadm -C

27                 /sbin/ipvsadm -Z

28                 ifconfig eth0:0 down

29                 ifconfig eth0:1 down

30                 route del $SNS_VIP

31                 route del $SS_VIP

32                 rm -rf /var/lock/subsys/ipvsadm >/dev/null 2>&1

33                 echo "ipvsadm stoped"

34                 ;;

35       

36      status)

37       

38                 if [ ! -e /var/lock/subsys/ipvsadm ];then

39                      echo "ipvsadm stoped"

40                      exit 1

41                 else

42                      echo "ipvsadm OK"

43                 fi

44                 ;;

45       

46      *)

47                echo "Usage: $0 {start|stop|status}"

48                 exit 1

49      esac

50      exit 0

复制代码

2． 配置Realserver脚本.

1                 [root@daniel ~]# vi /usr/local/sbin/realserver.sh

2          

3         #!/bin/bash

4         # description: Config realserver lo and apply noarp

5          

6         SNS_VIP=192.168.188.131

7         /etc/rc.d/init.d/functions

8          

9         case "$1" in

10      start)

11             ifconfig lo:0 $SNS_VIP netmask 255.255.255.255 up

12             /sbin/route add -host $SNS_VIP dev lo:0

13             echo "1" >/proc/sys/net/ipv4/conf/lo/arp_ignore

14             echo "2" >/proc/sys/net/ipv4/conf/lo/arp_announce

15             echo "1" >/proc/sys/net/ipv4/conf/all/arp_ignore

16             echo "2" >/proc/sys/net/ipv4/conf/all/arp_announce

17             sysctl -p >/dev/null 2>&1

18             echo "RealServer Start OK"

19       

20             ;;

21      stop)

22             ifconfig lo:0 down

23             route del $SNS_VIP >/dev/null 2>&1

24             echo "0" >/proc/sys/net/ipv4/conf/lo/arp_ignore

25             echo "0" >/proc/sys/net/ipv4/conf/lo/arp_announce

26             echo "0" >/proc/sys/net/ipv4/conf/all/arp_ignore

27             echo "0" >/proc/sys/net/ipv4/conf/all/arp_announce

28             echo "RealServer Stoped"

29             ;;

30      *)

31             echo "Usage: $0 {start|stop}"

32             exit 1

33      esac

34      exit 0

复制代码

或者采用secondary ip address方式配置
          # vi /etc/sysctl.conf

 

35                net.ipv4.conf.lo.arp_ignore = 1

36                net.ipv4.conf.lo.arp_announce = 2

37                net.ipv4.conf.all.arp_ignore = 1

38                net.ipv4.conf.all.arp_announce = 2

39                [code]

40                #sysctl –p

41                #ip addr add 192.168.188.131/24 dev lo：0 （在web服务器上也要做）

42                #ip add list 查看是否绑定

43        3. 启动lvs-dr脚本和realserver启本，在DR上可以查看LVS当前状态:

44              #watch ipvsadm –ln

复制代码

2. 配置虚地址: vi /etc/sysconfig/network-scripts/ifcfg-lo:0,

添加

DEVICE=lo:0

IPADDR=192.168.188.131

NETMASK=255.255.255.0

ONBOOT=yes

NAME=loopback

3. 关闭RealServer的ARP，

或者编辑/etc/sysctl.conf,修改系统变量

net.ipv4.conf.eth0.arp_ignore=1

net.ipv4.conf.eth0.arp_announce=1

net.ipv4.ip_forward = 1

#sysctl –p   ：   使配置生效

补充：打开

echo "1" > /proc/sys/net/ipv4/ip_forwarding

 

 

 

五．利用Keepalvied实现负载均衡和和高可用性
  1.配置在主负载均衡服务器上配置keepalived.conf

[root@daniel ~]# vi /etc/keepalived/keepalived.conf

 

/bin/bash: Configuration: command not found

bal_defs {

   notification_email {

     [e-mail]

   }

   notification_email_from [e-mail]

   smtp_server 127.0.0.1

 #  smtp_connect_timeout 30

   router_id LVS_DEVEL

}

###by daniel

vrrp_instance VI_1 {

    state MASTER  #备份服务器上将MASTER改为BACKUP 

    interface eth0

    virtual_router_id 51

    priority 100 # 备份服务上将100改为99

    advert_int 1

    authentication {

        auth_type PASS

        auth_pass 1111

    }

    virtual_ipaddress {

       192.168.188.131

  #(如果有多个VIP，继续换行填写.)

    }

}

 

virtual_server 192.168.188.131 80 {

1              delay_loop 6                  #(每隔10秒查询realserver状态)

2             lb_algo wrr                  #(lvs 算法)

3             lb_kind DR                  #(Direct Route)

4             persistence_timeout 60        #(同一IP的连接60秒内被分配到同一台realserver)

5             protocol TCP                #(用TCP协议检查realserver状态)

6          

7             real_server 192.168.188.133  80 {

8                 weight 3               #(权重)

9                 TCP_CHECK {

10              connect_timeout 10       #(10秒无响应超时)

            nb_get_retry 3

            delay_before_retry 3

            conect_port 80

        }

    }

real_server 192.168.188.136 80 {

        weight 3

           TCP_CHECK {

           connect_timeout 10

            nb_get_retry 3

            delay_before_retry 3

            conect_port 80

        }

    }

}
       

复制代码

2. BACKUP服务器同上配置，先安装lvs再按装keepalived,仍后配置/etc/keepalived/keepalived.conf，只需将红色标示的部分改一下即可.

3. 停掉lvs_dr.sh演示脚本
   vi /etc/rc.local
   #/usr/local/sbin/lvs-dr.sh  将lvs-dr.sh这个脚本注释掉。
   #/usr/local/sbin/lvs-dr.sh stop 停止lvs-dr脚本
   #/etc/init.d/keepalived start  启动keepalived 服务，keepalived就能利用keepalived.conf 配置文件，实现负载均衡和高可用.
     整个LVS负均衡HA方案，由keepalived.conf一个文件即可搞定！

4. 查看lvs服务是否正常
  #watch ipvsadm –ln

11      IP Virtual Server version 1.2.1 (size=4096)

12      Prot LocalAddressort Scheduler Flags

13        -> RemoteAddressort           Forward Weight ActiveConn InActConn

14      TCP  192.168.188131:80 wrr persistent 60

15        -> 192.168.188.133:80            Route   3      0          0

16        -> 192.168.188.136:80             Route   3      0          0

复制代码

#tail –f /var/log/message  监听日志，查看状态，测试LVS负载均衡及高可用性是否有效。

5．停Master服务器的keepalived服务，查看BAKCUP服务器是否能正常接管服务。


四．相关参考
  1．LVS 基础知识汇总
LVS的算法介绍              
学习LVS的三种转发模式      
LVS中的IP负载均衡技术      
更多的请到 负载均衡版查看

Keepalived 相关参考资料。
   http://www.keepalived.org/documentation.html

 

 

 

 

错误输出：

./lvs-dr.sh: line 21: ./etc/rc.d/init.d/functions: No such file or directory

SIOCSIFBRDADDR: Cannot assign requested address

SIOCSIFFLAGS: Cannot assign requested address

SIOCSIFFLAGS: Cannot assign requested address

SIOCADDRT: No such device

Service already exists

Destination already exists

Destination already exists

 

解决：ip地址冲突

 

2：[root@ipzhu67 ~]# /etc/init.d/ipvsadm start

Clearing the current IPVS table:                           [  OK  ]

Applying IPVS configuration: /etc/init.d/ipvsadm: line 62: /etc/sysconfig/ipvsadm: No such file or directo

 

 

解决：touch ipvsadm