利用LVS+Keepalived 实现高性能高可用负载均衡服务器【2】

利用LVS+Keepalived 实现高性能高可用负载均衡服务器

         作者：feiyang     blog：pkfeiyang.cublog.cn

服务器, 负载, Keepalived, LVS, 高性能

背景:

    随着你的网站业务量的增长你网站的服务器压力越来越大？需要负载均衡方案！商业的硬件如F5又太贵，你们又是创业型互联公司如何有效节约成本，节省不必要的浪费？同时实现商业硬件一样的高性能高可用的功能？有什么好的负载均衡可伸张可扩展的方案吗？答案是肯定的！有！我们利用LVS+Keepalived基于完整开源软件的架构可以为你提供一个负载均衡及高可用的服务器。

LVS+Keepalived 介绍

LVS是Linux Virtual Server的简写，意即Linux虚拟服务器，是一个虚拟的服务器集群系统。本项目在1998年5月由章文嵩博士成立，是中国国内最早出现的自由软件项目之一。目前有三种IP负载均衡技术（VS/NAT、VS/TUN和VS/DR）；

十种调度算法（rrr|wrr|lc|wlc|lblc|lblcr|dh|sh|sed|nq）。

Keepalvied

Keepalived在这里主要用作RealServer的健康状态检查以及LoadBalance主机和BackUP主机之间failover的实现

P信息列表：

名称        IP

LVS-DR-Master        192.168.188.138   （lvs本身的真实ip）   

LVS-DR-BACKUP        192.168.188.139    

LVS-DR-VIP             192.168.188.131       (虚拟ip)

WEB1-Realserver        10.10.10.133 （web的真实ip）    

WEB2-Realserver        10.10.10.136      

GateWay               192.168.188.1

 

三. 安装LVS和Keepalvied软件包

1. 下载相关软件包

#mkdir /usr/local/src/lvs

#cd /usr/local/src/lvs

wget

wget

yum install -y kernel-devel  如果没找内核目录时要进行安装

 

2. 安装LVS和Keepalived

#lsmod |grep ip_vs

#uname -r

2.6.18-164.el5

# ln -s /usr/src/kernels/2.6.18-194.11.3.el5-i686/ /usr/src/linux  （不能连接时 缺包 要装一个kernel-devel）

#tar zxvf ipvsadm-1.24.tar.gz

#cd ipvsadm-1.24

#make && make install

#find / -name ipvsadm  # 查看ipvsadm的位置

#tar zxvf keepalived-1.1.20.tar.gz

#cd keepalived-1.1.20

#./configure  && make && make install

#find / -name keepalived  # 查看keepalived位置              

#cp /usr/local/etc/rc.d/init.d/keepalived /etc/rc.d/init.d/

#cp /usr/local/etc/sysconfig/keepalived /etc/sysconfig/

#mkdir /etc/keepalived

#cp /usr/local/etc/keepalived/keepalived.conf /etc/keepalived/

#cp /usr/local/sbin/keepalived /usr/sbin/

#service keepalived start|stop     #做成系统启动服务方便管理

四. 配置LVS实现负载均衡

LVS-DR，配置LVS脚本实现负载均衡

注: 此脚本在于演示方便大家理解lvs,在keepalived方案中不要启动此脚本，所以看后面有注掉，很多朋友对此有问题。关于LVS的keepalvied的HA方案，完全由keepalived.conf一个文件搞定，特此声明！

 [root@daniel ~]# vi /usr/local/sbin/lvs-dr-sh

#!/bin/bash

GW=192.168.188.1

# website director vip.

SNS_VIP=192.168.188.131

SNS_RIP1=192.168.188.133

SNS_RIP2=192.168.188.136

/etc/rc.d/init.d/functions

logger $0 called with $1

case "$1" in

start)

# set squid vip

/sbin/ipvsadm --set 30 5 60

/sbin/ifconfig eth0:0 $SNS_VIP broadcast $SNS_VIP netmask 255.255.255.255  up

/sbin/route add -host $SNS_VIP dev eth0:0

/sbin/ipvsadm -A -t $SNS_VIP:80 -s wrr -p 3

/sbin/ipvsadm -a -t $SNS_VIP:80 -r $SNS_RIP1:80 -g -w 1

/sbin/ipvsadm -a -t $SNS_VIP:80 -r $SNS_RIP2:80 -g -w 1

touch /var/lock/subsys/ipvsadm >/dev/null 2>&1

;;

stop)

/sbin/ipvsadm -C

/sbin/ipvsadm -Z

ifconfig eth0:0 down

#ifconfig eth0:1 down

route del -host $SNS_VIP dev eth0:0

rm -rf /var/lock/subsys/ipvsadm >/dev/null 2>&1

echo "ipvsadm stoped"

;;

status)

if [ ! -e /var/lock/subsys/ipvsadm ];then

echo "ipvsadm stoped"

exit 1

else

echo "ipvsadm OK"

fi

;;

*)

echo "Usage: $0 {start|stop|status}"

exit 1

esac

exit 0

2． 配置Realserver脚本.

[root@daniel ~]# vi /usr/local/sbin/realserver.sh

#!/bin/bash

# description: Config realserver lo and apply noarp

SNS_VIP=192.168.188.131

/etc/rc.d/init.d/functions

case "$1" in

start)

ifconfig lo:0 $SNS_VIP netmask 255.255.255.255 up

/sbin/route add -host $SNS_VIP dev lo:0

echo "1" >/proc/sys/net/ipv4/conf/lo/arp_ignore

echo "2" >/proc/sys/net/ipv4/conf/lo/arp_announce

echo "1" >/proc/sys/net/ipv4/conf/all/arp_ignore

echo "2" >/proc/sys/net/ipv4/conf/all/arp_announce

sysctl -p >/dev/null 2>&1

echo "RealServer Start OK"

;;

stop)

ifconfig lo:0 down

route del -host $SNS_VIP dev lo:0>/dev/null 2>&1

echo "0" >/proc/sys/net/ipv4/conf/lo/arp_ignore

echo "0" >/proc/sys/net/ipv4/conf/lo/arp_announce

echo "0" >/proc/sys/net/ipv4/conf/all/arp_ignore

echo "0" >/proc/sys/net/ipv4/conf/all/arp_announce

echo "RealServer Stoped"

;;

*)

echo "Usage: $0 {start|stop}"

exit 1

esac

exit 0

或者采用secondary ip address方式配置

# vi /etc/sysctl.conf

net.ipv4.conf.lo.arp_ignore = 1

net.ipv4.conf.lo.arp_announce = 2

net.ipv4.conf.all.arp_ignore = 1

net.ipv4.conf.all.arp_announce = 2

[code]

#sysctl –p

#ip addr add 192.168.188.131/24 dev lo:0 （在web服务器上也要做）

ifconfig lo:0 192.168.188.131 netmask 255.255.255.255 broadcast 192.168.188.131 up

#ip add list 查看是否绑定

ipvsadm -A -t 192.168.188.131:80 -s rr

ipvsadm -a -t 192.168.188.131:80 -r 192.168.188.136:80 -g

ipvsadm -a -t 192.168.188.131:80 -r 192.168.188.133：80-g

service ipvsadm save

service ipvsadm start

3. 启动lvs-dr脚本和realserver启本，在DR上可以查看LVS当前状态:

#watch ipvsadm –ln

2. 配置虚地址: vi /etc/sysconfig/network-scripts/ifcfg-lo:0,

添加

DEVICE=lo:0

IPADDR=192.168.188.131

NETMASK=255.255.255.0

ONBOOT=yes

NAME=loopback

3. 关闭RealServer的ARP，

或者编辑/etc/sysctl.conf,修改系统变量

net.ipv4.conf.eth0.arp_ignore=1

net.ipv4.conf.eth0.arp_announce=1

net.ipv4.ip_forward = 1

#sysctl –p   ：   使配置生效

补充：也可以

echo "1" > /proc/sys/net/ipv4/ip_forwarding

五．利用Keepalvied实现负载均衡和和高可用性

  1.配置在主负载均衡服务器上配置keepalived.conf

Keepalived 在编译的时候指向内核，可以加载ipvs，负载均衡和高可用可有keepalived一个文件搞定。

[root@daniel ~]# vi /etc/keepalived/keepalived.conf

/bin/bash: Configuration: command not found

! Configuration File for keepalived

global_defs {

notification_email {

#接收邮件地址

# [e-mail]

# [e-mail]

}

notification_email_from [e-mail]

smtp_server 127.0.0.1 #邮件服务器

# smtp_connect_timeout 30

router_id LVS_DEVEL

}

# VIP1

vrrp_instance VI_1 {

state MASTER #备份服务器上将MASTER改为BACKUP

interface eth0

virtual_router_id 51

priority 100 # 备份服务上将100改为99

advert_int 1

authentication {

auth_type PASS

auth_pass 1111

}

virtual_ipaddress {

192.168.1.173

#(如果有多个VIP，继续换行填写.)

}

}

virtual_server 192.168.188.131 80 {

delay_loop 6 #(每隔10秒查询realserver状态)

lb_algo wrr #(lvs 算法)

lb_kind DR #(Direct Route)

persistence_timeout 60 #(同一IP的连接60秒内被分配到同一台realserver)

protocol TCP #(用TCP协议检查realserver状态)

real_server 192.168.188.133 80 {

#weight 3 #(权重)

TCP_CHECK {

connect_timeout 10 #(10秒无响应超时)

##persistence_timeout #注释掉 这样可以均衡的负载了

nb_get_retry 3

delay_before_retry 3

connect_port 80

}

}

real_server 192.168.188.136 80 {

#weight 3

TCP_CHECK {

connect_timeout 10

nb_get_retry 3

delay_before_retry 3

connect_port 80

}

}

}         

2. BACKUP服务器同上配置，先安装lvs再按装keepalived,仍后配置/etc/keepalived/keepalived.conf，只需将红色标示的部分改一下即可.

 

3. 停掉lvs_dr.sh演示脚本

   vi /etc/rc.local

   #/usr/local/sbin/lvs-dr.sh  将lvs-dr.sh这个脚本注释掉。

   #/usr/local/sbin/lvs-dr.sh stop 停止lvs-dr脚本

   #/etc/init.d/keepalived start  启动keepalived 服务，keepalived就能利用keepalived.conf 配置文件，实现负载均衡和高可用.

     整个LVS负均衡HA方案，由keepalived.conf一个文件即可搞定！

4. 查看lvs服务是否正常

#watch ipvsadm –ln

IP Virtual Server version 1.2.1 (size=4096)

Prot LocalAddressort Scheduler Flags

-> RemoteAddressort           Forward Weight ActiveConn InActConn

TCP  192.168.188131:80 wrr persistent 60

-> 192.168.188.133:80            Route   3      0          0

-> 192.168.188.136:80             Route   3      0          0

#tail –f /var/log/message  监听日志，查看状态，测试LVS负载均衡及高可用性是否有效。

 

5．停Master服务器的keepalived服务，查看BAKCUP服务器是否能正常接管服务。

四．相关参考

1．LVS 基础知识汇总

LVS的算法介绍              

学习LVS的三种转发模式      

LVS中的IP负载均衡技术      

更多的请到 负载均衡版查看

Keepalived 相关参考资料。

   http://www.keepalived.org/documentation.html

 

五：错误收集

错误输出：

./lvs-dr.sh: line 21: ./etc/rc.d/init.d/functions: No such file or directory

SIOCSIFBRDADDR: Cannot assign requested address

SIOCSIFFLAGS: Cannot assign requested address

SIOCSIFFLAGS: Cannot assign requested address

SIOCADDRT: No such device

Service already exists

Destination already exists

Destination already exists

解决：ip地址冲突

2：[root@ipzhu67 ~]# /etc/init.d/ipvsadm start

Clearing the current IPVS table:                           [  OK  ]

Applying IPVS configuration: /etc/init.d/ipvsadm: line 62: /etc/sysconfig/ipvsadm: No such file or directo

解决：touch ipvsadm

当不能访问后端真实服务器时，试着在添加如下规则

ifconfig lo:0 192.168.188.131 netmask 255.255.255.255  up

访问正常。

参考：http://blogimg.chinaunix.net/blog/upfile2/100919210840.pdf

田逸老师的blog：

http://blog.formyz.org/?p=46