利用LVS+Keepalived 实现高性能高可用负载均衡。

本文根据修改     同时，我的LVS也是使用下面文章里的配置，已经在生产上使用，没有什么问题，一周以来，很正常及稳定

背景:

随着你的网站业务量的增长你网站的服务器压力越来越大？需要负载均衡方案！商业的硬件如F5又太贵，你们又是创业型互联公司如何有效节约成本，节省不必要的浪费？同时实现商业硬件一样的高性能高可用的功能？有什么好的负载均衡可伸张可扩展的方案吗？答案是肯定的！有！我们利用LVS+Keepalived基于完整开源软件的架构可以为你提供一个负载均衡及高可用的服务器。

 

一． LVS+Keepalived 介绍

1. LVS

LVS是Linux Virtual Server的简写，意即Linux虚拟服务器，是一个虚拟的服务器集群系统。本项目在1998年5月由章文嵩博士成立，是中国国内最早出现的项目之一。目前有三种IP负载均衡技术（VS/NAT、VS/TUN和VS/DR）；

八种调度算法（rr,wrr,lc,wlc,lblc,lblcr,dh,sh）。

 

1. Keepalvied

Keepalived在这里主要用作RealServer的健康状态检查以及LoadBalance主机和BackUP主机之间failover的实现

二. 网站负载均衡拓朴图

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

IP信息列表：

名称	IP
LVS-DR-Master	192.168.1.110
LVS-DR-BACKUP	192.168.1.111
LVS-DR-VIP	192.168.1.113
WEB1-Realserver	192.168.1.112
WEB2-Realserver	192.168.1.115
GateWay	192.168.1.1

 

 

三. 安装LVS和Keepalvied软件包

1. 下载相关软件包

#mkdir /usr/local/src/lvs

#cd /usr/local/src/lvs

#wget 

#wget 

2. 安装LVS和Keepalived

#lsmod |grep ip_vs

#uname -r

2.6.18-53.el5PAE

#ln -s /usr/src/kernels/2.6.18-53.el5PAE-i686/  /usr/src/linux

 

#tar zxvf ipvsadm-1.24.tar.gz

#cd ipvsadm-1.24

#make && make install

#find / -name ipvsadm  # 查看ipvsadm的位置

 

#tar zxvf keepalived-1.1.15.tar.gz

#cd keepalived-1.1.15

#./configure  && make && make install

#find / -name keepalived  # 查看keepalived位置

 

#cp /usr/local/etc/rc.d/init.d/keepalived /etc/rc.d/init.d/

#cp /usr/local/etc/sysconfig/keepalived /etc/sysconfig/

#mkdir /etc/keepalived

#cp /usr/local/etc/keepalived/keepalived.conf /etc/keepalived/

#cp /usr/local/sbin/keepalived /usr/sbin/

#service keepalived start|stop     #做成系统启动服务方便管理.

 

 

四. 配置LVS实现负载均衡

1． LVS-DR，配置LVS脚本实现负载均衡

DR 和REAL SERVER 全部打开IP_FORWARD

编辑/etc/sysctl.conf ,把net.ipv4.ip_forward = 0 修改成net.ipv4.ip_forward = 1，保存退出

执行sysctl –p 使其生效

 

 

 

 

2． 配置Realserver脚本.指上面的WEB1-Realserver,WEB2-Realserver

#vi /usr/local/sbin/realserver.sh

#!/bin/bash

# description: Config realserver lo and apply noarp

#Written by :NetSeek

 

SNS_VIP=192.168.1.113

 

. /etc/rc.d/init.d/functions

 

case “$1″ in

start)

ifconfig lo:0 $SNS_VIP netmask 255.255.255.255 broadcast $SNS_VIP

/sbin/route add -host $SNS_VIP dev lo:0

echo “1″ >/proc/sys/net/ipv4/conf/lo/arp_ignore

echo “2″ >/proc/sys/net/ipv4/conf/lo/arp_announce

echo “1″ >/proc/sys/net/ipv4/conf/all/arp_ignore

echo “2″ >/proc/sys/net/ipv4/conf/all/arp_announce

sysctl -p >/dev/null 2>&1

echo “RealServer Start OK”

 

;;

stop)

ifconfig lo:0 down

route del $SNS_VIP >/dev/null 2>&1

echo “0″ >/proc/sys/net/ipv4/conf/lo/arp_ignore

echo “0″ >/proc/sys/net/ipv4/conf/lo/arp_announce

echo “0″ >/proc/sys/net/ipv4/conf/all/arp_ignore

echo “0″ >/proc/sys/net/ipv4/conf/all/arp_announce

echo “RealServer Stoped”

;;

*)

echo “Usage: $0 {start|stop}”

exit 1

esac

 

exit 0

或者采用secondary ip address方式配置

# vi /etc/sysctl.conf

net.ipv4.conf.lo.arp_ignore = 1
net.ipv4.conf.lo.arp_announce = 2
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2

#sysctl –p

#ip add list 查看是否绑定

1. 启动realserver脚本
2. /root/shell/real start

 

五．利用Keepalvied实现负载均衡和和高可用性

1.配置在主负载均衡服务器上配置keepalived.conf

#vi /etc/keepalived/keepalived.conf

 

! Configuration File for keepalived

 

global_defs {

notification_email {

[e-mail]

}

notification_email_from [e-mail]

smtp_server 127.0.0.1

smtp_connect_timeout 30

router_id LVS_DEVEL

}

 

# 20081013 written by :netseek

# VIP1

vrrp_instance VI_1 {

state BACKUP

nopreempt  //master 设置为BACKUP ，通过priority来判断哪成为主，设置nopreempt 是可以在MASTER 出现问题修复好后不抢用vip，直接充当slave的角色。这样，可以确保session不会丢失

interface eth0

virtual_router_id 100

priority 150   # 备份服务上将150改为100

advert_int 1

authentication {

auth_type PASS

auth_pass 1111

}

virtual_ipaddress {

192.168.1.113    (如果有多个VIP，继续换行填写.)

 

 

}

}

 

virtual_server 192.168.1.113 80 {

delay_loop 6        #（每隔6秒查询realserver 状态）

lb_algo wrr          #（lvs调度算法）

lb_kind DR                 #（采用直接路由方式）

persistence_timeout 60        #(同一IP的连接60秒内被分配到同一台realserver)

protocol TCP                #(用TCP协议检查realserver状态)

real_server 192.168.1.112 80 {

weight 3

TCP_CHECK {

connect_timeout 10

nb_get_retry 3

delay_before_retry 3

connect_port 80

}

}

 

real_server 192.168.1.115 80 {

weight 3

TCP_CHECK {

connect_timeout 10

nb_get_retry 3

delay_before_retry 3

connect_port 80

}

}

 

}

 

2. BACKUP服务器同上配置，先安装lvs再按装keepalived,仍后配置/etc/keepalived/keepalived.conf，BAKCUP 的priority 设置成100 ，不需要设置nopreempt

如下面是我的BAKCUP的配置文件

! Configuration File for keepalived

 

global_defs {

notification_email {

[e-mail]

#   [e-mail]

#   [e-mail]

}

notification_email_from [e-mail]

smtp_server 127.0.0.1

smtp_connect_timeout 30

router_id LVS_DEVEL

}

 

# 20081013 written by :netseek

# VIP1

vrrp_instance VI_1 {

state BACKUP

interface eth0

virtual_router_id 100

priority 100

advert_int 1

authentication {

auth_type PASS

auth_pass 1111

}

virtual_ipaddress {

192.168.1.113

 

}

}

 

virtual_server 192.168.1.113 80 {

delay_loop 6

lb_algo wrr

lb_kind DR

persistence_timeout 60

protocol TCP

real_server 192.168.1.112 80 {

weight 3

TCP_CHECK {

connect_timeout 10

nb_get_retry 3

delay_before_retry 3

connect_port 80

}

}

 

real_server 192.168.1.115 80 {

weight 3

TCP_CHECK {

connect_timeout 10

nb_get_retry 3

delay_before_retry 3

connect_port 80

}

}

 

}

 

 

3,在MASTER 及BACKUP 上启动keepalived

#/etc/init.d/keepalived start  启动keepalived 服务，keepalived就能利用keepalived.conf 配

置文件，实现负载均衡和高可用.

 

4. 查看lvs服务是否正常

#watch ipvsadm –ln

5．停Master服务器的keepalived服务，查看BAKCUP服务器是否能正常接管. 切换测试，现MASTER 关掉，如拔网线，当机，SLAVE会接管

[root@cluster1 keepalived]# ipvsadm -L -n

IP Virtual Server version 1.2.1 (size=4096)

Prot LocalAddress:Port Scheduler Flags

-> RemoteAddress:Port           Forward Weight ActiveConn InActConn

TCP  192.168.1.113:80 wrr persistent 60

-> 192.168.1.115:80             Route   3      0          0

-> 192.168.1.112:80             Route   3      0          0

root@cluster1keepalived]# service keepalived stop

Stopping keepalived:                                       [  OK  ]

[root@cluster2 keepalived]# tail /var/log/messages

Oct 27 15:11:09 cluster1 Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.1.113

Oct 27 15:11:09 cluster1 Keepalived_vrrp: Netlink reflector reports IP 192.168.1.113 added

Oct 27 15:11:09 cluster1 Keepalived_healthcheckers: Netlink reflector reports IP 192.168.1.113 added

Oct 27 15:11:14 cluster1 Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.1.113

Oct 27 15:13:56 cluster1 Keepalived: Terminating on signal

Oct 27 15:13:56 cluster1 Keepalived_vrrp: Terminating VRRP child process on signal

Oct 27 15:13:56 cluster1 Keepalived_vrrp: VRRP_Instance(VI_1) removing protocol VIPs.

Oct 27 15:13:56 cluster1Keepalived_healthcheckers: Netlink reflector reports IP 192.168.1.113 removed

Oct 27 15:13:56 cluster1 Keepalived_healthcheckers: Terminating Healthchecker child process on signal

Oct 27 15:13:56 cluste1 Keepalived: Stopping Keepalived v1.1.15 (07/29,2009)

[root@cluste1 keepalived]# ip add sh

1: lo: mtu 16436 qdisc noqueue

link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

inet 127.0.0.1/8 scope host lo

2: eth0: mtu 1500 qdisc pfifo_fast qlen 1000

link/ether 00:0c:29:0b:ca:18 brd ff:ff:ff:ff:ff:ff

inet 192.168.1.110/24 brd 192.168.1.255 scope global eth0

 

 

查看slave 上的状态

[root@cluster2keepalived]# ip add sh

1: lo: mtu 16436 qdisc noqueue

link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

inet 127.0.0.1/8 scope host lo

2: eth0: mtu 1500 qdisc pfifo_fast qlen 1000

link/ether 00:0c:29:39:c7:8b brd ff:ff:ff:ff:ff:ff

inet 192.168.1.111/24 brd 192.168.1.255 scope global eth0

inet 192.168.1.113/32 scope global eth0  //VIP转移到SLAVE上面来了

 

MASTER启动keepalived

root@cluster1 keepalived]# service keepalived start

Starting keepalived:                                       [  OK  ]

[root@cluster1 keepalived]# tail -f /var/log/messages

Oct 27 15:16:03 cluste1 Keepalived_vrrp: Registering gratutious ARP shared channel

Oct 27 15:16:03 cluster1 Keepalived: Starting VRRP child process, pid=2136

Oct 27 15:16:04 cluster2 Keepalived_healthcheckers: Opening file ‘/etc/keepalived/keepalived.conf’.

Oct 27 15:16:04 cluster1Keepalived_healthcheckers: Configuration is using : 11794 Bytes

Oct 27 15:16:04 cluster1 Keepalived_healthcheckers: Activating healtchecker for service [192.168.1.112:80]

Oct 27 15:16:04 cluster1 Keepalived_healthcheckers: Activating healtchecker for service [192.168.1.115:80]

Oct 27 15:16:04 cluster1 Keepalived_vrrp: Opening file ‘/etc/keepalived/keepalived.conf’.

Oct 27 15:16:04 cluster1 Keepalived_vrrp: Configuration is using : 36181 Bytes

Oct 27 15:16:04 cluster1 Keepalived_vrrp: VRRP_Instance(VI_1) Entering BACKUP STATE

Oct 27 15:16:04 cluster12 Keepalived_vrrp: VRRP sockpool: [ifindex(2), proto(112), fd(8,9)]

 

[root@cluster1 keepalived]# ip add sh //VIP并没有发生切换

1: lo: mtu 16436 qdisc noqueue

link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

inet 127.0.0.1/8 scope host lo

2: eth0: mtu 1500 qdisc pfifo_fast qlen 1000

link/ether 00:0c:29:0b:ca:18 brd ff:ff:ff:ff:ff:ff

inet 192.168.1.110/24 brd 192.168.1.255 scope global eth0

[root@cluster1 keepalived]#

 

 

通过以上测试，MASTER重新启动后，并没有再次成为MASTER，而是充当了SLAVE，这样SESSION会话就不会丢失。这也是设置nopreempt  的作用，记住，两边的state 都要设置BACKUP ，只是MASTER 的priority(优先级)要设置比SLAVE高一些。

#tail –f /var/log/message  监听日志，查看状态。

 

 

 

四．相关参考

1．LVS 基础知识汇总

• LVS的算法介绍               
• 学习LVS的三种转发模式       
• LVS中的IP负载均衡技术       

更多的请到 负载均衡版查看

 

1. Keepalived 相关参考资料。

http://www.keepalived.org/documentation.html