haproxy + keepalived “一键安装包” 配置与测试HOWTO

haproxy + keepalived “一键安装包” 配置与测试HOWTO

2011-12-29 09:24:42

标签：负载均衡 集群 高可用 keepalived haproxy

版权声明：原创作品，如需转载，请与作者联系。否则将追究法律责任。

本文为  “haproxy + keepalived 一键安装包”  的 第四篇

目的:帮助快速设置haproxy 与 keepalived 配置文档

第一篇 haproxy 安装与配置

第二篇 keepalived 安装与配置

第三篇 haproxy + keepalived 一键安装包(keepalived 为haproxy 做 HA 高可用）

第四篇 haproxy + keepalived “一键安装包”  配置与测试HOWTO

 

一 haproxy.sh 与 keepalived.sh 服务脚本 

	
[root@four cluster]# ls /usr/local/sbin/*.ah
haproxy.sh  keepalived.sh

 

二 设置自动启动 haproxy.sh 与 keepalived.sh

	
[root@four cluster]# cat /etc/rc.local
#!/bin/sh
#
# This script will be executed *after* all the other init scripts.
# You can put your own initialization stuff in here if you don't
# want to do the full Sys V style init stuff.

touch /var/lock/subsys/local
ulimit -SHn 65535

/usr/local/sbin/haproxy.sh start
/usr/local/sbin/keepalived.sh start

 

三 haproxy 相关配置

	
3.1 haroxy 程序目录
#ls /usr/local/haproxy
doc  haproxy.cfg  html  logs  sbin  share

3.2 haproxy 配置文档
[root@four cluster]# file /usr/local/haproxy/haproxy.cfg
/usr/local/haproxy/haproxy.cfg: UTF-8 Unicode text

3.3 编辑haproxy 配置文档
[root@four cluster]# vim /usr/local/haproxy/haproxy.cfg
第37 行 定义健康检查html文档，所以你需要在你的web服务上建立check.html文件用于haproxy
健康检查，也可以跟具需要自定义检查文档。
option  httpchk HEAD /check.html #用来做健康检查html文档
第38 39 行修10.0.1.252/253 替换成你的web 服务器ip地址
server server1 10.0.1.252:80 cookie server1 check inter 2000 rise 3     fall 3 weight 3
server server2 10.0.1.253:80 cookie server2 check inter 2000 rise 3     fall 3 maxconn 120 weight 3

3.4 启动haproxy 输入如下命令
[root@four cluster]# haproxy.sh start

3.5 检查是否有haproxy 进程
[root@four cluster]# ps aux | grep haproxy | grep -v 'grep'
nobody    4216  0.0  0.2   7436  1108 ?        Ss   20:35   0:00 /usr/local/haproxy/sbin/haproxy -f /usr/local/haproxy/haproxy.cfg

3.6 检查haproxy 是否监听80 端口
[root@four cluster]# lsof -i :80
COMMAND  PID   USER   FD   TYPE DEVICE SIZE NODE NAME
haproxy 4216 nobody    4u  IPv4   8229       TCP *:http (LISTEN)

3.7 简单测试 haproxy 看看否返回 ’200 OK‘
[root@four cluster]# curl -IL (换成你的ip )
HTTP/1.1 200 OK
Server: nginx/1.0.5
Date: Wed, 28 Dec 2011 12:42:30 GMT
Content-Type: text/html
Content-Length: 159
Last-Modified: Sat, 03 Dec 2011 11:36:10 GMT
Connection: close
Accept-Ranges: bytes

 

四 keepalived 相关配置

	
4.1 keepalived 程序目录
[root@four cluster]# ls /usr/local/keepalived
bin  etc  haproxy_check.sh  keepalived.conf  sbin  share

4.2 haproxy_check.sh shell脚本监控程序,用于检查 80端口，来判断haproxy 是否可用
[root@four cluster]# file /usr/local/keepalived/haproxy_check.sh
/usr/local/keepalived/haproxy_check.sh: Bourne-Again shell script text executable

4.3 keepalived 配置文档
[root@four cluster]# file /usr/local/keepalived/keepalived.conf
/usr/local/keepalived/keepalived.conf: ASCII text

4.4 编辑keepalived 配置文档
[root@four cluster]# vim /usr/local/keepalived/keepalived.conf
第16 行 interface 默认值 eth1 ,改成你的haproxy 外网卡(公网ip网卡)
interface eth0

第 34 行 默认值 10.0.1.250/24 dev eth1 ，10.0.1.250 替换成你virtual ip 地址,dev eth1 改成你的haproxy 外网卡(与第16行一致)
10.0.1.250/24 dev eth0 scope global

4.5 启动keepalived 输入如下命令
[root@four cluster]# keepalived.sh start
[root@four cluster]# 4242                #此处返回的是keepalived 进程id，敲下回车键即可

4.6 检查是否有keepalived 进程
# ps aux | grep keepalived | grep -v 'grep'
root      4238  0.0  0.0  63844   464 pts/0    S    20:52   0:00 /bin/bash /usr/local/sbin/keepalived.sh start
root      4240  0.0  0.1  35776   608 ?        Ss   20:52   0:00 /usr/local/keepalived/sbin/keepalived -f /usr/local/keepalived/keepalived.conf
root      4241  0.0  0.1  39936  1048 ?        S    20:52   0:00 /usr/local/keepalived/sbin/keepalived -f /usr/local/keepalived/keepalived.conf
root      4242  0.0  0.2  63848  1152 pts/0    S    20:52   0:00 /bin/bash /usr/local/keepalived/haproxy_check.sh start

4.7 查看keepalived 输出日志
[root@four cluster]# tail /var/log/messages
Dec 28 20:52:28 four Keepalived_vrrp: Registering Kernel netlink reflector
Dec 28 20:52:28 four Keepalived_vrrp: Registering Kernel netlink command channel
Dec 28 20:52:28 four Keepalived_vrrp: Registering gratutious ARP shared channel
Dec 28 20:52:28 four Keepalived: Starting VRRP child process, pid=4241
Dec 28 20:53:08 four Keepalived_vrrp: Opening file '/usr/local/keepalived/keepalived.conf'.
Dec 28 20:53:08 four Keepalived_vrrp: Configuration is using : 63577 Bytes
Dec 28 20:53:08 four Keepalived_vrrp: Using LinkWatch kernel netlink reflector...
Dec 28 20:53:08 four Keepalived_vrrp: VRRP_Instance(VI_1) Entering BACKUP STATE
Dec 28 20:53:12 four Keepalived_vrrp: VRRP_Instance(VI_1) Transition to MASTER STATE
Dec 28 20:53:13 four Keepalived_vrrp: VRRP_Instance(VI_1) Entering MASTER STATE              #当前keepalived 状态为master

 

五 haproxy 与 keepalived 整体测试

	
5.1 查看 haproxy 与 keepalived 进程树
[root@four cluster]# pstree
init─┬─crond
     ├─events/0
     ├─events/1
     ├─haproxy                                      #haproxy 进程
     ├─keepalived───keepalived                      #keepalived 进程(2个)
     ├─keepalived.sh───haproxy_check.s───sleep      #keepalived.sh 调用 haproxy_check.sh (监控脚本) 调用 sleep （用于控检查间隔时间,2s）


5.2 模拟网络故障 haproxy 切换测试
haproxy 1 日志
[root@one ~]# tail /var/log/messages -f
Dec 28 21:09:43 one kernel: NFSD: starting 90-second grace period
Dec 28 21:09:44 one Keepalived: Starting Keepalived v1.2.2 (12/25,2011)
Dec 28 21:09:44 one Keepalived_vrrp: Registering Kernel netlink reflector
Dec 28 21:09:44 one Keepalived_vrrp: Registering Kernel netlink command channel
Dec 28 21:09:44 one Keepalived_vrrp: Registering gratutious ARP shared channel
Dec 28 21:09:44 one Keepalived: Starting VRRP child process, pid=1993
Dec 28 21:10:24 one Keepalived_vrrp: Opening file '/usr/local/keepalived/keepalived.conf'.
Dec 28 21:10:24 one Keepalived_vrrp: Configuration is using : 63597 Bytes
Dec 28 21:10:24 one Keepalived_vrrp: Using LinkWatch kernel netlink reflector...
Dec 28 21:10:24 one Keepalived_vrrp: VRRP_Instance(VI_1) Entering BACKUP STATE

haproxy 2 日志
[root@four cluster]# ifdown eth0
[root@four cluster]# tail /var/log/messages
Dec 28 20:52:28 four Keepalived: Starting VRRP child process, pid=4241
Dec 28 20:53:08 four Keepalived_vrrp: Opening file '/usr/local/keepalived/keepalived.conf'.
Dec 28 20:53:08 four Keepalived_vrrp: Configuration is using : 63577 Bytes
Dec 28 20:53:08 four Keepalived_vrrp: Using LinkWatch kernel netlink reflector...
Dec 28 20:53:08 four Keepalived_vrrp: VRRP_Instance(VI_1) Entering BACKUP STATE
Dec 28 20:53:12 four Keepalived_vrrp: VRRP_Instance(VI_1) Transition to MASTER STATE
Dec 28 20:53:13 four Keepalived_vrrp: VRRP_Instance(VI_1) Entering MASTER STATE
Dec 28 21:12:07 four Keepalived_vrrp: Kernel is reporting: interface eth0 DOWN
Dec 28 21:12:07 four Keepalived_vrrp: VRRP_Instance(VI_1) Entering FAULT STATE
Dec 28 21:12:07 four Keepalived_vrrp: VRRP_Instance(VI_1) Now in FAULT state                 #keepalived fault 状态

haproxy 1 日志
[root@one ~]# tail /var/log/messages -f
Dec 28 21:09:43 one kernel: NFSD: starting 90-second grace period
Dec 28 21:09:44 one Keepalived: Starting Keepalived v1.2.2 (12/25,2011)
Dec 28 21:09:44 one Keepalived_vrrp: Registering Kernel netlink reflector
Dec 28 21:09:44 one Keepalived_vrrp: Registering Kernel netlink command channel
Dec 28 21:09:44 one Keepalived_vrrp: Registering gratutious ARP shared channel
Dec 28 21:09:44 one Keepalived: Starting VRRP child process, pid=1993
Dec 28 21:10:24 one Keepalived_vrrp: Opening file '/usr/local/keepalived/keepalived.conf'.
Dec 28 21:10:24 one Keepalived_vrrp: Configuration is using : 63597 Bytes
Dec 28 21:10:24 one Keepalived_vrrp: Using LinkWatch kernel netlink reflector...
Dec 28 21:10:24 one Keepalived_vrrp: VRRP_Instance(VI_1) Entering BACKUP STATE
Dec 28 21:12:08 one Keepalived_vrrp: VRRP_Instance(VI_1) Transition to MASTER STATE
Dec 28 21:12:09 one Keepalived_vrrp: VRRP_Instance(VI_1) Entering MASTER STATE             #成功切换为 master

haproxy 1 检查vip
[root@one ~]# ip add
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000
    link/ether 08:00:27:74:3c:05 brd ff:ff:ff:ff:ff:ff
    inet 10.0.2.251/24 brd 10.0.2.255 scope global eth0
    inet6 fe80::a00:27ff:fe74:3c05/64 scope link
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000
    link/ether 08:00:27:3e:9a:2b brd ff:ff:ff:ff:ff:ff
    inet 10.0.1.251/24 brd 10.0.1.255 scope global eth1
    inet 10.0.1.250/24 scope global secondary eth1  #virtual ip
    inet6 fe80::a00:27ff:fe3e:9a2b/64 scope link
       valid_lft forever preferred_lft forever
4: sit0: <NOARP> mtu 1480 qdisc noop
    link/sit 0.0.0.0 brd 0.0.0.0

5.3 模拟haproxy 80端口故障 keepalived 切换测试
haproxy 2 恢复
[root@four cluster]# ifup eth0
haproxy 2 日志
[root@four cluster]# tail /var/log/messages -f
Dec 28 20:53:08 four Keepalived_vrrp: Configuration is using : 63577 Bytes
Dec 28 20:53:08 four Keepalived_vrrp: Using LinkWatch kernel netlink reflector...
Dec 28 20:53:08 four Keepalived_vrrp: VRRP_Instance(VI_1) Entering BACKUP STATE
Dec 28 20:53:12 four Keepalived_vrrp: VRRP_Instance(VI_1) Transition to MASTER STATE
Dec 28 20:53:13 four Keepalived_vrrp: VRRP_Instance(VI_1) Entering MASTER STATE
Dec 28 21:12:07 four Keepalived_vrrp: Kernel is reporting: interface eth0 DOWN
Dec 28 21:12:07 four Keepalived_vrrp: VRRP_Instance(VI_1) Entering FAULT STATE
Dec 28 21:12:07 four Keepalived_vrrp: VRRP_Instance(VI_1) Now in FAULT state
Dec 28 21:15:34 four kernel: e1000: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
Dec 28 21:15:35 four Keepalived_vrrp: VRRP_Instance(VI_1) Entering BACKUP STATE


haproxy 1 进程
[root@one ~]# pstree
init─┬─crond
     ├─dbus-daemon
     ├─events/0
     ├─haproxy
     ├─keepalived───keepalived
     ├─keepalived.sh───haproxy_check.s───sleep

停止 haproxy 进程
[root@one ~]# killall haproxy                                        #停止haproxy 进程
[root@one ~]# tail /usr/local/keepalived/fault.log                   #fault log
 2011-12-28_21:17:50
 2011-12-28_21:17:51
 2011-12-28_21:17:52
 2011-12-28_21:17:53
 2011-12-28_21:17:54
 2011-12-28_21:17:55
 2011-12-28_21:17:56
 2011-12-28_21:17:57

haproxy 1 日志
[root@one ~]# tail /var/log/messages
Dec 28 21:12:09 one Keepalived_vrrp: VRRP_Instance(VI_1) Entering MASTER STATE
Dec 28 21:17:58 one Keepalived: Terminating on signal
Dec 28 21:17:58 one Keepalived: Stopping Keepalived v1.2.2 (12/25,2011)                   #keepalived 被haproxy_check.sh 监控脚本关闭
Dec 28 21:17:58 one Keepalived_vrrp: Terminating VRRP child process on signal
Dec 28 21:17:58 one kernel: ADDRCONF(NETDEV_UP): eth0: link is not ready
Dec 28 21:17:58 one kernel: e1000: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
Dec 28 21:17:58 one kernel: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
Dec 28 21:18:00 one kernel: ADDRCONF(NETDEV_UP): eth1: link is not ready
Dec 28 21:18:00 one kernel: e1000: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
Dec 28 21:18:00 one kernel: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready


haproxy 2 日志
[root@four cluster]# tail /var/log/messages -f
Dec 28 20:53:08 four Keepalived_vrrp: Configuration is using : 63577 Bytes
Dec 28 20:53:08 four Keepalived_vrrp: Using LinkWatch kernel netlink reflector...
Dec 28 20:53:08 four Keepalived_vrrp: VRRP_Instance(VI_1) Entering BACKUP STATE
Dec 28 20:53:12 four Keepalived_vrrp: VRRP_Instance(VI_1) Transition to MASTER STATE
Dec 28 20:53:13 four Keepalived_vrrp: VRRP_Instance(VI_1) Entering MASTER STATE
Dec 28 21:12:07 four Keepalived_vrrp: Kernel is reporting: interface eth0 DOWN
Dec 28 21:12:07 four Keepalived_vrrp: VRRP_Instance(VI_1) Entering FAULT STATE
Dec 28 21:12:07 four Keepalived_vrrp: VRRP_Instance(VI_1) Now in FAULT state
Dec 28 21:15:34 four kernel: e1000: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
Dec 28 21:15:35 four Keepalived_vrrp: VRRP_Instance(VI_1) Entering BACKUP STATE
Dec 28 21:18:02 four Keepalived_vrrp: VRRP_Instance(VI_1) Transition to MASTER STATE
Dec 28 21:18:03 four Keepalived_vrrp: VRRP_Instance(VI_1) Entering MASTER STATE          #keepalived 状态为 master

[root@four cluster]# ip add
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000
    link/ether 08:00:27:27:ef:a6 brd ff:ff:ff:ff:ff:ff
    inet 10.0.2.254/24 brd 10.0.2.255 scope global eth0
    inet6 fe80::a00:27ff:fe27:efa6/64 scope link
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000
    link/ether 08:00:27:cd:f3:db brd ff:ff:ff:ff:ff:ff
    inet 10.0.1.254/24 brd 10.0.1.255 scope global eth1
    inet 10.0.1.250/24 scope global secondary eth1  #virtual ip
    inet6 fe80::a00:27ff:fecd:f3db/64 scope link
       valid_lft forever preferred_lft forever
4: sit0: <NOARP> mtu 1480 qdisc noop
    link/sit 0.0.0.0 brd 0.0.0.0

 

六 haproxy_check.sh 监控脚本

	
脚本默认监控 eth1 网卡，所以你需要根据自己的需要调整，如果你haproxy virtual ip 在eth0 网卡，那么你需要做如下更改:
6.1 使用"#"号注释掉 第14行 第16行，取消掉 第15行 第17行
#eth1_ip=$(/sbin/ifconfig eth1  | sed -n '/inet addr:/ s/inet addr://pg' | awk -F" " '{print $1}')
eth0_ip=$(/sbin/ifconfig eth0  | sed -n '/inet addr:/ s/inet addr://pg' | awk -F" " '{print $1}')
#eth1_url="{eth1_ip}:80"
eth0_url="{eth0_ip}:80"

6.2 将第35行 $eth1_url,替换成 $eth0_url
if curl -IL -A "haproxy_check" "$eth1_url" > /dev/null 2>&1 ;then
6.3 将第48行 $eth1_url，替换成 $eth0_url
if ! curl -IL -A "haproxy_check" "$eth1_url" > /dev/null 2>&1 ;then

 

七 其它

	
5.2 模拟网络故障 haproxy 切换测试 与 5.3 模拟haproxy 80端口故障 keepalived 切换测试
五 haproxy 与 keepalived 整体测试，反过来测试也是没有问题的，不过要保证 haproxy.sh keepalived.sh 这两个服务脚本已经启动了，可以使用pstree 查看
如：[root@four cluster]# pstree
init─┬─crond
     ├─events/0
     ├─events/1
     ├─haproxy
     ├─keepalived───keepalived
     ├─keepalived.sh───haproxy_check.s───sleep

 

本文出自 “dongnan” 博客，转载请与作者联系！