keepalived+nginx -liubing35088-ChinaUnix博客

nginx+keepalived高可用性负载均衡

一、前言

最近因业务扩展，需要将当前的apache 转为nginx(web), 再在web前端放置nginx(负载均衡)。同时结合keepalived 对前端nginx实现HA。

nginx进程基于于Master+Slave(worker)多进程模型，自身具有非常稳定的子进程管理功能。在Master进程分配模式下，Master进程永远不进行业务处理，只是进行任务分发，从而达到Master进程的存活高可靠性，Slave(worker)进程所有的业务信号都由主进程发出，Slave(worker)进程所有的超时任务都会被Master中止，属于非阻塞式任务模型。

Keepalived是Linux下面实现VRRP 备份路由的高可靠性运行件。基于Keepalived设计的服务模式能够真正做到主服务器和备份服务器故障时IP瞬间无缝交接。二者结合，可以构架出比较稳定的软件lb方案。

二、nginx+keepalived架构实战

采用两台服务器做nginx主备，后端采用两个realserver（可以随意扩展）。

1、结构如下：

nginx master 172.31.1.100

nginx backup 172.31.1.135

vip 172.31.1.110

2、安装配置

(1)、安装keepalived（在nginx的mater和backup都安装）

wget
tar zxvf keepalived-1.1.19.tar.gz
cd keepalived-1.1.19
./configure --prefix=/usr/local/keepalived
make
make install
cp /usr/local/keepalived/sbin/keepalived /usr/sbin/
cp /usr/local/keepalived/etc/sysconfig/keepalived /etc/sysconfig/
cp /usr/local/keepalived/etc/rc.d/init.d/keepalived /etc/init.d/
mkdir /etc/keepalived
cd /etc/keepalived/

配置nginx master的keepalived配置文件

[root@zhang1 keepalived]# cat /etc/keepalived/keepalived.conf
############################################################

vrrp_script     chk_http_port {
script "/etc/keepalived/nginx_pid.sh"        /* 监控脚本 */
interval 2                                                    /* 监控时间间隔 */
weight 2
}

global_defs {
        notification_email {

        }
        notification_email_from
        smtp_server 172.31.1.101
        smtp_connect_timeout 30
        router_id LVS_Twioo
}

vrrp_instance VI_1 {
        state MASTER                        /* 设置为主 */
        interface eth0                           /* 监控网卡 */
        virtual_router_id 51                   /* 这个两台服务器必须一样 */
        mcast_src_ip 172.31.1.100    /* 多播的源IP，设置为本机IP，与VIP同一网卡*/
        priority 100                                /* 权重值 MASTRE 一定要高于 BAUCKUP */
        advert_int 1                                /* VRRP Multicast广播周期秒数*/
        smtp_alert

        authentication {
                auth_type   PASS            /* 加密 */
                auth_pass 123456          /* 加密的密码，两台服务器一定要一样，不然会出错 */
        }

track_script {
chk_http_port /* 执行监控的服务 */
}

virtual_ipaddress {
172.31.1.110
}
}
############################################################

配置nginx backup的keepalived配置文件

[root@zhang1 keepalived]# cat /etc/keepalived/keepalived.conf
############################################################

vrrp_script       chk_http_port {
script "/etc/keepalived/nginx_pid.sh"
interval 2
weight 2
}

global_defs {
        notification_email {

        }
        notification_email_from
        smtp_server 172.31.1.101
        smtp_connect_timeout 30
        router_id LVS_Twioo
}

vrrp_instance VI_1 {
        state BACKUP
        interface eth0
        virtual_router_id 51
mcast_src_ip 172.31.1.100
        priority 80
        advert_int 1
        smtp_alert

        authentication {
                auth_type PASS
                auth_pass 123456
        }

track_script {
chk_http_port
}

        virtual_ipaddress {
                172.31.1.110
        }

}

启动keepalived，查看虚拟IP是否绑定！

[root@zhang1 ~]# /etc/rc.d/init.d/keepalived start
Starting keepalived:                                       [ OK ]
1: lo: mtu 16436 qdisc noqueue
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: mtu 1500 qdisc pfifo_fast qlen 1000
    link/ether 00:26:55:1a:e5:28 brd ff:ff:ff:ff:ff:ff
    inet 172.31.1.100/24 brd 172.31.1.255 scope global eth0
    inet 172.31.1.110/32 scope global eth0
    inet6 fe80::226:55ff:fe1a:e528/64 scope link
       valid_lft forever preferred_lft forever
3: eth1: mtu 1500 qdisc pfifo_fast qlen 1000
    link/ether 00:26:55:1a:e5:2a brd ff:ff:ff:ff:ff:ff
4: sit0: mtu 1480 qdisc noop
    link/sit 0.0.0.0 brd 0.0.0.0

可以看到多了一个虚拟IP

查看master keepalived日志 tailf /var/log/message

Jul 23 14:50:38 nginx Keepalived_vrrp: Registering Kernel netlink command channel
Jul 23 14:50:38 nginx Keepalived_vrrp: Registering gratutious ARP shared channel
Jul 23 14:50:38 nginx Keepalived_vrrp: Opening file '/etc/keepalived/keepalived.conf'.
Jul 23 14:50:38 nginx Keepalived_vrrp: Configuration is using : 38512 Bytes
Jul 23 14:50:38 nginx Keepalived_vrrp: Using LinkWatch kernel netlink reflector...
Jul 23 14:50:38 nginx Keepalived_vrrp: VRRP sockpool: [ifindex(2), proto(112), fd(9,10)]
Jul 23 14:50:39 nginx Keepalived_vrrp: VRRP_Instance(VI_1)Transition to MASTER STATE
Jul 23 14:50:40 nginx Keepalived_vrrp: VRRP_Instance(VI_1) Entering MASTER STATE
Jul 23 14:50:40 nginx Keepalived_vrrp: VRRP_Instance(VI_1) setting protocol VIPs.
Jul 23 14:50:40 nginx Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 172.31.1.110
Jul 23 14:50:40 nginx Keepalived_vrrp: Remote SMTP server [172.31.1.101:25] connected.
Jul 23 14:50:40 nginx Keepalived_vrrp: Remote SMTP server [172.31.1.101:25] connected.
Jul 23 14:50:40 nginx Keepalived_vrrp: SMTP alert successfully sent.

可以看出状态为MASTER

此时在BACKUP机上启动keepalived，再查看日志

Jul 23 14:58:30 SvrTest Keepalived_vrrp: Opening file '/etc/keepalived/keepalived.conf'.
Jul 23 14:58:30 SvrTest Keepalived_vrrp: Configuration is using : 65052 Bytes
Jul 23 14:58:30 SvrTest Keepalived_vrrp: Using LinkWatch kernel netlink reflector...
Jul 23 14:58:30 SvrTest Keepalived_vrrp: VRRP_Instance(VI_1) Entering BACKUP STATE
Jul 23 14:58:30 SvrTest Keepalived_vrrp: VRRP sockpool: [ifindex(2), proto(112), fd(9,10)]
Jul 23 14:58:30 SvrTest Keepalived_vrrp: Remote SMTP server [172.31.1.101:25] connected.
Jul 23 14:58:30 SvrTest Keepalived_vrrp: SMTP alert successfully sent.
Jul 23 14:58:30 SvrTest Keepalived_vrrp: SMTP alert successfully sent.

此时如果停止MASTER上的keepalived ,那么在BACKUP的日志会出现

Jul 23 14:59:45 SvrTest Keepalived_vrrp: VRRP_Instance(VI_1)Transition to MASTER STATE
Jul 23 14:59:46 SvrTest Keepalived_vrrp: VRRP_Instance(VI_1)Entering MASTER STATE
Jul 23 14:59:46 SvrTest Keepalived_vrrp: VRRP_Instance(VI_1) setting protocol VIPs.
Jul 23 14:59:46 SvrTest Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 172.31.1.110
Jul 23 14:59:46 SvrTest avahi-daemon[3160]: Registering new address record for 172.31.1.110 on eth0.
Jul 23 14:59:46 SvrTest Keepalived_vrrp: Remote SMTP server [172.31.1.101:25] connected.
Jul 23 14:59:46 SvrTest Keepalived_vrrp: SMTP alert successfully sent.

此时BACKUP接管了VIP

若再启动MASTER上的keepalived,BACKUP上的日志会出现：

Jul 23 14:56:37 SvrTest Keepalived_vrrp: VRRP_Instance(VI_1) Received higher prio advert
Jul 23 14:56:37 SvrTest Keepalived_vrrp: VRRP_Instance(VI_1) Entering BACKUP STATE
Jul 23 14:56:37 SvrTest Keepalived_vrrp: VRRP_Instance(VI_1) removing protocol VIPs.
Jul 23 14:56:37 SvrTest Keepalived_vrrp: Remote SMTP server [172.31.1.101:25] connected.
Jul 23 14:56:37 SvrTest avahi-daemon[3160]: Withdrawing address record for 172.31.1.110 on eth0.
Jul 23 14:56:37 SvrTest Keepalived_vrrp: SMTP alert successfully sent.

则是说明有更高优先级的机器即MASTER启动，去除VIP，还给更优先级的机器

这样就实现了如果某一主机宕机，IP自动切换到备机上继续运行

################################### nginx_pid.sh 脚本

[root@SvrTest keepalived]# cat /etc/keepalived/nginx_pid.sh

#!/bin/bash

NGINX_PROCESS=`ps -C nginx --no-header | wc -l`

if [ $NGINX_PROCESS -eq 0 ];then

/usr/local/nginx/sbin/nginx

sleep 3

if [ `ps -C nginx --no-header | wc -l` -eq 0 ];then

/etc/init.d/keepalived stop

说明：

当主nginx 宕机后，会自动切换到从nginx上，这个时候可以配合以个报警的脚不来通告运维人员

当主修复好后，需要手动启动keepalived 程序，以便将服务重新切换回主服务器。

报警脚步我抽个时间补上。

keepalived+nginx 高可用性+负载均衡