nginx模块yaoweibin-nginx_upstream_check_module

nginx　后端失效管理
   -- 第三方模块测试 yaoweibin-nginx_upstream_check_module-7b0f364

1. 下载 yaoweibin-nginx_upstream_check_module-7b0f364

2. 解压后到nginx源码目录给nginx源码打补丁
cd nginx-0.8.54/
patch -p1 <  ../yaoweibin-nginx_upstream_check_module-7b0f364/check.patch

3.　编译nginx
./configure  
--prefix=/usr  
--sbin-path=/usr/sbin/nginx  
--conf-path=/etc/nginx/nginx.conf  
--error-log-path=/var/log/nginx/error.log  
--http-log-path=/var/log/nginx/access.log  
--pid-path=/var/run/nginx/nginx.pid   
--lock-path=/var/lock/nginx.lock  
--user=nginx  
--group=nginx
--with-http_ssl_module  
--with-http_flv_module  
--with-http_gzip_static_module  
--with-http_realip_module  
--with-http_stub_status_module  
--http-client-body-temp-path=/dev/shm/nginx/client/  
--http-proxy-temp-path=/dev/shm/nginx/proxy/  
--http-fastcgi-temp-path=/dev/shm/nginx/fcgi/  
--with-md5-asm
--with-md5=/usr/include  
--with-sha1-asm  
--with-sha1=/usr/include  
--add-module=../ngx_cache_purge-1.2 --add-module=../yaoweibin-nginx_upstream_check_module-7b0f364/

ngx_cache_purge-1.2 也要将模块下载解压

make

make install
　

4.　配置 /etc/nginx/nginx.conf 
主要几点，
a. timeout设置　前后端相配合
b.　压缩
c. 　缓存静态文件缓存于内存
d.　 log_format 方便被awstat解析的标准日志格式
e.　隐藏版本号
f. 　access log与cronolog相配合，自动分割每天的日志

参考了网络资料和官方手册以后，我的/etc/nginx/nginx.conf内容:
user      nginx nginx;
worker_processes  8;
worker_cpu_affinity 00000001 00000010 00000100 00001000 00010000 00100000 01000000 10000000;
error_log  /var/log/nginx/error.log;
pid        /var/run/nginx/nginx.pid;
worker_rlimit_nofile 65535;
 
events {
    use epoll;
    worker_connections 65535;
 }
 
http {
keepalive_timeout  180;
sendfile     on;
tcp_nopush   on;
tcp_nodelay  on;
client_header_buffer_size 4k;
open_file_cache max=102400 inactive=20s;
open_file_cache_valid 30s;
open_file_cache_min_uses 1;

gzip              on;
gzip_min_length   1000;
gzip_buffers      4 8k;
gzip_types    text/* text/css application/javascript application/x-javascript;
gzip_comp_level   9;
gzip_vary         on;
gzip_http_version 1.1;


include    mime.types;
include    fastcgi.conf;
default_type application/octet-stream;
log_format   main '$remote_addr - $remote_user [$time_local]  $status '
    '"$request" $body_bytes_sent "$http_referer" '
    '"$http_user_agent" "$http_x_forwarded_for"';
   
access_log   /var/log/nginx/access.log  main;

server_names_hash_bucket_size 128;

proxy_cache_path /dev/shm/nginx/proxy_cache levels=1:2 keys_zone=pnc:128m inactive=7d max_size=3m ;

proxy_redirect          off;
proxy_set_header        Host            $host;
proxy_set_header        X-Real-IP       $remote_addr;
proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Accept-Encoding "";

client_max_body_size    10m;
client_body_buffer_size 128k;

proxy_connect_timeout   90;
proxy_send_timeout      90;
proxy_read_timeout      90;
proxy_buffers           32 4k;

proxy_cache_key "$scheme://$host$request_uri";
proxy_cache_valid 200 302 1h;
proxy_cache_valid 301 1d;
proxy_cache_valid any 1m;


upstream one_server {
 server 10.0.0.198:80 weight=5;

 server 173.6.6.8:80 weight=1 backup;
 check interval=30000 rise=2 fall=3 timeout=5000 type=tcp;
}

upstream both_server {
 server 10.0.0.194:80 weight=5;
 server 10.0.0.198:80 weight=5;
 server 173.6.6.6:80 weight=1 backup;
 server 173.6.6.8:80 weight=1 backup;
 check interval=30000 rise=2 fall=3 timeout=5000 type=tcp;
}

include sites-enabled/*;

}

mime.types配置：基本保持默认设置

fastcgi.conf配置：
fastcgi_connect_timeout 120s;
fastcgi_send_timeout 120s;
fastcgi_read_timeout 120s;
fastcgi_buffer_size 128k;
fastcgi_buffers 8 128k;
fastcgi_busy_buffers_size 256k;
fastcgi_temp_file_write_size 256k;
fastcgi_intercept_errors on;

fastcgi_param  SCRIPT_FILENAME    $document_root$fastcgi_script_name;
fastcgi_param  QUERY_STRING       $query_string;
fastcgi_param  REQUEST_METHOD     $request_method;
fastcgi_param  CONTENT_TYPE       $content_type;
fastcgi_param  CONTENT_LENGTH     $content_length;
fastcgi_param  SCRIPT_NAME        $fastcgi_script_name;
fastcgi_param  REQUEST_URI        $request_uri;
fastcgi_param  DOCUMENT_URI       $document_uri;
fastcgi_param  DOCUMENT_ROOT      $document_root;
fastcgi_param  SERVER_PROTOCOL    $server_protocol;
fastcgi_param  GATEWAY_INTERFACE  CGI/1.1;
#fastcgi_param  SERVER_SOFTWARE    nginx/$nginx_version;
fastcgi_param  SERVER_SOFTWARE    nginx;
fastcgi_param  REMOTE_ADDR        $remote_addr;
fastcgi_param  REMOTE_PORT        $remote_port;
fastcgi_param  SERVER_ADDR        $server_addr;
fastcgi_param  SERVER_PORT        $server_port;
fastcgi_param  SERVER_NAME        $server_name;
 
fastcgi_index  index.php;
 
fastcgi_param  REDIRECT_STATUS    200;

sites-enabled各个域配置：
# cat sites-enabled/maindomain.com
server {
 listen       75.6.6.8:80;
 server_name   user.maindoamin.com;
 access_log   /var/log/nginx/access_log_pipe  main;
 
# serve static files
location ~* \.(jpg|png|gif|jpeg|css|js|mp3|wav|swf|mov|doc|pdf|xls|ppt|docx|pptx|xlsx|ico)$ {
proxy_pass     
proxy_cache pnc;
proxy_temp_path /dev/shm/nginx/proxy_temp2;
}

location ~.*\.(php|jsp|cgi)?$
{
 proxy_pass
}
location ~ /purge(/.*) {
allow 127.0.0.1;
deny all;
proxy_cache_purge pnc $1$is_args$args;
}
location /nginx_status {
stub_status on;
access_log off;
allow 127.0.0.1;
deny all;
}
 
# pass requests for dynamic content to rails/turbogears/zope, et al
location / {
 proxy_pass     
}
}

b. 内核网络参数优化
 /etc/sysctl.conf
net.ipv4.tcp_max_tw_buckets = 60000
net.ipv4.ip_local_port_range = 1024    65000
net.ipv4.tcp_tw_recycle = 1
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_syncookies = 1
net.core.somaxconn = 262144
net.core.netdev_max_backlog = 262144
net.ipv4.tcp_max_orphans = 262144
net.ipv4.tcp_max_syn_backlog = 262144
net.ipv4.tcp_timestamps = 0
net.ipv4.tcp_synack_retries = 1
net.ipv4.tcp_syn_retries = 1
net.ipv4.tcp_fin_timeout = 1
net.ipv4.tcp_keepalive_time = 30

sysctl -p

5.　启动
mkfifo /var/log/nginx/access_log_pipe
nginx -t　＃检查语法

adm_nginx_daemon.sh
#!/bin/bash
mkdir -p /dev/shm/nginx/proxy_cache
chown nginx.nginx -R /dev/shm/nginx/  /var/log/nginx/
ulimit -SHn 80000
sleep 1
nohup /usr/sbin/cronolog /var/log/nginx/user.maindomain.com.access_%Y%m%d.log < /var/log/nginx/access_log_pipe &
nohup nginx

重启 nginx -s reload

6. 模块介绍: yaoweibin-nginx_upstream_check_module-7b0f364
新参数:
check interval=30000 rise=2 fall=3 timeout=5000 type=tcp;

每30秒检测一次，超过3次失败则标记服务器状态为down。超过2次成功则将服务器状态为up,检测方式为tcp。
测试结果：
当服务器标记为down以后，所有请求向另一个后端机器转移。
当服务器标记为up以后，服务器回到正常的负载均衡。

nginx性能
静态:　httperf 获取1.8KB图片文件, 可达到 1w以上并发数，cpu/mem都非常小，netstat -an占用连接数也非常少。Nginx前端外网网卡发送速度可达到50MB/s

动态: 450~500并发数 ,严重受制于目前的db访问速度