Web-Proxy Web架构的部署(Nginx proxy)

1 服务器安排

Web服务器（至少2台）

Web Proxy服务器（至少1台）

 

 

2 服务器环境的部署

系统环境：Ubuntu 12.04.3 LTS \n \l  64bits

Web服务器要运行 

 Curl

     GD

     Geoip

     Gettext

     Memcache

     Memcached

     Mysql

     Mysqli

     Redis-Server

     FastDFS_Client

     Sphinx

     MySQL

     Nginx

     Php-Fpm

这些软件其中Ngixn-Memcache  Nginx-Redis Nginx 模块FastDFS_Client_Php  Redis_Php需要源码安装

 

 

安装部署步骤

 

所有的服务安装首选Aptitude默认安装，其次才是编译安装。

 

在Web Web Proxy服务器上部署的：

sudo aptitude update

sudo aptitude  install build-essential automake gcc rsync curl gettext libxml2 screen openssl e2fsprogs autoconf bison mysql-server mysql-client nginx php5-geoip libgeoip1 php5-gd  memcached php5-mysql php5-curl php5-gd php5-intl php-pear php5-imagick php5-imap php5-mcrypt php5-memcache php5-ming php5-ps php5-pspell php5-recode php5-snmp php5-sqlite php5-tidy php5-xmlrpc php5-xsl php5-fpm

 

需要源码安装的有Nginx 及Nginx-Memcache  Nginx-Redis模块

Fast-Client-Php Redis-Php模块

 

相关下载

 

Nginx:

        查看目前服务器相版本，到找到相对应的版本，然后下载备用

 

Ngixn-Memcache模块:

       到下载模块，备用(现在版本是0.14)

 

Nginx-Redis模块:

       到下载模块，备用，(现在版本是0.10)

 

Fast_Client_Php 模块:

       到下载，注意，Php的FastDFS模块是合成是FastDFS服务端中的，所以要下FastDFS服务端（现在版本是FastDFS4.6），另外独立下载FastDFS的Nginx模块(现在版本是V1.15)

 

Redis_Php 模块

到 下载

 

将下载下来的Nginx及模块解压统一放到/usr/loca/src/目录下，如Redis2-Nginx 0.1最新模块解压出来之后是的目录名为Redis2-Nginx-Module-0.10，全路径就是/usr/loca/src/nginx/ redis2-nginx-module-0.10,这个路径一会编译安装时要用到。

 

Nginx的编译安装

开始编译Nginx，由于之前 已经用Aptitude安装过Nginx了，所以我们直接运行

   /etc/nginx/sbin/nginx  –V，查看编译参数及一些自带模块的位置，这些参数可以直接复制给我们来编译Nginx下面我们开始编译Nginx，并带上Memcache,Redis等插件

./configure  --prefix=/etc/nginx --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-client-body-temp-path=/var/lib/nginx/body --http-fastcgi-temp-path=/var/lib/nginx/fastcgi --http-log-path=/var/log/nginx/access.log --http-proxy-temp-path=/var/lib/nginx/proxy --http-scgi-temp-path=/var/lib/nginx/scgi --http-uwsgi-temp-path=/var/lib/nginx/uwsgi --lock-path=/var/lock/nginx.lock --pid-path=/var/run/nginx.pid --without-select_module --without-poll_module --with-http_stub_status_module --with-http_ssl_module --with-pcre=/usr/local/src/pcre-8.31

--add-module=/usr/local/src/memc-nginx-module-0.14

--add-module=/usr/local/src/redis2-nginx-module-0.10 --add-module=/usr/local/src/fastdfs-nginx-module-0.15

--with-http_addition_module --with-http_dav_module --with-http_geoip_module --with-http_gzip_static_module --with-http_image_filter_module --with-http_realip_module --with-http_stub_status_module --with-http_ssl_module --with-http_sub_module --with-http_xslt_module --with-ipv6 --with-sha1=/usr/include/openssl --with-md5=/usr/include/openssl --with-mail --with-mail_ssl_module

 

注意红色字体部分是我们下载的模块，相对应 解压地址，这个地址一定要正确，接着是

make

sudo make install

完成了Nginx及相关插件的安装；

 

 

FastDFS Php模块

把下载的文件如FastDFS_V4.06.Tar.Gz，解压到/usr/local/src，进入目录/usr/local/src/fastdfs/，然后开始安装:

 

先安装FastDFS

 安装的前，先到系统中查找Libpthread.So

 locate  libpthread.so

记录下来这个路径，然后修改一下Make.Sh

vim  /usr/local/src/fastdfs/make.sh，找到TARGET_PREFIX，把它修改成target_prefix=/usr/local/fastdfs

然后继续查找

if  [ -f /usr/lib/libpthread.so ] ||………

把Libpthread.So的路径插进去，像下面这样：

if  [ -f /usr/lib/x86_64-linux-gnu/libpthread.so ] || [ -f /usr/lib/libpthread.so ] ||……

然后开始安装FastDFS:

     ./make.sh

sudo ./make.sh  install

完成FastDFS的安装。

接着进入/Usr/Local/Src/FastDFS/Php_Client目录，继续安装FastDFS的Php插件

phpize

./configure

make

sudo make install

即可完成安装。

 

 

Redis_Php 模块

  cd  /usr/local/src/phpredis-master

phpize

./configure

make

sudo make install

  完成安装

 

安装确认步骤

   Nginx:   /usr/nginx/sbin/nginx –V 在出来的参数中，请确认上边提到的红色字部分

           sudo /etc/init.d/nginx start，确认是否正常，如提示配置文件有错，请对应修改

 

   Php:    Php –M  在出来的模块列表，请确认有FastDFS 及Redis模块

 

   Mysql:  sudo /etc/init.d/mysql start 屏幕是否有出错信息 用netstat –antulp|grep 3306是否侦听在3306端口，如果不是，请根据配置文件找到错误日志，查看修正错误

 

 

 

 

3 相关的配置文件

Nginx

     首先是nginx的部分，位置在/etc/nginx/nginx.conf 以下是它的配置文件：

 

user www-data;

worker_processes 4;

worker_rlimit_nofile 65535;

pid /var/run/nginx.pid;

 

events {

        use epoll;

        worker_connections 65535;

}

 

 

http

{

  include       mime.types;

  default_type  application/octet-stream;

 

  access_log /var/log/nginx/access.log;

  error_log /var/log/nginx/error.log;

 

  server_names_hash_bucket_size    128;

  client_header_buffer_size        32k;

  large_client_header_buffers  4   32k;

  client_max_body_size            300m;

  client_body_temp_path   /dev/shm/client_body_temp;

 

  sendfile                          on;

  tcp_nopush                        on;

  keepalive_timeout                 60;

  server_tokens                    off;

  tcp_nodelay                       on;

 

  proxy_connect_timeout            180;

  proxy_read_timeout               180;

  proxy_send_timeout               180;

  proxy_buffer_size                16k;

  proxy_buffers                 4  64k;

  proxy_busy_buffers_size         128k;

  proxy_temp_file_write_size       64k;

  proxy_temp_path  /dev/shm/proxy_temp;

 

  fastcgi_connect_timeout          60;

  fastcgi_send_timeout             60;

  fastcgi_read_timeout             60;

  fastcgi_buffer_size              16k;

  fastcgi_buffers              4   64k;

  fastcgi_busy_buffers_size       128k;

  fastcgi_temp_file_write_size     64k;

  fastcgi_temp_path   /dev/shm/fastcgi;

 

  gzip                              on;

  gzip_min_length                   5k;

  gzip_buffers                 4   16k;

  gzip_http_version                1.0;

  gzip_comp_level                    6;

  gzip_types        text/plain application/x-javascript application/javascript text/css application/rss+xml;

  gzip_vary                         on;

 

  log_format raidcall '$remote_addr - $remote_user [$time_local] "$request" '

                                '$status $body_bytes_sent "$http_referer" '

                             '"$http_user_agent" $http_x_forwarded_for "$request_time"';

 

 

 include /etc/nginx/conf.d/*.conf;

 include /etc/nginx/sites-enabled/*;

 

}

其中

Worker_Processes 要根据系统相对应的Cpu个数，内存，及硬盘读写速度决定，通常来说，设置为4个或者8个较为合适，设置太大会造成Nginx被访问分配进程及回收等延迟反而影响效率;

 

Nginx Proxy

              位置在Nginx Proxy服务器的 /etc/nginx/nginx.conf

 

user www-data;

worker_processes 4;

worker_rlimit_nofile 65535;

pid /var/run/nginx.pid;

 

events {

        use epoll;

        worker_connections 65535;

}

 

 

http

{

  include       mime.types;

  default_type  application/octet-stream;

 

  access_log /var/log/nginx/access.log;

  error_log /var/log/nginx/error.log;

 

  server_names_hash_bucket_size    128;

  client_header_buffer_size        32k;

  large_client_header_buffers  4   32k;

  client_max_body_size            300m;

  client_body_temp_path   /dev/shm/client_body_temp;

 

  sendfile                          on;

  tcp_nopush                        on;

  keepalive_timeout                 60;

  server_tokens                    off;

  tcp_nodelay                       on;

 

  proxy_connect_timeout            180;

  proxy_read_timeout               180;

  proxy_send_timeout               180;

  proxy_buffer_size                16k;

  proxy_buffers                 4  64k;

  proxy_busy_buffers_size         128k;

  proxy_temp_file_write_size       64k;

  proxy_temp_path  /dev/shm/proxy_temp;

  proxy_cache_path  /dev/shm/proxy_cache_temp levels=1:2  keys_zone=cache_one:200m inactive=1d max_size=1g;

 

  fastcgi_connect_timeout          60;

  fastcgi_send_timeout             60;

  fastcgi_read_timeout             60;

  fastcgi_buffer_size              16k;

  fastcgi_buffers              4   64k;

  fastcgi_busy_buffers_size       128k;

  fastcgi_temp_file_write_size     64k;

  fastcgi_temp_path   /dev/shm/fastcgi;

 

  gzip                              on;

  gzip_min_length                   5k;

  gzip_buffers                 4   16k;

  gzip_http_version                1.0;

  gzip_comp_level                    6;

  gzip_types        text/plain application/x-javascript application/javascript text/css application/rss+xml;

  gzip_vary                         on;

 

  log_format raidcall '$remote_addr - $remote_user [$time_local] "$request" '

                                '$status $body_bytes_sent "$http_referer" '

                             '"$http_user_agent" $http_x_forwarded_for "$request_time"';

 

 

upstream backend_server {

    server   184.173.77.146:80 weight=2 max_fails=2 fail_timeout=30s;

    server   184.173.77.148:80 weight=2 max_fails=2 fail_timeout=30s;

  }

 

 

 

 

 include /etc/nginx/conf.d/*.conf;

 include /etc/nginx/sites-enabled/*;

 

}

  注意此配置文件配置文件权适用到Web Proxy服务器，开启了Proxy，并启用了1G内存做为Web缓存；其中Proxy_Cache_Path 为缓存路径，可随主机更改但一定要和Proxy_Temp_Path  在同一分区内；Upstream Backend_Server  中的Backend_Server 为Proxy 名称，可以随意更改，但在配置中调用一定要与此匹配。

 

 

 

Nginx Web Server

此配置文件在/etc/nginx/conf.d/

 

server

  {

    listen       80;

    server_name 

    index index.html index.htm;

    root  /www/www;

  

 location ~ \.php$ {

        root           /data/raidcall/www/;

        fastcgi_pass   127.0.0.1:9010;

        fastcgi_index  index.php;

        fastcgi_param script_filename  /data/raidcall/www//$fastcgi_script_name;

        include        fastcgi_params;

    }

    

    access_log  /data/weblog/ raidcall;

  }

 

注意域名要跟真实配置的一致，网站的位置Root …也要跟实际路径一致;

Fastcig_Pass 端口要跟Php-Fpm中的一致；

Raidcall 要跟Nginx配置的日志类型一致；

 

 

Nginx Web Proxy Server

          此文件位置于Web Proxy Server的/etc/nginx/conf.d/

server

  {

    listen       80;

    server_name 

    index index.html index.htm;

    root  /www/www;

  

 location ~ \.php$ {

        root           /data/raidcall/www/;

        fastcgi_pass   127.0.0.1:9010;

        fastcgi_index  index.php;

        fastcgi_param script_filename  /data/raidcall/www//$fastcgi_script_name;

        include        fastcgi_params;

}

 

 

    location /

    {

         proxy_next_upstream http_502 http_504 error timeout invalid_header;

         proxy_cache cache_one;

         proxy_cache_valid  200 304 12h;

         proxy_cache_key $host$uri$is_args$args;

         proxy_set_header host  $host;

         proxy_set_header x-forwarded-for  $remote_addr;

         proxy_pass ;

         expires      1d;

    }

 

 

    location ~ .*\.(php|jsp|cgi)?$

    {

         proxy_set_header host  $host;

         proxy_set_header x-forwarded-for  $remote_addr;

         proxy_pass ;

    }

    

    access_log  /data/weblog/ raidcall;

  }

 

上边两个第一个Location指定启用Proxy_Pass中的Backen_Server;

         第二个Location是此类动态的类型不做缓存

 

 

Redis

 

配置路径在：/Etc/Redis/Redis.Conf

具体配置为：

daemonize yes

pidfile /var/run/redis_6379.pid

port 6379

bind 127.0.0.1

timeout 0

loglevel notice

logfile /var/log/redis/redis_6379.log

databases 16

stop-writes-on-bgsave-error no

rdbcompression yes

rdbchecksum yes

dbfilename dump.rdb

dir /data/redis_data/6379

slave-serve-stale-data yes

slave-read-only yes

slave-priority 100

requirepass     redis-raidcall-cache

maxclients 10000

appendonly no

appendfsync everysec

no-appendfsync-on-rewrite no

auto-aof-rewrite-percentage 100

auto-aof-rewrite-min-size 64mb

lua-time-limit 5000

slowlog-log-slower-than 10000

slowlog-max-len 128

hash-max-ziplist-entries 512

hash-max-ziplist-value 64

list-max-ziplist-entries 512

list-max-ziplist-value 64

set-max-intset-entries 512

zset-max-ziplist-entries 128

zset-max-ziplist-value 64

activerehashing yes

client-output-buffer-limit normal 0 0 0

client-output-buffer-limit slave 256mb 64mb 60

client-output-buffer-limit pubsub 32mb 8mb 60

  

 注意Bind 这一段，因为Web服务器都是在本地读取，所以只在侦听在本地即可；

如果是开放给其它服务器读取，请设置为0.0.0.0

 

FastDFS_Client

配置文件的路径在：/etc/fdfs/client.conf

connect_timeout=30

network_timeout=60

base_path=/data/fastdfs_data/fdfs_client

tracker_server=67.228.132.195:22122

log_level=info

http.tracker_server_port=8080

 

需要注意的加粗黑色字体的部分，这部分要改成对应的FastDFS Tracker地址

 

Php

   配置文件在/etc/php5/fpm/php.ini ，需要根据不同版本配置成当地的时间，其它的保留原始配置即可，如国际版要改成

     date.timezone = america/chicago

Php-Fpm

   配置文件在/etc/php5/fpm/php-fpm.conf,原始配置即可，主要的配置文件在/etc/php5/fpm/pool.d/

    [www]

user = www-data

group = www-data

listen = 0.0.0.0:9010

pm = dynamic

pm.max_children = 1024

pm.start_servers = 30

pm.min_spare_servers = 25

pm.max_spare_servers = 100

pm.max_requests = 50000

pm.process_idle_timeout = 10s

slowlog = /var/log/php-fpm/php5-slow.log

request_slowlog_timeout = 3s

rlimit_files = 262140

 

注意黑色加粗的端口，Nginx的配置的Fastcgi_Pass端口要和这个一致

 

MySQL

         配置文件在/Etc/Mysql/My.Cnf

[client]

port            = 3306

socket          = /var/run/mysqld/mysqld.sock

[mysqld_safe]

socket          = /var/run/mysqld/mysqld.sock

nice            = 0

[mysqld]

user            = mysql

pid-file         = /var/run/mysqld/mysqld.pid

socket          = /var/run/mysqld/mysqld.sock

port            = 3306

basedir         = /usr

datadir         = /var/lib/mysql

tmpdir          = /tmp

lc-messages-dir = /usr/share/mysql

skip-external-locking

log_error = /var/log/mysql/error.log

skip-external-locking

skip-name-resolve

binlog_cache_size = 8m

key_buffer_size = 64m

max_allowed_packet = 64m

sort_buffer_size = 4m

read_buffer_size = 4m

read_rnd_buffer_size = 2m

myisam_sort_buffer_size = 2m

thread_cache_size = 16

back_log  = 500

max_connections = 4096

max_connect_errors = 200

table_cache = 4096

table_open_cache = 2048

character-set-server=utf8

init_connect='set names utf8'

query_cache_size = 128m

query_cache_limit = 2m

query_cache_min_res_unit = 2k

thread_stack = 192k

transaction_isolation = repeatable-read

tmp_table_size = 128m

max_heap_table_size = 128m

log-bin=mysql-bin

slow_query_log = on

long_query_time = 2

binlog_format = mixed

max_binlog_size = 1g

expire_logs_days = 7

innodb_data_file_path = ibdata1:16m:autoextend

innodb_io_capacity = 400

innodb_flush_log_at_trx_commit = 2

innodb_write_io_threads = 8

innodb_read_io_threads = 8

innodb_thread_concurrency = 16

innodb_max_dirty_pages_pct = 75

innodb_buffer_pool_size = 1024m

innodb_additional_mem_pool_size = 32m

innodb_log_file_size = 512m

innodb_log_buffer_size = 32m

innodb_log_files_in_group = 2

innodb_lock_wait_timeout = 50

innodb_flush_method = o_direct

[mysqldump]

quick

quote-names

max_allowed_packet      = 16m

[mysql]

[isamchk]

key_buffer              = 16m

!includedir /etc/mysql/conf.d/

4 其它：环境变量和日志分割等

环境变量

 多语言支持

 

因为我们都是很多都是国外版本，所以要多语言支持，要重新编译支持语言：

编辑语言支持文件如下：

 

sudo vim /var/lib/locales/supported.d/local

 

zh_cn.gb18030 gb18030

zh_cn.gbk gbk

zh_cn.utf-8 utf-8

zh_tw.utf-8 utf-8 en_us iso-8859-1

en_us.utf-8 utf-8

ru_ru.koi8-r koi8-r

ru_ru.utf-8 utf-8

ru_ru iso-8859-5

ru_ru.cp1251 cp1251

de_de.utf-8 utf-8

de_de iso-8859-1

de_de@euro iso-8859-15

pt_br.utf-8 utf-8

pt_br iso-8859-1

es_es.utf-8 utf-8

es_es iso-8859-1

es_es@euro iso-8859-15

 

清除掉可能的独立配置

sudo rm /var/lib/locales/supported.d/en

sudo rm /var/lib/locales/supported.d/zh

 

编译

sudo locale-gen –purge

 

等待完成即可

 

设置环境变量

  

sudo cp /etc/environment .

sudo echo 'language=="en_US.UTF-8"' >> environment

sudo echo 'lang=en_US.UTF-8' >> environment

sudo echo 'lc_all="en_US.UTF-8"' >> environment

sudo cp environment /etc/

 

如果是普通账号，没有权限直接对Environment操作，所以先复制出来所在目录，加上Locale设置，再覆盖回去；这里我们设置美国UTF8做为环境语言，有其它需求的可以按需要设置，如，可以改成zh_CN.UTF-8

 

日志分割

    

 利用系统的Logrotate来对 Nginx 日志进行分割，方便统计；配置文件/etc/logrotate.conf

用系统自带的配置即可，主要是我们要配置的自定义日志分割，位置/etc/logrotate.d/目录下

  

Nginx的分割配置：

   

/var/log/nginx/*.log

 

{

        daily

        missingok

        rotate 52

        compress

        delaycompress

        notifempty

        create 0640 www-data adm

        sharedscripts

        prerotate

                if [ -d /etc/logrotate.d/httpd-prerotate ]; then \

                        run-parts /etc/logrotate.d/httpd-prerotate; \

                fi; \

        endscript

        postrotate

                [ ! -f /var/run/nginx.pid ] || kill -usr1 `cat /var/run/nginx.pid`

        endscript

}

  这是定义为每天分割，日志的位置在/Var/Log/Nginx/,分割之后压缩保存， 分割之后日志保留52天

 

 

  MySQL的分割配置：

 

/var/log/mysql.log

/var/log/mysql/mysql.log

/var/log/mysql/mysql-slow.log

/var/log/mysql/error.log

 

 {

        daily

        rotate 7

        missingok

        create 640 mysql adm

        compress

        sharedscripts

        postrotate

                test -x /usr/bin/mysqladmin || exit 0

                # if this fails, check debian.conf!

                myadmin="/usr/bin/mysqladmin --defaults-file=/etc/mysql/debian.cnf"

                if [ -z "`$myadmin ping 2>/dev/null`" ]; then

                  # really no mysqld or rather a missing debian-sys-maint user?

                  # if this occurs and is not a error please report a bug.

                  #if ps cax | grep -q mysqld; then

                  if killall -q -s0 -umysql mysqld; then

                    exit 1

                  fi

                else

                  $myadmin flush-logs

                fi

        endscript

}