zabbix 应用系列之nginx tcp stream监控

1、实现思路

• 原生支持

• 通过分析日志获取

• 通过分析网络获取

2、原生支持和日志分析情况 nginx自身对监控的支持

• nginx提供status模块,目前只可以获取http相关信息，而tcp和udp相关的状态信息在当前 暂未实现
• 日志记录在稳定版本也未实现，经过查阅nginx文档发现后续版本会提供支持: [The ngx_stream_log_module module (1.11.4) writes session logs in the specified format.]

3、分析网络来监控

• nentstat 工具中关于连接状态介绍

State
       The state of the socket. Since there are no states in raw mode and usu‐
       ally no states used in UDP and UDPLite, this column may be left  blank.
       Normally this can be one of several values:

       ESTABLISHED
              The socket has an established connection.

       SYN_SENT
              The socket is actively attempting to establish a connection.

       SYN_RECV
              A connection request has been received from the network.

       FIN_WAIT1
              The socket is closed, and the connection is shutting down.

       FIN_WAIT2
              Connection  is  closed, and the socket is waiting for a shutdown
              from the remote end.

       TIME_WAIT
              The socket is waiting after close to handle packets still in the
              network.

       CLOSE  The socket is not being used.

       CLOSE_WAIT
              The remote end has shut down, waiting for the socket to close.

       LAST_ACK
              The  remote end has shut down, and the socket is closed. Waiting
              for acknowledgement.

       LISTEN The socket is listening for incoming connections.  Such  sockets
              are  not included in the output unless you specify the --listen‐
              ing (-l) or --all (-a) option.

       CLOSING
              Both sockets are shut down but we still don't have all our  data
              sent.

       UNKNOWN
              The state of the socket is unknown. 

• centos7 中替代netstat的网络工具ss中连接状态的介绍[ss STATE-FILTER]

STATE-FILTER allows to construct arbitrary set of states to match. Its syntax is sequence of keywords state and exclude followed by identifier of state.

Available identifiers are:
All standard TCP states: established, syn-sent, syn-recv, fin-wait-1, fin-wait-2, time-wait, closed, close-wait, last-ack, listen and closing.

all - for all the states

connected - all the states except for listen and closed

synchronized - all the connected states except for syn-sent

bucket - states, which are maintained as minisockets, i.e. time-wait and syn-recv

big - opposite to bucket

• ss 运行内容示例

State      Recv-Q Send-Q      Local Address:Port          Peer Address:Port
FIN-WAIT-1 0      1               10.0.1.11:59001         117.61.1.199:20060
ESTAB      0      0               10.0.1.11:http          117.61.3.172:38306
ESTAB      0      0               10.0.1.11:http        117.61.129.104:15315

4、脚本

#!/bin/sh
# nginx tcp stream  stats
# default two ports 59001 & 59002
# c: client to nginx
# s: nginx to backend server
function c59001 {
  ss -t -o state all '( sport = :59001 )' |tail -n +2 |wc -l
}
function s59001 {
  ss -t -o state all '( dport = :59001 )' |tail -n +2 |wc -l
}

function c59003 {
  ss -t -o state all '( sport = :59003 )' |tail -n +2 |wc -l
}
function s59003 {
  ss -t -o state all '( dport = :59003 )' |tail -n +2 |wc -l
}
function client {
  ss -t -o state all '( sport = :http or sport = :https or sport = :59001 or sport = :59003 )' |tail -n +2 |wc -l
}
function server {
  ss -t -o state all |tail -n +2|awk '{print $5}' |grep ^10.0 |wc -l
}
function all {
  client
  server

}
# Run the requested function
$1

• 备注

  脚本介绍的是tcp，将ss的参数-n 替换为-u即可统计udp 的stream

  上述脚本统计的是全部连接数（包含time wait等状态），如果统计建立的连接数（established）请将脚本中的ss命令替换为类似

  ss -o state established '( sport = :59001 or sport = :59003 )'

  脚本中用ss替换netstat有两个原因，一个是ss性能远高于netstat，特别是连接数很多的情况下（上万）；centos7已经优先安装使用ss（iproute包内），最小安装已经不安装netstat工具了:(

5、zabbix 设置

• agent端设置

  将脚本保存到/etc/zabbix/scripts

  vi /etc/zabbix/scripts/nginx-stream.sh

  将nginxstream.conf保存到/etc/zabbix/zabbixagentd.d/

  # cat nginx_stream.conf
    UserParameter=c59001,/etc/zabbix/scripts/nginx-stream.sh c59001
   UserParameter=c59003,/etc/zabbix/scripts/nginx-stream.sh c59003
   UserParameter=s59001,/etc/zabbix/scripts/nginx-stream.sh s59001
   UserParameter=s59003,/etc/zabbix/scripts/nginx-stream.sh s59003

  重启zabbix systemctl restart zabbix-agent 测试

  [root@nginx02 ~]# zabbix_agentd -t c59001
  c59001                                        [t|148]
  [root@nginx02 ~]# zabbix_agentd -t c59003
  c59003                                        [t|96]

  agent端已经完美工作！

• server端设置

  创建模板-创建项目（c59001，c59003，s59001，s59003）-设置触发器（可选）链接主机-创建图形-创建屏幕等等略过

  server 端测试如下

  [root@ops01 ~]# zabbix_get -s 10.0.1.12 -k c59003
  95
  [root@ops01 ~]# zabbix_get -s 10.0.1.12 -k s59003
  269 

  6、设置完成的图形展示