docker swarm mode 集群应用之五 prometheus+cadvisor+grafana 配置docker监控

docker监控设置

方案

prometheus + cadvisor + grafana 

修改docker增加实验特性和打开监听端口

打开实验特性，docker services logs 可以看到日志
打开度量地址和端口，方便docker-exporter采集数据
编辑或者增加如下的文件内容
cat /etc/docker/daemon.json
{
  "experimental": true,
  "metrics-addr": "0.0.0.0:4999"
}

下载相关docker images

docker pull prom/node-exporter
docker pull prom/node-exporter
docker pull prom/alertmanager
docker pull prometheus-swarm
docker pull basi/prometheus-swarm
docker pull grafana/grafana
docker pull basi/grafana
docker pull basi/node-exporter
docker pull basi/socat
docker pull basi/alertmanager
docker pull basi/grafana

docker tag 和push到本地私有仓库

docker tag prom/node-exporter reg.goluk.cn/prometheus/node-exporter
docker push reg.goluk.cn/prometheus/node-exporter

撰写 prometheus的stack yml文件

[root@swarm3 prometheus]# cat prometheus-swarm.yml
version: '3'
services:
  cadvisor:
   image: reg.goluk.cn/prometheus/cadvisor
   volumes:
     - /:/rootfs:ro
     - /var/run:/var/run:rw
     - /sys:/sys:ro
     - /var/lib/docker:/var/lib/docker:ro
   deploy:
     mode: global
     update_config:
       parallelism: 2
       delay: 10s
     restart_policy:
       condition: on-failure
node-exporter:
  image: reg.goluk.cn/basi/node-exporter
  environment:
    - HOST_HOSTNAME=/etc/host_hostname
  volumes:
    - /proc:/host/proc:ro
    - /sys:/host/sys:ro
    - /:/rootfs:ro
    - /etc/hostname:/etc/host_hostname
  deploy:
    mode: global
    update_config:
      parallelism: 2
      delay: 10s
    restart_policy:
    condition: on-failure
docker-exporter:
  image: reg.goluk.cn/basi/socat
  environment:
    - IN=172.19.0.1:4999
  ports:
    - 4999
  deploy:
    mode: global
    update_config:
      parallelism: 2
      delay: 10s
    restart_policy:
      condition: on-failure
alertmanager:
  image: reg.goluk.cn/basi/alertmanager
  ports:
    - 9093:9093
prometheus:
  image: reg.goluk.cn/basi/prometheus-swarm
  volumes:
    - /mnt/cephfs/app/prometheus/data2:/prometheus
  ports:
    - 9090:9090
grafana:
  image: reg.goluk.cn/basi/grafana
  environment:
    - GF_SERVER_ROOT_URL={CLUSTER_DOMAIN}
    - GF_SECURITY_ADMIN_PASSWORD=mobnote
    - PROMETHEUS_ENDPOINT=
    - ELASTICSEARCH_ENDPOINT=$ES_ADDRESS
    - ELASTICSEARCH_USER=$ES_USERNAME
    - ELASTICSEARCH_PASSWORD=$ES_PASSWORD
  ports:
    - 3000:3000

运行

docker stack deploy prometheus -c prometheus-swarm.yml

查看

[root@swarm1 ~]# docker stack ps prometheus
ID            NAME                                                  IMAGE                                      NODE    DESIRED STATE  CURRENT STATE       ERROR  PORTS
wky68btz4jgr  prometheus_docker-exporter.x971p4zni5ssccncx01s5dp33  reg.goluk.cn/basi/socat:latest             swarm2  Running        Running 6 days ago
une7pjtengjk  prometheus_docker-exporter.plphtb9km6qy4io9eurumreh2  reg.goluk.cn/basi/socat:latest             swarm3  Running        Running 6 days ago
t8va1t8nze86  prometheus_docker-exporter.cs58ccek6ehc521etji8f00nl  reg.goluk.cn/basi/socat:latest             swarm1  Running        Running 6 days ago
oxcdft8mi0tl  prometheus_node-exporter.x971p4zni5ssccncx01s5dp33    reg.goluk.cn/basi/node-exporter:latest     swarm2  Running        Running 6 days ago
tgoeis96r4or  prometheus_node-exporter.plphtb9km6qy4io9eurumreh2    reg.goluk.cn/basi/node-exporter:latest     swarm3  Running        Running 6 days ago
h6g77a1kyu5d  prometheus_node-exporter.cs58ccek6ehc521etji8f00nl    reg.goluk.cn/basi/node-exporter:latest     swarm1  Running        Running 6 days ago
m8n6ksico6vr  prometheus_cadvisor.x971p4zni5ssccncx01s5dp33         reg.goluk.cn/prometheus/cadvisor:latest    swarm2  Running        Running 6 days ago
628rmkmwwty2  prometheus_cadvisor.plphtb9km6qy4io9eurumreh2         reg.goluk.cn/prometheus/cadvisor:latest    swarm3  Running        Running 6 days ago
50yehykk8225  prometheus_cadvisor.cs58ccek6ehc521etji8f00nl         reg.goluk.cn/prometheus/cadvisor:latest    swarm1  Running        Running 6 days ago
du8lopn7lr5r  prometheus_grafana.1                                  reg.goluk.cn/basi/grafana:latest           swarm3  Running        Running 6 days ago
xdfy8xg3mfqj  prometheus_prometheus.1                               reg.goluk.cn/basi/prometheus-swarm:latest  swarm3  Running        Running 6 days ago
n0j7nt522upy  prometheus_alertmanager.1                             reg.goluk.cn/basi/alertmanager:latest      swarm3  Running        Running 6 days ago

访问方法

prometheus: 
grafana: