EFK(ElasticSearch+Fluentd+Kibana) 收集Docker日志

1. 下载安装包

2. 安装

rpm -ivh

rpm -ivh

3. 安装x-pack

/usr/share/elasticsearch/bin/elasticsearch-plugin install x-pack

/usr/share/elasticsearch/bin/elasticsearch-plugin install ingest-geoip

/usr/share/kibana/bin/kibana-plugin install x-pack

4. 生成认证用户名密码

/usr/share/elasticsearch/bin/x-pack/setup-passwords auto

输出类似：

Changed password for user kibana

PASSWORD kibana = Ww0XhLMZglXSvIfmSkUe

Changed password for user logstash_system

PASSWORD logstash_system = vOi2j6JfsKoIrcSEUzwf

Changed password for user elastic

PASSWORD elastic = 0jnHB3wQc45lwMmvOVe4

5. 配置ElasticSearch

[root@salt-st2 elasticsearch]# grep -v "^#" /etc/elasticsearch/elasticsearch.yml

path.data: /var/lib/elasticsearch

path.logs: /var/log/elasticsearch

network.host: 10.28.0.222 # 修改localhost 到IP

http.port: 9200

xpack.security.enabled: false

6. 配置Kibana

[root@salt-st2 kibana]# grep -v "^#" kibana.yml | grep -v "^$"

server.host: "10.28.0.222"

elasticsearch.url: "" # 修改localhost 到IP， 否则kibana连接拒绝

elasticsearch.username: "kibana" # 第四步生成的用户急及密码

elasticsearch.password: "Ww0XhLMZglXSvIfmSkUe"

7. 启动服务

systemctl start elasticsearch

systemctl start kibana

8. 配置Fluentd (所有Docker主机运行一个fluentd 容器来收集所有容器日志)

docker pull fluent/fluentd

docker run -d fluent/fluentd

docker exec -it sh

gem install fluent-plugin-elasticsearch # 安装插件fluent-plugin-elasticsearch

docker commit

docker tag wwyhy/fluentd

(已经将上述步骤完成并pull 到了docker hub, 可忽略以上步骤直接 docker pull wwyhy/fluentd.)

mkdir -p /opt/fluentd/etc

vim /opt/fluentd/etc/docker.conf

---------------------

@type forward

port 24224

bind 0.0.0.0

@type copy

@type elasticsearch

host 10.28.0.222

port 9200

logstash_format true

logstash_prefix fluentd

logstash_dateformat %Y%m%d

include_tag_key true

type_name access_log

tag_key @log_name

flush_interval 1s

@type stdout

-------------------------------

运行Fluentd agent.

docker run -d --name fluentd --restart=always -p 24224:24224 -p 24224:24224/udp -v /opt/fluentd/etc:/fluentd/etc -e FLUENTD_CONF=docker.conf wwyhy/fluentd

9. 配置Kibana Index

浏览器打开

创建index pattern, 输入 fluentd-*, 下一步， 选择@timestamp, 创建

10. 验证

docker pull registry.docker-cn.com/library/nginx

docker run -d --log-driver=fluentd --log-opt fluentd-address=172.17.0.1:24224 --log-opt tag={{.Name}} -p 80:80 registry.docker-cn.com/library/nginx

curl localhost (应该输出welcome to nginx)

浏览器打开

在 Discover下应该能看到NGINX的输出日志

结合Marathon(API)/Mesos/Docker, Json 文件或参数如下：

root@master:~/testjava# cat testapp.json

{

"id": "testjava",

"container": {

"type": "DOCKER",

"docker": {

"image": "",

"network": "BRIDGE",

"parameters": [

{ "key": "log-driver", "value": "fluentd" },

{ "key": "log-opt", "value": "tag='testjava'" },

{ "key": "log-opt", "value": "fluentd-address=172.17.0.1:24224" }

],

"portMappings": [

{ "containerPort": 80, "hostPort": 0 }

]

},

"volumes": []

}

}