更多python、Linux、网络安全学习内容，可移步：www.oldboyedu.com或关注\"老男孩Linux\"公众号
分类： LINUX
2022-10-31 14:38:54
filebeat是用于"转发"和"集中日志数据"的轻量级数据采集器。
filebeat会监视指定的日志文件路径，收集日志事件并将数据转发到elasticsearch，logstash，redis，kafka存储服务器。
当您要面对成百上千，甚至成千上万的服务器，虚拟机的容器生成的日志时，请告别SSH吧。
Filebeat将为您提供一种轻量级方法，用于转发和汇总日志与文件，让简单的事情不再繁杂。
| Filebeat的组件
Filebeat包含两个主要组件，input(输入)和Harvester(收割机)，两个组件协同工作将文件的尾部{BANNED}最佳新数据发送出去。
Harveste组件: 负责逐行读取单个文件的内容,然后将内容发送到输出。
input组件: 输入负责管理收割机并找到所有要读取的源。该参数的源文件路径需要使用者手动配置。
Spooler(缓冲区): 将Harvester组件采集的数据进行统一的缓存,并发往目的端,可以是 Elasticsearch, Logstash , kafka 和 Redis 等。
| Filebeat工作原理
filebeat工作流程如下:
1、filebeat启动后，filebeat通过Input读取指定的日志路径;
2、为该文件日志启动收割进程harvester,每个收割进程读取一个日志文件的新内容,并发送这些新的日志数据到处理程序spooler;
3、spooler会集合这些事件,{BANNED}最佳后filebeat会发送集合的数据到你指定的位置。
Filebeat如何保持文件的状态?
Filebeat保持每个文件的状态,并经常将状态刷新到注册表文件(data/registry/filebeat/log.json)中的磁盘。
该状态用于记住收割机读取的{BANNED}最佳后一个偏移量,并确保发送所有日志行。
Filebeat如何确保至少一次交付?
Filebeat保证事件将至少传送到配置的输出一次并且不会丢失数据。
Filebeat能够实现这种行为，因为它将每个事件的传递状态存储在注册表文件中。
部署Filebeat环境
| 安装Filebeat软件
# 编译安装Filebeat wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.12.1-linux-x86_64.tar.gz tar xf filebeat-7.12.1-linux-x86_64.tar.gz -C /oldboyedu/softwares/ cd /oldboyedu/softwares/ ln -s filebeat-7.12.1-linux-x86_64 filebeat vim /etc/profile.d/filebeat.sh # 添加Filebeat的环境变量 cat /etc/profile.d/filebeat.sh #!/bin/bash export FILE_BEAT=/oldboyedu/softwares/filebeat export PATH=$PATH:$FILE_BEAT # 使环境变量生效 source /etc/profile.d/filebeat.sh # 查看环境变量是否生效 which filebeat
| filebeat参数介绍
| 运行{BANNED}中国第一个实例
将标准输入的数据进行标准输出
vim stdin-to-console.yaml filebeat.inputs: - type: stdin enabled: true output.console: pretty: true enable: true # 查看filebeat的输出 filebeat -e -c stdin-to-console.yaml
企业实战
| nginx日志收集
安装nginx
yum -y install epel-release yum -y install nginx
创建配置文件
vim /etc/nginx/conf.d/elk103.oldboyedu.com.conf server { listen 80; server_name es.oldboyedu.com; root /oldboyedu/data/nginx/; location / { index index.html; } }
创建测试数据
mkdir -p /oldboyedu/data/nginx/ echo "老男孩教育
" > /oldboyedu/data/nginx/index.html
检查配置文件
nginx -t
启动nginx服务
systemctl start nginx
测试nginx服务
# 编写脚本 vim /server/scripts/nginx.sh #!/bin/bash while true do for i in "curl es.oldboyedu.com" do Time=$((RANDOM%5 +1 )) echo "本次间隔时间为：$Time" curl elk103.oldboyedu.com sleep $Time done done
配置nginx收集JSON并重启nginx
# 修改nginx的配置文件 vim /etc/nginx/nginx.conf ... # 自定义nginx的日志格式为json格式 log_format oldboyedu_nginx_json '{"@timestamp":"$time_iso8601",' '"host":"$server_addr",' '"clientip":"$remote_addr",' '"size":$body_bytes_sent,' '"responsetime":$request_time,' '"upstreamtime":"$upstream_response_time",' '"upstreamhost":"$upstream_addr",' '"http_host":"$host",' '"uri":"$uri",' '"domain":"$host",' '"xff":"$http_x_forwarded_for",' '"referer":"$http_referer",' '"tcp_xff":"$proxy_protocol_addr",' '"http_user_agent":"$http_user_agent",' '"status":"$status"}'; access_log /var/log/nginx/access.log oldboyedu_nginx_json; # 测试配置文件是否正常 nginx -t # 重新加载nginx systemctl restart nginx
配置filebeat的配置文件
vim 01-nginx-to-es.yaml filebeat.inputs: - type: log paths: - /var/log/nginx/access.log tags: "nginx" # 默认值为false，我们需要修改为true，即不会将消息存储至message字段! json.keys_under_root: true output.elasticsearch: hosts: ["192.168.56.130:9200","192.168.56.131:9200","192.168.56.132:9200"] #index: "oldboy-2022-%{[agent.version]}-%{+yyyy.MM.dd}" indices: - index: "oldboyedu-nginx2022-%{+yyyy.MM.dd}" when.contains: tags: "nginx" # 禁用索引的生命周期! setup.ilm.enabled: false # 指定索引模板的名称 setup.template.name: "oldboyedu" # 指定索引模板的匹配模式 setup.template.pattern: "oldboyedu-nginx*" # 指定索引模板的分片信息 setup.template.settings: index.number_of_shards: 5 index.number_of_replicas: 0
收集nginx的错误日志
vim 02-nginx-to-es.yaml filebeat.inputs: - type: log paths: - /var/log/nginx/access.log tags: "nginx-access" # 默认值为false，我们需要修改为true，即不会将消息存储至message字段! json.keys_under_root: true - type: log paths: - /var/log/nginx/error.log tags: "nginx-error" output.elasticsearch: hosts: ["192.168.56.130:9200","192.168.56.131:9200","192.168.56.132:9200"] #index: "oldboy-2022-%{[agent.version]}-%{+yyyy.MM.dd}" indices: - index: "oldboyedu-nginx-access-%{+yyyy.MM.dd}" when.contains: tags: "nginx-access" - index: "oldboyedu-nginx-error-%{+yyyy.MM.dd}" when.contains: tags: "nginx-error" # 禁用索引的生命周期! setup.ilm.enabled: false # 指定索引模板的名称 setup.template.name: "oldboyedu" # 指定索引模板的匹配模式 setup.template.pattern: "oldboyedu-nginx*" # 指定索引模板的分片信息 setup.template.settings: index.number_of_shards: 5 index.number_of_replicas: 0
| Nginx多虚拟主机
配置nginx的多虚拟主机
vim /etc/nginx/conf.d/bbs.oldboyedu.com.conf server { listen 80; server_name bbs.oldboyedu.com; root /oldboyedu/data/nginx/bbs; # 指定access.log的存储路径及日志格式. access_log /var/log/nginx/bbs.log oldboyedu_nginx_json; location / { index index.html; } } vim /etc/nginx/conf.d/blog.oldboyedu.com.conf server { listen 80; server_name blog.oldboyedu.com; root /oldboyedu/data/nginx/blog; # 指定access.log的存储路径及日志格式. access_log /var/log/nginx/blog.log oldboyedu_nginx_json; location / { index index.html; } }
创建测试数据
mkdir -p /oldboyedu/data/nginx/{blog,bbs} echo "blog
" > /oldboyedu/data/nginx/blog/index.html echo "bbs
" > /oldboyedu/data/nginx/bbs/index.html # 检查配置文件的语法 nginx -t # 修改主机名映射 vim /etc/hosts ... 192.168.56.132 blog.oldboyedu.com 192.168.56.132 bbs.oldboyedu.com # 重启nginx服务 systemctl restart nginx # 测试服务 curl blog.oldboyedu.com curl bbs.oldboyedu.com
编写fielbeat的yaml
vim nginx_vm_host.yaml filebeat.inputs: - type: log enabled: true paths: - /var/log/nginx/access.log # false会将json解析的格式存储至message，改为true则不存储至message json.keys_under_root: true # 覆盖默认的message字段，使用自定义json格式的key json.overwrite_keys: true # 为访问日志("access.log")打标签 tags: ["nginx-access"] - type: log enabled: true paths: - /var/log/nginx/blog.log # false会将json解析的格式存储至message，改为true则不存储至message json.keys_under_root: true # 覆盖默认的message字段，使用自定义json格式的key json.overwrite_keys: true # 为访问日志("access.log")打标签 tags: ["nginx-blog"] - type: log enabled: true paths: - /var/log/nginx/demo.log # false会将json解析的格式存储至message，改为true则不存储至message json.keys_under_root: true # 覆盖默认的message字段，使用自定义json格式的key json.overwrite_keys: true # 为访问日志("access.log")打标签 tags: ["nginx-demo"] - type: log enable: true paths: - /var/log/nginx/error.log # 为错误日志("error.log")打标签 tags: ["nginx-error"] output.elasticsearch: hosts: ["192.168.56.130:9200","192.168.56.131:9200","192.168.56.132:9200"] # index: "nginx-access-%{[agent.version]}-%{+yyyy.MM.dd}" # 注意哈，下面的标签不再是"index"啦~ indices: - index: "nginx-access-%{[agent.version]}-%{+yyyy.MM.dd}" when.contains: tags: "nginx-access" - index: "nginx-error-%{[agent.version]}-%{+yyyy.MM.dd}" when.contains: tags: "nginx-error" - index: "nginx-blog-%{[agent.version]}-%{+yyyy.MM.dd}" when.contains: tags: "nginx-blog" - index: "nginx-demo-%{[agent.version]}-%{+yyyy.MM.dd}" when.contains: tags: "nginx-demo" setup.ilm.enabled: false # 定义模板名称. setup.template.name: "nginx" # 定义模板的匹配索引名称. setup.template.pattern: "nginx-*" [root@oldboy-es03 project]# filebeat -e -c nginx_vm_host.yaml
| Tomcat日志收集
部署tomcat
tar zxf apache-tomcat-10.0.6.tar.gz -C /oldboy/softwares/ cd /oldboyedu/softwares/ ln -s apache-tomcat-10.0.6 tomcat # 配置JDK 的环境变量 vim /etc/profile.d/tomcat.sh #!/bin/bash export TOMCAT_HOME=/oldboyedu/softwares/tomcat export PATH=$PATH:$TOMCAT_HOME/bin # 让环境变量生效 . /etc/profile.d/tomcat.sh catalina.sh # 配置tomcat的JSON格式 vim /oldboyedu/softwares/tomcat/conf/server.xml ···(大概在133行哟~) <Host name="tomcat.oldboyedu.com" appBase="webapps" unpackWARs="true" autoDeploy="true"> ...(需要手动注释一下原内容) <!-- <Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs" prefix="localhost_access_log" suffix=".txt" pattern="%h %l %u %t "%r" %s %b" /> --> <Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs" prefix="tomcat.oldboyedu.com_access_log" suffix=".txt" pattern="{"clientip":"%h","ClientUser":"%l","authentica ted":"%u","AccessTime":"%t","request":"%r","status":"%s","SendBytes":"%b","Query?string":"%q","partner":"%{Referer}i","AgentVersion":"%{User-Agent}i"}"/> ... # 配置主机解析 vim /etc/hosts ... 19.168.56.132 tomcat.oldboyedu.com # 启动tomcat服务 catalina.sh start # 验证服务 略。
使用filebeat收集日志
vim ~/conf/project/tomcat01.tomcat-to-es.yaml filebeat.inputs: - type: log paths: - /oldboyedu/softwares/tomcat/logs/tomcat.oldboyedu.com_access_log.*.txt # false会将json解析的格式存储至message，改为true则不存储至message json.keys_under_root: true # 为访问日志("access.log")打标签 tags: "tomcat-access" output.elasticsearch: hosts: ["192.168.56.130:9200","192.168.56.131:9200","192.168.56.132:9200"] # 注意哈，下面的标签不再是"index"啦~ indices: - index: "tomcat-access-%{[agent.version]}-%{+yyyy.MM.dd}" when.contains: tags: "tomcat-access" setup.ilm.enabled: false # 定义模板名称. setup.template.name: "tomcat" # 定义模板的匹配索引名称. setup.template.pattern: "tomcat-*" # 指定索引模板的分片信息 setup.template.settings: index.number_of_shards: 3 index.number_of_replicas: 0 [root@oldboy-es03 ~]#
收集错误日志
vim ~/conf/project/tomcat/03.tomcat-to-es.yaml filebeat.inputs: - type: log paths: - /oldboyedu/softwares/tomcat/logs/tomcat.oldboyedu.com_access_log.*.txt json.keys_under_root: true tags: "tomcat-access" - type: log paths: - /oldboyedu/softwares/tomcat/logs/catalina* tags: "tomcat-error" multiline.type: pattern multiline.pattern: '^\d{2}' multiline.negate: true multiline.match: after multiline.max_lines: 1000 output.elasticsearch: hosts: ["192.168.56.130:9200","192.168.56.131:9200","192.168.56.132:9200"] indices: - index: "tomcat-access-%{[agent.version]}-%{+yyyy.MM.dd}" when.contains: tags: "tomcat-access" - index: "tomcat-error-%{[agent.version]}-%{+yyyy.MM.dd}" when.contains: tags: "tomcat-error" setup.ilm.enabled: false setup.template.name: "tomcat" setup.template.pattern: "tomcat-*" setup.template.settings: index.number_of_shards: 3 index.number_of_replicas: 0