kubernetes Logging Architecture

 
            第二次为kubernetest集群收集POD日志，kubernetes版本已更新为v1.14.2，还是选用sidecar模式旁挂filebeat, 通过 emptydir 共享日志目录，达到收集效果。同时测试一些env导入方式，引用downward api, 整合成一个filebeat配置文件。
                 
               
        此次收集filebeat.yml使用了很多env变量，配置文件自身是configmap, 通过valueFrom方式从Downward api , secret 导入数据，填充到环境变量中，来补全配置文件。话不多说配置文件奉上。

        filebeat连接集群所需要的用户名与密码

apiVersion: v1
kind: Secret
metadata:
  name: essecret
  namespace: basic-plat
type: Opaque
data:
  ES_USERNAME: YWRtaW4=
  ES_PASSWORD: Y2FpNjYxMTM2

        filebeat.yml configmap  // 由于ES集群版本是v5.5. 所以filebeat 也选用v5.5.2 

name: ${NODE_NAME}-${POD_IP}

filebeat.prospectors:
- input_type: log
  paths:
    - /usr/src/log/${SERVICE_NAME}_mpsp.log
  fields:
    service: ${SERVICE_NAME}_mpsp
  fields_under_root: true

- input_type: log
  paths:
    - /usr/src/log/${SERVICE_NAME}_detail.log
  fields:
    service: ${SERVICE_NAME}_detail
  fields_under_root: true

output.elasticsearch:
  hosts: '${ES_HOSTS}'
  username: '${ES_USERNAME}'
  password: '${ES_PASSWORD}'
  protocol: http
  path: /
  loadbalance: true
  worker: 2
  template.enabled: false
  indices:
    - index: "filebeat-${INDEX_NAME}_mpsp-log-%{+yyyy.MM.dd}"
      when.contains:
        service: "${SERVICE_NAME}_mpsp"
    - index: "filebeat-${INDEX_NAME}_detail-log-%{+yyyy.MM.dd}"
      when.contains:
        service: "${SERVICE_NAME}_detail"

       业务与filebreat容器的yaml

--- 业务配置内容省略

- name: filebeat
          image: 10.10.56.148/es-shipper-filebeat/filebeat:5.5.2
          imagePullPolicy: IfNotPresent
          command: ["filebeat"]
          args: ["-e","-path.config=/usr/local/src/"]
          env:
            - name: ES_HOSTS
              value: "10.10.38.18:9200,10.10.38.19:9200"
            - name: SERVICE_NAME
              value: "SaaSOperationPlatService"
            - name: INDEX_NAME
              value: "saasoperationplatservice"
            - name: NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
            - name: POD_IP
              valueFrom:
                fieldRef:
                  fieldPath: status.podIP
            - name: ES_USERNAME
              valueFrom:
                secretKeyRef:
                  name: essecret
                  key: ES_USERNAME
            - name: ES_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: essecret
                  key: ES_PASSWORD
          volumeMounts:
            - name: filebeat-config
              mountPath: /usr/local/src
            - name: logdir
              mountPath: /usr/src/log
      imagePullSecrets:
        - name: harbor148key
      volumes:
        - name: my-xxxx-service-config-dir
          configMap:
            name: xxxx-operationplat-service
        - name: logdir
          emptyDir: {}
        - name: filebeat-config
          configMap:
            name: filebeat-configmap

          通过downwad api 我们拿到了node节点的主机名称与POD_IP, 当出现问题时便于排障，通过secret导入所需的用户名密码，测试了新功能，收集日志成功。



                     
                                                                                   umpay_Cail



参考资料：