巧用shell+rsync服务实现日志自动过滤处理压缩并上传日志服务器，自动分类

分享一个自己很早以前先写的日志存储方案，简单实用，业务端上传完日志后自己清除已经上传过的日志，连清空间的工作都省了。（未完）

找出所有部署目录下的日志文件夹,压缩指定文件与文件夹,不改变日志压缩路径(排除gzip)
上传指定名称的日志压缩包到rsync服务器,根据ip自动生成目录,并上传后保持原来的目录结构.
上传文件权限进行自动修改,方便log用户访问.
自动创建用户,安装rsync,链接日志目录到log用户.


日志服务器:192.168.165.88,
同步服务器运行软件：rsync
日志上传到/back/chss并自动创建基于业务服务器IP的目录实现日志归类

[root@192_168_165_88 ~]# cat /etc/rsyncd/rsyncd.conf
# Minimal configuration file for rsync daemon
# See rsync(1) and rsyncd.conf(5) man pages for help
# This line is required by the /etc/init.d/rsyncd script
pid file = /var/run/rsyncd.pid
port = 873
address = 192.168.165.88
uid=root
gid=root
auth users = admin
secrets file = /etc/rsyncd/rsyncd.secrets
#incoming chmod = u+rwx,g+rwx,o+rx
use chroot = yes
read only = no
#limit access to private LANs
hosts allow=192.168.0.0/255.255.0.0
hosts deny=*
max connections = 500
#This will give you a separate log file
log file = /var/log/rsync.log
#This will log every file transferred - up to 85,000+ per user, per sync
transfer logging = yes
log format = %t %a %m %f %b
syslog facility = local3
timeout = 300
#ignore errors
[chss-t]
path = /back/chss
list = yes
[chss-tc]
path = /back/chss_config
list = yes

每个目录下都是跟业务服务器上的部署路径完全一样的完整的存储路径，这里不就不贴出来了。

[root@192_168_165_88 chss]# pwd
/back/chss
[root@192_168_165_88 chss]# ls
XXX XXX XXX XXX XXX ...
[root@192_168_165_88 chss]# cd chss-aaaaaa/
[root@192_168_165_88 chss-aaaaaa]# ls
192.168.171.72  192.168.174.45  192.168.174.46

各业务服务器中使用脚本处理过滤压缩后上传到日志服务器
把以下2个脚本放置在服务器的bin目录下
执行

[root@S192-168-171-134 ~]# nohup /bin/while.t &

[root@S192-168-171-134 tmp]# ls
find_192.168.171.134.log   rsync.exe.log  nohup.out rsync_192.168.171.134.log

/bin/while.t

#!/bin/bash
while :
do
/bin/bash /bin/rsync.t
sleep 0.5h
done

/bin/rsync.t

#!/bin/env bash
#set -x
#2013-08-18:	修复直接压缩201目录的bug
#2013-09-28:	新加项目分类目录,新加多重判断
#2015-10-19:	修复与glusterfs冲突问题.

export RSYNC_PASSWORD=skymobi
LOGPATH="opt app data app1"
SLEEPTIME=10s
IP=$(LC_ALL=C /sbin/ifconfig|grep "inet addr:"|grep -v "127.0.0.1"|cut -d: -f2|awk '{print $1}'|sed ':a;$!N;s/\n/_/g;ta;P;D')
USER=admin
RSYNCIP=192.168.165.88
PROJECT=chss-mis
RSYNCMODULE=chss-t
RSYNCCONFMODULE=chss-tc
RSYNCDIR=${PROJECT}/"$IP"
FINDLOGFILE=/tmp/find_"$IP".log
RSYNCLOGFILE=/tmp/rsync_"$IP".log
RSYNCEXELOGFILE=/tmp/rsync.exe.log
DATETIME=$(date +%F_%T)
mkdir -p /tmp/$RSYNCDIR
for rspath in $RSYNCMODULE $RSYNCCONFMODULE
do
rsync -atv --log-file=${RSYNCLOGFILE} /tmp/${PROJECT} ${USER}@"$RSYNCIP"::${rspath}
done

for Path in $LOGPATH
do
echo $DATETIME
if [ -d /$Path ];then
FINDER(){
find /${Path} -regex ".*logs/.*log.*201.*" -a ! -regex '.*tar.gz' -a -mmin +5 -exec date '+%F %T' \; -exec tar -zcvf `basename {}`_t.tar.gz {} --remove-files --exclude=*tar.gz \; >> $FINDLOGFILE
}
SPACE=$(df -P /${Path} |tail -n 1 |awk '{print $5}'|awk -F% '{print $1}')
if [ $SPACE -gt 80 ];then
find /${Path} -regex ".*logs/.*log.*201.*" -mtime +"${DAYS:=1}" -exec date '+%F %T' \; -exec rm -rfv {} \; >> ${FINDLOGFILE}
else
FINDER
fi
sleep $SLEEPTIME
rsync -aERtvz --no-p --chmod=Do+rX,Fo-X --log-file=${RSYNCLOGFILE} --remove-source-files --skip-compress=*.gz $(find /$Path -type f -name '*_t.tar.gz' -a ! -regex '.*/brick/.*') ${USER}@"$RSYNCIP"::"$RSYNCMODULE"/${RSYNCDIR}
rsync -aERtvz --no-p --chmod=Do+rX,Fo-X --log-file=${RSYNCLOGFILE} --remove-source-files --skip-compress=*.gz $(find /$Path -type f -name '*_tc.tar.gz' -a ! -regex '.*/brick/.*') ${USER}@"$RSYNCIP"::"$RSYNCCONFMODULE"/${RSYNCDIR}
else
echo /${Path} is not exist...
fi
done >> ${RSYNCEXELOGFILE}

在日志服务器上执行crontab任务
定期清理空间

#!/bin/env bash
set -x
LOGPATH="back/chss/chss-mis back/chss/zm-adv-mis opt app data app1 back"
IP=$(LC_ALL=C /sbin/ifconfig|grep "inet addr:"|grep -v "127.0.0.1"|cut -d: -f2|awk '{print $1}'|sed ':a;$!N;s/\n/_/g;ta;P;D')
FINDLOGFILE=/tmp/find_"$IP".log
RSYNCLOGFILE=/tmp/rm_"$IP".log
RSYNCEXELOGFILE=/tmp/rm.exe.log
DATETIME=$(date +%F_%T)
echo _start_ `date +%F_%T`
for Path in $LOGPATH
do
  echo $DATETIME
  if [ -d /$Path ];then
    SPACE=$(df -P /${Path} |tail -n 1 |awk '{print $5}'|awk -F% '{print $1}')
    if [ $SPACE -gt 80 ];then
      if [ ${Path} == 'back/chss/zm-adv-mis' ];then
        DATENUM=0
        MINNUM=360
      elif [ ${Path} == 'back/chss/chss-mis' ];then
        DATENUM=0
        MINNUM=7200
      else
        DATENUM=10
        MINNUM=10080
      fi
    else
      DATENUM=50
      MINNUM=43200
    fi
    FINDER(){
      #days
      #find /$Path -mtime +"${DATENUM:=20}" -a -regex '.*logs.*' -a ! -regex '.*chss_config' -a ! -empty -exec date +%F_%T \; -exec rm -rv {} \; >> $FINDLOGFILE
      #Mins
      find /$Path -mmin +"${MINNUM:=43200}" -a -regex '.*logs.*' -a ! -regex '.*chss_config' -a ! -empty -exec date +%F_%T \; -exec rm -rv {} \; >> $FINDLOGFILE
    }
    FINDER
    #echo ---- /${Path} $DATENUM days rm ok
    echo ---- /${Path} $(( $MINNUM / 60 / 24 )) days rm ok
  else
    echo ---- /${Path} is not exist...
  fi
done >> ${RSYNCEXELOGFILE}