对Squid proxy Server的完全分析程序:sarg-2.0.2.tar.gz

昨天在网上看到SARG这个软件,就download下来,装在我的代理服务器上,哇噻,功能真是很强大.上网的记录完全的展现在你的面前,你浏览过的网页,下载过的东东,你的访问量,流量.......很全呀,不过,我都害怕了,这么全,我的上网记录不是全都爆光了:)

安装:

sarg-2.0.2.tar.gz

#tar -zxvf sarg-2.0.2.tar.gz

#cd sarg-2.0.2

#./configure

#make;make install

#vi /usr/local/sarg/sarg.conf         #根据你的实际情况修改了

#vi /etc/cron.daily/sarg.daily            #包括/etc/cron.weekly/sarg.weekly,/etc/cron.monthly/sarg.monthly,这些就是你的Squid proxy Server的日报表,周报表,月报表呀,不过如果你的流量很大的话,就不要做日报表了,太大了,很浪费你的硬盘空间.

以下是几个英文的说明档:我的采用的是Example One,大家可以根据自己的需求修改

sarg默认的输出的目录/var/www/html/,所以要创建/var/www/html/daily,monthly,weekly三个目录,或手动执行上述shell命令,在IE中打开,出来了吧
大家可以这里看一下我的抓图

Example One:

sarg.cron
# Please edit the following lines to match your requirements
# Also edit sarg.* scripts if you want to customize reports folder
# To enable report generation put last 3 lines into your /etc/crontab
# file.

05 0 * * * root /usr/sbin/sarg.daily
15 0 * * 1 root /usr/sbin/sarg.weekly
30 1 1 * * root /usr/sbin/sarg.monthly

sarg.monthly
#!/bin/bash

#Get yesterday date
YESTERDAY=$(date --date "1 day ago" +%d/%m/%Y)

#Get 1 month ago date
MONTHAGO=$(date --date "1 month ago" +%d/%m/%Y)

/usr/sbin/sarg -o /var/www/html/squid/monthly -d $MONTHAGO-$YESTERDAY
# > /dev/null 2>&1

/usr/sbin/squid -k rotate

exit 0

sarg.daily
#!/bin/bash
 
#Get yesterday date
YESTERDAY=$(date --date "1 day ago" +%d/%m/%Y)
 
/usr/sbin/sarg -o /var/www/html/squid/daily -d $YESTERDAY > /dev/null 2>&1
 
exit 0

sarg.weekly
#!/bin/bash

#Generate Access.log for correct weekly reports
cat /var/log/squid/access.log.0 /var/log/squid/access.log > /var/log/squid/access.log.week

#Get yesterday date
YESTERDAY=$(date --date "1 days ago" +%d/%m/%Y)

#Get one week ago date
WEEKAGO=$(date --date "7 days ago" +%d/%m/%Y)

/usr/sbin/sarg -l /var/log/squid/access.log.week -o /var/www/html/squid/weekly -d $WEEKAGO-$YESTERDAY > /dev/null 2>&1

exit 0

Example Two:

#!/bin/sh
# SARG - Daily/Weekly/Monthly Squid usage reports creation tool
# Written by Ugo Viti <>
# Visit (Italian Page)
VER=20040228

## What is this?
# sarg-reports (this file) is a simple bash script to automate
# the SARG (a powerful squid log analyzer) reports and log management.
# Sarg it self, provide to end user a generic interface to create
# reports based on squid log (begin of log to current date).
# sarg-reports (this script) is useful because it allow you to easly
# create and manage Daily, Weekly and Monthly reports.
# Try it, within 5 minutes you will be ready to rule :-)
# using sarg-reports is very easy, read the following 3 steps to know how

## Requirements
# a) An unix system with bash shell (like GNU/Linux, FreeBSD, etc...)
# b) Squid -
# c) Sarg -

##
## Installation guide and configuration parameters
##

# 1) Download Squid and Sarg, Install, Configure and Tune
#    they before continue reading

# 2) In root crontab (crontab -e) insert the following lines:
#    (the today report creation time depend mostly of your squid server
#    load average, tune it):
#
#    --- BEGIN ROOT CRONTAB ---
#    PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin
#    00 08-18/1 * * * sarg-reports today
#    00 00 * * * sarg-reports daily
#    00 01 * * 1 sarg-reports weekly
#    30 02 1 * * sarg-reports monthly
#    --- END ROOT CRONTAB ---
#
#    REMEMBER: if you use logrotate, configure it to rotate the logs within MONTHLY basis,
#              AFTER sarg-reports created the monthly html report.

# 3) Customize the following variables:
#    (Please, configure accurately the sarg.conf file before)
#
#        (SARG) The sarg executable location
#      (CONFIG) The sarg main configuration file location
#     (HTMLOUT) Location where will be saved the reports
#   (PAGETITLE) The title of main index page
#     (LOGOIMG) Image logo to view in main index page
#    (LOGOLINK) HTTP web page link of logo
#       (DAILY) Word 'daily' translation, translate it to your language
#      (WEEKLY) Word 'weekly' translation, translate it to your language
#     (MONTHLY) Word 'monthly' translation, translate it to your language
# (EXCLUDELOG1) Exclude text from cron emails
#       +       (normally, sarg, during cron activity, if it don't find any valid records,
# (EXCLUDELOG2) it will output an error message (usually on 'today' reports).
#               I don't want to be warned by email about this, so, i wrote the 'text'
#               that will be never logged.
#               This is useful to receive email of real problems only (enjoy that)

       SARG=/usr/sbin/sarg
     CONFIG=/etc/sarg/sarg.conf
    HTMLOUT=/var/www/virtual/i-synapse.it/www/html/admin/log/proxy
  PAGETITLE="Statistiche Proxy Squid di $(hostname)"
    LOGOIMG=/images/synapse-logo.jpg
   LOGOLINK=
      DAILY=Giornaliero
     WEEKLY=Settimanale
    MONTHLY=Mensile
EXCLUDELOG1="SARG: Nessun records trovato."
EXCLUDELOG2="SARG: Fine"

######################################################################
## The configuration is end, so don't touch anything bellow

# TEMP Files
TMPFILE=/tmp/sarg-reports.$RANDOM
 ERRORS="${TMPFILE}.errors"

# Date Calc
if [ "$(uname)" = "FreeBSD" ]
  then
    MANUALDATE=$2
         TODAY=$(date +%d/%m/%Y)
     YESTERDAY=$(date -v-1d +%d/%m/%Y)
       WEEKAGO=$(date -v-1w +%d/%m/%Y)
      MONTHAGO=$(date -v-1m +01/%m/%Y)-$(date -v-1m +31/%m/%Y)
  else
    MANUALDATE=$2
         TODAY=$(date --date "today" +%d/%m/%Y)
     YESTERDAY=$(date --date "1 day ago" +%d/%m/%Y)
       WEEKAGO=$(date --date "1 week ago" +%d/%m/%Y)
      MONTHAGO=$(date --date "1 month ago" +01/%m/%Y)-$(date --date "1 month ago" +31/%m/%Y)
fi

# Fix for Red Hat 9 systems and coreutils prior to 5.0 version
export LC_ALL=C

# Main index.html creation
create_index_html ()
{
  echo -e "
  n
  n
  n
  n
  n
 

n
    n
    n
      n
        n
      n
      n
        n
      n
      n
        n
      n
      n
        n
      n
   

$PAGETITLE

$DAILY

$WEEKLY

$MONTHLY

n
 

n
  n
  " > $HTMLOUT/index.html
}

reports_autorm () {
  BAKLIST="${TMPFILE}.BAKLIST"
  BAKLISTTOLEAVELIST="${BAKLIST}.TOLEAVE.LIST"
  BAKLISTTOLEAVEFIND="${BAKLIST}.TOLEAVE.FIND"
  touch $BAKLIST $BAKLISTTOLEAVELIST $BAKLISTTOLEAVEFIND

  cd $DESTINATION

  find -name "*$BAKNAME*" -maxdepth 1 | sed -e 's/.///' | grep -wv "." | sort 1> $BAKLIST

  let TOLEAVEDAYS="$DAYBAK+1"

  if [ $TOLEAVEDAYS -ge "0" ]
    then
      cat $BAKLIST | tail -n $TOLEAVEDAYS > $BAKLISTTOLEAVELIST
      for BAK in $(cat $BAKLISTTOLEAVELIST)
        do
          echo "! -name $BAK" >> $BAKLISTTOLEAVEFIND
        done
  fi

  CURRENTH=$(date +%H)
  CURRENTM=$(date +%M)
  if [ ${CURRENTH:0:1} == 0 ]
  then
    CURRENTH=${CURRENTH:1:1}
  fi
  if [ ${CURRENTM:0:1} == 0 ]
    then
      CURRENTM=${CURRENTM:1:1}
  fi
  let MINUTES="((($DAYBAK*24)+$CURRENTH)*60)+$CURRENTM+1"

  # Purging old backups, only if backup return "OK" status
  find $DESTINATION -name "*$BAKNAME*" $(cat $BAKLISTTOLEAVEFIND) -maxdepth 1 -mmin +$MINUTES -exec rm -rf {} ;
  #find $DESTINATION $(cat $BAKLISTTOLEAVEFIND) -maxdepth 1 -mmin +$MINUTES -exec rm -rf {} ;
}

# Functions
exclude_from_log ()
{
  cat $ERRORS | grep -v "$EXCLUDELOG1" | grep -v "$EXCLUDELOG2"
  rm -f $TMPFILE*
}

manual ()
{
  DAILYOUT=$HTMLOUT/$DAILY
  mkdir -p $DAILYOUT
  create_index_html
  if [ -z "$MANUALDATE" ]
    then
      echo "No date given, please specify a valid date (DD/MM/YYYY)"
    else
      $SARG -f $CONFIG -d $MANUALDATE -o $DAILYOUT
  fi
}

today ()
{
  DAILYOUT=$HTMLOUT/$DAILY
  mkdir -p $DAILYOUT
  create_index_html
  $SARG -f $CONFIG -d $TODAY -o $DAILYOUT >$ERRORS 2>&1
  exclude_from_log
}

daily ()
{
  DAILYOUT=$HTMLOUT/$DAILY
  mkdir -p $DAILYOUT
  create_index_html
  $SARG -f $CONFIG -d $YESTERDAY -o $DAILYOUT >$ERRORS 2>&1
  exclude_from_log
}

weekly ()
{
  WEEKLYOUT=$HTMLOUT/$WEEKLY
  mkdir -p $WEEKLYOUT
  create_index_html
  $SARG -f $CONFIG -d $WEEKAGO-$YESTERDAY -o $WEEKLYOUT >$ERRORS 2>&1
  exclude_from_log
}

monthly ()
{
  MONTHLYOUT=$HTMLOUT/$MONTHLY
  mkdir -p $MONTHLYOUT
  create_index_html
  $SARG -f $CONFIG -d $MONTHAGO -o $MONTHLYOUT >$ERRORS 2>&1
  exclude_from_log
}

case $1 in
    manual)
        manual
        ;;
    today)
        today
        ;;
    daily)
        daily
        ;;
    weekly)
        weekly
        ;;
    monthly)
        monthly
        ;;
     *)
        echo "SARG - Daily / Weekly / Monthly - Squid proxy usage reports creation tool"
        echo "Written by Ugo Viti <>"
        echo "Version: $VER"
        echo
        echo "Usage: $0 [OPTIONS]"
        echo
        echo "Allowed options:"
        echo "    manual,  Create Manual report"
        echo "     today,  Create Today report"
        echo "     daily,  Create Daily report"
        echo "    weekly,  Create Weekly report"
        echo "    montly,  Create Monthly report"
        exit 0
esac

## HISTORY:
# 20030826 - FreeBSD support (thanks to martijn to let me coding on your FreeBSD server :-))
# 20030715 - Some cleanups
# 20030623 - Manual report creation
# 20030620 - Main Index creation
# 20030619 - Solved 'sort' bug on Red Hat 9 systems
# 20030618 - First Version

## TODO:
# - Smarty weekly recognition...
#   Like "begin of last week to end of last week",
#   doesn't like this script do: "7 days ago to yesterday"
# - Monthly recognition isn't so elegant (is very ugly, i know)
# - Suggestions are welcome :-)
# - If you Rotate the squid logs before sarg-reports will run,
#   it will not create any html reports
#   (TIPS: Rotate the logs after sarg-reports)

Daily (Jeremy′s script):
-------------------------------------
 
#!/bin/bash
 
#Get current date
TODAY=$(date +%d/%m/%Y)
 
#Get one week ago today
YESTERDAY=$(date --date "1 day ago" +%d/%m/%Y)
 
/usr/local/bin/sqmgrlog -l /usr/local/squid/logs/access.log -o /usr/local/apache/htdocs/reports/daily -z -d $YESTERDAY-$TODAY
 
exit 0

 
 
Weekly
-------------------------------------
 
 
 
#!/bin/bash
 
#Get current date
TODAY=$(date +%d/%m/%Y)
 
#Get one week ago today
YESTERDAY=$(date --date "1 week ago" +%d/%m/%Y)
 
/usr/local/bin/sqmgrlog -l /usr/local/squid/logs/access.log -o /usr/local/apache/htdocs/reports/weekly -z -d $YESTERDAY-$TODAY
 
exit 0

 
 
 
Monthly
 
#!/bin/bash
 
#Get current date
TODAY=$(date +%d/%m/%Y)
 
#Get one week ago today
YESTERDAY=$(date --date "1 month ago" +%d/%m/%Y)
 
/usr/local/bin/sqmgrlog -l /usr/local/squid/logs/access.log -o /usr/local/apache/htdocs/reports/monthly -z -d $YESTERDAY-$TODAY
 
/usr/local/squid/bin/squid -k rotate
 
exit 0

 
 
 
 
 
And the same for the admin who want the report as an email (example for daily)
 
#!/bin/bash
 
#Get current date
TODAY=$(date +%d/%m/%Y)
 
#Get one week ago today
YESTERDAY=$(date --date "1 day ago" +%d/%m/%Y)
 
/usr/local/bin/sqmgrlog -l /usr/local/squid/logs/access.log -e -z -d $YESTERDAY-$TODAY
 
exit 0

SARG RedHat Packages
What is it?
SARG is a tool that parses your squid logs and generates usage reports.

Why should I need something like that?
SARG is a great administration tool. With its help you'll always know how good your proxy is working,
generated reports will help you to tune your configuration, and to discover if somebody
in your company is accessing "unwanted" content :)

This RPM packages would make it easier to install this tool on your RedHat system

How to install it?
All you need to do is to get the RPM binary for Redhat 9.0 or rebuild the source RPM for your own linux system.
Starting from Sarg 1.4.1 I provide RPM Binary only for RedHat 9.0 If you're using an older version,
feel free to get the SRPM package and rebuild it for your system.

After you have installed the rpm, you may wish to tune the configuration for your own needs.
1. Modify /etc/sarg/sarg.conf if neccessary
2. If you wish to generate daily, weekly and monthly reports, modify your /etc/crontab file (look for sarg.cron at the bottom of this page or in /etc/sarg folder)
3. Modify /usr/sbin/sarg.daily(weekly,monthly) scripts if neccessary
4. If you wish to use the scripts provided with this rpm package, disable system squid log rotation. In order to do that, just delete file /etc/logrotate.d/squid
5. If you want to access reports from the web, you will also need to configure your apache server accordingly.

Default behaviour after installation:
Sarg configuration is located in /etc/sarg
Sample SARG scripts for report generation as well as the SARG binary are located in /usr/sbin
During installation /var/www/html/squid/daily(weekly,monthly) folders are created
Default output folder for reports is /var/www/html/squid

The following table would help you to find the RPM binary you are looking for:

RedHat Version Latest RPM Binary
9.0 sarg-1.4.1-3.i386.rpm (latest version)
7.X sarg-1.2.1-1.i386.rpm (old version)

Latest changes to RPM (1.4.1-3)
* Mon Apr 05 2004 Sergey Dushenkov
- Small error in sarg.weekly was fixed
- sarg.cron sample was fixed

* Fri Apr 02 2004 Sergey Dushenkov
- Updated to Sarg 1.4.1 with latest patch (lots of new features, see the Changelog)
- Tuned sarg.weekly and sarg.monthly scripts
- Added documentation
- Added man help

* Wed Dec 05 2001 Sergey Dushenkov
- Updated to Sarg 1.2.1 (some bugfixes and new reports)
- Fixed sarg.cron and sarg.weekly scripts

* Tue Apr 04 2001 Sergey Dushenkov
- Updated to Sarg 1.1.1 (with Latvian language support)
- Beginning from now in RPM package for 6.2 all paths are set top 6.2 defaults:
/home/httpd/html (in both scripts and conf)

* Fri Mar 29 2001 Sergey Dushenkov
- Updated to Sarg 1.1.1
- bin location changed to /usr/sbin
- sample crontab is located now in /etc/sarg/sarg.cron
- all scripts now located at /usr/sbin/sarg.daily,weekly,monthly
- corrections made to sarg.monthly script, now it works properly
- Eliminating local/sbin dir
- many fixes in rpm

What if I don't use RedHat or my redhat version is not 9.0?
In that case you will need to rebuild the source rpm (see next section)

Rebuilding the SRPM
When you have a pretty unique configuration and there is no binary for your plattform you have to build your own ;)
You can download the source RPM and rebuild it on your system
All you would need to do is to execute the following command:

rpmbuild --rebuild sarg-1.4.1-3.src.rpm
Then install the resulting binary RPM on your system.
Also don't forget to change the /etc/sarg/sarg.conf file to fit your system

Sample scripts for your Linux box
Below you will find a set of the scripts I'm using on my own system (they are included with RPM package).
These scripts generate reports and manage squid logs rotaion.

Script Name Short description
sarg.daily Generates daily reports
sarg.weekly Generates weekly reports
sarg.monthly Generates monthly reports
sarg.cron Sample scrontab entires for the scripts above

And one more note, the scripts above will generate reports in /var/www/html/squid/daily,weekly,monthly
If you want to generate reports in any other locations, then edit the appropriate scripts