Chinaunix首页 | 论坛 | 博客
  • 博客访问: 445737
  • 博文数量: 403
  • 博客积分: 0
  • 博客等级: 民兵
  • 技术积分: -70
  • 用 户 组: 普通用户
  • 注册时间: 2016-09-05 12:45
文章分类

全部博文(403)

文章存档

2014年(3)

2013年(1)

2012年(3)

2011年(21)

2010年(13)

2009年(64)

2008年(9)

2007年(36)

2006年(253)

分类:

2006-11-01 16:37:51

超強 squid log 分析器(Squid Analysis Report Generator)

作者:Eric
出處:小紅帽技術論壇
聲明:歡迎非商業行為轉載,轉載請註明出處。

前言:
squid 是一套功能與性能強大的免費代理伺服器,相信網路上不少代理伺服器都是使用 squid ,強大的 squid 唯一的缺點,記錄檔過於複雜,管理人員非常不容易分析使用記錄,還好有這套 Squid Analysis Report Generator ,其產生的報表可以讓我們輕鬆的掌握 squid 使用記錄。如果搭配 Linux NAT + Transparent Proxy ,內部電腦瀏覽網站分析將無所遁形,管理人員可以非常輕易分析使用者上網狀況,進而加以調整 squid and iptables !!

註:呵~~ 一般行政人員如果上xx網站,因此篇教學導致被抓包,不要來罵我!

軟體名稱:SARG (Squid Analysis Report Generator)
網站位址:
目前最新版本:Source 1.3.PRE2 ,RPM 1.2.1

安裝步驟:
1.
首先取得 SARG 檔案,請自行選擇 source 型式。

2.
安裝 SARG

code:

A. rpm 安裝
rpm : rpm -ivh 套件名
設定檔目錄預設為 /etc/sarg

B. 編譯 安裝
解壓 sarg-1.3-PRE2.tar.gz

執行
./configure
make
make install

安裝位置預設為 /usr/local/sarg



3.
設定 sarg.conf
註:rpm 安裝:/etc/sarg/sarg.conf
  編譯安裝:/usr/local/sarg/sarg.conf

以下只挑重點部份:
language 分析表使用之語言。
access_log 欲分析之 squid 之記錄檔位置。
title 分析報表之抬頭說明。
temporary_dir 分析時之資料暫存位置。
output_dir 分析後之報表輸出位置。
output_email 分析報表 email 寄送位置。
topuser_sort_field
user_sort_field 分析表排序欄位。
overwrite_report 分析表是不是要覆蓋。
charset 網頁語系設定。

範例:(其餘使用預設,或自行參考說明調整)
code:

language English
access_log /var/log/squid/access.log
title "Proxy 使用分析表"
temporary_dir /tmp
output_dir /var/www/html/squid
#output_email none
charset big5
overwrite_report no


4.
產生報表
方式有很多種:日報表、週報表、月報表、總報表....
以下以 rpm 安裝方式說明:
以 root run :
/usr/sbin/sarg (產生總報表)
/usr/sbin/sarg.daily (產生日報表)
/usr/sbin/sarg.weekly (產生週報表)
/usr/sbin/sarg.monthly (產生月報表)

以編譯安裝的可參考上述 scripts !!
sarg.daily
code:

#!/bin/bash

#Get yesterday date
YESTERDAY=$(date --date "1 day ago" +%d/%m/%Y)

/usr/sbin/sarg -o /var/www/html/squid/daily -d $YESTERDAY > /dev/null 2>&1

exit 0


sarg.weekly
code:

#!/bin/bash

#Get yesterday date
YESTERDAY=$(date --date "1 day ago" +%d/%m/%Y)

#Get one week ago date
WEEKAGO=$(date --date "1 week ago" +%d/%m/%Y)

/usr/sbin/sarg -o /var/www/html/squid/weekly -d $WEEKAGO-$YESTERDAY > /dev/null
2>&1

exit 0
[root@proxy daily]# more /usr/sbin/sarg.weekly
#!/bin/bash

#Get yesterday date
YESTERDAY=$(date --date "1 day ago" +%d/%m/%Y)

#Get one week ago date
WEEKAGO=$(date --date "1 week ago" +%d/%m/%Y)

/usr/sbin/sarg -o /var/www/html/squid/weekly -d $WEEKAGO-$YESTERDAY > /dev/null 2>&1

exit 0


sarg.monthly
code:

#!/bin/bash

#dynamic counter
cnt=2

if [ $cnt -eq 4 ]; then
#Get yesterday date
YESTERDAY=$(date --date "1 day ago" +%d/%m/%Y)

#Get 4 weeks ago date
WEEKSAGO=$(date --date "4 weeks ago" +%d/%m/%Y)

/usr/sbin/sarg -o /var/www/html/squid/monthly -d $WEEKSAGO-$YESTERDAY > /dev/null 2>&1

/usr/sbin/squid -k rotate

#don't move next line to upper, reason is that sed change the cnt assignment of the first 7 lines
cnt=1
else
let cnt++
fi
#echo Will rename itself \($0\) with cnt \($cnt\) increased. 1>&2
sargtmp=/var/tmp/`basename $0`
sed "1,7s/^cnt=.*/cnt=$cnt/
" $0 >|$sargtmp
chmod -f 775 $sargtmp
mv -f $sargtmp $0


5. 注意事項:
a. 以 RedHat 7.x 來說,squid access.log 會每周日零晨四點自動備份並清除,所以如果要做周報表與月報表就要去修改系統的 squid log 備份方式。
b. 一般我們只需做 日報 及 周報 表即可。
c. 上述 產生報表 scripts ,其報表存放位置為
日報表:/var/www/html/squid/daily
周報表:/var/www/html/squid/weekly
月報表:/var/www/html/squid/monthly
總報表:/var/www/html/squid

6. 特殊技巧
系統預設未有 中文語系檔,我們可以自行製作翻譯。
方法如下:
複製一份英文語系檔
cp /etc/sarg/languages/English /etc/sarg/languages/Taiwan
翻譯修改 /etc/sarg/languages/Taiwan (有人願意分享嗎??)
修改 sarg.conf
將 language English 修改為
language Taiwan (注意大小寫)

這樣就大功告成了,趕快用 IE 來查看一番囉~~

附錄(英文說明)
code:

Sarg is a Squid Analysis Report Generator that allow you to view "where" your users
are going to on the Internet.
Sarg generate reports in html, with many fields, like: users, IP Addresses, bytes, sites and times.

Support to SmartFilter added - 09/02/2000

Special thanks to:
Matteo Colombo - Italian language
Palamarchuk Eugen - Russian language
Csaba Kabai - Hungarian language
Evren Yurtesen - Turkish language
Andreas Piek - German language
Fred Pacquier - French language
Leonardo A. D'Angelo - Spanish language
Seth Mos - Dutch language
Milos Prudek - Czech language
Dima I. Allaverdov - Russian_windows1251 language
V Gatut Harijiso - Indonesian language
Ivan Minchev - Bulgarian_windows1251 language
Akira Kitamura - Japanese language
Radovan Drobnjakovic - Serbian language
Pieter Kooistra - Dutch language fix
Andrew Okhmat - Russian Koi8 language fixed
Nikolai V. Ivanyushin - Russian and Bulgarian language fix
Juris Valdovskis - Latvian language
Jordan Kanev - Bulgarian language fix
Wszebor Boksa - Polish language
Adi Cretu - Romanian language

Translating
. Copy include/English to YourLanguage
. Translate YourLanguage
. Send YourLanguage to to implement


If you use native squid log format, the elapsed time will be in reports (emulate_httpd_log off).

1. run ./configure

configure options: --enable-bindir=where sarg binary will be saved
default: /usr/bin

--enable-sysconfdir - where the configuration directory is
default: /usr/local/sarg

--enable-mandir - where the sarg man page will be saved
default: /usr/local/man/man1


3. make

4. make install

5. Go to /usr/local/sarg (or file entered with --sysconfdir on configure)
and change sarg.conf as you need.

6. Notes about sarg:

Date/Time report:
Every minute that a request is logged your time is incremented by the smaller
of 1 minute or the total time for the requests.

Usage: sarg -h

Source:

Any suggests and/or comments, plase:


# sarg.conf
#
# TAG: language
# Available languages:
# Bulgarian_windows1251
# Czech
# Dutch
# English
# French
# German
# Hungarian
# Indonesian
# Italian
# Japanese
# Latvian
# Polish
# Portuguese
# Romanian
# Russian_koi8
# Russian_windows1251
# Serbian
# Spanish
# Turkish
#
#language English

# TAG: access_log file
# Where is the access.log file
# sarg -l file
#
#access_log /usr/local/squid/logs/access.log
#access_log /var/log/squid/access.log # RedHat version

# TAG: title
# Especify the title for html page.
#
#title "Squid User Access Reports"

# TAG: font_face
# Especify the font for html page.
#
#font_face Arial

# TAG: header_color
# Especify the header color
#
#header_color darkblue

# TAG: header_bgcolor
# Especify the header bgcolor
#
#header_bgcolor blanchedalmond

# TAG: font_size
# TAG: font_size
# Especify the font size
#
#header_font_size -1

# TAG: background_color
# TAG: background_color
# Html page background color
#
#background_color white

# TAG: text_color
# Html page text color
#
#text_color black

# TAG: text_bgcolor
# Html page text background color
#
#text_bgcolor beige

# TAG: title_color
# Html page title color
#
#title_color green

# TAG: logo_image
# Html page logo.
#
#logo_image none

# TAG: logo_text
# Html page logo text.
#
#logo_text ""

# TAG: logo_text_color
# Html page logo texti color.
#
#logo_text_color black

# TAG: logo_image_size
# Html page logo image size.
# width height
#
#image_size 80 45

# TAG: background_image
# Html page background image
#
#background_image none

# TAG: password
# User password file used by authentication
# If you put here, reports will be generated only for users in.
#
#password none

# TAG: temporary_dir
# Temporary directory name
# sarg -w dir
#
#temporary_dir /tmp

# TAG: output_dir
# Where is the reports will ne stored.
# sarg -o dir
#
#output_dir /usr/local/etc/httpd/htdocs/squid-reports
#output_dir /home/httpd/html/squid-reports # RedHat version

# TAG: output_email
# Email address to send the reports
# sarg -e email
#
#output_email none

# TAG: resolve_ip yes/no
# Convert ip address to dns name
# sarg -n
#resolve_ip no

# TAG: user_ip yes/no
# Use Ip Address instead userid (reports)
# sarg -p
#user_ip no

# TAG: topuser_sort_field field normal/reverse
# Sort field for the Topuser Report.
# Allowed fields: USER CONNECT BYTES TIME
#
#topuser_sort_field BYTES reverse

# TAG: user_sort_field field normal/reverse
# Sort field for the User Report.
# Allowed fields: SITE CONNECT BYTES TIME
#
#user_sort_field BYTES reverse

# TAG: exclude_users file
# users within the file will be excluded from reports.
# you can use indexonly to have only index.html file.
#
#exclude_users none

# TAG: exclude_hosts file
# Hosts, domains or subnets will be excluded from reports.
#
# Eg.: 192.168.10.10 - exclude ip address only
# 192.168.10.0 - exclude full C class
# s1.acme.foo - exclude hostname only
# acme.foo - exclude full domain name
#
#exclude_hosts none

# TAG: useragent_log file
# Put here where is useragent.log to nable useragent report.
#
#useragent_log none

# TAG: date_format
# Date format in reports: e (Europe=dd/mm/yy), u (USA=mm/dd/yy), w (Weekly=yy.ww)
#date_format u

# TAG: per_user_limit file MB
# Save userid on file if download exceed n MB.
#
# This option can be used to disable user access if user exceed a download limit.
#per_user_limit none

# TAG: lastlog n
# How many reports files must be keept in reports directory.
# The oldest report file will be automatically removed.
# 0 - no limit.
#
#lastlog 0

# TAG: remove_temp_files yes
# Remove temporary files: geral, usuarios, top, periodo from root report directory.
#
#remove_temp_files yes

# TAG: index yes|no|only
# Generate the main index.html.
# only - generate only the main index.html
#
#index yes

# TAG: overwrite_report yes|no
# yes - if report date already exist then will be overwrited.
# no - if report date already exist then will be renamed to filename.n, filename.n+1
#
#overwrite_report no

# TAG: records_without_userid ignore|ip|everybody
# What can I do with records without user id (no authentication) in access.log file ?
#
# ignore - This record will be ignored.
# ip - Use ip address instead. (default)
# everybody - Use "everybody" instead.
#
#records_without_userid ip

# TAG: use_comma no|yes
# Use comma instead point in reports.
# Eg.: use_comma yes => 23,450,110
# use_comma no => 23.450.110
#
#use_comma no

# TAG: mail_utility mail|mailx
# Mail command to use to send reports via SMTP
#
#mail_utility mailx

# TAG: topsites_num n
# How many sites in topsites report.
#
#topsites_num 100

# TAG: topsites_sort_order CONNECT|BYTES A|D
# Sort for topsites report, where A=Ascendent, D=Descendent
#
#topsites_sort_order CONNECT D

# TAG: exclude_codes file
# Ignore records with these codes. Eg.: NONE/400
#
#exclude_codes /usr/local/sarg/exclude_codes

# TAG: replace_index string
# Replace "index.html" in the main index file with this string
# If null "index.html" is used
#
#replace_index

# TAG: max_elapsed milliseconds
# If elapsed time is recorded in log is greater than max_elapsed use 0 for elapsed time.
# Use 0 for no checking
#
#max_elapsed 0
# 8 Hours
max_elapsed 28800000

# TAG: report_type type
# What kind of reports to generate.
# topsites - shows the site, connect and bytes
# sites_users - shows which users were accessing a site
# users_sites - shows sites accessed by the user
# date_time - shows the amount of bytes used by day and hour
# denied - show all denied sites with full URL
# auth_failures - show autentication failures
#
# Eg.: report_type topsites denied
#
#report_type topsites users_sites sites_users date_time denied auth_failures site_user_time_date

# TAG: usertab filename
# You can change the "userid" or the "ip address" to be a real user name on the rpeorts.
# Table syntax:
# userid name or ip address name
# Eg:
# SirIsaac Isaac Newton
# vinci Leonardo da Vinci
# 192.168.10.1 Karol Wojtyla
#
# Each line must be terminated with '\n'
#
#usertab none

# TAG: long_url yes|no
# If yes, the full url is showed in report.
# If no, only the site will be showed
#
# YES option generate very big sort files and reports.
#
#long_url no

# TAG: date_time_by bytes|elap
# Date/Time reports will use bytes or elapsed time?
#
#date_time_by bytes

# TAG: charset name
# ISO 8859 is a full series of 10 standardized multilingual single-byte coded (8bit)
# graphic character sets for writing in alphabetic languages
# You can use the following charsets:
# Latin1 - West European
# Latin2 - East European
# Latin3 - South European
# Latin4 - North European
# Cyrillic
# Arabic
# Greek
# Hebrew
# Latin5 - Turkish
# Latin6
# Windows-1251
# Koi8-r
#
#charset Latin1

# TAG: user_invalid_char "&/"
# Records that contain invalid characters in userid will be ignored by Sarg.
#
#user_invalid_char "&/"

# TAG: privacy yes|no
# privacy_string "***.***.***.***"
# privacy_string_color blue
# In some countries the sysadm cannot see the visited sites by a restrictive law.
# Using privacy yes the visited url will be changes by privacy_string and the link
# will be removed from reports.
#
#privacy no
#privacy_string "***.***.***.***"
#privacy_string_color blue

# TAG: include_users "user1:user2:...:usern"
# Reports will be generated only for listed users.
#
#include_users none

# TAG: exclude_string "string1:string2:...:stringn"
# Records from access.log file that contain one of listed strings will be ignored.
#
#exclude_string none

# TAG: show_successful_message yes|no
# Show "Successful report generated on dir" at end of process.
#
# show_successful_message yes

# TAG: topuser_fields
# Which fields must be in Topuser report.
#
#topuser_fields NUM DATE_TIME USERID CONNECT BYTES %BYTES IN-CACHE-OUT USED_TIME MILISEC %TIME TOTAL AVERAGE

# TAG: topuser_num n
# How many users in topsites report. 0 = no limit
#
#topuser_num 0

 #by Pedro Lineu Orso
"Yes"是
"No"否
"English"英語
"Option"選項
"require an argument"須要一個引數
"Init"起始
"Reading access log file"讀取存取記錄檔
"Cannot open log file"無法開啟記錄檔
"Cannot open temporary file"無法開啟暫存檔
"Records read"讀記錄檔
"written"已寫入
"Log with mixed records format (squid and common log)"最大記錄格式
"Common log format"
"Squid log format"
"Log with invalid format"記錄無效格式
"No records found"沒有發現記錄
"Period"期間
"using"使用
"as temporary dir"暫存目錄
"Sorting file"檔案排序
"End"結束
"Parameters"參數
"Hostname or IP address"主機名稱或IP位址
"Date from-until"起止日期
"Date format"日期格式
"Europe"歐洲
"USA"美國
"IP report"IP報告
"Use Ip Address instead userid"使用IP位代替使用者名稱
"Accessed site"存取網站
"Time"時間
"User"使用者
"Squid version"Squid版本
"Temporary dir"暫存目錄
"Debug messages"除錯訊息
"Process messages"處理訊息
"Input log"輸入記錄
"Output dir"輸出目錄
"Usage"使用
"options"選項
"Email address to send reports"用Email寄記錄檔
"stdout for console"
"Reports by user and IP address"以使用者及IP位址報告
"reports"報告
"Cannot open file"無法開啟檔案
"DENIED"拒絕
"Successful report generated on"成功產生報告檔
"Successfull report generated and sent to"成功產生且傳出報告檔
"Making file"製作檔案
"Making period file"製作檔案期間
"File"檔案
"already exist, moved to"已經存在,移至
"Making index.html"製作index.html
"Sorting file"排序檔案
"Report"報告
"Squid not installed on this machine"這台機器上沒有安裝Squid
"Please, use -v option to set Squid version"請用-v的選項去設定Squid的版本
"Loading configuration from"載入組態從
"malloc error"malloc錯誤
"Loading password file from"載入密碼檔案從
"Making report"製作報告
"Decompressing log file"記錄檔案解壓縮
"Compressing log file"記錄檔案壓縮
"File not found"找不到檔案
"Resolve IP Address"解除IP住址
"Reading useragent log"讀取代理人記錄檔
"Loading exclude file from"載入執行檔
"excluded"排除
"Exclude file"排除的檔案
"Config file"組態檔
"Useragent log"代理記錄檔
"Making Useragent report"製作代理記錄檔
"version"版本
"limit exceeded"超過限制
"Added to file"加入檔案
"Convert the access.log file to a legible date"轉換access.log檔
"Split the log file by date in -d parameter"
"by"由
"reverse"相反
"normal"正常的
"Removing old report file"移除舊的報告檔
"Removing temporary files"移除舊的暫存檔
"Top"首要的
"sites"網站
"Sites & Users"網站&使用者
"Loading User table"載入使用者表格
"Cannot load. Memory fault"不能載入記憶體錯誤
"Squid User Access Report"Squid使用者存取報告
"Period"期間
"User"使用者
"ACCESSED SITE"存取網站
"CONNECT"連接
"BYTES"位元
"USED TIME"使用時間
"MILISEC"
"AVERAGE"平均
"Decreasing Access (bytes)"減少存取(位元)
"USERID"使用者代號
"TIME"時間
"NUM"編號
"FILE/PERIOD"檔案/期間
"CREATION DATE"統計日期
"USERS"使用者
"Sort"排序
"Squid Useragent's Report"Squid代理使用者報告
"AGENT"代理者
"TOTAL"總計
"Generated by"產生者
"on"在
"DATE/TIME"日期/時間
"IP/NAME"IP/名稱
"OUT"出
"IN"入
"CACHE"快取
"SitesUsers"網站使用者
"SmartFilter"精明的過濾
"Authentication Failures"認證失敗
"Denied"拒絕
"Topsites"熱門網站

阅读(4897) | 评论(0) | 转发(0) |
给主人留下些什么吧!~~