FreeBSD7.0下通过CARP+PF+PFSYNC实现服务器集群

硬件:BSD爱好者乐园 Z@V'? vP
hp_
服务器                5台(实际只用了4台, 2台做LoadBalance(主/备各一,每台至少需要三块网卡), 2台做应用服务.)
Z:nchb:A W*ccisco2950        2台BSD爱好者乐园2| u.E9M-s$Z5E
P

{@;R1wkP)|系统: 7.0 (最小化)
d-u:Usm P ~,N
w1H4}2X5j我部署环境的测试地址(不能长期保留, 群集部署是在内网中,所以外网通过重定向来carp0上的):
.q9|8@/Z4k
A2mBSD爱好者乐园 XKL Ply,cn o
一 网络拓扑:
h3gj4mlMR(qm K外网: 192.168.1.0/24
|w;MWU$\a8R内网: 192.168.10.0/24BSD爱好者乐园7|3pR;k(i6n,|@:s2?)W
HA专用: 10.10.10.250 / 10.10.10.251BSD爱好者乐园V"|#y1[
?"vY/k5O%S
BSD爱好者乐园!Z Xd p-K4JVBSD爱好者乐园*j0GP R;Dh

二 Load Balance配置:BSD爱好者乐园)v i FSB%\#L
BSD爱好者乐园8QO'r"?&\!y
A.内核设置:BSD爱好者乐园U+?+ug*tr
master/slave的配置是一样的
!V%Cq?,rz Edevice
P;I4dUL:Vdevice pf        #启动虚拟网络设备来记录流量(经由 bpf)
(N}1xU.R:I mdevice pflog     #启动虚拟网络设备来监视网络状态
R(ti9BQp P ?device pfsyncBSD爱好者乐园/k2G c4g+rF-j T

({^5PR$~rXoptions         ALTQBSD爱好者乐园{ Gu-h&s#@%A
options         ALTQ_CBQ        # 基于分类的排列 (CBQ)
wBs%rg \#C"nr%C(Eoptions         ALTQ_RED        # 随机先期检测 (RED)
4W6u/t0Tb |]Foptions         ALTQ_RIO        # 对进入和发出的包进行 RED
&e,d1?%w"_options         ALTQ_HFSC       # 带等级的包调度器 (HFSC)BSD爱好者乐园Z yw jD%G l
options         ALTQ_PRIQ       # 按优先级的排列 (PRIQ)
/ixp^{goptions         ALTQ_NOPCC      # 在联编 SMP 内核时必须使用，禁止读时钟BSD爱好者乐园F C9N.Q5A
BSD爱好者乐园W)@(} r{ n9oE
重编内核.BSD爱好者乐园F"H%B~$QW']n D
BSD爱好者乐园,ja}#@Cw b"E&t
B.配置rc.conf
aA\\[yj {这里只写与本章节相关的配置项了.BSD爱好者乐园x*HJ T3p.Pd N,Y
master部分:
5{4\1B6]~ Jgateway_enable="YES"
L1{1geO {:Sdefaultrouter="192.168.1.1"
8kMcjh6vBw"\#Wnhostname="master.cluster.org"
^
}0mC T#{cloned_interfaces="carp0 carp1"BSD爱好者乐园)j!hP,W;Xub LI

6w(k"Nb qh5p`# External Public Interface (for the secondary firewall use a different public ip.)BSD爱好者乐园YF$h1xa:oV(o
ifconfig_em0="inet 192.168.1.52 netmask 255.255.255.0"BSD爱好者乐园1t5~/iY`h B!Nv
# External Public Carp InterfaceBSD爱好者乐园 qX,[/N0xr9k1r
#ifconfig_carp0="vhid 1 pass 11111 192.168.1.51/24"BSD爱好者乐园,bFp V-m h)E4L5]M
ifconfig_carp0="vhid 1 pass 11111 192.168.1.51/24 advskew 10"BSD爱好者乐园UTW$R.nc1E6[5M

_doW-f# Internal Interface (for the secondary firewall change the ip address to 192.168.10.11)BSD爱好者乐园.I8U)d ?"w \ ed2{
ifconfig_em1="inet 192.168.10.10 netmask 255.255.255.0"
M
P^-|WWZj# Internal Carp Interface
|{`:r.?,Q9G0~ifconfig_carp1="vhid 1 pass 22222 192.168.10.100/24 advskew 10"
1lzXg5\QJT!Q t
,l }w0l3?a# Heartbeat Interface (for the secondary firewall, change the ip address to 10.10.10.251)
6~2qCuZuifconfig_vr0="10.10.10.250 netmask 255.255.255.0"BSD爱好者乐园?U K
`rJd
BSD爱好者乐园sJna!??(T7XH
# PFSync InterfaceBSD爱好者乐园4f0CF6a+B^V
ifconfig_pfsync0="up syncif vr0"
V G&`T#R0[XBSD爱好者乐园E7Qa"F"_ K
pf_enable="YES"BSD爱好者乐园E%s lWp}
pf_rules="/etc/pf.conf"BSD爱好者乐园#X5Iid W8B(E {nL
pf_flags=""
I3~0k G|h3V @9Cmpflog_enable="YES"BSD爱好者乐园 ri;b$Cx*u,}$xO_9h
pflog_logfile="/var/log/pflog"BSD爱好者乐园t#@ \:|"X
pflog_flags=""BSD爱好者乐园3d CluB&w4ZK d
BSD爱好者乐园*X[P I ^](kj
slave部分:BSD爱好者乐园OJ` IJ/g|
#slave的配置与master配置大概相近了. 改动的主要是本机IP和advskew的优先值. 注意: carp 的IP是公共的必须一致.
%AB,fU:N9jX;|gateway_enable="YES"
p/Hbe3S f!aWdefaultrouter="192.168.1.1"BSD爱好者乐园b:@U,Y9\#}[*Sw
hostname="slave.cluster.org"
UtV3U#DH|H}cloned_interfaces="carp0 carp1"
l c|bd}m
EBSD爱好者乐园P8{!IF;\C
# External Public Interface (for the primary firewall use a different public ip.)BSD爱好者乐园 m,?7Z {S
ifconfig_em0="inet 192.168.1.53 netmask 255.255.255.0"BSD爱好者乐园!O.rC9c l){$tz
# External Public Carp InterfaceBSD爱好者乐园 a6v|L:xk1zm
#ifconfig_carp0="vhid 1 pass 11111 192.168.1.51/24"BSD爱好者乐园qO6]`0VD(m*Q
ifconfig_carp0="vhid 1 pass 11111 192.168.1.51/24 advskew 20"BSD爱好者乐园Ao:rF
U
BSD爱好者乐园 mD Hxx-D
# Internal Interface (for the primary firewall change the ip address to 192.168.10.10)
4R4lVNRv5z C[ifconfig_em1="inet 192.168.10.11 netmask 255.255.255.0"BSD爱好者乐园z-J;jP0E
# Internal Carp Interface
P6U6c/l6t$`ifconfig_carp1="vhid 1 pass 22222 192.168.10.100/24 advskew 20"BSD爱好者乐园H5z:BrC
BSD爱好者乐园R ?/|^+X:H\:o'F
# Heartbeat Interface (for the primary firewall, change the ip address to 10.10.10.250)BSD爱好者乐园n/Z)V
l3Nj
ifconfig_fxp0="10.10.10.251 netmask 255.255.255.0"BSD爱好者乐园1lF7h1v9p-pfP

B'| yz:e ?D:F# PFSync Interface
yg*@P`QB.gifconfig_pfsync0="up syncif fxp0"
o*SM H/A PjBSD爱好者乐园!Y/DV^0L*jqe{
pf_enable="YES"BSD爱好者乐园@tJK5Y`
pf_rules="/etc/pf.conf"
&` xk,aelpf_flags=""
4nf:d ~ ?B-?Vpflog_enable="YES"
.@JaU|w8J m(b?Jnpflog_logfile="/var/log/pflog"BSD爱好者乐园-D
|H&o%n$t@ T
pflog_flags=""
:]l?4Bka
&oY*Iy kOC.pf.conf规则BSD爱好者乐园zMO8C[#Rp
master和slave除了网卡标识不一样.其他是一致的.BSD爱好者乐园)dD$X3v^:^ t
################################################################################
6G7\%@E/c# Macro and listsBSD爱好者乐园4J6Y9k$J:LZ [
################################################################################BSD爱好者乐园$tQaMw$FJ3{
lop_if = "lo0"

U c:Ex3B2}ext_if = "em0"
q'q2I*`3^r}i?6jJiint_if = "em1"
0v1cc9L|K9PNsync_if= "vr0"
a)G,n6c+Tf4~ext_carp = "carp0"
O
d9r/gxG'w
%gW_.n/D1S?web_ports = "{ 80, 443 }"
;j,zi.uZ G#i+rT#web_servers = "{ 192.168.10.20, 192.168.10.21, 192.168.10.22 }"BSD爱好者乐园-geX(n Ix
#web_servers = "{ 192.168.10.20 }"
#zA7X&r \p K}bweb_servers = "{ 192.168.10.20, 192.168.10.21 }"BSD爱好者乐园4Q#EKY*wUX1L
BSD爱好者乐园V|K:` ^+}|1xI


RO4f(n3vZ6Iv5PG?################################################################################BSD爱好者乐园.o_B%}LA4B hsd
# Options, scrub and NATBSD爱好者乐园BMigi{
################################################################################
aG
R nWd$et vO#{set block-policy dropBSD爱好者乐园P5h,Y~ _"J2NdN
set skip on $lop_if
/S5c8m+Dc5~8V"~gJBSD爱好者乐园"No$hqS)qVZWX
scrub inBSD爱好者乐园R.s:iS.[+h
BSD爱好者乐园(WM0j g7a Fq p)D7r
nat on $ext_if from $int_if:network to any -> $ext_ifBSD爱好者乐园&C5jMv(Ic
BSD爱好者乐园&u?P$AY.p+g'Ux
t ?
################################################################################
!d G l)Xc0r0]R# Redirection
,GMm~B6hz#u################################################################################
(P"\.|X0O!R#rdr on $ext_if proto tcp from any to any port 80 -> $web_servers round-robin sticky-addressBSD爱好者乐园Q.Q-M;hW&k+W
rdr on $ext_if proto tcp from any to $ext_carp port $web_ports -> $web_servers round-robin sticky-addressBSD爱好者乐园T6}1Op K0Gr

xu ]+b)W*cu
2^3U9d^0P'H.m8Int0O q################################################################################
F|l-V&u|$j# Filtering Rules
1T5k$zu3o+MU################################################################################BSD爱好者乐园,Ae5qS'L })t)E^:cr
BSD爱好者乐园3YD,E2x4UExr
pass quick on { $sync_if } proto pfsync keep state (no-sync)
@/|_s [J"]$`7Tpass on { $ext_if,$int_if } proto carp keep stateBSD爱好者乐园 uP nmv-UP!N:f(J
BSD爱好者乐园*G
Rfm4[1F"e[;M
D. sysctl.conf的相关配置
(r%^ pI)BsE-V |Z F#master和slave 一样.
5^a0y `9S~Bnet.inet.carp.preempt=1
H {^@Wt
TBSD爱好者乐园'pn3Bck4z
net.inet.tcp.blackhole=2BSD爱好者乐园j%xoC/g^
net.inet.udp.blackhole=1
'Q6Sww8?5re5o IBSD爱好者乐园9q:hLX/Y K
net.inet.tcp.sendspace=65536
o/e hZ%?7pnet.inet.tcp.recvspace=65536BSD爱好者乐园 Dt)D` n5ti P"{

*]9XYQAf3Tg~d(IE. 配置好后重启服务器.检查系统状况BSD爱好者乐园"zn^Jf'RhD
master上BSD爱好者乐园a
?9^hw {7T
master# ifconfigBSD爱好者乐园2U V$B-Vxv
vr0: flags=8843 metric 0 mtu 1500
?/p4aX"R        options=8
%Aq~G I
y*x        ether 00:05:5d:85:84:d8BSD爱好者乐园&@ta#q RTg{
        inet 10.10.10.250 netmask 0xffffff00 broadcast 10.10.10.255BSD爱好者乐园iH~ ~Pr
        media: Ethernet autoselect (100baseTX )
0P*Ye9O(W,B;im        status: activeBSD爱好者乐园n"A\ ^e*X"o2t \i
em0: flags=8943 metric 0 mtu 1500
F&`7f3B1I:j'Qp1u        options=9b
9s)[r3X4p[        ether 00:c0:9f:31:25:a2BSD爱好者乐园_B#l5DH TK
        inet 192.168.1.52 netmask 0xffffff00 broadcast 192.168.1.255
a3M]9Ps CX'q        media: Ethernet autoselect (100baseTX )
*H I i@w]
JT        status: activeBSD爱好者乐园+S3q2b Flb_v N2x"|y
em1: flags=8943 metric 0 mtu 1500BSD爱好者乐园;pfvT
p:m"vP y+K!^
        options=9bBSD爱好者乐园6X%zZ$gj
        ether 00:c0:9f:31:25:a3BSD爱好者乐园 K(]9O6UN5cF2uJA/R
        inet 192.168.10.10 netmask 0xffffff00 broadcast 192.168.10.255BSD爱好者乐园+x)S(i N7zq
        media: Ethernet autoselect (100baseTX )BSD爱好者乐园4D+i9y[;LZc3Y,h
        status: activeBSD爱好者乐园K(L(F}6Ml[
pflog0: flags=141 metric 0 mtu 33204BSD爱好者乐园*HPix } oK"P
lo0: flags=8049 metric 0 mtu 16384

GE)N {8W/Z4U2`$M        inet 127.0.0.1 netmask 0xff000000
6i \"w bN:X&F"nMpfsync0: flags=41 metric 0 mtu 1460BSD爱好者乐园 kA4Cp(N\ sL
        pfsync: syncdev: vr0 syncpeer: 224.0.0.240 maxupd: 128BSD爱好者乐园$Tc)b,j ?
carp0: flags=49 metric 0 mtu 1500BSD爱好者乐园8v0m6z*}P3S
        inet 192.168.1.51 netmask 0xffffff00
"@c%?J4a g        carp:MASTERvhid 1 advbase 1advskew 10
d(v {j8qocarp1: flags=49 metric 0 mtu 1500BSD爱好者乐园h|+c L"WO/R-f
        inet 192.168.10.100 netmask 0xffffff00
4k,?$G7p!wo        carp:MASTERvhid 1 advbase 1advskew 10
x!fe:`Lz0F
$f ~!N ]I@|:Qzyslave上:  BSD爱好者乐园kD5ny-`X6v)q(q$f
slave# ifconfig
LVE#ga6y&UC Xffxp0: flags=8843 metric 0 mtu 1500BSD爱好者乐园&d
Fg%e:vq
        options=8BSD爱好者乐园7n*v@AZ
        ether 00:07:e9:1b:4b:cdBSD爱好者乐园?$p ?^tE Mx
        inet 10.10.10.251 netmask 0xffffff00 broadcast 10.10.10.255BSD爱好者乐园7g-Z4[/H J)F nq
        media: Ethernet autoselect (100baseTX )
hAsH!K {        status: active
A1z w2V:A~ IOSdem0: flags=8943 metric 0 mtu 1500
8t/^EP`/H`$sN        options=9bBSD爱好者乐园{&k'y3@7?t7l ?
        ether 00:c0:9f:38:bd:afBSD爱好者乐园Oxq'Z,cP
        inet 192.168.1.53 netmask 0xffffff00 broadcast 192.168.1.255BSD爱好者乐园-H!_LFg]t r*J/@)[O
        media: Ethernet autoselect (100baseTX )BSD爱好者乐园1i*_7{3|x ^^(G6A|0g.@
        status: activeBSD爱好者乐园cC{/g/v5z*n
em1: flags=8943 metric 0 mtu 1500
1Lx cM9I
C6h,qt        options=9bBSD爱好者乐园 {;HB,qX
U{
        ether 00:c0:9f:38:bd:b0BSD爱好者乐园}Ss.vF)L;M
        inet 192.168.10.11 netmask 0xffffff00 broadcast 192.168.10.255
6O0G m6P%G%{ h-MZ        media: Ethernet autoselect (100baseTX )BSD爱好者乐园/v%?"W:Y[(?:ktT
        status: active
P(t z9T b*W-uuslo0: flags=8049 metric 0 mtu 16384BSD爱好者乐园 T.l2n{$BF X(_
        inet 127.0.0.1 netmask 0xff000000
E&p/tcqs2i2zZ ^pflog0: flags=141 metric 0 mtu 33204BSD爱好者乐园J"L YpTf.t:ro
pfsync0: flags=41 metric 0 mtu 1460
Q1g~D!E4N,DA3H        pfsync: syncdev: fxp0 syncpeer: 224.0.0.240 maxupd: 128BSD爱好者乐园6}r0f~&My(B-z
carp0: flags=49 metric 0 mtu 1500BSD爱好者乐园'A5_;^}:qYh6mT
        inet 192.168.1.51 netmask 0xffffff00BSD爱好者乐园K)g4[Wf7A
ix-t.G
        carp:BACKUPvhid 1 advbase 1advskew 20BSD爱好者乐园d}M-fF ]7K8?P
carp1: flags=49 metric 0 mtu 1500BSD爱好者乐园a&Py6g
V$V
        inet 192.168.10.100 netmask 0xffffff00
1[IeW&I        carp:BACKUPvhid 1 advbase 1advskew 20BSD爱好者乐园3`&b8o0Z1O7@AM@
BSD爱好者乐园Z?*]0n7A1?b3K
二. 服务器池的配置.
3~vKQ%S统一配置,把网关指向192.168.10.100就可以了.BSD爱好者乐园:y&GV&_Pq2E*TP1W
defaultrouter="192.168.10.100"
2EL}:{.F |%w U
Z+p4n I x-Vm+?/mN我在二台服务器上部署了相同的应用(lighttpd + php-fcgi), 配置我就不写了. 大家GG一下.BSD爱好者乐园N6HsU J
由于服务器不够,mysql我部署在其中的s2上.让两台服务器连到他上.BSD爱好者乐园PUYU \BX(EWD
BSD爱好者乐园-A1?Mi3sd$o Z
三 测试.BSD爱好者乐园Q,R#iy uh^P
A.静态页面:
^ z I L D~R d在s1和s2上分别与建立个index.htm页面.BSD爱好者乐园8n/~XU7Zh-g
s1上index.htm的内容是:
x+C7Y,k7?l:tW x"hi, this is No.1 server"BSD爱好者乐园$y#n1U!?wJ

Vk
e OJ9pBSD爱好者乐园's;AR `0_C
s2上index.htm的内容是:
DVp4V ]5Z5K"hi, this is No.2 server"BSD爱好者乐园'NV/Wuwx
BSD爱好者乐园0TL5r%C%MuluE
访问地址:  
"YZ#`.y.|返回的内容分别s1和s2的. 注意: 调度上做的状态保持的. 所以需要关闭浏览器再开才分别看得到不同的内容. 建议最好是几个人一起访问来返回不同的结果.
5zh R+{6vp-It
p1`Q"k4T3nsSB. 动态页面:
-I:t&t9M?6I
S7ot\^在s1和s2上简单部署了phpwind.  假如在./bbs下.
hb3UVlR-@6Ge Z访问地址:BSD爱好者乐园;EdCsk0R
返回的内容保持了一致性,BSD爱好者乐园F'STwTb%a
这里要提到的是的共享我没做, 现实的方法可以是通过nfs和iscsi. 暂时调不出iscsi的存储, 又不想用nfs所以就不作了. :)
^8NbD$L4T"mDRxN9[BSD爱好者乐园1Ea$A:R)L]-b5Z
BSD爱好者乐园:EW
j l D+_ mbg+^
master# pfctl -s state
E4th
TS6|Eall carp 192.168.1.52 -> 224.0.0.18       SINGLE:NO_TRAFFICBSD爱好者乐园1`:\0`~#Q6tX [ y
all pfsync 224.0.0.240 <- 10.10.10.251       NO_TRAFFIC:SINGLE
:bYq,CG+hn R N dball pfsync 10.10.10.250 -> 224.0.0.240       SINGLE:NO_TRAFFIC
Qq
O%@N4Z!_Eall carp 224.0.0.18 <- 192.168.10.10       NO_TRAFFIC:SINGLEBSD爱好者乐园 {2R|8jR@XR
P8z
all carp 224.0.0.18 <- 192.168.1.52       NO_TRAFFIC:SINGLEBSD爱好者乐园0?l Y%O.gn
all carp 192.168.10.10 -> 224.0.0.18       SINGLE:NO_TRAFFIC
h$}*AA-C?.[g.]_all tcp 192.168.10.21:65346 -> 192.168.1.52:61575 -> 130.104.5.67:25  FIN_WAIT_2:FIN_WAIT_2BSD爱好者乐园sWD2tF!U(|
all tcp 192.168.10.20:80 <- 192.168.1.51:80 <- 192.168.1.89:62414   TIME_WAIT:TIME_WAITBSD爱好者乐园&\ w7oW~ ~
all tcp 192.168.10.20:80 <- 192.168.1.51:80 <- 192.168.1.89:62416   FIN_WAIT_2:FIN_WAIT_2BSD爱好者乐园y,sx9Jh l
all tcp 192.168.10.20:80 <- 192.168.1.51:80 <- 192.168.1.89:62423   FIN_WAIT_2:FIN_WAIT_2
2S ](zR#Dj N_hWall tcp 192.168.10.20:80 <- 192.168.1.51:80 <- 192.168.1.89:62425   FIN_WAIT_2:ESTABLISHED
if^k
@ M1[ R2V H
9I5UkV}C. HA的测试BSD爱好者乐园wG oz!U)^$^V#r
拨掉master的网线, slave将自动接管. 状态保持.BSD爱好者乐园p%R
dq3i
ifconfig你可以看到BSD爱好者乐园}0x Rg,{
carp0: flags=49 metric 0 mtu 1500BSD爱好者乐园a$|^s]
fC-lV
        inet 192.168.1.51 netmask 0xffffff00
NFd5zx*} |"v@&CG        carp:MASTERvhid 1 advbase 1advskew 20
as;}B)U大家可以做些些测试了.
.KB8~Xj@ n4VGGBSD爱好者乐园8g)xe2F-SPjW
最后能有些:)
q5L$fQ,}U

:`
duy#mZ#F/?