处理Zenoss HA DRBD脑裂

今天做HA切换测试时，发现Zenoss数据只记录到很早以前的情况。判断是Drbd脑裂。主上显示：

# service drbd status
drbd driver loaded OK; device status:
version: 8.2.6 (api:88/proto:86-88)
GIT-hash: 3e69822d3bb4920a8c1bfdf7d647169eba7d2eb4 build by buildsvn@c5-x8664-build, 2008-10-03 11:30:17
m:res      cs          st               ds                 p  mounted           fstype
1:mysql    StandAlone  Primary/Unknown  UpToDate/DUnknown  -  /var/lib/mysql    ext3
2:zenhome  StandAlone  Primary/Unknown  UpToDate/DUnknown  -  /opt/zenoss       ext3
3:zenperf  StandAlone  Primary/Unknown  UpToDate/DUnknown  -  /opt/zenoss/perf  ext3

从上显示：

# service drbd status
drbd driver loaded OK; device status:
version: 8.2.6 (api:88/proto:86-88)
GIT-hash: 3e69822d3bb4920a8c1bfdf7d647169eba7d2eb4 build by buildsvn@c5-x8664-build, 2008-10-03 11:30:17
m:res      cs          st                 ds                 p  mounted  fstype
1:mysql    StandAlone  Secondary/Unknown  UpToDate/DUnknown  -
2:zenhome  StandAlone  Secondary/Unknown  UpToDate/DUnknown  -
3:zenperf  StandAlone  Secondary/Unknown  UpToDate/DUnknown  -

对于在从上入手，修复脑裂

# drbdadm disconnect mysql
# drbdadm disconnect zenhome
# drbdadm disconnect zenperf
# drbdadm secondary mysql
# drbdadm secondary zenhome
# drbdadm secondary zenperf
# drbdadm -- --discard-my-data connect mysql
# drbdadm -- --discard-my-data connect zenhome
# drbdadm -- --discard-my-data connect zenperf

查看从的Drbd状态

more /proc/drbd
version: 8.2.6 (api:88/proto:86-88)
GIT-hash: 3e69822d3bb4920a8c1bfdf7d647169eba7d2eb4 build by buildsvn@c5-x8664-bu
ild, 2008-10-03 11:30:17

1: cs:WFConnection st:Secondary/Unknown ds:UpToDate/DUnknown C r---
ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 oos:7584
2: cs:WFConnection st:Secondary/Unknown ds:UpToDate/DUnknown C r---
ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 oos:47888
3: cs:WFConnection st:Secondary/Unknown ds:UpToDate/DUnknown C r---
ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 oos:1936

显示WFConnection状态。看来，还得重新在主上进行连接。

# drbdadm connect mysql
# drbdadm connect zenhome
# drbdadm connect zenperf

再次查看主从状态，DRBD脑裂恢复。

more /proc/drbd
version: 8.2.6 (api:88/proto:86-88)
GIT-hash: 3e69822d3bb4920a8c1bfdf7d647169eba7d2eb4 build by buildsvn@c5-x8664-bu
ild, 2008-10-03 11:30:17

1: cs:SyncSource st:Primary/Secondary ds:UpToDate/Inconsistent C r---
ns:89456 nr:0 dw:45201920 dr:162898 al:97 bm:98 lo:1 pe:9 ua:253 ap:1 oos:16
6712
[======>.............] sync'ed: 36.6% (166712/255948)K
finish: 0:00:36 speed: 4,464 (6,864) K/sec
2: cs:SyncSource st:Primary/Secondary ds:UpToDate/Inconsistent C r---
ns:47360 nr:0 dw:4351072 dr:532070 al:229 bm:394 lo:1 pe:9 ua:1325 ap:2 oos:
339072
[=>..................] sync'ed: 13.7% (339072/386240)K
finish: 0:01:04 speed: 5,240 (5,240) K/sec
3: cs:SyncSource st:Primary/Secondary ds:UpToDate/Inconsistent C r---
ns:15372 nr:0 dw:8790796 dr:41474 al:66 bm:89 lo:2 pe:6 ua:344 ap:1 oos:1214
48
[=>..................] sync'ed: 14.8% (121448/136668)K
finish: 0:00:31 speed: 3,804 (3,804) K/sec

# more /proc/drbd
version: 8.2.6 (api:88/proto:86-88)
GIT-hash: 3e69822d3bb4920a8c1bfdf7d647169eba7d2eb4 build by buildsvn@c5-x8664-build, 2008-10-03 11:30:17

1: cs:SyncTarget st:Secondary/Primary ds:Inconsistent/UpToDate C r---
ns:0 nr:192920 dw:192920 dr:0 al:0 bm:14 lo:1 pe:1980 ua:0 ap:0 oos:63216
[==============>.....] sync'ed: 76.2% (63216/255948)K
finish: 0:00:26 speed: 2,368 (4,816) K/sec
2: cs:SyncTarget st:Secondary/Primary ds:Inconsistent/UpToDate C r---
ns:0 nr:150944 dw:150944 dr:0 al:0 bm:18 lo:2 pe:7320 ua:0 ap:0 oos:235328
[=======>............] sync'ed: 40.0% (235328/386240)K
finish: 0:00:53 speed: 4,332 (4,192) K/sec
3: cs:SyncTarget st:Secondary/Primary ds:Inconsistent/UpToDate C r---
ns:0 nr:119000 dw:119000 dr:0 al:0 bm:7 lo:1 pe:688 ua:0 ap:0 oos:17684
[================>...] sync'ed: 88.3% (17684/136668)K
finish: 0:00:03 speed: 4,600 (3,836) K/sec

看来，以后需要多注意DRBD状态，否则，Zenoss很有可能会切换到其它以住时期的运行状态。