以下是一次V880 硬盘故障(SVM)更换过程的log,纪念一下:
iostat -En
c0t6d0 Soft Errors: 1 Hard Errors: 0 Transport Errors: 0
Vendor: TOSHIBA Product: DVD-ROM SD-M1711 Revision: 1005 Serial No:
Size: 0.00GB <0 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 1 Predictive Failure Analysis: 0
c1t4d0 Soft Errors: 0 Hard Errors: 291 Transport Errors: 0 // 坏盘
Vendor: SEAGATE Product: ST373307FSUN72G Revision: 0307 Serial No: B
Size: 0.00GB <0 bytes>
Media Error: 0 Device Not Ready: 291 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
c1t5d0 Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: SEAGATE Product: ST373307FSUN72G Revision: 0307 Serial No: 0434B90MGW
Size: 73.40GB <73400057856 bytes>
# prtdiag -v
System Configuration: Sun Microsystems sun4u Sun Fire 880
System clock frequency: 150 MHz
Memory size: 8192 Megabytes
========================= CPUs ===============================================
Run E$ CPU CPU
Brd CPU MHz MB Impl. Mask
--- ----- ---- ---- ------- ----
A 0 1200 8.0 US-III+ 11.1
B 1 1200 8.0 US-III+ 11.1
A 2 1200 8.0 US-III+ 11.1
B 3 1200 8.0 US-III+ 11.1
========================= Memory Configuration ===============================
Logical Logical Logical
MC Bank Bank Bank DIMM Interleave Interleaved
Brd ID num size Status Size Factor with
---- --- ---- ------ ----------- ------ ---------- -----------
A 0 0 512MB no_status 256MB 8-way 0
A 0 1 512MB no_status 256MB 8-way 0
A 0 2 512MB no_status 256MB 8-way 0
A 0 3 512MB no_status 256MB 8-way 0
B 1 0 512MB no_status 256MB 8-way 1
B 1 1 512MB no_status 256MB 8-way 1
B 1 2 512MB no_status 256MB 8-way 1
B 1 3 512MB no_status 256MB 8-way 1
A 2 0 512MB no_status 256MB 8-way 0
A 2 1 512MB no_status 256MB 8-way 0
A 2 2 512MB no_status 256MB 8-way 0
A 2 3 512MB no_status 256MB 8-way 0
B 3 0 512MB no_status 256MB 8-way 1
B 3 1 512MB no_status 256MB 8-way 1
B 3 2 512MB no_status 256MB 8-way 1
B 3 3 512MB no_status 256MB 8-way 1
========================= IO Cards =========================
Bus Max
IO Port Bus Freq Bus Dev,
Brd Type ID Side Slot MHz Freq Func State Name Model
---- ---- ---- ---- ---- ---- ---- ---- ----- -------------------------------- ----------------------
I/O PCI 8 B 3 33 33 2,0 ok fibre-channel-pci10df,f900.10df.+
I/O PCI 8 B 2 33 33 3,0 ok fibre-channel-pci10df,f900.10df.+
I/O PCI 8 B 0 33 33 5,0 ok pci-pci8086,b154.0/network (netw+ PCI-BRIDGE
I/O PCI 8 B 0 33 33 0,0 ok network-pci108e,abba.20 SUNW,pci-ce/pci-bridge
I/O PCI 9 B 6 33 33 2,0 ok scsi-pci1000,f.1000.1000.14/disk+
I/O PCI 9 B 6 33 33 2,1 ok scsi-pci1000,f.1000.1000.14/disk+
I/O PCI 9 B 5 33 33 3,0 ok pci-pci8086,b154.0/network (netw+ PCI-BRIDGE
I/O PCI 9 B 5 33 33 0,0 ok network-pci108e,abba.20 SUNW,pci-ce/pci-bridge
I/O PCI 9 B 4 33 33 4,0 ok scsi-pci1000,f.1000.1000.14/disk+
I/O PCI 9 B 4 33 33 4,1 ok scsi-pci1000,f.1000.1000.14/disk+
I/O PCI 9 A 8 66 66 1,0 ok fibre-channel-pci10df,f900.10df.+
I/O PCI 9 A 7 66 66 2,0 ok fibre-channel-pci10df,f900.10df.+
No failures found in System
===========================
========================= Environmental Status =========================
System Temperatures (Celsius):
-------------------------------
Device Temperature Status
---------------------------------------
CPU0 59 OK
CPU1 58 OK
CPU2 57 OK
CPU3 60 OK
MB 27 OK
IOB 22 OK
DBP0 22 OK
=================================
Front Status Panel:
-------------------
Keyswitch position: NORMAL
System LED Status:
GEN FAULT REMOVE
[OFF] [OFF]
DISK FAULT POWER FAULT
[OFF] [OFF]
LEFT THERMAL FAULT RIGHT THERMAL FAULT
[OFF] [OFF]
LEFT DOOR RIGHT DOOR
[OFF] [OFF]
=================================
Disk Status:
Presence Fault LED Remove LED
DISK 0: [PRESENT] [OFF] [OFF]
DISK 1: [PRESENT] [OFF] [OFF]
DISK 2: [PRESENT] [OFF] [OFF]
DISK 3: [PRESENT] [OFF] [OFF]
DISK 4: [PRESENT] [OFF] [OFF]
DISK 5: [PRESENT] [OFF] [OFF]
DISK 6: [ EMPTY]
DISK 7: [ EMPTY]
DISK 8: [ EMPTY]
DISK 9: [ EMPTY]
DISK 10: [ EMPTY]
DISK 11: [ EMPTY]
=================================
Fan Bank :
----------
Bank Speed Status Fan State
( RPMS )
---- -------- --------- ---------
CPU0_PRIM_FAN 2054 [ENABLED] OK
CPU1_PRIM_FAN 2272 [ENABLED] OK
CPU0_SEC_FAN 0 [DISABLED] OK
CPU1_SEC_FAN 0 [DISABLED] OK
IO0_PRIM_FAN 2970 [ENABLED] OK
IO1_PRIM_FAN 2857 [ENABLED] OK
IO0_SEC_FAN 0 [DISABLED] OK
IO1_SEC_FAN 0 [DISABLED] OK
IO_BRIDGE_PRIM_FAN 3370 [ENABLED] OK
IO_BRIDGE_SEC_FAN 0 [DISABLED] OK
=================================
Power Supplies:
---------------
Supply Status Fan Fail Temp Fail CS Fail 3.3V 5V 12V 48V
------ ------------ -------- --------- ------- ---- -- --- ---
PS0 GOOD 8 4 2 3
PS1 GOOD 8 4 2 3
PS2 GOOD 8 5 2 3
========================= HW Revisions =======================================
System PROM revisions:
----------------------
OBP 4.15.1 2004/06/02 16:06
IO ASIC revisions:
------------------
Port
Brd Model ID Status Version
---- --------------- ---- ------ -------
IB-1 unknown 8 ok 7
IB-1 unknown 9 ok 7
# luxadm probe
Found Enclosure:
SUNWGS INT FCBPL Name:FCloop Node WWN:50800200001f0f88 Logical Path:/dev/es/ses0
#
# luxadm display FCloop
SUNWGS INT FCBPL
DISK STATUS
SLOT DISKS (Node WWN)
0 On (O.K.) 2000000c506746a6
1 On (O.K.) 2000000c50674637
2 On (O.K.) 2000000c50674545
3 On (O.K.) 2000000c5067490f
4 On (SCSI Error) 2000000c50674877 // 坏盘定位
5 On (O.K.) 2000000c506745ce
6 On (Login failed)
7 On (Login failed)
8 On (Login failed)
9 On (Login failed)
10 On (Login failed)
11 On (Login failed)
SUBSYSTEM STATUS
FW Revision:922A Box ID:0
Node WWN:50800200001f0f88 Enclosure Name:FCloop
SSC100's - 0=Base Bkpln, 1=Base LoopB, 2=Exp Bkpln, 3=Exp LoopB
SSC100 #0: O.K.(922A/ 8D3C)
SSC100 #1: O.K.(922A/ 8D3C)
SSC100 #2: Not Installed
SSC100 #3: Not Installed
Temperature Sensors - 0 Base, 1 Expansion
0:22篊
1Not Installed
Backplanes - A=Base, B=Expansion
A: O.K.
B: Not Installed
Default Language is USA English, ASCII
#
#
# luxadm remove_device -F FCloop ,s4
WARNING!!! Please ensure that no filesystems are mounted on these device(s).
All data on these devices should have been backed up.
The list of devices which will be removed is:
1: Box Name: "FCloop" slot 4
Node WWN: 2000000c50674877
Device Type:Disk device
Device Paths:
/dev/rdsk/c1t4d0s2
Please verify the above list of devices and
then enter 'c' or to Continue or 'q' to Quit. [Default: c]: c
stopping: Drive in "FCloop" slot 4....I/O error - FCloop,s4.
# # c: not found
# format
Searching for disks...done
AVAILABLE DISK SELECTIONS:
0. c1t0d0
1. c1t1d0
2. c1t2d0
3. c1t3d0
4. c1t4d0 usr
5. c1t5d0
Specify disk (enter its number): ^C^D
#
#
# luxadm display FCloop
SUNWGS INT FCBPL
DISK STATUS
SLOT DISKS (Node WWN)
0 On (O.K.) 2000000c506746a6
1 On (O.K.) 2000000c50674637
2 On (O.K.) 2000000c50674545
3 On (O.K.) 2000000c5067490f
4 On (O.K.) 20000004cf179daf //硬盘已更换
5 On (O.K.) 2000000c506745ce
6 On (Login failed)
7 On (Login failed)
8 On (Login failed)
9 On (Login failed)
10 On (Login failed)
11 On (Login failed)
SUBSYSTEM STATUS
FW Revision:922A Box ID:0
Node WWN:50800200001f0f88 Enclosure Name:FCloop
SSC100's - 0=Base Bkpln, 1=Base LoopB, 2=Exp Bkpln, 3=Exp LoopB
SSC100 #0: O.K.(922A/ 8D3C)
SSC100 #1: O.K.(922A/ 8D3C)
SSC100 #2: Not Installed
SSC100 #3: Not Installed
Temperature Sensors - 0 Base, 1 Expansion
0:22篊
1Not Installed
Backplanes - A=Base, B=Expansion
A: O.K.
B: Not Installed
Default Language is USA English, ASCII
# metastat -p
d90 -m d91 d92 1
d91 1 1 c1t2d0s0
d92 1 1 c1t5d0s0
d80 -m d81 d82 1
d81 1 1 c1t1d0s0
d82 1 1 c1t4d0s0
d50 -m d51 d52 1
d51 1 1 c1t0d0s5
d52 1 1 c1t3d0s5
d40 -m d41 d42 1
d41 1 1 c1t0d0s4
d42 1 1 c1t3d0s4
d20 -m d21 d22 1
d21 1 1 c1t0d0s1
d22 1 1 c1t3d0s1
d10 -m d11 d12 1
d11 1 1 c1t0d0s0
d12 1 1 c1t3d0s0
# prtvoc /dev/rdsk/c1t1d0s2 | fmthared -s - /dev/rdsk/c1t4d0s2
fmthard: New volume table of contents now in place.
# metareplace -e d80 c1t4d0s0 // 更换镜像盘
d80: device c1t4d0s0 is replaced with c1t4d0s0
# metastat |grep %
Resync in progress: 1 % done
# metastat |grep %
Resync in progress: 1 % done
# metastat d80
d80: Mirror
Submirror 0: d81
State: Okay
Submirror 1: d82
State: Resyncing
Resync in progress: 4 % done
Pass: 1
Read option: roundrobin (default)
Write option: parallel (default)
Size: 143278080 blocks (68 GB)
d81: Submirror of d80
State: Okay
Size: 143278080 blocks (68 GB)
Stripe 0:
Device Start Block Dbase State Reloc Hot Spare
c1t1d0s0 0 No Okay Yes
d82: Submirror of d80
State: Unavailable //换盘后,状态不对
Size: 143278080 blocks (68 GB)
Stripe 0:
Device Start Block Dbase State Reloc Hot Spare
c1t4d0s0 0 No - Yes
Device Relocation Information:
Device Reloc Device ID
c1t1d0 Yes id1,ssd@w2000000c50674637
c1t4d0 Yes id1,ssd@w20000004cf179daf
# metastat -i //刷新状态
# metastat d80
d80: Mirror
Submirror 0: d81
State: Okay
Submirror 1: d82
State: Resyncing
Resync in progress: 8 % done
Pass: 1
Read option: roundrobin (default)
Write option: parallel (default)
Size: 143278080 blocks (68 GB)
d81: Submirror of d80
State: Okay
Size: 143278080 blocks (68 GB)
Stripe 0:
Device Start Block Dbase State Reloc Hot Spare
c1t1d0s0 0 No Okay Yes
d82: Submirror of d80
State: Resyncing //状态正常
Size: 143278080 blocks (68 GB)
Stripe 0:
Device Start Block Dbase State Reloc Hot Spare
c1t4d0s0 0 No Resyncing Yes
Device Relocation Information:
Device Reloc Device ID
c1t1d0 Yes id1,ssd@w2000000c50674637
c1t4d0 Yes id1,ssd@w20000004cf179daf
# metastat |grep %
Resync in progress: 10 % done
# exit