aix平台 单机 10.2.0.4 突然实例宕机
alert.log 中记录
-
Mon Apr 12 13:25:58 2021
-
Thread 1 advanced to log sequence 140477 (LGWR switch)
-
Current log# 9 seq# 140477 mem# 0: /oradata/orcl/redo09.log
-
Mon Apr 12 13:26:38 2021
-
KCF: write/open error block=0x6706 online=1
-
file=38 /oradata/orcl/TSDAT02_08.dbf
-
error=27063 txt: 'IBM AIX RISC System/6000 Error: 16: Device busy
-
Additional information: -1
-
Additional information: 8192'
-
Automatic datafile offline due to write error on
-
file 38: /oradata/orcl/TSDAT02_08.dbf
-
Mon Apr 12 13:26:43 2021
-
Errors in file /home/oracle/admin/orcl/udump/orcl_ora_14155964.trc:
-
ORA-00376: file 38 cannot be read at this time
-
ORA-01110: data file 38: '/oradata/orcl/TSDAT02_08.dbf'
-
ORA-00376: file 38 cannot be read at this time
-
ORA-01110: data file 38: '/oradata/orcl/TSDAT02_08.dbf'
-
Mon Apr 12 13:26:49 2021
-
Errors in file /home/oracle/admin/orcl/bdump/orcl_pmon_34537506.trc:
-
ORA-00376: file 38 cannot be read at this time
-
ORA-01110: data file 38: '/oradata/orcl/TSDAT02_08.dbf'
-
Mon Apr 12 13:42:08 2021
-
Errors in file /home/oracle/admin/orcl/bdump/orcl_lgwr_53149820.trc:
-
ORA-00494: enqueue [CF] held for too long (more than 900 seconds) by 'inst 1, osid 36307002'
-
Mon Apr 12 13:42:09 2021
-
System State dumped to trace file /home/oracle/admin/orcl/bdump/orcl_lgwr_53149820.trc
-
Killing enqueue blocker (pid=36307002) on resource CF-00000000-00000000
-
by killing session 552.1
-
Mon Apr 12 13:47:11 2021
-
Errors in file /home/oracle/admin/orcl/bdump/orcl_lgwr_53149820.trc:
-
ORA-00494: enqueue [CF] held for too long (more than 900 seconds) by 'inst 1, osid 36307002'
-
Mon Apr 12 13:47:12 2021
-
System State dumped to trace file /home/oracle/admin/orcl/bdump/orcl_lgwr_53149820.trc
-
Killing enqueue blocker (pid=36307002) on resource CF-00000000-00000000
-
by terminating the process
-
LGWR: terminating instance due to error 2103
-
Instance terminated by LGWR, pid = 53149820
看到读写38号文件时 提示error=27063 txt: 'IBM AIX RISC System/6000 Error: 16: Device busy
查看os日志
-
[ oracle ]host1:/home/oracle/admin/orcl/bdump:> errpt|head
-
IDENTIFIER TIMESTAMP T C RESOURCE_NAME DESCRIPTION
-
DCB47997 0412142821 T H hdisk112 DISK OPERATION ERROR
-
DCB47997 0412142821 T H hdisk119 DISK OPERATION ERROR
-
B6267342 0412142821 P H hdisk119 DISK OPERATION ERROR
-
B6267342 0412142821 P H hdisk140 DISK OPERATION ERROR
-
B6267342 0412142821 P H hdisk119 DISK OPERATION ERROR
-
B6267342 0412142821 P H hdisk112 DISK OPERATION ERROR
-
DCB47997 0412142721 T H hdisk112 DISK OPERATION ERROR
-
DCB47997 0412142721 T H hdisk112 DISK OPERATION ERROR
-
B6267342 0412142721 P H hdisk119 DISK OPERATION ERROR
[ oracle ]host1:/home/oracle/admin/orcl/bdump:> errpt -aj DCB47997|more
---------------------------------------------------------------------------
LABEL: SC_DISK_ERR4
IDENTIFIER: DCB47997
Date/Time: Mon Apr 12 14:28:34 GMT+08:00 2021
Sequence Number: 847721
Machine Id: 00F710984C00
Node Id: host1
Class: H
Type: TEMP
WPAR: Global
Resource Name: hdisk112
Resource Class: disk
Resource Type: CLAR_FC_VRAID
Location: U5802.001.9K8N405-P1-C2-T2-W500601690960398D-L3000000000000
VPD:
Manufacturer................DGC
Machine Type and Model......VRAID
ROS Level and ID............0533
Serial Number...............CETV2173400018
Subsystem Vendor/Device ID..VNX5600
Device Specific.(PQ)........00
Device Specific.(VS)........255F081CECCL
Device Specific.(UI)........600601600A804500571D3D8381C2EA11
FRU Label...................0025
Device Specific.(Z0)........10
Device Specific.(Z1)........10
Description
DISK OPERATION ERROR
Probable Causes
MEDIA
DASD DEVICE
User Causes
MEDIA DEFECTIVE
Recommended Actions
FOR REMOVABLE MEDIA, CHANGE MEDIA AND RETRY
PERFORM PROBLEM DETERMINATION PROCEDURES
Failure Causes
MEDIA
DISK DRIVE
Recommended Actions
FOR REMOVABLE MEDIA, CHANGE MEDIA AND RETRY
PERFORM PROBLEM DETERMINATION PROCEDURES
Detail Data
PATH ID
0
SENSE DATA
0A00 2800 2117 0920 0004 0004 0000 0000 0000 0000 0000 0000 0118 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 034A 045C 0007 8000 0000 0000 0000 0000 0000 0000 0000 0083 0000
0000 003D 001D
---------------------------------------------------------------------------
据说是磁盘坏道或者链路闪断导致。
先处理故障，直接启动实例
startup 耗时20分钟终于起来了
实例启动后，38号文件一直报错
-
Mon Apr 12 14:22:33 2021
-
Errors in file /home/oracle/admin/orcl/bdump/orcl_smon_34537570.trc:
-
ORA-00376: file 38 cannot be read at this time
-
ORA-01110: data file 38: '/oradata/orcl/TSDAT02_08.dbf'
-
ORACLE Instance orcl (pid = 9) - Error 376 encountered while recovering transaction (30, 43) on object 1192947.
-
Mon Apr 12 14:22:34 2021
-
Errors in file /home/oracle/admin/orcl/bdump/orcl_smon_34537570.trc:
-
ORA-00376: file 38 cannot be read at this time
-
ORA-01110: data file 38: '/oradata/orcl/TSDAT02_08.dbf'
-
ORACLE Instance orcl (pid = 9) - Error 376 encountered while recovering transaction (30, 43) on object 1192947.
-
Mon Apr 12 14:22:35 2021
-
Errors in file /home/oracle/admin/orcl/bdump/orcl_smon_34537570.trc:
-
ORA-00376: file 38 cannot be read at this time
-
ORA-01110: data file 38: '/oradata/orcl/TSDAT02_08.dbf'
检查38号数据文件
select file#,status,bytes/1024/1024,name from v$datafile;
-
...
-
37 ONLINE 30720 /oradata/orcl/TSDAT03_12.dbf
-
38 RECOVER 30720 /oradata/orcl/TSDAT02_08.dbf
-
39 ONLINE 30720/oradata/orcl/TSDAT03_13.dbf
-
...
归档模式，问题发现的比较及时，赶紧修复
recover datafile 38;
alter database datafile 38 online;
很快顺利修复，又躲过一劫。
参考：
HOW TO RECOVER OFFLINE DROPPED DATAFILE IN ARCHIVELOG MODE (Doc ID 286355.1)
阅读(1342) | 评论(0) | 转发(0) |
