alert log
==================================
Wed Nov 19 08:36:00 2008
ksvcreate: Process(q000) creation failed
Wed Nov 19 08:37:03 2008
PMON failed to acquire latch, see PMON dump
Wed Nov 19 08:38:06 2008
PMON failed to acquire latch, see PMON dump
Wed Nov 19 08:39:07 2008
System State dumped to trace file
Wed Nov 19 08:39:08 2008
Errors in file d:\oracle\product\10.2.0\admin\fspdc\udump\fspdc_ora_2088.trc:
ORA-07445: exception encountered: core dump [ACCESS_VIOLATION] [kgxAolDump+208] [PC:0x50C15B6] [ADDR:0x88]
[UNABLE_TO_READ] []
Wed Nov 19 08:39:08 2008
PMON failed to acquire latch, see PMON dump
Wed Nov 19 08:39:24 2008
Incremental checkpoint up to RBA [0xcc51.1c995.0], current log tail at RBA [0xcc51.1c9f7.0]
Wed Nov 19 08:40:11 2008
PMON failed to acquire latch, see PMON dump
Wed Nov 19 08:41:13 2008
PMON failed to acquire latch, see PMON dump
Wed Nov 19 08:43:19 2008
PMON failed to acquire latch, see PMON dump
Wed Nov 19 08:44:21 2008
PMON failed to acquire latch, see PMON dump
Wed Nov 19 08:45:24 2008
PMON failed to acquire latch, see PMON dump
Wed Nov 19 08:46:03 2008
Errors in file d:\oracle\product\10.2.0\admin\fspdc\bdump\fspdc_pmon_4116.trc:
ORA-00601: cleanup lock conflict
Wed Nov 19 08:46:03 2008
Errors in file d:\oracle\product\10.2.0\admin\fspdc\bdump\fspdc_pmon_4116.trc:
ORA-00600: internal error code, arguments: [ksudlp1], [0x1BA2B39F0], [0x125F0B0E8], [], [], [], [], []
Wed Nov 19 08:46:17 2008
Errors in file d:\oracle\product\10.2.0\admin\fspdc\bdump\fspdc_pmon_4116.trc:
ORA-00600: internal error code, arguments: [ksudlp1], [0x1BA2B39F0], [0x125F0B0E8], [], [], [], [], []
Wed Nov 19 08:46:17 2008
PMON: terminating instance due to error 472
Wed Nov 19 08:46:17 2008
Errors in file d:\oracle\product\10.2.0\admin\fspdc\bdump\fspdc_dbw0_4100.trc:
ORA-00472: PMON process terminated with error
Wed Nov 19 08:46:18 2008
Errors in file d:\oracle\product\10.2.0\admin\fspdc\bdump\fspdc_lgwr_4548.trc:
ORA-00472: PMON process terminated with error
Wed Nov 19 08:46:18 2008
Errors in file d:\oracle\product\10.2.0\admin\fspdc\bdump\fspdc_ckpt_4552.trc:
ORA-00472: PMON process terminated with error
==================================
==================================
Windows thread id: 4116, image: ORACLE.EXE (PMON)
*** 2008-11-19 08:37:03.022
*** SERVICE NAME:(SYS$BACKGROUND) 2008-11-19 08:37:03.006
*** SESSION ID:(3305.1) 2008-11-19 08:37:03.006
PMON unable to acquire latch 48478c0 parameter table allocation management level=6
Location from where latch is held: ksp_param_table_free:
Context saved from call: 0
state=busy, wlstate=free
gotten 443273 times wait, failed first 108 sleeps 12
gotten 0 times nowait, failed: 0
possible holder pid = 1599 ospid=13728
----------------------------------------
SO: 00000001BA27C0E8, type: 2, owner: 0000000000000000, flag: INIT/-/-/0x00
(process) Oracle pid=1599, calls cur/top: 00000000F8BC77E8/00000000F8BC77E8, flag: (0) -
int error: 0, call error: 0, sess error: 0, txn error 0
(post info) last post received: 115 0 4
last post received-location: kslpsr
last process to post me: ba0e6488 1 6
last post sent: 0 0 24
last post sent-location: ksasnd
last process posted by me: ba0e6488 1 6
(latch info) wait_event=0 bits=40
Location from where call was made: kghfre: Chunk Header
Context saved from call: 4200013472
waiting for a6e41c0 Child shared pool level=7 child#=2
Location from where latch is held: kgh: quiesce extents:
Context saved from call: 0
state=busy, wlstate=free
waiters [orapid (seconds since: put on list, posted, alive check)]:
345 (213, 1227055020, 213)
1600 (213, 1227055020, 213)
1134 (213, 1227055020, 213)
170 (213, 1227055020, 213)
1471 (213, 1227055020, 213)
992 (213, 1227055020, 213)
1004 (213, 1227055020, 213)
1384 (213, 1227055020, 213)
479 (213, 1227055020, 213)
1184 (213, 1227055020, 213)
482 (213, 1227055020, 213)
1551 (213, 1227055020, 213)
52 (213, 1227055020, 213)
.
.
.
2121 (3, 1227055020, 3)
2122 (0, 1227055020, 0)
2124 (0, 1227055020, 0)
2125 (0, 1227055020, 0)
2127 (0, 1227055020, 0)
waiter count=214
gotten 4250018325 times wait, failed first 87272528 sleeps 9653822
gotten 0 times nowait, failed: 0
possible holder pid = 5 ospid=3632
on wait list for a6e41c0
holding (efd=2) 48478c0 parameter table allocation management level=6
Location from where latch is held: ksp_param_table_free:
Context saved from call: 0
state=busy, wlstate=free
Process Group: DEFAULT, pseudo proc: 00000001BA3E7440
O/S info: user: SYSTEM, term: DN2PROD201, ospid: 13728
OSD pid info: Windows thread id: 13728, image: ORACLE.EXE (J000)
Short stack dump:
=========================================
========================================
==============
HANG ANALYSIS:
==============
Found 159 objects waiting for
<0/3302/1/0xba0e5c60/3632/No Wait>
Found 78 objects waiting for
<0/2177/32351/0xb9151638/6672/latch: shared pool>
Found 111 objects waiting for
<0/2581/16069/0xba25b6e8/13772/latch: shared pool>
Found 103 objects waiting for
<0/1762/6126/0xb9204510/10592/latch: shared pool>
Found 117 objects waiting for
<0/2311/34085/0xba15e968/11952/latch: shared pool>
Found 143 objects waiting for
<0/1679/32270/0xb91d6700/6048/latch: shared pool>
Found 16 objects waiting for
<0/2193/49433/0xb92374b0/11812/latch: shared pool>
Found 16 objects waiting for
<0/1334/10775/0xba237bf8/776/latch: shared pool>
Open chains found:
Chain 1 : :
<0/3302/1/0xba0e5c60/3632/No Wait>
-- <0/1098/3/0xba313750/12128/latch: shared pool>
Other chains found:
Chain 2 : :
<0/1102/8577/0xb91e4b60/6320/latch: library cache>
Chain 3 : :
<0/1104/4844/0xba13ae78/13992/latch: library cache>
Chain 4 : :
<0/1111/49177/0xb9102e28/3872/latch: library cache>
Chain 5 : :
<0/1119/22790/0xb91431d8/7972/latch: library cache>
Chain 6 : :
<0/1134/30017/0xb91c3110/7044/latch: library cache>
Chain 7 : :
========================================
i) Atleast 2 systemstate dumps
sqlplus -prelim / as sysdba
oradebug setmypid
oradebug unlimit;
oradebug dump systemstate 266
oradebug dump systemstate 266
ii) Atleast 2 hanganalyze dumps:
sqlplus -prelim / as sysdba
oradebug setmypid
oradebug unlimit;
oradebug dump hanganalyze 3
oradebug dump hanganalyze 3
$ sqlplus
connect / as sysdba
oradebug setospid
oradebug unlimit
oradebug dump errorstack 3+J000 process inturn was waiting to acquire shared pool latch which was held by process with ospid=3632
Hangnalyze shows.. heavy waits for "larch : Library Cache" and "latch : shared pool". Almost all the processes
were directly or indirectly waiting for ospid=3632.
+The main culprit was process ospid=3632 which was currently NOT WAITING
+Database was hung as all the session including PMON was blocked by process with ospid=3632
select * from WRH$_ACTIVE_SESSION_HISTORY where session_sid = 3302 and snap_id > 3800 and snap_id < 3884;
SNAP_ID SAMPLE_ID SAMPLE_TIME SESSION_ID EVENT_ID WAIT_TIME TIME_WAITED PROGRAM
---------- ---------- ----------------------------- ---------- --------- --------- --------
--- ------------------
3879 6962250 19-NOV-08 08.20.57.274 AM 3302 696347763 432 0 ORACLE.EXE (MMAN)
3880 6963310 19-NOV-08 08.59.56.835 AM 3302 620694733 0 1146 ORACLE
$ sqlplus
connect / as sysdba
oradebug setospid
oradebug unlimit
oradebug dump errorstack 3
Database crashed due to bug 5476873
Process Explorer download
