ORA-07445: exception encountered: core dump [kslgetl()+40] [SIGSEGV] ...

ORA-07445: exception encountered: core dump [kslgetl()+40] [SIGSEGV] [ADDR:0x120000001D] [PC:0x74915B4] [Address not mapped to object] []

2012-12-04下午开始，邮箱受到大量alert日志错误信息。原以为不会影响数据库的使用，没想到会造成所有job停止运行。
所以一定对这个错误有足够的重视，出现问题后及时重启是解决问题的暂时有效方法，最好打补丁彻底解决问题！

alert.log
----------------------------------------
Wed Dec 05 10:49:22 2012
Sweep Incident[44252]: completed
Wed Dec 05 10:54:16 2012
Restarting dead background process CJQ0
Wed Dec 05 10:54:16 2012
CJQ0 started with pid=47, OS id=27318
Exception [type: SIGSEGV, Address not mapped to object] [ADDR:0x120000001D] [PC:0x74915B4, kslgetl()+40]
Errors in file /var/u01/app/oracle/diag/rdbms/xxx/yyy/trace/yyy_cjq0_27318.trc  (incident=44053):
ORA-07445: exception encountered: core dump [kslgetl()+40] [SIGSEGV] [ADDR:0x120000001D] [PC:0x74915B4] [Address not mapped to object] []
Wed Dec 05 10:54:24 2012
Sweep Incident[44053]: completed
Wed Dec 05 10:56:57 2012
Thread 1 advanced to log sequence 66826
Current log# 5 seq# 66826 mem# 0: /var/u01/app/oracle/oradata/xxx/redo05.log
Current log# 5 seq# 66826 mem# 1: /ext/oracle_backup/ora_redo_control_bk/redo05_b.log
Wed Dec 05 10:59:17 2012
Restarting dead background process CJQ0
Wed Dec 05 10:59:17 2012
CJQ0 started with pid=51, OS id=30478
Exception [type: SIGSEGV, Address not mapped to object] [ADDR:0x120000001D] [PC:0x74915B4, kslgetl()+40]
Errors in file /var/u01/app/oracle/diag/rdbms/xxx/yyy/trace/yyy_cjq0_30478.trc  (incident=41243):
ORA-07445: exception encountered: core dump [kslgetl()+40] [SIGSEGV] [ADDR:0x120000001D] [PC:0x74915B4] [Address not mapped to object] []
------------------------------------

trace文件
----------------------------------
Trace file /var/u01/app/oracle/diag/rdbms/xxx/yyy/trace/yyy_cjq0_30478.trc
Oracle Database 11g Enterprise Edition Release 11.1.0.6.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
ORACLE_HOME = /var/u01/app/oracle/product/11.1.0/db_1
System name:    Linux
Node name:      xxx
Release:        2.6.18-238.19.1.el5xen
Version:        #1 SMP Fri Jul 15 08:16:59 EDT 2011
Machine:        x86_64
Instance name: yyy
Redo thread mounted by this instance: 1
Oracle process number: 51
Unix process pid: 30478, image: oracle@xxx (CJQ0)


*** 2012-12-05 10:59:20.348
*** SESSION ID:(307.7515) 2012-12-05 10:59:20.348
*** CLIENT ID:() 2012-12-05 10:59:20.348
*** SERVICE NAME:(SYS$BACKGROUND) 2012-12-05 10:59:20.348
*** MODULE NAME:() 2012-12-05 10:59:20.348
*** ACTION NAME:() 2012-12-05 10:59:20.348

Exception [type: SIGSEGV, Address not mapped to object] [ADDR:0x120000001D] [PC:0x74915B4, Exception [type: SIGSEGV, Address not mapped to object] [ADDR:0x120000001D] [PC:0x74915B4, kslgetl()+40]

*** 2012-12-05 10:59:20.592
DDE: Problem Key 'ORA 7445 [kslgetl()+40]' was flood controlled (0x6) (incident: 41243)
ORA-07445: exception encountered: core dump [kslgetl()+40] [SIGSEGV] [ADDR:0x120000001D] [PC:0x74915B4] [Address not mapped to object] []
s***hd: crashing the process...
Background_Core_Dump = PARTIAL
-------------------------------

通过查看MOS，找到原因：
oracle触发了一个bug，oracle job进程（CJQ）中发生内存坏块，导致内部错误。
Details: This is a memory corruption that occurs in the CJQ process leading to a dump or internal error.

解决办法：
打补丁Patch:6890831;也可以重启数据库暂时解决这个问题。
Apply the 11.1.0.7 patch set (Patch:6890831)

注意：这个bug会造成所有job停止工作！！！所以出现这个错误后，最好尽快重启数据库。


MOS文档
Many ORA-7445 [kslgetl] or ORA-600 [ksl_invalid_latch], [kslges] Errors From CJQ0 Process [ID 579281.1]

Many ORA-7445 [kslgetl] or ORA-600 [ksl_invalid_latch], [kslges] Errors From CJQ0 Process(579281.1)

In this Document
Symptoms
Cause
Solution
References

Applies to:
Oracle Server - Enterprise Edition - Version: 10.2.0.1 to 11.1.0.6
This problem can occur on any platform.
Symptoms

The CJQ process may error frequently with any of the following errors:

ORA-7445 [kslgetl()+40]
ORA-7445 [kkjqfre()+46]
ORA-600 [ksl_invalid_latch], [kslges]

This is an example of what you may see in the alert log
Restarting dead background process CJQ0
CJQ0 started with pid=20, OS id=19744
Exception [type: SIGSEGV, Address not mapped to object] [ADDR:0x2000000026] [PC: 0x7471EE4, kslgetl()+40]
ORA-07445: exception encountered: core dump [kslgetl()+40] [SIGSEGV] [ADDR: 0x2000000026] [PC:0x7471EE4] [Address not mapped to object] []



The call stack in the trace file will resemble the following:

For ORA-7445 [kslgetl()] :  ... kslgetl kkjqspl kkjcjexe kkjssrh ksbcti ksbabs ksbrdp ...

For ORA-7445 [kkjqfre()] : ... kkjcjpr kkjssrh ksbcti ksbabs ksbrdp ...

For ORA-600 [ksl_invalid_latch], [kslges] :  ... kslges kslgetl kkjqspl kkjcjexe kkjssrh ksbcti ksbabs ksbrdp ...


Cause
This is due to Bug:7147411 which is fixed in 11.2 and included in the 11.1.0.7 patch set.

ABSTRACT: LOTS OF ORA-7445 [KSLGETL()+40] REPORTED IN ALERT.LOG

Details: This is a memory corruption that occurs in the CJQ process leading to a dump or internal error.


Solution

Apply the 11.1.0.7 patch set (Patch:6890831)

OR

Check My Oracle Support for the availability of a one-off patch (Patch 7147411) for your specific RDBMS version and platform.

Please note that this bug can also affect pre-11g releases.
References
BUG:6927134 - ORA-7445 [KSLGETL()+40] [SIGSEGV] FROM CJQ PROCESS
BUG:7147411 - LOTS OF ORA-7445 [KSLGETL()+40] REPORTED IN ALERT.LOG
BUG:7201195 - GET ORA-600 [KSL_INVALID_LATCH], [KSLGES]
NOTE:310413.1 - ORA-7445 (kslgetl)