fixup，Amgr两者的共同作用导致服务器宕机

今天碰到一个服务器宕机的情况，错误堆栈战如下：

###################################

## thread   4/6   :: amgr pid=23710, k-id=  49033  , pthr-id=772

## stack          :: k-state=wait, stk max-size=262144, cur-size=7500

###################################

ptrgl.$PTRGL() at 0xd01e01a8

raise.nsleep(??, ??) at 0xd01eebac

raise.nsleep(??, ??) at 0xd01eebac

sleep(??) at 0xd01f9a94

OSRunExternalScript(??) at 0xd3347a28

OSFaultCleanup(??, ??, ??) at 0xd3348a04

fatal_error(??, ??, ??) at 0xd425e744

pth_signal.pthread_kill(??, ??) at 0xd05e776c

pth_signal._p_raise(??) at 0xd05e71e8

raise.raise(??) at 0xd01eef30

Panic(??) at 0xd3179e70

LockMemHandle(??, ??, ??) at 0xd32746d8

OSMemoryLock(??) at 0xd3274c00

OSUnlockVARRAY(??) at 0xd32a99fc

SecondCheckDbForTasks(??) at 0x1001180c

ThreadWrapper(??) at 0xd3152c4c

./tmpfiles.16538.scr: 153: warning: Thread is in kernel mode, not all registers can be accessed.

 

在IBM support网站上找到与之相匹配的参堆栈信息可以参考：

 

是由于fixup运行时间过长导致不能amgr cache刷新，最终导致服务器宕机。需要升级Domino版本。

 

检查服务器的console log,发现的确存在该现象：

[23710:00006-00772] AmgrDbg: Agent 'XXX in database 'eds/docinter.nsf' is intended to run on ‘XXX, not XXX

[23710:00006-00772] AmgrDbg: CheckDbForTask - Found 1 task(s) in database abc.nsf

[23710:00006-00772] AmgrDbg: CheckDbForTask - Found 0 disabled task(s) in database abc.nsf

 

 

[18400:00010-01800] 2007-03-03 23:07:38   Performing consistency check on abc.nsf...

。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。

[18400:00010-01800] 2007-03-04 00:27:24   Completed consistency check on abc.nsf

 

 

参考：

Error: 'AMGR - PANIC: LookupHandle: handle out of range in SecondCheckDbForTasks'

 Technote (FAQ)

Problem

IBM Lotus Domino Server crashes on Amgr, and the following error message occurs:

"PANIC: LookupHandle: handle out of range in SecondCheckDbForTasks"

Stack:

Panic
LockMemHandle
OSMemoryLock
OSUnlockVARRAY
SecondCheckDbForTasks
ThreadWrapper
_lwp_start

Solution

This issue was reported to Quality Engineering as SPR #JPAI676KV5, and is fixed in Domino 6.5.4 Fix Pack 1 and Domino 6.5.5. This issue does not occur in Domino 7.0.
Excerpt from the Lotus Domino Release 6.5.4 Fix Pack 1 (FP1) and 6.5.5 MR fix lists (available at http://www.ibm.com/developerworks/lotus/):

Server
SPR# JPAI676KV5 - Fixed a crash during Fixup. This problem occurs when Fixup takes a long time to run and overlaps the Agent Manager cache refresh.