分类： 系统运维
2015-03-17 10:09:00
AMQ6119 An internal WebSphere MQ error has occurred 报错处理
单位的一台测试服务器搬迁之后，在启动服务器上的队列管理器时，有一个队列管理器起不来了。
首先，dspmq显示该队列管理器的状态是：status not available。
在启动时报错：AMQ6109 An internal WebSphere MQ error has occurred。
别的队列管理器启停都正常，新建的队列管理器启停也正常。
启动时生成的FDC内容如下：
+-----------------------------------------------------------------------------+
| |
| WebSphere MQ First Failure Symptom Report |
| ========================================= |
| |
| Date/Time :- Mon March 16 2015 19:12:24 CST |
| UTC Time :- 1426504344.385480 |
| UTC Time Offset :- 480 (CST) |
| Host Name :- uatcpe1 |
| Operating System :- AIX 7.1 |
| PIDS :- 5724H7221 |
| LVLS :- 7.5.0.3 |
| Product Long Name :- WebSphere MQ for AIX |
| Vendor :- IBM |
| Installation Path :- /usr/mqm |
| Installation Name :- Installation1 (1) |
| License Type :- Production |
| Probe Id :- XY324103 |
| Application Name :- MQM |
| Component :- xstGetSubpoolsLock |
| SCCS Info :- |
| /build/slot1/p750_P/src/lib/cs/unix/generic/amqxiinx.c, |
| Line Number :- 2058 |
| Build Date :- Jan 23 2014 |
| Build Level :- p750-003-140123 |
| Build Type :- IKAP - (Production) |
| Effective UserID :- 0 (root) |
| Real UserID :- 0 (root) |
| Program Name :- dspmq |
| Addressing mode :- 64-bit |
| LANG :- en_US |
| Process :- 11075780 |
| Thread :- 1 |
| QueueManager :- QMYXL03 |
| UserApp :- FALSE |
| ConnId(1) IPCC :- 0 |
| Last HQC :- 0.0.0-0 |
| Last HSHMEMB :- 0.0.0-0 |
| Major Errorcode :- xecF_E_UNEXPECTED_SYSTEM_RC |
| Minor Errorcode :- OK |
| Probe Type :- MSGAMQ6119 |
| Probe Severity :- 2 |
| Probe Description :- AMQ6119: An internal WebSphere MQ error has occurred |
| ('22 - A system call received a parameter that is not val' from semop.) |
| FDCSequenceNumber :- 0 |
| Arith1 :- 22 (0x16) |
| Comment1 :- '22 - A system call received a parameter that is not |
| val' from semop. |
| |
+-----------------------------------------------------------------------------+
在IBM官网查到已经修复的一个bug（IV47710）有类似现象。当队列管理器被异常关闭时，有可能触发这个bug。但根据官网的说明，7503版本已经修复了这个bug。
停止所有的队列管理器，执行amqiclen，没有效果（mqm用户和root用户都执行多次），最后用root用户执行ipcrm，清理掉了所有mqm的ipc资源。此时发现启动失败的那个队列管理器状态自动变成了running（极其诡异），执行endmqm之后再执行strmqm，队列管理器正常启动，然后依次启动其它的队列管理器。
结论：
搬迁时进行了野蛮停机，导致与问题队列管理器相关的ipc资源出现错误。清理ipc资源后恢复正常。
如果是生产环境，本次的处理操作（清理ipc资源）影响还是比较大的，加强日常备份队列管理器配置还是很重要的。那时候可以根据备份快速重建队列管理器。避免类似问题的影响范围太大。
最后附上IBM官网的一篇关于amqiclen和ipcrm操作的文章：
I have heard some people describe how they fixed a WebSphere MQ problem using "ipcrm" or "amqiclen". But is it safe to do this? What are the dangers?
It is not advisable to use these tools. See details below.
WebSphere MQ Support occasionally hears that customers are using ipcrm or amqiclen as part of their normal shutdown procedures.
For example, as part of a script that ends a queue manager using endmqm, they have added code after this command to call ipcrm against mqm-owned semaphore sets and shared memory segments.
WebSphere MQ Support advises that this should NOT be done. This document describes some of the reasoning for this advice.
Semaphores
It is potentially extremely dangerous for data integrity to remove MQ's semaphores while they might still be referenced by a running part of MQ.
MQ uses the semaphores to manage concurrent access to shared data, and to determine whether another copy of the queue manager is already running.
If the semaphores are removed without first ending the queue manager, then a second queue manager can be started on the same machine at the same time.
This is an extremely serious problem because the second copy of the queue manager, thinking it has exclusive access to its data, begins immediately to reference the same queue and log data as is already being referenced by the original copy of the queue manager.
If this happens, the said queue and log data instantly becomes corrupt beyond repair.
The advice from IBM is, leave all of MQ's semaphores in place, and never run ipcrm to remove them. This prevents the risk described above.
Shared memory
It is potentially harmful to remove MQ's shared memory segments while there is any process attached to them.
If no processes are attached then it is not usually harmful. But it still is not needed, so the general advice is still not to do it.
This is a typical scenario. An administrator might end their queue manager and listeners but forget to end one or more of their MQ applications.
A problem can arise if that application is bound to the queue manager using shared memory bindings (which is the norm [1]) and has not made an MQI call for a while. Application processes like this can remain attached to MQ's shared memory segments even though the queue manager has ended.
In this scenario, and supposing the application did not make another MQI call, it will remain attached to the shared memory. The administrator will find that the queue manager later refuses to start, and error messages will be written by the strmqm command. The reason is that, during strmqm, the queue manager startup code detects the presence of those old applications and fails the startup processing.
The error messages from strmqm do state that the application(s) should be ended to allow the queue manager to start. The error messages also mention the process id (pid) to make it easy to find the process concerned and end it.
Despite this, some system administrators have, in our experience, chosen to remove mqm-owned shared memory segments with ipcrm rather than end the applications, in order to allow strmqm to run. This does allow the queue manager to start, but later the applications can fail with a trap (SIGSEGV or similar), and an FDC file.
The advice from IBM is, leave all of MQ's shared memory segments in place. Follow the instructions given in the error messages from strmqm. End the MQ applications [2] that are still attached to the shared memory, and then strmqm will run successfully.
amqiclen
The amqiclen tool was developed to run as part of the WebSphere MQ installer. Its purpose is to check that it is safe to proceed with the install. The checks it makes are whether MQ's shared memory segments are in use by any running processes.
The amqiclen tool is potentially just as dangerous to an MQ system as the ipcrm command.
Therefore, for the same reason WebSphere MQ Support advises not to run ipcrm, we advise also not running amqiclen.
In fact amqiclen was not intended for use by MQ users or administrators directly. When used directly its effects are not supported by WebSphere MQ Support unless they have specifically advised it while working through a PMR.
Instead of running amqiclen to attempt to solve a perceived problem, please instead raise the matter with IBM in a PMR, so that WebSphere MQ Support can gather diagnostics and work through the issue to resolution.
Conclusion
There should never be a need for an administrator to manually remove MQ's shared memory or semaphore sets from the OS.
MQ has been written to tidy up after itself and so manage these things itself.
Therefore ipcrm and amqiclen should not need to be included in normal administration activities.
If you ever think MQ has not managed these resources successfully, they should raise the matter with IBM in a PMR, so that WebSphere MQ Support can gather diagnostics and work through the issue to resolution.
Footnotes
[1] This situation does not affect applications that are connected via "isolated bindings".
[2] Ending MQ applications is generally the cleanest way of causing them to disconnect from MQ's shared memory, but perhaps you do not want to end the applications. If this is the case then another method exists, though this method is only possible if the application is coded sympathetically. In this alternative method, the application must be made to go into a new MQI call. When it goes into the new MQI call it will inspect the data in MQ's shared memory and will find that the queue manager is not running. Upon finding this, the application will report a connection-broken style of error, and will disconnect from the shared memory.
