linux-HA 资源的failcount 问题,On loss of CCM Quorum: Ignore

先看一个例子：

[root@node96 ~]# ps -ef |grep postgres
postgres  4076     1  0 17:17 ?        00:00:00 /data/postgresql-9.2.0/bin/postgres -D /usr/local/pgsql/data -c config_file=/usr/local/pgsql/data/postgresql.conf
postgres  4128  4076  0 17:17 ?        00:00:00 postgres: logger process                                                                                        
postgres  4130  4076  0 17:17 ?        00:00:00 postgres: checkpointer process                                                                                  
postgres  4131  4076  0 17:17 ?        00:00:00 postgres: writer process                                                                                        
postgres  4132  4076  0 17:17 ?        00:00:00 postgres: wal writer process                                                                                    
postgres  4133  4076  0 17:17 ?        00:00:00 postgres: autovacuum launcher process                                                                           
postgres  4134  4076  0 17:17 ?        00:00:00 postgres: archiver process                                                                                      
postgres  4135  4076  0 17:17 ?        00:00:00 postgres: stats collector process                                                                               
postgres  4229  4076  0 17:17 ?        00:00:00 postgres: wal sender process repl 192.168.11.95(35071) streaming 0/170131D8                                     
root      4462  4420  0 17:17 pts/4    00:00:00 su - postgres
postgres  4463  4462  0 17:17 pts/4    00:00:00 -bash
postgres  4493  4463  0 17:17 pts/4    00:00:00 psql
postgres  4494  4076  0 17:17 ?        00:00:00 postgres: postgres postgres [local] idle                                                                        
root      6115 20538  0 17:23 pts/2    00:00:00 grep postgres

找到pg 数据库数据库主进程id  4076

先看看资源状态：

============
Last updated: Wed Sep 19 17:26:06 2012
Last change: Wed Sep 19 17:16:56 2012 via crmd on node95
Stack: openais
Current DC: node96 - partition with quorum
Version: 1.1.7-6.el6-148fccfd5985c5590cc601123c6c16e966b85d14
2 Nodes configured, 2 expected votes
5 Resources configured.
============

Node node95: online
        fence_vm96      (stonith:fence_vmware) Started
Node node96: online
        ClusterIp       (ocf::heartbeat:IPaddr2) Started
        fence_vm95      (stonith:fence_vmware) Started
        ping    (ocf::pacemaker:ping) Started
        postgres_res    (ocf::heartbeat:pgsql) Started

Inactive resources:


Migration summary:
* Node node95:
* Node node96:

然后我们kill -9   数据库的主进程。

看看资源状态

============
Last updated: Wed Sep 19 17:29:06 2012
Last change: Wed Sep 19 17:16:56 2012 via crmd on node95
Stack: openais
Current DC: node96 - partition with quorum
Version: 1.1.7-6.el6-148fccfd5985c5590cc601123c6c16e966b85d14
2 Nodes configured, 2 expected votes
5 Resources configured.
============

Node node95: online
        fence_vm96      (stonith:fence_vmware) Started
Node node96: online
        ClusterIp       (ocf::heartbeat:IPaddr2) Started
        fence_vm95      (stonith:fence_vmware) Started
        ping    (ocf::pacemaker:ping) Started
        postgres_res    (ocf::heartbeat:pgsql) Started

Inactive resources:


Migration summary:
* Node node95:
* Node node96:
   postgres_res: migration-threshold=1000000 fail-count=1

Failed actions:
    postgres_res_monitor_30000 (node=node96, call=45, rc=7, status=complete): not running

数据库资源在node96 重启了，下面多了个错误记录


看看日志：

Sep 19 17:29:05 node96 pgsql(postgres_res)[7479]: ERROR: command failed: su postgres -c cd /usr/local/pgsql/data; kill -s 0 4076 >/dev/null 2>&1             /* 数据库检测失败
Sep 19 17:29:05 node96 pgsql(postgres_res)[7479]: INFO: PostgreSQL is down
Sep 19 17:29:05 node96 crmd[1925]:     info: process_lrm_event: LRM operation postgres_res_monitor_30000 (call=45, rc=7, cib-update=380, confirmed=false) not running
Sep 19 17:29:05 node96 crmd[1925]:     info: process_graph_event: Action postgres_res_monitor_30000 arrived after a completed transition
Sep 19 17:29:05 node96 crmd[1925]:     info: abort_transition_graph: process_graph_event:481 - Triggered transition abort (complete=1, tag=lrm_rsc_op, id=postgres_res_last_failure_0, magic=0:7;5:128:0:87f9cd86-767b-4162-a0c8-da0217d89baf, cib=0.115.12) : Inactive graph
Sep 19 17:29:05 node96 crmd[1925]:  warning: update_failcount: Updating failcount for postgres_res on node96 after failed monitor: rc=7 (update=value++, time=1348046945)   /* 更新集群状态

--------/下面这些就是策略引擎的信息
Sep 19 17:29:05 node96 crmd[1925]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL rigin=abort_transition_graph ]
Sep 19 17:29:05 node96 attrd[1923]:   notice: attrd_trigger_update: Sending flush op to all hosts for: fail-count-postgres_res (1)
Sep 19 17:29:05 node96 attrd[1923]:   notice: attrd_perform_update: Sent update 133: fail-count-postgres_res=1
Sep 19 17:29:05 node96 attrd[1923]:   notice: attrd_trigger_update: Sending flush op to all hosts for: last-failure-postgres_res (1348046945)
Sep 19 17:29:05 node96 pengine[1924]:   notice: unpack_config: On loss of CCM Quorum: Ignore
Sep 19 17:29:05 node96 pengine[1924]:   notice: unpack_rsc_op: Operation monitor found resource postgres_res active on node95
Sep 19 17:29:05 node96 pengine[1924]:  warning: unpack_rsc_op: Processing failed op postgres_res_last_failure_0 on node96: not running (7)
Sep 19 17:29:05 node96 crmd[1925]:     info: abort_transition_graph: te_update_diff:176 - Triggered transition abort (complete=1, tag=nvpair, id=status-node96-fail-count-postgres_res, name=fail-count-postgres_res, value=1, magic=NA, cib=0.115.13) : Transient attribute: update
Sep 19 17:29:05 node96 attrd[1923]:   notice: attrd_perform_update: Sent update 135: last-failure-postgres_res=1348046945
Sep 19 17:29:05 node96 pengine[1924]:   notice: LogActions: Recover postgres_res#011(Started node96)
Sep 19 17:29:05 node96 crmd[1925]:     info: handle_response: pe_calc calculation pe_calc-dc-1348046945-250 is obsolete
Sep 19 17:29:05 node96 crmd[1925]:     info: abort_transition_graph: te_update_diff:176 - Triggered transition abort (complete=1, tag=nvpair, id=status-node96-last-failure-postgres_res, name=last-failure-postgres_res, value=1348046945, magic=NA, cib=0.115.14) : Transient attribute: update
Sep 19 17:29:05 node96 pengine[1924]:   notice: process_pe_message: Transition 129: PEngine Input stored in: /var/lib/pengine/pe-input-285.bz2
Sep 19 17:29:05 node96 pengine[1924]:   notice: unpack_config: On loss of CCM Quorum: Ignore
Sep 19 17:29:05 node96 pengine[1924]:   notice: unpack_rsc_op: Operation monitor found resource postgres_res active on node95
Sep 19 17:29:05 node96 pengine[1924]:  warning: unpack_rsc_op: Processing failed op postgres_res_last_failure_0 on node96: not running (7)
Sep 19 17:29:05 node96 pengine[1924]:   notice: common_apply_stickiness: postgres_res can fail 999999 more times on node96 before being forced off                   /*  这里提到了failcount 还剩下999999次失败的机会。

-----------/下面的日志就是在本地节点重新启动了服务。
Sep 19 17:29:05 node96 pengine[1924]:   notice: LogActions: Recover postgres_res#011(Started node96)
Sep 19 17:29:05 node96 crmd[1925]:   notice: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE rigin=handle_response ]
Sep 19 17:29:05 node96 crmd[1925]:     info: do_te_invoke: Processing graph 130 (ref=pe_calc-dc-1348046945-251) derived from /var/lib/pengine/pe-input-286.bz2
Sep 19 17:29:05 node96 crmd[1925]:     info: te_rsc_command: Initiating action 6: stop postgres_res_stop_0 on node96 (local)
Sep 19 17:29:05 node96 lrmd: [1922]: info: cancel_op: operation monitor[45] on ocf::pgsql::postgres_res for client 1925, its parameters: CRM_meta_depth=[0] pgdba=[postgres] pgdb=[postgres] pgdata=[/usr/local/pgsql/data] config=[/usr/local/pgsql/data/postgresql.conf] depth=[0] psql=[/usr/local/pgsql/bin/psql] pgctl=[/usr/local/pgsql/bin/pg_ctl] start_opt=[] crm_feature_set=[3.0.6] CRM_meta_on_fail=[standby] CRM_meta_name=[monitor] CRM_meta_interval=[30000] CRM_meta_timeout=[30000]  cancelled
Sep 19 17:29:05 node96 lrmd: [1922]: info: rsc:postgres_res:46: stop
Sep 19 17:29:05 node96 crmd[1925]:     info: process_lrm_event: LRM operation postgres_res_monitor_30000 (call=45, status=1, cib-update=0, confirmed=true) Cancelled
Sep 19 17:29:05 node96 pengine[1924]:   notice: process_pe_message: Transition 130: PEngine Input stored in: /var/lib/pengine/pe-input-286.bz2
Sep 19 17:29:05 node96 pgsql(postgres_res)[7523]: ERROR: command failed: su postgres -c cd /usr/local/pgsql/data; kill -s 0 4076 >/dev/null 2>&1
Sep 19 17:29:05 node96 crmd[1925]:     info: process_lrm_event: LRM operation postgres_res_stop_0 (call=46, rc=0, cib-update=384, confirmed=true) ok
Sep 19 17:29:05 node96 crmd[1925]:     info: te_rsc_command: Initiating action 17: start postgres_res_start_0 on node96 (local)
Sep 19 17:29:05 node96 lrmd: [1922]: info: rsc:postgres_res:47: start
Sep 19 17:29:05 node96 pgsql(postgres_res)[7561]: ERROR: command failed: su postgres -c cd /usr/local/pgsql/data; kill -s 0 4076 >/dev/null 2>&1
Sep 19 17:29:05 node96 pgsql(postgres_res)[7561]: INFO: server starting
Sep 19 17:29:05 node96 pgsql(postgres_res)[7561]: INFO: PostgreSQL start command sent.
Sep 19 17:29:05 node96 pgsql(postgres_res)[7561]: WARNING: psql: could not connect to server: Connection refused Is the server running locally and accepting connections on Unix domain socket "/tmp/.s.PGSQL.5432"?
Sep 19 17:29:05 node96 pgsql(postgres_res)[7561]: WARNING: PostgreSQL postgres isn't running
Sep 19 17:29:05 node96 pgsql(postgres_res)[7561]: WARNING: Connection error (connection to the server went bad and the session was not interactive) occurred while executing the psql command.
Sep 19 17:29:06 node96 pgsql(postgres_res)[7561]: INFO: PostgreSQL is started.


我们本来的的愿望是资源要切到备库去的，结果没有切，原因想必就是这个failcount 在做怪了。

pacemaker  里有个参数 ： migration-threshold=N 
设定的是本机上资源失败的次数达到了这个N 以后就会切到standby node 上去了。 并且再也不会切回来了。
除非管理员清理了这个节点上failcount。

就是这部分：
Migration summary:
* Node node95:
* Node node96:
   postgres_res: migration-threshold=1000000 fail-count=1

命令很简单：

cmr resource cleanup   

还有另外一个参数  failure-timeout=N
失败的超时时间

如果同时设置了这两个参数 ，如果达到了 migration-threshold的阀值，导致应用切换到从库，那么经过failure-timeout=N的时间后，可能会导致资源再切回到原来的主库，这个切回来的动作受到(stickiness and constraint scores) 这两个 条件的约束。

这里还涉及到了两个意外情况，在本机start-fail 和 stop-fail  如果是start-fail 会直接跟新failcount 到INFINITY ,  这会导致切换到备机。

如果是stop-fail  ，如果开启了fence设备，会导致fence 设备发生fence动作，然后把资源切到备机。

如果没有启动 fence 设备，那么集群会尝试不停的去stop 应用，但是这个应用不会切刀备机。