Pacemaker+Corosync集群与网络故障检测

Pacemaker+Corosync集群中，Corosync负责消息的传递，成员关系以及QUORUM服务，它实现了TOTEM协议确保应用也就是Pacemaker通过Corosync传递的消息是可靠有序的。由于Corosync负责通信，集群中各种网络故障的检测自然也是Corosync的事情。

TOTEM协议的作用和大名鼎鼎的Paxos，Raft一样同属是分布式共识协议。

http://blog.csdn.net/cxzhq2002/article/details/49563811

totem协议最简单的形象就是，他将多个节点组成一个令牌环。多个节点手拉手形成一个圈，大家依次的传递token。
只有获取到token的节点才有发送消息的权利。简单有效的解决了在分布式系统中各个节点的同步问题，因为只有一个节点会在一个时刻发送消息，
不会出现冲突。当然，如果有节点发生意外时，令牌环就会断掉，此时大家不能够通信，而是重新组建出一个新的令牌环。 

如果要详细了解TOTEM可以参考下面的PPT和Paper

TOTEM

TOTEM虽然使用不可靠的UDP通信方式，但本身有很好的网络容错机制。

• 断开和恢复

  断开时间超过token超时时间（默认1秒），进入Gather状态，形成新的ring。网络断开期间仍然发送数据包到隔离的节点探测节点状态，网络恢复后，再次进入Gather状态，形成新的ring。从corosync感知到网络断开到发起fialover需要时间，测试结果网络闪断在3秒以下通常不会触发failover。

• 丢包

  超过token_retransmit未收到token，触发节点重传，通过token包的seq也可以感知丢包，要求重传。测试结果90%以下的丢包不会对Corosync集群造成影响。

• 延迟

  触发token超时，形成新的ring。延迟的包到达后，收到foreign消息（环成员以外节点发送的消息）进入Gather状态，再次形成新的ring。测试结果网络延迟在1.5秒以下时通常不会触发failover。

• 乱序和重复

  通过ring number和seq可以识别包的顺序，确保投递给应用的消息是有序的。

token超时

调整Corosync行为最重要的一个参数就是token超时时间

与OP协议相关的Corosync选项
?token_retransmit
–Processor在转发完token后，在多长时间内没有收到token或消息后，将引发token重传。
–默认值：238ms
–如果设置了下面的token值，本值由程序自动计算。
?token
–Processor在多长时间内没有收到token（中间包含token重传）后，将触发token丢失事件（将激活MembershipProtocol，进入Gather状态）。
–默认值：1000ms
本值等于Token在Ring中循环一圈的时间，这个时间取决了三个因素：结点数，结点之间的网络速率，每个结点在拿到token后可以发送的max_messages。 

token超时时间默认为1秒，把它改成5秒后，至少可以容忍5秒的闪断和延迟。

vi /etc/corosync/corosync.conf

totem {
...
        token:5000
} 

下面是token超时时间为5秒，断开3节点集群中1个节点(dbhost03)和其它节点的网络时corosync的日志输出。

iptables -A INPUT -j DROP -s dbhost01
iptables -A OUTPUT -j DROP -d dbhost01
iptables -A INPUT -j DROP -s dbhost02
iptables -A OUTPUT -j DROP -d dbhost02 

tail -f /var/log/cluster/corosync.log|grep corosync

dbhost01和dbhost02上经过一段时间形成新的ring

Nov 01 09:44:38 [1680] dbhost01 corosync debug   [QB    ] Free'ing ringbuffer: /dev/shm/qb-cmap-response-1688-6915-25-header
Nov 01 09:44:38 [1680] dbhost01 corosync debug   [QB    ] Free'ing ringbuffer: /dev/shm/qb-cmap-event-1688-6915-25-header
Nov 01 09:44:38 [1680] dbhost01 corosync debug   [QB    ] Free'ing ringbuffer: /dev/shm/qb-cmap-request-1688-6915-25-header
Nov 01 09:44:38 [6915] dbhost01   crm_node:     info: corosync_node_name:   Unable to get node name for nodeid 1
Nov 01 09:44:38 [6915] dbhost01   crm_node:   notice: get_node_name:    Defaulting to uname -n for the local corosync node name
Nov 01 09:44:43 [1680] dbhost01 corosync debug   [TOTEM ] The token was lost in the OPERATIONAL state.
Nov 01 09:44:43 [1680] dbhost01 corosync notice  [TOTEM ] A processor failed, forming new configuration.
Nov 01 09:44:43 [1680] dbhost01 corosync debug   [TOTEM ] entering GATHER state from 2(The token was lost in the OPERATIONAL state.).
Nov 01 09:44:43 [7121] dbhost01   crm_node:     info: get_cluster_type: Verifying cluster type: 'corosync'
Nov 01 09:44:43 [7121] dbhost01   crm_node:     info: get_cluster_type: Assuming an active 'corosync' cluster
Nov 01 09:44:43 [1680] dbhost01 corosync debug   [QB    ] IPC credentials authenticated (1688-7121-25)
Nov 01 09:44:43 [1680] dbhost01 corosync debug   [QB    ] connecting to client [7121]
Nov 01 09:44:43 [1680] dbhost01 corosync debug   [QB    ] shm size:1048589; real_size:1052672; rb->word_size:263168
Nov 01 09:44:43 [1680] dbhost01 corosync debug   [QB    ] shm size:1048589; real_size:1052672; rb->word_size:263168
Nov 01 09:44:43 [1680] dbhost01 corosync debug   [QB    ] shm size:1048589; real_size:1052672; rb->word_size:263168
Nov 01 09:44:43 [1680] dbhost01 corosync debug   [MAIN  ] connection created
Nov 01 09:44:43 [1680] dbhost01 corosync debug   [CPG   ] lib_init_fn: conn=0x7f1ff377c7d0, cpd=0x7f1ff377ce84
Nov 01 09:44:43 [1680] dbhost01 corosync debug   [CPG   ] cpg finalize for conn=0x7f1ff377c7d0
Nov 01 09:44:43 [1680] dbhost01 corosync debug   [QB    ] HUP conn (1688-7121-25)
Nov 01 09:44:43 [1680] dbhost01 corosync debug   [QB    ] qb_ipcs_disconnect(1688-7121-25) state:2
Nov 01 09:44:43 [1680] dbhost01 corosync debug   [MAIN  ] cs_ipcs_connection_closed() 
Nov 01 09:44:43 [1680] dbhost01 corosync debug   [CPG   ] exit_fn for conn=0x7f1ff377c7d0
Nov 01 09:44:43 [1680] dbhost01 corosync debug   [MAIN  ] cs_ipcs_connection_destroyed() 
Nov 01 09:44:43 [1680] dbhost01 corosync debug   [QB    ] Free'ing ringbuffer: /dev/shm/qb-cpg-response-1688-7121-25-header
Nov 01 09:44:43 [1680] dbhost01 corosync debug   [QB    ] Free'ing ringbuffer: /dev/shm/qb-cpg-event-1688-7121-25-header
Nov 01 09:44:43 [1680] dbhost01 corosync debug   [QB    ] Free'ing ringbuffer: /dev/shm/qb-cpg-request-1688-7121-25-header
Nov 01 09:44:43 [1680] dbhost01 corosync debug   [QB    ] IPC credentials authenticated (1688-7121-25)
Nov 01 09:44:43 [1680] dbhost01 corosync debug   [QB    ] connecting to client [7121]
Nov 01 09:44:43 [1680] dbhost01 corosync debug   [QB    ] shm size:1048589; real_size:1052672; rb->word_size:263168
Nov 01 09:44:43 [1680] dbhost01 corosync debug   [QB    ] shm size:1048589; real_size:1052672; rb->word_size:263168
Nov 01 09:44:43 [1680] dbhost01 corosync debug   [QB    ] shm size:1048589; real_size:1052672; rb->word_size:263168
Nov 01 09:44:43 [1680] dbhost01 corosync debug   [MAIN  ] connection created
Nov 01 09:44:43 [1680] dbhost01 corosync debug   [CMAP  ] lib_init_fn: conn=0x7f1ff37794a0
Nov 01 09:44:43 [1680] dbhost01 corosync debug   [QB    ] HUP conn (1688-7121-25)
Nov 01 09:44:43 [1680] dbhost01 corosync debug   [QB    ] qb_ipcs_disconnect(1688-7121-25) state:2
Nov 01 09:44:43 [1680] dbhost01 corosync debug   [MAIN  ] cs_ipcs_connection_closed() 
Nov 01 09:44:43 [1680] dbhost01 corosync debug   [CMAP  ] exit_fn for conn=0x7f1ff37794a0
Nov 01 09:44:43 [7121] dbhost01   crm_node:     info: corosync_node_name:   Unable to get node name for nodeid 1
Nov 01 09:44:43 [1680] dbhost01 corosync debug   [MAIN  ] cs_ipcs_connection_destroyed() 
Nov 01 09:44:43 [1680] dbhost01 corosync debug   [QB    ] Free'ing ringbuffer: /dev/shm/qb-cmap-response-1688-7121-25-header
Nov 01 09:44:43 [7121] dbhost01   crm_node:   notice: get_node_name:    Defaulting to uname -n for the local corosync node name
Nov 01 09:44:43 [1680] dbhost01 corosync debug   [QB    ] Free'ing ringbuffer: /dev/shm/qb-cmap-event-1688-7121-25-header
Nov 01 09:44:43 [1680] dbhost01 corosync debug   [QB    ] Free'ing ringbuffer: /dev/shm/qb-cmap-request-1688-7121-25-header
Nov 01 09:44:49 [7196] dbhost01   crm_node:     info: get_cluster_type: Verifying cluster type: 'corosync'
Nov 01 09:44:49 [7196] dbhost01   crm_node:     info: get_cluster_type: Assuming an active 'corosync' cluster
Nov 01 09:44:49 [1680] dbhost01 corosync debug   [QB    ] IPC credentials authenticated (1688-7196-25)
Nov 01 09:44:49 [1680] dbhost01 corosync debug   [QB    ] connecting to client [7196]
Nov 01 09:44:49 [1680] dbhost01 corosync debug   [QB    ] shm size:1048589; real_size:1052672; rb->word_size:263168
Nov 01 09:44:49 [1680] dbhost01 corosync debug   [QB    ] shm size:1048589; real_size:1052672; rb->word_size:263168
Nov 01 09:44:49 [1680] dbhost01 corosync debug   [QB    ] shm size:1048589; real_size:1052672; rb->word_size:263168
Nov 01 09:44:49 [1680] dbhost01 corosync debug   [MAIN  ] connection created
Nov 01 09:44:49 [1680] dbhost01 corosync debug   [CPG   ] lib_init_fn: conn=0x7f1ff37794a0, cpd=0x7f1ff377a2f4
Nov 01 09:44:49 [1680] dbhost01 corosync debug   [CPG   ] cpg finalize for conn=0x7f1ff37794a0
Nov 01 09:44:49 [1680] dbhost01 corosync debug   [QB    ] HUP conn (1688-7196-25)
Nov 01 09:44:49 [1680] dbhost01 corosync debug   [QB    ] qb_ipcs_disconnect(1688-7196-25) state:2
Nov 01 09:44:49 [1680] dbhost01 corosync debug   [MAIN  ] cs_ipcs_connection_closed() 
Nov 01 09:44:49 [1680] dbhost01 corosync debug   [CPG   ] exit_fn for conn=0x7f1ff37794a0
Nov 01 09:44:49 [1680] dbhost01 corosync debug   [MAIN  ] cs_ipcs_connection_destroyed() 
Nov 01 09:44:49 [1680] dbhost01 corosync debug   [QB    ] Free'ing ringbuffer: /dev/shm/qb-cpg-response-1688-7196-25-header
Nov 01 09:44:49 [1680] dbhost01 corosync debug   [QB    ] Free'ing ringbuffer: /dev/shm/qb-cpg-event-1688-7196-25-header
Nov 01 09:44:49 [1680] dbhost01 corosync debug   [QB    ] Free'ing ringbuffer: /dev/shm/qb-cpg-request-1688-7196-25-header
Nov 01 09:44:49 [1680] dbhost01 corosync debug   [QB    ] IPC credentials authenticated (1688-7196-25)
Nov 01 09:44:49 [1680] dbhost01 corosync debug   [QB    ] connecting to client [7196]
Nov 01 09:44:49 [1680] dbhost01 corosync debug   [QB    ] shm size:1048589; real_size:1052672; rb->word_size:263168
Nov 01 09:44:49 [1680] dbhost01 corosync debug   [QB    ] shm size:1048589; real_size:1052672; rb->word_size:263168
Nov 01 09:44:49 [1680] dbhost01 corosync debug   [QB    ] shm size:1048589; real_size:1052672; rb->word_size:263168
Nov 01 09:44:49 [1680] dbhost01 corosync debug   [MAIN  ] connection created
Nov 01 09:44:49 [1680] dbhost01 corosync debug   [CMAP  ] lib_init_fn: conn=0x7f1ff37794a0
Nov 01 09:44:49 [1680] dbhost01 corosync debug   [QB    ] HUP conn (1688-7196-25)
Nov 01 09:44:49 [1680] dbhost01 corosync debug   [QB    ] qb_ipcs_disconnect(1688-7196-25) state:2
Nov 01 09:44:49 [1680] dbhost01 corosync debug   [MAIN  ] cs_ipcs_connection_closed() 
Nov 01 09:44:49 [1680] dbhost01 corosync debug   [CMAP  ] exit_fn for conn=0x7f1ff37794a0
Nov 01 09:44:49 [1680] dbhost01 corosync debug   [MAIN  ] cs_ipcs_connection_destroyed() 
Nov 01 09:44:49 [1680] dbhost01 corosync debug   [QB    ] Free'ing ringbuffer: /dev/shm/qb-cmap-response-1688-7196-25-header
Nov 01 09:44:49 [1680] dbhost01 corosync debug   [QB    ] Free'ing ringbuffer: /dev/shm/qb-cmap-event-1688-7196-25-header
Nov 01 09:44:49 [1680] dbhost01 corosync debug   [QB    ] Free'ing ringbuffer: /dev/shm/qb-cmap-request-1688-7196-25-header
Nov 01 09:44:49 [7196] dbhost01   crm_node:     info: corosync_node_name:   Unable to get node name for nodeid 1
Nov 01 09:44:49 [7196] dbhost01   crm_node:   notice: get_node_name:    Defaulting to uname -n for the local corosync node name
Nov 01 09:44:49 [1680] dbhost01 corosync debug   [TOTEM ] entering GATHER state from 0(consensus timeout).
Nov 01 09:44:49 [1680] dbhost01 corosync debug   [TOTEM ] Creating commit token because I am the rep.
Nov 01 09:44:49 [1680] dbhost01 corosync debug   [TOTEM ] Saving state aru 63b high seq received 63b
Nov 01 09:44:49 [1680] dbhost01 corosync debug   [TOTEM ] Storing new sequence id for ring 7f160
Nov 01 09:44:49 [1680] dbhost01 corosync debug   [TOTEM ] entering COMMIT state.
Nov 01 09:44:49 [1680] dbhost01 corosync debug   [TOTEM ] got commit token
Nov 01 09:44:49 [1680] dbhost01 corosync debug   [TOTEM ] entering RECOVERY state.
Nov 01 09:44:49 [1680] dbhost01 corosync debug   [TOTEM ] TRANS [0] member 10.37.20.193:
Nov 01 09:44:49 [1680] dbhost01 corosync debug   [TOTEM ] TRANS [1] member 10.37.20.195:
Nov 01 09:44:49 [1680] dbhost01 corosync debug   [TOTEM ] position [0] member 10.37.20.193:
Nov 01 09:44:49 [1680] dbhost01 corosync debug   [TOTEM ] previous ring seq 7f15c rep 10.37.20.193
Nov 01 09:44:49 [1680] dbhost01 corosync debug   [TOTEM ] aru 63b high delivered 63b received flag 1
Nov 01 09:44:49 [1680] dbhost01 corosync debug   [TOTEM ] position [1] member 10.37.20.195:
Nov 01 09:44:49 [1680] dbhost01 corosync debug   [TOTEM ] previous ring seq 7f15c rep 10.37.20.193
Nov 01 09:44:49 [1680] dbhost01 corosync debug   [TOTEM ] aru 63b high delivered 63b received flag 1
Nov 01 09:44:49 [1680] dbhost01 corosync debug   [TOTEM ] Did not need to originate any messages in recovery.
Nov 01 09:44:49 [1680] dbhost01 corosync debug   [TOTEM ] got commit token
Nov 01 09:44:49 [1680] dbhost01 corosync debug   [TOTEM ] Sending initial ORF token
Nov 01 09:44:49 [1680] dbhost01 corosync debug   [TOTEM ] token retrans flag is 0 my set retrans flag0 retrans queue empty 1 count 0, aru 0
Nov 01 09:44:49 [1680] dbhost01 corosync debug   [TOTEM ] install seq 0 aru 0 high seq received 0
Nov 01 09:44:49 [1680] dbhost01 corosync debug   [TOTEM ] token retrans flag is 0 my set retrans flag0 retrans queue empty 1 count 1, aru 0
Nov 01 09:44:49 [1680] dbhost01 corosync debug   [TOTEM ] install seq 0 aru 0 high seq received 0
Nov 01 09:44:49 [1680] dbhost01 corosync debug   [TOTEM ] token retrans flag is 0 my set retrans flag0 retrans queue empty 1 count 2, aru 0
Nov 01 09:44:49 [1680] dbhost01 corosync debug   [TOTEM ] install seq 0 aru 0 high seq received 0
Nov 01 09:44:49 [1680] dbhost01 corosync debug   [TOTEM ] token retrans flag is 0 my set retrans flag0 retrans queue empty 1 count 3, aru 0
Nov 01 09:44:49 [1680] dbhost01 corosync debug   [TOTEM ] install seq 0 aru 0 high seq received 0
Nov 01 09:44:49 [1680] dbhost01 corosync debug   [TOTEM ] retrans flag count 4 token aru 0 install seq 0 aru 0 0
Nov 01 09:44:49 [1680] dbhost01 corosync debug   [TOTEM ] Resetting old ring state
Nov 01 09:44:49 [1680] dbhost01 corosync debug   [TOTEM ] recovery to regular 1-0
Nov 01 09:44:49 [1680] dbhost01 corosync debug   [MAIN  ] Member left: r(0) ip(10.37.20.196) 
Nov 01 09:44:49 [1680] dbhost01 corosync debug   [TOTEM ] waiting_trans_ack changed to 1
Nov 01 09:44:49 [1680] dbhost01 corosync debug   [TOTEM ] entering OPERATIONAL state.
Nov 01 09:44:49 [1680] dbhost01 corosync notice  [TOTEM ] A new membership (10.37.20.193:520544) was formed. Members left: 3
Nov 01 09:44:49 [1680] dbhost01 corosync debug   [SYNC  ] Committing synchronization for corosync configuration map access
Nov 01 09:44:49 [1680] dbhost01 corosync debug   [CMAP  ] Not first sync -> no action
Nov 01 09:44:49 [1680] dbhost01 corosync debug   [CPG   ] got joinlist message from node 2
Nov 01 09:44:49 [1680] dbhost01 corosync debug   [CPG   ] comparing: sender r(0) ip(10.37.20.193) ; members(old:3 left:1)
Nov 01 09:44:49 [1680] dbhost01 corosync debug   [CPG   ] comparing: sender r(0) ip(10.37.20.195) ; members(old:3 left:1)
Nov 01 09:44:49 [1680] dbhost01 corosync debug   [CPG   ] chosen downlist: sender r(0) ip(10.37.20.193) ; members(old:3 left:1)
Nov 01 09:44:49 [1680] dbhost01 corosync debug   [CPG   ] left_list_entries:1
Nov 01 09:44:49 [1680] dbhost01 corosync debug   [CPG   ] left_list[0] group:attrd\x00, ip:r(0) ip(10.37.20.196) , pid:24157
Nov 01 09:44:49 [1792] dbhost01      attrd:     info: crm_update_peer_proc: pcmk_cpg_membership: Node dbhost03[3] - corosync-cpg is now offline
Nov 01 09:44:49 [1680] dbhost01 corosync debug   [CPG   ] left_list_entries:1
Nov 01 09:44:49 [1680] dbhost01 corosync debug   [CPG   ] left_list[0] group:cib\x00, ip:r(0) ip(10.37.20.196) , pid:24154
Nov 01 09:44:49 [1789] dbhost01        cib:     info: crm_update_peer_proc: pcmk_cpg_membership: Node dbhost03[3] - corosync-cpg is now offline
Nov 01 09:44:49 [1680] dbhost01 corosync debug   [CPG   ] left_list_entries:1
Nov 01 09:44:49 [1680] dbhost01 corosync debug   [CPG   ] left_list[0] group:crmd\x00, ip:r(0) ip(10.37.20.196) , pid:24159
Nov 01 09:44:49 [1794] dbhost01       crmd:     info: crm_update_peer_proc: pcmk_cpg_membership: Node dbhost03[3] - corosync-cpg is now offline
Nov 01 09:44:49 [1680] dbhost01 corosync debug   [CPG   ] left_list_entries:1
Nov 01 09:44:49 [1680] dbhost01 corosync debug   [CPG   ] left_list[0] group:pacemakerd\x00, ip:r(0) ip(10.37.20.196) , pid:24152
Nov 01 09:44:49 [1787] dbhost01 pacemakerd:     info: crm_update_peer_proc: pcmk_cpg_membership: Node dbhost03[3] - corosync-cpg is now offline
Nov 01 09:44:49 [1790] dbhost01 stonith-ng:     info: crm_update_peer_proc: pcmk_cpg_membership: Node dbhost03[3] - corosync-cpg is now offline
Nov 01 09:44:49 [1680] dbhost01 corosync debug   [CPG   ] left_list_entries:1
Nov 01 09:44:49 [1680] dbhost01 corosync debug   [CPG   ] left_list[0] group:stonith-ng\x00, ip:r(0) ip(10.37.20.196) , pid:24155
Nov 01 09:44:49 [1680] dbhost01 corosync debug   [CPG   ] got joinlist message from node 1
Nov 01 09:44:49 [1680] dbhost01 corosync debug   [SYNC  ] Committing synchronization for corosync cluster closed process group service v1.01
Nov 01 09:44:49 [1680] dbhost01 corosync debug   [CPG   ] joinlist_messages[0] group:crmd\x00, ip:r(0) ip(10.37.20.193) , pid:1794
Nov 01 09:44:49 [1680] dbhost01 corosync debug   [CPG   ] joinlist_messages[1] group:attrd\x00, ip:r(0) ip(10.37.20.193) , pid:1792
Nov 01 09:44:49 [1680] dbhost01 corosync debug   [CPG   ] joinlist_messages[2] group:stonith-ng\x00, ip:r(0) ip(10.37.20.193) , pid:1790
Nov 01 09:44:49 [1680] dbhost01 corosync debug   [CPG   ] joinlist_messages[3] group:cib\x00, ip:r(0) ip(10.37.20.193) , pid:1789
Nov 01 09:44:49 [1680] dbhost01 corosync debug   [CPG   ] joinlist_messages[4] group:pacemakerd\x00, ip:r(0) ip(10.37.20.193) , pid:1787
Nov 01 09:44:49 [1680] dbhost01 corosync debug   [CPG   ] joinlist_messages[5] group:crmd\x00, ip:r(0) ip(10.37.20.195) , pid:437
Nov 01 09:44:49 [1680] dbhost01 corosync debug   [CPG   ] joinlist_messages[6] group:attrd\x00, ip:r(0) ip(10.37.20.195) , pid:435
Nov 01 09:44:49 [1680] dbhost01 corosync debug   [CPG   ] joinlist_messages[7] group:stonith-ng\x00, ip:r(0) ip(10.37.20.195) , pid:433
Nov 01 09:44:49 [1680] dbhost01 corosync debug   [CPG   ] joinlist_messages[8] group:cib\x00, ip:r(0) ip(10.37.20.195) , pid:432
Nov 01 09:44:49 [1680] dbhost01 corosync debug   [CPG   ] joinlist_messages[9] group:pacemakerd\x00, ip:r(0) ip(10.37.20.195) , pid:430
Nov 01 09:44:49 [1680] dbhost01 corosync debug   [VOTEQ ] flags: quorate: Yes Leaving: No WFA Status: No First: No Qdevice: No QdeviceAlive: No QdeviceCastVote: No QdeviceMasterWins: No
Nov 01 09:44:49 [1680] dbhost01 corosync debug   [VOTEQ ] got nodeinfo message from cluster node 1
Nov 01 09:44:49 [1680] dbhost01 corosync debug   [VOTEQ ] nodeinfo message[1]: votes: 1, expected: 3 flags: 1
Nov 01 09:44:49 [1680] dbhost01 corosync debug   [VOTEQ ] flags: quorate: Yes Leaving: No WFA Status: No First: No Qdevice: No QdeviceAlive: No QdeviceCastVote: No QdeviceMasterWins: No
Nov 01 09:44:49 [1680] dbhost01 corosync debug   [VOTEQ ] total_votes=2, expected_votes=3
Nov 01 09:44:49 [1680] dbhost01 corosync debug   [VOTEQ ] node 1 state=1, votes=1, expected=3
Nov 01 09:44:49 [1680] dbhost01 corosync debug   [VOTEQ ] node 2 state=1, votes=1, expected=3
Nov 01 09:44:49 [1680] dbhost01 corosync debug   [VOTEQ ] node 3 state=2, votes=1, expected=3
Nov 01 09:44:49 [1680] dbhost01 corosync debug   [VOTEQ ] lowest node id: 1 us: 1
Nov 01 09:44:49 [1680] dbhost01 corosync debug   [VOTEQ ] got nodeinfo message from cluster node 1
Nov 01 09:44:49 [1680] dbhost01 corosync debug   [VOTEQ ] nodeinfo message[0]: votes: 0, expected: 0 flags: 0
Nov 01 09:44:49 [1680] dbhost01 corosync debug   [VOTEQ ] got nodeinfo message from cluster node 2
Nov 01 09:44:49 [1680] dbhost01 corosync debug   [VOTEQ ] nodeinfo message[2]: votes: 1, expected: 3 flags: 1
Nov 01 09:44:49 [1680] dbhost01 corosync debug   [VOTEQ ] flags: quorate: Yes Leaving: No WFA Status: No First: No Qdevice: No QdeviceAlive: No QdeviceCastVote: No QdeviceMasterWins: No
Nov 01 09:44:49 [1680] dbhost01 corosync debug   [VOTEQ ] got nodeinfo message from cluster node 2
Nov 01 09:44:49 [1680] dbhost01 corosync debug   [VOTEQ ] nodeinfo message[0]: votes: 0, expected: 0 flags: 0
Nov 01 09:44:49 [1680] dbhost01 corosync debug   [SYNC  ] Committing synchronization for corosync vote quorum service v1.0
Nov 01 09:44:49 [1680] dbhost01 corosync debug   [VOTEQ ] total_votes=2, expected_votes=3
Nov 01 09:44:49 [1680] dbhost01 corosync debug   [VOTEQ ] node 1 state=1, votes=1, expected=3
Nov 01 09:44:49 [1680] dbhost01 corosync debug   [VOTEQ ] node 2 state=1, votes=1, expected=3
Nov 01 09:44:49 [1680] dbhost01 corosync debug   [VOTEQ ] node 3 state=2, votes=1, expected=3
Nov 01 09:44:49 [1680] dbhost01 corosync debug   [VOTEQ ] lowest node id: 1 us: 1
Nov 01 09:44:49 [1680] dbhost01 corosync notice  [QUORUM] Members[2]: 1 2
Nov 01 09:44:49 [1680] dbhost01 corosync debug   [QUORUM] sending quorum notification to (nil), length = 56
Nov 01 09:44:49 [1680] dbhost01 corosync notice  [MAIN  ] Completed service synchronization, ready to provide service.
Nov 01 09:44:49 [1680] dbhost01 corosync debug   [TOTEM ] waiting_trans_ack changed to 0 

dbhost03上也形成只包含自己的ring

Nov 01 09:44:40 [1777] dbhost03   crm_node:     info: get_cluster_type: Verifying cluster type: 'corosync'
Nov 01 09:44:40 [1777] dbhost03   crm_node:     info: get_cluster_type: Assuming an active 'corosync' cluster
Nov 01 09:44:40 [24023] dbhost03 corosync debug   [QB    ] IPC credentials authenticated (24024-1777-25)
Nov 01 09:44:40 [24023] dbhost03 corosync debug   [QB    ] connecting to client [1777]
Nov 01 09:44:40 [24023] dbhost03 corosync debug   [QB    ] shm size:1048589; real_size:1052672; rb->word_size:263168
Nov 01 09:44:40 [24023] dbhost03 corosync debug   [QB    ] shm size:1048589; real_size:1052672; rb->word_size:263168
Nov 01 09:44:40 [24023] dbhost03 corosync debug   [QB    ] shm size:1048589; real_size:1052672; rb->word_size:263168
Nov 01 09:44:40 [24023] dbhost03 corosync debug   [MAIN  ] connection created
Nov 01 09:44:40 [24023] dbhost03 corosync debug   [CPG   ] lib_init_fn: conn=0x7f1243543fd0, cpd=0x7f124323c594
Nov 01 09:44:40 [24023] dbhost03 corosync debug   [CPG   ] cpg finalize for conn=0x7f1243543fd0
Nov 01 09:44:40 [24023] dbhost03 corosync debug   [QB    ] HUP conn (24024-1777-25)
Nov 01 09:44:40 [24023] dbhost03 corosync debug   [QB    ] qb_ipcs_disconnect(24024-1777-25) state:2
Nov 01 09:44:40 [24023] dbhost03 corosync debug   [MAIN  ] cs_ipcs_connection_closed() 
Nov 01 09:44:40 [24023] dbhost03 corosync debug   [CPG   ] exit_fn for conn=0x7f1243543fd0
Nov 01 09:44:40 [24023] dbhost03 corosync debug   [MAIN  ] cs_ipcs_connection_destroyed() 
Nov 01 09:44:40 [24023] dbhost03 corosync debug   [QB    ] Free'ing ringbuffer: /dev/shm/qb-cpg-response-24024-1777-25-header
Nov 01 09:44:40 [24023] dbhost03 corosync debug   [QB    ] Free'ing ringbuffer: /dev/shm/qb-cpg-event-24024-1777-25-header
Nov 01 09:44:40 [24023] dbhost03 corosync debug   [QB    ] Free'ing ringbuffer: /dev/shm/qb-cpg-request-24024-1777-25-header
Nov 01 09:44:40 [24023] dbhost03 corosync debug   [QB    ] IPC credentials authenticated (24024-1777-25)
Nov 01 09:44:40 [24023] dbhost03 corosync debug   [QB    ] connecting to client [1777]
Nov 01 09:44:40 [24023] dbhost03 corosync debug   [QB    ] shm size:1048589; real_size:1052672; rb->word_size:263168
Nov 01 09:44:40 [24023] dbhost03 corosync debug   [QB    ] shm size:1048589; real_size:1052672; rb->word_size:263168
Nov 01 09:44:40 [24023] dbhost03 corosync debug   [QB    ] shm size:1048589; real_size:1052672; rb->word_size:263168
Nov 01 09:44:40 [24023] dbhost03 corosync debug   [MAIN  ] connection created
Nov 01 09:44:40 [24023] dbhost03 corosync debug   [CMAP  ] lib_init_fn: conn=0x7f1243543fd0
Nov 01 09:44:40 [1777] dbhost03   crm_node:     info: corosync_node_name:   Unable to get node name for nodeid 3
Nov 01 09:44:40 [1777] dbhost03   crm_node:   notice: get_node_name:    Defaulting to uname -n for the local corosync node name
Nov 01 09:44:40 [24023] dbhost03 corosync debug   [QB    ] HUP conn (24024-1777-25)
Nov 01 09:44:40 [24023] dbhost03 corosync debug   [QB    ] qb_ipcs_disconnect(24024-1777-25) state:2
Nov 01 09:44:40 [24023] dbhost03 corosync debug   [MAIN  ] cs_ipcs_connection_closed() 
Nov 01 09:44:40 [24023] dbhost03 corosync debug   [CMAP  ] exit_fn for conn=0x7f1243543fd0
Nov 01 09:44:40 [24023] dbhost03 corosync debug   [MAIN  ] cs_ipcs_connection_destroyed() 
Nov 01 09:44:40 [24023] dbhost03 corosync debug   [QB    ] Free'ing ringbuffer: /dev/shm/qb-cmap-response-24024-1777-25-header
Nov 01 09:44:40 [24023] dbhost03 corosync debug   [QB    ] Free'ing ringbuffer: /dev/shm/qb-cmap-event-24024-1777-25-header
Nov 01 09:44:40 [24023] dbhost03 corosync debug   [QB    ] Free'ing ringbuffer: /dev/shm/qb-cmap-request-24024-1777-25-header
Nov 01 09:44:40 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(ucast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:40 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:40 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:41 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(ucast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:42 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(ucast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:43 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(ucast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:43 [24023] dbhost03 corosync debug   [TOTEM ] The token was lost in the OPERATIONAL state.
Nov 01 09:44:43 [24023] dbhost03 corosync notice  [TOTEM ] A processor failed, forming new configuration.
Nov 01 09:44:43 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:43 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:43 [24023] dbhost03 corosync debug   [TOTEM ] entering GATHER state from 2(The token was lost in the OPERATIONAL state.).
Nov 01 09:44:43 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:43 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:43 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:43 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:43 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:43 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:44 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:44 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:44 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:44 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:44 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:44 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:44 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:44 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:44 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:44 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:44 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:44 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:44 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:44 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:44 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:44 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:44 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:44 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:44 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:44 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:44 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:44 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:44 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:44 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:44 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:44 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:44 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:44 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:44 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:44 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:44 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:44 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:44 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:44 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:44 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:44 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:44 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:44 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:44 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:44 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:45 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:45 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:45 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:45 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:45 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:45 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:45 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:45 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:45 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:45 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:45 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:45 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:45 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:45 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:45 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:45 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:45 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:45 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:45 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:45 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:45 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:45 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:45 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:45 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:45 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:45 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:45 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:45 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:45 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:45 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:45 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:45 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:45 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:45 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:45 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:45 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:45 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:45 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:45 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:45 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:46 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:46 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:46 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:46 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:46 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:46 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:46 [2075] dbhost03   crm_node:     info: get_cluster_type: Verifying cluster type: 'corosync'
Nov 01 09:44:46 [2075] dbhost03   crm_node:     info: get_cluster_type: Assuming an active 'corosync' cluster
Nov 01 09:44:46 [24023] dbhost03 corosync debug   [QB    ] IPC credentials authenticated (24024-2075-25)
Nov 01 09:44:46 [24023] dbhost03 corosync debug   [QB    ] connecting to client [2075]
Nov 01 09:44:46 [24023] dbhost03 corosync debug   [QB    ] shm size:1048589; real_size:1052672; rb->word_size:263168
Nov 01 09:44:46 [24023] dbhost03 corosync debug   [QB    ] shm size:1048589; real_size:1052672; rb->word_size:263168
Nov 01 09:44:46 [24023] dbhost03 corosync debug   [QB    ] shm size:1048589; real_size:1052672; rb->word_size:263168
Nov 01 09:44:46 [24023] dbhost03 corosync debug   [MAIN  ] connection created
Nov 01 09:44:46 [24023] dbhost03 corosync debug   [CPG   ] lib_init_fn: conn=0x7f12435520a0, cpd=0x7f124323c594
Nov 01 09:44:46 [24023] dbhost03 corosync debug   [CPG   ] cpg finalize for conn=0x7f12435520a0
Nov 01 09:44:46 [24023] dbhost03 corosync debug   [QB    ] HUP conn (24024-2075-25)
Nov 01 09:44:46 [24023] dbhost03 corosync debug   [QB    ] qb_ipcs_disconnect(24024-2075-25) state:2
Nov 01 09:44:46 [24023] dbhost03 corosync debug   [MAIN  ] cs_ipcs_connection_closed() 
Nov 01 09:44:46 [24023] dbhost03 corosync debug   [CPG   ] exit_fn for conn=0x7f12435520a0
Nov 01 09:44:46 [24023] dbhost03 corosync debug   [MAIN  ] cs_ipcs_connection_destroyed() 
Nov 01 09:44:46 [24023] dbhost03 corosync debug   [QB    ] Free'ing ringbuffer: /dev/shm/qb-cpg-response-24024-2075-25-header
Nov 01 09:44:46 [24023] dbhost03 corosync debug   [QB    ] Free'ing ringbuffer: /dev/shm/qb-cpg-event-24024-2075-25-header
Nov 01 09:44:46 [24023] dbhost03 corosync debug   [QB    ] Free'ing ringbuffer: /dev/shm/qb-cpg-request-24024-2075-25-header
Nov 01 09:44:46 [24023] dbhost03 corosync debug   [QB    ] IPC credentials authenticated (24024-2075-25)
Nov 01 09:44:46 [24023] dbhost03 corosync debug   [QB    ] connecting to client [2075]
Nov 01 09:44:46 [24023] dbhost03 corosync debug   [QB    ] shm size:1048589; real_size:1052672; rb->word_size:263168
Nov 01 09:44:46 [24023] dbhost03 corosync debug   [QB    ] shm size:1048589; real_size:1052672; rb->word_size:263168
Nov 01 09:44:46 [24023] dbhost03 corosync debug   [QB    ] shm size:1048589; real_size:1052672; rb->word_size:263168
Nov 01 09:44:46 [24023] dbhost03 corosync debug   [MAIN  ] connection created
Nov 01 09:44:46 [24023] dbhost03 corosync debug   [CMAP  ] lib_init_fn: conn=0x7f12435520a0
Nov 01 09:44:46 [24023] dbhost03 corosync debug   [QB    ] HUP conn (24024-2075-25)
Nov 01 09:44:46 [24023] dbhost03 corosync debug   [QB    ] qb_ipcs_disconnect(24024-2075-25) state:2
Nov 01 09:44:46 [24023] dbhost03 corosync debug   [MAIN  ] cs_ipcs_connection_closed() 
Nov 01 09:44:46 [24023] dbhost03 corosync debug   [CMAP  ] exit_fn for conn=0x7f12435520a0
Nov 01 09:44:46 [24023] dbhost03 corosync debug   [MAIN  ] cs_ipcs_connection_destroyed() 
Nov 01 09:44:46 [24023] dbhost03 corosync debug   [QB    ] Free'ing ringbuffer: /dev/shm/qb-cmap-response-24024-2075-25-header
Nov 01 09:44:46 [24023] dbhost03 corosync debug   [QB    ] Free'ing ringbuffer: /dev/shm/qb-cmap-event-24024-2075-25-header
Nov 01 09:44:46 [24023] dbhost03 corosync debug   [QB    ] Free'ing ringbuffer: /dev/shm/qb-cmap-request-24024-2075-25-header
Nov 01 09:44:46 [2075] dbhost03   crm_node:     info: corosync_node_name:   Unable to get node name for nodeid 3
Nov 01 09:44:46 [2075] dbhost03   crm_node:   notice: get_node_name:    Defaulting to uname -n for the local corosync node name
Nov 01 09:44:46 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:46 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:46 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:46 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:46 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:46 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:46 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:46 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:46 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:46 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:46 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:46 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:46 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:46 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:46 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:46 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:46 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:46 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:46 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:46 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:46 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:46 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:46 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:46 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:46 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:46 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:46 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:46 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:46 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:46 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:46 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:46 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:46 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:46 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:47 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:47 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:47 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:47 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:47 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:47 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:47 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:47 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:47 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:47 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:47 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:47 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:47 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:47 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:47 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:47 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:47 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:47 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:47 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:47 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:47 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:47 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:47 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:47 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:47 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:47 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:47 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:47 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:47 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:47 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:47 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:47 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:47 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:47 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:47 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:47 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:47 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:47 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:47 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:47 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:48 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:48 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:48 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:48 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:48 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:48 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:48 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:48 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:48 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:48 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:48 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:48 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:48 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:48 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:48 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:48 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:48 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:48 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:48 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:48 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:48 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:48 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:48 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:48 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:48 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:48 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:48 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:48 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:48 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:48 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:48 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:48 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:48 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:48 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:48 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:48 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:48 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:48 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:48 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:48 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:49 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:49 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:49 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:49 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:49 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:49 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:49 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:49 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:49 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:49 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:49 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:49 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:49 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:49 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:49 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:49 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:49 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:49 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:49 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:49 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:49 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:49 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:49 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:49 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:49 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:49 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:49 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:49 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:49 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:49 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:49 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:49 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:49 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:49 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:49 [24023] dbhost03 corosync debug   [TOTEM ] entering GATHER state from 0(consensus timeout).
Nov 01 09:44:49 [24023] dbhost03 corosync debug   [TOTEM ] Creating commit token because I am the rep.
Nov 01 09:44:49 [24023] dbhost03 corosync debug   [TOTEM ] Saving state aru 63b high seq received 63b
Nov 01 09:44:49 [24023] dbhost03 corosync debug   [TOTEM ] Storing new sequence id for ring 7f160
Nov 01 09:44:49 [24023] dbhost03 corosync debug   [TOTEM ] entering COMMIT state.
Nov 01 09:44:49 [24023] dbhost03 corosync debug   [TOTEM ] got commit token
Nov 01 09:44:49 [24023] dbhost03 corosync debug   [TOTEM ] entering RECOVERY state.
Nov 01 09:44:49 [24023] dbhost03 corosync debug   [TOTEM ] TRANS [0] member 10.37.20.196:
Nov 01 09:44:49 [24023] dbhost03 corosync debug   [TOTEM ] position [0] member 10.37.20.196:
Nov 01 09:44:49 [24023] dbhost03 corosync debug   [TOTEM ] previous ring seq 7f15c rep 10.37.20.193
Nov 01 09:44:49 [24023] dbhost03 corosync debug   [TOTEM ] aru 63b high delivered 63b received flag 1
Nov 01 09:44:49 [24023] dbhost03 corosync debug   [TOTEM ] Did not need to originate any messages in recovery.
Nov 01 09:44:49 [24023] dbhost03 corosync debug   [TOTEM ] got commit token
Nov 01 09:44:49 [24023] dbhost03 corosync debug   [TOTEM ] Sending initial ORF token
Nov 01 09:44:49 [24023] dbhost03 corosync debug   [TOTEM ] token retrans flag is 0 my set retrans flag0 retrans queue empty 1 count 0, aru 0
Nov 01 09:44:49 [24023] dbhost03 corosync debug   [TOTEM ] install seq 0 aru 0 high seq received 0
Nov 01 09:44:49 [24023] dbhost03 corosync debug   [TOTEM ] token retrans flag is 0 my set retrans flag0 retrans queue empty 1 count 1, aru 0
Nov 01 09:44:49 [24023] dbhost03 corosync debug   [TOTEM ] install seq 0 aru 0 high seq received 0
Nov 01 09:44:49 [24023] dbhost03 corosync debug   [TOTEM ] token retrans flag is 0 my set retrans flag0 retrans queue empty 1 count 2, aru 0
Nov 01 09:44:49 [24023] dbhost03 corosync debug   [TOTEM ] install seq 0 aru 0 high seq received 0
Nov 01 09:44:49 [24023] dbhost03 corosync debug   [TOTEM ] token retrans flag is 0 my set retrans flag0 retrans queue empty 1 count 3, aru 0
Nov 01 09:44:49 [24023] dbhost03 corosync debug   [TOTEM ] install seq 0 aru 0 high seq received 0
Nov 01 09:44:49 [24023] dbhost03 corosync debug   [TOTEM ] retrans flag count 4 token aru 0 install seq 0 aru 0 0
Nov 01 09:44:49 [24023] dbhost03 corosync debug   [TOTEM ] Resetting old ring state
Nov 01 09:44:49 [24023] dbhost03 corosync debug   [TOTEM ] recovery to regular 1-0
Nov 01 09:44:49 [24023] dbhost03 corosync debug   [MAIN  ] Member left: r(0) ip(10.37.20.193) 
Nov 01 09:44:49 [24023] dbhost03 corosync debug   [MAIN  ] Member left: r(0) ip(10.37.20.195) 
Nov 01 09:44:49 [24023] dbhost03 corosync debug   [TOTEM ] waiting_trans_ack changed to 1
Nov 01 09:44:49 [24023] dbhost03 corosync debug   [TOTEM ] entering OPERATIONAL state.
Nov 01 09:44:49 [24023] dbhost03 corosync notice  [TOTEM ] A new membership (10.37.20.196:520544) was formed. Members left: 1 2
Nov 01 09:44:49 [24023] dbhost03 corosync debug   [SYNC  ] Committing synchronization for corosync configuration map access
Nov 01 09:44:49 [24023] dbhost03 corosync debug   [CMAP  ] Not first sync -> no action
Nov 01 09:44:49 [24023] dbhost03 corosync debug   [CPG   ] comparing: sender r(0) ip(10.37.20.196) ; members(old:3 left:2)
Nov 01 09:44:49 [24023] dbhost03 corosync debug   [CPG   ] chosen downlist: sender r(0) ip(10.37.20.196) ; members(old:3 left:2)
Nov 01 09:44:49 [24023] dbhost03 corosync debug   [CPG   ] left_list_entries:2
Nov 01 09:44:49 [24023] dbhost03 corosync debug   [CPG   ] left_list[0] group:attrd\x00, ip:r(0) ip(10.37.20.193) , pid:1792
Nov 01 09:44:49 [24023] dbhost03 corosync debug   [CPG   ] left_list[1] group:attrd\x00, ip:r(0) ip(10.37.20.195) , pid:435
Nov 01 09:44:49 [24157] dbhost03      attrd:     info: crm_update_peer_proc:    pcmk_cpg_membership: Node dbhost01[1] - corosync-cpg is now offline
Nov 01 09:44:49 [24157] dbhost03      attrd:     info: crm_update_peer_proc:    pcmk_cpg_membership: Node dbhost02[2] - corosync-cpg is now offline
Nov 01 09:44:49 [24023] dbhost03 corosync debug   [CPG   ] left_list_entries:2
Nov 01 09:44:49 [24023] dbhost03 corosync debug   [CPG   ] left_list[0] group:cib\x00, ip:r(0) ip(10.37.20.193) , pid:1789
Nov 01 09:44:49 [24023] dbhost03 corosync debug   [CPG   ] left_list[1] group:cib\x00, ip:r(0) ip(10.37.20.195) , pid:432
Nov 01 09:44:49 [24154] dbhost03        cib:     info: crm_update_peer_proc:    pcmk_cpg_membership: Node dbhost01[1] - corosync-cpg is now offline
Nov 01 09:44:49 [24154] dbhost03        cib:     info: crm_update_peer_proc:    pcmk_cpg_membership: Node dbhost02[2] - corosync-cpg is now offline
Nov 01 09:44:49 [24023] dbhost03 corosync debug   [CPG   ] left_list_entries:2
Nov 01 09:44:49 [24023] dbhost03 corosync debug   [CPG   ] left_list[0] group:crmd\x00, ip:r(0) ip(10.37.20.193) , pid:1794
Nov 01 09:44:49 [24023] dbhost03 corosync debug   [CPG   ] left_list[1] group:crmd\x00, ip:r(0) ip(10.37.20.195) , pid:437
Nov 01 09:44:49 [24159] dbhost03       crmd:     info: crm_update_peer_proc:    pcmk_cpg_membership: Node dbhost01[1] - corosync-cpg is now offline
Nov 01 09:44:49 [24159] dbhost03       crmd:     info: crm_update_peer_proc:    pcmk_cpg_membership: Node dbhost02[2] - corosync-cpg is now offline
Nov 01 09:44:49 [24023] dbhost03 corosync debug   [CPG   ] left_list_entries:2
Nov 01 09:44:49 [24023] dbhost03 corosync debug   [CPG   ] left_list[0] group:pacemakerd\x00, ip:r(0) ip(10.37.20.193) , pid:1787
Nov 01 09:44:49 [24023] dbhost03 corosync debug   [CPG   ] left_list[1] group:pacemakerd\x00, ip:r(0) ip(10.37.20.195) , pid:430
Nov 01 09:44:49 [24152] dbhost03 pacemakerd:     info: crm_update_peer_proc:    pcmk_cpg_membership: Node dbhost01[1] - corosync-cpg is now offline
Nov 01 09:44:49 [24152] dbhost03 pacemakerd:     info: crm_update_peer_proc:    pcmk_cpg_membership: Node dbhost02[2] - corosync-cpg is now offline
Nov 01 09:44:49 [24023] dbhost03 corosync debug   [CPG   ] left_list_entries:2
Nov 01 09:44:49 [24023] dbhost03 corosync debug   [CPG   ] left_list[0] group:stonith-ng\x00, ip:r(0) ip(10.37.20.193) , pid:1790
Nov 01 09:44:49 [24023] dbhost03 corosync debug   [CPG   ] left_list[1] group:stonith-ng\x00, ip:r(0) ip(10.37.20.195) , pid:433
Nov 01 09:44:49 [24155] dbhost03 stonith-ng:     info: crm_update_peer_proc:    pcmk_cpg_membership: Node dbhost01[1] - corosync-cpg is now offline
Nov 01 09:44:49 [24155] dbhost03 stonith-ng:     info: crm_update_peer_proc:    pcmk_cpg_membership: Node dbhost02[2] - corosync-cpg is now offline
Nov 01 09:44:49 [24023] dbhost03 corosync debug   [CPG   ] got joinlist message from node 3
Nov 01 09:44:49 [24023] dbhost03 corosync debug   [SYNC  ] Committing synchronization for corosync cluster closed process group service v1.01
Nov 01 09:44:49 [24023] dbhost03 corosync debug   [CPG   ] joinlist_messages[0] group:crmd\x00, ip:r(0) ip(10.37.20.196) , pid:24159
Nov 01 09:44:49 [24023] dbhost03 corosync debug   [CPG   ] joinlist_messages[1] group:attrd\x00, ip:r(0) ip(10.37.20.196) , pid:24157
Nov 01 09:44:49 [24023] dbhost03 corosync debug   [CPG   ] joinlist_messages[2] group:stonith-ng\x00, ip:r(0) ip(10.37.20.196) , pid:24155
Nov 01 09:44:49 [24023] dbhost03 corosync debug   [CPG   ] joinlist_messages[3] group:cib\x00, ip:r(0) ip(10.37.20.196) , pid:24154
Nov 01 09:44:49 [24023] dbhost03 corosync debug   [CPG   ] joinlist_messages[4] group:pacemakerd\x00, ip:r(0) ip(10.37.20.196) , pid:24152
Nov 01 09:44:49 [24023] dbhost03 corosync debug   [VOTEQ ] flags: quorate: Yes Leaving: No WFA Status: No First: No Qdevice: No QdeviceAlive: No QdeviceCastVote: No QdeviceMasterWins: No
Nov 01 09:44:49 [24023] dbhost03 corosync debug   [VOTEQ ] got nodeinfo message from cluster node 3
Nov 01 09:44:49 [24023] dbhost03 corosync debug   [VOTEQ ] nodeinfo message[3]: votes: 1, expected: 3 flags: 1
Nov 01 09:44:49 [24023] dbhost03 corosync debug   [VOTEQ ] flags: quorate: Yes Leaving: No WFA Status: No First: No Qdevice: No QdeviceAlive: No QdeviceCastVote: No QdeviceMasterWins: No
Nov 01 09:44:49 [24023] dbhost03 corosync debug   [VOTEQ ] total_votes=1, expected_votes=3
Nov 01 09:44:49 [24023] dbhost03 corosync debug   [VOTEQ ] node 1 state=2, votes=1, expected=3
Nov 01 09:44:49 [24023] dbhost03 corosync debug   [VOTEQ ] node 2 state=2, votes=1, expected=3
Nov 01 09:44:49 [24023] dbhost03 corosync debug   [VOTEQ ] node 3 state=1, votes=1, expected=3
Nov 01 09:44:49 [24023] dbhost03 corosync debug   [VOTEQ ] quorum lost, blocking activity
Nov 01 09:44:49 [24023] dbhost03 corosync debug   [VOTEQ ] got nodeinfo message from cluster node 3
Nov 01 09:44:49 [24023] dbhost03 corosync debug   [VOTEQ ] nodeinfo message[0]: votes: 0, expected: 0 flags: 0
Nov 01 09:44:49 [24023] dbhost03 corosync debug   [SYNC  ] Committing synchronization for corosync vote quorum service v1.0
Nov 01 09:44:49 [24023] dbhost03 corosync debug   [VOTEQ ] total_votes=1, expected_votes=3
Nov 01 09:44:49 [24023] dbhost03 corosync debug   [VOTEQ ] node 1 state=2, votes=1, expected=3
Nov 01 09:44:49 [24023] dbhost03 corosync debug   [VOTEQ ] node 2 state=2, votes=1, expected=3
Nov 01 09:44:49 [24023] dbhost03 corosync debug   [VOTEQ ] node 3 state=1, votes=1, expected=3
Nov 01 09:44:49 [24023] dbhost03 corosync notice  [QUORUM] This node is within the non-primary component and will NOT provide any services.
Nov 01 09:44:49 [24023] dbhost03 corosync notice  [QUORUM] Members[1]: 3
Nov 01 09:44:49 [24023] dbhost03 corosync debug   [QUORUM] sending quorum notification to (nil), length = 52
Nov 01 09:44:49 [24023] dbhost03 corosync notice  [MAIN  ] Completed service synchronization, ready to provide service.
Nov 01 09:44:49 [24023] dbhost03 corosync debug   [TOTEM ] waiting_trans_ack changed to 0
Nov 01 09:44:50 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:50 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:50 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:50 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:50 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:50 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:50 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:50 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:50 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)
Nov 01 09:44:50 [24023] dbhost03 corosync debug   [TOTEM ] sendmsg(mcast) failed (non-critical): Operation not permitted (1)