起因是这样的：笔者所在的公司通过keepalived来实现openstack中的L3 router的主备。原来期望在master down的时候，backup router能够迅速
接替master成为网关。可是在实际测试的时候发现并没有像期望的那样工作。
首先，环境中已经有一个router是master；
然后，再加一个router，期望成为backup；
最后，将原来的master干掉，期望原来的backup立即成为master；
期望的是这中间的断网时间很短，最多不超过3*keepalived adv_interval（环境中配置的是2秒），但是实际却断网了将近2分钟。
用tcpdump抓了下报文，发现master down的时候，priority为0的报文已经发出来了，backup那边的接口上也已经收到了，理论上应该立即切换为master。
但是没有。查了一下keepalived日志，发现我们期望的backup此时还处于init状态，直到差不多两分钟后才转为backup。
Aug 28 15:06:42 ca59 Keepalived_vrrp[16197]: Registering Kernel netlink command channel 这一步开始已经在init状态
Aug 28 15:06:42 ca59 Keepalived_vrrp[16197]: Registering gratuitous ARP shared channel
Aug 28 15:08:34 ca59 Keepalived_vrrp[16197]: Opening file '/data/neutron/ha_confs/5e6c047d-05f2-412b-86bc-7fdf8590b456/keepalived.conf'. 过了两分钟，还在init状态
Aug 28 15:08:34 ca59 Keepalived_vrrp[16197]: Configuration is using : 64147 Bytes
Aug 28 15:08:34 ca59 Keepalived_vrrp[16197]: -----< Global definitions >-----
Aug 28 15:08:34 ca59 Keepalived_vrrp[16197]: Smtp server connection timeout = 30
Aug 28 15:08:34 ca59 Keepalived_vrrp[16197]: VRRP IPv4 mcast group = 224.0.0.18
Aug 28 15:08:34 ca59 Keepalived_vrrp[16197]: VRRP IPv6 mcast group = 224.0.0.18
Aug 28 15:08:34 ca59 Keepalived_vrrp[16197]: SNMP Trap disabled
Aug 28 15:08:34 ca59 Keepalived_vrrp[16197]: -----< VRRP Topology >-----
Aug 28 15:08:34 ca59 Keepalived_vrrp[16197]: VRRP Instance = VR_1
Aug 28 15:08:34 ca59 Keepalived_vrrp[16197]: Want State = BACKUP
Aug 28 15:08:34 ca59 Keepalived_vrrp[16197]: Runing on device = ha-c16999d4-3e
Aug 28 15:08:34 ca59 Keepalived_vrrp[16197]: Virtual Router ID = 1
Aug 28 15:08:34 ca59 Keepalived_vrrp[16197]: Priority = 154
Aug 28 15:08:34 ca59 Keepalived_vrrp[16197]: Advert interval = 2sec
Aug 28 15:08:34 ca59 Keepalived_vrrp[16197]: Preempt disabled
Aug 28 15:08:34 ca59 Keepalived_vrrp[16197]: Tracked interfaces = 1
Aug 28 15:08:34 ca59 Keepalived_vrrp[16197]: ha-c16999d4-3e weight 0
Aug 28 15:08:34 ca59 Keepalived_vrrp[16197]: Virtual IP = 1
Aug 28 15:08:34 ca59 Keepalived_vrrp[16197]: 10.164.96.1/20 dev ha-c16999d4-3e scope global
Aug 28 15:08:34 ca59 Keepalived_vrrp[16197]: -----< VRRP Sync groups >-----
Aug 28 15:08:34 ca59 Keepalived_vrrp[16197]: VRRP Sync Group = VG_1, BACKUP
Aug 28 15:08:34 ca59 Keepalived_vrrp[16197]: monitor = VR_1
Aug 28 15:08:34 ca59 Keepalived_vrrp[16197]: Generic state transition script = '/data/neutron/ha_confs/5e6c047d-05f2-412b-86bc-7fdf8590b456/notify.sh'
Aug 28 15:08:34 ca59 Keepalived_vrrp[16197]: Using LinkWatch kernel netlink reflector...
Aug 28 15:08:34 ca59 Keepalived_vrrp[16197]: VRRP_Instance(VR_1) Entering BACKUP STATE 终于到了backup状态
Aug 28 15:08:37 ca59 Keepalived_vrrp[16197]: VRRP_Instance(VR_，1) Transition to MASTER STATE
这个就奇怪了，从vrrp和keepalived协议来看，从init状态到backup状态都应该是瞬间切换
看了一遍keepalived的代码，唯一怀疑的就是这个地方：
start_vrrp()-->alloc_global_data()-->set_default_values()-->set_default_router_id()-->get_local_name():
该函数里有个gethostbyname系统调用，start_vrrp()接口的其他调用都是普通的内存分配以及设置参数。
用strace跟踪了一下keepalived进程，果不其然
socket(PF_INET, SOCK_DGRAM|SOCK_NONBLOCK, IPPROTO_IP) = 10
connect(10, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("10.192.3.241")}, 16) = 0 53端口是DNS专用口
poll([{fd=10, events=POLLOUT}], 1, 0) = 1 ([{fd=10, revents=POLLOUT}])
sendto(10, "\372\253\1\0\0\1\0\0\0\0\0\0\4ca60\5cnhz1\003111\3org\0"..., 36, MSG_NOSIGNAL, NULL, 0) = 36
poll([{fd=10, events=POLLIN}], 1, 5000) = 0 (Timeout)
socket(PF_INET, SOCK_DGRAM|SOCK_NONBLOCK, IPPROTO_IP) = 11
connect(11, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("10.192.3.242")}, 16) = 0 尝试不同的nameserver
poll([{fd=11, events=POLLOUT}], 1, 0) = 1 ([{fd=11, revents=POLLOUT}])
sendto(11, "\372\253\1\0\0\1\0\0\0\0\0\0\4ca60\5cnhz1\003111\3org\0"..., 36, MSG_NOSIGNAL, NULL, 0) = 36
poll([{fd=11, events=POLLIN}], 1, 3000) = 0 (Timeout)
socket(PF_INET, SOCK_DGRAM|SOCK_NONBLOCK, IPPROTO_IP) = 12
connect(12, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("172.16.0.2")}, 16) = 0
poll([{fd=12, events=POLLOUT}], 1, 0) = 1 ([{fd=12, revents=POLLOUT}])
sendto(12, "\372\253\1\0\0\1\0\0\0\0\0\0\4ca60\5cnhz1\003111\3org\0"..., 36, MSG_NOSIGNAL, NULL, 0) = 36
poll([{fd=12, events=POLLIN}], 1, 6000) = 0 (Timeout)
poll([{fd=10, events=POLLOUT}], 1, 0) = 1 ([{fd=10, revents=POLLOUT}])
sendto(10, "\372\253\1\0\0\1\0\0\0\0\0\0\4ca60\5cnhz1\003111\3org\0"..., 36, MSG_NOSIGNAL, NULL, 0) = 36
poll([{fd=10, events=POLLIN}], 1, 5000) = 0 (Timeout)
poll([{fd=11, events=POLLOUT}], 1, 0) = 1 ([{fd=11, revents=POLLOUT}])
sendto(11, "\372\253\1\0\0\1\0\0\0\0\0\0\4ca60\5cnhz1\003111\3org\0"..., 36, MSG_NOSIGNAL, NULL, 0) = 36
poll([{fd=11, events=POLLIN}], 1, 3000) = 0 (Timeout)
poll([{fd=12, events=POLLOUT}], 1, 0) = 1 ([{fd=12, revents=POLLOUT}])
sendto(12, "\372\253\1\0\0\1\0\0\0\0\0\0\4ca60\5cnhz1\003111\3org\0"..., 36, MSG_NOSIGNAL, NULL, 0) = 36
poll([{fd=12, events=POLLIN}], 1, 6000) = 0 (Timeout)
从这个过程来看，keepalived启动的时候一直在尝试去解析ca60这个主机，domain name为cnhz1.111.org，
不停的去尝试3个nameserver：
10.192.3.241， 10.192.3.242， 172.16.0.2，但是每次都超时。所有的都尝试完后，终于退出到达backup状态。
再反过来看一下keepalived代码：
char *
get_local_name(void)
{
struct hostent *host;
struct utsname name;
if (uname(&name) < 0)
return NULL;
if (!(host = gethostbyname(name.nodename)))
return NULL;
return host->h_name;
}
单独将这段代码拎出来放到有问题的主机上执行，发现在namespace外面，执行很快；而在namespace里面，执行很慢，用strace看到了和上面一样的日志
再看一下namespace里面的路由表，发现没有到达10.192.3.241， 10.192.3.242， 172.16.0.2它们的路由。
那问题基本就确定了，应该是hosts文件里没有配置对应的主机名映射，而通过DNS又无法连接到对应的nameserver。
打开文件/etc/hosts文件和/etc/resolv.conf文件看了一下，果然如此。在/etc/hosts里面加了下ca60和其ip地址的映射，问题解决。
看了一下/etc/resolve.conf的manual文档，其实这里有很多参数是可以调的，比如：
timeout:n
sets the amount of time the resolver will wait for a response from a remote name server before retrying the query via
a different name server. Measured in seconds, the default is RES_TIMEOUT (currently 5, see ). The value
for this option is silently capped to 30.
attempts:n
sets the number of times the resolver will send a query to its name servers before giving up and returning an error to
the calling application. The default is RES_DFLRETRY (currently 2, see ). The value for this option is
silently capped to 5
在发生上述情况的时候，我们也可以通过调整options timeout:1 attempts:2来降低超时时间。
同样的另外两个例子来，说明DNS对于网络延时确实有不小的影响。
问题1：打开网页的时候很慢，ping 域名显示延时1000多毫秒，而直接ping ip地址则只有2个多毫秒。
查了一下就是resolve.conf配置了两个nameserver，但是第一个nameserver已经挂了，只有第2个nameserver是好的，
那么就需要等连接第一个nameserver超时后到第2个namserver才能域名解析成功。
问题2：tcpdump抓报文的时候，有时发现报文久久都不打印，加一个-n参数就好了，也是域名解析的问题在作怪。
阅读(3429) | 评论(0) | 转发(0) |
