重启ambari-agent服务器报错： Failed to start ping port listener of

维护过程中重启ambari-agent服务器 报错：

INFO 2016-03-08 13:04:24,070 main.py:74 - loglevel=logging.INFO
INFO 2016-03-08 13:04:24,076 DataCleaner.py:39 - Data cleanup thread started
INFO 2016-03-08 13:04:24,081 DataCleaner.py:120 - Data cleanup started
INFO 2016-03-08 13:04:24,087 DataCleaner.py:122 - Data cleanup finished
ERROR 2016-03-08 13:04:24,213 main.py:272 - Failed to start ping port listener of: [Errno 98] Address already in use
ERROR 2016-03-08 13:04:24,213 main.py:272 - Failed to start ping port listener of: [Errno 98] Address already in use
INFO 2016-03-08 13:04:24,214 PingPortListener.py:61 - Ping port listener killed
INFO 2016-03-08 13:04:24,214 ExitHelper.py:53 - Performing cleanup before exiting...
INFO 2016-03-08 13:07:26,875 main.py:74 - loglevel=logging.INFO
INFO 2016-03-08 13:07:26,875 main.py:74 - loglevel=logging.INFO
INFO 2016-03-08 13:07:26,877 DataCleaner.py:39 - Data cleanup thread started
INFO 2016-03-08 13:07:26,882 DataCleaner.py:120 - Data cleanup started
INFO 2016-03-08 13:07:26,884 DataCleaner.py:122 - Data cleanup finished
ERROR 2016-03-08 13:07:26,948 main.py:272 - Failed to start ping port listener of: [Errno 98] Address already in use
ERROR 2016-03-08 13:07:26,948 main.py:272 - Failed to start ping port listener of: [Errno 98] Address already in use
INFO 2016-03-08 13:07:26,948 PingPortListener.py:61 - Ping port listener killed
INFO 2016-03-08 13:07:26,949 ExitHelper.py:53 - Performing cleanup before exiting...
INFO 2016-03-08 13:08:54,429 main.py:74 - loglevel=logging.INFO
INFO 2016-03-08 13:08:54,429 main.py:74 - loglevel=logging.INFO
INFO 2016-03-08 13:08:54,431 DataCleaner.py:39 - Data cleanup thread started
INFO 2016-03-08 13:08:54,434 DataCleaner.py:120 - Data cleanup started
INFO 2016-03-08 13:08:54,437 DataCleaner.py:122 - Data cleanup finished
ERROR 2016-03-08 13:08:54,705 main.py:272 - Failed to start ping port listener of: [Errno 98] Address already in use
ERROR 2016-03-08 13:08:54,705 main.py:272 - Failed to start ping port listener of: [Errno 98] Address already in use
INFO 2016-03-08 13:08:54,705 PingPortListener.py:61 - Ping port listener killed
ng INFO 2016-03-08 13:08:54,706 ExitHelper.py:53 - Performing cleanup before exiting...

某个端口或者进程一直陪占用

解决方法：
发现df命令一直执行没有完成，

[root@testserver1 ~]# netstat -lanp|grep 8670
tcp        0      0 0.0.0.0:8670                0.0.0.0:*                   LISTEN      4771/df

[root@testserver1 ~]# kill -9 4771
kill后发现过一会又占用了，手动执行了df，发现现实不了，后来发现使用了nfs挂载其他服务器，nfs server服务器关机了，导致了这个问题。强制umount后恢复正常。

[root@testserver1 ~]# service ambari-agent restart
Verifying Python version compatibility...
Using python  /usr/bin/python2.6
ambari-agent is not running. No PID found at /var/run/ambari-agent/ambari-agent.pid
Verifying Python version compatibility...
Using python  /usr/bin/python2.6
Checking for previously running Ambari Agent...
Starting ambari-agent
Verifying ambari-agent process status...
Ambari Agent successfully started
Agent PID at: /var/run/ambari-agent/ambari-agent.pid
Agent out at: /var/log/ambari-agent/ambari-agent.out
Agent log at: /var/log/ambari-agent/ambari-agent.log

参考：