Mysql Bind on TCP/IP port异常处理一则

                                  Mysql Bind on TCP/IP port异常处理一则

       某日手动杀掉mysqld服务器进程以后，再次通过mysqld_safe启动mysql服务器，无法启动了，直程直接结束.

[root@localhost ~]# /bin/sh /usr/local/mysql/bin/mysqld_safe --datadir=/usr/local/mysql/data
141104 15:00:27 mysqld_safe Logging to '/usr/local/mysql/data/localhost.localdomain.err'.
141104 15:00:27 mysqld_safe Starting mysqld daemon with databases from /usr/local/mysql/data
141104 15:00:30 mysqld_safe mysqld from pid file /usr/local/mysql/data/localhost.localdomain.pid ended
[root@localhost ~]#

打开/usr/local/mysql/data/localhost.localdomain.err, 看到以下信息

2014-11-04 15:00:28 3038 [Note] Server hostname (bind-address): '*'; port: 3307
2014-11-04 15:00:28 3038 [Note] IPv6 is available.
2014-11-04 15:00:28 3038 [Note]   - '::' resolves to '::';
2014-11-04 15:00:28 3038 [Note] Server socket created on IP: '::'.
2014-11-04 15:00:28 3038 [ERROR] Can't start server: Bind on TCP/IP port: Address already in use
2014-11-04 15:00:28 3038 [ERROR] Do you already have another mysqld server running on port: 3307 ?
2014-11-04 15:00:28 3038 [ERROR] Aborting

根据提示信息来看，是端口被占用了.但是服务器已经没有其他mysql实例在跑了.通过查看进程和telnet可以排除这一点

[root@localhost 14394]# ps -eaf | grep mysqld
root      2636 21952  0 14:59 pts/3    00:00:00 grep mysqld

[root@localhost 14394]# telnet 127.0.0.1 3307
Trying 127.0.0.1...
telnet: connect to address 127.0.0.1: Connection refused

非常奇怪了，3307端口明明是没有被使用的，但是为什么mysql服务器还是报端口被占用的问题呢.难道是socket和
pid文件没有删除？手动删除这两个文件后,再次启动,问题依旧.

  排除iptables和selineux,发现都是停掉了.

[root@localhost 14394]# service iptables status
Firewall is stopped.
[root@localhost 14394]# getenforce
Disabled

通过netstat -nltp活动连接，也没有看到有异常的信息

[root@localhost 14394]# netstat -ntlp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address               Foreign Address             State       PID/Program name  
tcp        0      0 127.0.0.1:2208              0.0.0.0:*                   LISTEN      2241/hpiod         
tcp        0      0 0.0.0.0:10051               0.0.0.0:*                   LISTEN      30674/zabbix_server
tcp        0      0 127.0.0.1:2207              0.0.0.0:*                   LISTEN      2246/python        
tcp        0      0 :::3872                     :::*                        LISTEN      2771/java          
tcp        0      0 ::ffff:127.0.0.1:8005       :::*                        LISTEN      3040/java          
tcp        0      0 :::8009                     :::*                        LISTEN      3040/java          
tcp        0      0 :::80                       :::*                        LISTEN      1883/httpd         
tcp        0      0 :::8080                     :::*                        LISTEN      3040/java          
tcp        0      0 :::1521                     :::*                        LISTEN      18549/tnslsnr      
tcp        0      0 :::22                       :::*                        LISTEN      2264/sshd          
tcp        0      0 :::5501                     :::*                        LISTEN      18549/tnslsnr      
tcp        0      0 :::6878                     :::*                        LISTEN      14354/ora_d000_t2  

 

 通过netstat -antup选项，才发现真正的原因.

[root@localhost 14394]# netstat -antup
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address               Foreign Address             State       PID/Program name  
tcp        0      0 127.0.0.1:2208              0.0.0.0:*                   LISTEN      2241/hpiod         
tcp        0      0 0.0.0.0:10051               0.0.0.0:*                   LISTEN      30674/zabbix_server
tcp        0      0 127.0.0.1:2207              0.0.0.0:*                   LISTEN      2246/python        
tcp        0      0 127.0.0.1:49319             127.0.0.1:1521              ESTABLISHED 14346/ora_lreg_t2  
tcp        0      0 127.0.0.1:3307              127.0.0.1:3307              ESTABLISHED 30710/zabbix_server
tcp        0      0 :::3872                     :::*                        LISTEN      2771/java          
tcp        0      0 ::ffff:127.0.0.1:8005       :::*                        LISTEN      3040/java          
tcp        0      0 :::8009                     :::*                        LISTEN      3040/java          
tcp        0      0 :::80                       :::*                        LISTEN      1883/httpd         
tcp        0      0 :::8080                     :::*                        LISTEN      3040/java          
tcp        0      0 :::1521                     :::*                        LISTEN      18549/tnslsnr      
tcp        0      0 :::22                       :::*                        LISTEN      2264/sshd          
tcp        0      0 :::5501                     :::*                        LISTEN      18549/tnslsnr      
tcp        0      0 :::6878                     :::*                        LISTEN      14354/ora_d000_t2  
tcp        0      0 ::ffff:172.28.10.145:1521   ::ffff:172.28.12.23:51408   ESTABLISHED 9727/oraclet2      
tcp        0    372 ::ffff:172.28.10.145:22     ::ffff:172.28.12.23:50160   ESTABLISHED 17292/sshd: oracle 
tcp        0      0 ::ffff:172.28.10.145:1521   ::ffff:172.28.12.23:51443   ESTABLISHED 9742/oraclet2      
tcp        0      0 ::ffff:172.28.10.145:1521   ::ffff:172.28.10.145:37264  TIME_WAIT   -                  
tcp        0      0 ::ffff:127.0.0.1:1521       ::ffff:127.0.0.1:49319      ESTABLISHED 18549/tnslsnr      
tcp        0      0 ::ffff:172.28.10.145:1521   ::ffff:172.28.10.145:37288  TIME_WAIT   -                  
tcp        0      0 ::ffff:172.28.10.145:1521   ::ffff:172.28.10.145:37312  TIME_WAIT   -                  
udp        0      0 ::1:47116                   :::*                                    14356/ora_s000_t2  
udp        0      0 ::1:43933                   :::*                                    14354/ora_d000_t2  
udp        0      0 ::1:61855                   :::*                                    14346/ora_lreg_t2  

由于zabbix服务器在连接一直没有正常release,导致mysqld服务器在启动时调用bind函数一直不成功，返回EADDRINUSE错误

 EADDRINUSE
              (Internet domain sockets) The port number was specified as
              zero in the socket address structure, but, upon attempting to
              bind to an ephemeral port, it was determined that all port
              numbers in the ephemeral port range are currently in use

手动再kill掉zabbix_server服务器进程，稍等几分钟，等待连接自动被release掉后，再次在3307的端口上启动mysqld服务器进程成功.