apache的一个问题:Address already in use: make_sock:

前些天我们网站在高峰时候无法访问，后台重启apache报错：

 

(98)Address already in use: make_sock: could not bind to address 192.168.3.66:80
no listening sockets available, shutting down
Unable to open logs

 

经过详细的研究测试，发现是操作系统做了用户进程数的限制。测试如下：

 

 

 

正常状态下：max user processes    =          (-u) 20480
不断访问网站查看apache进程的变化：
[root@totoweb ~]# ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
file size               (blocks, -f) unlimited
pending signals                 (-i) 1024
max locked memory       (kbytes, -l) 32
max memory size         (kbytes, -m) unlimited
open files                      (-n) 65536
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 20480
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
[root@totoweb ~]#
[root@totoweb ~]# ps -u apache | wc -l
117
[root@totoweb ~]# ps -ef| grep  httpd | wc -l
118
[root@totoweb ~]# ps -u apache | wc -l
117
[root@totoweb ~]# ps -u apache | wc -l
130
[root@totoweb ~]# ps -u apache | wc -l
129
[root@totoweb ~]# ps -ef| grep  httpd | wc -l
130
[root@totoweb ~]# ps -ef| grep  httpd | wc -l
132
[root@totoweb ~]# ps -ef| grep  httpd | wc -l
162
[root@totoweb ~]# ps -u apache | wc -l
161
[root@totoweb ~]# ps -u apache | wc -l
159
[root@totoweb ~]# ps -ef| grep  httpd | wc -l
159
[root@totoweb ~]# ps -ef| grep  httpd | wc -l
159
[root@totoweb ~]# ps -u apache | wc -l
158
[root@totoweb ~]# ps -u apache | wc -l
157
[root@totoweb ~]# ps -u apache | wc -l
157
[root@totoweb ~]# ps -u apache | wc -l
153

重启apache，在不断访问网站查看apache进程变化：
[root@totoweb ~]# tstop
Using CATALINA_BASE:   /usr/local/tomcat
Using CATALINA_HOME:   /usr/local/tomcat
Using CATALINA_TMPDIR: /usr/local/tomcat/temp
Using JRE_HOME:       /usr/local/jdk1506
[root@totoweb ~]#
[root@totoweb ~]#
[root@totoweb ~]#
[root@totoweb ~]# tstart
Using CATALINA_BASE:   /usr/local/tomcat
Using CATALINA_HOME:   /usr/local/tomcat
Using CATALINA_TMPDIR: /usr/local/tomcat/temp
Using JRE_HOME:       /usr/local/jdk1506
[root@totoweb ~]#
[root@totoweb ~]#
[root@totoweb ~]# ps -u apache | wc -l
127
[root@totoweb ~]# ps -ef| grep  httpd | wc -l
128
[root@totoweb ~]# tstop
Using CATALINA_BASE:   /usr/local/tomcat
Using CATALINA_HOME:   /usr/local/tomcat
Using CATALINA_TMPDIR: /usr/local/tomcat/temp
Using JRE_HOME:       /usr/local/jdk1506
[root@totoweb ~]#
[root@totoweb ~]#
[root@totoweb ~]# ps -ef| grep  httpd | wc -l
1
[root@totoweb ~]# ps -u apache | wc -l
1
[root@totoweb ~]# tstart
Using CATALINA_BASE:   /usr/local/tomcat
Using CATALINA_HOME:   /usr/local/tomcat
Using CATALINA_TMPDIR: /usr/local/tomcat/temp
Using JRE_HOME:       /usr/local/jdk1506
[root@totoweb ~]# ps -u apache | wc -l
36
[root@totoweb ~]# ps -ef| grep  httpd | wc -l
112
[root@totoweb ~]# ps -u apache | wc -l
111
[root@totoweb ~]# ps -u apache | wc -l
111
[root@totoweb ~]# ps -ef| grep  httpd | wc -l
115
[root@totoweb ~]# ps -u apache | wc -l
142

修改limits，不重新登陆，在不断的访问网站，这种的情况下，max user processes没有改变：
[root@totoweb ~]# vi /etc/security/limits.conf

#        - stack - max stack size (KB)
#        - cpu - max CPU time (MIN)
#        - nproc - max number of processes
#        - as - address space limit
#        - maxlogins - max number of logins for this user
#        - priority - the priority to run user process with
#        - locks - max number of file locks the user can hold
#
#                
#

#*               soft    core            0
#*               hard    rss             10000
        hard    nproc           20
#@faculty        soft    nproc           20
#@faculty        hard    nproc           50
#ftp             hard    nproc           0
        -       maxlogins       4
# End of file
*               soft    nproc   50
*               hard    nproc   20480
*               soft    nofile  65536
*               hard    nofile  65536
"/etc/security/limits.conf" [dos] 48L, 1645C 已写入          
[root@totoweb ~]#
[root@totoweb ~]#
[root@totoweb ~]#
[root@totoweb ~]# ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
file size               (blocks, -f) unlimited
pending signals                 (-i) 1024
max locked memory       (kbytes, -l) 32
max memory size         (kbytes, -m) unlimited
open files                      (-n) 65536
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 20480
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
[root@totoweb ~]#
[root@totoweb ~]#
[root@totoweb ~]# ps -u apache | wc -l
122
[root@totoweb ~]# ps -ef| grep  httpd | wc -l
138
[root@totoweb ~]# ps -u apache | wc -l
153
[root@totoweb ~]# ps -u apache | wc -l
152
[root@totoweb ~]#

重启apache，不断访问网站，可以在此证明max user processes参数值对于apache用户没有改变：
[root@totoweb ~]# tstop
Using CATALINA_BASE:   /usr/local/tomcat
Using CATALINA_HOME:   /usr/local/tomcat
Using CATALINA_TMPDIR: /usr/local/tomcat/temp
Using JRE_HOME:       /usr/local/jdk1506
[root@totoweb ~]#
[root@totoweb ~]#
[root@totoweb ~]# tstart
Using CATALINA_BASE:   /usr/local/tomcat
Using CATALINA_HOME:   /usr/local/tomcat
Using CATALINA_TMPDIR: /usr/local/tomcat/temp
Using JRE_HOME:       /usr/local/jdk1506
[root@totoweb ~]#
[root@totoweb ~]#
[root@totoweb ~]# ps -u apache | wc -l
23
[root@totoweb ~]# ps -ef| grep  httpd | wc -l
80
[root@totoweb ~]# ps -u apache | wc -l
111
[root@totoweb ~]# ps -ef| grep  httpd | wc -l
112
[root@totoweb ~]# ps -u apache | wc -l
111
[root@totoweb ~]# ps -ef| grep  httpd | wc -l
112
[root@totoweb ~]# ps -ef| grep  httpd | wc -l
168
[root@totoweb ~]# ps -u apache | wc -l
164
[root@totoweb ~]#

退出，以root身份重新登陆，检查limits，可以看到max user processes已经改变：
[root@totoweb ~]# ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
file size               (blocks, -f) unlimited
pending signals                 (-i) 1024
max locked memory       (kbytes, -l) 32
max memory size         (kbytes, -m) unlimited
open files                      (-n) 65536
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 50
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
[root@totoweb ~]#

检查anache process情况：
[root@totoweb ~]# ps -u apache | wc -l
146
[root@totoweb ~]# ps -ef| grep  httpd | wc -l
147
[root@totoweb ~]# ps -ef| grep  httpd | wc -l
147
[root@totoweb ~]# ps -u apache | wc -l
146
[root@totoweb ~]# ps -u apache | wc -l
146
[root@totoweb ~]# ps -ef| grep  httpd | wc -l
147
[root@totoweb ~]# ps -ef| grep  httpd | wc -l
147
[root@totoweb ~]# ps -u apache | wc -l
146
[root@totoweb ~]# ps -u apache | wc -l
153
[root@totoweb ~]# ps -ef| grep  httpd | wc -l
154
[root@totoweb ~]# ps -ef| grep  httpd | wc -l
151
[root@totoweb ~]# ps -ef| grep  httpd | wc -l
148
[root@totoweb ~]# ps -u apache | wc -l
144
[root@totoweb ~]# ps -u apache | wc -l
130
[root@totoweb ~]# ps -ef| grep  httpd | wc -l
118
[root@totoweb ~]# ps -ef| grep  httpd | wc -l
129
[root@totoweb ~]# ps -ef| grep  httpd | wc -l
129
[root@totoweb ~]# ps -u apache | wc -l
127
[root@totoweb ~]#
[root@totoweb ~]# ps -u root | wc -l
290
[root@totoweb ~]#
[root@totoweb ~]# ps -u root | wc -l
290
[root@totoweb ~]# ps -u root | wc -l
290
[root@totoweb ~]# ps -u apache | wc -l
151
[root@totoweb ~]# ps -u root | wc -l
290
[root@totoweb ~]# ps -u root | wc -l
290
[root@totoweb ~]# ps -u apache | wc -l
可以看出apache进程限制没有受到影响，系统重新登陆，就重新读了一次limits配置文件，系统的maxuser参数限制

生效，但是apache没有重启，他所能用到的进程还没有受这个limits的限制，因此还会大于设置后的新的max user

processes值，但是应该会小于此时的系统的总进程数。而root系统进程不在发生变化，为什么？

重启apache，
[root@totoweb ~]# tstop
Using CATALINA_BASE:   /usr/local/tomcat
Using CATALINA_HOME:   /usr/local/tomcat
Using CATALINA_TMPDIR: /usr/local/tomcat/temp
Using JRE_HOME:       /usr/local/jdk1506
[root@totoweb ~]#
[root@totoweb ~]#
[root@totoweb ~]#
[root@totoweb ~]# tstart
Using CATALINA_BASE:   /usr/local/tomcat
Using CATALINA_HOME:   /usr/local/tomcat
Using CATALINA_TMPDIR: /usr/local/tomcat/temp
Using JRE_HOME:       /usr/local/jdk1506
[root@totoweb ~]#
[root@totoweb ~]# ps -u root | wc -l
289
[root@totoweb ~]# ps -ef| grep  httpd | wc -l
51
[root@totoweb ~]# ps -u apache | wc -l
51
[root@totoweb ~]# ps -u apache | wc -l
51
[root@totoweb ~]# ps -ef| grep  httpd | wc -l
51
[root@totoweb ~]# ps -u apache | wc -l
51
[root@totoweb ~]# ps -ef| grep  httpd | wc -l
51
[root@totoweb ~]# ps -u apache | wc -l
51
[root@totoweb ~]# ps -ef| grep  httpd | wc -l
51
[root@totoweb ~]#
[root@totoweb ~]# ps -u root | wc -l
289
[root@totoweb ~]# ps -u root | wc -l
289
[root@totoweb ~]#
我们看到apache进程已经被限制，这时已经无法访问网站，

再次重启apache，报错：
[root@totoweb ~]#
[root@totoweb ~]# tstop
Using CATALINA_BASE:   /usr/local/tomcat
Using CATALINA_HOME:   /usr/local/tomcat
Using CATALINA_TMPDIR: /usr/local/tomcat/temp
Using JRE_HOME:       /usr/local/jdk1506
httpd (pid 17178?) not running
[root@totoweb ~]#
[root@totoweb ~]#
[root@totoweb ~]# tstart
Using CATALINA_BASE:   /usr/local/tomcat
Using CATALINA_HOME:   /usr/local/tomcat
Using CATALINA_TMPDIR: /usr/local/tomcat/temp
Using JRE_HOME:       /usr/local/jdk1506
(98)Address already in use: make_sock: could not bind to address 192.168.3.66:80
no listening sockets available, shutting down
Unable to open logs
[root@totoweb ~]#
这个时候，系统分配给apache用户的进程已经达到最大，apache没有更多的进程去做做重启操作，这就是出错的原因

；

 

此时杀死apache用户所有http进程，就会可以成功启动apache：
[root@totoweb ~]# kill -9 $(ps -ef | grep httpd | awk '{print $2}')
-bash: kill: (18010) - 没有那个进程
[root@totoweb ~]#
[root@totoweb ~]# ps -ef| grep  httpd | wc -l
1
[root@totoweb ~]# tstart
Using CATALINA_BASE:   /usr/local/tomcat
Using CATALINA_HOME:   /usr/local/tomcat
Using CATALINA_TMPDIR: /usr/local/tomcat/temp
Using JRE_HOME:       /usr/local/jdk1506
[root@totoweb ~]#

 

但是如果网站访问访问量继续增大的话，到达max user processes限制值，还会出现同样的问题，无法访问网站：
[root@totoweb ~]# ps -ef| grep  httpd | wc -l
51
[root@totoweb ~]#
[root@totoweb ~]# ps -u root | wc -l
289
[root@totoweb ~]#

 

修改limits参数，不重新登陆：
[root@totoweb ~]# vi /etc/security/limits.conf

# /etc/security/limits.conf
#
#Each line describes a limit for a user in the form:
#
#           
#
#Where:
# can be:
#        - an user name
#        - a group name, with @group syntax
#        - the wildcard *, for default entry
#        - the wildcard %, can be also used with %group syntax,
#                 for maxlogin limit
#
# can have the two values:
#        - "soft" for enforcing the soft limits
#        - "hard" for enforcing hard limits
#
# can be one of the following:
#        - core - limits the core file size (KB)
#        - data - max data size (KB)
#        - fsize - maximum filesize (KB)
#        - memlock - max locked-in-memory address space (KB)
#        - nofile - max number of open files
#        - rss - max resident set size (KB)
#        - stack - max stack size (KB)
#        - cpu - max CPU time (MIN)
#        - nproc - max number of processes
#        - as - address space limit
#        - maxlogins - max number of logins for this user
#        - priority - the priority to run user process with
#        - locks - max number of file locks the user can hold
#
#                
#

#*               soft    core            0
#*               hard    rss             10000
        hard    nproc           20
#@faculty        soft    nproc           20
#@faculty        hard    nproc           50
#ftp             hard    nproc           0
        -       maxlogins       4
# End of file
*               soft    nproc   20480
*               hard    nproc   20480
*               soft    nofile  65536
*               hard    nofile  65536
"/etc/security/limits.conf" [dos] 48L, 1648C 已写入                                                   

          
[root@totoweb ~]#
[root@totoweb ~]#
[root@totoweb ~]# ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
file size               (blocks, -f) unlimited
pending signals                 (-i) 1024
max locked memory       (kbytes, -l) 32
max memory size         (kbytes, -m) unlimited
open files                      (-n) 65536
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 50
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
[root@totoweb ~]#
[root@totoweb ~]# tstop
Using CATALINA_BASE:   /usr/local/tomcat
Using CATALINA_HOME:   /usr/local/tomcat
Using CATALINA_TMPDIR: /usr/local/tomcat/temp
Using JRE_HOME:       /usr/local/jdk1506
httpd (pid 18039?) not running
[root@totoweb ~]#
[root@totoweb ~]#
[root@totoweb ~]# tstart
Using CATALINA_BASE:   /usr/local/tomcat
Using CATALINA_HOME:   /usr/local/tomcat
Using CATALINA_TMPDIR: /usr/local/tomcat/temp
Using JRE_HOME:       /usr/local/jdk1506
(98)Address already in use: make_sock: could not bind to address 192.168.3.66:80
no listening sockets available, shutting down
Unable to open logs
[root@totoweb ~]#
[root@totoweb ~]# ps -u root | wc -l
288
[root@totoweb ~]# ps -u apache | wc -l
32
[root@totoweb ~]# ps -ef| grep  httpd | wc -l
31
[root@totoweb ~]#

重新登陆但不重启apache：
[root@totoweb ~]# ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
file size               (blocks, -f) unlimited
pending signals                 (-i) 1024
max locked memory       (kbytes, -l) 32
max memory size         (kbytes, -m) unlimited
open files                      (-n) 65536
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 20480
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
[root@totoweb ~]# ps -ef| grep  httpd | wc -l
31
[root@totoweb ~]# ps -u apache | wc -l
31
[root@totoweb ~]# ps -u root | wc -l
288
[root@totoweb ~]# ps -u root | wc -l
288
[root@totoweb ~]# ps -u apache | wc -l
30
[root@totoweb ~]# ps -ef| grep  httpd | wc -l
30
[root@totoweb ~]#
可以看出apache还在被限制进程数

重启apache：
[root@totoweb ~]# tstop
Using CATALINA_BASE:   /usr/local/tomcat
Using CATALINA_HOME:   /usr/local/tomcat
Using CATALINA_TMPDIR: /usr/local/tomcat/temp
Using JRE_HOME:       /usr/local/jdk1506

httpd (pid 18039?) not running
[root@totoweb ~]#
[root@totoweb ~]#
[root@totoweb ~]# tstart
Using CATALINA_BASE:   /usr/local/tomcat
Using CATALINA_HOME:   /usr/local/tomcat
Using CATALINA_TMPDIR: /usr/local/tomcat/temp
Using JRE_HOME:       /usr/local/jdk1506
(98)Address already in use: make_sock: could not bind to address 192.168.3.66:80
no listening sockets available, shutting down
Unable to open logs
[root@totoweb ~]#
因为被修改limits之前，apache被限制，造成当前的apache进程无限循环死掉，所以重启无效

杀死所有apache进程，启动apache，正常工作：
[root@totoweb ~]# kill -9 $(ps -ef | grep httpd | awk '{print $2}')
-bash: kill: (19383) - 没有那个进程
[root@totoweb ~]#
[root@totoweb ~]#
[root@totoweb ~]# ps -ef| grep  httpd | wc -l
1
[root@totoweb ~]#
[root@totoweb ~]# tstart
Using CATALINA_BASE:   /usr/local/tomcat
Using CATALINA_HOME:   /usr/local/tomcat
Using CATALINA_TMPDIR: /usr/local/tomcat/temp
Using JRE_HOME:       /usr/local/jdk1506
[root@totoweb ~]#
[root@totoweb ~]#
[root@totoweb ~]# ps -ef| grep  httpd | wc -l
24
[root@totoweb ~]# ps -u apache | wc -l
143
[root@totoweb ~]# ps -ef| grep  httpd | wc -l
176
[root@totoweb ~]#
[root@totoweb ~]# ps -u root | wc -l
290
[root@totoweb ~]#