全部博文(317)
分类: LINUX
2015-07-31 11:25:39
服务器正常运行2个月,前天全站突然出现502 Bad Gateway,所有页面均打不开,采用的是nginx+php-fpm模式。
服务器情况:这个服务器流量比较大,每天PV大概30万,数据库使用的mysql5,最大表数据有80万,数据库总容量10G左右。
故障出现以后立即想到可能是php-fpm挂掉。ps下查看有php-fpm进程,然后查看端口也有9000运行,说明php-fpm运行正常。为什么还是出现502。重启nginx问题依旧,然后重启php-fpm问题还是没解决。
正在急切查找问题根源,突然服务器连接不上了,一问原来另一个同事在不知情的情况下居然重启了服务器。等服务器重启起来又正常了,太奇怪的问题。
nginx搭配的php-fpm突然停止响应的原因初步找到。
前几天发现一个服务器的php-fpm突然中止响应,而且重启php-fpm依然无效。
症状:
1。nginx运行正常,mysql运行正常。静态html文件可以正常访问。说明nginx服务没问题
2。只有php页面打不开。报错502 Bad getway,说明是nginx和php-fpm之间出现问题.
3。nginx的80端口,php-fpm的9000端口都在运行。php页面仍然无法响应。
4。重启php-fpm进程问题依旧。最后重启服务器才恢复,问题比较严重。
查看核心时间日志文件如下:
[01-Apr-2014 17:38:13] WARNING: [pool www] child 31153, script '/data/web/php/api/work.php' (request: "GET /api/work.php") execution timed out (55.098115 sec), terminating
[01-Apr-2014 17:38:13] WARNING: [pool www] child 31153 exited on signal 15 (SIGTERM) after 112950.411997 seconds from start
[01-Apr-2014 17:38:13] NOTICE: [pool www] child 1436 started
[01-Apr-2014 17:38:21] WARNING: [pool www] child 29802, script
'/data/web/php/api/work.php' (request: "GET /api/work.php") executing
too slow (3.019993 sec), logging
[01-Apr-2014 17:38:21] NOTICE: child 29802 stopped for tracing
[01-Apr-2014 17:38:21] NOTICE: about to trace 29802
[01-Apr-2014 17:38:21] ERROR: failed to ptrace(PEEKDATA) pid 29802: Input/output error (5)
[01-Apr-2014 17:38:21] NOTICE: finished trace of 29802
[01-Apr-2014 17:38:59] WARNING: [pool www] child 28407, script
'/data/web/php/api/work.php' (request: "GET /api/work.php") execution
timed out (55.150273 sec), terminating
[01-Apr-2014 17:38:59] WARNING: [pool www] child 28407 exited on signal 15 (SIGTERM) after 8480.945886 seconds from start
可以看到有些进程占用时间很长。55秒以上。然后就出现php-fpm停止响应。
初步判断应该是程序不合理引起的脚本长时间执行导致服务器资源崩溃,最后导致php-fpm出现不可恢复的异常。不知道是不是php-fpm的bug.
php版本为:PHP Version 5.4.23,采用php-fpm+Nginx方式,采用9000端口和nginx进行通信。
php-fpm突然不能响应并报错 ERROR: failed to ptrace(PEEKDATA) pid 3854: Input/output error
nginx+php-fpm(php版本是5.4的),php-fpm运行一段时间后突然停止响应,重启nginx和php-fpm都无效。只能重
启服务器,查看到php-fpm的log日志如下,很多类似ERROR: failed to ptrace(PEEKDATA) pid 23199:
Input/output error (5)这样的错误日志。
[03-Apr-2014 13:40:24] ERROR: failed to ptrace(PEEKDATA) pid 23199: Input/output error (5)
[03-Apr-2014 13:40:24] NOTICE: finished trace of 23199
[03-Apr-2014 13:40:44] WARNING: [pool www] child 3854, script
'/var/www/' (request: "POST /getacd.php") executing too
slow (3.618411 sec), logging
[03-Apr-2014 13:40:44] NOTICE: child 3854 stopped for tracing
[03-Apr-2014 13:40:44] NOTICE: about to trace 3854
[03-Apr-2014 13:40:44] ERROR: failed to ptrace(PEEKDATA) pid 3854: Input/output error (5)
[03-Apr-2014 13:40:44] NOTICE: finished trace of 3854
[03-Apr-2014 13:40:56] WARNING: [pool www] child 23375, script
'/var/www/' (request: "POST /getacd.php") executing too
slow (3.639160 sec), logging
[03-Apr-2014 13:40:56] NOTICE: child 23375 stopped for tracing
[03-Apr-2014 13:40:56] NOTICE: about to trace 23375
[03-Apr-2014 13:40:56] NOTICE: finished trace of 23375
[03-Apr-2014 13:41:16] WARNING: [pool www] child 23199, script
'/var/dweb/web5/api/acdsvr.php' (request: "GET /api/acdsvr.php")
execution timed out (55.197093 sec), terminating
[03-Apr-2014 13:41:16] WARNING: [pool www] child 23199 exited on signal 15 (SIGTERM) after 67025.486593 seconds from start
[03-Apr-2014 13:41:16] NOTICE: [pool www] child 29676 started
[03-Apr-2014 13:43:01] WARNING: [pool www] child 13723, script
'/var/www/' (request: "POST /getacd.php") executing too
slow (3.707757 sec), logging
[03-Apr-2014 13:43:01] NOTICE: child 13723 stopped for tracing
[03-Apr-2014 13:43:01] NOTICE: about to trace 13723
[03-Apr-2014 13:43:01] ERROR: failed to ptrace(PEEKDATA) pid 13723: Input/output error (5)
[03-Apr-2014 13:43:01] NOTICE: finished trace of 13723
[03-Apr-2014 13:44:01] WARNING: [pool www] child 28475, script
'/var/www/' (request: "POST /getacd.php") executing too
slow (3.134600 sec), logging
[03-Apr-2014 13:44:01] NOTICE: child 28475 stopped for tracing
[03-Apr-2014 13:44:01] NOTICE: about to trace 28475
[03-Apr-2014 13:44:01] ERROR: failed to ptrace(PEEKDATA) pid 28475: Input/output error (5)
问题正在跟踪!
loved999992015-10-21 21:55:04
<p><img alt=\"AB认牌粉:AB认牌粉与“挂花药膏”使用各有不同,AB认牌粉是白色粉面无味,戴眼镜后效果为深兰色。\" src=\"http://www.paiju.org/templates/cn/images/222.jpg\" width=\"772\" height=\"492\" /></p><p> 一直在被模仿的认牌粉,但从未被超越过!<span style=\"font-size:12px;\">原文地址:</span><a href=\"http://www.paiju.org/content/?914.htm\"><span style=\"font-size:12px;\">http://www.paiju.org/content/?914.htm</span></a><span style=\"font-size:12px;\"> <