Oracle RAC 11gR2数据库单节点linux操作系统无法启动

场景:新部署的RAC数据库，在做数据库初始化的时候，大量的并发导致操作，加之服务器/u01目录   写满(并发导入上TB数据)，在这个状态下，直接服务器reboot，导致linux服务器无法启动。

环境: Oracle RAC 11gR2
        Linux redhat 6.4

错误日志如下:

INFO: task bonnie++:31785 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

udevd[1368]:worker[1528] unexpectedly returned with status 0x0100
udevd[1368]:worker[1528] failed while handling '/devices/pci0000:00/0000:00:03.2/0000:04:00.1/host1/rport-2:0-5/target2:0:3/2:0:3:32/block/sdb/sdb5'

根源分析:

This is a know bug. By default Linux uses up to 40% of the available memory for file system caching. After this mark has been reached the file system flushes all outstanding data to disk causing all following IOs going synchronous. For flushing out this data to disk this there is a time limit of 120 seconds by default. In the case here the IO subsystem is not fast enough to flush the data withing 120 seconds. This especially happens on systems with a lof of memory.

The problem is solved in later kernels and there is not “fix” from Oracle. I fixed this by lowering the mark for flushing the cache from 40% to 10% by setting “vm.dirty_ratio=10” in /etc/sysctl.conf. This setting does not influence overall database performance since you hopefully use Direct IO and bypass the file system cache completely.

操作思路:

通过理解，刷新大量的数据(在缓存上面)写到硬盘上，默认的时间限制是120秒。大量的写操作，一时写不进去，就触发了这个issue。因此，修改了/etc/sysctl.conf文件的参数如下：

vm.dirty_ratio=10

-------------------------------------------分割线-------------------------------------------------------------
虽然修改了上述操作，还没有完全解决问题。
继续问题分析:

• The number of spawned udevd workers depends only on the amount of RAM. As a result, for machines with relatively big RAM sizes and lots of disks, a lot of udevd workers are running in parallel, maximizing CPU and I/O. This can cause udev events to timeout, because of hardware bottlenecks.
  【大量并发的udevd 工作进程，最大化的使用CPU和IO，最终导致udev事件超时，原因为：硬件瓶颈】

• A fix that helps govern multiple parallel driver loads that were occurring via modprobe to prevent unnecessary driver loads which contributed to high system resource use during device discovery which also could case the udev events to timeout.
  

• This and related isses were fixed within the udev-147-2.63.el6_7.1 via (private) bugzilla 1281469 and 1281467. Additional fixes are present from (private) bugzillas 1170313, 885978 and 816724 that address other related issues that contribute to 0x100 messages being displayed.
  【udev的补丁可以修复和改善这个issue】
  
  
  修改思路：
  在/lib/udev/rules.d/10-dm.rules文件，添加一行参数如下：
  
  OPTIONS+="event_timeout=600"
  
  然后，重启服务器的时候，注意，先关机。
  

  • shutdown -h now
  
  
  10分钟之后，在启动Oracle节点服务器，之后，linux服务器正常启动。

文献参考:
【1】http://blog.ronnyegner-consulting.de/2011/10/13/info-task-blocked-for-more-than-120-seconds/
【2】http://m.blog.csdn.net/vic_qxz/article/details/72781461