HOW TO FIX THE OOM KILLER CRASHES UNDER LINUX

Error log:
[ 347.960000] djpeg invoked oom-killer: gfp_mask=0x200da, order=0, oom_adj=0
[ 347.970000] [<8002d584>] (unwind_backtrace+0x0/0xcc) from [<8006a4a8>] (oom_kill_process.isra.12+0x70/0x1d8)
[ 347.980000] [<8006a4a8>] (oom_kill_process.isra.12+0x70/0x1d8) from [<8006a8e4>](__out_of_memory+0x134/0x154)
[ 347.990000] [<8006a8e4>] (__out_of_memory+0x134/0x154) from [<8006aadc>] (out_of_memory+0x5c/0x90)
[ 348.000000] [<8006aadc>] (out_of_memory+0x5c/0x90) from [<8006d1f8>] (__alloc_pages_nodemask+0x478/0x578)
[ 348.010000] [<8006d1f8>] (__alloc_pages_nodemask+0x478/0x578) from [<8007a83c>](handle_mm_fault+0x1b4/0xa54)
[ 348.020000] [<8007a83c>] (handle_mm_fault+0x1b4/0xa54) from [<8002e4a0>] (do_page_fault+0xe0/0x1dc)
[ 348.030000] [<8002e4a0>] (do_page_fault+0xe0/0x1dc) from [<8002722c>] (do_DataAbort+0x34/0x98)
[ 348.040000] [<8002722c>] (do_DataAbort+0x34/0x98) from [<80027ca0>] (ret_from_exception+0x0/0x10)
[ 348.050000] Exception stack(0x815c1fb0 to 0x815c1ff8)
[ 348.060000] 1fa0: 2cf36518 00000000 00000c10 2cf4d000
[ 348.060000] 1fc0: 00018ab0 00000104 0000040c 00000103 00000104 00017700 00000001 7eb47788
[ 348.070000] 1fe0: 00000000 7eb47680 2ab0a83c 2ac74588 20000010 ffffffff
[ 348.080000] Mem-info:
[ 348.080000] Normal per-cpu:
[ 348.090000] CPU 0: hi: 18, btch: 3 usd: 2
[ 348.090000] active_anon:18743 inactive_anon:2253 isolated_anon:0
[ 348.090000] active_ inactive_ isolated_
[ 348.090000] unevictable:0 dirty:0 writeback:0 unstable:0
[ 348.090000] free:312 slab_reclaimable:52 slab_unreclaimable:533
[ 348.090000] mapped:8 shmem:0 pagetables:56 bounce:0
[ 348.100000] Normal free:1248kB min:1248kB low:1560kB high:1872kB active_anon:74972kB inactive_anon:9012kB active_ inactive_ unevictable:0kB isolated(anon):0kB isolated(file):0kB present:97536kB mlocked:0kB dirty:0kB writeback:0kB mapped:32kB shmem:0kB slab_reclaimable:208kB slab_unreclaimable:2132kB kernel_stack:264kB pagetables:224kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:8 all_unreclaimable? no
[ 348.110000] lowmem_reserve[]: 0 0
[ 348.120000] Normal: 2*4kB 1*8kB 1*16kB 0*32kB 1*64kB 1*128kB 0*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 1248kB
[ 348.130000] 8 total pagecache pages
[ 348.140000] 24576 pages of RAM
[ 348.140000] 359 free pages
[ 348.150000] 2421 reserved pages
[ 348.150000] 585 slab pages
[ 348.150000] 23 pages shared
[ 348.160000] 0 pages swap cached
[ 348.160000] Out of memory: kill process 602 (djpeg) score 1871 or a child
[ 348.170000] Killed process 602 (djpeg)
Killed

So what’s happened? The reason can be explained shortly: The Linux kernel likes to always allocate memory if applications asking for it. Per default it doesn’t really check if there is enough memory available. Given that behavior applications can allocate more memory as really is available. At some point it can definitely cause an out of memory situation. As result the OOM killer will be invoked and will kill that process:

Jun 11 11:35:21 vsrv03 kernel: [378878.356858] php-cgi invoked oom-killer: gfp_mask=0x1280d2, order=0, oomkilladj=0
Jun 11 11:36:11 vsrv03 kernel: [378878.356880] Pid: 8490, comm: php-cgi Not tainted 2.6.26-2-xen-amd64 #1

The downside of this action is that all other running processes are also affected. As result the complete VM didn’t work and needed a restart.

To fix this problem the behavior of the kernel has to be changed, so it will no longer overcommit the memory for application requests. Finally I have included those mentioned values into the /etc/sysctl.conf file, so they get automatically applied on start-up:

vm.overcommit_memory = 2
vm.overcommit_ratio = 80

sh-4.2# sysctl -p
vm.overcommit_memory = 2
vm.overcommit_ratio = 80