全部博文（82）
分类： LINUX
2010-05-07 15:49:41
Major distribution kernels set the default value of /proc/sys/vm/overcommit_memory to zero, which means that processes can request more memory than is currently free in the system. This is done based on the heuristics that allocated memory is not used immediately, and that processes, over their lifetime, also do not use all of the memory they allocate. Without overcommit, a system will not fully utilize its memory, thus wasting some of it. Overcommiting memory allows the system to use the memory in a more efficient way, but at the risk of OOM situations. Memory-hogging programs can deplete the system's memory, bringing the whole system to a grinding halt. This can lead to a situation, when memory is so low, that even a single page cannot be allocated to a user process, to allow the administrator to kill an appropriate task, or to the kernel to carry out important operations such as freeing memory. In such a situation, the OOM-killer kicks in and identifies the process to be the sacrificial lamb for the benefit of the rest of the system.
Users and system administrators have often asked for ways to control the behavior of the OOM killer. To facilitate control, the /proc/
The process to be killed in an out-of-memory situation is selected based on its badness score. The badness score is reflected in /proc/
Any process unlucky enough to be in the swapoff() system call (which removes a swap file from the system) will be selected to be killed first. For the rest, the initial memory size becomes the original badness score of the process. Half of each child's memory size is added to the parent's score if they do not share the same memory. Thus forking servers are the prime candidates to be killed. Having only one "hungry" child will make the parent less preferable than the child. Finally, the following heuristics are applied to save important processes:
The task with the highest badness score is then selected and its children are killed. The process itself will be killed in an OOM situation when it does not have children.
/proc/
Alan Cox disliked this solution, that containers are the most appropriate way to control the problem. In response to this suggestion, the , contributed by Nikanth Karthikesan, provides control of the sequence of processes to be killed when the system runs out of memory. The patch introduces an OOM control group (cgroup) with an oom.priority field. The process to be killed is selected from the processes having the highest oom.priority value.
To take control of the OOM-killer, mount the cgroup OOM pseudo-filesystem introduced by the patch:
# mount -t cgroup -o oom oom /mnt/oom-killer
The OOM-killer directory contains the list of all processes in the file tasks, and their OOM priority in oom.priority. By default, oom.priority is set to one.
If you want to create a special control group containing the list of processes which should be the first to receive the OOM killer's attention, create a directory under /mnt/oom-killer to represent it:
# mkdir lambs
Set oom.priority to a value high enough:
# echo 256 > /mnt/oom-killer/lambs/oom.priority
oom.priority is a 64-bit unsigned integer, and can have a maximum value an unsigned 64-bit number can hold. While scanning for the process to be killed, the OOM-killer selects a process from the list of tasks with the highest oom.priority value.
Add the PID of the process to be added to the list of tasks:
# echo> /mnt/oom-killer/lambs/tasks
To create a list of processes, which will not be killed by the OOM-killer, make a directory to contain the processes:
# mkdir invincibles
Setting oom.priority to zero makes all the process in this cgroup to be excluded from the list of target processes to be killed.
# echo 0 > /mnt/oom-killer/invincibles/oom.priority
To add more processes to this group, add the pid of the task to the list of tasks in the invincible group:
# echo> /mnt/oom-killer/invincibles/tasks
Important processes, such as database processes and their controllers, can be added to this group, so they are ignored when OOM-killer searches for processes to be killed. All children of the processes listed in tasks automatically are added to the same control group and inherit the oom.priority of the parent. When multiple tasks have the highest oom.priority, the OOM killer selects the process based on the oom_score and oom_adj.
This approach did not appeal to cpuset users, though. Consider two cpusets, A and B. If a process in cpuset A has a high oom.priority value, it will be killed if cpuset B runs out of memory, even though there is enough memory in cpuset A. This calls for a different design to tame the OOM killer.
An interesting outcome of the discussion has been handling OOM situations in user space. The kernel sends notification to user space, and applications respond by dropping their user-space caches. In case the user-space processes are not able to free enough memory, or the processes ignore the kernel's requests to free memory, the kernel resorts to the good old method of killing processes. , developed by Kosaki Motohiro, is one such attempt made in the past. However, the cannot be applied to versions beyond 2.6.28 because the memory management reclaiming sequence have changed, but the design principles and goals can be reused. David Rientjes having one of the two hybrid solutions:
The other is /dev/mem_notify that allows you to poll() on a device file and be informed of low memory events. This can include the cgroup oom notifier behavior when a collection of tasks is completely out of memory, but can also warn when such a condition may be imminent. I suggested that this be implemented as a client of cgroups so that different handlers can be responsible for different aggregates of tasks.
Most developers prefer making /dev/mem_notify a client of control groups. This can be further extended to merge with the proposed oom-controller.
The Android developers required a greater degree of control over the low memory situation because the OOM killer does not kick in till late in the low memory situation, i.e. till all the cache is emptied. Android wanted a solution which would start early while the free memory is being depleted. So they introduced the "lowmemory" driver, which has multiple thresholds of low memory. In a low-memory situation, when the first thresholds are met, background processes are notified of the problem. They do not exit, but, instead, save their state. This affects the latency when switching applications, because the application has to reload on activation. On further pressure, the lowmemory killer kills the non-critical background processes whose state had been saved in the previous threshold and, finally, the foreground applications.
Keeping multiple low memory triggers gives the processes enough time to free memory from their caches because in an OOM situation, user-space processes may not be able to run at all. All it takes is a single allocation from the kernel's internal structures, or a page fault to make the system run out of memory. An earlier notification of a low-memory situation could avoid the OOM situation with a little help from the user space applications which respond to low memory notifications.
Killing processes based on kernel heuristics is not an optimal solution, and these new initiatives of offering better control to the user in selecting the process to be the sacrificial lamb are steps to a robust design to give more control to the user. However, it may take some time to come to a consensus on a final control solution.
Taming the OOM killer
Posted Feb 5, 2009 2:29 UTC (Thu) by brouhaha (subscriber, #1698) []
I'm still baffled as to why this is an issue at all. IMNSHO, the ability to overcommit memory should never have been created in the first place. If you need more memory, buy more memory, or create a larger swap partition or file.What user-space programs are allocating so much more memory than they actually need, anyhow?
I've been doing all of my software development and electrical engineering work (including schematic capture, PCB layout including autorouting, HDL simulation, FPGA synthesis, etc.) for years on a system with no swap space and with memory overcommit disabled, and I haven't run into any problem with it. On rare occasions I've been unable to start a synthesis or simulation run, which is perfectly fine with me, because I'd much rather be unable to start a run than to have a run get killed randomly after hours because the memory it thought it had allocated wasn't actually available, or worse yet, have some other random process killed.
Taming the OOM killer
Posted Feb 5, 2009 6:19 UTC (Thu) by cpeterso (guest, #305) []
Once the OOM Killer starts shooting down processes, I can't imagine that system will remain in a usable state much longer. You've ran out of memory and your processes (and work in progress) are gone.
Given that major distros default OOM Killer to off, who is the target market for the OOM Killer?
Taming the OOM killer
Posted Feb 8, 2009 19:05 UTC (Sun) by anton (subscriber, #25547) []
Once the OOM Killer starts shooting down processes, I can't imagine that system will remain in a usable state much longer.I have actually experienced several times that the system was stable and usable after the OOM killer had killed the right process. This typically involved killing only a pure user program that was not needed for any system job. In several cases these were compiler runs on a machine with 24GB of RAM and 48GB of swap, and buying more memory was not very practical, and probably would not have helped anyway: the memory consumption was probably due to a bug in the compiler.
Concerning memory overcommitment, I think that this is a good idea for most programs (which are not written to survive failing memory allocations). And relying on overcommitment can simplify programming: e.g., allocate a big chunk and put your growing structure there instead of reallocating all the time. And when you do it, do it right (i.e., echo 1 >/proc/sys/vm/overcommit_memory), not the half-hearted Linux default, which gives us the disadvantages of overcommitment (i.e., the OOM killer) combined with the disadvantages of no overcommitment (unpredictable allocation failures).
Concerning critical system programs, those should be written to survive failing memory allocations, should get really-committed memory if the allocation succeeds, and consequently should not be OOM-killed.
I have outlined this idea in more depth, and I think that AIX and/or Solaris implement something similar. Instead of my per-process idea, the MAP_NORESERVE flag allows to switch between commitment and overcommitment on a per-allocation basis (not sure how useful that is, as well as the default of committing memory).
Taming the OOM killer
Posted Feb 5, 2009 7:37 UTC (Thu) by dlang (subscriber, #313) []
the fork technically needs to allocate as much memory as the program is currently using.
this means that if you have a 512MB firefox process that needs to exec a 64k program to handle some mime time, you first allocate 512MB of additional memory, then exec the 64K program (at which time 511.95M of ram will be freed.
it used to be (in the bad old unix days) that when a fork happened the kernel would take the time to copy all 512MB of ram, only to throw it away a few ms later (the vfork call was invented to tell the system that it wanted to fork, but not really as a work-around for this)
modern *nix systems instead go through and mark those 500MB of memory as Copy-On-Write (COW), which isn't free, but is _FAR_ cheaper than actually touching all the memory (in extreme cases it changes from touching all 1G of ram (512MB of read, 512MB of write) to touching 16K of memory (1 bit per 8k page). this is a phenomenal speedup ( and it gets even better when you consider the amount of cpu cache that you avoid corrupting in the process.
in cases where the system doesn't exec something else you can get drastic savings due to the fact that the memory pages that contain the binary you are executing never change and therefor you never need to copy them. this is similar to the memory savings from doing threading, but without the risk of one thread affecting another thread
Taming the OOM killer
Posted Feb 5, 2009 8:00 UTC (Thu) by brouhaha (subscriber, #1698) []
Assuming that when a big process does a fork() that it is going to exec() a small process is completely absurd. Sure, that may happen fairly often, but the case where a small process does a fork() and does an exec() of a large process also happens fairly often.
Usually somewhere in this discussion someone says "but what about embedded systems", which I claim actually supports my position, because it is even MORE important in an embedded system for there to (1) be sufficient memory/swap, and (2) not let the OOM killer nuke some random process if there isn't.
Taming the OOM killer
Posted Feb 5, 2009 8:06 UTC (Thu) by dlang (subscriber, #313) []
also, actually _using_ swap can be extremely painful, sometime more painful than simply crashing the system (at least that you can have watchdogs for and failover/reboot)
in theory you are right, the system is unpredictable with overcommit enabled. in practice it is reliable enough for _many_ uses
Taming the OOM killer
Posted Feb 5, 2009 8:22 UTC (Thu) by brouhaha (subscriber, #1698) []
swap isn't free, especially on embedded systemsI have yet to use an embedded Linux system that didn't have substantially more "disk" than RAM, except one case in which there was no disk but plenty of RAM. I somewhat question the wisdom of designing an embedded Linux system for which there is little RAM and even less disk.
In an embedded system, I wouldn't expect there to be a large number of processes sleeping between a fork() and exec(). If there were, that would most likely be a sign of serious problems, so having a fork() fail under such circumstances seems like a good thing.
also, actually _using_ swap can be extremely painful, sometime more painful than simply crashing the systemSure, but when copy-on-write is used for the fork()/exec() case that seems to be what people are worried about, the swap won't actually be used. It will just be reserved until the exec().
If you're concerned about system performance degrading because of excessive swap usage, there's no reason why you can't have a user-space process to act as a watchdog for that problem, which may occur for reasons other than memory "overcommit".
I've been involved in engineering a number of embedded products that had to have high reliability in the field, and I would not dream of shipping such a product with the kernel configured to allow memory overcommitment. Even though you can _usually_ get away with it, "usually" isn't good enough. There simply needs to be enough memory (and/or swap) to handle the worst-case requirements of the system. Otherwise it _will_ fail in the field, and thus not be as reliable as intended.
Taming the OOM killer
Posted Feb 5, 2009 8:33 UTC (Thu) by dlang (subscriber, #313) []
if you could allocate the address space but then tell the kernel "don't really use it" you may be ok, but how is that different from the current overcommit?
you _are_ overcommiting (compared to what is acceptable to the system's performance) and counting on the efficiancies of COW to keep you from actually using the swap space you have comitted.
the only difference is that you overcommit up to a given point (at which time your allocations start failing, which may also cause the system to 'fail' as far as the user is concerned)
i fully agree that there are situations where disabling overcommit is the right thing to do. However, I am also seeing other cases where allowing overcommit is the right thing to do.
Taming the OOM killer
Posted Feb 5, 2009 9:07 UTC (Thu) by brouhaha (subscriber, #1698) []
if you could allocate the address space but then tell the kernel "don't really use it"I'm not telling the kernel "don't use it". If the kernel needs to, it will use it. For the primary case people seem concerned with, the time between fork() and exec(), it will be committed but due to COW, it won't actually get used. It may still get used for other cases, and within reason that's a good thing, but a user-space daemon can take some system-specific corrective action if it gets out of hand. This provides a whole lot more flexibility in error handling than a user-space daemon that would only control the behavior of the OOM killer.
you may be ok, but how is that different from the current overcommit?It's different because the kernel is NEVER going to kill an unrelated process selected by a heuristic. It is going to fail an allocation or fork, and the software can take some reasonable recovery action.
The system should not be designed or configured such that the kernel can fail to provide memory that has been committed, because there is NO reasonable recovery mechanism for that. It is far easier to handle memory allocation problems gracefully when the error is reported at the time of the attempt to commit the memory, rather than at some random future time.
Taming the OOM killer
Posted Feb 5, 2009 9:16 UTC (Thu) by dlang (subscriber, #313) []
there are times for that, but there are also times when 99.999% reliability is good enough.
Taming the OOM killer
Posted Feb 5, 2009 10:23 UTC (Thu) by epa (subscriber, #39769) []
(On a related note, the kernel is free to refuse any request for extra memory, and can do so for its own reasons. So for example if a process needs to fork() then the memory allocation would normally succeed, on the assumption that the extra memory probably won't be used, but provided there is enough swap space to back it up just in case. Whereas an explicit memory allocation 'I want ten gigabytes' could, as a matter of policy, be denied if the system doesn't have that much physical RAM.)
Taming the OOM killer
Posted Feb 5, 2009 21:03 UTC (Thu) by dlang (subscriber, #313) []
but if you have large programs that may need to fork, it's not nessasarily the case that it's 'a slightly larger amount of swap'. I've seen people arguing your point of view toss off that a large system should be willing to dedicate a full 1TB drive just to swap so that it can turn overcommit off. in practice, if you end up using more than a gig or so of swap your system slows to a crawl
Taming the OOM killer
Posted Feb 5, 2009 22:26 UTC (Thu) by epa (subscriber, #39769) []
It is rather a problem that merely adding swap space as available means it can then be used by applications just as willingly as physical RAM. Perhaps a per-process policy flag would say whether an app can have its memory allocation requests start going to swap (as opposed to getting 0 from malloc() when physical RAM is exhausted). Then sysadmins could switch this flag on for particular processes that need it.
Taming the OOM killer
Posted Feb 6, 2009 0:45 UTC (Fri) by nix (subscriber, #2304) []
Taming the OOM killer
Posted Feb 5, 2009 12:42 UTC (Thu) by michaeljt (subscriber, #39183) []
Taming the OOM killer
Posted Feb 5, 2009 15:44 UTC (Thu) by hppnq (subscriber, #14462) []
I would rather have my system slow to an unusable crawl if I was confident that it would come out of it again at some point. Even then, I can still press the reset button, which is what I have usually ended up doing in OOM situations anyway.
On your home system this makes some sense, but all this goes out the window once you have to take service levels into account.
Taming the OOM killer
Posted Feb 6, 2009 7:46 UTC (Fri) by michaeljt (subscriber, #39183) []
Taming the OOM killer
Posted Feb 6, 2009 8:58 UTC (Fri) by dlang (subscriber, #313) []
you probably already have tools in place to detect processes dieing and either restart them (if the memory preasure is temporary) or failover to another box (gracefully for all the other processes on the box)
Taming the OOM killer
Posted Feb 5, 2009 21:06 UTC (Thu) by dlang (subscriber, #313) []
if you are willing to hit reset in this condition then you should be willing to deal with the OOM killer killing the box under the same conditions.
Taming the OOM killer
Posted Feb 6, 2009 8:00 UTC (Fri) by michaeljt (subscriber, #39183) []
Taming the OOM killer
Posted Feb 12, 2009 19:14 UTC (Thu) by efexis (guest, #26355) []
Personally I have swap disabled or set very low, as a runaway process will basically mean I lose contact with a server, unable to log in to it or anything, until it has finished chewing through all available memory *and* swap (causing IO starvation, IO being the thing I need to log in and kill the offending task) until it hits the limit and gets killed.
Everything important is set to be restarted, either directly from init, or indirectly from daemontools or equivalent, which is restarted by init should it go down (which has never happened).
Taming the OOM killer
Posted Feb 13, 2009 23:33 UTC (Fri) by michaeljt (subscriber, #39183) []
Taming the OOM killer
Posted Feb 14, 2009 0:03 UTC (Sat) by dlang (subscriber, #313) []
how much swap did you allocate? any idea how much was used?
enabling overcommit with small amounts of swap will allow large programs to fork without problems, but will limit runaway processes. it's about the textbook case for using overcommit.
Taming the OOM killer
Posted Feb 16, 2009 9:04 UTC (Mon) by michaeljt (subscriber, #39183) []
Definitely too much (1 GB for 2 GB of RAM), as I realised after reading this: . That page was also what prompted my last comment. It seems a bit strange to me that increasing swap size should so badly affect system performance in this situation, and I wondered whether this could be fixed with the right tweak, such as limiting the amount of virtual memory available to processes, say to a default of 80 percent of physical RAM. This would still allow for large processes to fork, but might catch runaway processes a bit earlier. I think that if I find some time, I will try to work out how to do that (assuming you don't answer in the mean time to tell me why that is a really bad idea, or that there already is such a setting).
Taming the OOM killer
Posted Feb 16, 2009 15:38 UTC (Mon) by dlang (subscriber, #313) []
Taming the OOM killer
Posted Feb 17, 2009 8:23 UTC (Tue) by michaeljt (subscriber, #39183) []
Indeed. I set ulimit -v 1600000 (given that I have 2GB of physical RAM) and launched a known bad process (gnash on a page I know it can't cope with). gnash crashed after a few minutes, without even slowing down my system. I just wonder why this is not done by default. Of course, one could argue that this is a user or distribution problem, but given that knowledgeable people can change the value, why not in the kernel? (Again, to say 80% of physical RAM. I tried with 90% and gnash caused a noticeable performance degradation.) This is not a rhetorical question, I am genuinely curious.
Taming the OOM killer
Posted Feb 17, 2009 8:29 UTC (Tue) by dlang (subscriber, #313) []
the distro is in the same boat. if they configured it to do what you want, they would have other people screaming at them that they would rather see the computer slow down than have programs die (you even see people here arguing that)
Taming the OOM killer
Posted Feb 17, 2009 14:27 UTC (Tue) by michaeljt (subscriber, #39183) []
It does take a decision though - to allow all programmes to allocate as much RAM as they wish by default, even if it is not present, is very definitely a policy decision. Interestingly Wine fails to start if I set ulimit -v in this way (I can guess why). I wonder whether disabling overcommit would also prevent it from working?
Taming the OOM killer
Posted Feb 5, 2009 10:17 UTC (Thu) by epa (subscriber, #39769) []
any process that forks and execs allocates more memory than it needs.Quite. Which is why a single fork-and-exec-child-process system call is needed. With that, there would be much less need to overcommit memory and so a better chance of avoiding hacks like the OOM killer.
The classical Unix design of separate fork() and exec() is elegant at first glance, but in practice it has caused various unpleasant kludges to cope with the memory overcommit. (Another one was vfork(), which IIRC was a fork call that used less memory but only worked as long as you promise to call exec() immediately afterwards. Why they didn't make a single fork-plus-exec primitive rather than this crufty interface eludes me.)
Taming the OOM killer
Posted Feb 5, 2009 11:16 UTC (Thu) by iq-0 (subscriber, #36655) []
I don't know if filehandle closing is allowed after vfork, but this would also be a great help to ensure the right file handles are passed (which would be a tedious operation or really complex/verbose in the case of a single fork-and-exec call).
Taming the OOM killer
Posted Feb 5, 2009 14:08 UTC (Thu) by epa (subscriber, #39769) []
This way you can even implement a simple 'if exec(a) fails try exec(b) or otherwise flag error with (possibly program specific) meaningful data'...Oh sure, sometimes you will want to do something more complex like that. In these cases vfork() (as in classic BSD) doesn't work, because the child process is using memory belonging to the parent. The traditional fork() then exec() is best.
But this is a small minority of the cases when an external command is run. And running an external command accounts for a large proportion of total fork()s. I'm just suggesting to make something more robust (avoiding the need to overcommit memory) for the common case.
Taming the OOM killer
Posted Feb 5, 2009 17:03 UTC (Thu) by nix (subscriber, #2304) []
Taming the OOM killer
Posted Feb 5, 2009 22:05 UTC (Thu) by epa (subscriber, #39769) []
Perhaps another way to avoid the need for memory allocation would be to use a new vfork-like call (heck, it could even be called vfork) that has a fixed memory budget set as a matter of policy system-wide. So when you vfork(), the memory is set up as copy-on-write, but the child process has a budget of at most 1000 pages it can scribble on. That should be enough to set up the necessary file descriptors, but if it tries to dirty more than its allowance it is summarily killed.
That way, there is some upper limit to the amount of memory that needs to be allocated - when vfork()ing the kernel just needs to ensure 1000 free pages - and the kernel doesn't have to make a (possibly untrustworthy) promise that the whole process address space is available for normal use.
Taming the OOM killer
Posted Feb 5, 2009 11:49 UTC (Thu) by alonz (subscriber, #815) []
This isn't really simpler (except on block diagrams…)
Taming the OOM killer
Posted Feb 5, 2009 12:52 UTC (Thu) by michaeljt (subscriber, #39183) []
Taming the OOM killer
Posted Feb 5, 2009 20:00 UTC (Thu) by quotemstr (subscriber, #45331) []
(From SUSv2 / POSIX draft.)The vfork() function has the same effect as fork(2), except that the behavior is undefined if the process created by vfork() either modifies any data other than a variable of type pid_t used to store the return value from vfork(), or returns from the function in which vfork() was called, or calls any other function before successfully calling _exit(2) or one of the exec(3) family of functions.On the other hand,
pid_t child = clone(run_child_func, run_child_stack, CLONE_VM, run_child_data) would do the trick. The child would share memory with the parent, making overcommit unnecessary, but would have a different file descriptor table, allowing pipelines to be set up easily.
Taming the OOM killer
Posted Feb 5, 2009 16:55 UTC (Thu) by nix (subscriber, #2304) []
Taming the OOM killer
Posted Feb 5, 2009 17:40 UTC (Thu) by martinfick (subscriber, #4455) []
Taming the OOM killer
Posted Feb 5, 2009 22:22 UTC (Thu) by epa (subscriber, #39769) []
For those making embedded or high-availability systems who want to try harder and turn off overcommit altogether, fork-then-exec could be replaced in user space with posix_spawn or vfork-then-exec or similar.
Taming the OOM killer
Posted Feb 5, 2009 23:11 UTC (Thu) by martinfick (subscriber, #4455) []
The OOM killer kicks in when memory has been overcommitted through COW. Two processes are sharing the same memory region and one of them decides to write to that shared COW page requiring the page to now be copied. There is no memory allocating happening, simply a write to a memory page which is already allocated to a process (two of them actually).
Again, the fork then exec shortcut is not really the big deal, it is processes that fork and do not exec and then eventually write to a COW page.
Taming the OOM killer
Posted Feb 6, 2009 0:51 UTC (Fri) by nix (subscriber, #2304) []
- when a process stack is grown
- when a fork()ed process COWs
- when a page in a private file-backed mapping is written to for the
first time
- when a nonswappable kernel resource needs to be allocated (other than a
cache) which cannot be discarded when memory pressure is high
- if overcommit_memory is set, if a page from the heap or an anonymous
mmap() is requested for the first time
So the OOM killer is *always* needed, even if overcommitting were disabled
as much as possible. (You can overcommit disk space, too: thanks to sparse
files, you can run out of disk space writing to the middle of a file. With
some filesystems, e.g. NTFS, you can run out of disk space by renaming a
file, triggering a tree rebalance and node allocation when there's not
enough disk space left. NTFS maintains an emergency pool for this
situation, but it's only so large...)
Taming the OOM killer
Posted Feb 6, 2009 0:58 UTC (Fri) by martinfick (subscriber, #4455) []
Taming the OOM killer
Posted Feb 6, 2009 1:26 UTC (Fri) by dlang (subscriber, #313) []
how would you propose that programmers handle an error when they allocate a variable? (which is one way to grow the stack)
Taming the OOM killer
Posted Feb 6, 2009 1:38 UTC (Fri) by brouhaha (subscriber, #1698) []
The process should get a segfault or equivalent signal. If there is a handler for the signal, but the handler can't be invoked due to lack of stack space, the process should be killed. If the mechanism to signal the process in a potential out-of-stack situation is too complex to be practically implemented in the kernel, then the process should be killed without attempting to signal it.At no point should the OOM killer become involved, because there is no reason to propagate the error outside the process (other than by another process noticing that the process in question has exited). A principle of reliable systems is confining the consequences of an error to the minimum area necessary, and killing some other randomly-selected (or even heuristically-selected) process violates that principle.
Taming the OOM killer
Posted Feb 6, 2009 5:26 UTC (Fri) by njs (subscriber, #40338) []
This makes sense on the surface, but memory being a shared resource means that everything is horribly coupled no matter what and life isn't that simple.
You have 2 gigs of memory.
Process 1 and process 2 are each using 50 megabytes of RAM.
Then Process 1 allocates another 1948 megabytes.
Then Process 2 attempts to grow its stack by 1 page, but there is no memory.
The reason the OOM exists is that it makes no sense to blame Process 2 for this situation. And if you did blame Process 2, then the system would still be hosed and a few minutes later you'd have to kill off Process 3, Process 4, etc., until you got lucky and hit Process 1.
Taming the OOM killer
Posted Feb 7, 2009 17:52 UTC (Sat) by oak (guest, #2786) []
Taming the OOM killer
Posted Feb 8, 2009 15:26 UTC (Sun) by nix (subscriber, #2304) []
Taming the OOM killer
Posted Feb 12, 2009 14:32 UTC (Thu) by epa (subscriber, #39769) []
The OOM killer does not come into play when malloc is called. If malloc is called when there in no memory there is no need to kill any processes, malloc simply fails and return the appropriate error code.Ah, I didn't realize that. From the way people talk it sounded as though malloc() would always succeed and then the process would just blow up trying to use the memory. If the only memory overcommit is COW due to fork() then it's not so bad (though I still think some kind of vfork() would be a more hygienic practice).
Taming the OOM killer
Posted Feb 12, 2009 15:23 UTC (Thu) by tialaramex (subscriber, #21167) []
Taming the OOM killer
Posted Feb 12, 2009 16:06 UTC (Thu) by dlang (subscriber, #313) []
there are a growing number of such functions in C nowdays as people go back and figure out where programmers commonly get it wrong and provide functions that are much harder to misuse (the case that springs to mind are the string manipulation routines)
Taming the OOM killer
Posted Feb 5, 2009 22:49 UTC (Thu) by tbird20d (subscriber, #1901) []
I'm still baffled as to why this is an issue at all. IMNSHO, the ability to overcommit memory should never have been created in the first place. If you need more memory, buy more memory, or create a larger swap partition or file.This is not an option in embedded devices - particularly those that ship in the millions of units.
What user-space programs are allocating so much more memory than they actually need, anyhow?This simple answer is, in a modern system, all of them.
Nearly all programs have a virtual memory footprint greater than they will actually use (at one time, or even during their entire lifetime). This is true even ignoring the fork-and-exec issue. For large systems, this discrepancy is easily ignored or worked around, but for low-resource systems, the difference becomes a major issue. The ability to overcommit memory is actually one of the reasons Linux is selected over traditional RTOSes for some embedded projects.
Taming the OOM killer
Posted Feb 7, 2009 21:37 UTC (Sat) by giraffedata (subscriber, #1954) []
the ability to overcommit memory should never have been created in the first place.
I think you misunderstand the role overcommit plays in the issue. All it does is change which process gets arbitrarily killed when the unthinkable happens and the system exceeds the memory usage you planned for.
With overcommit, the OOM Killer uses a sophisticated algorithm to decide which process to kill. Without overcommit, the rule is simple: whatever next requests virtual memory (basically, malloc) dies. That process may be using very little memory, may be using the same amount it's used 100 times before without incident, and may be very important.
Oh, and with overcommit it's far less likely that anything at all will die.
There are systems (probably interactive, like the one you describe) where killing the next process to allocate memory is the least painful thing. There are plenty where it isn't.
Taming the OOM killer
Posted Feb 12, 2009 20:33 UTC (Thu) by xorbe (guest, #3165) []