Winodws下使用/3GB参数来调整Domino能够使用到的内存空间

最近在一个客户那边使用/3GB /PAE /Userva=2800来调整Domino已获得更多的内存寻址空间，但是在测试的过程中发现一些问题，导致服务器宕机。

在开启该参数后，并将测试客户端增加到9个的时候，每个客户端模拟560个用户去尝试发送邮件操作，测试发现内存使用情况是非线性增长的，在最初的1G的增长时间，仅仅使用了20分钟左右，然后每分钟增长1.8M左右。当增长到1.3G左右，以后的增加将非常的缓慢。

在后来增加测试用户数目后，Domino服务器出现不能响应和宕机的情况，后来根据宕机搜集到的日志来分析，得出如下的结论。

Ø         首先加入的/3GB参数已经起效。从nsd log中看到domino进程能够访问到0x7FFFFFFF以上的内存地址和象0xaefe0000这样高的内存地址。从memcheck看也显示了domino可用的内存限制达到了2.7GB，如下所示：

<@@ ------ System Data -> Performance Data -> Process Memory Mappings (summary) :: [ nserver:  0264] (Time 17:41:28) ------ @@>

COMMIT RESERVED    TOTAL

PRIVATE:   151.5M    98.1M   249.6M

MAPPED:     1.6G     4.6M     1.6G

IMAGE:    44.6M     0.0K    44.6M

TOTAL:     1.8G   102.7M     1.9G

Totals: Used= 1.9G, Free=847.7M, Fragmented=100.8M, Overall= 2.7G, maxFree= 0.0K, Limit=aefeffff ( 2.7G)

Ø     导致服务器宕机的原因是由于操作系统提示不能分配文件的句柄导致的服务器异常，该错误是操作系统产生的，最终Domino认为异常导致宕机，错误如下"(NT error = 1816): Not enough quota is available to process this command."

后来觉得这个问题涉及的东西太多，还要做很多调整，用户放弃了使用该参数。个人认为在以后的版本升级到8以后，如果采用64位的操作系统，这些问题都会迎刃而解。

有兴趣可以参考

  >    >  

Lotus Domino support for the Windows /3GB switch

 Technote (FAQ)

Question

On the Windows® 32-bit platform, heavily loaded Domino® servers, or servers with complex application designs may experience crashes due to high memory usage. More specifically, a specific process may reach the limit of its user address space. By default on Windows 32-bit, the users' address space is 2 gigabyte (GB) in size. This limit can be quickly reached on heavily loaded Domino servers.

Cause

The 4 GB address space limit is inherent to any 32-bit application. Installing more RAM, CPU or disk will not overcome this limit. The portion of the 4 GB address space given to the user mode (referred to as the user address space) is platform dependent. On W32, by default, the user address space is 2 GB, with the kernel reserving a separate 2 GB, resulting in a total of 4 GB of address space.

With ever increasing memory demands by complex Enterprise applications such as Domino, the 2 GB user address space can quickly become saturated, resulting in errors or crashes. This can become especially pronounced in cases where multiple enterprise components operate under one process, such as Domino, Oracle, and Java™. In these cases, it is possible to increase the user's address space size to 3 GB using the /3GB boot.ini switch for certain releases of Windows.

Does IBM® Lotus® recommends this course of action?

Answer

Before we provide the , it is important to provide the proper context for this discussion.

What does the /3GB switch do?

As mentioned above, on W32, the default size for the user address space is 2 GB, with the other 2 GB used as the kernel address space, for a total of 4 GB. In essence, you can think of the kernel address space as being shared across all processes (when they execute in kernel mode), with the same 2 GB being used by all. The user address space is mostly private to each process, unless the process chooses to allocate share-able memory.

On certain W32 platforms, it is possible to change the default percentages of the virtual address space by using the boot.ini parameter /3GB. Using the /3GB switch changes the split of the total virtual address space so that the user address space is increased to 3 GB, and the kernel address space is decreased to 1 GB. This setting in no way changes the overall virtual address space size (the process still only had 4 GB total), and it does not have an impact on how much virtual memory is managed by the system. The /3GB switch simply changes the split between user and kernel space for each process.

Below is a table that provides a list of W32 releases with which the /3GB switch is compatible. This information was taken from the Microsoft Article “Memory Support and Windows Operating Systems”:

W32 Versions that Support /3GB

Windows NT 4.0 Enterprise Edition

Windows 2000 Advanced Server

Windows 2000 Datacenter Server

Windows 2003 Server

Windows 2003 Enterprise Server

Windows 2003 Datacenter Server

Windows XP Professional

There are various implications when using this switch that must be considered, both application level (Domino) and kernel level (Operating System) factors:

Application (Domino) Level - The /3GB switch has its uses; however, even with this kernel-level setting enabled, a 32-bit process will not be able to use more than 2 GB user space unless it is compiled with the /LARGEADDRESSAWARE switch. This parameter must be specified when compiling the 32-bit executable.

If the kernel is configured with the /3GB setting, but an exe is not compiled with the Large Address Aware option, then the process’ user address space will still be restricted to 2 GB. Contrary to logic, the additional 1 GB is not given back to the kernel, but is instead flagged as a reserved block within the user address space. The result is that 1 GB of the virtual address space is lost. Hence, the /3GB switch should only be used if the process in question is compiled with /LARGEADDRESSAWARE. Misuse of the /3GB setting can cause more problems than it solves.

Beginning in Domino 7.0.1, the Domino server is compiled to be Large Address Aware, so that it can make use of the additional address space up to the full 3 GB. From an application-level perspective, this is all that is required to use the additional 1 GB of address space.

Kernel Level - The application level is not the only consideration. There is a far more important set of implications regarding kernel level performance. When the /3GB switch is used on W32, in actuality, one GB of the kernel's address space is taken, and given back to the user address space. This means the kernel's address space is cut in half.

While certain complex applications may need the extra room in their process address space, the drawback to using this switch is that the kernel must compress its memory usage into the remaining one GB. This will affect all processes system-wide, not only those that use the full 3 GB of user space, since it is the system that is booted into /3GB mode, and the kernel address space is shared across all processes. This can and will have substantial performance implications, particularly for certain kernel components such as the paged-pool, non-paged pool, and the System Page Table Entries (PTE).

Specifically, the use of /3GB will reduce the paged pool maximum size for Windows 2003 from 470 MB to below 200 MB. As default 80% page pool policy, the average paged pool would be around 160 MB, which is far below what is normally expected or desired, as Domino consumes a single 1 KB paged pool for each 1 MB database opening. In the worst scenario, Windows OS might log Event ID - 2020, or Domino might intermittently report the error "Hardware/OS error (Insufficient system resources exist to complete the requested service)."

In addition, when enabling the /3GB switch on Windows 32-bit, Lotus Domino performance testing indicates between a 10-15% increase in CPU usage (due to additional overhead from the constrained kernel). Domino testing indicates that this feature scales better under Windows 2003 than it does under Windows 2000.

/USERVA Switch

On Windows 2003, an additional switch is available to use in conjunction with the /3GB switch, known as the /userva switch. You can use /userva to tune the amount of the address space (in MB) that is given to user mode, using a value between 2048 MB (2 GB) and 3072 MB (3 GB). Since a given application may not require the full 3 GB space, but may perform well with 2.8 GB, this switch can be used and allow the remainder of the 3 GB to be returned to the kernel. This additional space for the kernel allows for an increase in the size of the PTE. For instance, you may specify a split of 2800 MB (2.73 GB) and 1296 MB (1.26 GB) with the following syntax:

/3GB /Userva=2800

Microsoft generally recommends that you keep the range of the /userva switch to between 2900 MB and 3030 MB. However, Domino performance testing indicates that a value of /userva=2800 works well. Again, this is only an option for Domino 7 and higher. For more information regarding the use of the /userva switch, see the following Microsoft article:

RAM vs. Address Space Size

Let's take a moment to stress one important point. The modification of the user and kernel address space sizes does NOT change the amount of RAM installed on the system. Furthermore, the relationship between RAM and the address space is not one-to-one. Pages accessed in the address space by a process can either exist in RAM, or the page file; the sum of these two media is referred to as virtual memory usage.

The fact that a process' user address space is increased to 3 GB does not mean that it will use 3 GB of RAM. It simply means that the process can allocate and access up to 3 GB of virtual memory. In fact, under the hood, the kernel makes the decision as to what data is paged out to disk, and what stays in RAM. The kernel is very conservative about what stays in RAM. So, in reality, very few pages in a process' user address space reside in RAM at any one time.

Another way to put it: the process cannot demand that it be given a certain amount of RAM. It is ultimately up to the kernel as to what stays in RAM and what gets paged out. In most instances, a process is not technically aware of which parts of its data are paged in and out of the page file. In short, the application knows no difference between RAM and page file. Therefore, we cannot directly discuss how much RAM a specific application requires, but in theory, only how much virtual memory it may use.

Misconceptions
You may be thinking: "OK, if I use the /3GB switch, and I have 8 GB of RAM, that should be 3 GB of RAM for Domino Partition #1, 3 GB of RAM for Domino Partition #2, leaving 2 GB of RAM for the kernel, right?"

Wrong.

It is important to refrain from drawing a one to one ratio between RAM and address space. While these concepts are related, they are not synonymous. Again, the majority of the user address space for a single process usually points to pages on disk at any given moment, with only a small portion of its address space pointing to RAM. The amount of RAM usage by a single process is not calculable in advance; it depends on process and system activity, and is ultimately decided by the kernel at runtime. The only way to get a rough figure of RAM usage within a particular environment is through a simulation of user load (as realistic as possible).

You might also be thinking: "OK, if I have less than 2 GB or RAM, how much RAM does the kernel get vs Domino? Does Domino get 3/4, and the kernel get 1/4, assuming I am using the /3GB switch?"

Again, you cannot draw a direct relationship between RAM and user address space. Even if you have less than 2 GB of RAM, the address space for the user is unchanged at 3 GB (assuming you are using the /3GB switch). The decreased amount of RAM just means that more if its address space points to the page file at any given moment, as controlled by the kernel.

The main point to take away from the above paragraphs is that the /3GB switch is used to adjust a process' address space, not its RAM usage, since low memory conditions reflect a limitation in address space rather than RAM usage.

The /PAE Switch

On Windows, by default the kernel can manage up to 4 GB of RAM (this does not include the page file). However, under certain versions of Windows, namely the Advanced Editions of 2000 and 2003, an administrator can increase this usage by using the setting /PAE within the boot.ini file. The /PAE (Physical Address Extension) allows the kernel to recognize and manage all of the installed RAM, up to 64 GB. The upper range depends on the exact version of Windows, but is no greater than 64 GB. Of course, in order to make use of the /PAE, it is necessary to have the correct memory hardware that contains the 36-address pin configuration.

The use of the /PAE setting does NOT affect the size of the virtual address space for each 32-bit application, but only affects the total amount of RAM that the kernel/CPU can manage system-wide.

To summarize:
• /3GB affects the virtual user address space of each process
• /PAE affects the amount of RAM that can be managed by the OS

The /PAE switch affects the amount of RAM that the CPU can address. The use of page file is not affected by this switch, since the page file is stored sequentially on disk. A system may commit far more than 4 GB of virtual memory (across all processes) without the /PAE switch enabled, where most of this memory usage is in the form of the page file, and where only a portion of RAM is in use. By its nature, the kernel can manage the full amount of RAM, and can allow the total virtual memory usage across all 32-bit applications to exceed far greater than 4 GB. In other words, you don’t need to have the /PAE switch enabled to commit more than 4 GB of total virtual memory (the sum of RAM and page file), but only if you want to make use of more than 4 GB of RAM across all processes.

Below is a table that provides RAM limits (using /PAE) for various versions of Windows Operating Systems. This information was taken from MSDN articles Q283037 and Q292934:

Windows 32-bit Version	Maximum RAM Support with /PAE
Windows XP Professional	4 GB
Windows 2000 Server	4 GB
Windows 2003 Server	4 GB
Windows 2000 Advanced Server	8 GB
Windows 2003 Enterprise Server	32 GB
Windows 2000 Datacenter Server	64 GB
Windows 2003 Datacenter Server	64 GB
Windows 2000/2003 Datacenter Server (with /3GB)	16 GB

/3GB vs. /PAE

The simultaneous use of /PAE and /3GB are to some degree at odds with each other. In order to support additional RAM, the /PAE switch requires more kernel space. However, when used in conjunction with the /3GB switch, the kernel actually has less address space to use. The net result is that a system running both /3GB and /PAE cannot address any more than 16 GB of memory at best, due to space considerations with the kernel. This limitation is described in MSDN article Q283037 titled “Large Memory Support is Available in Windows 2000” - it reads:

…A program that requests 3 GB of memory is more likely to be
able to have more of its memory remain in physical memory
rather than be paged out, which increases the performance of
programs that are capable of using the /3GB switch. The
exception is when the /3GB switch is used in conjunction with
the /PAE switch, the operating system does not use any memory
in excess of 16 GB. This behavior is caused by kernel virtual
memory space considerations. Thus, if the system restarts with
the /3GB entry in the Boot.ini file, and the system has more than
16 GB of physical memory, the additional physical random
access memory (RAM) is not used by the operating system…

IBM Lotus Domino recommendations

While the Domino 7 server is Large Address Aware, and while it may seem attractive to increase the address space of a Domino 7 server, Lotus recommends against the use of the /3GB switch, and recommends that you use this option only when there is no other alternative. Decreasing the kernel's address space can and will have a negative impact and essentially steals the kernel's resources to increase the user's resources. Since Domino 6 is not Large Address Aware, using the /3GB switch on Domino 6 has no affect, and only takes away the kernel's address space.

In cases where a server is crashing due to the 2 GB limit, there are various options available:

1. Use the /3GB Switch: As discussed above, this is the least desirable option, and is generally not recommended. If you use this option, consider using the /userva=2800 syntax. Due to its potentially constrained nature, it it strongly recommended that you test this configuration prior to implementing it in production.
   
   
2. Memory Tuning: Tune down memory usage, including reducing shared memory size, reducing thread counts, and redesigning applications and agents. In a number of cases, a defect or misconfiguration may be the cause for higher than expected memory usage, and action can be taken to remedy this situation. You should call into IBM Lotus Support for assistance on troubleshooting high memory usage.
   
   
3. Domino Partitioned Servers: In cases where no Domino defect is found or no further tuning or configuration changes can be made, another option is to spread the user load across multiple Domino partitions. Since each Domino partition has its own set of address spaces, dividing the application load across multiple partitions allows for significantly more virtual space inside which to maneuver (assuming that the Domino server is not originally hardware constrained). The same end can be accomplished through the use of Domino partitions on VMWare.
   
   
4. Windows 2003 64-bit: Another option is to run your 32-bit Domino Server on the Windows 2003 64-bit Operating System. In doing so, this moves the kernel's address space out of the 32-bit address space and into a separate 64-bit address space. This frees up the Domino 7 user address space to occupy nearly the entire 4 GB (32-bit) address space (since it is compiled as Large Address Aware). Since Domino 6 is not Large Address Aware by default, it still has a 2 GB user address space. See for additional caveats on W64.
   
   
5. Domino Native 64-bit: Once available, the Domino 8 Native 64-bit version will remove the 32-bit limitation on address space altogether, providing a much larger address space. However, this will not remove all Operating System limits or Domino limits, since these limits are enforced to maintain a balanced server. Don't forget, within the native 64-bit environment you may once again encounter your hardware limits (i.e. it will once again become theoretically possible for a single 64-bit process to exhaust system-wide RAM). For details on Domino 8 64-bit availability, see:
   
   (#1233872)
   
   (#1296452)

Related Microsoft Information

Under Windows 64-bit, it is no longer necessary to use either the /3GB switch or the /PAE switch, because, by default, the 64-bit Kernel can fully manage large amounts of RAM. In addition, as mentioned, each 32-bit process is able to use nearly all 4 GB in user mode (provided it is compiled as Large Address Aware).

Below is a table which has been taken and modified from Microsoft article #294418. For more information, please refer to the Microsoft article titled "Comparison of 32-bit and 64-bit memory architecture for 64-bit editions of Windows XP and Windows Server 2003" (see link below)

Windows Component	64-bit Windows	32-bit Windows
32-bit Total Address Space (per process)	4 GB	4 GB
32-bit User Address Space (per process)	4 GB (LAA) 2 GB (non-LAA)	3 GB (LAA with /3GB) 2 GB (default)
64-bit Total Address Space (per process)	16 terabytes	N/A
64-bit User Address Space (per process)	8 terabytes	N/A
Paging file size	512 terabytes	16 terabytes
Hyperspace	8 GB	4 MB
Paged pool	128 GB	470 MB
Non-paged pool	128 GB	256 MB
System cache	1 terabyte	1 GB
System PTEs	128 GB	660 MB

Comparison of 32-bit and 64-bit memory architecture for 64-bit editions of Windows XP and Windows Server 2003


Large memory support is available in Windows Server 2003 and in Windows 2000


How to use the /userva switch with the /3GB switch to tune the User-mode space to a value between 2 GB and 3 GB


Memory Support and Windows Operating Systems


Windows 2000 Datacenter Server Does Not Locate Memory Greater Than 16 GB


Use of the /3GB switch in Exchange Server 2003 on a Windows Server 2003-based system


Server is unable to allocate memory from the system paged pool