分类： 系统运维
2005-05-24 20:07:18
Squid requires a fair amount of kernel resources under moderate and high loads. In particular, you may need to configure your system with a higher-than-normal number of file descriptors and mbuf clusters. The file-descriptor limit can be especially annoying. You'd be better off to increase the limit before compiling Squid.
At this point, you might be tempted to get the precompiled binaries to avoid the hassle of building a new kernel.[1] Unfortunately, you need to make a new kernel, regardless. Squid and the kernel exchange information through data structures that must not exceed the set file-descriptor limits. Squid checks these limits at runtime and uses the safest (smallest) value. Thus, even if a precompiled binary has higher file descriptors than the kernel, the kernel value takes precedence.
Not all operating systems require building a new kernel. Some may be tunable at runtime.
To change some settings, you must build and install a new kernel. This procedure varies among different operating systems. Consult Unix System Administration Handbook (Prentice Hall) or your operating-system documentation if necessary. If you're using Linux, you probably don't need to recompile your kernel.
File descriptors are simply integers that identify each file and socket that a process has opened. The first opened file is 0, the second is 1, and so on. Unix operating systems usually impose a limit on the number of file descriptors that each process can open. Furthermore, Unix also normally has a systemwide limit.
Because of the way Squid works, the file-descriptor limits may adversely affect performance. When Squid uses up all the available file descriptors, it is unable to accept new connections from users. In other words, running out of file descriptors causes denial of service. Squid can't accept new requests until some of the current requests complete, and the corresponding files and sockets are closed. Squid issues a warning when it detects a file-descriptor shortage.
You can save yourself some trouble by making sure the file descriptor limits are appropriate before running ./configure. In most cases, 1024 file descriptors will be sufficient. Very busy caches may require 4096 or more. When configuring file descriptor limits, I recommend setting the systemwide limit to twice the per-process limit.
You can usually discover your system's file-descriptor limit from your Unix shell. All C shells and similar have the built-in limit command. Newer Bourne shells and similar have a command called ulimit. To find your file-descriptor limits, try running these commands:
csh% limit descriptors unlimited
csh% limit descriptors
descriptors 4096
or:
sh$ ulimit -n unlimited
sh$ ulimit -n
4096
On FreeBSD, you can also use the sysctl command:
% sysctl -a | grep maxfiles
kern.maxfiles: 8192
kern.maxfilesperproc: 4096
If you can't figure out the file-descriptor limit, Squid's ./configure script can do it for you. When you run ./configure, as described in Section 3.4, watch for output like this near the end:
checking Maximum number of file descriptors we can open... 4096
If either limit, ulimit, or ./configure report a value less than 1024, you should invest the time to increase the limit before compiling Squid. Otherwise, Squid's performance will be poor under a moderate load.
Increasing the file descriptor limit varies from system to system. The following sections offer some tips to help get you started.
Edit your kernel configuration file, and add a line like this:
options MAXFILES=8192
On OpenBSD, use option instead of options. Then, configure, compile, and install the new kernel. Reboot your system so the change takes effect.
Configuring file descriptors on Linux is a little complicated. You must edit one of the system include files, and execute some shell commands before compiling and running Squid. Start off by editing the file /usr/include/bits/types.h. Change the value for _ _FD_SETSIZE as follows:
#define _ _FD_SETSIZE 8192
Next, increase the kernel file descriptor limit with this command:
# echo 8192 > /proc/sys/fs/file-max
Finally, increase the process file-descriptor limit in the same shell in which you will configure and compile Squid:
sh# ulimit -Hn 8192
This command must be executed as root and only works from the bash shell. There is no need to reboot on Linux.
|
Add this line to your /etc/system file:
set rlim_fd_max = 4096
Then, reboot the system for the change to take effect.
The BSD-based networking code uses a data structure known as an mbuf (see W.R.Stevens' book, TCP/IP Illustrated, Vol 2). Mbufs are typically small (e.g., 128 octets) chunks of memory. The data for larger network packets are stored in mbuf clusters. The kernel may enforce an upper limit on the total number of mbuf clusters available in the system. You can find this limit with the netstat command:
% netstat -m
196/6368/32768 mbufs in use (current/peak/max):
146 mbufs allocated to data
50 mbufs allocated to packet headers
103/6182/8192 mbuf clusters in use (current/peak/max)
13956 Kbytes allocated to network (56% of mb_map in use)
0 requests for memory denied
0 requests for memory delayed
0 calls to protocol drain routines
In this example, there are 8,192 mbuf clusters available, but there are never more than 6,182 used at once. When the system runs out of mbuf clusters, I/O routines such as read( ) and write( ) return the "No buffer space available" error message.
NetBSD and OpenBSD don't display mbuf usage in netstat -m output. Instead, they report "WARNING: mclpool limit reached" via syslog.
To increase the number of mbuf clusters, you need to add an option to your kernel configuration file:
options NMBCLUSTERS=16384
Ephemeral ports are the local port numbers the TCP/IP stack assigns to outgoing connections. In other words, when Squid makes a connection to an origin server, the kernel assigns a port number to the local socket. These local port numbers fall within a certain range. On FreeBSD, for example, the default ephemeral port range is 1024-5000.
A shortage of ephemeral ports may adversely affect performance for very busy proxies (i.e., hundreds of requests per second). This is because some TCP connections enter a TIME_WAIT state when they are closed. An ephemeral port number can't be reused while the connection is in the TIME_WAIT state.
You can see how many connections are in this state with the netstat command:
% netstat -n | grep TIME_WAIT
Proto Recv-Q Send-Q Local Address Foreign Address (state)
tcp4 0 0 192.43.244.42.19583 212.67.202.80.80 TIME_WAIT
tcp4 0 0 192.43.244.42.19597 202.158.66.190.80 TIME_WAIT
tcp4 0 0 192.43.244.42.19600 207.99.19.230.80 TIME_WAIT
tcp4 0 0 192.43.244.42.19601 216.131.72.121.80 TIME_WAIT
tcp4 0 0 192.43.244.42.19602 209.61.183.115.80 TIME_WAIT
tcp4 0 0 192.43.244.42.3128 128.109.131.47.25666 TIME_WAIT
tcp4 0 0 192.43.244.42.3128 128.109.131.47.25795 TIME_WAIT
tcp4 0 0 192.43.244.42.3128 128.182.72.190.1488 TIME_WAIT
tcp4 0 0 192.43.244.42.3128 128.182.72.190.2194 TIME_WAIT
Note that this example has both client- and server-side connections. Client-side connections have 3128 as the local port number; server-side connections have 80 as the remote (foreign) port number. The ephemeral port numbers appear under the Local Address heading. In this example, they are in the 19,000s.
Unless you see thousands of ephemeral ports in the TIME_WAIT state, you probably don't need to increase the range. On FreeBSD, you can increase the range with this command:
# sysctl -w net.inet.ip.portrange.last=30000
On OpenBSD, the command is almost the same, but the sysctl variable has a different name:
# sysctl -w net.inet.ip.portlast=49151
On NetBSD, things work a little differently. The default range is 49,152-65,535. To increase the range, change the lower limit:
# sysctl -w net.inet.ip.anonportmin=10000
On Linux, simply write a pair of numbers to the following special file:
# echo "1024 40000" > /proc/sys/net/ipv4/ip_local_port_range
Don't forget to add these commands to your system startup scripts so that they take effect each time your machine reboots.
