分类： 系统运维
2009-03-23 09:35:33
There are many potential problems associated with the default configuration of mod_jk. While the default configuration may be adequate for a very low traffic website, when pushing any moderate to high load through mod_jk, there will be connection issues. This is not due to any bug in mod_jk, it is because the default configuration makes no assumption about your existing hardware or potential load, so, therefore, it is not tuned accordingly.
Note that the configuration recommendations here are to be used as an optimized base reference to avoid many of the common problems users experience with mod_jk. The optimizations here don't cover every single optimization but provide a solid base. See for details on all available mod_jk properties.
Let's take a look at a typical default configuration for Apache/Tomcat/mod_jk:
worker.list=loadbalancer,status
worker.node1.port=8009
worker.node1.host=node1.mydomain.com
worker.node1.type=ajp13
worker.node1.lbfactor=1
worker.node2.port=8009
worker.node2.host= node2.mydomain.com
worker.node2.type=ajp13
worker.node2.lbfactor=1
worker.loadbalancer.type=lb
worker.loadbalancer.balance_workers=node1,node2
worker.status.type=status
emptySessionPath="true" enableLookups="false" redirectPort="8443" />
StartServers 8
MinSpareServers 5
MaxSpareServers 20
ServerLimit 256
MaxClients 256
MaxRequestsPerChild 4000
The above configuration, under load, may cause mod_jk to be very slow and unresponsive, cause http errors, and cause half-closed connections. These problems can arise because there is no connection timeout specified to take care of 'orphaned' connections, no error handling properties defined in workers.properties, and no connection limits set in Apache and Tomcat.
First off, lets take care of Tomcat:
The main concern with server.xml is setting the connectionTimeout which sets the SO_TIMEOUT of the underlying socket. So when a connection in Tomcat hasn't had a request in the amount of time specified by connectionTimeout, then the connection dies off. This is necessary because if the connection hasn't been used for a certain period of time then there is the chance that it is half-close on the mod_jk end. If the connection isn't closed there will be an inflation of threads which can over time hit the maxThreads count in Tomcat then Tomcat will not be able to accept any new connections. A connectionTimeout of 600000 (10 minutes) is a good number to start out with. There may be a situation where the connections are not being recycled fast enough, in this instance the connectionTimeout could be lowered to 60000 or 1 minute.
The recommended value of maxThreads is 200 per CPU, so here we assume the server is a single core machine. If it had been quad core, we could push that value to 800, and more depending on RAM and other machine specs. The total threads is an aggregate value. If Apache and JBoss are on the same server, and that server has 4 cores, then you would half the maxThreads and MaxClients to 400 each. Always ensure the total threads possible doesn't exceed that 200 times the number of CPU cores.
emptySessionPath="true" enableLookups="false" redirectPort="8443"
maxThreads="200" connectionTimeout="600000"/>
See comments inline:
worker.list=loadbalancer,status
worker.template.port=8009
worker.template.type=ajp13
worker.template.lbfactor=1
#ping_mode was introduced in 1.2.27, if not using 1.2.27 please specify connect_timeout=10000 and prepost_timeout=10000 as an alternative
worker.template.ping_mode=A
worker.template.socket_timeout=10
#It is not necessary to specify connection_pool_timeout if you are running the worker mpm
worker.template.connection_pool_timeout=600
#Referencing the template worker properties makes the workers.properties shorter and more concise
worker.node1.reference=worker.template
worker.node1.host=192.168.1.2
worker.node2.reference=worker.template
worker.node2.host=192.168.1.3
worker.loadbalancer.type=lb
worker.loadbalancer.balance_workers=node1,node2
worker.loadbalancer.sticky_session=True
worker.status.type=status
The key points in the above workers.properties is we've added limits for the connections mod_jk makes. With the base configuration, socket timeouts default to infinite. The other important properties are ping_mode and connection_pool_timeout which must be set to equal server.xml's connectionTimeout when using the prefork mpm. When connectionTimeout and connection_pool_timeout are set, after a connection has been inactive for x amount of time, the connection in mod_jk and Tomcat will be closed at the same time, preventing a half-closed connection. Irregardless though, even if a connection issue happens, with the correct properties, like ping_mode=A set, then the bad connection will be detected and handled transparently.
Make note that maxThreads for the AJP connection should coincide with the MaxClients set in Apache's httpd.conf. MaxClients needs to be set in the correct module in Apache. This can be determined by running httpd -V. That is a capital V not a lowercase v.
# httpd -V
Server version: Apache/2.2.3
Server built: Sep 11 2006 09:43:05
Server's Module Magic Number: 20051115:3
Server loaded: APR 1.2.7, APR-Util 1.2.8
Compiled using: APR 1.2.7, APR-Util 1.2.7
Architecture: 32-bit
Server MPM: Prefork
threaded: no
forked: yes (variable process count)
Server compiled with....
-D APACHE_MPM_DIR="server/mpm/prefork"
-D APR_HAS_SENDFILE
-D APR_HAS_MMAP
-D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
-D APR_USE_SYSVSEM_SERIALIZE
-D APR_USE_PTHREAD_SERIALIZE
-D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
-D APR_HAS_OTHER_CHILD
-D AP_HAVE_RELIABLE_PIPED_LOGS
-D DYNAMIC_MODULE_LIMIT=128
-D HTTPD_ROOT="/etc/httpd"
-D SUEXEC_BIN="/usr/sbin/suexec"
-D DEFAULT_PIDLOG="logs/httpd.pid"
-D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
-D DEFAULT_LOCKFILE="logs/accept.lock"
-D DEFAULT_ERRORLOG="logs/error_log"
-D AP_TYPES_CONFIG_FILE="conf/mime.types"
-D SERVER_CONFIG_FILE="conf/httpd.conf"
Which tells me the Server MPM is Prefork. This is not always 100% accurate so you should also view the output of /etc/sysconfig/httpd to see if the following line is there: HTTPD=/usr/sbin/httpd.worker. If it is commented out you are running prefork, otherwise if uncommented worker.
From httpd.conf:
StartServers 8
MinSpareServers 5
MaxSpareServers 20
MaxClients 200
MaxRequestsPerChild 0
Or if Apache is using worker, it is:
StartServers 2
MaxClients 200
MinSpareThreads 25
MaxSpareThreads 75
ThreadsPerChild 25
MaxRequestsPerChild 0
MaxRequestsPerChild is 0, this is the recommended value when using mod_jk as mod_jk keeps open persistent connections. The key values in the above configuration are MaxClients and MaxRequestsPerChild, the rest of the values are left as default. Note that MaxRequestsPerChild is recommended to be 0 however the value may need to be greater than 0 depending on if Apache is used for other modules also, especially in the case of resource leakage. But for the vast majority of use cases, MaxRequestsPerChild should always be 0.
To get the most out of your mod_jk setup you should be using Apache's worker MPM which provides a definite performance improvement over the Prefork MPM. The following section will detail how to configure Apache/mod_jk/Tomcat with the worker MPM and the math behind the configuration.
Let's start out with the worker MPM configuration
ThreadLimit 100
StartServers 5
MaxClients 1000
MinSpareThreads 100
MaxSpareThreads 1000
ThreadsPerChild 100
MaxRequestsPerChild 0
The optimal configuration completely depends on the hardware being used and the load requirements. But a general rule of thumb, keep processes low and thread count high. To determine the number of processes Apache will use simply divide MaxClients by ThreadPerChild. So in this case MaxClients (1000) / ThreadsPerChild (100) = Processes (10), so Apache will allocate a maximum of 100 threads per each 10 child processes resulting in a total of 1000 possible clients.
Now to translate this to mod_jk, mod_jk maintains a connection pool for each worker defined in workers.properties. By default with Apache mod_jk sets connection_pool_size to ThreadsPerChild, so in the above case that would translate to 100, giving 1000 possible connections to JBoss. This may or may not be desired.
Let's take a common example, there will be 3 JBoss servers that combined needed to be able to handle 900 concurrent connections
worker.list=loadbalancer,status
worker.template.type=ajp13
worker.template.port=8009
worker.template.ping_mode=A
worker.template.connection_pool_size=30
worker.template.socket_timeout=10
worker.node1.reference=worker.template
worker.node1.host=192.168.0.101
worker.node2.reference=worker.template
worker.node2.host=192.168.0.102
worker.node3.reference=worker.template
worker.node3.host=192.168.0.103
worker.loadbalancer.type=lb
worker.loadbalancer.balance_workers=node1,node2, node3
worker.loadbalancer.sticky_session=True
worker.status.type=status
The above configuration tells mod_jk to multiplex 30 connections to the available Apache processes, which is 10 processes. So that means 30 connections multiplexed over 10 processes gives 300 possible connections to each backend worker. Furthermore, the total connections able to be used in this configuration from Apache is 900 which means 100 connections will be left over for static content or whatnot.
Next configure maxThreads in each AJP connector to match the above.
Nodes 1,2 and 3 AJP connectors:
emptySessionPath="true" enableLookups="false" redirectPort="8443"
maxThreads="300" connectionTimeout="600000" />
The last recommended setting is if the worker MPM is being used, then the following should be added to the httpd.conf file:
# Use a value of 600 with mod_jk 1.2.27 and a value of 60 with 1.2.28 and higher, there was a bug in 1.2.27 that calculated the value a tenth of the listed value
JkWatchdogInterval 60
Enabling the JkWatchdogInterval allows workers to be maintained periodically by a background thread. The maintenance includes checking for idle connections, correcting load status, and detecting back end health status. Note that JkWatchdogInterval was introduced and mod_jk 1.2.27 and will have no effect if used in prior versions.