JkWatchdogInterval

Posted on 2008-11-03 04:04:00.0 by Mladen Turk [ View original post ]

The newest mod_jk/isapi_redirector version 1.2.27 contains both many bug fixes over the previous official stable version and many new features.

It's been quite a long time (in terms of standard release processes go) since last version was released. Version 1.2.26 was released almost a year ago, but since there were no serious bugs found that would require us to make a new tag, we decided to test the development version as much as possible hoping the new version will be as stable as the previous version was.

So, what's new?

1. Watchdog thread and connection probing

One of the major problems with mod_jk is also its major feature, and this is persistent connections between mod_jk and backend application server. A persistent connection pool offers a performance increase because there is no need to establish a new connection each time a new client request comes in. However, being persistent, once the connection is established, it will stay opened for the server lifetime. This can lead to lots of problems when the web server needs to handle a much higher connection load than the backend application server. In such situations the backend often refuses client requests although the number of concurrent requests that it serves is much lower then maximum allowed.

The standard way of coping with this problem is by using the connectionTimeout inside the AJP/1.3 Connector in Tomcat's server.xml that will, after some time, close the opened persistent connections that were inactive after the specified time period.

The new feature added to mod_jk is a so called watchdog thread that runs inside each web server's child process and monitors for the opened connections that were not used for a long period of time. Instead of relying on the socket timeout on the backend side the users can now setup the watchdog thread on web server side. This thread will close all connections that were not used for a time longer than the connection_pool_timeout.

The watchdog thread can be used for one additional new feature called connection probing. Connection probing sends a standard CPING packet across the unused opened connections in the connection pool thus offering problem detection for connections before the actual request is made. By sending the actual data over the connection it also helps to deal with firewalls that are set to close the inactive connections, making the entire system more robust.

How to enable these features:

For Apache Httpd you will have to add the

JkWatchdogInterval 60

to the main httpd.conf file. The numeric value is the interval in seconds the background thread will run. Choosing a lower value will make the system more responsive but will, on the other hand, require more CPU time doing maintenance.

For Microsoft IIS the new numeric key
has to be added to the registry (watchdog_interval) with the same value as explained for Apache Httpd.

To configure the connection probing the connection_ping_interval has
to be defined for each ajp13 worker inside workers.properties file.

worker.node1.type=ajp13
# Run connection probing each 2 minutes
worker.node1.connection_ping_interval=120
# The connect and prepost timeout has to be specified
# as well for connection probing to operate
worker.node1.connect_timeout=1000
worker.node1.prepost_timeout=1000

Another way of enabling the connection probing is by using a new ping_timeout parameter. The ping_timeout when set will be used as a timeout for all ping operations. The new ping_mode parameter determines what ping modes will be used

worker.node1.type=ajp13
# Set the ping timeout to 1 second
worker.node1.ping_timeot=1000
# Set the ping mode to A (all)
worker.node1.ping_mode=A

The connection_pool_interval, if not specified, will be 10 times the ping_timeout and in this example will be 10 seconds.


2. Uri extension mechanism

Another new feature added is the Uri extension mechanism which allows the setting up of worker states on uri bases. When the application server needs to redeploy one of its applications it was not previously possible to do this in the case when there were multiple backend servers in mod_jk. The only solution was to temporarily disable the entire mapping for that application causing the 404 (Not Found) to be returned to the users. With the extension mechanism it is possible to temporary disable the Uri mapping only for a specific member of the loadbalancer.
If, for example, we have a loadbalancer consisting of two application servers and we wish to redeploy a new version of application to one of the nodes, we can now add the mapping rule inside uriworkermap.properties file stating that.

/foo/*=mybalancer;stopped=node1

Now whenever a request comes in for /foo/, the node1 will never be selected in the loadbalancer. In this case the other members of the loadbalancer will handle the load for that Uri. When the /foo/ application is redeployed on the node1 the stopped=node1 extension can be removed from the uriworkermap.properties file and the application /foo/ will continue to get served from the node1 again.

There are many other options that could be set as extensions for a particular Uri mapping. The complete documentation and a few examples are given inside the uriworkermap reference file.

One particularly handy option is the use of reply_timeout depending on the application itself. Before 1.2.27 the reply_timeout was a global per worker option, but in real life the applications themselves have a timeout that counts. On one node application A can take
four times longer to execute than application B for example. This can be fine tuned by using the reply_timeout extension mechanism

/foo/*=myworker;reply_timeout=10000
/bar/*=myworker;reply_timeout=60000

The /foo/ application will now have to respond within 10 seconds while application /bar/ will have to respond within 60 seconds.

3. Full IIS 6+ support

Microsoft IIS with version 6 and up introduced multiple processes for IIS. This is similar to Apache Httpd worker mpm concept. Isapi redirector 1.2.27 has a full support for this. The internal shared memory, object naming and locking was done to cope with multiple worker processes inside IIS.
Chuncked encoding support was added as a compile time feature. This means if you wish to use the chunked encoding support for IIS you'll have to install the corresponding binary build
(unless you are building this by yourself).

There are few other feature added to the IIS, so consult the change log for a complete list.

4. A new way of handling Loadbalancer retries

With version 1.2.27 the loadbalancer now keeps an internal table of its member states during the member election process. This allows mod_jk to effectively select the proper member without
breaking existing connections.

Also the handling of the 'retries' parameter has been changed inside version 1.2.27. Now, the number of retries means exactly what it says. Each member of the loadbalancer will be retried the specified amount of times before giving up and returning and error to the client. The new property retry_interval which defaults to 100 milliseconds is used as the delay before executing each retry. This allows mod_jk to determine the exact timeout the client will wait even if the all members of loadbalancer are in error state.

For workers that have a connection pool instead, mod_jk now has a new parameter to to determine the time to wait on a free connection from the connection pool. The new parameter is named connection_acquire_timeout and it defaults to retries times retry_interval (to allow previous configurations to continue to work). This parameter determines how long the loadbalancer will wait for a free connection from the worker's connection_pool before giving up and electing another member.

5. Programmable session routes

The last notable feature allows one to set custom session routes, instead of the default JSESSIONID, for tracking the session affinity marks. JSESSIONID is defined by the Servlet specification, but many users choose to change that default value in case they have a mix between various application servers that cannot share their session data. In such cases the only solution is to change the session identifier and two new parameters allows one to specify the identifiers for each particular worker. Those parameters are session_cookie and session_path which are used, again, if a different name than JSESSIONID or ;jsessionid is needed.

So this is brief overview of a important new features added to the Tomcat Connectors version 1.2.27. We encourage you to test out these new features and give us your feedback and any problems (hopefully none) you run in to.