RAC Transparent Application Failover (TAF)

 10g & 11g :Configuration of TAF(Transparent Application Failover) and Load Balancing (文档 ID 453293.1) 

In this Document

Goal

Solution

CONCEPTS

CONFIGURATION

Client Side Load Balancing

Connect Time Failover

Client Side Load Balance & Connect-time Failover Connect String

TAF - Client Side Configuration

TAF - Server Side Configuration

Troubleshooting TAF

References

---

APPLIES TO: 

Oracle Net Services - Version 10.2.0.1 to 11.2.0.4 [Release 10.2 to 11.2]
Information in this document applies to any platform.
***Checked for relevance on 16-OCT-2013***

GOAL

This note explains the configuration in 10g & 11g  for :

1) Client Side Connect Time Load Balance
2) Client Side Connect Time Failover
3) Server Side Listener Connection Load Balance 
4) Transparent Application Failover (TAF)

This note does not apply to 11gR2 SCAN configurations.

SOLUTION

 CONCEPTS

(1) Client Side Connect-Time Load Balance 

The client load balancing feature enables clients to randomize connection requests among the listeners. 

Tnsnames Parameter: LOAD_BALANCE 

The (load_balance=yes) instructs SQLNet to progress through the list of listener addresses in a random sequence, balancing the load on the various listeners. When set to OFF, instructs SQLNet to try the addresses sequentially until one succeeds. 

Load balancing can be specified for an ADDRESS_LIST or associated with a set of ADDRESSes or set DESCRIPTIONs. 

This parameter must be correctly coded in your net service name (connect descriptor). 
By default, this parameter is set to ON for DESCRIPTION_LISTs. 

If you use ADDRESS_LIST, (load_balance=yes) should be within the (ADDRESS_LIST=) portion. If you do not use ADDRESS_LIST, (load_balance=yes) should be within the (description=) portion. 

(2) Client Side Connect-Time failover 

The connect-time failover enables clients to connect to another listener if the initial connection to the first listener fails. The number of listener protocol addresses determines how many listeners are tried. Without 
connect-time failover, Oracle Net attempts a connection with only one listener. The default is on. 

Tnsnames Parameter: FAILOVER 

(failover=on) is default for ADDRESS_LISTs, DESCRIPTION_LISTs, and a set of 
DESCRIPTIONs., therefore, you do not have to specify it explicitly. 


(3) Server Side Listener Connection Load Balance 

The listener connection load balancing feature improves connection performance by balancing the number of active connections among multiple dispatchers and instances. In a single-instance environment, the listener selects the least loaded dispatcher to handle the incoming client requests. In an Oracle Real Application Clusters (RAC) environment, connection load balancing also has the capability to balance the number of active connections among multiple instances. 
1. Least-loaded node 
2. Least-loaded instance 
3. Least-loaded dispatcher for that instance (in case of Shared server configuration) 


There are two types of server-side load balancing: 

Load Based ??? Server side load balancing redirects connections by default depending on node load. From 10.2 onwards you can work this with Load balancing advisory (LBA). This will not be effective for login storms as the connections are already in the listener after which PMON update on the load comes to the listener. 
Session Based ??? Session based load balancing takes into account the number of sessions connected to each node and then distributes the connections to balance the number of sessions across the different nodes. 
Session count balancing is used when you set a listener parameter, PREFER_LEAST_LOADED_NODE_=OFF ( is the actual name of the listener which is different on each node in your cluster and by default is constructed as "listener_"). Please be aware that connections bursts (large number of connections initiated very quickly one after another) may not be properly balanced due to statistics update latency. 

(4) Transparent Application Failover (TAF) 

Transparent Application Failover (TAF) is a feature of the Oracle Call Interface (OCI) driver at client side. It enables the application to automatically reconnect to a database, if the database instance to which the connection is made fails. In this case, the active transactions roll back. 

Tnsnames Parameter: FAILOVER_MODE 

When an instance to which a connection is established fails or is shutdown, the connection on the client side becomes stale and would throw exceptions to the caller trying to use it. TAF enables the application to transparently reconnect to a preconfigured secondary instance creating a fresh connection, but identical to the connection that was established on the first original instance.

CONFIGURATION

 We will take a 2 node setup for this entire configuration example

Node1: 

Hostname: node1.idc.oracle.com  
VIP Hostname: node1-vip.idc.oracle.com  
Database Service_names: service.idc.oracle.com  
SID: sid1 


Node2:  

Hostname: node2.idc.oracle.com   
VIP Hostname: node2-vip.idc.oracle.com   
Database Service_names: service.idc.oracle.com   
SID: sid2

 

Note: Ensure that the node1-vip.idc.oracle.com and node2-vip.idc.oracle.com is resolvable & reachable from clients. If needed You may replace the hostnames with IP address in the entire configuration files.

Node1

Listener.ora

LISTENER_NODE1=   
(DESCRIPTION =   
 (ADDRESS_LIST =
  (ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC))   
  (ADDRESS = (PROTOCOL = TCP)(HOST = node1-vip.idc.oracle.com)(PORT = 1521)(IP=FIRST))
  (ADDRESS = (PROTOCOL = TCP)(HOST = node1.idc.oracle.com)(PORT = 1521)(IP=FIRST))     
 )   
)  

 Start the listener

$ lsnrctl start LISTENER_NODE1

Tnsnames.ora

NODE1_LOCAL= 
 (ADDRESS = (PROTOCOL = TCP)(HOST=node1-vip.idc.oracle.com)(PORT = 1521))     

 

Note: Make sure this tns alias NODE1_LOCAL is resolvable on the server, this ensures PMON picks up the entry without problem. 
The following command should succeed on node1 
$ tnsping node1_local

Set the local_listener for node1

sql > alter system set LOCAL_LISTENER='node1_local' scope=both sid='sid1' ;

 

Node2

Listener.ora

LISTENER_NODE2=    
(DESCRIPTION =   
 (ADDRESS_LIST =
 (ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC))            
 (ADDRESS=(PROTOCOL=TCP)(HOST=node2.idc.oracle.com)(PORT=1521)(IP=FIRST))   
 (ADDRESS=(PROTOCOL=TCP)(HOST=node2-vip.idc.oracle.com)(PORT=1521)(IP=FIRST))   
 )   
)

Start the listener

$ lsnrctl start LISTENER_NODE2

Tnsnames.ora

NODE2_LOCAL=  
   (ADDRESS = (PROTOCOL = TCP)(HOST = node2-vip.idc.oracle.com)(PORT = 1521)) 

 

Note: Make sure this tns alias NODE2_LOCAL is resolvable on the server, this ensures PMON picksup the entry without problem. 
The following command should succeed on node2 
$ tnsping node2_local

Set the local_listener for node2

sql > alter system set LOCAL_LISTENER='node2_local' scope=both sid='sid2' ;  

Enable the server side Load balancing (if required)
You should make the the listeners aware of adjacent nodes load to do the server side load balance.
To make PMON to notify the load information to adjacent nodes,  you should set the REMOTE_LISTENER parameter.

Add the following tnsalias in both the nodes tnsnames.ora 
Node1 & Node2 Tnsnames.ora

NODE_REMOTE = 
 (DESCRIPTION =      
  (ADDRESS = (PROTOCOL = TCP)(HOST=node1-vip.idc.oracle.com)(PORT = 1521))   
  (ADDRESS = (PROTOCOL = TCP)(HOST=node2-vip.idc.oracle.com)(PORT = 1521))    
 ) 

 

Note: Make sure this tns alias NODE_REMOTE is resolvable on all the servers, this ensures PMON picksup the entry without problem. 
The following command should succeed on all the nodes 
$ tnsping node_remote

Set the remote listeners to both the nodes

sql> alter system set REMOTE_LISTENER='node_remote' scope=both;

 Alternatively, set REMOTE_LISTENER to point the address string of the remote listener on the VIP:

SQL>alter system set REMOTE_LISTENER="(ADDRESS = (PROTOCOL = TCP)(HOST=node2-vip.idc.oracle.com)(PORT = 1521))"  scope=both   sid='instance_on_node1';

SQL>alter system set REMOTE_LISTENER="(ADDRESS = (PROTOCOL = TCP)(HOST=node1-vip.idc.oracle.com)(PORT = 1521))" scope=both sid='instance_on_node2';

Client Side Load Balancing

Following is an example of TNS net service alias for client side load balancing

CLIENT_LOADBALANCE=  
 (DESCRIPTION =  
  (LOAD_BALANCE = yes)  
  (ADDRESS = (PROTOCOL = TCP)(HOST = node1-vip.idc.oracle.com)(PORT = 1521))  
  (ADDRESS = (PROTOCOL = TCP)(HOST = node2-vip.idc.oracle.com)(PORT = 1521))  
  (CONNECT_DATA =  
   (SERVICE_NAME = service.idc.oracle.com)  
  )  
 )

Connect Time Failover

Sample connect time failover TNS net service alias
address_list defaults the failover=on

CLIENT_FAILOVER=  
(DESCRIPTION =  
 (ADDRESS_LIST =  
  (ADDRESS = (PROTOCOL = TCP)(HOST = node1-vip.idc.oracle.com)(PORT = 1521)) 
  (ADDRESS = (PROTOCOL = TCP)(HOST = node2-vip.idc.oracle.com)(PORT = 1521))  
 )  
 (CONNECT_DATA =  
   (SERVICE_NAME = service.idc.oracle.com)  
 )  
)

Client Side Load Balance & Connect-time Failover Connect String

CLIENT_LOAD_FAILOVER= 
(DESCRIPTION =  
 (ADDRESS_LIST =  
  (LOAD_BALANCE = yes)  
  (ADDRESS = (PROTOCOL = TCP)(HOST = node1-vip.idc.oracle.com)(PORT = 1521))  
  (ADDRESS = (PROTOCOL = TCP)(HOST = node2-vip.idc.oracle.com)(PORT = 1521))  
 )  
 (CONNECT_DATA =  
  (SERVICE_NAME = service.idc.oracle.com)  
 )  
) 

 

TAF - Client Side Configuration

Failover Mode Type can be Either SESSION or SELECT.
Session failover will have just the session to failed over to the next available node.The select query will not be resumed.

You can refer the below document for various options on TAF
Oracle?? Database Net Services Administrator's Guide
Chapter on Enabling Advanced Features of Oracle Net Services

Simple TAF can be configured on the client side tnsnames.ora with following entry

TAF= 
(DESCRIPTION =  
 (ADDRESS_LIST = 
  (LOAD_BALANCE = yes)  
  (ADDRESS = (PROTOCOL = TCP)(HOST = node1-vip.idc.oracle.com)(PORT = 1521))  
  (ADDRESS = (PROTOCOL = TCP)(HOST = node2-vip.idc.oracle.com)(PORT = 1521))  
 ) 
 (CONNECT_DATA =  
   (SERVICE_NAME = service.idc.oracle.com)  
   (failover_mode=(type=select)(method=basic))  
    )  
  ) 

TAF - Server Side Configuration

From Oracle 10.2 onwards, TAF can be configured with server side at the time of service creation also. 
Refer Note 460982.1 How To Configure Server Side Transparent Application 
Failover

Troubleshooting Load Balance

In case of server side load balancing not happening properly, to start diagnosing the issue you may collect the below set of information:

a. Is session load balance turned on ? 
b. What is the connection distribution on nodes ?
c. Is all the nodes are equal power w.r.t CPU / Memory / Hardware ?
d. Is there any application connecting locally (BEQueath connection) and doing big operations on any one of the node?
e. Are all the nodes are configured with proper settings of local / remote listeners?
Refer NOTE 263599.1 Understanding and Troubleshooting Instance Load Balancing

Troubleshooting TAF

In case of client connectivity failure the ideal start point to diagonise the issue is with Client Sqlnet tracing at support Level

To enable the client tracing  Refer 
NOTE 395525.1How to Enable Oracle SQLNet Client , Server , Listener , Kerberos and External procedure Tracing from Net Manager

While creating an Service Request with Oracle Support for TAF , provide the following information to diagonise the problem:

a. Failed connection client sqlnet tracing

b. Execute and provide the output of following commands from both the nodes:
    $ sql > show parameter Listener 
    $ sql> show parameter service
    $ lsnrctl  services

c. Upload the tnsnames.ora & Listener logs (by default created with .log) from both nodes.