How to manage DB Control 10.2 for RAC Database wit

:	395162.1	类型:	BULLETIN
	上次修订日期:	21-JUL-2008	状态:	PUBLISHED

In this Document
  
  
  
     
     
     
     
     
     
     
     
  

---

Applies to:

Enterprise Manager Grid Control - Version: 10.2.0.2 to 10.2.0.4
Information in this document applies to any platform.

Purpose

This bulletin will clarify the way to set up/configure/deconfigure DB Control 10.2.x.x for RAC database 10.2.x.x.
Examples of emca commands will be illustrated and the directory structure will be explained as well.

This bulletin is not intended to replace Oracle Documentation but to illustrate it.

For a complete overview of emca, please refer to the following documentation available at OTN:

Enterprise Manager Advanced Configuration (Available in both HTML and PDF format)
Chapter 1.2.6 Configuring Database Control During and After the Oracle Database 10g Installation
Topic 1.2.6.5 Using EMCA with Real Application Clusters

Scope and Application

• This article is intended for DBAs using DB Control to manage RAC databases 10.2.x.x.
• Before reading this article, the user must have a good understanding of RAC 10.2.x.x databases installation, configuration and functionalities
• This article is intended to illustrate the main emca commands to setup/configure DB Control for RAC datadase 10.2.x.x: installation/configuration/setup and usage of RAC database 10.2.x;x is beyond the scope of this article.
  Please refer to RAC documentation available on OTN for any RAC related topic.
  Section High Availability at
  

How to manage DB Control 10.2 for RAC Database with emca

Difference between DB Control for RAC 10.1.x.x and DB Control for RAC 10.2.x.x

As a reminder, DB Control 10.x.x.x includes 3 components:
- the dbconsole Management Service
- the dbconsole Management Agent
- the dbconsole Management Repository

In 10.1.x.x, when a DB Control was deployed on a RAC cluster with n nodes, a dbconsole was started on each node of the cluster. Each agent on each node reported to each dbconsole management service on the same node.

In 10.2.x.x, to improve performance and reduce the workload on the RAC database/instances,when a DB Control is first deployed on a RAC cluster with n nodes, a dbconsole is started only on the node from which the DB Control is deployed. Each agent on each node reports to that same unique dbconsole management service.
It is however possible to later reconfigure the DB Control to have more than one dbconsole started and to have agents reporting to several dbconsole.

Environment: RAC Database 10.2.0.2 with 3 instances running on a Cluster with 3 nodes

For the purpose of this article, we will illustrate DB Control on the following configuration
Cluster RAC 10.2.0.2 on linux Redhat 3.0 cluster crs with 3 nodes:
- node1.mycompany.com
- node2.mycompany.com
- node3.mycompany.com

RAC Database 10.2.0.2 EM102 using ASM with 3 instances:
- EM1021 running on node1.mycompany.com
- EM1022 running on node2.mycompany.com
- EM1023 running on node3.mycompany.com

The RAC database has been created manually without using dbca, therefore emca has not been run and there is no DB Control repository created in the RAC database.
The RAC database is not the hosting database for a Grid Control repository
This can be checked running the following SQL statement connected as a DBA user to the database:
SQL> select username from DBA_USERS where username = 'SYSMAN';
If the SQL Statement returns 'SYSMAN', it means that there is already a DB Control repository or a Grid Control repository present in the database.
To distinguish a Grid Control repository and a DB Control repository, you need to check the tablespaces. A DB Control repository is created with objects in SYSAUX tablespace. A Grid Control repository is created with objects in MGMT_TABLESPACE and MGMT_ECM_DEPOT_TS.
If
SQL> select tablespace_name from dba_tablespaces order by 1;

Returns MGMT_TABLESPACE and MGMT_ECM_DEPOT_TS among the tablespaces listed, it means that this database hosts a Grid Control repository. Therefore you cannot create a DB Control for that database.

If a Grid Control repository or a DB Control repository already exists and if you want to drop the existing repository because you are sure that this repository is decommissioned, you can use the following process:

To drop a DB Control repository or a to drop a Grid Control repository:
Note: This can be done from any node of the cluster, connected to whichever instance you prefer.

1. cd to RDBMS ORACLE_HOME/sysman/admin/emdrep/bin (for a DB Control Repository)
   or
   cd to the OMS ORACLE_HOME/sysman/admin/emdrep/bin (for a Grid Control Repository)
2. Issue the following command
   
   $ ./RepManager repository_host repository_port repository_SID
-sys_password password_for_sys_account -action drop

   * repository_host is the machine name where the Management Repository database is located
   * repository_port is the Management Repository database listener port address, usually 1521 or 1526
   * repository_SID is the Management Repository database system identifier
   * password_for_sys_account is the password of the SYS user for the database. For example, change_on_install.
   * -action drop indicates that you want to drop the Management Repository.
   
   Note: Dropping a Grid Control or DB Control repository quiesces and unquiesces the database.
   
   

Configure DB Control for a RAC database running 3 instances on a 3 RAC cluster node

Connect to and set the environment for the RDBMS RAC database ORACLE_HOME
ORACLE_HOME and ORACLE_SID must be set
ORACLE_HOME and ORACLE_HOME/bin are set in the environment variable $PATH

Run emca in interactive mode:

$ emca -config dbcontrol db -repos create -cluster
Enter the following information:
- Database unique name -- If you're not sure of the values for Database unique name
                          and service name, execute the following statement
                          connected as a DBA user to any instance of the RAC database:
                          SQL> show parameter db_unique_name
- Listener port
- SYS password
- SYSMAN password


or

Run emca in silent mode:
It is also possible to run emca in silent mode, providing a response file with all the parameters needed:

$emca -config dbcontrol db -repos create -cluster -si
lent -respfile /u01/app/oracle/admin/EM102/scripts/emca_configandrep.rsp

with content of file /u01/app/oracle/admin/EM102/scripts/emca_configandrep.rsp as follow:
CLUSTER_NAME=mycrsname -- if CLUSTER_NAME is not specified, will be the default cluster name (usually crs)
DB_UNIQUE_NAME=EM102
SERVICE_NAME=EM102
SYS_PWD=oracle
SYSMAN_PWD=oracle
DBSNMP_PWD=oracle
PORT=1521



This emca command will:
- create the DB Control repository in the RAC database
- configure the DB Control on the local node and deploy the DB Control on all nodes of the cluster
- start the DB Control on the local node (dbconsole and agent)
- start all the agents on all the other nodes of the cluster

The resulting sub-directories of the RDBMS ORACLE_HOME will be the same on each node of the cluster:

$ORACLE_HOME/node1.mycompany.com_EM1021
$ORACLE_HOME/oc4j/j2ee/OC4J_DBConsole_node1.mycompany.com_EM1021
$ORACLE_HOME/node2.mycompany.com_EM1022
$ORACLE_HOME/oc4j/j2ee/OC4J_DBConsole_node2.mycompany.com_EM1022
$ORACLE_HOME/node3.mycompany.com_EM1023
$ORACLE_HOME/oc4j/j2ee/OC4J_DBConsole_node3.mycompany.com_EM1023

On the cluster node node1.mycompany.com, the "active" sub-directories will be
$ORACLE_HOME/node1.mycompany.com_EM1021
$ORACLE_HOME/oc4j/j2ee/OC4J_DBConsole_node1.mycompany.com_EM1021

On the cluster node node2.mycompany.com, the "active" sub-directories will be
$ORACLE_HOME/node2.mycompany.com_EM1022
$ORACLE_HOME/oc4j/j2ee/OC4J_DBConsole_node2.mycompany.com_EM1022

On the cluster node node3.mycompany.com, the "active" sub-directories will be
$ORACLE_HOME/node3.mycompany.com_EM1023
$ORACLE_HOME/oc4j/j2ee/OC4J_DBConsole_node3.mycompany.com_EM1023

The other directories are just "witnesses" of the current DB Control configuration on the cluster.

Note: If you deploy another DB Control for another RAC database on the same cluster,
          you will have as well on each node of the cluster a new set of directories created under the
          RDBMS ORACLE_HOME.
          The format of the sub-directories is always hostnamei_SIDi.
          The active sub-directories on node i is always hostanmei_SIDi.

To check the configuration files, log files, targets.xml, contents of upload or recv directories
You must look only into the "active" directory.
This means that to check the file emd.properties of the agent on node1, you must login to node1 and look into $ORACLE_HOME/node1.mycompany.com_EM1021/sysman/config.
If you want to check the targets.xml of the agent on node3, you must login to node3 and look into or
Login to node3, set the environment variables
- ORACLE_HOME
- ORACLE_SID
- ORACLE_HOME and ORACLE_HOME/bin are set in the environment variable $PATH
- run

$ emctl config agent listtargets


Example of targets.xml file:



















































To check the DB Control configuration on the cluster:

$emca -displayConfig dbcontrol -cluster

INFO:

**************** Current Configuration ****************

INSTANCE NODE DBCONTROL_UPLOAD_HOST
---------- ---------- ---------------------
EM1021 node1 node1.mycompany.com
EM1022 node2 node1.mycompany.com
EM1023 node3 node1.mycompany.com


This table shows that:

• the db control is running on (DBCONTROL_UPLOAD_HOST)
• there are 3 instances monitored EM1021, EM1022 and EM1023 on nodes node1, node2 and node3 respectively.
• the agents on nodes node1, node2 and node3 are all reporting to the dbconsole running on node because emca was run from If emca had been run from all agents would have reported to
  

To check emca log files:

As displayed to the standard output, emca log files are create in $ORACLE_HOME/cfgtoollogs/emca/EM102.
The log files are only created on the local node (the node from which emca has been run).
The log file for DBControl configuration/setup and deployment is emca_.log
The log file for DB Control repository creation is emca_repos_create_.log

For more information on emca log files, please refer to:
How To Trace EMCA in RDBMS 10G

Login to DB Control

At the end of the deployment, emca gives the URL to login to the dbconsole.
This URL is also available in the dbconsole log file.
This URL is also available in the file $ORACLE_HOME/install/readme.txt
The Console ports and Agent ports can be checked in the file $ORACLE_HOME/install/portlist.ini

Reconfigure DB Control to have 2 dbconsole started and running to manage a RAC database running 3 instances on a 3 RAC cluster node

This does not really make sense for a 3 nodes RAC cluster, but it is possible to have several dbconsole running instead of one, and agents configured to report to different dbconsole.

For example we want to have a dbconsole running on node1 and node2 with agents on node1 reporting to the dbconsole on node1 and agent on node2 and node3 reporting to the dbconsole on node2.

This emca command can be run from any node in the cluster.

According to the documentation and the prompt help we would then run:

$ emca -reconfig dbcontrol -cluster -EM_NODE node2 -EM_SID_LIST EM1022,EM1023

or

$ emca -reconfig dbcontrol -cluster

Enter the following information:
Database unique name: EM102
Database Control node name (optional): node2
Agent SID list [comma separated] (optional): EM1022,EM1023


or

$ emca -reconfig dbcontrol -cluster -silent -respfile /u01/app/oracle/admin/EM102/scripts/emca_node2_23.rsp
With content of emca_node2_23.rsp as follow
DB_UNIQUE_NAME=EM102
EM_NODE=node2
EM_SID_LIST=EM1022,EM1023


This command will fail however with the error:
13-Oct-2006 15:23:48 oracle.sysman.emcp.EMDBPreConfig performDbcReconfiguration
SEVERE: Invalid value EM1022,EM1023 for parameter EM_SID_LIST
13-Oct-2006 15:23:48 oracle.sysman.emcp.EMConfig perform
SEVERE: Invalid value EM1022,EM1023 for parameter EM_SID_LIST
Refer to the log file at /u01/app/oracle/product/10.2.0/db/cfgtoollogs/emca/E
M102/emca_2006-10-13_03-22-29-PM.log for more details.

This issue has been logged in the bug:
EMCA FAILS WHEN SEVERAL SIDS ARE WRITTEN IN EM_SID_LIST PARAMETER
EM_SID_LIST accepts only one SID and fails to parse multiple SID separated by a comma.

The workaround is to reconfigure the DB Control for each SID.

$ emca -reconfig dbcontrol -cluster -EM_NODE node2 -EM_SID_LIST EM1022

$emca -displayConfig dbcontrol -cluster

INFO:

**************** Current Configuration ****************

INSTANCE NODE DBCONTROL_UPLOAD_HOST
---------- ---------- ---------------------
EM1021 node1 node1.mycompany.com
EM1022 node2 node2.mycompany.com
EM1023 node3 node1.mycompany.com


Then


$ emca -reconfig dbcontrol -cluster -EM_NODE node2 -EM_SID_LIST EM1023

$emca -displayConfig dbcontrol -cluster

INFO:

**************** Current Configuration ****************

INSTANCE NODE DBCONTROL_UPLOAD_HOST
---------- ---------- ---------------------
EM1021 node1 node1.mycompany.com
EM1022 node2 node2.mycompany.com
EM1023 node3 node2.mycompany.com

Remove an instance from DB Control monitoring

If you plan to remove an instance from the RAC database, you must first remove the instance from DB Control monitoring.
You can also remove an instance from DB Control monitoring simply because you are not interested to monitor this particular instance.
Removing an instance from DB Control monitoring does not remove the instance from the RAC database.

This command can be run from any node in the cluster, except from the node from where runs the instance for which we want to stop the monitoring.
Please refer to:
- emca -deleteInst db fails with Database Instance unavailable

For example if we want to stop monitoring the instance on node2, we can run the following command either from node1 or from node3:

$ emca -deleteInst db
Enter the following information:
Node name: node2
Database unique name: EM102
Database SID: EM1022



The command will:
- update the repository to remove all rows related to the instance
- remove all sub-directories related to DB Control on the node specified in node name (node2 here)
  - here the following sub-directories will be removed on node2
    $ORACLE_HOME/node1.mycompany.com_EM1021
    $ORACLE_HOME/oc4j/j2ee/OC4J_DBConsole_node1.mycompany.com_EM1021
    $ORACLE_HOME/node2.mycompany.com_EM1022
    $ORACLE_HOME/oc4j/j2ee/OC4J_DBConsole_node2.mycompany.com_EM1022
    $ORACLE_HOME/node3.mycompany.com_EM1023
    $ORACLE_HOME/oc4j/j2ee/OC4J_DBConsole_node3.mycompany.com_EM1023
- remove all sub-directories related to the instance EM1022 on all the other nodes of the cluster
  - here the following sub-directories will be removed on node1 and node3
    $ORACLE_HOME/node2.mycompany.com_EM1022
    $ORACLE_HOME/oc4j/j2ee/OC4J_DBConsole_node2.mycompany.com_EM1022
- reconfigure the agents which were reporting to the dbconsole removed if needed. This is what happened in our example:

• here we stopped monitoring the instance EM1022, which dropped the dbconsole on node2
•  however the agent on node3 was reporting to this dbconsole; emca has also reconfigured the agent on node3 to upload to the dbconsole on node1.


$emca -displayConfig dbcontrol -cluster

INFO:

**************** Current Configuration ****************

INSTANCE NODE DBCONTROL_UPLOAD_HOST
---------- ---------- ---------------------
EM1021 node1 node1.mycompany.com
EM1023 node3 node1.mycompany.com

Log files
The log files are only created on the local node (the node from which emca has been run).
emca log files for an instance operation (delete instance) are logged in:
RDBMS $ORACLE_HOME/cfgtoollogs/emca/EM102/EM1022
A new sub-directory is created, if it does not exists, to log emca operations at instance level, named with the SID on which the operation was performed.

Add an instance to DB Control monitoring

If you have added an instance to the RAC database and if you plan to monitor it with the DB Control, you must add the instance to DB Control monitoring.

$ emca -addInst db
Enter the following information:
Node name: node2
Database unique name: EM102
Database SID: EM1022



This emca command will:
- update the repository to add all rows related to the instance
- add all sub-directories related to DB Control on the node specified in node name (node2 here)
- here the following sub-directories will be created on node2
  $ORACLE_HOME/node1.mycompany.com_EM1021
  $ORACLE_HOME/oc4j/j2ee/OC4J_DBConsole_node1.mycompany.com_EM1021
  $ORACLE_HOME/node2.mycompany.com_EM1022
  $ORACLE_HOME/oc4j/j2ee/OC4J_DBConsole_node2.mycompany.com_EM1022
  $ORACLE_HOME/node3.mycompany.com_EM1023
  $ORACLE_HOME/oc4j/j2ee/OC4J_DBConsole_node3.mycompany.com_EM1023
- all sub-directories related to the instance EM1022 on all the nodes of the cluster
- here the following sub-directories will be added on node1 and node3
  $ORACLE_HOME/node2.mycompany.com_EM1022
  $ORACLE_HOME/oc4j/j2ee/OC4J_DBConsole_node2.mycompany.com_EM1022

$emca -displayConfig dbcontrol -cluster

INFO:

**************** Current Configuration ****************

INSTANCE NODE DBCONTROL_UPLOAD_HOST
---------- ---------- ---------------------
EM1021 node1 node1.mycompany.com
EM1022 node2 node1.mycompany.com
EM1023 node3 node1.mycompany.com

Log files
The log files are only created on the local node (the node from which emca has been run).
emca log files for an instance operation (add instance) are logged in:
RDBMS $ORACLE_HOME/cfgtoollogs/emca/EM102/EM1022
A new sub-directory is created, if it does not exist, to log emca operations at instance level, named with the SID on which the operation was performed.

Drop DB Control keeping the repository

$ emca -deconfig dbcontrol db -cluster
Enter the following information:
- Database unique name


This emca command does the following:
- stop the DB Control (dbconsole, agent) on all nodes of the cluster
- remove all DB Control related directories on all nodes of the cluster
  $ORACLE_HOME/node1.mycompany.com_EM1021
  $ORACLE_HOME/oc4j/j2ee/OC4J_DBConsole_node1.mycompany.com_EM1021
  $ORACLE_HOME/node2.mycompany.com_EM1022
  $ORACLE_HOME/oc4j/j2ee/OC4J_DBConsole_node2.mycompany.com_EM1022
  $ORACLE_HOME/node3.mycompany.com_EM1023
  $ORACLE_HOME/oc4j/j2ee/OC4J_DBConsole_node3.mycompany.com_EM1023

The repository has not been dropped.
You can then if needed recreate DBControl without recreating the repository.

Known issue
emca completes successfully but with the following warning:

06-Oct-2006 17:57:05 oracle.sysman.emcp.EMReposConfig stopDBMSJobs
WARNING: Error initializing SQL connection. SQL operations cannot be performed
06-Oct-2006 17:57:05 oracle.sysman.emcp.EMReposConfig invoke
WARNING: Unable to remove DBMS jobs.
Enterprise Manager configuration completed successfully
FINISHED EMCA at 06-Oct-2006 18:06:20

This issue has been logged in the internal bug (not visible through METALINK):
Bug: WHILE DECONFIG DBCONTROL FROM RAC DB, UNABLE TO REMOVE DBMS JOBS

Example: DB Control deconfiguration removing the repository

$ emca -deconfig dbcontrol db -repos drop -cluster
Enter the following information:
Database unique name: EM102
Listener port number: 1521
Password for SYS user:
Password for SYSMAN user:


This emca command does the following:

- stop the DB Control (dbconsole, agent) on all nodes of the cluster
- remove all DB Control related directories on all nodes of the cluster   
   $ORACLE_HOME/node1.mycompany.com_EM1021    
   $ORACLE_HOME/oc4j/j2ee/OC4J_DBConsole_node1.mycompany.com_EM1021 
   $ORACLE_HOME/node2.mycompany.com_EM1022 
   $ORACLE_HOME/oc4j/j2ee/OC4J_DBConsole_node2.mycompany.com_EM1022 
   $ORACLE_HOME/node3.mycompany.com_EM1023
   $ORACLE_HOME/oc4j/j2ee/OC4J_DBConsole_node3.mycompany.com_EM1023
- drop the repository
  The command (one line) launched by emca to drop the repository once the DB Control is dropped
   is:
   /oracle/app/oracle/product/10.2.0/db/sysman/admin/emdrep/bin/RepManager -connect (DESCRIPTION=
  ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=node1vip)
  (PORT=1521))(ADDRESS=(PROTOCOL=TCP)(HOST=ukp79142vip)(PORT=1521))
  (ADDRESS=(PROTOCOL=TCP)(HOST=node3vip)(PORT=1521))(LOAD_BALANCE=yes))
  (CONNECT_DATA=(SERVICE_NAME=EM102)))
  -repos_user SYSMAN -action drop -verbose
  -output_file /oracle/app/oracle/product/10.2.0/db/cfgtoollogs/emca/EM102/emca_repos_drop_2006-10-13_01-03-03-PM.log

Log files
The log files are only created on the local node (the node from which emca has been run).
The log file for DBControl configuration/setup and deployment is emca_timestamp.log
The log file for DB Control repository drop is emca_repos_drop_timestamp.log

References

- How to Trace / Debug the EMCA Tool in 10g and 11g
- Problem: emca -deleteInst db fails with Database Instance unavailable
Topic "Using EMCA with Real Application Clusters"