libclntsh.so: wrong ELF class: ELFCLASS64

RedHat 5.5 x86_64
10.2.0.1升级到10.2.0.3

启动时报CRS-0223,CRS-0215

[oracle@node1 racg]$ crs_start -all
node1 : CRS-1019: Resource ora.node2.ons (application) cannot run on node1
CRS-0223: Resource 'ora.node1.ons' has placement error.
CRS-0215: Could not start resource 'ora.node2.ons'.

看日志

[oracle@node2 racg]$ less ora.node2.ons.log
2012-02-09 22:31:26.175: [    RACG][2055623360] [19636][2055623360][ora.node2.ons]: libclntsh.so dlopen failed:
libclntsh.so: wrong ELF class: ELFCLASS64
Unable to open libclntsh.so
libclntsh.so dlopen failed:
libclntsh.so: wrong ELF class: ELFCLASS64
libclntsh.so dlopen failed:
libclntsh.so: wrong ELF class: ELFCLASS64
Unable to open li

修改vip报错

[root@node2 ~]# /u01/app/oracle/product/10.2.0/crs/bin/vipca
PRKH-1010 : Unable to communicate with CRS services.
[PRKH-1000 : Unable to load the SRVM HAS shared library
[PRKN-1008 : Unable to load the shared library "srvmhas10" or a dependent library, from
LD_LIBRARY_PATH="/u01/app/oracle/product/10.2.0/crs/jdk/jre/lib/i386/client:/u01/app/oracle/product/10.2.0/crs/jdk/jre/lib/i386:/u01/app/oracle/product/10.2.0/crs/jdk/jre/../lib/i386:/u01/app/oracle/product/10.2.0/crs/lib32:/u01/app/oracle/product/10.2.0/crs/srvm/lib32:/u01/app/oracle/product/10.2.0/crs/lib:/u01/app/oracle/product/10.2.0/crs/srvm/lib:"
[java.lang.UnsatisfiedLinkError: 
/u01/app/oracle/product/10.2.0/crs/lib32/libsrvmhas10.so: 
libclntsh.so.10.1: wrong ELF class: ELFCLASS64]]]
PRKH-1010 : Unable to communicate with CRS services.

查看，已经使用的是64位的库(在linux和unix操作系统里用来显示一个执行文件或共享库的动态依赖信息的。)

[oracle@node2 ~]$ ldd `which sqlplus`
libsqlplus.so => /u01/app/oracle/product/10.2.0/db_1/lib/libsqlplus.so (0x00002b6a7d201000)
libclntsh.so.10.1 => /u01/app/oracle/product/10.2.0/db_1/lib/libclntsh.so.10.1 (0x00002b6a7d3f1000)
libnnz10.so => /u01/app/oracle/product/10.2.0/db_1/lib/libnnz10.so (0x00002b6a7e86b000)
libdl.so.2 => /lib64/libdl.so.2 (0x0000003527400000)
libm.so.6 => /lib64/libm.so.6 (0x0000003527c00000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x0000003527800000)
libnsl.so.1 => /lib64/libnsl.so.1 (0x000000352b000000)
libc.so.6 => /lib64/libc.so.6 (0x0000003527000000)
/lib64/ld-linux-x86-64.so.2 (0x0000003526c00000)

[ID 342750.1]

$ rpm -qa |grep glibc-2
glibc-2.3.4-2.13
compat-glibc-2.3.2-95.30
compat-glibc-2.3.2-95.30
glibc-2.3.4-2.13
$ rpm -qa |grep glibc-devel
glibc-devel-2.3.4-2.13
glibc-devel-2.3.4-2.13

32/64位都装，可以使用下面这个命令检测

rpm -qa --queryformat "%{NAME}-%{VERSION}-%{RELEASE} (%{ARCH})\n"|grep glibc-devel

方法一
1. Login as root and shutdown crs on the node having problem.
      # crsctl stop crs
2. Backup files under $ORA_CRS_HOME/lib and $ORA_CRS_HOME/lib32
3. Log in as root user.
4. Set ORACLE_HOME to $ORA_CRS_HOME ie:
      # export ORACLE_HOME=/
5.  # cd $ORACLE_HOME/network/lib
6.  # make -f ins_net_client.mk client_sharedlib
7. For files under the $ORA_CRS_HOME/lib32 directory, Change owner, group, permission on files generated by step 6 to oracle:oinstall, 755 (check files generated by checking timestamp of files).
8. Check if symbolic link libclntsh.so pointing to $CRS_HOME/lib/libclntsh.so.10.1 exists under  $ORA_CRS_HOME/lib as before. If not, create one.
9. Startup crs as root:
   # crsctl start crs
10. rerun vipca if needed
我第8步检查时发现libclntsh.so.10.1文件没了.我没测试成功
方法二
安装包后,重装CRS