linux下安装CDH5

环境:

OS:Rad Hat Linux As5

CDH:5.3.2

1.安装步骤

1.1 节点角色分配

节点角色	ip	hostname
主节点	192.168.56.111	host01
从节点	192.168.56.112	host02

 

 

这里只有两个节点的环境，一个做主节点，另外一个做从节点

1.2配置hosts文件

 每台机器分配了ip和设定了主机名后,需要在hosts文件下添加如下项

 [root@host01 /]# more /etc/hosts

 # Do not remove the following line, or various programs

 # that require network functionality will fail.

 127.0.0.1               localhost.localdomain localhost

 ::1             localhost6.localdomain6 localhost6

 192.168.56.111   host01.com        host01

 192.168.56.112   host02.com        host02

 

1.3 建立等效连接

 [root@host01 ~]$ cd ~

 [root@host01 ~]$ mkdir ~/.ssh

 [root@host01 ~]$ chmod 700 ~/.ssh

 [root@host01 ~]$ ssh-keygen -t rsa

 Generating public/private rsa key pair.

 Enter file in which to save the key (/home/root/.ssh/id_rsa):

 Enter passphrase (empty for no passphrase):

 Enter same passphrase again:

 Your identification has been saved in /home/root/.ssh/id_rsa.

 Your public key has been saved in /home/root/.ssh/id_rsa.pub.

 The key fingerprint is:

 84:88:f0:dc:7c:a6:d0:e8:68:65:53:97:35:02:05:fb

 [root@host01 ~]$ ssh-keygen -t dsa

 Generating public/private dsa key pair.

 Enter file in which to save the key (/home/root/.ssh/id_dsa):

 Enter passphrase (empty for no passphrase):

 Enter same passphrase again:

 Your identification has been saved in /home/root/.ssh/id_dsa.

 Your public key has been saved in /home/root/.ssh/id_dsa.pub.

 The key fingerprint is:

 94:80:60:37:38:e6:aa:10:22:50:1a:c2:42:56:22:0a

 

[root@host01 .ssh]$ chmod 644 ~/.ssh /authorized_keys

[root@host02 ~]$ cd ~
[root@host02 ~]$ mkdir ~/.ssh
[root@host02 ~]$ chmod 700 ~/.ssh

[root@host02 ~]$ ssh-keygen -t rsa

Generating public/private rsa key pair.

Enter file in which to save the key (/home/root/.ssh/id_rsa):

Enter passphrase (empty for no passphrase):

Enter same passphrase again:

Your identification has been saved in /home/root/.ssh/id_rsa.

Your public key has been saved in /home/root/.ssh/id_rsa.pub.

The key fingerprint is:

00:ee:68:42:3a:16:a3:75:6c:d4:36:49:57:a5:5c:57 root@host02

[root@host02 ~]$ ssh-keygen -t dsa

Generating public/private dsa key pair.

Enter file in which to save the key (/home/root/.ssh/id_dsa):

Enter passphrase (empty for no passphrase):

Enter same passphrase again:

Your identification has been saved in /home/root/.ssh/id_dsa.

Your public key has been saved in /home/root/.ssh/id_dsa.pub.

The key fingerprint is:

21:47:cd:13:d2:17:a0:cb:c9:a0:c5:fc:39:3e:c4:bb root@host02

[root@host01 ~]$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

[root@host01 ~]$ cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys

[root@host01 ~]$ ssh host02 cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

The authenticity of host 'host02 (192.168.56.102)' can't be established.

RSA key fingerprint is 5b:13:97:1a:0c:4d:36:93:7b:b5:87:2f:ac:34:26:1f.

Are you sure you want to continue connecting (yes/no)? yes

Warning: Permanently added 'host02,192.168.56.102' (RSA) to the list of known hosts.

root@host02's password:

[root@host01 ~]$ ssh host02 cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys

root@host02's password:

[root@host01 ~]$ scp ~/.ssh/authorized_keys host02:~/.ssh/authorized_keys

root@host02's password:

authorized_keys                                                                                   100% 1992     2.0KB/s   00:00

[root@host01 .ssh]$ chmod 644 ~/.ssh /authorized_keys

1.4关闭防护墙

service iptables stop

chkconfig iptables off

关闭SELINUX:

修改 /etc/selinux/config下的SELINUX=disabled  (重启后永久生效)

在所有的机器上做同样的操作

1.5 安装jdk

目前系统的jdk是1.4的,需要安装1.6以上的版本

[hadoop@node1 ~]$ java -version

java version "1.4.2"

gij (GNU libgcj) version 4.1.2 20080704 (Red Hat 4.1.2-51)

可以到oracle官网下载jdk,下载地址下载地址：

我这里下载的是jdk-8u5-linux-i586.rpm

我这里下载的是rpm安装文件，可以先查看rpm包中java的安装路径

[root@node1 soft]# rpm -qpl jdk-8u5-linux-i586.rpm

/usr/java/jdk1.8.0_05/man/man1/javap.1

/usr/java/jdk1.8.0_05/man/man1/javaws.1

/usr/java/jdk1.8.0_05/man/man1/jcmd.1

/usr/java/jdk1.8.0_05/man/man1/jconsole.1

/usr/java/jdk1.8.0_05/man/man1/jdb.1

/usr/java/jdk1.8.0_05/man/man1/jdeps.1

/usr/java/jdk1.8.0_05/man/man1/jhat.1

/usr/java/jdk1.8.0_05/man/man1/jinfo.1

/usr/java/jdk1.8.0_05/man/man1/jjs.1

/usr/java/jdk1.8.0_05/man/man1/jmap.1

/usr/java/jdk1.8.0_05/man/man1/jmc.1

/usr/java/jdk1.8.0_05/man/man1/jps.1

/usr/java/jdk1.8.0_05/man/man1/jrunscript.1

/usr/java/jdk1.8.0_05/man/man1/jsadebugd.1

/usr/java/jdk1.8.0_05/man/man1/jstack.1

/usr/java/jdk1.8.0_05/man/man1/jstat.1

/usr/java/jdk1.8.0_05/man/man1/jstatd.1

/usr/java/jdk1.8.0_05/man/man1/jvisualvm.1

/usr/java/jdk1.8.0_05/man/man1/keytool.1

/usr/java/jdk1.8.0_05/man/man1/native2ascii.1

/usr/java/jdk1.8.0_05/man/man1/orbd.1

/usr/java/jdk1.8.0_05/man/man1/pack200.1

/usr/java/jdk1.8.0_05/man/man1/policytool.1

/usr/java/jdk1.8.0_05/man/man1/rmic.1

/usr/java/jdk1.8.0_05/man/man1/rmid.1

/usr/java/jdk1.8.0_05/man/man1/rmiregistry.1

/usr/java/jdk1.8.0_05/man/man1/schemagen.1

/usr/java/jdk1.8.0_05/man/man1/serialver.1

/usr/java/jdk1.8.0_05/man/man1/servertool.1

/usr/java/jdk1.8.0_05/man/man1/tnameserv.1

/usr/java/jdk1.8.0_05/man/man1/unpack200.1

/usr/java/jdk1.8.0_05/man/man1/wsgen.1

/usr/java/jdk1.8.0_05/man/man1/wsimport.1

/usr/java/jdk1.8.0_05/man/man1/xjc.1

/usr/java/jdk1.8.0_05/release

/usr/java/jdk1.8.0_05/src.zip

部分输出，可以看出直接安装rpm包的话，java会默认安装在/usr/java/目录下,为hadoop设置环境变量java_home的时候可以指定该目录,不指定的话,系统会默认去找系统默认安装的java目录。

使用如下的命令直接安装java

[root@node1 soft]# rpm -ivh jdk-8u5-linux-i586.rpm

Preparing...                ########################################### [100%]

   1:jdk                    ########################################### [100%]

Unpacking JAR files...

        rt.jar...

        jsse.jar...

        charsets.jar...

        tools.jar...

        localedata.jar...

        jfxrt.jar...

        plugin.jar...

        javaws.jar...

        deploy.jar...

 安装完成后，为hadoop设定环境变量,如下:

[hadoop@node1 ~]$ more .bash_profile

# .bash_profile

# Get the aliases and functions

if [ -f ~/.bashrc ]; then

        . ~/.bashrc

fi

# User specific environment and startup programs

export JAVA_HOME=/usr/java/jdk1.8.0_05

export JRE_HOME=/usr/java/jdk1.8.0_05/jre

export CLASSPATH=.:$CLASSPATH:$JAVA_HOME/lib:$JRE_HOME/lib

export PATH=$JAVA_HOME/bin:$JRE_HOME/bin:$PATH

1.6 安装mysql

这里安装过程省略，具体可以在网上找安装文档，安装完成后确保设置mysql为自动启动.

设置开机启动:chkconfig mysqld on

启动mysql服务:service mysqld start

创建用户

grant ALL on *.* to root@'192.168.56.%' identified by 'hadoop' WITH GRANT OPTION;

grant ALL on *.* to root@'host01' identified by 'hadoop' WITH GRANT OPTION;

grant ALL on *.* to scm@'192.168.56.%' identified by 'scm' WITH GRANT OPTION;

grant ALL on *.* to scm@'host01' identified by 'scm' WITH GRANT OPTION;

flush privileges;

 

创建数据库

#hive

create database hive DEFAULT CHARSET utf8 COLLATE utf8_general_ci;

#activity monitor

create database amon DEFAULT CHARSET utf8 COLLATE utf8_general_ci;

#hue

create database hue DEFAULT CHARSET utf8 COLLATE utf8_general_ci;

1.6 安装Cloudera Manager Server和Agent

1.6.1 下载安装介质

到官网下载cloudera-manager,下载地址为

我这里下载的是cloudera-manager-el5-cm5.3.2_x86_64.tar.gz

CDH安装包地址：

我这里下载的是

CDH-5.3.2-1.cdh5.3.2.p0.10-el5.parcel

CDH-5.3.2-1.cdh5.3.2.p0.10-el5.parcel.sha1

manifest.json

1.6.2 主节点解压安装

将下载下来的cloudera-manager-el5-cm5.3.2_x86_64.tar.gz上传到主节点/opt目录下，执行解压缩命令

tar –xvf cloudera-manager-el5-cm5.3.2_x86_64.tar.gz

1.6.3 Agent配置

修改/opt/cm-5.3.2/etc/cloudera-scm-agent/config.ini的server_host为主节点的主机名

如下：

[General]

# Hostname of the CM server.

server_host=host01

1.6.4 同步到其他节点

scp -r /opt/cm-5.3.2/ root@host02:/opt/

1.6.5 创建cloudera-scm用户

 在每个节点上执行如下的命令：

useradd --home=/opt/cm-5.3.2/run/cloudera-scm-server/ --shell=/bin/false --comment "Cloudera SCM User" cloudera-scm

1.6.6为Cloudera Manager建立数据库

将mysql-connector-java-5.1.30.jar放到/opt/cm-5.3.2/share/cmf/lib/中。

在主节点中初始化CM5的数据库：

root@host01 cloudera]# /opt/cm-5.3.2/share/cmf/schema/scm_prepare_database.sh mysql cm -h 127.0.0.1 -uroot -phadoop --scm-host 127.0.0.1 scm scm scm

1.6.7 准备Parcels,用以安装CDH5

将如下3个文件copy到主节点的/opt/cloudera/parcel-repo/目录下

CDH-5.3.2-1.cdh5.3.2.p0.10-el5.parcel

CDH-5.3.2-1.cdh5.3.2.p0.10-el5.parcel.sha1

manifest.json

然后将CDH-5.3.2-1.cdh5.3.2.p0.10-el5.parcel.sha1重命名为CDH-5.3.2-1.cdh5.3.2.p0.10-el5.parcel.sha

1.6.8 启动服务

/opt/cm-5.3.2/etc/init.d/cloudera-scm-server start启动服务端(主节点)

/opt/cm-5.3.2/etc/init.d/cloudera-scm-agent start启动Agent服务(所有节点)

1.6.9 安装cdh

在IE栏中输入地址，第一次登陆估计要等待一段时间，最后会弹出登陆提示框：

 

验证：

1.使用hdfs用户登录创建一个目录

hdfs用户在安装cdh的过程自动创建的用户，不需要用户手工创建

 [root@host01 /]# su - hdfs
[hdfs@host01 ~]$ hdfs dfs -mkdir -p /user/hadoop