Chinaunix首页 | 论坛 | 博客
  • 博客访问: 4825513
  • 博文数量: 971
  • 博客积分: 8199
  • 博客等级: 中将
  • 技术积分: 12717
  • 用 户 组: 普通用户
  • 注册时间: 2010-05-25 20:19
个人简介

脚踏实地、勇往直前!

文章分类

全部博文(971)

文章存档

2019年(62)

2018年(208)

2017年(81)

2016年(49)

2015年(50)

2014年(170)

2013年(52)

2012年(177)

2011年(93)

2010年(30)

分类: HADOOP

2015-04-27 14:48:00

环境:

OS:Rad Hat Linux As5

CDH:5.3.2



1.安装步骤

1.1 节点角色分配


节点角色

ip

hostname

主节点

192.168.56.111

host01

从节点

192.168.56.112

host02


 


 

这里只有两个节点的环境,一个做主节点,另外一个做从节点


1.2配置hosts文件

 每台机器分配了ip和设定了主机名后,需要在hosts文件下添加如下项

 [root@host01 /]# more /etc/hosts

 # Do not remove the following line, or various programs

 # that require network functionality will fail.

 127.0.0.1               localhost.localdomain localhost

 ::1             localhost6.localdomain6 localhost6

 192.168.56.111   host01.com        host01

 192.168.56.112   host02.com        host02

 

1.3 建立等效连接

 [root@host01 ~]$ cd ~

 [root@host01 ~]$ mkdir ~/.ssh

 [root@host01 ~]$ chmod 700 ~/.ssh

 [root@host01 ~]$ ssh-keygen -t rsa

 Generating public/private rsa key pair.

 Enter file in which to save the key (/home/root/.ssh/id_rsa):

 Enter passphrase (empty for no passphrase):

 Enter same passphrase again:

 Your identification has been saved in /home/root/.ssh/id_rsa.

 Your public key has been saved in /home/root/.ssh/id_rsa.pub.

 The key fingerprint is:

 84:88:f0:dc:7c:a6:d0:e8:68:65:53:97:35:02:05:fb root@host01

 [root@host01 ~]$ ssh-keygen -t dsa

 Generating public/private dsa key pair.

 Enter file in which to save the key (/home/root/.ssh/id_dsa):

 Enter passphrase (empty for no passphrase):

 Enter same passphrase again:

 Your identification has been saved in /home/root/.ssh/id_dsa.

 Your public key has been saved in /home/root/.ssh/id_dsa.pub.

 The key fingerprint is:

 94:80:60:37:38:e6:aa:10:22:50:1a:c2:42:56:22:0a root@host01

 

[root@host01 .ssh]$ chmod 644 ~/.ssh /authorized_keys

[root@host02 ~]$ cd ~
[root@host02 ~]$ mkdir ~/.ssh
[root@host02 ~]$ chmod 700 ~/.ssh


[root@host02 ~]$ ssh-keygen -t rsa

Generating public/private rsa key pair.

Enter file in which to save the key (/home/root/.ssh/id_rsa):

Enter passphrase (empty for no passphrase):

Enter same passphrase again:

Your identification has been saved in /home/root/.ssh/id_rsa.

Your public key has been saved in /home/root/.ssh/id_rsa.pub.

The key fingerprint is:

00:ee:68:42:3a:16:a3:75:6c:d4:36:49:57:a5:5c:57 root@host02

[root@host02 ~]$ ssh-keygen -t dsa

Generating public/private dsa key pair.

Enter file in which to save the key (/home/root/.ssh/id_dsa):

Enter passphrase (empty for no passphrase):

Enter same passphrase again:

Your identification has been saved in /home/root/.ssh/id_dsa.

Your public key has been saved in /home/root/.ssh/id_dsa.pub.

The key fingerprint is:

21:47:cd:13:d2:17:a0:cb:c9:a0:c5:fc:39:3e:c4:bb root@host02


[root@host01 ~]$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

[root@host01 ~]$ cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys

[root@host01 ~]$ ssh host02 cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

The authenticity of host 'host02 (192.168.56.102)' can't be established.

RSA key fingerprint is 5b:13:97:1a:0c:4d:36:93:7b:b5:87:2f:ac:34:26:1f.

Are you sure you want to continue connecting (yes/no)? yes

Warning: Permanently added 'host02,192.168.56.102' (RSA) to the list of known hosts.

root@host02's password:

[root@host01 ~]$ ssh host02 cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys

root@host02's password:

[root@host01 ~]$ scp ~/.ssh/authorized_keys host02:~/.ssh/authorized_keys

root@host02's password:

authorized_keys                                                                                   100% 1992     2.0KB/s   00:00

[root@host01 .ssh]$ chmod 644 ~/.ssh /authorized_keys


1.4
关闭防护墙

service iptables stop

chkconfig iptables off

关闭SELINUX:

修改 /etc/selinux/config下的SELINUX=disabled  (重启后永久生效)

在所有的机器上做同样的操作



1.5 安装jdk

目前系统的jdk1.4,需要安装1.6以上的版本

[hadoop@node1 ~]$ java -version

java version "1.4.2"

gij (GNU libgcj) version 4.1.2 20080704 (Red Hat 4.1.2-51)

可以到oracle官网下载jdk,下载地址下载地址http://www.oracle.com/technetwork/java/javase/index.html

我这里下载的是jdk-8u5-linux-i586.rpm



我这里下载的是rpm安装文件,可以先查看rpm包中java的安装路径

[root@node1 soft]# rpm -qpl jdk-8u5-linux-i586.rpm

/usr/java/jdk1.8.0_05/man/man1/javap.1

/usr/java/jdk1.8.0_05/man/man1/javaws.1

/usr/java/jdk1.8.0_05/man/man1/jcmd.1

/usr/java/jdk1.8.0_05/man/man1/jconsole.1

/usr/java/jdk1.8.0_05/man/man1/jdb.1

/usr/java/jdk1.8.0_05/man/man1/jdeps.1

/usr/java/jdk1.8.0_05/man/man1/jhat.1

/usr/java/jdk1.8.0_05/man/man1/jinfo.1

/usr/java/jdk1.8.0_05/man/man1/jjs.1

/usr/java/jdk1.8.0_05/man/man1/jmap.1

/usr/java/jdk1.8.0_05/man/man1/jmc.1

/usr/java/jdk1.8.0_05/man/man1/jps.1

/usr/java/jdk1.8.0_05/man/man1/jrunscript.1

/usr/java/jdk1.8.0_05/man/man1/jsadebugd.1

/usr/java/jdk1.8.0_05/man/man1/jstack.1

/usr/java/jdk1.8.0_05/man/man1/jstat.1

/usr/java/jdk1.8.0_05/man/man1/jstatd.1

/usr/java/jdk1.8.0_05/man/man1/jvisualvm.1

/usr/java/jdk1.8.0_05/man/man1/keytool.1

/usr/java/jdk1.8.0_05/man/man1/native2ascii.1

/usr/java/jdk1.8.0_05/man/man1/orbd.1

/usr/java/jdk1.8.0_05/man/man1/pack200.1

/usr/java/jdk1.8.0_05/man/man1/policytool.1

/usr/java/jdk1.8.0_05/man/man1/rmic.1

/usr/java/jdk1.8.0_05/man/man1/rmid.1

/usr/java/jdk1.8.0_05/man/man1/rmiregistry.1

/usr/java/jdk1.8.0_05/man/man1/schemagen.1

/usr/java/jdk1.8.0_05/man/man1/serialver.1

/usr/java/jdk1.8.0_05/man/man1/servertool.1

/usr/java/jdk1.8.0_05/man/man1/tnameserv.1

/usr/java/jdk1.8.0_05/man/man1/unpack200.1

/usr/java/jdk1.8.0_05/man/man1/wsgen.1

/usr/java/jdk1.8.0_05/man/man1/wsimport.1

/usr/java/jdk1.8.0_05/man/man1/xjc.1

/usr/java/jdk1.8.0_05/release

/usr/java/jdk1.8.0_05/src.zip


部分输出,可以看出直接安装
rpm包的话,java会默认安装在/usr/java/目录下,hadoop设置环境变量java_home的时候可以指定该目录,不指定的话,系统会默认去找系统默认安装的java目录。

使用如下的命令直接安装java

[root@node1 soft]# rpm -ivh jdk-8u5-linux-i586.rpm

Preparing...                ########################################### [100%]

   1:jdk                    ########################################### [100%]

Unpacking JAR files...

        rt.jar...

        jsse.jar...

        charsets.jar...

        tools.jar...

        localedata.jar...

        jfxrt.jar...

        plugin.jar...

        javaws.jar...

        deploy.jar...


 安装完成后,为hadoop设定环境变量,如下:

[hadoop@node1 ~]$ more .bash_profile

# .bash_profile

# Get the aliases and functions

if [ -f ~/.bashrc ]; then

        . ~/.bashrc

fi

# User specific environment and startup programs

export JAVA_HOME=/usr/java/jdk1.8.0_05

export JRE_HOME=/usr/java/jdk1.8.0_05/jre

export CLASSPATH=.:$CLASSPATH:$JAVA_HOME/lib:$JRE_HOME/lib

export PATH=$JAVA_HOME/bin:$JRE_HOME/bin:$PATH


1.6
安装mysql

这里安装过程省略,具体可以在网上找安装文档,安装完成后确保设置mysql为自动启动.

设置开机启动:chkconfig mysqld on

启动mysql服务:service mysqld start

创建用户

grant ALL on *.* to root@'192.168.56.%' identified by 'hadoop' WITH GRANT OPTION;

grant ALL on *.* to root@'host01' identified by 'hadoop' WITH GRANT OPTION;

grant ALL on *.* to scm@'192.168.56.%' identified by 'scm' WITH GRANT OPTION;

grant ALL on *.* to scm@'host01' identified by 'scm' WITH GRANT OPTION;

flush privileges;

 

创建数据库

#hive

create database hive DEFAULT CHARSET utf8 COLLATE utf8_general_ci;

#activity monitor

create database amon DEFAULT CHARSET utf8 COLLATE utf8_general_ci;

#hue

create database hue DEFAULT CHARSET utf8 COLLATE utf8_general_ci;


1.6
安装Cloudera Manager ServerAgent


1.6.1 下载安装介质

到官网下载cloudera-manager,下载地址为http://archive-primary.cloudera.com/cm5/cm/5/

我这里下载的是cloudera-manager-el5-cm5.3.2_x86_64.tar.gz

CDH安装包地址:http://archive.cloudera.com/cdh5/parcels/5.3.2/

我这里下载的是

CDH-5.3.2-1.cdh5.3.2.p0.10-el5.parcel

CDH-5.3.2-1.cdh5.3.2.p0.10-el5.parcel.sha1

manifest.json

1.6.2 主节点解压安装


将下载下来的cloudera-manager-el5-cm5.3.2_x86_64.tar.gz上传到主节点/opt目录下,执行解压缩命令

tar –xvf cloudera-manager-el5-cm5.3.2_x86_64.tar.gz


1.6.3 Agent
配置

修改/opt/cm-5.3.2/etc/cloudera-scm-agent/config.iniserver_host为主节点的主机名

如下:

[General]

# Hostname of the CM server.

server_host=host01

1.6.4 同步到其他节点

scp -r /opt/cm-5.3.2/ root@host02:/opt/


1.6.5
创建cloudera-scm用户


 在每个节点上执行如下的命令:

useradd --home=/opt/cm-5.3.2/run/cloudera-scm-server/ --shell=/bin/false --comment "Cloudera SCM User" cloudera-scm


1.6.6
Cloudera Manager建立数据库

mysql-connector-java-5.1.30.jar放到/opt/cm-5.3.2/share/cmf/lib/中。

在主节点中初始化CM5的数据库:

root@host01 cloudera]# /opt/cm-5.3.2/share/cmf/schema/scm_prepare_database.sh mysql cm -h 127.0.0.1 -uroot -phadoop --scm-host 127.0.0.1 scm scm scm

1.6.7 准备Parcels,用以安装CDH5


将如下3个文件copy到主节点的/opt/cloudera/parcel-repo/目录下

CDH-5.3.2-1.cdh5.3.2.p0.10-el5.parcel

CDH-5.3.2-1.cdh5.3.2.p0.10-el5.parcel.sha1

manifest.json

然后将CDH-5.3.2-1.cdh5.3.2.p0.10-el5.parcel.sha1重命名为CDH-5.3.2-1.cdh5.3.2.p0.10-el5.parcel.sha

1.6.8 启动服务


/opt/cm-5.3.2/etc/init.d/cloudera-scm-server start启动服务端(主节点)

/opt/cm-5.3.2/etc/init.d/cloudera-scm-agent start启动Agent服务(所有节点)

1.6.9 安装cdh

IE栏中输入地址http://192.168.56.111:7180/cmf/login,第一次登陆估计要等待一段时间,最后会弹出登陆提示框:







 

验证:

1.使用hdfs用户登录创建一个目录

hdfs用户在安装cdh的过程自动创建的用户,不需要用户手工创建

 [root@host01 /]# su - hdfs
[hdfs@host01 ~]$ hdfs dfs -mkdir -p /user/hadoop
 

阅读(4565) | 评论(0) | 转发(0) |
给主人留下些什么吧!~~
评论热议
请登录后评论。

登录 注册