ambari本地源自动化安装hortonworks hadoop

环境配置

Hostname	IP	OS
Ambari	172.16.235.164	Centos X64 base install
Maste.hadoop	172.16.235.165	Centos X64 base install
Datanode1.hadoop	172.16.235.166	Centos X64 base install

操作系统建议以software server 方式安装，base 方式安装后，需要安装Development tools，本文以base举例。

 

一：制作ambari，hadoop本地源

首先需要有一台能够联网的linux机器

1：制作ambari（1.4.1.25）本地源

[root@controller ~]# wget

[root@controller ~]# cp ambari.repo /etc/yum.repos.d/

将网络源中的包同步到本地：

reposync  -r HDP-UTILS-1.1.0.16

reposync  -r Updates-ambari-1.4.1.25

reposync  -r ambari-1.x

会得到相应的三个目录，每个目录中有相应的rpm包

准备一台http服务器，本文以172.16.235.164作为http服务器发布安装源

mkdir /var/www/html/ambari

将 HDP-UTILS-1.1.0.16 Updates-ambari-1.4.1.25 ambari-1.x复制到 /var/www/html/ambari下

cd /var/www/html/ambari

createrepo ./    #没有createrepo命令自己去安装

这样ambari安装源就做好了，下载好jdk-6u31-linux-x64.bin备用

 

2：准备hortonworks hadoop （HDP）2.0.6本地源

准备HDP网络源

vim /etc/yum.repos.d/HDP.repo，添加以下内容

[HDP-2.0.6]

name=HDP

baseurl=

enabled=1

gpgcheck=0

同步网络源到本地：

reposync  -r HDP-2.0.6  #将近1G

讲 HDP-2.0.6 复制到http server的/var/www/html/hdp下

cd /var/www/html/hdp

createrepo ./  

这样hadoop的本地源就做好了

 

3：制作系统本地源（安装ambari和hadoop时，可能会依赖些系统包，这时需要有系统的本地源来安装）

http server 挂载系统光盘

mount /dev/cdrom  /var/www/html/centos/

 

二：安装ambari

系统准备：关闭iptables、selinux,启动ntpd服务

[root@ambari ]vim /etc/hosts

172.16.235.164  ambari

172.16.235.165  master.hadoop

172.16.235.166  datanode1.hadoop

配置与所有主机的无密码登入

[root@ambari ~]# ssh-keygen  #一路回车（所有主机都要执行）

[root@ambari ~]# cd .ssh/

[root@ambari .ssh]# ls

id_rsa  id_rsa.pub

[root@ambari .ssh]# scp id_rsa.pub master.hadoop:/root/.ssh/1

The authenticity of host 'master.hadoop (172.16.235.165)' can't be established.

RSA key fingerprint is fc:33:53:5f:f2:20:86:92:ac:96:09:e2:dc:e7:33:ea.

Are you sure you want to continue connecting (yes/no)? yes

Warning: Permanently added 'master.hadoop,172.16.235.165' (RSA) to the list of known hosts.

[e-mail]'s password:

id_rsa.pub                                                                                        100%  393     0.4KB/s   00:00   

[root@ambari .ssh]# scp id_rsa.pub datanode1.hadoop:/root/.ssh/1

The authenticity of host 'datanode1.hadoop (172.16.235.166)' can't be established.

RSA key fingerprint is fc:33:53:5f:f2:20:86:92:ac:96:09:e2:dc:e7:33:ea.

Are you sure you want to continue connecting (yes/no)? yes

Warning: Permanently added 'datanode1.hadoop,172.16.235.166' (RSA) to the list of known hosts.

[e-mail]'s password:

id_rsa.pub                                                                                        100%  393     0.4KB/s   00:00  

登入master.hadoop

[root@master ~]# cd .ssh/

[root@master .ssh]# cat 1 >authorized_keys

登入datanode1.hadoop

[root@datanode1 ~]# cd .ssh/

[root@datanode1 .ssh]# cat 1 >authorized_keys

回到ambari 测试无密码登入

[root@ambari .ssh]# ssh master.hadoop

Last login: Wed Dec  4 11:11:15 2013 from 172.16.38.48

[root@master ~]# exit

logout

Connection to master.hadoop closed.

[root@ambari .ssh]# ssh datanode1.hadoop

Last login: Wed Dec  4 10:38:06 2013 from 172.16.38.48

[root@datanode1 ~]# exit

logout

Connection to datanode1.hadoop closed.

[root@ambari .ssh]#

 

安装与配置ambari服务：

[root@ambari ]vim /etc/yum.repos.d/ambari.repo  添加以下内容

[ambari-1.x]

name=Ambari 1.x

baseurl=

gpgcheck=0

enabled=1

 

[HDP-UTILS-1.1.0.16]

name=Hortonworks Data Platform Utils Version - HDP-UTILS-1.1.0.16

baseurl=

gpgcheck=0

enabled=1

 

[Updates-ambari-1.4.1.25]

name=ambari-1.4.1.25 - Updates

baseurl=

gpgcheck=

enabled=1

[root@ambari ]vim /etc/yum.repos.d/centos.repo  添加以下内容

[centos]

baseurl=

enabled=1

gpgcheck=0

执行安装ambari：

[root@ambari yum.repos.d]# yum -y install ambari-server

Loaded plugins: fastestmirror, security

Repository 'Updates-ambari-1.4.1.25': Error parsing config: Error parsing "gpgcheck = ''": invalid boolean value

Repository 'centos' is missing name in configuration, using id

Loading mirror speeds from cached hostfile

Setting up Install Process

Resolving Dependencies

--> Running transaction check

---> Package ambari-server.noarch 0:1.4.1.25-1 will be installed

--> Processing Dependency: postgresql-server >= 8.1 for package: ambari-server-1.4.1.25-1.noarch

--> Running transaction check

---> Package postgresql-server.x86_64 0:8.4.13-1.el6_3 will be installed

--> Processing Dependency: postgresql-libs(x86-64) = 8.4.13-1.el6_3 for package: postgresql-server-8.4.13-1.el6_3.x86_64

--> Processing Dependency: postgresql(x86-64) = 8.4.13-1.el6_3 for package: postgresql-server-8.4.13-1.el6_3.x86_64

--> Processing Dependency: libpq.so.5()(64bit) for package: postgresql-server-8.4.13-1.el6_3.x86_64

--> Running transaction check

---> Package postgresql.x86_64 0:8.4.13-1.el6_3 will be installed

---> Package postgresql-libs.x86_64 0:8.4.13-1.el6_3 will be installed

--> Finished Dependency Resolution

 

Dependencies Resolved

 

====================================================================================================================================

 Package                           Arch                   Version                          Repository                          Size

====================================================================================================================================

Installing:

 ambari-server                     noarch                 1.4.1.25-1                       HDP-UTILS-1.1.0.16                  37 M

Installing for dependencies:

 postgresql                        x86_64                 8.4.13-1.el6_3                   centos                             2.8 M

 postgresql-libs                   x86_64                 8.4.13-1.el6_3                   centos                             200 k

 postgresql-server                 x86_64                 8.4.13-1.el6_3                   centos                             3.4 M

 

Transaction Summary

====================================================================================================================================

Install       4 Package(s)

 

Total download size: 43 M

Installed size: 88 M

Downloading Packages:

(1/4): ambari-server-1.4.1.25-1.noarch.rpm                                                                   |  37 MB     00:00    

(2/4): postgresql-server-8.4.13-1.el6_3.x86_64.rpm                                                           | 3.4 MB     00:00    

(3/4): postgresql-libs-8.4.13-1.el6_3.x86_64.rpm                                                             | 200 kB     00:00    

(4/4): postgresql-8.4.13-1.el6_3.x86_64.rpm                                                                  | 2.8 MB     00:00    

------------------------------------------------------------------------------------------------------------------------------------

Total                                                                                                43 MB/s |  43 MB     00:00    

Running rpm_check_debug

Running Transaction Test

Transaction Test Succeeded

Running Transaction

  Installing : postgresql-libs-8.4.13-1.el6_3.x86_64                                                                            1/4

  Installing : postgresql-8.4.13-1.el6_3.x86_64                                                                                 2/4

  Installing : postgresql-server-8.4.13-1.el6_3.x86_64                                                                          3/4

  Installing : ambari-server-1.4.1.25-1.noarch                                                                                  4/4

  Verifying  : postgresql-server-8.4.13-1.el6_3.x86_64                                                                          1/4

  Verifying  : postgresql-libs-8.4.13-1.el6_3.x86_64                                                                            2/4

  Verifying  : ambari-server-1.4.1.25-1.noarch                                                                                  3/4

  Verifying  : postgresql-8.4.13-1.el6_3.x86_64                                                                                 4/4

 

Installed:

  ambari-server.noarch 0:1.4.1.25-1                                                                                                 

 

Dependency Installed:

  postgresql.x86_64 0:8.4.13-1.el6_3     postgresql-libs.x86_64 0:8.4.13-1.el6_3     postgresql-server.x86_64 0:8.4.13-1.el6_3   

 

Complete!

 

开始配置ambari 服务：

首先上传jdk文件

[root@ambari ]cd /var/lib/ambari-server/resources  将jdk-6u31-linux-x64.bin上传到该目录

[root@ambari resources]# chmod +x jdk-6u31-linux-x64.bin

[root@ambari yum.repos.d]# ambari-server setup

Using python  /usr/bin/python2.6

Initializing...

Setup ambari-server

Checking SELinux...

SELinux status is 'enabled'

SELinux mode is 'permissive'

WARNING: SELinux is set to 'permissive' mode and temporarily disabled.

OK to continue [y/n] (y)?

Customize user account for ambari-server daemon [y/n] (n)?

Adjusting ambari-server permissions and ownership...

Checking iptables...

WARNING: iptables is running. Confirm the necessary Ambari ports are accessible. Refer to the Ambari documentation for more details on ports.

OK to continue [y/n] (y)?

Checking JDK...

JDK already exists, using /var/lib/ambari-server/resources/jdk-6u31-linux-x64.bin

Installing JDK to /usr/jdk64

Successfully installed JDK to /usr/jdk64/jdk1.6.0_31

Downloading JCE Policy archive from to /var/lib/ambari-server/resources/jce_policy-6.zip

To download the JCE Policy files you must accept the license terms found at accepting will result in errors when configuring Kerberos security.

Do you accept the Oracle Binary Code License Agreement [y/n] (y)?

Successfully downloaded JCE Policy archive to /var/lib/ambari-server/resources/jce_policy-6.zip

Completing setup...

Configuring database...

Enter advanced database configuration [y/n] (n)?

Default properties detected. Using built-in database.

Checking PostgreSQL...

Running initdb: This may take upto a minute.

About to start PostgreSQL

Configuring local database...

Connecting to the database. Attempt 1...

Configuring PostgreSQL...

Restarting PostgreSQL

Ambari Server 'setup' completed successfully.

[root@ambari ~]# ambari-server start

Using python  /usr/bin/python2.6

Starting ambari-server

Ambari Server running with 'root' privileges.

Server PID at: /var/run/ambari-server/ambari-server.pid

Server out at: /var/log/ambari-server/ambari-server.out

Server log at: /var/log/ambari-server/ambari-server.log

Ambari Server 'start' completed successfully.

到此ambari安装好

 

三：本地安装hadoop

给ambari配置本地hadoo源地址：

[root@ambari ~]# cd /var/lib/ambari-server/resources/stacks/HDPLocal/2.0.6/repos/

[root@ambari repos]# vim repoinfo.xml  修改红色部分为本地源地址

 

   

     

      HDP-2.0.6

      HDP

   

 

这样在ambari执行hadoop安装时，会将本地源地址配置到所有主机上，在为所有主机安装ambari-agent时，会将ambari server下的/etc/yum.repo/ambari.repo文件复制到所有主机

 

主机准备（master.hadoop、datanode1.hadoop）配置相同，只举例master.hadoop

关闭iptables、selinux、启动ntpd服务（略）

[root@master ~]# vim /etc/hosts 添加以下

172.16.235.164  ambari

172.16.235.165  master.hadoop

172.16.235.166  datanode1.hadoop

[root@master ~]# vim /etc/yum.repos.d/centos.repo 添加以下内容

[root

[centos]

baseurl=

enabled=1

gpgcheck=0

@master ~]# yum -y groupinstall "Development tools"

 

登入ambari管理页面

账户：admin 密码：admin

配置集群名称

选择HDP版本

选择目标主机，一行一个主机。要让ambar server自动在所以主机安装ambari-agent，需要提供ambari server root账户的私钥，即/root/.ssh/ id_rsa文件，选中使用本地源安装hadoop，然后下一步

确认并且选择主机

确认OK

下一步选择要安装的组件，选好后下一步

选择组件分配在哪些主机，都是master 的服务

选择安装client，主机有限，master同时安装client

配置服务，其中nagios中要填写密码及报警接受邮箱必填，其他选择性配置

显示配置信息

开始安装，过程稍微有点长

安装过程

Google浏览器反应比较慢，用另一某浏览器已经显示succes了

Ganglia：

nagios：