Chinaunix首页 | 论坛 | 博客

MyLinuxBlog

首页　| 　博文目录　| 　关于我

OowarrioroO

  • 博客访问： 291920
  • 博文数量： 82
  • 博客积分： 0
  • 博客等级： 民兵
  • 技术积分： 874
  • 用 户 组： 普通用户
  • 注册时间： 2015-03-21 09:58
个人简介

traveling in cumputer science!!

文章分类

全部博文（82）

文章存档

2016年（13）

2015年（69）

我的朋友
最近访客
推荐博文
相关博文
setup cluster -- configure hadoop

分类： 云计算

2015-08-05 22:56:21

声明：文章原创，转载需注明出处。由于文章大多是学习过程中的记录内容，技术能力有限，希望大家指出其中错误，共同交流进步。由此原因，文章会不定期改善，看原文最新内容，请到：http://blog.chinaunix.net/uid/29454152.html
after above two step now we can configure our hadoop
at first we must install the "jdk" and then we can install hadoop
1. install jdk
    i use the command "sudo apt-get install openjdk-7-jdk"
    and you can down the new jdk package and install it
2. down the hadoop package  at below url
     />     and you can search "hadoop releases" by google to find
3. install and configure the file
    (1 install it
    uncompress the package
    at the "Downloads" directory use the below commands
    "tar-vxzf hadoop-2.6.0.tar.gz -C /home/warrior/bigData"
    "cd /home/warrior/bigData"
    "mv hadoop-2.6.0 hadoop"
    (2 configure it 
    at first set the "/etc/profile"
    add the content 

点击(此处)折叠或打开

  1. export JAVA_HOME=/usr/lib/jvm/java-7-openjdk-amd64/
  2. export HADOOP_INSTALL=/home/warrior/bigData/hadoop
  3. export PATH=$PATH:$HADOOP_INSTALL/bin
  4. export PATH=$PATH:$HADOOP_INSTALL/sbin
  5. export HADOOP_MAPRED_HOME=$HADOOP_INSTALL
  6. export HADOOP_COMMON_HOME=$HADOOP_INSTALL
  7. export HADOOP_HDFS_HOME=$HADOOP_INSTALL
  8. export YARN_HOME=$HADOOP_INSTALL
    (3 set the /home/warrior/bigData/hadoop/etc/hadoop/core-site.xml

点击(此处)折叠或打开

  1. <?xml version="1.0" encoding="UTF-8"?>
  2. <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
  3. <!--
  4.   Licensed under the Apache License, Version 2.0 (the "License");
  5.   you may not use this file except in compliance with the License.
  6.   You may obtain a copy of the License at

  8.     http://www.apache.org/licenses/LICENSE-2.0

  10.   Unless required by applicable law or agreed to in writing, software
  11.   distributed under the License is distributed on an "AS IS" BASIS,
  12.   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  13.   See the License for the specific language governing permissions and
  14.   limitations under the License. See accompanying LICENSE file.
  15. -->

  17. <!-- Put site-specific property overrides in this file. -->

  19. <configuration>
  20.     /* path of HDFS */
  21.     <property>
  22.         <name>fs.defaultFS</name>
  23.     </property>
  24.     /* buffer size */
  25.     <property>
  26.         <name>fs.default.name</name>
  27.         <value>hdfs://warrior:9000</value>
  28.     </property>
  29.     /* temp folder */
  30.     <property>
  31.         <name>hadoop.tmp.dir</name>
  32.         <value>file:/home/warrior/tmp</value>
  33.         <description>Abase for other temporary directories.</description>
  34.     </property>


  37. </configuration>
    (4 set the /home/warrior/bigData/hadoop/etc/hadoop/yarn-site.xml

点击(此处)折叠或打开

  1. <?xml version="1.0"?>
  2. <!--
  3.   Licensed under the Apache License, Version 2.0 (the "License");
  4.   you may not use this file except in compliance with the License.
  5.   You may obtain a copy of the License at

  7.     http://www.apache.org/licenses/LICENSE-2.0

  9.   Unless required by applicable law or agreed to in writing, software
  10.   distributed under the License is distributed on an "AS IS" BASIS,
  11.   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  12.   See the License for the specific language governing permissions and
  13.   limitations under the License. See accompanying LICENSE file.
  14. -->
  15. <configuration>

  17. <!-- Site specific YARN configuration properties -->
  18.     <property>
  19.         <name>yarn.nodemanager.aux-services</name>
  20.         <value>mapreduce_shuffle</value>
  21.     </property>
  22.     
  23.     <property>
  24.         <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
  25.         <value>org.apache.hadoop.mapred.ShuffleHandler</value>
  26.     </property>
  27.     
  28.     <property>
  29.         <name>yarn.resourcemanager.address</name>
  30.         <value>warrior:8032</value>
  31.     </property>
  32.     <property>
  33.         <name>yarn.resourcemanager.scheduler.address</name>
  34.         <value>warrior:8030</value>
  35.     </property>
  36.     
  37.     <property>
  38.         <name>yarn.resourcemanager.resource-tracker.address</name>
  39.         <value>warrior:8031</value>
  40.     </property>
  41.     
  42.     <property>
  43.         <name>yarn.resourcemanager.admin.address</name>
  44.         <value>warrior:8033</value>
  45.     </property>
  46.     
  47.     <property>
  48.         <name>yarn.resourcemanager.webapp.address</name>
  49.         <value>warrior:8088</value>
  50.     </property>
  51. </configuration>
    (5 set the /home/warrior/bigData/hadoop/etc/hadoop/mapres-site.xml

点击(此处)折叠或打开

  1. <configuration>
  2.     <property>
  3.         <name>mapreduce.framework.name</name>
  4.         <value>yarn</value>
  5.     </property>
  6.     
  7.     <property>
  8.         <name>mapreduce.jobhistory.address</name>
  9.         <value>warrior:10020</value>
  10.     </property>

  12.     <property>
  13.         <name>mapred.job.tracker</name>
  14.         <value>warrior:10001</value>
  15.     </property>
  16.     
  17.     <property>
  18.         <name>mapreduce.jobhistory.webapp.address</name>
  19.         <value>warrior:19888</value>
  20.     </property>
  21. </configuration>
    (6 make directory "namenode" "datanode" at "/home/warrior/bigData/"
    and set the /home/warrior/bigData/hadoop/etc/hadoop/hdfs-site.xml

点击(此处)折叠或打开

  1. <?xml version="1.0" encoding="UTF-8"?>
  2. <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
  3. <!--
  4.   Licensed under the Apache License, Version 2.0 (the "License");
  5.   you may not use this file except in compliance with the License.
  6.   You may obtain a copy of the License at

  8.     http://www.apache.org/licenses/LICENSE-2.0

  10.   Unless required by applicable law or agreed to in writing, software
  11.   distributed under the License is distributed on an "AS IS" BASIS,
  12.   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  13.   See the License for the specific language governing permissions and
  14.   limitations under the License. See accompanying LICENSE file.
  15. -->

  17. <!-- Put site-specific property overrides in this file. -->

  19. <configuration>
  20.     <property>
  21.         <name>dfs.namenode.secondary.http-address</name>
  22.         <value>warrior:9001</value>
  23.     </property>
  24.     
  25.     <property>
  26.         <name>dfs.namenode.name.dir</name>
  27.         <value>file:/home/warrior/bigData/hdfs//namenode</value>
  28.     </property>
  29.     
  30.     <property>
  31.         <name>dfs.datanode.data.dir</name>
  32.         <value>file:/home/warrior/bigData/hdfs//datanode</value>
  33.     </property>
  34.     
  35.     <property>
  36.         <name>dfs.replication</name>
  37.         <value>1</value>
  38.     </property>
  39.     
  40.     <property>
  41.         <name>dfs.webdfs.enabled</name>
  42.         <value>true</value>
  43.     </property>
  44. </configuration>
    (7 set the "master" and "slaves" 
    below is my set 
    set the /home/warrior/bigData/hadoop/etc/hadoop/master
    warrior@10.0.2.10
    set the /home/warrior/bigData/hadoop/etc/hadoop/slaves
    warrior@10.0.2.20
    warrior@10.0.2.30

4. scp the directory to all the other machines
    like at 10.0.2.10 to 10.0.2.20
    "scp /home/warrior/bigData/hadoop/  warrior@10.0.2.20:/home/warrior/bigData/hadoop/"
    to other like above
5. use below command test the hadoop cluster
    "./bin/hadoop namenode -format"
    "./sbin/start-all.sh"


阅读(1273) | 评论(0) | 转发(0) |
1

上一篇：setup cluster -- ssh to each other with no key ssh免密钥登陆配置

下一篇：搭建spark cluster过程中遇到的网络问题

给主人留下些什么吧！~~
关于我们 | 关于IT168 | 联系方式 | 广告合作 | 法律声明 | 免费注册

Copyright 2001-2010 ChinaUnix.net All Rights Reserved 北京皓辰网域网络信息技术有限公司. 版权所有

感谢所有关心和支持过ChinaUnix的朋友们

16024965号-6