Chinaunix首页 | 论坛 | 博客

尘世中一个迷途小书童riverhwp.blog.chinaunix.net

首页　| 　博文目录　| 　关于我

Larpenteur

  • 博客访问： 6320353
  • 博文数量： 2759
  • 博客积分： 1021
  • 博客等级： 中士
  • 技术积分： 4091
  • 用 户 组： 普通用户
  • 注册时间： 2012-03-11 14:14
文章分类

全部博文（2759）

文章存档

2019年（1）

2017年（84）

2016年（196）

2015年（204）

2014年（636）

2013年（1176）

2012年（463）

我的朋友
最近访客
推荐博文
相关博文
Hadoop与Hbase基本配置

分类： HADOOP

2013-08-17 02:49:15

原文地址：Hadoop与Hbase基本配置 作者：windhawkgyang

     在经历了几周的努力之后，终于选择放弃现在安装的稳定版本，转而安装旧版本来部署Hadoop系统。直到昨天，自己还一直为Inconsistent configuration的错误头疼不已。既然同事的版本已经装成功了，那自己也就先用跑通的系统试一下吧，毕竟先放下再回来看现在的问题可能有更好的解决思路吧。今天开始正式重新安装Hadoop与Hbase。之前的安装笔记比较凌乱，今天借着这样的机会把整个步骤重新梳理一遍。
一、安装Hadoop
     自己使用的软件版本是hadoop-1.0.3，比较早的一个版本，可以去hadoop的官方网站去下载。在安装hadoop以前首先要设置系统环境：
<1> 安装java-1.6版本，之前自己安装的java-1.7，但是没有成功，不晓得是不是java版本的缘故；无论如何，这次自己选择了比较保守的方案，从oracle官方注册后下载jdk-6u45-linux-i586.bin后解包得到【jdk-1.6.0_45】。注意到这里都是bin文件，因此需要chmod该文件以775的可执行权限，然后./filename.bin即可；
<2> 安装ssh，并且设置ssh无密码登录hadoop
<2.1>运行sudo apt-get install ssh/rsync，运行sudo apt-get install openjdk-6-jdk (for jps command)
<2.2>配置ssh本机免口令登录主要有两步：
运行ssh-keygen -t dsa -P '' -f /.ssh/id_dsa
   -t用来指定加密算法，可以选择dsa和rsa两种加密方式；
   -P用来指定密码，两个单引号表示空密码''；
   -f用来指定存放密钥的文件
运行cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
   这一步将公钥添加进本机的authorized_keys中，完成这两步后可以ssh localhost验证是否成功。接下来需要进入已经解好的hadoop-1.0.3中进行配置。hadoop的伪分布模式主要需要配置以下几个配置文件：
<3>conf/hbase-env.sh：主要用来配置hadoop的运行环境，这里需要修改JAVA_HOME到你的jdk1.6目录（见黑体）

点击(此处)折叠或打开

  1. #Set Hadoop-specific environment variables here.

  3. # The only required environment variable is JAVA_HOME. All others are
  4. # optional. When running a distributed configuration it is best to
  5. # set JAVA_HOME in this file, so that it is correctly defined on
  6. # remote nodes.

  8. # The java implementation to use. Required.
  9. export JAVA_HOME=/home/hadoop/platform/jdk1.6.0_45

  11. # Extra Java CLASSPATH elements. Optional.
  12. # export HADOOP_CLASSPATH=

  14. # The maximum amount of heap to use, in MB. Default is 1000.
  15. # export HADOOP_HEAPSIZE=2000

  17. # Extra Java runtime options. Empty by default.
  18. # export HADOOP_OPTS=-server

  20. # Command specific options appended to HADOOP_OPTS when specified
  21. export HADOOP_NAMENODE_OPTS="-Dcom.sun.management.jmxremote $HADOOP_NAMENODE_OPTS"
  22. export HADOOP_SECONDARYNAMENODE_OPTS="-Dcom.sun.management.jmxremote $HADOOP_SECONDARYNAMENODE_OPTS"
  23. export HADOOP_DATANODE_OPTS="-Dcom.sun.management.jmxremote $HADOOP_DATANODE_OPTS"
  24. export HADOOP_BALANCER_OPTS="-Dcom.sun.management.jmxremote $HADOOP_BALANCER_OPTS"
  25. export HADOOP_JOBTRACKER_OPTS="-Dcom.sun.management.jmxremote $HADOOP_JOBTRACKER_OPTS"
  26. # export HADOOP_TASKTRACKER_OPTS=
  27. # The following applies to multiple commands (fs, dfs, fsck, distcp etc)
  28. # export HADOOP_CLIENT_OPTS

  30. # Extra ssh options. Empty by default.
  31. # export HADOOP_SSH_OPTS="-o ConnectTimeout=1 -o SendEnv=HADOOP_CONF_DIR"

  33. # Where log files are stored. $HADOOP_HOME/logs by default.
  34. # export HADOOP_LOG_DIR=${HADOOP_HOME}/logs

  36. # File naming remote slave hosts. $HADOOP_HOME/conf/slaves by default.
  37. # export HADOOP_SLAVES=${HADOOP_HOME}/conf/slaves

  39. # host:path where hadoop code should be rsync'd from. Unset by default.
  40. # export HADOOP_MASTER=master:/home/$USER/src/hadoop

  42. # Seconds to sleep between slave commands. Unset by default. This
  43. # can be useful in large clusters, where, e.g., slave rsyncs can
  44. # otherwise arrive faster than the master can service them.
  45. # export HADOOP_SLAVE_SLEEP=0.1

  47. # The directory where pid files are stored. /tmp by default.
  48. # export HADOOP_PID_DIR=/var/hadoop/pids

  50. # A string representing this instance of hadoop. $USER by default.
  51. # export HADOOP_IDENT_STRING=$USER

  53. # The scheduling priority for daemon processes. See 'man nice'.
  54. # export HADOOP_NICENESS=10
<4>conf/core-site.xml
     这里主要配置fs.default.name（用来指定namenode）和hadoop.tmp.dir（默认的hdfs的tmp目录位置），这里可以不设置hadoop.tmp.dir，那么就会保存在默认的/tmp下，每次重启机器都会丢失数据。

点击(此处)折叠或打开

  1. <?xml version="1.0"?>
  2. <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
  3.   
  4. <!-- Put site-specific property overrides in this file. -->

  6. <configuration>
  7.    <property>
  8.        <name>fs.default.name</name>
  9.        <value>hdfs://localhost:9000</value>
  10.    </property>
  11.    <property>
  12.        <name>hadoop.tmp.dir</name>
  13.        <value>/home/hadoop/hdfs/tmp</value>
  14.    </property>

  16. </configuration>
<5>hdfs-site.xml
     这里的dfs.replication用来设置每份数据块的副本数目，默认是3，因为我们是在单机上配置的伪分布模式，因此设为1。dfs.name.dir和dfs.data.dir非常重要，用来设置存放hdfs中namenode和datanode数据的本地存放位置。这里如果设置不好，后续会出现多个错误。当然你也可以不设置采用默认的/tmp下的目录，但是同样重启会丢失数据。

点击(此处)折叠或打开

  1. <?xml version="1.0"?>
  2. <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

  4. <!-- Put site-specific property overrides in this file. -->

  6. <configuration>
  7.    <property>
  8.        <name>dfs.replication</name>
  9.        <value>1</value>
  10.    </property>
  11.    <property>
  12.        <name>dfs.name.dir</name>
  13.        <value>/home/hadoop/hdfs/name</value>
  14.    </property>
  15.    <property>
  16.        <name>dfs.data.dir</name>
  17.        <value>/home/hadoop/hdfs/data</value>
  18.    </property>

  20. </configuration>
<6>mapred-site.xml

点击(此处)折叠或打开

  1. <?xml version="1.0"?>
  2. <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

  4. <!-- Put site-specific property overrides in this file. -->

  6. <configuration>
  7.    <property>
  8.        <name>mapred.job.tracker</name>
  9.        <value>localhost:9001</value>
  10.    </property>

  12. </configuration>
      然后就是运行测试了，把hadoop-1.0.3/bin加入到/etc/profile中的PATH路径中，方便我们执行Hadoop命令。运行start-all.sh后出现了问题，jps查看namenode无法启动，使用hadoop namenode -format也不能成功，查看日志：

    提示我们存储的HDFS目录要么不存在要么没有权限：
FSNamesystem initialization failed.
org.apache.hadoop.hdfs.server.common.InconsistentFSStateException:Directory /home/hadoop/hdfs/name is in an inconsistent state: storage directory does not exist or is not accessible.
查看发现已经生成了hdfs目录，那么问题就是权限了，将/home下的hadoop目录权限由755设为775，然后重新运行hadoop，成功：

      在进行安装hbase之前，我们先来按照官方的方法测试一下伪分布式的hadoop，看看安装是否成功：首先将conf下的所有文件拷贝到hdfs上的input目录中，然后运行jar文件将结果存储到hdfs中的output目录中，最后从output目录中查看结果：


点击(此处)折叠或打开

  1. $ bin/hadoop fs -put conf input

  3. $ bin/hadoop jar hadoop-examples-*.jar grep input output 'dfs[a-z.]+'

  5. Copy the output files from the distributed filesystem to the local filesytem and examine them:
  6. $ bin/hadoop fs -get output output
  7. $ cat output/*

  9. or

  11. View the output files on the distributed filesystem:
  12. $ bin/hadoop fs -cat output/*

  14. When you're done, stop the daemons with:
  15. $ bin/stop-all.sh
二、安装hbase
     这里选择的版本是hbase-0.90.0，首先从google中直接搜索该版本，下载后解压得到hbase-0.90.0。同hadoop一样，这里我们的主要工作同样是修改配置文件：
<1> 修改/etc/hosts文件，将127.0.1.1 hadoop修改为127.0.0.1
<2> 设置ulimits：
修改/etc/security/limits.conf，添加：
hadoop -  nofile 32768
hadoop soft/hard nproc 32000
修改/etc/pam.d/common-session ，添加
session required pam_limits.so
<3>修改conf/hbase-env.xml
     这里主要设置JAVA_HOME目录，HBASE_LOG_DIR路径以及启用hbase自带的zookeeper

点击(此处)折叠或打开

  1. #
  2. #/**
  3. # * Copyright 2007 The Apache Software Foundation
  4. # *
  5. # * Licensed to the Apache Software Foundation (ASF) under one
  6. # * or more contributor license agreements. See the NOTICE file
  7. # * distributed with this work for additional information
  8. # * regarding copyright ownership. The ASF licenses this file
  9. # * to you under the Apache License, Version 2.0 (the
  10. # * "License"); you may not use this file except in compliance
  11. # * with the License. You may obtain a copy of the License at
  12. # *
  13. # * http://
  14. # *
  15. # * Unless required by applicable law or agreed to in writing, software
  16. # * distributed under the License is distributed on an "AS IS" BASIS,
  17. # * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  18. # * See the License for the specific language governing permissions and
  19. # * limitations under the License.
  20. # */

  22. # Set environment variables here.

  24. # The java implementation to use. Java 1.6 required.
  25.  export JAVA_HOME=/home/hadoop/platform/jdk1.6.0_45

  27. # Extra Java CLASSPATH elements. Optional.
  28. # export HBASE_CLASSPATH=

  30. # The maximum amount of heap to use, in MB. Default is 1000.
  31. # export HBASE_HEAPSIZE=1000

  33. # Extra Java runtime options.
  34. # Below are what we set by default. May only work with SUN JVM.
  35. # For more on why as well as other possible settings,
  36. # see http://wiki.apache.org/hadoop/PerformanceTuning
  37. export HBASE_OPTS="$HBASE_OPTS -ea -XX:+HeapDumpOnOutOfMemoryError -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode"

  39. # Uncomment below to enable java garbage collection logging.
  40. # export HBASE_OPTS="$HBASE_OPTS -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:$HBASE_HOME/logs/gc-hbase.log"

  42. # Uncomment and adjust to enable JMX exporting
  43. # See jmxremote.password and jmxremote.access in $JRE_HOME/lib/management to configure remote password access.
  44. # More details at: http://java.sun.com/javase/6/docs/technotes/guides/management/agent.html
  45. #
  46. # export HBASE_JMX_BASE="-Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false"
  47. # export HBASE_MASTER_OPTS="$HBASE_MASTER_OPTS $HBASE_JMX_BASE -Dcom.sun.management.jmxremote.port=10101"
  48. # export HBASE_REGIONSERVER_OPTS="$HBASE_REGIONSERVER_OPTS $HBASE_JMX_BASE -Dcom.sun.management.jmxremote.port=10102"
  49. # export HBASE_THRIFT_OPTS="$HBASE_THRIFT_OPTS $HBASE_JMX_BASE -Dcom.sun.management.jmxremote.port=10103"
  50. # export HBASE_ZOOKEEPER_OPTS="$HBASE_ZOOKEEPER_OPTS $HBASE_JMX_BASE -Dcom.sun.management.jmxremote.port=10104"

  52. # File naming hosts on which HRegionServers will run. $HBASE_HOME/conf/regionservers by default.
  53. # export HBASE_REGIONSERVERS=${HBASE_HOME}/conf/regionservers

  55. # Extra ssh options. Empty by default.
  56. # export HBASE_SSH_OPTS="-o ConnectTimeout=1 -o SendEnv=HBASE_CONF_DIR"

  58. # Where log files are stored. $HBASE_HOME/logs by default.
  59. export HBASE_LOG_DIR=${HBASE_HOME}/logs

  61. # A string representing this instance of hbase. $USER by default.
  62. # export HBASE_IDENT_STRING=$USER

  64. # The scheduling priority for daemon processes. See 'man nice'.
  65. # export HBASE_NICENESS=10

  67. # The directory where pid files are stored. /tmp by default.
  68. # export HBASE_PID_DIR=/var/hadoop/pids

  70. # Seconds to sleep between slave commands. Unset by default. This
  71. # can be useful in large clusters, where, e.g., slave rsyncs can
  72. # otherwise arrive faster than the master can service them.
  73. # export HBASE_SLAVE_SLEEP=0.1

  75. # Tell HBase whether it should manage it's own instance of Zookeeper or not.
  76.  export HBASE_MANAGES_ZK=true
<4>配置conf/hbase-site.xml
     这里要将我们的hbase设置为伪分布模式，因此除了设置hbase的根目录hbase.rootdir外，还需要设置hbase.cluster.distributed和hbase.zookeeper.quorum两个参数；至于zookeeper.znode.parent是指定zookeeper的相对目录。

点击(此处)折叠或打开

  1. <?xml version="1.0"?>
  2. <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
  3. <!--
  4. /**
  5.  * Copyright 2010 The Apache Software Foundation
  6.  *
  7.  * Licensed to the Apache Software Foundation (ASF) under one
  8.  * or more contributor license agreements. See the NOTICE file
  9.  * distributed with this work for additional information
  10.  * regarding copyright ownership. The ASF licenses this file
  11.  * to you under the Apache License, Version 2.0 (the
  12.  * "License"); you may not use this file except in compliance
  13.  * with the License. You may obtain a copy of the License at
  14.  *
  15.  * http://
  16.  *
  17.  * Unless required by applicable law or agreed to in writing, software
  18.  * distributed under the License is distributed on an "AS IS" BASIS,
  19.  * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  20.  * See the License for the specific language governing permissions and
  21.  * limitations under the License.
  22.  */
  23. -->
  24. <configuration>
  25.     <property>
  26.         <name>hbase.rootdir</name>
  27.         <value>hdfs://localhost:9000/hbase</value>
  28.     </property>
  29.         <property>
  30.         <name>hbase.cluster.distributed</name>
  31.         <value>true</value>
  32.     </property>
  33.     <property>
  34.         <name>hbase.zookeeper.quorum</name>
  35.         <value>localhost</value>
  36.     </property>
  37.     <property>
  38.         <name>zookeeper.znode.parent</name>
  39.         <value>/hbase</value>
  40.     </property>



  44. </configuration>

<5>运行测试      
      然后同样将hbase下的bin目录加入到PATH中，运行start-hbase.sh后发现HMaster未启动，查看日志发现以下提示：
FATAL: org.apache.hadoop.hbase.master.HMaster: Unhandled exception. Starting shutdown. java.io.IOException:
Call to localhost/127.0.0.1:9000 failed on local exception: java.io.EOFException

      原因是需要将hadoop中的hadoop-core-1.0.4.jar覆盖掉hbase-0.90.0/lib下的hadoop-core-0.20-append-r1056947.jar文件，再次运行提示了新错误：java.lang.NoClassDefFoundError


     看来是找不到相关的类，所以我们需要将hadoop-1.0.3下lib目录的jar文件全部拷贝到hbase下的lib目录下，再次运行，顺利通过：

<5>运行hbase shell
     Hbase为我们提供了shell接口进行测试，可以使用create命令创建一个新表，然后利用put命令添加新的行，使用scan命令查看建立的表：

     好了，hadoop和hbase的安装到此基本结束，明天开始配置Sleuthkit。
阅读(530) | 评论(0) | 转发(0) |
0

上一篇：c++复制构造函数的形参为什么规定为引用类型？

下一篇：是真佛只道家常——leo鉴书(15)

给主人留下些什么吧！~~
关于我们 | 关于IT168 | 联系方式 | 广告合作 | 法律声明 | 免费注册

Copyright 2001-2010 ChinaUnix.net All Rights Reserved 北京皓辰网域网络信息技术有限公司. 版权所有

感谢所有关心和支持过ChinaUnix的朋友们

16024965号-6