全部博文(372)
2012年(372)
分类: 虚拟化
2012-02-24 15:10:57
1.安装jdk(自带了jre),单纯的jre是不够的,MapReduce的编写和Hadoop的编译都依赖于jdk。(注意jdk必须1.6以上,本教程使用的是jdk1.6.0_24)
2、安装cygwin,下载地址 本教程采用的是1.7.9
必须选择的安装包: Net Category----openssh和openssl
可选包:Editors Category---vim,方便修改配置文件
Devel Category----subversion
3. 配置cygwin的环境变量
点击桌面的cygwin图标,启动cygwin,执行如下命令
$vim /etc/profile
在最后添加如下行:
export JAVA_HOME=/cygdrive/d/Java/jdk1.6.0_24
export HADOOP_HOME=/cygdrive/d/hadoop-0.21.0
export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin
export CLASSPATH=$CLASSPATH:.:$JAVA_HOME/lib:$JAVA_HOME/jre/lib
4.在cygwin中安装sshd
点击桌面的cygwin图标,启动cygwin,执行如下命令
$ssh-host-config,当要求输入yes/no时,选择no。当看到“Have fun”,一般表示sshd服务安装成功。
5.启动sshd服务
在桌面上的“我的电脑”图标上单击右键,点击“管理”,在打开的窗口左侧菜单中选择“服务和应用程序”,在右侧的列表中,在sshd行上点击右键,选择“启动”
6.配置ssh登录
点击桌面的cygwin图标,启动cygwin,执行如下命令
#生成密钥文件
$ssh-keygen 一路回车即可
#加为信任证书
$cd ~/.ssh
$cp id_rsa.pub authorized_keys
#exit
重新点击桌面的cygwin图标,启动cygwin
$ssh localhost
如果不再提示输入密码,则成功
7.下载hadoop-0.21.0的安装包,hadoop-0.21.0.tar.gz
8.安装hadoop
将hadoop-0.21.0.tar.gz 解压,如解压到d:\hadoop-0.21.0
配置conf下的hadoop-env.sh ,zhi
只需要将JAVA_HOME修改成jdk的安装路径即可(注意此处的路径不是windows风格的目录d:\java\dk1.6.0_24,而是linux风格/cygwin/d/java/jdk1.6.0_24)
配置conf下的mapred.site ,增加如下代码
9. format a new distributed-files system
$bin/hadoop namecode -format
10.start the hadoop daemons
$bin/start-all.sh
11.browser the web interface for NameCode and JobTracker,by default they are available at
NameCode-
JobTracker-
12.common Exception
12.1解决Window环境下启动Hadoop时出现的 java.lang.NoClassDefFoundError: org/apache/hadoop/util/PlatformName 异常在Window下启动Hadoop-0.21.0版本时,会出现下面这样的错误提示:
执行 $ bin/hadoop start-all.sh之后,无法启动.
异常一
Exception in thread "main" java.lang.IllegalArgumentException: Invalid URI for NameNode address (check fs.defaultFS): file:/// has no authority.
localhost: at org.apache.hadoop.hdfs.server.namenode.NameNode.getAddress(NameNode.java:214)
localhost: at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.initialize(SecondaryNameNode.java:135)
localhost: at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.
localhost: at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.main(SecondaryNameNode.java:481)
hadoop无法正常启动(2)
异常二、
starting namenode, logging to /home/xixitie/hadoop/bin/../logs/hadoop-root-namenode-aist.out
localhost: starting datanode, logging to /home/xixitie/hadoop/bin/../logs/hadoop-root-datanode-aist.out
localhost: starting secondarynamenode, logging to /home/xixitie/hadoop/bin/../logs/hadoop-root-secondarynamenode-aist.out
localhost: Exception in thread "main" java.lang.NullPointerException
localhost: at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:134)
localhost: at org.apache.hadoop.hdfs.server.namenode.NameNode.getAddress(NameNode.java:156)
localhost: at org.apache.hadoop.hdfs.server.namenode.NameNode.getAddress(NameNode.java:160)
localhost: at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.initialize(SecondaryNameNode.java:131)
localhost: at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.
localhost: at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.main(SecondaryNameNode.java:469)
starting jobtracker, logging to /home/xixitie/hadoop/bin/../logs/hadoop-root-jobtracker-aist.out
localhost: starting tasktracker, logging to /home/xixitie/hadoop/bin/../logs/hadoop-root-tasktracker-aist.out
hadoop无法正常启动(3)
异常三、
starting namenode, logging to /home/xixitie/hadoop/bin/../logs/hadoop-root-namenode-aist.out
localhost: starting datanode, logging to /home/xixitie/hadoop/bin/../logs/hadoop-root-datanode-aist.out
localhost: Error: JAVA_HOME is not set.
localhost: starting secondarynamenode, logging to /home/xixitie/hadoop/bin/../logs/hadoop-root-secondarynamenode-aist.out
localhost: Error: JAVA_HOME is not set.
starting jobtracker, logging to /home/xixitie/hadoop/bin/../logs/hadoop-root-jobtracker-aist.out
localhost: starting tasktracker, logging to /home/xixitie/hadoop/bin/../logs/hadoop-root-tasktracker-aist.out
localhost: Error: JAVA_HOME is not set.
解决方法:
请在$hadoop/conf/hadoop-env.sh文件中配置JDK的环境变量
JAVA_HOME=/home/xixitie/jdk
CLASSPATH=$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
export JAVA_HOME CLASSPATH
异常四:mapred-site.xml配置中使用hdfs://localhost:9001,而不使用localhost:9001的配置
异常信息如下:
11/04/20 23:33:25 INFO security.Groups: Group mapping impl=org.apache.hadoop.sec urity.ShellBasedUnixGroupsMapping; cacheTimeout=300000
11/04/20 23:33:25 WARN fs.FileSystem: "localhost:9000" is a deprecated filesyste m name. Use "hdfs://localhost:9000/" instead.
11/04/20 23:33:25 WARN conf.Configuration: mapred.task.id is deprecated. Instead , use mapreduce.task.attempt.id
11/04/20 23:33:25 WARN fs.FileSystem: "localhost:9000" is a deprecated filesyste m name. Use "hdfs://localhost:9000/" instead.
11/04/20 23:33:25 WARN fs.FileSystem: "localhost:9000" is a deprecated filesyste m name. Use "hdfs://localhost:9000/" instead.
解决方法:
mapred-site.xml配置中使用hdfs://localhost:9000,而不使用localhost:9000的配置
hadoop无法正常启动(5)
异常五、no namenode to stop 问题的解决:
异常信息如下:11/04/20 21:48:50 INFO ipc.Client: Retrying connect to server: localhost/127.0.0 .1:9000. Already tried 0 time(s).
11/04/20 21:48:51 INFO ipc.Client: Retrying connect to server: localhost/127.0.0 .1:9000. Already tried 1 time(s).
11/04/20 21:48:52 INFO ipc.Client: Retrying connect to server: localhost/127.0.0 .1:9000. Already tried 2 time(s).
11/04/20 21:48:53 INFO ipc.Client: Retrying connect to server: localhost/127.0.0 .1:9000. Already tried 3 time(s).
11/04/20 21:48:54 INFO ipc.Client: Retrying connect to server: localhost/127.0.0 .1:9000. Already tried 4 time(s).
11/04/20 21:48:55 INFO ipc.Client: Retrying connect to server: localhost/127.0.0 .1:9000. Already tried 5 time(s).
11/04/20 21:48:56 INFO ipc.Client: Retrying connect to server: localhost/127.0.0 .1:9000. Already tried 6 time(s).
11/04/20 21:48:57 INFO ipc.Client: Retrying connect to server: localhost/127.0.0 .1:9000. Already tried 7 time(s).
11/04/20 21:48:58 INFO ipc.Client: Retrying connect to server: localhost/127.0.0 .1:9000. Already tried 8 time(s).
解决方法:
这个问题是由namenode没有启动起来引起的,为什么no namenode to stop,可能之前的一些数据对namenode有影响,
你需要执行:
$ bin/hadoop namenode -format
然后
$bin/hadoop start-all.sh
hadoop无法正常启动(6)
异常五、no datanode to stop 问题的解决:
有时数据结构出现问题会产生无法启动datanode的问题。
然后用 hadoop namenode -format 重新格式化后仍然无效,/tmp中的文件并没有清楚。
其实还需要清除/tmp/hadoop*里的文件。
执行步骤:
一、先删除hadoop:///tmp
hadoop fs -rmr /tmp
二、停止 hadoop
stop-all.sh
三、删除/tmp/hadoop*
rm -rf /tmp/hadoop*
四、格式化hadoop
hadoop namenode -format
五、启动hadoop
start-all.sh
之后即可解决这个datanode没法启动的问题