Chinaunix首页 | 论坛 | 博客
  • 博客访问: 325926
  • 博文数量: 31
  • 博客积分: 393
  • 博客等级: 一等列兵
  • 技术积分: 388
  • 用 户 组: 普通用户
  • 注册时间: 2012-08-26 10:23
文章分类

全部博文(31)

文章存档

2013年(16)

2012年(15)

分类: 云计算

2012-09-24 18:29:28

hadoop平台搭起来首先想做的就是写个代码测试一下啦。但是貌似hadoop的java包有点搞不懂是虾米回事,这篇文章写得很清楚了,一看就懂!
编译好包后用一下命令执行mapreduce代码。
[@HADOOP_HOME]#bin/hadoop jar XXX.jar class_name inputfile outputfile
bin/hadoop jar #指定hadoop的命令为执行jar包中的程序
class_name     #XXX.jar包中的那个入口类名
inputfile      #这个是已经存在,位于:hdfs/user/root/inputfile
outputfile     #这个是hdfs为我们创建的用于存放结果的文件 hdfs/user/root/inputfile(系统自动创建)
 
一下是copy的具体的编译hadoop下java代码的方法:

首先确保hadoop已经正确安装、配置以及运行。

拷贝WordCount.java到我们的文件夹,下载的hadoop里带有WordCount.java,路径为:

hadoop-0.20.203.0/src/examples/org/apache/hadoop/examples/WordCount.java

进行拷贝操作

  1. [hadoop@localhost~]$ cp hadoop-0.20.203.0/src/examples/org/apache/hadoop/examples/WordCount.java ~
[hadoop@localhost~]$ cp hadoop-0.20.203.0/src/examples/org/apache/hadoop/examples/WordCount.java ~

  1. [hadoop@localhost~]$ ls
[hadoop@localhost~]$ ls

Desktop hadoop-0.20.203.0 WordCount.java

在当前目录下创建一个用来存放WordCount.class的文件夹

  1. [hadoop@localhost~]$ mkdir classes
[hadoop@localhost~]$ mkdir classes

编译WordCount.java

  1. [hadoop@localhost~]$ javac -classpath hadoop-0.20.203.0/hadoop-core-0.20.203.0.jar -d classesWordCount.java
[hadoop@localhost~]$ javac -classpath hadoop-0.20.203.0/hadoop-core-0.20.203.0.jar -d classesWordCount.java

上面的方法是按照hadoop自带的doc文档进行编译的,如果发现报错,出现如下异常

WordCount.java:53:error: cannot access Options

String[] otherArgs = new GenericOptionsParser(conf, args).getRemainingArgs();

^

class file for org.apache.commons.cli.Optionsnot found

1 error

则按如下进行编译

  1. [hadoop@localhost~]$ javac -classpath hadoop-0.20.203.0/hadoop-core-0.20.203.0.jar:hadoop-0.20.203.0/lib/commons-cli-1.2.jar -d classes WordCount.java
[hadoop@localhost~]$ javac -classpath hadoop-0.20.203.0/hadoop-core-0.20.203.0.jar:hadoop-0.20.203.0/lib/commons-cli-1.2.jar -d classes WordCount.java

编译成功,classes下会出现一个org的文件夹

  1. [hadoop@localhost~]$ ls classes
[hadoop@localhost~]$ ls classes

org

对编译好的class进行打包

  1. [hadoop@localhost~]$ jar -cvf wordcount.jar -C classes/ .
[hadoop@localhost~]$ jar -cvf wordcount.jar -C classes/ .

added manifest

adding: org/(in =0) (out= 0)(stored 0%)

adding:org/apache/(in = 0) (out= 0)(stored 0%)

adding:org/apache/hadoop/(in = 0) (out= 0)(stored 0%)

adding: org/apache/hadoop/examples/(in= 0) (out= 0)(stored 0%)

adding:org/apache/hadoop/examples/WordCount$TokenizerMapper.class(in = 1790) (out=765)(deflated 57%)

adding:org/apache/hadoop/examples/WordCount$IntSumReducer.class(in = 1793) (out=746)(deflated 58%)

adding:org/apache/hadoop/examples/WordCount.class(in = 1911) (out= 996)(deflated 47%)

  1. [hadoop@localhost~]$ ls
[hadoop@localhost~]$ ls

classes Desktop hadoop-0.20.203.0 wordcount.jar WordCount.java

到此为止,java文件的编译工作已经完成

现在为测试工作准备需要的测试文件,创建2个文件file01、file02,内容如下

  1. [hadoop@localhost~]$ ls
[hadoop@localhost~]$ ls

classes Desktop file01 file02 hadoop-0.20.203.0 wordcount.jar WordCount.java

  1. [hadoop@localhost~]$ cat file01
[hadoop@localhost~]$ cat file01

Hello World Bye World

  1. [hadoop@localhost~]$ cat file02
[hadoop@localhost~]$ cat file02

Hello Hadoop Goodbye Hadoop

启动hadoop,在hadoop中创建input文件夹

  1. [hadoop@localhost~]$ hadoop-0.20.203.0/bin/hadoop dfs -ls
  2. [hadoop@localhost~]$ hadoop-0.20.203.0/bin/hadoop dfs -mkdir input
  3. [hadoop@localhost~]$ hadoop-0.20.203.0/bin/hadoop dfs -ls
[hadoop@localhost~]$ hadoop-0.20.203.0/bin/hadoop dfs -ls [hadoop@localhost~]$ hadoop-0.20.203.0/bin/hadoop dfs -mkdir input [hadoop@localhost~]$ hadoop-0.20.203.0/bin/hadoop dfs -ls

Found 1 items

drwxr-xr-x - hadoop supergroup 0 2011-11-23 05:20 /user/hadoop/input

把file01、file02上传input中

  1. [hadoop@localhost~]$ hadoop-0.20.203.0/bin/hadoop fs -put file01 input
  2. [hadoop@localhost~]$ hadoop-0.20.203.0/bin/hadoop fs -put file02 input
  3. [hadoop@localhost~]$ hadoop-0.20.203.0/bin/hadoop fs -ls input
[hadoop@localhost~]$ hadoop-0.20.203.0/bin/hadoop fs -put file01 input [hadoop@localhost~]$ hadoop-0.20.203.0/bin/hadoop fs -put file02 input [hadoop@localhost~]$ hadoop-0.20.203.0/bin/hadoop fs -ls input

Found 2 items

-rw-r--r-- 1 hadoop supergroup 22 2011-11-23 05:22/user/hadoop/input/file01

-rw-r--r-- 1 hadoop supergroup 28 2011-11-23 05:22/user/hadoop/input/file02

运行及输出过程如下

  1. [hadoop@localhost~]$ hadoop-0.20.203.0/bin/hadoop jar wordcount.jar org.apache.hadoop.examples.WordCount input output
[hadoop@localhost~]$ hadoop-0.20.203.0/bin/hadoop jar wordcount.jar org.apache.hadoop.examples.WordCount input output

11/11/23 05:25:11INFO input.FileInputFormat: Total input paths to process : 2

11/11/23 05:25:12INFO mapred.JobClient: Running job: job_201111230519_0001

11/11/23 05:25:13INFO mapred.JobClient: map 0% reduce 0%

11/11/23 05:25:45INFO mapred.JobClient: map 50% reduce 0%

11/11/23 05:25:48INFO mapred.JobClient: map 100% reduce0%

11/11/23 05:26:03INFO mapred.JobClient: map 100% reduce 100%

11/11/23 05:26:07INFO mapred.JobClient: Job complete: job_201111230519_0001

11/11/23 05:26:07INFO mapred.JobClient: Counters: 25

11/11/23 05:26:07INFO mapred.JobClient: Job Counters

11/11/23 05:26:07INFO mapred.JobClient: Launchedreduce tasks=1

11/11/23 05:26:07INFO mapred.JobClient: SLOTS_MILLIS_MAPS=48562

11/11/23 05:26:07INFO mapred.JobClient: Total timespent by all reduces waiting after reserving slots (ms)=0

11/11/23 05:26:07INFO mapred.JobClient: Total timespent by all maps waiting after reserving slots (ms)=0

11/11/23 05:26:07INFO mapred.JobClient: Launched maptasks=2

11/11/23 05:26:07INFO mapred.JobClient: Data-local maptasks=2

11/11/23 05:26:07INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=16678

11/11/23 05:26:07INFO mapred.JobClient: File OutputFormat Counters

11/11/23 05:26:07INFO mapred.JobClient: BytesWritten=41

11/11/23 05:26:07INFO mapred.JobClient: FileSystemCounters

11/11/23 05:26:07INFO mapred.JobClient: FILE_BYTES_READ=79

11/11/23 05:26:07INFO mapred.JobClient: HDFS_BYTES_READ=272

11/11/23 05:26:07INFO mapred.JobClient: FILE_BYTES_WRITTEN=63583

11/11/23 05:26:07INFO mapred.JobClient: HDFS_BYTES_WRITTEN=41

11/11/23 05:26:07INFO mapred.JobClient: File Input FormatCounters

11/11/23 05:26:07INFO mapred.JobClient: Bytes Read=50

11/11/23 05:26:07INFO mapred.JobClient: Map-ReduceFramework

11/11/23 05:26:07INFO mapred.JobClient: Reduce inputgroups=5

11/11/23 05:26:07INFO mapred.JobClient: Map outputmaterialized bytes=85

11/11/23 05:26:07INFO mapred.JobClient: Combine outputrecords=6

11/11/23 05:26:07INFO mapred.JobClient: Map inputrecords=2

11/11/23 05:26:07INFO mapred.JobClient: Reduce shufflebytes=85

11/11/23 05:26:07INFO mapred.JobClient: Reduce outputrecords=5

11/11/23 05:26:07INFO mapred.JobClient: SpilledRecords=12

11/11/23 05:26:07INFO mapred.JobClient: Map outputbytes=82

11/11/23 05:26:07INFO mapred.JobClient: Combine inputrecords=8

11/11/23 05:26:07INFO mapred.JobClient: Map outputrecords=8

11/11/23 05:26:07INFO mapred.JobClient: SPLIT_RAW_BYTES=222

11/11/23 05:26:07INFO mapred.JobClient: Reduce inputrecords=6

进行结果的查看

  1. [hadoop@localhost~]$ hadoop-0.20.203.0/bin/hadoop fs -ls
[hadoop@localhost~]$ hadoop-0.20.203.0/bin/hadoop fs -ls

Found 2 items

drwxr-xr-x - hadoop supergroup 0 2011-11-23 05:22 /user/hadoop/input

drwxr-xr-x - hadoop supergroup 0 2011-11-23 05:26/user/hadoop/output

发现在hadoop中多了一个output文件夹,查看output中的文件信息

  1. [hadoop@localhost~]$ hadoop-0.20.203.0/bin/hadoop fs -ls output
[hadoop@localhost~]$ hadoop-0.20.203.0/bin/hadoop fs -ls output

Found 3 items

-rw-r--r-- 1 hadoop supergroup 0 2011-11-23 05:26/user/hadoop/output/_SUCCESS

drwxr-xr-x - hadoop supergroup 0 2011-11-23 05:25/user/hadoop/output/_logs

-rw-r--r-- 1 hadoop supergroup 41 2011-11-23 05:25/user/hadoop/output/part-r-00000

对WordCount的运行结果进行查看

  1. [hadoop@localhost~]$ hadoop-0.20.203.0/bin/hadoop fs -cat output/part-r-00000
[hadoop@localhost~]$ hadoop-0.20.203.0/bin/hadoop fs -cat output/part-r-00000

Bye 1

Goodbye 1

Hadoop 2

Hello 2

World 2

至此,hadoop下的WordCount实例运行结束,如果还想重新运行一遍,这需把hadoop下的output文件夹删除,因为hadoop为了保证结果的正确性,存在输出的文件夹的话,就会报异常,异常如下

  1. [hadoop@localhost~]$ hadoop-0.20.203.0/bin/hadoop jar wordcount.jar org.apache.hadoop.examples.WordCount input output
[hadoop@localhost~]$ hadoop-0.20.203.0/bin/hadoop jar wordcount.jar org.apache.hadoop.examples.WordCount input output

11/11/23 05:33:05INFO mapred.JobClient: Cleaning up the staging areahdfs://localhost:9000/tmp/hadoop-hadoop/mapred/staging/hadoop/.staging/job_201111230519_0002

Exception inthread "main" org.apache.hadoop.mapred.FileAlreadyExist***ception:Output directory output already exists

atorg.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:134)

atorg.apache.hadoop.mapred.JobClient$2.run(JobClient.java:830)

atorg.apache.hadoop.mapred.JobClient$2.run(JobClient.java:791)

atjava.security.AccessController.doPrivileged(Native Method)

at javax.security.auth.Subject.doAs(Subject.java:415)

atorg.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)

atorg.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:791)

atorg.apache.hadoop.mapreduce.Job.submit(Job.java:465)

atorg.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:494)

atorg.apache.hadoop.examples.WordCount.main(WordCount.java:67)

atsun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

atsun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

atjava.lang.reflect.Method.invoke(Method.java:601)

at org.apache.hadoop.util.RunJar.main(RunJar.java:156)

删除output文件夹的操作为

  1. [hadoop@localhost~]$ hadoop-0.20.203.0/bin/hadoop dfs -rmr output
[hadoop@localhost~]$ hadoop-0.20.203.0/bin/hadoop dfs -rmr output

Deletedhdfs://localhost:9000/user/hadoop/output

 
阅读(9880) | 评论(0) | 转发(0) |
给主人留下些什么吧!~~