windows下Eclipse开发MR程序出现的问题之二（插件形式）

  在Eclipse下开发MR程序时，有时候很容易犯一些小错误甚至是低级错误，以下就是一种比较常见的错误。 java.io.IOException: Type mismatch in value from map: expected org.apache.hadoop.io.DoubleWritable, recieved org.apache.hadoop.io.Text

2012-5-29 10:26:57 org.apache.hadoop.security.UserGroupInformation initialize
信息: JAAS Configuration already set up for Hadoop, not re-installing.
2012-5-29 10:26:58 org.apache.hadoop.mapreduce.lib.input.FileInputFormat listStatus
信息: Total input paths to process : 1
2012-5-29 10:26:58 org.apache.hadoop.util.NativeCodeLoader
警告: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2012-5-29 10:26:58 org.apache.hadoop.io.compress.snappy.LoadSnappy
警告: Snappy native library not loaded
2012-5-29 10:26:58 org.apache.hadoop.mapred.JobClient monitorAndPrintJob
信息: Running job: job_201205242029_0035
2012-5-29 10:26:59 org.apache.hadoop.mapred.JobClient monitorAndPrintJob
信息:  map 0% reduce 0%
2012-5-29 10:27:06 org.apache.hadoop.mapred.JobClient monitorAndPrintJob
信息: Task Id : attempt_201205242029_0035_m_000001_0, Status : FAILED
java.io.IOException: Type mismatch in value from map: expected org.apache.hadoop.io.DoubleWritable, recieved org.apache.hadoop.io.Text
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:876)
at org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:574)
at org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
at com.loogn.hadoop.practice.AverageByAttribute$AverageByAttributeMapper.map(AverageByAttribute.java:30)
at com.loogn.hadoop.practice.AverageByAttribute$AverageByAttributeMapper.map(AverageByAttribute.java:1)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:647)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:323)
at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGro
2012-5-29 10:27:06 org.apache.hadoop.mapred.JobClient monitorAndPrintJob
信息: Task Id : attempt_201205242029_0035_m_000003_0, Status : FAILED
java.io.IOException: Type mismatch in value from map: expected org.apache.hadoop.io.DoubleWritable, recieved org.apache.hadoop.io.Text
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:876)
at org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:574)
at org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
at com.loogn.hadoop.practice.AverageByAttribute$AverageByAttributeMapper.map(AverageByAttribute.java:30)
at com.loogn.hadoop.practice.AverageByAttribute$AverageByAttributeMapper.map(AverageByAttribute.java:1)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:647)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:323)
at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGro
2012-5-29 10:27:08 org.apache.hadoop.mapred.JobClient monitorAndPrintJob
信息: Task Id : attempt_201205242029_0035_m_000002_0, Status : FAILED
java.io.IOException: Type mismatch in value from map: expected org.apache.hadoop.io.DoubleWritable, recieved org.apache.hadoop.io.Text
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:876)
at org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:574)
at org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
at com.loogn.hadoop.practice.AverageByAttribute$AverageByAttributeMapper.map(AverageByAttribute.java:30)
at com.loogn.hadoop.practice.AverageByAttribute$AverageByAttributeMapper.map(AverageByAttribute.java:1)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:647)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:323)
at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGro
2012-5-29 10:27:10 org.apache.hadoop.mapred.JobClient monitorAndPrintJob
信息:  map 10% reduce 0%
2012-5-29 10:27:12 org.apache.hadoop.mapred.JobClient monitorAndPrintJob
信息: Task Id : attempt_201205242029_0035_m_000003_1, Status : FAILED
java.io.IOException: Type mismatch in value from map: expected org.apache.hadoop.io.DoubleWritable, recieved org.apache.hadoop.io.Text
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:876)
at org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:574)
at org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
at com.loogn.hadoop.practice.AverageByAttribute$AverageByAttributeMapper.map(AverageByAttribute.java:30)
at com.loogn.hadoop.practice.AverageByAttribute$AverageByAttributeMapper.map(AverageByAttribute.java:1)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:647)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:323)
at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGro
2012-5-29 10:27:12 org.apache.hadoop.mapred.JobClient monitorAndPrintJob
信息: Task Id : attempt_201205242029_0035_m_000001_1, Status : FAILED
java.io.IOException: Type mismatch in value from map: expected org.apache.hadoop.io.DoubleWritable, recieved org.apache.hadoop.io.Text
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:876)
at org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:574)
at org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
at com.loogn.hadoop.practice.AverageByAttribute$AverageByAttributeMapper.map(AverageByAttribute.java:30)
at com.loogn.hadoop.practice.AverageByAttribute$AverageByAttributeMapper.map(AverageByAttribute.java:1)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:647)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:323)
at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGro
2012-5-29 10:27:13 org.apache.hadoop.mapred.JobClient monitorAndPrintJob
信息:  map 16% reduce 0%
2012-5-29 10:27:13 org.apache.hadoop.mapred.JobClient monitorAndPrintJob
信息: Task Id : attempt_201205242029_0035_m_000002_1, Status : FAILED
java.io.IOException: Type mismatch in value from map: expected org.apache.hadoop.io.DoubleWritable, recieved org.apache.hadoop.io.Text
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:876)
at org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:574)
at org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
at com.loogn.hadoop.practice.AverageByAttribute$AverageByAttributeMapper.map(AverageByAttribute.java:30)
at com.loogn.hadoop.practice.AverageByAttribute$AverageByAttributeMapper.map(AverageByAttribute.java:1)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:647)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:323)
at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGro
2012-5-29 10:27:16 org.apache.hadoop.mapred.JobClient monitorAndPrintJob
信息: Task Id : attempt_201205242029_0035_m_000000_0, Status : FAILED
java.io.IOException: Type mismatch in value from map: expected org.apache.hadoop.io.DoubleWritable, recieved org.apache.hadoop.io.Text
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:876)
at org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:574)
at org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
at com.loogn.hadoop.practice.AverageByAttribute$AverageByAttributeMapper.map(AverageByAttribute.java:30)
at com.loogn.hadoop.practice.AverageByAttribute$AverageByAttributeMapper.map(AverageByAttribute.java:1)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:647)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:323)
at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGro
2012-5-29 10:27:17 org.apache.hadoop.mapred.JobClient monitorAndPrintJob
信息:  map 0% reduce 0%
2012-5-29 10:27:18 org.apache.hadoop.mapred.JobClient monitorAndPrintJob
信息: Task Id : attempt_201205242029_0035_m_000001_2, Status : FAILED
java.io.IOException: Type mismatch in value from map: expected org.apache.hadoop.io.DoubleWritable, recieved org.apache.hadoop.io.Text
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:876)
at org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:574)
at org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
at com.loogn.hadoop.practice.AverageByAttribute$AverageByAttributeMapper.map(AverageByAttribute.java:30)
at com.loogn.hadoop.practice.AverageByAttribute$AverageByAttributeMapper.map(AverageByAttribute.java:1)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:647)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:323)
at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGro
2012-5-29 10:27:18 org.apache.hadoop.mapred.JobClient monitorAndPrintJob
信息: Task Id : attempt_201205242029_0035_m_000002_2, Status : FAILED
java.io.IOException: Type mismatch in value from map: expected org.apache.hadoop.io.DoubleWritable, recieved org.apache.hadoop.io.Text
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:876)
at org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:574)
at org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
at com.loogn.hadoop.practice.AverageByAttribute$AverageByAttributeMapper.map(AverageByAttribute.java:30)
at com.loogn.hadoop.practice.AverageByAttribute$AverageByAttributeMapper.map(AverageByAttribute.java:1)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:647)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:323)
at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGro
2012-5-29 10:27:18 org.apache.hadoop.mapred.JobClient monitorAndPrintJob
信息: Task Id : attempt_201205242029_0035_m_000003_2, Status : FAILED
java.io.IOException: Type mismatch in value from map: expected org.apache.hadoop.io.DoubleWritable, recieved org.apache.hadoop.io.Text
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:876)
at org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:574)
at org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
at com.loogn.hadoop.practice.AverageByAttribute$AverageByAttributeMapper.map(AverageByAttribute.java:30)
at com.loogn.hadoop.practice.AverageByAttribute$AverageByAttributeMapper.map(AverageByAttribute.java:1)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:647)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:323)
at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGro
2012-5-29 10:27:24 org.apache.hadoop.mapred.JobClient monitorAndPrintJob
信息: Job complete: job_201205242029_0035
2012-5-29 10:27:24 org.apache.hadoop.mapred.Counters log
信息: Counters: 8
2012-5-29 10:27:24 org.apache.hadoop.mapred.Counters log
信息:   Job Counters
2012-5-29 10:27:24 org.apache.hadoop.mapred.Counters log
信息:     SLOTS_MILLIS_MAPS=85914
2012-5-29 10:27:24 org.apache.hadoop.mapred.Counters log
信息:     Total time spent by all reduces waiting after reserving slots (ms)=0
2012-5-29 10:27:24 org.apache.hadoop.mapred.Counters log
信息:     Total time spent by all maps waiting after reserving slots (ms)=0
2012-5-29 10:27:24 org.apache.hadoop.mapred.Counters log
信息:     Rack-local map tasks=7
2012-5-29 10:27:24 org.apache.hadoop.mapred.Counters log
信息:     Launched map tasks=14
2012-5-29 10:27:24 org.apache.hadoop.mapred.Counters log
信息:     Data-local map tasks=7
2012-5-29 10:27:24 org.apache.hadoop.mapred.Counters log
信息:     SLOTS_MILLIS_REDUCES=0
2012-5-29 10:27:24 org.apache.hadoop.mapred.Counters log
信息:     Failed map tasks=1

  之所以会出现这样的情况，这主要是因为程序中设置map输出格式的时候出了问题，如：

job.setOutputKeyClass(Text.class);
job.setOutputValueClass(DoubleWritable.class);

  在自定义mapper类中，输出格式是Text，所以与预期的DoubleWritable相悖，所以解决这样问题的方法是只需要将上面的第二行代码改为job.setOutputValueClass(Text.class)即可！

  出现这样的问题的主要原因是，没有搞清楚job.setOutputKeyClass和job.setOutputValueClass的意思，有的人会认为这是设置Reducer的输出格式，而其实是设置Map输出键值格式！