Chinaunix首页 | 论坛 | 博客

loogn_qiang的ChinaUnix博客loogn.blog.chinaunix.net

首页　| 　博文目录　| 　关于我

loogn_qiang

  • 博客访问： 302510
  • 博文数量： 22
  • 博客积分： 674
  • 博客等级： 上士
  • 技术积分： 792
  • 用 户 组： 普通用户
  • 注册时间： 2012-02-19 00:48
文章分类

全部博文（22）

文章存档

2013年（2）

2012年（20）

我的朋友
最近访客
推荐博文
相关博文
运行MR作业出现的问题之二

分类： 云计算

2012-04-06 14:23:40

  在运行《Hadoop开发者第三期》中利用hadoop访问传统关系型数据库系统时，出现了一系列的问题，其中一个问题是由代码中第43行引起的，此处会有两种情况：
  1）若在代码中将第43行代码写成：

点击(此处)折叠或打开

  1. DBConfiguration.configureDB(conf, "com.mysql.jdbc.Driver", "jdbc:mysql://localhost:3306/test", "root", "198921");
则会导致运行时出现以下错误：

点击(此处)折叠或打开

  1. 2012-04-06 14:38:31,831 INFO mapred.JobClient - Running job: job_201203301926_0052
  2. 2012-04-06 14:38:32,836 INFO mapred.JobClient - map 0% reduce 0%
  3. 2012-04-06 14:38:39,879 INFO mapred.JobClient - Task Id : attempt_201203301926_0052_m_000000_0, Status : FAILED
  4. java.lang.RuntimeException: java.lang.RuntimeException: com.mysql.jdbc.exceptions.jdbc4.Communication***ception: Communications link failure

  6. The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
  7.     at org.apache.hadoop.mapreduce.lib.db.DBInputFormat.setConf(DBInputFormat.java:158)
  8.     at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62)
  9.     at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
  10.     at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:606)
  11.     at org.apache.hadoop.mapred.MapTask.run(MapTask.java:323)
  12.     at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
  13.     at java.security.AccessController.doPrivileged(Native Method)
  14.     at javax.security.auth.Subject.doAs(Subject.java:396)
  15.     at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1157)
  16.     at org.apache.hadoop.mapred.Child.main(Child.java:264)
  17. Caused by: java.lang.RuntimeException: com.mysql.jdbc.exceptions
  18. 2012-04-06 14:38:39,944 INFO mapred.JobClient - Task Id : attempt_201203301926_0052_m_000001_0, Status : FAILED
  19. java.lang.RuntimeException: java.lang.RuntimeException: com.mysql.jdbc.exceptions.jdbc4.Communication***ception: Communications link failure

  21. The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
  22.     at org.apache.hadoop.mapreduce.lib.db.DBInputFormat.setConf(DBInputFormat.java:158)
  23.     at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62)
  24.     at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
  25.     at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:606)
  26.     at org.apache.hadoop.mapred.MapTask.run(MapTask.java:323)
  27.     at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
  28.     at java.security.AccessController.doPrivileged(Native Method)
  29.     at javax.security.auth.Subject.doAs(Subject.java:396)
  30.     at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1157)
  31.     at org.apache.hadoop.mapred.Child.main(Child.java:264)
  32. Caused by: java.lang.RuntimeException: com.mysql.jdbc.exceptions
  33. 2012-04-06 14:38:45,066 INFO mapred.JobClient - Task Id : attempt_201203301926_0052_m_000000_1, Status : FAILED
  34. java.lang.RuntimeException: java.lang.RuntimeException: com.mysql.jdbc.exceptions.jdbc4.Communication***ception: Communications link failure

  36. The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
  37.     at org.apache.hadoop.mapreduce.lib.db.DBInputFormat.setConf(DBInputFormat.java:158)
  38.     at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62)
  39.     at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
  40.     at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:606)
  41.     at org.apache.hadoop.mapred.MapTask.run(MapTask.java:323)
  42.     at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
  43.     at java.security.AccessController.doPrivileged(Native Method)
  44.     at javax.security.auth.Subject.doAs(Subject.java:396)
  45.     at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1157)
  46.     at org.apache.hadoop.mapred.Child.main(Child.java:264)
  47. Caused by: java.lang.RuntimeException: com.mysql.jdbc.exceptions
  48. 2012-04-06 14:38:45,153 INFO mapred.JobClient - Task Id : attempt_201203301926_0052_m_000001_1, Status : FAILED
  49. java.lang.RuntimeException: java.lang.RuntimeException: com.mysql.jdbc.exceptions.jdbc4.Communication***ception: Communications link failure

  51. The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
  52.     at org.apache.hadoop.mapreduce.lib.db.DBInputFormat.setConf(DBInputFormat.java:158)
  53.     at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62)
  54.     at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
  55.     at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:606)
  56.     at org.apache.hadoop.mapred.MapTask.run(MapTask.java:323)
  57.     at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
  58.     at java.security.AccessController.doPrivileged(Native Method)
  59.     at javax.security.auth.Subject.doAs(Subject.java:396)
  60.     at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1157)
  61.     at org.apache.hadoop.mapred.Child.main(Child.java:264)
  62. Caused by: java.lang.RuntimeException: com.mysql.jdbc.exceptions
  63. 2012-04-06 14:38:50,236 INFO mapred.JobClient - Task Id : attempt_201203301926_0052_m_000000_2, Status : FAILED
  64. java.lang.RuntimeException: java.lang.RuntimeException: com.mysql.jdbc.exceptions.jdbc4.Communication***ception: Communications link failure

  66. The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
  67.     at org.apache.hadoop.mapreduce.lib.db.DBInputFormat.setConf(DBInputFormat.java:158)
  68.     at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62)
  69.     at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
  70.     at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:606)
  71.     at org.apache.hadoop.mapred.MapTask.run(MapTask.java:323)
  72.     at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
  73.     at java.security.AccessController.doPrivileged(Native Method)
  74.     at javax.security.auth.Subject.doAs(Subject.java:396)
  75.     at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1157)
  76.     at org.apache.hadoop.mapred.Child.main(Child.java:264)
  77. Caused by: java.lang.RuntimeException: com.mysql.jdbc.exceptions
  78. 2012-04-06 14:38:50,306 INFO mapred.JobClient - Task Id : attempt_201203301926_0052_m_000001_2, Status : FAILED
  79. java.lang.RuntimeException: java.lang.RuntimeException: com.mysql.jdbc.exceptions.jdbc4.Communication***ception: Communications link failure

  81. The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
  82.     at org.apache.hadoop.mapreduce.lib.db.DBInputFormat.setConf(DBInputFormat.java:158)
  83.     at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62)
  84.     at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
  85.     at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:606)
  86.     at org.apache.hadoop.mapred.MapTask.run(MapTask.java:323)
  87.     at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
  88.     at java.security.AccessController.doPrivileged(Native Method)
  89.     at javax.security.auth.Subject.doAs(Subject.java:396)
  90.     at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1157)
  91.     at org.apache.hadoop.mapred.Child.main(Child.java:264)
  92. Caused by: java.lang.RuntimeException: com.mysql.jdbc.exceptions
  93. 2012-04-06 14:38:57,463 INFO mapred.JobClient - Job complete: job_201203301926_0052
  94. 2012-04-06 14:38:57,468 INFO mapred.JobClient - Counters: 6
  95. 2012-04-06 14:38:57,468 INFO mapred.JobClient - Job Counters
  96. 2012-04-06 14:38:57,469 INFO mapred.JobClient - SLOTS_MILLIS_MAPS=43062
  97. 2012-04-06 14:38:57,469 INFO mapred.JobClient - Total time spent by all reduces waiting after reserving slots (ms)=0
  98. 2012-04-06 14:38:57,469 INFO mapred.JobClient - Total time spent by all maps waiting after reserving slots (ms)=0
  99. 2012-04-06 14:38:57,469 INFO mapred.JobClient - Launched map tasks=8
  100. 2012-04-06 14:38:57,469 INFO mapred.JobClient - SLOTS_MILLIS_REDUCES=0
  101. 2012-04-06 14:38:57,469 INFO mapred.JobClient - Failed map tasks=1
这主要是因为其实访问RDBMS的不是本机，而是hadoop集群，设置成localhost当然会报错，这时你就需要将localhost替换成数据库所在主机的IP。
  2）解决上述问题之后，又出现下面的异常提示错误：  Host ' 10.110.24.28 ' is not allowed to connect to this MySQLserver。代码如下：

点击(此处)折叠或打开

  1. package com.loogn.rdbms;

  3. import java.io.File;

  5. import org.apache.hadoop.conf.Configuration;
  6. import org.apache.hadoop.fs.Path;
  7. import org.apache.hadoop.io.LongWritable;
  8. import org.apache.hadoop.io.Text;
  9. import org.apache.hadoop.mapreduce.lib.db.DBConfiguration;
  10. import org.apache.hadoop.mapreduce.lib.db.DBInputFormat;

  12. import org.apache.hadoop.mapreduce.Job;
  13. import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
  14. import org.apache.hadoop.util.GenericOptionsParser;


  17. import com.zjhcsoft.idata.hadoop.runHadoopJob.EJob;


  20. public class DBAccess {
  21.     public static void main(String[] args) throws Exception{
  22.         Configuration conf = new Configuration();
  23.         String[] otherargs = new GenericOptionsParser(conf, args).getRemainingArgs();
  24.         if(otherargs.length != 1) {
  25.             System.err.println("Usage: ");
  26.             System.exit(2);
  27.         }
  28.         
  29.         String HadoopHome = "E:\\hadoop\\hadoop-0.20.2-cdh3u3";
  30.         File fileJar = EJob.createTempJar("target\\classes");
  31.         EJob.addClasspath(HadoopHome + "\\conf");
  32.         ClassLoader classLoader = EJob.getClassLoader();
  33.         Thread.currentThread().setContextClassLoader(classLoader);
  34.         conf.set("mapred.jar", fileJar.toString());
  35.         
  36.         
  37.         //String driverName = "com.mysql.jdbc.Driver";
  38.         //Class.forName("com.mysql.jdbc.Driver");
  39.         
  40.         /**
  41.          * 下面这行代码提前设置
  42.          */
  43.         DBConfiguration.configureDB(conf, "com.mysql.jdbc.Driver", "jdbc:mysql://10.110.24.28:3306/test", "root", "198921");
  44.         Job job = new Job(conf, "DBAccess");
  45.         job.setJarByClass(DBAccess.class);
  46.         job.setInputFormatClass(DBInputFormat.class);
  47.         job.setOutputKeyClass(LongWritable.class);
  48.         job.setOutputValueClass(Text.class);
  49.         job.setMapperClass(DBMapper.class);
  50.         job.setReducerClass(IdentityReducer.class);
  51.         FileOutputFormat.setOutputPath(job, new Path(otherargs[0]));
  52.         
  53.         /**
  54.          * DB access three steps:
  55.          */

  57.         String[] fields = {"id", "name", "age"};
  58.         DBInputFormat.setInput(job, TestRecord.class, "test", null, "id", fields);
  59.         
  60.         
  61.         System.exit(job.waitForCompletion(true) ? 0 : 1);
  62.     }
  63. }
  对于这个问题的出现， 可能是你的帐号不允许从远程登陆，只能在localhost。这时有两种方法解决这个问题，一是改表法；二是授权法。第一种方法没有试过，不过应该也可行。我是使用第二种方法解决这个问题的： 例如，你想root使用密码123456从任何主机连接到mysql服务器的话。
GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' IDENTIFIED BY '123456' WITH GRANT OPTION;
  这样整个问题便得以解决！
阅读(1729) | 评论(3) | 转发(0) |
0

上一篇：windows下Eclipse开发MR程序出现的问题之一（插件形式）

下一篇：Oracle数据库初试——常见问题之一

给主人留下些什么吧！~~

loogn_qiang2012-10-29 09:40:56

xin_jmail: 大哥，我的错误如下，求帮助啊！
12/10/26 05:01:10 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xm.....
你这里的错误很明显啊，如下：
Caused by: java.lang.RuntimeException: java.lang.NoSuchMethodException: hadoop.test.DBInput$DBInputMapper.<init>()
        at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:115)
   &

回复 | 举报

xin_jmail2012-10-26 20:08:31

我的代码如下，我从远程可以访问192.168.1.135机子上的数据
package hadoop.test;

import java.io.DataInput;
import java.io.DataOutput;
import java.io.IOException;
import java.sql.PreparedStatement;
import java.sql.ResultSet;
import java.sql.SQLException;

import org.apache.hadoop.filecache.DistributedCache;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import o

回复 | 举报

xin_jmail2012-10-26 20:06:43

大哥，我的错误如下，求帮助啊！
12/10/26 05:01:10 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
12/10/26 05:01:16 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
12/10/26 05:01:46 INFO mapre

回复 | 举报
关于我们 | 关于IT168 | 联系方式 | 广告合作 | 法律声明 | 免费注册

Copyright 2001-2010 ChinaUnix.net All Rights Reserved 北京皓辰网域网络信息技术有限公司. 版权所有

感谢所有关心和支持过ChinaUnix的朋友们

16024965号-6