Chinaunix首页 | 论坛 | 博客
  • 博客访问: 298942
  • 博文数量: 22
  • 博客积分: 674
  • 博客等级: 上士
  • 技术积分: 792
  • 用 户 组: 普通用户
  • 注册时间: 2012-02-19 00:48
文章存档

2013年(2)

2012年(20)

我的朋友

分类: 云计算

2012-04-06 14:23:40

  在运行《Hadoop开发者第三期》中利用hadoop访问传统关系型数据库系统时,出现了一系列的问题,其中一个问题是由代码中第43行引起的,此处会有两种情况:
  1)若在代码中将第43行代码写成:

点击(此处)折叠或打开

  1. DBConfiguration.configureDB(conf, "com.mysql.jdbc.Driver", "jdbc:mysql://localhost:3306/test", "root", "198921");
则会导致运行时出现以下错误:

点击(此处)折叠或打开

  1. 2012-04-06 14:38:31,831 INFO mapred.JobClient - Running job: job_201203301926_0052
  2. 2012-04-06 14:38:32,836 INFO mapred.JobClient - map 0% reduce 0%
  3. 2012-04-06 14:38:39,879 INFO mapred.JobClient - Task Id : attempt_201203301926_0052_m_000000_0, Status : FAILED
  4. java.lang.RuntimeException: java.lang.RuntimeException: com.mysql.jdbc.exceptions.jdbc4.Communication***ception: Communications link failure

  5. The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
  6.     at org.apache.hadoop.mapreduce.lib.db.DBInputFormat.setConf(DBInputFormat.java:158)
  7.     at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62)
  8.     at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
  9.     at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:606)
  10.     at org.apache.hadoop.mapred.MapTask.run(MapTask.java:323)
  11.     at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
  12.     at java.security.AccessController.doPrivileged(Native Method)
  13.     at javax.security.auth.Subject.doAs(Subject.java:396)
  14.     at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1157)
  15.     at org.apache.hadoop.mapred.Child.main(Child.java:264)
  16. Caused by: java.lang.RuntimeException: com.mysql.jdbc.exceptions
  17. 2012-04-06 14:38:39,944 INFO mapred.JobClient - Task Id : attempt_201203301926_0052_m_000001_0, Status : FAILED
  18. java.lang.RuntimeException: java.lang.RuntimeException: com.mysql.jdbc.exceptions.jdbc4.Communication***ception: Communications link failure

  19. The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
  20.     at org.apache.hadoop.mapreduce.lib.db.DBInputFormat.setConf(DBInputFormat.java:158)
  21.     at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62)
  22.     at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
  23.     at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:606)
  24.     at org.apache.hadoop.mapred.MapTask.run(MapTask.java:323)
  25.     at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
  26.     at java.security.AccessController.doPrivileged(Native Method)
  27.     at javax.security.auth.Subject.doAs(Subject.java:396)
  28.     at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1157)
  29.     at org.apache.hadoop.mapred.Child.main(Child.java:264)
  30. Caused by: java.lang.RuntimeException: com.mysql.jdbc.exceptions
  31. 2012-04-06 14:38:45,066 INFO mapred.JobClient - Task Id : attempt_201203301926_0052_m_000000_1, Status : FAILED
  32. java.lang.RuntimeException: java.lang.RuntimeException: com.mysql.jdbc.exceptions.jdbc4.Communication***ception: Communications link failure

  33. The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
  34.     at org.apache.hadoop.mapreduce.lib.db.DBInputFormat.setConf(DBInputFormat.java:158)
  35.     at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62)
  36.     at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
  37.     at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:606)
  38.     at org.apache.hadoop.mapred.MapTask.run(MapTask.java:323)
  39.     at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
  40.     at java.security.AccessController.doPrivileged(Native Method)
  41.     at javax.security.auth.Subject.doAs(Subject.java:396)
  42.     at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1157)
  43.     at org.apache.hadoop.mapred.Child.main(Child.java:264)
  44. Caused by: java.lang.RuntimeException: com.mysql.jdbc.exceptions
  45. 2012-04-06 14:38:45,153 INFO mapred.JobClient - Task Id : attempt_201203301926_0052_m_000001_1, Status : FAILED
  46. java.lang.RuntimeException: java.lang.RuntimeException: com.mysql.jdbc.exceptions.jdbc4.Communication***ception: Communications link failure

  47. The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
  48.     at org.apache.hadoop.mapreduce.lib.db.DBInputFormat.setConf(DBInputFormat.java:158)
  49.     at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62)
  50.     at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
  51.     at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:606)
  52.     at org.apache.hadoop.mapred.MapTask.run(MapTask.java:323)
  53.     at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
  54.     at java.security.AccessController.doPrivileged(Native Method)
  55.     at javax.security.auth.Subject.doAs(Subject.java:396)
  56.     at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1157)
  57.     at org.apache.hadoop.mapred.Child.main(Child.java:264)
  58. Caused by: java.lang.RuntimeException: com.mysql.jdbc.exceptions
  59. 2012-04-06 14:38:50,236 INFO mapred.JobClient - Task Id : attempt_201203301926_0052_m_000000_2, Status : FAILED
  60. java.lang.RuntimeException: java.lang.RuntimeException: com.mysql.jdbc.exceptions.jdbc4.Communication***ception: Communications link failure

  61. The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
  62.     at org.apache.hadoop.mapreduce.lib.db.DBInputFormat.setConf(DBInputFormat.java:158)
  63.     at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62)
  64.     at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
  65.     at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:606)
  66.     at org.apache.hadoop.mapred.MapTask.run(MapTask.java:323)
  67.     at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
  68.     at java.security.AccessController.doPrivileged(Native Method)
  69.     at javax.security.auth.Subject.doAs(Subject.java:396)
  70.     at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1157)
  71.     at org.apache.hadoop.mapred.Child.main(Child.java:264)
  72. Caused by: java.lang.RuntimeException: com.mysql.jdbc.exceptions
  73. 2012-04-06 14:38:50,306 INFO mapred.JobClient - Task Id : attempt_201203301926_0052_m_000001_2, Status : FAILED
  74. java.lang.RuntimeException: java.lang.RuntimeException: com.mysql.jdbc.exceptions.jdbc4.Communication***ception: Communications link failure

  75. The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
  76.     at org.apache.hadoop.mapreduce.lib.db.DBInputFormat.setConf(DBInputFormat.java:158)
  77.     at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62)
  78.     at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
  79.     at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:606)
  80.     at org.apache.hadoop.mapred.MapTask.run(MapTask.java:323)
  81.     at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
  82.     at java.security.AccessController.doPrivileged(Native Method)
  83.     at javax.security.auth.Subject.doAs(Subject.java:396)
  84.     at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1157)
  85.     at org.apache.hadoop.mapred.Child.main(Child.java:264)
  86. Caused by: java.lang.RuntimeException: com.mysql.jdbc.exceptions
  87. 2012-04-06 14:38:57,463 INFO mapred.JobClient - Job complete: job_201203301926_0052
  88. 2012-04-06 14:38:57,468 INFO mapred.JobClient - Counters: 6
  89. 2012-04-06 14:38:57,468 INFO mapred.JobClient - Job Counters
  90. 2012-04-06 14:38:57,469 INFO mapred.JobClient - SLOTS_MILLIS_MAPS=43062
  91. 2012-04-06 14:38:57,469 INFO mapred.JobClient - Total time spent by all reduces waiting after reserving slots (ms)=0
  92. 2012-04-06 14:38:57,469 INFO mapred.JobClient - Total time spent by all maps waiting after reserving slots (ms)=0
  93. 2012-04-06 14:38:57,469 INFO mapred.JobClient - Launched map tasks=8
  94. 2012-04-06 14:38:57,469 INFO mapred.JobClient - SLOTS_MILLIS_REDUCES=0
  95. 2012-04-06 14:38:57,469 INFO mapred.JobClient - Failed map tasks=1
这主要是因为其实访问RDBMS的不是本机,而是hadoop集群,设置成localhost当然会报错,这时你就需要将localhost替换成数据库所在主机的IP。
  2)解决上述问题之后,又出现下面的异常提示错误:  Host ' 10.110.24.28 ' is not allowed to connect to this MySQLserver。代码如下:

点击(此处)折叠或打开

  1. package com.loogn.rdbms;

  2. import java.io.File;

  3. import org.apache.hadoop.conf.Configuration;
  4. import org.apache.hadoop.fs.Path;
  5. import org.apache.hadoop.io.LongWritable;
  6. import org.apache.hadoop.io.Text;
  7. import org.apache.hadoop.mapreduce.lib.db.DBConfiguration;
  8. import org.apache.hadoop.mapreduce.lib.db.DBInputFormat;

  9. import org.apache.hadoop.mapreduce.Job;
  10. import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
  11. import org.apache.hadoop.util.GenericOptionsParser;


  12. import com.zjhcsoft.idata.hadoop.runHadoopJob.EJob;


  13. public class DBAccess {
  14.     public static void main(String[] args) throws Exception{
  15.         Configuration conf = new Configuration();
  16.         String[] otherargs = new GenericOptionsParser(conf, args).getRemainingArgs();
  17.         if(otherargs.length != 1) {
  18.             System.err.println("Usage: ");
  19.             System.exit(2);
  20.         }
  21.         
  22.         String HadoopHome = "E:\\hadoop\\hadoop-0.20.2-cdh3u3";
  23.         File fileJar = EJob.createTempJar("target\\classes");
  24.         EJob.addClasspath(HadoopHome + "\\conf");
  25.         ClassLoader classLoader = EJob.getClassLoader();
  26.         Thread.currentThread().setContextClassLoader(classLoader);
  27.         conf.set("mapred.jar", fileJar.toString());
  28.         
  29.         
  30.         //String driverName = "com.mysql.jdbc.Driver";
  31.         //Class.forName("com.mysql.jdbc.Driver");
  32.         
  33.         /**
  34.          * 下面这行代码提前设置
  35.          */
  36.         DBConfiguration.configureDB(conf, "com.mysql.jdbc.Driver", "jdbc:mysql://10.110.24.28:3306/test", "root", "198921");
  37.         Job job = new Job(conf, "DBAccess");
  38.         job.setJarByClass(DBAccess.class);
  39.         job.setInputFormatClass(DBInputFormat.class);
  40.         job.setOutputKeyClass(LongWritable.class);
  41.         job.setOutputValueClass(Text.class);
  42.         job.setMapperClass(DBMapper.class);
  43.         job.setReducerClass(IdentityReducer.class);
  44.         FileOutputFormat.setOutputPath(job, new Path(otherargs[0]));
  45.         
  46.         /**
  47.          * DB access three steps:
  48.          */

  49.         String[] fields = {"id", "name", "age"};
  50.         DBInputFormat.setInput(job, TestRecord.class, "test", null, "id", fields);
  51.         
  52.         
  53.         System.exit(job.waitForCompletion(true) ? 0 : 1);
  54.     }
  55. }
  对于这个问题的出现, 可能是你的帐号不允许从远程登陆,只能在localhost。这时有两种方法解决这个问题,一是改表法;二是授权法。第一种方法没有试过,不过应该也可行。我是使用第二种方法解决这个问题的: 例如,你想root使用密码123456从任何主机连接到mysql服务器的话。
GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' IDENTIFIED BY '123456' WITH GRANT OPTION;
  这样整个问题便得以解决!
阅读(1680) | 评论(3) | 转发(0) |
给主人留下些什么吧!~~

loogn_qiang2012-10-29 09:40:56

xin_jmail: 大哥,我的错误如下,求帮助啊!
12/10/26 05:01:10 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xm.....
你这里的错误很明显啊,如下:
Caused by: java.lang.RuntimeException: java.lang.NoSuchMethodException: hadoop.test.DBInput$DBInputMapper.<init>()
        at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:115)
   &

xin_jmail2012-10-26 20:08:31

我的代码如下,我从远程可以访问192.168.1.135机子上的数据
package hadoop.test;

import java.io.DataInput;
import java.io.DataOutput;
import java.io.IOException;
import java.sql.PreparedStatement;
import java.sql.ResultSet;
import java.sql.SQLException;

import org.apache.hadoop.filecache.DistributedCache;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import o

xin_jmail2012-10-26 20:06:43

大哥,我的错误如下,求帮助啊!
12/10/26 05:01:10 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
12/10/26 05:01:16 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
12/10/26 05:01:46 INFO mapre