Chinaunix首页 | 论坛 | 博客
  • 博客访问: 1080962
  • 博文数量: 83
  • 博客积分: 159
  • 博客等级: 上尉
  • 技术积分: 2221
  • 用 户 组: 普通用户
  • 注册时间: 2010-05-15 17:08
个人简介

……致我那曾经苦逼的岁月……

文章分类
文章存档

2018年(1)

2017年(7)

2016年(13)

2014年(1)

2013年(12)

2012年(27)

2011年(22)

分类: 云计算

2012-06-20 11:44:02

环境说明:
搭建好的hadoop+hbase+zookeeper集群,因为hbase里面查询数据不支持select语句,所以搭建起了hive(数据仓库)。我的hive搭建过程也不做太多的介绍,用的是第三方数据库mysql存储hive的元数据。在hive里面我把hbase数据库的xyz表和hive里面的hbase_table_1表关联上,然后执行select * from table可以查到数据,但是select count(*) from table死活报错,结果是mapreduce的任务没跑成功。截图如下:
先查看hbase数据库的xyz表的数据
hbase(main):001:0> scan 'xyz'
ROW                  COLUMN+CELL                                               
10000                column=cf1:val, timestamp=1340091488116, value=China     
1 row(s) in 0.6730 seconds
hbase(main):002:0>
其次查看hive中的hbase_table_1表的数据
hive> select * from hbase_table_1;
OK
10000   China
Time taken: 4.133 seconds
hive>
最后我在hive里要做统计多少行命令和报错信息
hive> select count(*) from  hbase_table_1;
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
  set hive.exec.reducers.bytes.per.reducer=
In order to limit the maximum number of reducers:
  set hive.exec.reducers.max=
In order to set a constant number of reducers:
  set mapred.reduce.tasks=
Starting Job = job_201206190956_0001, Tracking URL =
Kill Command = /opt/hadoop/libexec/../bin/hadoop job  -Dmapred.job.tracker=master:9002 -kill job_201206190956_0001
Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 1
2012-06-20 10:46:58,214 Stage-1 map = 0%,  reduce = 0%
2012-06-20 10:47:58,795 Stage-1 map = 0%,  reduce = 0%
2012-06-20 10:48:03,875 Stage-1 map = 100%,  reduce = 100%
Ended Job = job_201206190956_0001 with errors
Error during job, obtaining debugging information...
Examining task ID: task_201206190956_0001_m_000002 (and more) from job job_201206190956_0001
Exception in thread "Thread-36" java.lang.RuntimeException: Error while reading from task log url
        at org.apache.hadoop.hive.ql.exec.errors.TaskLogProcessor.getErrors(TaskLogProcessor.java:130)
        at org.apache.hadoop.hive.ql.exec.JobDebugger.showJobFailDebugInfo(JobDebugger.java:211)
        at org.apache.hadoop.hive.ql.exec.JobDebugger.run(JobDebugger.java:81)
        at java.lang.Thread.run(Thread.java:619)
Caused by: java.io.IOException: Server returned HTTP response code: 400 for URL:
        at sun.net.(HttpURLConnection.java:1305)
        at java.net.URL.openStream(URL.java:1009)
        at org.apache.hadoop.hive.ql.exec.errors.TaskLogProcessor.getErrors(TaskLogProcessor.java:120)
        ... 3 more
FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask
MapReduce Jobs Launched:
Job 0: Map: 1  Reduce: 1   HDFS Read: 0 HDFS Write: 0 FAIL
Total MapReduce CPU Time Spent: 0 msec
经过几天上网查询和同事沟通的结果,解决上述问题需要有两个步骤:
1、在hive的配置文件hive-site.xml里面增加如下内容,当然value里面的值根据你自己的实际情况来写
 
    hive.aux.jars.path
 file:///opt/hive/lib/hive-hbase-handler-0.8.1.jar,file:///opt/hive/lib/h
base-0.92.1.jar,file:///opt/hive/lib/zookeeper-3.3.1.jar

 
2、然后将namenode节点的hbase配置文件hbase-site.xml拷贝到hadoop的conf目录下,最后将你的它用rsync同步到所有的datanode节点上。
最后我们在查一下试试?
hive> select count(*) from hbase_table_1;
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
  set hive.exec.reducers.bytes.per.reducer=
In order to limit the maximum number of reducers:
  set hive.exec.reducers.max=
In order to set a constant number of reducers:
  set mapred.reduce.tasks=
Starting Job = job_201206190956_0003, Tracking URL =
Kill Command = /opt/hadoop/libexec/../bin/hadoop job  -Dmapred.job.tracker=master:9002 -kill job_201206190956_0003
Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 1
2012-06-20 12:04:12,499 Stage-1 map = 0%,  reduce = 0%
2012-06-20 12:04:27,668 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU 2.82 sec
2012-06-20 12:04:28,682 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU 2.82 sec
2012-06-20 12:04:29,703 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU 2.82 sec
2012-06-20 12:04:30,713 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU 2.82 sec
2012-06-20 12:04:31,724 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU 2.82 sec
2012-06-20 12:04:32,734 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU 2.82 sec
2012-06-20 12:04:33,757 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU 2.82 sec
2012-06-20 12:04:34,768 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU 2.82 sec
2012-06-20 12:04:35,777 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU 2.82 sec
2012-06-20 12:04:36,788 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU 2.82 sec
2012-06-20 12:04:37,798 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU 2.82 sec
2012-06-20 12:04:38,808 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU 2.82 sec
2012-06-20 12:04:39,869 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU 2.82 sec
2012-06-20 12:04:40,880 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU 2.82 sec
2012-06-20 12:04:42,126 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU 2.82 sec
2012-06-20 12:04:43,136 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU 2.82 sec
2012-06-20 12:04:44,145 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU 2.82 sec
2012-06-20 12:04:45,155 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU 2.82 sec
2012-06-20 12:04:46,164 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU 2.82 sec
2012-06-20 12:04:47,174 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU 2.82 sec
2012-06-20 12:04:48,183 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU 2.82 sec
2012-06-20 12:04:49,236 Stage-1 map = 100%,  reduce = 100%, Cumulative CPU 7.48 sec
2012-06-20 12:04:50,247 Stage-1 map = 100%,  reduce = 100%, Cumulative CPU 7.48 sec
2012-06-20 12:04:51,267 Stage-1 map = 100%,  reduce = 100%, Cumulative CPU 7.48 sec
2012-06-20 12:04:52,277 Stage-1 map = 100%,  reduce = 100%, Cumulative CPU 7.48 sec
2012-06-20 12:04:53,288 Stage-1 map = 100%,  reduce = 100%, Cumulative CPU 7.48 sec
2012-06-20 12:04:54,320 Stage-1 map = 100%,  reduce = 100%, Cumulative CPU 7.48 sec
2012-06-20 12:04:55,330 Stage-1 map = 100%,  reduce = 100%, Cumulative CPU 7.48 sec
2012-06-20 12:04:56,341 Stage-1 map = 100%,  reduce = 100%, Cumulative CPU 7.48 sec
2012-06-20 12:04:57,364 Stage-1 map = 100%,  reduce = 100%, Cumulative CPU 7.48 sec
2012-06-20 12:04:58,375 Stage-1 map = 100%,  reduce = 100%, Cumulative CPU 7.48 sec
MapReduce Total cumulative CPU time: 7 seconds 480 msec
Ended Job = job_201206190956_0003
MapReduce Jobs Launched:
Job 0: Map: 1  Reduce: 1   Accumulative CPU: 7.48 sec   HDFS Read: 240 HDFS Write: 2 SUCESS
Total MapReduce CPU Time Spent: 7 seconds 480 msec
OK
1
Time taken: 92.04 seconds
可以了!因为我的表中只有一行数据!用的虚拟机比较慢,哎!!!
 


 
阅读(5581) | 评论(4) | 转发(1) |
给主人留下些什么吧!~~

split_two2012-08-20 14:19:08

xin_jmail: 坐等作者回答啊?!!!.....
能加我QQ细聊吗?316189480

xin_jmail2012-08-18 23:15:56

坐等作者回答啊?!!!

xin_jmail2012-08-18 17:13:26

我的zookeeper采用的是hbase自带的,集群中有三个节点hadoop1,hadoop2,hadoop3。hbase.zookeeper.quorum值配置的时候写的是hadoop1,hadoop2,hadoop3对应的IP地址。hbase的元数据库采用的是mysql。整合hive和hbase后,hive -hiveconf hbase.zookeeper.quorum= hadoop1,hadoop2,hadoop3命令启动,建表时就遇到刚才说的错误。jar包路径已写在配置文件里

xin_jmail2012-08-18 17:04:01

我的hive和hbase都能正常运行。但是hive+hbase整合时,出现错误:
FAILED: Error in metadata: java.lang.NullPointerException
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask,困扰了好长时间,希望您能解决一下,谢谢~