Chinaunix首页 | 论坛 | 认证专区 | 博客 登录 | 注册

kerlion的ChinaUnix博客kerlion.blog.chinaunix.net

甲骨文,Principal Oracle DBA, kerlion.he@hotmail.com

  • 博客访问: 768536
  • 博文数量: 134
  • 博客积分: 2286
  • 博客等级: 大尉
  • 技术积分: 1938
  • 用 户 组: 普通用户
  • 注册时间: 2010-06-10 10:37
文章分类

全部博文(134)

文章存档

2015年(2)

2014年(1)

2013年(34)

2012年(92)

2011年(5)

微信关注

IT168企业级官微



微信号:IT168qiye



系统架构师大会



微信号:SACC2013

订阅
热词专题
hive的安装和配置 2012-03-22 14:18:34

分类: 云计算

1. download
wget http://mirror.mel.bkb.net.au/pub/apache//hive/stable/hive-0.8.1.tar.gz
tar zxf hive-0.8.1.tar.gz
只需要在一个节点上安装


2. 设置环境变量

vi .bash_profile
export JAVA_HOME=/usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/jre
export HADOOP_HOME=/home/hadoop/hadoop-1.0.0
export HIVE_HOME=/home/hadoop/hive-0.8.1
export HADOOP_CONF_DIR=$HOME/conf
export HIVE_CONF_DIR=$HOME/hive-conf

export CLASSPATH=$HIVE_HOME/lib:$JAVA_HOME/lib:$JAVA_HOME/jre/lib:$HADOOP_HOME
export PATH=$HIVE_HOME/bin:$HADOOP_HOME/bin:$JAVA_HOME/bin:/sbin/:/bin:$PATH


3. 配置hive
cp -r hive-0.8.1/conf $HIVE_CONF_DIR/
cd $HIVE_CONF_DIR/
cp hive-default.xml.template hive-default.xml

cat hive-env.sh
export HADOOP_HEAPSIZE=512
export HIVE_CONF_DIR=/home/hadoop/hive-conf


3. 测试
$ hive
hive> show tables;
OK
Time taken: 4.824 seconds
hive> create table hwz(id int, name string);
OK
Time taken: 0.566 seconds

hive> select * from hwz;
OK
Time taken: 0.361 seconds



$ hadoop dfs -lsr /user/hive
Warning: $HADOOP_HOME is deprecated.

drwxr-xr-x - hadoop supergroup 0 2012-03-22 12:36 /user/hive/warehouse
drwxr-xr-x - hadoop supergroup 0 2012-03-22 12:36 /user/hive/warehouse/hwz

4. 配置Metastore用mysql数据库,这样才可以多用户同时访问
a. create user and database for hive in mysql
create database hive;
GRANT all ON hive.* TO hive@'%' IDENTIFIED BY 'hivepass';

b. change metastore to use mysql
cat hive-site.xml


hive.metastore.local
true


javax.jdo.option.ConnectionURL
jdbc:mysql://slave1:3306/hive?createDatabaseIfNotExist=true


javax.jdo.option.ConnectionDriverName
com.mysql.jdbc.Driver


javax.jdo.option.ConnectionUserName
hive


javax.jdo.option.ConnectionPassword
hivepass



c. 检查
$ hive
hive> use dw2;
OK

Time taken: 3.43 seconds

hive> create table hwz2(id int, name string ) row format delimited FIELDS TERMINATED BY ',';
OK
Time taken: 2.519 seconds

hive> show tables;

OK
hwz2
Time taken: 0.419 seconds
hive> load data local inpath 'demo.txt' overwrite into table hwz2;
Copying data from file:/home/hadoop/demo.txt
Copying file: file:/home/hadoop/demo.txt
Loading data to table dw2.hwz2
Deleted hdfs://master:9000/user/hive/warehouse/dw2.db/hwz2
OK
Time taken: 0.557 seconds
hive> select * from hwz2;
OK
12 jack
12 jack
12 jack
12 jack
12 jack
12 jack
12 jack
12 jack

$ hadoop dfs -lsr /user/hive
Warning: $HADOOP_HOME is deprecated.

drwxr-xr-x - hadoop supergroup 0 2012-03-22 15:36 /user/hive/warehouse
drwxr-xr-x - hadoop supergroup 0 2012-03-22 15:48 /user/hive/warehouse/dw2.db
drwxr-xr-x - hadoop supergroup 0 2012-03-22 15:48 /user/hive/warehouse/dw2.db/hwz2
-rw-r--r-- 2 hadoop supergroup 1201 2012-03-22 15:48 /user/hive/warehouse/dw2.db/hwz2/demo.txt
drwxr-xr-x - hadoop supergroup 0 2012-03-22 12:36 /user/hive/warehouse/hwz
drwxr-xr-x - hadoop supergroup 0 2012-03-22 15:36 /user/hive/warehouse/hwz2
-rw-r--r-- 2 hadoop supergroup 1201 2012-03-22 15:36 /user/hive/warehouse/hwz2/demo.txt
$ hadoop dfs -cat /user/hive/warehouse/dw2.db/hwz2/demo.txt |head
Warning: $HADOOP_HOME is deprecated.

12,jack
12,jack
12,jack
12,jack
12,jack
12,jack
12,jack
12,jack
12,jack
12,jack



d. 在mysql中验证建立的新表
mysql> use hive;
Database changed

mysql> show tables;
+-----------------+
| Tables_in_hive |
+-----------------+
| BUCKETING_COLS |
| CDS |
| COLUMNS_V2 |
| DATABASE_PARAMS |
| DBS |
| PARTITION_KEYS |
| SDS |
| SD_PARAMS |
| SEQUENCE_TABLE |
| SERDES |
| SERDE_PARAMS |
| SORT_COLS |
| TABLE_PARAMS |
| TBLS |
+-----------------+
14 rows in set (0.00 sec)



9. 常见错误
error 1:
-------------------------------------------------
hive> show tables;
FAILED: Error in metadata: javax.jdo.JDOFatalInternalException: Error creating transactional connection factory
Solution:
Hive不带mysql JDBC驱动,自己安装:


wget http://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-5.1.18.tar.gz/from/http://mysql.mirror.kangaroot.net/
tar zxf mysql-connector-java-5.1.18.tar.gz
cd mysql-connector-java-5.1.18
cp mysql-connector*.jar $HIVE_HOME/lib

error 2:
-------------------------------------------------
hive> show tables;
FAILED: Error in metadata: javax.jdo.JDOException: Couldnt obtain a new sequence (unique id) : Cannot execute statement: impossible to write to binary log since BINLOG_FORMAT = STATEMENT and at least one table uses a storage engine limited to row-based logging. InnoDB is limited to row-logging when transaction isolation level is READ COMMITTED or READ UNCOMMITTED.

Solution:
在mysql中设置 binlog_format='MIXED'






阅读(60577) | 评论(0) | 转发(0) |
给主人留下些什么吧!~~
评论热议
请登录后评论。

登录 注册