Hive从0.14版本开始支持事务和行级更新,但缺省是不支持的,需要一些附加的配置。要想支持行级insert、update、delete,需要配置Hive支持事务。
配置Hive支持事务(Hive 2.3.4版)
1. 在hive-site.xml文件中添加如下配置项
-
<property>
-
<name>hive.support.concurrency</name>
-
<value>true</value>
-
</property>
-
<property>
-
<name>hive.exec.dynamic.partition.mode</name>
-
<value>nonstrict</value>
-
</property>
-
<property>
-
<name>hive.txn.manager</name>
-
<value>org.apache.hadoop.hive.ql.lockmgr.DbTxnManager</value>
-
</property>
-
<property>
-
<name>hive.compactor.initiator.on</name>
-
<value>true</value>
-
</property>
-
<property>
-
<name>hive.compactor.worker.threads</name>
-
<value>1</value>
-
</property>
2. 添加Hive元数据(使用mysql存储),登录mysql,执行如下sql:
INSERT INTO NEXT_LOCK_ID VALUES(1);
INSERT INTO NEXT_COMPACTION_QUEUE_ID VALUES(1);
INSERT INTO NEXT_TXN_ID VALUES(1);
COMMIT;
3.
启动hadoop集群和mysql
$HADOOP_HOME/sbin/start-dfs.sh
$HADOOP_HOME/sbin/start-yarn.sh
~/mysql/bin/mysqld &
4.
建立测试表
create table t1(id int, name string)
clustered by (id) into 8 buckets
stored as orc TBLPROPERTIES ('transactional'='true');
说明:建表语句必须带有into buckets子句和stored as orc TBLPROPERTIES ('transactional'='true')子句,并且不能带有sorted by子句。
5.
测试insert、update、delete
-
insert into t1 values (1,'aaa');
-
insert into t1 values (2,'bbb');
-
update t1 set name='ccc' where id=1;
-
delete from t1 where id=2;
可以查看输入,update和delete都已经OK了
阅读(15388) | 评论(0) | 转发(0) |