hive 部署

hive

Created 星期三 04 三月 2015

hive : apache-hive-0.14.0-bin.tar.gz
hadoop : Apache Hadoop release 1.2.1
System : CentOS6.2 x64

在hdfs上建目录：

$ hadoop fs -mkdir /tmp
$ hadoop fs -mkdir /user/hive/warehouse

添加权限：
hrsjw1@hrsjw1-izp:~$ hadoop fs -chmod 777 /tmp ; hadoop fs -chmod 777 /tmp/hive/
hrsjw1@hrsjw1-izp:~$ hadoop fs -mkdir /user/hive ; hadoop fs -chmod 777 /user/hive
$ hadoop fs -chmod g+w /tmp
$ hadoop fs -chmod g+w /user/hive/warehouse

查看：
hrsjw1@hrsjw1-izp:~$ hadoop fs -ls /tmp ; hadoop fs -ls /user/hive/
Found 4 items
drwxrwxrwx - hrsjw1 supergroup 0 2015-03-04 10:23 /tmp/hive
drwx------ - hrsjw1 supergroup 0 2015-02-28 09:35 /tmp/temp-1911031721
drwxrwxrwx - hrsjw1 supergroup 0 2015-02-25 18:26 /tmp/temp-754243223
drwx------ - hrsjw1 supergroup 0 2015-02-28 16:41 /tmp/temp2073358314
Found 2 items
drwxrwxrwx - hrsjw1 hrsjw1 0 2015-03-03 16:56 /user/hive/tmp
drwxrwxr-x - hrsjw1 hrsjw1 0 2015-03-04 10:36 /user/hive/warehouse

下载：
hive ：
mysql jdbc :
解压：
unzip mysql-connector-java.zip
将mysql-connector-java-5.1.26-bin.jar拷贝到hive lib目录下：
hrsjw1@hrsjw1-izp:~/下载$ cp mysql-connector-java/mysql-connector-java-5.1.26/mysql-connector-java-5.1.26-bin.jar $HIVE_HOME/lib/

设置HADOOP_HOME、HIVE_HOME，并将其添加到 /etc/profile
hrsjw1@hrsjw1-izp:~$ tail /etc/profile
export PIG_HOME=/usr/local/pig
export HIVE_HOME=/usr/local/hive
export HADOOP_HOME=/usr/local/hadoop
export ZOOKEEPER_HOME=/usr/local/zookeeper
export HBASE_HOME=/usr/local/hbase
export JAVA_HOME=/usr/local/jdk1.7.0_51
export JRE_HOME=$JAVA_HOME/jre
export CLASSPATH=$JAVA_HOME/lib:$JRE_HOME/lib:$CLASSPATH
export PATH=$JAVA_HOME/bin:$JRE_HOME/bin:$ZOOKEEPER_HOME/bin:$HBASE_HOME/bin:$HIVE_HOME/bin:$PIG_HOME/bin:$HADOOP_HOME/bin:$PATH
export HADOOP_HOME_WARN_SUPPRESS=1

多用户支持
(确认已安装好mysql)
启动mysql：
$ mysql -u root -p
mysql>grant all on hive.* to root@localhost identified by '123456'
修改hive conf/hive-site.xml，如下：


javax.jdo.option.ConnectionURL jdbc:
JDBC connect string for a JDBC metastore


javax.jdo.option.ConnectionDriverName
com.mysql.jdbc.Driver
Driver class name for a JDBC metastore


javax.jdo.option.ConnectionUserName
root
username to use against metastore database


javax.jdo.option.ConnectionPassword
123456
password to use against metastore database

修改hive-env.sh
HADOOP_HOME=$HADOOP_HOME
修改hive-log4j.properties.template和hive-exec-log4j.properties.template
hive.log.dir=/usr/local/hive/logs

创建日志目录：
mkdir $HIVE_HOME/logs

建立数据库hive
create database hive

创建hive用户,并授权
grant all on hive.* to root@'%' identified by '123456';
flush privileges;

启动hive：
hrsjw1@hrsjw1-izp:~$ /usr/local/hive/bin/hive
Logging initialized using configuration in jar:file:/usr/local/hive/lib/hive-common-0.14.0.jar!/hive-log4j.properties
创建表：
hive> create table user_info(userid int, user_name string);
FAILED: Error in metadata: MetaException(message:javax.jdo.JDODataStoreException: An exception was thrown while adding/validating class(es) : Specified key was too long; max key length is 767 bytes
com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Specified key was too long; max key length is 767 bytes
...

搜了下,在mysql上执行:
在mysql机器的上运行:
root@hrsjw1-izp:/usr/local/hive/conf# mysql uroot -p
alter database hive character set latin1;
问题解决.

hive> create table user_info(userid int, user_name string);
OK
Time taken: 0.593 seconds
hive> show tables;
OK
user_info
Time taken: 0.451 seconds

hive> load data local inpath "/usr/local/hive/examples/files/kv1.txt" overwrite into table user_info;
Loading data to table default.user_info
Table default.user_info stats: [numFiles=1, numRows=0, totalSize=5812, rawDataSize=0]
OK
Time taken: 0.663 seconds

hive> select * from user_info limit 5;
OK
238 val_238
86 val_86
311 val_311
27 val_27
165 val_165
Time taken: 0.077 seconds, Fetched: 5 row(s)
hive> 

数据持久化：

hrsjw1@hrsjw1-izp:~$ hadoop fs -lsr /user/hive
drwxrwxrwx   - hrsjw1 hrsjw1          0 2015-03-03 16:56 /user/hive/tmp
drwx------   - hrsjw1 hrsjw1          0 2015-03-03 18:12 /user/hive/tmp/hrsjw1
drwxrwxr-x   - hrsjw1 hrsjw1          0 2015-03-04 10:36 /user/hive/warehouse
drwxr-xr-x   - hrsjw1 hrsjw1          0 2015-03-04 10:47 /user/hive/warehouse/user_info
-rw-r--r--   3 hrsjw1 hrsjw1       5812 2015-03-04 10:47 /user/hive/warehouse/user_info/kv1.txt

hrsjw1@hrsjw1-izp:/usr/local/hive/bin$ ./hive --service hwi
ls: 无法访问/usr/local/hive/lib/hive-hwi-*.war: 没有那个文件或目录
15/03/04 14:47:14 INFO hwi.HWIServer: HWI is starting up
15/03/04 14:47:15 INFO mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
15/03/04 14:47:15 INFO mortbay.log: jetty-6.1.26
15/03/04 14:47:16 INFO mortbay.log: Started SocketConnector@0.0.0.0:9999