使用 Spark-sql 导入数据到hadoop

Hive导入数据到hadoop

简介：
这里使用 hive，Hive一般在工作站上运行。它把SQL查询转换成一系列在 Hadoop 集群上运行的 MapReduce 作业。Hive把数据组织成表，通过
这种方式为存储在HDFS的数据赋予结构。元数据（如表模式）存储在 metastore 数据库中


Hive 模式把数据组织成表，创建表 hoop， 默认数据库为default

$ ./spark-sql --master=yarn --hiveconf hive.cli.print.header=false -S -e "create table hoop(id int, name string) row format delimited fields terminated by '\t';"

备注：
1）通过参数  hive.cli.print.header  可以控制在cli中是否显示表的列名，可以试试将其改为 true
2）row format delimited fields terminated by '\t'      -------> 其中这个是必须的，hive导入数据默认的模式

查看创建的表，这里使用 spark-sql 的JDBC Server查询刚刚创建的表
点击(此处)折叠或打开

$ ./beeline -u jdbc:hive2://2.hadoop.com:10008 -n biadmin
scan complete in 3ms
Connecting to jdbc:hive2://2.hadoop.com:10008
Connected to: Spark SQL (version 1.4.0)
Driver: Spark Project Core (version 1.4.0)
Transaction isolation: TRANSACTION_REPEATABLE_READ
Beeline version 1.4.0 by Apache Hive
0: jdbc:hive2://2.hadoop.com:10008> show databases;
+-----------+
|  result   |
+-----------+
| default |
| dp_click  |
| dp_dump   |
| dp_ods    |
| dp_tmp    |
+-----------+
0: jdbc:hive2://2.hadoop.com:10008> use default;
+---------+
| result  |
+---------+
+---------+
No rows selected (0.02 seconds)
0: jdbc:hive2://2.hadoop.com:10008> show tables;
+------------+--------------+
| tableName  | isTemporary  |
+------------+--------------+
| hoop       | false        |
+------------+--------------+
1 row selected (0.023 seconds)
0: jdbc:hive2://2.hadoop.com:10008> desc hoop;
+-----------+------------+----------+
| col_name  | data_type  | comment  |
+-----------+------------+----------+
| id        | int        | NULL     |
| name      | string     | NULL     |
+-----------+------------+----------+
2 rows selected (0.021 seconds)

导入数据，指定本地文件

$ spark-sql --master=yarn --hiveconf hive.cli.print.header=false -S -e "load data local inpath './hoop.txt' overwrite into table default.hoop;"
SET spark.sql.hive.version=0.13.1
SET spark.sql.hive.version=0.13.1
Copying data from file:hoop.txt
Copying file: file:hoop.txt
Loading data to table default.hoop
Table default.hoop stats: [numFiles=1, numRows=0, totalSize=81, rawDataSize=0]
OK


$ ./beeline -u jdbc:hive2://2.hadoop.com:10008 -n biadmin
scan complete in 2ms
Connecting to jdbc:hive2://2.hadoop.com:10008
Connected to: Spark SQL (version 1.4.0)
Driver: Spark Project Core (version 1.4.0)
Transaction isolation: TRANSACTION_REPEATABLE_READ
Beeline version 1.4.0 by Apache Hive
0: jdbc:hive2://2.hadoop.com:10008> select * from default.hoop;
+-----+---------+
| id  |  name   |
+-----+---------+
| 1   | qqqqqq  |
| 2   | wwwwww  |
| 3   | eeeeee  |
| 4   | rrrrrr  |
| 5   | tttttt  |
| 6   | yyyyyy  |
| 7   | uuuuuu  |
| 8   | iiiiii  |
| 9   | oooooo  |
+-----+---------+
9 rows selected (0.467 seconds)
0: jdbc:hive2://2.hadoop.com:10008>

待续。。。。。。