大数据学习笔记3-Sqoop

开始学习sqoop，用的版本1.99.7
1.下载解压sqoop

    export SQOOP_HOME=/opt/hadoop/sqoop-1.99.7
    export PATH=$PATH:$SQOOP_HOME/bin
    export SQOOP_SERVER_EXTRA_LIB=$SQOOP_HOME/extra
    export CATALINA_BASE=$SQOOP_HOME/server
    export LOGDIR=$SQOOP_HOME/logs/

2.修改hadoop/etc/hadoop/container-executor.cfg 
  allowed.system.users=wanghb  运行sqoop的用户名

3.vi core-site.xml

    
           hadoop.proxyuser.wanghb.hosts
           *
    
    
        hadoop.proxyuser.wanghb.groups 运行sqoop的用户名
        *
    

4.修改配置文件sqoop/conf/sqoop.properties 

org.apache.sqoop.submission.engine.mapreduce.configuration.directory=/study/hadoop/etc/hadoop

打开以下选项：
org.apache.sqoop.security.authentication.type=SIMPLE
org.apache.sqoop.security.authentication.handler=org.apache.sqoop.security.authentication.SimpleAuthenticationHandler
org.apache.sqoop.security.authentication.anonymous=true

然后替换@LOGDIR@ 和@BASEDIR@为logs和base,vi下切换成命令行模式(shift+:)输入：
0,$ s/@LOGDIR@/logs/g
0,$ s/@BASEDIR@/base/g

5.创建目录
mkdir /study/sqoop/extra
mkdir /study/sqoop/logs

6.mysql的驱动jar文件复制到这个目录下。 
cp mysql-connector-java-5.1.36-bin.jar /study/sqoop/extra

7.验证配置是否有效
使用sqoop2-tool工具进行验证：bin/sqoop2-tool verify 

8.开启服务器bin/sqoop2-server start

安装配置mysql：
9.安装mysql：sudo apt-get install mysql-server
   启动mysql服务：/etc/init.d/mysqld restart 
   连接数据库：mysql -uroot -p1234 //其中root是数据库的用户不是操作系统的,1234是密码
   创建用户
    grant all on *.* to wanghb@'%' identified by 'wanghb';
    grant all on *.* to wanghb@'localhost' identified by 'wanghb';
    grant all on *.* to wanghb@'master' identified by 'wanghb';
    flush privileges;

10.创建数据库：create database hadoop; 
   查看： show databases;  
   
   创建表：    
    use hadoop;//在hadoop数据库中创建或者在创建表时指定库
    create table person(
    number INT(11),
    name VARCHAR(255)
    )ENGINE=hadoop DEFAULT CHARSET=utf8;
    插入数据：
    INSERT INTO person 
    (number, name)
    VALUES
    (1001, "小明");    
   查看表单：show tables from hadoop；
   退出quit; 

**做个试验，使用sqoop从mysql中导出数据到hdfs中**

11.开启hdfs, 开启sqoop

突然发现sqoop2跟sqoop1完全不一样。未完待续