大数据学习笔记-环境相关常用命令

环境设置：
export JAVA_HOME=/study/jdk
export HADOOP_HOME=/study/hadoop
export HIVE_HOME=/study/hive
export SPARK_HOME=/study/spark

export CLASSPATH=$HIVE_HOME/lib:$JAVA_HOME/lib:$JAVA_HOME/jre/lib:$HADOOP_HOME
export PATH=.:$SPARK_HOME/bin:$HIVE_HOME/bin:$HADOOP_HOME/bin:$JAVA_HOME/bin:/sbin/:/bin:$PATH

格式化 HDFS：hadoop namenode -format  只使用一次

常用命令：
1.启动 HDFS： hadoop/sbin/start-dfs.sh 启动datanode，namenode
    等价于 hadoop-daemon.sh start datanode  
           hadoop-daemon.sh start namenode
2.启动 Yarn: hadoop/sbin/start-yarn.sh  启动resourcemanager，nodemanager
    等价于 yarn-daemon.sh start resourcemanager
           yarn-daemon.sh start nodemanager
3.启动HDFS+Yarn：hadoop/sbin/start-all.sh 相当于start-dfs.sh + start-yarn.sh

4.停止 HDFS： hadoop/sbin/stop-dfs.sh 停止datanode，namenode
    等价于 hadoop-daemon.sh stop datanode  
           hadoop-daemon.sh stop namenode
5.停止 Yarn: hadoop/sbin/stop-yarn.sh  停止resourcemanager，nodemanager
    等价于 yarn-daemon.sh stop resourcemanager
           yarn-daemon.sh stop nodemanager
6.停止HDFS+Yarn：hadoop/sbin/stop-all.sh 相当于stop-dfs.sh + stop-yarn.sh

7.启动mysql
  sudo service mysqld start   //ubuntu上:sudo  /etc/init.d/mysql start/restart
 
8.启动hive的metastore
  nohup hive --service metastore >> /study/hive/metastore.log 2>&1 &
 
9.启动hiveserver2，这样才能web访问hive
  nohup hive --service hiveserver2 >> /study/hive/hiveserver.log 2>&1 &

10.启动MR HistoryServer
  hadoop/sbin/mr-jobhistory-daemon.sh start historyserver
 
11.启动hadoop测试任务
  hadoop/bin/hadoop jar    /pathto/hadoop-mapreduce-examples-2.7.3.jar    pi    100    1000
 
12.查看yarn任务
  hadoop/bin/yarn application -status application_1505620174459_0001(mapreduce任务id)
  hadoop/bin/yarn application -kill application_1505620174459_0001
  hadoop/bin/yarn logs -applicationId application_1505620174459_0001  
 
13.启动和停止presto-server
/study/presto-server/bin/launcher start/stop

14.启动presto client
presto --server master:8080 --catalog hive --schema default

15.spark进入本地模式
spark-shell --master local

16.spark进入YARN分布式模式(yarn-cluster,yarn-client)
/study/spark/bin/spark-submit --master yarn-cluster/yarn-client --class org.training.examples.WordCount target/examples-1.0-SNAPSHOT.jar yarn-cluster/yarn-client /whb/data/README.txt

常见问题：
1.Ubuntu安装mysql：    
　　1. sudo apt-get install mysql-server
　　2. sudo apt-get install mysql-client
　　3. sudo apt-get install libmysqlclient-dev

2.mysql授权
  #mysql -uroot -pPASS
  mysql>GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' IDENTIFIED BY '1234' WITH GRANT OPTION;
  grant all on *.* to wanghb@'%' identified by 'wanghb';
    grant all on *.* to wanghb@'localhost' identified by 'wanghb';
    grant all on *.* to wanghb@'master' identified by 'wanghb';
  mysql> FLUSH PRIVILEGES;
 
3.配置好hive的hive-site.xml以后一定要先运行一次这个命令  
  schematool -dbType mysql -initSchema  
 
  如果执行失败报错：jdbc4.CommunicationsException : Communications link failure
  The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
 
  先尝试：mysql -h 主机名 -u root -p1234 看能否连接mysql  
  修改：/etc/mysql/my.cnf
  bind-address  = 0.0.0.0 默认只监听127.0.0.1