Chinaunix首页 | 论坛 | 博客
  • 博客访问: 2194693
  • 博文数量: 866
  • 博客积分: 14125
  • 博客等级: 上将
  • 技术积分: 10638
  • 用 户 组: 普通用户
  • 注册时间: 2007-07-27 16:53
个人简介

https://github.com/landuochong

文章分类

全部博文(866)

文章存档

2019年(3)

2018年(1)

2017年(10)

2015年(3)

2014年(8)

2013年(3)

2012年(70)

2011年(103)

2010年(360)

2009年(283)

2008年(22)

分类: 大数据

2017-11-04 15:49:09

环境设置:
export JAVA_HOME=/study/jdk
export HADOOP_HOME=/study/hadoop
export HIVE_HOME=/study/hive
export SPARK_HOME=/study/spark

export CLASSPATH=$HIVE_HOME/lib:$JAVA_HOME/lib:$JAVA_HOME/jre/lib:$HADOOP_HOME
export PATH=.:$SPARK_HOME/bin:$HIVE_HOME/bin:$HADOOP_HOME/bin:$JAVA_HOME/bin:/sbin/:/bin:$PATH

格式化 HDFS:hadoop namenode -format  只使用一次

常用命令:
1.启动 HDFS: hadoop/sbin/start-dfs.sh 启动datanode,namenode
    等价于 hadoop-daemon.sh start datanode  
           hadoop-daemon.sh start namenode
2.启动 Yarn: hadoop/sbin/start-yarn.sh  启动resourcemanager,nodemanager
    等价于 yarn-daemon.sh start resourcemanager
           yarn-daemon.sh start nodemanager
3.启动HDFS+Yarn:hadoop/sbin/start-all.sh 相当于start-dfs.sh + start-yarn.sh

4.停止 HDFS: hadoop/sbin/stop-dfs.sh 停止datanode,namenode
    等价于 hadoop-daemon.sh stop datanode  
           hadoop-daemon.sh stop namenode
5.停止 Yarn: hadoop/sbin/stop-yarn.sh  停止resourcemanager,nodemanager
    等价于 yarn-daemon.sh stop resourcemanager
           yarn-daemon.sh stop nodemanager
6.停止HDFS+Yarn:hadoop/sbin/stop-all.sh 相当于stop-dfs.sh + stop-yarn.sh

7.启动mysql
  sudo service mysqld start   //ubuntu上:sudo  /etc/init.d/mysql start/restart
 
8.启动hive的metastore
  nohup hive --service metastore >> /study/hive/metastore.log 2>&1 &
 
9.启动hiveserver2,这样才能web访问hive
  nohup hive --service hiveserver2 >> /study/hive/hiveserver.log 2>&1 &

10.启动MR HistoryServer
  hadoop/sbin/mr-jobhistory-daemon.sh start historyserver
 
11.启动hadoop测试任务
  hadoop/bin/hadoop jar    /pathto/hadoop-mapreduce-examples-2.7.3.jar    pi    100    1000
 
12.查看yarn任务
  hadoop/bin/yarn application -status application_1505620174459_0001(mapreduce任务id)
  hadoop/bin/yarn application -kill application_1505620174459_0001
  hadoop/bin/yarn logs -applicationId application_1505620174459_0001  
 
13.启动和停止presto-server
/study/presto-server/bin/launcher start/stop

14.启动presto client
presto --server master:8080 --catalog hive --schema default

15.spark进入本地模式
spark-shell --master local

16.spark进入YARN分布式模式(yarn-cluster,yarn-client)
/study/spark/bin/spark-submit --master yarn-cluster/yarn-client --class org.training.examples.WordCount target/examples-1.0-SNAPSHOT.jar yarn-cluster/yarn-client /whb/data/README.txt

常见问题:
1.Ubuntu安装mysql:    
  1. sudo apt-get install mysql-server
  2. sudo apt-get install mysql-client
  3. sudo apt-get install libmysqlclient-dev

2.mysql授权
  #mysql -uroot -pPASS
  mysql>GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' IDENTIFIED BY '1234' WITH GRANT OPTION;
  grant all on *.* to wanghb@'%' identified by 'wanghb';
    grant all on *.* to wanghb@'localhost' identified by 'wanghb';
    grant all on *.* to wanghb@'master' identified by 'wanghb';
  mysql> FLUSH PRIVILEGES;
 
3.配置好hive的hive-site.xml以后一定要先运行一次这个命令  
  schematool -dbType mysql -initSchema  
 
  如果执行失败报错:jdbc4.CommunicationsException : Communications link failure
  The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
 
  先尝试:mysql -h 主机名 -u root -p1234 看能否连接mysql  
  修改:/etc/mysql/my.cnf
  bind-address  = 0.0.0.0 默认只监听127.0.0.1
 


阅读(1571) | 评论(0) | 转发(0) |
给主人留下些什么吧!~~
评论热议
请登录后评论。

登录 注册