Chinaunix首页 | 论坛 | 博客
  • 博客访问: 646892
  • 博文数量: 121
  • 博客积分: 4034
  • 博客等级: 上校
  • 技术积分: 1439
  • 用 户 组: 普通用户
  • 注册时间: 2010-04-28 12:42
文章分类

全部博文(121)

文章存档

2017年(8)

2016年(10)

2013年(2)

2012年(3)

2011年(18)

2010年(80)

分类: 服务器与存储

2016-08-13 11:34:48

(1)禁用hugepage
使用命令cat /sys/kernel/mm/transparent_hugepage/defrag查看是不是打开,如果不是nerver,需要配置。
执行命令:runlevel,确定运行级别,ubuntu默认为2
添加文件:/etc/init.d/dishuge
  1. #!/bin/sh -e

  2. echo "[dancylog]enable transparent_hugepage"
  3. if test -f /sys/kernel/mm/transparent_hugepage/enabled; then
  4.     echo never > /sys/kernel/mm/transparent_hugepage/enabled
  5. fi
  6. if test -f /sys/kernel/mm/transparent_hugepage/defrag; then
  7.     echo never > /sys/kernel/mm/transparent_hugepage/defrag
  8. fi
设置可执行权限:chmod 777 dishuge

进入/etc/rc2.d目录(rc加运行级别),添加一个软连接:
ln -s ../init.d/dishuge S10dishuge
重启就可以了,可以使用命令:
cat  /sys/kernel/mm/transparent_hugepage/defrag确定是否设置成功了。

(2)禁用swap
echo "vm.swappiness=0" >> /etc/sysctl.conf
sysctl -p
重启使用命令:
cat /proc/sys/vm/swappines
查看是否设置成功。

(3)安装ssh,mysql,ntp
  1. sudo apt-get install openssh-client=1:6.6p1-2ubuntu1
  2. sudo apt-get install openssh-server
  3. sudo apt-get install mysql-server mysql-client libmysql-java ntp
(4)卸载iptables
sudo apt-get remove iptables

(5)安装jdk
sudo mkdir /usr/lib/jvm
进入jdk所在的目录
sudo tar zxvf jdk-7u79-linux-x64.tar.gz -C /usr/lib/jvm
sudo gedit /etc/profile

  1. export JAVA_HOME=/usr/lib/jvm/jdk1.7.0_79
  2. export JRE_HOME=${JAVA_HOME}/jre
  3. export CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib
  4. export PATH=${JAVA_HOME}/bin:$PATH
/etc/environment

  1. export JAVA_HOME=/usr/lib/jvm/jdk1.7.0_79
  2. export CLASSPATH=.:$JAVA_HOME/lib:$JAVA_HOME/jre/lib
  3. PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:$JAVA_HOME/bin"
配置jdk默认版本
  1. sudo update-alternatives --install /usr/bin/java java /usr/lib/jvm/jdk1.7.0_79/bin/java 300
  2. sudo update-alternatives --install /usr/bin/javac javac /usr/lib/jvm/jdk1.7.0_79/bin/javac 300
  3. sudo update-alternatives --install /usr/bin/jar jar /usr/lib/jvm/jdk1.7.0_79/bin/jar 300
  4. sudo update-alternatives --install /usr/bin/javah javah /usr/lib/jvm/jdk1.7.0_79/bin/javah 300
  5. sudo update-alternatives --install /usr/bin/javap javap /usr/lib/jvm/jdk1.7.0_79/bin/javap 300

  6. sudo update-alternatives --config java
java -version
echo $JAVA_HOME
(6)修改hostname
需要修改2个配置文件
/etc/hosts
/etc/hostname
根据需要更改就行了。
(7)ssh免交互
ssh-keygen -t rsa
cp id_rsa.pub authorized_keys
不用更改权限,生成的文件每个电脑需要拷贝一份。
(8)mysql 配置
需要修改配置文件:/etc/mysql/my.cnf
  1. bind-address = 0.0.0.0
  2. //////////////////////下面的复制。

  3. transaction-isolation = READ-COMMITTED

  4. key_buffer = 16M
  5. key_buffer_size = 32M
  6. max_allowed_packet = 32M
  7. thread_stack = 256K
  8. thread_cache_size = 64
  9. query_cache_limit = 8M
  10. query_cache_size = 64M
  11. query_cache_type = 1

  12. max_connections = 550

  13. log_bin=/var/lib/mysql/mysql_binary_log

  14. binlog_format = mixed

  15. read_buffer_size = 2M
  16. read_rnd_buffer_size = 16M
  17. sort_buffer_size = 8M
  18. join_buffer_size = 8M

  19. innodb_file_per_table = 1
  20. innodb_flush_log_at_trx_commit = 2
  21. innodb_log_buffer_size = 64M
  22. innodb_buffer_pool_size = 4G
  23. innodb_thread_concurrency = 8
  24. innodb_flush_method = O_DIRECT
  25. innodb_log_file_size = 52M

  26. [mysqld_safe]
  27. log-error=/var/log/mysqld.log
  28. pid-file=/var/run/mysqld/mysqld.pid

  29. sql_mode=STRICT_ALL_TABLES
重启mysql
service mysql restart
添加必要的数据库:
  1. create database hive DEFAULT CHARACTER SET utf8 COLLATE utf8_general_ci;
  2. create database hue DEFAULT CHARACTER SET utf8 COLLATE utf8_general_ci;
  3. create database am DEFAULT CHARACTER SET utf8 COLLATE utf8_general_ci;
  4. create database oozie DEFAULT CHARACTER SET utf8 COLLATE utf8_general_ci;
  5. grant all on hive.* to 'hl'@'%' identified by 'hl123';
  6. grant all on hue.* to 'hl'@'%' identified by 'hl123';
  7. grant all on am.* to 'hl'@'%' identified by 'hl123';
  8. grant all on oozie.* to 'hl'@'%' identified by 'hl123';
  9. grant all on scm.* to 'hl'@'%' identified by 'hl123';
  10. FLUSH PRIVILEGES;
  11. exit;
如果想删除对某个数据库的权限:revoke all on wifi.* from 'hl'@'%';
(9)添加用户
useradd --system --home=/opt/cm-5.4.0/run/cloudera-scm-server --no-create-home --shell=/bin/false --comment "Cloudera SCM User" cloudera-scm
(10) cm-5.4.0修改配置
更改:/opt/cm-5.4.0/etc/cloudera-scm-agent/config.ini
server_host改为hostname中的名字。
如果是agent,只需要更改config.ini,下面的不需要
配置数据库
/opt/cm-5.4.0/share/cmf/schema/scm_prepare_database.sh mysql scm -hlocalhost -uroot -p --scm-host localhost scm scm scm
查看/opt/cm-5.4.0/etc/cloudera-scm-server/db.properties是否修改正确,
如果不正确安装下面更改:
  1. com.cloudera.cmf.db.type=mysql
  2. com.cloudera.cmf.db.host=localhost
  3. com.cloudera.cmf.db.name=cloudera
  4. com.cloudera.cmf.db.user=root
  5. com.cloudera.cmf.db.password=hl123
(11)拷贝CDH5
mkdir -p /opt/cloudera/parcel-repo/
拷贝CDH5到这个目录下。
(12)添加必要的目录
mkdir -p /opt/cm-5.4.0/run/cloudera-scm-agent;mkdir -p /var/lib/cloudera-scm-server
(13)启动server和agent
/opt/cm-5.4.0/etc/init.d/cloudera-scm-server start
/opt/cm-5.4.0/etc/init.d/cloudera-scm-agent start
(14)卸载CDH5和重装
删除数据库:
  1. drop database scm;
  2. drop database hive;
  3. drop database hue;
  4. drop database oozie;
  5. drop database am;
删除用户:
  1. userdel -r hdfs
  2. userdel -r solr
  3. userdel -r zookeeper
  4. userdel -r llama
  5. userdel -r httpfs
  6. userdel -r mapred
  7. userdel -r sqoop
  8. userdel -r yarn
  9. userdel -r kms
  10. userdel -r hive
  11. userdel -r sqoop2
  12. userdel -r oozie
  13. userdel -r hbase
  14. userdel -r sentry
  15. userdel -r impala
  16. userdel -r spark
  17. userdel -r hue
  18. userdel -r cloudera-scm
  19. groupdel hadoop
  20. groupdel flume
  21. groupdel hdfs
  22. groupdel sqoop
  23. groupdel hive
删除文件:
  1. rm /impala/ /dfs/ /yarn/ -rf
  2. rm /var/log/hive/ /var/log/hue/ /var/log/impalad/ /var/log/oozie/ /var/log/sqoop2/ /var/log/zookeeper/ -rf
  3. rm /var/log/cloudera-scm-* /var/log/hadoop-* -rf
  4. rm /var/tmp -rf
  5. rm /var/lib/cloudera-* /var/lib/hadoop-* /var/lib/hue/ /var/lib/impala/ /var/lib/oozie/ /var/lib/sqoop2/ /var/lib/zookeeper/ /var/lib/spark/ -rf
阅读(1799) | 评论(0) | 转发(0) |
给主人留下些什么吧!~~