set up hadoop/hive/spark cluster on ec2

AWS EMR 启动的机器都很贵，想在3台t2.micro上搭建一个hadoop集群，因为t2.micro 内存1G不够跑spark，换用t2.small的2G内机器。

0. download java without auth:
wget --no-check-certificate -c --header "Cookie: oraclelicense=accept-securebackup-cookie"

1. choose AMI:
bitnami-hadoop-2.8.2-0-linux-debian-8-x86_64-hvm-ebs - ami-00654a65

2.configure security group ports ()
include: 22, 80, 443

Each daemon in Hadoop listens to a different port. The most relevant ones are:

• ResourceManager:
  • Service: 8032
  • Web UI: 8088
• NameNode:
  • Metadata: 9000
  • Web UI: 50070
• Secondary NameNode:
  • Metadata: 50090
• DataNode:
  • Data transfer: 50010
  • Metadata: 50020
  • WebUI: 50075
• Timeline Server:
  • Service: 10200
  • WebUI: 8188
• Hive:
  • Hiveserver2 binary: 10000
  • Hiveserver2 HTTP: 10001
  • Metastore: 9083
  • WebHCat: 50111
  • Derby DB: 1527

3.hadoop configurations:
ssh -i jameson-keypair.pem bitnami@13.59.230.131
/opt/bitnami/hadoop/etc/hadoop/core-site.xml
hdfs://localhost:9000
/opt/bitnami/hadoop/etc/hadoop/hdfs-site.xml
1
/opt/bitnami/hadoop/etc/hadoop/yarn-site.xml
yarn.resourcemanager.hostname NEED TO BE SET!
/opt/bitnami/hadoop/etc/hadoop/mapred-site.xml




1. Typically you want 2-4 partitions for each CPU in your cluster. Normally, Spark tries to set the number of partitions automatically based on your cluster. However, you can also set it manually by passing it as a second parameter to parallelize (e.g. sc.parallelize(data, 10)).
2.The textFile method also takes an optional second argument for controlling the number of partitions of the file. By default, Spark creates one partition for each block of the file (blocks being 128MB by default in HDFS), but you can also ask for a higher number of partitions by passing a larger value.