需要环境:
PC-1 Suse Linux 9 10.192.1.1
PC-2 Suse Linux 9 10.192.1.2
PC-3 Suse Linux 9 10.192.1.3
PC-4 Suse Linux 9 10.192.1.4
其中,PC-1做namenode节点,PC-2、PC-3和PC-4做datanode节点。
并且已经安装成功Hadoop-0.20.1及以上版本。 安装包准备
需要安装包:
zookeeper-3.2.1.tar.gz(stable版本)
hbase-0.20.1.tar.gz(stable版本) 安装步骤
安装和配置ZooKeeper
HBase从0.20.0开始,需要首先安装ZooKeeper。从apache上下载zookeeper-3.2.1.tar.gz(Stable版本),解压到/home/hdfs/目录下。
(1),在namenode节点新建zookeeper目录,在该目录下新建myid文件。
(2),在zookeeper-3.2.1/conf目录下,拷贝zoo_sample.cfg为zoo.cfg。在zoo.cfg中将dataDir改为/home/hdfs/zookeeper,在文件末位添加所有的主机:
server.1=10.192.1.1:2888:3888
server.2=10.192.1.2:2888:3888
server.3=10.192.1.3:2888:3888
server.4=10.192.1.4:2888:3888
server.5=10.192.1.5:2888:3888
server.6=10.192.1.62888:3888
(3)用scp命令将namenode节点的的/home/hdfs/ zookeeper-3.2.1和/home/hdfs/ zookeeper拷贝到其余所有主机的/home/hdfs目录下。
(4)参照zoo.cfg中的配置,在各主机myid文件中写入各自的编号。如:10.192.1.1入1,10.192.1.2写入2
(5)在所有节点上执行bin/zkServer.sh start,分别启动。
执行bin/zkCli.sh -server xxx.xxx.xxx.xxx:2181,检查指定服务器是否成功启动。
安装和配置HBase
下载HBase0.20.1版本,解压到namenode节点的/home/hdfs目录下。 配置说明
(1)系统所有配置项的默认设置在hbase-default.xml中查看,如果需要修改配置项的值,在hbase-site.xml中添加配置项。
在分布式模式下安装HBase,需要添加的最基本的配置项如下:
hbase.rootdir
hdfs://namenode.hdfs:54310/hbase
The directory shared by region servers.
hbase.cluster.distributed
true
The mode the cluster will be in. Possible values are
false: standalone and pseudo-distributed setups with managed Zookeeper
true: fully-distributed with unmanaged Zookeeper Quorum (see hbase-env.sh)
(2)在conf/hbase-env.sh中修改添加配置项:
export JAVA_HOME=/usr/java/jdk1.6.0_16
export HBASE_MANAGES_ZK=false
export HBASE_CLASSPATH=/home/hdfs/hadoop-0.20.1/conf
并把~/hadoop-0.20.1/conf/hdfs-site.xml拷贝至~/hbase-3.2.1/conf/目录下。
(3)将ZooKeeper的配置文件zoo.cfg添加到HBase所有主机的CLASSPATH中。
(4)在conf/regionservers中添加hadoop-0.20.1/conf/slaves中所有的datanode节点。 启动
Hadoop、ZooKeeper和HBase之间应该按照顺序启动和关闭:启动Hadoop—>启动ZooKeeper集群—>启动HBase—>停止HBase—>停止ZooKeeper集群—>停止Hadoop。
在namenode节点执行bin/hbase-daemon.sh,启动master。执行bin/start-hbase.sh和bin/stop-hbase.sh 脚本启动和停止HBase服务。 接口说明
HBase按列存储结构化数据,支持建表、插入记录、查询记录、删除记录和索引操作等等,不支持连接和更新操作。 开发步骤
引入JAR包
在Windows客户端编写JAVA程序操作HBase,需要引入一些JAR包。需要引入的JAR如下:hadoop-0.20.1-core.jar,commons-logging-1.0.4.jar,commons-logging-api-1.0.4.jar,zookeeper-3.2.1.jar,hbase-0.20.1.jar,log4j-1.2.15.jar。
开发模式
在分布式模式下开发,在程序中配置与HDFS和ZooKeeper的连接,即可对数据进行操作。 view plaincopy to clipboardprint? import java.util.Date; import java.text.SimpleDateFormat; import java.io.IOException; import java.awt.List; import java.util.Map; import java.util.NavigableMap;
import org.apache.hadoop.hbase.HBaseConfiguration; import org.apache.hadoop.hbase.HTableDescriptor; import org.apache.hadoop.hbase.HColumnDescriptor; import org.apache.hadoop.hbase.client.HBaseAdmin; import org.apache.hadoop.hbase.client.HTable;
import org.apache.hadoop.hbase.client.Put; import org.apache.hadoop.hbase.client.Get; import org.apache.hadoop.hbase.client.Scan; import org.apache.hadoop.hbase.client.ResultScanner; import org.apache.hadoop.hbase.client.Delete; import org.apache.hadoop.hbase.client.Result; import org.apache.hadoop.hbase.util.Bytes;
public class HBaseManager { public static void main(String[] args) throws Exception{ HBaseManager manager = new HBaseManager(); manager.testGet(); }
public void testQueryRS()throws Exception{ HBaseConfiguration config = new HBaseConfiguration(); config.set("hbase.master", "10.192.1.1:60000"); config.set("hbase.zookeeper.quorum", "10.192.1.1");
HTable table = new HTable(config, "commodity");
System.out.println("Get Spin's commodity info");
Scan scanner = new Scan(); scanner.addColumn(Bytes.toBytes("description")); scanner.setMaxVersions(); ResultScanner rsScanner = table.getScanner(scanner); System.out.println(rsScanner.toString()); Result rs = rsScanner.next(); while(null != rs){ System.out.println(rs.size()); NavigableMap>> nMap = rs.getMap(); NavigableMap> columnMap = nMap.get(Bytes.toBytes("description")); NavigableMap qualMap = columnMap.get(new byte[]{});
if(qualMap.entrySet().size() > 0){ System.out.println("---------------------------"); for(Map.Entry m :qualMap.entrySet()) { System.out.println("Value:"+ new String(m.getValue())); } } rs = rsScanner.next(); } }
public void testQueryCommodity()throws Exception{ HBaseConfiguration config = new HBaseConfiguration(); config.set("hbase.master", "10.192.1.1:60000"); config.set("hbase.zookeeper.quorum", "10.192.1.1.203");
HTable table = new HTable(config, "commodity");
System.out.println("Get Spin's commodity info"); Get mathGet = new Get(new String("Spin").getBytes()); mathGet.addColumn(Bytes.toBytes("widgetname")); mathGet.setMaxVersions(); Result rs = table.get(mathGet);
NavigableMap>> nMap = rs.getMap(); NavigableMap> columnMap = nMap.get(Bytes.toBytes("widgetname")); NavigableMap qualMap = columnMap.get(new byte[]{});
if(qualMap.entrySet().size() > 0){ for(Map.Entry m :qualMap.entrySet()) { System.out.println("Value:"+ new String(m.getValue()));
break; } } }
public void test()throws Exception{ HBaseConfiguration config = new HBaseConfiguration(); config.set("hbase.master", "10.192.1.1:60000"); config.set("hbase.zookeeper.quorum", "10.192.1.1"); HBaseAdmin admin = new HBaseAdmin(config); HTable table = new HTable(config, "scores");
if (admin.tableExists("scores")){ System.out.println("drop table"); admin.disableTable("scores"); admin.deleteTable("scores"); }
System.out.println("create table"); HTableDescriptor tableDescripter = new HTableDescriptor("scores".getBytes()); tableDescripter.addFamily(new HColumnDescriptor("grade")); tableDescripter.addFamily(new HColumnDescriptor("course"));
admin.createTable(tableDescripter);
System.out.println("add Tom's data");
Put tomPut = new Put(new String("Tom").getBytes()); tomPut.add(new String("grade").getBytes(), new byte[]{}, new String("1").getBytes()); tomPut.add(new String("grade").getBytes(), new String("math").getBytes(), new String("87").getBytes()); tomPut.add(new String("course").getBytes(), new String("math").getBytes(), new String("97").getBytes()); table.put(tomPut);
System.out.println("add Jerry's data");
Put jerryPut = new Put(new String("Jerry").getBytes()); jerryPut.add(new String("grade").getBytes(), new byte[]{}, new String("2").getBytes()); jerryPut.add(new String("grade").getBytes(), new String("math").getBytes(), new String("77").getBytes()); jerryPut.add(new String("course").getBytes(), new String("math").getBytes(), new String("92").getBytes()); table.put(jerryPut);
System.out.println("Get Tom's data"); Get tomGet = new Get(new String("Tom").getBytes()); Result tomResult = table.get(tomGet); System.out.println(tomResult.toString());
System.out.println("Get Tom's Math grade"); Get mathGet = new Get(new String("Tom").getBytes()); mathGet.addColumn(Bytes.toBytes("grade")); mathGet.setMaxVersions(); Result rs = table.get(mathGet);
NavigableMap>> nMap = rs.getMap(); NavigableMap> columnMap = nMap.get(Bytes.toBytes("grade")); NavigableMap qualMap = columnMap.get(Bytes.toBytes("math"));
for(Map.Entry m :qualMap.entrySet()) {
System.out.println("TimeStamp:"+m.getKey()); System.out.println("Value:"+ new String(m.getValue())); }
System.out.println("Delete a column"); Delete deleteArt = new Delete(Bytes.toBytes("Tom")); deleteArt.deleteColumn(Bytes.toBytes("grade"), Bytes.toBytes("math")); table.delete(deleteArt); }
public void testScanner() throws IOException{ HBaseConfiguration config = new HBaseConfiguration(); config.set("hbase.master", "10.192.1.1:60000"); config.set("hbase.zookeeper.quorum", "10.192.1.1");
HTable table = new HTable(config, "commodity");
System.out.println("Scan commodity info");
Scan scanner = new Scan(); scanner.addColumn(Bytes.toBytes("widgetname")); scanner.addColumn(Bytes.toBytes("filename")); scanner.addColumn(Bytes.toBytes("description")); scanner.addColumn(Bytes.toBytes("createtime")); //scanner.setMaxVersions(); //scanner.setMaxVersions(4); ResultScanner rsScanner = table.getScanner(scanner);
Result rs = rsScanner.next(); for(;null != rs; rs = rsScanner.next()){ System.out.println("rs.getRow()[" + new String(rs.getRow()) + "]"); System.out.println("[" + new String(rs.getValue(Bytes.toBytes("widgetname"))) + "]"); System.out.println("[" + new String(rs.getValue(Bytes.toBytes("filename"))) + "]"); System.out.println("[" + new String(rs.getValue(Bytes.toBytes("description"))) + "]"); String timeStr = new String(rs.getValue(Bytes.toBytes("createtime"))); System.out.println("[" + timeStr + "]");
SimpleDateFormat dateFormat = new SimpleDateFormat("yyyy-MM-dd hh:mm:ss"); try{ Date after = dateFormat.parse(timeStr); System.out.println(after); } catch(Exception exp){ exp.printStackTrace(); } } }
public void testGet()throws IOException{ HBaseConfiguration config = new HBaseConfiguration(); config.set("hbase.master", "10.192.1.1:60000"); config.set("hbase.zookeeper.quorum", "10.192.1.1");
HTable table = new HTable(config, "commodity");
Get get = new Get(new String("xxxx.wgt").getBytes()); get.addColumn(Bytes.toBytes("widgetname")); get.addColumn(Bytes.toBytes("filename")); get.addColumn(Bytes.toBytes("description")); get.addColumn(Bytes.toBytes("createtime"));
get.setMaxVersions(2); System.out.println("00000000000000"); Result dbResult = table.get(get);
System.out.println("11111111111111"); System.out.println(dbResult.size()); System.out.println("2222222222222222"); System.out.println(new String(dbResult.value())); System.out.println("3333333333333333"); System.out.println(dbResult.containsColumn(Bytes.toBytes("description"), new byte[]{})); System.out.println("44444444444444444"); System.out.println(dbResult.isEmpty()); System.out.println("55555555555555555"); System.out.println(dbResult.list()); System.out.println("66666666666666666"); System.out.println(dbResult.raw()); System.out.println("77777777777777777"); System.out.println(dbResult.toString()); System.out.println("88888888888888888"); System.out.println(dbResult.raw().clone()); System.out.println("99999999999999999"); } } |