【案例】dfs.datanode.max.xcievers参数导致hbase-0.92集群报错

文章来源：http://bkeep.blog.163.com/blog/static/123414290201272644422987/

场景:

15个datanode挂掉，只有2个存活

[dwhftp@dw-hbase-1 ~]$ hadoop dfsadmin -report   

Configured Capacity: 73837983129600 (67.16 TB)

Present Capacity: 69740285348454 (63.43 TB)

DFS Remaining: 61837580668928 (56.24 TB)

DFS Used: 7902704679526 (7.19 TB)

DFS Used%: 11.33%

Under replicated blocks: 0

Blocks with corrupt replicas: 0

Missing blocks: 0

 

14个regionserver只有2个存活

[dwhftp@dw-hbase-11 ~]$ hbase shell

HBase Shell; enter 'help' for list of supported commands.

Type "exit" to leave the HBase Shell

Version 0.92.0, r1231986, Mon Jan 16 13:16:35 UTC 2012

 

hbase(main):001:0> status

14 servers, 0 dead, 4739.8571 average load

 

服务进程没有挂掉，但不能对外提供服务，且服务器内存消耗16G

dwhftp   29754  1 15 Jul30 ?        3-07:58:27 /usr/alibaba/java/bin/java -XX:OnOutOfMemoryError=kill -9 %p -Xmx16000m -Dhbase.log.dir=/home/dwhftp/opt/hbase/logs -Dhbase.log.file=hbase-dwhftp-regionserver-dw-hbase-9.hst.ali.dw.alidc.net.log -Dhbase.home.dir=/home/dwhftp/opt/hbase -Dhbase.id.str=dwhftp -Dhbase.root.logger=INFO,DRFA -Djava.library.path=/home/dwhftp/opt/hbase/lib/native/Linux-amd64-64 -classpath /home/dwhftp/opt/hbase/conf:/usr/alibaba/java/lib/tools.jar:/home/dwhftp/opt/hbase:/home/dwhftp/opt/hbase/hbase-0.92.0.jar:/home/dwhftp/opt/hbase/hbase-0.92.0-tests.jar:/home/dwhftp/opt/hbase/lib/activation-1.1.jar:/home/dwhftp/opt/hbase/lib/asm-3.1.jar:/home/dwhftp/opt/hbase/lib/avro-1.5.3.jar:/home/dwhftp/opt/hbase/lib/avro-ipc-1.5.3.jar:/home/dwhftp/opt/hbase/lib/commons-beanutils-1.7.0.jar:/home/dwhftp/opt/hbase/lib/commons-beanutils-core-1.8.0.jar:/home/dwhftp/opt/hbase/lib/commons-cli-1.2.jar:/home/dwhftp/opt/hbase/lib/commons-codec-1.4.jar:/home/dwhftp/opt/hbase/lib/commons-collections-3.2.1.jar:/home/dwhftp/opt/hbase/lib/commons-configuration-1.6.jar:/home/dwhftp/opt/hbase/lib/commons-digester-1.8.jar:/home/dwhftp/opt/hbase/lib/commons-el-1.0.jar:/home/dwhftp/opt/hbase/lib/commons-httpclient-3.1.jar:/home/dwhftp/opt/hbase/lib/commons-lang-2.5.jar:/home/dwhftp/opt/hbase/lib/commons-logging-1.1.1.jar:/home/dwhftp/opt/hbase/lib/commons-math-2.1.jar:/home/dwhftp/opt/hbase/lib/commons-net-1.4.1.jar:/home/dwhftp/opt/hbase/lib/core-3.1.1.jar:/home/dwhftp/opt/hbase/lib/guava-r09.jar:/home/dwhftp/opt/hbase/lib/guava-r09-jarjar.jar:/home/dwhftp/opt/hbase/lib/hadoop-core-0.20.2-cdh3u3.jar:/home/dwhftp/opt/hbase/lib/high-scale-lib-1.1.1.jar:/home/dwhftp/opt/hbase/lib/httpclient-4.0.1.jar:/home/dwhftp/opt/hbase/lib/httpcore-4.0.1.jar:/home/dwhftp/opt/hbase/lib/jackson-core-asl-1.5.5.jar:/home/dwhftp/opt/hbase/lib/jackson-jaxrs-1.5.5.jar:/home/dwhftp/opt/hbase/lib/jackson-mapper-asl-1.5.5.jar:/home/dwhftp/opt/hbase/lib/jackson-xc-1.5.5.jar:/home/dwhftp/opt/hbase/lib/jamon-runtime-2.3.1.jar:/home/dwhftp/opt/hbase/lib/jasper-compiler-5.5.23.jar:/home/dwhftp/opt/hbase/lib/jasper-runtime-5.5.23.jar:/home/dwhftp/opt/hbase/lib/jaxb-api-2.1.jar:/home/dwhftp/opt/hbase/lib/jaxb-impl-2.1.12.jar:/home/dwhftp/opt/hbase/lib/jersey-core-1.4.jar:/home/dwhftp/opt/hbase/lib/jersey-json-1.4.jar:/home/dwhftp/opt/hbase/lib/jersey-server-1.4.jar:/home/dwhftp/opt/hbase/lib/jettison-1.1.jar:/home/dwhftp/opt/hbase/lib/jetty-6.1.26.jar:/home/dwhftp/opt/hbase/lib/jetty-util-6.1.26.jar:/home/dwhftp/opt/hbase/lib/jruby-complete-1.6.5.jar:/home/dwhftp/opt/hbase/lib/jsp-2.1-6.1.14.jar:/home/dwhftp/opt/hbase/lib/jsp-api-2.1-6.1.14.jar:/home/dwhftp/opt/hbase/lib/libthrift-0.7.0.jar:/home/dwhftp/opt/hbase/lib/log4j-1.2.16.jar:/home/dwhftp/opt/hbase/lib/netty-3.2.4.Final.jar:/home/dwhftp/opt/hbase/lib/protobuf-java-2.4.0a.jar:/home/dwhftp/opt/hbase/lib/servlet-api-2.5-6.1.14.jar:/home/dwhftp/opt/hbase/lib/servlet-api-2.5.jar:/home/dwhftp/opt/hbase/lib/slf4j-api-1.5.8.jar:/home/dwhftp/opt/hbase/lib/slf4j-log4j12-1.5.8.jar:/home/dwhftp/opt/hbase/lib/snappy-java-1.0.3.2.jar:/home/dwhftp/opt/hbase/lib/stax-api-1.0.1.jar:/home/dwhftp/opt/hbase/lib/velocity-1.7.jar:/home/dwhftp/opt/hbase/lib/xmlenc-0.52.jar:/home/dwhftp/opt/hbase/lib/zookeeper-3.4.2.jar::/home/dwhftp/opt/hadoop/conf:/home/dwhftp/opt/hadoop/conf org.apache.hadoop.hbase.regionserver.HRegionServer start

 

hbase查看表时报错如下

hbase(main):001:0> scan '20120819entry'

ROW                           COLUMN+CELL                                                                         

SLF4J: Class path contains multiple SLF4J bindings.

SLF4J: Found binding in [jar:file:/home/dwhftp/opt/install/hbase-0.92.0/lib/slf4j-log4j12-1.5.8.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: Found binding in [jar:file:/home/dwhftp/opt/install/hadoop-0.20/lib/slf4j-log4j12-1.4.3.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: See for an explanation.

 

ERROR: org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: .META.,,1

        at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2819)

        at org.apache.hadoop.hbase.regionserver.HRegionServer.getClosestRowBefore(HRegionServer.java:1755)

        at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

        at java.lang.reflect.Method.invoke(Method.java:597)

        at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)

        at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)

 

regionserver报错日志

org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease on /hbase/.logs/dw-hbase-17,60020,1345529427507/dw-hbase-17%2C60020%2C1345529427507.1345529431034 File does not exist. [Lease.  Holder: DFSClient_-1871695140, pendingcreates: 4]

省略部分文字...

2012-08-21 14:35:13,530 WARN org.apache.hadoop.hdfs.DFSClient: Error while syncing

java.io.IOException: Bad connect ack with firstBadLink as 172.16.197.18:50010

 

HMaster 上的报错

[dwhftp@dw-hbase-3 logs]$ pwd

/home/dwhftp/opt/hbase/logs

[dwhftp@dw-hbase-3 logs]$ tail -200f hbase-dwhftp-master-dw-hbase-3.hst.ali.dw.alidc.net.log

2012-08-21 10:37:56,548 INFO org.apache.hadoop.hbase.catalog.CatalogTracker: Failed verification of .META.,,1 at address=dw-hbase-9,60020,1343643389530; org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: .META.,,1

 

hadoop日志（问题终于找到了！）

[dwhftp@dw-hbase-13 logs]$ pwd

/home/dwhftp/opt/hadoop/logs

[dwhftp@dw-hbase-13 logs]$ vi hadoop-dwhftp-datanode-dw-hbase-13.hst.ali.dw.alidc.net.log

 

2012-08-21 14:35:00,203 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(172.16.197.25:50010, storageID=DS-2042732685-172.16.197.25-50010-1334122560477, infoPort=50075, ipcPort=50020):DataXceiver

java.io.IOException: xceiverCount 4097 exceeds the limit of concurrent xcievers 4096

        at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:156)

 

解决办法，调整xcievers参数

默认是4096，改为8192

vi /home/dwhftp/opt/hadoop/conf/hdfs-site.xml

  dfs.datanode.max.xcievers

  8192

 

dfs.datanode.max.xcievers 参数说明

一个 Hadoop HDFS Datanode 有一个同时处理文件的上限. 这个参数叫 xcievers (Hadoop的作者把这个单词拼错了). 在你加载之前，先确认下你有没有配置这个文件conf/hdfs-site.xml里面的xceivers参数，至少要有4096:

 

      

        dfs.datanode.max.xcievers

        4096

      

     

对于HDFS修改配置要记得重启.

 

如果没有这一项配置，你可能会遇到奇怪的失败。你会在Datanode的日志中看到xcievers exceeded，但是运行起来会报 missing blocks错误。例如: 10/12/08 20:10:31 INFO hdfs.DFSClient: Could not obtain block blk_XXXXXXXXXXXXXXXXXXXXXX_YYYYYYYY from any node: java.io.IOException: No live nodes contain current block. Will get new block locations from namenode and retry... [5]

 

重启hbase和hdfs

hbase-1

[dwhftp@dw-hbase-1 ~]$ start-dfs.sh

 

hbase-3

[dwhftp@dw-hbase-3 ~]$ start-hbase.sh

 

hbase-3

还要启动mapper/reduce用于和云梯同步数据

[dwhftp@dw-hbase-3 ~]$start-mapred.sh

后续动作

添加监控

端口监控、进程监控、

hbase-1 (namenode) 9000  50070  80  2181  3555

hbase-2 (SecondaryNameNode) 50090  2181  3555

hbase-3 (HMaster) 60010  80  2181  3555

hbase-4 (HMaster备) 60000  2181  3555

hbase-5 （HRegionServer） 60020  60030  2181  3555  datanode  50010  50075

hbase-6 ~ dw-hbase-18 （HRegionServer） 60020  60030  datanode  50010  50075

 

自己写脚本监控

 

监控活动的hbase和datanode

监控日志中的ERROR关键字

 

容量

 

跟开发人员约定可用性

namenode是单点，所以肯定不是7*24小时服务。单台机器挂掉hbase集群需要时间重建hbase，分钟级别的不可用。必须跟他们讲清楚。然后确定谁在维护hbase

 

另外创建一个hbase维护群，方便大家交流。

hbase资源申请

所以应用共同使用hbase集群，所以资源隔离。业务评审都需要有流程和规范。

 

日志级别调整

info、debug、warn、error放在同一个日志文件里面，信息量太多导致排错时间长

通过配置log4j将error级别日志打到单独文件里面。

 

向淘宝取经

日常维护动作

 

监控点