ERROR: org.apache.hadoop.hbase.NotServingRegionException: user_profile,,1677228015439.69f3f6a477f90bdc138e31f08ee909d8. is not online on hdp03,16020,1677229361798
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:3272)
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:3249)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:1414)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.newRegionScanner(RSRpcServices.java:2948)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3285)
at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42002)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
Count the number of rows in a table. Return value is the number of rows.
This operation may take a LONG time (Run '$HADOOP_HOME/bin/hadoop jar
hbase.jar rowcount' to run a counting mapreduce job). Current count is shown
every 1000 rows by default. Count interval may be optionally specified. Scan
caching is enabled on count scans by default. Default cache size is 10 rows.
If your rows are small in size, you may want to increase this
parameter. Examples:
hbase> count 'ns1:t1'
hbase> count 't1'
hbase> count 't1', INTERVAL => 100000
hbase> count 't1', CACHE => 1000
hbase> count 't1', INTERVAL => 10, CACHE => 1000
hbase> count 't1', FILTER => "
(QualifierFilter (>=, 'binary:xyz')) AND (TimestampsFilter ( 123, 456))"
ERROR: org.apache.hadoop.hbase.NotServingRegionException: user_profile,,1677228015439.69f3f6a477f90bdc138e31f08ee909d8. is not online on hdp03,16020,1677229361798
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:3272)
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:3249)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:1414)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.newRegionScanner(RSRpcServices.java:2948)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3285)
at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42002)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
Scan a table; pass table name and optionally a dictionary of scanner
specifications. Scanner specifications may include one or more of:
2023-02-24 17:34:49,265 INFO [main] zookeeper.RecoverableZooKeeper: Process identifier=hbase Fsck connecting to ZooKeeper ensemble=hdp01:2181,hdp02:2181,hdp03:2181
2023-02-24 17:34:49,273 INFO [main] zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.10-39d3a4f269333c922ed3db283be479f9deacaa0f, built on 03/23/2017 10:13 GMT
2023-02-24 17:34:49,273 INFO [main] zookeeper.ZooKeeper: Client environment:host.name=hdp01
2023-02-24 17:34:49,273 INFO [main] zookeeper.ZooKeeper: Client environment:java.version=1.8.0_212
2023-02-24 17:34:49,273 INFO [main] zookeeper.ZooKeeper: Client environment:java.vendor=Oracle Corporation
2023-02-24 17:34:49,273 INFO [main] zookeeper.ZooKeeper: Client environment:java.home=/opt/apps/jdk1.8.0_212/jre
2023-02-24 17:34:49,273 INFO [main] zookeeper.ZooKeeper: /hadoop/yarn/hadoop-yarn-server-web-proxy-3.1.3.jar:/opt/apps/hadoop-3.1.3/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-3.1.3.jar:/opt/apps/hadoop-3.1.3/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-3.1.3.jar:/opt/apps/hadoop-3.1.3/share/hadoop/yarn/hadoop-yarn-api-3.1.3.jar:/opt/apps/hadoop-3.1.3/share/hadoop/yarn/hadoop-yarn-client-3.1.3.jar:/opt/apps/hadoop-3.1.3/share/hadoop/yarn/hadoop-yarn-common-3.1.3.jar:/opt/apps/hadoop-3.1.3/share/hadoop/yarn/hadoop-yarn-server-tests-3.1.3.jar:/opt/apps/hadoop-3.1.3/share/hadoop/yarn/hadoop-yarn-services-api-3.1.3.jar:/opt/apps/hadoop-3.1.3/share/hadoop/yarn/hadoop-yarn-services-core-3.1.3.jar:/opt/apps/hadoop-3.1.3/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-3.1.3.jar:/opt/apps/hadoop-3.1.3/share/hadoop/yarn/hadoop-yarn-server-common-3.1.3.jar:/opt/apps/hadoop-3.1.3/share/hadoop/yarn/hadoop-yarn-server-nodemanager-3.1.3.jar:/opt/apps/hadoop-3.1.3/share/hadoop/yarn/hadoop-yarn-server-router-3.1.3.jar
2023-02-24 17:34:49,274 INFO [main] zookeeper.ZooKeeper: Client environment:java.library.path=/opt/apps/hadoop-3.1.3/lib/native
2023-02-24 17:34:49,274 INFO [main] zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
2023-02-24 17:34:49,274 INFO [main] zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
2023-02-24 17:34:49,274 INFO [main] zookeeper.ZooKeeper: Client environment:os.name=Linux
2023-02-24 17:34:49,274 INFO [main] zookeeper.ZooKeeper: Client environment:os.arch=amd64
2023-02-24 17:34:49,274 INFO [main] zookeeper.ZooKeeper: Client environment:os.version=3.10.0-862.el7.x86_64
2023-02-24 17:34:49,274 INFO [main] zookeeper.ZooKeeper: Client environment:user.name=whybigdata
2023-02-24 17:34:49,274 INFO [main] zookeeper.ZooKeeper: Client environment:user.home=/home/whybigdata
2023-02-24 17:34:49,274 INFO [main] zookeeper.ZooKeeper: Client environment:user.dir=/opt/apps/hbase-2.0.5
2023-02-24 17:34:49,290 INFO [main-SendThread(hdp02:2181)] zookeeper.ClientCnxn: Opening socket connection to server hdp02/192.168.10.12:2181. Will not attempt to authenticate using SASL (unknown error)
Allow checking/fixes for table: user_profile
HBaseFsck command line options: user_profile
2023-02-24 17:34:49,294 INFO [main] util.HBaseFsck: Launching hbck
2023-02-24 17:34:49,295 INFO [main-SendThread(hdp02:2181)] zookeeper.ClientCnxn: Socket connection established to hdp02/192.168.10.12:2181, initiating session
2023-02-24 17:34:49,306 INFO [main-SendThread(hdp02:2181)] zookeeper.ClientCnxn: Session establishment complete on server hdp02/192.168.10.12:2181, sessionid = 0x3000001abaf0003, negotiated timeout = 40000
2023-02-24 17:34:49,353 INFO [main] zookeeper.ReadOnlyZKClient: Connect 0x45fd9a4d to hdp01:2181,hdp02:2181,hdp03:2181 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms
2023-02-24 17:34:49,360 INFO [ReadOnlyZKClient-hdp01:2181,hdp02:2181,hdp03:2181@0x45fd9a4d-SendThread(hdp02:2181)] zookeeper.ClientCnxn: Opening socket connection to server hdp02/192.168.10.12:2181. Will not attempt to authenticate using SASL (unknown error)
2023-02-24 17:34:49,361 INFO [ReadOnlyZKClient-hdp01:2181,hdp02:2181,hdp03:2181@0x45fd9a4d-SendThread(hdp02:2181)] zookeeper.ClientCnxn: Socket connection established to hdp02/192.168.10.12:2181, initiating session
2023-02-24 17:34:49,365 INFO [ReadOnlyZKClient-hdp01:2181,hdp02:2181,hdp03:2181@0x45fd9a4d-SendThread(hdp02:2181)] zookeeper.ClientCnxn: Session establishment complete on server hdp02/192.168.10.12:2181, sessionid = 0x3000001abaf0004, negotiated timeout = 40000
Version: 2.0.5
2023-02-24 17:34:49,970 INFO [main] util.HBaseFsck: Computing mapping of all store files
2023-02-24 17:34:50,240 INFO [main] util.HBaseFsck: Validating mapping using HDFS state
2023-02-24 17:34:50,240 INFO [main] util.HBaseFsck: Computing mapping of all link files
.
2023-02-24 17:34:50,292 INFO [main] util.HBaseFsck: Validating mapping using HDFS state
Number of live region servers: 3
Number of dead region servers: 0
Master: hdp01,16000,1677229359514
Number of backup masters: 0
Average load: 1.0
Number of requests: 76
Number of regions: 3
Number of regions in transition: 0
2023-02-24 17:34:50,406 INFO [main] util.HBaseFsck: Loading regionsinfo from the hbase:meta table
Number of empty REGIONINFO_QUALIFIER rows in hbase:meta: 0
2023-02-24 17:34:50,501 INFO [main] util.HBaseFsck: getTableDescriptors == tableNames => [user_profile]
2023-02-24 17:34:50,502 INFO [main] zookeeper.ReadOnlyZKClient: Connect 0x1ce93c18 to hdp01:2181,hdp02:2181,hdp03:2181 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms
2023-02-24 17:34:50,505 INFO [ReadOnlyZKClient-hdp01:2181,hdp02:2181,hdp03:2181@0x1ce93c18-SendThread(hdp01:2181)] zookeeper.ClientCnxn: Opening socket connection to server hdp01/192.168.10.11:2181. Will not attempt to authenticate using SASL (unknown error)
2023-02-24 17:34:50,506 INFO [ReadOnlyZKClient-hdp01:2181,hdp02:2181,hdp03:2181@0x1ce93c18-SendThread(hdp01:2181)] zookeeper.ClientCnxn: Socket connection established to hdp01/192.168.10.11:2181, initiating session
2023-02-24 17:34:50,521 INFO [ReadOnlyZKClient-hdp01:2181,hdp02:2181,hdp03:2181@0x1ce93c18-SendThread(hdp01:2181)] zookeeper.ClientCnxn: Session establishment complete on server hdp01/192.168.10.11:2181, sessionid = 0x20000018ff60005, negotiated timeout = 40000
2023-02-24 17:34:50,537 INFO [main] client.ConnectionImplementation: Closing master protocol: MasterService
2023-02-24 17:34:50,537 INFO [main] zookeeper.ReadOnlyZKClient: Close zookeeper connection 0x1ce93c18 to hdp01:2181,hdp02:2181,hdp03:2181
Number of Tables: 1
2023-02-24 17:34:50,542 INFO [ReadOnlyZKClient-hdp01:2181,hdp02:2181,hdp03:2181@0x1ce93c18] zookeeper.ZooKeeper: Session: 0x20000018ff60005 closed
2023-02-24 17:34:50,542 INFO [ReadOnlyZKClient-hdp01:2181,hdp02:2181,hdp03:2181@0x1ce93c18-EventThread] zookeeper.ClientCnxn: EventThread shut down for session: 0x20000018ff60005
2023-02-24 17:34:50,550 INFO [main] util.HBaseFsck: Loading region directories from HDFS
2023-02-24 17:34:50,588 INFO [main] util.HBaseFsck: Loading region information from HDFS
2023-02-24 17:34:50,765 INFO [main] util.HBaseFsck: Checking and fixing region consistency
ERROR: Region { meta => user_profile,,1677228015439.69f3f6a477f90bdc138e31f08ee909d8., hdfs => hdfs://hdp01:8020/hbase/data/default/user_profile/69f3f6a477f90bdc138e31f08ee909d8, deployed => , replicaId => 0 } not deployed on any region server.
ERROR: Region { meta => user_profile,003155,1677228015439.690658266a0b11c87aada6935c91a1f7., hdfs => hdfs://hdp01:8020/hbase/data/default/user_profile/690658266a0b11c87aada6935c91a1f7, deployed => , replicaId => 0 } not deployed on any region server.
2023-02-24 17:34:50,804 INFO [main] util.HBaseFsck: Handling overlap merges in parallel. set hbasefsck.overlap.merge.parallel to false to run serially.
ERROR: There is a hole in the region chain between and . You need to create a new .regioninfo and region dir in hdfs to plug the hole.
ERROR: Found inconsistency in table user_profile
Summary:
Table user_profile is okay.
Number of regions: 0
Deployed on:
Table hbase:meta is okay.
Number of regions: 1
Deployed on: hdp02,16020,1677229361706
3 inconsistencies detected.
Status: INCONSISTENT
2023-02-24 17:34:50,881 INFO [main] zookeeper.ZooKeeper: Session: 0x3000001abaf0003 closed
2023-02-24 17:34:50,881 INFO [main-EventThread] zookeeper.ClientCnxn: EventThread shut down for session: 0x3000001abaf0003
2023-02-24 17:34:50,881 INFO [main] client.ConnectionImplementation: Closing master protocol: MasterService
2023-02-24 17:34:50,882 INFO [main] zookeeper.ReadOnlyZKClient: Close zookeeper connection 0x45fd9a4d to hdp01:2181,hdp02:2181,hdp03:2181
2023-02-24 17:34:50,887 INFO [ReadOnlyZKClient-hdp01:2181,hdp02:2181,hdp03:2181@0x45fd9a4d] zookeeper.ZooKeeper: Session: 0x3000001abaf0004 closed
2023-02-24 17:34:50,888 INFO [ReadOnlyZKClient-hdp01:2181,hdp02:2181,hdp03:2181@0x45fd9a4d-EventThread] zookeeper.ClientCnxn: EventThread shut down for session: 0x3000001abaf0004
复制代码
由上图可以看到,终极效果是:数据不一致(INCONSISTENT),同时还可以看到有一个 region hole的问题
对比操纵:
检测可以正常操纵的表:stu表的信息如下所示
[whybigdata@hdp01 hbase-2.0.5]$ hbase hbck 'stu'
2023-02-24 17:42:33,198 INFO [main] zookeeper.RecoverableZooKeeper: Process identifier=hbase Fsck connecting to ZooKeeper ensemble=hdp01:2181,hdp02:2181,hdp03:2181
2023-02-24 17:42:33,206 INFO [main] zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.10-39d3a4f269333c922ed3db283be479f9deacaa0f, built on 03/23/2017 10:13 GMT
2023-02-24 17:42:33,206 INFO [main] zookeeper.ZooKeeper: Client environment:host.name=hdp01
2023-02-24 17:42:33,206 INFO [main] zookeeper.ZooKeeper: Client environment:java.version=1.8.0_212
2023-02-24 17:42:33,206 INFO [main] zookeeper.ZooKeeper: Client environment:java.vendor=Oracle Corporation
2023-02-24 17:42:33,206 INFO [main] zookeeper.ZooKeeper: Client environment:java.home=/opt/apps/jdk1.8.0_212/jre
2023-02-24 17:42:33,207 INFO [main] zookeeper.ZooKeeper: /hadoop/yarn/hadoop-yarn-server-web-proxy-3.1.3.jar:/opt/apps/hadoop-3.1.3/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-3.1.3.jar:/opt/apps/hadoop-3.1.3/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-3.1.3.jar:/opt/apps/hadoop-3.1.3/share/hadoop/yarn/hadoop-yarn-api-3.1.3.jar:/opt/apps/hadoop-3.1.3/share/hadoop/yarn/hadoop-yarn-client-3.1.3.jar:/opt/apps/hadoop-3.1.3/share/hadoop/yarn/hadoop-yarn-common-3.1.3.jar:/opt/apps/hadoop-3.1.3/share/hadoop/yarn/hadoop-yarn-server-tests-3.1.3.jar:/opt/apps/hadoop-3.1.3/share/hadoop/yarn/hadoop-yarn-services-api-3.1.3.jar:/opt/apps/hadoop-3.1.3/share/hadoop/yarn/hadoop-yarn-services-core-3.1.3.jar:/opt/apps/hadoop-3.1.3/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-3.1.3.jar:/opt/apps/hadoop-3.1.3/share/hadoop/yarn/hadoop-yarn-server-common-3.1.3.jar:/opt/apps/hadoop-3.1.3/share/hadoop/yarn/hadoop-yarn-server-nodemanager-3.1.3.jar:/opt/apps/hadoop-3.1.3/share/hadoop/yarn/hadoop-yarn-server-router-3.1.3.jar
2023-02-24 17:42:33,207 INFO [main] zookeeper.ZooKeeper: Client environment:java.library.path=/opt/apps/hadoop-3.1.3/lib/native
2023-02-24 17:42:33,207 INFO [main] zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
2023-02-24 17:42:33,207 INFO [main] zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
2023-02-24 17:42:33,207 INFO [main] zookeeper.ZooKeeper: Client environment:os.name=Linux
2023-02-24 17:42:33,207 INFO [main] zookeeper.ZooKeeper: Client environment:os.arch=amd64
2023-02-24 17:42:33,207 INFO [main] zookeeper.ZooKeeper: Client environment:os.version=3.10.0-862.el7.x86_64
2023-02-24 17:42:33,207 INFO [main] zookeeper.ZooKeeper: Client environment:user.name=whybigdata
2023-02-24 17:42:33,207 INFO [main] zookeeper.ZooKeeper: Client environment:user.home=/home/whybigdata
2023-02-24 17:42:33,207 INFO [main] zookeeper.ZooKeeper: Client environment:user.dir=/opt/apps/hbase-2.0.5
2023-02-24 17:42:33,222 INFO [main-SendThread(hdp03:2181)] zookeeper.ClientCnxn: Opening socket connection to server hdp03/192.168.10.13:2181. Will not attempt to authenticate using SASL (unknown error)
Allow checking/fixes for table: stu
HBaseFsck command line options: stu
2023-02-24 17:42:33,226 INFO [main] util.HBaseFsck: Launching hbck
2023-02-24 17:42:33,227 INFO [main-SendThread(hdp03:2181)] zookeeper.ClientCnxn: Socket connection established to hdp03/192.168.10.13:2181, initiating session
2023-02-24 17:42:33,235 INFO [main-SendThread(hdp03:2181)] zookeeper.ClientCnxn: Session establishment complete on server hdp03/192.168.10.13:2181, sessionid = 0x40000034a3b0005, negotiated timeout = 40000
2023-02-24 17:42:33,281 INFO [main] zookeeper.ReadOnlyZKClient: Connect 0x45fd9a4d to hdp01:2181,hdp02:2181,hdp03:2181 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms
2023-02-24 17:42:33,288 INFO [ReadOnlyZKClient-hdp01:2181,hdp02:2181,hdp03:2181@0x45fd9a4d-SendThread(hdp02:2181)] zookeeper.ClientCnxn: Opening socket connection to server hdp02/192.168.10.12:2181. Will not attempt to authenticate using SASL (unknown error)
2023-02-24 17:42:33,289 INFO [ReadOnlyZKClient-hdp01:2181,hdp02:2181,hdp03:2181@0x45fd9a4d-SendThread(hdp02:2181)] zookeeper.ClientCnxn: Socket connection established to hdp02/192.168.10.12:2181, initiating session
2023-02-24 17:42:33,294 INFO [ReadOnlyZKClient-hdp01:2181,hdp02:2181,hdp03:2181@0x45fd9a4d-SendThread(hdp02:2181)] zookeeper.ClientCnxn: Session establishment complete on server hdp02/192.168.10.12:2181, sessionid = 0x3000001abaf0006, negotiated timeout = 40000
Version: 2.0.5
2023-02-24 17:42:33,788 INFO [main] util.HBaseFsck: Computing mapping of all store files
2023-02-24 17:42:34,024 INFO [main] util.HBaseFsck: Validating mapping using HDFS state
2023-02-24 17:42:34,025 INFO [main] util.HBaseFsck: Computing mapping of all link files
.
2023-02-24 17:42:34,075 INFO [main] util.HBaseFsck: Validating mapping using HDFS state
Number of live region servers: 3
Number of dead region servers: 0
Master: hdp01,16000,1677229359514
Number of backup masters: 0
Average load: 1.0
Number of requests: 97
Number of regions: 3
Number of regions in transition: 0
2023-02-24 17:42:34,168 INFO [main] util.HBaseFsck: Loading regionsinfo from the hbase:meta table
Number of empty REGIONINFO_QUALIFIER rows in hbase:meta: 0
2023-02-24 17:42:34,245 INFO [main] util.HBaseFsck: getTableDescriptors == tableNames => [stu]
2023-02-24 17:42:34,245 INFO [main] zookeeper.ReadOnlyZKClient: Connect 0x1ce93c18 to hdp01:2181,hdp02:2181,hdp03:2181 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms
2023-02-24 17:42:34,247 INFO [ReadOnlyZKClient-hdp01:2181,hdp02:2181,hdp03:2181@0x1ce93c18-SendThread(hdp02:2181)] zookeeper.ClientCnxn: Opening socket connection to server hdp02/192.168.10.12:2181. Will not attempt to authenticate using SASL (unknown error)
2023-02-24 17:42:34,248 INFO [ReadOnlyZKClient-hdp01:2181,hdp02:2181,hdp03:2181@0x1ce93c18-SendThread(hdp02:2181)] zookeeper.ClientCnxn: Socket connection established to hdp02/192.168.10.12:2181, initiating session
2023-02-24 17:42:34,252 INFO [ReadOnlyZKClient-hdp01:2181,hdp02:2181,hdp03:2181@0x1ce93c18-SendThread(hdp02:2181)] zookeeper.ClientCnxn: Session establishment complete on server hdp02/192.168.10.12:2181, sessionid = 0x3000001abaf0007, negotiated timeout = 40000
2023-02-24 17:42:34,264 INFO [main] client.ConnectionImplementation: Closing master protocol: MasterService
2023-02-24 17:42:34,264 INFO [main] zookeeper.ReadOnlyZKClient: Close zookeeper connection 0x1ce93c18 to hdp01:2181,hdp02:2181,hdp03:2181
Number of Tables: 1
2023-02-24 17:42:34,269 INFO [ReadOnlyZKClient-hdp01:2181,hdp02:2181,hdp03:2181@0x1ce93c18] zookeeper.ZooKeeper: Session: 0x3000001abaf0007 closed
2023-02-24 17:42:34,270 INFO [ReadOnlyZKClient-hdp01:2181,hdp02:2181,hdp03:2181@0x1ce93c18-EventThread] zookeeper.ClientCnxn: EventThread shut down for session: 0x3000001abaf0007
2023-02-24 17:42:34,273 INFO [main] util.HBaseFsck: Loading region directories from HDFS
2023-02-24 17:42:34,306 INFO [main] util.HBaseFsck: Loading region information from HDFS
2023-02-24 17:42:34,444 INFO [main] util.HBaseFsck: Checking and fixing region consistency
2023-02-24 17:42:34,471 INFO [main] util.HBaseFsck: Handling overlap merges in parallel. set hbasefsck.overlap.merge.parallel to false to run serially.
Summary:
Table hbase:meta is okay.
Number of regions: 1
Deployed on: hdp02,16020,1677229361706
Table stu is okay.
Number of regions: 1
Deployed on: hdp01,16020,1677229361937
0 inconsistencies detected.
Status: OK
2023-02-24 17:42:34,541 INFO [main] zookeeper.ZooKeeper: Session: 0x40000034a3b0005 closed
2023-02-24 17:42:34,542 INFO [main] client.ConnectionImplementation: Closing master protocol: MasterService
2023-02-24 17:42:34,541 INFO [main-EventThread] zookeeper.ClientCnxn: EventThread shut down for session: 0x40000034a3b0005
2023-02-24 17:42:34,542 INFO [main] zookeeper.ReadOnlyZKClient: Close zookeeper connection 0x45fd9a4d to hdp01:2181,hdp02:2181,hdp03:2181
2023-02-24 17:42:34,547 INFO [ReadOnlyZKClient-hdp01:2181,hdp02:2181,hdp03:2181@0x45fd9a4d] zookeeper.ZooKeeper: Session: 0x3000001abaf0006 closed
2023-02-24 17:42:34,547 INFO [ReadOnlyZKClient-hdp01:2181,hdp02:2181,hdp03:2181@0x45fd9a4d-EventThread] zookeeper.ClientCnxn: EventThread shut down for session: 0x3000001abaf0006
2023-02-24 18:17:24,321 INFO [main] zookeeper.RecoverableZooKeeper: Process identifier=hbase Fsck connecting to ZooKeeper ensemble=hdp01:2181,hdp02:2181,hdp03:2181
2023-02-24 18:17:24,328 INFO [main] zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.10-39d3a4f269333c922ed3db283be479f9deacaa0f, built on 03/23/2017 10:13 GMT
2023-02-24 18:17:24,329 INFO [main] zookeeper.ZooKeeper: Client environment:host.name=hdp01
2023-02-24 18:17:24,329 INFO [main] zookeeper.ZooKeeper: Client environment:java.version=1.8.0_212
2023-02-24 18:17:24,329 INFO [main] zookeeper.ZooKeeper: Client environment:java.vendor=Oracle Corporation
2023-02-24 18:17:24,329 INFO [main] zookeeper.ZooKeeper: Client environment:java.home=/opt/apps/jdk1.8.0_212/jre
2023-02-24 18:17:24,329 INFO [main] zookeeper.ZooKeeper: /hadoop/yarn/hadoop-yarn-server-web-proxy-3.1.3.jar:/opt/apps/hadoop-3.1.3/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-3.1.3.jar:/opt/apps/hadoop-3.1.3/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-3.1.3.jar:/opt/apps/hadoop-3.1.3/share/hadoop/yarn/hadoop-yarn-api-3.1.3.jar:/opt/apps/hadoop-3.1.3/share/hadoop/yarn/hadoop-yarn-client-3.1.3.jar:/opt/apps/hadoop-3.1.3/share/hadoop/yarn/hadoop-yarn-common-3.1.3.jar:/opt/apps/hadoop-3.1.3/share/hadoop/yarn/hadoop-yarn-server-tests-3.1.3.jar:/opt/apps/hadoop-3.1.3/share/hadoop/yarn/hadoop-yarn-services-api-3.1.3.jar:/opt/apps/hadoop-3.1.3/share/hadoop/yarn/hadoop-yarn-services-core-3.1.3.jar:/opt/apps/hadoop-3.1.3/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-3.1.3.jar:/opt/apps/hadoop-3.1.3/share/hadoop/yarn/hadoop-yarn-server-common-3.1.3.jar:/opt/apps/hadoop-3.1.3/share/hadoop/yarn/hadoop-yarn-server-nodemanager-3.1.3.jar:/opt/apps/hadoop-3.1.3/share/hadoop/yarn/hadoop-yarn-server-router-3.1.3.jar
2023-02-24 18:17:24,329 INFO [main] zookeeper.ZooKeeper: Client environment:java.library.path=/opt/apps/hadoop-3.1.3/lib/native
2023-02-24 18:17:24,329 INFO [main] zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
2023-02-24 18:17:24,329 INFO [main] zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
2023-02-24 18:17:24,329 INFO [main] zookeeper.ZooKeeper: Client environment:os.name=Linux
2023-02-24 18:17:24,329 INFO [main] zookeeper.ZooKeeper: Client environment:os.arch=amd64
2023-02-24 18:17:24,329 INFO [main] zookeeper.ZooKeeper: Client environment:os.version=3.10.0-862.el7.x86_64
2023-02-24 18:17:24,329 INFO [main] zookeeper.ZooKeeper: Client environment:user.name=whybigdata
2023-02-24 18:17:24,329 INFO [main] zookeeper.ZooKeeper: Client environment:user.home=/home/whybigdata
2023-02-24 18:17:24,329 INFO [main] zookeeper.ZooKeeper: Client environment:user.dir=/opt/apps/hbase-2.0.5
This option is deprecated, please use -fixAssignments instead.
2023-02-24 18:17:24,344 INFO [main-SendThread(hdp03:2181)] zookeeper.ClientCnxn: Opening socket connection to server hdp03/192.168.10.13:2181. Will not attempt to authenticate using SASL (unknown error)
Allow checking/fixes for table: user_profile
HBaseFsck command line options: -fix user_profile
2023-02-24 18:17:24,350 INFO [main-SendThread(hdp03:2181)] zookeeper.ClientCnxn: Socket connection established to hdp03/192.168.10.13:2181, initiating session
2023-02-24 18:17:24,360 INFO [main-SendThread(hdp03:2181)] zookeeper.ClientCnxn: Session establishment complete on server hdp03/192.168.10.13:2181, sessionid = 0x4000043ee360001, negotiated timeout = 40000
2023-02-24 18:17:24,462 INFO [pool-6-thread-1] util.HBaseFsck: Failed to create lock file hbase-hbck.lock, try=1 of 5
2023-02-24 18:17:24,665 INFO [pool-6-thread-1] util.HBaseFsck: Failed to create lock file hbase-hbck.lock, try=2 of 5
2023-02-24 18:17:25,069 INFO [pool-6-thread-1] util.HBaseFsck: Failed to create lock file hbase-hbck.lock, try=3 of 5
2023-02-24 18:17:25,873 INFO [pool-6-thread-1] util.HBaseFsck: Failed to create lock file hbase-hbck.lock, try=4 of 5
2023-02-24 18:17:27,476 INFO [pool-6-thread-1] util.HBaseFsck: Failed to create lock file hbase-hbck.lock, try=5 of 5
2023-02-24 18:17:30,677 ERROR [main] util.HBaseFsck: Another instance of hbck is fixing HBase, exiting this instance. [If you are sure no other instance is running, delete the lock file hdfs://hdp01:8020/hbase/.tmp/hbase-hbck.lock and rerun the tool]
Exception in thread "main" java.io.IOException: Duplicate hbck - Abort
at org.apache.hadoop.hbase.util.HBaseFsck.connect(HBaseFsck.java:555)
at org.apache.hadoop.hbase.util.HBaseFsck.exec(HBaseFsck.java:5105)
at org.apache.hadoop.hbase.util.HBaseFsck$HBaseFsckTool.run(HBaseFsck.java:4928)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:4916)
-fixReplication Deletes replication queues for removed peers
Metadata Repair options supported as of version 2.0: (expert features, use with caution!)
-fixVersionFile Try to fix missing hbase.version file in hdfs.
-fixReferenceFiles Try to offline lingering reference store files
-fixHFileLinks Try to offline lingering HFileLinks
-noHdfsChecking Don't load/check region info from HDFS. Assumes hbase:meta region info is good. Won't check/fix any HDFS issue, e.g. hole, orphan, or overlap