HBase有关ERROR: org.apache.hadoop.hbase.ipc.ServerNotRunningYetExcepti ...

大号在练葵花宝典  金牌会员 | 2024-11-21 02:06:11 | 显示全部楼层 | 阅读模式
打印 上一主题 下一主题
楼主

主题 900|帖子 900|积分 2700

HBase有关ERROR: org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet


  
写在前面



  • Linux版本：CentOS7.5
  • Hadoop版本：Hadoop-3.1.3
  • ZooKeeper版本：ZooKeeper3.5.7
  • HBase版本：HBase-2.0.5
  • 集群：完全分布式（三台节点：hdp01，hdp02，hdp03）
问题发现

实验场景

   模仿数据集导入到HBase，数据集为100w，但是由于磁盘空间不足，导致集群中的第三台节点hdp03的HBase被退出，hdp03节点被关闭，模仿数据失败。。。。。
  注意：hdp03因空间不足而被关机，重新启动并开启相应的服务，zk和hbase的进程都是完备的。这一点在下文中的「测试Error缘故原由」即可体现。
  



IDEA步伐实行报错如下

  1. Exception in thread "main" org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 20 actions: ConnectException: 20 times, servers with issues: hdp03,16020,1677226920150
  2.         at org.apache.hadoop.hbase.client.BatchErrors.makeException(BatchErrors.java:54)
  3.         at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.getErrors(AsyncRequestFutureImpl.java:1226)
  4.         at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:455)
  5.         at org.apache.hadoop.hbase.client.HTable.put(HTable.java:553)
  6.         at cn.whybigdata.dynamic_rule.datagen.UserProfileDataGen.main(UserProfileDataGen.java:59)
  8. Process finished with exit code 1
复制代码
  这就是由于磁盘空间不足导致的
  HBase Shell交互端报错展示

   查看HBase中的所有表
  –> 出现错误：ERROR: org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet
  1. hbase(main):001:0> list
  2. TABLE
  4. ERROR: org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet
  5.         at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2932)
  6.         at org.apache.hadoop.hbase.master.MasterRpcServices.isMasterRunning(MasterRpcServices.java:1084)
  7.         at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java)
  8.         at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
  9.         at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
  10.         at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
  11.         at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
  13. List all user tables in hbase. Optional regular expression parameter could
  14. be used to filter the output. Examples:
  16.   hbase> list
  17.   hbase> list 'abc.*'
  18.   hbase> list 'ns:abc.*'
  19.   hbase> list 'ns:.*'
  21. Took 8.9350 seconds
复制代码
解决方法



  • 进入到hadoop安装目录下的 bin目录，实行以下下令
  1. [whybigdata@hdp01 hbase-2.0.5]$ cd ../hadoop-3.1.3/bin/
  2. [whybigdata@hdp01 bin]$ pwd
  3. /opt/apps/hadoop-3.1.3/bin
复制代码



  • 移除hadoop的安全模式
  1. [whybigdata@hdp01 bin]$ ./hadoop dfsadmin -safemode leave
  2. WARNING: Use of this script to execute dfsadmin is deprecated.
  3. WARNING: Attempting to execute replacement "hdfs dfsadmin" instead.
  5. Safe mode is OFF
复制代码
  关闭Hadoop的安全模式之后，就可以list出HBase的表了，但是依旧不可以扫描scan大概查询表数据量等操纵
  

  • 其他操纵依旧不可行
  1. hbase(main):002:0> count 'user_profile'
  3. ERROR: org.apache.hadoop.hbase.NotServingRegionException: user_profile,,1677228015439.69f3f6a477f90bdc138e31f08ee909d8. is not online on hdp03,16020,1677229361798
  4.         at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:3272)
  5.         at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:3249)
  6.         at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:1414)
  7.         at org.apache.hadoop.hbase.regionserver.RSRpcServices.newRegionScanner(RSRpcServices.java:2948)
  8.         at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3285)
  9.         at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42002)
  10.         at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
  11.         at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
  12.         at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
  13.         at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
  15. Count the number of rows in a table.  Return value is the number of rows.
  16. This operation may take a LONG time (Run '$HADOOP_HOME/bin/hadoop jar
  17. hbase.jar rowcount' to run a counting mapreduce job). Current count is shown
  18. every 1000 rows by default. Count interval may be optionally specified. Scan
  19. caching is enabled on count scans by default. Default cache size is 10 rows.
  20. If your rows are small in size, you may want to increase this
  21. parameter. Examples:
  23. hbase> count 'ns1:t1'
  24. hbase> count 't1'
  25. hbase> count 't1', INTERVAL => 100000
  26. hbase> count 't1', CACHE => 1000
  27. hbase> count 't1', INTERVAL => 10, CACHE => 1000
  28. hbase> count 't1', FILTER => "
  29.     (QualifierFilter (>=, 'binary:xyz')) AND (TimestampsFilter ( 123, 456))"
  30. hbase> count 't1', COLUMNS => ['c1', 'c2'], STARTROW => 'abc', STOPROW => 'xyz'
  32. The same commands also can be run on a table reference. Suppose you had a reference
  33. t to table 't1', the corresponding commands would be:
  35. hbase> t.count
  36. hbase> t.count INTERVAL => 100000
  37. hbase> t.count CACHE => 1000
  38. hbase> t.count INTERVAL => 10, CACHE => 1000
  39. hbase> t.count FILTER => "
  40.     (QualifierFilter (>=, 'binary:xyz')) AND (TimestampsFilter ( 123, 456))"
  41. hbase> t.count COLUMNS => ['c1', 'c2'], STARTROW => 'abc', STOPROW => 'xyz'
  43. Took 8.8512 seconds
复制代码
  扫描表也是出现错误
  ERROR: org.apache.hadoop.hbase.NotServingRegionException:
  1. hbase(main):003:0>  scan 'user_profile',{LIMIT => 10}
  2. ROW                               COLUMN+CELL
  4. ERROR: org.apache.hadoop.hbase.NotServingRegionException: user_profile,,1677228015439.69f3f6a477f90bdc138e31f08ee909d8. is not online on hdp03,16020,1677229361798
  5.         at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:3272)
  6.         at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:3249)
  7.         at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:1414)
  8.         at org.apache.hadoop.hbase.regionserver.RSRpcServices.newRegionScanner(RSRpcServices.java:2948)
  9.         at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3285)
  10.         at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42002)
  11.         at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
  12.         at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
  13.         at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
  14.         at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
  16. Scan a table; pass table name and optionally a dictionary of scanner
  17. specifications.  Scanner specifications may include one or more of:
  18. TIMERANGE, FILTER, LIMIT, STARTROW, STOPROW, ROWPREFIXFILTER, TIMESTAMP,
  19. MAXLENGTH, COLUMNS, CACHE, RAW, VERSIONS, ALL_METRICS, METRICS,
  20. REGION_REPLICA_ID, ISOLATION_LEVEL, READ_TYPE, ALLOW_PARTIAL_RESULTS,
  21. BATCH or MAX_RESULT_SIZE
  23. If no columns are specified, all columns will be scanned.
  24. To scan all members of a column family, leave the qualifier empty as in
  25. 'col_family'.
  27. The filter can be specified in two ways:
  28. 1. Using a filterString - more information on this is available in the
  29. Filter Language document attached to the HBASE-4176 JIRA
  30. 2. Using the entire package name of the filter.
  32. If you wish to see metrics regarding the execution of the scan, the
  33. ALL_METRICS boolean should be set to true. Alternatively, if you would
  34. prefer to see only a subset of the metrics, the METRICS array can be
  35. defined to include the names of only the metrics you care about.
  37. Some examples:
  39.   hbase> scan 'hbase:meta'
  40.   hbase> scan 'hbase:meta', {COLUMNS => 'info:regioninfo'}
  41.   hbase> scan 'ns1:t1', {COLUMNS => ['c1', 'c2'], LIMIT => 10, STARTROW => 'xyz'}
  42.   hbase> scan 't1', {COLUMNS => ['c1', 'c2'], LIMIT => 10, STARTROW => 'xyz'}
  43.   hbase> scan 't1', {COLUMNS => 'c1', TIMERANGE => [1303668804000, 1303668904000]}
  44.   hbase> scan 't1', {REVERSED => true}
  45.   hbase> scan 't1', {ALL_METRICS => true}
  46.   hbase> scan 't1', {METRICS => ['RPC_RETRIES', 'ROWS_FILTERED']}
  47.   hbase> scan 't1', {ROWPREFIXFILTER => 'row2', FILTER => "
  48.     (QualifierFilter (>=, 'binary:xyz')) AND (TimestampsFilter ( 123, 456))"}
  49.   hbase> scan 't1', {FILTER =>
  50.     org.apache.hadoop.hbase.filter.ColumnPaginationFilter.new(1, 0)}
  51.   hbase> scan 't1', {CONSISTENCY => 'TIMELINE'}
  52.   hbase> scan 't1', {ISOLATION_LEVEL => 'READ_UNCOMMITTED'}
  53.   hbase> scan 't1', {MAX_RESULT_SIZE => 123456}
  54. For setting the Operation Attributes
  55.   hbase> scan 't1', { COLUMNS => ['c1', 'c2'], ATTRIBUTES => {'mykey' => 'myvalue'}}
  56.   hbase> scan 't1', { COLUMNS => ['c1', 'c2'], AUTHORIZATIONS => ['PRIVATE','SECRET']}
  57. For experts, there is an additional option -- CACHE_BLOCKS -- which
  58. switches block caching for the scanner on (true) or off (false).  By
  59. default it is enabled.  Examples:
  61.   hbase> scan 't1', {COLUMNS => ['c1', 'c2'], CACHE_BLOCKS => false}
  63. Also for experts, there is an advanced option -- RAW -- which instructs the
  64. scanner to return all cells (including delete markers and uncollected deleted
  65. cells). This option cannot be combined with requesting specific COLUMNS.
  66. Disabled by default.  Example:
  68.   hbase> scan 't1', {RAW => true, VERSIONS => 10}
  70. There is yet another option -- READ_TYPE -- which instructs the scanner to
  71. use a specific read type. Example:
  73.   hbase> scan 't1', {READ_TYPE => 'PREAD'}
  75. Besides the default 'toStringBinary' format, 'scan' supports custom formatting
  76. by column.  A user can define a FORMATTER by adding it to the column name in
  77. the scan specification.  The FORMATTER can be stipulated:
  79. 1. either as a org.apache.hadoop.hbase.util.Bytes method name (e.g, toInt, toString)
  80. 2. or as a custom class followed by method name: e.g. 'c(MyFormatterClass).format'.
  82. Example formatting cf:qualifier1 and cf:qualifier2 both as Integers:
  83.   hbase> scan 't1', {COLUMNS => ['cf:qualifier1:toInt',
  84.     'cf:qualifier2:c(org.apache.hadoop.hbase.util.Bytes).toInt'] }
  86. Note that you can specify a FORMATTER by column only (cf:qualifier). You can set a
  87. formatter for all columns (including, all key parts) using the "FORMATTER"
  88. and "FORMATTER_CLASS" options. The default "FORMATTER_CLASS" is
  89. "org.apache.hadoop.hbase.util.Bytes".
  91.   hbase> scan 't1', {FORMATTER => 'toString'}
  92.   hbase> scan 't1', {FORMATTER_CLASS => 'org.apache.hadoop.hbase.util.Bytes', FORMATTER => 'toString'}
  94. Scan can also be used directly from a table, by first getting a reference to a
  95. table, like such:
  97.   hbase> t = get_table 't'
  98.   hbase> t.scan
  100. Note in the above situation, you can still provide all the filtering, columns,
  101. options, etc as described above.
  104. Took 8.2657 seconds
复制代码
测试Error缘故原由

   新建一张表，来检测是否是表user_profile自己的问题，其实这里显而易见了，但是照旧操纵一下，稳妥些。
  

  • 创建新的表stu，成功创建并可以插入、查询数据
  1. hbase(main):004:0> create 'stu', 'info'
  2. Created table stu
  3. Took 4.3756 seconds
  4. => Hbase::Table - stu
  5. hbase(main):005:0> list
  6. TABLE
  7. stu
  8. user_profile
  9. 2 row(s)
  10. Took 0.0085 seconds
  11. => ["stu", "user_profile"]
  12. hbase(main):006:0> put 'stu', '1001', 'info:name', 'zhangsan'
  13. Took 0.1275 seconds
  14. hbase(main):007:0> scan 'stu'
  15. ROW                               COLUMN+CELL
  16. 1001                             column=info:name, timestamp=1677230857034, value=zhangsan
  17. 1 row(s)
  18. Took 0.0197 seconds
  19. hbase(main):008:0> count 'stu'
  20. 1 row(s)
  21. Took 0.0277 seconds
  22. => 1
复制代码
  证明是表user_profile自身的缘故原由；
    编写文章最后才发现，由于是表user_profile自身的缘故原由，以是我是不是可以直接把表user_profile删除，再新建一个同名表，然后再次模仿导入数据不就可以了？？？傻了！！！
  解决方案

   检测数据表user_profile的状态，控制台会陆续打印出集群的状态以及表的状态干系信息
  1. [whybigdata@hdp01 hbase-2.0.5]$ hbase hbck 'user_profile'
  2. 2023-02-24 17:34:49,265 INFO  [main] zookeeper.RecoverableZooKeeper: Process identifier=hbase Fsck connecting to ZooKeeper ensemble=hdp01:2181,hdp02:2181,hdp03:2181
  3. 2023-02-24 17:34:49,273 INFO  [main] zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.10-39d3a4f269333c922ed3db283be479f9deacaa0f, built on 03/23/2017 10:13 GMT
  4. 2023-02-24 17:34:49,273 INFO  [main] zookeeper.ZooKeeper: Client environment:host.name=hdp01
  5. 2023-02-24 17:34:49,273 INFO  [main] zookeeper.ZooKeeper: Client environment:java.version=1.8.0_212
  6. 2023-02-24 17:34:49,273 INFO  [main] zookeeper.ZooKeeper: Client environment:java.vendor=Oracle Corporation
  7. 2023-02-24 17:34:49,273 INFO  [main] zookeeper.ZooKeeper: Client environment:java.home=/opt/apps/jdk1.8.0_212/jre
  8. 2023-02-24 17:34:49,273 INFO  [main] zookeeper.ZooKeeper: /hadoop/yarn/hadoop-yarn-server-web-proxy-3.1.3.jar:/opt/apps/hadoop-3.1.3/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-3.1.3.jar:/opt/apps/hadoop-3.1.3/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-3.1.3.jar:/opt/apps/hadoop-3.1.3/share/hadoop/yarn/hadoop-yarn-api-3.1.3.jar:/opt/apps/hadoop-3.1.3/share/hadoop/yarn/hadoop-yarn-client-3.1.3.jar:/opt/apps/hadoop-3.1.3/share/hadoop/yarn/hadoop-yarn-common-3.1.3.jar:/opt/apps/hadoop-3.1.3/share/hadoop/yarn/hadoop-yarn-server-tests-3.1.3.jar:/opt/apps/hadoop-3.1.3/share/hadoop/yarn/hadoop-yarn-services-api-3.1.3.jar:/opt/apps/hadoop-3.1.3/share/hadoop/yarn/hadoop-yarn-services-core-3.1.3.jar:/opt/apps/hadoop-3.1.3/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-3.1.3.jar:/opt/apps/hadoop-3.1.3/share/hadoop/yarn/hadoop-yarn-server-common-3.1.3.jar:/opt/apps/hadoop-3.1.3/share/hadoop/yarn/hadoop-yarn-server-nodemanager-3.1.3.jar:/opt/apps/hadoop-3.1.3/share/hadoop/yarn/hadoop-yarn-server-router-3.1.3.jar
  9. 2023-02-24 17:34:49,274 INFO  [main] zookeeper.ZooKeeper: Client environment:java.library.path=/opt/apps/hadoop-3.1.3/lib/native
  10. 2023-02-24 17:34:49,274 INFO  [main] zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
  11. 2023-02-24 17:34:49,274 INFO  [main] zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
  12. 2023-02-24 17:34:49,274 INFO  [main] zookeeper.ZooKeeper: Client environment:os.name=Linux
  13. 2023-02-24 17:34:49,274 INFO  [main] zookeeper.ZooKeeper: Client environment:os.arch=amd64
  14. 2023-02-24 17:34:49,274 INFO  [main] zookeeper.ZooKeeper: Client environment:os.version=3.10.0-862.el7.x86_64
  15. 2023-02-24 17:34:49,274 INFO  [main] zookeeper.ZooKeeper: Client environment:user.name=whybigdata
  16. 2023-02-24 17:34:49,274 INFO  [main] zookeeper.ZooKeeper: Client environment:user.home=/home/whybigdata
  17. 2023-02-24 17:34:49,274 INFO  [main] zookeeper.ZooKeeper: Client environment:user.dir=/opt/apps/hbase-2.0.5
  18. 2023-02-24 17:34:49,275 INFO  [main] zookeeper.ZooKeeper: Initiating client connection, connectString=hdp01:2181,hdp02:2181,hdp03:2181 sessionTimeout=90000 watcher=org.apache.hadoop.hbase.zookeeper.PendingWatcher@7a362b6b
  19. 2023-02-24 17:34:49,290 INFO  [main-SendThread(hdp02:2181)] zookeeper.ClientCnxn: Opening socket connection to server hdp02/192.168.10.12:2181. Will not attempt to authenticate using SASL (unknown error)
  20. Allow checking/fixes for table: user_profile
  21. HBaseFsck command line options: user_profile
  22. 2023-02-24 17:34:49,294 INFO  [main] util.HBaseFsck: Launching hbck
  23. 2023-02-24 17:34:49,295 INFO  [main-SendThread(hdp02:2181)] zookeeper.ClientCnxn: Socket connection established to hdp02/192.168.10.12:2181, initiating session
  24. 2023-02-24 17:34:49,306 INFO  [main-SendThread(hdp02:2181)] zookeeper.ClientCnxn: Session establishment complete on server hdp02/192.168.10.12:2181, sessionid = 0x3000001abaf0003, negotiated timeout = 40000
  25. 2023-02-24 17:34:49,353 INFO  [main] zookeeper.ReadOnlyZKClient: Connect 0x45fd9a4d to hdp01:2181,hdp02:2181,hdp03:2181 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms
  26. 2023-02-24 17:34:49,359 INFO  [ReadOnlyZKClient-hdp01:2181,hdp02:2181,hdp03:2181@0x45fd9a4d] zookeeper.ZooKeeper: Initiating client connection, connectString=hdp01:2181,hdp02:2181,hdp03:2181 sessionTimeout=90000 watcher=org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$13/371452875@5321776b
  27. 2023-02-24 17:34:49,360 INFO  [ReadOnlyZKClient-hdp01:2181,hdp02:2181,hdp03:2181@0x45fd9a4d-SendThread(hdp02:2181)] zookeeper.ClientCnxn: Opening socket connection to server hdp02/192.168.10.12:2181. Will not attempt to authenticate using SASL (unknown error)
  28. 2023-02-24 17:34:49,361 INFO  [ReadOnlyZKClient-hdp01:2181,hdp02:2181,hdp03:2181@0x45fd9a4d-SendThread(hdp02:2181)] zookeeper.ClientCnxn: Socket connection established to hdp02/192.168.10.12:2181, initiating session
  29. 2023-02-24 17:34:49,365 INFO  [ReadOnlyZKClient-hdp01:2181,hdp02:2181,hdp03:2181@0x45fd9a4d-SendThread(hdp02:2181)] zookeeper.ClientCnxn: Session establishment complete on server hdp02/192.168.10.12:2181, sessionid = 0x3000001abaf0004, negotiated timeout = 40000
  30. Version: 2.0.5
  31. 2023-02-24 17:34:49,970 INFO  [main] util.HBaseFsck: Computing mapping of all store files
  33. 2023-02-24 17:34:50,240 INFO  [main] util.HBaseFsck: Validating mapping using HDFS state
  34. 2023-02-24 17:34:50,240 INFO  [main] util.HBaseFsck: Computing mapping of all link files
  35. .
  36. 2023-02-24 17:34:50,292 INFO  [main] util.HBaseFsck: Validating mapping using HDFS state
  37. Number of live region servers: 3
  38. Number of dead region servers: 0
  39. Master: hdp01,16000,1677229359514
  40. Number of backup masters: 0
  41. Average load: 1.0
  42. Number of requests: 76
  43. Number of regions: 3
  44. Number of regions in transition: 0
  45. 2023-02-24 17:34:50,406 INFO  [main] util.HBaseFsck: Loading regionsinfo from the hbase:meta table
  47. Number of empty REGIONINFO_QUALIFIER rows in hbase:meta: 0
  48. 2023-02-24 17:34:50,501 INFO  [main] util.HBaseFsck: getTableDescriptors == tableNames => [user_profile]
  49. 2023-02-24 17:34:50,502 INFO  [main] zookeeper.ReadOnlyZKClient: Connect 0x1ce93c18 to hdp01:2181,hdp02:2181,hdp03:2181 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms
  50. 2023-02-24 17:34:50,504 INFO  [ReadOnlyZKClient-hdp01:2181,hdp02:2181,hdp03:2181@0x1ce93c18] zookeeper.ZooKeeper: Initiating client connection, connectString=hdp01:2181,hdp02:2181,hdp03:2181 sessionTimeout=90000 watcher=org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$13/371452875@5321776b
  51. 2023-02-24 17:34:50,505 INFO  [ReadOnlyZKClient-hdp01:2181,hdp02:2181,hdp03:2181@0x1ce93c18-SendThread(hdp01:2181)] zookeeper.ClientCnxn: Opening socket connection to server hdp01/192.168.10.11:2181. Will not attempt to authenticate using SASL (unknown error)
  52. 2023-02-24 17:34:50,506 INFO  [ReadOnlyZKClient-hdp01:2181,hdp02:2181,hdp03:2181@0x1ce93c18-SendThread(hdp01:2181)] zookeeper.ClientCnxn: Socket connection established to hdp01/192.168.10.11:2181, initiating session
  53. 2023-02-24 17:34:50,521 INFO  [ReadOnlyZKClient-hdp01:2181,hdp02:2181,hdp03:2181@0x1ce93c18-SendThread(hdp01:2181)] zookeeper.ClientCnxn: Session establishment complete on server hdp01/192.168.10.11:2181, sessionid = 0x20000018ff60005, negotiated timeout = 40000
  54. 2023-02-24 17:34:50,537 INFO  [main] client.ConnectionImplementation: Closing master protocol: MasterService
  55. 2023-02-24 17:34:50,537 INFO  [main] zookeeper.ReadOnlyZKClient: Close zookeeper connection 0x1ce93c18 to hdp01:2181,hdp02:2181,hdp03:2181
  56. Number of Tables: 1
  57. 2023-02-24 17:34:50,542 INFO  [ReadOnlyZKClient-hdp01:2181,hdp02:2181,hdp03:2181@0x1ce93c18] zookeeper.ZooKeeper: Session: 0x20000018ff60005 closed
  58. 2023-02-24 17:34:50,542 INFO  [ReadOnlyZKClient-hdp01:2181,hdp02:2181,hdp03:2181@0x1ce93c18-EventThread] zookeeper.ClientCnxn: EventThread shut down for session: 0x20000018ff60005
  59. 2023-02-24 17:34:50,550 INFO  [main] util.HBaseFsck: Loading region directories from HDFS
  61. 2023-02-24 17:34:50,588 INFO  [main] util.HBaseFsck: Loading region information from HDFS
  62. 2023-02-24 17:34:50,638 INFO  [hbasefsck-pool1-t5] sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
  63. 2023-02-24 17:34:50,638 INFO  [hbasefsck-pool1-t9] sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
  64. 2023-02-24 17:34:50,638 INFO  [hbasefsck-pool1-t10] sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
  66. 2023-02-24 17:34:50,765 INFO  [main] util.HBaseFsck: Checking and fixing region consistency
  67. ERROR: Region { meta => user_profile,,1677228015439.69f3f6a477f90bdc138e31f08ee909d8., hdfs => hdfs://hdp01:8020/hbase/data/default/user_profile/69f3f6a477f90bdc138e31f08ee909d8, deployed => , replicaId => 0 } not deployed on any region server.
  68. ERROR: Region { meta => user_profile,003155,1677228015439.690658266a0b11c87aada6935c91a1f7., hdfs => hdfs://hdp01:8020/hbase/data/default/user_profile/690658266a0b11c87aada6935c91a1f7, deployed => , replicaId => 0 } not deployed on any region server.
  69. 2023-02-24 17:34:50,804 INFO  [main] util.HBaseFsck: Handling overlap merges in parallel. set hbasefsck.overlap.merge.parallel to false to run serially.
  70. ERROR: There is a hole in the region chain between  and .  You need to create a new .regioninfo and region dir in hdfs to plug the hole.
  71. ERROR: Found inconsistency in table user_profile
  72. Summary:
  73. Table user_profile is okay.
  74.     Number of regions: 0
  75.     Deployed on:
  76. Table hbase:meta is okay.
  77.     Number of regions: 1
  78.     Deployed on:  hdp02,16020,1677229361706
  79. 3 inconsistencies detected.
  80. Status: INCONSISTENT
  81. 2023-02-24 17:34:50,881 INFO  [main] zookeeper.ZooKeeper: Session: 0x3000001abaf0003 closed
  82. 2023-02-24 17:34:50,881 INFO  [main-EventThread] zookeeper.ClientCnxn: EventThread shut down for session: 0x3000001abaf0003
  83. 2023-02-24 17:34:50,881 INFO  [main] client.ConnectionImplementation: Closing master protocol: MasterService
  84. 2023-02-24 17:34:50,882 INFO  [main] zookeeper.ReadOnlyZKClient: Close zookeeper connection 0x45fd9a4d to hdp01:2181,hdp02:2181,hdp03:2181
  85. 2023-02-24 17:34:50,887 INFO  [ReadOnlyZKClient-hdp01:2181,hdp02:2181,hdp03:2181@0x45fd9a4d] zookeeper.ZooKeeper: Session: 0x3000001abaf0004 closed
  86. 2023-02-24 17:34:50,888 INFO  [ReadOnlyZKClient-hdp01:2181,hdp02:2181,hdp03:2181@0x45fd9a4d-EventThread] zookeeper.ClientCnxn: EventThread shut down for session: 0x3000001abaf0004
复制代码

   由上图可以看到，终极效果是：数据不一致（INCONSISTENT），同时还可以看到有一个 region hole的问题
  

  • 对比操纵：
   检测可以正常操纵的表：stu表的信息如下所示
  1. [whybigdata@hdp01 hbase-2.0.5]$ hbase hbck 'stu'
  2. 2023-02-24 17:42:33,198 INFO  [main] zookeeper.RecoverableZooKeeper: Process identifier=hbase Fsck connecting to ZooKeeper ensemble=hdp01:2181,hdp02:2181,hdp03:2181
  3. 2023-02-24 17:42:33,206 INFO  [main] zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.10-39d3a4f269333c922ed3db283be479f9deacaa0f, built on 03/23/2017 10:13 GMT
  4. 2023-02-24 17:42:33,206 INFO  [main] zookeeper.ZooKeeper: Client environment:host.name=hdp01
  5. 2023-02-24 17:42:33,206 INFO  [main] zookeeper.ZooKeeper: Client environment:java.version=1.8.0_212
  6. 2023-02-24 17:42:33,206 INFO  [main] zookeeper.ZooKeeper: Client environment:java.vendor=Oracle Corporation
  7. 2023-02-24 17:42:33,206 INFO  [main] zookeeper.ZooKeeper: Client environment:java.home=/opt/apps/jdk1.8.0_212/jre
  8. 2023-02-24 17:42:33,207 INFO  [main] zookeeper.ZooKeeper: /hadoop/yarn/hadoop-yarn-server-web-proxy-3.1.3.jar:/opt/apps/hadoop-3.1.3/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-3.1.3.jar:/opt/apps/hadoop-3.1.3/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-3.1.3.jar:/opt/apps/hadoop-3.1.3/share/hadoop/yarn/hadoop-yarn-api-3.1.3.jar:/opt/apps/hadoop-3.1.3/share/hadoop/yarn/hadoop-yarn-client-3.1.3.jar:/opt/apps/hadoop-3.1.3/share/hadoop/yarn/hadoop-yarn-common-3.1.3.jar:/opt/apps/hadoop-3.1.3/share/hadoop/yarn/hadoop-yarn-server-tests-3.1.3.jar:/opt/apps/hadoop-3.1.3/share/hadoop/yarn/hadoop-yarn-services-api-3.1.3.jar:/opt/apps/hadoop-3.1.3/share/hadoop/yarn/hadoop-yarn-services-core-3.1.3.jar:/opt/apps/hadoop-3.1.3/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-3.1.3.jar:/opt/apps/hadoop-3.1.3/share/hadoop/yarn/hadoop-yarn-server-common-3.1.3.jar:/opt/apps/hadoop-3.1.3/share/hadoop/yarn/hadoop-yarn-server-nodemanager-3.1.3.jar:/opt/apps/hadoop-3.1.3/share/hadoop/yarn/hadoop-yarn-server-router-3.1.3.jar
  9. 2023-02-24 17:42:33,207 INFO  [main] zookeeper.ZooKeeper: Client environment:java.library.path=/opt/apps/hadoop-3.1.3/lib/native
  10. 2023-02-24 17:42:33,207 INFO  [main] zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
  11. 2023-02-24 17:42:33,207 INFO  [main] zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
  12. 2023-02-24 17:42:33,207 INFO  [main] zookeeper.ZooKeeper: Client environment:os.name=Linux
  13. 2023-02-24 17:42:33,207 INFO  [main] zookeeper.ZooKeeper: Client environment:os.arch=amd64
  14. 2023-02-24 17:42:33,207 INFO  [main] zookeeper.ZooKeeper: Client environment:os.version=3.10.0-862.el7.x86_64
  15. 2023-02-24 17:42:33,207 INFO  [main] zookeeper.ZooKeeper: Client environment:user.name=whybigdata
  16. 2023-02-24 17:42:33,207 INFO  [main] zookeeper.ZooKeeper: Client environment:user.home=/home/whybigdata
  17. 2023-02-24 17:42:33,207 INFO  [main] zookeeper.ZooKeeper: Client environment:user.dir=/opt/apps/hbase-2.0.5
  18. 2023-02-24 17:42:33,208 INFO  [main] zookeeper.ZooKeeper: Initiating client connection, connectString=hdp01:2181,hdp02:2181,hdp03:2181 sessionTimeout=90000 watcher=org.apache.hadoop.hbase.zookeeper.PendingWatcher@7a362b6b
  19. 2023-02-24 17:42:33,222 INFO  [main-SendThread(hdp03:2181)] zookeeper.ClientCnxn: Opening socket connection to server hdp03/192.168.10.13:2181. Will not attempt to authenticate using SASL (unknown error)
  20. Allow checking/fixes for table: stu
  21. HBaseFsck command line options: stu
  22. 2023-02-24 17:42:33,226 INFO  [main] util.HBaseFsck: Launching hbck
  23. 2023-02-24 17:42:33,227 INFO  [main-SendThread(hdp03:2181)] zookeeper.ClientCnxn: Socket connection established to hdp03/192.168.10.13:2181, initiating session
  24. 2023-02-24 17:42:33,235 INFO  [main-SendThread(hdp03:2181)] zookeeper.ClientCnxn: Session establishment complete on server hdp03/192.168.10.13:2181, sessionid = 0x40000034a3b0005, negotiated timeout = 40000
  25. 2023-02-24 17:42:33,281 INFO  [main] zookeeper.ReadOnlyZKClient: Connect 0x45fd9a4d to hdp01:2181,hdp02:2181,hdp03:2181 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms
  26. 2023-02-24 17:42:33,286 INFO  [ReadOnlyZKClient-hdp01:2181,hdp02:2181,hdp03:2181@0x45fd9a4d] zookeeper.ZooKeeper: Initiating client connection, connectString=hdp01:2181,hdp02:2181,hdp03:2181 sessionTimeout=90000 watcher=org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$13/499703683@58ca425e
  27. 2023-02-24 17:42:33,288 INFO  [ReadOnlyZKClient-hdp01:2181,hdp02:2181,hdp03:2181@0x45fd9a4d-SendThread(hdp02:2181)] zookeeper.ClientCnxn: Opening socket connection to server hdp02/192.168.10.12:2181. Will not attempt to authenticate using SASL (unknown error)
  28. 2023-02-24 17:42:33,289 INFO  [ReadOnlyZKClient-hdp01:2181,hdp02:2181,hdp03:2181@0x45fd9a4d-SendThread(hdp02:2181)] zookeeper.ClientCnxn: Socket connection established to hdp02/192.168.10.12:2181, initiating session
  29. 2023-02-24 17:42:33,294 INFO  [ReadOnlyZKClient-hdp01:2181,hdp02:2181,hdp03:2181@0x45fd9a4d-SendThread(hdp02:2181)] zookeeper.ClientCnxn: Session establishment complete on server hdp02/192.168.10.12:2181, sessionid = 0x3000001abaf0006, negotiated timeout = 40000
  30. Version: 2.0.5
  31. 2023-02-24 17:42:33,788 INFO  [main] util.HBaseFsck: Computing mapping of all store files
  33. 2023-02-24 17:42:34,024 INFO  [main] util.HBaseFsck: Validating mapping using HDFS state
  34. 2023-02-24 17:42:34,025 INFO  [main] util.HBaseFsck: Computing mapping of all link files
  35. .
  36. 2023-02-24 17:42:34,075 INFO  [main] util.HBaseFsck: Validating mapping using HDFS state
  37. Number of live region servers: 3
  38. Number of dead region servers: 0
  39. Master: hdp01,16000,1677229359514
  40. Number of backup masters: 0
  41. Average load: 1.0
  42. Number of requests: 97
  43. Number of regions: 3
  44. Number of regions in transition: 0
  45. 2023-02-24 17:42:34,168 INFO  [main] util.HBaseFsck: Loading regionsinfo from the hbase:meta table
  47. Number of empty REGIONINFO_QUALIFIER rows in hbase:meta: 0
  48. 2023-02-24 17:42:34,245 INFO  [main] util.HBaseFsck: getTableDescriptors == tableNames => [stu]
  49. 2023-02-24 17:42:34,245 INFO  [main] zookeeper.ReadOnlyZKClient: Connect 0x1ce93c18 to hdp01:2181,hdp02:2181,hdp03:2181 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms
  50. 2023-02-24 17:42:34,246 INFO  [ReadOnlyZKClient-hdp01:2181,hdp02:2181,hdp03:2181@0x1ce93c18] zookeeper.ZooKeeper: Initiating client connection, connectString=hdp01:2181,hdp02:2181,hdp03:2181 sessionTimeout=90000 watcher=org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$13/499703683@58ca425e
  51. 2023-02-24 17:42:34,247 INFO  [ReadOnlyZKClient-hdp01:2181,hdp02:2181,hdp03:2181@0x1ce93c18-SendThread(hdp02:2181)] zookeeper.ClientCnxn: Opening socket connection to server hdp02/192.168.10.12:2181. Will not attempt to authenticate using SASL (unknown error)
  52. 2023-02-24 17:42:34,248 INFO  [ReadOnlyZKClient-hdp01:2181,hdp02:2181,hdp03:2181@0x1ce93c18-SendThread(hdp02:2181)] zookeeper.ClientCnxn: Socket connection established to hdp02/192.168.10.12:2181, initiating session
  53. 2023-02-24 17:42:34,252 INFO  [ReadOnlyZKClient-hdp01:2181,hdp02:2181,hdp03:2181@0x1ce93c18-SendThread(hdp02:2181)] zookeeper.ClientCnxn: Session establishment complete on server hdp02/192.168.10.12:2181, sessionid = 0x3000001abaf0007, negotiated timeout = 40000
  54. 2023-02-24 17:42:34,264 INFO  [main] client.ConnectionImplementation: Closing master protocol: MasterService
  55. 2023-02-24 17:42:34,264 INFO  [main] zookeeper.ReadOnlyZKClient: Close zookeeper connection 0x1ce93c18 to hdp01:2181,hdp02:2181,hdp03:2181
  56. Number of Tables: 1
  57. 2023-02-24 17:42:34,269 INFO  [ReadOnlyZKClient-hdp01:2181,hdp02:2181,hdp03:2181@0x1ce93c18] zookeeper.ZooKeeper: Session: 0x3000001abaf0007 closed
  58. 2023-02-24 17:42:34,270 INFO  [ReadOnlyZKClient-hdp01:2181,hdp02:2181,hdp03:2181@0x1ce93c18-EventThread] zookeeper.ClientCnxn: EventThread shut down for session: 0x3000001abaf0007
  59. 2023-02-24 17:42:34,273 INFO  [main] util.HBaseFsck: Loading region directories from HDFS
  61. 2023-02-24 17:42:34,306 INFO  [main] util.HBaseFsck: Loading region information from HDFS
  62. 2023-02-24 17:42:34,343 INFO  [hbasefsck-pool1-t1] sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
  63. 2023-02-24 17:42:34,343 INFO  [hbasefsck-pool1-t16] sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
  65. 2023-02-24 17:42:34,444 INFO  [main] util.HBaseFsck: Checking and fixing region consistency
  66. 2023-02-24 17:42:34,471 INFO  [main] util.HBaseFsck: Handling overlap merges in parallel. set hbasefsck.overlap.merge.parallel to false to run serially.
  67. Summary:
  68. Table hbase:meta is okay.
  69.     Number of regions: 1
  70.     Deployed on:  hdp02,16020,1677229361706
  71. Table stu is okay.
  72.     Number of regions: 1
  73.     Deployed on:  hdp01,16020,1677229361937
  74. 0 inconsistencies detected.
  75. Status: OK
  76. 2023-02-24 17:42:34,541 INFO  [main] zookeeper.ZooKeeper: Session: 0x40000034a3b0005 closed
  77. 2023-02-24 17:42:34,542 INFO  [main] client.ConnectionImplementation: Closing master protocol: MasterService
  78. 2023-02-24 17:42:34,541 INFO  [main-EventThread] zookeeper.ClientCnxn: EventThread shut down for session: 0x40000034a3b0005
  79. 2023-02-24 17:42:34,542 INFO  [main] zookeeper.ReadOnlyZKClient: Close zookeeper connection 0x45fd9a4d to hdp01:2181,hdp02:2181,hdp03:2181
  80. 2023-02-24 17:42:34,547 INFO  [ReadOnlyZKClient-hdp01:2181,hdp02:2181,hdp03:2181@0x45fd9a4d] zookeeper.ZooKeeper: Session: 0x3000001abaf0006 closed
  81. 2023-02-24 17:42:34,547 INFO  [ReadOnlyZKClient-hdp01:2181,hdp02:2181,hdp03:2181@0x45fd9a4d-EventThread] zookeeper.ClientCnxn: EventThread shut down for session: 0x3000001abaf0006
复制代码
  可以看到，正常表的检测效果的region servers个数正常且无死亡如下：
  

   以看到，正常表状态表现为OK
  



  • 实验修改表user_profile
  1. [whybigdata@hdp01 hbase-2.0.5]$ hbase hbck -fix "user_profile"
  2. 2023-02-24 18:17:24,321 INFO  [main] zookeeper.RecoverableZooKeeper: Process identifier=hbase Fsck connecting to ZooKeeper ensemble=hdp01:2181,hdp02:2181,hdp03:2181
  3. 2023-02-24 18:17:24,328 INFO  [main] zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.10-39d3a4f269333c922ed3db283be479f9deacaa0f, built on 03/23/2017 10:13 GMT
  4. 2023-02-24 18:17:24,329 INFO  [main] zookeeper.ZooKeeper: Client environment:host.name=hdp01
  5. 2023-02-24 18:17:24,329 INFO  [main] zookeeper.ZooKeeper: Client environment:java.version=1.8.0_212
  6. 2023-02-24 18:17:24,329 INFO  [main] zookeeper.ZooKeeper: Client environment:java.vendor=Oracle Corporation
  7. 2023-02-24 18:17:24,329 INFO  [main] zookeeper.ZooKeeper: Client environment:java.home=/opt/apps/jdk1.8.0_212/jre
  8. 2023-02-24 18:17:24,329 INFO  [main] zookeeper.ZooKeeper: /hadoop/yarn/hadoop-yarn-server-web-proxy-3.1.3.jar:/opt/apps/hadoop-3.1.3/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-3.1.3.jar:/opt/apps/hadoop-3.1.3/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-3.1.3.jar:/opt/apps/hadoop-3.1.3/share/hadoop/yarn/hadoop-yarn-api-3.1.3.jar:/opt/apps/hadoop-3.1.3/share/hadoop/yarn/hadoop-yarn-client-3.1.3.jar:/opt/apps/hadoop-3.1.3/share/hadoop/yarn/hadoop-yarn-common-3.1.3.jar:/opt/apps/hadoop-3.1.3/share/hadoop/yarn/hadoop-yarn-server-tests-3.1.3.jar:/opt/apps/hadoop-3.1.3/share/hadoop/yarn/hadoop-yarn-services-api-3.1.3.jar:/opt/apps/hadoop-3.1.3/share/hadoop/yarn/hadoop-yarn-services-core-3.1.3.jar:/opt/apps/hadoop-3.1.3/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-3.1.3.jar:/opt/apps/hadoop-3.1.3/share/hadoop/yarn/hadoop-yarn-server-common-3.1.3.jar:/opt/apps/hadoop-3.1.3/share/hadoop/yarn/hadoop-yarn-server-nodemanager-3.1.3.jar:/opt/apps/hadoop-3.1.3/share/hadoop/yarn/hadoop-yarn-server-router-3.1.3.jar
  9. 2023-02-24 18:17:24,329 INFO  [main] zookeeper.ZooKeeper: Client environment:java.library.path=/opt/apps/hadoop-3.1.3/lib/native
  10. 2023-02-24 18:17:24,329 INFO  [main] zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
  11. 2023-02-24 18:17:24,329 INFO  [main] zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
  12. 2023-02-24 18:17:24,329 INFO  [main] zookeeper.ZooKeeper: Client environment:os.name=Linux
  13. 2023-02-24 18:17:24,329 INFO  [main] zookeeper.ZooKeeper: Client environment:os.arch=amd64
  14. 2023-02-24 18:17:24,329 INFO  [main] zookeeper.ZooKeeper: Client environment:os.version=3.10.0-862.el7.x86_64
  15. 2023-02-24 18:17:24,329 INFO  [main] zookeeper.ZooKeeper: Client environment:user.name=whybigdata
  16. 2023-02-24 18:17:24,329 INFO  [main] zookeeper.ZooKeeper: Client environment:user.home=/home/whybigdata
  17. 2023-02-24 18:17:24,329 INFO  [main] zookeeper.ZooKeeper: Client environment:user.dir=/opt/apps/hbase-2.0.5
  18. 2023-02-24 18:17:24,330 INFO  [main] zookeeper.ZooKeeper: Initiating client connection, connectString=hdp01:2181,hdp02:2181,hdp03:2181 sessionTimeout=90000 watcher=org.apache.hadoop.hbase.zookeeper.PendingWatcher@7a362b6b
  19. This option is deprecated, please use  -fixAssignments instead.
  20. 2023-02-24 18:17:24,344 INFO  [main-SendThread(hdp03:2181)] zookeeper.ClientCnxn: Opening socket connection to server hdp03/192.168.10.13:2181. Will not attempt to authenticate using SASL (unknown error)
  21. Allow checking/fixes for table: user_profile
  22. HBaseFsck command line options: -fix user_profile
  23. 2023-02-24 18:17:24,350 INFO  [main-SendThread(hdp03:2181)] zookeeper.ClientCnxn: Socket connection established to hdp03/192.168.10.13:2181, initiating session
  24. 2023-02-24 18:17:24,360 INFO  [main-SendThread(hdp03:2181)] zookeeper.ClientCnxn: Session establishment complete on server hdp03/192.168.10.13:2181, sessionid = 0x4000043ee360001, negotiated timeout = 40000
  25. 2023-02-24 18:17:24,462 INFO  [pool-6-thread-1] util.HBaseFsck: Failed to create lock file hbase-hbck.lock, try=1 of 5
  26. 2023-02-24 18:17:24,665 INFO  [pool-6-thread-1] util.HBaseFsck: Failed to create lock file hbase-hbck.lock, try=2 of 5
  27. 2023-02-24 18:17:25,069 INFO  [pool-6-thread-1] util.HBaseFsck: Failed to create lock file hbase-hbck.lock, try=3 of 5
  28. 2023-02-24 18:17:25,873 INFO  [pool-6-thread-1] util.HBaseFsck: Failed to create lock file hbase-hbck.lock, try=4 of 5
  29. 2023-02-24 18:17:27,476 INFO  [pool-6-thread-1] util.HBaseFsck: Failed to create lock file hbase-hbck.lock, try=5 of 5
  30. 2023-02-24 18:17:30,677 ERROR [main] util.HBaseFsck: Another instance of hbck is fixing HBase, exiting this instance. [If you are sure no other instance is running, delete the lock file hdfs://hdp01:8020/hbase/.tmp/hbase-hbck.lock and rerun the tool]
  31. Exception in thread "main" java.io.IOException: Duplicate hbck - Abort
  32.         at org.apache.hadoop.hbase.util.HBaseFsck.connect(HBaseFsck.java:555)
  33.         at org.apache.hadoop.hbase.util.HBaseFsck.exec(HBaseFsck.java:5105)
  34.         at org.apache.hadoop.hbase.util.HBaseFsck$HBaseFsckTool.run(HBaseFsck.java:4928)
  35.         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
  36.         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
  37.         at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:4916)
复制代码

   可以看到错误：说是有另外一个运行的实例也在修复HBase，但是我是第一次实行修复操纵，理论上没有另一个运行的实例；按照其建议，删撤除指定的lock文件，即删除hdfs路径 /hbase/.tmp 下的 hbase-hbck.lock 文件，如下图所示：
  



  • 继续实行上述修复下令：
  1. [whybigdata@hdp01 hbase-2.0.5]$ hbase hbck -fix "user_profile"
  2. ......
  3. ERROR: option '-fix' is not supportted!
  5. -----------------------------------------------------------------------
  6. NOTE: As of HBase version 2.0, the hbck tool is significantly changed.
  7. In general, all Read-Only options are supported and can be be used
  8. safely. Most -fix/ -repair options are NOT supported. Please see usage
  9. below for details on which options are not supported.
  10. -----------------------------------------------------------------------
  11. Usage: fsck [opts] {only tables}
  12. where [opts] are:
  13.    -help Display help options (this)
  14.    -details Display full report of all regions.
  15.    -timelag <timeInSeconds>  Process only regions that  have not experienced any metadata updates in the last  <timeInSeconds> seconds.
  16.    -sleepBeforeRerun <timeInSeconds> Sleep this many seconds before checking if the fix worked if run with -fix
  17.    -summary Print only summary of the tables and status.
  18.    -metaonly Only check the state of the hbase:meta table.
  19.    -sidelineDir <hdfs://> HDFS path to backup existing meta.
  20.    -boundaries Verify that regions boundaries are the same between META and store files.
  21.    -exclusive Abort if another hbck is exclusive or fixing.
  23.   Datafile Repair options: (expert features, use with caution!)
  24.    -checkCorruptHFiles     Check all Hfiles by opening them to make sure they are valid
  25.    -sidelineCorruptHFiles  Quarantine corrupted HFiles.  implies -checkCorruptHFiles
  27. Replication options
  28.    -fixReplication   Deletes replication queues for removed peers
  30.   Metadata Repair options supported as of version 2.0: (expert features, use with caution!)
  31.    -fixVersionFile   Try to fix missing hbase.version file in hdfs.
  32.    -fixReferenceFiles  Try to offline lingering reference store files
  33.    -fixHFileLinks  Try to offline lingering HFileLinks
  34.    -noHdfsChecking   Don't load/check region info from HDFS. Assumes hbase:meta region info is good. Won't check/fix any HDFS issue, e.g. hole, orphan, or overlap
  35.    -ignorePreCheckPermission  ignore filesystem permission pre-check
  37. NOTE: Following options are NOT supported as of HBase version 2.0+.
  39.   UNSUPPORTED Metadata Repair options: (expert features, use with caution!)
  40.    -fix              Try to fix region assignments.  This is for backwards compatiblity
  41.    -fixAssignments   Try to fix region assignments.  Replaces the old -fix
  42.    -fixMeta          Try to fix meta problems.  This assumes HDFS region info is good.
  43.    -fixHdfsHoles     Try to fix region holes in hdfs.
  44.    -fixHdfsOrphans   Try to fix region dirs with no .regioninfo file in hdfs
  45.    -fixTableOrphans  Try to fix table dirs with no .tableinfo file in hdfs (online mode only)
  46.    -fixHdfsOverlaps  Try to fix region overlaps in hdfs.
  47.    -maxMerge <n>     When fixing region overlaps, allow at most <n> regions to merge. (n=5 by default)
  48.    -sidelineBigOverlaps  When fixing region overlaps, allow to sideline big overlaps
  49.    -maxOverlapsToSideline <n>  When fixing region overlaps, allow at most <n> regions to sideline per group. (n=2 by default)
  50.    -fixSplitParents  Try to force offline split parents to be online.
  51.    -removeParents    Try to offline and sideline lingering parents and keep daughter regions.
  52.    -fixEmptyMetaCells  Try to fix hbase:meta entries not referencing any region (empty REGIONINFO_QUALIFIER rows)
  54.   UNSUPPORTED Metadata Repair shortcuts
  55.    -repair           Shortcut for -fixAssignments -fixMeta -fixHdfsHoles -fixHdfsOrphans -fixHdfsOverlaps -fixVersionFile -sidelineBigOverlaps -fixReferenceFiles-fixHFileLinks
  56.    -repairHoles      Shortcut for -fixAssignments -fixMeta -fixHdfsHoles
  58. ......
复制代码
  可以看到， -fix选项在HBase2.x后不再支持
  

   HBase2.x支持的选项如下：
  

   查询了干系资料后，发现HBase2.x要利用修复功能必要利用得是hbck2，但是官方没有直接提供，必要自己去下载对应安装包的hbck2包，然后编译，选择自己必要的功能，才可以在HBase上利用。
  呃呃呃呃呃呃。。。。
  兜兜转转，照旧得卸载重装！！！
  鉴于本领有限，此处照旧之间卸载重装HBase吧
  彻底删除HBase数据



  • 毗连ZK，进入zk客户端
  1. [whybigdata@hadoop103 zookeeper-3.5.7]$ bin/zkCli.sh
复制代码


  • 查看当前znode的所包括的内容
  1. [zk: localhost:2181(CONNECTED) 4] ls -s /
  2. [admin, brokers, cluster, config, consumers, controller_epoch, hbase, isr_change_notification, latest_producer_id_block, log_dir_event_notification, zookeeper]cZxid = 0x0
  3. ctime = Thu Jan 01 08:00:00 CST 1970
  4. mZxid = 0x0
  5. mtime = Thu Jan 01 08:00:00 CST 1970
  6. pZxid = 0x600000002
  7. cversion = 19
  8. dataVersion = 0
  9. aclVersion = 0
  10. ephemeralOwner = 0x0
  11. dataLength = 0
  12. numChildren = 11
复制代码


  • 删除
  1. [zk: localhost:2181(CONNECTED) 5] rmr /hbase
  2. The command 'rmr' has been deprecated. Please use 'deleteall' instead.
  3. [zk: localhost:2181(CONNECTED) 6] ls -s /
  4. [admin, brokers, cluster, config, consumers, controller_epoch, isr_change_notification, latest_producer_id_block, log_dir_event_notification, zookeeper]cZxid = 0x0
  5. ctime = Thu Jan 01 08:00:00 CST 1970
  6. mZxid = 0x0
  7. mtime = Thu Jan 01 08:00:00 CST 1970
  8. pZxid = 0x800000396
  9. cversion = 20
  10. dataVersion = 0
  11. aclVersion = 0
  12. ephemeralOwner = 0x0
  13. dataLength = 0
  14. numChildren = 10
复制代码


  • 重启ZK、重启hadoop（hdfs、yarn）、重启hbase
   但是停止HBase集群，等待了好长时间，hdp01节点上的HMaster进程依旧存在，且停止过程并未中止。
  –> 我强制将HMaster进程给kill掉了（其实我也知道会有问题），果不其然，重启所有服务后，HBase的HMaster进程就观察不到了。害害害！！！
  卸载HBase

   卸载HBase前必要实行上述「彻底删除HBase数据」的步骤：
  

  • 删除hdfs上/hbase的目录（这个目录是在hbase-site.xml的参数hbase.rootdir设置的）
  • 删除zk上 /hbase的znode信息
  • 重启zk、hadoop集群
  • 删除三台节点的hbase安装目录
  • 重新解压安装hbase
  参考资料

https://www.playpi.org/2019101201.html
https://www.cnblogs.com/data-magnifier/p/15383318.html

免责声明：如果侵犯了您的权益，请联系站长，我们会及时删除侵权内容，谢谢合作！更多信息从访问主页：qidao123.com:ToB企服之家，中国第一个企服评测及商务社交产业平台。

本帖子中包含更多资源

您需要 登录 才可以下载或查看，没有账号？立即注册

x
回复

使用道具 举报

0 个回复

倒序浏览
返回列表

快速回复

高级模式
B Color Image Link Quote Code Smilies
您需要登录后才可以回帖 登录 or 立即注册

本版积分规则

大号在练葵花宝典

金牌会员
这个人很懒什么都没写!

楼主热帖

标签云

存储 挺好的 服务器
微信订阅号
微信服务号
微信客服
小程序
H5
关于我们 商务合作 网站地图
©Copyright 2022-2024 杭州祥升科技有限公司 版权所有 浙ICP备20004199号
快速回复 返回顶部 返回列表