Remove a region from HBase

Remove a region from HBase

Recently one of our hbase experienced outrage due to disk space full. Adding new nodes and balancing the cluster takes time.

After hdfs balancing with :

sudo -u hdfs hdfs dfsadmin -setBalancerBandwidth 65536000

sudo -u hdfs hdfs balancer -threshold 5

It gets back and functional. But when it runs MapReduce job from snapshot, it produces duplicate records in output. 

After digging into it a bit, I realized there are regions in HBase that are not online (shown in HBase UI), but since they are still residing in meta table and on the disk, the MapReduce job still running against them.

To remove those regions from HBase :

Find those regions from hbase:meta table

echo "scan 'hbase:meta'"| hbase shell &> dump.txt

And remove them:

deleteall 'hbase:meta', 'The name of the region'
Then run "hbase hbck PlatformData -repair" a couple times, until regions' data consistent.
Also find and remove the regions' file from hdfs :
sudo -u hdfs hadoop fs -rmr /hbase/data/default/user_profile/<regions hash>
Then the MapReduce job is good to go.